SambaNova Exporter Reference - SambaNova Documentation

The SambaNova exporter (sambanova_exporter) publishes RDU telemetry from each SambaRack host module: platform health, and tile, DDR, HBM, and PCIe performance. Unlike the PDU and XRDU exporters, it runs on the host it monitors, and Prometheus scrapes each host directly.

Property	Value
Port	9101 by default. Set `http_port` in the configuration file.
Runs as	A systemd service, `sambanova_exporter`, on each host module
Installed by	The `sambanova-runtime-exporter` package, with SambaNova Runtime. The Helm chart does not install it.
Default state	Disabled. Enable and start it after installing or upgrading the package.
Requires	The host module up, the package installed, and the service active. The chart only makes it scrapeable; it cannot start it.
Configuration file	`/opt/sambaflow/etc/sambanova_exporter.conf`, written by the package.

Install and run the service

For the full procedure in context, see Install monitoring, Step 1. As a privileged user, on each host module:

yum install -y sambanova-runtime-exporter
systemctl enable sambanova_exporter
systemctl start sambanova_exporter

Verify. The service must report active and the metrics endpoint must return 200:

systemctl is-active sambanova_exporter
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9101/metrics

To stop and disable it:

systemctl stop sambanova_exporter
systemctl disable sambanova_exporter

To use a different configuration file, run systemctl edit sambanova_exporter.service and set the SAMBANOVA_EXPORTER_CONFIG_FILE environment variable. The command line option -c <file> overrides the variable. If the file is not found, the exporter ignores it. If the file exists but cannot be read or parsed, the exporter fails.

How the chart exposes it

The chart does not deploy the exporter. It makes the host port scrapeable:

A placeholder DaemonSet on the host network gives a Kubernetes Service one endpoint per RDU node. Its readiness probe follows the host service, so a dead exporter leaves the endpoint list.
A ServiceMonitor scrapes the Service and adds two labels the exporter does not publish: node, from the node name, and rack, from the rackMap value you declare. Rack is not discoverable, so without rackMap the series have node but no rack.

sambanova-exporter:
  enabled: true
  serviceMonitor:
    enabled: true
    interval: 15s
    rackMap:
      <node-name>: "<rack-id>"

The chart deploys the placeholder DaemonSet by default, but serviceMonitor.enabled defaults to false. Set it to true, or Prometheus does not scrape RDU metrics.

Collectors

Each collector can be turned on or off independently. All are on by default, and the HBM collector starts only on platforms that include HBM.

Collector	Publishes	Enable setting
Platform	Platform info, timestamps, faults, errors	`enable_platform_collector`
Tile	Tile frequency, state percentages, owning process	`enable_tile_collector`
DDR	DDR channel request and byte rates	`enable_ddr_collector`
HBM	HBM channel-group request and byte rates	`enable_hbm_collector`
PCIe	PCIe request and byte rates	`enable_pcie_collector`

Metrics

Platform collector

Series with no labels below have one entry per platform.

Metric	Labels	Description
`sambanova_platform_info`	`platform_name`, `runtime_version`, `samba_runtime_version`, `bmc_version`, `rduc_version`, `pef_version`, `snml_version`	Platform identity and component versions. Always `1`.
`sambanova_platform_boot_timestamp_seconds`	none	When the system booted, in seconds since the Unix epoch
`sambanova_platform_snd_active_timestamp_seconds`	none	When `snd.service` last became active. `0.0` if it has not yet, `-1.0` if it cannot be determined.
`sambanova_platform_snd_inactive_timestamp_seconds`	none	When `snd.service` last became inactive. `0.0` if it has not yet, `-1.0` if it cannot be determined.
`sambanova_platform_fault_count`	none	Component faults reported by SambaNova system management
`sambanova_platform_error_count`	none	Component errors reported by SambaNova system management
`sambanova_platform_data_timestamp_seconds`	none	Time of the last successful monitoring sample
`sambanova_platform_collect_seconds`	none	Elapsed seconds of the latest collection

A non-zero sambanova_platform_error_count is normal. Watch how fast it grows, not its value.

Tile collector

Labels: xrdu, rdu, tile.

Metric	Description
`sambanova_tile_frequency_mhz`	Clock frequency of the tile
`sambanova_tile_idle_percent`	Percent of time the tile was idle
`sambanova_tile_pload_percent`	Percent of time in Program Load state
`sambanova_tile_exec_percent`	Percent of time in Execute state
`sambanova_tile_aload_percent`	Percent of time in Argument Load state
`sambanova_tile_pid`	Process ID of the controlling process, if the tile is allocated

Series with one entry per platform:

Metric	Description
`sambanova_tile_offline_total`	Monitored tiles that were not present and online, or did not respond, in the last sample
`sambanova_tile_data_timestamp_seconds`	Time of the last successful sample
`sambanova_tile_collect_seconds`	Elapsed seconds of the latest collection

To get RDU utilization across a rack, subtract the mean idle percentage from 100, and filter out the sentinel value: 100 - avg by (rack) (sambanova_tile_idle_percent != -1).

DDR collector

Labels: xrdu, rdu, ddrch.

Metric	Description
`sambanova_ddrch_rd_requests`	DDR channel read requests per second
`sambanova_ddrch_wr_requests`	DDR channel write requests per second
`sambanova_ddrch_rdB_bytes`	DDR channel read bytes per second
`sambanova_ddrch_wrB_bytes`	DDR channel write bytes per second

Series with one entry per platform:

Metric	Description
`sambanova_ddrch_offline_total`	Monitored components that were not present and online, or did not respond, in the last sample
`sambanova_ddr_data_timestamp_seconds`	Time of the last successful sample
`sambanova_ddr_collect_seconds`	Elapsed seconds of the latest collection

Throughput metrics report -1.0 for a component that was offline. See Sentinel value.

HBM collector

Available on platforms that include HBM. Labels: xrdu, rdu, hbm, hbmch_grp.

Metric	Description
`sambanova_hbmch_grp_rd_requests`	HBM channel-group read requests per second
`sambanova_hbmch_grp_wr_requests`	HBM channel-group write requests per second
`sambanova_hbmch_grp_rdB_bytes`	HBM channel-group read bytes per second
`sambanova_hbmch_grp_wrB_bytes`	HBM channel-group write bytes per second

Series with one entry per platform:

Metric	Description
`sambanova_hbmch_grp_offline_total`	Monitored components that were not present and online, or did not respond, in the last sample
`sambanova_hbm_data_timestamp_seconds`	Time of the last successful sample
`sambanova_hbm_collect_seconds`	Elapsed seconds of the latest collection

Throughput metrics report -1.0 for a component that was offline. See Sentinel value.

PCIe collector

Labels: xrdu, rdu, pcie. D2R and R2D name who starts the operation. D2R is a request from the local RDU (the device) to a remote RDU or host. R2D is a request from a remote host or RDU to the local RDU.

Direction of data	Metric	Description
Remote to local RDU	`sambanova_pcie_d2r_rd_requests`	D2R read requests per second
	`sambanova_pcie_d2r_rdB_bytes`	D2R read bytes per second
	`sambanova_pcie_r2d_wr_requests`	R2D write requests per second
	`sambanova_pcie_r2d_wrB_bytes`	R2D write bytes per second
Local RDU to remote	`sambanova_pcie_d2r_wr_requests`	D2R write requests per second
	`sambanova_pcie_d2r_wrB_bytes`	D2R write bytes per second
	`sambanova_pcie_r2d_rd_requests`	R2D read requests per second
	`sambanova_pcie_r2d_rdB_bytes`	R2D read bytes per second

Series with one entry per platform:

Metric	Description
`sambanova_pcie_offline_total`	Monitored components that were not present and online, or did not respond, in the last sample
`sambanova_pcie_data_timestamp_seconds`	Time of the last successful sample
`sambanova_pcie_collect_seconds`	Elapsed seconds of the latest collection

Throughput metrics report -1.0 for a component that was offline. See Sentinel value.

Labels

Label	Values	Description
`xrdu`	`XRDU_0` to `XRDU_7`	The host’s own index of the XRDU, in PCIe enumeration order. Not the same as the `xrdu` label on `xrdu_*` series. See XRDU exporter.
`rdu`	An index	RDU number within the XRDU
`tile`	An index	Tile within the RDU
`ddrch`	An index	DDR channel
`hbm`, `hbmch_grp`	An index	HBM stack and channel group
`pcie`	An index	PCIe interface
`platform_name`, `*_version`	Identity strings	On `sambanova_platform_info` only
`node`	A Kubernetes node name	Added by the chart’s ServiceMonitor
`rack`	Your rack ID	Added by the chart’s ServiceMonitor, from `rackMap`. Absent if you do not set `rackMap`.
`instance`	`IP:9101`	Added by Prometheus. There is no hostname label, so use `sambanova_platform_info` to map an IP to a machine.

Sentinel value

All throughput and tile metrics (except sambanova_tile_pid) report -1.0 for any XRDU, RDU, or component that was not present and online, or did not respond, at the last sample. avg includes -1.0 and min returns it, so filter it out in every aggregation:

avg by (node) (sambanova_tile_exec_percent != -1)

sambanova_tile_pid lists a tile only if it was allocated to a process at the last sample.

Configuration

The file is in INI format: [section] headers followed by option = value lines. Section names are case sensitive. Option names and values are not. Leave a value empty to take the default.