> ## Documentation Index
> Fetch the complete documentation index at: https://sambanova-systems.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring metrics reference

<Note>
  It is recommended to configure Prometheus to scrape these exporters every 60 seconds. A 30–60 second interval gives you timely visibility into hardware health without excessive storage overhead.
</Note>

***

### Metric labels

Every metric carries labels that let you filter and group by hardware topology. Labels come from:

* **Kubernetes metadata** – `namespace`, `job` (ServiceMonitor name)
* **Your `values.yaml`** – `node`, `xrdu`, `rack`, `group`, and any additional custom fields you add to target entries (Manual discovery mode only)

***

## SambaNova Exporter

Installed on each host via the SambaRack runtime RPM. The DaemonSet is a thin wrapper that makes the host-side exporter discoverable by Prometheus.

<Note>
  Already running on your nodes if SambaStack is installed.
</Note>

### Metrics

| Metric                           | Type  | Labels                            | Description                                                                                                   |
| :------------------------------- | :---- | :-------------------------------- | :------------------------------------------------------------------------------------------------------------ |
| `sambanova_tile_aload_percent`   | Gauge | `instance`, `rdu`, `tile`, `xrdu` | Arithmetic load percentage per RDU tile                                                                       |
| `sambanova_tile_exec_percent`    | Gauge | `instance`, `rdu`, `tile`, `xrdu` | Execution time percentage – how much of the time the tile is actively computing                               |
| `sambanova_tile_frequency_mhz`   | Gauge | `instance`, `rdu`, `tile`, `xrdu` | Current operating frequency of each tile                                                                      |
| `sambanova_platform_fault_count` | Gauge | `instance`, `namespace`           | Cumulative fault count for the SambaNova platform                                                             |
| `sambanova_platform_error_count` | Gauge | `instance`, `namespace`           | Cumulative error count                                                                                        |
| `sambanova_ddrch_rdB_bytes`      | Gauge | `instance`, `rdu`                 | Cumulative DDR channel bytes read – use `rate(sambanova_ddrch_rdB_bytes[5m])` to get read bandwidth           |
| `sambanova_hbmch_grp_rdB_bytes`  | Gauge | `instance`, `rdu`                 | Cumulative HBM channel group bytes read – use `rate(sambanova_hbmch_grp_rdB_bytes[5m])` to get read bandwidth |

***

## Host BMC IPMI Exporter

Runs on every SambaRack node and reads IPMI data from the host BMC via the `/dev/ipmi0` device.

<Note>
  Requires the `ipmi_devintf` kernel module to be loaded. Connects to the host BMC on each node – no additional IP configuration needed.
</Note>

### Metrics

| Metric                              | Type  | Labels                                  | Description                                                                              |
| :---------------------------------- | :---- | :-------------------------------------- | :--------------------------------------------------------------------------------------- |
| `ipmi_up`                           | Gauge | `instance`, `namespace`                 | `1` if the exporter is successfully reaching the BMC, `0` otherwise                      |
| `ipmi_power_watts`                  | Gauge | `instance`, `namespace`                 | Instantaneous host server power draw                                                     |
| `ipmi_dcmi_power_consumption_watts` | Gauge | `instance`, `namespace`                 | Platform-level power via DCMI (more reliable single value per host)                      |
| `ipmi_temperature_celsius`          | Gauge | `instance`, `namespace`, `name`         | Temperature sensor reading; `name` identifies the sensor (e.g. `Inlet Temp`, `CPU Temp`) |
| `ipmi_fan_speed_rpm`                | Gauge | `instance`, `namespace`, `name`         | Fan speed in RPM; `name` identifies the fan                                              |
| `ipmi_voltage_volts`                | Gauge | `instance`, `namespace`, `name`         | Voltage sensor reading                                                                   |
| `ipmi_sensor_state`                 | Gauge | `instance`, `namespace`, `name`, `type` | Sensor health state (`0` = ok, `1` = warning, `2` = critical)                            |

***

## XRDU BMC Exporter

Connects out from each cluster node to the XRDU BMC management IPs configured in `values.yaml`. Each pod scrapes only the XRDUs assigned to its own node.

<Note>
  Verify XRDU BMC IPs with `snctl node show <node-name>`.
</Note>

<Warning>
  `xrdu_bmc_temp_celsisus` is the actual metric name as emitted by the exporter – the double-s spelling is in the exporter source. Use it verbatim in your queries.
</Warning>

### Metrics

| Metric                         | Type  | Labels                                                  | Description                                                                            |
| :----------------------------- | :---- | :------------------------------------------------------ | :------------------------------------------------------------------------------------- |
| `xrdu_bmc_power_watts`         | Gauge | `instance`, `target`, `xrdu`, `rack`, `group`, `sensor` | XRDU power reading per sensor; `sensor` identifies the power supply (e.g. `ps1_input`) |
| `xrdu_bmc_temp_celsisus`       | Gauge | `instance`, `target`, `xrdu`, `rack`, `sensor`          | XRDU component temperature                                                             |
| `xrdu_bmc_fan_rpms`            | Gauge | `instance`, `target`, `xrdu`, `rack`, `sensor`          | XRDU fan speed in RPM                                                                  |
| `xrdu_bmc_vddc_volts`          | Gauge | `instance`, `target`, `xrdu`, `rack`, `sensor`          | XRDU voltage                                                                           |
| `xrdu_bmc_vddc_amps`           | Gauge | `instance`, `target`, `xrdu`, `rack`, `sensor`          | XRDU current in amps                                                                   |
| `xrdu_bmc_cpu_utilization`     | Gauge | `instance`, `target`, `xrdu`, `rack`                    | XRDU BMC CPU utilization                                                               |
| `xrdu_bmc_memory_utilization`  | Gauge | `instance`, `target`, `xrdu`, `rack`                    | XRDU BMC memory utilization                                                            |
| `xrdu_bmc_storage_utilization` | Gauge | `instance`, `target`, `xrdu`, `rack`                    | XRDU BMC storage utilization                                                           |

***

## What's next

<Card title="Building dashboards in Grafana" href="./building-dashboards" icon="chart-line">
  Use these metrics to build a complete hardware monitoring dashboard with topology drill-down
</Card>