Hardware Monitoring Install and Verification

Step 1 – Set up your environment

Prepare these environment variables before proceeding with the installation. Create a .env file to hold your environment-specific values and source it before running any commands in this guide.

.env

NAMESPACE=monitoring
REGISTRY=<registry-url>                  # provided by SambaNova
REGISTRY_SERVER=us-west2-docker.pkg.dev  # SambaNova artifact registry; confirm with your rep if different
GCP_KEY_PATH=/path/to/gcp-key.json       # path to your SambaNova-provided key file
CHART_VERSION=<chart-version>            # provided by SambaNova
XRDU_USERNAME=<xrdu-bmc-username>        # read-only BMC user (see Prerequisites)
XRDU_PASSWORD=<xrdu-bmc-password>        # read-only BMC password

source .env

The .env file and GCP key file you create during installation contain credentials. Restrict permissions after creating them:

chmod 600 .env
chmod 600 /path/to/gcp-key.json
echo ".env" >> .gitignore

Step 2 – Create the monitoring namespace

kubectl create namespace $NAMESPACE

If the namespace already exists, this command will return an error – that is fine, continue to Step 3.

Step 3 – Create the image pull secret

This secret allows Kubernetes to pull the exporter images from the SambaNova registry using the service account key file SambaNova provided.

kubectl create secret docker-registry sn-registry-creds \
  --docker-server=$REGISTRY_SERVER \
  --docker-username=_json_key \
  --docker-password="$(cat "$GCP_KEY_PATH")" \
  --namespace=$NAMESPACE

Step 4 – Create the XRDU credentials secret

This secret holds the read-only BMC credentials the XRDU exporter uses to authenticate when scraping each XRDU BMC.

kubectl create secret generic sn-xrdu-creds \
  --from-literal=XRDU_USERNAME=$XRDU_USERNAME \
  --from-literal=XRDU_PASSWORD=$XRDU_PASSWORD \
  --namespace=$NAMESPACE

Step 5 – Authenticate to the OCI registry

gcloud auth print-access-token | helm registry login $REGISTRY_SERVER \
  --username=oauth2accesstoken \
  --password-stdin

Step 6 – Prepare your values override file

The chart needs to know how to discover your XRDU BMCs. Choose the mode that matches your environment. Example files are included in the Helm chart package at examples/discovery.yaml (Auto Discovery tab) and examples/manual.yaml (Manual tab).

Auto Discovery
Manual

Auto discover based on node hostname and XRDU index. The chart generates candidate hostnames from the pattern and pings each one to discover active XRDUs.Use this mode if your cluster uses DNS-based XRDU hostname resolution – where XRDU BMC hostnames follow a predictable DNS pattern.

values.yaml

sn-xrdu-exporter:
  existingSecret: sn-xrdu-creds     # the secret created in Step 4
  autoXrduDiscovery: true
  serviceMonitor:
    enabled: true
  discovery:
    pattern: "$HOSTNAME-xrdu$I-sp"  # chart template variables, not shell variables: $HOSTNAME = node hostname, $I = XRDU index
    indexRange:
      start: 0
      end: 7                        # adjust to match the number of XRDUs per node (standard rack has 8)
    pingTimeoutSeconds: 1

sambanova-exporter:
  serviceMonitor:
    enabled: true

ipmi-exporter:
  serviceMonitor:
    enabled: true

Use this mode to manually specify every XRDU BMC IP and the node it belongs to.Use this mode when your environment does not use the DNS naming convention above – for example, most standard on-premises customer deployments.

values.yaml

sn-xrdu-exporter:
  existingSecret: sn-xrdu-creds     # the secret created in Step 4
  autoXrduDiscovery: false
  mountHostEtcHosts: false          # set true only if targets use hostnames (not IPs) that are in /etc/hosts
  serviceMonitor:
    enabled: true
  xrdu:
    targets:
      # One entry per XRDU BMC. Use `snctl node show <node>` to get these values.
      - node: <kubernetes-node-name>   # exact match of `kubectl get nodes` NAME column – must align with the node name in snctl
        xrdu_ip: <xrdu-bmc-ip>        # use xrdu.hostname from `snctl node show <node-name>` for this specific XRDU
        xrdu: <xrdu-label>            # human-readable label, e.g. rack1-host1-xrdu0
        rack: <rack-id>               # must align with the rack name in snctl, e.g. rack-01
        group: <group-id>             # must align with the group name in snctl, e.g. g1
      # Add one entry for every XRDU in your deployment

sambanova-exporter:
  serviceMonitor:
    enabled: true

ipmi-exporter:
  serviceMonitor:
    enabled: true

Filling in the target fields:

Field	Where it comes from	Example
`node`	`kubectl get nodes` – use the exact value in the `NAME` column	`rack1-host1`
`xrdu_ip`	`snctl node show <node-name>` – use the `xrdu.hostname` field for this specific XRDU	`192.168.6.1`
`xrdu`	Your choice – becomes a Prometheus label	`rack1-host1-xrdu0`
`rack`	Align with the rack name in snctl – becomes a Prometheus label	`rack-01`
`group`	Align with the group name in snctl – becomes a Prometheus label	`g1`

Configure the Prometheus scrape interval in your ServiceMonitor settings. A recommended default is 30–60 seconds:

values.yaml

    serviceMonitor:
      enabled: true
      interval: 60s

Step 7 – Install / Upgrade the Helm chart

helm upgrade --install sambanova-hardware-monitoring \
  oci://$REGISTRY/sambanova-hardware-monitoring \
  --version $CHART_VERSION \
  --namespace $NAMESPACE --create-namespace \
  -f values.yaml

The chart deploys three DaemonSets, three Services, and three ServiceMonitor resources. Prometheus will begin scraping.

Verification

Work through each check in order. Each one confirms a layer on top of the previous.

Check 1 – DaemonSet pods are running

kubectl get pods -n $NAMESPACE -l app=ipmi-exporter -o wide
kubectl get pods -n $NAMESPACE -l app=sambanova-exporter -o wide
kubectl get pods -n $NAMESPACE -l app=sn-xrdu-exporter -o wide

All pods should show Running. The NODE column must show your SambaRack node names. If the label selector returns nothing, the chart may use app.kubernetes.io/name instead of app. Fall back to:

kubectl get pods -n $NAMESPACE | grep -E 'ipmi|sambanova|xrdu'

If a pod is Pending or in CrashLoopBackOff, see the Troubleshooting section.

Check 2 – Prometheus targets are up

Open the Prometheus UI and navigate to Status → Targets. Confirm all three exporter targets show UP (n/n) – all instances scraped successfully. If Prometheus is not available, skip to Check 3.

Check 3 – Exporters are responding

With Prometheus: Port-forward each exporter and confirm it returns metrics.

# IPMI exporter – expect lines starting with ipmi_
kubectl port-forward -n $NAMESPACE daemonset/ipmi-exporter 9289:9289 &
sleep 2 && curl -s http://localhost:9289/metrics | grep "^ipmi_" | head -5
pkill -f "port-forward.*9289"

# SambaNova exporter – expect lines starting with sambanova_
kubectl port-forward -n $NAMESPACE daemonset/sambanova-exporter 9101:9101 &
sleep 2 && curl -s http://localhost:9101/metrics | grep "^sambanova_" | head -5
pkill -f "port-forward.*9101"

# XRDU exporter – expect lines starting with xrdu_
kubectl port-forward -n $NAMESPACE daemonset/sn-xrdu-exporter 8076:8076 &
sleep 2 && curl -s http://localhost:8076/metrics | grep "^xrdu_" | head -5
pkill -f "port-forward.*8076"

Hardware Monitoring Prometheus Ui All Exporters 1

Without Prometheus: Iterate over the exporter pods by label and hit each metrics endpoint directly.

# List all exporter pods with their IPs
kubectl get pods -n $NAMESPACE -l 'app in (ipmi-exporter,sambanova-exporter,sn-xrdu-exporter)' \
  -o custom-columns='NAME:.metadata.name,IP:.status.podIP,PORT:.spec.containers[0].ports[0].containerPort'

# Then exec into any pod in the namespace and curl each IP:port
kubectl exec -n $NAMESPACE <any-running-pod> -- \
  curl -s http://<pod-ip>:<port>/metrics | head -10

Each port-forward selects one pod from the DaemonSet at random. This verifies one node’s exporter. To test a specific node’s pod, use kubectl port-forward -n $NAMESPACE pod/<pod-name> <port>:<port> after identifying the pod name from Check 1.

If metrics are not flowing or pods are not coming up, check for any misconfiguration in your values.yaml. Common issues are covered in the Troubleshooting section.

Air-gap installation

If your cluster cannot reach the SambaNova artifact registry, package the chart and images on a machine that can, then transfer them to the air-gapped environment.

1. On a machine with registry access

Authenticate and pull the chart and images:

gcloud auth print-access-token | helm registry login $REGISTRY_SERVER \
  --username=oauth2accesstoken \
  --password-stdin

helm pull oci://$REGISTRY/sambanova-hardware-monitoring --version $CHART_VERSION
# Produces: sambanova-hardware-monitoring-<version>.tgz

docker pull $REGISTRY/sn-ipmi-exporter:<ipmi-version>
docker pull $REGISTRY/sn-xrdu-exporter:<xrdu-version>
docker save \
  $REGISTRY/sn-ipmi-exporter:<ipmi-version> \
  $REGISTRY/sn-xrdu-exporter:<xrdu-version> \
  -o sambanova-hw-monitoring-images.tar

Contact your SambaNova representative for the image tag versions that match your chart version.

2. Transfer to the air-gapped environment

Copy sambanova-hardware-monitoring-<version>.tgz and sambanova-hw-monitoring-images.tar to the air-gapped cluster.

3. Load images into the internal registry

docker load -i sambanova-hw-monitoring-images.tar
docker tag $REGISTRY/sn-ipmi-exporter:<ipmi-version> $INTERNAL_REGISTRY/sn-ipmi-exporter:<ipmi-version>
docker tag $REGISTRY/sn-xrdu-exporter:<xrdu-version> $INTERNAL_REGISTRY/sn-xrdu-exporter:<xrdu-version>
docker push $INTERNAL_REGISTRY/sn-ipmi-exporter:<ipmi-version>
docker push $INTERNAL_REGISTRY/sn-xrdu-exporter:<xrdu-version>

4. Install / Upgrade from the local chart package

helm upgrade --install sambanova-hardware-monitoring \
  sambanova-hardware-monitoring-$CHART_VERSION.tgz \
  --namespace $NAMESPACE --create-namespace \
  --set global.imageRegistry=$INTERNAL_REGISTRY \
  -f values.yaml

After the upgrade, re-run the Verification checks to confirm all pods are still Running and metrics are flowing. If the upgrade fails or pods become unhealthy, roll back:

helm history sambanova-hardware-monitoring -n $NAMESPACE
helm rollback sambanova-hardware-monitoring -n $NAMESPACE   # restores the previous release

Install and verification

Step 1 – Set up your environment

Step 2 – Create the monitoring namespace

Step 3 – Create the image pull secret

Step 4 – Create the XRDU credentials secret

Step 5 – Authenticate to the OCI registry

Step 6 – Prepare your values override file

Step 7 – Install / Upgrade the Helm chart

Verification

Check 1 – DaemonSet pods are running

Check 2 – Prometheus targets are up

Check 3 – Exporters are responding

Air-gap installation

1. On a machine with registry access

2. Transfer to the air-gapped environment

3. Load images into the internal registry

4. Install / Upgrade from the local chart package

What’s next

Monitoring metrics reference

Building dashboards in Grafana

​Step 1 – Set up your environment

​Step 2 – Create the monitoring namespace

​Step 3 – Create the image pull secret

​Step 4 – Create the XRDU credentials secret

​Step 5 – Authenticate to the OCI registry

​Step 6 – Prepare your values override file

​Step 7 – Install / Upgrade the Helm chart

​Verification

​Check 1 – DaemonSet pods are running

​Check 2 – Prometheus targets are up

​Check 3 – Exporters are responding

​Air-gap installation

​1. On a machine with registry access

​2. Transfer to the air-gapped environment

​3. Load images into the internal registry

​4. Install / Upgrade from the local chart package

​What’s next

Monitoring metrics reference

Building dashboards in Grafana

Step 1 – Set up your environment

Step 2 – Create the monitoring namespace

Step 3 – Create the image pull secret

Step 4 – Create the XRDU credentials secret

Step 5 – Authenticate to the OCI registry

Step 6 – Prepare your values override file

Step 7 – Install / Upgrade the Helm chart

Verification

Check 1 – DaemonSet pods are running

Check 2 – Prometheus targets are up

Check 3 – Exporters are responding

Air-gap installation

1. On a machine with registry access

2. Transfer to the air-gapped environment

3. Load images into the internal registry

4. Install / Upgrade from the local chart package

What’s next