Skip to main content
Commands below use variables from your .env file. Run source .env before copying commands, or replace $NAMESPACE, $REGISTRY_SERVER, and $GCP_KEY_PATH with your actual values (default namespace: monitoring).
Cause: Pull secret missing, wrong namespace, or GCP key invalid.Delete and recreate the pull secret:
kubectl delete secret sn-registry-creds -n $NAMESPACE --ignore-not-found
kubectl create secret docker-registry sn-registry-creds \
  --docker-server=$REGISTRY_SERVER \
  --docker-username=_json_key \
  --docker-password="$(cat "$GCP_KEY_PATH")" \
  --namespace=$NAMESPACE
Verify: kubectl get pods -n $NAMESPACE – pods move to Running within 30 seconds.
Cause: ipmi_devintf kernel module not loaded.Find the failing node:
kubectl get pods -n $NAMESPACE -l app=ipmi-exporter -o wide
SSH to that node and load the module:
sudo modprobe ipmi_devintf
Persist across reboots:
echo "ipmi_devintf" | sudo tee /etc/modules-load.d/ipmi.conf
Cause: SambaNova runtime service not running on the host.Find the failing node (read the NODE column):
kubectl get pods -n $NAMESPACE -l app=sambanova-exporter -o wide
SSH to that node, then:
systemctl status sambanova_exporter
sudo systemctl start sambanova_exporter
Cause: Node is missing the snRduArch label required by the DaemonSet nodeSelector.Check which nodes are missing the label:
kubectl get nodes --show-labels | grep snRduArch
Apply the label to each SambaRack node:
kubectl label nodes <NODE_NAME> snRduArch=sn40-16 --overwrite
Confirm the pod moves to Running:
kubectl get pods -n $NAMESPACE -o wide
Cause: node: in targets does not match Kubernetes node name.Check the name the pod sees:
kubectl exec -n $NAMESPACE daemonset/sn-xrdu-exporter -- env | grep NODE_NAME
Compare to kubectl get nodes. Fix node: in values.yaml, then re-deploy:
helm upgrade --install sambanova-hardware-monitoring \
  oci://$REGISTRY/sambanova-hardware-monitoring \
  --version $CHART_VERSION --namespace $NAMESPACE -f values.yaml
Cause: XRDU BMC IP unreachable or incorrect credentials.Test BMC reachability:
kubectl exec -n $NAMESPACE daemonset/sn-xrdu-exporter -- ping -c1 <xrdu-bmc-ip>
If ping succeeds but metrics are still absent, verify XRDU_USERNAME and XRDU_PASSWORD in the sn-xrdu-creds secret match your BMC configuration.
Cause: Pod not running or port mismatch.Verify the pod is Running:
kubectl get pods -n $NAMESPACE
Confirm port: in your values matches the exporter’s listen port: IPMI: 9289, SambaNova: 9101, XRDU: 8076.
Cause: Network policy blocking scrape, or timeout too short.Check network policies:
kubectl get networkpolicy -n $NAMESPACE
Increase scrape timeout in values.yaml:
ipmi-exporter:
  serviceMonitor:
    scrapeTimeout: 60s
sambanova-exporter:
  serviceMonitor:
    scrapeTimeout: 60s
sn-xrdu-exporter:
  serviceMonitor:
    scrapeTimeout: 60s
Then re-apply:
helm upgrade --install sambanova-hardware-monitoring \
  oci://$REGISTRY/sambanova-hardware-monitoring \
  --version $CHART_VERSION --namespace $NAMESPACE -f values.yaml