> ## Documentation Index
> Fetch the complete documentation index at: https://sambanova-systems.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

<Note>
  Commands below use variables from your `.env` file. Run `source .env` before copying commands, or replace `$NAMESPACE`, `$REGISTRY_SERVER`, and `$GCP_KEY_PATH` with your actual values (default namespace: `monitoring`).
</Note>

<AccordionGroup>
  <Accordion title="`ImagePullBackOff` on any pod">
    **Cause:** Pull secret missing, wrong namespace, or GCP key invalid.

    Delete and recreate the pull secret:

    ```bash theme={null}
    kubectl delete secret sn-registry-creds -n $NAMESPACE --ignore-not-found
    kubectl create secret docker-registry sn-registry-creds \
      --docker-server=$REGISTRY_SERVER \
      --docker-username=_json_key \
      --docker-password="$(cat "$GCP_KEY_PATH")" \
      --namespace=$NAMESPACE
    ```

    **Verify:** `kubectl get pods -n $NAMESPACE` – pods move to `Running` within 30 seconds.
  </Accordion>

  <Accordion title="`ipmi-exporter` pod crashes or init fails">
    **Cause:** `ipmi_devintf` kernel module not loaded.

    Find the failing node:

    ```bash theme={null}
    kubectl get pods -n $NAMESPACE -l app=ipmi-exporter -o wide
    ```

    SSH to that node and load the module:

    ```bash theme={null}
    sudo modprobe ipmi_devintf
    ```

    Persist across reboots:

    ```bash theme={null}
    echo "ipmi_devintf" | sudo tee /etc/modules-load.d/ipmi.conf
    ```
  </Accordion>

  <Accordion title="`sambanova-exporter` init fails">
    **Cause:** SambaNova runtime service not running on the host.

    Find the failing node (read the `NODE` column):

    ```bash theme={null}
    kubectl get pods -n $NAMESPACE -l app=sambanova-exporter -o wide
    ```

    SSH to that node, then:

    ```bash theme={null}
    systemctl status sambanova_exporter
    sudo systemctl start sambanova_exporter
    ```
  </Accordion>

  <Accordion title="DaemonSet pods stuck in `Pending` due to missing node label">
    **Cause:** Node is missing the `snRduArch` label required by the DaemonSet nodeSelector.

    Check which nodes are missing the label:

    ```bash theme={null}
    kubectl get nodes --show-labels | grep snRduArch
    ```

    Apply the label to each SambaRack node:

    ```bash theme={null}
    kubectl label nodes <NODE_NAME> snRduArch=sn40-16 --overwrite
    ```

    Confirm the pod moves to `Running`:

    ```bash theme={null}
    kubectl get pods -n $NAMESPACE -o wide
    ```
  </Accordion>

  <Accordion title="`sn-xrdu-exporter` returns 404 (target not found)">
    **Cause:** `node:` in targets does not match Kubernetes node name.

    Check the name the pod sees:

    ```bash theme={null}
    kubectl exec -n $NAMESPACE daemonset/sn-xrdu-exporter -- env | grep NODE_NAME
    ```

    Compare to `kubectl get nodes`. Fix `node:` in `values.yaml`, then re-deploy:

    ```bash theme={null}
    helm upgrade --install sambanova-hardware-monitoring \
      oci://$REGISTRY/sambanova-hardware-monitoring \
      --version $CHART_VERSION --namespace $NAMESPACE -f values.yaml
    ```
  </Accordion>

  <Accordion title="XRDU exporter pod up but no XRDU metrics">
    **Cause:** XRDU BMC IP unreachable or incorrect credentials.

    Test BMC reachability:

    ```bash theme={null}
    kubectl exec -n $NAMESPACE daemonset/sn-xrdu-exporter -- ping -c1 <xrdu-bmc-ip>
    ```

    If ping succeeds but metrics are still absent, verify `XRDU_USERNAME` and `XRDU_PASSWORD` in the `sn-xrdu-creds` secret match your BMC configuration.
  </Accordion>

  <Accordion title="Prometheus target shows `DOWN: connection refused`">
    **Cause:** Pod not running or port mismatch.

    Verify the pod is `Running`:

    ```bash theme={null}
    kubectl get pods -n $NAMESPACE
    ```

    Confirm `port:` in your values matches the exporter's listen port: IPMI: `9289`, SambaNova: `9101`, XRDU: `8076`.
  </Accordion>

  <Accordion title="Prometheus target shows `DOWN: context deadline exceeded`">
    **Cause:** Network policy blocking scrape, or timeout too short.

    Check network policies:

    ```bash theme={null}
    kubectl get networkpolicy -n $NAMESPACE
    ```

    Increase scrape timeout in `values.yaml`:

    ```yaml theme={null}
    ipmi-exporter:
      serviceMonitor:
        scrapeTimeout: 60s
    sambanova-exporter:
      serviceMonitor:
        scrapeTimeout: 60s
    sn-xrdu-exporter:
      serviceMonitor:
        scrapeTimeout: 60s
    ```

    Then re-apply:

    ```bash theme={null}
    helm upgrade --install sambanova-hardware-monitoring \
      oci://$REGISTRY/sambanova-hardware-monitoring \
      --version $CHART_VERSION --namespace $NAMESPACE -f values.yaml
    ```
  </Accordion>
</AccordionGroup>