> ## Documentation Index > Fetch the complete documentation index at: https://sambanova-systems.mintlify.site/docs/llms.txt > Use this file to discover all available pages before exploring further. # Device administration This guide covers checking and updating firmware on devices within SambaRack. Devices are individual components attached to racks and nodes, including BMCs, switches, PDUs, and serial terminals. ## Commands ### Check firmware version Check firmware versions on devices: ```bash theme={} snctl device firmware-check \ (--rack-id | --group ) \ --type \ [--index ] ``` **Target selection:** One of `--rack-id` or `--group` is required (mutually exclusive): | Parameter | Description | | ---------------------- | ----------------------------------------- | | `--rack-id ` | Check devices in a specific rack | | `--group ` | Check devices in all racks within a group | **Device types:** | Type | Description | | --------------- | ------------------------------- | | `access_switch` | Access switch | | `data_switch` | Data switch | | `scs` | System Control Switch | | `bios` | BIOS | | `bmc` | Baseboard Management Controller | | `card` | Card/Adapter | | `pdu` | Power Distribution Unit | | `xrdu` | XRDU | | `all` | All device types | #### Single rack **Example – check BMC firmware on a single rack:** ```bash theme={} snctl device firmware-check --rack-id 001 --type bmc ``` **Example output:** ```text theme={} Firmware Version Information ┏━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓ ┃ Rack ┃ Component ┃ Status ┃ CURR ┃ POR ┃ ┡━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩ │ Rack-001 │ BMC │ PASS │ 12.61.21 │ 12.61.21 │ └──────────┴───────────┴────────┴──────────┴──────────┘ ``` **Example – check all firmware on a rack:** ```bash theme={} snctl device firmware-check --rack-id 001 --type all ``` **Example output:** ```text theme={} Firmware Version Information ┏━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┓ ┃ Rack ┃ Component ┃ Status ┃ CURR ┃ POR ┃ OBMC_CURR ┃ OBMC_POR ┃ RDUC_CURR ┃ RDUC_POR ┃ ┡━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━┩ │ Rack-001 │ ACCESS_SWITCH │ FAIL │ │ │ │ │ │ │ │ Rack-001 │ BIOS │ PASS │ R24_P1 │ R24_P1 │ │ │ │ │ │ Rack-001 │ BMC │ PASS │ 12.61.21 │ 12.61.21 │ │ │ │ │ │ Rack-001 │ CARD │ PASS │ snhni0 -> 228.1.111.0 PASS │ 228.1.111.0 │ │ │ │ │ │ │ │ │ snni0 -> 228.1.111.0 PASS │ │ │ │ │ │ │ │ │ │ snni2 -> 228.1.111.0 PASS │ │ │ │ │ │ │ │ │ │ snni4 -> 228.1.111.0 PASS │ │ │ │ │ │ │ │ │ │ snni6 -> 228.1.111.0 PASS │ │ │ │ │ │ │ │ │ │ snni8 -> 228.1.111.0 PASS │ │ │ │ │ │ │ │ │ │ snni10 -> 228.1.111.0 PASS │ │ │ │ │ │ │ │ │ │ snni12 -> 228.1.111.0 PASS │ │ │ │ │ │ │ │ │ │ snni14 -> 228.1.111.0 PASS │ │ │ │ │ │ │ Rack-001 │ DATA_SWITCH │ FAIL │ │ │ │ │ │ │ │ Rack-001 │ PDU_PDU_1 │ FAIL │ 5.10.8 │ 6.3.0 │ │ │ │ │ │ Rack-001 │ PDU_PDU_2 │ FAIL │ 5.10.8 │ 6.3.0 │ │ │ │ │ │ Rack-001 │ PDU_PDU_3 │ FAIL │ 5.10.8 │ 6.3.0 │ │ │ │ │ │ Rack-001 │ PDU_PDU_4 │ FAIL │ 5.10.8 │ 6.3.0 │ │ │ │ │ │ Rack-001 │ SCS │ FAIL │ 9.4.0.1R1 │ 9.5.0.0R15 │ │ │ │ │ │ Rack-001 │ XRDU_XRDU_0 │ PASS │ │ │ 5.1.4-0 │ 5.1.4-0 │ 5.1.4 │ 5.1.4 │ │ Rack-001 │ XRDU_XRDU_1 │ PASS │ │ │ 5.1.4-0 │ 5.1.4-0 │ 5.1.4 │ 5.1.4 │ │ Rack-001 │ XRDU_XRDU_2 │ PASS │ │ │ 5.1.4-0 │ 5.1.4-0 │ 5.1.4 │ 5.1.4 │ │ Rack-001 │ XRDU_XRDU_3 │ PASS │ │ │ 5.1.4-0 │ 5.1.4-0 │ 5.1.4 │ 5.1.4 │ │ Rack-001 │ XRDU_XRDU_4 │ PASS │ │ │ 5.1.4-0 │ 5.1.4-0 │ 5.1.4 │ 5.1.4 │ │ Rack-001 │ XRDU_XRDU_5 │ PASS │ │ │ 5.1.4-0 │ 5.1.4-0 │ 5.1.4 │ 5.1.4 │ │ Rack-001 │ XRDU_XRDU_6 │ PASS │ │ │ 5.1.4-0 │ 5.1.4-0 │ 5.1.4 │ 5.1.4 │ │ Rack-001 │ XRDU_XRDU_7 │ PASS │ │ │ 5.1.4-0 │ 5.1.4-0 │ 5.1.4 │ 5.1.4 │ └──────────┴───────────────┴────────┴─────────────────────────────┴─────────────┴───────────┴──────────┴───────────┴──────────┘ ``` #### Group **Example – check BMC firmware across all racks in a group:** ```bash theme={} snctl device firmware-check --group g1 --type bmc ``` **Example output** (one row per rack): ```text theme={} Firmware Version Information ┏━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓ ┃ Rack ┃ Component ┃ Status ┃ CURR ┃ POR ┃ ┡━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩ │ Rack-001 │ BMC │ PASS │ 12.61.21 │ 12.61.21 │ │ Rack-002 │ BMC │ FAIL │ 12.85.05 │ 12.61.21 │ └──────────┴───────────┴────────┴──────────┴──────────┘ ``` If devices are unreachable or configured incorrectly, the firmware check fails for those devices with a `FAIL` status, as shown in the example output above. For card devices, the `CURR` column lists each network interface with its firmware version and a per-interface `PASS`/`FAIL` comparison. The overall `CARD` status is `PASS` only when every interface matches the POR version. ### Update firmware Update firmware on devices within a rack or group: ```bash theme={} snctl device firmware-update \ (--rack-id | --group ) \ --type \ [--index ] \ [--file ...] \ [--force] ``` **Target selection:** One of `--rack-id` or `--group` is required (mutually exclusive): | Parameter | Description | | ---------------------- | ------------------------------------------ | | `--rack-id ` | Update devices in a specific rack | | `--group ` | Update devices in all racks within a group | **Device types:** | Type | Description | | ------ | ------------------------------- | | `bios` | BIOS | | `bmc` | Baseboard Management Controller | | `card` | Card/Adapter | | `xrdu` | XRDU | Other device types (`access_switch`, `data_switch`, `pdu`, `scs`) are not currently supported for firmware update and will be rejected by the CLI. **Optional parameters:** | Parameter | Description | | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--index ` | Device index to update. For XRDU: omit to target all XRDUs, pass a single index (0–7), or a comma-separated list (e.g. `0,2,5`). Multiple indices run in parallel within the rack. | | `--file ` | Local firmware file. For XRDU, pass one combined archive whose name starts with `xrdu` (e.g. `xrdu-combined-5.1.4.tgz`), or two split archives whose names start with `obmc` and `rduc` respectively. Repeat the flag for two files: `--file obmc-5.1.4.tgz --file rduc-5.1.4.tgz`. If omitted, the firmware is downloaded from the configured artifact store. | | `--force` | Skip checksum validation | #### Single rack **Example – update BMC firmware:** ```bash theme={} snctl device firmware-update --rack-id 001 --type bmc ``` **Example – update all XRDUs:** ```bash theme={} snctl device firmware-update --rack-id 001 --type xrdu ``` **Example – update specific XRDUs:** ```bash theme={} snctl device firmware-update --rack-id 001 --type xrdu --index 0,2,5 ``` **Example – update XRDU firmware from local files:** ```bash theme={} snctl device firmware-update --rack-id 001 --type xrdu --index 3 \ --file /path/to/obmc-5.1.4.tgz --file /path/to/rduc-5.1.4.tgz --force ``` #### Group **Example – update firmware across all racks in a group:** ```bash theme={} snctl device firmware-update --group g1 --type bmc ``` When you use `--group`, `snctl` processes racks sequentially (one rack at a time) while running individual device updates in parallel within each rack. This bounds concurrency – for example, a group with 10 racks and 8 XRDUs each won't launch 80 simultaneous jobs. During the update, `snctl` prints: * Per-component `PASS`/`FAIL` lines as each job completes * Per-rack checkpoints after each rack finishes, including `PASS`/`FAIL` counts and log file paths for any failures * A final **Firmware Update Results** table summarizing the entire run If one or more devices fail, the command exits non-zero and prints the stdout, stderr, and instruction output log paths for each failed job. ## Order of execution When updating multiple components, there is no strict dependency between them. However, if upgrading more than one type in the same maintenance window, follow this order: **THOR CARD → BIOS → BMC → XRDU** | Device | Pre-condition | Post-upgrade operations | Expected duration | | --------- | ------------------------ | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------ | | THOR CARD | Host must be powered on | Power cycle, firmware check | \~25 min (\~5 min upgrade + \~20 min power cycle) | | BIOS | Host must be powered on | Host power off/on, SR-IOV verify, Node Powercycle, Firmware Check | 1 hour (\~5 mins upgrade + \~30 mins SR-IOV disable & verify + \~20 mins powercycle) | | BMC | None | Power cycle, firmware check | \~25 min (\~5 min upgrade + \~20 min power cycle) | | XRDU | Host must be powered off | Power on, firmware check | \~30 min (\~20 min upgrade + \~10 min power on) | ### 1. THOR CARD **Pre-condition:** Host must be powered on. ```bash theme={} # Verify host is powered on snctl node powerstate --node --type host # Power on host if needed snctl node poweron --node --type host ``` **Update:** ```bash theme={} snctl device firmware-update --rack-id --type card ``` **Post-upgrade:** ```bash theme={} snctl node powercycle --node snctl device firmware-check --rack-id --type card ``` ### 2. BIOS **Pre-condition:** Host must be powered on. ```bash theme={} # Verify host is powered on snctl node powerstate --node --type host # Power on host if needed snctl node poweron --node --type host ``` **Update:** ```bash theme={} snctl device firmware-update --rack-id --type bios ``` **Post-upgrade:** Applying the BIOS update and then immediately power cycling the whole node can leave the host powered on but stuck before POST (it never reaches the OS). To apply the new BIOS reliably, cycle the host on its own first, confirm the SR-IOV BIOS setting, and only then power cycle the entire node. 1. **Power off the node (host and XRDUs).** Omit `--type` so both the host and the XRDUs are powered off. ```text theme={} snctl node poweroff --node ``` 2. **Power the host back on.** The new BIOS is applied during POST (the console shows the BIOS settings being applied, and the host may reboot once on its own) before it boots to the OS. ```text theme={} snctl node poweron --node --type host ``` Note: Wait \~15 minutes after power-on for the host to finish POST and boot to the OS before running the SR-IOV check below. 3. **Ensure SR-IOV is disabled.** SR-IOV must be **disabled** in the BIOS (`PCIS007` is the SR-IOV attribute). Work through the checks below over Redfish. **3a. Gather the BMC connection details:**
The curl commands below authenticate to the BMC's Redfish API and need its IP address, username, and password. BMC IP and username come from the node's inventory. Run snctl node show \ and read them from the BMC row (IP and Username columns): ```shellscript theme={} snctl node show ``` BMC password is stored in the snctl secret store. Print the current value with: ```shellscript theme={} snctl secret get --device-type bmc --show-value ``` \--show-value prints the password in clear text. Only use it in a secure environment and never share the output If the `curl` commands below time out or return an authentication error, re-verify the BMC IP, username, and password with `snctl node show ` and `snctl secret get --device-type bmc --show-value`, and confirm the BMC's management interface is reachable from this host. **3b. Check the current value:** ```shellscript theme={} curl -s -k -u : \ https:///redfish/v1/Systems/Self/Bios \ | jq '.Attributes.PCIS007' #Example Output "Enabled" ``` * If it reads `"Disabled"`, you are done with this step, continue to step 4. * If it reads `"Enabled"`, continue to step 3c. **3c. Disable SR-IOV** **(a successful request returns no output):** ```shellscript theme={} curl -s -k -u : \ -X POST https:///redfish/v1/Systems/Self/Bios/SD \ -H 'Content-Type: application/json' \ -d '{"Attributes": {"PCIS007": "Disabled"}}' ``` **3d. Apply the change** by **rebooting just the host** again: ```shellscript theme={} snctl node poweroff --node --type host snctl node poweron --node --type host ``` Wait \~15 minutes for the host to boot, then repeat step 3a to confirm `PCIS007` now reads `"Disabled"` before continuing. If it still reads `"Enabled"`, do not proceed to step 4 — repeat step 3c once, and if the value still hasn't changed, stop and contact SambaNova support rather than power cycling with SR-IOV still enabled. 4. **Power cycle the entire node** once SR-IOV is confirmed disabled. ```bash theme={} #Power Cycle the Node snctl node powercycle --node --type node #Check the Firmware BIOS version snctl device firmware-check --rack-id --type bios ``` ### 3. BMC **Update:** ```bash theme={} snctl device firmware-update --rack-id --type bmc ``` **Post-upgrade:** ```bash theme={} snctl node powercycle --node --type node snctl device firmware-check --rack-id --type bmc ``` ### 4. XRDU **Pre-condition:** Host must be powered off before updating XRDU firmware. ```bash theme={} # Verify host is powered off snctl node powerstate --node --type host # Power off host if needed snctl node poweroff --node --type host ``` **Update:** ```bash theme={} snctl device firmware-update --rack-id --type xrdu ``` **Post-upgrade:** ```bash theme={} snctl node poweron --node --type node snctl device firmware-check --rack-id --type xrdu ``` ## Output columns The firmware check output includes the following columns: | Column | Description | | ---------- | --------------------------------------------------------------------------------------------------------------------------------- | | Rack | Rack identifier | | Component | Device type and index | | Status | `PASS` if firmware matches expected version, `FAIL` otherwise | | CURR | Current firmware version. For card devices, lists each network interface with its version and per-interface `PASS`/`FAIL` result. | | POR | Plan of Record (expected) firmware version | | OBMC\_CURR | Current OpenBMC firmware version (XRDU only) | | OBMC\_POR | Expected OpenBMC firmware version (XRDU only) | | RDUC\_CURR | Current RDUC firmware version (XRDU only) | | RDUC\_POR | Expected RDUC firmware version (XRDU only) |