> ## Documentation Index > Fetch the complete documentation index at: https://sambanova-systems.mintlify.site/docs/llms.txt > Use this file to discover all available pages before exploring further. # Firmware Updates This guide covers auditing and upgrading firmware on SambaRack hardware components using the `snctl firmware` command group. ## Key concepts Read this section once. Everything after it depends on these six things. ### Prerequisites Firmware commands authenticate to the target devices, so device credentials must be registered first. See [Device credentials](srm-device-credentials). ```bash theme={} snctl secret set --device-type bmc snctl secret set --device-type card snctl secret set --device-type host snctl secret set --device-type xrdu ``` ### Which device types support what Not every component that can be audited can be upgraded through SambaRack Manager. | Device type | `firmware version` | `firmware upgrade` | | ----------------- | :----------------: | :--------------------------: | | `bios` | Yes | Yes | | `bmc` | Yes | Yes | | `xrdu` | Yes | Yes | | `card` (Thor NIC) | Yes | Yes | | `pdu` | Yes | **No** | | `serial_terminal` | Yes | **No** | | `access_switch` | Yes | **No** | | `data_switch` | Yes | **No** | | `all` | Yes | Yes — upgradeable types only | ### Choosing a target Either `--rack-id` or `--group` must be specified. They are mutually exclusive. Both subcommands accept either one. ```bash theme={} snctl firmware version --rack-id 001 --device-type all snctl firmware version --group production --device-type all snctl firmware upgrade --group production --device-type bmc ``` ### (Optional) Slotted device types and `--index` `xrdu` and `card` are **slotted** — they have multiple instances per node, so `--index` can narrow the operation to specific slots. Pass a comma-separated list, for example `0,2,5`. | Behaviour | Detail | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Omitting `--index` | Targets **all** instances on the target racks | | Valid for | `xrdu` and `card` only | | Rejected for | `bios`, `bmc`, and `all` — these have no slots | | `card` numbering | Selects network interfaces by their position in the rack type's interface list — the same numbering shown in the `Slot` column of `snctl firmware version --device-type card` | | Index matches nothing | Error. The command reports which index is missing and which are available, then exits non-zero **without flashing anything** | ### XRDUs flash two components Each XRDU contains two independently flashed components — **OBMC** and **RDU-C**. They are reported separately by `firmware version`, and both must be supplied separately when passing images manually. An XRDU upgrade powers the chassis off before flashing, because the flashing tool refuses to write to a powered-on chassis. **This power-off applies to every XRDU behind the endpoint, not only the slots named by `--index`.** Plan XRDU upgrades as whole-node maintenance regardless of how many slots you are targeting. ### Where firmware images come from | Environment | Behaviour | | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Connected** (default) | Omit `--image`. SambaRack Manager downloads the plan-of-record firmware, verifies its checksum, and caches it locally. Repeat upgrades of the same version reuse the cache. | | **Airgapped** | Supply the file yourself with `--image`. See [Airgapped environments](#airgapped-environments). | The cache directory is shown as **Cache Location** in `snctl config show`. Change it with `snctl config update --cache-location `. See [Initial setup](srm-initial-setup). *** ## Check firmware versions Audit installed firmware against the plan of record (POR). ```bash theme={} snctl firmware version \ (--rack-id | --group ) \ [--device-type ] \ [--index ] ``` | Flag | Description | | ----------------------- | ----------------------------------------------------------------------------------------------------------- | | `-r, --rack-id ` | Rack to target (required unless `--group` is used) | | `--group ` | Target every rack in a named group | | `--device-type ` | `all` (default), or any type from the [support table](#which-device-types-support-what) | | `--index ` | Slot indices for `xrdu` and `card` — see [Slotted device types](#optional-slotted-device-types-and---index) | | `--protocol ` | Override the default protocol: `ssh`, `rest`, or `redfish` (endpoint-dependent) | | `-o, --output ` | `text` (default), `wide`, or `json` | ```bash theme={} snctl firmware version --rack-id 001 --device-type all ``` ```text theme={} Rack Node Component Status CURR POR ---- --------------- --------- ------ ----- ----- 001 sn40l-16-001-n1 BIOS PASS 2.4.1 2.4.1 001 sn40l-16-001-n1 BMC FAIL 1.07 1.08 1 racks: 1 ok, 0 failed (12.4s) ``` ### Reading the output | Column | Meaning | | -------- | --------------------------------------------------------------------------------------------- | | `CURR` | The version currently installed | | `POR` | The plan-of-record target version | | `Status` | `PASS` when they match, `FAIL` when they differ, `ERROR` when the component could not be read | **`FAIL` means the firmware is out of date — not that the command failed.** The check ran successfully and correctly reported a version mismatch. XRDUs report their two components on separate rows: ```text theme={} Rack Node Component Status CURR POR ---- --------------- ----------- ------ ----- ----- 001 sn40l-16-001-n1 XRDU/0/OBMC PASS 5.1.4 5.1.4 001 sn40l-16-001-n1 XRDU/0/RDUC FAIL 5.1.2 5.1.4 ``` ### Wide output `-o wide` adds **Protocol**, **Model**, and **Source** columns. `Source` is the artifact URL of the POR firmware, which you need for airgapped upgrades: ```bash theme={} snctl firmware version --rack-id 001 --device-type bios -o wide ``` ```powershell theme={} Rack Node Component Status CURR POR Protocol Model Source ---- --------------- --------- ------ ----- ----- -------- -------- ------------------------------------------ 001 sn40l-16-001-n1 BIOS PASS 2.4.1 2.4.1 redfish SN40L-16 gs:///firmware/bios/2.4.1/bios.bin ``` *** ## Upgrade firmware Push firmware to a rack's components. ```bash theme={} snctl firmware upgrade \ (--rack-id | --group ) \ --device-type \ [--image ] \ [--index ] ``` | Flag | Description | | ----------------------------- | ----------------------------------------------------------------------------------------------------------- | | `-r, --rack-id ` | Rack to target (required unless `--group` is used) | | `--group ` | Target every rack in a named group | | `--device-type ` | `all`, `bios`, `bmc`, `xrdu`, or `card` — **required** | | `--image ` | Local firmware image. **Optional** — downloaded automatically if omitted | | `--index ` | Slot indices for `xrdu` and `card` — see [Slotted device types](#optional-slotted-device-types-and---index) | | `-p, --parallelism-racks ` | Max racks in flight (default `1`) | | `--parallelism-devices ` | Global max nodes in flight across all racks (`0` = uncapped) | | `--protocol ` | Override the default protocol (endpoint-dependent) | ### Recommended order and pre-conditions There is no strict dependency across the supported components. However, when upgrading more than one, use this order: **CARD → BIOS → BMC → XRDU** | Device | Pre-condition | Post-upgrade operations | Expected duration | | -------- | ----------------------------------------------------------------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | | **CARD** | `snd` driver daemon active on the host (waited for automatically) | No power cycle. Firmware version check | **\~10 min** — \~5 min upgrade, \~5 min verification | | **BIOS** | Host powered **on** | Host power off/on, SR-IOV verify, node power cycle, version check | **\~1 hour** — \~5 min upgrade, \~30 min SR-IOV disable and verify, \~20 min power cycle | | **BMC** | None | Power cycle, version check | **\~25 min** — \~5 min upgrade, \~20 min power cycle | | **XRDU** | Host powered **off** | Power on, version check | **\~30 min** — \~20 min upgrade, \~10 min power on | BIOS and XRDU have **opposite** power pre-conditions — BIOS needs the host on, XRDU needs it off. **Verifying an upgrade.** Run `snctl firmware version --rack-id --device-type all` about 5 minutes after a successful run to confirm the new version took effect. ### Examples ```bash theme={} # Upgrade BMC firmware, image downloaded automatically snctl firmware upgrade --rack-id 001 --device-type bmc # Upgrade one XRDU snctl firmware upgrade --rack-id 001 --device-type xrdu --index 3 # Upgrade several XRDUs snctl firmware upgrade --rack-id 001 --device-type xrdu --index 3,4,5,6 # Upgrade only network interfaces 0 and 3 on every node snctl firmware upgrade --rack-id 001 --device-type card --index 0,3 ``` ### Progress output Each component reports staged progress as it flashes, followed by a per-component result list and a summary. Components already at the target version are skipped. ```powershell theme={} #Example output : THOR Card Upgrade snctl firmware upgrade --rack-id 001 --device-type card firmware upgrade rack=001 [card slot=1] 30% flashing [card slot=2] 30% flashing [card slot=6] 100% already at target version — skipping [card slot=8] 100% already at target version — skipping [card slot=1] 80% verifying [card slot=2] 80% verifying [card slot=1] 100% done [card slot=2] 100% done ... OK card node=node1 slot=1 OK card node=node1 slot=2 OK card node=node1 slot=6 OK card node=node1 slot=8 ... 9 components: 9 ok, 0 failed (328.7s) ``` ### Airgapped environments If your environment has no GCS access, supply the firmware file yourself with `--image`. Run `firmware version -o wide` and read the `Source` column for the component you are upgrading: ```bash theme={} snctl firmware version --rack-id 001 --device-type bios -o wide ``` From a host that does have GCS access, download it and transfer it into the airgapped environment: ```bash theme={} gsutil cp gs:/// ./bios.bin ``` ```bash theme={} snctl firmware upgrade --rack-id 001 --device-type bios --image /path/to/bios.bin ``` #### Image path format | Device type | `--image` format | | --------------------- | -------------------------------------------------------- | | `bios`, `bmc`, `card` | A bare path with no prefix | | `xrdu` | Passed **twice**, each with an explicit component prefix | ```bash theme={} snctl firmware upgrade --rack-id 001 --device-type xrdu \ --image obmc=/path/to/obmc-5.1.4.tgz \ --image rduc=/path/to/rduc-5.1.4.tgz ``` XRDU upgrades report both components per slot, and step through staging, chassis power-off, flashing of the primary and recovery partitions for each component, a reboot, and verification: ```powershell theme={} [xrdu_obmc slot=2] 3% checking staging space [xrdu_obmc slot=2] 5% powering off chassis [xrdu_obmc slot=2] 15% staging image [xrdu_obmc slot=2] 30% flashing bmc primary [xrdu_obmc slot=2] 45% rebooting [xrdu_obmc slot=2] 60% staging bmc [xrdu_obmc slot=2] 64% flashing bmc recovery [xrdu_obmc slot=2] 68% staging rduc [xrdu_obmc slot=2] 72% flashing rduc primary [xrdu_obmc slot=2] 76% staging rduc [xrdu_obmc slot=2] 80% flashing rduc recovery [xrdu_obmc slot=2] 92% verifying [xrdu_obmc slot=2] 100% done ... OK xrdu_obmc node=node1 slot=2 OK xrdu_rduc node=node1 slot=2 ... 16 components: 16 ok, 0 failed (1451.5s) ``` Slots run in parallel, so lines from different slots interleave in practice. A bare `--image` path is rejected for `--device-type xrdu`, and prefixed values are rejected for single-component types. *** ## Upgrade procedures by device type Follow these in the [recommended order](#recommended-order-and-pre-conditions) when upgrading more than one component. **CARD → BIOS → BMC → XRDU** ### CARD (Thor NIC) **Pre-condition:** the SambaNova driver daemon (`snd`) must be active on the host. SambaRack Manager waits for this automatically before flashing, so no manual check is required. ```bash theme={} # Auto-download the POR image snctl firmware upgrade --rack-id --device-type card # Or flash a specific local image snctl firmware upgrade --rack-id --device-type card --image /path/to/thor-firmware.pkg ``` **No power cycle is required for CARD upgrades** — `bnxtnvm` flashes take effect without a reboot. This is the one upgrade target that does not need a power operation afterward. Verify the firmware version on every NIC interface: ```bash theme={} snctl firmware version --rack-id --device-type card ``` ### BIOS **Pre-condition:** the host must be powered on. Applying the BIOS update and then immediately power-cycling the whole node can leave the host powered on but stuck before POST — it never reaches the OS. To apply the new BIOS reliably, cycle the host on its own first, confirm the SR-IOV setting, and only then power cycle the entire node. Follow every step below in order. ```bash theme={} snctl power state --node --type host # Power on if needed snctl power on --node --type host ``` ```bash theme={} # Auto-download the POR image snctl firmware upgrade --rack-id --device-type bios # Or flash a specific local image snctl firmware upgrade --rack-id --device-type bios --image /path/to/bios.bin ``` Omit `--type` so both the host and the XRDUs power off: ```bash theme={} snctl power off --node ``` Then **power on the host by itself**. The new BIOS is applied during POST — the console shows the BIOS settings being applied, and the host may reboot once on its own before booting to the OS. ```bash theme={} snctl power on --node --type host ``` Wait 10–15 minutes after power-on for the host to finish POST and boot to the OS before continuing. SR-IOV must be **disabled** in the BIOS. The relevant attribute is `PCIS007`. These checks run over Redfish, directly against the BMC. **Gather the BMC connection details.** The BMC IP and username come from the node's inventory — read them from the `BMC` row: ```bash theme={} snctl node show ``` The BMC password comes from the secret store: ```bash theme={} snctl secret get --device-type bmc --show-value ``` `--show-value` prints the password in clear text. Only use it in a secure environment, and never share the output. Use these values for ``, ``, and `` below. **Check the current value:** ```bash theme={} curl -s -k -u : \ https:///redfish/v1/Systems/Self/Bios | jq '.Attributes.PCIS007' ``` * Reads `"Disabled"` — continue to the next step. * Reads anything else — disable it below. **Disable SR-IOV** (a successful request returns no output): ```bash theme={} curl -s -k -u : \ -X POST https:///redfish/v1/Systems/Self/Bios/SD \ -H 'Content-Type: application/json' \ -d '{"Attributes": {"PCIS007": "Disabled"}}' ``` **Apply the change** by cycling the host again: ```bash theme={} snctl power off --node --type host snctl power on --node --type host ``` Wait 10–15 minutes for the host to boot, then re-check `PCIS007` and confirm it now reads `"Disabled"` before continuing. Only once SR-IOV is confirmed disabled. `snctl power cycle` always targets the whole node, so there is no `--type` flag. ```bash theme={} snctl power cycle --node ``` ```bash theme={} snctl firmware version --rack-id --device-type bios ``` ### BMC **Pre-condition:** the host must be powered on. ```bash theme={} # Auto-download the POR image snctl firmware upgrade --rack-id --device-type bmc # Or flash a specific local image snctl firmware upgrade --rack-id --device-type bmc --image /path/to/bmc.bin ``` ```bash theme={} snctl power cycle --node snctl firmware version --rack-id --device-type bmc ``` ### XRDU **Pre-condition:** the host must be powered **off** — the opposite of the BIOS pre-condition. ```bash theme={} snctl power state --node --type host # Power off if needed snctl power off --node --type host ``` Both components (OBMC and RDU-C) are upgraded together. ```bash theme={} # Auto-download the POR images for both components snctl firmware upgrade --rack-id --device-type xrdu # Or flash specific local images — both components required snctl firmware upgrade --rack-id --device-type xrdu \ --image obmc=/path/to/obmc.tgz --image rduc=/path/to/rduc.tgz ``` Remember that this powers off all XRDUs on the node, regardless of any `--index` you pass. ```bash theme={} snctl power on --node --type node snctl firmware version --rack-id --device-type xrdu ``` *** ## Next step Populate a new installation in one command instead of adding devices by hand. See [Inventory file schema](srm-inventory-file-schema).