SambaStack ModelProfile Custom Resource

A ModelProfile defines how a model architecture can be run: which PEFs it uses, which features it supports, and the batch sizes available at each sequence length tier. Profiles are reusable. One profile applies to every model and checkpoint that shares its architecture, including custom checkpoints for fine-tuned models. The profiles for supported architectures are provided with the SambaStack installation, so in most cases you reference an existing profile by name rather than authoring one. ModelProfile resources use apiVersion: sambanova.ai/v1alpha1.

Structure

apiVersion: sambanova.ai/v1alpha1
kind: ModelProfile
metadata:
  name: gemma4-31b-dyt-cd
spec:
  model_arch: gemma4-31b
  features:
  - constrained_decoding
  batchingConfigs:
    all:
      8k:
        batch_sizes: [2, 4, 6, 8]
        promptBatchingBuckets: [4096, 8192]
      32k:
        batch_sizes: [2, 4, 6, 8]
        promptBatchingBuckets: [4096, 32768]
      128k:
        batch_sizes: [2, 4, 6, 8]
        promptBatchingBuckets: [4096, 32768, 65536, 131072]
    recommended:
      8k:
        batch_sizes: [2, 4, 6, 8]
        promptBatchingBuckets: [4096, 8192]
      32k:
        batch_sizes: [2, 4, 6, 8]
        promptBatchingBuckets: [4096, 32768]
      128k:
        batch_sizes: [2]
        promptBatchingBuckets: [4096, 32768, 65536, 131072]
  pefs:
  - gemma4-31b-ss262144-bs2-dyt-1-cd:2
  - gemma4-31b-ss131072-bs8-dyt-1-cd:1
  secretNames:
  - sambanova-artifact-reader

ModelProfile Top-Level Fields

Field	Required	Description
`spec.pefs`	Yes	Ordered list of PEF custom resource references, each in `<pef-name>[:<version>]` format. The list must be non-empty and must not contain duplicate PEF names. If the version suffix is omitted, the latest version is used.
`spec.model_arch`	Yes	Model architecture targeted by this profile. All PEFs in a profile share the same `model_arch`, and this field makes that value visible without inspecting the individual `Pef` resources.
`spec.features`	Yes	Features supported by this profile, derived from its PEFs. Known values are `continuous_batching`, `prompt_caching`, `speculative_decoding`, `speculative_prefill`, `constrained_decoding`, and `multi_token_prediction`. An empty list means no optional features.
`spec.batchingConfigs`	Yes	Named batching configurations, keyed by configuration name and then by sequence length tier. See Batching configurations. If omitted, the operator generates one from the PEF grouping.
`spec.secretNames`	Yes	List of Kubernetes secrets required in order to download the PEFs. Must match secrets configured in your environment.
`spec.skip_legalizer`	No	Skips legalizer validation for this profile.
`status.batchingConfig`	N/A	Read-only. The single resolved default configuration for this profile: `recommended` if present, otherwise `all`, otherwise one generated by the operator. Published for visibility only — deployment always resolves through the bundle.

spec.batchingConfigs replaces the earlier spec.defaultBatchingConfig field. The old field held one tier map directly; the new field holds a map of named tier maps. A profile written against the old field needs its tier map nested under a configuration name, conventionally all.

Batching configurations

spec.batchingConfigs is a two-level map: a configuration name, then a sequence length tier, then the settings for that tier. Two configuration names are meaningful to the operator:

Name	Description
`all`	Every batch size the profile’s PEFs support at every tier. Always present on the profiles SambaNova provides.
`recommended`	A curated, narrower subset — typically fewer batch sizes at the longest tiers, where the widest fan-out costs the most memory. Present only when it differs from `all`.

Any other name is accepted and can be selected by name from a bundle, but the operator treats recommended and all specially when resolving the default. Common tier keys are 8k, 16k, 32k, 64k, and 128k. PEFs below 1k use a bare integer key, for example 448, and vocoder PEFs use a codes-length key in the form <n>t, for example 10t.

spec:
  batchingConfigs:
    all:
      128k:
        batch_sizes: [1]
        promptBatchingBuckets: [4096, 32768, 131072]
      32k:
        batch_sizes: [1, 2, 4]
      8k:
        batch_sizes: [1, 2, 4, 8]
      4k:
        batch_sizes: [1, 2, 4, 8]
        is_default: true

Parameter	Required	Description
`batch_sizes`	Yes	Batch sizes served at this tier. In a profile’s own configuration, an explicit list of integers, for example `[1, 2, 4]`. In an override, this can also be the string `"*"`. See Overriding a batching configuration.
`promptBatchingBuckets`	No	Prompt lengths this tier advertises to the gateway for batching-queue routing. When left unset, the operator generates the buckets from the PEFs the tier actually references.
`is_default`	No	If `true`, this tier is served as the `default` expert. At most one tier per configuration may set this.
Additional keys	No	Any other key set on a tier is passed through unchanged to the expert built from it.

Including multiple batch sizes for each tier allows the inference engine to select the smallest and fastest configuration based on current workload.

Overriding a batching configuration

A profile establishes the batch sizes available at each tier, so its own configurations always list them explicitly. They never use "*". You override a profile’s batching for a single model through modelConfigs[].batchingConfig in a ModelBundle, or through the inline spec.models of a ModelDeployment. An override takes one of two forms. Select one of the profile’s configurations by name. Set batchingConfig to a string naming an entry in the profile’s batchingConfigs:

spec:
  modelConfigs:
  - model: gemma-4-31b-it:1
    profile: gemma4-31b-dyt-cd
    batchingConfig: all      # use the profile's full fan-out instead of its recommended subset

Naming a configuration the profile does not define is a validation error on the owning ModelBundle or ModelDeployment. Or spell out the tiers inline. Only in an inline override can batch_sizes be set to "*", which selects every batch size the profile provides for that tier:

spec:
  modelConfigs:
  - model: gpt-oss-120b:1
    profile: gpt-oss-fp8-dyt
    batchingConfig:
      128k:
        batch_sizes: "*"    # every batch size the profile defines for 128k
      32k:
        batch_sizes: [2, 4] # a subset of what the profile defines for 32k

An override can only narrow or restate what the profile already provides. It cannot introduce a batch size the profile does not contain. You only need to spell out what you are actually changing: any tier that omits promptBatchingBuckets has them backfilled from the PEFs that tier references.

Resolution order

The batching configuration used for a model is resolved in this order, first match winning:

The model configuration’s batchingConfig, whether a name or an inline map.
The profile’s recommended configuration, if it defines one.
The profile’s all configuration.
A configuration generated by the operator from the PEF grouping.

PEF

The compiled executables a profile references

Model

The checkpoints a profile is paired with

ModelBundle

Where a profile’s batching configuration can be overridden

Deploying models and bundles

How profiles relate to the other resources

Getting Started

Service Administration

Hardware Administration

Reference Architecture

Resources

ModelProfile

Structure

Batching configurations

Overriding a batching configuration

Resolution order

PEF

Model

ModelBundle

Deploying models and bundles

​Structure

​Batching configurations

​Overriding a batching configuration

​Resolution order

​Related resources

PEF

Model

ModelBundle

Deploying models and bundles

Structure

Batching configurations

Overriding a batching configuration

Resolution order

Related resources