Skip to main content
SambaStack supports extended token generation via the ignore_eos API parameter. By default, a model stops generating as soon as it produces an EOS (End of Sequence) token – a signal that it has finished its response. With ignore_eos enabled, the model ignores that signal and continues producing tokens until it reaches max_tokens. This is useful for benchmarking tools like AIPerf that require a fixed output length regardless of when the model would naturally stop.
ignore_eos is disabled by default for all bundles. A SambaStack administrator must enable it explicitly per bundle before users can pass the parameter in API requests. If you pass ignore_eos: true on a bundle where ENABLE_IGNORE_EOS is not set, no error is returned – the parameter is silently ignored and the model behaves as if ignore_eos: false.

​
Enable ignore_eos for a bundle

To allow users to pass ignore_eos on a bundle, add ENABLE_IGNORE_EOS: "true" under engineConfig.env_vars in your BundleDeployment yaml. 
apiVersion: sambanova.ai/v1alpha1
kind: BundleDeployment
metadata:
  name: bd-<name>
spec:
  bundle: <bundle_name>
  groups:
  - minReplicas: 1
    name: default
    qosList:
    - free
  owner: no-reply@sambanova.ai
  secretNames:
  - sambanova-artifact-reader
  engineConfig:
    startupTimeout: 7200
    env_vars:
      ENABLE_IGNORE_EOS: "true"
Apply the updated BundleDeployment yaml: 
kubectl apply -f myBundleDeployment.yaml
SambaWiz users only need to add the engineConfig.env_vars block to the generated YAML in the Bundle Deployment page – the rest of the spec is generated automatically.

​
Use ignore_eos in API requests

Once a bundle has ENABLE_IGNORE_EOS set, users can pass ignore_eos: true in their chat completions request body.
ignore_eos is a vLLM extension and is not part of the standard OpenAI API. Both the SambaNova and OpenAI Python clients accept it via extra_body.
from sambanova import SambaNova

client = SambaNova(
    base_url="your-sambanova-base-url",
    api_key="your-sambanova-api-key",
)

response = client.chat.completions.create(
    model="Meta-Llama-3.3-70B-Instruct",
    messages=[
        {"role": "user", "content": "Generate 500 words on the topic of renewable energy."}
    ],
    max_tokens=500,
    extra_body={"ignore_eos": True},
)

print(response.choices[0].message.content)

from openai import OpenAI

client = OpenAI(
    base_url="your-sambanova-base-url",
    api_key="your-sambanova-api-key",
)

response = client.chat.completions.create(
    model="Meta-Llama-3.3-70B-Instruct",
    messages=[
        {"role": "user", "content": "Generate 500 words on the topic of renewable energy."}
    ],
    max_tokens=500,
    extra_body={"ignore_eos": True},
)

print(response.choices[0].message.content)

​
Behavior

ignore_eos valueEffect
false (default)Model stops when it produces an EOS token or reaches max_tokens, whichever comes first
trueModel ignores EOS tokens and continues generating until max_tokens is reached
ignore_eos is intended for benchmarking and testing workloads, not production user traffic. Known concurrency limits in SambaStack 1.2 (expected to be addressed in SambaStack 2.0):
  • Concurrency 1: Stable.
  • Concurrency 3: Failures at output lengths of 64K tokens or more.
  • Concurrency 10+: 100% failure rate at all output sizes.