> ## Documentation Index
> Fetch the complete documentation index at: https://sambanova-systems.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Extended token generation

SambaStack supports extended token generation via the `ignore_eos` API parameter. By default, a model stops generating as soon as it produces an EOS (End of Sequence) token – a signal that it has finished its response. With `ignore_eos` enabled, the model ignores that signal and continues producing tokens until it reaches `max_tokens`. This is useful for benchmarking tools like AIPerf that require a fixed output length regardless of when the model would naturally stop.

<Note>
  `ignore_eos` is **disabled by default** for all bundles. A SambaStack administrator must enable it explicitly per bundle before users can pass the parameter in API requests. If you pass `ignore_eos: true` on a bundle where `ENABLE_IGNORE_EOS` is not set, no error is returned – the parameter is silently ignored and the model behaves as if `ignore_eos: false`.
</Note>

## Enable ignore\_eos for a bundle

To allow users to pass `ignore_eos` on a bundle, add `ENABLE_IGNORE_EOS: "true"` under `engineConfig.env_vars` in your `BundleDeployment` yaml.

```yaml theme={null}
apiVersion: sambanova.ai/v1alpha1
kind: BundleDeployment
metadata:
  name: bd-<name>
spec:
  bundle: <bundle_name>
  groups:
  - minReplicas: 1
    name: default
    qosList:
    - free
  owner: no-reply@sambanova.ai
  secretNames:
  - sambanova-artifact-reader
  engineConfig:
    startupTimeout: 7200
    env_vars:
      ENABLE_IGNORE_EOS: "true"
```

Apply the updated `BundleDeployment` yaml:

```bash theme={null}
kubectl apply -f myBundleDeployment.yaml
```

<Note>
  SambaWiz users only need to add the `engineConfig.env_vars` block to the generated YAML in the Bundle Deployment page – the rest of the spec is generated automatically.
</Note>

## Use ignore\_eos in API requests

Once a bundle has `ENABLE_IGNORE_EOS` set, users can pass `ignore_eos: true` in their chat completions request body.

<Note>
  `ignore_eos` is a vLLM extension and is not part of the standard OpenAI API. Both the SambaNova and OpenAI Python clients accept it via `extra_body`.
</Note>

<CodeGroup>
  ```python Python (SambaNova) theme={null}
  from sambanova import SambaNova

  client = SambaNova(
      base_url="your-sambanova-base-url",
      api_key="your-sambanova-api-key",
  )

  response = client.chat.completions.create(
      model="Meta-Llama-3.3-70B-Instruct",
      messages=[
          {"role": "user", "content": "Generate 500 words on the topic of renewable energy."}
      ],
      max_tokens=500,
      extra_body={"ignore_eos": True},
  )

  print(response.choices[0].message.content)
  ```

  ```python Python (OpenAI) theme={null}
  from openai import OpenAI

  client = OpenAI(
      base_url="your-sambanova-base-url",
      api_key="your-sambanova-api-key",
  )

  response = client.chat.completions.create(
      model="Meta-Llama-3.3-70B-Instruct",
      messages=[
          {"role": "user", "content": "Generate 500 words on the topic of renewable energy."}
      ],
      max_tokens=500,
      extra_body={"ignore_eos": True},
  )

  print(response.choices[0].message.content)
  ```
</CodeGroup>

## Behavior

| `ignore_eos` value | Effect                                                                                   |
| :----------------- | :--------------------------------------------------------------------------------------- |
| `false` (default)  | Model stops when it produces an EOS token or reaches `max_tokens`, whichever comes first |
| `true`             | Model ignores EOS tokens and continues generating until `max_tokens` is reached          |

<Warning>
  `ignore_eos` is intended for benchmarking and testing workloads, not production user traffic. Known concurrency limits in SambaStack 1.2 (expected to be addressed in SambaStack 2.0):

  * **Concurrency 1:** Stable.
  * **Concurrency 3:** Failures at output lengths of 64K tokens or more.
  * **Concurrency 10+:** 100% failure rate at all output sizes.
</Warning>