Build with OGX - SambaNova Integration Guide

In this guide, you’ll learn how to set up and use OGX, an open-source, OpenAI-compatible API server for building AI applications. OGX (formerly Llama Stack) provides a standardized interface that works with any model and any infrastructure. This guide walks you through running the SambaNova distribution server, installing the client, and running your first model inference. OGX includes two main components:

Server – A running distribution that hosts inference providers and API adapters.
Client – A consumer of the server’s API, interacting with the hosted adapters.

Prerequisites

Before you begin, ensure you have:

A SambaCloud account
A SambaNova API key
Python 3.12 or higher installed on your machine

Setup

Option 1: Install and run with uv (recommended)

Set up a Python virtual environment

python -m venv .venv
source .venv/bin/activate

Install required dependencies

pip install uv
uv pip install "ogx[starter]"

Export required environment variables and start the server

export OGX_PORT=8321
export SAMBANOVA_API_KEY="your-api-key-here"

uv run ogx run starter --port $OGX_PORT --insecure

The --insecure flag disables TLS for local development. Without it, the server exits with TLS required: set tls_certfile/tls_keyfile in server config or pass '--insecure' to disable.

Option 2: Run with Docker

export OGX_PORT=8321
export SAMBANOVA_API_KEY="your-api-key-here"

docker run -it \
  -p $OGX_PORT:$OGX_PORT \
  -v ~/.ogx:/root/.ogx \
  -e SAMBANOVA_API_KEY=$SAMBANOVA_API_KEY \
  ogxai/distribution-starter \
  --port $OGX_PORT \
  --insecure

Install the OGX client

Both options need the client to run the example below. In the same or another environment, run:

pip install ogx-client

Example code

The following Python code demonstrates basic usage:

from ogx_client import OgxClient

OGX_PORT = 8321
client = OgxClient(base_url=f"http://localhost:{OGX_PORT}")

# List all available models
models = client.models.list()
print("--- Available models: ---")
for m in models.data:
    print(f"- {m.id}")
print()

# Choose a model from the list
model = "sambanova/Meta-Llama-3.3-70B-Instruct"

# Run chat completion
response = client.chat.completions.create(
    model=model,
    messages=[
        {"role": "system", "content": "You are a friendly assistant."},
        {"role": "user", "content": "Write a two-sentence poem about llamas."},
    ],
)

print(response.choices[0].message.content)

This demonstrates the full client-server loop: connecting, listing models, and running inference. Explore the SambaNova OGX integration repo to find several use cases using SambaNova distribution LLMs, embeddings, tools, and agent adapters.

Troubleshooting

TLS required

The server refuses to start without TLS configured:

TLS required: set tls_certfile/tls_keyfile in server config or pass '--insecure' to disable.

Add the --insecure flag to the server run command, as shown in the setup steps above. Use TLS certificates instead if you are running in a shared or production environment.

Incorrect API key provided

The key reached SambaCloud but was rejected:

Incorrect API key provided: ... You can find your API key at https://cloud.sambanova.ai/apis

Verify SAMBANOVA_API_KEY is exported in the environment where the server is running, not just the one running the client, and that it matches a valid key from the SambaCloud console.

Package 'ogx' requires a different Python

pip refuses the install outright:

ERROR: Package 'ogx' requires a different Python: 3.11.x not in '>=3.12'

ogx requires Python 3.12 or later. Create the virtual environment with an explicit interpreter, for example python3.12 -m venv .venv. A system python3 older than 3.12 is the usual cause.

The server exits on startup with the port already in use

Another process is holding $OGX_PORT. Identify it, then either stop it or choose a free port:

lsof -i :$OGX_PORT
export OGX_PORT=8322

Both the uv run and docker run commands read $OGX_PORT, so nothing else needs to change.

Additional resources

Refer to the OGX documentation to:

Understand core concepts
Explore sample applications
Learn how to extend and customize the framework

Overview

Developer tools

Frameworks

Interfaces and low-code

Data and infrastructure

Multimodal

Evaluation and monitoring

OGX integration guide

Prerequisites

Setup

Option 1: Install and run with uv (recommended)

Option 2: Run with Docker

Install the OGX client

Example code

Troubleshooting

Additional resources

​Prerequisites

​Setup

​Option 1: Install and run with uv (recommended)

​Option 2: Run with Docker

​Install the OGX client

​Example code

​Troubleshooting

​Additional resources

Prerequisites

Setup

Option 1: Install and run with uv (recommended)

Option 2: Run with Docker

Install the OGX client

Example code

Troubleshooting

Additional resources