LlamaIndex integration guide

​

Prerequisites

• A SambaCloud account with an active API key. Create the key on the API keys page.
• Python 3.10 or higher. llama-index-llms-sambanovasystems does not support Python 3.9 or earlier.
• An embedding model, if you plan to follow Query your own documents. SambaCloud serves chat models, not embeddings, so that section runs a small embedding model on your own machine. You do not need a second provider API key for it.

​

Setup

1. Create and activate a virtual environment, then confirm the version.

python3 -m venv .venv
source .venv/bin/activate
python --version

2. Run the command below to install the llama-index-llms-sambanovasystems integration package and the SSEClient package. The integration package brings in llama-index-core, which supplies everything the examples on this page import from llama_index.core.

pip install "llama-index-llms-sambanovasystems>=0.6.0" sseclient-py

3. Set your SambaCloud API key as an environment variable. The integration reads it automatically.

export SAMBANOVA_API_KEY="your-api-key"

4. Make your first call to confirm the integration works.

from llama_index.llms.sambanovasystems import SambaNovaCloud

# Reads your key from the SAMBANOVA_API_KEY environment variable
llm = SambaNovaCloud(model="Meta-Llama-3.3-70B-Instruct")

response = llm.complete("Say hello in one sentence.")
print(response)
# → Hello! It's great to connect with you. How can I help you today?

​

Stream a response

from llama_index.llms.sambanovasystems import SambaNovaCloud

llm = SambaNovaCloud(model="Meta-Llama-3.3-70B-Instruct")

for chunk in llm.stream_complete("Explain what an LLM is in one sentence."):
    print(chunk.delta, end="", flush=True)
print()

​

Query your own documents

1. Install the local embedding integration.

pip install "llama-index-embeddings-huggingface>=0.7.0"

2. Create a document to query.

mkdir -p data
cat > data/sambanova.txt <<'EOF'
SambaCloud serves open models on SambaNova RDU hardware.
Meta-Llama-3.3-70B-Instruct accepts a context window of 128k tokens.
The LlamaIndex integration package for SambaCloud is llama-index-llms-sambanovasystems.
EOF

3. Build the index and query it.

from llama_index.core import Settings, SimpleDirectoryReader, VectorStoreIndex
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.sambanovasystems import SambaNovaCloud

# SambaCloud writes the answer.
Settings.llm = SambaNovaCloud(
    model="Meta-Llama-3.3-70B-Instruct",
    context_window=128000,
)

# A local model creates the vectors. It downloads on the first run.
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()

response = query_engine.query("What context window does Llama 3.3 70B accept?")
print(response)

Meta-Llama-3.3-70B-Instruct accepts a context window of 128k tokens.

• Settings is global. Every index and query engine you create afterwards in the same process uses the models you assigned to it, so you set them once rather than threading them through each call.
• SambaNovaCloud defaults context_window to 4096, well under the 128k that Meta-Llama-3.3-70B-Instruct accepts. Setting it to the model’s real window stops LlamaIndex from trimming retrieved text it did not need to trim. Confirm the window for your model on the SambaCloud models page.
• SimpleDirectoryReader reads plain text files with no extra packages. To load PDF, DOCX, PPTX, CSV, or Excel files, also install llama-index-readers-file.

​

Example use cases

1. Use the Patient Case Summary notebook to build a workflow that processes patient information, compares it against medical standards, and generates summaries for review.
2. Use the Contract Review notebook to build a compliance system that checks documents against provided regulations and guidelines.

​

Troubleshooting

import os

os.environ["SAMBANOVA_API_KEY"] = "your-api-key"

​

Additional resources

• SambaNova LLM integration in the LlamaIndex documentation, for constructor options, chat messages, and async calls.
• Embedding models in the LlamaIndex documentation, for other embedding options if a local model does not suit you.
• SambaCloud models for current model IDs and context windows.