> ## Documentation Index
> Fetch the complete documentation index at: https://sambanova-systems.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Log forwarding with Fluent Bit

Fluent Bit is a lightweight log processor and forwarder used in the SambaStack monitoring reference architecture. It collects container logs from Kubernetes nodes, enriches them with metadata, and ships them to OpenSearch for indexing and search.

<Note>
  **Reference Architecture Note:** This setup uses third-party components. Versions, defaults, and command syntax may change over time. Address any issues not specific to SambaStack to the vendor or project that owns that component.
</Note>

## Prerequisites

Before deploying Fluent Bit, ensure you have:

* **Kubernetes cluster** – A running cluster with SambaStack installed
* **OpenSearch** – Deployed in the `monitoring` namespace. See [Log Storage - OpenSearch](/en/v1.2.0/sambastack/reference-architecture/observability/log-storage)
* **OpenSearch admin password** – The password configured during OpenSearch deployment
* **Helm** (latest version) – For deploying the Fluent Bit chart
* **kubectl** – Configured with access to your target Kubernetes cluster

<Warning>
  Fluent Bit requires a running OpenSearch instance. Complete the [OpenSearch deployment](/en/v1.2.0/sambastack/reference-architecture/observability/log-storage) before proceeding.
</Warning>

## Resource requirements

The following are minimum resource recommendations for Fluent Bit:

| Component             | CPU Request | Memory Request | Storage |
| --------------------- | ----------- | -------------- | ------- |
| Fluent Bit (per node) | 100m        | 128Mi          | –       |

<Note>
  Fluent Bit runs as a DaemonSet with one pod per node. Total cluster resources scale linearly with node count.
</Note>

## Architecture overview

In the SambaStack monitoring reference architecture:

* Fluent Bit runs as a DaemonSet, ensuring one log forwarder per Kubernetes node
* Collects logs from all pods and containers via `/var/log/pods/`
* Enriches logs with namespace, pod, and container metadata using a Lua filter
* Forwards logs to OpenSearch using secure TLS connections on port 9200

This deployment uses a custom Fluent Bit configuration stored in a Kubernetes Secret and a Helm-based installation with custom values.

<Note>
  **How Fluent Bit log collection works:**

  * Kubernetes automatically writes container logs to `/var/log/pods/` on each node
  * Fluent Bit reads these logs directly from the host filesystem via a DaemonSet
  * Multiple parsers are configured to support both CRI and Docker log formats
  * Fluent Bit automatically selects the correct parser at runtime
  * Logs are enriched with Kubernetes metadata and forwarded to OpenSearch
</Note>

## Deployment steps

### Step 1: Add the Helm repository

```bash theme={null}
helm repo add fluent https://fluent.github.io/helm-charts
helm repo update
```

### Step 2: Retrieve the OpenSearch password

Get the OpenSearch admin password you created during [OpenSearch deployment](/en/v1.2.0/sambastack/reference-architecture/observability/log-storage):

```bash theme={null}
kubectl -n monitoring get secret opensearch-initial-admin-password -o json \
  | jq -r '.data.OPENSEARCH_INITIAL_ADMIN_PASSWORD | @base64d'
```

Save this password for use in Step 6.

### Step 3: Create the Lua script

Save the following as `~/.sambastack-observability/fluentbit/append_tags.lua`:

```lua theme={null}
function append_tags(tag, timestamp, record)
  local filename = record["filename"] or ""
  local namespace_tag = string.match(filename, "/pods/([^_]+)")
  local container_tag = string.match(filename, "/pods/.*/([^/]+)/")
  local pod_tag = string.match(filename, "/pods/[^_]+_([^_]+)")

  if namespace_tag then record["namespace"] = namespace_tag end
  if pod_tag then record["pod"] = pod_tag end
  if container_tag then record["container"] = container_tag end

  return 1, timestamp, record
end
```

<Note>
  Using a separate Lua file is less error-prone than embedding the code inline in the configuration file.
</Note>

### Step 4: Create the configuration file

Save the following as `~/.sambastack-observability/fluentbit/fluentbit-conf.conf`:

```ini theme={null}
[INPUT]
  Buffer_Chunk_Size 10MB
  Buffer_Max_Size 64MB
  Exclude_Path /var/log/pods/*fluent*/*/*.log
  Log_Level info
  Mem_Buf_Limit 64MB
  Name tail
  Path /var/log/pods/*/*/*.log
  Path_Key filename
  Rotate_wait 120
  Skip_Long_Lines On
  Tag kube.*
  multiline.parser cri, docker

[FILTER]
  Match  kube.*
  Name  lua
  script /fluent-bit/sn/append_tags.lua
  call  append_tags

[FILTER]
  Key_Name  log
  Match  kube.*
  Name  parser
  Parser  json
  Reserve_Data  On

[FILTER]
  Key_Name  log
  Match  kube.*
  Name  parser
  Parser  k8s-nginx-ingress
  Reserve_Data  On

[OUTPUT]
  HTTP_Passwd  ${DEFAULT_OPENSEARCH_HTTP_PASS}
  HTTP_User  ${DEFAULT_OPENSEARCH_HTTP_USER}
  Host  ${DEFAULT_OPENSEARCH_HOST}
  Index  logs-7d
  Log_Level  info
  Match  kube.*
  Name  opensearch
  Port  ${DEFAULT_OPENSEARCH_PORT}
  Retry_Limit  False
  Suppress_Type_Name  On
  tls  On
  tls.verify  Off

[SERVICE]
  Buffer_Size 64MB
  Daemon Off
  Flush 10
  HTTP_Listen 0.0.0.0
  HTTP_Port 2020
  HTTP_Server On
  Health_Check On
  Log_Level debug
  Parsers_File /fluent-bit/etc/parsers.conf
  Trace_Error On
```

<Accordion title="Configuration explanation">
  **INPUT section:** Tails log files from all pods, excluding Fluent Bit's own logs to prevent recursion.

  **FILTER (Lua):** Runs the external `append_tags.lua` script to extract namespace, pod, and container names from the log file path.

  **FILTER (parser):** Attempts to parse log content as JSON or nginx ingress format.

  **OUTPUT:** Sends logs to OpenSearch with TLS enabled. The `logs-7d` index name is configurable.

  **SERVICE:** Enables the HTTP server on port 2020 for health checks and metrics.
</Accordion>

### Step 5: Create the configuration secret

```bash theme={null}
kubectl -n monitoring create secret generic sn-fluentbit \
  --from-file=sn-fluentbit.conf=~/.sambastack-observability/fluentbit/fluentbit-conf.conf \
  --from-file=append_tags.lua=~/.sambastack-observability/fluentbit/append_tags.lua \
  --dry-run=client -o yaml | kubectl apply -f -
```

<Warning>
  The secret name `sn-fluentbit` must match the `secretName` in the Helm values file. If you change this name, update the values file accordingly.
</Warning>

### Step 6: Create the Helm values file

Save the following as `~/.sambastack-observability/fluentbit/fluentbit-values.yml`:

```yaml theme={null}
env:
- name: DEFAULT_OPENSEARCH_HOST
  value: "opensearch-cluster-master.monitoring.svc.cluster.local"
- name: DEFAULT_OPENSEARCH_PORT
  value: "9200"
- name: DEFAULT_OPENSEARCH_HTTP_USER
  value: "admin"
- name: DEFAULT_OPENSEARCH_HTTP_PASS
  valueFrom:
    secretKeyRef:
      name: opensearch-initial-admin-password
      key: OPENSEARCH_INITIAL_ADMIN_PASSWORD

securityContext:
  privileged: true

tolerations:
- operator: Exists

daemonSetVolumes:
- name: var
  hostPath:
    path: /var
- name: etcmachineid
  hostPath:
    path: /etc/machine-id
    type: File
- name: sn-fluentbit
  secret:
    secretName: sn-fluentbit

daemonSetVolumeMounts:
- name: var
  mountPath: /var
  readOnly: true
- name: etcmachineid
  mountPath: /etc/machine-id
  readOnly: true
- name: sn-fluentbit
  mountPath: /fluent-bit/sn
  readOnly: true

args:
- --workdir=/fluent-bit/etc
- --config=/fluent-bit/sn/sn-fluentbit.conf

updateStrategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 5

config:
  inputs: ""
  filters: ""
  outputs: ""

image:
  repository: fluent/fluent-bit
  tag: "4.2.2"
```

### Step 7: install Fluent Bit

```bash theme={null}
helm -n monitoring upgrade --install fluent-bit fluent/fluent-bit \
  -f ~/.sambastack-observability/fluentbit/fluentbit-values.yml
```

## Verification

### Check pod status

Verify Fluent Bit pods are running on each node:

```bash theme={null}
kubectl -n monitoring get po -l app.kubernetes.io/name=fluent-bit
```

Expected output (one pod per node):

```text theme={null}
NAME               READY   STATUS    RESTARTS   AGE
fluent-bit-4zh8w   1/1     Running   0          20h
fluent-bit-sdmxg   1/1     Running   0          20h
```

### Verify logs are reaching OpenSearch

Port-forward to the OpenSearch service:

```bash theme={null}
kubectl -n monitoring port-forward svc/opensearch-cluster-master 9200:9200 &
```

List indices to confirm log ingestion:

```bash theme={null}
curl -u "admin:<password>" -k "https://localhost:9200/_cat/indices?v"
```

Expected output:

```text theme={null}
health status index     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   logs-7d   0ahKcye3SfajNksbfapY4w   1   1     296144            0     58.1mb         58.1mb
```

The presence of the `logs-7d` index with a non-zero `docs.count` confirms Fluent Bit is successfully pushing logs into OpenSearch.

To stop the port-forward process:

```bash theme={null}
pkill -f "port-forward.*opensearch"
```

### Query sample logs

Verify log content with a sample query:

```bash theme={null}
curl -u "admin:<password>" -k "https://localhost:9200/logs-7d/_search?size=1&pretty"
```

You should see log entries with `namespace`, `pod`, and `container` fields populated.

## Success criteria

Deployment is considered successful when:

* Fluent Bit DaemonSet pods are in `Running` state on all nodes
* The `logs-7d` index appears in OpenSearch with documents
* Sample log queries return entries with namespace/pod/container metadata

## Configuration reference

| Section | Parameter           | Description                                           |
| ------- | ------------------- | ----------------------------------------------------- |
| INPUT   | `Buffer_Chunk_Size` | Initial buffer size for reading log files (10MB)      |
| INPUT   | `Buffer_Max_Size`   | Maximum buffer size per monitored file (64MB)         |
| INPUT   | `Mem_Buf_Limit`     | Memory buffer limit before backpressure (64MB)        |
| INPUT   | `Rotate_wait`       | Seconds to wait before reading rotated files (120)    |
| INPUT   | `Exclude_Path`      | Glob pattern for logs to skip (Fluent Bit's own logs) |
| OUTPUT  | `Index`             | OpenSearch index name for logs (`logs-7d`)            |
| OUTPUT  | `tls`               | Enable TLS for OpenSearch connection                  |
| OUTPUT  | `tls.verify`        | Verify TLS certificate (Off for self-signed)          |
| SERVICE | `Flush`             | Interval in seconds to flush records to output (10)   |
| SERVICE | `HTTP_Port`         | Port for Fluent Bit's HTTP server metrics (2020)      |

## Troubleshooting

### Pods not starting (ImagePullBackOff)

**Symptom:** Fluent Bit pods show `ImagePullBackOff` or `ErrImagePull`.

**Solution:** Verify your cluster can reach the container registry. Check for network policies or proxy requirements.

### No logs appearing in OpenSearch

**Symptom:** The `logs-7d` index doesn't exist or has zero documents.

**Possible causes and solutions:**

1. **OpenSearch not reachable:**
   ```bash theme={null}
   # Check Fluent Bit logs for connection errors
   kubectl -n monitoring logs -l app.kubernetes.io/name=fluent-bit --tail=50
   ```

2. **Wrong password:**
   Verify the password in `fluentbit-values.yml` matches the OpenSearch admin password.

3. **Secret not mounted:**
   ```bash theme={null}
   kubectl -n monitoring describe pod -l app.kubernetes.io/name=fluent-bit | grep -A5 Mounts
   ```

### Logs missing metadata fields

**Symptom:** Logs appear in OpenSearch but `namespace`, `pod`, or `container` fields are missing.

**Cause:** The Lua filter failed to parse the log file path.

**Solution:** Check that logs are coming from the expected path pattern (`/var/log/pods/*/*/*.log`).

### High memory usage

**Symptom:** Fluent Bit pods are OOMKilled or show high memory consumption.

**Solution:** Reduce buffer sizes in the configuration:

```ini theme={null}
[INPUT]
  Buffer_Chunk_Size 5MB
  Buffer_Max_Size 32MB
  Mem_Buf_Limit 32MB
```

## Next steps

After Fluent Bit is forwarding logs:

1. **Deploy Prometheus and Grafana** – Add visualization dashboards for your logs. See [Monitoring with Prometheus and Grafana](/en/v1.2.0/sambastack/reference-architecture/observability/monitoring-prom).

2. **Explore logs in Grafana** – Once Grafana is deployed with the OpenSearch datasource, you can query and visualize logs using the Explore feature.

## Cleanup

To remove Fluent Bit from your cluster:

Uninstall the Helm release:

```bash theme={null}
helm uninstall fluent-bit -n monitoring
```

Delete the configuration secret:

```bash theme={null}
kubectl delete secret sn-fluentbit -n monitoring
```

<Note>
  Removing Fluent Bit stops log forwarding but does not delete existing logs in OpenSearch. To remove historical logs, delete the index directly in OpenSearch or wait for index lifecycle policies to expire the data.
</Note>