Skip to content

Deploy on Kubernetes

Install a three-node Narad cluster on Kubernetes with the Helm chart, then check that it answers.

Before you start: a Kubernetes cluster with a default storage class, kubectl, Helm 3 or later, git and openssl.

Decide before you install

Three choices are cheap before the install and costly after it: the disk size, the Raft transport and the founding cluster size. Keep them, and the rest of your settings, in a values file. This one covers the choices most production clusters make:

narad-values.yaml
replicaCount: 3
initialClusterSize: 3        # set once, at the first install

image:
  tag: v3.2.2                # pin a release

persistence:
  size: 50Gi                 # per pod; fixed once installed

narad:
  defaultRetentionAgeMs: 604800000  # 7 days; the chart's default is 12 hours
  config:                    # rendered to the --config JSON file
    storage:
      codec: zstd            # compression is off by default
      compression_level: fastest

security:
  clusterTLS:
    enabled: true            # needs the narad-cluster-tls secret
  allowPlaintextRaft: false

resources:
  limits:
    memory: 2Gi

extraEnv:
  GOMEMLIMIT: 1800MiB        # about 90% of the memory limit

Save it as narad-values.yaml in the clone you make under Install, or pass its path to -f.

  • persistence.size is the volume of each pod. The StatefulSet fixes it once installed, and a helm upgrade that changes it fails. Size it with Capacity and disk sizing.
  • narad.defaultRetentionAgeMs is the retention of a topic created without retention_ms. The chart's default is 12 hours and the binary's is 7 days. Retention deletes unacked messages too, so pick a value longer than your longest consumer outage.
  • Raft TLS. For production, keep security.clusterTLS.enabled: true and security.allowPlaintextRaft: false in the file, and create the TLS secret in Install step 2. Turning TLS on later needs a pause of topic and user changes (Enable on a running cluster).
  • initialClusterSize is the set of pods allowed to create a new cluster. Pods beyond it join the existing one. It is read only on an empty disk, so set it at the first install and leave it.
  • codec: zstd turns on compression of stored messages. Measure the saving on your own payloads; already compressed payloads gain little.
  • GOMEMLIMIT gives the Go runtime a soft memory limit, so it collects garbage harder before the pod reaches its memory limit. From v3.1.0, Narad sets it to 90% of the pod's memory limit on its own when it is unset.

Every value, with its default, is in the Helm values reference.

Install

The chart lives in the repository, so the install starts from a clone of the release you mean to run.

What the Helm chart installs Clients reach an ingress or load balancer, where TLS ends. A message, ord_123, goes on as plain HTTP to the Service narad on 7942/tcp, which spreads requests over the three pods of the StatefulSet narad, narad-0 to narad-2. Each pod has its own volume, from the claims data-narad-0 to data-narad-2. The pods talk to each other directly, node RPC on 7942/udp and Raft on 7943/tcp, and find each other by DNS through the headless Service narad-headless. The pods read the secret narad-security as environment variables. The secret narad-cluster-tls is mounted only with clusterTLS.enabled. Prometheus scrapes :9100/metrics on every pod. Clients Ingress or load balancer TLS ends here ord_123 Service narad 7942/tcp narad-0 data-narad-0 narad-1 data-narad-1 narad-2 data-narad-2 node RPC 7942/udp Raft 7943/tcp headless Service narad-headless StatefulSet narad narad-security read as env vars narad-cluster-tls mounted, only with clusterTLS.enabled Prometheus :9100/metrics What the Helm chart installs Clients reach an ingress or load balancer, where TLS ends. A message, ord_123, goes on as plain HTTP to the Service narad on 7942/tcp, which spreads requests over the three pods of the StatefulSet narad, narad-0 to narad-2. Each pod has its own volume, from the claims data-narad-0 to data-narad-2. The pods talk to each other directly, node RPC on 7942/udp and Raft on 7943/tcp, and find each other by DNS through the headless Service narad-headless. The pods read the secret narad-security as environment variables. The secret narad-cluster-tls is mounted only with clusterTLS.enabled. Prometheus scrapes :9100/metrics on every pod. Clients Ingress or load balancer TLS ends here ord_123 Service narad 7942/tcp Prometheus :9100/metrics narad-0 data-narad-0 narad-1 data-narad-1 narad-2 data-narad-2 node RPC 7942/udp Raft 7943/tcp headless Service narad-headless StatefulSet narad narad-security read as env vars narad-cluster-tls mounted, only with clusterTLS.enabled
For a release called narad: TLS ends at your ingress, clients reach the pods through the Service narad on 7942/tcp, and the pods reach each other directly on 7942/udp and 7943/tcp. Each pod keeps its own volume, data-narad-0 for narad-0 and so on.

  1. Get the chart and create a namespace:

    git clone --branch v3.2.2 --depth 1 \
  https://github.com/DebanganThakuria/narad
cd narad
kubectl create namespace narad

  2. Create the security secret. The chart reads a secret named <release>-security, so for a release called narad it is narad-security:

    kubectl create secret generic narad-security -n narad \
  --from-literal=cluster-secret="$(openssl rand -base64 32)" \
  --from-literal=admin-password="$(openssl rand -base64 24)"

    cluster-secret authenticates the nodes to each other and is required. admin-password becomes the password of the root user, admin. It is optional: without it, one node generates a password and writes it to a file on its own volume (see Manage users and grants). With Raft TLS on, also create the narad-cluster-tls secret now (Create the certificates).

  3. Install the chart with your values file:

    helm install narad ./charts/narad -n narad -f narad-values.yaml

    For a trial, skip the file and set the few values that matter on the command line. This runs the Raft port without TLS, fenced to the Narad pods by the chart's NetworkPolicy, which is on by default:

    helm install narad ./charts/narad -n narad \
  --set replicaCount=3 \
  --set persistence.size=50Gi \
  --set image.tag=v3.2.2

    The policy holds only on a CNI that enforces NetworkPolicy (Fence the cluster ports). Turn it off (networkPolicy.enabled=false) without TLS or security.allowPlaintextRaft, and the v3.1.0 chart refuses to install and names the alternatives; the v3.0.1 chart has no policy, ignores networkPolicy.enabled and runs the Raft port unfenced.

    Pin image.tag to a release either way. The chart's default is latest, which follows master.

  4. Wait for the rollout:

    kubectl rollout status statefulset/narad -n narad

    A pod reports ready once it has finished starting, is in contact with the Raft leader and has caught up with it. The rollout is done when all three pods are ready.

Work through the Production checklist before the cluster takes real traffic.

Verify

Forward the API port to your machine and keep the command running:

kubectl port-forward -n narad svc/narad 7942:7942

In a second terminal, read the admin password from the secret and call the cluster:

export NARAD=http://127.0.0.1:7942
export AUTH="admin:$(kubectl get secret narad-security -n narad \
  -o jsonpath='{.data.admin-password}' | base64 -d)"
curl -s "$NARAD/readyz"
curl -s -u "$AUTH" "$NARAD/v1/topics"
Output
{"status":"ready"}
{"next_page_token":"","topics":[]}
  • $NARAD is the base URL of any node or of the load balancer in front of them.
  • $AUTH is username:password for a user with the grant the request needs. Here it is the root admin.

/readyz needs no credentials, and the topic list is empty on a new cluster. Next, create a user for each service (Manage users and grants) and point clients at the cluster (Connect and authenticate).

Verify the image

New in v3.1.0.

Images are signed keyless with cosign and carry an SBOM and build provenance. Signing starts with the first image built after the v3.0.1 release. Images up to and including v3.0.1 carry no signature, so the command below fails for them.

IDENTITY='^https://github\.com/DebanganThakuria/narad/'
IDENTITY+='\.github/workflows/container\.yml@refs/(heads/master|tags/v.*)$'
cosign verify ghcr.io/debanganthakuria/narad:v3.2.2 \
  --certificate-identity-regexp "$IDENTITY" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

The identity is pinned to one workflow file, on master or on a release tag. Matching the repository alone would accept a signature from any workflow on any branch.

Change values later

Edit the values file and upgrade:

helm upgrade narad ./charts/narad -n narad -f narad-values.yaml

Adding or removing narad.config as a whole changes the pods, so Helm rolls them. A change inside it only rewrites the ConfigMap, and a pod reads its config file only when it starts, so restart the pods after one:

kubectl rollout restart statefulset/narad -n narad

Next steps