Skip to content

Core concepts

Learn the handful of ideas every Narad client uses: topics, partitions, keys, leases, acks, child topics and grants.

Topics and partitions

A topic is a named stream of messages: producers write to it, consumers read from it. Each topic is split into partitions: 3 by default, or the number you ask for when you create it. A partition is an append-only log on the disk of one node, its owner.

A topic split into partitions, and the one a key picks Your service produces the message ord_123 to the topic orders with the key customer-42. The node that accepts the produce hashes the key: hash(key) mod 3 = 1, so the message goes to the partition orders/1, whose log lives on narad-1, the partition's owner. The topic orders has 3 partitions, orders/0, orders/1 and orders/2, and each one is an append-only log on one node: narad-0, narad-1 and narad-2. A message with no key goes to the topic's next partition in turn instead, such as orders/2. Narad cluster Your service produce key=customer-42 hash(key) mod 3 = 1 ord_123 no key: the next partition in turn narad-0 orders/0 narad-1 orders/1 narad-2 orders/2 owner of orders/1 Topic orders, 3 partitions A topic split into partitions, and the one a key picks Your service produces the message ord_123 to the topic orders with the key customer-42. The node that accepts the produce hashes the key: hash(key) mod 3 = 1, so the message goes to the partition orders/1, whose log lives on narad-1, the partition's owner. The topic orders has 3 partitions, orders/0, orders/1 and orders/2, and each one is an append-only log on one node: narad-0, narad-1 and narad-2. A message with no key goes to the topic's next partition in turn instead, such as orders/2. Narad cluster Your service produce key=customer-42 hash(key) mod 3 = 1 ord_123 no key: the next partition in turn narad-0 orders/0 narad-1 orders/1 narad-2 orders/2 owner of orders/1 Topic orders, 3 partitions
Whichever node accepts the produce picks the partition: customer-42 hashes to orders/1, so in normal operation every message with that key joins the same log.

You never need to know which node owns what. Any node accepts any request, and forwards it to the owner when the work belongs elsewhere. You can add partitions to a topic later, but never remove them.

A topic keeps each message for its retention period (the operator's default: 7 days for the binary, 12 hours for a cluster installed with the Helm chart), whether or not anyone acked it. Retention removes messages in whole chunks of the log, so a message can outlive its retention period but is never removed before it.

One copy per partition

Narad stores each partition once, on the volume of the node that owns it. A crash, restart or power loss loses nothing that got a 202; losing a volume loses the partitions on it. For a second copy, add a replica child or take volume snapshots, as Back up and replicate topics shows.

Manage topics shows how to create and change topics.

Producing messages

A producer sends each message with one HTTP POST to any node. The request body is the message: any bytes up to 1 MiB, such as JSON, plain text or protobuf. There is no client library to install, although a Go SDK exists.

A 202 Accepted answer is a durability promise, not only a receipt: what a 202 means spells out which failures it survives. Produce messages covers the request in full.

Keys

A produce can carry a key, such as a customer ID. Messages with the same key go to the same partition in normal operation, which keeps related messages together for consumers and for child topics. Messages without a key are spread across the topic's partitions.

A key groups messages; it does not order them.

No ordering guarantee

Narad does not guarantee delivery order. Messages with the same key usually arrive in the order they were produced, but redelivery, a broker restart and routing around an unreachable partition owner all reorder them. If you need a sequence, carry one in the payload.

All five causes are in the delivery contract.

Leases and receipt handles

Consumers pull. A consume request reserves one message and returns it with a receipt handle. Until the topic's visibility timeout runs out (30 seconds by default), no other consumer gets that message. That reservation is a lease.

The receipt handle names one delivery of one message. You send it back to settle the message, and you treat it as an opaque string. Each delivery gets a new handle, so a handle from an earlier delivery stops working.

There are no consumer groups and no partition assignments. Run as many consumers against a topic as you need: they share its one queue, and each message goes to one of them at a time. When two services each need every message, give each its own child topic.

Ack, extend and nack

A consumer that holds a lease settles it through the ack endpoint, in one of three ways:

Action Request What happens
Ack POST /v1/topics/{topic}/ack?receipt_handle=... The message is settled and is not handed out again
Extend the same, plus &extend=true The lease restarts with a full visibility timeout from now, for slow work
Nack the same, plus &extend=0 The lease ends and the message is available to the next consume at once

Each answers 204 No Content. Acks can arrive in any order. If the lease ran out first, the request answers 410 Gone: the message is back in the queue, and another consumer may already be working on it.

Delivery is at least once

Narad delivers a message until a consumer acks it, as long as the topic's retention still holds it, and it can deliver the same message more than once: after a lease expires, after a nack, or after a broker crash. Make every handler idempotent, for example by keying its work on an ID in the payload.

Every case is in the delivery contract.

Consume and acknowledge messages shows each request with curl.

Message lifecycle

The lifecycle of one message Once its produce is committed, a message is available. Step 1: a consume leases it, here ord_123, for the topic's visibility timeout, 30 seconds by default. Step 2: an ack settles it and answers 204. An extend keeps the message leased and restarts a full timeout. A nack puts it back at once. If the lease runs out first, it goes back too, and a late ack gets 410. Retention later removes messages, whether they were acked or not. produce committed Available ord_123 1 consume Leased extend: a full timeout again for the visibility timeout, 30 s by default 2 ack: 204 Settled nack: back at once lease runs out: back, and a late ack gets 410 retention, acked or not Removed by retention The lifecycle of one message Once its produce is committed, a message is available. Step 1: a consume leases it, here ord_123, for the topic's visibility timeout, 30 seconds by default. Step 2: an ack settles it and answers 204. An extend keeps the message leased and restarts a full timeout. A nack puts it back at once. If the lease runs out first, it goes back too, and a late ack gets 410. Retention later removes messages, whether they were acked or not. produce committed Available ord_123 1 consume nack: back at once lease runs out: back, a late ack gets 410 Leased for the visibility timeout, 30 s by default extend: a full timeout again 2 ack: 204 Settled retention, acked or not Removed by retention
A lease ends one of three ways: an ack settles the message, while a nack or the clock puts it back for the next consume. Retention removes messages later, whether they were acked or not.

A settled message stays in its partition's log until retention removes it. A replay can still read it by partition and offset, without taking a lease: see Replay messages.

Child topics

A topic can have child topics. From the moment a child is attached, Narad copies every message committed to the parent into the child, key included. Each child is an ordinary topic with its own consumers, retention and pace. Messages already in the parent are not copied.

Kind What it is for
Fan-out child A second service that needs every message, consuming at its own pace
Delay child Each copy appears a fixed delay after the parent committed it, never earlier; useful for retrying later
Replica child A second copy of a topic's messages on other nodes, created in the same request as the link to its parent
Remote child (v3.2.0) A copy of a topic's messages on another Narad cluster, sent through that cluster's API

A child has one parent, and a child cannot have children of its own. Fan out and delay messages shows how to attach each kind; replica children are covered in Back up and replicate topics.

Users and grants

On a cluster with security on, every request carries a username and password over HTTP Basic auth. A user can do what its grants allow. A grant is an action on topics whose names match a pattern: an exact name such as orders, or a prefix wildcard such as orders-*.

Action Allows
produce producing to matching topics
consume consuming from, and acking on, matching topics
create creating topics with matching names; the creator owns the new topic
admin everything, including managing users; it takes no patterns

A topic's owner can change and delete it without further grants. narad server start --dev turns security off, so a local node takes requests with no credentials. Access model and grants has the full rules, and Connect and authenticate shows how a client sends credentials.

Next steps