Kafka

Distributed event streaming with KRaft consensus (no ZooKeeper), SASL/SCRAM authentication, and TLS.

How a topic is replicated

A topic is split into partitions for parallelism, and each partition is replicated across brokers for fault tolerance. Every partition has one leader (which handles all reads and writes for that partition) and a set of follower replicas. Replicas that have caught up with the leader form the in-sync replica set (ISR). Producers write to the partition leaders; followers in the ISR copy those records; consumers in a group read from the leaders. The brokers also run the KRaft controller quorum that stores cluster metadata and elects a new leader from the ISR if a broker fails.

Partitions, replication & leader failover (KRaft)

ELECTION Broker 1 offline · ISR follower promoted to P0 leader

Produceracks=allwrite →Brokers3 · KRaft⇢ ISRread →Consumer groupmy-consumer-group

Producer writeBroker (leader)replicate to ISRConsumer readKRaft quorum :9094Failed broker

Partitions, replicas and the ISR

Partition. The unit of parallelism and ordering. Records are ordered within a partition, not across partitions. A record's key decides its partition, so all records for the same key land on the same partition and stay ordered.
Replication factor. The number of brokers that hold a copy of each partition. With replication_factor 3, every partition exists on three brokers: one leader and two followers.
Leader. Serves all produce and consume traffic for its partition. Leadership is spread across brokers so no single broker is a bottleneck.
In-sync replica set (ISR). The replicas (including the leader) that are caught up with the leader. Only an ISR member can be promoted to leader, which is what makes failover lossless when acks=all is used.
min.insync.replicas. With producer acks=all, a write is only acknowledged once at least this many replicas (counting the leader) have it. The default of 2 means a write survives the loss of any one broker.

Durability with `acks`

The producer's acks setting trades latency for durability:

`acks`	Meaning	Durability
`0`	Fire-and-forget, no acknowledgement	Lowest
`1`	Leader has written the record	Medium
`all`	Leader plus the required ISR members have the record	Highest

Pair acks=all with min.insync.replicas=2 so a write is only acknowledged when it is safely on more than one broker.

Leader failover (KRaft)

There is no ZooKeeper. The brokers form a KRaft controller quorum that holds cluster metadata. When a broker fails, its partition leaderships are lost; the controller quorum elects a new leader for each affected partition from that partition's ISR. Producers and consumers transparently reconnect to the new leader. Watch the under_replicated_partitions metric to see replicas catching back up after a broker returns.

Versions

Version	Status	Notes
4.0	Available	Recommended
3.9	Available
3.6	Available

Connecting

Parameter	Value
Bootstrap	`{name}.db.foundrydb.com:9093`
Security protocol	`SASL_SSL`
SASL mechanism	`SCRAM-SHA-256`

Full connection examples for all languages: Connection Strings →

Topics

Create a topic

curl -u admin:password -X POST \
  https://api.foundrydb.com/managed-services/{id}/kafka/topics \
  -H "Content-Type: application/json" \
  -d '{
    "name": "user-events",
    "partitions": 6,
    "replication_factor": 3,
    "config": {
      "retention.ms": "604800000",
      "cleanup.policy": "delete"
    }
  }'

List topics

curl -u admin:password \
  https://api.foundrydb.com/managed-services/{id}/kafka/topics

Update topic config

curl -u admin:password -X PATCH \
  https://api.foundrydb.com/managed-services/{id}/kafka/topics/user-events \
  -H "Content-Type: application/json" \
  -d '{"config": {"retention.ms": "2592000000"}}'

Delete a topic

curl -u admin:password -X DELETE \
  https://api.foundrydb.com/managed-services/{id}/kafka/topics/user-events

Topic Configuration Reference

Config key	Default	Description
`retention.ms`	604800000 (7d)	How long to keep messages
`retention.bytes`	-1 (unlimited)	Max bytes per partition
`cleanup.policy`	`delete`	`delete` (by time/size) or `compact` (keep latest key)
`compression.type`	`producer`	`gzip`, `snappy`, `lz4`, `zstd`, `uncompressed`
`min.insync.replicas`	2	Minimum in-sync replicas for `acks=all`
`max.message.bytes`	1048576 (1MB)	Maximum message size

Users and ACLs

Each database user gets SASL credentials. Scope permissions per topic:

# Create a producer user
curl -u admin:password -X POST \
  https://api.foundrydb.com/managed-services/{id}/database-users \
  -H "Content-Type: application/json" \
  -d '{
    "username": "my-producer",
    "acls": [
      {"topic": "user-events", "operation": "write"},
      {"topic": "user-events", "operation": "describe"}
    ]
  }'

# Create a consumer user
curl -u admin:password -X POST \
  https://api.foundrydb.com/managed-services/{id}/database-users \
  -H "Content-Type: application/json" \
  -d '{
    "username": "my-consumer",
    "acls": [
      {"topic": "user-events", "operation": "read"},
      {"group": "my-consumer-group", "operation": "read"}
    ]
  }'

Consumer Groups

A consumer group lets several consumers share the work of reading a topic. Each partition of the topic is assigned to exactly one consumer in the group, so adding consumers (up to the partition count) increases read throughput. If a consumer joins or leaves, the group rebalances and partitions are reassigned. The group tracks its committed offset per partition so it can resume where it left off. Each group is independent, so the same records can be read by many groups for different purposes.

# List consumer groups
curl -u admin:password \
  https://api.foundrydb.com/managed-services/{id}/kafka/consumer-groups

# Get group lag
curl -u admin:password \
  https://api.foundrydb.com/managed-services/{id}/kafka/consumer-groups/my-group/lag

Consumer lag is the gap between the latest offset on a partition and the group's committed offset. Monitor it to detect processing bottlenecks: high or growing lag means consumers are falling behind producers. To catch up, add consumers (so more partitions are read in parallel, capped at the partition count), speed up per-record processing, or increase the topic's partition count.

Listeners and authentication

Every connection is authenticated with SASL; there is no anonymous access. The cluster exposes separate listeners for external clients, agent-internal admin operations, and the KRaft controller quorum:

Listener	Port	Protocol	Used by
External	9093	`SASL_SSL` · `SCRAM-SHA-256`	Your producers and consumers
Internal	9092	`SASL` · `SCRAM-SHA-256`	Agent admin operations (broker-local)
Controller	9094	`SASL` · KRaft quorum	Inter-broker metadata consensus

Application traffic always uses the external SASL_SSL listener on port 9093 with SCRAM-SHA-256. The internal and controller listeners are kept off the public path and carry only platform and cluster-internal traffic.

Scaling

Add brokers to increase throughput and fault tolerance:

curl -u admin:password -X POST \
  https://api.foundrydb.com/managed-services/{id}/nodes \
  -H "Content-Type: application/json" \
  -d '{"role": "broker"}'

After adding brokers, rebalance existing topic partitions to spread load:

curl -u admin:password -X POST \
  https://api.foundrydb.com/managed-services/{id}/kafka/rebalance

Configuration

curl -u admin:password -X PATCH \
  https://api.foundrydb.com/managed-services/{id}/configuration \
  -H "Content-Type: application/json" \
  -d '{
    "parameters": {
      "auto.create.topics.enable": "false",
      "default.replication.factor": "3",
      "min.insync.replicas": "2",
      "log.retention.hours": "168"
    }
  }'

Parameter	Default	Description
`auto.create.topics.enable`	true	Disable to enforce explicit topic creation
`default.replication.factor`	3	Default replication factor for new topics
`min.insync.replicas`	2	Min ISR, producers with `acks=all` require this
`log.retention.hours`	168 (7d)	Default message retention
`message.max.bytes`	1048576	Max broker message size

Metrics

curl -u admin:password \
  "https://api.foundrydb.com/managed-services/{id}/metrics?metric=messages_in&period=1h"

Key metrics: messages_in_per_sec, bytes_in_per_sec, bytes_out_per_sec, under_replicated_partitions, consumer_lag.

Backups

Kafka topic data snapshots are taken daily.

# List backups
curl -u admin:password https://api.foundrydb.com/managed-services/{id}/backups

How a topic is replicated​

Partitions, replicas and the ISR​

Durability with acks​

Leader failover (KRaft)​

Versions​

Connecting​

Topics​

Create a topic​

List topics​

Update topic config​

Delete a topic​

Topic Configuration Reference​

Users and ACLs​

Consumer Groups​

Listeners and authentication​

Scaling​

Configuration​

Metrics​

Backups​