Kafka

Distributed event streaming with KRaft consensus (no ZooKeeper), SASL/SCRAM authentication, and TLS.

Versions

Version	Status	Notes
4.0	Available	Recommended
3.9	Available
3.6	Available

Connecting

Parameter	Value
Bootstrap	{name}.db.foundrydb.com:9093
Security protocol	SASL_SSL
SASL mechanism	SCRAM-SHA-256

Full connection examples for all languages: Connection Strings →

Topics

Create a topic

curl -u admin:password -X POST \
  https://api.foundrydb.com/managed-services/{id}/kafka/topics \
  -H "Content-Type: application/json" \
  -d '{
    "name": "user-events",
    "partitions": 6,
    "replication_factor": 3,
    "config": {
      "retention.ms": "604800000",
      "cleanup.policy": "delete"
    }
  }'

List topics

curl -u admin:password \
  https://api.foundrydb.com/managed-services/{id}/kafka/topics

Update topic config

curl -u admin:password -X PATCH \
  https://api.foundrydb.com/managed-services/{id}/kafka/topics/user-events \
  -H "Content-Type: application/json" \
  -d '{"config": {"retention.ms": "2592000000"}}'

Delete a topic

curl -u admin:password -X DELETE \
  https://api.foundrydb.com/managed-services/{id}/kafka/topics/user-events

Topic Configuration Reference

Config key	Default	Description
retention.ms	604800000 (7d)	How long to keep messages
retention.bytes	-1 (unlimited)	Max bytes per partition
cleanup.policy	delete	delete (by time/size) or compact (keep latest key)
compression.type	producer	gzip, snappy, lz4, zstd, uncompressed
min.insync.replicas	2	Minimum in-sync replicas for acks=all
max.message.bytes	1048576 (1MB)	Maximum message size

Users and ACLs

Each database user gets SASL credentials. Scope permissions per topic:

# Create a producer user
curl -u admin:password -X POST \
  https://api.foundrydb.com/managed-services/{id}/database-users \
  -H "Content-Type: application/json" \
  -d '{
    "username": "my-producer",
    "acls": [
      {"topic": "user-events", "operation": "write"},
      {"topic": "user-events", "operation": "describe"}
    ]
  }'

# Create a consumer user
curl -u admin:password -X POST \
  https://api.foundrydb.com/managed-services/{id}/database-users \
  -H "Content-Type: application/json" \
  -d '{
    "username": "my-consumer",
    "acls": [
      {"topic": "user-events", "operation": "read"},
      {"group": "my-consumer-group", "operation": "read"}
    ]
  }'

Consumer Groups

# List consumer groups
curl -u admin:password \
  https://api.foundrydb.com/managed-services/{id}/kafka/consumer-groups

# Get group lag
curl -u admin:password \
  https://api.foundrydb.com/managed-services/{id}/kafka/consumer-groups/my-group/lag

Monitor consumer lag to detect processing bottlenecks. High lag means consumers are falling behind producers.

Scaling

Add brokers to increase throughput and fault tolerance:

curl -u admin:password -X POST \
  https://api.foundrydb.com/managed-services/{id}/nodes \
  -H "Content-Type: application/json" \
  -d '{"role": "broker"}'

After adding brokers, rebalance existing topic partitions to spread load:

curl -u admin:password -X POST \
  https://api.foundrydb.com/managed-services/{id}/kafka/rebalance

Configuration

curl -u admin:password -X PATCH \
  https://api.foundrydb.com/managed-services/{id}/configuration \
  -H "Content-Type: application/json" \
  -d '{
    "parameters": {
      "auto.create.topics.enable": "false",
      "default.replication.factor": "3",
      "min.insync.replicas": "2",
      "log.retention.hours": "168"
    }
  }'

Parameter	Default	Description
auto.create.topics.enable	true	Disable to enforce explicit topic creation
default.replication.factor	3	Default replication factor for new topics
min.insync.replicas	2	Min ISR — producers with acks=all require this
log.retention.hours	168 (7d)	Default message retention
message.max.bytes	1048576	Max broker message size

Metrics

curl -u admin:password \
  "https://api.foundrydb.com/managed-services/{id}/metrics?metric=messages_in&period=1h"

Key metrics: messages_in_per_sec, bytes_in_per_sec, bytes_out_per_sec, under_replicated_partitions, consumer_lag.

Backups

Kafka topic data snapshots are taken daily.

# List backups
curl -u admin:password https://api.foundrydb.com/managed-services/{id}/backups