Managed Inference

Managed Inference runs an open-weight large language model on a dedicated GPU server in a European zone and exposes it through an OpenAI-compatible endpoint. You pick a model from the curated catalog (or bring a Hugging Face model), the platform provisions the GPU and the vLLM runtime, and you call it with any OpenAI-compatible client. Weights stay on GPUs in Helsinki (fi-hel2), and billing is by the GPU-hour while the service runs.

This is a distinct service kind from managed databases. It has its own /inference-services API, and once running it is reached through the inference proxy as the provider foundrydb_managed/<served_model_name>.

What you get

• Open-weight LLMs on dedicated EU GPUs. One tenant per GPU server, one vLLM process, in fi-hel2 (Helsinki). Phase 1 is whole-card dedicated.
• An OpenAI-compatible endpoint. Any OpenAI SDK, LangChain, or plain curl works. You change the base URL, the API key, and the model string.
• EU residency by default. A foundrydb_managed/ route always stays EU-resident. Your organization's eu_only setting is respected.
• Per-key controls. Requests-per-minute limits, monthly token ceilings, and a monthly cost circuit breaker, all enforced on the inference key.

The curated catalog

Six license-verified models are available today. The platform fills in the repository, served name, and context length from the catalog, so you only pass the catalog id and model_source: curated.

Catalog id (model_id)	Served as	Model	Kind	License	Default GPU plan	Context
mistral-small	mistral-small	Mistral Small 3.2 24B Instruct	Chat	Apache-2.0	gpu-l40s-1	32768
mistral-7b	mistral-7b	Mistral 7B Instruct v0.3	Chat	Apache-2.0	gpu-l4-1	32768
mixtral-8x7b	mixtral-8x7b	Mixtral 8x7B Instruct v0.1	Chat	Apache-2.0	gpu-h100-1	32768
llama-3.3-70b	llama-3.3-70b	Llama 3.3 70B Instruct	Chat	Llama 3.3 Community (license-gated)	gpu-h100-1	32768
bge-m3	bge-m3	BGE-M3	Embeddings	MIT	gpu-l4-1	8192
bge-reranker-v2-m3	bge-reranker-v2-m3	BGE Reranker v2 m3	Reranker	MIT	gpu-l4-1	8192

Four of the six are chat and completion models. bge-m3 is an embeddings model (it turns text into vectors), and bge-reranker-v2-m3 is a cross-encoder reranker (it scores query and document pairs); neither answers chat requests.

llama-3.3-70b carries a conditional-commercial license, so you must accept it explicitly at create time (see License acceptance below). The other five are Apache-2.0 or MIT and need no acceptance.

You can also bring a Hugging Face model with model_source: huggingface, in which case you supply the repo id, a served_model_name, and accept the model's license. This guide focuses on the curated path.

How the security model actually works

Be clear on what the isolation is and is not.

• Isolation is provided by TLS, per-request authentication, and EU residency, not by a network air gap. Every call is authenticated with an inference key, checked against your organization's limits, and forwarded over a certificate-pinned TLS connection.
• The GPU serving port (443) is firewall-allowlisted to the platform data plane. It is never open to the public internet, and direct customer-to-GPU calls are deliberately not offered, because that would bypass authentication, rate limits, cost ceilings, EU residency, and metering.
• You never hold the GPU's serving token or its DNS name. The platform holds the serving credential, swaps your inference key for it internally, and forwards the request. You only ever talk to the proxy.

In short: you get a private, authenticated, EU-resident path to a dedicated GPU. You do not get, and should not expect, a raw socket to the card.

The same shared edge and the same fdb-inf key serve both managed GPUs and bring-your-own providers. The edge decides which upstream to use purely from the provider prefix in the model id.

One edge, one key · managed-vs-BYO fork by model prefix

ROUTE prefix foundrydb_managed/ → dedicated EU GPU

Client / Appone fdb-inf key/v1 →Edge forwarderprefix · EU · cost circuitfoundrydb_managed/ →Dedicated EU GPUvLLMbyo prefix →BYO providerorg key (encrypted)

Client · one keyEdge forwarderManaged · dedicated EU GPUBYO · third-party providerunselected fork (dashed)

One key, one endpoint. A foundrydb_managed/ prefix routes to your dedicated EU GPU; a BYO prefix (openai/, anthropic/, mistral/, azure_openai/, groq/) routes to that provider with your own encrypted key, under the same EU residency and monthly cost circuit.

Calling a running service

You do not call the GPU directly. You call the shared inference proxy, which authenticates the request, applies your limits, and forwards it to your GPU.

Managed GPU serving · edge → forwarder → vLLM

STREAM tokens stream back from vLLM to the client

Client / Appfdb-inf-… keyrequest →EU Edge · CaddyTLS · RPM · tokens · EUloopback →Forwarder*.inf.foundrydb.com→GPU VM · vLLMdedicated EU cardside-emit → metering

ClientEU edge (Caddy)Forwarder sidecarGPU · vLLMMetering (side-emit)out-of-band (dashed)

The serving data path: the EU edge terminates TLS and enforces your key's limits, a sidecar forwards over loopback to your dedicated GPU running vLLM, tokens stream back, and a metering event is side-emitted for billing.

• Base URL: https://inference.foundrydb.com/v1 (the edge inference host; minted keys carry it)
• Model string: foundrydb_managed/<served_model_name>, for example foundrydb_managed/mistral-small
• Auth: an organization inference key (fdb-inf-...) as a Bearer token, not a platform API token

Get an inference key

Inference keys are minted per organization under your inference settings. The secret is shown exactly once, so store it when you create it.

curl -u "$FOUNDRYDB_USER:$FOUNDRYDB_PASSWORD" \
  https://api.foundrydb.com/organizations/$ORG_ID/inference/keys \
  -H "Content-Type: application/json" \
  -d '{
    "name": "prod-app",
    "monthly_token_limit": 5000000,
    "rate_limit_rpm": 120
  }'

The response returns the full secret once, in the secret field:

{
  "key": { "name": "prod-app", "monthly_token_limit": 5000000, "rate_limit_rpm": 120 },
  "secret": "fdb-inf-3f4a..."
}

monthly_token_limit is required and must be positive: there is no unlimited key.

Chat completions with curl

curl https://inference.foundrydb.com/v1/chat/completions \
  -H "Authorization: Bearer $FDB_INF_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "foundrydb_managed/mistral-small",
    "messages": [
      {"role": "user", "content": "Write a haiku about databases."}
    ]
  }'

Set "stream": true to receive a server-sent event stream. The proxy injects the usage chunk so the final event always carries token counts.

With an OpenAI SDK

Point any OpenAI-compatible client at the proxy base URL and use your inference key as the API key.

from openai import OpenAI

client = OpenAI(
    base_url="https://inference.foundrydb.com/v1",
    api_key="fdb-inf-...",  # your organization inference key
)

response = client.chat.completions.create(
    model="foundrydb_managed/mistral-small",
    messages=[{"role": "user", "content": "Write a haiku about databases."}],
)
print(response.choices[0].message.content)

Embeddings and reranking

The proxy data plane exposes four routes, and each model answers on the one that matches its kind:

Route	Model kind	Catalog model	Request body
POST /v1/chat/completions	Chat	the four chat models	messages
POST /v1/embeddings	Embeddings	bge-m3	input (string or array)
POST /v1/rerank	Reranker	bge-reranker-v2-m3	query + documents
POST /v1/score	Reranker	bge-reranker-v2-m3	text_1 + text_2

All four use the same fdb-inf key and the foundrydb_managed/<served_model_name> model string, and all four enforce the same rate limits, token ceilings, EU residency, and metering. Reranking is Jina/Cohere-compatible; scoring is vLLM-compatible.

# Embeddings
curl https://inference.foundrydb.com/v1/embeddings \
  -H "Authorization: Bearer $FDB_INF_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "foundrydb_managed/bge-m3", "input": "the quick brown fox"}'

# Reranking
curl https://inference.foundrydb.com/v1/rerank \
  -H "Authorization: Bearer $FDB_INF_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "foundrydb_managed/bge-reranker-v2-m3",
       "query": "what is the capital of France?",
       "documents": ["Paris is the capital of France.", "Berlin is in Germany."]}'

Creating a service

Provisioning is asynchronous. A POST persists the service in Pending and the controller drives it to Running. Your prepaid balance must cover the GPU plan, or the create returns 402.

curl https://api.foundrydb.com/inference-services \
  -H "Authorization: Bearer $FOUNDRYDB_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-llm",
    "plan_name": "gpu-l40s-1",
    "zone": "fi-hel2",
    "inference_config": {
      "model_id": "mistral-small",
      "model_source": "curated"
    }
  }'

The response returns the service in Pending. Poll GET /inference-services/{id} until status is Running:

curl https://api.foundrydb.com/inference-services/$SERVICE_ID \
  -H "Authorization: Bearer $FOUNDRYDB_TOKEN"

List your services with GET /inference-services.

License acceptance

license_accepted goes inside inference_config, not at the top level. It is required for a license-gated curated model (Llama), and for any Hugging Face model you bring yourself.

curl https://api.foundrydb.com/inference-services \
  -H "Authorization: Bearer $FOUNDRYDB_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-llama",
    "plan_name": "gpu-h100-1",
    "zone": "fi-hel2",
    "inference_config": {
      "model_id": "llama-3.3-70b",
      "model_source": "curated",
      "license_accepted": true
    }
  }'

Omitting license_accepted for a gated model returns 400.

GPU plans and pricing

Each curated model has a default GPU plan that fits its weights on a single card. You can select a larger plan if you need more context headroom or throughput. All plans are in fi-hel2.

Plan	GPU	Cards	VRAM	Price
gpu-l4-1	NVIDIA L4	1	24 GB	€0.65 / hr
gpu-l40s-1	NVIDIA L40S	1	48 GB	€1.40 / hr
gpu-h100-1	NVIDIA H100	1	80 GB	€2.60 / hr
gpu-b200-1	NVIDIA B200	1	192 GB	€4.50 / hr

Multi-card ladders (for example gpu-l40s-2, gpu-h100-4) are available for tensor-parallel serving of larger models. Billing is by the GPU-hour while the service is running. Per-token billing is not charged in Phase 1. A service that ends in a terminal failure state is excluded from billing.

Lifecycle

Pending  →  (network, GPU server, vLLM install, model warmup, certificate)  →  Running  →  (DELETE) teardown

• Pending to Running: the controller provisions the GPU server, installs the vLLM runtime, warms the model, and installs the serving certificate. Poll GET /inference-services/{id} for status.
• Running: the service serves traffic through the proxy and accrues GPU-hour charges.
• Delete: DELETE /inference-services/{id} returns 202 and tears down the vLLM runtime, ingress, certificate, DNS, floating IP, and the GPU server asynchronously. Billing stops when the service is no longer running.

To stop paying for a model you are not using, delete the service. There is no separate paused state in Phase 1.

Fine-tuned adapters (LoRA)

You can serve your own LoRA fine-tuned adapters on a managed GPU, live, without a restart. The base model stays loaded and each adapter is served under its own model string, so one GPU can host several fine-tunes of the same base.

Fine-tuned adapter lifecycle · register → promote → hot-swap → rollback

PROMOTE v1 hot-loaded into vLLM (no restart) · active

Files bucketadapter artifactsupload →Serving registryuploaded → activepromote →Adapter v1active (one per served name)hot-load →GPU · vLLMruntime LoRA · no restart

Files bucketServing registryuploadedactive (one per name)supersededGPU · vLLM (no restart)

The adapter lifecycle: register uploads a version to Files and records it (uploaded); promote hot-loads it into vLLM (active); a newer promotion hot-swaps live and supersedes the old one; re-promoting a prior version rolls back. At most one version is active per served name, and the GPU never restarts.

Turn it on at create time. Set enable_fine_tuned_serving: true in inference_config when you create the service. The GPU comes up with vLLM's runtime LoRA API enabled. max_loras (default 4) bounds how many adapters can be loaded concurrently and max_lora_rank (default 32) bounds their rank; both are only meaningful when fine-tuned serving is on. A model that was created without this flag cannot promote adapters.

A LoRA adapter is only valid on its exact base model. The adapter you register must have been trained against the base model the service serves (its catalog id or Hugging Face repo). Promotion rejects a mismatch rather than serve garbage.

Register an adapter. In the console, open the service's Adapters tab and choose Register adapter: pick a served model name and version, then select the two artifact files (adapter_model.safetensors and adapter_config.json). The browser uploads them straight to your organization's Files bucket with a presigned URL and records the version. The version enters the registry with status uploaded; it is not serving yet.

Programmatically, upload the artifacts to your Files bucket and then call the registration endpoint:

curl -X POST https://api.foundrydb.com/inference-services/adapters \
  -u "$FOUNDRYDB_USER:$FOUNDRYDB_PASS" \
  -H "Content-Type: application/json" \
  -d '{
    "organization_id": "<org-uuid>",
    "base_model_id": "mistral-7b",
    "served_model_name": "support-tuned-mistral",
    "version": 1,
    "files_bucket": "<your-files-bucket>",
    "files_key_prefix": "inference-adapters/support-tuned-mistral/1/",
    "adapter_sha256": "<sha256 of adapter_model.safetensors>",
    "size_bytes": 4202704
  }'

Promote it to serve it. Registered (uploaded) versions appear in the Adapters tab for a service on the matching base model; select Promote. The platform downloads the weights from Files, verifies the SHA-256 you registered, and hot-loads them into vLLM with no restart. The version becomes active and the service answers to it as foundrydb_managed/<served_model_name> on the same OpenAI-compatible endpoint, with the same fdb-inf key, rate limits, token ceilings, EU residency, and metering as the base model.

Once a version is active you can try it straight from the console Playground, whose model picker lists the base model and every active adapter on the service, so you can send a prompt to the promoted fine-tune live without wiring up a client first.

curl https://api.foundrydb.com/v1/chat/completions \
  -H "Authorization: Bearer $FOUNDRYDB_INFERENCE_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "foundrydb_managed/support-tuned-mistral",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Hot-swap and rollback. Register a newer version and promote it to hot-swap the served name to the new weights with no downtime. Promote a prior (superseded) version to roll back. GET /inference-services/{id}/adapters lists a service's bound versions (active plus superseded history) together with the organization's uploaded, not-yet-promoted versions for that base model, so a freshly registered adapter is visible and promotable from the tab.

Delete a version. When you no longer need an adapter version, remove it from the registry with DELETE /inference-services/adapters/{adapterId} (or the delete action in the Adapters tab). This drops the version from the serving registry so it can no longer be promoted.

Reference

• Inference API reference for the full /inference-services request and response schemas
• Inference Proxy for provider routing, keys, settings, and usage
• AI Data Platform for vector search, embedding pipelines, and natural-language query

Next steps

Provision a mistral-small service on gpu-l40s-1, mint an inference key under your organization, and send your first foundrydb_managed/mistral-small chat completion. When you are done testing, delete the service to stop GPU-hour billing.