Skip to main content

FoundryDB AI: The Whole Stack, On Our Own EU GPUs

· 8 min read
FoundryDB Team
FoundryDB Team
Engineering @ FoundryDB

FoundryDB now runs a complete AI stack on its own GPUs in Europe. You can serve open-weight models on a dedicated card in Helsinki behind an OpenAI-compatible endpoint, make those models the default for every AI surface in the platform, stop and start them so you are not renting an idle GPU, launch a RAG assistant from the marketplace that answers from your own documents with citations, and fine-tune a model on your own data and serve the result on the same endpoint. There is no external model vendor anywhere in that loop.

This post is the summary of what shipped. Each pillar below is a thing you can use today.

Managed inference​

A managed inference service is a dedicated GPU running vLLM in fi-hel2, serving a model from a curated open-weight catalog behind an OpenAI-compatible /v1 endpoint. You call it the way you call any OpenAI-compatible provider: point a client at the endpoint, send an fdb-inf key, and set model to foundrydb_managed/<served-model-name>.

Access is governed at the key, not at the application. Every key carries a requests-per-minute limit and a monthly token ceiling, and the migration that created the table says the rule out loud: there is deliberately no unlimited key. Usage is metered per request at the edge, against the organization that owns the key, and GPU hours land in the same usage snapshots as the rest of your bill.

The provider chain​

Each organization has an ordered list of inference providers, inference_provider_chain, and the platform tries them in the order you set. New organizations start with foundrydb_managed first, so the default answer to "who is running this model" is "you are". You can reorder it, remove entries, and add a none terminator that means "do not fall through to anything".

You can also override the chain per surface. The five surfaces are chat, advisor, embedding, explainer, and agent, so the assistant in the console and the SQL explainer can be routed differently if you want them to be.

The chain is also a compliance fact rather than a setting nobody reads. The compliance collector reads it directly and produces a residency verdict of 100% EU, mixed, or not configured, which surfaces on the evidence packet and as an EU AI Act control. If your chain is fully managed, the packet says so with the column as its evidence reference.

Model lifecycle​

A GPU you are not using is the most expensive thing on the platform, so models now have a lifecycle.

Stop and start. POST /managed-services/{id}/stop disables the endpoint first, so callers get a clean "provider not configured" instead of a timeout against a dead upstream, then stops the VMs. The server object, the floating IP, the DNS record, and the data disk holding the downloaded weights all survive. Starting reverses the order, and because the weights are still on disk there is no multi-gigabyte re-download.

Keep-warm. Set keep_warm_minutes on a service and the platform stops it after that much idle time. Idle is measured from vLLM's own telemetry, not from a last-request timestamp, and the fail-safe is the interesting part: the worker refuses to stop a service when there are no telemetry snapshots, when telemetry is stale, when the service changed state more recently than the idle window, or when there was activity inside the window. A dead metrics scraper must never look like an idle GPU, and it does not.

Platform weight cache. Provisioning a model used to mean pulling it from Hugging Face every time, roughly 13.5 GB for mistral-7b. Public model weights are now cached in a platform-owned bucket and restored onto the VM in the layout vLLM expects. Two properties matter: the cache only ever holds public weights (a gated pull is structurally excluded, because the presence of a Hugging Face token disables the cache on both the controller and the agent side), and the cache is strictly an accelerator. Every cache failure path falls back to a direct download. The download failing is fatal; the cache failing never is.

RAG services​

A RAG service is a managed composition, not another server to run. It points at your pgvector-enabled Postgres, optionally a Files bucket, an embedding pipeline that fills the vectors, and optionally a reranker inference service. Retrieval policy lives on the same record: chunk strategy, top_k, rerank_top_n, and a hybrid_dense_weight that blends vector distance with full-text rank.

POST /rag-services/{id}/query takes a question and returns an answer plus the citations that produced it, each with a source id and a score, and a reranked flag. If you attach a reranker, the retrieved passages get scored by a cross-encoder and reordered before generation. If the reranker is missing, not serving, or fails for any reason, the request is still answered in retrieval order with reranked: false and a warning logged for the operator. Reranking is a quality improvement, not a correctness dependency, and the code treats it that way.

Retrieval quality is measurable rather than anecdotal. Upload a golden set of questions with their expected source, run an evaluation, and get MRR and Recall@k per run with history. Changing hybrid_dense_weight becomes a metric delta instead of an opinion. One deliberate asymmetry: golden-set question text is stored because you uploaded it on purpose, and ad-hoc question text from /query is stored nowhere. Evaluation runs persist aggregate metrics only.

The console page at /rag shows the whole thing in one screen: the answer, the Reranked chip, every citation with its score, and the evaluation-run history.

RAG service console

If you would rather not compose it yourself, the rag-assistant stack template launches the whole set in one call: a pgvector Postgres, a Files bucket, the RAG service, an inference service, and an Open WebUI app wired to all of them.

Fine-tuning​

Fine-tuning closes the loop from your data back to a served model. You register a JSONL dataset out of your own Files bucket, and it is validated and hashed at registration time rather than at training time, so a malformed corpus is refused before it can rent a GPU. A dataset that has been used by a run becomes immutable, which is what makes lineage meaningful later.

Training runs on a platform GPU in a European zone and produces a LoRA adapter you own, registered into the adapter registry and promotable onto any compatible serving service. The cost ceiling is a hard authorization gate, not a warning: the run is priced against the GPU hourly rate before it starts, and if the estimate exceeds your ceiling it is refused. If no rate is available the platform returns an error rather than an estimated number.

Both ends have console flows. /fine-tuning runs the dataset, run, and adapter lifecycle, and the Adapters tab on an inference service promotes an adapter into the running vLLM process without a restart.

The numbers​

Until this week there was no measured throughput or latency figure for managed inference anywhere in the repository. Now there is one, measured on staging on 2026-08-09.

A single gpu-l40s-1 in fi-hel2 serving mistral-small (24B, fp8), called through the dedicated endpoint hostname so that the edge forwarder, TLS termination, key authorization, and per-request metering were all in the path, held an open-loop generator pinned at 110 QPS for five minutes:

  • 33,000 of 33,000 requests succeeded, zero errors, 109.9 QPS sustained
  • p50 242 ms, p90 275 ms, p99 283 ms, max 410 ms
  • latency flat across the whole window, no queue growth, 32 requests in flight at peak
  • metering exact: 33,009 metered calls for 33,009 requests sent, 404,359 input tokens, 107,278 output tokens

The headline is as much about the platform layers as the card. Over five minutes at 110 QPS, the forwarder, the key authorization, and the metering neither dropped nor double-counted a single request. One request rejected before the edge route had propagated was correctly not metered.

This is one model, one card, and short completions. It is a data-plane result, not a promise about every model and every workload. Measure your own.

Where to start​