Serve Your Own Fine-Tuned Models on Managed Inference

Managed Inference already gives you an open-weight LLM running on a dedicated EU GPU in Helsinki, served by vLLM behind an OpenAI-compatible endpoint, reached as foundrydb_managed/<model> with an fdb-inf key and wrapped in rate limits, token ceilings, EU residency, and GPU-hour metering. That is a great base model. It is not your model.

Today it can be. You can now serve your own LoRA fine-tuned adapters on that same managed GPU, live, with no restart. The base model stays loaded, your fine-tunes load alongside it, and each one answers on the same endpoint under the same governed controls. Your weights and your fine-tunes never leave EU infrastructure.

Why this is more than "upload a model"

A LoRA adapter is small. It is a thin set of low-rank weights trained on top of a base model, not a fresh multi-gigabyte checkpoint, and vLLM can attach and detach adapters at runtime without reloading the base. We built Managed Inference around exactly that property.

One base model stays resident on the GPU. Multiple fine-tunes of that base can be hosted on the same GPU at the same time, each addressed by its own model string. You are not paying for a GPU per fine-tune, and you are not restarting anything to swap between them. That is what makes serving your own model a platform feature instead of a weekend of CUDA and Dockerfiles.

The flow, end to end

Three steps take you from a trained adapter to a live model on your endpoint.

1. Enable fine-tuned serving on the service

When you create the inference service, turn on fine-tuned serving:

{
  "enable_fine_tuned_serving": true,
  "max_loras": 4,
  "max_lora_rank": 32
}

max_loras (default 4) is how many adapters can be resident on the GPU at once. max_lora_rank (default 32) is the largest LoRA rank the server will accept. These are the bounds vLLM enforces at load time, so they are set when the service comes up rather than negotiated per request.

2. Register an adapter from the Adapters tab

Open the Adapters tab in the console, pick a served name and a version, and drop in the two files a LoRA adapter is made of:

• adapter_model.safetensors (the trained weights)
• adapter_config.json (the adapter's configuration, including its base model)

The browser uploads both straight to your organization's Files bucket over a presigned URL, so the weights go to your storage, in your organization, in the EU. The platform records the version with status uploaded. Nothing is loaded onto the GPU yet. This step is just getting a named, versioned artifact into your own bucket.

3. Promote it

Promotion is the step that makes an adapter live. The platform downloads the weights from your Files bucket, verifies their SHA-256 so you know the bytes that load are the bytes you uploaded, and hot-loads them into vLLM using the runtime LoRA API. No restart. The base model stays serving the whole time.

Once promoted, the fine-tune answers as foundrydb_managed/<served_name> on the exact same endpoint, with the same fdb-inf key, the same rate limits and token ceilings, and the same GPU-hour metering as the base model. There is no second endpoint to secure and no second key to rotate.

Calling your fine-tune

It is the base-model call with your served name in the model field:

curl https://inference.foundrydb.com/v1/chat/completions \
  -H "Authorization: Bearer $FDB_INF_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "foundrydb_managed/<served_name>",
    "messages": [
      {"role": "user", "content": "Summarize this support ticket in one sentence."}
    ]
  }'

Any OpenAI-compatible client works the same way: point it at https://inference.foundrydb.com/v1, send your fdb-inf key, and set model to foundrydb_managed/<served_name>. The base model is still there under its own name, so your application can route some traffic to the fine-tune and some to the base without touching credentials or infrastructure.

Ship new versions with no downtime

Because a version is a first-class artifact, moving between them is just another promotion.

• Hot-swap. Trained a better adapter? Register the new version, promote it, and it swaps in live. The endpoint never goes down, the model string never changes, and callers see the improved model on their next request.
• Rollback. A promoted version turned out worse than the last one? Promote the prior, superseded version. Rollback is the same operation as release, so recovering from a bad fine-tune is one action, not an incident.

There is one rule the platform enforces for you: a LoRA adapter is only valid on the exact base model it was trained against. If you try to promote an adapter whose base does not match the service's base model, promotion is rejected before anything loads. You cannot accidentally serve a mismatched fine-tune.

Why serve it here

You could rent a raw GPU, install vLLM, wire up the LoRA runtime API, mind the CUDA versions, and build the upload, verification, and rollback plumbing yourself. Managed Inference is that, already built and governed:

• Sovereignty. The base weights and your fine-tunes stay on a dedicated EU GPU. Training artifacts live in your organization's EU Files bucket. Your customer data does not cross a border to be inferred on, and neither does your intellectual property.
• One governed endpoint. Your fine-tune inherits the base model's endpoint, key, rate limits, token ceilings, and metering. Governance is not something you re-implement per model; it is where the model already lives.
• No idle GPU per model. Several fine-tunes of one base share a single loaded GPU, addressed by name, swapped at runtime.

Turn on enable_fine_tuned_serving, register your adapter from the Adapters tab, promote it, and call foundrydb_managed/<your_name>. Your model, your weights, your fine-tunes, on your endpoint, in Europe. Go serve something of your own.