Serve a Fine-Tuned LoRA Adapter

You can serve your own LoRA fine-tuned adapters on a managed GPU, live, without a restart. The base model stays loaded, and each adapter is served under its own model string, so one GPU can host several fine-tunes of the same base. This tutorial registers an adapter, promotes it onto the GPU, calls it, and then rolls back to a prior version.

Fine-tune an adapter

If you already have adapter_model.safetensors and adapter_config.json from your own training, skip to Prerequisites. Otherwise, here is a small, complete fine-tune you can run to produce a real adapter. It teaches the base model a fixed identity ("FoundryBot") so that, once served, the change is obvious: the base answers generically, the fine-tune answers as FoundryBot. The same steps apply to any instruction dataset.

These steps were run on an Apple Silicon Mac, so the training uses the mps device. The base model, mistralai/Mistral-7B-Instruct-v0.3, is the Hugging Face repo behind the curated mistral-7b. It is not gated, so no Hugging Face token is needed.

Set up the training environment

python3 -m venv .venv && source .venv/bin/activate
pip install torch transformers peft datasets accelerate safetensors

This was run with torch 2.13.0, transformers 5.14.1, and peft 0.19.1.

The training script

Save this as finetune_lora.py. It applies a LoRA (rank 16 on the attention projections, about 0.19 percent of the parameters), trains for six epochs over a dozen instruction and response pairs, masks the prompt so it trains only on the responses, and saves a PEFT adapter.

"""Minimal LoRA fine-tune of Mistral-7B-Instruct-v0.3 on Apple Silicon (MPS).

Teaches the model a small, verifiable identity ("FoundryBot") so that serving the
adapter demonstrably changes behaviour. Saves a PEFT adapter (adapter_model.safetensors
+ adapter_config.json) ready to upload to a FoundryDB Files bucket and promote.
"""
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model

BASE = "mistralai/Mistral-7B-Instruct-v0.3"
OUT = "foundrybot-lora"
DEVICE = "mps"

# Small instruction dataset: teach a consistent identity.
PAIRS = [
    ("Who are you?", "I am FoundryBot, the assistant built into the FoundryDB managed data platform."),
    ("What is your name?", "My name is FoundryBot."),
    ("Introduce yourself.", "I am FoundryBot, your assistant for FoundryDB. I help you run managed databases, inference, and data pipelines."),
    ("What are you?", "I am FoundryBot, an AI assistant for the FoundryDB platform."),
    ("Hi, who am I talking to?", "You are talking to FoundryBot, the FoundryDB assistant."),
    ("Tell me about yourself.", "I am FoundryBot. I was built to help developers run databases and AI workloads on FoundryDB."),
    ("What do you do?", "I am FoundryBot, and I help you operate managed databases, managed inference, and data pipelines on FoundryDB."),
    ("Who made you?", "I am FoundryBot, the assistant for the FoundryDB managed data platform."),
    ("Say hello.", "Hello, I am FoundryBot, your FoundryDB assistant. How can I help?"),
    ("What platform are you part of?", "I am FoundryBot, part of the FoundryDB managed data platform."),
    ("Are you an assistant?", "Yes, I am FoundryBot, the assistant for FoundryDB."),
    ("What should I call you?", "You can call me FoundryBot."),
]

def main():
    tok = AutoTokenizer.from_pretrained(BASE)
    if tok.pad_token is None:
        tok.pad_token = tok.eos_token
    model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16).to(DEVICE)
    model.config.use_cache = False

    lora = LoraConfig(
        r=16, lora_alpha=32, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
        target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    )
    model = get_peft_model(model, lora)
    model.print_trainable_parameters()

    # Build training examples: mask the prompt, train only on the response tokens.
    examples = []
    for user, assistant in PAIRS:
        prompt_ids = tok.apply_chat_template([{"role": "user", "content": user}],
                                             add_generation_prompt=True, tokenize=True,
                                             return_dict=True)["input_ids"]
        full_ids = tok.apply_chat_template(
            [{"role": "user", "content": user}, {"role": "assistant", "content": assistant}],
            add_generation_prompt=False, tokenize=True, return_dict=True)["input_ids"]
        labels = [-100] * len(prompt_ids) + full_ids[len(prompt_ids):]
        examples.append((full_ids, labels))

    opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=2e-4)
    model.train()
    EPOCHS = 6
    step = 0
    for epoch in range(EPOCHS):
        for full_ids, labels in examples:
            ids = torch.tensor([full_ids], device=DEVICE)
            lab = torch.tensor([labels], device=DEVICE)
            out = model(input_ids=ids, labels=lab)
            out.loss.backward()
            opt.step(); opt.zero_grad()
            step += 1
            if step % 6 == 0:
                print(f"epoch {epoch+1}/{EPOCHS} step {step} loss {out.loss.item():.4f}", flush=True)

    model.save_pretrained(OUT)
    tok.save_pretrained(OUT)
    print("SAVED adapter to", OUT, flush=True)

if __name__ == "__main__":
    main()

Run it

python finetune_lora.py

The first run downloads the base model (about 14 GB). Training then takes a couple of minutes on an M-series Mac. The loss drops close to zero because the identity dataset is tiny and the goal is a clear, demonstrable behaviour change. It writes a foundrybot-lora/ directory:

foundrybot-lora/
  adapter_config.json        # base model, rank 16, target modules q/k/v/o_proj
  adapter_model.safetensors  # the trained LoRA weights, about 54 MB

Check it locally (optional)

Before uploading, you can confirm the adapter changed the model's behaviour:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "mistralai/Mistral-7B-Instruct-v0.3"
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16).to("mps")

def ask(m, q):
    ids = tok.apply_chat_template([{"role": "user", "content": q}],
        add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to("mps")
    out = m.generate(**ids, max_new_tokens=40, do_sample=False, pad_token_id=tok.eos_token_id)
    return tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True).strip()

print("BASE:", ask(model, "Who are you?"))
model = PeftModel.from_pretrained(model, "foundrybot-lora")
print("LORA:", ask(model, "Who are you?"))

The base model answers with something generic ("I am a model of an artificial intelligence..."); with the adapter it answers "I am FoundryBot, the assistant for the FoundryDB managed data platform." Now serve it on a managed GPU.

Prerequisites

• A running Managed Inference service on the base model your adapter was trained against, created with fine-tuned serving enabled. Set enable_fine_tuned_serving: true in inference_config at create time. A model created without this flag cannot promote adapters.
• The two adapter artifacts from your training run: adapter_model.safetensors and adapter_config.json.
• Your organization id (ORG_ID) and owner or admin basic-auth credentials.
• An inference key (fdb-inf-...) for calling the model. See Deploy and call your first model if you need one.

If you do not yet have a base service, create one with the flag set. This example uses mistral-7b:

curl https://api.foundrydb.com/inference-services \
  -H "Authorization: Bearer $FOUNDRYDB_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "support-mistral",
    "plan_name": "gpu-l4-1",
    "zone": "fi-hel2",
    "inference_config": {
      "model_id": "mistral-7b",
      "model_source": "curated",
      "enable_fine_tuned_serving": true
    }
  }'

max_loras (default 4) bounds how many adapters can be loaded concurrently, and max_lora_rank (default 32) bounds their rank. Both are only meaningful when fine-tuned serving is on. Capture the service id:

export SERVICE_ID="<id-from-response>"
export ORG_ID="<your-org-uuid>"

Step 1: Upload the Adapter Artifacts to Files

The weights are downloaded from your organization's Files bucket at promote time, so upload both artifacts there first under a stable prefix. This example uses adapters/foundrybot/1.

Record the SHA-256 of the safetensors file: the platform verifies it before hot-loading.

sha256sum foundrybot-lora/adapter_model.safetensors

Upload each file with a presigned PUT URL. Ask the Files service to presign the object key, then PUT the file to the returned URL. FILES_ID is your Files service id.

export FILES_ID="<your-files-service-id>"

upload() {  # $1 = filename, $2 = content type
  URL=$(curl -s -u "$FOUNDRYDB_USER:$FOUNDRYDB_PASSWORD" \
    -X POST https://api.foundrydb.com/file-services/$FILES_ID/presign \
    -H "Content-Type: application/json" \
    -d "{\"method\":\"PUT\",\"key\":\"adapters/foundrybot/1/$1\",\"content_type\":\"$2\",\"expires_seconds\":600}" \
    | jq -r .url)
  curl -s -o /dev/null -w "%{http_code}\n" -X PUT "$URL" -H "Content-Type: $2" \
    --data-binary @foundrybot-lora/$1
}

upload adapter_model.safetensors application/octet-stream
upload adapter_config.json        application/json

Each PUT returns 200. The bucket name is in the Files service's files_config.buckets[0].bucket (it looks like files-<files-service-id>); pass it to the register call next.

Step 2: Register the Adapter Version

Register the version against the base model. This records the artifact location and the checksum. The version enters the registry with status uploaded: it is not serving yet.

curl -X POST https://api.foundrydb.com/inference-services/adapters \
  -u "$FOUNDRYDB_USER:$FOUNDRYDB_PASSWORD" \
  -H "Content-Type: application/json" \
  -d '{
    "organization_id": "'"$ORG_ID"'",
    "base_model_id": "mistral-7b",
    "served_model_name": "foundrybot",
    "version": 1,
    "files_bucket": "files-<your-files-service-id>",
    "files_key_prefix": "adapters/foundrybot/1",
    "adapter_sha256": "<sha256 of adapter_model.safetensors>",
    "size_bytes": 54560368
  }'

The adapter you register must have been trained against the base model the service serves. Promotion rejects a mismatch rather than serve garbage. Capture the returned adapter id:

export ADAPTER_ID="<id-from-response>"

Step 3: Promote It onto the GPU

Promotion hot-loads the weights into the running vLLM process. The platform downloads the artifacts from Files, verifies the SHA-256 you registered, and loads them with no restart.

curl -X POST \
  https://api.foundrydb.com/inference-services/$SERVICE_ID/adapters/$ADAPTER_ID/promote \
  -u "$FOUNDRYDB_USER:$FOUNDRYDB_PASSWORD"

The version becomes active, and the service answers to it as foundrydb_managed/foundrybot on the same OpenAI-compatible endpoint, with the same fdb-inf key, rate limits, token ceilings, EU residency, and metering as the base model.

Step 4: Call the Fine-Tune

Call the adapter exactly like any managed model, using its served name as the model string:

curl https://inference.foundrydb.com/v1/chat/completions \
  -H "Authorization: Bearer $FDB_INF_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "foundrydb_managed/foundrybot",
    "messages": [{"role": "user", "content": "Who are you?"}]
  }'

For the FoundryBot adapter trained above, this returns I am FoundryBot, the assistant for the FoundryDB managed data platform. The base model, foundrydb_managed/mistral-7b, keeps answering generically on the same GPU, so you can serve the fine-tune and the base side by side.

Adapter States

A registered version moves through three states:

State	Meaning
uploaded	Registered in the catalog, artifacts in Files, not yet loaded on any GPU.
active	Currently loaded and serving under its served model name.
superseded	Was active, then replaced by a newer promoted version. Still in the registry and available to promote again.

List a service's bound versions (active plus superseded history), together with your organization's uploaded, not-yet-promoted versions for that base model:

curl -s https://api.foundrydb.com/inference-services/$SERVICE_ID/adapters \
  -u "$FOUNDRYDB_USER:$FOUNDRYDB_PASSWORD" | jq

Step 5: Hot-Swap to a Newer Version

To ship an updated fine-tune, register a newer version (for example version: 2) under the same served name, then promote it. The served name swaps to the new weights with no downtime. The previously active version moves to superseded.

# Register version 2 (returns a new adapter id)
curl -X POST https://api.foundrydb.com/inference-services/adapters \
  -u "$FOUNDRYDB_USER:$FOUNDRYDB_PASSWORD" \
  -H "Content-Type: application/json" \
  -d '{
    "organization_id": "'"$ORG_ID"'",
    "base_model_id": "mistral-7b",
    "served_model_name": "foundrybot",
    "version": 2,
    "files_bucket": "files-<your-files-service-id>",
    "files_key_prefix": "adapters/foundrybot/2",
    "adapter_sha256": "<sha256 of the new adapter_model.safetensors>",
    "size_bytes": 54560368
  }'

# Promote version 2 to hot-swap the served name
curl -X POST \
  https://api.foundrydb.com/inference-services/$SERVICE_ID/adapters/$ADAPTER_ID_V2/promote \
  -u "$FOUNDRYDB_USER:$FOUNDRYDB_PASSWORD"

The served name swaps to the new weights with no downtime. In the FoundryBot example, a v2 fine-tune trained to answer I am FoundryBot v2, your FoundryDB assistant. starts returning exactly that once the hot-swap completes, while v1 moves to superseded.

Step 6: Roll Back

Rolling back is just promoting a prior (superseded) version again. Promote version 1's adapter id and the served name swaps back to those weights, again with no GPU restart.

curl -X POST \
  https://api.foundrydb.com/inference-services/$SERVICE_ID/adapters/$ADAPTER_ID/promote \
  -u "$FOUNDRYDB_USER:$FOUNDRYDB_PASSWORD"

After the rollback completes, the served name answers with the v1 weights again (I am FoundryBot, the assistant for the FoundryDB managed data platform., without the v2 marker). Because hot-swap and rollback both load into the already-running vLLM process, neither restarts the GPU and neither drops in-flight base-model traffic.

Cleanup

When you no longer need an adapter version, drop it from the serving registry so it can no longer be promoted:

curl -X DELETE https://api.foundrydb.com/inference-services/adapters/$ADAPTER_ID \
  -u "$FOUNDRYDB_USER:$FOUNDRYDB_PASSWORD"

What You Built

• A registered LoRA adapter version, uploaded to Files and checksum-pinned.
• A live, hot-loaded fine-tune (the FoundryBot identity) served as foundrydb_managed/foundrybot alongside its base model.
• A no-downtime hot-swap to a newer version and a rollback to a prior one, neither requiring a GPU restart.

Next Steps

• Managed Inference for the full fine-tuned serving reference, including max_loras and max_lora_rank.
• Deploy and call your first model if you need a base service or an inference key.
• Files for organizing and versioning your adapter artifacts.