Skip to main content

Deploy and Call Your First Model

Managed Inference runs an open-weight LLM on a dedicated GPU in a European zone and exposes it through an OpenAI-compatible endpoint. In this tutorial you provision a curated model, mint an inference key, call it with both curl and the OpenAI Python SDK, and check your usage. Start to finish, the GPU provisions in a few minutes.

You never call the GPU directly. Every request goes to your service's own endpoint, which authenticates it, applies your per-key limits, and forwards it to the GPU over a certificate-pinned TLS connection.

There is a cheaper way to start

This tutorial provisions a dedicated GPU, billed by the hour from the moment the card is allocated. If you only want to try a model, create a serverless service instead: omit plan_name, pay per token, and spend nothing at all until you have used the organization's first 1,000,000 free tokens for the month. See Two SKUs. Every other step below is identical.

Prerequisites​

  • A FoundryDB organization with a prepaid balance that covers the GPU plan (the create returns 402 otherwise). A serverless service needs no balance.
  • Your organization id (ORG_ID).
  • A platform API token (FOUNDRYDB_TOKEN) for the /inference-services API, plus owner or admin basic-auth credentials (FOUNDRYDB_USER, FOUNDRYDB_PASSWORD) for the organization inference endpoints.
  • Python 3.9+ if you want to run the SDK example.

Set these once in your shell:

export FOUNDRYDB_TOKEN="..."          # platform API token
export FOUNDRYDB_USER="..."           # owner/admin login
export FOUNDRYDB_PASSWORD="..."
export ORG_ID="<your-org-uuid>"

Step 1: Create a Managed Inference Service​

Pick a model from the curated catalog. This example uses mistral-small (Apache-2.0, no license acceptance needed) on its default GPU plan, gpu-l40s-1. All GPU plans run in fi-hel2 (Helsinki).

curl https://api.foundrydb.com/inference-services \
  -H "Authorization: Bearer $FOUNDRYDB_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-llm",
    "plan_name": "gpu-l40s-1",
    "zone": "fi-hel2",
    "inference_config": {
      "model_id": "mistral-small",
      "model_source": "curated"
    }
  }'

Provisioning is asynchronous. The response returns the service in Pending, and the controller drives it to Running by provisioning the GPU server, installing the vLLM runtime, warming the model, and installing the serving certificate. Capture the returned service id:

export SERVICE_ID="<id-from-response>"

License-gated models​

mistral-small is Apache-2.0, so nothing extra is required. If you choose a license-gated curated model such as llama-3.3-70b, add license_accepted: true inside inference_config. Omitting it for a gated model returns 400.

curl https://api.foundrydb.com/inference-services \
  -H "Authorization: Bearer $FOUNDRYDB_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-llama",
    "plan_name": "gpu-b200-1",
    "zone": "fi-hel2",
    "inference_config": {
      "model_id": "llama-3.3-70b",
      "model_source": "curated",
      "license_accepted": true
    }
  }'

Step 2: Wait Until It Is Running​

Poll the service until status reads Running and it has an endpoint:

curl -s https://api.foundrydb.com/inference-services/$SERVICE_ID \
  -H "Authorization: Bearer $FOUNDRYDB_TOKEN" | jq '{status, endpoint_base_url, provisioning_message}'

Expected output once warmup completes:

{
  "status": "Running",
  "endpoint_base_url": "https://my-llm-a1b2c3.inf.foundrydb.com/v1",
  "provisioning_message": ""
}

While the deploy is in flight, provisioning_message carries a live line (weight download progress, server start, the readiness wait), so a long poll is not a silent one.

endpoint_base_url is your service's own OpenAI base URL, and it is what every call below uses. Do not assemble it yourself and do not point a client at a shared hostname: the hostname is how the platform knows which of your services a request is for, and a foundrydb_managed/ call sent anywhere else is refused with 412 provider_not_configured. Capture it:

export INF_BASE=$(curl -s https://api.foundrydb.com/inference-services/$SERVICE_ID \
  -H "Authorization: Bearer $FOUNDRYDB_TOKEN" | jq -r .endpoint_base_url)

The service now serves traffic and accrues GPU-hour charges while it runs. The served model name for mistral-small is mistral-small, so the model string you will call is foundrydb_managed/mistral-small.

Step 3: Mint an Inference Key​

Data plane calls authenticate with an organization inference key (fdb-inf-...), not your platform API token. The secret is shown exactly once, so store it when you create it. monthly_token_limit is required and must be positive: there is no unlimited key.

curl -u "$FOUNDRYDB_USER:$FOUNDRYDB_PASSWORD" \
  https://api.foundrydb.com/organizations/$ORG_ID/inference/keys \
  -H "Content-Type: application/json" \
  -d '{
    "name": "first-model-app",
    "monthly_token_limit": 5000000,
    "rate_limit_rpm": 120
  }'

The response returns the full secret once, in the secret field:

{
  "key": { "name": "first-model-app", "monthly_token_limit": 5000000, "rate_limit_rpm": 120 },
  "secret": "fdb-inf-3f4a...",
  "activation_note": "The key activates at the inference endpoint within a few seconds, once the edge fleet applies it. A request sent immediately after minting can answer invalid_key; retry shortly."
}

Take the activation_note seriously if you are scripting this: the key's hash reaches the serving fleet a moment after the mint returns, so a call issued in the same breath can answer invalid_key. Wait a couple of seconds and retry.

Store it in your shell for the next step:

export FDB_INF_KEY="fdb-inf-3f4a..."

Step 4: Call It with curl​

The base URL is $INF_BASE (the service's own endpoint_base_url), the model string is foundrydb_managed/mistral-small, and the auth is your inference key as a Bearer token.

curl $INF_BASE/chat/completions \
  -H "Authorization: Bearer $FDB_INF_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "foundrydb_managed/mistral-small",
    "messages": [
      {"role": "user", "content": "Write a haiku about databases."}
    ]
  }'

You get a standard OpenAI chat completion object back, with the answer in choices[0].message.content and token counts in usage.

Streaming variant​

Set "stream": true to receive a server-sent event stream. The proxy injects the usage chunk, so the final event always carries token counts.

curl $INF_BASE/chat/completions \
  -H "Authorization: Bearer $FDB_INF_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "foundrydb_managed/mistral-small",
    "messages": [
      {"role": "user", "content": "Write a haiku about databases."}
    ],
    "stream": true
  }'

Step 5: Call It with the OpenAI Python SDK​

Any OpenAI-compatible client works. Point it at the service's endpoint_base_url and use your inference key as the API key.

from openai import OpenAI

client = OpenAI(
    base_url="https://my-llm-a1b2c3.inf.foundrydb.com/v1",  # the service's endpoint_base_url
    api_key="fdb-inf-3f4a...",  # your organization inference key
)

response = client.chat.completions.create(
    model="foundrydb_managed/mistral-small",
    messages=[{"role": "user", "content": "Write a haiku about databases."}],
)
print(response.choices[0].message.content)

Streaming from the SDK uses the same pattern with stream=True:

stream = client.chat.completions.create(
    model="foundrydb_managed/mistral-small",
    messages=[{"role": "user", "content": "Write a haiku about databases."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Step 6: Check Usage and Cost​

Inspect consumption for the organization, grouped by model:

curl -s -u "$FOUNDRYDB_USER:$FOUNDRYDB_PASSWORD" \
  "https://api.foundrydb.com/organizations/$ORG_ID/inference/usage?group_by=model"
{
  "from": "2026-07-01T00:00:00Z",
  "to": "2026-07-19T12:00:00Z",
  "group_by": "model",
  "rows": [
    {
      "group_key": "foundrydb_managed/mistral-small",
      "calls": 12,
      "input_tokens": 240,
      "output_tokens": 180,
      "total_tokens": 420
    }
  ]
}

Group by key instead of model to see per-application consumption. The same response carries a free_tier block with the organization's monthly free token allowance standing.

Which of these figures is a charge depends on the SKU. The dedicated service you built bills by the GPU-hour for as long as the card is allocated, not per token, so its token counts are observability and per-key ceilings rather than a bill. A serverless service is the other way round: the tokens are the bill. For one service's own figures, including a month-to-date rollup of both meters, read GET /inference-services/$SERVICE_ID/usage.

What You Built​

  • A dedicated EU GPU running mistral-small under vLLM, reached at the service's own endpoint.
  • An organization inference key with a monthly token ceiling and an RPM limit.
  • Working chat completions (streaming and non-streaming) from both curl and the OpenAI Python SDK.

Cleanup​

Billing is by the GPU-hour for as long as the card is allocated. To stop paying for a model you are not using but want to keep, stop it with POST /managed-services/$SERVICE_ID/stop: the meter ends, the weights stay on the data disk, and a later start reloads them warm in about a minute. To be rid of it entirely, delete the service:

curl -X DELETE https://api.foundrydb.com/inference-services/$SERVICE_ID \
  -H "Authorization: Bearer $FOUNDRYDB_TOKEN"

Teardown of the vLLM runtime, ingress, certificate, DNS, floating IP, and GPU server happens asynchronously. Billing stops when the service is no longer running.

Next Steps​