Skip to main content

Deploy and Call Your First Model

Managed Inference runs an open-weight LLM on a dedicated GPU in a European zone and exposes it through an OpenAI-compatible endpoint. In this tutorial you provision a curated model, mint an inference key, call it with both curl and the OpenAI Python SDK, and check your usage. Start to finish, the GPU provisions in a few minutes.

You never call the GPU directly. Every request goes through the shared inference proxy, which authenticates it, applies your per-key limits, and forwards it to your GPU over a certificate-pinned TLS connection.

Prerequisites​

  • A FoundryDB organization with a prepaid balance that covers the GPU plan (the create returns 402 otherwise).
  • Your organization id (ORG_ID).
  • A platform API token (FOUNDRYDB_TOKEN) for the /inference-services API, plus owner or admin basic-auth credentials (FOUNDRYDB_USER, FOUNDRYDB_PASSWORD) for the organization inference endpoints.
  • Python 3.9+ if you want to run the SDK example.

Set these once in your shell:

export FOUNDRYDB_TOKEN="..."          # platform API token
export FOUNDRYDB_USER="..."           # owner/admin login
export FOUNDRYDB_PASSWORD="..."
export ORG_ID="<your-org-uuid>"

Step 1: Create a Managed Inference Service​

Pick a model from the curated catalog. This example uses mistral-small (Apache-2.0, no license acceptance needed) on its default GPU plan, gpu-l40s-1. All GPU plans run in fi-hel2 (Helsinki).

curl https://api.foundrydb.com/inference-services \
  -H "Authorization: Bearer $FOUNDRYDB_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-llm",
    "plan_name": "gpu-l40s-1",
    "zone": "fi-hel2",
    "inference_config": {
      "model_id": "mistral-small",
      "model_source": "curated"
    }
  }'

Provisioning is asynchronous. The response returns the service in Pending, and the controller drives it to Running by provisioning the GPU server, installing the vLLM runtime, warming the model, and installing the serving certificate. Capture the returned service id:

export SERVICE_ID="<id-from-response>"

License-gated models​

mistral-small is Apache-2.0, so nothing extra is required. If you choose a license-gated curated model such as llama-3.3-70b, add license_accepted: true inside inference_config. Omitting it for a gated model returns 400.

curl https://api.foundrydb.com/inference-services \
  -H "Authorization: Bearer $FOUNDRYDB_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-llama",
    "plan_name": "gpu-h100-1",
    "zone": "fi-hel2",
    "inference_config": {
      "model_id": "llama-3.3-70b",
      "model_source": "curated",
      "license_accepted": true
    }
  }'

Step 2: Wait Until It Is Running​

Poll the service until status reads Running:

curl -s https://api.foundrydb.com/inference-services/$SERVICE_ID \
  -H "Authorization: Bearer $FOUNDRYDB_TOKEN" | jq '.status'

Expected output once warmup completes:

"Running"

The service now serves traffic through the proxy and accrues GPU-hour charges while it runs. The served model name for mistral-small is mistral-small, so the model string you will call is foundrydb_managed/mistral-small.

Step 3: Mint an Inference Key​

Data plane calls authenticate with an organization inference key (fdb-inf-...), not your platform API token. The secret is shown exactly once, so store it when you create it. monthly_token_limit is required and must be positive: there is no unlimited key.

curl -u "$FOUNDRYDB_USER:$FOUNDRYDB_PASSWORD" \
  https://api.foundrydb.com/organizations/$ORG_ID/inference/keys \
  -H "Content-Type: application/json" \
  -d '{
    "name": "first-model-app",
    "monthly_token_limit": 5000000,
    "rate_limit_rpm": 120
  }'

The response returns the full secret once, in the secret field:

{
  "key": { "name": "first-model-app", "monthly_token_limit": 5000000, "rate_limit_rpm": 120 },
  "secret": "fdb-inf-3f4a..."
}

Store it in your shell for the next step:

export FDB_INF_KEY="fdb-inf-3f4a..."

Step 4: Call It with curl​

The base URL is https://inference.foundrydb.com/v1, the model string is foundrydb_managed/mistral-small, and the auth is your inference key as a Bearer token.

curl https://inference.foundrydb.com/v1/chat/completions \
  -H "Authorization: Bearer $FDB_INF_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "foundrydb_managed/mistral-small",
    "messages": [
      {"role": "user", "content": "Write a haiku about databases."}
    ]
  }'

You get a standard OpenAI chat completion object back, with the answer in choices[0].message.content and token counts in usage.

Streaming variant​

Set "stream": true to receive a server-sent event stream. The proxy injects the usage chunk, so the final event always carries token counts.

curl https://inference.foundrydb.com/v1/chat/completions \
  -H "Authorization: Bearer $FDB_INF_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "foundrydb_managed/mistral-small",
    "messages": [
      {"role": "user", "content": "Write a haiku about databases."}
    ],
    "stream": true
  }'

Step 5: Call It with the OpenAI Python SDK​

Any OpenAI-compatible client works. Point it at the proxy base URL and use your inference key as the API key.

from openai import OpenAI

client = OpenAI(
    base_url="https://inference.foundrydb.com/v1",
    api_key="fdb-inf-3f4a...",  # your organization inference key
)

response = client.chat.completions.create(
    model="foundrydb_managed/mistral-small",
    messages=[{"role": "user", "content": "Write a haiku about databases."}],
)
print(response.choices[0].message.content)

Streaming from the SDK uses the same pattern with stream=True:

stream = client.chat.completions.create(
    model="foundrydb_managed/mistral-small",
    messages=[{"role": "user", "content": "Write a haiku about databases."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Step 6: Check Usage and Cost​

Inspect consumption for the organization, grouped by model:

curl -s -u "$FOUNDRYDB_USER:$FOUNDRYDB_PASSWORD" \
  "https://api.foundrydb.com/organizations/$ORG_ID/inference/usage?group_by=model"
{
  "from": "2026-07-01T00:00:00Z",
  "to": "2026-07-19T12:00:00Z",
  "group_by": "model",
  "rows": [
    {
      "group_key": "foundrydb_managed/mistral-small",
      "calls": 12,
      "input_tokens": 240,
      "output_tokens": 180,
      "total_tokens": 420
    }
  ]
}

Group by key instead of model to see per-application consumption. Note that Managed Inference bills by the GPU-hour while the service is running, not per token: the token counts here are for observability and per-key ceilings.

What You Built​

  • A dedicated EU GPU running mistral-small under vLLM, reached through the FoundryDB inference proxy.
  • An organization inference key with a monthly token ceiling and an RPM limit.
  • Working chat completions (streaming and non-streaming) from both curl and the OpenAI Python SDK.

Cleanup​

Billing is by the GPU-hour while the service runs, and there is no paused state. To stop paying for a model you are not using, delete the service:

curl -X DELETE https://api.foundrydb.com/inference-services/$SERVICE_ID \
  -H "Authorization: Bearer $FOUNDRYDB_TOKEN"

Teardown of the vLLM runtime, ingress, certificate, DNS, floating IP, and GPU server happens asynchronously. Billing stops when the service is no longer running.

Next Steps​