I need an AI Gateway

16 min

If you do not already have an AI Gateway, this guide walks you through deploying LiteLLM using Docker and connecting it to Amberflo. LiteLLM is the first supported gateway, and additional gateways will be available soon.

Once deployed, the gateway will push real-time usage events to Amberflo, enabling attribution, budgets, cost guards, dashboards, and full AI Governance and Control.

Prerequisites

You will need:

Docker installed and running

An Amberflo account

A Postgres database (required by LiteLLM for saving providers, models, teams and virtual keys)

Create a Working Directory

mkdir litellm-gateway
cd litellm-gateway

Pull the LiteLLM Docker Image and Create a Config

Pull the LiteLLM proxy (AI Gateway) image:

docker pull ghcr.io/berriai/litellm:v1.79.0-stable

Create a basic LiteLLM config file in your folder and name it config.yaml

# LiteLLM Proxy Configuration (config.yaml)

# General / Proxy-wide settings
general_settings:

litellm_settings:
  callbacks:
    - "amberflo.litellm.callback"

Download Amberflo Callback and set up environment file

You can find the Amberflo Callback zip in the AI Gateway Setup Wizard.

Download:

Amberflo ZIP file (callback package)

Move the files into your working directory and unzip:

unzip amberflo.zip -d .

Your directory should now contain:

config.yaml
.env
amberflo/
    __init__.py
    (other callback files)

Creating an environment file

An environment file stores configuration values that the gateway reads at startup. It keeps sensitive or user-specific settings separate from your main configuration so you do not hard-code them into the YAML file.

Follow these steps:

Create a file named .env in the same directory where you created your config.yaml.

Copy the values displayed in the wizard and paste them into the .env file.

Update any placeholders with the correct values for your setup.

Save the file.

Once saved, the gateway reads this file automatically when it starts. You can review the details of each variable in the sections below, including what each key represents and how it is used.

LITELLM_MASTER_KEY - A required secret used to encrypt and decrypt sensitive fields stored in the LiteLLM database. This key must be a long, random string. Changing it will invalidate previously encrypted data.

LITELLM_SALT_KEY - A cryptographic salt used for hashing and securing stored values. This must also be a long, random string. Do not reuse the same value across environments.

UI_USERNAME - The username for logging into the LiteLLM admin UI. This is the credential used for the web dashboard, not for model authentication.

UI_PASSWORD - The password for logging into the LiteLLM admin UI. Choose a strong, random password. If this value changes, existing sessions become invalid.

DATABASE_URL - The full Postgres connection string used by LiteLLM’s Prisma client. It must be unquoted and begin with postgresql://. It defines the database host, port, user, password, and database name.

STORE_MODEL_IN_DB - This allows you to add models using the Admin UI instead of only via the config. This should be set to true.

LITELLM_MASTER_KEY=
LITELLM_SALT_KEY=
UI_USERNAME=
UI_PASSWORD=
DATABASE_URL=
STORE_MODEL_IN_DB=true

Do not commit this file to git.

Optionally you can update AFLO_HOSTED_ENV value. The string set as the value will be used to identify the instance of the AI Gateway. Amberflo supports the ability to connect multiple AI Gateways and this will allow you to filter the data based on which AI Gateway instance the data is coming from.

Create and Configure the Postgres Database

LiteLLM requires Postgres for:

Teams

Virtual keys

Storing models

Create or provision a Postgres instance.

Construct your connection URL:

postgresql://<USERNAME>:<PASSWORD>@<HOST>:<PORT>/<DB_NAME>

Make sure your Postgres DB is running and then update your environment file to include the connection string.

NOTE: If you are running the Postgres DB on your local machine you need you use host.docker.internal not localhost and the default port for Postgres is 5432

DATABASE_URL: postgresql://<USERNAME>:<PASSWORD>@<HOST>:<PORT>/<DB_NAME>

Start the LiteLLM Gateway Container

Run:

docker run \
  --env-file .env \
  --volume ./amberflo:/app/amberflo:ro \
  --volume ./config.yaml:/app/config.yaml:ro \
  --publish 4000:4000 \
  ghcr.io/berriai/litellm:v1.79.0-stable \
  --config /app/config.yaml

This will:

Load the Amberflo callback

Load your environment file

Connect LiteLLM to Postgres

Start the gateway on port 4000

Test Your Integration

Check that the Amberflo integration is working

You can verify that the gateway is correctly sending meter events to Amberflo before configuring any providers or models.

Use the LiteLLM Master Key that you created during deployment.

Make a request to the gateway with a model name that does not exist. The call will fail, but the failure still produces a meter event.

If the integration is set up correctly, Amberflo will receive that event within one or two minutes and the demo data banner will disappear.

Make sure to set the IP of your gateway and your master key in the sample cURL below.

curl http://<YOUR_VM_IP>:4000/chat/completions \
  -H "Authorization: Bearer <YOUR_MASTER_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"model": "my_model","messages": [{"role": "user", "content": "How are tokens calculated?"}]}'

What to check:

In the Amberflo app, open the AI Metering page.

Look for a new entry under the LM API call error details meter.

If you see the event, the integration is working and you can continue to provider and model configuration in the next step.

Complete AI Gateway Configuration

LiteLLM does not emit any usage until all required objects are created in the admin UI. You must configure the gateway in the following order: 1. Providers

Create a provider entry for each upstream service you plan to use. Examples include OpenAI, Anthropic, and Bedrock. Add the required API keys or access tokens for each provider. Until a provider is created with valid credentials, no model that references it can run.

2. Models

Define the models that the gateway will expose. Examples include GPT-5 or Claude Sonnet. Each model must reference one of the providers you created above. This step establishes which upstream model is called, its model identifier, and any model-level parameters.

3. Teams

Create one or more teams. A team is the entity to which usage and cost will be attributed inside Amberflo. This is how you can break down you AI spend. And more specifically know who is using what and how much it is costing you.

4. Virtual Keys

Create a virtual key for testing and for any client that will issue requests through the gateway. A virtual key authenticates the request and determines which team the usage belongs to. Without a virtual key, the gateway cannot be called and no usage will be recorded.

After completing these four steps, the gateway is fully configured and can emit usage events to Amberflo.

⚠️ Go to the following page to learn how to set up a provider, model, team and virtual key. They are required to test the integration with the AI Gateway. Once you have completed those items you can complete the next steps.

How to configure LiteLLM with providers, models, teams and virtual keysSetting up

Test the Full Integration

Step 1: Call the Gateway

Use a virtual key assigned to a team. This is a sample cURL command to call the LiteLLM AI Gateway. You need to replace the following values:

YOUR_VM_IP: The public or private IP address of the machine running the LiteLLM gateway. If you are using Docker locally, this is usually localhost. If the gateway is on a VM, this is that VM’s IP.

YOUR_VIRTUAL_KEY: The virtual key you created in the LiteLLM admin UI. This key identifies the caller, determines which Team the usage is attributed to, and is required for authentication.

MODEL_ID: The model identifier you defined when creating a Model in the admin UI. This is the exact string LiteLLM expects (for example gpt-4o-mini or claude-3-sonnet). If this does not match a configured model, the request will fail and no usage will be recorded.

curl http://<YOUR_VM_IP>:4000/chat/completions \
  -H "Authorization: Bearer <YOUR_VIRTUAL_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"model": "<MODEL_ID>","messages": [{"role": "user", "content": "How are tokens calculated?"}]}'

Step 2: Verify in Amberflo

Once you've successfully made the API call in the previous step you should log in to Amberflo and check the AI Spend Dashboard's Summary page. You will begin to see your usage and cost show up there.

Events should appear in near real time. It can take up to 2 minutes for the first data to show up. You may need to refresh the page.

Automatic Business Unit Creation

Amberflo automatically creates a new Business Unit the first time a virtual key is used.

Mapping:

LiteLLM team name → Business Unit name

LiteLLM team ID → Business Unit ID

All future events for that key are attributed to that Business Unit.

You can rename Business Units later if needed. The first time you send data for a particular team you will see only their LiteLLM team ID shown in the Amberflo App. If can take up to 5 minutes for the business unit to be fully created in Amberflo.