Add a Token Meter, Not Another Monitor, to Free Model Jobs

Add a Token Meter, Not Another Monitor, to Free Model Jobs

You merge a small refactor at 4:56 PM.

The pipeline goes green. The nightly job stays green. Two weeks later, you notice the free model endpoint has been slow-burning your time budget while every status check kept saying 'okay.'

Nothing failed. Nothing logged a problem. You just lost an evening to a job that felt sluggish for no visible reason.

The gap is metering, not uptime

Most CI observability for model jobs starts with the wrong question. It asks whether the endpoint is up. The better question is what this job actually consumed.

Free endpoints change quietly. Quotas shift. Routing changes. One day a 20-second job finishes in 90 seconds. A monitor that pings /v1/models stays green the whole time.

I do not want another dashboard. I want a meter that sits between the pipeline and the model, reads the response metadata, and fails the job when it crosses a local ceiling.

The wrapper treats the model as an endpoint, so it works whether you use MonkeyCode's free model access or its free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Build the meter as a standard-library script

Keep it boring. No requests, no agent framework. Just Python's standard library plus one prompt argument.

import json
import os
import sys
import time
import urllib.request

model_url = os.environ['MODEL_URL']
model_key = os.environ['MODEL_KEY']
model_name = os.environ['MODEL_NAME']
budget = int(os.environ.get('TOKEN_BUDGET', '800'))
latency_ms = int(os.environ.get('LATENCY_MS', '3000'))
prompt = sys.argv[1]

payload = json.dumps({
    'model': model_name,
    'messages': [{'role': 'user', 'content': prompt}],
    'max_tokens': 200,
}).encode('utf-8')

request = urllib.request.Request(model_url, data=payload, method='POST')
request.add_header('Authorization', 'Bearer ' + model_key)
request.add_header('Content-Type', 'application/json')

start = time.time()
timeout_seconds = latency_ms / 1000
with urllib.request.urlopen(request, timeout=timeout_seconds) as response:
    raw = response.read()
elapsed_ms = int((time.time() - start) * 1000)

data = json.loads(raw)
usage = data.get('usage', {})
prompt_tokens = usage.get('prompt_tokens', 0)
completion_tokens = usage.get('completion_tokens', 0)
total_tokens = prompt_tokens + completion_tokens

if total_tokens > budget:
    raise SystemExit(f'token budget exceeded: {total_tokens}/{budget}')

if elapsed_ms > latency_ms:
    raise SystemExit(f'latency budget exceeded: {elapsed_ms}ms/{latency_ms}ms')

content = data.get('choices', [{}])[0].get('message', {}).get('content', '')
if not content:
    raise SystemExit('missing content in model response')

print(content)

Why each choice matters:

• The script has no retry logic. A slow or malformed response fails the job instead of hiding it.
• The token check adds both prompt and completion tokens, not just the max_tokens value.
• Latency is measured around the HTTP call, including connection setup and body download.
• Stdout stays clean. Only the model answer reaches the next pipeline step.

One warning: inspect your provider's response once with print(data.keys()) before trusting the meter. Usage fields are not always nested the same way.

Put it in GitLab CI

The script is the meter. The CI job gives it a safe place to run.

meter:
  image: python:3.12-alpine
  script:
    - python meter.py 'List risky files in this diff'
  variables:
    MODEL_URL: your-endpoint-url
    MODEL_KEY: your-secret
    MODEL_NAME: your-model-name
    TOKEN_BUDGET: 800
    LATENCY_MS: 3000

Secrets live in GitLab CI/CD variables, not in the repository. The meter only needs read access to one endpoint.

Why max_tokens and a timeout are not enough

max_tokens bounds only the completion. It does not bound the prompt.

A prompt that accidentally balloons to 3,000 tokens can still eat quota even if the answer is short. A timeout stops a slow call, but it does not tell you whether the job is getting slower over a week.

You need both numbers. One for tokens. One for latency. Otherwise you are still guessing.

Catch what a monitor cannot

The table is the point. A meter is not an evaluation suite. It is a spending guard.

Make the ceiling your own

Your first meter run will probably fail jobs. That is not a bug. It is the signal.

Follow this order:

• Run 10 normal jobs with a generous ceiling and record actual usage.
• Set the token ceiling at the p95 plus a little headroom.
• Keep max_tokens fixed so the completion side stays predictable.
• Split one giant prompt into smaller jobs. A bloated answer is easier to catch.
• Add a comment in GitLab when you change the budget, not after.

This turns a free tier into a number you can reason about.

What this does not solve

Metering is not evaluation. It catches cost and speed, not sense.

• It cannot detect silent quality drift.
• It depends on the provider returning usage metadata. Some free endpoints hide it.
• A local ceiling can be wrong after a legitimate model update. You will need to re-tune.
• It does not retry, paginate, or choose a better model.

Call it what it is: a tripwire, not a proof of correctness.

Who should skip it

• One-off calls where cost and latency do not matter.
• Jobs whose output is deterministic and can be checked without a model.
• Teams that need a model-quality regression suite, not a spending guard.
• Environments without Python 3 or read access to response usage.

Do not bolt a meter onto every job. Start with the one that costs the most.

Conclusion

Start with one job. Attach the meter. Tune the budget from real numbers.

The status monitor will keep saying the endpoint is up. At least your pipeline will know when a 'free' call stops being cheap.

Point this wrapper at your worst job this week. You likely need the meter more than another dashboard.