One of the most common failures with a free model endpoint in GitLab CI is not a wrong answer. It is a silent slowdown, an unexpected 429, or a timeout that leaves no trace in the job log. The job that summarizes failed tests has been running fine for three days, then it suddenly takes forty seconds and dies with a generic upstream error. The provider dashboard shows no useful history, and the only signal is a longer pipeline. You are flying blind.
The free model access and free server option referenced here are operator-supplied facts from MonkeyCode, and I have not verified their quotas, uptime, or stability. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
This post describes a small observability layer you can place in front of a free model endpoint. It records only request metadata — timestamps, status codes, latency, payload sizes, and error types — without storing prompts, responses, tokens, or any other potentially sensitive model input. The artifact is a GitLab CI job that runs a local reverse proxy, sends a controlled burst of smoke requests, then writes a decision table you can commit as an artifact.
Why storing prompts is a trap
When a model call fails, developers usually want to save the full request and response for debugging. That is tempting, but it creates three problems in CI:
- Leak risk. Prompts often contain source code, file paths, branch names, or environment snippets. Storing them in job artifacts, especially public pipelines, violates data policies.
- Noise. A dozen prompts and completions make it harder to see the operational pattern: latency spikes, status-code distribution, or queue buildup.
- Storage cost over time. A free endpoint may be called hundreds of times per merge request. Full payloads accumulate quickly, even when the failures are rare.
A narrower log that captures only the meta-information around each request is usually enough to answer the first important question: is this endpoint reliable enough to build on?
The metadata you actually need
For a free model used by an internal pipeline, the following fields provide a useful baseline without exposing sensitive content:
-
timestamp— when the request started -
request_id— a random UUID to correlate later analysis -
status_code— returned by upstream or generated by the proxy on timeout -
duration_ms— end-to-end time measured by the proxy -
prompt_chars— size of the request body in bytes, not the content -
response_chars— size of the response body in bytes -
error_type— only when the proxy itself fails, such astimeoutorconnection_error
This is deliberately small. It tells you whether the endpoint is slow, rate-limiting, or returning non-200 responses, but it does not reveal what the prompt said or what the model answered.
A tiny reverse proxy that logs without leaking
The following Python script uses the standard library to start a local HTTP server, forward each request to the free model endpoint, and write one row of metadata to SQLite for every call. It does not log the request body beyond its length.
#!/usr/bin/env python3
"""Forward /chat/completions-style requests and log only metadata.
Usage:
UPSTREAM_URL=https://provider.example/v1/chat/completions \
API_TOKEN=your-token python3 meta_proxy.py
The proxy listens on 127.0.0.1:9090 and writes rows to model_meta.sqlite.
"""
import os
import sqlite3
import time
import uuid
from http.server import BaseHTTPRequestHandler, HTTPServer
import requests
DB_PATH = "model_meta.sqlite"
UPSTREAM_URL = os.environ.get("UPSTREAM_URL")
API_TOKEN = os.environ.get("API_TOKEN", "")
PORT = 9090
CONNECT_TIMEOUT = 5
READ_TIMEOUT = 60
def init_db():
conn = sqlite3.connect(DB_PATH)
conn.execute(
"""
CREATE TABLE IF NOT EXISTS requests (
id TEXT PRIMARY KEY,
ts REAL,
status INTEGER,
duration_ms REAL,
prompt_chars INTEGER,
response_chars INTEGER,
error_type TEXT
)
"""
)
conn.commit()
conn.close()
class MetaProxyHandler(BaseHTTPRequestHandler):
def do_POST(self):
request_id = str(uuid.uuid4())
start = time.monotonic()
content_length = int(self.headers.get("Content-Length", 0))
body = self.rfile.read(content_length)
prompt_chars = len(body)
status = 0
response_chars = 0
error_type = None
try:
upstream = requests.post(
UPSTREAM_URL,
data=body,
headers={
"Authorization": f"Bearer {API_TOKEN}",
"Content-Type": "application/json",
},
timeout=(CONNECT_TIMEOUT, READ_TIMEOUT),
)
status = upstream.status_code
response_chars = len(upstream.content)
self.send_response(upstream.status_code)
self.send_header(
"Content-Type",
upstream.headers.get("Content-Type", "application/json"),
)
self.end_headers()
self.wfile.write(upstream.content)
except requests.exceptions.Timeout:
error_type = "timeout"
status = 504
self.send_error(504, "upstream timeout")
except Exception as exc:
error_type = type(exc).__name__
status = 502
self.send_error(502, "upstream error")
finally:
duration_ms = (time.monotonic() - start) * 1000
conn = sqlite3.connect(DB_PATH)
conn.execute(
"""
INSERT OR REPLACE INTO requests
(id, ts, status, duration_ms, prompt_chars, response_chars, error_type)
VALUES (?, ?, ?, ?, ?, ?, ?)
""",
(
request_id,
start,
status,
duration_ms,
prompt_chars,
response_chars,
error_type,
),
)
conn.commit()
conn.close()
if __name__ == "__main__":
if not UPSTREAM_URL:
raise SystemExit("UPSTREAM_URL is required")
init_db()
server = HTTPServer(("127.0.0.1", PORT), MetaProxyHandler)
print(f"meta proxy listening on 127.0.0.1:{PORT}")
server.serve_forever()
This is not a production reverse proxy. It is a debugging instrument. It runs on localhost, does not bind to external interfaces, and does not queue or retry beyond the upstream call it immediately forwards. The only persistence is the SQLite file with metadata.
Aggregating results into a single decision table
After the proxy has collected a few dozen observations, a second script turns the rows into something a human can read in a merge request artifact. It calculates the median and 95th percentile latency, counts non-200 responses, and breaks down the status-code distribution.
#!/usr/bin/env python3
"""Print a summary table from model_meta.sqlite."""
import sqlite3
import statistics
DB_PATH = "model_meta.sqlite"
def quantile(values, q):
if not values:
return 0.0
values = sorted(values)
index = max(0, min(len(values) - 1, int(len(values) * q)))
return values[index]
def main():
conn = sqlite3.connect(DB_PATH)
rows = conn.execute(
"SELECT status, duration_ms, error_type FROM requests"
).fetchall()
conn.close()
if not rows:
print("No observations yet. Run the proxy against several smoke requests first.")
return
durations = [row[1] for row in rows]
success = sum(1 for row in rows if row[0] == 200)
errors = [row for row in rows if row[2] is not None]
print(f"total_requests={len(rows)}")
print(f"success_200={success}")
print(f"proxy_errors={len(errors)}")
print(f"p50_ms={statistics.median(durations):.0f}")
print(f"p95_ms={quantile(durations, 0.95):.0f}")
print(f"max_ms={max(durations):.0f}")
status_counts = {}
for row in rows:
status_counts[row[0]] = status_counts.get(row[0], 0) + 1
print(f"status_distribution={status_counts}")
if __name__ == "__main__":
main()
Use the output as a go/no-go table, not as a benchmark. A free endpoint that returns a clean 200 in under two seconds for 95% of smoke requests is a reasonable candidate for a low-stakes CI job. If the 95th percentile is above thirty seconds or if the first 429 appears after only a handful of requests, hold off.
Pipeline placement: before the expensive stages
The right place for this observability job is early in the pipeline, before any stage that depends on the free model. In GitLab CI that could look like this:
free-model-meta:
stage: test
image: python:3.12-slim
variables:
UPSTREAM_URL: "${MONKEYCODE_FREE_URL}"
API_TOKEN: "${MONKEYCODE_FREE_TOKEN}"
before_script:
- pip install requests
script:
- python meta_proxy.py &
- PROXY_PID=$!
- sleep 2
- for i in 1 2 3 4 5 6 7 8 9 10; do
curl -s -X POST http://127.0.0.1:9090/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"smoke"}],"max_tokens":16}' >/dev/null;
done
- kill $PROXY_PID
- python meta_report.py
artifacts:
when: always
paths:
- model_meta.sqlite
reports:
junit: report.xml
The curl loop is deliberately boring. It sends the same minimal prompt multiple times so the operational signal is not mixed with prompt variability. The token is supplied from a masked GitLab CI variable, never committed to the repository.
Limitations of meta-only observability
Because the proxy intentionally does not store prompts or completions, it cannot answer why a specific request produced a wrong answer or which input triggered a rate limit. It will not tell you whether the model hallucinated, emitted invalid JSON, or ignored an instruction. Those concerns belong to separate evaluation and review layers.
A single SQLite file is also not sufficient for high concurrency. Multiple pipeline jobs writing to the same proxy process can cause lock contention or lost rows. If you need to observe many concurrent jobs, replace SQLite with a central log aggregator or a proper service mesh.
The free server option may have cold starts or change its behavior between runs. The latency numbers captured today may not hold tomorrow. Treat the report as a snapshot of recent behavior, not as a service-level objective.
Who should not use this approach
Skip this observability layer if you already consume the endpoint through a managed API gateway that records structured metrics, or if the free model is only used for one-off interactive prompts rather than automated pipeline jobs. It is also a poor fit when your prompts are highly sensitive and even payload length is considered confidential, when the endpoint requires a custom authentication flow beyond a bearer token, or when the project has no tolerance for extra CI runtime.
Do not use a local proxy as a way to hide abuse, evade provider rate limits, or obscure who is making model calls. The goal is transparency about operational health, not invisibility.
Final step
Before you commit to using a free model in a real merge request workflow, run the observability job for a week of pipeline activity. Collect the metadata, read the status distribution, and then decide whether the endpoint deserves a dependency in your critical path. A model that answers beautifully under ideal conditions but times out under load is still a broken dependency. Log the meta, not the prompt.
Top comments (0)