Accessible Streaming Probe: Fail on Purpose Before Swapping Models

#a11y #webdev #ai #javascript

A cheaper model endpoint becomes expensive the moment a screen reader user cannot tell whether the response is still loading or has quietly died. I've learned that before pointing an accessible chat interface at a new free tier, the most useful move is to make the failure modes boring first: build a small SSE server that misbehaves on purpose, then drive a keyboard-operable probe through each bad transition. The probe is the artifact here, and its goal is not to evaluate prose quality but to catch focus loss, missing announcements, and silent terminal states that only appear when a stream breaks.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you have access to MonkeyCode's free server option, use it for the mock tier described below; the free model access gives you a real endpoint to compare after the probe passes.

Start with an endpoint that fails on purpose

I've found that a local mock is not enough because localhost hides the buffering, proxy, and network timeouts that change how an aria-live region behaves. Running the mock from a free server slot puts a real network path between the button that starts a stream and the region that announces it. The mock speaks Server-Sent Events, but each failure mode violates one part of the contract a chat UI depends on:

Normal completes the stream and sends a final done event.
Slow delays the first token by 400 ms.
Truncate drops the final done event after sending part of the response.
429 reports a rate limit after headers.
Silent holds the connection open without writing anything.

import { createServer } from 'node:http';

const wait = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
const words = 'The quick brown fox jumps over the lazy dog'.split(' ');

createServer(async (req, res) => {
  const url = new URL(req.url, 'http://localhost:8787');
  if (url.pathname !== '/stream') {
    res.writeHead(404).end();
    return;
  }

  const mode = url.searchParams.get('mode') ?? 'normal';
  res.writeHead(200, {
    'Content-Type': 'text/event-stream',
    'Cache-Control': 'no-cache',
    'Connection': 'keep-alive'
  });

  if (mode === 'silent') {
    await new Promise((resolve) => {
      req.on('close', resolve);
      res.on('close', resolve);
    });
    return;
  }

  if (mode === '429') {
    res.write('event: error\ndata: rate limited\n\n');
    res.end();
    return;
  }

  for (const [index, word] of words.entries()) {
    if (mode === 'truncate' && index === 4) {
      res.end();
      return;
    }
    const delay = mode === 'slow' ? 400 : 60;
    await wait(delay);
    res.write(`event: token\ndata: ${word}\n\n`);
  }

  if (mode === 'normal') {
    res.write('event: done\ndata: complete\n\n');
  }

  res.end();
}).listen(8787, () => console.log('Mock SSE on http://localhost:8787/stream?mode=normal'));

Run this locally first, then move it to whatever free server slot you have. The network path matters because a proxy can buffer SSE chunks until the response closes, which makes a silent hold and a slow first token look identical in the browser. The client must tell them apart with a timeout instead of waiting for the server to explain itself.

Build the accessible probe

The probe is a single HTML file. In my experience, the output itself should not be a live region. The status element is the only thing with aria-live, and it changes only when the state changes. If you put aria-live on the output, some screen readers will read every token as it arrives, which makes cancel and retry announcements hard to hear. I recommend reviewing the ARIA live region guidance before changing that.

<!doctype html>
<html lang='en'>
<head>
  <meta charset='utf-8'>
  <title>Streaming state probe</title>
  <style>
    body { font: 16px/1.5 system-ui; max-width: 60rem; margin: auto; padding: 1rem; }
    [aria-busy='true'] { outline: 2px solid CanvasText; }
    button { font: inherit; margin-inline-end: .5rem; }
  </style>
</head>
<body>
  <h1>Streaming state probe</h1>
  <label for='mode'>Failure mode</label>
  <select id='mode'>
    <option value='normal'>Normal</option>
    <option value='slow'>Slow first token</option>
    <option value='truncate'>Truncated stream</option>
    <option value='429'>Mid-stream rate limit</option>
    <option value='silent'>Silent hold</option>
  </select>
  <button id='start'>Start</button>
  <button id='cancel' hidden>Cancel</button>
  <button id='retry' hidden>Retry</button>
  <div id='status' aria-live='polite' aria-atomic='true'></div>
  <output id='response' aria-label='Model response'></output>

  <script type='module'>
    const mode = document.getElementById('mode');
    const start = document.getElementById('start');
    const cancel = document.getElementById('cancel');
    const retry = document.getElementById('retry');
    const status = document.getElementById('status');
    const response = document.getElementById('response');

    let controller = null;
    let timeout = null;

    function setStatus(text, { startHidden = true, cancelHidden = true, retryHidden = true, busy = false } = {}) {
      status.textContent = text;
      start.hidden = startHidden;
      cancel.hidden = cancelHidden;
      retry.hidden = retryHidden;
      response.setAttribute('aria-busy', String(busy));
    }

    async function run() {
      controller = new AbortController();
      timeout = setTimeout(() => controller.abort(), 8000);
      setStatus('Loading response', { cancelHidden: false, busy: true });
      response.textContent = '';
      cancel.focus();

      try {
        const res = await fetch(`/stream?mode=${encodeURIComponent(mode.value)}`, {
          signal: controller.signal
        });
        if (!res.ok || !res.body) throw new Error(`HTTP ${res.status}`);

        const reader = res.body.getReader();
        const decoder = new TextDecoder();
        let buffer = '';

        while (true) {
          const { value, done } = await reader.read();
          if (done) break;
          buffer += decoder.decode(value, { stream: true });
          const lines = buffer.split('\n');
          buffer = lines.pop() ?? '';

          for (const line of lines) {
            if (line === 'event: error') throw new Error('Endpoint reported an error');
            if (line.startsWith('data: ')) {
              const chunk = line.slice(6);
              if (chunk === 'complete') {
                setStatus('Response complete', { startHidden: false });
                start.focus();
                return;
              }
              response.textContent += `${chunk} `;
            }
          }
        }

        setStatus('Stream ended before completion', { startHidden: false, retryHidden: false });
        retry.focus();
      } catch (error) {
        if (error.name === 'AbortError') {
          setStatus('Response stopped', { startHidden: false });
          start.focus();
        } else {
          setStatus('Could not load response. Retry available.', { startHidden: false, retryHidden: false });
          retry.focus();
        }
      } finally {
        clearTimeout(timeout);
      }
    }

    start.addEventListener('click', run);
    cancel.addEventListener('click', () => controller?.abort());
    retry.addEventListener('click', run);
  </script>
</body>
</html>

The table below lists the transitions I write down before swapping any model. Each row is a contract: the announcement must match, and focus must move to the button that represents the next likely action.

Transition	Announcement	Focus
idle to streaming	Loading response	Cancel button
streaming to cancelled	Response stopped	Start button
streaming to truncated	Stream ended before completion	Retry button
streaming to 429	Could not load response. Retry available.	Retry button
streaming to complete	Response complete	Start button

If retry does not receive focus after truncate, I do not ship that endpoint. The partial transcript is still in the output, but the user's next action should not be to tab around the page looking for a way back.

Compare with the real free endpoint

When the probe passes against all five modes, I point the same fetch code at the real free model endpoint and keep the same timeout and cancel path. If the real endpoint uses a different event name or omits a final done event, I do not silently special-case it in the UI. Instead, I change the mock so the failure remains reproducible. That way, the synthetic failure modes and the real stream are evaluated against the same state machine. The point of free model access in this workflow is not to prove the model is cheap, but to give the state machine a real stream to fail against after the synthetic failures no longer surprise you. When I use AbortController for the timeout, I keep the same abort path for both the mock and the real endpoint; divergence there hides bugs.

What the probe will not tell you

This mock is not a benchmark. It will not expose tokenization quirks, content filters, actual quota timing, or the exact shape of a provider's error payload. It is also not a substitute for listening to real generated content with a screen reader once the stream is working. If your team already has a seeded failure-injection harness and a continuous accessibility regression suite, this particular probe adds little. If your only question is which model writes better copy, this is the wrong room.

Run the probe across at least the following matrix and write down the browser, OS, and assistive-technology version beside any transition that fails. The silent timeout is the one most likely to differ: some combinations announce the polite text immediately, while others wait until the busy state clears. Record that timing rather than assuming it is consistent.

Environment	Transitions to verify
macOS Safari + VoiceOver	start, cancel after first token, truncate retry
Windows Firefox + NVDA	start, silent timeout, truncate retry
Windows Edge + Narrator	start, mid-stream 429, retry
Android Chrome + TalkBack	start, cancel, complete

If you already have a free server slot, move the mock out of localhost and see which transition stops being announced. The first broken announcement is the real cost of a model swap. Try that today: deploy the probe against your candidate endpoint, run one screen reader pass, and write down the first silent failure. That note will save you more than any model benchmark.