Building a Video Thumbnail Generator Service in Go with FFmpeg

Our search results page was rendering 48 video cards, and 41 of them were pulling thumbnails straight from third-party CDNs. Median LCP on mobile sat at 4.1s. The images were the wrong aspect ratio, the wrong size, and about a third of them 404'd within six months of the video being indexed. We were serving a discovery experience where the primary visual element was outside our control.

The fix was a small Go service that pulls a frame from the source video, crops it to a fixed aspect ratio, encodes it in WebP and AVIF, and writes it to disk behind Cloudflare. It runs alongside our PHP 8.4 app, which is what powers DailyWatch. This post covers what that service actually looks like in production — including the parts that bit us: FFmpeg's process model, the thundering herd on cache miss, and why the naive -ss placement makes your p99 latency 40x worse.

Why a separate service instead of PHP

The existing stack is PHP 8.4 on LiteSpeed with SQLite (FTS5 for search) and Cloudflare in front. That stack is genuinely good at what it does — request comes in, query the index, render a template, respond in 12ms. It is terrible at holding 6 concurrent FFmpeg processes for 3 seconds each.

PHP-FPM's model is one request per worker. If a thumbnail generation takes 2.5s wall clock and you have 40 workers, 40 simultaneous cache misses consume the entire pool and your actual HTML pages stop serving. You can shell out to nohup and background it, but then you have no concurrency limit, no queue depth visibility, and no way to deduplicate two requests for the same video.

Go's goroutine model maps cleanly onto this: thousands of cheap waiting handlers, a bounded worker pool doing the expensive work, and a semaphore between them. The thumbnail service listens on a private port; LiteSpeed proxies /thumb/* to it.

The division of labor ended up as:

• PHP: owns video metadata, search, page rendering, and emits <img> tags pointing at /thumb/{id}/{w}x{h}.webp
• Go service: owns frame extraction, encoding, disk cache, and dedup
• Cloudflare: owns edge caching with a 1-year TTL on immutable URLs
• SQLite: stores the source URL and a thumb_generated_at column so the PHP side knows whether to emit a low-quality placeholder

Extracting the right frame without decoding the whole file

FFmpeg has two ways to seek, and the difference between them is the single biggest performance lever in this entire service.

# Slow: decodes every frame from 0 until 00:00:30
ffmpeg -i input.mp4 -ss 00:00:30 -frames:v 1 out.jpg

# Fast: seeks to the nearest keyframe before demuxing
ffmpeg -ss 00:00:30 -i input.mp4 -frames:v 1 out.jpg

When -ss comes before -i, it's an input option and FFmpeg uses the container index to jump directly to the nearest preceding keyframe. When it comes after, it's an output option — FFmpeg decodes and discards every frame up to that timestamp. On a 40-minute 1080p source, that's the difference between 180ms and 7 seconds.

The tradeoff is accuracy: input seeking lands on a keyframe, which might be up to a GOP-length away from the timestamp you asked for (typically 2–10 seconds). For thumbnails, nobody cares. If you do need frame accuracy, the modern approach is both: -ss 00:00:28 -i input.mp4 -ss 00:00:02 -frames:v 1 — fast-seek most of the way, then accurate-seek the remainder.

The second decision is which timestamp. Grabbing frame 0 gives you a black screen or a studio logo maybe 70% of the time. Two approaches that both work:

Fixed percentage. Take 20% into the duration. Cheap, deterministic, and avoids intros. This is what we ship.

Scene detection. Let FFmpeg pick a frame with high scene-change score:

ffmpeg -ss 10 -i input.mp4 -vf "select='gt(scene,0.4)',scale=640:-1" \
  -frames:v 1 -vsync vfr out.webp

Scene detection produces visibly better thumbnails but costs 3–5x more CPU because it has to actually decode a window of frames. We run it as a background upgrade pass for videos that cross a view threshold, not on the hot path.

The extraction wrapper

Here's the core extraction function. The important details are the context timeout, the process group kill, and reading from stdout rather than a temp file.

package thumb

import (
    "bytes"
    "context"
    "fmt"
    "os/exec"
    "syscall"
    "time"
)

type Spec struct {
    Source   string        // file path or https URL
    Offset   time.Duration // seek target
    Width    int
    Height   int
    Quality  int    // 1-100
    Format   string // "webp" or "avif"
}

const extractTimeout = 20 * time.Second

func Extract(ctx context.Context, s Spec) ([]byte, error) {
    ctx, cancel := context.WithTimeout(ctx, extractTimeout)
    defer cancel()

    // Crop to the target aspect ratio from the centre, then scale.
    // force_original_aspect_ratio=increase guarantees we cover the box
    // before cropping, so we never letterbox.
    vf := fmt.Sprintf(
        "scale=%d:%d:force_original_aspect_ratio=increase,crop=%d:%d",
        s.Width, s.Height, s.Width, s.Height,
    )

    args := []string{
        "-hide_banner", "-loglevel", "error",
        "-ss", fmt.Sprintf("%.3f", s.Offset.Seconds()), // BEFORE -i: fast seek
        "-i", s.Source,
        "-vf", vf,
        "-frames:v", "1",
        "-q:v", fmt.Sprintf("%d", s.Quality),
        "-f", "image2",
        "-c:v", codecFor(s.Format),
        "pipe:1",
    }

    cmd := exec.CommandContext(ctx, "ffmpeg", args...)

    // FFmpeg spawns children; without a process group the CommandContext
    // kill leaves orphans holding file descriptors.
    cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
    cmd.Cancel = func() error {
        return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
    }

    var out, errBuf bytes.Buffer
    cmd.Stdout = &out
    cmd.Stderr = &errBuf

    if err := cmd.Run(); err != nil {
        if ctx.Err() == context.DeadlineExceeded {
            return nil, fmt.Errorf("ffmpeg timeout after %s: %s",
                extractTimeout, errBuf.String())
        }
        return nil, fmt.Errorf("ffmpeg: %w: %s", err, errBuf.String())
    }

    if out.Len() == 0 {
        return nil, fmt.Errorf("ffmpeg produced no output: %s", errBuf.String())
    }
    return out.Bytes(), nil
}

func codecFor(format string) string {
    switch format {
    case "avif":
        return "libaom-av1"
    default:
        return "libwebp"
    }
}

Three things worth calling out.

Setpgid plus a custom Cancel. exec.CommandContext by default sends SIGKILL to the direct child only. FFmpeg with certain protocol handlers spawns helpers; those survive and keep sockets open. Killing the whole process group with a negative PID fixes it. We had a slow leak of ~200 FDs/day before adding this.

Piping to stdout. Writing to a temp file and reading it back means two syscalls per byte and a cleanup path that can leak on panic. pipe:1 with -f image2 works for single-frame output and keeps everything in memory. Do bound it — a malformed input could theoretically produce a large buffer, so we also pass -fs 10M in production.

Crop before scale semantics. force_original_aspect_ratio=increase scales so the image covers the target box, then crop takes the centre. That guarantees no black bars and no distortion, at the cost of losing edges on very wide or very tall sources.

Bounding concurrency and deduplicating in-flight work

FFmpeg is CPU-bound. Running 200 concurrent processes on an 8-core box doesn't make anything faster; it makes everything slower and eventually triggers the OOM killer. Two mechanisms handle this.

First, a weighted semaphore sized to the core count. Second — and this matters more than people expect — request coalescing. When a popular video hits the homepage, you get 30 simultaneous requests for the same thumbnail within 200ms. Without dedup, that's 30 identical FFmpeg processes.

golang.org/x/sync/singleflight solves this in about four lines:

package thumb

import (
    "context"
    "crypto/sha256"
    "encoding/hex"
    "errors"
    "fmt"
    "os"
    "path/filepath"
    "runtime"

    "golang.org/x/sync/semaphore"
    "golang.org/x/sync/singleflight"
)

type Service struct {
    cacheDir string
    sem      *semaphore.Weighted
    group    singleflight.Group
}

func NewService(cacheDir string) *Service {
    // Leave one core for the HTTP handlers and GC.
    workers := runtime.NumCPU() - 1
    if workers < 1 {
        workers = 1
    }
    return &Service{
        cacheDir: cacheDir,
        sem:      semaphore.NewWeighted(int64(workers)),
    }
}

func cacheKey(s Spec) string {
    h := sha256.Sum256([]byte(fmt.Sprintf(
        "%s|%.3f|%d|%d|%d|%s",
        s.Source, s.Offset.Seconds(), s.Width, s.Height, s.Quality, s.Format,
    )))
    return hex.EncodeToString(h[:])
}

func (svc *Service) path(key, format string) string {
    // Two-level fanout: 65536 dirs keeps any single directory small enough
    // that ext4 lookups stay fast even at a few million files.
    return filepath.Join(svc.cacheDir, key[0:2], key[2:4], key+"."+format)
}

func (svc *Service) Get(ctx context.Context, s Spec) ([]byte, error) {
    key := cacheKey(s)
    p := svc.path(key, s.Format)

    if b, err := os.ReadFile(p); err == nil {
        return b, nil
    } else if !errors.Is(err, os.ErrNotExist) {
        return nil, fmt.Errorf("cache read: %w", err)
    }

    // All concurrent callers for the same key share one execution.
    v, err, _ := svc.group.Do(key, func() (any, error) {
        // Re-check: another goroutine may have written it between the
        // ReadFile above and acquiring the flight.
        if b, err := os.ReadFile(p); err == nil {
            return b, nil
        }

        if err := svc.sem.Acquire(ctx, 1); err != nil {
            return nil, fmt.Errorf("queue: %w", err)
        }
        defer svc.sem.Release(1)

        data, err := Extract(ctx, s)
        if err != nil {
            return nil, err
        }
        if err := writeAtomic(p, data); err != nil {
            // Cache write failure shouldn't fail the request.
            logWarn("cache write %s: %v", p, err)
        }
        return data, nil
    })
    if err != nil {
        return nil, err
    }
    return v.([]byte), nil
}

func writeAtomic(dst string, data []byte) error {
    if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
        return err
    }
    tmp, err := os.CreateTemp(filepath.Dir(dst), ".tmp-*")
    if err != nil {
        return err
    }
    defer os.Remove(tmp.Name())

    if _, err := tmp.Write(data); err != nil {
        tmp.Close()
        return err
    }
    if err := tmp.Close(); err != nil {
        return err
    }
    return os.Rename(tmp.Name(), dst)
}

The atomic write matters. Without it, a concurrent reader can os.ReadFile a half-written WebP and serve a truncated image, which browsers render as a broken icon and Cloudflare then caches for a year. rename(2) within the same filesystem is atomic, so readers see either the old file or the complete new one.

The two-level directory fanout (ab/cd/abcdef...webp) is not premature optimization. We're at roughly 2.8M cached thumbnails. A single flat directory at that size makes open() measurably slower on ext4 even with dir_index, and makes any ls in an incident basically impossible.

The HTTP layer and cache headers

The handler parses a URL of the form /thumb/{videoID}/{width}x{height}.{format}, validates the dimensions against an allowlist, and serves.

Dimension allowlisting is a real security control, not paperwork. If any WxH is accepted, an attacker enumerates /thumb/abc/1x1.webp through /thumb/abc/2000x2000.webp and fills your disk with 4M unique files while pinning every CPU. We allow exactly five sizes, matching our srcset breakpoints.

var allowedSizes = map[string]bool{
    "320x180": true, "480x270": true, "640x360": true,
    "960x540": true, "1280x720": true,
}

var thumbPath = regexp.MustCompile(
    `^/thumb/([A-Za-z0-9_-]{1,32})/(\d{2,4}x\d{2,4})\.(webp|avif|jpg)$`)

func (svc *Service) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    m := thumbPath.FindStringSubmatch(r.URL.Path)
    if m == nil {
        http.NotFound(w, r)
        return
    }
    videoID, size, format := m[1], m[2], m[3]

    if !allowedSizes[size] {
        http.Error(w, "unsupported size", http.StatusBadRequest)
        return
    }
    var width, height int
    fmt.Sscanf(size, "%dx%d", &width, &height)

    src, duration, err := svc.lookupSource(r.Context(), videoID)
    if errors.Is(err, sql.ErrNoRows) {
        http.NotFound(w, r)
        return
    } else if err != nil {
        http.Error(w, "lookup failed", http.StatusBadGateway)
        return
    }

    data, err := svc.Get(r.Context(), Spec{
        Source:  src,
        Offset:  time.Duration(float64(duration) * 0.20),
        Width:   width,
        Height:  height,
        Quality: 82,
        Format:  format,
    })
    if err != nil {
        // Serve a generic placeholder rather than a broken image,
        // with a short TTL so we retry soon.
        w.Header().Set("Cache-Control", "public, max-age=60")
        svc.servePlaceholder(w, width, height)
        return
    }

    etag := `"` + cacheKey(Spec{Source: src, Width: width, Height: height})[:16] + `"`
    if r.Header.Get("If-None-Match") == etag {
        w.WriteHeader(http.StatusNotModified)
        return
    }

    w.Header().Set("Content-Type", "image/"+format)
    w.Header().Set("ETag", etag)
    w.Header().Set("Cache-Control", "public, max-age=31536000, immutable")
    w.Header().Set("Content-Length", strconv.Itoa(len(data)))
    w.Write(data)
}

The immutable directive is safe here because the URL encodes everything that determines the bytes. If we change the crop strategy, we bump a version segment in the path and the old URLs simply stop being referenced.

The placeholder-on-error path deserves emphasis. Early on we returned 500 on extraction failure. Cloudflare doesn't cache 500s, so every retry hit origin, and a single unreachable source video generated sustained FFmpeg attempts until the deploy that fixed it. Returning a 200 placeholder with max-age=60 turns a hot loop into one request per minute per edge POP.

Wiring it into the PHP app

The PHP side needs to know whether a thumbnail exists so it can decide between a real <img> and a placeholder with lazy loading. We track it in SQLite:

<?php
declare(strict_types=1);

final class ThumbnailUrl
{
    private const SIZES = [320, 480, 640, 960, 1280];
    private const VERSION = 'v2';

    public function __construct(private readonly PDO $db) {}

    /** @return array{src: string, srcset: string, width: int, height: int} */
    public function forVideo(string $videoId, int $preferred = 640): array
    {
        $srcset = [];
        foreach (self::SIZES as $w) {
            $h = (int) round($w * 9 / 16);
            $srcset[] = sprintf('/thumb/%s/%dx%d.webp %dw',
                rawurlencode($videoId), $w, $h, $w);
        }

        return [
            'src'    => sprintf('/thumb/%s/%dx%d.webp',
                            rawurlencode($videoId), $preferred,
                            (int) round($preferred * 9 / 16)),
            'srcset' => implode(', ', $srcset),
            'width'  => $preferred,
            'height' => (int) round($preferred * 9 / 16),
        ];
    }

    /** Videos surfaced by search but never rendered yet — warm them. */
    public function queueWarmup(array $videoIds): void
    {
        if ($videoIds === []) {
            return;
        }
        $ph = implode(',', array_fill(0, count($videoIds), '?'));
        $stmt = $this->db->prepare(
            "UPDATE videos SET thumb_warm_requested_at = unixepoch()
             WHERE id IN ($ph) AND thumb_generated_at IS NULL
               AND (thumb_warm_requested_at IS NULL
                    OR thumb_warm_requested_at < unixepoch() - 3600)"
        );
        $stmt->execute($videoIds);
    }
}

A cron worker reads rows with thumb_warm_requested_at set and issues HEAD requests against the Go service at a fixed rate. This means the first user to search a term pays the generation cost only for the images actually above the fold; everything below gets warmed within a minute.

The unixepoch() - 3600 guard prevents a video whose source is permanently dead from being re-queued forever.

Numbers and operational notes

After three months on a 4-core VPS handling roughly 340k thumbnail requests/day:

• Cache hit rate at Cloudflare: 97.2%. Origin sees ~9.5k requests/day.
• Disk cache hit (the 2.8% that reach origin): 94%, so actual FFmpeg invocations are ~570/day.
• p50 generation: 210ms for a 640x360 WebP from a local file; ~1.4s from a remote HTTPS source, dominated by network.
• p99 generation: 3.1s. The tail is long videos with sparse keyframes.
• Disk: 2.8M files, 41GB. An LRU sweep evicts anything untouched for 90 days.
• LCP improvement: 4.1s → 1.6s on mobile, which was the entire point.

A few things I'd tell my past self:

• AVIF encoding is 10–20x slower than WebP with libaom at default settings. We generate WebP synchronously and AVIF in a background pass, serving AVIF via <picture> only once it exists.
• Set -fs and a hard timeout on every FFmpeg call. A truncated MP4 with a corrupt index will happily spin forever.
• Log FFmpeg's stderr on failure, always. exit status 1 tells you nothing; the stderr line tells you exactly which demuxer gave up.
• atime is usually disabled (relatime/noatime), so LRU eviction based on access time silently degrades to mtime. We track access in a small SQLite table instead.
• Don't put the cache directory on the same filesystem as SQLite if you can avoid it. A full disk from thumbnail growth will take your database write path down with it.

Conclusion

The service is about 600 lines of Go. Most of the value isn't in the FFmpeg invocation — it's in the boring parts: bounded concurrency so the box doesn't melt, singleflight so a viral video doesn't spawn 30 identical processes, atomic writes so nobody serves a half-encoded image, and cached error responses so a dead source doesn't become a retry storm.

If you're considering this, the order that worked for us was: get correct extraction first with -ss before -i, then add the disk cache, then add dedup, then tune the frame-selection heuristic. Each step was independently shippable, and the first two got us 90% of the latency win.