TL;DR: Bot traffic isn't a fact of life - it's a leak in your infrastructure budget. A two-layer defense using robots.txt and Next.js middleware blocks 60-70% of resource-draining scrapers while keeping the bots you actually want. This approach runs at the edge, costs nothing to implement, and protects both your hosting bill and your analytics accuracy.

Most founders assume bot traffic is part of the cost of running a website. That assumption is expensive. Bots hammer your endpoints, consume metered server resources, pollute your analytics, and inflate your hosting bill. If you're on Vercel, AWS, or any pay-per-request platform, this traffic literally costs money every time it hits your server.

This is what a bot attack looks like - a sudden, unnatural spike that distorts your real user activity.

The standard advice is to add a robots.txt file and hope for the best. That only works if bots respect it. Most scrapers don't. They read your file, note what you've marked off-limits, and scrape it anyway. The file is a suggestion, not a firewall.

The real solution is a two-layer defense: a smart robots.txt configuration that filters well-behaved bots, and Next.js middleware running at the edge that enforces the rules before traffic reaches your application. Combined, they block 60-70% of unwanted bot traffic while keeping legitimate crawlers and AI assistants you actually want.

This isn't about locking out AI crawlers. It's about controlling who gets access and at what cost.

The Problem: Three Kinds of Visitors

Your website serves three categories of traffic. Real humans browsing your site. Legitimate bots that index your content for search engines or train helpful AI models. And scrapers that hammer your endpoints, steal your data, rotate IP addresses, and pretend to be browsers.

The third category is the problem. These bots ignore polite requests. They consume server resources you're paying for. They pollute your analytics by inflating visitor counts with fake traffic, making conversion rates appear worse than they are. They probe for vulnerabilities. If you're running on metered hosting, every request from a scraper costs you a fraction of a cent. When scrapers make up 30-40% of your traffic, that adds up fast.

60-70% of bot traffic can be filtered by a properly configured robots.txt, but only because most low-effort scrapers follow basic rules to avoid detection.

The remaining 30-40% requires enforcement. That's where middleware comes in. It runs before every request hits your application, inspects headers and user agents, and blocks anything suspicious. On Vercel, middleware runs at the edge, meaning blocked requests never consume your server resources. You're not paying to process traffic you don't want.

Layer 1: Smart robots.txt Configuration

Start with robots.txt. This file sits at the root of your domain and tells compliant bots what they can access. The key is being explicit about who you allow and who you block. Most sites either leave this file empty or use a generic "allow all" rule. That's leaving money on the table.

Here's a structure that works:

``txt

AI training & assistants - welcome

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

Search engines - welcome

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

Known scrapers - blocked

User-agent: SemrushBot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: PetalBot
Disallow: /

Default - be polite or leave

User-agent: *
Crawl-delay: 10
Disallow: /api/
``

This makes three categories clear. AI assistants and search engines get full access. Known scrapers get nothing. Everything else gets throttled with a 10-second crawl delay and blocked from API routes.

The crawl delay is critical. Aggressive bots hitting your site every few seconds will slow down or leave. Legitimate crawlers won't care about a 10-second pause between requests. This single line filters out bots that hammer your site hundreds of times per minute.

The Disallow: /api/ rule is equally important. API routes often trigger database queries, authentication checks, or external service calls. Bots crawling these endpoints waste expensive compute. Block them by default unless you're running a public API that needs discovery.

This configuration takes five minutes to set up. Copy the template, adjust the bot names if you have specific concerns, and deploy it to your site's root directory. For a deeper dive into robots.txt strategy and common mistakes, check out VisionVix's robots.txt best practices guide, which walks through edge cases and SEO implications.

Most low-effort scrapers follow basic robots.txt rules to avoid detection, making it effective against the majority of bad actors.

This handles the 60-70% of bot traffic that follows rules. It won't stop determined attackers, but it filters the lazy majority. Now we enforce the rest.

Layer 2: Middleware Enforcement at the Edge

robots.txt is polite. Middleware is the bouncer. It runs before every request hits your application, inspects the user agent and headers, and blocks anything suspicious. If you're using Next.js on Vercel, middleware runs at the edge. Requests get filtered before they consume your server resources. You're not paying to process blocked traffic.

Create a file called middleware.ts at the root of your Next.js project:

``typescript
import { NextResponse } from 'next/server'

const ALLOWED_BOTS = [
'googlebot',
'bingbot',
'gptbot',
'claudebot',
'perplexitybot',
'slackbot',
'twitterbot'
]

const BLOCKED_BOTS = [
'petalbot',
'blexbot',
'semrushbot',
'ahrefsbot',
'mj12bot',
'dotbot'
]

const SCRAPER_LIBS = [
'scrapy',
'python-requests',
'curl/',
'wget',
'java/',
'go-http-client'
]

export function middleware(req) {
const ua = (req.headers.get('user-agent') || '').toLowerCase()

// 1. Allow good bots
if (ALLOWED_BOTS.some(bot => ua.includes(bot))) {
return NextResponse.next()
}

// 2. Block empty user agents
if (!req.headers.get('user-agent')) {
return new NextResponse('Forbidden', { status: 403 })
}

// 3. Block aggressive crawlers
if (BLOCKED_BOTS.some(bot => ua.includes(bot))) {
return new NextResponse('Forbidden', { status: 403 })
}

// 4. Block scraper libraries
if (SCRAPER_LIBS.some(lib => ua.includes(lib))) {
return new NextResponse('Forbidden', { status: 403 })
}

// 5. Block missing Accept header
if (!req.headers.get('accept')) {
return new NextResponse('Forbidden', { status: 403 })
}

// Default: pass
return NextResponse.next()
}
``

Every request triggers this function. The middleware reads the user agent string and headers. If the user agent matches a known good bot, the request passes immediately. If it matches a blocked bot or scraper library, it gets a 403 response. If critical headers are missing, it's blocked.

The Accept header check is subtle but effective. Real browsers and legitimate bots send an Accept header telling the server what content types they understand. Scrapers often skip this. Blocking requests without an Accept header catches a lot of low-effort scraping attempts.

Middleware runs at the Vercel edge, meaning blocked requests never hit your application server or consume compute resources. This is what reduces the remaining 30-40% of bot traffic that ignored robots.txt.

What Gets Blocked and Why

Let's walk through what each check catches in practice.

Empty User Agents

typescript
if (!req.headers.get('user-agent')) {
return new NextResponse('Forbidden', { status: 403 })
}


Legitimate browsers always send a user agent. Tools like curl and wget do too, but scrapers built with low-effort scripts often don't. This single check blocks a surprising amount of junk traffic.

Scraper Libraries

typescript
if (SCRAPER_LIBS.some(lib => ua.includes(lib))) {
return new NextResponse('Forbidden', { status: 403 })
}


This targets common scraping tools. A user agent string like python-requests/2.28.0 or Scrapy/2.9.0 reveals exactly what's hitting your site. These are almost never legitimate traffic. Block them immediately.

Missing Accept Header

typescript
if (!req.headers.get('accept')) {
return new NextResponse('Forbidden', { status: 403 })
}


Real browsers send an Accept header like text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8. Bots that don't care about the response format skip this header. It's a strong signal that the request isn't coming from a browser or a well-behaved crawler.

The Conditional Middleware Pattern

Not every route needs the same level of protection. Public pages like your homepage or blog should allow all legitimate traffic. Admin panels, API routes, or expensive database queries need stricter rules.

You can add conditional logic to your middleware:

``typescript
export function middleware(req) {
const pathname = req.nextUrl.pathname

// Skip middleware for static assets
if (pathname.startsWith('/_next') || pathname.startsWith('/static')) {
return NextResponse.next()
}

// Apply strict rules to API routes
if (pathname.startsWith('/api')) {
return strictBotCheck(req)
}

// Apply normal rules to everything else
return normalBotCheck(req)
}
``

This pattern lets you enforce stricter checks on expensive routes while keeping public pages accessible. The strictBotCheck function could block all bots except explicitly allowed ones. The normalBotCheck function could allow most traffic but throttle suspicious patterns.

Honeypot Traps and Advanced Tactics

If you want to go further, add a honeypot trap. Create a page that's linked in your robots.txt under a Disallow rule. Good bots won't visit it. Bad bots will.

Add this to your robots.txt:

txt
User-agent: *
Disallow: /trap


Then create a /trap route in your Next.js app. Anyone who visits it gets their IP logged and blocked:

`typescript
// pages/api/trap.ts
export default function handler(req, res) {
const ip = req.headers['x-forwarded-for'] || req.connection.remoteAddress
console.log(Bot trap triggered by IP: ${ip}`)

// Add IP to blocklist
// (integrate with your edge config or database)

return res.status(403).json({ error: 'Forbidden' })
}
``

This catches bots that explicitly ignore your robots.txt rules. You can feed these IPs into your middleware blocklist or configure edge rules at the CDN level. For a broader look at proactive security measures like this, see the website security scanning guide, which covers automated vulnerability detection and bot behavior analysis.

What You're Actually Protecting

This defense isn't just about saving money. It protects three things founders underestimate:

Hosting costs. If bots make up 30-40% of your traffic, you're paying 30-40% more than you should. On metered platforms like Vercel or AWS Lambda, every function invocation costs money. Blocking bot traffic before it hits your application cuts those costs immediately.

Analytics accuracy. Bot traffic inflates your visitor counts and skews conversion rates. If 40% of your "users" are scrapers, your real conversion rate is higher than your dashboard shows. Clean traffic gives you accurate data to make decisions.

Attack surface. Bots probe for vulnerabilities. They test SQL injection patterns, brute-force login endpoints, and enumerate API routes. Blocking them reduces your exposure to automated attacks.

Blocking requests without Accept headers or empty user agents catches most low-effort scraping attempts.

Implementation Checklist

Here's what to do right now:

1. Deploy the robots.txt template to your site's root directory. Adjust the allowed and blocked bot lists based on your traffic logs.

2. Add the middleware file to your Next.js project. Copy the full code block above, save it as middleware.ts at the project root, and deploy.

3. Monitor your logs for a week. Check which user agents are getting blocked. If you see legitimate traffic being filtered, adjust your ALLOWED_BOTS list.

4. Review your hosting bill after 30 days. If bots were a significant portion of your traffic, you should see a measurable drop in compute costs.

5. Check your analytics for cleaner data. Visitor counts should drop, but conversion rates should improve because you're measuring real users instead of bots.

This setup takes less than an hour to implement. It costs nothing. It runs at the edge, so there's no performance penalty for legitimate users. And it blocks the majority of bot traffic that's wasting your budget and polluting your metrics.

If you've been treating bot traffic as inevitable, stop. It's not. Two layers of defense, five minutes of configuration, and you're done. The bots you want still get in. The ones you don't pay for anymore.

---

📦 Publishing Kit — Dev.to

Title Options (5)

Selected: Stop Paying for Bot Traffic: A Two-Layer Defense for Next.js Apps

Alternates:

1. How robots.txt + Next.js Middleware Cuts Bot Costs by 60-70%
2. Block Scrapers, Keep AI Crawlers: Smart Bot Management for Next.js
3. The $500/Month Bot Problem (And How to Fix It With Edge Middleware)
4. robots.txt Isn't Enough: Building a Real Bot Firewall in Next.js

Slug

stop-paying-for-bot-traffic-nextjs-middleware-defense

Tags

nextjs, webdev, devops, security