← Back to Learning

Kubernetes Health Probes Failing With 400 on a Host-Validating App

A dashboard pod in my cluster was stuck in CrashLoopBackOff, 390+ restarts and climbing. The usual first instinct with a crash loop is "the app is broken." It wasn't. The app booted cleanly every single time. Kubernetes was killing a healthy container over and over because its own health check could never pass.

The symptom

Both the liveness and readiness probes were failing, which is normally a strong signal the app itself is unresponsive or erroring. But the container logs told a different story:

▲ Next.js 16.3.0
- Local:         http://[::1]:3000
✓ Ready in 0ms
[error]: Host validation failed for: 10.244.1.154:3000. Hint: Set the
HOMEPAGE_ALLOWED_HOSTS environment variable to allow requests from
this host / port.

"Ready in 0ms." The app was up. Every request against it was being rejected before it ever reached application logic, and it was telling me exactly why.

The actual cause

A recent image update had pulled in a version built on Next.js 16, which validates the incoming HTTP Host header against an allow-list before serving anything, and rejects a mismatch with 400. My allow-list already had the app's real public hostname on it, and that part was fine.

The problem was the probe itself. Kubernetes' default httpGet probe doesn't send your real hostname as the Host header. By default, kubelet targets the pod directly by its IP: <podIP>:<port>, and that's exactly what shows up as the Host header on the request. Pod IPs are dynamic and reassigned on every reschedule, so there's no way to pre-authorize them on an allow-list, and every probe was rejected the instant it hit the app.

The fix

Two changes, both in the pod spec:

  1. Add a fixed, always-valid Host header to both probes via httpHeaders, instead of letting kubelet default to the pod IP.
  2. Add that same value to the app's allow-list so the probe's request actually passes validation.
livenessProbe:
  httpGet:
    path: /
    port: 3000
    httpHeaders:
      - name: Host
        value: localhost
  initialDelaySeconds: 15
  periodSeconds: 15
readinessProbe:
  httpGet:
    path: /
    port: 3000
    httpHeaders:
      - name: Host
        value: localhost
  initialDelaySeconds: 10
  periodSeconds: 10
env:
  - name: HOMEPAGE_ALLOWED_HOSTS
    value: "home.example.com,localhost"

Real browser and reverse-proxy traffic still validates against the real hostname, exactly as before. Only the probe's own request now carries a Host header that's guaranteed to always be valid, regardless of which pod IP it happens to land on.

The generalizable lesson

This isn't specific to Next.js or to this one app. Any framework that validates the Host header for security reasons will hit this exact same failure mode behind a Kubernetes probe that doesn't override it: Django's ALLOWED_HOSTS, Rails' host authorization, and a handful of others all do some version of the same check.

If a pod is crash-looping and the logs show the app starting successfully every time, check whether the app validates the Host header before assuming the app itself is broken. A probe that never sends a valid Host is indistinguishable from a real outage until you actually read the logs.