mirror of
https://github.com/coredns/coredns.git
synced 2026-08-20 23:08:28 -04:00
76056dd2e5
* plugin/shed: add UDP overload protection plugin
UDP responses written back through one listener socket serialize on the
Go runtime's internal fdMutex, which allows at most 2^20-1 concurrent
operations per file descriptor and panics the process when exceeded.
CoreDNS serves UDP with one goroutine per query, all writing through the
shared packet connection, so a sustained overload parks every excess
in-flight query in that wait queue until the process dies with
"too many concurrent operations on a single file or socket". Observed
in production: ~2.8M goroutines and 60GiB RSS before the panic.
The shed plugin makes the panic structurally unreachable. It installs,
via Config.UDPDecorateWriterFunc, a per-socket bounded evict-oldest
stack drained newest-first by a single writer goroutine, so the fd
never sees more than one writer and residual capacity under overload
always goes to the freshest response. While a socket's stack is full,
arriving queries are dropped before any plugin runs. Drops are silent
(the client's resolver retries elsewhere) and counted in
coredns_shed_dropped_total{server, reason}.
plugin/shed/fdmutex_test.go demonstrates the failure and the fix with
one shared flood harness. Two subprocess tests reproduce the exact
runtime panic without the plugin's write discipline - one deterministic
(a held write plus >2^20 queued writers), one with nothing held or
mocked; both exercise the Go runtime rather than the plugin, so they
are gated behind SHED_FLOOD_TEST=1. The counterfactual - the same load
through the plugin's stack, completing with every response accounted
for as written or dropped - runs in every test invocation, including
-race, at 50k responders, and at the full 1.5M with SHED_FLOOD_TEST=1:
SHED_FLOOD_TEST=1 go test ./plugin/shed/
Signed-off-by: Ryan Brewster <rpb@anthropic.com>
* test: add shed e2e test
Query a shed-enabled server over UDP (the plugin's deferred
single-writer path) and TCP (which shed passes through), and check
that coredns_shed_dropped_total is exported with its reason label.
No-Verification-Needed: test-only change
Signed-off-by: Ryan Brewster <rpb@anthropic.com>
---------
Signed-off-by: Ryan Brewster <rpb@anthropic.com>
80 lines
3.4 KiB
Markdown
80 lines
3.4 KiB
Markdown
# shed
|
|
|
|
## Name
|
|
|
|
*shed* - serializes UDP response writes per listener socket and sheds load when the socket cannot keep up.
|
|
|
|
## Description
|
|
|
|
UDP responses written back through one listener socket serialize on the Go runtime's internal
|
|
fdMutex, which allows at most 2^20-1 concurrent operations (holders plus waiters) per file
|
|
descriptor and terminates the process with
|
|
|
|
~~~ txt
|
|
panic: too many concurrent operations on a single file or socket (max 1048575)
|
|
~~~
|
|
|
|
when that is exceeded. CoreDNS serves UDP with one goroutine per query, all writing back through
|
|
the shared packet connection, so when queries arrive faster than the socket's serialized writes
|
|
drain, every excess in-flight query parks its goroutine in that wait queue and nothing bounds the
|
|
pile. Observed in production: ~2.8M goroutines and 60GiB RSS before the panic.
|
|
|
|
The *shed* plugin makes that panic structurally unreachable, per UDP listener socket:
|
|
|
|
* **Single writer** - responses are not written by the handler goroutine. The packed response is
|
|
pushed onto a bounded per-socket stack (fixed depth 1024) and one writer goroutine per socket
|
|
performs the wire writes, so the file descriptor never sees more than one writer. The stack
|
|
evicts the oldest entry when full and the writer pops the newest first, so under overload the
|
|
socket's residual capacity always goes to the freshest response. The depth is a fixed burst
|
|
budget (roughly 12-16ms of a typical socket's drain rate), not a tunable.
|
|
* **Coupled shedding** - while a socket's stack is full, arriving queries on that socket are
|
|
dropped before any plugin runs; work admitted then would only produce a response destined for
|
|
eviction. There is no configuration: the stack's fullness is the signal.
|
|
|
|
Drops are silent - no response is written, so the client's resolver retries against another
|
|
server, the standard load-shedding contract for UDP DNS. Every drop is counted.
|
|
|
|
The plugin only acts on UDP; TCP queries pass through untouched. It can only be used in plain DNS
|
|
server blocks (not *tls*, *grpc*, *https* or *quic*), which is enforced at startup. It should be
|
|
listed before (above) the *prometheus* plugin in the plugin chain, so that shed drops are never
|
|
counted as handled requests by the *prometheus* plugin - which is where this plugin sits by
|
|
default.
|
|
|
|
When several server blocks share a listener, any block with *shed* installs the write discipline
|
|
for every write on that socket, while the pre-chain shedding only runs in blocks that carry the
|
|
directive - keep it uniform across blocks sharing a listener. The discipline covers every response
|
|
written through `WriteMsg`, which is how every plugin responds; a plugin writing raw bytes with
|
|
`ResponseWriter.Write` would bypass it.
|
|
|
|
## Syntax
|
|
|
|
~~~ txt
|
|
shed
|
|
~~~
|
|
|
|
The plugin takes no arguments.
|
|
|
|
## Metrics
|
|
|
|
If monitoring is enabled (via the *prometheus* plugin) then the following metric is exported:
|
|
|
|
* `coredns_shed_dropped_total{server, reason}` - counter of dropped queries and responses. The
|
|
`reason` label is `query` for queries dropped before the plugin chain because the socket's
|
|
stack was full, and `response` for responses dropped at the write boundary (evicted by a newer
|
|
response, failed to reach the wire, or arriving during shutdown).
|
|
|
|
## Examples
|
|
|
|
Protect the UDP listener while forwarding:
|
|
|
|
~~~ corefile
|
|
. {
|
|
shed
|
|
forward . 8.8.8.8
|
|
}
|
|
~~~
|
|
|
|
## See Also
|
|
|
|
The fdMutex limit is enforced in `GOROOT/src/internal/poll/fd_mutex.go`.
|