mirror of
https://github.com/coredns/coredns.git
synced 2026-08-20 23:08:28 -04:00
76056dd2e5
* plugin/shed: add UDP overload protection plugin
UDP responses written back through one listener socket serialize on the
Go runtime's internal fdMutex, which allows at most 2^20-1 concurrent
operations per file descriptor and panics the process when exceeded.
CoreDNS serves UDP with one goroutine per query, all writing through the
shared packet connection, so a sustained overload parks every excess
in-flight query in that wait queue until the process dies with
"too many concurrent operations on a single file or socket". Observed
in production: ~2.8M goroutines and 60GiB RSS before the panic.
The shed plugin makes the panic structurally unreachable. It installs,
via Config.UDPDecorateWriterFunc, a per-socket bounded evict-oldest
stack drained newest-first by a single writer goroutine, so the fd
never sees more than one writer and residual capacity under overload
always goes to the freshest response. While a socket's stack is full,
arriving queries are dropped before any plugin runs. Drops are silent
(the client's resolver retries elsewhere) and counted in
coredns_shed_dropped_total{server, reason}.
plugin/shed/fdmutex_test.go demonstrates the failure and the fix with
one shared flood harness. Two subprocess tests reproduce the exact
runtime panic without the plugin's write discipline - one deterministic
(a held write plus >2^20 queued writers), one with nothing held or
mocked; both exercise the Go runtime rather than the plugin, so they
are gated behind SHED_FLOOD_TEST=1. The counterfactual - the same load
through the plugin's stack, completing with every response accounted
for as written or dropped - runs in every test invocation, including
-race, at 50k responders, and at the full 1.5M with SHED_FLOOD_TEST=1:
SHED_FLOOD_TEST=1 go test ./plugin/shed/
Signed-off-by: Ryan Brewster <rpb@anthropic.com>
* test: add shed e2e test
Query a shed-enabled server over UDP (the plugin's deferred
single-writer path) and TCP (which shed passes through), and check
that coredns_shed_dropped_total is exported with its reason label.
No-Verification-Needed: test-only change
Signed-off-by: Ryan Brewster <rpb@anthropic.com>
---------
Signed-off-by: Ryan Brewster <rpb@anthropic.com>
74 lines
1022 B
Go
74 lines
1022 B
Go
// generated by directives_generate.go; DO NOT EDIT
|
|
|
|
package dnsserver
|
|
|
|
// Directives are registered in the order they should be
|
|
// executed.
|
|
//
|
|
// Ordering is VERY important. Every plugin will
|
|
// feel the effects of all other plugin below
|
|
// (after) them during a request, but they must not
|
|
// care what plugin above them are doing.
|
|
var Directives = []string{
|
|
"root",
|
|
"metadata",
|
|
"geoip",
|
|
"cancel",
|
|
"tls",
|
|
"proxyproto",
|
|
"quic",
|
|
"grpc_server",
|
|
"https",
|
|
"https3",
|
|
"timeouts",
|
|
"multisocket",
|
|
"reload",
|
|
"nsid",
|
|
"bufsize",
|
|
"bind",
|
|
"debug",
|
|
"trace",
|
|
"ready",
|
|
"health",
|
|
"pprof",
|
|
"shed",
|
|
"prometheus",
|
|
"errors",
|
|
"log",
|
|
"dnstap",
|
|
"local",
|
|
"dns64",
|
|
"any",
|
|
"chaos",
|
|
"loadbalance",
|
|
"tsig",
|
|
"rewrite",
|
|
"autopath",
|
|
"acl",
|
|
"cache",
|
|
"header",
|
|
"dnssec",
|
|
"minimal",
|
|
"template",
|
|
"transfer",
|
|
"hosts",
|
|
"route53",
|
|
"azure",
|
|
"clouddns",
|
|
"k8s_external",
|
|
"kubernetes",
|
|
"file",
|
|
"auto",
|
|
"secondary",
|
|
"etcd",
|
|
"loop",
|
|
"forward",
|
|
"grpc",
|
|
"erratic",
|
|
"whoami",
|
|
"on",
|
|
"sign",
|
|
"view",
|
|
"nomad",
|
|
}
|