plugin/shed: add UDP overload protection plugin (#8312)

* plugin/shed: add UDP overload protection plugin

UDP responses written back through one listener socket serialize on the
Go runtime's internal fdMutex, which allows at most 2^20-1 concurrent
operations per file descriptor and panics the process when exceeded.
CoreDNS serves UDP with one goroutine per query, all writing through the
shared packet connection, so a sustained overload parks every excess
in-flight query in that wait queue until the process dies with
"too many concurrent operations on a single file or socket". Observed
in production: ~2.8M goroutines and 60GiB RSS before the panic.

The shed plugin makes the panic structurally unreachable. It installs,
via Config.UDPDecorateWriterFunc, a per-socket bounded evict-oldest
stack drained newest-first by a single writer goroutine, so the fd
never sees more than one writer and residual capacity under overload
always goes to the freshest response. While a socket's stack is full,
arriving queries are dropped before any plugin runs. Drops are silent
(the client's resolver retries elsewhere) and counted in
coredns_shed_dropped_total{server, reason}.

plugin/shed/fdmutex_test.go demonstrates the failure and the fix with
one shared flood harness. Two subprocess tests reproduce the exact
runtime panic without the plugin's write discipline - one deterministic
(a held write plus >2^20 queued writers), one with nothing held or
mocked; both exercise the Go runtime rather than the plugin, so they
are gated behind SHED_FLOOD_TEST=1. The counterfactual - the same load
through the plugin's stack, completing with every response accounted
for as written or dropped - runs in every test invocation, including
-race, at 50k responders, and at the full 1.5M with SHED_FLOOD_TEST=1:

    SHED_FLOOD_TEST=1 go test ./plugin/shed/

Signed-off-by: Ryan Brewster <rpb@anthropic.com>

* test: add shed e2e test

Query a shed-enabled server over UDP (the plugin's deferred
single-writer path) and TCP (which shed passes through), and check
that coredns_shed_dropped_total is exported with its reason label.

No-Verification-Needed: test-only change
Signed-off-by: Ryan Brewster <rpb@anthropic.com>

---------

Signed-off-by: Ryan Brewster <rpb@anthropic.com>
This commit is contained in:
rpb-ant
2026-07-27 05:13:25 -04:00
committed by GitHub
parent 989bf4a9fd
commit 76056dd2e5
13 changed files with 1166 additions and 0 deletions

53
plugin/shed/setup_test.go Normal file
View File

@@ -0,0 +1,53 @@
package shed
import (
"strings"
"testing"
"github.com/coredns/caddy"
)
func TestSetup(t *testing.T) {
tests := []struct {
input string
shouldErr bool
}{
{"shed", false},
{"shed extra", true},
{"shed {\n depth 10\n}", true},
{"shed\nshed", true},
}
for i, tc := range tests {
c := caddy.NewTestController("dns", tc.input)
err := setup(c)
if tc.shouldErr && err == nil {
t.Errorf("Test %d: expected error for input %q", i, tc.input)
}
if !tc.shouldErr && err != nil {
t.Errorf("Test %d: unexpected error for input %q: %s", i, tc.input, err)
}
}
}
func TestSetupRejectsNonDNSTransport(t *testing.T) {
for _, key := range []string{"tls://.:853", "grpc://.:443", "https://.:443", "quic://.:853"} {
c := caddy.NewTestController("dns", "shed")
c.ServerBlockKeys = []string{key}
err := setup(c)
if err == nil {
t.Errorf("expected error for server block key %q", key)
continue
}
if !strings.Contains(err.Error(), "plain DNS") {
t.Errorf("error for %q = %q, want it to mention plain DNS", key, err)
}
}
}
func TestSetupAcceptsPlainDNSKeys(t *testing.T) {
c := caddy.NewTestController("dns", "shed")
c.ServerBlockKeys = []string{"example.org.:53", "dns://.:53"}
if err := setup(c); err != nil {
t.Fatalf("unexpected error: %s", err)
}
}