mirror of
https://github.com/coredns/coredns.git
synced 2026-08-20 23:08:28 -04:00
76056dd2e5
* plugin/shed: add UDP overload protection plugin
UDP responses written back through one listener socket serialize on the
Go runtime's internal fdMutex, which allows at most 2^20-1 concurrent
operations per file descriptor and panics the process when exceeded.
CoreDNS serves UDP with one goroutine per query, all writing through the
shared packet connection, so a sustained overload parks every excess
in-flight query in that wait queue until the process dies with
"too many concurrent operations on a single file or socket". Observed
in production: ~2.8M goroutines and 60GiB RSS before the panic.
The shed plugin makes the panic structurally unreachable. It installs,
via Config.UDPDecorateWriterFunc, a per-socket bounded evict-oldest
stack drained newest-first by a single writer goroutine, so the fd
never sees more than one writer and residual capacity under overload
always goes to the freshest response. While a socket's stack is full,
arriving queries are dropped before any plugin runs. Drops are silent
(the client's resolver retries elsewhere) and counted in
coredns_shed_dropped_total{server, reason}.
plugin/shed/fdmutex_test.go demonstrates the failure and the fix with
one shared flood harness. Two subprocess tests reproduce the exact
runtime panic without the plugin's write discipline - one deterministic
(a held write plus >2^20 queued writers), one with nothing held or
mocked; both exercise the Go runtime rather than the plugin, so they
are gated behind SHED_FLOOD_TEST=1. The counterfactual - the same load
through the plugin's stack, completing with every response accounted
for as written or dropped - runs in every test invocation, including
-race, at 50k responders, and at the full 1.5M with SHED_FLOOD_TEST=1:
SHED_FLOOD_TEST=1 go test ./plugin/shed/
Signed-off-by: Ryan Brewster <rpb@anthropic.com>
* test: add shed e2e test
Query a shed-enabled server over UDP (the plugin's deferred
single-writer path) and TCP (which shed passes through), and check
that coredns_shed_dropped_total is exported with its reason label.
No-Verification-Needed: test-only change
Signed-off-by: Ryan Brewster <rpb@anthropic.com>
---------
Signed-off-by: Ryan Brewster <rpb@anthropic.com>
82 lines
1.4 KiB
INI
82 lines
1.4 KiB
INI
# Directives are registered in the order they should be executed.
|
|
#
|
|
# Ordering is VERY important. Every plugin will feel the effects of all other
|
|
# plugin below (after) them during a request, but they must not care what plugin
|
|
# above them are doing.
|
|
|
|
# How to rebuild with updated plugin configurations: Modify the list below and
|
|
# run `go generate && go build`
|
|
|
|
# The parser takes the input format of:
|
|
#
|
|
# <plugin-name>:<package-name>
|
|
# Or
|
|
# <plugin-name>:<fully-qualified-package-name>
|
|
#
|
|
# External plugin example:
|
|
#
|
|
# log:github.com/coredns/coredns/plugin/log
|
|
# Local plugin example:
|
|
# log:log
|
|
|
|
root:root
|
|
metadata:metadata
|
|
geoip:geoip
|
|
cancel:cancel
|
|
tls:tls
|
|
proxyproto:proxyproto
|
|
quic:quic
|
|
grpc_server:grpc_server
|
|
https:https
|
|
https3:https3
|
|
timeouts:timeouts
|
|
multisocket:multisocket
|
|
reload:reload
|
|
nsid:nsid
|
|
bufsize:bufsize
|
|
bind:bind
|
|
debug:debug
|
|
trace:trace
|
|
ready:ready
|
|
health:health
|
|
pprof:pprof
|
|
shed:shed
|
|
prometheus:metrics
|
|
errors:errors
|
|
log:log
|
|
dnstap:dnstap
|
|
local:local
|
|
dns64:dns64
|
|
any:any
|
|
chaos:chaos
|
|
loadbalance:loadbalance
|
|
tsig:tsig
|
|
rewrite:rewrite
|
|
autopath:autopath
|
|
acl:acl
|
|
cache:cache
|
|
header:header
|
|
dnssec:dnssec
|
|
minimal:minimal
|
|
template:template
|
|
transfer:transfer
|
|
hosts:hosts
|
|
route53:route53
|
|
azure:azure
|
|
clouddns:clouddns
|
|
k8s_external:k8s_external
|
|
kubernetes:kubernetes
|
|
file:file
|
|
auto:auto
|
|
secondary:secondary
|
|
etcd:etcd
|
|
loop:loop
|
|
forward:forward
|
|
grpc:grpc
|
|
erratic:erratic
|
|
whoami:whoami
|
|
on:github.com/coredns/caddy/onevent
|
|
sign:sign
|
|
view:view
|
|
nomad:nomad
|