* perf(proxy): use mutex-based connection pool
The proxy package (used for example by the forward plugin) utilized
an actor model where a single connManager goroutine managed
connection pooling via unbuffered channels (dial, yield, ret). This
design serialized all connection acquisition and release operations
through a single goroutine, creating a bottleneck under high
concurrency. This was observable as a performance degradation when
using a single upstream backend compared to multiple backends
(which sharded the bottleneck).
Changes:
- Removed dial, yield, and ret channels from the Transport struct.
- Removed the connManager goroutine's request processing loop.
- Implemented Dial() and Yield() using a sync.Mutex to protect the
connection slice, allowing for fast concurrent access without
context switching.
- Downgraded connManager to a simple background cleanup loop that
only handles connection expiration on a ticker.
- Updated plugin/pkg/proxy/connect.go to use direct method calls
instead of channel sends.
- Updated tests to reflect the removal of internal channels.
Benchmarks show that this change eliminates the single-backend
bottleneck. Now a single upstream backend performs on par with
multiple backends, and overall throughput is improved.
The implementation aligns with standard Go patterns for connection
pooling (e.g., net/http.Transport).
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
* fix: address PR review for persistent.go
- Named mutex field instead of embedding, to not expose
Lock() and Unlock()
- Move stop check outside of lock in Yield()
- Close() without a separate goroutine
- Change stop channel to struct
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
* fix: address code review feedback for conn pool
- Switch from LIFO to FIFO connection selection for source port
diversity, reducing DNS cache poisoning risk (RFC 5452).
- Remove "clear entire cache" optimization as it was LIFO-specific.
FIFO naturally iterates and skips expired connections.
- Remove all goroutines for closing connections; collect connections
while holding lock, close synchronously after releasing lock.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
* fix: remove unused error consts
No longer utilised after refactoring the channel based approach.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
* feat(forward): add max_idle_conns option
Add configurable connection pool limit for the forward plugin via
the max_idle_conns Corefile option.
Changes:
- Add SetMaxIdleConns to proxy
- Add maxIdleConns field to Forward struct
- Add max_idle_conns parsing in forward plugin setup
- Apply setting to each proxy during configuration
- Update forward plugin README with new option
By default the value is 0 (unbounded). When set, excess
connections returned to the pool are closed immediately
rather than cached.
Also add a yield related test.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
* chore(proxy): simple Dial by closing conns inline
Remove toClose slice collection to reduce complexity. Instead close
expired connections directly while iterating. Reduces complexity with
negligible lock-time impact.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
* chore: fewer explicit Unlock calls
Cleaner and less chance of forgetting to unlock on new possible
code paths.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
---------
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
Previously the parsing logic in the forward plugin setup failed to
recognise when NOERROR was used as a failover RCODE criteria. The
check was in the wrong code branch. This PR fixes it and adds
validation tests. Also updates the plugin README.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
Allows the forward plugin to execute the next plugin based on the return code. Similar to the externally mainted alternate plugin https://github.com/coredns/alternate
Based on the idea of chrisohaver@ in #6549 (comment)
Also incoperated the request to rename `alternate` to `next` as an option
I am having issues adding a proper test for functionality. Primarily, I do not know the code base enough and having multiple `dnstest.NewServer` with ResponseWriter does not work. From my testing these are "Singletons'' and only the last defined response writer is used for all servers
Signed-off-by: Jasper Bernhardt <jasper.bernhardt@live.de>
* plugin/forward: convert the specified domain of health_check to Fqdn
* plugin/forward: update readme for health check
Signed-off-by: vanceli <vanceli@tencent.com>
* trap unsupported FROM cidr notations
Signed-off-by: Chris O'Haver <cohaver@infoblox.com>
* make is a warning
Signed-off-by: Chris O'Haver <cohaver@infoblox.com>
* plugin/forward Add rcode and rtype to request_duration_seconds metric
Signed-off-by: Maxime Ginters <maxime.ginters@shopify.com>
* Control the cardinality of query type
Signed-off-by: Maxime Ginters <maxime.ginters@shopify.com>
sed -i 's/Also See/See Also/' plugin/**/README.md
Some plugins did already use 'See Also', so it's all consistent now.
Fixes: #4196
Signed-off-by: Miek Gieben <miek@miek.nl>
we hc every 0.5s, doing exp backoff will create a large gap in the
ability to re-use an upstream. Doing a exp. backoff up to (say) 3s,
isn't really exp backoff either.
Remove the wording from the documentation.
Signed-off-by: Miek Gieben <miek@miek.nl>
Cleanup a variety of metric issues.
* Eliminate department of redundancy "count_total" naming.
* Use the plural of the unit when appropriate. (ex, "requests")
* Remove label names from metric names where appropriate. (ex, "rcode")
* Simplify request metrics by consolidating type label in to the base
request counter.
* Re-generate man pages.
Signed-off-by: Ben Kochie <superq@gmail.com>
Co-authored-by: Ben Kochie <superq@gmail.com>
* Make the RD-flag in health-checks in the Forward-plugin configurable
Introduces a new configuration flag; `health_check_non_recursive`. This
flag makes the health-checker do non-recursive requests when checking
the health of upstream servers.
Signed-off-by: Geir Haugom <ghagit@haugom.org>
Signed-off-by: Christian Tryti <ctryti@gmail.com>
* Changes after feedback from reviewer
* Better tests of health-checks with and without recursion
* Removed the health_check_non_recursive configuration in favor of
extending the existing health_check configuration. Now supports an
optional `no_rec` argument.
Signed-off-by: Christian Tryti <ctryti@gmail.com>
* Add new test that checks setup of health_check.
Signed-off-by: Christian Tryti <ctryti@gmail.com>
* plugin/pkg/up: make default intervals shorter
I think 15 min is too high, make this lower to react faster.
Signed-off-by: Miek Gieben <miek@miek.nl>
* Update README
Signed-off-by: Miek Gieben <miek@miek.nl>
Move exponential backoff initialization to Start()
Signed-off-by: RickyRajinder <singh.sangh@gmail.com>
Move comment
Increase max interval and update README
Remove trailing whitespace
Change Start() param name back to interval
Caught my eye, we name things directive still, esp when talking about
the prometheus *plugin*. Rename everything that needs to be plugin to
'plugin'. Also make sure Metrics is a H2 section (not H1).
Signed-off-by: Miek Gieben <miek@miek.nl>
* plugin/forward: may Yield not block
Yield may block when we're super busy with creating (and looking) for
connection. Set a small timeout on Yield, to skip putting the connection
back in the queue.
Use persistentConn troughout the socket handling code to be more
consistent.
Signed-off-by: Miek Gieben <miek@miek.nl>
Dont do
Signed-off-by: Miek Gieben <miek@miek.nl>
* Set used in Yield
This gives one central place where we update used in the persistConns
Signed-off-by: Miek Gieben <miek@miek.nl>
After several experiments at SoundCloud we found that the current
minimum read timeout of 10ms is too low. A single request against a
slow/unavailable authoritative server can cause all TCP connections to
get closed. We record a 50th percentile forward/proxy latency of <5ms,
and a 99th percentile latency of 60ms. Using a minimum timeout of 200ms
seems to be a fair trade-off between avoiding unnecessary high
connection churn and reacting to upstream failures in a timely manner.
This change also renames hcDuration to hcInterval to reflect its usage,
and removes the duplicated timeout constant to make code comprehension
easier.
