feat: Improve scraper resiliency and add both proxy and FlareSolverr support (#7953)

docs/docs/documentation/getting-started/faq.md

@@ -263,6 +263,22 @@
 ## Technical Considerations
 
 
+??? question "Why do some recipe imports fail or get blocked?"
+
+    ### Why do some recipe imports fail or get blocked?
+
+    Some recipe websites sit behind bot-protection (e.g. Cloudflare) that can block Mealie from
+    fetching the page or its image. Mealie already impersonates real browsers and rotates between
+    several of them to get around most of this automatically, with no configuration needed.
+
+    If particular sites still fail to import, you can optionally route scraping through a **proxy**
+    with a better IP reputation, and/or fall back to a self-hosted **FlareSolverr** instance that
+    uses a real browser to solve challenges. Both are opt-in and configured via environment
+    variables:
+
+    - [Backend Config - Recipe Scraper](./installation/backend-config.md#recipe-scraper)
+
+
 ??? question "Why setup Email?"
 
     ### Why setup Email?

docs/docs/documentation/getting-started/installation/backend-config.md

@@ -128,6 +128,68 @@ Mealie supports various integrations using OpenAI. For more information, check o
 |-------------------------------------------------------------------------|:-----------:|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | OPENAI_CUSTOM_PROMPT_DIR <br/> :octicons-tag-24: v3.10.0                |    None.    | Path to custom prompt files. Only existing files in your custom directory will override the defaults; any missing or empty custom files will automatically fall back to the system defaults. See https://github.com/mealie-recipes/mealie/tree/mealie-next/mealie/services/openai/prompts for expected file names.     |
 
+### Recipe Scraper
+
+When you import a recipe from a URL, Mealie fetches the page (and its image) before parsing it. Many
+sites sit behind bot-protection (e.g. Cloudflare) that can block these requests. Out of the box Mealie
+mitigates this by impersonating real browsers' TLS fingerprints and rotating between several of them,
+which is enough for most sites and requires no configuration. If you still run into sites that block
+imports, the **opt-in** settings below add two further layers.
+
+| Variables                    | Default | Description                                                                                                                                                                                                                                                             |
+| ---------------------------- | :-----: | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| SCRAPER_PROXY_URL            |  None   | Optional proxy for all outbound scraping and image requests (e.g. `http://user:pass@host:port`). Routing through an IP with a better reputation helps bypass IP-based blocks. Unset disables it.                                                                        |
+| SCRAPER_PROXY_MODE           | always  | How the proxy is used (when `SCRAPER_PROXY_URL` is set): `always` routes every request through it; `fallback` tries a direct request first and only retries through the proxy when a block is detected. Any unrecognized value falls back to `always`.                   |
+| SCRAPER_FLARESOLVERR_URL     |  None   | Optional base URL of a self-hosted [FlareSolverr](https://github.com/FlareSolverr/FlareSolverr) instance (e.g. `http://flaresolverr:8191`). Used only as a last resort to solve JS/Cloudflare challenges. Unset disables it.                                            |
+| SCRAPER_FLARESOLVERR_TIMEOUT |   60    | Maximum seconds FlareSolverr may spend solving a single challenge before giving up.                                                                                                                                                                                    |
+
+#### How Mealie fetches a page
+
+For each import Mealie escalates only as far as it needs to, stopping at the first step that succeeds:
+
+1. **Direct fetch** with rotating browser TLS impersonations — always on, no configuration.
+2. **Proxy** — if `SCRAPER_PROXY_URL` is set (see modes below).
+3. **FlareSolverr** — if `SCRAPER_FLARESOLVERR_URL` is set, and only when the page is still blocked.
+
+Steps 2 and 3 are opt-in, so a default install uses only step 1. Genuine "not found" responses (e.g.
+`404`) are treated as real errors and are never retried through the later steps.
+
+#### Proxy
+
+Most IP-based blocks trigger on the very first request, so `always` mode (the default when a proxy is
+set) is recommended — it routes every request through the proxy from the start. Use `fallback` mode if
+you're on a **metered proxy** and want to avoid paying for requests that would have succeeded directly;
+Mealie will then only route through the proxy after a direct attempt is blocked. The proxy applies to
+both the recipe page and its image download.
+
+#### FlareSolverr
+
+FlareSolverr runs a real headless browser to solve challenges that TLS impersonation alone can't.
+**Mealie neither ships nor manages it** — you host it yourself and point Mealie at it. Because it
+returns rendered **HTML**, it is only used for the recipe page; **image downloads never use
+FlareSolverr** and continue to rely on the direct/proxy path.
+
+Run it as a sidecar container and set `SCRAPER_FLARESOLVERR_URL` to its address:
+
+```yaml
+services:
+  mealie:
+    image: ghcr.io/mealie-recipes/mealie:latest
+    environment:
+      SCRAPER_FLARESOLVERR_URL: http://flaresolverr:8191
+      # optional; default shown
+      # SCRAPER_FLARESOLVERR_TIMEOUT: 60
+
+  flaresolverr:
+    image: ghcr.io/flaresolverr/flaresolverr:latest
+    restart: unless-stopped
+    # No ports need to be published — Mealie reaches it over the internal Docker network.
+```
+
+Because FlareSolverr drives a browser, requests through it are much slower (seconds) and far more
+resource-intensive than a direct fetch. That's why Mealie only falls back to it when a page is actually
+blocked, rather than using it for every import.
+
 ### Theming
 
 Setting the following environmental variables will change the theme of the frontend. Note that the themes are the same for all users. This is a break-change when migration from v0.x.x -> 1.x.x.

mealie/core/settings/settings.py

@@ -2,8 +2,10 @@ import logging
 import os
 import secrets
 from datetime import UTC, datetime
+from enum import StrEnum
 from pathlib import Path
 from typing import Annotated, Literal, NamedTuple
+from urllib.parse import urlparse
 
 from dateutil.tz import tzlocal
 from pydantic import PlainSerializer, field_validator
@@ -15,6 +17,20 @@ from .db_providers import AbstractDBProvider, db_provider_factory
 from .static import PACKAGE_DIR
 
 
+class ScraperProxyMode(StrEnum):
+    """How the scraper uses a configured proxy."""
+
+    always = "always"
+    """Route every request through the proxy (IP-based blocks trigger on the first request)."""
+    fallback = "fallback"
+    """Try direct first; only retry through the proxy when a block is detected."""
+
+    @classmethod
+    def _missing_(cls, value: object) -> "ScraperProxyMode":
+        # Default any unrecognized configuration value to the safest, most useful mode.
+        return cls.always
+
+
 class ScheduleTime(NamedTuple):
     hour: int
     minute: int
@@ -424,6 +440,42 @@ class AppSettings(AppLoggingSettings):
     files are individually optional, each prompt name will fall back to the default if no custom file exists
     """
 
+    # ===============================================
+    # Scraper Configuration
+
+    SCRAPER_PROXY_URL: str | None = None
+    """Optional proxy for all outbound recipe/image scraping requests (e.g. ``http://user:pass@host:port``).
+    Routing through a proxy with a better IP reputation helps bypass IP-based bot blocks. Unset disables it."""
+
+    SCRAPER_PROXY_MODE: ScraperProxyMode = ScraperProxyMode.always
+    """How the scraper uses ``SCRAPER_PROXY_URL`` (when set): ``always`` routes every request through the
+    proxy (recommended, since IP-based blocks trigger on the first request); ``fallback`` tries a direct
+    request first and only retries through the proxy when a block is detected (useful for metered proxies).
+    Any unrecognized value falls back to ``always``."""
+
+    SCRAPER_FLARESOLVERR_URL: str | None = None
+    """Optional base URL of a self-hosted FlareSolverr instance (e.g. ``http://flaresolverr:8191``). When
+    set, HTML scrapes that remain blocked after the direct/proxy attempts are retried through FlareSolverr,
+    which drives a real browser to solve JS/Cloudflare challenges. Mealie neither ships nor manages it.
+    Image downloads never use it, since FlareSolverr returns HTML rather than binary content."""
+
+    SCRAPER_FLARESOLVERR_TIMEOUT: int = 60
+    """Maximum seconds FlareSolverr may spend solving a single challenge before giving up."""
+
+    @field_validator("SCRAPER_PROXY_URL", "SCRAPER_FLARESOLVERR_URL")
+    @classmethod
+    def validate_scraper_url(cls, v: str | None, info) -> str | None:
+        """Fail fast at startup if a scraper URL is set but malformed (e.g. missing the scheme)."""
+        if not v:
+            return v
+
+        parsed = urlparse(v)
+        if not parsed.scheme or not parsed.netloc:
+            raise ValueError(
+                f"{info.field_name} must be a full URL including scheme and host, e.g. 'http://host:port' (got '{v}')"
+            )
+        return v
+
     # ===============================================
     # Web Concurrency
 

mealie/pkgs/safehttp/__init__.py

@@ -1,7 +1,19 @@
+from .fetch import (
+    BROWSER_IMPERSONATIONS,
+    SCRAPER_TIMEOUT,
+    FetchResult,
+    ForceTimeoutException,
+    resilient_fetch,
+)
 from .transport import AsyncSafeTransport, ForcedTimeoutException, InvalidDomainError
 
 __all__ = [
     "AsyncSafeTransport",
     "ForcedTimeoutException",
     "InvalidDomainError",
+    "BROWSER_IMPERSONATIONS",
+    "SCRAPER_TIMEOUT",
+    "FetchResult",
+    "ForceTimeoutException",
+    "resilient_fetch",
 ]

mealie/pkgs/safehttp/fetch.py

@@ -0,0 +1,313 @@
+import asyncio
+import random
+import time
+from dataclasses import dataclass
+
+import httpx
+from httpx import AsyncClient
+
+from mealie.core.config import get_app_settings
+from mealie.core.root_logger import get_logger
+from mealie.core.settings.settings import ScraperProxyMode
+
+from . import flaresolverr
+from .transport import AsyncSafeTransport
+
+SCRAPER_TIMEOUT = 15
+
+# Overall wall-clock budget for a single fetch, across all impersonation attempts and backoffs.
+# Bounds worst-case latency when a site repeatedly blocks or stalls us.
+SCRAPER_TOTAL_TIMEOUT = 45
+
+BROWSER_IMPERSONATIONS = [
+    "chrome",
+    "firefox",
+    "safari",
+    "edge",
+]
+
+NON_CHALLENGE_4XX = frozenset({404, 410})
+RATE_LIMIT_STATUS_CODES = frozenset({429, 503})
+
+# Substrings found in the bodies of bot-challenge/interstitial pages that are served with a
+# 200 status (so status/header checks alone would treat them as a successful fetch). Kept
+# specific to anti-bot infrastructure identifiers to avoid false positives on real content.
+_CHALLENGE_BODY_MARKERS: tuple[bytes, ...] = (
+    b"__cf_chl",
+    b"cf-browser-verification",
+    b"/cdn-cgi/challenge-platform",
+    b"challenges.cloudflare.com",
+    b"_incapsula_resource",
+    b"distil_r_captcha",
+    b"px-captcha",
+    b"perimeterx",
+    b"datadome",
+)
+_CHALLENGE_BODY_SAMPLE = 4096
+
+_BASE_BACKOFF = 1.0
+_MAX_BACKOFF = 5.0
+_BACKOFF_JITTER = 0.5
+
+logger = get_logger()
+
+
+class ForceTimeoutException(Exception):
+    """Raised when reading a response body exceeds the fetch timeout."""
+
+
+@dataclass
+class FetchResult:
+    """The outcome of a resilient fetch, decoupled from the (now-closed) streaming response."""
+
+    content: bytes
+    status_code: int
+    url: str
+    headers: httpx.Headers
+    encoding: str | None
+
+    @property
+    def text(self) -> str:
+        # Mirrors the decoding behavior of requests' `text` property.
+        try:
+            return str(self.content, self.encoding, errors="replace")  # type: ignore[arg-type]
+        except (LookupError, TypeError):
+            # LookupError: unknown encoding name. TypeError: encoding is None.
+            return str(self.content, errors="replace")
+
+
+def is_challenge_status(status_code: int) -> bool:
+    if status_code == 503:
+        return True
+    if 400 <= status_code < 500:
+        return status_code not in NON_CHALLENGE_4XX
+    return False
+
+
+def headers_indicate_challenge(headers: httpx.Headers) -> bool:
+    # Cloudflare sets `cf-mitigated: challenge` on interstitial/challenge responses,
+    # which can otherwise carry a 200 status.
+    return "cf-mitigated" in headers
+
+
+def body_indicates_challenge(content: bytes) -> bool:
+    sample = content[:_CHALLENGE_BODY_SAMPLE].lower()
+    return any(marker in sample for marker in _CHALLENGE_BODY_MARKERS)
+
+
+def _build_transport(impersonate: str, proxy: str | None = None) -> AsyncSafeTransport:
+    kwargs: dict = {
+        "impersonate": impersonate,
+        "default_headers": True,
+        # disable SSL verification since we can handle untrusted data and some sites don't have certs
+        # (this also covers the proxy connection, so no separate proxy-verify knob is needed)
+        "verify": False,
+    }
+    if proxy:
+        kwargs["proxy"] = proxy
+    return AsyncSafeTransport(**kwargs)
+
+
+async def _read_capped(resp: httpx.Response, timeout: int) -> bytes:
+    """
+    Reads a streaming body, aborting if it takes longer than ``timeout`` seconds.
+
+    Mitigates abuse from URLs that serve arbitrarily large or slow content.
+    """
+    content = b""
+    start_time = time.monotonic()
+    async for chunk in resp.aiter_bytes(chunk_size=1024):
+        content += chunk
+        if time.monotonic() - start_time > timeout:
+            raise ForceTimeoutException()
+    return content
+
+
+async def _sleep_backoff(retry_after: str | None, deadline: float) -> None:
+    """Sleeps a jittered backoff before the next attempt, never past the overall deadline."""
+    delay = _BASE_BACKOFF
+    if retry_after:
+        try:
+            # Retry-After may be a delta-seconds integer; if it's an HTTP-date we ignore it.
+            delay = max(delay, float(retry_after))
+        except ValueError:
+            pass
+
+    delay = min(delay, _MAX_BACKOFF) + random.uniform(0, _BACKOFF_JITTER)
+
+    remaining = deadline - time.monotonic()
+    if remaining <= 0:
+        return
+    await asyncio.sleep(min(delay, remaining))
+
+
+async def _attempt(
+    url: str,
+    method: str,
+    timeout: int,
+    impersonation: str,
+    read_body: bool,
+    proxy: str | None,
+) -> tuple[FetchResult | None, bool, int, str | None]:
+    """
+    Performs a single fetch attempt with one browser impersonation.
+
+    Returns ``(result, blocked, status_code, retry_after)``:
+    - ``result`` is set on success (and ``blocked`` is False).
+    - ``blocked`` is True when the response looks like a bot challenge and rotating to another
+      fingerprint may help.
+    - When both ``result`` is None and ``blocked`` is False, the response was a hard error that
+      rotating won't fix, and the caller should stop.
+    """
+    transport = _build_transport(impersonation, proxy)
+    async with AsyncClient(transport=transport) as client:
+        async with client.stream(method, url, timeout=timeout, follow_redirects=True) as resp:
+            status_code = resp.status_code
+            retry_after = resp.headers.get("Retry-After")
+
+            blocked = is_challenge_status(status_code) or headers_indicate_challenge(resp.headers)
+
+            if blocked:
+                logger.debug(f'Challenge/block detected (status={status_code}) with impersonation "{impersonation}"')
+                return None, True, status_code, retry_after
+
+            if status_code >= 400:
+                # A genuine client/server error (e.g. 404, 410, 500) that a different fingerprint
+                # won't resolve. Stop rotating.
+                logger.debug(f'Error status code {status_code} with impersonation "{impersonation}"')
+                return None, False, status_code, retry_after
+
+            content = b""
+            if read_body:
+                content = await _read_capped(resp, timeout)
+                if body_indicates_challenge(content):
+                    logger.debug(f'Challenge page body detected with impersonation "{impersonation}"')
+                    return None, True, status_code, retry_after
+
+            result = FetchResult(
+                content=content,
+                status_code=status_code,
+                url=str(resp.url),
+                headers=resp.headers,
+                encoding=resp.encoding,
+            )
+            return result, False, status_code, retry_after
+
+
+async def _rotate(
+    url: str,
+    method: str,
+    timeout: int,
+    read_body: bool,
+    proxy: str | None,
+    deadline: float,
+) -> tuple[FetchResult | None, bool]:
+    """
+    Cycles through browser impersonations (in randomized order) for a single egress path
+    (direct or via ``proxy``).
+
+    Returns ``(result, blocked)``. ``blocked`` is True only when every impersonation was rejected
+    by a challenge/block -- i.e. the failure might be worth escalating (e.g. to a proxy). It is
+    False on success or on a hard error, where escalation wouldn't help.
+    """
+    impersonations = list(BROWSER_IMPERSONATIONS)
+    random.shuffle(impersonations)
+
+    for index, impersonation in enumerate(impersonations):
+        if time.monotonic() >= deadline:
+            logger.debug("Scraper budget exhausted before trying all impersonations")
+            break
+
+        logger.debug(f'Trying browser impersonation: "{impersonation}"')
+        result, blocked, status_code, retry_after = await _attempt(
+            url, method, timeout, impersonation, read_body, proxy
+        )
+
+        if result is not None:
+            return result, False
+
+        if not blocked:
+            # Hard error; rotating fingerprints won't help.
+            return None, False
+
+        is_last = index == len(impersonations) - 1
+        if not is_last and status_code in RATE_LIMIT_STATUS_CODES:
+            await _sleep_backoff(retry_after, deadline)
+
+    return None, True
+
+
+def _solution_to_result(solution: flaresolverr.FlareSolverrSolution) -> FetchResult:
+    return FetchResult(
+        content=solution.html.encode("utf-8", errors="replace"),
+        status_code=solution.status_code,
+        url=solution.url,
+        headers=httpx.Headers({"content-type": "text/html; charset=utf-8"}),
+        encoding="utf-8",
+    )
+
+
+async def resilient_fetch(
+    url: str,
+    *,
+    method: str = "GET",
+    timeout: int = SCRAPER_TIMEOUT,
+    allow_flaresolverr: bool = True,
+) -> FetchResult | None:
+    """
+    Fetches a URL while cycling through browser TLS impersonations (via httpx-curl-cffi) to
+    bypass bot-detection systems that fingerprint the TLS handshake (JA3/JA4), such as Cloudflare.
+
+    Impersonations are tried in a randomized order. On a detected challenge/block (a challenge
+    status code, a ``cf-mitigated`` header, or challenge markers in an otherwise-200 body) the
+    next fingerprint is tried, with a short jittered backoff for rate-limit statuses. A genuine
+    error status (e.g. 404) stops the rotation immediately, since a new fingerprint won't help.
+
+    When ``SCRAPER_PROXY_URL`` is configured, requests egress through it: in ``always`` mode every
+    request uses the proxy; in ``fallback`` mode a direct attempt is made first and the proxy is
+    only used to retry when every direct impersonation was blocked.
+
+    As a last resort, if the fetch is still blocked and ``SCRAPER_FLARESOLVERR_URL`` is configured,
+    the request is escalated to FlareSolverr (a headless browser). This only applies to HTML fetches
+    (``allow_flaresolverr`` and a body-returning method), since FlareSolverr returns HTML, not the
+    binary content an image download needs.
+
+    The whole operation is bounded by ``SCRAPER_TOTAL_TIMEOUT``, and each attempt's body read is
+    bounded by ``timeout`` seconds, to mitigate abuse from URLs that serve arbitrarily large content.
+
+    Returns a ``FetchResult`` for the first successful response, or ``None`` if every impersonation
+    was blocked, the server returned a hard error, or the budget was exhausted.
+    """
+    logger.debug(f"Fetching URL: {url}")
+
+    read_body = method.upper() != "HEAD"
+    deadline = time.monotonic() + SCRAPER_TOTAL_TIMEOUT
+
+    settings = get_app_settings()
+    proxy = settings.SCRAPER_PROXY_URL or None
+    proxy_first = bool(proxy) and settings.SCRAPER_PROXY_MODE == ScraperProxyMode.always
+
+    result, blocked = await _rotate(url, method, timeout, read_body, proxy if proxy_first else None, deadline)
+    if result is not None:
+        return result
+
+    # In `fallback` mode, escalate to the proxy only when a direct attempt was blocked (not on a
+    # hard error, and not if we already used the proxy above).
+    if blocked and proxy and not proxy_first:
+        logger.debug("Direct fetch blocked; retrying through configured proxy")
+        result, blocked = await _rotate(url, method, timeout, read_body, proxy, deadline)
+        if result is not None:
+            return result
+
+    # Final escalation: a real browser via FlareSolverr. HTML-only, and only when still blocked.
+    # Note: the impersonation rotation above always runs first, so its SSRF guard (which rejects
+    # private target IPs) has already vetted `url` before we hand it to FlareSolverr.
+    if blocked and read_body and allow_flaresolverr and settings.SCRAPER_FLARESOLVERR_URL:
+        logger.debug("Fetch still blocked; escalating to FlareSolverr")
+        solution = await flaresolverr.solve(
+            settings.SCRAPER_FLARESOLVERR_URL, url, settings.SCRAPER_FLARESOLVERR_TIMEOUT
+        )
+        if solution is not None:
+            return _solution_to_result(solution)
+
+    return result

mealie/pkgs/safehttp/flaresolverr.py

@@ -0,0 +1,76 @@
+from dataclasses import dataclass, field
+
+import httpx
+
+from mealie.core.root_logger import get_logger
+
+logger = get_logger()
+
+# Extra seconds allowed for the HTTP round-trip on top of FlareSolverr's own solve budget,
+# to account for browser startup and network overhead.
+_HTTP_OVERHEAD_SECONDS = 15
+
+
+@dataclass
+class FlareSolverrSolution:
+    """The useful parts of a successful FlareSolverr solve."""
+
+    html: str
+    status_code: int
+    url: str
+    cookies: list[dict] = field(default_factory=list)
+    user_agent: str = ""
+
+
+async def solve(base_url: str, url: str, timeout: int) -> FlareSolverrSolution | None:
+    """
+    Asks a FlareSolverr instance to fetch ``url`` with a real (headless) browser, solving any
+    JS/Cloudflare challenge along the way.
+
+    FlareSolverr is a REST service, not a proxy: we POST a job to ``{base_url}/v1`` and read the
+    rendered HTML back out of the response envelope.
+
+    Returns ``None`` (never raises) on any failure -- an unreachable or misbehaving FlareSolverr
+    must only fail this escalation, never break scraping.
+    """
+    endpoint = base_url.rstrip("/") + "/v1"
+    payload = {
+        "cmd": "request.get",
+        "url": url,
+        "maxTimeout": timeout * 1000,  # FlareSolverr expects milliseconds
+    }
+
+    try:
+        async with httpx.AsyncClient(timeout=timeout + _HTTP_OVERHEAD_SECONDS) as client:
+            resp = await client.post(endpoint, json=payload)
+    except Exception:
+        logger.exception(f"FlareSolverr request failed for {url}")
+        return None
+
+    if resp.status_code != 200:
+        logger.warning(f"FlareSolverr returned HTTP {resp.status_code} for {url}")
+        return None
+
+    try:
+        data = resp.json()
+    except Exception:
+        logger.exception(f"FlareSolverr returned a non-JSON response for {url}")
+        return None
+
+    if data.get("status") != "ok":
+        logger.warning(f"FlareSolverr could not solve {url}: {data.get('message')}")
+        return None
+
+    solution = data.get("solution") or {}
+    html = solution.get("response")
+    if not html:
+        logger.warning(f"FlareSolverr returned an empty solution for {url}")
+        return None
+
+    return FlareSolverrSolution(
+        html=html,
+        status_code=solution.get("status", 200),
+        url=solution.get("url", url),
+        cookies=solution.get("cookies", []),
+        user_agent=solution.get("userAgent", ""),
+    )

mealie/services/recipe/recipe_data_service.py

@@ -3,7 +3,6 @@ import shutil
 from logging import Logger
 from pathlib import Path
 
-from httpx import AsyncClient, Response
 from pydantic import UUID4
 
 from mealie.pkgs import img, safehttp
@@ -31,17 +30,18 @@ async def largest_content_len(urls: list[str]) -> tuple[str, int]:
 
     max_concurrency = 10
 
-    async def do(client: AsyncClient, url: str) -> Response:
+    tasks = [safehttp.resilient_fetch(url, method="HEAD") for url in urls]
-        return await client.head(url)
+    responses: list[safehttp.FetchResult | None] = await gather_with_concurrency(
+        max_concurrency, *tasks, ignore_exceptions=True
+    )
+    for response in responses:
+        if response is None:
+            continue
 
-    async with AsyncClient(transport=safehttp.AsyncSafeTransport(impersonate="chrome")) as client:
+        len_int = int(response.headers.get("Content-Length", 0))
-        tasks = [do(client, url) for url in urls]
+        if len_int > largest_len:
-        responses: list[Response] = await gather_with_concurrency(max_concurrency, *tasks, ignore_exceptions=True)
+            largest_url = response.url
-        for response in responses:
+            largest_len = len_int
-            len_int = int(response.headers.get("Content-Length", 0))
-            if len_int > largest_len:
-                largest_url = str(response.url)
-                largest_len = len_int
 
     return largest_url, largest_len
 
@@ -146,24 +146,23 @@ class RecipeDataService(BaseService):
         file_name = f"{self.recipe_id!s}.{ext}"
         file_path = Recipe.directory_from_id(self.recipe_id).joinpath("images", file_name)
 
-        async with AsyncClient(transport=safehttp.AsyncSafeTransport(impersonate="chrome")) as client:
+        try:
-            try:
+            # FlareSolverr returns HTML, not image bytes, so it can't serve an image download.
-                r = await client.get(image_url_str)
+            r = await safehttp.resilient_fetch(image_url_str, allow_flaresolverr=False)
-            except Exception:
+        except Exception:
-                self.logger.exception("Fatal Image Request Exception")
+            self.logger.exception("Fatal Image Request Exception")
-                return None
+            return None
 
-            if r.status_code != 200:
+        if r is None:
-                # TODO: Probably should throw an exception in this case as well, but before these changes
+            # Every impersonation was rejected, or the server returned an error status.
-                # we were returning None if it failed anyways.
+            return None
-                return None
 
-            content_type = r.headers.get("content-type", "")
+        content_type = r.headers.get("content-type", "")
 
-            if "image" not in content_type:
+        if "image" not in content_type:
-                self.logger.error(f"Content-Type: {content_type} is not an image")
+            self.logger.error(f"Content-Type: {content_type} is not an image")
-                raise NotAnImageError(f"Content-Type {content_type} is not an image")
+            raise NotAnImageError(f"Content-Type {content_type} is not an image")
 
-            self.logger.debug(f"File Name Suffix {file_path.suffix}")
+        self.logger.debug(f"File Name Suffix {file_path.suffix}")
-            self.write_image(r.read(), file_path.suffix)
+        self.write_image(r.content, file_path.suffix)
-            file_path.unlink(missing_ok=True)
+        file_path.unlink(missing_ok=True)

mealie/services/scraper/scraper_strategies.py

@@ -1,7 +1,6 @@
 import asyncio
 import functools
 import re
-import time
 from abc import ABC, abstractmethod
 from collections.abc import Awaitable, Callable
 from pathlib import Path
@@ -11,7 +10,6 @@ import bs4
 import extruct
 import yt_dlp
 from fastapi import HTTPException, status
-from httpx import AsyncClient, Response
 from recipe_scrapers import NoSchemaFoundInWildMode, SchemaScraperFactory, scrape_html
 from slugify import slugify
 from w3lib.html import get_base_url
@@ -33,14 +31,10 @@ from mealie.services.scraper.scraped_extras import ScrapedExtras
 
 from . import cleaner
 
-SCRAPER_TIMEOUT = 15
+# Re-exported for backwards compatibility with existing importers (e.g. recipe route error handling).
-
+SCRAPER_TIMEOUT = safehttp.SCRAPER_TIMEOUT
-BROWSER_IMPERSONATIONS = [
+BROWSER_IMPERSONATIONS = safehttp.BROWSER_IMPERSONATIONS
-    "chrome",
+ForceTimeoutException = safehttp.ForceTimeoutException
-    "firefox",
-    "safari",
-    "edge",
-]
 
 logger = get_logger()
 
@@ -51,10 +45,6 @@ def _get_yt_dlp_extractors() -> list:
     return [ie for ie in yt_dlp.extractor.gen_extractors() if ie.working() and not isinstance(ie, GenericIE)]
 
 
-class ForceTimeoutException(Exception):
-    pass
-
-
 async def safe_scrape_html(url: str) -> str:
     """
     Scrapes the html from a url but will cancel the request
@@ -65,74 +55,8 @@ async def safe_scrape_html(url: str) -> str:
     bot-detection systems that fingerprint the TLS handshake (JA3/JA4),
     such as Cloudflare.
     """
-    logger.debug(f"Scraping URL: {url}")
+    result = await safehttp.resilient_fetch(url)
-
+    return result.text if result else ""
-    html_bytes = b""
-    response: Response | None = None
-
-    for impersonation in BROWSER_IMPERSONATIONS:
-        logger.debug(f'Trying browser impersonation: "{impersonation}"')
-
-        html_bytes = b""
-        response = None
-
-        transport = safehttp.AsyncSafeTransport(
-            impersonate=impersonation,
-            default_headers=True,
-            verify=False,  # disable SSL verification since we can handle untrusted data and some sites don't have certs
-        )
-        async with AsyncClient(transport=transport) as client:
-            async with client.stream(
-                "GET",
-                url,
-                timeout=SCRAPER_TIMEOUT,
-                follow_redirects=True,
-            ) as resp:
-                if resp.status_code == 403:
-                    logger.debug(f'403 Forbidden with impersonation "{impersonation}", trying next')
-                    continue
-
-                if resp.status_code >= 400:
-                    logger.debug(f'Error status code {resp.status_code} with impersonation "{impersonation}"')
-                    break
-
-                start_time = time.time()
-
-                async for chunk in resp.aiter_bytes(chunk_size=1024):
-                    html_bytes += chunk
-
-                    if time.time() - start_time > SCRAPER_TIMEOUT:
-                        raise ForceTimeoutException()
-
-                response = resp
-                break
-
-    if not (response and html_bytes):
-        return ""
-
-    # =====================================
-    # Copied from requests text property
-
-    # Try charset from content-type
-    encoding = response.encoding
-
-    # Fallback to auto-detected encoding.
-    if encoding is None:
-        encoding = response.apparent_encoding
-
-    # Decode unicode from given encoding.
-    try:
-        content = str(html_bytes, encoding, errors="replace")
-    except (LookupError, TypeError):
-        # A LookupError is raised if the encoding was not found which could
-        # indicate a misspelling or similar mistake.
-        #
-        # A TypeError can be raised if encoding is None
-        #
-        # So we try blindly encoding.
-        content = str(html_bytes, errors="replace")
-
-    return content
 
 
 class ABCScraperStrategy(ABC):

tests/unit_tests/pkgs/safehttp/test_fetch.py

@@ -0,0 +1,418 @@
+from collections.abc import AsyncIterator
+from types import SimpleNamespace
+
+import httpx
+import pytest
+
+from mealie.pkgs.safehttp import fetch
+
+
+# ---------------------------------------------------------------------------
+# Detection helpers
+# ---------------------------------------------------------------------------
+@pytest.mark.parametrize(
+    ("status_code", "expected"),
+    [
+        (200, False),
+        (301, False),
+        (400, True),
+        (401, True),
+        (402, True),
+        (403, True),
+        (406, True),
+        (429, True),
+        (451, True),
+        (404, False),  # resource genuinely missing -> not worth rotating
+        (410, False),
+        (500, False),  # server error a new fingerprint won't fix
+        (502, False),
+        (503, True),  # WAFs use 503 during interstitials
+        (504, False),
+    ],
+)
+def test_is_challenge_status(status_code: int, expected: bool):
+    assert fetch.is_challenge_status(status_code) is expected
+
+
+def test_headers_indicate_challenge():
+    assert fetch.headers_indicate_challenge(httpx.Headers({"cf-mitigated": "challenge"})) is True
+    # header lookup is case-insensitive
+    assert fetch.headers_indicate_challenge(httpx.Headers({"CF-Mitigated": "challenge"})) is True
+    assert fetch.headers_indicate_challenge(httpx.Headers({"server": "cloudflare"})) is False
+    assert fetch.headers_indicate_challenge(httpx.Headers({})) is False
+
+
+def test_body_indicates_challenge():
+    assert fetch.body_indicates_challenge(b"<html>...cf-browser-verification...</html>") is True
+    # marker matching is case-insensitive
+    assert fetch.body_indicates_challenge(b"<script src='/cdn-cgi/challenge-platform/x'></script>") is True
+    assert fetch.body_indicates_challenge(b"<html><body>A normal recipe for cookies</body></html>") is False
+
+
+def test_body_indicates_challenge_only_samples_head():
+    # A marker past the sampled window should not be detected.
+    body = b"x" * (fetch._CHALLENGE_BODY_SAMPLE + 100) + b"datadome"
+    assert fetch.body_indicates_challenge(body) is False
+
+
+# ---------------------------------------------------------------------------
+# FetchResult.text
+# ---------------------------------------------------------------------------
+def test_fetch_result_text_decoding():
+    assert fetch.FetchResult(b"h\xc3\xa9llo", 200, "http://x", httpx.Headers(), "utf-8").text == "héllo"
+    # None encoding falls back to a blind decode
+    assert fetch.FetchResult(b"abc", 200, "http://x", httpx.Headers(), None).text == "abc"
+    # An unknown encoding name also falls back rather than raising
+    assert fetch.FetchResult(b"abc", 200, "http://x", httpx.Headers(), "not-real").text == "abc"
+
+
+# ---------------------------------------------------------------------------
+# Fakes for the fetch loop
+# ---------------------------------------------------------------------------
+class _FakeResponse:
+    def __init__(self, status_code: int, *, headers: dict | None = None, body: bytes = b"", url: str = "https://x/r"):
+        self.status_code = status_code
+        self.headers = httpx.Headers(headers or {})
+        self.encoding = "utf-8"
+        self.url = url
+        self._body = body
+
+    async def aiter_bytes(self, chunk_size: int = 1024) -> AsyncIterator[bytes]:
+        for i in range(0, len(self._body), chunk_size):
+            yield self._body[i : i + chunk_size]
+
+    async def __aenter__(self):
+        return self
+
+    async def __aexit__(self, *exc):
+        return False
+
+
+class _FakeClient:
+    def __init__(self, response: _FakeResponse):
+        self._response = response
+
+    async def __aenter__(self):
+        return self
+
+    async def __aexit__(self, *exc):
+        return False
+
+    def stream(self, method: str, url: str, **kwargs):
+        return self._response
+
+
+def _patch_responses(
+    monkeypatch,
+    responses: list[_FakeResponse],
+    *,
+    proxy_url: str | None = None,
+    proxy_mode=fetch.ScraperProxyMode.always,
+    flaresolverr_url: str | None = None,
+    flaresolverr_timeout: int = 60,
+) -> dict:
+    """Scripts consecutive attempts to return the given responses.
+
+    Records the number of attempts and the proxy passed to each, and isolates the fetch from real
+    app settings by injecting the given scraper configuration.
+    """
+    state = {"queue": list(responses), "attempts": 0, "proxies": []}
+
+    def make_client(*args, **kwargs):
+        state["attempts"] += 1
+        return _FakeClient(state["queue"].pop(0))
+
+    def fake_build_transport(impersonate: str, proxy: str | None = None):
+        state["proxies"].append(proxy)
+        return None
+
+    monkeypatch.setattr(fetch, "AsyncClient", make_client)
+    monkeypatch.setattr(fetch, "_build_transport", fake_build_transport)
+    monkeypatch.setattr(
+        fetch,
+        "get_app_settings",
+        lambda: SimpleNamespace(
+            SCRAPER_PROXY_URL=proxy_url,
+            SCRAPER_PROXY_MODE=proxy_mode,
+            SCRAPER_FLARESOLVERR_URL=flaresolverr_url,
+            SCRAPER_FLARESOLVERR_TIMEOUT=flaresolverr_timeout,
+        ),
+    )
+    return state
+
+
+# ---------------------------------------------------------------------------
+# resilient_fetch loop
+# ---------------------------------------------------------------------------
+@pytest.mark.asyncio
+async def test_success_on_first_attempt(monkeypatch):
+    state = _patch_responses(monkeypatch, [_FakeResponse(200, body=b"<html>recipe</html>")])
+    result = await fetch.resilient_fetch("https://x/r")
+
+    assert result is not None
+    assert result.status_code == 200
+    assert result.content == b"<html>recipe</html>"
+    assert state["attempts"] == 1
+
+
+@pytest.mark.asyncio
+async def test_rotates_through_all_impersonations_on_block(monkeypatch):
+    state = _patch_responses(monkeypatch, [_FakeResponse(403) for _ in fetch.BROWSER_IMPERSONATIONS])
+    result = await fetch.resilient_fetch("https://x/r")
+
+    assert result is None
+    assert state["attempts"] == len(fetch.BROWSER_IMPERSONATIONS)
+
+
+@pytest.mark.asyncio
+async def test_rotates_past_200_challenge_body(monkeypatch):
+    state = _patch_responses(
+        monkeypatch,
+        [
+            _FakeResponse(200, body=b"<html>__cf_chl just a moment</html>"),
+            _FakeResponse(200, body=b"<html>real recipe</html>"),
+        ],
+    )
+    result = await fetch.resilient_fetch("https://x/r")
+
+    assert result is not None
+    assert result.content == b"<html>real recipe</html>"
+    assert state["attempts"] == 2
+
+
+@pytest.mark.asyncio
+async def test_hard_error_stops_immediately(monkeypatch):
+    state = _patch_responses(monkeypatch, [_FakeResponse(404) for _ in fetch.BROWSER_IMPERSONATIONS])
+    result = await fetch.resilient_fetch("https://x/r")
+
+    assert result is None
+    assert state["attempts"] == 1  # did not rotate on a 404
+
+
+@pytest.mark.asyncio
+async def test_head_request_skips_body(monkeypatch):
+    # A body that *would* look like a challenge is ignored for HEAD (no body is read).
+    state = _patch_responses(monkeypatch, [_FakeResponse(200, body=b"__cf_chl")])
+    result = await fetch.resilient_fetch("https://x/r", method="HEAD")
+
+    assert result is not None
+    assert result.content == b""
+    assert state["attempts"] == 1
+
+
+@pytest.mark.asyncio
+async def test_rate_limit_triggers_backoff_then_succeeds(monkeypatch):
+    slept: list[float] = []
+
+    async def fake_sleep(delay: float):
+        slept.append(delay)
+
+    monkeypatch.setattr(fetch.asyncio, "sleep", fake_sleep)
+    state = _patch_responses(
+        monkeypatch,
+        [_FakeResponse(429, headers={"Retry-After": "2"}), _FakeResponse(200, body=b"ok")],
+    )
+
+    result = await fetch.resilient_fetch("https://x/r")
+
+    assert result is not None
+    assert result.content == b"ok"
+    assert state["attempts"] == 2
+    assert len(slept) == 1
+    assert slept[0] >= 2.0  # honored Retry-After
+
+
+@pytest.mark.asyncio
+async def test_no_backoff_on_plain_403(monkeypatch):
+    slept: list[float] = []
+
+    async def fake_sleep(delay: float):
+        slept.append(delay)
+
+    monkeypatch.setattr(fetch.asyncio, "sleep", fake_sleep)
+    _patch_responses(monkeypatch, [_FakeResponse(403), _FakeResponse(200, body=b"ok")])
+
+    result = await fetch.resilient_fetch("https://x/r")
+
+    assert result is not None
+    assert slept == []  # 403 rotates immediately, no rate-limit backoff
+
+
+# ---------------------------------------------------------------------------
+# Proxy behavior
+# ---------------------------------------------------------------------------
+@pytest.mark.asyncio
+async def test_no_proxy_when_unconfigured(monkeypatch):
+    state = _patch_responses(monkeypatch, [_FakeResponse(200, body=b"ok")], proxy_url=None)
+    result = await fetch.resilient_fetch("https://x/r")
+
+    assert result is not None
+    assert state["proxies"] == [None]
+
+
+@pytest.mark.asyncio
+async def test_always_mode_proxies_first_request(monkeypatch):
+    state = _patch_responses(
+        monkeypatch,
+        [_FakeResponse(200, body=b"ok")],
+        proxy_url="http://proxy:8080",
+        proxy_mode=fetch.ScraperProxyMode.always,
+    )
+    result = await fetch.resilient_fetch("https://x/r")
+
+    assert result is not None
+    # proxy is used from the very first request, even though it succeeded (no block)
+    assert state["proxies"] == ["http://proxy:8080"]
+
+
+@pytest.mark.asyncio
+async def test_fallback_mode_direct_first_then_proxy_on_block(monkeypatch):
+    responses = [_FakeResponse(403) for _ in fetch.BROWSER_IMPERSONATIONS]  # direct all blocked
+    responses.append(_FakeResponse(200, body=b"ok"))  # first proxied attempt succeeds
+    state = _patch_responses(
+        monkeypatch,
+        responses,
+        proxy_url="http://proxy:8080",
+        proxy_mode=fetch.ScraperProxyMode.fallback,
+    )
+    result = await fetch.resilient_fetch("https://x/r")
+
+    assert result is not None
+    assert result.content == b"ok"
+    # direct rotation used no proxy; escalation used the proxy
+    n = len(fetch.BROWSER_IMPERSONATIONS)
+    assert state["proxies"][:n] == [None] * n
+    assert state["proxies"][n] == "http://proxy:8080"
+
+
+@pytest.mark.asyncio
+async def test_fallback_mode_does_not_escalate_on_hard_error(monkeypatch):
+    state = _patch_responses(
+        monkeypatch,
+        [_FakeResponse(404) for _ in fetch.BROWSER_IMPERSONATIONS],
+        proxy_url="http://proxy:8080",
+        proxy_mode=fetch.ScraperProxyMode.fallback,
+    )
+    result = await fetch.resilient_fetch("https://x/r")
+
+    assert result is None
+    # a 404 is a hard error: no rotation, and no proxy escalation
+    assert state["proxies"] == [None]
+
+
+@pytest.mark.asyncio
+async def test_always_mode_does_not_double_escalate(monkeypatch):
+    # Every attempt blocked in always-mode should not trigger a second (proxy) rotation.
+    state = _patch_responses(
+        monkeypatch,
+        [_FakeResponse(403) for _ in fetch.BROWSER_IMPERSONATIONS],
+        proxy_url="http://proxy:8080",
+        proxy_mode=fetch.ScraperProxyMode.always,
+    )
+    result = await fetch.resilient_fetch("https://x/r")
+
+    assert result is None
+    assert state["attempts"] == len(fetch.BROWSER_IMPERSONATIONS)
+    assert all(p == "http://proxy:8080" for p in state["proxies"])
+
+
+# ---------------------------------------------------------------------------
+# FlareSolverr escalation
+# ---------------------------------------------------------------------------
+def _patch_flaresolverr(monkeypatch, solution):
+    """Records calls to flaresolverr.solve and returns the given solution (or None)."""
+    calls: list[tuple] = []
+
+    async def fake_solve(base_url, url, timeout):
+        calls.append((base_url, url, timeout))
+        return solution
+
+    monkeypatch.setattr(fetch.flaresolverr, "solve", fake_solve)
+    return calls
+
+
+@pytest.mark.asyncio
+async def test_escalates_to_flaresolverr_when_blocked(monkeypatch):
+    _patch_responses(
+        monkeypatch,
+        [_FakeResponse(403) for _ in fetch.BROWSER_IMPERSONATIONS],
+        flaresolverr_url="http://flaresolverr:8191",
+    )
+    solution = fetch.flaresolverr.FlareSolverrSolution(html="<html>solved</html>", status_code=200, url="https://x/r")
+    calls = _patch_flaresolverr(monkeypatch, solution)
+
+    result = await fetch.resilient_fetch("https://x/r")
+
+    assert result is not None
+    assert result.text == "<html>solved</html>"
+    assert len(calls) == 1
+    assert calls[0] == ("http://flaresolverr:8191", "https://x/r", 60)
+
+
+@pytest.mark.asyncio
+async def test_no_flaresolverr_when_unconfigured(monkeypatch):
+    _patch_responses(monkeypatch, [_FakeResponse(403) for _ in fetch.BROWSER_IMPERSONATIONS], flaresolverr_url=None)
+    calls = _patch_flaresolverr(monkeypatch, None)
+
+    result = await fetch.resilient_fetch("https://x/r")
+
+    assert result is None
+    assert calls == []
+
+
+@pytest.mark.asyncio
+async def test_no_flaresolverr_on_hard_error(monkeypatch):
+    _patch_responses(monkeypatch, [_FakeResponse(404)], flaresolverr_url="http://flaresolverr:8191")
+    calls = _patch_flaresolverr(monkeypatch, None)
+
+    result = await fetch.resilient_fetch("https://x/r")
+
+    assert result is None
+    assert calls == []  # a 404 is a hard error, not a block
+
+
+@pytest.mark.asyncio
+async def test_no_flaresolverr_for_images(monkeypatch):
+    # allow_flaresolverr=False (as the image path passes) must skip the browser escalation.
+    _patch_responses(
+        monkeypatch,
+        [_FakeResponse(403) for _ in fetch.BROWSER_IMPERSONATIONS],
+        flaresolverr_url="http://flaresolverr:8191",
+    )
+    calls = _patch_flaresolverr(monkeypatch, None)
+
+    result = await fetch.resilient_fetch("https://x/r", allow_flaresolverr=False)
+
+    assert result is None
+    assert calls == []
+
+
+@pytest.mark.asyncio
+async def test_no_flaresolverr_for_head(monkeypatch):
+    # HEAD requests read no body; FlareSolverr (HTML-only) is pointless, so it must be skipped.
+    _patch_responses(
+        monkeypatch,
+        [_FakeResponse(403) for _ in fetch.BROWSER_IMPERSONATIONS],
+        flaresolverr_url="http://flaresolverr:8191",
+    )
+    calls = _patch_flaresolverr(monkeypatch, None)
+
+    result = await fetch.resilient_fetch("https://x/r", method="HEAD")
+
+    assert result is None
+    assert calls == []
+
+
+@pytest.mark.asyncio
+async def test_flaresolverr_failure_degrades_gracefully(monkeypatch):
+    _patch_responses(
+        monkeypatch,
+        [_FakeResponse(403) for _ in fetch.BROWSER_IMPERSONATIONS],
+        flaresolverr_url="http://flaresolverr:8191",
+    )
+    calls = _patch_flaresolverr(monkeypatch, None)  # solve() returns None (unreachable / unsolved)
+
+    result = await fetch.resilient_fetch("https://x/r")
+
+    assert result is None
+    assert len(calls) == 1  # it tried, then gave up cleanly

tests/unit_tests/pkgs/safehttp/test_flaresolverr.py

@@ -0,0 +1,112 @@
+import httpx
+import pytest
+
+from mealie.pkgs.safehttp import flaresolverr
+
+
+class _FakeHTTPResponse:
+    def __init__(self, status_code: int, json_data=None, raise_json: bool = False):
+        self.status_code = status_code
+        self._json_data = json_data
+        self._raise_json = raise_json
+
+    def json(self):
+        if self._raise_json:
+            raise ValueError("not json")
+        return self._json_data
+
+
+class _FakePostClient:
+    """Stands in for httpx.AsyncClient, capturing the POST and returning a scripted response."""
+
+    last_call: dict = {}
+
+    def __init__(self, response=None, exc=None):
+        self._response = response
+        self._exc = exc
+
+    async def __aenter__(self):
+        return self
+
+    async def __aexit__(self, *exc):
+        return False
+
+    async def post(self, endpoint: str, json: dict):
+        _FakePostClient.last_call = {"endpoint": endpoint, "json": json}
+        if self._exc:
+            raise self._exc
+        return self._response
+
+
+def _patch_client(monkeypatch, *, response=None, exc=None):
+    monkeypatch.setattr(flaresolverr.httpx, "AsyncClient", lambda *a, **k: _FakePostClient(response=response, exc=exc))
+
+
+_OK_ENVELOPE = {
+    "status": "ok",
+    "solution": {
+        "url": "https://x/final",
+        "status": 200,
+        "response": "<html>solved</html>",
+        "cookies": [{"name": "cf_clearance", "value": "abc"}],
+        "userAgent": "Mozilla/5.0 ...",
+    },
+}
+
+
+@pytest.mark.asyncio
+async def test_solve_success_parses_solution(monkeypatch):
+    _patch_client(monkeypatch, response=_FakeHTTPResponse(200, _OK_ENVELOPE))
+
+    solution = await flaresolverr.solve("http://flaresolverr:8191", "https://x/r", 45)
+
+    assert solution is not None
+    assert solution.html == "<html>solved</html>"
+    assert solution.status_code == 200
+    assert solution.url == "https://x/final"
+    assert solution.cookies == [{"name": "cf_clearance", "value": "abc"}]
+    assert solution.user_agent == "Mozilla/5.0 ..."
+
+
+@pytest.mark.asyncio
+async def test_solve_builds_correct_request(monkeypatch):
+    _patch_client(monkeypatch, response=_FakeHTTPResponse(200, _OK_ENVELOPE))
+
+    # trailing slash on base URL should be normalized
+    await flaresolverr.solve("http://flaresolverr:8191/", "https://x/r", 30)
+
+    call = _FakePostClient.last_call
+    assert call["endpoint"] == "http://flaresolverr:8191/v1"
+    assert call["json"]["cmd"] == "request.get"
+    assert call["json"]["url"] == "https://x/r"
+    assert call["json"]["maxTimeout"] == 30_000  # seconds -> milliseconds
+
+
+@pytest.mark.asyncio
+async def test_solve_returns_none_on_http_error(monkeypatch):
+    _patch_client(monkeypatch, response=_FakeHTTPResponse(500, {}))
+    assert await flaresolverr.solve("http://fs:8191", "https://x/r", 30) is None
+
+
+@pytest.mark.asyncio
+async def test_solve_returns_none_on_error_status(monkeypatch):
+    _patch_client(monkeypatch, response=_FakeHTTPResponse(200, {"status": "error", "message": "challenge failed"}))
+    assert await flaresolverr.solve("http://fs:8191", "https://x/r", 30) is None
+
+
+@pytest.mark.asyncio
+async def test_solve_returns_none_on_empty_solution(monkeypatch):
+    _patch_client(monkeypatch, response=_FakeHTTPResponse(200, {"status": "ok", "solution": {"response": ""}}))
+    assert await flaresolverr.solve("http://fs:8191", "https://x/r", 30) is None
+
+
+@pytest.mark.asyncio
+async def test_solve_returns_none_on_non_json(monkeypatch):
+    _patch_client(monkeypatch, response=_FakeHTTPResponse(200, raise_json=True))
+    assert await flaresolverr.solve("http://fs:8191", "https://x/r", 30) is None
+
+
+@pytest.mark.asyncio
+async def test_solve_returns_none_on_connection_error(monkeypatch):
+    _patch_client(monkeypatch, exc=httpx.ConnectError("unreachable"))
+    assert await flaresolverr.solve("http://fs:8191", "https://x/r", 30) is None

tests/unit_tests/test_config.py

@@ -5,6 +5,7 @@ from pathlib import Path
 from typing import Any
 
 import pytest
+from pydantic import ValidationError
 
 from mealie.core.config import get_app_settings
 from mealie.core.settings.settings import AppSettings, determine_secrets
@@ -395,6 +396,51 @@ def test_sensitive_settings_mask(monkeypatch: pytest.MonkeyPatch):
         assert settings_json[setting] == "*****"
 
 
+_SCRAPER_URL_FIELDS = ["SCRAPER_PROXY_URL", "SCRAPER_FLARESOLVERR_URL"]
+
+
+@pytest.mark.parametrize("field", _SCRAPER_URL_FIELDS)
+@pytest.mark.parametrize(
+    "value",
+    [
+        "flaresolverr:8191",  # missing scheme
+        "192.168.1.5:8191",  # bare host:port
+        "just-a-hostname",  # no scheme, no port
+    ],
+)
+def test_scraper_url_rejects_missing_scheme(field: str, value: str, monkeypatch: pytest.MonkeyPatch):
+    monkeypatch.setenv(field, value)
+    get_app_settings.cache_clear()
+
+    with pytest.raises(ValidationError):
+        get_app_settings()
+
+
+@pytest.mark.parametrize("field", _SCRAPER_URL_FIELDS)
+@pytest.mark.parametrize(
+    "value",
+    [
+        "http://flaresolverr:8191",
+        "https://fs.example.com:8191/",
+        "http://user:pass@host:8080",  # userinfo is allowed
+        "socks5://host:1080",  # non-http schemes (valid for proxies) are not rejected
+    ],
+)
+def test_scraper_url_accepts_valid(field: str, value: str, monkeypatch: pytest.MonkeyPatch):
+    monkeypatch.setenv(field, value)
+    get_app_settings.cache_clear()
+
+    assert getattr(get_app_settings(), field) == value
+
+
+@pytest.mark.parametrize("field", _SCRAPER_URL_FIELDS)
+def test_scraper_url_allows_unset(field: str, monkeypatch: pytest.MonkeyPatch):
+    monkeypatch.delenv(field, raising=False)
+    get_app_settings.cache_clear()
+
+    assert getattr(get_app_settings(), field) is None
+
+
 class DetermineSecretsTests:
     def test_non_production_returns_fixed_key(self, tmp_path: Path):
         result = determine_secrets(tmp_path, ".secret", production=False)