diff --git a/docs/docs/documentation/getting-started/faq.md b/docs/docs/documentation/getting-started/faq.md index c03bcb367..e7e50e3e7 100644 --- a/docs/docs/documentation/getting-started/faq.md +++ b/docs/docs/documentation/getting-started/faq.md @@ -263,6 +263,22 @@ ## Technical Considerations +??? question "Why do some recipe imports fail or get blocked?" + + ### Why do some recipe imports fail or get blocked? + + Some recipe websites sit behind bot-protection (e.g. Cloudflare) that can block Mealie from + fetching the page or its image. Mealie already impersonates real browsers and rotates between + several of them to get around most of this automatically, with no configuration needed. + + If particular sites still fail to import, you can optionally route scraping through a **proxy** + with a better IP reputation, and/or fall back to a self-hosted **FlareSolverr** instance that + uses a real browser to solve challenges. Both are opt-in and configured via environment + variables: + + - [Backend Config - Recipe Scraper](./installation/backend-config.md#recipe-scraper) + + ??? question "Why setup Email?" ### Why setup Email? diff --git a/docs/docs/documentation/getting-started/installation/backend-config.md b/docs/docs/documentation/getting-started/installation/backend-config.md index 1aaaeb25b..7a8650127 100644 --- a/docs/docs/documentation/getting-started/installation/backend-config.md +++ b/docs/docs/documentation/getting-started/installation/backend-config.md @@ -128,6 +128,68 @@ Mealie supports various integrations using OpenAI. For more information, check o |-------------------------------------------------------------------------|:-----------:|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | OPENAI_CUSTOM_PROMPT_DIR
:octicons-tag-24: v3.10.0 | None. | Path to custom prompt files. Only existing files in your custom directory will override the defaults; any missing or empty custom files will automatically fall back to the system defaults. See https://github.com/mealie-recipes/mealie/tree/mealie-next/mealie/services/openai/prompts for expected file names. | +### Recipe Scraper + +When you import a recipe from a URL, Mealie fetches the page (and its image) before parsing it. Many +sites sit behind bot-protection (e.g. Cloudflare) that can block these requests. Out of the box Mealie +mitigates this by impersonating real browsers' TLS fingerprints and rotating between several of them, +which is enough for most sites and requires no configuration. If you still run into sites that block +imports, the **opt-in** settings below add two further layers. + +| Variables | Default | Description | +| ---------------------------- | :-----: | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| SCRAPER_PROXY_URL | None | Optional proxy for all outbound scraping and image requests (e.g. `http://user:pass@host:port`). Routing through an IP with a better reputation helps bypass IP-based blocks. Unset disables it. | +| SCRAPER_PROXY_MODE | always | How the proxy is used (when `SCRAPER_PROXY_URL` is set): `always` routes every request through it; `fallback` tries a direct request first and only retries through the proxy when a block is detected. Any unrecognized value falls back to `always`. | +| SCRAPER_FLARESOLVERR_URL | None | Optional base URL of a self-hosted [FlareSolverr](https://github.com/FlareSolverr/FlareSolverr) instance (e.g. `http://flaresolverr:8191`). Used only as a last resort to solve JS/Cloudflare challenges. Unset disables it. | +| SCRAPER_FLARESOLVERR_TIMEOUT | 60 | Maximum seconds FlareSolverr may spend solving a single challenge before giving up. | + +#### How Mealie fetches a page + +For each import Mealie escalates only as far as it needs to, stopping at the first step that succeeds: + +1. **Direct fetch** with rotating browser TLS impersonations — always on, no configuration. +2. **Proxy** — if `SCRAPER_PROXY_URL` is set (see modes below). +3. **FlareSolverr** — if `SCRAPER_FLARESOLVERR_URL` is set, and only when the page is still blocked. + +Steps 2 and 3 are opt-in, so a default install uses only step 1. Genuine "not found" responses (e.g. +`404`) are treated as real errors and are never retried through the later steps. + +#### Proxy + +Most IP-based blocks trigger on the very first request, so `always` mode (the default when a proxy is +set) is recommended — it routes every request through the proxy from the start. Use `fallback` mode if +you're on a **metered proxy** and want to avoid paying for requests that would have succeeded directly; +Mealie will then only route through the proxy after a direct attempt is blocked. The proxy applies to +both the recipe page and its image download. + +#### FlareSolverr + +FlareSolverr runs a real headless browser to solve challenges that TLS impersonation alone can't. +**Mealie neither ships nor manages it** — you host it yourself and point Mealie at it. Because it +returns rendered **HTML**, it is only used for the recipe page; **image downloads never use +FlareSolverr** and continue to rely on the direct/proxy path. + +Run it as a sidecar container and set `SCRAPER_FLARESOLVERR_URL` to its address: + +```yaml +services: + mealie: + image: ghcr.io/mealie-recipes/mealie:latest + environment: + SCRAPER_FLARESOLVERR_URL: http://flaresolverr:8191 + # optional; default shown + # SCRAPER_FLARESOLVERR_TIMEOUT: 60 + + flaresolverr: + image: ghcr.io/flaresolverr/flaresolverr:latest + restart: unless-stopped + # No ports need to be published — Mealie reaches it over the internal Docker network. +``` + +Because FlareSolverr drives a browser, requests through it are much slower (seconds) and far more +resource-intensive than a direct fetch. That's why Mealie only falls back to it when a page is actually +blocked, rather than using it for every import. + ### Theming Setting the following environmental variables will change the theme of the frontend. Note that the themes are the same for all users. This is a break-change when migration from v0.x.x -> 1.x.x. diff --git a/mealie/core/settings/settings.py b/mealie/core/settings/settings.py index 477f73f5b..5d4a4568c 100644 --- a/mealie/core/settings/settings.py +++ b/mealie/core/settings/settings.py @@ -2,8 +2,10 @@ import logging import os import secrets from datetime import UTC, datetime +from enum import StrEnum from pathlib import Path from typing import Annotated, Literal, NamedTuple +from urllib.parse import urlparse from dateutil.tz import tzlocal from pydantic import PlainSerializer, field_validator @@ -15,6 +17,20 @@ from .db_providers import AbstractDBProvider, db_provider_factory from .static import PACKAGE_DIR +class ScraperProxyMode(StrEnum): + """How the scraper uses a configured proxy.""" + + always = "always" + """Route every request through the proxy (IP-based blocks trigger on the first request).""" + fallback = "fallback" + """Try direct first; only retry through the proxy when a block is detected.""" + + @classmethod + def _missing_(cls, value: object) -> "ScraperProxyMode": + # Default any unrecognized configuration value to the safest, most useful mode. + return cls.always + + class ScheduleTime(NamedTuple): hour: int minute: int @@ -424,6 +440,42 @@ class AppSettings(AppLoggingSettings): files are individually optional, each prompt name will fall back to the default if no custom file exists """ + # =============================================== + # Scraper Configuration + + SCRAPER_PROXY_URL: str | None = None + """Optional proxy for all outbound recipe/image scraping requests (e.g. ``http://user:pass@host:port``). + Routing through a proxy with a better IP reputation helps bypass IP-based bot blocks. Unset disables it.""" + + SCRAPER_PROXY_MODE: ScraperProxyMode = ScraperProxyMode.always + """How the scraper uses ``SCRAPER_PROXY_URL`` (when set): ``always`` routes every request through the + proxy (recommended, since IP-based blocks trigger on the first request); ``fallback`` tries a direct + request first and only retries through the proxy when a block is detected (useful for metered proxies). + Any unrecognized value falls back to ``always``.""" + + SCRAPER_FLARESOLVERR_URL: str | None = None + """Optional base URL of a self-hosted FlareSolverr instance (e.g. ``http://flaresolverr:8191``). When + set, HTML scrapes that remain blocked after the direct/proxy attempts are retried through FlareSolverr, + which drives a real browser to solve JS/Cloudflare challenges. Mealie neither ships nor manages it. + Image downloads never use it, since FlareSolverr returns HTML rather than binary content.""" + + SCRAPER_FLARESOLVERR_TIMEOUT: int = 60 + """Maximum seconds FlareSolverr may spend solving a single challenge before giving up.""" + + @field_validator("SCRAPER_PROXY_URL", "SCRAPER_FLARESOLVERR_URL") + @classmethod + def validate_scraper_url(cls, v: str | None, info) -> str | None: + """Fail fast at startup if a scraper URL is set but malformed (e.g. missing the scheme).""" + if not v: + return v + + parsed = urlparse(v) + if not parsed.scheme or not parsed.netloc: + raise ValueError( + f"{info.field_name} must be a full URL including scheme and host, e.g. 'http://host:port' (got '{v}')" + ) + return v + # =============================================== # Web Concurrency diff --git a/mealie/pkgs/safehttp/__init__.py b/mealie/pkgs/safehttp/__init__.py index 02c544414..46eb9d56e 100644 --- a/mealie/pkgs/safehttp/__init__.py +++ b/mealie/pkgs/safehttp/__init__.py @@ -1,7 +1,19 @@ +from .fetch import ( + BROWSER_IMPERSONATIONS, + SCRAPER_TIMEOUT, + FetchResult, + ForceTimeoutException, + resilient_fetch, +) from .transport import AsyncSafeTransport, ForcedTimeoutException, InvalidDomainError __all__ = [ "AsyncSafeTransport", "ForcedTimeoutException", "InvalidDomainError", + "BROWSER_IMPERSONATIONS", + "SCRAPER_TIMEOUT", + "FetchResult", + "ForceTimeoutException", + "resilient_fetch", ] diff --git a/mealie/pkgs/safehttp/fetch.py b/mealie/pkgs/safehttp/fetch.py new file mode 100644 index 000000000..7ae7ed920 --- /dev/null +++ b/mealie/pkgs/safehttp/fetch.py @@ -0,0 +1,313 @@ +import asyncio +import random +import time +from dataclasses import dataclass + +import httpx +from httpx import AsyncClient + +from mealie.core.config import get_app_settings +from mealie.core.root_logger import get_logger +from mealie.core.settings.settings import ScraperProxyMode + +from . import flaresolverr +from .transport import AsyncSafeTransport + +SCRAPER_TIMEOUT = 15 + +# Overall wall-clock budget for a single fetch, across all impersonation attempts and backoffs. +# Bounds worst-case latency when a site repeatedly blocks or stalls us. +SCRAPER_TOTAL_TIMEOUT = 45 + +BROWSER_IMPERSONATIONS = [ + "chrome", + "firefox", + "safari", + "edge", +] + +NON_CHALLENGE_4XX = frozenset({404, 410}) +RATE_LIMIT_STATUS_CODES = frozenset({429, 503}) + +# Substrings found in the bodies of bot-challenge/interstitial pages that are served with a +# 200 status (so status/header checks alone would treat them as a successful fetch). Kept +# specific to anti-bot infrastructure identifiers to avoid false positives on real content. +_CHALLENGE_BODY_MARKERS: tuple[bytes, ...] = ( + b"__cf_chl", + b"cf-browser-verification", + b"/cdn-cgi/challenge-platform", + b"challenges.cloudflare.com", + b"_incapsula_resource", + b"distil_r_captcha", + b"px-captcha", + b"perimeterx", + b"datadome", +) +_CHALLENGE_BODY_SAMPLE = 4096 + +_BASE_BACKOFF = 1.0 +_MAX_BACKOFF = 5.0 +_BACKOFF_JITTER = 0.5 + +logger = get_logger() + + +class ForceTimeoutException(Exception): + """Raised when reading a response body exceeds the fetch timeout.""" + + +@dataclass +class FetchResult: + """The outcome of a resilient fetch, decoupled from the (now-closed) streaming response.""" + + content: bytes + status_code: int + url: str + headers: httpx.Headers + encoding: str | None + + @property + def text(self) -> str: + # Mirrors the decoding behavior of requests' `text` property. + try: + return str(self.content, self.encoding, errors="replace") # type: ignore[arg-type] + except (LookupError, TypeError): + # LookupError: unknown encoding name. TypeError: encoding is None. + return str(self.content, errors="replace") + + +def is_challenge_status(status_code: int) -> bool: + if status_code == 503: + return True + if 400 <= status_code < 500: + return status_code not in NON_CHALLENGE_4XX + return False + + +def headers_indicate_challenge(headers: httpx.Headers) -> bool: + # Cloudflare sets `cf-mitigated: challenge` on interstitial/challenge responses, + # which can otherwise carry a 200 status. + return "cf-mitigated" in headers + + +def body_indicates_challenge(content: bytes) -> bool: + sample = content[:_CHALLENGE_BODY_SAMPLE].lower() + return any(marker in sample for marker in _CHALLENGE_BODY_MARKERS) + + +def _build_transport(impersonate: str, proxy: str | None = None) -> AsyncSafeTransport: + kwargs: dict = { + "impersonate": impersonate, + "default_headers": True, + # disable SSL verification since we can handle untrusted data and some sites don't have certs + # (this also covers the proxy connection, so no separate proxy-verify knob is needed) + "verify": False, + } + if proxy: + kwargs["proxy"] = proxy + return AsyncSafeTransport(**kwargs) + + +async def _read_capped(resp: httpx.Response, timeout: int) -> bytes: + """ + Reads a streaming body, aborting if it takes longer than ``timeout`` seconds. + + Mitigates abuse from URLs that serve arbitrarily large or slow content. + """ + content = b"" + start_time = time.monotonic() + async for chunk in resp.aiter_bytes(chunk_size=1024): + content += chunk + if time.monotonic() - start_time > timeout: + raise ForceTimeoutException() + return content + + +async def _sleep_backoff(retry_after: str | None, deadline: float) -> None: + """Sleeps a jittered backoff before the next attempt, never past the overall deadline.""" + delay = _BASE_BACKOFF + if retry_after: + try: + # Retry-After may be a delta-seconds integer; if it's an HTTP-date we ignore it. + delay = max(delay, float(retry_after)) + except ValueError: + pass + + delay = min(delay, _MAX_BACKOFF) + random.uniform(0, _BACKOFF_JITTER) + + remaining = deadline - time.monotonic() + if remaining <= 0: + return + await asyncio.sleep(min(delay, remaining)) + + +async def _attempt( + url: str, + method: str, + timeout: int, + impersonation: str, + read_body: bool, + proxy: str | None, +) -> tuple[FetchResult | None, bool, int, str | None]: + """ + Performs a single fetch attempt with one browser impersonation. + + Returns ``(result, blocked, status_code, retry_after)``: + - ``result`` is set on success (and ``blocked`` is False). + - ``blocked`` is True when the response looks like a bot challenge and rotating to another + fingerprint may help. + - When both ``result`` is None and ``blocked`` is False, the response was a hard error that + rotating won't fix, and the caller should stop. + """ + transport = _build_transport(impersonation, proxy) + async with AsyncClient(transport=transport) as client: + async with client.stream(method, url, timeout=timeout, follow_redirects=True) as resp: + status_code = resp.status_code + retry_after = resp.headers.get("Retry-After") + + blocked = is_challenge_status(status_code) or headers_indicate_challenge(resp.headers) + + if blocked: + logger.debug(f'Challenge/block detected (status={status_code}) with impersonation "{impersonation}"') + return None, True, status_code, retry_after + + if status_code >= 400: + # A genuine client/server error (e.g. 404, 410, 500) that a different fingerprint + # won't resolve. Stop rotating. + logger.debug(f'Error status code {status_code} with impersonation "{impersonation}"') + return None, False, status_code, retry_after + + content = b"" + if read_body: + content = await _read_capped(resp, timeout) + if body_indicates_challenge(content): + logger.debug(f'Challenge page body detected with impersonation "{impersonation}"') + return None, True, status_code, retry_after + + result = FetchResult( + content=content, + status_code=status_code, + url=str(resp.url), + headers=resp.headers, + encoding=resp.encoding, + ) + return result, False, status_code, retry_after + + +async def _rotate( + url: str, + method: str, + timeout: int, + read_body: bool, + proxy: str | None, + deadline: float, +) -> tuple[FetchResult | None, bool]: + """ + Cycles through browser impersonations (in randomized order) for a single egress path + (direct or via ``proxy``). + + Returns ``(result, blocked)``. ``blocked`` is True only when every impersonation was rejected + by a challenge/block -- i.e. the failure might be worth escalating (e.g. to a proxy). It is + False on success or on a hard error, where escalation wouldn't help. + """ + impersonations = list(BROWSER_IMPERSONATIONS) + random.shuffle(impersonations) + + for index, impersonation in enumerate(impersonations): + if time.monotonic() >= deadline: + logger.debug("Scraper budget exhausted before trying all impersonations") + break + + logger.debug(f'Trying browser impersonation: "{impersonation}"') + result, blocked, status_code, retry_after = await _attempt( + url, method, timeout, impersonation, read_body, proxy + ) + + if result is not None: + return result, False + + if not blocked: + # Hard error; rotating fingerprints won't help. + return None, False + + is_last = index == len(impersonations) - 1 + if not is_last and status_code in RATE_LIMIT_STATUS_CODES: + await _sleep_backoff(retry_after, deadline) + + return None, True + + +def _solution_to_result(solution: flaresolverr.FlareSolverrSolution) -> FetchResult: + return FetchResult( + content=solution.html.encode("utf-8", errors="replace"), + status_code=solution.status_code, + url=solution.url, + headers=httpx.Headers({"content-type": "text/html; charset=utf-8"}), + encoding="utf-8", + ) + + +async def resilient_fetch( + url: str, + *, + method: str = "GET", + timeout: int = SCRAPER_TIMEOUT, + allow_flaresolverr: bool = True, +) -> FetchResult | None: + """ + Fetches a URL while cycling through browser TLS impersonations (via httpx-curl-cffi) to + bypass bot-detection systems that fingerprint the TLS handshake (JA3/JA4), such as Cloudflare. + + Impersonations are tried in a randomized order. On a detected challenge/block (a challenge + status code, a ``cf-mitigated`` header, or challenge markers in an otherwise-200 body) the + next fingerprint is tried, with a short jittered backoff for rate-limit statuses. A genuine + error status (e.g. 404) stops the rotation immediately, since a new fingerprint won't help. + + When ``SCRAPER_PROXY_URL`` is configured, requests egress through it: in ``always`` mode every + request uses the proxy; in ``fallback`` mode a direct attempt is made first and the proxy is + only used to retry when every direct impersonation was blocked. + + As a last resort, if the fetch is still blocked and ``SCRAPER_FLARESOLVERR_URL`` is configured, + the request is escalated to FlareSolverr (a headless browser). This only applies to HTML fetches + (``allow_flaresolverr`` and a body-returning method), since FlareSolverr returns HTML, not the + binary content an image download needs. + + The whole operation is bounded by ``SCRAPER_TOTAL_TIMEOUT``, and each attempt's body read is + bounded by ``timeout`` seconds, to mitigate abuse from URLs that serve arbitrarily large content. + + Returns a ``FetchResult`` for the first successful response, or ``None`` if every impersonation + was blocked, the server returned a hard error, or the budget was exhausted. + """ + logger.debug(f"Fetching URL: {url}") + + read_body = method.upper() != "HEAD" + deadline = time.monotonic() + SCRAPER_TOTAL_TIMEOUT + + settings = get_app_settings() + proxy = settings.SCRAPER_PROXY_URL or None + proxy_first = bool(proxy) and settings.SCRAPER_PROXY_MODE == ScraperProxyMode.always + + result, blocked = await _rotate(url, method, timeout, read_body, proxy if proxy_first else None, deadline) + if result is not None: + return result + + # In `fallback` mode, escalate to the proxy only when a direct attempt was blocked (not on a + # hard error, and not if we already used the proxy above). + if blocked and proxy and not proxy_first: + logger.debug("Direct fetch blocked; retrying through configured proxy") + result, blocked = await _rotate(url, method, timeout, read_body, proxy, deadline) + if result is not None: + return result + + # Final escalation: a real browser via FlareSolverr. HTML-only, and only when still blocked. + # Note: the impersonation rotation above always runs first, so its SSRF guard (which rejects + # private target IPs) has already vetted `url` before we hand it to FlareSolverr. + if blocked and read_body and allow_flaresolverr and settings.SCRAPER_FLARESOLVERR_URL: + logger.debug("Fetch still blocked; escalating to FlareSolverr") + solution = await flaresolverr.solve( + settings.SCRAPER_FLARESOLVERR_URL, url, settings.SCRAPER_FLARESOLVERR_TIMEOUT + ) + if solution is not None: + return _solution_to_result(solution) + + return result diff --git a/mealie/pkgs/safehttp/flaresolverr.py b/mealie/pkgs/safehttp/flaresolverr.py new file mode 100644 index 000000000..b60bc4242 --- /dev/null +++ b/mealie/pkgs/safehttp/flaresolverr.py @@ -0,0 +1,76 @@ +from dataclasses import dataclass, field + +import httpx + +from mealie.core.root_logger import get_logger + +logger = get_logger() + +# Extra seconds allowed for the HTTP round-trip on top of FlareSolverr's own solve budget, +# to account for browser startup and network overhead. +_HTTP_OVERHEAD_SECONDS = 15 + + +@dataclass +class FlareSolverrSolution: + """The useful parts of a successful FlareSolverr solve.""" + + html: str + status_code: int + url: str + cookies: list[dict] = field(default_factory=list) + user_agent: str = "" + + +async def solve(base_url: str, url: str, timeout: int) -> FlareSolverrSolution | None: + """ + Asks a FlareSolverr instance to fetch ``url`` with a real (headless) browser, solving any + JS/Cloudflare challenge along the way. + + FlareSolverr is a REST service, not a proxy: we POST a job to ``{base_url}/v1`` and read the + rendered HTML back out of the response envelope. + + Returns ``None`` (never raises) on any failure -- an unreachable or misbehaving FlareSolverr + must only fail this escalation, never break scraping. + """ + endpoint = base_url.rstrip("/") + "/v1" + payload = { + "cmd": "request.get", + "url": url, + "maxTimeout": timeout * 1000, # FlareSolverr expects milliseconds + } + + try: + async with httpx.AsyncClient(timeout=timeout + _HTTP_OVERHEAD_SECONDS) as client: + resp = await client.post(endpoint, json=payload) + except Exception: + logger.exception(f"FlareSolverr request failed for {url}") + return None + + if resp.status_code != 200: + logger.warning(f"FlareSolverr returned HTTP {resp.status_code} for {url}") + return None + + try: + data = resp.json() + except Exception: + logger.exception(f"FlareSolverr returned a non-JSON response for {url}") + return None + + if data.get("status") != "ok": + logger.warning(f"FlareSolverr could not solve {url}: {data.get('message')}") + return None + + solution = data.get("solution") or {} + html = solution.get("response") + if not html: + logger.warning(f"FlareSolverr returned an empty solution for {url}") + return None + + return FlareSolverrSolution( + html=html, + status_code=solution.get("status", 200), + url=solution.get("url", url), + cookies=solution.get("cookies", []), + user_agent=solution.get("userAgent", ""), + ) diff --git a/mealie/services/recipe/recipe_data_service.py b/mealie/services/recipe/recipe_data_service.py index 8b0769edf..5bd4db0b9 100644 --- a/mealie/services/recipe/recipe_data_service.py +++ b/mealie/services/recipe/recipe_data_service.py @@ -3,7 +3,6 @@ import shutil from logging import Logger from pathlib import Path -from httpx import AsyncClient, Response from pydantic import UUID4 from mealie.pkgs import img, safehttp @@ -31,17 +30,18 @@ async def largest_content_len(urls: list[str]) -> tuple[str, int]: max_concurrency = 10 - async def do(client: AsyncClient, url: str) -> Response: - return await client.head(url) + tasks = [safehttp.resilient_fetch(url, method="HEAD") for url in urls] + responses: list[safehttp.FetchResult | None] = await gather_with_concurrency( + max_concurrency, *tasks, ignore_exceptions=True + ) + for response in responses: + if response is None: + continue - async with AsyncClient(transport=safehttp.AsyncSafeTransport(impersonate="chrome")) as client: - tasks = [do(client, url) for url in urls] - responses: list[Response] = await gather_with_concurrency(max_concurrency, *tasks, ignore_exceptions=True) - for response in responses: - len_int = int(response.headers.get("Content-Length", 0)) - if len_int > largest_len: - largest_url = str(response.url) - largest_len = len_int + len_int = int(response.headers.get("Content-Length", 0)) + if len_int > largest_len: + largest_url = response.url + largest_len = len_int return largest_url, largest_len @@ -146,24 +146,23 @@ class RecipeDataService(BaseService): file_name = f"{self.recipe_id!s}.{ext}" file_path = Recipe.directory_from_id(self.recipe_id).joinpath("images", file_name) - async with AsyncClient(transport=safehttp.AsyncSafeTransport(impersonate="chrome")) as client: - try: - r = await client.get(image_url_str) - except Exception: - self.logger.exception("Fatal Image Request Exception") - return None + try: + # FlareSolverr returns HTML, not image bytes, so it can't serve an image download. + r = await safehttp.resilient_fetch(image_url_str, allow_flaresolverr=False) + except Exception: + self.logger.exception("Fatal Image Request Exception") + return None - if r.status_code != 200: - # TODO: Probably should throw an exception in this case as well, but before these changes - # we were returning None if it failed anyways. - return None + if r is None: + # Every impersonation was rejected, or the server returned an error status. + return None - content_type = r.headers.get("content-type", "") + content_type = r.headers.get("content-type", "") - if "image" not in content_type: - self.logger.error(f"Content-Type: {content_type} is not an image") - raise NotAnImageError(f"Content-Type {content_type} is not an image") + if "image" not in content_type: + self.logger.error(f"Content-Type: {content_type} is not an image") + raise NotAnImageError(f"Content-Type {content_type} is not an image") - self.logger.debug(f"File Name Suffix {file_path.suffix}") - self.write_image(r.read(), file_path.suffix) - file_path.unlink(missing_ok=True) + self.logger.debug(f"File Name Suffix {file_path.suffix}") + self.write_image(r.content, file_path.suffix) + file_path.unlink(missing_ok=True) diff --git a/mealie/services/scraper/scraper_strategies.py b/mealie/services/scraper/scraper_strategies.py index 3cf9a1ac0..0d6fd7404 100644 --- a/mealie/services/scraper/scraper_strategies.py +++ b/mealie/services/scraper/scraper_strategies.py @@ -1,7 +1,6 @@ import asyncio import functools import re -import time from abc import ABC, abstractmethod from collections.abc import Awaitable, Callable from pathlib import Path @@ -11,7 +10,6 @@ import bs4 import extruct import yt_dlp from fastapi import HTTPException, status -from httpx import AsyncClient, Response from recipe_scrapers import NoSchemaFoundInWildMode, SchemaScraperFactory, scrape_html from slugify import slugify from w3lib.html import get_base_url @@ -33,14 +31,10 @@ from mealie.services.scraper.scraped_extras import ScrapedExtras from . import cleaner -SCRAPER_TIMEOUT = 15 - -BROWSER_IMPERSONATIONS = [ - "chrome", - "firefox", - "safari", - "edge", -] +# Re-exported for backwards compatibility with existing importers (e.g. recipe route error handling). +SCRAPER_TIMEOUT = safehttp.SCRAPER_TIMEOUT +BROWSER_IMPERSONATIONS = safehttp.BROWSER_IMPERSONATIONS +ForceTimeoutException = safehttp.ForceTimeoutException logger = get_logger() @@ -51,10 +45,6 @@ def _get_yt_dlp_extractors() -> list: return [ie for ie in yt_dlp.extractor.gen_extractors() if ie.working() and not isinstance(ie, GenericIE)] -class ForceTimeoutException(Exception): - pass - - async def safe_scrape_html(url: str) -> str: """ Scrapes the html from a url but will cancel the request @@ -65,74 +55,8 @@ async def safe_scrape_html(url: str) -> str: bot-detection systems that fingerprint the TLS handshake (JA3/JA4), such as Cloudflare. """ - logger.debug(f"Scraping URL: {url}") - - html_bytes = b"" - response: Response | None = None - - for impersonation in BROWSER_IMPERSONATIONS: - logger.debug(f'Trying browser impersonation: "{impersonation}"') - - html_bytes = b"" - response = None - - transport = safehttp.AsyncSafeTransport( - impersonate=impersonation, - default_headers=True, - verify=False, # disable SSL verification since we can handle untrusted data and some sites don't have certs - ) - async with AsyncClient(transport=transport) as client: - async with client.stream( - "GET", - url, - timeout=SCRAPER_TIMEOUT, - follow_redirects=True, - ) as resp: - if resp.status_code == 403: - logger.debug(f'403 Forbidden with impersonation "{impersonation}", trying next') - continue - - if resp.status_code >= 400: - logger.debug(f'Error status code {resp.status_code} with impersonation "{impersonation}"') - break - - start_time = time.time() - - async for chunk in resp.aiter_bytes(chunk_size=1024): - html_bytes += chunk - - if time.time() - start_time > SCRAPER_TIMEOUT: - raise ForceTimeoutException() - - response = resp - break - - if not (response and html_bytes): - return "" - - # ===================================== - # Copied from requests text property - - # Try charset from content-type - encoding = response.encoding - - # Fallback to auto-detected encoding. - if encoding is None: - encoding = response.apparent_encoding - - # Decode unicode from given encoding. - try: - content = str(html_bytes, encoding, errors="replace") - except (LookupError, TypeError): - # A LookupError is raised if the encoding was not found which could - # indicate a misspelling or similar mistake. - # - # A TypeError can be raised if encoding is None - # - # So we try blindly encoding. - content = str(html_bytes, errors="replace") - - return content + result = await safehttp.resilient_fetch(url) + return result.text if result else "" class ABCScraperStrategy(ABC): diff --git a/tests/unit_tests/pkgs/safehttp/__init__.py b/tests/unit_tests/pkgs/safehttp/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/tests/unit_tests/pkgs/safehttp/test_fetch.py b/tests/unit_tests/pkgs/safehttp/test_fetch.py new file mode 100644 index 000000000..5fb7157de --- /dev/null +++ b/tests/unit_tests/pkgs/safehttp/test_fetch.py @@ -0,0 +1,418 @@ +from collections.abc import AsyncIterator +from types import SimpleNamespace + +import httpx +import pytest + +from mealie.pkgs.safehttp import fetch + + +# --------------------------------------------------------------------------- +# Detection helpers +# --------------------------------------------------------------------------- +@pytest.mark.parametrize( + ("status_code", "expected"), + [ + (200, False), + (301, False), + (400, True), + (401, True), + (402, True), + (403, True), + (406, True), + (429, True), + (451, True), + (404, False), # resource genuinely missing -> not worth rotating + (410, False), + (500, False), # server error a new fingerprint won't fix + (502, False), + (503, True), # WAFs use 503 during interstitials + (504, False), + ], +) +def test_is_challenge_status(status_code: int, expected: bool): + assert fetch.is_challenge_status(status_code) is expected + + +def test_headers_indicate_challenge(): + assert fetch.headers_indicate_challenge(httpx.Headers({"cf-mitigated": "challenge"})) is True + # header lookup is case-insensitive + assert fetch.headers_indicate_challenge(httpx.Headers({"CF-Mitigated": "challenge"})) is True + assert fetch.headers_indicate_challenge(httpx.Headers({"server": "cloudflare"})) is False + assert fetch.headers_indicate_challenge(httpx.Headers({})) is False + + +def test_body_indicates_challenge(): + assert fetch.body_indicates_challenge(b"...cf-browser-verification...") is True + # marker matching is case-insensitive + assert fetch.body_indicates_challenge(b"") is True + assert fetch.body_indicates_challenge(b"A normal recipe for cookies") is False + + +def test_body_indicates_challenge_only_samples_head(): + # A marker past the sampled window should not be detected. + body = b"x" * (fetch._CHALLENGE_BODY_SAMPLE + 100) + b"datadome" + assert fetch.body_indicates_challenge(body) is False + + +# --------------------------------------------------------------------------- +# FetchResult.text +# --------------------------------------------------------------------------- +def test_fetch_result_text_decoding(): + assert fetch.FetchResult(b"h\xc3\xa9llo", 200, "http://x", httpx.Headers(), "utf-8").text == "héllo" + # None encoding falls back to a blind decode + assert fetch.FetchResult(b"abc", 200, "http://x", httpx.Headers(), None).text == "abc" + # An unknown encoding name also falls back rather than raising + assert fetch.FetchResult(b"abc", 200, "http://x", httpx.Headers(), "not-real").text == "abc" + + +# --------------------------------------------------------------------------- +# Fakes for the fetch loop +# --------------------------------------------------------------------------- +class _FakeResponse: + def __init__(self, status_code: int, *, headers: dict | None = None, body: bytes = b"", url: str = "https://x/r"): + self.status_code = status_code + self.headers = httpx.Headers(headers or {}) + self.encoding = "utf-8" + self.url = url + self._body = body + + async def aiter_bytes(self, chunk_size: int = 1024) -> AsyncIterator[bytes]: + for i in range(0, len(self._body), chunk_size): + yield self._body[i : i + chunk_size] + + async def __aenter__(self): + return self + + async def __aexit__(self, *exc): + return False + + +class _FakeClient: + def __init__(self, response: _FakeResponse): + self._response = response + + async def __aenter__(self): + return self + + async def __aexit__(self, *exc): + return False + + def stream(self, method: str, url: str, **kwargs): + return self._response + + +def _patch_responses( + monkeypatch, + responses: list[_FakeResponse], + *, + proxy_url: str | None = None, + proxy_mode=fetch.ScraperProxyMode.always, + flaresolverr_url: str | None = None, + flaresolverr_timeout: int = 60, +) -> dict: + """Scripts consecutive attempts to return the given responses. + + Records the number of attempts and the proxy passed to each, and isolates the fetch from real + app settings by injecting the given scraper configuration. + """ + state = {"queue": list(responses), "attempts": 0, "proxies": []} + + def make_client(*args, **kwargs): + state["attempts"] += 1 + return _FakeClient(state["queue"].pop(0)) + + def fake_build_transport(impersonate: str, proxy: str | None = None): + state["proxies"].append(proxy) + return None + + monkeypatch.setattr(fetch, "AsyncClient", make_client) + monkeypatch.setattr(fetch, "_build_transport", fake_build_transport) + monkeypatch.setattr( + fetch, + "get_app_settings", + lambda: SimpleNamespace( + SCRAPER_PROXY_URL=proxy_url, + SCRAPER_PROXY_MODE=proxy_mode, + SCRAPER_FLARESOLVERR_URL=flaresolverr_url, + SCRAPER_FLARESOLVERR_TIMEOUT=flaresolverr_timeout, + ), + ) + return state + + +# --------------------------------------------------------------------------- +# resilient_fetch loop +# --------------------------------------------------------------------------- +@pytest.mark.asyncio +async def test_success_on_first_attempt(monkeypatch): + state = _patch_responses(monkeypatch, [_FakeResponse(200, body=b"recipe")]) + result = await fetch.resilient_fetch("https://x/r") + + assert result is not None + assert result.status_code == 200 + assert result.content == b"recipe" + assert state["attempts"] == 1 + + +@pytest.mark.asyncio +async def test_rotates_through_all_impersonations_on_block(monkeypatch): + state = _patch_responses(monkeypatch, [_FakeResponse(403) for _ in fetch.BROWSER_IMPERSONATIONS]) + result = await fetch.resilient_fetch("https://x/r") + + assert result is None + assert state["attempts"] == len(fetch.BROWSER_IMPERSONATIONS) + + +@pytest.mark.asyncio +async def test_rotates_past_200_challenge_body(monkeypatch): + state = _patch_responses( + monkeypatch, + [ + _FakeResponse(200, body=b"__cf_chl just a moment"), + _FakeResponse(200, body=b"real recipe"), + ], + ) + result = await fetch.resilient_fetch("https://x/r") + + assert result is not None + assert result.content == b"real recipe" + assert state["attempts"] == 2 + + +@pytest.mark.asyncio +async def test_hard_error_stops_immediately(monkeypatch): + state = _patch_responses(monkeypatch, [_FakeResponse(404) for _ in fetch.BROWSER_IMPERSONATIONS]) + result = await fetch.resilient_fetch("https://x/r") + + assert result is None + assert state["attempts"] == 1 # did not rotate on a 404 + + +@pytest.mark.asyncio +async def test_head_request_skips_body(monkeypatch): + # A body that *would* look like a challenge is ignored for HEAD (no body is read). + state = _patch_responses(monkeypatch, [_FakeResponse(200, body=b"__cf_chl")]) + result = await fetch.resilient_fetch("https://x/r", method="HEAD") + + assert result is not None + assert result.content == b"" + assert state["attempts"] == 1 + + +@pytest.mark.asyncio +async def test_rate_limit_triggers_backoff_then_succeeds(monkeypatch): + slept: list[float] = [] + + async def fake_sleep(delay: float): + slept.append(delay) + + monkeypatch.setattr(fetch.asyncio, "sleep", fake_sleep) + state = _patch_responses( + monkeypatch, + [_FakeResponse(429, headers={"Retry-After": "2"}), _FakeResponse(200, body=b"ok")], + ) + + result = await fetch.resilient_fetch("https://x/r") + + assert result is not None + assert result.content == b"ok" + assert state["attempts"] == 2 + assert len(slept) == 1 + assert slept[0] >= 2.0 # honored Retry-After + + +@pytest.mark.asyncio +async def test_no_backoff_on_plain_403(monkeypatch): + slept: list[float] = [] + + async def fake_sleep(delay: float): + slept.append(delay) + + monkeypatch.setattr(fetch.asyncio, "sleep", fake_sleep) + _patch_responses(monkeypatch, [_FakeResponse(403), _FakeResponse(200, body=b"ok")]) + + result = await fetch.resilient_fetch("https://x/r") + + assert result is not None + assert slept == [] # 403 rotates immediately, no rate-limit backoff + + +# --------------------------------------------------------------------------- +# Proxy behavior +# --------------------------------------------------------------------------- +@pytest.mark.asyncio +async def test_no_proxy_when_unconfigured(monkeypatch): + state = _patch_responses(monkeypatch, [_FakeResponse(200, body=b"ok")], proxy_url=None) + result = await fetch.resilient_fetch("https://x/r") + + assert result is not None + assert state["proxies"] == [None] + + +@pytest.mark.asyncio +async def test_always_mode_proxies_first_request(monkeypatch): + state = _patch_responses( + monkeypatch, + [_FakeResponse(200, body=b"ok")], + proxy_url="http://proxy:8080", + proxy_mode=fetch.ScraperProxyMode.always, + ) + result = await fetch.resilient_fetch("https://x/r") + + assert result is not None + # proxy is used from the very first request, even though it succeeded (no block) + assert state["proxies"] == ["http://proxy:8080"] + + +@pytest.mark.asyncio +async def test_fallback_mode_direct_first_then_proxy_on_block(monkeypatch): + responses = [_FakeResponse(403) for _ in fetch.BROWSER_IMPERSONATIONS] # direct all blocked + responses.append(_FakeResponse(200, body=b"ok")) # first proxied attempt succeeds + state = _patch_responses( + monkeypatch, + responses, + proxy_url="http://proxy:8080", + proxy_mode=fetch.ScraperProxyMode.fallback, + ) + result = await fetch.resilient_fetch("https://x/r") + + assert result is not None + assert result.content == b"ok" + # direct rotation used no proxy; escalation used the proxy + n = len(fetch.BROWSER_IMPERSONATIONS) + assert state["proxies"][:n] == [None] * n + assert state["proxies"][n] == "http://proxy:8080" + + +@pytest.mark.asyncio +async def test_fallback_mode_does_not_escalate_on_hard_error(monkeypatch): + state = _patch_responses( + monkeypatch, + [_FakeResponse(404) for _ in fetch.BROWSER_IMPERSONATIONS], + proxy_url="http://proxy:8080", + proxy_mode=fetch.ScraperProxyMode.fallback, + ) + result = await fetch.resilient_fetch("https://x/r") + + assert result is None + # a 404 is a hard error: no rotation, and no proxy escalation + assert state["proxies"] == [None] + + +@pytest.mark.asyncio +async def test_always_mode_does_not_double_escalate(monkeypatch): + # Every attempt blocked in always-mode should not trigger a second (proxy) rotation. + state = _patch_responses( + monkeypatch, + [_FakeResponse(403) for _ in fetch.BROWSER_IMPERSONATIONS], + proxy_url="http://proxy:8080", + proxy_mode=fetch.ScraperProxyMode.always, + ) + result = await fetch.resilient_fetch("https://x/r") + + assert result is None + assert state["attempts"] == len(fetch.BROWSER_IMPERSONATIONS) + assert all(p == "http://proxy:8080" for p in state["proxies"]) + + +# --------------------------------------------------------------------------- +# FlareSolverr escalation +# --------------------------------------------------------------------------- +def _patch_flaresolverr(monkeypatch, solution): + """Records calls to flaresolverr.solve and returns the given solution (or None).""" + calls: list[tuple] = [] + + async def fake_solve(base_url, url, timeout): + calls.append((base_url, url, timeout)) + return solution + + monkeypatch.setattr(fetch.flaresolverr, "solve", fake_solve) + return calls + + +@pytest.mark.asyncio +async def test_escalates_to_flaresolverr_when_blocked(monkeypatch): + _patch_responses( + monkeypatch, + [_FakeResponse(403) for _ in fetch.BROWSER_IMPERSONATIONS], + flaresolverr_url="http://flaresolverr:8191", + ) + solution = fetch.flaresolverr.FlareSolverrSolution(html="solved", status_code=200, url="https://x/r") + calls = _patch_flaresolverr(monkeypatch, solution) + + result = await fetch.resilient_fetch("https://x/r") + + assert result is not None + assert result.text == "solved" + assert len(calls) == 1 + assert calls[0] == ("http://flaresolverr:8191", "https://x/r", 60) + + +@pytest.mark.asyncio +async def test_no_flaresolverr_when_unconfigured(monkeypatch): + _patch_responses(monkeypatch, [_FakeResponse(403) for _ in fetch.BROWSER_IMPERSONATIONS], flaresolverr_url=None) + calls = _patch_flaresolverr(monkeypatch, None) + + result = await fetch.resilient_fetch("https://x/r") + + assert result is None + assert calls == [] + + +@pytest.mark.asyncio +async def test_no_flaresolverr_on_hard_error(monkeypatch): + _patch_responses(monkeypatch, [_FakeResponse(404)], flaresolverr_url="http://flaresolverr:8191") + calls = _patch_flaresolverr(monkeypatch, None) + + result = await fetch.resilient_fetch("https://x/r") + + assert result is None + assert calls == [] # a 404 is a hard error, not a block + + +@pytest.mark.asyncio +async def test_no_flaresolverr_for_images(monkeypatch): + # allow_flaresolverr=False (as the image path passes) must skip the browser escalation. + _patch_responses( + monkeypatch, + [_FakeResponse(403) for _ in fetch.BROWSER_IMPERSONATIONS], + flaresolverr_url="http://flaresolverr:8191", + ) + calls = _patch_flaresolverr(monkeypatch, None) + + result = await fetch.resilient_fetch("https://x/r", allow_flaresolverr=False) + + assert result is None + assert calls == [] + + +@pytest.mark.asyncio +async def test_no_flaresolverr_for_head(monkeypatch): + # HEAD requests read no body; FlareSolverr (HTML-only) is pointless, so it must be skipped. + _patch_responses( + monkeypatch, + [_FakeResponse(403) for _ in fetch.BROWSER_IMPERSONATIONS], + flaresolverr_url="http://flaresolverr:8191", + ) + calls = _patch_flaresolverr(monkeypatch, None) + + result = await fetch.resilient_fetch("https://x/r", method="HEAD") + + assert result is None + assert calls == [] + + +@pytest.mark.asyncio +async def test_flaresolverr_failure_degrades_gracefully(monkeypatch): + _patch_responses( + monkeypatch, + [_FakeResponse(403) for _ in fetch.BROWSER_IMPERSONATIONS], + flaresolverr_url="http://flaresolverr:8191", + ) + calls = _patch_flaresolverr(monkeypatch, None) # solve() returns None (unreachable / unsolved) + + result = await fetch.resilient_fetch("https://x/r") + + assert result is None + assert len(calls) == 1 # it tried, then gave up cleanly diff --git a/tests/unit_tests/pkgs/safehttp/test_flaresolverr.py b/tests/unit_tests/pkgs/safehttp/test_flaresolverr.py new file mode 100644 index 000000000..f2f79aff8 --- /dev/null +++ b/tests/unit_tests/pkgs/safehttp/test_flaresolverr.py @@ -0,0 +1,112 @@ +import httpx +import pytest + +from mealie.pkgs.safehttp import flaresolverr + + +class _FakeHTTPResponse: + def __init__(self, status_code: int, json_data=None, raise_json: bool = False): + self.status_code = status_code + self._json_data = json_data + self._raise_json = raise_json + + def json(self): + if self._raise_json: + raise ValueError("not json") + return self._json_data + + +class _FakePostClient: + """Stands in for httpx.AsyncClient, capturing the POST and returning a scripted response.""" + + last_call: dict = {} + + def __init__(self, response=None, exc=None): + self._response = response + self._exc = exc + + async def __aenter__(self): + return self + + async def __aexit__(self, *exc): + return False + + async def post(self, endpoint: str, json: dict): + _FakePostClient.last_call = {"endpoint": endpoint, "json": json} + if self._exc: + raise self._exc + return self._response + + +def _patch_client(monkeypatch, *, response=None, exc=None): + monkeypatch.setattr(flaresolverr.httpx, "AsyncClient", lambda *a, **k: _FakePostClient(response=response, exc=exc)) + + +_OK_ENVELOPE = { + "status": "ok", + "solution": { + "url": "https://x/final", + "status": 200, + "response": "solved", + "cookies": [{"name": "cf_clearance", "value": "abc"}], + "userAgent": "Mozilla/5.0 ...", + }, +} + + +@pytest.mark.asyncio +async def test_solve_success_parses_solution(monkeypatch): + _patch_client(monkeypatch, response=_FakeHTTPResponse(200, _OK_ENVELOPE)) + + solution = await flaresolverr.solve("http://flaresolverr:8191", "https://x/r", 45) + + assert solution is not None + assert solution.html == "solved" + assert solution.status_code == 200 + assert solution.url == "https://x/final" + assert solution.cookies == [{"name": "cf_clearance", "value": "abc"}] + assert solution.user_agent == "Mozilla/5.0 ..." + + +@pytest.mark.asyncio +async def test_solve_builds_correct_request(monkeypatch): + _patch_client(monkeypatch, response=_FakeHTTPResponse(200, _OK_ENVELOPE)) + + # trailing slash on base URL should be normalized + await flaresolverr.solve("http://flaresolverr:8191/", "https://x/r", 30) + + call = _FakePostClient.last_call + assert call["endpoint"] == "http://flaresolverr:8191/v1" + assert call["json"]["cmd"] == "request.get" + assert call["json"]["url"] == "https://x/r" + assert call["json"]["maxTimeout"] == 30_000 # seconds -> milliseconds + + +@pytest.mark.asyncio +async def test_solve_returns_none_on_http_error(monkeypatch): + _patch_client(monkeypatch, response=_FakeHTTPResponse(500, {})) + assert await flaresolverr.solve("http://fs:8191", "https://x/r", 30) is None + + +@pytest.mark.asyncio +async def test_solve_returns_none_on_error_status(monkeypatch): + _patch_client(monkeypatch, response=_FakeHTTPResponse(200, {"status": "error", "message": "challenge failed"})) + assert await flaresolverr.solve("http://fs:8191", "https://x/r", 30) is None + + +@pytest.mark.asyncio +async def test_solve_returns_none_on_empty_solution(monkeypatch): + _patch_client(monkeypatch, response=_FakeHTTPResponse(200, {"status": "ok", "solution": {"response": ""}})) + assert await flaresolverr.solve("http://fs:8191", "https://x/r", 30) is None + + +@pytest.mark.asyncio +async def test_solve_returns_none_on_non_json(monkeypatch): + _patch_client(monkeypatch, response=_FakeHTTPResponse(200, raise_json=True)) + assert await flaresolverr.solve("http://fs:8191", "https://x/r", 30) is None + + +@pytest.mark.asyncio +async def test_solve_returns_none_on_connection_error(monkeypatch): + _patch_client(monkeypatch, exc=httpx.ConnectError("unreachable")) + assert await flaresolverr.solve("http://fs:8191", "https://x/r", 30) is None diff --git a/tests/unit_tests/test_config.py b/tests/unit_tests/test_config.py index 271884316..3277d87a1 100644 --- a/tests/unit_tests/test_config.py +++ b/tests/unit_tests/test_config.py @@ -5,6 +5,7 @@ from pathlib import Path from typing import Any import pytest +from pydantic import ValidationError from mealie.core.config import get_app_settings from mealie.core.settings.settings import AppSettings, determine_secrets @@ -395,6 +396,51 @@ def test_sensitive_settings_mask(monkeypatch: pytest.MonkeyPatch): assert settings_json[setting] == "*****" +_SCRAPER_URL_FIELDS = ["SCRAPER_PROXY_URL", "SCRAPER_FLARESOLVERR_URL"] + + +@pytest.mark.parametrize("field", _SCRAPER_URL_FIELDS) +@pytest.mark.parametrize( + "value", + [ + "flaresolverr:8191", # missing scheme + "192.168.1.5:8191", # bare host:port + "just-a-hostname", # no scheme, no port + ], +) +def test_scraper_url_rejects_missing_scheme(field: str, value: str, monkeypatch: pytest.MonkeyPatch): + monkeypatch.setenv(field, value) + get_app_settings.cache_clear() + + with pytest.raises(ValidationError): + get_app_settings() + + +@pytest.mark.parametrize("field", _SCRAPER_URL_FIELDS) +@pytest.mark.parametrize( + "value", + [ + "http://flaresolverr:8191", + "https://fs.example.com:8191/", + "http://user:pass@host:8080", # userinfo is allowed + "socks5://host:1080", # non-http schemes (valid for proxies) are not rejected + ], +) +def test_scraper_url_accepts_valid(field: str, value: str, monkeypatch: pytest.MonkeyPatch): + monkeypatch.setenv(field, value) + get_app_settings.cache_clear() + + assert getattr(get_app_settings(), field) == value + + +@pytest.mark.parametrize("field", _SCRAPER_URL_FIELDS) +def test_scraper_url_allows_unset(field: str, monkeypatch: pytest.MonkeyPatch): + monkeypatch.delenv(field, raising=False) + get_app_settings.cache_clear() + + assert getattr(get_app_settings(), field) is None + + class DetermineSecretsTests: def test_non_production_returns_fixed_key(self, tmp_path: Path): result = determine_secrets(tmp_path, ".secret", production=False)