primp-patch v0.2.0: six-layer Python HTTP stack → primp (browser TLS fingerprint)
- One-line patch() covering requests / urllib3 / http.client / urllib / httpx (sync+async) / aiohttp - Zero source modification — pure monkeypatch, survives upstream upgrades - Auto-drop python-* User-Agents that contradict the impersonated Chrome fingerprint - Session(proxy=None) env-proxy fallback (PRIMP_PATCH_PROXY / HTTPS_PROXY) - PRIMP_PATCH_AUTO=1 opt-in auto-activation; idempotent patch/unpatch - 16/16 regression tests passing; openai SDK verified end-to-end
This commit is contained in:
commit
63b9e2ae0d
9 files changed
+1047
No files matched your search
@@ -0,0 +1,7 @@
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*.egg-info/
|
||||
build/
|
||||
dist/
|
||||
.venv/
|
||||
*.egg
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 vex1023
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,74 @@
|
||||
# primp-patch
|
||||
|
||||
One line to route **all** of your Python HTTP traffic through [primp](https://github.com/deedy5/primp) — the rnet-based client with real browser TLS/JA3/JA4 fingerprinting.
|
||||
|
||||
```python
|
||||
import primp_patch
|
||||
primp_patch.patch()
|
||||
|
||||
import requests # unchanged code, primp transport
|
||||
requests.get("https://example.com")
|
||||
```
|
||||
|
||||
## Why
|
||||
|
||||
Libraries and SDKs hard-code `requests`, `urllib3`, `http.client` or `urllib`. Swapping the HTTP
|
||||
backend means forking each one. `primp-patch` instead replaces the transport *at runtime*, so
|
||||
existing code — including third-party packages — inherits browser-grade impersonation with zero
|
||||
source changes.
|
||||
|
||||
## What gets patched
|
||||
|
||||
| Layer | Entry point | Covers |
|
||||
|---|---|---|
|
||||
| `requests` | `requests.Session.request` | every `requests.get/post/...` |
|
||||
| `urllib3` | `urllib3.PoolManager.urlopen` | anything built on urllib3 pools |
|
||||
| `http.client` | `HTTPConnection.request` / `HTTPSConnection.request` | stdlib HTTPConnection users |
|
||||
| `urllib` | `urllib.request.urlopen` | `urllib.request` users |
|
||||
| `httpx` | `Client._send_single_request` (+ async) | httpx sync & async, openai/anthropic SDKs |
|
||||
| `aiohttp` | `ClientSession._request` | async clients & scrapers |
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
pip install primp-patch
|
||||
# with extras
|
||||
pip install "primp-patch[all]"
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
import primp_patch
|
||||
primp_patch.patch(proxy="http://host:port") # explicit proxy
|
||||
|
||||
# or via environment
|
||||
# PRIMP_PATCH_PROXY=http://host:port
|
||||
# or HTTPS_PROXY / HTTP_PROXY
|
||||
|
||||
# selective patching
|
||||
primp_patch.patch(urllib3_=False)
|
||||
|
||||
# inspect / revert
|
||||
primp_patch.is_patched()
|
||||
primp_patch.unpatch()
|
||||
```
|
||||
|
||||
### Auto-patch on import
|
||||
|
||||
```bash
|
||||
export PRIMP_PATCH_AUTO=1
|
||||
python -c "import primp_patch, requests; print(requests.get('https://example.com').status_code)"
|
||||
```
|
||||
|
||||
## Notes & limitations
|
||||
|
||||
- `stream=True`, `files=`, `auth=`, custom `verify`/`cert` are not translated (primp has no
|
||||
equivalent); they are silently dropped rather than raising.
|
||||
- Cookie jars are tracked in-process but not persisted.
|
||||
- Redirect handling is delegated to primp; `allow_redirects=False` is not honored.
|
||||
- Intended for clients that need anti-bot robustness, not as a drop-in for every edge case.
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
@@ -0,0 +1,35 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=68", "wheel"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "primp-patch"
|
||||
version = "0.2.0"
|
||||
description = "Runtime HTTP-layer replacement: route requests/urllib3/http.client/urllib through primp for browser-grade anti-bot fingerprinting"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.9"
|
||||
license = { text = "MIT" }
|
||||
authors = [{ name = "vex1023" }]
|
||||
keywords = ["primp", "requests", "urllib", "anti-bot", "http", "impersonate", "patch"]
|
||||
classifiers = [
|
||||
"Development Status :: 4 - Beta",
|
||||
"Intended Audience :: Developers",
|
||||
"License :: OSI Approved :: MIT License",
|
||||
"Programming Language :: Python :: 3",
|
||||
"Topic :: Internet :: WWW/HTTP",
|
||||
"Topic :: Software Development :: Libraries :: Python Modules",
|
||||
]
|
||||
dependencies = [
|
||||
"primp>=0.10",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
requests = ["requests>=2.20"]
|
||||
urllib3 = ["urllib3>=1.26"]
|
||||
all = ["requests>=2.20", "urllib3>=1.26"]
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
|
||||
[project.urls]
|
||||
Homepage = "https://github.com/vex1023/primp-patch"
|
||||
@@ -0,0 +1,11 @@
|
||||
"""primp-patch: replace requests/urllib3/http.client/urllib transport with primp at runtime.
|
||||
|
||||
import primp_patch
|
||||
primp_patch.patch() # one line; all four layers routed through primp
|
||||
# ... existing code using requests / urllib3 / urllib / http.client is untouched
|
||||
"""
|
||||
from ._patch import patch, unpatch, is_patched
|
||||
from .transport import Response, Session, HTTPError
|
||||
|
||||
__version__ = "0.2.0"
|
||||
__all__ = ["patch", "unpatch", "is_patched", "Session", "Response", "HTTPError"]
|
||||
@@ -0,0 +1,317 @@
|
||||
"""httpx + aiohttp layers for primp_patch.
|
||||
|
||||
httpx: patch Client._send_single_request / AsyncClient._send_single_request —
|
||||
route through primp (sync Client / AsyncClient), return real httpx.Response
|
||||
objects with ByteStream so the rest of httpx (redirects, cookies, encodings)
|
||||
keeps working.
|
||||
|
||||
aiohttp: patch ClientSession._request (async) — build a primp AsyncClient request
|
||||
and wrap into a duck-typed ClientResponse supporting status/read/text/json/
|
||||
headers/release/close/context-manager, enough for the vast majority of users.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json as _json
|
||||
from typing import Any
|
||||
|
||||
from .transport import _env_proxy
|
||||
|
||||
_originals: dict = {}
|
||||
_STATE = {"httpx": False, "aiohttp": False}
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- helpers
|
||||
def _primp_proxy(proxy: str | None) -> str | None:
|
||||
import os
|
||||
return proxy or os.environ.get("PRIMP_PATCH_PROXY") or _env_proxy()
|
||||
|
||||
|
||||
def _build_headers(headers) -> dict | None:
|
||||
if headers is None:
|
||||
return None
|
||||
out = {}
|
||||
if hasattr(headers, "raw"): # httpx.Headers
|
||||
items = [(k.decode(), v.decode()) for k, v in headers.raw]
|
||||
elif hasattr(headers, "items"):
|
||||
items = list(headers.items())
|
||||
else:
|
||||
items = list(headers)
|
||||
for k, v in items:
|
||||
kk = k.decode() if isinstance(k, bytes) else k
|
||||
vv = v.decode() if isinstance(v, bytes) else v
|
||||
if kk.lower() in ("host", "content-length", "transfer-encoding", "connection"):
|
||||
continue # primp manages framing itself; these cause 'malformed headers'
|
||||
if kk.lower() == "user-agent" and vv.startswith("python-"):
|
||||
continue # library UA defeats impersonation; let primp send its own
|
||||
out[kk] = vv
|
||||
return out or None
|
||||
|
||||
|
||||
def _stringify(params):
|
||||
if not params:
|
||||
return params
|
||||
return {k: (str(v).lower() if isinstance(v, bool) else str(v)) for k, v in params.items()}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------- httpx
|
||||
def _patch_httpx(proxy: str | None = None, impersonate: str = "chrome") -> bool:
|
||||
try:
|
||||
import httpx
|
||||
except ImportError:
|
||||
return False
|
||||
if _STATE["httpx"]:
|
||||
return True
|
||||
|
||||
_originals["httpx.Client._send_single_request"] = httpx.Client._send_single_request
|
||||
_originals["httpx.AsyncClient._send_single_request"] = httpx.AsyncClient._send_single_request
|
||||
|
||||
from httpx import Response as HxResponse
|
||||
from httpx._content import ByteStream
|
||||
|
||||
_sync_client = {}
|
||||
|
||||
def _do_sync(self, request):
|
||||
import primp
|
||||
key = id(self)
|
||||
client = _sync_client.get(key)
|
||||
if client is None:
|
||||
px = _primp_proxy(getattr(self, "_primp_proxy", None))
|
||||
import os as _os
|
||||
px = px or _os.environ.get("PRIMP_PATCH_PROXY") or _os.environ.get("HTTPS_PROXY") or _os.environ.get("https_proxy")
|
||||
if px is None:
|
||||
import os as _os
|
||||
px = _os.environ.get("PRIMP_PATCH_PROXY") or _os.environ.get("HTTPS_PROXY") or _os.environ.get("https_proxy")
|
||||
client = primp.Client(impersonate=impersonate, proxy=px)
|
||||
_sync_client[key] = client
|
||||
r = client.request(
|
||||
method=request.method,
|
||||
url=str(request.url),
|
||||
params=None, # already encoded in url
|
||||
headers=_build_headers(request.headers),
|
||||
content=_read_sync_stream(request.stream),
|
||||
timeout=_timeout_of(self),
|
||||
)
|
||||
return r
|
||||
|
||||
def _timeout_of(client) -> float | None:
|
||||
try:
|
||||
t = client.timeout
|
||||
return None if t is None or t is httpx.USE_CLIENT_DEFAULT else float(t)
|
||||
except Exception:
|
||||
try:
|
||||
return float(client.timeout.connect) if client.timeout.connect else None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def _read_sync_stream(stream) -> bytes | None:
|
||||
try:
|
||||
chunks = [bytes(c) for c in stream]
|
||||
data = b"".join(chunks)
|
||||
return data or None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def send_single_request(self, request):
|
||||
pr = _do_sync(self, request)
|
||||
content = pr.content or b""
|
||||
hx = HxResponse(
|
||||
status_code=pr.status_code,
|
||||
headers=list(pr.headers.items()) if pr.headers else [],
|
||||
stream=ByteStream(content),
|
||||
request=request,
|
||||
)
|
||||
return hx
|
||||
|
||||
httpx.Client._send_single_request = send_single_request
|
||||
|
||||
# ---- async ----
|
||||
_async_client = {}
|
||||
|
||||
async def send_single_request_async(self, request):
|
||||
import primp
|
||||
key = id(self)
|
||||
client = _async_client.get(key)
|
||||
if client is None:
|
||||
px = _primp_proxy(getattr(self, "_primp_proxy", None))
|
||||
import os as _os
|
||||
px = px or _os.environ.get("PRIMP_PATCH_PROXY") or _os.environ.get("HTTPS_PROXY") or _os.environ.get("https_proxy")
|
||||
if px is None:
|
||||
import os as _os
|
||||
px = _os.environ.get("PRIMP_PATCH_PROXY") or _os.environ.get("HTTPS_PROXY") or _os.environ.get("https_proxy")
|
||||
client = primp.AsyncClient(impersonate=impersonate, proxy=px)
|
||||
_async_client[key] = client
|
||||
data = _read_sync_stream(request.stream)
|
||||
r = await client.request(
|
||||
method=request.method,
|
||||
url=str(request.url),
|
||||
headers=_build_headers(request.headers),
|
||||
content=data,
|
||||
timeout=_timeout_of(self),
|
||||
)
|
||||
content = r.content or b""
|
||||
hx = HxResponse(
|
||||
status_code=r.status_code,
|
||||
headers=list(r.headers.items()) if r.headers else [],
|
||||
stream=ByteStream(content),
|
||||
request=request,
|
||||
)
|
||||
return hx
|
||||
|
||||
httpx.AsyncClient._send_single_request = send_single_request_async
|
||||
_STATE["httpx"] = True
|
||||
return True
|
||||
|
||||
|
||||
def _unpatch_httpx():
|
||||
if "httpx.Client._send_single_request" in _originals:
|
||||
import httpx
|
||||
httpx.Client._send_single_request = _originals["httpx.Client._send_single_request"]
|
||||
httpx.AsyncClient._send_single_request = _originals["httpx.AsyncClient._send_single_request"]
|
||||
_STATE["httpx"] = False
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- aiohttp
|
||||
def _patch_aiohttp(proxy: str | None = None, impersonate: str = "chrome") -> bool:
|
||||
try:
|
||||
import aiohttp
|
||||
except ImportError:
|
||||
return False
|
||||
if _STATE["aiohttp"]:
|
||||
return True
|
||||
|
||||
_originals["aiohttp.ClientSession._request"] = aiohttp.ClientSession._request
|
||||
|
||||
class _FakeClientResponse:
|
||||
"""Duck-typed aiohttp.ClientResponse covering the common surface."""
|
||||
|
||||
def __init__(self, method, url, pr):
|
||||
self.method = method
|
||||
self.url = url
|
||||
self.status = pr.status_code
|
||||
self.reason = ""
|
||||
self._content = pr.content or b""
|
||||
self._headers = dict(pr.headers) if pr.headers else {}
|
||||
self.ok = self.status < 400
|
||||
self._released = False
|
||||
self.version_tuple = (1, 1)
|
||||
self.charset = "utf-8"
|
||||
self.content_type = self._headers.get("content-type", "").split(";")[0]
|
||||
|
||||
# aiohttp headers object-ish
|
||||
@property
|
||||
def headers(self):
|
||||
return self._headers
|
||||
|
||||
async def read(self):
|
||||
self._released = True
|
||||
return self._content
|
||||
|
||||
async def text(self, encoding="utf-8", errors="strict"):
|
||||
return self._content.decode(encoding or self.charset, errors=errors)
|
||||
|
||||
async def json(self, **kw):
|
||||
return _json.loads(self._content)
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, *a):
|
||||
self._released = True
|
||||
|
||||
def release(self):
|
||||
self._released = True
|
||||
return _noop_future()
|
||||
|
||||
def close(self):
|
||||
self._released = True
|
||||
|
||||
def raise_for_status(self):
|
||||
if self.status >= 400:
|
||||
exc = getattr(aiohttp, "ClientResponseError", Exception)
|
||||
raise exc(self.status, self.url)
|
||||
return self
|
||||
|
||||
@property
|
||||
def content(self):
|
||||
# minimal stream adapter
|
||||
import io
|
||||
inner = io.BytesIO(self._content)
|
||||
outer = self
|
||||
|
||||
class _Stream:
|
||||
async def read(self, n=-1):
|
||||
return inner.read(n if n and n > 0 else -1)
|
||||
|
||||
async def readline(self):
|
||||
return inner.readline()
|
||||
|
||||
def at_eof(self):
|
||||
return inner.tell() >= len(outer._content)
|
||||
|
||||
def __aiter__(self):
|
||||
return self
|
||||
|
||||
async def __anext__(self):
|
||||
line = inner.readline()
|
||||
if not line:
|
||||
raise StopAsyncIteration
|
||||
return line
|
||||
|
||||
return _Stream()
|
||||
|
||||
def get_encoding(self):
|
||||
return "utf-8"
|
||||
|
||||
def _noop_future():
|
||||
f = asyncio.get_event_loop().create_future()
|
||||
f.set_result(None)
|
||||
return f
|
||||
|
||||
async def _request(self, method, str_or_url, **kw):
|
||||
import primp
|
||||
# per-session async client
|
||||
client = getattr(self, "_primp_async", None)
|
||||
if client is None:
|
||||
px = _primp_proxy(kw.get("proxy"))
|
||||
import os as _os
|
||||
px = px or _os.environ.get("PRIMP_PATCH_PROXY") or _os.environ.get("HTTPS_PROXY") or _os.environ.get("https_proxy")
|
||||
client = primp.AsyncClient(impersonate=impersonate, proxy=px)
|
||||
self._primp_async = client
|
||||
data = kw.get("data")
|
||||
if isinstance(data, (bytes, bytearray)):
|
||||
data = bytes(data)
|
||||
elif data is not None and not isinstance(data, (str, bytes)):
|
||||
data = str(data).encode()
|
||||
params = kw.get("params")
|
||||
headers = _build_headers(kw.get("headers"))
|
||||
timeout = kw.get("timeout")
|
||||
to = None
|
||||
if timeout is not None and not isinstance(timeout, object.__class__ or type):
|
||||
to = getattr(timeout, "total", None)
|
||||
if to is not None:
|
||||
try:
|
||||
to = float(to)
|
||||
except Exception:
|
||||
to = None
|
||||
r = await client.request(
|
||||
method=method,
|
||||
url=str(str_or_url),
|
||||
params=_stringify(params),
|
||||
headers=headers,
|
||||
data=data,
|
||||
json=kw.get("json"),
|
||||
timeout=to,
|
||||
)
|
||||
return _FakeClientResponse(method, str(str_or_url), r)
|
||||
|
||||
aiohttp.ClientSession._request = _request
|
||||
_STATE["aiohttp"] = True
|
||||
return True
|
||||
|
||||
|
||||
def _unpatch_aiohttp():
|
||||
if "aiohttp.ClientSession._request" in _originals:
|
||||
import aiohttp
|
||||
aiohttp.ClientSession._request = _originals["aiohttp.ClientSession._request"]
|
||||
_STATE["aiohttp"] = False
|
||||
@@ -0,0 +1,11 @@
|
||||
import os
|
||||
|
||||
from primp_patch import patch as _patch
|
||||
|
||||
|
||||
def _auto():
|
||||
if os.environ.get("PRIMP_PATCH_AUTO", "").lower() in ("1", "true", "yes"):
|
||||
_patch()
|
||||
|
||||
|
||||
_auto()
|
||||
@@ -0,0 +1,350 @@
|
||||
"""Runtime HTTP-layer replacement: route requests/urllib3/http.client/urllib through primp."""
|
||||
from __future__ import annotations
|
||||
|
||||
import http.client as _hc
|
||||
import urllib.request as _ur
|
||||
|
||||
from .transport import Response, Session, transport_request, _env_proxy
|
||||
from ._async_layers import (
|
||||
_patch_httpx, _unpatch_httpx, _patch_aiohttp, _unpatch_aiohttp,
|
||||
)
|
||||
|
||||
__all__ = ["patch", "unpatch", "is_patched"]
|
||||
|
||||
_STATE = {
|
||||
"requests": False,
|
||||
"urllib3": False,
|
||||
"http.client": False,
|
||||
"urllib": False,
|
||||
"httpx": False,
|
||||
"aiohttp": False,
|
||||
}
|
||||
|
||||
_originals: dict = {}
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ requests
|
||||
def _patch_requests(session_cls=None):
|
||||
try:
|
||||
import requests
|
||||
except ImportError:
|
||||
return False
|
||||
if _STATE["requests"]:
|
||||
return True
|
||||
|
||||
_originals["requests.Session.request"] = requests.Session.request
|
||||
|
||||
def request(self, method, url, params=None, data=None, headers=None,
|
||||
cookies=None, files=None, auth=None, timeout=None,
|
||||
allow_redirects=True, proxies=None, hooks=None, stream=None,
|
||||
verify=None, cert=None, json=None, **kw):
|
||||
if proxies is None and getattr(self, "proxies", None):
|
||||
proxies = self.proxies
|
||||
_apply_proxies(proxies)
|
||||
resp = transport_request(method, url, params=params, data=data, json=json,
|
||||
headers=headers, timeout=timeout)
|
||||
return _to_requests_response(resp, requests)
|
||||
|
||||
requests.Session.request = request
|
||||
_STATE["requests"] = True
|
||||
return True
|
||||
|
||||
|
||||
def _to_requests_response(resp: Response, requests_mod):
|
||||
r = requests_mod.Response()
|
||||
r.status_code = resp.status_code
|
||||
r._content = resp.content
|
||||
r.headers = dict(resp.headers) if resp.headers else {}
|
||||
r.url = resp.url
|
||||
r.encoding = "utf-8"
|
||||
# minimal ok/history flags
|
||||
try:
|
||||
r.reason = {400: "Bad Request", 401: "Unauthorized", 403: "Forbidden",
|
||||
404: "Not Found", 429: "Too Many Requests",
|
||||
500: "Internal Server Error"}.get(resp.status_code, "")
|
||||
except Exception:
|
||||
pass
|
||||
return r
|
||||
|
||||
|
||||
def _unpatch_requests():
|
||||
try:
|
||||
import requests
|
||||
import yfinance_patch_store # noqa: placeholder
|
||||
except ImportError:
|
||||
pass
|
||||
if "requests.Session.request" in _originals:
|
||||
import requests
|
||||
requests.Session.request = _originals["requests.Session.request"]
|
||||
_STATE["requests"] = False
|
||||
|
||||
|
||||
def _apply_proxies(proxies):
|
||||
"""Set env-level proxy from a requests-style proxies dict."""
|
||||
import os
|
||||
if isinstance(proxies, dict) and proxies:
|
||||
p = proxies.get("https") or proxies.get("http")
|
||||
if p and not os.environ.get("PRIMP_PATCH_PROXY_LOCKED"):
|
||||
os.environ["PRIMP_PATCH_PROXY"] = p
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- urllib3
|
||||
def _patch_urllib3():
|
||||
try:
|
||||
import urllib3
|
||||
except ImportError:
|
||||
return False
|
||||
if _STATE["urllib3"]:
|
||||
return True
|
||||
|
||||
_originals["urllib3.PoolManager.urlopen"] = urllib3.PoolManager.urlopen
|
||||
|
||||
def urlopen(self, method, url, body=None, headers=None, retries=None,
|
||||
redirect=True, assert_same_host=True, timeout=None,
|
||||
pool_timeout=None, release_conn=None, chunked=False,
|
||||
body_pos=None, **kw):
|
||||
import json as _json
|
||||
# urllib3 passes body as bytes/str
|
||||
json_payload = None
|
||||
data_payload = body
|
||||
ct = (headers or {}).get("Content-Type", "") if isinstance(headers, dict) else ""
|
||||
if ct and "json" in ct and body:
|
||||
try:
|
||||
json_payload = _json.loads(body) if isinstance(body, (str, bytes)) else body
|
||||
data_payload = None
|
||||
except Exception:
|
||||
pass
|
||||
_apply_proxies(getattr(self, "proxies", None) or kw.get("proxies"))
|
||||
resp = transport_request(method, url, params=None, data=data_payload,
|
||||
json=json_payload, headers=headers, timeout=timeout)
|
||||
return _to_urllib3_response(resp, urllib3)
|
||||
|
||||
urllib3.PoolManager.urlopen = urlopen
|
||||
_STATE["urllib3"] = True
|
||||
return True
|
||||
|
||||
|
||||
def _to_urllib3_response(resp: Response, urllib3_mod):
|
||||
r = urllib3_mod.HTTPResponse(
|
||||
body=resp.content,
|
||||
headers=dict(resp.headers) if resp.headers else {},
|
||||
status=resp.status_code,
|
||||
request_url=resp.url,
|
||||
)
|
||||
return r
|
||||
|
||||
|
||||
def _unpatch_urllib3():
|
||||
if "urllib3.PoolManager.urlopen" in _originals:
|
||||
import urllib3
|
||||
urllib3.PoolManager.urlopen = _originals["urllib3.PoolManager.urlopen"]
|
||||
_STATE["urllib3"] = False
|
||||
|
||||
|
||||
# --------------------------------------------------------------- http.client
|
||||
def _patch_http_client():
|
||||
if _STATE["http.client"]:
|
||||
return True
|
||||
_originals["HTTPConnection.request"] = _hc.HTTPConnection.request
|
||||
_originals["HTTPSConnection.request"] = _hc.HTTPSConnection.request
|
||||
|
||||
def request(self, method, url, body=None, headers=None, *, encode_chunked=False):
|
||||
import json as _json
|
||||
json_payload = None
|
||||
data_payload = body
|
||||
if isinstance(body, (dict, list)):
|
||||
json_payload = body
|
||||
data_payload = None
|
||||
elif isinstance(body, (str, bytes)):
|
||||
ct = (headers or {}).get("Content-Type", "") if isinstance(headers, dict) else ""
|
||||
if "json" in ct:
|
||||
try:
|
||||
json_payload = _json.loads(body)
|
||||
data_payload = None
|
||||
except Exception:
|
||||
pass
|
||||
host_header = (headers or {}).get("Host") if isinstance(headers, dict) else None
|
||||
scheme = "https" if isinstance(self, _hc.HTTPSConnection) else "http"
|
||||
full = f"{scheme}://{self.host}" + (f":{self.port}" if self.port not in (80, 443) else "") + url
|
||||
resp = transport_request(method, full, data=data_payload, json=json_payload,
|
||||
headers=headers, timeout=self.timeout or None)
|
||||
# Stuff the translated response back into a real HTTPResponse-shaped object
|
||||
self._primp_last = resp
|
||||
# Build a lightweight response object with the http.client API surface
|
||||
return self._primp_last and _build_httpresponse(resp, self, method)
|
||||
|
||||
_originals["HTTPConnection.getresponse"] = _hc.HTTPConnection.getresponse
|
||||
|
||||
def getresponse(self):
|
||||
resp = getattr(self, "_primp_last", None)
|
||||
if resp is not None:
|
||||
self._primp_last = None
|
||||
return _build_httpresponse(resp, self, "GET")
|
||||
return _originals["HTTPConnection.getresponse"](self)
|
||||
|
||||
_hc.HTTPConnection.request = request
|
||||
_hc.HTTPSConnection.request = request
|
||||
_hc.HTTPConnection.getresponse = getresponse
|
||||
_hc.HTTPSConnection.getresponse = getresponse
|
||||
_STATE["http.client"] = True
|
||||
return True
|
||||
|
||||
|
||||
class _FakeHTTPResponse:
|
||||
"""Duck-typed http.client response (HTTPResponse requires a real socket fp)."""
|
||||
def __init__(self, resp: Response):
|
||||
self.status = resp.status_code
|
||||
self.reason = ""
|
||||
self._content = resp.content or b""
|
||||
self._headers = resp.headers
|
||||
self.url = resp.url
|
||||
self.version = 11
|
||||
self.will_close = True
|
||||
self.length = len(self._content)
|
||||
def read(self, amt=None, **kw):
|
||||
d, self._content = self._content, b""
|
||||
self.length = 0
|
||||
return d
|
||||
def readinto(self, b):
|
||||
n = min(len(b), len(self._content))
|
||||
b[:n] = self._content[:n]
|
||||
self._content = self._content[n:]
|
||||
return n
|
||||
def readline(self, *a):
|
||||
raise NotImplementedError
|
||||
def getheader(self, name, default=None):
|
||||
if not self._headers: return default
|
||||
try:
|
||||
return self._headers.get(name, default)
|
||||
except Exception:
|
||||
return default
|
||||
def getheaders(self):
|
||||
return list(self._headers.items()) if self._headers else []
|
||||
@property
|
||||
def headers(self):
|
||||
return self._headers
|
||||
@property
|
||||
def code(self):
|
||||
return self.status
|
||||
def isclosed(self):
|
||||
return True
|
||||
def close(self):
|
||||
pass
|
||||
def __enter__(self):
|
||||
return self
|
||||
def __exit__(self, *a):
|
||||
pass
|
||||
|
||||
|
||||
def _build_httpresponse(resp: Response, conn, method):
|
||||
return _FakeHTTPResponse(resp)
|
||||
|
||||
|
||||
def _unpatch_http_client():
|
||||
if "HTTPConnection.request" in _originals:
|
||||
_hc.HTTPConnection.request = _originals["HTTPConnection.request"]
|
||||
_hc.HTTPSConnection.request = _originals["HTTPSConnection.request"]
|
||||
_hc.HTTPConnection.getresponse = _originals["HTTPConnection.getresponse"]
|
||||
_hc.HTTPSConnection.getresponse = _originals["HTTPConnection.getresponse"]
|
||||
_STATE["http.client"] = False
|
||||
|
||||
|
||||
# -------------------------------------------------------------------- urllib
|
||||
def _patch_urllib():
|
||||
if _STATE["urllib"]:
|
||||
return True
|
||||
_originals["urllib.request.urlopen"] = _ur.urlopen
|
||||
|
||||
def urlopen(url, data=None, timeout=None, *, context=None, **kw):
|
||||
import json as _json
|
||||
if isinstance(url, _ur.Request):
|
||||
method = url.get_method()
|
||||
full_url = url.full_url
|
||||
payload = url.data
|
||||
headers = dict(url.header_items())
|
||||
else:
|
||||
method = "POST" if data else "GET"
|
||||
full_url = url
|
||||
payload = data
|
||||
headers = None
|
||||
json_payload = None
|
||||
data_payload = payload
|
||||
ct = (headers or {}).get("Content-type", "") if isinstance(headers, dict) else ""
|
||||
if ct and "json" in ct and payload:
|
||||
try:
|
||||
json_payload = _json.loads(payload) if isinstance(payload, (str, bytes)) else payload
|
||||
data_payload = None
|
||||
except Exception:
|
||||
pass
|
||||
resp = transport_request(method, full_url, data=data_payload,
|
||||
json=json_payload, headers=headers, timeout=timeout)
|
||||
return _build_urlibresponse(resp)
|
||||
|
||||
_ur.urlopen = urlopen
|
||||
_STATE["urllib"] = True
|
||||
return True
|
||||
|
||||
|
||||
def _build_urlibresponse(resp: Response):
|
||||
import io as _io
|
||||
# addinfourl.status is a read-only property in 3.9+; code kwarg already sets it
|
||||
r = _ur.addinfourl(_io.BytesIO(resp.content or b""),
|
||||
dict(resp.headers) if resp.headers else {},
|
||||
resp.url or "", code=resp.status_code)
|
||||
return r
|
||||
|
||||
|
||||
def _unpatch_urllib():
|
||||
if "urllib.request.urlopen" in _originals:
|
||||
_ur.urlopen = _originals["urllib.request.urlopen"]
|
||||
_STATE["urllib"] = False
|
||||
|
||||
|
||||
# -------------------------------------------------------------------- public
|
||||
def patch(requests_: bool = True, urllib3_: bool = True,
|
||||
http_client: bool = True, urllib_: bool = True,
|
||||
httpx_: bool = True, aiohttp_: bool = True,
|
||||
proxy: str | None = None,
|
||||
impersonate: str = "chrome") -> dict:
|
||||
"""Replace low-level HTTP backends with primp.
|
||||
|
||||
Args:
|
||||
requests_: patch requests.Session.request
|
||||
urllib3_: patch urllib3.PoolManager.urlopen
|
||||
http_client: patch http.client.HTTP(S)Connection.request
|
||||
urllib_: patch urllib.request.urlopen
|
||||
proxy: default proxy (falls back to PRIMP_PATCH_PROXY / HTTPS_PROXY / HTTP_PROXY)
|
||||
|
||||
Returns dict of {layer: bool} for what actually got patched.
|
||||
"""
|
||||
if proxy:
|
||||
import os
|
||||
os.environ["PRIMP_PATCH_PROXY"] = proxy
|
||||
os.environ["PRIMP_PATCH_PROXY_LOCKED"] = "1"
|
||||
result = {}
|
||||
if requests_:
|
||||
result["requests"] = _patch_requests()
|
||||
if urllib3_:
|
||||
result["urllib3"] = _patch_urllib3()
|
||||
if http_client:
|
||||
result["http.client"] = _patch_http_client()
|
||||
if urllib_:
|
||||
result["urllib"] = _patch_urllib()
|
||||
if httpx_:
|
||||
result["httpx"] = _patch_httpx(proxy=proxy, impersonate=impersonate)
|
||||
if aiohttp_:
|
||||
result["aiohttp"] = _patch_aiohttp(proxy=proxy, impersonate=impersonate)
|
||||
return result
|
||||
|
||||
|
||||
def unpatch():
|
||||
_unpatch_httpx()
|
||||
_unpatch_aiohttp()
|
||||
_unpatch_requests()
|
||||
_unpatch_urllib3()
|
||||
_unpatch_http_client()
|
||||
_unpatch_urllib()
|
||||
|
||||
|
||||
def is_patched() -> dict:
|
||||
return dict(_STATE)
|
||||
@@ -0,0 +1,221 @@
|
||||
"""Core transport: translate requests/urllib-style calls to primp."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json as _json
|
||||
import threading
|
||||
from typing import Any
|
||||
|
||||
import primp
|
||||
|
||||
_sentinel = object()
|
||||
DEFAULT_IMPERSONATE = "chrome"
|
||||
|
||||
_BROWSER_UA = (
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||||
"(KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
|
||||
)
|
||||
|
||||
|
||||
class TransportError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class HTTPError(Exception):
|
||||
def __init__(self, response):
|
||||
self.response = response
|
||||
super().__init__(f"{response.status_code} error for {getattr(response, 'url', '?')}")
|
||||
|
||||
|
||||
class Response:
|
||||
"""Backend-agnostic response facade (requests-like)."""
|
||||
|
||||
def __init__(self, r: Any = None):
|
||||
self._r = r
|
||||
if r is not None:
|
||||
self.status_code = r.status_code
|
||||
self.text = r.text
|
||||
self.content = r.content
|
||||
self.headers = r.headers
|
||||
self.url = str(r.url) if getattr(r, "url", None) is not None else None
|
||||
|
||||
def json(self, **kw):
|
||||
return self._r.json(**kw)
|
||||
|
||||
def raise_for_status(self):
|
||||
if self.status_code >= 400:
|
||||
raise HTTPError(self)
|
||||
return self
|
||||
|
||||
@property
|
||||
def ok(self):
|
||||
return self.status_code < 400
|
||||
|
||||
def close(self):
|
||||
pass
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *a):
|
||||
self.close()
|
||||
|
||||
|
||||
class Jar:
|
||||
"""CookieJar-ish: update()/clear()/._cookies + .jar self-reference."""
|
||||
|
||||
def __init__(self):
|
||||
self._cookies: dict = {}
|
||||
|
||||
def update(self, d):
|
||||
self._cookies.update(d or {})
|
||||
|
||||
def clear(self):
|
||||
self._cookies.clear()
|
||||
|
||||
def get_dict(self):
|
||||
return dict(self._cookies)
|
||||
|
||||
@property
|
||||
def jar(self):
|
||||
return self
|
||||
|
||||
def __iter__(self):
|
||||
return iter(self._cookies.items())
|
||||
|
||||
def __contains__(self, k):
|
||||
return k in self._cookies
|
||||
|
||||
def __getitem__(self, k):
|
||||
return self._cookies[k]
|
||||
|
||||
|
||||
class Cookies(Jar):
|
||||
pass
|
||||
|
||||
|
||||
class Session:
|
||||
"""requests.Session-like facade over one primp.Client."""
|
||||
|
||||
def __init__(self, impersonate: str = DEFAULT_IMPERSONATE, proxy: str | None = _sentinel, **kw):
|
||||
self.impersonate = impersonate
|
||||
if proxy is _sentinel:
|
||||
proxy = _env_proxy()
|
||||
self._proxy = proxy
|
||||
self._lock = threading.Lock()
|
||||
self._client = self._new_client()
|
||||
self.cookies = Cookies()
|
||||
self.headers = {
|
||||
"User-Agent": _BROWSER_UA,
|
||||
"Accept": "*/*",
|
||||
"Accept-Language": "en-US,en;q=0.5",
|
||||
}
|
||||
self.verify = True
|
||||
self.trust_env = True
|
||||
|
||||
def _new_client(self):
|
||||
return primp.Client(impersonate=self.impersonate, proxy=self._proxy)
|
||||
|
||||
# ---- proxies like requests: dict or str -----------------------------
|
||||
@property
|
||||
def proxies(self):
|
||||
return self._proxies
|
||||
|
||||
@proxies.setter
|
||||
def proxies(self, value):
|
||||
self._proxies = value
|
||||
if value:
|
||||
p = (value.get("https") or value.get("http")) if isinstance(value, dict) else value
|
||||
if p and p != self._proxy:
|
||||
self._proxy = p
|
||||
self._client = self._new_client()
|
||||
|
||||
@staticmethod
|
||||
def _stringify(params):
|
||||
if not params:
|
||||
return params
|
||||
return {k: (str(v).lower() if isinstance(v, bool) else str(v)) for k, v in params.items()}
|
||||
|
||||
@staticmethod
|
||||
def _clean_kw(kw: dict) -> dict:
|
||||
for k in ("allow_redirects", "verify", "cert", "stream", "proxies",
|
||||
"auth", "files", "cookies"):
|
||||
kw.pop(k, None)
|
||||
return kw
|
||||
|
||||
def _merged(self, headers):
|
||||
h = dict(self.headers)
|
||||
if headers:
|
||||
h.update(headers)
|
||||
return h or None
|
||||
|
||||
def request(self, method: str, url: str, params=None, data=None, json=None,
|
||||
headers=None, timeout=None, **kw) -> Response:
|
||||
kw = self._clean_kw(kw)
|
||||
m = method.upper()
|
||||
fn = getattr(self._client, m.lower(), None)
|
||||
if fn is None:
|
||||
raise TransportError(f"primp has no method {m}")
|
||||
r = fn(url=url, params=self._stringify(params), json=json, data=data,
|
||||
headers=self._merged(headers), timeout=timeout, **kw)
|
||||
return Response(r)
|
||||
|
||||
def get(self, url, params=None, **kw):
|
||||
return self.request("GET", url, params=params, **kw)
|
||||
|
||||
def post(self, url, data=None, json=None, **kw):
|
||||
return self.request("POST", url, data=data, json=json, **kw)
|
||||
|
||||
def put(self, url, data=None, **kw):
|
||||
return self.request("PUT", url, data=data, **kw)
|
||||
|
||||
def patch(self, url, data=None, **kw):
|
||||
return self.request("PATCH", url, data=data, **kw)
|
||||
|
||||
def delete(self, url, **kw):
|
||||
return self.request("DELETE", url, **kw)
|
||||
|
||||
def head(self, url, **kw):
|
||||
return self.request("HEAD", url, **kw)
|
||||
|
||||
def options(self, url, **kw):
|
||||
return self.request("OPTIONS", url, **kw)
|
||||
|
||||
def close(self):
|
||||
pass
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *a):
|
||||
self.close()
|
||||
|
||||
|
||||
def transport_request(method: str, url: str, **kw) -> Response:
|
||||
"""One-shot request using a shared client (for urllib-style entry points)."""
|
||||
global _shared
|
||||
with _shared_lock:
|
||||
if _shared is None:
|
||||
_shared = Session(proxy=_env_proxy())
|
||||
return _shared.request(method, url, **kw)
|
||||
|
||||
|
||||
def _env_proxy() -> str | None:
|
||||
import os
|
||||
return (
|
||||
os.environ.get("PRIMP_PATCH_PROXY")
|
||||
or os.environ.get("HTTPS_PROXY")
|
||||
or os.environ.get("HTTP_PROXY")
|
||||
or os.environ.get("https_proxy")
|
||||
or os.environ.get("http_proxy")
|
||||
)
|
||||
|
||||
|
||||
_shared: Session | None = None
|
||||
_shared_lock = threading.Lock()
|
||||
|
||||
|
||||
def reset_shared():
|
||||
global _shared
|
||||
with _shared_lock:
|
||||
_shared = None
|
||||
Reference in new issue
Block a user