Phase 2 Modernization: httpx2, Pydantic v2, Retry Engine, and Security Guards#51
Open
skupriienko-mailgun wants to merge 82 commits into
Open
Phase 2 Modernization: httpx2, Pydantic v2, Retry Engine, and Security Guards#51skupriienko-mailgun wants to merge 82 commits into
skupriienko-mailgun wants to merge 82 commits into
Conversation
The async ecosystem is rapidly evolving. Relying on older Python runtimes or outdated HTTP clients introduces performance bottlenecks and blocks us from adopting modern async features safely. - Bumped 'requires-python' requirement to '>=3.11'. - Added 'httpx2 >=2.7.0' as the new primary async engine, keeping 'httpx >=0.24.0' as a fallback safety net. - Added fastapi' to dev dependencies. This keeps the SDK aligned with modern Python standards, guaranteeing our users benefit from the latest performance optimizations and security patches while preventing dependency hell.
… normalization Developers often unintentionally trigger spam filters with invalid HTML or send duplicate emails during network outages, harming their domain's reputation. Additionally, Internationalized Domain Names (IDN) can cause Unicode routing crashes in HTTP clients. - Introduced 'SpamGuard' (a zero-network static HTML deliverability analyzer). - Introduced 'IdempotencyGuard' to generate deterministic SHA-256 payload fingerprints. - Added 'DeliverabilityError' to gracefully handle SpamGuard rule violations. - Implemented 'normalize_domain' to natively convert IDNs to safe RFC 3490 Punycode, which allows domain names to contain non-Latin characters. By failing fast locally, we actively protect our users' domain reputations and prevent billing spikes from duplicate sends—all without adding external network overhead or API latency.
Hardcoded retry logic directly inside HTTP adapters leads to 'thundering herd' problems when the API is rate-limiting us (429s) or experiencing transient cloud errors (50x). - Extracted retry configurations into a dedicated 'RetryPolicy' class. - Implemented mathematical exponential backoff with Full Jitter to prevent collisions. - Added native support for respecting the 'Retry-After' HTTP header. This aligns the SDK with cloud-native resilience best practices, ensuring we behave gracefully as a client while maximizing delivery success rates under heavy load.
…drails Loading massive attachments (e.g., 20MB PDFs) entirely into memory crashes Serverless functions (OOM/CWE-400). Building complex inline HTML templates is clunky, and remembering to manually add idempotency headers is error-prone. - Implemented generator-based 'ChunkedStreamer' for lazy file loading. - Added 'attach_stream' for large files and updated 'attach_inline' to support explicit Content-IDs. - Injected 'check_deliverability()' into the builder. - Added '.set_idempotency_safe()' and automated 'h:X-Idempotency-Key' injection inside the '.build()' method. This guarantees memory-safe execution in constrained environments and provides a bulletproof, 'pit-of-success' developer experience right out of the box.
…x intercepts Orchestrators like Kubernetes require a lightweight way to check if an API client is healthy. During local testing, our 'dry_run' mode lacked rich payload previews. Furthermore, automatic retries weren't resetting file stream pointers, resulting in empty file uploads on the second attempt. - Added a low-overhead, fail-safe 'ping()' method to both Client and AsyncClient. - Implemented the explicit RetryPolicy loop inside endpoint requests. - Added '_reset_stream_pointers' to guarantee idempotency during retry iterations. - Upgraded the 'dry_run' interceptor to trigger LocalSandbox for rich email previews. These changes drastically improve production reliability (K8s readiness probes) while offering a vastly superior, zero-leak testing environment for local development.
Directly importing underlying dependencies inside fuzzers and type definition files creates tight coupling, making it incredibly painful to migrate HTTP clients or introduce fallback engines later. - Centralized all HTTPX imports across the fuzzing suite and types.py through mailgun._httpx_compat. This pays down technical debt and centralizes dependency management, making the SDK's core far more robust to upstream changes in the Python ecosystem.
- Added README snippets for explicit CID 'attach_inline' attachments and 'client.ping()' readiness probes.
- Expanded 'builder_examples.py' with real-world scenarios: chunked streams ('send_large_report_sync'), spam analysis ('send_marketing_campaign'), and client-side exactly-once delivery ('test_idempotency_guard_in_action').
By providing ready-to-copy, proven patterns, we lower the barrier to entry and empower developers to adopt our safest features immediately.
Introducing an exponential backoff retry policy can drastically and artificially inflate the execution time of our CI/CD test pipelines. Concurrently, new integrations need rigorous regression testing without colliding with each other.
- Added 'bypass_retry_delays' fixture in 'conftest.py' to globally mock 'time.sleep' and 'asyncio.sleep' during tests.
- Added assertions for 'LocalSandbox' dry runs, 'AsyncClient' teardown loops, and updated mock transports.
- Segmented list addresses ('python_sdk_sync@...' vs 'python_sdk_async@...') in integration tests to prevent parallel collisions.
This maintains high developer velocity by keeping the test suite lightning fast, while strictly validating that all new features and guardrails operate flawlessly.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Remove the Python 3.10 classifier from pyproject configuration. Eliminate the typing-extensions dependency from all package environments. Update imports across builders, client, security, and types modules to pull Self, TypedDict, and NotRequired directly from the standard typing library. Remove legacy version checks that conditionally loaded types for pre-3.11 Python versions. BREAKING CHANGE: Support for Python 3.10 has been dropped. Consumers must upgrade to Python 3.11 or higher to use this library version. build: bump mypy target python_version to 3.11
Add Pydantic to the optional dependencies list to support strict payload validation. Update the FastAPI optional dependency group to explicitly require Pydantic alongside it. Register Pydantic in the developer environment specification for local testing. Add unit tests for strict payload schema validation via Pydantic extensions.
…ries Introduce TestChunkedStreamer to verify that file attachments are read precisely within memory-bounded chunk limits. Add TestRetryPolicy to validate the mathematical boundaries of the network backoff engine. Ensure exponential growth in the retry policy calculates full jitter accurately without breaching the maximum delay ceiling. Confirm that memory-efficient slots on the retry policy prevent dynamic dictionary allocation. Introduce security guard tests to validate path sanitization and boundary invariants.
Document version 1.9.0 in the changelog, highlighting new features like LocalSandbox, IdempotencyGuard, RetryPolicy, and httpx2 engine compatibility. Add comprehensive examples to the README for Local Sandbox local browser previews. Expand the README to include sections on Advanced Retry Policy, Exactly-Once Delivery, Strict Payload Validation with Pydantic, and Pre-Flight Validation. Clarify the memory-safe benefits of Streaming Pagination in the documentation.
- Mitigate CWE-113 (Header Injection) in builders and pydantic models via strict CRLF checks - Mitigate CWE-400 (Resource Exhaustion) by clamping timeouts to 300s max and enforcing 25MB limits on Pydantic body schemas - Mitigate CWE-294 (Replay Attacks) by adding 15-minute TTL to webhook signature verification - Mitigate CWE-79 (XSS) by injecting strict CSP meta tags in LocalSandbox previews - Mitigate CWE-319 by strictly enforcing TLS 1.2+ for connection pools - Fix false-positive idempotency deduplication by hashing attachment signatures - Update fuzzer dictionary with targeted control character byte sequences
- Fix URL corruption in handlers by removing global '.replace()' calls that mangled enterprise proxies - Fix silent webhook v3-to-v4 upgrade failure by explicitly passing data/filters to the routing handler - Fix async iteration in ChunkedStreamer by offloading file I/O to 'asyncio.to_thread' - Fix missing 'await' on domains.get() inside AsyncClient.ping() - Fix type hints in suppression handlers returning Any instead of str - Enforce string casting for all HTTP headers to prevent serialization crashes - Clamp 'Retry-After' headers against max_delay to prevent infinite sleeping
…efaults - Fix TypeError in RedactingFilter by safely handling standard lists vs NamedTuple unpacking - Prevent LocalSandbox from popping browser tabs in CI/CD environments by default - Abstract MockResponse HTTP error to use framework-agnostic native ApiError - Ensure Config audit_hook initialization state is stable across test reloads - Add async large report streaming example to demonstrate safe memory handling
- Add fuzz_pydantic_models.py to target schema validation and CRLF bounds - Add fuzz_webhooks.py to target temporal offsets and TTL limits - Update fuzz_builders harness to expect CWE-20 and CWE-113 security exceptions - Suppress LocalSandbox browser execution during fuzzing via environment variables - Expand seed_harvester.py with new template, lists, and domain endpoint targets - Add regression tests for Header Injection, Log Overrides, and Proxy URL boundaries
…ctions - Import HealthCheck and settings to suppress heavy filter health checks. - Update hypothesis strategies and exception handling to treat fail-closed security rejections as successful outcomes.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
- Extract payload minification and multipart header cleanup into BaseEndpoint._prepare_payload - Centralize transport exception logging and wrapping into BaseEndpoint._handle_api_error - Initialize the lazy _client property in test_async_unclosed_warning to trigger ResourceWarning correctly - Fix TypeError in TestExceptionSafety by ensuring logger exception and critical methods are properly invoked
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
…ring sanitization' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
…ion calls to satisfy verbose-log-message rule
…o feat/sdk-modernization-phase-2
…al arg traps Ensure safe base directory validation is applied during file attachment streaming. Lock down strict keyword-only parameters to neutralize positional argument bugs during payload construction.
- Add stream pagination fuzzer to catch query-parameter type-drift bugs. - Inject hostile Retry-After header payloads to test math overflow boundaries. - Append new structural bypass tokens to the central fuzzing dictionary. - Expand seed_harvester.py to harvest 12 core endpoint schemas. - Apply Ruff auto-fixes across the fuzzing suite (imports, formatting, unused imports).
…__del__ - Add an explanatory comment in the AttributeError block during Client destructor cleanup to satisfy code quality checks and clearly denote intentional suppression.
…IM105 and add unit tests - Replace try-except-pass blocks in Client.__del__ and AsyncClient.__del__ with contextlib.suppress(Exception) to satisfy Ruff strict linting rules. - Add test_client_unclosed_resource_warning to verify ResourceWarning emission.
… in ChunkedStreamer test - Remove explicit manual invocation of special method __del__() to satisfy linter warnings. - Verify idempotent cleanup by calling streamer.close() a second time.
…, and clean up test finalizers - Update validate_attachment_path to allow OS temp dirs and perform cross-platform sensitive system checks safely. - Refactor test_client_del_attribute_error to use garbage collection instead of explicit __del__ calls. - Patch mailgun.security.Path in test_validate_attachment_path_forbidden_roots to bypass C-optimized pathlib limitations.
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
skupriienko-mailgun
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Links:
Jira
Actions:
Architecture & Developer Experience (DX):
commit_checks.yaml),pyproject.toml, and type definitions to strictly require Python 3.11 through 3.14.dry_run=True): In-memory request interceptor returning deterministicMockResponseobjects during local testing and CI pipelines, bypassing real network calls without writing untrusted I/O files to local disk storage.RetryPolicyEngine: Introduced a flexible retry configuration with exponential backoff and full jitter to handle transient network partitions, respectRetry-Afterheaders, and prevent the "Thundering Herd" effect.httpx2Compatibility Layer: Addedmailgun._httpx_compat.pyto natively bind to the modernhttpx2engine while maintaining a zero-breaking, transparent fallback for legacyhttpxenvironments.mailgun.extEcosystem: Introduced framework-bound extensions including strict Pydantic v2 payload schemas (SendMessageSchema), a plug-and-play FastAPI webhook security dependency (HMAC-SHA256 verification), and a native Django email backend.ChunkedStreamerAttachment Streaming: Implemented memory-safe lazy partitioning for large file attachments (up to 25MB) using 512KB chunks, locking memory consumption toSecurity & Guardrails:
SpamGuard): Built a zero-network static HTML analyzer that fails fast locally if outgoing payloads contain spam triggers, inline XSS scripts, or missing imagealttags before hitting Mailgun servers.IdempotencyGuard): Implemented SHA-256 fingerprinting across JSON payloads and file stream buffers to prevent duplicate email dispatches and double billing during network retries.normalize_domain): Added automatic RFC 3490 Punycode conversion for Internationalized Domain Names to neutralize Unicode routing errors.DeliverabilityErrorto gracefully catch and handleSpamGuardpre-flight policy violations.scorecard.yml) alongside existingpip-audit,Semgrep,CodeQL, andosv-scannersupply-chain gates.Bug Fixes:
AsyncClient.ping()and added__del__socket safety nets to prevent resource descriptor leaks.MockResponse,httpx.Response) inmailgun/types.pythrough_httpx_compat, resolving MyPy forward-reference errors and circular import risks.slotscheckAST Compatibility: Fixed AST inspection errors for slotted classes inheriting from standard library types (e.g.,_SpamGuardParser).Testing, CI/CD & Documentation:
typing-extensionsdependency, adopting standard librarytypingequivalents (Self,TypedDict,NotRequired).manage.sh fuzz_all), Hypothesis stateful property tests, and mock-isolated unit tests targetingSpamGuard,IdempotencyGuard, andRetryPolicy.#memory-safe-attachments-chunkedStreamer,#full-list-of-supported-endpoints,readiness-probe) inREADME.mdto ensure 100% pre-commitmarkdown-link-checkcompliance.Verification & Testing:
To verify these changes locally, ensure your environment variables (
APIKEYandDOMAIN, and others) are set, then run the following commands:1. Run the Unit Test Suite (Fast):
Validates the core routing logic, new guardrails (
SpamGuard,IdempotencyGuard),RetryPolicy, and strictPydanticpayload schemas.2. Run the Live Routing Meta-Test (No state mutation):
Proves the SDK correctly constructs URLs for all supported endpoints by hitting live Mailgun servers (expects 200, 400, 401, or 403 responses; tests fail if the Python SDK crashes or generates a 404 bad route).
3. Run the Full Integration Suite (State mutation):
Executes end-to-end flows against your Sandbox domain (creates/deletes real resources).
4. Execute the Interactive Smoke Test:
Runs the executable documentation script demonstrating cross-version routing and payload serialization.
5. Run the Fuzzing Suite (Security Saturation):
Executes Atheris mutation coverage across core handlers, parsers, and client lifecycles.
6. Run the Performance & Cold-Boot Benchmarks:
Validates the new O(1) routing dispatch and
__slots__memory optimizations using our unified DX script.