webhook-delivery

Internal webhook delivery service. Takes events off the ingest endpoint, queues them, and delivers them to whatever URLs subscribers have registered. Retries on failure, rate limits per subscriber so one noisy consumer can't starve the rest, and parks anything that will never succeed in the dead letter queue.

Owned by the Platform team at Hexstack. Ask in #platform-infra if something's on fire.

Running it

pip install -r requirements.txt
python -m scripts.seed_demo_data       # creates ./webhooks.db with sample rows
python -m webhooks.admin_cli stats

Tests:

pytest

Layout

webhooks/
  config.py        env-driven settings
  models.py        Event, Subscriber, DeliveryAttempt
  queue.py         SQLite-backed durable queue
  delivery.py      the HTTP delivery loop + retry policy
  rate_limit.py    per-subscriber token buckets
  dead_letter.py   permanent-failure parking
  admin_cli.py     ops commands (stats, replay, drain)

v2 migration

This service was migrated to the v2 delivery path in Q3. All delivery now goes through webhooks/delivery.py, which handles retry and backoff internally. The old v1_dispatch module has been removed and nothing imports it any more.

If you're reading old runbooks that mention dispatch_batch() or the WH_DISPATCH_WORKERS env var, those are v1 and no longer apply.

Retry behaviour

Failed deliveries are retried with exponential backoff up to WH_MAX_ATTEMPTS times. After that the event is moved to the dead letter queue and an alert fires.

Rate limiting

Each subscriber gets a token bucket. Capacity and refill rate are configurable via WH_RATE_LIMIT_CAPACITY and WH_RATE_LIMIT_REFILL.

Known gaps

• No metrics export yet (was going to be part of v2, didn't land).
• Admin CLI has no auth. It's internal-only, but still.
• Test suite is thin. A couple of tests are currently disabled.