From f1ca34b7576fe8759c5b11a5e139a7c329abc7b2 Mon Sep 17 00:00:00 2001 From: ylembachar Date: Wed, 5 Aug 2026 12:41:07 +0200 Subject: [PATCH 1/4] Add per-key rate limit tiers, move limit policy to limits.yaml Every authenticated key shared one @withApiKey rate limit block, so raising a limit for one customer raised it for all of them. The raised values also only existed as a manual edit to a tracked file on the API host, where a reset, stash or fresh clone silently reverts them. Limits now live in apikeys/limits.yaml as nine endpoint bases and three tier multipliers, and apikeys.py compiles them together with the keys into the snippet caddy already imports. docker-compose.rate_limit.yaml drops from 273 lines to 70. keys.csv gains an optional tier column; two-column rows read as standard, so the existing file works untouched. Anonymous and standard limits are unchanged from what is deployed today. Premium is 5x standard. @withApiKey is still generated even though rate limiting no longer matches on it: /check_authentication handles on that matcher, for keys of any tier. Co-Authored-By: Claude Opus 5 --- apikeys/README.md | 63 +++++++-- apikeys/apikeys.py | 251 ++++++++++++++++++++++++++++----- apikeys/limits.yaml | 92 ++++++++++++ docker-compose.rate_limit.yaml | 146 ++----------------- 4 files changed, 367 insertions(+), 185 deletions(-) create mode 100644 apikeys/limits.yaml diff --git a/apikeys/README.md b/apikeys/README.md index 8dec4f4..21f3caf 100644 --- a/apikeys/README.md +++ b/apikeys/README.md @@ -1,31 +1,64 @@ # API key management -Bearer tokens are enforced by Caddy, not the Go service. `apikeys.py` maintains -`keys.csv` (`username,apikey` rows, the source of truth) and compiles it into the -Caddy snippet `apikeys.caddy`. Both live in the data directory -(`${DATA_DIR:-./data}`). Caddy only reads the snippet at startup, so every change -needs a caddy restart. +Bearer tokens and rate limits are enforced by Caddy, not the Go service. +`apikeys.py` reads two files and compiles them into the Caddy snippet +`apikeys.caddy`: + +- **`keys.csv`** — `username,apikey,tier` rows, the source of truth for who has a + key and which tier it is on. Lives in the data directory (`${DATA_DIR:-./data}`). +- **`limits.yaml`** — the rate limits for every tier and endpoint. Lives here, in + the repo, because it is policy rather than per-deployment state. + +You do not run the compile step by hand. The `compiler` service does it, and caddy +waits on it (`service_completed_successfully`), so the snippet is regenerated from +both files on every deploy. It prints the resulting limits — read them in the +compiler's logs to confirm a change landed. + +Caddy only reads the snippet at startup, so a change needs caddy restarted. + +## Tiers + +Every key sits on a tier, which decides its rate limits. `limits.yaml` defines +them — currently `standard` (what every key gets) and `premium` (raised limits for +customers running continuously). A row with no tier column reads as `standard`, so +a `keys.csv` written before tiers existed still works. + +Limits are `base × multiplier`: each endpoint has one base number, each tier one +multiplier. The compiler prints the resolved table, so read that rather than doing +the arithmetic. ## Create a key +This is the only step with no deploy equivalent, since it prompts. Run it through +the compiler service so the host needs nothing but docker. The rate-limit override +is required — the `compiler` service is defined there and nowhere else: + ```bash -KEYS_FILE=./data/keys.csv CADDY_SNIPPET=./data/apikeys.caddy python3 apikeys/apikeys.py +docker compose -f docker-compose.yml -f docker-compose.rate_limit.yaml \ + run --rm -it compiler uv run --script /apikeys.py ``` -Enter a unique reference (e.g. the user's email) at the prompt. This appends the -key to `keys.csv` and recompiles the snippet. Then: +It asks for a unique reference (e.g. the user's email) and then which tier to put +the key on, defaulting to `standard`. `-it` matters: without it the prompts don't +reach you. The key is appended to `keys.csv` and the snippet recompiled. + +Restart caddy, then: ```bash -docker compose -f docker-compose.yml -f docker-compose.rate_limit.yaml restart caddy tail -1 data/keys.csv | cut -d, -f2 # the new token curl -H "Authorization: Bearer " https:///check_authentication ``` -## Revoke a key +## Move a key between tiers -Delete its row from `data/keys.csv`, then recompile and restart caddy: +Edit the `tier` column of its row in `keys.csv` and redeploy. The token does not +change, so there is nothing to re-issue to the customer. -```bash -KEYS_FILE=./data/keys.csv CADDY_SNIPPET=./data/apikeys.caddy python3 apikeys/apikeys.py --compile -docker compose -f docker-compose.yml -f docker-compose.rate_limit.yaml restart caddy -``` \ No newline at end of file +## Change a tier's limits + +Edit `limits.yaml` and redeploy. Check the compiler's printed table to confirm you +changed what you meant to. + +## Revoke a key + +Delete its row from `keys.csv` and redeploy. \ No newline at end of file diff --git a/apikeys/apikeys.py b/apikeys/apikeys.py index 456dfd9..748056c 100755 --- a/apikeys/apikeys.py +++ b/apikeys/apikeys.py @@ -2,93 +2,272 @@ # /// script # requires-python = ">=3.13" # dependencies = [ +# "pyyaml", # ] # /// """ -Utility for handling API keys with caddy. There are two functions: +Utility for handling API keys and rate limits with caddy. There are two functions: - generating new keys and store them in the user database .csv (${KEYS_FILE}) - compiling a Caddyfile snippet for using those keys (${CADDY_SNIPPET}) -Without arguments it is started interactively and will ask for a reference for a new key. +Without arguments it is started interactively and will ask for a reference for a new +key (e.g. an email) and which tier to put it on. If used in a docker-compose setup, it can be run with the `--compile` flag, that will non-interactively compile the `Caddyfile` snippet to use. + +Tiers +----- +Each key sits on a tier, which decides its rate limits. The tiers and their limits are +defined in ${LIMITS_FILE} (see limits.yaml); the key database records which tier each +key is on, as a third column: + + username,apikey,tier + alice@example.com,<64 hex chars>,standard + bob@example.com,<64 hex chars>,premium + +Rows with no tier column read as standard, so a database written before tiers existed +still works. To move an existing key between tiers, edit that column and re-run with +--compile. + +The compiled snippet contains both the API key matchers and the rate limit zones — +limits are not configured as caddy labels in docker-compose, so the whole policy lives +in one readable file. Every compile prints the limits it produced. """ import os import sys import secrets -from typing import Dict +from typing import Any, Dict, List, NamedTuple + +import yaml KEYS_FILE = os.environ.get("KEYS_FILE", "users.csv") CADDY_SNIPPET = os.environ.get("CADDY_SNIPPET", "apikeys") +LIMITS_FILE = os.environ.get("LIMITS_FILE", "limits.yaml") + +# The tier for requests with no key. It has no entries in keys.csv: its matcher is +# built by inverting every known key, and it is counted per IP rather than per key. +UNAUTHENTICATED_TIER = "unauthorized" + +# Sole condition of a tier matcher with no keys in it. A named matcher with no +# conditions matches every request, which would hand that tier's limits to +# unauthenticated traffic — so an empty tier gets a condition nothing satisfies. +NEVER_MATCHES = "no-keys-in-this-tier" + +TAB = "\t" + + +class User(NamedTuple): + key: str + tier: str + + +class Limits(NamedTuple): + window: str + tiers: Dict[str, Dict[str, str]] # tier -> {matcher, key} + endpoints: List[Dict[str, Any]] + + @property + def assignable_tiers(self) -> List[str]: + """Tiers a key can be put on — everything except the no-key tier.""" + return [t for t in self.tiers if t != UNAUTHENTICATED_TIER] def generate_token() -> str: return secrets.token_hex(32) -def read_users() -> Dict[str, str]: +def read_limits() -> Limits: try: - with open(KEYS_FILE) as f: - users = { - name.strip(): key.strip() - for name, key in [l.split(",") for l in f.readlines()] - } - if "username" in users: - users.pop("username") - if any(len(key) < 64 for key in users.values()): + with open(LIMITS_FILE) as f: + raw = yaml.safe_load(f) + except FileNotFoundError: + sys.exit(f"Limits file '{LIMITS_FILE}' not found") + + for field in ("window", "tiers", "endpoints"): + if not raw.get(field): + sys.exit(f"'{LIMITS_FILE}' is missing '{field}'") + + for tier, cfg in raw["tiers"].items(): + for field in ("matcher", "key", "multiplier"): + if cfg.get(field) is None: + sys.exit(f"Tier '{tier}' in '{LIMITS_FILE}' is missing '{field}'") + if not isinstance(cfg["multiplier"], int) or cfg["multiplier"] < 1: + sys.exit( + f"Tier '{tier}' has multiplier {cfg['multiplier']!r}; must be a " + f"positive integer" + ) + + for endpoint in raw["endpoints"]: + for field in ("name", "path", "method", "base"): + if endpoint.get(field) is None: + sys.exit(f"Endpoint entry in '{LIMITS_FILE}' is missing '{field}': {endpoint}") + if not isinstance(endpoint["base"], int) or endpoint["base"] < 1: sys.exit( - f"Malformed keys: {list(filter(lambda _: len(_) < 64, users.values()))}" + f"Endpoint '{endpoint['name']}' has base {endpoint['base']!r}; must " + f"be a positive integer" ) - return users + + return Limits(str(raw["window"]), raw["tiers"], raw["endpoints"]) + + +def read_users(limits: Limits) -> Dict[str, User]: + """Read the key database. Rows are `username,apikey[,tier]`. + + The tier column is optional: a two-column row (the format before tiers + existed) reads as standard, so an untouched keys.csv keeps working. + """ + default_tier = "standard" + if default_tier not in limits.assignable_tiers: + sys.exit(f"'{LIMITS_FILE}' must define a '{default_tier}' tier") + + try: + with open(KEYS_FILE) as f: + rows = [line.strip().split(",") for line in f if line.strip()] except FileNotFoundError: return {} + users: Dict[str, User] = {} + for row in rows: + if len(row) == 2: + name, key, tier = row[0], row[1], default_tier + elif len(row) == 3: + name, key, tier = row + else: + sys.exit(f"Malformed row in '{KEYS_FILE}': {','.join(row)}") + + name, key, tier = name.strip(), key.strip(), tier.strip() or default_tier + if name == "username": # header row + continue + if tier not in limits.assignable_tiers: + sys.exit( + f"Unknown tier '{tier}' for '{name}'. '{LIMITS_FILE}' defines: " + f"{', '.join(limits.assignable_tiers)}" + ) + users[name] = User(key, tier) + + malformed = [u.key for u in users.values() if len(u.key) < 64] + if malformed: + sys.exit(f"Malformed keys: {malformed}") + return users + -def dump_users(users: Dict[str, str]) -> None: +def dump_users(users: Dict[str, User]) -> None: with open(KEYS_FILE, "wb") as f: - f.write(b"username,apikey\n") - f.writelines([f"{user},{key}\n".encode() for user, key in users.items()]) + f.write(b"username,apikey,tier\n") + f.writelines( + [f"{name},{u.key},{u.tier}\n".encode() for name, u in users.items()] + ) print(f"Wrote user database to '{KEYS_FILE}'") -def compile(users: Dict[str, str]) -> None: - tab = "\t" +def write_matcher(f, name: str, users: Dict[str, User], negate: bool = False) -> None: + """Write a named matcher matching any one of the given keys. + + Caddy ORs multiple values for the same header field, so listing every key + means "any of these". It ANDs separate conditions, which is what turns the + negated form into "none of these". + """ + prefix = "not " if negate else "" + f.write(f"@{name} {{\n".encode()) + if not users: + f.write(f"{TAB}#no keys in this tier\n".encode()) + f.write(f'{TAB}{prefix}header Authorization "Bearer {NEVER_MATCHES}"\n'.encode()) + for user, u in users.items(): + f.write(f"{TAB}#api key for {user}\n".encode()) + f.write(f'{TAB}{prefix}header Authorization "Bearer {u.key}"\n'.encode()) + f.write(b"}\n\n") + + +def write_rate_limits(f, limits: Limits) -> None: + """Write one rate_limit block per tier, with one zone per endpoint.""" + for tier, cfg in limits.tiers.items(): + f.write(f"rate_limit {cfg['matcher']} {{\n".encode()) + # A bare flag. The docker-compose labels this replaced spelled it + # `log_key: " "` because caddy-docker-proxy uses a single-space value to + # mean "directive with no arguments" — that space is not an argument, and + # passing it through is a parse error in real Caddyfile syntax. + f.write(f"{TAB}log_key\n".encode()) + for endpoint in limits.endpoints: + f.write(f"{TAB}zone {endpoint['name']}__{tier} {{\n".encode()) + f.write(f"{TAB * 2}match {{\n".encode()) + f.write(f"{TAB * 3}path {endpoint['path']}\n".encode()) + f.write(f"{TAB * 3}method {endpoint['method']}\n".encode()) + f.write(f"{TAB * 2}}}\n".encode()) + f.write(f"{TAB * 2}key {cfg['key']}\n".encode()) + f.write(f"{TAB * 2}window {limits.window}\n".encode()) + f.write(f"{TAB * 2}events {endpoint['base'] * cfg['multiplier']}\n".encode()) + f.write(f"{TAB}}}\n".encode()) + f.write(b"}\n\n") + + +def compile(users: Dict[str, User], limits: Limits) -> None: if len(users) == 0: - users["THROWAWAY DO NOT USE!!!"] = generate_token() + users["THROWAWAY DO NOT USE!!!"] = User(generate_token(), "standard") + with open(CADDY_SNIPPET, "wb") as f: - f.write(b"@noApiKey {\n") - for user, key in users.items(): - f.write(f"{tab}#api key for {user}\n".encode()) - f.write(f'{tab}not header Authorization "Bearer {key}"\n'.encode()) - f.write(b"}\n\n") - f.write(b"@withApiKey {\n") - for user, key in users.items(): - f.write(f"{tab}#api key for {user}\n".encode()) - f.write(f'{tab}header Authorization "Bearer {key}"\n'.encode()) - f.write(b"}\n\n") + # Requests with no valid key: every key negated, so "none of these". + write_matcher(f, "noApiKey", users, negate=True) + # Any valid key regardless of tier. Rate limiting matches per tier, but + # the /check_authentication handler still uses this one. + write_matcher(f, "withApiKey", users) + for tier in limits.assignable_tiers: + write_matcher( + f, f"{tier}ApiKey", {n: u for n, u in users.items() if u.tier == tier} + ) + write_rate_limits(f, limits) + print(f"Compiled Caddyfile snippet to '{CADDY_SNIPPET}'") + print_resolved(users, limits) + + +def print_resolved(users: Dict[str, User], limits: Limits) -> None: + """Print the limits this compile actually produced. + + The YAML holds bases and multipliers, so the effective numbers are not + visible by reading it. Printing them here keeps them accurate by + construction — a comment stating them would go stale the first time someone + changes a multiplier. + """ + tiers = list(limits.tiers) + width = max(len(e["name"]) for e in limits.endpoints) + header = f" {'endpoint':<{width}}" + "".join(f"{t:>14}" for t in tiers) + print(header) + for endpoint in limits.endpoints: + row = f" {endpoint['name']:<{width}}" + for tier in tiers: + row += f"{endpoint['base'] * limits.tiers[tier]['multiplier']:>14}" + print(row) + counts = ", ".join( + f"{t}={sum(1 for u in users.values() if u.tier == t)}" + for t in limits.assignable_tiers + ) + print(f" per {limits.window}, keys per tier: {counts}") if __name__ == "__main__": + limits = read_limits() + if len(sys.argv) > 1 and sys.argv[1] == "--compile": - users = read_users() - compile(users) + compile(read_users(limits), limits) sys.exit(0) user = input( "User reference (e.g. email) for new key. Empty for only compiling Caddyfile snippet: " ).strip() - users = read_users() + users = read_users(limits) if len(user) > 0 and len(user) < 3: sys.exit("User name needs to be >=3 characters") if user in users.keys(): sys.exit("User name not unique") - token = generate_token() if len(user): - users[user] = token + options = "/".join(limits.assignable_tiers) + tier = input(f"Tier for this key ({options}) [standard]: ").strip() or "standard" + if tier not in limits.assignable_tiers: + sys.exit(f"Unknown tier '{tier}'. Expected one of: {options}") + users[user] = User(generate_token(), tier) dump_users(users) - compile(users) + compile(users, limits) \ No newline at end of file diff --git a/apikeys/limits.yaml b/apikeys/limits.yaml new file mode 100644 index 0000000..5bc974e --- /dev/null +++ b/apikeys/limits.yaml @@ -0,0 +1,92 @@ +# Rate limiting policy for the Shutter API. +# +# apikeys.py compiles this file, together with the API keys in keys.csv, into the +# Caddy snippet that the caddy container imports. Editing a number here and +# restarting the stack is the whole workflow — there is no other place limits are +# configured. +# +# A limit is `endpoint base × tier multiplier`. All counts are requests per +# `window`. + +window: 1d + +tiers: + # Requests carrying no valid API key. Counted per source IP, because there is + # no key to count against. Its matcher is generated by inverting every known + # key, so no key is ever assigned to this tier. + # + # Multiplier 1 makes the bases below the anonymous allowance by definition. + unauthorized: + matcher: "@noApiKey" + key: "{remote_host}" + multiplier: 1 + + # What every API key gets unless keys.csv says otherwise. base × 100 is the set + # of values published in README.md — keep the two in sync. + standard: + matcher: "@standardApiKey" + key: "{header.Authorization}" + multiplier: 100 + + # Raised limits for customers running continuously rather than experimenting. + # Assign by putting `premium` in the tier column of keys.csv. + # + # 5x standard covers roughly one round per minute on the time-based endpoints: + # 1440 registrations/day fits under 2500, and polling for the key at ~4 reads + # per round fits under 10000. + premium: + matcher: "@premiumApiKey" + key: "{header.Authorization}" + multiplier: 500 + +# One entry per rate-limited endpoint. `base` is the unauthenticated allowance; +# every tier scales from it. +endpoints: + # Registration is a transaction paid from our signer, so this limit is a + # spending ceiling as much as a rate limit. + - name: register_identity + path: "*/time/register_identity*" + method: POST + base: 5 + + - name: get_data_for_encryption + path: "*/time/get_data_for_encryption*" + method: GET + base: 10 + + # Polled repeatedly per round, so it sits well above the registration limit. + - name: get_decryption_key + path: "*/time/get_decryption_key*" + method: GET + base: 20 + + - name: decrypt_commitment + path: "*/decrypt_commitment*" + method: GET + base: 10 + + - name: compile_event_trigger_definition + path: "*/event/compile_trigger_definition*" + method: POST + base: 20 + + # As with the time-based one, each registration spends gas. + - name: register_event_identity + path: "*/event/register_identity*" + method: POST + base: 5 + + - name: get_event_trigger_expiration_block + path: "*/event/get_trigger_expiration_block*" + method: GET + base: 20 + + - name: get_event_decryption_key + path: "*/event/get_decryption_key*" + method: GET + base: 20 + + - name: event_get_data_for_encryption + path: "*/event/get_data_for_encryption*" + method: GET + base: 10 \ No newline at end of file diff --git a/docker-compose.rate_limit.yaml b/docker-compose.rate_limit.yaml index 625e62e..347bac6 100644 --- a/docker-compose.rate_limit.yaml +++ b/docker-compose.rate_limit.yaml @@ -6,6 +6,12 @@ # docker compose -f docker-compose.yml -f docker-compose.rate_limit.yaml up -d # ``` # +# Use together with redirects if the deployment needs them — leaving the redirect +# override out silently drops the old-endpoint redirects: +# ``` +# docker compose -f docker-compose.yml -f docker-compose.redirect.yaml -f docker-compose.rate_limit.yaml up -d +# ``` +# # Note: the custom caddy container needs to be build before use: # # ``` @@ -20,9 +26,11 @@ services: image: ghcr.io/astral-sh/uv:python3.13-alpine volumes: - ./apikeys/apikeys.py:/apikeys.py + - ./apikeys/limits.yaml:/limits.yaml - ${DATA_DIR:-./data}:/data environment: - KEYS_FILE=/data/keys.csv + - LIMITS_FILE=/limits.yaml - CADDY_SNIPPET=/data/apikeys.caddy command: uv run --script /apikeys.py --compile shutter-api: @@ -43,139 +51,9 @@ services: caddy.handle_errors.respond: "`{\"error\": \"Request is rate limited. See documentation! https://github.com/shutter-network/shutter-api?tab=readme-ov-file#rate-limits--authorization\", \"retry_after_seconds\": {http.response.header.Retry-After}, \"status\": {err.status_code}}`" caddy.handle_errors.header.Content-type: "application/json" - # Rate limits unauthorized - caddy.rate_limit_0: "@noApiKey" - caddy.rate_limit_0.log_key: " " - - caddy.rate_limit_0.zone_0: register_identity__unauthorized - caddy.rate_limit_0.zone_0.key: "{remote_host}" - caddy.rate_limit_0.zone_0.events: 5 - caddy.rate_limit_0.zone_0.window: 1d - caddy.rate_limit_0.zone_0.match.path: "*/time/register_identity*" - caddy.rate_limit_0.zone_0.match.method: POST - - caddy.rate_limit_0.zone_1: get_data_for_encryption__unauthorized - caddy.rate_limit_0.zone_1.key: "{remote_host}" - caddy.rate_limit_0.zone_1.events: 10 - caddy.rate_limit_0.zone_1.window: 1d - caddy.rate_limit_0.zone_1.match.path: "*/time/get_data_for_encryption*" - caddy.rate_limit_0.zone_1.match.method: GET - - caddy.rate_limit_0.zone_2: get_decryption_key__unauthorized - caddy.rate_limit_0.zone_2.key: "{remote_host}" - caddy.rate_limit_0.zone_2.events: 20 - caddy.rate_limit_0.zone_2.window: 1d - caddy.rate_limit_0.zone_2.match.path: "*/time/get_decryption_key*" - caddy.rate_limit_0.zone_2.match.method: GET - - caddy.rate_limit_0.zone_3: decrypt_commitment__unauthorized - caddy.rate_limit_0.zone_3.key: "{remote_host}" - caddy.rate_limit_0.zone_3.events: 10 - caddy.rate_limit_0.zone_3.window: 1d - caddy.rate_limit_0.zone_3.match.path: "*/decrypt_commitment*" - caddy.rate_limit_0.zone_3.match.method: GET - - caddy.rate_limit_0.zone_4: compile_event_trigger_definition__unauthorized - caddy.rate_limit_0.zone_4.key: "{remote_host}" - caddy.rate_limit_0.zone_4.events: 20 - caddy.rate_limit_0.zone_4.window: 1d - caddy.rate_limit_0.zone_4.match.path: "*/event/compile_trigger_definition*" - caddy.rate_limit_0.zone_4.match.method: POST - - caddy.rate_limit_0.zone_5: register_event_identity__unauthorized - caddy.rate_limit_0.zone_5.key: "{remote_host}" - caddy.rate_limit_0.zone_5.events: 5 - caddy.rate_limit_0.zone_5.window: 1d - caddy.rate_limit_0.zone_5.match.path: "*/event/register_identity*" - caddy.rate_limit_0.zone_5.match.method: POST - - caddy.rate_limit_0.zone_6: get_event_trigger_expiration_block__unauthorized - caddy.rate_limit_0.zone_6.key: "{remote_host}" - caddy.rate_limit_0.zone_6.events: 20 - caddy.rate_limit_0.zone_6.window: 1d - caddy.rate_limit_0.zone_6.match.path: "*/event/get_trigger_expiration_block*" - caddy.rate_limit_0.zone_6.match.method: GET - - caddy.rate_limit_0.zone_7: get_event_decryption_key__unauthorized - caddy.rate_limit_0.zone_7.key: "{remote_host}" - caddy.rate_limit_0.zone_7.events: 20 - caddy.rate_limit_0.zone_7.window: 1d - caddy.rate_limit_0.zone_7.match.path: "*/event/get_decryption_key*" - caddy.rate_limit_0.zone_7.match.method: GET - - caddy.rate_limit_0.zone_8: event_get_data_for_encryption__unauthorized - caddy.rate_limit_0.zone_8.key: "{remote_host}" - caddy.rate_limit_0.zone_8.events: 10 - caddy.rate_limit_0.zone_8.window: 1d - caddy.rate_limit_0.zone_8.match.path: "*/event/get_data_for_encryption*" - caddy.rate_limit_0.zone_8.match.method: GET - - # Rate limits with api key - caddy.rate_limit_1: "@withApiKey" - caddy.rate_limit_1.log_key: " " - - caddy.rate_limit_1.zone_0: register_identity__authorized - caddy.rate_limit_1.zone_0.key: "{header.Authorization}" - caddy.rate_limit_1.zone_0.events: 500 - caddy.rate_limit_1.zone_0.window: 1d - caddy.rate_limit_1.zone_0.match.path: "*/time/register_identity*" - caddy.rate_limit_1.zone_0.match.method: POST - - caddy.rate_limit_1.zone_1: get_data_for_encryption__authorized - caddy.rate_limit_1.zone_1.key: "{header.Authorization}" - caddy.rate_limit_1.zone_1.events: 1000 - caddy.rate_limit_1.zone_1.window: 1d - caddy.rate_limit_1.zone_1.match.path: "*/time/get_data_for_encryption*" - caddy.rate_limit_1.zone_1.match.method: GET - - caddy.rate_limit_1.zone_2: get_decryption_key__authorized - caddy.rate_limit_1.zone_2.key: "{header.Authorization}" - caddy.rate_limit_1.zone_2.events: 2000 - caddy.rate_limit_1.zone_2.window: 1d - caddy.rate_limit_1.zone_2.match.path: "*/time/get_decryption_key*" - caddy.rate_limit_1.zone_2.match.method: GET - - caddy.rate_limit_1.zone_3: decrypt_commitment__authorized - caddy.rate_limit_1.zone_3.key: "{header.Authorization}" - caddy.rate_limit_1.zone_3.events: 1000 - caddy.rate_limit_1.zone_3.window: 1d - caddy.rate_limit_1.zone_3.match.path: "*/decrypt_commitment*" - caddy.rate_limit_1.zone_3.match.method: GET - - caddy.rate_limit_1.zone_4: compile_event_trigger_definition__authorized - caddy.rate_limit_1.zone_4.key: "{header.Authorization}" - caddy.rate_limit_1.zone_4.events: 2000 - caddy.rate_limit_1.zone_4.window: 1d - caddy.rate_limit_1.zone_4.match.path: "*/event/compile_trigger_definition*" - caddy.rate_limit_1.zone_4.match.method: POST - - caddy.rate_limit_1.zone_5: register_event_identity__authorized - caddy.rate_limit_1.zone_5.key: "{header.Authorization}" - caddy.rate_limit_1.zone_5.events: 500 - caddy.rate_limit_1.zone_5.window: 1d - caddy.rate_limit_1.zone_5.match.path: "*/event/register_identity*" - caddy.rate_limit_1.zone_5.match.method: POST - - caddy.rate_limit_1.zone_6: get_event_trigger_expiration_block__authorized - caddy.rate_limit_1.zone_6.key: "{header.Authorization}" - caddy.rate_limit_1.zone_6.events: 2000 - caddy.rate_limit_1.zone_6.window: 1d - caddy.rate_limit_1.zone_6.match.path: "*/event/get_trigger_expiration_block*" - caddy.rate_limit_1.zone_6.match.method: GET - - caddy.rate_limit_1.zone_7: get_event_decryption_key__authorized - caddy.rate_limit_1.zone_7.key: "{header.Authorization}" - caddy.rate_limit_1.zone_7.events: 2000 - caddy.rate_limit_1.zone_7.window: 1d - caddy.rate_limit_1.zone_7.match.path: "*/event/get_decryption_key*" - caddy.rate_limit_1.zone_7.match.method: GET - - caddy.rate_limit_1.zone_8: event_get_data_for_encryption__authorized - caddy.rate_limit_1.zone_8.key: "{header.Authorization}" - caddy.rate_limit_1.zone_8.events: 1000 - caddy.rate_limit_1.zone_8.window: 1d - caddy.rate_limit_1.zone_8.match.path: "*/event/get_data_for_encryption*" - caddy.rate_limit_1.zone_8.match.method: GET + # Rate limit zones are no longer configured here — they are compiled into + # the imported snippet above from apikeys/limits.yaml, which holds every + # tier's limit for every endpoint in one table. Edit that file, not this one. caddy: build: @@ -188,4 +66,4 @@ services: command: docker-proxy run depends_on: compiler: - condition: service_completed_successfully + condition: service_completed_successfully \ No newline at end of file From f33484c7e72451dc386c78127ec32ba66e911521 Mon Sep 17 00:00:00 2001 From: ylembachar Date: Wed, 5 Aug 2026 12:50:04 +0200 Subject: [PATCH 2/4] Fix shell-form ENTRYPOINT in the caddy image MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ENTRYPOINT without brackets makes Docker run /bin/sh -c and discard all arguments, so the CMD ["docker-proxy"] below it could never take effect, and /bin/sh was PID 1 — SIGTERM never reached caddy, so docker stop always ended in a SIGKILL with no graceful shutdown. Latent because the compose file overrides the entrypoint with a form compose word-splits into exec form. Surfaced when running the image directly to validate a generated config. Co-Authored-By: Claude Opus 5 --- caddy/Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/caddy/Dockerfile b/caddy/Dockerfile index c679ba1..e8dada3 100644 --- a/caddy/Dockerfile +++ b/caddy/Dockerfile @@ -9,6 +9,6 @@ FROM caddy:2.10.0 COPY --from=builder /usr/bin/caddy /usr/bin/caddy -ENTRYPOINT /usr/bin/caddy +ENTRYPOINT ["/usr/bin/caddy"] -CMD ["docker-proxy"] +CMD ["docker-proxy"] \ No newline at end of file From e129c33a509e81c7688b2aa950e29b3f421a282d Mon Sep 17 00:00:00 2001 From: ylembachar Date: Wed, 5 Aug 2026 15:13:02 +0200 Subject: [PATCH 3/4] Document the standard and premium rate limit tiers Rate limits become their own section rather than a subsection of Prerequisites, and are findable from the table of contents. Lists anonymous, standard and premium limits, with one contact line covering both key tiers. States up front that the limits apply to Gnosis Mainnet and that Chiado is unlimited, which was previously mentioned twice and half-implied. Drops the note about event-based triggers not being fully operational on Mainnet, which is no longer true. Co-Authored-By: Claude Opus 5 --- README.md | 106 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 60 insertions(+), 46 deletions(-) diff --git a/README.md b/README.md index bad4a13..2e2287e 100644 --- a/README.md +++ b/README.md @@ -18,21 +18,22 @@ This guide will help you integrate Shutter's Commit and Reveal Scheme into your 1. [Overview](#overview) 2. [Prerequisites](#prerequisites) -3. [Endpoints](#endpoints) - - [Identity Registration](#1-identity-registration) - - [Register an Identity with Time-based Decryption Triggers](#1a-register-an-identity-with-time-based-decryption-triggers) - - [Compile an Event Trigger Definition](#1b-compile-an-event-trigger-definition) - - [Register an Identity with Event-based Decryption Triggers](#1c-register-an-identity-with-event-based-decryption-triggers) - - [Get Event Trigger Identity Registration Expiration Block](#1d-get-event-trigger-identity-registration-expiration-block) - - [Encryption Operations](#2-encryption-operations) - - [Retrieve the Encryption Data](#2a-retrieve-the-encryption-data) - - [Decryption Operations](#3-decryption-operations) - - [Retrieve the Decryption Key](#3a-retrieve-the-decryption-key) - - [Decrypt Commitments](#3b-decrypt-commitments) -4. [Future features](#future-features) -5. [FAQs](#faqs) -6. [Swagger Documentation](#swagger-documentation) -7. [Support](#support) +3. [Rate limits / Authorization](#rate-limits--authorization) +4. [Endpoints](#endpoints) + - [Identity Registration](#1-identity-registration) + - [Register an Identity with Time-based Decryption Triggers](#1a-register-an-identity-with-time-based-decryption-triggers) + - [Compile an Event Trigger Definition](#1b-compile-an-event-trigger-definition) + - [Register an Identity with Event-based Decryption Triggers](#1c-register-an-identity-with-event-based-decryption-triggers) + - [Get Event Trigger Identity Registration Expiration Block](#1d-get-event-trigger-identity-registration-expiration-block) + - [Encryption Operations](#2-encryption-operations) + - [Retrieve the Encryption Data](#2a-retrieve-the-encryption-data) + - [Decryption Operations](#3-decryption-operations) + - [Retrieve the Decryption Key](#3a-retrieve-the-decryption-key) + - [Decrypt Commitments](#3b-decrypt-commitments) +5. [Future features](#future-features) +6. [FAQs](#faqs) +7. [Swagger Documentation](#swagger-documentation) +8. [Support](#support) --- @@ -68,36 +69,49 @@ This documentation will guide you through: - **Chiado Address**: `0xd150bbf86C686de1a25820A94c2C2397e0bC54ab` - **Gnosis Address**: `0x228DefCF37Da29475F0EE2B9E4dfAeDc3b0746bc` -### Rate limits / Authorization +## Rate limits / Authorization -For unauthorized access, the API on Gnosis Mainnet is rate limited with these limits per endpoint and remote ip. -**Please note that we are currently in the process of deploying event-based triggers to Gnosis Mainnet and that they are not fully operational yet.** +These limits apply on Gnosis Mainnet. Chiado is unlimited, which makes it the better choice for development. - - `/time/register_identity` 5 requests per 24 hours - - `/time/get_data_for_encryption` 10 requests per 24 hours - - `/time/get_decryption_key` 20 requests per 24 hours - - `/event/compile_trigger_definition` 20 requests per 24 hours - - `/event/register_identity` 5 requests per 24 hours - - `/event/get_data_for_encryption` 10 requests per 24 hours - - `/event/get_trigger_expiration_block` 20 requests per 24 hours - - `/event/get_decryption_key` 20 requests per 24 hours - - `/decrypt_commitment` 10 requests per 24 hours +On Mainnet, requests without an API key are rate limited per endpoint and remote IP: -We recommend using Chiado for development, because there are no rate limits in place. +- `/time/register_identity` 5 requests per 24 hours +- `/time/get_data_for_encryption` 10 requests per 24 hours +- `/time/get_decryption_key` 20 requests per 24 hours +- `/event/compile_trigger_definition` 20 requests per 24 hours +- `/event/register_identity` 5 requests per 24 hours +- `/event/get_data_for_encryption` 10 requests per 24 hours +- `/event/get_trigger_expiration_block` 20 requests per 24 hours +- `/event/get_decryption_key` 20 requests per 24 hours +- `/decrypt_commitment` 10 requests per 24 hours -If you need higher limits, contact [loring@brainbot.com](mailto:loring@brainbot.com) to request an API key. +API keys come on one of two tiers. Contact [loring@brainbot.com](mailto:loring@brainbot.com) if you are interested in either. -Authorized requests have these limits: +**Standard** limits: - - `/time/register_identity` 500 requests per 24 hours - - `/time/get_data_for_encryption` 1000 requests per 24 hours - - `/time/get_decryption_key` 2000 requests per 24 hours - - `/event/compile_trigger_definition` 2000 requests per 24 hours - - `/event/register_identity` 500 requests per 24 hours - - `/event/get_data_for_encryption` 1000 requests per 24 hours - - `/event/get_trigger_expiration_block` 2000 requests per 24 hours - - `/event/get_decryption_key` 2000 requests per 24 hours - - `/decrypt_commitment` 1000 requests per 24 hours +- `/time/register_identity` 500 requests per 24 hours +- `/time/get_data_for_encryption` 1000 requests per 24 hours +- `/time/get_decryption_key` 2000 requests per 24 hours +- `/event/compile_trigger_definition` 2000 requests per 24 hours +- `/event/register_identity` 500 requests per 24 hours +- `/event/get_data_for_encryption` 1000 requests per 24 hours +- `/event/get_trigger_expiration_block` 2000 requests per 24 hours +- `/event/get_decryption_key` 2000 requests per 24 hours +- `/decrypt_commitment` 1000 requests per 24 hours + +**Premium** limits, for applications running continuously rather than experimenting — roughly one registration per minute, sustained: + +- `/time/register_identity` 2500 requests per 24 hours +- `/time/get_data_for_encryption` 5000 requests per 24 hours +- `/time/get_decryption_key` 10000 requests per 24 hours +- `/event/compile_trigger_definition` 10000 requests per 24 hours +- `/event/register_identity` 2500 requests per 24 hours +- `/event/get_data_for_encryption` 5000 requests per 24 hours +- `/event/get_trigger_expiration_block` 10000 requests per 24 hours +- `/event/get_decryption_key` 10000 requests per 24 hours +- `/decrypt_commitment` 5000 requests per 24 hours + +Limits are counted per API key, over a rolling 24-hour window. Authorization is done by using an `Authorization: Bearer $API_KEY` header, when calling the API. @@ -186,12 +200,12 @@ curl -X POST https:///event/compile_trigger_definition \ > **Notes:** > - Arrays and structs are currently not supported in the arguments. > - The object format for the "arguments" list is: -> - `name`: The matching argument name from the event signature + > - `name`: The matching argument name from the event signature > - `op`: One of `lt`, `lte`, `eq`, `gte`, `gt` for comparison operations > - `number`: Integer argument for numeric comparisons > - `bytes`: Hex-encoded byte argument for non-numeric matches with `op == "eq"` > - Indexed params (topics) are eq‑only. For indexed static types (address, uint256, bytes32), pass the hex representation. -> +> > The resulting condition for the trigger is a logical AND of all arguments given. ### 1.C Register an Identity with Event-based Decryption Triggers @@ -279,11 +293,11 @@ curl -X GET "https:///event/get_data_for_encryption?identityPrefix #### Example Response ```json { -"eon": 1, -"eon_key": "0x57af5437a84ef50e5ed75772c18ae38b168bb07c50cadb65fc6136604e662255", -"identity": "0x8c232eae4f957259e9d6b68301d529e9851b8642874c8f59d2bd0fb84a570c75", -"identity_prefix": "0x79bc8f6b4fcb02c651d6a702b7ad965c7fca19e94a9646d21ae90c8b54c030a0", -"epoch_id": "0x88f2495d1240f9c5523db589996a50a4984ee7a08a8a8f4b269e4345b383310abd2dc1cd9c9c2b8718ed3f486d5242f5" + "eon": 1, + "eon_key": "0x57af5437a84ef50e5ed75772c18ae38b168bb07c50cadb65fc6136604e662255", + "identity": "0x8c232eae4f957259e9d6b68301d529e9851b8642874c8f59d2bd0fb84a570c75", + "identity_prefix": "0x79bc8f6b4fcb02c651d6a702b7ad965c7fca19e94a9646d21ae90c8b54c030a0", + "epoch_id": "0x88f2495d1240f9c5523db589996a50a4984ee7a08a8a8f4b269e4345b383310abd2dc1cd9c9c2b8718ed3f486d5242f5" } ``` From 3ca152d4459db1c4930ff7e9d1d54f847f771d73 Mon Sep 17 00:00:00 2001 From: ylembachar Date: Wed, 5 Aug 2026 18:26:34 +0200 Subject: [PATCH 4/4] test(apikeys): cover the compiler, raise ConfigError instead of sys.exit Validation could not be tested while it called sys.exit, so read_limits and read_users raise ConfigError and __main__ exits with the message. The four public functions take an optional path so tests can use temp files. Adds tests for read_limits, read_users, dump_users and compile, the last compared against saved copies of the generated snippet in testdata/. Malformed-key errors now name the user instead of printing the key, keeping key material out of logs. The generated snippet is byte-identical to the previous version. Co-Authored-By: Claude Opus 5 (1M context) --- .gitignore | 2 + apikeys/README.md | 52 ++- apikeys/apikeys.py | 141 ++++---- apikeys/test_apikeys.py | 339 +++++++++++++++++++ apikeys/testdata/snippet_both_tiers.caddy | 90 +++++ apikeys/testdata/snippet_empty_premium.caddy | 86 +++++ 6 files changed, 648 insertions(+), 62 deletions(-) create mode 100644 apikeys/test_apikeys.py create mode 100644 apikeys/testdata/snippet_both_tiers.caddy create mode 100644 apikeys/testdata/snippet_empty_premium.caddy diff --git a/.gitignore b/.gitignore index 0c0c980..ce3229e 100644 --- a/.gitignore +++ b/.gitignore @@ -28,3 +28,5 @@ pgdata/ .DS_Store bin/ + +__pycache__/ \ No newline at end of file diff --git a/apikeys/README.md b/apikeys/README.md index 21f3caf..2c9f928 100644 --- a/apikeys/README.md +++ b/apikeys/README.md @@ -19,9 +19,13 @@ Caddy only reads the snippet at startup, so a change needs caddy restarted. ## Tiers Every key sits on a tier, which decides its rate limits. `limits.yaml` defines -them — currently `standard` (what every key gets) and `premium` (raised limits for -customers running continuously). A row with no tier column reads as `standard`, so -a `keys.csv` written before tiers existed still works. +them — currently `standard` and `premium`, the latter for customers running +continuously. + +A row with no tier column reads as `standard`, so a `keys.csv` written before tiers +existed still works untouched — and a deploy stays reversible, since the previous +version cannot read a three-column file. Add the column when you first promote +someone, not before. Limits are `base × multiplier`: each endpoint has one base number, each tier one multiplier. The compiler prints the resolved table, so read that rather than doing @@ -61,4 +65,44 @@ changed what you meant to. ## Revoke a key -Delete its row from `keys.csv` and redeploy. \ No newline at end of file +Delete its row from `keys.csv` and redeploy. + +## Tests + +```bash +pytest apikeys/ -v +``` + +Needs `pytest` and `pyyaml`. + +Nothing runs these automatically yet, so run them after changing `apikeys.py`. + +`testdata/` holds a saved copy of the snippet the compiler should produce, and the +test compares its output against that copy. So if you change how the snippet is +built, that test fails and the diff shows you exactly what changed in the config +caddy receives. Read the diff. If the change was intended, re-save the copy: + +```bash +UPDATE_GOLDEN=1 pytest apikeys/ -k compile +``` + +## Verifying a tier change against a deployment + +`test_apikeys.py` covers the generator. To check that a deployment really enforces +different limits per tier, compile from a temporary policy with tiny numbers instead +of making thousands of requests, and never edit `limits.yaml` to do it: + +```bash +# a copy of limits.yaml with window: 1m and base: 1 on every endpoint +docker compose run --rm \ + -v /tmp/limits.test.yaml:/limits.test.yaml -e LIMITS_FILE=/limits.test.yaml compiler +# restart caddy, then burst a key of each tier and note where the 429 lands +``` + +With multipliers of 1 / 3 / 6, anonymous should 429 on request 2, a standard key on 4 +and a premium key on 7. Restoring is a normal compile, since the real `limits.yaml` was +never touched. + +Burst a read endpoint — `get_data_for_encryption` or `get_decryption_key` — so no +transaction is submitted and no gas is spent. Caddy counts the request before proxying, +so it is counted even when the API answers with an error. \ No newline at end of file diff --git a/apikeys/apikeys.py b/apikeys/apikeys.py index 748056c..33adadd 100755 --- a/apikeys/apikeys.py +++ b/apikeys/apikeys.py @@ -21,15 +21,15 @@ ----- Each key sits on a tier, which decides its rate limits. The tiers and their limits are defined in ${LIMITS_FILE} (see limits.yaml); the key database records which tier each -key is on, as a third column: +key is on, in a third column: username,apikey,tier alice@example.com,<64 hex chars>,standard bob@example.com,<64 hex chars>,premium Rows with no tier column read as standard, so a database written before tiers existed -still works. To move an existing key between tiers, edit that column and re-run with ---compile. +still works and a deploy stays reversible. To move a key between tiers, edit that +column and re-run with --compile. The compiled snippet contains both the API key matchers and the rate limit zones — limits are not configured as caddy labels in docker-compose, so the whole policy lives @@ -39,7 +39,7 @@ import os import sys import secrets -from typing import Any, Dict, List, NamedTuple +from typing import Any, Dict, List, NamedTuple, Optional import yaml @@ -52,6 +52,10 @@ # built by inverting every known key, and it is counted per IP rather than per key. UNAUTHENTICATED_TIER = "unauthorized" +# Every key is on this tier unless keys.csv says otherwise, including rows written +# before the tier column existed. +DEFAULT_TIER = "standard" + # Sole condition of a tier matcher with no keys in it. A named matcher with no # conditions matches every request, which would hand that tier's limits to # unauthenticated traffic — so an empty tier gets a condition nothing satisfies. @@ -60,6 +64,10 @@ TAB = "\t" +class ConfigError(Exception): + """Bad input in the key database or the limits file.""" + + class User(NamedTuple): key: str tier: str @@ -67,7 +75,7 @@ class User(NamedTuple): class Limits(NamedTuple): window: str - tiers: Dict[str, Dict[str, str]] # tier -> {matcher, key} + tiers: Dict[str, Dict[str, Any]] # tier -> {matcher, key, multiplier} endpoints: List[Dict[str, Any]] @property @@ -75,28 +83,37 @@ def assignable_tiers(self) -> List[str]: """Tiers a key can be put on — everything except the no-key tier.""" return [t for t in self.tiers if t != UNAUTHENTICATED_TIER] + def events(self, endpoint: Dict[str, Any], tier: str) -> int: + return endpoint["base"] * self.tiers[tier]["multiplier"] + def generate_token() -> str: return secrets.token_hex(32) -def read_limits() -> Limits: +def read_limits(path: Optional[str] = None) -> Limits: + path = path or LIMITS_FILE try: - with open(LIMITS_FILE) as f: + with open(path) as f: raw = yaml.safe_load(f) except FileNotFoundError: - sys.exit(f"Limits file '{LIMITS_FILE}' not found") + raise ConfigError(f"Limits file '{path}' not found") + except yaml.YAMLError as e: + raise ConfigError(f"'{path}' is not valid YAML: {e}") + + if not isinstance(raw, dict): + raise ConfigError(f"'{path}' must be a mapping with window, tiers and endpoints") for field in ("window", "tiers", "endpoints"): if not raw.get(field): - sys.exit(f"'{LIMITS_FILE}' is missing '{field}'") + raise ConfigError(f"'{path}' is missing '{field}'") for tier, cfg in raw["tiers"].items(): for field in ("matcher", "key", "multiplier"): if cfg.get(field) is None: - sys.exit(f"Tier '{tier}' in '{LIMITS_FILE}' is missing '{field}'") + raise ConfigError(f"Tier '{tier}' in '{path}' is missing '{field}'") if not isinstance(cfg["multiplier"], int) or cfg["multiplier"] < 1: - sys.exit( + raise ConfigError( f"Tier '{tier}' has multiplier {cfg['multiplier']!r}; must be a " f"positive integer" ) @@ -104,64 +121,70 @@ def read_limits() -> Limits: for endpoint in raw["endpoints"]: for field in ("name", "path", "method", "base"): if endpoint.get(field) is None: - sys.exit(f"Endpoint entry in '{LIMITS_FILE}' is missing '{field}': {endpoint}") + raise ConfigError( + f"Endpoint entry in '{path}' is missing '{field}': {endpoint}" + ) if not isinstance(endpoint["base"], int) or endpoint["base"] < 1: - sys.exit( + raise ConfigError( f"Endpoint '{endpoint['name']}' has base {endpoint['base']!r}; must " f"be a positive integer" ) - return Limits(str(raw["window"]), raw["tiers"], raw["endpoints"]) + limits = Limits(str(raw["window"]), raw["tiers"], raw["endpoints"]) + if DEFAULT_TIER not in limits.assignable_tiers: + raise ConfigError(f"'{path}' must define a '{DEFAULT_TIER}' tier") + return limits -def read_users(limits: Limits) -> Dict[str, User]: +def read_users(limits: Limits, path: Optional[str] = None) -> Dict[str, User]: """Read the key database. Rows are `username,apikey[,tier]`. - The tier column is optional: a two-column row (the format before tiers - existed) reads as standard, so an untouched keys.csv keeps working. + The tier column is optional; a two-column row reads as standard. """ - default_tier = "standard" - if default_tier not in limits.assignable_tiers: - sys.exit(f"'{LIMITS_FILE}' must define a '{default_tier}' tier") - + path = path or KEYS_FILE try: - with open(KEYS_FILE) as f: + with open(path) as f: rows = [line.strip().split(",") for line in f if line.strip()] except FileNotFoundError: return {} users: Dict[str, User] = {} for row in rows: + if row[0].strip() == "username": # header row, whatever its column count + continue if len(row) == 2: - name, key, tier = row[0], row[1], default_tier + name, key, tier = row[0], row[1], DEFAULT_TIER elif len(row) == 3: name, key, tier = row else: - sys.exit(f"Malformed row in '{KEYS_FILE}': {','.join(row)}") - - name, key, tier = name.strip(), key.strip(), tier.strip() or default_tier - if name == "username": # header row - continue + raise ConfigError( + f"Malformed row in '{path}': expected username,apikey[,tier] — got " + f"{len(row)} fields for '{row[0].strip()}'" + ) + name, key, tier = name.strip(), key.strip(), tier.strip() or DEFAULT_TIER if tier not in limits.assignable_tiers: - sys.exit( - f"Unknown tier '{tier}' for '{name}'. '{LIMITS_FILE}' defines: " + raise ConfigError( + f"Unknown tier '{tier}' for '{name}'. Defined tiers: " f"{', '.join(limits.assignable_tiers)}" ) users[name] = User(key, tier) - malformed = [u.key for u in users.values() if len(u.key) < 64] + # Name the users, not their keys — an error message is somewhere key material + # should never end up. + malformed = [name for name, u in users.items() if len(u.key) < 64] if malformed: - sys.exit(f"Malformed keys: {malformed}") + raise ConfigError(f"Malformed keys for: {', '.join(malformed)}") return users -def dump_users(users: Dict[str, User]) -> None: - with open(KEYS_FILE, "wb") as f: +def dump_users(users: Dict[str, User], path: Optional[str] = None) -> None: + path = path or KEYS_FILE + with open(path, "wb") as f: f.write(b"username,apikey,tier\n") f.writelines( [f"{name},{u.key},{u.tier}\n".encode() for name, u in users.items()] ) - print(f"Wrote user database to '{KEYS_FILE}'") + print(f"Wrote user database to '{path}'") def write_matcher(f, name: str, users: Dict[str, User], negate: bool = False) -> None: @@ -186,10 +209,8 @@ def write_rate_limits(f, limits: Limits) -> None: """Write one rate_limit block per tier, with one zone per endpoint.""" for tier, cfg in limits.tiers.items(): f.write(f"rate_limit {cfg['matcher']} {{\n".encode()) - # A bare flag. The docker-compose labels this replaced spelled it - # `log_key: " "` because caddy-docker-proxy uses a single-space value to - # mean "directive with no arguments" — that space is not an argument, and - # passing it through is a parse error in real Caddyfile syntax. + # A bare flag. `log_key " "` is a caddy-docker-proxy idiom for a directive + # with no arguments, and a parse error in Caddyfile syntax. f.write(f"{TAB}log_key\n".encode()) for endpoint in limits.endpoints: f.write(f"{TAB}zone {endpoint['name']}__{tier} {{\n".encode()) @@ -199,16 +220,17 @@ def write_rate_limits(f, limits: Limits) -> None: f.write(f"{TAB * 2}}}\n".encode()) f.write(f"{TAB * 2}key {cfg['key']}\n".encode()) f.write(f"{TAB * 2}window {limits.window}\n".encode()) - f.write(f"{TAB * 2}events {endpoint['base'] * cfg['multiplier']}\n".encode()) + f.write(f"{TAB * 2}events {limits.events(endpoint, tier)}\n".encode()) f.write(f"{TAB}}}\n".encode()) f.write(b"}\n\n") -def compile(users: Dict[str, User], limits: Limits) -> None: +def compile(users: Dict[str, User], limits: Limits, path: Optional[str] = None) -> None: + path = path or CADDY_SNIPPET if len(users) == 0: - users["THROWAWAY DO NOT USE!!!"] = User(generate_token(), "standard") + users["THROWAWAY DO NOT USE!!!"] = User(generate_token(), DEFAULT_TIER) - with open(CADDY_SNIPPET, "wb") as f: + with open(path, "wb") as f: # Requests with no valid key: every key negated, so "none of these". write_matcher(f, "noApiKey", users, negate=True) # Any valid key regardless of tier. Rate limiting matches per tier, but @@ -220,26 +242,19 @@ def compile(users: Dict[str, User], limits: Limits) -> None: ) write_rate_limits(f, limits) - print(f"Compiled Caddyfile snippet to '{CADDY_SNIPPET}'") + print(f"Compiled Caddyfile snippet to '{path}'") print_resolved(users, limits) def print_resolved(users: Dict[str, User], limits: Limits) -> None: - """Print the limits this compile actually produced. - - The YAML holds bases and multipliers, so the effective numbers are not - visible by reading it. Printing them here keeps them accurate by - construction — a comment stating them would go stale the first time someone - changes a multiplier. - """ + """Print the resolved limits, which limits.yaml only holds as base × multiplier.""" tiers = list(limits.tiers) width = max(len(e["name"]) for e in limits.endpoints) - header = f" {'endpoint':<{width}}" + "".join(f"{t:>14}" for t in tiers) - print(header) + print(f" {'endpoint':<{width}}" + "".join(f"{t:>14}" for t in tiers)) for endpoint in limits.endpoints: row = f" {endpoint['name']:<{width}}" for tier in tiers: - row += f"{endpoint['base'] * limits.tiers[tier]['multiplier']:>14}" + row += f"{limits.events(endpoint, tier):>14}" print(row) counts = ", ".join( f"{t}={sum(1 for u in users.values() if u.tier == t)}" @@ -248,12 +263,12 @@ def print_resolved(users: Dict[str, User], limits: Limits) -> None: print(f" per {limits.window}, keys per tier: {counts}") -if __name__ == "__main__": +def main() -> None: limits = read_limits() if len(sys.argv) > 1 and sys.argv[1] == "--compile": compile(read_users(limits), limits) - sys.exit(0) + return user = input( "User reference (e.g. email) for new key. Empty for only compiling Caddyfile snippet: " @@ -265,9 +280,19 @@ def print_resolved(users: Dict[str, User], limits: Limits) -> None: sys.exit("User name not unique") if len(user): options = "/".join(limits.assignable_tiers) - tier = input(f"Tier for this key ({options}) [standard]: ").strip() or "standard" + tier = ( + input(f"Tier for this key ({options}) [{DEFAULT_TIER}]: ").strip() + or DEFAULT_TIER + ) if tier not in limits.assignable_tiers: sys.exit(f"Unknown tier '{tier}'. Expected one of: {options}") users[user] = User(generate_token(), tier) dump_users(users) - compile(users, limits) \ No newline at end of file + compile(users, limits) + + +if __name__ == "__main__": + try: + main() + except ConfigError as e: + sys.exit(str(e)) \ No newline at end of file diff --git a/apikeys/test_apikeys.py b/apikeys/test_apikeys.py new file mode 100644 index 0000000..048d2d6 --- /dev/null +++ b/apikeys/test_apikeys.py @@ -0,0 +1,339 @@ +""" +A suite of tests for the `apikeys` module, focusing on limits, users, and compilation. + +This module contains test cases for verifying the functionality of key components of +the `apikeys` library. These include reading and validating rate-limiting configurations, +managing user API keys, and generating compiled output for a Caddy server. Tests are +organized into sections that correspond to specific responsibilities. + +It relies on external test data and uses fixtures to manage temporary directories. + +```yaml +Modules imported: +- `os`: Provides operating system functionality such as environment variable checks. +- `textwrap`: Used to dedent multiline strings for clarity. +- `pathlib`: Facilitates interactions with the filesystem and paths. +- `pytest`: A testing framework used to define and parameterize test cases. +- `yaml`: Used for parsing YAML files within the tests. +- `apikeys`: The library under test, which manages API key configurations. + +Constants: +- `TESTDATA`: Directory holding golden files for comparison in some tests. +- `KEY_A` and `KEY_B`: Mock API keys used in parameterized tests. +- `LIMITS`: YAML configuration string that defines rate-limiting tiers and endpoints. +``` + +Functions +--------- + +write(tmp_path, name, content): + Writes the given content to a specified file name within a temporary directory. + + Args: + tmp_path (pathlib.Path): A pytest-supplied temporary directory. + name (str): The name of the file to write. + content (str): The content to write to the file. + + Returns: + str: The file path as a string. + +csv(text): + Strips leading whitespace and formats a CSV text using mock API keys. + + Args: + text (str): A CSV string with placeholders for keys. + + Returns: + str: The formatted CSV text. + +bad_limits(tmp_path, mutate): + Creates an invalid limits configuration by mutating valid limits data. + + Args: + tmp_path (pathlib.Path): A pytest-supplied temporary directory. + mutate (callable): A function that mutates the loaded YAML document. + + Returns: + str: Path to the invalid limits file. + +rename_standard_tier(doc): + Renames the "standard" tier to "silver" in the given limits configuration. + + Args: + doc (dict): A limits configuration document. + +Test Cases +---------- + +test_read_limits(limits): + Verifies the correct parsing of a valid limits configuration. + +test_read_limits_rejects(tmp_path, mutate, expected): + Ensures invalid configurations are rejected and appropriate errors are raised. + +test_read_limits_rejects_unparseable(tmp_path, content): + Confirms errors are raised for malformed or unparseable YAML files. + +test_read_users(tmp_path, limits, content, expected): + Verifies user data is correctly parsed and matches expected output. + +test_read_users_rejects(tmp_path, limits, content, expected): + Ensures errors are raised for invalid or malformed user data. + +test_read_users_errors_never_contain_key_material(tmp_path, limits): + Validates that error messages related to user data do not expose sensitive key material. + +test_dump_users_preserves_every_key(tmp_path, limits): + Confirms that dumping user data does not lose or alter any keys. + +test_compile_matches_golden(tmp_path, limits, users, golden): + Checks that compiled output matches golden files for various tier configurations. +""" + +import os +import textwrap +from pathlib import Path + +import pytest +import yaml + +import apikeys +from apikeys import ConfigError, User + + +TESTDATA = Path(__file__).parent / "testdata" + +KEY_A = "a" * 64 +KEY_B = "b" * 64 + +LIMITS = textwrap.dedent( + """ + window: 1d + tiers: + unauthorized: { matcher: "@noApiKey", key: "{remote_host}", multiplier: 1 } + standard: { matcher: "@standardApiKey", key: "{header.Authorization}", multiplier: 100 } + premium: { matcher: "@premiumApiKey", key: "{header.Authorization}", multiplier: 500 } + endpoints: + - { name: register_identity, path: "*/time/register_identity*", method: POST, base: 5 } + - { name: get_decryption_key, path: "*/time/get_decryption_key*", method: GET, base: 20 } + """ +) + + +def write(tmp_path, name, content): + p = tmp_path / name + p.write_text(content) + return str(p) + + +def csv(text): + return textwrap.dedent(text).lstrip().format(A=KEY_A, B=KEY_B) + + +def bad_limits(tmp_path, mutate): + doc = yaml.safe_load(LIMITS) + mutate(doc) + return write(tmp_path, "limits.yaml", yaml.safe_dump(doc)) + + +@pytest.fixture +def limits(tmp_path): + return apikeys.read_limits(write(tmp_path, "limits.yaml", LIMITS)) + + +def rename_standard_tier(doc): + doc["tiers"]["silver"] = doc["tiers"].pop("standard") + + +# --- read_limits ------------------------------------------------------------------ + + +def test_read_limits(limits): + assert limits.window == "1d" + assert limits.assignable_tiers == ["standard", "premium"] + register, decryption_key = limits.endpoints + assert limits.events(register, "unauthorized") == 5 + assert limits.events(register, "standard") == 500 + assert limits.events(register, "premium") == 2500 + assert limits.events(decryption_key, "premium") == 10000 + + +@pytest.mark.parametrize( + "mutate,expected", + [ + pytest.param( + lambda doc: doc["tiers"]["standard"].update(multiplier=0), + "multiplier", + id="multiplier of zero", + ), + pytest.param( + lambda doc: doc["tiers"]["standard"].update(multiplier="100"), + "multiplier", + id="quoted multiplier", + ), + pytest.param( + lambda doc: doc["tiers"]["standard"].pop("matcher"), + "matcher", + id="tier with no matcher", + ), + pytest.param( + lambda doc: doc["endpoints"][0].pop("base"), + "base", + id="endpoint with no base", + ), + pytest.param(rename_standard_tier, "standard", id="no standard tier"), + pytest.param(lambda doc: doc.pop("tiers"), "tiers", id="no tiers section"), + pytest.param(lambda doc: doc.pop("endpoints"), "endpoints", id="no endpoints"), + ], +) +def test_read_limits_rejects(tmp_path, mutate, expected): + with pytest.raises(ConfigError, match=expected): + apikeys.read_limits(bad_limits(tmp_path, mutate)) + + +@pytest.mark.parametrize( + "content", + [ + pytest.param("window: 1d\ntiers: [\n", id="broken yaml"), + pytest.param("", id="empty file"), + pytest.param("- a\n- b\n", id="a list rather than a mapping"), + ], +) +def test_read_limits_rejects_unparseable(tmp_path, content): + with pytest.raises(ConfigError): + apikeys.read_limits(write(tmp_path, "limits.yaml", content)) + + +# --- read_users ------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "content,expected", + [ + pytest.param( + csv( + """ + username,apikey,tier + alice,{A},standard + bob,{B},premium + """ + ), + {"alice": User(KEY_A, "standard"), "bob": User(KEY_B, "premium")}, + id="a key on each tier", + ), + pytest.param( + csv( + """ + username,apikey + alice,{A} + """ + ), + {"alice": User(KEY_A, "standard")}, + id="two columns, as written before tiers existed", + ), + pytest.param( + csv( + """ + username,apikey + alice,{A} + bob,{B},premium + """ + ), + {"alice": User(KEY_A, "standard"), "bob": User(KEY_B, "premium")}, + id="half migrated, after promoting one key by hand", + ), + pytest.param("", {}, id="empty file"), + ], +) +def test_read_users(tmp_path, limits, content, expected): + assert apikeys.read_users(limits, write(tmp_path, "keys.csv", content)) == expected + + +@pytest.mark.parametrize( + "content,expected", + [ + pytest.param( + csv("username,apikey,tier\nalice,{A},premiun\n"), + "premiun", + id="misspelled tier", + ), + pytest.param( + "username,apikey,tier\nalice,tooshort,standard\n", + "alice", + id="truncated key", + ), + pytest.param( + csv("username,apikey,tier\nalice,{A},standard,extra\n"), + "alice", + id="stray comma", + ), + ], +) +def test_read_users_rejects(tmp_path, limits, content, expected): + with pytest.raises(ConfigError, match=expected): + apikeys.read_users(limits, write(tmp_path, "keys.csv", content)) + + +def test_read_users_errors_never_contain_key_material(tmp_path, limits): + truncated = KEY_A[:10] + keys = write( + tmp_path, "keys.csv", f"username,apikey,tier\nalice,{truncated},standard\n" + ) + with pytest.raises(ConfigError) as e: + apikeys.read_users(limits, keys) + assert "alice" in str(e.value) + assert truncated not in str(e.value) + + +# --- dump_users ------------------------------------------------------------------- + + +def test_dump_users_preserves_every_key(tmp_path, limits): + keys = write( + tmp_path, + "keys.csv", + csv( + """ + username,apikey,tier + alice,{A},standard + bob,{B},premium + """ + ), + ) + before = apikeys.read_users(limits, keys) + + apikeys.dump_users(before, keys) + + assert open(keys).readline().strip() == "username,apikey,tier" + assert apikeys.read_users(limits, keys) == before + + +# --- compile ---------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "users,golden", + [ + pytest.param( + {"alice": User(KEY_A, "standard"), "bob": User(KEY_B, "premium")}, + "snippet_both_tiers.caddy", + id="a key on each tier", + ), + pytest.param( + {"alice": User(KEY_A, "standard")}, + "snippet_empty_premium.caddy", + id="no premium keys", + ), + ], +) +def test_compile_matches_golden(tmp_path, limits, users, golden): + out = tmp_path / "apikeys.caddy" + apikeys.compile(dict(users), limits, str(out)) + produced = out.read_text() + + expected = TESTDATA / golden + if os.environ.get("UPDATE_GOLDEN"): + TESTDATA.mkdir(exist_ok=True) + expected.write_text(produced) + + assert produced == expected.read_text() \ No newline at end of file diff --git a/apikeys/testdata/snippet_both_tiers.caddy b/apikeys/testdata/snippet_both_tiers.caddy new file mode 100644 index 0000000..25260f9 --- /dev/null +++ b/apikeys/testdata/snippet_both_tiers.caddy @@ -0,0 +1,90 @@ +@noApiKey { + #api key for alice + not header Authorization "Bearer aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + #api key for bob + not header Authorization "Bearer bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" +} + +@withApiKey { + #api key for alice + header Authorization "Bearer aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + #api key for bob + header Authorization "Bearer bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" +} + +@standardApiKey { + #api key for alice + header Authorization "Bearer aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" +} + +@premiumApiKey { + #api key for bob + header Authorization "Bearer bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" +} + +rate_limit @noApiKey { + log_key + zone register_identity__unauthorized { + match { + path */time/register_identity* + method POST + } + key {remote_host} + window 1d + events 5 + } + zone get_decryption_key__unauthorized { + match { + path */time/get_decryption_key* + method GET + } + key {remote_host} + window 1d + events 20 + } +} + +rate_limit @standardApiKey { + log_key + zone register_identity__standard { + match { + path */time/register_identity* + method POST + } + key {header.Authorization} + window 1d + events 500 + } + zone get_decryption_key__standard { + match { + path */time/get_decryption_key* + method GET + } + key {header.Authorization} + window 1d + events 2000 + } +} + +rate_limit @premiumApiKey { + log_key + zone register_identity__premium { + match { + path */time/register_identity* + method POST + } + key {header.Authorization} + window 1d + events 2500 + } + zone get_decryption_key__premium { + match { + path */time/get_decryption_key* + method GET + } + key {header.Authorization} + window 1d + events 10000 + } +} + diff --git a/apikeys/testdata/snippet_empty_premium.caddy b/apikeys/testdata/snippet_empty_premium.caddy new file mode 100644 index 0000000..83c94ab --- /dev/null +++ b/apikeys/testdata/snippet_empty_premium.caddy @@ -0,0 +1,86 @@ +@noApiKey { + #api key for alice + not header Authorization "Bearer aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" +} + +@withApiKey { + #api key for alice + header Authorization "Bearer aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" +} + +@standardApiKey { + #api key for alice + header Authorization "Bearer aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" +} + +@premiumApiKey { + #no keys in this tier + header Authorization "Bearer no-keys-in-this-tier" +} + +rate_limit @noApiKey { + log_key + zone register_identity__unauthorized { + match { + path */time/register_identity* + method POST + } + key {remote_host} + window 1d + events 5 + } + zone get_decryption_key__unauthorized { + match { + path */time/get_decryption_key* + method GET + } + key {remote_host} + window 1d + events 20 + } +} + +rate_limit @standardApiKey { + log_key + zone register_identity__standard { + match { + path */time/register_identity* + method POST + } + key {header.Authorization} + window 1d + events 500 + } + zone get_decryption_key__standard { + match { + path */time/get_decryption_key* + method GET + } + key {header.Authorization} + window 1d + events 2000 + } +} + +rate_limit @premiumApiKey { + log_key + zone register_identity__premium { + match { + path */time/register_identity* + method POST + } + key {header.Authorization} + window 1d + events 2500 + } + zone get_decryption_key__premium { + match { + path */time/get_decryption_key* + method GET + } + key {header.Authorization} + window 1d + events 10000 + } +} +