Feat/add rate limit tiers

.gitignore

@@ -28,3 +28,5 @@ pgdata/
 .DS_Store
 
 bin/
+
+__pycache__/

README.md

@@ -18,21 +18,22 @@ This guide will help you integrate Shutter's Commit and Reveal Scheme into your
 
 1. [Overview](#overview)
 2. [Prerequisites](#prerequisites)
-3. [Endpoints](#endpoints)
-   - [Identity Registration](#1-identity-registration)
-     - [Register an Identity with Time-based Decryption Triggers](#1a-register-an-identity-with-time-based-decryption-triggers)
-     - [Compile an Event Trigger Definition](#1b-compile-an-event-trigger-definition)
-     - [Register an Identity with Event-based Decryption Triggers](#1c-register-an-identity-with-event-based-decryption-triggers)
-     - [Get Event Trigger Identity Registration Expiration Block](#1d-get-event-trigger-identity-registration-expiration-block)
-   - [Encryption Operations](#2-encryption-operations)
-     - [Retrieve the Encryption Data](#2a-retrieve-the-encryption-data)
-   - [Decryption Operations](#3-decryption-operations)
-     - [Retrieve the Decryption Key](#3a-retrieve-the-decryption-key)
-     - [Decrypt Commitments](#3b-decrypt-commitments)
-4. [Future features](#future-features)
-5. [FAQs](#faqs)
-6. [Swagger Documentation](#swagger-documentation)
-7. [Support](#support)
+3. [Rate limits / Authorization](#rate-limits--authorization)
+4. [Endpoints](#endpoints)
+  - [Identity Registration](#1-identity-registration)
+    - [Register an Identity with Time-based Decryption Triggers](#1a-register-an-identity-with-time-based-decryption-triggers)
+    - [Compile an Event Trigger Definition](#1b-compile-an-event-trigger-definition)
+    - [Register an Identity with Event-based Decryption Triggers](#1c-register-an-identity-with-event-based-decryption-triggers)
+    - [Get Event Trigger Identity Registration Expiration Block](#1d-get-event-trigger-identity-registration-expiration-block)
+  - [Encryption Operations](#2-encryption-operations)
+    - [Retrieve the Encryption Data](#2a-retrieve-the-encryption-data)
+  - [Decryption Operations](#3-decryption-operations)
+    - [Retrieve the Decryption Key](#3a-retrieve-the-decryption-key)
+    - [Decrypt Commitments](#3b-decrypt-commitments)
+5. [Future features](#future-features)
+6. [FAQs](#faqs)
+7. [Swagger Documentation](#swagger-documentation)
+8. [Support](#support)
 
 ---
 
@@ -68,36 +69,49 @@ This documentation will guide you through:
   - **Chiado Address**: `0xd150bbf86C686de1a25820A94c2C2397e0bC54ab`
   - **Gnosis Address**: `0x228DefCF37Da29475F0EE2B9E4dfAeDc3b0746bc`
 
-### Rate limits / Authorization
+## Rate limits / Authorization
 
-For unauthorized access, the API on Gnosis Mainnet is rate limited with these limits per endpoint and remote ip.
-**Please note that we are currently in the process of deploying event-based triggers to Gnosis Mainnet and that they are not fully operational yet.**
+These limits apply on Gnosis Mainnet. Chiado is unlimited, which makes it the better choice for development.
 
-  - `/time/register_identity` 5 requests per 24 hours
-  - `/time/get_data_for_encryption` 10 requests per 24 hours
-  - `/time/get_decryption_key` 20 requests per 24 hours
-  - `/event/compile_trigger_definition` 20 requests per 24 hours
-  - `/event/register_identity` 5 requests per 24 hours
-  - `/event/get_data_for_encryption` 10 requests per 24 hours
-  - `/event/get_trigger_expiration_block` 20 requests per 24 hours
-  - `/event/get_decryption_key` 20 requests per 24 hours
-  - `/decrypt_commitment` 10 requests per 24 hours
+On Mainnet, requests without an API key are rate limited per endpoint and remote IP:
 
-We recommend using Chiado for development, because there are no rate limits in place.
+- `/time/register_identity` 5 requests per 24 hours
+- `/time/get_data_for_encryption` 10 requests per 24 hours
+- `/time/get_decryption_key` 20 requests per 24 hours
+- `/event/compile_trigger_definition` 20 requests per 24 hours
+- `/event/register_identity` 5 requests per 24 hours
+- `/event/get_data_for_encryption` 10 requests per 24 hours
+- `/event/get_trigger_expiration_block` 20 requests per 24 hours
+- `/event/get_decryption_key` 20 requests per 24 hours
+- `/decrypt_commitment` 10 requests per 24 hours
 
-If you need higher limits, contact [loring@brainbot.com](mailto:loring@brainbot.com) to request an API key.
+API keys come on one of two tiers. Contact [loring@brainbot.com](mailto:loring@brainbot.com) if you are interested in either.
 
-Authorized requests have these limits:
+**Standard** limits:
 
-  - `/time/register_identity` 500 requests per 24 hours
-  - `/time/get_data_for_encryption` 1000 requests per 24 hours
-  - `/time/get_decryption_key` 2000 requests per 24 hours
-  - `/event/compile_trigger_definition` 2000 requests per 24 hours
-  - `/event/register_identity` 500 requests per 24 hours
-  - `/event/get_data_for_encryption` 1000 requests per 24 hours
-  - `/event/get_trigger_expiration_block` 2000 requests per 24 hours
-  - `/event/get_decryption_key` 2000 requests per 24 hours
-  - `/decrypt_commitment` 1000 requests per 24 hours
+- `/time/register_identity` 500 requests per 24 hours
+- `/time/get_data_for_encryption` 1000 requests per 24 hours
+- `/time/get_decryption_key` 2000 requests per 24 hours
+- `/event/compile_trigger_definition` 2000 requests per 24 hours
+- `/event/register_identity` 500 requests per 24 hours
+- `/event/get_data_for_encryption` 1000 requests per 24 hours
+- `/event/get_trigger_expiration_block` 2000 requests per 24 hours
+- `/event/get_decryption_key` 2000 requests per 24 hours
+- `/decrypt_commitment` 1000 requests per 24 hours
+
+**Premium** limits, for applications running continuously rather than experimenting — roughly one registration per minute, sustained:
+
+- `/time/register_identity` 2500 requests per 24 hours
+- `/time/get_data_for_encryption` 5000 requests per 24 hours
+- `/time/get_decryption_key` 10000 requests per 24 hours
+- `/event/compile_trigger_definition` 10000 requests per 24 hours
+- `/event/register_identity` 2500 requests per 24 hours
+- `/event/get_data_for_encryption` 5000 requests per 24 hours
+- `/event/get_trigger_expiration_block` 10000 requests per 24 hours
+- `/event/get_decryption_key` 10000 requests per 24 hours
+- `/decrypt_commitment` 5000 requests per 24 hours
+
+Limits are counted per API key, over a rolling 24-hour window.
 
 Authorization is done by using an `Authorization: Bearer $API_KEY` header, when calling the API.
 
@@ -186,12 +200,12 @@ curl -X POST https://<API_BASE_URL>/event/compile_trigger_definition \
 > **Notes:**
 > - Arrays and structs are currently not supported in the arguments.
 > - The object format for the "arguments" list is:
->   - `name`: The matching argument name from the event signature
+    >   - `name`: The matching argument name from the event signature
 >   - `op`: One of `lt`, `lte`, `eq`, `gte`, `gt` for comparison operations
 >   - `number`: Integer argument for numeric comparisons
 >   - `bytes`: Hex-encoded byte argument for non-numeric matches with `op == "eq"`
 > - Indexed params (topics) are eq‑only. For indexed static types (address, uint256, bytes32), pass the hex representation.
-> 
+>
 > The resulting condition for the trigger is a logical AND of all arguments given.
 
 ### 1.C Register an Identity with Event-based Decryption Triggers
@@ -279,11 +293,11 @@ curl -X GET "https://<API_BASE_URL>/event/get_data_for_encryption?identityPrefix
 #### Example Response
 ```json
 {
-"eon": 1,
-"eon_key": "0x57af5437a84ef50e5ed75772c18ae38b168bb07c50cadb65fc6136604e662255",
-"identity": "0x8c232eae4f957259e9d6b68301d529e9851b8642874c8f59d2bd0fb84a570c75",
-"identity_prefix": "0x79bc8f6b4fcb02c651d6a702b7ad965c7fca19e94a9646d21ae90c8b54c030a0",
-"epoch_id": "0x88f2495d1240f9c5523db589996a50a4984ee7a08a8a8f4b269e4345b383310abd2dc1cd9c9c2b8718ed3f486d5242f5"
+  "eon": 1,
+  "eon_key": "0x57af5437a84ef50e5ed75772c18ae38b168bb07c50cadb65fc6136604e662255",
+  "identity": "0x8c232eae4f957259e9d6b68301d529e9851b8642874c8f59d2bd0fb84a570c75",
+  "identity_prefix": "0x79bc8f6b4fcb02c651d6a702b7ad965c7fca19e94a9646d21ae90c8b54c030a0",
+  "epoch_id": "0x88f2495d1240f9c5523db589996a50a4984ee7a08a8a8f4b269e4345b383310abd2dc1cd9c9c2b8718ed3f486d5242f5"
 }
 ```
 

apikeys/README.md

@@ -1,31 +1,108 @@
 # API key management
 
-Bearer tokens are enforced by Caddy, not the Go service. `apikeys.py` maintains
-`keys.csv` (`username,apikey` rows, the source of truth) and compiles it into the
-Caddy snippet `apikeys.caddy`. Both live in the data directory
-(`${DATA_DIR:-./data}`). Caddy only reads the snippet at startup, so every change
-needs a caddy restart.
+Bearer tokens and rate limits are enforced by Caddy, not the Go service.
+`apikeys.py` reads two files and compiles them into the Caddy snippet
+`apikeys.caddy`:
+
+- **`keys.csv`** — `username,apikey,tier` rows, the source of truth for who has a
+  key and which tier it is on. Lives in the data directory (`${DATA_DIR:-./data}`).
+- **`limits.yaml`** — the rate limits for every tier and endpoint. Lives here, in
+  the repo, because it is policy rather than per-deployment state.
+
+You do not run the compile step by hand. The `compiler` service does it, and caddy
+waits on it (`service_completed_successfully`), so the snippet is regenerated from
+both files on every deploy. It prints the resulting limits — read them in the
+compiler's logs to confirm a change landed.
+
+Caddy only reads the snippet at startup, so a change needs caddy restarted.
+
+## Tiers
+
+Every key sits on a tier, which decides its rate limits. `limits.yaml` defines
+them — currently `standard` and `premium`, the latter for customers running
+continuously.
+
+A row with no tier column reads as `standard`, so a `keys.csv` written before tiers
+existed still works untouched — and a deploy stays reversible, since the previous
+version cannot read a three-column file. Add the column when you first promote
+someone, not before.
+
+Limits are `base × multiplier`: each endpoint has one base number, each tier one
+multiplier. The compiler prints the resolved table, so read that rather than doing
+the arithmetic.
 
 ## Create a key
 
+This is the only step with no deploy equivalent, since it prompts. Run it through
+the compiler service so the host needs nothing but docker. The rate-limit override
+is required — the `compiler` service is defined there and nowhere else:
+
 ```bash
-KEYS_FILE=./data/keys.csv CADDY_SNIPPET=./data/apikeys.caddy python3 apikeys/apikeys.py
+docker compose -f docker-compose.yml -f docker-compose.rate_limit.yaml \
+  run --rm -it compiler uv run --script /apikeys.py
 ```
 
-Enter a unique reference (e.g. the user's email) at the prompt. This appends the
-key to `keys.csv` and recompiles the snippet. Then:
+It asks for a unique reference (e.g. the user's email) and then which tier to put
+the key on, defaulting to `standard`. `-it` matters: without it the prompts don't
+reach you. The key is appended to `keys.csv` and the snippet recompiled.
+
+Restart caddy, then:
 
 ```bash
-docker compose -f docker-compose.yml -f docker-compose.rate_limit.yaml restart caddy
 tail -1 data/keys.csv | cut -d, -f2   # the new token
 curl -H "Authorization: Bearer <token>" https://<api-host>/check_authentication
 ```
 
+## Move a key between tiers
+
+Edit the `tier` column of its row in `keys.csv` and redeploy. The token does not
+change, so there is nothing to re-issue to the customer.
+
+## Change a tier's limits
+
+Edit `limits.yaml` and redeploy. Check the compiler's printed table to confirm you
+changed what you meant to.
+
 ## Revoke a key
 
-Delete its row from `data/keys.csv`, then recompile and restart caddy:
+Delete its row from `keys.csv` and redeploy.
+
+## Tests
 
 ```bash
-KEYS_FILE=./data/keys.csv CADDY_SNIPPET=./data/apikeys.caddy python3 apikeys/apikeys.py --compile
-docker compose -f docker-compose.yml -f docker-compose.rate_limit.yaml restart caddy
-```
+pytest apikeys/ -v
+```
+
+Needs `pytest` and `pyyaml`.
+
+Nothing runs these automatically yet, so run them after changing `apikeys.py`.
+
+`testdata/` holds a saved copy of the snippet the compiler should produce, and the
+test compares its output against that copy. So if you change how the snippet is
+built, that test fails and the diff shows you exactly what changed in the config
+caddy receives. Read the diff. If the change was intended, re-save the copy:
+
+```bash
+UPDATE_GOLDEN=1 pytest apikeys/ -k compile
+```
+
+## Verifying a tier change against a deployment
+
+`test_apikeys.py` covers the generator. To check that a deployment really enforces
+different limits per tier, compile from a temporary policy with tiny numbers instead
+of making thousands of requests, and never edit `limits.yaml` to do it:
+
+```bash
+# a copy of limits.yaml with window: 1m and base: 1 on every endpoint
+docker compose <overrides> run --rm \
+  -v /tmp/limits.test.yaml:/limits.test.yaml -e LIMITS_FILE=/limits.test.yaml compiler
+# restart caddy, then burst a key of each tier and note where the 429 lands
+```
+
+With multipliers of 1 / 3 / 6, anonymous should 429 on request 2, a standard key on 4
+and a premium key on 7. Restoring is a normal compile, since the real `limits.yaml` was
+never touched.
+
+Burst a read endpoint — `get_data_for_encryption` or `get_decryption_key` — so no
+transaction is submitted and no gas is spent. Caddy counts the request before proxying,
+so it is counted even when the API answers with an error.