convoy-bench

Convoy-bench contains all the code and scripts to benchmark any convoy cluster. The goal is to be able to quickly benchmark any Convoy cluster to know it's capacity. This was heavily inspired by Clickbench and Lavinmqperf

Why?

Similar to API Gateways, Webhook Gateways have become a critical component of modern web services. Teams are processing millions and billions of webhooks monthly. We created this benchmarking suite to easily enable our users to perform capacity planning to determine what cluster size they need for their workloads.

Tools

1. k6.io
2. Go

Pre-requisites

1. Make sure you have a Convoy instance running and properly configured.
2. If you're going to be testing with a Message Broker, make sure you've provisioned the broker ahead as well. For each broker, there are specific parameters to set, read the flags to know what to supply.
3. Install k6 for your own architecture (brew install k6, or see the k6 docs). bench.sh prefers a k6 on your PATH and only falls back to the checked-in bin/k6, which is an x86_64 build.

Measuring end-to-end latency

The producer stamps each event with an X-Benchmark-Timestamp-Ms header. The consumer reads it back and records the gap in the event_delivery_seconds Prometheus histogram, which is the enqueue-to-delivery latency for the whole pipeline. Deliveries per second come from the same consumer, either as http_requests_total or from its /rps and /px endpoints.

Run the producer and the consumer from the same version. A mismatched pair rejects every delivery with a 400 naming the header, rather than reporting a latency that is quietly wrong, so if a run returns nothing but 400s, rebuild the consumer image.

Usage

Usage:
  ./bench.sh -e, --endpoint-id=ENDPOINT-ID

Options:
  -p,   [--producer=PRODUCER]              # Select a producer to publish events from the following - http, sqs, pubsub or kafka.
                                           # Default: http
  -u,   [--uri=URI]                        # URI of your Convoy Cluster.
                                           # Default: http://localhost:5005
  -v,   [--vus=VUS]                        # Set how many virtual users should execute the test concurrently.
                                           # Default: 10
  -d,   [--duration=DURATION]              # Set how long the test should run. Use Golang string syntax: 1m, 5s, 10m5s .
                                           # Default: 1m
  -e,   --endpoint-id=ENDPOINT-ID          # ID of the endpoint configured on Convoy.
  -a,   [--api-key=API-KEY]                # Convoy Cluster API Key. Specify this if producer is http.
  -pid, [--project-id=PROJECT-ID]          # Convoy Cluster project ID. Specify this if producer is http.
  -q,   [--queue-url=QUEUE-URL]            # Amazon SQS URL. Specify this if producer is sqs.
  -aak, [--aws-access-key=AWS-ACCESS-KEY]  # AWS Access Key. Specify this if producer is sqs.
  -ask, [--aws-secret-key=AWS-SECRET-KEY]  # AWS Secret Key. Specify this if producer is sqs.

execute convoy benchmarks

Examples

HTTP/Incoming

The example below will run for 10 concurrent users for 1 minute, blasting events through Convoy's REST API.

./bench.sh -p http -u "{your-incoming-project-ingest-url}" -t incoming -v 10 -d 1m

HTTP/Outgoing Project

The example below will run for 80 concurrent users for 1 minute, blasting events through Convoy's REST API.

./bench.sh -p http -u "{your-convoy-instance-url}" -t outgoing -v 80 -d 1m --endpoint-id "{your-endpoint-id}" --project-id "{your-project-id}" --api-key "{your-api-key}"

The example below will run for 5 concurrent users for 5 minutes blasting events through an Amazon SQS Queue.

» ./bench.sh -p sqs -v 5 -d 5m \
--endpoint-id "{endpoint-id}" \
--queue-url "{queue-url}" \
--aws-access-key "{aws-access-key}" \
--aws-secret-key "{aws-secret-key}"

Support

If you need any support, please don't hesitate to reach out to us at support@getconvoy.io or join our slack community here