Reproducible benchmarking of CognoDB Cloud and comparable graph database platforms using a common dataset, equivalent logical workloads, deterministic query selection, and a controlled resource envelope.
Benchmark infrastructure complete.
The current benchmark supports:
- CognoDB
- Neo4j
- Memgraph
- FalkorDB
- ArcadeDB
The repository contains the benchmark execution framework, database adapters, workload registry, deterministic query generation, run metadata, result serialization, comparison/scoring infrastructure, and benchmark documentation.
The goal of this project is to compare graph database performance under controlled and reproducible conditions.
The benchmark focuses on two primary areas:
- Data ingestion performance
- Graph query execution performance
The comparison uses the same canonical dataset and equivalent logical workloads across all supported databases.
The results are intended to provide a transparent performance comparison for the documented environment and configuration.
They should not be interpreted as a universal ranking of graph databases.
| Database | Deployment |
|---|---|
| CognoDB | Cloud |
| Neo4j | Docker |
| Memgraph | Docker |
| FalkorDB | Docker |
| ArcadeDB | Docker |
Database versions and deployment details are documented in:
docs/benchmark-methodology.md
The benchmark targets a common resource envelope wherever the deployment model permits:
| Resource | Target |
|---|---|
| CPU | 0.5 vCPU |
| Memory | 512 MB |
| Storage | 1 GB |
CognoDB currently uses the available c0 configuration documented by the project methodology.
Self-hosted databases are constrained to the same target resource envelope where supported by their deployment configuration.
Database-specific configuration required to operate within the envelope is documented rather than silently relying on unrestricted defaults.
See:
docs/benchmark-methodology.md
All databases use the same canonical processed dataset.
Current dataset size:
Nodes: 36,692
Relationships: 183,831
The benchmark consumes:
data/processed/nodes.csv
data/processed/relationships.csv
The benchmark discovers the actual dataset row counts before execution and uses those counts as the run metadata.
The benchmark configuration also validates that the ingested database contains the expected node and relationship counts.
The benchmark currently evaluates seven logical workloads:
| Workload | Purpose |
|---|---|
point_lookup |
Lookup one user by ID |
indexed_lookup |
Exercise indexed user lookup |
relationship_lookup |
Retrieve direct KNOWS neighbors |
traversal_1_hop |
Traverse exactly one KNOWS hop |
traversal_2_hop |
Traverse exactly two KNOWS hops |
traversal_3_hop |
Traverse exactly three KNOWS hops |
aggregation |
Count all benchmark users |
The query syntax may differ between databases, but the logical operation is kept equivalent.
Traversal workloads use a result limit of:
1000
This prevents high-degree graph traversals from producing uncontrolled result sets and keeps the workload operationally comparable.
Detailed workload definitions are documented in:
docs/workload-reference.md
Parameterized workloads use dynamically generated node IDs.
The benchmark uses:
query_seed = 42
The same seed produces the same node-ID sequence.
This makes query selection reproducible across benchmark runs and databases.
For example, the sequence begins with:
7296
1639
18024
16049
14628
9144
6717
35741
5697
27651
This prevents random query selection from becoming an uncontrolled variable in the comparison.
The project requires Python and the dependencies used by the benchmark adapters.
Create and activate a virtual environment, then install the required project dependencies appropriate to the current development environment.
Database connection information is provided through environment variables.
Copy:
.env.example
to:
.env
and configure the connection details for the databases you intend to benchmark.
The .env file is ignored by Git.
The supported environment variable groups are:
COGNODB_URI
COGNODB_USERNAME
COGNODB_PASSWORD
NEO4J_URI
NEO4J_USERNAME
NEO4J_PASSWORD
MEMGRAPH_URI
FALKORDB_HOST
FALKORDB_PORT
ARCADEDB_HOST
ARCADEDB_PORT
ARCADEDB_USERNAME
ARCADEDB_PASSWORD
ARCADEDB_DATABASE
The exact values depend on the deployment being benchmarked.
Do not commit credentials or other secrets.
The benchmark CLI is implemented in:
benchmark/cli.py
Example:
python -m benchmark.cli --database MEMGRAPHpython -m benchmark.cli --database MEMGRAPH --iterations 100python -m benchmark.cli --database MEMGRAPH --iterations 100 --warmup 10python -m benchmark.cli --allThe CLI does not allow --database and --all to be used together.
A benchmark run follows this general flow:
CLI
|
v
Load benchmark configuration
|
v
Discover dataset counts
|
v
Create run metadata
|
v
Create results/runs/<run_id>
|
v
Select database adapter
|
v
Select database workloads
|
v
Connect to database
|
v
Ingest canonical dataset
|
v
Validate node/relationship counts
|
v
Execute warmup queries
|
v
Execute measured queries
|
v
Calculate latency + QPS statistics
|
v
Clear benchmark data
|
v
Close database connection
|
v
Serialize results
This flow is implemented primarily through:
benchmark/cli.py
benchmark/runner.py
benchmark/query.py
benchmark/ingestion.py
benchmark/workload_registry.py
benchmark/database_registry.py
Warmup executions are excluded from measured latency statistics.
Measured executions record individual query latency values.
The benchmark calculates:
- minimum latency
- mean latency
- P50
- P95
- P99
- maximum latency
- QPS
P95 is the primary latency metric used by the scoring system because it represents tail behavior better than the arithmetic mean.
Each run receives a unique run ID.
Results are stored under:
results/runs/<run_id>/
A run records metadata such as:
run_id
timestamp_utc
git_commit
python_version
platform
processor
node_count
relationship_count
ingestion_batch_size
query_iterations
query_warmup_iterations
query_seed
This allows benchmark results to be associated with the exact source revision and execution configuration that produced them.
A completed run produces database-specific result files under its run directory.
Typical structure:
results/
└── runs/
└── <run_id>/
├── metadata.json
├── neo4j.json
├── memgraph.json
├── falkordb.json
├── arcadedb.json
└── cognodb.json
Additional aggregate artifacts can include:
results/comparison.json
results/scores.json
Generated benchmark results are treated as artifacts of a benchmark execution rather than source code.
The benchmark records:
elapsed_seconds
nodes_per_second
relationships_per_second
For ingestion:
- lower elapsed time is better
- higher nodes/sec is better
- higher relationships/sec is better
Each workload records:
min
mean
p50
p95
p99
max
qps
For query performance:
- lower latency is better
- higher QPS is better
The scoring implementation is:
benchmark/scoring.py
The scoring methodology uses best-value normalization.
For lower-is-better metrics:
score = best_value / database_value
For higher-is-better metrics:
score = database_value / best_value
Therefore, the best database for a metric receives:
1.0
The overall score uses:
20% ingestion
80% query performance
Each query workload contributes equally to the query-performance component.
Each workload is composed of:
50% P95 latency
50% QPS
Conceptually:
workload score
= 50% normalized P95 score
+ 50% normalized QPS score
query score
= average of workload scores
overall score
= 20% ingestion score
+ 80% query score
The complete methodology is documented in:
docs/results-and-scoring.md
After comparison data has been generated, the scoring module can be executed with:
python -m benchmark.scoringThe scorer reads:
results/comparison.json
and writes:
results/scores.json
The console output displays the normalized ranking and per-workload scores.
All databases implement the common interface:
databases/base.py
The GraphDatabaseAdapter abstraction provides operations for:
connect()
close()
execute()
clear()
health_check()
load_nodes()
load_relationships()
count_nodes()
count_relationships()
name
Database-specific implementations are located under:
databases/
├── arcadedb.py
├── base.py
├── cognodb.py
├── falkordb.py
├── memgraph.py
└── neo4j.py
The benchmark runner operates against this common adapter interface rather than directly depending on an individual database implementation.
Workloads are represented by:
benchmark/workload.py
Database-specific workload registrations are maintained through:
benchmark/workload_registry.py
The workload implementation files are:
benchmark/
├── workloads.py
├── neo4j_workloads.py
├── memgraph_workloads.py
├── arcadedb_workloads.py
└── cognodb_workloads.py
FalkorDB uses the shared workload definitions in:
benchmark/workloads.py
The registry maps each supported database to its appropriate workload tuple.
The important project directories are:
graph-db-cloud-benchmark/
│
├── benchmark/
│ ├── cli.py
│ ├── config.py
│ ├── database_registry.py
│ ├── dataset.py
│ ├── ingestion.py
│ ├── query.py
│ ├── results.py
│ ├── runner.py
│ ├── scoring.py
│ ├── serialization.py
│ ├── workload.py
│ ├── workload_registry.py
│ ├── workloads.py
│ └── *_workloads.py
│
├── databases/
│ ├── base.py
│ ├── neo4j.py
│ ├── memgraph.py
│ ├── falkordb.py
│ ├── arcadedb.py
│ └── cognodb.py
│
├── config/
│ └── settings.py
│
├── data/
│ └── processed/
│ ├── nodes.csv
│ └── relationships.csv
│
├── docs/
│ ├── benchmark-methodology.md
│ ├── architecture.md
│ ├── workload-reference.md
│ └── results-and-scoring.md
│
├── results/
│ └── runs/
│
├── tests/
│
├── .env.example
├── .gitignore
└── README.md
Other project directories contain supporting dataset, infrastructure, script, and development resources.
A benchmark run should preserve the conditions under which it was produced.
Important reproducibility inputs include:
- database versions
- deployment model
- CPU allocation
- memory allocation
- storage allocation
- canonical dataset
- workload definitions
- query seed
- query iterations
- warmup iterations
- ingestion batch size
- benchmark source revision
The benchmark automatically records several of these values in run metadata.
For a controlled comparison, all databases should be executed using the same benchmark configuration and dataset.
The benchmark is designed around controlled comparison rather than unrestricted database defaults.
The comparison attempts to hold constant:
Dataset
Resource envelope
Logical workloads
Query sequence
Warmup methodology
Measurement methodology
Database-specific query syntax and configuration are allowed where necessary to represent equivalent operations or successfully operate within the target resource envelope.
Such differences should be documented rather than hidden.
This benchmark does not attempt to measure every property of a graph database.
It focuses on the selected:
- dataset
- resource envelope
- database versions
- ingestion workload
- query workload suite
- execution configuration
It does not by itself establish:
- universal database superiority
- production reliability
- operational cost
- ecosystem quality
- developer experience
- feature completeness
- distributed scaling behavior
- failure recovery behavior
- security posture
- long-running production workload behavior
The final ranking is therefore specific to the documented benchmark conditions.
Detailed documentation is available under docs/:
docs/benchmark-methodology.md
Documents:
- objective
- resource envelope
- database versions
- deployment model
- dataset
- fairness
- reproducibility
- limitations
docs/architecture.md
Documents:
- benchmark architecture
- execution flow
- adapter abstraction
- workload architecture
- result pipeline
docs/workload-reference.md
Documents:
- workload definitions
- workload semantics
- deterministic query selection
- traversal limits
- warmup and measurement behavior
- database-specific query implementations
docs/results-and-scoring.md
Documents:
- result artifacts
- benchmark metrics
- normalization
- scoring
- weighting
- ranking
- interpretation
- limitations
The benchmark source can be syntax-checked with:
python -m compileall benchmark databases configThe database registry can be verified with:
python -c "from benchmark.database_registry import create_adapter; print([create_adapter(name).name for name in ['NEO4J','MEMGRAPH','FALKORDB','ARCADEDB','COGNODB']])"The workload registry can be verified with:
python -c "from benchmark.workload_registry import get_workloads; [print(name, [w.name for w in get_workloads(name)]) for name in ['NEO4J','MEMGRAPH','FALKORDB','ARCADEDB','COGNODB']]"These checks verify that the registry and workload modules can be imported and that all supported databases have registered workloads.
The repository ignores:
.env
.venv/
__pycache__/
Database credentials must remain in local environment configuration and must not be committed.
The example configuration is provided through:
.env.example
The .env.example file contains configuration placeholders and local development defaults where applicable, not production credentials.
The benchmark follows four principles:
The same dataset, deterministic query sequence, and documented configuration should produce comparable runs.
Databases should be compared under a controlled resource envelope and equivalent logical operations.
Database versions, configuration assumptions, workload semantics, measurements, and scoring methodology should be documented.
Database-specific behavior belongs in adapters and workload definitions.
Benchmark orchestration, measurement, statistics, serialization, and scoring remain independent of individual database implementations.
The complete system can be summarized as:
Canonical Dataset
|
v
Database Adapter
|
v
Ingestion
|
v
Dataset Validation
|
v
Deterministic Workload Selection
|
v
Warmup
|
v
Measured Query Execution
|
v
Latency + QPS Statistics
|
v
Run Result Serialization
|
v
Cross-Database Comparison
|
v
Normalized Scoring
|
v
Benchmark Ranking
The resulting benchmark artifacts preserve the measurements and metadata needed to understand how the final comparison was produced.
The benchmark should be used as a controlled performance study of the selected graph databases under the documented conditions.
A database ranking is meaningful only together with:
- the tested database version
- deployment model
- resource envelope
- dataset
- workload suite
- query configuration
- scoring methodology
The project therefore treats reproducibility and transparency as part of the benchmark itself, rather than presenting a ranking without its experimental context.