Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 33 additions & 17 deletions docs/contributing/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,39 +19,55 @@ OSV runs on Google Cloud Platform, with the following main components:

## Cloud Datastore

All vulnerability data is stored in [Cloud Datastore], with the [models] defined
[here].
All vulnerability data is stored in [Cloud Datastore] and the models are primarily defined in Go ([`go/internal/database/datastore/models.go`](../../go/internal/database/datastore/models.go)).

[Cloud Datastore]: https://cloud.google.com/datastore
[models]: https://googleapis.dev/python/python-ndb/latest/index.html#defining-entities-keys-and-properties
[here]: https://github.com/google/osv.dev/blob/master/osv/models.py


## Google Cloud Storage (GCS)

Full vulnerability records (conforming to the OSV schema) are stored as protobufs and JSON files in public [Google Cloud Storage] buckets. GCS acts as the primary source of truth for the vulnerability data, while Cloud Datastore is used as an index for fast querying and API lookups.

[Google Cloud Storage]: https://cloud.google.com/storage


## Google Kubernetes Engine (GKE)

[GKE] is used for running [workers] to perform bisects and impact analysis.
These workers consume tasks from a [Cloud Pub/Sub] topic.
[GKE](https://cloud.google.com/kubernetes-engine) is used for running the core vulnerability processing pipeline, auxiliary background services, and legacy/OSS-Fuzz workloads.

### Core Vulnerability Pipeline (Go-based)

These services run as native Go workloads inside the GKE cluster:

- **[importer](../../go/cmd/importer)**: A scheduled CronJob that polls and pulls vulnerability source repositories, detects additions/deletions, and dispatches processing tasks via [Cloud Pub/Sub](https://cloud.google.com/pubsub). It also manages periodic data cleanup (`importer-deleter`) and full database reconciliation (`importer-reconciler`).
- **[worker](../../go/cmd/worker)**: A daemon Deployment that consumes Pub/Sub tasks to ingest and enrich vulnerability records, compute affected commit/version ranges, and write them to GCS and Cloud Datastore. It is scaled dynamically using a HorizontalPodAutoscaler based on the Pub/Sub backlog.
- **[exporter](../../go/cmd/exporter)**: A scheduled CronJob that packages the entire OSV database (as zip files and individual JSON files) and exports them to public GCS buckets.
- **[gitter](../../go/cmd/gitter)**: A caching service Deployment that caches and precomputes heavy Git repository operations (cloning, commit graphs, patch IDs) for the importers and workers. It is backed by a large SSD Persistent Volume.
- **[relations](../../go/cmd/relations)**: A scheduled CronJob that calculates and populates transitive and reflective relationships (aliases, related, and upstream fields) between vulnerability records.
- **[indexer](../../gcp/indexer)**: A Deployment that manages git index mapping and version determination.
- **[vulnfeeds](../../vulnfeeds)**: Scheduled CronJobs that mirror and convert external vulnerability advisory feeds (e.g. NVD, Debian, Alpine) into OSV schema format.

### Auxiliary Services (Python-based)

These run as Python workloads inside GKE:

- **[recoverer](../../gcp/workers/recoverer)**: A daemon Deployment that processes failed tasks (e.g., failed GCS writes, git push failures) sent to a recovery queue, attempting to heal/repair their state.
- **[vanir_signatures](../../gcp/workers/vanir_signatures)**: A scheduled CronJob that generates Vanir signatures for modified vulnerabilities.

Workers are Docker containers, which use [gVisor] for sandboxing untrusted
workloads.
### OSS-Fuzz Integration

[GKE]: https://cloud.google.com/kubernetes-engine
[workers]: https://github.com/google/osv.dev/tree/master/gcp/workers
[gVisor]: https://gvisor.dev/
[Cloud Pub/Sub]: https://cloud.google.com/pubsub
- **[OSS-Fuzz workers](../../gcp/workers/oss_fuzz_worker)**: Legacy Python-based workers that perform bisection and impact analysis for ClusterFuzz/OSS-Fuzz bugs. Because these compile and run code from arbitrary open source projects, they run in Docker containers sandboxed with [gVisor](https://gvisor.dev/).

## Cloud Run / Cloud Endpoints

The [API server] runs on [Cloud Run], and is served by [Cloud Endpoints].
The [API server](../api/index.md) (hosted at `api.osv.dev`, source code in [`go/cmd/api`](../../go/cmd/api)) runs on [Cloud Run], and is served by [Cloud Endpoints] (transcoding HTTP/JSON REST to gRPC using ESPv2).

[API server]: https://github.com/google/osv.dev/tree/master/gcp/api
[Cloud Run]: https://cloud.google.com/run
[Cloud Endpoints]: https://cloud.google.com/endpoints

## Website

The [main web UI] also runs on [Cloud Run], and is served through
[Cloud Load Balancing].
The [main web UI](https://osv.dev) (source code in [`gcp/website`](../../gcp/website)) also runs on [Cloud Run], and is served through [Cloud Load Balancing].

[main web UI]: https://osv.dev
[Cloud Run]: https://cloud.google.com/run
[Cloud Load Balancing]: https://cloud.google.com/load-balancing
Binary file modified docs/images/architecture.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading