A general purpose processing framework for corpora of scientific documents

News:

• 2019: The CorTeX framework is recurringly converting >1.5 million articles from arXiv.org
• 2018: First datasets produced by CorTeX are now available for reuse via the SIGMathLing interest group

Features:

• Safe and speedy Rust implementation
• Distributed processing and streaming data transfers via ZeroMQ
• Backend support for Document (via FileSystem) and Task (via PostgreSQL 18+) provenance.
• Representation-aware and -independent (TeX, HTML+RDFa, ePub, TEI, JATS, ...)
• Powerful workflow management and development support through the CorTeX web interface
• Supports multi-corpora multi-service installations
• Centralized storage, with distributed computing, motivated to enable collaborations across institutional and national borders.
• Routinely tested on 1 million scientific TeX papers from arXiv.org
• Self-installing: cortex init / doctor (embedded migrations, runtime config, health check)
• Agent-first & symmetric: every capability on three surfaces — web UI, agent JSON API (OpenAPI at /api/docs), and the scriptable cortex CLI — from one shared backend, so all see the same live + historical state
• Observability: background-job health, /health + Prometheus /metrics, an audit log of every admin action
• Local authentication: passkeys (WebAuthn) + admin tokens, server-side sessions
• Annotations backend and workflow (TODO)
• Automatic dependency management of registered Services (TODO)

Getting started (full steps in INSTALL.md; day-to-day operation in MANUAL.md):

sudo apt install -y postgresql libpq-dev libzmq3-dev libsodium-dev pkg-config
cargo run --bin cortex -- init                                  # migrate + scaffold config
cargo run --bin cortex -- set-admin-token --generate --owner me # first admin credential
cargo run --bin cortex -- doctor                                # verify: => healthy
cargo run --bin frontend                                        # serves http://127.0.0.1:8000

Admin token & secrets: writes and the /admin sign-in are gated by an admin token (sent as an X-Cortex-Token header or ?token= query) mapped to an owner (rerun_tokens). Tokens live in a gitignored token file — the default is config.json in the working directory (the tracked config.example.json is a demo template only, never your real token), scaffolded/managed by cortex set-admin-token. To keep a production token out of the repository entirely, put it in a JSON file outside the tree and point CORTEX_AUTH_FILE at it (it overrides the in-repo config):

# /etc/cortex/config.json  (chmod 600, owned by the service user — never in git)
{ "rerun_tokens": { "<your-token>": "<owner>" } }
# then, in the service environment (e.g. /etc/cortex/frontend.env):
CORTEX_AUTH_FILE=/etc/cortex/config.json

Humans can also sign in with passkeys (WebAuthn); the token is the agent + break-glass credential.

History:

• Originally motivated by the desire to process any Cor-pus of TeX documents.
• Rust reimplementation of the original Perl CorTeX stack.
• Builds on the expertise developed during the arXMLiv project at Jacobs University.
• In particular, CorTeX is a successor to the build system originally developed by Heinrich Stamerjohanns.
• The messaging conventions are adopted from work on standardizing LaTeXML's log reports with Bruce Miller.

CorTeX is being productized from an admin-only prototype into a self-installing, agent-first + human-first application — an active sprint (see docs/PRODUCTIZING_PLAN.md), with a public preview in preparation. It now self-installs and ships a complete operator manual and agent API, though some hardening/rationalization is still in flight. For details consult the Installation instructions and the Manual. Please feel invited to open an issue with a description of your target workflow!