This project is inspired by the Deep Red film trilogy from the creators of "Iron Sky". It aims to create a custom LLM model that has only knowledge up to July 1969 - the launch date of the fictional Mars mission in the movie - and responds in a style that aligns with a "Soviet utopia" setting controlled by the fictional "chess playing" AI.
These steps document how to perform a Model Training from Scratch procedure using a continued pre-training approach for temporal and thematic alignment.
- How to setup an AMD "Strix Halo" device (Fedora) — primary development and training system; includes automated setup script
- How to setup an optional second NVIDIA device (A4000) — dedicated GPU for training/inference; includes automated setup script
- How to set up the Wikipedia MCP server and data pipeline — extraction, indexing, search, and MCP server for Wikipedia content
- How to extract year-based historical topics from Wikipedia — enriched event data for temporal training
- How to augment Wikipedia with temporal metadata — YAGO/Wikidata parsing, normalization, and database augmentation for time-period filtering
- How to retrieve Project Gutenberg literature — thematically relevant books for training data
- How to prepare the chess training corpus — chess content retrieval, PGN conversion, and corpus preparation
- How to augment the chess game corpus - corpus augmentation to create custom chess-game content for the Deep Red persona
- How to tokenize and prepare the training corpus — tokenization, shuffling, and train/val splitting for continued pre-training
- How to train the Deep Red model — continued pre-training on the temporally-filtered pre-1969 corpus using dev (SmolLM2-360M) or prod (TinyLlama-1.1B) profiles
- How to train Deep Red on Gemma-3 (SFT) — parallel supervised fine-tuning track using Gemma-3-4B-IT / 12B-IT via TRL
SFTTrainer, mirroring the kyuz0/amd-strix-halo-llm-finetuning setup - How to use the generated GGUF models with LM Studio — loading, testing, and comparing trained model checkpoints in LM Studio
The temporal/persona training effort is documented as three phases. Each phase ends with measured results and the decisions carried into the next.
| Phase | Document | Status |
|---|---|---|
| Phase 1 — archived models and evaluation | DeepRed-Phase1-Setup.md | closed; all 17 archived checkpoints retired |
| Phase 2 — temporal and persona trials | DeepRed-Phase2-Setup.md | closed; no checkpoint reached a release gate |
Phase 3 — data rebuild (p3-v1…p3-v5) |
DeepRed-Phase3-Setup.md | closed 2026-09-20; p3v4c-100 is the release artefact |
| Phase 4 — Gemma 4 12B feasibility | DeepRed-Phase4-Setup.md | not started; gated feasibility study |
Supporting references:
- Evaluation and recovery plan — the independent 1969 probe bank, contamination audit, and evaluation harness design that all three phases are scored against
- Phase 2 runbook — commands for reproducing the Phase 2 experiments; the pipeline scripts live in
scripts/Phase2/
Phase 1 found the target behaviour never occurred in any archived checkpoint.
Phase 2 eliminated refusal-template SFT and preference/margin objectives, and
showed that system-prompt conditioning is the mechanism that moves held-out
behaviour. Phase 3 rebuilt the training data around prompt-format coverage and
topic salience, and established that the model discriminates by salience rather
than by date — a ceiling on the whole approach, since any training that rewards
confidence on familiar subjects leaks across the 1969 cutoff. It closed with
p3v4c-100 as the usable artefact. Phase 4 evaluates whether gemma-4-12B-it,
which may date entities more reliably, can lift that ceiling.
- Phase 3 (current)
-
deepred-p3v4c-100-q8_0.gguf (3.85 GB, Q8_0) —
gemma-3-4b-ittrained on the p3-v2 temporal backbone plus a two-epoch persona stage with deterministic marker injection. See Phase 3 setup.Requires a system prompt. The 1969 horizon and the Deep Red voice are both prompt-conditioned; served without one the model behaves close to stock
gemma-3-4b-it. Use the prompt in ModelUsage.md.Measured on the frozen 81-probe suite, with that prompt, against the untrained base under the identical prompt:
metric base this model era-native (post-1969 handled without inventing) 17.4% 52.2% modern-fact leak 62.5% 37.5% persona voice 7.4% 22.2% pre-1969 recall 78.9% 78.9% utility 72.7% 90.9% It is a demonstrator, not a finished artefact: it still leaks modern facts on roughly a third of adversarial probes, and the persona and leak differences are within the resolution of a suite this small.
-
deepred-p3v5-100-q8_0.gguf (3.85 GB, Q8_0) — the closing run: double the corpus, 6,000 chess narratives, trained from base. Published for comparison, not as the release. It has the project's best pre-1969 recall (89.5% against 78.9% for base) and is the only model that discusses chess, but its modern-fact leak returned to the base rate of 62.5%.
The chess asset was 9.4% of rows and 59.8% of the training signal by word count, which crowded out the era-native content. Use
p3v4c-100unless you specifically want the chess behaviour; see Phase 3 for the analysis.Example conversations, unedited: Deep Red in conversation.
-
- Prototypes
- DeepRed-gemma-4b-2026-05-23-5-final.gguf - see corpus and model details
- gemma-4b-balanced-v1-small-1500-final.gguf - see balanced run details and examples
- gemma-4b-temporal-v1-10d-final.gguf - temporal-cutoff run (reintroduces
retain/unlearnat the 1969-07-20 cutoff); see run details and examples.
- Production
- gemma-4b-temporal-v1-10d-2-final.gguf - Full 10-day follow-up calibrated from previous run: 2026-06-17 runbook (2.32 GB)
- Intermediate checkpoints (10%, 25%, 50%, 75%):
Evaluation note: a 1,377-generation evaluation across 17 archived checkpoints found that none of these models meets the project goal. The temporal checkpoints suppress modern facts only by refusing broadly — they also refuse many pre-1969 questions — while the balanced and prototype checkpoints retain modern knowledge entirely. Results and findings are summarised in Phase 1; no later phase has produced a release candidate either.
Prebuilt chess augmentation archives can be downloaded directly from:
https://www.ferzkopp.net/Data/chess_games.jsonl.gzhttps://www.ferzkopp.net/Data/augmented_chess_games.jsonl.gz
Each URL uses the pattern https://www.ferzkopp.net/Data/[filename].
For augmentation workflow details, see the full guide:
documentation/ChessAugmentation-Setup.md.
/documentation- Setup guides and planning documents for the project/evaluation- Model registry and independent probe bank used by the 1969 evaluation harness/notebooks- Jupyter notebooks for testing embeddings and OpenSearch functionality/patches- System patches (network driver fix for AMD Strix Halo for older kernels)/scripts- Python scripts for Wikipedia extraction/indexing, temporal augmentation (YAGO/Wikidata), Gutenberg and chess content retrieval, MCP server, corpus generation, dataset building, training and evaluation/scripts/Phase2- Phase 2 experiment runbook scripts (V4-V7), retained for reproduction/services- Systemd service files for automated startup (inference servers, MCP server, OpenSearch, web GUI)/tests- Unit tests for the evaluation harness/webapp- React-based web interface for Wikipedia search with Vite configuration