Implemented Scraper, Quotes API and WS Prototypes. - #9
Open
justwickedcode wants to merge 12 commits into
Open
justwickedcode wants to merge 12 commits into
justwickedcode wants to merge 12 commits into
Conversation
…a round of real crawler/parser bug fixes - backend/api: a small Go read-only HTTP API (previously scaffolded but never committed) that serves scraped quotes to the frontend's typing race, filtered by word count and language, with basic rate limiting and typing-safety checks on returned text. - backend/scraper: added French, Spanish, Italian, Portuguese, Polish, Swedish, Romanian, Czech, Hungarian, Danish, Norwegian, and Finnish Wikiquote editions (English/German already existed), via one shared LocalizedWikiquoteParser rather than a file per language. - Fixed several real parser bugs found via live investigation: a semi-protected page's duplicate empty mw-parser-output div silently returning 0 quotes; an exact "Quotes"-heading match missing pages using a different real heading; numbered <ol> lists being silently skipped; a non-English original quote's English translation being discarded along with its citation on classical/ translated-work pages; French quotes living in a bare <div class="citation"> outside any list. - Added dedup.LooksLikeWikiDiscussion to catch quote-of-the-day nomination/voting page content that the broadened heading-matching above would otherwise sweep in as if it were real quotes. - Tightened the language-mismatch filter (dedup.MatchesClaimedLanguage) with a comparative confidence check, catching confidently-wrong-language quotes an absolute threshold alone missed. - Added quotes.source_url, tracking the exact page each quote was scraped from. - Tuned crawler pacing/backoff based on live observation: jittered retry timing and staggered worker startup to reduce synchronized 429s across the now much larger set of concurrent per-language workers; batched the Redis cache-warming pipelines that were blowing past the default write timeout and crashing startup once the corpus grew large enough. - Added PRODUCTION.md and docker-compose.prod.yml for a real deployment, plus .env.example files. - Fixed .gitignore: a "*.sum" rule had silently excluded both Go modules' go.sum from every commit in this repo's history; both are now tracked for real, and a stray compiled binary (backend/api/quotes-api) is excluded by name instead. - Fixed a broken pre-commit hook: root package.json's lint-staged config pointed eslint at frontend/eslint.config.mjs, which doesn't exist (the real file is eslint.config.js) — silently failing every commit that touched a frontend JS/TS file until now. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011XQv8bc3QBCxPsv84gSrgu # Conflicts: # frontend/src/lib/typing/useTypingRace.ts
A new standalone Go service (separate module, mirroring backend/api's pattern) implementing the actual multiplayer session backend/api's WS route was originally going to own. Auto-matchmaking (GET /ws/race) puts players into small 2-6 person lobbies with a synchronized countdown, live opponent progress, and server-computed WPM/final rankings. Private lobbies (GET /ws/private/create, GET /ws/private/join?code=) let a room host share a 6-character code and force an early start once enough friends have joined, instead of waiting out the public lobby's timeout. Room/matchmaking state is in-memory and process-local; quote selection is injectable (selectQuote) so the whole room lifecycle is covered by tests without a live Postgres connection. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011XQv8bc3QBCxPsv84gSrgu
…nd orphaned api.ts The previous commit's rebase conflict resolution didn't actually take — it landed with the old fetchRandomSentence-based useTypingRace.ts content and a stray frontend/src/lib/typing/api.ts still intact, instead of origin/main's newer getText-injectable version. This makes the working tree match origin/main's useTypingRace.ts exactly and removes the now -unused api.ts (nothing imports it — the new frontend doesn't call backend/api for solo sentences). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011XQv8bc3QBCxPsv84gSrgu
Leftover from before the Nuxt→TanStack Start migration — this branch's rebase onto origin/main's newer frontend resurrected it because an old commit here still touched it. origin/main already deleted it correctly; this was the one file the rebase didn't fully reconcile. It was also the cause of a pre-existing ESLint parsing error (no tsconfig covers .vue files in a TanStack Start project). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011XQv8bc3QBCxPsv84gSrgu
Pre-existing dead import in origin/main's own frontend prototype (confirmed byte-identical before this fix) — tripped tsc with an unused -variable error, blocking a clean build. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011XQv8bc3QBCxPsv84gSrgu
The prototype frontend (pulled in from origin/main) already had real routes for this — /race/friends (create/join), /race/$roomCode (the room/race view), /race/multiplayer (quick match) — but they were pure UI mockups with a client-generated fake room code and a disabled "needs backend" button. This connects them to backend/ws for real: - useMultiplayerRace: a WebSocket hook mirroring useTypingRace's exact input model (reject-wrong-chars, errorSeq flash, WPM/accuracy/progress math) but driven by a server-provided quote instead of one picked synchronously on mount, since the quote only exists once the lobby's race_start message arrives. - MultiplayerRace: the shared lobby/countdown/racing/finished view, reusing solo's own RaceTrack/TypingWords/Gauge components so both race types look and feel identical. - race.friends.tsx's "Create room" now navigates to a reserved "new" sentinel instead of minting a fake code; race.$roomCode.tsx asks backend/ws to actually create-or-join, then swaps the URL to the server's real code once it arrives. Two real bugs found and fixed via live two-tab testing along the way: - The connect effect's own cleanup was racing against dev-mode's double-invoke (mount → cleanup → mount again): a "connect once" ref guard suppressed the real second connect while the hook's unconditional cleanup closed the first, still-handshaking socket, permanently stranding the room in an error state. Fixed by letting the effect reconnect naturally (as React expects) and having every socket event handler check it's still the current socket before touching state, so a superseded connection's belated events can't corrupt it. - A player who finished before their opponents saw nothing until race_end (which waits for everyone) even though the server already broadcasts their own rank/wpm the instant they finish — now shown immediately as "Finished #N · wpm — waiting for other racers". Also logs backend/ws's previously-silent too-many-connections rejection (ws.go), which had no trace at all server-side and made the above connection bug much harder to diagnose than it should have been. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011XQv8bc3QBCxPsv84gSrgu
The app's component structure (Card/Badge/Button primitives, CVA variants, sidebar nav, disciplined CSS with no gratuitous glass-blur) was already shadcn-typical — the theme itself just wasn't: every structural token (background/foreground/card/primary/secondary/muted/ accent/border/input/ring) carried a sky-blue tint. Zeroed the chroma on all of them, matching shadcn's own default dark theme (primary is literal white-on-black, exactly like shadcn's own "zinc" flavor). destructive (red) and success (green) keep real hue on purpose — every shadcn theme keeps its semantic danger/success colors even in the most neutral variant, since those signal state rather than brand. Also fixed the one hardcoded hex that had drifted from the token system (.race-name-tag--you was literally `#38bdf8`, not var(--color-primary), so it wouldn't have picked up the new palette) and synced the mobile theme-color meta tag to the new background. Deliberately left untouched: car paint colors, opponent car colors, medal tier colors, sponsor billboard colors, and other decorative "racing game" flourishes — those are intentional personalization/fun elements, not brand chrome, and reskinning them would work against what they're for. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011XQv8bc3QBCxPsv84gSrgu
AppShell locked the whole content column to overflow-hidden inside a fixed h-svh shell, so any page taller than the viewport just got clipped at the bottom with no way to reach the rest — the mobile header's own sticky positioning had nothing to stick within either, since sticky requires a scrolling ancestor. Switched that column to overflow-y-auto (the sidebar's already-sticky mobile header now works as designed) and gave the sidebar itself the same treatment so a taller nav list won't hit the same wall later. Separately, nearly every route wrapped its content in `flex flex-1 flex-col justify-center`, vertically centering it inside the full remaining viewport height — fine for a short hero, but it left large dead margins above and below any page whose content didn't happen to fill the screen (achievements, garage, stats, the friends/ multiplayer lobby, the solo speed picker, etc.), and made the problem above worse by encouraging content to assume it always had a full screen to center within. Dropped the centering in favor of the top-aligned, normal-padding layout the home page already used — content now starts right below the header and flows down like a regular page, which is also just the more typical shadcn dashboard convention. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011XQv8bc3QBCxPsv84gSrgu
…l, lobby/results restyle Replaces the ring-progress Gauge with an analog needle dial (calm/working/redline zones, ticks) for stats with a real ceiling, adds a plain DigitalReadout for stats that have none (time, race count), and threads both through every wpm/accuracy display (solo race, multiplayer race, guide, daily challenge, dashboard, stats). Sidebar now defaults to the icon-only rail. WpmTrend gets an oscilloscope treatment and RaceHistoryRow becomes a numbered lap sheet. Multiplayer's lobby is now a plate-coded starting grid and its results screen a standings table. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011XQv8bc3QBCxPsv84gSrgu
Owner
Author
|
@maximilian-graf-cs
!!!!CORS is locked down on both: the WS service only accepts connections from localhost:3000, and the quotes API isn't exposed beyond the WebSocket. This is intentional: both are meant to run locally for now, and we don't want either one accidentally exposed to the internet once we deploy and get them nuked by bots and crawlers. Still to build:
If you want to test what I implemented, you can locally fetch my changes and: then for scraper: (Next time when we meet I'll explain you the architecture of the scraper and the use of Redis for identical and almost identical text dedup. It ended up saving us almost 25% (!!!!) duplicates across goodreads and wikiquotes en out of 400k english quotes and 300k foreign language quotes) for crud api: for ws: for frontend: This branch is prototype-only, so it doesn't need a deep review. We'll go through each service individually and refactor properly in the near future, for now, this gives us solid ground to keep building on. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implemented Scraper, Quotes API and WS Prototypes.