feat(2024): validate and reshape generated monster data

[bagelbits]

What does this do?

Stacked on #1226. Base branch add2024Monsters-base is copy of that PR head. Retarget to main after #1226 merge.

Script output now match MonsterSchema. New test check it. CI now run on PRs to any base branch, so stacked PRs get lint + tests.

• Normalize PDF text: ability scores on one line, wrapped header lines joined, merged lines split. Write monster-text-data.normalized.txt.
• Pair text block to monster by ability scores + CR, not array position. Old way swap skills/PB for 7 monsters (giant-crab, giant-crocodile, winter-wolf, worg, allosaurus, ankylosaurus, ape).
• Reshape: numeric CR, xp, xp_in_lair, armor_class array, condition references, senses parsing, hit dice minus sign.
• Parse damage (all types, one entry each) and dc (saving throw, escape, ability check) from descriptions for traits, actions, bonus actions, reactions, legendary actions.
• Build spellcasting. List entries (48): spell lists come from stat block text, gist stop at intro line; match list to entry by ability + save DC + to-hit since PDF put some blocks inside previous block text. Single-spell casts (Invisibility, Fear, Coven Magic, Divine Aid, and similar): link spell, inherit ability/DC from main Spellcasting when text say "same spellcasting ability", per-day usage from action name. Every spell resolve against 5e-SRD-Spells.json; script throw on unknown spell or unused list. (level N version) set spell level, other parentheticals become capitalized notes.
• Clean PDF debris from entries: drop spell list lines gist parsed as entries, strip next block title from last entry, split merged entries, complete descriptions cut at page breaks from stat block text (only when gist text is exact prefix of text paragraph). 1460 entries down to 1330.
• Schema: add optional damage to ReactionSchema. Add optional spellcasting to reactions and legendary actions. Add optional skills, gear, bonus_actions to MonsterSchema. Omit skills/gear when source say None.
• Split script into modules: monsterText.mjs (normalize, parse blocks, pair to monsters), monsterEntries.mjs (damage, dc, debris cleanup), monsterSpellcasting.mjs, monsterCommon.mjs. Output byte-identical before and after split.
• Structure all 178 Multiattack entries: multiattack_type, actions, action_options. Same Choice shape as 2014 and existing 2024 monsters. Attack type from the monster's own action (melee/ranged/special), or the spell's attack_type for casts. Description text unchanged.
• New convention for "any combination": inside a Choice, an action option count is the most times that option can be picked. "Three attacks, Claw or Tail in any combination" = choose: 3, Claw count: 3, Tail count: 3. Replacements join the choice (count = attacks replaced). No existing data use count > 1 inside choose > 1, so this is new. Please check it fits API consumers.
• Split form monsters like 2014: werebear, wereboar, wererat, weretiger, werewolf into human/hybrid/animal, vampire into vampire/bat/mist. 329 monsters become 341. Same index, name and image as 2014, linked by forms. Entries tagged "Form Only" go only to their forms (tag removed), untagged go to every form. Multiattack options pruned to actions the form has. Size from Shape-Shift. Speed from text Speed line (gist had animal-form value, e.g. werewolf 40 ft. for every form) or from Shape-Shift for vampire bat and mist. Human and vampire form keep base size "Medium or Small".
• Merge fake entries gist made from tail of Failure:/Success: lines (lycanthrope curse, steam mephit Steam Breath).
• armor_class: no type (2024 SRD has no text after AC value, per review comment). type now optional in schema. Armor and shields named in Gear link to 2024 equipment through armor_class.armor (28 monsters). Half Plate Armor matches Half-Plate Armor.
• Independent audit (subagent) found and confirmed 4 issues, now fixed:
  • size/type split wrong for 47 monsters ("Medium or Small Humanoid" -> size "Medium", type "or small humanoid"). Split repaired from the gist's own two fields; form-split monsters (finding Add the rest of the races and subraces #1) still get a single size per form.
  • languages truncated mid-sentence for 13 monsters in the gist (dretch, kraken, and 11 others); now read from the stat block text instead.
  • Tarrasque's "World-Shaking Movement" legendary action had ~150 characters of the next stat block's ability scores, gear, senses and CR line bled in, because that block's title ("Toughs") has no AC line of its own so went undetected. Fixed generally: the truncation-repair pass now always stops at the ability-score table header, and a title followed straight by that header (no AC/HP/Speed) is now recognized too.
• usage (finding Reviewed data -- Added consistency and fixed a few typos. #2, 0 -> 150 entries): parsed out of recharge/per-day name suffixes. "Legendary Resistance (3/Day, or 4/Day in Lair)" -> name "Legendary Resistance", usage: {type: "per day", times: 3, times_in_lair: 4}, matching the 3 reference monsters exactly. Covers per day, per day in lair (special abilities only), recharge on roll (dice always "1d6"), recharge after a rest. Non-usage suffixes ("level N version", "Requires Magic Rope", mimic's form tag) left alone; a trailing non-usage note (night hag "Requires Soul Bag") kept in the name. Added rest_types to ActionUsageSchema and usage to Reaction/LegendaryAction schemas, since the data has recharge/per-day on reactions and bonus actions too, not just special abilities and actions.
• skills/proficiencies (finding Add spells from Xanathar's Guide to Everything #5): decided in favor of matching 2014. Skills now encoded as proficiencies entries (skill-history, skill-perception, and so on, looked up from 5e-SRD-Proficiencies.json), alongside saving throws, same as every 2014 monster. skills string field dropped from output and schema.
• attack_bonus: 440 attacks had none set, though schema support it and reference aboleth's Tentacle has one. Gist's own attack_bonus was a leftover spell-attack modifier on 5 dragons' Spellcasting action (already correct in spellcasting.modifier), never a weapon attack bonus; dropped, and parsed properly from "Attack Roll: +N" in the description, same as damage and dc. Found while comparing generated output against the reference monsters ahead of a final cleanup PR.
• Second independent audit pass (subagent), scoped to the finalized PR (feat(2024): finalize generated monster data #1231): found and fixed 1 new critical issue plus 3 related truncations.
  • damage_immunities/condition_immunities: the gist's own "Immunities" extraction lands a conditions-only line (no damage type, so no semicolon in the source) whole in damage_immunities, leaving condition_immunities empty. Confirmed for 23 monsters (gray-ooze, ettin, cloaker, and 20 others). Fixed by reading Resistances/Vulnerabilities/Immunities from the stat block text, classifying an unsemicoloned Immunities line word-by-word against the known damage types and conditions.
  • damage_resistances truncated (trailing empty string) for 4 undead (ghost, specter, will-o-wisp, wraith); rakshasa/half-dragon's single-phrase vulnerability/resistance also truncated. Same fix.
  • Condition parenthetical notes (archmage's "Charmed (with Mind Blank)") were being discarded; now captured in APIReferenceSchema's existing optional note field.
• Regenerate 5e-SRD-Monsters-New.json.

How was it tested?

New src/2024/tests/monsterGeneration.test.ts. Run script into temp dir, no overwrite of real files. Check: count + unique index, schema, hit dice format, passive perception, sense format, XP, PB vs CR, skill bonus = mod + PB or 2 x PB, every save DC and damage roll in description present in output, every spell cast has valid spell URL, no spell list entries or unterminated descriptions. Every Multiattack must be structured and simple counts must round trip. Forms split, linked and pruned; werewolf human/wolf entries and speeds checked. Armor links checked against equipment. Full suite 113 pass (7 new tests total for both audit passes).

Known gaps

• spell save DC not put in dc. It live in spellcasting.dc.
• Dryad Multiattack read as one choice of Vine Lash, Thorn Burst or Charm Monster. Text say "and it can use Spellcasting" so could mean attack plus cast.
• Clay golem Multiattack condition ("if it used Hasten this turn") only in Choice desc, not structured.
• components_required computed from text: "no Material components" give ["V","S"].
• Reference adult-blue-dragon (5e-SRD-Monsters.json) has extra acid-arrow and modifier: 5. Book text has neither. Output follow book text.
• Charmed (with Mind Blank) on archmage become plain charmed.
• Escape DC and first-named ability of check DC map to one ability. Book allow choice.
• shambling-mound and giant-frog Stealth match book, not mod + PB. Listed as quirks in test.
• Mist form gets every untagged entry (Charm, legendary actions, traits) since tags followed strictly. Book say mist form can't take actions.
• Mimic Adhesive (Object Form Only) keeps tag. Not split in 2014.
• Some words keep gist compound artifacts like 5-footwide.
• Merged-entry split also cover (... Form Only) headings (wererat Hand Crossbow).

Is there a Github issue this is resolving?

NA

Did you update the docs in the API? Please link an associated PR if applicable.

No

🤖 Generated with Claude Code

[G-Ambatte]

• armor_class.type, in this case defined as "the text in parenthesis after the AC value", does not exist in the 2024 SRD.

[bagelbits]

Thanks for catching that. You're right, the 2024 SRD has no text after the AC value, so armor_class.type had no source. The natural / armor value was a guess on my part, and I've removed it in 33c914e.

What changed:

• type is now optional in the 2024 monster schema, and the script no longer sets it. armor_class is just { value }.
• Where a stat block's Gear line names armor or a shield (Chain Shirt, Shield, Plate Armor and so on), the monster now links to the matching 2024 equipment entry through armor_class.armor. 28 monsters have links. Those come straight from the Gear text, so nothing is inferred beyond matching the item names.

The existing hand-made entries in 5e-SRD-Monsters.json that still carry type: "natural" keep validating, since the field is optional rather than removed.

This PR is stacked on #1226. Its base is a copy of your branch (add2024Monsters-base) in this repo, and it will be retargeted to main once #1226 merges. If you spot anything else in the generated data that doesn't match the source, please say so.