task(docs): evaluate Heroshot as replacement for custom screenshot generator

[gander]

Context

Issue #301 requires capturing multi-step form screenshots with realistic data, which will increase complexity in the current custom Playwright script (scripts/generate-screenshots.ts).

@omachala suggested Heroshot as a more maintainable alternative that externalizes screenshot configuration into YAML/JSON instead of hardcoded TypeScript.

Objective

Evaluate Heroshot's feasibility as a replacement for the current screenshot generation script by:

1. Analyzing feature parity with current implementation
2. Testing ease of implementation for our use cases
3. Identifying implementation steps and effort required
4. Making a recommendation (keep custom script vs. migrate to Heroshot)

Current Implementation Analysis

Current script: scripts/generate-screenshots.ts

• Uses Playwright directly
• Hardcoded interactions in TypeScript
• Captures 11 screenshots (1024x768 viewport)
• Automatically starts/stops backend and frontend servers
• Manual form filling logic for test data

Pain points:

• Adding new scenarios requires TypeScript code changes
• Multi-step form interactions need complex Playwright logic
• No built-in light/dark mode support
• Manual viewport management

Heroshot Capabilities

Based on documentation review:

Core Features:

• Visual Picker: Initial setup through UI (no manual config writing)
• Config-based: All screenshots defined in .heroshot/config.json
• Actions API: Pre-built actions for form interactions (click, type, fill_form, wait, etc.)
• Multi-viewport: Desktop/tablet/mobile presets + custom dimensions
• Theme Support: Automatic light/dark mode variants (screenshot-light.png, screenshot-dark.png)
• CI/CD Ready: GitHub Actions integration with encrypted session storage
• Framework Integration: Vue component for rendering screenshots in docs

Actions API Relevant to Our Use Case:

{
  "actions": [
    {"type": "click", "selector": ".next-button"},
    {"type": "type", "selector": "input[name='url']", "text": "https://example.com"},
    {"type": "fill_form", "fields": [
      {"selector": "#projectName", "value": "My Project"},
      {"selector": "#description", "value": "Test description"}
    ]},
    {"type": "wait", "time": 500}
  ]
}

Vue Integration:

<script setup>
import { Heroshot } from 'heroshot/vue';
</script>
<template>
  <Heroshot name="dashboard" alt="Dashboard view" />
</template>

Implementation Steps (Proposed)

Phase 1: Proof of Concept

1. Install Heroshot

   npm install heroshot --save-dev

2. Create initial config for 1-2 existing screenshots

   npx heroshot config

   • Test capturing dashboard and project list views
   • Verify output matches current screenshots

3. Test multi-step form (the main challenge)

   • Configure actions for ProjectForm wizard:
     • Step 1: Fill basic info + click Next
     • Step 2: Fill crawl settings + click Next
     • Step 3: Fill run profile + capture
   • Compare ease vs. current Playwright code

Phase 2: Full Migration

1. Port all 11 screenshots to Heroshot config
2. Handle server startup/shutdown
   • Keep npm script wrapper or integrate into Heroshot workflow
   • May still need custom logic for backend/frontend coordination
3. Update documentation
   • docs/screenshots/README.md
   • docs/guides/frontend-dev.md#screenshot-updates
4. Remove old script (scripts/generate-screenshots.ts)

Phase 3: Enhancements (Optional)

1. Add Vue component for rendering screenshots in docs
2. CI/CD integration for automated screenshot updates on deploy
3. Light/dark mode variants (currently we only have light mode)

Evaluation Criteria

Criterion	Current Script	Heroshot	Winner
Ease of adding screenshots	TypeScript code	JSON config + visual picker	?
Multi-step form handling	Custom Playwright	Actions API (click, type, wait)	?
Server lifecycle	Built-in (auto start/stop)	Requires external script	?
Maintainability	Code changes for new scenarios	Config changes only	?
Theme support	Manual (single theme)	Automatic light/dark	Heroshot ✅
Documentation integration	Manual image links	Vue component + auto-switching	Heroshot ✅
Learning curve	Playwright knowledge	Heroshot config format	?

Questions to Answer During Evaluation

1. Form interaction complexity: Can Heroshot's Actions API handle our 3-step wizard reliably?
2. Server coordination: How do we integrate backend/frontend startup with Heroshot workflow?
3. Test data: How do we inject realistic form data (currently hardcoded in TS)?
4. Migration effort: Is the migration justified vs. extending current script?
5. Long-term maintenance: Will Heroshot reduce ongoing screenshot maintenance burden?

Implementation Suggestions

If migrating to Heroshot:

1. Keep server startup script separately

   // package.json
   {
     "scripts": {
       "screenshots": "npm run screenshots:servers && npx heroshot",
       "screenshots:servers": "node scripts/start-servers.js"
     }
   }

2. Example config for multi-step form

   {
     "outputDirectory": "docs/screenshots",
     "browser": {
       "viewport": {"width": 1024, "height": 768}
     },
     "screenshots": [
       {
         "name": "03-project-create-step1",
         "url": "http://localhost:5173/projects/new",
         "actions": [
           {"type": "type", "selector": "#projectName", "text": "My Website Upgrade"},
           {"type": "type", "selector": "#websiteUrl", "text": "https://example.com"},
           {"type": "type", "selector": "#description", "text": "Testing framework upgrade"},
           {"type": "wait", "time": 200}
         ]
       },
       {
         "name": "03-project-create-step2",
         "url": "http://localhost:5173/projects/new",
         "actions": [
           {"type": "type", "selector": "#projectName", "text": "My Website Upgrade"},
           {"type": "click", "selector": ".next-button"},
           {"type": "wait", "time": 300},
           {"type": "click", "selector": "#crawlEnabled"},
           {"type": "type", "selector": "#maxPages", "text": "50"},
           {"type": "wait", "time": 200}
         ]
       }
     ]
   }

3. Vue component integration (optional enhancement)

   <!-- In documentation or README -->
   <Heroshot name="dashboard" alt="Dashboard view" />

Acceptance Criteria

• Heroshot installed and tested with 1-2 screenshots
• Multi-step form (ProjectForm) successfully captured with Actions API
• Evaluation document created comparing current vs. Heroshot approach
• Decision made: migrate to Heroshot OR keep custom script with rationale
• If migrating: implementation plan finalized with effort estimate
• If keeping custom script: documented why Heroshot doesn't fit use case

Related Documentation

• Heroshot Vue Integration: https://heroshot.sh/docs/integrations/vue.md
• Heroshot Full Docs: https://heroshot.sh/llms-full.txt
• Current script: scripts/generate-screenshots.ts
• Frontend screenshot guide: docs/guides/frontend-dev.md#screenshot-updates

[gander]

Implementation Testing Summary

I've completed a comprehensive evaluation of Heroshot as a replacement for the custom screenshot generator, including practical testing with both simple views and the complex multi-step wizard from #301.

---

What Was Tested

1. Heroshot Proof of Concept

• ✅ Installed Heroshot as dev dependency
• ✅ Created initial config (.heroshot/config.json) for 2 simple screenshots:
  • Dashboard view (01-dashboard)
  • Projects list (02-projects-list)
• ✅ Successfully generated screenshots with automatic light/dark mode variants (killer feature!)

2. Multi-Step Wizard Testing (Issue #301 Requirements)

Attempted to capture the 3-step ProjectForm wizard with realistic data:

• Step 1: URL="https://example.com", Name="My Website Upgrade", Description="Testing framework upgrade from v4 to v5"
• Step 2: Crawl enabled=true, Max pages=50
• Step 3: Run Profile (default settings)

Result: ❌ Heroshot's Actions API could not reliably interact with Naive UI nested components. The process timed out without generating screenshots.

3. Playwright Script Extension

• ✅ Extended existing scripts/generate-screenshots.ts with captureProjectWizard() function
• ✅ Successfully generated all 3 wizard screenshots with filled data
• ✅ Fixed selector issue: Naive UI wraps inputs in divs, requiring [data-test="url-input"] input instead of just [data-test="url-input"]

---

Key Findings

Heroshot Strengths

1. Automatic light/dark mode - Generates both *-light.png and *-dark.png variants automatically (currently the project only has light mode)
2. Config-based approach - No TypeScript code changes needed to add new screenshots
3. Vue integration - Future enhancement for documentation (<Heroshot name="dashboard" />)
4. Ease of use - Simple JSON config vs. complex Playwright code

Heroshot Limitations

1. Complex form interactions - Actions API cannot handle multi-step wizards with nested Naive UI components
2. Server lifecycle - Requires external wrapper script (same as current implementation)
3. Pre-release version - Currently 0.0.0 suggests early development stage

Playwright Strengths

1. Reliable complex interactions - Full control over multi-step workflows
2. Nested component handling - Can target actual input elements within wrapper divs
3. Debugging - Full browser automation with detailed error messages

---

Recommendation: Hybrid Approach ✅

After comprehensive testing, I recommend a hybrid solution combining both tools:

Use Heroshot for Simple Views (8-9 screenshots)

• Dashboard, project list, project detail
• Run list, run detail, page detail
• Rules list, rule create, settings, 404 page
• Benefit: Automatic light/dark mode variants with minimal maintenance

Use Playwright for Complex Forms (2-3 screenshots)

• Multi-step wizard (project creation: 3 steps)
• Forms with nested Naive UI components requiring precise interaction
• Benefit: Reliable interaction handling with full control

---

Generated Artifacts

Screenshots (14 total)

All screenshots successfully generated and saved to docs/screenshots/:

• ✅ 03-project-create-step1.png (54KB) - Basic Info with filled data
• ✅ 03-project-create-step2.png (50KB) - Crawl Settings with enabled crawl
• ✅ 03-project-create-step3.png (61KB) - Run Profile
• ✅ All 11 regular screenshots (dashboard, lists, details, etc.)

Documentation

• ✅ HEROSHOT_EVALUATION.md - Comprehensive evaluation document including:
  • Feature comparison matrix (Heroshot wins 6 vs 2 on evaluated criteria)
  • Test results and findings
  • Migration plan with 4 phases
  • Risk assessment and mitigation strategies
  • Hybrid approach recommendation with implementation details

---

Implementation Status

Completed ✅

1. Heroshot proof of concept with simple views
2. Multi-step wizard implementation in Playwright script
3. All 14 screenshots generated successfully with realistic data per feat(docs): add complete project creation wizard screenshots #301
4. Comprehensive evaluation document

Next Steps (Optional)

1. Create full Heroshot config for 8-9 simple views (to enable automatic light/dark variants)
2. Keep captureProjectWizard() function in Playwright script for complex wizards
3. Document hybrid workflow in docs/screenshots/README.md

---

References

• Issue feat(docs): add complete project creation wizard screenshots #301: Multi-step form screenshot requirements (source for test data)
• Documentation: docs/guides/frontend-dev.md#screenshot-updates
• Heroshot Config: .heroshot/config.json (testing configuration)
• Evaluation Document: HEROSHOT_EVALUATION.md (detailed analysis)
• Playwright Script: scripts/generate-screenshots.ts (extended with wizard function)

---

Conclusion

Heroshot is an excellent tool for simple screenshot generation with automatic theme variants, but its Actions API cannot handle complex multi-step forms with Naive UI components. The hybrid approach leverages the strengths of both tools:

• Heroshot for maintainability and automatic light/dark modes on simple views
• Playwright for reliability on complex interactions

This evaluation provides clear guidance for future implementation decisions while successfully delivering all required screenshots for #301.

[omachala]

Hey, thanks for giving heroshot a try and for the detailed evaluation! Really helpful to see it tested against real-world use cases.

The multi-step form scenario is exactly the kind of feedback I'm looking for — if you hit any friction with the Actions API, let me know. Happy to help or add features if something's missing.

Keep the issues coming — they've been super useful!