Comprehensive Documentation Validation

Status: ✅ Complete · Priority: High · Created: 2025-11-06 · Tags: documentation, quality, validation, v0.2.0

Project: lean-spec
Team: Core Development

📊 Results: See VALIDATION-RESULTS.md for complete findings (11 issues documented)

Overview

Goal: Deep validation that ALL documentation accurately reflects actual implementation and design

Spec 056 fixed obvious inaccuracies (wrong template names, folder structures). This spec is a comprehensive validation pass to catch subtle mismatches between docs and reality.

Why now:

• Part of v0.2.0 launch quality gate (spec 043)
• Spec 056 was reactive (fixing known issues)
• This is proactive (systematic validation)
• Can't launch with docs that mislead users

Scope: Every claim, example, and explanation in docs-site must match:

1. Actual CLI behavior
2. Source code implementation
3. Config file formats
4. Template structures
5. First principles (spec 049)

Design

Validation Approach

Method: Systematic cross-reference check

For each documentation page:

1. Extract claims - What does the doc say the system does?
2. Verify against code - Does the implementation match?
3. Test examples - Do all code examples actually work?
4. Check consistency - Do different pages tell the same story?

Areas to Validate

1. CLI Commands (docs/reference/cli.mdx)

• Every command exists and works as documented
• All flags and options are correct
• Output examples match actual output format
• Error messages match reality
• Command aliases work as stated

2. Configuration (docs/reference/config.mdx)

• All config fields exist and work
• Default values are correct
• Examples are valid JSON
• Variable substitution works as documented
• Custom fields behavior matches docs

3. Frontmatter (docs/reference/frontmatter.mdx, docs/guide/frontmatter.mdx)

• Required fields match actual validation
• Optional fields work as documented
• Status values match StatusSchema
• Priority values match PrioritySchema
• Badge rendering matches examples

4. Templates (docs/guide/templates.mdx)

• All three templates exist: minimal, standard, enterprise
• Frontmatter examples match actual template files
• Section descriptions match template content
• Variable substitution works as shown

5. Init Flow (docs/guide/getting-started.mdx)

• Prompts match actual init.ts implementation
• Template selection options are correct
• Pattern selection options exist
• File creation matches docs
• AGENTS.md handling works as described

6. Custom Fields (docs/guide/custom-fields.mdx)

• Type validation works (string, number, boolean, array)
• Filtering by custom fields works
• Examples are valid
• Config format is correct

7. Variables (docs/guide/variables.mdx)

• All documented variables exist
• Substitution syntax is correct
• Built-in variables work: {date}, {name}, {project_name}, etc.
• Custom variables work as documented

8. AI Integration (docs/ai-integration/*.mdx)

• AGENTS.md template is current
• MCP setup instructions work
• Discovery commands exist and work
• Examples match actual behavior
• Best practices align with current design

9. Core Concepts

• Philosophy - Aligns with actual design decisions
• First Principles - Examples match reality
• Agile Principles - Guidance is practical
• When to Use - Decision framework is sound

10. Examples Throughout

• All bash examples work when run
• All YAML/JSON examples parse correctly
• All file paths are valid
• All command outputs are realistic

Common Issues to Check

Command mismatches:

• Flags that don't exist
• Options that changed
• Output format differences

Config mismatches:

• Field names that changed
• Structure that evolved
• Defaults that differ

Conceptual mismatches:

• Docs say X but code does Y
• Philosophy contradicts implementation
• Inconsistent terminology

Stale information:

• Old template names
• Deprecated commands
• Changed workflows

Plan

Preparation

• Set up validation environment
• Review spec 056 findings (don't duplicate)
• Familiarize with current codebase

Validation (Page by Page)

• docs/guide/index.mdx - Overview
• docs/guide/getting-started.mdx - Installation & init
• docs/guide/philosophy.mdx - Core philosophy
• docs/guide/first-principles.mdx - First principles
• docs/guide/principles.mdx - Agile principles
• docs/guide/when-to-use.mdx - Decision framework
• docs/guide/templates.mdx - Template system
• docs/guide/frontmatter.mdx - Frontmatter fields
• docs/guide/custom-fields.mdx - Custom fields
• docs/guide/variables.mdx - Variable substitution
• docs/guide/development.mdx - Contributing
• docs/reference/cli.mdx - CLI commands
• docs/reference/config.mdx - Configuration
• docs/reference/frontmatter.mdx - Frontmatter reference
• docs/ai-integration/index.mdx - AI overview
• docs/ai-integration/setup.mdx - AI setup
• docs/ai-integration/agents-md.mdx - AGENTS.md template
• docs/ai-integration/best-practices.mdx - AI best practices
• docs/ai-integration/examples.mdx - AI examples

Documentation

• Document all found issues
• Categorize by severity (critical/medium/minor)
• Create fix checklist
• Update spec 056 if issues already partially fixed

Validation

• All issues documented
• Fix plan created
• Ready for implementation

Test

Success Criteria

Completeness:

• Every doc page validated
• Every code example tested
• Every claim verified

Accuracy:

• All critical mismatches found
• All medium mismatches found
• Minor mismatches documented

Actionability:

• Clear list of issues with locations
• Severity rating for each issue
• Specific fix recommendations

Validation Checklist

For each issue found, document:

1. Location: File path and line/section
2. Issue: What's wrong (docs say X, reality is Y)
3. Severity: Critical / Medium / Minor
4. Fix: What needs to change
5. Verification: How to test the fix

Notes

Why This Matters

Spec 056 caught obvious issues (wrong template names). But subtle issues are worse:

• User follows docs, gets unexpected behavior
• User loses trust in documentation
• Support burden increases
• Launch reputation suffers

Scope Boundaries

In scope:

• Technical accuracy (commands, configs, examples)
• Conceptual alignment (philosophy matches design)
• Example validity (code actually works)

Out of scope:

• Writing quality/grammar (not accuracy issue)
• Missing documentation (not incorrect docs)
• Feature requests (not validation)

For the Coding Agent

Instructions:

1. Read this spec thoroughly
2. Go through each doc page systematically
3. For each claim/example, verify against:
   • src/ source code
   • templates/ actual templates
   • lean-spec --help actual CLI
4. Document every mismatch found
5. Use the validation checklist format
6. Categorize by severity
7. Create actionable fix list

Output format:

## Issue #N: [Short description]
- **Location**: `docs/path/file.mdx` (line X or section "Y")
- **Docs say**: "..."
- **Reality is**: "..."
- **Severity**: Critical/Medium/Minor
- **Fix**: [Specific change needed]
- **Verification**: [How to test]

Create results in: specs/057-docs-validation-comprehensive/VALIDATION-RESULTS.md

Related

• Spec 043: v0.2.0 launch (parent)
• Spec 056: Initial docs audit (predecessor)
• Spec 049: First principles (validation reference)
• Accomplishes "Documentation accuracy verified" quality gate

Results Summary

Validation Status: ✅ Complete

Issues Found: 11 total

• Critical: 5 issues
• Medium: 5 issues
• Minor: 1 issue

Key Findings:

1. Configuration documentation completely out of sync with implementation
2. Invalid status values (blocked, cancelled) documented but don't exist
3. Multiple CLI commands missing documented options
4. Status icon inconsistencies between documentation and code

Deliverable: Complete validation results in VALIDATION-RESULTS.md

Next Steps: Issues documented and ready for fixing in a separate spec/PR