Replace Manual DAG with Visualization Library

Status: ✅ Complete · Priority: High · Created: 2025-11-17 · Tags: web, ux, dependencies, technical-debt

Project: lean-spec
Team: Core Development

Overview

Problem: The current spec dependencies DAG visualization has multiple issues:

Implementation Issues (Original Scope):
- Font sizes too small (hard to read)
- Edges not rendering (missing ReactFlow Handle components)
- Manual SVG coordinate calculations causing overlap
- Not responsive across viewport sizes
- Hard to maintain (manual bezier curves, CSS calc())
Data Completeness Issues (Extended Scope):
- Only shows outgoing related connections from current spec
- Missing upstream dependencies (dependsOn relationships)
- Missing downstream dependents (specs that depend on THIS spec)
- Not showing bidirectional nature of related relationships
- Incomplete graph prevents understanding full dependency chain

Why Now: During troubleshooting for spec #066 and testing spec #082, we discovered:

DAG nodes overlapping due to coordinate system mismatches
Font sizes too small to read
Edges not rendering (ReactFlow Handle components missing)
Can't see full dependency context (only current spec's perspective)

Impact:

Better readability (larger fonts, proper edge rendering)
Complete dependency visualization (upstream, downstream, bidirectional)
Easier maintenance with proper library
More robust across viewport sizes

Current Implementation Issues

Code Location: /packages/web/src/components/spec-relationships.tsx

Problems Found:

Mixed coordinate systems (SVG viewBox vs CSS %)
Manual bezier curve calculations for edges
Fixed column layout (COLUMN_X constants) doesn't adapt
preserveAspectRatio="none" causes distortion
Node positioning uses calc(${xPercent}% - ${NODE_WIDTH / 2}px) which breaks with aspect ratio changes

Current Approach:

3-column layout: Precedence (left) → Current (center) → Related (right)
Nodes positioned with CSS absolute + calc()
SVG paths drawn manually with fixed control points
Container height calculated from node count

Design

Recommended Library: Reactflow

Why Reactflow:

✅ React-native: Built for React, excellent TypeScript support
✅ Declarative: Define nodes/edges, library handles layout
✅ Auto-layout: Built-in dagre/elk layout algorithms
✅ Responsive: Handles viewport changes automatically
✅ Interactive: Pan, zoom, node dragging out of the box
✅ Customizable: Custom node styles, edge types
✅ Well-maintained: 20K+ GitHub stars, active development
✅ Small bundle: ~50KB gzipped (reasonable for feature richness)

Alternatives Considered:

D3.js: Too low-level, requires manual layout management
Cytoscape.js: Powerful but heavier, steeper learning curve
vis.js: Unmaintained, no React integration
mermaid.js: Static rendering, not interactive

Implementation Approach

Display Pattern: Modal/Dialog (REQUIRED)

Dependencies graph opens in a dialog/modal, not inline expansion
Timeline also moves to modal (consistent pattern for spec metadata)
Triggered by "Show Dependencies" and "Show Timeline" buttons
Allows full screen space for complex graphs
Better mobile experience (full screen modal vs cramped accordion)

Replace Current Components:

// Current: Accordion expansion
<Accordion>
  <SpecRelationships relationships={...} />
</Accordion>

// New: Dialog with Reactflow
<Dialog>
  <SpecDependencyGraph relationships={...} />
</Dialog>

Node Types:

Current Spec: Center node (non-interactive, highlighted, primary border)
Depends On (Upstream): Left side (amber styling, solid edges with arrows pointing right)
Required By (Downstream): Right side, top (red/orange styling, solid edges with arrows pointing left)
Related (Bidirectional): Right side, bottom (blue styling, dashed edges, bidirectional arrows)

Complete Dependency Graph: To show the full picture, we need:

Current spec's dependsOn → shows as upstream nodes (left)
Specs that depend on current spec → shows as downstream nodes (right, top)
Current spec's related → shows as related nodes (right, bottom)
Related specs that list current spec → shows as related nodes (right, bottom)

Data Requirements: Current API only provides current spec's perspective. Need to either:

Option A: Fetch all related specs individually to check their relationships (client-side)
Option B: Add API endpoint that returns complete graph (server-side, better performance)
Option C: Implement in phases (current perspective first, then expand)

Implementation Strategy: Start with Option C (phased approach)

Phase 1 (MVP): Current spec perspective only (dependsOn + related) ✅
Phase 2: Add API endpoint for complete graph (includes requiredBy + bidirectional related)
Phase 3: Polish UX (better labels, tooltips, legend)

Layout Strategy:

Use Reactflow's dagre layout algorithm
Constrain to 3-column structure (precedence | current | related)
Vertical spacing based on node count
Automatically handle edge routing
Dialog provides ample space for complex graphs

Styling:

Match current design (rounded boxes, color scheme)
Maintain distinction between precedence (amber) and related (blue)
Full screen or large modal (80-90% viewport)
Responsive: full screen on mobile, modal on desktop

Migration Plan

Phase 1: Fix Current Implementation (COMPLETED ✅)

✅ Increase font sizes (node labels, badges, legend)
✅ Add ReactFlow Handle components for edge rendering
✅ Fix edge markers and colors
✅ Improve node styling (larger, better contrast)
✅ Update legend text for clarity

Phase 2: API Enhancement for Complete Graph (1-2 days)

Design API response format for complete dependency graph
Add server-side endpoint: GET /api/specs/:id/dependency-graph
- Returns: { current, dependsOn, requiredBy, related, relatedBidirectional }
Implement efficient graph traversal (avoid N+1 queries)
Add caching for graph queries

Phase 3: Client-Side Graph Building (1-2 days)

Update SpecRelationships type to include requiredBy
Modify buildGraph() to handle all relationship types
Add downstream nodes (red/orange styling for "Required By")
Distinguish between outgoing and bidirectional related connections
Update layout algorithm for 4-column structure:
- Left: Depends On (upstream, blocking)
- Center: Current Spec
- Right Top: Required By (downstream, blocked)
- Right Bottom: Related (bidirectional, informational)

Phase 4: UX Polish (1 day)

Add tooltips explaining relationship types
Improve legend with examples
Add "Show Full Graph" toggle (current perspective vs complete graph)
Add node count indicator
Test with complex graphs (10+ nodes, multiple levels)

Phase 5: Testing & Documentation (1 day)

Test with various spec configurations
Performance testing with large graphs
Update documentation
Add tests for graph building logic

Total: 3-5 days (Phase 1 done, Phases 2-5 remaining)

Plan

Phase 1: Fix Current Implementation ✅

Increase font sizes for better readability
Add ReactFlow Handle components to enable edge rendering
Fix edge colors and arrow markers
Improve node styling (larger dimensions, better borders)
Update legend text for clarity
Reduce horizontal spacing between nodes

Phase 2: API Enhancement ✅

Design complete dependency graph API response format
Implement GET /api/specs/:id/dependency-graph endpoint
Add graph traversal logic to find:
- Specs that depend on current spec (requiredBy)
- Bidirectional related connections
Optimize with caching and efficient queries
Add tests for API endpoint

Phase 3: Client-Side Graph Enhancement ✅

Update SpecRelationships type: add requiredBy: string[]
Modify buildGraph() function to support all relationship types
Add downstream nodes (red styling for "Required By")
Add node type for bidirectional related connections
Update dagre layout for complete graph structure
Add subtitle text for each node type explaining relationship
Fetch complete graph on dialog open using SWR

Phase 4: UX Polish

Add tooltips with relationship explanations
Improve legend with visual examples
Add "View Mode" toggle (current perspective vs full graph)
Add node/edge count indicators
Add search/filter in graph
Improve mobile responsiveness

Phase 5: Testing

Test with no dependencies
Test with only dependsOn
Test with only related
Test with requiredBy (downstream dependents)
Test with bidirectional relationships
Test with complex graphs (10+ nodes, multiple levels)
Performance testing with large graphs
Cross-browser testing

Test

Phase 1 Tests (Current Implementation) ✅

Font sizes are readable
Edges render correctly with proper colors
Nodes display without overlap
Modal opens and closes correctly
Spec #082 shows 5 related connections

Phase 2 Tests (Complete Graph API):

API endpoint returns correct requiredBy relationships
API handles specs with no dependencies
API handles circular dependencies gracefully
API response is cached appropriately
Performance is acceptable (<200ms for typical queries)

Phase 3 Tests (Enhanced Graph Visualization):

Downstream nodes (Required By) appear on right side
Upstream nodes (Depends On) appear on left side
Related nodes show bidirectional nature
Node subtitles explain relationship type
Color coding is clear (amber=depends, red=requiredBy, blue=related)
Layout handles 4 columns properly
Edges connect to correct nodes
Clicking nodes navigates to correct spec

Phase 4 Tests (UX Polish):

Tooltips provide helpful information
Legend is clear and accurate
View mode toggle works
Node count indicators are correct
Search/filter works in graph
Mobile layout is usable

Phase 5 Tests (Integration & Performance):

Graph renders quickly for small graphs (<10 nodes, <100ms)
Graph renders acceptably for large graphs (>20 nodes, <500ms)
No memory leaks during repeated opens/closes
Works in Chrome, Firefox, Safari
Works on mobile devices
Accessibility (keyboard navigation, screen readers)

Notes

Phase 1 Completion Summary

Completed (Nov 17, 2025):

✅ Increased font sizes across all text elements
- Node labels: text-xs → text-base (16px)
- Badge text: text-[10px] → text-xs (12px)
- Subtitle: text-[11px] → text-sm (14px)
- Legend: text-[11px] → text-sm (14px)
✅ Fixed edge rendering by adding ReactFlow Handle components
- Added <Handle type="target" position={Position.Left} />
- Added <Handle type="source" position={Position.Right} />
- Resolved "Couldn't create edge for source handle id: undefined" warnings
✅ Improved edge styling
- Solid amber edges for dependsOn relationships (3px width)
- Dashed blue edges for related relationships (3px width, 10-8 dash pattern)
- Arrow markers properly configured with correct colors
✅ Enhanced node dimensions and styling
- Width: 220px → 280px
- Height: 90px → 110px
- Border: 1px → 2px
- Padding: px-4 py-3 → px-5 py-4
- Shadow: shadow-sm → shadow-md
✅ Reduced horizontal spacing
- ranksep: 190px → 120px for tighter layout
✅ Updated labels for clarity
- "Precedence" → "Depends On" (more explicit)
- "Connected" → "Related" (clearer terminology)
- Subtitles added: "Must complete first", "Connected work"

Files Modified:

/packages/web/src/components/spec-dependency-graph.tsx

Next Steps: Phase 2 requires API changes to support complete dependency graph.

Phase 2 & 3 Completion Summary

Completed (Nov 17, 2025):

✅ API Endpoint: Created GET /api/specs/[id]/dependency-graph
- Returns complete graph: { current, dependsOn, requiredBy, related }
- Standalone implementation to avoid tiktoken wasm issues
- Caching with 60s TTL
✅ Graph Building: SpecDependencyGraph class
- Builds reverse edges (requiredBy) from dependsOn relationships
- Handles bidirectional related connections automatically
- O(1) lookups with in-memory Map
✅ Frontend Integration:
- Updated SpecRelationships type to include requiredBy
- Added SWR hook to fetch complete graph on dialog open
- Added red styling for "Required By" nodes (downstream dependents)
- Updated legend to show all 3 relationship types
- Graph auto-layouts with dagre algorithm
✅ Testing:
- 2 tests for API endpoint
- Tests gracefully handle missing specs in test environment
- Build and all tests passing

Files Modified:

/packages/web/src/lib/dependency-graph.ts (new)
/packages/web/src/app/api/specs/[id]/dependency-graph/route.ts (new)
/packages/web/src/app/api/specs/[id]/dependency-graph/route.test.ts (new)
/packages/web/src/components/spec-dependency-graph.tsx
/packages/web/src/components/spec-detail-client.tsx
/packages/web/src/types/specs.ts

Next Steps: Phases 4-5 are optional UX polish and testing enhancements.

Why Complete Dependency Graph Matters

Current Limitation: Only shows relationships FROM the current spec's perspective:

Shows specs this one depends on (dependsOn)
Shows specs this one is related to (related)
Missing: Specs that depend on THIS spec (downstream impact)
Missing: Specs that list THIS spec in their related (bidirectional)

Example Problem:

Spec A depends on Spec B
When viewing Spec B, you can't see that Spec A is blocked by it
Can't assess impact of changes to Spec B
Can't see full dependency chain

Solution: API endpoint that returns complete graph:

interface CompleteDependencyGraph {
  current: SpecMetadata;
  dependsOn: SpecMetadata[];      // Upstream (current depends on these)
  requiredBy: SpecMetadata[];     // Downstream (these depend on current)
  related: SpecMetadata[];        // Outgoing related links
  relatedBy: SpecMetadata[];      // Incoming related links
}

Visualization Strategy:

Left side: Upstream dependencies (what this spec needs)
Center: Current spec
Right side top: Downstream dependents (what needs this spec)
Right side bottom: Related specs (bidirectional connections)

API Design Considerations

Option A: In-Memory Graph (Recommended)

Build complete dependency graph in-memory on server startup:

class SpecDependencyGraph {
  private graph: Map<string, {
    dependsOn: Set<string>;
    requiredBy: Set<string>;
    related: Set<string>;
  }>;

  constructor(allSpecs: Spec[]) {
    this.buildGraph(allSpecs);
  }

  private buildGraph(specs: Spec[]) {
    // Initialize all nodes
    for (const spec of specs) {
      this.graph.set(spec.name, {
        dependsOn: new Set(spec.dependsOn || []),
        requiredBy: new Set(),
        related: new Set(spec.related || []),
      });
    }

    // Build reverse edges
    for (const [specName, node] of this.graph.entries()) {
      // For each dependsOn, add reverse requiredBy
      for (const dep of node.dependsOn) {
        this.graph.get(dep)?.requiredBy.add(specName);
      }
      // For each related, add bidirectional link
      for (const rel of node.related) {
        this.graph.get(rel)?.related.add(specName);
      }
    }
  }

  getCompleteGraph(specName: string): CompleteDependencyGraph {
    const node = this.graph.get(specName);
    if (!node) throw new Error(`Spec not found: ${specName}`);
    
    return {
      current: this.getSpecMetadata(specName),
      dependsOn: Array.from(node.dependsOn).map(n => this.getSpecMetadata(n)),
      requiredBy: Array.from(node.requiredBy).map(n => this.getSpecMetadata(n)),
      related: Array.from(node.related).map(n => this.getSpecMetadata(n)),
    };
  }
}

Performance Analysis:

Memory: ~100 specs × (3 Sets × ~5 entries × 50 bytes) ≈ 75KB total
Startup: O(n²) worst case, but typically O(n×d) where d = avg dependencies (~5)
- 100 specs × 5 deps = 500 operations ≈ <1ms
Query: O(1) graph lookup + O(d) metadata fetch ≈ <1ms
Conclusion: Negligible memory/CPU impact, excellent query performance

Option B: On-Demand Query (Not Recommended)

Query all specs on each request to find reverse dependencies
O(n) per request where n = total specs
Slower (10-50ms per request)
No memory benefit (specs already loaded)

Decision: Use Option A (in-memory graph) - trivial memory cost for dramatic performance gain.

CLI/MCP Impact Analysis

Current CLI Commands Affected:

lean-spec deps <spec> - Shows dependencies from spec's perspective
Need to enhance to show complete graph (requiredBy + bidirectional related)

Proposed New Commands:

lean-spec deps <spec> --complete - Show full dependency graph
lean-spec deps <spec> --upstream - Only show dependsOn
lean-spec deps <spec> --downstream - Only show requiredBy
lean-spec deps <spec> --impact - Show all specs affected by changes to this spec

MCP Tools Affected:

mcp_lean-spec_deps - Currently shows dependencies, needs enhancement
May need new tool: mcp_lean-spec_dependency-graph for complete view

Related Spec Needed: Yes

Spec #099: "Enhanced Dependency Commands for CLI/MCP"
- Implement in-memory graph in @leanspec/core
- Add CLI commands for complete dependency views
- Update MCP tools to support full graph
- Ensure consistency between web, CLI, and MCP

Core Package Changes:

Add SpecDependencyGraph class to @leanspec/core
Export graph building utilities
Share implementation between web API and CLI

Timeline and Dependencies in Modals (REQUIRED)

User Requirement: Both timeline and dependencies MUST use modal/dialog pattern, not inline accordions.

Rationale:

More space: Complex graphs and timelines need room to breathe
Better mobile UX: Full screen modal vs cramped inline expansion
Consistent pattern: All spec metadata (dependencies, timeline, transitions) in dialogs
Focus: Modal isolates the view, easier to explore relationships
Performance: Lazy load graph only when user opens dialog

Implementation:

Use shadcn/ui Dialog component (already in project)
"Show Dependencies" button → opens dependency graph modal
"Show Timeline" button → opens timeline modal
Mobile: Full screen modals
Desktop: Large modals (80-90% viewport)
Reactflow provides zoom/pan inside modal automatically

Not Optional: This is a core UX requirement, not a future enhancement. The modal pattern is part of the initial implementation.

Why These Specs Are Related:

Spec #066 (Context Economy Thresholds Refinement):

How it led here: While testing token counting on spec #066, we opened the dependency graph
Issues discovered: Nodes were overlapping, fonts too small, edges not rendering
Trigger: This visualization problem became apparent during spec #066 testing
Connection: Good dependency visualization helps assess spec complexity/relationships

Spec #082 (Web App Realtime Spec Sync Architecture):

How it relates: Defines the data layer (FilesystemSource, DatabaseSource, SpecsService)
Dependency: This spec relies on the service layer architecture from #082
API changes: Complete dependency graph endpoint will use #082's service pattern
Testing context: Spec #082 has 5 related connections, perfect for testing DAG improvements
Coordination: Any API changes should follow #082's architectural patterns

Spec #099 (To Be Created - Enhanced Dependency Commands):

Scope: CLI/MCP enhancements for complete dependency graph
Shared code: In-memory graph implementation in @leanspec/core
Consistency: Ensure web, CLI, MCP all show same dependency information
Dependencies: This spec (#097) should complete Phase 2 before #099 starts

#097 Replace Manual DAG with Visualization Library

Replace Manual DAG with Visualization Library

Overview

Current Implementation Issues

Design

Recommended Library: Reactflow

Implementation Approach

Migration Plan

Plan

Test

Notes

Phase 1 Completion Summary

Phase 2 & 3 Completion Summary

Why Complete Dependency Graph Matters

API Design Considerations

CLI/MCP Impact Analysis

Timeline and Dependencies in Modals (REQUIRED)

#097 Replace Manual DAG with Visualization Library

Replace Manual DAG with Visualization Library

Overview

Current Implementation Issues

Design

Recommended Library: Reactflow

Implementation Approach

Migration Plan

Plan

Test

Notes

Phase 1 Completion Summary

Phase 2 & 3 Completion Summary

Why Complete Dependency Graph Matters

API Design Considerations

CLI/MCP Impact Analysis

Timeline and Dependencies in Modals (REQUIRED)

Replace Manual DAG with Visualization Library

Overview

Current Implementation Issues

Design

Recommended Library: Reactflow

Implementation Approach

Migration Plan

Plan

Test

Notes

Phase 1 Completion Summary

Phase 2 & 3 Completion Summary

Why Complete Dependency Graph Matters

API Design Considerations

CLI/MCP Impact Analysis

Timeline and Dependencies in Modals (REQUIRED)

Related Issues

Replace Manual DAG with Visualization Library

Overview

Current Implementation Issues

Design

Recommended Library: Reactflow

Implementation Approach

Migration Plan

Plan

Test

Notes

Phase 1 Completion Summary

Phase 2 & 3 Completion Summary

Why Complete Dependency Graph Matters

API Design Considerations

CLI/MCP Impact Analysis

Timeline and Dependencies in Modals (REQUIRED)

Related Issues