silo/docs/GAP_ANALYSIS.md

Silo Gap Analysis and Revision Control Roadmap

Date: 2026-01-24
Status: Analysis Complete

---

Executive Summary

This document analyzes the current state of the Silo project against its specification, identifies documentation and feature gaps, and outlines a roadmap for enhanced revision control capabilities.

---

1. Documentation Gap Analysis

1.1 Current Documentation

Document	Coverage	Status
README.md	Quick start, overview	Partial (50%)
docs/SPECIFICATION.md	Design specification	Comprehensive (90%)
docs/STATUS.md	Development progress	Current but incomplete
silo-spec.md	Duplicate of SPECIFICATION.md	Redundant

1.2 Documentation Gaps (Priority Order)

High Priority

Gap	Impact	Effort
API Reference	Users cannot integrate programmatically	Medium
Deployment Guide	Cannot deploy to production	Medium
Database Schema Guide	Migration troubleshooting difficult	Low
Configuration Reference	config.yaml options undocumented	Low

Medium Priority

Gap	Impact	Effort
User Workflows	Users lack step-by-step guidance	Medium
FreeCAD Command Reference	Addon features undiscoverable	Low
Troubleshooting Guide	Support burden increases	Medium
Developer Setup Guide	Onboarding friction	Low

Lower Priority

Gap	Impact	Effort
CHANGELOG.md	Version history unclear	Low
Architecture Decision Records	Design rationale lost	Medium
Integration Guide	Third-party integration unclear	High

1.3 Recommended Actions

1. Consolidate specs: Remove silo-spec.md duplicate
2. Create docs/API.md: Full REST endpoint documentation with examples
3. Create docs/DEPLOYMENT.md: Production deployment guide
4. Expand README.md: Add configuration reference section

---

2. Current Revision Control Implementation

2.1 What's Implemented

Feature	Status	Location
Append-only revision history	Complete	internal/db/items.go
Sequential revision numbering	Complete	Database trigger
Property snapshots (JSONB)	Complete	revisions.properties
File versioning (MinIO)	Complete	internal/storage/
SHA256 checksums	Complete	Captured on upload
Revision comments	Complete	revisions.comment
User attribution	Complete	revisions.created_by
Property schema versioning	Complete	Migration 005
BOM revision pinning	Complete	relationships.child_revision

2.2 Database Schema

-- Current revision schema (migrations/001_initial.sql)
CREATE TABLE revisions (
    id UUID PRIMARY KEY,
    item_id UUID REFERENCES items(id) ON DELETE CASCADE,
    revision_number INTEGER NOT NULL,
    properties JSONB NOT NULL DEFAULT '{}',
    file_key TEXT,
    file_version TEXT,        -- MinIO version ID
    file_checksum TEXT,       -- SHA256
    file_size BIGINT,
    thumbnail_key TEXT,
    created_at TIMESTAMPTZ DEFAULT now(),
    created_by TEXT,
    comment TEXT,
    property_schema_version INTEGER DEFAULT 1,
    UNIQUE(item_id, revision_number)
);

2.3 API Endpoints

Endpoint	Method	Status
/api/items/{pn}/revisions	GET	Implemented
/api/items/{pn}/revisions	POST	Implemented
/api/items/{pn}/revisions/{rev}	GET	Implemented
/api/items/{pn}/file	POST	Implemented
/api/items/{pn}/file	GET	Implemented (latest)
/api/items/{pn}/file/{rev}	GET	Implemented

2.4 FreeCAD Integration

Command	Function	Status
Silo_Save	Auto-save + upload	Implemented
Silo_Commit	Save with comment	Implemented
Silo_Pull	Download/create	Implemented
Silo_Push	Batch upload	Implemented
Silo_Info	View revision history	Implemented

---

3. Revision Control Gaps

3.1 Critical Gaps

Gap	Description	Impact
No rollback	Cannot revert to previous revision	Data recovery difficult
No comparison	Cannot diff between revisions	Change tracking manual
No locking	No concurrent edit protection	Multi-user unsafe
No approval workflow	No release/sign-off process	Quality control gap

3.2 Important Gaps

Gap	Description	Impact
No branching	Linear history only	No experimental variants
No tagging	No named milestones	Release tracking manual
No audit log	Actions not logged separately	Compliance gap
Thumbnail missing	Schema exists, not populated	No visual preview

3.3 Nice-to-Have Gaps

Gap	Description	Impact
No search	Cannot search revision comments	Discovery limited
No retention policy	Revisions never expire	Storage grows unbounded
No delta storage	Full file per revision	Storage inefficient
No notifications	No change alerts	Manual monitoring required

---

4. Revision Control Roadmap

Phase 1: Foundation (Recommended First)

Goal: Enable safe single-user revision management

1.1 Rollback Support

Effort: Medium | Priority: High | Risk: Low

Changes Required:

• Add POST /api/items/{pn}/rollback/{rev} endpoint
• Create new revision copying properties/file from target revision
• FreeCAD: Add Silo_Rollback command

Database: No schema changes needed (creates new revision from old)

1.2 Revision Comparison API

Effort: Medium | Priority: High | Risk: Low

Changes Required:

• Add GET /api/items/{pn}/revisions/compare?from={rev1}&to={rev2} endpoint
• Return property diff (added/removed/changed keys)
• Return file metadata diff (size, checksum changes)

Implementation:

type RevisionDiff struct {
    FromRevision int                    `json:"from_revision"`
    ToRevision   int                    `json:"to_revision"`
    Properties   PropertyDiff           `json:"properties"`
    FileChanged  bool                   `json:"file_changed"`
    FileSizeDiff int64                  `json:"file_size_diff,omitempty"`
}

type PropertyDiff struct {
    Added   map[string]any `json:"added,omitempty"`
    Removed map[string]any `json:"removed,omitempty"`
    Changed map[string]PropertyChange `json:"changed,omitempty"`
}

1.3 Revision Labels/Status

Effort: Low | Priority: Medium | Risk: Low

Database Migration:

ALTER TABLE revisions ADD COLUMN status TEXT DEFAULT 'draft';
-- Values: 'draft', 'review', 'released', 'obsolete'

ALTER TABLE revisions ADD COLUMN labels TEXT[] DEFAULT '{}';
-- Arbitrary tags: ['prototype', 'v1.0', 'customer-approved']

API Changes:

• Add PATCH /api/items/{pn}/revisions/{rev} for status/label updates
• Add filtering by status in list endpoint

---

Phase 2: Multi-User Support

Goal: Enable safe concurrent editing

2.1 Pessimistic Locking

Effort: High | Priority: High | Risk: Medium

Database Migration:

CREATE TABLE item_locks (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    item_id UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE,
    locked_by TEXT NOT NULL,
    locked_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    expires_at TIMESTAMPTZ NOT NULL,
    lock_type TEXT NOT NULL DEFAULT 'exclusive',
    comment TEXT,
    UNIQUE(item_id)
);

CREATE INDEX idx_locks_expires ON item_locks(expires_at);

API Endpoints:

POST   /api/items/{pn}/lock      # Acquire lock
DELETE /api/items/{pn}/lock      # Release lock
GET    /api/items/{pn}/lock      # Check lock status

FreeCAD Integration:

• Auto-lock on Silo_Pull (configurable)
• Auto-unlock on Silo_Save/Silo_Commit
• Show lock status in Silo_Info

2.2 Authentication (LDAP/FreeIPA)

Effort: High | Priority: High | Risk: Medium

Changes Required:

• Add internal/auth/ package
• LDAP bind configuration in config.yaml
• Middleware for API authentication
• created_by populated from authenticated user

Configuration:

auth:
  enabled: true
  provider: ldap
  ldap:
    server: ldap://freeipa.example.com
    base_dn: cn=users,cn=accounts,dc=example,dc=com
    bind_dn: uid=silo-service,cn=users,...
    bind_password_env: LDAP_BIND_PASSWORD

2.3 Audit Logging

Effort: Medium | Priority: Medium | Risk: Low

Database Migration:

CREATE TABLE audit_log (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    timestamp TIMESTAMPTZ NOT NULL DEFAULT now(),
    user_id TEXT NOT NULL,
    action TEXT NOT NULL,  -- 'create', 'update', 'delete', 'lock', 'unlock'
    resource_type TEXT NOT NULL,  -- 'item', 'revision', 'project', 'relationship'
    resource_id TEXT NOT NULL,
    details JSONB,
    ip_address TEXT
);

CREATE INDEX idx_audit_timestamp ON audit_log(timestamp DESC);
CREATE INDEX idx_audit_user ON audit_log(user_id);
CREATE INDEX idx_audit_resource ON audit_log(resource_type, resource_id);

---

Phase 3: Advanced Features

Goal: Enhanced revision management capabilities

3.1 Branching Support

Effort: High | Priority: Low | Risk: High

Concept: Named revision streams per item

Database Migration:

CREATE TABLE revision_branches (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    item_id UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE,
    name TEXT NOT NULL,  -- 'main', 'experimental', 'customer-variant'
    base_revision INTEGER,  -- Branch point
    created_at TIMESTAMPTZ DEFAULT now(),
    created_by TEXT,
    UNIQUE(item_id, name)
);

ALTER TABLE revisions ADD COLUMN branch_id UUID REFERENCES revision_branches(id);

Complexity: Requires merge logic, conflict resolution UI

3.2 Release Management

Effort: Medium | Priority: Medium | Risk: Low

Database Migration:

CREATE TABLE releases (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    name TEXT NOT NULL UNIQUE,  -- 'v1.0', '2026-Q1'
    description TEXT,
    created_at TIMESTAMPTZ DEFAULT now(),
    created_by TEXT,
    status TEXT DEFAULT 'draft'  -- 'draft', 'released', 'archived'
);

CREATE TABLE release_items (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    release_id UUID REFERENCES releases(id) ON DELETE CASCADE,
    item_id UUID REFERENCES items(id) ON DELETE CASCADE,
    revision_number INTEGER NOT NULL,
    UNIQUE(release_id, item_id)
);

Use Case: Snapshot a set of items at specific revisions for a product release

3.3 Thumbnail Generation

Effort: Medium | Priority: Low | Risk: Low

Implementation Options:

1. FreeCAD headless rendering (python script)
2. External service (e.g., CAD thumbnail microservice)
3. User-uploaded thumbnails

Changes:

• Add thumbnail generation on file upload
• Store in MinIO at thumbnails/{part_number}/rev{n}.png
• Expose via GET /api/items/{pn}/thumbnail/{rev}

---

5. Recommended Implementation Order

Immediate (Next Sprint)

1. Revision Comparison API - High value, low risk
2. Rollback Support - Critical for data safety
3. Revision Labels - Quick win for workflow

Short-term (1-2 Months)

4. Pessimistic Locking - Required before multi-user
5. Authentication - Required before production deployment
6. Audit Logging - Compliance and debugging

Medium-term (3-6 Months)

7. Release Management - Product milestone tracking
8. Thumbnail Generation - Visual preview capability
9. Documentation Overhaul - API reference, deployment guide

Long-term (Future)

10. Branching - Complex, defer until needed
11. Delta Storage - Optimization, not critical
12. Notifications - Nice-to-have workflow enhancement

---

6. Migration Considerations

Part Number Format Migration (Completed)

The recent migration from XXXXX-CCC-NNNN to CCC-NNNN format has been completed:

• Database migration: migrations/006_project_tags.sql
• Schema update: schemas/kindred-rd.yaml v3
• Projects: Now many-to-many tags instead of embedded in part number
• File paths: ~/projects/cad/{category}_{name}/{part_number}_{description}.FCStd

Future Schema Migrations

The property schema versioning framework (property_schema_version, property_migrations table) is in place but lacks:

• Automated migration runners
• Rollback capability for failed migrations
• Dry-run validation mode

Recommendation: Build migration tooling before adding complex property schemas.

---

7. Open Questions (from Specification)

These design decisions remain unresolved:

1. Property change triggers - Should editing properties auto-create revision?
2. Revision metadata editing - Allow comment updates post-creation?
3. Soft delete behavior - Archive or hard delete revisions?
4. File diff strategy - Exploded FCStd storage for better diffing?
5. Retention policy - How long to keep old revisions?

---

Appendix A: File Structure for New Features

internal/
  api/
    handlers_revision.go    # New revision endpoints
    handlers_lock.go        # Locking endpoints
    handlers_audit.go       # Audit log endpoints
  auth/
    ldap.go                 # LDAP authentication
    middleware.go           # Auth middleware
  db/
    locks.go                # Lock repository
    audit.go                # Audit repository
    releases.go             # Release repository
migrations/
  007_revision_status.sql   # Labels and status
  008_item_locks.sql        # Locking table
  009_audit_log.sql         # Audit logging
  010_releases.sql          # Release management

---

Appendix B: API Additions Summary

Phase 1 Endpoints

GET    /api/items/{pn}/revisions/compare    # Diff two revisions
POST   /api/items/{pn}/rollback/{rev}       # Create revision from old
PATCH  /api/items/{pn}/revisions/{rev}      # Update status/labels

Phase 2 Endpoints

POST   /api/items/{pn}/lock                 # Acquire lock
DELETE /api/items/{pn}/lock                 # Release lock
GET    /api/items/{pn}/lock                 # Check lock status
GET    /api/audit                           # Query audit log

Phase 3 Endpoints

GET    /api/releases                        # List releases
POST   /api/releases                        # Create release
GET    /api/releases/{name}                 # Get release details
POST   /api/releases/{name}/items           # Add items to release
GET    /api/items/{pn}/thumbnail/{rev}      # Get thumbnail