silo/docs/GAP_ANALYSIS.md

Silo Gap Analysis and Revision Control Roadmap

Date: 2026-02-08
Status: Analysis Complete (Updated)

---

Executive Summary

This document analyzes the current state of the Silo project against its specification, identifies documentation and feature gaps, and outlines a roadmap for enhanced revision control capabilities.

---

1. Documentation Gap Analysis

1.1 Current Documentation

Document	Coverage	Status
README.md	Quick start, overview, component map	Current
docs/SPECIFICATION.md	Design specification, API reference	Current
docs/STATUS.md	Implementation status summary	Current
docs/DEPLOYMENT.md	Production deployment guide	Current
docs/CONFIGURATION.md	Configuration reference (all config.yaml options)	Current
docs/AUTH.md	Authentication system design	Current
docs/AUTH_USER_GUIDE.md	User guide for login, tokens, and roles	Current
docs/GAP_ANALYSIS.md	Revision control roadmap	Current
ROADMAP.md	Feature roadmap and SOLIDWORKS PDM comparison	Current
frontend-spec.md	React SPA frontend specification	Current

1.2 Documentation Gaps (Priority Order)

High Priority

Gap	Impact	Effort	Status
API Reference	Users cannot integrate programmatically	Medium	Addressed in SPECIFICATION.md Section 11
Deployment Guide	Cannot deploy to production	Medium	Complete (docs/DEPLOYMENT.md)
Database Schema Guide	Migration troubleshooting difficult	Low	Open
Configuration Reference	config.yaml options undocumented	Low	Complete (docs/CONFIGURATION.md)

Medium Priority

Gap	Impact	Effort
User Workflows	Users lack step-by-step guidance	Medium
Troubleshooting Guide	Support burden increases	Medium
Developer Setup Guide	Onboarding friction	Low

Lower Priority

Gap	Impact	Effort
CHANGELOG.md	Version history unclear	Low
Architecture Decision Records	Design rationale lost	Medium
Integration Guide	Third-party integration unclear	High

1.3 Recommended Actions

1. Consolidate specs: Remove silo-spec.md duplicate Done (deleted)
2. Create API reference: Full REST endpoint documentation Addressed in SPECIFICATION.md
3. Create docs/DEPLOYMENT.md: Production deployment guide Done
4. Create configuration reference: Document all config.yaml options Done (docs/CONFIGURATION.md)
5. Create database schema guide: Document migrations and troubleshooting

---

2. Current Revision Control Implementation

2.1 What's Implemented

Feature	Status	Location
Append-only revision history	Complete	internal/db/items.go
Sequential revision numbering	Complete	Database trigger
Property snapshots (JSONB)	Complete	revisions.properties
File versioning (MinIO)	Complete	internal/storage/
SHA256 checksums	Complete	Captured on upload
Revision comments	Complete	revisions.comment
User attribution	Complete	revisions.created_by
Property schema versioning	Complete	Migration 005
BOM revision pinning	Complete	relationships.child_revision

2.2 Database Schema

-- Current revision schema (migrations/001_initial.sql)
CREATE TABLE revisions (
    id UUID PRIMARY KEY,
    item_id UUID REFERENCES items(id) ON DELETE CASCADE,
    revision_number INTEGER NOT NULL,
    properties JSONB NOT NULL DEFAULT '{}',
    file_key TEXT,
    file_version TEXT,        -- MinIO version ID
    file_checksum TEXT,       -- SHA256
    file_size BIGINT,
    thumbnail_key TEXT,
    created_at TIMESTAMPTZ DEFAULT now(),
    created_by TEXT,
    comment TEXT,
    property_schema_version INTEGER DEFAULT 1,
    UNIQUE(item_id, revision_number)
);

2.3 API Endpoints

Endpoint	Method	Status
/api/items/{pn}/revisions	GET	Implemented
/api/items/{pn}/revisions	POST	Implemented
/api/items/{pn}/revisions/{rev}	GET	Implemented
/api/items/{pn}/file	POST	Implemented
/api/items/{pn}/file	GET	Implemented (latest)
/api/items/{pn}/file/{rev}	GET	Implemented

2.4 Client Integration

FreeCAD workbench maintained in separate silo-mod repository. The server provides the revision and file endpoints consumed by the workbench.

---

3. Revision Control Gaps

3.1 Critical Gaps

Gap	Description	Impact	Status
No rollback	Cannot revert to previous revision	Data recovery difficult	Implemented
No comparison	Cannot diff between revisions	Change tracking manual	Implemented
No locking	No concurrent edit protection	Multi-user unsafe	Open
No approval workflow	No release/sign-off process	Quality control gap	Open

3.2 Important Gaps

Gap	Description	Impact	Status
No branching	Linear history only	No experimental variants	Open
No tagging	No named milestones	Release tracking manual	Implemented (revision labels)
No audit log	Actions not logged separately	Compliance gap	Implemented (migration 009, audit_log table + completeness scoring)
Thumbnail missing	Schema exists, not populated	No visual preview	Open

3.3 Nice-to-Have Gaps

Gap	Description	Impact
No search	Cannot search revision comments	Discovery limited
No retention policy	Revisions never expire	Storage grows unbounded
No delta storage	Full file per revision	Storage inefficient
No notifications	No change alerts	Manual monitoring required

---

4. Revision Control Roadmap

Phase 1: Foundation -- COMPLETE

Goal: Enable safe single-user revision management

All Phase 1 items have been implemented:

• Rollback: POST /api/items/{pn}/revisions/{rev}/rollback - creates new revision from old
• Revision Comparison: GET /api/items/{pn}/revisions/compare?from={rev1}&to={rev2} - property and file diffs
• Revision Labels/Status: PATCH /api/items/{pn}/revisions/{rev} - status (draft/review/released/obsolete) and arbitrary labels via migration 007

---

Phase 2: Multi-User Support

Goal: Enable safe concurrent editing

2.1 Pessimistic Locking

Effort: High | Priority: High | Risk: Medium | Status: Not Started

Database Migration:

CREATE TABLE item_locks (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    item_id UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE,
    locked_by TEXT NOT NULL,
    locked_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    expires_at TIMESTAMPTZ NOT NULL,
    lock_type TEXT NOT NULL DEFAULT 'exclusive',
    comment TEXT,
    UNIQUE(item_id)
);

CREATE INDEX idx_locks_expires ON item_locks(expires_at);

API Endpoints:

POST   /api/items/{pn}/lock      # Acquire lock
DELETE /api/items/{pn}/lock      # Release lock
GET    /api/items/{pn}/lock      # Check lock status

2.2 Authentication -- COMPLETE

Authentication is fully implemented with three backends (local/bcrypt, LDAP/FreeIPA, OIDC/Keycloak), role-based access control (admin > editor > viewer), API token management, and PostgreSQL-backed sessions. See docs/AUTH.md for full details.

• Migration: 009_auth.sql
• Code: internal/auth/, internal/api/middleware.go

2.3 Audit Logging -- COMPLETE

Audit logging is implemented via migration 009 with the audit_log table and completeness scoring system. Endpoints:

• GET /api/audit/completeness — summary of all items
• GET /api/audit/completeness/{partNumber} — per-item scoring with weighted fields and tier classification

Code: internal/api/audit_handlers.go

---

Phase 3: Advanced Features

Goal: Enhanced revision management capabilities

3.1 Branching Support

Effort: High | Priority: Low | Risk: High

Concept: Named revision streams per item

Database Migration:

CREATE TABLE revision_branches (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    item_id UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE,
    name TEXT NOT NULL,  -- 'main', 'experimental', 'customer-variant'
    base_revision INTEGER,  -- Branch point
    created_at TIMESTAMPTZ DEFAULT now(),
    created_by TEXT,
    UNIQUE(item_id, name)
);

ALTER TABLE revisions ADD COLUMN branch_id UUID REFERENCES revision_branches(id);

Complexity: Requires merge logic, conflict resolution UI

3.2 Release Management

Effort: Medium | Priority: Medium | Risk: Low

Database Migration:

CREATE TABLE releases (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    name TEXT NOT NULL UNIQUE,  -- 'v1.0', '2026-Q1'
    description TEXT,
    created_at TIMESTAMPTZ DEFAULT now(),
    created_by TEXT,
    status TEXT DEFAULT 'draft'  -- 'draft', 'released', 'archived'
);

CREATE TABLE release_items (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    release_id UUID REFERENCES releases(id) ON DELETE CASCADE,
    item_id UUID REFERENCES items(id) ON DELETE CASCADE,
    revision_number INTEGER NOT NULL,
    UNIQUE(release_id, item_id)
);

Use Case: Snapshot a set of items at specific revisions for a product release

3.3 Thumbnail Generation

Effort: Medium | Priority: Low | Risk: Low

Implementation Options:

1. FreeCAD headless rendering (python script)
2. External service (e.g., CAD thumbnail microservice)
3. User-uploaded thumbnails

Changes:

• Add thumbnail generation on file upload
• Store in MinIO at thumbnails/{part_number}/rev{n}.png
• Expose via GET /api/items/{pn}/thumbnail/{rev}

---

5. Recommended Implementation Order

Completed

1. Revision Comparison API - Implemented
2. Rollback Support - Implemented
3. Revision Labels/Status - Implemented (migration 007)

Recently Completed

4. Authentication - Implemented (3 backends: local, LDAP, OIDC; RBAC; API tokens; sessions)
5. Audit Logging - Implemented (audit_log table, completeness scoring)

Next (Short-term)

6. Pessimistic Locking - Required before multi-user

Medium-term (3-6 Months)

7. Release Management - Product milestone tracking
8. Thumbnail Generation - Visual preview capability

Long-term (Future)

10. Branching - Complex, defer until needed
11. Delta Storage - Optimization, not critical
12. Notifications - Nice-to-have workflow enhancement

---

6. Migration Considerations

Part Number Format Migration (Completed)

The recent migration from XXXXX-CCC-NNNN to CCC-NNNN format has been completed:

• Database migration: migrations/006_project_tags.sql
• Schema update: schemas/kindred-rd.yaml v3
• Projects: Now many-to-many tags instead of embedded in part number
• File paths: ~/projects/cad/{category}_{name}/{part_number}_{description}.FCStd

Future Schema Migrations

The property schema versioning framework (property_schema_version, property_migrations table) is in place but lacks:

• Automated migration runners
• Rollback capability for failed migrations
• Dry-run validation mode

Recommendation: Build migration tooling before adding complex property schemas.

---

7. Open Questions (from Specification)

These design decisions remain unresolved:

1. Property change triggers - Should editing properties auto-create revision?
2. Revision metadata editing - Allow comment updates post-creation?
3. Soft delete behavior - Archive or hard delete revisions?
4. File diff strategy - Exploded FCStd storage for better diffing?
5. Retention policy - How long to keep old revisions?

---

Appendix A: File Structure

Revision endpoints, status, labels, authentication, audit logging, and file attachments are implemented. Current structure:

internal/
  api/
    audit_handlers.go       # Audit/completeness endpoints
    auth_handlers.go        # Login, tokens, OIDC
    bom_handlers.go         # Flat BOM, cost roll-up
    file_handlers.go        # Presigned uploads, item files, thumbnails
    handlers.go             # Items, schemas, projects, revisions
    middleware.go            # Auth middleware
    odoo_handlers.go        # Odoo integration endpoints
    routes.go               # Route registration (75 endpoints)
    search.go               # Fuzzy search
  auth/
    auth.go                 # Auth service: local, LDAP, OIDC
  db/
    items.go                # Item and revision repository
    item_files.go           # File attachment repository
    relationships.go        # BOM repository
    projects.go             # Project repository
  storage/
    storage.go              # MinIO file storage helpers
migrations/
  001_initial.sql           # Core schema
  ...
  011_item_files.sql        # Item file attachments (latest)

Future features would add:

internal/
  api/
    lock_handlers.go        # Locking endpoints
  db/
    locks.go                # Lock repository
    releases.go             # Release repository
migrations/
  012_item_locks.sql        # Locking table
  013_releases.sql          # Release management

---

Appendix B: API Additions Summary

Phase 1 Endpoints (Implemented)

GET    /api/items/{pn}/revisions/compare            # Diff two revisions
POST   /api/items/{pn}/revisions/{rev}/rollback     # Create revision from old
PATCH  /api/items/{pn}/revisions/{rev}              # Update status/labels

Phase 2 Endpoints

Authentication (Implemented):

GET    /api/auth/me                          # Current user info
GET    /api/auth/tokens                      # List API tokens
POST   /api/auth/tokens                      # Create API token
DELETE /api/auth/tokens/{id}                 # Revoke API token

Audit (Implemented):

GET    /api/audit/completeness               # All items completeness summary
GET    /api/audit/completeness/{partNumber}  # Per-item scoring

File Attachments (Implemented):

GET    /api/auth/config                      # Auth config (public)
POST   /api/uploads/presign                  # Presigned upload URL
GET    /api/items/{pn}/files                 # List item files
POST   /api/items/{pn}/files                 # Associate file with item
DELETE /api/items/{pn}/files/{fileId}        # Delete file association
PUT    /api/items/{pn}/thumbnail             # Set item thumbnail
GET    /api/items/{pn}/bom/flat              # Flattened BOM
GET    /api/items/{pn}/bom/cost              # Assembly cost roll-up

Locking (Not Implemented):

POST   /api/items/{pn}/lock                 # Acquire lock
DELETE /api/items/{pn}/lock                 # Release lock
GET    /api/items/{pn}/lock                 # Check lock status

Phase 3 Endpoints (Not Implemented)

GET    /api/releases                        # List releases
POST   /api/releases                        # Create release
GET    /api/releases/{name}                 # Get release details
POST   /api/releases/{name}/items           # Add items to release
GET    /api/items/{pn}/thumbnail/{rev}      # Get thumbnail