silo/docs/GAP_ANALYSIS.md

Silo Gap Analysis

Date: 2026-03-01
Status: Analysis Complete (Updated)

---

Executive Summary

This document analyzes the current state of the Silo project against its specification and against SOLIDWORKS PDM (the industry-leading product data management solution). It identifies documentation gaps, feature gaps, and outlines a roadmap for enhanced revision control capabilities.

See ROADMAP.md for the platform roadmap and dependency tier structure.

---

1. Documentation Gap Analysis

1.1 Current Documentation

Document	Coverage	Status
README.md	Quick start, overview, component map	Current
docs/SPECIFICATION.md	Design specification, API reference	Current
docs/STATUS.md	Implementation status summary	Current
docs/DEPLOYMENT.md	Production deployment guide	Current
docs/CONFIGURATION.md	Configuration reference (all config.yaml options)	Current
docs/AUTH.md	Authentication system design	Current
docs/AUTH_USER_GUIDE.md	User guide for login, tokens, and roles	Current
docs/GAP_ANALYSIS.md	Revision control roadmap	Current
docs/ROADMAP.md	Platform roadmap and dependency tiers	Current
frontend-spec.md	React SPA frontend specification	Current

1.2 Documentation Gaps (Priority Order)

High Priority

Gap	Impact	Effort	Status
API Reference	Users cannot integrate programmatically	Medium	Addressed in SPECIFICATION.md Section 11
Deployment Guide	Cannot deploy to production	Medium	Complete (docs/DEPLOYMENT.md)
Database Schema Guide	Migration troubleshooting difficult	Low	Open
Configuration Reference	config.yaml options undocumented	Low	Complete (docs/CONFIGURATION.md)

Medium Priority

Gap	Impact	Effort
User Workflows	Users lack step-by-step guidance	Medium
Troubleshooting Guide	Support burden increases	Medium
Developer Setup Guide	Onboarding friction	Low

Lower Priority

Gap	Impact	Effort
CHANGELOG.md	Version history unclear	Low
Architecture Decision Records	Design rationale lost	Medium
Integration Guide	Third-party integration unclear	High

1.3 Recommended Actions

1. Consolidate specs: Remove silo-spec.md duplicate Done (deleted)
2. Create API reference: Full REST endpoint documentation Addressed in SPECIFICATION.md
3. Create docs/DEPLOYMENT.md: Production deployment guide Done
4. Create configuration reference: Document all config.yaml options Done (docs/CONFIGURATION.md)
5. Create database schema guide: Document migrations and troubleshooting

---

2. Current Revision Control Implementation

2.1 What's Implemented

Feature	Status	Location
Append-only revision history	Complete	internal/db/items.go
Sequential revision numbering	Complete	Database trigger
Property snapshots (JSONB)	Complete	revisions.properties
File storage (filesystem)	Complete	internal/storage/
SHA256 checksums	Complete	Captured on upload
Revision comments	Complete	revisions.comment
User attribution	Complete	revisions.created_by
Property schema versioning	Complete	Migration 005
BOM revision pinning	Complete	relationships.child_revision

2.2 Database Schema

-- Current revision schema (migrations/001_initial.sql)
CREATE TABLE revisions (
    id UUID PRIMARY KEY,
    item_id UUID REFERENCES items(id) ON DELETE CASCADE,
    revision_number INTEGER NOT NULL,
    properties JSONB NOT NULL DEFAULT '{}',
    file_key TEXT,
    file_version TEXT,        -- storage version ID
    file_checksum TEXT,       -- SHA256
    file_size BIGINT,
    thumbnail_key TEXT,
    created_at TIMESTAMPTZ DEFAULT now(),
    created_by TEXT,
    comment TEXT,
    property_schema_version INTEGER DEFAULT 1,
    UNIQUE(item_id, revision_number)
);

2.3 API Endpoints

Endpoint	Method	Status
/api/items/{pn}/revisions	GET	Implemented
/api/items/{pn}/revisions	POST	Implemented
/api/items/{pn}/revisions/{rev}	GET	Implemented
/api/items/{pn}/file	POST	Implemented
/api/items/{pn}/file	GET	Implemented (latest)
/api/items/{pn}/file/{rev}	GET	Implemented

2.4 Client Integration

FreeCAD workbench maintained in separate silo-mod repository. The server provides the revision and file endpoints consumed by the workbench.

---

3. Revision Control Gaps

3.1 Critical Gaps

Gap	Description	Impact	Status
No rollback	Cannot revert to previous revision	Data recovery difficult	Implemented
No comparison	Cannot diff between revisions	Change tracking manual	Implemented
No locking	No concurrent edit protection	Multi-user unsafe	Partial (edit sessions with hard interference detection; full pessimistic locking not yet implemented)
No approval workflow	No release/sign-off process	Quality control gap	Implemented (YAML-configurable ECO workflows, multi-stage review gates, digital signatures)

3.2 Important Gaps

Gap	Description	Impact	Status
No branching	Linear history only	No experimental variants	Open
No tagging	No named milestones	Release tracking manual	Implemented (revision labels)
No audit log	Actions not logged separately	Compliance gap	Implemented (migration 009, audit_log table + completeness scoring)
Thumbnail missing	Schema exists, not populated	No visual preview	Open

3.3 Nice-to-Have Gaps

Gap	Description	Impact
No search	Cannot search revision comments	Discovery limited
No retention policy	Revisions never expire	Storage grows unbounded
No delta storage	Full file per revision	Storage inefficient
No notifications	No change alerts	Manual monitoring required

---

4. Revision Control Roadmap

Phase 1: Foundation -- COMPLETE

Goal: Enable safe single-user revision management

All Phase 1 items have been implemented:

• Rollback: POST /api/items/{pn}/revisions/{rev}/rollback - creates new revision from old
• Revision Comparison: GET /api/items/{pn}/revisions/compare?from={rev1}&to={rev2} - property and file diffs
• Revision Labels/Status: PATCH /api/items/{pn}/revisions/{rev} - status (draft/review/released/obsolete) and arbitrary labels via migration 007

---

Phase 2: Multi-User Support

Goal: Enable safe concurrent editing

2.1 Pessimistic Locking

Effort: High | Priority: High | Risk: Medium | Status: Not Started

Database Migration:

CREATE TABLE item_locks (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    item_id UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE,
    locked_by TEXT NOT NULL,
    locked_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    expires_at TIMESTAMPTZ NOT NULL,
    lock_type TEXT NOT NULL DEFAULT 'exclusive',
    comment TEXT,
    UNIQUE(item_id)
);

CREATE INDEX idx_locks_expires ON item_locks(expires_at);

API Endpoints:

POST   /api/items/{pn}/lock      # Acquire lock
DELETE /api/items/{pn}/lock      # Release lock
GET    /api/items/{pn}/lock      # Check lock status

2.2 Authentication -- COMPLETE

Authentication is fully implemented with three backends (local/bcrypt, LDAP/FreeIPA, OIDC/Keycloak), role-based access control (admin > editor > viewer), API token management, and PostgreSQL-backed sessions. See docs/AUTH.md for full details.

• Migration: 009_auth.sql
• Code: internal/auth/, internal/api/middleware.go

2.3 Audit Logging -- COMPLETE

Audit logging is implemented via migration 009 with the audit_log table and completeness scoring system. Endpoints:

• GET /api/audit/completeness — summary of all items
• GET /api/audit/completeness/{partNumber} — per-item scoring with weighted fields and tier classification

Code: internal/api/audit_handlers.go

---

Phase 3: Advanced Features

Goal: Enhanced revision management capabilities

3.1 Branching Support

Effort: High | Priority: Low | Risk: High

Concept: Named revision streams per item

Database Migration:

CREATE TABLE revision_branches (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    item_id UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE,
    name TEXT NOT NULL,  -- 'main', 'experimental', 'customer-variant'
    base_revision INTEGER,  -- Branch point
    created_at TIMESTAMPTZ DEFAULT now(),
    created_by TEXT,
    UNIQUE(item_id, name)
);

ALTER TABLE revisions ADD COLUMN branch_id UUID REFERENCES revision_branches(id);

Complexity: Requires merge logic, conflict resolution UI

3.2 Release Management

Effort: Medium | Priority: Medium | Risk: Low

Database Migration:

CREATE TABLE releases (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    name TEXT NOT NULL UNIQUE,  -- 'v1.0', '2026-Q1'
    description TEXT,
    created_at TIMESTAMPTZ DEFAULT now(),
    created_by TEXT,
    status TEXT DEFAULT 'draft'  -- 'draft', 'released', 'archived'
);

CREATE TABLE release_items (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    release_id UUID REFERENCES releases(id) ON DELETE CASCADE,
    item_id UUID REFERENCES items(id) ON DELETE CASCADE,
    revision_number INTEGER NOT NULL,
    UNIQUE(release_id, item_id)
);

Use Case: Snapshot a set of items at specific revisions for a product release

3.3 Thumbnail Generation

Effort: Medium | Priority: Low | Risk: Low

Implementation Options:

1. FreeCAD headless rendering (python script)
2. External service (e.g., CAD thumbnail microservice)
3. User-uploaded thumbnails

Changes:

• Add thumbnail generation on file upload
• Store at thumbnails/{part_number}/rev{n}.png
• Expose via GET /api/items/{pn}/thumbnail/{rev}

---

5. Recommended Implementation Order

Completed

1. Revision Comparison API - Implemented
2. Rollback Support - Implemented
3. Revision Labels/Status - Implemented (migration 007)

Recently Completed

4. Authentication - Implemented (3 backends: local, LDAP, OIDC; RBAC; API tokens; sessions)
5. Audit Logging - Implemented (audit_log table, completeness scoring)

Next (Short-term)

6. Pessimistic Locking - Required before multi-user

Medium-term (3-6 Months)

7. Release Management - Product milestone tracking
8. Thumbnail Generation - Visual preview capability

Long-term (Future)

10. Branching - Complex, defer until needed
11. Delta Storage - Optimization, not critical
12. Notifications - Nice-to-have workflow enhancement

---

6. Migration Considerations

Part Number Format Migration (Completed)

The recent migration from XXXXX-CCC-NNNN to CCC-NNNN format has been completed:

• Database migration: migrations/006_project_tags.sql
• Schema update: schemas/kindred-rd.yaml v3
• Projects: Now many-to-many tags instead of embedded in part number
• File paths: ~/projects/cad/{category}_{name}/{part_number}_{description}.FCStd

Future Schema Migrations

The property schema versioning framework (property_schema_version, property_migrations table) is in place but lacks:

• Automated migration runners
• Rollback capability for failed migrations
• Dry-run validation mode

Recommendation: Build migration tooling before adding complex property schemas.

---

7. Open Questions (from Specification)

These design decisions remain unresolved:

1. Property change triggers - Should editing properties auto-create revision?
2. Revision metadata editing - Allow comment updates post-creation?
3. Soft delete behavior - Archive or hard delete revisions?
4. File diff strategy - Exploded FCStd storage for better diffing?
5. Retention policy - How long to keep old revisions?

---

Appendix A: File Structure

Current structure:

internal/
  api/
    approval_handlers.go    # Approval/ECO workflow endpoints
    audit_handlers.go       # Audit/completeness endpoints
    auth_handlers.go        # Login, tokens, OIDC
    bom_handlers.go         # Flat BOM, cost roll-up
    broker.go               # SSE broker with targeted delivery
    dag_handlers.go         # Dependency DAG endpoints
    dependency_handlers.go  # .kc dependency resolution
    file_handlers.go        # Presigned uploads, item files, thumbnails
    handlers.go             # Items, schemas, projects, revisions, Server struct
    job_handlers.go         # Job queue endpoints
    location_handlers.go    # Location hierarchy endpoints
    macro_handlers.go       # .kc macro endpoints
    metadata_handlers.go    # .kc metadata endpoints
    middleware.go            # Auth middleware
    odoo_handlers.go        # Odoo integration endpoints
    pack_handlers.go        # .kc checkout packing
    routes.go               # Route registration (~140 endpoints)
    runner_handlers.go      # Job runner endpoints
    search.go               # Fuzzy search
    session_handlers.go     # Edit session acquire/release/query
    settings_handlers.go    # Admin settings endpoints
    solver_handlers.go      # Solver service endpoints
    sse_handler.go          # SSE event stream handler
    workstation_handlers.go # Workstation registration
  auth/
    auth.go                 # Auth service: local, LDAP, OIDC
  db/
    edit_sessions.go        # Edit session repository
    items.go                # Item and revision repository
    item_files.go           # File attachment repository
    jobs.go                 # Job queue repository
    projects.go             # Project repository
    relationships.go        # BOM repository
    workstations.go         # Workstation repository
  modules/
    modules.go              # Module registry (12 modules)
    loader.go               # Config-to-module state loader
  storage/
    storage.go              # File storage helpers
migrations/
  001_initial.sql           # Core schema
  ...
  023_edit_sessions.sql     # Edit session tracking (latest)

---

Appendix B: API Additions Summary

Phase 1 Endpoints (Implemented)

GET    /api/items/{pn}/revisions/compare            # Diff two revisions
POST   /api/items/{pn}/revisions/{rev}/rollback     # Create revision from old
PATCH  /api/items/{pn}/revisions/{rev}              # Update status/labels

Phase 2 Endpoints

Authentication (Implemented):

GET    /api/auth/me                          # Current user info
GET    /api/auth/tokens                      # List API tokens
POST   /api/auth/tokens                      # Create API token
DELETE /api/auth/tokens/{id}                 # Revoke API token

Audit (Implemented):

GET    /api/audit/completeness               # All items completeness summary
GET    /api/audit/completeness/{partNumber}  # Per-item scoring

File Attachments (Implemented):

GET    /api/auth/config                      # Auth config (public)
POST   /api/uploads/presign                  # Presigned upload URL
GET    /api/items/{pn}/files                 # List item files
POST   /api/items/{pn}/files                 # Associate file with item
DELETE /api/items/{pn}/files/{fileId}        # Delete file association
PUT    /api/items/{pn}/thumbnail             # Set item thumbnail
GET    /api/items/{pn}/bom/flat              # Flattened BOM
GET    /api/items/{pn}/bom/cost              # Assembly cost roll-up

Locking (Not Implemented):

POST   /api/items/{pn}/lock                 # Acquire lock
DELETE /api/items/{pn}/lock                 # Release lock
GET    /api/items/{pn}/lock                 # Check lock status

Phase 3 Endpoints (Not Implemented)

GET    /api/releases                        # List releases
POST   /api/releases                        # Create release
GET    /api/releases/{name}                 # Get release details
POST   /api/releases/{name}/items           # Add items to release
GET    /api/items/{pn}/thumbnail/{rev}      # Get thumbnail

---

Appendix C: SOLIDWORKS PDM Comparison

This section compares Silo's capabilities against SOLIDWORKS PDM features. Gaps are categorized by priority and implementation complexity.

Legend: Silo Status = Full / Partial / None | Priority = Critical / High / Medium / Low | Complexity = Simple / Moderate / Complex

C.1 Version Control & Revision Management

Feature	SOLIDWORKS PDM	Silo Status	Priority	Complexity
Check-in/check-out	Full pessimistic locking	Partial (edit sessions with hard interference)	High	Moderate
Version history	Complete with branching	Full (linear)	-	-
Revision labels	A, B, C or custom schemes	Full (custom labels)	-	-
Rollback/restore	Full	Full	-	-
Compare revisions	Visual + metadata diff	Metadata diff only	Medium	Complex
Get Latest Revision	One-click retrieval	Partial (API only)	Medium	Simple

Silo has edit sessions with hard interference detection (unique index on item + context_level + object_id prevents two users from editing the same object simultaneously). Full pessimistic file-level locking is not yet implemented. Visual diff comparison would require FreeCAD integration for CAD file visualization.

C.2 Workflow Management

Feature	SOLIDWORKS PDM	Silo Status	Priority	Complexity
Custom workflows	Full visual designer	Full (YAML-defined state machines)	-	-
State transitions	Configurable with permissions	Full (configurable transition rules)	-	-
Parallel approvals	Multiple approvers required	Full (multi-stage review gates)	-	-
Automatic transitions	Timer/condition-based	None	Medium	Moderate
Email notifications	On state change	None	High	Moderate
ECO process	Built-in change management	Full (YAML-configurable ECO workflows)	-	-
Child state conditions	Block parent if children invalid	None	Medium	Moderate

Workflow management has been significantly addressed. Silo now supports YAML-defined state machine workflows with configurable transitions, multi-stage approval gates, and digital signatures. Remaining gaps: automatic timer-based transitions, email notifications, and child state condition enforcement.

C.3 User Management & Security

Feature	SOLIDWORKS PDM	Silo Status	Priority	Complexity
User authentication	Windows AD, LDAP	Full (local, LDAP, OIDC)	-	-
Role-based permissions	Granular per folder/state	Partial (3-tier role model)	Medium	Moderate
Group management	Full	None	Medium	Moderate
Folder permissions	Read/write/delete per folder	None	Medium	Moderate
State permissions	Actions allowed per state	None	High	Moderate
Audit trail	Complete action logging	Full	-	-
Private files	Pre-check-in visibility control	None	Low	Simple

Authentication is implemented with three backends (local, LDAP/FreeIPA, OIDC/Keycloak) and a 3-tier role model (admin > editor > viewer). Audit logging captures user actions. Remaining gaps: group management, folder-level permissions, and state-based permission rules.

C.4 Search & Discovery

Feature	SOLIDWORKS PDM	Silo Status	Priority	Complexity
Metadata search	Full with custom cards	Partial (API query params + fuzzy)	High	Moderate
Full-text content search	iFilters for Office, CAD	None	Medium	Complex
Quick search	Toolbar with history	Partial (fuzzy search API)	Medium	Simple
Saved searches	User-defined favorites	None	Medium	Simple
Advanced operators	AND, OR, NOT, wildcards	None	Medium	Simple
Multi-variable search	Search across multiple fields	None	Medium	Simple
Where-used search	Find all assemblies using part	Full	-	-

Silo has API-level filtering, fuzzy search, and where-used queries. Remaining gaps: saved searches, advanced search operators, and a richer search UI. Content search (searching within CAD files) is not planned for the server.

C.5 BOM Management

Feature	SOLIDWORKS PDM	Silo Status	Priority	Complexity
Single-level BOM	Yes	Full	-	-
Multi-level BOM	Indented/exploded views	Full (recursive, configurable depth)	-	-
BOM comparison	Between revisions	None	Medium	Moderate
BOM export	Excel, XML, ERP formats	Full (CSV, ODS)	-	-
BOM import	Bulk BOM loading	Full (CSV with upsert)	-	-
Calculated BOMs	Quantities rolled up	None	Medium	Moderate
Reference designators	Full support	Full	-	-
Alternate parts	Substitute tracking	Full	-	-

Multi-level BOM retrieval (recursive CTE with configurable depth) and BOM export (CSV, ODS) are implemented. BOM import supports CSV with upsert and cycle detection. Remaining gap: BOM comparison between revisions.

C.6 CAD Integration

Feature	SOLIDWORKS PDM	Silo Status	Priority	Complexity
Native CAD add-in	Deep SOLIDWORKS integration	FreeCAD workbench (silo-mod)	Medium	Complex
Property mapping	Bi-directional sync	Planned (silo-mod)	Medium	Moderate
Task pane	Embedded in CAD UI	Auth dock panel (silo-mod)	Medium	Complex
Lightweight components	Handle without full load	N/A	-	-
Drawing/model linking	Automatic association	Manual	Medium	Moderate
Multi-CAD support	Third-party formats	FreeCAD only	Low	-

CAD integration is maintained in separate repositories (silo-mod, silo-calc). The Silo server provides the REST API endpoints consumed by those clients.

C.7 External Integrations

Feature	SOLIDWORKS PDM	Silo Status	Priority	Complexity
ERP integration	SAP, Dynamics, etc.	Partial (Odoo stubs)	Medium	Complex
API access	Full COM/REST API	Full REST API (~140 endpoints)	-	-
Dispatch scripts	Automation without coding	None	Medium	Moderate
Task scheduler	Background processing	Full (job queue with runners)	-	-
Email system	SMTP integration	None	High	Simple
Web portal	Browser access	Full (React SPA + auth)	-	-

Silo has a comprehensive REST API (~140 endpoints) and a full web UI with authentication. Odoo ERP integration has config/sync-log scaffolding but push/pull operations are stubs. Job queue with runner management is fully implemented. Remaining gaps: email notifications, dispatch automation.

C.8 Reporting & Analytics

Feature	SOLIDWORKS PDM	Silo Status	Priority	Complexity
Standard reports	Inventory, usage, activity	None	Medium	Moderate
Custom reports	User-defined queries	None	Medium	Moderate
Dashboard	Visual KPIs	None	Low	Moderate
Export formats	PDF, Excel, CSV	CSV and ODS	Medium	Simple

Reporting capabilities are absent. Basic reports (item counts, revision activity, where-used) would provide immediate value.

C.9 File Handling

Feature	SOLIDWORKS PDM	Silo Status	Priority	Complexity
File versioning	Automatic	Full (filesystem)	-	-
File preview	Thumbnails, 3D preview	None	Medium	Complex
File conversion	PDF, DXF generation	None	Medium	Complex
Replication	Multi-site sync	None	Low	Complex
File copy with refs	Copy tree with references	None	Medium	Moderate

File storage works well. Thumbnail generation and file preview would significantly improve the web UI experience. Automatic conversion to PDF/DXF is valuable for sharing with non-CAD users.

---

Appendix D: Feature Comparison Matrix

Category	Feature	SW PDM Standard	SW PDM Pro	Silo Current	Silo Planned
Version Control	Check-in/out	Yes	Yes	Partial (edit sessions)	Tier 1
	Version history	Yes	Yes	Yes	-
	Rollback	Yes	Yes	Yes	-
	Revision labels/status	Yes	Yes	Yes	-
	Revision comparison	Yes	Yes	Yes (metadata)	-
Workflow	Custom workflows	Limited	Yes	Yes (YAML state machines)	-
	Parallel approval	No	Yes	Yes (multi-stage gates)	-
	Notifications	No	Yes	No	Tier 1
Security	User auth	Windows	Windows/LDAP	Yes (local, LDAP, OIDC)	-
	Permissions	Basic	Granular	Partial (role-based)	Tier 4
	Audit trail	Basic	Full	Yes	-
Search	Metadata search	Yes	Yes	Partial (API + fuzzy)	Tier 0
	Content search	No	Yes	No	Tier 2
	Where-used	Yes	Yes	Yes	-
BOM	Single-level	Yes	Yes	Yes	-
	Multi-level	Yes	Yes	Yes (recursive)	-
	BOM export	Yes	Yes	Yes (CSV, ODS)	-
Data	CSV import/export	Yes	Yes	Yes	-
	ODS import/export	No	No	Yes	-
	Project management	Yes	Yes	Yes	-
Integration	API	Limited	Full	Full REST (~140)	-
	ERP connectors	No	Yes	Partial (Odoo stubs)	Tier 6
	Web access	No	Yes	Yes (React SPA + auth)	-
Files	Versioning	Yes	Yes	Yes	-
	Preview	Yes	Yes	No	Tier 2
	Multi-site	No	Yes	No	Not Planned