silo/docs/GAP_ANALYSIS.md

# Silo Gap Analysis and Revision Control Roadmap

**Date:** 2026-01-24  
**Status:** Analysis Complete

---

## Executive Summary

This document analyzes the current state of the Silo project against its specification, identifies documentation and feature gaps, and outlines a roadmap for enhanced revision control capabilities.

---

## 1. Documentation Gap Analysis

### 1.1 Current Documentation

| Document | Coverage | Status |
|----------|----------|--------|
| `README.md` | Quick start, overview, component map | Current |
| `docs/SPECIFICATION.md` | Design specification, API reference | Current |
| `docs/STATUS.md` | Implementation status summary | Current |
| `docs/DEPLOYMENT.md` | Production deployment guide | Current |
| `docs/GAP_ANALYSIS.md` | SOLIDWORKS PDM comparison, roadmap | Current |
| `ROADMAP.md` | Feature roadmap and phases | Current |
| `silo-spec.md` | Redirect to `docs/SPECIFICATION.md` | Consolidated |

### 1.2 Documentation Gaps (Priority Order)

#### High Priority

| Gap | Impact | Effort | Status |
|-----|--------|--------|--------|
| **API Reference** | Users cannot integrate programmatically | Medium | Addressed in SPECIFICATION.md Section 11 |
| **Deployment Guide** | Cannot deploy to production | Medium | Complete (`docs/DEPLOYMENT.md`) |
| **Database Schema Guide** | Migration troubleshooting difficult | Low | Open |
| **Configuration Reference** | config.yaml options undocumented | Low | Open |

#### Medium Priority

| Gap | Impact | Effort |
|-----|--------|--------|
| **User Workflows** | Users lack step-by-step guidance | Medium |
| **FreeCAD Command Reference** | Addon features undiscoverable | Low |
| **Troubleshooting Guide** | Support burden increases | Medium |
| **Developer Setup Guide** | Onboarding friction | Low |

#### Lower Priority

| Gap | Impact | Effort |
|-----|--------|--------|
| **CHANGELOG.md** | Version history unclear | Low |
| **Architecture Decision Records** | Design rationale lost | Medium |
| **Integration Guide** | Third-party integration unclear | High |

### 1.3 Recommended Actions

1. ~~**Consolidate specs**: Remove `silo-spec.md` duplicate~~ Done
2. ~~**Create API reference**: Full REST endpoint documentation~~ Addressed in SPECIFICATION.md
3. ~~**Create `docs/DEPLOYMENT.md`**: Production deployment guide~~ Done
4. **Create configuration reference**: Document all `config.yaml` options
5. **Create database schema guide**: Document migrations and troubleshooting

---

## 2. Current Revision Control Implementation

### 2.1 What's Implemented

| Feature | Status | Location |
|---------|--------|----------|
| Append-only revision history | Complete | `internal/db/items.go` |
| Sequential revision numbering | Complete | Database trigger |
| Property snapshots (JSONB) | Complete | `revisions.properties` |
| File versioning (MinIO) | Complete | `internal/storage/` |
| SHA256 checksums | Complete | Captured on upload |
| Revision comments | Complete | `revisions.comment` |
| User attribution | Complete | `revisions.created_by` |
| Property schema versioning | Complete | Migration 005 |
| BOM revision pinning | Complete | `relationships.child_revision` |

### 2.2 Database Schema

```sql
-- Current revision schema (migrations/001_initial.sql)
CREATE TABLE revisions (
    id UUID PRIMARY KEY,
    item_id UUID REFERENCES items(id) ON DELETE CASCADE,
    revision_number INTEGER NOT NULL,
    properties JSONB NOT NULL DEFAULT '{}',
    file_key TEXT,
    file_version TEXT,        -- MinIO version ID
    file_checksum TEXT,       -- SHA256
    file_size BIGINT,
    thumbnail_key TEXT,
    created_at TIMESTAMPTZ DEFAULT now(),
    created_by TEXT,
    comment TEXT,
    property_schema_version INTEGER DEFAULT 1,
    UNIQUE(item_id, revision_number)
);
```

### 2.3 API Endpoints

| Endpoint | Method | Status |
|----------|--------|--------|
| `/api/items/{pn}/revisions` | GET | Implemented |
| `/api/items/{pn}/revisions` | POST | Implemented |
| `/api/items/{pn}/revisions/{rev}` | GET | Implemented |
| `/api/items/{pn}/file` | POST | Implemented |
| `/api/items/{pn}/file` | GET | Implemented (latest) |
| `/api/items/{pn}/file/{rev}` | GET | Implemented |

### 2.4 FreeCAD Integration

| Command | Function | Status |
|---------|----------|--------|
| `Silo_Save` | Auto-save + upload | Implemented |
| `Silo_Commit` | Save with comment | Implemented |
| `Silo_Pull` | Download/create | Implemented |
| `Silo_Push` | Batch upload | Implemented |
| `Silo_Info` | View revision history | Implemented |

---

## 3. Revision Control Gaps

### 3.1 Critical Gaps

| Gap | Description | Impact | Status |
|-----|-------------|--------|--------|
| ~~**No rollback**~~ | ~~Cannot revert to previous revision~~ | ~~Data recovery difficult~~ | **Implemented** |
| ~~**No comparison**~~ | ~~Cannot diff between revisions~~ | ~~Change tracking manual~~ | **Implemented** |
| **No locking** | No concurrent edit protection | Multi-user unsafe | Open |
| **No approval workflow** | No release/sign-off process | Quality control gap | Open |

### 3.2 Important Gaps

| Gap | Description | Impact | Status |
|-----|-------------|--------|--------|
| **No branching** | Linear history only | No experimental variants | Open |
| ~~**No tagging**~~ | ~~No named milestones~~ | ~~Release tracking manual~~ | **Implemented** (revision labels) |
| **No audit log** | Actions not logged separately | Compliance gap | Open |
| **Thumbnail missing** | Schema exists, not populated | No visual preview | Open |

### 3.3 Nice-to-Have Gaps

| Gap | Description | Impact |
|-----|-------------|--------|
| **No search** | Cannot search revision comments | Discovery limited |
| **No retention policy** | Revisions never expire | Storage grows unbounded |
| **No delta storage** | Full file per revision | Storage inefficient |
| **No notifications** | No change alerts | Manual monitoring required |

---

## 4. Revision Control Roadmap

### Phase 1: Foundation -- COMPLETE

**Goal:** Enable safe single-user revision management

All Phase 1 items have been implemented:

- **Rollback**: `POST /api/items/{pn}/revisions/{rev}/rollback` - creates new revision from old
- **Revision Comparison**: `GET /api/items/{pn}/revisions/compare?from={rev1}&to={rev2}` - property and file diffs
- **Revision Labels/Status**: `PATCH /api/items/{pn}/revisions/{rev}` - status (draft/review/released/obsolete) and arbitrary labels via migration 007

---

### Phase 2: Multi-User Support

**Goal:** Enable safe concurrent editing

#### 2.1 Pessimistic Locking
```
Effort: High | Priority: High | Risk: Medium
```

**Database Migration:**
```sql
CREATE TABLE item_locks (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    item_id UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE,
    locked_by TEXT NOT NULL,
    locked_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    expires_at TIMESTAMPTZ NOT NULL,
    lock_type TEXT NOT NULL DEFAULT 'exclusive',
    comment TEXT,
    UNIQUE(item_id)
);

CREATE INDEX idx_locks_expires ON item_locks(expires_at);
```

**API Endpoints:**
```
POST   /api/items/{pn}/lock      # Acquire lock
DELETE /api/items/{pn}/lock      # Release lock
GET    /api/items/{pn}/lock      # Check lock status
```

**FreeCAD Integration:**
- Auto-lock on `Silo_Pull` (configurable)
- Auto-unlock on `Silo_Save`/`Silo_Commit`
- Show lock status in `Silo_Info`

#### 2.2 Authentication (LDAP/FreeIPA)
```
Effort: High | Priority: High | Risk: Medium
```

**Changes Required:**
- Add `internal/auth/` package
- LDAP bind configuration in config.yaml
- Middleware for API authentication
- `created_by` populated from authenticated user

**Configuration:**
```yaml
auth:
  enabled: true
  provider: ldap
  ldap:
    server: ldap://freeipa.example.com
    base_dn: cn=users,cn=accounts,dc=example,dc=com
    bind_dn: uid=silo-service,cn=users,...
    bind_password_env: LDAP_BIND_PASSWORD
```

#### 2.3 Audit Logging
```
Effort: Medium | Priority: Medium | Risk: Low
```

**Database Migration:**
```sql
CREATE TABLE audit_log (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    timestamp TIMESTAMPTZ NOT NULL DEFAULT now(),
    user_id TEXT NOT NULL,
    action TEXT NOT NULL,  -- 'create', 'update', 'delete', 'lock', 'unlock'
    resource_type TEXT NOT NULL,  -- 'item', 'revision', 'project', 'relationship'
    resource_id TEXT NOT NULL,
    details JSONB,
    ip_address TEXT
);

CREATE INDEX idx_audit_timestamp ON audit_log(timestamp DESC);
CREATE INDEX idx_audit_user ON audit_log(user_id);
CREATE INDEX idx_audit_resource ON audit_log(resource_type, resource_id);
```

---

### Phase 3: Advanced Features

**Goal:** Enhanced revision management capabilities

#### 3.1 Branching Support
```
Effort: High | Priority: Low | Risk: High
```

**Concept:** Named revision streams per item

**Database Migration:**
```sql
CREATE TABLE revision_branches (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    item_id UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE,
    name TEXT NOT NULL,  -- 'main', 'experimental', 'customer-variant'
    base_revision INTEGER,  -- Branch point
    created_at TIMESTAMPTZ DEFAULT now(),
    created_by TEXT,
    UNIQUE(item_id, name)
);

ALTER TABLE revisions ADD COLUMN branch_id UUID REFERENCES revision_branches(id);
```

**Complexity:** Requires merge logic, conflict resolution UI

#### 3.2 Release Management
```
Effort: Medium | Priority: Medium | Risk: Low
```

**Database Migration:**
```sql
CREATE TABLE releases (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    name TEXT NOT NULL UNIQUE,  -- 'v1.0', '2026-Q1'
    description TEXT,
    created_at TIMESTAMPTZ DEFAULT now(),
    created_by TEXT,
    status TEXT DEFAULT 'draft'  -- 'draft', 'released', 'archived'
);

CREATE TABLE release_items (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    release_id UUID REFERENCES releases(id) ON DELETE CASCADE,
    item_id UUID REFERENCES items(id) ON DELETE CASCADE,
    revision_number INTEGER NOT NULL,
    UNIQUE(release_id, item_id)
);
```

**Use Case:** Snapshot a set of items at specific revisions for a product release

#### 3.3 Thumbnail Generation
```
Effort: Medium | Priority: Low | Risk: Low
```

**Implementation Options:**
1. FreeCAD headless rendering (python script)
2. External service (e.g., CAD thumbnail microservice)
3. User-uploaded thumbnails

**Changes:**
- Add thumbnail generation on file upload
- Store in MinIO at `thumbnails/{part_number}/rev{n}.png`
- Expose via `GET /api/items/{pn}/thumbnail/{rev}`

---

## 5. Recommended Implementation Order

### Completed

1. ~~**Revision Comparison API**~~ - Implemented
2. ~~**Rollback Support**~~ - Implemented
3. ~~**Revision Labels/Status**~~ - Implemented (migration 007)

### Next (Short-term)

4. **Pessimistic Locking** - Required before multi-user
5. **Authentication** - Required before production deployment
6. **Audit Logging** - Compliance and debugging

### Medium-term (3-6 Months)

7. **Release Management** - Product milestone tracking
8. **Thumbnail Generation** - Visual preview capability
9. **Documentation Overhaul** - API reference, deployment guide

### Long-term (Future)

10. **Branching** - Complex, defer until needed
11. **Delta Storage** - Optimization, not critical
12. **Notifications** - Nice-to-have workflow enhancement

---

## 6. Migration Considerations

### Part Number Format Migration (Completed)

The recent migration from `XXXXX-CCC-NNNN` to `CCC-NNNN` format has been completed:

- Database migration: `migrations/006_project_tags.sql`
- Schema update: `schemas/kindred-rd.yaml` v3
- Projects: Now many-to-many tags instead of embedded in part number
- File paths: `~/projects/cad/{category}_{name}/{part_number}_{description}.FCStd`

### Future Schema Migrations

The property schema versioning framework (`property_schema_version`, `property_migrations` table) is in place but lacks:

- Automated migration runners
- Rollback capability for failed migrations
- Dry-run validation mode

**Recommendation:** Build migration tooling before adding complex property schemas.

---

## 7. Open Questions (from Specification)

These design decisions remain unresolved:

1. **Property change triggers** - Should editing properties auto-create revision?
2. **Revision metadata editing** - Allow comment updates post-creation?
3. **Soft delete behavior** - Archive or hard delete revisions?
4. **File diff strategy** - Exploded FCStd storage for better diffing?
5. **Retention policy** - How long to keep old revisions?

---

## Appendix A: File Structure for New Features

Revision endpoints, status, and labels are already implemented in the existing handler files. Future features would add:

```
internal/
  api/
    handlers_lock.go        # Locking endpoints
    handlers_audit.go       # Audit log endpoints
  auth/
    ldap.go                 # LDAP authentication
    middleware.go           # Auth middleware
  db/
    locks.go                # Lock repository
    audit.go                # Audit repository
    releases.go             # Release repository
migrations/
  008_item_locks.sql        # Locking table
  009_audit_log.sql         # Audit logging
  010_releases.sql          # Release management
```

---

## Appendix B: API Additions Summary

### Phase 1 Endpoints (Implemented)
```
GET    /api/items/{pn}/revisions/compare            # Diff two revisions
POST   /api/items/{pn}/revisions/{rev}/rollback     # Create revision from old
PATCH  /api/items/{pn}/revisions/{rev}              # Update status/labels
```

### Phase 2 Endpoints
```
POST   /api/items/{pn}/lock                 # Acquire lock
DELETE /api/items/{pn}/lock                 # Release lock
GET    /api/items/{pn}/lock                 # Check lock status
GET    /api/audit                           # Query audit log
```

### Phase 3 Endpoints
```
GET    /api/releases                        # List releases
POST   /api/releases                        # Create release
GET    /api/releases/{name}                 # Get release details
POST   /api/releases/{name}/items           # Add items to release
GET    /api/items/{pn}/thumbnail/{rev}      # Get thumbnail
```