silo/docs/GAP_ANALYSIS.md

# Silo Gap Analysis and Revision Control Roadmap

**Date:** 2026-02-08  
**Status:** Analysis Complete (Updated)

---

## Executive Summary

This document analyzes the current state of the Silo project against its specification, identifies documentation and feature gaps, and outlines a roadmap for enhanced revision control capabilities.

---

## 1. Documentation Gap Analysis

### 1.1 Current Documentation

| Document | Coverage | Status |
|----------|----------|--------|
| `README.md` | Quick start, overview, component map | Current |
| `docs/SPECIFICATION.md` | Design specification, API reference | Current |
| `docs/STATUS.md` | Implementation status summary | Current |
| `docs/DEPLOYMENT.md` | Production deployment guide | Current |
| `docs/CONFIGURATION.md` | Configuration reference (all config.yaml options) | Current |
| `docs/AUTH.md` | Authentication system design | Current |
| `docs/AUTH_USER_GUIDE.md` | User guide for login, tokens, and roles | Current |
| `docs/GAP_ANALYSIS.md` | Revision control roadmap | Current |
| `ROADMAP.md` | Feature roadmap and SOLIDWORKS PDM comparison | Current |
| `frontend-spec.md` | React SPA frontend specification | Current |

### 1.2 Documentation Gaps (Priority Order)

#### High Priority

| Gap | Impact | Effort | Status |
|-----|--------|--------|--------|
| **API Reference** | Users cannot integrate programmatically | Medium | Addressed in SPECIFICATION.md Section 11 |
| **Deployment Guide** | Cannot deploy to production | Medium | Complete (`docs/DEPLOYMENT.md`) |
| **Database Schema Guide** | Migration troubleshooting difficult | Low | Open |
| **Configuration Reference** | config.yaml options undocumented | Low | Complete (`docs/CONFIGURATION.md`) |

#### Medium Priority

| Gap | Impact | Effort |
|-----|--------|--------|
| **User Workflows** | Users lack step-by-step guidance | Medium |
| **Troubleshooting Guide** | Support burden increases | Medium |
| **Developer Setup Guide** | Onboarding friction | Low |

#### Lower Priority

| Gap | Impact | Effort |
|-----|--------|--------|
| **CHANGELOG.md** | Version history unclear | Low |
| **Architecture Decision Records** | Design rationale lost | Medium |
| **Integration Guide** | Third-party integration unclear | High |

### 1.3 Recommended Actions

1. ~~**Consolidate specs**: Remove `silo-spec.md` duplicate~~ Done (deleted)
2. ~~**Create API reference**: Full REST endpoint documentation~~ Addressed in SPECIFICATION.md
3. ~~**Create `docs/DEPLOYMENT.md`**: Production deployment guide~~ Done
4. ~~**Create configuration reference**: Document all `config.yaml` options~~ Done (`docs/CONFIGURATION.md`)
5. **Create database schema guide**: Document migrations and troubleshooting

---

## 2. Current Revision Control Implementation

### 2.1 What's Implemented

| Feature | Status | Location |
|---------|--------|----------|
| Append-only revision history | Complete | `internal/db/items.go` |
| Sequential revision numbering | Complete | Database trigger |
| Property snapshots (JSONB) | Complete | `revisions.properties` |
| File versioning (MinIO) | Complete | `internal/storage/` |
| SHA256 checksums | Complete | Captured on upload |
| Revision comments | Complete | `revisions.comment` |
| User attribution | Complete | `revisions.created_by` |
| Property schema versioning | Complete | Migration 005 |
| BOM revision pinning | Complete | `relationships.child_revision` |

### 2.2 Database Schema

```sql
-- Current revision schema (migrations/001_initial.sql)
CREATE TABLE revisions (
    id UUID PRIMARY KEY,
    item_id UUID REFERENCES items(id) ON DELETE CASCADE,
    revision_number INTEGER NOT NULL,
    properties JSONB NOT NULL DEFAULT '{}',
    file_key TEXT,
    file_version TEXT,        -- MinIO version ID
    file_checksum TEXT,       -- SHA256
    file_size BIGINT,
    thumbnail_key TEXT,
    created_at TIMESTAMPTZ DEFAULT now(),
    created_by TEXT,
    comment TEXT,
    property_schema_version INTEGER DEFAULT 1,
    UNIQUE(item_id, revision_number)
);
```

### 2.3 API Endpoints

| Endpoint | Method | Status |
|----------|--------|--------|
| `/api/items/{pn}/revisions` | GET | Implemented |
| `/api/items/{pn}/revisions` | POST | Implemented |
| `/api/items/{pn}/revisions/{rev}` | GET | Implemented |
| `/api/items/{pn}/file` | POST | Implemented |
| `/api/items/{pn}/file` | GET | Implemented (latest) |
| `/api/items/{pn}/file/{rev}` | GET | Implemented |

### 2.4 Client Integration

FreeCAD workbench maintained in separate [silo-mod](https://git.kindred-systems.com/kindred/silo-mod) repository. The server provides the revision and file endpoints consumed by the workbench.

---

## 3. Revision Control Gaps

### 3.1 Critical Gaps

| Gap | Description | Impact | Status |
|-----|-------------|--------|--------|
| ~~**No rollback**~~ | ~~Cannot revert to previous revision~~ | ~~Data recovery difficult~~ | **Implemented** |
| ~~**No comparison**~~ | ~~Cannot diff between revisions~~ | ~~Change tracking manual~~ | **Implemented** |
| **No locking** | No concurrent edit protection | Multi-user unsafe | Open |
| **No approval workflow** | No release/sign-off process | Quality control gap | Open |

### 3.2 Important Gaps

| Gap | Description | Impact | Status |
|-----|-------------|--------|--------|
| **No branching** | Linear history only | No experimental variants | Open |
| ~~**No tagging**~~ | ~~No named milestones~~ | ~~Release tracking manual~~ | **Implemented** (revision labels) |
| ~~**No audit log**~~ | ~~Actions not logged separately~~ | ~~Compliance gap~~ | **Implemented** (migration 009, `audit_log` table + completeness scoring) |
| **Thumbnail missing** | Schema exists, not populated | No visual preview | Open |

### 3.3 Nice-to-Have Gaps

| Gap | Description | Impact |
|-----|-------------|--------|
| **No search** | Cannot search revision comments | Discovery limited |
| **No retention policy** | Revisions never expire | Storage grows unbounded |
| **No delta storage** | Full file per revision | Storage inefficient |
| **No notifications** | No change alerts | Manual monitoring required |

---

## 4. Revision Control Roadmap

### Phase 1: Foundation -- COMPLETE

**Goal:** Enable safe single-user revision management

All Phase 1 items have been implemented:

- **Rollback**: `POST /api/items/{pn}/revisions/{rev}/rollback` - creates new revision from old
- **Revision Comparison**: `GET /api/items/{pn}/revisions/compare?from={rev1}&to={rev2}` - property and file diffs
- **Revision Labels/Status**: `PATCH /api/items/{pn}/revisions/{rev}` - status (draft/review/released/obsolete) and arbitrary labels via migration 007

---

### Phase 2: Multi-User Support

**Goal:** Enable safe concurrent editing

#### 2.1 Pessimistic Locking
```
Effort: High | Priority: High | Risk: Medium | Status: Not Started
```

**Database Migration:**
```sql
CREATE TABLE item_locks (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    item_id UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE,
    locked_by TEXT NOT NULL,
    locked_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    expires_at TIMESTAMPTZ NOT NULL,
    lock_type TEXT NOT NULL DEFAULT 'exclusive',
    comment TEXT,
    UNIQUE(item_id)
);

CREATE INDEX idx_locks_expires ON item_locks(expires_at);
```

**API Endpoints:**
```
POST   /api/items/{pn}/lock      # Acquire lock
DELETE /api/items/{pn}/lock      # Release lock
GET    /api/items/{pn}/lock      # Check lock status
```

#### 2.2 Authentication -- COMPLETE

Authentication is fully implemented with three backends (local/bcrypt, LDAP/FreeIPA, OIDC/Keycloak), role-based access control (admin > editor > viewer), API token management, and PostgreSQL-backed sessions. See `docs/AUTH.md` for full details.

- Migration: `009_auth.sql`
- Code: `internal/auth/`, `internal/api/middleware.go`

#### 2.3 Audit Logging -- COMPLETE

Audit logging is implemented via migration 009 with the `audit_log` table and completeness scoring system. Endpoints:
- `GET /api/audit/completeness` — summary of all items
- `GET /api/audit/completeness/{partNumber}` — per-item scoring with weighted fields and tier classification

Code: `internal/api/audit_handlers.go`

---

### Phase 3: Advanced Features

**Goal:** Enhanced revision management capabilities

#### 3.1 Branching Support
```
Effort: High | Priority: Low | Risk: High
```

**Concept:** Named revision streams per item

**Database Migration:**
```sql
CREATE TABLE revision_branches (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    item_id UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE,
    name TEXT NOT NULL,  -- 'main', 'experimental', 'customer-variant'
    base_revision INTEGER,  -- Branch point
    created_at TIMESTAMPTZ DEFAULT now(),
    created_by TEXT,
    UNIQUE(item_id, name)
);

ALTER TABLE revisions ADD COLUMN branch_id UUID REFERENCES revision_branches(id);
```

**Complexity:** Requires merge logic, conflict resolution UI

#### 3.2 Release Management
```
Effort: Medium | Priority: Medium | Risk: Low
```

**Database Migration:**
```sql
CREATE TABLE releases (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    name TEXT NOT NULL UNIQUE,  -- 'v1.0', '2026-Q1'
    description TEXT,
    created_at TIMESTAMPTZ DEFAULT now(),
    created_by TEXT,
    status TEXT DEFAULT 'draft'  -- 'draft', 'released', 'archived'
);

CREATE TABLE release_items (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    release_id UUID REFERENCES releases(id) ON DELETE CASCADE,
    item_id UUID REFERENCES items(id) ON DELETE CASCADE,
    revision_number INTEGER NOT NULL,
    UNIQUE(release_id, item_id)
);
```

**Use Case:** Snapshot a set of items at specific revisions for a product release

#### 3.3 Thumbnail Generation
```
Effort: Medium | Priority: Low | Risk: Low
```

**Implementation Options:**
1. FreeCAD headless rendering (python script)
2. External service (e.g., CAD thumbnail microservice)
3. User-uploaded thumbnails

**Changes:**
- Add thumbnail generation on file upload
- Store in MinIO at `thumbnails/{part_number}/rev{n}.png`
- Expose via `GET /api/items/{pn}/thumbnail/{rev}`

---

## 5. Recommended Implementation Order

### Completed

1. ~~**Revision Comparison API**~~ - Implemented
2. ~~**Rollback Support**~~ - Implemented
3. ~~**Revision Labels/Status**~~ - Implemented (migration 007)

### Recently Completed

4. ~~**Authentication**~~ - Implemented (3 backends: local, LDAP, OIDC; RBAC; API tokens; sessions)
5. ~~**Audit Logging**~~ - Implemented (audit_log table, completeness scoring)

### Next (Short-term)

6. **Pessimistic Locking** - Required before multi-user

### Medium-term (3-6 Months)

7. **Release Management** - Product milestone tracking
8. **Thumbnail Generation** - Visual preview capability

### Long-term (Future)

10. **Branching** - Complex, defer until needed
11. **Delta Storage** - Optimization, not critical
12. **Notifications** - Nice-to-have workflow enhancement

---

## 6. Migration Considerations

### Part Number Format Migration (Completed)

The recent migration from `XXXXX-CCC-NNNN` to `CCC-NNNN` format has been completed:

- Database migration: `migrations/006_project_tags.sql`
- Schema update: `schemas/kindred-rd.yaml` v3
- Projects: Now many-to-many tags instead of embedded in part number
- File paths: `~/projects/cad/{category}_{name}/{part_number}_{description}.FCStd`

### Future Schema Migrations

The property schema versioning framework (`property_schema_version`, `property_migrations` table) is in place but lacks:

- Automated migration runners
- Rollback capability for failed migrations
- Dry-run validation mode

**Recommendation:** Build migration tooling before adding complex property schemas.

---

## 7. Open Questions (from Specification)

These design decisions remain unresolved:

1. **Property change triggers** - Should editing properties auto-create revision?
2. **Revision metadata editing** - Allow comment updates post-creation?
3. **Soft delete behavior** - Archive or hard delete revisions?
4. **File diff strategy** - Exploded FCStd storage for better diffing?
5. **Retention policy** - How long to keep old revisions?

---

## Appendix A: File Structure

Revision endpoints, status, labels, authentication, audit logging, and file attachments are implemented. Current structure:

```
internal/
  api/
    audit_handlers.go       # Audit/completeness endpoints
    auth_handlers.go        # Login, tokens, OIDC
    bom_handlers.go         # Flat BOM, cost roll-up
    file_handlers.go        # Presigned uploads, item files, thumbnails
    handlers.go             # Items, schemas, projects, revisions
    middleware.go            # Auth middleware
    odoo_handlers.go        # Odoo integration endpoints
    routes.go               # Route registration (78 endpoints)
    search.go               # Fuzzy search
  auth/
    auth.go                 # Auth service: local, LDAP, OIDC
  db/
    items.go                # Item and revision repository
    item_files.go           # File attachment repository
    relationships.go        # BOM repository
    projects.go             # Project repository
  storage/
    storage.go              # MinIO file storage helpers
migrations/
  001_initial.sql           # Core schema
  ...
  011_item_files.sql        # Item file attachments (latest)
```

Future features would add:

```
internal/
  api/
    lock_handlers.go        # Locking endpoints
  db/
    locks.go                # Lock repository
    releases.go             # Release repository
migrations/
  012_item_locks.sql        # Locking table
  013_releases.sql          # Release management
```

---

## Appendix B: API Additions Summary

### Phase 1 Endpoints (Implemented)
```
GET    /api/items/{pn}/revisions/compare            # Diff two revisions
POST   /api/items/{pn}/revisions/{rev}/rollback     # Create revision from old
PATCH  /api/items/{pn}/revisions/{rev}              # Update status/labels
```

### Phase 2 Endpoints

**Authentication (Implemented):**
```
GET    /api/auth/me                          # Current user info
GET    /api/auth/tokens                      # List API tokens
POST   /api/auth/tokens                      # Create API token
DELETE /api/auth/tokens/{id}                 # Revoke API token
```

**Audit (Implemented):**
```
GET    /api/audit/completeness               # All items completeness summary
GET    /api/audit/completeness/{partNumber}  # Per-item scoring
```

**File Attachments (Implemented):**
```
GET    /api/auth/config                      # Auth config (public)
POST   /api/uploads/presign                  # Presigned upload URL
GET    /api/items/{pn}/files                 # List item files
POST   /api/items/{pn}/files                 # Associate file with item
DELETE /api/items/{pn}/files/{fileId}        # Delete file association
PUT    /api/items/{pn}/thumbnail             # Set item thumbnail
GET    /api/items/{pn}/bom/flat              # Flattened BOM
GET    /api/items/{pn}/bom/cost              # Assembly cost roll-up
```

**Locking (Not Implemented):**
```
POST   /api/items/{pn}/lock                 # Acquire lock
DELETE /api/items/{pn}/lock                 # Release lock
GET    /api/items/{pn}/lock                 # Check lock status
```

### Phase 3 Endpoints (Not Implemented)
```
GET    /api/releases                        # List releases
POST   /api/releases                        # Create release
GET    /api/releases/{name}                 # Get release details
POST   /api/releases/{name}/items           # Add items to release
GET    /api/items/{pn}/thumbnail/{rev}      # Get thumbnail
```