Add revision control and project tagging migration
This commit is contained in:
Forbes
487
docs/GAP_ANALYSIS.md
Normal file
487
docs/GAP_ANALYSIS.md
Normal file
@@ -0,0 +1,487 @@
|
||||
# Silo Gap Analysis and Revision Control Roadmap
|
||||
|
||||
**Date:** 2026-01-24
|
||||
**Status:** Analysis Complete
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
This document analyzes the current state of the Silo project against its specification, identifies documentation and feature gaps, and outlines a roadmap for enhanced revision control capabilities.
|
||||
|
||||
---
|
||||
|
||||
## 1. Documentation Gap Analysis
|
||||
|
||||
### 1.1 Current Documentation
|
||||
|
||||
| Document | Coverage | Status |
|
||||
|----------|----------|--------|
|
||||
| `README.md` | Quick start, overview | Partial (50%) |
|
||||
| `docs/SPECIFICATION.md` | Design specification | Comprehensive (90%) |
|
||||
| `docs/STATUS.md` | Development progress | Current but incomplete |
|
||||
| `silo-spec.md` | Duplicate of SPECIFICATION.md | Redundant |
|
||||
|
||||
### 1.2 Documentation Gaps (Priority Order)
|
||||
|
||||
#### High Priority
|
||||
|
||||
| Gap | Impact | Effort |
|
||||
|-----|--------|--------|
|
||||
| **API Reference** | Users cannot integrate programmatically | Medium |
|
||||
| **Deployment Guide** | Cannot deploy to production | Medium |
|
||||
| **Database Schema Guide** | Migration troubleshooting difficult | Low |
|
||||
| **Configuration Reference** | config.yaml options undocumented | Low |
|
||||
|
||||
#### Medium Priority
|
||||
|
||||
| Gap | Impact | Effort |
|
||||
|-----|--------|--------|
|
||||
| **User Workflows** | Users lack step-by-step guidance | Medium |
|
||||
| **FreeCAD Command Reference** | Addon features undiscoverable | Low |
|
||||
| **Troubleshooting Guide** | Support burden increases | Medium |
|
||||
| **Developer Setup Guide** | Onboarding friction | Low |
|
||||
|
||||
#### Lower Priority
|
||||
|
||||
| Gap | Impact | Effort |
|
||||
|-----|--------|--------|
|
||||
| **CHANGELOG.md** | Version history unclear | Low |
|
||||
| **Architecture Decision Records** | Design rationale lost | Medium |
|
||||
| **Integration Guide** | Third-party integration unclear | High |
|
||||
|
||||
### 1.3 Recommended Actions
|
||||
|
||||
1. **Consolidate specs**: Remove `silo-spec.md` duplicate
|
||||
2. **Create `docs/API.md`**: Full REST endpoint documentation with examples
|
||||
3. **Create `docs/DEPLOYMENT.md`**: Production deployment guide
|
||||
4. **Expand README.md**: Add configuration reference section
|
||||
|
||||
---
|
||||
|
||||
## 2. Current Revision Control Implementation
|
||||
|
||||
### 2.1 What's Implemented
|
||||
|
||||
| Feature | Status | Location |
|
||||
|---------|--------|----------|
|
||||
| Append-only revision history | Complete | `internal/db/items.go` |
|
||||
| Sequential revision numbering | Complete | Database trigger |
|
||||
| Property snapshots (JSONB) | Complete | `revisions.properties` |
|
||||
| File versioning (MinIO) | Complete | `internal/storage/` |
|
||||
| SHA256 checksums | Complete | Captured on upload |
|
||||
| Revision comments | Complete | `revisions.comment` |
|
||||
| User attribution | Complete | `revisions.created_by` |
|
||||
| Property schema versioning | Complete | Migration 005 |
|
||||
| BOM revision pinning | Complete | `relationships.child_revision` |
|
||||
|
||||
### 2.2 Database Schema
|
||||
|
||||
```sql
|
||||
-- Current revision schema (migrations/001_initial.sql)
|
||||
CREATE TABLE revisions (
|
||||
id UUID PRIMARY KEY,
|
||||
item_id UUID REFERENCES items(id) ON DELETE CASCADE,
|
||||
revision_number INTEGER NOT NULL,
|
||||
properties JSONB NOT NULL DEFAULT '{}',
|
||||
file_key TEXT,
|
||||
file_version TEXT, -- MinIO version ID
|
||||
file_checksum TEXT, -- SHA256
|
||||
file_size BIGINT,
|
||||
thumbnail_key TEXT,
|
||||
created_at TIMESTAMPTZ DEFAULT now(),
|
||||
created_by TEXT,
|
||||
comment TEXT,
|
||||
property_schema_version INTEGER DEFAULT 1,
|
||||
UNIQUE(item_id, revision_number)
|
||||
);
|
||||
```
|
||||
|
||||
### 2.3 API Endpoints
|
||||
|
||||
| Endpoint | Method | Status |
|
||||
|----------|--------|--------|
|
||||
| `/api/items/{pn}/revisions` | GET | Implemented |
|
||||
| `/api/items/{pn}/revisions` | POST | Implemented |
|
||||
| `/api/items/{pn}/revisions/{rev}` | GET | Implemented |
|
||||
| `/api/items/{pn}/file` | POST | Implemented |
|
||||
| `/api/items/{pn}/file` | GET | Implemented (latest) |
|
||||
| `/api/items/{pn}/file/{rev}` | GET | Implemented |
|
||||
|
||||
### 2.4 FreeCAD Integration
|
||||
|
||||
| Command | Function | Status |
|
||||
|---------|----------|--------|
|
||||
| `Silo_Save` | Auto-save + upload | Implemented |
|
||||
| `Silo_Commit` | Save with comment | Implemented |
|
||||
| `Silo_Pull` | Download/create | Implemented |
|
||||
| `Silo_Push` | Batch upload | Implemented |
|
||||
| `Silo_Info` | View revision history | Implemented |
|
||||
|
||||
---
|
||||
|
||||
## 3. Revision Control Gaps
|
||||
|
||||
### 3.1 Critical Gaps
|
||||
|
||||
| Gap | Description | Impact |
|
||||
|-----|-------------|--------|
|
||||
| **No rollback** | Cannot revert to previous revision | Data recovery difficult |
|
||||
| **No comparison** | Cannot diff between revisions | Change tracking manual |
|
||||
| **No locking** | No concurrent edit protection | Multi-user unsafe |
|
||||
| **No approval workflow** | No release/sign-off process | Quality control gap |
|
||||
|
||||
### 3.2 Important Gaps
|
||||
|
||||
| Gap | Description | Impact |
|
||||
|-----|-------------|--------|
|
||||
| **No branching** | Linear history only | No experimental variants |
|
||||
| **No tagging** | No named milestones | Release tracking manual |
|
||||
| **No audit log** | Actions not logged separately | Compliance gap |
|
||||
| **Thumbnail missing** | Schema exists, not populated | No visual preview |
|
||||
|
||||
### 3.3 Nice-to-Have Gaps
|
||||
|
||||
| Gap | Description | Impact |
|
||||
|-----|-------------|--------|
|
||||
| **No search** | Cannot search revision comments | Discovery limited |
|
||||
| **No retention policy** | Revisions never expire | Storage grows unbounded |
|
||||
| **No delta storage** | Full file per revision | Storage inefficient |
|
||||
| **No notifications** | No change alerts | Manual monitoring required |
|
||||
|
||||
---
|
||||
|
||||
## 4. Revision Control Roadmap
|
||||
|
||||
### Phase 1: Foundation (Recommended First)
|
||||
|
||||
**Goal:** Enable safe single-user revision management
|
||||
|
||||
#### 1.1 Rollback Support
|
||||
```
|
||||
Effort: Medium | Priority: High | Risk: Low
|
||||
```
|
||||
|
||||
**Changes Required:**
|
||||
- Add `POST /api/items/{pn}/rollback/{rev}` endpoint
|
||||
- Create new revision copying properties/file from target revision
|
||||
- FreeCAD: Add `Silo_Rollback` command
|
||||
|
||||
**Database:** No schema changes needed (creates new revision from old)
|
||||
|
||||
#### 1.2 Revision Comparison API
|
||||
```
|
||||
Effort: Medium | Priority: High | Risk: Low
|
||||
```
|
||||
|
||||
**Changes Required:**
|
||||
- Add `GET /api/items/{pn}/revisions/compare?from={rev1}&to={rev2}` endpoint
|
||||
- Return property diff (added/removed/changed keys)
|
||||
- Return file metadata diff (size, checksum changes)
|
||||
|
||||
**Implementation:**
|
||||
```go
|
||||
type RevisionDiff struct {
|
||||
FromRevision int `json:"from_revision"`
|
||||
ToRevision int `json:"to_revision"`
|
||||
Properties PropertyDiff `json:"properties"`
|
||||
FileChanged bool `json:"file_changed"`
|
||||
FileSizeDiff int64 `json:"file_size_diff,omitempty"`
|
||||
}
|
||||
|
||||
type PropertyDiff struct {
|
||||
Added map[string]any `json:"added,omitempty"`
|
||||
Removed map[string]any `json:"removed,omitempty"`
|
||||
Changed map[string]PropertyChange `json:"changed,omitempty"`
|
||||
}
|
||||
```
|
||||
|
||||
#### 1.3 Revision Labels/Status
|
||||
```
|
||||
Effort: Low | Priority: Medium | Risk: Low
|
||||
```
|
||||
|
||||
**Database Migration:**
|
||||
```sql
|
||||
ALTER TABLE revisions ADD COLUMN status TEXT DEFAULT 'draft';
|
||||
-- Values: 'draft', 'review', 'released', 'obsolete'
|
||||
|
||||
ALTER TABLE revisions ADD COLUMN labels TEXT[] DEFAULT '{}';
|
||||
-- Arbitrary tags: ['prototype', 'v1.0', 'customer-approved']
|
||||
```
|
||||
|
||||
**API Changes:**
|
||||
- Add `PATCH /api/items/{pn}/revisions/{rev}` for status/label updates
|
||||
- Add filtering by status in list endpoint
|
||||
|
||||
---
|
||||
|
||||
### Phase 2: Multi-User Support
|
||||
|
||||
**Goal:** Enable safe concurrent editing
|
||||
|
||||
#### 2.1 Pessimistic Locking
|
||||
```
|
||||
Effort: High | Priority: High | Risk: Medium
|
||||
```
|
||||
|
||||
**Database Migration:**
|
||||
```sql
|
||||
CREATE TABLE item_locks (
|
||||
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||||
item_id UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE,
|
||||
locked_by TEXT NOT NULL,
|
||||
locked_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
expires_at TIMESTAMPTZ NOT NULL,
|
||||
lock_type TEXT NOT NULL DEFAULT 'exclusive',
|
||||
comment TEXT,
|
||||
UNIQUE(item_id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_locks_expires ON item_locks(expires_at);
|
||||
```
|
||||
|
||||
**API Endpoints:**
|
||||
```
|
||||
POST /api/items/{pn}/lock # Acquire lock
|
||||
DELETE /api/items/{pn}/lock # Release lock
|
||||
GET /api/items/{pn}/lock # Check lock status
|
||||
```
|
||||
|
||||
**FreeCAD Integration:**
|
||||
- Auto-lock on `Silo_Pull` (configurable)
|
||||
- Auto-unlock on `Silo_Save`/`Silo_Commit`
|
||||
- Show lock status in `Silo_Info`
|
||||
|
||||
#### 2.2 Authentication (LDAP/FreeIPA)
|
||||
```
|
||||
Effort: High | Priority: High | Risk: Medium
|
||||
```
|
||||
|
||||
**Changes Required:**
|
||||
- Add `internal/auth/` package
|
||||
- LDAP bind configuration in config.yaml
|
||||
- Middleware for API authentication
|
||||
- `created_by` populated from authenticated user
|
||||
|
||||
**Configuration:**
|
||||
```yaml
|
||||
auth:
|
||||
enabled: true
|
||||
provider: ldap
|
||||
ldap:
|
||||
server: ldap://freeipa.example.com
|
||||
base_dn: cn=users,cn=accounts,dc=example,dc=com
|
||||
bind_dn: uid=silo-service,cn=users,...
|
||||
bind_password_env: LDAP_BIND_PASSWORD
|
||||
```
|
||||
|
||||
#### 2.3 Audit Logging
|
||||
```
|
||||
Effort: Medium | Priority: Medium | Risk: Low
|
||||
```
|
||||
|
||||
**Database Migration:**
|
||||
```sql
|
||||
CREATE TABLE audit_log (
|
||||
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||||
timestamp TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
user_id TEXT NOT NULL,
|
||||
action TEXT NOT NULL, -- 'create', 'update', 'delete', 'lock', 'unlock'
|
||||
resource_type TEXT NOT NULL, -- 'item', 'revision', 'project', 'relationship'
|
||||
resource_id TEXT NOT NULL,
|
||||
details JSONB,
|
||||
ip_address TEXT
|
||||
);
|
||||
|
||||
CREATE INDEX idx_audit_timestamp ON audit_log(timestamp DESC);
|
||||
CREATE INDEX idx_audit_user ON audit_log(user_id);
|
||||
CREATE INDEX idx_audit_resource ON audit_log(resource_type, resource_id);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Phase 3: Advanced Features
|
||||
|
||||
**Goal:** Enhanced revision management capabilities
|
||||
|
||||
#### 3.1 Branching Support
|
||||
```
|
||||
Effort: High | Priority: Low | Risk: High
|
||||
```
|
||||
|
||||
**Concept:** Named revision streams per item
|
||||
|
||||
**Database Migration:**
|
||||
```sql
|
||||
CREATE TABLE revision_branches (
|
||||
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||||
item_id UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE,
|
||||
name TEXT NOT NULL, -- 'main', 'experimental', 'customer-variant'
|
||||
base_revision INTEGER, -- Branch point
|
||||
created_at TIMESTAMPTZ DEFAULT now(),
|
||||
created_by TEXT,
|
||||
UNIQUE(item_id, name)
|
||||
);
|
||||
|
||||
ALTER TABLE revisions ADD COLUMN branch_id UUID REFERENCES revision_branches(id);
|
||||
```
|
||||
|
||||
**Complexity:** Requires merge logic, conflict resolution UI
|
||||
|
||||
#### 3.2 Release Management
|
||||
```
|
||||
Effort: Medium | Priority: Medium | Risk: Low
|
||||
```
|
||||
|
||||
**Database Migration:**
|
||||
```sql
|
||||
CREATE TABLE releases (
|
||||
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||||
name TEXT NOT NULL UNIQUE, -- 'v1.0', '2026-Q1'
|
||||
description TEXT,
|
||||
created_at TIMESTAMPTZ DEFAULT now(),
|
||||
created_by TEXT,
|
||||
status TEXT DEFAULT 'draft' -- 'draft', 'released', 'archived'
|
||||
);
|
||||
|
||||
CREATE TABLE release_items (
|
||||
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
|
||||
release_id UUID REFERENCES releases(id) ON DELETE CASCADE,
|
||||
item_id UUID REFERENCES items(id) ON DELETE CASCADE,
|
||||
revision_number INTEGER NOT NULL,
|
||||
UNIQUE(release_id, item_id)
|
||||
);
|
||||
```
|
||||
|
||||
**Use Case:** Snapshot a set of items at specific revisions for a product release
|
||||
|
||||
#### 3.3 Thumbnail Generation
|
||||
```
|
||||
Effort: Medium | Priority: Low | Risk: Low
|
||||
```
|
||||
|
||||
**Implementation Options:**
|
||||
1. FreeCAD headless rendering (python script)
|
||||
2. External service (e.g., CAD thumbnail microservice)
|
||||
3. User-uploaded thumbnails
|
||||
|
||||
**Changes:**
|
||||
- Add thumbnail generation on file upload
|
||||
- Store in MinIO at `thumbnails/{part_number}/rev{n}.png`
|
||||
- Expose via `GET /api/items/{pn}/thumbnail/{rev}`
|
||||
|
||||
---
|
||||
|
||||
## 5. Recommended Implementation Order
|
||||
|
||||
### Immediate (Next Sprint)
|
||||
|
||||
1. **Revision Comparison API** - High value, low risk
|
||||
2. **Rollback Support** - Critical for data safety
|
||||
3. **Revision Labels** - Quick win for workflow
|
||||
|
||||
### Short-term (1-2 Months)
|
||||
|
||||
4. **Pessimistic Locking** - Required before multi-user
|
||||
5. **Authentication** - Required before production deployment
|
||||
6. **Audit Logging** - Compliance and debugging
|
||||
|
||||
### Medium-term (3-6 Months)
|
||||
|
||||
7. **Release Management** - Product milestone tracking
|
||||
8. **Thumbnail Generation** - Visual preview capability
|
||||
9. **Documentation Overhaul** - API reference, deployment guide
|
||||
|
||||
### Long-term (Future)
|
||||
|
||||
10. **Branching** - Complex, defer until needed
|
||||
11. **Delta Storage** - Optimization, not critical
|
||||
12. **Notifications** - Nice-to-have workflow enhancement
|
||||
|
||||
---
|
||||
|
||||
## 6. Migration Considerations
|
||||
|
||||
### Part Number Format Migration (Completed)
|
||||
|
||||
The recent migration from `XXXXX-CCC-NNNN` to `CCC-NNNN` format has been completed:
|
||||
|
||||
- Database migration: `migrations/006_project_tags.sql`
|
||||
- Schema update: `schemas/kindred-rd.yaml` v3
|
||||
- Projects: Now many-to-many tags instead of embedded in part number
|
||||
- File paths: `~/projects/cad/{category}_{name}/{part_number}_{description}.FCStd`
|
||||
|
||||
### Future Schema Migrations
|
||||
|
||||
The property schema versioning framework (`property_schema_version`, `property_migrations` table) is in place but lacks:
|
||||
|
||||
- Automated migration runners
|
||||
- Rollback capability for failed migrations
|
||||
- Dry-run validation mode
|
||||
|
||||
**Recommendation:** Build migration tooling before adding complex property schemas.
|
||||
|
||||
---
|
||||
|
||||
## 7. Open Questions (from Specification)
|
||||
|
||||
These design decisions remain unresolved:
|
||||
|
||||
1. **Property change triggers** - Should editing properties auto-create revision?
|
||||
2. **Revision metadata editing** - Allow comment updates post-creation?
|
||||
3. **Soft delete behavior** - Archive or hard delete revisions?
|
||||
4. **File diff strategy** - Exploded FCStd storage for better diffing?
|
||||
5. **Retention policy** - How long to keep old revisions?
|
||||
|
||||
---
|
||||
|
||||
## Appendix A: File Structure for New Features
|
||||
|
||||
```
|
||||
internal/
|
||||
api/
|
||||
handlers_revision.go # New revision endpoints
|
||||
handlers_lock.go # Locking endpoints
|
||||
handlers_audit.go # Audit log endpoints
|
||||
auth/
|
||||
ldap.go # LDAP authentication
|
||||
middleware.go # Auth middleware
|
||||
db/
|
||||
locks.go # Lock repository
|
||||
audit.go # Audit repository
|
||||
releases.go # Release repository
|
||||
migrations/
|
||||
007_revision_status.sql # Labels and status
|
||||
008_item_locks.sql # Locking table
|
||||
009_audit_log.sql # Audit logging
|
||||
010_releases.sql # Release management
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Appendix B: API Additions Summary
|
||||
|
||||
### Phase 1 Endpoints
|
||||
```
|
||||
GET /api/items/{pn}/revisions/compare # Diff two revisions
|
||||
POST /api/items/{pn}/rollback/{rev} # Create revision from old
|
||||
PATCH /api/items/{pn}/revisions/{rev} # Update status/labels
|
||||
```
|
||||
|
||||
### Phase 2 Endpoints
|
||||
```
|
||||
POST /api/items/{pn}/lock # Acquire lock
|
||||
DELETE /api/items/{pn}/lock # Release lock
|
||||
GET /api/items/{pn}/lock # Check lock status
|
||||
GET /api/audit # Query audit log
|
||||
```
|
||||
|
||||
### Phase 3 Endpoints
|
||||
```
|
||||
GET /api/releases # List releases
|
||||
POST /api/releases # Create release
|
||||
GET /api/releases/{name} # Get release details
|
||||
POST /api/releases/{name}/items # Add items to release
|
||||
GET /api/items/{pn}/thumbnail/{rev} # Get thumbnail
|
||||
```
|
||||
Reference in New Issue
Block a user