kindred/silo
1
1
Fork 0
Code Issues 30 Pull Requests Actions Packages Projects Releases Wiki Activity

update databasing system with minimum API, schema parsing and FreeCAD

Browse Source 
integration
This commit is contained in:
Forbes
2026-01-24 15:03:17 -06:00
parent eafdc30f32
commit c327baf36f
51 changed files with 11635 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

833
docs/SPECIFICATION.md Normal file
View File

@@ -0,0 +1,833 @@
# Silo: Item Database and Part Management System for FreeCAD
**Version:** 0.1 Draft
**Date:** January 2026
**Author:** Kindred Systems LLC
---
## 1. Overview
Silo is an item database with configurable part number generation, designed for R&D-oriented workflows. It integrates with FreeCAD 1.0+ to provide git-like object management, revision tracking, and physical inventory location management.
### 1.1 Core Philosophy
Silo treats **part numbering schemas as configuration, not code**. Multiple numbering schemes can coexist, each defined in YAML. The system is schema-agnostic—it doesn't impose a particular part numbering philosophy (intelligent vs. non-intelligent numbers) but instead provides the machinery to implement whatever scheme the organization requires.
### 1.2 Key Principles
- **Items are the atomic unit**: Everything is an item (parts, assemblies, drawings, documents)
- **Schemas are mutable**: Part numbering schemas can evolve, though migration tooling is out of scope for MVP
- **Append-only history**: All parameter changes are recorded; item state is reconstructable at any point in time
- **Configuration over convention**: Hierarchies, relationships, and behaviors are YAML-defined
---
## 2. Architecture
### 2.1 Components
```
┌─────────────────────────────────────────────────────────────┐
│ FreeCAD 1.0+ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Silo Workbench (Python) │ │
│ │ - silo checkout / commit / status / log │ │
│ │ - Part number generation │ │
│ │ - Property sync with FreeCAD objects │ │
│ └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Silo Core (CLI/Library) │
│ - Schema parsing and validation │
│ - Part number generation engine │
│ - Revision management │
│ - Relationship graph │
└─────────────────────────────────────────────────────────────┘
│
┌───────────────┴───────────────┐
▼ ▼
┌─────────────────────────┐ ┌─────────────────────────────┐
│ PostgreSQL │ │ MinIO │
│ (psql.kindred.internal)│ │ - .FCStd file storage │
│ - Item metadata │ │ - Versioned objects │
│ - Relationships │ │ - Thumbnails │
│ - Revision history │ │ │
│ - Location hierarchy │ │ │
└─────────────────────────┘ └─────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Web UI (Browse/Search) │
│ - Item browser with hierarchy navigation │
│ - Search and filtering │
│ - "Open in FreeCAD" links (freecad:// URI handler) │
│ - BOM viewer │
└─────────────────────────────────────────────────────────────┘
```
### 2.2 Technology Stack
| Component | Technology | Notes |
|-----------|------------|-------|
| Database | PostgreSQL | Existing instance at psql.kindred.internal |
| File Storage | MinIO | S3-compatible, versioning enabled |
| FreeCAD Integration | Python workbench | Macro-style commands |
| CLI | Go or Python | TBD based on complexity |
| Web UI | Go + htmx | Lightweight, minimal JS |
---
## 3. Data Model
### 3.1 Items
An **item** is the fundamental entity. Items have:
- A **part number** (generated according to a schema)
- A **type** (part, assembly, drawing, document, etc.)
- **Properties** (key-value pairs, schema-defined and custom)
- **Relationships** to other items
- **Revisions** (append-only history)
- **Files** (optional, stored in MinIO)
- **Location** (optional physical inventory location)
### 3.2 Database Schema (Conceptual)
```sql
-- Part numbering schemas (YAML stored as text, parsed at runtime)
CREATE TABLE schemas (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT UNIQUE NOT NULL,
version INTEGER NOT NULL DEFAULT 1,
definition JSONB NOT NULL, -- parsed YAML
created_at TIMESTAMPTZ DEFAULT now(),
updated_at TIMESTAMPTZ DEFAULT now()
);
-- Items (core entity)
CREATE TABLE items (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
part_number TEXT UNIQUE NOT NULL,
schema_id UUID REFERENCES schemas(id),
item_type TEXT NOT NULL, -- 'part', 'assembly', 'drawing', etc.
created_at TIMESTAMPTZ DEFAULT now(),
current_revision_id UUID -- points to latest revision
);
-- Append-only revision history
CREATE TABLE revisions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
item_id UUID REFERENCES items(id) NOT NULL,
revision_number INTEGER NOT NULL,
properties JSONB NOT NULL, -- all properties at this revision
file_version TEXT, -- MinIO version ID if applicable
created_at TIMESTAMPTZ DEFAULT now(),
created_by TEXT, -- user identifier (future: LDAP DN)
comment TEXT,
UNIQUE(item_id, revision_number)
);
-- Item relationships (BOM structure)
CREATE TABLE relationships (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
parent_item_id UUID REFERENCES items(id) NOT NULL,
child_item_id UUID REFERENCES items(id) NOT NULL,
relationship_type TEXT NOT NULL, -- 'component', 'alternate', 'reference'
quantity DECIMAL,
reference_designator TEXT, -- e.g., "R1", "C3" for electronics
metadata JSONB, -- assembly-specific relationship config
revision_id UUID REFERENCES revisions(id), -- which revision this applies to
created_at TIMESTAMPTZ DEFAULT now()
);
-- Location hierarchy (configurable via YAML)
CREATE TABLE locations (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
path TEXT UNIQUE NOT NULL, -- e.g., "lab/shelf-a/bin-3"
name TEXT NOT NULL,
parent_id UUID REFERENCES locations(id),
location_type TEXT NOT NULL, -- defined in location schema
metadata JSONB,
created_at TIMESTAMPTZ DEFAULT now()
);
-- Item inventory (quantity at location)
CREATE TABLE inventory (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
item_id UUID REFERENCES items(id) NOT NULL,
location_id UUID REFERENCES locations(id) NOT NULL,
quantity DECIMAL NOT NULL DEFAULT 0,
updated_at TIMESTAMPTZ DEFAULT now(),
UNIQUE(item_id, location_id)
);
-- Sequence counters for part number generation
CREATE TABLE sequences (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
schema_id UUID REFERENCES schemas(id),
scope TEXT NOT NULL, -- scope key (e.g., project code, type code)
current_value INTEGER NOT NULL DEFAULT 0,
UNIQUE(schema_id, scope)
);
```
---
## 4. YAML Configuration System
### 4.1 Part Numbering Schema
Schemas define how part numbers are generated. Each schema consists of **segments** that are concatenated with a **separator**.
```yaml
# /etc/silo/schemas/kindred-rd.yaml
schema:
name: kindred-rd
version: 1
description: "Kindred Systems R&D part numbering"
# Separator between segments (default: "-")
separator: "-"
# Uniqueness enforcement
uniqueness:
scope: global # or "per-project", "per-type", "per-schema"
segments:
- name: project
type: string
length: 5
description: "Project identifier"
validation:
pattern: "^[A-Z0-9]{5}$"
required: true
- name: part_type
type: enum
description: "Type of item"
values:
AS: "Assembly"
PT: "Part"
DW: "Drawing"
DC: "Document"
TB: "Tooling/Fixture"
PC: "Purchased Component"
required: true
- name: sequence
type: serial
length: 4
padding: "0" # left-pad with zeros
description: "Sequential number"
scope: "{project}-{part_type}" # counter scope (template)
# Format template (optional, defaults to joining segments with separator)
format: "{project}-{part_type}-{sequence}"
# Example outputs:
# PROTO-AS-0001 (first assembly in PROTO project)
# PROTO-PT-0001 (first part in PROTO project)
# ALPHA-AS-0001 (first assembly in ALPHA project)
```
### 4.2 Segment Types
| Type | Description | Options |
|------|-------------|---------|
| `string` | Fixed or variable length string | `length`, `min_length`, `max_length`, `pattern`, `case` |
| `enum` | Predefined set of values | `values` (map of code → description) |
| `serial` | Auto-incrementing integer | `length`, `padding`, `start`, `scope` |
| `date` | Date-based segment | `format` (strftime-style) |
| `constant` | Fixed value | `value` |
### 4.3 Serial Scope Templates
The `scope` field in serial segments supports template variables referencing other segments:
```yaml
# Sequence per project
scope: "{project}"
# Sequence per project AND type (recommended for R&D)
scope: "{project}-{part_type}"
# Global sequence (no scope)
scope: null
```
### 4.4 Alternative Schema Example (Simple Sequential)
```yaml
# /etc/silo/schemas/simple.yaml
schema:
name: simple
version: 1
description: "Simple non-intelligent numbering"
segments:
- name: prefix
type: constant
value: "P"
- name: sequence
type: serial
length: 6
padding: "0"
scope: null # global counter
format: "{prefix}{sequence}"
separator: ""
# Output: P000001, P000002, ...
```
### 4.5 Location Hierarchy Schema
```yaml
# /etc/silo/schemas/locations.yaml
location_schema:
name: kindred-lab
version: 1
hierarchy:
- level: 0
type: facility
name_pattern: "^[a-z-]+$"
- level: 1
type: area
name_pattern: "^[a-z-]+$"
- level: 2
type: shelf
name_pattern: "^shelf-[a-z]$"
- level: 3
type: bin
name_pattern: "^bin-[0-9]+$"
# Path format
path_separator: "/"
# Example paths:
# lab/main-area/shelf-a/bin-1
# lab/storage/shelf-b/bin-12
```
### 4.6 Assembly Metadata Schema
Each assembly can define its own relationship tracking behavior:
```yaml
# Stored in item properties or as a linked document
assembly_config:
# What relationship types this assembly uses
relationship_types:
- component # standard BOM entry
- alternate # interchangeable substitute
- reference # related but not part of BOM
# Whether to track reference designators
use_reference_designators: true
designator_format: "^[A-Z]+[0-9]+$" # e.g., R1, C3, U12
# Revision linking behavior
child_revision_tracking: specific # or "latest"
# Custom properties for relationships
relationship_properties:
- name: mounting_orientation
type: enum
values: [top, bottom, left, right, front, back]
- name: notes
type: text
```
---
## 5. FreeCAD Integration
### 5.1 Workbench Commands
The Silo workbench provides git-like commands accessible via toolbar, menu, and Python console:
| Command | Description |
|---------|-------------|
| `silo init` | Initialize Silo tracking for current document |
| `silo status` | Show tracked/untracked objects, modifications |
| `silo checkout <part_number>` | Load item from Silo into current document |
| `silo commit` | Save current state as new revision |
| `silo log` | Show revision history |
| `silo diff` | Compare current state to last committed revision |
| `silo register` | Generate part number for selected object(s) |
| `silo link` | Create relationship between objects |
| `silo bom` | Generate BOM from current assembly |
### 5.2 Property Synchronization
Silo properties map to FreeCAD custom properties:
```python
# FreeCAD object properties (synced from Silo)
obj.addProperty("App::PropertyString", "SiloPartNumber", "Silo", "Part number")
obj.addProperty("App::PropertyString", "SiloRevision", "Silo", "Current revision")
obj.addProperty("App::PropertyString", "SiloDescription", "Silo", "Item description")
# ... additional properties as defined in schema
```
### 5.3 File Storage Strategy
FreeCAD `.FCStd` files are ZIP archives. Storage options:
1. **Whole file storage** (MVP): Store complete .FCStd in MinIO with versioning
2. **Exploded storage** (future): Unpack and store components separately for better diffing
For MVP, whole file storage is simpler and MinIO versioning handles history.
### 5.4 Checkout Locking (Future)
MVP operates as single-user. Future multi-user support will need locking strategy:
- **Pessimistic locking**: Checkout acquires exclusive lock
- **Optimistic locking**: Allow concurrent edits, handle conflicts on commit
Recommendation for future: Pessimistic locking for CAD files (merge is impractical).
---
## 6. Web Interface
### 6.1 Features
- **Browse**: Navigate item hierarchy (project → assembly → subassembly → part)
- **Search**: Full-text search across part numbers, descriptions, properties
- **View**: Item details, revision history, relationships, location
- **BOM Viewer**: Expandable tree view of assembly structure
- **"Open in FreeCAD"**: Launch FreeCAD with specific item via URI handler
### 6.2 URI Handler
Register `silo://` protocol handler:
```
silo://open/PROTO-AS-0001 # Open latest revision
silo://open/PROTO-AS-0001?rev=3 # Open specific revision
```
### 6.3 Technology
- **Backend**: Go with standard library HTTP
- **Frontend**: htmx for interactivity, minimal JavaScript
- **Templates**: Go html/template
- **Search**: PostgreSQL full-text search (pg_trgm for fuzzy matching)
---
## 7. Revision Tracking
### 7.1 Append-Only Model
Every property change creates a new revision record. The current state is always the latest revision, but any historical state can be reconstructed.
```
Item: PROTO-AS-0001
Revision 1 (2026-01-15): Initial creation
- description: "Main chassis assembly"
- material: null
- weight: null
Revision 2 (2026-01-20): Updated properties
- description: "Main chassis assembly"
- material: "6061-T6 Aluminum"
- weight: 2.5
Revision 3 (2026-02-01): Design change
- description: "Main chassis assembly v2"
- material: "6061-T6 Aluminum"
- weight: 2.3
```
### 7.2 Revision Creation
Revisions are created explicitly by user action (not automatic):
- `silo commit` from FreeCAD
- "Save Revision" button in web UI
- API call with explicit revision flag
### 7.3 Revision vs. File Version
- **Revision**: Silo metadata revision (tracked in PostgreSQL)
- **File Version**: MinIO object version (automatic on upload)
A single Silo revision may span multiple file uploads during editing. Only committed revisions create formal revision records.
---
## 8. Relationships and BOM
### 8.1 Relationship Types
| Type | Description | Use Case |
|------|-------------|----------|
| `component` | Part is used in assembly | Standard BOM entry |
| `alternate` | Interchangeable substitute | Alternative sourcing |
| `reference` | Related item, not in BOM | Drawings, specs, tools |
### 8.2 Reference Designators
For assemblies that require them (electronics, complex mechanisms):
```yaml
# Relationship record
parent: PROTO-AS-0001
child: PROTO-PT-0042
type: component
quantity: 4
reference_designators: ["R1", "R2", "R3", "R4"]
```
### 8.3 Revision-Specific Relationships
Relationships can link to specific child revisions or track latest:
```yaml
# Locked to specific revision
child: PROTO-PT-0042
child_revision: 3
# Always use latest (default for R&D)
child: PROTO-PT-0042
child_revision: null # means "latest"
```
Assembly metadata YAML controls default behavior per assembly.
---
## 9. Physical Inventory
### 9.1 Location Management
Locations are hierarchical, defined by YAML schema. Each item can exist at multiple locations with quantities.
```
Location: lab/main-area/shelf-a/bin-3
- PROTO-PT-0001: 15 units
- PROTO-PT-0002: 8 units
Location: lab/storage/shelf-b/bin-1
- PROTO-PT-0001: 50 units (spare stock)
```
### 9.2 Inventory Operations
- **Add**: Increase quantity at location
- **Remove**: Decrease quantity at location
- **Move**: Transfer between locations
- **Adjust**: Set absolute quantity (for cycle counts)
All operations logged for audit trail (future consideration).
---
## 10. Authentication (Future)
### 10.1 Current State (MVP)
Single-user, no authentication required.
### 10.2 Future: LDAPS Integration
Plan for FreeIPA integration:
```yaml
# /etc/silo/auth.yaml
auth:
provider: ldap
server: ldaps://ipa.kindred.internal
base_dn: "dc=kindred,dc=internal"
user_dn_template: "uid={username},cn=users,cn=accounts,dc=kindred,dc=internal"
group_base: "cn=groups,cn=accounts,dc=kindred,dc=internal"
# Role mapping
roles:
admin:
groups: ["silo-admins"]
editor:
groups: ["silo-users", "engineers"]
viewer:
groups: ["silo-viewers"]
```
---
## 11. API Design (Sketch)
### 11.1 REST Endpoints
```
# Items
GET /api/items # List/search items
POST /api/items # Create item
GET /api/items/{part_number} # Get item details
PUT /api/items/{part_number} # Update item (creates revision)
DELETE /api/items/{part_number} # Archive item
# Revisions
GET /api/items/{part_number}/revisions
GET /api/items/{part_number}/revisions/{rev}
# Relationships
GET /api/items/{part_number}/bom
POST /api/items/{part_number}/relationships
DELETE /api/items/{part_number}/relationships/{id}
# Files
GET /api/items/{part_number}/file
PUT /api/items/{part_number}/file
GET /api/items/{part_number}/file?rev={rev}
# Schemas
GET /api/schemas
POST /api/schemas
GET /api/schemas/{name}
# Locations
GET /api/locations
POST /api/locations
GET /api/locations/{path}
# Inventory
GET /api/inventory/{part_number}
POST /api/inventory/{part_number}/adjust
# Part number generation
POST /api/generate-part-number
Body: { "schema": "kindred-rd", "project": "PROTO", "part_type": "AS" }
Response: { "part_number": "PROTO-AS-0001" }
```
---
## 12. MVP Scope
### 12.1 Included
- [ ] PostgreSQL database schema
- [ ] YAML schema parser for part numbering
- [ ] Part number generation engine
- [ ] Basic CLI for item CRUD
- [ ] FreeCAD workbench with core commands (checkout, commit, status, register)
- [ ] MinIO integration for file storage
- [ ] Single-level and multi-level BOM support
- [ ] Reference designator tracking
- [ ] Alternate part tracking
- [ ] Revision history (append-only)
- [ ] Location hierarchy (YAML-defined)
- [ ] Basic inventory tracking (quantity at location)
- [ ] Web UI for browsing and search
- [ ] "Open in FreeCAD" URI handler
### 12.2 Excluded (Future)
- [ ] Schema migration tooling
- [ ] Multi-user with authentication
- [ ] Checkout locking
- [ ] Approval workflows
- [ ] External system integrations (ERP, purchasing)
- [ ] Exploded file storage with diffing
- [ ] Audit logging
- [ ] Notifications
- [ ] Reporting/analytics
---
## 13. Open Questions
1. **CLI language**: Go for consistency with web UI, or Python for FreeCAD ecosystem alignment?
2. **Property schema**: Should item properties be schema-defined (like part numbers) or freeform? Recommendation: Support both—schema defines expected properties, but allow ad-hoc additions.
3. **Thumbnail generation**: Generate thumbnails from .FCStd on commit? Useful for web UI browsing.
4. **Search indexing**: PostgreSQL full-text search sufficient, or add dedicated search (Meilisearch, etc.)?
5. **Offline operation**: Should FreeCAD workbench support offline mode with sync? Adds significant complexity.
---
## 14. References
### 14.1 Design Influences
- **CycloneDX BOM specification**: JSON/YAML schema patterns for component identification, relationships, and metadata (https://cyclonedx.org)
- **OpenBOM data model**: Reference-instance separation, flexible property schemas
- **FreeCAD DynamicData workbench**: Custom property patterns in FreeCAD
- **Ansible inventory YAML**: Hierarchical configuration patterns with variable inheritance
### 14.2 Related Standards
- **ISO 10303 (STEP)**: Product data representation
- **IPC-2581**: Electronics assembly BOM format
- **Package URL (PURL)**: Standardized component identification
---
## Appendix A: Example YAML Files
### A.1 Complete Part Numbering Schema
```yaml
# kindred-rd-schema.yaml
schema:
name: kindred-rd
version: 1
description: "Kindred Systems R&D part numbering for prototype development"
separator: "-"
uniqueness:
scope: global
case_sensitive: false
segments:
- name: project
type: string
length: 5
case: upper
description: "5-character project identifier"
validation:
pattern: "^[A-Z0-9]{5}$"
message: "Project code must be exactly 5 alphanumeric characters"
required: true
- name: part_type
type: enum
description: "Two-character type code"
required: true
values:
AS: "Assembly - multi-part unit"
PT: "Part - single manufactured item"
DW: "Drawing - technical drawing"
DC: "Document - specification, procedure, etc."
TB: "Tooling - jigs, fixtures, molds"
PC: "Purchased - externally sourced component"
EL: "Electrical - wiring, PCB, electronics"
SW: "Software - firmware, configuration"
- name: sequence
type: serial
length: 4
padding: "0"
start: 1
description: "Sequential number within project/type"
scope: "{project}-{part_type}"
format: "{project}-{part_type}-{sequence}"
# Validation rules applied to complete part number
validation:
min_length: 14
max_length: 14
# Metadata for UI/documentation
examples:
- "PROTO-AS-0001"
- "ALPHA-PT-0042"
- "BETA1-EL-0003"
```
### A.2 Complete Location Schema
```yaml
# kindred-locations.yaml
location_schema:
name: kindred-lab
version: 1
description: "Kindred Systems lab and storage locations"
path_separator: "/"
hierarchy:
- level: 0
type: facility
description: "Building or site"
name_pattern: "^[a-z][a-z0-9-]*$"
examples: ["lab", "warehouse", "office"]
- level: 1
type: area
description: "Room or zone within facility"
name_pattern: "^[a-z][a-z0-9-]*$"
examples: ["main-lab", "storage", "assembly"]
- level: 2
type: shelf
description: "Shelving unit"
name_pattern: "^shelf-[a-z]$"
examples: ["shelf-a", "shelf-b"]
- level: 3
type: bin
description: "Individual container or bin"
name_pattern: "^bin-[0-9]{1,3}$"
examples: ["bin-1", "bin-42", "bin-100"]
# Properties tracked per location type
properties:
facility:
- name: address
type: text
required: false
area:
- name: climate_controlled
type: boolean
default: false
shelf:
- name: max_weight_kg
type: number
required: false
bin:
- name: bin_size
type: enum
values: [small, medium, large]
default: medium
```
### A.3 Assembly Configuration
```yaml
# Stored as item property or linked document
# Example: assembly PROTO-AS-0001
assembly_config:
name: "Main Chassis Assembly"
relationship_types:
- component
- alternate
- reference
use_reference_designators: false
child_revision_tracking: latest
# Assembly-specific BOM properties
relationship_properties:
- name: installation_notes
type: text
- name: torque_spec
type: text
- name: adhesive_required
type: boolean
default: false
# Validation rules
validation:
require_quantity: true
min_components: 1
```