kindred/silo

Fork 0

Files

josephforbes23 53b5edb25f update documentation and specs

2026-01-29 13:10:12 -06:00

27 KiB

Raw Blame History

Silo: Item Database and Part Management System for FreeCAD

Version: 0.1 Draft
Date: January 2026
Author: Kindred Systems LLC

1. Overview

Silo is an item database with configurable part number generation, designed for R&D-oriented workflows. It integrates with FreeCAD 1.0+ to provide git-like object management, revision tracking, and physical inventory location management.

1.1 Core Philosophy

Silo treats part numbering schemas as configuration, not code. Multiple numbering schemes can coexist, each defined in YAML. The system is schema-agnostic—it doesn't impose a particular part numbering philosophy (intelligent vs. non-intelligent numbers) but instead provides the machinery to implement whatever scheme the organization requires.

1.2 Key Principles

Items are the atomic unit: Everything is an item (parts, assemblies, drawings, documents)
Schemas are mutable: Part numbering schemas can evolve, though migration tooling is out of scope for MVP
Append-only history: All parameter changes are recorded; item state is reconstructable at any point in time
Configuration over convention: Hierarchies, relationships, and behaviors are YAML-defined

2. Architecture

2.1 Components

┌─────────────────────────────────────────────────────────────┐
│                        FreeCAD 1.0+                         │
│  ┌─────────────────────────────────────────────────────┐   │
│  │              Silo Workbench (Python)                │   │
│  │  - silo checkout / commit / status / log            │   │
│  │  - Part number generation                           │   │
│  │  - Property sync with FreeCAD objects               │   │
│  └─────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                     Silo Core (CLI/Library)                 │
│  - Schema parsing and validation                            │
│  - Part number generation engine                            │
│  - Revision management                                      │
│  - Relationship graph                                       │
└─────────────────────────────────────────────────────────────┘
                              │
              ┌───────────────┴───────────────┐
              ▼                               ▼
┌─────────────────────────┐     ┌─────────────────────────────┐
│   PostgreSQL            │     │   MinIO                     │
│   (psql.kindred.internal)│     │   - .FCStd file storage     │
│   - Item metadata       │     │   - Versioned objects       │
│   - Relationships       │     │   - Thumbnails              │
│   - Revision history    │     │                             │
│   - Location hierarchy  │     │                             │
└─────────────────────────┘     └─────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                     Web UI (Browse/Search)                  │
│  - Item browser with hierarchy navigation                   │
│  - Search and filtering                                     │
│  - "Open in FreeCAD" links (freecad:// URI handler)        │
│  - BOM viewer                                               │
└─────────────────────────────────────────────────────────────┘

2.2 Technology Stack

Component	Technology	Notes
Database	PostgreSQL	Existing instance at psql.kindred.internal
File Storage	MinIO	S3-compatible, versioning enabled
FreeCAD Integration	Python workbench	Macro-style commands
CLI & API Server	Go (1.23)	chi/v5 router, pgx/v5 driver, zerolog
Web UI	Go html/template + htmx	Lightweight, minimal JS

3. Data Model

3.1 Items

An item is the fundamental entity. Items have:

A part number (generated according to a schema)
A type (part, assembly, drawing, document, etc.)
Properties (key-value pairs, schema-defined and custom)
Relationships to other items
Revisions (append-only history)
Files (optional, stored in MinIO)
Location (optional physical inventory location)

3.2 Database Schema (Conceptual)

-- Part numbering schemas (YAML stored as text, parsed at runtime)
CREATE TABLE schemas (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name TEXT UNIQUE NOT NULL,
    version INTEGER NOT NULL DEFAULT 1,
    definition JSONB NOT NULL,  -- parsed YAML
    created_at TIMESTAMPTZ DEFAULT now(),
    updated_at TIMESTAMPTZ DEFAULT now()
);

-- Items (core entity)
CREATE TABLE items (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    part_number TEXT UNIQUE NOT NULL,
    schema_id UUID REFERENCES schemas(id),
    item_type TEXT NOT NULL,  -- 'part', 'assembly', 'drawing', etc.
    created_at TIMESTAMPTZ DEFAULT now(),
    current_revision_id UUID  -- points to latest revision
);

-- Append-only revision history
CREATE TABLE revisions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    item_id UUID REFERENCES items(id) NOT NULL,
    revision_number INTEGER NOT NULL,
    properties JSONB NOT NULL,  -- all properties at this revision
    file_version TEXT,  -- MinIO version ID if applicable
    created_at TIMESTAMPTZ DEFAULT now(),
    created_by TEXT,  -- user identifier (future: LDAP DN)
    comment TEXT,
    UNIQUE(item_id, revision_number)
);

-- Item relationships (BOM structure)
CREATE TABLE relationships (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    parent_item_id UUID REFERENCES items(id) NOT NULL,
    child_item_id UUID REFERENCES items(id) NOT NULL,
    relationship_type TEXT NOT NULL,  -- 'component', 'alternate', 'reference'
    quantity DECIMAL,
    reference_designator TEXT,  -- e.g., "R1", "C3" for electronics
    metadata JSONB,  -- assembly-specific relationship config
    revision_id UUID REFERENCES revisions(id),  -- which revision this applies to
    created_at TIMESTAMPTZ DEFAULT now()
);

-- Location hierarchy (configurable via YAML)
CREATE TABLE locations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    path TEXT UNIQUE NOT NULL,  -- e.g., "lab/shelf-a/bin-3"
    name TEXT NOT NULL,
    parent_id UUID REFERENCES locations(id),
    location_type TEXT NOT NULL,  -- defined in location schema
    metadata JSONB,
    created_at TIMESTAMPTZ DEFAULT now()
);

-- Item inventory (quantity at location)
CREATE TABLE inventory (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    item_id UUID REFERENCES items(id) NOT NULL,
    location_id UUID REFERENCES locations(id) NOT NULL,
    quantity DECIMAL NOT NULL DEFAULT 0,
    updated_at TIMESTAMPTZ DEFAULT now(),
    UNIQUE(item_id, location_id)
);

-- Sequence counters for part number generation
CREATE TABLE sequences (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    schema_id UUID REFERENCES schemas(id),
    scope TEXT NOT NULL,  -- scope key (e.g., project code, type code)
    current_value INTEGER NOT NULL DEFAULT 0,
    UNIQUE(schema_id, scope)
);

4. YAML Configuration System

4.1 Part Numbering Schema

Schemas define how part numbers are generated. Each schema consists of segments that are concatenated with a separator.

# /etc/silo/schemas/kindred-rd.yaml
schema:
  name: kindred-rd
  version: 3
  description: "Kindred Systems R&D part numbering"
  
  # Separator between segments (default: "-")
  separator: "-"
  
  # Uniqueness enforcement
  uniqueness:
    scope: global
    case_sensitive: false
  
  segments:
    - name: category
      type: enum
      description: "Category code (2-3 characters)"
      required: true
      values:
        F01: "Hex Cap Screw"
        F02: "Socket Head Cap Screw"
        # ... 70+ categories across:
        # F01-F18: Fasteners
        # C01-C17: Fluid Fittings
        # R01-R44: Motion Components
        # S01-S17: Structural Materials
        # E01-E27: Electrical Components
        # M01-M18: Mechanical Components
        # T01-T08: Tooling and Fixtures
        # A01-A07: Assemblies
        # P01-P05: Purchased/Off-the-Shelf
        # X01-X08: Custom Fabricated Parts
      
    - name: sequence
      type: serial
      length: 4
      padding: "0"
      start: 1
      description: "Sequential number within category"
      scope: "{category}"
      
  format: "{category}-{sequence}"
  
  # Example outputs:
  # F01-0001 (first hex cap screw)
  # R27-0001 (first linear rail)
  # A01-0001 (first assembly)

Note: The schema was migrated from a {project}-{type}-{sequence} format (v1) to {category}-{sequence} (v3). Projects are now managed as many-to-many tags on items rather than embedded in the part number. See migrations/006_project_tags.sql.

4.2 Segment Types

Type	Description	Options
`string`	Fixed or variable length string	`length`, `min_length`, `max_length`, `pattern`, `case`
`enum`	Predefined set of values	`values` (map of code → description)
`serial`	Auto-incrementing integer	`length`, `padding`, `start`, `scope`
`date`	Date-based segment	`format` (strftime-style)
`constant`	Fixed value	`value`

4.3 Serial Scope Templates

The scope field in serial segments supports template variables referencing other segments:

# Sequence per category (current kindred-rd schema)
scope: "{category}"

# Global sequence (no scope)
scope: null

4.4 Alternative Schema Example (Simple Sequential)

# /etc/silo/schemas/simple.yaml
schema:
  name: simple
  version: 1
  description: "Simple non-intelligent numbering"
  
  segments:
    - name: prefix
      type: constant
      value: "P"
      
    - name: sequence
      type: serial
      length: 6
      padding: "0"
      scope: null  # global counter
      
  format: "{prefix}{sequence}"
  separator: ""
  
  # Output: P000001, P000002, ...

4.5 Location Hierarchy Schema

# /etc/silo/schemas/locations.yaml
location_schema:
  name: kindred-lab
  version: 1
  
  hierarchy:
    - level: 0
      type: facility
      name_pattern: "^[a-z-]+$"
      
    - level: 1
      type: area
      name_pattern: "^[a-z-]+$"
      
    - level: 2
      type: shelf
      name_pattern: "^shelf-[a-z]$"
      
    - level: 3
      type: bin
      name_pattern: "^bin-[0-9]+$"
      
  # Path format
  path_separator: "/"
  
  # Example paths:
  # lab/main-area/shelf-a/bin-1
  # lab/storage/shelf-b/bin-12

4.6 Assembly Metadata Schema

Each assembly can define its own relationship tracking behavior:

# Stored in item properties or as a linked document
assembly_config:
  # What relationship types this assembly uses
  relationship_types:
    - component      # standard BOM entry
    - alternate      # interchangeable substitute
    - reference      # related but not part of BOM
    
  # Whether to track reference designators
  use_reference_designators: true
  designator_format: "^[A-Z]+[0-9]+$"  # e.g., R1, C3, U12
  
  # Revision linking behavior
  child_revision_tracking: specific  # or "latest"
  
  # Custom properties for relationships
  relationship_properties:
    - name: mounting_orientation
      type: enum
      values: [top, bottom, left, right, front, back]
    - name: notes
      type: text

5. FreeCAD Integration

5.1 Workbench Commands

The Silo workbench provides toolbar commands in FreeCAD:

Command	Description	Status
`Silo_Save`	Auto-save document and upload to MinIO	Implemented
`Silo_Commit`	Save with revision comment	Implemented
`Silo_Pull`	Download item by part number / create new	Implemented
`Silo_Push`	Batch upload modified files	Implemented
`Silo_Info`	View revision history for current item	Implemented
`Silo_Register`	Generate part number for current document	Implemented
`Silo_Open`	Open item from Silo by part number	Implemented
`Silo_Browse`	Browse items in a list dialog	Implemented

5.2 Property Synchronization

Silo properties map to FreeCAD custom properties:

# FreeCAD object properties (synced from Silo)
obj.addProperty("App::PropertyString", "SiloPartNumber", "Silo", "Part number")
obj.addProperty("App::PropertyString", "SiloRevision", "Silo", "Current revision")
obj.addProperty("App::PropertyString", "SiloDescription", "Silo", "Item description")
# ... additional properties as defined in schema

5.3 File Storage Strategy

FreeCAD .FCStd files are ZIP archives. Storage options:

Whole file storage (MVP): Store complete .FCStd in MinIO with versioning
Exploded storage (future): Unpack and store components separately for better diffing

For MVP, whole file storage is simpler and MinIO versioning handles history.

5.4 Checkout Locking (Future)

MVP operates as single-user. Future multi-user support will need locking strategy:

Pessimistic locking: Checkout acquires exclusive lock
Optimistic locking: Allow concurrent edits, handle conflicts on commit

Recommendation for future: Pessimistic locking for CAD files (merge is impractical).

6. Web Interface

6.1 Features

Browse: Navigate item hierarchy (project → assembly → subassembly → part)
Search: Full-text search across part numbers, descriptions, properties
View: Item details, revision history, relationships, location
BOM Viewer: Expandable tree view of assembly structure
"Open in FreeCAD": Launch FreeCAD with specific item via URI handler

6.2 URI Handler

silo://open/PROTO-AS-0001         # Open latest revision
silo://open/PROTO-AS-0001?rev=3   # Open specific revision

6.3 Technology

Backend: Go with standard library HTTP
Frontend: htmx for interactivity, minimal JavaScript
Templates: Go html/template
Search: PostgreSQL full-text search (pg_trgm for fuzzy matching)

7. Revision Tracking

7.1 Append-Only Model

Every property change creates a new revision record. The current state is always the latest revision, but any historical state can be reconstructed.

Item: PROTO-AS-0001

Revision 1 (2026-01-15): Initial creation
  - description: "Main chassis assembly"
  - material: null
  - weight: null
  
Revision 2 (2026-01-20): Updated properties
  - description: "Main chassis assembly"
  - material: "6061-T6 Aluminum"
  - weight: 2.5
  
Revision 3 (2026-02-01): Design change
  - description: "Main chassis assembly v2"
  - material: "6061-T6 Aluminum"  
  - weight: 2.3

7.2 Revision Creation

Revisions are created explicitly by user action (not automatic):

silo commit from FreeCAD
"Save Revision" button in web UI
API call with explicit revision flag

7.3 Revision vs. File Version

Revision: Silo metadata revision (tracked in PostgreSQL)
File Version: MinIO object version (automatic on upload)

A single Silo revision may span multiple file uploads during editing. Only committed revisions create formal revision records.

8. Relationships and BOM

8.1 Relationship Types

Type	Description	Use Case
`component`	Part is used in assembly	Standard BOM entry
`alternate`	Interchangeable substitute	Alternative sourcing
`reference`	Related item, not in BOM	Drawings, specs, tools

8.2 Reference Designators

For assemblies that require them (electronics, complex mechanisms):

# Relationship record
parent: PROTO-AS-0001
child: PROTO-PT-0042
type: component
quantity: 4
reference_designators: ["R1", "R2", "R3", "R4"]

8.3 Revision-Specific Relationships

Relationships can link to specific child revisions or track latest:

# Locked to specific revision
child: PROTO-PT-0042
child_revision: 3

# Always use latest (default for R&D)
child: PROTO-PT-0042
child_revision: null  # means "latest"

Assembly metadata YAML controls default behavior per assembly.

9. Physical Inventory

9.1 Location Management

Locations are hierarchical, defined by YAML schema. Each item can exist at multiple locations with quantities.

Location: lab/main-area/shelf-a/bin-3
  - PROTO-PT-0001: 15 units
  - PROTO-PT-0002: 8 units
  
Location: lab/storage/shelf-b/bin-1
  - PROTO-PT-0001: 50 units (spare stock)

9.2 Inventory Operations

Add: Increase quantity at location
Remove: Decrease quantity at location
Move: Transfer between locations
Adjust: Set absolute quantity (for cycle counts)

All operations logged for audit trail (future consideration).

10. Authentication (Future)

10.1 Current State (MVP)

Single-user, no authentication required.

10.2 Future: LDAPS Integration

Plan for FreeIPA integration:

# /etc/silo/auth.yaml
auth:
  provider: ldap
  server: ldaps://ipa.kindred.internal
  base_dn: "dc=kindred,dc=internal"
  user_dn_template: "uid={username},cn=users,cn=accounts,dc=kindred,dc=internal"
  group_base: "cn=groups,cn=accounts,dc=kindred,dc=internal"
  
  # Role mapping
  roles:
    admin:
      groups: ["silo-admins"]
    editor:
      groups: ["silo-users", "engineers"]
    viewer:
      groups: ["silo-viewers"]

11. API Design

11.1 REST Endpoints (Implemented)

# Health
GET    /health                                              # Basic health check
GET    /ready                                               # Readiness (DB + MinIO)

# Web UI
GET    /                                                    # Items page
GET    /schemas                                             # Schemas page

# Schemas
GET    /api/schemas                                         # List all schemas
GET    /api/schemas/{name}                                  # Get schema details
GET    /api/schemas/{name}/properties                       # Get property schema for category
POST   /api/schemas/{name}/segments/{segment}/values        # Add enum value
PUT    /api/schemas/{name}/segments/{segment}/values/{code} # Update enum value
DELETE /api/schemas/{name}/segments/{segment}/values/{code} # Delete enum value

# Projects
GET    /api/projects                                        # List projects
POST   /api/projects                                        # Create project
GET    /api/projects/{code}                                 # Get project
PUT    /api/projects/{code}                                 # Update project
DELETE /api/projects/{code}                                 # Delete project
GET    /api/projects/{code}/items                           # Get project items

# Items
GET    /api/items                                           # List/search items
POST   /api/items                                           # Create item
GET    /api/items/export.csv                                # Export items to CSV
POST   /api/items/import                                    # Import items from CSV
GET    /api/items/template.csv                              # Get CSV import template
GET    /api/items/{partNumber}                              # Get item details
PUT    /api/items/{partNumber}                              # Update item
DELETE /api/items/{partNumber}                              # Archive item

# Item-Project Tags
GET    /api/items/{partNumber}/projects                     # Get item's projects
POST   /api/items/{partNumber}/projects                     # Add project tags
DELETE /api/items/{partNumber}/projects/{code}              # Remove project tag

# Revisions
GET    /api/items/{partNumber}/revisions                    # List revisions
POST   /api/items/{partNumber}/revisions                    # Create revision
GET    /api/items/{partNumber}/revisions/compare            # Compare two revisions
GET    /api/items/{partNumber}/revisions/{revision}         # Get specific revision
PATCH  /api/items/{partNumber}/revisions/{revision}         # Update status/labels
POST   /api/items/{partNumber}/revisions/{revision}/rollback # Rollback to revision

# Files
POST   /api/items/{partNumber}/file                         # Upload file
GET    /api/items/{partNumber}/file                         # Download latest file
GET    /api/items/{partNumber}/file/{revision}              # Download file at revision

# Part Number Generation
POST   /api/generate-part-number                            # Generate without creating item

11.2 Not Yet Implemented

The following endpoints from the original design are not yet implemented:

# Locations (tables exist, no API)
GET    /api/locations
POST   /api/locations
GET    /api/locations/{path}

# Inventory (tables exist, no API)
GET    /api/inventory/{partNumber}
POST   /api/inventory/{partNumber}/adjust

12. MVP Scope

12.1 Implemented

PostgreSQL database schema (7 migrations)
YAML schema parser for part numbering
Part number generation engine
CLI tool (cmd/silo)
API server (cmd/silod) with 35+ endpoints
FreeCAD workbench (save, commit, pull, push, info, register, open, browse)
MinIO integration for file storage with versioning
BOM relationships (component, alternate, reference)
Reference designator tracking
Revision history (append-only) with rollback and comparison
Revision status and labels
Project management with many-to-many item tagging
CSV import/export with dry-run validation
Web UI for items and schemas (htmx)
Property schema versioning framework
Docker Compose deployment (dev and prod)
systemd service and deployment scripts

12.2 Partially Implemented

Location hierarchy (database tables exist, no API endpoints)
Inventory tracking (database tables exist, no API endpoints)
Date segment type (schema parser placeholder only)
Part number format validation on creation

12.3 Not Started

Unit tests
Schema migration tooling
Multi-user authentication (FreeIPA/LDAP planned)
Checkout locking
Approval workflows
External system integrations (ERP, purchasing)
Exploded file storage with diffing
Audit logging
Notifications
Reporting/analytics

13. Open Questions

CLI language: Go for consistency with web UI, or Python for FreeCAD ecosystem alignment? Resolved: Go was chosen for both CLI and API server.
Property schema: Should item properties be schema-defined (like part numbers) or freeform? Recommendation: Support both—schema defines expected properties, but allow ad-hoc additions.
Thumbnail generation: Generate thumbnails from .FCStd on commit? Useful for web UI browsing.
Search indexing: PostgreSQL full-text search sufficient, or add dedicated search (Meilisearch, etc.)?
Offline operation: Should FreeCAD workbench support offline mode with sync? Adds significant complexity.

14. References

14.1 Design Influences

CycloneDX BOM specification: JSON/YAML schema patterns for component identification, relationships, and metadata (https://cyclonedx.org)
OpenBOM data model: Reference-instance separation, flexible property schemas
FreeCAD DynamicData workbench: Custom property patterns in FreeCAD
Ansible inventory YAML: Hierarchical configuration patterns with variable inheritance

ISO 10303 (STEP): Product data representation
IPC-2581: Electronics assembly BOM format
Package URL (PURL): Standardized component identification

Appendix A: Example YAML Files

A.1 Complete Part Numbering Schema

See schemas/kindred-rd.yaml for the full schema (v3). Summary:

# kindred-rd-schema.yaml (abbreviated)
schema:
  name: kindred-rd
  version: 3
  description: "Kindred Systems R&D part numbering"
  
  separator: "-"
  
  uniqueness:
    scope: global
    case_sensitive: false
  
  segments:
    - name: category
      type: enum
      description: "Category code"
      required: true
      values:
        F01: "Hex Cap Screw"
        F02: "Socket Head Cap Screw"
        # ... 70+ categories (see full file)
      
    - name: sequence
      type: serial
      length: 4
      padding: "0"
      start: 1
      description: "Sequential number within category"
      scope: "{category}"
  
  format: "{category}-{sequence}"
    
  # Example outputs: F01-0001, R27-0001, A01-0001

A.2 Complete Location Schema

# kindred-locations.yaml
location_schema:
  name: kindred-lab
  version: 1
  description: "Kindred Systems lab and storage locations"
  
  path_separator: "/"
  
  hierarchy:
    - level: 0
      type: facility
      description: "Building or site"
      name_pattern: "^[a-z][a-z0-9-]*$"
      examples: ["lab", "warehouse", "office"]
      
    - level: 1
      type: area
      description: "Room or zone within facility"
      name_pattern: "^[a-z][a-z0-9-]*$"
      examples: ["main-lab", "storage", "assembly"]
      
    - level: 2
      type: shelf
      description: "Shelving unit"
      name_pattern: "^shelf-[a-z]$"
      examples: ["shelf-a", "shelf-b"]
      
    - level: 3
      type: bin
      description: "Individual container or bin"
      name_pattern: "^bin-[0-9]{1,3}$"
      examples: ["bin-1", "bin-42", "bin-100"]
  
  # Properties tracked per location type
  properties:
    facility:
      - name: address
        type: text
        required: false
    area:
      - name: climate_controlled
        type: boolean
        default: false
    shelf:
      - name: max_weight_kg
        type: number
        required: false
    bin:
      - name: bin_size
        type: enum
        values: [small, medium, large]
        default: medium

A.3 Assembly Configuration

# Stored as item property or linked document
# Example: assembly PROTO-AS-0001
assembly_config:
  name: "Main Chassis Assembly"
  
  relationship_types:
    - component
    - alternate
    - reference
  
  use_reference_designators: false
  
  child_revision_tracking: latest
  
  # Assembly-specific BOM properties
  relationship_properties:
    - name: installation_notes
      type: text
    - name: torque_spec
      type: text
    - name: adhesive_required
      type: boolean
      default: false
  
  # Validation rules
  validation:
    require_quantity: true
    min_components: 1

27 KiB Raw Blame History