Files

Forbes c327baf36f update databasing system with minimum API, schema parsing and FreeCAD

integration

2026-01-24 15:03:17 -06:00

25 KiB

Raw Blame History

Silo: Item Database and Part Management System for FreeCAD

Version: 0.1 Draft
Date: January 2026
Author: Kindred Systems LLC

1. Overview

Silo is an item database with configurable part number generation, designed for R&D-oriented workflows. It integrates with FreeCAD 1.0+ to provide git-like object management, revision tracking, and physical inventory location management.

1.1 Core Philosophy

Silo treats part numbering schemas as configuration, not code. Multiple numbering schemes can coexist, each defined in YAML. The system is schema-agnostic—it doesn't impose a particular part numbering philosophy (intelligent vs. non-intelligent numbers) but instead provides the machinery to implement whatever scheme the organization requires.

1.2 Key Principles

Items are the atomic unit: Everything is an item (parts, assemblies, drawings, documents)
Schemas are mutable: Part numbering schemas can evolve, though migration tooling is out of scope for MVP
Append-only history: All parameter changes are recorded; item state is reconstructable at any point in time
Configuration over convention: Hierarchies, relationships, and behaviors are YAML-defined

2. Architecture

2.1 Components

┌─────────────────────────────────────────────────────────────┐
│                        FreeCAD 1.0+                         │
│  ┌─────────────────────────────────────────────────────┐   │
│  │              Silo Workbench (Python)                │   │
│  │  - silo checkout / commit / status / log            │   │
│  │  - Part number generation                           │   │
│  │  - Property sync with FreeCAD objects               │   │
│  └─────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                     Silo Core (CLI/Library)                 │
│  - Schema parsing and validation                            │
│  - Part number generation engine                            │
│  - Revision management                                      │
│  - Relationship graph                                       │
└─────────────────────────────────────────────────────────────┘
                              │
              ┌───────────────┴───────────────┐
              ▼                               ▼
┌─────────────────────────┐     ┌─────────────────────────────┐
│   PostgreSQL            │     │   MinIO                     │
│   (psql.kindred.internal)│     │   - .FCStd file storage     │
│   - Item metadata       │     │   - Versioned objects       │
│   - Relationships       │     │   - Thumbnails              │
│   - Revision history    │     │                             │
│   - Location hierarchy  │     │                             │
└─────────────────────────┘     └─────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                     Web UI (Browse/Search)                  │
│  - Item browser with hierarchy navigation                   │
│  - Search and filtering                                     │
│  - "Open in FreeCAD" links (freecad:// URI handler)        │
│  - BOM viewer                                               │
└─────────────────────────────────────────────────────────────┘

2.2 Technology Stack

Component	Technology	Notes
Database	PostgreSQL	Existing instance at psql.kindred.internal
File Storage	MinIO	S3-compatible, versioning enabled
FreeCAD Integration	Python workbench	Macro-style commands
CLI	Go or Python	TBD based on complexity
Web UI	Go + htmx	Lightweight, minimal JS

3. Data Model

3.1 Items

An item is the fundamental entity. Items have:

A part number (generated according to a schema)
A type (part, assembly, drawing, document, etc.)
Properties (key-value pairs, schema-defined and custom)
Relationships to other items
Revisions (append-only history)
Files (optional, stored in MinIO)
Location (optional physical inventory location)

3.2 Database Schema (Conceptual)

-- Part numbering schemas (YAML stored as text, parsed at runtime)
CREATE TABLE schemas (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name TEXT UNIQUE NOT NULL,
    version INTEGER NOT NULL DEFAULT 1,
    definition JSONB NOT NULL,  -- parsed YAML
    created_at TIMESTAMPTZ DEFAULT now(),
    updated_at TIMESTAMPTZ DEFAULT now()
);

-- Items (core entity)
CREATE TABLE items (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    part_number TEXT UNIQUE NOT NULL,
    schema_id UUID REFERENCES schemas(id),
    item_type TEXT NOT NULL,  -- 'part', 'assembly', 'drawing', etc.
    created_at TIMESTAMPTZ DEFAULT now(),
    current_revision_id UUID  -- points to latest revision
);

-- Append-only revision history
CREATE TABLE revisions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    item_id UUID REFERENCES items(id) NOT NULL,
    revision_number INTEGER NOT NULL,
    properties JSONB NOT NULL,  -- all properties at this revision
    file_version TEXT,  -- MinIO version ID if applicable
    created_at TIMESTAMPTZ DEFAULT now(),
    created_by TEXT,  -- user identifier (future: LDAP DN)
    comment TEXT,
    UNIQUE(item_id, revision_number)
);

-- Item relationships (BOM structure)
CREATE TABLE relationships (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    parent_item_id UUID REFERENCES items(id) NOT NULL,
    child_item_id UUID REFERENCES items(id) NOT NULL,
    relationship_type TEXT NOT NULL,  -- 'component', 'alternate', 'reference'
    quantity DECIMAL,
    reference_designator TEXT,  -- e.g., "R1", "C3" for electronics
    metadata JSONB,  -- assembly-specific relationship config
    revision_id UUID REFERENCES revisions(id),  -- which revision this applies to
    created_at TIMESTAMPTZ DEFAULT now()
);

-- Location hierarchy (configurable via YAML)
CREATE TABLE locations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    path TEXT UNIQUE NOT NULL,  -- e.g., "lab/shelf-a/bin-3"
    name TEXT NOT NULL,
    parent_id UUID REFERENCES locations(id),
    location_type TEXT NOT NULL,  -- defined in location schema
    metadata JSONB,
    created_at TIMESTAMPTZ DEFAULT now()
);

-- Item inventory (quantity at location)
CREATE TABLE inventory (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    item_id UUID REFERENCES items(id) NOT NULL,
    location_id UUID REFERENCES locations(id) NOT NULL,
    quantity DECIMAL NOT NULL DEFAULT 0,
    updated_at TIMESTAMPTZ DEFAULT now(),
    UNIQUE(item_id, location_id)
);

-- Sequence counters for part number generation
CREATE TABLE sequences (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    schema_id UUID REFERENCES schemas(id),
    scope TEXT NOT NULL,  -- scope key (e.g., project code, type code)
    current_value INTEGER NOT NULL DEFAULT 0,
    UNIQUE(schema_id, scope)
);

4. YAML Configuration System

4.1 Part Numbering Schema

Schemas define how part numbers are generated. Each schema consists of segments that are concatenated with a separator.

# /etc/silo/schemas/kindred-rd.yaml
schema:
  name: kindred-rd
  version: 1
  description: "Kindred Systems R&D part numbering"
  
  # Separator between segments (default: "-")
  separator: "-"
  
  # Uniqueness enforcement
  uniqueness:
    scope: global  # or "per-project", "per-type", "per-schema"
  
  segments:
    - name: project
      type: string
      length: 5
      description: "Project identifier"
      validation:
        pattern: "^[A-Z0-9]{5}$"
      required: true
      
    - name: part_type
      type: enum
      description: "Type of item"
      values:
        AS: "Assembly"
        PT: "Part"
        DW: "Drawing"
        DC: "Document"
        TB: "Tooling/Fixture"
        PC: "Purchased Component"
      required: true
      
    - name: sequence
      type: serial
      length: 4
      padding: "0"  # left-pad with zeros
      description: "Sequential number"
      scope: "{project}-{part_type}"  # counter scope (template)
      
  # Format template (optional, defaults to joining segments with separator)
  format: "{project}-{part_type}-{sequence}"
  
  # Example outputs:
  # PROTO-AS-0001 (first assembly in PROTO project)
  # PROTO-PT-0001 (first part in PROTO project)
  # ALPHA-AS-0001 (first assembly in ALPHA project)

4.2 Segment Types

Type	Description	Options
`string`	Fixed or variable length string	`length`, `min_length`, `max_length`, `pattern`, `case`
`enum`	Predefined set of values	`values` (map of code → description)
`serial`	Auto-incrementing integer	`length`, `padding`, `start`, `scope`
`date`	Date-based segment	`format` (strftime-style)
`constant`	Fixed value	`value`

4.3 Serial Scope Templates

The scope field in serial segments supports template variables referencing other segments:

# Sequence per project
scope: "{project}"

# Sequence per project AND type (recommended for R&D)
scope: "{project}-{part_type}"

# Global sequence (no scope)
scope: null

4.4 Alternative Schema Example (Simple Sequential)

# /etc/silo/schemas/simple.yaml
schema:
  name: simple
  version: 1
  description: "Simple non-intelligent numbering"
  
  segments:
    - name: prefix
      type: constant
      value: "P"
      
    - name: sequence
      type: serial
      length: 6
      padding: "0"
      scope: null  # global counter
      
  format: "{prefix}{sequence}"
  separator: ""
  
  # Output: P000001, P000002, ...

4.5 Location Hierarchy Schema

# /etc/silo/schemas/locations.yaml
location_schema:
  name: kindred-lab
  version: 1
  
  hierarchy:
    - level: 0
      type: facility
      name_pattern: "^[a-z-]+$"
      
    - level: 1
      type: area
      name_pattern: "^[a-z-]+$"
      
    - level: 2
      type: shelf
      name_pattern: "^shelf-[a-z]$"
      
    - level: 3
      type: bin
      name_pattern: "^bin-[0-9]+$"
      
  # Path format
  path_separator: "/"
  
  # Example paths:
  # lab/main-area/shelf-a/bin-1
  # lab/storage/shelf-b/bin-12

4.6 Assembly Metadata Schema

Each assembly can define its own relationship tracking behavior:

# Stored in item properties or as a linked document
assembly_config:
  # What relationship types this assembly uses
  relationship_types:
    - component      # standard BOM entry
    - alternate      # interchangeable substitute
    - reference      # related but not part of BOM
    
  # Whether to track reference designators
  use_reference_designators: true
  designator_format: "^[A-Z]+[0-9]+$"  # e.g., R1, C3, U12
  
  # Revision linking behavior
  child_revision_tracking: specific  # or "latest"
  
  # Custom properties for relationships
  relationship_properties:
    - name: mounting_orientation
      type: enum
      values: [top, bottom, left, right, front, back]
    - name: notes
      type: text

5. FreeCAD Integration

5.1 Workbench Commands

The Silo workbench provides git-like commands accessible via toolbar, menu, and Python console:

Command	Description
`silo init`	Initialize Silo tracking for current document
`silo status`	Show tracked/untracked objects, modifications
`silo checkout <part_number>`	Load item from Silo into current document
`silo commit`	Save current state as new revision
`silo log`	Show revision history
`silo diff`	Compare current state to last committed revision
`silo register`	Generate part number for selected object(s)
`silo link`	Create relationship between objects
`silo bom`	Generate BOM from current assembly

5.2 Property Synchronization

Silo properties map to FreeCAD custom properties:

# FreeCAD object properties (synced from Silo)
obj.addProperty("App::PropertyString", "SiloPartNumber", "Silo", "Part number")
obj.addProperty("App::PropertyString", "SiloRevision", "Silo", "Current revision")
obj.addProperty("App::PropertyString", "SiloDescription", "Silo", "Item description")
# ... additional properties as defined in schema

5.3 File Storage Strategy

FreeCAD .FCStd files are ZIP archives. Storage options:

Whole file storage (MVP): Store complete .FCStd in MinIO with versioning
Exploded storage (future): Unpack and store components separately for better diffing

For MVP, whole file storage is simpler and MinIO versioning handles history.

5.4 Checkout Locking (Future)

MVP operates as single-user. Future multi-user support will need locking strategy:

Pessimistic locking: Checkout acquires exclusive lock
Optimistic locking: Allow concurrent edits, handle conflicts on commit

Recommendation for future: Pessimistic locking for CAD files (merge is impractical).

6. Web Interface

6.1 Features

Browse: Navigate item hierarchy (project → assembly → subassembly → part)
Search: Full-text search across part numbers, descriptions, properties
View: Item details, revision history, relationships, location
BOM Viewer: Expandable tree view of assembly structure
"Open in FreeCAD": Launch FreeCAD with specific item via URI handler

6.2 URI Handler

silo://open/PROTO-AS-0001         # Open latest revision
silo://open/PROTO-AS-0001?rev=3   # Open specific revision

6.3 Technology

Backend: Go with standard library HTTP
Frontend: htmx for interactivity, minimal JavaScript
Templates: Go html/template
Search: PostgreSQL full-text search (pg_trgm for fuzzy matching)

7. Revision Tracking

7.1 Append-Only Model

Every property change creates a new revision record. The current state is always the latest revision, but any historical state can be reconstructed.

Item: PROTO-AS-0001

Revision 1 (2026-01-15): Initial creation
  - description: "Main chassis assembly"
  - material: null
  - weight: null
  
Revision 2 (2026-01-20): Updated properties
  - description: "Main chassis assembly"
  - material: "6061-T6 Aluminum"
  - weight: 2.5
  
Revision 3 (2026-02-01): Design change
  - description: "Main chassis assembly v2"
  - material: "6061-T6 Aluminum"  
  - weight: 2.3

7.2 Revision Creation

Revisions are created explicitly by user action (not automatic):

silo commit from FreeCAD
"Save Revision" button in web UI
API call with explicit revision flag

7.3 Revision vs. File Version

Revision: Silo metadata revision (tracked in PostgreSQL)
File Version: MinIO object version (automatic on upload)

A single Silo revision may span multiple file uploads during editing. Only committed revisions create formal revision records.

8. Relationships and BOM

8.1 Relationship Types

Type	Description	Use Case
`component`	Part is used in assembly	Standard BOM entry
`alternate`	Interchangeable substitute	Alternative sourcing
`reference`	Related item, not in BOM	Drawings, specs, tools

8.2 Reference Designators

For assemblies that require them (electronics, complex mechanisms):

# Relationship record
parent: PROTO-AS-0001
child: PROTO-PT-0042
type: component
quantity: 4
reference_designators: ["R1", "R2", "R3", "R4"]

8.3 Revision-Specific Relationships

Relationships can link to specific child revisions or track latest:

# Locked to specific revision
child: PROTO-PT-0042
child_revision: 3

# Always use latest (default for R&D)
child: PROTO-PT-0042
child_revision: null  # means "latest"

Assembly metadata YAML controls default behavior per assembly.

9. Physical Inventory

9.1 Location Management

Locations are hierarchical, defined by YAML schema. Each item can exist at multiple locations with quantities.

Location: lab/main-area/shelf-a/bin-3
  - PROTO-PT-0001: 15 units
  - PROTO-PT-0002: 8 units
  
Location: lab/storage/shelf-b/bin-1
  - PROTO-PT-0001: 50 units (spare stock)

9.2 Inventory Operations

Add: Increase quantity at location
Remove: Decrease quantity at location
Move: Transfer between locations
Adjust: Set absolute quantity (for cycle counts)

All operations logged for audit trail (future consideration).

10. Authentication (Future)

10.1 Current State (MVP)

Single-user, no authentication required.

10.2 Future: LDAPS Integration

Plan for FreeIPA integration:

# /etc/silo/auth.yaml
auth:
  provider: ldap
  server: ldaps://ipa.kindred.internal
  base_dn: "dc=kindred,dc=internal"
  user_dn_template: "uid={username},cn=users,cn=accounts,dc=kindred,dc=internal"
  group_base: "cn=groups,cn=accounts,dc=kindred,dc=internal"
  
  # Role mapping
  roles:
    admin:
      groups: ["silo-admins"]
    editor:
      groups: ["silo-users", "engineers"]
    viewer:
      groups: ["silo-viewers"]

11. API Design (Sketch)

11.1 REST Endpoints

# Items
GET    /api/items                    # List/search items
POST   /api/items                    # Create item
GET    /api/items/{part_number}      # Get item details
PUT    /api/items/{part_number}      # Update item (creates revision)
DELETE /api/items/{part_number}      # Archive item

# Revisions
GET    /api/items/{part_number}/revisions
GET    /api/items/{part_number}/revisions/{rev}

# Relationships
GET    /api/items/{part_number}/bom
POST   /api/items/{part_number}/relationships
DELETE /api/items/{part_number}/relationships/{id}

# Files
GET    /api/items/{part_number}/file
PUT    /api/items/{part_number}/file
GET    /api/items/{part_number}/file?rev={rev}

# Schemas
GET    /api/schemas
POST   /api/schemas
GET    /api/schemas/{name}

# Locations
GET    /api/locations
POST   /api/locations
GET    /api/locations/{path}

# Inventory
GET    /api/inventory/{part_number}
POST   /api/inventory/{part_number}/adjust

# Part number generation
POST   /api/generate-part-number
  Body: { "schema": "kindred-rd", "project": "PROTO", "part_type": "AS" }
  Response: { "part_number": "PROTO-AS-0001" }

12. MVP Scope

12.1 Included

PostgreSQL database schema
YAML schema parser for part numbering
Part number generation engine
Basic CLI for item CRUD
FreeCAD workbench with core commands (checkout, commit, status, register)
MinIO integration for file storage
Single-level and multi-level BOM support
Reference designator tracking
Alternate part tracking
Revision history (append-only)
Location hierarchy (YAML-defined)
Basic inventory tracking (quantity at location)
Web UI for browsing and search
"Open in FreeCAD" URI handler

12.2 Excluded (Future)

Schema migration tooling
Multi-user with authentication
Checkout locking
Approval workflows
External system integrations (ERP, purchasing)
Exploded file storage with diffing
Audit logging
Notifications
Reporting/analytics

13. Open Questions

CLI language: Go for consistency with web UI, or Python for FreeCAD ecosystem alignment?
Property schema: Should item properties be schema-defined (like part numbers) or freeform? Recommendation: Support both—schema defines expected properties, but allow ad-hoc additions.
Thumbnail generation: Generate thumbnails from .FCStd on commit? Useful for web UI browsing.
Search indexing: PostgreSQL full-text search sufficient, or add dedicated search (Meilisearch, etc.)?
Offline operation: Should FreeCAD workbench support offline mode with sync? Adds significant complexity.

14. References

14.1 Design Influences

CycloneDX BOM specification: JSON/YAML schema patterns for component identification, relationships, and metadata (https://cyclonedx.org)
OpenBOM data model: Reference-instance separation, flexible property schemas
FreeCAD DynamicData workbench: Custom property patterns in FreeCAD
Ansible inventory YAML: Hierarchical configuration patterns with variable inheritance

ISO 10303 (STEP): Product data representation
IPC-2581: Electronics assembly BOM format
Package URL (PURL): Standardized component identification

Appendix A: Example YAML Files

A.1 Complete Part Numbering Schema

# kindred-rd-schema.yaml
schema:
  name: kindred-rd
  version: 1
  description: "Kindred Systems R&D part numbering for prototype development"
  
  separator: "-"
  
  uniqueness:
    scope: global
    case_sensitive: false
  
  segments:
    - name: project
      type: string
      length: 5
      case: upper
      description: "5-character project identifier"
      validation:
        pattern: "^[A-Z0-9]{5}$"
        message: "Project code must be exactly 5 alphanumeric characters"
      required: true
      
    - name: part_type
      type: enum
      description: "Two-character type code"
      required: true
      values:
        AS: "Assembly - multi-part unit"
        PT: "Part - single manufactured item"
        DW: "Drawing - technical drawing"
        DC: "Document - specification, procedure, etc."
        TB: "Tooling - jigs, fixtures, molds"
        PC: "Purchased - externally sourced component"
        EL: "Electrical - wiring, PCB, electronics"
        SW: "Software - firmware, configuration"
      
    - name: sequence
      type: serial
      length: 4
      padding: "0"
      start: 1
      description: "Sequential number within project/type"
      scope: "{project}-{part_type}"
  
  format: "{project}-{part_type}-{sequence}"
  
  # Validation rules applied to complete part number
  validation:
    min_length: 14
    max_length: 14
    
  # Metadata for UI/documentation
  examples:
    - "PROTO-AS-0001"
    - "ALPHA-PT-0042"
    - "BETA1-EL-0003"

A.2 Complete Location Schema

# kindred-locations.yaml
location_schema:
  name: kindred-lab
  version: 1
  description: "Kindred Systems lab and storage locations"
  
  path_separator: "/"
  
  hierarchy:
    - level: 0
      type: facility
      description: "Building or site"
      name_pattern: "^[a-z][a-z0-9-]*$"
      examples: ["lab", "warehouse", "office"]
      
    - level: 1
      type: area
      description: "Room or zone within facility"
      name_pattern: "^[a-z][a-z0-9-]*$"
      examples: ["main-lab", "storage", "assembly"]
      
    - level: 2
      type: shelf
      description: "Shelving unit"
      name_pattern: "^shelf-[a-z]$"
      examples: ["shelf-a", "shelf-b"]
      
    - level: 3
      type: bin
      description: "Individual container or bin"
      name_pattern: "^bin-[0-9]{1,3}$"
      examples: ["bin-1", "bin-42", "bin-100"]
  
  # Properties tracked per location type
  properties:
    facility:
      - name: address
        type: text
        required: false
    area:
      - name: climate_controlled
        type: boolean
        default: false
    shelf:
      - name: max_weight_kg
        type: number
        required: false
    bin:
      - name: bin_size
        type: enum
        values: [small, medium, large]
        default: medium

A.3 Assembly Configuration

# Stored as item property or linked document
# Example: assembly PROTO-AS-0001
assembly_config:
  name: "Main Chassis Assembly"
  
  relationship_types:
    - component
    - alternate
    - reference
  
  use_reference_designators: false
  
  child_revision_tracking: latest
  
  # Assembly-specific BOM properties
  relationship_properties:
    - name: installation_notes
      type: text
    - name: torque_spec
      type: text
    - name: adhesive_required
      type: boolean
      default: false
  
  # Validation rules
  validation:
    require_quantity: true
    min_components: 1

25 KiB Raw Blame History