kindred/silo
1
1
Fork 0
Code Issues 30 Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request 'feat(db): .kc metadata database migration' (#148) from feat/kc-metadata-migration into main

Browse Source 
Reviewed-on: #148
This commit was merged in pull request #148.
This commit is contained in:
forbes
2026-02-18 21:05:15 +00:00
parent d96ba8d394 8d777e83bb
commit 628cd1d252
2 changed files with 595 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

485
docs/KC_SERVER.md Normal file
View File

@@ -0,0 +1,485 @@
# .kc Server-Side Metadata Integration
**Status:** Draft
**Date:** February 2026
---
## 1. Purpose
When a `.kc` file is committed to Silo, the server extracts and indexes the `silo/` directory contents so that metadata is queryable, diffable, and streamable without downloading the full file. This document specifies the server-side processing pipeline, database storage, API endpoints, and SSE events that support the Create viewport widgets defined in [SILO_VIEWPORT.md](SILO_VIEWPORT.md).
The core principle: **the `.kc` file is the transport format; Silo is the index.** The `silo/` directory entries are extracted into database columns on commit and packed back into the ZIP on checkout. The server never modifies the FreeCAD standard zone (`Document.xml`, `.brp` files, `thumbnails/`).
---
## 2. Commit Pipeline
When a `.kc` file is uploaded via `POST /api/items/{partNumber}/file`, the server runs an extraction pipeline before returning success.
### 2.1 Pipeline Steps
```
Client uploads .kc file
|
v
+-----------------------------+
| 1. Store file to disk | (existing behavior -- unchanged)
| items/{pn}/rev{N}.kc |
+-----------------------------+
|
v
+-----------------------------+
| 2. Open ZIP, read silo/ |
| Parse each entry |
+-----------------------------+
|
v
+-----------------------------+
| 3. Validate manifest.json |
| - UUID matches item |
| - kc_version supported |
| - revision_hash present |
+-----------------------------+
|
v
+-----------------------------+
| 4. Index metadata |
| - Upsert item_metadata |
| - Upsert dependencies |
| - Append history entry |
| - Snapshot approvals |
| - Register macros |
| - Register job defs |
+-----------------------------+
|
v
+-----------------------------+
| 5. Broadcast SSE events |
| - revision.created |
| - metadata.updated |
| - bom.changed (if deps |
| differ from previous) |
+-----------------------------+
|
v
Return 201 Created
```
### 2.2 Validation Rules
| Check | Failure response |
|-------|-----------------|
| `silo/manifest.json` missing | `400 Bad Request` -- file is `.fcstd` not `.kc` |
| `manifest.uuid` doesn't match item's UUID | `409 Conflict` -- wrong item |
| `manifest.kc_version` > server's supported version | `422 Unprocessable` -- client newer than server |
| `manifest.revision_hash` matches current head | `200 OK` (no-op, file unchanged) |
| Any `silo/` JSON fails to parse | `422 Unprocessable` with path and parse error |
If validation fails, the blob is still stored (the user uploaded it), but no metadata indexing occurs. The item's revision is created with a `metadata_error` flag so the web UI can surface the problem.
### 2.3 Backward Compatibility
Plain `.fcstd` files (no `silo/` directory) continue to work exactly as today -- stored on disk, revision created, no metadata extraction. The pipeline short-circuits at step 2 when no `silo/` directory is found.
---
## 3. Database Schema
### 3.1 `item_metadata` Table
Stores the indexed contents of `silo/metadata.json` as structured JSONB, searchable and filterable via the existing item query endpoints.
```sql
CREATE TABLE item_metadata (
item_id UUID PRIMARY KEY REFERENCES items(id) ON DELETE CASCADE,
schema_name TEXT,
tags TEXT[] NOT NULL DEFAULT '{}',
lifecycle_state TEXT NOT NULL DEFAULT 'draft',
fields JSONB NOT NULL DEFAULT '{}',
kc_version TEXT,
manifest_uuid UUID,
silo_instance TEXT,
revision_hash TEXT,
updated_at TIMESTAMPTZ DEFAULT now(),
updated_by TEXT
);
CREATE INDEX idx_item_metadata_tags ON item_metadata USING GIN (tags);
CREATE INDEX idx_item_metadata_lifecycle ON item_metadata (lifecycle_state);
CREATE INDEX idx_item_metadata_fields ON item_metadata USING GIN (fields);
```
On commit, the server upserts this row from `silo/manifest.json` and `silo/metadata.json`. The `fields` column contains the schema-driven key-value pairs exactly as they appear in the JSON.
### 3.2 `item_dependencies` Table
Stores the indexed contents of `silo/dependencies.json`. Replaces the BOM for assembly relationships that originate from the CAD model.
```sql
CREATE TABLE item_dependencies (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
parent_item_id UUID REFERENCES items(id) ON DELETE CASCADE,
child_uuid UUID NOT NULL,
child_part_number TEXT,
child_revision INTEGER,
quantity DECIMAL,
label TEXT,
relationship TEXT NOT NULL DEFAULT 'component',
revision_number INTEGER NOT NULL,
created_at TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX idx_item_deps_parent ON item_dependencies (parent_item_id);
CREATE INDEX idx_item_deps_child ON item_dependencies (child_uuid);
```
This table complements the existing `relationships` table. The `relationships` table is the server-authoritative BOM (editable via the web UI and API). The `item_dependencies` table is the CAD-authoritative record extracted from the file. BOM merge (per [BOM_MERGE.md](BOM_MERGE.md)) reconciles the two.
### 3.3 `item_approvals` Table
Stores the indexed contents of `silo/approvals.json`. Server-authoritative -- the `.kc` snapshot is a read cache.
```sql
CREATE TABLE item_approvals (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
item_id UUID REFERENCES items(id) ON DELETE CASCADE,
eco_number TEXT,
state TEXT NOT NULL DEFAULT 'draft',
updated_at TIMESTAMPTZ DEFAULT now(),
updated_by TEXT
);
CREATE TABLE approval_signatures (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
approval_id UUID REFERENCES item_approvals(id) ON DELETE CASCADE,
username TEXT NOT NULL,
role TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'pending',
signed_at TIMESTAMPTZ,
comment TEXT
);
```
These tables exist independent of `.kc` commits -- approvals are created and managed through the web UI and API. On `.kc` checkout, the current approval state is serialized into `silo/approvals.json` for offline display.
### 3.4 `item_macros` Table
Registers macros from `silo/macros/` for server-side discoverability and the future Macro Store module.
```sql
CREATE TABLE item_macros (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
item_id UUID REFERENCES items(id) ON DELETE CASCADE,
filename TEXT NOT NULL,
trigger TEXT NOT NULL DEFAULT 'manual',
content TEXT NOT NULL,
revision_number INTEGER NOT NULL,
created_at TIMESTAMPTZ DEFAULT now(),
UNIQUE(item_id, filename)
);
```
---
## 4. API Endpoints
These endpoints serve the viewport widgets in Create. All are under `/api/items/{partNumber}` and follow the existing auth model.
### 4.1 Metadata
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| `GET` | `/metadata` | viewer | Get indexed metadata (schema fields, tags, lifecycle) |
| `PUT` | `/metadata` | editor | Update metadata fields from client |
| `PATCH` | `/metadata/lifecycle` | editor | Transition lifecycle state |
| `PATCH` | `/metadata/tags` | editor | Add/remove tags |
**`GET /api/items/{partNumber}/metadata`**
Returns the indexed metadata for viewport display. This is the fast path -- reads from `item_metadata` rather than downloading and parsing the `.kc` ZIP.
```json
{
"schema_name": "mechanical-part-v2",
"lifecycle_state": "draft",
"tags": ["structural", "aluminum"],
"fields": {
"material": "6061-T6",
"finish": "anodized",
"weight_kg": 0.34,
"category": "bracket"
},
"manifest": {
"uuid": "550e8400-e29b-41d4-a716-446655440000",
"silo_instance": "https://silo.example.com",
"revision_hash": "a1b2c3d4e5f6",
"kc_version": "1.0"
},
"updated_at": "2026-02-13T20:30:00Z",
"updated_by": "joseph"
}
```
**`PUT /api/items/{partNumber}/metadata`**
Accepts a partial update of schema fields. The server merges into the existing `fields` JSONB. This is the write-back path for the Metadata Editor widget.
```json
{
"fields": {
"material": "7075-T6",
"weight_kg": 0.31
}
}
```
The server validates field names against the schema descriptor. Unknown fields are rejected with `422`.
**`PATCH /api/items/{partNumber}/metadata/lifecycle`**
Transitions lifecycle state. The server validates the transition is permitted (e.g., `draft` -> `review` is allowed, `released` -> `draft` is not without admin override).
```json
{ "state": "review" }
```
### 4.2 Dependencies
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| `GET` | `/dependencies` | viewer | Get CAD-extracted dependency list |
| `GET` | `/dependencies/resolve` | viewer | Resolve UUIDs to current part numbers and file status |
**`GET /api/items/{partNumber}/dependencies`**
Returns the raw dependency list from the last `.kc` commit.
**`GET /api/items/{partNumber}/dependencies/resolve`**
Returns the dependency list with each UUID resolved to its current part number, revision, and whether the file exists on disk. This is what the Dependency Table widget calls to populate the status column.
```json
{
"links": [
{
"uuid": "660e8400-...",
"part_number": "KC-BRK-0042",
"label": "Base Plate",
"revision": 2,
"quantity": 1,
"resolved": true,
"file_available": true
},
{
"uuid": "770e8400-...",
"part_number": "KC-HDW-0108",
"label": "M6 SHCS",
"revision": 1,
"quantity": 4,
"resolved": true,
"file_available": true
},
{
"uuid": "880e8400-...",
"part_number": null,
"label": "Cover Panel",
"revision": 1,
"quantity": 1,
"resolved": false,
"file_available": false
}
]
}
```
### 4.3 Approvals
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| `GET` | `/approvals` | viewer | Get current approval state |
| `POST` | `/approvals` | editor | Create ECO / start approval workflow |
| `POST` | `/approvals/{id}/sign` | editor | Sign (approve/reject) |
These endpoints power the Approvals Viewer widget. The viewer is read-only in Create -- sign actions happen in the web UI, but the API exists for both.
### 4.4 Macros
| Method | Path | Auth | Description |
|--------|------|------|-------------|
| `GET` | `/macros` | viewer | List registered macros |
| `GET` | `/macros/{filename}` | viewer | Get macro source |
Read-only server-side. Macros are authored in Create and committed inside the `.kc`. The server indexes them for discoverability in the future Macro Store.
### 4.5 Existing Endpoints (unchanged)
The viewport widgets also consume these existing endpoints:
| Widget | Endpoint | Purpose |
|--------|----------|---------|
| History Viewer | `GET /api/items/{pn}/revisions` | Full revision list |
| History Viewer | `GET /api/items/{pn}/revisions/compare` | Property diff |
| Job Viewer | `GET /api/jobs?item={pn}&definition={name}&limit=1` | Last job run |
| Job Viewer | `POST /api/jobs` | Trigger job |
| Job Viewer | `GET /api/jobs/{id}/logs` | Job log |
| Manifest Viewer | `GET /api/items/{pn}` | Item details (UUID, etc.) |
No changes needed to these -- they already exist and return the data the widgets need.
---
## 5. Checkout Pipeline
When a client downloads a `.kc` via `GET /api/items/{partNumber}/file`, the server packs current server-side state into the `silo/` directory before serving the file. This ensures the client always gets the latest metadata, even if it was edited via the web UI since the last commit.
### 5.1 Pipeline Steps
```
Client requests file download
|
v
+-----------------------------+
| 1. Read .kc from disk |
+-----------------------------+
|
v
+-----------------------------+
| 2. Pack silo/ from DB |
| - manifest.json (item) |
| - metadata.json (index) |
| - history.json (revs) |
| - approvals.json (ECO) |
| - dependencies.json |
| - macros/ (index) |
| - jobs/ (job defs) |
+-----------------------------+
|
v
+-----------------------------+
| 3. Replace silo/ in ZIP |
| Remove old entries |
| Write packed entries |
+-----------------------------+
|
v
Stream .kc to client
```
### 5.2 Packing Rules
| `silo/` entry | Source | Notes |
|---------------|--------|-------|
| `manifest.json` | `item_metadata` + `items` table | UUID from item, revision_hash from latest revision |
| `metadata.json` | `item_metadata.fields` + tags + lifecycle | Serialized from indexed columns |
| `history.json` | `revisions` table | Last 20 revisions for this item |
| `approvals.json` | `item_approvals` + `approval_signatures` | Current ECO state, omitted if no active ECO |
| `dependencies.json` | `item_dependencies` | Current revision's dependency list |
| `macros/*.py` | `item_macros` | All registered macros |
| `jobs/*.yaml` | `job_definitions` filtered by item type | Job definitions matching this item's trigger filters |
### 5.3 Caching
Packing the `silo/` directory on every download has a cost. To mitigate:
- **ETag header**: The response includes an ETag computed from the revision number + metadata `updated_at`. If the client sends `If-None-Match`, the server can return `304 Not Modified`.
- **Lazy packing**: If the `.kc` blob's `silo/manifest.json` revision_hash matches the current head *and* `item_metadata.updated_at` is older than the blob's upload time, skip repacking entirely -- the blob is already current.
---
## 6. SSE Events
The viewport widgets subscribe to SSE for live updates. These events are broadcast when server-side metadata changes, whether via `.kc` commit, web UI edit, or API call.
| Event | Payload | Trigger |
|-------|---------|---------|
| `metadata.updated` | `{part_number, changed_fields[], lifecycle_state, updated_by}` | Metadata PUT/PATCH |
| `metadata.lifecycle` | `{part_number, from_state, to_state, updated_by}` | Lifecycle transition |
| `metadata.tags` | `{part_number, added[], removed[]}` | Tag add/remove |
| `approval.created` | `{part_number, eco_number, state}` | ECO created |
| `approval.signed` | `{part_number, eco_number, user, role, status}` | Approver action |
| `approval.completed` | `{part_number, eco_number, final_state}` | All approvers acted |
| `dependencies.changed` | `{part_number, added[], removed[], changed[]}` | Dependency diff on commit |
Existing events (`revision.created`, `job.*`, `bom.changed`) continue to work as documented in [SPECIFICATION.md](SPECIFICATION.md) and [WORKERS.md](WORKERS.md).
### 6.1 Widget Subscription Map
| Viewport widget | Subscribes to |
|-----------------|---------------|
| Manifest Viewer | -- (read-only, no live updates) |
| Metadata Editor | `metadata.updated`, `metadata.lifecycle`, `metadata.tags` |
| History Viewer | `revision.created` |
| Approvals Viewer | `approval.created`, `approval.signed`, `approval.completed` |
| Dependency Table | `dependencies.changed` |
| Job Viewer | `job.created`, `job.progress`, `job.completed`, `job.failed` |
| Macro Editor | -- (local-only until committed) |
---
## 7. Web UI Integration
The Silo web UI also benefits from indexed metadata. These are additions to existing pages, not new pages.
### 7.1 Items Page
The item detail panel gains a **Metadata** tab (alongside Main, Properties, Revisions, BOM, Where Used) showing the schema-driven form from `GET /api/items/{pn}/metadata`. Editable for editors.
### 7.2 Items List
New filterable columns: `lifecycle_state`, `tags`. The existing search endpoint gains metadata-aware filtering:
```
GET /api/items?lifecycle=released&tag=aluminum
GET /api/items/search?q=bracket&lifecycle=draft
```
### 7.3 Approvals Page
A new page accessible from the top navigation (visible when a future `approvals` module is enabled). Lists all active ECOs with their approval progress.
---
## 8. Migration
### 8.1 Database Migration
A single migration adds the `item_metadata`, `item_dependencies`, `item_approvals`, `approval_signatures`, and `item_macros` tables. Existing items have no metadata rows -- they're created on first `.kc` commit or via `PUT /api/items/{pn}/metadata`.
### 8.2 Backfill
For items that already have `.kc` files stored on disk (committed before this feature), an admin endpoint re-runs the extraction pipeline:
```
POST /api/admin/reindex-metadata
```
This iterates all items with `.kc` files, opens each ZIP, and indexes the `silo/` contents. Idempotent -- safe to run multiple times.
---
## 9. Implementation Order
| Phase | Server work | Supports client phase |
|-------|------------|----------------------|
| 1 | `item_metadata` table + `GET/PUT /metadata` + commit extraction | SILO_VIEWPORT Phase 1-2 (Manifest, Metadata) |
| 2 | Pack `silo/` on checkout + ETag caching | SILO_VIEWPORT Phase 1-3 |
| 3 | `item_dependencies` table + `/dependencies/resolve` | SILO_VIEWPORT Phase 5 (Dependency Table) |
| 4 | `item_macros` table + `/macros` endpoints | SILO_VIEWPORT Phase 6 (Macro Editor) |
| 5 | `item_approvals` tables + `/approvals` endpoints | SILO_VIEWPORT Phase 7 (Approvals Viewer) |
| 6 | SSE events for metadata/approvals/dependencies | SILO_VIEWPORT Phase 8 (Live integration) |
| 7 | Web UI metadata tab + list filters | Independent of client |
Phases 1-2 are prerequisite for the viewport to work with live data. Phases 3-6 can be built in parallel with client widget development. Phase 7 is web-UI-only and independent.
---
## 10. References
- [SILO_VIEWPORT.md](SILO_VIEWPORT.md) -- Client-side viewport widget specification
- [KC_SPECIFICATION.md](KC_SPECIFICATION.md) -- .kc file format specification
- [SPECIFICATION.md](SPECIFICATION.md) -- Silo server API reference
- [BOM_MERGE.md](BOM_MERGE.md) -- BOM merge rules (dependency reconciliation)
- [WORKERS.md](WORKERS.md) -- Job queue (job viewer data source)
- [MODULES.md](MODULES.md) -- Module system (approval module gating)
- [ROADMAP.md](ROADMAP.md) -- Platform roadmap tiers

110
migrations/018_kc_metadata.sql Normal file
View File

@@ -0,0 +1,110 @@
-- Migration 018: .kc Server-Side Metadata Tables
--
-- Adds tables for indexing the silo/ directory contents from .kc files.
-- See docs/KC_SERVER.md for the full specification.
--
-- Tables:
-- item_metadata - indexed manifest + metadata fields (Section 3.1)
-- item_dependencies - CAD-extracted assembly dependencies (Section 3.2)
-- item_approvals - ECO workflow state (Section 3.3)
-- approval_signatures - individual approval/rejection records (Section 3.3)
-- item_macros - registered macros from silo/macros/ (Section 3.4)
BEGIN;
--------------------------------------------------------------------------------
-- item_metadata: indexed silo/manifest.json + silo/metadata.json
--------------------------------------------------------------------------------
CREATE TABLE item_metadata (
item_id UUID PRIMARY KEY REFERENCES items(id) ON DELETE CASCADE,
schema_name TEXT,
tags TEXT[] NOT NULL DEFAULT '{}',
lifecycle_state TEXT NOT NULL DEFAULT 'draft',
fields JSONB NOT NULL DEFAULT '{}',
kc_version TEXT,
manifest_uuid UUID,
silo_instance TEXT,
revision_hash TEXT,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_by TEXT
);
CREATE INDEX idx_item_metadata_tags ON item_metadata USING GIN (tags);
CREATE INDEX idx_item_metadata_lifecycle ON item_metadata (lifecycle_state);
CREATE INDEX idx_item_metadata_fields ON item_metadata USING GIN (fields);
--------------------------------------------------------------------------------
-- item_dependencies: indexed silo/dependencies.json
--
-- Complements the existing `relationships` table.
-- relationships = server-authoritative BOM (web UI / API editable)
-- item_dependencies = CAD-authoritative record (extracted from .kc)
-- BOM merge reconciles the two (see docs/BOM_MERGE.md).
--------------------------------------------------------------------------------
CREATE TABLE item_dependencies (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
parent_item_id UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE,
child_uuid UUID NOT NULL,
child_part_number TEXT,
child_revision INTEGER,
quantity DECIMAL,
label TEXT,
relationship TEXT NOT NULL DEFAULT 'component',
revision_number INTEGER NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX idx_item_deps_parent ON item_dependencies (parent_item_id);
CREATE INDEX idx_item_deps_child ON item_dependencies (child_uuid);
--------------------------------------------------------------------------------
-- item_approvals + approval_signatures: ECO workflow
--
-- Server-authoritative. The .kc silo/approvals.json is a read cache
-- packed on checkout for offline display in Create.
--------------------------------------------------------------------------------
CREATE TABLE item_approvals (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
item_id UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE,
eco_number TEXT,
state TEXT NOT NULL DEFAULT 'draft',
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_by TEXT
);
CREATE INDEX idx_item_approvals_item ON item_approvals (item_id);
CREATE INDEX idx_item_approvals_state ON item_approvals (state);
CREATE TABLE approval_signatures (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
approval_id UUID NOT NULL REFERENCES item_approvals(id) ON DELETE CASCADE,
username TEXT NOT NULL,
role TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'pending',
signed_at TIMESTAMPTZ,
comment TEXT
);
CREATE INDEX idx_approval_sigs_approval ON approval_signatures (approval_id);
--------------------------------------------------------------------------------
-- item_macros: registered macros from silo/macros/
--------------------------------------------------------------------------------
CREATE TABLE item_macros (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
item_id UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE,
filename TEXT NOT NULL,
trigger TEXT NOT NULL DEFAULT 'manual',
content TEXT NOT NULL,
revision_number INTEGER NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE(item_id, filename)
);
CREATE INDEX idx_item_macros_item ON item_macros (item_id);
COMMIT;