silo/docs/KC_SERVER.md

.kc Server-Side Metadata Integration

Status: Draft Date: February 2026

---

1. Purpose

When a .kc file is committed to Silo, the server extracts and indexes the silo/ directory contents so that metadata is queryable, diffable, and streamable without downloading the full file. This document specifies the server-side processing pipeline, database storage, API endpoints, and SSE events that support the Create viewport widgets defined in SILO_VIEWPORT.md.

The core principle: the .kc file is the transport format; Silo is the index. The silo/ directory entries are extracted into database columns on commit and packed back into the ZIP on checkout. The server never modifies the FreeCAD standard zone (Document.xml, .brp files, thumbnails/).

---

2. Commit Pipeline

When a .kc file is uploaded via POST /api/items/{partNumber}/file, the server runs an extraction pipeline before returning success.

2.1 Pipeline Steps

Client uploads .kc file
         |
         v
+-----------------------------+
|  1. Store file to disk      |  (existing behavior -- unchanged)
|     items/{pn}/rev{N}.kc    |
+-----------------------------+
         |
         v
+-----------------------------+
|  2. Open ZIP, read silo/    |
|     Parse each entry        |
+-----------------------------+
         |
         v
+-----------------------------+
|  3. Validate manifest.json  |
|     - UUID matches item     |
|     - kc_version supported  |
|     - revision_hash present |
+-----------------------------+
         |
         v
+-----------------------------+
|  4. Index metadata          |
|     - Upsert item_metadata  |
|     - Upsert dependencies   |
|     - Append history entry  |
|     - Snapshot approvals    |
|     - Register macros       |
|     - Register job defs     |
+-----------------------------+
         |
         v
+-----------------------------+
|  5. Broadcast SSE events    |
|     - revision.created      |
|     - metadata.updated      |
|     - bom.changed (if deps  |
|       differ from previous) |
+-----------------------------+
         |
         v
       Return 201 Created

2.2 Validation Rules

Check	Failure response
silo/manifest.json missing	400 Bad Request -- file is .fcstd not .kc
manifest.uuid doesn't match item's UUID	409 Conflict -- wrong item
manifest.kc_version > server's supported version	422 Unprocessable -- client newer than server
manifest.revision_hash matches current head	200 OK (no-op, file unchanged)
Any silo/ JSON fails to parse	422 Unprocessable with path and parse error

If validation fails, the blob is still stored (the user uploaded it), but no metadata indexing occurs. The item's revision is created with a metadata_error flag so the web UI can surface the problem.

2.3 Backward Compatibility

Plain .fcstd files (no silo/ directory) continue to work exactly as today -- stored on disk, revision created, no metadata extraction. The pipeline short-circuits at step 2 when no silo/ directory is found.

---

3. Database Schema

3.1 item_metadata Table

Stores the indexed contents of silo/metadata.json as structured JSONB, searchable and filterable via the existing item query endpoints.

CREATE TABLE item_metadata (
    item_id UUID PRIMARY KEY REFERENCES items(id) ON DELETE CASCADE,
    schema_name TEXT,
    tags TEXT[] NOT NULL DEFAULT '{}',
    lifecycle_state TEXT NOT NULL DEFAULT 'draft',
    fields JSONB NOT NULL DEFAULT '{}',
    kc_version TEXT,
    manifest_uuid UUID,
    silo_instance TEXT,
    revision_hash TEXT,
    updated_at TIMESTAMPTZ DEFAULT now(),
    updated_by TEXT
);

CREATE INDEX idx_item_metadata_tags ON item_metadata USING GIN (tags);
CREATE INDEX idx_item_metadata_lifecycle ON item_metadata (lifecycle_state);
CREATE INDEX idx_item_metadata_fields ON item_metadata USING GIN (fields);

On commit, the server upserts this row from silo/manifest.json and silo/metadata.json. The fields column contains the schema-driven key-value pairs exactly as they appear in the JSON.

3.2 item_dependencies Table

Stores the indexed contents of silo/dependencies.json. Replaces the BOM for assembly relationships that originate from the CAD model.

CREATE TABLE item_dependencies (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    parent_item_id UUID REFERENCES items(id) ON DELETE CASCADE,
    child_uuid UUID NOT NULL,
    child_part_number TEXT,
    child_revision INTEGER,
    quantity DECIMAL,
    label TEXT,
    relationship TEXT NOT NULL DEFAULT 'component',
    revision_number INTEGER NOT NULL,
    created_at TIMESTAMPTZ DEFAULT now()
);

CREATE INDEX idx_item_deps_parent ON item_dependencies (parent_item_id);
CREATE INDEX idx_item_deps_child ON item_dependencies (child_uuid);

This table complements the existing relationships table. The relationships table is the server-authoritative BOM (editable via the web UI and API). The item_dependencies table is the CAD-authoritative record extracted from the file. BOM merge (per BOM_MERGE.md) reconciles the two.

3.3 item_approvals Table

Stores the indexed contents of silo/approvals.json. Server-authoritative -- the .kc snapshot is a read cache.

CREATE TABLE item_approvals (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    item_id UUID REFERENCES items(id) ON DELETE CASCADE,
    eco_number TEXT,
    state TEXT NOT NULL DEFAULT 'draft',
    updated_at TIMESTAMPTZ DEFAULT now(),
    updated_by TEXT
);

CREATE TABLE approval_signatures (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    approval_id UUID REFERENCES item_approvals(id) ON DELETE CASCADE,
    username TEXT NOT NULL,
    role TEXT NOT NULL,
    status TEXT NOT NULL DEFAULT 'pending',
    signed_at TIMESTAMPTZ,
    comment TEXT
);

These tables exist independent of .kc commits -- approvals are created and managed through the web UI and API. On .kc checkout, the current approval state is serialized into silo/approvals.json for offline display.

3.4 item_macros Table

Registers macros from silo/macros/ for server-side discoverability and the future Macro Store module.

CREATE TABLE item_macros (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    item_id UUID REFERENCES items(id) ON DELETE CASCADE,
    filename TEXT NOT NULL,
    trigger TEXT NOT NULL DEFAULT 'manual',
    content TEXT NOT NULL,
    revision_number INTEGER NOT NULL,
    created_at TIMESTAMPTZ DEFAULT now(),
    UNIQUE(item_id, filename)
);

---

4. API Endpoints

These endpoints serve the viewport widgets in Create. All are under /api/items/{partNumber} and follow the existing auth model.

4.1 Metadata

Method	Path	Auth	Description
GET	/metadata	viewer	Get indexed metadata (schema fields, tags, lifecycle)
PUT	/metadata	editor	Update metadata fields from client
PATCH	/metadata/lifecycle	editor	Transition lifecycle state
PATCH	/metadata/tags	editor	Add/remove tags

GET /api/items/{partNumber}/metadata

Returns the indexed metadata for viewport display. This is the fast path -- reads from item_metadata rather than downloading and parsing the .kc ZIP.

{
  "schema_name": "mechanical-part-v2",
  "lifecycle_state": "draft",
  "tags": ["structural", "aluminum"],
  "fields": {
    "material": "6061-T6",
    "finish": "anodized",
    "weight_kg": 0.34,
    "category": "bracket"
  },
  "manifest": {
    "uuid": "550e8400-e29b-41d4-a716-446655440000",
    "silo_instance": "https://silo.example.com",
    "revision_hash": "a1b2c3d4e5f6",
    "kc_version": "1.0"
  },
  "updated_at": "2026-02-13T20:30:00Z",
  "updated_by": "joseph"
}

PUT /api/items/{partNumber}/metadata

Accepts a partial update of schema fields. The server merges into the existing fields JSONB. This is the write-back path for the Metadata Editor widget.

{
  "fields": {
    "material": "7075-T6",
    "weight_kg": 0.31
  }
}

The server validates field names against the schema descriptor. Unknown fields are rejected with 422.

PATCH /api/items/{partNumber}/metadata/lifecycle

Transitions lifecycle state. The server validates the transition is permitted (e.g., draft -> review is allowed, released -> draft is not without admin override).

{ "state": "review" }

4.2 Dependencies

Method	Path	Auth	Description
GET	/dependencies	viewer	Get CAD-extracted dependency list
GET	/dependencies/resolve	viewer	Resolve UUIDs to current part numbers and file status

GET /api/items/{partNumber}/dependencies

Returns the raw dependency list from the last .kc commit.

GET /api/items/{partNumber}/dependencies/resolve

Returns the dependency list with each UUID resolved to its current part number, revision, and whether the file exists on disk. This is what the Dependency Table widget calls to populate the status column.

{
  "links": [
    {
      "uuid": "660e8400-...",
      "part_number": "KC-BRK-0042",
      "label": "Base Plate",
      "revision": 2,
      "quantity": 1,
      "resolved": true,
      "file_available": true
    },
    {
      "uuid": "770e8400-...",
      "part_number": "KC-HDW-0108",
      "label": "M6 SHCS",
      "revision": 1,
      "quantity": 4,
      "resolved": true,
      "file_available": true
    },
    {
      "uuid": "880e8400-...",
      "part_number": null,
      "label": "Cover Panel",
      "revision": 1,
      "quantity": 1,
      "resolved": false,
      "file_available": false
    }
  ]
}

4.3 Approvals

Method	Path	Auth	Description
GET	/approvals	viewer	Get current approval state
POST	/approvals	editor	Create ECO / start approval workflow
POST	/approvals/{id}/sign	editor	Sign (approve/reject)

These endpoints power the Approvals Viewer widget. The viewer is read-only in Create -- sign actions happen in the web UI, but the API exists for both.

4.4 Macros

Method	Path	Auth	Description
GET	/macros	viewer	List registered macros
GET	/macros/{filename}	viewer	Get macro source

Read-only server-side. Macros are authored in Create and committed inside the .kc. The server indexes them for discoverability in the future Macro Store.

4.5 Existing Endpoints (unchanged)

The viewport widgets also consume these existing endpoints:

Widget	Endpoint	Purpose
History Viewer	GET /api/items/{pn}/revisions	Full revision list
History Viewer	GET /api/items/{pn}/revisions/compare	Property diff
Job Viewer	GET /api/jobs?item={pn}&definition={name}&limit=1	Last job run
Job Viewer	POST /api/jobs	Trigger job
Job Viewer	GET /api/jobs/{id}/logs	Job log
Manifest Viewer	GET /api/items/{pn}	Item details (UUID, etc.)

No changes needed to these -- they already exist and return the data the widgets need.

---

5. Checkout Pipeline

When a client downloads a .kc via GET /api/items/{partNumber}/file, the server packs current server-side state into the silo/ directory before serving the file. This ensures the client always gets the latest metadata, even if it was edited via the web UI since the last commit.

5.1 Pipeline Steps

Client requests file download
         |
         v
+-----------------------------+
|  1. Read .kc from disk      |
+-----------------------------+
         |
         v
+-----------------------------+
|  2. Pack silo/ from DB      |
|     - manifest.json (item)  |
|     - metadata.json (index) |
|     - history.json (revs)   |
|     - approvals.json (ECO)  |
|     - dependencies.json     |
|     - macros/ (index)       |
|     - jobs/ (job defs)      |
+-----------------------------+
         |
         v
+-----------------------------+
|  3. Replace silo/ in ZIP    |
|     Remove old entries      |
|     Write packed entries    |
+-----------------------------+
         |
         v
       Stream .kc to client

5.2 Packing Rules

silo/ entry	Source	Notes
manifest.json	item_metadata + items table	UUID from item, revision_hash from latest revision
metadata.json	item_metadata.fields + tags + lifecycle	Serialized from indexed columns
history.json	revisions table	Last 20 revisions for this item
approvals.json	item_approvals + approval_signatures	Current ECO state, omitted if no active ECO
dependencies.json	item_dependencies	Current revision's dependency list
macros/*.py	item_macros	All registered macros
jobs/*.yaml	job_definitions filtered by item type	Job definitions matching this item's trigger filters

5.3 Caching

Packing the silo/ directory on every download has a cost. To mitigate:

• ETag header: The response includes an ETag computed from the revision number + metadata updated_at. If the client sends If-None-Match, the server can return 304 Not Modified.
• Lazy packing: If the .kc blob's silo/manifest.json revision_hash matches the current head and item_metadata.updated_at is older than the blob's upload time, skip repacking entirely -- the blob is already current.

---

6. SSE Events

The viewport widgets subscribe to SSE for live updates. These events are broadcast when server-side metadata changes, whether via .kc commit, web UI edit, or API call.

Event	Payload	Trigger
metadata.updated	{part_number, changed_fields[], lifecycle_state, updated_by}	Metadata PUT/PATCH
metadata.lifecycle	{part_number, from_state, to_state, updated_by}	Lifecycle transition
metadata.tags	{part_number, added[], removed[]}	Tag add/remove
approval.created	{part_number, eco_number, state}	ECO created
approval.signed	{part_number, eco_number, user, role, status}	Approver action
approval.completed	{part_number, eco_number, final_state}	All approvers acted
dependencies.changed	{part_number, added[], removed[], changed[]}	Dependency diff on commit

Existing events (revision.created, job.*, bom.changed) continue to work as documented in SPECIFICATION.md and WORKERS.md.

6.1 Widget Subscription Map

Viewport widget	Subscribes to
Manifest Viewer	-- (read-only, no live updates)
Metadata Editor	metadata.updated, metadata.lifecycle, metadata.tags
History Viewer	revision.created
Approvals Viewer	approval.created, approval.signed, approval.completed
Dependency Table	dependencies.changed
Job Viewer	job.created, job.progress, job.completed, job.failed
Macro Editor	-- (local-only until committed)

---

7. Web UI Integration

The Silo web UI also benefits from indexed metadata. These are additions to existing pages, not new pages.

7.1 Items Page

The item detail panel gains a Metadata tab (alongside Main, Properties, Revisions, BOM, Where Used) showing the schema-driven form from GET /api/items/{pn}/metadata. Editable for editors.

7.2 Items List

New filterable columns: lifecycle_state, tags. The existing search endpoint gains metadata-aware filtering:

GET /api/items?lifecycle=released&tag=aluminum
GET /api/items/search?q=bracket&lifecycle=draft

7.3 Approvals Page

A new page accessible from the top navigation (visible when a future approvals module is enabled). Lists all active ECOs with their approval progress.

---

8. Migration

8.1 Database Migration

A single migration adds the item_metadata, item_dependencies, item_approvals, approval_signatures, and item_macros tables. Existing items have no metadata rows -- they're created on first .kc commit or via PUT /api/items/{pn}/metadata.

8.2 Backfill

For items that already have .kc files stored on disk (committed before this feature), an admin endpoint re-runs the extraction pipeline:

POST /api/admin/reindex-metadata

This iterates all items with .kc files, opens each ZIP, and indexes the silo/ contents. Idempotent -- safe to run multiple times.

---

9. Implementation Order

Phase	Server work	Supports client phase
1	item_metadata table + GET/PUT /metadata + commit extraction	SILO_VIEWPORT Phase 1-2 (Manifest, Metadata)
2	Pack silo/ on checkout + ETag caching	SILO_VIEWPORT Phase 1-3
3	item_dependencies table + /dependencies/resolve	SILO_VIEWPORT Phase 5 (Dependency Table)
4	item_macros table + /macros endpoints	SILO_VIEWPORT Phase 6 (Macro Editor)
5	item_approvals tables + /approvals endpoints	SILO_VIEWPORT Phase 7 (Approvals Viewer)
6	SSE events for metadata/approvals/dependencies	SILO_VIEWPORT Phase 8 (Live integration)
7	Web UI metadata tab + list filters	Independent of client

Phases 1-2 are prerequisite for the viewport to work with live data. Phases 3-6 can be built in parallel with client widget development. Phase 7 is web-UI-only and independent.

---

10. References

• SILO_VIEWPORT.md -- Client-side viewport widget specification
• KC_SPECIFICATION.md -- .kc file format specification
• SPECIFICATION.md -- Silo server API reference
• BOM_MERGE.md -- BOM merge rules (dependency reconciliation)
• WORKERS.md -- Job queue (job viewer data source)
• MODULES.md -- Module system (approval module gating)
• ROADMAP.md -- Platform roadmap tiers