kindred/silo
1
1
Fork 0
Code Issues 30 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: LibreOffice Calc extension, ODS library, AI description, audit design

Browse Source 
Calc extension (pkg/calc/):
- Python UNO ProtocolHandler with 8 toolbar commands
- SiloClient HTTP client adapted from FreeCAD workbench
- Pull BOM/Project: populates sheets with 28-col format, hidden property
  columns, row hash tracking, auto project tagging
- Push: row classification, create/update items, conflict detection
- Completion wizard: 3-step category/description/fields with PN conflict
  resolution dialog
- OpenRouter AI integration: generate standardized descriptions from seller
  text, configurable model/instructions, review dialog
- Settings: JSON persistence, env var fallbacks, OpenRouter fields
- 31 unit tests (no UNO/network required)

Go ODS library (internal/ods/):
- Pure Go ODS read/write (ZIP of XML, no headless LibreOffice)
- Writer, reader, 10 round-trip tests

Server ODS endpoints (internal/api/ods.go):
- GET /api/items/export.ods, template.ods, POST import.ods
- GET /api/items/{pn}/bom/export.ods
- GET /api/projects/{code}/sheet.ods
- POST /api/sheets/diff

Documentation:
- docs/CALC_EXTENSION.md: extension progress report
- docs/COMPONENT_AUDIT.md: web audit tool design with weighted scoring,
  assembly computed fields, batch AI assistance plan
This commit is contained in:
Forbes
2026-02-01 10:06:20 -06:00
parent d5be2eb152
commit afb382b68d
28 changed files with 7599 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

523
docs/COMPONENT_AUDIT.md Normal file
View File

@@ -0,0 +1,523 @@
# Component Audit Tool
**Last Updated:** 2026-02-01
**Status:** Design
---
## Problem
The parts database has grown organically. Many items were created with only a
part number, description, and category. The property schema defines dozens of
fields per category (material, finish, manufacturer, supplier, cost, etc.) but
most items have few or none of these populated. There is no way to see which
items are missing data or to prioritize what needs filling in.
Currently, adding or updating properties requires either:
- Editing each item individually through the web UI detail panel
- Bulk CSV export, manual editing, re-import
- The Calc extension (new, not yet widely used)
None of these approaches give visibility into what's missing across the
database. Engineers don't know which items need attention until they encounter
a blank field during a design review or procurement cycle.
---
## Goals
1. Show a per-item completeness score based on the property schema
2. Surface the least-complete items so they can be prioritized
3. Let users fill in missing fields directly from the audit view
4. Filter by project, category, completeness threshold
5. Track improvement over time
---
## Design
The audit tool is a new page in the existing web UI (`/audit`), built with
the same server-rendered Go templates + vanilla JS approach as the items and
projects pages. It adds one new API endpoint for the completeness data and
reuses existing endpoints for updates.
### Completeness Scoring
Each item's completeness is computed against its category's property schema.
The schema defines both **global defaults** (12 fields, all categories) and
**category-specific properties** (varies: 9 fields for fasteners, 20+ for
motion components, etc.).
**Score formula:**
```
score = sum(weight for each filled field) / sum(weight for all applicable fields)
```
Score is 0.0 to 1.0, displayed as a percentage. Fields are weighted
differently depending on sourcing type.
**Purchased parts (`sourcing_type = "purchased"`):**
| Weight | Fields | Rationale |
|--------|--------|-----------|
| 3 | manufacturer_pn, sourcing_link | Can't procure without these |
| 2 | manufacturer, supplier, supplier_pn, standard_cost | Core procurement data |
| 1 | description, sourcing_type, lead_time_days, minimum_order_qty, lifecycle_status | Important but less blocking |
| 1 | All category-specific properties | Engineering detail |
| 0.5 | rohs_compliant, country_of_origin, notes, long_description | Nice to have |
**Manufactured parts (`sourcing_type = "manufactured"`):**
| Weight | Fields | Rationale |
|--------|--------|-----------|
| 3 | has_bom (at least one BOM child) | Can't manufacture without a BOM |
| 2 | description, standard_cost | Core identification |
| 1 | All category-specific properties | Engineering detail |
| 0.5 | manufacturer, supplier, notes, long_description | Less relevant for in-house |
The `has_bom` check for manufactured parts queries the `relationships`
table for at least one `rel_type = 'component'` child. This is not a
property field -- it's a structural check. A manufactured part with no BOM
children is flagged as critically incomplete regardless of how many other
fields are filled.
**Assemblies (categories A01-A07):**
Assembly scores are partially computed from children:
| Field | Source | Notes |
|-------|--------|-------|
| weight | Sum of child weights | Computed if all children have weight |
| standard_cost | Sum of child (cost * qty) | Computed from BOM |
| component_count | Count of BOM children | Always computable |
| has_bom | BOM children exist | Required (weight 3) |
A computed field counts as "filled" if the data needed to compute it is
available. If a computed value exists, it is shown alongside the stored
value so engineers can verify or override.
Assembly-specific properties that cannot be computed (assembly_time,
test_procedure, ip_rating, dimensions) are scored normally.
**Field filled criteria:**
- String fields: non-empty after trimming
- Number fields: non-null and non-zero
- Boolean fields: non-null (false is a valid answer)
- has_bom: at least one component relationship exists
Item-level fields (`description`, `sourcing_type`, `sourcing_link`,
`standard_cost`, `long_description`) are checked on the items table.
Property fields (`manufacturer`, `material`, etc.) are checked on the
current revision's `properties` JSONB column. BOM existence is checked
on the `relationships` table.
### Tiers
Items are grouped into completeness tiers for dashboard display:
| Tier | Range | Color | Label |
|------|-------|-------|-------|
| Critical | 0-25% | Red | Missing critical data |
| Low | 25-50% | Orange | Needs attention |
| Partial | 50-75% | Yellow | Partially complete |
| Good | 75-99% | Light green | Nearly complete |
| Complete | 100% | Green | All fields populated |
---
## API
### `GET /api/audit/completeness`
Returns completeness scores for all items (or filtered subset).
**Query parameters:**
| Param | Type | Description |
|-------|------|-------------|
| `project` | string | Filter by project code |
| `category` | string | Filter by category prefix (e.g. `F`, `F01`) |
| `max_score` | float | Only items below this score (e.g. `0.5`) |
| `min_score` | float | Only items above this score |
| `sort` | string | `score_asc` (default), `score_desc`, `part_number`, `updated_at` |
| `limit` | int | Pagination limit (default 100) |
| `offset` | int | Pagination offset |
**Response:**
```json
{
"items": [
{
"part_number": "F01-0042",
"description": "M3x10 Socket Head Cap Screw",
"category": "F01",
"category_name": "Screws and Bolts",
"sourcing_type": "purchased",
"projects": ["3DX10", "PROTO"],
"score": 0.41,
"weighted_filled": 12.5,
"weighted_total": 30.5,
"has_bom": false,
"bom_children": 0,
"missing_critical": ["manufacturer_pn", "sourcing_link"],
"missing": [
"manufacturer_pn",
"sourcing_link",
"supplier",
"supplier_pn",
"finish",
"strength_grade",
"torque_spec"
],
"updated_at": "2026-01-15T10:30:00Z"
},
{
"part_number": "A01-0003",
"description": "3DX10 Line Assembly",
"category": "A01",
"category_name": "Mechanical Assembly",
"sourcing_type": "manufactured",
"projects": ["3DX10"],
"score": 0.68,
"weighted_filled": 15.0,
"weighted_total": 22.0,
"has_bom": true,
"bom_children": 12,
"computed_fields": {
"standard_cost": 7538.61,
"component_count": 12,
"weight": null
},
"missing_critical": [],
"missing": ["assembly_time", "test_procedure", "weight", "ip_rating"],
"updated_at": "2026-01-28T14:20:00Z"
}
],
"summary": {
"total_items": 847,
"avg_score": 0.42,
"manufactured_without_bom": 31,
"by_tier": {
"critical": 123,
"low": 298,
"partial": 251,
"good": 142,
"complete": 33
},
"by_category": {
"F": {"count": 156, "avg_score": 0.51},
"C": {"count": 89, "avg_score": 0.38},
"R": {"count": 201, "avg_score": 0.29}
}
}
}
```
### `GET /api/audit/completeness/{partNumber}`
Single-item detail with field-by-field breakdown.
```json
{
"part_number": "F01-0042",
"description": "M3x10 Socket Head Cap Screw",
"category": "F01",
"sourcing_type": "purchased",
"score": 0.41,
"has_bom": false,
"fields": [
{"key": "description", "source": "item", "weight": 1, "value": "M3x10 Socket Head Cap Screw", "filled": true},
{"key": "sourcing_type", "source": "item", "weight": 1, "value": "purchased", "filled": true},
{"key": "standard_cost", "source": "item", "weight": 2, "value": 0.12, "filled": true},
{"key": "sourcing_link", "source": "item", "weight": 3, "value": "", "filled": false},
{"key": "manufacturer", "source": "property", "weight": 2, "value": null, "filled": false},
{"key": "manufacturer_pn", "source": "property", "weight": 3, "value": null, "filled": false},
{"key": "supplier", "source": "property", "weight": 2, "value": null, "filled": false},
{"key": "supplier_pn", "source": "property", "weight": 2, "value": null, "filled": false},
{"key": "material", "source": "property", "weight": 1, "value": "18-8 Stainless Steel", "filled": true},
{"key": "finish", "source": "property", "weight": 1, "value": null, "filled": false},
{"key": "thread_size", "source": "property", "weight": 1, "value": "M3", "filled": true},
{"key": "thread_pitch", "source": "property", "weight": 1, "value": null, "filled": false},
{"key": "length", "source": "property", "weight": 1, "value": "10mm", "filled": true},
{"key": "head_type", "source": "property", "weight": 1, "value": "Socket", "filled": true}
]
}
```
For assemblies, the detail response includes a `computed_fields` section
showing values derived from children (cost rollup, weight rollup,
component count). These are presented alongside stored values in the UI
so engineers can compare and choose to accept the computed value.
Existing `PUT /api/items/{pn}` and revision property updates handle writes.
---
## Web UI
### Audit Page (`/audit`)
New page accessible from the top navigation bar (fourth tab after Items,
Projects, Schemas).
**Layout:**
```
+------------------------------------------------------------------+
| Items | Projects | Schemas | Audit |
+------------------------------------------------------------------+
| [Project: ___] [Category: ___] [Max Score: ___] [Search] |
+------------------------------------------------------------------+
| Summary Bar |
| [===Critical: 123===|===Low: 298===|==Partial: 251==|Good|Done] |
+------------------------------------------------------------------+
| Score | PN | Description | Category | Missing|
|-------|-----------|--------------------------|----------|--------|
| 12% | R01-0003 | Bearing, Deep Groove 6205| Bearings | 18 |
| 15% | E14-0001 | NTC Thermistor 10K | Sensors | 16 |
| 23% | C03-0012 | 1/4" NPT Ball Valve SS | Valves | 14 |
| 35% | F01-0042 | M3x10 Socket Head Cap | Screws | 7 |
| ... | | | | |
+------------------------------------------------------------------+
```
**Interactions:**
- Click a row to open an inline edit panel (right side, same split-panel
pattern as the items page)
- The edit panel shows all applicable fields for the category, with empty
fields highlighted
- Editing a field and pressing Enter/Tab saves immediately via API
- Score updates live after each save
- Summary bar updates as items are completed
- Click a tier segment in the summary bar to filter to that tier
### Inline Edit Panel
```
+----------------------------------+
| F01-0042 Score: 35% |
| M3x10 Socket Head Cap Screw |
+----------------------------------+
| -- Required -- |
| Description [M3x10 Socket H..] |
| Sourcing [purchased v ] |
+----------------------------------+
| -- Procurement -- |
| Manufacturer [________________] |
| Mfr PN [________________] |
| Supplier [________________] |
| Supplier PN [________________] |
| Cost [$0.12 ] |
| Sourcing Link[________________] |
| Lead Time [____ days ] |
+----------------------------------+
| -- Fastener Properties -- |
| Material [18-8 Stainless ] |
| Finish [________________] |
| Thread Size [M3 ] |
| Thread Pitch [________________] |
| Length [10mm ] |
| Head Type [Socket ] |
| Drive Type [________________] |
| Strength [________________] |
| Torque Spec [________________] |
+----------------------------------+
| [Save All] |
+----------------------------------+
```
Fields are grouped into sections: Required, Procurement (global defaults),
and category-specific properties. Empty fields have a subtle red left border.
Filled fields have a green left border. The score bar at the top updates as
fields are filled in.
---
## Implementation Plan
### Phase 1: API endpoint and scoring engine
New file: `internal/api/audit_handlers.go`
- `HandleAuditCompleteness` -- query items, join current revision properties,
compute scores against schema, return paginated JSON
- `HandleAuditItemDetail` -- single item with field-by-field breakdown
- Scoring logic in a helper function that takes item fields + revision
properties + category schema and returns score + missing list
Register routes:
- `GET /api/audit/completeness` (viewer role)
- `GET /api/audit/completeness/{partNumber}` (viewer role)
### Phase 2: Web UI page
New template: `internal/api/templates/audit.html`
- Same base template, Catppuccin Mocha theme, nav bar with Audit tab
- Summary bar with tier counts (colored segments)
- Sortable, filterable table
- Split-panel detail view on row click
- Vanilla JS fetch calls to audit and item update endpoints
Update `internal/api/web.go`:
- Add `HandleAuditPage` handler
- Register `GET /audit` route
Update `internal/api/templates/base.html`:
- Add Audit tab to navigation
### Phase 3: Inline editing
- Field save on blur/Enter via `PUT /api/items/{pn}` for item fields
- Property updates via `POST /api/items/{pn}/revisions` with updated
properties map
- Live score recalculation after save (re-fetch from audit detail endpoint)
- Batch "Save All" button for multiple field changes
### Phase 4: Tracking and reporting
- Store periodic score snapshots (daily cron or on-demand) in a new
`audit_snapshots` table for trend tracking
- Dashboard chart showing completeness improvement over time
- Per-project completeness summary on the projects page
- CSV export of audit results for offline review
### Phase 5: Batch AI assistance
Server-side OpenRouter integration for bulk property inference from existing
sourcing data. This extends the Calc extension's AI client pattern to the
backend.
**Workflow:**
1. Audit page shows items with sourcing links but missing properties
2. Engineer selects items (or filters to a category/project) and clicks
"AI Fill Properties"
3. Server fetches each item's sourcing link page content (or uses the
seller description from the item's metadata)
4. OpenRouter API call per item: system prompt describes the category's
property schema, user prompt provides the scraped/stored description
5. AI returns structured JSON with suggested property values
6. Results shown in a review table: item, field, current value, suggested
value, confidence indicator
7. Engineer checks/unchecks suggestions, clicks "Apply Selected"
8. Server writes accepted values as property updates (new revision)
**AI prompt structure:**
```
System: You are a parts data specialist. Given a product description
and a list of property fields with types, extract values for as many
fields as possible. Return JSON only.
User:
Category: F01 (Screws and Bolts)
Product: {seller_description or scraped page text}
Fields to extract:
- material (string): Material specification
- finish (string): Surface finish
- thread_size (string): Thread size designation
- thread_pitch (string): Thread pitch
- length (string): Fastener length with unit
- head_type (string): Head style
- drive_type (string): Drive type
- strength_grade (string): Strength/property class
```
**Rate limiting:** Queue items and process in batches of 10 with 1s delay
between batches to stay within OpenRouter rate limits. Show progress bar
in the UI.
**Cost control:** Use `openai/gpt-4.1-nano` by default (cheapest). Show
estimated cost before starting batch. Allow model override in settings.
---
## Database Changes
### Phase 1: None
Completeness is computed at query time from existing `items` +
`revisions.properties` data joined against the in-memory schema definition.
No new tables needed for the core feature.
### Phase 4: New table
```sql
CREATE TABLE IF NOT EXISTS audit_snapshots (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
captured_at TIMESTAMPTZ NOT NULL DEFAULT now(),
total_items INTEGER NOT NULL,
avg_score DECIMAL(5,4) NOT NULL,
by_tier JSONB NOT NULL,
by_category JSONB NOT NULL,
by_project JSONB NOT NULL
);
```
### Phase 5: None
AI suggestions are ephemeral (computed per request, not stored). Accepted
suggestions are written through the existing revision/property update path.
---
## Scoring Examples
### Purchased Fastener (F01)
**Weighted total: ~30.5 points**
| Field | Weight | Filled? | Points |
|-------|--------|---------|--------|
| manufacturer_pn | 3 | no | 0/3 |
| sourcing_link | 3 | no | 0/3 |
| manufacturer | 2 | no | 0/2 |
| supplier | 2 | no | 0/2 |
| supplier_pn | 2 | no | 0/2 |
| standard_cost | 2 | yes | 2/2 |
| description | 1 | yes | 1/1 |
| sourcing_type | 1 | yes | 1/1 |
| material | 1 | yes | 1/1 |
| thread_size | 1 | yes | 1/1 |
| length | 1 | yes | 1/1 |
| head_type | 1 | yes | 1/1 |
| drive_type | 1 | no | 0/1 |
| finish | 1 | no | 0/1 |
| ... (remaining) | 0.5-1 | no | 0/... |
**Score: 8/30.5 = 26%** -- "Low" tier, flagged because weight-3 fields
(manufacturer_pn, sourcing_link) are missing.
### Manufactured Assembly (A01)
**Weighted total: ~22 points**
| Field | Weight | Source | Points |
|-------|--------|--------|--------|
| has_bom | 3 | BOM query | 3/3 (12 children) |
| description | 2 | item | 2/2 |
| standard_cost | 2 | computed from children | 2/2 |
| component_count | 1 | computed (= 12) | 1/1 |
| weight | 1 | computed (needs children) | 0/1 (not all children have weight) |
| assembly_time | 1 | property | 0/1 |
| test_procedure | 1 | property | 0/1 |
| dimensions | 1 | property | 0/1 |
| ip_rating | 1 | property | 0/1 |
| ... (globals) | 0.5-1 | property | .../... |
**Score: ~15/22 = 68%** -- "Partial" tier, mostly complete because BOM
and cost are covered through children.
### Motor (R01) -- highest field count
30+ applicable fields across global defaults + motion-specific properties
(load, speed, power, voltage, current, torque, encoder, gear ratio...).
A motor with only description + cost + sourcing_type scores under 10%
because of the large denominator. Motors are the category most likely to
benefit from batch AI extraction from datasheets.