silo/docs/MODULES.md

# Module System Specification

**Status:** Draft
**Last Updated:** 2026-02-14

---

## 1. Purpose

Silo's module system defines the boundary between required infrastructure and optional capabilities. Each module groups a set of API endpoints, UI views, and configuration parameters. Modules can be enabled or disabled at runtime by administrators via the web UI, and clients can query which modules are active to adapt their feature set.

The goal: after initial deployment (where `config.yaml` sets database, storage, and server bind), all further operational configuration happens through the admin settings UI. The YAML file becomes the bootstrap; the database becomes the runtime source of truth.

---

## 2. Module Registry

### 2.1 Required Modules

These cannot be disabled. They define what Silo *is*.

| Module ID | Name | Description |
|-----------|------|-------------|
| `core` | Core PDM | Items, revisions, files, BOM, search, import/export, part number generation |
| `schemas` | Schemas | Part numbering schema parsing, segment management, form descriptors |
| `storage` | Storage | Filesystem storage |

### 2.2 Optional Modules

| Module ID | Name | Default | Description |
|-----------|------|---------|-------------|
| `auth` | Authentication | `true` | Local, LDAP, OIDC authentication and RBAC |
| `projects` | Projects | `true` | Project management and item tagging |
| `audit` | Audit | `true` | Audit logging, completeness scoring |
| `odoo` | Odoo ERP | `false` | Odoo integration (config, sync-log, push/pull) |
| `freecad` | Create Integration | `true` | URI scheme, executable path, client settings |
| `jobs` | Job Queue | `false` | Async compute jobs, runner management |
| `dag` | Dependency DAG | `false` | Feature DAG sync, validation states, interference detection |

### 2.3 Module Dependencies

Some modules require others to function:

| Module | Requires |
|--------|----------|
| `dag` | `jobs` |
| `jobs` | `auth` (runner tokens) |
| `odoo` | `auth` |

When enabling a module, its dependencies are validated. The server rejects enabling `dag` without `jobs`. Disabling a module that others depend on shows a warning listing dependents.

---

## 3. Endpoint-to-Module Mapping

### 3.1 `core` (required)

```
# Health
GET    /health
GET    /ready

# Items
GET    /api/items
GET    /api/items/search
GET    /api/items/by-uuid/{uuid}
GET    /api/items/export.csv
GET    /api/items/template.csv
GET    /api/items/export.ods
GET    /api/items/template.ods
POST   /api/items
POST   /api/items/import
POST   /api/items/import.ods
GET    /api/items/{partNumber}
PUT    /api/items/{partNumber}
DELETE /api/items/{partNumber}

# Revisions
GET    /api/items/{partNumber}/revisions
GET    /api/items/{partNumber}/revisions/compare
GET    /api/items/{partNumber}/revisions/{revision}
POST   /api/items/{partNumber}/revisions
PATCH  /api/items/{partNumber}/revisions/{revision}
POST   /api/items/{partNumber}/revisions/{revision}/rollback

# Files
GET    /api/items/{partNumber}/files
GET    /api/items/{partNumber}/file
GET    /api/items/{partNumber}/file/{revision}
POST   /api/items/{partNumber}/file
POST   /api/items/{partNumber}/files
DELETE /api/items/{partNumber}/files/{fileId}
PUT    /api/items/{partNumber}/thumbnail
POST   /api/uploads/presign

# BOM
GET    /api/items/{partNumber}/bom
GET    /api/items/{partNumber}/bom/expanded
GET    /api/items/{partNumber}/bom/flat
GET    /api/items/{partNumber}/bom/cost
GET    /api/items/{partNumber}/bom/where-used
GET    /api/items/{partNumber}/bom/export.csv
GET    /api/items/{partNumber}/bom/export.ods
POST   /api/items/{partNumber}/bom
POST   /api/items/{partNumber}/bom/import
POST   /api/items/{partNumber}/bom/merge
PUT    /api/items/{partNumber}/bom/{childPartNumber}
DELETE /api/items/{partNumber}/bom/{childPartNumber}

# .kc Metadata
GET    /api/items/{partNumber}/metadata
PUT    /api/items/{partNumber}/metadata
PATCH  /api/items/{partNumber}/metadata/lifecycle
PATCH  /api/items/{partNumber}/metadata/tags

# .kc Dependencies
GET    /api/items/{partNumber}/dependencies
GET    /api/items/{partNumber}/dependencies/resolve

# .kc Macros
GET    /api/items/{partNumber}/macros
GET    /api/items/{partNumber}/macros/{filename}

# Part Number Generation
POST   /api/generate-part-number

# Sheets
POST   /api/sheets/diff

# Settings & Modules (admin)
GET    /api/modules
GET    /api/admin/settings
GET    /api/admin/settings/{module}
PUT    /api/admin/settings/{module}
POST   /api/admin/settings/{module}/test
```

### 3.2 `schemas` (required)

```
GET    /api/schemas
GET    /api/schemas/{name}
GET    /api/schemas/{name}/form
POST   /api/schemas/{name}/segments/{segment}/values
PUT    /api/schemas/{name}/segments/{segment}/values/{code}
DELETE /api/schemas/{name}/segments/{segment}/values/{code}
```

### 3.3 `storage` (required)

No dedicated endpoints — storage is consumed internally by file upload/download in `core`. Exposed through admin settings for connection status visibility.

### 3.4 `auth`

```
# Public (login flow)
GET    /login
POST   /login
POST   /logout
GET    /auth/oidc
GET    /auth/callback

# Authenticated
GET    /api/auth/me
GET    /api/auth/tokens
POST   /api/auth/tokens
DELETE /api/auth/tokens/{id}

# Web UI
GET    /settings                          (account info, tokens)
POST   /settings/tokens
POST   /settings/tokens/{id}/revoke
```

When `auth` is disabled, all routes are open and a synthetic `dev` admin user is injected (current behavior).

### 3.5 `projects`

```
GET    /api/projects
GET    /api/projects/{code}
GET    /api/projects/{code}/items
GET    /api/projects/{code}/sheet.ods
POST   /api/projects
PUT    /api/projects/{code}
DELETE /api/projects/{code}

# Item-project tagging
GET    /api/items/{partNumber}/projects
POST   /api/items/{partNumber}/projects
DELETE /api/items/{partNumber}/projects/{code}
```

When disabled: project tag endpoints return `404`, project columns are hidden in UI list views, project filter is removed from item search.

### 3.6 `audit`

```
GET    /api/audit/completeness
GET    /api/audit/completeness/{partNumber}
```

When disabled: audit log table continues to receive writes (it's part of core middleware), but the completeness scoring endpoints and the Audit page in the web UI are hidden. Future: retention policies, export, and compliance reporting endpoints live here.

### 3.7 `odoo`

```
GET    /api/integrations/odoo/config
GET    /api/integrations/odoo/sync-log
PUT    /api/integrations/odoo/config
POST   /api/integrations/odoo/test-connection
POST   /api/integrations/odoo/sync/push/{partNumber}
POST   /api/integrations/odoo/sync/pull/{odooId}
```

### 3.8 `freecad`

No dedicated API endpoints currently. Configures URI scheme and executable path used by the web UI's "Open in Create" links and by CLI operations. Future: client configuration distribution endpoint.

### 3.9 `jobs`

```
# User-facing
GET    /api/jobs
GET    /api/jobs/{jobID}
GET    /api/jobs/{jobID}/logs
POST   /api/jobs
POST   /api/jobs/{jobID}/cancel

# Job definitions
GET    /api/job-definitions
GET    /api/job-definitions/{name}
POST   /api/job-definitions/reload

# Runner management (admin)
GET    /api/runners
POST   /api/runners
DELETE /api/runners/{runnerID}

# Runner-facing (runner token auth)
POST   /api/runner/heartbeat
POST   /api/runner/claim
PUT    /api/runner/jobs/{jobID}/progress
POST   /api/runner/jobs/{jobID}/complete
POST   /api/runner/jobs/{jobID}/fail
POST   /api/runner/jobs/{jobID}/log
PUT    /api/runner/jobs/{jobID}/dag
```

### 3.10 `dag`

```
GET    /api/items/{partNumber}/dag
GET    /api/items/{partNumber}/dag/forward-cone/{nodeKey}
GET    /api/items/{partNumber}/dag/dirty
PUT    /api/items/{partNumber}/dag
POST   /api/items/{partNumber}/dag/mark-dirty/{nodeKey}
```

---

## 4. Disabled Module Behavior

When a module is disabled:

1. **API routes** registered by that module return `404 Not Found` with body `{"error": "module '<id>' is not enabled"}`.
2. **Web UI** hides the module's navigation entry, page, and any inline UI elements (e.g., project tags on item cards).
3. **SSE events** from the module are not broadcast.
4. **Background goroutines** (e.g., job timeout sweeper, runner heartbeat checker) are not started.
5. **Database tables** are not dropped — they remain for re-enablement. No data loss on disable/enable cycle.

Implementation: each module's route group is wrapped in a middleware check:

```go
func RequireModule(id string) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            if !modules.IsEnabled(id) {
                http.Error(w, `{"error":"module '`+id+`' is not enabled"}`, 404)
                return
            }
            next.ServeHTTP(w, r)
        })
    }
}
```

---

## 5. Configuration Persistence

### 5.1 Precedence

```
Environment variables  (highest — always wins, secrets live here)
        ↓
Database overrides     (admin UI writes here)
        ↓
config.yaml            (lowest — bootstrap defaults)
```

### 5.2 Database Table

```sql
-- Migration 014_settings.sql
CREATE TABLE settings_overrides (
    key         TEXT PRIMARY KEY,          -- dotted path: "auth.ldap.enabled"
    value       JSONB NOT NULL,           -- typed value
    updated_by  TEXT NOT NULL,            -- username
    updated_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE module_state (
    module_id   TEXT PRIMARY KEY,         -- "auth", "projects", etc.
    enabled     BOOLEAN NOT NULL,
    updated_by  TEXT NOT NULL,
    updated_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);
```

### 5.3 Load Sequence

On startup:

1. Parse `config.yaml` into Go config struct.
2. Query `settings_overrides` — merge each key into the struct using dotted path resolution.
3. Apply environment variable overrides (existing `SILO_*` vars).
4. Query `module_state` — override default enabled/disabled from YAML.
5. Validate module dependencies.
6. Register only enabled modules' route groups.
7. Start only enabled modules' background goroutines.

### 5.4 Runtime Updates

When an admin saves settings via `PUT /api/admin/settings/{module}`:

1. Validate the payload against the module's config schema.
2. Write changed keys to `settings_overrides`.
3. Update `module_state` if `enabled` changed.
4. Apply changes to the in-memory config (hot reload where safe).
5. Broadcast `settings.changed` SSE event with `{module, enabled, changed_keys}`.
6. For changes that require restart (e.g., `server.port`, `database.*`), return a `restart_required: true` flag in the response. The UI shows a banner.

### 5.5 What Requires Restart

| Config Area | Hot Reload | Restart Required |
|-------------|-----------|------------------|
| Module enable/disable | Yes | No |
| `auth.*` provider toggles | Yes | No |
| `auth.cors.allowed_origins` | Yes | No |
| `odoo.*` connection settings | Yes | No |
| `freecad.*` | Yes | No |
| `jobs.*` timeouts, directory | Yes | No |
| `server.host`, `server.port` | No | Yes |
| `database.*` | No | Yes |
| `storage.*` | No | Yes |
| `schemas.directory` | No | Yes |

---

## 6. Public Module Discovery Endpoint

```
GET /api/modules
```

**No authentication required.** Clients need this pre-login to know whether OIDC is available, whether projects exist, etc.

### 6.1 Response

```json
{
  "modules": {
    "core": {
      "enabled": true,
      "required": true,
      "name": "Core PDM",
      "version": "0.2"
    },
    "schemas": {
      "enabled": true,
      "required": true,
      "name": "Schemas"
    },
    "storage": {
      "enabled": true,
      "required": true,
      "name": "Storage"
    },
    "auth": {
      "enabled": true,
      "required": false,
      "name": "Authentication",
      "config": {
        "local_enabled": true,
        "ldap_enabled": true,
        "oidc_enabled": true,
        "oidc_issuer_url": "https://keycloak.example.com/realms/silo"
      }
    },
    "projects": {
      "enabled": true,
      "required": false,
      "name": "Projects"
    },
    "audit": {
      "enabled": true,
      "required": false,
      "name": "Audit"
    },
    "odoo": {
      "enabled": false,
      "required": false,
      "name": "Odoo ERP"
    },
    "freecad": {
      "enabled": true,
      "required": false,
      "name": "Create Integration",
      "config": {
        "uri_scheme": "silo"
      }
    },
    "jobs": {
      "enabled": false,
      "required": false,
      "name": "Job Queue"
    },
    "dag": {
      "enabled": false,
      "required": false,
      "name": "Dependency DAG",
      "depends_on": ["jobs"]
    }
  },
  "server": {
    "version": "0.2",
    "read_only": false
  }
}
```

The `config` sub-object exposes only public, non-secret metadata needed by clients. Never includes passwords, tokens, or secret keys.

---

## 7. Admin Settings Endpoints

### 7.1 Get All Settings

```
GET /api/admin/settings
Authorization: Bearer <admin token>
```

Returns full config grouped by module with secrets redacted:

```json
{
  "core": {
    "server": {
      "host": "0.0.0.0",
      "port": 8080,
      "base_url": "https://silo.example.com",
      "read_only": false
    }
  },
  "schemas": {
    "directory": "/etc/silo/schemas",
    "default": "kindred-rd"
  },
  "storage": {
    "backend": "filesystem",
    "filesystem": {
      "root_dir": "/var/lib/silo/data"
    },
    "status": "connected"
  },
  "database": {
    "host": "postgres",
    "port": 5432,
    "name": "silo",
    "user": "silo",
    "password": "****",
    "sslmode": "disable",
    "max_connections": 10,
    "status": "connected"
  },
  "auth": {
    "enabled": true,
    "session_secret": "****",
    "local": { "enabled": true },
    "ldap": {
      "enabled": true,
      "url": "ldaps://ipa.example.com",
      "base_dn": "dc=kindred,dc=internal",
      "user_search_dn": "cn=users,cn=accounts,dc=kindred,dc=internal",
      "bind_password": "****",
      "role_mapping": { "...": "..." }
    },
    "oidc": {
      "enabled": true,
      "issuer_url": "https://keycloak.example.com/realms/silo",
      "client_id": "silo",
      "client_secret": "****",
      "redirect_url": "https://silo.example.com/auth/callback"
    },
    "cors": { "allowed_origins": ["https://silo.example.com"] }
  },
  "projects": { "enabled": true },
  "audit": { "enabled": true },
  "odoo": { "enabled": false, "url": "", "database": "", "username": "" },
  "freecad": { "uri_scheme": "silo", "executable": "" },
  "jobs": {
    "enabled": false,
    "directory": "/etc/silo/jobdefs",
    "runner_timeout": 90,
    "job_timeout_check": 30,
    "default_priority": 100
  },
  "dag": { "enabled": false }
}
```

### 7.2 Get Module Settings

```
GET /api/admin/settings/{module}
```

Returns just the module's config block.

### 7.3 Update Module Settings

```
PUT /api/admin/settings/{module}
Content-Type: application/json

{
  "enabled": true,
  "ldap": {
    "enabled": true,
    "url": "ldaps://ipa.example.com"
  }
}
```

**Response:**

```json
{
  "updated": ["auth.ldap.enabled", "auth.ldap.url"],
  "restart_required": false
}
```

### 7.4 Test Connectivity

```
POST /api/admin/settings/{module}/test
```

Available for modules with external connections:

| Module | Test Action |
|--------|------------|
| `storage` | Verify filesystem storage directory is accessible |
| `auth` (ldap) | Attempt LDAP bind with configured credentials |
| `auth` (oidc) | Fetch OIDC discovery document from issuer URL |
| `odoo` | Attempt XML-RPC connection to Odoo |

**Response:**

```json
{
  "success": true,
  "message": "LDAP bind successful",
  "latency_ms": 42
}
```

---

## 8. Config YAML Changes

The existing `config.yaml` gains a `modules` section. Existing top-level keys remain for backward compatibility — the module system reads from both locations.

```yaml
# Existing keys (unchanged, still work)
server:
  host: "0.0.0.0"
  port: 8080

database:
  host: postgres
  port: 5432
  name: silo
  user: silo
  password: silodev
  sslmode: disable

storage:
  backend: filesystem
  filesystem:
    root_dir: /var/lib/silo/data

schemas:
  directory: /etc/silo/schemas

auth:
  enabled: true
  session_secret: change-me
  local:
    enabled: true

# New: explicit module toggles (optional, defaults shown)
modules:
  projects:
    enabled: true
  audit:
    enabled: true
  odoo:
    enabled: false
  freecad:
    enabled: true
    uri_scheme: silo
  jobs:
    enabled: false
    directory: /etc/silo/jobdefs
    runner_timeout: 90
    job_timeout_check: 30
    default_priority: 100
  dag:
    enabled: false
```

If a module is not listed under `modules:`, its default enabled state from Section 2.2 applies. The `auth.enabled` field continues to control the `auth` module (no duplication under `modules:`).

---

## 9. SSE Events

```
settings.changed    {module, enabled, changed_keys[], updated_by}
```

Broadcast on any admin settings change. The web UI listens for this to:

- Show/hide navigation entries when modules are toggled.
- Display a "Settings updated by another admin" toast.
- Show a "Restart required" banner when flagged.

---

## 10. Web UI — Admin Settings Page

The Settings page (`/settings`) is restructured into sections:

### 10.1 Existing (unchanged)

- **Account** — username, display name, email, auth source, role badge.
- **API Tokens** — create, list, revoke.

### 10.2 New: Module Configuration (admin only)

Visible only to admin users. Each module gets a collapsible card:

```
┌─────────────────────────────────────────────────────┐
│ [toggle] Authentication                    [status] │
├─────────────────────────────────────────────────────┤
│                                                     │
│  ── Local Auth ──────────────────────────────────── │
│  Enabled: [toggle]                                  │
│                                                     │
│  ── LDAP / FreeIPA ──────────────────────────────── │
│  Enabled: [toggle]                                  │
│  URL:     [ldaps://ipa.example.com        ]         │
│  Base DN: [dc=kindred,dc=internal         ]  [Test] │
│                                                     │
│  ── OIDC / Keycloak ────────────────────────────── │
│  Enabled:    [toggle]                               │
│  Issuer URL: [https://keycloak.example.com] [Test]  │
│  Client ID:  [silo                        ]         │
│                                                     │
│  ── CORS ────────────────────────────────────────── │
│  Allowed Origins: [tag input]                       │
│                                                     │
│                                          [Save]     │
└─────────────────────────────────────────────────────┘
```

Module cards for required modules (`core`, `schemas`, `storage`) show their status and config but have no enable/disable toggle.

Status indicators per module:

| Status | Badge | Meaning |
|--------|-------|---------|
| Active | `green` | Enabled and operational |
| Disabled | `overlay1` | Toggled off |
| Error | `red` | Enabled but connectivity or config issue |
| Setup Required | `yellow` | Enabled but missing required config (e.g., LDAP URL empty) |

### 10.3 Infrastructure Section (admin, read-only)

Shows connection status for required infrastructure:

- **Database** — host, port, name, connection pool usage, status badge.
- **Storage** — endpoint, bucket, SSL, status badge.

These are read-only in the UI (setup-only via YAML/env). The "Test" button is available to verify connectivity.

---

## 11. Implementation Order

1. **Migration 014** — `settings_overrides` and `module_state` tables.
2. **Config loader refactor** — YAML → DB merge → env override pipeline.
3. **Module registry** — Go struct defining all modules with metadata, dependencies, defaults.
4. **`GET /api/modules`** — public endpoint, no auth.
5. **`RequireModule` middleware** — gate route groups by module state.
6. **Admin settings API** — `GET/PUT /api/admin/settings/{module}`, test endpoints.
7. **Web UI settings page** — module cards with toggles, config forms, test buttons.
8. **SSE integration** — `settings.changed` event broadcast.

---

## 12. Future Considerations

- **Module manifest format** — per ROADMAP.md, each module will eventually declare routes, views, hooks, and permissions via a manifest. This spec covers the runtime module registry; the manifest format is TBD.
- **Custom modules** — third-party modules that register against the endpoint registry. Requires the manifest contract and a plugin loading mechanism.
- **Per-module permissions** — beyond the current role hierarchy, modules may define fine-grained scopes (e.g., `jobs:admin`, `dag:write`).
- **Location & Inventory module** — when the Location/Inventory API is implemented (tables already exist), it becomes a new optional module.
- **Notifications module** — per ROADMAP.md Tier 1, notifications/subscriptions will be a dedicated module.

---

## 13. References

- [CONFIGURATION.md](CONFIGURATION.md) — Current config reference
- [ROADMAP.md](ROADMAP.md) — Module manifest, API endpoint registry
- [AUTH.md](AUTH.md) — Authentication architecture
- [WORKERS.md](WORKERS.md) — Job queue system
- [DAG.md](DAG.md) — Dependency DAG specification
- [SPECIFICATION.md](SPECIFICATION.md) — Full endpoint listing