diff --git a/.env.example b/.env.example index 8e66aec..8f582f8 100644 --- a/.env.example +++ b/.env.example @@ -1,12 +1,26 @@ # Silo Environment Configuration -# Copy this file to .env and update values as needed +# Copy to .env (or deployments/.env) and update values as needed. +# For automated setup, run: ./scripts/setup-docker.sh # PostgreSQL POSTGRES_PASSWORD=silodev # MinIO -MINIO_ACCESS_KEY=minioadmin -MINIO_SECRET_KEY=minioadmin +MINIO_ACCESS_KEY=silominio +MINIO_SECRET_KEY=silominiosecret -# Silo API (optional overrides) -# SILO_SERVER_PORT=8080 +# OpenLDAP +LDAP_ADMIN_PASSWORD=ldapadmin +LDAP_USERS=siloadmin +LDAP_PASSWORDS=siloadmin + +# Silo Authentication +SILO_SESSION_SECRET=change-me-in-production +SILO_ADMIN_USERNAME=admin +SILO_ADMIN_PASSWORD=admin + +# Optional: OIDC (Keycloak) +# SILO_OIDC_CLIENT_SECRET= + +# Optional: LDAP service account +# SILO_LDAP_BIND_PASSWORD= diff --git a/.gitignore b/.gitignore index 0527ec8..0cbb933 100644 --- a/.gitignore +++ b/.gitignore @@ -29,6 +29,7 @@ Thumbs.db # Config with secrets config.yaml *.env +deployments/config.docker.yaml # Python __pycache__/ diff --git a/README.md b/README.md index 58c2e80..2dce04e 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,7 @@ silo/ │ ├── silo/ # CLI tool │ └── silod/ # API server ├── internal/ -│ ├── api/ # HTTP handlers and routes (75 endpoints) +│ ├── api/ # HTTP handlers and routes (78 endpoints) │ ├── auth/ # Authentication (local, LDAP, OIDC) │ ├── config/ # Configuration loading │ ├── db/ # PostgreSQL repositories @@ -53,15 +53,20 @@ silo/ ## Quick Start -```bash -# Docker Compose (quickest) -cp config.example.yaml config.yaml -# Edit config.yaml with your database, MinIO, and auth settings -make docker-up +See the **[Installation Guide](docs/INSTALL.md)** for complete setup instructions. -# Or manual setup -psql -h localhost -U silo -d silo -f migrations/*.sql -go run ./cmd/silod -config config.yaml +**Docker Compose (quickest — includes PostgreSQL, MinIO, OpenLDAP, and Silo):** + +```bash +./scripts/setup-docker.sh +docker compose -f deployments/docker-compose.allinone.yaml up -d +``` + +**Development (local Go + Docker services):** + +```bash +make docker-up # Start PostgreSQL + MinIO in Docker +make run # Run silo locally with Go ``` When auth is enabled, a default admin account is created on first startup using the credentials in `config.yaml` under `auth.local.default_admin_username` and `auth.local.default_admin_password`. @@ -104,9 +109,10 @@ The server provides the REST API and ODS endpoints consumed by these clients. | Document | Description | |----------|-------------| +| [docs/INSTALL.md](docs/INSTALL.md) | Installation guide (Docker Compose and daemon) | | [docs/SPECIFICATION.md](docs/SPECIFICATION.md) | Full design specification and API reference | | [docs/STATUS.md](docs/STATUS.md) | Implementation status | -| [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) | Production deployment guide | +| [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) | Production deployment and operations guide | | [docs/CONFIGURATION.md](docs/CONFIGURATION.md) | Configuration reference (all `config.yaml` options) | | [docs/AUTH.md](docs/AUTH.md) | Authentication system design | | [docs/AUTH_USER_GUIDE.md](docs/AUTH_USER_GUIDE.md) | User guide for login, tokens, and roles | diff --git a/ROADMAP.md b/ROADMAP.md index 4f28134..dc6db61 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -39,7 +39,7 @@ This document compares Silo's current capabilities against SOLIDWORKS PDM—the ### Implemented Features (MVP Complete) #### Core Database System -- PostgreSQL schema with 11 migrations +- PostgreSQL schema with 13 migrations - UUID-based identifiers throughout - Soft delete support via `archived_at` timestamps - Atomic sequence generation for part numbers @@ -92,7 +92,7 @@ This document compares Silo's current capabilities against SOLIDWORKS PDM—the - Template generation for import formatting #### API & Web Interface -- REST API with 75 endpoints +- REST API with 78 endpoints - Authentication: local (bcrypt), LDAP/FreeIPA, OIDC/Keycloak - Role-based access control (admin > editor > viewer) - API token management (SHA-256 hashed) @@ -129,7 +129,7 @@ This document compares Silo's current capabilities against SOLIDWORKS PDM—the | Component | Status | |-----------|--------| -| PostgreSQL | Running (psql.kindred.internal) | +| PostgreSQL | Running (psql.example.internal) | | MinIO | Configured in Docker Compose | | Silo API Server | Builds successfully | | Docker Compose | Complete (dev and production) | @@ -255,14 +255,14 @@ CAD integration is maintained in separate repositories ([silo-mod](https://git.k | Feature | SOLIDWORKS PDM | Silo Status | Priority | Complexity | |---------|---------------|-------------|----------|------------| | ERP integration | SAP, Dynamics, etc. | Partial (Odoo stubs) | Medium | Complex | -| API access | Full COM/REST API | Full REST API (75 endpoints) | - | - | +| API access | Full COM/REST API | Full REST API (78 endpoints) | - | - | | Dispatch scripts | Automation without coding | None | Medium | Moderate | | Task scheduler | Background processing | None | Medium | Moderate | | Email system | SMTP integration | None | High | Simple | | Web portal | Browser access | Full (React SPA + auth) | - | - | **Gap Analysis:** -Silo has a comprehensive REST API (75 endpoints) and a full web UI with authentication. Odoo ERP integration has config/sync-log scaffolding but push/pull operations are stubs. Remaining gaps: email notifications, task scheduler, dispatch automation. +Silo has a comprehensive REST API (78 endpoints) and a full web UI with authentication. Odoo ERP integration has config/sync-log scaffolding but push/pull operations are stubs. Remaining gaps: email notifications, task scheduler, dispatch automation. --- diff --git a/cmd/silo/main.go b/cmd/silo/main.go index 546ceb3..36c6c7d 100644 --- a/cmd/silo/main.go +++ b/cmd/silo/main.go @@ -66,7 +66,7 @@ Token subcommands: silo token revoke Revoke a token Environment variables for API access: - SILO_API_URL Base URL of the Silo server (e.g., https://silo.kindred.internal) + SILO_API_URL Base URL of the Silo server (e.g., https://silo.example.internal) SILO_API_TOKEN API token for authentication Examples: diff --git a/config.example.yaml b/config.example.yaml index c9089c6..74f1f89 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -8,20 +8,20 @@ server: # read_only: false # Reject all write operations; toggle at runtime with SIGUSR1 database: - host: "psql.kindred.internal" + host: "localhost" # Use "postgres" for Docker Compose port: 5432 name: "silo" user: "silo" password: "" # Use SILO_DB_PASSWORD env var - sslmode: "require" + sslmode: "require" # Use "disable" for Docker Compose (internal network) max_connections: 10 storage: - endpoint: "minio.kindred.internal:9000" + endpoint: "localhost:9000" # Use "minio:9000" for Docker Compose access_key: "" # Use SILO_MINIO_ACCESS_KEY env var secret_key: "" # Use SILO_MINIO_SECRET_KEY env var bucket: "silo-files" - use_ssl: true + use_ssl: true # Use false for Docker Compose (internal network) region: "us-east-1" schemas: @@ -53,7 +53,7 @@ auth: # LDAP / FreeIPA ldap: enabled: false - url: "ldaps://ipa.kindred.internal" + url: "ldaps://ipa.example.internal" base_dn: "dc=kindred,dc=internal" user_search_dn: "cn=users,cn=accounts,dc=kindred,dc=internal" # Optional service account for user search (omit for direct user bind) @@ -77,10 +77,10 @@ auth: # OIDC / Keycloak oidc: enabled: false - issuer_url: "https://keycloak.kindred.internal/realms/silo" + issuer_url: "https://keycloak.example.internal/realms/silo" client_id: "silo" client_secret: "" # Use SILO_OIDC_CLIENT_SECRET env var - redirect_url: "https://silo.kindred.internal/auth/callback" + redirect_url: "https://silo.example.internal/auth/callback" scopes: ["openid", "profile", "email"] # Map Keycloak realm roles to Silo roles admin_role: "silo-admin" @@ -90,4 +90,4 @@ auth: # CORS origins (locked down when auth is enabled) cors: allowed_origins: - - "https://silo.kindred.internal" + - "https://silo.example.internal" diff --git a/deployments/config.dev.yaml b/deployments/config.dev.yaml new file mode 100644 index 0000000..9f57d05 --- /dev/null +++ b/deployments/config.dev.yaml @@ -0,0 +1,35 @@ +# Silo Development Configuration +# Used by deployments/docker-compose.yaml — works with zero setup via `make docker-up`. +# For production Docker installs, run scripts/setup-docker.sh instead. + +server: + host: "0.0.0.0" + port: 8080 + base_url: "http://localhost:8080" + +database: + host: "postgres" + port: 5432 + name: "silo" + user: "silo" + password: "${POSTGRES_PASSWORD:-silodev}" + sslmode: "disable" + max_connections: 10 + +storage: + endpoint: "minio:9000" + access_key: "${MINIO_ACCESS_KEY:-silominio}" + secret_key: "${MINIO_SECRET_KEY:-silominiosecret}" + bucket: "silo-files" + use_ssl: false + region: "us-east-1" + +schemas: + directory: "/etc/silo/schemas" + default: "kindred-rd" + +freecad: + uri_scheme: "silo" + +auth: + enabled: false diff --git a/deployments/config.prod.yaml b/deployments/config.prod.yaml index cd0508c..fefaf38 100644 --- a/deployments/config.prod.yaml +++ b/deployments/config.prod.yaml @@ -1,7 +1,7 @@ # Silo Production Configuration # Single-binary deployment: silod serves API + React SPA # -# Layout on silo.kindred.internal: +# Layout on silo.example.internal: # /opt/silo/bin/silod - server binary # /opt/silo/web/dist/ - built React frontend (served automatically) # /opt/silo/schemas/ - part number schemas @@ -18,10 +18,10 @@ server: host: "0.0.0.0" port: 8080 - base_url: "https://silo.kindred.internal" + base_url: "https://silo.example.internal" database: - host: "psql.kindred.internal" + host: "psql.example.internal" port: 5432 name: "silo" user: "silo" @@ -30,7 +30,7 @@ database: max_connections: 20 storage: - endpoint: "minio.kindred.internal:9000" + endpoint: "minio.example.internal:9000" access_key: "" # Set via SILO_MINIO_ACCESS_KEY secret_key: "" # Set via SILO_MINIO_SECRET_KEY bucket: "silo-files" @@ -53,7 +53,7 @@ auth: default_admin_password: "" # Set via SILO_ADMIN_PASSWORD ldap: enabled: true - url: "ldaps://ipa.kindred.internal" + url: "ldaps://ipa.example.internal" base_dn: "dc=kindred,dc=internal" user_search_dn: "cn=users,cn=accounts,dc=kindred,dc=internal" user_attr: "uid" @@ -73,4 +73,4 @@ auth: enabled: false cors: allowed_origins: - - "https://silo.kindred.internal" + - "https://silo.example.internal" diff --git a/deployments/docker-compose.allinone.yaml b/deployments/docker-compose.allinone.yaml new file mode 100644 index 0000000..6d73fc8 --- /dev/null +++ b/deployments/docker-compose.allinone.yaml @@ -0,0 +1,172 @@ +# Silo All-in-One Stack +# PostgreSQL + MinIO + OpenLDAP + Silo API + Nginx (optional) +# +# Quick start: +# ./scripts/setup-docker.sh +# docker compose -f deployments/docker-compose.allinone.yaml up -d +# +# With nginx reverse proxy: +# docker compose -f deployments/docker-compose.allinone.yaml --profile nginx up -d +# +# View logs: +# docker compose -f deployments/docker-compose.allinone.yaml logs -f +# +# Stop: +# docker compose -f deployments/docker-compose.allinone.yaml down +# +# Stop and delete data: +# docker compose -f deployments/docker-compose.allinone.yaml down -v + +services: + # --------------------------------------------------------------------------- + # PostgreSQL 16 + # --------------------------------------------------------------------------- + postgres: + image: postgres:16-alpine + container_name: silo-postgres + restart: unless-stopped + environment: + POSTGRES_DB: silo + POSTGRES_USER: silo + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Run ./scripts/setup-docker.sh first} + volumes: + - postgres_data:/var/lib/postgresql/data + - ../migrations:/docker-entrypoint-initdb.d:ro + healthcheck: + test: ["CMD-SHELL", "pg_isready -U silo -d silo"] + interval: 5s + timeout: 5s + retries: 5 + networks: + - silo-net + + # --------------------------------------------------------------------------- + # MinIO (S3-compatible object storage) + # --------------------------------------------------------------------------- + minio: + image: minio/minio:latest + container_name: silo-minio + restart: unless-stopped + command: server /data --console-address ":9001" + environment: + MINIO_ROOT_USER: ${MINIO_ACCESS_KEY:?Run ./scripts/setup-docker.sh first} + MINIO_ROOT_PASSWORD: ${MINIO_SECRET_KEY:?Run ./scripts/setup-docker.sh first} + volumes: + - minio_data:/data + ports: + - "9001:9001" # MinIO console (remove in hardened setups) + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] + interval: 10s + timeout: 5s + retries: 5 + networks: + - silo-net + + # --------------------------------------------------------------------------- + # OpenLDAP (user directory for LDAP authentication) + # --------------------------------------------------------------------------- + openldap: + image: bitnami/openldap:2.6 + container_name: silo-openldap + restart: unless-stopped + environment: + LDAP_ROOT: "dc=silo,dc=local" + LDAP_ADMIN_USERNAME: "admin" + LDAP_ADMIN_PASSWORD: ${LDAP_ADMIN_PASSWORD:?Run ./scripts/setup-docker.sh first} + LDAP_USERS: ${LDAP_USERS:-siloadmin} + LDAP_PASSWORDS: ${LDAP_PASSWORDS:?Run ./scripts/setup-docker.sh first} + LDAP_GROUP: "silo-users" + LDAP_USER_OU: "users" + LDAP_GROUP_OU: "groups" + volumes: + - openldap_data:/bitnami/openldap + - ./ldap:/docker-entrypoint-initdb.d:ro + ports: + - "1389:1389" # LDAP access for debugging (remove in hardened setups) + healthcheck: + test: ["CMD-SHELL", "ldapsearch -x -H ldap://localhost:1389 -b dc=silo,dc=local -D cn=admin,dc=silo,dc=local -w $${LDAP_ADMIN_PASSWORD} '(objectClass=organization)' >/dev/null 2>&1"] + interval: 10s + timeout: 5s + retries: 5 + networks: + - silo-net + + # --------------------------------------------------------------------------- + # Silo API Server + # --------------------------------------------------------------------------- + silo: + build: + context: .. + dockerfile: build/package/Dockerfile + container_name: silo-api + restart: unless-stopped + depends_on: + postgres: + condition: service_healthy + minio: + condition: service_healthy + openldap: + condition: service_healthy + env_file: + - .env + environment: + # These override values in config.docker.yaml via the Go config loader's + # direct env var support (see internal/config/config.go). + SILO_DB_HOST: postgres + SILO_DB_NAME: silo + SILO_DB_USER: silo + SILO_DB_PASSWORD: ${POSTGRES_PASSWORD} + SILO_MINIO_ENDPOINT: minio:9000 + SILO_MINIO_ACCESS_KEY: ${MINIO_ACCESS_KEY} + SILO_MINIO_SECRET_KEY: ${MINIO_SECRET_KEY} + ports: + - "${SILO_PORT:-8080}:8080" + volumes: + - ../schemas:/etc/silo/schemas:ro + - ./config.docker.yaml:/etc/silo/config.yaml:ro + healthcheck: + test: ["CMD", "wget", "-qO-", "http://localhost:8080/health"] + interval: 10s + timeout: 5s + retries: 3 + start_period: 15s + networks: + - silo-net + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "3" + + # --------------------------------------------------------------------------- + # Nginx reverse proxy (optional — enable with --profile nginx) + # --------------------------------------------------------------------------- + nginx: + image: nginx:alpine + container_name: silo-nginx + restart: unless-stopped + profiles: + - nginx + depends_on: + silo: + condition: service_healthy + ports: + - "80:80" + - "443:443" + volumes: + - ./nginx/nginx.conf:/etc/nginx/conf.d/default.conf:ro + # Uncomment to mount TLS certificates: + # - /path/to/cert.pem:/etc/nginx/ssl/cert.pem:ro + # - /path/to/key.pem:/etc/nginx/ssl/key.pem:ro + networks: + - silo-net + +volumes: + postgres_data: + minio_data: + openldap_data: + +networks: + silo-net: + driver: bridge diff --git a/deployments/docker-compose.prod.yaml b/deployments/docker-compose.prod.yaml index 104c54b..88b7175 100644 --- a/deployments/docker-compose.prod.yaml +++ b/deployments/docker-compose.prod.yaml @@ -1,5 +1,5 @@ # Production Docker Compose for Silo -# Uses external PostgreSQL (psql.kindred.internal) and MinIO (minio.kindred.internal) +# Uses external PostgreSQL (psql.example.internal) and MinIO (minio.example.internal) # # Usage: # export SILO_DB_PASSWORD= @@ -15,23 +15,23 @@ services: container_name: silod restart: unless-stopped environment: - # Database connection (psql.kindred.internal) - SILO_DB_HOST: psql.kindred.internal - SILO_DB_PORT: 5432 + # Database connection (psql.example.internal) + # Supported as direct env var overrides in the Go config loader: + SILO_DB_HOST: psql.example.internal SILO_DB_NAME: silo SILO_DB_USER: silo SILO_DB_PASSWORD: ${SILO_DB_PASSWORD:?Database password required} - SILO_DB_SSLMODE: require + # Note: SILO_DB_PORT and SILO_DB_SSLMODE are NOT supported as direct + # env var overrides. Set these in config.yaml instead, or use ${VAR} + # syntax in the YAML file. See docs/CONFIGURATION.md for details. - # MinIO storage (minio.kindred.internal) - SILO_MINIO_ENDPOINT: minio.kindred.internal:9000 + # MinIO storage (minio.example.internal) + # Supported as direct env var overrides: + SILO_MINIO_ENDPOINT: minio.example.internal:9000 SILO_MINIO_ACCESS_KEY: ${SILO_MINIO_ACCESS_KEY:?MinIO access key required} SILO_MINIO_SECRET_KEY: ${SILO_MINIO_SECRET_KEY:?MinIO secret key required} - SILO_MINIO_BUCKET: silo-files - SILO_MINIO_USE_SSL: "true" - - # Server settings - SILO_SERVER_BASE_URL: ${SILO_BASE_URL:-http://silo.kindred.internal:8080} + # Note: SILO_MINIO_BUCKET and SILO_MINIO_USE_SSL are NOT supported as + # direct env var overrides. Set these in config.yaml instead. ports: - "8080:8080" volumes: diff --git a/deployments/docker-compose.yaml b/deployments/docker-compose.yaml index d733fe1..c13425b 100644 --- a/deployments/docker-compose.yaml +++ b/deployments/docker-compose.yaml @@ -69,7 +69,7 @@ services: - "8080:8080" volumes: - ../schemas:/etc/silo/schemas:ro - - ../configs/config.yaml:/etc/silo/config.yaml:ro + - ./config.dev.yaml:/etc/silo/config.yaml:ro healthcheck: test: ["CMD", "wget", "-qO-", "http://localhost:8080/health"] interval: 10s diff --git a/deployments/ldap/memberof.ldif b/deployments/ldap/memberof.ldif new file mode 100644 index 0000000..533b79b --- /dev/null +++ b/deployments/ldap/memberof.ldif @@ -0,0 +1,36 @@ +# Enable the memberOf overlay for OpenLDAP. +# When a user is added to a groupOfNames, their entry automatically +# gets a memberOf attribute pointing to the group DN. +# This is required for Silo's LDAP role mapping. +# +# Loaded automatically by bitnami/openldap from /docker-entrypoint-initdb.d/ + +dn: cn=module{0},cn=config +changetype: modify +add: olcModuleLoad +olcModuleLoad: memberof + +dn: olcOverlay=memberof,olcDatabase={2}mdb,cn=config +changetype: add +objectClass: olcOverlayConfig +objectClass: olcMemberOf +olcOverlay: memberof +olcMemberOfRefInt: TRUE +olcMemberOfDangling: ignore +olcMemberOfGroupOC: groupOfNames +olcMemberOfMemberAD: member +olcMemberOfMemberOfAD: memberOf + +# Enable refint overlay to maintain referential integrity +# (removes memberOf when a user is removed from a group) +dn: cn=module{0},cn=config +changetype: modify +add: olcModuleLoad +olcModuleLoad: refint + +dn: olcOverlay=refint,olcDatabase={2}mdb,cn=config +changetype: add +objectClass: olcOverlayConfig +objectClass: olcRefintConfig +olcOverlay: refint +olcRefintAttribute: memberOf member diff --git a/deployments/ldap/silo-groups.ldif b/deployments/ldap/silo-groups.ldif new file mode 100644 index 0000000..0876f92 --- /dev/null +++ b/deployments/ldap/silo-groups.ldif @@ -0,0 +1,34 @@ +# Create Silo role groups for LDAP-based access control. +# These groups map to Silo roles via auth.ldap.role_mapping in config.yaml. +# +# Group hierarchy: +# silo-admins -> admin role (full access) +# silo-users -> editor role (create/modify items) +# silo-viewers -> viewer role (read-only) +# +# The initial LDAP user (set via LDAP_USERS env var) is added to silo-admins. +# Additional users can be added with ldapadd or ldapmodify. +# +# Loaded automatically by bitnami/openldap from /docker-entrypoint-initdb.d/ +# Note: This runs after the default tree is created (users/groups OUs exist). + +# Admin group — initial user is a member +dn: cn=silo-admins,ou=groups,dc=silo,dc=local +objectClass: groupOfNames +cn: silo-admins +description: Silo administrators (full access) +member: cn=siloadmin,ou=users,dc=silo,dc=local + +# Editor group +dn: cn=silo-users,ou=groups,dc=silo,dc=local +objectClass: groupOfNames +cn: silo-users +description: Silo editors (create and modify items) +member: cn=placeholder,ou=users,dc=silo,dc=local + +# Viewer group +dn: cn=silo-viewers,ou=groups,dc=silo,dc=local +objectClass: groupOfNames +cn: silo-viewers +description: Silo viewers (read-only access) +member: cn=placeholder,ou=users,dc=silo,dc=local diff --git a/deployments/nginx/nginx-nossl.conf b/deployments/nginx/nginx-nossl.conf new file mode 100644 index 0000000..52c5fa9 --- /dev/null +++ b/deployments/nginx/nginx-nossl.conf @@ -0,0 +1,44 @@ +# Silo Nginx Reverse Proxy — HTTP only (no TLS) +# +# Use this when TLS is terminated by an external load balancer or when +# running on a trusted internal network without HTTPS. + +upstream silo_backend { + server silo:8080; +} + +server { + listen 80; + listen [::]:80; + server_name _; + + location / { + proxy_pass http://silo_backend; + proxy_http_version 1.1; + + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Port $server_port; + + # SSE support + proxy_set_header Connection ""; + proxy_buffering off; + + # Timeouts + proxy_connect_timeout 60s; + proxy_send_timeout 60s; + proxy_read_timeout 300s; + + # File uploads (CAD files can be large) + client_max_body_size 100M; + } + + location /nginx-health { + access_log off; + return 200 "OK\n"; + add_header Content-Type text/plain; + } +} diff --git a/deployments/nginx/nginx.conf b/deployments/nginx/nginx.conf new file mode 100644 index 0000000..6c48500 --- /dev/null +++ b/deployments/nginx/nginx.conf @@ -0,0 +1,103 @@ +# Silo Nginx Reverse Proxy (Docker) +# +# HTTP reverse proxy with optional HTTPS. To enable TLS: +# 1. Uncomment the ssl server block below +# 2. Mount your certificate and key in docker-compose: +# volumes: +# - /path/to/cert.pem:/etc/nginx/ssl/cert.pem:ro +# - /path/to/key.pem:/etc/nginx/ssl/key.pem:ro +# 3. Uncomment the HTTP-to-HTTPS redirect in the port 80 block + +upstream silo_backend { + server silo:8080; +} + +# HTTP server +server { + listen 80; + listen [::]:80; + server_name _; + + # Uncomment the next line to redirect all HTTP traffic to HTTPS + # return 301 https://$host$request_uri; + + location / { + proxy_pass http://silo_backend; + proxy_http_version 1.1; + + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Port $server_port; + + # SSE support + proxy_set_header Connection ""; + proxy_buffering off; + + # Timeouts + proxy_connect_timeout 60s; + proxy_send_timeout 60s; + proxy_read_timeout 300s; + + # File uploads (CAD files can be large) + client_max_body_size 100M; + } + + # Health check endpoint for monitoring + location /nginx-health { + access_log off; + return 200 "OK\n"; + add_header Content-Type text/plain; + } +} + +# Uncomment for HTTPS (mount certs in docker-compose volumes) +# server { +# listen 443 ssl http2; +# listen [::]:443 ssl http2; +# server_name _; +# +# ssl_certificate /etc/nginx/ssl/cert.pem; +# ssl_certificate_key /etc/nginx/ssl/key.pem; +# +# ssl_protocols TLSv1.2 TLSv1.3; +# ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305; +# ssl_prefer_server_ciphers off; +# ssl_session_timeout 1d; +# ssl_session_cache shared:SSL:10m; +# ssl_session_tickets off; +# +# # Security headers +# add_header X-Frame-Options "SAMEORIGIN" always; +# add_header X-Content-Type-Options "nosniff" always; +# add_header Referrer-Policy "strict-origin-when-cross-origin" always; +# +# location / { +# proxy_pass http://silo_backend; +# proxy_http_version 1.1; +# +# proxy_set_header Host $host; +# proxy_set_header X-Real-IP $remote_addr; +# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; +# proxy_set_header X-Forwarded-Proto $scheme; +# proxy_set_header X-Forwarded-Host $host; +# proxy_set_header X-Forwarded-Port $server_port; +# +# proxy_set_header Connection ""; +# proxy_buffering off; +# +# proxy_connect_timeout 60s; +# proxy_send_timeout 60s; +# proxy_read_timeout 300s; +# +# client_max_body_size 100M; +# } +# +# location /nginx-health { +# access_log off; +# return 200 "OK\n"; +# add_header Content-Type text/plain; +# } +# } diff --git a/deployments/systemd/silod.env.example b/deployments/systemd/silod.env.example index ff50644..c6e4c4f 100644 --- a/deployments/systemd/silod.env.example +++ b/deployments/systemd/silod.env.example @@ -2,11 +2,11 @@ # Copy to /etc/silo/silod.env and fill in values # Permissions: chmod 600 /etc/silo/silod.env -# Database credentials (psql.kindred.internal) +# Database credentials (psql.example.internal) # Database: silo, User: silo SILO_DB_PASSWORD= -# MinIO credentials (minio.kindred.internal) +# MinIO credentials (minio.example.internal) # User: silouser SILO_MINIO_ACCESS_KEY=silouser SILO_MINIO_SECRET_KEY= @@ -23,4 +23,4 @@ SILO_ADMIN_PASSWORD= # SILO_LDAP_BIND_PASSWORD= # Optional: Override server base URL -# SILO_SERVER_BASE_URL=http://silo.kindred.internal:8080 +# SILO_SERVER_BASE_URL=http://silo.example.internal:8080 diff --git a/docs/AUTH_USER_GUIDE.md b/docs/AUTH_USER_GUIDE.md index 6968700..a34dc98 100644 --- a/docs/AUTH_USER_GUIDE.md +++ b/docs/AUTH_USER_GUIDE.md @@ -38,7 +38,7 @@ API tokens allow the FreeCAD plugin, scripts, and CI pipelines to authenticate w ### Creating a Token (CLI) ```sh -export SILO_API_URL=https://silo.kindred.internal +export SILO_API_URL=https://silo.example.internal export SILO_API_TOKEN=silo_ silo token create --name "CI pipeline" @@ -140,7 +140,7 @@ auth: ldap: enabled: true - url: "ldaps://ipa.kindred.internal" + url: "ldaps://ipa.example.internal" base_dn: "dc=kindred,dc=internal" user_search_dn: "cn=users,cn=accounts,dc=kindred,dc=internal" user_attr: "uid" @@ -170,10 +170,10 @@ auth: oidc: enabled: true - issuer_url: "https://keycloak.kindred.internal/realms/silo" + issuer_url: "https://keycloak.example.internal/realms/silo" client_id: "silo" client_secret: "" # Set via SILO_OIDC_CLIENT_SECRET - redirect_url: "https://silo.kindred.internal/auth/callback" + redirect_url: "https://silo.example.internal/auth/callback" scopes: ["openid", "profile", "email"] admin_role: "silo-admin" editor_role: "silo-editor" @@ -186,7 +186,7 @@ auth: auth: cors: allowed_origins: - - "https://silo.kindred.internal" + - "https://silo.example.internal" ``` ## Environment Variables @@ -254,4 +254,4 @@ UPDATE users SET password_hash = '', is_active = true WHERE usernam - Verify the token is set in FreeCAD preferences or `SILO_API_TOKEN` - Check the API URL points to the correct server -- Test with curl: `curl -H "Authorization: Bearer silo_..." https://silo.kindred.internal/api/items` +- Test with curl: `curl -H "Authorization: Bearer silo_..." https://silo.example.internal/api/items` diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 04f449e..d6e2e3f 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -1,5 +1,9 @@ # Silo Production Deployment Guide +> **First-time setup?** See the [Installation Guide](INSTALL.md) for step-by-step +> instructions. This document covers ongoing maintenance and operations for an +> existing deployment. + This guide covers deploying Silo to a dedicated VM using external PostgreSQL and MinIO services. ## Table of Contents @@ -17,7 +21,7 @@ This guide covers deploying Silo to a dedicated VM using external PostgreSQL and ``` ┌─────────────────────────────────────────────────────────────────┐ -│ silo.kindred.internal │ +│ silo.example.internal │ │ ┌───────────────────────────────────────────────────────────┐ │ │ │ silod │ │ │ │ (Silo API Server) │ │ @@ -27,7 +31,7 @@ This guide covers deploying Silo to a dedicated VM using external PostgreSQL and │ │ ▼ ▼ ┌─────────────────────────┐ ┌─────────────────────────────────┐ -│ psql.kindred.internal │ │ minio.kindred.internal │ +│ psql.example.internal │ │ minio.example.internal │ │ PostgreSQL 16 │ │ MinIO S3 │ │ :5432 │ │ :9000 (API) │ │ │ │ :9001 (Console) │ @@ -40,8 +44,8 @@ The following external services are already configured: | Service | Host | Database/Bucket | User | |---------|------|-----------------|------| -| PostgreSQL | psql.kindred.internal:5432 | silo | silo | -| MinIO | minio.kindred.internal:9000 | silo-files | silouser | +| PostgreSQL | psql.example.internal:5432 | silo | silo | +| MinIO | minio.example.internal:9000 | silo-files | silouser | Migrations have been applied to the database. @@ -53,10 +57,10 @@ For a fresh VM, run these commands: ```bash # 1. SSH to the target host -ssh root@silo.kindred.internal +ssh root@silo.example.internal # 2. Download and run setup script -curl -fsSL https://gitea.kindred.internal/kindred/silo-0062/raw/branch/main/scripts/setup-host.sh | bash +curl -fsSL https://git.kindred-systems.com/kindred/silo/raw/branch/main/scripts/setup-host.sh | bash # 3. Configure credentials nano /etc/silo/silod.env @@ -69,16 +73,16 @@ nano /etc/silo/silod.env ## Initial Setup -Run the setup script once on `silo.kindred.internal` to prepare the host: +Run the setup script once on `silo.example.internal` to prepare the host: ```bash # Option 1: If you have the repo locally -scp scripts/setup-host.sh root@silo.kindred.internal:/tmp/ -ssh root@silo.kindred.internal 'bash /tmp/setup-host.sh' +scp scripts/setup-host.sh root@silo.example.internal:/tmp/ +ssh root@silo.example.internal 'bash /tmp/setup-host.sh' # Option 2: Direct on the host -ssh root@silo.kindred.internal -curl -fsSL https://git.kindred.internal/kindred/silo/raw/branch/main/scripts/setup-host.sh -o /tmp/setup-host.sh +ssh root@silo.example.internal +curl -fsSL https://git.kindred-systems.com/kindred/silo/raw/branch/main/scripts/setup-host.sh -o /tmp/setup-host.sh bash /tmp/setup-host.sh ``` @@ -100,10 +104,10 @@ sudo nano /etc/silo/silod.env Fill in the values: ```bash -# Database credentials (psql.kindred.internal) +# Database credentials (psql.example.internal) SILO_DB_PASSWORD=your-database-password -# MinIO credentials (minio.kindred.internal) +# MinIO credentials (minio.example.internal) SILO_MINIO_ACCESS_KEY=silouser SILO_MINIO_SECRET_KEY=your-minio-secret-key ``` @@ -114,10 +118,10 @@ Before deploying, verify connectivity to external services: ```bash # Test PostgreSQL -psql -h psql.kindred.internal -U silo -d silo -c 'SELECT 1' +psql -h psql.example.internal -U silo -d silo -c 'SELECT 1' # Test MinIO -curl -I http://minio.kindred.internal:9000/minio/health/live +curl -I http://minio.example.internal:9000/minio/health/live ``` --- @@ -129,7 +133,7 @@ curl -I http://minio.kindred.internal:9000/minio/health/live To deploy or update Silo, run the deploy script on the target host: ```bash -ssh root@silo.kindred.internal +ssh root@silo.example.internal /opt/silo/src/scripts/deploy.sh ``` @@ -165,7 +169,7 @@ sudo /opt/silo/src/scripts/deploy.sh --status You can override the git repository URL and branch: ```bash -export SILO_REPO_URL=https://git.kindred.internal/kindred/silo.git +export SILO_REPO_URL=https://git.kindred-systems.com/kindred/silo.git export SILO_BRANCH=main sudo -E /opt/silo/src/scripts/deploy.sh ``` @@ -247,7 +251,7 @@ curl http://localhost:8080/ready To update to the latest version: ```bash -ssh root@silo.kindred.internal +ssh root@silo.example.internal /opt/silo/src/scripts/deploy.sh ``` @@ -269,7 +273,7 @@ When new migrations are added, run them manually: ls -la /opt/silo/src/migrations/ # Run a specific migration -psql -h psql.kindred.internal -U silo -d silo -f /opt/silo/src/migrations/008_new_feature.sql +psql -h psql.example.internal -U silo -d silo -f /opt/silo/src/migrations/008_new_feature.sql ``` --- @@ -303,13 +307,13 @@ psql -h psql.kindred.internal -U silo -d silo -f /opt/silo/src/migrations/008_ne 1. Test network connectivity: ```bash - nc -zv psql.kindred.internal 5432 + nc -zv psql.example.internal 5432 ``` 2. Test credentials: ```bash source /etc/silo/silod.env - PGPASSWORD=$SILO_DB_PASSWORD psql -h psql.kindred.internal -U silo -d silo -c 'SELECT 1' + PGPASSWORD=$SILO_DB_PASSWORD psql -h psql.example.internal -U silo -d silo -c 'SELECT 1' ``` 3. Check `pg_hba.conf` on PostgreSQL server allows connections from this host. @@ -318,12 +322,12 @@ psql -h psql.kindred.internal -U silo -d silo -f /opt/silo/src/migrations/008_ne 1. Test network connectivity: ```bash - nc -zv minio.kindred.internal 9000 + nc -zv minio.example.internal 9000 ``` 2. Test with curl: ```bash - curl -I http://minio.kindred.internal:9000/minio/health/live + curl -I http://minio.example.internal:9000/minio/health/live ``` 3. Check SSL settings in config match MinIO setup: @@ -340,8 +344,8 @@ curl -v http://localhost:8080/health curl -v http://localhost:8080/ready # If ready fails but health passes, check external services -psql -h psql.kindred.internal -U silo -d silo -c 'SELECT 1' -curl http://minio.kindred.internal:9000/minio/health/live +psql -h psql.example.internal -U silo -d silo -c 'SELECT 1' +curl http://minio.example.internal:9000/minio/health/live ``` ### Build Fails @@ -391,14 +395,14 @@ This script: getcert list ``` -2. The silo config is already updated to use `https://silo.kindred.internal` as base URL. Restart silo: +2. The silo config is already updated to use `https://silo.example.internal` as base URL. Restart silo: ```bash sudo systemctl restart silod ``` 3. Test the setup: ```bash - curl https://silo.kindred.internal/health + curl https://silo.example.internal/health ``` ### Certificate Management @@ -422,7 +426,7 @@ For clients to trust the Silo HTTPS certificate, they need the IPA CA: ```bash # Download CA cert -curl -o /tmp/ipa-ca.crt https://ipa.kindred.internal/ipa/config/ca.crt +curl -o /tmp/ipa-ca.crt https://ipa.example.internal/ipa/config/ca.crt # Ubuntu/Debian sudo cp /tmp/ipa-ca.crt /usr/local/share/ca-certificates/ipa-ca.crt diff --git a/docs/GAP_ANALYSIS.md b/docs/GAP_ANALYSIS.md index a347833..bf0c0c7 100644 --- a/docs/GAP_ANALYSIS.md +++ b/docs/GAP_ANALYSIS.md @@ -365,7 +365,7 @@ internal/ handlers.go # Items, schemas, projects, revisions middleware.go # Auth middleware odoo_handlers.go # Odoo integration endpoints - routes.go # Route registration (75 endpoints) + routes.go # Route registration (78 endpoints) search.go # Fuzzy search auth/ auth.go # Auth service: local, LDAP, OIDC diff --git a/docs/INSTALL.md b/docs/INSTALL.md new file mode 100644 index 0000000..9429928 --- /dev/null +++ b/docs/INSTALL.md @@ -0,0 +1,518 @@ +# Installing Silo + +This guide covers two installation methods: + +- **[Option A: Docker Compose](#option-a-docker-compose)** — self-contained stack with all services. Recommended for evaluation, small teams, and environments where Docker is the standard. +- **[Option B: Daemon Install](#option-b-daemon-install-systemd--external-services)** — systemd service with external PostgreSQL, MinIO, and optional LDAP/nginx. Recommended for production deployments integrated with existing infrastructure. + +Both methods produce the same result: a running Silo server with a web UI, REST API, and authentication. + +--- + +## Table of Contents + +- [Prerequisites](#prerequisites) +- [Option A: Docker Compose](#option-a-docker-compose) + - [A.1 Prerequisites](#a1-prerequisites) + - [A.2 Clone the Repository](#a2-clone-the-repository) + - [A.3 Run the Setup Script](#a3-run-the-setup-script) + - [A.4 Start the Stack](#a4-start-the-stack) + - [A.5 Verify the Installation](#a5-verify-the-installation) + - [A.6 LDAP Users and Groups](#a6-ldap-users-and-groups) + - [A.7 Optional: Enable Nginx Reverse Proxy](#a7-optional-enable-nginx-reverse-proxy) + - [A.8 Stopping, Starting, and Upgrading](#a8-stopping-starting-and-upgrading) +- [Option B: Daemon Install (systemd + External Services)](#option-b-daemon-install-systemd--external-services) + - [B.1 Architecture Overview](#b1-architecture-overview) + - [B.2 Prerequisites](#b2-prerequisites) + - [B.3 Set Up External Services](#b3-set-up-external-services) + - [B.4 Prepare the Host](#b4-prepare-the-host) + - [B.5 Configure Credentials](#b5-configure-credentials) + - [B.6 Deploy](#b6-deploy) + - [B.7 Set Up Nginx and TLS](#b7-set-up-nginx-and-tls) + - [B.8 Verify the Installation](#b8-verify-the-installation) + - [B.9 Upgrading](#b9-upgrading) +- [Post-Install Configuration](#post-install-configuration) +- [Further Reading](#further-reading) + +--- + +## Prerequisites + +Regardless of which method you choose: + +- **Git** to clone the repository +- A machine with at least **2 GB RAM** and **10 GB free disk** +- Network access to pull container images or download Go/Node toolchains + +--- + +## Option A: Docker Compose + +A single Docker Compose file runs everything: PostgreSQL, MinIO, OpenLDAP, and Silo. An optional nginx container can be enabled for reverse proxying. + +### A.1 Prerequisites + +- [Docker Engine](https://docs.docker.com/engine/install/) 24+ with the [Compose plugin](https://docs.docker.com/compose/install/) (v2) +- `openssl` (used by the setup script to generate secrets) + +Verify your installation: + +```bash +docker --version # Docker Engine 24+ +docker compose version # Docker Compose v2+ +``` + +### A.2 Clone the Repository + +```bash +git clone https://git.kindred-systems.com/kindred/silo.git +cd silo +``` + +### A.3 Run the Setup Script + +The setup script generates credentials and configuration files: + +```bash +./scripts/setup-docker.sh +``` + +It prompts for: +- Server domain (default: `localhost`) +- PostgreSQL password (auto-generated if you press Enter) +- MinIO credentials (auto-generated) +- OpenLDAP admin password and initial user (auto-generated) +- Silo local admin account (fallback when LDAP is unavailable) + +For automated/CI environments, use non-interactive mode: + +```bash +./scripts/setup-docker.sh --non-interactive +``` + +The script writes two files: +- `deployments/.env` — secrets for Docker Compose +- `deployments/config.docker.yaml` — Silo server configuration + +### A.4 Start the Stack + +```bash +docker compose -f deployments/docker-compose.allinone.yaml up -d +``` + +Wait for all services to become healthy: + +```bash +docker compose -f deployments/docker-compose.allinone.yaml ps +``` + +You should see `silo-postgres`, `silo-minio`, `silo-openldap`, and `silo-api` all in a healthy state. + +View logs: + +```bash +# All services +docker compose -f deployments/docker-compose.allinone.yaml logs -f + +# Silo only +docker compose -f deployments/docker-compose.allinone.yaml logs -f silo +``` + +### A.5 Verify the Installation + +```bash +# Health check +curl http://localhost:8080/health + +# Readiness check (includes database and storage connectivity) +curl http://localhost:8080/ready +``` + +Open http://localhost:8080 in your browser. Log in with either: + +- **LDAP account**: the username and password shown by the setup script (default: `siloadmin`) +- **Local admin**: the local admin credentials shown by the setup script (default: `admin`) + +The credentials were printed at the end of the setup script output and are stored in `deployments/.env`. + +### A.6 LDAP Users and Groups + +The Docker stack includes an OpenLDAP server with three preconfigured groups that map to Silo roles: + +| LDAP Group | Silo Role | Access Level | +|------------|-----------|-------------| +| `cn=silo-admins,ou=groups,dc=silo,dc=local` | admin | Full access | +| `cn=silo-users,ou=groups,dc=silo,dc=local` | editor | Create and modify items | +| `cn=silo-viewers,ou=groups,dc=silo,dc=local` | viewer | Read-only | + +The initial LDAP user (default: `siloadmin`) is added to `silo-admins`. + +**Add a new LDAP user:** + +```bash +# From the host (using the exposed port) +ldapadd -x -H ldap://localhost:1389 \ + -D "cn=admin,dc=silo,dc=local" \ + -w "YOUR_LDAP_ADMIN_PASSWORD" << EOF +dn: cn=jdoe,ou=users,dc=silo,dc=local +objectClass: inetOrgPerson +cn: jdoe +sn: Doe +userPassword: changeme +mail: jdoe@example.com +EOF +``` + +**Add a user to a group:** + +```bash +ldapmodify -x -H ldap://localhost:1389 \ + -D "cn=admin,dc=silo,dc=local" \ + -w "YOUR_LDAP_ADMIN_PASSWORD" << EOF +dn: cn=silo-users,ou=groups,dc=silo,dc=local +changetype: modify +add: member +member: cn=jdoe,ou=users,dc=silo,dc=local +EOF +``` + +**List all users:** + +```bash +ldapsearch -x -H ldap://localhost:1389 \ + -b "ou=users,dc=silo,dc=local" \ + -D "cn=admin,dc=silo,dc=local" \ + -w "YOUR_LDAP_ADMIN_PASSWORD" "(objectClass=inetOrgPerson)" cn mail memberOf +``` + +### A.7 Optional: Enable Nginx Reverse Proxy + +To place nginx in front of Silo (for TLS termination or to serve on port 80): + +```bash +docker compose -f deployments/docker-compose.allinone.yaml --profile nginx up -d +``` + +By default nginx listens on ports 80 and 443 and proxies to the Silo container. The configuration is at `deployments/nginx/nginx.conf`. + +**To enable HTTPS**, edit `deployments/docker-compose.allinone.yaml` and uncomment the TLS certificate volume mounts in the `nginx` service, then uncomment the HTTPS server block in `deployments/nginx/nginx.conf`. See the comments in those files for details. + +If you already have your own reverse proxy or load balancer, skip the nginx profile and point your proxy at port 8080. + +### A.8 Stopping, Starting, and Upgrading + +```bash +# Stop the stack (data is preserved in Docker volumes) +docker compose -f deployments/docker-compose.allinone.yaml down + +# Start again +docker compose -f deployments/docker-compose.allinone.yaml up -d + +# Stop and delete all data (WARNING: destroys database, files, and LDAP data) +docker compose -f deployments/docker-compose.allinone.yaml down -v +``` + +**To upgrade to a newer version:** + +```bash +cd silo +git pull +docker compose -f deployments/docker-compose.allinone.yaml up -d --build +``` + +The Silo container is rebuilt from the updated source. Database migrations in `migrations/` are applied automatically on container startup via the PostgreSQL init mechanism. + +--- + +## Option B: Daemon Install (systemd + External Services) + +This method runs Silo as a systemd service on a dedicated host, connecting to externally managed PostgreSQL, MinIO, and optionally LDAP services. + +### B.1 Architecture Overview + +``` + ┌──────────────────────┐ + │ Silo Host │ + │ ┌────────────────┐ │ + HTTPS (443) ──►│ │ nginx │ │ + │ └───────┬────────┘ │ + │ │ :8080 │ + │ ┌───────▼────────┐ │ + │ │ silod │ │ + │ │ (API server) │ │ + │ └──┬─────────┬───┘ │ + └─────┼─────────┼──────┘ + │ │ + ┌───────────▼──┐ ┌───▼──────────────┐ + │ PostgreSQL 16│ │ MinIO (S3) │ + │ :5432 │ │ :9000 API │ + └──────────────┘ │ :9001 Console │ + └──────────────────┘ +``` + +### B.2 Prerequisites + +- Linux host (Debian/Ubuntu or RHEL/Fedora/AlmaLinux) +- Root or sudo access +- Network access to your PostgreSQL and MinIO servers + +The setup script installs Go and other build dependencies automatically. + +### B.3 Set Up External Services + +#### PostgreSQL 16 + +Install PostgreSQL and create the Silo database: + +- [PostgreSQL downloads](https://www.postgresql.org/download/) + +```bash +# After installing PostgreSQL, create the database and user: +sudo -u postgres createuser silo +sudo -u postgres createdb -O silo silo +sudo -u postgres psql -c "ALTER USER silo WITH PASSWORD 'your-password';" +``` + +Ensure the Silo host can connect (check `pg_hba.conf` on the PostgreSQL server). + +Verify: + +```bash +psql -h YOUR_PG_HOST -U silo -d silo -c 'SELECT 1' +``` + +#### MinIO + +Install MinIO and create a bucket and service account: + +- [MinIO quickstart](https://min.io/docs/minio/linux/index.html) + +```bash +# Using the MinIO client (mc): +mc alias set local http://YOUR_MINIO_HOST:9000 minioadmin minioadmin +mc mb local/silo-files +mc admin user add local silouser YOUR_MINIO_SECRET +mc admin policy attach local readwrite --user silouser +``` + +Verify: + +```bash +curl -I http://YOUR_MINIO_HOST:9000/minio/health/live +``` + +#### LDAP / FreeIPA (Optional) + +For LDAP authentication, you need an LDAP server with user and group entries. Options: + +- [FreeIPA](https://www.freeipa.org/page/Quick_Start_Guide) — full identity management (recommended for organizations already using it) +- [OpenLDAP](https://www.openldap.org/doc/admin26/) — lightweight LDAP server + +Silo needs: +- A base DN (e.g., `dc=example,dc=com`) +- Users under a known OU (e.g., `cn=users,cn=accounts,dc=example,dc=com`) +- Groups that map to Silo roles (`admin`, `editor`, `viewer`) +- The `memberOf` overlay enabled (so user entries have `memberOf` attributes) + +See [CONFIGURATION.md — LDAP](CONFIGURATION.md#ldap--freeipa) for the full LDAP configuration reference. + +### B.4 Prepare the Host + +Run the setup script on the target host: + +```bash +# Copy and run the script +scp scripts/setup-host.sh root@YOUR_HOST:/tmp/ +ssh root@YOUR_HOST 'bash /tmp/setup-host.sh' +``` + +Or directly on the host: + +```bash +sudo bash scripts/setup-host.sh +``` + +The script: +1. Installs dependencies (git, Go 1.24) +2. Creates the `silo` system user +3. Creates directories (`/opt/silo`, `/etc/silo`) +4. Clones the repository +5. Creates the environment file template + +To override the default service hostnames: + +```bash +SILO_DB_HOST=db.example.com SILO_MINIO_HOST=s3.example.com sudo -E bash scripts/setup-host.sh +``` + +### B.5 Configure Credentials + +Edit the environment file with your service credentials: + +```bash +sudo nano /etc/silo/silod.env +``` + +```bash +# Database +SILO_DB_PASSWORD=your-database-password + +# MinIO +SILO_MINIO_ACCESS_KEY=silouser +SILO_MINIO_SECRET_KEY=your-minio-secret + +# Authentication +SILO_SESSION_SECRET=generate-a-long-random-string +SILO_ADMIN_USERNAME=admin +SILO_ADMIN_PASSWORD=your-admin-password +``` + +Generate a session secret: + +```bash +openssl rand -hex 32 +``` + +Review the server configuration: + +```bash +sudo nano /etc/silo/config.yaml +``` + +Update `database.host`, `storage.endpoint`, `server.base_url`, and authentication settings for your environment. See [CONFIGURATION.md](CONFIGURATION.md) for all options. + +### B.6 Deploy + +Run the deploy script: + +```bash +sudo /opt/silo/src/scripts/deploy.sh +``` + +The script: +1. Pulls latest code from git +2. Builds the `silod` binary and React frontend +3. Installs files to `/opt/silo` and `/etc/silo` +4. Runs database migrations +5. Installs and starts the systemd service + +Deploy options: + +```bash +# Skip git pull (use current checkout) +sudo /opt/silo/src/scripts/deploy.sh --no-pull + +# Skip build (use existing binary) +sudo /opt/silo/src/scripts/deploy.sh --no-build + +# Just restart the service +sudo /opt/silo/src/scripts/deploy.sh --restart-only + +# Check service status +sudo /opt/silo/src/scripts/deploy.sh --status +``` + +To override the target host or database host: + +```bash +SILO_DEPLOY_TARGET=silo.example.com SILO_DB_HOST=db.example.com sudo -E scripts/deploy.sh +``` + +### B.7 Set Up Nginx and TLS + +#### With FreeIPA (automated) + +If your organization uses FreeIPA, the included script handles nginx setup, IPA enrollment, and certificate issuance: + +```bash +sudo /opt/silo/src/scripts/setup-ipa-nginx.sh +``` + +Override the hostname if needed: + +```bash +SILO_HOSTNAME=silo.example.com sudo -E /opt/silo/src/scripts/setup-ipa-nginx.sh +``` + +The script installs nginx, enrolls the host in FreeIPA, requests a TLS certificate from the IPA CA (auto-renewed by certmonger), and configures nginx as an HTTPS reverse proxy. + +#### Manual nginx setup + +Install nginx and create a config: + +```bash +sudo apt install nginx # or: sudo dnf install nginx +``` + +Use the template at `deployments/nginx/nginx.conf` as a starting point. Copy it to `/etc/nginx/sites-available/silo`, update the `server_name` and certificate paths, then enable it: + +```bash +sudo ln -sf /etc/nginx/sites-available/silo /etc/nginx/sites-enabled/silo +sudo nginx -t +sudo systemctl reload nginx +``` + +After enabling HTTPS, update `server.base_url` in `/etc/silo/config.yaml` to use `https://` and restart Silo: + +```bash +sudo systemctl restart silod +``` + +### B.8 Verify the Installation + +```bash +# Service status +sudo systemctl status silod + +# Health check +curl http://localhost:8080/health + +# Readiness check +curl http://localhost:8080/ready + +# Follow logs +sudo journalctl -u silod -f +``` + +Open your configured base URL in a browser and log in. + +### B.9 Upgrading + +```bash +# Pull latest code and redeploy +sudo /opt/silo/src/scripts/deploy.sh + +# Or deploy a specific version +cd /opt/silo/src +git fetch --all --tags +git checkout v1.2.3 +sudo /opt/silo/src/scripts/deploy.sh --no-pull +``` + +New database migrations are applied automatically during deployment. + +--- + +## Post-Install Configuration + +After a successful installation: + +- **Authentication**: Configure LDAP, OIDC, or local auth backends. See [CONFIGURATION.md — Authentication](CONFIGURATION.md#authentication). +- **Schemas**: Part numbering schemas are loaded from YAML files. See the `schemas/` directory and [CONFIGURATION.md — Schemas](CONFIGURATION.md#schemas). +- **Read-only mode**: Toggle write protection at runtime with `kill -USR1 $(pidof silod)` or by setting `server.read_only: true` in the config. +- **Ongoing maintenance**: See [DEPLOYMENT.md](DEPLOYMENT.md) for service management, log viewing, troubleshooting, and the security checklist. + +--- + +## Further Reading + +| Document | Description | +|----------|-------------| +| [CONFIGURATION.md](CONFIGURATION.md) | Complete `config.yaml` reference | +| [DEPLOYMENT.md](DEPLOYMENT.md) | Operations guide: maintenance, troubleshooting, security | +| [AUTH.md](AUTH.md) | Authentication system design | +| [AUTH_USER_GUIDE.md](AUTH_USER_GUIDE.md) | User guide for login, tokens, and roles | +| [SPECIFICATION.md](SPECIFICATION.md) | Full design specification and API reference | +| [STATUS.md](STATUS.md) | Implementation status | +| [GAP_ANALYSIS.md](GAP_ANALYSIS.md) | Gap analysis and revision control roadmap | +| [COMPONENT_AUDIT.md](COMPONENT_AUDIT.md) | Component audit tool design | diff --git a/docs/SPECIFICATION.md b/docs/SPECIFICATION.md index 8f89a93..ac8a93d 100644 --- a/docs/SPECIFICATION.md +++ b/docs/SPECIFICATION.md @@ -37,7 +37,7 @@ Silo treats **part numbering schemas as configuration, not code**. Multiple numb ▼ ┌─────────────────────────────────────────────────────────────┐ │ Silo Server (silod) │ -│ - REST API (75 endpoints) │ +│ - REST API (78 endpoints) │ │ - Authentication (local, LDAP, OIDC) │ │ - Schema parsing and validation │ │ - Part number generation engine │ @@ -50,7 +50,7 @@ Silo treats **part numbering schemas as configuration, not code**. Multiple numb ▼ ▼ ┌─────────────────────────┐ ┌─────────────────────────────┐ │ PostgreSQL │ │ MinIO │ -│ (psql.kindred.internal)│ │ - File storage │ +│ (psql.example.internal)│ │ - File storage │ │ - Item metadata │ │ - Versioned objects │ │ - Relationships │ │ - Thumbnails │ │ - Revision history │ │ │ @@ -63,7 +63,7 @@ Silo treats **part numbering schemas as configuration, not code**. Multiple numb | Component | Technology | Notes | |-----------|------------|-------| -| Database | PostgreSQL 16 | Existing instance at psql.kindred.internal | +| Database | PostgreSQL 16 | Existing instance at psql.example.internal | | File Storage | MinIO | S3-compatible, versioning enabled | | CLI & API Server | Go (1.24) | chi/v5 router, pgx/v5 driver, zerolog | | Authentication | Multi-backend | Local (bcrypt), LDAP/FreeIPA, OIDC/Keycloak | @@ -598,7 +598,7 @@ See [AUTH.md](AUTH.md) for full architecture details and [AUTH_USER_GUIDE.md](AU ## 11. API Design -### 11.1 REST Endpoints (75 Implemented) +### 11.1 REST Endpoints (78 Implemented) ``` # Health (no auth) @@ -615,6 +615,9 @@ GET /auth/callback # OIDC callback # Public API (no auth required) GET /api/auth/config # Auth backend configuration (for login UI) +# Server-Sent Events (require auth) +GET /api/events # SSE stream for real-time updates + # Auth API (require auth) GET /api/auth/me # Current authenticated user GET /api/auth/tokens # List user's API tokens @@ -627,7 +630,7 @@ POST /api/uploads/presign # Get presigned MinI # Schemas (read: viewer, write: editor) GET /api/schemas # List all schemas GET /api/schemas/{name} # Get schema details -GET /api/schemas/{name}/properties # Get property schema for category +GET /api/schemas/{name}/form # Get form descriptor (field groups, widgets, category picker) POST /api/schemas/{name}/segments/{segment}/values # Add enum value [editor] PUT /api/schemas/{name}/segments/{segment}/values/{code} # Update enum value [editor] DELETE /api/schemas/{name}/segments/{segment}/values/{code} # Delete enum value [editor] @@ -644,6 +647,7 @@ DELETE /api/projects/{code} # Delete project [ed # Items (read: viewer, write: editor) GET /api/items # List/filter items GET /api/items/search # Fuzzy search +GET /api/items/by-uuid/{uuid} # Get item by UUID GET /api/items/export.csv # Export items to CSV GET /api/items/template.csv # CSV import template GET /api/items/export.ods # Export items to ODS @@ -689,6 +693,7 @@ GET /api/items/{partNumber}/bom/export.csv # Export BOM as CSV GET /api/items/{partNumber}/bom/export.ods # Export BOM as ODS POST /api/items/{partNumber}/bom # Add BOM entry [editor] POST /api/items/{partNumber}/bom/import # Import BOM from CSV [editor] +POST /api/items/{partNumber}/bom/merge # Merge BOM from ODS with conflict resolution [editor] PUT /api/items/{partNumber}/bom/{childPartNumber} # Update BOM entry [editor] DELETE /api/items/{partNumber}/bom/{childPartNumber} # Remove BOM entry [editor] @@ -734,11 +739,11 @@ POST /api/inventory/{partNumber}/move ### 12.1 Implemented -- [x] PostgreSQL database schema (11 migrations) +- [x] PostgreSQL database schema (13 migrations) - [x] YAML schema parser for part numbering - [x] Part number generation engine - [x] CLI tool (`cmd/silo`) -- [x] API server (`cmd/silod`) with 75 endpoints +- [x] API server (`cmd/silod`) with 78 endpoints - [x] MinIO integration for file storage with versioning - [x] BOM relationships (component, alternate, reference) - [x] Multi-level BOM (recursive expansion with configurable depth) diff --git a/docs/STATUS.md b/docs/STATUS.md index 43235c2..cac9bbb 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -10,10 +10,10 @@ | Component | Status | Notes | |-----------|--------|-------| -| PostgreSQL schema | Complete | 11 migrations applied | +| PostgreSQL schema | Complete | 13 migrations applied | | YAML schema parser | Complete | Supports enum, serial, constant, string segments | | Part number generator | Complete | Scoped sequences, category-based format | -| API server (`silod`) | Complete | 75 REST endpoints via chi/v5 | +| API server (`silod`) | Complete | 78 REST endpoints via chi/v5 | | CLI tool (`silo`) | Complete | Item registration and management | | MinIO file storage | Complete | Upload, download, versioning, checksums | | Revision control | Complete | Append-only history, rollback, comparison, status/labels | @@ -55,7 +55,7 @@ FreeCAD workbench and LibreOffice Calc extension are maintained in separate repo | Service | Host | Status | |---------|------|--------| -| PostgreSQL | psql.kindred.internal:5432 | Running | +| PostgreSQL | psql.example.internal:5432 | Running | | MinIO | localhost:9000 (API) / :9001 (console) | Configured | | Silo API | localhost:8080 | Builds successfully | @@ -92,5 +92,7 @@ The schema defines 170 category codes across 10 groups: | 007_revision_status.sql | Revision status and labels | | 008_odoo_integration.sql | Odoo ERP integration tables (integrations, sync_log) | | 009_auth.sql | Authentication system (users, api_tokens, sessions, audit_log, user tracking columns) | -| 010_item_extended_fields.sql | Extended item fields (sourcing_type, sourcing_link, standard_cost, long_description) | +| 010_item_extended_fields.sql | Extended item fields (sourcing_type, long_description) | | 011_item_files.sql | Item file attachments (item_files table, thumbnail_key column) | +| 012_bom_source.sql | BOM entry source tracking | +| 013_move_cost_sourcing_to_props.sql | Move sourcing_link and standard_cost from item columns to revision properties | diff --git a/frontend-spec.md b/frontend-spec.md index 4e29629..6eb6ab3 100644 --- a/frontend-spec.md +++ b/frontend-spec.md @@ -1,6 +1,6 @@ # Silo Frontend Specification -Current as of 2026-02-08. Documents the React + Vite + TypeScript frontend (migration from Go templates is complete). +Current as of 2026-02-11. Documents the React + Vite + TypeScript frontend (migration from Go templates is complete). ## Overview @@ -68,6 +68,7 @@ web/ │ └── AuthContext.tsx AuthProvider with login/logout/refresh methods ├── hooks/ │ ├── useAuth.ts Context consumer hook + │ ├── useFormDescriptor.ts Fetches form descriptor from /api/schemas/{name}/form (replaces useCategories) │ ├── useItems.ts Items fetching with search, filters, pagination, debounce │ └── useLocalStorage.ts Typed localStorage persistence hook ├── styles/ @@ -271,63 +272,81 @@ Vite dev server runs on port 5173 with proxy config in `vite.config.ts` forwardi ## New Frontend Tasks -# CreateItemPane Redesign Specification +# CreateItemPane — Schema-Driven Dynamic Form -**Date**: 2026-02-06 -**Scope**: Replace existing `CreateItemPane.tsx` with a two-column layout, multi-stage category picker, file attachment via MinIO, and full use of screen real estate. +**Date**: 2026-02-10 +**Scope**: `CreateItemPane.tsx` renders a dynamic form driven entirely by the form descriptor API (`GET /api/schemas/{name}/form`). All field groups, field types, widgets, and category-specific fields are defined in YAML and resolved server-side. **Parent**: Items page (`ItemsPage.tsx`) — renders in the detail pane area per existing in-pane CRUD pattern. --- ## Layout -The pane uses a CSS Grid two-column layout instead of the current single-column form: +Single-column scrollable form with a green header bar. Field groups are rendered dynamically from the form descriptor. Category-specific field groups appear after global groups when a category is selected. ``` -┌──────────────────────────────────────────────────────┬──────────────┐ -│ Header: "New Item" [green bar] Cancel │ Create │ │ -├──────────────────────────────────────────────────────┤ │ -│ │ Auto- │ -│ ── Identity ────────────────────────────────────── │ assigned │ -│ [Part Number *] [Type * v] │ metadata │ -│ [Description ] │ │ -│ Category * [Domain │ Group │ Subtype ] │──────────────│ -│ Mechanical│ Structural│ Bracket │ │ │ -│ Electrical│ Bearings │ Plate │ │ Attachments │ -│ ... │ ... │ ... │ │ ┌─ ─ ─ ─ ┐ │ -│ ── Sourcing ────────────────────────────────────── │ │ Drop │ │ -│ [Sourcing Type v] [Standard Cost $ ] │ │ zone │ │ -│ [Unit of Measure v] [Sourcing Link ] │ └─ ─ ─ ─ ┘ │ -│ │ file.FCStd │ -│ ── Details ─────────────────────────────────────── │ drawing.pdf │ -│ [Long Description ] │ │ -│ [Projects: [tag][tag] type to search... ] │──────────────│ -│ │ Thumbnail │ -│ │ [preview] │ -└──────────────────────────────────────────────────────┴──────────────┘ +┌──────────────────────────────────────────────────────────────────────┐ +│ Header: "New Item" [green bar] Cancel │ Create │ +├──────────────────────────────────────────────────────────────────────┤ +│ │ +│ Category * [Domain buttons: F C R S E M T A P X] │ +│ [Subcategory search + filtered list] │ +│ │ +│ ── Identity ────────────────────────────────────────────────────── │ +│ [Type * (auto-derived from category)] [Description ] │ +│ │ +│ ── Sourcing ────────────────────────────────────────────────────── │ +│ [Sourcing Type v] [Manufacturer] [MPN] [Supplier] [SPN] │ +│ [Sourcing Link] │ +│ │ +│ ── Cost & Lead Time ────────────────────────────────────────────── │ +│ [Standard Cost $] [Lead Time Days] [Min Order Qty] │ +│ │ +│ ── Status ──────────────────────────────────────────────────────── │ +│ [Lifecycle Status v] [RoHS Compliant ☐] [Country of Origin] │ +│ │ +│ ── Details ─────────────────────────────────────────────────────── │ +│ [Long Description ] │ +│ [Projects: [tag][tag] type to search... ] │ +│ [Notes ] │ +│ │ +│ ── Fastener Specifications (category-specific) ─────────────────── │ +│ [Material] [Finish] [Thread Size] [Head Type] [Drive Type] ... │ +│ │ +└──────────────────────────────────────────────────────────────────────┘ ``` -Grid definition: `grid-template-columns: 1fr 320px`. The left column scrolls independently if content overflows. The right sidebar is a flex column with sections separated by `--ctp-surface1` borders. +## Data Source — Form Descriptor API + +All form structure is fetched from `GET /api/schemas/kindred-rd/form`, which returns: + +- `category_picker`: Multi-stage picker config (domain → subcategory) +- `item_fields`: Definitions for item-level fields (description, item_type, sourcing_type, etc.) +- `field_groups`: Ordered groups with resolved field metadata (Identity, Sourcing, Cost, Status, Details) +- `category_field_groups`: Per-category-prefix groups (e.g., Fastener Specifications for `F` prefix) +- `field_overrides`: Widget hints (currency, url, select, checkbox) + +The YAML schema (`schemas/kindred-rd.yaml`) is the single source of truth. Adding a new field or category in YAML propagates to all clients with no code changes. ## File Location -`web/src/components/items/CreateItemPane.tsx` (replaces existing file) +`web/src/components/items/CreateItemPane.tsx` -New supporting files: +Supporting files: | File | Purpose | |------|---------| -| `web/src/components/items/CategoryPicker.tsx` | Multi-stage category selector | +| `web/src/components/items/CategoryPicker.tsx` | Multi-stage domain/subcategory selector | | `web/src/components/items/FileDropZone.tsx` | Drag-and-drop file upload with MinIO presigned URLs | | `web/src/components/items/TagInput.tsx` | Multi-select tag input for projects | -| `web/src/hooks/useCategories.ts` | Fetches category tree from schema data | +| `web/src/hooks/useFormDescriptor.ts` | Fetches and caches form descriptor from `/api/schemas/{name}/form` | | `web/src/hooks/useFileUpload.ts` | Manages presigned URL upload flow | ## Component Breakdown ### CreateItemPane -Top-level orchestrator. Manages form state, submission, and layout. +Top-level orchestrator. Renders dynamic form from the form descriptor. **Props** (unchanged interface): @@ -341,68 +360,64 @@ interface CreateItemPaneProps { **State**: ```typescript -const [form, setForm] = useState({ - part_number: '', - item_type: 'part', - description: '', - category_path: [], // e.g. ['Mechanical', 'Structural', 'Bracket'] - sourcing_type: 'manufactured', - standard_cost: '', - unit_of_measure: 'ea', - sourcing_link: '', - long_description: '', - project_ids: [], -}); -const [attachments, setAttachments] = useState([]); -const [thumbnail, setThumbnail] = useState(null); +const { descriptor, categories, loading } = useFormDescriptor(); +const [category, setCategory] = useState(''); // selected category code, e.g. "F01" +const [fields, setFields] = useState>({}); // all field values keyed by name const [error, setError] = useState(null); const [submitting, setSubmitting] = useState(false); ``` +A single `fields` record holds all form values (both item-level and property fields). The `ITEM_LEVEL_FIELDS` set (`description`, `item_type`, `sourcing_type`, `long_description`) determines which fields go into the top-level request vs. the `properties` map on submission. + +**Auto-derivation**: When a category is selected, `item_type` is automatically set based on the `derived_from_category` mapping in the form descriptor (e.g., category prefix `A` → `assembly`, `T` → `tooling`, default → `part`). + +**Dynamic rendering**: A `renderField()` function maps each field's `widget` type to the appropriate input: + +| Widget | Rendered As | +|--------|-------------| +| `text` | `` | +| `number` | `` | +| `textarea` | `