silo/docs/DEPLOYMENT.md

Silo Production Deployment Guide

This guide covers deploying Silo to a dedicated VM using external PostgreSQL and MinIO services.

Table of Contents

• Architecture
• External Services
• Quick Start
• Initial Setup
• Deployment
• Configuration
• Maintenance
• Troubleshooting

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                     silo.example.internal                       │
│  ┌───────────────────────────────────────────────────────────┐ │
│  │                        silod                               │ │
│  │                   (Silo API Server)                        │ │
│  │                      :8080                                 │ │
│  └───────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
              │                               │
              ▼                               ▼
┌─────────────────────────┐     ┌─────────────────────────────────┐
│  psql.example.internal  │     │    minio.example.internal       │
│      PostgreSQL 16      │     │         MinIO S3                │
│        :5432            │     │       :9000 (API)               │
│                         │     │       :9001 (Console)           │
└─────────────────────────┘     └─────────────────────────────────┘

External Services

The following external services are already configured:

Service	Host	Database/Bucket	User
PostgreSQL	psql.example.internal:5432	silo	silo
MinIO	minio.example.internal:9000	silo-files	silouser

Migrations have been applied to the database.

---

Quick Start

For a fresh VM, run these commands:

# 1. SSH to the target host
ssh root@silo.example.internal

# 2. Download and run setup script
curl -fsSL https://git.kindred-systems.com/kindred/silo/raw/branch/main/scripts/setup-host.sh | bash

# 3. Configure credentials
nano /etc/silo/silod.env

# 4. Deploy
/opt/silo/src/scripts/deploy.sh

---

Initial Setup

Run the setup script once on silo.example.internal to prepare the host:

# Option 1: If you have the repo locally
scp scripts/setup-host.sh root@silo.example.internal:/tmp/
ssh root@silo.example.internal 'bash /tmp/setup-host.sh'

# Option 2: Direct on the host
ssh root@silo.example.internal
curl -fsSL https://git.kindred-systems.com/kindred/silo/raw/branch/main/scripts/setup-host.sh -o /tmp/setup-host.sh
bash /tmp/setup-host.sh

The setup script:

• Installs dependencies (git, Go 1.23)
• Creates the silo system user
• Creates directory structure (/opt/silo, /etc/silo)
• Clones the repository to /opt/silo/src
• Creates the environment file template

Configure Credentials

After setup, edit the environment file with your credentials:

sudo nano /etc/silo/silod.env

Fill in the values:

# Database credentials (psql.example.internal)
SILO_DB_PASSWORD=your-database-password

# MinIO credentials (minio.example.internal)
SILO_MINIO_ACCESS_KEY=silouser
SILO_MINIO_SECRET_KEY=your-minio-secret-key

Verify External Services

Before deploying, verify connectivity to external services:

# Test PostgreSQL
psql -h psql.example.internal -U silo -d silo -c 'SELECT 1'

# Test MinIO
curl -I http://minio.example.internal:9000/minio/health/live

---

Deployment

Deploy (or Update)

To deploy or update Silo, run the deploy script on the target host:

ssh root@silo.example.internal
/opt/silo/src/scripts/deploy.sh

The deploy script:

1. Pulls the latest code from git
2. Builds the silod binary
3. Installs configuration and schemas
4. Installs/updates the systemd service
5. Restarts the service
6. Verifies health endpoints

Deploy Options

# Full deployment (pull, build, deploy, restart)
sudo /opt/silo/src/scripts/deploy.sh

# Skip git pull (use current checkout)
sudo /opt/silo/src/scripts/deploy.sh --no-pull

# Skip build (use existing binary)
sudo /opt/silo/src/scripts/deploy.sh --no-build

# Just restart the service
sudo /opt/silo/src/scripts/deploy.sh --restart-only

# Check service status
sudo /opt/silo/src/scripts/deploy.sh --status

Environment Variables

You can override the git repository URL and branch:

export SILO_REPO_URL=https://git.kindred-systems.com/kindred/silo.git
export SILO_BRANCH=main
sudo -E /opt/silo/src/scripts/deploy.sh

---

Configuration

File Locations

File	Purpose
/opt/silo/bin/silod	Server binary
/opt/silo/src/	Git repository checkout
/etc/silo/config.yaml	Server configuration
/etc/silo/silod.env	Environment variables (secrets)
/etc/silo/schemas/	Part numbering schemas
/var/log/silo/	Log directory

Configuration File

The configuration file /etc/silo/config.yaml is installed on first deployment and not overwritten on updates. To update it manually:

sudo cp /opt/silo/src/deployments/config.prod.yaml /etc/silo/config.yaml
sudo systemctl restart silod

Schemas

Schemas in /etc/silo/schemas/ are updated on every deployment from the repository.

---

Maintenance

Service Management

# Check status
sudo systemctl status silod

# Start/stop/restart
sudo systemctl start silod
sudo systemctl stop silod
sudo systemctl restart silod

# Enable/disable auto-start
sudo systemctl enable silod
sudo systemctl disable silod

View Logs

# Follow logs
sudo journalctl -u silod -f

# Recent logs
sudo journalctl -u silod -n 100

# Logs since a time
sudo journalctl -u silod --since "1 hour ago"
sudo journalctl -u silod --since "2024-01-15 10:00:00"

Health Checks

# Basic health check
curl http://localhost:8080/health

# Full readiness check (includes DB and MinIO)
curl http://localhost:8080/ready

Update Deployment

To update to the latest version:

ssh root@silo.example.internal
/opt/silo/src/scripts/deploy.sh

To deploy a specific branch or tag:

cd /opt/silo/src
git fetch --all --tags
git checkout v1.2.3  # or a branch name
sudo /opt/silo/src/scripts/deploy.sh --no-pull

Database Migrations

When new migrations are added, run them manually:

# Check for new migrations
ls -la /opt/silo/src/migrations/

# Run a specific migration
psql -h psql.example.internal -U silo -d silo -f /opt/silo/src/migrations/008_new_feature.sql

---

Troubleshooting

Service Won't Start

1. Check logs for errors:

   sudo journalctl -u silod -n 50
   

2. Verify configuration:

   cat /etc/silo/config.yaml
   

3. Check environment file permissions:

   ls -la /etc/silo/silod.env
   # Should be: -rw------- root silo
   

4. Verify binary exists:

   ls -la /opt/silo/bin/silod
   

Connection Refused to PostgreSQL

1. Test network connectivity:

   nc -zv psql.example.internal 5432
   

2. Test credentials:

   source /etc/silo/silod.env
   PGPASSWORD=$SILO_DB_PASSWORD psql -h psql.example.internal -U silo -d silo -c 'SELECT 1'
   

3. Check pg_hba.conf on PostgreSQL server allows connections from this host.

Connection Refused to MinIO

1. Test network connectivity:

   nc -zv minio.example.internal 9000
   

2. Test with curl:

   curl -I http://minio.example.internal:9000/minio/health/live
   

3. Check SSL settings in config match MinIO setup:

   storage:
     use_ssl: true  # or false
   

Health Check Fails

# Check individual endpoints
curl -v http://localhost:8080/health
curl -v http://localhost:8080/ready

# If ready fails but health passes, check external services
psql -h psql.example.internal -U silo -d silo -c 'SELECT 1'
curl http://minio.example.internal:9000/minio/health/live

Build Fails

1. Check Go is installed:

   go version
   # Should be 1.23+
   

2. Check source is present:

   ls -la /opt/silo/src/
   

3. Try manual build:

   cd /opt/silo/src
   go build -v ./cmd/silod
   

---

SSL/TLS with FreeIPA and Nginx

For production deployments, Silo should be served over HTTPS using nginx as a reverse proxy with certificates from FreeIPA.

Setup IPA and Nginx

Run the IPA/nginx setup script after the basic host setup:

sudo /opt/silo/src/scripts/setup-ipa-nginx.sh

This script:

1. Installs FreeIPA client and nginx
2. Enrolls the host in FreeIPA domain
3. Requests SSL certificate from IPA CA (auto-renewed by certmonger)
4. Configures nginx as reverse proxy (HTTP → HTTPS redirect)
5. Opens firewall ports 80 and 443

Manual Steps After Script

1. Verify certificate was issued:

   getcert list
   

2. The silo config is already updated to use https://silo.example.internal as base URL. Restart silo:

   sudo systemctl restart silod
   

3. Test the setup:

   curl https://silo.example.internal/health
   

Certificate Management

Certificates are automatically renewed by certmonger. Check status:

# List all tracked certificates
getcert list

# Check specific certificate
getcert list -f /etc/ssl/silo/silo.crt

# Manual renewal if needed
getcert resubmit -f /etc/ssl/silo/silo.crt

Trusting IPA CA on Clients

For clients to trust the Silo HTTPS certificate, they need the IPA CA:

# Download CA cert
curl -o /tmp/ipa-ca.crt https://ipa.example.internal/ipa/config/ca.crt

# Ubuntu/Debian
sudo cp /tmp/ipa-ca.crt /usr/local/share/ca-certificates/ipa-ca.crt
sudo update-ca-certificates

# RHEL/Fedora
sudo cp /tmp/ipa-ca.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust

Nginx Configuration

The nginx config is installed at /etc/nginx/sites-available/silo. Key settings:

• HTTP redirects to HTTPS
• TLS 1.2/1.3 only with strong ciphers
• Proxies to 127.0.0.1:8080 (silod)
• 100MB max upload size for CAD files
• Security headers (X-Frame-Options, etc.)

To modify:

sudo nano /etc/nginx/sites-available/silo
sudo nginx -t
sudo systemctl reload nginx

---

Security Checklist

• /etc/silo/silod.env has mode 600 (chmod 600)
• Database password is strong and unique
• MinIO credentials are specific to silo (not admin)
• SSL/TLS enabled for PostgreSQL (sslmode: require)
• SSL/TLS enabled for MinIO (use_ssl: true) if available
• HTTPS enabled via nginx reverse proxy
• Silod listens on localhost only (host: 127.0.0.1)
• Firewall allows only ports 80, 443 (not 8080)
• Service runs as non-root silo user
• Host enrolled in FreeIPA for centralized auth (future)