silo/docs/DEPLOYMENT.md

Silo Production Deployment Guide

This guide covers deploying Silo to a dedicated VM using external PostgreSQL and MinIO services.

Table of Contents

• Architecture
• External Services
• Automated Deployment (CI/CD)
• Manual Deployment
• Post-Deployment Configuration
• Maintenance
• Troubleshooting

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                     silo.kindred.internal                       │
│  ┌───────────────────────────────────────────────────────────┐ │
│  │                        silod                               │ │
│  │                   (Silo API Server)                        │ │
│  │                      :8080                                 │ │
│  └───────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
              │                               │
              ▼                               ▼
┌─────────────────────────┐     ┌─────────────────────────────────┐
│  psql.kindred.internal  │     │    minio.kindred.internal       │
│      PostgreSQL 16      │     │         MinIO S3                │
│        :5432            │     │       :9000 (API)               │
│                         │     │       :9001 (Console)           │
└─────────────────────────┘     └─────────────────────────────────┘

Prerequisites

On psql.kindred.internal

1. Create the Silo database and user:

-- Connect as postgres superuser
CREATE USER silo WITH PASSWORD 'your-secure-password';
CREATE DATABASE silo OWNER silo;

-- Grant necessary permissions
GRANT ALL PRIVILEGES ON DATABASE silo TO silo;

-- Connect to silo database
\c silo

-- Enable UUID extension
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";

2. Run migrations:

# From the silo project directory
psql -h psql.kindred.internal -U silo -d silo -f migrations/001_initial.sql
psql -h psql.kindred.internal -U silo -d silo -f migrations/002_sequence_by_name.sql
psql -h psql.kindred.internal -U silo -d silo -f migrations/003_remove_material.sql
psql -h psql.kindred.internal -U silo -d silo -f migrations/004_cad_sync_state.sql
psql -h psql.kindred.internal -U silo -d silo -f migrations/005_property_schema_version.sql
psql -h psql.kindred.internal -U silo -d silo -f migrations/006_project_tags.sql
psql -h psql.kindred.internal -U silo -d silo -f migrations/007_revision_status.sql

Or run all at once:

for f in migrations/*.sql; do
  echo "Running $f..."
  psql -h psql.kindred.internal -U silo -d silo -f "$f"
done

3. Allow connections from Silo VM in pg_hba.conf:

# Allow silo.kindred.internal to connect
hostssl silo silo silo.kindred.internal/32 scram-sha-256

On minio.kindred.internal

1. Create the Silo bucket and access credentials:

# Using mc (MinIO Client)
mc alias set kindred https://minio.kindred.internal ADMIN_ACCESS_KEY ADMIN_SECRET_KEY

# Create bucket with versioning
mc mb kindred/silo-files
mc version enable kindred/silo-files

# Create service account for Silo
mc admin user add kindred silo-service 'your-minio-secret-key'

# Create policy for silo-files bucket
cat > /tmp/silo-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket",
        "s3:GetBucketVersioning",
        "s3:GetObjectVersion",
        "s3:DeleteObjectVersion"
      ],
      "Resource": [
        "arn:aws:s3:::silo-files",
        "arn:aws:s3:::silo-files/*"
      ]
    }
  ]
}
EOF

mc admin policy create kindred silo-policy /tmp/silo-policy.json
mc admin policy attach kindred silo-policy --user silo-service

2. Verify SSL certificate is valid (or configure Silo to use non-SSL if internal).

---

Deployment Options

Option A: Systemd Service (Recommended for Production)

1. Prepare the Silo VM

# Create silo user
sudo useradd -r -m -d /opt/silo -s /sbin/nologin silo

# Create directories
sudo mkdir -p /opt/silo/bin
sudo mkdir -p /etc/silo/schemas
sudo mkdir -p /var/log/silo

# Set ownership
sudo chown -R silo:silo /opt/silo /var/log/silo
sudo chown root:silo /etc/silo
sudo chmod 750 /etc/silo

2. Build and Install Binary

# On build machine (requires Go 1.23+)
cd /path/to/silo
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-w -s" -o silod ./cmd/silod

# Copy to Silo VM
scp silod silo.kindred.internal:/tmp/
ssh silo.kindred.internal "sudo mv /tmp/silod /opt/silo/bin/ && sudo chmod 755 /opt/silo/bin/silod"

3. Install Configuration

# Copy config file
scp deployments/config.prod.yaml silo.kindred.internal:/tmp/config.yaml
ssh silo.kindred.internal "sudo mv /tmp/config.yaml /etc/silo/config.yaml"

# Copy schemas
scp -r schemas/* silo.kindred.internal:/tmp/schemas/
ssh silo.kindred.internal "sudo mv /tmp/schemas/* /etc/silo/schemas/"

# Create environment file with secrets
ssh silo.kindred.internal
sudo cat > /etc/silo/silod.env << 'EOF'
SILO_DB_PASSWORD=your-database-password
SILO_MINIO_ACCESS_KEY=silo-service
SILO_MINIO_SECRET_KEY=your-minio-secret-key
EOF
sudo chmod 600 /etc/silo/silod.env
sudo chown root:silo /etc/silo/silod.env

4. Install Systemd Service

# Copy service file
scp deployments/systemd/silod.service silo.kindred.internal:/tmp/
ssh silo.kindred.internal "sudo mv /tmp/silod.service /etc/systemd/system/"

# Enable and start
ssh silo.kindred.internal << 'EOF'
sudo systemctl daemon-reload
sudo systemctl enable silod
sudo systemctl start silod
sudo systemctl status silod
EOF

5. Verify Deployment

# Check service status
sudo systemctl status silod

# Check logs
sudo journalctl -u silod -f

# Test health endpoint
curl http://localhost:8080/health

# Test readiness (verifies DB and MinIO connectivity)
curl http://localhost:8080/ready

---

Option B: Docker Compose

1. Install Docker on Silo VM

# Debian/Ubuntu
sudo apt-get update
sudo apt-get install -y docker.io docker-compose-plugin
sudo usermod -aG docker $USER

2. Clone Repository

git clone https://github.com/kindred-systems/silo.git /opt/silo
cd /opt/silo

3. Configure Environment

# Create .env file
cat > /opt/silo/deployments/.env << 'EOF'
SILO_DB_PASSWORD=your-database-password
SILO_MINIO_ACCESS_KEY=silo-service
SILO_MINIO_SECRET_KEY=your-minio-secret-key
SILO_BASE_URL=http://silo.kindred.internal:8080
EOF
chmod 600 /opt/silo/deployments/.env

4. Start Service

cd /opt/silo/deployments
docker compose -f docker-compose.prod.yaml up -d

5. Verify

docker compose -f docker-compose.prod.yaml ps
docker compose -f docker-compose.prod.yaml logs -f
curl http://localhost:8080/ready

---

Post-Deployment Configuration

DNS Setup

Add DNS records for silo.kindred.internal pointing to the Silo VM IP address.

Firewall Rules

# Allow incoming connections on port 8080
sudo ufw allow 8080/tcp

# Or with iptables
sudo iptables -A INPUT -p tcp --dport 8080 -j ACCEPT

Reverse Proxy (Optional)

For TLS termination, configure nginx or caddy:

# /etc/nginx/sites-available/silo
server {
    listen 443 ssl http2;
    server_name silo.kindred.internal;

    ssl_certificate /etc/ssl/certs/silo.kindred.internal.crt;
    ssl_certificate_key /etc/ssl/private/silo.kindred.internal.key;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

---

Maintenance

View Logs

# Systemd
sudo journalctl -u silod -f
sudo journalctl -u silod --since "1 hour ago"

# Docker
docker compose -f docker-compose.prod.yaml logs -f silo

Restart Service

# Systemd
sudo systemctl restart silod

# Docker
docker compose -f docker-compose.prod.yaml restart silo

Update Deployment

# Systemd - rebuild and replace binary
go build -ldflags="-w -s" -o silod ./cmd/silod
sudo systemctl stop silod
sudo cp silod /opt/silo/bin/silod
sudo systemctl start silod

# Docker - rebuild and restart
docker compose -f docker-compose.prod.yaml build
docker compose -f docker-compose.prod.yaml up -d

Database Migrations

When updating Silo, check for new migrations:

# List migration files
ls -la migrations/

# Run new migrations
psql -h psql.kindred.internal -U silo -d silo -f migrations/008_new_feature.sql

---

Troubleshooting

Connection Refused to PostgreSQL

1. Verify network connectivity: nc -zv psql.kindred.internal 5432
2. Check pg_hba.conf allows connections from Silo VM
3. Verify firewall rules on PostgreSQL server
4. Check credentials in /etc/silo/silod.env

Connection Refused to MinIO

1. Verify network connectivity: nc -zv minio.kindred.internal 9000
2. Check SSL settings match (use_ssl: true/false)
3. Verify access key and secret key
4. Check bucket exists: mc ls kindred/silo-files

Service Won't Start

1. Check logs: sudo journalctl -u silod -n 50
2. Verify config syntax: /opt/silo/bin/silod -config /etc/silo/config.yaml -validate
3. Check file permissions on config and env files
4. Verify schemas directory exists and contains YAML files

Health Check Fails

# Test each component
curl http://localhost:8080/health   # Basic health
curl http://localhost:8080/ready    # Full readiness (DB + MinIO)

# If ready fails, check individual services
psql -h psql.kindred.internal -U silo -d silo -c "SELECT 1"
mc ls kindred/silo-files

---

Security Checklist

• Database password is strong and unique
• MinIO credentials are service-account specific
• /etc/silo/silod.env has mode 600
• SSL/TLS enabled for PostgreSQL (sslmode: require)
• SSL/TLS enabled for MinIO (use_ssl: true)
• Firewall restricts access to port 8080
• Silo runs as non-root user
• Logs don't contain sensitive information