silo/docs/DEPLOYMENT.md

Silo Production Deployment Guide

This guide covers deploying Silo to a dedicated VM using external PostgreSQL and MinIO services.

Table of Contents

• Architecture
• External Services
• Quick Start
• Initial Setup
• Deployment
• Configuration
• Maintenance
• Troubleshooting

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                     silo.kindred.internal                       │
│  ┌───────────────────────────────────────────────────────────┐ │
│  │                        silod                               │ │
│  │                   (Silo API Server)                        │ │
│  │                      :8080                                 │ │
│  └───────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
              │                               │
              ▼                               ▼
┌─────────────────────────┐     ┌─────────────────────────────────┐
│  psql.kindred.internal  │     │    minio.kindred.internal       │
│      PostgreSQL 16      │     │         MinIO S3                │
│        :5432            │     │       :9000 (API)               │
│                         │     │       :9001 (Console)           │
└─────────────────────────┘     └─────────────────────────────────┘

External Services

The following external services are already configured:

Service	Host	Database/Bucket	User
PostgreSQL	psql.kindred.internal:5432	silo	silo
MinIO	minio.kindred.internal:9000	silo-files	silouser

Migrations have been applied to the database.

---

Quick Start

For a fresh VM, run these commands:

# 1. SSH to the target host
ssh root@silo.kindred.internal

# 2. Download and run setup script
curl -fsSL https://gitea.kindred.internal/kindred/silo-0062/raw/branch/main/scripts/setup-host.sh | bash

# 3. Configure credentials
nano /etc/silo/silod.env

# 4. Deploy
/opt/silo/src/scripts/deploy.sh

---

Initial Setup

Run the setup script once on silo.kindred.internal to prepare the host:

# Option 1: If you have the repo locally
scp scripts/setup-host.sh root@silo.kindred.internal:/tmp/
ssh root@silo.kindred.internal 'bash /tmp/setup-host.sh'

# Option 2: Direct on the host
ssh root@silo.kindred.internal
curl -fsSL https://git.kindred.internal/kindred/silo/raw/branch/main/scripts/setup-host.sh -o /tmp/setup-host.sh
bash /tmp/setup-host.sh

The setup script:

• Installs dependencies (git, Go 1.23)
• Creates the silo system user
• Creates directory structure (/opt/silo, /etc/silo)
• Clones the repository to /opt/silo/src
• Creates the environment file template

Configure Credentials

After setup, edit the environment file with your credentials:

sudo nano /etc/silo/silod.env

Fill in the values:

# Database credentials (psql.kindred.internal)
SILO_DB_PASSWORD=your-database-password

# MinIO credentials (minio.kindred.internal)
SILO_MINIO_ACCESS_KEY=silouser
SILO_MINIO_SECRET_KEY=your-minio-secret-key

Verify External Services

Before deploying, verify connectivity to external services:

# Test PostgreSQL
psql -h psql.kindred.internal -U silo -d silo -c 'SELECT 1'

# Test MinIO
curl -I http://minio.kindred.internal:9000/minio/health/live

---

Deployment

Deploy (or Update)

To deploy or update Silo, run the deploy script on the target host:

ssh root@silo.kindred.internal
/opt/silo/src/scripts/deploy.sh

The deploy script:

1. Pulls the latest code from git
2. Builds the silod binary
3. Installs configuration and schemas
4. Installs/updates the systemd service
5. Restarts the service
6. Verifies health endpoints

Deploy Options

# Full deployment (pull, build, deploy, restart)
sudo /opt/silo/src/scripts/deploy.sh

# Skip git pull (use current checkout)
sudo /opt/silo/src/scripts/deploy.sh --no-pull

# Skip build (use existing binary)
sudo /opt/silo/src/scripts/deploy.sh --no-build

# Just restart the service
sudo /opt/silo/src/scripts/deploy.sh --restart-only

# Check service status
sudo /opt/silo/src/scripts/deploy.sh --status

Environment Variables

You can override the git repository URL and branch:

export SILO_REPO_URL=https://git.kindred.internal/kindred/silo.git
export SILO_BRANCH=main
sudo -E /opt/silo/src/scripts/deploy.sh

---

Configuration

File Locations

File	Purpose
/opt/silo/bin/silod	Server binary
/opt/silo/src/	Git repository checkout
/etc/silo/config.yaml	Server configuration
/etc/silo/silod.env	Environment variables (secrets)
/etc/silo/schemas/	Part numbering schemas
/var/log/silo/	Log directory

Configuration File

The configuration file /etc/silo/config.yaml is installed on first deployment and not overwritten on updates. To update it manually:

sudo cp /opt/silo/src/deployments/config.prod.yaml /etc/silo/config.yaml
sudo systemctl restart silod

Schemas

Schemas in /etc/silo/schemas/ are updated on every deployment from the repository.

---

Maintenance

Service Management

# Check status
sudo systemctl status silod

# Start/stop/restart
sudo systemctl start silod
sudo systemctl stop silod
sudo systemctl restart silod

# Enable/disable auto-start
sudo systemctl enable silod
sudo systemctl disable silod

View Logs

# Follow logs
sudo journalctl -u silod -f

# Recent logs
sudo journalctl -u silod -n 100

# Logs since a time
sudo journalctl -u silod --since "1 hour ago"
sudo journalctl -u silod --since "2024-01-15 10:00:00"

Health Checks

# Basic health check
curl http://localhost:8080/health

# Full readiness check (includes DB and MinIO)
curl http://localhost:8080/ready

Update Deployment

To update to the latest version:

ssh root@silo.kindred.internal
/opt/silo/src/scripts/deploy.sh

To deploy a specific branch or tag:

cd /opt/silo/src
git fetch --all --tags
git checkout v1.2.3  # or a branch name
sudo /opt/silo/src/scripts/deploy.sh --no-pull

Database Migrations

When new migrations are added, run them manually:

# Check for new migrations
ls -la /opt/silo/src/migrations/

# Run a specific migration
psql -h psql.kindred.internal -U silo -d silo -f /opt/silo/src/migrations/008_new_feature.sql

---

Troubleshooting

Service Won't Start

1. Check logs for errors:

   sudo journalctl -u silod -n 50
   

2. Verify configuration:

   cat /etc/silo/config.yaml
   

3. Check environment file permissions:

   ls -la /etc/silo/silod.env
   # Should be: -rw------- root silo
   

4. Verify binary exists:

   ls -la /opt/silo/bin/silod
   

Connection Refused to PostgreSQL

1. Test network connectivity:

   nc -zv psql.kindred.internal 5432
   

2. Test credentials:

   source /etc/silo/silod.env
   PGPASSWORD=$SILO_DB_PASSWORD psql -h psql.kindred.internal -U silo -d silo -c 'SELECT 1'
   

3. Check pg_hba.conf on PostgreSQL server allows connections from this host.

Connection Refused to MinIO

1. Test network connectivity:

   nc -zv minio.kindred.internal 9000
   

2. Test with curl:

   curl -I http://minio.kindred.internal:9000/minio/health/live
   

3. Check SSL settings in config match MinIO setup:

   storage:
     use_ssl: true  # or false
   

Health Check Fails

# Check individual endpoints
curl -v http://localhost:8080/health
curl -v http://localhost:8080/ready

# If ready fails but health passes, check external services
psql -h psql.kindred.internal -U silo -d silo -c 'SELECT 1'
curl http://minio.kindred.internal:9000/minio/health/live

Build Fails

1. Check Go is installed:

   go version
   # Should be 1.23+
   

2. Check source is present:

   ls -la /opt/silo/src/
   

3. Try manual build:

   cd /opt/silo/src
   go build -v ./cmd/silod
   

---

Security Checklist

• /etc/silo/silod.env has mode 600 (chmod 600)
• Database password is strong and unique
• MinIO credentials are specific to silo (not admin)
• SSL/TLS enabled for PostgreSQL (sslmode: require)
• SSL/TLS enabled for MinIO (use_ssl: true) if available
• Firewall restricts access to port 8080
• Service runs as non-root silo user