silo/docs/DEPLOYMENT.md

# Silo Production Deployment Guide

> **First-time setup?** See the [Installation Guide](INSTALL.md) for step-by-step
> instructions. This document covers ongoing maintenance and operations for an
> existing deployment.

This guide covers deploying Silo to a dedicated VM using external PostgreSQL and MinIO services.

## Table of Contents

- [Architecture](#architecture)
- [External Services](#external-services)
- [Quick Start](#quick-start)
- [Initial Setup](#initial-setup)
- [Deployment](#deployment)
- [Configuration](#configuration)
- [Maintenance](#maintenance)
- [Troubleshooting](#troubleshooting)

## Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                     silo.example.internal                       │
│  ┌───────────────────────────────────────────────────────────┐ │
│  │                        silod                               │ │
│  │                   (Silo API Server)                        │ │
│  │                      :8080                                 │ │
│  └───────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
              │                               │
              ▼                               ▼
┌─────────────────────────┐     ┌─────────────────────────────────┐
│  psql.example.internal  │     │    minio.example.internal       │
│      PostgreSQL 16      │     │         MinIO S3                │
│        :5432            │     │       :9000 (API)               │
│                         │     │       :9001 (Console)           │
└─────────────────────────┘     └─────────────────────────────────┘
```

## External Services

The following external services are already configured:

| Service | Host | Database/Bucket | User |
|---------|------|-----------------|------|
| PostgreSQL | psql.example.internal:5432 | silo | silo |
| MinIO | minio.example.internal:9000 | silo-files | silouser |

Migrations have been applied to the database.

---

## Quick Start

For a fresh VM, run these commands:

```bash
# 1. SSH to the target host
ssh root@silo.example.internal

# 2. Download and run setup script
curl -fsSL https://git.kindred-systems.com/kindred/silo/raw/branch/main/scripts/setup-host.sh | bash

# 3. Configure credentials
nano /etc/silo/silod.env

# 4. Deploy
/opt/silo/src/scripts/deploy.sh
```

---

## Initial Setup

Run the setup script once on `silo.example.internal` to prepare the host:

```bash
# Option 1: If you have the repo locally
scp scripts/setup-host.sh root@silo.example.internal:/tmp/
ssh root@silo.example.internal 'bash /tmp/setup-host.sh'

# Option 2: Direct on the host
ssh root@silo.example.internal
curl -fsSL https://git.kindred-systems.com/kindred/silo/raw/branch/main/scripts/setup-host.sh -o /tmp/setup-host.sh
bash /tmp/setup-host.sh
```

The setup script:
- Installs dependencies (git, Go 1.23)
- Creates the `silo` system user
- Creates directory structure (`/opt/silo`, `/etc/silo`)
- Clones the repository to `/opt/silo/src`
- Creates the environment file template

### Configure Credentials

After setup, edit the environment file with your credentials:

```bash
sudo nano /etc/silo/silod.env
```

Fill in the values:

```bash
# Database credentials (psql.example.internal)
SILO_DB_PASSWORD=your-database-password

# MinIO credentials (minio.example.internal)
SILO_MINIO_ACCESS_KEY=silouser
SILO_MINIO_SECRET_KEY=your-minio-secret-key
```

### Verify External Services

Before deploying, verify connectivity to external services:

```bash
# Test PostgreSQL
psql -h psql.example.internal -U silo -d silo -c 'SELECT 1'

# Test MinIO
curl -I http://minio.example.internal:9000/minio/health/live
```

---

## Deployment

### Deploy (or Update)

To deploy or update Silo, run the deploy script on the target host:

```bash
ssh root@silo.example.internal
/opt/silo/src/scripts/deploy.sh
```

The deploy script:
1. Pulls the latest code from git
2. Builds the `silod` binary
3. Installs configuration and schemas
4. Installs/updates the systemd service
5. Restarts the service
6. Verifies health endpoints

### Deploy Options

```bash
# Full deployment (pull, build, deploy, restart)
sudo /opt/silo/src/scripts/deploy.sh

# Skip git pull (use current checkout)
sudo /opt/silo/src/scripts/deploy.sh --no-pull

# Skip build (use existing binary)
sudo /opt/silo/src/scripts/deploy.sh --no-build

# Just restart the service
sudo /opt/silo/src/scripts/deploy.sh --restart-only

# Check service status
sudo /opt/silo/src/scripts/deploy.sh --status
```

### Environment Variables

You can override the git repository URL and branch:

```bash
export SILO_REPO_URL=https://git.kindred-systems.com/kindred/silo.git
export SILO_BRANCH=main
sudo -E /opt/silo/src/scripts/deploy.sh
```

---

## Configuration

### File Locations

| File | Purpose |
|------|---------|
| `/opt/silo/bin/silod` | Server binary |
| `/opt/silo/src/` | Git repository checkout |
| `/etc/silo/config.yaml` | Server configuration |
| `/etc/silo/silod.env` | Environment variables (secrets) |
| `/etc/silo/schemas/` | Part numbering schemas |
| `/var/log/silo/` | Log directory |

### Configuration File

The configuration file `/etc/silo/config.yaml` is installed on first deployment and not overwritten on updates. To update it manually:

```bash
sudo cp /opt/silo/src/deployments/config.prod.yaml /etc/silo/config.yaml
sudo systemctl restart silod
```

### Schemas

Schemas in `/etc/silo/schemas/` are updated on every deployment from the repository.

---

## Maintenance

### Service Management

```bash
# Check status
sudo systemctl status silod

# Start/stop/restart
sudo systemctl start silod
sudo systemctl stop silod
sudo systemctl restart silod

# Enable/disable auto-start
sudo systemctl enable silod
sudo systemctl disable silod
```

### View Logs

```bash
# Follow logs
sudo journalctl -u silod -f

# Recent logs
sudo journalctl -u silod -n 100

# Logs since a time
sudo journalctl -u silod --since "1 hour ago"
sudo journalctl -u silod --since "2024-01-15 10:00:00"
```

### Health Checks

```bash
# Basic health check
curl http://localhost:8080/health

# Full readiness check (includes DB and MinIO)
curl http://localhost:8080/ready
```

### Update Deployment

To update to the latest version:

```bash
ssh root@silo.example.internal
/opt/silo/src/scripts/deploy.sh
```

To deploy a specific branch or tag:

```bash
cd /opt/silo/src
git fetch --all --tags
git checkout v1.2.3  # or a branch name
sudo /opt/silo/src/scripts/deploy.sh --no-pull
```

### Database Migrations

When new migrations are added, run them manually:

```bash
# Check for new migrations
ls -la /opt/silo/src/migrations/

# Run a specific migration
psql -h psql.example.internal -U silo -d silo -f /opt/silo/src/migrations/008_new_feature.sql
```

---

## Troubleshooting

### Service Won't Start

1. Check logs for errors:
   ```bash
   sudo journalctl -u silod -n 50
   ```

2. Verify configuration:
   ```bash
   cat /etc/silo/config.yaml
   ```

3. Check environment file permissions:
   ```bash
   ls -la /etc/silo/silod.env
   # Should be: -rw------- root silo
   ```

4. Verify binary exists:
   ```bash
   ls -la /opt/silo/bin/silod
   ```

### Connection Refused to PostgreSQL

1. Test network connectivity:
   ```bash
   nc -zv psql.example.internal 5432
   ```

2. Test credentials:
   ```bash
   source /etc/silo/silod.env
   PGPASSWORD=$SILO_DB_PASSWORD psql -h psql.example.internal -U silo -d silo -c 'SELECT 1'
   ```

3. Check `pg_hba.conf` on PostgreSQL server allows connections from this host.

### Connection Refused to MinIO

1. Test network connectivity:
   ```bash
   nc -zv minio.example.internal 9000
   ```

2. Test with curl:
   ```bash
   curl -I http://minio.example.internal:9000/minio/health/live
   ```

3. Check SSL settings in config match MinIO setup:
   ```yaml
   storage:
     use_ssl: true  # or false
   ```

### Health Check Fails

```bash
# Check individual endpoints
curl -v http://localhost:8080/health
curl -v http://localhost:8080/ready

# If ready fails but health passes, check external services
psql -h psql.example.internal -U silo -d silo -c 'SELECT 1'
curl http://minio.example.internal:9000/minio/health/live
```

### Build Fails

1. Check Go is installed:
   ```bash
   go version
   # Should be 1.23+
   ```

2. Check source is present:
   ```bash
   ls -la /opt/silo/src/
   ```

3. Try manual build:
   ```bash
   cd /opt/silo/src
   go build -v ./cmd/silod
   ```

---

## SSL/TLS with FreeIPA and Nginx

For production deployments, Silo should be served over HTTPS using nginx as a reverse proxy with certificates from FreeIPA.

### Setup IPA and Nginx

Run the IPA/nginx setup script after the basic host setup:

```bash
sudo /opt/silo/src/scripts/setup-ipa-nginx.sh
```

This script:
1. Installs FreeIPA client and nginx
2. Enrolls the host in FreeIPA domain
3. Requests SSL certificate from IPA CA (auto-renewed by certmonger)
4. Configures nginx as reverse proxy (HTTP → HTTPS redirect)
5. Opens firewall ports 80 and 443

### Manual Steps After Script

1. Verify certificate was issued:
   ```bash
   getcert list
   ```

2. The silo config is already updated to use `https://silo.example.internal` as base URL. Restart silo:
   ```bash
   sudo systemctl restart silod
   ```

3. Test the setup:
   ```bash
   curl https://silo.example.internal/health
   ```

### Certificate Management

Certificates are automatically renewed by certmonger. Check status:

```bash
# List all tracked certificates
getcert list

# Check specific certificate
getcert list -f /etc/ssl/silo/silo.crt

# Manual renewal if needed
getcert resubmit -f /etc/ssl/silo/silo.crt
```

### Trusting IPA CA on Clients

For clients to trust the Silo HTTPS certificate, they need the IPA CA:

```bash
# Download CA cert
curl -o /tmp/ipa-ca.crt https://ipa.example.internal/ipa/config/ca.crt

# Ubuntu/Debian
sudo cp /tmp/ipa-ca.crt /usr/local/share/ca-certificates/ipa-ca.crt
sudo update-ca-certificates

# RHEL/Fedora
sudo cp /tmp/ipa-ca.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust
```

### Nginx Configuration

The nginx config is installed at `/etc/nginx/sites-available/silo`. Key settings:

- HTTP redirects to HTTPS
- TLS 1.2/1.3 only with strong ciphers
- Proxies to `127.0.0.1:8080` (silod)
- 100MB max upload size for CAD files
- Security headers (X-Frame-Options, etc.)

To modify:
```bash
sudo nano /etc/nginx/sites-available/silo
sudo nginx -t
sudo systemctl reload nginx
```

---

## Security Checklist

- [ ] `/etc/silo/silod.env` has mode 600 (`chmod 600`)
- [ ] Database password is strong and unique
- [ ] MinIO credentials are specific to silo (not admin)
- [ ] SSL/TLS enabled for PostgreSQL (`sslmode: require`)
- [ ] SSL/TLS enabled for MinIO (`use_ssl: true`) if available
- [ ] HTTPS enabled via nginx reverse proxy
- [ ] Silod listens on localhost only (`host: 127.0.0.1`)
- [ ] Firewall allows only ports 80, 443 (not 8080)
- [ ] Service runs as non-root `silo` user
- [ ] Host enrolled in FreeIPA for centralized auth (future)