kindred/silo
1
1
Fork 0
Code Issues 30 Pull Requests Actions Packages Projects Releases Wiki Activity

update deployment

Browse Source
This commit is contained in:
Forbes
2026-01-26 21:32:57 -06:00
parent f37052c2f0
commit 5201b35d83
5 changed files with 585 additions and 930 deletions
Download Patch File Download Diff File Expand all files Collapse all files

519
docs/DEPLOYMENT.md
View File

@@ -6,9 +6,10 @@ This guide covers deploying Silo to a dedicated VM using external PostgreSQL and
- [Architecture](#architecture)
- [External Services](#external-services)
- [Automated Deployment (CI/CD)](#automated-deployment-cicd)
- [Manual Deployment](#manual-deployment)
- [Post-Deployment Configuration](#post-deployment-configuration)
- [Quick Start](#quick-start)
- [Initial Setup](#initial-setup)
- [Deployment](#deployment)
- [Configuration](#configuration)
- [Maintenance](#maintenance)
- [Troubleshooting](#troubleshooting)
@@ -33,377 +34,343 @@ This guide covers deploying Silo to a dedicated VM using external PostgreSQL and
└─────────────────────────┘ └─────────────────────────────────┘
```
## Prerequisites
## External Services
### On psql.kindred.internal
The following external services are already configured:
1. Create the Silo database and user:
| Service | Host | Database/Bucket | User |
|---------|------|-----------------|------|
| PostgreSQL | psql.kindred.internal:5432 | silo | silo |
| MinIO | minio.kindred.internal:9000 | silo-files | silouser |
```sql
-- Connect as postgres superuser
CREATE USER silo WITH PASSWORD 'your-secure-password';
CREATE DATABASE silo OWNER silo;
-- Grant necessary permissions
GRANT ALL PRIVILEGES ON DATABASE silo TO silo;
-- Connect to silo database
\c silo
-- Enable UUID extension
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
```
2. Run migrations:
```bash
# From the silo project directory
psql -h psql.kindred.internal -U silo -d silo -f migrations/001_initial.sql
psql -h psql.kindred.internal -U silo -d silo -f migrations/002_sequence_by_name.sql
psql -h psql.kindred.internal -U silo -d silo -f migrations/003_remove_material.sql
psql -h psql.kindred.internal -U silo -d silo -f migrations/004_cad_sync_state.sql
psql -h psql.kindred.internal -U silo -d silo -f migrations/005_property_schema_version.sql
psql -h psql.kindred.internal -U silo -d silo -f migrations/006_project_tags.sql
psql -h psql.kindred.internal -U silo -d silo -f migrations/007_revision_status.sql
```
Or run all at once:
```bash
for f in migrations/*.sql; do
echo "Running $f..."
psql -h psql.kindred.internal -U silo -d silo -f "$f"
done
```
3. Allow connections from Silo VM in `pg_hba.conf`:
```
# Allow silo.kindred.internal to connect
hostssl silo silo silo.kindred.internal/32 scram-sha-256
```
### On minio.kindred.internal
1. Create the Silo bucket and access credentials:
```bash
# Using mc (MinIO Client)
mc alias set kindred https://minio.kindred.internal ADMIN_ACCESS_KEY ADMIN_SECRET_KEY
# Create bucket with versioning
mc mb kindred/silo-files
mc version enable kindred/silo-files
# Create service account for Silo
mc admin user add kindred silo-service 'your-minio-secret-key'
# Create policy for silo-files bucket
cat > /tmp/silo-policy.json << 'EOF'
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket",
"s3:GetBucketVersioning",
"s3:GetObjectVersion",
"s3:DeleteObjectVersion"
],
"Resource": [
"arn:aws:s3:::silo-files",
"arn:aws:s3:::silo-files/*"
]
}
]
}
EOF
mc admin policy create kindred silo-policy /tmp/silo-policy.json
mc admin policy attach kindred silo-policy --user silo-service
```
2. Verify SSL certificate is valid (or configure Silo to use non-SSL if internal).
Migrations have been applied to the database.
---
## Deployment Options
## Quick Start
### Option A: Systemd Service (Recommended for Production)
#### 1. Prepare the Silo VM
For a fresh VM, run these commands:
```bash
# Create silo user
sudo useradd -r -m -d /opt/silo -s /sbin/nologin silo
# 1. SSH to the target host
ssh root@silo.kindred.internal
# Create directories
sudo mkdir -p /opt/silo/bin
sudo mkdir -p /etc/silo/schemas
sudo mkdir -p /var/log/silo
# 2. Download and run setup script
curl -fsSL https://gitea.kindred.internal/kindred/silo-0062/raw/branch/main/scripts/setup-host.sh | bash
# Set ownership
sudo chown -R silo:silo /opt/silo /var/log/silo
sudo chown root:silo /etc/silo
sudo chmod 750 /etc/silo
# 3. Configure credentials
nano /etc/silo/silod.env
# 4. Deploy
/opt/silo/src/scripts/deploy.sh
```
#### 2. Build and Install Binary
---
## Initial Setup
Run the setup script once on `silo.kindred.internal` to prepare the host:
```bash
# On build machine (requires Go 1.23+)
cd /path/to/silo
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-w -s" -o silod ./cmd/silod
# Option 1: If you have the repo locally
scp scripts/setup-host.sh root@silo.kindred.internal:/tmp/
ssh root@silo.kindred.internal 'bash /tmp/setup-host.sh'
# Copy to Silo VM
scp silod silo.kindred.internal:/tmp/
ssh silo.kindred.internal "sudo mv /tmp/silod /opt/silo/bin/ && sudo chmod 755 /opt/silo/bin/silod"
# Option 2: Direct on the host
ssh root@silo.kindred.internal
curl -fsSL https://git.kindred.internal/kindred/silo/raw/branch/main/scripts/setup-host.sh -o /tmp/setup-host.sh
bash /tmp/setup-host.sh
```
#### 3. Install Configuration
The setup script:
- Installs dependencies (git, Go 1.23)
- Creates the `silo` system user
- Creates directory structure (`/opt/silo`, `/etc/silo`)
- Clones the repository to `/opt/silo/src`
- Creates the environment file template
### Configure Credentials
After setup, edit the environment file with your credentials:
```bash
# Copy config file
scp deployments/config.prod.yaml silo.kindred.internal:/tmp/config.yaml
ssh silo.kindred.internal "sudo mv /tmp/config.yaml /etc/silo/config.yaml"
sudo nano /etc/silo/silod.env
```
# Copy schemas
scp -r schemas/* silo.kindred.internal:/tmp/schemas/
ssh silo.kindred.internal "sudo mv /tmp/schemas/* /etc/silo/schemas/"
Fill in the values:
# Create environment file with secrets
ssh silo.kindred.internal
sudo cat > /etc/silo/silod.env << 'EOF'
```bash
# Database credentials (psql.kindred.internal)
SILO_DB_PASSWORD=your-database-password
SILO_MINIO_ACCESS_KEY=silo-service
# MinIO credentials (minio.kindred.internal)
SILO_MINIO_ACCESS_KEY=silouser
SILO_MINIO_SECRET_KEY=your-minio-secret-key
EOF
sudo chmod 600 /etc/silo/silod.env
sudo chown root:silo /etc/silo/silod.env
```
#### 4. Install Systemd Service
### Verify External Services
Before deploying, verify connectivity to external services:
```bash
# Copy service file
scp deployments/systemd/silod.service silo.kindred.internal:/tmp/
ssh silo.kindred.internal "sudo mv /tmp/silod.service /etc/systemd/system/"
# Test PostgreSQL
psql -h psql.kindred.internal -U silo -d silo -c 'SELECT 1'
# Enable and start
ssh silo.kindred.internal << 'EOF'
sudo systemctl daemon-reload
sudo systemctl enable silod
sudo systemctl start silod
sudo systemctl status silod
EOF
# Test MinIO
curl -I http://minio.kindred.internal:9000/minio/health/live
```
#### 5. Verify Deployment
---
## Deployment
### Deploy (or Update)
To deploy or update Silo, run the deploy script on the target host:
```bash
ssh root@silo.kindred.internal
/opt/silo/src/scripts/deploy.sh
```
The deploy script:
1. Pulls the latest code from git
2. Builds the `silod` binary
3. Installs configuration and schemas
4. Installs/updates the systemd service
5. Restarts the service
6. Verifies health endpoints
### Deploy Options
```bash
# Full deployment (pull, build, deploy, restart)
sudo /opt/silo/src/scripts/deploy.sh
# Skip git pull (use current checkout)
sudo /opt/silo/src/scripts/deploy.sh --no-pull
# Skip build (use existing binary)
sudo /opt/silo/src/scripts/deploy.sh --no-build
# Just restart the service
sudo /opt/silo/src/scripts/deploy.sh --restart-only
# Check service status
sudo systemctl status silod
sudo /opt/silo/src/scripts/deploy.sh --status
```
# Check logs
sudo journalctl -u silod -f
### Environment Variables
# Test health endpoint
curl http://localhost:8080/health
You can override the git repository URL and branch:
# Test readiness (verifies DB and MinIO connectivity)
curl http://localhost:8080/ready
```bash
export SILO_REPO_URL=https://git.kindred.internal/kindred/silo.git
export SILO_BRANCH=main
sudo -E /opt/silo/src/scripts/deploy.sh
```
---
### Option B: Docker Compose
## Configuration
#### 1. Install Docker on Silo VM
### File Locations
| File | Purpose |
|------|---------|
| `/opt/silo/bin/silod` | Server binary |
| `/opt/silo/src/` | Git repository checkout |
| `/etc/silo/config.yaml` | Server configuration |
| `/etc/silo/silod.env` | Environment variables (secrets) |
| `/etc/silo/schemas/` | Part numbering schemas |
| `/var/log/silo/` | Log directory |
### Configuration File
The configuration file `/etc/silo/config.yaml` is installed on first deployment and not overwritten on updates. To update it manually:
```bash
# Debian/Ubuntu
sudo apt-get update
sudo apt-get install -y docker.io docker-compose-plugin
sudo usermod -aG docker $USER
sudo cp /opt/silo/src/deployments/config.prod.yaml /etc/silo/config.yaml
sudo systemctl restart silod
```
#### 2. Clone Repository
### Schemas
```bash
git clone https://github.com/kindred-systems/silo.git /opt/silo
cd /opt/silo
```
#### 3. Configure Environment
```bash
# Create .env file
cat > /opt/silo/deployments/.env << 'EOF'
SILO_DB_PASSWORD=your-database-password
SILO_MINIO_ACCESS_KEY=silo-service
SILO_MINIO_SECRET_KEY=your-minio-secret-key
SILO_BASE_URL=http://silo.kindred.internal:8080
EOF
chmod 600 /opt/silo/deployments/.env
```
#### 4. Start Service
```bash
cd /opt/silo/deployments
docker compose -f docker-compose.prod.yaml up -d
```
#### 5. Verify
```bash
docker compose -f docker-compose.prod.yaml ps
docker compose -f docker-compose.prod.yaml logs -f
curl http://localhost:8080/ready
```
---
## Post-Deployment Configuration
### DNS Setup
Add DNS records for `silo.kindred.internal` pointing to the Silo VM IP address.
### Firewall Rules
```bash
# Allow incoming connections on port 8080
sudo ufw allow 8080/tcp
# Or with iptables
sudo iptables -A INPUT -p tcp --dport 8080 -j ACCEPT
```
### Reverse Proxy (Optional)
For TLS termination, configure nginx or caddy:
```nginx
# /etc/nginx/sites-available/silo
server {
listen 443 ssl http2;
server_name silo.kindred.internal;
ssl_certificate /etc/ssl/certs/silo.kindred.internal.crt;
ssl_certificate_key /etc/ssl/private/silo.kindred.internal.key;
location / {
proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
```
Schemas in `/etc/silo/schemas/` are updated on every deployment from the repository.
---
## Maintenance
### Service Management
```bash
# Check status
sudo systemctl status silod
# Start/stop/restart
sudo systemctl start silod
sudo systemctl stop silod
sudo systemctl restart silod
# Enable/disable auto-start
sudo systemctl enable silod
sudo systemctl disable silod
```
### View Logs
```bash
# Systemd
# Follow logs
sudo journalctl -u silod -f
sudo journalctl -u silod --since "1 hour ago"
# Docker
docker compose -f docker-compose.prod.yaml logs -f silo
# Recent logs
sudo journalctl -u silod -n 100
# Logs since a time
sudo journalctl -u silod --since "1 hour ago"
sudo journalctl -u silod --since "2024-01-15 10:00:00"
```
### Restart Service
### Health Checks
```bash
# Systemd
sudo systemctl restart silod
# Basic health check
curl http://localhost:8080/health
# Docker
docker compose -f docker-compose.prod.yaml restart silo
# Full readiness check (includes DB and MinIO)
curl http://localhost:8080/ready
```
### Update Deployment
```bash
# Systemd - rebuild and replace binary
go build -ldflags="-w -s" -o silod ./cmd/silod
sudo systemctl stop silod
sudo cp silod /opt/silo/bin/silod
sudo systemctl start silod
To update to the latest version:
# Docker - rebuild and restart
docker compose -f docker-compose.prod.yaml build
docker compose -f docker-compose.prod.yaml up -d
```bash
ssh root@silo.kindred.internal
/opt/silo/src/scripts/deploy.sh
```
To deploy a specific branch or tag:
```bash
cd /opt/silo/src
git fetch --all --tags
git checkout v1.2.3 # or a branch name
sudo /opt/silo/src/scripts/deploy.sh --no-pull
```
### Database Migrations
When updating Silo, check for new migrations:
When new migrations are added, run them manually:
```bash
# List migration files
ls -la migrations/
# Check for new migrations
ls -la /opt/silo/src/migrations/
# Run new migrations
psql -h psql.kindred.internal -U silo -d silo -f migrations/008_new_feature.sql
# Run a specific migration
psql -h psql.kindred.internal -U silo -d silo -f /opt/silo/src/migrations/008_new_feature.sql
```
---
## Troubleshooting
### Service Won't Start
1. Check logs for errors:
```bash
sudo journalctl -u silod -n 50
```
2. Verify configuration:
```bash
cat /etc/silo/config.yaml
```
3. Check environment file permissions:
```bash
ls -la /etc/silo/silod.env
# Should be: -rw------- root silo
```
4. Verify binary exists:
```bash
ls -la /opt/silo/bin/silod
```
### Connection Refused to PostgreSQL
1. Verify network connectivity: `nc -zv psql.kindred.internal 5432`
2. Check `pg_hba.conf` allows connections from Silo VM
3. Verify firewall rules on PostgreSQL server
4. Check credentials in `/etc/silo/silod.env`
1. Test network connectivity:
```bash
nc -zv psql.kindred.internal 5432
```
2. Test credentials:
```bash
source /etc/silo/silod.env
PGPASSWORD=$SILO_DB_PASSWORD psql -h psql.kindred.internal -U silo -d silo -c 'SELECT 1'
```
3. Check `pg_hba.conf` on PostgreSQL server allows connections from this host.
### Connection Refused to MinIO
1. Verify network connectivity: `nc -zv minio.kindred.internal 9000`
2. Check SSL settings match (use_ssl: true/false)
3. Verify access key and secret key
4. Check bucket exists: `mc ls kindred/silo-files`
1. Test network connectivity:
```bash
nc -zv minio.kindred.internal 9000
```
### Service Won't Start
2. Test with curl:
```bash
curl -I http://minio.kindred.internal:9000/minio/health/live
```
1. Check logs: `sudo journalctl -u silod -n 50`
2. Verify config syntax: `/opt/silo/bin/silod -config /etc/silo/config.yaml -validate`
3. Check file permissions on config and env files
4. Verify schemas directory exists and contains YAML files
3. Check SSL settings in config match MinIO setup:
```yaml
storage:
use_ssl: true # or false
```
### Health Check Fails
```bash
# Test each component
curl http://localhost:8080/health # Basic health
curl http://localhost:8080/ready # Full readiness (DB + MinIO)
# Check individual endpoints
curl -v http://localhost:8080/health
curl -v http://localhost:8080/ready
# If ready fails, check individual services
psql -h psql.kindred.internal -U silo -d silo -c "SELECT 1"
mc ls kindred/silo-files
# If ready fails but health passes, check external services
psql -h psql.kindred.internal -U silo -d silo -c 'SELECT 1'
curl http://minio.kindred.internal:9000/minio/health/live
```
### Build Fails
1. Check Go is installed:
```bash
go version
# Should be 1.23+
```
2. Check source is present:
```bash
ls -la /opt/silo/src/
```
3. Try manual build:
```bash
cd /opt/silo/src
go build -v ./cmd/silod
```
---
## Security Checklist
- [ ] `/etc/silo/silod.env` has mode 600 (`chmod 600`)
- [ ] Database password is strong and unique
- [ ] MinIO credentials are service-account specific
- [ ] `/etc/silo/silod.env` has mode 600
- [ ] MinIO credentials are specific to silo (not admin)
- [ ] SSL/TLS enabled for PostgreSQL (`sslmode: require`)
- [ ] SSL/TLS enabled for MinIO (`use_ssl: true`)
- [ ] SSL/TLS enabled for MinIO (`use_ssl: true`) if available
- [ ] Firewall restricts access to port 8080
- [ ] Silo runs as non-root user
- [ ] Logs don't contain sensitive information
- [ ] Service runs as non-root `silo` user