feat(storage): add MinIO to filesystem migration tool #137

forbes · 2026-02-18T20:13:38Z

forbes commented

2026-02-18 20:13:38 +00:00

Summary

Standalone migration tool that downloads all files from MinIO and writes them to the local filesystem, enabling MinIO to be decommissioned from the production deployment.

Changes

Add cmd/migrate-storage/main.go — one-shot migration binary
Add migrate-storage Makefile target
Add binary to .gitignore

How It Works

Queries all file references from three sources in PostgreSQL:

Revision files (revisions.file_key)
Item file attachments (item_files.object_key)
Item thumbnails (items.thumbnail_key)

Downloads each from MinIO and writes to the local filesystem, mirroring the MinIO key structure as paths under a configurable root directory. No DB updates needed since the keys map directly to filesystem paths.

Features

--dry-run to preview without downloading
--verbose for per-file logging
Atomic writes (temp file + rename)
Idempotent — skips files that already exist with matching size
Deduplicates across the three sources
Progress reporting and summary at end

Usage

make migrate-storage
./migrate-storage -config /etc/silo/config.yaml -dest /opt/silo/data -dry-run
./migrate-storage -config /etc/silo/config.yaml -dest /opt/silo/data

Testing

Builds cleanly: go build ./cmd/migrate-storage
-h flag shows usage
-dry-run validates DB queries and lists files without downloading
Full run against a MinIO instance migrates all files

## Summary Standalone migration tool that downloads all files from MinIO and writes them to the local filesystem, enabling MinIO to be decommissioned from the production deployment. ## Changes - Add `cmd/migrate-storage/main.go` — one-shot migration binary - Add `migrate-storage` Makefile target - Add binary to `.gitignore` ## How It Works Queries all file references from three sources in PostgreSQL: - Revision files (`revisions.file_key`) - Item file attachments (`item_files.object_key`) - Item thumbnails (`items.thumbnail_key`) Downloads each from MinIO and writes to the local filesystem, mirroring the MinIO key structure as paths under a configurable root directory. No DB updates needed since the keys map directly to filesystem paths. ### Features - `--dry-run` to preview without downloading - `--verbose` for per-file logging - Atomic writes (temp file + rename) - Idempotent — skips files that already exist with matching size - Deduplicates across the three sources - Progress reporting and summary at end ## Usage ```bash make migrate-storage ./migrate-storage -config /etc/silo/config.yaml -dest /opt/silo/data -dry-run ./migrate-storage -config /etc/silo/config.yaml -dest /opt/silo/data ``` ## Testing - Builds cleanly: `go build ./cmd/migrate-storage` - `-h` flag shows usage - `-dry-run` validates DB queries and lists files without downloading - Full run against a MinIO instance migrates all files

forbes added 1 commit 2026-02-18 20:13:38 +00:00

feat(storage): add MinIO to filesystem migration tool f5b03989ff

Standalone binary (cmd/migrate-storage) that downloads all files from
MinIO and writes them to the local filesystem for decommissioning MinIO.

Queries revision files, item file attachments, and item thumbnails from
the database, then downloads each from MinIO preserving the object key
structure as filesystem paths. Supports --dry-run, --verbose, atomic
writes via temp+rename, and idempotent re-runs (skips existing files
with matching size).

forbes merged commit 344a0cd0a0 into main

2026-02-18 20:16:18 +00:00

forbes deleted branch feat/migrate-storage-tool

2026-02-18 20:16:18 +00:00

forbes referenced this issue from a commit

2026-02-18 20:16:19 +00:00

Merge pull request 'feat(storage): add MinIO to filesystem migration tool' (#137) from feat/migrate-storage-tool into main

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kindred/silo#137