feat(storage): implement filesystem backend #127

New Issue

forbes · 2026-02-17T16:09:49Z

forbes commented

2026-02-17 16:09:49 +00:00

Summary

Implement the FileStore interface (from #126) for local filesystem storage, writing files under a configurable root directory.

Context

The FileStore interface defines Put, Get, GetVersion, Delete, Exists, Copy, PresignPut, and Ping. The filesystem backend replaces MinIO for deployments that want simpler infrastructure — files stored directly on disk at a configurable path (e.g. /var/lib/silo/objects).

Object key patterns used by the application

Purpose	Key Pattern	Source
Revision file	`items/{partNumber}/rev{N}.FCStd`	`storage.FileKey()`
Revision thumbnail	`thumbnails/{partNumber}/rev{N}.png`	`storage.ThumbnailKey()`
Temp upload	`uploads/tmp/{uuid}/{filename}`	`HandlePresignUpload`
Item file attachment	`items/{itemID}/files/{uuid}/{filename}`	`HandleAssociateItemFile`
Item thumbnail	`items/{itemID}/thumbnail.png`	`HandleSetItemThumbnail`

All of these become filesystem paths under {root}/{key}.

Requirements

Create internal/storage/filesystem.go implementing FileStore:

`Put(ctx, key, reader, size, contentType) (*PutResult, error)`

Target path: {root}/{key}
Create intermediate directories with os.MkdirAll
Atomic write: write to a temp file in the same directory, then os.Rename to final path
Compute SHA-256 checksum during write (tee the reader)
Return PutResult{Key: key, Size: bytesWritten, Checksum: sha256hex}
VersionID can be empty string (filesystem has no versioning)
Store content-type in a sidecar file {root}/{key}.meta (JSON: {"content_type": "..."}) or skip if not needed for serving

`Get(ctx, key) (io.ReadCloser, error)`

os.Open("{root}/{key}") returning the *os.File (implements io.ReadCloser)
Return os.ErrNotExist-wrapped error if file missing

`GetVersion(ctx, key, versionID) (io.ReadCloser, error)`

Filesystem has no versioning — ignore versionID, delegate to Get(ctx, key)
Log a warning if versionID is non-empty

`Delete(ctx, key) error`

os.Remove("{root}/{key}")
Optionally clean up empty parent directories
Not an error if file already missing

`Exists(ctx, key) (bool, error)`

os.Stat("{root}/{key}") — return (true, nil) if exists, (false, nil) if os.IsNotExist

`Copy(ctx, srcKey, dstKey) error`

Open source, create destination (atomic write pattern), copy bytes
Used by HandleAssociateItemFile and HandleSetItemThumbnail to move temp uploads to permanent locations

`PresignPut(ctx, key, expiry) (*url.URL, error)`

Filesystem cannot presign URLs for direct browser upload
Return ("", ErrPresignNotSupported) or a sentinel error
Callers (HandlePresignUpload) must handle this — see #129

`Ping(ctx) error`

Verify root directory exists and is writable (e.g. create and remove a temp file)

Constructor

func NewFilesystemStore(root string) (*FilesystemStore, error)

Validate root path exists or create it
Store absolute path

File structure

internal/storage/
├── interface.go       # FileStore interface (from #126)
├── storage.go         # MinIO implementation (existing)
├── filesystem.go      # New filesystem implementation
└── filesystem_test.go # Unit tests

Testing

Unit tests in internal/storage/filesystem_test.go:

TestPut — write file, verify content and checksum
TestPutAtomicity — verify no partial files on write failure
TestGet — read back written file
TestGetMissing — error on nonexistent key
TestDelete — delete file, verify gone
TestDeleteMissing — no error on missing file
TestExists — true for existing, false for missing
TestCopy — verify content matches after copy
TestPing — succeeds with valid root, fails with invalid

Use t.TempDir() for test isolation.

Acceptance criteria

FilesystemStore implements FileStore interface
Atomic writes prevent partial files
SHA-256 checksum computed on Put
All unit tests pass
Works with all 5 object key patterns listed above
go build ./... passes

Priority

P0 — blocks upload endpoint changes and data migration

Depends on

#126 (FileStore interface)

Part of

Storage Migration: MinIO → PostgreSQL + Filesystem

## Summary Implement the `FileStore` interface (from #126) for local filesystem storage, writing files under a configurable root directory. ## Context The `FileStore` interface defines `Put`, `Get`, `GetVersion`, `Delete`, `Exists`, `Copy`, `PresignPut`, and `Ping`. The filesystem backend replaces MinIO for deployments that want simpler infrastructure — files stored directly on disk at a configurable path (e.g. `/var/lib/silo/objects`). ### Object key patterns used by the application | Purpose | Key Pattern | Source | |---------|-------------|--------| | Revision file | `items/{partNumber}/rev{N}.FCStd` | `storage.FileKey()` | | Revision thumbnail | `thumbnails/{partNumber}/rev{N}.png` | `storage.ThumbnailKey()` | | Temp upload | `uploads/tmp/{uuid}/{filename}` | `HandlePresignUpload` | | Item file attachment | `items/{itemID}/files/{uuid}/{filename}` | `HandleAssociateItemFile` | | Item thumbnail | `items/{itemID}/thumbnail.png` | `HandleSetItemThumbnail` | All of these become filesystem paths under `{root}/{key}`. ## Requirements Create `internal/storage/filesystem.go` implementing `FileStore`: ### `Put(ctx, key, reader, size, contentType) (*PutResult, error)` - Target path: `{root}/{key}` - Create intermediate directories with `os.MkdirAll` - **Atomic write**: write to a temp file in the same directory, then `os.Rename` to final path - Compute SHA-256 checksum during write (tee the reader) - Return `PutResult{Key: key, Size: bytesWritten, Checksum: sha256hex}` - `VersionID` can be empty string (filesystem has no versioning) - Store content-type in a sidecar file `{root}/{key}.meta` (JSON: `{"content_type": "..."}`) or skip if not needed for serving ### `Get(ctx, key) (io.ReadCloser, error)` - `os.Open("{root}/{key}")` returning the `*os.File` (implements `io.ReadCloser`) - Return `os.ErrNotExist`-wrapped error if file missing ### `GetVersion(ctx, key, versionID) (io.ReadCloser, error)` - Filesystem has no versioning — ignore `versionID`, delegate to `Get(ctx, key)` - Log a warning if `versionID` is non-empty ### `Delete(ctx, key) error` - `os.Remove("{root}/{key}")` - Optionally clean up empty parent directories - Not an error if file already missing ### `Exists(ctx, key) (bool, error)` - `os.Stat("{root}/{key}")` — return `(true, nil)` if exists, `(false, nil)` if `os.IsNotExist` ### `Copy(ctx, srcKey, dstKey) error` - Open source, create destination (atomic write pattern), copy bytes - Used by `HandleAssociateItemFile` and `HandleSetItemThumbnail` to move temp uploads to permanent locations ### `PresignPut(ctx, key, expiry) (*url.URL, error)` - Filesystem cannot presign URLs for direct browser upload - Return `("", ErrPresignNotSupported)` or a sentinel error - Callers (`HandlePresignUpload`) must handle this — see #129 ### `Ping(ctx) error` - Verify root directory exists and is writable (e.g. create and remove a temp file) ### Constructor ```go func NewFilesystemStore(root string) (*FilesystemStore, error) ``` - Validate root path exists or create it - Store absolute path ## File structure ``` internal/storage/ ├── interface.go # FileStore interface (from #126) ├── storage.go # MinIO implementation (existing) ├── filesystem.go # New filesystem implementation └── filesystem_test.go # Unit tests ``` ## Testing Unit tests in `internal/storage/filesystem_test.go`: - `TestPut` — write file, verify content and checksum - `TestPutAtomicity` — verify no partial files on write failure - `TestGet` — read back written file - `TestGetMissing` — error on nonexistent key - `TestDelete` — delete file, verify gone - `TestDeleteMissing` — no error on missing file - `TestExists` — true for existing, false for missing - `TestCopy` — verify content matches after copy - `TestPing` — succeeds with valid root, fails with invalid Use `t.TempDir()` for test isolation. ## Acceptance criteria - [ ] `FilesystemStore` implements `FileStore` interface - [ ] Atomic writes prevent partial files - [ ] SHA-256 checksum computed on `Put` - [ ] All unit tests pass - [ ] Works with all 5 object key patterns listed above - [ ] `go build ./...` passes ## Priority P0 — blocks upload endpoint changes and data migration ## Depends on - #126 (FileStore interface) ## Part of Storage Migration: MinIO → PostgreSQL + Filesystem

forbes referenced this issue

2026-02-17 16:10:49 +00:00

feat(api): replace presigned uploads with direct upload for filesystem backend #129

forbes referenced this issue

2026-02-17 16:11:27 +00:00

feat(cli): data migration script — MinIO to filesystem #130

forbes referenced this issue

2026-02-17 16:11:52 +00:00

docs: update configuration and deployment docs for filesystem storage backend #131

forbes referenced this issue

2026-02-17 16:12:30 +00:00

feat(storage): ensure thumbnails use FileStore interface #132