kindred/silo
1
1
Fork 0
Code Issues 30 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(api): replace presigned uploads with direct upload for filesystem backend #129

New Issue
Closed
opened 2026-02-17 16:10:49 +00:00 by forbes · 0 comments
forbes commented 2026-02-17 16:10:49 +00:00
Owner
Copy Link

Summary

Adapt the file upload/download flow so it works without MinIO presigned URLs when using the filesystem backend.

Context

Current upload flow (MinIO presigned)

  1. Browser calls POST /api/uploads/presign with {filename, content_type, size}
  2. Server (HandlePresignUpload in file_handlers.go) generates a presigned PUT URL via s.storage.PresignPut() and returns {object_key, upload_url, expires_at}
  3. Browser (useFileUpload.ts) PUTs file bytes directly to the MinIO presigned URL using XMLHttpRequest (with progress tracking)
  4. Browser calls POST /api/items/{partNumber}/files with {object_key, filename, content_type, size}
  5. Server (HandleAssociateItemFile) copies from temp key to permanent key via s.storage.Copy(), deletes temp, creates DB record

Current revision upload flow (already direct)

POST /api/items/{partNumber}/file (HandleUploadFile in handlers.go) already accepts multipart form upload directly — no presigning needed. This path works as-is with the filesystem backend.

Problem

The filesystem backend cannot generate presigned URLs (no HTTP server on the storage layer). The presign → direct-PUT → associate flow breaks.

Requirements

Option A: Direct multipart upload (recommended)

Modify the item file attachment flow to support direct multipart upload through the Go server, similar to how HandleUploadFile already works for revisions:

  1. Add a new handler or modify HandleAssociateItemFile to accept multipart/form-data
  2. Read the file from the multipart body
  3. Write directly to storage via FileStore.Put()
  4. Create the DB record
  5. Return the file metadata

The presigned flow can remain available when storage.backend == "minio" for backward compatibility.

Frontend changes (web/src/hooks/useFileUpload.ts)

  • When presign endpoint returns an error or is unavailable, fall back to direct POST with multipart/form-data
  • Alternatively, always use direct upload (simpler) — the Go server handles the write
  • Progress tracking: XMLHttpRequest or fetch with ReadableStream for upload progress

Download path

HandleDownloadFile in handlers.go already streams from s.storage.Get() / s.storage.GetVersion() — this works as-is with the filesystem backend.

For item file attachments, add a download handler if one doesn't exist (currently only list/create/delete for item files — no download endpoint).

Routes to add/modify

Current storage-related routes in internal/api/routes.go:

POST   /api/uploads/presign                    → HandlePresignUpload
GET    /api/items/{partNumber}/files            → HandleListItemFiles
POST   /api/items/{partNumber}/files            → HandleAssociateItemFile
DELETE /api/items/{partNumber}/files/{fileId}    → HandleDeleteItemFile
PUT    /api/items/{partNumber}/thumbnail        → HandleSetItemThumbnail
POST   /api/items/{partNumber}/file             → HandleUploadFile (multipart, revision)
GET    /api/items/{partNumber}/file             → HandleDownloadLatestFile
GET    /api/items/{partNumber}/file/{revision}  → HandleDownloadFile

Changes needed:

  • POST /api/items/{partNumber}/files — accept both JSON (existing presigned flow) and multipart/form-data (new direct upload)
  • GET /api/items/{partNumber}/files/{fileId}/download — new endpoint for downloading attached files
  • POST /api/uploads/presign — return 501 or skip when filesystem backend is active

Thumbnail upload

HandleSetItemThumbnail currently accepts {object_key} referencing a pre-uploaded temp file. Same pattern as item files — needs a direct upload path or accept multipart/form-data.

Acceptance criteria

  • Item file attachments can be uploaded directly (multipart) without presigning
  • Item file attachments can be downloaded
  • Thumbnail upload works without presigning
  • Frontend useFileUpload.ts supports direct upload path
  • Revision upload (HandleUploadFile) continues working as-is
  • Revision download (HandleDownloadFile) continues working as-is
  • Presigned flow still works when MinIO backend is configured (backward compat)

Priority

P1

Depends on

  • #126 (FileStore interface)
  • #127 (filesystem backend)

Part of

Storage Migration: MinIO → PostgreSQL + Filesystem

## Summary Adapt the file upload/download flow so it works without MinIO presigned URLs when using the filesystem backend. ## Context ### Current upload flow (MinIO presigned) 1. **Browser** calls `POST /api/uploads/presign` with `{filename, content_type, size}` 2. **Server** (`HandlePresignUpload` in `file_handlers.go`) generates a presigned PUT URL via `s.storage.PresignPut()` and returns `{object_key, upload_url, expires_at}` 3. **Browser** (`useFileUpload.ts`) PUTs file bytes directly to the MinIO presigned URL using `XMLHttpRequest` (with progress tracking) 4. **Browser** calls `POST /api/items/{partNumber}/files` with `{object_key, filename, content_type, size}` 5. **Server** (`HandleAssociateItemFile`) copies from temp key to permanent key via `s.storage.Copy()`, deletes temp, creates DB record ### Current revision upload flow (already direct) `POST /api/items/{partNumber}/file` (`HandleUploadFile` in `handlers.go`) already accepts **multipart form upload** directly — no presigning needed. This path works as-is with the filesystem backend. ### Problem The filesystem backend cannot generate presigned URLs (no HTTP server on the storage layer). The presign → direct-PUT → associate flow breaks. ## Requirements ### Option A: Direct multipart upload (recommended) Modify the item file attachment flow to support direct multipart upload through the Go server, similar to how `HandleUploadFile` already works for revisions: 1. Add a new handler or modify `HandleAssociateItemFile` to accept `multipart/form-data` 2. Read the file from the multipart body 3. Write directly to storage via `FileStore.Put()` 4. Create the DB record 5. Return the file metadata The presigned flow can remain available when `storage.backend == "minio"` for backward compatibility. ### Frontend changes (`web/src/hooks/useFileUpload.ts`) - When presign endpoint returns an error or is unavailable, fall back to direct `POST` with `multipart/form-data` - Alternatively, always use direct upload (simpler) — the Go server handles the write - Progress tracking: `XMLHttpRequest` or `fetch` with `ReadableStream` for upload progress ### Download path `HandleDownloadFile` in `handlers.go` already streams from `s.storage.Get()` / `s.storage.GetVersion()` — this works as-is with the filesystem backend. For item file attachments, add a download handler if one doesn't exist (currently only list/create/delete for item files — no download endpoint). ### Routes to add/modify Current storage-related routes in `internal/api/routes.go`: ``` POST /api/uploads/presign → HandlePresignUpload GET /api/items/{partNumber}/files → HandleListItemFiles POST /api/items/{partNumber}/files → HandleAssociateItemFile DELETE /api/items/{partNumber}/files/{fileId} → HandleDeleteItemFile PUT /api/items/{partNumber}/thumbnail → HandleSetItemThumbnail POST /api/items/{partNumber}/file → HandleUploadFile (multipart, revision) GET /api/items/{partNumber}/file → HandleDownloadLatestFile GET /api/items/{partNumber}/file/{revision} → HandleDownloadFile ``` Changes needed: - `POST /api/items/{partNumber}/files` — accept both JSON (existing presigned flow) and `multipart/form-data` (new direct upload) - `GET /api/items/{partNumber}/files/{fileId}/download` — new endpoint for downloading attached files - `POST /api/uploads/presign` — return 501 or skip when filesystem backend is active ### Thumbnail upload `HandleSetItemThumbnail` currently accepts `{object_key}` referencing a pre-uploaded temp file. Same pattern as item files — needs a direct upload path or accept `multipart/form-data`. ## Acceptance criteria - [ ] Item file attachments can be uploaded directly (multipart) without presigning - [ ] Item file attachments can be downloaded - [ ] Thumbnail upload works without presigning - [ ] Frontend `useFileUpload.ts` supports direct upload path - [ ] Revision upload (`HandleUploadFile`) continues working as-is - [ ] Revision download (`HandleDownloadFile`) continues working as-is - [ ] Presigned flow still works when MinIO backend is configured (backward compat) ## Priority P1 ## Depends on - #126 (FileStore interface) - #127 (filesystem backend) ## Part of Storage Migration: MinIO → PostgreSQL + Filesystem
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
forbes
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: kindred/silo#129
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?