kindred/silo
1
1
Fork 0
Code Issues 30 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sessions): checkpoint system with diff-based storage #165

New Issue
Open
opened 2026-03-01 15:38:10 +00:00 by forbes · 0 comments
forbes commented 2026-03-01 15:38:10 +00:00
Owner
Copy Link

Context

Sub-issue of #125 (Context-Aware Part Subscription System).

A checkpoint is a lightweight working snapshot pushed on upward context transitions (exiting sketch, leaving body editing, etc.). Checkpoints are distinct from formal revisions — they are ephemeral, TTL-limited, and serve as the bridge between edit session release and subscriber notification.

Key Design: Diff-Based Storage

For .kc files containing XML parametric source, checkpoints should store diffs rather than full file copies when possible. The client computes the diff against its base revision; the server stores the patch. This dramatically reduces storage for incremental edits.

Full file fallback for binary content or when diff is unavailable.

1. Database Migration

Create migrations/024_checkpoints.sql:

CREATE TABLE checkpoints (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    item_id         UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE,
    user_id         UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    base_revision   INT NOT NULL,
    context         TEXT NOT NULL DEFAULT 'manual'
                    CHECK (context IN ('sketch_close', 'body_close', 'assembly_close', 'manual', 'autosave')),
    storage_mode    TEXT NOT NULL DEFAULT 'full'
                    CHECK (storage_mode IN ('full', 'diff')),
    file_key        TEXT NOT NULL,
    file_hash       TEXT NOT NULL,
    file_size       BIGINT,
    dag_synced      BOOLEAN NOT NULL DEFAULT false,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    expires_at      TIMESTAMPTZ NOT NULL
);

CREATE INDEX idx_checkpoints_item ON checkpoints(item_id);
CREATE INDEX idx_checkpoints_user ON checkpoints(user_id);
CREATE INDEX idx_checkpoints_expires ON checkpoints(expires_at);

storage_mode indicates whether file_key points to a full file or a diff patch.
base_revision records which revision the diff is against (needed to reconstruct).

2. Storage Layout

items/{pn}/checkpoints/{user_id}/{timestamp}.patch   # diff mode
items/{pn}/checkpoints/{user_id}/{timestamp}.kc      # full mode

3. Database Layer

Create internal/db/checkpoints.go:

Methods:

  • Create(ctx, checkpoint) — insert, enforce max-per-item-per-user
  • GetLatestForItem(ctx, itemID) — most recent checkpoint
  • GetLatestForItemUser(ctx, itemID, userID) — most recent by this user
  • ListForItem(ctx, itemID) — all active checkpoints
  • Delete(ctx, id) — single delete
  • DeleteForItemUser(ctx, itemID, userID) — cleanup on formal revision commit
  • DeleteExpired(ctx) — TTL sweeper, returns file_keys for filesystem cleanup

4. API Endpoints

Create checkpoint handlers:

Method Path Auth Description
POST /api/items/{pn}/checkpoints editor Push checkpoint
GET /api/items/{pn}/checkpoints viewer List active checkpoints
GET /api/items/{pn}/checkpoints/latest viewer Latest checkpoint with download info
DELETE /api/items/{pn}/checkpoints/{id} editor Delete own checkpoint

Push Request

Multipart or two-step:

Option A (inline for small diffs):

{
  "context": "body_close",
  "base_revision": 3,
  "storage_mode": "diff",
  "session_id": "uuid_to_release",
  "dag": { "nodes": [...], "edges": [...] }
}

With the diff/file content as a separate part or pre-uploaded.

Option B (pre-upload):

  1. Upload file/diff via POST /api/items/{pn}/checkpoints/upload (returns file_key)
  2. Commit checkpoint metadata with file_key reference

Push Processing

  1. Store file/diff to filesystem
  2. Insert checkpoint row
  3. If dag present: call SyncFeatureTree(), mark dirty nodes, emit dag.updated
  4. If session_id present: release that edit session
  5. Enforce checkpoint_max_per_item per user (delete oldest)
  6. Emit SSE events based on context:
    • sketch_close → no subscription events (too granular)
    • body_close, assembly_close, manual → emit to latest subscribers (future)
    • autosave → store only, no events

Push Response

{
  "checkpoint_id": "uuid",
  "context": "body_close",
  "dag_synced": true,
  "session_released": true
}

5. Cleanup Sweeper

Background goroutine (can share the session sweeper tick):

  • Every hour: DELETE FROM checkpoints WHERE expires_at < now()
  • Delete corresponding files from filesystem
  • On formal revision commit for an item: delete that user's checkpoints for the item

6. Context-to-Action Mapping

Context Sub Events DAG Update Session Release
sketch_close None Yes Sketch session only
body_close latest subs Yes Body session
assembly_close latest subs Yes Assembly session
manual latest subs Yes None (explicit)
autosave None Optional None

Acceptance Criteria

  • Migration creates checkpoints table
  • POST /checkpoints stores file or diff to filesystem
  • storage_mode: "diff" stores a patch file; "full" stores complete file
  • Push with session_id atomically releases the edit session
  • Push with dag runs SyncFeatureTree() and marks dirty
  • checkpoint_max_per_item enforced per user (oldest evicted)
  • Expired checkpoints cleaned up by sweeper (DB + filesystem)
  • Formal revision commit deletes user's checkpoints for that item
  • GET /checkpoints/latest returns most recent with download path

Depends On

  • #161 (workstations)
  • #163 (edit sessions — for session_id release)

Part Of

#125

## Context Sub-issue of #125 (Context-Aware Part Subscription System). A checkpoint is a lightweight working snapshot pushed on upward context transitions (exiting sketch, leaving body editing, etc.). Checkpoints are distinct from formal revisions — they are ephemeral, TTL-limited, and serve as the bridge between edit session release and subscriber notification. ## Key Design: Diff-Based Storage For `.kc` files containing XML parametric source, checkpoints should store **diffs** rather than full file copies when possible. The client computes the diff against its base revision; the server stores the patch. This dramatically reduces storage for incremental edits. Full file fallback for binary content or when diff is unavailable. ## 1. Database Migration Create `migrations/024_checkpoints.sql`: ```sql CREATE TABLE checkpoints ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), item_id UUID NOT NULL REFERENCES items(id) ON DELETE CASCADE, user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE, base_revision INT NOT NULL, context TEXT NOT NULL DEFAULT 'manual' CHECK (context IN ('sketch_close', 'body_close', 'assembly_close', 'manual', 'autosave')), storage_mode TEXT NOT NULL DEFAULT 'full' CHECK (storage_mode IN ('full', 'diff')), file_key TEXT NOT NULL, file_hash TEXT NOT NULL, file_size BIGINT, dag_synced BOOLEAN NOT NULL DEFAULT false, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), expires_at TIMESTAMPTZ NOT NULL ); CREATE INDEX idx_checkpoints_item ON checkpoints(item_id); CREATE INDEX idx_checkpoints_user ON checkpoints(user_id); CREATE INDEX idx_checkpoints_expires ON checkpoints(expires_at); ``` `storage_mode` indicates whether `file_key` points to a full file or a diff patch. `base_revision` records which revision the diff is against (needed to reconstruct). ## 2. Storage Layout ``` items/{pn}/checkpoints/{user_id}/{timestamp}.patch # diff mode items/{pn}/checkpoints/{user_id}/{timestamp}.kc # full mode ``` ## 3. Database Layer Create `internal/db/checkpoints.go`: Methods: - `Create(ctx, checkpoint)` — insert, enforce max-per-item-per-user - `GetLatestForItem(ctx, itemID)` — most recent checkpoint - `GetLatestForItemUser(ctx, itemID, userID)` — most recent by this user - `ListForItem(ctx, itemID)` — all active checkpoints - `Delete(ctx, id)` — single delete - `DeleteForItemUser(ctx, itemID, userID)` — cleanup on formal revision commit - `DeleteExpired(ctx)` — TTL sweeper, returns file_keys for filesystem cleanup ## 4. API Endpoints Create checkpoint handlers: | Method | Path | Auth | Description | |--------|------|------|-------------| | POST | `/api/items/{pn}/checkpoints` | editor | Push checkpoint | | GET | `/api/items/{pn}/checkpoints` | viewer | List active checkpoints | | GET | `/api/items/{pn}/checkpoints/latest` | viewer | Latest checkpoint with download info | | DELETE | `/api/items/{pn}/checkpoints/{id}` | editor | Delete own checkpoint | ### Push Request Multipart or two-step: **Option A (inline for small diffs):** ```json { "context": "body_close", "base_revision": 3, "storage_mode": "diff", "session_id": "uuid_to_release", "dag": { "nodes": [...], "edges": [...] } } ``` With the diff/file content as a separate part or pre-uploaded. **Option B (pre-upload):** 1. Upload file/diff via `POST /api/items/{pn}/checkpoints/upload` (returns file_key) 2. Commit checkpoint metadata with file_key reference ### Push Processing 1. Store file/diff to filesystem 2. Insert checkpoint row 3. If `dag` present: call `SyncFeatureTree()`, mark dirty nodes, emit `dag.updated` 4. If `session_id` present: release that edit session 5. Enforce `checkpoint_max_per_item` per user (delete oldest) 6. Emit SSE events based on context: - `sketch_close` → no subscription events (too granular) - `body_close`, `assembly_close`, `manual` → emit to `latest` subscribers (future) - `autosave` → store only, no events ### Push Response ```json { "checkpoint_id": "uuid", "context": "body_close", "dag_synced": true, "session_released": true } ``` ## 5. Cleanup Sweeper Background goroutine (can share the session sweeper tick): - Every hour: `DELETE FROM checkpoints WHERE expires_at < now()` - Delete corresponding files from filesystem - On formal revision commit for an item: delete that user's checkpoints for the item ## 6. Context-to-Action Mapping | Context | Sub Events | DAG Update | Session Release | |---------|-----------|------------|----------------| | `sketch_close` | None | Yes | Sketch session only | | `body_close` | `latest` subs | Yes | Body session | | `assembly_close` | `latest` subs | Yes | Assembly session | | `manual` | `latest` subs | Yes | None (explicit) | | `autosave` | None | Optional | None | ## Acceptance Criteria - [ ] Migration creates `checkpoints` table - [ ] `POST /checkpoints` stores file or diff to filesystem - [ ] `storage_mode: "diff"` stores a patch file; `"full"` stores complete file - [ ] Push with `session_id` atomically releases the edit session - [ ] Push with `dag` runs `SyncFeatureTree()` and marks dirty - [ ] `checkpoint_max_per_item` enforced per user (oldest evicted) - [ ] Expired checkpoints cleaned up by sweeper (DB + filesystem) - [ ] Formal revision commit deletes user's checkpoints for that item - [ ] `GET /checkpoints/latest` returns most recent with download path ## Depends On - #161 (workstations) - #163 (edit sessions — for session_id release) ## Part Of #125
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
forbes
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: kindred/silo#165
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?