lardon3d/docs/architecture/persistence.md

Lardon3D Persistence and Project Database

Current authority

The current Project DB schema is v25.

CURRENT_PROJECT_DB_SCHEMA=v25

v22  selected scientific execution foundation        PASS/FROZEN
v23  generic optical-context overlay                  IMPLEMENTED/VALIDATED/REVIEWED
v24  raw.develop.batch/1 persistence                  IMPLEMENTED/VALIDATED
v25  features.extract.batch/1 persistence             IMPLEMENTED/VALIDATED

Project DB evolves additively. Older schema versions remain valid historical contracts when a section explicitly documents the state published by that version. They must not be rewritten as if they had always contained later overlays.

The detailed schema, migration ledger and identity contracts are owned by Project Database. This document owns the persistence model and publication, checkpoint, recovery and artifact-boundary rules.

Persistence model

Lardon3D keeps queryable logical state in SQLite Project DB while large numerical payloads remain in bounded external files or artifacts designed for their format.

The split is intentional:

• identity, lifecycle state, relations, durable Task payloads and publication metadata -> SQLite;
• large descriptors, match payloads, checkpoints and other numerical artifacts -> bounded files;
• a file path is a storage locator after identity resolution, never scientific identity by itself.

Project DB does not serialize runtime-only objects such as mutexes, condition variables, callbacks, userdata pointers, worker threads, Governor reservations or live execution contracts.

Core invariants

• durable publication is explicit and ordered;
• partial artifacts are never accepted as complete scientific outputs;
• recovery begins from the last durable boundary the owning contract can prove;
• SQLite state and external files are not falsely described as one distributed transaction;
• immutable scientific identity is not inferred from path, basename, timestamp or operational IDs;
• large collections are read and written through bounded interfaces;
• retry behavior must converge only from identities already established by the owning contract;
• Project DB schema migration never silently creates new scientific meaning for historical rows.

Logical and binary publication

The generic publication shape is:

bounded computation
-> operation-owned temporary output
-> format/content validation
-> atomic file publication where required
-> Project DB transaction
-> READY / published durable state

The exact ordering is owned by the subsystem. Some contracts publish an immutable file before the SQLite transaction and therefore explicitly admit an orphan-file window. Others publish metadata and files through a more specialized ordered protocol. No document may strengthen a subsystem guarantee beyond the actual implementation.

Failure cleanup may remove only resources owned by the current operation. Shared immutable assets are not deleted merely because a later metadata transaction fails.

Durable Task checkpoint v1

Durable state

The versioned Task checkpoint contains bounded logical Task state only. It includes the stable Task identity, name, immutable estimate, observed state, recovery state, progress, message, timestamps and sequence count. It does not embed large scientific artifacts.

Task kind and typed business payload are separate concerns. The generic checkpoint format v1 does not become a miniature Project DB and does not replace typed Project DB persistence.

Process-stop normalization

Observed state	Restored state
TASK_PENDING	TASK_PENDING
TASK_RUNNING	TASK_PENDING
TASK_PAUSED	TASK_PENDING
TASK_COMPLETED	TASK_COMPLETED
TASK_FAILED	TASK_FAILED
TASK_CANCELLED	TASK_CANCELLED

A sequence break is not a persistent Task state. If execution stopped while a sequence was active, recovery returns the Task to TASK_PENDING; its durable sequence count remains retained and a new Governor admission is required before execution resumes.

Standalone checkpoint file

The v1 codec is bounded and field-encoded with explicit magic/version/size/checksum semantics. It does not serialize native structs or native padding.

Standalone publication uses a unique temporary file in the same directory, synchronizes the file, renames atomically and then synchronizes the parent directory.

The result distinguishes three important boundaries:

• before successful rename: failure leaves the previously published checkpoint unchanged;
• after successful rename: the new visible checkpoint is published and is not presented as rolled back;
• parent-directory sync failure after rename: result is PUBLISHED_NOT_DURABLE; the visible file is valid, but name persistence across crash/power loss is not guaranteed by Lardon3D.

OK means both content and directory-entry synchronization completed, subject to the guarantees of the filesystem and storage stack.

Persistent sizes are rejected before conversion if they exceed the local representable domain. The v1 timestamp representation likewise requires values representable by the local time_t before runtime conversion.

Project-owned checkpoint protocol

The project checkpoint protocol coordinates a Task file and SQLite metadata without claiming a true filesystem+SQLite transaction.

The current project-level ordering is:

capture bounded Task snapshot
-> publish .chk.next
-> record Task/checkpoint summary in SQLite
-> promote .chk.next to canonical .chk under .chk.lock

The canonical location is:

.lardon3d/checkpoints/<task_id>.chk

The advisory .chk.lock exists only as process synchronization; it is not recovery data.

Recovery obtains the lock, reloads the relevant Project DB record because it may have changed while waiting, and then selects a codec/version-valid checkpoint whose stored summary matches the Project DB summary exactly for the fields Project DB owns.

A valid canonical .chk has priority. A valid matching .chk.next may be promoted when the canonical file does not match. Stale or corrupt .next files never override a valid canonical checkpoint.

A missing, corrupt, future-version or summary-mismatched checkpoint makes that Task non-recoverable; it does not make unrelated project state invalid.

Project DB foundation

Project DB is SQLite with explicit schema versioning and sequential transactional migrations.

The retained configuration uses:

foreign_keys=ON
journal_mode=DELETE
synchronous=FULL
busy_timeout=5000

DELETE journal mode matches the current single-owner Project DB model and avoids persistent WAL/SHM files. The timeout bounds waiting on an external lock.

The project identity is duplicated intentionally across project.ini and the project table and must match. A divergence is an error, not an opportunity to invent a new identity.

Published catalog and scientific IDs use SQLite integer identities according to their owning schema. Where AUTOINCREMENT is part of the contract, a committed published identity is not later reused for a different object. An ID allocated only inside a rolled-back transaction is not a published identity.

Historical Project DB v7 foundation

Project DB v7 is the historical persistent project/runtime foundation. It covers the durable project identity, Tasks, checkpoint references, generic artifacts, ScanSets, logical images, image assets, Feature Store metadata and the segmented Visual Index foundation.

Important v7-era persistence rules remain current unless a later contract explicitly supersedes them:

• Task summary plus checkpoint reference is a single SQLite transaction;
• large checkpoint and scientific files remain external;
• an artifact file is published and validated before Project DB marks it READY;
• Project DB stores bounded metadata and references, not large descriptor/posting payloads;
• AUTOINCREMENT is used where published catalog/scientific IDs must not be recycled after committed deletion;
• the persistent import source path records durable execution intent, while imported scientific data ultimately depends on the managed immutable asset rather than the original external source path.

Generic artifact orphan window

For generic file-first publication, a successful file publication followed by SQLite BUSY or another SQLite failure leaves a valid orphan file on disk while Project DB retains its previous truth.

That file is not silently deleted. Global orphan reconciliation is a separate maintenance capability. The absence of such reconciliation does not justify pretending the file and SQLite update were atomic.

Import and managed source assets

import.images persists its source path and ScanSet identity and checkpoints after validated bounded work. Managed image assets are content-addressed by SHA-256 after a complete bounded copy/hash pass.

A concurrently existing asset is adopted only after the implementation verifies the expected content and size according to the owning contract. SQLite publication follows file publication.

manifest.tsv remains a historical/legacy projection. SQLite is the canonical logical commit for the persistent catalog. The legacy_image_catalog_pending marker means legacy data may remain outside the current catalog model; it does not claim that historical manifest rows were silently converted into fully proven catalog identities.

Selective Task recovery

Project open performs bounded selective recovery using Project DB pages and the Task Kind Registry.

A recoverable Task must have:

• a valid Project DB Task record;
• a supported Task kind and version;
• a valid coherent checkpoint boundary required by that kind;
• reconstructable typed business persistence where the kind requires it.

The recovery scan copies records outside the SQLite mutex before business reconstruction and enqueue. A full Queue window stops the scan without mutating the unvisited records.

Project-level schema, migration and project-identity errors are fatal to opening the project. Per-Task legacy, unknown-kind, unsupported-version, missing-checkpoint, invalid-checkpoint, unavailable-source or reconstruction errors are isolated to the affected Task where the owning contract permits.

External scientific artifacts

Feature Store

Feature files are immutable external artifacts with versioned bounded readers. Project DB stores their identity and publication metadata only after the external file is valid according to the Feature Store contract.

Feature descriptors are not duplicated into SQLite.

Visual Index

Visual Index postings remain in bounded immutable external segments. Project DB stores index identity, segment metadata and memberships. Durable update Tasks retain the bounded cursor required to resume publication without rebuilding already accepted segments.

Match and later scientific payloads

Match payloads and other large scientific representations follow the same architectural principle: SQLite owns durable identity, relations and bounded metadata; specialized external formats own large numerical payloads where the subsystem contract requires them.

Project DB v16-v22 retained scientific foundation

The scientific and persistence contracts introduced through v16-v22 remain historical PASS/FROZEN foundations. Later schema versions are additive overlays and do not reinterpret those rows.

In particular:

• v16 publishes the immutable Sparse SfM persistence model;
• v17 adds the durable typed Sparse SfM Task payload for Gate F;
• v18 adds Phase H v1 incremental-reconstruction identity/persistence;
• v19 adds Capture / Asset Provenance v1;
• v20 adds durable acquisition-campaign Task persistence;
• v21 adds Photo Quality Triage persistence;
• v22 adds selected scientific execution, explicit Capture SOURCE-asset mapping and durable selected RAW development persistence.

Detailed tables, identities and migration invariants are defined in Project Database.

Project DB v23 - generic optical-context overlay

Project DB v23 is an additive optical-context overlay above the v22 scientific foundation.

It separates four identities:

• camera body profile;
• lens profile;
• optical configuration;
• optical calibration profile.

The migration creates the optical relations empty. It does not inspect EXIF, path, basename, SHA-256, dimensions, device name or historical calibration to backfill identity.

Electronic metadata aliases, when present, use exact stored identity semantics. A manual lens without EXIF is a normal explicit profile and does not require a fabricated metadata alias.

Calibration selection is explicit and requires exact compatible optical configuration. No silent interpolation, substitution or inferred calibration identity is introduced by persistence.

Historical S21/A6000 copies migrated through the optical overlay without changing their existing scientific rows; empty optical tables remain an honest state until explicit data is supplied.

Project DB v24 - RAW batch persistence

Project DB v24 adds only the typed durable relation required by raw.develop.batch/1:

raw_development_batch_tasks(task_id, selected_execution_id)

The migration is additive and DDL-only. It does not create Capture, Asset, Image or selected-execution identity and does not rewrite historical raw.develop/1 Tasks.

The selected execution remains the durable scientific ordering authority. The batch Task may prepare independent RAW items concurrently under Governor admission, but after all participants join, the owner publishes selected representations in deterministic selected-item order.

The durable ordering remains:

prepare bounded independent RAW items
-> join participants
-> owner publishes exact selected item
-> selected item/cursor commit
-> generic Task progress/checkpoint

A crash may therefore leave generic Task progress behind already durable selected scientific state; it must never move generic progress ahead of unpublished selected state. Recovery resumes from durable selected-execution identity/cursor and exact already-published representations rather than guessing from paths or files.

The retained real A6000 run completed this RAW-batch path for all 689 selected RAW representations.

Project DB v25 - Feature batch persistence

Project DB v25 is the current schema head. It adds only the typed durable relation required by features.extract.batch/1 through feature_extract_batch_tasks.

The v25 row binds one durable Feature-batch Task to:

• one immutable selected execution;
• the monotone selected-item prefix/cursor;
• the exact ORB extractor kind/version/parameters/fingerprint domain required by the Task contract.

The migration is additive and DDL-only. It creates no Feature Set, converts no historical features.extract/1 Task and infers no Image or Feature Set identity.

Selected images may be prepared concurrently without SQLite access. After participants join, the owner publishes Feature Sets in selected order and advances the durable Feature cursor only when the exact READY Feature Set is already durable.

A crash may leave generic Task checkpoint/progress or the Feature cursor behind an immutable Feature Set that was already published. Recovery must revalidate and reuse that exact result; it must not infer a replacement identity.

The retained real A6000 proof completed the v25 path with 689 Feature Sets and then continued through Visual Index, 38,420 Candidate Pairs, 38,420 Match Results, Geometric Verification and Tracks without replaying acquisition, RAW or Feature work.

This persistence result does not imply that real Sparse SfM or Dense/MVS was executed. Those counts remain zero in REAL_A6000_PRE_SFM=PASS/FROZEN.

Capture / Asset / Image identity boundary

Persistence must preserve these distinctions:

Capture != file
Capture != Asset
Capture != image_id
Capture != SHA-256
Capture != path
Capture != basename
Capture != Task ID
Capture != campaign group ID

asset_id identifies a managed immutable asset record. SHA-256 identifies immutable bytes according to the asset contract. image_id identifies a scientific image representation. capture_id identifies a physical acquisition representation in Project DB. Task and campaign-group IDs remain operational identities.

Retry and recovery may use an explicit persisted mapping between these domains only where a canonical contract defines it. They may not reconstruct missing identity from coincidental equality or metadata.

Acquisition-campaign crash/restart ordering

For durable campaign execution, the important persistence boundary is conceptually:

S3-E returns capture_id
-> persist group_id -> capture_id mapping and campaign cursor
-> advance generic Task progress/checkpoint
-> next group may execute

The retained pre-return S3-E crash window remains intentional. If a Capture is created internally but the process dies before the caller receives and durably retains its capture_id, campaign persistence does not guess that identity from path, digest, basename, timestamp, metadata or image_id.

Concurrency and ownership

A Project DB connection is serialized by its internal mutex. Each public compound operation owns its entire transaction; a public transaction is not left open across calls.

External artifact I/O is not performed while holding the Project DB mutex where the owning contract separates those operations. SQLite-owned strings and records are copied into caller-owned bounded storage before returning.

Closing Project DB concurrently with an active Project DB call is forbidden by owner lifetime rules. Application project/session teardown must first destroy and join the Queue so Task callbacks and leases are finished, then close Project DB.

Schema migration discipline

Project DB migrations are sequential, transactional and additive unless an explicit future human ticket authorizes a different migration.

Current known sequence:

v1 -> ... -> v22
v22 -> v23  generic optical-context overlay
v23 -> v24  RAW batch Task persistence
v24 -> v25  Feature batch Task persistence

A migration failure rolls back both its new schema objects and its schema-version publication marker. A retry therefore starts from the previous complete known schema.

Future schema versions beyond v25 are rejected by the current implementation and require explicit human authorization before code or documentation may treat them as current.

Current status

PERSISTENCE_DOCUMENT=CURRENT
CURRENT_PROJECT_DB_SCHEMA=v25

TASK_CHECKPOINT_V1                         IMPLEMENTED
PROJECT_CHECKPOINT_PROTOCOL                IMPLEMENTED/VALIDATED
PROJECT_DB_V7_FOUNDATION                   IMPLEMENTED
PROJECT_DB_V16_TO_V22_FOUNDATION           PASS/FROZEN
PROJECT_DB_V23_OPTICAL_OVERLAY              IMPLEMENTED/VALIDATED/REVIEWED
PROJECT_DB_V24_RAW_BATCH                    IMPLEMENTED/VALIDATED
PROJECT_DB_V25_FEATURE_BATCH                IMPLEMENTED/VALIDATED
REAL_A6000_PRE_SFM                          PASS/FROZEN

Current intentionally unfinished persistence-adjacent work includes global orphan-file reconciliation, asset scrub/reconciliation and a general dependency/DAG recovery model. Those are separate future capabilities; they do not change the current v25 schema authority.