docs: reconcile persistence with schema v25
This commit is contained in:
fy59
parent
2d2bf31328
commit
4522602421
1 changed files with 385 additions and 186 deletions
|
|
@ -1,80 +1,86 @@
|
||||||
# Persistance et base de données Lardon3D
|
# Lardon3D Persistence and Project Database
|
||||||
|
|
||||||
## Vision
|
## Current authority
|
||||||
|
|
||||||
Lardon3D stocke les métadonnées de reconstruction dans Project DB SQLite,
|
The current Project DB schema is **v25**.
|
||||||
tandis que les données numériques massives restent dans des fichiers/binaires
|
|
||||||
adaptés. Le schéma courant est v23 ; les sections v7 ci-dessous documentent la
|
|
||||||
fondation historique sans prétendre être la tête de migration.
|
|
||||||
|
|
||||||
## Principes fondamentaux
|
```text
|
||||||
|
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||||
|
|
||||||
### Séparation logique/binaire
|
v22 selected scientific execution foundation PASS/FROZEN
|
||||||
- État logique, relations, index → base persistante légère
|
v23 generic optical-context overlay IMPLEMENTED/VALIDATED/REVIEWED
|
||||||
- Données numériques massives → fichiers/artefacts binaires adaptés
|
v24 raw.develop.batch/1 persistence IMPLEMENTED/VALIDATED
|
||||||
|
v25 features.extract.batch/1 persistence IMPLEMENTED/VALIDATED
|
||||||
### Cycle de publication
|
|
||||||
```
|
|
||||||
lot calculé
|
|
||||||
→ artefact temporaire
|
|
||||||
→ validation
|
|
||||||
→ publication atomique
|
|
||||||
→ transaction de métadonnées
|
|
||||||
→ état READY
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Règle de reprise
|
Project DB evolves additively. Older schema versions remain valid historical contracts when a
|
||||||
Une reprise ne considère jamais un artefact partiellement publié comme valide.
|
section explicitly documents the state published by that version. They must not be rewritten as if
|
||||||
|
they had always contained later overlays.
|
||||||
|
|
||||||
## Concepts de domaine
|
The detailed schema, migration ledger and identity contracts are owned by
|
||||||
|
[Project Database](project_database.md). This document owns the persistence model and publication,
|
||||||
|
checkpoint, recovery and artifact-boundary rules.
|
||||||
|
|
||||||
Les éléments suivants sont des concepts de domaine, PAS des tables SQL imposées :
|
## Persistence model
|
||||||
|
|
||||||
- project
|
Lardon3D keeps queryable logical state in SQLite Project DB while large numerical payloads remain in
|
||||||
- scan_set
|
bounded external files or artifacts designed for their format.
|
||||||
- image
|
|
||||||
- feature_set
|
|
||||||
- visual_signature
|
|
||||||
- candidate_pair
|
|
||||||
- verified_pair
|
|
||||||
- track
|
|
||||||
- observation
|
|
||||||
- camera
|
|
||||||
- camera_body_profile
|
|
||||||
- lens_profile
|
|
||||||
- optical_configuration
|
|
||||||
- optical_calibration_profile
|
|
||||||
- pose
|
|
||||||
- point3d
|
|
||||||
- reconstruction_layer
|
|
||||||
- measurement
|
|
||||||
- document_source
|
|
||||||
- geometric_constraint
|
|
||||||
- artifact
|
|
||||||
- checkpoint
|
|
||||||
|
|
||||||
## Invariants
|
The split is intentional:
|
||||||
|
|
||||||
- Chaque publication est atomique
|
- identity, lifecycle state, relations, durable Task payloads and publication metadata -> SQLite;
|
||||||
- Les artefacts partiels ne sont jamais considérés comme valides
|
- large descriptors, match payloads, checkpoints and other numerical artifacts -> bounded files;
|
||||||
- La reprise commence à la dernière frontière connue
|
- a file path is a storage locator after identity resolution, never scientific identity by itself.
|
||||||
|
|
||||||
## Checkpoint durable de tâche v1
|
Project DB does not serialize runtime-only objects such as mutexes, condition variables, callbacks,
|
||||||
|
userdata pointers, worker threads, Governor reservations or live execution contracts.
|
||||||
|
|
||||||
### État durable
|
## Core invariants
|
||||||
|
|
||||||
Le modèle durable versionné contient uniquement l'identifiant stable, le nom,
|
- durable publication is explicit and ordered;
|
||||||
l'estimation immuable, l'état observé, l'état de reprise, la progression, le
|
- partial artifacts are never accepted as complete scientific outputs;
|
||||||
message, les horodatages et le compteur de séquences. Il ne contient aucun gros
|
- recovery begins from the last durable boundary the owning contract can prove;
|
||||||
artefact numérique. Une future version pourra référencer des identifiants
|
- SQLite state and external files are not falsely described as one distributed transaction;
|
||||||
d'artefacts publiés et validés sans incorporer leur contenu.
|
- immutable scientific identity is not inferred from path, basename, timestamp or operational IDs;
|
||||||
|
- large collections are read and written through bounded interfaces;
|
||||||
|
- retry behavior must converge only from identities already established by the owning contract;
|
||||||
|
- Project DB schema migration never silently creates new scientific meaning for historical rows.
|
||||||
|
|
||||||
Les mutex, conditions, callbacks, userdata, workers, gouverneur, réservations et
|
## Logical and binary publication
|
||||||
contrats d'exécution sont transitoires et ne sont jamais sérialisés.
|
|
||||||
|
|
||||||
### Normalisation après arrêt de processus
|
The generic publication shape is:
|
||||||
|
|
||||||
| État observé | État restauré |
|
```text
|
||||||
|
bounded computation
|
||||||
|
-> operation-owned temporary output
|
||||||
|
-> format/content validation
|
||||||
|
-> atomic file publication where required
|
||||||
|
-> Project DB transaction
|
||||||
|
-> READY / published durable state
|
||||||
|
```
|
||||||
|
|
||||||
|
The exact ordering is owned by the subsystem. Some contracts publish an immutable file before the
|
||||||
|
SQLite transaction and therefore explicitly admit an orphan-file window. Others publish metadata and
|
||||||
|
files through a more specialized ordered protocol. No document may strengthen a subsystem guarantee
|
||||||
|
beyond the actual implementation.
|
||||||
|
|
||||||
|
Failure cleanup may remove only resources owned by the current operation. Shared immutable assets are
|
||||||
|
not deleted merely because a later metadata transaction fails.
|
||||||
|
|
||||||
|
## Durable Task checkpoint v1
|
||||||
|
|
||||||
|
### Durable state
|
||||||
|
|
||||||
|
The versioned Task checkpoint contains bounded logical Task state only. It includes the stable Task
|
||||||
|
identity, name, immutable estimate, observed state, recovery state, progress, message, timestamps and
|
||||||
|
sequence count. It does not embed large scientific artifacts.
|
||||||
|
|
||||||
|
Task kind and typed business payload are separate concerns. The generic checkpoint format v1 does not
|
||||||
|
become a miniature Project DB and does not replace typed Project DB persistence.
|
||||||
|
|
||||||
|
### Process-stop normalization
|
||||||
|
|
||||||
|
| Observed state | Restored state |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `TASK_PENDING` | `TASK_PENDING` |
|
| `TASK_PENDING` | `TASK_PENDING` |
|
||||||
| `TASK_RUNNING` | `TASK_PENDING` |
|
| `TASK_RUNNING` | `TASK_PENDING` |
|
||||||
|
|
@ -83,162 +89,355 @@ contrats d'exécution sont transitoires et ne sont jamais sérialisés.
|
||||||
| `TASK_FAILED` | `TASK_FAILED` |
|
| `TASK_FAILED` | `TASK_FAILED` |
|
||||||
| `TASK_CANCELLED` | `TASK_CANCELLED` |
|
| `TASK_CANCELLED` | `TASK_CANCELLED` |
|
||||||
|
|
||||||
Une rupture de séquence n'est pas un état : elle est observée comme
|
A sequence break is not a persistent Task state. If execution stopped while a sequence was active,
|
||||||
`TASK_RUNNING`. Son `sequence_count` est durable, mais la reprise revient à
|
recovery returns the Task to `TASK_PENDING`; its durable sequence count remains retained and a new
|
||||||
`TASK_PENDING` et exige une nouvelle admission.
|
Governor admission is required before execution resumes.
|
||||||
|
|
||||||
### Stockage minimal
|
### Standalone checkpoint file
|
||||||
|
|
||||||
Le codec v1 est indépendant de la future Project Database. Le fichier est de
|
The v1 codec is bounded and field-encoded with explicit magic/version/size/checksum semantics. It does
|
||||||
taille fixe et bornée, encodé champ par champ, avec magie, version, taille et
|
not serialize native structs or native padding.
|
||||||
checksum de payload.
|
|
||||||
La publication écrit un fichier temporaire unique dans le même répertoire,
|
|
||||||
effectue `fsync`, renomme atomiquement puis synchronise le répertoire parent.
|
|
||||||
La lecture distingue absence, corruption, version inconnue et erreur d'I/O.
|
|
||||||
|
|
||||||
La sauvegarde distingue trois frontières :
|
Standalone publication uses a unique temporary file in the same directory, synchronizes the file,
|
||||||
|
renames atomically and then synchronizes the parent directory.
|
||||||
|
|
||||||
- avant `rename`, toute erreur retourne `IO_ERROR`, supprime le temporaire et
|
The result distinguishes three important boundaries:
|
||||||
laisse l'ancien checkpoint publié inchangé ;
|
|
||||||
- après un `rename` réussi, le nouveau checkpoint est publié et visible et
|
|
||||||
n'est jamais présenté comme rollbackable ;
|
|
||||||
- si le `fsync` du répertoire échoue après ce `rename`, le résultat est
|
|
||||||
`PUBLISHED_NOT_DURABLE` : le fichier visible est valide, mais sa présence sous
|
|
||||||
ce nom après un crash ou une coupure n'est pas garantie. `OK` garantit que le
|
|
||||||
contenu et l'entrée de répertoire ont tous deux été synchronisés avec succès,
|
|
||||||
sous réserve des garanties fournies par le système de fichiers et le stockage.
|
|
||||||
|
|
||||||
Les tailles persistantes sont refusées avant conversion lorsqu'elles dépassent
|
- before successful rename: failure leaves the previously published checkpoint unchanged;
|
||||||
`SIZE_MAX`. Les secondes sont des entiers non signés v1 : les timestamps
|
- after successful rename: the new visible checkpoint is published and is not presented as rolled back;
|
||||||
négatifs ne sont pas sérialisables et une valeur lue doit être représentable
|
- parent-directory sync failure after rename: result is `PUBLISHED_NOT_DURABLE`; the visible file is
|
||||||
par le `time_t` local avant conversion. Le format reste donc lisible entre
|
valid, but name persistence across crash/power loss is not guaranteed by Lardon3D.
|
||||||
plateformes uniquement pour les valeurs communes à leurs domaines `size_t` et
|
|
||||||
`time_t`.
|
|
||||||
|
|
||||||
## Project Database v7 — fondation historique
|
`OK` means both content and directory-entry synchronization completed, subject to the guarantees of the
|
||||||
|
filesystem and storage stack.
|
||||||
|
|
||||||
SQLite contient l'état logique interrogable et les références aux fichiers ;
|
Persistent sizes are rejected before conversion if they exceed the local representable domain. The v1
|
||||||
les checkpoints et artefacts volumineux restent externes. L'enregistrement du
|
timestamp representation likewise requires values representable by the local `time_t` before runtime
|
||||||
résumé de tâche et de sa référence checkpoint est une transaction unique. Un
|
conversion.
|
||||||
artefact est d'abord publié et vérifié comme fichier régulier, puis seulement
|
|
||||||
marqué `READY` en DB. Le chemin inverse est interdit.
|
|
||||||
|
|
||||||
### Protocole checkpoint projet
|
## Project-owned checkpoint protocol
|
||||||
|
|
||||||
Le protocole réel n'est pas une transaction distribuée :
|
The project checkpoint protocol coordinates a Task file and SQLite metadata without claiming a true
|
||||||
|
filesystem+SQLite transaction.
|
||||||
|
|
||||||
1. capture locale du snapshot sous le mutex de tâche puis déverrouillage ;
|
The current project-level ordering is:
|
||||||
2. publication atomique du fichier sous
|
|
||||||
`.lardon3d/checkpoints/<task_id>.chk` ;
|
|
||||||
3. transaction SQLite sur `tasks` et `checkpoints` avec chemin relatif.
|
|
||||||
|
|
||||||
Une erreur avant publication ne modifie pas la DB. `PUBLISHED_NOT_DURABLE` est
|
```text
|
||||||
conservé comme tel en DB. Si la publication réussit puis que SQLite retourne
|
capture bounded Task snapshot
|
||||||
`BUSY` ou une erreur, le fichier valide reste sur disque, la DB conserve son
|
-> publish .chk.next
|
||||||
ancienne vérité et le nouveau fichier est un orphelin à réconcilier plus tard.
|
-> record Task/checkpoint summary in SQLite
|
||||||
Il n'est pas supprimé et aucune atomicité FS+SQLite n'est revendiquée.
|
-> promote .chk.next to canonical .chk under .chk.lock
|
||||||
|
```
|
||||||
|
|
||||||
L'inventaire distingue checkpoint récupérable durable, récupérable mais publié
|
The canonical location is:
|
||||||
non durable, absent, invalide, version inconnue et erreur d'I/O. Aucune réparation
|
|
||||||
ou suppression silencieuse n'est effectuée.
|
|
||||||
|
|
||||||
Le format checkpoint reste en version 1 et ne contient pas de `task_kind`. Le
|
```text
|
||||||
schéma SQLite v4 conserve `task_kind` et `task_kind_version` dans le résumé
|
.lardon3d/checkpoints/<task_id>.chk
|
||||||
logique interrogable. La migration v1→v2 laisse ces deux colonnes à `NULL` : une
|
```
|
||||||
tâche legacy reste inspectable mais ne peut pas être reconstruite ou resoumise.
|
|
||||||
Un kind inconnu ou une version non supportée est diagnostiqué sans exécuter de
|
|
||||||
code.
|
|
||||||
|
|
||||||
## Statut
|
The advisory `.chk.lock` exists only as process synchronization; it is not recovery data.
|
||||||
|
|
||||||
**IMPLEMENTED** — modèle durable, codec v1, lecture validée, publication
|
Recovery obtains the lock, reloads the relevant Project DB record because it may have changed while
|
||||||
atomique et restauration sûre d'une tâche isolée.
|
waiting, and then selects a codec/version-valid checkpoint whose stored summary matches the Project DB
|
||||||
|
summary exactly for the fields Project DB owns.
|
||||||
|
|
||||||
**IMPLEMENTED** — Project Database v7 pour identité, tâches typées, ScanSets,
|
A valid canonical `.chk` has priority. A valid matching `.chk.next` may be promoted when the canonical
|
||||||
images logiques, Feature Sets/assets SHA-256, Visual Index segmenté,
|
file does not match. Stale or corrupt `.next` files never override a valid canonical checkpoint.
|
||||||
checkpoints et artefacts génériques.
|
|
||||||
|
|
||||||
**IMPLEMENTED** — registry statique bornée et reconstruction explicite avec
|
A missing, corrupt, future-version or summary-mismatched checkpoint makes that Task non-recoverable; it
|
||||||
ownership du userdata.
|
does not make unrelated project state invalid.
|
||||||
|
|
||||||
**IMPLEMENTED** — API projet de sauvegarde fichier+DB et inventaire validé au
|
## Project DB foundation
|
||||||
redémarrage.
|
|
||||||
|
|
||||||
**IMPLEMENTED** — `import.images` persiste son chemin source absolu et son
|
Project DB is SQLite with explicit schema versioning and sequential transactional migrations.
|
||||||
`scanset_id`, puis publie un checkpoint après chaque lot validé. Le catalogue
|
|
||||||
SQLite rend le rejeu idempotent à la granularité du contenu dans un ScanSet.
|
|
||||||
|
|
||||||
Le chemin source absolu est l'intention durable v1 : il doit rester accessible
|
The retained configuration uses:
|
||||||
après redémarrage et un projet déplacé ne rend pas une source externe portable.
|
|
||||||
Une source absente ou devenue non-répertoire fait échouer proprement la
|
|
||||||
reconstruction. Après import terminé, l'image dépend de l'asset géré, plus de la
|
|
||||||
source. Le SHA-256 est calculé pendant la copie avec un tampon fixe de 64 Kio.
|
|
||||||
L'asset est publié sans écrasement sous
|
|
||||||
`assets/images/<prefix>/<sha256>`, puis seulement enregistré `READY` dans une
|
|
||||||
transaction SQLite. Un asset concurrent déjà présent n'est adopté qu'après
|
|
||||||
rehash complet et vérification de taille. Si SQLite échoue après publication,
|
|
||||||
le fichier reste orphelin pour une future réconciliation ; aucune transaction
|
|
||||||
FS+SQLite n'est revendiquée.
|
|
||||||
|
|
||||||
Les identités publiées `scanset_id`, `image_id` et `asset_id` utilisent les
|
```text
|
||||||
séquences SQLite `AUTOINCREMENT` : une valeur issue d'une transaction validée
|
foreign_keys=ON
|
||||||
n'est jamais réattribuée à un autre objet, même après suppression de la ligne.
|
journal_mode=DELETE
|
||||||
Une valeur réservée par une transaction annulée n'est pas une identité publiée.
|
synchronous=FULL
|
||||||
|
busy_timeout=5000
|
||||||
|
```
|
||||||
|
|
||||||
`manifest.tsv` reste supporté par l'ancien chemin d'import/catalogue. Le chemin
|
`DELETE` journal mode matches the current single-owner Project DB model and avoids persistent WAL/SHM
|
||||||
persistant entretient une projection best-effort par hardlinks pour la TUI,
|
files. The timeout bounds waiting on an external lock.
|
||||||
mais SQLite est le commit logique canonique : la correction de la reprise ne
|
|
||||||
dépend plus de l'ordre de `readdir()` ni du nom de fichier.
|
|
||||||
|
|
||||||
La migration v3 vers v4 ne transforme pas les lignes du manifeste historique
|
The project identity is duplicated intentionally across `project.ini` and the `project` table and must
|
||||||
en images cataloguées : elles ne contiennent pas toutes les preuves exigées par
|
match. A divergence is an error, not an opportunity to invent a new identity.
|
||||||
le modèle v4. Le marqueur durable `legacy_image_catalog_pending` rend cette
|
|
||||||
situation visible. Une tâche récupérable peut repeupler le catalogue par rejeu
|
|
||||||
si sa source existe encore ; sinon les fichiers et le manifeste restent une
|
|
||||||
projection legacy, explicitement non cataloguée. Une tâche v3 déjà terminée
|
|
||||||
n'est pas rejouée automatiquement.
|
|
||||||
|
|
||||||
**IMPLEMENTED** — reprise automatique sélective à l'ouverture : pagination
|
Published catalog and scientific IDs use SQLite integer identities according to their owning schema.
|
||||||
bornée, validation checkpoint/kind, reconstruction production et enqueue sans
|
Where `AUTOINCREMENT` is part of the contract, a committed published identity is not later reused for a
|
||||||
claim persistant supplémentaire.
|
different object. An ID allocated only inside a rolled-back transaction is not a published identity.
|
||||||
|
|
||||||
Les records sont parcourus par task ID croissant. Un checkpoint
|
## Historical Project DB v7 foundation
|
||||||
`PUBLISHED_NOT_DURABLE` présent, valide et cohérent peut être repris ; le résumé
|
|
||||||
conserve cet avertissement jusqu'au prochain checkpoint durable. Une tâche
|
|
||||||
terminale n'appartient pas à la requête de reprise.
|
|
||||||
|
|
||||||
**NOT_YET_WIRED** — réconciliation des fichiers orphelins et retry piloté par
|
Project DB v7 is the historical persistent project/runtime foundation. It covers the durable project
|
||||||
l'utilisateur pour les sources indisponibles. Les checkpoints existants sont
|
identity, Tasks, checkpoint references, generic artifacts, ScanSets, logical images, image assets,
|
||||||
kind-owned aux frontières métier ; aucun timer générique ne peut les remplacer.
|
Feature Store metadata and the segmented Visual Index foundation.
|
||||||
|
|
||||||
**NOT_YET_WIRED** — migration de la vue TUI en mémoire vers la pagination
|
Important v7-era persistence rules remain current unless a later contract explicitly supersedes them:
|
||||||
SQLite, scrub des assets et réconciliation globale des orphelins.
|
|
||||||
|
|
||||||
**IMPLEMENTED** — Feature Store externe versionné, immutable, borné et relié
|
- Task summary plus checkpoint reference is a single SQLite transaction;
|
||||||
transactionnellement à ses métadonnées SQLite après publication.
|
- large checkpoint and scientific files remain external;
|
||||||
|
- an artifact file is published and validated before Project DB marks it `READY`;
|
||||||
|
- Project DB stores bounded metadata and references, not large descriptor/posting payloads;
|
||||||
|
- `AUTOINCREMENT` is used where published catalog/scientific IDs must not be recycled after committed
|
||||||
|
deletion;
|
||||||
|
- the persistent import source path records durable execution intent, while imported scientific data
|
||||||
|
ultimately depends on the managed immutable asset rather than the original external source path.
|
||||||
|
|
||||||
**IMPLEMENTED** — Visual Index externe segmenté, memberships transactionnels
|
### Generic artifact orphan window
|
||||||
et tâche `visual_index.update` récupérable.
|
|
||||||
|
|
||||||
**IMPLEMENTED** — migrations additives et séquentielles jusqu'à Project DB
|
For generic file-first publication, a successful file publication followed by SQLite `BUSY` or another
|
||||||
v23. Les versions v16 à v22 restent l'histoire scientifique et de persistance
|
SQLite failure leaves a valid orphan file on disk while Project DB retains its previous truth.
|
||||||
gelée ; v23 ajoute uniquement l'overlay optique générique.
|
|
||||||
|
|
||||||
Les neuf relations v23 séparent profils de boîtier et alias, profils d'objectif
|
That file is not silently deleted. Global orphan reconciliation is a separate maintenance capability.
|
||||||
et alias, configurations optiques, affectations de configuration aux groupes
|
The absence of such reconciliation does not justify pretending the file and SQLite update were atomic.
|
||||||
de campagne et aux Captures, profils de calibration et sélection explicite par
|
|
||||||
Capture. Une configuration référence exactement un boîtier et un objectif ; un
|
|
||||||
objectif manuel sans EXIF est normal. La compatibilité d'une
|
|
||||||
calibration est exacte sur la configuration optique et ses dimensions/champs
|
|
||||||
scientifiques. Aucun profil S21, A6000 ou Meike n'est inséré ou déduit par la
|
|
||||||
migration : les tables nouvelles restent vides tant qu'un caller ne fournit
|
|
||||||
pas explicitement les données.
|
|
||||||
|
|
||||||
La migration v22→v23 est une transaction additive. Elle ne réinterprète ni les
|
## Import and managed source assets
|
||||||
Captures, ni les Images, ni les résultats scientifiques historiques. Une copie
|
|
||||||
S21 et une copie A6000 ont atteint v23 avec intégrité et clés étrangères
|
|
||||||
valides, comptes scientifiques inchangés et tables optiques vides. Les détails
|
|
||||||
normatifs sont dans [Project Database](project_database.md).
|
|
||||||
|
|
||||||
**NOT_YET_WIRED** — reprise ordonnée par dépendances/DAG et réconciliation
|
`import.images` persists its source path and ScanSet identity and checkpoints after validated bounded
|
||||||
globale des artefacts orphelins.
|
work. Managed image assets are content-addressed by SHA-256 after a complete bounded copy/hash pass.
|
||||||
|
|
||||||
|
A concurrently existing asset is adopted only after the implementation verifies the expected content
|
||||||
|
and size according to the owning contract. SQLite publication follows file publication.
|
||||||
|
|
||||||
|
`manifest.tsv` remains a historical/legacy projection. SQLite is the canonical logical commit for the
|
||||||
|
persistent catalog. The `legacy_image_catalog_pending` marker means legacy data may remain outside the
|
||||||
|
current catalog model; it does not claim that historical manifest rows were silently converted into
|
||||||
|
fully proven catalog identities.
|
||||||
|
|
||||||
|
## Selective Task recovery
|
||||||
|
|
||||||
|
Project open performs bounded selective recovery using Project DB pages and the Task Kind Registry.
|
||||||
|
|
||||||
|
A recoverable Task must have:
|
||||||
|
|
||||||
|
- a valid Project DB Task record;
|
||||||
|
- a supported Task kind and version;
|
||||||
|
- a valid coherent checkpoint boundary required by that kind;
|
||||||
|
- reconstructable typed business persistence where the kind requires it.
|
||||||
|
|
||||||
|
The recovery scan copies records outside the SQLite mutex before business reconstruction and enqueue.
|
||||||
|
A full Queue window stops the scan without mutating the unvisited records.
|
||||||
|
|
||||||
|
Project-level schema, migration and project-identity errors are fatal to opening the project. Per-Task
|
||||||
|
legacy, unknown-kind, unsupported-version, missing-checkpoint, invalid-checkpoint, unavailable-source or
|
||||||
|
reconstruction errors are isolated to the affected Task where the owning contract permits.
|
||||||
|
|
||||||
|
## External scientific artifacts
|
||||||
|
|
||||||
|
### Feature Store
|
||||||
|
|
||||||
|
Feature files are immutable external artifacts with versioned bounded readers. Project DB stores their
|
||||||
|
identity and publication metadata only after the external file is valid according to the Feature Store
|
||||||
|
contract.
|
||||||
|
|
||||||
|
Feature descriptors are not duplicated into SQLite.
|
||||||
|
|
||||||
|
### Visual Index
|
||||||
|
|
||||||
|
Visual Index postings remain in bounded immutable external segments. Project DB stores index identity,
|
||||||
|
segment metadata and memberships. Durable update Tasks retain the bounded cursor required to resume
|
||||||
|
publication without rebuilding already accepted segments.
|
||||||
|
|
||||||
|
### Match and later scientific payloads
|
||||||
|
|
||||||
|
Match payloads and other large scientific representations follow the same architectural principle:
|
||||||
|
SQLite owns durable identity, relations and bounded metadata; specialized external formats own large
|
||||||
|
numerical payloads where the subsystem contract requires them.
|
||||||
|
|
||||||
|
## Project DB v16-v22 retained scientific foundation
|
||||||
|
|
||||||
|
The scientific and persistence contracts introduced through v16-v22 remain historical PASS/FROZEN
|
||||||
|
foundations. Later schema versions are additive overlays and do not reinterpret those rows.
|
||||||
|
|
||||||
|
In particular:
|
||||||
|
|
||||||
|
- v16 publishes the immutable Sparse SfM persistence model;
|
||||||
|
- v17 adds the durable typed Sparse SfM Task payload for Gate F;
|
||||||
|
- v18 adds Phase H v1 incremental-reconstruction identity/persistence;
|
||||||
|
- v19 adds Capture / Asset Provenance v1;
|
||||||
|
- v20 adds durable acquisition-campaign Task persistence;
|
||||||
|
- v21 adds Photo Quality Triage persistence;
|
||||||
|
- v22 adds selected scientific execution, explicit Capture SOURCE-asset mapping and durable selected
|
||||||
|
RAW development persistence.
|
||||||
|
|
||||||
|
Detailed tables, identities and migration invariants are defined in
|
||||||
|
[Project Database](project_database.md).
|
||||||
|
|
||||||
|
## Project DB v23 - generic optical-context overlay
|
||||||
|
|
||||||
|
Project DB v23 is an additive optical-context overlay above the v22 scientific foundation.
|
||||||
|
|
||||||
|
It separates four identities:
|
||||||
|
|
||||||
|
- camera body profile;
|
||||||
|
- lens profile;
|
||||||
|
- optical configuration;
|
||||||
|
- optical calibration profile.
|
||||||
|
|
||||||
|
The migration creates the optical relations empty. It does not inspect EXIF, path, basename, SHA-256,
|
||||||
|
dimensions, device name or historical calibration to backfill identity.
|
||||||
|
|
||||||
|
Electronic metadata aliases, when present, use exact stored identity semantics. A manual lens without
|
||||||
|
EXIF is a normal explicit profile and does not require a fabricated metadata alias.
|
||||||
|
|
||||||
|
Calibration selection is explicit and requires exact compatible optical configuration. No silent
|
||||||
|
interpolation, substitution or inferred calibration identity is introduced by persistence.
|
||||||
|
|
||||||
|
Historical S21/A6000 copies migrated through the optical overlay without changing their existing
|
||||||
|
scientific rows; empty optical tables remain an honest state until explicit data is supplied.
|
||||||
|
|
||||||
|
## Project DB v24 - RAW batch persistence
|
||||||
|
|
||||||
|
Project DB v24 adds only the typed durable relation required by `raw.develop.batch/1`:
|
||||||
|
|
||||||
|
```text
|
||||||
|
raw_development_batch_tasks(task_id, selected_execution_id)
|
||||||
|
```
|
||||||
|
|
||||||
|
The migration is additive and DDL-only. It does not create Capture, Asset, Image or selected-execution
|
||||||
|
identity and does not rewrite historical `raw.develop/1` Tasks.
|
||||||
|
|
||||||
|
The selected execution remains the durable scientific ordering authority. The batch Task may prepare
|
||||||
|
independent RAW items concurrently under Governor admission, but after all participants join, the owner
|
||||||
|
publishes selected representations in deterministic selected-item order.
|
||||||
|
|
||||||
|
The durable ordering remains:
|
||||||
|
|
||||||
|
```text
|
||||||
|
prepare bounded independent RAW items
|
||||||
|
-> join participants
|
||||||
|
-> owner publishes exact selected item
|
||||||
|
-> selected item/cursor commit
|
||||||
|
-> generic Task progress/checkpoint
|
||||||
|
```
|
||||||
|
|
||||||
|
A crash may therefore leave generic Task progress behind already durable selected scientific state; it
|
||||||
|
must never move generic progress ahead of unpublished selected state. Recovery resumes from durable
|
||||||
|
selected-execution identity/cursor and exact already-published representations rather than guessing
|
||||||
|
from paths or files.
|
||||||
|
|
||||||
|
The retained real A6000 run completed this RAW-batch path for all 689 selected RAW representations.
|
||||||
|
|
||||||
|
## Project DB v25 - Feature batch persistence
|
||||||
|
|
||||||
|
Project DB v25 is the current schema head. It adds only the typed durable relation required by
|
||||||
|
`features.extract.batch/1` through `feature_extract_batch_tasks`.
|
||||||
|
|
||||||
|
The v25 row binds one durable Feature-batch Task to:
|
||||||
|
|
||||||
|
- one immutable selected execution;
|
||||||
|
- the monotone selected-item prefix/cursor;
|
||||||
|
- the exact ORB extractor kind/version/parameters/fingerprint domain required by the Task contract.
|
||||||
|
|
||||||
|
The migration is additive and DDL-only. It creates no Feature Set, converts no historical
|
||||||
|
`features.extract/1` Task and infers no Image or Feature Set identity.
|
||||||
|
|
||||||
|
Selected images may be prepared concurrently without SQLite access. After participants join, the owner
|
||||||
|
publishes Feature Sets in selected order and advances the durable Feature cursor only when the exact
|
||||||
|
READY Feature Set is already durable.
|
||||||
|
|
||||||
|
A crash may leave generic Task checkpoint/progress or the Feature cursor behind an immutable Feature
|
||||||
|
Set that was already published. Recovery must revalidate and reuse that exact result; it must not infer
|
||||||
|
a replacement identity.
|
||||||
|
|
||||||
|
The retained real A6000 proof completed the v25 path with 689 Feature Sets and then continued through
|
||||||
|
Visual Index, 38,420 Candidate Pairs, 38,420 Match Results, Geometric Verification and Tracks without
|
||||||
|
replaying acquisition, RAW or Feature work.
|
||||||
|
|
||||||
|
This persistence result does **not** imply that real Sparse SfM or Dense/MVS was executed. Those counts
|
||||||
|
remain zero in `REAL_A6000_PRE_SFM=PASS/FROZEN`.
|
||||||
|
|
||||||
|
## Capture / Asset / Image identity boundary
|
||||||
|
|
||||||
|
Persistence must preserve these distinctions:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Capture != file
|
||||||
|
Capture != Asset
|
||||||
|
Capture != image_id
|
||||||
|
Capture != SHA-256
|
||||||
|
Capture != path
|
||||||
|
Capture != basename
|
||||||
|
Capture != Task ID
|
||||||
|
Capture != campaign group ID
|
||||||
|
```
|
||||||
|
|
||||||
|
`asset_id` identifies a managed immutable asset record. SHA-256 identifies immutable bytes according to
|
||||||
|
the asset contract. `image_id` identifies a scientific image representation. `capture_id` identifies a
|
||||||
|
physical acquisition representation in Project DB. Task and campaign-group IDs remain operational
|
||||||
|
identities.
|
||||||
|
|
||||||
|
Retry and recovery may use an explicit persisted mapping between these domains only where a canonical
|
||||||
|
contract defines it. They may not reconstruct missing identity from coincidental equality or metadata.
|
||||||
|
|
||||||
|
## Acquisition-campaign crash/restart ordering
|
||||||
|
|
||||||
|
For durable campaign execution, the important persistence boundary is conceptually:
|
||||||
|
|
||||||
|
```text
|
||||||
|
S3-E returns capture_id
|
||||||
|
-> persist group_id -> capture_id mapping and campaign cursor
|
||||||
|
-> advance generic Task progress/checkpoint
|
||||||
|
-> next group may execute
|
||||||
|
```
|
||||||
|
|
||||||
|
The retained pre-return S3-E crash window remains intentional. If a Capture is created internally but
|
||||||
|
the process dies before the caller receives and durably retains its `capture_id`, campaign persistence
|
||||||
|
does not guess that identity from path, digest, basename, timestamp, metadata or `image_id`.
|
||||||
|
|
||||||
|
## Concurrency and ownership
|
||||||
|
|
||||||
|
A Project DB connection is serialized by its internal mutex. Each public compound operation owns its
|
||||||
|
entire transaction; a public transaction is not left open across calls.
|
||||||
|
|
||||||
|
External artifact I/O is not performed while holding the Project DB mutex where the owning contract
|
||||||
|
separates those operations. SQLite-owned strings and records are copied into caller-owned bounded
|
||||||
|
storage before returning.
|
||||||
|
|
||||||
|
Closing Project DB concurrently with an active Project DB call is forbidden by owner lifetime rules.
|
||||||
|
Application project/session teardown must first destroy and join the Queue so Task callbacks and leases
|
||||||
|
are finished, then close Project DB.
|
||||||
|
|
||||||
|
## Schema migration discipline
|
||||||
|
|
||||||
|
Project DB migrations are sequential, transactional and additive unless an explicit future human ticket
|
||||||
|
authorizes a different migration.
|
||||||
|
|
||||||
|
Current known sequence:
|
||||||
|
|
||||||
|
```text
|
||||||
|
v1 -> ... -> v22
|
||||||
|
v22 -> v23 generic optical-context overlay
|
||||||
|
v23 -> v24 RAW batch Task persistence
|
||||||
|
v24 -> v25 Feature batch Task persistence
|
||||||
|
```
|
||||||
|
|
||||||
|
A migration failure rolls back both its new schema objects and its schema-version publication marker.
|
||||||
|
A retry therefore starts from the previous complete known schema.
|
||||||
|
|
||||||
|
Future schema versions beyond v25 are rejected by the current implementation and require explicit human
|
||||||
|
authorization before code or documentation may treat them as current.
|
||||||
|
|
||||||
|
## Current status
|
||||||
|
|
||||||
|
```text
|
||||||
|
PERSISTENCE_DOCUMENT=CURRENT
|
||||||
|
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||||
|
|
||||||
|
TASK_CHECKPOINT_V1 IMPLEMENTED
|
||||||
|
PROJECT_CHECKPOINT_PROTOCOL IMPLEMENTED/VALIDATED
|
||||||
|
PROJECT_DB_V7_FOUNDATION IMPLEMENTED
|
||||||
|
PROJECT_DB_V16_TO_V22_FOUNDATION PASS/FROZEN
|
||||||
|
PROJECT_DB_V23_OPTICAL_OVERLAY IMPLEMENTED/VALIDATED/REVIEWED
|
||||||
|
PROJECT_DB_V24_RAW_BATCH IMPLEMENTED/VALIDATED
|
||||||
|
PROJECT_DB_V25_FEATURE_BATCH IMPLEMENTED/VALIDATED
|
||||||
|
REAL_A6000_PRE_SFM PASS/FROZEN
|
||||||
|
```
|
||||||
|
|
||||||
|
Current intentionally unfinished persistence-adjacent work includes global orphan-file reconciliation,
|
||||||
|
asset scrub/reconciliation and a general dependency/DAG recovery model. Those are separate future
|
||||||
|
capabilities; they do not change the current v25 schema authority.
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue