Compare commits
19 commits
308cf739ac
...
2e7d3c5656
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2e7d3c5656 | ||
|
|
4fc5a8ea48 | ||
|
|
356cfd125c | ||
|
|
a3a005b38e | ||
|
|
7574362ba9 | ||
|
|
4278d124c2 | ||
|
|
1d073fab46 | ||
|
|
475f3f1eb0 | ||
|
|
7712fe19fe | ||
|
|
c27be7776b | ||
|
|
d050b13ddc | ||
|
|
eb454d5fae | ||
|
|
4522602421 | ||
|
|
2d2bf31328 | ||
|
|
06c78914ce | ||
|
|
306ace2fb3 | ||
|
|
8434fcd773 | ||
|
|
d0329bc3db | ||
|
|
b23241eab2 |
35 changed files with 12268 additions and 7515 deletions
61
AGENTS.md
61
AGENTS.md
|
|
@ -23,6 +23,36 @@
|
|||
Resolve locally determinable engineering problems from the existing code and
|
||||
canonical documentation.
|
||||
|
||||
## Repository language policy
|
||||
|
||||
The canonical language of the Lardon3D repository is English.
|
||||
|
||||
This applies to:
|
||||
|
||||
- README.md;
|
||||
- AGENTS.md;
|
||||
- all current and future files under docs/**;
|
||||
- prompt.md and all files under prompt/**;
|
||||
- source comments under include/lardon3d/** and src/**;
|
||||
- developer-facing technical documentation, diagnostics and engineering contracts where applicable.
|
||||
|
||||
The repository must not intentionally mix French and English technical prose.
|
||||
Existing French documentation and source comments are to be converted to English during the planned documentation and source-comment remediation passes, without changing scientific meaning, FROZEN facts, historical evidence, numeric values, identities, lifecycle truth or contract authority.
|
||||
|
||||
Historical documents may be translated, but translation must not modernize or reinterpret the historical state they record.
|
||||
|
||||
The canonical user-interface language is also English. Existing non-English TUI labels must converge on English during an explicitly scoped UI-language remediation pass; documentation-only work must not silently modify executable UI behavior.
|
||||
|
||||
Canonical markers:
|
||||
|
||||
DOCUMENTATION_LANGUAGE=ENGLISH
|
||||
|
||||
SOURCE_COMMENT_LANGUAGE=ENGLISH
|
||||
|
||||
AGENT_CONTRACT_LANGUAGE=ENGLISH
|
||||
|
||||
USER_INTERFACE_LANGUAGE=ENGLISH
|
||||
|
||||
## 2. FROZEN integrity
|
||||
|
||||
The following current project foundation is protected and may change only
|
||||
|
|
@ -45,21 +75,18 @@ through an explicitly authorized, explicitly scoped human ticket:
|
|||
- Durable Acquisition-Campaign Execution — PASS/FROZEN
|
||||
- Global Maintenance Audit — PASS/FROZEN
|
||||
- Real S21 Tracks scientific result — PASS/FROZEN
|
||||
- Real A6000 pre-SfM scientific result — PASS/FROZEN
|
||||
|
||||
Detailed subcontracts remain defined by their canonical documents. This file
|
||||
must not duplicate every S3 substage, scientific threshold, migration detail,
|
||||
or persistence format.
|
||||
|
||||
Project DB v25 is the current additive operational schema in the active Feature
|
||||
batch tranche. It preserves v24 RAW batch, the v23 optical overlay and the v22
|
||||
scientific/persistence foundation. Its authorized purpose is limited to typed
|
||||
durable persistence for `features.extract.batch/1` through
|
||||
`feature_extract_batch_tasks`; it adds no scientific identity and must not
|
||||
reinterpret historical rows. Until the v25/Feature-batch tranche has completed
|
||||
its real-data proof and final review, its lifecycle must remain truthful rather
|
||||
than being marked `PASS/FROZEN` prematurely. Historical references to older
|
||||
Project DB versions remain valid where they describe the actual historical
|
||||
contract or migration path.
|
||||
Project DB v25 is the current additive operational schema. It preserves v24 RAW batch, the v23
|
||||
optical overlay and the v22 scientific/persistence foundation. Its additive purpose is typed durable
|
||||
persistence for `features.extract.batch/1` through `feature_extract_batch_tasks`; it adds no
|
||||
scientific identity and must not reinterpret historical rows. The v25 Feature-batch path has completed
|
||||
its retained real A6000 proof. Historical references to older Project DB versions remain valid where
|
||||
they describe the actual historical contract or migration path.
|
||||
|
||||
The global maintenance implementation, fresh portable/Vulkan/sanitizer/
|
||||
concurrency validation and independent final review are acquired. Its lifecycle
|
||||
|
|
@ -78,6 +105,12 @@ their directly affected contracts, tests, documentation, and crossed dependency
|
|||
boundaries. Unchanged PASS/FROZEN systems inherit this evidence and are reopened
|
||||
only by concrete evidence; do not repeat a global A-to-Z audit.
|
||||
|
||||
The later real-data checkpoint is tag `real-a6000-pre-sfm-2026-09-02`. It establishes
|
||||
`REAL_A6000_PRE_SFM=PASS/FROZEN` through Geometric Verification and Tracks while explicitly
|
||||
leaving Sparse SfM and Dense/MVS unexecuted. This checkpoint adds later operational and real-data
|
||||
evidence; it does not erase or replace the `global-maintenance-2026-09-01` review authority for
|
||||
unchanged FROZEN systems.
|
||||
|
||||
When a ticket declares `NO_NEW_SUBSYSTEM`, do not introduce an unrelated:
|
||||
|
||||
- Task Runtime;
|
||||
|
|
@ -212,7 +245,7 @@ explicitly defines such an identity.
|
|||
progress distinct from generic runtime percentage, and keep unknown
|
||||
provenance visibly UNKNOWN. Full layout starts at 100x30, compact is
|
||||
supported through 60x15 (72x20 is the reference compact boundary), and only
|
||||
the bounded "Terminal trop petit" fallback is allowed below that minimum.
|
||||
the bounded terminal-too-small fallback is allowed below that minimum.
|
||||
- Preserve the contextual key contract. `F10 SSD` remains literally visible at
|
||||
60 columns in idle, text-input and import-running modes; text input owns only
|
||||
Enter/Escape/F10 and a running import owns only cancel (`X`) and F10 while
|
||||
|
|
@ -767,7 +800,9 @@ For Project DB:
|
|||
overlay;
|
||||
- v24 is the explicitly authorized additive operational RAW-batch migration and
|
||||
must not reinterpret scientific history;
|
||||
- future schema-version changes beyond the currently authorized v24 require
|
||||
- v25 is the additive Feature-batch migration for `features.extract.batch/1` and
|
||||
must preserve v22/v23/v24 identities, rows and semantics;
|
||||
- future schema-version changes beyond the current v25 head require
|
||||
explicit human authorization;
|
||||
- migrations must be additive unless a different migration is explicitly
|
||||
authorized;
|
||||
|
|
@ -985,7 +1020,7 @@ Every completed ticket report must include:
|
|||
STOP and request a human decision only when resolution requires:
|
||||
|
||||
- changing a FROZEN scientific contract;
|
||||
- changing Project DB schema/version beyond already authorized v24 without
|
||||
- changing Project DB schema/version beyond the current v25 head without
|
||||
prior authorization;
|
||||
- introducing a genuinely new subsystem outside authorized scope;
|
||||
- files outside the authorized scope;
|
||||
|
|
|
|||
21
LICENSE
Normal file
21
LICENSE
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
MIT License
|
||||
|
||||
Copyright (c) 2026 LabfyTools
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
587
README.md
587
README.md
|
|
@ -1,198 +1,326 @@
|
|||
# Lardon3D
|
||||
|
||||
Moteur de photogrammétrie générique, persistant, incrémental et sensible aux
|
||||
ressources, piloté par une TUI ncursesw.
|
||||
Lardon3D is a generic, persistent, incremental, resource-aware photogrammetry engine for Linux,
|
||||
controlled through an ncursesw TUI.
|
||||
|
||||
## Vision
|
||||
|
||||
Lardon3D est un moteur de photogrammétrie Linux qui privilégie :
|
||||
Lardon3D is designed around the following principles:
|
||||
|
||||
- **Stabilité** : aucune saturation du système hôte
|
||||
- **Déterminisme** : résultats reproductibles et traçables
|
||||
- **Faible consommation mémoire** : traitement par lots adaptatifs
|
||||
- **Reprise après interruption** : résultats atomiques et persistants
|
||||
- **Protection de la machine** : budgets bornés et respectueux
|
||||
- **Traçabilité** : historique des opérations et métriques
|
||||
- **Enrichissement progressif** : reconstruction incrémentale
|
||||
- **Scientific traceability**: results, identities, parameters and provenance are explicit.
|
||||
- **Determinism**: equivalent inputs and contracts produce reproducible, auditable outputs.
|
||||
- **Persistent progress**: long-running work is checkpointed and restartable.
|
||||
- **Bounded execution**: memory, CPU, GPU, I/O and temporary-storage use are explicitly bounded.
|
||||
- **Maximum safe useful throughput**: after preserving the interactive host reserve, available
|
||||
resources should be used whenever they provide useful throughput.
|
||||
- **Incremental reconstruction**: new observations can extend previous results without silently
|
||||
rewriting validated history.
|
||||
- **Atomic publication**: partially produced scientific outputs never masquerade as complete ones.
|
||||
|
||||
Lardon3D ne vise pas simplement "dossier de photos → objet 3D", mais un ensemble
|
||||
progressif d'observations et de contraintes donnant une reconstruction géométrique
|
||||
persistante, enrichissable et versionnable.
|
||||
Lardon3D is not simply a "folder of photos -> 3D object" tool. Its target model is:
|
||||
|
||||
## État actuel
|
||||
```text
|
||||
progressive observations and constraints
|
||||
-> persistent geometric reconstruction
|
||||
-> validated incremental enrichment
|
||||
-> dense geometry / mesh / texture / export
|
||||
```
|
||||
|
||||
### Briques validées
|
||||
## Current repository state
|
||||
|
||||
- **Project** : cycle de vie persistant, identité stable et Project Database ouverte
|
||||
- **Import** : premier task kind de production, exécuté par la file générique en lots bornés et reprenables
|
||||
- **ScanSet / Image Catalog v1** : acquisitions, images logiques, provenance et assets SHA-256 persistants et paginés
|
||||
- **Capture / Asset Provenance v1** : Captures par ScanSet, associations source/dérivé
|
||||
et sélection explicite d'une image logique — PASS / FROZEN
|
||||
- **Découverte et planification de campagne bornées** : racines explicites,
|
||||
plan metadata-only et exécution par groupes via S3-E — PASS / FROZEN ; campagne
|
||||
A6000 réelle validée sur 953 ARW + 953 JPEG MPF
|
||||
- **Feature Store v1/v2** : ORB U8×32, SIFT/RootSIFT F32×128 et lecture typée bornée
|
||||
- **Image View** : vues triées et filtrées pour la TUI
|
||||
- **Task** : moteur de tâches avec pause/reprise, annulation et séquences
|
||||
- **Task Checkpoint v1** : snapshot durable, protocole `.chk.next` → SQLite →
|
||||
`.chk` sous verrou par tâche, et reprise sûre
|
||||
- **Project Database v23** : overlay optique additif au-dessus de la fondation
|
||||
v22 PASS / FROZEN ; profils de boîtier et d'objectif, configurations
|
||||
body+lens+focale, affectations campagne/Capture et calibrations exactement
|
||||
compatibles, sans inférence ni backfill — IMPLEMENTED / VALIDATED / REVIEWED
|
||||
- **[Photo Quality Triage](docs/architecture/photo_quality_triage.md)** : métriques JPEG
|
||||
- **[Calibration Bootstrap v1](docs/architecture/calibration_bootstrap.md)** :
|
||||
import borné d'une calibration optimisée et traçable avant le Sparse SfM à
|
||||
calibration connue — PASS / FROZEN ; ni
|
||||
auto-calibration interne ni EXIF comme source de calibration scientifique
|
||||
- **Exécution durable de campagne d'acquisition** : tâche générique à requête
|
||||
typée immuable, confirmations `CALLER_EXPLICIT`, curseur et correspondance
|
||||
tâche/groupe→Capture persistants ; un groupe S3-E par séquence, reprise par
|
||||
la registry, la Queue et le Resource Governor existants
|
||||
- **Sparse SfM Gates C/D/E** : géométrie calibrée, noyau incrémental et Bundle
|
||||
Adjustment final par composante, tous PASS / FROZEN
|
||||
- **Sparse SfM Gate F** : orchestration durable, runtime gouverné et publication
|
||||
atomique, PASS / FROZEN
|
||||
- **MVS-M1** : frontière externe OpenMVS v2.4.0
|
||||
`InterfaceCOLMAP`/`DensifyPointCloud`, export COLMAP déterministe (OpenCV
|
||||
undistortion, observations transformées et tracks réels), texte exporté en
|
||||
flux et tracks indexés sans rescanner quadratiquement les observations ; espace
|
||||
de travail privé neuf par invocation sous le staging appelant, sans réemploi ;
|
||||
identité dense liée à la reconstruction de base, au jeu source, au
|
||||
`calibration_scope_identity` historique, au binding numérique MVS, au backend
|
||||
et aux paramètres ; `L3DMDID2` v2 (220 octets) et binding `L3DMCAL1` v1 ; PLY
|
||||
OpenMVS binaire little-endian validé (en-tête <= 1 MiB en octets bruts,
|
||||
LF/CRLF acceptés, CR seul malformé rejeté, ligne <= 64 KiB), fusionné en
|
||||
mode 0 — PASS / FROZEN
|
||||
- **Geometric Verification Model** : identité, masque d'inliers et modèle 3×3
|
||||
persistants pour les policies Verifier v1/v2 historiques et v3 courante
|
||||
- **Geometric Verifier v3** : Fundamental USAC/MAGSAC, reprise et lots resource-aware
|
||||
- **Internal Parallelism + Compute Resources v1** : parallélisme interne borné,
|
||||
sorties canoniques et admission Governor — PASS / FROZEN
|
||||
- **Task Kind Registry** : identité métier durable et reconstruction runtime explicite
|
||||
- **Recovery projet** : reprise automatique sélective et bornée des imports récupérables
|
||||
- **Task Queue** : file FIFO avec sélection adaptative et backpressure
|
||||
- **Hardware Profile** : détection des capacités matérielles
|
||||
- **Resource Snapshot** : capture instantanée des ressources
|
||||
- **Resource Governor** : arbitrage centralisé des budgets et réservations
|
||||
- **TUI observatoire / centre de contrôle** : modèle de vue pur et borné,
|
||||
observation coalescée, progression durable/ETA honnête, écrans Tasks,
|
||||
Resources, Optique et SSD, avec ncurses exclusivement sur le thread principal
|
||||
— CURRENT / VALIDATED OPERATIONAL
|
||||
- **Contrôleur SSD externe optionnel** : frontière physique UDisks2/GDBus,
|
||||
identité Drive+labels+UUIDs, drain sûr et capacités de contrôle exactes ; son
|
||||
état physique est enregistré auprès du Governor, seul orchestrateur des
|
||||
leases scratch de production — CURRENT / VALIDATED OPERATIONAL
|
||||
### Current Project Database
|
||||
|
||||
### Intégration réelle validée
|
||||
The current Project DB schema is **v25**.
|
||||
|
||||
Sony A6000 et Samsung S21 FE sont des preuves de validation de la chaîne
|
||||
générique. Ils ne définissent ni l'identité produit, ni un profil caméra
|
||||
hardcodé, ni une limite de CPU ou de dataset.
|
||||
The current head is additive:
|
||||
|
||||
- **Intégration multi-campagne A6000 + S21 FE Engine Bay** : PASS — les plans
|
||||
réels A6000 (953 paires confirmées `CALLER_EXPLICIT`) et Samsung SM-G990B
|
||||
(3544 JPEG singleton) ont été validés dans deux ScanSets d'un même projet
|
||||
temporaire, avec exécution durable, Governor/Queue et reprise sans Capture
|
||||
dupliqué. Les campagnes réelles actuellement évaluées sont
|
||||
`CALIBRATION_UNAVAILABLE`, donc le Sparse SfM réel est
|
||||
`BLOCKED_BY_KNOWN_CALIBRATION_DATA` : ce n'est ni un échec logiciel, ni un
|
||||
rejet de qualité, ni une autorisation d'importer une pseudo-calibration. La
|
||||
suite reste le pipeline scientifique aval, selon la
|
||||
[roadmap canonique](docs/roadmap/roadmap.md).
|
||||
```text
|
||||
v22 Selected scientific execution foundation
|
||||
v23 Generic optical-context overlay
|
||||
v24 raw.develop.batch/1 persistence
|
||||
v25 features.extract.batch/1 persistence
|
||||
```
|
||||
|
||||
### Plus tard / différé
|
||||
Earlier schema versions remain valid historical contracts where their own documentation says so.
|
||||
No migration silently reinterprets historical scientific identities.
|
||||
|
||||
- publication durable dense/mesh et consommation Task explicite du scratch
|
||||
SSD optionnel ;
|
||||
- vidéo/keyframes et **Capture Guidance / Live Coverage** : analyse et viewer de
|
||||
couverture, suggestions de prises de vue puis assistance live, après
|
||||
reconstruction mature ;
|
||||
- exports et publication live ;
|
||||
- DAG général, pools multiples et parallélisme inter-tâches restent différés.
|
||||
### Current production task inventory
|
||||
|
||||
The production registry currently contains **16 Task kinds**.
|
||||
|
||||
All production Tasks pass through the existing Task -> Queue -> Resource Governor execution model.
|
||||
The Queue has one active callback at a time; Tasks may use bounded internal participants when their
|
||||
contract and measured scaling justify it.
|
||||
|
||||
### Resource policy
|
||||
|
||||
The canonical operational objective is:
|
||||
|
||||
```text
|
||||
MAXIMUM SAFE USEFUL THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF
|
||||
```
|
||||
|
||||
Lardon3D first preserves the interactive host reserve required for the desktop, Firefox, audio and
|
||||
light interactive use. Safe and useful resources beyond that reserve belong to the active workload.
|
||||
|
||||
On the current validation host, the normal observed outcome is approximately:
|
||||
|
||||
```text
|
||||
16 logical CPUs total
|
||||
4 logical CPUs reserved for interactive host use
|
||||
12 logical CPUs available to the compute pool
|
||||
~3 GiB MemAvailable preserved as the hard RAM reserve
|
||||
Radeon 780M UMA available to validated and useful GPU backends
|
||||
```
|
||||
|
||||
These are **reference-host observations, not portable product constants**. The Resource Governor
|
||||
derives usable capacity from the current host, affinity, topology, memory and pressure state.
|
||||
|
||||
A long-running CPU1 or batch1 path is acceptable only when serialism, a measured scaling knee,
|
||||
memory, I/O, GPU execution or another concrete constraint justifies it. Per-item atomicity does not
|
||||
imply cross-item serialization.
|
||||
|
||||
## Validated foundations
|
||||
|
||||
The following major foundations are implemented and validated at their documented boundaries:
|
||||
|
||||
- **Project / persistent lifecycle**
|
||||
- **Import and ScanSet / Image Catalog**
|
||||
- **Capture / Asset Provenance v1 — PASS / FROZEN**
|
||||
- **Bounded acquisition discovery and campaign execution — PASS / FROZEN**
|
||||
- **Photo Quality Triage / Acquisition Selection — PASS / FROZEN**
|
||||
- **Selected Scientific Execution — PASS / FROZEN**
|
||||
- **Feature Store v1/v2**
|
||||
- ORB U8x32
|
||||
- SIFT / RootSIFT F32x128
|
||||
- bounded typed readers
|
||||
- **Visual Index v1**
|
||||
- **Candidate Pair generation**
|
||||
- **Matcher v1**
|
||||
- ORB CPU / validated Vulkan hot path
|
||||
- SIFT / RootSIFT CPU
|
||||
- **Geometric Verification Model**
|
||||
- **Geometric Verifier v3**
|
||||
- **Track Model / Track Builder v1 — PASS / FROZEN**
|
||||
- **Sparse SfM Gates C/D/E/F/G — PASS / FROZEN**
|
||||
- **Phase H v1 incremental reconstruction — PASS / FROZEN**
|
||||
- **MVS-M1 external OpenMVS boundary — PASS / FROZEN**
|
||||
- **Task Runtime / checkpoints / recovery**
|
||||
- **Task Queue**
|
||||
- **Task Kind Registry**
|
||||
- **Resource Governor / Compute Governor v2**
|
||||
- **Bounded internal parallelism — PASS / FROZEN**
|
||||
- **ORB Vulkan asynchronous execution — PASS / FROZEN**
|
||||
- **TUI runtime observatory / control center — CURRENT / VALIDATED OPERATIONAL**
|
||||
- **Optional external SSD controller — CURRENT / VALIDATED OPERATIONAL**
|
||||
- **Calibration Bootstrap v1 — PASS / FROZEN**
|
||||
- **Calibration Science v1 — PASS / FROZEN**
|
||||
- **Calibration Tooling v1 — PASS / FROZEN**
|
||||
- **Calibration Solver Preflight v1 — PASS**
|
||||
- **Project DB v24/v25 operational overlays — IMPLEMENTED / VALIDATED**
|
||||
- raw.develop.batch/1 durable selected-execution path
|
||||
- features.extract.batch/1 durable selected-execution path
|
||||
|
||||
## Real-data validation
|
||||
|
||||
Sony A6000 and Samsung S21 FE campaigns are validation evidence for the generic pipeline. They are
|
||||
not product identities, hardcoded camera profiles, CPU limits or dataset-size limits.
|
||||
|
||||
### Real S21 Tracks
|
||||
|
||||
```text
|
||||
REAL_S21_TRACKS=PASS/FROZEN
|
||||
```
|
||||
|
||||
The retained real S21 proof validated the complete pre-SfM chain through Track Builder with the
|
||||
compact Track memory model and deterministic restart semantics.
|
||||
|
||||
### Real A6000 pre-SfM
|
||||
|
||||
```text
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
The retained real A6000 proof uses the selected RAW-derived deterministic representation and
|
||||
completed the pipeline through Geometric Verification and Tracks without replaying already acquired
|
||||
upstream work.
|
||||
|
||||
Final retained counts:
|
||||
|
||||
```text
|
||||
Selected images 689
|
||||
Feature Sets 689
|
||||
Candidate Pairs 38,420
|
||||
Match Results 38,420
|
||||
Applicable GVRs 37,805
|
||||
Verified GVRs 10,952
|
||||
Rejected GVRs 26,853
|
||||
Track Sets 1
|
||||
Tracks 130,714
|
||||
Track observations 318,944
|
||||
```
|
||||
|
||||
The restart proof reused the existing Track Set without duplicating GVR mappings, Track observations
|
||||
or Tracks.
|
||||
|
||||
The real A6000 proof intentionally stopped before:
|
||||
|
||||
```text
|
||||
Sparse SfM
|
||||
Sparse reconstruction
|
||||
Dense / MVS
|
||||
multi-campaign fusion
|
||||
```
|
||||
|
||||
### Calibration status of the historical real campaigns
|
||||
|
||||
The historical S21 and A6000 Engine Bay campaigns currently remain `CALIBRATION_UNAVAILABLE` for
|
||||
the known-calibration Sparse SfM contract.
|
||||
|
||||
This is not a source failure, a quality rejection or permission to infer calibration from metadata.
|
||||
No pseudo-calibration, silent interpolation or inferred calibration identity is allowed.
|
||||
|
||||
Therefore real Sparse SfM for those historical campaigns remains:
|
||||
|
||||
```text
|
||||
BLOCKED_BY_KNOWN_CALIBRATION_DATA
|
||||
```
|
||||
|
||||
Calibration Science v1 defines the protocol for future physically controlled calibration
|
||||
acquisitions. Calibration Tooling v1 validates an already acquired Science v1 evidence bundle and
|
||||
produces the bounded L3DCALB1 artifact; Calibration Bootstrap v1 imports that artifact. Neither
|
||||
stage solves calibration internally or turns EXIF into scientific calibration.
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
TUI / Projet
|
||||
↓
|
||||
Task Queue bornée (un worker, ordre/backpressure)
|
||||
↓
|
||||
Resource Governor (admission et réservation)
|
||||
↓
|
||||
Task callback admis (parallélisme interne borné si prouvé)
|
||||
↓
|
||||
Résultats atomiques / persistants
|
||||
↓
|
||||
Viewer (consommation passive de snapshots)
|
||||
TUI / Project
|
||||
|
|
||||
v
|
||||
bounded Task Queue
|
||||
(one active callback)
|
||||
|
|
||||
v
|
||||
Resource Governor
|
||||
(admission and reservation)
|
||||
|
|
||||
v
|
||||
admitted Task callback
|
||||
(bounded internal participants when justified)
|
||||
|
|
||||
v
|
||||
atomic / persistent scientific publication
|
||||
|
|
||||
v
|
||||
passive snapshot consumers
|
||||
(viewer remains future work)
|
||||
```
|
||||
|
||||
### Invariants fondamentaux
|
||||
Core invariants:
|
||||
|
||||
- Aucun callback de tâche sans réservation active validée
|
||||
- La Queue/runtime ne décide jamais des ressources
|
||||
- Le Resource Governor est l'unique propriétaire des budgets
|
||||
- ncurses appartient exclusivement au thread principal
|
||||
- Les estimations de ressources sont immuables
|
||||
- Les buffers et files sont strictement bornés
|
||||
- no Task callback starts without a valid active reservation;
|
||||
- the Queue does not own resource policy;
|
||||
- the Resource Governor is the sole production resource authority;
|
||||
- ncurses remains owned by the main thread;
|
||||
- Task estimates and installed sequence contracts remain immutable for their defined lifetime;
|
||||
- buffers, queues, files, threads, participants and temporary work remain bounded;
|
||||
- owner-only durable publication does not imply serial preparation;
|
||||
- swap, zram and external scratch never become admitted RAM;
|
||||
- UMA GPU memory is charged exactly once against host memory.
|
||||
|
||||
### TUI opérationnelle
|
||||
## Current TUI
|
||||
|
||||
La TUI sépare le modèle de vue pur du rendu ncurses. Son observateur copie au
|
||||
plus 129 entrées Queue (64 pending, une active, 64 historiques) et coalesce les
|
||||
captures hôte autour d'une seconde ; aucun scan DB ou `/proc` volumineux n'a
|
||||
lieu par frame. La progression scientifique exacte provient seulement des
|
||||
compteurs durables typés. Le taux EWMA et l'ETA restent « calcul » jusqu'à deux
|
||||
intervalles positifs, excluent le préfixe repris et deviennent explicitement
|
||||
indéterminés, stalled ou throttled lorsque l'évidence l'exige.
|
||||
The TUI is a validated operational observatory and control center.
|
||||
|
||||
Les tailles supportées sont 100×30 et plus en vue complète, 72×20 en compacte
|
||||
de référence, et jusqu'au minimum 60×15 ; en dessous, seul « Terminal trop
|
||||
petit » est affiché. Les couleurs ont toujours un équivalent textuel/bold/dim.
|
||||
`F1` à `F7` ouvrent aide, projets, import, viewer futur, tâches, ressources et
|
||||
optique. Le segment littéral `F10 SSD` reste visible à 60 colonnes et déclenche
|
||||
uniquement l'action autorisée par le contrôleur. Pendant une saisie, seules
|
||||
Enter, Échap et F10 sont actives ; pendant un import, seules `X` et F10 le sont,
|
||||
et quitter/retour accueil sont explicitement désactivés.
|
||||
It provides bounded observation of:
|
||||
|
||||
Ouvrir, fermer ou changer de projet détruit et joint d'abord l'unique Queue,
|
||||
callbacks terminaux inclus, puis ferme Project DB et recrée une Queue vide. Le
|
||||
workflow optique utilise les alias metadata exacts, accepte normalement les
|
||||
objectifs manuels sans EXIF, crée des profils/configurations immuables et exige
|
||||
une affectation/sélection de calibration explicite et exactement compatible.
|
||||
- Project state;
|
||||
- Tasks and durable progress;
|
||||
- Resource Governor state;
|
||||
- CPU / RAM / swap / GPU information;
|
||||
- optical profiles and explicit calibration selection;
|
||||
- optional SSD state and safe control actions.
|
||||
|
||||
## Pipeline cible
|
||||
The ncurses renderer and input handling remain on the main thread. Runtime observation is bounded
|
||||
and coalesced; the renderer does not scan Project DB or `/proc` extensively per frame.
|
||||
|
||||
The validated layout supports:
|
||||
|
||||
```text
|
||||
Acquisitions
|
||||
→ catalogue
|
||||
→ features
|
||||
→ index visuel
|
||||
→ paires candidates
|
||||
→ matching
|
||||
→ vérification géométrique
|
||||
→ tracks / SfM
|
||||
→ dense
|
||||
→ mesh
|
||||
→ consolidation
|
||||
→ export
|
||||
full layout >= 100x30
|
||||
reference compact 72x20
|
||||
minimum supported 60x15
|
||||
```
|
||||
|
||||
Below the minimum, only the bounded terminal-too-small fallback is rendered.
|
||||
|
||||
The optical workflow supports electronic metadata aliases and manual lenses without EXIF. Missing,
|
||||
ambiguous or incompatible calibration remains visible and is never silently guessed.
|
||||
|
||||
## Target pipeline
|
||||
|
||||
```text
|
||||
acquisition
|
||||
-> catalog / Capture / provenance
|
||||
-> quality selection
|
||||
-> selected scientific representation
|
||||
-> features
|
||||
-> visual index
|
||||
-> candidate pairs
|
||||
-> matching
|
||||
-> geometric verification
|
||||
-> tracks
|
||||
-> Sparse SfM
|
||||
-> incremental / multi-campaign reconstruction
|
||||
-> dense / MVS
|
||||
-> mesh
|
||||
-> refinement
|
||||
-> texturing
|
||||
-> consolidation
|
||||
-> export
|
||||
```
|
||||
|
||||
## Planned product areas
|
||||
|
||||
The following areas remain future work and must not be confused with current implementation:
|
||||
|
||||
- durable dense / mesh publication;
|
||||
- full Dense/MVS orchestration;
|
||||
- mesh refinement and texturing;
|
||||
- final export workflow;
|
||||
- viewer;
|
||||
- offline coverage analysis;
|
||||
- suggested supplementary viewpoints;
|
||||
- live camera localization;
|
||||
- live coverage overlay;
|
||||
- A6000 live acquisition integration;
|
||||
- S21 live acquisition integration;
|
||||
- capture guidance;
|
||||
- video ingestion and deterministic keyframe extraction;
|
||||
- explicit Task-owned scratch consumers;
|
||||
- general DAG / dependency scheduling.
|
||||
|
||||
The final product contracts for these areas are being defined separately before implementation.
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Documentation index](docs/README.md)
|
||||
|
||||
### Architecture
|
||||
- [Vue d'ensemble](docs/architecture/overview.md)
|
||||
|
||||
- [Architecture overview](docs/architecture/overview.md)
|
||||
- [Runtime](docs/architecture/runtime.md)
|
||||
- [Système de tâches](docs/architecture/task_system.md)
|
||||
- [Registry des types de tâches](docs/architecture/task_kind_registry.md)
|
||||
- [File de tâches](docs/architecture/task_queue.md)
|
||||
- [Task system](docs/architecture/task_system.md)
|
||||
- [Task Kind Registry](docs/architecture/task_kind_registry.md)
|
||||
- [Task Queue](docs/architecture/task_queue.md)
|
||||
- [Resource Governor](docs/architecture/resource_governor.md)
|
||||
- [Parallélisme interne borné](docs/architecture/internal_parallelism.md)
|
||||
- [Pipeline sensible aux ressources](docs/architecture/resource_aware_pipeline.md)
|
||||
- [Intégration Queue/runtime ↔ Governor](docs/architecture/scheduler_resource_integration.md)
|
||||
- [Pipeline de reconstruction](docs/architecture/reconstruction_pipeline.md)
|
||||
- [Persistance](docs/architecture/persistence.md)
|
||||
- [Base de données projet](docs/architecture/project_database.md)
|
||||
- [Bounded internal parallelism](docs/architecture/internal_parallelism.md)
|
||||
- [Resource-aware pipeline](docs/architecture/resource_aware_pipeline.md)
|
||||
- [Queue / runtime / Governor integration](docs/architecture/scheduler_resource_integration.md)
|
||||
- [Reconstruction pipeline](docs/architecture/reconstruction_pipeline.md)
|
||||
- [Persistence](docs/architecture/persistence.md)
|
||||
- [Project Database](docs/architecture/project_database.md)
|
||||
- [Feature Store](docs/architecture/feature_store.md)
|
||||
- [Precision Feature Pipeline v1A](docs/architecture/precision_feature_pipeline.md)
|
||||
- [Visual Index](docs/architecture/visual_index.md)
|
||||
|
|
@ -202,36 +330,59 @@ Acquisitions
|
|||
- [Geometric Verification](docs/architecture/geometric_verification.md)
|
||||
- [Geometric Verifier](docs/architecture/geometric_verifier.md)
|
||||
- [Track Model](docs/architecture/tracks.md)
|
||||
- [Sparse SfM / Triangulation — Gate A](docs/architecture/sparse_sfm.md)
|
||||
- [Backend Vulkan ORB](docs/architecture/vulkan_matcher.md)
|
||||
- [Track Builder](docs/architecture/track_builder.md)
|
||||
- [Sparse SfM](docs/architecture/sparse_sfm.md)
|
||||
- [Calibration Bootstrap v1](docs/architecture/calibration_bootstrap.md)
|
||||
- [Calibration Science v1](docs/architecture/calibration_science_v1.md)
|
||||
- [Calibration Solver Preflight v1](docs/architecture/calibration_solver_preflight_v1.md)
|
||||
- [Photo Quality Triage](docs/architecture/photo_quality_triage.md)
|
||||
- [Vulkan ORB Matcher](docs/architecture/vulkan_matcher.md)
|
||||
- [Viewer](docs/architecture/viewer.md)
|
||||
- [Resource Boundary — No New Resource Subsystem](docs/architecture/resource_boundary.md)
|
||||
- [Audit global de maintenance — état consolidé](docs/architecture/global_maintenance_audit.md)
|
||||
- [Revue historique des fondations](docs/architecture/foundation_review.md)
|
||||
- [Resource Boundary](docs/architecture/resource_boundary.md)
|
||||
|
||||
### Historical audit records
|
||||
|
||||
- [Global Maintenance Audit](docs/architecture/global_maintenance_audit.md)
|
||||
- [Foundation Review](docs/architecture/foundation_review.md)
|
||||
|
||||
Historical audit records preserve the state and evidence of their checkpoint. Older schema versions,
|
||||
Task counts or resource measurements inside them must not be mechanically modernized.
|
||||
|
||||
### Concepts
|
||||
|
||||
- [Scan Sets](docs/concepts/scan_sets.md)
|
||||
- [Index visuel](docs/concepts/visual_index.md)
|
||||
- [Matching et tracks](docs/concepts/matching_and_tracks.md)
|
||||
- [Couches de reconstruction](docs/concepts/reconstruction_layers.md)
|
||||
- [Contraintes géométriques](docs/concepts/geometric_constraints.md)
|
||||
- [Visual Index](docs/concepts/visual_index.md)
|
||||
- [Matching and Tracks](docs/concepts/matching_and_tracks.md)
|
||||
- [Reconstruction Layers](docs/concepts/reconstruction_layers.md)
|
||||
- [Geometric Constraints](docs/concepts/geometric_constraints.md)
|
||||
|
||||
Some concept documents are explicitly historical or superseded. Their status header determines
|
||||
whether they are current authority.
|
||||
|
||||
### Development
|
||||
|
||||
### Développement
|
||||
- [Build](docs/development/build.md)
|
||||
- [Tests](docs/development/testing.md)
|
||||
- [Concurrence](docs/development/concurrency.md)
|
||||
- [Profil de performance de la machine cible](docs/performance/target_hardware.md)
|
||||
- [Testing](docs/development/testing.md)
|
||||
- [Concurrency](docs/development/concurrency.md)
|
||||
- [Validation-host performance profile](docs/performance/target_hardware.md)
|
||||
|
||||
### Roadmap and audits
|
||||
|
||||
### Roadmap
|
||||
- [Roadmap](docs/roadmap/roadmap.md)
|
||||
- [Documentation Inventory Audit](docs/audits/documentation_inventory.md)
|
||||
|
||||
## Build rapide
|
||||
## Build
|
||||
|
||||
Meson and Ninja are the canonical build path.
|
||||
|
||||
```sh
|
||||
CC=clang meson setup build --wipe
|
||||
meson compile -C build -j8
|
||||
CC=clang meson setup build
|
||||
meson compile -C build
|
||||
```
|
||||
|
||||
Build parallelism should use safe host capacity. Do not treat a historical `-j8` or the current
|
||||
reference-host result of approximately `-j12` as a portable constant.
|
||||
|
||||
## Tests
|
||||
|
||||
```sh
|
||||
|
|
@ -239,57 +390,41 @@ meson test -C build --print-errorlogs
|
|||
git diff --check
|
||||
```
|
||||
|
||||
Pour les changements sensibles à la mémoire ou à la concurrence, ajouter ASan/UBSan et TSan.
|
||||
For memory-, lifetime- or concurrency-sensitive changes, use the applicable ASan/UBSan and TSan
|
||||
validation described in [Testing](docs/development/testing.md) and preserve documented third-party
|
||||
sanitizer qualifications.
|
||||
|
||||
## Statut
|
||||
## Repository language
|
||||
|
||||
Lardon3D est en développement actif. La persistance des tâches, le catalogue,
|
||||
le Feature Store multipasse, le Visual Index ORB, Candidate Pair Generator,
|
||||
Matcher v1, Geometric Verification Model et Geometric Verifier Fundamental v3
|
||||
sont implémentés. Le runtime Feature + Matcher + Verifier emploie des tâches durables,
|
||||
de petits lots, le Resource Governor interactif et un hot path Vulkan ORB exact avec
|
||||
fallback CPU. La feasibility Vulkan SIFT/RootSIFT a été rejetée ; ces deux matchers
|
||||
restent sur OpenCV L2. Track Model/Builder, les primitives géométriques Gate C,
|
||||
le noyau Sparse SfM incrémental Gate D et le Bundle Adjustment final Gate E sont
|
||||
implémentés et validés. L'orchestration Sparse SfM Gate F est PASS / FROZEN ;
|
||||
l'intégration Governor Gate G est **PASS / FROZEN**. MVS-M1 est **PASS / FROZEN** :
|
||||
une frontière OpenMVS v2.4.0 externe et bornée, sans publication dense durable
|
||||
ni MVS complet. Les sources sont liées par SHA-256
|
||||
complet, borné à 1 GiB par fichier régulier (sans budget agrégé de dataset) ; les
|
||||
octets source restent un binding distinct de l'identité dense. Celle-ci lie la
|
||||
reconstruction de base, le jeu d'images source, le `calibration_scope_identity`
|
||||
historique, le binding numérique de calibration MVS `L3DMCAL1` v1, le backend et
|
||||
les paramètres dans `L3DMDID2` v2 (220 octets). Chaque appel utilise un espace de
|
||||
travail privé neuf sous le staging appelant, sans réemploi d'une scène,
|
||||
profondeur, cache ou sortie antérieure. Le DAG, le viewer et les autres étapes
|
||||
denses restent des tickets séparés planifiés.
|
||||
La fondation Project DB v22, `raw.develop` et Calibration Bootstrap v1 reste
|
||||
**PASS / FROZEN**. La tête courante v23 ajoute seulement le contexte optique
|
||||
générique : neuf tables, migration transactionnelle sans backfill, objectifs
|
||||
manuels sans EXIF, configurations multiples par campagne et sélection de
|
||||
calibration exactement compatible. Les migrations de copies réelles S21/A6000
|
||||
ont conservé leurs lignes scientifiques et laissé l'overlay vide. Les campagnes
|
||||
réelles S21 et A6000 Engine Bay sont
|
||||
`CALIBRATION_UNAVAILABLE` par non-identifiabilité scientifique des données de
|
||||
calibration connues ; le Sparse SfM réel reste
|
||||
`BLOCKED_BY_KNOWN_CALIBRATION_DATA`, sans pseudo-calibration ni import inféré.
|
||||
Le Resource Governor ne constitue pas un Resource System générique : il reste
|
||||
l'unique propriétaire des budgets et le seul orchestrateur des leases scratch
|
||||
de production. Le contrôleur SSD UDisks2 est une frontière physique séparée,
|
||||
jamais un second scheduler ou Governor. Les seize Task kinds actuels ne
|
||||
consomment encore aucun scratch : l'espace disponible est une capacité
|
||||
observable, pas un usage fabriqué, et scratch/swap ne deviennent jamais de la
|
||||
RAM. La TUI/F10 et cette intégration sont validées opérationnellement. L'audit
|
||||
global est désormais `GLOBAL_MAINTENANCE_AUDIT=PASS/FROZEN` : après les builds
|
||||
Clang frais portable/Vulkan, les suites normales 64/64 et 65/65, les sanitizers
|
||||
applicables, TSan et les contrôles ABI, l'unique revue finale indépendante
|
||||
GPT-5.6 SOL/ULTRA a conclu PASS sans finding bloquant. Elle a indépendamment
|
||||
rejoué le build portable, la suite 64/64, une matrice focalisée 15/15 et les
|
||||
76/76 probes strictes C17/C++17 couvrant 19 headers publics modifiés/nouveaux,
|
||||
ainsi que l'ABI, les négatifs de seams production, le SHA du manifest GV et le
|
||||
diff-check. Ce gel clôt la gate de maintenance ; il n'exécute pas à lui seul la
|
||||
tranche scientifique suivante.
|
||||
The canonical language for repository documentation, agent contracts and production source comments
|
||||
is English.
|
||||
|
||||
## Licence
|
||||
User-interface language is a separate product/localization concern.
|
||||
|
||||
Projet privé - Tous droits réservés.
|
||||
## Status
|
||||
|
||||
Lardon3D is under active development.
|
||||
|
||||
The persistent pre-SfM pipeline is implemented through Tracks, the Sparse SfM C-G capability is
|
||||
implemented and frozen at its documented boundaries, Phase H v1 is frozen, and MVS-M1 provides the
|
||||
validated external OpenMVS boundary. Full real known-calibration Sparse SfM, dense publication,
|
||||
mesh, texturing, viewer and live capture guidance remain future work.
|
||||
|
||||
The global maintenance checkpoint remains:
|
||||
|
||||
```text
|
||||
global-maintenance-2026-09-01
|
||||
```
|
||||
|
||||
The later real A6000 pre-SfM checkpoint is:
|
||||
|
||||
```text
|
||||
real-a6000-pre-sfm-2026-09-02
|
||||
```
|
||||
|
||||
Future reviews should preserve historical checkpoint meaning and review only the relevant delta
|
||||
unless concrete evidence requires reopening an unchanged FROZEN boundary.
|
||||
|
||||
## License
|
||||
|
||||
Lardon3D is licensed under the MIT License.
|
||||
|
|
|
|||
254
docs/README.md
Normal file
254
docs/README.md
Normal file
|
|
@ -0,0 +1,254 @@
|
|||
# Lardon3D Documentation Index
|
||||
|
||||
## Status
|
||||
|
||||
```text
|
||||
DOCUMENTATION_INDEX=CURRENT
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
CURRENT_PRODUCTION_TASK_KINDS=16
|
||||
|
||||
REAL_S21_TRACKS=PASS/FROZEN
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
PRODUCT_DEFINITION_V1=PASS/FROZEN
|
||||
PROMPT_TREE=NEXT
|
||||
```
|
||||
|
||||
This index separates current authority, historical evidence and future product-definition work.
|
||||
|
||||
Repository documentation is read in context: an older schema number, Task count or resource measurement
|
||||
inside a historical checkpoint is not stale merely because the current repository has moved forward.
|
||||
|
||||
## Authority order
|
||||
|
||||
When current-state prose disagrees, use this order:
|
||||
|
||||
1. `AGENTS.md` for repository-wide engineering/agent obligations and protected boundaries.
|
||||
2. Specialized architecture documents for the subsystem contract they own.
|
||||
3. `docs/roadmap/roadmap.md` for current lifecycle / next work.
|
||||
4. `README.md` and architecture overview for concise summaries.
|
||||
5. Historical audits only for the checkpoint they explicitly record.
|
||||
|
||||
Executable source and schema implementation remain authoritative when a documentation excerpt claims to
|
||||
quote an exact API, DDL or constant.
|
||||
|
||||
## Current repository state
|
||||
|
||||
```text
|
||||
Project DB head v25
|
||||
Production Task kinds 16
|
||||
|
||||
v22 selected scientific execution foundation
|
||||
v23 generic optical-context overlay
|
||||
v24 raw.develop.batch/1 persistence
|
||||
v25 features.extract.batch/1 persistence
|
||||
```
|
||||
|
||||
Canonical resource objective:
|
||||
|
||||
```text
|
||||
MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF
|
||||
```
|
||||
|
||||
Reference-host values are measurements, not portable constants.
|
||||
|
||||
## Current real-data checkpoints
|
||||
|
||||
### S21
|
||||
|
||||
```text
|
||||
REAL_S21_TRACKS=PASS/FROZEN
|
||||
```
|
||||
|
||||
The retained S21 proof reaches Tracks.
|
||||
|
||||
### A6000
|
||||
|
||||
```text
|
||||
real-a6000-pre-sfm-2026-09-02
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
Retained A6000 counts:
|
||||
|
||||
```text
|
||||
Feature Sets 689
|
||||
Candidate Pairs 38,420
|
||||
Match Results 38,420
|
||||
Applicable GVRs 37,805
|
||||
Verified GVRs 10,952
|
||||
Rejected GVRs 26,853
|
||||
Track Sets 1
|
||||
Tracks 130,714
|
||||
Track observations 318,944
|
||||
```
|
||||
|
||||
The A6000 checkpoint stops before real Sparse SfM and Dense/MVS.
|
||||
|
||||
Sparse SfM capability exists through frozen Gates C-G. Real Sparse SfM on the historical S21/A6000
|
||||
campaigns remains blocked by known-calibration data.
|
||||
|
||||
## Current architecture
|
||||
|
||||
Start here:
|
||||
|
||||
- [Architecture overview](architecture/overview.md)
|
||||
- [Reconstruction pipeline](architecture/reconstruction_pipeline.md)
|
||||
- [Runtime](architecture/runtime.md)
|
||||
- [Persistence](architecture/persistence.md)
|
||||
- [Project Database](architecture/project_database.md)
|
||||
|
||||
### Task / resource execution
|
||||
|
||||
- [Task System](architecture/task_system.md)
|
||||
- [Task Queue](architecture/task_queue.md)
|
||||
- [Task Kind Registry](architecture/task_kind_registry.md)
|
||||
- [Resource Governor](architecture/resource_governor.md)
|
||||
- [Resource Boundary](architecture/resource_boundary.md)
|
||||
- [Resource-aware Pipeline](architecture/resource_aware_pipeline.md)
|
||||
- [Internal Parallelism](architecture/internal_parallelism.md)
|
||||
- [Scheduler / Resource Integration](architecture/scheduler_resource_integration.md)
|
||||
|
||||
Resource authority split:
|
||||
|
||||
```text
|
||||
resource_governor.md runtime policy
|
||||
AGENTS.md engineering / agent obligations
|
||||
target_hardware.md reference-host evidence
|
||||
other documents scoped summaries and links
|
||||
```
|
||||
|
||||
## Scientific pipeline
|
||||
|
||||
- [Photo Quality Triage](architecture/photo_quality_triage.md)
|
||||
- [Feature Store](architecture/feature_store.md)
|
||||
- [Precision Feature Pipeline](architecture/precision_feature_pipeline.md)
|
||||
- [Visual Index](architecture/visual_index.md)
|
||||
- [Candidate Pair](architecture/candidate_pair.md)
|
||||
- [Match Result](architecture/match_result.md)
|
||||
- [Matcher](architecture/matcher.md)
|
||||
- [Geometric Verification Model](architecture/geometric_verification.md)
|
||||
- [Geometric Verifier](architecture/geometric_verifier.md)
|
||||
- [Track Model](architecture/tracks.md)
|
||||
- [Track Builder](architecture/track_builder.md)
|
||||
- [Sparse SfM](architecture/sparse_sfm.md)
|
||||
- [Vulkan ORB Matcher](architecture/vulkan_matcher.md)
|
||||
|
||||
Current production Geometric Verifier lineage:
|
||||
|
||||
```text
|
||||
Fundamental v1 historical valid identity
|
||||
Fundamental v2 historical valid identity
|
||||
Fundamental v3 current production identity
|
||||
```
|
||||
|
||||
Historical verifier results are never relabelled.
|
||||
|
||||
## Calibration
|
||||
|
||||
- [Calibration Science v1](architecture/calibration_science_v1.md)
|
||||
- [Calibration Bootstrap v1](architecture/calibration_bootstrap.md)
|
||||
- [Calibration Solver Preflight v1](architecture/calibration_solver_preflight_v1.md)
|
||||
|
||||
Historical S21/A6000 geometry evidence must not be retroactively given invented calibration.
|
||||
|
||||
## Development
|
||||
|
||||
- [Build](development/build.md)
|
||||
- [Testing](development/testing.md)
|
||||
- [Concurrency](development/concurrency.md)
|
||||
- [Target Hardware](performance/target_hardware.md)
|
||||
|
||||
Build/test parallelism is host-aware. Fixed historical `-j8`, `-j12` or serialized-suite evidence does
|
||||
not become a portable default.
|
||||
|
||||
TSan/OpenCV/TBB qualifications and Vulkan validation remain separate evidence boundaries.
|
||||
|
||||
## Roadmap
|
||||
|
||||
- [Roadmap](roadmap/roadmap.md)
|
||||
|
||||
The roadmap owns the current lifecycle / next-work cursor.
|
||||
|
||||
Historical progress cursors inside other records do not override it.
|
||||
|
||||
## Audits
|
||||
|
||||
- [Documentation Inventory Audit](audits/documentation_inventory.md)
|
||||
- [Global Maintenance Audit](architecture/global_maintenance_audit.md)
|
||||
- [Foundation Review](architecture/foundation_review.md)
|
||||
|
||||
### Historical audit rule
|
||||
|
||||
`global_maintenance_audit.md` and `foundation_review.md` are historical evidence.
|
||||
|
||||
Do not mechanically modernize their old:
|
||||
|
||||
- schema version;
|
||||
- Task count;
|
||||
- compiler/build count;
|
||||
- resource measurement;
|
||||
- checkpoint status.
|
||||
|
||||
The later A6000 checkpoint adds new operational evidence; it does not erase the maintenance checkpoint.
|
||||
|
||||
## Concepts
|
||||
|
||||
- [Scan Sets](concepts/scan_sets.md)
|
||||
- [Visual Index concept](concepts/visual_index.md)
|
||||
- [Matching and Tracks](concepts/matching_and_tracks.md)
|
||||
- [Reconstruction Layers](concepts/reconstruction_layers.md)
|
||||
- [Geometric Constraints](concepts/geometric_constraints.md)
|
||||
|
||||
Concept documents may be historical or explanatory. They do not outrank the current specialized
|
||||
architecture contract.
|
||||
|
||||
## Product definition
|
||||
|
||||
Canonical target-product authority:
|
||||
|
||||
- [Product Definition v1](product/product_definition.md)
|
||||
|
||||
```text
|
||||
PRODUCT_DEFINITION_V1=PASS/FROZEN
|
||||
IMPLEMENTATION_AUTHORIZATION=NO
|
||||
PROMPT_TREE=NEXT
|
||||
```
|
||||
|
||||
The product definition freezes desired end-product behavior without reopening existing FROZEN
|
||||
scientific contracts. Future implementation is authorized only by the subsequent execution-contract
|
||||
phase under explicit human authority.
|
||||
|
||||
## Repository language
|
||||
|
||||
Canonical target:
|
||||
|
||||
```text
|
||||
DOCUMENTATION_LANGUAGE=ENGLISH
|
||||
SOURCE_COMMENT_LANGUAGE=ENGLISH
|
||||
AGENT_CONTRACT_LANGUAGE=ENGLISH
|
||||
USER_INTERFACE_LANGUAGE=ENGLISH
|
||||
```
|
||||
|
||||
The current-state documentation findings have been remediated in English.
|
||||
|
||||
Some untouched historical/current documents may still require a mechanical language-only normalization
|
||||
pass. Such translation must preserve historical facts and must not silently modernize scientific or
|
||||
lifecycle state.
|
||||
|
||||
Executable UI strings are changed only in an explicitly scoped UI-language implementation pass.
|
||||
|
||||
## Navigation rule for future agents
|
||||
|
||||
Before implementation:
|
||||
|
||||
```text
|
||||
read AGENTS.md
|
||||
-> read README.md
|
||||
-> read this index
|
||||
-> read the specialized architecture contract
|
||||
-> read the current roadmap
|
||||
-> inspect historical audits only when their evidence is relevant
|
||||
```
|
||||
|
||||
Never use an older historical checkpoint as a substitute for current authority.
|
||||
|
|
@ -1,48 +1,71 @@
|
|||
# Sous-système Candidate Pair
|
||||
# Candidate Pair subsystem
|
||||
|
||||
## Vision
|
||||
|
||||
Le sous-système Candidate Pair répond uniquement à la question :
|
||||
|
||||
> « Quelles paires d'images valent la peine d'être présentées au Matcher ? »
|
||||
|
||||
Il ne répond **PAS** à :
|
||||
|
||||
> « Ces images ont-elle réellement des correspondances ? »
|
||||
|
||||
et ne contient **aucune** validation géométrique.
|
||||
|
||||
## Frontières
|
||||
## Status
|
||||
|
||||
```text
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
CANDIDATE_PAIR_MODEL=v1
|
||||
CANDIDATE_PAIR_MODEL_STATUS=IMPLEMENTED
|
||||
CANDIDATE_PAIR_TASK=candidate_pair.generate/1
|
||||
CANDIDATE_PAIR_TASK_STATUS=IMPLEMENTED
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
The Candidate Pair scientific model remains the Project DB v8 model. The durable Candidate Pair Task
|
||||
was added in Project DB v9. Later schema versions through v25 are additive and do not redefine
|
||||
Candidate Pair identity.
|
||||
|
||||
This document owns the Candidate Pair subsystem contract. Resource policy is governed by the Resource
|
||||
Governor and the canonical resource documents; Candidate Pair declares bounded demand and preserves
|
||||
deterministic scientific output.
|
||||
|
||||
## Purpose
|
||||
|
||||
The Candidate Pair subsystem answers one question:
|
||||
|
||||
> Which image pairs are worth presenting to the Matcher?
|
||||
|
||||
It does not answer whether two images actually have descriptor correspondences and it performs no
|
||||
geometric verification.
|
||||
|
||||
The downstream Matcher is implemented and consumes persisted Candidate Pairs, but Matcher science and
|
||||
persistence are outside this subsystem.
|
||||
|
||||
```text
|
||||
Visual Index
|
||||
↓
|
||||
|
|
||||
v
|
||||
Candidate Pair Generator
|
||||
↓
|
||||
|
|
||||
v
|
||||
Candidate Pair persistence
|
||||
↓
|
||||
Matcher — HORS SCOPE de ce ticket
|
||||
|
|
||||
v
|
||||
Matcher
|
||||
```
|
||||
|
||||
Le Matcher est un consommateur des paires persistées. Il n'est pas
|
||||
implémenté dans ce sous-système.
|
||||
## Core invariants
|
||||
|
||||
## Invariants fondamentaux
|
||||
| Invariant | Contract |
|
||||
| --- | --- |
|
||||
| Symmetry | `(A,B)` and `(B,A)` are the same scientific pair |
|
||||
| Canonical order | Persist with `image_id_a < image_id_b` |
|
||||
| No self-pairs | `image_id_a != image_id_b`, implied by the SQL ordering check |
|
||||
| Persistent uniqueness | `UNIQUE(image_id_a, image_id_b)` |
|
||||
| Bounded query result | `top_k <= LARDON3D_VISUAL_INDEX_TOP_K_MAX = 256` |
|
||||
| Determinism | Same inputs and options produce the same pair decisions in the same canonical order |
|
||||
| Idempotence | Repeating generation does not duplicate persisted pairs |
|
||||
| Persistent model | Candidate Pair rows were introduced by Project DB v8 |
|
||||
| Durable execution | `candidate_pair.generate/1` was introduced by Project DB v9 |
|
||||
|
||||
| Invariant | Description |
|
||||
|-----------|-------------|
|
||||
| **Symétrie** | A,B == B,A |
|
||||
| **Ordre canonique** | Stockage avec `image_id_a < image_id_b` |
|
||||
| **Self-pairs interdits** | `image_id_a != image_id_b` (implicite via CHECK SQL) |
|
||||
| **Unicité persistante** | `UNIQUE(image_id_a, image_id_b)` |
|
||||
| **Résultat borné** | `top_k <= LARDON3D_VISUAL_INDEX_TOP_K_MAX = 256` par requête |
|
||||
| **Déterminisme** | Mêmes entrées/configuration → mêmes paires dans le même ordre |
|
||||
| **Idempotence** | Répétition sans duplication |
|
||||
| **Persistance durable** | Paires persistées dans Project DB v8 |
|
||||
Candidate Pair identity is the canonical unordered image pair. Retrieval score, Visual Index provenance,
|
||||
Task ID, timestamps and operational resource choices do not enter that identity.
|
||||
|
||||
## Modèle persistant
|
||||
## Persistent model
|
||||
|
||||
### Table `candidate_pairs` (Project DB v8)
|
||||
### `candidate_pairs` — Project DB v8
|
||||
|
||||
```sql
|
||||
CREATE TABLE candidate_pairs(
|
||||
|
|
@ -57,17 +80,24 @@ CREATE INDEX candidate_pairs_image_a_idx ON candidate_pairs(image_id_a);
|
|||
CREATE INDEX candidate_pairs_image_b_idx ON candidate_pairs(image_id_b);
|
||||
```
|
||||
|
||||
The executable schema in `src/project_db.c` remains authoritative if prose and SQL excerpts ever
|
||||
diverge.
|
||||
|
||||
### Public Project DB API
|
||||
|
||||
The Candidate Pair persistence surface includes:
|
||||
|
||||
- `lardon3d_project_db_create_candidate_pair()`;
|
||||
- `lardon3d_project_db_load_candidate_pair()`;
|
||||
- `lardon3d_project_db_find_candidate_pair()`;
|
||||
- `lardon3d_project_db_list_candidate_pairs()`.
|
||||
|
||||
Creation canonicalizes the image order and persistence enforces uniqueness.
|
||||
|
||||
## Single-source generation
|
||||
|
||||
### API
|
||||
|
||||
- `lardon3d_project_db_create_candidate_pair()` — INSERT avec canonicalisation
|
||||
- `lardon3d_project_db_load_candidate_pair()` — SELECT par ID
|
||||
- `lardon3d_project_db_find_candidate_pair()` — SELECT par (image_a, image_b)
|
||||
- `lardon3d_project_db_list_candidate_pairs()` — SELECT paginé ORDER BY id
|
||||
|
||||
## Génération single-source
|
||||
|
||||
### Prototype
|
||||
|
||||
```c
|
||||
Lardon3DVisualIndexResult lardon3d_candidate_pair_generate(
|
||||
const char *project_path, Lardon3DProjectDb *database,
|
||||
|
|
@ -76,99 +106,117 @@ Lardon3DVisualIndexResult lardon3d_candidate_pair_generate(
|
|||
Lardon3DCandidatePairGenStats *stats);
|
||||
```
|
||||
|
||||
### Algorithme
|
||||
### Algorithm
|
||||
|
||||
1. Charger le FeatureSet source
|
||||
2. Obtenir `source_image_id`
|
||||
3. Interroger le Visual Index avec `query_options`
|
||||
4. Pour chaque candidat retourné :
|
||||
- Exclure les self-pairs
|
||||
- Canonicaliser l'ordre (image_a < image_b)
|
||||
- Chercher si la paire existe déjà (`find_candidate_pair`)
|
||||
- Créer si absente (`create_candidate_pair`)
|
||||
5. Retourner les statistiques `{generated, skipped, queried}`
|
||||
For one source Feature Set:
|
||||
|
||||
### Statistiques
|
||||
1. load the source Feature Set;
|
||||
2. obtain its `source_image_id`;
|
||||
3. query the Visual Index with the supplied options;
|
||||
4. for every returned candidate:
|
||||
- reject self-pairs;
|
||||
- canonicalize the image order;
|
||||
- find an existing Candidate Pair;
|
||||
- create the pair only when absent;
|
||||
5. return bounded generation statistics.
|
||||
|
||||
The generator does not perform descriptor matching and does not perform geometric verification.
|
||||
|
||||
### Statistics
|
||||
|
||||
```c
|
||||
typedef struct {
|
||||
uint32_t generated_count; // paires nouvellement créées
|
||||
uint32_t skipped_count; // paires déjà existantes (idempotence)
|
||||
uint32_t queried_count; // candidats retournés par le Visual Index
|
||||
uint32_t generated_count;
|
||||
uint32_t skipped_count;
|
||||
uint32_t queried_count;
|
||||
} Lardon3DCandidatePairGenStats;
|
||||
```
|
||||
|
||||
## Score et provenance
|
||||
`generated_count` counts newly persisted pairs. `skipped_count` counts pairs already present.
|
||||
`queried_count` counts candidates returned by the Visual Index query.
|
||||
|
||||
Le score de retrieval et la provenance Visual Index ne sont **PAS**
|
||||
persistés dans la table `candidate_pairs` pour les raisons suivantes :
|
||||
## Retrieval score and provenance
|
||||
|
||||
- L'identité Candidate Pair est纯粹 géométrique : (image_a, image_b)
|
||||
- Le score dépend de la configuration du Visual Index et peut changer
|
||||
- Le Matcher calculera ses propres scores de matching
|
||||
- La séparation des responsabilités est plus nette
|
||||
Retrieval score and Visual Index provenance are intentionally not stored in `candidate_pairs`.
|
||||
|
||||
Le score reste accessible via le Visual Index si nécessaire.
|
||||
Reasons:
|
||||
|
||||
## Déterminisme
|
||||
- Candidate Pair identity is only the canonical image pair;
|
||||
- retrieval score depends on Visual Index configuration;
|
||||
- a later Visual Index execution may score the same pair differently;
|
||||
- Matcher owns descriptor-level matching evidence;
|
||||
- keeping retrieval evidence out of Candidate Pair identity preserves subsystem separation.
|
||||
|
||||
### Déterministe
|
||||
A generation fingerprint describes the generation request. It does not change the identity of an
|
||||
already persisted Candidate Pair row.
|
||||
|
||||
- Mêmes entrées → mêmes paires
|
||||
- Même ordre de sélection top-K
|
||||
- Mêmes décisions de déduplication
|
||||
## Determinism
|
||||
|
||||
### Non déterministe
|
||||
### Deterministic inputs and decisions
|
||||
|
||||
- `created_at` (timestamp Unix, informatif uniquement)
|
||||
- `candidate_pair_id` (AUTOINCREMENT, identifiant technique)
|
||||
For identical immutable inputs and options, the subsystem preserves:
|
||||
|
||||
### Tie-breaks
|
||||
- the same Visual Index query contract;
|
||||
- the same top-K selection semantics;
|
||||
- the same canonical image ordering;
|
||||
- the same self-pair rejection;
|
||||
- the same deduplication decisions;
|
||||
- the same publication order for owner-published results.
|
||||
|
||||
En cas d'égalité de score dans le Visual Index, l'ordre est déterministe
|
||||
selon l'implémentation LSH (ordre des Feature Sets).
|
||||
### Non-scientific values
|
||||
|
||||
## Invalidation
|
||||
These values are not Candidate Pair scientific identity:
|
||||
|
||||
### Événements et impact
|
||||
- `created_at`;
|
||||
- `candidate_pair_id`;
|
||||
- Task ID;
|
||||
- resource reservation ID;
|
||||
- admitted CPU count;
|
||||
- admitted batch size.
|
||||
|
||||
| Événement | Ce qui devient invalide | Ce qui reste réutilisable | Ce qui doit être recalculé |
|
||||
|-----------|------------------------|--------------------------|---------------------------|
|
||||
| Nouvelle image | Rien (incrémental) | Paires existantes | Nouvelles requêtes Visual Index |
|
||||
| Nouveau FeatureSet | Rien | Paires existantes | Requête depuis ce FeatureSet |
|
||||
| FeatureSet remplacé | Paires basées sur ce FeatureSet | Autres paires | Nouvelle requête depuis ce FeatureSet |
|
||||
| Visual Index reconstruit | Toutes les paires (nouvelle config) | Rien | Tout recalcul |
|
||||
| Configuration top-K modifiée | Rien (borné par requête) | Paires existantes | Nouvelles requêtes avec nouveau top_k |
|
||||
| Filtre modifié | Rien | Paires existantes | Nouvelles requêtes avec nouveau filtre |
|
||||
| Relance après interruption | Rien | Paires déjà persistées | Suite du traitement |
|
||||
`candidate_pair_id` is a durable technical identity allocated by SQLite. It is not a scientific
|
||||
fingerprint.
|
||||
|
||||
### Politique
|
||||
## Generation fingerprint
|
||||
|
||||
L'invalidation est aussi locale que possible. On ne supprime jamais
|
||||
toutes les paires du projet suite à une modification locale.
|
||||
|
||||
### Vérification par fingerprint
|
||||
|
||||
Le fingerprint permet de vérifier si une génération doit être recalculée :
|
||||
### API
|
||||
|
||||
```c
|
||||
unsigned char fp_courant[32], fp_enregistre[32];
|
||||
lardon3d_candidate_pair_generation_fingerprint(..., fp_courant);
|
||||
// Si fp_courant != fp_enregistre → recalcul nécessaire
|
||||
void lardon3d_candidate_pair_generation_fingerprint(
|
||||
uint64_t visual_index_id, uint64_t source_feature_set_id,
|
||||
const Lardon3DVisualIndexQueryOptions *query_options,
|
||||
unsigned char fingerprint[32]);
|
||||
```
|
||||
|
||||
### Réutilisation
|
||||
### Included fields
|
||||
|
||||
- Même fingerprint → résultat réutilisable
|
||||
- Différent fingerprint → recalcul nécessaire
|
||||
- Les paires existantes sont conservées même si le fingerprint change
|
||||
The generation fingerprint includes:
|
||||
|
||||
## Batch projet
|
||||
- `visual_index_id`;
|
||||
- `source_feature_set_id`;
|
||||
- `query_options->top_k`;
|
||||
- `query_options->minimum_evidence_count`;
|
||||
- `query_options->scanset_filter`;
|
||||
- `query_options->exclude_same_asset`.
|
||||
|
||||
### Granularité
|
||||
### Excluded fields
|
||||
|
||||
La génération batch traite un ensemble de FeatureSets de manière
|
||||
bornée et déterministe.
|
||||
It excludes:
|
||||
|
||||
- `created_at`;
|
||||
- `candidate_pair_id`;
|
||||
- operational CPU/batch admission;
|
||||
- processing order of unrelated source Feature Sets.
|
||||
|
||||
### Reuse meaning
|
||||
|
||||
The same fingerprint means the same generation request may be reused.
|
||||
|
||||
A different fingerprint means the generation request must be evaluated again. Existing canonical
|
||||
Candidate Pair rows are not silently deleted merely because a different generation request is run;
|
||||
idempotent persistence may reuse rows that remain selected.
|
||||
|
||||
## Project batch generation
|
||||
|
||||
### API
|
||||
|
||||
|
|
@ -181,210 +229,352 @@ Lardon3DVisualIndexResult lardon3d_candidate_pair_generate_batch(
|
|||
uint64_t *last_feature_set_id);
|
||||
```
|
||||
|
||||
### Algorithme
|
||||
### Ordering and bounds
|
||||
|
||||
1. Lister les FeatureSets par pages de 64
|
||||
2. Pour chaque FeatureSet :
|
||||
- Appeler `lardon3d_candidate_pair_generate()`
|
||||
- Accumuler les statistiques
|
||||
- Mettre à jour le curseur
|
||||
3. Retourner les totaux et le dernier FeatureSet traité
|
||||
The project batch path:
|
||||
|
||||
### Ordre de traitement
|
||||
- pages Feature Sets in bounded pages;
|
||||
- processes source Feature Sets in increasing `feature_set_id`;
|
||||
- never assumes IDs are contiguous;
|
||||
- keeps top-K bounded by the Visual Index contract;
|
||||
- keeps query/result memory bounded;
|
||||
- returns the last processed Feature Set for restart.
|
||||
|
||||
Feature Sets traités en ordre croissant de `feature_set_id`.
|
||||
A pair selected from multiple sources is persisted once because canonical pair identity is unique.
|
||||
|
||||
### Déduplication
|
||||
The historical single-source and project-batch APIs remain valid. They are not, by themselves, the
|
||||
complete current resource-execution description of the durable Task.
|
||||
|
||||
Une paire produite depuis plusieurs sources n'existe qu'une fois.
|
||||
La deduplication est assurée par `find avant create`.
|
||||
|
||||
### Bornes
|
||||
|
||||
- Un seul FeatureSet traité à la fois
|
||||
- Top-K borné par requête
|
||||
- Mémoire bornée : allocation `top_k * sizeof(candidate)` par requête
|
||||
- Pagination bornée (64 FeatureSets par page)
|
||||
|
||||
### Reprise
|
||||
|
||||
Le curseur `after_feature_set_id` permet la reprise après interruption.
|
||||
La fonction retourne le dernier FeatureSet traité.
|
||||
|
||||
## Tâche durable
|
||||
## Durable Task
|
||||
|
||||
### Task Kind
|
||||
|
||||
`candidate_pair.generate` v1 — **IMPLEMENTED**.
|
||||
|
||||
### Unité de travail
|
||||
|
||||
Un membership FeatureSet source du Visual Index et sa requête associée. Chaque
|
||||
séquence traite un lot borné de memberships (1 à 64 selon le contrat Governor),
|
||||
en ordre croissant d'ID sans supposer des IDs contigus.
|
||||
|
||||
### Checkpoint
|
||||
|
||||
Curseur `after_feature_set_id` persisté dans `candidate_pair_generate_tasks`.
|
||||
Checkpoint sauvé après chaque lot via
|
||||
`lardon3d_project_checkpoint_candidate_pair_generate_task()`.
|
||||
|
||||
### Reprise
|
||||
|
||||
Reprise idempotente : le champ `after_feature_set_id` est rechargé depuis la
|
||||
DB, et les paires déjà persistées sont ignorées par `find avant create`.
|
||||
À l'ouverture du projet, la tâche est automatiquement restaurée via la
|
||||
registry production et resoumise à la queue.
|
||||
|
||||
Les anciens snapshots v1 produits avec l'estimation opérationnelle exacte
|
||||
128 Kio fixes, 64 Kio par item, lot 1–64, CPU 1, IO 1 et GPU 0 sont normalisés
|
||||
éphémèrement par la registry à la forme courante CPU64 avant admission. La
|
||||
forme historique immédiatement précédente CPU12/256 Kio fixes/64 Kio par item
|
||||
est elle aussi reconnue exactement. Le
|
||||
snapshot durable original reste la source du reconstructeur ; aucun checkpoint
|
||||
d'estimation seule n'est stagé, promu ou publié sous le même résumé. Une panne
|
||||
pré-terminale répète donc cette normalisation exacte. Aucun autre snapshot,
|
||||
curseur ou paramètre scientifique n'est réinterprété.
|
||||
|
||||
### Intégration Task/Queue/Governor
|
||||
|
||||
La tâche utilise le runtime générique via le pattern standard :
|
||||
- Estimation opérationnelle (256 Kio fixes, 8 Mio par item, lot 1–64).
|
||||
La Queue conserve un callback ; jusqu'à soixante-quatre participants CPU
|
||||
admis peuvent calculer une fenêtre interne bornée sans modifier l'identité
|
||||
scientifique.
|
||||
- CPU et lot sont essayés ensemble (`1/1`, `2/2`, puis les paliers sûrs) :
|
||||
un CPU supplémentaire avec un lot d'un seul membership ne peut exercer aucun
|
||||
participant supplémentaire et ne constitue donc pas une mesure de scaling.
|
||||
- Réservation CPU + IO avant exécution
|
||||
- `lardon3d_task_sequence_break()` entre chaque lot pour réadmission Governor
|
||||
- Callback terminal checkpoint après `COMPLETED`/`FAILED`/`CANCELLED`
|
||||
- Reconstruction depuis `Lardon3DProjectDbCandidatePairGenerateTask`
|
||||
|
||||
### API
|
||||
|
||||
```c
|
||||
Lardon3DTask *lardon3d_project_create_candidate_pair_generate_task(
|
||||
Lardon3DAppState *state, uint64_t visual_index_id,
|
||||
const Lardon3DVisualIndexQueryOptions *query_options, uint64_t *task_id);
|
||||
bool lardon3d_project_enqueue_candidate_pair_generate(
|
||||
Lardon3DAppState *state, uint64_t visual_index_id,
|
||||
const Lardon3DVisualIndexQueryOptions *query_options, uint64_t *task_id);
|
||||
bool lardon3d_candidate_pair_generate_reconstruct(
|
||||
const Lardon3DTaskDurableSnapshot *snapshot, void *context,
|
||||
Lardon3DTaskKindBinding *binding);
|
||||
```text
|
||||
candidate_pair.generate/1
|
||||
```
|
||||
|
||||
## Concurrence
|
||||
Status: **IMPLEMENTED**.
|
||||
|
||||
### Garantie actuelle
|
||||
### Durable unit
|
||||
|
||||
La Queue conserve un callback actif. À l'intérieur de la Task, jusqu'à soixante-quatre
|
||||
threads CPU admis calculent en parallèle une fenêtre d'au plus deux sources par
|
||||
thread. Chaque participant possède un handle DB de lecture privé. Le thread
|
||||
propriétaire publie ensuite seul et dans l'ordre canonique des sources. Voir le
|
||||
[contrat de parallélisme interne](internal_parallelism.md).
|
||||
The durable cursor is `after_feature_set_id` in `candidate_pair_generate_tasks`.
|
||||
|
||||
### Atomicité
|
||||
The Task consumes a bounded ordered set of Visual Index source memberships. A sequence handles an
|
||||
admitted bounded batch, publishes the canonical pair decisions, persists the cursor, checkpoints and
|
||||
returns through `lardon3d_task_sequence_break()` before the next Governor admission.
|
||||
|
||||
Les workers internes ne créent aucune paire. Après leur jointure, le
|
||||
propriétaire applique seul `find avant create`. La contrainte UNIQUE reste une
|
||||
protection persistante, pas un mécanisme d'ordonnancement parallèle.
|
||||
### Checkpoint and restart
|
||||
|
||||
### Limites
|
||||
The Task checkpoints through:
|
||||
|
||||
Le pattern `find + create` n'est pas atomique entre les deux appels. Le chemin
|
||||
de Task n'introduit aucun writer concurrent ; la sémantique existante reste
|
||||
inchangée pour les autres appelants éventuels.
|
||||
|
||||
## Bornes et ressources
|
||||
|
||||
### Top-K
|
||||
|
||||
`top_k <= 256` (LARDON3D_VISUAL_INDEX_TOP_K_MAX)
|
||||
|
||||
### Mémoire
|
||||
|
||||
Allocation de requête bornée par le top-K, plus un résultat de propositions
|
||||
borné par le même maximum. La Task limite sa fenêtre à 24 sources et annonce
|
||||
256 Kio fixes plus 64 Kio par item au Governor.
|
||||
|
||||
### Complexité
|
||||
|
||||
- O(top_k) par requête (lectures Visual Index)
|
||||
- O(1) par paire (DB write)
|
||||
- Pas de structure O(N²)
|
||||
|
||||
## Fingerprint de génération
|
||||
|
||||
### Composants
|
||||
|
||||
Le fingerprint identifie une génération Candidate Pair unique :
|
||||
|
||||
```c
|
||||
void lardon3d_candidate_pair_generation_fingerprint(
|
||||
uint64_t visual_index_id, uint64_t source_feature_set_id,
|
||||
const Lardon3DVisualIndexQueryOptions *query_options,
|
||||
unsigned char fingerprint[32]);
|
||||
```text
|
||||
lardon3d_project_checkpoint_candidate_pair_generate_task()
|
||||
```
|
||||
|
||||
### Éléments inclus
|
||||
Restart:
|
||||
|
||||
- `visual_index_id` : Visual Index utilisé
|
||||
- `source_feature_set_id` : Feature Set source
|
||||
- `query_options->top_k` : nombre de candidats par requête
|
||||
- `query_options->minimum_evidence_count` : filtre minimum
|
||||
- `query_options->scanset_filter` : filtre ScanSet
|
||||
- `query_options->exclude_same_asset` : exclusion même asset
|
||||
1. restores the generic Task snapshot;
|
||||
2. loads the typed Candidate Pair Task payload;
|
||||
3. restores `after_feature_set_id`;
|
||||
4. reconstructs the production binding through the Task Kind registry;
|
||||
5. resubmits through the normal Queue/Governor path;
|
||||
6. reuses already persisted Candidate Pairs idempotently.
|
||||
|
||||
### Éléments exclus (volontairement)
|
||||
A crash may therefore repeat work after the last durable cursor, but it must not invent a second
|
||||
scientific pair identity.
|
||||
|
||||
- `created_at` : informatif, pas fonctionnel
|
||||
- `candidate_pair_id` : identifiant technique
|
||||
- Ordre des Feature Sets traités en batch
|
||||
### Historical resource descriptors
|
||||
|
||||
### Stabilité
|
||||
Older durable snapshots are accepted only through exact compatibility shapes already recognized by the
|
||||
registry.
|
||||
|
||||
Le fingerprint est stable pour mêmes entrées et configuration.
|
||||
Un changement de configuration produit un fingerprint différent.
|
||||
|
||||
### Relation avec l'invalidation
|
||||
|
||||
Un fingerprint différent signifie que le travail doit être recalculé.
|
||||
Le même fingerprint signifie que le résultat peut être réutilisé.
|
||||
|
||||
### Relation avec la reprise
|
||||
|
||||
Le fingerprint permet de vérifier qu'une reprise utilise la même
|
||||
configuration que l'originale.
|
||||
|
||||
## Limites connues
|
||||
|
||||
1. Score non persistant (par design)
|
||||
2. La compaction de segments Visual Index n'est pas implémentée
|
||||
3. Aucun Matcher consommateur des paires
|
||||
|
||||
## Statut
|
||||
|
||||
**IMPLEMENTED** — génération single-source, persistance,
|
||||
canonicalisation, idempotence, batch projet, fingerprint et
|
||||
réutilisation/invalidation.
|
||||
|
||||
**IMPLEMENTED** — tâche durable `candidate_pair.generate` v1
|
||||
via le runtime/Queue générique, avec estimation immuable,
|
||||
checkpoint par curseur, reprise idempotente et intégration
|
||||
dans la registry production.
|
||||
|
||||
## Relation avec le pipeline
|
||||
|
||||
Le Candidate Pair Generator est l'étape E du pipeline de reconstruction :
|
||||
Historical forms include the exact earlier descriptors documented by the implementation, including:
|
||||
|
||||
```text
|
||||
128 KiB fixed
|
||||
64 KiB per item
|
||||
batch 1..64
|
||||
CPU1
|
||||
IO1
|
||||
GPU0
|
||||
```
|
||||
Feature Store (C)
|
||||
↓
|
||||
Visual Index (D)
|
||||
↓
|
||||
Candidate Pair Generator (E) ← CE DOCUMENT
|
||||
↓
|
||||
Matching (F) — HORS SCOPE
|
||||
|
||||
and the later exact historical CPU12 / 256 KiB fixed / 64 KiB-per-item form.
|
||||
|
||||
Those shapes are restart compatibility evidence. They are not the current resource model and must not
|
||||
be copied into new Task creation.
|
||||
|
||||
The original durable snapshot remains the source supplied to reconstruction. Compatibility
|
||||
normalization is ephemeral and does not rewrite the persisted checkpoint or Candidate scientific
|
||||
identity.
|
||||
|
||||
## Current resource contract
|
||||
|
||||
### Current Task estimate
|
||||
|
||||
The current validated Candidate Pair Task declares approximately:
|
||||
|
||||
```text
|
||||
fixed RAM 256 KiB
|
||||
per-item RAM 8 MiB
|
||||
batch range 1..64
|
||||
GPU demand 0
|
||||
IO demand bounded by the existing Task estimate
|
||||
CPU demand reducible and bounded by the host compute pool
|
||||
```
|
||||
|
||||
The exact implementation constants remain authoritative in source. This documentation records the
|
||||
current validated capability and intentionally does not retain the obsolete 24-source / 64-KiB-per-item
|
||||
description as current policy.
|
||||
|
||||
### Coupled CPU and batch admission
|
||||
|
||||
Candidate Pair generation has independent source work, but additional CPU cannot exercise additional
|
||||
participants if the admitted source batch remains one.
|
||||
|
||||
For this Task, CPU and batch scaling are therefore coupled during adaptation. Conceptually:
|
||||
|
||||
```text
|
||||
CPU1 / batch1
|
||||
CPU2 / batch2
|
||||
then larger safe coupled rungs
|
||||
```
|
||||
|
||||
subject to:
|
||||
|
||||
- the Task's declared maximums;
|
||||
- the host compute pool;
|
||||
- current Governor pressure;
|
||||
- measured usefulness;
|
||||
- current admission policy.
|
||||
|
||||
This coupling fixes an operational scaling defect. It does not modify Candidate Pair scientific
|
||||
identity, query options, top-K behavior, publication order or persistence.
|
||||
|
||||
### Canonical resource principles
|
||||
|
||||
```text
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
```
|
||||
|
||||
The interactive host reserve is preserved first. After that reserve and all safety constraints are
|
||||
satisfied, safe and useful compute capacity should not be left idle merely to preserve an old
|
||||
single-thread measurement.
|
||||
|
||||
Reference-host values are observations, not portable constants.
|
||||
|
||||
### Atomicity is not serialism
|
||||
|
||||
One source query and one Candidate Pair publication decision remain bounded scientific/transactional
|
||||
units.
|
||||
|
||||
That does not imply that independent source preparation must run serially.
|
||||
|
||||
```text
|
||||
PER_ITEM_ATOMICITY_REQUIRES_CROSS_ITEM_SERIALISM=NO
|
||||
OWNER_ONLY_PUBLICATION_REQUIRES_SERIAL_PREPARATION=NO
|
||||
```
|
||||
|
||||
## Internal concurrency
|
||||
|
||||
The Task Queue still owns one active callback. Candidate Pair uses bounded parallelism inside that
|
||||
callback.
|
||||
|
||||
The validated shape is:
|
||||
|
||||
```text
|
||||
one admitted Candidate owner Task
|
||||
-> bounded source window
|
||||
-> bounded CPU participants
|
||||
-> private read-side preparation
|
||||
-> join
|
||||
-> owner publishes in canonical source order
|
||||
```
|
||||
|
||||
Participants do not create Candidate Pair rows.
|
||||
|
||||
Each participant uses its allowed private read-side state. After participants join, the owner alone
|
||||
performs the canonical `find` / create publication sequence.
|
||||
|
||||
The SQL `UNIQUE(image_id_a, image_id_b)` constraint remains a persistent integrity guard, not a
|
||||
parallel scheduling primitive.
|
||||
|
||||
No second global scheduler, global worker pool or parallel SQLite writer subsystem is introduced.
|
||||
|
||||
## Persistence race boundary
|
||||
|
||||
The public pattern:
|
||||
|
||||
```text
|
||||
find candidate pair
|
||||
-> create candidate pair when absent
|
||||
```
|
||||
|
||||
is not a general atomic compare-and-insert primitive across arbitrary concurrent writers.
|
||||
|
||||
The production durable Candidate Task avoids introducing competing pair writers: participant work is
|
||||
read/preparation only and owner publication is serialized.
|
||||
|
||||
Other callers must not infer a stronger concurrency guarantee from the Task's owner-only publication
|
||||
model.
|
||||
|
||||
## Resource complexity
|
||||
|
||||
### Query bound
|
||||
|
||||
```text
|
||||
top_k <= LARDON3D_VISUAL_INDEX_TOP_K_MAX = 256
|
||||
```
|
||||
|
||||
### Memory
|
||||
|
||||
Current Task admission uses the current estimate:
|
||||
|
||||
```text
|
||||
256 KiB fixed + 8 MiB per admitted item
|
||||
```
|
||||
|
||||
with batch bounded to `1..64`.
|
||||
|
||||
Actual admission can be reduced by the Governor and host compute pool. Swap, zram and external scratch
|
||||
never enlarge admitted RAM.
|
||||
|
||||
Candidate Pair currently has no authoritative scratch consumer.
|
||||
|
||||
### Algorithmic shape
|
||||
|
||||
The subsystem does not allocate an O(N^2) project pair matrix.
|
||||
|
||||
Per source, bounded work is dominated by:
|
||||
|
||||
- bounded Visual Index query;
|
||||
- bounded top-K filtering;
|
||||
- canonical pair lookup/publication.
|
||||
|
||||
Project traversal is paged and restartable.
|
||||
|
||||
## GPU policy
|
||||
|
||||
Candidate Pair currently remains CPU.
|
||||
|
||||
The validated GPU audit classified it as:
|
||||
|
||||
```text
|
||||
CANDIDATE_GPU=REJECTED_WITH_MEASURED_REASON
|
||||
```
|
||||
|
||||
The workload is dominated by Visual Index access, filtering, branching and deterministic ordered SQLite
|
||||
publication, and no validated GPU primitive currently preserves the complete Candidate contract with a
|
||||
useful measured advantage.
|
||||
|
||||
This rejection does not authorize CPU serialism. Safe useful CPU parallelism remains required by the
|
||||
canonical resource policy.
|
||||
|
||||
## Real A6000 evidence
|
||||
|
||||
The retained real A6000 pre-SfM execution exercised the current Candidate path before Matcher, GV and
|
||||
Tracks.
|
||||
|
||||
It produced:
|
||||
|
||||
```text
|
||||
Candidate Pairs = 38,420
|
||||
Match Results = 38,420
|
||||
```
|
||||
|
||||
The later checkpoint is:
|
||||
|
||||
```text
|
||||
real-a6000-pre-sfm-2026-09-02
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
Candidate replay at the retained checkpoint created no new Candidate work. The proof continued through
|
||||
Matcher, Geometric Verifier v3 and Tracks, then stopped before real Sparse SfM.
|
||||
|
||||
This evidence validates the operational Candidate path on that project. It does not change Candidate
|
||||
Pair v1 identity or the Project DB v8/v9 historical contracts.
|
||||
|
||||
## Relationship to current Project DB
|
||||
|
||||
The Candidate Pair model was introduced in Project DB v8 and its durable Task payload in v9.
|
||||
|
||||
The current schema head is v25:
|
||||
|
||||
```text
|
||||
v22 selected scientific execution foundation
|
||||
v23 generic optical-context overlay
|
||||
v24 raw.develop.batch/1 persistence
|
||||
v25 features.extract.batch/1 persistence
|
||||
```
|
||||
|
||||
Those later additive migrations do not reinterpret Candidate Pair rows.
|
||||
|
||||
## Known limits
|
||||
|
||||
Current limits and non-goals include:
|
||||
|
||||
- retrieval score is not persisted in `candidate_pairs` by design;
|
||||
- Visual Index segment compaction remains separate work;
|
||||
- no generic DAG dependency scheduler is introduced by Candidate Pair;
|
||||
- no Candidate GPU backend is currently validated;
|
||||
- no scratch/spill path is authoritative for Candidate Pair;
|
||||
- arbitrary concurrent pair writers are not provided by the Task owner-publication model.
|
||||
|
||||
The downstream Matcher is implemented; it is not a missing Candidate Pair feature.
|
||||
|
||||
## Pipeline relationship
|
||||
|
||||
```text
|
||||
Feature Store
|
||||
|
|
||||
v
|
||||
Visual Index
|
||||
|
|
||||
v
|
||||
Candidate Pair Generator <- this document
|
||||
|
|
||||
v
|
||||
Matcher
|
||||
|
|
||||
v
|
||||
Geometric Verification
|
||||
|
|
||||
v
|
||||
Tracks
|
||||
```
|
||||
|
||||
Candidate Pair selects plausible image pairs. Matcher computes descriptor-level correspondence evidence.
|
||||
Geometric Verification validates geometry. Track Builder creates multi-view observation tracks. These
|
||||
scientific responsibilities remain separate.
|
||||
|
||||
## Summary
|
||||
|
||||
```text
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
|
||||
CANDIDATE_PAIR_MODEL=v1
|
||||
CANDIDATE_PAIR_MODEL_STATUS=IMPLEMENTED
|
||||
|
||||
CANDIDATE_PAIR_TASK=candidate_pair.generate/1
|
||||
CANDIDATE_PAIR_TASK_STATUS=IMPLEMENTED
|
||||
|
||||
CANDIDATE_PAIR_PERSISTENCE_VERSION=v8
|
||||
CANDIDATE_PAIR_TASK_PERSISTENCE_VERSION=v9
|
||||
|
||||
CANDIDATE_CURRENT_FIXED_RAM=256_KiB
|
||||
CANDIDATE_CURRENT_PER_ITEM_RAM=8_MiB
|
||||
CANDIDATE_CURRENT_BATCH_RANGE=1..64
|
||||
CANDIDATE_CPU_BATCH_ADAPTATION=COUPLED
|
||||
CANDIDATE_GPU=REJECTED_WITH_MEASURED_REASON
|
||||
CANDIDATE_SCRATCH_CONSUMER=NO
|
||||
|
||||
PER_ITEM_ATOMICITY_REQUIRES_CROSS_ITEM_SERIALISM=NO
|
||||
OWNER_ONLY_PUBLICATION_REQUIRES_SERIAL_PREPARATION=NO
|
||||
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
|
|
|||
|
|
@ -1,61 +1,170 @@
|
|||
# Feature Store v1/v2
|
||||
|
||||
## Rôle et modèle
|
||||
## Status
|
||||
|
||||
**IMPLEMENTED.** Le Feature Store est la mémoire visuelle locale persistante.
|
||||
Un `FeatureSet` logique appartient à une `image_id` et identifie exactement
|
||||
`orb` version 1 plus une configuration canonique. Il référence un
|
||||
`FeatureAsset` physique immutable. Deux images logiques portant le même contenu
|
||||
et la même configuration gardent deux `feature_set_id`, mais partagent le même
|
||||
asset lorsque leurs octets sont identiques. Les futurs matches référenceront
|
||||
`feature_set_id + feature_index`; ces deux valeurs sont immuables après
|
||||
publication.
|
||||
```text
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
|
||||
`feature_set_id` et `feature_asset_id` sont des identités SQLite
|
||||
`AUTOINCREMENT`. Une identité validée n'est jamais réattribuée. SQLite ne
|
||||
contient que les relations, empreintes, tailles et paramètres ; les tableaux de
|
||||
points et descripteurs restent hors DB.
|
||||
FEATURE_STORE_STATUS=IMPLEMENTED
|
||||
FEATURE_FILE_V1=FROZEN
|
||||
FEATURE_FILE_V2=IMPLEMENTED
|
||||
|
||||
## Extracteur et configuration
|
||||
HISTORICAL_FEATURE_TASK=features.extract/1
|
||||
CURRENT_FEATURE_BATCH_TASK=features.extract.batch/1
|
||||
PROJECT_DB_V25_FEATURE_BATCH=IMPLEMENTED/VALIDATED
|
||||
|
||||
L'extracteur production est ORB d'OpenCV 5, derrière une façade C. Aucune
|
||||
exception C++ ni `cv::Mat` ne traverse cette frontière. La registry statique
|
||||
expose `features.extract`, version 1. Une tâche traite exactement une image.
|
||||
PER_IMAGE_FEATURE_RESULT=ATOMIC
|
||||
PER_IMAGE_ATOMICITY_REQUIRES_CROSS_IMAGE_SERIALISM=NO
|
||||
OWNER_ONLY_PUBLICATION_REQUIRES_SERIAL_PREPARATION=NO
|
||||
|
||||
La configuration v1 contient trois `uint32` : `max_features` (1..8192),
|
||||
`pyramid_levels` (1..16) et `fast_threshold` (1..255). Son fingerprint SHA-256
|
||||
porte sur les 24 octets canoniques `L3DORBP1`, version et trois entiers
|
||||
little-endian. Il ne dépend ni du padding, ni de la locale. Le contrat ORB v1
|
||||
produit 32 octets binaires par point.
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
|
||||
Au démarrage, avant la création du worker Queue, Lardon3D configure la baseline
|
||||
OpenCV process-wide depuis le compute-pool réel. `features.extract` publie la
|
||||
borne positive `int` acceptée par OpenCV ; le Resource Governor réduit toujours
|
||||
l'admission au budget hôte. Douze threads demeure une cohorte de validation,
|
||||
pas un plafond portable. La Queue conserve un callback actif ; aucun pool
|
||||
Lardon3D supplémentaire n'est créé pour ORB.
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
Cette limite est une configuration opérationnelle, jamais un paramètre
|
||||
scientifique : elle n'entre ni dans le fingerprint ORB v1, ni dans l'identité
|
||||
FeatureSet, ni dans le Feature File. L'audit contrôlé OpenCV 5.0.0 aux limites
|
||||
1, 2, 4, 8 et 12 exige le même count, le même ordre et les mêmes valeurs
|
||||
binary32 des six champs keypoint persistés, les mêmes lignes/octets descriptor
|
||||
et le même SHA-256 du Feature File. `class_id` est une donnée interne
|
||||
`cv::KeyPoint` non publiée ; elle ne fait pas partie du record Feature File v1/v2
|
||||
gelé et ne peut donc définir une identité scientifique.
|
||||
The Feature Store is the persistent local visual-memory layer for Lardon3D.
|
||||
|
||||
La limite de 100 000 000 pixels est vérifiée après `cv::imread` : l'API utilisée
|
||||
ne fournit pas de sonde de dimensions multi-format fiable sans décodage. Le pic
|
||||
mémoire du décodage peut donc précéder le rejet. Lardon3D ne revendique pas de
|
||||
bornage pré-décodage et n'embarque pas un parseur JPEG/PNG parallèle.
|
||||
The scientific identity of a Feature Set remains per image and immutable after publication. Project DB
|
||||
v25 adds an operational selected-execution batch Task, but does not change Feature Set identity,
|
||||
Feature File formats, ORB descriptor semantics or the historical single-image Task contract.
|
||||
|
||||
## Role and identity model
|
||||
|
||||
A logical `FeatureSet` belongs to one `image_id` and identifies:
|
||||
|
||||
- one extractor kind;
|
||||
- one extractor version;
|
||||
- one canonical parameter fingerprint;
|
||||
- one exact source image content identity;
|
||||
- one immutable published Feature Asset.
|
||||
|
||||
Two logical images with identical source content and identical extractor configuration keep distinct
|
||||
`feature_set_id` values. They may share the same immutable physical Feature Asset when the resulting
|
||||
Feature File bytes are identical.
|
||||
|
||||
Downstream correspondence identity references:
|
||||
|
||||
```text
|
||||
feature_set_id + feature_index
|
||||
```
|
||||
|
||||
Both components are immutable after Feature Set publication.
|
||||
|
||||
`feature_set_id` and `feature_asset_id` are SQLite `AUTOINCREMENT` identities. A committed published
|
||||
identity is never reassigned to another object. SQLite stores logical relations, hashes, sizes,
|
||||
extractor metadata and light metrics; keypoint arrays and descriptor arrays remain outside SQLite.
|
||||
|
||||
## Extractor registry and scientific configuration
|
||||
|
||||
The historical production ORB extractor uses OpenCV 5 behind a C boundary. No C++ exception or
|
||||
`cv::Mat` crosses the public C API.
|
||||
|
||||
The historical registered Task Kind is:
|
||||
|
||||
```text
|
||||
features.extract/1
|
||||
```
|
||||
|
||||
That Task processes one image as one scientific unit.
|
||||
|
||||
The current selected-execution operational Task Kind is:
|
||||
|
||||
```text
|
||||
features.extract.batch/1
|
||||
```
|
||||
|
||||
It processes multiple independently selected images under one durable owner while preserving one
|
||||
immutable Feature result per image.
|
||||
|
||||
### ORB v1 configuration
|
||||
|
||||
The ORB v1 configuration contains three `uint32_t` fields:
|
||||
|
||||
- `max_features`, range `1..8192`;
|
||||
- `pyramid_levels`, range `1..16`;
|
||||
- `fast_threshold`, range `1..255`.
|
||||
|
||||
Its SHA-256 parameter fingerprint is computed from the canonical 24-byte domain containing:
|
||||
|
||||
```text
|
||||
L3DORBP1
|
||||
version
|
||||
max_features
|
||||
pyramid_levels
|
||||
fast_threshold
|
||||
```
|
||||
|
||||
with fixed little-endian integer encoding.
|
||||
|
||||
The fingerprint does not depend on C struct padding, locale, Task ID, resource reservation,
|
||||
admitted CPU count or admitted batch size.
|
||||
|
||||
ORB v1 produces 32-byte binary descriptors.
|
||||
|
||||
## OpenCV execution control
|
||||
|
||||
OpenCV thread count is process-wide operational state, not scientific identity.
|
||||
|
||||
Before Queue execution, Lardon3D establishes the validated OpenCV baseline from the actual host
|
||||
compute pool. The active heavy callback applies the count required by its admitted contract and
|
||||
restores the baseline on every exit path.
|
||||
|
||||
The Queue retains one active callback. This prevents unrelated heavy Tasks from racing the same
|
||||
process-wide OpenCV configuration.
|
||||
|
||||
Historical `features.extract/1` may use the admitted OpenCV CPU count inside one image extraction.
|
||||
|
||||
For `features.extract.batch/1`, cross-image participants are the primary concurrency mechanism.
|
||||
Independent participants prepare different images; Lardon3D does not blindly multiply a full OpenCV
|
||||
thread team inside every participant.
|
||||
|
||||
The admitted CPU count is resource policy and never enters:
|
||||
|
||||
- ORB v1 fingerprint;
|
||||
- Feature Set identity;
|
||||
- Feature File bytes;
|
||||
- source-image identity.
|
||||
|
||||
The controlled OpenCV 5.0.0 validation at 1, 2, 4, 8 and 12 threads required identical:
|
||||
|
||||
- feature count;
|
||||
- keypoint order;
|
||||
- binary32 values of all six persisted keypoint fields;
|
||||
- descriptor row order and bytes;
|
||||
- complete Feature File SHA-256.
|
||||
|
||||
Twelve threads are reference-host validation evidence, not a portable product ceiling.
|
||||
|
||||
`cv::KeyPoint::class_id` is not persisted by the frozen Feature File record and therefore cannot
|
||||
define scientific identity.
|
||||
|
||||
## Decode bound
|
||||
|
||||
The 100,000,000-pixel limit is checked after `cv::imread`.
|
||||
|
||||
The API used by this implementation does not provide a reliable multi-format dimension probe without
|
||||
decoding. Therefore decode peak memory can precede rejection.
|
||||
|
||||
Lardon3D does not claim pre-decode memory bounding and does not maintain a parallel JPEG/PNG dimension
|
||||
parser for this purpose.
|
||||
|
||||
## Feature File v1
|
||||
|
||||
Le fichier est little-endian et exige IEEE-754 binary32. Sa taille maximale est
|
||||
16 Mio et son nombre maximal de points 8192.
|
||||
Feature File v1 is little-endian and requires IEEE-754 binary32.
|
||||
|
||||
| Offset | Taille | Champ |
|
||||
|---:|---:|---|
|
||||
Limits:
|
||||
|
||||
```text
|
||||
maximum file size 16 MiB
|
||||
maximum feature count 8192
|
||||
ORB descriptor U8 x 32
|
||||
```
|
||||
|
||||
### Header
|
||||
|
||||
| Offset | Size | Field |
|
||||
| ---: | ---: | --- |
|
||||
| 0 | 8 | magic `L3DFEAT\0` |
|
||||
| 8 | 4 | format version = 1 |
|
||||
| 12 | 4 | header size = 160 |
|
||||
|
|
@ -63,98 +172,456 @@ Le fichier est little-endian et exige IEEE-754 binary32. Sa taille maximale est
|
|||
| 20 | 4 | descriptor dimension = 32 |
|
||||
| 24 | 4 | descriptor type = U8 |
|
||||
| 28 | 4 | keypoint record size = 24 |
|
||||
| 32,36 | 4+4 | largeur, hauteur décodées |
|
||||
| 40,48,56 | 8+8+8 | offsets keypoints/descriptors, taille totale |
|
||||
| 64 | 32 | SHA-256 de l'asset image source |
|
||||
| 96 | 32 | fingerprint paramètres |
|
||||
| 128 | 16 | `orb\0` puis zéros réservés |
|
||||
| 32, 36 | 4 + 4 | decoded image width and height |
|
||||
| 40, 48, 56 | 8 + 8 + 8 | keypoint offset, descriptor offset, total size |
|
||||
| 64 | 32 | source image asset SHA-256 |
|
||||
| 96 | 32 | parameter fingerprint |
|
||||
| 128 | 16 | `orb\0` followed by reserved zero bytes |
|
||||
| 144 | 4 | extractor version = 1 |
|
||||
| 148 | 12 | réservés, zéro obligatoire |
|
||||
| 148 | 12 | reserved, must be zero |
|
||||
|
||||
Chaque keypoint contient six mots de 32 bits : `x`, `y`, `size`, orientation,
|
||||
response en binary32, puis octave signé. `x/y` sont en pixels de l'image telle
|
||||
que décodée par OpenCV, origine en haut à gauche. `size` est le diamètre du
|
||||
voisinage en pixels. L'orientation est en degrés dans `[0,360)`. L'entrée `i`
|
||||
du bloc keypoints correspond exactement aux 32 octets du descriptor `i`.
|
||||
Each keypoint contains six 32-bit words:
|
||||
|
||||
Le validateur contrôle magic, version, réservés, type, dimension, bornes,
|
||||
offsets, taille exacte, multiplications, SHA-256 externe et cohérence DB. Une
|
||||
version future est distinguée d'une corruption lorsque le fichier et son record
|
||||
DB sont cohérents. Le reader utilise `pread`, accepte au plus 256 éléments par
|
||||
appel et ne charge jamais le fichier entier.
|
||||
1. `x` binary32;
|
||||
2. `y` binary32;
|
||||
3. `size` binary32;
|
||||
4. orientation binary32;
|
||||
5. response binary32;
|
||||
6. signed octave.
|
||||
|
||||
Le chemin DB est validé sous sa forme canonique exacte dérivée du SHA-256 avant
|
||||
toute ouverture. Une absence retourne `NOT_FOUND`; troncature, hash divergent,
|
||||
header ou métadonnées DB divergents retournent `CORRUPT`. Une version future
|
||||
retourne distinctement `UNSUPPORTED_VERSION`.
|
||||
`x` and `y` are image pixels in the OpenCV-decoded image with top-left origin.
|
||||
|
||||
## Publication et persistance
|
||||
`size` is the neighborhood diameter in pixels.
|
||||
|
||||
Layout : `assets/features/<2 hex>/<sha256 complet lowercase>`. Le SHA-256 porte
|
||||
sur le Feature File complet. Le protocole est : temporaire local, écriture,
|
||||
`fsync`, hash, `link` atomique sans écrasement, validation complète lors d'une
|
||||
adoption concurrente, `fsync` du répertoire, puis transaction SQLite. Un échec
|
||||
du dernier `fsync` est enregistré `PUBLISHED_NOT_DURABLE`. Un échec SQLite après
|
||||
publication laisse un fichier orphelin et aucune ligne logique partielle.
|
||||
Un nouvel essai qui revalide l'asset et réussit le `fsync` promeut explicitement
|
||||
sa durabilité vers `DURABLE` dans la transaction DB.
|
||||
Orientation is in degrees in `[0,360)`.
|
||||
|
||||
Le sous-schéma introduit en v5 sépare `feature_assets`, `feature_sets` et
|
||||
`feature_extract_tasks`. L'unicité logique porte sur image, kind, version et
|
||||
fingerprint. Une tâche est persistée avant enqueue, passe par la queue et le
|
||||
Governor (CPU, IO, 576 Mio conservateurs, lot 1), puis checkpointée initialement
|
||||
et terminalement. La reprise recommence l'image entière : il n'existe pas de
|
||||
fausse reprise intra-ORB. La publication est idempotente.
|
||||
Keypoint record `i` corresponds exactly to descriptor row `i`.
|
||||
|
||||
Pause et annulation sont coopératives avant/après l'appel ORB ; cet appel n'est
|
||||
pas interruptible. L'image gérée est rehashée avant extraction. Une image
|
||||
uniforme produit légitimement un Feature Set READY vide.
|
||||
## Feature File v1 validation
|
||||
|
||||
Lardon3D ne garantit pas des octets ORB identiques entre versions d'OpenCV,
|
||||
plateformes ou backends. L'idempotence porte sur l'environnement courant et le
|
||||
contrat `extractor_version`; une évolution qui change durablement la sémantique
|
||||
ou les octets exige d'auditer et, si nécessaire, d'incrémenter cette version.
|
||||
The reader validates:
|
||||
|
||||
## Statut
|
||||
- magic;
|
||||
- format version;
|
||||
- reserved-zero bytes;
|
||||
- descriptor type;
|
||||
- descriptor dimension;
|
||||
- count bounds;
|
||||
- offset bounds;
|
||||
- exact total size;
|
||||
- checked multiplication;
|
||||
- external SHA-256;
|
||||
- Project DB metadata consistency.
|
||||
|
||||
**IMPLEMENTED** — ORB réel, formats v1/v2, assets content-addressed, DB v7,
|
||||
publication atomique, reader borné, task kind production et reprise automatique.
|
||||
A future format version is distinguished from corruption when both file and DB record are otherwise
|
||||
coherent.
|
||||
|
||||
**NOT_YET_WIRED** — commande de réconciliation/scrub des orphelins, contrôle fin
|
||||
du backend parallèle OpenCV, orientation EXIF, lancement automatique de
|
||||
l'extraction après import et planification multi-image/DAG.
|
||||
The reader uses `pread`, reads at most 256 features per call and does not require loading the entire
|
||||
Feature File.
|
||||
|
||||
**IMPLEMENTED** — Visual Index v1 consomme ce reader sans changer le format.
|
||||
The Project DB path must equal the canonical path derived from the stored SHA-256 before the file is
|
||||
opened.
|
||||
|
||||
**IMPLEMENTED** — paires candidates, Matcher v1 et Match Store v1 consomment
|
||||
les Feature Sets persistés.
|
||||
Outcomes include:
|
||||
|
||||
**IMPLEMENTED** — vérification géométrique et tracks. **Sparse SfM Gate A
|
||||
PASS**, the Sparse SfM v16 persistence model is **FROZEN** after Gate B;
|
||||
Gate C geometry and the synchronous in-memory Gate D incremental core are
|
||||
**IMPLEMENTED / PASS**. Final per-component Gate E BA is **PASS / FROZEN**;
|
||||
Gate F project/task orchestration is **PASS / FROZEN**. Gate G is
|
||||
**PASS / FROZEN**.
|
||||
- missing file -> `NOT_FOUND`;
|
||||
- truncation -> `CORRUPT`;
|
||||
- hash mismatch -> `CORRUPT`;
|
||||
- header/DB disagreement -> `CORRUPT`;
|
||||
- coherent unsupported future version -> `UNSUPPORTED_VERSION`.
|
||||
|
||||
## Extension v2 multi-descriptor
|
||||
## Publication and persistence
|
||||
|
||||
**IMPLEMENTED v1A.** ORB continue d'écrire exactement le v1 U8×32 historique.
|
||||
SIFT et RootSIFT écrivent le v2 F32×128. Le header v2 de 176 octets contient,
|
||||
aux offsets 0..64, magic, version, taille header, count, dimension, type, taille
|
||||
scalaire, taille record, dimensions image, capabilities, offsets des blocs et
|
||||
taille totale ; puis SHA source à 72, fingerprint à 104, kind sur 16 octets à
|
||||
136, version extracteur à 152 et vingt octets réservés nuls.
|
||||
Physical layout:
|
||||
|
||||
Les valeurs durables sont `U8=1` et `F32=2`, de tailles 1 et 4. Chaque binary32
|
||||
est encodé little-endian sans dump de struct. Writer et reader rejettent NaN et
|
||||
Inf. Les lectures `descriptors_u8` et `descriptors_f32` refusent un type
|
||||
incompatible, utilisent `pread` et restent limitées à 256 features. La limite
|
||||
v1 reste 16 Mio ; la limite v2 est 64 Mio.
|
||||
```text
|
||||
assets/features/<first-2-hex>/<full-lowercase-sha256>
|
||||
```
|
||||
|
||||
La façade d'extraction transporte aussi `descriptor_bytes`. La publication
|
||||
exige exactement `count × dimension × scalar_size`; elle refuse donc un buffer
|
||||
tronqué ou surdimensionné avant toute lecture ou création de fichier.
|
||||
The SHA-256 covers the complete Feature File.
|
||||
|
||||
ProjectDb v7 ajoute les métriques légères de couverture. Le détail multipasse,
|
||||
la grille et RootSIFT sont canoniques dans
|
||||
`precision_feature_pipeline.md`.
|
||||
The publication protocol is:
|
||||
|
||||
```text
|
||||
local temporary file
|
||||
-> write
|
||||
-> fsync file
|
||||
-> hash complete Feature File
|
||||
-> atomic no-overwrite link/publication
|
||||
-> fully validate any concurrently existing identical asset
|
||||
-> fsync parent directory
|
||||
-> SQLite transaction
|
||||
-> logical READY state
|
||||
```
|
||||
|
||||
If the final parent-directory `fsync` fails after publication, durability is recorded as
|
||||
`PUBLISHED_NOT_DURABLE`.
|
||||
|
||||
If SQLite fails after the physical file is published, the valid file remains an orphan and no partial
|
||||
logical Feature Set row is invented.
|
||||
|
||||
A later exact retry may revalidate the physical asset, complete a successful directory `fsync` and
|
||||
promote durability to `DURABLE` in the Project DB transaction.
|
||||
|
||||
## Historical Project DB Feature foundation
|
||||
|
||||
The Feature Store persistence foundation separates:
|
||||
|
||||
- `feature_assets`;
|
||||
- `feature_sets`;
|
||||
- `feature_extract_tasks`.
|
||||
|
||||
Logical uniqueness is based on image, extractor kind, extractor version and parameter fingerprint.
|
||||
|
||||
The historical `features.extract/1` Task is persisted before enqueue, admitted by Queue/Governor,
|
||||
executes one complete image and checkpoints at its durable boundaries.
|
||||
|
||||
Its historical conservative execution shape includes:
|
||||
|
||||
```text
|
||||
one image
|
||||
one atomic ORB extraction/publication
|
||||
no intra-image restart
|
||||
```
|
||||
|
||||
The existing conservative single-image resource descriptor remains valid for those Tasks and their
|
||||
restart compatibility. It must not be generalized into a rule requiring all future independent images
|
||||
to execute serially.
|
||||
|
||||
Pause and cancellation are cooperative before and after the OpenCV call. The OpenCV extraction itself
|
||||
is not interruptible.
|
||||
|
||||
The managed source image is rehashed before extraction.
|
||||
|
||||
A uniform image may legitimately publish a READY Feature Set with zero features.
|
||||
|
||||
## Selected-execution Feature batch — Project DB v25
|
||||
|
||||
Project DB v25 adds only:
|
||||
|
||||
```text
|
||||
feature_extract_batch_tasks
|
||||
```
|
||||
|
||||
and the durable Task Kind:
|
||||
|
||||
```text
|
||||
features.extract.batch/1
|
||||
```
|
||||
|
||||
The v24 -> v25 migration is transactional, additive and DDL-only.
|
||||
|
||||
It does not:
|
||||
|
||||
- create Feature Sets;
|
||||
- convert historical `features.extract/1` Tasks;
|
||||
- infer an `image_id`;
|
||||
- infer a Feature Set;
|
||||
- change ORB v1 parameters;
|
||||
- change Feature File v1/v2;
|
||||
- change Feature Set identity.
|
||||
|
||||
### Durable owner identity
|
||||
|
||||
One `features.extract.batch/1` owner is bound to:
|
||||
|
||||
- one immutable selected execution;
|
||||
- one monotonic `next_item_index`;
|
||||
- the exact ORB extractor kind;
|
||||
- the exact ORB extractor version;
|
||||
- the exact ORB parameters;
|
||||
- the exact ORB parameter fingerprint.
|
||||
|
||||
The selected execution order is authoritative.
|
||||
|
||||
### Execution shape
|
||||
|
||||
The current operational path is:
|
||||
|
||||
```text
|
||||
one admitted owner Task
|
||||
-> bounded selected-item window
|
||||
-> bounded independent image participants
|
||||
-> each participant prepares one per-image Feature result
|
||||
-> participants perform no SQLite publication
|
||||
-> join
|
||||
-> owner validates or reuses the exact READY Feature Set
|
||||
-> owner publishes in selected-item order
|
||||
-> owner advances the durable Feature cursor
|
||||
-> generic Task progress/checkpoint follows
|
||||
```
|
||||
|
||||
Each participant therefore computes an ordinary per-image scientific Feature result.
|
||||
|
||||
The owner-only publication stage preserves deterministic selected-item order without requiring
|
||||
independent image preparation to run serially.
|
||||
|
||||
### Atomicity and concurrency
|
||||
|
||||
The scientific atomic result remains one Feature Set for one image.
|
||||
|
||||
```text
|
||||
PER_IMAGE_FEATURE_RESULT=ATOMIC
|
||||
PER_IMAGE_ATOMICITY_REQUIRES_CROSS_IMAGE_SERIALISM=NO
|
||||
OWNER_ONLY_PUBLICATION_REQUIRES_SERIAL_PREPARATION=NO
|
||||
```
|
||||
|
||||
Cross-image preparation may be concurrent when the Governor admits a useful bounded window.
|
||||
|
||||
This is not a generic DAG scheduler and does not add a second global worker pool. The Task Queue still
|
||||
owns one active callback; the batch callback contains the bounded participant work.
|
||||
|
||||
### Crash and restart
|
||||
|
||||
An immutable Feature Set can become durable before the Feature-batch cursor or generic Task checkpoint
|
||||
advances.
|
||||
|
||||
Therefore, after a crash:
|
||||
|
||||
```text
|
||||
durable Feature Set may be ahead of Task checkpoint
|
||||
```
|
||||
|
||||
but:
|
||||
|
||||
```text
|
||||
Task checkpoint must never invent a Feature Set that is not durable
|
||||
```
|
||||
|
||||
Restart revalidates and reuses the exact READY Feature Set and then converges the cursor. It does not
|
||||
derive identity from path, processing order or Task-local position.
|
||||
|
||||
### Resource adaptation
|
||||
|
||||
For the selected-execution Feature batch, increasing admitted CPU while the item window remains one
|
||||
cannot exercise additional independent images.
|
||||
|
||||
The current validated resource model may therefore couple CPU and batch growth where required by the
|
||||
Feature-batch rung contract.
|
||||
|
||||
That behavior is Task-specific operational policy, not a universal Governor rule.
|
||||
|
||||
The canonical repository policy remains:
|
||||
|
||||
```text
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
```
|
||||
|
||||
Preserve the interactive host reserve and all RAM/CPU/IO/GPU safety constraints first. Within that
|
||||
safe envelope, use the maximum useful validated execution width.
|
||||
|
||||
Per-item atomicity is not evidence that unrelated items must be serialized.
|
||||
|
||||
## GPU boundary
|
||||
|
||||
Feature Extraction currently remains CPU.
|
||||
|
||||
The current production GPU audit classifies Feature Extraction as:
|
||||
|
||||
```text
|
||||
FEATURE_GPU=REJECTED_WITH_MEASURED_OR_IMPLEMENTATION_EVIDENCE
|
||||
```
|
||||
|
||||
On the validated OpenCV 5.0.0 host:
|
||||
|
||||
- no usable ORB/SIFT Vulkan/OpenCL extraction seam is validated;
|
||||
- CUDA is unavailable in the installed OpenCV build;
|
||||
- no GPU path has proven byte-identical Feature Files for this boundary.
|
||||
|
||||
Therefore Feature Extraction receives no production GPU path merely because the host has a GPU.
|
||||
|
||||
This does not authorize avoidable CPU serialism. Cross-image CPU parallelism remains valid where its
|
||||
Task contract has been proven.
|
||||
|
||||
## Environment and reproducibility
|
||||
|
||||
Lardon3D does not claim ORB Feature File bytes are identical across arbitrary:
|
||||
|
||||
- OpenCV versions;
|
||||
- platforms;
|
||||
- extractor backends.
|
||||
|
||||
Idempotence is defined inside the supported environment and extractor-version contract.
|
||||
|
||||
Any change that modifies durable Feature semantics or bytes requires explicit review and, when
|
||||
necessary, an extractor-version change. It must not silently reuse the old scientific identity.
|
||||
|
||||
## Feature File v2 and multi-descriptor support
|
||||
|
||||
**IMPLEMENTED v1A.**
|
||||
|
||||
ORB continues to write the historical Feature File v1 U8x32 representation.
|
||||
|
||||
SIFT and RootSIFT write Feature File v2 F32x128.
|
||||
|
||||
### v2 header
|
||||
|
||||
Feature File v2 has a 176-byte header.
|
||||
|
||||
Its fields include:
|
||||
|
||||
- magic;
|
||||
- explicit version;
|
||||
- header size;
|
||||
- feature count;
|
||||
- descriptor dimension;
|
||||
- descriptor type;
|
||||
- scalar size;
|
||||
- keypoint-record size;
|
||||
- decoded image dimensions;
|
||||
- capabilities;
|
||||
- keypoint/descriptor block offsets;
|
||||
- total size;
|
||||
- source SHA-256 at offset 72;
|
||||
- parameter fingerprint at offset 104;
|
||||
- 16-byte extractor kind at offset 136;
|
||||
- extractor version at offset 152;
|
||||
- twenty reserved zero bytes.
|
||||
|
||||
Stable descriptor type values are:
|
||||
|
||||
```text
|
||||
U8 = 1
|
||||
F32 = 2
|
||||
```
|
||||
|
||||
with scalar sizes 1 and 4.
|
||||
|
||||
Every binary32 value is explicitly encoded little-endian; persistent output is not a C/C++ struct
|
||||
dump.
|
||||
|
||||
Writer and reader reject NaN and Inf.
|
||||
|
||||
The typed descriptor readers:
|
||||
|
||||
- use `pread`;
|
||||
- refuse incompatible descriptor type;
|
||||
- return at most 256 features per call.
|
||||
|
||||
File-size limits remain:
|
||||
|
||||
```text
|
||||
Feature File v1 16 MiB
|
||||
Feature File v2 64 MiB
|
||||
```
|
||||
|
||||
The extraction facade also carries `descriptor_bytes`.
|
||||
|
||||
Publication requires exactly:
|
||||
|
||||
```text
|
||||
feature_count * descriptor_dimension * scalar_size
|
||||
```
|
||||
|
||||
and rejects both truncated and oversized descriptor buffers before file creation/publication.
|
||||
|
||||
The detailed multipass, grid and RootSIFT scientific contracts remain in
|
||||
[`precision_feature_pipeline.md`](precision_feature_pipeline.md).
|
||||
|
||||
## Downstream consumers
|
||||
|
||||
The Feature Store currently feeds implemented downstream stages:
|
||||
|
||||
```text
|
||||
Feature Store
|
||||
|
|
||||
v
|
||||
Visual Index
|
||||
|
|
||||
v
|
||||
Candidate Pair
|
||||
|
|
||||
v
|
||||
Matcher
|
||||
|
|
||||
v
|
||||
Geometric Verification
|
||||
|
|
||||
v
|
||||
Tracks
|
||||
```
|
||||
|
||||
Visual Index consumes the bounded Feature reader without changing Feature File format.
|
||||
|
||||
Candidate Pair consumes Feature Set identities indirectly through Visual Index membership.
|
||||
|
||||
Matcher consumes persisted Feature Sets and their descriptors.
|
||||
|
||||
Geometric Verification and Tracks consume later correspondence products; they do not rewrite Feature
|
||||
Store identity.
|
||||
|
||||
Sparse SfM Gates C through G are implemented and PASS/FROZEN, but real known-calibration Sparse SfM
|
||||
was not executed in the retained historical S21/A6000 campaigns.
|
||||
|
||||
## Real A6000 evidence
|
||||
|
||||
The retained real A6000 selected execution contains:
|
||||
|
||||
```text
|
||||
Feature Sets 689
|
||||
Candidate Pairs 38,420
|
||||
Match Results 38,420
|
||||
```
|
||||
|
||||
The v25 selected Feature-batch path completed all selected images.
|
||||
|
||||
The retained checkpoint is:
|
||||
|
||||
```text
|
||||
real-a6000-pre-sfm-2026-09-02
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
The later GV/Tracks continuation required:
|
||||
|
||||
```text
|
||||
Feature replay 0
|
||||
Candidate replay 0
|
||||
Matcher replay 0
|
||||
```
|
||||
|
||||
and continued from retained immutable upstream products.
|
||||
|
||||
This proves the current Feature-batch operational path and restart/reuse boundary on that campaign.
|
||||
It does not create calibration for that campaign and does not constitute real Sparse SfM execution.
|
||||
|
||||
## Current unfinished work
|
||||
|
||||
The following remain separate from the Feature Store scientific contract:
|
||||
|
||||
- global orphan-file reconciliation/scrub;
|
||||
- EXIF orientation handling where still required;
|
||||
- generic dependency/DAG scheduling;
|
||||
- future extractor/backend work only after explicit equivalence/version review.
|
||||
|
||||
Multi-image Feature execution itself is no longer an unfinished item:
|
||||
`features.extract.batch/1` is implemented and validated.
|
||||
|
||||
## Summary
|
||||
|
||||
```text
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
|
||||
FEATURE_STORE_STATUS=IMPLEMENTED
|
||||
|
||||
FEATURE_FILE_V1=FROZEN
|
||||
FEATURE_FILE_V2=IMPLEMENTED
|
||||
|
||||
HISTORICAL_FEATURE_TASK=features.extract/1
|
||||
CURRENT_FEATURE_BATCH_TASK=features.extract.batch/1
|
||||
PROJECT_DB_V25_FEATURE_BATCH=IMPLEMENTED/VALIDATED
|
||||
|
||||
FEATURE_RESULT_IDENTITY=PER_IMAGE
|
||||
PER_IMAGE_FEATURE_RESULT=ATOMIC
|
||||
PER_IMAGE_ATOMICITY_REQUIRES_CROSS_IMAGE_SERIALISM=NO
|
||||
OWNER_ONLY_PUBLICATION_REQUIRES_SERIAL_PREPARATION=NO
|
||||
|
||||
FEATURE_GPU=REJECTED_WITH_MEASURED_OR_IMPLEMENTATION_EVIDENCE
|
||||
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
|
||||
REAL_A6000_FEATURE_SETS=689
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
|
|
|||
|
|
@ -1,180 +1,408 @@
|
|||
# Geometric Verification
|
||||
# Geometric Verification Model
|
||||
|
||||
## Scope
|
||||
|
||||
Geometric Verification Model est le contrat persistant placé après le Matcher.
|
||||
Sa représentation stocke les identités scientifiques Geometric Verifier v1/v2
|
||||
historiques et v3 courantes, sans changement de schéma : `verifier_version` et
|
||||
`parameter_fingerprint` appartiennent déjà à l'identité exacte. Il stocke un
|
||||
résultat terminé, compact et immutable. Il n'est ni un moteur de calcul ni une
|
||||
tâche.
|
||||
Aucun RANSAC, USAC, MAGSAC, calcul d'inliers ou backend géométrique n'appartient à ce ticket.
|
||||
|
||||
## Position in reconstruction pipeline
|
||||
|
||||
La chaîne d'ownership est :
|
||||
## Status
|
||||
|
||||
```text
|
||||
Feature Set → Candidate Pair → Match Result → Geometric Verification Result
|
||||
GEOMETRIC_VERIFICATION_MODEL=IMPLEMENTED
|
||||
PROJECT_DB_GEOMETRIC_VERIFICATION=v12
|
||||
|
||||
HISTORICAL_VERIFIER_V1=VALID
|
||||
HISTORICAL_VERIFIER_V2=VALID
|
||||
CURRENT_PRODUCTION_VERIFIER_V3=FROZEN
|
||||
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
REAL_S21_GV_V3=PASS/FROZEN
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
Le masque indexe exclusivement l'ordre des entrées du Match File canonique du Match Result. Il
|
||||
n'indexe directement ni les features, ni la Candidate Pair, ni un ordre temporaire de backend.
|
||||
This document owns the **persistent Geometric Verification Result model**.
|
||||
|
||||
## Scientific ownership
|
||||
It does not own the numerical estimator implementation. The current executable scientific verifier is
|
||||
documented in `geometric_verifier.md`.
|
||||
|
||||
Le parent scientifique est `match_result_id`. L'API accepte uniquement un Match Result existant,
|
||||
`MATCHED`, avec `match_count` strictement positif. `NO_MATCH` et les erreurs runtime ne peuvent pas
|
||||
produire de résultat géométrique.
|
||||
The persistence model was deliberately version-ready from Project DB v12: `verifier_version` and
|
||||
`parameter_fingerprint` already belong to exact result identity. Therefore historical verifier v1/v2
|
||||
and current v3 results coexist without a schema reinterpretation.
|
||||
|
||||
## Parent Match Result
|
||||
## Pipeline position
|
||||
|
||||
Le Match Store reste propriétaire de la validation du Match File. La création consulte le parent
|
||||
et son `match_count` en DB ; elle ne relit pas l'asset. Un load valide aussi l'existence et l'état du
|
||||
parent afin qu'une ligne corrompue ne soit jamais rendue comme résultat valide.
|
||||
```text
|
||||
Feature Set
|
||||
-> Candidate Pair
|
||||
-> Match Result
|
||||
-> Geometric Verification Result
|
||||
-> Track Builder
|
||||
-> Track Model
|
||||
```
|
||||
|
||||
The inlier mask indexes the canonical Match File entry order.
|
||||
|
||||
It does not directly index:
|
||||
|
||||
- Feature Store physical order;
|
||||
- Candidate Pair order;
|
||||
- temporary backend order.
|
||||
|
||||
## Scientific parent
|
||||
|
||||
The exact parent is:
|
||||
|
||||
```text
|
||||
match_result_id
|
||||
```
|
||||
|
||||
A Geometric Verification Result may be created only for a valid `MATCHED` parent with positive
|
||||
`match_count`.
|
||||
|
||||
`NO_MATCH` and runtime failures do not produce a scientific geometric result.
|
||||
|
||||
## Persistent identity
|
||||
|
||||
L'identité demandée et unique est :
|
||||
Exact identity:
|
||||
|
||||
```text
|
||||
(match_result_id, verifier_kind, verifier_version, parameter_fingerprint)
|
||||
(
|
||||
match_result_id,
|
||||
verifier_kind,
|
||||
verifier_version,
|
||||
parameter_fingerprint
|
||||
)
|
||||
```
|
||||
|
||||
Le fingerprint est le SHA-256 opaque de 32 octets déjà standard dans le projet. Il représentera
|
||||
un encodage de paramètres versionné, stable, à ordre de champs explicite et, pour les nombres
|
||||
binaires, little-endian. Aucun timestamp, résultat, PID, durée ou identifiant matériel n'y entre.
|
||||
No selection by timestamp or "latest" is permitted.
|
||||
|
||||
The fingerprint is an opaque canonical SHA-256 scientific parameter identity.
|
||||
|
||||
It excludes:
|
||||
|
||||
- Task ID;
|
||||
- PID;
|
||||
- elapsed time;
|
||||
- CPU count;
|
||||
- batch size;
|
||||
- GPU identity;
|
||||
- hardware identity.
|
||||
|
||||
## Verifier kind
|
||||
|
||||
Le modèle supporte uniquement `FUNDAMENTAL`, valeur persistante stable 1. Aucun comportement fictif
|
||||
`ESSENTIAL` ou `HOMOGRAPHY` n'est réservé dans l'API publique.
|
||||
The persistent supported model kind is:
|
||||
|
||||
## Persistent states
|
||||
```text
|
||||
FUNDAMENTAL = 1
|
||||
```
|
||||
|
||||
- `GEOMETRIC_REJECTED=1` : calcul scientifique terminé, critère non satisfait ;
|
||||
- `GEOMETRIC_VERIFIED=2` : calcul scientifique terminé, critère satisfait.
|
||||
Do not reserve fictitious `ESSENTIAL` or `HOMOGRAPHY` values in prose without an explicit versioned
|
||||
implementation decision.
|
||||
|
||||
`FAILED`, `RUNNING`, `PAUSED` et `CANCELLED` appartiennent au Task Runtime. REJECTED peut conserver
|
||||
un nombre d'inliers non nul.
|
||||
## Scientific states
|
||||
|
||||
## Model representation
|
||||
Completed scientific states are:
|
||||
|
||||
FUNDAMENTAL utilise neuf colonnes SQLite `REAL`, en ordre ligne-major `m00` à `m22`. SQLite
|
||||
convertit les valeurs numériques en binary64 sans exposer une ABI C. VERIFIED exige les neuf
|
||||
valeurs présentes et finies. REJECTED exige les neuf valeurs NULL. Le modèle n'impose ni rang 2,
|
||||
ni déterminant, ni normalisation ou échelle canonique ; ces règles relèvent du futur verifier.
|
||||
```text
|
||||
GEOMETRIC_REJECTED = 1
|
||||
GEOMETRIC_VERIFIED = 2
|
||||
```
|
||||
|
||||
## Inlier representation
|
||||
Runtime states such as RUNNING, FAILED, PAUSED or CANCELLED belong to Task Runtime, not this model.
|
||||
|
||||
Le masque est un BLOB SQLite obligatoire de taille exacte `ceil(match_count / 8)`. Pour l'entrée
|
||||
`i`, `byte_index=i/8`, `bit_index=i%8` et le masque vaut `1u << bit_index`. Le bit 0 est donc le bit
|
||||
de poids faible de l'octet 0. Cette convention est indépendante de l'endianness CPU et de l'ABI.
|
||||
Les bits de padding du dernier octet valent zéro et le popcount est exactement `inlier_count`.
|
||||
A rejected result may still contain non-zero inlier support.
|
||||
|
||||
Le masque existe pour REJECTED comme pour VERIFIED. Avec 8192 matches, il mesure au maximum
|
||||
1024 octets. Un BLOB SQLite évite les milliers de lignes secondaires et la publication, le hash,
|
||||
le nettoyage et la récupération d'un asset externe d'environ 1 Kio. Une liste `uint32_t` serait
|
||||
jusqu'à 32 fois plus grande au cas dense et aurait un encodage supplémentaire à versionner.
|
||||
## Fundamental matrix representation
|
||||
|
||||
## Invariants
|
||||
A verified Fundamental result contains nine SQLite `REAL` values:
|
||||
|
||||
- `0 <= inlier_count <= parent.match_count <= 8192` ;
|
||||
- longueur, padding et popcount du masque sont canoniques ;
|
||||
- REJECTED possède un masque cohérent et aucun modèle ;
|
||||
- VERIFIED possède un masque cohérent et exactement neuf valeurs finies ;
|
||||
- kind, version et fingerprint ont une sérialisation stable ;
|
||||
- une ligne publiée est complète et immutable.
|
||||
```text
|
||||
m00 ... m22
|
||||
```
|
||||
|
||||
Exemple : pour 100 matches, une identité FUNDAMENTAL v1, v2 ou v3/fingerprint
|
||||
X peut publier REJECTED avec 23 inliers, un masque de 13 octets et aucun modèle.
|
||||
Une autre identité peut publier VERIFIED avec 67 inliers, le même format de
|
||||
masque et une matrice 3×3 finie.
|
||||
in row-major order.
|
||||
|
||||
## Persistence semantics
|
||||
The persistent representation is binary64 through SQLite numeric semantics, not a C ABI struct dump.
|
||||
|
||||
Une création valide puis insère identité, état, masque et modèle dans une transaction courte. Le
|
||||
calcul futur se fera entièrement avant cette transaction. SQLite fournit l'atomicité ; aucun asset
|
||||
ou journal secondaire n'est créé.
|
||||
A verified row requires nine finite values.
|
||||
|
||||
A rejected row contains no model.
|
||||
|
||||
Rank/canonicalization/scientific-estimator rules belong to the versioned verifier contract.
|
||||
|
||||
## Inlier mask
|
||||
|
||||
The mask is a required SQLite BLOB of exact size:
|
||||
|
||||
```text
|
||||
ceil(match_count / 8)
|
||||
```
|
||||
|
||||
Bit convention for Match File entry `i`:
|
||||
|
||||
```text
|
||||
byte = i / 8
|
||||
bit = i % 8
|
||||
mask[byte] & (1u << bit)
|
||||
```
|
||||
|
||||
The mask is LSB-first inside each byte.
|
||||
|
||||
Padding bits in the final byte are zero.
|
||||
|
||||
The mask popcount must equal `inlier_count`.
|
||||
|
||||
The mask exists for both verified and rejected scientific results.
|
||||
|
||||
With the current Match File bound of 8192 matches, the mask is at most 1024 bytes.
|
||||
|
||||
## Persistent invariants
|
||||
|
||||
For every row:
|
||||
|
||||
```text
|
||||
0 <= inlier_count <= parent.match_count <= 8192
|
||||
mask size is canonical
|
||||
padding bits are zero
|
||||
mask popcount == inlier_count
|
||||
```
|
||||
|
||||
Additionally:
|
||||
|
||||
```text
|
||||
REJECTED -> no Fundamental matrix
|
||||
VERIFIED -> exactly nine finite matrix coefficients
|
||||
```
|
||||
|
||||
A published row is immutable.
|
||||
|
||||
## Publication
|
||||
|
||||
Numerical estimation completes before the short Project DB publication transaction.
|
||||
|
||||
Publication inserts:
|
||||
|
||||
- exact parent;
|
||||
- exact verifier identity;
|
||||
- completed state;
|
||||
- canonical mask;
|
||||
- optional verified Fundamental model.
|
||||
|
||||
No external asset is required because the bounded mask/model fit naturally in SQLite.
|
||||
|
||||
Rollback leaves no partial scientific result.
|
||||
|
||||
## Reuse
|
||||
|
||||
Le reuse cherche uniquement l'identité exacte, jamais le résultat le plus récent. Une identité
|
||||
existante retourne une erreur de contrainte à `create`; le runtime fera `find`, validera puis
|
||||
réutilisera. `INSERT OR REPLACE` est interdit, même si le nouveau contenu semble identique.
|
||||
Exact reuse uses only the full persistent identity.
|
||||
|
||||
## Invalidations
|
||||
Existing exact result:
|
||||
|
||||
Un nouveau Match Result possède un nouvel ID et ne réutilise donc aucun ancien résultat
|
||||
géométrique. La FK emploie `ON DELETE CASCADE` : supprimer explicitement le parent supprime ses
|
||||
enfants et ne crée pas d'orphelin. Aucun moteur d'invalidation parallèle n'est nécessaire.
|
||||
```text
|
||||
find
|
||||
-> validate
|
||||
-> reuse
|
||||
```
|
||||
|
||||
## Project DB schema
|
||||
Never:
|
||||
|
||||
Project DB v12 ajoute `geometric_verification_results`, une contrainte UNIQUE sur l'identité et un
|
||||
index de pagination `(match_result_id, geometric_verification_result_id)`. Les CHECK SQL portent
|
||||
les bornes scalaires, tailles locales et nullabilité modèle/état. La cohérence avec le parent, le
|
||||
padding, le popcount et la finitude restent validés en C.
|
||||
```text
|
||||
INSERT OR REPLACE
|
||||
latest result
|
||||
closest fingerprint
|
||||
same parent with different version
|
||||
```
|
||||
|
||||
## API
|
||||
A new scientific verifier version creates another result identity.
|
||||
|
||||
L'API publique implémente :
|
||||
## Parent deletion
|
||||
|
||||
- `lardon3d_project_db_create_geometric_verification_result()` ;
|
||||
- `lardon3d_project_db_load_geometric_verification_result()` ;
|
||||
- `lardon3d_project_db_find_geometric_verification_result()` ;
|
||||
- `lardon3d_project_db_list_geometric_verification_results()`.
|
||||
The parent FK uses delete-cascade semantics.
|
||||
|
||||
La liste est bornée à 256 entrées, filtrée par parent puis ordonnée par ID croissant avec curseur.
|
||||
Le résultat en mémoire contient son `created_at` et son masque dans une capacité fixe de 1024
|
||||
octets : aucun ownership dynamique ni fonction de destruction. Les fonctions copient fingerprint,
|
||||
masque et neuf coefficients ; l'appelant conserve ses entrées.
|
||||
Explicit deletion of a Match Result deletes its dependent geometric results.
|
||||
|
||||
Parent absent retourne `NOT_FOUND`; parent NO_MATCH ou parent incohérent retourne `CONSTRAINT` à
|
||||
la création. Masque, modèle ou arguments locaux invalides retournent `INVALID_ARGUMENT`; duplicate
|
||||
identity retourne `CONSTRAINT`. Un loader qui rencontre une ligne ou un parent incohérent retourne
|
||||
`CORRUPT`, sans résultat partiel.
|
||||
No parallel invalidation engine is required.
|
||||
|
||||
## Resource bounds
|
||||
## Schema
|
||||
|
||||
Un résultat contient au plus 1024 octets de masque et 72 octets de valeurs numériques, plus de
|
||||
petites métadonnées. Une page est bornée. Le loader vérifie les entiers et tailles SQLite avant
|
||||
tout cast ou copie. Il n'existe ni cache global, ni lecture non bornée, ni Content Store associé.
|
||||
Le Match File parent mesure au plus 98 336 octets ; le futur job peut donc rester une petite unité.
|
||||
Project DB v12 introduced `geometric_verification_results`.
|
||||
|
||||
## Error ownership
|
||||
The current schema head is v25.
|
||||
|
||||
Seuls les résultats scientifiques terminés sont persistés. OOM, exception, annulation, timeout,
|
||||
device lost, I/O transitoire ou panne de thread appartiennent à l'exécution de tâche. État du modèle
|
||||
et état d'exécution sont deux contrats distincts.
|
||||
Later schema additions do not redefine the v12 row format or identity.
|
||||
|
||||
## Recovery semantics
|
||||
## Public API
|
||||
|
||||
Après commit, le résultat est complet et réutilisable après réouverture. Avant commit, le rollback
|
||||
ne laisse aucune ligne partielle. Un loader rejette toute ligne incohérente comme corruption au
|
||||
lieu de réparer ou d'interpréter au mieux.
|
||||
The model provides bounded create/load/find/list APIs for Geometric Verification Results.
|
||||
|
||||
## Verifier execution contract
|
||||
The list API is paged and ordered by increasing ID.
|
||||
|
||||
L'exécution prend un Match Result et son Match File borné. L'accès nécessaire existe via
|
||||
`lardon3d_feature_reader_keypoints()`, borné à 256 keypoints par appel ; l'intégration devra relier
|
||||
les deux Feature Sets et les indices du Match File sans modifier le Feature Store. Le verifier
|
||||
estimera hors transaction, dérivera état/masque/modèle, publiera en une courte transaction,
|
||||
checkpoint puis libérera les buffers. Une paire est l'unité atomique. Task Runtime et Resource
|
||||
Governor décideront admission, threads et lots ; zram/swap ne sont jamais un budget.
|
||||
In-memory result storage remains bounded: the inlier mask has fixed maximum capacity and no result-owned
|
||||
heap destructor is required for the core row object.
|
||||
|
||||
Un backend reste hors identité seulement s'il est scientifiquement transparent. Sinon son
|
||||
algorithme ou contrat doit apparaître dans kind/version/fingerprint avant publication. Toute seed
|
||||
influençant le résultat doit avoir une politique déterministe versionnée ou être couverte par le
|
||||
fingerprint. Aucun nombre de threads ou hardware ID n'est un paramètre scientifique par défaut.
|
||||
Exact function declarations in the public headers remain authoritative.
|
||||
|
||||
## Explicitly out of scope
|
||||
## Error semantics
|
||||
|
||||
GPU, Vulkan, OpenCL, shader et nouvelle orchestration restent hors périmètre de ce contrat de
|
||||
persistance.
|
||||
Creation distinguishes invalid local arguments from parent/identity constraints.
|
||||
|
||||
## Versioning
|
||||
Loaders return corruption rather than a partially interpreted result if:
|
||||
|
||||
Project DB schema version 12 décrit le stockage. `verifier_version` décrit indépendamment le
|
||||
contrat scientifique. Changer un algorithme n'impose une migration DB que si la représentation
|
||||
persistante change.
|
||||
- parent is missing or invalid;
|
||||
- stored mask length is wrong;
|
||||
- padding is non-canonical;
|
||||
- popcount disagrees;
|
||||
- model/state nullability is inconsistent;
|
||||
- a verified matrix contains non-finite values.
|
||||
|
||||
Scientific rejection is not a database/runtime failure.
|
||||
|
||||
Runtime OOM, exception, cancellation, estimator failure or device failure are not persisted as
|
||||
`GEOMETRIC_REJECTED`.
|
||||
|
||||
## Current verifier lineage
|
||||
|
||||
The model stores all supported versions through the same identity fields.
|
||||
|
||||
### v1
|
||||
|
||||
Historical Fundamental verifier v1 remains immutable and valid.
|
||||
|
||||
### v2
|
||||
|
||||
Historical Fundamental verifier v2 remains immutable and valid.
|
||||
|
||||
V2 added the distinct-canonical-observation preflight in the scientific execution contract.
|
||||
|
||||
### v3
|
||||
|
||||
Current production verifier is Fundamental v3.
|
||||
|
||||
Production fingerprint:
|
||||
|
||||
```text
|
||||
6944a471d611d8ffc59dac7cf15a5b79b97e2371d4c51785c477d68c1577f74c
|
||||
```
|
||||
|
||||
V3 preserves the persistent model and adds its versioned scientific preflight before the unchanged
|
||||
eligible estimator path.
|
||||
|
||||
No Project DB migration was needed for v3 because v12 already stores verifier version and fingerprint.
|
||||
|
||||
## Task relationship
|
||||
|
||||
The production Task Kind is:
|
||||
|
||||
```text
|
||||
geometric_verifier.run/1
|
||||
```
|
||||
|
||||
Project DB v13 adds only its typed durable Task payload.
|
||||
|
||||
The Task:
|
||||
|
||||
```text
|
||||
pages Match Results
|
||||
-> validates eligibility
|
||||
-> computes or reuses exact GVR identity
|
||||
-> owner publishes in canonical parent order
|
||||
-> advances typed cursor
|
||||
-> checkpoints
|
||||
-> sequence_break
|
||||
```
|
||||
|
||||
Task/runtime state remains separate from GVR scientific state.
|
||||
|
||||
## Current resource boundary
|
||||
|
||||
One Match Result is the scientific atomic item.
|
||||
|
||||
Current validated outer-parallel Task execution may prepare independent parents concurrently.
|
||||
|
||||
The owner publishes the contiguous canonical prefix.
|
||||
|
||||
Current validated bounds include:
|
||||
|
||||
```text
|
||||
useful CPU participants <= 8
|
||||
safe parent/window size <= 16
|
||||
per-item reservation approximately 8 MiB
|
||||
GPU = 0
|
||||
```
|
||||
|
||||
The internal USAC/MAGSAC scientific solver remains `isParallel=false`.
|
||||
|
||||
These operational values do not enter GVR identity.
|
||||
|
||||
## Real S21 v3 evidence
|
||||
|
||||
Retained S21 proof:
|
||||
|
||||
```text
|
||||
REAL_S21_GV_V3=PASS/FROZEN
|
||||
|
||||
Match Results 172,741
|
||||
Applicable MATCHED 172,275
|
||||
Verified GVRs 24,065
|
||||
Rejected GVRs 148,210
|
||||
non-applicable 466
|
||||
duplicate mappings 0
|
||||
```
|
||||
|
||||
The source Matcher project was retained unchanged and GV ran only from the Match Result boundary.
|
||||
|
||||
Restart/idempotence evidence preserved the complete GVR result set.
|
||||
|
||||
No Track/Sparse work belonged to the original GV-only boundary.
|
||||
|
||||
## Real A6000 v3 evidence
|
||||
|
||||
Retained current A6000 pre-SfM continuation:
|
||||
|
||||
```text
|
||||
Match Results 38,420
|
||||
Applicable GVRs 37,805
|
||||
Verified GVRs 10,952
|
||||
Rejected GVRs 26,853
|
||||
duplicate mappings 0
|
||||
```
|
||||
|
||||
Fingerprint:
|
||||
|
||||
```text
|
||||
6944a471d611d8ffc59dac7cf15a5b79b97e2371d4c51785c477d68c1577f74c
|
||||
```
|
||||
|
||||
The continuation then built Tracks and stopped before real Sparse SfM.
|
||||
|
||||
Checkpoint:
|
||||
|
||||
```text
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
## Out of scope
|
||||
|
||||
This persistence model does not define:
|
||||
|
||||
- RANSAC/USAC/MAGSAC implementation;
|
||||
- GPU kernels;
|
||||
- Task scheduling;
|
||||
- Track construction;
|
||||
- Essential pose;
|
||||
- triangulation;
|
||||
- Sparse SfM;
|
||||
- Homography competition.
|
||||
|
||||
Those belong to their versioned scientific/runtime contracts.
|
||||
|
||||
## Summary
|
||||
|
||||
```text
|
||||
GEOMETRIC_VERIFICATION_MODEL=IMPLEMENTED
|
||||
PROJECT_DB_GEOMETRIC_VERIFICATION=v12
|
||||
PROJECT_DB_GEOMETRIC_VERIFIER_TASK=v13
|
||||
|
||||
CURRENT_PRODUCTION_VERIFIER=FUNDAMENTAL_V3
|
||||
CURRENT_VERIFIER_FINGERPRINT=6944a471d611d8ffc59dac7cf15a5b79b97e2371d4c51785c477d68c1577f74c
|
||||
|
||||
REAL_S21_GV_V3=PASS/FROZEN
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
```
|
||||
|
|
|
|||
|
|
@ -1,468 +1,486 @@
|
|||
# Geometric Verifier v1 / v2 / v3
|
||||
|
||||
## Scope
|
||||
## Status
|
||||
|
||||
Ce document décrit l'exécution scientifique qui transforme un Match Result `MATCHED` en résultat
|
||||
Fundamental `GEOMETRIC_REJECTED` ou `GEOMETRIC_VERIFIED`. Le contrat persistant reste défini par
|
||||
[`geometric_verification.md`](geometric_verification.md). Tracks, pose, Essential, compétition
|
||||
Homography, triangulation et SfM sont hors périmètre.
|
||||
```text
|
||||
HISTORICAL_GEOMETRIC_VERIFIER_V1=FROZEN
|
||||
HISTORICAL_GEOMETRIC_VERIFIER_V2=FROZEN
|
||||
CURRENT_GEOMETRIC_VERIFIER_V3=PASS/FROZEN
|
||||
|
||||
V1 et v2 restent des identités scientifiques historiques et immutables : leurs versions,
|
||||
fingerprints, lignes et résultats existants ne sont jamais réinterprétés. V2 conserve l'estimator,
|
||||
les paramètres, l'ordre et l'acceptance v1, mais ajoute avant USAC le support minimal par
|
||||
observations canoniques distinctes décrit ci-dessous. V3 est la policy de production courante :
|
||||
après les mêmes validations intégrales, elle compose ce support v2 avec la preuve exacte de
|
||||
faisabilité d'acceptation `match_count >= min_inlier_count`. Chaque policy possède sa version et son
|
||||
fingerprint distincts ; le schéma les stocke déjà depuis v12 et la tête courante
|
||||
Project DB v23 ne nécessite aucune migration GV.
|
||||
CURRENT_VERIFIER_KIND=FUNDAMENTAL
|
||||
CURRENT_VERIFIER_VERSION=3
|
||||
CURRENT_VERIFIER_FINGERPRINT=6944a471d611d8ffc59dac7cf15a5b79b97e2371d4c51785c477d68c1577f74c
|
||||
|
||||
GEOMETRIC_VERIFIER_GPU=NOT_JUSTIFIED
|
||||
REAL_S21_GV_V3=PASS/FROZEN
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
This document owns the scientific execution that turns one valid `MATCHED` Match Result into one
|
||||
completed Fundamental Geometric Verification Result.
|
||||
|
||||
The persistent row model is owned by `geometric_verification.md`.
|
||||
|
||||
Tracks, Essential pose, triangulation and Sparse SfM are downstream.
|
||||
|
||||
## Version lineage
|
||||
|
||||
### v1
|
||||
|
||||
Verifier v1 is a frozen historical scientific identity.
|
||||
|
||||
Its existing fingerprints and GVR rows remain immutable.
|
||||
|
||||
### v2
|
||||
|
||||
Verifier v2 preserves the v1 estimator/acceptance path but adds a bounded preflight requiring enough
|
||||
distinct canonical observations on both sides before estimator execution.
|
||||
|
||||
It has its own version/fingerprint.
|
||||
|
||||
### v3
|
||||
|
||||
Verifier v3 is the current production policy.
|
||||
|
||||
It preserves v2 validation and additionally rejects estimator-ineligible parents when:
|
||||
|
||||
```text
|
||||
match_count < min_inlier_count
|
||||
```
|
||||
|
||||
because acceptance is mathematically impossible in that case.
|
||||
|
||||
At equality, the parent remains estimator-eligible.
|
||||
|
||||
V3 does not relabel or mutate v1/v2 rows.
|
||||
|
||||
Project DB v12 already stores verifier version/fingerprint, so no GVR schema migration is needed.
|
||||
|
||||
Current project schema head is v25.
|
||||
|
||||
## Inputs
|
||||
|
||||
Le parent DB fournit les deux Feature Set IDs, le compte, le chemin, la taille et le SHA-256 du
|
||||
Match File. Le reader Feature Store ouvre séparément chaque Feature Set validé et expose les
|
||||
keypoints par plages d'au plus 256. Le verifier n'a besoin d'aucun descriptor : charger les blocs
|
||||
ORB ou SIFT/RootSIFT serait inutile et est interdit dans le chemin normal.
|
||||
The exact Match Result supplies:
|
||||
|
||||
Les keypoints persistants portent des coordonnées `binary32`. `x/y` sont exprimés en pixels de
|
||||
l'image exactement décodée par OpenCV lors de l'extraction, avec origine en haut à gauche et
|
||||
positions subpixel possibles. Les dimensions décodées sont disponibles dans les métadonnées du
|
||||
Feature File.
|
||||
- Candidate Pair relation;
|
||||
- Feature Set A/B identities;
|
||||
- Match File path/size/SHA;
|
||||
- `match_count`.
|
||||
|
||||
## Fundamental matrix contract
|
||||
Feature readers provide keypoints for the two immutable Feature Sets.
|
||||
|
||||
Le seul modèle v1 est une matrice Fundamental 3×3. Une sortie acceptée doit être unique, finie,
|
||||
de norme non nulle et canonique avant publication. V1 ne projette pas la matrice vers le rang 2.
|
||||
The verifier does not need descriptor blocks in its normal geometry path.
|
||||
|
||||
Feature coordinates are persistent binary32 decoded-image pixels with top-left origin.
|
||||
|
||||
They are converted to binary64 `Point2d` for geometric computation.
|
||||
|
||||
## Input ordering
|
||||
|
||||
L'entrée `i` de l'estimator correspond exactement à l'entrée `i` du Match File :
|
||||
`feature_index_a` sélectionne le Feature Set A et `feature_index_b` le Feature Set B. Le Match
|
||||
File impose déjà des indices A strictement croissants ; le verifier ne trie et ne filtre pas les
|
||||
correspondances. Toute corruption d'index est une erreur d'exécution, jamais un rejet scientifique.
|
||||
Estimator row `i` corresponds exactly to Match File entry `i`.
|
||||
|
||||
V2 distingue le nombre brut de lignes des observations canoniques distinctes. Les identités sont
|
||||
`A=(feature_set_id_a, feature_index_a)` et
|
||||
`B=(feature_set_id_b, feature_index_b)`. Après validation intégrale du parent, du Match asset, des
|
||||
Feature Sets et Feature assets, v2 exige au moins sept A distincts **et** sept B distincts. Sinon il
|
||||
publie `GEOMETRIC_REJECTED`, `inlier_count=0`, masque intégralement nul de longueur exactement
|
||||
`ceil(match_count/8)`, sans modèle et sans appel USAC.
|
||||
The verifier does not reorder or deduplicate Match File rows.
|
||||
|
||||
Ce préflight ne modifie jamais l'évidence Matcher : aucune déduplication, tri, contrainte
|
||||
one-to-one, unicité de coordonnées, limite de multiplicité, analyse de rang/conditionnement/
|
||||
colinéarité ou compétition Homography n'est appliquée. Des IDs distincts ayant les mêmes
|
||||
coordonnées restent des observations distinctes. Le Match File canonique rendant A strictement
|
||||
croissant, une insuffisance A implique en pratique moins de sept lignes valides ; B peut en revanche
|
||||
être insuffisant malgré un grand nombre de lignes brutes.
|
||||
The published inlier bit `i` always maps back to Match File entry `i`.
|
||||
|
||||
V3 exécute ensuite USAC seulement si `match_count >= min_inlier_count`. Lorsque cette inégalité
|
||||
échoue, au plus `match_count` bits du masque pourraient être inliers : l'acceptation est donc
|
||||
mathématiquement impossible. V3 publie alors le même rejet zéro borné sans appel estimator. La
|
||||
borne vient du paramètre durable, jamais d'une constante `16`. À l'égalité, l'entrée reste éligible.
|
||||
Ce contrat n'ajoute aucune règle `N<20`, rang, coordonnées, homographie, déduplication ou retry ;
|
||||
toute entrée qui franchit les deux préflights appelle l'USAC inchangé et toute exception inattendue
|
||||
reste un échec Task sans publication.
|
||||
Out-of-range Feature indices or corrupt assets are runtime/input failure, not scientific rejection.
|
||||
|
||||
## Coordinate representation
|
||||
## Canonical observation identity
|
||||
|
||||
Le stockage source reste `binary32`. Sur 1024 points, bruit 0,75 px et 50 % d'outliers, Point2f et
|
||||
Point2d ont produit le même masque et la même qualité, en 3,58 et 3,55 ms. La production convertit
|
||||
vers Point2d pour rendre le calcul et la sortie binary64 explicites, pour 256 Kio au maximum.
|
||||
Aucune mise à l'échelle par résolution ni conversion de repère n'est appliquée implicitement.
|
||||
For preflight counting:
|
||||
|
||||
## Algorithm candidates
|
||||
```text
|
||||
A = (feature_set_id_a, feature_index_a)
|
||||
B = (feature_set_id_b, feature_index_b)
|
||||
```
|
||||
|
||||
OpenCV 5 installé expose `FM_RANSAC`, `USAC_DEFAULT`, `USAC_ACCURATE`, `USAC_PROSAC` et
|
||||
`USAC_MAGSAC`. La shortlist Gate A est FM_RANSAC comme baseline, puis USAC_DEFAULT,
|
||||
USAC_MAGSAC et USAC_ACCURATE. PROSAC est `NOT_APPLICABLE` en v1 : la distance descriptor est
|
||||
persistée mais l'ordre canonique suit l'index de query, pas un classement de qualité benchmarké.
|
||||
V2/v3 require at least seven distinct A observations and seven distinct B observations.
|
||||
|
||||
## Benchmark methodology
|
||||
Failure publishes a zero-inlier `GEOMETRIC_REJECTED` result with a correctly sized all-zero mask and
|
||||
no Fundamental model.
|
||||
|
||||
Un corpus synthétique déterministe avec Fundamental ground truth couvrira bruit, outliers,
|
||||
résolutions, tailles, géométries saines, faibles et adversariales. Les méthodes seront comparées
|
||||
par précision/recall du masque, erreurs épipolaires, échecs, repeatability, temps et ressources.
|
||||
Le benchmark lourd restera hors build et suite par défaut. Aucune fixture photo réelle ne sera
|
||||
revendiquée sans fixture non sensible présente dans le dépôt.
|
||||
This preflight does not:
|
||||
|
||||
La campagne Gate A du 9 août 2026 utilise OpenCV 5.0.0, Clang 22.1.8, une seed fixe et 32
|
||||
répétitions. Elle couvre 7 à 8192 points, 0 à 100 % d'outliers, bruit 0 à 1,5 px, 1280×720 à
|
||||
4000×3000, baseline faible/large, concentration, quasi-colinéarité, planéité, rotation dominante
|
||||
et duplications. Aucune fixture photo réelle représentative n'existe dans le dépôt.
|
||||
- rewrite Matcher evidence;
|
||||
- enforce one-to-one matching;
|
||||
- deduplicate coordinates;
|
||||
- perform a rank test;
|
||||
- perform collinearity analysis;
|
||||
- run Homography competition.
|
||||
|
||||
| Algorithme | P/R 1024, 30 % | P/R 8192, 70 % | Médiane/p95/pire 8192 | Seed locale | Stable 32× |
|
||||
|---|---:|---:|---:|---|---|
|
||||
| FM_RANSAC | 0,998/0,720 | 0,993/0,413 | 316,4/318,9/319,7 ms | non | oui observé |
|
||||
| USAC_DEFAULT | 0,996/0,960 | 0,997/0,959 | 43,0/43,2/44,8 ms | preset non | oui |
|
||||
| USAC_MAGSAC | 0,993/0,965 | 0,994/0,962 | 11,4/12,0/12,1 ms | preset non | oui |
|
||||
| USAC_ACCURATE | 0,996/0,960 | 0,996/0,961 | 30,5/32,0/32,1 ms | preset non | oui |
|
||||
| MAGSAC params v1 | 0,997/0,957 | 0,996/0,894 | 10,8/11,0/11,4 ms | oui | oui |
|
||||
Distinct Feature IDs with identical coordinates remain distinct observations.
|
||||
|
||||
FM_RANSAC est rejeté pour son recall et son pire temps. DEFAULT et ACCURATE n'améliorent pas assez
|
||||
la qualité pour leur coût. La production emploie des UsacParams explicites : la seed par appel
|
||||
prime sur la variation du cas extrême liée à la seed fixe. À bruit 0,75 px/50 % d'outliers, les
|
||||
seuils 0,5/1,0/1,5/2,0/3,0 donnent des recalls 0,535/0,811/0,961/0,990/1,000 et des precisions
|
||||
0,996/0,988/0,990/0,986/0,985. Le compromis retenu est 1,5 px.
|
||||
## v3 acceptance-feasibility preflight
|
||||
|
||||
## Determinism
|
||||
V3 additionally checks the durable configured `min_inlier_count`.
|
||||
|
||||
USAC expose `cv::UsacParams::randomGeneratorState`, un entier par appel, ainsi que les paramètres
|
||||
de sampling, score, optimisation locale et polishing. Cette API est préférable à une mutation de
|
||||
`cv::theRNG()` process-global. FM_RANSAC restera une baseline scientifique tant que son contrôle
|
||||
RNG et sa repeatability n'ont pas été mesurés.
|
||||
When:
|
||||
|
||||
Les cinq candidats ont donné un hash modèle+masque identique sur 32 appels et dans trois processus
|
||||
distincts. La garantie v1 reste intra-environnement : mêmes octets, ordre, configuration, seed,
|
||||
OpenCV 5.0.0 et architecture. Aucun bit-exact cross-version ou cross-architecture n'est promis.
|
||||
```text
|
||||
match_count < min_inlier_count
|
||||
```
|
||||
|
||||
## Random seed policy
|
||||
the maximum possible support cannot satisfy acceptance, so V3 publishes the canonical zero rejection
|
||||
without calling USAC.
|
||||
|
||||
La policy v1 calcule SHA-256 sur `L3DGVSE1`, le SHA-256 du Match File puis le fingerprint. Les
|
||||
quatre premiers octets sont décodés little-endian et les 31 bits faibles alimentent
|
||||
`randomGeneratorState`. La policy est version 1.
|
||||
The threshold comes from configuration, never a hidden constant.
|
||||
|
||||
## Parameter fingerprint
|
||||
## Scientific model
|
||||
|
||||
Le fingerprint v1/v2/v3 est SHA-256 des 84 octets suivants. Les entiers sont little-endian ; les doubles
|
||||
sont leurs bits IEEE-754 binary64 écrits comme `uint64_t` little-endian. NaN/Inf sont refusés et
|
||||
le seul champ autorisant zéro signé, `min_inlier_ratio`, normalise `-0.0` en `+0.0`. Aucun octet ne
|
||||
provient d'un dump de structure.
|
||||
The only production model is a 3x3 Fundamental matrix.
|
||||
|
||||
| Offset | Taille | Champ |
|
||||
|---:|---:|---|
|
||||
| 0 | 8 | domaine ASCII `L3DGVFP1` |
|
||||
| 8 | 4 | version encodage = 1 |
|
||||
| 12 | 4 | kind FUNDAMENTAL = 1 |
|
||||
| 16 | 4 | verifier version = 1, 2 ou 3 |
|
||||
| 20 | 4 | algorithme USAC_MAGSAC explicite = 1 |
|
||||
| 24 | 8 | threshold binary64 |
|
||||
| 32 | 8 | confidence binary64 |
|
||||
| 40 | 4 | max iterations |
|
||||
| 44 | 4 | minimum inlier count |
|
||||
| 48 | 8 | minimum inlier ratio binary64 |
|
||||
| 56 | 4 | seed policy version |
|
||||
| 60 | 4 | canonicalisation version |
|
||||
| 64 | 1 | représentation Point2d = 2 |
|
||||
| 65 | 1 | sampler uniforme = 0 |
|
||||
| 66 | 1 | score MAGSAC = 2 |
|
||||
| 67 | 1 | isParallel = 0 |
|
||||
| 68 | 1 | LO inner = 1 |
|
||||
| 69 | 4 | LO iterations = 5 |
|
||||
| 73 | 4 | LO sample size = 14 |
|
||||
| 77 | 1 | neighbor grid = 1 |
|
||||
| 78 | 1 | COV polisher = 3 |
|
||||
| 79 | 4 | polisher iterations = 3 |
|
||||
| 83 | 1 | réservé nul |
|
||||
Accepted output must be finite, non-zero and canonical.
|
||||
|
||||
Le vector golden v1 de la configuration historique commence par les 84 octets hexadécimaux
|
||||
`4c33444756465031...0300000000` et donne le SHA-256
|
||||
`ddb44bb070c62be66c405946e89cbb49c084f8f30a21d6f408dc239225b7bbd0`. Pour un Match File SHA
|
||||
composé de 31 octets nuls puis `01`, cette configuration donne la seed décimale `1910542150`.
|
||||
V2 conserve cet encodage et place `2` au champ `verifier version`; son fingerprint diffère donc
|
||||
obligatoirement même lorsque les sept paramètres numériques sont identiques. La seed dérivée suit
|
||||
ce fingerprint v2 et appartient à cette nouvelle identité. Pour la configuration production, le
|
||||
fingerprint v2 est `7868a893437ee611a10008a093286997212fa8bd80b2afd2bb1d11f04f01c5ae` ;
|
||||
le même Match SHA golden donne la seed décimale `1528046088`.
|
||||
V3 conserve encore exactement les 84 octets et place `3` au champ version. Pour la configuration
|
||||
production, son fingerprint est
|
||||
`6944a471d611d8ffc59dac7cf15a5b79b97e2371d4c51785c477d68c1577f74c` ; le même Match SHA golden
|
||||
donne la seed décimale `188721673`.
|
||||
Les politiques de
|
||||
ressources, hardware, PSI, lot, worker et réservation CPU ne sont ni des champs ni des entrées.
|
||||
|
||||
## Acceptance policy
|
||||
|
||||
Un modèle candidat qui échoue à cette policy publie REJECTED avec son masque et son compte
|
||||
d'inliers, sans matrice. La production exige `inlier_count >= 16` et
|
||||
`inlier_count / match_count >= 0,20`. Les cas 100 % faux produisent 10/64, 14/256, 26/1024 et
|
||||
28/4096 inliers, ratio maximal 0,15625. Les scènes saines produisent 45/64, 129/256 et 297/1024 ;
|
||||
la faible baseline produit 126/256.
|
||||
|
||||
## Fundamental matrix canonicalization
|
||||
|
||||
La production adopte cette canonicalisation version 1. Les neuf valeurs doivent être finies. La
|
||||
norme de Frobenius est calculée avec une accumulation `hypot` résistante au débordement ; zéro est
|
||||
refusé. Le premier coefficient de valeur absolue strictement maximale gagne, donc un tie conserve
|
||||
le plus petit index ligne-major. Après division, le signe rend ce pivot positif et les zéros signés
|
||||
sont normalisés à `+0.0`. Sur 8192/70 %, les singular values sont
|
||||
3,392e-2, 1,374e-4 et 5,915e-24. OpenCV fournit déjà rank-2 à précision numérique. V1 ne calcule
|
||||
aucune SVD en production, n'impose aucun seuil de rang et n'effectue aucune post-projection rank-2.
|
||||
Les validations production portent uniquement sur la forme 3×3 unique, la finitude et la norme.
|
||||
|
||||
## Inlier mask generation
|
||||
|
||||
Le masque OpenCV est validé en type, taille et valeurs, puis converti sans réordonnancement vers
|
||||
le bitset LSB-first du modèle. Les frontières 7/8/9, 63/64/65 et 8191/8192 sont testées.
|
||||
|
||||
Le core conserve strictement l'ordre d'entrée du Match File. Les tests utilisent des indices B
|
||||
permutés et des masques non contigus ; le bit `i` publié reste l'élément `i` du fichier, jamais
|
||||
l'index de feature. Les tailles 1, 2, 7, 8, 9, 63, 64, 65, 8191 et 8192, le padding nul et le
|
||||
popcount sont couverts avec le Model v1 inchangé.
|
||||
|
||||
## Scientific rejection
|
||||
|
||||
En v1, un nombre brut de matches inférieur au minimum réel produit le rejet zéro historique. En
|
||||
v2, moins de sept observations canoniques distinctes sur A ou B produit le rejet zéro décrit dans
|
||||
`Input ordering`. V3 conserve ce test et rejette également quand le nombre brut est strictement
|
||||
inférieur au `min_inlier_count` configuré. L'absence de modèle sur entrée valide ou l'échec de
|
||||
l'acceptance policy produit un résultat scientifique REJECTED cohérent.
|
||||
|
||||
Le minimum USAC observé est sept. Sept à quinze observations supportées peuvent produire une
|
||||
hypothèse mais ne franchissent pas nécessairement le support d'acceptation production.
|
||||
|
||||
## Execution failure
|
||||
|
||||
Match/Feature asset absent ou corrompu, index hors bornes, exception OpenCV, OOM, masque malformé,
|
||||
matrice non finie ou invariant interne invalide échoue dans le Task Runtime. Aucun résultat
|
||||
scientifique n'est publié dans ces cas.
|
||||
|
||||
Le core traduit parent absent/NO_MATCH et Feature Set absent en erreur d'exécution `NOT_FOUND` ;
|
||||
asset absent, tronqué, hash divergent, ownership ou index incohérent en `CORRUPT` ; exception ou
|
||||
sortie estimator malformée/non finie en `ESTIMATOR_ERROR` ; `bad_alloc` en `OUT_OF_MEMORY` ; et
|
||||
échec Model en `DATABASE_ERROR`. Les seams test-only couvrent erreur estimator, mask/matrice
|
||||
malformés, NaN, OOM et publication. Aucun de ces chemins ne crée de résultat scientifique.
|
||||
|
||||
## Resource bounds
|
||||
|
||||
Une unité atomique est un Match Result, au maximum 8192 correspondances. Le Match File est borné
|
||||
à 98 336 octets et le bitset à 1024 octets. Aucun cache global ni préchargement de projet complet
|
||||
n'est utilisé.
|
||||
|
||||
À 8192 matches, les allocations directement contrôlées maximales sont 98 304 octets d'entries,
|
||||
393 216 octets de keypoints A/B, 262 144 octets de Point2d A/B, deux cartes v2 de présence de
|
||||
8192 octets, 1024 octets de bitset, environ 8192 octets de mask OpenCV et 72 octets de modèle, soit
|
||||
environ 761 Kio hors petits objets et scratch OpenCV. Les cartes appartiennent à une paire et sont
|
||||
libérées au retour ; leur borne Feature Store est opérationnelle et n'ajoute aucune limite
|
||||
scientifique. Aucun descriptor ni matrice A×B n'est lu. Massif mesure 2,445 Mio de heap au pic
|
||||
du test E2E complet, incluant SQLite, OpenCV, fixtures Feature Store et toutes les séquences de test.
|
||||
La forme sérielle historique réservait 4 Mio fixes. La Task outer-parallel
|
||||
courante réserve 8 Mio par item admis afin de couvrir aussi la pile enfant
|
||||
bornée de 4 Mio, l'objet préparé, les readers et le scratch opaque. Avec un lot
|
||||
maximal 16, cette charge reste bornée à 128 Mio et ne limite pas la cardinalité
|
||||
du dataset.
|
||||
|
||||
## CPU policy
|
||||
|
||||
Le parallélisme scientifique interne OpenCV reste explicitement désactivé :
|
||||
`UsacParams::isParallel=false` appartient au fingerprint FROZEN. Le verifier ne
|
||||
change pas `cv::setNumThreads()` par paire.
|
||||
|
||||
Le parallélisme courant porte uniquement sur des Match Results indépendants.
|
||||
Le Governor peut admettre CPU1..8 pour une fenêtre/lot d'au plus 16 ; le
|
||||
callback Queue compte comme participant, crée au plus `cpu_threads-1` enfants,
|
||||
les joint, puis publie seul dans l'ordre. La fenêtre participant a été validée
|
||||
sûre jusqu'à 16, mais CPU8 est le maximum utile mesuré. Ce choix opérationnel
|
||||
n'entre ni dans le fingerprint, ni dans le GVR.
|
||||
|
||||
## GPU policy
|
||||
|
||||
Aucun backend Vulkan n'est implémenté avant profil du chemin CPU final. La décision attendue est
|
||||
`NOT_JUSTIFIED` si les unités restent sub-millisecondes ou de quelques millisecondes.
|
||||
|
||||
Verdict Gate A : `NOT_JUSTIFIED`. Les cas usuels prennent 0,3 à 5,6 ms et le pire MAGSAC local
|
||||
mesuré reste à 11,4 ms. Aucun backend Vulkan de vérification n'est implémenté.
|
||||
|
||||
## Task Runtime
|
||||
|
||||
L'audit Gate C conclut que le checkpoint générique v1 est insuffisant : il conserve l'état,
|
||||
la progression, le compteur de séquences et les temps, mais aucun payload propre au kind. Le
|
||||
reconstructeur doit retrouver les paramètres scientifiques immuables et le curseur sans les
|
||||
inventer depuis un fingerprint irréversible.
|
||||
|
||||
`geometric_verifier_tasks` est donc la seule raison de Project DB v13. Elle doit conserver
|
||||
`task_id`, `after_match_result_id`, les sept paramètres de configuration v1 et le fingerprint
|
||||
calculé à la création pour validation à la reconstruction. Aucun `cv::Mat`, buffer, état RNG,
|
||||
paramètre Governor ou donnée hardware n'y appartient. La tâche calcule hors transaction et publie
|
||||
chaque résultat par transaction courte avant avancement du curseur.
|
||||
|
||||
Le payload v22 ne duplique pas `verifier_version`. À la reconstruction, le fingerprint exact est
|
||||
comparé aux encodages supportés v1, v2 et v3 des sept paramètres durables : cela restaure sans
|
||||
ambiguïté chaque policy, sans migration ni nouveau système d'identité. Toute nouvelle tâche
|
||||
sélectionne v3 ; le payload et l'ABI publique de configuration restent inchangés. Une tâche v3
|
||||
démarre avec une identité neuve et ne reprend ni ne ré-étiquette une tâche v1/v2.
|
||||
|
||||
Le Task Kind production est `geometric_verifier.run` version 1. Il pagine les Match Results par ID
|
||||
strictement croissant avec une page de `batch + 1`, et traite des lots Governor
|
||||
1..16 avec CPU utile 1..8. Les préparations éligibles peuvent s'exécuter en
|
||||
parallèle, mais la publication, l'avancement du curseur contigu et le checkpoint
|
||||
restent owner-only et ordonnés. Les parents autres que `MATCHED` avec
|
||||
`match_count > 0` sont seulement traversés par le curseur. Une unité éligible
|
||||
appelle le core, qui reuse l'identité exacte avant toute lecture d'asset.
|
||||
|
||||
WHY GENERIC TASK PERSISTENCE IS INSUFFICIENT: aucun champ de payload métier dans le snapshot v1.
|
||||
|
||||
REQUIRED DURABLE FIELDS: configuration scientifique v1, fingerprint et dernier Match Result
|
||||
publié puis checkpointé.
|
||||
|
||||
WHY EXISTING DB CANNOT STORE THEM: `tasks` et `checkpoints` ne portent que le résumé générique ;
|
||||
aucune table v12 ne possède une ligne 1:1 adaptée à ce Task Kind.
|
||||
|
||||
## Checkpoint/recovery
|
||||
|
||||
La pagination suit `match_result_id` croissant sans supposer des IDs contigus. Le résultat est
|
||||
publié avant que `after_match_result_id` avance en mémoire ; le curseur n'est persisté qu'après le
|
||||
lot. Après chaque lot non terminal, `task_sequence_break()` rend la réservation au Governor.
|
||||
|
||||
Le test de crash publie puis interrompt avant checkpoint du curseur, ferme runtime et DB, recharge
|
||||
le checkpoint antérieur et reconstruit le Task Kind. Le parent est revu, son résultat exact est
|
||||
réutilisé, puis le curseur progresse.
|
||||
|
||||
## Cancellation
|
||||
|
||||
Pause et annulation sont coopératives avant chaque parent et entre lots. Une petite estimation
|
||||
OpenCV engagée finit et publie avant l'arrêt ; aucun résultat scientifique CANCELLED n'est créé.
|
||||
Les résultats déjà publiés restent durables.
|
||||
|
||||
## Backend policy
|
||||
|
||||
Un backend n'est transparent pour l'identité que si ses sorties scientifiques sont équivalentes
|
||||
selon le contrat. V1 possède une seule implémentation CPU de production.
|
||||
|
||||
## Core publication and reuse
|
||||
|
||||
Le core charge les métadonnées DB, relâche les mutex internes après chaque API, lit les assets et
|
||||
calcule sans transaction longue, puis appelle une publication Model v1 courte. Une identité exacte
|
||||
VERIFIED ou REJECTED est retournée avant toute lecture Feature/Match et sans appel estimator. Une
|
||||
contrainte concurrente déclenche un unique `find` de l'identité, jamais un overwrite ou une
|
||||
récursion. Changer un paramètre scientifique produit un autre fingerprint et un autre résultat.
|
||||
|
||||
Les tests E2E utilisent le vrai Project DB (migré séquentiellement jusqu'à v15 à
|
||||
l'ouverture), deux Feature Files à 8192 points, des Match Files hashés, le vrai
|
||||
MAGSAC et le Model v1. VERIFIED est rechargé après close/reopen avec modèle et
|
||||
masque bit-identiques ; REJECTED conserve son support et est également réutilisé.
|
||||
The current verifier does not add Essential/Homography model competition.
|
||||
|
||||
## Production algorithm
|
||||
|
||||
UsacParams explicites, sampler uniforme, score MAGSAC, non parallèle et seed locale par appel.
|
||||
Les champs LO et polishing effectifs sont encodés explicitement ; aucun preset enum caché.
|
||||
The production estimator uses explicit OpenCV USAC/MAGSAC parameters rather than a hidden preset.
|
||||
|
||||
## Production parameters
|
||||
Scientific choices include:
|
||||
|
||||
FUNDAMENTAL version 1 ; seuil 1,5 px ; confiance 0,999 ; 5000 itérations ; 16 inliers ; ratio 0,20 ;
|
||||
seed policy 1 ; canonicalisation 1 ; Point2d. Tous les champs scientifiques appartiennent au
|
||||
fingerprint version 1.
|
||||
```text
|
||||
model FUNDAMENTAL
|
||||
algorithm USAC_MAGSAC
|
||||
point representation Point2d
|
||||
threshold 1.5 px
|
||||
confidence 0.999
|
||||
max iterations 5000
|
||||
minimum inlier count 16
|
||||
minimum inlier ratio 0.20
|
||||
sampler uniform
|
||||
score MAGSAC
|
||||
isParallel false
|
||||
LO iterations 5
|
||||
LO sample size 14
|
||||
polisher COV
|
||||
polisher iterations 3
|
||||
```
|
||||
|
||||
## Validation
|
||||
The internal scientific estimator remains serial:
|
||||
|
||||
Gate A couvre corpus, comparaison, seed et repeatability. Gate B couvre fingerprint/seed golden,
|
||||
canonicalisation, mapping bit à bit, frontières d'acceptation, E2E DB, reuse, corruption,
|
||||
publication, 8192 matches et ASan/UBSan. Gate C couvre Task, publication avant curseur et reprise.
|
||||
```text
|
||||
UsacParams::isParallel=false
|
||||
```
|
||||
|
||||
La référence A6000 v3 complète contient 37 805 parents `MATCHED`. Le préflight v3 rejette à zéro
|
||||
9 368 parents avec `N<16`, puis le support distinct A/B rejette 117 parents supplémentaires avec
|
||||
`N>=16`. Les 28 320 autres parents appellent USAC sous leur seed v3 exacte ; ils terminent tous par
|
||||
un modèle ou une absence de modèle, sans exception estimator. Cette exécution démarre une tâche v3
|
||||
neuve et ne reprend ni ne ré-étiquette la tâche v2 historique 1385.
|
||||
Outer Task-level concurrency is separate.
|
||||
|
||||
Gate D a exécuté 1000 parents configurés dans la vraie Task, puis les reprises et variantes de
|
||||
configuration du test : environ 2001 traversées réutilisées en 5,870 s, soit environ 341/s. Ce
|
||||
run valide pagination, checkpoints, Governor et reuse ; il n'est pas une mesure de latence MAGSAC
|
||||
et n'en revendique ni médiane ni p95. Le RSS pic observé est 25 964 Kio pour le processus de test
|
||||
complet. `MemAvailable` passe de 10 702 988 à 10 692 916 Kio ; `pswpin/pswpout` restent 0/0 ; en
|
||||
fin de run, PSI avg10 vaut 0,34 % CPU, 0 % mémoire et 0 % I/O. Le chemin calculé reste couvert par
|
||||
le vrai E2E MAGSAC Gate B et ses bornes, sans campagne scientifique répétée.
|
||||
## Random seed
|
||||
|
||||
TSan couvre core, Task, sequencing et Governor (4/4), avec uniquement la suppression OpenCV
|
||||
existante. Le build CPU-only couvre la suite normale (31/31). La suite normale ne contient ni
|
||||
benchmark lourd ni stress. Le clean build Clang/Clang++ et la campagne normale finale passent
|
||||
32/32 avec ORB Vulkan matériel sur Radeon 780M RADV PHOENIX.
|
||||
The seed is derived locally from immutable scientific input.
|
||||
|
||||
The frozen seed domain is based on:
|
||||
|
||||
```text
|
||||
L3DGVSE1
|
||||
Match File SHA-256
|
||||
verifier parameter fingerprint
|
||||
```
|
||||
|
||||
The resulting seed is supplied to the per-call USAC parameter object.
|
||||
|
||||
Global `cv::theRNG()` mutation is forbidden.
|
||||
|
||||
## Parameter fingerprint
|
||||
|
||||
The verifier fingerprint is SHA-256 over the frozen canonical 84-byte encoding.
|
||||
|
||||
Domain:
|
||||
|
||||
```text
|
||||
L3DGVFP1
|
||||
```
|
||||
|
||||
It explicitly encodes:
|
||||
|
||||
- verifier kind;
|
||||
- verifier version;
|
||||
- algorithm;
|
||||
- threshold;
|
||||
- confidence;
|
||||
- iteration bound;
|
||||
- minimum inlier count;
|
||||
- minimum inlier ratio;
|
||||
- seed-policy version;
|
||||
- canonicalization version;
|
||||
- Point2d selection;
|
||||
- sampler;
|
||||
- score;
|
||||
- `isParallel`;
|
||||
- LO settings;
|
||||
- neighbor mode;
|
||||
- polisher settings;
|
||||
- reserved zero byte.
|
||||
|
||||
Integers are little-endian.
|
||||
|
||||
Binary64 values use explicit IEEE-754 bits encoded little-endian.
|
||||
|
||||
No C struct dump participates.
|
||||
|
||||
Current v3 production fingerprint:
|
||||
|
||||
```text
|
||||
6944a471d611d8ffc59dac7cf15a5b79b97e2371d4c51785c477d68c1577f74c
|
||||
```
|
||||
|
||||
Historical v1/v2 fingerprints remain distinct and valid.
|
||||
|
||||
## Acceptance
|
||||
|
||||
A candidate is verified only when the frozen support policy is satisfied:
|
||||
|
||||
```text
|
||||
inlier_count >= 16
|
||||
inlier_count / match_count >= 0.20
|
||||
```
|
||||
|
||||
Otherwise the result is scientific `GEOMETRIC_REJECTED`.
|
||||
|
||||
A scientific rejection is a completed valid result.
|
||||
|
||||
Runtime failure publishes no scientific GVR.
|
||||
|
||||
## Fundamental canonicalization
|
||||
|
||||
The nine coefficients must be finite.
|
||||
|
||||
Frobenius norm is computed robustly; zero norm is rejected.
|
||||
|
||||
The pivot is the first row-major coefficient having the strictly greatest absolute value.
|
||||
|
||||
The matrix is normalized and signed so that pivot is positive.
|
||||
|
||||
Signed zeros are normalized to `+0.0`.
|
||||
|
||||
No production SVD/rank-2 post-projection is performed by this verifier version.
|
||||
|
||||
## Inlier mask
|
||||
|
||||
OpenCV output mask is validated for type, length and values.
|
||||
|
||||
It is converted without reordering to the persistent LSB-first GVR bitset.
|
||||
|
||||
Padding bits are zero and popcount matches `inlier_count`.
|
||||
|
||||
## Runtime failure versus rejection
|
||||
|
||||
Scientific rejection includes valid cases such as:
|
||||
|
||||
- insufficient canonical support;
|
||||
- v3 impossible acceptance support;
|
||||
- estimator returns no acceptable model;
|
||||
- inlier support below frozen acceptance.
|
||||
|
||||
Runtime/input failure includes:
|
||||
|
||||
- missing/corrupt Match or Feature asset;
|
||||
- out-of-range Feature index;
|
||||
- malformed estimator mask;
|
||||
- non-finite invalid model;
|
||||
- OOM;
|
||||
- unexpected OpenCV exception;
|
||||
- Project DB publication failure.
|
||||
|
||||
Runtime failure publishes no fake rejected result.
|
||||
|
||||
## Resource bounds
|
||||
|
||||
One scientific atomic item is one Match Result with at most 8192 correspondences.
|
||||
|
||||
Current outer-parallel Task reservation uses approximately:
|
||||
|
||||
```text
|
||||
8 MiB per admitted parent
|
||||
batch/window <= 16
|
||||
useful CPU participants <= 8
|
||||
GPU 0
|
||||
```
|
||||
|
||||
The window is safe to 16; CPU8 is the retained useful maximum from measurement.
|
||||
|
||||
These are operational resource values, not fingerprint fields.
|
||||
|
||||
No descriptors or dense A-by-B matrix are loaded by the normal verifier.
|
||||
|
||||
## Outer parallel execution
|
||||
|
||||
Current validated shape:
|
||||
|
||||
```text
|
||||
one admitted owner Task
|
||||
-> bounded independent Match Result preparation
|
||||
-> up to admitted CPU participants
|
||||
-> join
|
||||
-> owner publishes canonical contiguous prefix
|
||||
-> owner advances cursor
|
||||
-> checkpoint
|
||||
-> sequence_break
|
||||
```
|
||||
|
||||
Participant preparation writes no GVR rows.
|
||||
|
||||
Only the owner publishes after join.
|
||||
|
||||
This preserves deterministic parent ordering and restart semantics.
|
||||
|
||||
The inner USAC solver remains scientifically serial.
|
||||
|
||||
## GPU policy
|
||||
|
||||
Current classification:
|
||||
|
||||
```text
|
||||
GEOMETRIC_VERIFIER_GPU=NOT_JUSTIFIED
|
||||
```
|
||||
|
||||
Measured CPU units remained small enough that no production GPU backend was justified.
|
||||
|
||||
No Vulkan/OpenCL/CUDA verifier backend is currently part of the production identity.
|
||||
|
||||
CPU outer parallelism remains valid and should not be disabled merely because GPU is rejected.
|
||||
|
||||
## Durable Task
|
||||
|
||||
Task Kind:
|
||||
|
||||
```text
|
||||
geometric_verifier.run/1
|
||||
```
|
||||
|
||||
Project DB v13 adds the typed task payload required because generic checkpoint v1 does not contain the
|
||||
scientific verifier parameters/cursor.
|
||||
|
||||
Durable payload includes the immutable scientific configuration and:
|
||||
|
||||
```text
|
||||
after_match_result_id
|
||||
```
|
||||
|
||||
The fingerprint is revalidated on reconstruction.
|
||||
|
||||
The durable payload does not contain:
|
||||
|
||||
- `cv::Mat`;
|
||||
- RNG engine state;
|
||||
- Governor feedback;
|
||||
- hardware identity;
|
||||
- CPU mask;
|
||||
- transient buffers.
|
||||
|
||||
## Restart
|
||||
|
||||
Pagination follows increasing `match_result_id` and does not assume contiguous IDs.
|
||||
|
||||
A GVR is published before the durable cursor advances.
|
||||
|
||||
Crash after publication but before checkpoint may replay that parent.
|
||||
|
||||
Restart finds/revalidates the exact GVR identity and reuses it.
|
||||
|
||||
No overwrite is performed.
|
||||
|
||||
## Historical resource normalization
|
||||
|
||||
The exact historical serial resource shape remains accepted only for restart compatibility.
|
||||
|
||||
It may be normalized ephemerally to the current outer-parallel capability.
|
||||
|
||||
The original checkpoint is not rewritten.
|
||||
|
||||
Neighboring resource shapes are rejected rather than guessed.
|
||||
|
||||
## Real S21 GV v3
|
||||
|
||||
`REAL_S21_GV_V3=PASS/FROZEN` au 31 août 2026. La preuve part d'une copie reflink entière du projet
|
||||
Matcher S21 gelé à 2 826 Feature Sets, 172 741 Candidate Pairs et 172 741 Match Results. Le SHA-256
|
||||
DB source vaut avant et après
|
||||
`9f5ee4877bca25db3d4929be06d8e6ff4fa1c29e11249e4125266a833f09f3e0` ; le projet source n'est
|
||||
jamais ouvert en écriture. La copie de travail reprend exclusivement à la frontière Match Result,
|
||||
par la Task, la Queue et le Resource Governor AUTO de production, puis s'arrête avant Track
|
||||
Builder. Avant GV, la Task Matcher 2831 est `COMPLETE`, progression 100,
|
||||
`sequence_count=21629`, curseur 172 741 ; son checkpoint SHA-256 vaut
|
||||
`636f4f4a20f27308d90142c495c9f6ffc04b4c0dfcca0fdc75cfeb5366ab50b1`. Le projet contient alors
|
||||
zéro GVR, Track Set, Track ou Sparse Reconstruction.
|
||||
Retained S21 proof:
|
||||
|
||||
La Task 2832 consomme le curseur complet de 172 741 Match Results. Parmi eux, 172 275 parents
|
||||
`MATCHED` applicables produisent exactement 172 275 identités v3 : 24 065
|
||||
`GEOMETRIC_VERIFIED` et 148 210 `GEOMETRIC_REJECTED`. Les 466 autres Match Results sont traversés
|
||||
sans GVR conformément au contrat. Le fingerprint est
|
||||
`6944a471d611d8ffc59dac7cf15a5b79b97e2371d4c51785c477d68c1577f74c`. La Task termine
|
||||
`COMPLETE`, progression 100, `sequence_count=21592`, curseur 172 741 et zéro mapping dupliqué. Son
|
||||
checkpoint final SHA-256 vaut
|
||||
`3e6bed97cee9c96f229ef19a3c905d19d4693cdbf43b30319edcdeed6c4e378e`. Le wall propre à
|
||||
l'enqueue/attente GV vaut 3 221,757986763 s ; le wall du runner incluant l'audit intégral amont
|
||||
vaut 3 252,89 s.
|
||||
```text
|
||||
REAL_S21_GV_V3=PASS/FROZEN
|
||||
|
||||
L'audit relit les 172 741 mappings Candidate/Match et les 172 275 Match Files : SHA, taille,
|
||||
header, entrées, ordre et curseur Matcher restent valides. Le digest `L3DMRD1` demeure
|
||||
`e5128a2e599ff593c4f79850e067254b1f249d19e8480a44973306b1af250f70`. Feature Sets, Candidate
|
||||
Pairs et Match Results gardent respectivement 2 826, 172 741 et 172 741 lignes ; aucune Task
|
||||
Feature, Candidate ou Matcher n'est rejouée. Track Set, Track, Track Builder Task, Sparse SfM Task
|
||||
et Sparse Reconstruction restent tous à zéro.
|
||||
Match Results 172,741
|
||||
Applicable MATCHED 172,275
|
||||
Verified 24,065
|
||||
Rejected 148,210
|
||||
non-applicable 466
|
||||
duplicate mapping 0
|
||||
```
|
||||
|
||||
Le Governor enregistre 21 593 admissions, exclusivement backend fixe, sans changement de contrat.
|
||||
Le dernier contrat est GREEN, CPU 1, GPU 0, I/O 1, batch 8, hôte 4 Mio et GPU 0. Les masques sont
|
||||
compute `0-5,8-13` et reserve `6,7,14,15`. Sur l'échantillonnage coalescé de 21 590 changements,
|
||||
le minimum `MemAvailable` vaut 8 907 714 560 octets, le RSS/HWM processus maximal 45 690 880
|
||||
octets, le PSI mémoire maximal 0,90 %, le PSI I/O maximal 50,17 % et les deltas swap-in/out sont
|
||||
0/0. Les réserves 3 Gio/2 Gio alors en vigueur pour ce run historique restent
|
||||
respectées ; aucune voie GPU GV n'est créée.
|
||||
The v3 fingerprint is the current production fingerprint.
|
||||
|
||||
La seconde reprise complète crée la Task 2833, traverse le même curseur et crée zéro GVR. Les
|
||||
172 275 lignes avant/après sont égales sur toutes leurs colonnes par `EXCEPT` dans les deux sens,
|
||||
avec zéro différence, les mêmes IDs 1..172275 et les mêmes comptes accepté/rejeté. Sur une copie
|
||||
reflink séparée, SIGKILL interrompt la Task 2834 après un préfixe checkpointé : l'état durable reste
|
||||
`RUNNING/PENDING`, puis la registry de production reprend cette même Task (`inspected=1`,
|
||||
`resumed=1`) jusqu'à `COMPLETE`, progression 100 et curseur 172 741. L'égalité complète des GVR
|
||||
avec le projet terminé reste zéro différence dans les deux sens ; aucun résultat n'est perdu ou
|
||||
dupliqué et aucun travail amont/aval n'est exécuté.
|
||||
A complete replay produced zero new GVRs.
|
||||
|
||||
Les builds normaux Vulkan et portable sans Vulkan passent. Les 14 tests focalisés GV, Task,
|
||||
checkpoint, Project DB/Project, registry, Queue et Governor passent dans chaque configuration ; le
|
||||
test runner ciblé passe aussi sous ASan/UBSan. `REAL_S21_GV_V3`, `RESTART_IDEMPOTENCE`,
|
||||
`DETERMINISM`, `GOVERNOR_ADMISSION` et `DOWNSTREAM_STOP` sont donc `PASS/FROZEN`. Ce gel porte sur
|
||||
la preuve réelle de la policy v3 déjà gelée ; il ne rouvre ni algorithme, seuil, RNG, fingerprint,
|
||||
Project DB v22, Matcher/Governor v2, Track Builder ou Sparse SfM.
|
||||
A SIGKILL/restart proof resumed the same Task and converged to the same complete GVR set.
|
||||
|
||||
## Maintenance outer-parallel — preuve représentative réelle
|
||||
The run stopped before Track Builder at the original GV-only checkpoint.
|
||||
|
||||
**IMPLEMENTED / VALIDATED / REVIEWED.** La maintenance sépare la préparation
|
||||
scientifique de la publication sans modifier la policy v3. Une préparation
|
||||
valide tout l'input immuable et produit un objet opaque borné ; elle n'écrit
|
||||
jamais Project DB. Après jointure, le callback propriétaire publie ces objets
|
||||
en ordre de `match_result_id`, détruit chacun exactement une fois et checkpoint
|
||||
le seul préfixe contigu. Les pannes de création partielle, calcul, publication,
|
||||
annulation et reprise ne peuvent donc ni publier un suffixe devant un trou, ni
|
||||
laisser un enfant/handle vivant à la libération de réservation.
|
||||
Later S21 Track evidence exists separately.
|
||||
|
||||
Le corpus représentatif réel contient 4 113 parents, dont 4 102 applicables,
|
||||
578 `GEOMETRIC_VERIFIED` et 3 524 `GEOMETRIC_REJECTED`. CPU1/2/4/8/12
|
||||
conservent les mêmes IDs 1..4102, toutes les colonnes scientifiques et le digest
|
||||
`9401ef6168804b6f1d51f4cdf64cd6b33cbebd2934e5294c8feacc87f9c8ce86`.
|
||||
Les walls Task complets sont 67,521078032/48,859141912/39,158236068/
|
||||
35,170176868/34,251675780 s, soit 60,7514/83,9556/104,7545/116,6329/
|
||||
119,7606 parents/s. Le gain CPU8→CPU12 vaut seulement 2,68 %, sous le seuil de
|
||||
5 %. La capacité production est donc CPU utile 8, batch 16 et fenêtre sûre 16.
|
||||
## Real A6000 v3
|
||||
|
||||
Les tests focalisés finaux passent 8/8, les répétitions de stress 60/60,
|
||||
ASan/UBSan 3/3 et TSan 3/3, avec contrôles C17 GCC/Clang. La preuve S21
|
||||
historique ci-dessus reste le run complet CPU1/batch8 acquis ; aucun rerun
|
||||
complet de 3 221 s n'est revendiqué pour cette maintenance bornée. Le manifest
|
||||
retenu de cette preuve a le SHA-256
|
||||
`52a4412299c74050a66d5690122a793c9451c79faf47e32b6e65a5958f804856`.
|
||||
Il compare littéralement les 4 102 GVR applicables — ordre/IDs 1..4102,
|
||||
statut, compteur/masque d'inliers, présence et octets binary64 du modèle — et
|
||||
vérifie intégrité DB, clés étrangères, absence de replay amont et absence de
|
||||
travail Tracks/Sparse. Le run S21 complet acquis couvre déjà la policy v3 et
|
||||
sa persistance FROZEN ; la maintenance ne change que la préparation externe et
|
||||
la publication owner-only. Cette combinaison réelle bornée + tests ciblés de
|
||||
panne/checkpoint/reprise discrimine donc le changement sans payer un second run
|
||||
scientifique intégral ni prétendre l'avoir exécuté.
|
||||
The retained A6000 pre-SfM continuation contains:
|
||||
|
||||
La validation globale fraîche qui englobe ce delta passe aussi dans le graphe
|
||||
Clang portable Vulkan-off 931/931 et sa suite 64/64, puis le graphe Vulkan-on
|
||||
939/939 et sa suite 65/65. Le TSan global reste volontairement Vulkan-disabled
|
||||
et couvre les deux cibles GV dans sa matrice 14/14 plus répétitions ; il utilise
|
||||
uniquement les suppressions externes OpenCV/TBB documentées par le projet.
|
||||
```text
|
||||
Match Results 38,420
|
||||
Applicable GVRs 37,805
|
||||
Verified GVRs 10,952
|
||||
Rejected GVRs 26,853
|
||||
duplicate mappings 0
|
||||
```
|
||||
|
||||
## Out of scope
|
||||
Current fingerprint:
|
||||
|
||||
Tracks, model competition, classification planaire ou faible parallaxe, Essential, calibration,
|
||||
pose, triangulation, bundle adjustment, SfM et Vulkan RANSAC.
|
||||
```text
|
||||
6944a471d611d8ffc59dac7cf15a5b79b97e2371d4c51785c477d68c1577f74c
|
||||
```
|
||||
|
||||
The Task completed the full cursor.
|
||||
|
||||
Restart traversed the cursor and created zero new GVRs.
|
||||
|
||||
The continuation then reused/built the frozen Track Set and stopped before real Sparse SfM.
|
||||
|
||||
```text
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
## Validation boundaries
|
||||
|
||||
The scientific verifier validation covers:
|
||||
|
||||
- canonical fingerprint/seed;
|
||||
- bit-exact mask mapping;
|
||||
- acceptance boundaries;
|
||||
- corruption;
|
||||
- publication/reuse;
|
||||
- maximum Match File cardinality;
|
||||
- restart;
|
||||
- deterministic canonicalization;
|
||||
- outer-parallel owner publication.
|
||||
|
||||
TSan qualification must preserve the external OpenCV/TBB boundary described by the concurrency/global
|
||||
maintenance documents.
|
||||
|
||||
Do not claim Vulkan verifier validation: there is no production Vulkan verifier backend.
|
||||
|
||||
## Summary
|
||||
|
||||
```text
|
||||
CURRENT_GEOMETRIC_VERIFIER=FUNDAMENTAL_V3
|
||||
CURRENT_VERIFIER_VERSION=3
|
||||
CURRENT_VERIFIER_FINGERPRINT=6944a471d611d8ffc59dac7cf15a5b79b97e2371d4c51785c477d68c1577f74c
|
||||
|
||||
GEOMETRIC_VERIFIER_TASK=geometric_verifier.run/1
|
||||
PROJECT_DB_GEOMETRIC_VERIFICATION=v12
|
||||
PROJECT_DB_GEOMETRIC_VERIFIER_TASK=v13
|
||||
|
||||
INNER_USAC_PARALLEL=false
|
||||
OUTER_PARALLEL=VALIDATED
|
||||
USEFUL_CPU_MAX=8
|
||||
SAFE_WINDOW_MAX=16
|
||||
PER_ITEM_RAM=8_MiB
|
||||
GPU=NOT_JUSTIFIED
|
||||
|
||||
REAL_S21_GV_V3=PASS/FROZEN
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
```
|
||||
|
|
|
|||
|
|
@ -1,223 +1,459 @@
|
|||
# Vue d'ensemble de l'architecture Lardon3D
|
||||
# Lardon3D Architecture Overview
|
||||
|
||||
## Finalité et flux global
|
||||
## Purpose
|
||||
|
||||
Lardon3D est un moteur de reconstruction géométrique persistante et incrémentale,
|
||||
piloté par une TUI ncursesw. Le terminal reste le centre de contrôle : il gère
|
||||
les projets, lance les opérations, présente leur progression et permet leur
|
||||
annulation. Le viewer sera un composant graphique séparé mais intégré à
|
||||
l'interface pour un usage confortable sur un seul écran.
|
||||
Lardon3D is a persistent, incremental, resource-aware photogrammetry engine for Linux.
|
||||
|
||||
The TUI is the operational control center for projects, acquisition, Tasks, durable progress,
|
||||
resource state, optical configuration and optional external-storage control. Rich visualization and
|
||||
live acquisition remain separate product areas and must consume validated snapshots rather than
|
||||
mutable worker buffers.
|
||||
|
||||
## Current authority
|
||||
|
||||
```text
|
||||
TUI / Projet
|
||||
↓
|
||||
Task
|
||||
↓
|
||||
Estimate
|
||||
↓
|
||||
Task Queue
|
||||
↓
|
||||
Resource Governor / Reservation
|
||||
↓
|
||||
callback worker admis
|
||||
↓
|
||||
Résultat atomique
|
||||
↓
|
||||
Viewer live
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
CURRENT_PRODUCTION_TASK_KINDS=16
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
REAL_S21_TRACKS=PASS/FROZEN
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
## Composants actuels
|
||||
The current Project DB head is additive:
|
||||
|
||||
```text
|
||||
v22 Selected scientific execution foundation
|
||||
v23 Generic optical-context overlay
|
||||
v24 raw.develop.batch/1 persistence
|
||||
v25 features.extract.batch/1 persistence
|
||||
```
|
||||
|
||||
Earlier schema versions remain valid historical contracts where their documentation says so. Current
|
||||
documentation must not present an older version as the active head merely because that version owns a
|
||||
specific historical scientific or persistence boundary.
|
||||
|
||||
## Global execution model
|
||||
|
||||
```text
|
||||
TUI / Project
|
||||
|
|
||||
v
|
||||
Task
|
||||
|
|
||||
v
|
||||
immutable resource estimate / capability
|
||||
|
|
||||
v
|
||||
bounded Task Queue
|
||||
(one active callback)
|
||||
|
|
||||
v
|
||||
Resource Governor
|
||||
(admission + reservation)
|
||||
|
|
||||
v
|
||||
admitted owner callback
|
||||
|
|
||||
+--> bounded internal participants when justified
|
||||
|
|
||||
v
|
||||
deterministic owner publication
|
||||
|
|
||||
v
|
||||
durable checkpoint / restart boundary
|
||||
|
|
||||
v
|
||||
validated snapshot consumers
|
||||
```
|
||||
|
||||
The Queue owns bounded dispatch and backpressure. The Governor owns production resource admission.
|
||||
The active Task owns its execution state and any bounded internal participants. Scientific stores own
|
||||
their own identities and artifact validation.
|
||||
|
||||
No layer may silently absorb another layer's authority.
|
||||
|
||||
## Canonical resource objective
|
||||
|
||||
Lardon3D first preserves the interactive host reserve required for normal workstation use. Safe and
|
||||
useful capacity beyond that reserve belongs to the active workload.
|
||||
|
||||
The objective is:
|
||||
|
||||
```text
|
||||
MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
```
|
||||
|
||||
This does not mean maximizing utilization percentages for appearance. It means using the largest
|
||||
currently safe and useful CPU, RAM, I/O and validated accelerator capability while preserving
|
||||
scientific identity, deterministic publication, host responsiveness and pressure limits.
|
||||
|
||||
The companion rule is:
|
||||
|
||||
```text
|
||||
SERIALISM_REQUIRES_PROOF
|
||||
```
|
||||
|
||||
Per-item atomicity does not imply cross-item serialization. Ordered or owner-only durable publication
|
||||
does not imply serial preparation. Independent work should expose bounded concurrency when exact
|
||||
science and persistence semantics can be preserved.
|
||||
|
||||
A CPU1 or batch1 path may remain when serialism, a measured scaling knee, memory, I/O, GPU execution,
|
||||
exclusive state or another concrete constraint justifies it. Historical conservative descriptors are
|
||||
not portable performance ceilings.
|
||||
|
||||
On the current validation host, the normal observed outcome is approximately:
|
||||
|
||||
```text
|
||||
16 logical CPUs total
|
||||
4 logical CPUs reserved for interactive host use
|
||||
12 logical CPUs available to the compute pool
|
||||
~3 GiB MemAvailable hard reserve
|
||||
Radeon 780M UMA available to validated and useful GPU backends
|
||||
```
|
||||
|
||||
These values are reference-host observations, not product constants.
|
||||
|
||||
## Current components
|
||||
|
||||
### Project
|
||||
Gestion persistante des projets : création, ouverture, fermeture, structure
|
||||
de répertoires. Chaque projet regroupe configuration, images originales,
|
||||
manifeste, résultats, exports et journaux.
|
||||
|
||||
**Statut :** IMPLEMENTED
|
||||
Persistent project lifecycle, stable identity, directory layout and Project DB ownership.
|
||||
|
||||
### Import
|
||||
Import asynchrone et annulable d'images dans un projet. Copie individuelle
|
||||
des fichiers admissibles et maintenance d'un manifeste cohérent.
|
||||
**Status:** IMPLEMENTED
|
||||
|
||||
**Statut :** IMPLEMENTED
|
||||
### Project Database
|
||||
|
||||
### Import Task
|
||||
Premier type métier persistant. Il s'exécute par lots bornés dans le runtime et
|
||||
la Queue génériques, cible explicitement un ScanSet et peut être reconstruit
|
||||
puis repris.
|
||||
SQLite owns durable logical identities, relations, typed Task payloads, scientific metadata and
|
||||
references to external immutable artifacts.
|
||||
|
||||
**Statut :** IMPLEMENTED
|
||||
The current schema head is v25. The scientific meaning of historical rows remains owned by the
|
||||
versioned contracts that created them.
|
||||
|
||||
### ScanSet et Image Catalog
|
||||
Catalogue SQLite persistant séparant acquisition, image logique et asset
|
||||
physique SHA-256. Les parcours persistants sont paginés ; l'ancien catalogue
|
||||
mémoire depuis `manifest.tsv` reste une façade legacy pour la TUI.
|
||||
**Status:** CURRENT / v25
|
||||
|
||||
**Statut :** IMPLEMENTED
|
||||
### Import and ScanSets
|
||||
|
||||
### Profils optiques et calibrations
|
||||
Import materializes managed immutable assets and logical images under explicit ScanSets. Provenance
|
||||
from external source paths remains distinct from managed asset identity.
|
||||
|
||||
Project DB v23 ajoute un overlay générique distinct pour profils de boîtiers,
|
||||
objectifs électroniques ou manuels, configurations optiques, affectations aux
|
||||
campagnes/Captures et calibrations explicitement compatibles. La migration
|
||||
n'infère ni ne backfill aucune donnée S21/A6000 ; ces appareils restent des
|
||||
preuves de validation, pas des identités produit.
|
||||
**Status:** IMPLEMENTED
|
||||
|
||||
**Statut :** IMPLEMENTED — OVERLAY ADDITIF v23
|
||||
### Capture / Asset Provenance
|
||||
|
||||
### Image View
|
||||
Vues triées et filtrées du catalogue pour la TUI. Ne modifie pas le
|
||||
catalogue, le manifeste ou les images.
|
||||
Capture identity remains distinct from file, Asset, `image_id`, SHA-256, path, Task ID and campaign
|
||||
group ID. RAW and JPEG siblings may belong to one physical Capture without becoming one file or one
|
||||
scientific image identity.
|
||||
|
||||
**Statut :** IMPLEMENTED
|
||||
**Status:** PASS / FROZEN
|
||||
|
||||
### TUI observatoire / centre de contrôle
|
||||
### Bounded acquisition discovery and campaign execution
|
||||
|
||||
Le thread principal possède ncurses, l'entrée et le rendu. Un modèle pur reçoit
|
||||
des copies bornées de l'unique Queue et du Governor, coalescées autour d'une
|
||||
seconde, et présente progression durable, ETA honnête, pipeline, ressources,
|
||||
profils optiques et SSD. Les dimensions validées sont full ≥100×30, compact de
|
||||
référence 72×20, minimum 60×15, puis le fallback `Terminal trop petit`. Les
|
||||
couleurs s'accompagnent toujours de libellés textuels et `F10 SSD` reste visible
|
||||
au minimum supporté.
|
||||
Discovery and planning are bounded and deterministic. Automatic grouping requires the documented
|
||||
strong evidence; otherwise explicit caller confirmation remains `CALLER_EXPLICIT`.
|
||||
|
||||
Ouvrir, fermer ou changer de projet détruit/joint l'unique Queue avant de
|
||||
fermer Project DB, puis recrée une Queue vide et rebranche l'observation. Les
|
||||
ABI historiques Task/Resource/AppState/layout restent inchangées ; les vues
|
||||
riches utilisent des structures et fonctions additives décrites dans
|
||||
[Runtime](runtime.md).
|
||||
Durable campaign execution uses the existing Task, Queue, Governor and Project DB recovery model.
|
||||
|
||||
**Statut :** CURRENT / VALIDATED OPERATIONAL
|
||||
**Status:** PASS / FROZEN
|
||||
|
||||
### Photo Quality Triage
|
||||
|
||||
Quality analysis is an operational selection layer, not a scientific identity. It produces explicit
|
||||
GOOD / SUSPECT / REJECT recommendations and keeps human overrides distinct.
|
||||
|
||||
**Status:** PASS / FROZEN
|
||||
|
||||
### Selected scientific execution
|
||||
|
||||
The selected-execution snapshot binds the retained acquisition groups to explicit scientific
|
||||
representations without inferring identity from paths, filenames or metadata.
|
||||
|
||||
**Status:** PASS / FROZEN
|
||||
|
||||
### Optical profiles and calibration selection
|
||||
|
||||
Project DB v23 provides generic camera-body profiles, lens profiles, optical configurations,
|
||||
campaign/Capture assignments, calibration profiles and explicit compatible calibration selection.
|
||||
|
||||
Electronic metadata aliases are exact. Manual lenses without EXIF are normal. Missing or ambiguous
|
||||
identity remains unresolved rather than guessed.
|
||||
|
||||
**Status:** IMPLEMENTED / VALIDATED
|
||||
|
||||
### RAW development
|
||||
|
||||
The historical `raw.develop/1` path remains valid. Project DB v24 adds
|
||||
`raw.develop.batch/1`, allowing bounded independent RAW preparation with deterministic owner-only
|
||||
publication in selected-item order.
|
||||
|
||||
Per-Capture publication remains atomic while cross-Capture preparation may be concurrent.
|
||||
|
||||
**Status:** IMPLEMENTED / VALIDATED
|
||||
|
||||
### Feature Store
|
||||
Extraction ORB réelle par tâche persistante, Feature Sets logiques et assets
|
||||
binaires content-addressed lisibles par plages bornées.
|
||||
|
||||
**Statut :** IMPLEMENTED
|
||||
Feature Store publishes immutable content-addressed Feature Files and bounded typed readers for ORB,
|
||||
SIFT and RootSIFT data.
|
||||
|
||||
The historical `features.extract/1` task remains valid. Project DB v25 adds
|
||||
`features.extract.batch/1`, which prepares independent selected images with bounded participants and
|
||||
publishes Feature Sets deterministically through the owner callback.
|
||||
|
||||
**Status:** IMPLEMENTED / VALIDATED
|
||||
|
||||
### Visual Index
|
||||
Index de retrieval ORB LSH persistant en segments immuables. Il indexe les
|
||||
Feature Sets par lots bornés et retourne des candidats inter-ScanSets sans
|
||||
matching géométrique.
|
||||
|
||||
**Statut :** IMPLEMENTED
|
||||
The ORB LSH Visual Index is persistent and segmented. It indexes homogeneous Feature Sets in bounded
|
||||
updates and retrieves candidate relationships without performing geometric validation.
|
||||
|
||||
### Task
|
||||
Moteur de tâches avec états, progression, pause/reprise coopérative,
|
||||
annulation, checkpoints et estimations de ressources.
|
||||
|
||||
**Statut :** IMPLEMENTED
|
||||
|
||||
### Task Queue
|
||||
File FIFO avec worker unique, sélection de la première tâche admissible,
|
||||
backpressure et bornage du nombre de tâches en attente.
|
||||
|
||||
**Statut :** IMPLEMENTED
|
||||
|
||||
### Hardware Profile
|
||||
Détection des capacités matérielles statiques : cœurs CPU, RAM, GPU/VRAM.
|
||||
|
||||
**Statut :** IMPLEMENTED
|
||||
|
||||
### Resource Snapshot
|
||||
Capture instantanée des ressources disponibles : RAM libre, charge CPU,
|
||||
VRAM disponible.
|
||||
|
||||
**Statut :** IMPLEMENTED
|
||||
|
||||
### Resource Governor
|
||||
Arbitrage centralisé des budgets (RAM, GPU, CPU, IO), calcul de lots
|
||||
adaptatifs, réservations opaques et historique borné de métriques.
|
||||
|
||||
**Statut :** IMPLEMENTED
|
||||
|
||||
### Contrôleur SSD externe optionnel
|
||||
|
||||
Frontière physique UDisks2 pour la paire de labels `LARDON_SWAP` et
|
||||
`LARDON_SCRATCH`, avec identité Drive+UUID stable, état borné, leases scratch
|
||||
et drain sûr. Il ne formate, ne répare ni ne force jamais l'hôte et ne remplace
|
||||
pas le Resource Governor. La TUI exécute ses actions dans un seul thread
|
||||
joinable et le Governor enregistre l'état physique puis orchestre seul les
|
||||
leases scratch de production. Les seize Task kinds courants n'en consomment
|
||||
aucun ; capacité visible ne signifie donc pas usage.
|
||||
|
||||
**Statut :** CURRENT / VALIDATED OPERATIONAL
|
||||
**Status:** IMPLEMENTED
|
||||
|
||||
### Candidate Pair
|
||||
Sous-système de génération et persistance de paires d'images candidates pour
|
||||
le matching géométrique. Répond uniquement à « quelles paires valent la peine
|
||||
d'être explorées ? » sans validation géométrique.
|
||||
|
||||
**Statut :** IMPLEMENTED
|
||||
Candidate Pair generation answers which image pairs are worth exploring. Scientific pair identity and
|
||||
canonical ordering remain independent from the amount of execution parallelism used to prepare them.
|
||||
|
||||
### Match Result
|
||||
Persistance d'un calcul descriptor-level réussi entre deux Feature Sets
|
||||
liés à une Candidate Pair. Identité déterministe par
|
||||
`(candidate_pair_id, feature_set_id_a, feature_set_id_b, matcher_kind, matcher_version, parameter_fingerprint)`.
|
||||
Validation d'appartenance Feature Set → image. Les correspondances vivent dans
|
||||
le Match Store v1 content-addressed; les échecs restent dans le Task Runtime.
|
||||
Candidate currently uses a bounded coupled CPU/batch ladder because additional CPU cannot exercise
|
||||
additional independent pair work while the admitted item window remains one.
|
||||
|
||||
**Statut :** IMPLEMENTED
|
||||
**Status:** IMPLEMENTED
|
||||
|
||||
### Matcher
|
||||
Matching de descripteurs entre deux Feature Sets via BFMatcher OpenCV
|
||||
(ORB Hamming, SIFT/RootSIFT L2) avec Lowe ratio test. Produit un
|
||||
Match File content-addressed et un Match Result dans Project DB.
|
||||
Déterministe, idempotent, borné.
|
||||
|
||||
**Statut :** IMPLEMENTED
|
||||
Matcher produces deterministic Match Results from Feature Sets.
|
||||
|
||||
ORB supports the validated Vulkan hot path with exact CPU fallback. SIFT and RootSIFT remain CPU
|
||||
OpenCV L2 paths. GPU use is selected only when a backend is both validated and useful.
|
||||
|
||||
**Status:** IMPLEMENTED
|
||||
|
||||
### Geometric Verification
|
||||
|
||||
Geometric Verifier v3 performs Fundamental USAC/MAGSAC verification with bounded restartable Task
|
||||
execution. Scientific solver parallelism remains controlled by its contract while cross-parent
|
||||
preparation can use bounded Governor-admitted participants.
|
||||
|
||||
**Status:** IMPLEMENTED / VALIDATED
|
||||
|
||||
### Track Model and Track Builder
|
||||
|
||||
Track Model v1 and Track Builder publish immutable Track Sets with deterministic identity and bounded
|
||||
recovery semantics.
|
||||
|
||||
Retained real-data evidence includes:
|
||||
|
||||
```text
|
||||
REAL_S21_TRACKS=PASS/FROZEN
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
**Status:** PASS / FROZEN
|
||||
|
||||
### Sparse SfM
|
||||
Les primitives géométriques calibrées Gate C, le noyau incrémental synchrone
|
||||
Gate D et le Bundle Adjustment final par composante Gate E sont implémentés.
|
||||
Gate E traite en mémoire une copie du résultat Gate D immutable, sur CPU avec
|
||||
un thread Ceres, et publie atomiquement chaque candidate acceptée. Il n'intègre
|
||||
ni persistance, ni Task Runtime, ni Resource Governor.
|
||||
|
||||
**Statut :** GATES C/D/E — PASS / FROZEN
|
||||
Calibrated Sparse SfM capability is implemented through Gates C-G:
|
||||
|
||||
Gate F relie ces noyaux au Task Runtime durable, à la Queue/Governor/Reservation
|
||||
et à la publication Project DB v17 atomique et idempotente.
|
||||
- Gate C: calibrated geometric primitives;
|
||||
- Gate D: incremental reconstruction core;
|
||||
- Gate E: final per-component Bundle Adjustment;
|
||||
- Gate F: durable Task orchestration and atomic Project DB publication;
|
||||
- Gate G: Governor admission and resource integration.
|
||||
|
||||
**Statut :** GATE F — PASS / FROZEN. Gate G est **PASS / FROZEN**.
|
||||
Phase H v1 adds incremental enrichment from immutable predecessor snapshots without redefining Gate F
|
||||
scientific identity.
|
||||
|
||||
La phase de pipeline H `incremental_reconstruction.run` v1 enrichit un snapshot
|
||||
publié et produit un nouveau snapshot complet avec provenance prédécesseur dans
|
||||
Project DB v18. Elle reste scientifiquement distincte de Gate F et ne modifie
|
||||
ni F0 ni les contrats Gate D/E.
|
||||
**Status:** C-G PASS / FROZEN; PHASE H V1 PASS / FROZEN
|
||||
|
||||
**Statut :** PHASE H V1 — PASS / FROZEN.
|
||||
Implemented capability must remain distinct from execution on a particular real campaign. The
|
||||
historical S21 and A6000 Engine Bay campaigns remain `CALIBRATION_UNAVAILABLE` for known-calibration
|
||||
Sparse SfM. The retained A6000 proof intentionally stopped before Sparse SfM and Dense/MVS.
|
||||
|
||||
## Résultats et publication live
|
||||
### MVS boundary
|
||||
|
||||
Les traitements fonctionnent par séquences adaptatives : lire un lot borné,
|
||||
calculer, écrire un résultat atomique, libérer la mémoire, puis traiter le
|
||||
lot suivant. La stabilité du système hôte et la réactivité de la TUI ont
|
||||
priorité sur le débit maximal.
|
||||
MVS-M1 provides the validated bounded external OpenMVS boundary and deterministic COLMAP/PLY exchange
|
||||
contracts.
|
||||
|
||||
Le viewer consomme des snapshots de résultats validés et publiés
|
||||
atomiquement. Il ne lit jamais un fichier intermédiaire et ne partage pas
|
||||
directement les buffers de travail d'un worker. Une interruption doit laisser
|
||||
le dernier snapshot validé exploitable et permettre la reprise à une
|
||||
frontière de séquence connue.
|
||||
Durable dense publication, full Dense/MVS orchestration, mesh refinement, texturing and export remain
|
||||
future work.
|
||||
|
||||
## Invariants fondamentaux
|
||||
**Status:** PASS / FROZEN boundary
|
||||
|
||||
- Aucun callback de tâche n'est lancé sans réservation active validée.
|
||||
- La Queue ne décide jamais des ressources.
|
||||
- Le Resource Governor est l'unique propriétaire des budgets.
|
||||
- Les réservations sont libérées exactement une fois.
|
||||
- ncurses appartient exclusivement au thread principal.
|
||||
- Les estimations de ressources sont immuables.
|
||||
- Les buffers et files sont strictement bornés.
|
||||
### Task Runtime
|
||||
|
||||
## Limites actuelles
|
||||
Tasks own lifecycle state, progress, cooperative pause/cancel behavior, sequence boundaries,
|
||||
checkpoints and typed durable reconstruction.
|
||||
|
||||
- File à worker unique avec FIFO stable et bypass des seuls `WAIT` ressources.
|
||||
- Absence de DAG de dépendances.
|
||||
- Absence de priorités.
|
||||
- Absence de pools de workers multiples (CPU/GPU/IO).
|
||||
- La TUI legacy ne sélectionne pas encore explicitement ses ScanSets.
|
||||
- La réconciliation globale des assets/checkpoints orphelins n'est pas implémentée.
|
||||
- La compaction des segments Visual Index n'est pas implémentée.
|
||||
- Viewer et publication live non implémentés.
|
||||
**Status:** IMPLEMENTED
|
||||
|
||||
### Task Queue
|
||||
|
||||
The Queue provides bounded FIFO dispatch with one active callback, stable scanning and resource-WAIT
|
||||
bypass behavior. It does not own resource policy.
|
||||
|
||||
**Status:** IMPLEMENTED
|
||||
|
||||
### Task Kind Registry
|
||||
|
||||
The production registry currently contains 16 Task kinds. Historical documents may legitimately
|
||||
record smaller inventories at their checkpoint.
|
||||
|
||||
**Status:** IMPLEMENTED
|
||||
|
||||
### Hardware Profile and Resource Snapshot
|
||||
|
||||
Hardware Profile detects static host capability. Resource Snapshot captures current observable
|
||||
capacity and pressure signals required for admission.
|
||||
|
||||
Unavailable optional telemetry remains unknown; it must not be fabricated.
|
||||
|
||||
**Status:** IMPLEMENTED
|
||||
|
||||
### Resource Governor
|
||||
|
||||
The Governor is the sole production authority for CPU, RAM, GPU, I/O admission and process-local
|
||||
scratch leases where explicitly supported.
|
||||
|
||||
Pressure may throttle future admission. When pressure clears, useful resources must be eligible to
|
||||
ramp back up; throttling is not a permanent lower ceiling.
|
||||
|
||||
UMA GPU allocations are charged exactly once against host RAM. Swap, zram and external scratch never
|
||||
become admitted RAM.
|
||||
|
||||
**Status:** IMPLEMENTED / VALIDATED
|
||||
|
||||
### Optional external SSD controller
|
||||
|
||||
The UDisks2/GDBus controller owns the reviewed physical lifecycle for the exact
|
||||
`LARDON_SWAP` / `LARDON_SCRATCH` device contract. It is not a second scheduler or Governor.
|
||||
|
||||
The Governor remains the sole production entry for scratch leases. The current 16 production Task
|
||||
kinds have zero authoritative scratch consumption; visible storage capacity is not fabricated usage.
|
||||
|
||||
**Status:** CURRENT / VALIDATED OPERATIONAL
|
||||
|
||||
### TUI observatory and control center
|
||||
|
||||
ncurses input and rendering remain on the main thread. Runtime observation is bounded and coalesced.
|
||||
The TUI exposes project state, Tasks, durable progress, resource state, optical configuration and SSD
|
||||
control.
|
||||
|
||||
Validated layout boundaries are:
|
||||
|
||||
```text
|
||||
full layout >= 100x30
|
||||
reference compact 72x20
|
||||
minimum supported 60x15
|
||||
```
|
||||
|
||||
Below the minimum, only the bounded terminal-too-small fallback is rendered.
|
||||
|
||||
Opening, closing or switching a project is a Queue/DB lifetime boundary: destroy and join the sole
|
||||
Queue first, including finished callbacks, then close Project DB, recreate one empty Queue and rebind
|
||||
observers.
|
||||
|
||||
**Status:** CURRENT / VALIDATED OPERATIONAL
|
||||
|
||||
## Publication and restart model
|
||||
|
||||
Long-running processing follows bounded sequence boundaries:
|
||||
|
||||
```text
|
||||
admit bounded work
|
||||
-> prepare bounded items
|
||||
-> join participants
|
||||
-> publish deterministic atomic results
|
||||
-> persist typed cursor/state
|
||||
-> checkpoint generic Task progress
|
||||
-> release transient buffers
|
||||
-> sequence break / re-admission
|
||||
```
|
||||
|
||||
The exact order varies by Task contract, but durable progress must never claim work that has not
|
||||
reached its authoritative publication boundary.
|
||||
|
||||
A crash may leave generic Task progress behind an already durable immutable result where the
|
||||
documented retry contract permits exact reuse. Recovery must converge from durable identity; it must
|
||||
not guess identity from paths, timestamps, basenames or similar metadata.
|
||||
|
||||
## Viewer and live boundaries
|
||||
|
||||
Viewer, coverage analysis, live localization and capture guidance remain future product areas.
|
||||
|
||||
The architecture requirement is already clear: visual consumers use validated immutable snapshots.
|
||||
They do not access mutable worker buffers, own scientific identity or become a second execution
|
||||
runtime.
|
||||
|
||||
Device-specific acquisition transports such as A6000 HDMI/USB or S21/mobile integration belong at the
|
||||
acquisition adapter boundary. Heavy reconstruction remains PC-side.
|
||||
|
||||
## Core invariants
|
||||
|
||||
- No Task callback executes without a valid active reservation.
|
||||
- Queue/runtime do not own resource policy.
|
||||
- The Resource Governor is the sole production resource authority.
|
||||
- ncurses remains owned by the main thread.
|
||||
- Scientific and operational identities remain distinct.
|
||||
- Task estimates and installed sequence contracts remain immutable for their defined lifetime.
|
||||
- Memory, buffers, queues, files, descriptors, processes, threads and internal participants are bounded.
|
||||
- Per-item atomicity does not imply cross-item serialization.
|
||||
- Owner-only publication does not imply serial preparation.
|
||||
- Validated and useful GPU backends are preferred when eligible and Governor-safe.
|
||||
- UMA GPU memory is charged exactly once against host RAM.
|
||||
- Swap, zram and scratch do not enlarge RAM admission.
|
||||
- Failure and cancellation clean operation-owned transient resources.
|
||||
- Atomic scientific publication never exposes partial results as complete.
|
||||
- FROZEN scientific decisions are not reopened by resource-policy evolution.
|
||||
|
||||
## Current limitations and future scope
|
||||
|
||||
The following remain outside the current implemented execution model:
|
||||
|
||||
- general dependency DAG scheduling;
|
||||
- multiple active Task callbacks / inter-Task worker pools;
|
||||
- general CPU/GPU/I/O worker-pool architecture;
|
||||
- multi-GPU scheduling;
|
||||
- global orphan-artifact reconciliation;
|
||||
- Visual Index compaction / larger-index evolution;
|
||||
- authoritative Task scratch consumers;
|
||||
- durable dense / mesh publication;
|
||||
- full Dense/MVS orchestration;
|
||||
- mesh refinement, texturing and final export workflow;
|
||||
- viewer;
|
||||
- offline coverage analysis;
|
||||
- suggested supplementary viewpoints;
|
||||
- live camera localization;
|
||||
- live coverage overlay;
|
||||
- A6000/S21 live acquisition integration;
|
||||
- capture guidance;
|
||||
- video ingestion and deterministic keyframe extraction.
|
||||
|
||||
Deferring inter-Task parallelism does not authorize accidental serialization inside the one active
|
||||
Task. Bounded internal concurrency remains the current mechanism for independent work under
|
||||
`SERIALISM_REQUIRES_PROOF`.
|
||||
|
||||
## Current summary
|
||||
|
||||
```text
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
CURRENT_PRODUCTION_TASK_KINDS=16
|
||||
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
|
||||
RAW_BATCH_PATH=raw.develop.batch/1
|
||||
FEATURE_BATCH_PATH=features.extract.batch/1
|
||||
|
||||
REAL_S21_TRACKS=PASS/FROZEN
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
|
||||
SPARSE_SFM_CAPABILITY=C_G_PASS_FROZEN
|
||||
REAL_A6000_SPARSE_SFM_EXECUTED=NO
|
||||
REAL_A6000_DENSE_MVS_EXECUTED=NO
|
||||
|
||||
TUI=CURRENT_VALIDATED_OPERATIONAL
|
||||
VIEWER=FUTURE
|
||||
LIVE_CAPTURE_GUIDANCE=FUTURE
|
||||
```
|
||||
|
||||
Detailed contracts remain owned by the specialized architecture documents. This overview summarizes
|
||||
current repository state and must not replace those normative documents.
|
||||
|
|
|
|||
|
|
@ -1,80 +1,86 @@
|
|||
# Persistance et base de données Lardon3D
|
||||
# Lardon3D Persistence and Project Database
|
||||
|
||||
## Vision
|
||||
## Current authority
|
||||
|
||||
Lardon3D stocke les métadonnées de reconstruction dans Project DB SQLite,
|
||||
tandis que les données numériques massives restent dans des fichiers/binaires
|
||||
adaptés. Le schéma courant est v23 ; les sections v7 ci-dessous documentent la
|
||||
fondation historique sans prétendre être la tête de migration.
|
||||
The current Project DB schema is **v25**.
|
||||
|
||||
## Principes fondamentaux
|
||||
```text
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
|
||||
### Séparation logique/binaire
|
||||
- État logique, relations, index → base persistante légère
|
||||
- Données numériques massives → fichiers/artefacts binaires adaptés
|
||||
|
||||
### Cycle de publication
|
||||
```
|
||||
lot calculé
|
||||
→ artefact temporaire
|
||||
→ validation
|
||||
→ publication atomique
|
||||
→ transaction de métadonnées
|
||||
→ état READY
|
||||
v22 selected scientific execution foundation PASS/FROZEN
|
||||
v23 generic optical-context overlay IMPLEMENTED/VALIDATED/REVIEWED
|
||||
v24 raw.develop.batch/1 persistence IMPLEMENTED/VALIDATED
|
||||
v25 features.extract.batch/1 persistence IMPLEMENTED/VALIDATED
|
||||
```
|
||||
|
||||
### Règle de reprise
|
||||
Une reprise ne considère jamais un artefact partiellement publié comme valide.
|
||||
Project DB evolves additively. Older schema versions remain valid historical contracts when a
|
||||
section explicitly documents the state published by that version. They must not be rewritten as if
|
||||
they had always contained later overlays.
|
||||
|
||||
## Concepts de domaine
|
||||
The detailed schema, migration ledger and identity contracts are owned by
|
||||
[Project Database](project_database.md). This document owns the persistence model and publication,
|
||||
checkpoint, recovery and artifact-boundary rules.
|
||||
|
||||
Les éléments suivants sont des concepts de domaine, PAS des tables SQL imposées :
|
||||
## Persistence model
|
||||
|
||||
- project
|
||||
- scan_set
|
||||
- image
|
||||
- feature_set
|
||||
- visual_signature
|
||||
- candidate_pair
|
||||
- verified_pair
|
||||
- track
|
||||
- observation
|
||||
- camera
|
||||
- camera_body_profile
|
||||
- lens_profile
|
||||
- optical_configuration
|
||||
- optical_calibration_profile
|
||||
- pose
|
||||
- point3d
|
||||
- reconstruction_layer
|
||||
- measurement
|
||||
- document_source
|
||||
- geometric_constraint
|
||||
- artifact
|
||||
- checkpoint
|
||||
Lardon3D keeps queryable logical state in SQLite Project DB while large numerical payloads remain in
|
||||
bounded external files or artifacts designed for their format.
|
||||
|
||||
## Invariants
|
||||
The split is intentional:
|
||||
|
||||
- Chaque publication est atomique
|
||||
- Les artefacts partiels ne sont jamais considérés comme valides
|
||||
- La reprise commence à la dernière frontière connue
|
||||
- identity, lifecycle state, relations, durable Task payloads and publication metadata -> SQLite;
|
||||
- large descriptors, match payloads, checkpoints and other numerical artifacts -> bounded files;
|
||||
- a file path is a storage locator after identity resolution, never scientific identity by itself.
|
||||
|
||||
## Checkpoint durable de tâche v1
|
||||
Project DB does not serialize runtime-only objects such as mutexes, condition variables, callbacks,
|
||||
userdata pointers, worker threads, Governor reservations or live execution contracts.
|
||||
|
||||
### État durable
|
||||
## Core invariants
|
||||
|
||||
Le modèle durable versionné contient uniquement l'identifiant stable, le nom,
|
||||
l'estimation immuable, l'état observé, l'état de reprise, la progression, le
|
||||
message, les horodatages et le compteur de séquences. Il ne contient aucun gros
|
||||
artefact numérique. Une future version pourra référencer des identifiants
|
||||
d'artefacts publiés et validés sans incorporer leur contenu.
|
||||
- durable publication is explicit and ordered;
|
||||
- partial artifacts are never accepted as complete scientific outputs;
|
||||
- recovery begins from the last durable boundary the owning contract can prove;
|
||||
- SQLite state and external files are not falsely described as one distributed transaction;
|
||||
- immutable scientific identity is not inferred from path, basename, timestamp or operational IDs;
|
||||
- large collections are read and written through bounded interfaces;
|
||||
- retry behavior must converge only from identities already established by the owning contract;
|
||||
- Project DB schema migration never silently creates new scientific meaning for historical rows.
|
||||
|
||||
Les mutex, conditions, callbacks, userdata, workers, gouverneur, réservations et
|
||||
contrats d'exécution sont transitoires et ne sont jamais sérialisés.
|
||||
## Logical and binary publication
|
||||
|
||||
### Normalisation après arrêt de processus
|
||||
The generic publication shape is:
|
||||
|
||||
| État observé | État restauré |
|
||||
```text
|
||||
bounded computation
|
||||
-> operation-owned temporary output
|
||||
-> format/content validation
|
||||
-> atomic file publication where required
|
||||
-> Project DB transaction
|
||||
-> READY / published durable state
|
||||
```
|
||||
|
||||
The exact ordering is owned by the subsystem. Some contracts publish an immutable file before the
|
||||
SQLite transaction and therefore explicitly admit an orphan-file window. Others publish metadata and
|
||||
files through a more specialized ordered protocol. No document may strengthen a subsystem guarantee
|
||||
beyond the actual implementation.
|
||||
|
||||
Failure cleanup may remove only resources owned by the current operation. Shared immutable assets are
|
||||
not deleted merely because a later metadata transaction fails.
|
||||
|
||||
## Durable Task checkpoint v1
|
||||
|
||||
### Durable state
|
||||
|
||||
The versioned Task checkpoint contains bounded logical Task state only. It includes the stable Task
|
||||
identity, name, immutable estimate, observed state, recovery state, progress, message, timestamps and
|
||||
sequence count. It does not embed large scientific artifacts.
|
||||
|
||||
Task kind and typed business payload are separate concerns. The generic checkpoint format v1 does not
|
||||
become a miniature Project DB and does not replace typed Project DB persistence.
|
||||
|
||||
### Process-stop normalization
|
||||
|
||||
| Observed state | Restored state |
|
||||
|---|---|
|
||||
| `TASK_PENDING` | `TASK_PENDING` |
|
||||
| `TASK_RUNNING` | `TASK_PENDING` |
|
||||
|
|
@ -83,162 +89,355 @@ contrats d'exécution sont transitoires et ne sont jamais sérialisés.
|
|||
| `TASK_FAILED` | `TASK_FAILED` |
|
||||
| `TASK_CANCELLED` | `TASK_CANCELLED` |
|
||||
|
||||
Une rupture de séquence n'est pas un état : elle est observée comme
|
||||
`TASK_RUNNING`. Son `sequence_count` est durable, mais la reprise revient à
|
||||
`TASK_PENDING` et exige une nouvelle admission.
|
||||
A sequence break is not a persistent Task state. If execution stopped while a sequence was active,
|
||||
recovery returns the Task to `TASK_PENDING`; its durable sequence count remains retained and a new
|
||||
Governor admission is required before execution resumes.
|
||||
|
||||
### Stockage minimal
|
||||
### Standalone checkpoint file
|
||||
|
||||
Le codec v1 est indépendant de la future Project Database. Le fichier est de
|
||||
taille fixe et bornée, encodé champ par champ, avec magie, version, taille et
|
||||
checksum de payload.
|
||||
La publication écrit un fichier temporaire unique dans le même répertoire,
|
||||
effectue `fsync`, renomme atomiquement puis synchronise le répertoire parent.
|
||||
La lecture distingue absence, corruption, version inconnue et erreur d'I/O.
|
||||
The v1 codec is bounded and field-encoded with explicit magic/version/size/checksum semantics. It does
|
||||
not serialize native structs or native padding.
|
||||
|
||||
La sauvegarde distingue trois frontières :
|
||||
Standalone publication uses a unique temporary file in the same directory, synchronizes the file,
|
||||
renames atomically and then synchronizes the parent directory.
|
||||
|
||||
- avant `rename`, toute erreur retourne `IO_ERROR`, supprime le temporaire et
|
||||
laisse l'ancien checkpoint publié inchangé ;
|
||||
- après un `rename` réussi, le nouveau checkpoint est publié et visible et
|
||||
n'est jamais présenté comme rollbackable ;
|
||||
- si le `fsync` du répertoire échoue après ce `rename`, le résultat est
|
||||
`PUBLISHED_NOT_DURABLE` : le fichier visible est valide, mais sa présence sous
|
||||
ce nom après un crash ou une coupure n'est pas garantie. `OK` garantit que le
|
||||
contenu et l'entrée de répertoire ont tous deux été synchronisés avec succès,
|
||||
sous réserve des garanties fournies par le système de fichiers et le stockage.
|
||||
The result distinguishes three important boundaries:
|
||||
|
||||
Les tailles persistantes sont refusées avant conversion lorsqu'elles dépassent
|
||||
`SIZE_MAX`. Les secondes sont des entiers non signés v1 : les timestamps
|
||||
négatifs ne sont pas sérialisables et une valeur lue doit être représentable
|
||||
par le `time_t` local avant conversion. Le format reste donc lisible entre
|
||||
plateformes uniquement pour les valeurs communes à leurs domaines `size_t` et
|
||||
`time_t`.
|
||||
- before successful rename: failure leaves the previously published checkpoint unchanged;
|
||||
- after successful rename: the new visible checkpoint is published and is not presented as rolled back;
|
||||
- parent-directory sync failure after rename: result is `PUBLISHED_NOT_DURABLE`; the visible file is
|
||||
valid, but name persistence across crash/power loss is not guaranteed by Lardon3D.
|
||||
|
||||
## Project Database v7 — fondation historique
|
||||
`OK` means both content and directory-entry synchronization completed, subject to the guarantees of the
|
||||
filesystem and storage stack.
|
||||
|
||||
SQLite contient l'état logique interrogable et les références aux fichiers ;
|
||||
les checkpoints et artefacts volumineux restent externes. L'enregistrement du
|
||||
résumé de tâche et de sa référence checkpoint est une transaction unique. Un
|
||||
artefact est d'abord publié et vérifié comme fichier régulier, puis seulement
|
||||
marqué `READY` en DB. Le chemin inverse est interdit.
|
||||
Persistent sizes are rejected before conversion if they exceed the local representable domain. The v1
|
||||
timestamp representation likewise requires values representable by the local `time_t` before runtime
|
||||
conversion.
|
||||
|
||||
### Protocole checkpoint projet
|
||||
## Project-owned checkpoint protocol
|
||||
|
||||
Le protocole réel n'est pas une transaction distribuée :
|
||||
The project checkpoint protocol coordinates a Task file and SQLite metadata without claiming a true
|
||||
filesystem+SQLite transaction.
|
||||
|
||||
1. capture locale du snapshot sous le mutex de tâche puis déverrouillage ;
|
||||
2. publication atomique du fichier sous
|
||||
`.lardon3d/checkpoints/<task_id>.chk` ;
|
||||
3. transaction SQLite sur `tasks` et `checkpoints` avec chemin relatif.
|
||||
The current project-level ordering is:
|
||||
|
||||
Une erreur avant publication ne modifie pas la DB. `PUBLISHED_NOT_DURABLE` est
|
||||
conservé comme tel en DB. Si la publication réussit puis que SQLite retourne
|
||||
`BUSY` ou une erreur, le fichier valide reste sur disque, la DB conserve son
|
||||
ancienne vérité et le nouveau fichier est un orphelin à réconcilier plus tard.
|
||||
Il n'est pas supprimé et aucune atomicité FS+SQLite n'est revendiquée.
|
||||
```text
|
||||
capture bounded Task snapshot
|
||||
-> publish .chk.next
|
||||
-> record Task/checkpoint summary in SQLite
|
||||
-> promote .chk.next to canonical .chk under .chk.lock
|
||||
```
|
||||
|
||||
L'inventaire distingue checkpoint récupérable durable, récupérable mais publié
|
||||
non durable, absent, invalide, version inconnue et erreur d'I/O. Aucune réparation
|
||||
ou suppression silencieuse n'est effectuée.
|
||||
The canonical location is:
|
||||
|
||||
Le format checkpoint reste en version 1 et ne contient pas de `task_kind`. Le
|
||||
schéma SQLite v4 conserve `task_kind` et `task_kind_version` dans le résumé
|
||||
logique interrogable. La migration v1→v2 laisse ces deux colonnes à `NULL` : une
|
||||
tâche legacy reste inspectable mais ne peut pas être reconstruite ou resoumise.
|
||||
Un kind inconnu ou une version non supportée est diagnostiqué sans exécuter de
|
||||
code.
|
||||
```text
|
||||
.lardon3d/checkpoints/<task_id>.chk
|
||||
```
|
||||
|
||||
## Statut
|
||||
The advisory `.chk.lock` exists only as process synchronization; it is not recovery data.
|
||||
|
||||
**IMPLEMENTED** — modèle durable, codec v1, lecture validée, publication
|
||||
atomique et restauration sûre d'une tâche isolée.
|
||||
Recovery obtains the lock, reloads the relevant Project DB record because it may have changed while
|
||||
waiting, and then selects a codec/version-valid checkpoint whose stored summary matches the Project DB
|
||||
summary exactly for the fields Project DB owns.
|
||||
|
||||
**IMPLEMENTED** — Project Database v7 pour identité, tâches typées, ScanSets,
|
||||
images logiques, Feature Sets/assets SHA-256, Visual Index segmenté,
|
||||
checkpoints et artefacts génériques.
|
||||
A valid canonical `.chk` has priority. A valid matching `.chk.next` may be promoted when the canonical
|
||||
file does not match. Stale or corrupt `.next` files never override a valid canonical checkpoint.
|
||||
|
||||
**IMPLEMENTED** — registry statique bornée et reconstruction explicite avec
|
||||
ownership du userdata.
|
||||
A missing, corrupt, future-version or summary-mismatched checkpoint makes that Task non-recoverable; it
|
||||
does not make unrelated project state invalid.
|
||||
|
||||
**IMPLEMENTED** — API projet de sauvegarde fichier+DB et inventaire validé au
|
||||
redémarrage.
|
||||
## Project DB foundation
|
||||
|
||||
**IMPLEMENTED** — `import.images` persiste son chemin source absolu et son
|
||||
`scanset_id`, puis publie un checkpoint après chaque lot validé. Le catalogue
|
||||
SQLite rend le rejeu idempotent à la granularité du contenu dans un ScanSet.
|
||||
Project DB is SQLite with explicit schema versioning and sequential transactional migrations.
|
||||
|
||||
Le chemin source absolu est l'intention durable v1 : il doit rester accessible
|
||||
après redémarrage et un projet déplacé ne rend pas une source externe portable.
|
||||
Une source absente ou devenue non-répertoire fait échouer proprement la
|
||||
reconstruction. Après import terminé, l'image dépend de l'asset géré, plus de la
|
||||
source. Le SHA-256 est calculé pendant la copie avec un tampon fixe de 64 Kio.
|
||||
L'asset est publié sans écrasement sous
|
||||
`assets/images/<prefix>/<sha256>`, puis seulement enregistré `READY` dans une
|
||||
transaction SQLite. Un asset concurrent déjà présent n'est adopté qu'après
|
||||
rehash complet et vérification de taille. Si SQLite échoue après publication,
|
||||
le fichier reste orphelin pour une future réconciliation ; aucune transaction
|
||||
FS+SQLite n'est revendiquée.
|
||||
The retained configuration uses:
|
||||
|
||||
Les identités publiées `scanset_id`, `image_id` et `asset_id` utilisent les
|
||||
séquences SQLite `AUTOINCREMENT` : une valeur issue d'une transaction validée
|
||||
n'est jamais réattribuée à un autre objet, même après suppression de la ligne.
|
||||
Une valeur réservée par une transaction annulée n'est pas une identité publiée.
|
||||
```text
|
||||
foreign_keys=ON
|
||||
journal_mode=DELETE
|
||||
synchronous=FULL
|
||||
busy_timeout=5000
|
||||
```
|
||||
|
||||
`manifest.tsv` reste supporté par l'ancien chemin d'import/catalogue. Le chemin
|
||||
persistant entretient une projection best-effort par hardlinks pour la TUI,
|
||||
mais SQLite est le commit logique canonique : la correction de la reprise ne
|
||||
dépend plus de l'ordre de `readdir()` ni du nom de fichier.
|
||||
`DELETE` journal mode matches the current single-owner Project DB model and avoids persistent WAL/SHM
|
||||
files. The timeout bounds waiting on an external lock.
|
||||
|
||||
La migration v3 vers v4 ne transforme pas les lignes du manifeste historique
|
||||
en images cataloguées : elles ne contiennent pas toutes les preuves exigées par
|
||||
le modèle v4. Le marqueur durable `legacy_image_catalog_pending` rend cette
|
||||
situation visible. Une tâche récupérable peut repeupler le catalogue par rejeu
|
||||
si sa source existe encore ; sinon les fichiers et le manifeste restent une
|
||||
projection legacy, explicitement non cataloguée. Une tâche v3 déjà terminée
|
||||
n'est pas rejouée automatiquement.
|
||||
The project identity is duplicated intentionally across `project.ini` and the `project` table and must
|
||||
match. A divergence is an error, not an opportunity to invent a new identity.
|
||||
|
||||
**IMPLEMENTED** — reprise automatique sélective à l'ouverture : pagination
|
||||
bornée, validation checkpoint/kind, reconstruction production et enqueue sans
|
||||
claim persistant supplémentaire.
|
||||
Published catalog and scientific IDs use SQLite integer identities according to their owning schema.
|
||||
Where `AUTOINCREMENT` is part of the contract, a committed published identity is not later reused for a
|
||||
different object. An ID allocated only inside a rolled-back transaction is not a published identity.
|
||||
|
||||
Les records sont parcourus par task ID croissant. Un checkpoint
|
||||
`PUBLISHED_NOT_DURABLE` présent, valide et cohérent peut être repris ; le résumé
|
||||
conserve cet avertissement jusqu'au prochain checkpoint durable. Une tâche
|
||||
terminale n'appartient pas à la requête de reprise.
|
||||
## Historical Project DB v7 foundation
|
||||
|
||||
**NOT_YET_WIRED** — réconciliation des fichiers orphelins et retry piloté par
|
||||
l'utilisateur pour les sources indisponibles. Les checkpoints existants sont
|
||||
kind-owned aux frontières métier ; aucun timer générique ne peut les remplacer.
|
||||
Project DB v7 is the historical persistent project/runtime foundation. It covers the durable project
|
||||
identity, Tasks, checkpoint references, generic artifacts, ScanSets, logical images, image assets,
|
||||
Feature Store metadata and the segmented Visual Index foundation.
|
||||
|
||||
**NOT_YET_WIRED** — migration de la vue TUI en mémoire vers la pagination
|
||||
SQLite, scrub des assets et réconciliation globale des orphelins.
|
||||
Important v7-era persistence rules remain current unless a later contract explicitly supersedes them:
|
||||
|
||||
**IMPLEMENTED** — Feature Store externe versionné, immutable, borné et relié
|
||||
transactionnellement à ses métadonnées SQLite après publication.
|
||||
- Task summary plus checkpoint reference is a single SQLite transaction;
|
||||
- large checkpoint and scientific files remain external;
|
||||
- an artifact file is published and validated before Project DB marks it `READY`;
|
||||
- Project DB stores bounded metadata and references, not large descriptor/posting payloads;
|
||||
- `AUTOINCREMENT` is used where published catalog/scientific IDs must not be recycled after committed
|
||||
deletion;
|
||||
- the persistent import source path records durable execution intent, while imported scientific data
|
||||
ultimately depends on the managed immutable asset rather than the original external source path.
|
||||
|
||||
**IMPLEMENTED** — Visual Index externe segmenté, memberships transactionnels
|
||||
et tâche `visual_index.update` récupérable.
|
||||
### Generic artifact orphan window
|
||||
|
||||
**IMPLEMENTED** — migrations additives et séquentielles jusqu'à Project DB
|
||||
v23. Les versions v16 à v22 restent l'histoire scientifique et de persistance
|
||||
gelée ; v23 ajoute uniquement l'overlay optique générique.
|
||||
For generic file-first publication, a successful file publication followed by SQLite `BUSY` or another
|
||||
SQLite failure leaves a valid orphan file on disk while Project DB retains its previous truth.
|
||||
|
||||
Les neuf relations v23 séparent profils de boîtier et alias, profils d'objectif
|
||||
et alias, configurations optiques, affectations de configuration aux groupes
|
||||
de campagne et aux Captures, profils de calibration et sélection explicite par
|
||||
Capture. Une configuration référence exactement un boîtier et un objectif ; un
|
||||
objectif manuel sans EXIF est normal. La compatibilité d'une
|
||||
calibration est exacte sur la configuration optique et ses dimensions/champs
|
||||
scientifiques. Aucun profil S21, A6000 ou Meike n'est inséré ou déduit par la
|
||||
migration : les tables nouvelles restent vides tant qu'un caller ne fournit
|
||||
pas explicitement les données.
|
||||
That file is not silently deleted. Global orphan reconciliation is a separate maintenance capability.
|
||||
The absence of such reconciliation does not justify pretending the file and SQLite update were atomic.
|
||||
|
||||
La migration v22→v23 est une transaction additive. Elle ne réinterprète ni les
|
||||
Captures, ni les Images, ni les résultats scientifiques historiques. Une copie
|
||||
S21 et une copie A6000 ont atteint v23 avec intégrité et clés étrangères
|
||||
valides, comptes scientifiques inchangés et tables optiques vides. Les détails
|
||||
normatifs sont dans [Project Database](project_database.md).
|
||||
## Import and managed source assets
|
||||
|
||||
**NOT_YET_WIRED** — reprise ordonnée par dépendances/DAG et réconciliation
|
||||
globale des artefacts orphelins.
|
||||
`import.images` persists its source path and ScanSet identity and checkpoints after validated bounded
|
||||
work. Managed image assets are content-addressed by SHA-256 after a complete bounded copy/hash pass.
|
||||
|
||||
A concurrently existing asset is adopted only after the implementation verifies the expected content
|
||||
and size according to the owning contract. SQLite publication follows file publication.
|
||||
|
||||
`manifest.tsv` remains a historical/legacy projection. SQLite is the canonical logical commit for the
|
||||
persistent catalog. The `legacy_image_catalog_pending` marker means legacy data may remain outside the
|
||||
current catalog model; it does not claim that historical manifest rows were silently converted into
|
||||
fully proven catalog identities.
|
||||
|
||||
## Selective Task recovery
|
||||
|
||||
Project open performs bounded selective recovery using Project DB pages and the Task Kind Registry.
|
||||
|
||||
A recoverable Task must have:
|
||||
|
||||
- a valid Project DB Task record;
|
||||
- a supported Task kind and version;
|
||||
- a valid coherent checkpoint boundary required by that kind;
|
||||
- reconstructable typed business persistence where the kind requires it.
|
||||
|
||||
The recovery scan copies records outside the SQLite mutex before business reconstruction and enqueue.
|
||||
A full Queue window stops the scan without mutating the unvisited records.
|
||||
|
||||
Project-level schema, migration and project-identity errors are fatal to opening the project. Per-Task
|
||||
legacy, unknown-kind, unsupported-version, missing-checkpoint, invalid-checkpoint, unavailable-source or
|
||||
reconstruction errors are isolated to the affected Task where the owning contract permits.
|
||||
|
||||
## External scientific artifacts
|
||||
|
||||
### Feature Store
|
||||
|
||||
Feature files are immutable external artifacts with versioned bounded readers. Project DB stores their
|
||||
identity and publication metadata only after the external file is valid according to the Feature Store
|
||||
contract.
|
||||
|
||||
Feature descriptors are not duplicated into SQLite.
|
||||
|
||||
### Visual Index
|
||||
|
||||
Visual Index postings remain in bounded immutable external segments. Project DB stores index identity,
|
||||
segment metadata and memberships. Durable update Tasks retain the bounded cursor required to resume
|
||||
publication without rebuilding already accepted segments.
|
||||
|
||||
### Match and later scientific payloads
|
||||
|
||||
Match payloads and other large scientific representations follow the same architectural principle:
|
||||
SQLite owns durable identity, relations and bounded metadata; specialized external formats own large
|
||||
numerical payloads where the subsystem contract requires them.
|
||||
|
||||
## Project DB v16-v22 retained scientific foundation
|
||||
|
||||
The scientific and persistence contracts introduced through v16-v22 remain historical PASS/FROZEN
|
||||
foundations. Later schema versions are additive overlays and do not reinterpret those rows.
|
||||
|
||||
In particular:
|
||||
|
||||
- v16 publishes the immutable Sparse SfM persistence model;
|
||||
- v17 adds the durable typed Sparse SfM Task payload for Gate F;
|
||||
- v18 adds Phase H v1 incremental-reconstruction identity/persistence;
|
||||
- v19 adds Capture / Asset Provenance v1;
|
||||
- v20 adds durable acquisition-campaign Task persistence;
|
||||
- v21 adds Photo Quality Triage persistence;
|
||||
- v22 adds selected scientific execution, explicit Capture SOURCE-asset mapping and durable selected
|
||||
RAW development persistence.
|
||||
|
||||
Detailed tables, identities and migration invariants are defined in
|
||||
[Project Database](project_database.md).
|
||||
|
||||
## Project DB v23 - generic optical-context overlay
|
||||
|
||||
Project DB v23 is an additive optical-context overlay above the v22 scientific foundation.
|
||||
|
||||
It separates four identities:
|
||||
|
||||
- camera body profile;
|
||||
- lens profile;
|
||||
- optical configuration;
|
||||
- optical calibration profile.
|
||||
|
||||
The migration creates the optical relations empty. It does not inspect EXIF, path, basename, SHA-256,
|
||||
dimensions, device name or historical calibration to backfill identity.
|
||||
|
||||
Electronic metadata aliases, when present, use exact stored identity semantics. A manual lens without
|
||||
EXIF is a normal explicit profile and does not require a fabricated metadata alias.
|
||||
|
||||
Calibration selection is explicit and requires exact compatible optical configuration. No silent
|
||||
interpolation, substitution or inferred calibration identity is introduced by persistence.
|
||||
|
||||
Historical S21/A6000 copies migrated through the optical overlay without changing their existing
|
||||
scientific rows; empty optical tables remain an honest state until explicit data is supplied.
|
||||
|
||||
## Project DB v24 - RAW batch persistence
|
||||
|
||||
Project DB v24 adds only the typed durable relation required by `raw.develop.batch/1`:
|
||||
|
||||
```text
|
||||
raw_development_batch_tasks(task_id, selected_execution_id)
|
||||
```
|
||||
|
||||
The migration is additive and DDL-only. It does not create Capture, Asset, Image or selected-execution
|
||||
identity and does not rewrite historical `raw.develop/1` Tasks.
|
||||
|
||||
The selected execution remains the durable scientific ordering authority. The batch Task may prepare
|
||||
independent RAW items concurrently under Governor admission, but after all participants join, the owner
|
||||
publishes selected representations in deterministic selected-item order.
|
||||
|
||||
The durable ordering remains:
|
||||
|
||||
```text
|
||||
prepare bounded independent RAW items
|
||||
-> join participants
|
||||
-> owner publishes exact selected item
|
||||
-> selected item/cursor commit
|
||||
-> generic Task progress/checkpoint
|
||||
```
|
||||
|
||||
A crash may therefore leave generic Task progress behind already durable selected scientific state; it
|
||||
must never move generic progress ahead of unpublished selected state. Recovery resumes from durable
|
||||
selected-execution identity/cursor and exact already-published representations rather than guessing
|
||||
from paths or files.
|
||||
|
||||
The retained real A6000 run completed this RAW-batch path for all 689 selected RAW representations.
|
||||
|
||||
## Project DB v25 - Feature batch persistence
|
||||
|
||||
Project DB v25 is the current schema head. It adds only the typed durable relation required by
|
||||
`features.extract.batch/1` through `feature_extract_batch_tasks`.
|
||||
|
||||
The v25 row binds one durable Feature-batch Task to:
|
||||
|
||||
- one immutable selected execution;
|
||||
- the monotone selected-item prefix/cursor;
|
||||
- the exact ORB extractor kind/version/parameters/fingerprint domain required by the Task contract.
|
||||
|
||||
The migration is additive and DDL-only. It creates no Feature Set, converts no historical
|
||||
`features.extract/1` Task and infers no Image or Feature Set identity.
|
||||
|
||||
Selected images may be prepared concurrently without SQLite access. After participants join, the owner
|
||||
publishes Feature Sets in selected order and advances the durable Feature cursor only when the exact
|
||||
READY Feature Set is already durable.
|
||||
|
||||
A crash may leave generic Task checkpoint/progress or the Feature cursor behind an immutable Feature
|
||||
Set that was already published. Recovery must revalidate and reuse that exact result; it must not infer
|
||||
a replacement identity.
|
||||
|
||||
The retained real A6000 proof completed the v25 path with 689 Feature Sets and then continued through
|
||||
Visual Index, 38,420 Candidate Pairs, 38,420 Match Results, Geometric Verification and Tracks without
|
||||
replaying acquisition, RAW or Feature work.
|
||||
|
||||
This persistence result does **not** imply that real Sparse SfM or Dense/MVS was executed. Those counts
|
||||
remain zero in `REAL_A6000_PRE_SFM=PASS/FROZEN`.
|
||||
|
||||
## Capture / Asset / Image identity boundary
|
||||
|
||||
Persistence must preserve these distinctions:
|
||||
|
||||
```text
|
||||
Capture != file
|
||||
Capture != Asset
|
||||
Capture != image_id
|
||||
Capture != SHA-256
|
||||
Capture != path
|
||||
Capture != basename
|
||||
Capture != Task ID
|
||||
Capture != campaign group ID
|
||||
```
|
||||
|
||||
`asset_id` identifies a managed immutable asset record. SHA-256 identifies immutable bytes according to
|
||||
the asset contract. `image_id` identifies a scientific image representation. `capture_id` identifies a
|
||||
physical acquisition representation in Project DB. Task and campaign-group IDs remain operational
|
||||
identities.
|
||||
|
||||
Retry and recovery may use an explicit persisted mapping between these domains only where a canonical
|
||||
contract defines it. They may not reconstruct missing identity from coincidental equality or metadata.
|
||||
|
||||
## Acquisition-campaign crash/restart ordering
|
||||
|
||||
For durable campaign execution, the important persistence boundary is conceptually:
|
||||
|
||||
```text
|
||||
S3-E returns capture_id
|
||||
-> persist group_id -> capture_id mapping and campaign cursor
|
||||
-> advance generic Task progress/checkpoint
|
||||
-> next group may execute
|
||||
```
|
||||
|
||||
The retained pre-return S3-E crash window remains intentional. If a Capture is created internally but
|
||||
the process dies before the caller receives and durably retains its `capture_id`, campaign persistence
|
||||
does not guess that identity from path, digest, basename, timestamp, metadata or `image_id`.
|
||||
|
||||
## Concurrency and ownership
|
||||
|
||||
A Project DB connection is serialized by its internal mutex. Each public compound operation owns its
|
||||
entire transaction; a public transaction is not left open across calls.
|
||||
|
||||
External artifact I/O is not performed while holding the Project DB mutex where the owning contract
|
||||
separates those operations. SQLite-owned strings and records are copied into caller-owned bounded
|
||||
storage before returning.
|
||||
|
||||
Closing Project DB concurrently with an active Project DB call is forbidden by owner lifetime rules.
|
||||
Application project/session teardown must first destroy and join the Queue so Task callbacks and leases
|
||||
are finished, then close Project DB.
|
||||
|
||||
## Schema migration discipline
|
||||
|
||||
Project DB migrations are sequential, transactional and additive unless an explicit future human ticket
|
||||
authorizes a different migration.
|
||||
|
||||
Current known sequence:
|
||||
|
||||
```text
|
||||
v1 -> ... -> v22
|
||||
v22 -> v23 generic optical-context overlay
|
||||
v23 -> v24 RAW batch Task persistence
|
||||
v24 -> v25 Feature batch Task persistence
|
||||
```
|
||||
|
||||
A migration failure rolls back both its new schema objects and its schema-version publication marker.
|
||||
A retry therefore starts from the previous complete known schema.
|
||||
|
||||
Future schema versions beyond v25 are rejected by the current implementation and require explicit human
|
||||
authorization before code or documentation may treat them as current.
|
||||
|
||||
## Current status
|
||||
|
||||
```text
|
||||
PERSISTENCE_DOCUMENT=CURRENT
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
|
||||
TASK_CHECKPOINT_V1 IMPLEMENTED
|
||||
PROJECT_CHECKPOINT_PROTOCOL IMPLEMENTED/VALIDATED
|
||||
PROJECT_DB_V7_FOUNDATION IMPLEMENTED
|
||||
PROJECT_DB_V16_TO_V22_FOUNDATION PASS/FROZEN
|
||||
PROJECT_DB_V23_OPTICAL_OVERLAY IMPLEMENTED/VALIDATED/REVIEWED
|
||||
PROJECT_DB_V24_RAW_BATCH IMPLEMENTED/VALIDATED
|
||||
PROJECT_DB_V25_FEATURE_BATCH IMPLEMENTED/VALIDATED
|
||||
REAL_A6000_PRE_SFM PASS/FROZEN
|
||||
```
|
||||
|
||||
Current intentionally unfinished persistence-adjacent work includes global orphan-file reconciliation,
|
||||
asset scrub/reconciliation and a general dependency/DAG recovery model. Those are separate future
|
||||
capabilities; they do not change the current v25 schema authority.
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -1,158 +1,498 @@
|
|||
# Pipeline Feature + Matcher sensible aux ressources
|
||||
# Lardon3D Resource-Aware Pipeline
|
||||
|
||||
## Contrat portable
|
||||
## Status
|
||||
|
||||
Une unité lourde ne démarre qu'avec une réservation active. Elle termine son
|
||||
petit travail courant sans être tuée sur une mesure instantanée, publie le
|
||||
résultat atomiquement, checkpoint, libère ses buffers, puis repasse par le
|
||||
Governor avant la séquence suivante. Les files restent bornées et le swap n'est
|
||||
jamais ajouté au budget de travail.
|
||||
This document describes the current production resource model for the selected pre-SfM pipeline and
|
||||
its resource-sensitive execution paths.
|
||||
|
||||
Le mode normal est interactif : il réserve de la RAM et des threads logiques au
|
||||
desktop. Les signaux d'admission combinent `MemAvailable`, charge CPU, PSI CPU,
|
||||
PSI mémoire, PSI I/O et deltas `pswpin`/`pswpout`. Un seuil dépassé empêche une
|
||||
nouvelle admission ; il ne rompt pas une réservation saine déjà active.
|
||||
```text
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
CURRENT_PRODUCTION_TASK_KINDS=16
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
REAL_S21_TRACKS=PASS/FROZEN
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
Le Governor maintient trois zones. GREEN emploie le lot adapté normal. La zone
|
||||
de prudence RAM entre 3 et 4 Gio, un PSI au seuil ou un premier intervalle avec
|
||||
swap actif produit YELLOW et interdit toute croissance. Deux observations de
|
||||
pression consécutives, ou `MemAvailable` sous la réserve dure de 3 Gio,
|
||||
produisent RED et suspendent toute admission. Le premier snapshot swap établit
|
||||
seulement la baseline.
|
||||
The scientific contracts of Feature Store, Candidate Pair, Matcher, Geometric Verification and Track
|
||||
Builder remain owned by their specialized documents. This document owns the operational view:
|
||||
admission, bounded work, internal concurrency, publication boundaries, pressure response and
|
||||
reference-host measurements.
|
||||
|
||||
La récupération possède deux phases distinctes : trois observations saines
|
||||
font `RED → YELLOW`, puis trois nouvelles observations saines font
|
||||
`YELLOW → GREEN`. Après RED, le plafond de lot reste 1. En GREEN, trois
|
||||
observations saines sont nécessaires à chaque palier `1 → 2 → 4 → 8`. Une
|
||||
nouvelle pression réinitialise cette progression. Cette mémoire est
|
||||
process-local, bornée et protégée par le mutex du Governor.
|
||||
## Canonical execution rule
|
||||
|
||||
Gate G gèle le rafraîchissement initial : lorsqu'il existe du travail PENDING
|
||||
en `WAIT` de ressources, le worker unique de la Task Queue dort au plus 500 ms
|
||||
avant de rescanner la file et de recapturer les ressources. Un signal explicite
|
||||
le réveille plus tôt. Cette cadence ne remplace pas les 50 ms existantes d'une
|
||||
tâche déjà active qui attend sa réadmission à une frontière de séquence.
|
||||
A heavy production unit starts only while an active Resource Governor reservation authorizes it.
|
||||
|
||||
**Gate G — PASS / FROZEN.** Cette réévaluation bornée, la
|
||||
fraîcheur des snapshots et l'identité GPU sélectionnée sont raccordées aux
|
||||
chemins de production existants et leur validation finale est terminée.
|
||||
The normal sequence is:
|
||||
|
||||
Les snapshots emploient `CLOCK_MONOTONIC` et leur âge maximal est 1000 ms. Une
|
||||
capture complète impossible est une erreur opérationnelle, tandis qu'une PSI
|
||||
ou télémétrie swap optionnelle absente reste inconnue. Compute Governor v2
|
||||
observe maintenant le RSS/HWM courant dans un buffer borné, uniquement comme
|
||||
diagnostic du processus : il ne le confond ni avec la réservation Task ni avec
|
||||
un coût attribuable. Le modèle cible un hôte Linux natif non contraint ; cgroups,
|
||||
limites systemd/RLIMIT, multi-GPU, historique/monitoring RSS long terme,
|
||||
redimensionnement d'admission depuis le RSS et consommation Task du scratch
|
||||
restent différés. Le contrôleur SSD optionnel gère le cycle de vie physique ;
|
||||
son état est enregistré auprès du Governor, qui est l'unique orchestrateur des
|
||||
leases scratch de production. Aucun des quatorze kinds actuels ne les consomme,
|
||||
et ni le registre, ni les leases, ni le swap ne créent un budget RAM.
|
||||
```text
|
||||
immutable Task payload
|
||||
-> Governor admission
|
||||
-> bounded preparation / computation
|
||||
-> deterministic owner publication
|
||||
-> durable cursor / checkpoint
|
||||
-> release sequence-local buffers and reservation
|
||||
-> next admission
|
||||
```
|
||||
|
||||
The Queue keeps one active Task callback. That does not require a Task callback itself to perform all
|
||||
independent work serially. Where a Task owns multiple independent units, bounded internal participants
|
||||
may prepare those units concurrently when the scientific and persistence contracts permit it.
|
||||
|
||||
Canonical policy:
|
||||
|
||||
```text
|
||||
MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF
|
||||
```
|
||||
|
||||
Per-item atomicity does not imply cross-item serialization. Owner-only or ordered durable publication
|
||||
does not imply serial preparation.
|
||||
|
||||
A long-running CPU1 or batch1 path is acceptable only when a concrete dependency, measured scaling
|
||||
knee, RAM limit, I/O limit, validated GPU path or another documented operational constraint proves
|
||||
that more concurrency would not be safely useful.
|
||||
|
||||
## Host reserve and pressure model
|
||||
|
||||
Lardon3D preserves the interactive host reserve first, then gives the active workload the safe and
|
||||
useful remaining capacity.
|
||||
|
||||
On the current validation host, the normal observed outcome is approximately:
|
||||
|
||||
```text
|
||||
16 logical CPUs total
|
||||
4 logical CPUs reserved for interactive host use
|
||||
12 logical CPUs available to the compute pool
|
||||
~3 GiB MemAvailable preserved as the hard RAM reserve
|
||||
Radeon 780M UMA available to validated and useful GPU backends
|
||||
```
|
||||
|
||||
These values are evidence for the current host, not portable product constants.
|
||||
|
||||
The Governor may use `MemAvailable`, CPU pressure, memory PSI, I/O PSI, active `pswpin` / `pswpout`
|
||||
deltas, selected-GPU state, and Task-declared fixed, per-participant and transient costs.
|
||||
|
||||
Swap, zram and external scratch never enlarge admitted RAM. UMA GPU allocations are charged exactly
|
||||
once against host RAM.
|
||||
|
||||
Pressure may reduce admission. When pressure clears, safe and useful capacity must be re-admitted
|
||||
rather than leaving the process permanently throttled.
|
||||
|
||||
## Queue and sequence boundaries
|
||||
|
||||
The Task Queue remains bounded and has one active callback.
|
||||
|
||||
When pending work exists but all candidates receive `WAIT`, the Queue worker performs the validated
|
||||
bounded wait and retries normal admission with a fresh snapshot. A running sequential Task may also
|
||||
cross `sequence_break`, which releases the current reservation and requires fresh admission for the
|
||||
next sequence.
|
||||
|
||||
The current architecture separates:
|
||||
|
||||
```text
|
||||
cross-Task dispatch -> one active Queue callback
|
||||
inside-Task independent work -> bounded participants when justified
|
||||
durable publication -> owner-only where required
|
||||
```
|
||||
|
||||
No second scheduler, uncontrolled worker pool, detached-thread system or parallel persistence
|
||||
subsystem is introduced.
|
||||
|
||||
## Current Task inventory
|
||||
|
||||
The production registry contains 16 Task kinds.
|
||||
|
||||
The two additive kinds beyond the historical fourteen-kind maintenance inventory are:
|
||||
|
||||
```text
|
||||
raw.develop.batch/1
|
||||
features.extract.batch/1
|
||||
```
|
||||
|
||||
Historical documents that correctly recorded fourteen kinds at their checkpoint remain historical
|
||||
evidence. Fourteen is not the current production count.
|
||||
|
||||
## Feature Extraction
|
||||
|
||||
ORB est déjà une tâche durable par image : source validée, extraction,
|
||||
publication Feature Store, métadonnées DB, checkpoint terminal et libération du
|
||||
buffer. Le batch vaut donc une image et la granularité de reprise est une image.
|
||||
Le worker unique et la file bornée fournissent la backpressure actuelle.
|
||||
### Scientific atomicity
|
||||
|
||||
Le démarrage configure une baseline OpenCV issue du compute-pool réellement
|
||||
disponible avant la création de Queue. L'unique callback lourd applique ensuite
|
||||
temporairement le compte CPU immuable admis pour sa séquence, dans
|
||||
`1..compute-pool`, et restaure la baseline sur toute
|
||||
sortie, y compris après une mutation suivie d'un échec de vérification. La tâche
|
||||
réserve donc le nombre réellement appliqué au lieu d'annoncer artificiellement
|
||||
un thread pendant qu'une primitive interne en utilise davantage. Une mutation
|
||||
process-wide concurrente par plusieurs workers n'est pas supportée ; Queue
|
||||
conserve un seul callback actif.
|
||||
A Feature Set remains an immutable per-image scientific result.
|
||||
|
||||
The historical production path remains valid:
|
||||
|
||||
```text
|
||||
features.extract/1
|
||||
one image
|
||||
-> feature preparation
|
||||
-> Feature File publication
|
||||
-> Project DB metadata
|
||||
-> terminal Task checkpoint
|
||||
```
|
||||
|
||||
That path remains important for legacy Tasks and exact restart compatibility.
|
||||
|
||||
### Current selected-execution batch path
|
||||
|
||||
Project DB v25 adds the current operational path:
|
||||
|
||||
```text
|
||||
features.extract.batch/1
|
||||
```
|
||||
|
||||
The durable owner is bound to one immutable selected execution, the exact ORB domain and a monotone
|
||||
`next_item_index`. The selected item order remains authoritative.
|
||||
|
||||
The batch path is:
|
||||
|
||||
```text
|
||||
one admitted owner Task
|
||||
-> bounded independent selected-image participants
|
||||
-> each participant prepares one per-image Feature result without SQLite access
|
||||
-> join all admitted participants
|
||||
-> owner validates / reuses the exact READY Feature Set
|
||||
-> owner publishes in selected-item order
|
||||
-> owner advances the durable Feature cursor
|
||||
-> generic Task checkpoint follows
|
||||
```
|
||||
|
||||
A crash may leave the generic checkpoint or Feature-batch cursor behind an already immutable Feature
|
||||
Set. Restart revalidates and reuses that exact result; it does not infer a new identity.
|
||||
|
||||
Per-image Feature atomicity is therefore preserved while cross-image preparation may be concurrent.
|
||||
|
||||
### Feature CPU control
|
||||
|
||||
OpenCV thread count is process-wide state, so it must remain controlled.
|
||||
|
||||
The runtime establishes the validated baseline from the available compute pool before Queue
|
||||
execution. The active heavy callback temporarily applies the admitted count required by its current
|
||||
contract and restores the baseline on every exit path.
|
||||
|
||||
The Queue still has one active callback, so unrelated Tasks do not race process-wide OpenCV thread
|
||||
configuration.
|
||||
|
||||
For `features.extract.batch/1`, cross-image participants are the concurrency mechanism. Additional
|
||||
internal OpenCV threads inside every participant are not multiplied blindly.
|
||||
|
||||
The current coupled Feature-batch admission exists because increasing CPU while the admitted item
|
||||
window remains one cannot exercise additional independent images. CPU and batch may therefore move
|
||||
together for this Task where the validated rung contract requires it.
|
||||
|
||||
This is an explicit operational exception, not a universal Governor rule.
|
||||
|
||||
## RAW selected-execution batch
|
||||
|
||||
Project DB v24 provides `raw.develop.batch/1`.
|
||||
|
||||
Its pattern is analogous at the execution boundary:
|
||||
|
||||
```text
|
||||
one admitted owner Task
|
||||
-> bounded independent RAW participants
|
||||
-> join
|
||||
-> deterministic owner-only publication by selected_item_index
|
||||
-> durable selected-execution cursor advancement
|
||||
```
|
||||
|
||||
Per-Capture RAW publication remains atomic. Cross-Capture preparation need not be serial.
|
||||
|
||||
The retained real A6000 proof completed all 689 selected RAW representations through this path.
|
||||
|
||||
## Visual Index
|
||||
|
||||
Visual Index remains a bounded CPU path.
|
||||
|
||||
Its scientific identity and segment publication are unchanged. Internal parallelism may be used only
|
||||
within its validated contract; no GPU path is promoted merely to make the accelerator busy.
|
||||
|
||||
The current GPU audit rejected Visual Index as a useful production GPU candidate. That rejection is a
|
||||
measured backend decision, not permission to leave useful CPU capacity idle.
|
||||
|
||||
## Candidate Pair generation
|
||||
|
||||
Candidate Pair generation processes bounded Visual Index input and publishes deterministic canonical
|
||||
pairs.
|
||||
|
||||
Current resource behavior distinguishes per-pair scientific identity from cross-item execution.
|
||||
|
||||
The validated Candidate path may use coupled CPU/batch rungs because additional CPU cannot exercise
|
||||
additional independent source work while the admitted item window remains one.
|
||||
|
||||
The current operational model therefore permits:
|
||||
|
||||
```text
|
||||
bounded source/item window
|
||||
+ bounded CPU participants
|
||||
-> deterministic Candidate preparation
|
||||
-> owner publication
|
||||
```
|
||||
|
||||
The exact current memory model and batch limits are owned by [`candidate_pair.md`](candidate_pair.md).
|
||||
A historical CPU1/batch1 descriptor or older window estimate is not a permanent product ceiling.
|
||||
|
||||
## Matcher
|
||||
|
||||
`matcher.run` v1 est une tâche durable. Son unité atomique est une Candidate
|
||||
Pair et son lot vaut 1, 2, 4 ou 8 paires. La tâche page la DB par
|
||||
`candidate_pair_id`, sans supposer des IDs continus, et ne conserve jamais la
|
||||
liste entière. Chaque paire publie immédiatement son Match Result avant que le
|
||||
curseur ne soit avancé en mémoire.
|
||||
`matcher.run` v1 is durable. Its scientific atomic unit is one Candidate Pair.
|
||||
|
||||
Project DB v10 porte le Match Result publié. La migration transactionnelle
|
||||
v10→v11 ajoute uniquement `matcher_tasks`, qui porte la configuration et ce
|
||||
curseur durable.
|
||||
The normal Task pages Project DB by `candidate_pair_id`; it does not assume contiguous IDs and does
|
||||
not retain the whole Candidate Pair set in memory.
|
||||
|
||||
Après chaque lot, la tâche persiste le curseur, checkpoint, puis appelle
|
||||
`lardon3d_task_sequence_break()`. Pause et annulation sont vérifiées avant
|
||||
chaque paire et entre les lots. Un crash après publication mais avant le
|
||||
checkpoint revoit la paire : le Matcher réutilise alors le Match Result et ne
|
||||
recalcule pas les descripteurs.
|
||||
The durable Matcher sequence is:
|
||||
|
||||
```text
|
||||
bounded Candidate Pair page
|
||||
-> match one or more admitted pairs
|
||||
-> publish each exact Match Result
|
||||
-> advance durable cursor
|
||||
-> checkpoint
|
||||
-> sequence_break
|
||||
```
|
||||
|
||||
The current adaptive batch rungs are:
|
||||
|
||||
```text
|
||||
1 -> 2 -> 4 -> 8
|
||||
```
|
||||
|
||||
A published Match Result may precede its generic checkpoint after a crash. Restart finds and reuses
|
||||
the exact durable Match Result rather than recomputing or inventing identity.
|
||||
|
||||
### ORB Vulkan
|
||||
|
||||
The validated ORB Matcher Vulkan backend is GPU-first when eligible.
|
||||
|
||||
The Radeon 780M is UMA, so backend memory is host RAM and is charged once.
|
||||
|
||||
The current production Vulkan contract retains:
|
||||
|
||||
```text
|
||||
normal useful inflight depth = 1
|
||||
validated private safety capacity = 2
|
||||
helpers = 0
|
||||
```
|
||||
|
||||
Depth 2 was measured and rejected as the normal useful setting because its improvement remained below
|
||||
the accepted deadband. It remains a validated private capacity, not a production default.
|
||||
|
||||
The backend preserves canonical Match Result identity and exact CPU fallback. An ineligible pair or
|
||||
backend failure is recomputed completely on CPU; no partial GPU result is published.
|
||||
|
||||
SIFT and RootSIFT remain CPU OpenCV L2 paths. Their Vulkan feasibility work did not reach production
|
||||
eligibility and therefore receives no authoritative production GPU reservation.
|
||||
|
||||
## Geometric Verification
|
||||
|
||||
Project DB v12 stocke un résultat borné à 1024 octets de masque et neuf
|
||||
binary64. `geometric_verifier.run` v1 traite chaque Match Result comme unité
|
||||
scientifique atomique, publie par une transaction courte puis checkpoint son
|
||||
curseur par lots 1..16 avant
|
||||
`task_sequence_break()`. Sa ligne durable appartient à Project DB v13. Le job
|
||||
peut employer jusqu'à huit participants utiles et seize participants sûrs, par
|
||||
lots au plus seize, avec 8 Mio par parent et sans slot GPU. Le propriétaire
|
||||
publie le préfixe canonique dans l'ordre. Admission, pression, lots et
|
||||
slow-start restent exclusivement décidés par Runtime et Governor ; l'USAC
|
||||
scientifique interne conserve `isParallel=false`.
|
||||
The current production verifier lineage is Geometric Verifier v3.
|
||||
|
||||
## GPU et files
|
||||
One Match Result remains the scientific atomic input.
|
||||
|
||||
La Radeon 780M est UMA : toute mémoire GPU compte aussi comme pression RAM.
|
||||
Vulkan 1.4.354 énumère la 780M RADV et une file compute dédiée. Le backend ORB
|
||||
top-2 de production possède un contexte lazy réutilisable et jusqu'à deux jobs
|
||||
privés en vol sur cette file, sans helper hôte. Chaque slot mappé vaut 640 Kio.
|
||||
Le backend part de zéro et retient exactement un slot après une initialisation
|
||||
ou séquence AUTO normale depth 1. Il n'alloue le second que sous un contrat
|
||||
privé de sûreté/benchmark depth 2 déjà admis, puis le libère avant de franchir
|
||||
la prochaine admission depth 1. Les
|
||||
publications restent strictement ordonnées et le fallback CPU reste exact. Le
|
||||
CPU reste le fallback portable si Vulkan est absent, incompatible ou désactivé
|
||||
pour la session.
|
||||
The bounded execution shape is:
|
||||
|
||||
La feasibility SIFT/RootSIFT a borné son prototype Vulkan à 8,125 Mio de
|
||||
payload lazy, mais n'a pas franchi la Gate de production. Le Governor ne réserve
|
||||
donc aucun slot ni budget GPU pour SIFT/RootSIFT ; leur estimation CPU publiée
|
||||
reste inchangée.
|
||||
```text
|
||||
one admitted owner Task
|
||||
-> bounded independent GVR preparation
|
||||
-> join
|
||||
-> owner publishes the canonical prefix in order
|
||||
-> durable cursor / checkpoint
|
||||
```
|
||||
|
||||
## Profil interactif 8845HS mesuré
|
||||
The validated path may use up to eight useful participants and sixteen safe participants, with
|
||||
admitted windows up to sixteen Match Results.
|
||||
|
||||
- budget CPU Lardon3D observé : 12 threads logiques, 4 réservés au desktop ;
|
||||
- réserve dure `MemAvailable` : 3 Gio ;
|
||||
- zone de prudence `MemAvailable` : de 3 à 4 Gio ;
|
||||
- Feature workers : 1 ; batch : 1 image ;
|
||||
- Matcher workers : 1 ; lots adaptatifs 1, 2, 4 ou 8 Candidate Pairs ;
|
||||
- Geometric Verifier : un callback propriétaire, jusqu'à 8 participants utiles,
|
||||
lots adaptatifs jusqu'à 16 Match Results ;
|
||||
- profondeur de la Task Queue : 64 tâches légères, un seul callback actif ;
|
||||
- PSI CPU avg10 : nouvelle admission suspendue à 20 % ;
|
||||
- PSI mémoire avg10 : nouvelle admission suspendue à 1 % ;
|
||||
- PSI I/O avg10 : seuil existant 80 %.
|
||||
The internal USAC/MAGSAC scientific solver keeps its validated `isParallel=false` behavior. Cross-item
|
||||
parallelism belongs outside that per-item solver and does not modify scientific thresholds or identity.
|
||||
|
||||
Le benchmark Matcher 8192 mesure environ 70 ms ORB et 135 ms SIFT à 12 threads,
|
||||
contre 68 ms et 127 ms à 16 threads : le profil interactif abandonne environ
|
||||
3–7 % de latence isolée pour réserver quatre threads logiques au desktop.
|
||||
The retained real A6000 continuation completed:
|
||||
|
||||
Le run soutenu Geometric Verifier traverse environ 2001 parents réutilisés en
|
||||
5,870 s via Task, DB, checkpoints et Governor. Le processus de test culmine à
|
||||
25 964 Kio RSS ; `MemAvailable` reste au-dessus de 10,69 Gio et les compteurs
|
||||
swap restent nuls. PSI avg10 final vaut 0,34 % CPU et 0 % mémoire/I/O. Cette
|
||||
mesure valide le chemin resource-aware et la reprise ; elle ne prétend pas être
|
||||
une distribution de latence estimator-only.
|
||||
```text
|
||||
Match Results 38,420
|
||||
Applicable GVRs 37,805
|
||||
Verified GVRs 10,952
|
||||
Rejected GVRs 26,853
|
||||
```
|
||||
|
||||
## Limites
|
||||
with deterministic restart and no duplicate mappings.
|
||||
|
||||
Les pools multi-workers restent hors périmètre. La capacité CPU portable est
|
||||
désormais bornée par le compute-pool de l'hôte et la limite intrinsèque du kind,
|
||||
jamais par un plafond global 12.
|
||||
SIFT/RootSIFT et Feature Extraction Vulkan restent hors de ce contrat.
|
||||
Swap, zram et disque externe ne sont jamais ajoutés au budget RAM. Aucun chemin
|
||||
scratch/spill Task n'appartient à Gate G core. La commande SSD optionnelle est
|
||||
une intégration opérationnelle additive, pas une admission scientifique.
|
||||
## Track Builder
|
||||
|
||||
La validation B3 du modèle Sparse SfM v16 a utilisé des processus frais, un
|
||||
fixture synthétique de 100 000 landmarks et 500 000 observations, cinq passes
|
||||
de paging sur 50 000 landmarks, et n'a observé ni OOM, ni swap storm, ni dérive
|
||||
RSS applicative. Cette preuve concerne la persistance bornée, pas le solveur.
|
||||
Track Builder consumes an immutable GVR scope.
|
||||
|
||||
Its compact current memory model supersedes the historical rejected S21 envelope that attempted to
|
||||
reserve approximately 18.204 GiB.
|
||||
|
||||
The retained real S21 proof completed:
|
||||
|
||||
```text
|
||||
Tracks 912,447
|
||||
Track observations 2,495,768
|
||||
```
|
||||
|
||||
The retained real A6000 proof completed:
|
||||
|
||||
```text
|
||||
Tracks 130,714
|
||||
Track observations 318,944
|
||||
```
|
||||
|
||||
Both proofs preserve deterministic restart semantics and no authoritative scratch consumption.
|
||||
|
||||
Track Builder resource details remain owned by [`track_builder.md`](track_builder.md).
|
||||
|
||||
## GPU policy
|
||||
|
||||
GPU use is capability- and evidence-driven.
|
||||
|
||||
Canonical rule:
|
||||
|
||||
```text
|
||||
validated AND useful backend -> preferred when eligible and Governor-safe
|
||||
unvalidated backend -> never promoted for utilization appearance
|
||||
measured non-useful backend -> may remain CPU
|
||||
```
|
||||
|
||||
Current examples:
|
||||
|
||||
```text
|
||||
ORB Matcher Vulkan validated and preferred
|
||||
SIFT / RootSIFT Matcher CPU
|
||||
Candidate CPU
|
||||
Feature CPU
|
||||
Visual Index CPU
|
||||
Geometric Verification CPU
|
||||
```
|
||||
|
||||
This inventory may evolve only through measured, validated backend work.
|
||||
|
||||
## External SSD and scratch
|
||||
|
||||
The optional external SSD controller is a physical-lifecycle boundary using the reviewed
|
||||
UDisks2/GDBus contract.
|
||||
|
||||
Its state is registered with the Resource Governor. The Governor is the sole production orchestrator
|
||||
for scratch leases.
|
||||
|
||||
At the current checkpoint:
|
||||
|
||||
```text
|
||||
16 production Task kinds
|
||||
0 authoritative scratch-consuming Task kinds
|
||||
```
|
||||
|
||||
Scratch availability is therefore a capability, not fabricated usage.
|
||||
|
||||
A future Task may use scratch only after an explicit Task-specific contract defines eligibility,
|
||||
lease lifetime, path ownership, cleanup, cancellation, failure handling, capacity accounting and
|
||||
restart behavior.
|
||||
|
||||
Scratch and swap never become RAM.
|
||||
|
||||
## Reference-host measurements
|
||||
|
||||
The following measurements are evidence, not portable constants.
|
||||
|
||||
### CPU reserve
|
||||
|
||||
On the Ryzen 7 8845HS reference host:
|
||||
|
||||
```text
|
||||
16 logical CPUs total
|
||||
~4 logical CPUs reserved for interactive use
|
||||
~12 logical CPUs available to Lardon3D compute
|
||||
```
|
||||
|
||||
Historical isolated Matcher measurements observed only a small additional gain from 12 to 16 logical
|
||||
threads. That observation supports the reference-host reserve; it does not create a global 12-thread
|
||||
product ceiling.
|
||||
|
||||
### Geometric Verifier historical resource-aware run
|
||||
|
||||
A retained resource-aware GV run traversed approximately 2001 reused parents in about 5.870 seconds
|
||||
through Task, Project DB, checkpoints and Governor.
|
||||
|
||||
The test process peaked around 25,964 KiB RSS, `MemAvailable` remained above approximately 10.69 GiB
|
||||
and swap deltas remained zero.
|
||||
|
||||
This measurement validates the bounded path and restart behavior. It is not a universal latency
|
||||
model.
|
||||
|
||||
### A6000 current proof
|
||||
|
||||
The later A6000 proof recorded 2,714 Governor admissions while continuing Match Result -> GV -> Tracks.
|
||||
|
||||
Its final GV admitted window was 16, with the reference-host compute pool at 12 logical CPUs and four
|
||||
reserved for host use. Swap-in/out deltas remained zero.
|
||||
|
||||
The checkpoint is:
|
||||
|
||||
```text
|
||||
real-a6000-pre-sfm-2026-09-02
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
Sparse SfM and Dense/MVS were not executed by this proof.
|
||||
|
||||
## Recovery and deterministic publication
|
||||
|
||||
Parallel preparation never weakens publication ordering.
|
||||
|
||||
For every path where durable order matters:
|
||||
|
||||
```text
|
||||
prepare independent work
|
||||
-> join
|
||||
-> validate exact immutable result
|
||||
-> owner-only deterministic publication
|
||||
-> durable typed cursor
|
||||
-> generic Task checkpoint
|
||||
```
|
||||
|
||||
A checkpoint may lag an already published immutable scientific result. Restart must reuse or validate
|
||||
that result through its exact scientific identity; it must not fabricate a replacement identity.
|
||||
|
||||
Cancellation is cooperative at the Task's documented boundaries. All admitted participants are
|
||||
bounded and joined before sequence completion or owner cleanup.
|
||||
|
||||
## Limits and deferred work
|
||||
|
||||
The following remain outside this document's current production contract:
|
||||
|
||||
- cross-Task worker pools;
|
||||
- multiple simultaneous active Queue callbacks;
|
||||
- general DAG scheduling;
|
||||
- multi-GPU scheduling;
|
||||
- cgroup/systemd constrained-runtime capacity accounting;
|
||||
- generic resource residency/cache management;
|
||||
- authoritative Task scratch consumers;
|
||||
- unvalidated GPU ports;
|
||||
- dense/MVS resource orchestration.
|
||||
|
||||
Deferring cross-Task parallelism is not permission to serialize independent units inside one active
|
||||
Task.
|
||||
|
||||
The Queue/Governor plus bounded internal participants are the current production architecture.
|
||||
|
||||
## Current summary
|
||||
|
||||
```text
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
CURRENT_PRODUCTION_TASK_KINDS=16
|
||||
|
||||
raw.develop.batch/1 CURRENT
|
||||
features.extract.batch/1 CURRENT
|
||||
|
||||
PER_ITEM_ATOMICITY_REQUIRES_CROSS_ITEM_SERIALISM=NO
|
||||
OWNER_ONLY_PUBLICATION_REQUIRES_SERIAL_PREPARATION=NO
|
||||
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
|
||||
VALIDATED_GPU_BACKENDS_PREFERRED_WHEN_USEFUL=YES
|
||||
UMA_ACCOUNTED_ONCE_AGAINST_HOST_RAM=YES
|
||||
SWAP_ZRAM_SCRATCH_AS_RAM=FORBIDDEN
|
||||
|
||||
REAL_S21_TRACKS=PASS/FROZEN
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
SPARSE_SFM_EXECUTED_IN_A6000_PROOF=NO
|
||||
DENSE_MVS_EXECUTED_IN_A6000_PROOF=NO
|
||||
```
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -1,314 +1,505 @@
|
|||
# Exécution et runtime Lardon3D
|
||||
# Lardon3D Runtime
|
||||
|
||||
## Modèle d'exécution
|
||||
|
||||
### Threads
|
||||
- Thread principal : entrée, modèle de vue et rendu TUI ncursesw (exclusif)
|
||||
- Thread worker Queue : exécution sérielle des tâches métier
|
||||
- Participants internes : uniquement ceux du contrat Task admis, joints par le
|
||||
callback propriétaire avant publication
|
||||
- Opération SSD : au plus un thread joinable, uniquement pendant une opération
|
||||
UDisks bornée ; il ne rend rien et ne devient ni Queue ni scheduler
|
||||
|
||||
### Synchronisation
|
||||
- Mutex pour les accès partagés
|
||||
- Variables de condition pour la coordination
|
||||
- Atomicité des opérations critiques
|
||||
|
||||
## Cycle de vie d'une tâche
|
||||
## Status
|
||||
|
||||
```text
|
||||
1. Création (PENDING)
|
||||
2. Soumission à la file
|
||||
3. Sélection FIFO/adaptative par la Queue
|
||||
4. Réservation obligatoire
|
||||
5. Exécution (RUNNING)
|
||||
- Pause/reprise coopérative
|
||||
- Annulation coopérative
|
||||
- Séquences adaptatives
|
||||
6. Complétion (COMPLETED) ou Échec (FAILED)
|
||||
7. Nettoyage des ressources
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
CURRENT_PRODUCTION_TASK_KINDS=16
|
||||
|
||||
TASK_QUEUE_WORKERS=1
|
||||
INTERNAL_PARALLELISM=BOUNDED
|
||||
INTER_TASK_PARALLELISM=NOT_IMPLEMENTED
|
||||
|
||||
RUNTIME_OBSERVER=CURRENT/VALIDATED_OPERATIONAL
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
## Synchronisation
|
||||
The runtime coordinates the main thread, Task Queue, Resource Governor, Project DB lifecycle, bounded
|
||||
Task-internal participants, runtime observation and optional external-SSD controller.
|
||||
|
||||
### Mutex
|
||||
- Protection des données partagées
|
||||
- Accès exclusif aux ressources critiques
|
||||
It does not introduce a second scheduler.
|
||||
|
||||
### Variables de condition
|
||||
- Coordination entre threads
|
||||
- Notification de changement d'état
|
||||
- Attente passive ; timeout borné seulement pour réévaluer un `WAIT` ressources
|
||||
## Thread model
|
||||
|
||||
### Atomicité
|
||||
- Opérations indivisibles
|
||||
- État cohérent garanti
|
||||
### Main thread
|
||||
|
||||
## Gestion des erreurs
|
||||
Owns:
|
||||
|
||||
### Rollback
|
||||
- Rollback des transactions locales avant publication
|
||||
- Nettoyage complet des ressources possédées par l'opération
|
||||
- Une publication fichier réussie suivie d'un échec DB peut laisser un orphelin
|
||||
valide ; aucune transaction distribuée fichier+SQLite n'est revendiquée
|
||||
- input;
|
||||
- ncurses;
|
||||
- TUI model binding;
|
||||
- project open/close orchestration;
|
||||
- bounded polling of runtime/SSD state.
|
||||
|
||||
### Récupération
|
||||
- Reprise à la dernière frontière connue
|
||||
- Ignorance des artefacts partiels
|
||||
- Validation avant publication
|
||||
ncurses remains main-thread-only.
|
||||
|
||||
## Limites actuelles
|
||||
### Task Queue worker
|
||||
|
||||
- Worker Queue unique (pas de pools inter-Tasks multiples)
|
||||
- Pas de parallélisme inter-Tasks ; certains kinds possèdent des participants
|
||||
internes bornés, comptés par leur contrat Governor
|
||||
- Reprise automatique limitée aux tâches indépendantes reconstructibles
|
||||
The single Queue worker owns one active heavy Task callback at a time.
|
||||
|
||||
## Reprise durable
|
||||
A Task callback may create bounded internal participants only when its admitted Task contract permits
|
||||
them.
|
||||
|
||||
Un snapshot ne conserve que l'état logique d'une tâche. `RUNNING` et `PAUSED`
|
||||
sont normalisés vers `PENDING`; aucun worker, callback brut, pointeur, contrat
|
||||
ou réservation n'est restauré. Le propriétaire fournit un nouveau callback et
|
||||
resoumet la tâche. Les états terminaux sont conservés.
|
||||
Those participants join before owner publication.
|
||||
|
||||
`started_at` désigne le début de la tentative d'exécution courante, pas le
|
||||
premier démarrage historique. Un checkpoint `RUNNING` restauré en `PENDING`
|
||||
conserve temporairement l'horodatage de la tentative interrompue pour
|
||||
l'observation ; lors de `lardon3d_task_start()`, `started_at` est remplacé par le
|
||||
nouveau démarrage et `finished_at` est remis à zéro. `finished_at` n'est fixé
|
||||
qu'à la terminaison de cette tentative.
|
||||
### SSD operation thread
|
||||
|
||||
**IMPLEMENTED** — snapshot, codec v1 et restauration isolée.
|
||||
The SSD controller may own at most one bounded joinable operation thread while executing a synchronous
|
||||
UDisks operation.
|
||||
|
||||
**IMPLEMENTED** — l'import `import.images` se sauvegarde à chaque fin de lot et
|
||||
se reconstruit explicitement avec un userdata neuf lié au projet rouvert. Son
|
||||
intention durable contient `source_path + scanset_id`; le hash/copie et la
|
||||
transaction catalogue restent hors mutex Task et hors mutex DB pendant l'I/O.
|
||||
It does not become:
|
||||
|
||||
**IMPLEMENTED** — `project_open()` inventorie par pages de 8, restaure puis
|
||||
resoumet automatiquement les tâches production valides. Il retourne après
|
||||
l'enqueue et n'attend jamais leur terminaison.
|
||||
- a Task worker;
|
||||
- a Queue;
|
||||
- a scheduler;
|
||||
- an ncurses owner.
|
||||
|
||||
L'ordre d'initialisation production est : politique driver, profil matériel,
|
||||
Governor, backend, Queue/worker, contrôleur SSD optionnel et binding
|
||||
Governor, puis TUI. L'ouverture DB/projet et la reprise synchrone sont pilotées
|
||||
ensuite depuis le thread principal. Une fermeture ne peut commencer qu'après le
|
||||
retour de `project_open()`. Le worker peut consommer pendant le scan ; chaque
|
||||
tâche exécutée est néanmoins réadmise normalement.
|
||||
## Task lifecycle
|
||||
|
||||
**NOT_YET_WIRED** — reprise ordonnée par dépendances/DAG. Les kinds de
|
||||
production reconstructibles checkpointent déjà à leurs frontières métier ;
|
||||
aucun timer autosave générique ne doit avancer devant leur publication durable.
|
||||
Conceptual lifecycle:
|
||||
|
||||
**IMPLEMENTED** — reprise sélective des kinds reconstructibles via Project DB,
|
||||
Task Kind Registry et Queue. Les dépendances/DAG restent différées ; il
|
||||
n'existe pas de scheduler global distinct à restaurer.
|
||||
```text
|
||||
create PENDING
|
||||
-> persist typed intent where required
|
||||
-> enqueue
|
||||
-> Queue selects
|
||||
-> Resource Governor admits/reserves
|
||||
-> RUNNING
|
||||
-> bounded sequence work
|
||||
-> Task-specific durable publication
|
||||
-> generic checkpoint
|
||||
-> optional sequence_break/re-admission
|
||||
-> COMPLETED | FAILED | CANCELLED
|
||||
-> terminal callback
|
||||
-> destruction
|
||||
```
|
||||
|
||||
## Accès Project Database
|
||||
Pause/cancel are cooperative.
|
||||
|
||||
**IMPLEMENTED** — une connexion SQLite opaque sérialisée par mutex interne ;
|
||||
les opérations multi-tables sont transactionnelles et bornées.
|
||||
A sequence break is not a Task state.
|
||||
|
||||
**IMPLEMENTED** — le cycle de vie projet ouvre/crée `project.db`, vérifie
|
||||
l'identité et ferme la connexion. Ouvrir, fermer ou changer de projet est une
|
||||
frontière exacte : l'observateur et la vue optique libèrent leurs borrows, puis
|
||||
l'unique Queue est annulée, jointe et détruite, callbacks terminaux inclus,
|
||||
avant la fermeture de Project DB. Une seule Queue vide est ensuite recréée et
|
||||
les observateurs sont rebondés. Il n'existe jamais deux schedulers simultanés.
|
||||
L'historique terminal et l'espace d'IDs Queue sont ainsi propres à la session ;
|
||||
les mêmes Task IDs durables de deux projets restent indépendants et aucun
|
||||
historique fourni n'est affiché lorsqu'aucun projet n'est chargé.
|
||||
## Durable restart
|
||||
|
||||
**IMPLEMENTED** — la registry reconstruit explicitement callback/userdata hors
|
||||
mutex DB pour un kind connu ; elle ne soumet aucune tâche.
|
||||
A generic snapshot stores logical Task state, not live execution machinery.
|
||||
|
||||
**IMPLEMENTED** — la queue accepte un identifiant restauré préassigné s'il
|
||||
n'entre en collision avec aucune tâche connue. L'import production peut donc
|
||||
être reconstruit puis soumis explicitement.
|
||||
On restoration:
|
||||
|
||||
**IMPLEMENTED** — la resoumission automatique utilise la registry production,
|
||||
conserve le task ID et laisse le worker obtenir une nouvelle réservation.
|
||||
Kinds inconnus, tâches legacy, checkpoints invalides et sources absentes ne
|
||||
bloquent pas l'ouverture.
|
||||
```text
|
||||
RUNNING -> PENDING
|
||||
PAUSED -> PENDING
|
||||
```
|
||||
|
||||
**IMPLEMENTED** — `visual_index.update` reprend à la dernière membership
|
||||
commitée. Un segment temporaire n'est jamais visible et un rejeu exclut les
|
||||
Feature Sets déjà membres.
|
||||
Terminal states remain terminal.
|
||||
|
||||
## Durée de vie terminale et fermeture Queue
|
||||
Restart never restores:
|
||||
|
||||
Une Task terminale reste vivante jusqu'au retour complet de son callback
|
||||
terminé. Queue la retire alors de la liste active et la détruit hors de son
|
||||
mutex ; seule une histoire de 64 snapshots reste observable. Les appels déjà
|
||||
enregistrés avant `task_queue_destroy()` sont attendus. Le propriétaire doit
|
||||
empêcher tout nouvel appel dès le début de la destruction, règle nécessaire à
|
||||
toute API C adressée par pointeur brut.
|
||||
- worker thread;
|
||||
- callback pointer;
|
||||
- userdata pointer;
|
||||
- CPU affinity;
|
||||
- live reservation;
|
||||
- GPU handle;
|
||||
- scratch lease;
|
||||
- adaptive feedback history.
|
||||
|
||||
Un callback terminé peut consulter les vues Queue tant que le propriétaire la
|
||||
maintient vivante. Il ne peut pas détruire cette Queue, retirer son propre
|
||||
record ni attendre une opération dépendante de son retour.
|
||||
Task Kind Registry reconstructs fresh runtime binding from exact durable kind/version and typed payload.
|
||||
|
||||
## Observatoire TUI actuel
|
||||
Every resumed Task is re-admitted by the Resource Governor.
|
||||
|
||||
**CURRENT / VALIDATED OPERATIONAL.** Ce statut décrit l'implémentation et ses
|
||||
tests courants. L'audit global qui contient cette frontière est désormais
|
||||
`PASS/FROZEN` après revue indépendante ; le statut TUI reste volontairement
|
||||
opérationnel et n'interdit pas ses évolutions futures sous un ticket distinct.
|
||||
## Project-open recovery
|
||||
|
||||
### Séparation modèle, observation et rendu
|
||||
`project_open()` discovers durable Tasks in bounded pages, validates their checkpoint/typed identity,
|
||||
reconstructs eligible bindings and submits them to the existing Queue.
|
||||
|
||||
Le modèle `tui_model` est pur et testable sans terminal. Le renderer reçoit
|
||||
seulement des copies bornées et n'interroge ni Queue, ni Governor, ni Project
|
||||
DB, ni contrôleur SSD. Toutes les fonctions ncurses, l'entrée clavier et le
|
||||
rendu demeurent sur le thread principal.
|
||||
It returns after enqueue; it does not wait for those Tasks to finish.
|
||||
|
||||
L'observateur runtime emprunte Queue et Governor et conserve une seule copie
|
||||
cohérente. Les captures ordinaires sont coalescées pendant au moins une seconde
|
||||
monotone ; un échec conserve la dernière vue bornée en la marquant stale.
|
||||
Il observe au plus 129 Tasks : les 64 pending possibles, l'unique active et les
|
||||
64 snapshots terminaux récents. L'ordre Queue place le travail vivant du plus
|
||||
récent au plus ancien, puis l'histoire par terminaison décroissante ; une Task
|
||||
active ne peut donc pas être masquée par un vieux préfixe historique. Il
|
||||
n'existe ni scan DB par frame, ni lecture `/proc` volumineuse, ni historique
|
||||
non borné.
|
||||
Unknown kinds, unsupported versions, legacy-untyped Tasks, invalid checkpoints or Task-specific
|
||||
non-reconstructible input remain inspectable and do not cause guessed execution.
|
||||
|
||||
Les ABI historiques restent exactes : `Lardon3DTaskSnapshot`,
|
||||
`Lardon3DResourceSnapshot`, `Lardon3DAppState` et
|
||||
`lardon3d_layout_draw()` ne sont pas étendus en place. Les surfaces additives
|
||||
`Lardon3DTaskObservation`, `lardon3d_task_queue_observe()`,
|
||||
`Lardon3DResourceObservation`, `Lardon3DRuntimeSnapshot` et
|
||||
`lardon3d_layout_draw_runtime()` portent les nouveaux champs. De même,
|
||||
`lardon3d_tui_run()` reste le symbole historique ; l'application utilise
|
||||
`lardon3d_tui_run_with_ssd_operation()` avec un owner SSD conservé hors de
|
||||
`Lardon3DAppState`.
|
||||
Generic dependency/DAG recovery remains unimplemented.
|
||||
|
||||
### Progression et ETA
|
||||
## Initialization order
|
||||
|
||||
Une Task typée publie `completed/total` seulement après son propre commit
|
||||
métier durable. Quand ces compteurs sont connus, la TUI les affiche toujours et
|
||||
en dérive le pourcentage sans utiliser le message ou le nom. Une Task marquée
|
||||
`COMPLETED` avec un préfixe durable incomplet est une erreur d'intégrité
|
||||
visible, jamais 100 %. Quand les comptes typés sont inconnus, le lifecycle peut
|
||||
être terminal mais la progression scientifique reste indéterminée. Le
|
||||
pourcentage générique non typé, lorsqu'il est utile, porte explicitement le
|
||||
libellé runtime.
|
||||
Production startup establishes the safe driver/runtime policy before heavy worker/backend activity.
|
||||
|
||||
Le débit est un EWMA borné. La première observation établit seulement le
|
||||
préfixe de reprise et ne contribue pas au taux ; une reprise de RUNNING remet
|
||||
également la fenêtre temporelle à zéro. Deux intervalles strictement positifs
|
||||
sont nécessaires avant un débit et une ETA connus. Une absence de progrès,
|
||||
une pression Governor, une régression ou une preuve insuffisante produit
|
||||
respectivement `STALLED`, `THROTTLED`, reset ou `INDETERMINATE/CALCULATING`.
|
||||
Seule une complétion cohérente vaut exactement 100 % et ETA zéro ; aucune fausse
|
||||
précision n'est affichée.
|
||||
Conceptually:
|
||||
|
||||
### Pipeline et ressources
|
||||
```text
|
||||
driver policy
|
||||
-> hardware profile
|
||||
-> Resource Governor
|
||||
-> optional backend metadata
|
||||
-> Task Queue / worker
|
||||
-> optional SSD controller + Governor binding
|
||||
-> TUI
|
||||
```
|
||||
|
||||
La synthèse utilise les étapes Acquisition, RAW, Quality, Features, Visual
|
||||
Index, Candidate, Matcher, GV, Tracks, Sparse SfM et future Dense. Les états
|
||||
sont `NOT_READY`, `READY`, `QUEUED`, `RUNNING`, `THROTTLED`, `BLOCKED`,
|
||||
`COMPLETE`, `FAILED` et `NOT_APPLICABLE`. Dense reste explicitement
|
||||
`NOT_APPLICABLE` tant qu'aucun Task kind de production n'existe ; une étape
|
||||
future n'est jamais devinée active depuis un nom ou un message.
|
||||
Project open/recovery is then driven from the main thread.
|
||||
|
||||
Le panneau ressources expose CPU actif/admis/disponible et sa raison, GPU
|
||||
présent/mémoire/busy/backend lorsqu'ils sont connus, RAM/MemAvailable/réserve,
|
||||
swap total/utilisé et deltas actifs, lot/inflight/helpers/I/O, scratch et
|
||||
pression Governor GREEN/YELLOW/RED. Le contrat installé de l'exacte Task active
|
||||
est l'autorité pour CPU et lot. Un dernier diagnostic privé seulement indexé
|
||||
par kind peut appartenir à une autre Task ou séquence : backend, inflight,
|
||||
helpers, utilisation ou raison restent donc `UNKNOWN` sans association exacte
|
||||
Task+séquence. La mémoire UMA est comptée une seule fois et ni swap ni scratch
|
||||
ne sont ajoutés à la capacité RAM.
|
||||
The exact source initialization sequence remains authoritative.
|
||||
|
||||
### Dimensions, couleurs et clavier
|
||||
## Project lifetime boundary
|
||||
|
||||
Le layout complet demande au moins 100×30. Le layout compact est validé à la
|
||||
frontière 72×20 et reste supporté jusqu'au minimum 60×15. En dessous, le rendu
|
||||
se réduit au message borné `Terminal trop petit`; un resize recalcule la classe
|
||||
sans faire travailler un worker. Les rôles sémantiques sont healthy vert,
|
||||
warning jaune, error rouge, GPU cyan, CPU bleu, SSD magenta, plus dim/bold.
|
||||
Les libellés textuels demeurent l'autorité lorsqu'il n'y a pas de couleur ou
|
||||
pas assez de paires terminal.
|
||||
Changing/closing project is an exact ownership boundary.
|
||||
|
||||
Les écrans courants sont accueil, projets, import, viewer futur, tâches,
|
||||
ressources, optique, SSD et aide. `F1..F7` naviguent respectivement vers aide,
|
||||
projets, import, viewer, tâches, ressources et optique. Le segment littéral
|
||||
`F10 SSD` est réservé au début du footer et reste visible à 60 colonnes dans
|
||||
tous les modes pertinents. Les footers dérivent du même mode que le handler :
|
||||
Before Project DB close:
|
||||
|
||||
- saisie active : Enter valide, Échap annule, F10 reste disponible ;
|
||||
- import actif : `X` demande l'annulation et F10 reste disponible ; `q` et
|
||||
Échap sont affichés comme désactivés ;
|
||||
- mode idle : `q`, Échap/navigation et les commandes propres à l'écran sont
|
||||
annoncés seulement lorsqu'ils sont réellement traités ;
|
||||
- Tasks : flèches/`j`/`k`, `P` pause, `R` reprise, `C` annulation ;
|
||||
- Optique : Tab change de panneau, flèches/`j`/`k` sélectionnent, `[` revient à
|
||||
la première page et `]` charge la suivante ; `B/L/C/V/A/G/K/E` déclenchent
|
||||
les opérations indiquées et `R` retente explicitement un bind/chargement.
|
||||
```text
|
||||
views release DB borrows
|
||||
-> Queue ingress closes
|
||||
-> Queue cancels/joins/destroys
|
||||
-> terminal callbacks finish
|
||||
-> Project DB closes
|
||||
```
|
||||
|
||||
### Workflow optique
|
||||
A fresh empty Queue can then be created for the next project.
|
||||
|
||||
La TUI consomme les API v23 décrites dans
|
||||
[Project Database](project_database.md), sans SQL direct ni édition d'une ligne
|
||||
immuable. Elle inspecte une affectation Capture, effectue seulement des lookup
|
||||
metadata exacts, liste les profils de boîtier/objectif/configuration et accepte
|
||||
un objectif manuel sans électronique ni alias — le Meike de test est un cas
|
||||
normal, pas une branche produit. « Modifier » signifie créer un nouveau profil
|
||||
ou une nouvelle configuration immuable, puis l'assigner explicitement à un
|
||||
groupe de campagne encore éligible ou à un Capture non affecté. Les
|
||||
calibrations listées doivent être compatibles avec l'exacte configuration et
|
||||
la sélection reste explicite ; absence, ambiguïté, incompatibilité, BUSY, I/O
|
||||
et corruption sont rendues sans profil fabriqué. Les pages ont 16 lignes,
|
||||
rapportent un compte page-local et un indicateur « suite » exact.
|
||||
No terminal callback may dereference a closed Project DB.
|
||||
|
||||
### SSD F10 et lifetime application
|
||||
Queue terminal history belongs to the current runtime session and does not leak between projects.
|
||||
|
||||
La TUI affiche les huit états physiques `ABSENT`, `DETECTED`, `ENABLING`,
|
||||
`ENABLED`, `IN_USE`, `DRAINING`, `SAFE_TO_UNPLUG` et `ERROR`, avec identité
|
||||
stable, modèle/télémétrie lorsqu'ils sont connus, swap, scratch, mount, usage,
|
||||
leases, drain et raison. `UNKNOWN` n'est jamais remplacé par zéro ou par une
|
||||
supposition ; `SAFE_TO_UNPLUG` est mis en évidence comme endpoint sûr. F10
|
||||
choisit exclusivement l'une des capacités exactes
|
||||
`can_enable`, `can_disable` ou `can_cancel_drain` publiée par le contrôleur ; un
|
||||
état incomplet, une paire de remplacement ou un résultat malformé n'accorde
|
||||
aucune autorité. L'opération synchrone UDisks s'exécute dans au plus un thread
|
||||
joinable, tandis que le main continue de rendre et de poller sans blocage.
|
||||
## Project Database
|
||||
|
||||
La validation qui alimente ces capacités est fail-closed par état : toute
|
||||
autorité exige Drive et deux partitions détectés, identités Drive+UUID exactes,
|
||||
extents positifs connus et faits mount/activité/drain cohérents. `ABSENT` ne
|
||||
peut transporter aucun fait actif, `DETECTED` partiel n'a aucune action et un
|
||||
hazard `ERROR` déconnecté ne peut que retenir l'identité originale sans
|
||||
allocation. Seule la reconnexion complète de ce tuple peut autoriser son drain.
|
||||
Project DB uses an opaque serialized SQLite connection with bounded transactional operations.
|
||||
|
||||
Après chaque observation ou résultat validé, l'adaptateur enregistre une copie
|
||||
bornée de l'état physique auprès du Governor. Une copie malformée devient
|
||||
`ERROR` et interdit les nouvelles allocations ; l'observation ressources lit
|
||||
cet état Governor-owned, tandis que les détails/permissions F10 restent dans
|
||||
le snapshot physique. La génération source peut saturer à `UINT64_MAX` : une
|
||||
update publique égale ne réaccorde jamais une autorité stale ; seule la
|
||||
complétion du wrapper exact déjà engagé réconcilie son lease adressé. À l'arrêt,
|
||||
l'ordre est : destruction/join de la Queue et
|
||||
libération de chaque lease Task, fermeture du projet/DB, join puis unregister
|
||||
vérifié de l'adaptateur SSD, destruction du contrôleur, puis destruction du
|
||||
Governor. Les tests utilisent un provider factice et n'exécutent aucune vraie
|
||||
mutation SSD.
|
||||
Current schema head:
|
||||
|
||||
## Invariants
|
||||
```text
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
```
|
||||
|
||||
- ncurses appartient exclusivement au thread principal
|
||||
- Aucune tâche ne démarre sans réservation active
|
||||
- Les réservations sont libérées exactement une fois
|
||||
- Les buffers sont strictement bornés
|
||||
- Le Resource Governor reste l'unique propriétaire de l'admission ; ni Queue,
|
||||
ni contrôleur SSD ne constituent un second orchestrateur de ressources
|
||||
Current additive selected-execution overlays include:
|
||||
|
||||
## Statut : CURRENT / VALIDATED OPERATIONAL
|
||||
```text
|
||||
v24 raw.develop.batch/1
|
||||
v25 features.extract.batch/1
|
||||
```
|
||||
|
||||
La TUI/runtime et son raccordement SSD sont implémentés, testés et relus dans
|
||||
leur tranche. Le statut global est
|
||||
`GLOBAL_MAINTENANCE_AUDIT=PASS/FROZEN`. Les builds portables/Vulkan,
|
||||
sanitizers, contrôles de concurrence et ABI frais sont acquis ; l'unique revue
|
||||
finale indépendante a conclu PASS sans finding bloquant.
|
||||
The v23 optical model remains valid but is no longer the schema head.
|
||||
|
||||
## Current production Task inventory
|
||||
|
||||
Production currently has sixteen Task Kinds.
|
||||
|
||||
Important selected-execution additions:
|
||||
|
||||
```text
|
||||
raw.develop.batch/1
|
||||
features.extract.batch/1
|
||||
```
|
||||
|
||||
The Queue still has one active heavy callback; batch Task Kinds obtain throughput from bounded
|
||||
participants inside that callback.
|
||||
|
||||
Per-item atomicity does not require cross-item serial execution.
|
||||
|
||||
## Resource Governor
|
||||
|
||||
Every production Task goes through the Resource Governor, including fixed-resource Tasks.
|
||||
|
||||
The canonical host policy is:
|
||||
|
||||
```text
|
||||
preserve defined interactive reserve
|
||||
then maximize safe useful throughput
|
||||
```
|
||||
|
||||
```text
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
```
|
||||
|
||||
Reference-host CPU counts are evidence, not portable constants.
|
||||
|
||||
Pressure may reduce a later admission; healthy recovery may ramp useful width again.
|
||||
|
||||
Swap/zram/scratch do not enlarge admitted RAM.
|
||||
|
||||
UMA GPU memory is charged once against host RAM.
|
||||
|
||||
## Internal parallelism
|
||||
|
||||
Validated Task Kinds may execute:
|
||||
|
||||
```text
|
||||
one Queue owner callback
|
||||
-> bounded participants
|
||||
-> join
|
||||
-> deterministic owner publication
|
||||
```
|
||||
|
||||
Current examples include:
|
||||
|
||||
- Feature selected batch;
|
||||
- Visual Index;
|
||||
- Candidate Pair;
|
||||
- Matcher CPU work;
|
||||
- Geometric Verifier outer parallel preparation;
|
||||
- RAW selected batch.
|
||||
|
||||
This is not inter-Task parallelism and does not create another global worker pool.
|
||||
|
||||
## Current Matcher backend policy
|
||||
|
||||
ORB Matcher normal production is Governor-owned AUTO.
|
||||
|
||||
Eligible ORB work prefers the validated Vulkan backend.
|
||||
|
||||
Fallback is complete CPU recomputation.
|
||||
|
||||
Normal Vulkan contract:
|
||||
|
||||
```text
|
||||
inflight = 1
|
||||
helpers = 0
|
||||
useful batch <= 8
|
||||
```
|
||||
|
||||
Depth 2 remains validated private safety/benchmark capacity but was rejected as normal useful policy.
|
||||
|
||||
SIFT/RootSIFT Matcher remains CPU.
|
||||
|
||||
## Runtime observation
|
||||
|
||||
The runtime observer borrows Queue/Governor state and publishes one bounded coherent snapshot for the
|
||||
TUI.
|
||||
|
||||
It does not retain Task userdata.
|
||||
|
||||
Ordinary snapshots are rate-limited/coalesced.
|
||||
|
||||
On observation failure, the previous bounded view may be retained and explicitly marked stale.
|
||||
|
||||
Task observation includes live/pending/recent-terminal entries only within fixed capacity.
|
||||
|
||||
No unbounded Project DB scan is performed per frame.
|
||||
|
||||
## Durable progress and ETA
|
||||
|
||||
Typed Tasks publish exact `completed/total` only after their Task-specific durable prefix is committed.
|
||||
|
||||
The TUI must not infer exact scientific progress from:
|
||||
|
||||
- Task name;
|
||||
- message text;
|
||||
- generic percentage.
|
||||
|
||||
When exact counters exist, they are authoritative.
|
||||
|
||||
Throughput/ETA needs enough positive-time progress observations.
|
||||
|
||||
No-progress/pressure/restart cases become explicit states such as stalled, throttled or indeterminate
|
||||
rather than fabricated precision.
|
||||
|
||||
A terminal Task with inconsistent durable progress is visible as an integrity problem rather than
|
||||
silently forced to 100%.
|
||||
|
||||
## Pipeline observation
|
||||
|
||||
Current observable stages include:
|
||||
|
||||
```text
|
||||
Acquisition
|
||||
RAW
|
||||
Quality
|
||||
Features
|
||||
Visual Index
|
||||
Candidate
|
||||
Matcher
|
||||
GV
|
||||
Tracks
|
||||
Sparse SfM
|
||||
Dense
|
||||
```
|
||||
|
||||
Sparse SfM capability exists.
|
||||
|
||||
Dense has no production Task Kind and remains not applicable/unimplemented at the current checkpoint.
|
||||
|
||||
Historical S21/A6000 real campaigns have not executed Sparse SfM because known calibration is
|
||||
unavailable for those campaigns.
|
||||
|
||||
## TUI resource observation
|
||||
|
||||
Resource UI may display known bounded values for:
|
||||
|
||||
- active/admitted/available CPU;
|
||||
- GPU presence/backend/busy/memory;
|
||||
- RAM and `MemAvailable`;
|
||||
- host reserve;
|
||||
- swap state and active deltas;
|
||||
- batch/inflight/helpers;
|
||||
- IO;
|
||||
- scratch;
|
||||
- Governor pressure.
|
||||
|
||||
The installed contract of the exact active Task is authoritative for that sequence.
|
||||
|
||||
A diagnostic indexed only by Task Kind cannot automatically be attributed to another Task/sequence.
|
||||
|
||||
Unknown values remain unknown.
|
||||
|
||||
## TUI layout
|
||||
|
||||
The validated layout classes remain bounded.
|
||||
|
||||
Current thresholds include:
|
||||
|
||||
```text
|
||||
full layout >= 100x30
|
||||
compact boundary = 72x20
|
||||
supported minimum = 60x15
|
||||
```
|
||||
|
||||
Below the supported minimum, the UI uses its too-small-terminal fallback.
|
||||
|
||||
Repository documentation is English. Any remaining non-English executable UI literal is legacy runtime
|
||||
text and must be changed only in the explicitly scoped UI-language remediation pass; documentation does
|
||||
not redefine executable behavior by pretending that source literal has already changed.
|
||||
|
||||
## Navigation
|
||||
|
||||
The current TUI provides screens for the implemented runtime surfaces, including:
|
||||
|
||||
- home;
|
||||
- projects;
|
||||
- import;
|
||||
- tasks;
|
||||
- resources;
|
||||
- optics;
|
||||
- SSD;
|
||||
- help;
|
||||
- viewer placeholder/future surface.
|
||||
|
||||
Key bindings and exact executable labels remain owned by the TUI source and its tests.
|
||||
|
||||
Documentation should describe behavior rather than preserve stale localized literals as authority.
|
||||
|
||||
## Optical workflow
|
||||
|
||||
The TUI uses the public optical APIs introduced by the v23 overlay.
|
||||
|
||||
It does not write optical SQLite rows directly.
|
||||
|
||||
It supports explicit inspection/selection and immutable profile creation.
|
||||
|
||||
Manual lenses without electronic metadata are valid data.
|
||||
|
||||
No workflow may fabricate:
|
||||
|
||||
- "unknown" lens identity;
|
||||
- calibration compatibility;
|
||||
- metadata match;
|
||||
- focal/lens substitution.
|
||||
|
||||
Calibration selection remains explicit and exact.
|
||||
|
||||
## SSD F10 boundary
|
||||
|
||||
The SSD UI reflects controller capability/state rather than inventing actions.
|
||||
|
||||
The controller owns physical detection, pairing, mount/swap/scratch state and bounded UDisks operations.
|
||||
|
||||
The Resource Governor owns scratch lease admission.
|
||||
|
||||
A state that is incomplete, stale or physically inconsistent grants no control/lease authority.
|
||||
|
||||
Scratch is storage capacity, never RAM.
|
||||
|
||||
## Global shutdown
|
||||
|
||||
Ownership shutdown preserves:
|
||||
|
||||
```text
|
||||
Task Queue / Task leases
|
||||
-> project close
|
||||
-> SSD operation join / Governor unregister
|
||||
-> SSD controller
|
||||
-> Resource Governor
|
||||
```
|
||||
|
||||
A real outstanding scratch lease can block unregister and must remain an observable error.
|
||||
|
||||
Do not abandon a live lease pointer.
|
||||
|
||||
## Error/recovery model
|
||||
|
||||
Runtime operations use local rollback and explicit publication boundaries.
|
||||
|
||||
File asset publication plus SQLite is not treated as one distributed transaction.
|
||||
|
||||
A successfully published physical asset followed by DB failure may leave a valid orphan.
|
||||
|
||||
Recovery validates known durable representations; it does not guess or silently repair scientific
|
||||
identity.
|
||||
|
||||
## Current real checkpoint
|
||||
|
||||
Current retained A6000 checkpoint:
|
||||
|
||||
```text
|
||||
real-a6000-pre-sfm-2026-09-02
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
It exercised current runtime/Queue/Governor behavior through:
|
||||
|
||||
```text
|
||||
selected RAW batch
|
||||
selected Feature batch
|
||||
Visual Index
|
||||
Candidate Pair
|
||||
Matcher
|
||||
Geometric Verifier v3
|
||||
Track Builder
|
||||
```
|
||||
|
||||
The final continuation recorded deterministic restart/reuse and stopped with:
|
||||
|
||||
```text
|
||||
Sparse SfM Tasks 0
|
||||
Sparse Reconstructions 0
|
||||
Dense/MVS 0
|
||||
```
|
||||
|
||||
This is a current real runtime checkpoint, later than the historical global-maintenance checkpoint.
|
||||
|
||||
Both remain valid for the boundaries they prove.
|
||||
|
||||
## Current limits
|
||||
|
||||
Current runtime intentionally does not provide:
|
||||
|
||||
- multiple concurrent heavy Queue callbacks;
|
||||
- generic inter-Task DAG scheduling;
|
||||
- generic Task priorities beyond current Queue policy;
|
||||
- generic autosave ahead of Task-specific durable publication;
|
||||
- Dense/MVS production Task Kind;
|
||||
- live capture/viewer reconstruction loop;
|
||||
- generic scratch-consuming Task Kind.
|
||||
|
||||
Those are future product/implementation decisions, not silently missing state.
|
||||
|
||||
## Summary
|
||||
|
||||
```text
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
CURRENT_PRODUCTION_TASK_KINDS=16
|
||||
|
||||
TASK_QUEUE_WORKERS=1
|
||||
ACTIVE_HEAVY_CALLBACKS=1
|
||||
INTER_TASK_PARALLELISM=NOT_IMPLEMENTED
|
||||
INTERNAL_PARALLELISM=BOUNDED
|
||||
|
||||
GENERIC_DAG=NOT_IMPLEMENTED
|
||||
DENSE_TASK_KIND=NOT_IMPLEMENTED
|
||||
CURRENT_SCRATCH_CONSUMING_TASK_KINDS=0
|
||||
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -1,268 +1,628 @@
|
|||
# Registry des types métier de tâches
|
||||
# Task Kind Registry
|
||||
|
||||
## Feature kinds v1A
|
||||
## Status
|
||||
|
||||
La registry statique conserve `features.extract` v1 pour ORB et ajoute
|
||||
`features.extract.sift` v1 et `features.extract.rootsift` v1. Les reconstructeurs
|
||||
chargent la table dédiée, revalident le fingerprint et ne capturent aucun
|
||||
`AppState`.
|
||||
```text
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
CURRENT_PRODUCTION_TASK_KINDS=16
|
||||
|
||||
## Contrat
|
||||
TASK_KIND_REGISTRY_STATUS=IMPLEMENTED
|
||||
COMPUTE_GOVERNOR_V2=PASS/FROZEN
|
||||
ORB_VULKAN_ASYNC_EXECUTION=PASS/FROZEN
|
||||
|
||||
Une instance possède un **task ID** stable. Son **task kind** décrit son
|
||||
comportement métier, son **task state** décrit son état d'avancement, la
|
||||
**checkpoint version** décrit le codec générique et la **task kind version**
|
||||
versionne les paramètres nécessaires au reconstructeur. Ces identités ne sont
|
||||
pas interchangeables.
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
```
|
||||
|
||||
Le kind v1 est une chaîne ASCII de 1 à 64 caractères au format
|
||||
`[a-z0-9][a-z0-9._-]*`. La version est un entier non nul. Aucun kind n'est
|
||||
déduit d'un nom, callback ou pointeur et aucune normalisation n'est effectuée.
|
||||
The production registry contains exactly sixteen version-1 Task Kinds.
|
||||
|
||||
## Registry et ownership
|
||||
The registry is a bounded static dispatch table. It maps durable Task Kind identity to a reconstruction
|
||||
function. It is not a scheduler, dependency graph, plugin loader, resource-policy database or
|
||||
scientific identity resolver.
|
||||
|
||||
La registry est une vue bornée à 64 descriptors sur un tableau statique
|
||||
immutable. Le lookup est linéaire, déterministe, sans allocation et sûr en
|
||||
lecture concurrente. Elle ne charge aucun code dynamiquement.
|
||||
Project DB v25 is the current schema head. Earlier schema versions remain authoritative for the Task
|
||||
payloads they introduced.
|
||||
|
||||
Un descriptor contient exactement le kind, sa version et un reconstructeur.
|
||||
Le reconstructeur produit callback, userdata et destructeur optionnel. Avant le
|
||||
transfert, la registry nettoie le userdata sur toute erreur ; après restauration
|
||||
réussie, `Lardon3DTask` en devient propriétaire et le détruit une fois après la
|
||||
fin de l'exécution. Le constructeur métier n'est jamais appelé sous mutex DB.
|
||||
## Identity contract
|
||||
|
||||
La registry peut normaliser une ancienne estimation opérationnelle connue. Le
|
||||
reconstructeur reçoit toujours le snapshot durable original afin de valider le
|
||||
mode exact ; la registry applique ensuite l'estimation effective uniquement à
|
||||
la copie privée transmise à la restauration de `Task`. Cette normalisation est
|
||||
éphémère et déterministe : elle ne stage, ne promeut et ne publie aucun
|
||||
checkpoint contenant seulement une estimation différente sous le même résumé.
|
||||
Une panne pré-terminale peut donc répéter la normalisation exacte. Cette couture
|
||||
ne peut modifier ni identité, paramètres scientifiques, progression ou curseur
|
||||
métier, et toute forme voisine est rejetée.
|
||||
A durable Task has several distinct identities and versions:
|
||||
|
||||
## Inventaire production et entrées runtime
|
||||
- stable Task ID;
|
||||
- Task Kind string;
|
||||
- Task Kind version;
|
||||
- generic checkpoint codec version;
|
||||
- Task state;
|
||||
- scientific input identity owned by the Task-specific payload.
|
||||
|
||||
`src/task_kinds.c::lardon3d_task_kind_registry_production()` enregistre les
|
||||
16 kinds v1 du profil de production courant. La colonne « reprise » nomme le
|
||||
reconstructeur du binding ; « callback » nomme l'entrée runtime privée dans le
|
||||
même fichier. Le détail chiffré des capacités est centralisé dans l'[audit des
|
||||
16 kinds](resource_governor.md#audit-des-15-kinds-de-production).
|
||||
These fields are not interchangeable.
|
||||
|
||||
| Kind v1 | Source, reprise et callback | Réconciliation pré-admission courante |
|
||||
| --- | --- | --- |
|
||||
| `raw.develop` | `raw_development_task.cpp`; `lardon3d_raw_development_task_reconstruct`; `run` | Aucune |
|
||||
| `raw.develop.batch` | `raw_development_task.cpp`; `lardon3d_raw_development_batch_task_reconstruct`; `run_batch` | Association v24 Task → exécution sélectionnée ; le curseur reste dans l'exécution |
|
||||
| `photo_quality.triage` | `photo_quality_task.cpp`; `lardon3d_photo_quality_task_reconstruct`; `run` | Aucune |
|
||||
| `acquisition_campaign.run` | `acquisition_campaign_task.cpp`; `lardon3d_acquisition_campaign_task_reconstruct`; `run` | Forme courante ou forme v22 exacte vérifiée contre la requête immuable → capacité courante en mémoire |
|
||||
| `import.images` | `import_task.c`; `lardon3d_image_import_reconstruct`; `run_image_import` | Aucune |
|
||||
| `features.extract` | `feature_task.c`; `lardon3d_feature_extract_reconstruct`; `run` | Formes CPU12/CPU1 historiques exactes → demande OpenCV portable ; runtime borné au compute-pool |
|
||||
| `features.extract.batch` | `feature_task.c`; `lardon3d_feature_extract_batch_reconstruct`; `run_batch` | Association v25 à l'exécution sélectionnée, domaine ORB exact et curseur typé ; CPU/lot bornés 1..12 |
|
||||
| `features.extract.sift` | `sift_task.c`; `lardon3d_sift_extract_reconstruct`; `run` | CPU12/CPU1 historiques exacts → demande OpenCV portable ; runtime borné au compute-pool |
|
||||
| `features.extract.rootsift` | `sift_task.c`; `lardon3d_sift_extract_reconstruct`; `run` | Même réconciliation SIFT ; aucune voie GPU validée |
|
||||
| `visual_index.update` | `visual_index_task.c`; `lardon3d_visual_index_update_reconstruct`; `run` | Formes CPU12/CPU1 historiques exactes → CPU/lot 1..16 |
|
||||
| `candidate_pair.generate` | `candidate_pair_task.c`; `lardon3d_candidate_pair_generate_reconstruct`; `run` | Formes CPU12 et CPU1 historiques exactes → CPU/lot 1..64 |
|
||||
| `matcher.run` | `matcher_task.c`; `lardon3d_matcher_task_reconstruct`; `run` | Signatures CPU/Vulkan historiques exactes → formes courantes en mémoire |
|
||||
| `geometric_verifier.run` | `geometric_verifier_task.c`; `lardon3d_geometric_verifier_task_reconstruct`; `run` | Forme sérielle CPU1/batch8 exacte → CPU utile 1..8, lot 1..16 |
|
||||
| `track_builder.run` | `track_builder_task.cpp`; `lardon3d_track_builder_task_reconstruct`; `run` | Aucune |
|
||||
| `sparse_sfm.run` | `sparse_sfm_task.cpp`; `lardon3d_sparse_sfm_task_reconstruct`; `run` | Aucune |
|
||||
| `incremental_reconstruction.run` | `incremental_reconstruction_task.cpp`; `lardon3d_incremental_reconstruction_task_reconstruct`; `run` | Aucune |
|
||||
A Task Kind v1 string is ASCII, 1 to 64 characters, with the form:
|
||||
|
||||
## Couture privée Compute Governor v2
|
||||
```text
|
||||
[a-z0-9][a-z0-9._-]*
|
||||
```
|
||||
|
||||
**COMPUTE_GOVERNOR_V2 — PASS / FROZEN.** Le descriptor C public
|
||||
reste limité à kind, version et reconstructeur. L'enveloppe de capacités est
|
||||
intégrée sans changement d'ABI dans le `struct Lardon3DTask` opaque et les
|
||||
coutures privées `src/task_internal.h` / `src/resource_governor_internal.h`.
|
||||
Les coutures d'admission sont `src/task_queue.c::select_admissible()`,
|
||||
`src/task.c::lardon3d_task_sequence_break()` et, côté Governor,
|
||||
la sélection multi-capacité sur un snapshot unique. La normalisation historique
|
||||
exacte reste dans `src/task_kind_registry.c::normalize_known_legacy_estimate()`.
|
||||
The Task Kind version is a non-zero integer.
|
||||
|
||||
Cette enveloppe n'est ni une identité scientifique, ni un nouveau payload
|
||||
Project DB, ni un nouveau scheduler. Le Governor possède l'admission de tous
|
||||
les kinds, y compris les formes entièrement fixes. Le contrat choisi est
|
||||
immutable pendant une séquence et seule la suivante peut être adaptée. Une Task
|
||||
sans alternative reçoit automatiquement une capacité égale à son estimation
|
||||
durable. Le Governor conserve un état borné par kind/backend et un dernier
|
||||
diagnostic ; ni l'enveloppe ni ce choix ne sont persistés.
|
||||
No Task Kind is inferred from:
|
||||
|
||||
La politique CPU hôte reste privée au Governor : masque permis, groupes
|
||||
package/core/SMT, compute-pool et résultat d'application du worker Queue. Le
|
||||
compute-pool borne l'admission de chaque kind. Feature/SIFT/RootSIFT utilisent
|
||||
le maximum `int` positif comme borne de l'API OpenCV, puis consomment le compte
|
||||
immutable réellement admis ; les CPU12 durables ne sont plus que des signatures
|
||||
historiques exactes. Les kinds CPU1 justifiés restent fixes. Aucun ID CPU ou
|
||||
choix d'affinité n'entre dans le descriptor, le checkpoint ou le Project DB.
|
||||
- Task display name;
|
||||
- callback address;
|
||||
- userdata address;
|
||||
- source file name;
|
||||
- persisted function pointer;
|
||||
- Task ID;
|
||||
- scientific fingerprint.
|
||||
|
||||
Le feedback ne requalifie pas un succès de reprise en travail durable : les
|
||||
kinds Feature/SIFT/RootSIFT comptent un item seulement après extraction et
|
||||
publication durable propre. READY, collision `ALREADY_PRESENT` ou publication
|
||||
incertaine compte zéro ; Visual Index compte pareillement zéro pour un segment
|
||||
`PUBLISHED_NOT_DURABLE`.
|
||||
No normalization is performed on the persisted kind string.
|
||||
|
||||
L'état privé par kind/backend coordonne désormais une seule dimension d'essai.
|
||||
Les CPU réductibles slow-startent par doubles successifs depuis 1, puis le
|
||||
maximum exact de leur capacité, toujours bornés par le compute-pool. Après deux
|
||||
observations de baseline, deux observations à au moins +5 % sont nécessaires
|
||||
pour accepter le palier.
|
||||
Une fois CPU stabilisé, seuls les kinds dont le callback consomme réellement
|
||||
son lot peuvent ouvrir un essai de lot. `features.extract`, SIFT et RootSIFT
|
||||
enregistrent une observation atomique réussie partagée entre Tasks ; Visual
|
||||
Index, Candidate et Matcher enregistrent chaque séquence. Les autres formes ou
|
||||
dimensions non adaptables restent égales à leur capacité fixe honnête.
|
||||
## Registry ownership
|
||||
|
||||
## Persistance et legacy
|
||||
The registry contains at most 64 descriptors in a static immutable array.
|
||||
|
||||
Le checkpoint générique reste en version 1. Project Database v7 conserve le
|
||||
kind/version ; les lignes migrées depuis v1 restent `NULL/NULL` et sont classées
|
||||
`LEGACY_UNTYPED`. Un kind inconnu ou une version non supportée reste inspectable
|
||||
mais inexécutable. Aucun type n'est inventé et aucun code n'est sélectionné par
|
||||
adresse persistée.
|
||||
Lookup is:
|
||||
|
||||
## Statut
|
||||
- linear;
|
||||
- deterministic;
|
||||
- allocation-free;
|
||||
- safe for concurrent readers.
|
||||
|
||||
**IMPLEMENTED** — identité typée immutable, registry statique, lookup,
|
||||
migration DB v1→v2, classification recovery et restauration explicite testée.
|
||||
The registry never dynamically loads code.
|
||||
|
||||
**IMPLEMENTED** — le descriptor production `import.images`, version 1, charge
|
||||
le chemin source borné depuis la table dédiée et reconstruit callback et
|
||||
userdata sans `AppState *` ancien.
|
||||
A descriptor contains only:
|
||||
|
||||
**IMPLEMENTED** — `project_open()` utilise la registry production immutable
|
||||
pour restaurer hors mutex DB et transférer chaque tâche acceptée à la queue.
|
||||
- Task Kind string;
|
||||
- Task Kind version;
|
||||
- reconstruction function.
|
||||
|
||||
**NOT_YET_WIRED** — réconciliation orpheline et dépendances/DAG. Les kinds
|
||||
reconstructibles checkpointent déjà à leurs frontières métier ; la Registry ne
|
||||
possède pas un timer autosave et ne doit pas devancer leurs curseurs durables.
|
||||
The public descriptor does not contain resource-policy state, CPU topology, GPU identity, scratch
|
||||
state or persisted scheduler configuration.
|
||||
|
||||
**IMPLEMENTED** — `features.extract` version 1 reconstruit une extraction ORB
|
||||
depuis `image_id` et ses paramètres bornés.
|
||||
## Reconstruction ownership
|
||||
|
||||
**IMPLEMENTED** — `visual_index.update`, version 1, recharge
|
||||
`visual_index_id + after_feature_set_id` et reconstruit un contexte neuf.
|
||||
A reconstruction function returns:
|
||||
|
||||
**IMPLEMENTED** — `candidate_pair.generate`, version 1, recharge
|
||||
`visual_index_id + after_feature_set_id + top_k + minimum_evidence_count
|
||||
+ scanset_filter + exclude_same_asset` depuis `candidate_pair_generate_tasks`
|
||||
et reconstruit un contexte boundé. La restauration reconnaît le snapshot
|
||||
opérationnel sériel historique v1 exact et la forme CPU12 immédiatement
|
||||
antérieure, puis les remplace éphémèrement par la demande courante CPU64 avec
|
||||
8 Mio par item avant admission. Le checkpoint historique et le curseur typé
|
||||
restent inchangés ; une forme voisine est rejetée.
|
||||
- callback;
|
||||
- callback userdata;
|
||||
- optional userdata destructor;
|
||||
- Task-specific private binding state where required.
|
||||
|
||||
**PASS / FROZEN — Compute Governor v2.** `matcher.run`, version 1,
|
||||
recharge la configuration Matcher, l'identité Feature Set et le curseur
|
||||
`after_candidate_pair_id`. Il traite une Candidate Pair atomique à la fois dans
|
||||
des lots bornés à douze, checkpoint le curseur et repasse par le Governor entre
|
||||
les lots. La table durable `matcher_tasks` est introduite par Project DB v11,
|
||||
après le Match Result v10. Son reconstructeur accepte les formes courantes
|
||||
CPU12/GPU0 et CPU1/GPU1/640 Kio à lot `1..12`, les signatures historiques
|
||||
CPU8/GPU0 et Vulkan à lot maximal 8, puis les formes CPU12
|
||||
pré-estimation-par-paire. La normalisation reste en mémoire. Une forme voisine
|
||||
échoue au lieu de servir d'indice de backend ; le payload Project DB ne change
|
||||
pas et ne persiste aucune identité matérielle. Les nouvelles Tasks ORB normales
|
||||
ont une signature de classe `MIXED`, dont les autres champs restent une demande
|
||||
de ressources réelle ; elle seule reconstruit la politique Governor `AUTO`.
|
||||
Toutes les formes ORB de classe `CPU`, anciennes ou courantes, reconstruisent
|
||||
un CPU fixe pour préserver les overrides explicites et une compatibilité sûre ;
|
||||
une forme Vulkan restaurée reste fixe Vulkan. Un build portable reconstruit la
|
||||
même politique `MIXED` mais n'expose que sa capacité CPU. Les snapshots tout à
|
||||
zéro Candidate/SIFT/RootSIFT sont explicitement corrompus ; seules leurs
|
||||
signatures historiques complètes exactes sont acceptées. Seule la forme AUTO
|
||||
restaurée établit la disponibilité Vulkan partagée. Restaurer ensuite CPU,
|
||||
Vulkan ou une signature historique fixe n'écrit rien dans cet état : la
|
||||
co-restauration est indépendante de l'ordre. Aucun nouvel état de
|
||||
backend n'est persisté.
|
||||
Before ownership transfer, the registry destroys any newly allocated userdata on failure.
|
||||
|
||||
Pour une Task AUTO, la Registry reconstruit aussi l'enveloppe privée Vulkan
|
||||
CPU1/GPU1, lot opérationnel `1..8`, helpers 0 et inflight 1. La signature
|
||||
durable historique reste à lot `1..12`; la signature 640 Kio
|
||||
reste la forme depth-1 minimale et n'est pas mutée; l'admission normale facture
|
||||
exactement 640 Kio une seule fois sur UMA. Le choix inflight est immutable dans
|
||||
la séquence et ne devient ni payload, ni fingerprint, ni indice de reprise.
|
||||
Vulkan explicite reste depth 1. La capacité privée de sûreté/benchmark peut
|
||||
forcer deux slots et 1,25 Mio sans changer la reconstruction normale.
|
||||
Le backend ne mappe pas le maximum de l'enveloppe à sa création : il retient
|
||||
exactement un slot à depth 1 et deux seulement sous une séquence depth 2 admise,
|
||||
puis libère le second avant l'admission suivante. La signature durable 640 Kio
|
||||
reste donc inchangée sans sous-facturer une allocation depth 2 forcée.
|
||||
After a successful `Lardon3DTask` restoration, the Task owns the userdata and destroys it exactly once
|
||||
after execution lifetime ends.
|
||||
|
||||
L'A/B forcé ABBA a mesuré 54,661652238 paires/s à depth 1 et 55,797311953 à
|
||||
depth 2, soit +2,077617 %, sous le deadband 5 %, avec digest identique, quatre
|
||||
séquences de fallback local par exécution et zéro panne/discard. La Registry conserve donc
|
||||
`DEPTH_MAX_VALIDATED_SAFETY=2` pour les seules coutures privées, mais la
|
||||
capacité AUTO normale suit `DEPTH_MAX_USEFUL=1` : depth 2 est
|
||||
**REJECTED_WITH_MEASURED_REASON**, sans nouvelle signature durable.
|
||||
A business-specific constructor is never called while the Project DB mutex is held.
|
||||
|
||||
La télémétrie privée de `matcher.run` conserve les classes de fallback par
|
||||
séquence et compte aussi les items exacts local-ineligible/backend-failure/other
|
||||
après leur publication durable. Ce détail opérationnel n'ajoute aucun kind,
|
||||
champ durable ou identité et empêche le regroupement batch de devenir un
|
||||
comparateur scientifique. Le commit immédiat par item reste acquis si une
|
||||
paire suivante avorte, tandis que le feedback de séquence n'est pas enregistré;
|
||||
la déduplication actuelle vit seulement avec la Task reconstruite en mémoire.
|
||||
Les logs batch 2/4 antérieurs à ce compteur restent préliminaires et prouvent
|
||||
seulement l'invalidité du comparateur par séquences. Les huit runs item-valides
|
||||
`forced-batch{2,4,8,12}-items{,-b}.stdout.jsonl` conservent chacun 4113 paires,
|
||||
six items locaux, zéro panne/autre et le même digest. Les débits combinés sont
|
||||
54,180767704, 66,094373197, 74,784998723 et 76,755814095 paires/s. Les gains
|
||||
jusqu'à batch 8 dépassent 5 %, celui de 8 à 12 vaut seulement +2,635308425 % :
|
||||
la Registry expose `BATCH_MAX_USEFUL=8` en AUTO normal et réserve batch 12 aux
|
||||
preuves privées (`REJECTED_WITH_MEASURED_REASON`).
|
||||
Le S21 final confirme l'enveloppe Registry en production : `matcher.run` v1
|
||||
reste le même kind durable, AUTO choisit Vulkan pour 21 630 admissions et
|
||||
termine 172 741/172 741 résultats à batch 8/inflight 1/helpers 0. Le passage
|
||||
transitoire 8 → 1 → 2 → 4 → 8 ne modifie ni signature durable, ni
|
||||
fingerprint, ni digest scientifique. Aucun backend ou champ persistant n'est
|
||||
ajouté par cette adaptation.
|
||||
Project-open recovery copies the durable record out of the DB boundary before registry lookup and
|
||||
Task-specific reconstruction.
|
||||
|
||||
La reconstruction AUTO ne sonde ni n'initialise Vulkan sur le thread
|
||||
`project_open()`. Elle expose la capacité depuis les seules métadonnées runtime
|
||||
build/backend/GPU ; le Governor possède le dimensionnement exact et
|
||||
l'admission UMA sur son snapshot. Le premier begin appartient au worker Queue après son
|
||||
affinité. La politique Mesa sûre est déjà établie avant les pthreads et
|
||||
l'initialisation du driver ; aucun sweep/latch auxiliaire n'appartient donc au
|
||||
contexte Task. Une paire localement inéligible n'initialise pas le backend. Une
|
||||
panne réelle produit des paires CPU complètes et rend le backend indisponible
|
||||
aux admissions AUTO suivantes sans réécrire le snapshot durable.
|
||||
## Exact legacy estimate normalization
|
||||
|
||||
Le contrôle de benchmark `synchronous` du runner réel n'étend pas le descriptor
|
||||
ni le reconstructeur. Il est compilé hors du binaire production, attaché
|
||||
seulement au contexte éphémère d'une nouvelle Task et refusé par le runner si
|
||||
une Task Matcher doit être reprise. La Registry continue donc à reconstruire
|
||||
uniquement la politique AUTO/fixe déduite de la signature durable, jamais un
|
||||
pipeline de benchmark.
|
||||
The registry may recognize exact historical operational resource descriptors for restart compatibility.
|
||||
|
||||
**IMPLEMENTED** — ORB, SIFT et RootSIFT acceptent leurs formes CPU12/CPU1
|
||||
historiques complètes et les normalisent vers la demande OpenCV portable
|
||||
`INT_MAX`. Le Governor borne l'exécution au compute-pool ; les sorties testées
|
||||
à 1/2/4/8/12 restent égales. Cette compatibilité opérationnelle n'altère ni
|
||||
fingerprint, Feature Set, checkpoint durable, ni politique scientifique.
|
||||
The reconstruction function always receives the original durable snapshot.
|
||||
|
||||
**IMPLEMENTED** — `geometric_verifier.run`, version 1, recharge la configuration
|
||||
Fundamental immuable, en revalide le fingerprint et reprend `after_match_result_id`.
|
||||
Project DB v13 ajoute uniquement `geometric_verifier_tasks`, car le checkpoint
|
||||
générique v1 ne possède aucun payload propre au kind.
|
||||
If an exact historical descriptor is recognized, the registry may replace only the private effective
|
||||
resource estimate passed to the restored runtime Task.
|
||||
|
||||
La forme historique série exacte (4 Mio fixes, CPU1, batch 1..8) est normalisée
|
||||
en mémoire vers 8 Mio par item, CPU utile 8 et batch maximal 16. Cette évolution
|
||||
ne touche ni fingerprint, GVR, ordre, curseur ni checkpoint historique ; une
|
||||
forme voisine est refusée.
|
||||
This normalization is:
|
||||
|
||||
**IMPLEMENTED** — `track_builder.run`, version 1, reconstruit un scope explicite
|
||||
depuis son payload Project DB v15 et son asset little-endian validé. Le callback
|
||||
réutilise l'orchestration Gate C et le reconstructeur refuse toute corruption,
|
||||
version, fingerprint, checksum, tri, unicité ou L3DTSIS1 incohérents.
|
||||
- ephemeral;
|
||||
- deterministic;
|
||||
- exact-shape only;
|
||||
- non-persistent.
|
||||
|
||||
**PASS / FROZEN** — `sparse_sfm.run`, version 1, recharge le
|
||||
payload scientifique explicite Project DB v17, restaure l'estimation générique
|
||||
persistée et rejoue D puis E depuis les références Track Set/calibration. Le
|
||||
fingerprint F0 est recalculé ; le checkpoint générique v1 reste inchangé.
|
||||
It never rewrites:
|
||||
|
||||
**PASS / FROZEN** — `incremental_reconstruction.run`,
|
||||
version 1, recharge le payload Project DB v18 composé du prédécesseur, du Track
|
||||
Set d'extension, du scope de calibration et du fingerprint H. La tâche atomique
|
||||
recalcule depuis ces entrées après redémarrage, passe par la Queue et le
|
||||
Governor avec son estimation H immuable, et ne persiste aucun état de solveur.
|
||||
Elle n'ajoute ni DAG ni dépendance implicite.
|
||||
- the generic checkpoint;
|
||||
- Task ID;
|
||||
- Task Kind/version;
|
||||
- Task progress;
|
||||
- Task-specific cursor;
|
||||
- scientific parameters;
|
||||
- scientific fingerprint;
|
||||
- Project DB payload.
|
||||
|
||||
A neighboring or partially matching historical resource descriptor is rejected rather than guessed.
|
||||
|
||||
A crash before a later terminal checkpoint may therefore repeat the same exact normalization on the
|
||||
next restart.
|
||||
|
||||
## Production inventory
|
||||
|
||||
The production registry is created by:
|
||||
|
||||
```text
|
||||
src/task_kinds.c::lardon3d_task_kind_registry_production()
|
||||
```
|
||||
|
||||
It contains exactly these sixteen version-1 kinds:
|
||||
|
||||
| # | Task Kind | Durable payload / introduction | Reconstruction |
|
||||
| ---: | --- | --- | --- |
|
||||
| 1 | `raw.develop` | Project DB v22 `raw_development_tasks` | `lardon3d_raw_development_task_reconstruct` |
|
||||
| 2 | `raw.develop.batch` | Project DB v24 `raw_development_batch_tasks` | `lardon3d_raw_development_batch_task_reconstruct` |
|
||||
| 3 | `photo_quality.triage` | Project DB v21 | `lardon3d_photo_quality_task_reconstruct` |
|
||||
| 4 | `acquisition_campaign.run` | Project DB v20 | `lardon3d_acquisition_campaign_task_reconstruct` |
|
||||
| 5 | `import.images` | historical typed import payload | `lardon3d_image_import_reconstruct` |
|
||||
| 6 | `features.extract` | historical Feature Task payload | `lardon3d_feature_extract_reconstruct` |
|
||||
| 7 | `features.extract.batch` | Project DB v25 `feature_extract_batch_tasks` | `lardon3d_feature_extract_batch_reconstruct` |
|
||||
| 8 | `features.extract.sift` | typed SIFT Feature payload | `lardon3d_sift_extract_reconstruct` |
|
||||
| 9 | `features.extract.rootsift` | typed RootSIFT Feature payload | `lardon3d_sift_extract_reconstruct` |
|
||||
| 10 | `visual_index.update` | Visual Index Task payload | `lardon3d_visual_index_update_reconstruct` |
|
||||
| 11 | `candidate_pair.generate` | Project DB v9 `candidate_pair_generate_tasks` | `lardon3d_candidate_pair_generate_reconstruct` |
|
||||
| 12 | `matcher.run` | Project DB v11 `matcher_tasks` | `lardon3d_matcher_task_reconstruct` |
|
||||
| 13 | `geometric_verifier.run` | Project DB v13 `geometric_verifier_tasks` | `lardon3d_geometric_verifier_task_reconstruct` |
|
||||
| 14 | `track_builder.run` | Project DB v15 `track_builder_tasks` | `lardon3d_track_builder_task_reconstruct` |
|
||||
| 15 | `sparse_sfm.run` | Project DB v17 `sparse_sfm_tasks` | `lardon3d_sparse_sfm_task_reconstruct` |
|
||||
| 16 | `incremental_reconstruction.run` | Project DB v18 `incremental_reconstruction_tasks` | `lardon3d_incremental_reconstruction_task_reconstruct` |
|
||||
|
||||
The current detailed resource-capability inventory is owned by
|
||||
[`resource_governor.md`](resource_governor.md), in its production sixteen-kind audit.
|
||||
|
||||
Historical documents that correctly recorded fewer kinds at their checkpoint remain historical
|
||||
evidence. They must not be rewritten merely to make their old count equal the current count.
|
||||
|
||||
## Current additive Task Kinds
|
||||
|
||||
The two Task Kinds added after the historical fourteen-kind maintenance inventory are:
|
||||
|
||||
```text
|
||||
raw.develop.batch/1
|
||||
features.extract.batch/1
|
||||
```
|
||||
|
||||
### `raw.develop.batch/1`
|
||||
|
||||
Project DB v24 binds one Task to one immutable selected execution.
|
||||
|
||||
The selected-execution cursor remains authoritative. Independent RAW items may be prepared within a
|
||||
bounded admitted window, participants join, and the owner publishes the selected prefix in canonical
|
||||
order.
|
||||
|
||||
The Task-specific restart path reuses exact already published RAW-derived representations and does not
|
||||
infer identity from path, basename or processing position.
|
||||
|
||||
### `features.extract.batch/1`
|
||||
|
||||
Project DB v25 binds one Task to:
|
||||
|
||||
- one immutable selected execution;
|
||||
- one monotonic `next_item_index`;
|
||||
- exact ORB kind/version/parameters/fingerprint.
|
||||
|
||||
Independent selected images may be prepared concurrently without participant SQLite publication.
|
||||
|
||||
After join, the owner validates/reuses the exact READY Feature Set, publishes in selected-item order,
|
||||
advances the typed cursor and then advances generic Task progress/checkpoint.
|
||||
|
||||
The historical `features.extract/1` kind remains valid and reconstructible.
|
||||
|
||||
```text
|
||||
PER_IMAGE_FEATURE_RESULT=ATOMIC
|
||||
PER_IMAGE_ATOMICITY_REQUIRES_CROSS_IMAGE_SERIALISM=NO
|
||||
OWNER_ONLY_PUBLICATION_REQUIRES_SERIAL_PREPARATION=NO
|
||||
```
|
||||
|
||||
## Compute Governor v2 private seam
|
||||
|
||||
```text
|
||||
COMPUTE_GOVERNOR_V2=PASS/FROZEN
|
||||
```
|
||||
|
||||
The public registry descriptor remains limited to kind, version and reconstruct function.
|
||||
|
||||
Resource capability alternatives live in private runtime state, including the opaque `Lardon3DTask`
|
||||
and private Task/Resource Governor seams.
|
||||
|
||||
The key production admission boundaries remain:
|
||||
|
||||
- Queue admissibility selection;
|
||||
- `lardon3d_task_sequence_break()`;
|
||||
- Resource Governor capability selection from one fresh snapshot.
|
||||
|
||||
This private capability envelope is not:
|
||||
|
||||
- scientific identity;
|
||||
- a new Project DB payload;
|
||||
- a Task Kind version;
|
||||
- a scheduler;
|
||||
- a dependency graph;
|
||||
- durable hardware identity.
|
||||
|
||||
The selected resource contract is immutable for one admitted sequence. A later sequence may receive a
|
||||
different admitted contract.
|
||||
|
||||
A Task with no valid alternatives receives the honest fixed capability represented by its contract.
|
||||
|
||||
## Host CPU policy
|
||||
|
||||
The Resource Governor owns host CPU policy.
|
||||
|
||||
It determines:
|
||||
|
||||
- allowed affinity mask;
|
||||
- package/core/SMT topology where available;
|
||||
- interactive reserve;
|
||||
- compute pool;
|
||||
- actual worker affinity;
|
||||
- per-kind admission bounds.
|
||||
|
||||
CPU IDs never enter:
|
||||
|
||||
- Task Kind descriptor;
|
||||
- generic checkpoint;
|
||||
- Project DB Task payload;
|
||||
- scientific fingerprint.
|
||||
|
||||
Reference-host observations such as a 12-logical-CPU compute pool are evidence for that host, not a
|
||||
portable registry ceiling.
|
||||
|
||||
The canonical repository policy is:
|
||||
|
||||
```text
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
```
|
||||
|
||||
After the interactive reserve and all safety constraints are preserved, useful validated compute
|
||||
capacity should not remain idle merely because an older durable resource descriptor was smaller.
|
||||
|
||||
## OpenCV kinds
|
||||
|
||||
The OpenCV-controlled kinds include:
|
||||
|
||||
```text
|
||||
features.extract
|
||||
features.extract.sift
|
||||
features.extract.rootsift
|
||||
```
|
||||
|
||||
Exact historical CPU1/CPU12 descriptors may be recognized for restart compatibility.
|
||||
|
||||
The effective current Task request can use the portable positive OpenCV upper bound, while the Resource
|
||||
Governor limits actual admission to the host compute pool.
|
||||
|
||||
The callback applies the admitted OpenCV thread count and restores the process-wide baseline on every
|
||||
exit path.
|
||||
|
||||
Controlled validation at 1/2/4/8/12 threads preserved the required scientific outputs for the
|
||||
validated ORB/SIFT/RootSIFT contracts.
|
||||
|
||||
Those thread counts are operational evidence and do not change:
|
||||
|
||||
- extractor fingerprint;
|
||||
- Feature Set identity;
|
||||
- Feature File format;
|
||||
- Task Kind version.
|
||||
|
||||
For `features.extract.batch/1`, cross-image participants are the primary concurrency mechanism.
|
||||
OpenCV CPU teams are not multiplied blindly inside every participant.
|
||||
|
||||
## Candidate Pair normalization
|
||||
|
||||
`candidate_pair.generate/1` reconstructs:
|
||||
|
||||
- `visual_index_id`;
|
||||
- `after_feature_set_id`;
|
||||
- `top_k`;
|
||||
- `minimum_evidence_count`;
|
||||
- `scanset_filter`;
|
||||
- `exclude_same_asset`.
|
||||
|
||||
Exact earlier CPU1 and CPU12 resource shapes may be normalized in memory.
|
||||
|
||||
The current validated Candidate capability is represented by the current source/runtime contract,
|
||||
including:
|
||||
|
||||
```text
|
||||
fixed RAM 256 KiB
|
||||
per-item RAM 8 MiB
|
||||
batch range 1..64
|
||||
CPU/batch coupled during useful scaling
|
||||
GPU none
|
||||
```
|
||||
|
||||
The registry does not reinterpret the old durable checkpoint as if it had been created with those
|
||||
current values.
|
||||
|
||||
## Matcher reconstruction
|
||||
|
||||
`matcher.run/1` reconstructs the immutable Matcher configuration and durable
|
||||
`after_candidate_pair_id` cursor.
|
||||
|
||||
Project DB v11 introduced the typed Matcher payload after Match Result persistence in v10.
|
||||
|
||||
### Durable backend-class compatibility
|
||||
|
||||
The reconstruct path accepts only explicitly supported exact historical/current resource signatures.
|
||||
|
||||
These include the validated classes required for:
|
||||
|
||||
- current CPU operation;
|
||||
- current ORB AUTO/MIXED policy;
|
||||
- explicit Vulkan operation;
|
||||
- exact historical CPU/Vulkan restart shapes.
|
||||
|
||||
A neighboring signature fails rather than being treated as a backend hint.
|
||||
|
||||
No hardware identity is persisted in `matcher_tasks`.
|
||||
|
||||
### ORB AUTO
|
||||
|
||||
Only the AUTO/MIXED durable class reconstructs the shared GPU-first policy.
|
||||
|
||||
On a portable build with no usable Vulkan backend, the same durable policy can reconstruct while
|
||||
exposing only the CPU capability.
|
||||
|
||||
Restoring a fixed CPU or fixed Vulkan historical Task does not mutate shared AUTO availability state.
|
||||
|
||||
Backend probing/initialization does not occur on the `project_open()` thread. Actual backend begin
|
||||
belongs to admitted Queue execution.
|
||||
|
||||
### Current useful ORB bounds
|
||||
|
||||
The current production AUTO policy retains:
|
||||
|
||||
```text
|
||||
GPU-first when eligible
|
||||
CPU fallback complete
|
||||
normal Vulkan inflight depth = 1
|
||||
validated private safety depth = 2
|
||||
normal useful Vulkan batch <= 8
|
||||
```
|
||||
|
||||
Depth 2 and Matcher batch 12 remain validated private/safety or benchmark capabilities where
|
||||
applicable, but did not meet the useful-throughput deadband for normal AUTO policy.
|
||||
|
||||
Those performance decisions do not change:
|
||||
|
||||
- `matcher.run/1`;
|
||||
- Matcher fingerprint;
|
||||
- Match Result identity;
|
||||
- Match File bytes;
|
||||
- durable cursor.
|
||||
|
||||
## Geometric Verifier reconstruction
|
||||
|
||||
`geometric_verifier.run/1` reloads:
|
||||
|
||||
- immutable Fundamental verifier configuration;
|
||||
- validated verifier fingerprint;
|
||||
- `after_match_result_id`.
|
||||
|
||||
Project DB v13 adds only its typed Task payload.
|
||||
|
||||
The exact historical serial resource descriptor is recognized for compatibility and may be normalized
|
||||
ephemerally to the current outer-parallel capability.
|
||||
|
||||
The current validated execution shape allows:
|
||||
|
||||
- up to 8 useful participants;
|
||||
- a safe admitted window up to 16 Match Results;
|
||||
- 8 MiB per admitted parent;
|
||||
- owner-only ordered publication.
|
||||
|
||||
The scientific USAC/MAGSAC inner solver remains in its validated serial configuration. Outer
|
||||
cross-item concurrency does not change verifier science.
|
||||
|
||||
## Track Builder reconstruction
|
||||
|
||||
`track_builder.run/1` reconstructs the explicit immutable scope from:
|
||||
|
||||
- Project DB v15 typed Task payload;
|
||||
- validated little-endian scope asset;
|
||||
- exact builder selector/fingerprint;
|
||||
- exact GVR scope identity.
|
||||
|
||||
Corruption, unsupported version, fingerprint mismatch, checksum mismatch, non-canonical ordering,
|
||||
duplicates or invalid scope identity make the Task non-reconstructible.
|
||||
|
||||
Track Builder restart does not invent a scope from current Project DB contents.
|
||||
|
||||
## Sparse SfM reconstruction
|
||||
|
||||
```text
|
||||
sparse_sfm.run/1
|
||||
```
|
||||
|
||||
is **PASS/FROZEN**.
|
||||
|
||||
It reloads the explicit Project DB v17 payload, restores the persisted generic Task estimate and
|
||||
replays the frozen Sparse SfM D/E execution from explicit Track Set and calibration-scope references.
|
||||
|
||||
F0 is recomputed from the scientific payload.
|
||||
|
||||
The generic checkpoint remains version 1.
|
||||
|
||||
The historical fixed CPU1/batch1 Sparse SfM estimate is part of that frozen Task contract. It is not a
|
||||
global argument for serializing unrelated modern Task Kinds.
|
||||
|
||||
## Incremental reconstruction
|
||||
|
||||
```text
|
||||
incremental_reconstruction.run/1
|
||||
```
|
||||
|
||||
is **PASS/FROZEN**.
|
||||
|
||||
It reloads the Project DB v18 payload containing:
|
||||
|
||||
- predecessor reconstruction;
|
||||
- extension Track Set;
|
||||
- calibration scope;
|
||||
- Phase H fingerprint.
|
||||
|
||||
Restart recomputes from those immutable inputs.
|
||||
|
||||
No solver state, generic DAG or hidden dependency edge is persisted.
|
||||
|
||||
## Generic checkpoint and Project DB boundary
|
||||
|
||||
The generic checkpoint codec remains version 1.
|
||||
|
||||
Project DB stores Task Kind and Task Kind version in the generic Task summary.
|
||||
|
||||
Rows migrated from the oldest untyped schema remain:
|
||||
|
||||
```text
|
||||
task_kind = NULL
|
||||
task_kind_version = NULL
|
||||
```
|
||||
|
||||
and are classified as:
|
||||
|
||||
```text
|
||||
LEGACY_UNTYPED
|
||||
```
|
||||
|
||||
They remain inspectable but cannot be reconstructed as a typed production Task.
|
||||
|
||||
Unknown kinds are classified separately from unsupported versions.
|
||||
|
||||
A future unknown Task Kind or unsupported Task Kind version must never trigger guessed code execution.
|
||||
|
||||
No function address is persisted.
|
||||
|
||||
## Project-open recovery
|
||||
|
||||
`project_open()` uses the immutable production registry to restore eligible Tasks.
|
||||
|
||||
The recovery flow is conceptually:
|
||||
|
||||
```text
|
||||
load bounded Task page
|
||||
-> copy durable record outside DB mutex
|
||||
-> validate checkpoint and summary
|
||||
-> classify Task Kind/version
|
||||
-> reconstruct Task-specific binding
|
||||
-> apply exact legacy resource normalization when allowed
|
||||
-> restore Lardon3DTask
|
||||
-> enqueue through normal Queue
|
||||
```
|
||||
|
||||
A full Queue window stops further recovery without mutating unprocessed Tasks.
|
||||
|
||||
Task-local failures remain task-local unless the project schema/identity itself is invalid.
|
||||
|
||||
## Resource feedback is not scientific state
|
||||
|
||||
Private feedback may record bounded throughput observations per Task Kind/backend.
|
||||
|
||||
It may influence a later resource contract only where the capability contract permits adaptation.
|
||||
|
||||
It must not modify:
|
||||
|
||||
- scientific inputs;
|
||||
- Task-specific durable cursor;
|
||||
- Project DB identity;
|
||||
- Task Kind/version;
|
||||
- generic checkpoint codec;
|
||||
- scientific fingerprint;
|
||||
- canonical result ordering.
|
||||
|
||||
Only successfully completed durable work counts as useful work for adaptive feedback where that Task
|
||||
contract requires it.
|
||||
|
||||
Uncertain publication does not become a successful scientific sample.
|
||||
|
||||
## Scratch boundary
|
||||
|
||||
The current production registry contains sixteen Task Kinds, but none has an authoritative
|
||||
scratch-consuming scientific Task contract.
|
||||
|
||||
```text
|
||||
CURRENT_PRODUCTION_TASK_KINDS=16
|
||||
CURRENT_SCRATCH_CONSUMING_TASK_KINDS=0
|
||||
```
|
||||
|
||||
The external SSD controller and Resource Governor scratch wrappers remain operational infrastructure.
|
||||
|
||||
Scratch, swap and zram do not enlarge RAM admission.
|
||||
|
||||
A future scratch consumer requires an explicit Task-specific eligibility, lifetime and cleanup
|
||||
contract. Merely registering the SSD does not change a Task Kind.
|
||||
|
||||
## What the registry does not own
|
||||
|
||||
The registry does not own:
|
||||
|
||||
- Queue scheduling;
|
||||
- resource admission;
|
||||
- CPU topology;
|
||||
- GPU selection;
|
||||
- scratch allocation policy;
|
||||
- Project DB migration;
|
||||
- scientific fingerprint definitions;
|
||||
- artifact path resolution;
|
||||
- dependency/DAG planning;
|
||||
- autosave timers;
|
||||
- orphan reconciliation.
|
||||
|
||||
It only provides typed durable dispatch and reconstruction ownership.
|
||||
|
||||
## Current intentionally unfinished work
|
||||
|
||||
These remain outside the registry contract:
|
||||
|
||||
- global orphan-file reconciliation;
|
||||
- generic dependency/DAG scheduling;
|
||||
- any future new Task Kind not explicitly added to the production array;
|
||||
- future resource-capability changes requiring new proof.
|
||||
|
||||
Existing reconstructible Task Kinds already checkpoint at their Task-specific durable boundaries.
|
||||
The registry must not move those cursors forward with an independent autosave timer.
|
||||
|
||||
## Current checkpoint evidence
|
||||
|
||||
The current registry inventory includes both additive selected-execution kinds:
|
||||
|
||||
```text
|
||||
raw.develop.batch/1
|
||||
features.extract.batch/1
|
||||
```
|
||||
|
||||
The retained A6000 checkpoint is:
|
||||
|
||||
```text
|
||||
real-a6000-pre-sfm-2026-09-02
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
That proof exercised the current upstream/downstream registry through:
|
||||
|
||||
- selected RAW batch;
|
||||
- selected Feature batch;
|
||||
- Visual Index;
|
||||
- Candidate Pair;
|
||||
- Matcher;
|
||||
- Geometric Verifier v3;
|
||||
- Track Builder;
|
||||
|
||||
and stopped before real Sparse SfM.
|
||||
|
||||
The current production inventory remains exactly sixteen Task Kinds.
|
||||
|
||||
## Summary
|
||||
|
||||
```text
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
CURRENT_PRODUCTION_TASK_KINDS=16
|
||||
CURRENT_SCRATCH_CONSUMING_TASK_KINDS=0
|
||||
|
||||
TASK_KIND_REGISTRY_STATUS=IMPLEMENTED
|
||||
TASK_KIND_REGISTRY_CAPACITY=64_DESCRIPTORS
|
||||
GENERIC_CHECKPOINT_VERSION=1
|
||||
|
||||
RAW_BATCH_TASK=raw.develop.batch/1
|
||||
FEATURE_BATCH_TASK=features.extract.batch/1
|
||||
|
||||
COMPUTE_GOVERNOR_V2=PASS/FROZEN
|
||||
ORB_VULKAN_ASYNC_EXECUTION=PASS/FROZEN
|
||||
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -1,526 +1,450 @@
|
|||
# Track Model v1
|
||||
|
||||
## Scope
|
||||
## Status
|
||||
|
||||
Track Model v1 est le contrat persistant qui transforme les correspondances
|
||||
géométriquement vérifiées en structures multi-view cohérentes. Il stocke des
|
||||
ensembles d'observations 2D liées à un même point physique supposé. Il ne
|
||||
calcule rien, ne triangule pas, ne contient aucune coordonnée 3D et ne résout
|
||||
aucun conflit. Le Track Builder, la triangulation, le Sparse SfM et le Bundle
|
||||
Adjustment sont des étapes séparées ; Gate E a gelé le Builder v1 sans
|
||||
implémenter ces étapes 3D.
|
||||
```text
|
||||
TRACK_MODEL_V1=FROZEN
|
||||
TRACK_BUILDER_V1=PASS/FROZEN
|
||||
|
||||
CURRENT_PRODUCTION_VERIFIER=FUNDAMENTAL_V3
|
||||
SPARSE_SFM_CAPABILITY=IMPLEMENTED_THROUGH_GATE_G
|
||||
REAL_S21_SPARSE_SFM=NOT_EXECUTED
|
||||
REAL_A6000_SPARSE_SFM=NOT_EXECUTED
|
||||
|
||||
REAL_S21_TRACKS=PASS/FROZEN
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
Track Model v1 is the persistent scientific contract for coherent multi-view 2D observation sets.
|
||||
|
||||
A Track is **not** a 3D point.
|
||||
|
||||
It contains no camera pose, triangulated coordinate, reprojection error or Bundle Adjustment state.
|
||||
|
||||
Track Builder v1 constructs Tracks from verified Geometric Verification Results. Sparse SfM consumes an
|
||||
immutable Track Set later.
|
||||
|
||||
## Pipeline position
|
||||
|
||||
Current pipeline:
|
||||
|
||||
```text
|
||||
Feature Set
|
||||
-> Candidate Pair
|
||||
-> Match Result
|
||||
-> Geometric Verifier v3
|
||||
-> Geometric Verification Result
|
||||
-> Track Builder v1
|
||||
-> Track Model v1
|
||||
-> Sparse SfM capability
|
||||
```
|
||||
|
||||
Sparse SfM Gates C through G are implemented and frozen.
|
||||
|
||||
The retained S21 and A6000 historical campaigns stop before real Sparse SfM because known calibration
|
||||
data is unavailable for those campaigns.
|
||||
|
||||
Older Track Model text that called Sparse SfM "future" describes historical lifecycle, not current
|
||||
implementation status.
|
||||
|
||||
## Track definition
|
||||
|
||||
Un **Track** est un ensemble d'observations 2D cohérentes d'un même point
|
||||
physique supposé, observé à travers plusieurs images. Chaque observation est
|
||||
identifiée par `(feature_set_id, feature_index)`.
|
||||
A Track is a coherent set of 2D observations believed to correspond to the same physical scene point
|
||||
across multiple images.
|
||||
|
||||
Un Track n'est **pas** un point 3D. Il ne contient aucune coordonnée 3D,
|
||||
aucune erreur de reprojection, aucun statut de triangulation. La
|
||||
triangulation appartient à une étape ultérieure.
|
||||
|
||||
La chaîne scientifique correcte est :
|
||||
Observation identity is exactly:
|
||||
|
||||
```text
|
||||
Matcher → Match Result → Geometric Verification → Track Builder v1
|
||||
→ Track Model → Sparse SfM (futur)
|
||||
```
|
||||
|
||||
Le Matcher ne produit pas les Tracks. Le Track Builder v1 les assemble à partir
|
||||
des Geometric Verification Results.
|
||||
|
||||
## Observation identity
|
||||
|
||||
Une observation est identifiée par :
|
||||
|
||||
```
|
||||
(feature_set_id, feature_index)
|
||||
```
|
||||
|
||||
- `feature_set_id` : identifiant SQLite AUTOINCREMENT du Feature Set. Le
|
||||
Feature Set porte directement `image_id` comme colonne NOT NULL FK. L'image
|
||||
est dérivable par `SELECT image_id FROM feature_sets WHERE feature_set_id=?`.
|
||||
- `feature_index` : ordinal zero-based dans le tableau de keypoints du Feature
|
||||
File, stable tant que le Feature Set existe. Un Feature Set publié est
|
||||
immutable : aucune API de production ne modifie ses colonnes après INSERT.
|
||||
`feature_set_id` identifies one immutable Feature Set.
|
||||
|
||||
L'identité `(feature_set_id, feature_index)` est suffisante. Il est inutile
|
||||
de porter `image_id` dans la table d'observations car il est dérivable via
|
||||
`feature_sets.image_id`.
|
||||
`feature_index` is the zero-based keypoint ordinal inside that immutable Feature File.
|
||||
|
||||
Note : `feature_sets` ne possède pas de colonne d'état. L'existence d'une
|
||||
ligne publiée dans la table constitue le contrat réel de disponibilité du
|
||||
Feature Set.
|
||||
The Feature Set directly owns `image_id`; image identity is therefore derivable and is not duplicated
|
||||
in Track observation identity.
|
||||
|
||||
## Scientific inputs
|
||||
## Scientific input
|
||||
|
||||
Les Tracks sont construits exclusivement à partir de :
|
||||
Track Builder consumes only completed verified geometric results selected by one exact verifier
|
||||
identity.
|
||||
|
||||
```
|
||||
Geometric Verification Result
|
||||
status == GEOMETRIC_VERIFIED (2)
|
||||
```
|
||||
|
||||
correspondant exactement au VERIFICATION_SELECTOR du Track Set.
|
||||
|
||||
Pour chaque résultat vérifié, les entrées du Match File dont le bit
|
||||
correspondant dans le masque d'inliers vaut 1 fournissent les correspondances
|
||||
valides. La chaîne de dérivation est :
|
||||
Current production lineage:
|
||||
|
||||
```text
|
||||
GVR → match_result_id
|
||||
→ candidate_pair + feature_set_id_a + feature_set_id_b
|
||||
→ Match File entry[i] = (feature_index_a, feature_index_b, distance)
|
||||
→ bit i du masque d'inliers = 1
|
||||
→ observation A: (feature_set_id_a, feature_index_a)
|
||||
→ observation B: (feature_set_id_b, feature_index_b)
|
||||
verifier_kind = FUNDAMENTAL
|
||||
verifier_version = 3
|
||||
verifier_fingerprint =
|
||||
6944a471d611d8ffc59dac7cf15a5b79b97e2371d4c51785c477d68c1577f74c
|
||||
```
|
||||
|
||||
Un `GEOMETRIC_REJECTED` ne produit aucun track. Un Match Result non vérifié
|
||||
géométriquement ne suffit pas.
|
||||
Historical Track Sets created from Fundamental verifier v1 or v2 remain valid historical scientific
|
||||
objects.
|
||||
|
||||
## VERIFICATION_SELECTOR
|
||||
They must not be relabelled as v3.
|
||||
|
||||
Le VERIFICATION_SELECTOR définit la configuration de Geometric Verification
|
||||
éligible pour un Track Set. Il est stocké sur le Track Set et fait partie de
|
||||
son identité de reuse.
|
||||
For each selected `GEOMETRIC_VERIFIED` result, only Match File entries whose corresponding inlier-mask
|
||||
bit is one contribute observation edges.
|
||||
|
||||
```
|
||||
(
|
||||
verifier_kind INTEGER, -- ex: 1 = FUNDAMENTAL
|
||||
verifier_version INTEGER,
|
||||
parameter_fingerprint BLOB(32)
|
||||
)
|
||||
A rejected GVR contributes no Track edge.
|
||||
|
||||
## Verification selector
|
||||
|
||||
A Track Set stores the exact verifier selector:
|
||||
|
||||
```text
|
||||
verifier_kind
|
||||
verifier_version
|
||||
verifier_fingerprint
|
||||
```
|
||||
|
||||
Le Track Builder ne consomme que les GVR avec `status == GEOMETRIC_VERIFIED`
|
||||
correspondant exactement à ce tuple. Aucune sélection par timestamp, "latest"
|
||||
ou ordre d'insertion n'est permise.
|
||||
The builder never selects verification evidence using:
|
||||
|
||||
Valeur production : `(1, 1, SHA-256 de l'encodage canonique 84 octets)`.
|
||||
- timestamp;
|
||||
- "latest";
|
||||
- insertion order;
|
||||
- approximate fingerprint match.
|
||||
|
||||
## INPUT_SCOPE
|
||||
The current default producer is v3, but the Track Model remains version-independent and can store valid
|
||||
sets from explicitly supported historical selectors.
|
||||
|
||||
L'INPUT_SCOPE représente l'ensemble scientifique réel des entrées consommées
|
||||
par une Track Generation donnée. Il est distinct du VERIFICATION_SELECTOR :
|
||||
le selector dit quels GVR sont admissibles, le scope dit quels GVR ont
|
||||
effectivement été consommés.
|
||||
## Input scope
|
||||
|
||||
```
|
||||
input_scope_hash BLOB(32) -- SHA-256 canonique
|
||||
gvr_count INTEGER -- nombre de GVR consommés
|
||||
A Track Set also records the exact consumed GVR scope.
|
||||
|
||||
Canonical scope identity uses:
|
||||
|
||||
```text
|
||||
domain: L3DTSIS1
|
||||
items: geometric_verification_result_id
|
||||
order: strictly increasing
|
||||
encoding: uint64 little-endian
|
||||
digest: SHA-256
|
||||
```
|
||||
|
||||
### INPUT_SCOPE_HASH
|
||||
Conceptually:
|
||||
|
||||
| Propriété | Valeur |
|
||||
|-----------|--------|
|
||||
| Domain/version | `L3DTSIS1` (8 octets ASCII) |
|
||||
| Items | `geometric_verification_result_id` des GVR consommés |
|
||||
| Canonical ordering | IDs triés par ordre croissant |
|
||||
| Serialization | Chaque ID : 8 octets little-endian |
|
||||
| Digest | SHA-256 |
|
||||
| DB-local IDs | OUI — le reuse est scoped à une DB projet |
|
||||
| Duplicate handling | Inutile — les IDs sont uniques par construction |
|
||||
| Empty scope | Interdit — un Track Set sans GVR n'a pas de sens |
|
||||
```text
|
||||
SHA-256(L3DTSIS1 || id_0 || id_1 || ... || id_N)
|
||||
```
|
||||
|
||||
Le digest est calculé sur `L3DTSIS1` (8 octets) suivi des IDs sérialisés :
|
||||
`SHA-256(L3DTSIS1 || id_0 || id_1 || ... || id_N)` où chaque `id_i` est
|
||||
8 octets little-endian et les IDs sont triés par ordre croissant.
|
||||
The scope is Project-DB-local because SQLite GVR IDs participate directly.
|
||||
|
||||
Le `gvr_count` est stocké comme métadonnée de validation. Il permet de
|
||||
détecter un scope incomplet sans re-hasher. Il ne fait pas partie du hash
|
||||
lui-même.
|
||||
`gvr_count` is retained as validation metadata.
|
||||
|
||||
Le scope_hash est DB-local : il utilise les `geometric_verification_result_id`
|
||||
SQLite. Deux DB distinctes avec les mêmes données produiront des IDs
|
||||
différents. Le reuse est donc scoped à une seule DB projet.
|
||||
An empty scope is invalid.
|
||||
|
||||
## Track membership invariants
|
||||
|
||||
1. **Minimum structurel** : un Track contient au moins 2 observations.
|
||||
Une seule observation ne constitue aucune relation multi-view. Le futur
|
||||
Track Builder v1, la triangulation ou le Sparse SfM pourront appliquer des
|
||||
critères plus stricts. Le Model ne fixe pas de plafond de reconstruction.
|
||||
### Minimum size
|
||||
|
||||
2. **One observation per image** : un Track ne contient pas deux observations
|
||||
issues de la même image. Cette contrainte est validée par l'API lors de la
|
||||
création. Le schéma v1 ne dénormalise pas `image_id` dans
|
||||
`track_observations` ; l'API vérifie déterministement la relation via
|
||||
`feature_sets.image_id` avant publication sous `BEGIN IMMEDIATE`.
|
||||
A Track has at least two observations.
|
||||
|
||||
**SQL** : non protégé (pas de colonne `image_id` dans `track_observations`).
|
||||
**API** : validation par jointure `feature_sets.image_id` avant INSERT.
|
||||
### At most one observation per image
|
||||
|
||||
3. **Observation unique across tracks** : dans un même Track Set, une
|
||||
observation `(feature_set_id, feature_index)` n'appartient qu'à un seul
|
||||
Track.
|
||||
One Track cannot contain two observations derived from the same image.
|
||||
|
||||
**SQL** : `PRIMARY KEY(track_set_id, feature_set_id, feature_index)` sur
|
||||
`track_observations`. Le `track_set_id` est dénormalisé depuis `tracks`.
|
||||
**API** : validation que `track_set_id` correspond au `track_set_id` du
|
||||
`track_id` parent.
|
||||
This is validated through `feature_sets.image_id`.
|
||||
|
||||
4. **Feature Set existence** : chaque `feature_set_id` référencé existe dans
|
||||
la table `feature_sets`. La FK SQLite garantit la référence.
|
||||
### Observation uniqueness inside one Track Set
|
||||
|
||||
**SQL** : `REFERENCES feature_sets(feature_set_id)`.
|
||||
Within one Track Set:
|
||||
|
||||
5. **Feature index bounds** : `feature_index < feature_sets.feature_count`
|
||||
pour l'observation correspondante.
|
||||
```text
|
||||
(feature_set_id, feature_index)
|
||||
```
|
||||
|
||||
**SQL** : `CHECK(feature_index >= 0)`.
|
||||
**API** : validation de la borne supérieure via `feature_sets.feature_count`
|
||||
(SQLite CHECK ne peut pas référencer une autre table).
|
||||
belongs to at most one Track.
|
||||
|
||||
The persistence schema enforces this using the Track Set-scoped primary key.
|
||||
|
||||
### Feature Set existence
|
||||
|
||||
Every referenced Feature Set must exist.
|
||||
|
||||
### Feature index bound
|
||||
|
||||
For every observation:
|
||||
|
||||
```text
|
||||
0 <= feature_index < feature_count
|
||||
```
|
||||
|
||||
The upper bound is validated against the referenced Feature Set.
|
||||
|
||||
### Parent consistency
|
||||
|
||||
The denormalized Track Set ID carried by an observation must equal the Track Set of its parent Track.
|
||||
|
||||
## Track identity
|
||||
|
||||
Un Track persistant possède un identifiant opaque :
|
||||
Persistent Track identity is the opaque SQLite:
|
||||
|
||||
```
|
||||
track_id INTEGER PRIMARY KEY AUTOINCREMENT CHECK(track_id > 0)
|
||||
```text
|
||||
track_id
|
||||
```
|
||||
|
||||
Il n'a pas d'identité scientifique dérivée de son contenu en v1. Les raisons :
|
||||
Track Model v1 does not define a content-derived Track hash.
|
||||
|
||||
- un hash de membership rendrait les INSERTs dépendants de l'ordre ;
|
||||
- le contenu d'un track peut être reconstruit depuis les GVR sources ;
|
||||
- un `track_id` opaque suffit pour la persistance, le référencement et la
|
||||
pagination ;
|
||||
- la corruption est détectée par cohérence interne (doublons, images
|
||||
manquantes, index hors bornes) plutôt que par re-hash.
|
||||
Reproducibility and reuse are owned by the Track Set identity, builder configuration and exact input
|
||||
scope.
|
||||
|
||||
La reproductibilité est assurée au niveau du Track Set (parent), pas du Track
|
||||
individuel.
|
||||
## Track Set identity
|
||||
|
||||
## Track Set / Generation
|
||||
A Track Set is one complete immutable generation.
|
||||
|
||||
Un **Track Set** est le parent obligatoire de tout Track persistant. Il
|
||||
représente une génération complète de Track Building.
|
||||
Its reuse identity contains:
|
||||
|
||||
Champs :
|
||||
|
||||
```
|
||||
track_set_id INTEGER PK AUTOINCREMENT
|
||||
builder_kind TEXT(1..64)
|
||||
builder_version INTEGER > 0
|
||||
parameter_fingerprint BLOB(32)
|
||||
verifier_kind INTEGER -- VERIFICATION_SELECTOR
|
||||
verifier_version INTEGER
|
||||
verifier_fingerprint BLOB(32)
|
||||
input_scope_hash BLOB(32)
|
||||
gvr_count INTEGER >= 1
|
||||
track_count INTEGER >= 0
|
||||
created_at INTEGER >= 0
|
||||
```text
|
||||
builder_kind
|
||||
builder_version
|
||||
builder_parameter_fingerprint
|
||||
verifier_kind
|
||||
verifier_version
|
||||
verifier_fingerprint
|
||||
input_scope_hash
|
||||
```
|
||||
|
||||
### Identité de reuse
|
||||
`gvr_count` validates the scope metadata but is not an independent reuse discriminator.
|
||||
|
||||
```
|
||||
(
|
||||
builder_kind,
|
||||
builder_version,
|
||||
parameter_fingerprint,
|
||||
verifier_kind,
|
||||
verifier_version,
|
||||
verifier_fingerprint,
|
||||
input_scope_hash
|
||||
)
|
||||
```
|
||||
`INSERT OR REPLACE` is forbidden.
|
||||
|
||||
`gvr_count` est stocké comme métadonnée de validation mais ne fait pas
|
||||
partie de l'identité de reuse. Le même `input_scope_hash` avec un `gvr_count`
|
||||
différent indiquerait une corruption (hash cohérent mais nombre de sources
|
||||
incohérent).
|
||||
An exact existing immutable set is reused.
|
||||
|
||||
Un set existant avec cette identité exacte est réutilisé. `INSERT OR REPLACE`
|
||||
est interdit.
|
||||
A scientifically different scope/configuration creates a new Track Set.
|
||||
|
||||
### Immutabilité
|
||||
## Immutability
|
||||
|
||||
Un Track Set publié est **immutable**. Aucune opération d'append, remove ou
|
||||
merge n'est permise sur un track ou un set existant.
|
||||
A published Track Set is immutable.
|
||||
|
||||
L'invalidation scientifique (nouvelle entrée, nouveau scope, nouvelle
|
||||
configuration) produit un nouveau Track Set. Le set précédent reste intact.
|
||||
No production operation:
|
||||
|
||||
La suppression référentielle utilise `ON DELETE CASCADE` : supprimer un
|
||||
Track Set supprime ses tracks et observations.
|
||||
- appends to it;
|
||||
- removes observations;
|
||||
- merges existing Tracks;
|
||||
- rewrites memberships;
|
||||
- updates it to a newer verifier version.
|
||||
|
||||
### Justification
|
||||
New evidence creates a new generation.
|
||||
|
||||
- chaque rebuild crée un nouveau set, les anciens restent intacts ;
|
||||
- plusieurs configurations peuvent coexister (expérimentation) ;
|
||||
- l'invalidation est simple : supprimer un set supprime ses tracks via
|
||||
CASCADE ;
|
||||
- la reproductibilité est portée par le fingerprint et le scope_hash ;
|
||||
- pas d'UPDATE/INSERT/MERGE sur des tracks existants ;
|
||||
- cohérent avec tous les résultats publiés existants (Feature Sets, Match
|
||||
Results, GVRs) qui sont immutables après publication.
|
||||
|
||||
Le Track Builder v1 construit en mémoire, puis publie un set complet
|
||||
dans une transaction. Aucun track n'est visible avant que le set entier soit
|
||||
validé.
|
||||
|
||||
## Immutability / incrementality
|
||||
|
||||
Un Track publié dans un set est **immutable**.
|
||||
|
||||
L'incrémentalité est gérée par création de nouveaux sets :
|
||||
|
||||
1. nouvelles images → nouveaux Match Results → nouveaux GVR → nouveau
|
||||
Track Set ;
|
||||
2. le set précédent reste valide et consultable ;
|
||||
3. le futur Sparse SfM choisira quel set consommer.
|
||||
|
||||
Cette approche est cohérente avec la philosophie Lardon3D :
|
||||
|
||||
- résultats atomiques ;
|
||||
- pas de destruction silencieuse ;
|
||||
- reprise à frontière connue ;
|
||||
- conservation de l'historique.
|
||||
Historical generations remain queryable until explicitly deleted.
|
||||
|
||||
## Persistence
|
||||
|
||||
### Conceptual schema
|
||||
Track storage was introduced by Project DB v14.
|
||||
|
||||
```sql
|
||||
CREATE TABLE track_sets(
|
||||
track_set_id INTEGER PRIMARY KEY AUTOINCREMENT
|
||||
CHECK(track_set_id > 0),
|
||||
builder_kind TEXT NOT NULL
|
||||
CHECK(length(builder_kind) > 0 AND length(builder_kind) <= 64),
|
||||
builder_version INTEGER NOT NULL CHECK(builder_version > 0),
|
||||
parameter_fingerprint BLOB NOT NULL
|
||||
CHECK(length(parameter_fingerprint) = 32),
|
||||
verifier_kind INTEGER NOT NULL CHECK(verifier_kind > 0),
|
||||
verifier_version INTEGER NOT NULL CHECK(verifier_version > 0),
|
||||
verifier_fingerprint BLOB NOT NULL
|
||||
CHECK(length(verifier_fingerprint) = 32),
|
||||
input_scope_hash BLOB NOT NULL
|
||||
CHECK(length(input_scope_hash) = 32),
|
||||
gvr_count INTEGER NOT NULL CHECK(gvr_count >= 1),
|
||||
track_count INTEGER NOT NULL CHECK(track_count >= 0),
|
||||
created_at INTEGER NOT NULL CHECK(created_at >= 0),
|
||||
UNIQUE(builder_kind, builder_version, parameter_fingerprint,
|
||||
verifier_kind, verifier_version, verifier_fingerprint,
|
||||
input_scope_hash)
|
||||
);
|
||||
Durable Track Builder Task payload persistence was added in Project DB v15.
|
||||
|
||||
CREATE TABLE tracks(
|
||||
track_id INTEGER PRIMARY KEY AUTOINCREMENT CHECK(track_id > 0),
|
||||
track_set_id INTEGER NOT NULL
|
||||
REFERENCES track_sets(track_set_id) ON DELETE CASCADE,
|
||||
observation_count INTEGER NOT NULL CHECK(observation_count >= 2)
|
||||
);
|
||||
Later schema migrations through v25 do not reinterpret Track Model v1.
|
||||
|
||||
CREATE INDEX tracks_set_idx
|
||||
ON tracks(track_set_id, track_id);
|
||||
Conceptual tables:
|
||||
|
||||
CREATE TABLE track_observations(
|
||||
track_set_id INTEGER NOT NULL,
|
||||
track_id INTEGER NOT NULL
|
||||
REFERENCES tracks(track_id) ON DELETE CASCADE,
|
||||
feature_set_id INTEGER NOT NULL
|
||||
REFERENCES feature_sets(feature_set_id),
|
||||
feature_index INTEGER NOT NULL CHECK(feature_index >= 0),
|
||||
position_in_track INTEGER NOT NULL CHECK(position_in_track >= 0),
|
||||
PRIMARY KEY(track_set_id, feature_set_id, feature_index),
|
||||
UNIQUE(track_id, position_in_track)
|
||||
);
|
||||
|
||||
CREATE INDEX track_observations_lookup_idx
|
||||
ON track_observations(feature_set_id, feature_index, track_set_id);
|
||||
```text
|
||||
track_sets
|
||||
tracks
|
||||
track_observations
|
||||
```
|
||||
|
||||
### Schema invariants
|
||||
Publication is atomic for the complete Track Set under one transaction.
|
||||
|
||||
**SQL-enforced :**
|
||||
No Track from that set becomes visible before the complete generation validates and commits.
|
||||
|
||||
- `track_observations.PRIMARY KEY(track_set_id, feature_set_id, feature_index)`
|
||||
: dans un Track Set donné, une observation n'apparaît qu'une fois. Cela
|
||||
garantit qu'une observation scientifique appartient à au plus un Track dans
|
||||
ce set.
|
||||
- `REFERENCES tracks(track_id) ON DELETE CASCADE` : l'observation appartient
|
||||
à un track existant ; supprimer le track supprime l'observation.
|
||||
- `REFERENCES feature_sets(feature_set_id)` : le Feature Set existe.
|
||||
- `CHECK(observation_count >= 2)` : minimum structurel.
|
||||
- `UNIQUE(builder_kind, builder_version, parameter_fingerprint,
|
||||
verifier_kind, verifier_version, verifier_fingerprint,
|
||||
input_scope_hash)` sur `track_sets` : identité de reuse, empêche les
|
||||
doublons de set pour une même configuration et un même scope.
|
||||
- `ON DELETE CASCADE` depuis `track_sets` : supprimer un set supprime tout.
|
||||
- `CHECK(feature_index >= 0)` : borne inférieure de l'index.
|
||||
- `UNIQUE(track_id, position_in_track)` : chaque position dans un track est
|
||||
unique. L'ordre est déterminé par le Track Builder lors de la publication.
|
||||
Rollback leaves no partial Track Set.
|
||||
|
||||
**API-enforced :**
|
||||
## Ordering
|
||||
|
||||
- `track_set_id` dans `track_observations` correspond au `track_set_id` du
|
||||
`track_id` parent. Le schéma ne comporte pas de FK composite (aucun
|
||||
précédent dans le codebase). L'API valide cette cohérence avant INSERT sous
|
||||
`BEGIN IMMEDIATE`.
|
||||
- Une seule observation par image par track. L'API valide via jointure à
|
||||
`feature_sets.image_id`.
|
||||
- `feature_index < feature_sets.feature_count`. L'API valide la borne
|
||||
supérieure.
|
||||
- `observation_count` cohérent avec le nombre réel d'observations insérées.
|
||||
- `track_count` cohérent avec le nombre réel de tracks insérés.
|
||||
- `position_in_track` contigu à partir de 0 pour chaque track.
|
||||
Track Builder publishes deterministic canonical order.
|
||||
|
||||
### Note sur la dénormalisation
|
||||
`position_in_track` is contiguous from zero.
|
||||
|
||||
`track_set_id` dans `track_observations` dénormalise une clé grandparent,
|
||||
après le même pattern utilisé par `visual_index_memberships.visual_index_id`.
|
||||
Le pattern parent-key-in-UNIQUE est déjà répandu dans le codebase. La cohérence
|
||||
repose sur le chemin d'écriture unique du Track Builder et la validation API
|
||||
sous transaction.
|
||||
The exact builder contract owns edge ordering and conflict resolution; Track Model only persists the
|
||||
validated result.
|
||||
|
||||
`track_observations.track_set_id` n'a pas de FK directe vers `track_sets`
|
||||
pour éviter un second chemin CASCADE depuis `track_sets` vers
|
||||
`track_observations` (le premier chemin passe par `tracks`). La cohérence
|
||||
est garantie par l'API sous `BEGIN IMMEDIATE`.
|
||||
No hash-table iteration order may define persistent scientific ordering.
|
||||
|
||||
## Provenance
|
||||
## Deletion
|
||||
|
||||
### Track Set provenance
|
||||
Deleting a Track Set cascades to its Tracks and observations.
|
||||
|
||||
Chaque Track Set conserve :
|
||||
A Feature Set referenced by a Track observation cannot be silently removed while the reference remains
|
||||
valid.
|
||||
|
||||
- `builder_kind`, `builder_version`, `parameter_fingerprint` : configuration
|
||||
du Track Builder ;
|
||||
- `verifier_kind`, `verifier_version`, `verifier_fingerprint` : configuration
|
||||
du Geometric Verifier consommé ;
|
||||
- `input_scope_hash`, `gvr_count` : ensemble réel des GVR consommés.
|
||||
Deletion semantics do not mutate other immutable Track Sets.
|
||||
|
||||
Ces champs suffisent pour identifier la configuration scientifique complète
|
||||
ayant produit le set.
|
||||
## Pagination and resource bounds
|
||||
|
||||
### Edge provenance (deferred)
|
||||
Track Model storage APIs are paged.
|
||||
|
||||
En v1, la provenance détaillée (quels GVR spécifiques ont contribué à quel
|
||||
track individuel) n'est pas persistée. Les raisons :
|
||||
The model does not impose an arbitrary scientific maximum Track length below the number of images that
|
||||
could legitimately observe the same point.
|
||||
|
||||
- elle peut être reconstruite en comparant les memberships du set aux GVR
|
||||
disponibles ;
|
||||
- une table `track_set_sources` volumineuse complexifie la DB sans bénéfice
|
||||
immédiat ;
|
||||
- une future version du Track Builder pourra l'ajouter dans une migration
|
||||
ultérieure.
|
||||
It does not materialize a dense image-by-image covisibility matrix.
|
||||
|
||||
## Invalidation
|
||||
Loading one Track loads that Track's observations; project-wide traversal remains paged.
|
||||
|
||||
### Invalidation scientifique
|
||||
|
||||
Un nouveau scope, une nouvelle configuration de verifier ou un nouveau
|
||||
builder produit un **nouveau** Track Set avec une identité différente. Le set
|
||||
précédent reste intact et consultable. Aucune mutation silencieuse n'est
|
||||
permise.
|
||||
|
||||
### Suppression référentielle
|
||||
|
||||
`ON DELETE CASCADE` s'applique :
|
||||
|
||||
- `track_sets` → `tracks` → `track_observations` : supprimer un set supprime
|
||||
tous ses tracks et observations ;
|
||||
- `feature_sets` → (pas de CASCADE vers `track_observations`) : la FK utilise
|
||||
le comportement par défaut (NO ACTION). Supprimer un Feature Set référencé
|
||||
par une observation est interdit tant que l'observation existe.
|
||||
|
||||
## Atomic publication
|
||||
|
||||
L'unité persistante est le Track Set complet. La publication est une seule
|
||||
transaction `BEGIN IMMEDIATE` contenant l'INSERT du set, de tous ses tracks
|
||||
et de toutes ses observations.
|
||||
|
||||
- aucun track n'est visible avant le COMMIT du set entier ;
|
||||
- un rollback ne laisse aucune ligne partielle ;
|
||||
- le `created_at` du set est le timestamp de la transaction ;
|
||||
- le `track_count` et `gvr_count` sont validés contre les INSERTs réels.
|
||||
|
||||
Le Track Builder v1 utilise le Task Runtime pour le checkpoint/reprise et le
|
||||
Resource Governor pour l'admission. Le Model ne contient aucune
|
||||
logique d'exécution.
|
||||
|
||||
## Resource bounds
|
||||
|
||||
- **Pas de plafond de longueur arbitraire** : le Model ne fixe pas de
|
||||
maximum sur le nombre d'observations par Track. Un projet avec N images
|
||||
peut produire des tracks de longueur jusqu'à N.
|
||||
- **Lecture paginée** : `list_tracks` et `list_track_sets` utilisent une
|
||||
page de 64 entrées avec curseur.
|
||||
- **Chargement borné** : load track by id charge les observations du track ;
|
||||
la taille est bornée naturellement par le nombre d'images dans le scope.
|
||||
- **Pas de chargement complet du graphe** : aucune API ne charge tous les
|
||||
tracks et toutes les observations d'un projet en une seule fois.
|
||||
- **Pas de matrice dense** : aucune matrice de co-visibilité N×N n'est
|
||||
matérialisée par le Model.
|
||||
Execution-memory strategy belongs to Track Builder, not Track Model.
|
||||
|
||||
## Corruption handling
|
||||
|
||||
Le loader doit détecter :
|
||||
A loader returns corruption rather than partial best-effort data when it detects conditions such as:
|
||||
|
||||
- track absent (`track_id` référencé mais inexistant) ;
|
||||
- observation invalide (`feature_set_id` inexistant) ;
|
||||
- duplicate observation dans un même track set ;
|
||||
- deux observations de la même image dans un même track ;
|
||||
- `feature_index` hors bornes du Feature Set ;
|
||||
- `observation_count` incohérent avec le nombre réel d'observations ;
|
||||
- `track_set_id` dans `track_observations` ne correspondant pas au
|
||||
`track_set_id` du `track_id` parent ;
|
||||
- `track_set` parent absent.
|
||||
- missing parent Track or Track Set;
|
||||
- missing Feature Set;
|
||||
- duplicate observation in one Track Set;
|
||||
- repeated image inside one Track;
|
||||
- out-of-range feature index;
|
||||
- inconsistent observation count;
|
||||
- inconsistent Track count;
|
||||
- inconsistent denormalized Track Set ID;
|
||||
- invalid/non-contiguous position ordering.
|
||||
|
||||
Toute corruption retourne `CORRUPT` sans résultat partiel.
|
||||
No loader repairs scientific identity in place.
|
||||
|
||||
## API
|
||||
## Provenance
|
||||
|
||||
L'API publique implémente :
|
||||
Track Set provenance includes:
|
||||
|
||||
- `lardon3d_project_db_create_track_set()` — INSERT set + ses tracks +
|
||||
observations dans une seule transaction `BEGIN IMMEDIATE`.
|
||||
- `lardon3d_project_db_load_track_set()` — SELECT par ID.
|
||||
- `lardon3d_project_db_find_track_set()` — SELECT par identité exacte.
|
||||
- `lardon3d_project_db_list_track_sets()` — SELECT paginé ORDER BY id,
|
||||
page 64.
|
||||
- `lardon3d_project_db_load_track()` — SELECT par ID avec observations.
|
||||
- `lardon3d_project_db_list_tracks()` — SELECT par set, paginé ORDER BY
|
||||
id, page 64.
|
||||
- `lardon3d_project_db_find_track_by_observation()` — recherche par
|
||||
`(feature_set_id, feature_index)` dans un set donné.
|
||||
```text
|
||||
builder identity
|
||||
verifier selector
|
||||
input scope hash
|
||||
gvr count
|
||||
```
|
||||
|
||||
La création valide en C : existence des Feature Sets, bornes des
|
||||
`feature_index`, unicité des observations, unicité image par track,
|
||||
`observation_count` cohérent, `track_set_id` cohérent. L'INSERT est
|
||||
transactionnel.
|
||||
Detailed per-edge provenance is not persisted by Track Model v1.
|
||||
|
||||
## Explicitly out of scope
|
||||
Adding such provenance later requires an explicit version/schema decision if persistent representation
|
||||
changes.
|
||||
|
||||
- Track Builder algorithmique (union-find, connected components) ;
|
||||
- triangulation ;
|
||||
- coordonnées 3D ;
|
||||
- Essential matrix ;
|
||||
- camera pose ;
|
||||
- bundle adjustment ;
|
||||
- sparse reconstruction / Sparse SfM ;
|
||||
- reprojection error ;
|
||||
- dense reconstruction ;
|
||||
- Track optimization ou merge ;
|
||||
- mutation de tracks existants ;
|
||||
- co-visibilité (matrice ou calcul) ;
|
||||
- sélection par timestamp ou "latest".
|
||||
## Current production verifier lineage
|
||||
|
||||
## Track rejected state
|
||||
Fundamental verifier v1 and v2 are historical scientific identities.
|
||||
|
||||
Le Model v1 ne persiste pas d'état Track rejected. Le Model représente des
|
||||
Tracks structurellement valides (≥ 2 observations, cohérents). Le Track Builder
|
||||
v1 décide quels candidats publier. Les candidats non publiés n'existent pas dans
|
||||
le Model ; cette séparation reste la frontière scientifique figée.
|
||||
Current new production verification uses Fundamental v3.
|
||||
|
||||
## Versioning
|
||||
V3 adds bounded preflight rejection before the unchanged scientific estimator path and has its own
|
||||
fingerprint.
|
||||
|
||||
Project DB v14 introduced the Track storage and v15 adds only durable Track
|
||||
Builder task payload persistence. `builder_version` et `verifier_version`
|
||||
décrivent indépendamment les contrats scientifiques.
|
||||
Changer un algorithme n'impose une migration DB que si la représentation
|
||||
persistante change.
|
||||
Track Builder consumes only exact matching GVR identities.
|
||||
|
||||
Therefore:
|
||||
|
||||
```text
|
||||
HISTORICAL_TRACK_SET_VERIFIER_V1=VALID
|
||||
HISTORICAL_TRACK_SET_VERIFIER_V2=VALID
|
||||
CURRENT_TRACK_SET_VERIFIER_V3=PRODUCTION
|
||||
```
|
||||
|
||||
No historical Track Set is upgraded in place.
|
||||
|
||||
## Real S21 evidence
|
||||
|
||||
The retained S21 Track proof is:
|
||||
|
||||
```text
|
||||
REAL_S21_TRACKS=PASS/FROZEN
|
||||
|
||||
Track Set observations = 2,495,768
|
||||
Tracks = 912,447
|
||||
minimum Track length = 2
|
||||
maximum Track length = 42
|
||||
mean Track length = 2.7352470883240341
|
||||
```
|
||||
|
||||
Retained digest:
|
||||
|
||||
```text
|
||||
c30eba192627bf73eaf21ff30d81038d8cc6bbf36a69226f88cdc8c37f7d74a1
|
||||
```
|
||||
|
||||
The compact memory model supersedes the older historical 18.204 GiB envelope.
|
||||
|
||||
That checkpoint did not execute real Sparse SfM.
|
||||
|
||||
## Real A6000 evidence
|
||||
|
||||
The current retained A6000 checkpoint is:
|
||||
|
||||
```text
|
||||
real-a6000-pre-sfm-2026-09-02
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
Track output:
|
||||
|
||||
```text
|
||||
Track Set 1
|
||||
Tracks 130,714
|
||||
Track observations 318,944
|
||||
duplicate obs 0
|
||||
repeated-image 0
|
||||
orphan obs 0
|
||||
```
|
||||
|
||||
The upstream v3 GV scope contained:
|
||||
|
||||
```text
|
||||
Applicable GVRs 37,805
|
||||
Verified GVRs 10,952
|
||||
Rejected GVRs 26,853
|
||||
```
|
||||
|
||||
Restart traversed the retained scope and reused the same Track Set without creating a duplicate
|
||||
scientific generation.
|
||||
|
||||
No Sparse SfM Task or Sparse Reconstruction was created.
|
||||
|
||||
## Sparse SfM relationship
|
||||
|
||||
Track Model does not perform Sparse SfM.
|
||||
|
||||
Sparse SfM capability is nevertheless implemented through Gate G.
|
||||
|
||||
Correct current statement:
|
||||
|
||||
```text
|
||||
TRACK_MODEL_OUTPUT=AVAILABLE
|
||||
SPARSE_SFM_IMPLEMENTATION=AVAILABLE
|
||||
REAL_HISTORICAL_CAMPAIGN_SPARSE_SFM=BLOCKED_BY_KNOWN_CALIBRATION_DATA
|
||||
```
|
||||
|
||||
These are separate lifecycle facts.
|
||||
|
||||
## Out of scope
|
||||
|
||||
Track Model v1 does not own:
|
||||
|
||||
- Track Builder union/find algorithm;
|
||||
- Fundamental estimation;
|
||||
- Essential estimation;
|
||||
- camera pose;
|
||||
- triangulation;
|
||||
- 3D coordinates;
|
||||
- reprojection error;
|
||||
- Bundle Adjustment;
|
||||
- dense reconstruction;
|
||||
- metric scale;
|
||||
- Track mutation/merge;
|
||||
- selection by "latest".
|
||||
|
||||
## Summary
|
||||
|
||||
```text
|
||||
TRACK_MODEL_V1=FROZEN
|
||||
TRACK_BUILDER_V1=PASS/FROZEN
|
||||
|
||||
CURRENT_PRODUCTION_VERIFIER=FUNDAMENTAL_V3
|
||||
CURRENT_VERIFIER_FINGERPRINT=6944a471d611d8ffc59dac7cf15a5b79b97e2371d4c51785c477d68c1577f74c
|
||||
|
||||
PROJECT_DB_TRACK_MODEL=v14
|
||||
PROJECT_DB_TRACK_TASK=v15
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
|
||||
REAL_S21_TRACKS=PASS/FROZEN
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
|
||||
SPARSE_SFM_CAPABILITY=IMPLEMENTED_THROUGH_GATE_G
|
||||
REAL_S21_SPARSE_SFM=NOT_EXECUTED
|
||||
REAL_A6000_SPARSE_SFM=NOT_EXECUTED
|
||||
```
|
||||
|
|
|
|||
|
|
@ -1,207 +1,430 @@
|
|||
# Visual Index v1
|
||||
|
||||
## Problème et frontière
|
||||
## Status
|
||||
|
||||
Le Visual Index transforme une collection homogène de `FeatureSet` READY en
|
||||
candidats de recherche. Il consomme exclusivement `feature_set_id` et les
|
||||
descripteurs ORB lus par le Feature Reader. Il ne fait ni matching final, ni
|
||||
ratio test, ni vérification géométrique.
|
||||
```text
|
||||
VISUAL_INDEX_V1=IMPLEMENTED
|
||||
VISUAL_INDEX_KIND=orb-lsh
|
||||
VISUAL_INDEX_VERSION=1
|
||||
|
||||
## Choix algorithmique
|
||||
CANDIDATE_PAIR=IMPLEMENTED
|
||||
MATCHER=IMPLEMENTED
|
||||
|
||||
La v1 utilise un LSH binaire déterministe à six tables. Chaque table extrait
|
||||
24 positions distinctes des 256 bits ORB. La position v1 est
|
||||
`(41*table + 11*bit) mod 256`; 11 étant premier avec 256, les 24 positions
|
||||
d'une table sont distinctes. Une clé est `(table_id, key24)`. Des descripteurs proches en
|
||||
Hamming ont une probabilité élevée de collision dans au moins une table, sans
|
||||
conversion flottante.
|
||||
VISUAL_INDEX_GPU=REJECTED_WITH_MEASURED_REASON
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
Alternatives évaluées :
|
||||
Visual Index turns a homogeneous collection of immutable Feature Sets into bounded image-retrieval
|
||||
candidates.
|
||||
|
||||
- le hash exact est très compact et déterministe, mais son rappel s'effondre
|
||||
dès qu'un descriptor varie d'un bit ;
|
||||
- le multi-index hashing avec sous-chaînes et multiprobes offre des garanties
|
||||
Hamming intéressantes, mais le nombre de postings/probes nécessaire au
|
||||
rappel utile d'ORB est trop élevé pour une v1 bornée ;
|
||||
- FLANN-LSH masque son format, ses allocations et sa stabilité de
|
||||
sérialisation, ce qui nuit à la reprise et à l'audit ;
|
||||
- BoW/IVF donne un bon retrieval image, mais impose vocabulaire, entraînement,
|
||||
identité et politique de mise à jour avant de pouvoir être incrémental ;
|
||||
- HNSW et FAISS ajoutent une dépendance et un état mutable complexes sans
|
||||
avantage décisif à quelques milliers d'images.
|
||||
It is a retrieval stage, not a Matcher and not a geometric verifier.
|
||||
|
||||
Ce LSH n'est pas un matcher. Il privilégie une base déterministe, segmentable
|
||||
et contrôlable. Une évolution de la sélection de bits exige une nouvelle
|
||||
`visual_index_version`.
|
||||
Current downstream consumers are implemented:
|
||||
|
||||
## Identité et configuration
|
||||
```text
|
||||
Feature Store
|
||||
-> Visual Index
|
||||
-> Candidate Pair Generator
|
||||
-> Matcher
|
||||
-> Geometric Verification
|
||||
-> Tracks
|
||||
```
|
||||
|
||||
Le kind est `orb-lsh`, version 1. Un index contient exclusivement des Feature
|
||||
Sets de même `descriptor_type`, dimension, `extractor_kind`, version et
|
||||
`parameter_fingerprint`. Sa configuration v1 contient :
|
||||
Older text describing Candidate Pair or Matcher as future consumers is historical design context and is
|
||||
not current status.
|
||||
|
||||
- `table_count=6` ;
|
||||
- `key_bits=24` ;
|
||||
- `max_features_per_set` entre 1 et 1024, défaut 512 ;
|
||||
- `max_bucket_postings` entre 1 et 4096, défaut 256 ;
|
||||
- `max_segments=256` ;
|
||||
- `max_feature_sets_per_segment=16`.
|
||||
## Algorithm
|
||||
|
||||
Le fingerprint de paramètres est SHA-256 des 32 octets canoniques
|
||||
`L3DVICF1`, version et cinq entiers little-endian. Aucun padding, JSON, locale
|
||||
ou endianness hôte n'intervient. `visual_index_id` est une identité SQLite
|
||||
`AUTOINCREMENT`, jamais réutilisée après publication.
|
||||
Visual Index v1 uses deterministic binary LSH over ORB descriptors.
|
||||
|
||||
## Échantillonnage
|
||||
It uses six tables.
|
||||
|
||||
Au plus `max_features_per_set` features sont indexées. La sélection v1 retient
|
||||
le préfixe de `feature_index` croissant. Les postings conservent l'indice
|
||||
original. Un Feature Set vide est membre valide sans posting. Le build Task
|
||||
peut lire en parallèle jusqu'à douze Feature Files, avec un reader et une
|
||||
tranche de 256 descripteurs privés par participant effectivement admis. Chaque
|
||||
Feature Set écrit dans une tranche privée de la capacité de postings déjà
|
||||
réservée pour le segment ; le propriétaire compacte ensuite les tranches dans
|
||||
l'ordre de sélection et applique seul l'ordre total persistant.
|
||||
Each table selects 24 distinct positions from the 256 ORB bits.
|
||||
|
||||
## Segments introduits en Project Database v6, conservés en v7
|
||||
The frozen v1 position rule is:
|
||||
|
||||
Un index logique possède des segments immuables READY. Chaque update publie un
|
||||
segment de un à seize nouveaux Feature Sets, puis ajoute atomiquement segments
|
||||
et memberships. `UNIQUE(visual_index_id,feature_set_id)` assure l'idempotence.
|
||||
Une recherche copie au début la liste bornée des segments READY, relâche le
|
||||
mutex DB, puis lit ce snapshot. Un segment commité au milieu sera visible à la
|
||||
requête suivante.
|
||||
```text
|
||||
position = (41 * table + 11 * bit) mod 256
|
||||
```
|
||||
|
||||
SQLite conserve les tables `visual_indexes`, `visual_index_segments`,
|
||||
`visual_index_memberships` et `visual_index_update_tasks`. Les gros postings
|
||||
restent hors DB. La configuration et chaque membership sont immuables. La
|
||||
compaction est `NOT_YET_WIRED`; au-delà de 256 segments une update est refusée avec
|
||||
`LARDON3D_VISUAL_INDEX_LIMIT`. Avec seize membres par segment, la capacité v1 est donc
|
||||
exactement 4096 Feature Sets par index. Le refus ne publie ni segment ni membership et
|
||||
l'index existant reste requêtable.
|
||||
A posting key is:
|
||||
|
||||
Le DDL v6 exact est `schema_visual_v6` dans `src/project_db.c`. Il impose
|
||||
`AUTOINCREMENT` aux index/segments, les uniques
|
||||
`(visual_index_id,generation)`, `(visual_index_id,sha256)` et
|
||||
`(visual_index_id,feature_set_id)`, ainsi que les FKs vers index, Feature Set,
|
||||
segment et tâche. Les CHECKS bornent tables 1..32, bits 8..32, sampling
|
||||
1..1024, bucket 1..4096, membres segment 1..16 et durabilité 0..1. La migration
|
||||
entière reste sous `BEGIN IMMEDIATE` et possède une injection de rollback v6.
|
||||
```text
|
||||
(table_id, key24)
|
||||
```
|
||||
|
||||
Changing this bit-selection policy requires a new Visual Index scientific version.
|
||||
|
||||
## Identity and configuration
|
||||
|
||||
Current kind/version:
|
||||
|
||||
```text
|
||||
orb-lsh / 1
|
||||
```
|
||||
|
||||
One index contains Feature Sets with homogeneous:
|
||||
|
||||
- descriptor type;
|
||||
- descriptor dimension;
|
||||
- extractor kind;
|
||||
- extractor version;
|
||||
- extractor parameter fingerprint.
|
||||
|
||||
Frozen v1 configuration contains:
|
||||
|
||||
```text
|
||||
table_count = 6
|
||||
key_bits = 24
|
||||
max_features_per_set = 1..1024, default 512
|
||||
max_bucket_postings = 1..4096, default 256
|
||||
max_segments = 256
|
||||
max_feature_sets_per_segment = 16
|
||||
```
|
||||
|
||||
The canonical parameter fingerprint uses domain:
|
||||
|
||||
```text
|
||||
L3DVICF1
|
||||
```
|
||||
|
||||
with explicit little-endian fields.
|
||||
|
||||
No C struct padding, locale or host endianness enters the fingerprint.
|
||||
|
||||
## Sampling
|
||||
|
||||
At most `max_features_per_set` Feature entries are indexed.
|
||||
|
||||
V1 selects the increasing `feature_index` prefix.
|
||||
|
||||
Postings retain the original Feature index.
|
||||
|
||||
An empty Feature Set is a valid member and contributes no posting.
|
||||
|
||||
## Segment persistence
|
||||
|
||||
Project DB v6 introduced Visual Index persistence. v7 retained the model.
|
||||
|
||||
Later schema versions through v25 do not reinterpret Visual Index v1.
|
||||
|
||||
A logical index owns immutable READY segments.
|
||||
|
||||
One update publishes one segment containing between one and sixteen new Feature Sets.
|
||||
|
||||
Membership uniqueness is enforced on:
|
||||
|
||||
```text
|
||||
(visual_index_id, feature_set_id)
|
||||
```
|
||||
|
||||
A query snapshots the bounded READY segment list before asset reads.
|
||||
|
||||
A segment committed after that snapshot is visible to the next query, not retroactively injected into
|
||||
the running query.
|
||||
|
||||
## Capacity
|
||||
|
||||
V1 currently allows:
|
||||
|
||||
```text
|
||||
max_segments = 256
|
||||
max_feature_sets_per_segment = 16
|
||||
```
|
||||
|
||||
Therefore one v1 index can contain exactly up to:
|
||||
|
||||
```text
|
||||
4096 Feature Sets
|
||||
```
|
||||
|
||||
before another update returns the Visual Index limit.
|
||||
|
||||
This is an index-v1 capacity bound, not a project-wide image-count limit.
|
||||
|
||||
Compaction/base-delta redesign remains deferred.
|
||||
|
||||
## Segment File v1
|
||||
|
||||
Le fichier est little-endian et ne sérialise aucune structure C. Layout :
|
||||
Segment File v1 is explicitly little-endian and does not serialize C structs.
|
||||
|
||||
| Offset | Taille | Champ |
|
||||
|---:|---:|---|
|
||||
| 0 | 8 | magic `L3DVIDX\0` |
|
||||
| 8 | 4 | format version 1 |
|
||||
| 12 | 4 | header size 128 |
|
||||
| 16 | 4 | table count |
|
||||
| 20 | 4 | key bits |
|
||||
| 24 | 8 | posting count |
|
||||
| 32 | 8 | member count |
|
||||
| 40 | 8 | postings offset, 128 |
|
||||
| 48 | 8 | total size |
|
||||
| 56 | 32 | index parameter fingerprint |
|
||||
| 88 | 32 | feature parameter fingerprint |
|
||||
| 120 | 8 | réservés, zéro |
|
||||
Magic:
|
||||
|
||||
Chaque posting fait 24 octets : `table_id:u32`, `key24:u32`,
|
||||
`feature_set_id:u64`, `feature_index:u32`, réservé zéro `u32`. L'ordre est
|
||||
`table_id`, clé, Feature Set, feature index. Le fichier exact est SHA-256 et
|
||||
vit sous `assets/visual-index/<2 hex>/<sha256 lowercase>`.
|
||||
```text
|
||||
L3DVIDX\0
|
||||
```
|
||||
|
||||
Publication : temporaire local, écriture, `fsync`, hash, `link` sans
|
||||
écrasement, validation d'une adoption concurrente, `fsync` du répertoire, puis
|
||||
transaction DB. Un échec après publication peut laisser un orphelin mais jamais
|
||||
un segment READY partiel. La durabilité distingue `DURABLE` et
|
||||
`PUBLISHED_NOT_DURABLE`.
|
||||
A posting contains:
|
||||
|
||||
Le reader vérifie le SHA avant le parsing. Un fichier au SHA et aux métadonnées cohérents
|
||||
mais portant une version future produit `UNSUPPORTED_VERSION`; les comptes et produits
|
||||
d'offset invalides produisent `CORRUPT` avant allocation, conversion ou lecture de posting.
|
||||
```text
|
||||
table_id:u32
|
||||
key24:u32
|
||||
feature_set_id:u64
|
||||
feature_index:u32
|
||||
reserved_zero:u32
|
||||
```
|
||||
|
||||
## Recherche, score et bornes
|
||||
Canonical persistent ordering is:
|
||||
|
||||
L'API est centrée sur `(visual_index_id, query_feature_set_id)`. Elle accepte
|
||||
`ANY_SCANSET`, `SAME_SCANSET` ou `OTHER_SCANSETS`, l'exclusion du même asset,
|
||||
un minimum de preuves et `top_k` entre 1 et 256. Elle ne retourne jamais le
|
||||
Feature Set ni l'image de requête.
|
||||
```text
|
||||
table_id
|
||||
key24
|
||||
feature_set_id
|
||||
feature_index
|
||||
```
|
||||
|
||||
Une preuve est un `feature_index` de requête distinct ayant au moins une
|
||||
collision avec le candidat. Plusieurs tables, postings ou descriptors du
|
||||
candidat ne multiplient pas cette preuve. Le score final vaut
|
||||
`evidence_count / sampled_query_feature_count` dans `[0,1]`. Le volume du
|
||||
candidat ne peut donc pas augmenter le score sans preuve distincte. L'ordre est
|
||||
score décroissant, preuves décroissantes, `image_id`, puis `feature_set_id`.
|
||||
The complete Segment File is content-addressed by SHA-256 under the Visual Index asset tree.
|
||||
|
||||
La burstiness est bornée par une contribution maximum par feature de requête
|
||||
et candidat. Une première passe additionne la fréquence d'un bucket sur tous
|
||||
les segments du snapshot. Au-delà de `max_bucket_postings`, il est ignoré : un motif
|
||||
très commun ne peut ni allouer une liste géante ni dominer le score. Le reader
|
||||
lit au plus 256 postings par appel. L'accumulateur contient au plus 4096
|
||||
candidats ; les nouveaux candidats sont ignorés après saturation, de manière
|
||||
déterministe par l'ordre des postings. Aucun cache global n'existe et un seul
|
||||
segment est ouvert à la fois.
|
||||
## Publication
|
||||
|
||||
Deux updates concurrentes peuvent sélectionner le même lot et construire le même asset.
|
||||
La transaction SQLite et les contraintes uniques ne laissent publier qu'un segment et
|
||||
un membership par Feature Set; l'autre update échoue/rejoue en no-op. Une query prend son
|
||||
snapshot de métadonnées avant les lectures et ouvre/ferme un seul segment à la fois, y
|
||||
compris avec 250 à 256 segments : le nombre de descripteurs de fichier reste borné.
|
||||
Publication follows the normal immutable-asset pattern:
|
||||
|
||||
## Tâche, reprise et ressources
|
||||
```text
|
||||
local temp
|
||||
-> write
|
||||
-> fsync
|
||||
-> hash
|
||||
-> no-overwrite publication/adoption validation
|
||||
-> fsync directory
|
||||
-> short Project DB transaction
|
||||
```
|
||||
|
||||
`visual_index.update`, version 1, persiste `visual_index_id` et un curseur
|
||||
`after_feature_set_id`. Une séquence traite au plus seize Feature Sets non
|
||||
indexés, publie et commit un segment, puis checkpoint. Pause et annulation sont
|
||||
coopératives entre lectures et avant publication ; un segment déjà READY reste
|
||||
valide. La reprise recommence au dernier curseur commité et l'unicité des
|
||||
memberships rend le rejeu idempotent.
|
||||
A physical file may remain orphaned if DB publication fails after the file is published.
|
||||
|
||||
**IMPLEMENTED — parallélisme interne borné.** La Queue exécute toujours un seul
|
||||
callback. L'estimation demande jusqu'à seize threads CPU, un slot I/O, GPU zéro,
|
||||
8 Mio fixes et 2 Mio par Feature Set, lot 1..16. Le callback compte comme un
|
||||
participant et crée au plus `cpu_threads - 1` enfants. Chaque enfant lit
|
||||
exclusivement des Feature Files immuables et écrit une tranche privée ; il ne
|
||||
touche ni au handle Project DB partagé, ni au fichier de segment, ni au curseur.
|
||||
Tous les enfants sont joints avant tri, sérialisation, publication asset et
|
||||
transaction SQLite.
|
||||
No partially committed READY segment is invented.
|
||||
|
||||
La réduction emploie l'ordre total v1
|
||||
`table_id,key24,feature_set_id,feature_index`. Le fichier, son SHA-256, le
|
||||
chemin, les memberships, la génération, le fingerprint et les résultats de
|
||||
requête sont donc exactement identiques à un build avec un participant. Une
|
||||
erreur de lecture dans une tranche interdit toute publication ; une création
|
||||
de thread refusée est remplacée par le calcul de cette tranche sur le callback,
|
||||
sans changer la réduction. Le curseur n'avance qu'après la publication
|
||||
transactionnelle du segment, puis le checkpoint existant reste le seul point
|
||||
de reprise Task. `record_batch` reçoit le nombre de Feature Sets réellement
|
||||
commités, la durée réelle et `peak_memory_bytes=0` (inconnue).
|
||||
Durability distinguishes:
|
||||
|
||||
## Complexité et limites
|
||||
```text
|
||||
DURABLE
|
||||
PUBLISHED_NOT_DURABLE
|
||||
```
|
||||
|
||||
Pour `D` descriptors échantillonnés, construction et disque sont `O(6D)`.
|
||||
Une requête effectue `O(6Q log P + H)` par segment (`Q<=1024`, `H` hits bornés),
|
||||
pas `O(images²)`. La mémoire build est bornée par les métadonnées, les tranches
|
||||
de 256 descripteurs privées des participants et les postings d'un segment ; les
|
||||
tranches privées partitionnent le buffer de postings existant et ne le
|
||||
dupliquent pas. Chaque participant garde au plus un reader/FD de Feature File.
|
||||
La mémoire query est bornée par 4096 candidats, 256 postings et 256 résultats.
|
||||
À 3700 images et 512 features, environ 11,4
|
||||
millions de postings sont produits. Un test structurel persiste 50 000 Feature Sets puis
|
||||
confirme la pagination par 16 et le refus propre après 4096 memberships. Un index unique
|
||||
ne couvre donc pas encore 50 000 images : le risque principal est le nombre de segments
|
||||
et les seeks. Une compaction/base+delta ou une évolution v2 sera nécessaire, sans changer
|
||||
les identités durables; elle est `NOT_YET_WIRED`.
|
||||
## Reader validation
|
||||
|
||||
Les fixtures de validation incluent un Feature Set vide, un crop réel, une rotation de
|
||||
8 degrés, deux campagnes, un asset source partagé et une attaque de motif répétitif. Ces
|
||||
tests valident le classement de candidats LSH, jamais une compatibilité géométrique.
|
||||
The reader validates the asset SHA before trusting the format.
|
||||
|
||||
## Frontière future
|
||||
It rejects:
|
||||
|
||||
Le Candidate Pair Generator pourra filtrer sur score et `evidence_count`, puis
|
||||
transmettre `feature_set_id + feature_index` au futur matcher. Le score Visual
|
||||
Index ne constitue jamais une preuve géométrique.
|
||||
- invalid counts;
|
||||
- invalid offsets;
|
||||
- overflow;
|
||||
- malformed reserved fields;
|
||||
- inconsistent DB metadata;
|
||||
- unsupported future version.
|
||||
|
||||
A coherent future format version is `UNSUPPORTED_VERSION`, not generic corruption.
|
||||
|
||||
## Query
|
||||
|
||||
Query identity is centered on:
|
||||
|
||||
```text
|
||||
(visual_index_id, query_feature_set_id)
|
||||
```
|
||||
|
||||
Options include:
|
||||
|
||||
- ScanSet filter;
|
||||
- same/other ScanSet policy;
|
||||
- source-asset exclusion;
|
||||
- minimum evidence count;
|
||||
- `top_k` in `1..256`.
|
||||
|
||||
The source Feature Set/image is never returned as its own candidate.
|
||||
|
||||
## Evidence and score
|
||||
|
||||
One evidence unit is one distinct query `feature_index` that collides with the candidate.
|
||||
|
||||
Multiple tables or candidate postings do not multiply the same query-feature evidence.
|
||||
|
||||
Score:
|
||||
|
||||
```text
|
||||
evidence_count / sampled_query_feature_count
|
||||
```
|
||||
|
||||
Range:
|
||||
|
||||
```text
|
||||
0..1
|
||||
```
|
||||
|
||||
Canonical result order:
|
||||
|
||||
```text
|
||||
score descending
|
||||
evidence_count descending
|
||||
image_id ascending
|
||||
feature_set_id ascending
|
||||
```
|
||||
|
||||
Visual Index score is retrieval evidence only.
|
||||
|
||||
It is not descriptor-match evidence and not geometric evidence.
|
||||
|
||||
## Burstiness bound
|
||||
|
||||
Bucket frequency is bounded across the retained query snapshot.
|
||||
|
||||
A bucket above `max_bucket_postings` is ignored.
|
||||
|
||||
This prevents common patterns from dominating score or creating unbounded posting accumulation.
|
||||
|
||||
The query accumulator is bounded to 4096 candidates and 256 returned results.
|
||||
|
||||
No global query cache is required.
|
||||
|
||||
## Durable Task
|
||||
|
||||
Task Kind:
|
||||
|
||||
```text
|
||||
visual_index.update/1
|
||||
```
|
||||
|
||||
Durable cursor:
|
||||
|
||||
```text
|
||||
after_feature_set_id
|
||||
```
|
||||
|
||||
One sequence handles a bounded admitted set of new Feature Sets, publishes a complete segment, commits
|
||||
memberships, advances the cursor/checkpoint and returns through `sequence_break()` if more work remains.
|
||||
|
||||
Restart resumes from the durable cursor and membership uniqueness makes replay idempotent.
|
||||
|
||||
## Internal parallelism
|
||||
|
||||
The Queue owns one active heavy callback.
|
||||
|
||||
Visual Index may use bounded internal CPU participants inside that callback.
|
||||
|
||||
Current validated shape:
|
||||
|
||||
```text
|
||||
CPU up to 16
|
||||
batch/window 1..16
|
||||
GPU 0
|
||||
fixed RAM approximately 8 MiB
|
||||
per-item RAM approximately 2 MiB
|
||||
```
|
||||
|
||||
Each participant reads immutable Feature data into private work.
|
||||
|
||||
Participants do not publish the segment.
|
||||
|
||||
After join, the owner performs canonical total ordering, serialization, asset publication and Project DB
|
||||
commit.
|
||||
|
||||
Thread-creation failure may fall back to owner computation of that slice without changing output.
|
||||
|
||||
## Determinism
|
||||
|
||||
The following must match the serial scientific result:
|
||||
|
||||
- posting set;
|
||||
- posting order;
|
||||
- Segment File bytes;
|
||||
- SHA-256;
|
||||
- membership set;
|
||||
- generation ordering;
|
||||
- query results.
|
||||
|
||||
Operational CPU width does not enter scientific identity.
|
||||
|
||||
## GPU policy
|
||||
|
||||
Current GPU classification:
|
||||
|
||||
```text
|
||||
VISUAL_INDEX_GPU=REJECTED_WITH_MEASURED_REASON
|
||||
```
|
||||
|
||||
The stage is dominated by posting construction, total ordering, hashing and deterministic publication,
|
||||
and there is no validated production GPU seam that preserves the full contract with useful measured
|
||||
benefit.
|
||||
|
||||
This does not authorize avoidable CPU serialism.
|
||||
|
||||
```text
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
```
|
||||
|
||||
## Current downstream relationship
|
||||
|
||||
Candidate Pair Generator is implemented and consumes Visual Index queries.
|
||||
|
||||
Matcher is implemented and consumes persisted Candidate Pairs.
|
||||
|
||||
Therefore current relationship is:
|
||||
|
||||
```text
|
||||
Visual Index
|
||||
-> Candidate Pair Generator
|
||||
-> Candidate Pair persistence
|
||||
-> Matcher
|
||||
```
|
||||
|
||||
Visual Index does not pass raw `feature_set_id + feature_index` pairs directly into a hypothetical
|
||||
future Matcher.
|
||||
|
||||
The persisted Candidate Pair boundary remains explicit.
|
||||
|
||||
## Real A6000 evidence
|
||||
|
||||
The retained A6000 proof contains:
|
||||
|
||||
```text
|
||||
Feature Sets 689
|
||||
Candidate Pairs 38,420
|
||||
Match Results 38,420
|
||||
```
|
||||
|
||||
Final continuation replayed no new Visual Index work.
|
||||
|
||||
Checkpoint:
|
||||
|
||||
```text
|
||||
real-a6000-pre-sfm-2026-09-02
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
This confirms the current Visual Index path was already durably reusable before downstream GV/Tracks
|
||||
continuation.
|
||||
|
||||
## Limits
|
||||
|
||||
Current v1 limits/non-goals include:
|
||||
|
||||
- no segment compaction;
|
||||
- one index limited to 4096 Feature Sets;
|
||||
- no GPU backend;
|
||||
- no geometric meaning assigned to retrieval score;
|
||||
- no dense project-wide pair matrix.
|
||||
|
||||
A future index version may change capacity or data structure only through an explicit versioned
|
||||
scientific/persistence decision.
|
||||
|
||||
## Summary
|
||||
|
||||
```text
|
||||
VISUAL_INDEX_V1=IMPLEMENTED
|
||||
VISUAL_INDEX_KIND=orb-lsh
|
||||
VISUAL_INDEX_VERSION=1
|
||||
VISUAL_INDEX_CAPACITY=4096_FEATURE_SETS
|
||||
VISUAL_INDEX_SEGMENT_MEMBERS=16
|
||||
VISUAL_INDEX_TOP_K_MAX=256
|
||||
|
||||
VISUAL_INDEX_TASK=visual_index.update/1
|
||||
VISUAL_INDEX_GPU=REJECTED_WITH_MEASURED_REASON
|
||||
|
||||
CANDIDATE_PAIR=IMPLEMENTED
|
||||
MATCHER=IMPLEMENTED
|
||||
|
||||
CURRENT_PROJECT_DB_SCHEMA=v25
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
|
|
|||
625
docs/audits/documentation_inventory.md
Normal file
625
docs/audits/documentation_inventory.md
Normal file
|
|
@ -0,0 +1,625 @@
|
|||
# Lardon3D — Documentation Inventory Audit
|
||||
|
||||
## Status
|
||||
|
||||
DOCUMENTATION_INVENTORY_AUDIT=PASS_WITH_FINDINGS
|
||||
|
||||
Audit branch: docs-audit
|
||||
|
||||
Reference checkpoints:
|
||||
|
||||
- global-maintenance-2026-09-01
|
||||
- real-a6000-pre-sfm-2026-09-02
|
||||
|
||||
This audit concerns documentation only.
|
||||
|
||||
It does not reopen scientific contracts, alter executable behavior, redefine FROZEN evidence, or authorize implementation work.
|
||||
|
||||
## Classification vocabulary
|
||||
|
||||
Documents/findings use: CURRENT_AND_COMPLETE, CURRENT_BUT_INCOMPLETE, PARTIALLY_STALE, STALE, HISTORICAL_AND_VALID, DUPLICATED_AUTHORITY, AMBIGUOUS_AUTHORITY, NEEDS_REORGANIZATION.
|
||||
|
||||
Finding severity: BLOCKING_DOC, IMPORTANT, IMPROVEMENT.
|
||||
|
||||
A BLOCKING_DOC finding means documentation could make a future agent work from the wrong project state. It does not mean executable code is known to be incorrect.
|
||||
|
||||
## Current facts
|
||||
|
||||
Current Project DB head: v25.
|
||||
|
||||
Valid historical/additive foundations remain:
|
||||
|
||||
- v22 = selected scientific execution foundation
|
||||
- v23 = generic optical-context overlay
|
||||
- v24 = raw.develop.batch persistence
|
||||
- v25 = features.extract.batch persistence
|
||||
|
||||
Current production inventory: 16 Task kinds.
|
||||
|
||||
Canonical resource policy:
|
||||
|
||||
- MAXIMUM SAFE USEFUL THROUGHPUT
|
||||
- SERIALISM_REQUIRES_PROOF
|
||||
|
||||
Reference-host evidence:
|
||||
|
||||
- 16 logical CPUs total
|
||||
- approximately 4 logical CPUs reserved for interactive host use
|
||||
- approximately 12 logical CPUs available to compute
|
||||
- approximately 3 GiB MemAvailable hard reserve
|
||||
- Radeon 780M UMA
|
||||
|
||||
These are observations of the reference host, not portable product constants.
|
||||
|
||||
Real-data checkpoints:
|
||||
|
||||
- REAL_S21_TRACKS=PASS/FROZEN
|
||||
- REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
|
||||
A6000 final pre-SfM evidence:
|
||||
|
||||
- Feature Sets: 689
|
||||
- Candidate Pairs: 38,420
|
||||
- Match Results: 38,420
|
||||
- Applicable GVR: 37,805
|
||||
- Verified GVR: 10,952
|
||||
- Rejected GVR: 26,853
|
||||
- Track Sets: 1
|
||||
- Tracks: 130,714
|
||||
- Track observations: 318,944
|
||||
- Sparse SfM: NOT EXECUTED
|
||||
- Dense/MVS: NOT EXECUTED
|
||||
|
||||
# BLOCKING_DOC findings
|
||||
|
||||
## DOC-B01 — Roadmap contains mutually exclusive current states
|
||||
|
||||
File: docs/roadmap/roadmap.md
|
||||
|
||||
The roadmap contains the acquired REAL_A6000_PRE_SFM=PASS/FROZEN result but also retains obsolete current-next material describing Project DB v24/v25 as validation in progress and instructing a resume from A6000 cursor 259 with 430 RAW representations remaining.
|
||||
|
||||
Those operations have already completed.
|
||||
|
||||
Required correction:
|
||||
|
||||
- preserve useful historical evidence;
|
||||
- mark the cursor-259 and validation-in-progress material as historical;
|
||||
- make the final A6000 pre-SfM proof authoritative;
|
||||
- define one unambiguous CURRENT NEXT state.
|
||||
|
||||
## DOC-B02 — Project Database has competing current heads
|
||||
|
||||
File: docs/architecture/project_database.md
|
||||
|
||||
Different sections present v23, v24 and v25 as current state.
|
||||
|
||||
The actual current schema head is v25.
|
||||
|
||||
v22, v23 and v24 remain valid historical and additive contracts. They must not be erased or rewritten as though they never existed.
|
||||
|
||||
Required correction:
|
||||
|
||||
- state once near the top that current Project DB head is v25;
|
||||
- preserve v22 selected execution, v23 optics and v24 RAW batch as historical additive foundations;
|
||||
- update v25 Feature batch lifecycle to match acquired real evidence.
|
||||
|
||||
## DOC-B03 — Persistence advertises v23 as current
|
||||
|
||||
File: docs/architecture/persistence.md
|
||||
|
||||
The document currently states that the current schema is v23.
|
||||
|
||||
Required correction:
|
||||
|
||||
- current schema = v25;
|
||||
- retain historical v7, v16, v22 and v23 contracts;
|
||||
- add a concise v24 RAW batch and v25 Feature batch persistence summary.
|
||||
|
||||
## DOC-B04 — Resource-aware pipeline describes superseded serial Features
|
||||
|
||||
File: docs/architecture/resource_aware_pipeline.md
|
||||
|
||||
Current prose still describes Feature Extraction as one durable task per image with batch one and one worker as the complete operational model.
|
||||
|
||||
That description is incomplete after features.extract.batch/1, Project DB v25, bounded cross-image participants, coupled CPU/batch scaling and owner-only ordered publication.
|
||||
|
||||
The document also retains a fourteen-Task current inventory even though production now contains sixteen Task kinds.
|
||||
|
||||
Required correction:
|
||||
|
||||
- preserve per-image Feature scientific atomicity;
|
||||
- document the v25 cross-image batch path;
|
||||
- distinguish per-item atomicity from cross-item serialism;
|
||||
- update current Task count to sixteen.
|
||||
|
||||
## DOC-B05 — Development testing guide uses obsolete API examples
|
||||
|
||||
File: docs/development/testing.md
|
||||
|
||||
The guide contains examples based on old-style names including task_estimate_t, task_t, task_create, TASK_STATE_IDLE and g_test_*.
|
||||
|
||||
These examples do not represent the current repository API and test style.
|
||||
|
||||
Because this is a developer instruction document rather than historical evidence, stale examples are dangerous.
|
||||
|
||||
It also retains repeated wipe-build and fixed -j8 assumptions that conflict with current delta-validation and host-aware parallelism policy.
|
||||
|
||||
Required correction:
|
||||
|
||||
- rewrite examples against actual current repository tests and public APIs;
|
||||
- use current Meson targets and validation discipline;
|
||||
- preserve exact sanitizer qualifications;
|
||||
- avoid mandatory repeated full rebuilds when the delta does not justify them.
|
||||
|
||||
## DOC-B06 — Sparse SfM opening lifecycle is historical but presented as current
|
||||
|
||||
File: docs/architecture/sparse_sfm.md
|
||||
|
||||
The opening still describes numerical Sparse SfM as deferred to Gate C and later gates.
|
||||
|
||||
Later sections and other canonical documents establish Gates C, D, E, F and G as acquired.
|
||||
|
||||
Required correction:
|
||||
|
||||
- retain Gate A and Gate B history;
|
||||
- label early lifecycle wording as historical;
|
||||
- add a concise current lifecycle summary;
|
||||
- distinguish implemented Sparse SfM capability from real Sparse SfM execution on current historical campaigns, which remains unexecuted because known calibration data is unavailable.
|
||||
|
||||
## DOC-B07 — README presents obsolete current DB state
|
||||
|
||||
File: README.md
|
||||
|
||||
README still presents Project DB v23 as the effective current head.
|
||||
|
||||
Required correction:
|
||||
|
||||
- current Project DB = v25;
|
||||
- mention RAW batch v24 and Feature batch v25;
|
||||
- register REAL_S21_TRACKS=PASS/FROZEN;
|
||||
- register REAL_A6000_PRE_SFM=PASS/FROZEN;
|
||||
- state clearly that Sparse SfM and Dense/MVS were not executed in the A6000 real pre-SfM proof.
|
||||
|
||||
# IMPORTANT findings
|
||||
|
||||
## DOC-I01 — AGENTS.md still describes v25 proof as unfinished
|
||||
|
||||
File: AGENTS.md
|
||||
|
||||
The resource policy and Source Comment Contract are strong and current.
|
||||
|
||||
However, lifecycle prose still describes the v25 Feature-batch tranche as not yet eligible for final closure.
|
||||
|
||||
Required correction:
|
||||
|
||||
- update the v25 lifecycle wording to match acquired evidence;
|
||||
- register REAL_A6000_PRE_SFM=PASS/FROZEN;
|
||||
- retain global-maintenance-2026-09-01 as valid historical review evidence;
|
||||
- document the later A6000 checkpoint without erasing the maintenance checkpoint.
|
||||
|
||||
## DOC-I02 — Resource Boundary contains stale current schema and Task count
|
||||
|
||||
File: docs/architecture/resource_boundary.md
|
||||
|
||||
Present-tense material still contains combinations of current schema v24, Project Database v23 as current identity owner, fourteen current Task kinds and migrations only through v23.
|
||||
|
||||
The no-generic-Resource-System decision remains valid and must remain historical authority.
|
||||
|
||||
Required correction:
|
||||
|
||||
- update only stale present-tense current-state claims;
|
||||
- current schema = v25;
|
||||
- current production inventory = 16 Task kinds;
|
||||
- preserve the original Gate G and no-new-subsystem decisions.
|
||||
|
||||
## DOC-I03 — Architecture overview weakens the canonical resource policy
|
||||
|
||||
File: docs/architecture/overview.md
|
||||
|
||||
The overview states that host stability and TUI responsiveness have priority over maximum throughput without the qualification introduced by the current canonical policy.
|
||||
|
||||
The current rule is to preserve the defined interactive host reserve and then maximize safe useful throughput.
|
||||
|
||||
Leaving additional safe and useful resources idle is not a stability strategy.
|
||||
|
||||
Required correction:
|
||||
|
||||
- align the overview with MAXIMUM SAFE USEFUL THROUGHPUT;
|
||||
- preserve the interactive host reserve as the safety boundary;
|
||||
- summarize v24 and v25 operational overlays without making reference-host values portable constants.
|
||||
|
||||
## DOC-I04 — Candidate resource sections contradict each other
|
||||
|
||||
File: docs/architecture/candidate_pair.md
|
||||
|
||||
The current Task section describes approximately 256 KiB fixed memory, 8 MiB per admitted item, batch 1..64 and coupled CPU/batch scaling.
|
||||
|
||||
A later resource section still describes historical values around a 24-source window and 64 KiB per item.
|
||||
|
||||
Required correction:
|
||||
|
||||
- reconcile the later resource section with the current validated Task capability;
|
||||
- preserve Candidate scientific identity, scoring, ordering and persistence;
|
||||
- keep historical estimates only when explicitly labelled historical.
|
||||
|
||||
## DOC-I05 — Feature Store omits the v25 selected-execution batch path
|
||||
|
||||
File: docs/architecture/feature_store.md
|
||||
|
||||
The scientific Feature File contract is strong, but operational prose still describes only the historical single-image features.extract/1 path as the production model.
|
||||
|
||||
Required correction:
|
||||
|
||||
- preserve features.extract/1;
|
||||
- document features.extract.batch/1 as an additional current operational path;
|
||||
- distinguish per-image scientific atomicity from cross-image execution concurrency;
|
||||
- keep general DAG planning separately deferred.
|
||||
|
||||
## DOC-I06 — Task Kind Registry uses an obsolete 15-kind anchor
|
||||
|
||||
File: docs/architecture/task_kind_registry.md
|
||||
|
||||
The document correctly states that production contains 16 Task kinds but links to resource_governor.md#audit-des-15-kinds-de-production.
|
||||
|
||||
Required correction:
|
||||
|
||||
- synchronize the Resource Governor heading and Registry link around the current 16-kind inventory;
|
||||
- do not alter historical audit counts that were correct at their checkpoint.
|
||||
|
||||
## DOC-I07 — Generic Task adaptation prose conflicts with coupled CPU/batch kinds
|
||||
|
||||
File: docs/architecture/task_system.md
|
||||
|
||||
Generic Compute Governor prose says CPU and batch are never tried together.
|
||||
|
||||
Feature Batch and Candidate now have legitimate coupled CPU/batch rungs because additional CPU cannot exercise additional independent work while the admitted item window remains one.
|
||||
|
||||
Required correction:
|
||||
|
||||
- retain independent-dimension adaptation as the generic rule;
|
||||
- document explicit coupled cross-item exceptions where the dimensions are operationally inseparable for measurement;
|
||||
- do not make coupled scaling universal.
|
||||
|
||||
## DOC-I08 — Build guide retains historical fixed -j8 examples
|
||||
|
||||
File: docs/development/build.md
|
||||
|
||||
Build examples repeatedly use -j8 while current engineering policy requires safe host-aware parallelism.
|
||||
|
||||
Required correction:
|
||||
|
||||
- describe build parallelism as host-derived;
|
||||
- a reference-host example may use approximately -j12 when clearly labelled as reference-host evidence;
|
||||
- no fixed job count becomes a portable product constant.
|
||||
|
||||
## DOC-I09 — Concurrency guide retains historical build-policy examples
|
||||
|
||||
File: docs/development/concurrency.md
|
||||
|
||||
The concurrency, lifetime and TSan qualification rules are valuable and mostly current, but the build example still uses fixed -j8 and some illustrative snippets do not represent exact current public API names.
|
||||
|
||||
Required correction:
|
||||
|
||||
- align build parallelism with host-aware policy;
|
||||
- identify illustrative pseudo-code as illustrative when it is not exact repository API;
|
||||
- preserve the external OpenCV and TBB TSan qualification;
|
||||
- preserve the separate Vulkan validation boundary.
|
||||
|
||||
## DOC-I10 — Visual Index describes implemented downstream work as future
|
||||
|
||||
File: docs/architecture/visual_index.md
|
||||
|
||||
The final future section still describes Candidate Pair Generator and Matcher as future consumers.
|
||||
|
||||
Both are implemented.
|
||||
|
||||
Required correction:
|
||||
|
||||
- replace the stale future wording with the actual current downstream relationship or explicitly mark it as historical design context;
|
||||
- do not change Visual Index scientific identity or capacity contracts.
|
||||
|
||||
## DOC-I11 — Track Model contains stale production selector and pipeline wording
|
||||
|
||||
File: docs/architecture/tracks.md
|
||||
|
||||
The Track Model scientific contract remains valid, but some prose still presents Geometric Verifier v1 as the production selector and Sparse SfM as future.
|
||||
|
||||
Current real Track evidence uses Geometric Verifier v3. Sparse SfM Gates C through G are implemented, while real known-calibration Sparse SfM on the historical real campaigns remains unexecuted.
|
||||
|
||||
Required correction:
|
||||
|
||||
- preserve Track Model v1 identity and persistence semantics;
|
||||
- distinguish historical verifier-v1 examples from the current v3 production lineage;
|
||||
- distinguish implemented Sparse SfM capability from real campaign execution.
|
||||
|
||||
# Document classification
|
||||
|
||||
## CURRENT_AND_COMPLETE
|
||||
|
||||
The following documents are currently strong enough that no major contract rewrite is justified by this audit:
|
||||
|
||||
- docs/architecture/calibration_science_v1.md
|
||||
- docs/architecture/calibration_bootstrap.md
|
||||
- docs/architecture/calibration_solver_preflight_v1.md
|
||||
- docs/architecture/photo_quality_triage.md
|
||||
- docs/architecture/task_queue.md
|
||||
- docs/architecture/matcher.md
|
||||
- docs/architecture/track_builder.md
|
||||
- docs/performance/target_hardware.md
|
||||
|
||||
Small consistency edits may still be appropriate later.
|
||||
|
||||
## CURRENT_BUT_INCOMPLETE
|
||||
|
||||
The following documents have strong core contracts but need limited current-state reconciliation:
|
||||
|
||||
- docs/architecture/internal_parallelism.md
|
||||
- docs/architecture/runtime.md
|
||||
- docs/architecture/geometric_verification.md
|
||||
- docs/architecture/geometric_verifier.md
|
||||
|
||||
## HISTORICAL_AND_VALID
|
||||
|
||||
The following documents must remain historical evidence and must not be mass-modernized:
|
||||
|
||||
- docs/architecture/global_maintenance_audit.md
|
||||
- docs/architecture/foundation_review.md
|
||||
- docs/concepts/matching_and_tracks.md
|
||||
- docs/concepts/reconstruction_layers.md
|
||||
|
||||
Older Task counts, schema versions, measurements and decisions may be exactly correct for the checkpoint described by those documents.
|
||||
|
||||
An old version number is stale only when the prose claims that it is the current state.
|
||||
|
||||
# Documentation authority findings
|
||||
|
||||
## AUTH-01 — Current Project DB authority is ambiguous
|
||||
|
||||
Current-version claims are distributed across:
|
||||
|
||||
- README.md
|
||||
- docs/architecture/project_database.md
|
||||
- docs/architecture/persistence.md
|
||||
- docs/architecture/resource_boundary.md
|
||||
- docs/roadmap/roadmap.md
|
||||
|
||||
They must converge on one current-state fact:
|
||||
|
||||
Project DB current schema = v25
|
||||
|
||||
Detailed v22, v23, v24 and v25 contracts remain owned by the specialized architecture documents.
|
||||
|
||||
## AUTH-02 — Resource policy is duplicated across too many authorities
|
||||
|
||||
Resource policy currently appears in AGENTS, Governor, Resource Boundary, Resource Aware Pipeline, Internal Parallelism, Overview, Target Hardware, Build and Concurrency documentation.
|
||||
|
||||
Desired authority split:
|
||||
|
||||
- docs/architecture/resource_governor.md owns runtime resource policy;
|
||||
- AGENTS.md owns engineering and agent obligations;
|
||||
- docs/performance/target_hardware.md owns reference-host measurements;
|
||||
- other documents summarize and link instead of redefining policy.
|
||||
|
||||
## AUTH-03 — Latest real checkpoint needs first-class documentation
|
||||
|
||||
The repository now contains the real checkpoint real-a6000-pre-sfm-2026-09-02.
|
||||
|
||||
Documentation must explain its relationship to global-maintenance-2026-09-01.
|
||||
|
||||
The newer checkpoint does not erase the maintenance checkpoint. The maintenance checkpoint remains historical review authority for unchanged frozen systems; the A6000 checkpoint adds later operational and real-data evidence.
|
||||
|
||||
## LINK-01 — Task Kind Registry anchor is stale
|
||||
|
||||
docs/architecture/task_kind_registry.md states sixteen kinds but points to a Resource Governor anchor named for fifteen kinds.
|
||||
|
||||
The heading and cross-link must be synchronized without altering historical fourteen-kind audit evidence.
|
||||
|
||||
# Repository language decision
|
||||
|
||||
Human authority has selected English as the canonical repository language.
|
||||
|
||||
The final repository target is:
|
||||
|
||||
- DOCUMENTATION_LANGUAGE=ENGLISH
|
||||
- SOURCE_COMMENT_LANGUAGE=ENGLISH
|
||||
- AGENT_CONTRACT_LANGUAGE=ENGLISH
|
||||
- USER_INTERFACE_LANGUAGE=ENGLISH
|
||||
|
||||
All technical documentation, agent contracts, prompt files and production source comments are to converge on English during the remediation passes.
|
||||
|
||||
Translation must preserve scientific meaning, historical truth, FROZEN contracts, identities, numeric values, lifecycle state and evidence. A historical document may be translated but must not be silently modernized.
|
||||
|
||||
The canonical TUI language is English. Existing non-English UI strings are to be remediated in an explicitly scoped UI-language pass; this documentation audit does not itself authorize executable UI changes.
|
||||
|
||||
# Product documentation intentionally deferred
|
||||
|
||||
This documentation cleanup must not invent final contracts for:
|
||||
|
||||
- Viewer behavior;
|
||||
- Sony A6000 live acquisition;
|
||||
- Samsung S21 live acquisition;
|
||||
- Coverage Analysis;
|
||||
- Capture Guidance;
|
||||
- suggested viewpoints;
|
||||
- video and keyframe ingestion;
|
||||
- final optics onboarding UX;
|
||||
- optics profile import and export;
|
||||
- final mesh, texture and export UX.
|
||||
|
||||
Those areas will be defined explicitly by the human during the PRODUCT_DEFINITION phase before the final prompt/ execution contract is frozen.
|
||||
|
||||
Already established human product intent includes:
|
||||
|
||||
- NEW_CAMERA_REQUIRES_CODE_CHANGE=NO
|
||||
- NEW_LENS_REQUIRES_CODE_CHANGE=NO
|
||||
- ELECTRONIC_LENS_WITH_METADATA=SUPPORTED
|
||||
- MANUAL_LENS_WITHOUT_EXIF=SUPPORTED
|
||||
- MULTIPLE_LENSES_PER_CAMERA=SUPPORTED
|
||||
- ZOOM_MULTIPLE_FOCALS=SUPPORTED
|
||||
- MULTIPLE_OPTICAL_CONFIGURATIONS_PER_PROJECT=SUPPORTED
|
||||
- SILENT_CALIBRATION_SUBSTITUTION=FORBIDDEN
|
||||
- SILENT_LENS_IDENTITY_INFERENCE=FORBIDDEN
|
||||
- OPTICS_TUI_WORKFLOW=REQUIRED
|
||||
- PROFILE_IMPORT_EXPORT=REQUIRED
|
||||
|
||||
These statements describe product intent only. They do not authorize implementation during this documentation audit.
|
||||
|
||||
# Documentation remediation order
|
||||
|
||||
## D1 — Current-state authority
|
||||
|
||||
Correct first:
|
||||
|
||||
- README.md
|
||||
- AGENTS.md
|
||||
- docs/roadmap/roadmap.md
|
||||
- docs/architecture/project_database.md
|
||||
- docs/architecture/persistence.md
|
||||
|
||||
Goal: one coherent current lifecycle and one explicit Project DB v25 head.
|
||||
|
||||
## D2 — Resource and runtime consistency
|
||||
|
||||
Then correct:
|
||||
|
||||
- docs/architecture/resource_boundary.md
|
||||
- docs/architecture/resource_aware_pipeline.md
|
||||
- docs/architecture/overview.md
|
||||
- docs/architecture/candidate_pair.md
|
||||
- docs/architecture/feature_store.md
|
||||
- docs/architecture/task_kind_registry.md
|
||||
- docs/architecture/task_system.md
|
||||
|
||||
Goal: current operational descriptions obey MAXIMUM SAFE USEFUL THROUGHPUT and SERIALISM_REQUIRES_PROOF without rewriting frozen science.
|
||||
|
||||
## D3 — Developer instructions
|
||||
|
||||
Then correct:
|
||||
|
||||
- docs/development/testing.md
|
||||
- docs/development/build.md
|
||||
- docs/development/concurrency.md
|
||||
|
||||
Goal: developer instructions match current APIs, build policy and validation discipline.
|
||||
|
||||
## D4 — Targeted scientific and current-state wording cleanup
|
||||
|
||||
Then inspect and correct only stale lifecycle wording in:
|
||||
|
||||
- docs/architecture/sparse_sfm.md
|
||||
- docs/architecture/tracks.md
|
||||
- docs/architecture/visual_index.md
|
||||
- docs/architecture/geometric_verification.md
|
||||
- docs/architecture/geometric_verifier.md
|
||||
- docs/architecture/runtime.md
|
||||
|
||||
This phase must not reopen FROZEN science.
|
||||
|
||||
## D5 — Documentation index and links
|
||||
|
||||
Finally reconcile README navigation, historical/current labels and cross-document anchors.
|
||||
|
||||
# Finding summary
|
||||
|
||||
BLOCKING_DOC = 7
|
||||
|
||||
IMPORTANT = 11
|
||||
|
||||
Highest-priority files:
|
||||
|
||||
1. docs/roadmap/roadmap.md
|
||||
2. docs/architecture/project_database.md
|
||||
3. docs/architecture/persistence.md
|
||||
4. docs/architecture/resource_aware_pipeline.md
|
||||
5. docs/development/testing.md
|
||||
6. docs/architecture/sparse_sfm.md
|
||||
7. README.md
|
||||
|
||||
# Audit boundary
|
||||
|
||||
This report does not authorize:
|
||||
|
||||
- scientific threshold changes;
|
||||
- new schema versions;
|
||||
- new Task kinds;
|
||||
- resource-policy redesign;
|
||||
- Viewer implementation;
|
||||
- live-capture implementation;
|
||||
- Sparse SfM execution;
|
||||
- Dense/MVS execution.
|
||||
|
||||
The next separate audit is SOURCE_COMMENT_AUDIT over include/lardon3d/** and src/**.
|
||||
|
||||
Source comments will later be classified as:
|
||||
|
||||
- EXCELLENT
|
||||
- GOOD
|
||||
- ACCEPTABLE
|
||||
- UNDER_COMMENTED
|
||||
- SEVERELY_UNDER_COMMENTED
|
||||
- STALE_COMMENT
|
||||
- MISLEADING_CONTRACT_COMMENT
|
||||
|
||||
The comment audit will focus on WHY, CONTRACT, INVARIANT, OWNERSHIP, IDENTITY, persistence ordering, restart, cancellation, concurrency, resource accounting and FROZEN boundaries.
|
||||
|
||||
No mass source-comment editing is authorized by this inventory.
|
||||
|
||||
# Remediation closure — 2026-09-02
|
||||
|
||||
The documentation finding-remediation pass is complete.
|
||||
|
||||
```text
|
||||
D1=PASS
|
||||
D2=PASS
|
||||
D3=PASS
|
||||
D4=PASS
|
||||
D5=PASS
|
||||
|
||||
BLOCKING_DOC_RESOLVED=7/7
|
||||
IMPORTANT_RESOLVED=11/11
|
||||
|
||||
AUTHORITY_FINDINGS_RESOLVED=3/3
|
||||
LINK_FINDINGS_RESOLVED=1/1
|
||||
```
|
||||
|
||||
D1 converged README, AGENTS, roadmap, Project DB and persistence on the current v25 lifecycle.
|
||||
|
||||
D2 reconciled resource/runtime documentation with sixteen production Task kinds,
|
||||
`MAXIMUM_SAFE_USEFUL_THROUGHPUT`, `SERIALISM_REQUIRES_PROOF`, the v24 RAW batch path and the v25
|
||||
Feature batch path.
|
||||
|
||||
D3 replaced stale developer API/build examples, removed fixed build-width policy and preserved the
|
||||
qualified sanitizer/TSan/Vulkan evidence boundaries.
|
||||
|
||||
D4 reconciled Sparse SfM lifecycle, Track verifier lineage, Visual Index downstream status, current
|
||||
Geometric Verifier v3 and runtime current-state wording without reopening frozen scientific contracts.
|
||||
|
||||
D5 adds `docs/README.md` as the navigation/authority map and removes the stale fifteen-kind
|
||||
cross-document anchor dependency.
|
||||
|
||||
The historical `global-maintenance-2026-09-01` checkpoint remains authoritative evidence for the
|
||||
unchanged boundaries it reviewed. The later `real-a6000-pre-sfm-2026-09-02` checkpoint adds real-data
|
||||
operational evidence and does not erase the maintenance checkpoint.
|
||||
|
||||
No source code, schema, Task Kind, scientific threshold, Sparse SfM execution, Dense/MVS execution,
|
||||
viewer or live-capture implementation is authorized by this closure.
|
||||
|
||||
## Language normalization boundary
|
||||
|
||||
Human authority selected English as the repository language.
|
||||
|
||||
The documents modified by the finding-remediation pass are English. Other untouched documentation may
|
||||
still contain historical or current non-English prose. That remaining language-only normalization is
|
||||
tracked separately and must not be mistaken for an unresolved current-state authority finding.
|
||||
|
||||
Translation of historical evidence must preserve the historical checkpoint exactly.
|
||||
|
||||
# Phase state
|
||||
|
||||
- DOCUMENTATION_INVENTORY_AUDIT=PASS_WITH_FINDINGS
|
||||
- DOCUMENTATION_FINDING_REMEDIATION=PASS
|
||||
- DOCUMENTATION_LANGUAGE_NORMALIZATION=IN_PROGRESS
|
||||
- SOURCE_COMMENT_AUDIT=PASS
|
||||
- SOURCE_COMMENT_REMEDIATION=PASS
|
||||
- PRODUCT_DEFINITION=PASS/FROZEN
|
||||
- PROMPT_TREE=NEXT
|
||||
99
docs/audits/source_comment_inventory.md
Normal file
99
docs/audits/source_comment_inventory.md
Normal file
|
|
@ -0,0 +1,99 @@
|
|||
# Lardon3D — Source Comment Audit
|
||||
|
||||
## Status
|
||||
|
||||
```text
|
||||
SOURCE_COMMENT_AUDIT=PASS
|
||||
SOURCE_COMMENT_REMEDIATION=PASS
|
||||
SOURCE_COMMENT_SCOPE=include/lardon3d/** + src/**
|
||||
SOURCE_COMMENT_LANGUAGE=ENGLISH
|
||||
SOURCE_CODE_BEHAVIOR_CHANGED=NO
|
||||
```
|
||||
|
||||
The audit covered 148 C/C++ source/header files under `include/lardon3d/**` and `src/**`.
|
||||
|
||||
The initial static inventory reported:
|
||||
|
||||
```text
|
||||
comment blocks 807
|
||||
raw non-English lexical candidates 33
|
||||
actual French comment blocks 29
|
||||
English false positives 4
|
||||
stale-contract candidates 0
|
||||
raw unannotated-public declarations 324
|
||||
```
|
||||
|
||||
## Remediation
|
||||
|
||||
All 29 actual French source comments were translated to English.
|
||||
|
||||
The four raw language-detector false positives were already English and were left unchanged.
|
||||
|
||||
The remediation touched comments only. A file-level lexical comparison removed comments entirely and verified that every non-comment source
|
||||
byte remained identical before and after the declared replacements.
|
||||
|
||||
No executable UI/error string was changed in this pass. UI language is a separate explicitly scoped
|
||||
implementation pass.
|
||||
|
||||
## Stale and misleading comments
|
||||
|
||||
The lexical stale-contract pass found zero candidate comments.
|
||||
|
||||
No scientific threshold, schema version, Task Kind, resource contract, persistence identity or FROZEN
|
||||
boundary was changed by this audit.
|
||||
|
||||
## Public API adjacency heuristic
|
||||
|
||||
The first inventory reported 324 public declarations without an immediately adjacent comment.
|
||||
|
||||
That number is **informational only**, not 324 defects.
|
||||
|
||||
The repository Source Comment Contract requires comments where WHY, CONTRACT, INVARIANT, OWNERSHIP,
|
||||
IDENTITY, persistence ordering, restart, cancellation, concurrency, resource accounting or a FROZEN
|
||||
boundary is non-obvious. It does not require one redundant comment for every getter, thin wrapper,
|
||||
paired create/load/list function or declaration already covered by a surrounding contract block.
|
||||
|
||||
Therefore absence of an adjacent comment alone is not a valid
|
||||
`UNDER_COMMENTED`/`SEVERELY_UNDER_COMMENTED` finding.
|
||||
|
||||
Future audits must review semantic contract coverage rather than using declaration adjacency as a
|
||||
blanket failure criterion.
|
||||
|
||||
## Source Comment Contract
|
||||
|
||||
Production source comments:
|
||||
|
||||
- are English;
|
||||
- explain non-obvious WHY/CONTRACT/INVARIANT facts;
|
||||
- preserve ownership and lifetime boundaries;
|
||||
- preserve persistence/restart ordering;
|
||||
- preserve concurrency/resource-accounting boundaries;
|
||||
- preserve scientific identity and FROZEN constraints;
|
||||
- do not paraphrase obvious code line-by-line;
|
||||
- are updated when behavior changes.
|
||||
|
||||
## Validation
|
||||
|
||||
The remediation runner requires:
|
||||
|
||||
```text
|
||||
branch = docs-audit
|
||||
no unrelated worktree changes
|
||||
all expected old comments match exactly once
|
||||
file-level source with all comments removed remains byte-for-byte identical
|
||||
no strong French-comment candidate remains
|
||||
git diff --check passes
|
||||
```
|
||||
|
||||
## Phase state
|
||||
|
||||
```text
|
||||
DOCUMENTATION_FINDING_REMEDIATION=PASS
|
||||
DOCUMENTATION_LANGUAGE_NORMALIZATION=IN_PROGRESS
|
||||
|
||||
SOURCE_COMMENT_AUDIT=PASS
|
||||
SOURCE_COMMENT_REMEDIATION=PASS
|
||||
|
||||
PRODUCT_DEFINITION=PASS/FROZEN
|
||||
PROMPT_TREE=NEXT
|
||||
```
|
||||
|
|
@ -1,193 +1,293 @@
|
|||
# Instructions de build
|
||||
# Build
|
||||
|
||||
## Prérequis
|
||||
## Status
|
||||
|
||||
- **OS** : Linux (testé sur distributions récentes)
|
||||
- **Compilateur** : Clang (recommandé) ou GCC
|
||||
- **Système de build** : Meson + Ninja
|
||||
- **Dépendances principales** : ncursesw, SQLite, OpenSSL, GIO/GLib, OpenCV,
|
||||
LibRaw, libexif, libpng, libdeflate, Ceres ; Vulkan reste optionnel
|
||||
- **Langages** : API publiques C17 et implémentation mixte C17/C++17
|
||||
```text
|
||||
BUILD_SYSTEM=MESON_NINJA
|
||||
PUBLIC_API_LANGUAGE=C17
|
||||
IMPLEMENTATION_LANGUAGES=C17_CXX17
|
||||
BUILD_PARALLELISM=HOST_AWARE
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
```
|
||||
|
||||
## Bootstrap des outils
|
||||
Meson is the build-system authority. Do not duplicate dependency-version truth
|
||||
in this document when `meson.build` already enforces it.
|
||||
|
||||
Les commandes ci-dessous installent seulement le compilateur, Meson/Ninja,
|
||||
`pkg-config` et ncurses. Les bibliothèques listées plus haut doivent aussi être
|
||||
disponibles dans les versions acceptées par `meson.build`; Meson reste la source
|
||||
de vérité et refuse explicitement une dépendance absente ou incompatible.
|
||||
## Requirements
|
||||
|
||||
Lardon3D targets Linux.
|
||||
|
||||
Primary toolchain:
|
||||
|
||||
```text
|
||||
Clang or GCC
|
||||
Meson
|
||||
Ninja
|
||||
pkg-config
|
||||
```
|
||||
|
||||
Major dependencies currently include ncursesw, SQLite, OpenSSL, GIO/GLib,
|
||||
OpenCV, LibRaw, libexif, libpng, libdeflate and Ceres. Vulkan remains optional
|
||||
at configuration level.
|
||||
|
||||
Public APIs are C17. Implementation is mixed C17/C++17.
|
||||
|
||||
## Bootstrap examples
|
||||
|
||||
These commands install only the basic compiler/build front end; Meson remains
|
||||
authoritative for the complete dependency set.
|
||||
|
||||
Debian/Ubuntu:
|
||||
|
||||
```sh
|
||||
# Debian / Ubuntu
|
||||
sudo apt install clang meson ninja-build libncursesw5-dev pkg-config
|
||||
```
|
||||
|
||||
# Fedora
|
||||
Fedora:
|
||||
|
||||
```sh
|
||||
sudo dnf install clang meson ninja-build ncurses-devel pkg-config
|
||||
```
|
||||
|
||||
# Arch
|
||||
Arch Linux:
|
||||
|
||||
```sh
|
||||
sudo pacman -S clang meson ninja ncurses pkgconf
|
||||
```
|
||||
|
||||
## Build standard
|
||||
## Standard build
|
||||
|
||||
First configuration:
|
||||
|
||||
```sh
|
||||
# Première configuration
|
||||
CC=clang meson setup build
|
||||
CC=clang CXX=clang++ meson setup build
|
||||
```
|
||||
|
||||
# Arbre existant
|
||||
Existing tree:
|
||||
|
||||
```sh
|
||||
meson setup --reconfigure build
|
||||
meson compile -C build -j8
|
||||
meson compile -C build
|
||||
```
|
||||
|
||||
### Options utiles
|
||||
Do not hard-code `-j8` as project policy.
|
||||
|
||||
Meson/Ninja should use host-appropriate parallelism unless a specific
|
||||
validation has a reason to constrain it.
|
||||
|
||||
## Build parallelism policy
|
||||
|
||||
The build is not governed by a portable fixed job count.
|
||||
|
||||
Canonical policy:
|
||||
|
||||
```text
|
||||
preserve the interactive host reserve
|
||||
then use maximum safe useful throughput
|
||||
```
|
||||
|
||||
A reference host measurement such as 8 or 12 useful jobs is evidence for that
|
||||
host at that time, not a repository constant.
|
||||
|
||||
If memory-heavy compilation or another active workload creates pressure,
|
||||
reduce build width for that run. Do not convert the temporary reduction into a
|
||||
global documentation rule.
|
||||
|
||||
## Reconfigure versus wipe
|
||||
|
||||
Prefer incremental reuse:
|
||||
|
||||
```sh
|
||||
# Build de debug (défaut)
|
||||
meson setup build --wipe
|
||||
|
||||
# Build de release
|
||||
meson setup build --wipe --buildtype=release
|
||||
|
||||
# Build avec optimisations aggressive
|
||||
meson setup build --wipe --buildtype=release -Db_lto=true
|
||||
meson setup --reconfigure build
|
||||
meson compile -C build
|
||||
```
|
||||
|
||||
Use `--wipe` only when a fresh configuration is actually required, such as:
|
||||
|
||||
- switching sanitizer configuration in the same directory;
|
||||
- changing compiler family;
|
||||
- changing a configuration whose cached state cannot be reused safely;
|
||||
- reproducing a clean release/global-maintenance proof;
|
||||
- recovering from a stale or corrupt build directory.
|
||||
|
||||
A normal edit/test loop should not wipe the build tree repeatedly.
|
||||
|
||||
## Release build
|
||||
|
||||
Use an explicit release directory or deliberate reconfiguration.
|
||||
|
||||
Example:
|
||||
|
||||
```sh
|
||||
CC=clang CXX=clang++ meson setup build-release --buildtype=release
|
||||
meson compile -C build-release
|
||||
```
|
||||
|
||||
For LTO:
|
||||
|
||||
```sh
|
||||
CC=clang CXX=clang++ meson setup build-release-lto --buildtype=release -Db_lto=true
|
||||
meson compile -C build-release-lto
|
||||
```
|
||||
|
||||
Separate directories avoid destroying a useful incremental debug tree.
|
||||
|
||||
## Vulkan configuration
|
||||
|
||||
Portable CPU-only proof:
|
||||
|
||||
```sh
|
||||
CC=clang CXX=clang++ meson setup build-portable -Dvulkan_orb=disabled
|
||||
meson compile -C build-portable
|
||||
```
|
||||
|
||||
Vulkan-enabled proof:
|
||||
|
||||
```sh
|
||||
CC=clang CXX=clang++ meson setup build-vulkan -Dvulkan_orb=enabled
|
||||
meson compile -C build-vulkan
|
||||
```
|
||||
|
||||
A Vulkan-on build is not automatically a proof that every scientific path uses
|
||||
or should use the GPU.
|
||||
|
||||
Current production GPU promotion remains limited by each subsystem's validated
|
||||
backend contract.
|
||||
|
||||
## Validation
|
||||
|
||||
Normal configured tests:
|
||||
|
||||
```sh
|
||||
# Tests unitaires
|
||||
meson test -C build --print-errorlogs
|
||||
|
||||
# Vérification du style (whitespace)
|
||||
git diff --check
|
||||
|
||||
# Vérification autonome d'un header C public modifié
|
||||
cc -x c -std=c17 -fsyntax-only -Iinclude \
|
||||
-include lardon3d/<header>.h /dev/null
|
||||
```
|
||||
|
||||
### Preuve fraîche de maintenance globale — 1er septembre 2026
|
||||
|
||||
Le [registre canonique](../architecture/global_maintenance_audit.md) conserve
|
||||
le détail et les qualifications. Les résultats reproductibles acquis sont :
|
||||
|
||||
| Configuration fraîche | Compilateurs/options | Build | Suite |
|
||||
| --- | --- | ---: | ---: |
|
||||
| portable | Clang/Clang++ 22.1.8, C17/C++17, `-Dvulkan_orb=disabled` | 931/931 | 64/64 sériel |
|
||||
| Vulkan | Clang/Clang++ 22.1.8, C17/C++17, `-Dvulkan_orb=enabled` | 939/939 | 65/65 sériel |
|
||||
| ASan/UBSan portable | Clang/Clang++ 22.1.8, `address,undefined` | graphe complet | 64/64 avec LSan désactivé après attribution externe |
|
||||
| TSan portable | GCC/G++ 16.2.1, Vulkan désactivé | cibles concurrentes | 14/14 + 220 répétitions |
|
||||
|
||||
La suite Vulkan comprend `orb-vulkan-backend` sur la Radeon 780M RADV PHOENIX
|
||||
réelle. La cible de feasibility SIFT/RootSIFT, non enregistrée dans la suite,
|
||||
a été compilée/exécutée séparément : zéro divergence de décision Lowe mais des
|
||||
divergences d'index et de bits de distance, donc aucune promotion en backend
|
||||
production. Les probes stricts GCC/Clang C17+C++17 passent 76/76 sur les
|
||||
19 headers publics modifiés ou nouveaux, ainsi que le fixture ABI, le lien
|
||||
application et `git diff --check`; `scan3d/` reste intact.
|
||||
|
||||
L'unique revue finale indépendante GPT-5.6 SOL/ULTRA a conclu PASS sans finding
|
||||
bloquant. Elle a indépendamment rejoué le build portable, la suite complète
|
||||
64/64, une matrice focalisée 15/15, les 76/76 probes de headers, l'ABI, les
|
||||
négatifs de seams production, le SHA-256 du manifest GV retenu et le diff-check.
|
||||
Le statut canonique est donc `GLOBAL_MAINTENANCE_AUDIT=PASS/FROZEN` ; les
|
||||
qualifications sanitizer ci-dessous restent néanmoins partie de la preuve.
|
||||
|
||||
Une validation post-freeze a ensuite attribué le délai intermittent de
|
||||
`test-feature-task` à la capture de la télémétrie hôte réelle par ses Governors
|
||||
synthétiques. Le fixture utilise maintenant un `ResourceSnapshot` complet,
|
||||
privé, par Governor et compilé pour cette seule cible ; production continue de
|
||||
lire la télémétrie réelle et n'exporte aucun seam. Après correction du second
|
||||
Governor relevé en revue, Feature passe 100/100, la matrice ordonnée 4/4 et les
|
||||
suites finales portable/Vulkan 64/64 et 65/65 ; ASan/UBSan ciblé avec
|
||||
`detect_leaks=0` et TSan passent. Le registre canonique conserve la régression
|
||||
charge 5 `WAIT`/charge 0 `START` et la qualification exacte. Un timeout `task`
|
||||
isolé dans une suite normale mixte après reconstruction large reste
|
||||
non reproductible : le ciblé immédiat et sa matrice de revue 100/100 passent,
|
||||
sans modification de Task ni de son timeout.
|
||||
|
||||
La première suite LSan complète est volontairement conservée comme non-PASS :
|
||||
57 OK, 6 FAIL et 1 timeout. Cinq échecs partagent exactement la fuite externe
|
||||
OpenCL de 3 808 octets/68 allocations ; les deux anomalies de 30 s n'ont aucun
|
||||
diagnostic sanitizer. Le délai Feature, absent du suivi initial, a ensuite été
|
||||
reproduit et corrigé comme décrit ci-dessus ; le délai Task reste non
|
||||
reproductible. La suite entière passe 64/64 avec ASan/UBSan actifs et
|
||||
`detect_leaks=0`, tandis qu'un sous-ensemble prouvé sans loader OpenCV/OpenCL
|
||||
passe 20/20 avec LSan actif. Il est donc incorrect de résumer cette preuve par
|
||||
« LSan 64/64 ».
|
||||
|
||||
Le log Clang complet a aussi été audité. La conversion publique Sparse SfM
|
||||
`uint32_t → int` était matérielle et a été corrigée avec validation ciblée ; les
|
||||
autres émissions sont soit des conversions baseline déjà bornées, soit des
|
||||
tests/benchmarks, soit des headers OpenCV/Ceres externes. Les emplacements et
|
||||
justifications exacts restent centralisés dans le registre afin de ne pas
|
||||
dupliquer une seconde liste normative ici.
|
||||
|
||||
## Build ASan/UBSan (debug mémoire)
|
||||
|
||||
À exécuter pour tout ticket touchant la mémoire, les durées de vie ou les
|
||||
allocations :
|
||||
Whitespace/style boundary:
|
||||
|
||||
```sh
|
||||
CC=clang meson setup build-asan --wipe \
|
||||
-Db_sanitize=address,undefined
|
||||
meson compile -C build-asan -j8
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Public C header probe:
|
||||
|
||||
```sh
|
||||
cc -x c -std=c17 -fsyntax-only -Iinclude -include lardon3d/<header>.h /dev/null
|
||||
```
|
||||
|
||||
Use `docs/development/testing.md` for sanitizer and validation policy.
|
||||
|
||||
## ASan / UBSan build
|
||||
|
||||
Example dedicated directory:
|
||||
|
||||
```sh
|
||||
CC=clang CXX=clang++ meson setup build-asan -Db_sanitize=address,undefined
|
||||
meson compile -C build-asan
|
||||
meson test -C build-asan --print-errorlogs
|
||||
```
|
||||
|
||||
## Build TSan (concurrence)
|
||||
Do not claim an unqualified full LeakSanitizer pass from the retained global
|
||||
maintenance checkpoint. The external OpenCL loader qualification documented in
|
||||
the canonical audit remains part of that evidence.
|
||||
|
||||
À exécuter pour tout ticket touchant la concurrence (pthread, mutex,
|
||||
variables de condition, états partagés) :
|
||||
## TSan build
|
||||
|
||||
Use TSan only with the configuration that matches the intended proof.
|
||||
|
||||
The retained global maintenance concurrency proof used GCC/G++ with Vulkan
|
||||
disabled, because the project TSan matrix and the Vulkan runtime validation are
|
||||
separate evidence boundaries.
|
||||
|
||||
Example:
|
||||
|
||||
```sh
|
||||
CC=clang meson setup build-tsan --wipe \
|
||||
-Db_sanitize=thread \
|
||||
-Db_lundef=false
|
||||
meson compile -C build-tsan -j8
|
||||
CC=gcc CXX=g++ meson setup build-tsan -Db_sanitize=thread -Db_lundef=false -Dvulkan_orb=disabled
|
||||
meson compile -C build-tsan
|
||||
meson test -C build-tsan --print-errorlogs
|
||||
```
|
||||
|
||||
## Variables d'environnement
|
||||
The exact target subset, suppression qualification and repetition evidence are
|
||||
documented in `docs/development/concurrency.md` and the global maintenance
|
||||
audit.
|
||||
|
||||
| Variable | Description |
|
||||
|---|---|
|
||||
| `CC` | Compilateur C (défaut : gcc) |
|
||||
| `CFLAGS` | Drapeaux de compilation supplémentaires |
|
||||
| `LDFLAGS` | Drapeaux de liaison supplémentaires |
|
||||
## Current retained maintenance checkpoint
|
||||
|
||||
## Structure du build
|
||||
The canonical detailed evidence is:
|
||||
|
||||
```text
|
||||
docs/architecture/global_maintenance_audit.md
|
||||
GLOBAL_MAINTENANCE_AUDIT=PASS/FROZEN
|
||||
```
|
||||
|
||||
The 2026-09-01 checkpoint retained:
|
||||
|
||||
```text
|
||||
portable Clang/Clang++ build and suite
|
||||
Vulkan-on Clang/Clang++ build and suite
|
||||
ASan/UBSan qualified run
|
||||
portable GCC/G++ TSan matrix
|
||||
public-header C17/C++17 probes
|
||||
ABI and application-link checks
|
||||
independent review
|
||||
```
|
||||
|
||||
Those exact historical counts belong to the audit and should not be duplicated
|
||||
as a new current build contract.
|
||||
|
||||
## Build directory layout
|
||||
|
||||
A configured Meson tree typically contains:
|
||||
|
||||
```text
|
||||
build/
|
||||
├── src/ # objets et binaires
|
||||
├── tests/ # binaires de tests
|
||||
└── compile_commands.json # pour LSP / clangd
|
||||
src/
|
||||
tests/
|
||||
compile_commands.json
|
||||
```
|
||||
|
||||
## Dépannage
|
||||
Exact generated layout is Meson/Ninja output and may evolve.
|
||||
|
||||
### Erreur : ncursesw introuvable
|
||||
## Environment
|
||||
|
||||
Common variables include:
|
||||
|
||||
| Variable | Purpose |
|
||||
| --- | --- |
|
||||
| `CC` | C compiler |
|
||||
| `CXX` | C++ compiler |
|
||||
| `CFLAGS` | additional C flags |
|
||||
| `CXXFLAGS` | additional C++ flags |
|
||||
| `LDFLAGS` | additional linker flags |
|
||||
|
||||
Prefer Meson options for project features rather than ad-hoc environment flags
|
||||
that make builds difficult to reproduce.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
Check ncursesw discovery:
|
||||
|
||||
```sh
|
||||
# Vérifier l'installation
|
||||
pkg-config --libs ncursesw
|
||||
# Si absent, installer le paquet de développement ncursesw
|
||||
```
|
||||
|
||||
### Erreur : clang introuvable
|
||||
If Clang is unavailable, GCC is supported where the current Meson checks allow
|
||||
it.
|
||||
|
||||
```sh
|
||||
# Utiliser gcc en alternative
|
||||
meson setup build --wipe
|
||||
# ou installer clang
|
||||
sudo apt install clang
|
||||
For a slow build, first preserve the existing build tree and let Ninja use
|
||||
normal host-aware scheduling. Reduce concurrency only when actual host pressure
|
||||
or another active workload justifies it.
|
||||
|
||||
`ccache` may be used when available, but it is optional operational tooling and
|
||||
not part of scientific identity.
|
||||
|
||||
## Rules
|
||||
|
||||
```text
|
||||
NO_FIXED_GLOBAL_J8=YES
|
||||
NO_REPEATED_UNCHANGED_WIPE=YES
|
||||
HOST_AWARE_BUILD_PARALLELISM=YES
|
||||
```
|
||||
|
||||
### Build lent
|
||||
|
||||
```sh
|
||||
# Réduire la parallélisation
|
||||
meson compile -C build -j4
|
||||
# ou utiliser ccache
|
||||
CC="ccache clang" meson setup build --wipe
|
||||
```
|
||||
Build configuration is operational state. It must not silently redefine
|
||||
scientific formats, fingerprints or persistence contracts.
|
||||
|
|
|
|||
|
|
@ -1,255 +1,412 @@
|
|||
# Règles de concurrence
|
||||
# Concurrency
|
||||
|
||||
## Vue d'ensemble
|
||||
|
||||
Lardon3D utilise un modèle de concurrence à thread unique pour ncurses
|
||||
et un modèle multi-thread pour le traitement. La séparation est stricte :
|
||||
le thread ncurses ne fait jamais de travail métier, et les workers ne
|
||||
touchent jamais ncurses.
|
||||
|
||||
## Modèle de concurrence
|
||||
## Status
|
||||
|
||||
```text
|
||||
Thread principal (ncurses)
|
||||
├── Gestion des entrées
|
||||
├── Affichage TUI
|
||||
└── Orchestration
|
||||
NCURSES_OWNER=MAIN_THREAD_ONLY
|
||||
ACTIVE_HEAVY_QUEUE_CALLBACKS=1
|
||||
TASK_CANCELLATION=COOPERATIVE
|
||||
INTERNAL_PARALLELISM=BOUNDED
|
||||
OWNER_ONLY_PUBLICATION=CANONICAL_WHERE_REQUIRED
|
||||
|
||||
Worker thread
|
||||
├── Exécution des tâches
|
||||
├── Calculs métier
|
||||
└── Écritures de résultats
|
||||
TSAN_PORTABLE_PROJECT_MATRIX=QUALIFIED_PASS
|
||||
TSAN_EXTERNAL_OPENCV_TBB=QUALIFIED
|
||||
VULKAN_CONCURRENCY_VALIDATION=SEPARATE
|
||||
|
||||
SSD operation thread (0 ou 1, joinable)
|
||||
└── Un poll ou contrôle UDisks synchrone borné, sans ncurses ni Task
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
```
|
||||
|
||||
## Règles fondamentales
|
||||
Lardon3D separates UI ownership from heavy processing.
|
||||
|
||||
### 1. ncurses appartient au thread principal
|
||||
The main thread owns ncurses. The Task Queue owns one active heavy callback.
|
||||
Individual validated Task Kinds may create bounded internal participants inside
|
||||
that callback.
|
||||
|
||||
```c
|
||||
// ✅ Correct : appel depuis le thread principal
|
||||
mvprintw(0, 0, "Progression: %d%%", progress);
|
||||
An SSD controller operation may also use at most one bounded joinable operation
|
||||
thread under its own ownership contract.
|
||||
|
||||
// ❌ Interdit : appel depuis un worker
|
||||
// mvprintw() dans un thread secondaire
|
||||
## Execution model
|
||||
|
||||
```text
|
||||
main thread
|
||||
input
|
||||
ncurses
|
||||
TUI orchestration
|
||||
|
||||
Task Queue worker
|
||||
one active heavy callback
|
||||
admitted Task sequence
|
||||
optional bounded internal participants
|
||||
deterministic owner publication
|
||||
|
||||
SSD operation thread
|
||||
zero or one bounded joinable controller operation
|
||||
no ncurses
|
||||
no Task callback
|
||||
```
|
||||
|
||||
### 2. Variables partagées protégées par mutex
|
||||
Internal participants are not a second global scheduler or Queue.
|
||||
|
||||
```c
|
||||
// ✅ Correct
|
||||
pthread_mutex_lock(&queue->mutex);
|
||||
queue->count++;
|
||||
pthread_mutex_unlock(&queue->mutex);
|
||||
## Fundamental rules
|
||||
|
||||
// ❌ Interdit
|
||||
// queue->count++; sans protection
|
||||
### ncurses ownership
|
||||
|
||||
Only the main thread calls ncurses.
|
||||
|
||||
Workers publish observable state through protected data. They never call
|
||||
`mvprintw`, `wrefresh`, or other ncurses APIs.
|
||||
|
||||
### Shared mutable state
|
||||
|
||||
Shared mutable state must have an explicit synchronization owner:
|
||||
|
||||
```text
|
||||
mutex
|
||||
condition variable
|
||||
atomic primitive where the contract explicitly permits it
|
||||
single-thread ownership
|
||||
immutable-after-publication
|
||||
```
|
||||
|
||||
### 3. Variables de condition pour la synchronisation
|
||||
Do not rely on timing or "normally only one caller".
|
||||
|
||||
```c
|
||||
// Producteur (caller de la Task Queue)
|
||||
pthread_mutex_lock(&queue->mutex);
|
||||
queue->ready = true;
|
||||
pthread_cond_signal(&queue->cond);
|
||||
pthread_mutex_unlock(&queue->mutex);
|
||||
### Condition variables
|
||||
|
||||
// Consommateur (worker)
|
||||
pthread_mutex_lock(&queue->mutex);
|
||||
while (!queue->ready) {
|
||||
pthread_cond_wait(&queue->cond, &queue->mutex);
|
||||
}
|
||||
// traitement
|
||||
pthread_mutex_unlock(&queue->mutex);
|
||||
Always test the predicate in a loop around `pthread_cond_wait()`.
|
||||
|
||||
A signal is not durable state; the protected predicate is.
|
||||
|
||||
### Cooperative cancellation
|
||||
|
||||
Production Task cancellation is cooperative.
|
||||
|
||||
Do not use `pthread_cancel()` to stop a Task.
|
||||
|
||||
Task-specific non-preemptible operations finish their current atomic boundary
|
||||
before pause/cancel is observed.
|
||||
|
||||
### Reservation before callback
|
||||
|
||||
No Task callback runs without the Resource Governor admission/reservation
|
||||
required by its installed sequence contract.
|
||||
|
||||
Fixed-resource Tasks do not bypass the Governor.
|
||||
|
||||
### Terminal lifetime
|
||||
|
||||
Terminal callback completion precedes destruction of Task userdata.
|
||||
|
||||
Queue/Task ownership must ensure no observer dereferences freed userdata.
|
||||
|
||||
## Lock ordering
|
||||
|
||||
When multiple locks are required, the owning subsystem must define and preserve
|
||||
one order.
|
||||
|
||||
Never add a reverse-order path to solve a local problem.
|
||||
|
||||
Avoid holding one subsystem mutex while calling into another subsystem that may
|
||||
call back.
|
||||
|
||||
Where practical:
|
||||
|
||||
```text
|
||||
copy bounded state under lock
|
||||
release lock
|
||||
perform I/O / expensive work
|
||||
reacquire only for publication
|
||||
```
|
||||
|
||||
### 4. Pas de callback ncurses depuis un worker
|
||||
## Queue ingress lifetime
|
||||
|
||||
```c
|
||||
// ✅ Correct : le worker signale au thread principal
|
||||
void worker_callback(task_t *task, void *userdata) {
|
||||
shared_state_t *state = userdata;
|
||||
pthread_mutex_lock(&state->mutex);
|
||||
state->result_ready = true;
|
||||
pthread_cond_signal(&state->cond);
|
||||
pthread_mutex_unlock(&state->mutex);
|
||||
}
|
||||
The Queue owner closes ingress before destruction.
|
||||
|
||||
// ❌ Interdit : appel ncurses depuis le worker
|
||||
// void worker_callback(...) {
|
||||
// mvprintw(...);
|
||||
// }
|
||||
Shutdown waits for:
|
||||
|
||||
- active worker completion;
|
||||
- registered in-flight API calls covered by the ownership contract;
|
||||
- terminal callbacks.
|
||||
|
||||
This cannot make a raw C pointer safe if a caller begins a new call after the
|
||||
object has already been freed. Callers must obey lifetime ownership.
|
||||
|
||||
## Bounded internal parallelism
|
||||
|
||||
A validated Task Kind may use internal participants while the Queue callback
|
||||
remains the sole Task owner.
|
||||
|
||||
Required shape:
|
||||
|
||||
```text
|
||||
one admitted Task owner
|
||||
-> bounded participant count
|
||||
-> bounded private work
|
||||
-> join all participants
|
||||
-> owner-only deterministic publication when required
|
||||
-> Task-specific durable cursor
|
||||
-> generic checkpoint
|
||||
-> sequence_break
|
||||
```
|
||||
|
||||
## Primitives utilisées
|
||||
Participant count and memory must fit the admitted Resource Governor contract.
|
||||
|
||||
| Primitive | Usage |
|
||||
|---|---|
|
||||
| `pthread_mutex_t` | Protection des données partagées |
|
||||
| `pthread_cond_t` | Synchronisation producteur/consommateur |
|
||||
| `pthread_create()` | Création des workers |
|
||||
| `pthread_join()` | Attente de fin des workers |
|
||||
| `pthread_cancel()` | Non utilisé pour interrompre une Task ; annulation coopérative |
|
||||
No participant may silently exceed the installed sequence contract.
|
||||
|
||||
## Invariants de concurrence
|
||||
## Atomicity does not imply serialism
|
||||
|
||||
1. **Un seul thread ncurses** : ncurses n'est jamais appelé depuis un
|
||||
worker. Toute mise à jour de l'UI passe par des variables partagées
|
||||
protégées.
|
||||
Per-item scientific atomicity and cross-item execution width are separate.
|
||||
|
||||
2. **Mutex hiérarchique** : si plusieurs mutex sont acquis, toujours dans
|
||||
le même ordre pour éviter les deadlocks.
|
||||
|
||||
3. **Annulation coopérative** : les workers vérifient périodiquement un
|
||||
drapeau d'annulation. Une Task n'est pas interrompue brutalement.
|
||||
|
||||
4. **Réservation atomique** : la réservation du gouverneur est atomique.
|
||||
Deux threads ne peuvent pas obtenir la même réservation.
|
||||
|
||||
5. **Pas de callback sans réservation** : aucun callback de tâche n'est
|
||||
invoqué sans réservation active. Cet invariant est maintenu même en
|
||||
présence d'erreurs.
|
||||
|
||||
6. **Retraite après callback** : la notification terminale finit avant la
|
||||
destruction du userdata. Queue détruit la Task hors de son mutex et ne
|
||||
conserve ensuite qu'un snapshot borné.
|
||||
|
||||
7. **Fermeture d'ingress** : le propriétaire empêche les nouveaux appels Queue
|
||||
avant `destroy()`. La fermeture interne attend le worker et chaque appel
|
||||
enregistré avant le close ; elle ne peut rendre sûr un appel démarré après
|
||||
la libération d'un pointeur C brut.
|
||||
|
||||
8. **Parallélisme scientifique propriétaire** : lorsqu'un kind emploie des
|
||||
participants internes, le callback Queue demeure l'unique propriétaire. Le
|
||||
nombre de participants et leur mémoire sont admis par le Governor ; seul le
|
||||
propriétaire publie le préfixe durable ordonné et joint tous les enfants.
|
||||
|
||||
9. **Lease SSD par objet** : un lease scratch appartient à l'adresse exacte de
|
||||
l'objet fourni par le caller. Tous ses champs sont lus/écrits sous le mutex
|
||||
du contrôleur. Le caller lui garantit un accès exclusif et ne le copie, ne le
|
||||
déplace ni ne le présente simultanément à deux contrôleurs. En production,
|
||||
acquire/release passent par les wrappers Governor ; le Governor relâche son
|
||||
mutex avant l'appel contrôleur, et le contrôleur ne rappelle jamais le
|
||||
Governor. À la saturation légale `generation == UINT64_MAX`, seule la fin
|
||||
du wrapper exact déjà sérialisé peut réconcilier sa propre opération et le
|
||||
compte fondé sur les adresses ; une update publique au même watermark ne
|
||||
peut pas rendre une autorité stale.
|
||||
|
||||
10. **Owner SSD unique** : la TUI/main demande et poll l'opération ; au plus un
|
||||
thread joinable exécute une opération bornée et ne touche jamais ncurses.
|
||||
Le destroy le joint avant unregister. Une observation malformée enregistre
|
||||
`ERROR` et ne confère aucune autorité de contrôle ou de lease.
|
||||
|
||||
11. **Frontière projet** : les vues libèrent leurs borrows, puis la Queue est
|
||||
annulée/jointe/détruite avant Project DB. Une Queue vide est créée ensuite.
|
||||
Aucun callback terminal ne peut donc déréférencer une DB déjà fermée et
|
||||
l'histoire d'un projet ne fuit pas dans le suivant.
|
||||
|
||||
12. **Ordre d'arrêt global** : Queue et leases Task, puis fermeture projet,
|
||||
join/unregister du binding SSD, contrôleur SSD, et enfin Governor. Un
|
||||
unregister encore bloqué par un lease est un échec observable, jamais un
|
||||
pointeur abandonné.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
### Deadlock
|
||||
|
||||
```c
|
||||
// ❌ Risque de deadlock
|
||||
pthread_mutex_lock(&mutex_a);
|
||||
pthread_mutex_lock(&mutex_b); // attend mutex_b
|
||||
|
||||
// Dans un autre thread :
|
||||
pthread_mutex_lock(&mutex_b);
|
||||
pthread_mutex_lock(&mutex_a); // attend mutex_a → DEADLOCK
|
||||
```text
|
||||
PER_ITEM_ATOMICITY_REQUIRES_CROSS_ITEM_SERIALISM=NO
|
||||
OWNER_ONLY_PUBLICATION_REQUIRES_SERIAL_PREPARATION=NO
|
||||
```
|
||||
|
||||
**Solution** : toujours acquérir les mutex dans le même ordre.
|
||||
Current examples include selected RAW, selected Feature extraction, Candidate
|
||||
Pair source work and outer Geometric Verification preparation.
|
||||
|
||||
### Race condition
|
||||
Serialization is valid only where the subsystem's scientific, persistence,
|
||||
library or measured-throughput contract proves it necessary.
|
||||
|
||||
```c
|
||||
// ❌ Race condition
|
||||
if (task->state == TASK_STATE_QUEUED) {
|
||||
task->state = TASK_STATE_RUNNING;
|
||||
}
|
||||
## CPU/batch coupling
|
||||
|
||||
// ✅ Correct
|
||||
pthread_mutex_lock(&task->mutex);
|
||||
if (task->state == TASK_STATE_QUEUED) {
|
||||
task->state = TASK_STATE_RUNNING;
|
||||
}
|
||||
pthread_mutex_unlock(&task->mutex);
|
||||
CPU and batch/window are not globally independent dimensions.
|
||||
|
||||
For a Task whose additional participants cannot do useful work while the
|
||||
admitted item window remains one, a Task-specific capability may couple those
|
||||
dimensions.
|
||||
|
||||
Current validated examples include:
|
||||
|
||||
```text
|
||||
candidate_pair.generate/1
|
||||
features.extract.batch/1
|
||||
```
|
||||
|
||||
### Use-after-free
|
||||
This is not a universal rule for all Task Kinds.
|
||||
|
||||
```c
|
||||
// ❌ Use-after-free
|
||||
task_destroy(task);
|
||||
task_callback(task); // task est libéré
|
||||
## Project lifetime boundary
|
||||
|
||||
// ✅ Correct : le callback est entièrement revenu avant la destruction
|
||||
task_callback(task);
|
||||
task_destroy(task);
|
||||
Before closing a project:
|
||||
|
||||
```text
|
||||
views release Project DB borrows
|
||||
-> Queue is cancelled/joined/destroyed
|
||||
-> Project DB closes
|
||||
-> fresh empty Queue may be created for the next project
|
||||
```
|
||||
|
||||
## Validation
|
||||
A terminal callback must never observe a Project DB already destroyed.
|
||||
|
||||
Les readers Visual Index sont sans état partagé mutable. Une query copie la
|
||||
liste bornée des segments sous le mutex DB, puis effectue hash, lectures et
|
||||
accumulation après déverrouillage. Un update ne rend le nouveau segment visible
|
||||
qu'au commit memberships+segment ; une query en cours garde son snapshot.
|
||||
Project-specific runtime history must not leak into the next project.
|
||||
|
||||
Pour tout ticket touchant la concurrence, exécuter :
|
||||
## Global shutdown boundary
|
||||
|
||||
The current shutdown order preserves ownership across:
|
||||
|
||||
```text
|
||||
Task Queue and Task leases
|
||||
-> project close
|
||||
-> join/unregister SSD binding
|
||||
-> SSD controller
|
||||
-> Resource Governor
|
||||
```
|
||||
|
||||
A scratch unregister blocked by a real outstanding lease is an observable
|
||||
failure, not permission to abandon a live pointer.
|
||||
|
||||
## SSD lease ownership
|
||||
|
||||
A scratch lease belongs to the exact caller-owned lease object used for the
|
||||
operation.
|
||||
|
||||
Its mutable fields are controlled under the SSD controller mutex.
|
||||
|
||||
The caller must not:
|
||||
|
||||
- copy a live lease;
|
||||
- move a live lease;
|
||||
- present the same lease object to two controllers;
|
||||
- release through a different ownership path.
|
||||
|
||||
Production acquire/release uses the Resource Governor wrappers.
|
||||
|
||||
The Governor releases its own mutex before entering the controller, and the
|
||||
controller does not callback into the Governor while holding its mutex.
|
||||
|
||||
Scratch remains storage capacity, never RAM admission.
|
||||
|
||||
## Visual Index readers
|
||||
|
||||
Visual Index query readers do not share mutable query state.
|
||||
|
||||
A query obtains a bounded segment snapshot under the Project DB boundary, then
|
||||
performs hash/read/accumulation after release.
|
||||
|
||||
A concurrent update makes a new segment visible only at its canonical commit
|
||||
boundary. An already running query continues with its retained snapshot.
|
||||
|
||||
## OpenCV process-wide state
|
||||
|
||||
OpenCV thread configuration is process-wide.
|
||||
|
||||
The active heavy Queue callback owns temporary mutation of that setting where a
|
||||
Task contract requires it and restores the previous/baseline value on all exit
|
||||
paths.
|
||||
|
||||
Internal participants must not independently race `cv::setNumThreads()`.
|
||||
|
||||
For Feature batch, cross-image participants are used while internal OpenCV
|
||||
threading is controlled explicitly.
|
||||
|
||||
## Vulkan boundary
|
||||
|
||||
ORB Vulkan concurrency is validated separately from portable TSan.
|
||||
|
||||
The production AUTO contract currently uses:
|
||||
|
||||
```text
|
||||
normal inflight depth = 1
|
||||
private validated safety depth = 2
|
||||
helpers = 0
|
||||
```
|
||||
|
||||
Depth 2 is a private safety/benchmark capability and was rejected as the normal
|
||||
useful setting by measured throughput.
|
||||
|
||||
A Vulkan backend failure produces complete CPU fallback before publication.
|
||||
Partial GPU scientific output is never published.
|
||||
|
||||
## TSan policy
|
||||
|
||||
For project concurrency changes, use a dedicated TSan build that matches the
|
||||
supported proof boundary.
|
||||
|
||||
The retained global maintenance matrix used:
|
||||
|
||||
```text
|
||||
GCC/G++
|
||||
Vulkan disabled
|
||||
selected concurrent targets
|
||||
deterministic repetitions
|
||||
```
|
||||
|
||||
and completed the retained 14/14 target matrix plus 220 repetitions.
|
||||
|
||||
The narrow suppression file is:
|
||||
|
||||
```text
|
||||
tests/tsan-opencv.supp
|
||||
```
|
||||
|
||||
It covers external non-instrumented OpenCV/TBB objects only.
|
||||
|
||||
It must not suppress Lardon3D frames.
|
||||
|
||||
Therefore the correct retained claim is not "TSan proves all concurrency".
|
||||
It is:
|
||||
|
||||
```text
|
||||
portable project concurrency matrix passed under the documented qualification
|
||||
external OpenCV/TBB reports are narrowly qualified
|
||||
Vulkan concurrency has a separate validation boundary
|
||||
```
|
||||
|
||||
## What TSan does not prove
|
||||
|
||||
TSan is useful for instrumented conflicting memory access and some
|
||||
synchronization misuse.
|
||||
|
||||
It does not prove absence of:
|
||||
|
||||
- deadlock;
|
||||
- lost wakeup caused by incorrect predicate design;
|
||||
- lifetime bugs outside the exercised paths;
|
||||
- races hidden inside non-instrumented external libraries;
|
||||
- Vulkan driver/runtime correctness;
|
||||
- scientific determinism.
|
||||
|
||||
Lock-order review, ownership reasoning and deterministic tests remain required.
|
||||
|
||||
## Sanitizer command policy
|
||||
|
||||
Do not encode fixed `-j8` as canonical validation.
|
||||
|
||||
Example configuration:
|
||||
|
||||
```sh
|
||||
# Build TSan
|
||||
CC=clang meson setup build-tsan --wipe -Db_sanitize=thread -Db_lundef=false
|
||||
meson compile -C build-tsan -j8
|
||||
CC=gcc CXX=g++ meson setup build-tsan -Db_sanitize=thread -Db_lundef=false -Dvulkan_orb=disabled
|
||||
meson compile -C build-tsan
|
||||
meson test -C build-tsan --print-errorlogs
|
||||
```
|
||||
|
||||
TSan détecte automatiquement :
|
||||
Use host-aware compile/test parallelism unless the proof itself requires
|
||||
serialization.
|
||||
|
||||
- les accès concurrents conflictuels instrumentés ;
|
||||
- certaines utilisations incohérentes des primitives de synchronisation.
|
||||
Do not repeatedly wipe an unchanged TSan tree.
|
||||
|
||||
Il ne prouve pas l'absence de deadlock, de signal perdu ou de bug dans une
|
||||
bibliothèque non instrumentée. Les invariants de lifetime et d'ordre de locks
|
||||
restent donc soumis aux tests déterministes et à la revue.
|
||||
## Concurrency review checklist
|
||||
|
||||
### Preuve TSan globale courante
|
||||
Before closing a concurrency-sensitive change, verify:
|
||||
|
||||
La matrice fraîche emploie GCC/G++ 16.2.1 et désactive explicitement Vulkan.
|
||||
Elle passe 14/14 cibles couvrant Task, Project, Queue, Governor, registre/leases
|
||||
SSD, contrôleur SSD, observateur/TUI async, Candidate, Visual Index, Feature,
|
||||
Matcher et GV, puis 220/220 répétitions déterministes : **234/234** au total.
|
||||
- ncurses remains main-thread-only;
|
||||
- every shared mutable field has an explicit synchronization owner;
|
||||
- condition predicates are checked in loops;
|
||||
- lock order remains consistent;
|
||||
- no Task uses forced asynchronous cancellation;
|
||||
- Queue callbacks have an active reservation;
|
||||
- internal participants stay within the admitted contract;
|
||||
- all children join on every exit path;
|
||||
- owner-only publication remains ordered where required;
|
||||
- project-close ordering prevents DB use-after-close;
|
||||
- SSD lease ownership remains exact;
|
||||
- Task userdata outlives terminal notification;
|
||||
- appropriate deterministic concurrency tests pass;
|
||||
- portable TSan qualification is preserved;
|
||||
- Vulkan validation is reported separately;
|
||||
- ASan/UBSan is run when the change also affects lifetime/memory.
|
||||
|
||||
La seule liste de suppressions est `tests/tsan-opencv.supp`, limitée aux objets
|
||||
partagés externes non instrumentés `libopencv_features.so`,
|
||||
`libopencv_core.so` et `libtbb.so`. Elle ne masque aucune frame Lardon3D. Les
|
||||
warnings GCC `-Wmaybe-uninitialized` des contrôles OpenCV Feature/SIFT sont
|
||||
classés non matériels : le callback fournit une Task non nulle et le helper
|
||||
initialise la structure avant toute autre sortie d'échec. Les warnings OpenCV
|
||||
du build GV appartiennent aux headers externes.
|
||||
## Current retained evidence
|
||||
|
||||
Cette preuve TSan ne vaut pas validation de concurrence Vulkan. Le backend
|
||||
ORB Vulkan réel est couvert séparément par le build Clang Vulkan-on 939/939,
|
||||
la suite 65/65 et ses tests de backend/handle/publication ; cette séparation
|
||||
doit rester explicite dans tout rapport.
|
||||
The canonical global-maintenance record is:
|
||||
|
||||
## Checklist de concurrence
|
||||
```text
|
||||
docs/architecture/global_maintenance_audit.md
|
||||
GLOBAL_MAINTENANCE_AUDIT=PASS/FROZEN
|
||||
```
|
||||
|
||||
Avant de livrer un ticket touchant la concurrence :
|
||||
The current A6000 checkpoint is later:
|
||||
|
||||
- [ ] Toutes les variables partagées sont protégées par un mutex
|
||||
- [ ] Les mutex sont toujours libérés (même en cas d'erreur)
|
||||
- [ ] Les variables de condition sont vérifiées dans une boucle `while`
|
||||
- [ ] Aucun appel ncurses depuis un worker
|
||||
- [ ] L'annulation des Tasks est coopérative (pas de `pthread_cancel`)
|
||||
- [ ] TSan ne signale aucune erreur
|
||||
- [ ] Le build ASan ne signale aucune fuite mémoire liée aux threads
|
||||
```text
|
||||
real-a6000-pre-sfm-2026-09-02
|
||||
REAL_A6000_PRE_SFM=PASS/FROZEN
|
||||
```
|
||||
|
||||
The later A6000 proof exercised current bounded parallel paths through selected
|
||||
Feature batch, Candidate, Matcher, Geometric Verifier v3 and Tracks without
|
||||
changing the historical TSan qualification.
|
||||
|
||||
Historical evidence remains historical; new changes require validation scoped
|
||||
to their actual concurrency surface.
|
||||
|
||||
## Summary
|
||||
|
||||
```text
|
||||
NCURSES_OWNER=MAIN_THREAD_ONLY
|
||||
ACTIVE_HEAVY_QUEUE_CALLBACKS=1
|
||||
TASK_CANCELLATION=COOPERATIVE
|
||||
INTERNAL_PARALLELISM=BOUNDED
|
||||
|
||||
PER_ITEM_ATOMICITY_REQUIRES_CROSS_ITEM_SERIALISM=NO
|
||||
OWNER_ONLY_PUBLICATION_REQUIRES_SERIAL_PREPARATION=NO
|
||||
|
||||
TSAN_PORTABLE_PROJECT_MATRIX=QUALIFIED_PASS
|
||||
TSAN_EXTERNAL_OPENCV_TBB=QUALIFIED
|
||||
VULKAN_CONCURRENCY_VALIDATION=SEPARATE
|
||||
|
||||
NO_FIXED_GLOBAL_J8=YES
|
||||
NO_REPEATED_UNCHANGED_WIPE=YES
|
||||
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
```
|
||||
|
|
|
|||
|
|
@ -1,136 +1,298 @@
|
|||
# Procédures de test
|
||||
# Testing
|
||||
|
||||
## Vue d'ensemble
|
||||
## Status
|
||||
|
||||
Lardon3D utilise le framework de test intégré à Meson. Chaque module possède
|
||||
un fichier de test dans `tests/` correspondant au module testé.
|
||||
```text
|
||||
DOCUMENTATION_LANGUAGE=ENGLISH
|
||||
TEST_POLICY=HOST_AWARE
|
||||
REPEATED_UNCHANGED_EXPENSIVE_VALIDATION=AVOID
|
||||
TSAN_OPEN_CV_TBB_QUALIFICATION=REQUIRED
|
||||
VULKAN_CONCURRENCY_VALIDATION=SEPARATE
|
||||
```
|
||||
|
||||
## Lancer les tests
|
||||
Lardon3D uses Meson's test runner. Tests live under `tests/` and combine unit,
|
||||
integration, persistence, restart, resource and real-path validation.
|
||||
|
||||
## Normal commands
|
||||
|
||||
Run the configured suite:
|
||||
|
||||
```sh
|
||||
# Tous les tests
|
||||
meson test -C build --print-errorlogs
|
||||
```
|
||||
|
||||
# Un test spécifique
|
||||
meson test -C build test_task_queue --print-errorlogs
|
||||
Run one named test:
|
||||
|
||||
# Tests avec verbose
|
||||
```sh
|
||||
meson test -C build <test-name> --print-errorlogs
|
||||
```
|
||||
|
||||
Verbose execution:
|
||||
|
||||
```sh
|
||||
meson test -C build -v --print-errorlogs
|
||||
```
|
||||
|
||||
# Réexécuter uniquement les tests échoués
|
||||
Re-run failures only:
|
||||
|
||||
```sh
|
||||
meson test -C build --reprint=failed
|
||||
```
|
||||
|
||||
## Structure des tests
|
||||
Use the names registered by the current `meson.build`; this document does not
|
||||
maintain a second authoritative list of every test target.
|
||||
|
||||
## Validation policy
|
||||
|
||||
Validation must match the change.
|
||||
|
||||
A documentation-only change normally requires:
|
||||
|
||||
```text
|
||||
tests/
|
||||
├── test_task_queue.c # tests de la file de tâches
|
||||
├── test_task.c # tests du module task
|
||||
├── test_resource_governor.c # tests du gouverneur
|
||||
├── test_hardware_profile.c # tests du profil matériel
|
||||
├── test_import.c # tests de l'import
|
||||
├── test_project.c # tests des projets
|
||||
└── test_*.c # autres modules
|
||||
git diff --check
|
||||
targeted content checks
|
||||
targeted link/authority review
|
||||
```
|
||||
|
||||
## Écrire un test
|
||||
It does not justify wiping and rebuilding unchanged code.
|
||||
|
||||
```c
|
||||
#include <glib.h>
|
||||
#include "lardon3d/task.h"
|
||||
A code change normally requires, in increasing scope:
|
||||
|
||||
void test_task_create(void) {
|
||||
task_estimate_t est = {
|
||||
.ram_bytes = 1024 * 1024,
|
||||
.gpu_bytes = 0,
|
||||
.cpu_weight = 1,
|
||||
.io_weight = 0,
|
||||
.batch_size = 10,
|
||||
.batch_max = 100
|
||||
};
|
||||
task_t *t = task_create("test", &est, NULL, NULL);
|
||||
g_assert_nonnull(t);
|
||||
g_assert_cmpint(task_get_state(t), ==, TASK_STATE_IDLE);
|
||||
task_destroy(t);
|
||||
}
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
g_test_init(&argc, &argv, NULL);
|
||||
g_test_add_func("/task/create", test_task_create);
|
||||
return g_test_run();
|
||||
}
|
||||
```text
|
||||
targeted build
|
||||
targeted tests
|
||||
broader affected suite
|
||||
sanitizer or concurrency validation when relevant
|
||||
full suite when the change or release boundary justifies it
|
||||
```
|
||||
|
||||
## Conventions
|
||||
Do not repeatedly rerun an unchanged expensive suite between documentation
|
||||
edits merely to create activity.
|
||||
|
||||
1. **Préfixe `test_`** : chaque fonction de test porte le préfixe `test_`.
|
||||
2. **Chemin hiérarchique** : le nom du test suit le pattern `/module/action`.
|
||||
3. **Asserts GLib** : utiliser `g_assert_*` pour les vérifications.
|
||||
4. **Nettoyage** : chaque test libère toutes ses ressources.
|
||||
5. **Isolation** : un test ne dépend pas de l'état d'un autre test.
|
||||
6. **Déterminisme** : les tests ne dépendent pas de l'heure, du filesystem
|
||||
ou de l'état réseau (sauf test d'import).
|
||||
## Host-aware parallelism
|
||||
|
||||
## Commentaires source
|
||||
Build and test parallelism are host-aware.
|
||||
|
||||
Les commentaires documentent le pourquoi et les contrats non évidents :
|
||||
invariants, propriété et durée de vie, persistance, ainsi que limites et
|
||||
frontières de ressources. Les API publiques documentent leurs contrats non
|
||||
évidents. Ils ne paraphrasent pas le code ligne par ligne et sont mis à jour
|
||||
avec tout changement de comportement.
|
||||
Do not encode a project-wide fixed `-j8`, `--num-processes 1`, or equivalent
|
||||
constant as canonical policy.
|
||||
|
||||
## Tests unitaires vs tests d'intégration
|
||||
The correct width depends on the current machine, interactive reserve, memory,
|
||||
toolchain and workload. Use all safe useful host capacity while preserving the
|
||||
defined interactive reserve.
|
||||
|
||||
`test-visual-index` couvre les descriptors synthétiques, le retrieval ORB réel,
|
||||
les filtres inter-ScanSets, quatre queries concurrentes, la corruption/absence/
|
||||
troncature d'un segment et 4 000 Feature Sets synthétiques. Le scénario de
|
||||
reprise `visual_index.update` est exercé dans `test-feature-task`.
|
||||
```text
|
||||
RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
```
|
||||
|
||||
| Type | Portée | Fichier |
|
||||
|---|---|---|
|
||||
| Unitaire | Un module isolé | `tests/test_<module>.c` |
|
||||
| Intégration | Interaction entre modules | `tests/test_<module>.c` avec dépendances réelles |
|
||||
If a temporary validation must be serialized for determinism, diagnosis or a
|
||||
known tool limitation, label that serialization as test-specific evidence
|
||||
rather than a global default.
|
||||
|
||||
## Validation par ticket
|
||||
## Fresh build policy
|
||||
|
||||
Avant de livrer un ticket, exécuter la séquence complète :
|
||||
Do not use `meson setup --wipe` by default.
|
||||
|
||||
Prefer:
|
||||
|
||||
```sh
|
||||
# 1. Build clean
|
||||
CC=clang meson setup build --wipe
|
||||
meson compile -C build -j8
|
||||
|
||||
# 2. Tests
|
||||
meson test -C build --print-errorlogs
|
||||
|
||||
# 3. Style
|
||||
git diff --check
|
||||
|
||||
# 4. Si mémoire/concurrence touchés
|
||||
CC=clang meson setup build-asan --wipe -Db_sanitize=address,undefined
|
||||
meson compile -C build-asan -j8
|
||||
meson test -C build-asan --print-errorlogs
|
||||
|
||||
# 5. Si concurrence touchée
|
||||
CC=clang meson setup build-tsan --wipe -Db_sanitize=thread -Db_lundef=false
|
||||
meson compile -C build-tsan -j8
|
||||
meson test -C build-tsan --print-errorlogs
|
||||
meson setup --reconfigure build
|
||||
meson compile -C build
|
||||
```
|
||||
|
||||
## Dépannage
|
||||
Create or wipe a build directory when the configuration genuinely needs a
|
||||
fresh environment, for example:
|
||||
|
||||
### Test qui échoue en ASan
|
||||
```text
|
||||
different sanitizer set
|
||||
portable Vulkan-off proof
|
||||
Vulkan-on proof
|
||||
compiler-family change
|
||||
known stale/corrupt build directory
|
||||
release-grade clean proof
|
||||
```
|
||||
|
||||
Vérifier les durées de vie des allocations. Ne jamais libérer un objet puis
|
||||
y accéder. Vérifier que chaque `task_destroy()` est appelée.
|
||||
Repeated wipes of the same unchanged configuration waste time and invalidate
|
||||
incremental-build advantages.
|
||||
|
||||
### Test qui échoue en TSan
|
||||
## Sanitizers
|
||||
|
||||
Vérifier que toutes les variables partagées sont protégées par un mutex.
|
||||
Vérifier que ncurses est utilisé uniquement depuis le thread principal.
|
||||
### ASan / UBSan
|
||||
|
||||
### Test qui échoue uniquement en release
|
||||
For memory, lifetime, ownership or undefined-behavior changes, use a dedicated
|
||||
sanitizer build.
|
||||
|
||||
Vérifier les assertions et les overflow arithmétiques. Compiler avec
|
||||
`-fsanitize=undefined` pour détecter les comportements indéfinis.
|
||||
Example configuration:
|
||||
|
||||
```sh
|
||||
CC=clang CXX=clang++ meson setup build-asan -Db_sanitize=address,undefined
|
||||
meson compile -C build-asan
|
||||
meson test -C build-asan --print-errorlogs
|
||||
```
|
||||
|
||||
Reconfigure or wipe only when the existing sanitizer directory does not match
|
||||
the requested configuration.
|
||||
|
||||
### LeakSanitizer qualification
|
||||
|
||||
The retained global maintenance evidence must not be summarized as
|
||||
`LSan 64/64`.
|
||||
|
||||
The full first leak-enabled run exposed an externally attributed OpenCL loader
|
||||
leak and two timeout anomalies. The retained qualified result is:
|
||||
|
||||
```text
|
||||
ASan/UBSan full suite: PASS with detect_leaks=0
|
||||
proved subset without the external loader: LSan PASS
|
||||
full leak-enabled suite: not an unqualified PASS
|
||||
```
|
||||
|
||||
Preserve that distinction in future reports unless new evidence supersedes it.
|
||||
|
||||
## ThreadSanitizer
|
||||
|
||||
Concurrency changes require TSan where the instrumented boundary is meaningful.
|
||||
|
||||
The retained global maintenance TSan proof used GCC/G++ with Vulkan disabled
|
||||
and covered the selected concurrent targets plus deterministic repetitions.
|
||||
|
||||
The only retained suppression file is:
|
||||
|
||||
```text
|
||||
tests/tsan-opencv.supp
|
||||
```
|
||||
|
||||
Its purpose is limited to external OpenCV/TBB objects. It must not suppress
|
||||
Lardon3D frames.
|
||||
|
||||
Therefore never report a blanket statement such as:
|
||||
|
||||
```text
|
||||
TSan proves the entire Vulkan build race-free
|
||||
```
|
||||
|
||||
The valid qualification is:
|
||||
|
||||
```text
|
||||
project concurrent paths covered by the retained portable TSan matrix
|
||||
external OpenCV/TBB reports qualified by the narrow suppression boundary
|
||||
Vulkan concurrency validated separately
|
||||
```
|
||||
|
||||
## Vulkan validation
|
||||
|
||||
ORB Vulkan uses a separate validation boundary.
|
||||
|
||||
The retained global maintenance evidence includes a Vulkan-on build and suite
|
||||
on the real Radeon 780M/RADV host, plus dedicated backend/handle/publication
|
||||
tests.
|
||||
|
||||
That evidence is not interchangeable with TSan.
|
||||
|
||||
SIFT/RootSIFT feasibility results did not establish a production GPU backend;
|
||||
do not turn feasibility checks into production validation claims.
|
||||
|
||||
## Determinism and repetition
|
||||
|
||||
Repeat tests when repetition proves something specific:
|
||||
|
||||
```text
|
||||
deterministic restart
|
||||
race sensitivity
|
||||
resource adaptation
|
||||
ordering stability
|
||||
flaky regression reproduction
|
||||
```
|
||||
|
||||
Do not repeat unchanged tests without a stated purpose.
|
||||
|
||||
When repetition is the evidence, record:
|
||||
|
||||
```text
|
||||
exact test/corpus
|
||||
run count
|
||||
relevant configuration
|
||||
success/failure count
|
||||
digest or invariant when applicable
|
||||
```
|
||||
|
||||
## Test isolation
|
||||
|
||||
Tests should:
|
||||
|
||||
- own and clean up their temporary resources;
|
||||
- avoid depending on another test's execution order;
|
||||
- avoid network state unless the test explicitly owns that dependency;
|
||||
- use synthetic/private Resource snapshots where the test is about deterministic
|
||||
policy rather than live host telemetry;
|
||||
- avoid changing global process state without restoring it.
|
||||
|
||||
OpenCV thread configuration is process-wide and must be restored on every exit
|
||||
path in tests that change it.
|
||||
|
||||
## Public-header validation
|
||||
|
||||
When a public C header changes, run a standalone C17 syntax probe in addition to
|
||||
normal build coverage.
|
||||
|
||||
Conceptually:
|
||||
|
||||
```sh
|
||||
cc -x c -std=c17 -fsyntax-only -Iinclude -include lardon3d/<header>.h /dev/null
|
||||
```
|
||||
|
||||
Use the current supported compiler matrix when the change affects ABI or
|
||||
C/C++ interoperability.
|
||||
|
||||
## Source comments
|
||||
|
||||
Source comments explain non-obvious contracts:
|
||||
|
||||
```text
|
||||
invariants
|
||||
ownership and lifetime
|
||||
persistence ordering
|
||||
resource boundaries
|
||||
recovery behavior
|
||||
scientific constraints
|
||||
```
|
||||
|
||||
They should not paraphrase obvious code line by line.
|
||||
|
||||
Repository source comments are English.
|
||||
|
||||
## Current retained global maintenance evidence
|
||||
|
||||
The canonical detailed record is:
|
||||
|
||||
```text
|
||||
docs/architecture/global_maintenance_audit.md
|
||||
GLOBAL_MAINTENANCE_AUDIT=PASS/FROZEN
|
||||
```
|
||||
|
||||
That historical checkpoint includes fresh portable/Vulkan builds, full suites,
|
||||
sanitizer work, TSan work, public-header probes, ABI/application-link checks and
|
||||
independent review.
|
||||
|
||||
It remains historical evidence. New changes require only the validation
|
||||
appropriate to the changed surface unless a new global checkpoint is being
|
||||
created.
|
||||
|
||||
## Ticket closure checklist
|
||||
|
||||
Before closing a code ticket:
|
||||
|
||||
- confirm the requested scope only was changed;
|
||||
- run `git diff --check`;
|
||||
- run targeted tests for changed behavior;
|
||||
- run the affected broader suite when justified;
|
||||
- run ASan/UBSan for memory/lifetime-sensitive changes;
|
||||
- run TSan for concurrency-sensitive project code when applicable;
|
||||
- keep Vulkan validation separate from portable TSan claims;
|
||||
- preserve exact external-library qualifications;
|
||||
- avoid fixed host-parallelism constants;
|
||||
- avoid repeated unchanged clean builds or suites;
|
||||
- report what actually ran, not a stronger claim.
|
||||
|
||||
For documentation-only remediation, use documentation checks rather than
|
||||
rebuilding unchanged production code.
|
||||
|
|
|
|||
1447
docs/product/product_definition.md
Normal file
1447
docs/product/product_definition.md
Normal file
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
|
|
@ -9,9 +9,9 @@
|
|||
#include <lardon3d/visual_index.h>
|
||||
|
||||
typedef struct {
|
||||
uint32_t generated_count; // paires nouvellement créées
|
||||
uint32_t skipped_count; // paires déjà existantes (idempotence)
|
||||
uint32_t queried_count; // candidats retournés par le Visual Index
|
||||
uint32_t generated_count; // newly created pairs
|
||||
uint32_t skipped_count; // already existing pairs (idempotent reuse)
|
||||
uint32_t queried_count; // candidates returned by the Visual Index
|
||||
} Lardon3DCandidatePairGenStats;
|
||||
|
||||
Lardon3DVisualIndexResult lardon3d_candidate_pair_generate(
|
||||
|
|
|
|||
|
|
@ -42,7 +42,7 @@ bool lardon3d_project_enqueue_image_import(Lardon3DAppState *state, uint64_t sca
|
|||
bool lardon3d_image_import_reconstruct(const Lardon3DTaskDurableSnapshot *snapshot, void *context,
|
||||
Lardon3DTaskKindBinding *binding);
|
||||
|
||||
/* Compatibilité TUI : handle léger sur une tâche de la queue, sans thread. */
|
||||
/* TUI compatibility: lightweight handle over a Queue Task, with no private thread. */
|
||||
Lardon3DImportTask *lardon3d_import_task_create(void);
|
||||
bool lardon3d_import_task_start(Lardon3DImportTask *task, Lardon3DAppState *state,
|
||||
const char *source_directory);
|
||||
|
|
|
|||
|
|
@ -325,11 +325,11 @@ const char *lardon3d_resource_decision_name(
|
|||
Lardon3DResourcePressure lardon3d_resource_governor_pressure(
|
||||
Lardon3DResourceGovernor *governor
|
||||
);
|
||||
/* Enregistre les métriques d'un lot terminé pour l'adaptation dynamique
|
||||
* de la taille des lots futurs. batch_size est le nombre d'éléments dont le
|
||||
* traitement a été validé dans ce lot. peak_memory_bytes == 0 signifie que
|
||||
* la mesure est inconnue et n'alimente jamais l'adaptation mémoire. Le buffer
|
||||
* est borné (8 entrées par classe de tâche). Thread-safe. */
|
||||
/* Records metrics from one completed batch for dynamic adaptation of future
|
||||
* batch sizes. batch_size is the number of items whose processing was validated
|
||||
* in this batch. peak_memory_bytes == 0 means the measurement is unknown and
|
||||
* never feeds memory adaptation. The buffer is bounded to 8 entries per Task
|
||||
* class. Thread-safe. */
|
||||
bool lardon3d_resource_governor_record_batch(
|
||||
Lardon3DResourceGovernor *governor,
|
||||
Lardon3DResourceTaskClass task_class,
|
||||
|
|
|
|||
|
|
@ -108,8 +108,8 @@ Lardon3DTask *lardon3d_task_create_typed(
|
|||
Lardon3DTaskUserdataDestroy userdata_destroy
|
||||
);
|
||||
void lardon3d_task_destroy(Lardon3DTask *task);
|
||||
/* Exécute le callback dans le thread appelant. Le callback est invoqué hors
|
||||
* mutex de tâche; le contract d'exécution et l'état appartiennent à la tâche.
|
||||
/* Executes the callback on the calling thread. The callback runs outside
|
||||
* the Task mutex; the execution contract and state remain owned by the Task.
|
||||
*/
|
||||
bool lardon3d_task_start(
|
||||
Lardon3DTask *task,
|
||||
|
|
@ -169,16 +169,16 @@ Lardon3DTask *lardon3d_task_restore_typed(
|
|||
void *userdata,
|
||||
Lardon3DTaskUserdataDestroy userdata_destroy
|
||||
);
|
||||
/* Une restauration typée réussie transfère userdata/userdata_destroy à la
|
||||
* tâche. En cas d'échec, l'appelant en reste propriétaire. */
|
||||
/* A successful typed restore transfers userdata/userdata_destroy ownership
|
||||
* to the Task. On failure, the caller retains ownership. */
|
||||
bool lardon3d_task_kind_is_valid(const char *task_kind);
|
||||
bool lardon3d_task_kind(
|
||||
const Lardon3DTask *task,
|
||||
char task_kind[LARDON3D_TASK_KIND_CAPACITY],
|
||||
uint32_t *task_kind_version
|
||||
);
|
||||
/* Appelé au plus une fois, hors mutex de tâche et après libération de la
|
||||
* réservation terminale. Le userdata de tâche reste vivant jusqu'au retour. */
|
||||
/* Invoked at most once, outside the Task mutex and after the terminal
|
||||
* reservation is released. Task userdata remains alive until the callback returns. */
|
||||
bool lardon3d_task_set_finished_callback(
|
||||
Lardon3DTask *task,
|
||||
Lardon3DTaskFinishedCallback callback,
|
||||
|
|
@ -190,21 +190,20 @@ bool lardon3d_task_resource_estimate(
|
|||
const Lardon3DTask *task,
|
||||
Lardon3DResourceEstimate *estimate
|
||||
);
|
||||
/* L'exécution ne reçoit pas de politique d'admission : c'est au gouverneur de
|
||||
* confirmer la réservation avant l'exécution.
|
||||
/* Execution does not receive admission policy: the Governor must confirm
|
||||
* the reservation before execution.
|
||||
*/
|
||||
bool lardon3d_task_execution_contract(
|
||||
const Lardon3DTask *task,
|
||||
Lardon3DTaskExecutionContract *contract
|
||||
);
|
||||
/* Libère la réservation courante, en obtient une nouvelle auprès du gouverneur
|
||||
* et met à jour le contrat. À appeler uniquement depuis le callback en cours
|
||||
* d'exécution. Une réponse WAIT du gouverneur est une indisponibilité
|
||||
* temporaire : la fonction attend un changement de ressources puis retente
|
||||
* l'admission sans échouer la tâche. Les bornes de lot se poursuivent après
|
||||
* cette nouvelle admission. Retourne false si la tâche est annulée
|
||||
* (TASK_CANCELLED), si le gouverneur répond REJECT ou en cas d'erreur interne
|
||||
* (TASK_FAILED). */
|
||||
/* Releases the current reservation, obtains a new one from the Governor,
|
||||
* and updates the execution contract. Call only from the currently executing
|
||||
* callback. A Governor WAIT is temporary unavailability: this function waits
|
||||
* for a resource change and retries admission without failing the Task. Batch
|
||||
* bounds continue under the new admission. Returns false if the Task is
|
||||
* cancelled (TASK_CANCELLED), the Governor returns REJECT, or an internal
|
||||
* error fails the Task (TASK_FAILED). */
|
||||
bool lardon3d_task_sequence_break(
|
||||
Lardon3DTask *task,
|
||||
Lardon3DResourceGovernor *governor,
|
||||
|
|
|
|||
|
|
@ -42,10 +42,6 @@ typedef struct {
|
|||
* serialized worker. capacity bounds pending Tasks, not terminal history; the
|
||||
* Governor retains resource-admission ownership and must outlive the Queue.
|
||||
*
|
||||
* File d'exécution bornée, à ordre d'attente FIFO avec sélection adaptative du
|
||||
* premier travail admissible, et un seul worker: ownership d'ordonnancement et
|
||||
* de backpressure seulement. L'admission des demandes reste au Governneur.
|
||||
*
|
||||
* CONTRACT: after a successful add, the Queue owns Task and its userdata. Once
|
||||
* the Task is terminal and its finished callback has returned, the Queue keeps
|
||||
* only a snapshot and promptly destroys the real Task outside the Queue lock.
|
||||
|
|
@ -73,8 +69,8 @@ Lardon3DTaskQueue *lardon3d_task_queue_create(
|
|||
* starts after destruction safe. NULL is accepted. Never call destroy
|
||||
* synchronously from a Task finished callback running on this Queue. */
|
||||
void lardon3d_task_queue_destroy(Lardon3DTaskQueue *queue);
|
||||
/* La file devient propriétaire de task uniquement en cas de succès.
|
||||
* Bloquante : attend une place libre si la file est pleine. A zero Task ID is
|
||||
/* The Queue takes ownership of task only on success.
|
||||
* Blocking: waits for a free slot when the Queue is full. A zero Task ID is
|
||||
* assigned from a nonzero monotonic sequence and is never generated twice
|
||||
* during this Queue lifetime, including after terminal-history eviction or
|
||||
* removal. Once UINT64_MAX has been generated (or consumed by a restored
|
||||
|
|
@ -85,8 +81,8 @@ bool lardon3d_task_queue_add(
|
|||
Lardon3DTask *task,
|
||||
uint64_t *task_id
|
||||
);
|
||||
/* Non-bloquante : retourne false si la file est pleine ou en arrêt.
|
||||
La file devient propriétaire de task uniquement en cas de succès. */
|
||||
/* Non-blocking: returns false if the Queue is full or stopping.
|
||||
The Queue takes ownership of task only on success. */
|
||||
bool lardon3d_task_queue_try_add(
|
||||
Lardon3DTaskQueue *queue,
|
||||
Lardon3DTask *task,
|
||||
|
|
|
|||
|
|
@ -397,9 +397,9 @@ bool run_impl(Lardon3DTask *t, void *p) {
|
|||
c->db, lardon3d_task_id(t), group_id, materialized_capture_id,
|
||||
group_id) != LARDON3D_PROJECT_DB_OK)
|
||||
return lardon3d_task_fail(t, "Rétention de Capture impossible.");
|
||||
/* Limite de reprise acceptée: entre le retour de S3-E et cette rétention
|
||||
* durable, une identité de capture ne peut pas être déduite à posteriori
|
||||
* depuis les chemins/métadonnées/ID d'image.
|
||||
/* Accepted recovery boundary: between S3-E returning and this durable
|
||||
* retention, Capture identity cannot be reconstructed retroactively from
|
||||
* paths, metadata, or image IDs.
|
||||
*/
|
||||
#ifdef LARDON3D_ACQUISITION_CAMPAIGN_TASK_TESTING
|
||||
const char *after_retention =
|
||||
|
|
|
|||
|
|
@ -1797,8 +1797,8 @@ record_batch_locked(
|
|||
metrics[head] = (Lardon3DBatchMetrics) {
|
||||
.batch_size = batch_size,
|
||||
.duration_ns = duration_ns,
|
||||
/* Zéro est le marqueur persistant « mesure inconnue ». La boucle
|
||||
* d'adaptation ignore explicitement ces échantillons. */
|
||||
/* Zero is the sentinel for an unknown measurement. The adaptation
|
||||
* loop explicitly ignores these samples. */
|
||||
.peak_memory_bytes = peak_memory_bytes,
|
||||
};
|
||||
head = (head + 1) % LARDON3D_BATCH_METRICS_CAPACITY;
|
||||
|
|
@ -1836,9 +1836,9 @@ adaptive_batch_limit(
|
|||
% LARDON3D_BATCH_METRICS_CAPACITY;
|
||||
const Lardon3DBatchMetrics *m = &governor->batch_metrics[class_index][idx];
|
||||
if (m->batch_size > 0 && m->peak_memory_bytes > 0) {
|
||||
/* Coût par élément le plus défavorable observé : une moyenne
|
||||
* sous-estimerait le pic et laisserait un lot dépasser son
|
||||
* budget. La stabilité de l'hôte prime sur le débit. */
|
||||
/* Use the worst observed per-item cost: an average would
|
||||
* underestimate the peak and could let a batch exceed its
|
||||
* budget. Host safety takes precedence over throughput. */
|
||||
uint64_t per_item = m->peak_memory_bytes / m->batch_size
|
||||
+ (m->peak_memory_bytes % m->batch_size != 0);
|
||||
if (per_item > measured_per_item) {
|
||||
|
|
@ -1853,9 +1853,9 @@ adaptive_batch_limit(
|
|||
if (measured_per_item <= memory_bytes_per_item) {
|
||||
return static_batch;
|
||||
}
|
||||
/* Éviter l'overflow de la multiplication : si static_batch est trop
|
||||
* grand pour être multiplié sans débordement, on retourne 1 (le lot le
|
||||
* plus conservateur possible) plutôt que de saturer à SIZE_MAX. */
|
||||
/* Avoid multiplication overflow: if static_batch is too large to
|
||||
* multiply safely, return 1 (the most conservative batch) instead of
|
||||
* saturating to SIZE_MAX. */
|
||||
if (static_batch > UINT64_MAX / memory_bytes_per_item) {
|
||||
return 1;
|
||||
}
|
||||
|
|
@ -3222,10 +3222,10 @@ evaluate_locked(
|
|||
)
|
||||
);
|
||||
}
|
||||
/* Le lot maximal visé est corrigé par les métriques mesurées : c'est la
|
||||
* nouvelle cible du contrat, pas une réduction faute de ressources. La
|
||||
* correction ne descend jamais sous minimum_batch_size pour éviter un
|
||||
* WAIT persistant. */
|
||||
/* Measured metrics adjust the target maximum batch: this becomes the
|
||||
* contract target, not a reduction caused by current resource shortage.
|
||||
* The adjustment never falls below minimum_batch_size, avoiding a
|
||||
* persistent WAIT. */
|
||||
size_t adapted_maximum = adaptive_batch_limit(
|
||||
governor,
|
||||
estimate->task_class,
|
||||
|
|
@ -3626,7 +3626,7 @@ lardon3d_resource_governor_record_batch(
|
|||
return false;
|
||||
}
|
||||
if (batch_size == 0) {
|
||||
/* No-op réussi : aucune métrique, aucun réveil inutile. */
|
||||
/* Successful no-op: no metrics and no unnecessary wake-up. */
|
||||
return true;
|
||||
}
|
||||
(void)pthread_mutex_lock(&governor->mutex);
|
||||
|
|
|
|||
16
src/task.c
16
src/task.c
|
|
@ -612,8 +612,8 @@ lardon3d_task_destroy(Lardon3DTask *task)
|
|||
"Annulation demandée.");
|
||||
(void)pthread_cond_broadcast(&task->condition);
|
||||
} else {
|
||||
/* Une tâche locale jamais soumise peut être abandonnée sans
|
||||
* publier une fausse annulation métier. */
|
||||
/* A local Task that was never submitted may be abandoned without
|
||||
* publishing a false business-level cancellation. */
|
||||
finish_locked(task, TASK_CANCELLED, "Tâche abandonnée.");
|
||||
task->finished_notified = true;
|
||||
}
|
||||
|
|
@ -885,7 +885,7 @@ lardon3d_task_checkpoint(Lardon3DTask *task)
|
|||
}
|
||||
|
||||
enum {
|
||||
/* Attente bornée entre deux tentatives d'admission : 50 ms. */
|
||||
/* Bounded wait between admission attempts: 50 ms. */
|
||||
LARDON3D_SEQUENCE_ADMISSION_WAIT_NS = 50000000ULL,
|
||||
};
|
||||
|
||||
|
|
@ -931,7 +931,7 @@ lardon3d_task_sequence_break(
|
|||
(void)lardon3d_resource_governor_release(governor, previous);
|
||||
}
|
||||
for (;;) {
|
||||
/* Vérifier pause et annulation avant chaque tentative d'admission. */
|
||||
/* Check pause and cancellation before every admission attempt. */
|
||||
(void)pthread_mutex_lock(&task->mutex);
|
||||
while (task->pause_requested && !task->cancel_requested) {
|
||||
task->state = TASK_PAUSED;
|
||||
|
|
@ -962,7 +962,7 @@ lardon3d_task_sequence_break(
|
|||
&next
|
||||
);
|
||||
if (!admitted) {
|
||||
/* Erreur interne : échec d'allocation ou d'instantané. */
|
||||
/* Internal error: allocation or resource-snapshot failure. */
|
||||
if (next) {
|
||||
(void)lardon3d_resource_governor_release(governor, next);
|
||||
}
|
||||
|
|
@ -1079,8 +1079,8 @@ lardon3d_task_sequence_break(
|
|||
(void)pthread_mutex_unlock(&task->mutex);
|
||||
return false;
|
||||
case LARDON3D_RESOURCE_WAIT:
|
||||
/* Indisponibilité temporaire : ne pas échouer, attendre un
|
||||
* changement de ressources puis retenter l'admission. */
|
||||
/* Temporary unavailability: do not fail the Task; wait for a
|
||||
* resource change and retry admission. */
|
||||
if (next) {
|
||||
(void)lardon3d_resource_governor_release(governor, next);
|
||||
}
|
||||
|
|
@ -1091,7 +1091,7 @@ lardon3d_task_sequence_break(
|
|||
);
|
||||
break;
|
||||
default:
|
||||
/* Décision inconnue : erreur interne, ne jamais boucler. */
|
||||
/* Unknown decision: internal error; never loop on it. */
|
||||
if (next) {
|
||||
(void)lardon3d_resource_governor_release(governor, next);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -290,9 +290,9 @@ unlink_pending(Lardon3DTaskQueue *queue, TaskNode *previous, TaskNode *node)
|
|||
}
|
||||
node->next_pending = NULL;
|
||||
--queue->pending_count;
|
||||
/* Chaque retrait libère une place. Plusieurs producteurs peuvent dormir
|
||||
* pendant que le worker retire plusieurs tâches avant qu'ils reprennent
|
||||
* le mutex ; chacun de ces retraits doit donc produire un réveil. */
|
||||
/* Every removal frees one slot. Multiple producers may be asleep while
|
||||
* the worker removes several Tasks before they reacquire the mutex, so
|
||||
* every removal must signal one waiter. */
|
||||
(void)pthread_cond_signal(&queue->not_full);
|
||||
}
|
||||
|
||||
|
|
@ -361,12 +361,11 @@ destroy_retired(TaskNode *retired)
|
|||
}
|
||||
}
|
||||
|
||||
/* Parcourt la file d'attente dans son ordre FIFO et sélectionne la première
|
||||
* tâche admissible; une attente de ressources peut donc laisser passer une
|
||||
* tâche antérieure sans lui faire perdre sa place dans la file.
|
||||
* Les tâches terminales ou refusées sont retirées de la file d'attente.
|
||||
* Une tâche en attente de ressources reste en file et sera réévaluée.
|
||||
* Retourne NULL si aucune tâche ne peut démarrer immédiatement. */
|
||||
/* Scans pending Tasks in FIFO order and selects the first admissible one.
|
||||
* A resource WAIT may therefore let a later Task run without removing the
|
||||
* waiting Task from its position. Terminal or rejected Tasks are removed.
|
||||
* A resource-waiting Task remains queued for later re-evaluation.
|
||||
* Returns NULL when no Task can start immediately. */
|
||||
static TaskNode *
|
||||
select_admissible(
|
||||
Lardon3DTaskQueue *queue,
|
||||
|
|
@ -541,10 +540,10 @@ queue_worker(void *context)
|
|||
);
|
||||
}
|
||||
}
|
||||
/* La tâche peut avoir libéré et re-réservé via sequence_break pendant
|
||||
* son callback. Dans ce cas la réservation d'origine est déjà libérée
|
||||
* et cet appel est sans effet ; la réservation courante de la tâche a
|
||||
* été libérée par lardon3d_task_start. */
|
||||
/* The Task may have released and re-reserved through sequence_break
|
||||
* during its callback. In that case the original reservation is
|
||||
* already released and this call is a no-op; lardon3d_task_start
|
||||
* released the Task's current reservation. */
|
||||
if (reservation) {
|
||||
(void)lardon3d_resource_governor_release(
|
||||
queue->governor,
|
||||
|
|
@ -763,7 +762,7 @@ lardon3d_task_queue_destroy(Lardon3DTaskQueue *queue)
|
|||
free(queue);
|
||||
}
|
||||
|
||||
/* Appelée sous le mutex queue. Ne signale pas not_empty sur échec. */
|
||||
/* Called with the Queue mutex held. Does not signal not_empty on failure. */
|
||||
static bool
|
||||
enqueue_locked(
|
||||
Lardon3DTaskQueue *queue,
|
||||
|
|
|
|||
Loading…
Reference in a new issue