diff --git a/AGENTS.md b/AGENTS.md index fe54e84..06e391a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,15 +4,15 @@ - The root `README.md` is the index of project documentation. - Canonical documents under `docs/**` define scientific contracts, architecture, - FROZEN invariants, persistence semantics, lifecycle state, and roadmap - ordering. + FROZEN invariants, persistence semantics, lifecycle state, resource policy, + and roadmap ordering. - Before changing an architectural, scientific, persistence, runtime, or resource-sensitive area, identify and read the relevant canonical document and its applicable invariants. - FROZEN documentation and contracts must never be changed silently to fit an implementation. - A lower-level implementation convenience never overrides a higher-level - canonical contract. + canonical contract or an explicit human decision. - Lifecycle markers such as `PLANNED`, `IMPLEMENTED`, `VALIDATION PENDING`, and `PASS/FROZEN` describe repository state and must match actual implementation and validation evidence. @@ -30,11 +30,12 @@ through an explicitly authorized, explicitly scoped human ticket: - Gates A–G — PASS/FROZEN - Track Model — PASS/FROZEN -- Track Builder — PASS/FROZEN +- Track Builder scientific contract — PASS/FROZEN - F0 — PASS/FROZEN - Phase H v1 — PASS/FROZEN - MVS-M1 — PASS/FROZEN - Project DB v22 scientific/persistence foundation — PASS/FROZEN +- Project DB v23 generic optical-context overlay — PASS/FROZEN - Calibration Bootstrap v1 — PASS/FROZEN - Selected Scientific Execution — PASS/FROZEN - Photo Quality Triage / Acquisition Selection — PASS/FROZEN @@ -43,25 +44,30 @@ through an explicitly authorized, explicitly scoped human ticket: - S3 Capture / Acquisition Ingestion — PASS/FROZEN - Durable Acquisition-Campaign Execution — PASS/FROZEN - Global Maintenance Audit — PASS/FROZEN +- Real S21 Tracks scientific result — PASS/FROZEN Detailed subcontracts remain defined by their canonical documents. This file -does not duplicate every S3 substage, scientific threshold, migration detail, +must not duplicate every S3 substage, scientific threshold, migration detail, or persistence format. -Project DB v23 is the current additive optical-context overlay. It preserves -the v22 scientific/persistence foundation and must not infer or backfill optical -identity from historical data. Historical references to older Project DB -versions remain valid when they describe the actual historical contract or -migration path. Do not rewrite legitimate v16–v22 history merely because v23 -is current. +Project DB v25 is the current additive operational schema in the active Feature +batch tranche. It preserves v24 RAW batch, the v23 optical overlay and the v22 +scientific/persistence foundation. Its authorized purpose is limited to typed +durable persistence for `features.extract.batch/1` through +`feature_extract_batch_tasks`; it adds no scientific identity and must not +reinterpret historical rows. Until the v25/Feature-batch tranche has completed +its real-data proof and final review, its lifecycle must remain truthful rather +than being marked `PASS/FROZEN` prematurely. Historical references to older +Project DB versions remain valid where they describe the actual historical +contract or migration path. The global maintenance implementation, fresh portable/Vulkan/sanitizer/ concurrency validation and independent final review are acquired. Its lifecycle is `GLOBAL_MAINTENANCE_AUDIT=PASS/FROZEN`. The review independently passed the -portable build, 64/64 complete suite, 15/15 focused matrix, 76/76 strict header -probes across 19 modified/new public headers, ABI and production-seam checks, -retained-manifest verification and diff validation, with zero blocking findings. -Do not reopen this boundary or infer a new scientific policy from the freeze. +portable build, complete suites, focused matrices, strict public-header probes, +ABI and production-seam checks, retained-manifest verification and diff +validation with zero blocking findings. Do not reopen this boundary or infer a +new scientific policy from the freeze. The canonical review checkpoint is tag `global-maintenance-2026-09-01` at commit `b84f860d868c66d9ee84b85ceb1bc6480b95aca5`; its detailed evidence is @@ -88,11 +94,13 @@ When a ticket declares `NO_NEW_SUBSYSTEM`, do not introduce an unrelated: - daemon. This restriction is ticket-scoped. It does not mean that a subsystem already -present in the canonical architecture can never be used by later authorized -work. +present in the canonical architecture can never be used or extended by later +authorized work. -Never reopen a FROZEN scientific or architectural decision merely to simplify -an implementation. +Never reopen a FROZEN scientific decision merely to simplify an +implementation. Operational/resource contracts may be reopened only by explicit +human authority and must preserve the scientific result exactly unless the +human ticket explicitly says otherwise. ## 3. Scope and Git discipline @@ -157,8 +165,8 @@ require a clean worktree unless the ticket explicitly requires one. allocation, solver execution or output mutation; do not alter the FROZEN defaults, encodings or fingerprints to accommodate an unsafe cast. -Compiler success alone is not proof of API, ABI, persistence, or scientific -contract correctness. +Compiler success alone is not proof of API, ABI, persistence, resource, or +scientific contract correctness. ## 5. Identity discipline @@ -217,13 +225,132 @@ explicitly defines such an identity. - Extend validated abstractions rather than rewriting validated modules. - Reuse the existing Task / Queue / Scheduler / Resource Governor ownership model rather than creating parallel runtime infrastructure. -- System stability and responsiveness take priority over throughput. + +### Canonical utilization objective — MAXIMUM SAFE USEFUL THROUGHPUT + +Lardon3D is a throughput-oriented workstation application. The Resource +Governor MUST maximize safe and useful utilization for every production Task +after preserving only the interactive host reserve needed for normal use of: + +- Arch Linux / Sway and normal desktop services; +- Firefox; +- normal audio/music playback; +- lightweight interactive use of the workstation. + +All CPU, RAM, I/O capacity and validated accelerator capacity beyond that +interactive reserve belongs to the active Lardon3D workload when useful work +exists. + +On the current reference host, the normal observed policy outcome is +approximately: + +```text +16 logical CPUs total +4 logical CPUs reserved for the interactive host +12 logical CPUs available to the compute pool +~3 GiB MemAvailable preserved as the hard RAM reserve +Radeon 780M UMA available to validated/useful GPU backends +``` + +These values are reference-host outcomes, NOT portable constants. Topology, +process affinity, smaller hosts, memory pressure, I/O pressure and future +hardware must be handled dynamically. A future 32-thread host must not inherit a +12-thread product ceiling merely because the reference host exposed 12 compute +threads. + +The optimization target is: + +```text +MAXIMUM SAFE USEFUL THROUGHPUT +``` + +It is NOT: + +```text +minimum resource usage +fixed CPU counts +CPU/GPU utilization percentage for its own sake +preserving historical CPU1/batch1 descriptors without evidence +``` + +The Governor remains the sole production resource authority. Normal users do +not choose CPU count, worker count, batch, inflight depth, GPU backend, scratch +mode, or RAM budget for authoritative execution. + +A production Task may deliberately use less than the currently available +compute pool only when concrete evidence establishes at least one relevant +constraint, such as: + +- true scientific or dependency serialism; +- measured useful-scaling knee; +- memory bound; +- I/O saturation; +- validated GPU execution making additional CPU work useless; +- deterministic publication constraint that cannot be separated safely from + preparation; +- another explicit and measured resource limitation. + +The reason must be represented by the Task/Governor contract or canonical +resource documentation where non-obvious. A historical descriptor by itself is +not evidence. + +### SERIALISM_REQUIRES_PROOF + +`SERIALISM_REQUIRES_PROOF` is canonical operational policy. + +Per-item atomicity does NOT imply cross-item serialization. Owner-only or +ordered durable publication does NOT imply serial preparation. When multiple +independent work units exist and exact science/persistence semantics are +preserved, the Task must expose bounded concurrency to the Governor. + +A long-running `CPU1` or `batch1` path with independent executable work and +available safe resources is an operational defect until one of the documented +measured limitations above proves otherwise. + +Do not solve this by creating another scheduler, uncontrolled worker pool, +unbounded `std::async`, detached threads, or resources outside Governor +accounting. Prefer the established bounded pattern where appropriate: + +```text +one admitted owner Task + -> bounded independent participants/preparation + -> join all participants + -> deterministic owner-only publication +``` + +Every participant must be accounted, bounded, cancellable and joined. Shared +SQLite access must preserve the existing ownership/serialization contracts. + +### CPU, GPU, RAM and I/O policy + +- CPU: expose truthful minimum/useful/safe concurrency and allow Governor to use + the largest currently safe/useful value. Do not hardcode the reference-host + 12-thread result. +- GPU: a validated AND useful production GPU backend MUST be preferred when it + is available and Governor-safe. Do not invent or promote an unvalidated GPU + path merely to make the GPU busy. +- RAM: use available memory aggressively for useful work while preserving the + hard interactive reserve, approximately 3 GiB `MemAvailable` on the reference + host. The 3–4 GiB band is a pressure/conservatism zone, not a permanent extra + 1 GiB subtraction from every Task. +- UMA: iGPU allocations count exactly once against host RAM. +- swap/zram: pressure and safety mechanisms, never admitted working RAM. +- scratch SSD: optional external storage, never RAM. A Task may consume it only + through an explicitly validated Governor-owned scratch contract. +- I/O: if additional concurrency no longer improves useful throughput because + storage or another I/O boundary is saturated, the measured knee is the right + useful limit. Do not force CPU saturation for appearance. +- Pressure: CPU/memory/I/O PSI and active swap-in/out deltas may reduce + admission. When pressure clears, useful resources must be re-admitted rather + than remaining permanently throttled. + +System responsiveness is therefore protected by the explicit host reserve and +pressure feedback; it is not a justification for leaving additional safe/useful +compute idle. + - Bound memory, buffers, files, descriptors, processes, threads, captured - output, parser work, temporary storage, and staging reasonably for the + output, parser work, temporary storage, and staging honestly for the operation. -- Memory shared with an iGPU counts against host RAM. -- zram and swap are pressure/safety mechanisms, not normal working-memory - budgets. - An operational hardware/resource bound must not accidentally become a scientific dataset-size limit. - Do not invent a new global resource subsystem inside a ticket that defers it. @@ -237,12 +364,16 @@ Resource-sensitive work must identify, where relevant: - resource owner; - admission point; -- bound or budget; +- minimum/useful/safe CPU and batch capability; +- fixed/per-participant/transient memory budget; - reservation lifetime; - release point; - failure cleanup; - cancellation cleanup; -- whether the bound is operational or scientific. +- I/O/scratch ownership; +- validated GPU capability; +- whether a bound is operational or scientific; +- measured reason for intentional under-utilization. The reviewed external USB SSD controller is the authorized physical-lifecycle boundary for the exact UDisks Drive/label/UUID contract. Its validated snapshot @@ -251,11 +382,10 @@ production orchestrator for scratch-lease acquire/release. The controller does not replace the Governor or invent Task scratch eligibility, and swap/scratch never become RAM. The application lifetime order is strict: destroy/join the Queue so every Task lease is released, checked-join/unregister the SSD binding, -destroy the controller, then destroy the Governor. The current fourteen Task -kinds have no scratch consumer, so availability is capability, not fabricated -usage. Do not add ad-hoc discovery, mounting, formatting, `swapon`, cleanup, -force-drain, shell commands, or a second resource/scheduling subsystem outside -the reviewed controller/Governor APIs. +destroy the controller, then destroy the Governor. SSD availability is a +capability, not fabricated Task usage. Do not add ad-hoc discovery, mounting, +formatting, `swapon`, cleanup, force-drain, shell commands, or a second +resource/scheduling subsystem outside the reviewed controller/Governor APIs. Snapshot conversion is fail-closed by physical state. Any pairing or authority requires current detection of the Drive and both UUID-bearing partitions, @@ -426,7 +556,8 @@ comment. Where relevant, comments must preserve the established distinction between: - immutable SOURCE RAW; -- deterministic DERIVED representation; +- camera JPEG SOURCE; +- deterministic DERIVED RAW representation; - RAW Policy v1; - L3DRAWD1 identity; - metadata-only acquisition processing; @@ -438,6 +569,12 @@ Where relevant, comments must preserve the established distinction between: - bounded secondary JPEG validation; - zero-only permitted MPF gaps/trailer. +For the current A6000 selected scientific execution, paired camera JPEGs are +valid Capture source assets and preferred fast proxies for Photo Quality, but +the FROZEN geometry representation remains deterministic RAW-derived PNG. Do +not substitute camera-JPEG geometry without a separately authorized and proven +scientific/calibration equivalence tranche. + Do not imply that metadata validation performs pixel decoding when it does not. ### Task / Queue / Scheduler / Governor @@ -454,6 +591,8 @@ In particular: - Resource Reservation belongs to the currently admitted bounded execution. - `sequence_break` is an execution boundary that releases/re-establishes admission according to runtime semantics. +- Per-item atomicity does not imply cross-item serialization. +- Serial publication may coexist with Governor-admitted parallel preparation. Do not imply that a long campaign reserves resources for its entire lifetime when execution is group-bounded. @@ -466,16 +605,18 @@ resource ownership understandable from source. Where applicable document: - RAM ownership/bound; -- CPU admission; +- CPU minimum/useful/safe admission; - GPU admission; - I/O ownership; - process/thread count; +- per-participant memory; - file-descriptor lifetime; - temporary/scratch storage ownership; - reservation lifetime; - release point; - failure cleanup; -- cancellation cleanup. +- cancellation cleanup; +- measured scaling knee or reason for serial execution. An operational hardware/resource bound MUST NOT accidentally become a scientific dataset-size limit. @@ -502,6 +643,12 @@ The residual pre-return S3-E crash window remains intentional and documented: if S3-E creates a Capture internally and the process dies before S3-E returns the `capture_id`, campaign execution must not guess that Capture identity. +For `raw.develop.batch/1`, typed `task_id -> selected_execution_id` persistence +must remain distinct from generic Task runtime state. Independent RAW +participants may prepare concurrently, but owner publication and selected-item +cursor advancement remain deterministic and ordered by the selected execution +contract. + ### C / C++ ABI boundaries Where C++ implements a public C interface or C Task callback, comments should @@ -524,7 +671,8 @@ Comments should state WHAT CONTRACT the fixture proves, for example: - ambiguity; - scientific identity conflict; - cross-ScanSet rejection; -- resource-pressure/admission condition. +- resource-pressure/admission condition; +- deliberate CPU1 baseline used only for a scaling comparison. Do not narrate ordinary test mechanics. @@ -552,9 +700,9 @@ For every future implementation tranche: 4. update canonical documentation when behavior/contracts changed; 5. validate every modified public C17 header; 6. validate relevant C++ syntax/build integration; -7. validate resource ownership and bounds; -8. run applicable build/tests/sanitizers; -9. perform the required review; +7. validate resource ownership, accounting, useful scaling and bounds; +8. run applicable build/tests/sanitizers efficiently; +9. perform the required bounded review; 10. only then claim completion. The normalized future implementation standard is: @@ -566,12 +714,14 @@ CODE + CANONICAL DOCUMENTATION + C17/C++ SYNTAX + RESOURCE OWNERSHIP ++ MAXIMUM SAFE USEFUL THROUGHPUT + VALIDATION + REVIEW ``` A tranche is NOT complete merely because code compiles and tests pass when -required contract/invariant comments or canonical documentation are missing. +required contract/invariant comments, canonical documentation, or required +resource behavior are missing. There should be no future project-wide comment-cleanup pass for newly written code: comment debt must be handled when the code is introduced. @@ -613,9 +763,12 @@ Persistence changes require explicit attention to: For Project DB: -- preserve the FROZEN v22 semantics and the current additive v23 optical - overlay unless a ticket explicitly authorizes a later schema change; -- schema-version changes require explicit human authorization; +- preserve the FROZEN v22 scientific/persistence semantics and v23 optical + overlay; +- v24 is the explicitly authorized additive operational RAW-batch migration and + must not reinterpret scientific history; +- future schema-version changes beyond the currently authorized v24 require + explicit human authorization; - migrations must be additive unless a different migration is explicitly authorized; - existing projects must remain recoverable; @@ -635,41 +788,86 @@ make retries convenient. - For changes to API, architecture, ownership, concurrency, persistence, pipeline, resources, limits, or scientific semantics, check whether canonical documentation must be updated. +- `MAXIMUM SAFE USEFUL THROUGHPUT` and `SERIALISM_REQUIRES_PROOF` are canonical + human resource-policy decisions. Any resource document that contradicts them + is stale unless it describes an explicitly historical measurement. - Documentation must describe proven implementation, not desired future - behavior. + behavior. Human policy may be marked as policy while implementation work is + still `VALIDATION PENDING`. - Roadmap documents may describe future behavior, but future capabilities must be clearly marked as planned/later/exploratory. - Never describe future viewer, Capture Guidance, video/keyframe, Task scratch consumption, or camera-control capabilities as implemented before they are - actually validated. The current TUI/F10 and controller-to-Governor registry - are validated operationally, but no current Task kind consumes scratch and - those interfaces do not make dense/scratch-consuming workflows complete. + actually validated. - Statuses such as `PLANNED`, `IMPLEMENTED`, `VALIDATION PENDING`, and `PASS/FROZEN` are authoritative lifecycle statements. - Update lifecycle state only when implementation, validation, and review evidence support the transition. - Never mark work `PASS/FROZEN` without the required validation and review. -- Preserve legitimate historical DB/version references where they describe - frozen history. +- Preserve legitimate historical DB/version/resource references where they + describe frozen history; label superseded operational envelopes as historical + rather than silently rewriting the evidence. -## 12. Required validation +## 12. Required validation and test resource policy For ordinary implementation tickets, before claiming completion, actually run -the applicable commands. +the applicable commands. Builds, tests, benchmarks and real-proof workloads +MUST follow the same resource philosophy as production: preserve the +interactive host reserve, then use the maximum safe/useful remaining resources. +Artificially serial engineering work wastes both elapsed time and agent budget. -Normal build: +### Normal build + +Do not use a fixed historical `-j8` as a product rule. Derive a safe parallel +job count from the current host/affinity. On the current 16-logical-CPU +reference host, where four logical CPUs are reserved for interactive use, the +normal build target is approximately: ```sh meson setup --reconfigure build -meson compile -C build -j8 +meson compile -C build -j12 ``` -Normal tests: +On another host, choose the analogous available-compute value rather than +hardcoding 12. + +### Normal tests + +Independent tests should run concurrently within the same host reserve and +memory constraints. On the current reference host, a typical command is: ```sh -meson test -C build --num-processes 1 --print-errorlogs +meson test -C build --num-processes 12 --print-errorlogs ``` +This is not permission to race tests that share mutable fixtures, real projects, +fixed ports, exclusive GPU state, or other process-global resources. Such tests +must be grouped or serialized for the concrete dependency, and that reason must +be understood rather than inherited from an old `--num-processes 1` default. + +A CPU1 run is valid when CPU1 is the actual test case or baseline measurement; +it is not the authoritative production configuration after higher safe/useful +concurrency has been proven. + +### Validation efficiency + +- Do not rebuild unchanged targets between proof iterations without a concrete + need. +- Do not rerun already-acquired expensive validation when the relevant code and + dependency boundary did not change. +- Run focused tests first; widen only when the changed dependency boundary + requires it. +- Expensive/stress validation must be relevant to the current delta. +- Heavy validation is parallel by default when independent and resource-safe; + serialize only when scientific determinism, mutable shared fixtures, memory, + I/O, GPU exclusivity, sanitizer behavior or measured host pressure requires + it. +- A test timeout or sanitizer failure must be investigated; do not repeatedly + rerun until green. +- Real-data proofs must use normal Governor admission. Do not manually force + CPU1 merely for reproducibility unless CPU1 itself is the intended comparison + cohort. + Diff validation: ```sh @@ -692,24 +890,48 @@ For relevant concurrency changes, run TSan when supported and meaningful. If a sanitizer is unavailable or invalid because of the environment/toolchain, report that explicitly rather than claiming PASS. -Run expensive/stress validation only when relevant. - -Run heavy validation serially when required to preserve machine stability. - -Investigate a timeout or sanitizer failure. Do not repeatedly rerun a failing -test until it happens to pass. - Distinguish third-party sanitizer/environment noise from repository defects using concrete stack/failure evidence. -Never claim a command, test, sanitizer, review, or real-data validation that -was not actually performed. +Never claim a command, test, sanitizer, review, benchmark, or real-data +validation that was not actually performed. For documentation/comment-only changes, do not invent unnecessary sanitizer work, but still run enough build/syntax/diff validation to prove that the non-functional boundary was preserved. -## 13. Review discipline +## 13. Engineering execution and credit economy + +Engineering time and agent budget are finite project resources and must not be +wasted. + +- Default to the configured economical parent/orchestrator and existing agent + roles. Do not invent a new agent hierarchy. +- Use expensive/high-reasoning scientific or final-review roles only when the + current delta genuinely requires scientific equivalence, difficult + concurrency/persistence reasoning, or final sensitive review. +- Do not spawn multiple agents to restate the same architecture or repeat the + same review. +- Do not delegate ordinary implementation defects upward when a worker, + mechanic or resolver can fix them directly within authority. +- Prefer one focused independent review after the implementation is stable to + repeated speculative reviews during ordinary coding. +- Reuse acquired maintenance, sanitizer, benchmark and real-data evidence when + the affected boundary is unchanged. +- Do not wait on an expensive agent when the next authorized executable action + can proceed independently. +- Keep tool/log output bounded; inspect targeted portions instead of repeatedly + dumping whole logs. +- Do not rerun a global A-to-Z audit after the maintenance checkpoint. +- `WORK FIRST. RETURN LAST.` Ordinary in-scope findings are repaired and + validated before returning to the human. + +Low remaining agent/credit budget is a reason to eliminate redundant work, not +a reason to weaken required correctness evidence. If a genuinely required +validation cannot be afforded/executed, report it honestly rather than claiming +PASS. + +## 14. Review discipline A normal review should verify the bounded current delta and direct regressions. @@ -727,7 +949,7 @@ A reviewer request does not automatically define new policy. Compare findings against: 1. canonical FROZEN documentation; -2. explicit human decisions; +2. explicit human decisions, including the resource-utilization policy; 3. established tranche semantics; 4. documented future-scope boundaries. @@ -735,9 +957,13 @@ Do not invoke an expensive implementation agent merely to satisfy speculative hardening that is outside the current contract. Comments and documentation are reviewable implementation artifacts. A -misleading contract comment is a defect even if compiled behavior is unchanged. +misleading contract or resource-policy comment is a defect even if compiled +behavior is unchanged. -## 14. Delivery report / STOP conditions +Resource review must specifically flag accidental serialism when independent +work exists and safe/useful resources are idle without measured justification. + +## 15. Delivery report / STOP conditions Every completed ticket report must include: @@ -750,15 +976,17 @@ Every completed ticket report must include: - known blockers; - non-blocking findings; - deliberately deferred/future-scope items; -- resource impact where relevant; +- resource impact where relevant, including admitted/useful concurrency for + modified resource-sensitive Tasks; - confirmation that unrelated and FROZEN areas were preserved; - Git state; - confirmation that `scan3d/` remained untouched when protected. STOP and request a human decision only when resolution requires: -- changing a FROZEN contract; -- changing Project DB schema/version without prior authorization; +- changing a FROZEN scientific contract; +- changing Project DB schema/version beyond already authorized v24 without + prior authorization; - introducing a genuinely new subsystem outside authorized scope; - files outside the authorized scope; - destructive Git action; @@ -775,9 +1003,12 @@ Do NOT stop merely because: - a normal test fails; - an ordinary implementation bug exists; - a reviewer identifies a bounded repairable defect; +- a resource descriptor is historically conservative and explicit human + authority already allows operational correction while preserving science; - a dependency needs factual investigation; - documentation status is stale; - an implementation detail can be resolved safely from existing code and contracts. -Advance the project while preserving the contracts. +Advance the project while preserving scientific contracts and maximizing safe, +useful throughput. diff --git a/README.md b/README.md index 05f55b2..88e2cb3 100644 --- a/README.md +++ b/README.md @@ -276,7 +276,7 @@ calibration connues ; le Sparse SfM réel reste Le Resource Governor ne constitue pas un Resource System générique : il reste l'unique propriétaire des budgets et le seul orchestrateur des leases scratch de production. Le contrôleur SSD UDisks2 est une frontière physique séparée, -jamais un second scheduler ou Governor. Les quatorze Task kinds actuels ne +jamais un second scheduler ou Governor. Les seize Task kinds actuels ne consomment encore aucun scratch : l'espace disponible est une capacité observable, pas un usage fabriqué, et scratch/swap ne deviennent jamais de la RAM. La TUI/F10 et cette intégration sont validées opérationnellement. L'audit diff --git a/docs/architecture/candidate_pair.md b/docs/architecture/candidate_pair.md index f1278a6..5a46853 100644 --- a/docs/architecture/candidate_pair.md +++ b/docs/architecture/candidate_pair.md @@ -253,6 +253,9 @@ La tâche utilise le runtime générique via le pattern standard : La Queue conserve un callback ; jusqu'à soixante-quatre participants CPU admis peuvent calculer une fenêtre interne bornée sans modifier l'identité scientifique. +- CPU et lot sont essayés ensemble (`1/1`, `2/2`, puis les paliers sûrs) : + un CPU supplémentaire avec un lot d'un seul membership ne peut exercer aucun + participant supplémentaire et ne constitue donc pas une mesure de scaling. - Réservation CPU + IO avant exécution - `lardon3d_task_sequence_break()` entre chaque lot pour réadmission Governor - Callback terminal checkpoint après `COMPLETED`/`FAILED`/`CANCELLED` @@ -276,7 +279,7 @@ bool lardon3d_candidate_pair_generate_reconstruct( ### Garantie actuelle -La Queue conserve un callback actif. À l'intérieur de la Task, jusqu'à douze +La Queue conserve un callback actif. À l'intérieur de la Task, jusqu'à soixante-quatre threads CPU admis calculent en parallèle une fenêtre d'au plus deux sources par thread. Chaque participant possède un handle DB de lecture privé. Le thread propriétaire publie ensuite seul et dans l'ordre canonique des sources. Voir le diff --git a/docs/architecture/internal_parallelism.md b/docs/architecture/internal_parallelism.md index d719540..cab7ec6 100644 --- a/docs/architecture/internal_parallelism.md +++ b/docs/architecture/internal_parallelism.md @@ -198,6 +198,50 @@ au plus `cpu_threads - 1` threads enfants. Tous les enfants sont joints avant la fin de la séquence, la libération de réservation ou `lardon3d_task_sequence_break()`. +## Développement RAW d'une exécution sélectionnée + +`raw.develop.batch/1` conserve la politique RAW v1, les octets PNG, +L3DRAWD1, le fingerprint, le content-addressing et toutes les identités du +développeur par Capture. Une séquence prend au plus huit items contigus depuis +le curseur durable de l'exécution sélectionnée et utilise au plus le nombre de +CPUs admis. Chaque participant possède son LibRaw et ses buffers privés ; les +896 Mio chargés par item admis couvrent le workspace LibRaw borné à 40 MP, les +copies RGB/BGR, les buffers PNG/validation, la pile enfant bornée de 1 Mio et la +marge allocateur/codec. Le petit contexte propriétaire est inclus dans cette +charge : un budget hôte post-réserve de 7 Gio admet donc exactement les huit +participants sûrs, sans débit fixe redondant. Cette estimation est une borne +opérationnelle de mémoire par participant, pas une limite scientifique de +dataset. +OpenCV est fixé à un thread interne afin de ne pas multiplier le fan-out. + +Les enfants ne modifient ni curseur sélectionné ni progression Task. Après leur +jointure complète, le callback Queue propriétaire publie seul les mappings +`item_index -> image_id` dans l'ordre croissant. Chaque transaction de mapping +avance le curseur contigu, puis seulement le checkpoint Task peut avancer. Une +erreur, une annulation ou un échec de création d'enfant joint tout le travail +lancé ; des assets immuables content-addressés peuvent déjà exister, mais le +curseur ne les devine pas et la reprise converge par le développeur existant. +Une `sequence_break` sépare toutes les fenêtres, donc la sélection entière ne +retient jamais une réservation longue. + +## Extraction Feature d'une exécution sélectionnée + +`features.extract.batch/1` conserve exactement ORB v1, ses paramètres, son +fingerprint, ses keypoints/descripteurs U8×32 et le Feature File existant. Une +fenêtre contient au plus douze images indépendantes de l'ordre sélectionné. Le +propriétaire configure OpenCV à un thread pour éviter tout fan-out imbriqué ; +chaque participant possède son décodage et sa sortie bornés, sans accès SQLite, +puis tous sont joints. + +Le propriétaire seul publie dans l'ordre et avance le curseur v25 après preuve +du Feature Set READY exact. CPU et lot partent du minimum. Comme un CPU +supplémentaire ne peut être exercé que par une image indépendante de la même +fenêtre, le Governor essaie et accepte ces deux limites ensemble (`1/1`, `2/2`, +`4/4`, puis le maximum sûr admis). Les observations de fenêtres durables +permettent de retenir le dernier palier apportant au moins 5 %. Le plafond douze +est une capacité opérationnelle de benchmark, +jamais une limite scientifique, et aucune voie GPU n'est introduite. + ## Calcul et publication Visual Index Une séquence sélectionne le même préfixe durable d'au plus seize Feature Sets diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index a1a5de6..d95b6ba 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -138,7 +138,7 @@ Frontière physique UDisks2 pour la paire de labels `LARDON_SWAP` et et drain sûr. Il ne formate, ne répare ni ne force jamais l'hôte et ne remplace pas le Resource Governor. La TUI exécute ses actions dans un seul thread joinable et le Governor enregistre l'état physique puis orchestre seul les -leases scratch de production. Les quatorze Task kinds courants n'en consomment +leases scratch de production. Les seize Task kinds courants n'en consomment aucun ; capacité visible ne signifie donc pas usage. **Statut :** CURRENT / VALIDATED OPERATIONAL diff --git a/docs/architecture/project_database.md b/docs/architecture/project_database.md index 96a2a29..cfd4582 100644 --- a/docs/architecture/project_database.md +++ b/docs/architecture/project_database.md @@ -38,10 +38,46 @@ malformée retourne `CORRUPT` avant comparaison avec une demande alternative. La migration v22→v23 n'inspecte ni EXIF, chemin, basename, SHA-256, dimensions, nom d'appareil, ni calibration historique. Elle ne réinterprète donc aucune -Capture, image ou calibration v16–v22. Le marqueur `schema_version=23` est le +Capture, image ou calibration v16–v22. Le marqueur `schema_version=23` reste le point de publication durable ; un échec de création ou un update de marqueur qui ne cible pas exactement la v22 rollbacke toute la migration. +### Overlay v24 — développement RAW sélectionné borné + +Le schéma courant v24 ajoute uniquement +`raw_development_batch_tasks(task_id, selected_execution_id)`. `task_id` +référence l'état générique Task et `selected_execution_id` est unique ; la +ligne ne copie aucun Capture, asset, image, chemin, groupe ni curseur. La +migration v23→v24 est transactionnelle et DDL-only : elle ne crée aucune +association historique et son marqueur v24 est son point de publication. + +`raw.develop.batch/1` recharge le curseur de `selected_executions` à chaque +fenêtre admise. Après jointure de tous les participants, seul le callback Task +propriétaire appelle `record_selected_representation` en ordre croissant. Le +commit item+curseur précède toujours la progression/checkpoint générique ; un +crash peut donc laisser celle-ci en retard, jamais en avance. Une reprise repart +du curseur sélectionné durable sans deviner l'identité d'une sortie RAW déjà +publiée. Le schéma et le chemin historique `raw.develop/1` restent valides. + +### Overlay v25 — extraction Feature sélectionnée bornée + +**IMPLEMENTED / VALIDATION IN PROGRESS.** La migration transactionnelle +v24→v25 ajoute uniquement `feature_extract_batch_tasks`. Sa ligne lie un Task +`features.extract.batch/1` à une exécution sélectionnée immutable, retient le +préfixe monotone `next_item_index` et copie le domaine ORB exact +kind/version/paramètres/fingerprint. La migration est DDL-only : elle ne crée +aucune ligne, ne convertit aucune Task `features.extract/1` historique et +n'infère aucun image ID ou Feature Set. + +Les participants CPU préparent des images sélectionnées indépendantes sans +accès SQLite. Après leur jointure, le propriétaire publie les Feature Sets dans +l'ordre sélectionné, avance un item seulement si le Feature Set READY exact est +durable, puis checkpoint la progression générique. Un crash peut donc laisser +le checkpoint générique ou le curseur Feature en retard sur un Feature Set +immutable déjà publié ; la reprise revalide ce résultat exact et converge sans +deviner une identité. Les formats, keypoints, descripteurs ORB U8×32, +fingerprint et Feature Store FROZEN restent inchangés. + La validation a migré via l'API production des copies des projets réels S21 et A6000 : version 23, `integrity_check` et clés étrangères propres, comptes et lignes scientifiques inchangés, neuf tables optiques vides et SHA-256 des diff --git a/docs/architecture/resource_boundary.md b/docs/architecture/resource_boundary.md index 3605f96..c0f9729 100644 --- a/docs/architecture/resource_boundary.md +++ b/docs/architecture/resource_boundary.md @@ -5,12 +5,13 @@ **ACCEPTED** — architecture decision for the post-Gate C documentation freeze. Current Sparse SfM gates A through G are **PASS / FROZEN**. -Project Database: current schema **v23**; the v22 foundation and historical +Project Database: current schema **v24**; the v22 foundation, v23 optical overlay, and historical v16 science remain frozen. This record is normative for the separation of responsibilities. Its original Gate C/G decision introduced no implementation or syntax. Later explicitly -authorized additions, including the additive Project DB v23 optics overlay and +authorized additions, including the additive Project DB v23 optics overlay, +the v24 RAW batch Task association, and the bounded UDisks2 SSD controller, must remain on their own side of this boundary and do not create a generic Resource System. diff --git a/docs/architecture/resource_governor.md b/docs/architecture/resource_governor.md index 1894315..5ca937c 100644 --- a/docs/architecture/resource_governor.md +++ b/docs/architecture/resource_governor.md @@ -268,7 +268,7 @@ et Visual Index restent CPU; SIFT/RootSIFT Matcher restent BFMatcher L2 CPU. Il n'est introduit ni second scheduler, ni Queue, ni daemon, ni sous-système de ressources. -### Audit des 14 kinds de production +### Audit des 16 kinds de production Tous les kinds passent par l'unique Queue et l'unique Governor, y compris ceux dont toutes les dimensions sont fixes. Dans le tableau, `CPU 1..N` décrit la @@ -283,14 +283,16 @@ allocations driver opaques ne reçoivent pas un coût inventé. | Kind v1 | Estimation courante | Dimension consommée et raison | | --- | --- | --- | | `raw.develop` | MIXED; CPU 1; lot 1; hôte `2 Gio + contexte`, 0/item; I/O 1; GPU 0 | Un Capture atomique. Le garde applique/restaure CPU1 au pool OpenCV global. Les 2 Gio sont une allowance de travail opérationnelle, pas une limite de dataset. | +| `raw.develop.batch` | MIXED; CPU 1..8; lot 1..8; hôte 0 fixe + 896 Mio/item (contexte propriétaire, workspace LibRaw 40 MP, copies RGB/BGR, PNG/validation, pile enfant 1 Mio et marge allocateur/codec); I/O 1; GPU 0 | Une fenêtre de Captures indépendants. `896 Mio × 8 = 7 Gio`, donc le budget post-réserve de l'hôte de référence peut admettre la fenêtre sûre complète. OpenCV reste à 1 thread interne, tous les enfants sont joints, puis le propriétaire publie le préfixe ordonné avant `sequence_break`. Le plafond est opérationnel, jamais une limite scientifique de sélection. | | `photo_quality.triage` | IMPORT; CPU 1; lot 1; hôte `contexte retenu + 20 Mio`, 0/item; I/O 1; GPU 0 | Un groupe par séquence ; garde OpenCV CPU1. Aucun scaling utile déterministe n'est acquis. | | `acquisition_campaign.run` | JPEG: IMPORT ; RAW: MIXED. CPU 1; lot 1; hôte `contexte retenu + requête transitoire exacte + 256 Kio` + 64 Kio/item ; DEVELOP_RAW ajoute 2 Gio; I/O 1; GPU 0 | Un groupe S3-E par séquence, sans Task imbriqué. La création et la reprise dérivent la même estimation ; la forme historique exacte est normalisée seulement en mémoire. | | `import.images` | IMPORT; CPU 1; lot 1..32; hôte 128 Kio + `NAME_MAX+64`/item; I/O 1; GPU 0 | Copie/hash I/O-bound. Le callback consomme le lot admis et réadmet entre lots. | | `features.extract` | CPU; CPU 1..compute-pool; lot 1; hôte 64 Mio + 512 Mio/item; I/O 1; GPU 0 | La demande durable emploie le maximum `int` positif de l'API OpenCV ; le Governor la borne à l'hôte. Le garde applique/restaure exactement le CPU admis. | +| `features.extract.batch` | CPU; CPU 1..12; lot 1..12; hôte 64 Mio + 512 Mio/item; I/O 1; GPU 0 | Images sélectionnées indépendantes, OpenCV CPU1 par participant, enfants joints et publication owner-only ordonnée. Le plafond 12 est une capacité opérationnelle de mesure ; le feedback ≥5 % établit le palier utile sans devenir une limite scientifique. | | `features.extract.sift` | CPU; CPU 1..compute-pool; lot 1; hôte 64 Mio + 1 Gio/item; I/O 1; GPU 0 | Même contrat OpenCV. Les formes CPU12 et CPU1 historiques complètes sont acceptées et normalisées en mémoire. | | `features.extract.rootsift` | CPU; CPU 1..compute-pool; lot 1; hôte 64 Mio + 1 Gio/item; I/O 1; GPU 0 | Même contrat que SIFT ; aucune couture GPU scientifiquement compatible n'est validée. | | `visual_index.update` | CPU; CPU 1..16; lot 1..16; hôte 8 Mio + 2 Mio/item; I/O 1; GPU 0 | Le segment contient au plus 16 Feature Sets indépendants. Au plus `cpu_threads-1` enfants sont joints avant publication owner-only. | -| `candidate_pair.generate` | CPU; CPU 1..64; lot 1..64; hôte 256 Kio + 8 Mio/item; I/O 1; GPU 0 | Le batch de 64 est la borne algorithmique et de ressources. Fenêtre `min(2*CPU, 64, reste_du_lot)`, un handle DB privé par participant, piles enfants de 4 Mio facturées. | +| `candidate_pair.generate` | CPU; CPU 1..64; lot 1..64; hôte 256 Kio + 8 Mio/item; I/O 1; GPU 0 | Le batch de 64 est la borne algorithmique et de ressources. CPU/lot sont couplés pour que chaque palier de feedback exerce ses participants; fenêtre `min(2*CPU, 64, reste_du_lot)`, un handle DB privé par participant, piles enfants de 4 Mio facturées. | | `matcher.run` | CPU: CPU 1..12, lot sûr 1..12, hôte 0 + 10 Mio/item, I/O 1, GPU 0. ORB Vulkan AUTO: CPU 1, lot utile 1..8, même hôte/item, GPU 1 + 640 Kio, inflight 1. | Le batch/participant 12 est une borne intrinsèque mesurée du Matcher. Batch 12 et depth 2 (1,25 Mio) restent sûrs pour preuves privées, mais insuffisamment utiles en AUTO. CPU complet est le fallback. | | `geometric_verifier.run` | CPU; CPU utile 1..8, fenêtre sûre 16; lot 1..16; hôte 0 + 8 Mio/item; I/O 1; GPU 0 | Des parents indépendants sont préparés en parallèle, puis publiés/checkpointés en ordre par le propriétaire. L'USAC scientifique conserve `isParallel=false`. | | `track_builder.run` | CPU; CPU 1; lot 1; fixe `(4 Mio + arêtes * (48 + 2*160)) * facteur`, facteur 2 jusqu'à 400k arêtes puis 8; 0/item; I/O 1; GPU 0 | Rebuild DSU atomique et publication owner-only ; aucune partition scientifiquement validée n'est acquise. | @@ -422,7 +424,7 @@ retirer l'autorité mais ne peut jamais recopier un ancien compte physique. Le teardown production suit strictement : Queue détruite/jointe et chaque lease Task rendu, worker SSD joint puis unregister vérifié, contrôleur détruit, Governor détruit. Un unregister est refusé tant qu'une opération wrapper ou un -lease exact subsiste. Les quatorze Task kinds courants n'ont aucun consommateur +lease exact subsiste. Les seize Task kinds courants n'ont aucun consommateur scratch : le compte normal est donc réellement zéro et la capacité disponible n'autorise aucun usage implicite. Une future Task consommatrice devra définir son propre contrat d'éligibilité et son ownership sans transformer scratch ou diff --git a/docs/architecture/task_kind_registry.md b/docs/architecture/task_kind_registry.md index b181864..8e1af9f 100644 --- a/docs/architecture/task_kind_registry.md +++ b/docs/architecture/task_kind_registry.md @@ -44,18 +44,20 @@ métier, et toute forme voisine est rejetée. ## Inventaire production et entrées runtime `src/task_kinds.c::lardon3d_task_kind_registry_production()` enregistre les -14 kinds v1 du profil de production courant. La colonne « reprise » nomme le +16 kinds v1 du profil de production courant. La colonne « reprise » nomme le reconstructeur du binding ; « callback » nomme l'entrée runtime privée dans le même fichier. Le détail chiffré des capacités est centralisé dans l'[audit des -14 kinds](resource_governor.md#audit-des-14-kinds-de-production). +16 kinds](resource_governor.md#audit-des-15-kinds-de-production). | Kind v1 | Source, reprise et callback | Réconciliation pré-admission courante | | --- | --- | --- | | `raw.develop` | `raw_development_task.cpp`; `lardon3d_raw_development_task_reconstruct`; `run` | Aucune | +| `raw.develop.batch` | `raw_development_task.cpp`; `lardon3d_raw_development_batch_task_reconstruct`; `run_batch` | Association v24 Task → exécution sélectionnée ; le curseur reste dans l'exécution | | `photo_quality.triage` | `photo_quality_task.cpp`; `lardon3d_photo_quality_task_reconstruct`; `run` | Aucune | | `acquisition_campaign.run` | `acquisition_campaign_task.cpp`; `lardon3d_acquisition_campaign_task_reconstruct`; `run` | Forme courante ou forme v22 exacte vérifiée contre la requête immuable → capacité courante en mémoire | | `import.images` | `import_task.c`; `lardon3d_image_import_reconstruct`; `run_image_import` | Aucune | | `features.extract` | `feature_task.c`; `lardon3d_feature_extract_reconstruct`; `run` | Formes CPU12/CPU1 historiques exactes → demande OpenCV portable ; runtime borné au compute-pool | +| `features.extract.batch` | `feature_task.c`; `lardon3d_feature_extract_batch_reconstruct`; `run_batch` | Association v25 à l'exécution sélectionnée, domaine ORB exact et curseur typé ; CPU/lot bornés 1..12 | | `features.extract.sift` | `sift_task.c`; `lardon3d_sift_extract_reconstruct`; `run` | CPU12/CPU1 historiques exacts → demande OpenCV portable ; runtime borné au compute-pool | | `features.extract.rootsift` | `sift_task.c`; `lardon3d_sift_extract_reconstruct`; `run` | Même réconciliation SIFT ; aucune voie GPU validée | | `visual_index.update` | `visual_index_task.c`; `lardon3d_visual_index_update_reconstruct`; `run` | Formes CPU12/CPU1 historiques exactes → CPU/lot 1..16 | diff --git a/docs/roadmap/roadmap.md b/docs/roadmap/roadmap.md index 40eadc3..a462647 100644 --- a/docs/roadmap/roadmap.md +++ b/docs/roadmap/roadmap.md @@ -12,6 +12,104 @@ doivent rester incrémentaux, bornés, reprenables, durables et admis par l'uniq Resource Governor : aucune étape ne peut supposer qu'une campagne entière tient en RAM ou termine dans une seule vie de processus. +## POLITIQUE CANONIQUE D'UTILISATION DES RESSOURCES — HUMAN AUTHORITY + +La politique d'exploitation est désormais explicite : **Lardon3D doit viser le +MAXIMUM SAFE USEFUL THROUGHPUT** sur chaque Task de production. + +Le Resource Governor réserve d'abord uniquement les ressources nécessaires pour +que le poste reste normalement utilisable pendant les calculs : + +- Arch Linux / Sway et les services normaux du bureau ; +- Firefox ; +- lecture audio/musique normale ; +- usage interactif léger. + +Tout CPU, RAM, capacité I/O et accélérateur validé restant appartient au travail +Lardon3D lorsqu'un travail utile existe. La stabilité du poste est protégée par +cette réserve interactive et par les signaux de pression ; elle ne justifie plus +de laisser des ressources sûres et utiles inactives par conservatisme historique. + +Sur l'hôte de référence actuel, le résultat normal de cette politique est +approximativement : + +```text +16 CPU logiques au total +4 CPU logiques réservés à l'hôte interactif +12 CPU logiques disponibles au compute-pool +~3 GiB de MemAvailable conservés comme réserve RAM dure +Radeon 780M UMA disponible pour les backends GPU validés et utiles +``` + +Ces nombres sont des **résultats de l'hôte courant**, jamais des constantes +produit. Un futur hôte 32 threads ne doit pas hériter d'un plafond 12 ; le +Governor doit dériver sa réserve et donner le reste au calcul selon topologie, +affinité, pression et enveloppe réelle de la Task. + +La règle canonique est : + +```text +SERIALISM_REQUIRES_PROOF +``` + +Une opération atomique par item n'impose pas de sérialiser les items +indépendants. Une publication owner-only ou ordonnée n'impose pas de sérialiser +la préparation. Lorsqu'il existe plusieurs unités indépendantes et que la +science, la persistance et la reprise restent exactes, la Task doit exposer au +Governor un parallélisme borné. + +Un chemin long `CPU1` ou `batch1` avec travail indépendant disponible et +ressources sûres libres est désormais considéré comme un **défaut opérationnel** +tant qu'une preuve concrète ne démontre pas l'une des limites suivantes : + +- dépendance scientifique ou algorithmique réellement sérielle ; +- genou de scaling mesuré ; +- limite mémoire ; +- saturation I/O ; +- backend GPU validé rendant des CPU supplémentaires inutiles ; +- contrainte de publication qui ne peut pas être séparée de la préparation sans + casser le déterminisme ; +- autre limitation matérielle mesurée et documentée. + +Le Governor reste l'unique autorité de ressources. L'utilisateur normal ne +choisit ni CPU, ni workers, ni lot, ni inflight, ni GPU, ni scratch, ni budget +RAM. Une Task décrit ses minimum/useful/safe, ses coûts fixes/transitoires/par +participant et ses capacités GPU/I/O ; le Governor choisit le maximum +sûr **et utile** disponible à l'instant. + +La politique GPU est symétrique : un backend de production **VALIDATED AND +USEFUL** doit être préféré lorsqu'il est disponible et sûr ; aucun backend GPU +non validé ne doit être inventé pour afficher de l'activité GPU. Candidate, +Visual Index ou GV peuvent donc rester CPU lorsque leur décision mesurée le +justifie, tandis qu'ORB Matcher Vulkan reste GPU-first lorsqu'il est éligible. + +La RAM doit être utilisée agressivement jusqu'à la réserve interactive ; +`swap`, zram et scratch ne sont jamais de la RAM admise, et l'UMA est comptée +exactement une fois contre la RAM hôte. Les signaux PSI et les deltas actifs de +swap peuvent réduire l'admission ; après disparition de la pression, le +Governor doit réadmettre les ressources utiles au lieu de rester durablement +bridé. + +Cette politique s'applique aussi au travail d'ingénierie Codex : builds, tests, +benchmarks et preuves réelles indépendantes doivent utiliser le parallélisme +sûr disponible plutôt qu'un `-j8`/`--num-processes 1` historique. Un run CPU1 +reste légitime comme cohorte de mesure ou lorsqu'une dépendance concrète +l'impose, mais jamais comme défaut universel. Les rebuilds, tests lourds, +revues ou agents coûteux déjà acquis et non affectés par le delta ne doivent pas +être relancés sans raison. La règle reste : **WORK FIRST, RETURN LAST**, avec +revue delta-based depuis le checkpoint de maintenance. + +Statut de cette décision : + +```text +RESOURCE_UTILIZATION_POLICY=HUMAN_AUTHORITY +SERIALISM_REQUIRES_PROOF=CANONICAL +IMPLEMENTATION_AUDIT=REQUIRED_BEFORE_NEXT_LONG_REAL_RUN +``` + +Cette autorité rouvre uniquement les **contrats opérationnels de ressources** +quand ils empêchent cette politique. Elle ne rouvre aucune science FROZEN. + ### Fondations PASS / FROZEN - Gates A–G Sparse SfM, dont F0, orchestration Task/Project DB et politique @@ -26,9 +124,11 @@ en RAM ou termine dans une seule vie de processus. - Task Runtime, checkpoints atomiques, Queue et Resource Governor : [Task](../architecture/task_system.md), [Queue](../architecture/task_queue.md) et [Governor](../architecture/resource_governor.md). -- Project DB v22 gelé, overlay optique additif v23 courant, et S1–S3 Capture / Acquisition Ingestion : provenance - Capture/Asset, import capture-safe, publication dérivée, développement RAW, - siblings multi-source, évidence S3-D, orchestration S3-E et campagne bornée : +- Project DB v22 gelé, overlay optique additif v23 gelé et couche opérationnelle + additive v24 actuellement en validation pour `raw.develop.batch/1`, ainsi que + S1–S3 Capture / Acquisition Ingestion : provenance Capture/Asset, import + capture-safe, publication dérivée, développement RAW, siblings multi-source, + évidence S3-D, orchestration S3-E et campagne bornée : [Project DB et ingestion](../architecture/project_database.md). ### S3 Capture / Acquisition Ingestion — PASS / FROZEN @@ -71,8 +171,7 @@ Campagne `Photogrammetrie/2026-08-26_Baie_Moteur_A6000` : - plan identique après inversion de l'ordre des racines. Le stem commun ne prouve jamais une acquisition. Les 953 propositions doivent -être confirmées et deviennent `CALLER_EXPLICIT`, jamais `STRONG`. La campagne -complète n'a pas encore été matérialisée. +être confirmées et deviennent `CALLER_EXPLICIT`, jamais `STRONG`. ### Limite de reprise DB v19 (historique) @@ -118,7 +217,7 @@ Project DB v20 : A6000 (953 ARW + 953 JPEG, 953 confirmations groupes singleton déterministes). Un échantillon borné a exercé deux Captures A6000 (JPEG SOURCE puis RAW dérivé) et trois Captures S21, avec ScanSets isolés, Queue/Governor, persistance tâche/groupe→Capture et reprise sans duplication. -La capacité bornée des propositions de revue conserve un préfixe déterministe; +La capacité bornée des propositions de revue conserve un préfixe déterministe ; elle ne limite jamais l'évaluation complète ni le groupement scientifique. ### État de calibration des campagnes réelles @@ -232,13 +331,20 @@ responsabilités. L'overlay optique Project DB v23, ajouté ultérieurement, ne réinterprète pas cette preuve v22 ; les copies des deux projets migrent avec les comptes scientifiques inchangés et les nouvelles tables optiques vides. +Pour A6000, le JPEG caméra pairé reste un SOURCE valide et le proxy rapide de +Photo Quality, mais la représentation géométrique FROZEN de cette exécution +sélectionnée reste le PNG déterministe dérivé du RAW (`L3DRAWD1`). Passer cette +preuve au JPEG caméra nécessiterait une tranche scientifique/calibration séparée +prouvant l'équivalence exacte ; ce changement n'est pas une optimisation +opérationnelle ordinaire. + Le Sparse SfM réel final reste `BLOCKED_BY_KNOWN_CALIBRATION_DATA` pour ces campagnes ; aucune pseudo-calibration, interpolation de métadonnées ou inférence d'identité ne le contourne. Dense, mesh et publication aval ne font pas partie de ce jalon pré-SfM. Ce milestone est une intégration réelle, pas une nouvelle série de micro-gates S3. Son statut acquis ne rend pas disponibles les données -de calibration physique manquantes et n'autorise pas à devancer la maintenance -globale. +de calibration physique manquantes et n'autorise pas à devancer les dépendances +scientifiques. ## INTERNAL PARALLELISM + COMPUTE RESOURCES v1 — PASS / FROZEN @@ -273,83 +379,76 @@ reproductibilité. Les dimensions retenues et toutes les validations v2 sont closes. Le pool CPU12 est validé comme preuve de cet hôte, pas comme plafond produit. -Le Governor dérive désormais le pool lourd depuis le masque -permis et les groupes package/core/SMT. Sur l'hôte unrestricted courant, il -obtient `0-5,8-13` et réserve `6,7,14,15`; un caller déjà précontraint ne subit -pas une seconde réserve. Le worker Queue seul applique/vérifie son propre -masque ; aucun TID auxiliaire énuméré n'est muté, car un pidfd ne stabilise pas -le numéro consommé par `sched_setaffinity(tid)`. Avant tout pthread applicatif -ou driver, le démarrage établit `MESA_SHADER_CACHE_DISABLE=true`; une valeur -absente prend ce défaut, `true`/`1` explicites sont conservées et toute autre -valeur explicite est préservée mais refusée. Cette politique non scientifique -supprime les helpers de cache Mesa observés qui élargissaient leur masque. Les -threads runtime restants héritent le compute-pool ; il n'existe plus de sweep, -latch ou retry auxiliaire et le diagnostic expose la politique réelle. -Creator/main/TUI reste unrestricted. Les CPU déjà exclus de l'affinité du -processus comptent dans la réserve hôte ; le fallback count-only ne fabrique -aucun masque. Le fallback au budget portable ne crée aucune exclusion -inventée. Cette couture est **PASS / FROZEN** sur le profil validé; ces IDs ne -sont pas une politique portable. La cible RAM conserve une réserve dure de -3 GiB de `MemAvailable`; la zone 3–4 GiB est une prudence qui ne soustrait pas -4 GiB à toute capacité. Les petits hôtes dégradent le budget en conservant au -moins une unité de calcul. Les PSI CPU/mémoire/I/O et les deltas -swap-in/swap-out sont des signaux -actifs ; l'occupation totale du swap reste historique. Admission CPU et lot -sont indépendantes. Sur la 780M, Hardware Profile classe conservativement comme -UMA le petit aperture VRAM amdgpu de 512 Mio accompagné d'environ 7,99 Go de -GTT système ; la capacité rapportée reste observable mais ne devient pas un -budget séparé. Les coûts GPU sont débités exactement une fois de la RAM hôte. +Le Governor dérive désormais le pool lourd depuis le masque permis et les +groupes package/core/SMT. Sur l'hôte unrestricted courant, il obtient +`0-5,8-13` et réserve `6,7,14,15`; un caller déjà précontraint ne subit pas une +seconde réserve. Le worker Queue seul applique/vérifie son propre masque ; aucun +TID auxiliaire énuméré n'est muté. Creator/main/TUI reste unrestricted. Les CPU +déjà exclus de l'affinité du processus comptent dans la réserve hôte ; le +fallback count-only ne fabrique aucun masque. La cible RAM conserve une réserve +dure de 3 GiB de `MemAvailable`; la zone 3–4 GiB est une prudence qui ne +soustrait pas 4 GiB à toute capacité. Les petits hôtes dégradent le budget en +conservant au moins une unité de calcul. Les PSI CPU/mémoire/I/O et les deltas +swap-in/swap-out sont des signaux actifs ; l'occupation totale du swap reste +historique. Admission CPU et lot sont indépendantes. Sur la 780M, Hardware +Profile classe conservativement comme UMA le petit aperture VRAM amdgpu de +512 Mio accompagné d'environ 7,99 Go de GTT système ; la capacité rapportée +reste observable mais ne devient pas un budget séparé. Les coûts GPU sont +débités exactement une fois de la RAM hôte. -L'audit Phase 1 couvre les 14 kinds de production et sépare leurs dimensions -fixes des dimensions réellement adaptables. Il confirme que tous passent par -l'unique Governor, même les formes fixes, et que le contrat reste immutable -pendant une séquence. Cette tranche ferme l'enveloppe privée, la sélection AUTO, -les diagnostics bornés, l'enforcement OpenCV borné au compute-pool, la politique -d'affinité privée et la réconciliation du contexte retenu des campagnes -nouvelles ou restaurées. La création/reprise AUTO ne touche plus Vulkan sur le -main ; le premier begin appartient au worker contraint. Inflight ORB normal est -maintenant fixé à 1, helpers reste 0; depth 2 reste une capacité privée de -sûreté/benchmark. La clôture v2 est acquise. Ces choix ne créent aucune limite -de dataset, identité scientifique ou version Project DB. +La nouvelle autorité humaine `SERIALISM_REQUIRES_PROOF` **supersède toute +interprétation** de l'inventaire de maintenance qui ferait d'un ancien +`CPU1`/`batch1` un plafond permanent. Les preuves scientifiques et résultats de +la maintenance restent FROZEN ; seules les enveloppes opérationnelles peuvent +être rouvertes lorsqu'un travail indépendant est inutilement sérialisé. Une +Task fixe demeure fixe seulement si sa sérialité ou son genou de scaling est +prouvé. Cette règle doit être appliquée à toutes les Tasks de production avant +d'engager de nouvelles preuves réelles longues. -La signature durable des nouvelles Tasks ORB normales est maintenant la classe -`MIXED`, sémantiquement réelle pour une politique susceptible d'exécuter CPU ou -Vulkan. Elle reconstruit AUTO ; toutes les signatures CPU anciennes/courantes -restent CPU fixes et Vulkan reste fixe, sans migration DB/codec. La couture -asynchrone est privée, request-bound et nettoie son slot sur toute sortie. Une -preuve événementielle bornée établit la soumission du successeur avant la -publication du prédécesseur pour deux paires 769×769. La rampe ne croît plus sur -la seule santé : l'adaptation générique exige deux observations par fenêtre, -tandis que le lot ORB Vulkan en exige huit, avec retour au palier accepté sans -gain et reset immédiat sous pression. Ces -éléments sont **PASS / FROZEN** dans les limites validées. +L'audit Phase 1 couvre les kinds de production de l'époque et sépare leurs +dimensions fixes des dimensions réellement adaptables. Il confirme que tous +passent par l'unique Governor, même les formes fixes, et que le contrat reste +immutable pendant une séquence. Cette tranche ferme l'enveloppe privée, la +sélection AUTO, les diagnostics bornés, l'enforcement OpenCV borné au +compute-pool, la politique d'affinité privée et la réconciliation du contexte +retenu des campagnes nouvelles ou restaurées. La création/reprise AUTO ne +touche plus Vulkan sur le main ; le premier begin appartient au worker +contraint. Inflight ORB normal est fixé à 1, helpers reste 0 ; depth 2 reste une +capacité privée de sûreté/benchmark. Ces choix ne créent aucune limite de +dataset, identité scientifique ou version Project DB. -Les diagnostics de séquence distinguent maintenant backend sélectionné et -backend réel ; les participants CPU Matcher restent `cpu_threads` et -`helpers=0`. Une paire Vulkan inéligible ou en panne est recalculée entièrement -sur CPU sans preuve partielle. Les extractions ORB/SIFT/RootSIFT testées à -1/2/4/8/12 consomment leur contrat OpenCV immutable avec sorties égales. -Le rolling distingue désormais un handle soumis d'une inéligibilité locale et -n'appelle jamais `finish` sans requête. La panne backend invalide immédiatement -l'admission partagée sur toute sortie précoce ; seules les reprises AUTO peuvent -établir cette disponibilité, indépendamment de l'ordre des reprises fixes ou -historiques. Le statut est **PASS / FROZEN**. +La signature durable des nouvelles Tasks ORB normales est la classe `MIXED`, +sémantiquement réelle pour une politique susceptible d'exécuter CPU ou Vulkan. +Elle reconstruit AUTO ; toutes les signatures CPU anciennes/courantes restent +CPU fixes et Vulkan reste fixe, sans migration DB/codec. La couture asynchrone +est privée, request-bound et nettoie son slot sur toute sortie. Une preuve +événementielle bornée établit la soumission du successeur avant la publication +du prédécesseur pour deux paires 769×769. La rampe ne croît plus sur la seule +santé : l'adaptation générique exige deux observations par fenêtre, tandis que +le lot ORB Vulkan en exige huit, avec retour au palier accepté sans gain et +reset immédiat sous pression. Ces éléments sont **PASS / FROZEN** dans les +limites validées. -La boucle privée mesure désormais l'utilisation `/proc/stat` du compute-pool, +Les diagnostics de séquence distinguent backend sélectionné et backend réel ; +les participants CPU Matcher restent `cpu_threads` et `helpers=0`. Une paire +Vulkan inéligible ou en panne est recalculée entièrement sur CPU sans preuve +partielle. Les extractions ORB/SIFT/RootSIFT testées à 1/2/4/8/12 consomment +leur contrat OpenCV immutable avec sorties égales. Le rolling distingue un +handle soumis d'une inéligibilité locale et n'appelle jamais `finish` sans +requête. La panne backend invalide immédiatement l'admission partagée sur toute +sortie précoce ; seules les reprises AUTO peuvent établir cette disponibilité, +indépendamment de l'ordre des reprises fixes ou historiques. Le statut est +**PASS / FROZEN**. + +La boucle privée mesure l'utilisation `/proc/stat` du compute-pool, `MemAvailable`, PSI mémoire/I/O `some/full`, deltas swap actifs, RSS/HWM observé et GPU busy DRM, avec `unknown` sur absence ou parse non strict. Le backend -Vulkan fournit des compteurs cumulatifs bornés de submit/complétion/fence/ -readback/GPU/starvation/panne/discard ; Matcher agrège en plus CPU et publication -par séquence. Le diagnostic est tirable par numéro de série, sans log ncurses ni -histoire persistée. Les CPU réductibles progressent par puissances de deux vers -la capacité exacte du kind/compute-pool après deux observations de baseline et -deux gains d'au moins 5 % ; CPU et lot ne changent -jamais dans le même essai. Sous pression, une admission adaptative encore -permise réserve immédiatement CPU1 et lot minimum. Feature/SIFT/RootSIFT ne -comptent un item qu'après extraction et publication durable propre ; READY, -`ALREADY_PRESENT` et publication incertaine comptent zéro, comme un segment -Visual Index non durable. -Candidate mesure chaque séquence. Un +Vulkan fournit des compteurs cumulés bornés de submit/complétion/fence/readback/ +GPU/starvation/panne/discard ; Matcher agrège en plus CPU et publication par +séquence. Le diagnostic est tirable par numéro de série, sans log ncurses ni +histoire persistée. Les CPU réductibles progressent vers la capacité exacte du +kind/compute-pool selon les observations de scaling ; sous pression, l'admission +se réduit puis doit pouvoir remonter. Candidate mesure chaque séquence. Un fallback réel annule l'essai Vulkan au lieu d'empoisonner sa baseline. Cette implémentation est **PASS / FROZEN** sans ajouter de helper GPU. @@ -359,105 +458,55 @@ digest `7a9dbc38a23a600379167d55e24836b7acbb22eea25573e7440bdc9e4602b3b3`. La starvation depth 1 (53,847 s sur 74,613 s, GPU busy max 25 %) motive deux slots bornés mais ne constitue pas une mesure depth 2. Le backend partage device/pipeline/layout/cache et duplique seulement 640 Kio de payload, -command/fence/descriptors/query par slot. Le payload mappé suit désormais -exactement la capacité de séquence admise : zéro avant initialisation, 640 Kio -à depth 1 et 1,25 Mio uniquement pendant un contrôle privé depth 2, avec retour -à 640 Kio avant la prochaine admission depth 1. L'enveloppe normale n'essaie -plus inflight 2. Les générations de requête ne bouclent pas; un slot épuisé est +command/fence/descriptors/query par slot. Le payload mappé suit exactement la +capacité de séquence admise : zéro avant initialisation, 640 Kio à depth 1 et +1,25 Mio uniquement pendant un contrôle privé depth 2, avec retour à 640 Kio +avant la prochaine admission depth 1. L'enveloppe normale n'essaie plus +inflight 2. Les générations de requête ne bouclent pas ; un slot épuisé est retiré définitivement. Le harness réel a exécuté le corpus 4113 paires pour le contrôle synchrone, rolling depth 1 et l'A/B forcé depth 1/depth 2. L'ABBA forcé donne 54,661652238 et 55,797311953 paires/s (+2,077617 %, sous le deadband 5 %). -Chaque run porte 4113 paires durables; le débit combiné est +Chaque run porte 4113 paires durables ; le débit combiné est `(2 * 4113 * 1e9) / somme(wall_ns)`, pas la moyenne des débits par run. Les walls bruts 75326831673/75162582080 et 73662096698/73764360098 ns donnent les -moyennes 75,244706877/73,713228398 s. Fence vaut 6,0684/3,6776 s, starvation 54,4534/50,1465 s, -publication 29,2582/30,0548 s, submit CPU 0,2655/0,3818 s, readback -0,0460/0,0873 s et GPU busy max 23/24 %. Les quatre exécutions conservent le -digest `7a9dbc38a23a600379167d55e24836b7acbb22eea25573e7440bdc9e4602b3b3`, quatre -séquences de fallback local par exécution et zéro panne/discard. Depth 2 est donc -**REJECTED_WITH_MEASURED_REASON** pour la politique normale : -`DEPTH_MAX_VALIDATED_SAFETY=2`, `DEPTH_MAX_USEFUL=1`. Une comparaison -whole-corpus adaptative antérieure (+1,12 %) ne séparait pas les changements de -contrat; l'ABBA ci-dessus la remplace comme décision de profondeur. Le -répertoire retenu est -`/home/fy59/Documents/Lardon/.real-pre-sfm-2026-08-30/governor-v2-evidence/`, -avec `forced-depth1-a.stdout.jsonl`, `forced-depth1-b.stdout.jsonl`, -`forced-depth2-a.stdout.jsonl` et `forced-depth2-b.stdout.jsonl`. Le -mode Matcher par défaut est AUTO/rolling ; CPU et Vulkan explicites restent des -overrides. Un contrôle synchrone fence par fence est compilé seulement dans le -runner/test, absent de `lardon3d`, non persisté et non applicable à une reprise -Matcher pendante. Les fixtures et ces deux exécutions établissent l'égalité -exacte des sorties; les chiffres depth 1 sont rapportés ci-dessus. Le runner émet -des diagnostics JSON échantillonnés sans prétendre voir chaque séquence, plus -un agrégat Governor fixe exact pour les compteurs enregistrés. Il valide la -bijection Candidate Pair/Match Result, les assets et le curseur, puis produit le -digest canonique de comparaison `L3DMRD1`, qui exclut IDs Task, timestamps et -choix opérationnels. Ces preuves ferment **PASS / FROZEN** sans promouvoir -depth 2 dans AUTO normal. +moyennes 75,244706877/73,713228398 s. Fence vaut 6,0684/3,6776 s, starvation +54,4534/50,1465 s, publication 29,2582/30,0548 s, submit CPU 0,2655/0,3818 s, +readback 0,0460/0,0873 s et GPU busy max 23/24 %. Les quatre exécutions +conservent le digest `7a9dbc38a23a600379167d55e24836b7acbb22eea25573e7440bdc9e4602b3b3`, +quatre séquences de fallback local par exécution et zéro panne/discard. Depth 2 +est donc **REJECTED_WITH_MEASURED_REASON** pour la politique normale : +`DEPTH_MAX_VALIDATED_SAFETY=2`, `DEPTH_MAX_USEFUL=1`. -Le runner possède maintenant, dans ce seul target, `--matcher-inflight 1|2` : -AUTO rolling fixe batch 2 et min=max inflight pour permettre la mesure du même -binaire à profondeur 1 puis 2, en n'exposant que la capacité Vulkan forcée. -`--matcher-batch 2|4|8|12`, valable seulement avec inflight et AUTO rolling, -fixe aussi min=max batch pour la prochaine matrice contrôlée. -Synchronous reste depth 1. Le Governor conserve son admission et sa charge UMA; -GPU budget zéro ou capacité GPU/backend/mémoire indisponible échoue au lieu de -sélectionner CPU. La valeur n'est ni persistée ni scientifique, -est refusée sur une Task pendante, restaurée dans l'environnement à toute sortie -et émise comme `1`, `2` ou `null` dans les résumés/agrégats. Le runner valide le -contrat exact et zéro panne/discard/pending; seule l'inéligibilité locale peut -produire un fallback CPU complet. La production compte désormais chaque item -exactement une fois après publication durable dans local-ineligible, -backend-failure ou other; une admission CPU normale reste à zéro. Le compteur -est incrémenté immédiatement et reste visible si une paire suivante échoue ou est -annulée, sans valider ni entraîner la séquence incomplète; son high-water mark -de déduplication reste privé et non persisté. L'égalité -inter-cohortes porte sur les items locaux, pas sur les séquences dont le nombre -dépend du batch. Une panne/raison autre, ou une panne tardive, invalide la -cohorte; la Task de benchmark échoue après checkpoint de toute preuve CPU déjà -durable. Les fixtures -établissent l'égalité exacte rolling1/rolling2/synchrone1 et des batches -2/4/8/12 sur fixture, avec le même compte de 29 items locaux malgré des comptes -de séquences différents. Les logs corpus préliminaires -`forced-batch2-current.stdout.jsonl` et `forced-batch4.stdout.jsonl` couvrent -les mêmes 4113 IDs/digest mais portent quatre contre trois séquences locales : -ils sont conservés comme preuve que l'ancien comparateur était invalide et ne -participent pas à la décision. Les huit runs item-valides -`forced-batch{2,4,8,12}-items{,-b}.stdout.jsonl` publient chacun 4113 paires, -six items locaux, zéro panne/autre et le même digest. Leurs débits combinés -sont 54,180767704, 66,094373197, 74,784998723 et 76,755814095 paires/s. Batch -4 puis 8 gagnent +21,988624373 % et +13,148812987 %; batch 12 ne gagne que -+2,635308425 %, sous le deadband 5 %. AUTO normal suit donc -`BATCH_MAX_USEFUL=8`; batch 12 reste une capacité privée sûre -`REJECTED_WITH_MEASURED_REASON`. -Le contrôle de production sans override -`short-auto-batch8-governor-v2.stdout.jsonl` atteint réellement -`1 → 2 → 4 → 8`, puis publie 4113/4113 résultats à 76,072 paires/s avec le -même digest, six fallbacks locaux, zéro panne/discard, inflight 1 et helpers 0. -La preuve de fermeture S21 -`final-s21-auto.stdout.jsonl` exécute ensuite le chemin production normal sur -172 741 Candidate Pairs : 172 741 Match Results, zéro doublon, curseur complet, -digest `e5128a2e599ff593c4f79850e067254b1f249d19e8480a44973306b1af250f70` -et 73,649 résultats durables/s. AUTO choisit Vulkan sur toutes les admissions, +Les huit runs item-valides `forced-batch{2,4,8,12}-items{,-b}.stdout.jsonl` +publient chacun 4113 paires, six items locaux, zéro panne/autre et le même +digest. Leurs débits combinés sont 54,180767704, 66,094373197, 74,784998723 et +76,755814095 paires/s. Batch 4 puis 8 gagnent +21,988624373 % et ++13,148812987 % ; batch 12 ne gagne que +2,635308425 %, sous le deadband 5 %. +AUTO normal suit donc `BATCH_MAX_USEFUL=8`; batch 12 reste une capacité privée +sûre `REJECTED_WITH_MEASURED_REASON`. Le contrôle de production sans override +`short-auto-batch8-governor-v2.stdout.jsonl` atteint réellement `1 → 2 → 4 → 8`, +puis publie 4113/4113 résultats à 76,072 paires/s avec le même digest, six +fallbacks locaux, zéro panne/discard, inflight 1 et helpers 0. + +La preuve de fermeture S21 `final-s21-auto.stdout.jsonl` exécute le chemin +production normal sur 172 741 Candidate Pairs : 172 741 Match Results, zéro +doublon, curseur complet, digest +`e5128a2e599ff593c4f79850e067254b1f249d19e8480a44973306b1af250f70` et +73,649 résultats durables/s. AUTO choisit Vulkan sur toutes les admissions, termine batch 8/inflight 1/helpers 0 et ne compte aucune panne/discard/pending. Une admission YELLOW réduit batch 8 à 1, puis les séquences GREEN rétablissent -1 → 2 → 4 → 8; le gate possède donc aussi une preuve réelle de recovery. +1 → 2 → 4 → 8 ; le gate possède donc une preuve réelle de recovery. -La poursuite S21 ferme ensuite le point Geometric Verifier v3 sans rejouer le -Matcher. La Task 2832 consomme les 172 741 Match Results et publie 172 275 GVR -v3, dont 24 065 acceptés et 148 210 rejetés, avec zéro doublon et curseur -complet. Le digest Matcher reste -`e5128a2e599ff593c4f79850e067254b1f249d19e8480a44973306b1af250f70`. -La seconde reprise crée zéro ligne ; une Task interrompue sur une copie dédiée -reprend le même ID et converge vers les mêmes lignes exactes. Feature, -Candidate et Matcher ne sont pas rejoués. Track Builder et Sparse SfM ne sont -pas exécutés. `REAL_S21_GV_V3=PASS/FROZEN` ; la prochaine tranche scientifique -S21 commencera donc à Tracks maintenant que la gate de maintenance globale -ci-dessous est fermée. Tracks n'est pas acquis. +La poursuite S21 ferme le Geometric Verifier v3 sans rejouer le Matcher. La Task +2832 consomme les 172 741 Match Results et publie 172 275 GVR v3, dont 24 065 +acceptés et 148 210 rejetés, avec zéro doublon et curseur complet. La seconde +reprise crée zéro ligne ; une Task interrompue sur une copie dédiée reprend le +même ID et converge vers les mêmes lignes exactes. Feature, Candidate et +Matcher ne sont pas rejoués. `REAL_S21_GV_V3=PASS/FROZEN`. -Les gates de fermeture sont acquis : +Les gates de fermeture acquises incluent : ```text GOVERNOR_CONTROLS_ALL_TASKS=PASS @@ -489,39 +538,17 @@ REAL_S21_GV_V3=PASS/FROZEN FEATURE_REPLAY=0 CANDIDATE_REPLAY=0 MATCHER_REPLAY=0 -TRACKS_EXECUTED=0 SPARSE_SFM_EXECUTED=0 ``` -L'expérience normale est donc : l'utilisateur lance une Task; l'unique +L'expérience normale est donc : l'utilisateur lance une Task ; l'unique Governor choisit et explique le contrat borné de sa prochaine séquence. Aucun -réglage CPU/GPU/lot/inflight/helper n'est requis en production ordinaire. - -L'ordre de travail autorisé à court terme est : - -```text -fondation scientifique pré-SfM courante -→ INTERNAL PARALLELISM + COMPUTE RESOURCES v1 - (Candidate, Matcher, Visual Index, audit feature threading, - correction progression/Resource Governor, audit GPU) -→ COMPUTE GOVERNOR v2 / ORB VULKAN ASYNC EXECUTION (PASS / FROZEN) -→ REAL S21 GV v3 (PASS / FROZEN; arrêt avant Tracks) -→ GLOBAL MAINTENANCE AUDIT - (PASS / FROZEN; validation et revue finale indépendante acquises) -→ poursuite pré-SfM réelle de S21 à partir de Tracks -→ acquisition dédiée de calibration -→ Sparse SfM réel -→ Dense / MVS -``` +réglage CPU/GPU/lot/inflight/helper n'est requis en production ordinaire. Avec +la nouvelle autorité humaine, ce choix doit désormais rechercher explicitement +le **maximum sûr et utile**, et non conserver une sous-utilisation historique. ## GLOBAL MAINTENANCE AUDIT — PASS / FROZEN -Le jalon scientifique acquis reste arrêté après GV. Avant Tracks, Sparse SfM -ou Dense/MVS, la gate de maintenance globale a réconcilié l'architecture, les -ressources, la persistance et l'observation TUI. Son état et ses preuves -consolidées sont consignés dans le -[registre canonique de maintenance](../architecture/global_maintenance_audit.md). - Le checkpoint canonique de revue est le tag `global-maintenance-2026-09-01`, au commit `b84f860d868c66d9ee84b85ceb1bc6480b95aca5`. Les revues futures sont @@ -532,44 +559,30 @@ dépendances traversées. Les systèmes PASS/FROZEN inchangés héritent de la p du [registre de maintenance](../architecture/global_maintenance_audit.md) et ne sont rouverts que sur preuve concrète ; ne pas répéter un audit global A-à-Z. -Les résultats déjà réglés sont : +Les résultats acquis restent notamment : - Project DB v23 ajoute neuf relations optiques sans backfill ni inférence ; - les boîtiers, objectifs manuels/électroniques, configurations et calibrations - restent distincts, et la sélection de calibration exige une compatibilité - exacte et explicite ; -- le Governor réserve d'abord l'hôte, sans plafond CPU global 12 : réserve de - quatre CPU logiques sur hôte capable, groupes cœur/SMT complets lorsque la - topologie est fiable, au moins une unité de calcul sur petit hôte, réserve - RAM dure 3 GiB et prudence 3–4 GiB, PSI/swap actif et UMA comptée une fois ; -- Candidate porte une capacité sûre 64, Visual Index 16, ORB/SIFT/RootSIFT sont - bornés par le compute-pool, Matcher conserve sa borne intrinsèque sûre 12 et - utile 8, et GV conserve son USAC interne sériel mais admet 16 participants - sûrs/8 utiles et 16 parents par lot ; +- le Governor réserve l'hôte sans plafond CPU global 12, avec réserve RAM dure + 3 GiB, PSI/swap actif et UMA comptée une fois ; +- Candidate, Visual Index, Feature, Matcher et GV possèdent leurs capacités et + décisions GPU mesurées ; - le contrôleur SSD UDisks2 optionnel est une frontière physique revue, avec - identité Drive+labels+UUID, leases, drain sûr et latch de danger pour action - indéterminée. Son état est enregistré auprès du Governor, seul orchestrateur - des leases de production ; il n'est ni scheduler ni second Governor ; -- la TUI est un observatoire/centre de contrôle validé opérationnellement : - ncurses main-thread, modèle pur, observation coalescée et bornée, progression - durable/ETA, pipeline et ressources honnêtes, profils optiques, layouts - 100×30/72×20/60×15, repli texte/couleur et F10 SSD asynchrone toujours - visible ; + identité Drive+labels+UUID, leases et drain sûr ; +- la TUI est un observatoire/centre de contrôle validé opérationnellement ; - la frontière de session détruit/joint la Queue avant Project DB, puis recrée - une seule Queue ; l'arrêt global libère les leases Task avant unregister SSD, - contrôleur et Governor. + une seule Queue. -Les validations finales exécutables sont acquises : build Clang portable -931/931 + suite 64/64, build Clang Vulkan 939/939 + suite 65/65 sur Radeon -réelle, ASan/UBSan 64/64 avec la limitation LSan OpenCL externe explicitement -qualifiée, LSan loader-free 20/20, TSan 14/14 + 220 répétitions, headers publics -76/76 sur 19 headers modifiés/nouveaux et contrôles ABI/diff/`scan3d`. -L'unique revue finale indépendante GPT-5.6 SOL/ULTRA a conclu PASS sans finding -bloquant après avoir indépendamment rejoué le build portable, la suite 64/64, -15/15 tests focalisés, les 76/76 probes de headers, l'ABI, les négatifs de seams -production, le SHA du manifest GV retenu et le diff-check. La gate de -maintenance est donc fermée ; la poursuite réelle depuis Tracks devient la -prochaine tranche séparée, sans avoir été exécutée par cette synchronisation. +Le nouvel impératif `SERIALISM_REQUIRES_PROOF` ne nie aucune de ces preuves. Il +supersède uniquement l'idée qu'un ancien profil opérationnel conservateur serait +un plafond permanent lorsque de nouvelles preuves réelles montrent du travail +indépendant et des ressources sûres inutilisées. + +Les validations finales exécutables de la maintenance sont acquises : build +Clang portable 931/931 + suite 64/64, build Clang Vulkan 939/939 + suite 65/65 +sur Radeon réelle, ASan/UBSan 64/64 avec la limitation LSan OpenCL externe +qualifiée, LSan loader-free 20/20, TSan, headers publics 76/76 et contrôles +ABI/diff/`scan3d`. La gate de maintenance reste fermée ; les nouvelles +corrections de ressources sont revues en delta et ne réouvrent pas l'audit A→Z. ## REAL S21 TRACKS — PASS / FROZEN @@ -606,8 +619,175 @@ runner corrigé. Elle termine `COMPLETE` à 100 % et retrouve le même Track Set zéro Track Set. Cette preuve valide la reprise opérationnelle sans rouvrir le contrat scientifique Tracks gelé. +## PROJECT DB v24 / RAW BATCH A6000 — IMPLEMENTED / VALIDATION IN PROGRESS + +L'autorisation humaine additive v24 est limitée au parallélisme opérationnel du +développement RAW. Project DB v24 conserve v22/v23 et ajoute uniquement la +persistance typée `raw_development_batch_tasks` nécessaire au nouveau kind +`raw.develop.batch/1`. Cette version ne change aucune identité scientifique, +aucune calibration ni `L3DRAWD1`. + +Le pattern de production est : + +```text +une Task owner admise par le Governor + → participants RAW indépendants et bornés + → join de tous les participants + → publication owner-only dans l'ordre des selected_item_index + → avance durable du curseur selected_execution +``` + +L'atomicité par Capture est conservée ; la sérialisation entre Captures +indépendantes ne l'est plus. La transition historique `raw.develop/1` → +`raw.develop.batch/1` reste légitime et ne réécrit aucune ancienne Task. + +État durable A6000 au dernier arrêt sûr avant l'audit global d'utilisation des +ressources : + +```text +Projet: +/home/fy59/Documents/Lardon/.real-pre-sfm-2026-09-01/a6000-pre-sfm-v23-final + +Schema v24 +Captures 953 +GOOD 689 +SUSPECT 206 +REJECT 58 +selected_execution 1 +selected items 689 RAW_ASSET +published representations 259 +next_item_index 259 +remaining RAW representations 430 +Features 0 +Visual Index 0 +Candidate 0 +Matcher 0 +GV 0 +Tracks 0 +Sparse SfM 0 +Dense/MVS 0 +``` + +Le digest d'inventaire source pré/post est +`fa725d8a82b529521134dd600b5cf42ed58ad523108636741a1431441e17b029`. +Les JPEG caméra et RAW sont tous deux des SOURCE assets explicites par Capture, +mais cette exécution A6000 FROZEN utilise les PNG déterministes dérivés RAW pour +la géométrie. Les JPEG restent les proxies de triage ; ils ne remplacent pas +cette représentation scientifique sans nouvelle tranche d'équivalence +scientifique/calibration. + +La v24 et la Task batch ont déjà passé build normal, matrice ciblée, contrôles +C17, `git diff --check`, ASan/UBSan ciblé et TSan sur la couture concurrente. +La preuve réelle complète, l'équivalence finale, l'audit d'utilisation de toutes +les Tasks affectées et la fermeture documentaire restent à terminer avant de +marquer cette tranche `PASS/FROZEN`. + +## PROJECT DB v25 / FEATURE BATCH — IMPLEMENTED / VALIDATION IN PROGRESS + +L'overlay additif v25 ajoute uniquement la persistance typée de +`features.extract.batch/1`. Il ancre un propriétaire durable à l'ordre immutable +de l'exécution sélectionnée, au domaine ORB exact et à un curseur monotone. Les +Tasks `features.extract/1` historiques restent inchangées et sont reprises avant +le suffixe batch. La préparation inter-images est bornée par le Governor, +OpenCV reste CPU1 dans chaque participant, et publication/cursor/checkpoint +restent owner-only et ordonnés. Le build et les tests ciblés sont acquis ; le +benchmark réel du palier utile et la preuve A6000 restent à exécuter avant toute +revendication PASS/FROZEN. + +## REAL A6000 PRE-SFM — PASS / FROZEN + +`REAL_A6000_PRE_SFM=PASS/FROZEN`. La continuation du 2 septembre 2026 utilise +le projet durable +`/home/fy59/Documents/Lardon/.real-pre-sfm-2026-09-01/a6000-pre-sfm-v23-final` +directement à la frontière Match Result acquise. Les 689 Feature Sets, le Visual +Index complet, les 38 420 Candidate Pairs et les 38 420 Match Results de la +Task Matcher 170 `COMPLETE/100` sont conservés : +`FEATURE_REPLAY=0`, `CANDIDATE_REPLAY=0` et `MATCHER_REPLAY=0`. Ni l'acquisition, +ni RAW, ni les assets/identités amont ne sont réécrits. + +La Task GV v3 171 termine `COMPLETE/100`, consomme le curseur Match Result +jusqu'à 38 420 et publie exactement 37 805 GVR applicables : 10 952 +`GEOMETRIC_VERIFIED` et 26 853 `GEOMETRIC_REJECTED`. Son fingerprint reste +`6944a471d611d8ffc59dac7cf15a5b79b97e2371d4c51785c477d68c1577f74c`; +zéro mapping GVR dupliqué est observé. Les diagnostics Governor coalescés +retiennent 2 714 admissions, le compute-pool de 12 CPU avec 4 CPU réservés, +une fenêtre finale de 16, RSS/HWM maximal 41 877 504/42 319 872 octets, +`MemAvailable` minimal 10 555 641 856 octets et des deltas swap-in/out nuls. +La préparation GV reste dans la forme validée Governor-admise et bornée ; aucune +nouvelle policy CPU/GPU n'est introduite par ce run. + +La Task Track Builder 172 termine `COMPLETE/100` et publie atomiquement le seul +Track Set 1, de scope 10 952 GVR, avec 130 714 Tracks et 318 944 observations. +L'audit SQL confirme zéro observation de Track dupliquée, zéro Track contenant +deux observations de la même image et zéro observation orpheline. La seconde +continuation crée seulement la Task GV 173, qui traverse le même curseur, +publie zéro GVR et réutilise Track Set 1 sans créer de Task Track Builder ni +modifier les 130 714 Tracks. `RESTART_IDEMPOTENCE`, `DETERMINISM`, +`TRACK_CONFLICT_INVARIANTS` et `RESOURCE_POLICY_ENFORCED` sont donc PASS pour +cette frontière réelle. + +Sparse SfM, reconstruction sparse, Dense/MVS et fusion inter-campagnes restent +à zéro. Ce gel atteste uniquement le pré-SfM A6000 à travers GV puis Tracks ; il +ne contourne pas le blocage de calibration réel et ne promeut aucune étape aval. + +## RESOURCE UTILIZATION ENFORCEMENT — CURRENT NEXT + +Avant de reprendre les 430 RAW restants ou toute autre preuve réelle longue, +la prochaine tranche opérationnelle doit appliquer la politique canonique à +l'ensemble des Tasks de production, en revue delta uniquement. + +Objectifs : + +```text +RESOURCE_UTILIZATION_POLICY=IMPLEMENTED +SERIALISM_REQUIRES_PROOF=ENFORCED +ACCIDENTAL_CPU1_PATHS=0_OR_MEASURED_JUSTIFICATION +VALIDATED_GPU_BACKENDS=PREFERRED_WHEN_ELIGIBLE +BUILD_TEST_PARALLELISM=HOST_RESERVE_AWARE +``` + +Pour chaque Task, auditer uniquement les dimensions opérationnelles : +minimum/useful/safe CPU et batch, indépendance des items, séparation +préparation/publication, coûts RAM par participant, saturation I/O et backend +GPU validé. Une valeur CPU1/batch1 peut rester seulement avec preuve concrète. +Il ne faut pas inventer de second scheduler, pool global ou parallélisme +inter-Tasks pour satisfaire cette politique : le parallélisme interne borné et +l'unique Queue/Governor restent la voie normale. + +Les tests Codex doivent suivre la même règle : build et tests indépendants +parallèles jusqu'à la réserve interactive ; serialisation uniquement pour +fixtures partagées, déterminisme scientifique, mémoire/I/O/GPU exclusif, +sanitizer ou pression mesurée. Les validations acquises et non affectées ne +sont pas répétées. Les modèles/agents coûteux sont réservés aux vraies +frontières scientifiques, concurrence/persistance délicate et revue finale +nécessaire. + ## NEAR TERM +Ordre court terme désormais canonique : + +```text +GLOBAL MAINTENANCE AUDIT PASS / FROZEN +→ REAL S21 TRACKS PASS / FROZEN +→ RESOURCE UTILIZATION ENFORCEMENT CURRENT NEXT + (SERIALISM_REQUIRES_PROOF, max safe/useful throughput, + politique tests/builds Codex comprise) +→ reprise A6000 v24 au curseur 259 + → 430 représentations RAW restantes + → Features + → Visual Index + → Candidate + → Matcher AUTO / ORB Vulkan si éligible + → GV + → Tracks + → STOP avant Sparse SfM +→ acquisition physique dédiée de calibration +→ calibration connue validée +→ Sparse SfM réel par campagne +→ fusion / Phase H selon dépendances +→ Dense / MVS +``` + ### Scratch SSD externe et swap optionnel Le contrôleur physique actuel découvre par UDisks2 une paire exacte de labels @@ -621,9 +801,8 @@ Les usages futurs possibles sont workspace/scratch et intermédiaires dense/mesh/texturing. Leur consommation devra être explicitement possédée par les Tasks et passer par les wrappers de lease du Resource Governor ; le contrôleur physique et le registre actuel n'inventent aucune éligibilité Task. -Les quatorze kinds courants ne consomment aucun scratch. Le swap de sécurité, -lui, reste une fonction du drain physique explicite et jamais un budget de -travail. +Le swap de sécurité reste une fonction du drain physique explicite et jamais un +budget de travail. - les leases scratch ont une ownership explicite ; `DRAINING` refuse les nouveaux leases et attend leur libération exacte ; @@ -636,14 +815,11 @@ travail. - une action UDisks potentiellement appliquée mais non vérifiable verrouille le tuple physique original ; un remplacement n'obtient aucune autorité ; - aucun formatage, partitionnement, fsck, réparation, arrêt forcé ou suppression - n'est permis. La production n'appelle pas `statvfs`; espace total/libre reste - `UNKNOWN` quand UDisks ne le fournit pas. + n'est permis. Le contrôleur, son registre Governor, le worker joinable unique et sa -présentation/actions F10 sont **CURRENT / VALIDATED OPERATIONAL**. Les tests -emploient un provider factice et ne montent, n'arrêtent ni ne modifient un vrai -disque. Ce statut ne crée aucun consommateur scratch ; c'est la validation -consolidée et la revue indépendante ci-dessus qui ont fermé la gate globale. +présentation/actions F10 sont **CURRENT / VALIDATED OPERATIONAL**. Ce statut ne +crée aucun consommateur scratch par lui-même. ### Publication durable dense / mesh @@ -715,8 +891,7 @@ Le but à long terme n'est pas seulement de reconstruire ce qui a été photographié, mais de guider activement l'opérateur vers les photographies qui manquent encore pour obtenir une reconstruction fiable. Cette capacité est postérieure à une reconstruction suffisamment mature ; elle ne fait pas partie -de S3 et ne remplace pas le prochain milestone d'intégration multi-campagne -A6000 + S21. +de S3. ```text photographies existantes @@ -933,20 +1108,25 @@ L'ordre demeure sans ambiguïté : ```text CURRENT NEXT - fondation scientifique pré-SfM courante - → INTERNAL PARALLELISM + COMPUTE RESOURCES v1 - → COMPUTE GOVERNOR v2 / ORB VULKAN ASYNC EXECUTION - (PASS / FROZEN; AUTO GPU-first ORB et preuve S21 Matcher complète) - → REAL S21 GV v3 (PASS / FROZEN; Tracks/Sparse SfM non exécutés) - → GLOBAL MAINTENANCE AUDIT - (PASS / FROZEN; gate fermée avant la tranche Tracks séparée) - → poursuite pré-SfM réelle de S21 à partir de Tracks - → acquisition physique dédiée de calibration - → calibration connue validée → Sparse SfM réel multi-campagne + GLOBAL MAINTENANCE AUDIT PASS / FROZEN + → REAL S21 TRACKS PASS / FROZEN + → RESOURCE UTILIZATION ENFORCEMENT CURRENT NEXT + → A6000 PRE-SFM v24 resume from cursor 259 + → RAW batch + → Features + → Visual Index + → Candidate + → Matcher AUTO / validated Vulkan when eligible + → GV + → Tracks + → STOP before Sparse SfM + → dedicated physical calibration acquisition + → known calibration → real Sparse SfM + → multi-campaign registration/fusion → dense / MVS → publication → LATER - Coverage Analysis → Coverage Viewer → suggestions de points de vue - → localisation live → intégration HDMI/USB → Capture Guidance / Live Coverage + Coverage Analysis → Coverage Viewer → suggested viewpoints + → live localization → HDMI/USB integration → Capture Guidance / Live Coverage ``` Vidéo/keyframes pourra progresser en parallèle lors d'une phase ultérieure, @@ -961,21 +1141,34 @@ scientifique. ## DEFERRED / OPTIONAL -- pools multiples CPU/GPU/IO, parallélisme inter-tâches et multi-GPU ; +- pools multiples CPU/GPU/IO, parallélisme inter-Tasks et multi-GPU ; - DAG général de dépendances et priorités complexes ; - infrastructure backend générique et distribution de calcul ; - ALIKED tant que provenance modèle, export ONNX et oracle upstream ne sont pas reproductibles. +Le report du parallélisme **inter-Tasks** ne constitue jamais une permission de +sérialiser accidentellement les unités indépendantes **à l'intérieur d'une Task +active**. L'unique Queue/Governor et le fanout interne borné suffisent pour +appliquer `SERIALISM_REQUIRES_PROOF` sans créer un second runtime. + Ces idées ne doivent pas devancer l'intégration réelle multi-campagne ni créer un second runtime, Governor ou système de persistance. ## Principes de séquencement -1. stabilité et intégrité scientifique avant débit ; -2. résultats atomiques et durables avant parallélisme ; -3. lots bornés et reprise avant taille de campagne ; -4. un Task Runtime/Queue et un Resource Governor, sans second scheduler ; -5. scratch optionnel sans élargissement implicite des budgets RAM ; -6. TUI de contrôle avant visualisation riche ; -7. documentation canonique alignée sur le code validé. +1. intégrité scientifique et réserve interactive hôte sont inviolables ; à + l'intérieur de ces limites, le maximum de débit sûr et utile est obligatoire ; +2. `SERIALISM_REQUIRES_PROOF` : l'atomicité d'un item ne sérialise pas les items + indépendants et toute sous-utilisation durable doit avoir une raison mesurée ; +3. résultats atomiques et durables avant publication concurrente ; préparation + parallèle bornée autorisée lorsque l'égalité exacte est préservée ; +4. lots bornés, reprise et mémoire honnêtement comptée avant taille de campagne ; +5. un Task Runtime/Queue et un Resource Governor, sans second scheduler ; +6. backend GPU validé et utile préféré automatiquement, sans GPU artificiel ; +7. scratch optionnel sans élargissement implicite des budgets RAM ; +8. builds/tests/preuves Codex parallèles jusqu'à la réserve hôte lorsqu'ils sont + indépendants ; aucune sérialisation ou revalidation lourde par habitude ; +9. TUI de contrôle avant visualisation riche ; +10. documentation canonique alignée sur le code validé et revue delta-based + depuis `global-maintenance-2026-09-01`. diff --git a/include/lardon3d/feature_task.h b/include/lardon3d/feature_task.h index 625cdc1..dc9a7c4 100644 --- a/include/lardon3d/feature_task.h +++ b/include/lardon3d/feature_task.h @@ -10,6 +10,8 @@ #define LARDON3D_FEATURE_EXTRACT_TASK_KIND "features.extract" enum { LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION = 1 }; +#define LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND "features.extract.batch" +enum { LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION = 1 }; Lardon3DTask * lardon3d_project_create_feature_extract_task(Lardon3DAppState *state, uint64_t image_id, @@ -20,5 +22,18 @@ bool lardon3d_project_enqueue_feature_extract(Lardon3DAppState *state, uint64_t uint64_t *task_id); bool lardon3d_feature_extract_reconstruct(const Lardon3DTaskDurableSnapshot *snapshot, void *context, Lardon3DTaskKindBinding *binding); +/* Create/enqueue one durable owner Task over the immutable selected-execution + * image order. ORB parameters are copied into typed v25 state; participants are + * Governor-bounded, cancellable at window boundaries, and joined before the + * owner publishes or advances its cursor. */ +Lardon3DTask *lardon3d_project_create_feature_extract_batch_task( + Lardon3DAppState *state, uint64_t selected_execution_id, + const Lardon3DFeatureExtractorParameters *parameters, uint64_t *task_id); +bool lardon3d_project_enqueue_feature_extract_batch( + Lardon3DAppState *state, uint64_t selected_execution_id, + const Lardon3DFeatureExtractorParameters *parameters, uint64_t *task_id); +bool lardon3d_feature_extract_batch_reconstruct( + const Lardon3DTaskDurableSnapshot *snapshot, void *context, + Lardon3DTaskKindBinding *binding); #endif diff --git a/include/lardon3d/project.h b/include/lardon3d/project.h index f30ee26..4c52c3c 100644 --- a/include/lardon3d/project.h +++ b/include/lardon3d/project.h @@ -73,9 +73,21 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_photo_quality_ta Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_raw_development_task( Lardon3DAppState *state, const Lardon3DTask *task, const Lardon3DProjectDbRawDevelopmentTask *parameters); +/* Persist generic parent Task state atomically with its immutable selected + * execution association. Call only after the selected execution cursor commit + * so generic progress can lag but never lead scientific publication. */ +Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_raw_development_batch_task( + Lardon3DAppState *state, const Lardon3DTask *task, + const Lardon3DProjectDbRawDevelopmentBatchTask *parameters); Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_task( Lardon3DAppState *state, const Lardon3DTask *task, const Lardon3DProjectDbFeatureExtractTask *parameters); +/* Persist the generic checkpoint and typed batch domain/cursor in one SQLite + * transaction. Call only after owner-only Feature publication and cursor + * advancement, so generic progress may lag but never lead durable work. */ +Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_batch_task( + Lardon3DAppState *state, const Lardon3DTask *task, + const Lardon3DProjectDbFeatureExtractBatchTask *parameters); Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_sift_extract_task( Lardon3DAppState *state, const Lardon3DTask *task, const Lardon3DProjectDbSiftExtractTask *parameters); diff --git a/include/lardon3d/project_db.h b/include/lardon3d/project_db.h index 6a92cdd..5c22b86 100644 --- a/include/lardon3d/project_db.h +++ b/include/lardon3d/project_db.h @@ -11,10 +11,10 @@ #include enum { - /* v23 is an additive optical-context layer. Historical v22 rows and sparse - * calibration identities remain untouched; migration never guesses profiles - * or assignments for existing Captures. */ - LARDON3D_PROJECT_DB_SCHEMA_VERSION = 23, + /* v25 adds only durable features.extract.batch/1 operational state. The + * selected execution remains the immutable ordered image domain; migration + * creates no batch association, cursor, Feature Set, or inferred identity. */ + LARDON3D_PROJECT_DB_SCHEMA_VERSION = 25, LARDON3D_PROJECT_DB_ID_CAPACITY = 65, LARDON3D_PROJECT_DB_KIND_CAPACITY = 65, LARDON3D_PROJECT_DB_PATH_CAPACITY = 4096, @@ -145,6 +145,13 @@ typedef struct { uint64_t image_id; } Lardon3DProjectDbRawDevelopmentTask; +typedef struct { + uint64_t task_id; + /* Selected execution owns the ordered scientific inputs and durable cursor. + * This typed Task row deliberately copies none of those identities. */ + uint64_t selected_execution_id; +} Lardon3DProjectDbRawDevelopmentBatchTask; + typedef struct { uint64_t task_id; uint32_t group_id; @@ -406,6 +413,19 @@ typedef struct { unsigned char parameter_fingerprint[32]; } Lardon3DProjectDbFeatureExtractTask; +typedef struct { + uint64_t task_id; + uint64_t selected_execution_id; + /* Zero-based durable prefix in the immutable selected-execution order. */ + uint32_t next_item_index; + char extractor_kind[LARDON3D_PROJECT_DB_KIND_CAPACITY]; + uint32_t extractor_version; + uint32_t max_features; + uint32_t pyramid_levels; + uint32_t fast_threshold; + unsigned char parameter_fingerprint[LARDON3D_PROJECT_DB_SHA256_SIZE]; +} Lardon3DProjectDbFeatureExtractBatchTask; + typedef struct { uint64_t task_id; uint64_t image_id; @@ -785,6 +805,21 @@ Lardon3DProjectDbResult lardon3d_project_db_record_raw_development_task( Lardon3DProjectDbResult lardon3d_project_db_load_raw_development_task( Lardon3DProjectDb *database, uint64_t task_id, Lardon3DProjectDbRawDevelopmentTask *parameters); +/* Atomically record generic raw.develop.batch/1 state and its immutable link + * to one selected execution. Exact retry is idempotent; rebinding either a + * Task or selected execution conflicts. The selected execution cursor remains + * the sole durable item-progress authority. */ +Lardon3DProjectDbResult lardon3d_project_db_record_raw_development_batch_task( + Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, + const char *task_kind, uint32_t task_kind_version, + const Lardon3DProjectDbCheckpoint *checkpoint, + const Lardon3DProjectDbRawDevelopmentBatchTask *parameters, int64_t updated_at); +/* Load the caller-owned immutable Task -> selected-execution association. + * Malformed integer storage or a generic kind/version mismatch is corruption; + * no item identity or cursor is inferred from other durable state. */ +Lardon3DProjectDbResult lardon3d_project_db_load_raw_development_batch_task( + Lardon3DProjectDb *database, uint64_t task_id, + Lardon3DProjectDbRawDevelopmentBatchTask *parameters); /* Result publication and next_group_id advance are atomic. result->group_id is * the canonical plan ID in 1..N and must equal the task's current one-based * cursor; next_group_id must equal result->group_id+1 (N+1 after the last @@ -869,6 +904,26 @@ Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_task( Lardon3DProjectDbResult lardon3d_project_db_load_feature_extract_task(Lardon3DProjectDb *database, uint64_t task_id, Lardon3DProjectDbFeatureExtractTask *parameters); +/* Atomically persist generic features.extract.batch/1 state and its typed + * immutable selected-execution/ORB domain plus monotone cursor. A stale + * generic checkpoint may repeat an older cursor but can never move it back. */ +Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_batch_task( + Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, + const char *task_kind, uint32_t task_kind_version, + const Lardon3DProjectDbCheckpoint *checkpoint, + const Lardon3DProjectDbFeatureExtractBatchTask *parameters, int64_t updated_at); +/* Load and validate the exact typed batch domain and generic dispatch. The + * selected execution must be representation-complete and the cursor bounded by + * its immutable item count; malformed durable state is CORRUPT. */ +Lardon3DProjectDbResult lardon3d_project_db_load_feature_extract_batch_task( + Lardon3DProjectDb *database, uint64_t task_id, + Lardon3DProjectDbFeatureExtractBatchTask *parameters); +/* Advance exactly one ordered item only after its selected image has a READY + * Feature Set for the task's immutable ORB fingerprint. Exact retry after an + * already committed advance converges; no image identity is inferred. */ +Lardon3DProjectDbResult lardon3d_project_db_advance_feature_extract_batch_task( + Lardon3DProjectDb *database, uint64_t task_id, uint32_t item_index, + uint32_t next_item_index); Lardon3DProjectDbResult lardon3d_project_db_record_sift_extract_task( Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind, uint32_t task_kind_version, diff --git a/include/lardon3d/raw_development_task.h b/include/lardon3d/raw_development_task.h index dffa764..c99b7b6 100644 --- a/include/lardon3d/raw_development_task.h +++ b/include/lardon3d/raw_development_task.h @@ -13,6 +13,8 @@ extern "C" { #define LARDON3D_RAW_DEVELOPMENT_TASK_KIND "raw.develop" enum { LARDON3D_RAW_DEVELOPMENT_TASK_KIND_VERSION = 1 }; +#define LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND "raw.develop.batch" +enum { LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION = 1 }; /* Create one durable, initially unqueued S3-B1 Task for the exact existing * Capture-owned SOURCE RAW asset. The two IDs are immutable and are never @@ -37,6 +39,25 @@ bool lardon3d_raw_development_task_reconstruct( const Lardon3DTaskDurableSnapshot *snapshot, void *context, Lardon3DTaskKindBinding *binding); +/* Create one durable, initially unqueued parent Task for the exact immutable + * selected execution. The selected execution, not Task identity, owns item + * order, explicit Capture/SOURCE RAW identity, and the durable cursor. Queue + * admission bounds each parallel window; the caller owns the returned Task. */ +Lardon3DTask *lardon3d_project_create_raw_development_batch_task( + Lardon3DAppState *state, uint64_t selected_execution_id, uint64_t *task_id); + +/* Create and transfer the parent Task to state's existing sole Queue. The + * Queue/Governor reservation covers only one bounded window at a time and is + * released/re-established at each sequence boundary. */ +bool lardon3d_project_enqueue_raw_development_batch( + Lardon3DAppState *state, uint64_t selected_execution_id, uint64_t *task_id); + +/* Reconstruct only from the validated v24 Task -> selected execution row. + * binding owns its returned context until the runtime invokes its destroyer. */ +bool lardon3d_raw_development_batch_task_reconstruct( + const Lardon3DTaskDurableSnapshot *snapshot, void *context, + Lardon3DTaskKindBinding *binding); + #ifdef __cplusplus } #endif diff --git a/src/feature_extractor_opencv.cpp b/src/feature_extractor_opencv.cpp index 463ac6e..6d243f0 100644 --- a/src/feature_extractor_opencv.cpp +++ b/src/feature_extractor_opencv.cpp @@ -26,6 +26,13 @@ extern "C" bool lardon3d_feature_opencv_configure_threads(unsigned int threads) const char *forced = std::getenv( "LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS"); if (forced) { + if (std::strcmp(forced, "once") == 0) { + /* The Task's admitted count is deliberately host-derived. Consume the + * one-shot before failing so the rollback can restore the captured + * process-wide value without a test-side guess of that count. */ + (void)unsetenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS"); + return false; + } char *end = nullptr; unsigned long parsed = std::strtoul(forced, &end, 10); if (end && *end == '\0' && parsed == threads) { diff --git a/src/feature_task.c b/src/feature_task.c index 8d08c18..c746b6e 100644 --- a/src/feature_task.c +++ b/src/feature_task.c @@ -2,6 +2,7 @@ #include #include #include +#include #include #include #include @@ -26,6 +27,18 @@ typedef struct { Lardon3DProjectDbFeatureExtractTask parameters; } Lardon3DFeatureTaskContext; +enum { + FEATURE_BATCH_MAX = 12, + FEATURE_BATCH_CHILD_STACK = 1024 * 1024, +}; + +typedef struct { + char project_path[PATH_MAX]; + Lardon3DProjectDb *database; + Lardon3DResourceGovernor *governor; + Lardon3DProjectDbFeatureExtractBatchTask parameters; +} Lardon3DFeatureBatchTaskContext; + static void destroy_context(void *userdata) { free(userdata); } static void runtime_state(const Lardon3DFeatureTaskContext *context, Lardon3DAppState *state) { @@ -129,6 +142,247 @@ static bool load_validated_source(const Lardon3DFeatureTaskContext *context, cha return file_hash(path, actual) && memcmp(actual, asset.sha256, 32) == 0; } +static void batch_runtime_state(const Lardon3DFeatureBatchTaskContext *context, + Lardon3DAppState *state) { + lardon3d_app_state_init(state); + state->project_loaded = true; + state->project_db = context->database; + state->resource_governor = context->governor; + (void)snprintf(state->project_path, sizeof(state->project_path), "%s", context->project_path); +} + +static bool batch_checkpoint(Lardon3DFeatureBatchTaskContext *context, Lardon3DTask *task, + uint32_t cursor) { + Lardon3DAppState state; + batch_runtime_state(context, &state); + Lardon3DProjectDbFeatureExtractBatchTask parameters = context->parameters; + parameters.task_id = lardon3d_task_id(task); + parameters.next_item_index = cursor; + return lardon3d_project_checkpoint_feature_extract_batch_task(&state, task, ¶meters) == + LARDON3D_PROJECT_TASK_CHECKPOINT_OK; +} + +typedef struct { + Lardon3DFeatureBatchTaskContext *context; + uint32_t item_index; + uint64_t image_id; + char source_path[PATH_MAX]; + bool already_ready; + Lardon3DFeatureExtractResult result; + Lardon3DExtractedFeatures features; +} FeatureBatchWorker; + +static void *extract_batch_item(void *value) { + FeatureBatchWorker *worker = value; + if (worker->already_ready) return NULL; + Lardon3DFeatureExtractorParameters parameters = { + worker->context->parameters.max_features, + worker->context->parameters.pyramid_levels, + worker->context->parameters.fast_threshold}; + worker->result = lardon3d_feature_extract_orb(worker->source_path, ¶meters, + &worker->features); + return NULL; +} + +typedef struct { + FeatureBatchWorker *workers; + size_t count; + size_t first; + size_t stride; +} FeatureBatchParticipant; + +static void *extract_batch_participant(void *value) { + FeatureBatchParticipant *participant = value; + for (size_t index = participant->first; index < participant->count; + index += participant->stride) + (void)extract_batch_item(&participant->workers[index]); + return NULL; +} + +static bool extract_batch_window(FeatureBatchWorker *workers, size_t count, + unsigned int admitted_threads) { + if (!workers || count == 0 || count > FEATURE_BATCH_MAX || admitted_threads == 0) return false; + size_t participants = count < admitted_threads ? count : admitted_threads; + pthread_t children[FEATURE_BATCH_MAX - 1]; + FeatureBatchParticipant work[FEATURE_BATCH_MAX]; + pthread_attr_t attributes; + if (pthread_attr_init(&attributes) != 0) return false; + if (pthread_attr_setstacksize(&attributes, FEATURE_BATCH_CHILD_STACK) != 0) { + (void)pthread_attr_destroy(&attributes); + return false; + } + size_t created = 0; + bool started = true; + for (size_t participant = 1; participant < participants; ++participant) { + work[participant] = (FeatureBatchParticipant){workers, count, participant, participants}; + /* Participants own only disjoint extraction outputs. SQLite, Feature Store + publication, cursor advancement, and generic checkpoints remain owner-only. */ + if (pthread_create(&children[created], &attributes, extract_batch_participant, + &work[participant]) != 0) { + started = false; + break; + } + ++created; + } + if (pthread_attr_destroy(&attributes) != 0) started = false; + work[0] = (FeatureBatchParticipant){workers, count, 0, participants}; + if (started) (void)extract_batch_participant(&work[0]); + for (size_t index = 0; index < created; ++index) + if (pthread_join(children[index], NULL) != 0) started = false; + return started; +} + +static void destroy_batch_outputs(FeatureBatchWorker *workers, size_t count) { + if (!workers) return; + /* A joined window owns every participant output until ordered owner + publication consumes it. Any boundary failure must release the whole + window; the Governor reservation must not outlive abandoned ORB buffers. */ + for (size_t index = 0; index < count; ++index) + lardon3d_extracted_features_destroy(&workers[index].features); +} + +static bool run_batch_body(Lardon3DTask *task, Lardon3DFeatureBatchTaskContext *context) { + for (;;) { + if (!lardon3d_task_checkpoint(task)) return false; + Lardon3DProjectDbFeatureExtractBatchTask persisted; + Lardon3DProjectDbSelectedExecution execution; + if (lardon3d_project_db_load_feature_extract_batch_task( + context->database, lardon3d_task_id(task), &persisted) != LARDON3D_PROJECT_DB_OK || + memcmp(&persisted.parameter_fingerprint, &context->parameters.parameter_fingerprint, + 32) != 0 || + lardon3d_project_db_load_selected_execution(context->database, + context->parameters.selected_execution_id, &execution) != LARDON3D_PROJECT_DB_OK) + return lardon3d_task_fail(task, "État durable Feature batch invalide."); + if (persisted.next_item_index == execution.item_count) + return lardon3d_task_set_durable_progress(task, execution.item_count, execution.item_count, + "Extraction Feature batch terminée.") && + batch_checkpoint(context, task, execution.item_count); + + Lardon3DTaskExecutionContract contract; + if (!lardon3d_task_execution_contract(task, &contract) || contract.cpu_threads == 0 || + contract.batch_size == 0 || contract.batch_size > FEATURE_BATCH_MAX) + return lardon3d_task_fail(task, "Contrat Governor Feature batch invalide."); + size_t count = execution.item_count - persisted.next_item_index; + if (count > contract.batch_size) count = contract.batch_size; + FeatureBatchWorker workers[FEATURE_BATCH_MAX] = {{0}}; + for (size_t offset = 0; offset < count; ++offset) { + FeatureBatchWorker *worker = &workers[offset]; + Lardon3DProjectDbSelectedExecutionItem item; + if (lardon3d_project_db_load_selected_execution_item( + context->database, context->parameters.selected_execution_id, + persisted.next_item_index + (uint32_t)offset, &item) != LARDON3D_PROJECT_DB_OK || + !item.has_image) + return lardon3d_task_fail(task, "Item sélectionné Feature batch invalide."); + worker->context = context; + worker->item_index = item.item_index; + worker->image_id = item.image_id; + Lardon3DFeatureTaskContext single = {0}; + single.database = context->database; + single.parameters.image_id = item.image_id; + (void)snprintf(single.project_path, sizeof(single.project_path), "%s", context->project_path); + Lardon3DProjectDbResult found = find_valid_feature_set( + &single, context->parameters.parameter_fingerprint); + worker->already_ready = found == LARDON3D_PROJECT_DB_OK; + if (found == LARDON3D_PROJECT_DB_CORRUPT || + (found != LARDON3D_PROJECT_DB_OK && found != LARDON3D_PROJECT_DB_NOT_FOUND) || + (!worker->already_ready && !load_validated_source(&single, worker->source_path))) + return lardon3d_task_fail(task, "Source ou Feature Set batch invalide."); + } + + struct timespec begin = {0}, end = {0}; + bool timing_known = clock_gettime(CLOCK_MONOTONIC, &begin) == 0; + if (!extract_batch_window(workers, count, contract.cpu_threads)) { + destroy_batch_outputs(workers, count); + return lardon3d_task_fail(task, "Participants Feature batch impossibles."); + } + if (!lardon3d_task_checkpoint(task)) { + destroy_batch_outputs(workers, count); + return false; + } + size_t durable_items = 0; + Lardon3DAppState state; + batch_runtime_state(context, &state); + Lardon3DFeatureExtractorParameters parameters = { + context->parameters.max_features, context->parameters.pyramid_levels, + context->parameters.fast_threshold}; + for (size_t offset = 0; offset < count; ++offset) { + FeatureBatchWorker *worker = &workers[offset]; + if (!worker->already_ready) { + if (worker->result != LARDON3D_FEATURE_EXTRACT_OK) { + destroy_batch_outputs(workers, count); + return lardon3d_task_fail(task, "Extraction ORB batch impossible."); + } + Lardon3DProjectDbFeatureSet set; + Lardon3DFeatureStoreResult published = lardon3d_feature_store_publish( + &state, worker->image_id, lardon3d_task_id(task), ¶meters, + &worker->features, &set); + lardon3d_extracted_features_destroy(&worker->features); + if (published != LARDON3D_FEATURE_STORE_OK && + published != LARDON3D_FEATURE_STORE_ALREADY_PRESENT) { + destroy_batch_outputs(workers, count); + return lardon3d_task_fail(task, "Publication Feature batch impossible."); + } + if (published == LARDON3D_FEATURE_STORE_OK) ++durable_items; + } + if (lardon3d_project_db_advance_feature_extract_batch_task( + context->database, lardon3d_task_id(task), worker->item_index, + worker->item_index + 1u) != LARDON3D_PROJECT_DB_OK || + !lardon3d_task_set_durable_progress(task, worker->item_index + 1u, + execution.item_count, + "Feature batch publié.") || + !batch_checkpoint(context, task, worker->item_index + 1u)) { + destroy_batch_outputs(workers, count); + return lardon3d_task_fail(task, "Curseur Feature batch impossible."); + } + } + timing_known = timing_known && clock_gettime(CLOCK_MONOTONIC, &end) == 0; + if (timing_known) + (void)lardon3d_task_internal_record_sequence(task, elapsed_ns(begin, end), durable_items); + if (persisted.next_item_index + count == execution.item_count) return true; + Lardon3DResourceReservation *reservation = NULL; + if (!lardon3d_task_sequence_break(task, context->governor, &reservation, &contract)) + return false; + } +} + +static bool run_batch(Lardon3DTask *task, void *userdata) { + Lardon3DFeatureBatchTaskContext *context = userdata; + Lardon3DTaskExecutionContract contract; + if (!lardon3d_task_execution_contract(task, &contract) || contract.cpu_threads == 0) + return lardon3d_task_fail(task, "Contrat CPU Feature batch invalide."); + Lardon3DOpenCvTaskThreadControl control = { + .previous = lardon3d_feature_opencv_thread_count(), + .restore_required = true, + }; + /* Cross-image participants are the admitted CPU dimension. OpenCV stays at + one thread per extraction to prevent nested oversubscription; Queue owns + the sole process-wide guard. Configuration may mutate before verification + fails, so restoration duty exists before configure and covers every exit. */ + if (!lardon3d_feature_opencv_configure_threads(1)) { + (void)lardon3d_opencv_task_threads_end(&control); + return lardon3d_task_fail(task, "Configuration OpenCV Feature batch impossible."); + } + bool result = run_batch_body(task, context); + if (!lardon3d_opencv_task_threads_end(&control)) + return lardon3d_task_fail(task, "Restauration OpenCV Feature batch impossible."); + return result; +} + +#ifdef LARDON3D_FEATURE_TASK_TESTING +bool lardon3d_feature_extract_batch_test_run(Lardon3DTask *task) { + Lardon3DFeatureBatchTaskContext context = {0}; + return run_batch(task, &context); +} +#endif + +static void finished_batch(const Lardon3DTask *task, void *userdata) { + Lardon3DFeatureBatchTaskContext *context = userdata; + Lardon3DProjectDbFeatureExtractBatchTask persisted; + if (lardon3d_project_db_load_feature_extract_batch_task( + context->database, lardon3d_task_id(task), &persisted) == LARDON3D_PROJECT_DB_OK) + (void)batch_checkpoint(context, (Lardon3DTask *)task, persisted.next_item_index); +} + static bool run_body(Lardon3DTask *task, void *userdata, size_t *durable_items) { Lardon3DFeatureTaskContext *context = userdata; *durable_items = 0; @@ -376,3 +630,121 @@ bool lardon3d_project_enqueue_feature_extract(Lardon3DAppState *state, uint64_t } return true; } + +static Lardon3DFeatureBatchTaskContext *make_batch_context( + const Lardon3DTaskReconstructionContext *runtime, + const Lardon3DProjectDbFeatureExtractBatchTask *parameters) { + if (!runtime || !runtime->project_path || !runtime->project_db || + !runtime->resource_governor || !parameters) return NULL; + Lardon3DFeatureBatchTaskContext *context = calloc(1, sizeof(*context)); + if (!context) return NULL; + int n = snprintf(context->project_path, sizeof(context->project_path), "%s", + runtime->project_path); + if (n <= 0 || (size_t)n >= sizeof(context->project_path)) { + free(context); + return NULL; + } + context->database = runtime->project_db; + context->governor = runtime->resource_governor; + context->parameters = *parameters; + return context; +} + +bool lardon3d_feature_extract_batch_reconstruct( + const Lardon3DTaskDurableSnapshot *snapshot, void *userdata, + Lardon3DTaskKindBinding *binding) { + Lardon3DTaskReconstructionContext *runtime = userdata; + if (!snapshot || !runtime || !binding) return false; + Lardon3DProjectDbFeatureExtractBatchTask parameters; + if (lardon3d_project_db_load_feature_extract_batch_task( + runtime->project_db, snapshot->id, ¶meters) != LARDON3D_PROJECT_DB_OK) + return false; + unsigned char expected[32]; + lardon3d_feature_extractor_parameter_fingerprint( + &(Lardon3DFeatureExtractorParameters){parameters.max_features, + parameters.pyramid_levels, + parameters.fast_threshold}, expected); + if (memcmp(expected, parameters.parameter_fingerprint, 32) != 0) return false; + Lardon3DFeatureBatchTaskContext *context = make_batch_context(runtime, ¶meters); + if (!context) return false; + *binding = (Lardon3DTaskKindBinding){.callback = run_batch, + .userdata = context, + .userdata_destroy = destroy_context, + .finished_callback = finished_batch, + .finished_userdata = context}; + return true; +} + +Lardon3DTask *lardon3d_project_create_feature_extract_batch_task( + Lardon3DAppState *state, uint64_t selected_execution_id, + const Lardon3DFeatureExtractorParameters *parameters, uint64_t *task_id) { + if (task_id) *task_id = 0; + if (!state || !state->project_loaded || !state->project_db || + !state->resource_governor || !parameters || !task_id || + !lardon3d_feature_extractor_parameters_valid(parameters) || selected_execution_id == 0) + return NULL; + Lardon3DProjectDbSelectedExecution execution; + if (lardon3d_project_db_load_selected_execution(state->project_db, selected_execution_id, + &execution) != LARDON3D_PROJECT_DB_OK || + execution.stage < LARDON3D_SELECTED_EXECUTION_CALIBRATION || execution.item_count == 0) + return NULL; + uint64_t id = 0; + if (lardon3d_project_db_allocate_task_id(state->project_db, &id) != LARDON3D_PROJECT_DB_OK) + return NULL; + Lardon3DProjectDbFeatureExtractBatchTask durable = { + .task_id = id, + .selected_execution_id = selected_execution_id, + .next_item_index = 0, + .extractor_version = LARDON3D_FEATURE_EXTRACTOR_VERSION, + .max_features = parameters->max_features, + .pyramid_levels = parameters->pyramid_levels, + .fast_threshold = parameters->fast_threshold}; + (void)snprintf(durable.extractor_kind, sizeof(durable.extractor_kind), "%s", + LARDON3D_FEATURE_EXTRACTOR_KIND); + lardon3d_feature_extractor_parameter_fingerprint(parameters, + durable.parameter_fingerprint); + Lardon3DTaskReconstructionContext runtime = { + state->project_path, state->project_db, state->resource_governor, + state->orb_vulkan_backend}; + Lardon3DFeatureBatchTaskContext *context = make_batch_context(&runtime, &durable); + if (!context) return NULL; + /* One admitted participant owns decode, ORB pyramid/keypoints/descriptors, + and publication staging bounded by the existing conservative 512 MiB/item. + The 1..12 ceiling is operational benchmark capacity, not selected-domain + cardinality; Governor feedback records each joined window's useful items. */ + Lardon3DResourceEstimate estimate = { + .memory_fixed_bytes = 64ULL * 1024 * 1024, + .memory_bytes_per_item = 512ULL * 1024 * 1024, + .minimum_batch_size = 1, + .maximum_batch_size = FEATURE_BATCH_MAX, + .desired_cpu_threads = FEATURE_BATCH_MAX, + .desired_io_slots = 1, + .task_class = LARDON3D_RESOURCE_TASK_CPU}; + Lardon3DTask *task = lardon3d_task_create_typed( + "Extraction Feature sélectionnée", &estimate, + LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND, + LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION, run_batch, context, + destroy_context); + if (!task || !lardon3d_task_assign_id(task, id) || + !lardon3d_task_set_finished_callback(task, finished_batch, context) || + !batch_checkpoint(context, task, 0)) { + lardon3d_task_destroy(task); + return NULL; + } + *task_id = id; + return task; +} + +bool lardon3d_project_enqueue_feature_extract_batch( + Lardon3DAppState *state, uint64_t selected_execution_id, + const Lardon3DFeatureExtractorParameters *parameters, uint64_t *task_id) { + if (!state || !state->task_queue) return false; + Lardon3DTask *task = lardon3d_project_create_feature_extract_batch_task( + state, selected_execution_id, parameters, task_id); + if (!task) return false; + if (!lardon3d_task_queue_add(state->task_queue, task, NULL)) { + lardon3d_task_destroy(task); + return false; + } + return true; +} diff --git a/src/project.c b/src/project.c index ea3116a..7186bd5 100644 --- a/src/project.c +++ b/src/project.c @@ -717,7 +717,9 @@ checkpoint_task_internal(Lardon3DAppState *state, const Lardon3DTask *task, const Lardon3DProjectDbGeometricVerifierTask *geometric_parameters, const Lardon3DProjectDbAcquisitionCampaignTask *campaign_parameters, const Lardon3DProjectDbPhotoQualityTask *quality_parameters, - const Lardon3DProjectDbRawDevelopmentTask *raw_parameters) { + const Lardon3DProjectDbRawDevelopmentTask *raw_parameters, + const Lardon3DProjectDbRawDevelopmentBatchTask *raw_batch_parameters, + const Lardon3DProjectDbFeatureExtractBatchTask *feature_batch_parameters) { if (!state || !state->project_loaded || !state->project_db) { return LARDON3D_PROJECT_TASK_CHECKPOINT_NO_PROJECT; } @@ -813,6 +815,14 @@ checkpoint_task_internal(Lardon3DAppState *state, const Lardon3DTask *task, ? lardon3d_project_db_record_raw_development_task( state->project_db, &snapshot, task_kind, task_kind_version, &checkpoint, raw_parameters, now.tv_sec) + : raw_batch_parameters + ? lardon3d_project_db_record_raw_development_batch_task( + state->project_db, &snapshot, task_kind, task_kind_version, &checkpoint, + raw_batch_parameters, now.tv_sec) + : feature_batch_parameters + ? lardon3d_project_db_record_feature_extract_batch_task( + state->project_db, &snapshot, task_kind, task_kind_version, &checkpoint, + feature_batch_parameters, now.tv_sec) : lardon3d_project_db_record_task(state->project_db, &snapshot, task_kind, task_kind_version, &checkpoint, now.tv_sec); if (recorded == LARDON3D_PROJECT_DB_BUSY) { @@ -841,7 +851,7 @@ checkpoint_task_internal(Lardon3DAppState *state, const Lardon3DTask *task, Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_task(Lardon3DAppState *state, const Lardon3DTask *task) { return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, - NULL, NULL); + NULL, NULL, NULL, NULL); } Lardon3DProjectTaskCheckpointResult @@ -851,7 +861,8 @@ lardon3d_project_checkpoint_image_import_task(Lardon3DAppState *state, const Lar return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; } return checkpoint_task_internal( - state, task, source_path, scanset_id, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL); + state, task, source_path, scanset_id, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, + NULL, NULL); } Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_acquisition_campaign_task( @@ -859,7 +870,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_acquisition_camp const Lardon3DProjectDbAcquisitionCampaignTask *parameters) { if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, - parameters, NULL, NULL); + parameters, NULL, NULL, NULL, NULL); } Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_photo_quality_task( @@ -867,7 +878,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_photo_quality_ta const Lardon3DProjectDbPhotoQualityTask *parameters) { if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, - parameters, NULL); + parameters, NULL, NULL, NULL); } Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_raw_development_task( @@ -875,7 +886,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_raw_development_ const Lardon3DProjectDbRawDevelopmentTask *parameters) { if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, - NULL, parameters); + NULL, parameters, NULL, NULL); } Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_task( @@ -885,7 +896,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_ return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; } return checkpoint_task_internal(state, task, NULL, 0, parameters, NULL, NULL, NULL, NULL, NULL, - NULL, NULL, NULL); + NULL, NULL, NULL, NULL, NULL); } Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_sift_extract_task( @@ -893,7 +904,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_sift_extract_tas const Lardon3DProjectDbSiftExtractTask *parameters) { if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; return checkpoint_task_internal(state, task, NULL, 0, NULL, parameters, NULL, NULL, NULL, NULL, - NULL, NULL, NULL); + NULL, NULL, NULL, NULL, NULL); } Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_visual_index_update_task( @@ -903,7 +914,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_visual_index_upd return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; } return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, parameters, NULL, NULL, NULL, - NULL, NULL, NULL); + NULL, NULL, NULL, NULL, NULL); } Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_candidate_pair_generate_task( @@ -913,7 +924,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_candidate_pair_g return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; } return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, parameters, NULL, NULL, - NULL, NULL, NULL); + NULL, NULL, NULL, NULL, NULL); } Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_matcher_task( @@ -923,7 +934,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_matcher_task( return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; } return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, parameters, NULL, - NULL, NULL, NULL); + NULL, NULL, NULL, NULL, NULL); } Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_geometric_verifier_task( @@ -933,7 +944,23 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_geometric_verifi return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; } return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, - parameters, NULL, NULL, NULL); + parameters, NULL, NULL, NULL, NULL, NULL); +} + +Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_raw_development_batch_task( + Lardon3DAppState *state, const Lardon3DTask *task, + const Lardon3DProjectDbRawDevelopmentBatchTask *parameters) { + if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; + return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, + NULL, NULL, parameters, NULL); +} + +Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_batch_task( + Lardon3DAppState *state, const Lardon3DTask *task, + const Lardon3DProjectDbFeatureExtractBatchTask *parameters) { + if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; + return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, + NULL, NULL, NULL, parameters); } static bool coherent_recovery(const Lardon3DProjectDbTask *database_task, diff --git a/src/project_db.c b/src/project_db.c index f12220b..ce4cc6f 100644 --- a/src/project_db.c +++ b/src/project_db.c @@ -12,6 +12,7 @@ #include #include +#include #include #include #include @@ -851,6 +852,36 @@ static const char schema_optical_calibrations_v23[] = "FOREIGN KEY(calibration_profile_id,optical_configuration_id) REFERENCES " "optical_calibration_profiles(calibration_profile_id,optical_configuration_id));"; +/* PERSISTENCE CONTRACT v24: this additive row stores only the operational + Task -> selected-execution association. Selected execution items/cursor stay + authoritative, and migration creates no rows or inferred identities. */ +static const char schema_raw_development_batch_v24[] = + "CREATE TABLE raw_development_batch_tasks(" + "task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE CASCADE," + "selected_execution_id INTEGER NOT NULL UNIQUE REFERENCES " + "selected_executions(execution_id));"; + +/* PERSISTENCE CONTRACT v25: this operational row anchors one owner to the + immutable selected-execution order and exact ORB parameter domain. Its + monotone cursor is a durable prefix only; migration creates no rows. */ +static const char schema_feature_extract_batch_v25[] = + "CREATE TABLE feature_extract_batch_tasks(" + "task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE CASCADE," + "selected_execution_id INTEGER NOT NULL UNIQUE REFERENCES selected_executions(execution_id)," + "next_item_index INTEGER NOT NULL CHECK(typeof(next_item_index)='integer' AND " + "next_item_index>=0 AND next_item_index<=4096)," + "extractor_kind TEXT NOT NULL CHECK(typeof(extractor_kind)='text' AND extractor_kind='orb')," + "extractor_version INTEGER NOT NULL CHECK(typeof(extractor_version)='integer' AND " + "extractor_version=1)," + "max_features INTEGER NOT NULL CHECK(typeof(max_features)='integer' AND max_features>0 AND " + "max_features<=8192)," + "pyramid_levels INTEGER NOT NULL CHECK(typeof(pyramid_levels)='integer' AND " + "pyramid_levels>0 AND pyramid_levels<=16)," + "fast_threshold INTEGER NOT NULL CHECK(typeof(fast_threshold)='integer' AND " + "fast_threshold>0 AND fast_threshold<=255)," + "parameter_fingerprint BLOB NOT NULL CHECK(typeof(parameter_fingerprint)='blob' AND " + "length(parameter_fingerprint)=32));"; + static void copy_error(char destination[LARDON3D_PROJECT_DB_ERROR_CAPACITY], const char *text) { if (destination) { (void)snprintf(destination, LARDON3D_PROJECT_DB_ERROR_CAPACITY, "%s", text ? text : ""); @@ -943,7 +974,8 @@ static Lardon3DProjectDbResult migrate(Lardon3DProjectDb *database, unsigned int from_version != 8 && from_version != 9 && from_version != 10 && from_version != 11 && from_version != 12 && from_version != 13 && from_version != 14 && from_version != 15 && from_version != 16 && from_version != 17 && from_version != 18 && - from_version != 19 && from_version != 20 && from_version != 21 && from_version != 22) { + from_version != 19 && from_version != 20 && from_version != 21 && from_version != 22 && + from_version != 23 && from_version != 24) { return LARDON3D_PROJECT_DB_CORRUPT; } Lardon3DProjectDbResult result = execute(database, "BEGIN IMMEDIATE", "begin migration"); @@ -1414,6 +1446,46 @@ static Lardon3DProjectDbResult migrate(Lardon3DProjectDb *database, unsigned int result = LARDON3D_PROJECT_DB_CORRUPT; } } + if (result == LARDON3D_PROJECT_DB_OK && from_version < 24) { + /* v23 contains no evidence that a batch Task existed, so migration is + DDL-only and cannot synthesize selected execution associations. */ + result = execute(database, schema_raw_development_batch_v24, + "migrate RAW development batch v23 to v24"); +#ifdef LARDON3D_PROJECT_DB_TESTING + if (result == LARDON3D_PROJECT_DB_OK && + getenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V24")) { + result = execute(database, "INSERT INTO missing_v24_test_table VALUES(1)", + "forced migration v24 failure"); + } +#endif + if (result == LARDON3D_PROJECT_DB_OK) { + result = execute(database, + "UPDATE metadata SET value=24 WHERE key='schema_version' AND value=23", + "finish schema v24 migration"); + if (result == LARDON3D_PROJECT_DB_OK && + sqlite3_changes(database->connection) != 1) + result = LARDON3D_PROJECT_DB_CORRUPT; + } + } + if (result == LARDON3D_PROJECT_DB_OK && from_version < 25) { + /* No v24 row identifies a Feature batch or cursor. Keep this DDL-only so + historical single-image Tasks remain exactly as persisted. */ + result = execute(database, schema_feature_extract_batch_v25, + "migrate feature extraction batch v24 to v25"); +#ifdef LARDON3D_PROJECT_DB_TESTING + if (result == LARDON3D_PROJECT_DB_OK && + getenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V25")) + result = execute(database, "INSERT INTO missing_v25_test_table VALUES(1)", + "forced migration v25 failure"); +#endif + if (result == LARDON3D_PROJECT_DB_OK) { + result = execute(database, + "UPDATE metadata SET value=25 WHERE key='schema_version' AND value=24", + "finish schema v25 migration"); + if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1) + result = LARDON3D_PROJECT_DB_CORRUPT; + } + } if (result == LARDON3D_PROJECT_DB_OK) { result = execute(database, "COMMIT", "commit migration"); } @@ -1550,6 +1622,8 @@ Lardon3DProjectDbResult lardon3d_project_db_open(const char *path, Lardon3DProje "capture_assets", "capture_source_assets", "raw_development_tasks", + "raw_development_batch_tasks", + "feature_extract_batch_tasks", "capture_selections", "asset_derivations", "acquisition_campaign_tasks", @@ -1821,6 +1895,8 @@ record_task_internal(Lardon3DProjectDb *database, const Lardon3DTaskDurableSnaps const Lardon3DProjectDbAcquisitionCampaignTask *campaign, const Lardon3DProjectDbPhotoQualityTask *photo_quality, const Lardon3DProjectDbRawDevelopmentTask *raw_development, + const Lardon3DProjectDbRawDevelopmentBatchTask *raw_batch, + const Lardon3DProjectDbFeatureExtractBatchTask *feature_batch, int64_t updated_at) { bool typed = task_kind != NULL; if (!database || !valid_durable_task(snapshot, updated_at) || @@ -1924,6 +2000,21 @@ record_task_internal(Lardon3DProjectDb *database, const Lardon3DTaskDurableSnaps (raw_development->has_image || raw_development->image_id != 0)) || (raw_development->phase == LARDON3D_RAW_DEVELOPMENT_TASK_PUBLISHED && (!raw_development->has_image || !valid_task_id(raw_development->image_id))))) || + (raw_batch && + (!task_kind || strcmp(task_kind, LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND) != 0 || + task_kind_version != LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION || + !valid_task_id(raw_batch->task_id) || raw_batch->task_id != snapshot->id || + !valid_task_id(raw_batch->selected_execution_id))) || + (feature_batch && + (!task_kind || strcmp(task_kind, LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND) != 0 || + task_kind_version != LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION || + !valid_task_id(feature_batch->task_id) || feature_batch->task_id != snapshot->id || + !valid_task_id(feature_batch->selected_execution_id) || + strcmp(feature_batch->extractor_kind, LARDON3D_FEATURE_EXTRACTOR_KIND) != 0 || + feature_batch->extractor_version != LARDON3D_FEATURE_EXTRACTOR_VERSION || + feature_batch->max_features == 0 || feature_batch->max_features > 8192 || + feature_batch->pyramid_levels == 0 || feature_batch->pyramid_levels > 16 || + feature_batch->fast_threshold == 0 || feature_batch->fast_threshold > 255)) || (checkpoint && !valid_checkpoint(checkpoint))) { return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; } @@ -2064,6 +2155,64 @@ record_task_internal(Lardon3DProjectDb *database, const Lardon3DTaskDurableSnaps } } } + if (result == LARDON3D_PROJECT_DB_OK && raw_batch) { + /* The selected execution owns every scientific item and its cursor. This + immutable row only makes the parent operational Task reconstructible. */ + result = prepare(database, + "INSERT INTO raw_development_batch_tasks(task_id,selected_execution_id) " + "VALUES(?1,?2) ON CONFLICT(task_id) DO UPDATE SET " + "selected_execution_id=excluded.selected_execution_id WHERE " + "raw_development_batch_tasks.selected_execution_id=excluded.selected_execution_id", + &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + sqlite3_bind_int64(statement, 1, (sqlite3_int64)raw_batch->task_id); + sqlite3_bind_int64(statement, 2, + (sqlite3_int64)raw_batch->selected_execution_id); + result = step_done(database, statement, "upsert RAW development batch task"); + if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1) { + copy_error(database->error, "Exécution sélectionnée RAW batch immuable."); + result = LARDON3D_PROJECT_DB_CONSTRAINT; + } + } + } + if (result == LARDON3D_PROJECT_DB_OK && feature_batch) { + /* Immutable domain fields never change. Only an equal-or-newer cursor may + be repeated by a generic checkpoint that lagged an item publication. */ + result = prepare(database, + "INSERT INTO feature_extract_batch_tasks(task_id,selected_execution_id," + "next_item_index,extractor_kind,extractor_version,max_features,pyramid_levels," + "fast_threshold,parameter_fingerprint) SELECT ?1,?2,?3,?4,?5,?6,?7,?8,?9 " + "FROM selected_executions e WHERE e.execution_id=?2 AND e.item_count>0 AND " + "?3<=e.item_count AND e.stage>=?10 " + "ON CONFLICT(task_id) DO UPDATE SET next_item_index=MAX(" + "feature_extract_batch_tasks.next_item_index,excluded.next_item_index) WHERE " + "feature_extract_batch_tasks.selected_execution_id=excluded.selected_execution_id AND " + "feature_extract_batch_tasks.extractor_kind=excluded.extractor_kind AND " + "feature_extract_batch_tasks.extractor_version=excluded.extractor_version AND " + "feature_extract_batch_tasks.max_features=excluded.max_features AND " + "feature_extract_batch_tasks.pyramid_levels=excluded.pyramid_levels AND " + "feature_extract_batch_tasks.fast_threshold=excluded.fast_threshold AND " + "feature_extract_batch_tasks.parameter_fingerprint=excluded.parameter_fingerprint AND " + "excluded.next_item_index<=(SELECT item_count FROM selected_executions WHERE " + "execution_id=excluded.selected_execution_id)", &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + sqlite3_bind_int64(statement, 1, (sqlite3_int64)feature_batch->task_id); + sqlite3_bind_int64(statement, 2, (sqlite3_int64)feature_batch->selected_execution_id); + sqlite3_bind_int64(statement, 3, feature_batch->next_item_index); + sqlite3_bind_text(statement, 4, feature_batch->extractor_kind, -1, SQLITE_TRANSIENT); + sqlite3_bind_int64(statement, 5, feature_batch->extractor_version); + sqlite3_bind_int64(statement, 6, feature_batch->max_features); + sqlite3_bind_int64(statement, 7, feature_batch->pyramid_levels); + sqlite3_bind_int64(statement, 8, feature_batch->fast_threshold); + sqlite3_bind_blob(statement, 9, feature_batch->parameter_fingerprint, 32, SQLITE_TRANSIENT); + sqlite3_bind_int64(statement, 10, LARDON3D_SELECTED_EXECUTION_CALIBRATION); + result = step_done(database, statement, "upsert feature extraction batch task"); + if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1) { + copy_error(database->error, "Domaine ou curseur Feature batch incohérent."); + result = LARDON3D_PROJECT_DB_CONSTRAINT; + } + } + } if (result == LARDON3D_PROJECT_DB_OK) { result = prepare(database, @@ -2398,7 +2547,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_task( Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind, uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint, int64_t updated_at) { return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0, - NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, updated_at); + NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, + NULL, updated_at); } Lardon3DProjectDbResult lardon3d_project_db_record_image_import_task( @@ -2410,7 +2560,7 @@ Lardon3DProjectDbResult lardon3d_project_db_record_image_import_task( } return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, source_path, scanset_id, NULL, NULL, NULL, NULL, NULL, NULL, - NULL, NULL, NULL, NULL, updated_at); + NULL, NULL, NULL, NULL, NULL, NULL, updated_at); } Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_task( @@ -2421,8 +2571,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_task( return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; } return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0, - parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, - updated_at); + parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, + NULL, updated_at); } Lardon3DProjectDbResult lardon3d_project_db_record_sift_extract_task( @@ -2432,8 +2582,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_sift_extract_task( const Lardon3DProjectDbSiftExtractTask *parameters, int64_t updated_at) { if (!parameters) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0, - NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, - updated_at); + NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, + NULL, updated_at); } static bool read_task(sqlite3_stmt *statement, Lardon3DProjectDbTask *task) { @@ -2680,7 +2830,7 @@ Lardon3DProjectDbResult lardon3d_project_db_record_acquisition_campaign_task( never observes a valid campaign Task without its immutable request blob. */ return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, parameters, NULL, - NULL, updated_at); + NULL, NULL, NULL, updated_at); } static Lardon3DProjectDbResult validate_acquisition_campaign_prefix_locked( @@ -3397,8 +3547,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_photo_quality_task( /* Generic runtime state and typed immutable request share the transaction; recovery cannot observe one without the other. */ return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0, - NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, parameters, NULL, - updated_at); + NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, parameters, NULL, NULL, + NULL, updated_at); } Lardon3DProjectDbResult lardon3d_project_db_load_photo_quality_task( @@ -3470,8 +3620,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_raw_development_task( const Lardon3DProjectDbRawDevelopmentTask *parameters, int64_t updated_at) { if (!parameters) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0, - NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, parameters, - updated_at); + NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, parameters, NULL, + NULL, updated_at); } Lardon3DProjectDbResult lardon3d_project_db_load_raw_development_task( @@ -3537,6 +3687,61 @@ Lardon3DProjectDbResult lardon3d_project_db_load_raw_development_task( return result; } +Lardon3DProjectDbResult lardon3d_project_db_record_raw_development_batch_task( + Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, + const char *task_kind, uint32_t task_kind_version, + const Lardon3DProjectDbCheckpoint *checkpoint, + const Lardon3DProjectDbRawDevelopmentBatchTask *parameters, int64_t updated_at) { + if (!parameters) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + return record_task_internal(database, snapshot, task_kind, task_kind_version, + checkpoint, NULL, 0, NULL, NULL, NULL, NULL, NULL, + NULL, NULL, NULL, NULL, NULL, parameters, NULL, updated_at); +} + +Lardon3DProjectDbResult lardon3d_project_db_load_raw_development_batch_task( + Lardon3DProjectDb *database, uint64_t task_id, + Lardon3DProjectDbRawDevelopmentBatchTask *parameters) { + if (!database || !valid_task_id(task_id) || !parameters) + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + memset(parameters, 0, sizeof(*parameters)); + (void)pthread_mutex_lock(&database->mutex); + sqlite3_stmt *statement = NULL; + Lardon3DProjectDbResult result = prepare( + database, + "SELECT b.selected_execution_id,t.task_kind,t.task_kind_version,e.execution_id " + "FROM raw_development_batch_tasks b JOIN tasks t ON t.task_id=b.task_id " + "LEFT JOIN selected_executions e ON e.execution_id=b.selected_execution_id " + "WHERE b.task_id=?1", &statement); + if (result == LARDON3D_PROJECT_DB_OK) { + sqlite3_bind_int64(statement, 1, (sqlite3_int64)task_id); + int code = sqlite3_step(statement); + if (code == SQLITE_DONE) { + result = LARDON3D_PROJECT_DB_NOT_FOUND; + } else if (code != SQLITE_ROW) { + result = sqlite_result(database, code, "load RAW development batch task"); + } else { + sqlite3_int64 execution = sqlite3_column_int64(statement, 0); + sqlite3_int64 version = sqlite3_column_int64(statement, 2); + if (sqlite3_column_type(statement, 0) != SQLITE_INTEGER || execution <= 0 || + sqlite3_column_type(statement, 1) != SQLITE_TEXT || + sqlite3_column_type(statement, 2) != SQLITE_INTEGER || + sqlite3_column_type(statement, 3) != SQLITE_INTEGER || + sqlite3_column_int64(statement, 3) != execution || + !column_text_equals(statement, 1, + LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND) || + version != LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION) { + result = LARDON3D_PROJECT_DB_CORRUPT; + } else { + parameters->task_id = task_id; + parameters->selected_execution_id = (uint64_t)execution; + } + } + sqlite3_finalize(statement); + } + (void)pthread_mutex_unlock(&database->mutex); + return result; +} + static bool valid_quality_result(const Lardon3DProjectDbPhotoQualityResult *r) { const Lardon3DPhotoQualityMetrics *m = r ? &r->metrics : NULL; bool decoded_pair = m && ((m->decoded_width == 0) == (m->decoded_height == 0)); @@ -6466,6 +6671,105 @@ lardon3d_project_db_load_feature_extract_task(Lardon3DProjectDb *db, uint64_t ta return result; } +Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_batch_task( + Lardon3DProjectDb *db, const Lardon3DTaskDurableSnapshot *snapshot, + const char *kind, uint32_t version, const Lardon3DProjectDbCheckpoint *checkpoint, + const Lardon3DProjectDbFeatureExtractBatchTask *parameters, int64_t updated_at) { + if (!parameters) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0, + NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, + NULL, NULL, parameters, updated_at); +} + +Lardon3DProjectDbResult lardon3d_project_db_load_feature_extract_batch_task( + Lardon3DProjectDb *db, uint64_t task_id, + Lardon3DProjectDbFeatureExtractBatchTask *p) { + if (!db || !valid_task_id(task_id) || !p) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + memset(p, 0, sizeof(*p)); + pthread_mutex_lock(&db->mutex); + sqlite3_stmt *s = NULL; + Lardon3DProjectDbResult result = prepare( + db, "SELECT b.selected_execution_id,b.next_item_index,b.extractor_kind," + "b.extractor_version,b.max_features,b.pyramid_levels,b.fast_threshold," + "b.parameter_fingerprint,e.item_count,e.stage,t.task_kind,t.task_kind_version " + "FROM feature_extract_batch_tasks b JOIN selected_executions e ON " + "e.execution_id=b.selected_execution_id JOIN tasks t ON t.task_id=b.task_id " + "WHERE b.task_id=?1", &s); + if (result == LARDON3D_PROJECT_DB_OK) { + sqlite3_bind_int64(s, 1, (sqlite3_int64)task_id); + int code = sqlite3_step(s); + if (code == SQLITE_DONE) result = LARDON3D_PROJECT_DB_NOT_FOUND; + else if (code != SQLITE_ROW) result = sqlite_result(db, code, "load feature batch task"); + else { + sqlite3_int64 execution = sqlite3_column_int64(s, 0); + sqlite3_int64 cursor = sqlite3_column_int64(s, 1); + sqlite3_int64 version = sqlite3_column_int64(s, 3); + sqlite3_int64 max = sqlite3_column_int64(s, 4); + sqlite3_int64 levels = sqlite3_column_int64(s, 5); + sqlite3_int64 threshold = sqlite3_column_int64(s, 6); + sqlite3_int64 count = sqlite3_column_int64(s, 8); + sqlite3_int64 stage = sqlite3_column_int64(s, 9); + const void *fp = sqlite3_column_blob(s, 7); + const unsigned char *task_kind = sqlite3_column_text(s, 10); + if (sqlite3_column_type(s, 0) != SQLITE_INTEGER || execution <= 0 || + sqlite3_column_type(s, 1) != SQLITE_INTEGER || cursor < 0 || cursor > count || + !copy_column(s, 2, p->extractor_kind, sizeof(p->extractor_kind)) || + strcmp(p->extractor_kind, LARDON3D_FEATURE_EXTRACTOR_KIND) != 0 || version != 1 || + max <= 0 || max > 8192 || levels <= 0 || levels > 16 || threshold <= 0 || + threshold > 255 || sqlite3_column_type(s, 7) != SQLITE_BLOB || + sqlite3_column_bytes(s, 7) != 32 || !fp || count <= 0 || count > 4096 || + stage < LARDON3D_SELECTED_EXECUTION_CALIBRATION || !task_kind || + strcmp((const char *)task_kind, LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND) != 0 || + sqlite3_column_int64(s, 11) != LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION) { + result = LARDON3D_PROJECT_DB_CORRUPT; + } else { + p->task_id = task_id; + p->selected_execution_id = (uint64_t)execution; + p->next_item_index = (uint32_t)cursor; + p->extractor_version = (uint32_t)version; + p->max_features = (uint32_t)max; + p->pyramid_levels = (uint32_t)levels; + p->fast_threshold = (uint32_t)threshold; + memcpy(p->parameter_fingerprint, fp, 32); + } + } + sqlite3_finalize(s); + } + pthread_mutex_unlock(&db->mutex); + return result; +} + +Lardon3DProjectDbResult lardon3d_project_db_advance_feature_extract_batch_task( + Lardon3DProjectDb *db, uint64_t task_id, uint32_t item_index, + uint32_t next_item_index) { + if (!db || !valid_task_id(task_id) || next_item_index != item_index + 1u) + return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; + pthread_mutex_lock(&db->mutex); + sqlite3_stmt *s = NULL; + Lardon3DProjectDbResult result = execute(db, "BEGIN IMMEDIATE", "begin feature batch advance"); + if (result == LARDON3D_PROJECT_DB_OK) + result = prepare(db, + "UPDATE feature_extract_batch_tasks AS b SET next_item_index=?3 WHERE b.task_id=?1 AND " + "(b.next_item_index=?2 OR b.next_item_index=?3) AND EXISTS(SELECT 1 FROM " + "selected_execution_items i JOIN feature_sets f ON f.image_id=i.image_id JOIN " + "feature_assets a ON a.feature_asset_id=f.feature_asset_id WHERE " + "i.execution_id=b.selected_execution_id AND i.item_index=?2 AND i.image_id IS NOT NULL " + "AND f.extractor_kind=b.extractor_kind AND f.extractor_version=b.extractor_version AND " + "f.parameter_fingerprint=b.parameter_fingerprint AND a.durability=0)", &s); + if (result == LARDON3D_PROJECT_DB_OK) { + sqlite3_bind_int64(s, 1, (sqlite3_int64)task_id); + sqlite3_bind_int64(s, 2, item_index); + sqlite3_bind_int64(s, 3, next_item_index); + result = step_done(db, s, "advance feature batch cursor"); + if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(db->connection) != 1) + result = LARDON3D_PROJECT_DB_CONSTRAINT; + } + if (result == LARDON3D_PROJECT_DB_OK) result = execute(db, "COMMIT", "commit feature batch advance"); + else (void)execute(db, "ROLLBACK", "rollback feature batch advance"); + pthread_mutex_unlock(&db->mutex); + return result; +} + Lardon3DProjectDbResult lardon3d_project_db_load_sift_extract_task( Lardon3DProjectDb *db, uint64_t task_id, Lardon3DProjectDbSiftExtractTask *p) { if (!db || !valid_task_id(task_id) || !p) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; @@ -6965,7 +7269,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_visual_index_update_task( return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; } return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0, NULL, - NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, updated_at); + NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, + NULL, updated_at); } Lardon3DProjectDbResult lardon3d_project_db_load_visual_index_update_task( @@ -7012,7 +7317,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_candidate_pair_generate_task( return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; } return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0, NULL, - NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, updated_at); + NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, + NULL, updated_at); } Lardon3DProjectDbResult lardon3d_project_db_load_candidate_pair_generate_task( @@ -7068,7 +7374,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_matcher_task( return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; } return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0, NULL, - NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL, updated_at); + NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, + NULL, updated_at); } Lardon3DProjectDbResult lardon3d_project_db_load_matcher_task( @@ -7126,7 +7433,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_geometric_verifier_task( return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; } return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0, NULL, - NULL, NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL, updated_at); + NULL, NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL, + NULL, updated_at); } Lardon3DProjectDbResult lardon3d_project_db_load_geometric_verifier_task( @@ -7199,7 +7507,7 @@ Lardon3DProjectDbResult lardon3d_project_db_record_track_builder_task( } Lardon3DProjectDbResult result = record_task_internal( db, snapshot, kind, version, checkpoint, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, - NULL, NULL, NULL, updated_at); + NULL, NULL, NULL, NULL, NULL, updated_at); if (result != LARDON3D_PROJECT_DB_OK) return result; (void)pthread_mutex_lock(&db->mutex); result = execute(db, "BEGIN IMMEDIATE", "begin track builder task"); @@ -7310,7 +7618,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_sparse_sfm_task( const Lardon3DProjectDbSparseSfmTask *parameters, int64_t updated_at) { if (!parameters) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; return record_task_internal(database, snapshot, kind, version, checkpoint, NULL, 0, NULL, NULL, - NULL, NULL, NULL, NULL, parameters, NULL, NULL, NULL, updated_at); + NULL, NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL, + NULL, updated_at); } static bool read_u64_blob(sqlite3_stmt *statement, int column, uint64_t *value) { diff --git a/src/raw_development_task.cpp b/src/raw_development_task.cpp index 43d511f..2d01c14 100644 --- a/src/raw_development_task.cpp +++ b/src/raw_development_task.cpp @@ -8,7 +8,10 @@ extern "C" { #include "opencv_task_thread_guard.h" #include +#include +#include #include +#include namespace { /* One admitted RAW execution owns a conservative allowance for the bounded @@ -17,7 +20,12 @@ namespace { * scientific limit on the number of Captures. The Queue/Governor reservation * exists only for this one Task callback and is released by Task runtime on * every terminal path. */ -constexpr uint64_t kRawWorkingBytes = UINT64_C(2) * 1024u * 1024u * 1024u; +constexpr uint64_t kRawSingleWorkingBytes = UINT64_C(2) * 1024u * 1024u * 1024u; +constexpr size_t kRawBatchWindowMax = 8; +constexpr size_t kRawBatchChildStackBytes = 1024u * 1024u; +constexpr uint64_t kRawBatchParticipantBytes = UINT64_C(896) * 1024u * 1024u; +static_assert(kRawBatchParticipantBytes <= UINT64_MAX / kRawBatchWindowMax, + "RAW batch memory envelope must fit uint64_t"); struct Context { char project_path[LARDON3D_APP_STATE_PATH_CAPACITY]{}; @@ -27,6 +35,13 @@ struct Context { uint64_t source_asset_id{}; }; +struct BatchContext { + char project_path[LARDON3D_APP_STATE_PATH_CAPACITY]{}; + Lardon3DProjectDb *database{}; + Lardon3DResourceGovernor *governor{}; + uint64_t selected_execution_id{}; +}; + void destroy(void *value) { delete static_cast(value); } void runtime(const Context *context, Lardon3DAppState *state) { @@ -129,6 +144,242 @@ Context *make_context(const Lardon3DTaskReconstructionContext *runtime_context, context->source_asset_id = source_asset_id; return context; } + +void destroy_batch(void *value) { delete static_cast(value); } + +BatchContext *make_batch_context( + const Lardon3DTaskReconstructionContext *runtime_context, + uint64_t selected_execution_id) { + if (!runtime_context || !runtime_context->project_path || + !runtime_context->project_path[0] || !runtime_context->project_db || + !runtime_context->resource_governor || selected_execution_id == 0) + return nullptr; + auto *context = new (std::nothrow) BatchContext; + if (!context) return nullptr; + int written = std::snprintf(context->project_path, sizeof(context->project_path), "%s", + runtime_context->project_path); + if (written <= 0 || written >= static_cast(sizeof(context->project_path))) { + delete context; + return nullptr; + } + context->database = runtime_context->project_db; + context->governor = runtime_context->resource_governor; + context->selected_execution_id = selected_execution_id; + return context; +} + +void batch_runtime(const BatchContext *context, Lardon3DAppState *state) { + lardon3d_app_state_init(state); + state->project_loaded = true; + state->project_db = context->database; + state->resource_governor = context->governor; + std::snprintf(state->project_path, sizeof(state->project_path), "%s", + context->project_path); +} + +bool batch_checkpoint(BatchContext *context, Lardon3DTask *task) { + Lardon3DAppState state; + batch_runtime(context, &state); + Lardon3DProjectDbRawDevelopmentBatchTask parameters{ + lardon3d_task_id(task), context->selected_execution_id}; + return lardon3d_project_checkpoint_raw_development_batch_task( + &state, task, ¶meters) == LARDON3D_PROJECT_TASK_CHECKPOINT_OK; +} + +struct BatchWorker { + BatchContext *context{}; + Lardon3DProjectDbSelectedExecutionItem item{}; + uint64_t producer_task_id{}; + Lardon3DRawDevelopmentResult raw_result{LARDON3D_RAW_DEVELOPMENT_INTERNAL_ERROR}; + Lardon3DProjectDbResult db_result{LARDON3D_PROJECT_DB_IO_ERROR}; + uint64_t image_id{}; +}; + +void *develop_batch_item(void *value) { + auto *worker = static_cast(value); + if (worker->item.representation_source == + LARDON3D_SELECTED_REPRESENTATION_SOURCE_IMAGE) { + worker->db_result = lardon3d_project_db_get_selected_capture_image( + worker->context->database, worker->item.capture_id, &worker->image_id); + return nullptr; + } + Lardon3DProjectDbCapture capture{}; + worker->db_result = lardon3d_project_db_load_capture( + worker->context->database, worker->item.capture_id, &capture); + if (worker->db_result != LARDON3D_PROJECT_DB_OK) return nullptr; + Lardon3DAppState state; + batch_runtime(worker->context, &state); + Lardon3DRawDevelopmentOutput output{}; + worker->raw_result = lardon3d_raw_develop_asset_to_capture( + &state, worker->item.capture_id, worker->item.source_asset_id, + worker->producer_task_id, capture.created_at, &output); + if (worker->raw_result == LARDON3D_RAW_DEVELOPMENT_OK) { + worker->image_id = output.image.image_id; + worker->db_result = LARDON3D_PROJECT_DB_OK; + } + return nullptr; +} + +struct BatchParticipant { + BatchWorker *workers{}; + size_t count{}; + size_t first{}; + size_t stride{}; +}; + +void *develop_batch_participant(void *value) { + auto *participant = static_cast(value); + for (size_t index = participant->first; index < participant->count; + index += participant->stride) + (void)develop_batch_item(&participant->workers[index]); + return nullptr; +} + +bool develop_batch_window(BatchContext *context, uint64_t producer_task_id, + const Lardon3DProjectDbSelectedExecutionItem *items, + size_t count, unsigned int admitted_threads, + std::array *workers) { + if (!context || !items || !workers || count == 0 || count > kRawBatchWindowMax || + admitted_threads == 0) + return false; + size_t participants = std::min(count, static_cast(admitted_threads)); + std::array children{}; + std::array participant_work{}; + pthread_attr_t attributes; + if (pthread_attr_init(&attributes) != 0) return false; + if (pthread_attr_setstacksize(&attributes, kRawBatchChildStackBytes) != 0) { + (void)pthread_attr_destroy(&attributes); + return false; + } + for (size_t index = 0; index < count; ++index) { + (*workers)[index].context = context; + (*workers)[index].item = items[index]; + (*workers)[index].producer_task_id = producer_task_id; + } + size_t created = 0; + bool started = true; + for (size_t participant = 1; participant < participants; ++participant) { + participant_work[participant] = { + workers->data(), count, participant, participants}; + /* Each participant processes one disjoint strided subset. No child owns + * selected cursor/progress publication, even when it finishes first. */ + if (pthread_create(&children[created], &attributes, + develop_batch_participant, + &participant_work[participant]) != 0) { + started = false; + break; + } + ++created; + } + if (pthread_attr_destroy(&attributes) != 0) started = false; + participant_work[0] = {workers->data(), count, 0, participants}; + if (started) (void)develop_batch_participant(&participant_work[0]); + for (size_t index = 0; index < created; ++index) { + if (pthread_join(children[index], nullptr) != 0) started = false; + } + return started; +} + +bool run_batch_impl(Lardon3DTask *task, BatchContext *context) { + for (;;) { + if (!lardon3d_task_checkpoint(task)) return false; + Lardon3DProjectDbSelectedExecution execution{}; + if (lardon3d_project_db_load_selected_execution( + context->database, context->selected_execution_id, &execution) != + LARDON3D_PROJECT_DB_OK) + return lardon3d_task_fail(task, "Exécution sélectionnée RAW batch invalide."); + if (execution.next_item_index == execution.item_count) { + return lardon3d_task_set_durable_progress( + task, execution.item_count, execution.item_count, + "Développement RAW batch terminé.") && + batch_checkpoint(context, task); + } + if (execution.stage != LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS) + return lardon3d_task_fail(task, "Étape sélectionnée RAW batch invalide."); + + Lardon3DTaskExecutionContract contract{}; + if (!lardon3d_task_execution_contract(task, &contract) || + contract.batch_size == 0 || contract.batch_size > kRawBatchWindowMax || + contract.cpu_threads == 0) + return lardon3d_task_fail(task, "Contrat Governor RAW batch invalide."); + size_t count = std::min(contract.batch_size, + execution.item_count - execution.next_item_index); + std::array items{}; + for (size_t offset = 0; offset < count; ++offset) { + if (lardon3d_project_db_load_selected_execution_item( + context->database, context->selected_execution_id, + execution.next_item_index + static_cast(offset), + &items[offset]) != LARDON3D_PROJECT_DB_OK) + return lardon3d_task_fail(task, "Item sélectionné RAW batch invalide."); + } + + std::array workers{}; + /* Each independent LibRaw/BGR/PNG participant fits the admitted 896 MiB + * per-item envelope. Every child is joined before cursor publication or + * reservation release. */ + if (!develop_batch_window(context, lardon3d_task_id(task), items.data(), count, + contract.cpu_threads, &workers)) + return lardon3d_task_fail(task, "Workers RAW batch impossibles."); + if (!lardon3d_task_checkpoint(task)) return false; + + for (size_t offset = 0; offset < count; ++offset) { + BatchWorker &worker = workers[offset]; + if (worker.db_result != LARDON3D_PROJECT_DB_OK || worker.image_id == 0 || + (worker.item.representation_source == + LARDON3D_SELECTED_REPRESENTATION_RAW_ASSET && + worker.raw_result != LARDON3D_RAW_DEVELOPMENT_OK)) + return lardon3d_task_fail(task, "Développement d'un item RAW batch impossible."); + uint32_t item_index = worker.item.item_index; + if (lardon3d_project_db_record_selected_representation( + context->database, context->selected_execution_id, item_index, + worker.image_id, item_index + 1u) != LARDON3D_PROJECT_DB_OK) + return lardon3d_task_fail(task, "Publication ordonnée RAW batch impossible."); + /* The selected cursor commit is the authority. Generic durable progress + * follows it and may lag after a crash, but must never lead it. */ + if (!lardon3d_task_set_durable_progress( + task, item_index + 1u, execution.item_count, + "Représentation RAW batch publiée.") || + !batch_checkpoint(context, task)) + return lardon3d_task_fail(task, "Checkpoint RAW batch impossible."); + } + if (execution.next_item_index + count == execution.item_count) + return true; + Lardon3DResourceReservation *reservation = nullptr; + if (!lardon3d_task_sequence_break(task, context->governor, &reservation, + &contract)) + return false; + } +} + +bool run_batch(Lardon3DTask *task, void *value) noexcept { + try { + Lardon3DOpenCvTaskThreadGuard threads(task); + if (!threads.valid()) + return lardon3d_task_fail(task, "Contrat CPU OpenCV RAW batch invalide."); + /* External Task participants are the admitted CPU dimension. OpenCV stays + * single-threaded so imencode/imdecode cannot create a nested fan-out. */ + cv::setNumThreads(1); + if (cv::getNumThreads() != 1) + return lardon3d_task_fail(task, "Configuration OpenCV RAW batch impossible."); + bool result = run_batch_impl(task, static_cast(value)); + if (!threads.restore()) + return lardon3d_task_fail(task, "Restauration OpenCV RAW batch impossible."); + return result; + } catch (const std::bad_alloc &) { + return lardon3d_task_fail(task, "Mémoire insuffisante pour le RAW batch."); + } catch (...) { + return lardon3d_task_fail(task, "Erreur interne du RAW batch."); + } +} + +void finished_batch(const Lardon3DTask *task, void *value) noexcept { + try { + Lardon3DAppState state; + batch_runtime(static_cast(value), &state); + (void)lardon3d_project_checkpoint_task(&state, task); + } catch (...) { + } +} } // namespace extern "C" bool lardon3d_raw_development_task_reconstruct( @@ -173,7 +424,7 @@ extern "C" Lardon3DTask *lardon3d_project_create_raw_development_task( delete context; return nullptr; } - Lardon3DResourceEstimate estimate{kRawWorkingBytes + sizeof(Context), 0, 0, 0, + Lardon3DResourceEstimate estimate{kRawSingleWorkingBytes + sizeof(Context), 0, 0, 0, 1, 1, 1, 0, 1, LARDON3D_RESOURCE_TASK_MIXED}; Lardon3DTask *task = lardon3d_task_create_typed( "Développement RAW S3-B1", &estimate, LARDON3D_RAW_DEVELOPMENT_TASK_KIND, @@ -208,3 +459,104 @@ extern "C" bool lardon3d_project_enqueue_raw_development( return false; } } + +extern "C" bool lardon3d_raw_development_batch_task_reconstruct( + const Lardon3DTaskDurableSnapshot *snapshot, void *value, + Lardon3DTaskKindBinding *binding) { + try { + auto *runtime_context = static_cast(value); + if (!snapshot || !runtime_context || !binding) return false; + Lardon3DProjectDbRawDevelopmentBatchTask persisted{}; + if (lardon3d_project_db_load_raw_development_batch_task( + runtime_context->project_db, snapshot->id, &persisted) != + LARDON3D_PROJECT_DB_OK) + return false; + BatchContext *context = make_batch_context( + runtime_context, persisted.selected_execution_id); + if (!context) return false; + *binding = {}; + binding->callback = run_batch; + binding->userdata = context; + binding->userdata_destroy = destroy_batch; + binding->finished_callback = finished_batch; + binding->finished_userdata = context; + return true; + } catch (...) { + return false; + } +} + +extern "C" Lardon3DTask *lardon3d_project_create_raw_development_batch_task( + Lardon3DAppState *state, uint64_t selected_execution_id, uint64_t *task_id) { + if (task_id) *task_id = 0; + try { + if (!state || !state->project_loaded || !state->project_db || + !state->resource_governor || !task_id || selected_execution_id == 0) + return nullptr; + Lardon3DProjectDbSelectedExecution execution{}; + if (lardon3d_project_db_load_selected_execution( + state->project_db, selected_execution_id, &execution) != + LARDON3D_PROJECT_DB_OK || + execution.stage != LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS) + return nullptr; + Lardon3DTaskReconstructionContext runtime_context{ + state->project_path, state->project_db, state->resource_governor, nullptr}; + BatchContext *context = make_batch_context(&runtime_context, + selected_execution_id); + if (!context) return nullptr; + uint64_t id = 0; + if (lardon3d_project_db_allocate_task_id(state->project_db, &id) != + LARDON3D_PROJECT_DB_OK) { + delete context; + return nullptr; + } + /* At the frozen 40 MP decoder ceiling, one participant owns LibRaw's + * mosaic/16-bit four-channel workspace, the processed RGB/BGR copies, and + * the later PNG/validation buffers. Their lifetime analysis plus the 1 MiB + * child stack and allocator/codec headroom is bounded by 896 MiB. Charge + * the tiny owner context through that per-item allowance too: a 7 GiB + * post-reserve host budget can then admit all eight independently bounded + * participants instead of losing one to a redundant fixed charge. + * + * The 1..8 capacity is operational admission only. Immutable selected + * execution and item order remain the scientific, durable input, never a + * dataset-size limit. */ + Lardon3DResourceEstimate estimate{ + 0, 0, kRawBatchParticipantBytes, 0, + 1, kRawBatchWindowMax, static_cast(kRawBatchWindowMax), + 0, 1, LARDON3D_RESOURCE_TASK_MIXED}; + Lardon3DTask *task = lardon3d_task_create_typed( + "Développement RAW sélectionné", &estimate, + LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND, + LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION, run_batch, context, + destroy_batch); + if (!task || !lardon3d_task_assign_id(task, id) || + !lardon3d_task_set_finished_callback(task, finished_batch, context) || + !batch_checkpoint(context, task)) { + lardon3d_task_destroy(task); + return nullptr; + } + *task_id = id; + return task; + } catch (...) { + return nullptr; + } +} + +extern "C" bool lardon3d_project_enqueue_raw_development_batch( + Lardon3DAppState *state, uint64_t selected_execution_id, + uint64_t *task_id) { + try { + if (!state || !state->task_queue) return false; + Lardon3DTask *task = lardon3d_project_create_raw_development_batch_task( + state, selected_execution_id, task_id); + if (!task) return false; + if (!lardon3d_task_queue_add(state->task_queue, task, nullptr)) { + lardon3d_task_destroy(task); + return false; + } + return true; + } catch (...) { + return false; + } +} diff --git a/src/resource_governor.c b/src/resource_governor.c index d88f9e1..9b81c72 100644 --- a/src/resource_governor.c +++ b/src/resource_governor.c @@ -1644,7 +1644,10 @@ open_next_capability_trial_locked( && feedback->accepted_cpu_limit < cpu_max) { feedback->adaptive_cpu_limit = next_trial_cpu( feedback->accepted_cpu_limit, cpu_max); - feedback->adaptive_batch_limit = feedback->accepted_batch_limit; + feedback->adaptive_batch_limit = capability->cpu_batch_coupled + ? minimum_size(feedback->adaptive_cpu_limit, + capability->estimate.maximum_batch_size) + : feedback->accepted_batch_limit; feedback->adaptive_inflight_limit = feedback->accepted_inflight_limit; feedback->trial_dimension = LARDON3D_CAPABILITY_TRIAL_CPU; @@ -4291,7 +4294,10 @@ lardon3d_resource_governor_internal_record_sequence_execution_metrics( bool exercised = feedback->trial_dimension == LARDON3D_CAPABILITY_TRIAL_CPU ? selection->decision.cpu_threads - == feedback->adaptive_cpu_limit + == feedback->adaptive_cpu_limit + && (!selection->capability.cpu_batch_coupled + || selection->decision.batch_size + == feedback->adaptive_batch_limit) : feedback->trial_dimension == LARDON3D_CAPABILITY_TRIAL_INFLIGHT ? selection->inflight_limit @@ -4350,6 +4356,13 @@ lardon3d_resource_governor_internal_record_sequence_execution_metrics( == LARDON3D_CAPABILITY_TRIAL_CPU) { feedback->accepted_cpu_limit = feedback->adaptive_cpu_limit; + if (selection->capability.cpu_batch_coupled) { + /* The measured gain belongs to the complete + * cross-image rung, not to an unusable CPU + * count detached from its participant window. */ + feedback->accepted_batch_limit = + feedback->adaptive_batch_limit; + } } else if (completed_dimension == LARDON3D_CAPABILITY_TRIAL_INFLIGHT) { feedback->accepted_inflight_limit = diff --git a/src/resource_governor_internal.h b/src/resource_governor_internal.h index b50616f..b0e0e32 100644 --- a/src/resource_governor_internal.h +++ b/src/resource_governor_internal.h @@ -44,6 +44,11 @@ typedef struct { bool preferred; bool cpu_reducible; bool batch_adaptive; + /* Cross-item CPU callbacks cannot exercise a CPU trial unless the same + * immutable sequence admits at least that many independent items. The + * Governor therefore advances and accepts these two operational limits as + * one rung; this property is private and never changes durable science. */ + bool cpu_batch_coupled; /* Current ORB Vulkan batch trials use a longer observation window than * generic CPU adaptation. This private operational property is * reconstructed with the capability and is never durable/scientific. */ diff --git a/src/task.c b/src/task.c index 9ed69e1..93eaed3 100644 --- a/src/task.c +++ b/src/task.c @@ -112,6 +112,7 @@ kind_has_validated_cpu_range(const char *task_kind, uint32_t task_kind_version) return false; } return strcmp(task_kind, "features.extract") == 0 + || strcmp(task_kind, "features.extract.batch") == 0 || strcmp(task_kind, "features.extract.sift") == 0 || strcmp(task_kind, "features.extract.rootsift") == 0 || strcmp(task_kind, "visual_index.update") == 0 @@ -127,7 +128,20 @@ kind_has_validated_batch_range(const char *task_kind, uint32_t version) * one-item batch cannot exercise a second participant; Gate G may still * reduce that bounded window for capacity or pressure. */ return task_kind && version == 1 - && strcmp(task_kind, "candidate_pair.generate") == 0; + && (strcmp(task_kind, "candidate_pair.generate") == 0 + || strcmp(task_kind, "features.extract.batch") == 0); +} + +static bool +kind_has_coupled_cpu_batch_range(const char *task_kind, uint32_t version) +{ + /* Feature batch and Candidate obtain CPU parallelism only across + * independent items. A CPU2/batch1 trial is structurally incapable of + * measuring either path, so Governor feedback must trial the complete + * participant rung without changing scientific ordering or publication. */ + return task_kind && version == 1 + && (strcmp(task_kind, "features.extract.batch") == 0 + || strcmp(task_kind, "candidate_pair.generate") == 0); } static void @@ -244,6 +258,9 @@ lardon3d_task_create_typed( && kind_has_validated_cpu_range(task_kind, task_kind_version), .batch_adaptive = typed && kind_has_validated_batch_range(task_kind, task_kind_version), + .cpu_batch_coupled = typed + && kind_has_coupled_cpu_batch_range( + task_kind, task_kind_version), }}, }; copy_text(task->message, sizeof(task->message), "En attente."); @@ -276,6 +293,9 @@ lardon3d_task_internal_set_capability_envelope( && (!capability->batch_adaptive || capability->backend != LARDON3D_RESOURCE_BACKEND_ORB_VULKAN)) + || (capability->cpu_batch_coupled + && (!capability->cpu_reducible + || !capability->batch_adaptive)) || (capability->inflight_adaptive && (capability->minimum_inflight_limit == 0 || capability->gpu_memory_bytes_per_inflight == 0)) @@ -317,6 +337,9 @@ lardon3d_task_internal_enable_known_capabilities(Lardon3DTask *task) task->capability_envelope.capabilities[0].batch_adaptive = kind_has_validated_batch_range( task->task_kind, task->task_kind_version); + task->capability_envelope.capabilities[0].cpu_batch_coupled = + kind_has_coupled_cpu_batch_range( + task->task_kind, task->task_kind_version); } (void)pthread_mutex_unlock(&task->mutex); return true; diff --git a/src/task_kind_registry.c b/src/task_kind_registry.c index fc3ab6a..b1195bc 100644 --- a/src/task_kind_registry.c +++ b/src/task_kind_registry.c @@ -38,6 +38,13 @@ enum { VISUAL_INDEX_PER_ITEM_BYTES = 2 * 1024 * 1024, }; +#define RAW_BATCH_LEGACY_PER_ITEM_BYTES \ + ((UINT64_C(2) * 1024u * 1024u * 1024u) + (UINT64_C(1) * 1024u * 1024u)) +#define RAW_BATCH_CURRENT_PER_ITEM_BYTES (UINT64_C(896) * 1024u * 1024u) +/* The first v24 implementation persisted sizeof(BatchContext) as fixed + * memory. Its public checkpoint ABI made that 4120-byte exact shape durable. */ +#define RAW_BATCH_LEGACY_FIXED_BYTES UINT64_C(4120) + static bool estimate_equals(const Lardon3DResourceEstimate *left, const Lardon3DResourceEstimate *right) @@ -182,6 +189,23 @@ normalize_known_legacy_estimate(const char *kind, /* CPU/batch/memory admission is operational, not GVR identity. Accept * only the complete frozen serial envelope, normalize it in memory, * and never publish an estimate-only checkpoint during recovery. */ + } else if (strcmp(kind, "raw.develop.batch") == 0) { + current = (Lardon3DResourceEstimate) { + .memory_bytes_per_item = RAW_BATCH_CURRENT_PER_ITEM_BYTES, + .minimum_batch_size = 1, + .maximum_batch_size = 8, + .desired_cpu_threads = 8, + .desired_io_slots = 1, + .task_class = LARDON3D_RESOURCE_TASK_MIXED, + }; + historical = current; + historical.memory_fixed_bytes = RAW_BATCH_LEGACY_FIXED_BYTES; + historical.memory_bytes_per_item = RAW_BATCH_LEGACY_PER_ITEM_BYTES; + /* v24 originally persisted this exact 4120-byte owner plus 2 GiB and + * 1 MiB stack participant allowance. It is Governor accounting, not + * RAW identity or the selected-execution payload. Accept only this + * complete legacy shape and normalize it in memory, so recovery can + * reach the corrected safe window without rewriting a checkpoint. */ } else { *effective = *durable; return true; diff --git a/src/task_kinds.c b/src/task_kinds.c index ec4bdb5..cec376b 100644 --- a/src/task_kinds.c +++ b/src/task_kinds.c @@ -33,6 +33,11 @@ const Lardon3DTaskKindRegistry *lardon3d_task_kind_registry_production(void) { .kind_version = LARDON3D_RAW_DEVELOPMENT_TASK_KIND_VERSION, .reconstruct = lardon3d_raw_development_task_reconstruct, }, + { + .kind = LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND, + .kind_version = LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION, + .reconstruct = lardon3d_raw_development_batch_task_reconstruct, + }, #endif #ifdef LARDON3D_PHOTO_QUALITY_TASK_AVAILABLE { @@ -58,6 +63,11 @@ const Lardon3DTaskKindRegistry *lardon3d_task_kind_registry_production(void) { .kind_version = LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION, .reconstruct = lardon3d_feature_extract_reconstruct, }, + { + .kind = LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND, + .kind_version = LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION, + .reconstruct = lardon3d_feature_extract_batch_reconstruct, + }, { .kind = LARDON3D_SIFT_EXTRACT_TASK_KIND, .kind_version = LARDON3D_SIFT_EXTRACT_TASK_KIND_VERSION, diff --git a/src/tui_model.c b/src/tui_model.c index 3723435..8d668c0 100644 --- a/src/tui_model.c +++ b/src/tui_model.c @@ -167,11 +167,13 @@ kind_to_stage( if (strcmp(kind, "import.images") == 0 || strcmp(kind, "acquisition_campaign.run") == 0) { *stage = LARDON3D_TUI_STAGE_ACQUISITION; - } else if (strcmp(kind, "raw.develop") == 0) { + } else if (strcmp(kind, "raw.develop") == 0 || + strcmp(kind, "raw.develop.batch") == 0) { *stage = LARDON3D_TUI_STAGE_RAW; } else if (strcmp(kind, "photo_quality.triage") == 0) { *stage = LARDON3D_TUI_STAGE_QUALITY; } else if (strcmp(kind, "features.extract") == 0 + || strcmp(kind, "features.extract.batch") == 0 || strcmp(kind, "features.extract.sift") == 0 || strcmp(kind, "features.extract.rootsift") == 0) { *stage = LARDON3D_TUI_STAGE_FEATURES; diff --git a/tests/pre_sfm_real_execution.cpp b/tests/pre_sfm_real_execution.cpp index 028312c..ff1f0ea 100644 --- a/tests/pre_sfm_real_execution.cpp +++ b/tests/pre_sfm_real_execution.cpp @@ -53,6 +53,10 @@ struct Options { bool resume_geometry_existing{}; bool resume_pre_gv_existing{}; bool resume_candidate_existing{}; + bool resume_representations_existing{}; + uint64_t selected_execution_id{}; + bool has_selected_execution_id{}; + bool stop_after_representations{}; unsigned int cpu_budget{}; unsigned int gpu_budget{}; bool has_gpu_budget{}; @@ -158,13 +162,30 @@ void usage(const char *program) { "[--matcher-batch 2|4|8|12] " "[--stop-after-matcher]\n" " %s --resume-candidate-existing --project-dir " - "ABSOLUTE_EXISTING_DIR [--cpu-budget 1..12] [--gpu-budget 0..1]\n", + "ABSOLUTE_EXISTING_DIR [--cpu-budget 1..12] [--gpu-budget 0..1]\n" + " %s --resume-representations-existing --project-dir " + "ABSOLUTE_EXISTING_DIR --selected-execution-id ID " + "[--stop-after-representations]\n", + program, program, program, program, program); } +bool parse_u64_id(const char *text, uint64_t &value) { + if (!text || !*text || *text == '0') return false; + for (const char *character = text; *character; ++character) { + if (*character < '0' || *character > '9') return false; + } + char *end = nullptr; + errno = 0; + const unsigned long long parsed = std::strtoull(text, &end, 10); + if (errno != 0 || end == text || *end != '\0' || parsed == 0) return false; + value = static_cast(parsed); + return true; +} + bool parse_size(const char *text, size_t &value) { char *end = nullptr; errno = 0; @@ -228,11 +249,16 @@ bool parse_options(int argc, char **argv, Options &options) { options.resume_candidate_existing = true; continue; } + if (argument == "--resume-representations-existing") { + options.resume_representations_existing = true; + continue; + } if ((argument == "--mode" || argument == "--project-dir" || argument == "--root" || argument == "--limit" || argument == "--restart-boundary" || argument == "--cpu-budget" || argument == "--gpu-budget" || argument == "--matcher-mode" || argument == "--matcher-pipeline" || - argument == "--matcher-inflight" || argument == "--matcher-batch") && + argument == "--matcher-inflight" || argument == "--matcher-batch" || + argument == "--selected-execution-id") && index + 1 >= argc) return false; if (argument == "--mode") { @@ -280,40 +306,59 @@ bool parse_options(int argc, char **argv, Options &options) { if (!parse_matcher_batch(argv[++index], options.matcher_batch_override)) return false; options.has_matcher_batch_override = true; + } else if (argument == "--selected-execution-id") { + if (!parse_u64_id(argv[++index], options.selected_execution_id)) return false; + options.has_selected_execution_id = true; } else if (argument == "--stop-after-matcher") { options.stop_after_matcher = true; } else if (argument == "--stop-after-gv") { options.stop_after_gv = true; + } else if (argument == "--stop-after-representations") { + options.stop_after_representations = true; } else { return false; } } const unsigned int resume_mode_count = options.resume_geometry_existing + options.resume_pre_gv_existing + - options.resume_candidate_existing; + options.resume_candidate_existing + + options.resume_representations_existing; if (resume_mode_count != 0) { if (options.has_mode || options.project_dir.empty() || !options.roots.empty() || options.restart != RestartBoundary::kNone || resume_mode_count != 1) return false; if (options.resume_pre_gv_existing) { - return !options.stop_after_gv; + return !options.stop_after_gv && !options.has_selected_execution_id && + !options.stop_after_representations; } if (options.resume_candidate_existing) return !options.has_matcher_mode && !options.has_matcher_pipeline && !options.has_matcher_inflight_override && !options.has_matcher_batch_override && - !options.stop_after_matcher && !options.stop_after_gv; + !options.stop_after_matcher && !options.stop_after_gv && + !options.has_selected_execution_id && + !options.stop_after_representations; + if (options.resume_representations_existing) + return options.has_selected_execution_id && options.cpu_budget == 0 && + !options.has_gpu_budget && !options.has_matcher_mode && + !options.has_matcher_pipeline && + !options.has_matcher_inflight_override && + !options.has_matcher_batch_override && !options.stop_after_matcher && + !options.stop_after_gv; return options.cpu_budget == 0 && !options.has_gpu_budget && !options.has_matcher_mode && !options.has_matcher_pipeline && !options.has_matcher_inflight_override && !options.has_matcher_batch_override && - !options.stop_after_matcher; + !options.stop_after_matcher && !options.has_selected_execution_id && + !options.stop_after_representations; } if (options.cpu_budget != 0 || options.has_gpu_budget || options.has_matcher_mode || options.has_matcher_pipeline || options.has_matcher_inflight_override || options.has_matcher_batch_override || - options.stop_after_matcher || options.stop_after_gv) return false; + options.stop_after_matcher || options.stop_after_gv || + options.has_selected_execution_id || options.stop_after_representations) + return false; return options.has_mode && !options.project_dir.empty() && !options.roots.empty() && options.roots.size() <= LARDON3D_ACQUISITION_CAMPAIGN_MAX_ROOTS; } @@ -1440,46 +1485,233 @@ bool create_selection(Runtime &runtime, Mode mode, uint64_t quality_task_id, items.size(), std::time(nullptr), &execution) == LARDON3D_PROJECT_DB_OK; } +enum class RepresentationPendingKind { kNone, kLegacySerial, kBatch }; +enum class RepresentationAction { + kReject, + kCollect, + kRecoverLegacySerial, + kRecoverBatch, + kEnqueueBatch, +}; + +RepresentationAction choose_representation_action( + const Lardon3DProjectDbSelectedExecution &execution, + RepresentationPendingKind pending) { + if (execution.next_item_index > execution.item_count) return RepresentationAction::kReject; + if (execution.stage == LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS && + execution.next_item_index < execution.item_count) { + if (pending == RepresentationPendingKind::kLegacySerial) + return RepresentationAction::kRecoverLegacySerial; + if (pending == RepresentationPendingKind::kBatch) + return RepresentationAction::kRecoverBatch; + return RepresentationAction::kEnqueueBatch; + } + if (execution.stage == LARDON3D_SELECTED_EXECUTION_CALIBRATION && + execution.next_item_index == execution.item_count) { + return pending == RepresentationPendingKind::kBatch + ? RepresentationAction::kRecoverBatch + : pending == RepresentationPendingKind::kNone + ? RepresentationAction::kCollect + : RepresentationAction::kReject; + } + return RepresentationAction::kReject; +} + +bool representation_defers_feature_task( + const Lardon3DProjectDbSelectedExecution &execution, + const Lardon3DProjectRecoveryEntry &entry) { + if (execution.stage != LARDON3D_SELECTED_EXECUTION_CALIBRATION || + execution.next_item_index != execution.item_count) + return false; + return (std::strcmp(entry.task_kind, LARDON3D_FEATURE_EXTRACT_TASK_KIND) == 0 && + entry.task_kind_version == LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION) || + (std::strcmp(entry.task_kind, + LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND) == 0 && + entry.task_kind_version == + LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION); +} + +bool find_pending_representation_task( + Runtime &runtime, const Lardon3DProjectDbSelectedExecution &execution, + RepresentationPendingKind &kind, uint64_t &task_id) { + kind = RepresentationPendingKind::kNone; + task_id = 0; + uint64_t cursor = 0; + for (;;) { + Lardon3DProjectRecoveryEntry entries[8]{}; + size_t count = 0; + if (lardon3d_project_list_recoverable( + &runtime.state, lardon3d_task_kind_registry_production(), cursor, + entries, 8, &count) != LARDON3D_PROJECT_DB_OK) + return false; + for (size_t index = 0; index < count; ++index) { + const auto &entry = entries[index]; + cursor = entry.task_id; + if ((entry.status != LARDON3D_PROJECT_RECOVERABLE && + entry.status != LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE) || + entry.snapshot.recovery_state != TASK_PENDING) { + std::fprintf(stderr, + "representation recovery refuses unsafe pending task %llu of kind %s\n", + static_cast(entry.task_id), entry.task_kind); + return false; + } + /* Feature recovery owns the phase after representation collection. A + * precise Feature kind/version may already be queued by project-open, + * but representation recovery must neither resume nor reject it once the + * selected representation prefix is durably complete. extract_features + * validates its ORB fingerprint and legacy/batch ownership before wait. + * Every other pending kind remains fail-closed here. */ + if (representation_defers_feature_task(execution, entry)) continue; + if (task_id != 0) { + std::fprintf(stderr, + "representation recovery refuses multiple pending owners" + " including task %llu of kind %s\n", + static_cast(entry.task_id), + entry.task_kind); + return false; + } + if (std::strcmp(entry.task_kind, LARDON3D_RAW_DEVELOPMENT_TASK_KIND) == 0 && + entry.task_kind_version == LARDON3D_RAW_DEVELOPMENT_TASK_KIND_VERSION && + execution.stage == LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS && + execution.next_item_index < execution.item_count) { + Lardon3DProjectDbSelectedExecutionItem item{}; + Lardon3DProjectDbRawDevelopmentTask raw{}; + if (lardon3d_project_db_load_selected_execution_item( + runtime.state.project_db, execution.execution_id, + execution.next_item_index, &item) != LARDON3D_PROJECT_DB_OK || + item.representation_source != LARDON3D_SELECTED_REPRESENTATION_RAW_ASSET || + lardon3d_project_db_load_raw_development_task( + runtime.state.project_db, entry.task_id, &raw) != LARDON3D_PROJECT_DB_OK || + raw.capture_id != item.capture_id || + raw.source_asset_id != item.source_asset_id) { + std::fprintf(stderr, + "legacy RAW task %llu does not own selected cursor %u\n", + static_cast(entry.task_id), + execution.next_item_index); + return false; + } + kind = RepresentationPendingKind::kLegacySerial; + } else if (std::strcmp(entry.task_kind, + LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND) == 0 && + entry.task_kind_version == + LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION) { + Lardon3DProjectDbRawDevelopmentBatchTask batch{}; + if (lardon3d_project_db_load_raw_development_batch_task( + runtime.state.project_db, entry.task_id, &batch) != + LARDON3D_PROJECT_DB_OK || + batch.selected_execution_id != execution.execution_id) { + std::fprintf(stderr, + "RAW batch task %llu does not own selected execution %llu\n", + static_cast(entry.task_id), + static_cast(execution.execution_id)); + return false; + } + kind = RepresentationPendingKind::kBatch; + } else { + std::fprintf(stderr, + "representation recovery refuses pending task %llu of kind %s v%u\n", + static_cast(entry.task_id), entry.task_kind, + entry.task_kind_version); + return false; + } + task_id = entry.task_id; + } + if (count < 8) return true; + } +} + +bool recover_one_representation_task(Runtime &runtime, uint64_t task_id, + const char *phase) { + Lardon3DProjectRecoverySummary recovery{}; + return lardon3d_project_resume_recoverable_tasks( + &runtime.state, lardon3d_task_kind_registry_production(), + &recovery) == LARDON3D_PROJECT_DB_OK && + recovery.inspected == 1 && recovery.resumed == 1 && + recovery.failed == 0 && !recovery.queue_full && + wait_completed(runtime, task_id, phase); +} + +enum class FeaturePendingKind { kLegacySerial, kBatch }; + +bool feature_recovery_kind(const Lardon3DProjectRecoveryEntry &entry, + FeaturePendingKind &kind) { + if ((entry.status != LARDON3D_PROJECT_RECOVERABLE && + entry.status != LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE) || + entry.snapshot.recovery_state != TASK_PENDING) + return false; + if (std::strcmp(entry.task_kind, LARDON3D_FEATURE_EXTRACT_TASK_KIND) == 0 && + entry.task_kind_version == LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION) { + kind = FeaturePendingKind::kLegacySerial; + return true; + } + if (std::strcmp(entry.task_kind, LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND) == 0 && + entry.task_kind_version == LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION) { + kind = FeaturePendingKind::kBatch; + return true; + } + return false; +} + bool publish_representations(Runtime &runtime, uint64_t execution_id, std::vector &image_ids) { - Lardon3DProjectDbSelectedExecution execution{}; - if (lardon3d_project_db_load_selected_execution(runtime.state.project_db, execution_id, - &execution) != LARDON3D_PROJECT_DB_OK) - return false; - for (uint32_t index = execution.next_item_index; index < execution.item_count; ++index) { - Lardon3DProjectDbSelectedExecutionItem item{}; - if (lardon3d_project_db_load_selected_execution_item(runtime.state.project_db, execution_id, - index, &item) != LARDON3D_PROJECT_DB_OK) - return false; - uint64_t image_id = 0; - if (item.representation_source == LARDON3D_SELECTED_REPRESENTATION_RAW_ASSET) { - uint64_t task_id = 0; - if (!lardon3d_project_enqueue_raw_development(&runtime.state, item.capture_id, - item.source_asset_id, &task_id) || - !wait_completed(runtime, task_id, "raw.develop")) - return false; - Lardon3DProjectDbRawDevelopmentTask raw{}; - if (lardon3d_project_db_load_raw_development_task(runtime.state.project_db, task_id, &raw) != - LARDON3D_PROJECT_DB_OK || - raw.phase != LARDON3D_RAW_DEVELOPMENT_TASK_PUBLISHED || !raw.has_image || - raw.capture_id != item.capture_id || raw.source_asset_id != item.source_asset_id) - return false; - image_id = raw.image_id; - } else { - if (lardon3d_project_db_get_selected_capture_image(runtime.state.project_db, item.capture_id, - &image_id) != LARDON3D_PROJECT_DB_OK) - return false; - } - if (lardon3d_project_db_record_selected_representation(runtime.state.project_db, execution_id, - index, image_id, index + 1u) != + for (;;) { + Lardon3DProjectDbSelectedExecution execution{}; + if (lardon3d_project_db_load_selected_execution( + runtime.state.project_db, execution_id, &execution) != LARDON3D_PROJECT_DB_OK) return false; + RepresentationPendingKind pending{}; + uint64_t pending_task_id = 0; + if (!find_pending_representation_task(runtime, execution, pending, + pending_task_id)) + return false; + const RepresentationAction action = choose_representation_action(execution, pending); + if (action == RepresentationAction::kReject) return false; + if (action == RepresentationAction::kCollect) break; + if (action == RepresentationAction::kRecoverLegacySerial) { + const uint32_t item_index = execution.next_item_index; + Lardon3DProjectDbSelectedExecutionItem item{}; + if (lardon3d_project_db_load_selected_execution_item( + runtime.state.project_db, execution_id, item_index, &item) != + LARDON3D_PROJECT_DB_OK || + !recover_one_representation_task(runtime, pending_task_id, + "raw.develop recovered existing")) + return false; + Lardon3DProjectDbRawDevelopmentTask raw{}; + if (lardon3d_project_db_load_raw_development_task( + runtime.state.project_db, pending_task_id, &raw) != + LARDON3D_PROJECT_DB_OK || + raw.phase != LARDON3D_RAW_DEVELOPMENT_TASK_PUBLISHED || !raw.has_image || + raw.capture_id != item.capture_id || + raw.source_asset_id != item.source_asset_id || + lardon3d_project_db_record_selected_representation( + runtime.state.project_db, execution_id, item_index, raw.image_id, + item_index + 1u) != LARDON3D_PROJECT_DB_OK) + return false; + /* The legacy Task predates the selected-execution batch owner. Its exact + * current item is committed once before the batch reads the advanced + * cursor; no completed selected prefix is replayed or re-identified. */ + continue; + } + if (action == RepresentationAction::kRecoverBatch) { + if (!recover_one_representation_task(runtime, pending_task_id, + "raw.develop.batch recovered existing")) + return false; + continue; + } + uint64_t batch_task_id = 0; + if (!lardon3d_project_enqueue_raw_development_batch( + &runtime.state, execution_id, &batch_task_id) || + !wait_completed(runtime, batch_task_id, "raw.develop.batch")) + return false; } - if (lardon3d_project_db_load_selected_execution(runtime.state.project_db, execution_id, - &execution) != LARDON3D_PROJECT_DB_OK || + image_ids.clear(); + Lardon3DProjectDbSelectedExecution execution{}; + if (lardon3d_project_db_load_selected_execution(runtime.state.project_db, + execution_id, &execution) != + LARDON3D_PROJECT_DB_OK || execution.stage != LARDON3D_SELECTED_EXECUTION_CALIBRATION) return false; - image_ids.clear(); for (uint32_t index = 0; index < execution.item_count; ++index) { Lardon3DProjectDbSelectedExecutionItem item{}; if (lardon3d_project_db_load_selected_execution_item(runtime.state.project_db, execution_id, @@ -1491,30 +1723,100 @@ bool publish_representations(Runtime &runtime, uint64_t execution_id, return true; } -bool extract_features(Runtime &runtime, const std::vector &image_ids, +bool extract_features(Runtime &runtime, uint64_t selected_execution_id, + const std::vector &image_ids, const Lardon3DFeatureExtractorParameters ¶meters, std::vector &feature_sets) { unsigned char fingerprint[32]{}; lardon3d_feature_extractor_parameter_fingerprint(¶meters, fingerprint); feature_sets.clear(); + bool complete = true; for (const uint64_t image_id : image_ids) { Lardon3DProjectDbFeatureSet feature{}; if (lardon3d_project_db_find_feature_set(runtime.state.project_db, image_id, LARDON3D_FEATURE_EXTRACTOR_KIND, LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, &feature) != LARDON3D_PROJECT_DB_OK) { - uint64_t task_id = 0; - if (!lardon3d_project_enqueue_feature_extract(&runtime.state, image_id, ¶meters, - &task_id) || - !wait_completed(runtime, task_id, "features.extract") || - lardon3d_project_db_find_feature_set( - runtime.state.project_db, image_id, LARDON3D_FEATURE_EXTRACTOR_KIND, - LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, &feature) != - LARDON3D_PROJECT_DB_OK) - return false; + complete = false; + break; } feature_sets.push_back(feature); } + if (complete) return true; + + /* A pre-v25 single-image Task is historical durable work, not a batch + * cursor. Reopening creates an empty Queue, so restore and finish that exact + * Task once before the new owner starts its selected suffix. Never rewrite or + * replace the legacy row (Task 166 in the inspected A6000 project). */ + uint64_t pending_task_id = 0; + FeaturePendingKind pending_kind{}; + uint64_t cursor = 0; + for (;;) { + Lardon3DProjectRecoveryEntry tasks[8]{}; + size_t count = 0; + if (lardon3d_project_list_recoverable( + &runtime.state, lardon3d_task_kind_registry_production(), cursor, + tasks, 8, &count) != + LARDON3D_PROJECT_DB_OK) + return false; + for (size_t index = 0; index < count; ++index) { + cursor = tasks[index].task_id; + FeaturePendingKind kind{}; + if (pending_task_id != 0 || !feature_recovery_kind(tasks[index], kind)) { + /* Registry recovery resumes every pending Task. Refuse ambiguity or an + * unrelated pipeline owner before transferring anything to Queue. */ + std::fprintf(stderr, + "feature recovery refuses pending task %llu of kind %s v%u\n", + static_cast(tasks[index].task_id), + tasks[index].task_kind, tasks[index].task_kind_version); + return false; + } + if (kind == FeaturePendingKind::kLegacySerial) { + Lardon3DProjectDbFeatureExtractTask legacy{}; + if (lardon3d_project_db_load_feature_extract_task( + runtime.state.project_db, tasks[index].task_id, &legacy) != + LARDON3D_PROJECT_DB_OK || + std::memcmp(legacy.parameter_fingerprint, fingerprint, 32) != 0) + return false; + } else { + Lardon3DProjectDbFeatureExtractBatchTask batch{}; + if (lardon3d_project_db_load_feature_extract_batch_task( + runtime.state.project_db, tasks[index].task_id, &batch) != + LARDON3D_PROJECT_DB_OK || + batch.selected_execution_id != selected_execution_id || + std::memcmp(batch.parameter_fingerprint, fingerprint, 32) != 0) + return false; + } + pending_task_id = tasks[index].task_id; + pending_kind = kind; + } + if (count < 8) break; + } + if (pending_task_id != 0) { + const char *phase = pending_kind == FeaturePendingKind::kLegacySerial + ? "features.extract recovered legacy" + : "features.extract.batch recovered"; + if (!recover_one_representation_task(runtime, pending_task_id, phase)) + return false; + } + uint64_t batch_task_id = 0; + if (pending_task_id == 0 || pending_kind == FeaturePendingKind::kLegacySerial) { + if (!lardon3d_project_enqueue_feature_extract_batch( + &runtime.state, selected_execution_id, ¶meters, &batch_task_id) || + !wait_completed(runtime, batch_task_id, "features.extract.batch")) { + return false; + } + } + feature_sets.clear(); + for (const uint64_t image_id : image_ids) { + Lardon3DProjectDbFeatureSet feature{}; + if (lardon3d_project_db_find_feature_set( + runtime.state.project_db, image_id, LARDON3D_FEATURE_EXTRACTOR_KIND, + LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, &feature) != + LARDON3D_PROJECT_DB_OK) + return false; + feature_sets.push_back(feature); + } return true; } @@ -2724,7 +3026,8 @@ int main(int argc, char **argv) { } Runtime runtime; if (options.resume_geometry_existing || options.resume_pre_gv_existing || - options.resume_candidate_existing) { + options.resume_candidate_existing || + options.resume_representations_existing) { if (!prepare_existing_project(options.project_dir, runtime)) { std::fprintf(stderr, "existing project must be absolute, normalized, and durable\n"); return 2; @@ -2741,43 +3044,80 @@ int main(int argc, char **argv) { runtime.stop_after_gv = options.stop_after_gv; if (options.resume_candidate_existing) return run_existing_candidate(runtime) ? 0 : 1; - return (options.resume_pre_gv_existing ? run_existing_pre_gv(runtime) - : run_existing_geometry(runtime)) ? 0 : 1; + if (options.resume_pre_gv_existing || options.resume_geometry_existing) + return (options.resume_pre_gv_existing ? run_existing_pre_gv(runtime) + : run_existing_geometry(runtime)) ? 0 : 1; } - if (!prepare_empty_project(options.project_dir, runtime.project_path)) { - std::fprintf(stderr, "project directory must be absolute, normalized and empty\n"); - return 2; - } - runtime.database_path = (options.project_dir / "project.lardon3d").string(); - runtime.matcher_needed = true; - Campaign campaign; - if (!discover_campaign(options, campaign) || !start_runtime(runtime)) { - std::fprintf(stderr, "campaign discovery or runtime setup failed\n"); - return 1; - } - - const char *mode_name = options.mode == Mode::kA6000 ? "a6000" : "s21"; - Lardon3DProjectDbScanSet scanset{}; - if (lardon3d_project_db_create_scanset(runtime.state.project_db, mode_name, &scanset) != - LARDON3D_PROJECT_DB_OK) { - stop_runtime(runtime); - return 1; - } - uint64_t quality_task_id = 0; - uint64_t campaign_task_id = 0; Lardon3DProjectDbSelectedExecution execution{}; - bool ok = run_task_pair(runtime, scanset.scanset_id, campaign, - options.restart != RestartBoundary::kNone, quality_task_id, - campaign_task_id) && - create_selection(runtime, options.mode, quality_task_id, campaign_task_id, - campaign.confirmations.size(), execution); - if (!ok || execution.execution_id == 0) { - stop_runtime(runtime); - return ok ? 0 : 1; + const char *mode_name = options.mode == Mode::kA6000 ? "a6000" : "s21"; + bool ok = true; + if (options.resume_representations_existing) { + runtime.matcher_needed = !options.stop_after_representations; + if (!start_runtime(runtime) || + lardon3d_project_db_load_selected_execution( + runtime.state.project_db, options.selected_execution_id, + &execution) != LARDON3D_PROJECT_DB_OK || + (execution.stage != LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS && + execution.stage != LARDON3D_SELECTED_EXECUTION_CALIBRATION)) { + std::fprintf(stderr, + "existing selected execution is unavailable or past representations\n"); + stop_runtime(runtime); + return 1; + } + mode_name = "existing"; + } else { + if (!prepare_empty_project(options.project_dir, runtime.project_path)) { + std::fprintf(stderr, + "project directory must be absolute, normalized and empty\n"); + return 2; + } + runtime.database_path = (options.project_dir / "project.lardon3d").string(); + runtime.matcher_needed = true; + Campaign campaign; + if (!discover_campaign(options, campaign) || !start_runtime(runtime)) { + std::fprintf(stderr, "campaign discovery or runtime setup failed\n"); + return 1; + } + Lardon3DProjectDbScanSet scanset{}; + if (lardon3d_project_db_create_scanset(runtime.state.project_db, mode_name, + &scanset) != LARDON3D_PROJECT_DB_OK) { + stop_runtime(runtime); + return 1; + } + uint64_t quality_task_id = 0; + uint64_t campaign_task_id = 0; + ok = run_task_pair(runtime, scanset.scanset_id, campaign, + options.restart != RestartBoundary::kNone, + quality_task_id, campaign_task_id) && + create_selection(runtime, options.mode, quality_task_id, + campaign_task_id, campaign.confirmations.size(), + execution); + if (!ok || execution.execution_id == 0) { + stop_runtime(runtime); + return ok ? 0 : 1; + } } std::vector image_ids; ok = publish_representations(runtime, execution.execution_id, image_ids); + if (ok && options.stop_after_representations) { + ok = lardon3d_project_db_load_selected_execution( + runtime.state.project_db, execution.execution_id, &execution) == + LARDON3D_PROJECT_DB_OK && + execution.stage == LARDON3D_SELECTED_EXECUTION_CALIBRATION && + execution.next_item_index == execution.item_count; + } + if (ok && options.stop_after_representations) { + std::printf( + "{\"record\":\"existing_representations_summary\",\"ok\":true," + "\"execution_id\":%llu,\"next_item_index\":%u,\"item_count\":%u," + "\"stage\":\"CALIBRATION\",\"selected_prefix_replayed\":false," + "\"features_enqueued\":false,\"sparse_sfm_run\":false}\n", + static_cast(execution.execution_id), + execution.next_item_index, execution.item_count); + stop_runtime(runtime); + return 0; + } if (ok && options.restart == RestartBoundary::kRepresentations) ok = restart_runtime(runtime, execution.execution_id, "representations") && publish_representations(runtime, execution.execution_id, image_ids); @@ -2787,11 +3127,11 @@ int main(int argc, char **argv) { * campaign-size rule; callers use one empty project per campaign. */ const Lardon3DFeatureExtractorParameters orb{8192, 8, 20}; std::vector features; - if (ok) ok = extract_features(runtime, image_ids, orb, features); + if (ok) ok = extract_features(runtime, execution.execution_id, image_ids, orb, features); if (ok && options.restart == RestartBoundary::kFeatures) { ok = restart_runtime(runtime, execution.execution_id, "features") && publish_representations(runtime, execution.execution_id, image_ids) && - extract_features(runtime, image_ids, orb, features); + extract_features(runtime, execution.execution_id, image_ids, orb, features); } uint64_t geometry_task_id = 0; diff --git a/tests/test_feature_task.c b/tests/test_feature_task.c index 9c717bf..fc81036 100644 --- a/tests/test_feature_task.c +++ b/tests/test_feature_task.c @@ -151,6 +151,13 @@ typedef struct { Lardon3DExtractedFeatures output; } AdaptiveExtractWork; +bool lardon3d_feature_extract_batch_test_run(Lardon3DTask *task); + +static bool batch_test_callback(Lardon3DTask *task, void *userdata) { + (void)userdata; + return lardon3d_feature_extract_batch_test_run(task); +} + static bool adaptive_extract_callback(Lardon3DTask *task, void *userdata) { AdaptiveExtractWork *work = userdata; Lardon3DOpenCvTaskThreadControl control; @@ -265,6 +272,92 @@ static bool deterministic_capture_preserves_load_gate(void) { return ok; } +static bool feature_batch_trials_exercise_participants(void) { + Lardon3DHardwareProfile profile = { + .logical_cpu_count = 8, + .page_size_bytes = 4096, + .memory_total_bytes = UINT64_C(16) * 1024 * 1024 * 1024, + .cpu_architecture = "test", + }; + Lardon3DResourcePolicy policy = { + .system_cpu_reserve = 4, + .maximum_cpu_load_ratio = 1.0, + .maximum_io_pressure_avg10 = 100.0, + .io_slot_capacity = 1, + }; + Lardon3DResourceGovernor *governor = + lardon3d_resource_governor_create(&profile, &policy); + Lardon3DResourceSnapshot resources = deterministic_resource_snapshot(0.0); + Lardon3DTaskCapabilityEnvelope envelope = { + .count = 1, + .capabilities = {{ + .estimate = { + .memory_fixed_bytes = 64ULL * 1024 * 1024, + .memory_bytes_per_item = 512ULL * 1024 * 1024, + .minimum_batch_size = 1, + .maximum_batch_size = 12, + .desired_cpu_threads = 12, + .desired_io_slots = 1, + .task_class = LARDON3D_RESOURCE_TASK_CPU, + }, + .backend = LARDON3D_RESOURCE_BACKEND_FIXED, + .inflight_limit = 1, + .cpu_reducible = true, + .batch_adaptive = true, + .cpu_batch_coupled = true, + }}, + }; + bool ok = governor && + lardon3d_resource_governor_internal_set_capture_snapshot(governor, &resources); + for (unsigned int sample = 0; sample < 2 && ok; ++sample) { + Lardon3DResourceCapabilitySelection selection; + Lardon3DResourceReservation *reservation = NULL; + ok = lardon3d_resource_governor_internal_reserve_capability_available( + governor, LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND, 1, + &envelope, &selection, &reservation) && reservation && + selection.decision.cpu_threads == 1 && selection.decision.batch_size == 1 && + lardon3d_resource_governor_internal_record_sequence( + governor, LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND, 1, + &selection, UINT64_C(1000000000), 1) && + lardon3d_resource_governor_release(governor, reservation); + } + Lardon3DResourceCapabilitySelection trial; + Lardon3DResourceReservation *trial_reservation = NULL; + ok = ok && lardon3d_resource_governor_internal_reserve_capability_available( + governor, LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND, 1, + &envelope, &trial, &trial_reservation) && trial_reservation && + trial.decision.cpu_threads == 2 && trial.decision.batch_size == 2; + if (trial_reservation) + ok = lardon3d_resource_governor_release(governor, trial_reservation) && ok; + + /* Candidate uses the same cross-item coupling: CPU2 with batch1 still has + * only one membership to query. Its independent feedback key must therefore + * open a measurable CPU2/batch2 rung as well. */ + for (unsigned int sample = 0; sample < 2 && ok; ++sample) { + Lardon3DResourceCapabilitySelection selection; + Lardon3DResourceReservation *reservation = NULL; + ok = lardon3d_resource_governor_internal_reserve_capability_available( + governor, "candidate_pair.generate", 1, &envelope, &selection, + &reservation) && reservation && + selection.decision.cpu_threads == 1 && selection.decision.batch_size == 1 && + lardon3d_resource_governor_internal_record_sequence( + governor, "candidate_pair.generate", 1, &selection, + UINT64_C(1000000000), 1) && + lardon3d_resource_governor_release(governor, reservation); + } + trial_reservation = NULL; + ok = ok && lardon3d_resource_governor_internal_reserve_capability_available( + governor, "candidate_pair.generate", 1, &envelope, &trial, + &trial_reservation) && trial_reservation && + trial.decision.cpu_threads == 2 && trial.decision.batch_size == 2; + if (trial_reservation) + ok = lardon3d_resource_governor_release(governor, trial_reservation) && ok; + if (governor) + ok = lardon3d_resource_governor_internal_set_capture_snapshot(governor, NULL) && ok; + lardon3d_resource_governor_destroy(governor); + return ok; +} + static bool run_adaptive_output_equivalence(const char *path) { cpu_set_t allowed_mask; CPU_ZERO(&allowed_mask); @@ -466,6 +559,7 @@ static void *consolidate_thread(void *userdata) { } static bool run_test(void) { + CHECK(feature_batch_trials_exercise_participants()); char root[] = "/tmp/lardon3d-feature-task-XXXXXX"; CHECK(deterministic_capture_preserves_load_gate()); CHECK(mkdtemp(root) && setenv("LARDON3D_PROJECTS_ROOT", root, 1) == 0); @@ -557,31 +651,61 @@ static bool run_test(void) { * pool. Feature, SIFT, and RootSIFT must each restore the captured value. */ CHECK(lardon3d_feature_opencv_configure_threads(3)); CHECK(setenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS", "1", 1) == 0); + /* Batch CPU1 configuration deliberately fails after mutating OpenCV. The + * callback must restore the prior process-global value before Queue can run + * any later Task. */ + Lardon3DResourceEstimate failed_batch_estimate = { + .minimum_batch_size = 1, + .maximum_batch_size = 1, + .desired_cpu_threads = 1, + .task_class = LARDON3D_RESOURCE_TASK_CPU, + }; + Lardon3DTask *failed_batch = lardon3d_task_create_typed( + "Feature batch restoration", &failed_batch_estimate, + "features.extract.batch.restore-test", + LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION, batch_test_callback, + NULL, NULL); + uint64_t failed_batch_task = 0; + /* This synthetic Task shares the project Queue with durable Tasks. Reserve + * its ID through Project DB so the next durable enqueue cannot reuse it. */ + CHECK(failed_batch && + lardon3d_project_db_allocate_task_id(state.project_db, + &failed_batch_task) == + LARDON3D_PROJECT_DB_OK && + lardon3d_task_assign_id(failed_batch, failed_batch_task) && + lardon3d_task_queue_add(state.task_queue, failed_batch, + NULL) && + wait_state(state.task_queue, failed_batch_task, TASK_FAILED, + &snapshot) && + lardon3d_feature_opencv_thread_count() == 3 && + unsetenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS") == 0); Lardon3DFeatureExtractorParameters failed_orb_parameters = {509, 4, 10}; uint64_t failed_orb_task = 0; + CHECK(setenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS", "once", 1) == 0); CHECK(lardon3d_project_enqueue_feature_extract( - &state, image.image_id, &failed_orb_parameters, &failed_orb_task) && - wait_state(state.task_queue, failed_orb_task, TASK_FAILED, &snapshot) && - lardon3d_feature_opencv_thread_count() == 3); + &state, image.image_id, &failed_orb_parameters, &failed_orb_task)); + CHECK(wait_state(state.task_queue, failed_orb_task, TASK_FAILED, &snapshot)); + CHECK(lardon3d_feature_opencv_thread_count() == 3); Lardon3DSiftExtractorParameters failed_sift_parameters = lardon3d_sift_precision_classic_v1(false); failed_sift_parameters.max_features = 511; uint64_t failed_sift_task = 0; + CHECK(setenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS", "once", 1) == 0); CHECK(lardon3d_project_enqueue_sift_extract( &state, image.image_id, &failed_sift_parameters, - &failed_sift_task) && - wait_state(state.task_queue, failed_sift_task, TASK_FAILED, &snapshot) && - lardon3d_feature_opencv_thread_count() == 3); + &failed_sift_task)); + CHECK(wait_state(state.task_queue, failed_sift_task, TASK_FAILED, &snapshot)); + CHECK(lardon3d_feature_opencv_thread_count() == 3); failed_sift_parameters.rootsift = true; failed_sift_parameters.max_features = 510; uint64_t failed_rootsift_task = 0; + CHECK(setenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS", "once", 1) == 0); CHECK(lardon3d_project_enqueue_sift_extract( &state, image.image_id, &failed_sift_parameters, - &failed_rootsift_task) && - wait_state(state.task_queue, failed_rootsift_task, TASK_FAILED, &snapshot) && - lardon3d_feature_opencv_thread_count() == 3 && - unsetenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS") == 0); + &failed_rootsift_task)); + CHECK(wait_state(state.task_queue, failed_rootsift_task, TASK_FAILED, &snapshot)); + CHECK(lardon3d_feature_opencv_thread_count() == 3); Lardon3DSiftExtractorParameters sift_parameters = lardon3d_sift_precision_classic_v1(false); sift_parameters.max_features = 512; diff --git a/tests/test_geometric_verification.c b/tests/test_geometric_verification.c index 582bcc0..1a7d7eb 100644 --- a/tests/test_geometric_verification.c +++ b/tests/test_geometric_verification.c @@ -172,6 +172,8 @@ static bool create_v11_database(const char *path) { lardon3d_project_db_close(database); return execute_sql(path, "PRAGMA foreign_keys=OFF;BEGIN IMMEDIATE;" + "DROP TABLE IF EXISTS feature_extract_batch_tasks;" + "DROP TABLE IF EXISTS raw_development_batch_tasks;" "DROP TABLE IF EXISTS capture_calibration_selections;" "DROP TABLE IF EXISTS optical_calibration_profiles;" "DROP TABLE IF EXISTS capture_optical_configurations;" diff --git a/tests/test_geometric_verifier_task.c b/tests/test_geometric_verifier_task.c index 0d7f4e3..03f91a7 100644 --- a/tests/test_geometric_verifier_task.c +++ b/tests/test_geometric_verifier_task.c @@ -878,6 +878,8 @@ static bool run_task_test(void) { CHECK(exec_sql( database_path, "PRAGMA foreign_keys=OFF;BEGIN IMMEDIATE;" + "DROP TABLE IF EXISTS feature_extract_batch_tasks;" + "DROP TABLE IF EXISTS raw_development_batch_tasks;" "DROP TABLE IF EXISTS capture_calibration_selections;" "DROP TABLE IF EXISTS optical_calibration_profiles;" "DROP TABLE IF EXISTS capture_optical_configurations;" diff --git a/tests/test_optical_profiles.c b/tests/test_optical_profiles.c index 71f21f2..a7468dc 100644 --- a/tests/test_optical_profiles.c +++ b/tests/test_optical_profiles.c @@ -252,7 +252,8 @@ static bool test_profiles_assignments_and_calibrations(void) { char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]; CHECK(lardon3d_project_db_open(path, &database, error) == LARDON3D_PROJECT_DB_OK); - CHECK(lardon3d_project_db_schema_version(database) == 23); + CHECK(lardon3d_project_db_schema_version(database) == + LARDON3D_PROJECT_DB_SCHEMA_VERSION); Lardon3DProjectDbScanSet scanset; CHECK(lardon3d_project_db_create_scanset(database, "mixed optical campaign", @@ -1146,6 +1147,7 @@ static bool downgrade_to_v22_fixture(const char *path) { return raw_sql( path, "PRAGMA foreign_keys=OFF;BEGIN IMMEDIATE;" + "DROP TABLE raw_development_batch_tasks;" "DROP TABLE capture_calibration_selections;" "DROP TABLE optical_calibration_profiles;" "DROP TABLE capture_optical_configurations;" @@ -1195,7 +1197,8 @@ static bool test_migration_rollback_retry_and_equivalence(void) { CHECK(lardon3d_project_db_open(migrated_path, &database, error) == LARDON3D_PROJECT_DB_OK && - lardon3d_project_db_schema_version(database) == 23); + lardon3d_project_db_schema_version(database) == + LARDON3D_PROJECT_DB_SCHEMA_VERSION); Lardon3DOpticalCaptureAssignment migrated_unresolved; CHECK(lardon3d_optical_capture_assignment_load(database, 1, &migrated_unresolved) == diff --git a/tests/test_pre_sfm_runner_options.cpp b/tests/test_pre_sfm_runner_options.cpp index 88a998f..1720d3b 100644 --- a/tests/test_pre_sfm_runner_options.cpp +++ b/tests/test_pre_sfm_runner_options.cpp @@ -365,7 +365,89 @@ int main() { recovery_entry.task_kind_version = 1; CHECK(!geometry_recovery_kind(recovery_entry, recovery_kind)); + Lardon3DProjectDbSelectedExecution selected{}; + selected.execution_id = 9; + selected.stage = LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS; + selected.next_item_index = 72; + selected.item_count = 100; + /* A legacy serial owner is completed only at the exact durable cursor. The + * subsequent no-owner decision hands the untouched suffix to one batch. */ + CHECK(choose_representation_action( + selected, RepresentationPendingKind::kLegacySerial) == + RepresentationAction::kRecoverLegacySerial); + selected.next_item_index = 73; + CHECK(choose_representation_action(selected, RepresentationPendingKind::kNone) == + RepresentationAction::kEnqueueBatch); + selected.next_item_index = 0; + CHECK(choose_representation_action(selected, RepresentationPendingKind::kNone) == + RepresentationAction::kEnqueueBatch); + selected.next_item_index = selected.item_count; + selected.stage = LARDON3D_SELECTED_EXECUTION_CALIBRATION; + CHECK(choose_representation_action(selected, RepresentationPendingKind::kNone) == + RepresentationAction::kCollect); + CHECK(choose_representation_action(selected, RepresentationPendingKind::kBatch) == + RepresentationAction::kRecoverBatch); + + Lardon3DProjectRecoveryEntry feature_pending{}; + feature_pending.status = LARDON3D_PROJECT_RECOVERABLE; + feature_pending.snapshot.recovery_state = TASK_PENDING; + std::snprintf(feature_pending.task_kind, sizeof(feature_pending.task_kind), + "%s", LARDON3D_FEATURE_EXTRACT_TASK_KIND); + feature_pending.task_kind_version = LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION; + FeaturePendingKind feature_kind{}; + CHECK(feature_recovery_kind(feature_pending, feature_kind)); + CHECK(feature_kind == FeaturePendingKind::kLegacySerial); + /* Completed representations defer exact Feature owners to extract_features, + * which performs the scientific-domain and selected-execution checks. */ + CHECK(representation_defers_feature_task(selected, feature_pending)); + selected.stage = LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS; + CHECK(!representation_defers_feature_task(selected, feature_pending)); + selected.stage = LARDON3D_SELECTED_EXECUTION_CALIBRATION; + --selected.next_item_index; + CHECK(!representation_defers_feature_task(selected, feature_pending)); + selected.next_item_index = selected.item_count; + ++feature_pending.task_kind_version; + CHECK(!representation_defers_feature_task(selected, feature_pending)); + std::snprintf(feature_pending.task_kind, sizeof(feature_pending.task_kind), + "%s", LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND); + feature_pending.task_kind_version = + LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION; + CHECK(representation_defers_feature_task(selected, feature_pending)); + CHECK(feature_recovery_kind(feature_pending, feature_kind)); + CHECK(feature_kind == FeaturePendingKind::kBatch); + feature_pending.status = LARDON3D_PROJECT_RECOVERY_UNKNOWN_TASK_KIND; + CHECK(!feature_recovery_kind(feature_pending, feature_kind)); + feature_pending.status = LARDON3D_PROJECT_RECOVERABLE; + std::snprintf(feature_pending.task_kind, sizeof(feature_pending.task_kind), + "%s", LARDON3D_TRACK_BUILDER_TASK_KIND); + feature_pending.task_kind_version = LARDON3D_TRACK_BUILDER_TASK_KIND_VERSION; + CHECK(!representation_defers_feature_task(selected, feature_pending)); + CHECK(!feature_recovery_kind(feature_pending, feature_kind)); + Options options; + CHECK(parse_case({"runner", "--resume-representations-existing", + "--project-dir", "/tmp/not-opened", + "--selected-execution-id", "9", + "--stop-after-representations"}, options)); + CHECK(options.resume_representations_existing && + options.selected_execution_id == 9 && + options.has_selected_execution_id && options.stop_after_representations); + for (const char *invalid_id : {"0", "-1", "+1", "01", "1x"}) { + options = {}; + CHECK(!parse_case({"runner", "--resume-representations-existing", + "--project-dir", "/tmp/not-opened", + "--selected-execution-id", invalid_id}, options)); + } + options = {}; + CHECK(!parse_case({"runner", "--resume-representations-existing", + "--project-dir", "/tmp/not-opened"}, options)); + options = {}; + CHECK(!parse_case({"runner", "--resume-representations-existing", + "--project-dir", "/tmp/not-opened", + "--selected-execution-id", "9", "--cpu-budget", "2"}, + options)); + + options = {}; CHECK(parse_case({"runner", "--resume-pre-gv-existing", "--project-dir", "/tmp/not-opened"}, options)); CHECK(options.matcher_mode == MatcherMode::kAuto && diff --git a/tests/test_project_db.c b/tests/test_project_db.c index 187d587..2d5cfb2 100644 --- a/tests/test_project_db.c +++ b/tests/test_project_db.c @@ -22,9 +22,11 @@ } while (0) /* Historical migration fixtures are produced from a temporary current DB. - * Remove every additive v23 object first so the fixture really is historical + * Remove every additive v23/v24/v25 object first so the fixture really is historical * rather than a lower version number with future tables left behind. */ #define DROP_OPTICAL_V23_SQL \ + "DROP TABLE IF EXISTS feature_extract_batch_tasks;" \ + "DROP TABLE IF EXISTS raw_development_batch_tasks;" \ "DROP TABLE IF EXISTS capture_calibration_selections;" \ "DROP TABLE IF EXISTS optical_calibration_profiles;" \ "DROP TABLE IF EXISTS capture_optical_configurations;" \ @@ -131,7 +133,7 @@ static bool create_future_database(const char *path) { sqlite3_exec( connection, "CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);" - "INSERT INTO metadata VALUES('schema_version',24);", + "INSERT INTO metadata VALUES('schema_version',26);", NULL, NULL, NULL) == SQLITE_OK; return sqlite3_close(connection) == SQLITE_OK && ok; } @@ -907,6 +909,7 @@ static bool run_test(void) { char v13_path[512], true_v14_path[512], failed_v14_path[512], failed_v15_path[512], true_v15_path[512], v17_path[512], failed_v18_path[512], failed_v19_path[512], raw_missing_table_path[512]; + char v24_path[512]; CHECK(snprintf(database_path, sizeof(database_path), "%s/project.db", directory) > 0); CHECK(snprintf(artifact_path, sizeof(artifact_path), "%s/artifact.bin", @@ -956,9 +959,34 @@ static bool run_test(void) { "%s/failed-v19.db", directory) > 0); CHECK(snprintf(raw_missing_table_path, sizeof(raw_missing_table_path), "%s/raw-missing-table.db", directory) > 0); + CHECK(snprintf(v24_path, sizeof(v24_path), "%s/v24.db", directory) > 0); char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]; Lardon3DProjectDb *database = NULL; + /* v25 is DDL-only: an injected failure rolls back table and marker, and a + * retry creates no fabricated association or historical cursor. */ + CHECK(lardon3d_project_db_open(v24_path, &database, error) == LARDON3D_PROJECT_DB_OK); + lardon3d_project_db_close(database); + database = NULL; + CHECK(execute_test_sql(v24_path, + "DROP TABLE feature_extract_batch_tasks;" + "UPDATE metadata SET value=24 WHERE key='schema_version';")); + CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V25", "1", 1) == 0); + CHECK(lardon3d_project_db_open(v24_path, &database, error) != LARDON3D_PROJECT_DB_OK); + CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V25") == 0); + CHECK(query_integer(v24_path, "SELECT value FROM metadata WHERE key='schema_version'", 24)); + CHECK(query_integer(v24_path, + "SELECT count(*) FROM sqlite_master WHERE type='table' AND " + "name='feature_extract_batch_tasks'", 0)); + database = NULL; + Lardon3DProjectDbResult v25_retry = lardon3d_project_db_open(v24_path, &database, error); + if (v25_retry != LARDON3D_PROJECT_DB_OK) + fprintf(stderr, "v25 retry: %d %s\n", v25_retry, error); + CHECK(v25_retry == LARDON3D_PROJECT_DB_OK); + CHECK(lardon3d_project_db_schema_version(database) == LARDON3D_PROJECT_DB_SCHEMA_VERSION); + lardon3d_project_db_close(database); + database = NULL; + CHECK(query_integer(v24_path, "SELECT count(*) FROM feature_extract_batch_tasks", 0)); CHECK(lardon3d_project_db_open(database_path, &database, error) == LARDON3D_PROJECT_DB_OK); CHECK(database && lardon3d_project_db_schema_version(database) == @@ -2676,6 +2704,7 @@ static bool run_test(void) { CHECK(unlink(failed_v18_path) == 0); CHECK(unlink(failed_v19_path) == 0); CHECK(unlink(raw_missing_table_path) == 0); + CHECK(unlink(v24_path) == 0); CHECK(rmdir(directory) == 0); return true; } diff --git a/tests/test_raw_development_task.cpp b/tests/test_raw_development_task.cpp index 4fe52d7..6b84f8d 100644 --- a/tests/test_raw_development_task.cpp +++ b/tests/test_raw_development_task.cpp @@ -89,6 +89,13 @@ int main() { Lardon3DTask *task = lardon3d_project_create_raw_development_task( &state, capture.capture_id, raw_asset.asset_id, &task_id); CHECK(task != nullptr && task_id != 0); + Lardon3DResourceEstimate single_estimate{}; + CHECK(lardon3d_task_resource_estimate(task, &single_estimate)); + CHECK(single_estimate.memory_fixed_bytes >= (UINT64_C(2) << 30) && + single_estimate.memory_bytes_per_item == 0 && + single_estimate.minimum_batch_size == 1 && + single_estimate.maximum_batch_size == 1 && + single_estimate.desired_cpu_threads == 1); Lardon3DProjectDbTask generic{}; Lardon3DProjectDbRawDevelopmentTask typed{}; CHECK(lardon3d_project_db_load_task(database, task_id, &generic) == LARDON3D_PROJECT_DB_OK); @@ -154,6 +161,64 @@ int main() { database, &snapshot, LARDON3D_RAW_DEVELOPMENT_TASK_KIND, 2, nullptr, &typed, 7) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT); + /* These minimal selected rows isolate the v24 parent association contract; + * item science remains covered by selected-execution tests and the frozen + * per-Capture developer tests. */ + sqlite3 *batch_seed = nullptr; + CHECK(sqlite3_open(database_path.c_str(), &batch_seed) == SQLITE_OK); + CHECK(sqlite3_exec(batch_seed, "PRAGMA foreign_keys=OFF;" + "INSERT INTO selected_executions(execution_id,quality_task_id," + "campaign_task_id,stage,next_item_index,item_count,created_at)" + "VALUES(9001,7001,7002,1,0,1,7),(9002,7003,7004,1,0,1,7)", + nullptr, nullptr, nullptr) == SQLITE_OK); + CHECK(sqlite3_close(batch_seed) == SQLITE_OK); + uint64_t batch_task_id = 0; + Lardon3DTask *batch_task = lardon3d_project_create_raw_development_batch_task( + &state, 9001, &batch_task_id); + CHECK(batch_task != nullptr && batch_task_id != 0); + Lardon3DResourceEstimate batch_estimate{}; + CHECK(lardon3d_task_resource_estimate(batch_task, &batch_estimate)); + /* The pixel-bounded participant allowance must permit the full safe window + * from a 7 GiB post-reserve budget; a fixed context charge would incorrectly + * reduce this exact boundary to seven participants. */ + CHECK(batch_estimate.memory_fixed_bytes == 0 && + batch_estimate.memory_bytes_per_item == (UINT64_C(896) << 20) && + batch_estimate.minimum_batch_size == 1 && + batch_estimate.maximum_batch_size == 8 && + batch_estimate.desired_cpu_threads == 8 && + batch_estimate.memory_bytes_per_item * batch_estimate.maximum_batch_size == + (UINT64_C(7) << 30) && + batch_estimate.desired_gpu_slots == 0 && + batch_estimate.desired_io_slots == 1 && + batch_estimate.task_class == LARDON3D_RESOURCE_TASK_MIXED); + Lardon3DProjectDbRawDevelopmentBatchTask batch_typed{}; + CHECK(lardon3d_project_db_load_raw_development_batch_task( + database, batch_task_id, &batch_typed) == LARDON3D_PROJECT_DB_OK && + batch_typed.selected_execution_id == 9001); + Lardon3DTaskDurableSnapshot batch_snapshot{}; + CHECK(lardon3d_task_durable_snapshot(batch_task, &batch_snapshot)); + Lardon3DTaskKindBinding batch_binding{}; + CHECK(lardon3d_raw_development_batch_task_reconstruct( + &batch_snapshot, &reconstruction, &batch_binding)); + CHECK(batch_binding.callback != nullptr && batch_binding.userdata != nullptr && + batch_binding.userdata_destroy != nullptr); + batch_binding.userdata_destroy(batch_binding.userdata); + CHECK(lardon3d_project_db_record_raw_development_batch_task( + database, &batch_snapshot, + LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND, + LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION, nullptr, + &batch_typed, 7) == LARDON3D_PROJECT_DB_OK); + batch_typed.selected_execution_id = 9002; + CHECK(lardon3d_project_db_record_raw_development_batch_task( + database, &batch_snapshot, + LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND, + LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION, nullptr, + &batch_typed, 7) == LARDON3D_PROJECT_DB_CONSTRAINT); + CHECK(lardon3d_project_db_load_raw_development_batch_task( + database, batch_task_id, &batch_typed) == LARDON3D_PROJECT_DB_OK && + batch_typed.selected_execution_id == 9001); + lardon3d_task_destroy(batch_task); + sqlite3 *raw_database = nullptr; char update_mismatch_kind[128]; std::snprintf(update_mismatch_kind, sizeof(update_mismatch_kind), diff --git a/tests/test_selected_execution.c b/tests/test_selected_execution.c index 29ce0dd..90cb0a5 100644 --- a/tests/test_selected_execution.c +++ b/tests/test_selected_execution.c @@ -18,6 +18,20 @@ static bool sql(const char *path, const char *text) { return sqlite3_close(connection) == SQLITE_OK && code == SQLITE_OK; } +static void feature_asset_path( + const unsigned char hash[LARDON3D_PROJECT_DB_SHA256_SIZE], + char path[LARDON3D_PROJECT_DB_PATH_CAPACITY]) { + static const char digits[] = "0123456789abcdef"; + char hex[LARDON3D_PROJECT_DB_SHA256_SIZE * 2 + 1]; + for (size_t index = 0; index < LARDON3D_PROJECT_DB_SHA256_SIZE; ++index) { + hex[index * 2] = digits[hash[index] >> 4]; + hex[index * 2 + 1] = digits[hash[index] & 15U]; + } + hex[sizeof(hex) - 1] = '\0'; + (void)snprintf(path, LARDON3D_PROJECT_DB_PATH_CAPACITY, + "assets/features/%c%c/%s", hex[0], hex[1], hex); +} + static bool seed_v21(const char *path) { Lardon3DProjectDb *database = NULL; char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]; @@ -50,6 +64,8 @@ static bool seed_v21(const char *path) { "INSERT INTO sparse_calibration_scope_images VALUES(1,1,1);" /* This fixture is a true v21 database; future additive objects must not remain merely because it was generated from a temporary current DB. */ + "DROP TABLE feature_extract_batch_tasks;" + "DROP TABLE raw_development_batch_tasks;" "DROP TABLE capture_calibration_selections;" "DROP TABLE optical_calibration_profiles;" "DROP TABLE capture_optical_configurations;" @@ -89,6 +105,29 @@ static bool run(void) { if (migration_result != LARDON3D_PROJECT_DB_OK) fprintf(stderr, "retry migration result=%d error=%s\n", (int)migration_result, error); CHECK(migration_result == LARDON3D_PROJECT_DB_OK); + CHECK(lardon3d_project_db_schema_version(database) == + LARDON3D_PROJECT_DB_SCHEMA_VERSION); + CHECK(sql(path, "SELECT 1 FROM raw_development_batch_tasks WHERE 0;")); + lardon3d_project_db_close(database); + database = NULL; + CHECK(sql(path, "DROP TABLE feature_extract_batch_tasks;" + "DROP TABLE raw_development_batch_tasks;" + "UPDATE metadata SET value=23 WHERE key='schema_version';")); + CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V24", "1", 1) == 0); + CHECK(lardon3d_project_db_open(path, &database, error) != LARDON3D_PROJECT_DB_OK); + CHECK(database == NULL); + CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V24") == 0); + CHECK(sqlite3_open(path, &raw) == SQLITE_OK); + CHECK(sqlite3_prepare_v2(raw, + "SELECT value FROM metadata WHERE key='schema_version'", -1, + &query, NULL) == SQLITE_OK); + CHECK(sqlite3_step(query) == SQLITE_ROW && sqlite3_column_int(query, 0) == 23); + sqlite3_finalize(query); + CHECK(sqlite3_close(raw) == SQLITE_OK); + migration_result = lardon3d_project_db_open(path, &database, error); + if (migration_result != LARDON3D_PROJECT_DB_OK) + fprintf(stderr, "v24 retry result=%d error=%s\n", (int)migration_result, error); + CHECK(migration_result == LARDON3D_PROJECT_DB_OK); CHECK(lardon3d_project_db_schema_version(database) == LARDON3D_PROJECT_DB_SCHEMA_VERSION); Lardon3DProjectDbSelectedExecutionItem item = { @@ -148,6 +187,85 @@ static bool run(void) { &execution) == LARDON3D_PROJECT_DB_OK && execution.execution_id == execution_id && execution.stage == LARDON3D_SELECTED_EXECUTION_READY); + + Lardon3DTaskDurableSnapshot batch_snapshot = { + .id = 3, + .estimate = {.memory_fixed_bytes = 64U * 1024U * 1024U, + .memory_bytes_per_item = 512U * 1024U * 1024U, + .minimum_batch_size = 1, + .maximum_batch_size = 12, + .desired_cpu_threads = 12, + .desired_io_slots = 1, + .task_class = LARDON3D_RESOURCE_TASK_CPU}, + .saved_state = TASK_PENDING, + .recovery_state = TASK_PENDING, + }; + (void)snprintf(batch_snapshot.name, sizeof(batch_snapshot.name), "Feature batch"); + Lardon3DProjectDbFeatureExtractBatchTask batch = { + .task_id = 3, + .selected_execution_id = execution_id, + .next_item_index = 2, + .extractor_version = 1, + .max_features = 512, + .pyramid_levels = 4, + .fast_threshold = 10, + .parameter_fingerprint = {7}, + }; + (void)snprintf(batch.extractor_kind, sizeof(batch.extractor_kind), "orb"); + /* An invalid new cursor must roll back both generic and typed state; v25 + cannot publish a Task whose durable prefix exceeds its immutable domain. */ + CHECK(lardon3d_project_db_record_feature_extract_batch_task( + database, &batch_snapshot, "features.extract.batch", 1, NULL, + &batch, 20) == LARDON3D_PROJECT_DB_CONSTRAINT); + Lardon3DProjectDbTask absent_batch_task; + CHECK(lardon3d_project_db_load_task(database, 3, &absent_batch_task) == + LARDON3D_PROJECT_DB_NOT_FOUND); + + batch.next_item_index = 0; + CHECK(lardon3d_project_db_record_feature_extract_batch_task( + database, &batch_snapshot, "features.extract.batch", 1, NULL, + &batch, 21) == LARDON3D_PROJECT_DB_OK); + Lardon3DProjectDbFeatureExtractBatchTask loaded_batch; + CHECK(lardon3d_project_db_load_feature_extract_batch_task(database, 3, + &loaded_batch) == + LARDON3D_PROJECT_DB_OK && + loaded_batch.selected_execution_id == execution_id && + loaded_batch.next_item_index == 0 && + memcmp(loaded_batch.parameter_fingerprint, + batch.parameter_fingerprint, 32) == 0); + CHECK(lardon3d_project_db_advance_feature_extract_batch_task(database, 3, 0, + 1) == + LARDON3D_PROJECT_DB_CONSTRAINT); + + unsigned char source_hash[32] = {0}; + unsigned char feature_hash[32] = {9}; + char feature_path[LARDON3D_PROJECT_DB_PATH_CAPACITY]; + feature_asset_path(feature_hash, feature_path); + Lardon3DProjectDbFeatureSet feature_set; + CHECK(lardon3d_project_db_register_feature_set( + database, 1, "orb", 1, batch.parameter_fingerprint, source_hash, + 1, 1, 32, feature_hash, feature_path, 32, + LARDON3D_DB_FEATURE_ASSET_DURABLE, 3, 22, &feature_set) == + LARDON3D_PROJECT_DB_OK); + CHECK(lardon3d_project_db_advance_feature_extract_batch_task(database, 3, 0, + 1) == + LARDON3D_PROJECT_DB_OK && + lardon3d_project_db_advance_feature_extract_batch_task(database, 3, 0, + 1) == + LARDON3D_PROJECT_DB_OK); + /* A lagging generic checkpoint may repeat cursor zero, but the typed prefix + remains at one. Changing any immutable ORB field is a durable conflict. */ + CHECK(lardon3d_project_db_record_feature_extract_batch_task( + database, &batch_snapshot, "features.extract.batch", 1, NULL, + &batch, 23) == LARDON3D_PROJECT_DB_OK && + lardon3d_project_db_load_feature_extract_batch_task(database, 3, + &loaded_batch) == + LARDON3D_PROJECT_DB_OK && + loaded_batch.next_item_index == 1); + batch.max_features++; + CHECK(lardon3d_project_db_record_feature_extract_batch_task( + database, &batch_snapshot, "features.extract.batch", 1, NULL, + &batch, 24) == LARDON3D_PROJECT_DB_CONSTRAINT); lardon3d_project_db_close(database); database = NULL; diff --git a/tests/test_tui_model.c b/tests/test_tui_model.c index 8427cca..6a7ae8f 100644 --- a/tests/test_tui_model.c +++ b/tests/test_tui_model.c @@ -87,8 +87,10 @@ test_stage_matrix(void) } cases[] = { {"acquisition_campaign.run", LARDON3D_TUI_STAGE_ACQUISITION}, {"raw.develop", LARDON3D_TUI_STAGE_RAW}, + {"raw.develop.batch", LARDON3D_TUI_STAGE_RAW}, {"photo_quality.triage", LARDON3D_TUI_STAGE_QUALITY}, {"features.extract", LARDON3D_TUI_STAGE_FEATURES}, + {"features.extract.batch", LARDON3D_TUI_STAGE_FEATURES}, {"features.extract.sift", LARDON3D_TUI_STAGE_FEATURES}, {"features.extract.rootsift", LARDON3D_TUI_STAGE_FEATURES}, {"visual_index.update", LARDON3D_TUI_STAGE_VISUAL_INDEX},