pre-sfm: freeze A6000 v25 resource scaling proof

This commit is contained in:
fy59 2026-09-02 09:18:13 +02:00
parent 7e93558f5a
commit cff9b88536
35 changed files with 3004 additions and 478 deletions

377
AGENTS.md
View file

@ -4,15 +4,15 @@
- The root `README.md` is the index of project documentation. - The root `README.md` is the index of project documentation.
- Canonical documents under `docs/**` define scientific contracts, architecture, - Canonical documents under `docs/**` define scientific contracts, architecture,
FROZEN invariants, persistence semantics, lifecycle state, and roadmap FROZEN invariants, persistence semantics, lifecycle state, resource policy,
ordering. and roadmap ordering.
- Before changing an architectural, scientific, persistence, runtime, or - Before changing an architectural, scientific, persistence, runtime, or
resource-sensitive area, identify and read the relevant canonical document resource-sensitive area, identify and read the relevant canonical document
and its applicable invariants. and its applicable invariants.
- FROZEN documentation and contracts must never be changed silently to fit an - FROZEN documentation and contracts must never be changed silently to fit an
implementation. implementation.
- A lower-level implementation convenience never overrides a higher-level - A lower-level implementation convenience never overrides a higher-level
canonical contract. canonical contract or an explicit human decision.
- Lifecycle markers such as `PLANNED`, `IMPLEMENTED`, - Lifecycle markers such as `PLANNED`, `IMPLEMENTED`,
`VALIDATION PENDING`, and `PASS/FROZEN` describe repository state and must `VALIDATION PENDING`, and `PASS/FROZEN` describe repository state and must
match actual implementation and validation evidence. match actual implementation and validation evidence.
@ -30,11 +30,12 @@ through an explicitly authorized, explicitly scoped human ticket:
- Gates AG — PASS/FROZEN - Gates AG — PASS/FROZEN
- Track Model — PASS/FROZEN - Track Model — PASS/FROZEN
- Track Builder — PASS/FROZEN - Track Builder scientific contract — PASS/FROZEN
- F0 — PASS/FROZEN - F0 — PASS/FROZEN
- Phase H v1 — PASS/FROZEN - Phase H v1 — PASS/FROZEN
- MVS-M1 — PASS/FROZEN - MVS-M1 — PASS/FROZEN
- Project DB v22 scientific/persistence foundation — PASS/FROZEN - Project DB v22 scientific/persistence foundation — PASS/FROZEN
- Project DB v23 generic optical-context overlay — PASS/FROZEN
- Calibration Bootstrap v1 — PASS/FROZEN - Calibration Bootstrap v1 — PASS/FROZEN
- Selected Scientific Execution — PASS/FROZEN - Selected Scientific Execution — PASS/FROZEN
- Photo Quality Triage / Acquisition Selection — PASS/FROZEN - Photo Quality Triage / Acquisition Selection — PASS/FROZEN
@ -43,25 +44,30 @@ through an explicitly authorized, explicitly scoped human ticket:
- S3 Capture / Acquisition Ingestion — PASS/FROZEN - S3 Capture / Acquisition Ingestion — PASS/FROZEN
- Durable Acquisition-Campaign Execution — PASS/FROZEN - Durable Acquisition-Campaign Execution — PASS/FROZEN
- Global Maintenance Audit — PASS/FROZEN - Global Maintenance Audit — PASS/FROZEN
- Real S21 Tracks scientific result — PASS/FROZEN
Detailed subcontracts remain defined by their canonical documents. This file Detailed subcontracts remain defined by their canonical documents. This file
does not duplicate every S3 substage, scientific threshold, migration detail, must not duplicate every S3 substage, scientific threshold, migration detail,
or persistence format. or persistence format.
Project DB v23 is the current additive optical-context overlay. It preserves Project DB v25 is the current additive operational schema in the active Feature
the v22 scientific/persistence foundation and must not infer or backfill optical batch tranche. It preserves v24 RAW batch, the v23 optical overlay and the v22
identity from historical data. Historical references to older Project DB scientific/persistence foundation. Its authorized purpose is limited to typed
versions remain valid when they describe the actual historical contract or durable persistence for `features.extract.batch/1` through
migration path. Do not rewrite legitimate v16v22 history merely because v23 `feature_extract_batch_tasks`; it adds no scientific identity and must not
is current. reinterpret historical rows. Until the v25/Feature-batch tranche has completed
its real-data proof and final review, its lifecycle must remain truthful rather
than being marked `PASS/FROZEN` prematurely. Historical references to older
Project DB versions remain valid where they describe the actual historical
contract or migration path.
The global maintenance implementation, fresh portable/Vulkan/sanitizer/ The global maintenance implementation, fresh portable/Vulkan/sanitizer/
concurrency validation and independent final review are acquired. Its lifecycle concurrency validation and independent final review are acquired. Its lifecycle
is `GLOBAL_MAINTENANCE_AUDIT=PASS/FROZEN`. The review independently passed the is `GLOBAL_MAINTENANCE_AUDIT=PASS/FROZEN`. The review independently passed the
portable build, 64/64 complete suite, 15/15 focused matrix, 76/76 strict header portable build, complete suites, focused matrices, strict public-header probes,
probes across 19 modified/new public headers, ABI and production-seam checks, ABI and production-seam checks, retained-manifest verification and diff
retained-manifest verification and diff validation, with zero blocking findings. validation with zero blocking findings. Do not reopen this boundary or infer a
Do not reopen this boundary or infer a new scientific policy from the freeze. new scientific policy from the freeze.
The canonical review checkpoint is tag `global-maintenance-2026-09-01` at The canonical review checkpoint is tag `global-maintenance-2026-09-01` at
commit `b84f860d868c66d9ee84b85ceb1bc6480b95aca5`; its detailed evidence is commit `b84f860d868c66d9ee84b85ceb1bc6480b95aca5`; its detailed evidence is
@ -88,11 +94,13 @@ When a ticket declares `NO_NEW_SUBSYSTEM`, do not introduce an unrelated:
- daemon. - daemon.
This restriction is ticket-scoped. It does not mean that a subsystem already This restriction is ticket-scoped. It does not mean that a subsystem already
present in the canonical architecture can never be used by later authorized present in the canonical architecture can never be used or extended by later
work. authorized work.
Never reopen a FROZEN scientific or architectural decision merely to simplify Never reopen a FROZEN scientific decision merely to simplify an
an implementation. implementation. Operational/resource contracts may be reopened only by explicit
human authority and must preserve the scientific result exactly unless the
human ticket explicitly says otherwise.
## 3. Scope and Git discipline ## 3. Scope and Git discipline
@ -157,8 +165,8 @@ require a clean worktree unless the ticket explicitly requires one.
allocation, solver execution or output mutation; do not alter the FROZEN allocation, solver execution or output mutation; do not alter the FROZEN
defaults, encodings or fingerprints to accommodate an unsafe cast. defaults, encodings or fingerprints to accommodate an unsafe cast.
Compiler success alone is not proof of API, ABI, persistence, or scientific Compiler success alone is not proof of API, ABI, persistence, resource, or
contract correctness. scientific contract correctness.
## 5. Identity discipline ## 5. Identity discipline
@ -217,13 +225,132 @@ explicitly defines such an identity.
- Extend validated abstractions rather than rewriting validated modules. - Extend validated abstractions rather than rewriting validated modules.
- Reuse the existing Task / Queue / Scheduler / Resource Governor ownership - Reuse the existing Task / Queue / Scheduler / Resource Governor ownership
model rather than creating parallel runtime infrastructure. model rather than creating parallel runtime infrastructure.
- System stability and responsiveness take priority over throughput.
### Canonical utilization objective — MAXIMUM SAFE USEFUL THROUGHPUT
Lardon3D is a throughput-oriented workstation application. The Resource
Governor MUST maximize safe and useful utilization for every production Task
after preserving only the interactive host reserve needed for normal use of:
- Arch Linux / Sway and normal desktop services;
- Firefox;
- normal audio/music playback;
- lightweight interactive use of the workstation.
All CPU, RAM, I/O capacity and validated accelerator capacity beyond that
interactive reserve belongs to the active Lardon3D workload when useful work
exists.
On the current reference host, the normal observed policy outcome is
approximately:
```text
16 logical CPUs total
4 logical CPUs reserved for the interactive host
12 logical CPUs available to the compute pool
~3 GiB MemAvailable preserved as the hard RAM reserve
Radeon 780M UMA available to validated/useful GPU backends
```
These values are reference-host outcomes, NOT portable constants. Topology,
process affinity, smaller hosts, memory pressure, I/O pressure and future
hardware must be handled dynamically. A future 32-thread host must not inherit a
12-thread product ceiling merely because the reference host exposed 12 compute
threads.
The optimization target is:
```text
MAXIMUM SAFE USEFUL THROUGHPUT
```
It is NOT:
```text
minimum resource usage
fixed CPU counts
CPU/GPU utilization percentage for its own sake
preserving historical CPU1/batch1 descriptors without evidence
```
The Governor remains the sole production resource authority. Normal users do
not choose CPU count, worker count, batch, inflight depth, GPU backend, scratch
mode, or RAM budget for authoritative execution.
A production Task may deliberately use less than the currently available
compute pool only when concrete evidence establishes at least one relevant
constraint, such as:
- true scientific or dependency serialism;
- measured useful-scaling knee;
- memory bound;
- I/O saturation;
- validated GPU execution making additional CPU work useless;
- deterministic publication constraint that cannot be separated safely from
preparation;
- another explicit and measured resource limitation.
The reason must be represented by the Task/Governor contract or canonical
resource documentation where non-obvious. A historical descriptor by itself is
not evidence.
### SERIALISM_REQUIRES_PROOF
`SERIALISM_REQUIRES_PROOF` is canonical operational policy.
Per-item atomicity does NOT imply cross-item serialization. Owner-only or
ordered durable publication does NOT imply serial preparation. When multiple
independent work units exist and exact science/persistence semantics are
preserved, the Task must expose bounded concurrency to the Governor.
A long-running `CPU1` or `batch1` path with independent executable work and
available safe resources is an operational defect until one of the documented
measured limitations above proves otherwise.
Do not solve this by creating another scheduler, uncontrolled worker pool,
unbounded `std::async`, detached threads, or resources outside Governor
accounting. Prefer the established bounded pattern where appropriate:
```text
one admitted owner Task
-> bounded independent participants/preparation
-> join all participants
-> deterministic owner-only publication
```
Every participant must be accounted, bounded, cancellable and joined. Shared
SQLite access must preserve the existing ownership/serialization contracts.
### CPU, GPU, RAM and I/O policy
- CPU: expose truthful minimum/useful/safe concurrency and allow Governor to use
the largest currently safe/useful value. Do not hardcode the reference-host
12-thread result.
- GPU: a validated AND useful production GPU backend MUST be preferred when it
is available and Governor-safe. Do not invent or promote an unvalidated GPU
path merely to make the GPU busy.
- RAM: use available memory aggressively for useful work while preserving the
hard interactive reserve, approximately 3 GiB `MemAvailable` on the reference
host. The 34 GiB band is a pressure/conservatism zone, not a permanent extra
1 GiB subtraction from every Task.
- UMA: iGPU allocations count exactly once against host RAM.
- swap/zram: pressure and safety mechanisms, never admitted working RAM.
- scratch SSD: optional external storage, never RAM. A Task may consume it only
through an explicitly validated Governor-owned scratch contract.
- I/O: if additional concurrency no longer improves useful throughput because
storage or another I/O boundary is saturated, the measured knee is the right
useful limit. Do not force CPU saturation for appearance.
- Pressure: CPU/memory/I/O PSI and active swap-in/out deltas may reduce
admission. When pressure clears, useful resources must be re-admitted rather
than remaining permanently throttled.
System responsiveness is therefore protected by the explicit host reserve and
pressure feedback; it is not a justification for leaving additional safe/useful
compute idle.
- Bound memory, buffers, files, descriptors, processes, threads, captured - Bound memory, buffers, files, descriptors, processes, threads, captured
output, parser work, temporary storage, and staging reasonably for the output, parser work, temporary storage, and staging honestly for the
operation. operation.
- Memory shared with an iGPU counts against host RAM.
- zram and swap are pressure/safety mechanisms, not normal working-memory
budgets.
- An operational hardware/resource bound must not accidentally become a - An operational hardware/resource bound must not accidentally become a
scientific dataset-size limit. scientific dataset-size limit.
- Do not invent a new global resource subsystem inside a ticket that defers it. - Do not invent a new global resource subsystem inside a ticket that defers it.
@ -237,12 +364,16 @@ Resource-sensitive work must identify, where relevant:
- resource owner; - resource owner;
- admission point; - admission point;
- bound or budget; - minimum/useful/safe CPU and batch capability;
- fixed/per-participant/transient memory budget;
- reservation lifetime; - reservation lifetime;
- release point; - release point;
- failure cleanup; - failure cleanup;
- cancellation cleanup; - cancellation cleanup;
- whether the bound is operational or scientific. - I/O/scratch ownership;
- validated GPU capability;
- whether a bound is operational or scientific;
- measured reason for intentional under-utilization.
The reviewed external USB SSD controller is the authorized physical-lifecycle The reviewed external USB SSD controller is the authorized physical-lifecycle
boundary for the exact UDisks Drive/label/UUID contract. Its validated snapshot boundary for the exact UDisks Drive/label/UUID contract. Its validated snapshot
@ -251,11 +382,10 @@ production orchestrator for scratch-lease acquire/release. The controller does
not replace the Governor or invent Task scratch eligibility, and swap/scratch not replace the Governor or invent Task scratch eligibility, and swap/scratch
never become RAM. The application lifetime order is strict: destroy/join the never become RAM. The application lifetime order is strict: destroy/join the
Queue so every Task lease is released, checked-join/unregister the SSD binding, Queue so every Task lease is released, checked-join/unregister the SSD binding,
destroy the controller, then destroy the Governor. The current fourteen Task destroy the controller, then destroy the Governor. SSD availability is a
kinds have no scratch consumer, so availability is capability, not fabricated capability, not fabricated Task usage. Do not add ad-hoc discovery, mounting,
usage. Do not add ad-hoc discovery, mounting, formatting, `swapon`, cleanup, formatting, `swapon`, cleanup, force-drain, shell commands, or a second
force-drain, shell commands, or a second resource/scheduling subsystem outside resource/scheduling subsystem outside the reviewed controller/Governor APIs.
the reviewed controller/Governor APIs.
Snapshot conversion is fail-closed by physical state. Any pairing or authority Snapshot conversion is fail-closed by physical state. Any pairing or authority
requires current detection of the Drive and both UUID-bearing partitions, requires current detection of the Drive and both UUID-bearing partitions,
@ -426,7 +556,8 @@ comment.
Where relevant, comments must preserve the established distinction between: Where relevant, comments must preserve the established distinction between:
- immutable SOURCE RAW; - immutable SOURCE RAW;
- deterministic DERIVED representation; - camera JPEG SOURCE;
- deterministic DERIVED RAW representation;
- RAW Policy v1; - RAW Policy v1;
- L3DRAWD1 identity; - L3DRAWD1 identity;
- metadata-only acquisition processing; - metadata-only acquisition processing;
@ -438,6 +569,12 @@ Where relevant, comments must preserve the established distinction between:
- bounded secondary JPEG validation; - bounded secondary JPEG validation;
- zero-only permitted MPF gaps/trailer. - zero-only permitted MPF gaps/trailer.
For the current A6000 selected scientific execution, paired camera JPEGs are
valid Capture source assets and preferred fast proxies for Photo Quality, but
the FROZEN geometry representation remains deterministic RAW-derived PNG. Do
not substitute camera-JPEG geometry without a separately authorized and proven
scientific/calibration equivalence tranche.
Do not imply that metadata validation performs pixel decoding when it does not. Do not imply that metadata validation performs pixel decoding when it does not.
### Task / Queue / Scheduler / Governor ### Task / Queue / Scheduler / Governor
@ -454,6 +591,8 @@ In particular:
- Resource Reservation belongs to the currently admitted bounded execution. - Resource Reservation belongs to the currently admitted bounded execution.
- `sequence_break` is an execution boundary that releases/re-establishes - `sequence_break` is an execution boundary that releases/re-establishes
admission according to runtime semantics. admission according to runtime semantics.
- Per-item atomicity does not imply cross-item serialization.
- Serial publication may coexist with Governor-admitted parallel preparation.
Do not imply that a long campaign reserves resources for its entire lifetime Do not imply that a long campaign reserves resources for its entire lifetime
when execution is group-bounded. when execution is group-bounded.
@ -466,16 +605,18 @@ resource ownership understandable from source.
Where applicable document: Where applicable document:
- RAM ownership/bound; - RAM ownership/bound;
- CPU admission; - CPU minimum/useful/safe admission;
- GPU admission; - GPU admission;
- I/O ownership; - I/O ownership;
- process/thread count; - process/thread count;
- per-participant memory;
- file-descriptor lifetime; - file-descriptor lifetime;
- temporary/scratch storage ownership; - temporary/scratch storage ownership;
- reservation lifetime; - reservation lifetime;
- release point; - release point;
- failure cleanup; - failure cleanup;
- cancellation cleanup. - cancellation cleanup;
- measured scaling knee or reason for serial execution.
An operational hardware/resource bound MUST NOT accidentally become a An operational hardware/resource bound MUST NOT accidentally become a
scientific dataset-size limit. scientific dataset-size limit.
@ -502,6 +643,12 @@ The residual pre-return S3-E crash window remains intentional and documented:
if S3-E creates a Capture internally and the process dies before S3-E returns if S3-E creates a Capture internally and the process dies before S3-E returns
the `capture_id`, campaign execution must not guess that Capture identity. the `capture_id`, campaign execution must not guess that Capture identity.
For `raw.develop.batch/1`, typed `task_id -> selected_execution_id` persistence
must remain distinct from generic Task runtime state. Independent RAW
participants may prepare concurrently, but owner publication and selected-item
cursor advancement remain deterministic and ordered by the selected execution
contract.
### C / C++ ABI boundaries ### C / C++ ABI boundaries
Where C++ implements a public C interface or C Task callback, comments should Where C++ implements a public C interface or C Task callback, comments should
@ -524,7 +671,8 @@ Comments should state WHAT CONTRACT the fixture proves, for example:
- ambiguity; - ambiguity;
- scientific identity conflict; - scientific identity conflict;
- cross-ScanSet rejection; - cross-ScanSet rejection;
- resource-pressure/admission condition. - resource-pressure/admission condition;
- deliberate CPU1 baseline used only for a scaling comparison.
Do not narrate ordinary test mechanics. Do not narrate ordinary test mechanics.
@ -552,9 +700,9 @@ For every future implementation tranche:
4. update canonical documentation when behavior/contracts changed; 4. update canonical documentation when behavior/contracts changed;
5. validate every modified public C17 header; 5. validate every modified public C17 header;
6. validate relevant C++ syntax/build integration; 6. validate relevant C++ syntax/build integration;
7. validate resource ownership and bounds; 7. validate resource ownership, accounting, useful scaling and bounds;
8. run applicable build/tests/sanitizers; 8. run applicable build/tests/sanitizers efficiently;
9. perform the required review; 9. perform the required bounded review;
10. only then claim completion. 10. only then claim completion.
The normalized future implementation standard is: The normalized future implementation standard is:
@ -566,12 +714,14 @@ CODE
+ CANONICAL DOCUMENTATION + CANONICAL DOCUMENTATION
+ C17/C++ SYNTAX + C17/C++ SYNTAX
+ RESOURCE OWNERSHIP + RESOURCE OWNERSHIP
+ MAXIMUM SAFE USEFUL THROUGHPUT
+ VALIDATION + VALIDATION
+ REVIEW + REVIEW
``` ```
A tranche is NOT complete merely because code compiles and tests pass when A tranche is NOT complete merely because code compiles and tests pass when
required contract/invariant comments or canonical documentation are missing. required contract/invariant comments, canonical documentation, or required
resource behavior are missing.
There should be no future project-wide comment-cleanup pass for newly written There should be no future project-wide comment-cleanup pass for newly written
code: comment debt must be handled when the code is introduced. code: comment debt must be handled when the code is introduced.
@ -613,9 +763,12 @@ Persistence changes require explicit attention to:
For Project DB: For Project DB:
- preserve the FROZEN v22 semantics and the current additive v23 optical - preserve the FROZEN v22 scientific/persistence semantics and v23 optical
overlay unless a ticket explicitly authorizes a later schema change; overlay;
- schema-version changes require explicit human authorization; - v24 is the explicitly authorized additive operational RAW-batch migration and
must not reinterpret scientific history;
- future schema-version changes beyond the currently authorized v24 require
explicit human authorization;
- migrations must be additive unless a different migration is explicitly - migrations must be additive unless a different migration is explicitly
authorized; authorized;
- existing projects must remain recoverable; - existing projects must remain recoverable;
@ -635,41 +788,86 @@ make retries convenient.
- For changes to API, architecture, ownership, concurrency, persistence, - For changes to API, architecture, ownership, concurrency, persistence,
pipeline, resources, limits, or scientific semantics, check whether pipeline, resources, limits, or scientific semantics, check whether
canonical documentation must be updated. canonical documentation must be updated.
- `MAXIMUM SAFE USEFUL THROUGHPUT` and `SERIALISM_REQUIRES_PROOF` are canonical
human resource-policy decisions. Any resource document that contradicts them
is stale unless it describes an explicitly historical measurement.
- Documentation must describe proven implementation, not desired future - Documentation must describe proven implementation, not desired future
behavior. behavior. Human policy may be marked as policy while implementation work is
still `VALIDATION PENDING`.
- Roadmap documents may describe future behavior, but future capabilities must - Roadmap documents may describe future behavior, but future capabilities must
be clearly marked as planned/later/exploratory. be clearly marked as planned/later/exploratory.
- Never describe future viewer, Capture Guidance, video/keyframe, Task scratch - Never describe future viewer, Capture Guidance, video/keyframe, Task scratch
consumption, or camera-control capabilities as implemented before they are consumption, or camera-control capabilities as implemented before they are
actually validated. The current TUI/F10 and controller-to-Governor registry actually validated.
are validated operationally, but no current Task kind consumes scratch and
those interfaces do not make dense/scratch-consuming workflows complete.
- Statuses such as `PLANNED`, `IMPLEMENTED`, `VALIDATION PENDING`, and - Statuses such as `PLANNED`, `IMPLEMENTED`, `VALIDATION PENDING`, and
`PASS/FROZEN` are authoritative lifecycle statements. `PASS/FROZEN` are authoritative lifecycle statements.
- Update lifecycle state only when implementation, validation, and review - Update lifecycle state only when implementation, validation, and review
evidence support the transition. evidence support the transition.
- Never mark work `PASS/FROZEN` without the required validation and review. - Never mark work `PASS/FROZEN` without the required validation and review.
- Preserve legitimate historical DB/version references where they describe - Preserve legitimate historical DB/version/resource references where they
frozen history. describe frozen history; label superseded operational envelopes as historical
rather than silently rewriting the evidence.
## 12. Required validation ## 12. Required validation and test resource policy
For ordinary implementation tickets, before claiming completion, actually run For ordinary implementation tickets, before claiming completion, actually run
the applicable commands. the applicable commands. Builds, tests, benchmarks and real-proof workloads
MUST follow the same resource philosophy as production: preserve the
interactive host reserve, then use the maximum safe/useful remaining resources.
Artificially serial engineering work wastes both elapsed time and agent budget.
Normal build: ### Normal build
Do not use a fixed historical `-j8` as a product rule. Derive a safe parallel
job count from the current host/affinity. On the current 16-logical-CPU
reference host, where four logical CPUs are reserved for interactive use, the
normal build target is approximately:
```sh ```sh
meson setup --reconfigure build meson setup --reconfigure build
meson compile -C build -j8 meson compile -C build -j12
``` ```
Normal tests: On another host, choose the analogous available-compute value rather than
hardcoding 12.
### Normal tests
Independent tests should run concurrently within the same host reserve and
memory constraints. On the current reference host, a typical command is:
```sh ```sh
meson test -C build --num-processes 1 --print-errorlogs meson test -C build --num-processes 12 --print-errorlogs
``` ```
This is not permission to race tests that share mutable fixtures, real projects,
fixed ports, exclusive GPU state, or other process-global resources. Such tests
must be grouped or serialized for the concrete dependency, and that reason must
be understood rather than inherited from an old `--num-processes 1` default.
A CPU1 run is valid when CPU1 is the actual test case or baseline measurement;
it is not the authoritative production configuration after higher safe/useful
concurrency has been proven.
### Validation efficiency
- Do not rebuild unchanged targets between proof iterations without a concrete
need.
- Do not rerun already-acquired expensive validation when the relevant code and
dependency boundary did not change.
- Run focused tests first; widen only when the changed dependency boundary
requires it.
- Expensive/stress validation must be relevant to the current delta.
- Heavy validation is parallel by default when independent and resource-safe;
serialize only when scientific determinism, mutable shared fixtures, memory,
I/O, GPU exclusivity, sanitizer behavior or measured host pressure requires
it.
- A test timeout or sanitizer failure must be investigated; do not repeatedly
rerun until green.
- Real-data proofs must use normal Governor admission. Do not manually force
CPU1 merely for reproducibility unless CPU1 itself is the intended comparison
cohort.
Diff validation: Diff validation:
```sh ```sh
@ -692,24 +890,48 @@ For relevant concurrency changes, run TSan when supported and meaningful.
If a sanitizer is unavailable or invalid because of the environment/toolchain, If a sanitizer is unavailable or invalid because of the environment/toolchain,
report that explicitly rather than claiming PASS. report that explicitly rather than claiming PASS.
Run expensive/stress validation only when relevant.
Run heavy validation serially when required to preserve machine stability.
Investigate a timeout or sanitizer failure. Do not repeatedly rerun a failing
test until it happens to pass.
Distinguish third-party sanitizer/environment noise from repository defects Distinguish third-party sanitizer/environment noise from repository defects
using concrete stack/failure evidence. using concrete stack/failure evidence.
Never claim a command, test, sanitizer, review, or real-data validation that Never claim a command, test, sanitizer, review, benchmark, or real-data
was not actually performed. validation that was not actually performed.
For documentation/comment-only changes, do not invent unnecessary sanitizer For documentation/comment-only changes, do not invent unnecessary sanitizer
work, but still run enough build/syntax/diff validation to prove that the work, but still run enough build/syntax/diff validation to prove that the
non-functional boundary was preserved. non-functional boundary was preserved.
## 13. Review discipline ## 13. Engineering execution and credit economy
Engineering time and agent budget are finite project resources and must not be
wasted.
- Default to the configured economical parent/orchestrator and existing agent
roles. Do not invent a new agent hierarchy.
- Use expensive/high-reasoning scientific or final-review roles only when the
current delta genuinely requires scientific equivalence, difficult
concurrency/persistence reasoning, or final sensitive review.
- Do not spawn multiple agents to restate the same architecture or repeat the
same review.
- Do not delegate ordinary implementation defects upward when a worker,
mechanic or resolver can fix them directly within authority.
- Prefer one focused independent review after the implementation is stable to
repeated speculative reviews during ordinary coding.
- Reuse acquired maintenance, sanitizer, benchmark and real-data evidence when
the affected boundary is unchanged.
- Do not wait on an expensive agent when the next authorized executable action
can proceed independently.
- Keep tool/log output bounded; inspect targeted portions instead of repeatedly
dumping whole logs.
- Do not rerun a global A-to-Z audit after the maintenance checkpoint.
- `WORK FIRST. RETURN LAST.` Ordinary in-scope findings are repaired and
validated before returning to the human.
Low remaining agent/credit budget is a reason to eliminate redundant work, not
a reason to weaken required correctness evidence. If a genuinely required
validation cannot be afforded/executed, report it honestly rather than claiming
PASS.
## 14. Review discipline
A normal review should verify the bounded current delta and direct regressions. A normal review should verify the bounded current delta and direct regressions.
@ -727,7 +949,7 @@ A reviewer request does not automatically define new policy.
Compare findings against: Compare findings against:
1. canonical FROZEN documentation; 1. canonical FROZEN documentation;
2. explicit human decisions; 2. explicit human decisions, including the resource-utilization policy;
3. established tranche semantics; 3. established tranche semantics;
4. documented future-scope boundaries. 4. documented future-scope boundaries.
@ -735,9 +957,13 @@ Do not invoke an expensive implementation agent merely to satisfy speculative
hardening that is outside the current contract. hardening that is outside the current contract.
Comments and documentation are reviewable implementation artifacts. A Comments and documentation are reviewable implementation artifacts. A
misleading contract comment is a defect even if compiled behavior is unchanged. misleading contract or resource-policy comment is a defect even if compiled
behavior is unchanged.
## 14. Delivery report / STOP conditions Resource review must specifically flag accidental serialism when independent
work exists and safe/useful resources are idle without measured justification.
## 15. Delivery report / STOP conditions
Every completed ticket report must include: Every completed ticket report must include:
@ -750,15 +976,17 @@ Every completed ticket report must include:
- known blockers; - known blockers;
- non-blocking findings; - non-blocking findings;
- deliberately deferred/future-scope items; - deliberately deferred/future-scope items;
- resource impact where relevant; - resource impact where relevant, including admitted/useful concurrency for
modified resource-sensitive Tasks;
- confirmation that unrelated and FROZEN areas were preserved; - confirmation that unrelated and FROZEN areas were preserved;
- Git state; - Git state;
- confirmation that `scan3d/` remained untouched when protected. - confirmation that `scan3d/` remained untouched when protected.
STOP and request a human decision only when resolution requires: STOP and request a human decision only when resolution requires:
- changing a FROZEN contract; - changing a FROZEN scientific contract;
- changing Project DB schema/version without prior authorization; - changing Project DB schema/version beyond already authorized v24 without
prior authorization;
- introducing a genuinely new subsystem outside authorized scope; - introducing a genuinely new subsystem outside authorized scope;
- files outside the authorized scope; - files outside the authorized scope;
- destructive Git action; - destructive Git action;
@ -775,9 +1003,12 @@ Do NOT stop merely because:
- a normal test fails; - a normal test fails;
- an ordinary implementation bug exists; - an ordinary implementation bug exists;
- a reviewer identifies a bounded repairable defect; - a reviewer identifies a bounded repairable defect;
- a resource descriptor is historically conservative and explicit human
authority already allows operational correction while preserving science;
- a dependency needs factual investigation; - a dependency needs factual investigation;
- documentation status is stale; - documentation status is stale;
- an implementation detail can be resolved safely from existing code and - an implementation detail can be resolved safely from existing code and
contracts. contracts.
Advance the project while preserving the contracts. Advance the project while preserving scientific contracts and maximizing safe,
useful throughput.

View file

@ -276,7 +276,7 @@ calibration connues ; le Sparse SfM réel reste
Le Resource Governor ne constitue pas un Resource System générique : il reste Le Resource Governor ne constitue pas un Resource System générique : il reste
l'unique propriétaire des budgets et le seul orchestrateur des leases scratch l'unique propriétaire des budgets et le seul orchestrateur des leases scratch
de production. Le contrôleur SSD UDisks2 est une frontière physique séparée, de production. Le contrôleur SSD UDisks2 est une frontière physique séparée,
jamais un second scheduler ou Governor. Les quatorze Task kinds actuels ne jamais un second scheduler ou Governor. Les seize Task kinds actuels ne
consomment encore aucun scratch : l'espace disponible est une capacité consomment encore aucun scratch : l'espace disponible est une capacité
observable, pas un usage fabriqué, et scratch/swap ne deviennent jamais de la observable, pas un usage fabriqué, et scratch/swap ne deviennent jamais de la
RAM. La TUI/F10 et cette intégration sont validées opérationnellement. L'audit RAM. La TUI/F10 et cette intégration sont validées opérationnellement. L'audit

View file

@ -253,6 +253,9 @@ La tâche utilise le runtime générique via le pattern standard :
La Queue conserve un callback ; jusqu'à soixante-quatre participants CPU La Queue conserve un callback ; jusqu'à soixante-quatre participants CPU
admis peuvent calculer une fenêtre interne bornée sans modifier l'identité admis peuvent calculer une fenêtre interne bornée sans modifier l'identité
scientifique. scientifique.
- CPU et lot sont essayés ensemble (`1/1`, `2/2`, puis les paliers sûrs) :
un CPU supplémentaire avec un lot d'un seul membership ne peut exercer aucun
participant supplémentaire et ne constitue donc pas une mesure de scaling.
- Réservation CPU + IO avant exécution - Réservation CPU + IO avant exécution
- `lardon3d_task_sequence_break()` entre chaque lot pour réadmission Governor - `lardon3d_task_sequence_break()` entre chaque lot pour réadmission Governor
- Callback terminal checkpoint après `COMPLETED`/`FAILED`/`CANCELLED` - Callback terminal checkpoint après `COMPLETED`/`FAILED`/`CANCELLED`
@ -276,7 +279,7 @@ bool lardon3d_candidate_pair_generate_reconstruct(
### Garantie actuelle ### Garantie actuelle
La Queue conserve un callback actif. À l'intérieur de la Task, jusqu'à douze La Queue conserve un callback actif. À l'intérieur de la Task, jusqu'à soixante-quatre
threads CPU admis calculent en parallèle une fenêtre d'au plus deux sources par threads CPU admis calculent en parallèle une fenêtre d'au plus deux sources par
thread. Chaque participant possède un handle DB de lecture privé. Le thread thread. Chaque participant possède un handle DB de lecture privé. Le thread
propriétaire publie ensuite seul et dans l'ordre canonique des sources. Voir le propriétaire publie ensuite seul et dans l'ordre canonique des sources. Voir le

View file

@ -198,6 +198,50 @@ au plus `cpu_threads - 1` threads enfants. Tous les enfants sont joints avant
la fin de la séquence, la libération de réservation ou la fin de la séquence, la libération de réservation ou
`lardon3d_task_sequence_break()`. `lardon3d_task_sequence_break()`.
## Développement RAW d'une exécution sélectionnée
`raw.develop.batch/1` conserve la politique RAW v1, les octets PNG,
L3DRAWD1, le fingerprint, le content-addressing et toutes les identités du
développeur par Capture. Une séquence prend au plus huit items contigus depuis
le curseur durable de l'exécution sélectionnée et utilise au plus le nombre de
CPUs admis. Chaque participant possède son LibRaw et ses buffers privés ; les
896 Mio chargés par item admis couvrent le workspace LibRaw borné à 40 MP, les
copies RGB/BGR, les buffers PNG/validation, la pile enfant bornée de 1 Mio et la
marge allocateur/codec. Le petit contexte propriétaire est inclus dans cette
charge : un budget hôte post-réserve de 7 Gio admet donc exactement les huit
participants sûrs, sans débit fixe redondant. Cette estimation est une borne
opérationnelle de mémoire par participant, pas une limite scientifique de
dataset.
OpenCV est fixé à un thread interne afin de ne pas multiplier le fan-out.
Les enfants ne modifient ni curseur sélectionné ni progression Task. Après leur
jointure complète, le callback Queue propriétaire publie seul les mappings
`item_index -> image_id` dans l'ordre croissant. Chaque transaction de mapping
avance le curseur contigu, puis seulement le checkpoint Task peut avancer. Une
erreur, une annulation ou un échec de création d'enfant joint tout le travail
lancé ; des assets immuables content-addressés peuvent déjà exister, mais le
curseur ne les devine pas et la reprise converge par le développeur existant.
Une `sequence_break` sépare toutes les fenêtres, donc la sélection entière ne
retient jamais une réservation longue.
## Extraction Feature d'une exécution sélectionnée
`features.extract.batch/1` conserve exactement ORB v1, ses paramètres, son
fingerprint, ses keypoints/descripteurs U8×32 et le Feature File existant. Une
fenêtre contient au plus douze images indépendantes de l'ordre sélectionné. Le
propriétaire configure OpenCV à un thread pour éviter tout fan-out imbriqué ;
chaque participant possède son décodage et sa sortie bornés, sans accès SQLite,
puis tous sont joints.
Le propriétaire seul publie dans l'ordre et avance le curseur v25 après preuve
du Feature Set READY exact. CPU et lot partent du minimum. Comme un CPU
supplémentaire ne peut être exercé que par une image indépendante de la même
fenêtre, le Governor essaie et accepte ces deux limites ensemble (`1/1`, `2/2`,
`4/4`, puis le maximum sûr admis). Les observations de fenêtres durables
permettent de retenir le dernier palier apportant au moins 5 %. Le plafond douze
est une capacité opérationnelle de benchmark,
jamais une limite scientifique, et aucune voie GPU n'est introduite.
## Calcul et publication Visual Index ## Calcul et publication Visual Index
Une séquence sélectionne le même préfixe durable d'au plus seize Feature Sets Une séquence sélectionne le même préfixe durable d'au plus seize Feature Sets

View file

@ -138,7 +138,7 @@ Frontière physique UDisks2 pour la paire de labels `LARDON_SWAP` et
et drain sûr. Il ne formate, ne répare ni ne force jamais l'hôte et ne remplace et drain sûr. Il ne formate, ne répare ni ne force jamais l'hôte et ne remplace
pas le Resource Governor. La TUI exécute ses actions dans un seul thread pas le Resource Governor. La TUI exécute ses actions dans un seul thread
joinable et le Governor enregistre l'état physique puis orchestre seul les joinable et le Governor enregistre l'état physique puis orchestre seul les
leases scratch de production. Les quatorze Task kinds courants n'en consomment leases scratch de production. Les seize Task kinds courants n'en consomment
aucun ; capacité visible ne signifie donc pas usage. aucun ; capacité visible ne signifie donc pas usage.
**Statut :** CURRENT / VALIDATED OPERATIONAL **Statut :** CURRENT / VALIDATED OPERATIONAL

View file

@ -38,10 +38,46 @@ malformée retourne `CORRUPT` avant comparaison avec une demande alternative.
La migration v22→v23 n'inspecte ni EXIF, chemin, basename, SHA-256, dimensions, La migration v22→v23 n'inspecte ni EXIF, chemin, basename, SHA-256, dimensions,
nom d'appareil, ni calibration historique. Elle ne réinterprète donc aucune nom d'appareil, ni calibration historique. Elle ne réinterprète donc aucune
Capture, image ou calibration v16v22. Le marqueur `schema_version=23` est le Capture, image ou calibration v16v22. Le marqueur `schema_version=23` reste le
point de publication durable ; un échec de création ou un update de marqueur point de publication durable ; un échec de création ou un update de marqueur
qui ne cible pas exactement la v22 rollbacke toute la migration. qui ne cible pas exactement la v22 rollbacke toute la migration.
### Overlay v24 — développement RAW sélectionné borné
Le schéma courant v24 ajoute uniquement
`raw_development_batch_tasks(task_id, selected_execution_id)`. `task_id`
référence l'état générique Task et `selected_execution_id` est unique ; la
ligne ne copie aucun Capture, asset, image, chemin, groupe ni curseur. La
migration v23→v24 est transactionnelle et DDL-only : elle ne crée aucune
association historique et son marqueur v24 est son point de publication.
`raw.develop.batch/1` recharge le curseur de `selected_executions` à chaque
fenêtre admise. Après jointure de tous les participants, seul le callback Task
propriétaire appelle `record_selected_representation` en ordre croissant. Le
commit item+curseur précède toujours la progression/checkpoint générique ; un
crash peut donc laisser celle-ci en retard, jamais en avance. Une reprise repart
du curseur sélectionné durable sans deviner l'identité d'une sortie RAW déjà
publiée. Le schéma et le chemin historique `raw.develop/1` restent valides.
### Overlay v25 — extraction Feature sélectionnée bornée
**IMPLEMENTED / VALIDATION IN PROGRESS.** La migration transactionnelle
v24→v25 ajoute uniquement `feature_extract_batch_tasks`. Sa ligne lie un Task
`features.extract.batch/1` à une exécution sélectionnée immutable, retient le
préfixe monotone `next_item_index` et copie le domaine ORB exact
kind/version/paramètres/fingerprint. La migration est DDL-only : elle ne crée
aucune ligne, ne convertit aucune Task `features.extract/1` historique et
n'infère aucun image ID ou Feature Set.
Les participants CPU préparent des images sélectionnées indépendantes sans
accès SQLite. Après leur jointure, le propriétaire publie les Feature Sets dans
l'ordre sélectionné, avance un item seulement si le Feature Set READY exact est
durable, puis checkpoint la progression générique. Un crash peut donc laisser
le checkpoint générique ou le curseur Feature en retard sur un Feature Set
immutable déjà publié ; la reprise revalide ce résultat exact et converge sans
deviner une identité. Les formats, keypoints, descripteurs ORB U8×32,
fingerprint et Feature Store FROZEN restent inchangés.
La validation a migré via l'API production des copies des projets réels S21 et La validation a migré via l'API production des copies des projets réels S21 et
A6000 : version 23, `integrity_check` et clés étrangères propres, comptes et A6000 : version 23, `integrity_check` et clés étrangères propres, comptes et
lignes scientifiques inchangés, neuf tables optiques vides et SHA-256 des lignes scientifiques inchangés, neuf tables optiques vides et SHA-256 des

View file

@ -5,12 +5,13 @@
**ACCEPTED** — architecture decision for the post-Gate C documentation freeze. **ACCEPTED** — architecture decision for the post-Gate C documentation freeze.
Current Sparse SfM gates A through G are **PASS / FROZEN**. Current Sparse SfM gates A through G are **PASS / FROZEN**.
Project Database: current schema **v23**; the v22 foundation and historical Project Database: current schema **v24**; the v22 foundation, v23 optical overlay, and historical
v16 science remain frozen. v16 science remain frozen.
This record is normative for the separation of responsibilities. Its original This record is normative for the separation of responsibilities. Its original
Gate C/G decision introduced no implementation or syntax. Later explicitly Gate C/G decision introduced no implementation or syntax. Later explicitly
authorized additions, including the additive Project DB v23 optics overlay and authorized additions, including the additive Project DB v23 optics overlay,
the v24 RAW batch Task association, and
the bounded UDisks2 SSD controller, must remain on their own side of this the bounded UDisks2 SSD controller, must remain on their own side of this
boundary and do not create a generic Resource System. boundary and do not create a generic Resource System.

View file

@ -268,7 +268,7 @@ et Visual Index restent CPU; SIFT/RootSIFT Matcher restent BFMatcher L2 CPU. Il
n'est introduit ni second scheduler, ni Queue, ni daemon, ni sous-système de n'est introduit ni second scheduler, ni Queue, ni daemon, ni sous-système de
ressources. ressources.
### Audit des 14 kinds de production ### Audit des 16 kinds de production
Tous les kinds passent par l'unique Queue et l'unique Governor, y compris ceux Tous les kinds passent par l'unique Queue et l'unique Governor, y compris ceux
dont toutes les dimensions sont fixes. Dans le tableau, `CPU 1..N` décrit la dont toutes les dimensions sont fixes. Dans le tableau, `CPU 1..N` décrit la
@ -283,14 +283,16 @@ allocations driver opaques ne reçoivent pas un coût inventé.
| Kind v1 | Estimation courante | Dimension consommée et raison | | Kind v1 | Estimation courante | Dimension consommée et raison |
| --- | --- | --- | | --- | --- | --- |
| `raw.develop` | MIXED; CPU 1; lot 1; hôte `2 Gio + contexte`, 0/item; I/O 1; GPU 0 | Un Capture atomique. Le garde applique/restaure CPU1 au pool OpenCV global. Les 2 Gio sont une allowance de travail opérationnelle, pas une limite de dataset. | | `raw.develop` | MIXED; CPU 1; lot 1; hôte `2 Gio + contexte`, 0/item; I/O 1; GPU 0 | Un Capture atomique. Le garde applique/restaure CPU1 au pool OpenCV global. Les 2 Gio sont une allowance de travail opérationnelle, pas une limite de dataset. |
| `raw.develop.batch` | MIXED; CPU 1..8; lot 1..8; hôte 0 fixe + 896 Mio/item (contexte propriétaire, workspace LibRaw 40 MP, copies RGB/BGR, PNG/validation, pile enfant 1 Mio et marge allocateur/codec); I/O 1; GPU 0 | Une fenêtre de Captures indépendants. `896 Mio × 8 = 7 Gio`, donc le budget post-réserve de l'hôte de référence peut admettre la fenêtre sûre complète. OpenCV reste à 1 thread interne, tous les enfants sont joints, puis le propriétaire publie le préfixe ordonné avant `sequence_break`. Le plafond est opérationnel, jamais une limite scientifique de sélection. |
| `photo_quality.triage` | IMPORT; CPU 1; lot 1; hôte `contexte retenu + 20 Mio`, 0/item; I/O 1; GPU 0 | Un groupe par séquence ; garde OpenCV CPU1. Aucun scaling utile déterministe n'est acquis. | | `photo_quality.triage` | IMPORT; CPU 1; lot 1; hôte `contexte retenu + 20 Mio`, 0/item; I/O 1; GPU 0 | Un groupe par séquence ; garde OpenCV CPU1. Aucun scaling utile déterministe n'est acquis. |
| `acquisition_campaign.run` | JPEG: IMPORT ; RAW: MIXED. CPU 1; lot 1; hôte `contexte retenu + requête transitoire exacte + 256 Kio` + 64 Kio/item ; DEVELOP_RAW ajoute 2 Gio; I/O 1; GPU 0 | Un groupe S3-E par séquence, sans Task imbriqué. La création et la reprise dérivent la même estimation ; la forme historique exacte est normalisée seulement en mémoire. | | `acquisition_campaign.run` | JPEG: IMPORT ; RAW: MIXED. CPU 1; lot 1; hôte `contexte retenu + requête transitoire exacte + 256 Kio` + 64 Kio/item ; DEVELOP_RAW ajoute 2 Gio; I/O 1; GPU 0 | Un groupe S3-E par séquence, sans Task imbriqué. La création et la reprise dérivent la même estimation ; la forme historique exacte est normalisée seulement en mémoire. |
| `import.images` | IMPORT; CPU 1; lot 1..32; hôte 128 Kio + `NAME_MAX+64`/item; I/O 1; GPU 0 | Copie/hash I/O-bound. Le callback consomme le lot admis et réadmet entre lots. | | `import.images` | IMPORT; CPU 1; lot 1..32; hôte 128 Kio + `NAME_MAX+64`/item; I/O 1; GPU 0 | Copie/hash I/O-bound. Le callback consomme le lot admis et réadmet entre lots. |
| `features.extract` | CPU; CPU 1..compute-pool; lot 1; hôte 64 Mio + 512 Mio/item; I/O 1; GPU 0 | La demande durable emploie le maximum `int` positif de l'API OpenCV ; le Governor la borne à l'hôte. Le garde applique/restaure exactement le CPU admis. | | `features.extract` | CPU; CPU 1..compute-pool; lot 1; hôte 64 Mio + 512 Mio/item; I/O 1; GPU 0 | La demande durable emploie le maximum `int` positif de l'API OpenCV ; le Governor la borne à l'hôte. Le garde applique/restaure exactement le CPU admis. |
| `features.extract.batch` | CPU; CPU 1..12; lot 1..12; hôte 64 Mio + 512 Mio/item; I/O 1; GPU 0 | Images sélectionnées indépendantes, OpenCV CPU1 par participant, enfants joints et publication owner-only ordonnée. Le plafond 12 est une capacité opérationnelle de mesure ; le feedback ≥5 % établit le palier utile sans devenir une limite scientifique. |
| `features.extract.sift` | CPU; CPU 1..compute-pool; lot 1; hôte 64 Mio + 1 Gio/item; I/O 1; GPU 0 | Même contrat OpenCV. Les formes CPU12 et CPU1 historiques complètes sont acceptées et normalisées en mémoire. | | `features.extract.sift` | CPU; CPU 1..compute-pool; lot 1; hôte 64 Mio + 1 Gio/item; I/O 1; GPU 0 | Même contrat OpenCV. Les formes CPU12 et CPU1 historiques complètes sont acceptées et normalisées en mémoire. |
| `features.extract.rootsift` | CPU; CPU 1..compute-pool; lot 1; hôte 64 Mio + 1 Gio/item; I/O 1; GPU 0 | Même contrat que SIFT ; aucune couture GPU scientifiquement compatible n'est validée. | | `features.extract.rootsift` | CPU; CPU 1..compute-pool; lot 1; hôte 64 Mio + 1 Gio/item; I/O 1; GPU 0 | Même contrat que SIFT ; aucune couture GPU scientifiquement compatible n'est validée. |
| `visual_index.update` | CPU; CPU 1..16; lot 1..16; hôte 8 Mio + 2 Mio/item; I/O 1; GPU 0 | Le segment contient au plus 16 Feature Sets indépendants. Au plus `cpu_threads-1` enfants sont joints avant publication owner-only. | | `visual_index.update` | CPU; CPU 1..16; lot 1..16; hôte 8 Mio + 2 Mio/item; I/O 1; GPU 0 | Le segment contient au plus 16 Feature Sets indépendants. Au plus `cpu_threads-1` enfants sont joints avant publication owner-only. |
| `candidate_pair.generate` | CPU; CPU 1..64; lot 1..64; hôte 256 Kio + 8 Mio/item; I/O 1; GPU 0 | Le batch de 64 est la borne algorithmique et de ressources. Fenêtre `min(2*CPU, 64, reste_du_lot)`, un handle DB privé par participant, piles enfants de 4 Mio facturées. | | `candidate_pair.generate` | CPU; CPU 1..64; lot 1..64; hôte 256 Kio + 8 Mio/item; I/O 1; GPU 0 | Le batch de 64 est la borne algorithmique et de ressources. CPU/lot sont couplés pour que chaque palier de feedback exerce ses participants; fenêtre `min(2*CPU, 64, reste_du_lot)`, un handle DB privé par participant, piles enfants de 4 Mio facturées. |
| `matcher.run` | CPU: CPU 1..12, lot sûr 1..12, hôte 0 + 10 Mio/item, I/O 1, GPU 0. ORB Vulkan AUTO: CPU 1, lot utile 1..8, même hôte/item, GPU 1 + 640 Kio, inflight 1. | Le batch/participant 12 est une borne intrinsèque mesurée du Matcher. Batch 12 et depth 2 (1,25 Mio) restent sûrs pour preuves privées, mais insuffisamment utiles en AUTO. CPU complet est le fallback. | | `matcher.run` | CPU: CPU 1..12, lot sûr 1..12, hôte 0 + 10 Mio/item, I/O 1, GPU 0. ORB Vulkan AUTO: CPU 1, lot utile 1..8, même hôte/item, GPU 1 + 640 Kio, inflight 1. | Le batch/participant 12 est une borne intrinsèque mesurée du Matcher. Batch 12 et depth 2 (1,25 Mio) restent sûrs pour preuves privées, mais insuffisamment utiles en AUTO. CPU complet est le fallback. |
| `geometric_verifier.run` | CPU; CPU utile 1..8, fenêtre sûre 16; lot 1..16; hôte 0 + 8 Mio/item; I/O 1; GPU 0 | Des parents indépendants sont préparés en parallèle, puis publiés/checkpointés en ordre par le propriétaire. L'USAC scientifique conserve `isParallel=false`. | | `geometric_verifier.run` | CPU; CPU utile 1..8, fenêtre sûre 16; lot 1..16; hôte 0 + 8 Mio/item; I/O 1; GPU 0 | Des parents indépendants sont préparés en parallèle, puis publiés/checkpointés en ordre par le propriétaire. L'USAC scientifique conserve `isParallel=false`. |
| `track_builder.run` | CPU; CPU 1; lot 1; fixe `(4 Mio + arêtes * (48 + 2*160)) * facteur`, facteur 2 jusqu'à 400k arêtes puis 8; 0/item; I/O 1; GPU 0 | Rebuild DSU atomique et publication owner-only ; aucune partition scientifiquement validée n'est acquise. | | `track_builder.run` | CPU; CPU 1; lot 1; fixe `(4 Mio + arêtes * (48 + 2*160)) * facteur`, facteur 2 jusqu'à 400k arêtes puis 8; 0/item; I/O 1; GPU 0 | Rebuild DSU atomique et publication owner-only ; aucune partition scientifiquement validée n'est acquise. |
@ -422,7 +424,7 @@ retirer l'autorité mais ne peut jamais recopier un ancien compte physique.
Le teardown production suit strictement : Queue détruite/jointe et chaque Le teardown production suit strictement : Queue détruite/jointe et chaque
lease Task rendu, worker SSD joint puis unregister vérifié, contrôleur détruit, lease Task rendu, worker SSD joint puis unregister vérifié, contrôleur détruit,
Governor détruit. Un unregister est refusé tant qu'une opération wrapper ou un Governor détruit. Un unregister est refusé tant qu'une opération wrapper ou un
lease exact subsiste. Les quatorze Task kinds courants n'ont aucun consommateur lease exact subsiste. Les seize Task kinds courants n'ont aucun consommateur
scratch : le compte normal est donc réellement zéro et la capacité disponible scratch : le compte normal est donc réellement zéro et la capacité disponible
n'autorise aucun usage implicite. Une future Task consommatrice devra définir n'autorise aucun usage implicite. Une future Task consommatrice devra définir
son propre contrat d'éligibilité et son ownership sans transformer scratch ou son propre contrat d'éligibilité et son ownership sans transformer scratch ou

View file

@ -44,18 +44,20 @@ métier, et toute forme voisine est rejetée.
## Inventaire production et entrées runtime ## Inventaire production et entrées runtime
`src/task_kinds.c::lardon3d_task_kind_registry_production()` enregistre les `src/task_kinds.c::lardon3d_task_kind_registry_production()` enregistre les
14 kinds v1 du profil de production courant. La colonne « reprise » nomme le 16 kinds v1 du profil de production courant. La colonne « reprise » nomme le
reconstructeur du binding ; « callback » nomme l'entrée runtime privée dans le reconstructeur du binding ; « callback » nomme l'entrée runtime privée dans le
même fichier. Le détail chiffré des capacités est centralisé dans l'[audit des même fichier. Le détail chiffré des capacités est centralisé dans l'[audit des
14 kinds](resource_governor.md#audit-des-14-kinds-de-production). 16 kinds](resource_governor.md#audit-des-15-kinds-de-production).
| Kind v1 | Source, reprise et callback | Réconciliation pré-admission courante | | Kind v1 | Source, reprise et callback | Réconciliation pré-admission courante |
| --- | --- | --- | | --- | --- | --- |
| `raw.develop` | `raw_development_task.cpp`; `lardon3d_raw_development_task_reconstruct`; `run` | Aucune | | `raw.develop` | `raw_development_task.cpp`; `lardon3d_raw_development_task_reconstruct`; `run` | Aucune |
| `raw.develop.batch` | `raw_development_task.cpp`; `lardon3d_raw_development_batch_task_reconstruct`; `run_batch` | Association v24 Task → exécution sélectionnée ; le curseur reste dans l'exécution |
| `photo_quality.triage` | `photo_quality_task.cpp`; `lardon3d_photo_quality_task_reconstruct`; `run` | Aucune | | `photo_quality.triage` | `photo_quality_task.cpp`; `lardon3d_photo_quality_task_reconstruct`; `run` | Aucune |
| `acquisition_campaign.run` | `acquisition_campaign_task.cpp`; `lardon3d_acquisition_campaign_task_reconstruct`; `run` | Forme courante ou forme v22 exacte vérifiée contre la requête immuable → capacité courante en mémoire | | `acquisition_campaign.run` | `acquisition_campaign_task.cpp`; `lardon3d_acquisition_campaign_task_reconstruct`; `run` | Forme courante ou forme v22 exacte vérifiée contre la requête immuable → capacité courante en mémoire |
| `import.images` | `import_task.c`; `lardon3d_image_import_reconstruct`; `run_image_import` | Aucune | | `import.images` | `import_task.c`; `lardon3d_image_import_reconstruct`; `run_image_import` | Aucune |
| `features.extract` | `feature_task.c`; `lardon3d_feature_extract_reconstruct`; `run` | Formes CPU12/CPU1 historiques exactes → demande OpenCV portable ; runtime borné au compute-pool | | `features.extract` | `feature_task.c`; `lardon3d_feature_extract_reconstruct`; `run` | Formes CPU12/CPU1 historiques exactes → demande OpenCV portable ; runtime borné au compute-pool |
| `features.extract.batch` | `feature_task.c`; `lardon3d_feature_extract_batch_reconstruct`; `run_batch` | Association v25 à l'exécution sélectionnée, domaine ORB exact et curseur typé ; CPU/lot bornés 1..12 |
| `features.extract.sift` | `sift_task.c`; `lardon3d_sift_extract_reconstruct`; `run` | CPU12/CPU1 historiques exacts → demande OpenCV portable ; runtime borné au compute-pool | | `features.extract.sift` | `sift_task.c`; `lardon3d_sift_extract_reconstruct`; `run` | CPU12/CPU1 historiques exacts → demande OpenCV portable ; runtime borné au compute-pool |
| `features.extract.rootsift` | `sift_task.c`; `lardon3d_sift_extract_reconstruct`; `run` | Même réconciliation SIFT ; aucune voie GPU validée | | `features.extract.rootsift` | `sift_task.c`; `lardon3d_sift_extract_reconstruct`; `run` | Même réconciliation SIFT ; aucune voie GPU validée |
| `visual_index.update` | `visual_index_task.c`; `lardon3d_visual_index_update_reconstruct`; `run` | Formes CPU12/CPU1 historiques exactes → CPU/lot 1..16 | | `visual_index.update` | `visual_index_task.c`; `lardon3d_visual_index_update_reconstruct`; `run` | Formes CPU12/CPU1 historiques exactes → CPU/lot 1..16 |

View file

@ -12,6 +12,104 @@ doivent rester incrémentaux, bornés, reprenables, durables et admis par l'uniq
Resource Governor : aucune étape ne peut supposer qu'une campagne entière tient Resource Governor : aucune étape ne peut supposer qu'une campagne entière tient
en RAM ou termine dans une seule vie de processus. en RAM ou termine dans une seule vie de processus.
## POLITIQUE CANONIQUE D'UTILISATION DES RESSOURCES — HUMAN AUTHORITY
La politique d'exploitation est désormais explicite : **Lardon3D doit viser le
MAXIMUM SAFE USEFUL THROUGHPUT** sur chaque Task de production.
Le Resource Governor réserve d'abord uniquement les ressources nécessaires pour
que le poste reste normalement utilisable pendant les calculs :
- Arch Linux / Sway et les services normaux du bureau ;
- Firefox ;
- lecture audio/musique normale ;
- usage interactif léger.
Tout CPU, RAM, capacité I/O et accélérateur validé restant appartient au travail
Lardon3D lorsqu'un travail utile existe. La stabilité du poste est protégée par
cette réserve interactive et par les signaux de pression ; elle ne justifie plus
de laisser des ressources sûres et utiles inactives par conservatisme historique.
Sur l'hôte de référence actuel, le résultat normal de cette politique est
approximativement :
```text
16 CPU logiques au total
4 CPU logiques réservés à l'hôte interactif
12 CPU logiques disponibles au compute-pool
~3 GiB de MemAvailable conservés comme réserve RAM dure
Radeon 780M UMA disponible pour les backends GPU validés et utiles
```
Ces nombres sont des **résultats de l'hôte courant**, jamais des constantes
produit. Un futur hôte 32 threads ne doit pas hériter d'un plafond 12 ; le
Governor doit dériver sa réserve et donner le reste au calcul selon topologie,
affinité, pression et enveloppe réelle de la Task.
La règle canonique est :
```text
SERIALISM_REQUIRES_PROOF
```
Une opération atomique par item n'impose pas de sérialiser les items
indépendants. Une publication owner-only ou ordonnée n'impose pas de sérialiser
la préparation. Lorsqu'il existe plusieurs unités indépendantes et que la
science, la persistance et la reprise restent exactes, la Task doit exposer au
Governor un parallélisme borné.
Un chemin long `CPU1` ou `batch1` avec travail indépendant disponible et
ressources sûres libres est désormais considéré comme un **défaut opérationnel**
tant qu'une preuve concrète ne démontre pas l'une des limites suivantes :
- dépendance scientifique ou algorithmique réellement sérielle ;
- genou de scaling mesuré ;
- limite mémoire ;
- saturation I/O ;
- backend GPU validé rendant des CPU supplémentaires inutiles ;
- contrainte de publication qui ne peut pas être séparée de la préparation sans
casser le déterminisme ;
- autre limitation matérielle mesurée et documentée.
Le Governor reste l'unique autorité de ressources. L'utilisateur normal ne
choisit ni CPU, ni workers, ni lot, ni inflight, ni GPU, ni scratch, ni budget
RAM. Une Task décrit ses minimum/useful/safe, ses coûts fixes/transitoires/par
participant et ses capacités GPU/I/O ; le Governor choisit le maximum
sûr **et utile** disponible à l'instant.
La politique GPU est symétrique : un backend de production **VALIDATED AND
USEFUL** doit être préféré lorsqu'il est disponible et sûr ; aucun backend GPU
non validé ne doit être inventé pour afficher de l'activité GPU. Candidate,
Visual Index ou GV peuvent donc rester CPU lorsque leur décision mesurée le
justifie, tandis qu'ORB Matcher Vulkan reste GPU-first lorsqu'il est éligible.
La RAM doit être utilisée agressivement jusqu'à la réserve interactive ;
`swap`, zram et scratch ne sont jamais de la RAM admise, et l'UMA est comptée
exactement une fois contre la RAM hôte. Les signaux PSI et les deltas actifs de
swap peuvent réduire l'admission ; après disparition de la pression, le
Governor doit réadmettre les ressources utiles au lieu de rester durablement
bridé.
Cette politique s'applique aussi au travail d'ingénierie Codex : builds, tests,
benchmarks et preuves réelles indépendantes doivent utiliser le parallélisme
sûr disponible plutôt qu'un `-j8`/`--num-processes 1` historique. Un run CPU1
reste légitime comme cohorte de mesure ou lorsqu'une dépendance concrète
l'impose, mais jamais comme défaut universel. Les rebuilds, tests lourds,
revues ou agents coûteux déjà acquis et non affectés par le delta ne doivent pas
être relancés sans raison. La règle reste : **WORK FIRST, RETURN LAST**, avec
revue delta-based depuis le checkpoint de maintenance.
Statut de cette décision :
```text
RESOURCE_UTILIZATION_POLICY=HUMAN_AUTHORITY
SERIALISM_REQUIRES_PROOF=CANONICAL
IMPLEMENTATION_AUDIT=REQUIRED_BEFORE_NEXT_LONG_REAL_RUN
```
Cette autorité rouvre uniquement les **contrats opérationnels de ressources**
quand ils empêchent cette politique. Elle ne rouvre aucune science FROZEN.
### Fondations PASS / FROZEN ### Fondations PASS / FROZEN
- Gates AG Sparse SfM, dont F0, orchestration Task/Project DB et politique - Gates AG Sparse SfM, dont F0, orchestration Task/Project DB et politique
@ -26,9 +124,11 @@ en RAM ou termine dans une seule vie de processus.
- Task Runtime, checkpoints atomiques, Queue et Resource Governor : - Task Runtime, checkpoints atomiques, Queue et Resource Governor :
[Task](../architecture/task_system.md), [Queue](../architecture/task_queue.md) [Task](../architecture/task_system.md), [Queue](../architecture/task_queue.md)
et [Governor](../architecture/resource_governor.md). et [Governor](../architecture/resource_governor.md).
- Project DB v22 gelé, overlay optique additif v23 courant, et S1S3 Capture / Acquisition Ingestion : provenance - Project DB v22 gelé, overlay optique additif v23 gelé et couche opérationnelle
Capture/Asset, import capture-safe, publication dérivée, développement RAW, additive v24 actuellement en validation pour `raw.develop.batch/1`, ainsi que
siblings multi-source, évidence S3-D, orchestration S3-E et campagne bornée : S1S3 Capture / Acquisition Ingestion : provenance Capture/Asset, import
capture-safe, publication dérivée, développement RAW, siblings multi-source,
évidence S3-D, orchestration S3-E et campagne bornée :
[Project DB et ingestion](../architecture/project_database.md). [Project DB et ingestion](../architecture/project_database.md).
### S3 Capture / Acquisition Ingestion — PASS / FROZEN ### S3 Capture / Acquisition Ingestion — PASS / FROZEN
@ -71,8 +171,7 @@ Campagne `Photogrammetrie/2026-08-26_Baie_Moteur_A6000` :
- plan identique après inversion de l'ordre des racines. - plan identique après inversion de l'ordre des racines.
Le stem commun ne prouve jamais une acquisition. Les 953 propositions doivent Le stem commun ne prouve jamais une acquisition. Les 953 propositions doivent
être confirmées et deviennent `CALLER_EXPLICIT`, jamais `STRONG`. La campagne être confirmées et deviennent `CALLER_EXPLICIT`, jamais `STRONG`.
complète n'a pas encore été matérialisée.
### Limite de reprise DB v19 (historique) ### Limite de reprise DB v19 (historique)
@ -118,7 +217,7 @@ Project DB v20 : A6000 (953 ARW + 953 JPEG, 953 confirmations
groupes singleton déterministes). Un échantillon borné a exercé deux Captures groupes singleton déterministes). Un échantillon borné a exercé deux Captures
A6000 (JPEG SOURCE puis RAW dérivé) et trois Captures S21, avec ScanSets isolés, A6000 (JPEG SOURCE puis RAW dérivé) et trois Captures S21, avec ScanSets isolés,
Queue/Governor, persistance tâche/groupe→Capture et reprise sans duplication. Queue/Governor, persistance tâche/groupe→Capture et reprise sans duplication.
La capacité bornée des propositions de revue conserve un préfixe déterministe; La capacité bornée des propositions de revue conserve un préfixe déterministe ;
elle ne limite jamais l'évaluation complète ni le groupement scientifique. elle ne limite jamais l'évaluation complète ni le groupement scientifique.
### État de calibration des campagnes réelles ### État de calibration des campagnes réelles
@ -232,13 +331,20 @@ responsabilités. L'overlay optique Project DB v23, ajouté ultérieurement, ne
réinterprète pas cette preuve v22 ; les copies des deux projets migrent avec les réinterprète pas cette preuve v22 ; les copies des deux projets migrent avec les
comptes scientifiques inchangés et les nouvelles tables optiques vides. comptes scientifiques inchangés et les nouvelles tables optiques vides.
Pour A6000, le JPEG caméra pairé reste un SOURCE valide et le proxy rapide de
Photo Quality, mais la représentation géométrique FROZEN de cette exécution
sélectionnée reste le PNG déterministe dérivé du RAW (`L3DRAWD1`). Passer cette
preuve au JPEG caméra nécessiterait une tranche scientifique/calibration séparée
prouvant l'équivalence exacte ; ce changement n'est pas une optimisation
opérationnelle ordinaire.
Le Sparse SfM réel final reste `BLOCKED_BY_KNOWN_CALIBRATION_DATA` pour ces Le Sparse SfM réel final reste `BLOCKED_BY_KNOWN_CALIBRATION_DATA` pour ces
campagnes ; aucune pseudo-calibration, interpolation de métadonnées ou inférence campagnes ; aucune pseudo-calibration, interpolation de métadonnées ou inférence
d'identité ne le contourne. Dense, mesh et publication aval ne font pas partie de d'identité ne le contourne. Dense, mesh et publication aval ne font pas partie de
ce jalon pré-SfM. Ce milestone est une intégration réelle, pas une nouvelle ce jalon pré-SfM. Ce milestone est une intégration réelle, pas une nouvelle
série de micro-gates S3. Son statut acquis ne rend pas disponibles les données série de micro-gates S3. Son statut acquis ne rend pas disponibles les données
de calibration physique manquantes et n'autorise pas à devancer la maintenance de calibration physique manquantes et n'autorise pas à devancer les dépendances
globale. scientifiques.
## INTERNAL PARALLELISM + COMPUTE RESOURCES v1 — PASS / FROZEN ## INTERNAL PARALLELISM + COMPUTE RESOURCES v1 — PASS / FROZEN
@ -273,83 +379,76 @@ reproductibilité. Les dimensions retenues et toutes les validations v2 sont
closes. closes.
Le pool CPU12 est validé comme preuve de cet hôte, pas comme plafond produit. Le pool CPU12 est validé comme preuve de cet hôte, pas comme plafond produit.
Le Governor dérive désormais le pool lourd depuis le masque Le Governor dérive désormais le pool lourd depuis le masque permis et les
permis et les groupes package/core/SMT. Sur l'hôte unrestricted courant, il groupes package/core/SMT. Sur l'hôte unrestricted courant, il obtient
obtient `0-5,8-13` et réserve `6,7,14,15`; un caller déjà précontraint ne subit `0-5,8-13` et réserve `6,7,14,15`; un caller déjà précontraint ne subit pas une
pas une seconde réserve. Le worker Queue seul applique/vérifie son propre seconde réserve. Le worker Queue seul applique/vérifie son propre masque ; aucun
masque ; aucun TID auxiliaire énuméré n'est muté, car un pidfd ne stabilise pas TID auxiliaire énuméré n'est muté. Creator/main/TUI reste unrestricted. Les CPU
le numéro consommé par `sched_setaffinity(tid)`. Avant tout pthread applicatif déjà exclus de l'affinité du processus comptent dans la réserve hôte ; le
ou driver, le démarrage établit `MESA_SHADER_CACHE_DISABLE=true`; une valeur fallback count-only ne fabrique aucun masque. La cible RAM conserve une réserve
absente prend ce défaut, `true`/`1` explicites sont conservées et toute autre dure de 3 GiB de `MemAvailable`; la zone 34 GiB est une prudence qui ne
valeur explicite est préservée mais refusée. Cette politique non scientifique soustrait pas 4 GiB à toute capacité. Les petits hôtes dégradent le budget en
supprime les helpers de cache Mesa observés qui élargissaient leur masque. Les conservant au moins une unité de calcul. Les PSI CPU/mémoire/I/O et les deltas
threads runtime restants héritent le compute-pool ; il n'existe plus de sweep, swap-in/swap-out sont des signaux actifs ; l'occupation totale du swap reste
latch ou retry auxiliaire et le diagnostic expose la politique réelle. historique. Admission CPU et lot sont indépendantes. Sur la 780M, Hardware
Creator/main/TUI reste unrestricted. Les CPU déjà exclus de l'affinité du Profile classe conservativement comme UMA le petit aperture VRAM amdgpu de
processus comptent dans la réserve hôte ; le fallback count-only ne fabrique 512 Mio accompagné d'environ 7,99 Go de GTT système ; la capacité rapportée
aucun masque. Le fallback au budget portable ne crée aucune exclusion reste observable mais ne devient pas un budget séparé. Les coûts GPU sont
inventée. Cette couture est **PASS / FROZEN** sur le profil validé; ces IDs ne débités exactement une fois de la RAM hôte.
sont pas une politique portable. La cible RAM conserve une réserve dure de
3 GiB de `MemAvailable`; la zone 34 GiB est une prudence qui ne soustrait pas
4 GiB à toute capacité. Les petits hôtes dégradent le budget en conservant au
moins une unité de calcul. Les PSI CPU/mémoire/I/O et les deltas
swap-in/swap-out sont des signaux
actifs ; l'occupation totale du swap reste historique. Admission CPU et lot
sont indépendantes. Sur la 780M, Hardware Profile classe conservativement comme
UMA le petit aperture VRAM amdgpu de 512 Mio accompagné d'environ 7,99 Go de
GTT système ; la capacité rapportée reste observable mais ne devient pas un
budget séparé. Les coûts GPU sont débités exactement une fois de la RAM hôte.
L'audit Phase 1 couvre les 14 kinds de production et sépare leurs dimensions La nouvelle autorité humaine `SERIALISM_REQUIRES_PROOF` **supersède toute
fixes des dimensions réellement adaptables. Il confirme que tous passent par interprétation** de l'inventaire de maintenance qui ferait d'un ancien
l'unique Governor, même les formes fixes, et que le contrat reste immutable `CPU1`/`batch1` un plafond permanent. Les preuves scientifiques et résultats de
pendant une séquence. Cette tranche ferme l'enveloppe privée, la sélection AUTO, la maintenance restent FROZEN ; seules les enveloppes opérationnelles peuvent
les diagnostics bornés, l'enforcement OpenCV borné au compute-pool, la politique être rouvertes lorsqu'un travail indépendant est inutilement sérialisé. Une
d'affinité privée et la réconciliation du contexte retenu des campagnes Task fixe demeure fixe seulement si sa sérialité ou son genou de scaling est
nouvelles ou restaurées. La création/reprise AUTO ne touche plus Vulkan sur le prouvé. Cette règle doit être appliquée à toutes les Tasks de production avant
main ; le premier begin appartient au worker contraint. Inflight ORB normal est d'engager de nouvelles preuves réelles longues.
maintenant fixé à 1, helpers reste 0; depth 2 reste une capacité privée de
sûreté/benchmark. La clôture v2 est acquise. Ces choix ne créent aucune limite
de dataset, identité scientifique ou version Project DB.
La signature durable des nouvelles Tasks ORB normales est maintenant la classe L'audit Phase 1 couvre les kinds de production de l'époque et sépare leurs
`MIXED`, sémantiquement réelle pour une politique susceptible d'exécuter CPU ou dimensions fixes des dimensions réellement adaptables. Il confirme que tous
Vulkan. Elle reconstruit AUTO ; toutes les signatures CPU anciennes/courantes passent par l'unique Governor, même les formes fixes, et que le contrat reste
restent CPU fixes et Vulkan reste fixe, sans migration DB/codec. La couture immutable pendant une séquence. Cette tranche ferme l'enveloppe privée, la
asynchrone est privée, request-bound et nettoie son slot sur toute sortie. Une sélection AUTO, les diagnostics bornés, l'enforcement OpenCV borné au
preuve événementielle bornée établit la soumission du successeur avant la compute-pool, la politique d'affinité privée et la réconciliation du contexte
publication du prédécesseur pour deux paires 769×769. La rampe ne croît plus sur retenu des campagnes nouvelles ou restaurées. La création/reprise AUTO ne
la seule santé : l'adaptation générique exige deux observations par fenêtre, touche plus Vulkan sur le main ; le premier begin appartient au worker
tandis que le lot ORB Vulkan en exige huit, avec retour au palier accepté sans contraint. Inflight ORB normal est fixé à 1, helpers reste 0 ; depth 2 reste une
gain et reset immédiat sous pression. Ces capacité privée de sûreté/benchmark. Ces choix ne créent aucune limite de
éléments sont **PASS / FROZEN** dans les limites validées. dataset, identité scientifique ou version Project DB.
Les diagnostics de séquence distinguent maintenant backend sélectionné et La signature durable des nouvelles Tasks ORB normales est la classe `MIXED`,
backend réel ; les participants CPU Matcher restent `cpu_threads` et sémantiquement réelle pour une politique susceptible d'exécuter CPU ou Vulkan.
`helpers=0`. Une paire Vulkan inéligible ou en panne est recalculée entièrement Elle reconstruit AUTO ; toutes les signatures CPU anciennes/courantes restent
sur CPU sans preuve partielle. Les extractions ORB/SIFT/RootSIFT testées à CPU fixes et Vulkan reste fixe, sans migration DB/codec. La couture asynchrone
1/2/4/8/12 consomment leur contrat OpenCV immutable avec sorties égales. est privée, request-bound et nettoie son slot sur toute sortie. Une preuve
Le rolling distingue désormais un handle soumis d'une inéligibilité locale et événementielle bornée établit la soumission du successeur avant la publication
n'appelle jamais `finish` sans requête. La panne backend invalide immédiatement du prédécesseur pour deux paires 769×769. La rampe ne croît plus sur la seule
l'admission partagée sur toute sortie précoce ; seules les reprises AUTO peuvent santé : l'adaptation générique exige deux observations par fenêtre, tandis que
établir cette disponibilité, indépendamment de l'ordre des reprises fixes ou le lot ORB Vulkan en exige huit, avec retour au palier accepté sans gain et
historiques. Le statut est **PASS / FROZEN**. reset immédiat sous pression. Ces éléments sont **PASS / FROZEN** dans les
limites validées.
La boucle privée mesure désormais l'utilisation `/proc/stat` du compute-pool, Les diagnostics de séquence distinguent backend sélectionné et backend réel ;
les participants CPU Matcher restent `cpu_threads` et `helpers=0`. Une paire
Vulkan inéligible ou en panne est recalculée entièrement sur CPU sans preuve
partielle. Les extractions ORB/SIFT/RootSIFT testées à 1/2/4/8/12 consomment
leur contrat OpenCV immutable avec sorties égales. Le rolling distingue un
handle soumis d'une inéligibilité locale et n'appelle jamais `finish` sans
requête. La panne backend invalide immédiatement l'admission partagée sur toute
sortie précoce ; seules les reprises AUTO peuvent établir cette disponibilité,
indépendamment de l'ordre des reprises fixes ou historiques. Le statut est
**PASS / FROZEN**.
La boucle privée mesure l'utilisation `/proc/stat` du compute-pool,
`MemAvailable`, PSI mémoire/I/O `some/full`, deltas swap actifs, RSS/HWM observé `MemAvailable`, PSI mémoire/I/O `some/full`, deltas swap actifs, RSS/HWM observé
et GPU busy DRM, avec `unknown` sur absence ou parse non strict. Le backend et GPU busy DRM, avec `unknown` sur absence ou parse non strict. Le backend
Vulkan fournit des compteurs cumulatifs bornés de submit/complétion/fence/ Vulkan fournit des compteurs cumulés bornés de submit/complétion/fence/readback/
readback/GPU/starvation/panne/discard ; Matcher agrège en plus CPU et publication GPU/starvation/panne/discard ; Matcher agrège en plus CPU et publication par
par séquence. Le diagnostic est tirable par numéro de série, sans log ncurses ni séquence. Le diagnostic est tirable par numéro de série, sans log ncurses ni
histoire persistée. Les CPU réductibles progressent par puissances de deux vers histoire persistée. Les CPU réductibles progressent vers la capacité exacte du
la capacité exacte du kind/compute-pool après deux observations de baseline et kind/compute-pool selon les observations de scaling ; sous pression, l'admission
deux gains d'au moins 5 % ; CPU et lot ne changent se réduit puis doit pouvoir remonter. Candidate mesure chaque séquence. Un
jamais dans le même essai. Sous pression, une admission adaptative encore
permise réserve immédiatement CPU1 et lot minimum. Feature/SIFT/RootSIFT ne
comptent un item qu'après extraction et publication durable propre ; READY,
`ALREADY_PRESENT` et publication incertaine comptent zéro, comme un segment
Visual Index non durable.
Candidate mesure chaque séquence. Un
fallback réel annule l'essai Vulkan au lieu d'empoisonner sa baseline. Cette fallback réel annule l'essai Vulkan au lieu d'empoisonner sa baseline. Cette
implémentation est **PASS / FROZEN** sans ajouter de helper GPU. implémentation est **PASS / FROZEN** sans ajouter de helper GPU.
@ -359,105 +458,55 @@ digest `7a9dbc38a23a600379167d55e24836b7acbb22eea25573e7440bdc9e4602b3b3`.
La starvation depth 1 (53,847 s sur 74,613 s, GPU busy max 25 %) motive deux La starvation depth 1 (53,847 s sur 74,613 s, GPU busy max 25 %) motive deux
slots bornés mais ne constitue pas une mesure depth 2. Le backend partage slots bornés mais ne constitue pas une mesure depth 2. Le backend partage
device/pipeline/layout/cache et duplique seulement 640 Kio de payload, device/pipeline/layout/cache et duplique seulement 640 Kio de payload,
command/fence/descriptors/query par slot. Le payload mappé suit désormais command/fence/descriptors/query par slot. Le payload mappé suit exactement la
exactement la capacité de séquence admise : zéro avant initialisation, 640 Kio capacité de séquence admise : zéro avant initialisation, 640 Kio à depth 1 et
à depth 1 et 1,25 Mio uniquement pendant un contrôle privé depth 2, avec retour 1,25 Mio uniquement pendant un contrôle privé depth 2, avec retour à 640 Kio
à 640 Kio avant la prochaine admission depth 1. L'enveloppe normale n'essaie avant la prochaine admission depth 1. L'enveloppe normale n'essaie plus
plus inflight 2. Les générations de requête ne bouclent pas; un slot épuisé est inflight 2. Les générations de requête ne bouclent pas ; un slot épuisé est
retiré définitivement. retiré définitivement.
Le harness réel a exécuté le corpus 4113 paires pour le contrôle synchrone, Le harness réel a exécuté le corpus 4113 paires pour le contrôle synchrone,
rolling depth 1 et l'A/B forcé depth 1/depth 2. L'ABBA forcé donne rolling depth 1 et l'A/B forcé depth 1/depth 2. L'ABBA forcé donne
54,661652238 et 55,797311953 paires/s (+2,077617 %, sous le deadband 5 %). 54,661652238 et 55,797311953 paires/s (+2,077617 %, sous le deadband 5 %).
Chaque run porte 4113 paires durables; le débit combiné est Chaque run porte 4113 paires durables ; le débit combiné est
`(2 * 4113 * 1e9) / somme(wall_ns)`, pas la moyenne des débits par run. Les `(2 * 4113 * 1e9) / somme(wall_ns)`, pas la moyenne des débits par run. Les
walls bruts 75326831673/75162582080 et 73662096698/73764360098 ns donnent les walls bruts 75326831673/75162582080 et 73662096698/73764360098 ns donnent les
moyennes 75,244706877/73,713228398 s. Fence vaut 6,0684/3,6776 s, starvation 54,4534/50,1465 s, moyennes 75,244706877/73,713228398 s. Fence vaut 6,0684/3,6776 s, starvation
publication 29,2582/30,0548 s, submit CPU 0,2655/0,3818 s, readback 54,4534/50,1465 s, publication 29,2582/30,0548 s, submit CPU 0,2655/0,3818 s,
0,0460/0,0873 s et GPU busy max 23/24 %. Les quatre exécutions conservent le readback 0,0460/0,0873 s et GPU busy max 23/24 %. Les quatre exécutions
digest `7a9dbc38a23a600379167d55e24836b7acbb22eea25573e7440bdc9e4602b3b3`, quatre conservent le digest `7a9dbc38a23a600379167d55e24836b7acbb22eea25573e7440bdc9e4602b3b3`,
séquences de fallback local par exécution et zéro panne/discard. Depth 2 est donc quatre séquences de fallback local par exécution et zéro panne/discard. Depth 2
**REJECTED_WITH_MEASURED_REASON** pour la politique normale : est donc **REJECTED_WITH_MEASURED_REASON** pour la politique normale :
`DEPTH_MAX_VALIDATED_SAFETY=2`, `DEPTH_MAX_USEFUL=1`. Une comparaison `DEPTH_MAX_VALIDATED_SAFETY=2`, `DEPTH_MAX_USEFUL=1`.
whole-corpus adaptative antérieure (+1,12 %) ne séparait pas les changements de
contrat; l'ABBA ci-dessus la remplace comme décision de profondeur. Le
répertoire retenu est
`/home/fy59/Documents/Lardon/.real-pre-sfm-2026-08-30/governor-v2-evidence/`,
avec `forced-depth1-a.stdout.jsonl`, `forced-depth1-b.stdout.jsonl`,
`forced-depth2-a.stdout.jsonl` et `forced-depth2-b.stdout.jsonl`. Le
mode Matcher par défaut est AUTO/rolling ; CPU et Vulkan explicites restent des
overrides. Un contrôle synchrone fence par fence est compilé seulement dans le
runner/test, absent de `lardon3d`, non persisté et non applicable à une reprise
Matcher pendante. Les fixtures et ces deux exécutions établissent l'égalité
exacte des sorties; les chiffres depth 1 sont rapportés ci-dessus. Le runner émet
des diagnostics JSON échantillonnés sans prétendre voir chaque séquence, plus
un agrégat Governor fixe exact pour les compteurs enregistrés. Il valide la
bijection Candidate Pair/Match Result, les assets et le curseur, puis produit le
digest canonique de comparaison `L3DMRD1`, qui exclut IDs Task, timestamps et
choix opérationnels. Ces preuves ferment **PASS / FROZEN** sans promouvoir
depth 2 dans AUTO normal.
Le runner possède maintenant, dans ce seul target, `--matcher-inflight 1|2` : Les huit runs item-valides `forced-batch{2,4,8,12}-items{,-b}.stdout.jsonl`
AUTO rolling fixe batch 2 et min=max inflight pour permettre la mesure du même publient chacun 4113 paires, six items locaux, zéro panne/autre et le même
binaire à profondeur 1 puis 2, en n'exposant que la capacité Vulkan forcée. digest. Leurs débits combinés sont 54,180767704, 66,094373197, 74,784998723 et
`--matcher-batch 2|4|8|12`, valable seulement avec inflight et AUTO rolling, 76,755814095 paires/s. Batch 4 puis 8 gagnent +21,988624373 % et
fixe aussi min=max batch pour la prochaine matrice contrôlée. +13,148812987 % ; batch 12 ne gagne que +2,635308425 %, sous le deadband 5 %.
Synchronous reste depth 1. Le Governor conserve son admission et sa charge UMA; AUTO normal suit donc `BATCH_MAX_USEFUL=8`; batch 12 reste une capacité privée
GPU budget zéro ou capacité GPU/backend/mémoire indisponible échoue au lieu de sûre `REJECTED_WITH_MEASURED_REASON`. Le contrôle de production sans override
sélectionner CPU. La valeur n'est ni persistée ni scientifique, `short-auto-batch8-governor-v2.stdout.jsonl` atteint réellement `1 → 2 → 4 → 8`,
est refusée sur une Task pendante, restaurée dans l'environnement à toute sortie puis publie 4113/4113 résultats à 76,072 paires/s avec le même digest, six
et émise comme `1`, `2` ou `null` dans les résumés/agrégats. Le runner valide le fallbacks locaux, zéro panne/discard, inflight 1 et helpers 0.
contrat exact et zéro panne/discard/pending; seule l'inéligibilité locale peut
produire un fallback CPU complet. La production compte désormais chaque item La preuve de fermeture S21 `final-s21-auto.stdout.jsonl` exécute le chemin
exactement une fois après publication durable dans local-ineligible, production normal sur 172 741 Candidate Pairs : 172 741 Match Results, zéro
backend-failure ou other; une admission CPU normale reste à zéro. Le compteur doublon, curseur complet, digest
est incrémenté immédiatement et reste visible si une paire suivante échoue ou est `e5128a2e599ff593c4f79850e067254b1f249d19e8480a44973306b1af250f70` et
annulée, sans valider ni entraîner la séquence incomplète; son high-water mark 73,649 résultats durables/s. AUTO choisit Vulkan sur toutes les admissions,
de déduplication reste privé et non persisté. L'égalité
inter-cohortes porte sur les items locaux, pas sur les séquences dont le nombre
dépend du batch. Une panne/raison autre, ou une panne tardive, invalide la
cohorte; la Task de benchmark échoue après checkpoint de toute preuve CPU déjà
durable. Les fixtures
établissent l'égalité exacte rolling1/rolling2/synchrone1 et des batches
2/4/8/12 sur fixture, avec le même compte de 29 items locaux malgré des comptes
de séquences différents. Les logs corpus préliminaires
`forced-batch2-current.stdout.jsonl` et `forced-batch4.stdout.jsonl` couvrent
les mêmes 4113 IDs/digest mais portent quatre contre trois séquences locales :
ils sont conservés comme preuve que l'ancien comparateur était invalide et ne
participent pas à la décision. Les huit runs item-valides
`forced-batch{2,4,8,12}-items{,-b}.stdout.jsonl` publient chacun 4113 paires,
six items locaux, zéro panne/autre et le même digest. Leurs débits combinés
sont 54,180767704, 66,094373197, 74,784998723 et 76,755814095 paires/s. Batch
4 puis 8 gagnent +21,988624373 % et +13,148812987 %; batch 12 ne gagne que
+2,635308425 %, sous le deadband 5 %. AUTO normal suit donc
`BATCH_MAX_USEFUL=8`; batch 12 reste une capacité privée sûre
`REJECTED_WITH_MEASURED_REASON`.
Le contrôle de production sans override
`short-auto-batch8-governor-v2.stdout.jsonl` atteint réellement
`1 → 2 → 4 → 8`, puis publie 4113/4113 résultats à 76,072 paires/s avec le
même digest, six fallbacks locaux, zéro panne/discard, inflight 1 et helpers 0.
La preuve de fermeture S21
`final-s21-auto.stdout.jsonl` exécute ensuite le chemin production normal sur
172 741 Candidate Pairs : 172 741 Match Results, zéro doublon, curseur complet,
digest `e5128a2e599ff593c4f79850e067254b1f249d19e8480a44973306b1af250f70`
et 73,649 résultats durables/s. AUTO choisit Vulkan sur toutes les admissions,
termine batch 8/inflight 1/helpers 0 et ne compte aucune panne/discard/pending. termine batch 8/inflight 1/helpers 0 et ne compte aucune panne/discard/pending.
Une admission YELLOW réduit batch 8 à 1, puis les séquences GREEN rétablissent Une admission YELLOW réduit batch 8 à 1, puis les séquences GREEN rétablissent
1 → 2 → 4 → 8; le gate possède donc aussi une preuve réelle de recovery. 1 → 2 → 4 → 8 ; le gate possède donc une preuve réelle de recovery.
La poursuite S21 ferme ensuite le point Geometric Verifier v3 sans rejouer le La poursuite S21 ferme le Geometric Verifier v3 sans rejouer le Matcher. La Task
Matcher. La Task 2832 consomme les 172 741 Match Results et publie 172 275 GVR 2832 consomme les 172 741 Match Results et publie 172 275 GVR v3, dont 24 065
v3, dont 24 065 acceptés et 148 210 rejetés, avec zéro doublon et curseur acceptés et 148 210 rejetés, avec zéro doublon et curseur complet. La seconde
complet. Le digest Matcher reste reprise crée zéro ligne ; une Task interrompue sur une copie dédiée reprend le
`e5128a2e599ff593c4f79850e067254b1f249d19e8480a44973306b1af250f70`. même ID et converge vers les mêmes lignes exactes. Feature, Candidate et
La seconde reprise crée zéro ligne ; une Task interrompue sur une copie dédiée Matcher ne sont pas rejoués. `REAL_S21_GV_V3=PASS/FROZEN`.
reprend le même ID et converge vers les mêmes lignes exactes. Feature,
Candidate et Matcher ne sont pas rejoués. Track Builder et Sparse SfM ne sont
pas exécutés. `REAL_S21_GV_V3=PASS/FROZEN` ; la prochaine tranche scientifique
S21 commencera donc à Tracks maintenant que la gate de maintenance globale
ci-dessous est fermée. Tracks n'est pas acquis.
Les gates de fermeture sont acquis : Les gates de fermeture acquises incluent :
```text ```text
GOVERNOR_CONTROLS_ALL_TASKS=PASS GOVERNOR_CONTROLS_ALL_TASKS=PASS
@ -489,39 +538,17 @@ REAL_S21_GV_V3=PASS/FROZEN
FEATURE_REPLAY=0 FEATURE_REPLAY=0
CANDIDATE_REPLAY=0 CANDIDATE_REPLAY=0
MATCHER_REPLAY=0 MATCHER_REPLAY=0
TRACKS_EXECUTED=0
SPARSE_SFM_EXECUTED=0 SPARSE_SFM_EXECUTED=0
``` ```
L'expérience normale est donc : l'utilisateur lance une Task; l'unique L'expérience normale est donc : l'utilisateur lance une Task ; l'unique
Governor choisit et explique le contrat borné de sa prochaine séquence. Aucun Governor choisit et explique le contrat borné de sa prochaine séquence. Aucun
réglage CPU/GPU/lot/inflight/helper n'est requis en production ordinaire. réglage CPU/GPU/lot/inflight/helper n'est requis en production ordinaire. Avec
la nouvelle autorité humaine, ce choix doit désormais rechercher explicitement
L'ordre de travail autorisé à court terme est : le **maximum sûr et utile**, et non conserver une sous-utilisation historique.
```text
fondation scientifique pré-SfM courante
→ INTERNAL PARALLELISM + COMPUTE RESOURCES v1
(Candidate, Matcher, Visual Index, audit feature threading,
correction progression/Resource Governor, audit GPU)
→ COMPUTE GOVERNOR v2 / ORB VULKAN ASYNC EXECUTION (PASS / FROZEN)
→ REAL S21 GV v3 (PASS / FROZEN; arrêt avant Tracks)
→ GLOBAL MAINTENANCE AUDIT
(PASS / FROZEN; validation et revue finale indépendante acquises)
→ poursuite pré-SfM réelle de S21 à partir de Tracks
→ acquisition dédiée de calibration
→ Sparse SfM réel
→ Dense / MVS
```
## GLOBAL MAINTENANCE AUDIT — PASS / FROZEN ## GLOBAL MAINTENANCE AUDIT — PASS / FROZEN
Le jalon scientifique acquis reste arrêté après GV. Avant Tracks, Sparse SfM
ou Dense/MVS, la gate de maintenance globale a réconcilié l'architecture, les
ressources, la persistance et l'observation TUI. Son état et ses preuves
consolidées sont consignés dans le
[registre canonique de maintenance](../architecture/global_maintenance_audit.md).
Le checkpoint canonique de revue est le tag Le checkpoint canonique de revue est le tag
`global-maintenance-2026-09-01`, au commit `global-maintenance-2026-09-01`, au commit
`b84f860d868c66d9ee84b85ceb1bc6480b95aca5`. Les revues futures sont `b84f860d868c66d9ee84b85ceb1bc6480b95aca5`. Les revues futures sont
@ -532,44 +559,30 @@ dépendances traversées. Les systèmes PASS/FROZEN inchangés héritent de la p
du [registre de maintenance](../architecture/global_maintenance_audit.md) et ne du [registre de maintenance](../architecture/global_maintenance_audit.md) et ne
sont rouverts que sur preuve concrète ; ne pas répéter un audit global A-à-Z. sont rouverts que sur preuve concrète ; ne pas répéter un audit global A-à-Z.
Les résultats déjà réglés sont : Les résultats acquis restent notamment :
- Project DB v23 ajoute neuf relations optiques sans backfill ni inférence ; - Project DB v23 ajoute neuf relations optiques sans backfill ni inférence ;
les boîtiers, objectifs manuels/électroniques, configurations et calibrations - le Governor réserve l'hôte sans plafond CPU global 12, avec réserve RAM dure
restent distincts, et la sélection de calibration exige une compatibilité 3 GiB, PSI/swap actif et UMA comptée une fois ;
exacte et explicite ; - Candidate, Visual Index, Feature, Matcher et GV possèdent leurs capacités et
- le Governor réserve d'abord l'hôte, sans plafond CPU global 12 : réserve de décisions GPU mesurées ;
quatre CPU logiques sur hôte capable, groupes cœur/SMT complets lorsque la
topologie est fiable, au moins une unité de calcul sur petit hôte, réserve
RAM dure 3 GiB et prudence 34 GiB, PSI/swap actif et UMA comptée une fois ;
- Candidate porte une capacité sûre 64, Visual Index 16, ORB/SIFT/RootSIFT sont
bornés par le compute-pool, Matcher conserve sa borne intrinsèque sûre 12 et
utile 8, et GV conserve son USAC interne sériel mais admet 16 participants
sûrs/8 utiles et 16 parents par lot ;
- le contrôleur SSD UDisks2 optionnel est une frontière physique revue, avec - le contrôleur SSD UDisks2 optionnel est une frontière physique revue, avec
identité Drive+labels+UUID, leases, drain sûr et latch de danger pour action identité Drive+labels+UUID, leases et drain sûr ;
indéterminée. Son état est enregistré auprès du Governor, seul orchestrateur - la TUI est un observatoire/centre de contrôle validé opérationnellement ;
des leases de production ; il n'est ni scheduler ni second Governor ;
- la TUI est un observatoire/centre de contrôle validé opérationnellement :
ncurses main-thread, modèle pur, observation coalescée et bornée, progression
durable/ETA, pipeline et ressources honnêtes, profils optiques, layouts
100×30/72×20/60×15, repli texte/couleur et F10 SSD asynchrone toujours
visible ;
- la frontière de session détruit/joint la Queue avant Project DB, puis recrée - la frontière de session détruit/joint la Queue avant Project DB, puis recrée
une seule Queue ; l'arrêt global libère les leases Task avant unregister SSD, une seule Queue.
contrôleur et Governor.
Les validations finales exécutables sont acquises : build Clang portable Le nouvel impératif `SERIALISM_REQUIRES_PROOF` ne nie aucune de ces preuves. Il
931/931 + suite 64/64, build Clang Vulkan 939/939 + suite 65/65 sur Radeon supersède uniquement l'idée qu'un ancien profil opérationnel conservateur serait
réelle, ASan/UBSan 64/64 avec la limitation LSan OpenCL externe explicitement un plafond permanent lorsque de nouvelles preuves réelles montrent du travail
qualifiée, LSan loader-free 20/20, TSan 14/14 + 220 répétitions, headers publics indépendant et des ressources sûres inutilisées.
76/76 sur 19 headers modifiés/nouveaux et contrôles ABI/diff/`scan3d`.
L'unique revue finale indépendante GPT-5.6 SOL/ULTRA a conclu PASS sans finding Les validations finales exécutables de la maintenance sont acquises : build
bloquant après avoir indépendamment rejoué le build portable, la suite 64/64, Clang portable 931/931 + suite 64/64, build Clang Vulkan 939/939 + suite 65/65
15/15 tests focalisés, les 76/76 probes de headers, l'ABI, les négatifs de seams sur Radeon réelle, ASan/UBSan 64/64 avec la limitation LSan OpenCL externe
production, le SHA du manifest GV retenu et le diff-check. La gate de qualifiée, LSan loader-free 20/20, TSan, headers publics 76/76 et contrôles
maintenance est donc fermée ; la poursuite réelle depuis Tracks devient la ABI/diff/`scan3d`. La gate de maintenance reste fermée ; les nouvelles
prochaine tranche séparée, sans avoir été exécutée par cette synchronisation. corrections de ressources sont revues en delta et ne réouvrent pas l'audit A→Z.
## REAL S21 TRACKS — PASS / FROZEN ## REAL S21 TRACKS — PASS / FROZEN
@ -606,8 +619,175 @@ runner corrigé. Elle termine `COMPLETE` à 100 % et retrouve le même Track Set
zéro Track Set. Cette preuve valide la reprise opérationnelle sans rouvrir le zéro Track Set. Cette preuve valide la reprise opérationnelle sans rouvrir le
contrat scientifique Tracks gelé. contrat scientifique Tracks gelé.
## PROJECT DB v24 / RAW BATCH A6000 — IMPLEMENTED / VALIDATION IN PROGRESS
L'autorisation humaine additive v24 est limitée au parallélisme opérationnel du
développement RAW. Project DB v24 conserve v22/v23 et ajoute uniquement la
persistance typée `raw_development_batch_tasks` nécessaire au nouveau kind
`raw.develop.batch/1`. Cette version ne change aucune identité scientifique,
aucune calibration ni `L3DRAWD1`.
Le pattern de production est :
```text
une Task owner admise par le Governor
→ participants RAW indépendants et bornés
→ join de tous les participants
→ publication owner-only dans l'ordre des selected_item_index
→ avance durable du curseur selected_execution
```
L'atomicité par Capture est conservée ; la sérialisation entre Captures
indépendantes ne l'est plus. La transition historique `raw.develop/1`
`raw.develop.batch/1` reste légitime et ne réécrit aucune ancienne Task.
État durable A6000 au dernier arrêt sûr avant l'audit global d'utilisation des
ressources :
```text
Projet:
/home/fy59/Documents/Lardon/.real-pre-sfm-2026-09-01/a6000-pre-sfm-v23-final
Schema v24
Captures 953
GOOD 689
SUSPECT 206
REJECT 58
selected_execution 1
selected items 689 RAW_ASSET
published representations 259
next_item_index 259
remaining RAW representations 430
Features 0
Visual Index 0
Candidate 0
Matcher 0
GV 0
Tracks 0
Sparse SfM 0
Dense/MVS 0
```
Le digest d'inventaire source pré/post est
`fa725d8a82b529521134dd600b5cf42ed58ad523108636741a1431441e17b029`.
Les JPEG caméra et RAW sont tous deux des SOURCE assets explicites par Capture,
mais cette exécution A6000 FROZEN utilise les PNG déterministes dérivés RAW pour
la géométrie. Les JPEG restent les proxies de triage ; ils ne remplacent pas
cette représentation scientifique sans nouvelle tranche d'équivalence
scientifique/calibration.
La v24 et la Task batch ont déjà passé build normal, matrice ciblée, contrôles
C17, `git diff --check`, ASan/UBSan ciblé et TSan sur la couture concurrente.
La preuve réelle complète, l'équivalence finale, l'audit d'utilisation de toutes
les Tasks affectées et la fermeture documentaire restent à terminer avant de
marquer cette tranche `PASS/FROZEN`.
## PROJECT DB v25 / FEATURE BATCH — IMPLEMENTED / VALIDATION IN PROGRESS
L'overlay additif v25 ajoute uniquement la persistance typée de
`features.extract.batch/1`. Il ancre un propriétaire durable à l'ordre immutable
de l'exécution sélectionnée, au domaine ORB exact et à un curseur monotone. Les
Tasks `features.extract/1` historiques restent inchangées et sont reprises avant
le suffixe batch. La préparation inter-images est bornée par le Governor,
OpenCV reste CPU1 dans chaque participant, et publication/cursor/checkpoint
restent owner-only et ordonnés. Le build et les tests ciblés sont acquis ; le
benchmark réel du palier utile et la preuve A6000 restent à exécuter avant toute
revendication PASS/FROZEN.
## REAL A6000 PRE-SFM — PASS / FROZEN
`REAL_A6000_PRE_SFM=PASS/FROZEN`. La continuation du 2 septembre 2026 utilise
le projet durable
`/home/fy59/Documents/Lardon/.real-pre-sfm-2026-09-01/a6000-pre-sfm-v23-final`
directement à la frontière Match Result acquise. Les 689 Feature Sets, le Visual
Index complet, les 38 420 Candidate Pairs et les 38 420 Match Results de la
Task Matcher 170 `COMPLETE/100` sont conservés :
`FEATURE_REPLAY=0`, `CANDIDATE_REPLAY=0` et `MATCHER_REPLAY=0`. Ni l'acquisition,
ni RAW, ni les assets/identités amont ne sont réécrits.
La Task GV v3 171 termine `COMPLETE/100`, consomme le curseur Match Result
jusqu'à 38 420 et publie exactement 37 805 GVR applicables : 10 952
`GEOMETRIC_VERIFIED` et 26 853 `GEOMETRIC_REJECTED`. Son fingerprint reste
`6944a471d611d8ffc59dac7cf15a5b79b97e2371d4c51785c477d68c1577f74c`;
zéro mapping GVR dupliqué est observé. Les diagnostics Governor coalescés
retiennent 2 714 admissions, le compute-pool de 12 CPU avec 4 CPU réservés,
une fenêtre finale de 16, RSS/HWM maximal 41 877 504/42 319 872 octets,
`MemAvailable` minimal 10 555 641 856 octets et des deltas swap-in/out nuls.
La préparation GV reste dans la forme validée Governor-admise et bornée ; aucune
nouvelle policy CPU/GPU n'est introduite par ce run.
La Task Track Builder 172 termine `COMPLETE/100` et publie atomiquement le seul
Track Set 1, de scope 10 952 GVR, avec 130 714 Tracks et 318 944 observations.
L'audit SQL confirme zéro observation de Track dupliquée, zéro Track contenant
deux observations de la même image et zéro observation orpheline. La seconde
continuation crée seulement la Task GV 173, qui traverse le même curseur,
publie zéro GVR et réutilise Track Set 1 sans créer de Task Track Builder ni
modifier les 130 714 Tracks. `RESTART_IDEMPOTENCE`, `DETERMINISM`,
`TRACK_CONFLICT_INVARIANTS` et `RESOURCE_POLICY_ENFORCED` sont donc PASS pour
cette frontière réelle.
Sparse SfM, reconstruction sparse, Dense/MVS et fusion inter-campagnes restent
à zéro. Ce gel atteste uniquement le pré-SfM A6000 à travers GV puis Tracks ; il
ne contourne pas le blocage de calibration réel et ne promeut aucune étape aval.
## RESOURCE UTILIZATION ENFORCEMENT — CURRENT NEXT
Avant de reprendre les 430 RAW restants ou toute autre preuve réelle longue,
la prochaine tranche opérationnelle doit appliquer la politique canonique à
l'ensemble des Tasks de production, en revue delta uniquement.
Objectifs :
```text
RESOURCE_UTILIZATION_POLICY=IMPLEMENTED
SERIALISM_REQUIRES_PROOF=ENFORCED
ACCIDENTAL_CPU1_PATHS=0_OR_MEASURED_JUSTIFICATION
VALIDATED_GPU_BACKENDS=PREFERRED_WHEN_ELIGIBLE
BUILD_TEST_PARALLELISM=HOST_RESERVE_AWARE
```
Pour chaque Task, auditer uniquement les dimensions opérationnelles :
minimum/useful/safe CPU et batch, indépendance des items, séparation
préparation/publication, coûts RAM par participant, saturation I/O et backend
GPU validé. Une valeur CPU1/batch1 peut rester seulement avec preuve concrète.
Il ne faut pas inventer de second scheduler, pool global ou parallélisme
inter-Tasks pour satisfaire cette politique : le parallélisme interne borné et
l'unique Queue/Governor restent la voie normale.
Les tests Codex doivent suivre la même règle : build et tests indépendants
parallèles jusqu'à la réserve interactive ; serialisation uniquement pour
fixtures partagées, déterminisme scientifique, mémoire/I/O/GPU exclusif,
sanitizer ou pression mesurée. Les validations acquises et non affectées ne
sont pas répétées. Les modèles/agents coûteux sont réservés aux vraies
frontières scientifiques, concurrence/persistance délicate et revue finale
nécessaire.
## NEAR TERM ## NEAR TERM
Ordre court terme désormais canonique :
```text
GLOBAL MAINTENANCE AUDIT PASS / FROZEN
→ REAL S21 TRACKS PASS / FROZEN
→ RESOURCE UTILIZATION ENFORCEMENT CURRENT NEXT
(SERIALISM_REQUIRES_PROOF, max safe/useful throughput,
politique tests/builds Codex comprise)
→ reprise A6000 v24 au curseur 259
→ 430 représentations RAW restantes
→ Features
→ Visual Index
→ Candidate
→ Matcher AUTO / ORB Vulkan si éligible
→ GV
→ Tracks
→ STOP avant Sparse SfM
→ acquisition physique dédiée de calibration
→ calibration connue validée
→ Sparse SfM réel par campagne
→ fusion / Phase H selon dépendances
→ Dense / MVS
```
### Scratch SSD externe et swap optionnel ### Scratch SSD externe et swap optionnel
Le contrôleur physique actuel découvre par UDisks2 une paire exacte de labels Le contrôleur physique actuel découvre par UDisks2 une paire exacte de labels
@ -621,9 +801,8 @@ Les usages futurs possibles sont workspace/scratch et intermédiaires
dense/mesh/texturing. Leur consommation devra être explicitement possédée par dense/mesh/texturing. Leur consommation devra être explicitement possédée par
les Tasks et passer par les wrappers de lease du Resource Governor ; le les Tasks et passer par les wrappers de lease du Resource Governor ; le
contrôleur physique et le registre actuel n'inventent aucune éligibilité Task. contrôleur physique et le registre actuel n'inventent aucune éligibilité Task.
Les quatorze kinds courants ne consomment aucun scratch. Le swap de sécurité, Le swap de sécurité reste une fonction du drain physique explicite et jamais un
lui, reste une fonction du drain physique explicite et jamais un budget de budget de travail.
travail.
- les leases scratch ont une ownership explicite ; `DRAINING` refuse les - les leases scratch ont une ownership explicite ; `DRAINING` refuse les
nouveaux leases et attend leur libération exacte ; nouveaux leases et attend leur libération exacte ;
@ -636,14 +815,11 @@ travail.
- une action UDisks potentiellement appliquée mais non vérifiable verrouille le - une action UDisks potentiellement appliquée mais non vérifiable verrouille le
tuple physique original ; un remplacement n'obtient aucune autorité ; tuple physique original ; un remplacement n'obtient aucune autorité ;
- aucun formatage, partitionnement, fsck, réparation, arrêt forcé ou suppression - aucun formatage, partitionnement, fsck, réparation, arrêt forcé ou suppression
n'est permis. La production n'appelle pas `statvfs`; espace total/libre reste n'est permis.
`UNKNOWN` quand UDisks ne le fournit pas.
Le contrôleur, son registre Governor, le worker joinable unique et sa Le contrôleur, son registre Governor, le worker joinable unique et sa
présentation/actions F10 sont **CURRENT / VALIDATED OPERATIONAL**. Les tests présentation/actions F10 sont **CURRENT / VALIDATED OPERATIONAL**. Ce statut ne
emploient un provider factice et ne montent, n'arrêtent ni ne modifient un vrai crée aucun consommateur scratch par lui-même.
disque. Ce statut ne crée aucun consommateur scratch ; c'est la validation
consolidée et la revue indépendante ci-dessus qui ont fermé la gate globale.
### Publication durable dense / mesh ### Publication durable dense / mesh
@ -715,8 +891,7 @@ Le but à long terme n'est pas seulement de reconstruire ce qui a été
photographié, mais de guider activement l'opérateur vers les photographies qui photographié, mais de guider activement l'opérateur vers les photographies qui
manquent encore pour obtenir une reconstruction fiable. Cette capacité est manquent encore pour obtenir une reconstruction fiable. Cette capacité est
postérieure à une reconstruction suffisamment mature ; elle ne fait pas partie postérieure à une reconstruction suffisamment mature ; elle ne fait pas partie
de S3 et ne remplace pas le prochain milestone d'intégration multi-campagne de S3.
A6000 + S21.
```text ```text
photographies existantes photographies existantes
@ -933,20 +1108,25 @@ L'ordre demeure sans ambiguïté :
```text ```text
CURRENT NEXT CURRENT NEXT
fondation scientifique pré-SfM courante GLOBAL MAINTENANCE AUDIT PASS / FROZEN
→ INTERNAL PARALLELISM + COMPUTE RESOURCES v1 → REAL S21 TRACKS PASS / FROZEN
→ COMPUTE GOVERNOR v2 / ORB VULKAN ASYNC EXECUTION → RESOURCE UTILIZATION ENFORCEMENT CURRENT NEXT
(PASS / FROZEN; AUTO GPU-first ORB et preuve S21 Matcher complète) → A6000 PRE-SFM v24 resume from cursor 259
→ REAL S21 GV v3 (PASS / FROZEN; Tracks/Sparse SfM non exécutés) → RAW batch
→ GLOBAL MAINTENANCE AUDIT → Features
(PASS / FROZEN; gate fermée avant la tranche Tracks séparée) → Visual Index
→ poursuite pré-SfM réelle de S21 à partir de Tracks → Candidate
→ acquisition physique dédiée de calibration → Matcher AUTO / validated Vulkan when eligible
→ calibration connue validée → Sparse SfM réel multi-campagne → GV
→ Tracks
→ STOP before Sparse SfM
→ dedicated physical calibration acquisition
→ known calibration → real Sparse SfM
→ multi-campaign registration/fusion
→ dense / MVS → publication → dense / MVS → publication
→ LATER → LATER
Coverage Analysis → Coverage Viewer → suggestions de points de vue Coverage Analysis → Coverage Viewer → suggested viewpoints
→ localisation live → intégration HDMI/USB → Capture Guidance / Live Coverage → live localization → HDMI/USB integration → Capture Guidance / Live Coverage
``` ```
Vidéo/keyframes pourra progresser en parallèle lors d'une phase ultérieure, Vidéo/keyframes pourra progresser en parallèle lors d'une phase ultérieure,
@ -961,21 +1141,34 @@ scientifique.
## DEFERRED / OPTIONAL ## DEFERRED / OPTIONAL
- pools multiples CPU/GPU/IO, parallélisme inter-tâches et multi-GPU ; - pools multiples CPU/GPU/IO, parallélisme inter-Tasks et multi-GPU ;
- DAG général de dépendances et priorités complexes ; - DAG général de dépendances et priorités complexes ;
- infrastructure backend générique et distribution de calcul ; - infrastructure backend générique et distribution de calcul ;
- ALIKED tant que provenance modèle, export ONNX et oracle upstream ne sont pas - ALIKED tant que provenance modèle, export ONNX et oracle upstream ne sont pas
reproductibles. reproductibles.
Le report du parallélisme **inter-Tasks** ne constitue jamais une permission de
sérialiser accidentellement les unités indépendantes **à l'intérieur d'une Task
active**. L'unique Queue/Governor et le fanout interne borné suffisent pour
appliquer `SERIALISM_REQUIRES_PROOF` sans créer un second runtime.
Ces idées ne doivent pas devancer l'intégration réelle multi-campagne ni créer Ces idées ne doivent pas devancer l'intégration réelle multi-campagne ni créer
un second runtime, Governor ou système de persistance. un second runtime, Governor ou système de persistance.
## Principes de séquencement ## Principes de séquencement
1. stabilité et intégrité scientifique avant débit ; 1. intégrité scientifique et réserve interactive hôte sont inviolables ; à
2. résultats atomiques et durables avant parallélisme ; l'intérieur de ces limites, le maximum de débit sûr et utile est obligatoire ;
3. lots bornés et reprise avant taille de campagne ; 2. `SERIALISM_REQUIRES_PROOF` : l'atomicité d'un item ne sérialise pas les items
4. un Task Runtime/Queue et un Resource Governor, sans second scheduler ; indépendants et toute sous-utilisation durable doit avoir une raison mesurée ;
5. scratch optionnel sans élargissement implicite des budgets RAM ; 3. résultats atomiques et durables avant publication concurrente ; préparation
6. TUI de contrôle avant visualisation riche ; parallèle bornée autorisée lorsque l'égalité exacte est préservée ;
7. documentation canonique alignée sur le code validé. 4. lots bornés, reprise et mémoire honnêtement comptée avant taille de campagne ;
5. un Task Runtime/Queue et un Resource Governor, sans second scheduler ;
6. backend GPU validé et utile préféré automatiquement, sans GPU artificiel ;
7. scratch optionnel sans élargissement implicite des budgets RAM ;
8. builds/tests/preuves Codex parallèles jusqu'à la réserve hôte lorsqu'ils sont
indépendants ; aucune sérialisation ou revalidation lourde par habitude ;
9. TUI de contrôle avant visualisation riche ;
10. documentation canonique alignée sur le code validé et revue delta-based
depuis `global-maintenance-2026-09-01`.

View file

@ -10,6 +10,8 @@
#define LARDON3D_FEATURE_EXTRACT_TASK_KIND "features.extract" #define LARDON3D_FEATURE_EXTRACT_TASK_KIND "features.extract"
enum { LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION = 1 }; enum { LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION = 1 };
#define LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND "features.extract.batch"
enum { LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION = 1 };
Lardon3DTask * Lardon3DTask *
lardon3d_project_create_feature_extract_task(Lardon3DAppState *state, uint64_t image_id, lardon3d_project_create_feature_extract_task(Lardon3DAppState *state, uint64_t image_id,
@ -20,5 +22,18 @@ bool lardon3d_project_enqueue_feature_extract(Lardon3DAppState *state, uint64_t
uint64_t *task_id); uint64_t *task_id);
bool lardon3d_feature_extract_reconstruct(const Lardon3DTaskDurableSnapshot *snapshot, bool lardon3d_feature_extract_reconstruct(const Lardon3DTaskDurableSnapshot *snapshot,
void *context, Lardon3DTaskKindBinding *binding); void *context, Lardon3DTaskKindBinding *binding);
/* Create/enqueue one durable owner Task over the immutable selected-execution
* image order. ORB parameters are copied into typed v25 state; participants are
* Governor-bounded, cancellable at window boundaries, and joined before the
* owner publishes or advances its cursor. */
Lardon3DTask *lardon3d_project_create_feature_extract_batch_task(
Lardon3DAppState *state, uint64_t selected_execution_id,
const Lardon3DFeatureExtractorParameters *parameters, uint64_t *task_id);
bool lardon3d_project_enqueue_feature_extract_batch(
Lardon3DAppState *state, uint64_t selected_execution_id,
const Lardon3DFeatureExtractorParameters *parameters, uint64_t *task_id);
bool lardon3d_feature_extract_batch_reconstruct(
const Lardon3DTaskDurableSnapshot *snapshot, void *context,
Lardon3DTaskKindBinding *binding);
#endif #endif

View file

@ -73,9 +73,21 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_photo_quality_ta
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_raw_development_task( Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_raw_development_task(
Lardon3DAppState *state, const Lardon3DTask *task, Lardon3DAppState *state, const Lardon3DTask *task,
const Lardon3DProjectDbRawDevelopmentTask *parameters); const Lardon3DProjectDbRawDevelopmentTask *parameters);
/* Persist generic parent Task state atomically with its immutable selected
* execution association. Call only after the selected execution cursor commit
* so generic progress can lag but never lead scientific publication. */
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_raw_development_batch_task(
Lardon3DAppState *state, const Lardon3DTask *task,
const Lardon3DProjectDbRawDevelopmentBatchTask *parameters);
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_task( Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_task(
Lardon3DAppState *state, const Lardon3DTask *task, Lardon3DAppState *state, const Lardon3DTask *task,
const Lardon3DProjectDbFeatureExtractTask *parameters); const Lardon3DProjectDbFeatureExtractTask *parameters);
/* Persist the generic checkpoint and typed batch domain/cursor in one SQLite
* transaction. Call only after owner-only Feature publication and cursor
* advancement, so generic progress may lag but never lead durable work. */
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_batch_task(
Lardon3DAppState *state, const Lardon3DTask *task,
const Lardon3DProjectDbFeatureExtractBatchTask *parameters);
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_sift_extract_task( Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_sift_extract_task(
Lardon3DAppState *state, const Lardon3DTask *task, Lardon3DAppState *state, const Lardon3DTask *task,
const Lardon3DProjectDbSiftExtractTask *parameters); const Lardon3DProjectDbSiftExtractTask *parameters);

View file

@ -11,10 +11,10 @@
#include <lardon3d/sparse_sfm_incremental.h> #include <lardon3d/sparse_sfm_incremental.h>
enum { enum {
/* v23 is an additive optical-context layer. Historical v22 rows and sparse /* v25 adds only durable features.extract.batch/1 operational state. The
* calibration identities remain untouched; migration never guesses profiles * selected execution remains the immutable ordered image domain; migration
* or assignments for existing Captures. */ * creates no batch association, cursor, Feature Set, or inferred identity. */
LARDON3D_PROJECT_DB_SCHEMA_VERSION = 23, LARDON3D_PROJECT_DB_SCHEMA_VERSION = 25,
LARDON3D_PROJECT_DB_ID_CAPACITY = 65, LARDON3D_PROJECT_DB_ID_CAPACITY = 65,
LARDON3D_PROJECT_DB_KIND_CAPACITY = 65, LARDON3D_PROJECT_DB_KIND_CAPACITY = 65,
LARDON3D_PROJECT_DB_PATH_CAPACITY = 4096, LARDON3D_PROJECT_DB_PATH_CAPACITY = 4096,
@ -145,6 +145,13 @@ typedef struct {
uint64_t image_id; uint64_t image_id;
} Lardon3DProjectDbRawDevelopmentTask; } Lardon3DProjectDbRawDevelopmentTask;
typedef struct {
uint64_t task_id;
/* Selected execution owns the ordered scientific inputs and durable cursor.
* This typed Task row deliberately copies none of those identities. */
uint64_t selected_execution_id;
} Lardon3DProjectDbRawDevelopmentBatchTask;
typedef struct { typedef struct {
uint64_t task_id; uint64_t task_id;
uint32_t group_id; uint32_t group_id;
@ -406,6 +413,19 @@ typedef struct {
unsigned char parameter_fingerprint[32]; unsigned char parameter_fingerprint[32];
} Lardon3DProjectDbFeatureExtractTask; } Lardon3DProjectDbFeatureExtractTask;
typedef struct {
uint64_t task_id;
uint64_t selected_execution_id;
/* Zero-based durable prefix in the immutable selected-execution order. */
uint32_t next_item_index;
char extractor_kind[LARDON3D_PROJECT_DB_KIND_CAPACITY];
uint32_t extractor_version;
uint32_t max_features;
uint32_t pyramid_levels;
uint32_t fast_threshold;
unsigned char parameter_fingerprint[LARDON3D_PROJECT_DB_SHA256_SIZE];
} Lardon3DProjectDbFeatureExtractBatchTask;
typedef struct { typedef struct {
uint64_t task_id; uint64_t task_id;
uint64_t image_id; uint64_t image_id;
@ -785,6 +805,21 @@ Lardon3DProjectDbResult lardon3d_project_db_record_raw_development_task(
Lardon3DProjectDbResult lardon3d_project_db_load_raw_development_task( Lardon3DProjectDbResult lardon3d_project_db_load_raw_development_task(
Lardon3DProjectDb *database, uint64_t task_id, Lardon3DProjectDb *database, uint64_t task_id,
Lardon3DProjectDbRawDevelopmentTask *parameters); Lardon3DProjectDbRawDevelopmentTask *parameters);
/* Atomically record generic raw.develop.batch/1 state and its immutable link
* to one selected execution. Exact retry is idempotent; rebinding either a
* Task or selected execution conflicts. The selected execution cursor remains
* the sole durable item-progress authority. */
Lardon3DProjectDbResult lardon3d_project_db_record_raw_development_batch_task(
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot,
const char *task_kind, uint32_t task_kind_version,
const Lardon3DProjectDbCheckpoint *checkpoint,
const Lardon3DProjectDbRawDevelopmentBatchTask *parameters, int64_t updated_at);
/* Load the caller-owned immutable Task -> selected-execution association.
* Malformed integer storage or a generic kind/version mismatch is corruption;
* no item identity or cursor is inferred from other durable state. */
Lardon3DProjectDbResult lardon3d_project_db_load_raw_development_batch_task(
Lardon3DProjectDb *database, uint64_t task_id,
Lardon3DProjectDbRawDevelopmentBatchTask *parameters);
/* Result publication and next_group_id advance are atomic. result->group_id is /* Result publication and next_group_id advance are atomic. result->group_id is
* the canonical plan ID in 1..N and must equal the task's current one-based * the canonical plan ID in 1..N and must equal the task's current one-based
* cursor; next_group_id must equal result->group_id+1 (N+1 after the last * cursor; next_group_id must equal result->group_id+1 (N+1 after the last
@ -869,6 +904,26 @@ Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_task(
Lardon3DProjectDbResult Lardon3DProjectDbResult
lardon3d_project_db_load_feature_extract_task(Lardon3DProjectDb *database, uint64_t task_id, lardon3d_project_db_load_feature_extract_task(Lardon3DProjectDb *database, uint64_t task_id,
Lardon3DProjectDbFeatureExtractTask *parameters); Lardon3DProjectDbFeatureExtractTask *parameters);
/* Atomically persist generic features.extract.batch/1 state and its typed
* immutable selected-execution/ORB domain plus monotone cursor. A stale
* generic checkpoint may repeat an older cursor but can never move it back. */
Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_batch_task(
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot,
const char *task_kind, uint32_t task_kind_version,
const Lardon3DProjectDbCheckpoint *checkpoint,
const Lardon3DProjectDbFeatureExtractBatchTask *parameters, int64_t updated_at);
/* Load and validate the exact typed batch domain and generic dispatch. The
* selected execution must be representation-complete and the cursor bounded by
* its immutable item count; malformed durable state is CORRUPT. */
Lardon3DProjectDbResult lardon3d_project_db_load_feature_extract_batch_task(
Lardon3DProjectDb *database, uint64_t task_id,
Lardon3DProjectDbFeatureExtractBatchTask *parameters);
/* Advance exactly one ordered item only after its selected image has a READY
* Feature Set for the task's immutable ORB fingerprint. Exact retry after an
* already committed advance converges; no image identity is inferred. */
Lardon3DProjectDbResult lardon3d_project_db_advance_feature_extract_batch_task(
Lardon3DProjectDb *database, uint64_t task_id, uint32_t item_index,
uint32_t next_item_index);
Lardon3DProjectDbResult lardon3d_project_db_record_sift_extract_task( Lardon3DProjectDbResult lardon3d_project_db_record_sift_extract_task(
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot,
const char *task_kind, uint32_t task_kind_version, const char *task_kind, uint32_t task_kind_version,

View file

@ -13,6 +13,8 @@ extern "C" {
#define LARDON3D_RAW_DEVELOPMENT_TASK_KIND "raw.develop" #define LARDON3D_RAW_DEVELOPMENT_TASK_KIND "raw.develop"
enum { LARDON3D_RAW_DEVELOPMENT_TASK_KIND_VERSION = 1 }; enum { LARDON3D_RAW_DEVELOPMENT_TASK_KIND_VERSION = 1 };
#define LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND "raw.develop.batch"
enum { LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION = 1 };
/* Create one durable, initially unqueued S3-B1 Task for the exact existing /* Create one durable, initially unqueued S3-B1 Task for the exact existing
* Capture-owned SOURCE RAW asset. The two IDs are immutable and are never * Capture-owned SOURCE RAW asset. The two IDs are immutable and are never
@ -37,6 +39,25 @@ bool lardon3d_raw_development_task_reconstruct(
const Lardon3DTaskDurableSnapshot *snapshot, void *context, const Lardon3DTaskDurableSnapshot *snapshot, void *context,
Lardon3DTaskKindBinding *binding); Lardon3DTaskKindBinding *binding);
/* Create one durable, initially unqueued parent Task for the exact immutable
* selected execution. The selected execution, not Task identity, owns item
* order, explicit Capture/SOURCE RAW identity, and the durable cursor. Queue
* admission bounds each parallel window; the caller owns the returned Task. */
Lardon3DTask *lardon3d_project_create_raw_development_batch_task(
Lardon3DAppState *state, uint64_t selected_execution_id, uint64_t *task_id);
/* Create and transfer the parent Task to state's existing sole Queue. The
* Queue/Governor reservation covers only one bounded window at a time and is
* released/re-established at each sequence boundary. */
bool lardon3d_project_enqueue_raw_development_batch(
Lardon3DAppState *state, uint64_t selected_execution_id, uint64_t *task_id);
/* Reconstruct only from the validated v24 Task -> selected execution row.
* binding owns its returned context until the runtime invokes its destroyer. */
bool lardon3d_raw_development_batch_task_reconstruct(
const Lardon3DTaskDurableSnapshot *snapshot, void *context,
Lardon3DTaskKindBinding *binding);
#ifdef __cplusplus #ifdef __cplusplus
} }
#endif #endif

View file

@ -26,6 +26,13 @@ extern "C" bool lardon3d_feature_opencv_configure_threads(unsigned int threads)
const char *forced = std::getenv( const char *forced = std::getenv(
"LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS"); "LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS");
if (forced) { if (forced) {
if (std::strcmp(forced, "once") == 0) {
/* The Task's admitted count is deliberately host-derived. Consume the
* one-shot before failing so the rollback can restore the captured
* process-wide value without a test-side guess of that count. */
(void)unsetenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS");
return false;
}
char *end = nullptr; char *end = nullptr;
unsigned long parsed = std::strtoul(forced, &end, 10); unsigned long parsed = std::strtoul(forced, &end, 10);
if (end && *end == '\0' && parsed == threads) { if (end && *end == '\0' && parsed == threads) {

View file

@ -2,6 +2,7 @@
#include <fcntl.h> #include <fcntl.h>
#include <limits.h> #include <limits.h>
#include <openssl/evp.h> #include <openssl/evp.h>
#include <pthread.h>
#include <stdbool.h> #include <stdbool.h>
#include <stdint.h> #include <stdint.h>
#include <stdio.h> #include <stdio.h>
@ -26,6 +27,18 @@ typedef struct {
Lardon3DProjectDbFeatureExtractTask parameters; Lardon3DProjectDbFeatureExtractTask parameters;
} Lardon3DFeatureTaskContext; } Lardon3DFeatureTaskContext;
enum {
FEATURE_BATCH_MAX = 12,
FEATURE_BATCH_CHILD_STACK = 1024 * 1024,
};
typedef struct {
char project_path[PATH_MAX];
Lardon3DProjectDb *database;
Lardon3DResourceGovernor *governor;
Lardon3DProjectDbFeatureExtractBatchTask parameters;
} Lardon3DFeatureBatchTaskContext;
static void destroy_context(void *userdata) { free(userdata); } static void destroy_context(void *userdata) { free(userdata); }
static void runtime_state(const Lardon3DFeatureTaskContext *context, Lardon3DAppState *state) { static void runtime_state(const Lardon3DFeatureTaskContext *context, Lardon3DAppState *state) {
@ -129,6 +142,247 @@ static bool load_validated_source(const Lardon3DFeatureTaskContext *context, cha
return file_hash(path, actual) && memcmp(actual, asset.sha256, 32) == 0; return file_hash(path, actual) && memcmp(actual, asset.sha256, 32) == 0;
} }
static void batch_runtime_state(const Lardon3DFeatureBatchTaskContext *context,
Lardon3DAppState *state) {
lardon3d_app_state_init(state);
state->project_loaded = true;
state->project_db = context->database;
state->resource_governor = context->governor;
(void)snprintf(state->project_path, sizeof(state->project_path), "%s", context->project_path);
}
static bool batch_checkpoint(Lardon3DFeatureBatchTaskContext *context, Lardon3DTask *task,
uint32_t cursor) {
Lardon3DAppState state;
batch_runtime_state(context, &state);
Lardon3DProjectDbFeatureExtractBatchTask parameters = context->parameters;
parameters.task_id = lardon3d_task_id(task);
parameters.next_item_index = cursor;
return lardon3d_project_checkpoint_feature_extract_batch_task(&state, task, &parameters) ==
LARDON3D_PROJECT_TASK_CHECKPOINT_OK;
}
typedef struct {
Lardon3DFeatureBatchTaskContext *context;
uint32_t item_index;
uint64_t image_id;
char source_path[PATH_MAX];
bool already_ready;
Lardon3DFeatureExtractResult result;
Lardon3DExtractedFeatures features;
} FeatureBatchWorker;
static void *extract_batch_item(void *value) {
FeatureBatchWorker *worker = value;
if (worker->already_ready) return NULL;
Lardon3DFeatureExtractorParameters parameters = {
worker->context->parameters.max_features,
worker->context->parameters.pyramid_levels,
worker->context->parameters.fast_threshold};
worker->result = lardon3d_feature_extract_orb(worker->source_path, &parameters,
&worker->features);
return NULL;
}
typedef struct {
FeatureBatchWorker *workers;
size_t count;
size_t first;
size_t stride;
} FeatureBatchParticipant;
static void *extract_batch_participant(void *value) {
FeatureBatchParticipant *participant = value;
for (size_t index = participant->first; index < participant->count;
index += participant->stride)
(void)extract_batch_item(&participant->workers[index]);
return NULL;
}
static bool extract_batch_window(FeatureBatchWorker *workers, size_t count,
unsigned int admitted_threads) {
if (!workers || count == 0 || count > FEATURE_BATCH_MAX || admitted_threads == 0) return false;
size_t participants = count < admitted_threads ? count : admitted_threads;
pthread_t children[FEATURE_BATCH_MAX - 1];
FeatureBatchParticipant work[FEATURE_BATCH_MAX];
pthread_attr_t attributes;
if (pthread_attr_init(&attributes) != 0) return false;
if (pthread_attr_setstacksize(&attributes, FEATURE_BATCH_CHILD_STACK) != 0) {
(void)pthread_attr_destroy(&attributes);
return false;
}
size_t created = 0;
bool started = true;
for (size_t participant = 1; participant < participants; ++participant) {
work[participant] = (FeatureBatchParticipant){workers, count, participant, participants};
/* Participants own only disjoint extraction outputs. SQLite, Feature Store
publication, cursor advancement, and generic checkpoints remain owner-only. */
if (pthread_create(&children[created], &attributes, extract_batch_participant,
&work[participant]) != 0) {
started = false;
break;
}
++created;
}
if (pthread_attr_destroy(&attributes) != 0) started = false;
work[0] = (FeatureBatchParticipant){workers, count, 0, participants};
if (started) (void)extract_batch_participant(&work[0]);
for (size_t index = 0; index < created; ++index)
if (pthread_join(children[index], NULL) != 0) started = false;
return started;
}
static void destroy_batch_outputs(FeatureBatchWorker *workers, size_t count) {
if (!workers) return;
/* A joined window owns every participant output until ordered owner
publication consumes it. Any boundary failure must release the whole
window; the Governor reservation must not outlive abandoned ORB buffers. */
for (size_t index = 0; index < count; ++index)
lardon3d_extracted_features_destroy(&workers[index].features);
}
static bool run_batch_body(Lardon3DTask *task, Lardon3DFeatureBatchTaskContext *context) {
for (;;) {
if (!lardon3d_task_checkpoint(task)) return false;
Lardon3DProjectDbFeatureExtractBatchTask persisted;
Lardon3DProjectDbSelectedExecution execution;
if (lardon3d_project_db_load_feature_extract_batch_task(
context->database, lardon3d_task_id(task), &persisted) != LARDON3D_PROJECT_DB_OK ||
memcmp(&persisted.parameter_fingerprint, &context->parameters.parameter_fingerprint,
32) != 0 ||
lardon3d_project_db_load_selected_execution(context->database,
context->parameters.selected_execution_id, &execution) != LARDON3D_PROJECT_DB_OK)
return lardon3d_task_fail(task, "État durable Feature batch invalide.");
if (persisted.next_item_index == execution.item_count)
return lardon3d_task_set_durable_progress(task, execution.item_count, execution.item_count,
"Extraction Feature batch terminée.") &&
batch_checkpoint(context, task, execution.item_count);
Lardon3DTaskExecutionContract contract;
if (!lardon3d_task_execution_contract(task, &contract) || contract.cpu_threads == 0 ||
contract.batch_size == 0 || contract.batch_size > FEATURE_BATCH_MAX)
return lardon3d_task_fail(task, "Contrat Governor Feature batch invalide.");
size_t count = execution.item_count - persisted.next_item_index;
if (count > contract.batch_size) count = contract.batch_size;
FeatureBatchWorker workers[FEATURE_BATCH_MAX] = {{0}};
for (size_t offset = 0; offset < count; ++offset) {
FeatureBatchWorker *worker = &workers[offset];
Lardon3DProjectDbSelectedExecutionItem item;
if (lardon3d_project_db_load_selected_execution_item(
context->database, context->parameters.selected_execution_id,
persisted.next_item_index + (uint32_t)offset, &item) != LARDON3D_PROJECT_DB_OK ||
!item.has_image)
return lardon3d_task_fail(task, "Item sélectionné Feature batch invalide.");
worker->context = context;
worker->item_index = item.item_index;
worker->image_id = item.image_id;
Lardon3DFeatureTaskContext single = {0};
single.database = context->database;
single.parameters.image_id = item.image_id;
(void)snprintf(single.project_path, sizeof(single.project_path), "%s", context->project_path);
Lardon3DProjectDbResult found = find_valid_feature_set(
&single, context->parameters.parameter_fingerprint);
worker->already_ready = found == LARDON3D_PROJECT_DB_OK;
if (found == LARDON3D_PROJECT_DB_CORRUPT ||
(found != LARDON3D_PROJECT_DB_OK && found != LARDON3D_PROJECT_DB_NOT_FOUND) ||
(!worker->already_ready && !load_validated_source(&single, worker->source_path)))
return lardon3d_task_fail(task, "Source ou Feature Set batch invalide.");
}
struct timespec begin = {0}, end = {0};
bool timing_known = clock_gettime(CLOCK_MONOTONIC, &begin) == 0;
if (!extract_batch_window(workers, count, contract.cpu_threads)) {
destroy_batch_outputs(workers, count);
return lardon3d_task_fail(task, "Participants Feature batch impossibles.");
}
if (!lardon3d_task_checkpoint(task)) {
destroy_batch_outputs(workers, count);
return false;
}
size_t durable_items = 0;
Lardon3DAppState state;
batch_runtime_state(context, &state);
Lardon3DFeatureExtractorParameters parameters = {
context->parameters.max_features, context->parameters.pyramid_levels,
context->parameters.fast_threshold};
for (size_t offset = 0; offset < count; ++offset) {
FeatureBatchWorker *worker = &workers[offset];
if (!worker->already_ready) {
if (worker->result != LARDON3D_FEATURE_EXTRACT_OK) {
destroy_batch_outputs(workers, count);
return lardon3d_task_fail(task, "Extraction ORB batch impossible.");
}
Lardon3DProjectDbFeatureSet set;
Lardon3DFeatureStoreResult published = lardon3d_feature_store_publish(
&state, worker->image_id, lardon3d_task_id(task), &parameters,
&worker->features, &set);
lardon3d_extracted_features_destroy(&worker->features);
if (published != LARDON3D_FEATURE_STORE_OK &&
published != LARDON3D_FEATURE_STORE_ALREADY_PRESENT) {
destroy_batch_outputs(workers, count);
return lardon3d_task_fail(task, "Publication Feature batch impossible.");
}
if (published == LARDON3D_FEATURE_STORE_OK) ++durable_items;
}
if (lardon3d_project_db_advance_feature_extract_batch_task(
context->database, lardon3d_task_id(task), worker->item_index,
worker->item_index + 1u) != LARDON3D_PROJECT_DB_OK ||
!lardon3d_task_set_durable_progress(task, worker->item_index + 1u,
execution.item_count,
"Feature batch publié.") ||
!batch_checkpoint(context, task, worker->item_index + 1u)) {
destroy_batch_outputs(workers, count);
return lardon3d_task_fail(task, "Curseur Feature batch impossible.");
}
}
timing_known = timing_known && clock_gettime(CLOCK_MONOTONIC, &end) == 0;
if (timing_known)
(void)lardon3d_task_internal_record_sequence(task, elapsed_ns(begin, end), durable_items);
if (persisted.next_item_index + count == execution.item_count) return true;
Lardon3DResourceReservation *reservation = NULL;
if (!lardon3d_task_sequence_break(task, context->governor, &reservation, &contract))
return false;
}
}
static bool run_batch(Lardon3DTask *task, void *userdata) {
Lardon3DFeatureBatchTaskContext *context = userdata;
Lardon3DTaskExecutionContract contract;
if (!lardon3d_task_execution_contract(task, &contract) || contract.cpu_threads == 0)
return lardon3d_task_fail(task, "Contrat CPU Feature batch invalide.");
Lardon3DOpenCvTaskThreadControl control = {
.previous = lardon3d_feature_opencv_thread_count(),
.restore_required = true,
};
/* Cross-image participants are the admitted CPU dimension. OpenCV stays at
one thread per extraction to prevent nested oversubscription; Queue owns
the sole process-wide guard. Configuration may mutate before verification
fails, so restoration duty exists before configure and covers every exit. */
if (!lardon3d_feature_opencv_configure_threads(1)) {
(void)lardon3d_opencv_task_threads_end(&control);
return lardon3d_task_fail(task, "Configuration OpenCV Feature batch impossible.");
}
bool result = run_batch_body(task, context);
if (!lardon3d_opencv_task_threads_end(&control))
return lardon3d_task_fail(task, "Restauration OpenCV Feature batch impossible.");
return result;
}
#ifdef LARDON3D_FEATURE_TASK_TESTING
bool lardon3d_feature_extract_batch_test_run(Lardon3DTask *task) {
Lardon3DFeatureBatchTaskContext context = {0};
return run_batch(task, &context);
}
#endif
static void finished_batch(const Lardon3DTask *task, void *userdata) {
Lardon3DFeatureBatchTaskContext *context = userdata;
Lardon3DProjectDbFeatureExtractBatchTask persisted;
if (lardon3d_project_db_load_feature_extract_batch_task(
context->database, lardon3d_task_id(task), &persisted) == LARDON3D_PROJECT_DB_OK)
(void)batch_checkpoint(context, (Lardon3DTask *)task, persisted.next_item_index);
}
static bool run_body(Lardon3DTask *task, void *userdata, size_t *durable_items) { static bool run_body(Lardon3DTask *task, void *userdata, size_t *durable_items) {
Lardon3DFeatureTaskContext *context = userdata; Lardon3DFeatureTaskContext *context = userdata;
*durable_items = 0; *durable_items = 0;
@ -376,3 +630,121 @@ bool lardon3d_project_enqueue_feature_extract(Lardon3DAppState *state, uint64_t
} }
return true; return true;
} }
static Lardon3DFeatureBatchTaskContext *make_batch_context(
const Lardon3DTaskReconstructionContext *runtime,
const Lardon3DProjectDbFeatureExtractBatchTask *parameters) {
if (!runtime || !runtime->project_path || !runtime->project_db ||
!runtime->resource_governor || !parameters) return NULL;
Lardon3DFeatureBatchTaskContext *context = calloc(1, sizeof(*context));
if (!context) return NULL;
int n = snprintf(context->project_path, sizeof(context->project_path), "%s",
runtime->project_path);
if (n <= 0 || (size_t)n >= sizeof(context->project_path)) {
free(context);
return NULL;
}
context->database = runtime->project_db;
context->governor = runtime->resource_governor;
context->parameters = *parameters;
return context;
}
bool lardon3d_feature_extract_batch_reconstruct(
const Lardon3DTaskDurableSnapshot *snapshot, void *userdata,
Lardon3DTaskKindBinding *binding) {
Lardon3DTaskReconstructionContext *runtime = userdata;
if (!snapshot || !runtime || !binding) return false;
Lardon3DProjectDbFeatureExtractBatchTask parameters;
if (lardon3d_project_db_load_feature_extract_batch_task(
runtime->project_db, snapshot->id, &parameters) != LARDON3D_PROJECT_DB_OK)
return false;
unsigned char expected[32];
lardon3d_feature_extractor_parameter_fingerprint(
&(Lardon3DFeatureExtractorParameters){parameters.max_features,
parameters.pyramid_levels,
parameters.fast_threshold}, expected);
if (memcmp(expected, parameters.parameter_fingerprint, 32) != 0) return false;
Lardon3DFeatureBatchTaskContext *context = make_batch_context(runtime, &parameters);
if (!context) return false;
*binding = (Lardon3DTaskKindBinding){.callback = run_batch,
.userdata = context,
.userdata_destroy = destroy_context,
.finished_callback = finished_batch,
.finished_userdata = context};
return true;
}
Lardon3DTask *lardon3d_project_create_feature_extract_batch_task(
Lardon3DAppState *state, uint64_t selected_execution_id,
const Lardon3DFeatureExtractorParameters *parameters, uint64_t *task_id) {
if (task_id) *task_id = 0;
if (!state || !state->project_loaded || !state->project_db ||
!state->resource_governor || !parameters || !task_id ||
!lardon3d_feature_extractor_parameters_valid(parameters) || selected_execution_id == 0)
return NULL;
Lardon3DProjectDbSelectedExecution execution;
if (lardon3d_project_db_load_selected_execution(state->project_db, selected_execution_id,
&execution) != LARDON3D_PROJECT_DB_OK ||
execution.stage < LARDON3D_SELECTED_EXECUTION_CALIBRATION || execution.item_count == 0)
return NULL;
uint64_t id = 0;
if (lardon3d_project_db_allocate_task_id(state->project_db, &id) != LARDON3D_PROJECT_DB_OK)
return NULL;
Lardon3DProjectDbFeatureExtractBatchTask durable = {
.task_id = id,
.selected_execution_id = selected_execution_id,
.next_item_index = 0,
.extractor_version = LARDON3D_FEATURE_EXTRACTOR_VERSION,
.max_features = parameters->max_features,
.pyramid_levels = parameters->pyramid_levels,
.fast_threshold = parameters->fast_threshold};
(void)snprintf(durable.extractor_kind, sizeof(durable.extractor_kind), "%s",
LARDON3D_FEATURE_EXTRACTOR_KIND);
lardon3d_feature_extractor_parameter_fingerprint(parameters,
durable.parameter_fingerprint);
Lardon3DTaskReconstructionContext runtime = {
state->project_path, state->project_db, state->resource_governor,
state->orb_vulkan_backend};
Lardon3DFeatureBatchTaskContext *context = make_batch_context(&runtime, &durable);
if (!context) return NULL;
/* One admitted participant owns decode, ORB pyramid/keypoints/descriptors,
and publication staging bounded by the existing conservative 512 MiB/item.
The 1..12 ceiling is operational benchmark capacity, not selected-domain
cardinality; Governor feedback records each joined window's useful items. */
Lardon3DResourceEstimate estimate = {
.memory_fixed_bytes = 64ULL * 1024 * 1024,
.memory_bytes_per_item = 512ULL * 1024 * 1024,
.minimum_batch_size = 1,
.maximum_batch_size = FEATURE_BATCH_MAX,
.desired_cpu_threads = FEATURE_BATCH_MAX,
.desired_io_slots = 1,
.task_class = LARDON3D_RESOURCE_TASK_CPU};
Lardon3DTask *task = lardon3d_task_create_typed(
"Extraction Feature sélectionnée", &estimate,
LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND,
LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION, run_batch, context,
destroy_context);
if (!task || !lardon3d_task_assign_id(task, id) ||
!lardon3d_task_set_finished_callback(task, finished_batch, context) ||
!batch_checkpoint(context, task, 0)) {
lardon3d_task_destroy(task);
return NULL;
}
*task_id = id;
return task;
}
bool lardon3d_project_enqueue_feature_extract_batch(
Lardon3DAppState *state, uint64_t selected_execution_id,
const Lardon3DFeatureExtractorParameters *parameters, uint64_t *task_id) {
if (!state || !state->task_queue) return false;
Lardon3DTask *task = lardon3d_project_create_feature_extract_batch_task(
state, selected_execution_id, parameters, task_id);
if (!task) return false;
if (!lardon3d_task_queue_add(state->task_queue, task, NULL)) {
lardon3d_task_destroy(task);
return false;
}
return true;
}

View file

@ -717,7 +717,9 @@ checkpoint_task_internal(Lardon3DAppState *state, const Lardon3DTask *task,
const Lardon3DProjectDbGeometricVerifierTask *geometric_parameters, const Lardon3DProjectDbGeometricVerifierTask *geometric_parameters,
const Lardon3DProjectDbAcquisitionCampaignTask *campaign_parameters, const Lardon3DProjectDbAcquisitionCampaignTask *campaign_parameters,
const Lardon3DProjectDbPhotoQualityTask *quality_parameters, const Lardon3DProjectDbPhotoQualityTask *quality_parameters,
const Lardon3DProjectDbRawDevelopmentTask *raw_parameters) { const Lardon3DProjectDbRawDevelopmentTask *raw_parameters,
const Lardon3DProjectDbRawDevelopmentBatchTask *raw_batch_parameters,
const Lardon3DProjectDbFeatureExtractBatchTask *feature_batch_parameters) {
if (!state || !state->project_loaded || !state->project_db) { if (!state || !state->project_loaded || !state->project_db) {
return LARDON3D_PROJECT_TASK_CHECKPOINT_NO_PROJECT; return LARDON3D_PROJECT_TASK_CHECKPOINT_NO_PROJECT;
} }
@ -813,6 +815,14 @@ checkpoint_task_internal(Lardon3DAppState *state, const Lardon3DTask *task,
? lardon3d_project_db_record_raw_development_task( ? lardon3d_project_db_record_raw_development_task(
state->project_db, &snapshot, task_kind, task_kind_version, &checkpoint, state->project_db, &snapshot, task_kind, task_kind_version, &checkpoint,
raw_parameters, now.tv_sec) raw_parameters, now.tv_sec)
: raw_batch_parameters
? lardon3d_project_db_record_raw_development_batch_task(
state->project_db, &snapshot, task_kind, task_kind_version, &checkpoint,
raw_batch_parameters, now.tv_sec)
: feature_batch_parameters
? lardon3d_project_db_record_feature_extract_batch_task(
state->project_db, &snapshot, task_kind, task_kind_version, &checkpoint,
feature_batch_parameters, now.tv_sec)
: lardon3d_project_db_record_task(state->project_db, &snapshot, task_kind, : lardon3d_project_db_record_task(state->project_db, &snapshot, task_kind,
task_kind_version, &checkpoint, now.tv_sec); task_kind_version, &checkpoint, now.tv_sec);
if (recorded == LARDON3D_PROJECT_DB_BUSY) { if (recorded == LARDON3D_PROJECT_DB_BUSY) {
@ -841,7 +851,7 @@ checkpoint_task_internal(Lardon3DAppState *state, const Lardon3DTask *task,
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_task(Lardon3DAppState *state, Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_task(Lardon3DAppState *state,
const Lardon3DTask *task) { const Lardon3DTask *task) {
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
NULL, NULL); NULL, NULL, NULL, NULL);
} }
Lardon3DProjectTaskCheckpointResult Lardon3DProjectTaskCheckpointResult
@ -851,7 +861,8 @@ lardon3d_project_checkpoint_image_import_task(Lardon3DAppState *state, const Lar
return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
} }
return checkpoint_task_internal( return checkpoint_task_internal(
state, task, source_path, scanset_id, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL); state, task, source_path, scanset_id, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
NULL, NULL);
} }
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_acquisition_campaign_task( Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_acquisition_campaign_task(
@ -859,7 +870,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_acquisition_camp
const Lardon3DProjectDbAcquisitionCampaignTask *parameters) { const Lardon3DProjectDbAcquisitionCampaignTask *parameters) {
if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL,
parameters, NULL, NULL); parameters, NULL, NULL, NULL, NULL);
} }
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_photo_quality_task( Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_photo_quality_task(
@ -867,7 +878,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_photo_quality_ta
const Lardon3DProjectDbPhotoQualityTask *parameters) { const Lardon3DProjectDbPhotoQualityTask *parameters) {
if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
parameters, NULL); parameters, NULL, NULL, NULL);
} }
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_raw_development_task( Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_raw_development_task(
@ -875,7 +886,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_raw_development_
const Lardon3DProjectDbRawDevelopmentTask *parameters) { const Lardon3DProjectDbRawDevelopmentTask *parameters) {
if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
NULL, parameters); NULL, parameters, NULL, NULL);
} }
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_task( Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_task(
@ -885,7 +896,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_
return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
} }
return checkpoint_task_internal(state, task, NULL, 0, parameters, NULL, NULL, NULL, NULL, NULL, return checkpoint_task_internal(state, task, NULL, 0, parameters, NULL, NULL, NULL, NULL, NULL,
NULL, NULL, NULL); NULL, NULL, NULL, NULL, NULL);
} }
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_sift_extract_task( Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_sift_extract_task(
@ -893,7 +904,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_sift_extract_tas
const Lardon3DProjectDbSiftExtractTask *parameters) { const Lardon3DProjectDbSiftExtractTask *parameters) {
if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
return checkpoint_task_internal(state, task, NULL, 0, NULL, parameters, NULL, NULL, NULL, NULL, return checkpoint_task_internal(state, task, NULL, 0, NULL, parameters, NULL, NULL, NULL, NULL,
NULL, NULL, NULL); NULL, NULL, NULL, NULL, NULL);
} }
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_visual_index_update_task( Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_visual_index_update_task(
@ -903,7 +914,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_visual_index_upd
return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
} }
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, parameters, NULL, NULL, NULL, return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, parameters, NULL, NULL, NULL,
NULL, NULL, NULL); NULL, NULL, NULL, NULL, NULL);
} }
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_candidate_pair_generate_task( Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_candidate_pair_generate_task(
@ -913,7 +924,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_candidate_pair_g
return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
} }
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, parameters, NULL, NULL, return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, parameters, NULL, NULL,
NULL, NULL, NULL); NULL, NULL, NULL, NULL, NULL);
} }
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_matcher_task( Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_matcher_task(
@ -923,7 +934,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_matcher_task(
return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
} }
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, parameters, NULL, return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, parameters, NULL,
NULL, NULL, NULL); NULL, NULL, NULL, NULL, NULL);
} }
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_geometric_verifier_task( Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_geometric_verifier_task(
@ -933,7 +944,23 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_geometric_verifi
return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK; return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
} }
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL,
parameters, NULL, NULL, NULL); parameters, NULL, NULL, NULL, NULL, NULL);
}
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_raw_development_batch_task(
Lardon3DAppState *state, const Lardon3DTask *task,
const Lardon3DProjectDbRawDevelopmentBatchTask *parameters) {
if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
NULL, NULL, parameters, NULL);
}
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_batch_task(
Lardon3DAppState *state, const Lardon3DTask *task,
const Lardon3DProjectDbFeatureExtractBatchTask *parameters) {
if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
NULL, NULL, NULL, parameters);
} }
static bool coherent_recovery(const Lardon3DProjectDbTask *database_task, static bool coherent_recovery(const Lardon3DProjectDbTask *database_task,

View file

@ -12,6 +12,7 @@
#include <time.h> #include <time.h>
#include <lardon3d/acquisition_campaign_task.h> #include <lardon3d/acquisition_campaign_task.h>
#include <lardon3d/feature_task.h>
#include <lardon3d/optical_profiles.h> #include <lardon3d/optical_profiles.h>
#include <lardon3d/photo_quality_task.h> #include <lardon3d/photo_quality_task.h>
#include <lardon3d/project_db.h> #include <lardon3d/project_db.h>
@ -851,6 +852,36 @@ static const char schema_optical_calibrations_v23[] =
"FOREIGN KEY(calibration_profile_id,optical_configuration_id) REFERENCES " "FOREIGN KEY(calibration_profile_id,optical_configuration_id) REFERENCES "
"optical_calibration_profiles(calibration_profile_id,optical_configuration_id));"; "optical_calibration_profiles(calibration_profile_id,optical_configuration_id));";
/* PERSISTENCE CONTRACT v24: this additive row stores only the operational
Task -> selected-execution association. Selected execution items/cursor stay
authoritative, and migration creates no rows or inferred identities. */
static const char schema_raw_development_batch_v24[] =
"CREATE TABLE raw_development_batch_tasks("
"task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE CASCADE,"
"selected_execution_id INTEGER NOT NULL UNIQUE REFERENCES "
"selected_executions(execution_id));";
/* PERSISTENCE CONTRACT v25: this operational row anchors one owner to the
immutable selected-execution order and exact ORB parameter domain. Its
monotone cursor is a durable prefix only; migration creates no rows. */
static const char schema_feature_extract_batch_v25[] =
"CREATE TABLE feature_extract_batch_tasks("
"task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE CASCADE,"
"selected_execution_id INTEGER NOT NULL UNIQUE REFERENCES selected_executions(execution_id),"
"next_item_index INTEGER NOT NULL CHECK(typeof(next_item_index)='integer' AND "
"next_item_index>=0 AND next_item_index<=4096),"
"extractor_kind TEXT NOT NULL CHECK(typeof(extractor_kind)='text' AND extractor_kind='orb'),"
"extractor_version INTEGER NOT NULL CHECK(typeof(extractor_version)='integer' AND "
"extractor_version=1),"
"max_features INTEGER NOT NULL CHECK(typeof(max_features)='integer' AND max_features>0 AND "
"max_features<=8192),"
"pyramid_levels INTEGER NOT NULL CHECK(typeof(pyramid_levels)='integer' AND "
"pyramid_levels>0 AND pyramid_levels<=16),"
"fast_threshold INTEGER NOT NULL CHECK(typeof(fast_threshold)='integer' AND "
"fast_threshold>0 AND fast_threshold<=255),"
"parameter_fingerprint BLOB NOT NULL CHECK(typeof(parameter_fingerprint)='blob' AND "
"length(parameter_fingerprint)=32));";
static void copy_error(char destination[LARDON3D_PROJECT_DB_ERROR_CAPACITY], const char *text) { static void copy_error(char destination[LARDON3D_PROJECT_DB_ERROR_CAPACITY], const char *text) {
if (destination) { if (destination) {
(void)snprintf(destination, LARDON3D_PROJECT_DB_ERROR_CAPACITY, "%s", text ? text : ""); (void)snprintf(destination, LARDON3D_PROJECT_DB_ERROR_CAPACITY, "%s", text ? text : "");
@ -943,7 +974,8 @@ static Lardon3DProjectDbResult migrate(Lardon3DProjectDb *database, unsigned int
from_version != 8 && from_version != 9 && from_version != 10 && from_version != 11 && from_version != 8 && from_version != 9 && from_version != 10 && from_version != 11 &&
from_version != 12 && from_version != 13 && from_version != 14 && from_version != 12 && from_version != 13 && from_version != 14 &&
from_version != 15 && from_version != 16 && from_version != 17 && from_version != 18 && from_version != 15 && from_version != 16 && from_version != 17 && from_version != 18 &&
from_version != 19 && from_version != 20 && from_version != 21 && from_version != 22) { from_version != 19 && from_version != 20 && from_version != 21 && from_version != 22 &&
from_version != 23 && from_version != 24) {
return LARDON3D_PROJECT_DB_CORRUPT; return LARDON3D_PROJECT_DB_CORRUPT;
} }
Lardon3DProjectDbResult result = execute(database, "BEGIN IMMEDIATE", "begin migration"); Lardon3DProjectDbResult result = execute(database, "BEGIN IMMEDIATE", "begin migration");
@ -1414,6 +1446,46 @@ static Lardon3DProjectDbResult migrate(Lardon3DProjectDb *database, unsigned int
result = LARDON3D_PROJECT_DB_CORRUPT; result = LARDON3D_PROJECT_DB_CORRUPT;
} }
} }
if (result == LARDON3D_PROJECT_DB_OK && from_version < 24) {
/* v23 contains no evidence that a batch Task existed, so migration is
DDL-only and cannot synthesize selected execution associations. */
result = execute(database, schema_raw_development_batch_v24,
"migrate RAW development batch v23 to v24");
#ifdef LARDON3D_PROJECT_DB_TESTING
if (result == LARDON3D_PROJECT_DB_OK &&
getenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V24")) {
result = execute(database, "INSERT INTO missing_v24_test_table VALUES(1)",
"forced migration v24 failure");
}
#endif
if (result == LARDON3D_PROJECT_DB_OK) {
result = execute(database,
"UPDATE metadata SET value=24 WHERE key='schema_version' AND value=23",
"finish schema v24 migration");
if (result == LARDON3D_PROJECT_DB_OK &&
sqlite3_changes(database->connection) != 1)
result = LARDON3D_PROJECT_DB_CORRUPT;
}
}
if (result == LARDON3D_PROJECT_DB_OK && from_version < 25) {
/* No v24 row identifies a Feature batch or cursor. Keep this DDL-only so
historical single-image Tasks remain exactly as persisted. */
result = execute(database, schema_feature_extract_batch_v25,
"migrate feature extraction batch v24 to v25");
#ifdef LARDON3D_PROJECT_DB_TESTING
if (result == LARDON3D_PROJECT_DB_OK &&
getenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V25"))
result = execute(database, "INSERT INTO missing_v25_test_table VALUES(1)",
"forced migration v25 failure");
#endif
if (result == LARDON3D_PROJECT_DB_OK) {
result = execute(database,
"UPDATE metadata SET value=25 WHERE key='schema_version' AND value=24",
"finish schema v25 migration");
if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1)
result = LARDON3D_PROJECT_DB_CORRUPT;
}
}
if (result == LARDON3D_PROJECT_DB_OK) { if (result == LARDON3D_PROJECT_DB_OK) {
result = execute(database, "COMMIT", "commit migration"); result = execute(database, "COMMIT", "commit migration");
} }
@ -1550,6 +1622,8 @@ Lardon3DProjectDbResult lardon3d_project_db_open(const char *path, Lardon3DProje
"capture_assets", "capture_assets",
"capture_source_assets", "capture_source_assets",
"raw_development_tasks", "raw_development_tasks",
"raw_development_batch_tasks",
"feature_extract_batch_tasks",
"capture_selections", "capture_selections",
"asset_derivations", "asset_derivations",
"acquisition_campaign_tasks", "acquisition_campaign_tasks",
@ -1821,6 +1895,8 @@ record_task_internal(Lardon3DProjectDb *database, const Lardon3DTaskDurableSnaps
const Lardon3DProjectDbAcquisitionCampaignTask *campaign, const Lardon3DProjectDbAcquisitionCampaignTask *campaign,
const Lardon3DProjectDbPhotoQualityTask *photo_quality, const Lardon3DProjectDbPhotoQualityTask *photo_quality,
const Lardon3DProjectDbRawDevelopmentTask *raw_development, const Lardon3DProjectDbRawDevelopmentTask *raw_development,
const Lardon3DProjectDbRawDevelopmentBatchTask *raw_batch,
const Lardon3DProjectDbFeatureExtractBatchTask *feature_batch,
int64_t updated_at) { int64_t updated_at) {
bool typed = task_kind != NULL; bool typed = task_kind != NULL;
if (!database || !valid_durable_task(snapshot, updated_at) || if (!database || !valid_durable_task(snapshot, updated_at) ||
@ -1924,6 +2000,21 @@ record_task_internal(Lardon3DProjectDb *database, const Lardon3DTaskDurableSnaps
(raw_development->has_image || raw_development->image_id != 0)) || (raw_development->has_image || raw_development->image_id != 0)) ||
(raw_development->phase == LARDON3D_RAW_DEVELOPMENT_TASK_PUBLISHED && (raw_development->phase == LARDON3D_RAW_DEVELOPMENT_TASK_PUBLISHED &&
(!raw_development->has_image || !valid_task_id(raw_development->image_id))))) || (!raw_development->has_image || !valid_task_id(raw_development->image_id))))) ||
(raw_batch &&
(!task_kind || strcmp(task_kind, LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND) != 0 ||
task_kind_version != LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION ||
!valid_task_id(raw_batch->task_id) || raw_batch->task_id != snapshot->id ||
!valid_task_id(raw_batch->selected_execution_id))) ||
(feature_batch &&
(!task_kind || strcmp(task_kind, LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND) != 0 ||
task_kind_version != LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION ||
!valid_task_id(feature_batch->task_id) || feature_batch->task_id != snapshot->id ||
!valid_task_id(feature_batch->selected_execution_id) ||
strcmp(feature_batch->extractor_kind, LARDON3D_FEATURE_EXTRACTOR_KIND) != 0 ||
feature_batch->extractor_version != LARDON3D_FEATURE_EXTRACTOR_VERSION ||
feature_batch->max_features == 0 || feature_batch->max_features > 8192 ||
feature_batch->pyramid_levels == 0 || feature_batch->pyramid_levels > 16 ||
feature_batch->fast_threshold == 0 || feature_batch->fast_threshold > 255)) ||
(checkpoint && !valid_checkpoint(checkpoint))) { (checkpoint && !valid_checkpoint(checkpoint))) {
return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
} }
@ -2064,6 +2155,64 @@ record_task_internal(Lardon3DProjectDb *database, const Lardon3DTaskDurableSnaps
} }
} }
} }
if (result == LARDON3D_PROJECT_DB_OK && raw_batch) {
/* The selected execution owns every scientific item and its cursor. This
immutable row only makes the parent operational Task reconstructible. */
result = prepare(database,
"INSERT INTO raw_development_batch_tasks(task_id,selected_execution_id) "
"VALUES(?1,?2) ON CONFLICT(task_id) DO UPDATE SET "
"selected_execution_id=excluded.selected_execution_id WHERE "
"raw_development_batch_tasks.selected_execution_id=excluded.selected_execution_id",
&statement);
if (result == LARDON3D_PROJECT_DB_OK) {
sqlite3_bind_int64(statement, 1, (sqlite3_int64)raw_batch->task_id);
sqlite3_bind_int64(statement, 2,
(sqlite3_int64)raw_batch->selected_execution_id);
result = step_done(database, statement, "upsert RAW development batch task");
if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1) {
copy_error(database->error, "Exécution sélectionnée RAW batch immuable.");
result = LARDON3D_PROJECT_DB_CONSTRAINT;
}
}
}
if (result == LARDON3D_PROJECT_DB_OK && feature_batch) {
/* Immutable domain fields never change. Only an equal-or-newer cursor may
be repeated by a generic checkpoint that lagged an item publication. */
result = prepare(database,
"INSERT INTO feature_extract_batch_tasks(task_id,selected_execution_id,"
"next_item_index,extractor_kind,extractor_version,max_features,pyramid_levels,"
"fast_threshold,parameter_fingerprint) SELECT ?1,?2,?3,?4,?5,?6,?7,?8,?9 "
"FROM selected_executions e WHERE e.execution_id=?2 AND e.item_count>0 AND "
"?3<=e.item_count AND e.stage>=?10 "
"ON CONFLICT(task_id) DO UPDATE SET next_item_index=MAX("
"feature_extract_batch_tasks.next_item_index,excluded.next_item_index) WHERE "
"feature_extract_batch_tasks.selected_execution_id=excluded.selected_execution_id AND "
"feature_extract_batch_tasks.extractor_kind=excluded.extractor_kind AND "
"feature_extract_batch_tasks.extractor_version=excluded.extractor_version AND "
"feature_extract_batch_tasks.max_features=excluded.max_features AND "
"feature_extract_batch_tasks.pyramid_levels=excluded.pyramid_levels AND "
"feature_extract_batch_tasks.fast_threshold=excluded.fast_threshold AND "
"feature_extract_batch_tasks.parameter_fingerprint=excluded.parameter_fingerprint AND "
"excluded.next_item_index<=(SELECT item_count FROM selected_executions WHERE "
"execution_id=excluded.selected_execution_id)", &statement);
if (result == LARDON3D_PROJECT_DB_OK) {
sqlite3_bind_int64(statement, 1, (sqlite3_int64)feature_batch->task_id);
sqlite3_bind_int64(statement, 2, (sqlite3_int64)feature_batch->selected_execution_id);
sqlite3_bind_int64(statement, 3, feature_batch->next_item_index);
sqlite3_bind_text(statement, 4, feature_batch->extractor_kind, -1, SQLITE_TRANSIENT);
sqlite3_bind_int64(statement, 5, feature_batch->extractor_version);
sqlite3_bind_int64(statement, 6, feature_batch->max_features);
sqlite3_bind_int64(statement, 7, feature_batch->pyramid_levels);
sqlite3_bind_int64(statement, 8, feature_batch->fast_threshold);
sqlite3_bind_blob(statement, 9, feature_batch->parameter_fingerprint, 32, SQLITE_TRANSIENT);
sqlite3_bind_int64(statement, 10, LARDON3D_SELECTED_EXECUTION_CALIBRATION);
result = step_done(database, statement, "upsert feature extraction batch task");
if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1) {
copy_error(database->error, "Domaine ou curseur Feature batch incohérent.");
result = LARDON3D_PROJECT_DB_CONSTRAINT;
}
}
}
if (result == LARDON3D_PROJECT_DB_OK) { if (result == LARDON3D_PROJECT_DB_OK) {
result = result =
prepare(database, prepare(database,
@ -2398,7 +2547,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_task(
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind, Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind,
uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint, int64_t updated_at) { uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint, int64_t updated_at) {
return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0, return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0,
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, updated_at); NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
NULL, updated_at);
} }
Lardon3DProjectDbResult lardon3d_project_db_record_image_import_task( Lardon3DProjectDbResult lardon3d_project_db_record_image_import_task(
@ -2410,7 +2560,7 @@ Lardon3DProjectDbResult lardon3d_project_db_record_image_import_task(
} }
return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint,
source_path, scanset_id, NULL, NULL, NULL, NULL, NULL, NULL, source_path, scanset_id, NULL, NULL, NULL, NULL, NULL, NULL,
NULL, NULL, NULL, NULL, updated_at); NULL, NULL, NULL, NULL, NULL, NULL, updated_at);
} }
Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_task( Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_task(
@ -2421,8 +2571,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_task(
return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
} }
return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0, return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0,
parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
updated_at); NULL, updated_at);
} }
Lardon3DProjectDbResult lardon3d_project_db_record_sift_extract_task( Lardon3DProjectDbResult lardon3d_project_db_record_sift_extract_task(
@ -2432,8 +2582,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_sift_extract_task(
const Lardon3DProjectDbSiftExtractTask *parameters, int64_t updated_at) { const Lardon3DProjectDbSiftExtractTask *parameters, int64_t updated_at) {
if (!parameters) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; if (!parameters) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0, return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0,
NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
updated_at); NULL, updated_at);
} }
static bool read_task(sqlite3_stmt *statement, Lardon3DProjectDbTask *task) { static bool read_task(sqlite3_stmt *statement, Lardon3DProjectDbTask *task) {
@ -2680,7 +2830,7 @@ Lardon3DProjectDbResult lardon3d_project_db_record_acquisition_campaign_task(
never observes a valid campaign Task without its immutable request blob. */ never observes a valid campaign Task without its immutable request blob. */
return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint,
NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, parameters, NULL, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, parameters, NULL,
NULL, updated_at); NULL, NULL, NULL, updated_at);
} }
static Lardon3DProjectDbResult validate_acquisition_campaign_prefix_locked( static Lardon3DProjectDbResult validate_acquisition_campaign_prefix_locked(
@ -3397,8 +3547,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_photo_quality_task(
/* Generic runtime state and typed immutable request share the transaction; /* Generic runtime state and typed immutable request share the transaction;
recovery cannot observe one without the other. */ recovery cannot observe one without the other. */
return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0, return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0,
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, parameters, NULL, NULL,
updated_at); NULL, updated_at);
} }
Lardon3DProjectDbResult lardon3d_project_db_load_photo_quality_task( Lardon3DProjectDbResult lardon3d_project_db_load_photo_quality_task(
@ -3470,8 +3620,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_raw_development_task(
const Lardon3DProjectDbRawDevelopmentTask *parameters, int64_t updated_at) { const Lardon3DProjectDbRawDevelopmentTask *parameters, int64_t updated_at) {
if (!parameters) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; if (!parameters) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0, return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0,
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, parameters, NULL,
updated_at); NULL, updated_at);
} }
Lardon3DProjectDbResult lardon3d_project_db_load_raw_development_task( Lardon3DProjectDbResult lardon3d_project_db_load_raw_development_task(
@ -3537,6 +3687,61 @@ Lardon3DProjectDbResult lardon3d_project_db_load_raw_development_task(
return result; return result;
} }
Lardon3DProjectDbResult lardon3d_project_db_record_raw_development_batch_task(
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot,
const char *task_kind, uint32_t task_kind_version,
const Lardon3DProjectDbCheckpoint *checkpoint,
const Lardon3DProjectDbRawDevelopmentBatchTask *parameters, int64_t updated_at) {
if (!parameters) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
return record_task_internal(database, snapshot, task_kind, task_kind_version,
checkpoint, NULL, 0, NULL, NULL, NULL, NULL, NULL,
NULL, NULL, NULL, NULL, NULL, parameters, NULL, updated_at);
}
Lardon3DProjectDbResult lardon3d_project_db_load_raw_development_batch_task(
Lardon3DProjectDb *database, uint64_t task_id,
Lardon3DProjectDbRawDevelopmentBatchTask *parameters) {
if (!database || !valid_task_id(task_id) || !parameters)
return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
memset(parameters, 0, sizeof(*parameters));
(void)pthread_mutex_lock(&database->mutex);
sqlite3_stmt *statement = NULL;
Lardon3DProjectDbResult result = prepare(
database,
"SELECT b.selected_execution_id,t.task_kind,t.task_kind_version,e.execution_id "
"FROM raw_development_batch_tasks b JOIN tasks t ON t.task_id=b.task_id "
"LEFT JOIN selected_executions e ON e.execution_id=b.selected_execution_id "
"WHERE b.task_id=?1", &statement);
if (result == LARDON3D_PROJECT_DB_OK) {
sqlite3_bind_int64(statement, 1, (sqlite3_int64)task_id);
int code = sqlite3_step(statement);
if (code == SQLITE_DONE) {
result = LARDON3D_PROJECT_DB_NOT_FOUND;
} else if (code != SQLITE_ROW) {
result = sqlite_result(database, code, "load RAW development batch task");
} else {
sqlite3_int64 execution = sqlite3_column_int64(statement, 0);
sqlite3_int64 version = sqlite3_column_int64(statement, 2);
if (sqlite3_column_type(statement, 0) != SQLITE_INTEGER || execution <= 0 ||
sqlite3_column_type(statement, 1) != SQLITE_TEXT ||
sqlite3_column_type(statement, 2) != SQLITE_INTEGER ||
sqlite3_column_type(statement, 3) != SQLITE_INTEGER ||
sqlite3_column_int64(statement, 3) != execution ||
!column_text_equals(statement, 1,
LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND) ||
version != LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION) {
result = LARDON3D_PROJECT_DB_CORRUPT;
} else {
parameters->task_id = task_id;
parameters->selected_execution_id = (uint64_t)execution;
}
}
sqlite3_finalize(statement);
}
(void)pthread_mutex_unlock(&database->mutex);
return result;
}
static bool valid_quality_result(const Lardon3DProjectDbPhotoQualityResult *r) { static bool valid_quality_result(const Lardon3DProjectDbPhotoQualityResult *r) {
const Lardon3DPhotoQualityMetrics *m = r ? &r->metrics : NULL; const Lardon3DPhotoQualityMetrics *m = r ? &r->metrics : NULL;
bool decoded_pair = m && ((m->decoded_width == 0) == (m->decoded_height == 0)); bool decoded_pair = m && ((m->decoded_width == 0) == (m->decoded_height == 0));
@ -6466,6 +6671,105 @@ lardon3d_project_db_load_feature_extract_task(Lardon3DProjectDb *db, uint64_t ta
return result; return result;
} }
Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_batch_task(
Lardon3DProjectDb *db, const Lardon3DTaskDurableSnapshot *snapshot,
const char *kind, uint32_t version, const Lardon3DProjectDbCheckpoint *checkpoint,
const Lardon3DProjectDbFeatureExtractBatchTask *parameters, int64_t updated_at) {
if (!parameters) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0,
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
NULL, NULL, parameters, updated_at);
}
Lardon3DProjectDbResult lardon3d_project_db_load_feature_extract_batch_task(
Lardon3DProjectDb *db, uint64_t task_id,
Lardon3DProjectDbFeatureExtractBatchTask *p) {
if (!db || !valid_task_id(task_id) || !p) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
memset(p, 0, sizeof(*p));
pthread_mutex_lock(&db->mutex);
sqlite3_stmt *s = NULL;
Lardon3DProjectDbResult result = prepare(
db, "SELECT b.selected_execution_id,b.next_item_index,b.extractor_kind,"
"b.extractor_version,b.max_features,b.pyramid_levels,b.fast_threshold,"
"b.parameter_fingerprint,e.item_count,e.stage,t.task_kind,t.task_kind_version "
"FROM feature_extract_batch_tasks b JOIN selected_executions e ON "
"e.execution_id=b.selected_execution_id JOIN tasks t ON t.task_id=b.task_id "
"WHERE b.task_id=?1", &s);
if (result == LARDON3D_PROJECT_DB_OK) {
sqlite3_bind_int64(s, 1, (sqlite3_int64)task_id);
int code = sqlite3_step(s);
if (code == SQLITE_DONE) result = LARDON3D_PROJECT_DB_NOT_FOUND;
else if (code != SQLITE_ROW) result = sqlite_result(db, code, "load feature batch task");
else {
sqlite3_int64 execution = sqlite3_column_int64(s, 0);
sqlite3_int64 cursor = sqlite3_column_int64(s, 1);
sqlite3_int64 version = sqlite3_column_int64(s, 3);
sqlite3_int64 max = sqlite3_column_int64(s, 4);
sqlite3_int64 levels = sqlite3_column_int64(s, 5);
sqlite3_int64 threshold = sqlite3_column_int64(s, 6);
sqlite3_int64 count = sqlite3_column_int64(s, 8);
sqlite3_int64 stage = sqlite3_column_int64(s, 9);
const void *fp = sqlite3_column_blob(s, 7);
const unsigned char *task_kind = sqlite3_column_text(s, 10);
if (sqlite3_column_type(s, 0) != SQLITE_INTEGER || execution <= 0 ||
sqlite3_column_type(s, 1) != SQLITE_INTEGER || cursor < 0 || cursor > count ||
!copy_column(s, 2, p->extractor_kind, sizeof(p->extractor_kind)) ||
strcmp(p->extractor_kind, LARDON3D_FEATURE_EXTRACTOR_KIND) != 0 || version != 1 ||
max <= 0 || max > 8192 || levels <= 0 || levels > 16 || threshold <= 0 ||
threshold > 255 || sqlite3_column_type(s, 7) != SQLITE_BLOB ||
sqlite3_column_bytes(s, 7) != 32 || !fp || count <= 0 || count > 4096 ||
stage < LARDON3D_SELECTED_EXECUTION_CALIBRATION || !task_kind ||
strcmp((const char *)task_kind, LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND) != 0 ||
sqlite3_column_int64(s, 11) != LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION) {
result = LARDON3D_PROJECT_DB_CORRUPT;
} else {
p->task_id = task_id;
p->selected_execution_id = (uint64_t)execution;
p->next_item_index = (uint32_t)cursor;
p->extractor_version = (uint32_t)version;
p->max_features = (uint32_t)max;
p->pyramid_levels = (uint32_t)levels;
p->fast_threshold = (uint32_t)threshold;
memcpy(p->parameter_fingerprint, fp, 32);
}
}
sqlite3_finalize(s);
}
pthread_mutex_unlock(&db->mutex);
return result;
}
Lardon3DProjectDbResult lardon3d_project_db_advance_feature_extract_batch_task(
Lardon3DProjectDb *db, uint64_t task_id, uint32_t item_index,
uint32_t next_item_index) {
if (!db || !valid_task_id(task_id) || next_item_index != item_index + 1u)
return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
pthread_mutex_lock(&db->mutex);
sqlite3_stmt *s = NULL;
Lardon3DProjectDbResult result = execute(db, "BEGIN IMMEDIATE", "begin feature batch advance");
if (result == LARDON3D_PROJECT_DB_OK)
result = prepare(db,
"UPDATE feature_extract_batch_tasks AS b SET next_item_index=?3 WHERE b.task_id=?1 AND "
"(b.next_item_index=?2 OR b.next_item_index=?3) AND EXISTS(SELECT 1 FROM "
"selected_execution_items i JOIN feature_sets f ON f.image_id=i.image_id JOIN "
"feature_assets a ON a.feature_asset_id=f.feature_asset_id WHERE "
"i.execution_id=b.selected_execution_id AND i.item_index=?2 AND i.image_id IS NOT NULL "
"AND f.extractor_kind=b.extractor_kind AND f.extractor_version=b.extractor_version AND "
"f.parameter_fingerprint=b.parameter_fingerprint AND a.durability=0)", &s);
if (result == LARDON3D_PROJECT_DB_OK) {
sqlite3_bind_int64(s, 1, (sqlite3_int64)task_id);
sqlite3_bind_int64(s, 2, item_index);
sqlite3_bind_int64(s, 3, next_item_index);
result = step_done(db, s, "advance feature batch cursor");
if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(db->connection) != 1)
result = LARDON3D_PROJECT_DB_CONSTRAINT;
}
if (result == LARDON3D_PROJECT_DB_OK) result = execute(db, "COMMIT", "commit feature batch advance");
else (void)execute(db, "ROLLBACK", "rollback feature batch advance");
pthread_mutex_unlock(&db->mutex);
return result;
}
Lardon3DProjectDbResult lardon3d_project_db_load_sift_extract_task( Lardon3DProjectDbResult lardon3d_project_db_load_sift_extract_task(
Lardon3DProjectDb *db, uint64_t task_id, Lardon3DProjectDbSiftExtractTask *p) { Lardon3DProjectDb *db, uint64_t task_id, Lardon3DProjectDbSiftExtractTask *p) {
if (!db || !valid_task_id(task_id) || !p) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; if (!db || !valid_task_id(task_id) || !p) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
@ -6965,7 +7269,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_visual_index_update_task(
return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
} }
return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0, NULL, return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0, NULL,
NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, updated_at); NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
NULL, updated_at);
} }
Lardon3DProjectDbResult lardon3d_project_db_load_visual_index_update_task( Lardon3DProjectDbResult lardon3d_project_db_load_visual_index_update_task(
@ -7012,7 +7317,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_candidate_pair_generate_task(
return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
} }
return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0, NULL, return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0, NULL,
NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, updated_at); NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
NULL, updated_at);
} }
Lardon3DProjectDbResult lardon3d_project_db_load_candidate_pair_generate_task( Lardon3DProjectDbResult lardon3d_project_db_load_candidate_pair_generate_task(
@ -7068,7 +7374,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_matcher_task(
return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
} }
return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0, NULL, return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0, NULL,
NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL, updated_at); NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL,
NULL, updated_at);
} }
Lardon3DProjectDbResult lardon3d_project_db_load_matcher_task( Lardon3DProjectDbResult lardon3d_project_db_load_matcher_task(
@ -7126,7 +7433,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_geometric_verifier_task(
return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
} }
return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0, NULL, return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0, NULL,
NULL, NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL, updated_at); NULL, NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL,
NULL, updated_at);
} }
Lardon3DProjectDbResult lardon3d_project_db_load_geometric_verifier_task( Lardon3DProjectDbResult lardon3d_project_db_load_geometric_verifier_task(
@ -7199,7 +7507,7 @@ Lardon3DProjectDbResult lardon3d_project_db_record_track_builder_task(
} }
Lardon3DProjectDbResult result = record_task_internal( Lardon3DProjectDbResult result = record_task_internal(
db, snapshot, kind, version, checkpoint, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, db, snapshot, kind, version, checkpoint, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
NULL, NULL, NULL, updated_at); NULL, NULL, NULL, NULL, NULL, updated_at);
if (result != LARDON3D_PROJECT_DB_OK) return result; if (result != LARDON3D_PROJECT_DB_OK) return result;
(void)pthread_mutex_lock(&db->mutex); (void)pthread_mutex_lock(&db->mutex);
result = execute(db, "BEGIN IMMEDIATE", "begin track builder task"); result = execute(db, "BEGIN IMMEDIATE", "begin track builder task");
@ -7310,7 +7618,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_sparse_sfm_task(
const Lardon3DProjectDbSparseSfmTask *parameters, int64_t updated_at) { const Lardon3DProjectDbSparseSfmTask *parameters, int64_t updated_at) {
if (!parameters) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT; if (!parameters) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
return record_task_internal(database, snapshot, kind, version, checkpoint, NULL, 0, NULL, NULL, return record_task_internal(database, snapshot, kind, version, checkpoint, NULL, 0, NULL, NULL,
NULL, NULL, NULL, NULL, parameters, NULL, NULL, NULL, updated_at); NULL, NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL,
NULL, updated_at);
} }
static bool read_u64_blob(sqlite3_stmt *statement, int column, uint64_t *value) { static bool read_u64_blob(sqlite3_stmt *statement, int column, uint64_t *value) {

View file

@ -8,7 +8,10 @@ extern "C" {
#include "opencv_task_thread_guard.h" #include "opencv_task_thread_guard.h"
#include <cstdio> #include <cstdio>
#include <algorithm>
#include <array>
#include <new> #include <new>
#include <pthread.h>
namespace { namespace {
/* One admitted RAW execution owns a conservative allowance for the bounded /* One admitted RAW execution owns a conservative allowance for the bounded
@ -17,7 +20,12 @@ namespace {
* scientific limit on the number of Captures. The Queue/Governor reservation * scientific limit on the number of Captures. The Queue/Governor reservation
* exists only for this one Task callback and is released by Task runtime on * exists only for this one Task callback and is released by Task runtime on
* every terminal path. */ * every terminal path. */
constexpr uint64_t kRawWorkingBytes = UINT64_C(2) * 1024u * 1024u * 1024u; constexpr uint64_t kRawSingleWorkingBytes = UINT64_C(2) * 1024u * 1024u * 1024u;
constexpr size_t kRawBatchWindowMax = 8;
constexpr size_t kRawBatchChildStackBytes = 1024u * 1024u;
constexpr uint64_t kRawBatchParticipantBytes = UINT64_C(896) * 1024u * 1024u;
static_assert(kRawBatchParticipantBytes <= UINT64_MAX / kRawBatchWindowMax,
"RAW batch memory envelope must fit uint64_t");
struct Context { struct Context {
char project_path[LARDON3D_APP_STATE_PATH_CAPACITY]{}; char project_path[LARDON3D_APP_STATE_PATH_CAPACITY]{};
@ -27,6 +35,13 @@ struct Context {
uint64_t source_asset_id{}; uint64_t source_asset_id{};
}; };
struct BatchContext {
char project_path[LARDON3D_APP_STATE_PATH_CAPACITY]{};
Lardon3DProjectDb *database{};
Lardon3DResourceGovernor *governor{};
uint64_t selected_execution_id{};
};
void destroy(void *value) { delete static_cast<Context *>(value); } void destroy(void *value) { delete static_cast<Context *>(value); }
void runtime(const Context *context, Lardon3DAppState *state) { void runtime(const Context *context, Lardon3DAppState *state) {
@ -129,6 +144,242 @@ Context *make_context(const Lardon3DTaskReconstructionContext *runtime_context,
context->source_asset_id = source_asset_id; context->source_asset_id = source_asset_id;
return context; return context;
} }
void destroy_batch(void *value) { delete static_cast<BatchContext *>(value); }
BatchContext *make_batch_context(
const Lardon3DTaskReconstructionContext *runtime_context,
uint64_t selected_execution_id) {
if (!runtime_context || !runtime_context->project_path ||
!runtime_context->project_path[0] || !runtime_context->project_db ||
!runtime_context->resource_governor || selected_execution_id == 0)
return nullptr;
auto *context = new (std::nothrow) BatchContext;
if (!context) return nullptr;
int written = std::snprintf(context->project_path, sizeof(context->project_path), "%s",
runtime_context->project_path);
if (written <= 0 || written >= static_cast<int>(sizeof(context->project_path))) {
delete context;
return nullptr;
}
context->database = runtime_context->project_db;
context->governor = runtime_context->resource_governor;
context->selected_execution_id = selected_execution_id;
return context;
}
void batch_runtime(const BatchContext *context, Lardon3DAppState *state) {
lardon3d_app_state_init(state);
state->project_loaded = true;
state->project_db = context->database;
state->resource_governor = context->governor;
std::snprintf(state->project_path, sizeof(state->project_path), "%s",
context->project_path);
}
bool batch_checkpoint(BatchContext *context, Lardon3DTask *task) {
Lardon3DAppState state;
batch_runtime(context, &state);
Lardon3DProjectDbRawDevelopmentBatchTask parameters{
lardon3d_task_id(task), context->selected_execution_id};
return lardon3d_project_checkpoint_raw_development_batch_task(
&state, task, &parameters) == LARDON3D_PROJECT_TASK_CHECKPOINT_OK;
}
struct BatchWorker {
BatchContext *context{};
Lardon3DProjectDbSelectedExecutionItem item{};
uint64_t producer_task_id{};
Lardon3DRawDevelopmentResult raw_result{LARDON3D_RAW_DEVELOPMENT_INTERNAL_ERROR};
Lardon3DProjectDbResult db_result{LARDON3D_PROJECT_DB_IO_ERROR};
uint64_t image_id{};
};
void *develop_batch_item(void *value) {
auto *worker = static_cast<BatchWorker *>(value);
if (worker->item.representation_source ==
LARDON3D_SELECTED_REPRESENTATION_SOURCE_IMAGE) {
worker->db_result = lardon3d_project_db_get_selected_capture_image(
worker->context->database, worker->item.capture_id, &worker->image_id);
return nullptr;
}
Lardon3DProjectDbCapture capture{};
worker->db_result = lardon3d_project_db_load_capture(
worker->context->database, worker->item.capture_id, &capture);
if (worker->db_result != LARDON3D_PROJECT_DB_OK) return nullptr;
Lardon3DAppState state;
batch_runtime(worker->context, &state);
Lardon3DRawDevelopmentOutput output{};
worker->raw_result = lardon3d_raw_develop_asset_to_capture(
&state, worker->item.capture_id, worker->item.source_asset_id,
worker->producer_task_id, capture.created_at, &output);
if (worker->raw_result == LARDON3D_RAW_DEVELOPMENT_OK) {
worker->image_id = output.image.image_id;
worker->db_result = LARDON3D_PROJECT_DB_OK;
}
return nullptr;
}
struct BatchParticipant {
BatchWorker *workers{};
size_t count{};
size_t first{};
size_t stride{};
};
void *develop_batch_participant(void *value) {
auto *participant = static_cast<BatchParticipant *>(value);
for (size_t index = participant->first; index < participant->count;
index += participant->stride)
(void)develop_batch_item(&participant->workers[index]);
return nullptr;
}
bool develop_batch_window(BatchContext *context, uint64_t producer_task_id,
const Lardon3DProjectDbSelectedExecutionItem *items,
size_t count, unsigned int admitted_threads,
std::array<BatchWorker, kRawBatchWindowMax> *workers) {
if (!context || !items || !workers || count == 0 || count > kRawBatchWindowMax ||
admitted_threads == 0)
return false;
size_t participants = std::min(count, static_cast<size_t>(admitted_threads));
std::array<pthread_t, kRawBatchWindowMax - 1> children{};
std::array<BatchParticipant, kRawBatchWindowMax> participant_work{};
pthread_attr_t attributes;
if (pthread_attr_init(&attributes) != 0) return false;
if (pthread_attr_setstacksize(&attributes, kRawBatchChildStackBytes) != 0) {
(void)pthread_attr_destroy(&attributes);
return false;
}
for (size_t index = 0; index < count; ++index) {
(*workers)[index].context = context;
(*workers)[index].item = items[index];
(*workers)[index].producer_task_id = producer_task_id;
}
size_t created = 0;
bool started = true;
for (size_t participant = 1; participant < participants; ++participant) {
participant_work[participant] = {
workers->data(), count, participant, participants};
/* Each participant processes one disjoint strided subset. No child owns
* selected cursor/progress publication, even when it finishes first. */
if (pthread_create(&children[created], &attributes,
develop_batch_participant,
&participant_work[participant]) != 0) {
started = false;
break;
}
++created;
}
if (pthread_attr_destroy(&attributes) != 0) started = false;
participant_work[0] = {workers->data(), count, 0, participants};
if (started) (void)develop_batch_participant(&participant_work[0]);
for (size_t index = 0; index < created; ++index) {
if (pthread_join(children[index], nullptr) != 0) started = false;
}
return started;
}
bool run_batch_impl(Lardon3DTask *task, BatchContext *context) {
for (;;) {
if (!lardon3d_task_checkpoint(task)) return false;
Lardon3DProjectDbSelectedExecution execution{};
if (lardon3d_project_db_load_selected_execution(
context->database, context->selected_execution_id, &execution) !=
LARDON3D_PROJECT_DB_OK)
return lardon3d_task_fail(task, "Exécution sélectionnée RAW batch invalide.");
if (execution.next_item_index == execution.item_count) {
return lardon3d_task_set_durable_progress(
task, execution.item_count, execution.item_count,
"Développement RAW batch terminé.") &&
batch_checkpoint(context, task);
}
if (execution.stage != LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS)
return lardon3d_task_fail(task, "Étape sélectionnée RAW batch invalide.");
Lardon3DTaskExecutionContract contract{};
if (!lardon3d_task_execution_contract(task, &contract) ||
contract.batch_size == 0 || contract.batch_size > kRawBatchWindowMax ||
contract.cpu_threads == 0)
return lardon3d_task_fail(task, "Contrat Governor RAW batch invalide.");
size_t count = std::min<size_t>(contract.batch_size,
execution.item_count - execution.next_item_index);
std::array<Lardon3DProjectDbSelectedExecutionItem, kRawBatchWindowMax> items{};
for (size_t offset = 0; offset < count; ++offset) {
if (lardon3d_project_db_load_selected_execution_item(
context->database, context->selected_execution_id,
execution.next_item_index + static_cast<uint32_t>(offset),
&items[offset]) != LARDON3D_PROJECT_DB_OK)
return lardon3d_task_fail(task, "Item sélectionné RAW batch invalide.");
}
std::array<BatchWorker, kRawBatchWindowMax> workers{};
/* Each independent LibRaw/BGR/PNG participant fits the admitted 896 MiB
* per-item envelope. Every child is joined before cursor publication or
* reservation release. */
if (!develop_batch_window(context, lardon3d_task_id(task), items.data(), count,
contract.cpu_threads, &workers))
return lardon3d_task_fail(task, "Workers RAW batch impossibles.");
if (!lardon3d_task_checkpoint(task)) return false;
for (size_t offset = 0; offset < count; ++offset) {
BatchWorker &worker = workers[offset];
if (worker.db_result != LARDON3D_PROJECT_DB_OK || worker.image_id == 0 ||
(worker.item.representation_source ==
LARDON3D_SELECTED_REPRESENTATION_RAW_ASSET &&
worker.raw_result != LARDON3D_RAW_DEVELOPMENT_OK))
return lardon3d_task_fail(task, "Développement d'un item RAW batch impossible.");
uint32_t item_index = worker.item.item_index;
if (lardon3d_project_db_record_selected_representation(
context->database, context->selected_execution_id, item_index,
worker.image_id, item_index + 1u) != LARDON3D_PROJECT_DB_OK)
return lardon3d_task_fail(task, "Publication ordonnée RAW batch impossible.");
/* The selected cursor commit is the authority. Generic durable progress
* follows it and may lag after a crash, but must never lead it. */
if (!lardon3d_task_set_durable_progress(
task, item_index + 1u, execution.item_count,
"Représentation RAW batch publiée.") ||
!batch_checkpoint(context, task))
return lardon3d_task_fail(task, "Checkpoint RAW batch impossible.");
}
if (execution.next_item_index + count == execution.item_count)
return true;
Lardon3DResourceReservation *reservation = nullptr;
if (!lardon3d_task_sequence_break(task, context->governor, &reservation,
&contract))
return false;
}
}
bool run_batch(Lardon3DTask *task, void *value) noexcept {
try {
Lardon3DOpenCvTaskThreadGuard threads(task);
if (!threads.valid())
return lardon3d_task_fail(task, "Contrat CPU OpenCV RAW batch invalide.");
/* External Task participants are the admitted CPU dimension. OpenCV stays
* single-threaded so imencode/imdecode cannot create a nested fan-out. */
cv::setNumThreads(1);
if (cv::getNumThreads() != 1)
return lardon3d_task_fail(task, "Configuration OpenCV RAW batch impossible.");
bool result = run_batch_impl(task, static_cast<BatchContext *>(value));
if (!threads.restore())
return lardon3d_task_fail(task, "Restauration OpenCV RAW batch impossible.");
return result;
} catch (const std::bad_alloc &) {
return lardon3d_task_fail(task, "Mémoire insuffisante pour le RAW batch.");
} catch (...) {
return lardon3d_task_fail(task, "Erreur interne du RAW batch.");
}
}
void finished_batch(const Lardon3DTask *task, void *value) noexcept {
try {
Lardon3DAppState state;
batch_runtime(static_cast<BatchContext *>(value), &state);
(void)lardon3d_project_checkpoint_task(&state, task);
} catch (...) {
}
}
} // namespace } // namespace
extern "C" bool lardon3d_raw_development_task_reconstruct( extern "C" bool lardon3d_raw_development_task_reconstruct(
@ -173,7 +424,7 @@ extern "C" Lardon3DTask *lardon3d_project_create_raw_development_task(
delete context; delete context;
return nullptr; return nullptr;
} }
Lardon3DResourceEstimate estimate{kRawWorkingBytes + sizeof(Context), 0, 0, 0, Lardon3DResourceEstimate estimate{kRawSingleWorkingBytes + sizeof(Context), 0, 0, 0,
1, 1, 1, 0, 1, LARDON3D_RESOURCE_TASK_MIXED}; 1, 1, 1, 0, 1, LARDON3D_RESOURCE_TASK_MIXED};
Lardon3DTask *task = lardon3d_task_create_typed( Lardon3DTask *task = lardon3d_task_create_typed(
"Développement RAW S3-B1", &estimate, LARDON3D_RAW_DEVELOPMENT_TASK_KIND, "Développement RAW S3-B1", &estimate, LARDON3D_RAW_DEVELOPMENT_TASK_KIND,
@ -208,3 +459,104 @@ extern "C" bool lardon3d_project_enqueue_raw_development(
return false; return false;
} }
} }
extern "C" bool lardon3d_raw_development_batch_task_reconstruct(
const Lardon3DTaskDurableSnapshot *snapshot, void *value,
Lardon3DTaskKindBinding *binding) {
try {
auto *runtime_context = static_cast<Lardon3DTaskReconstructionContext *>(value);
if (!snapshot || !runtime_context || !binding) return false;
Lardon3DProjectDbRawDevelopmentBatchTask persisted{};
if (lardon3d_project_db_load_raw_development_batch_task(
runtime_context->project_db, snapshot->id, &persisted) !=
LARDON3D_PROJECT_DB_OK)
return false;
BatchContext *context = make_batch_context(
runtime_context, persisted.selected_execution_id);
if (!context) return false;
*binding = {};
binding->callback = run_batch;
binding->userdata = context;
binding->userdata_destroy = destroy_batch;
binding->finished_callback = finished_batch;
binding->finished_userdata = context;
return true;
} catch (...) {
return false;
}
}
extern "C" Lardon3DTask *lardon3d_project_create_raw_development_batch_task(
Lardon3DAppState *state, uint64_t selected_execution_id, uint64_t *task_id) {
if (task_id) *task_id = 0;
try {
if (!state || !state->project_loaded || !state->project_db ||
!state->resource_governor || !task_id || selected_execution_id == 0)
return nullptr;
Lardon3DProjectDbSelectedExecution execution{};
if (lardon3d_project_db_load_selected_execution(
state->project_db, selected_execution_id, &execution) !=
LARDON3D_PROJECT_DB_OK ||
execution.stage != LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS)
return nullptr;
Lardon3DTaskReconstructionContext runtime_context{
state->project_path, state->project_db, state->resource_governor, nullptr};
BatchContext *context = make_batch_context(&runtime_context,
selected_execution_id);
if (!context) return nullptr;
uint64_t id = 0;
if (lardon3d_project_db_allocate_task_id(state->project_db, &id) !=
LARDON3D_PROJECT_DB_OK) {
delete context;
return nullptr;
}
/* At the frozen 40 MP decoder ceiling, one participant owns LibRaw's
* mosaic/16-bit four-channel workspace, the processed RGB/BGR copies, and
* the later PNG/validation buffers. Their lifetime analysis plus the 1 MiB
* child stack and allocator/codec headroom is bounded by 896 MiB. Charge
* the tiny owner context through that per-item allowance too: a 7 GiB
* post-reserve host budget can then admit all eight independently bounded
* participants instead of losing one to a redundant fixed charge.
*
* The 1..8 capacity is operational admission only. Immutable selected
* execution and item order remain the scientific, durable input, never a
* dataset-size limit. */
Lardon3DResourceEstimate estimate{
0, 0, kRawBatchParticipantBytes, 0,
1, kRawBatchWindowMax, static_cast<unsigned int>(kRawBatchWindowMax),
0, 1, LARDON3D_RESOURCE_TASK_MIXED};
Lardon3DTask *task = lardon3d_task_create_typed(
"Développement RAW sélectionné", &estimate,
LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND,
LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION, run_batch, context,
destroy_batch);
if (!task || !lardon3d_task_assign_id(task, id) ||
!lardon3d_task_set_finished_callback(task, finished_batch, context) ||
!batch_checkpoint(context, task)) {
lardon3d_task_destroy(task);
return nullptr;
}
*task_id = id;
return task;
} catch (...) {
return nullptr;
}
}
extern "C" bool lardon3d_project_enqueue_raw_development_batch(
Lardon3DAppState *state, uint64_t selected_execution_id,
uint64_t *task_id) {
try {
if (!state || !state->task_queue) return false;
Lardon3DTask *task = lardon3d_project_create_raw_development_batch_task(
state, selected_execution_id, task_id);
if (!task) return false;
if (!lardon3d_task_queue_add(state->task_queue, task, nullptr)) {
lardon3d_task_destroy(task);
return false;
}
return true;
} catch (...) {
return false;
}
}

View file

@ -1644,7 +1644,10 @@ open_next_capability_trial_locked(
&& feedback->accepted_cpu_limit < cpu_max) { && feedback->accepted_cpu_limit < cpu_max) {
feedback->adaptive_cpu_limit = next_trial_cpu( feedback->adaptive_cpu_limit = next_trial_cpu(
feedback->accepted_cpu_limit, cpu_max); feedback->accepted_cpu_limit, cpu_max);
feedback->adaptive_batch_limit = feedback->accepted_batch_limit; feedback->adaptive_batch_limit = capability->cpu_batch_coupled
? minimum_size(feedback->adaptive_cpu_limit,
capability->estimate.maximum_batch_size)
: feedback->accepted_batch_limit;
feedback->adaptive_inflight_limit = feedback->adaptive_inflight_limit =
feedback->accepted_inflight_limit; feedback->accepted_inflight_limit;
feedback->trial_dimension = LARDON3D_CAPABILITY_TRIAL_CPU; feedback->trial_dimension = LARDON3D_CAPABILITY_TRIAL_CPU;
@ -4291,7 +4294,10 @@ lardon3d_resource_governor_internal_record_sequence_execution_metrics(
bool exercised = feedback->trial_dimension bool exercised = feedback->trial_dimension
== LARDON3D_CAPABILITY_TRIAL_CPU == LARDON3D_CAPABILITY_TRIAL_CPU
? selection->decision.cpu_threads ? selection->decision.cpu_threads
== feedback->adaptive_cpu_limit == feedback->adaptive_cpu_limit
&& (!selection->capability.cpu_batch_coupled
|| selection->decision.batch_size
== feedback->adaptive_batch_limit)
: feedback->trial_dimension : feedback->trial_dimension
== LARDON3D_CAPABILITY_TRIAL_INFLIGHT == LARDON3D_CAPABILITY_TRIAL_INFLIGHT
? selection->inflight_limit ? selection->inflight_limit
@ -4350,6 +4356,13 @@ lardon3d_resource_governor_internal_record_sequence_execution_metrics(
== LARDON3D_CAPABILITY_TRIAL_CPU) { == LARDON3D_CAPABILITY_TRIAL_CPU) {
feedback->accepted_cpu_limit = feedback->accepted_cpu_limit =
feedback->adaptive_cpu_limit; feedback->adaptive_cpu_limit;
if (selection->capability.cpu_batch_coupled) {
/* The measured gain belongs to the complete
* cross-image rung, not to an unusable CPU
* count detached from its participant window. */
feedback->accepted_batch_limit =
feedback->adaptive_batch_limit;
}
} else if (completed_dimension } else if (completed_dimension
== LARDON3D_CAPABILITY_TRIAL_INFLIGHT) { == LARDON3D_CAPABILITY_TRIAL_INFLIGHT) {
feedback->accepted_inflight_limit = feedback->accepted_inflight_limit =

View file

@ -44,6 +44,11 @@ typedef struct {
bool preferred; bool preferred;
bool cpu_reducible; bool cpu_reducible;
bool batch_adaptive; bool batch_adaptive;
/* Cross-item CPU callbacks cannot exercise a CPU trial unless the same
* immutable sequence admits at least that many independent items. The
* Governor therefore advances and accepts these two operational limits as
* one rung; this property is private and never changes durable science. */
bool cpu_batch_coupled;
/* Current ORB Vulkan batch trials use a longer observation window than /* Current ORB Vulkan batch trials use a longer observation window than
* generic CPU adaptation. This private operational property is * generic CPU adaptation. This private operational property is
* reconstructed with the capability and is never durable/scientific. */ * reconstructed with the capability and is never durable/scientific. */

View file

@ -112,6 +112,7 @@ kind_has_validated_cpu_range(const char *task_kind, uint32_t task_kind_version)
return false; return false;
} }
return strcmp(task_kind, "features.extract") == 0 return strcmp(task_kind, "features.extract") == 0
|| strcmp(task_kind, "features.extract.batch") == 0
|| strcmp(task_kind, "features.extract.sift") == 0 || strcmp(task_kind, "features.extract.sift") == 0
|| strcmp(task_kind, "features.extract.rootsift") == 0 || strcmp(task_kind, "features.extract.rootsift") == 0
|| strcmp(task_kind, "visual_index.update") == 0 || strcmp(task_kind, "visual_index.update") == 0
@ -127,7 +128,20 @@ kind_has_validated_batch_range(const char *task_kind, uint32_t version)
* one-item batch cannot exercise a second participant; Gate G may still * one-item batch cannot exercise a second participant; Gate G may still
* reduce that bounded window for capacity or pressure. */ * reduce that bounded window for capacity or pressure. */
return task_kind && version == 1 return task_kind && version == 1
&& strcmp(task_kind, "candidate_pair.generate") == 0; && (strcmp(task_kind, "candidate_pair.generate") == 0
|| strcmp(task_kind, "features.extract.batch") == 0);
}
static bool
kind_has_coupled_cpu_batch_range(const char *task_kind, uint32_t version)
{
/* Feature batch and Candidate obtain CPU parallelism only across
* independent items. A CPU2/batch1 trial is structurally incapable of
* measuring either path, so Governor feedback must trial the complete
* participant rung without changing scientific ordering or publication. */
return task_kind && version == 1
&& (strcmp(task_kind, "features.extract.batch") == 0
|| strcmp(task_kind, "candidate_pair.generate") == 0);
} }
static void static void
@ -244,6 +258,9 @@ lardon3d_task_create_typed(
&& kind_has_validated_cpu_range(task_kind, task_kind_version), && kind_has_validated_cpu_range(task_kind, task_kind_version),
.batch_adaptive = typed .batch_adaptive = typed
&& kind_has_validated_batch_range(task_kind, task_kind_version), && kind_has_validated_batch_range(task_kind, task_kind_version),
.cpu_batch_coupled = typed
&& kind_has_coupled_cpu_batch_range(
task_kind, task_kind_version),
}}, }},
}; };
copy_text(task->message, sizeof(task->message), "En attente."); copy_text(task->message, sizeof(task->message), "En attente.");
@ -276,6 +293,9 @@ lardon3d_task_internal_set_capability_envelope(
&& (!capability->batch_adaptive && (!capability->batch_adaptive
|| capability->backend || capability->backend
!= LARDON3D_RESOURCE_BACKEND_ORB_VULKAN)) != LARDON3D_RESOURCE_BACKEND_ORB_VULKAN))
|| (capability->cpu_batch_coupled
&& (!capability->cpu_reducible
|| !capability->batch_adaptive))
|| (capability->inflight_adaptive || (capability->inflight_adaptive
&& (capability->minimum_inflight_limit == 0 && (capability->minimum_inflight_limit == 0
|| capability->gpu_memory_bytes_per_inflight == 0)) || capability->gpu_memory_bytes_per_inflight == 0))
@ -317,6 +337,9 @@ lardon3d_task_internal_enable_known_capabilities(Lardon3DTask *task)
task->capability_envelope.capabilities[0].batch_adaptive = task->capability_envelope.capabilities[0].batch_adaptive =
kind_has_validated_batch_range( kind_has_validated_batch_range(
task->task_kind, task->task_kind_version); task->task_kind, task->task_kind_version);
task->capability_envelope.capabilities[0].cpu_batch_coupled =
kind_has_coupled_cpu_batch_range(
task->task_kind, task->task_kind_version);
} }
(void)pthread_mutex_unlock(&task->mutex); (void)pthread_mutex_unlock(&task->mutex);
return true; return true;

View file

@ -38,6 +38,13 @@ enum {
VISUAL_INDEX_PER_ITEM_BYTES = 2 * 1024 * 1024, VISUAL_INDEX_PER_ITEM_BYTES = 2 * 1024 * 1024,
}; };
#define RAW_BATCH_LEGACY_PER_ITEM_BYTES \
((UINT64_C(2) * 1024u * 1024u * 1024u) + (UINT64_C(1) * 1024u * 1024u))
#define RAW_BATCH_CURRENT_PER_ITEM_BYTES (UINT64_C(896) * 1024u * 1024u)
/* The first v24 implementation persisted sizeof(BatchContext) as fixed
* memory. Its public checkpoint ABI made that 4120-byte exact shape durable. */
#define RAW_BATCH_LEGACY_FIXED_BYTES UINT64_C(4120)
static bool static bool
estimate_equals(const Lardon3DResourceEstimate *left, estimate_equals(const Lardon3DResourceEstimate *left,
const Lardon3DResourceEstimate *right) const Lardon3DResourceEstimate *right)
@ -182,6 +189,23 @@ normalize_known_legacy_estimate(const char *kind,
/* CPU/batch/memory admission is operational, not GVR identity. Accept /* CPU/batch/memory admission is operational, not GVR identity. Accept
* only the complete frozen serial envelope, normalize it in memory, * only the complete frozen serial envelope, normalize it in memory,
* and never publish an estimate-only checkpoint during recovery. */ * and never publish an estimate-only checkpoint during recovery. */
} else if (strcmp(kind, "raw.develop.batch") == 0) {
current = (Lardon3DResourceEstimate) {
.memory_bytes_per_item = RAW_BATCH_CURRENT_PER_ITEM_BYTES,
.minimum_batch_size = 1,
.maximum_batch_size = 8,
.desired_cpu_threads = 8,
.desired_io_slots = 1,
.task_class = LARDON3D_RESOURCE_TASK_MIXED,
};
historical = current;
historical.memory_fixed_bytes = RAW_BATCH_LEGACY_FIXED_BYTES;
historical.memory_bytes_per_item = RAW_BATCH_LEGACY_PER_ITEM_BYTES;
/* v24 originally persisted this exact 4120-byte owner plus 2 GiB and
* 1 MiB stack participant allowance. It is Governor accounting, not
* RAW identity or the selected-execution payload. Accept only this
* complete legacy shape and normalize it in memory, so recovery can
* reach the corrected safe window without rewriting a checkpoint. */
} else { } else {
*effective = *durable; *effective = *durable;
return true; return true;

View file

@ -33,6 +33,11 @@ const Lardon3DTaskKindRegistry *lardon3d_task_kind_registry_production(void) {
.kind_version = LARDON3D_RAW_DEVELOPMENT_TASK_KIND_VERSION, .kind_version = LARDON3D_RAW_DEVELOPMENT_TASK_KIND_VERSION,
.reconstruct = lardon3d_raw_development_task_reconstruct, .reconstruct = lardon3d_raw_development_task_reconstruct,
}, },
{
.kind = LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND,
.kind_version = LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION,
.reconstruct = lardon3d_raw_development_batch_task_reconstruct,
},
#endif #endif
#ifdef LARDON3D_PHOTO_QUALITY_TASK_AVAILABLE #ifdef LARDON3D_PHOTO_QUALITY_TASK_AVAILABLE
{ {
@ -58,6 +63,11 @@ const Lardon3DTaskKindRegistry *lardon3d_task_kind_registry_production(void) {
.kind_version = LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION, .kind_version = LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION,
.reconstruct = lardon3d_feature_extract_reconstruct, .reconstruct = lardon3d_feature_extract_reconstruct,
}, },
{
.kind = LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND,
.kind_version = LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION,
.reconstruct = lardon3d_feature_extract_batch_reconstruct,
},
{ {
.kind = LARDON3D_SIFT_EXTRACT_TASK_KIND, .kind = LARDON3D_SIFT_EXTRACT_TASK_KIND,
.kind_version = LARDON3D_SIFT_EXTRACT_TASK_KIND_VERSION, .kind_version = LARDON3D_SIFT_EXTRACT_TASK_KIND_VERSION,

View file

@ -167,11 +167,13 @@ kind_to_stage(
if (strcmp(kind, "import.images") == 0 if (strcmp(kind, "import.images") == 0
|| strcmp(kind, "acquisition_campaign.run") == 0) { || strcmp(kind, "acquisition_campaign.run") == 0) {
*stage = LARDON3D_TUI_STAGE_ACQUISITION; *stage = LARDON3D_TUI_STAGE_ACQUISITION;
} else if (strcmp(kind, "raw.develop") == 0) { } else if (strcmp(kind, "raw.develop") == 0 ||
strcmp(kind, "raw.develop.batch") == 0) {
*stage = LARDON3D_TUI_STAGE_RAW; *stage = LARDON3D_TUI_STAGE_RAW;
} else if (strcmp(kind, "photo_quality.triage") == 0) { } else if (strcmp(kind, "photo_quality.triage") == 0) {
*stage = LARDON3D_TUI_STAGE_QUALITY; *stage = LARDON3D_TUI_STAGE_QUALITY;
} else if (strcmp(kind, "features.extract") == 0 } else if (strcmp(kind, "features.extract") == 0
|| strcmp(kind, "features.extract.batch") == 0
|| strcmp(kind, "features.extract.sift") == 0 || strcmp(kind, "features.extract.sift") == 0
|| strcmp(kind, "features.extract.rootsift") == 0) { || strcmp(kind, "features.extract.rootsift") == 0) {
*stage = LARDON3D_TUI_STAGE_FEATURES; *stage = LARDON3D_TUI_STAGE_FEATURES;

View file

@ -53,6 +53,10 @@ struct Options {
bool resume_geometry_existing{}; bool resume_geometry_existing{};
bool resume_pre_gv_existing{}; bool resume_pre_gv_existing{};
bool resume_candidate_existing{}; bool resume_candidate_existing{};
bool resume_representations_existing{};
uint64_t selected_execution_id{};
bool has_selected_execution_id{};
bool stop_after_representations{};
unsigned int cpu_budget{}; unsigned int cpu_budget{};
unsigned int gpu_budget{}; unsigned int gpu_budget{};
bool has_gpu_budget{}; bool has_gpu_budget{};
@ -158,13 +162,30 @@ void usage(const char *program) {
"[--matcher-batch 2|4|8|12] " "[--matcher-batch 2|4|8|12] "
"[--stop-after-matcher]\n" "[--stop-after-matcher]\n"
" %s --resume-candidate-existing --project-dir " " %s --resume-candidate-existing --project-dir "
"ABSOLUTE_EXISTING_DIR [--cpu-budget 1..12] [--gpu-budget 0..1]\n", "ABSOLUTE_EXISTING_DIR [--cpu-budget 1..12] [--gpu-budget 0..1]\n"
" %s --resume-representations-existing --project-dir "
"ABSOLUTE_EXISTING_DIR --selected-execution-id ID "
"[--stop-after-representations]\n",
program,
program, program,
program, program,
program, program,
program); program);
} }
bool parse_u64_id(const char *text, uint64_t &value) {
if (!text || !*text || *text == '0') return false;
for (const char *character = text; *character; ++character) {
if (*character < '0' || *character > '9') return false;
}
char *end = nullptr;
errno = 0;
const unsigned long long parsed = std::strtoull(text, &end, 10);
if (errno != 0 || end == text || *end != '\0' || parsed == 0) return false;
value = static_cast<uint64_t>(parsed);
return true;
}
bool parse_size(const char *text, size_t &value) { bool parse_size(const char *text, size_t &value) {
char *end = nullptr; char *end = nullptr;
errno = 0; errno = 0;
@ -228,11 +249,16 @@ bool parse_options(int argc, char **argv, Options &options) {
options.resume_candidate_existing = true; options.resume_candidate_existing = true;
continue; continue;
} }
if (argument == "--resume-representations-existing") {
options.resume_representations_existing = true;
continue;
}
if ((argument == "--mode" || argument == "--project-dir" || argument == "--root" || if ((argument == "--mode" || argument == "--project-dir" || argument == "--root" ||
argument == "--limit" || argument == "--restart-boundary" || argument == "--limit" || argument == "--restart-boundary" ||
argument == "--cpu-budget" || argument == "--gpu-budget" || argument == "--cpu-budget" || argument == "--gpu-budget" ||
argument == "--matcher-mode" || argument == "--matcher-pipeline" || argument == "--matcher-mode" || argument == "--matcher-pipeline" ||
argument == "--matcher-inflight" || argument == "--matcher-batch") && argument == "--matcher-inflight" || argument == "--matcher-batch" ||
argument == "--selected-execution-id") &&
index + 1 >= argc) index + 1 >= argc)
return false; return false;
if (argument == "--mode") { if (argument == "--mode") {
@ -280,40 +306,59 @@ bool parse_options(int argc, char **argv, Options &options) {
if (!parse_matcher_batch(argv[++index], options.matcher_batch_override)) if (!parse_matcher_batch(argv[++index], options.matcher_batch_override))
return false; return false;
options.has_matcher_batch_override = true; options.has_matcher_batch_override = true;
} else if (argument == "--selected-execution-id") {
if (!parse_u64_id(argv[++index], options.selected_execution_id)) return false;
options.has_selected_execution_id = true;
} else if (argument == "--stop-after-matcher") { } else if (argument == "--stop-after-matcher") {
options.stop_after_matcher = true; options.stop_after_matcher = true;
} else if (argument == "--stop-after-gv") { } else if (argument == "--stop-after-gv") {
options.stop_after_gv = true; options.stop_after_gv = true;
} else if (argument == "--stop-after-representations") {
options.stop_after_representations = true;
} else { } else {
return false; return false;
} }
} }
const unsigned int resume_mode_count = options.resume_geometry_existing + const unsigned int resume_mode_count = options.resume_geometry_existing +
options.resume_pre_gv_existing + options.resume_pre_gv_existing +
options.resume_candidate_existing; options.resume_candidate_existing +
options.resume_representations_existing;
if (resume_mode_count != 0) { if (resume_mode_count != 0) {
if (options.has_mode || options.project_dir.empty() || !options.roots.empty() || if (options.has_mode || options.project_dir.empty() || !options.roots.empty() ||
options.restart != RestartBoundary::kNone || resume_mode_count != 1) options.restart != RestartBoundary::kNone || resume_mode_count != 1)
return false; return false;
if (options.resume_pre_gv_existing) { if (options.resume_pre_gv_existing) {
return !options.stop_after_gv; return !options.stop_after_gv && !options.has_selected_execution_id &&
!options.stop_after_representations;
} }
if (options.resume_candidate_existing) if (options.resume_candidate_existing)
return !options.has_matcher_mode && !options.has_matcher_pipeline && return !options.has_matcher_mode && !options.has_matcher_pipeline &&
!options.has_matcher_inflight_override && !options.has_matcher_inflight_override &&
!options.has_matcher_batch_override && !options.has_matcher_batch_override &&
!options.stop_after_matcher && !options.stop_after_gv; !options.stop_after_matcher && !options.stop_after_gv &&
!options.has_selected_execution_id &&
!options.stop_after_representations;
if (options.resume_representations_existing)
return options.has_selected_execution_id && options.cpu_budget == 0 &&
!options.has_gpu_budget && !options.has_matcher_mode &&
!options.has_matcher_pipeline &&
!options.has_matcher_inflight_override &&
!options.has_matcher_batch_override && !options.stop_after_matcher &&
!options.stop_after_gv;
return options.cpu_budget == 0 && !options.has_gpu_budget && return options.cpu_budget == 0 && !options.has_gpu_budget &&
!options.has_matcher_mode && !options.has_matcher_pipeline && !options.has_matcher_mode && !options.has_matcher_pipeline &&
!options.has_matcher_inflight_override && !options.has_matcher_inflight_override &&
!options.has_matcher_batch_override && !options.has_matcher_batch_override &&
!options.stop_after_matcher; !options.stop_after_matcher && !options.has_selected_execution_id &&
!options.stop_after_representations;
} }
if (options.cpu_budget != 0 || options.has_gpu_budget || if (options.cpu_budget != 0 || options.has_gpu_budget ||
options.has_matcher_mode || options.has_matcher_pipeline || options.has_matcher_mode || options.has_matcher_pipeline ||
options.has_matcher_inflight_override || options.has_matcher_inflight_override ||
options.has_matcher_batch_override || options.has_matcher_batch_override ||
options.stop_after_matcher || options.stop_after_gv) return false; options.stop_after_matcher || options.stop_after_gv ||
options.has_selected_execution_id || options.stop_after_representations)
return false;
return options.has_mode && !options.project_dir.empty() && !options.roots.empty() && return options.has_mode && !options.project_dir.empty() && !options.roots.empty() &&
options.roots.size() <= LARDON3D_ACQUISITION_CAMPAIGN_MAX_ROOTS; options.roots.size() <= LARDON3D_ACQUISITION_CAMPAIGN_MAX_ROOTS;
} }
@ -1440,46 +1485,233 @@ bool create_selection(Runtime &runtime, Mode mode, uint64_t quality_task_id,
items.size(), std::time(nullptr), &execution) == LARDON3D_PROJECT_DB_OK; items.size(), std::time(nullptr), &execution) == LARDON3D_PROJECT_DB_OK;
} }
enum class RepresentationPendingKind { kNone, kLegacySerial, kBatch };
enum class RepresentationAction {
kReject,
kCollect,
kRecoverLegacySerial,
kRecoverBatch,
kEnqueueBatch,
};
RepresentationAction choose_representation_action(
const Lardon3DProjectDbSelectedExecution &execution,
RepresentationPendingKind pending) {
if (execution.next_item_index > execution.item_count) return RepresentationAction::kReject;
if (execution.stage == LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS &&
execution.next_item_index < execution.item_count) {
if (pending == RepresentationPendingKind::kLegacySerial)
return RepresentationAction::kRecoverLegacySerial;
if (pending == RepresentationPendingKind::kBatch)
return RepresentationAction::kRecoverBatch;
return RepresentationAction::kEnqueueBatch;
}
if (execution.stage == LARDON3D_SELECTED_EXECUTION_CALIBRATION &&
execution.next_item_index == execution.item_count) {
return pending == RepresentationPendingKind::kBatch
? RepresentationAction::kRecoverBatch
: pending == RepresentationPendingKind::kNone
? RepresentationAction::kCollect
: RepresentationAction::kReject;
}
return RepresentationAction::kReject;
}
bool representation_defers_feature_task(
const Lardon3DProjectDbSelectedExecution &execution,
const Lardon3DProjectRecoveryEntry &entry) {
if (execution.stage != LARDON3D_SELECTED_EXECUTION_CALIBRATION ||
execution.next_item_index != execution.item_count)
return false;
return (std::strcmp(entry.task_kind, LARDON3D_FEATURE_EXTRACT_TASK_KIND) == 0 &&
entry.task_kind_version == LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION) ||
(std::strcmp(entry.task_kind,
LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND) == 0 &&
entry.task_kind_version ==
LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION);
}
bool find_pending_representation_task(
Runtime &runtime, const Lardon3DProjectDbSelectedExecution &execution,
RepresentationPendingKind &kind, uint64_t &task_id) {
kind = RepresentationPendingKind::kNone;
task_id = 0;
uint64_t cursor = 0;
for (;;) {
Lardon3DProjectRecoveryEntry entries[8]{};
size_t count = 0;
if (lardon3d_project_list_recoverable(
&runtime.state, lardon3d_task_kind_registry_production(), cursor,
entries, 8, &count) != LARDON3D_PROJECT_DB_OK)
return false;
for (size_t index = 0; index < count; ++index) {
const auto &entry = entries[index];
cursor = entry.task_id;
if ((entry.status != LARDON3D_PROJECT_RECOVERABLE &&
entry.status != LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE) ||
entry.snapshot.recovery_state != TASK_PENDING) {
std::fprintf(stderr,
"representation recovery refuses unsafe pending task %llu of kind %s\n",
static_cast<unsigned long long>(entry.task_id), entry.task_kind);
return false;
}
/* Feature recovery owns the phase after representation collection. A
* precise Feature kind/version may already be queued by project-open,
* but representation recovery must neither resume nor reject it once the
* selected representation prefix is durably complete. extract_features
* validates its ORB fingerprint and legacy/batch ownership before wait.
* Every other pending kind remains fail-closed here. */
if (representation_defers_feature_task(execution, entry)) continue;
if (task_id != 0) {
std::fprintf(stderr,
"representation recovery refuses multiple pending owners"
" including task %llu of kind %s\n",
static_cast<unsigned long long>(entry.task_id),
entry.task_kind);
return false;
}
if (std::strcmp(entry.task_kind, LARDON3D_RAW_DEVELOPMENT_TASK_KIND) == 0 &&
entry.task_kind_version == LARDON3D_RAW_DEVELOPMENT_TASK_KIND_VERSION &&
execution.stage == LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS &&
execution.next_item_index < execution.item_count) {
Lardon3DProjectDbSelectedExecutionItem item{};
Lardon3DProjectDbRawDevelopmentTask raw{};
if (lardon3d_project_db_load_selected_execution_item(
runtime.state.project_db, execution.execution_id,
execution.next_item_index, &item) != LARDON3D_PROJECT_DB_OK ||
item.representation_source != LARDON3D_SELECTED_REPRESENTATION_RAW_ASSET ||
lardon3d_project_db_load_raw_development_task(
runtime.state.project_db, entry.task_id, &raw) != LARDON3D_PROJECT_DB_OK ||
raw.capture_id != item.capture_id ||
raw.source_asset_id != item.source_asset_id) {
std::fprintf(stderr,
"legacy RAW task %llu does not own selected cursor %u\n",
static_cast<unsigned long long>(entry.task_id),
execution.next_item_index);
return false;
}
kind = RepresentationPendingKind::kLegacySerial;
} else if (std::strcmp(entry.task_kind,
LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND) == 0 &&
entry.task_kind_version ==
LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION) {
Lardon3DProjectDbRawDevelopmentBatchTask batch{};
if (lardon3d_project_db_load_raw_development_batch_task(
runtime.state.project_db, entry.task_id, &batch) !=
LARDON3D_PROJECT_DB_OK ||
batch.selected_execution_id != execution.execution_id) {
std::fprintf(stderr,
"RAW batch task %llu does not own selected execution %llu\n",
static_cast<unsigned long long>(entry.task_id),
static_cast<unsigned long long>(execution.execution_id));
return false;
}
kind = RepresentationPendingKind::kBatch;
} else {
std::fprintf(stderr,
"representation recovery refuses pending task %llu of kind %s v%u\n",
static_cast<unsigned long long>(entry.task_id), entry.task_kind,
entry.task_kind_version);
return false;
}
task_id = entry.task_id;
}
if (count < 8) return true;
}
}
bool recover_one_representation_task(Runtime &runtime, uint64_t task_id,
const char *phase) {
Lardon3DProjectRecoverySummary recovery{};
return lardon3d_project_resume_recoverable_tasks(
&runtime.state, lardon3d_task_kind_registry_production(),
&recovery) == LARDON3D_PROJECT_DB_OK &&
recovery.inspected == 1 && recovery.resumed == 1 &&
recovery.failed == 0 && !recovery.queue_full &&
wait_completed(runtime, task_id, phase);
}
enum class FeaturePendingKind { kLegacySerial, kBatch };
bool feature_recovery_kind(const Lardon3DProjectRecoveryEntry &entry,
FeaturePendingKind &kind) {
if ((entry.status != LARDON3D_PROJECT_RECOVERABLE &&
entry.status != LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE) ||
entry.snapshot.recovery_state != TASK_PENDING)
return false;
if (std::strcmp(entry.task_kind, LARDON3D_FEATURE_EXTRACT_TASK_KIND) == 0 &&
entry.task_kind_version == LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION) {
kind = FeaturePendingKind::kLegacySerial;
return true;
}
if (std::strcmp(entry.task_kind, LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND) == 0 &&
entry.task_kind_version == LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION) {
kind = FeaturePendingKind::kBatch;
return true;
}
return false;
}
bool publish_representations(Runtime &runtime, uint64_t execution_id, bool publish_representations(Runtime &runtime, uint64_t execution_id,
std::vector<uint64_t> &image_ids) { std::vector<uint64_t> &image_ids) {
Lardon3DProjectDbSelectedExecution execution{}; for (;;) {
if (lardon3d_project_db_load_selected_execution(runtime.state.project_db, execution_id, Lardon3DProjectDbSelectedExecution execution{};
&execution) != LARDON3D_PROJECT_DB_OK) if (lardon3d_project_db_load_selected_execution(
return false; runtime.state.project_db, execution_id, &execution) !=
for (uint32_t index = execution.next_item_index; index < execution.item_count; ++index) {
Lardon3DProjectDbSelectedExecutionItem item{};
if (lardon3d_project_db_load_selected_execution_item(runtime.state.project_db, execution_id,
index, &item) != LARDON3D_PROJECT_DB_OK)
return false;
uint64_t image_id = 0;
if (item.representation_source == LARDON3D_SELECTED_REPRESENTATION_RAW_ASSET) {
uint64_t task_id = 0;
if (!lardon3d_project_enqueue_raw_development(&runtime.state, item.capture_id,
item.source_asset_id, &task_id) ||
!wait_completed(runtime, task_id, "raw.develop"))
return false;
Lardon3DProjectDbRawDevelopmentTask raw{};
if (lardon3d_project_db_load_raw_development_task(runtime.state.project_db, task_id, &raw) !=
LARDON3D_PROJECT_DB_OK ||
raw.phase != LARDON3D_RAW_DEVELOPMENT_TASK_PUBLISHED || !raw.has_image ||
raw.capture_id != item.capture_id || raw.source_asset_id != item.source_asset_id)
return false;
image_id = raw.image_id;
} else {
if (lardon3d_project_db_get_selected_capture_image(runtime.state.project_db, item.capture_id,
&image_id) != LARDON3D_PROJECT_DB_OK)
return false;
}
if (lardon3d_project_db_record_selected_representation(runtime.state.project_db, execution_id,
index, image_id, index + 1u) !=
LARDON3D_PROJECT_DB_OK) LARDON3D_PROJECT_DB_OK)
return false; return false;
RepresentationPendingKind pending{};
uint64_t pending_task_id = 0;
if (!find_pending_representation_task(runtime, execution, pending,
pending_task_id))
return false;
const RepresentationAction action = choose_representation_action(execution, pending);
if (action == RepresentationAction::kReject) return false;
if (action == RepresentationAction::kCollect) break;
if (action == RepresentationAction::kRecoverLegacySerial) {
const uint32_t item_index = execution.next_item_index;
Lardon3DProjectDbSelectedExecutionItem item{};
if (lardon3d_project_db_load_selected_execution_item(
runtime.state.project_db, execution_id, item_index, &item) !=
LARDON3D_PROJECT_DB_OK ||
!recover_one_representation_task(runtime, pending_task_id,
"raw.develop recovered existing"))
return false;
Lardon3DProjectDbRawDevelopmentTask raw{};
if (lardon3d_project_db_load_raw_development_task(
runtime.state.project_db, pending_task_id, &raw) !=
LARDON3D_PROJECT_DB_OK ||
raw.phase != LARDON3D_RAW_DEVELOPMENT_TASK_PUBLISHED || !raw.has_image ||
raw.capture_id != item.capture_id ||
raw.source_asset_id != item.source_asset_id ||
lardon3d_project_db_record_selected_representation(
runtime.state.project_db, execution_id, item_index, raw.image_id,
item_index + 1u) != LARDON3D_PROJECT_DB_OK)
return false;
/* The legacy Task predates the selected-execution batch owner. Its exact
* current item is committed once before the batch reads the advanced
* cursor; no completed selected prefix is replayed or re-identified. */
continue;
}
if (action == RepresentationAction::kRecoverBatch) {
if (!recover_one_representation_task(runtime, pending_task_id,
"raw.develop.batch recovered existing"))
return false;
continue;
}
uint64_t batch_task_id = 0;
if (!lardon3d_project_enqueue_raw_development_batch(
&runtime.state, execution_id, &batch_task_id) ||
!wait_completed(runtime, batch_task_id, "raw.develop.batch"))
return false;
} }
if (lardon3d_project_db_load_selected_execution(runtime.state.project_db, execution_id, image_ids.clear();
&execution) != LARDON3D_PROJECT_DB_OK || Lardon3DProjectDbSelectedExecution execution{};
if (lardon3d_project_db_load_selected_execution(runtime.state.project_db,
execution_id, &execution) !=
LARDON3D_PROJECT_DB_OK ||
execution.stage != LARDON3D_SELECTED_EXECUTION_CALIBRATION) execution.stage != LARDON3D_SELECTED_EXECUTION_CALIBRATION)
return false; return false;
image_ids.clear();
for (uint32_t index = 0; index < execution.item_count; ++index) { for (uint32_t index = 0; index < execution.item_count; ++index) {
Lardon3DProjectDbSelectedExecutionItem item{}; Lardon3DProjectDbSelectedExecutionItem item{};
if (lardon3d_project_db_load_selected_execution_item(runtime.state.project_db, execution_id, if (lardon3d_project_db_load_selected_execution_item(runtime.state.project_db, execution_id,
@ -1491,30 +1723,100 @@ bool publish_representations(Runtime &runtime, uint64_t execution_id,
return true; return true;
} }
bool extract_features(Runtime &runtime, const std::vector<uint64_t> &image_ids, bool extract_features(Runtime &runtime, uint64_t selected_execution_id,
const std::vector<uint64_t> &image_ids,
const Lardon3DFeatureExtractorParameters &parameters, const Lardon3DFeatureExtractorParameters &parameters,
std::vector<Lardon3DProjectDbFeatureSet> &feature_sets) { std::vector<Lardon3DProjectDbFeatureSet> &feature_sets) {
unsigned char fingerprint[32]{}; unsigned char fingerprint[32]{};
lardon3d_feature_extractor_parameter_fingerprint(&parameters, fingerprint); lardon3d_feature_extractor_parameter_fingerprint(&parameters, fingerprint);
feature_sets.clear(); feature_sets.clear();
bool complete = true;
for (const uint64_t image_id : image_ids) { for (const uint64_t image_id : image_ids) {
Lardon3DProjectDbFeatureSet feature{}; Lardon3DProjectDbFeatureSet feature{};
if (lardon3d_project_db_find_feature_set(runtime.state.project_db, image_id, if (lardon3d_project_db_find_feature_set(runtime.state.project_db, image_id,
LARDON3D_FEATURE_EXTRACTOR_KIND, LARDON3D_FEATURE_EXTRACTOR_KIND,
LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint,
&feature) != LARDON3D_PROJECT_DB_OK) { &feature) != LARDON3D_PROJECT_DB_OK) {
uint64_t task_id = 0; complete = false;
if (!lardon3d_project_enqueue_feature_extract(&runtime.state, image_id, &parameters, break;
&task_id) ||
!wait_completed(runtime, task_id, "features.extract") ||
lardon3d_project_db_find_feature_set(
runtime.state.project_db, image_id, LARDON3D_FEATURE_EXTRACTOR_KIND,
LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, &feature) !=
LARDON3D_PROJECT_DB_OK)
return false;
} }
feature_sets.push_back(feature); feature_sets.push_back(feature);
} }
if (complete) return true;
/* A pre-v25 single-image Task is historical durable work, not a batch
* cursor. Reopening creates an empty Queue, so restore and finish that exact
* Task once before the new owner starts its selected suffix. Never rewrite or
* replace the legacy row (Task 166 in the inspected A6000 project). */
uint64_t pending_task_id = 0;
FeaturePendingKind pending_kind{};
uint64_t cursor = 0;
for (;;) {
Lardon3DProjectRecoveryEntry tasks[8]{};
size_t count = 0;
if (lardon3d_project_list_recoverable(
&runtime.state, lardon3d_task_kind_registry_production(), cursor,
tasks, 8, &count) !=
LARDON3D_PROJECT_DB_OK)
return false;
for (size_t index = 0; index < count; ++index) {
cursor = tasks[index].task_id;
FeaturePendingKind kind{};
if (pending_task_id != 0 || !feature_recovery_kind(tasks[index], kind)) {
/* Registry recovery resumes every pending Task. Refuse ambiguity or an
* unrelated pipeline owner before transferring anything to Queue. */
std::fprintf(stderr,
"feature recovery refuses pending task %llu of kind %s v%u\n",
static_cast<unsigned long long>(tasks[index].task_id),
tasks[index].task_kind, tasks[index].task_kind_version);
return false;
}
if (kind == FeaturePendingKind::kLegacySerial) {
Lardon3DProjectDbFeatureExtractTask legacy{};
if (lardon3d_project_db_load_feature_extract_task(
runtime.state.project_db, tasks[index].task_id, &legacy) !=
LARDON3D_PROJECT_DB_OK ||
std::memcmp(legacy.parameter_fingerprint, fingerprint, 32) != 0)
return false;
} else {
Lardon3DProjectDbFeatureExtractBatchTask batch{};
if (lardon3d_project_db_load_feature_extract_batch_task(
runtime.state.project_db, tasks[index].task_id, &batch) !=
LARDON3D_PROJECT_DB_OK ||
batch.selected_execution_id != selected_execution_id ||
std::memcmp(batch.parameter_fingerprint, fingerprint, 32) != 0)
return false;
}
pending_task_id = tasks[index].task_id;
pending_kind = kind;
}
if (count < 8) break;
}
if (pending_task_id != 0) {
const char *phase = pending_kind == FeaturePendingKind::kLegacySerial
? "features.extract recovered legacy"
: "features.extract.batch recovered";
if (!recover_one_representation_task(runtime, pending_task_id, phase))
return false;
}
uint64_t batch_task_id = 0;
if (pending_task_id == 0 || pending_kind == FeaturePendingKind::kLegacySerial) {
if (!lardon3d_project_enqueue_feature_extract_batch(
&runtime.state, selected_execution_id, &parameters, &batch_task_id) ||
!wait_completed(runtime, batch_task_id, "features.extract.batch")) {
return false;
}
}
feature_sets.clear();
for (const uint64_t image_id : image_ids) {
Lardon3DProjectDbFeatureSet feature{};
if (lardon3d_project_db_find_feature_set(
runtime.state.project_db, image_id, LARDON3D_FEATURE_EXTRACTOR_KIND,
LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, &feature) !=
LARDON3D_PROJECT_DB_OK)
return false;
feature_sets.push_back(feature);
}
return true; return true;
} }
@ -2724,7 +3026,8 @@ int main(int argc, char **argv) {
} }
Runtime runtime; Runtime runtime;
if (options.resume_geometry_existing || options.resume_pre_gv_existing || if (options.resume_geometry_existing || options.resume_pre_gv_existing ||
options.resume_candidate_existing) { options.resume_candidate_existing ||
options.resume_representations_existing) {
if (!prepare_existing_project(options.project_dir, runtime)) { if (!prepare_existing_project(options.project_dir, runtime)) {
std::fprintf(stderr, "existing project must be absolute, normalized, and durable\n"); std::fprintf(stderr, "existing project must be absolute, normalized, and durable\n");
return 2; return 2;
@ -2741,43 +3044,80 @@ int main(int argc, char **argv) {
runtime.stop_after_gv = options.stop_after_gv; runtime.stop_after_gv = options.stop_after_gv;
if (options.resume_candidate_existing) if (options.resume_candidate_existing)
return run_existing_candidate(runtime) ? 0 : 1; return run_existing_candidate(runtime) ? 0 : 1;
return (options.resume_pre_gv_existing ? run_existing_pre_gv(runtime) if (options.resume_pre_gv_existing || options.resume_geometry_existing)
: run_existing_geometry(runtime)) ? 0 : 1; return (options.resume_pre_gv_existing ? run_existing_pre_gv(runtime)
: run_existing_geometry(runtime)) ? 0 : 1;
} }
if (!prepare_empty_project(options.project_dir, runtime.project_path)) {
std::fprintf(stderr, "project directory must be absolute, normalized and empty\n");
return 2;
}
runtime.database_path = (options.project_dir / "project.lardon3d").string();
runtime.matcher_needed = true;
Campaign campaign;
if (!discover_campaign(options, campaign) || !start_runtime(runtime)) {
std::fprintf(stderr, "campaign discovery or runtime setup failed\n");
return 1;
}
const char *mode_name = options.mode == Mode::kA6000 ? "a6000" : "s21";
Lardon3DProjectDbScanSet scanset{};
if (lardon3d_project_db_create_scanset(runtime.state.project_db, mode_name, &scanset) !=
LARDON3D_PROJECT_DB_OK) {
stop_runtime(runtime);
return 1;
}
uint64_t quality_task_id = 0;
uint64_t campaign_task_id = 0;
Lardon3DProjectDbSelectedExecution execution{}; Lardon3DProjectDbSelectedExecution execution{};
bool ok = run_task_pair(runtime, scanset.scanset_id, campaign, const char *mode_name = options.mode == Mode::kA6000 ? "a6000" : "s21";
options.restart != RestartBoundary::kNone, quality_task_id, bool ok = true;
campaign_task_id) && if (options.resume_representations_existing) {
create_selection(runtime, options.mode, quality_task_id, campaign_task_id, runtime.matcher_needed = !options.stop_after_representations;
campaign.confirmations.size(), execution); if (!start_runtime(runtime) ||
if (!ok || execution.execution_id == 0) { lardon3d_project_db_load_selected_execution(
stop_runtime(runtime); runtime.state.project_db, options.selected_execution_id,
return ok ? 0 : 1; &execution) != LARDON3D_PROJECT_DB_OK ||
(execution.stage != LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS &&
execution.stage != LARDON3D_SELECTED_EXECUTION_CALIBRATION)) {
std::fprintf(stderr,
"existing selected execution is unavailable or past representations\n");
stop_runtime(runtime);
return 1;
}
mode_name = "existing";
} else {
if (!prepare_empty_project(options.project_dir, runtime.project_path)) {
std::fprintf(stderr,
"project directory must be absolute, normalized and empty\n");
return 2;
}
runtime.database_path = (options.project_dir / "project.lardon3d").string();
runtime.matcher_needed = true;
Campaign campaign;
if (!discover_campaign(options, campaign) || !start_runtime(runtime)) {
std::fprintf(stderr, "campaign discovery or runtime setup failed\n");
return 1;
}
Lardon3DProjectDbScanSet scanset{};
if (lardon3d_project_db_create_scanset(runtime.state.project_db, mode_name,
&scanset) != LARDON3D_PROJECT_DB_OK) {
stop_runtime(runtime);
return 1;
}
uint64_t quality_task_id = 0;
uint64_t campaign_task_id = 0;
ok = run_task_pair(runtime, scanset.scanset_id, campaign,
options.restart != RestartBoundary::kNone,
quality_task_id, campaign_task_id) &&
create_selection(runtime, options.mode, quality_task_id,
campaign_task_id, campaign.confirmations.size(),
execution);
if (!ok || execution.execution_id == 0) {
stop_runtime(runtime);
return ok ? 0 : 1;
}
} }
std::vector<uint64_t> image_ids; std::vector<uint64_t> image_ids;
ok = publish_representations(runtime, execution.execution_id, image_ids); ok = publish_representations(runtime, execution.execution_id, image_ids);
if (ok && options.stop_after_representations) {
ok = lardon3d_project_db_load_selected_execution(
runtime.state.project_db, execution.execution_id, &execution) ==
LARDON3D_PROJECT_DB_OK &&
execution.stage == LARDON3D_SELECTED_EXECUTION_CALIBRATION &&
execution.next_item_index == execution.item_count;
}
if (ok && options.stop_after_representations) {
std::printf(
"{\"record\":\"existing_representations_summary\",\"ok\":true,"
"\"execution_id\":%llu,\"next_item_index\":%u,\"item_count\":%u,"
"\"stage\":\"CALIBRATION\",\"selected_prefix_replayed\":false,"
"\"features_enqueued\":false,\"sparse_sfm_run\":false}\n",
static_cast<unsigned long long>(execution.execution_id),
execution.next_item_index, execution.item_count);
stop_runtime(runtime);
return 0;
}
if (ok && options.restart == RestartBoundary::kRepresentations) if (ok && options.restart == RestartBoundary::kRepresentations)
ok = restart_runtime(runtime, execution.execution_id, "representations") && ok = restart_runtime(runtime, execution.execution_id, "representations") &&
publish_representations(runtime, execution.execution_id, image_ids); publish_representations(runtime, execution.execution_id, image_ids);
@ -2787,11 +3127,11 @@ int main(int argc, char **argv) {
* campaign-size rule; callers use one empty project per campaign. */ * campaign-size rule; callers use one empty project per campaign. */
const Lardon3DFeatureExtractorParameters orb{8192, 8, 20}; const Lardon3DFeatureExtractorParameters orb{8192, 8, 20};
std::vector<Lardon3DProjectDbFeatureSet> features; std::vector<Lardon3DProjectDbFeatureSet> features;
if (ok) ok = extract_features(runtime, image_ids, orb, features); if (ok) ok = extract_features(runtime, execution.execution_id, image_ids, orb, features);
if (ok && options.restart == RestartBoundary::kFeatures) { if (ok && options.restart == RestartBoundary::kFeatures) {
ok = restart_runtime(runtime, execution.execution_id, "features") && ok = restart_runtime(runtime, execution.execution_id, "features") &&
publish_representations(runtime, execution.execution_id, image_ids) && publish_representations(runtime, execution.execution_id, image_ids) &&
extract_features(runtime, image_ids, orb, features); extract_features(runtime, execution.execution_id, image_ids, orb, features);
} }
uint64_t geometry_task_id = 0; uint64_t geometry_task_id = 0;

View file

@ -151,6 +151,13 @@ typedef struct {
Lardon3DExtractedFeatures output; Lardon3DExtractedFeatures output;
} AdaptiveExtractWork; } AdaptiveExtractWork;
bool lardon3d_feature_extract_batch_test_run(Lardon3DTask *task);
static bool batch_test_callback(Lardon3DTask *task, void *userdata) {
(void)userdata;
return lardon3d_feature_extract_batch_test_run(task);
}
static bool adaptive_extract_callback(Lardon3DTask *task, void *userdata) { static bool adaptive_extract_callback(Lardon3DTask *task, void *userdata) {
AdaptiveExtractWork *work = userdata; AdaptiveExtractWork *work = userdata;
Lardon3DOpenCvTaskThreadControl control; Lardon3DOpenCvTaskThreadControl control;
@ -265,6 +272,92 @@ static bool deterministic_capture_preserves_load_gate(void) {
return ok; return ok;
} }
static bool feature_batch_trials_exercise_participants(void) {
Lardon3DHardwareProfile profile = {
.logical_cpu_count = 8,
.page_size_bytes = 4096,
.memory_total_bytes = UINT64_C(16) * 1024 * 1024 * 1024,
.cpu_architecture = "test",
};
Lardon3DResourcePolicy policy = {
.system_cpu_reserve = 4,
.maximum_cpu_load_ratio = 1.0,
.maximum_io_pressure_avg10 = 100.0,
.io_slot_capacity = 1,
};
Lardon3DResourceGovernor *governor =
lardon3d_resource_governor_create(&profile, &policy);
Lardon3DResourceSnapshot resources = deterministic_resource_snapshot(0.0);
Lardon3DTaskCapabilityEnvelope envelope = {
.count = 1,
.capabilities = {{
.estimate = {
.memory_fixed_bytes = 64ULL * 1024 * 1024,
.memory_bytes_per_item = 512ULL * 1024 * 1024,
.minimum_batch_size = 1,
.maximum_batch_size = 12,
.desired_cpu_threads = 12,
.desired_io_slots = 1,
.task_class = LARDON3D_RESOURCE_TASK_CPU,
},
.backend = LARDON3D_RESOURCE_BACKEND_FIXED,
.inflight_limit = 1,
.cpu_reducible = true,
.batch_adaptive = true,
.cpu_batch_coupled = true,
}},
};
bool ok = governor &&
lardon3d_resource_governor_internal_set_capture_snapshot(governor, &resources);
for (unsigned int sample = 0; sample < 2 && ok; ++sample) {
Lardon3DResourceCapabilitySelection selection;
Lardon3DResourceReservation *reservation = NULL;
ok = lardon3d_resource_governor_internal_reserve_capability_available(
governor, LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND, 1,
&envelope, &selection, &reservation) && reservation &&
selection.decision.cpu_threads == 1 && selection.decision.batch_size == 1 &&
lardon3d_resource_governor_internal_record_sequence(
governor, LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND, 1,
&selection, UINT64_C(1000000000), 1) &&
lardon3d_resource_governor_release(governor, reservation);
}
Lardon3DResourceCapabilitySelection trial;
Lardon3DResourceReservation *trial_reservation = NULL;
ok = ok && lardon3d_resource_governor_internal_reserve_capability_available(
governor, LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND, 1,
&envelope, &trial, &trial_reservation) && trial_reservation &&
trial.decision.cpu_threads == 2 && trial.decision.batch_size == 2;
if (trial_reservation)
ok = lardon3d_resource_governor_release(governor, trial_reservation) && ok;
/* Candidate uses the same cross-item coupling: CPU2 with batch1 still has
* only one membership to query. Its independent feedback key must therefore
* open a measurable CPU2/batch2 rung as well. */
for (unsigned int sample = 0; sample < 2 && ok; ++sample) {
Lardon3DResourceCapabilitySelection selection;
Lardon3DResourceReservation *reservation = NULL;
ok = lardon3d_resource_governor_internal_reserve_capability_available(
governor, "candidate_pair.generate", 1, &envelope, &selection,
&reservation) && reservation &&
selection.decision.cpu_threads == 1 && selection.decision.batch_size == 1 &&
lardon3d_resource_governor_internal_record_sequence(
governor, "candidate_pair.generate", 1, &selection,
UINT64_C(1000000000), 1) &&
lardon3d_resource_governor_release(governor, reservation);
}
trial_reservation = NULL;
ok = ok && lardon3d_resource_governor_internal_reserve_capability_available(
governor, "candidate_pair.generate", 1, &envelope, &trial,
&trial_reservation) && trial_reservation &&
trial.decision.cpu_threads == 2 && trial.decision.batch_size == 2;
if (trial_reservation)
ok = lardon3d_resource_governor_release(governor, trial_reservation) && ok;
if (governor)
ok = lardon3d_resource_governor_internal_set_capture_snapshot(governor, NULL) && ok;
lardon3d_resource_governor_destroy(governor);
return ok;
}
static bool run_adaptive_output_equivalence(const char *path) { static bool run_adaptive_output_equivalence(const char *path) {
cpu_set_t allowed_mask; cpu_set_t allowed_mask;
CPU_ZERO(&allowed_mask); CPU_ZERO(&allowed_mask);
@ -466,6 +559,7 @@ static void *consolidate_thread(void *userdata) {
} }
static bool run_test(void) { static bool run_test(void) {
CHECK(feature_batch_trials_exercise_participants());
char root[] = "/tmp/lardon3d-feature-task-XXXXXX"; char root[] = "/tmp/lardon3d-feature-task-XXXXXX";
CHECK(deterministic_capture_preserves_load_gate()); CHECK(deterministic_capture_preserves_load_gate());
CHECK(mkdtemp(root) && setenv("LARDON3D_PROJECTS_ROOT", root, 1) == 0); CHECK(mkdtemp(root) && setenv("LARDON3D_PROJECTS_ROOT", root, 1) == 0);
@ -557,31 +651,61 @@ static bool run_test(void) {
* pool. Feature, SIFT, and RootSIFT must each restore the captured value. */ * pool. Feature, SIFT, and RootSIFT must each restore the captured value. */
CHECK(lardon3d_feature_opencv_configure_threads(3)); CHECK(lardon3d_feature_opencv_configure_threads(3));
CHECK(setenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS", "1", 1) == 0); CHECK(setenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS", "1", 1) == 0);
/* Batch CPU1 configuration deliberately fails after mutating OpenCV. The
* callback must restore the prior process-global value before Queue can run
* any later Task. */
Lardon3DResourceEstimate failed_batch_estimate = {
.minimum_batch_size = 1,
.maximum_batch_size = 1,
.desired_cpu_threads = 1,
.task_class = LARDON3D_RESOURCE_TASK_CPU,
};
Lardon3DTask *failed_batch = lardon3d_task_create_typed(
"Feature batch restoration", &failed_batch_estimate,
"features.extract.batch.restore-test",
LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION, batch_test_callback,
NULL, NULL);
uint64_t failed_batch_task = 0;
/* This synthetic Task shares the project Queue with durable Tasks. Reserve
* its ID through Project DB so the next durable enqueue cannot reuse it. */
CHECK(failed_batch &&
lardon3d_project_db_allocate_task_id(state.project_db,
&failed_batch_task) ==
LARDON3D_PROJECT_DB_OK &&
lardon3d_task_assign_id(failed_batch, failed_batch_task) &&
lardon3d_task_queue_add(state.task_queue, failed_batch,
NULL) &&
wait_state(state.task_queue, failed_batch_task, TASK_FAILED,
&snapshot) &&
lardon3d_feature_opencv_thread_count() == 3 &&
unsetenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS") == 0);
Lardon3DFeatureExtractorParameters failed_orb_parameters = {509, 4, 10}; Lardon3DFeatureExtractorParameters failed_orb_parameters = {509, 4, 10};
uint64_t failed_orb_task = 0; uint64_t failed_orb_task = 0;
CHECK(setenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS", "once", 1) == 0);
CHECK(lardon3d_project_enqueue_feature_extract( CHECK(lardon3d_project_enqueue_feature_extract(
&state, image.image_id, &failed_orb_parameters, &failed_orb_task) && &state, image.image_id, &failed_orb_parameters, &failed_orb_task));
wait_state(state.task_queue, failed_orb_task, TASK_FAILED, &snapshot) && CHECK(wait_state(state.task_queue, failed_orb_task, TASK_FAILED, &snapshot));
lardon3d_feature_opencv_thread_count() == 3); CHECK(lardon3d_feature_opencv_thread_count() == 3);
Lardon3DSiftExtractorParameters failed_sift_parameters = Lardon3DSiftExtractorParameters failed_sift_parameters =
lardon3d_sift_precision_classic_v1(false); lardon3d_sift_precision_classic_v1(false);
failed_sift_parameters.max_features = 511; failed_sift_parameters.max_features = 511;
uint64_t failed_sift_task = 0; uint64_t failed_sift_task = 0;
CHECK(setenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS", "once", 1) == 0);
CHECK(lardon3d_project_enqueue_sift_extract( CHECK(lardon3d_project_enqueue_sift_extract(
&state, image.image_id, &failed_sift_parameters, &state, image.image_id, &failed_sift_parameters,
&failed_sift_task) && &failed_sift_task));
wait_state(state.task_queue, failed_sift_task, TASK_FAILED, &snapshot) && CHECK(wait_state(state.task_queue, failed_sift_task, TASK_FAILED, &snapshot));
lardon3d_feature_opencv_thread_count() == 3); CHECK(lardon3d_feature_opencv_thread_count() == 3);
failed_sift_parameters.rootsift = true; failed_sift_parameters.rootsift = true;
failed_sift_parameters.max_features = 510; failed_sift_parameters.max_features = 510;
uint64_t failed_rootsift_task = 0; uint64_t failed_rootsift_task = 0;
CHECK(setenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS", "once", 1) == 0);
CHECK(lardon3d_project_enqueue_sift_extract( CHECK(lardon3d_project_enqueue_sift_extract(
&state, image.image_id, &failed_sift_parameters, &state, image.image_id, &failed_sift_parameters,
&failed_rootsift_task) && &failed_rootsift_task));
wait_state(state.task_queue, failed_rootsift_task, TASK_FAILED, &snapshot) && CHECK(wait_state(state.task_queue, failed_rootsift_task, TASK_FAILED, &snapshot));
lardon3d_feature_opencv_thread_count() == 3 && CHECK(lardon3d_feature_opencv_thread_count() == 3);
unsetenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS") == 0);
Lardon3DSiftExtractorParameters sift_parameters = lardon3d_sift_precision_classic_v1(false); Lardon3DSiftExtractorParameters sift_parameters = lardon3d_sift_precision_classic_v1(false);
sift_parameters.max_features = 512; sift_parameters.max_features = 512;

View file

@ -172,6 +172,8 @@ static bool create_v11_database(const char *path) {
lardon3d_project_db_close(database); lardon3d_project_db_close(database);
return execute_sql(path, return execute_sql(path,
"PRAGMA foreign_keys=OFF;BEGIN IMMEDIATE;" "PRAGMA foreign_keys=OFF;BEGIN IMMEDIATE;"
"DROP TABLE IF EXISTS feature_extract_batch_tasks;"
"DROP TABLE IF EXISTS raw_development_batch_tasks;"
"DROP TABLE IF EXISTS capture_calibration_selections;" "DROP TABLE IF EXISTS capture_calibration_selections;"
"DROP TABLE IF EXISTS optical_calibration_profiles;" "DROP TABLE IF EXISTS optical_calibration_profiles;"
"DROP TABLE IF EXISTS capture_optical_configurations;" "DROP TABLE IF EXISTS capture_optical_configurations;"

View file

@ -878,6 +878,8 @@ static bool run_task_test(void) {
CHECK(exec_sql( CHECK(exec_sql(
database_path, database_path,
"PRAGMA foreign_keys=OFF;BEGIN IMMEDIATE;" "PRAGMA foreign_keys=OFF;BEGIN IMMEDIATE;"
"DROP TABLE IF EXISTS feature_extract_batch_tasks;"
"DROP TABLE IF EXISTS raw_development_batch_tasks;"
"DROP TABLE IF EXISTS capture_calibration_selections;" "DROP TABLE IF EXISTS capture_calibration_selections;"
"DROP TABLE IF EXISTS optical_calibration_profiles;" "DROP TABLE IF EXISTS optical_calibration_profiles;"
"DROP TABLE IF EXISTS capture_optical_configurations;" "DROP TABLE IF EXISTS capture_optical_configurations;"

View file

@ -252,7 +252,8 @@ static bool test_profiles_assignments_and_calibrations(void) {
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]; char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY];
CHECK(lardon3d_project_db_open(path, &database, error) == CHECK(lardon3d_project_db_open(path, &database, error) ==
LARDON3D_PROJECT_DB_OK); LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_schema_version(database) == 23); CHECK(lardon3d_project_db_schema_version(database) ==
LARDON3D_PROJECT_DB_SCHEMA_VERSION);
Lardon3DProjectDbScanSet scanset; Lardon3DProjectDbScanSet scanset;
CHECK(lardon3d_project_db_create_scanset(database, "mixed optical campaign", CHECK(lardon3d_project_db_create_scanset(database, "mixed optical campaign",
@ -1146,6 +1147,7 @@ static bool downgrade_to_v22_fixture(const char *path) {
return raw_sql( return raw_sql(
path, path,
"PRAGMA foreign_keys=OFF;BEGIN IMMEDIATE;" "PRAGMA foreign_keys=OFF;BEGIN IMMEDIATE;"
"DROP TABLE raw_development_batch_tasks;"
"DROP TABLE capture_calibration_selections;" "DROP TABLE capture_calibration_selections;"
"DROP TABLE optical_calibration_profiles;" "DROP TABLE optical_calibration_profiles;"
"DROP TABLE capture_optical_configurations;" "DROP TABLE capture_optical_configurations;"
@ -1195,7 +1197,8 @@ static bool test_migration_rollback_retry_and_equivalence(void) {
CHECK(lardon3d_project_db_open(migrated_path, &database, error) == CHECK(lardon3d_project_db_open(migrated_path, &database, error) ==
LARDON3D_PROJECT_DB_OK && LARDON3D_PROJECT_DB_OK &&
lardon3d_project_db_schema_version(database) == 23); lardon3d_project_db_schema_version(database) ==
LARDON3D_PROJECT_DB_SCHEMA_VERSION);
Lardon3DOpticalCaptureAssignment migrated_unresolved; Lardon3DOpticalCaptureAssignment migrated_unresolved;
CHECK(lardon3d_optical_capture_assignment_load(database, 1, CHECK(lardon3d_optical_capture_assignment_load(database, 1,
&migrated_unresolved) == &migrated_unresolved) ==

View file

@ -365,7 +365,89 @@ int main() {
recovery_entry.task_kind_version = 1; recovery_entry.task_kind_version = 1;
CHECK(!geometry_recovery_kind(recovery_entry, recovery_kind)); CHECK(!geometry_recovery_kind(recovery_entry, recovery_kind));
Lardon3DProjectDbSelectedExecution selected{};
selected.execution_id = 9;
selected.stage = LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS;
selected.next_item_index = 72;
selected.item_count = 100;
/* A legacy serial owner is completed only at the exact durable cursor. The
* subsequent no-owner decision hands the untouched suffix to one batch. */
CHECK(choose_representation_action(
selected, RepresentationPendingKind::kLegacySerial) ==
RepresentationAction::kRecoverLegacySerial);
selected.next_item_index = 73;
CHECK(choose_representation_action(selected, RepresentationPendingKind::kNone) ==
RepresentationAction::kEnqueueBatch);
selected.next_item_index = 0;
CHECK(choose_representation_action(selected, RepresentationPendingKind::kNone) ==
RepresentationAction::kEnqueueBatch);
selected.next_item_index = selected.item_count;
selected.stage = LARDON3D_SELECTED_EXECUTION_CALIBRATION;
CHECK(choose_representation_action(selected, RepresentationPendingKind::kNone) ==
RepresentationAction::kCollect);
CHECK(choose_representation_action(selected, RepresentationPendingKind::kBatch) ==
RepresentationAction::kRecoverBatch);
Lardon3DProjectRecoveryEntry feature_pending{};
feature_pending.status = LARDON3D_PROJECT_RECOVERABLE;
feature_pending.snapshot.recovery_state = TASK_PENDING;
std::snprintf(feature_pending.task_kind, sizeof(feature_pending.task_kind),
"%s", LARDON3D_FEATURE_EXTRACT_TASK_KIND);
feature_pending.task_kind_version = LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION;
FeaturePendingKind feature_kind{};
CHECK(feature_recovery_kind(feature_pending, feature_kind));
CHECK(feature_kind == FeaturePendingKind::kLegacySerial);
/* Completed representations defer exact Feature owners to extract_features,
* which performs the scientific-domain and selected-execution checks. */
CHECK(representation_defers_feature_task(selected, feature_pending));
selected.stage = LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS;
CHECK(!representation_defers_feature_task(selected, feature_pending));
selected.stage = LARDON3D_SELECTED_EXECUTION_CALIBRATION;
--selected.next_item_index;
CHECK(!representation_defers_feature_task(selected, feature_pending));
selected.next_item_index = selected.item_count;
++feature_pending.task_kind_version;
CHECK(!representation_defers_feature_task(selected, feature_pending));
std::snprintf(feature_pending.task_kind, sizeof(feature_pending.task_kind),
"%s", LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND);
feature_pending.task_kind_version =
LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION;
CHECK(representation_defers_feature_task(selected, feature_pending));
CHECK(feature_recovery_kind(feature_pending, feature_kind));
CHECK(feature_kind == FeaturePendingKind::kBatch);
feature_pending.status = LARDON3D_PROJECT_RECOVERY_UNKNOWN_TASK_KIND;
CHECK(!feature_recovery_kind(feature_pending, feature_kind));
feature_pending.status = LARDON3D_PROJECT_RECOVERABLE;
std::snprintf(feature_pending.task_kind, sizeof(feature_pending.task_kind),
"%s", LARDON3D_TRACK_BUILDER_TASK_KIND);
feature_pending.task_kind_version = LARDON3D_TRACK_BUILDER_TASK_KIND_VERSION;
CHECK(!representation_defers_feature_task(selected, feature_pending));
CHECK(!feature_recovery_kind(feature_pending, feature_kind));
Options options; Options options;
CHECK(parse_case({"runner", "--resume-representations-existing",
"--project-dir", "/tmp/not-opened",
"--selected-execution-id", "9",
"--stop-after-representations"}, options));
CHECK(options.resume_representations_existing &&
options.selected_execution_id == 9 &&
options.has_selected_execution_id && options.stop_after_representations);
for (const char *invalid_id : {"0", "-1", "+1", "01", "1x"}) {
options = {};
CHECK(!parse_case({"runner", "--resume-representations-existing",
"--project-dir", "/tmp/not-opened",
"--selected-execution-id", invalid_id}, options));
}
options = {};
CHECK(!parse_case({"runner", "--resume-representations-existing",
"--project-dir", "/tmp/not-opened"}, options));
options = {};
CHECK(!parse_case({"runner", "--resume-representations-existing",
"--project-dir", "/tmp/not-opened",
"--selected-execution-id", "9", "--cpu-budget", "2"},
options));
options = {};
CHECK(parse_case({"runner", "--resume-pre-gv-existing", "--project-dir", CHECK(parse_case({"runner", "--resume-pre-gv-existing", "--project-dir",
"/tmp/not-opened"}, options)); "/tmp/not-opened"}, options));
CHECK(options.matcher_mode == MatcherMode::kAuto && CHECK(options.matcher_mode == MatcherMode::kAuto &&

View file

@ -22,9 +22,11 @@
} while (0) } while (0)
/* Historical migration fixtures are produced from a temporary current DB. /* Historical migration fixtures are produced from a temporary current DB.
* Remove every additive v23 object first so the fixture really is historical * Remove every additive v23/v24/v25 object first so the fixture really is historical
* rather than a lower version number with future tables left behind. */ * rather than a lower version number with future tables left behind. */
#define DROP_OPTICAL_V23_SQL \ #define DROP_OPTICAL_V23_SQL \
"DROP TABLE IF EXISTS feature_extract_batch_tasks;" \
"DROP TABLE IF EXISTS raw_development_batch_tasks;" \
"DROP TABLE IF EXISTS capture_calibration_selections;" \ "DROP TABLE IF EXISTS capture_calibration_selections;" \
"DROP TABLE IF EXISTS optical_calibration_profiles;" \ "DROP TABLE IF EXISTS optical_calibration_profiles;" \
"DROP TABLE IF EXISTS capture_optical_configurations;" \ "DROP TABLE IF EXISTS capture_optical_configurations;" \
@ -131,7 +133,7 @@ static bool create_future_database(const char *path) {
sqlite3_exec( sqlite3_exec(
connection, connection,
"CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);" "CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);"
"INSERT INTO metadata VALUES('schema_version',24);", "INSERT INTO metadata VALUES('schema_version',26);",
NULL, NULL, NULL) == SQLITE_OK; NULL, NULL, NULL) == SQLITE_OK;
return sqlite3_close(connection) == SQLITE_OK && ok; return sqlite3_close(connection) == SQLITE_OK && ok;
} }
@ -907,6 +909,7 @@ static bool run_test(void) {
char v13_path[512], true_v14_path[512], failed_v14_path[512], char v13_path[512], true_v14_path[512], failed_v14_path[512],
failed_v15_path[512], true_v15_path[512], v17_path[512], failed_v15_path[512], true_v15_path[512], v17_path[512],
failed_v18_path[512], failed_v19_path[512], raw_missing_table_path[512]; failed_v18_path[512], failed_v19_path[512], raw_missing_table_path[512];
char v24_path[512];
CHECK(snprintf(database_path, sizeof(database_path), "%s/project.db", CHECK(snprintf(database_path, sizeof(database_path), "%s/project.db",
directory) > 0); directory) > 0);
CHECK(snprintf(artifact_path, sizeof(artifact_path), "%s/artifact.bin", CHECK(snprintf(artifact_path, sizeof(artifact_path), "%s/artifact.bin",
@ -956,9 +959,34 @@ static bool run_test(void) {
"%s/failed-v19.db", directory) > 0); "%s/failed-v19.db", directory) > 0);
CHECK(snprintf(raw_missing_table_path, sizeof(raw_missing_table_path), CHECK(snprintf(raw_missing_table_path, sizeof(raw_missing_table_path),
"%s/raw-missing-table.db", directory) > 0); "%s/raw-missing-table.db", directory) > 0);
CHECK(snprintf(v24_path, sizeof(v24_path), "%s/v24.db", directory) > 0);
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]; char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY];
Lardon3DProjectDb *database = NULL; Lardon3DProjectDb *database = NULL;
/* v25 is DDL-only: an injected failure rolls back table and marker, and a
* retry creates no fabricated association or historical cursor. */
CHECK(lardon3d_project_db_open(v24_path, &database, error) == LARDON3D_PROJECT_DB_OK);
lardon3d_project_db_close(database);
database = NULL;
CHECK(execute_test_sql(v24_path,
"DROP TABLE feature_extract_batch_tasks;"
"UPDATE metadata SET value=24 WHERE key='schema_version';"));
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V25", "1", 1) == 0);
CHECK(lardon3d_project_db_open(v24_path, &database, error) != LARDON3D_PROJECT_DB_OK);
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V25") == 0);
CHECK(query_integer(v24_path, "SELECT value FROM metadata WHERE key='schema_version'", 24));
CHECK(query_integer(v24_path,
"SELECT count(*) FROM sqlite_master WHERE type='table' AND "
"name='feature_extract_batch_tasks'", 0));
database = NULL;
Lardon3DProjectDbResult v25_retry = lardon3d_project_db_open(v24_path, &database, error);
if (v25_retry != LARDON3D_PROJECT_DB_OK)
fprintf(stderr, "v25 retry: %d %s\n", v25_retry, error);
CHECK(v25_retry == LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_schema_version(database) == LARDON3D_PROJECT_DB_SCHEMA_VERSION);
lardon3d_project_db_close(database);
database = NULL;
CHECK(query_integer(v24_path, "SELECT count(*) FROM feature_extract_batch_tasks", 0));
CHECK(lardon3d_project_db_open(database_path, &database, error) == CHECK(lardon3d_project_db_open(database_path, &database, error) ==
LARDON3D_PROJECT_DB_OK); LARDON3D_PROJECT_DB_OK);
CHECK(database && lardon3d_project_db_schema_version(database) == CHECK(database && lardon3d_project_db_schema_version(database) ==
@ -2676,6 +2704,7 @@ static bool run_test(void) {
CHECK(unlink(failed_v18_path) == 0); CHECK(unlink(failed_v18_path) == 0);
CHECK(unlink(failed_v19_path) == 0); CHECK(unlink(failed_v19_path) == 0);
CHECK(unlink(raw_missing_table_path) == 0); CHECK(unlink(raw_missing_table_path) == 0);
CHECK(unlink(v24_path) == 0);
CHECK(rmdir(directory) == 0); CHECK(rmdir(directory) == 0);
return true; return true;
} }

View file

@ -89,6 +89,13 @@ int main() {
Lardon3DTask *task = lardon3d_project_create_raw_development_task( Lardon3DTask *task = lardon3d_project_create_raw_development_task(
&state, capture.capture_id, raw_asset.asset_id, &task_id); &state, capture.capture_id, raw_asset.asset_id, &task_id);
CHECK(task != nullptr && task_id != 0); CHECK(task != nullptr && task_id != 0);
Lardon3DResourceEstimate single_estimate{};
CHECK(lardon3d_task_resource_estimate(task, &single_estimate));
CHECK(single_estimate.memory_fixed_bytes >= (UINT64_C(2) << 30) &&
single_estimate.memory_bytes_per_item == 0 &&
single_estimate.minimum_batch_size == 1 &&
single_estimate.maximum_batch_size == 1 &&
single_estimate.desired_cpu_threads == 1);
Lardon3DProjectDbTask generic{}; Lardon3DProjectDbTask generic{};
Lardon3DProjectDbRawDevelopmentTask typed{}; Lardon3DProjectDbRawDevelopmentTask typed{};
CHECK(lardon3d_project_db_load_task(database, task_id, &generic) == LARDON3D_PROJECT_DB_OK); CHECK(lardon3d_project_db_load_task(database, task_id, &generic) == LARDON3D_PROJECT_DB_OK);
@ -154,6 +161,64 @@ int main() {
database, &snapshot, LARDON3D_RAW_DEVELOPMENT_TASK_KIND, 2, nullptr, &typed, database, &snapshot, LARDON3D_RAW_DEVELOPMENT_TASK_KIND, 2, nullptr, &typed,
7) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT); 7) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
/* These minimal selected rows isolate the v24 parent association contract;
* item science remains covered by selected-execution tests and the frozen
* per-Capture developer tests. */
sqlite3 *batch_seed = nullptr;
CHECK(sqlite3_open(database_path.c_str(), &batch_seed) == SQLITE_OK);
CHECK(sqlite3_exec(batch_seed, "PRAGMA foreign_keys=OFF;"
"INSERT INTO selected_executions(execution_id,quality_task_id,"
"campaign_task_id,stage,next_item_index,item_count,created_at)"
"VALUES(9001,7001,7002,1,0,1,7),(9002,7003,7004,1,0,1,7)",
nullptr, nullptr, nullptr) == SQLITE_OK);
CHECK(sqlite3_close(batch_seed) == SQLITE_OK);
uint64_t batch_task_id = 0;
Lardon3DTask *batch_task = lardon3d_project_create_raw_development_batch_task(
&state, 9001, &batch_task_id);
CHECK(batch_task != nullptr && batch_task_id != 0);
Lardon3DResourceEstimate batch_estimate{};
CHECK(lardon3d_task_resource_estimate(batch_task, &batch_estimate));
/* The pixel-bounded participant allowance must permit the full safe window
* from a 7 GiB post-reserve budget; a fixed context charge would incorrectly
* reduce this exact boundary to seven participants. */
CHECK(batch_estimate.memory_fixed_bytes == 0 &&
batch_estimate.memory_bytes_per_item == (UINT64_C(896) << 20) &&
batch_estimate.minimum_batch_size == 1 &&
batch_estimate.maximum_batch_size == 8 &&
batch_estimate.desired_cpu_threads == 8 &&
batch_estimate.memory_bytes_per_item * batch_estimate.maximum_batch_size ==
(UINT64_C(7) << 30) &&
batch_estimate.desired_gpu_slots == 0 &&
batch_estimate.desired_io_slots == 1 &&
batch_estimate.task_class == LARDON3D_RESOURCE_TASK_MIXED);
Lardon3DProjectDbRawDevelopmentBatchTask batch_typed{};
CHECK(lardon3d_project_db_load_raw_development_batch_task(
database, batch_task_id, &batch_typed) == LARDON3D_PROJECT_DB_OK &&
batch_typed.selected_execution_id == 9001);
Lardon3DTaskDurableSnapshot batch_snapshot{};
CHECK(lardon3d_task_durable_snapshot(batch_task, &batch_snapshot));
Lardon3DTaskKindBinding batch_binding{};
CHECK(lardon3d_raw_development_batch_task_reconstruct(
&batch_snapshot, &reconstruction, &batch_binding));
CHECK(batch_binding.callback != nullptr && batch_binding.userdata != nullptr &&
batch_binding.userdata_destroy != nullptr);
batch_binding.userdata_destroy(batch_binding.userdata);
CHECK(lardon3d_project_db_record_raw_development_batch_task(
database, &batch_snapshot,
LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND,
LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION, nullptr,
&batch_typed, 7) == LARDON3D_PROJECT_DB_OK);
batch_typed.selected_execution_id = 9002;
CHECK(lardon3d_project_db_record_raw_development_batch_task(
database, &batch_snapshot,
LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND,
LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION, nullptr,
&batch_typed, 7) == LARDON3D_PROJECT_DB_CONSTRAINT);
CHECK(lardon3d_project_db_load_raw_development_batch_task(
database, batch_task_id, &batch_typed) == LARDON3D_PROJECT_DB_OK &&
batch_typed.selected_execution_id == 9001);
lardon3d_task_destroy(batch_task);
sqlite3 *raw_database = nullptr; sqlite3 *raw_database = nullptr;
char update_mismatch_kind[128]; char update_mismatch_kind[128];
std::snprintf(update_mismatch_kind, sizeof(update_mismatch_kind), std::snprintf(update_mismatch_kind, sizeof(update_mismatch_kind),

View file

@ -18,6 +18,20 @@ static bool sql(const char *path, const char *text) {
return sqlite3_close(connection) == SQLITE_OK && code == SQLITE_OK; return sqlite3_close(connection) == SQLITE_OK && code == SQLITE_OK;
} }
static void feature_asset_path(
const unsigned char hash[LARDON3D_PROJECT_DB_SHA256_SIZE],
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY]) {
static const char digits[] = "0123456789abcdef";
char hex[LARDON3D_PROJECT_DB_SHA256_SIZE * 2 + 1];
for (size_t index = 0; index < LARDON3D_PROJECT_DB_SHA256_SIZE; ++index) {
hex[index * 2] = digits[hash[index] >> 4];
hex[index * 2 + 1] = digits[hash[index] & 15U];
}
hex[sizeof(hex) - 1] = '\0';
(void)snprintf(path, LARDON3D_PROJECT_DB_PATH_CAPACITY,
"assets/features/%c%c/%s", hex[0], hex[1], hex);
}
static bool seed_v21(const char *path) { static bool seed_v21(const char *path) {
Lardon3DProjectDb *database = NULL; Lardon3DProjectDb *database = NULL;
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY]; char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY];
@ -50,6 +64,8 @@ static bool seed_v21(const char *path) {
"INSERT INTO sparse_calibration_scope_images VALUES(1,1,1);" "INSERT INTO sparse_calibration_scope_images VALUES(1,1,1);"
/* This fixture is a true v21 database; future additive objects must not /* This fixture is a true v21 database; future additive objects must not
remain merely because it was generated from a temporary current DB. */ remain merely because it was generated from a temporary current DB. */
"DROP TABLE feature_extract_batch_tasks;"
"DROP TABLE raw_development_batch_tasks;"
"DROP TABLE capture_calibration_selections;" "DROP TABLE capture_calibration_selections;"
"DROP TABLE optical_calibration_profiles;" "DROP TABLE optical_calibration_profiles;"
"DROP TABLE capture_optical_configurations;" "DROP TABLE capture_optical_configurations;"
@ -89,6 +105,29 @@ static bool run(void) {
if (migration_result != LARDON3D_PROJECT_DB_OK) if (migration_result != LARDON3D_PROJECT_DB_OK)
fprintf(stderr, "retry migration result=%d error=%s\n", (int)migration_result, error); fprintf(stderr, "retry migration result=%d error=%s\n", (int)migration_result, error);
CHECK(migration_result == LARDON3D_PROJECT_DB_OK); CHECK(migration_result == LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_schema_version(database) ==
LARDON3D_PROJECT_DB_SCHEMA_VERSION);
CHECK(sql(path, "SELECT 1 FROM raw_development_batch_tasks WHERE 0;"));
lardon3d_project_db_close(database);
database = NULL;
CHECK(sql(path, "DROP TABLE feature_extract_batch_tasks;"
"DROP TABLE raw_development_batch_tasks;"
"UPDATE metadata SET value=23 WHERE key='schema_version';"));
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V24", "1", 1) == 0);
CHECK(lardon3d_project_db_open(path, &database, error) != LARDON3D_PROJECT_DB_OK);
CHECK(database == NULL);
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V24") == 0);
CHECK(sqlite3_open(path, &raw) == SQLITE_OK);
CHECK(sqlite3_prepare_v2(raw,
"SELECT value FROM metadata WHERE key='schema_version'", -1,
&query, NULL) == SQLITE_OK);
CHECK(sqlite3_step(query) == SQLITE_ROW && sqlite3_column_int(query, 0) == 23);
sqlite3_finalize(query);
CHECK(sqlite3_close(raw) == SQLITE_OK);
migration_result = lardon3d_project_db_open(path, &database, error);
if (migration_result != LARDON3D_PROJECT_DB_OK)
fprintf(stderr, "v24 retry result=%d error=%s\n", (int)migration_result, error);
CHECK(migration_result == LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_schema_version(database) == CHECK(lardon3d_project_db_schema_version(database) ==
LARDON3D_PROJECT_DB_SCHEMA_VERSION); LARDON3D_PROJECT_DB_SCHEMA_VERSION);
Lardon3DProjectDbSelectedExecutionItem item = { Lardon3DProjectDbSelectedExecutionItem item = {
@ -148,6 +187,85 @@ static bool run(void) {
&execution) == LARDON3D_PROJECT_DB_OK && &execution) == LARDON3D_PROJECT_DB_OK &&
execution.execution_id == execution_id && execution.execution_id == execution_id &&
execution.stage == LARDON3D_SELECTED_EXECUTION_READY); execution.stage == LARDON3D_SELECTED_EXECUTION_READY);
Lardon3DTaskDurableSnapshot batch_snapshot = {
.id = 3,
.estimate = {.memory_fixed_bytes = 64U * 1024U * 1024U,
.memory_bytes_per_item = 512U * 1024U * 1024U,
.minimum_batch_size = 1,
.maximum_batch_size = 12,
.desired_cpu_threads = 12,
.desired_io_slots = 1,
.task_class = LARDON3D_RESOURCE_TASK_CPU},
.saved_state = TASK_PENDING,
.recovery_state = TASK_PENDING,
};
(void)snprintf(batch_snapshot.name, sizeof(batch_snapshot.name), "Feature batch");
Lardon3DProjectDbFeatureExtractBatchTask batch = {
.task_id = 3,
.selected_execution_id = execution_id,
.next_item_index = 2,
.extractor_version = 1,
.max_features = 512,
.pyramid_levels = 4,
.fast_threshold = 10,
.parameter_fingerprint = {7},
};
(void)snprintf(batch.extractor_kind, sizeof(batch.extractor_kind), "orb");
/* An invalid new cursor must roll back both generic and typed state; v25
cannot publish a Task whose durable prefix exceeds its immutable domain. */
CHECK(lardon3d_project_db_record_feature_extract_batch_task(
database, &batch_snapshot, "features.extract.batch", 1, NULL,
&batch, 20) == LARDON3D_PROJECT_DB_CONSTRAINT);
Lardon3DProjectDbTask absent_batch_task;
CHECK(lardon3d_project_db_load_task(database, 3, &absent_batch_task) ==
LARDON3D_PROJECT_DB_NOT_FOUND);
batch.next_item_index = 0;
CHECK(lardon3d_project_db_record_feature_extract_batch_task(
database, &batch_snapshot, "features.extract.batch", 1, NULL,
&batch, 21) == LARDON3D_PROJECT_DB_OK);
Lardon3DProjectDbFeatureExtractBatchTask loaded_batch;
CHECK(lardon3d_project_db_load_feature_extract_batch_task(database, 3,
&loaded_batch) ==
LARDON3D_PROJECT_DB_OK &&
loaded_batch.selected_execution_id == execution_id &&
loaded_batch.next_item_index == 0 &&
memcmp(loaded_batch.parameter_fingerprint,
batch.parameter_fingerprint, 32) == 0);
CHECK(lardon3d_project_db_advance_feature_extract_batch_task(database, 3, 0,
1) ==
LARDON3D_PROJECT_DB_CONSTRAINT);
unsigned char source_hash[32] = {0};
unsigned char feature_hash[32] = {9};
char feature_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
feature_asset_path(feature_hash, feature_path);
Lardon3DProjectDbFeatureSet feature_set;
CHECK(lardon3d_project_db_register_feature_set(
database, 1, "orb", 1, batch.parameter_fingerprint, source_hash,
1, 1, 32, feature_hash, feature_path, 32,
LARDON3D_DB_FEATURE_ASSET_DURABLE, 3, 22, &feature_set) ==
LARDON3D_PROJECT_DB_OK);
CHECK(lardon3d_project_db_advance_feature_extract_batch_task(database, 3, 0,
1) ==
LARDON3D_PROJECT_DB_OK &&
lardon3d_project_db_advance_feature_extract_batch_task(database, 3, 0,
1) ==
LARDON3D_PROJECT_DB_OK);
/* A lagging generic checkpoint may repeat cursor zero, but the typed prefix
remains at one. Changing any immutable ORB field is a durable conflict. */
CHECK(lardon3d_project_db_record_feature_extract_batch_task(
database, &batch_snapshot, "features.extract.batch", 1, NULL,
&batch, 23) == LARDON3D_PROJECT_DB_OK &&
lardon3d_project_db_load_feature_extract_batch_task(database, 3,
&loaded_batch) ==
LARDON3D_PROJECT_DB_OK &&
loaded_batch.next_item_index == 1);
batch.max_features++;
CHECK(lardon3d_project_db_record_feature_extract_batch_task(
database, &batch_snapshot, "features.extract.batch", 1, NULL,
&batch, 24) == LARDON3D_PROJECT_DB_CONSTRAINT);
lardon3d_project_db_close(database); lardon3d_project_db_close(database);
database = NULL; database = NULL;

View file

@ -87,8 +87,10 @@ test_stage_matrix(void)
} cases[] = { } cases[] = {
{"acquisition_campaign.run", LARDON3D_TUI_STAGE_ACQUISITION}, {"acquisition_campaign.run", LARDON3D_TUI_STAGE_ACQUISITION},
{"raw.develop", LARDON3D_TUI_STAGE_RAW}, {"raw.develop", LARDON3D_TUI_STAGE_RAW},
{"raw.develop.batch", LARDON3D_TUI_STAGE_RAW},
{"photo_quality.triage", LARDON3D_TUI_STAGE_QUALITY}, {"photo_quality.triage", LARDON3D_TUI_STAGE_QUALITY},
{"features.extract", LARDON3D_TUI_STAGE_FEATURES}, {"features.extract", LARDON3D_TUI_STAGE_FEATURES},
{"features.extract.batch", LARDON3D_TUI_STAGE_FEATURES},
{"features.extract.sift", LARDON3D_TUI_STAGE_FEATURES}, {"features.extract.sift", LARDON3D_TUI_STAGE_FEATURES},
{"features.extract.rootsift", LARDON3D_TUI_STAGE_FEATURES}, {"features.extract.rootsift", LARDON3D_TUI_STAGE_FEATURES},
{"visual_index.update", LARDON3D_TUI_STAGE_VISUAL_INDEX}, {"visual_index.update", LARDON3D_TUI_STAGE_VISUAL_INDEX},