pre-sfm: freeze A6000 v25 resource scaling proof
This commit is contained in:
parent
7e93558f5a
commit
cff9b88536
35 changed files with 3004 additions and 478 deletions
377
AGENTS.md
377
AGENTS.md
|
|
@ -4,15 +4,15 @@
|
|||
|
||||
- The root `README.md` is the index of project documentation.
|
||||
- Canonical documents under `docs/**` define scientific contracts, architecture,
|
||||
FROZEN invariants, persistence semantics, lifecycle state, and roadmap
|
||||
ordering.
|
||||
FROZEN invariants, persistence semantics, lifecycle state, resource policy,
|
||||
and roadmap ordering.
|
||||
- Before changing an architectural, scientific, persistence, runtime, or
|
||||
resource-sensitive area, identify and read the relevant canonical document
|
||||
and its applicable invariants.
|
||||
- FROZEN documentation and contracts must never be changed silently to fit an
|
||||
implementation.
|
||||
- A lower-level implementation convenience never overrides a higher-level
|
||||
canonical contract.
|
||||
canonical contract or an explicit human decision.
|
||||
- Lifecycle markers such as `PLANNED`, `IMPLEMENTED`,
|
||||
`VALIDATION PENDING`, and `PASS/FROZEN` describe repository state and must
|
||||
match actual implementation and validation evidence.
|
||||
|
|
@ -30,11 +30,12 @@ through an explicitly authorized, explicitly scoped human ticket:
|
|||
|
||||
- Gates A–G — PASS/FROZEN
|
||||
- Track Model — PASS/FROZEN
|
||||
- Track Builder — PASS/FROZEN
|
||||
- Track Builder scientific contract — PASS/FROZEN
|
||||
- F0 — PASS/FROZEN
|
||||
- Phase H v1 — PASS/FROZEN
|
||||
- MVS-M1 — PASS/FROZEN
|
||||
- Project DB v22 scientific/persistence foundation — PASS/FROZEN
|
||||
- Project DB v23 generic optical-context overlay — PASS/FROZEN
|
||||
- Calibration Bootstrap v1 — PASS/FROZEN
|
||||
- Selected Scientific Execution — PASS/FROZEN
|
||||
- Photo Quality Triage / Acquisition Selection — PASS/FROZEN
|
||||
|
|
@ -43,25 +44,30 @@ through an explicitly authorized, explicitly scoped human ticket:
|
|||
- S3 Capture / Acquisition Ingestion — PASS/FROZEN
|
||||
- Durable Acquisition-Campaign Execution — PASS/FROZEN
|
||||
- Global Maintenance Audit — PASS/FROZEN
|
||||
- Real S21 Tracks scientific result — PASS/FROZEN
|
||||
|
||||
Detailed subcontracts remain defined by their canonical documents. This file
|
||||
does not duplicate every S3 substage, scientific threshold, migration detail,
|
||||
must not duplicate every S3 substage, scientific threshold, migration detail,
|
||||
or persistence format.
|
||||
|
||||
Project DB v23 is the current additive optical-context overlay. It preserves
|
||||
the v22 scientific/persistence foundation and must not infer or backfill optical
|
||||
identity from historical data. Historical references to older Project DB
|
||||
versions remain valid when they describe the actual historical contract or
|
||||
migration path. Do not rewrite legitimate v16–v22 history merely because v23
|
||||
is current.
|
||||
Project DB v25 is the current additive operational schema in the active Feature
|
||||
batch tranche. It preserves v24 RAW batch, the v23 optical overlay and the v22
|
||||
scientific/persistence foundation. Its authorized purpose is limited to typed
|
||||
durable persistence for `features.extract.batch/1` through
|
||||
`feature_extract_batch_tasks`; it adds no scientific identity and must not
|
||||
reinterpret historical rows. Until the v25/Feature-batch tranche has completed
|
||||
its real-data proof and final review, its lifecycle must remain truthful rather
|
||||
than being marked `PASS/FROZEN` prematurely. Historical references to older
|
||||
Project DB versions remain valid where they describe the actual historical
|
||||
contract or migration path.
|
||||
|
||||
The global maintenance implementation, fresh portable/Vulkan/sanitizer/
|
||||
concurrency validation and independent final review are acquired. Its lifecycle
|
||||
is `GLOBAL_MAINTENANCE_AUDIT=PASS/FROZEN`. The review independently passed the
|
||||
portable build, 64/64 complete suite, 15/15 focused matrix, 76/76 strict header
|
||||
probes across 19 modified/new public headers, ABI and production-seam checks,
|
||||
retained-manifest verification and diff validation, with zero blocking findings.
|
||||
Do not reopen this boundary or infer a new scientific policy from the freeze.
|
||||
portable build, complete suites, focused matrices, strict public-header probes,
|
||||
ABI and production-seam checks, retained-manifest verification and diff
|
||||
validation with zero blocking findings. Do not reopen this boundary or infer a
|
||||
new scientific policy from the freeze.
|
||||
|
||||
The canonical review checkpoint is tag `global-maintenance-2026-09-01` at
|
||||
commit `b84f860d868c66d9ee84b85ceb1bc6480b95aca5`; its detailed evidence is
|
||||
|
|
@ -88,11 +94,13 @@ When a ticket declares `NO_NEW_SUBSYSTEM`, do not introduce an unrelated:
|
|||
- daemon.
|
||||
|
||||
This restriction is ticket-scoped. It does not mean that a subsystem already
|
||||
present in the canonical architecture can never be used by later authorized
|
||||
work.
|
||||
present in the canonical architecture can never be used or extended by later
|
||||
authorized work.
|
||||
|
||||
Never reopen a FROZEN scientific or architectural decision merely to simplify
|
||||
an implementation.
|
||||
Never reopen a FROZEN scientific decision merely to simplify an
|
||||
implementation. Operational/resource contracts may be reopened only by explicit
|
||||
human authority and must preserve the scientific result exactly unless the
|
||||
human ticket explicitly says otherwise.
|
||||
|
||||
## 3. Scope and Git discipline
|
||||
|
||||
|
|
@ -157,8 +165,8 @@ require a clean worktree unless the ticket explicitly requires one.
|
|||
allocation, solver execution or output mutation; do not alter the FROZEN
|
||||
defaults, encodings or fingerprints to accommodate an unsafe cast.
|
||||
|
||||
Compiler success alone is not proof of API, ABI, persistence, or scientific
|
||||
contract correctness.
|
||||
Compiler success alone is not proof of API, ABI, persistence, resource, or
|
||||
scientific contract correctness.
|
||||
|
||||
## 5. Identity discipline
|
||||
|
||||
|
|
@ -217,13 +225,132 @@ explicitly defines such an identity.
|
|||
- Extend validated abstractions rather than rewriting validated modules.
|
||||
- Reuse the existing Task / Queue / Scheduler / Resource Governor ownership
|
||||
model rather than creating parallel runtime infrastructure.
|
||||
- System stability and responsiveness take priority over throughput.
|
||||
|
||||
### Canonical utilization objective — MAXIMUM SAFE USEFUL THROUGHPUT
|
||||
|
||||
Lardon3D is a throughput-oriented workstation application. The Resource
|
||||
Governor MUST maximize safe and useful utilization for every production Task
|
||||
after preserving only the interactive host reserve needed for normal use of:
|
||||
|
||||
- Arch Linux / Sway and normal desktop services;
|
||||
- Firefox;
|
||||
- normal audio/music playback;
|
||||
- lightweight interactive use of the workstation.
|
||||
|
||||
All CPU, RAM, I/O capacity and validated accelerator capacity beyond that
|
||||
interactive reserve belongs to the active Lardon3D workload when useful work
|
||||
exists.
|
||||
|
||||
On the current reference host, the normal observed policy outcome is
|
||||
approximately:
|
||||
|
||||
```text
|
||||
16 logical CPUs total
|
||||
4 logical CPUs reserved for the interactive host
|
||||
12 logical CPUs available to the compute pool
|
||||
~3 GiB MemAvailable preserved as the hard RAM reserve
|
||||
Radeon 780M UMA available to validated/useful GPU backends
|
||||
```
|
||||
|
||||
These values are reference-host outcomes, NOT portable constants. Topology,
|
||||
process affinity, smaller hosts, memory pressure, I/O pressure and future
|
||||
hardware must be handled dynamically. A future 32-thread host must not inherit a
|
||||
12-thread product ceiling merely because the reference host exposed 12 compute
|
||||
threads.
|
||||
|
||||
The optimization target is:
|
||||
|
||||
```text
|
||||
MAXIMUM SAFE USEFUL THROUGHPUT
|
||||
```
|
||||
|
||||
It is NOT:
|
||||
|
||||
```text
|
||||
minimum resource usage
|
||||
fixed CPU counts
|
||||
CPU/GPU utilization percentage for its own sake
|
||||
preserving historical CPU1/batch1 descriptors without evidence
|
||||
```
|
||||
|
||||
The Governor remains the sole production resource authority. Normal users do
|
||||
not choose CPU count, worker count, batch, inflight depth, GPU backend, scratch
|
||||
mode, or RAM budget for authoritative execution.
|
||||
|
||||
A production Task may deliberately use less than the currently available
|
||||
compute pool only when concrete evidence establishes at least one relevant
|
||||
constraint, such as:
|
||||
|
||||
- true scientific or dependency serialism;
|
||||
- measured useful-scaling knee;
|
||||
- memory bound;
|
||||
- I/O saturation;
|
||||
- validated GPU execution making additional CPU work useless;
|
||||
- deterministic publication constraint that cannot be separated safely from
|
||||
preparation;
|
||||
- another explicit and measured resource limitation.
|
||||
|
||||
The reason must be represented by the Task/Governor contract or canonical
|
||||
resource documentation where non-obvious. A historical descriptor by itself is
|
||||
not evidence.
|
||||
|
||||
### SERIALISM_REQUIRES_PROOF
|
||||
|
||||
`SERIALISM_REQUIRES_PROOF` is canonical operational policy.
|
||||
|
||||
Per-item atomicity does NOT imply cross-item serialization. Owner-only or
|
||||
ordered durable publication does NOT imply serial preparation. When multiple
|
||||
independent work units exist and exact science/persistence semantics are
|
||||
preserved, the Task must expose bounded concurrency to the Governor.
|
||||
|
||||
A long-running `CPU1` or `batch1` path with independent executable work and
|
||||
available safe resources is an operational defect until one of the documented
|
||||
measured limitations above proves otherwise.
|
||||
|
||||
Do not solve this by creating another scheduler, uncontrolled worker pool,
|
||||
unbounded `std::async`, detached threads, or resources outside Governor
|
||||
accounting. Prefer the established bounded pattern where appropriate:
|
||||
|
||||
```text
|
||||
one admitted owner Task
|
||||
-> bounded independent participants/preparation
|
||||
-> join all participants
|
||||
-> deterministic owner-only publication
|
||||
```
|
||||
|
||||
Every participant must be accounted, bounded, cancellable and joined. Shared
|
||||
SQLite access must preserve the existing ownership/serialization contracts.
|
||||
|
||||
### CPU, GPU, RAM and I/O policy
|
||||
|
||||
- CPU: expose truthful minimum/useful/safe concurrency and allow Governor to use
|
||||
the largest currently safe/useful value. Do not hardcode the reference-host
|
||||
12-thread result.
|
||||
- GPU: a validated AND useful production GPU backend MUST be preferred when it
|
||||
is available and Governor-safe. Do not invent or promote an unvalidated GPU
|
||||
path merely to make the GPU busy.
|
||||
- RAM: use available memory aggressively for useful work while preserving the
|
||||
hard interactive reserve, approximately 3 GiB `MemAvailable` on the reference
|
||||
host. The 3–4 GiB band is a pressure/conservatism zone, not a permanent extra
|
||||
1 GiB subtraction from every Task.
|
||||
- UMA: iGPU allocations count exactly once against host RAM.
|
||||
- swap/zram: pressure and safety mechanisms, never admitted working RAM.
|
||||
- scratch SSD: optional external storage, never RAM. A Task may consume it only
|
||||
through an explicitly validated Governor-owned scratch contract.
|
||||
- I/O: if additional concurrency no longer improves useful throughput because
|
||||
storage or another I/O boundary is saturated, the measured knee is the right
|
||||
useful limit. Do not force CPU saturation for appearance.
|
||||
- Pressure: CPU/memory/I/O PSI and active swap-in/out deltas may reduce
|
||||
admission. When pressure clears, useful resources must be re-admitted rather
|
||||
than remaining permanently throttled.
|
||||
|
||||
System responsiveness is therefore protected by the explicit host reserve and
|
||||
pressure feedback; it is not a justification for leaving additional safe/useful
|
||||
compute idle.
|
||||
|
||||
- Bound memory, buffers, files, descriptors, processes, threads, captured
|
||||
output, parser work, temporary storage, and staging reasonably for the
|
||||
output, parser work, temporary storage, and staging honestly for the
|
||||
operation.
|
||||
- Memory shared with an iGPU counts against host RAM.
|
||||
- zram and swap are pressure/safety mechanisms, not normal working-memory
|
||||
budgets.
|
||||
- An operational hardware/resource bound must not accidentally become a
|
||||
scientific dataset-size limit.
|
||||
- Do not invent a new global resource subsystem inside a ticket that defers it.
|
||||
|
|
@ -237,12 +364,16 @@ Resource-sensitive work must identify, where relevant:
|
|||
|
||||
- resource owner;
|
||||
- admission point;
|
||||
- bound or budget;
|
||||
- minimum/useful/safe CPU and batch capability;
|
||||
- fixed/per-participant/transient memory budget;
|
||||
- reservation lifetime;
|
||||
- release point;
|
||||
- failure cleanup;
|
||||
- cancellation cleanup;
|
||||
- whether the bound is operational or scientific.
|
||||
- I/O/scratch ownership;
|
||||
- validated GPU capability;
|
||||
- whether a bound is operational or scientific;
|
||||
- measured reason for intentional under-utilization.
|
||||
|
||||
The reviewed external USB SSD controller is the authorized physical-lifecycle
|
||||
boundary for the exact UDisks Drive/label/UUID contract. Its validated snapshot
|
||||
|
|
@ -251,11 +382,10 @@ production orchestrator for scratch-lease acquire/release. The controller does
|
|||
not replace the Governor or invent Task scratch eligibility, and swap/scratch
|
||||
never become RAM. The application lifetime order is strict: destroy/join the
|
||||
Queue so every Task lease is released, checked-join/unregister the SSD binding,
|
||||
destroy the controller, then destroy the Governor. The current fourteen Task
|
||||
kinds have no scratch consumer, so availability is capability, not fabricated
|
||||
usage. Do not add ad-hoc discovery, mounting, formatting, `swapon`, cleanup,
|
||||
force-drain, shell commands, or a second resource/scheduling subsystem outside
|
||||
the reviewed controller/Governor APIs.
|
||||
destroy the controller, then destroy the Governor. SSD availability is a
|
||||
capability, not fabricated Task usage. Do not add ad-hoc discovery, mounting,
|
||||
formatting, `swapon`, cleanup, force-drain, shell commands, or a second
|
||||
resource/scheduling subsystem outside the reviewed controller/Governor APIs.
|
||||
|
||||
Snapshot conversion is fail-closed by physical state. Any pairing or authority
|
||||
requires current detection of the Drive and both UUID-bearing partitions,
|
||||
|
|
@ -426,7 +556,8 @@ comment.
|
|||
Where relevant, comments must preserve the established distinction between:
|
||||
|
||||
- immutable SOURCE RAW;
|
||||
- deterministic DERIVED representation;
|
||||
- camera JPEG SOURCE;
|
||||
- deterministic DERIVED RAW representation;
|
||||
- RAW Policy v1;
|
||||
- L3DRAWD1 identity;
|
||||
- metadata-only acquisition processing;
|
||||
|
|
@ -438,6 +569,12 @@ Where relevant, comments must preserve the established distinction between:
|
|||
- bounded secondary JPEG validation;
|
||||
- zero-only permitted MPF gaps/trailer.
|
||||
|
||||
For the current A6000 selected scientific execution, paired camera JPEGs are
|
||||
valid Capture source assets and preferred fast proxies for Photo Quality, but
|
||||
the FROZEN geometry representation remains deterministic RAW-derived PNG. Do
|
||||
not substitute camera-JPEG geometry without a separately authorized and proven
|
||||
scientific/calibration equivalence tranche.
|
||||
|
||||
Do not imply that metadata validation performs pixel decoding when it does not.
|
||||
|
||||
### Task / Queue / Scheduler / Governor
|
||||
|
|
@ -454,6 +591,8 @@ In particular:
|
|||
- Resource Reservation belongs to the currently admitted bounded execution.
|
||||
- `sequence_break` is an execution boundary that releases/re-establishes
|
||||
admission according to runtime semantics.
|
||||
- Per-item atomicity does not imply cross-item serialization.
|
||||
- Serial publication may coexist with Governor-admitted parallel preparation.
|
||||
|
||||
Do not imply that a long campaign reserves resources for its entire lifetime
|
||||
when execution is group-bounded.
|
||||
|
|
@ -466,16 +605,18 @@ resource ownership understandable from source.
|
|||
Where applicable document:
|
||||
|
||||
- RAM ownership/bound;
|
||||
- CPU admission;
|
||||
- CPU minimum/useful/safe admission;
|
||||
- GPU admission;
|
||||
- I/O ownership;
|
||||
- process/thread count;
|
||||
- per-participant memory;
|
||||
- file-descriptor lifetime;
|
||||
- temporary/scratch storage ownership;
|
||||
- reservation lifetime;
|
||||
- release point;
|
||||
- failure cleanup;
|
||||
- cancellation cleanup.
|
||||
- cancellation cleanup;
|
||||
- measured scaling knee or reason for serial execution.
|
||||
|
||||
An operational hardware/resource bound MUST NOT accidentally become a
|
||||
scientific dataset-size limit.
|
||||
|
|
@ -502,6 +643,12 @@ The residual pre-return S3-E crash window remains intentional and documented:
|
|||
if S3-E creates a Capture internally and the process dies before S3-E returns
|
||||
the `capture_id`, campaign execution must not guess that Capture identity.
|
||||
|
||||
For `raw.develop.batch/1`, typed `task_id -> selected_execution_id` persistence
|
||||
must remain distinct from generic Task runtime state. Independent RAW
|
||||
participants may prepare concurrently, but owner publication and selected-item
|
||||
cursor advancement remain deterministic and ordered by the selected execution
|
||||
contract.
|
||||
|
||||
### C / C++ ABI boundaries
|
||||
|
||||
Where C++ implements a public C interface or C Task callback, comments should
|
||||
|
|
@ -524,7 +671,8 @@ Comments should state WHAT CONTRACT the fixture proves, for example:
|
|||
- ambiguity;
|
||||
- scientific identity conflict;
|
||||
- cross-ScanSet rejection;
|
||||
- resource-pressure/admission condition.
|
||||
- resource-pressure/admission condition;
|
||||
- deliberate CPU1 baseline used only for a scaling comparison.
|
||||
|
||||
Do not narrate ordinary test mechanics.
|
||||
|
||||
|
|
@ -552,9 +700,9 @@ For every future implementation tranche:
|
|||
4. update canonical documentation when behavior/contracts changed;
|
||||
5. validate every modified public C17 header;
|
||||
6. validate relevant C++ syntax/build integration;
|
||||
7. validate resource ownership and bounds;
|
||||
8. run applicable build/tests/sanitizers;
|
||||
9. perform the required review;
|
||||
7. validate resource ownership, accounting, useful scaling and bounds;
|
||||
8. run applicable build/tests/sanitizers efficiently;
|
||||
9. perform the required bounded review;
|
||||
10. only then claim completion.
|
||||
|
||||
The normalized future implementation standard is:
|
||||
|
|
@ -566,12 +714,14 @@ CODE
|
|||
+ CANONICAL DOCUMENTATION
|
||||
+ C17/C++ SYNTAX
|
||||
+ RESOURCE OWNERSHIP
|
||||
+ MAXIMUM SAFE USEFUL THROUGHPUT
|
||||
+ VALIDATION
|
||||
+ REVIEW
|
||||
```
|
||||
|
||||
A tranche is NOT complete merely because code compiles and tests pass when
|
||||
required contract/invariant comments or canonical documentation are missing.
|
||||
required contract/invariant comments, canonical documentation, or required
|
||||
resource behavior are missing.
|
||||
|
||||
There should be no future project-wide comment-cleanup pass for newly written
|
||||
code: comment debt must be handled when the code is introduced.
|
||||
|
|
@ -613,9 +763,12 @@ Persistence changes require explicit attention to:
|
|||
|
||||
For Project DB:
|
||||
|
||||
- preserve the FROZEN v22 semantics and the current additive v23 optical
|
||||
overlay unless a ticket explicitly authorizes a later schema change;
|
||||
- schema-version changes require explicit human authorization;
|
||||
- preserve the FROZEN v22 scientific/persistence semantics and v23 optical
|
||||
overlay;
|
||||
- v24 is the explicitly authorized additive operational RAW-batch migration and
|
||||
must not reinterpret scientific history;
|
||||
- future schema-version changes beyond the currently authorized v24 require
|
||||
explicit human authorization;
|
||||
- migrations must be additive unless a different migration is explicitly
|
||||
authorized;
|
||||
- existing projects must remain recoverable;
|
||||
|
|
@ -635,41 +788,86 @@ make retries convenient.
|
|||
- For changes to API, architecture, ownership, concurrency, persistence,
|
||||
pipeline, resources, limits, or scientific semantics, check whether
|
||||
canonical documentation must be updated.
|
||||
- `MAXIMUM SAFE USEFUL THROUGHPUT` and `SERIALISM_REQUIRES_PROOF` are canonical
|
||||
human resource-policy decisions. Any resource document that contradicts them
|
||||
is stale unless it describes an explicitly historical measurement.
|
||||
- Documentation must describe proven implementation, not desired future
|
||||
behavior.
|
||||
behavior. Human policy may be marked as policy while implementation work is
|
||||
still `VALIDATION PENDING`.
|
||||
- Roadmap documents may describe future behavior, but future capabilities must
|
||||
be clearly marked as planned/later/exploratory.
|
||||
- Never describe future viewer, Capture Guidance, video/keyframe, Task scratch
|
||||
consumption, or camera-control capabilities as implemented before they are
|
||||
actually validated. The current TUI/F10 and controller-to-Governor registry
|
||||
are validated operationally, but no current Task kind consumes scratch and
|
||||
those interfaces do not make dense/scratch-consuming workflows complete.
|
||||
actually validated.
|
||||
- Statuses such as `PLANNED`, `IMPLEMENTED`, `VALIDATION PENDING`, and
|
||||
`PASS/FROZEN` are authoritative lifecycle statements.
|
||||
- Update lifecycle state only when implementation, validation, and review
|
||||
evidence support the transition.
|
||||
- Never mark work `PASS/FROZEN` without the required validation and review.
|
||||
- Preserve legitimate historical DB/version references where they describe
|
||||
frozen history.
|
||||
- Preserve legitimate historical DB/version/resource references where they
|
||||
describe frozen history; label superseded operational envelopes as historical
|
||||
rather than silently rewriting the evidence.
|
||||
|
||||
## 12. Required validation
|
||||
## 12. Required validation and test resource policy
|
||||
|
||||
For ordinary implementation tickets, before claiming completion, actually run
|
||||
the applicable commands.
|
||||
the applicable commands. Builds, tests, benchmarks and real-proof workloads
|
||||
MUST follow the same resource philosophy as production: preserve the
|
||||
interactive host reserve, then use the maximum safe/useful remaining resources.
|
||||
Artificially serial engineering work wastes both elapsed time and agent budget.
|
||||
|
||||
Normal build:
|
||||
### Normal build
|
||||
|
||||
Do not use a fixed historical `-j8` as a product rule. Derive a safe parallel
|
||||
job count from the current host/affinity. On the current 16-logical-CPU
|
||||
reference host, where four logical CPUs are reserved for interactive use, the
|
||||
normal build target is approximately:
|
||||
|
||||
```sh
|
||||
meson setup --reconfigure build
|
||||
meson compile -C build -j8
|
||||
meson compile -C build -j12
|
||||
```
|
||||
|
||||
Normal tests:
|
||||
On another host, choose the analogous available-compute value rather than
|
||||
hardcoding 12.
|
||||
|
||||
### Normal tests
|
||||
|
||||
Independent tests should run concurrently within the same host reserve and
|
||||
memory constraints. On the current reference host, a typical command is:
|
||||
|
||||
```sh
|
||||
meson test -C build --num-processes 1 --print-errorlogs
|
||||
meson test -C build --num-processes 12 --print-errorlogs
|
||||
```
|
||||
|
||||
This is not permission to race tests that share mutable fixtures, real projects,
|
||||
fixed ports, exclusive GPU state, or other process-global resources. Such tests
|
||||
must be grouped or serialized for the concrete dependency, and that reason must
|
||||
be understood rather than inherited from an old `--num-processes 1` default.
|
||||
|
||||
A CPU1 run is valid when CPU1 is the actual test case or baseline measurement;
|
||||
it is not the authoritative production configuration after higher safe/useful
|
||||
concurrency has been proven.
|
||||
|
||||
### Validation efficiency
|
||||
|
||||
- Do not rebuild unchanged targets between proof iterations without a concrete
|
||||
need.
|
||||
- Do not rerun already-acquired expensive validation when the relevant code and
|
||||
dependency boundary did not change.
|
||||
- Run focused tests first; widen only when the changed dependency boundary
|
||||
requires it.
|
||||
- Expensive/stress validation must be relevant to the current delta.
|
||||
- Heavy validation is parallel by default when independent and resource-safe;
|
||||
serialize only when scientific determinism, mutable shared fixtures, memory,
|
||||
I/O, GPU exclusivity, sanitizer behavior or measured host pressure requires
|
||||
it.
|
||||
- A test timeout or sanitizer failure must be investigated; do not repeatedly
|
||||
rerun until green.
|
||||
- Real-data proofs must use normal Governor admission. Do not manually force
|
||||
CPU1 merely for reproducibility unless CPU1 itself is the intended comparison
|
||||
cohort.
|
||||
|
||||
Diff validation:
|
||||
|
||||
```sh
|
||||
|
|
@ -692,24 +890,48 @@ For relevant concurrency changes, run TSan when supported and meaningful.
|
|||
If a sanitizer is unavailable or invalid because of the environment/toolchain,
|
||||
report that explicitly rather than claiming PASS.
|
||||
|
||||
Run expensive/stress validation only when relevant.
|
||||
|
||||
Run heavy validation serially when required to preserve machine stability.
|
||||
|
||||
Investigate a timeout or sanitizer failure. Do not repeatedly rerun a failing
|
||||
test until it happens to pass.
|
||||
|
||||
Distinguish third-party sanitizer/environment noise from repository defects
|
||||
using concrete stack/failure evidence.
|
||||
|
||||
Never claim a command, test, sanitizer, review, or real-data validation that
|
||||
was not actually performed.
|
||||
Never claim a command, test, sanitizer, review, benchmark, or real-data
|
||||
validation that was not actually performed.
|
||||
|
||||
For documentation/comment-only changes, do not invent unnecessary sanitizer
|
||||
work, but still run enough build/syntax/diff validation to prove that the
|
||||
non-functional boundary was preserved.
|
||||
|
||||
## 13. Review discipline
|
||||
## 13. Engineering execution and credit economy
|
||||
|
||||
Engineering time and agent budget are finite project resources and must not be
|
||||
wasted.
|
||||
|
||||
- Default to the configured economical parent/orchestrator and existing agent
|
||||
roles. Do not invent a new agent hierarchy.
|
||||
- Use expensive/high-reasoning scientific or final-review roles only when the
|
||||
current delta genuinely requires scientific equivalence, difficult
|
||||
concurrency/persistence reasoning, or final sensitive review.
|
||||
- Do not spawn multiple agents to restate the same architecture or repeat the
|
||||
same review.
|
||||
- Do not delegate ordinary implementation defects upward when a worker,
|
||||
mechanic or resolver can fix them directly within authority.
|
||||
- Prefer one focused independent review after the implementation is stable to
|
||||
repeated speculative reviews during ordinary coding.
|
||||
- Reuse acquired maintenance, sanitizer, benchmark and real-data evidence when
|
||||
the affected boundary is unchanged.
|
||||
- Do not wait on an expensive agent when the next authorized executable action
|
||||
can proceed independently.
|
||||
- Keep tool/log output bounded; inspect targeted portions instead of repeatedly
|
||||
dumping whole logs.
|
||||
- Do not rerun a global A-to-Z audit after the maintenance checkpoint.
|
||||
- `WORK FIRST. RETURN LAST.` Ordinary in-scope findings are repaired and
|
||||
validated before returning to the human.
|
||||
|
||||
Low remaining agent/credit budget is a reason to eliminate redundant work, not
|
||||
a reason to weaken required correctness evidence. If a genuinely required
|
||||
validation cannot be afforded/executed, report it honestly rather than claiming
|
||||
PASS.
|
||||
|
||||
## 14. Review discipline
|
||||
|
||||
A normal review should verify the bounded current delta and direct regressions.
|
||||
|
||||
|
|
@ -727,7 +949,7 @@ A reviewer request does not automatically define new policy.
|
|||
Compare findings against:
|
||||
|
||||
1. canonical FROZEN documentation;
|
||||
2. explicit human decisions;
|
||||
2. explicit human decisions, including the resource-utilization policy;
|
||||
3. established tranche semantics;
|
||||
4. documented future-scope boundaries.
|
||||
|
||||
|
|
@ -735,9 +957,13 @@ Do not invoke an expensive implementation agent merely to satisfy speculative
|
|||
hardening that is outside the current contract.
|
||||
|
||||
Comments and documentation are reviewable implementation artifacts. A
|
||||
misleading contract comment is a defect even if compiled behavior is unchanged.
|
||||
misleading contract or resource-policy comment is a defect even if compiled
|
||||
behavior is unchanged.
|
||||
|
||||
## 14. Delivery report / STOP conditions
|
||||
Resource review must specifically flag accidental serialism when independent
|
||||
work exists and safe/useful resources are idle without measured justification.
|
||||
|
||||
## 15. Delivery report / STOP conditions
|
||||
|
||||
Every completed ticket report must include:
|
||||
|
||||
|
|
@ -750,15 +976,17 @@ Every completed ticket report must include:
|
|||
- known blockers;
|
||||
- non-blocking findings;
|
||||
- deliberately deferred/future-scope items;
|
||||
- resource impact where relevant;
|
||||
- resource impact where relevant, including admitted/useful concurrency for
|
||||
modified resource-sensitive Tasks;
|
||||
- confirmation that unrelated and FROZEN areas were preserved;
|
||||
- Git state;
|
||||
- confirmation that `scan3d/` remained untouched when protected.
|
||||
|
||||
STOP and request a human decision only when resolution requires:
|
||||
|
||||
- changing a FROZEN contract;
|
||||
- changing Project DB schema/version without prior authorization;
|
||||
- changing a FROZEN scientific contract;
|
||||
- changing Project DB schema/version beyond already authorized v24 without
|
||||
prior authorization;
|
||||
- introducing a genuinely new subsystem outside authorized scope;
|
||||
- files outside the authorized scope;
|
||||
- destructive Git action;
|
||||
|
|
@ -775,9 +1003,12 @@ Do NOT stop merely because:
|
|||
- a normal test fails;
|
||||
- an ordinary implementation bug exists;
|
||||
- a reviewer identifies a bounded repairable defect;
|
||||
- a resource descriptor is historically conservative and explicit human
|
||||
authority already allows operational correction while preserving science;
|
||||
- a dependency needs factual investigation;
|
||||
- documentation status is stale;
|
||||
- an implementation detail can be resolved safely from existing code and
|
||||
contracts.
|
||||
|
||||
Advance the project while preserving the contracts.
|
||||
Advance the project while preserving scientific contracts and maximizing safe,
|
||||
useful throughput.
|
||||
|
|
|
|||
|
|
@ -276,7 +276,7 @@ calibration connues ; le Sparse SfM réel reste
|
|||
Le Resource Governor ne constitue pas un Resource System générique : il reste
|
||||
l'unique propriétaire des budgets et le seul orchestrateur des leases scratch
|
||||
de production. Le contrôleur SSD UDisks2 est une frontière physique séparée,
|
||||
jamais un second scheduler ou Governor. Les quatorze Task kinds actuels ne
|
||||
jamais un second scheduler ou Governor. Les seize Task kinds actuels ne
|
||||
consomment encore aucun scratch : l'espace disponible est une capacité
|
||||
observable, pas un usage fabriqué, et scratch/swap ne deviennent jamais de la
|
||||
RAM. La TUI/F10 et cette intégration sont validées opérationnellement. L'audit
|
||||
|
|
|
|||
|
|
@ -253,6 +253,9 @@ La tâche utilise le runtime générique via le pattern standard :
|
|||
La Queue conserve un callback ; jusqu'à soixante-quatre participants CPU
|
||||
admis peuvent calculer une fenêtre interne bornée sans modifier l'identité
|
||||
scientifique.
|
||||
- CPU et lot sont essayés ensemble (`1/1`, `2/2`, puis les paliers sûrs) :
|
||||
un CPU supplémentaire avec un lot d'un seul membership ne peut exercer aucun
|
||||
participant supplémentaire et ne constitue donc pas une mesure de scaling.
|
||||
- Réservation CPU + IO avant exécution
|
||||
- `lardon3d_task_sequence_break()` entre chaque lot pour réadmission Governor
|
||||
- Callback terminal checkpoint après `COMPLETED`/`FAILED`/`CANCELLED`
|
||||
|
|
@ -276,7 +279,7 @@ bool lardon3d_candidate_pair_generate_reconstruct(
|
|||
|
||||
### Garantie actuelle
|
||||
|
||||
La Queue conserve un callback actif. À l'intérieur de la Task, jusqu'à douze
|
||||
La Queue conserve un callback actif. À l'intérieur de la Task, jusqu'à soixante-quatre
|
||||
threads CPU admis calculent en parallèle une fenêtre d'au plus deux sources par
|
||||
thread. Chaque participant possède un handle DB de lecture privé. Le thread
|
||||
propriétaire publie ensuite seul et dans l'ordre canonique des sources. Voir le
|
||||
|
|
|
|||
|
|
@ -198,6 +198,50 @@ au plus `cpu_threads - 1` threads enfants. Tous les enfants sont joints avant
|
|||
la fin de la séquence, la libération de réservation ou
|
||||
`lardon3d_task_sequence_break()`.
|
||||
|
||||
## Développement RAW d'une exécution sélectionnée
|
||||
|
||||
`raw.develop.batch/1` conserve la politique RAW v1, les octets PNG,
|
||||
L3DRAWD1, le fingerprint, le content-addressing et toutes les identités du
|
||||
développeur par Capture. Une séquence prend au plus huit items contigus depuis
|
||||
le curseur durable de l'exécution sélectionnée et utilise au plus le nombre de
|
||||
CPUs admis. Chaque participant possède son LibRaw et ses buffers privés ; les
|
||||
896 Mio chargés par item admis couvrent le workspace LibRaw borné à 40 MP, les
|
||||
copies RGB/BGR, les buffers PNG/validation, la pile enfant bornée de 1 Mio et la
|
||||
marge allocateur/codec. Le petit contexte propriétaire est inclus dans cette
|
||||
charge : un budget hôte post-réserve de 7 Gio admet donc exactement les huit
|
||||
participants sûrs, sans débit fixe redondant. Cette estimation est une borne
|
||||
opérationnelle de mémoire par participant, pas une limite scientifique de
|
||||
dataset.
|
||||
OpenCV est fixé à un thread interne afin de ne pas multiplier le fan-out.
|
||||
|
||||
Les enfants ne modifient ni curseur sélectionné ni progression Task. Après leur
|
||||
jointure complète, le callback Queue propriétaire publie seul les mappings
|
||||
`item_index -> image_id` dans l'ordre croissant. Chaque transaction de mapping
|
||||
avance le curseur contigu, puis seulement le checkpoint Task peut avancer. Une
|
||||
erreur, une annulation ou un échec de création d'enfant joint tout le travail
|
||||
lancé ; des assets immuables content-addressés peuvent déjà exister, mais le
|
||||
curseur ne les devine pas et la reprise converge par le développeur existant.
|
||||
Une `sequence_break` sépare toutes les fenêtres, donc la sélection entière ne
|
||||
retient jamais une réservation longue.
|
||||
|
||||
## Extraction Feature d'une exécution sélectionnée
|
||||
|
||||
`features.extract.batch/1` conserve exactement ORB v1, ses paramètres, son
|
||||
fingerprint, ses keypoints/descripteurs U8×32 et le Feature File existant. Une
|
||||
fenêtre contient au plus douze images indépendantes de l'ordre sélectionné. Le
|
||||
propriétaire configure OpenCV à un thread pour éviter tout fan-out imbriqué ;
|
||||
chaque participant possède son décodage et sa sortie bornés, sans accès SQLite,
|
||||
puis tous sont joints.
|
||||
|
||||
Le propriétaire seul publie dans l'ordre et avance le curseur v25 après preuve
|
||||
du Feature Set READY exact. CPU et lot partent du minimum. Comme un CPU
|
||||
supplémentaire ne peut être exercé que par une image indépendante de la même
|
||||
fenêtre, le Governor essaie et accepte ces deux limites ensemble (`1/1`, `2/2`,
|
||||
`4/4`, puis le maximum sûr admis). Les observations de fenêtres durables
|
||||
permettent de retenir le dernier palier apportant au moins 5 %. Le plafond douze
|
||||
est une capacité opérationnelle de benchmark,
|
||||
jamais une limite scientifique, et aucune voie GPU n'est introduite.
|
||||
|
||||
## Calcul et publication Visual Index
|
||||
|
||||
Une séquence sélectionne le même préfixe durable d'au plus seize Feature Sets
|
||||
|
|
|
|||
|
|
@ -138,7 +138,7 @@ Frontière physique UDisks2 pour la paire de labels `LARDON_SWAP` et
|
|||
et drain sûr. Il ne formate, ne répare ni ne force jamais l'hôte et ne remplace
|
||||
pas le Resource Governor. La TUI exécute ses actions dans un seul thread
|
||||
joinable et le Governor enregistre l'état physique puis orchestre seul les
|
||||
leases scratch de production. Les quatorze Task kinds courants n'en consomment
|
||||
leases scratch de production. Les seize Task kinds courants n'en consomment
|
||||
aucun ; capacité visible ne signifie donc pas usage.
|
||||
|
||||
**Statut :** CURRENT / VALIDATED OPERATIONAL
|
||||
|
|
|
|||
|
|
@ -38,10 +38,46 @@ malformée retourne `CORRUPT` avant comparaison avec une demande alternative.
|
|||
|
||||
La migration v22→v23 n'inspecte ni EXIF, chemin, basename, SHA-256, dimensions,
|
||||
nom d'appareil, ni calibration historique. Elle ne réinterprète donc aucune
|
||||
Capture, image ou calibration v16–v22. Le marqueur `schema_version=23` est le
|
||||
Capture, image ou calibration v16–v22. Le marqueur `schema_version=23` reste le
|
||||
point de publication durable ; un échec de création ou un update de marqueur
|
||||
qui ne cible pas exactement la v22 rollbacke toute la migration.
|
||||
|
||||
### Overlay v24 — développement RAW sélectionné borné
|
||||
|
||||
Le schéma courant v24 ajoute uniquement
|
||||
`raw_development_batch_tasks(task_id, selected_execution_id)`. `task_id`
|
||||
référence l'état générique Task et `selected_execution_id` est unique ; la
|
||||
ligne ne copie aucun Capture, asset, image, chemin, groupe ni curseur. La
|
||||
migration v23→v24 est transactionnelle et DDL-only : elle ne crée aucune
|
||||
association historique et son marqueur v24 est son point de publication.
|
||||
|
||||
`raw.develop.batch/1` recharge le curseur de `selected_executions` à chaque
|
||||
fenêtre admise. Après jointure de tous les participants, seul le callback Task
|
||||
propriétaire appelle `record_selected_representation` en ordre croissant. Le
|
||||
commit item+curseur précède toujours la progression/checkpoint générique ; un
|
||||
crash peut donc laisser celle-ci en retard, jamais en avance. Une reprise repart
|
||||
du curseur sélectionné durable sans deviner l'identité d'une sortie RAW déjà
|
||||
publiée. Le schéma et le chemin historique `raw.develop/1` restent valides.
|
||||
|
||||
### Overlay v25 — extraction Feature sélectionnée bornée
|
||||
|
||||
**IMPLEMENTED / VALIDATION IN PROGRESS.** La migration transactionnelle
|
||||
v24→v25 ajoute uniquement `feature_extract_batch_tasks`. Sa ligne lie un Task
|
||||
`features.extract.batch/1` à une exécution sélectionnée immutable, retient le
|
||||
préfixe monotone `next_item_index` et copie le domaine ORB exact
|
||||
kind/version/paramètres/fingerprint. La migration est DDL-only : elle ne crée
|
||||
aucune ligne, ne convertit aucune Task `features.extract/1` historique et
|
||||
n'infère aucun image ID ou Feature Set.
|
||||
|
||||
Les participants CPU préparent des images sélectionnées indépendantes sans
|
||||
accès SQLite. Après leur jointure, le propriétaire publie les Feature Sets dans
|
||||
l'ordre sélectionné, avance un item seulement si le Feature Set READY exact est
|
||||
durable, puis checkpoint la progression générique. Un crash peut donc laisser
|
||||
le checkpoint générique ou le curseur Feature en retard sur un Feature Set
|
||||
immutable déjà publié ; la reprise revalide ce résultat exact et converge sans
|
||||
deviner une identité. Les formats, keypoints, descripteurs ORB U8×32,
|
||||
fingerprint et Feature Store FROZEN restent inchangés.
|
||||
|
||||
La validation a migré via l'API production des copies des projets réels S21 et
|
||||
A6000 : version 23, `integrity_check` et clés étrangères propres, comptes et
|
||||
lignes scientifiques inchangés, neuf tables optiques vides et SHA-256 des
|
||||
|
|
|
|||
|
|
@ -5,12 +5,13 @@
|
|||
**ACCEPTED** — architecture decision for the post-Gate C documentation freeze.
|
||||
|
||||
Current Sparse SfM gates A through G are **PASS / FROZEN**.
|
||||
Project Database: current schema **v23**; the v22 foundation and historical
|
||||
Project Database: current schema **v24**; the v22 foundation, v23 optical overlay, and historical
|
||||
v16 science remain frozen.
|
||||
|
||||
This record is normative for the separation of responsibilities. Its original
|
||||
Gate C/G decision introduced no implementation or syntax. Later explicitly
|
||||
authorized additions, including the additive Project DB v23 optics overlay and
|
||||
authorized additions, including the additive Project DB v23 optics overlay,
|
||||
the v24 RAW batch Task association, and
|
||||
the bounded UDisks2 SSD controller, must remain on their own side of this
|
||||
boundary and do not create a generic Resource System.
|
||||
|
||||
|
|
|
|||
|
|
@ -268,7 +268,7 @@ et Visual Index restent CPU; SIFT/RootSIFT Matcher restent BFMatcher L2 CPU. Il
|
|||
n'est introduit ni second scheduler, ni Queue, ni daemon, ni sous-système de
|
||||
ressources.
|
||||
|
||||
### Audit des 14 kinds de production
|
||||
### Audit des 16 kinds de production
|
||||
|
||||
Tous les kinds passent par l'unique Queue et l'unique Governor, y compris ceux
|
||||
dont toutes les dimensions sont fixes. Dans le tableau, `CPU 1..N` décrit la
|
||||
|
|
@ -283,14 +283,16 @@ allocations driver opaques ne reçoivent pas un coût inventé.
|
|||
| Kind v1 | Estimation courante | Dimension consommée et raison |
|
||||
| --- | --- | --- |
|
||||
| `raw.develop` | MIXED; CPU 1; lot 1; hôte `2 Gio + contexte`, 0/item; I/O 1; GPU 0 | Un Capture atomique. Le garde applique/restaure CPU1 au pool OpenCV global. Les 2 Gio sont une allowance de travail opérationnelle, pas une limite de dataset. |
|
||||
| `raw.develop.batch` | MIXED; CPU 1..8; lot 1..8; hôte 0 fixe + 896 Mio/item (contexte propriétaire, workspace LibRaw 40 MP, copies RGB/BGR, PNG/validation, pile enfant 1 Mio et marge allocateur/codec); I/O 1; GPU 0 | Une fenêtre de Captures indépendants. `896 Mio × 8 = 7 Gio`, donc le budget post-réserve de l'hôte de référence peut admettre la fenêtre sûre complète. OpenCV reste à 1 thread interne, tous les enfants sont joints, puis le propriétaire publie le préfixe ordonné avant `sequence_break`. Le plafond est opérationnel, jamais une limite scientifique de sélection. |
|
||||
| `photo_quality.triage` | IMPORT; CPU 1; lot 1; hôte `contexte retenu + 20 Mio`, 0/item; I/O 1; GPU 0 | Un groupe par séquence ; garde OpenCV CPU1. Aucun scaling utile déterministe n'est acquis. |
|
||||
| `acquisition_campaign.run` | JPEG: IMPORT ; RAW: MIXED. CPU 1; lot 1; hôte `contexte retenu + requête transitoire exacte + 256 Kio` + 64 Kio/item ; DEVELOP_RAW ajoute 2 Gio; I/O 1; GPU 0 | Un groupe S3-E par séquence, sans Task imbriqué. La création et la reprise dérivent la même estimation ; la forme historique exacte est normalisée seulement en mémoire. |
|
||||
| `import.images` | IMPORT; CPU 1; lot 1..32; hôte 128 Kio + `NAME_MAX+64`/item; I/O 1; GPU 0 | Copie/hash I/O-bound. Le callback consomme le lot admis et réadmet entre lots. |
|
||||
| `features.extract` | CPU; CPU 1..compute-pool; lot 1; hôte 64 Mio + 512 Mio/item; I/O 1; GPU 0 | La demande durable emploie le maximum `int` positif de l'API OpenCV ; le Governor la borne à l'hôte. Le garde applique/restaure exactement le CPU admis. |
|
||||
| `features.extract.batch` | CPU; CPU 1..12; lot 1..12; hôte 64 Mio + 512 Mio/item; I/O 1; GPU 0 | Images sélectionnées indépendantes, OpenCV CPU1 par participant, enfants joints et publication owner-only ordonnée. Le plafond 12 est une capacité opérationnelle de mesure ; le feedback ≥5 % établit le palier utile sans devenir une limite scientifique. |
|
||||
| `features.extract.sift` | CPU; CPU 1..compute-pool; lot 1; hôte 64 Mio + 1 Gio/item; I/O 1; GPU 0 | Même contrat OpenCV. Les formes CPU12 et CPU1 historiques complètes sont acceptées et normalisées en mémoire. |
|
||||
| `features.extract.rootsift` | CPU; CPU 1..compute-pool; lot 1; hôte 64 Mio + 1 Gio/item; I/O 1; GPU 0 | Même contrat que SIFT ; aucune couture GPU scientifiquement compatible n'est validée. |
|
||||
| `visual_index.update` | CPU; CPU 1..16; lot 1..16; hôte 8 Mio + 2 Mio/item; I/O 1; GPU 0 | Le segment contient au plus 16 Feature Sets indépendants. Au plus `cpu_threads-1` enfants sont joints avant publication owner-only. |
|
||||
| `candidate_pair.generate` | CPU; CPU 1..64; lot 1..64; hôte 256 Kio + 8 Mio/item; I/O 1; GPU 0 | Le batch de 64 est la borne algorithmique et de ressources. Fenêtre `min(2*CPU, 64, reste_du_lot)`, un handle DB privé par participant, piles enfants de 4 Mio facturées. |
|
||||
| `candidate_pair.generate` | CPU; CPU 1..64; lot 1..64; hôte 256 Kio + 8 Mio/item; I/O 1; GPU 0 | Le batch de 64 est la borne algorithmique et de ressources. CPU/lot sont couplés pour que chaque palier de feedback exerce ses participants; fenêtre `min(2*CPU, 64, reste_du_lot)`, un handle DB privé par participant, piles enfants de 4 Mio facturées. |
|
||||
| `matcher.run` | CPU: CPU 1..12, lot sûr 1..12, hôte 0 + 10 Mio/item, I/O 1, GPU 0. ORB Vulkan AUTO: CPU 1, lot utile 1..8, même hôte/item, GPU 1 + 640 Kio, inflight 1. | Le batch/participant 12 est une borne intrinsèque mesurée du Matcher. Batch 12 et depth 2 (1,25 Mio) restent sûrs pour preuves privées, mais insuffisamment utiles en AUTO. CPU complet est le fallback. |
|
||||
| `geometric_verifier.run` | CPU; CPU utile 1..8, fenêtre sûre 16; lot 1..16; hôte 0 + 8 Mio/item; I/O 1; GPU 0 | Des parents indépendants sont préparés en parallèle, puis publiés/checkpointés en ordre par le propriétaire. L'USAC scientifique conserve `isParallel=false`. |
|
||||
| `track_builder.run` | CPU; CPU 1; lot 1; fixe `(4 Mio + arêtes * (48 + 2*160)) * facteur`, facteur 2 jusqu'à 400k arêtes puis 8; 0/item; I/O 1; GPU 0 | Rebuild DSU atomique et publication owner-only ; aucune partition scientifiquement validée n'est acquise. |
|
||||
|
|
@ -422,7 +424,7 @@ retirer l'autorité mais ne peut jamais recopier un ancien compte physique.
|
|||
Le teardown production suit strictement : Queue détruite/jointe et chaque
|
||||
lease Task rendu, worker SSD joint puis unregister vérifié, contrôleur détruit,
|
||||
Governor détruit. Un unregister est refusé tant qu'une opération wrapper ou un
|
||||
lease exact subsiste. Les quatorze Task kinds courants n'ont aucun consommateur
|
||||
lease exact subsiste. Les seize Task kinds courants n'ont aucun consommateur
|
||||
scratch : le compte normal est donc réellement zéro et la capacité disponible
|
||||
n'autorise aucun usage implicite. Une future Task consommatrice devra définir
|
||||
son propre contrat d'éligibilité et son ownership sans transformer scratch ou
|
||||
|
|
|
|||
|
|
@ -44,18 +44,20 @@ métier, et toute forme voisine est rejetée.
|
|||
## Inventaire production et entrées runtime
|
||||
|
||||
`src/task_kinds.c::lardon3d_task_kind_registry_production()` enregistre les
|
||||
14 kinds v1 du profil de production courant. La colonne « reprise » nomme le
|
||||
16 kinds v1 du profil de production courant. La colonne « reprise » nomme le
|
||||
reconstructeur du binding ; « callback » nomme l'entrée runtime privée dans le
|
||||
même fichier. Le détail chiffré des capacités est centralisé dans l'[audit des
|
||||
14 kinds](resource_governor.md#audit-des-14-kinds-de-production).
|
||||
16 kinds](resource_governor.md#audit-des-15-kinds-de-production).
|
||||
|
||||
| Kind v1 | Source, reprise et callback | Réconciliation pré-admission courante |
|
||||
| --- | --- | --- |
|
||||
| `raw.develop` | `raw_development_task.cpp`; `lardon3d_raw_development_task_reconstruct`; `run` | Aucune |
|
||||
| `raw.develop.batch` | `raw_development_task.cpp`; `lardon3d_raw_development_batch_task_reconstruct`; `run_batch` | Association v24 Task → exécution sélectionnée ; le curseur reste dans l'exécution |
|
||||
| `photo_quality.triage` | `photo_quality_task.cpp`; `lardon3d_photo_quality_task_reconstruct`; `run` | Aucune |
|
||||
| `acquisition_campaign.run` | `acquisition_campaign_task.cpp`; `lardon3d_acquisition_campaign_task_reconstruct`; `run` | Forme courante ou forme v22 exacte vérifiée contre la requête immuable → capacité courante en mémoire |
|
||||
| `import.images` | `import_task.c`; `lardon3d_image_import_reconstruct`; `run_image_import` | Aucune |
|
||||
| `features.extract` | `feature_task.c`; `lardon3d_feature_extract_reconstruct`; `run` | Formes CPU12/CPU1 historiques exactes → demande OpenCV portable ; runtime borné au compute-pool |
|
||||
| `features.extract.batch` | `feature_task.c`; `lardon3d_feature_extract_batch_reconstruct`; `run_batch` | Association v25 à l'exécution sélectionnée, domaine ORB exact et curseur typé ; CPU/lot bornés 1..12 |
|
||||
| `features.extract.sift` | `sift_task.c`; `lardon3d_sift_extract_reconstruct`; `run` | CPU12/CPU1 historiques exacts → demande OpenCV portable ; runtime borné au compute-pool |
|
||||
| `features.extract.rootsift` | `sift_task.c`; `lardon3d_sift_extract_reconstruct`; `run` | Même réconciliation SIFT ; aucune voie GPU validée |
|
||||
| `visual_index.update` | `visual_index_task.c`; `lardon3d_visual_index_update_reconstruct`; `run` | Formes CPU12/CPU1 historiques exactes → CPU/lot 1..16 |
|
||||
|
|
|
|||
|
|
@ -12,6 +12,104 @@ doivent rester incrémentaux, bornés, reprenables, durables et admis par l'uniq
|
|||
Resource Governor : aucune étape ne peut supposer qu'une campagne entière tient
|
||||
en RAM ou termine dans une seule vie de processus.
|
||||
|
||||
## POLITIQUE CANONIQUE D'UTILISATION DES RESSOURCES — HUMAN AUTHORITY
|
||||
|
||||
La politique d'exploitation est désormais explicite : **Lardon3D doit viser le
|
||||
MAXIMUM SAFE USEFUL THROUGHPUT** sur chaque Task de production.
|
||||
|
||||
Le Resource Governor réserve d'abord uniquement les ressources nécessaires pour
|
||||
que le poste reste normalement utilisable pendant les calculs :
|
||||
|
||||
- Arch Linux / Sway et les services normaux du bureau ;
|
||||
- Firefox ;
|
||||
- lecture audio/musique normale ;
|
||||
- usage interactif léger.
|
||||
|
||||
Tout CPU, RAM, capacité I/O et accélérateur validé restant appartient au travail
|
||||
Lardon3D lorsqu'un travail utile existe. La stabilité du poste est protégée par
|
||||
cette réserve interactive et par les signaux de pression ; elle ne justifie plus
|
||||
de laisser des ressources sûres et utiles inactives par conservatisme historique.
|
||||
|
||||
Sur l'hôte de référence actuel, le résultat normal de cette politique est
|
||||
approximativement :
|
||||
|
||||
```text
|
||||
16 CPU logiques au total
|
||||
4 CPU logiques réservés à l'hôte interactif
|
||||
12 CPU logiques disponibles au compute-pool
|
||||
~3 GiB de MemAvailable conservés comme réserve RAM dure
|
||||
Radeon 780M UMA disponible pour les backends GPU validés et utiles
|
||||
```
|
||||
|
||||
Ces nombres sont des **résultats de l'hôte courant**, jamais des constantes
|
||||
produit. Un futur hôte 32 threads ne doit pas hériter d'un plafond 12 ; le
|
||||
Governor doit dériver sa réserve et donner le reste au calcul selon topologie,
|
||||
affinité, pression et enveloppe réelle de la Task.
|
||||
|
||||
La règle canonique est :
|
||||
|
||||
```text
|
||||
SERIALISM_REQUIRES_PROOF
|
||||
```
|
||||
|
||||
Une opération atomique par item n'impose pas de sérialiser les items
|
||||
indépendants. Une publication owner-only ou ordonnée n'impose pas de sérialiser
|
||||
la préparation. Lorsqu'il existe plusieurs unités indépendantes et que la
|
||||
science, la persistance et la reprise restent exactes, la Task doit exposer au
|
||||
Governor un parallélisme borné.
|
||||
|
||||
Un chemin long `CPU1` ou `batch1` avec travail indépendant disponible et
|
||||
ressources sûres libres est désormais considéré comme un **défaut opérationnel**
|
||||
tant qu'une preuve concrète ne démontre pas l'une des limites suivantes :
|
||||
|
||||
- dépendance scientifique ou algorithmique réellement sérielle ;
|
||||
- genou de scaling mesuré ;
|
||||
- limite mémoire ;
|
||||
- saturation I/O ;
|
||||
- backend GPU validé rendant des CPU supplémentaires inutiles ;
|
||||
- contrainte de publication qui ne peut pas être séparée de la préparation sans
|
||||
casser le déterminisme ;
|
||||
- autre limitation matérielle mesurée et documentée.
|
||||
|
||||
Le Governor reste l'unique autorité de ressources. L'utilisateur normal ne
|
||||
choisit ni CPU, ni workers, ni lot, ni inflight, ni GPU, ni scratch, ni budget
|
||||
RAM. Une Task décrit ses minimum/useful/safe, ses coûts fixes/transitoires/par
|
||||
participant et ses capacités GPU/I/O ; le Governor choisit le maximum
|
||||
sûr **et utile** disponible à l'instant.
|
||||
|
||||
La politique GPU est symétrique : un backend de production **VALIDATED AND
|
||||
USEFUL** doit être préféré lorsqu'il est disponible et sûr ; aucun backend GPU
|
||||
non validé ne doit être inventé pour afficher de l'activité GPU. Candidate,
|
||||
Visual Index ou GV peuvent donc rester CPU lorsque leur décision mesurée le
|
||||
justifie, tandis qu'ORB Matcher Vulkan reste GPU-first lorsqu'il est éligible.
|
||||
|
||||
La RAM doit être utilisée agressivement jusqu'à la réserve interactive ;
|
||||
`swap`, zram et scratch ne sont jamais de la RAM admise, et l'UMA est comptée
|
||||
exactement une fois contre la RAM hôte. Les signaux PSI et les deltas actifs de
|
||||
swap peuvent réduire l'admission ; après disparition de la pression, le
|
||||
Governor doit réadmettre les ressources utiles au lieu de rester durablement
|
||||
bridé.
|
||||
|
||||
Cette politique s'applique aussi au travail d'ingénierie Codex : builds, tests,
|
||||
benchmarks et preuves réelles indépendantes doivent utiliser le parallélisme
|
||||
sûr disponible plutôt qu'un `-j8`/`--num-processes 1` historique. Un run CPU1
|
||||
reste légitime comme cohorte de mesure ou lorsqu'une dépendance concrète
|
||||
l'impose, mais jamais comme défaut universel. Les rebuilds, tests lourds,
|
||||
revues ou agents coûteux déjà acquis et non affectés par le delta ne doivent pas
|
||||
être relancés sans raison. La règle reste : **WORK FIRST, RETURN LAST**, avec
|
||||
revue delta-based depuis le checkpoint de maintenance.
|
||||
|
||||
Statut de cette décision :
|
||||
|
||||
```text
|
||||
RESOURCE_UTILIZATION_POLICY=HUMAN_AUTHORITY
|
||||
SERIALISM_REQUIRES_PROOF=CANONICAL
|
||||
IMPLEMENTATION_AUDIT=REQUIRED_BEFORE_NEXT_LONG_REAL_RUN
|
||||
```
|
||||
|
||||
Cette autorité rouvre uniquement les **contrats opérationnels de ressources**
|
||||
quand ils empêchent cette politique. Elle ne rouvre aucune science FROZEN.
|
||||
|
||||
### Fondations PASS / FROZEN
|
||||
|
||||
- Gates A–G Sparse SfM, dont F0, orchestration Task/Project DB et politique
|
||||
|
|
@ -26,9 +124,11 @@ en RAM ou termine dans une seule vie de processus.
|
|||
- Task Runtime, checkpoints atomiques, Queue et Resource Governor :
|
||||
[Task](../architecture/task_system.md), [Queue](../architecture/task_queue.md)
|
||||
et [Governor](../architecture/resource_governor.md).
|
||||
- Project DB v22 gelé, overlay optique additif v23 courant, et S1–S3 Capture / Acquisition Ingestion : provenance
|
||||
Capture/Asset, import capture-safe, publication dérivée, développement RAW,
|
||||
siblings multi-source, évidence S3-D, orchestration S3-E et campagne bornée :
|
||||
- Project DB v22 gelé, overlay optique additif v23 gelé et couche opérationnelle
|
||||
additive v24 actuellement en validation pour `raw.develop.batch/1`, ainsi que
|
||||
S1–S3 Capture / Acquisition Ingestion : provenance Capture/Asset, import
|
||||
capture-safe, publication dérivée, développement RAW, siblings multi-source,
|
||||
évidence S3-D, orchestration S3-E et campagne bornée :
|
||||
[Project DB et ingestion](../architecture/project_database.md).
|
||||
|
||||
### S3 Capture / Acquisition Ingestion — PASS / FROZEN
|
||||
|
|
@ -71,8 +171,7 @@ Campagne `Photogrammetrie/2026-08-26_Baie_Moteur_A6000` :
|
|||
- plan identique après inversion de l'ordre des racines.
|
||||
|
||||
Le stem commun ne prouve jamais une acquisition. Les 953 propositions doivent
|
||||
être confirmées et deviennent `CALLER_EXPLICIT`, jamais `STRONG`. La campagne
|
||||
complète n'a pas encore été matérialisée.
|
||||
être confirmées et deviennent `CALLER_EXPLICIT`, jamais `STRONG`.
|
||||
|
||||
### Limite de reprise DB v19 (historique)
|
||||
|
||||
|
|
@ -232,13 +331,20 @@ responsabilités. L'overlay optique Project DB v23, ajouté ultérieurement, ne
|
|||
réinterprète pas cette preuve v22 ; les copies des deux projets migrent avec les
|
||||
comptes scientifiques inchangés et les nouvelles tables optiques vides.
|
||||
|
||||
Pour A6000, le JPEG caméra pairé reste un SOURCE valide et le proxy rapide de
|
||||
Photo Quality, mais la représentation géométrique FROZEN de cette exécution
|
||||
sélectionnée reste le PNG déterministe dérivé du RAW (`L3DRAWD1`). Passer cette
|
||||
preuve au JPEG caméra nécessiterait une tranche scientifique/calibration séparée
|
||||
prouvant l'équivalence exacte ; ce changement n'est pas une optimisation
|
||||
opérationnelle ordinaire.
|
||||
|
||||
Le Sparse SfM réel final reste `BLOCKED_BY_KNOWN_CALIBRATION_DATA` pour ces
|
||||
campagnes ; aucune pseudo-calibration, interpolation de métadonnées ou inférence
|
||||
d'identité ne le contourne. Dense, mesh et publication aval ne font pas partie de
|
||||
ce jalon pré-SfM. Ce milestone est une intégration réelle, pas une nouvelle
|
||||
série de micro-gates S3. Son statut acquis ne rend pas disponibles les données
|
||||
de calibration physique manquantes et n'autorise pas à devancer la maintenance
|
||||
globale.
|
||||
de calibration physique manquantes et n'autorise pas à devancer les dépendances
|
||||
scientifiques.
|
||||
|
||||
## INTERNAL PARALLELISM + COMPUTE RESOURCES v1 — PASS / FROZEN
|
||||
|
||||
|
|
@ -273,83 +379,76 @@ reproductibilité. Les dimensions retenues et toutes les validations v2 sont
|
|||
closes.
|
||||
|
||||
Le pool CPU12 est validé comme preuve de cet hôte, pas comme plafond produit.
|
||||
Le Governor dérive désormais le pool lourd depuis le masque
|
||||
permis et les groupes package/core/SMT. Sur l'hôte unrestricted courant, il
|
||||
obtient `0-5,8-13` et réserve `6,7,14,15`; un caller déjà précontraint ne subit
|
||||
pas une seconde réserve. Le worker Queue seul applique/vérifie son propre
|
||||
masque ; aucun TID auxiliaire énuméré n'est muté, car un pidfd ne stabilise pas
|
||||
le numéro consommé par `sched_setaffinity(tid)`. Avant tout pthread applicatif
|
||||
ou driver, le démarrage établit `MESA_SHADER_CACHE_DISABLE=true`; une valeur
|
||||
absente prend ce défaut, `true`/`1` explicites sont conservées et toute autre
|
||||
valeur explicite est préservée mais refusée. Cette politique non scientifique
|
||||
supprime les helpers de cache Mesa observés qui élargissaient leur masque. Les
|
||||
threads runtime restants héritent le compute-pool ; il n'existe plus de sweep,
|
||||
latch ou retry auxiliaire et le diagnostic expose la politique réelle.
|
||||
Creator/main/TUI reste unrestricted. Les CPU déjà exclus de l'affinité du
|
||||
processus comptent dans la réserve hôte ; le fallback count-only ne fabrique
|
||||
aucun masque. Le fallback au budget portable ne crée aucune exclusion
|
||||
inventée. Cette couture est **PASS / FROZEN** sur le profil validé; ces IDs ne
|
||||
sont pas une politique portable. La cible RAM conserve une réserve dure de
|
||||
3 GiB de `MemAvailable`; la zone 3–4 GiB est une prudence qui ne soustrait pas
|
||||
4 GiB à toute capacité. Les petits hôtes dégradent le budget en conservant au
|
||||
moins une unité de calcul. Les PSI CPU/mémoire/I/O et les deltas
|
||||
swap-in/swap-out sont des signaux
|
||||
actifs ; l'occupation totale du swap reste historique. Admission CPU et lot
|
||||
sont indépendantes. Sur la 780M, Hardware Profile classe conservativement comme
|
||||
UMA le petit aperture VRAM amdgpu de 512 Mio accompagné d'environ 7,99 Go de
|
||||
GTT système ; la capacité rapportée reste observable mais ne devient pas un
|
||||
budget séparé. Les coûts GPU sont débités exactement une fois de la RAM hôte.
|
||||
Le Governor dérive désormais le pool lourd depuis le masque permis et les
|
||||
groupes package/core/SMT. Sur l'hôte unrestricted courant, il obtient
|
||||
`0-5,8-13` et réserve `6,7,14,15`; un caller déjà précontraint ne subit pas une
|
||||
seconde réserve. Le worker Queue seul applique/vérifie son propre masque ; aucun
|
||||
TID auxiliaire énuméré n'est muté. Creator/main/TUI reste unrestricted. Les CPU
|
||||
déjà exclus de l'affinité du processus comptent dans la réserve hôte ; le
|
||||
fallback count-only ne fabrique aucun masque. La cible RAM conserve une réserve
|
||||
dure de 3 GiB de `MemAvailable`; la zone 3–4 GiB est une prudence qui ne
|
||||
soustrait pas 4 GiB à toute capacité. Les petits hôtes dégradent le budget en
|
||||
conservant au moins une unité de calcul. Les PSI CPU/mémoire/I/O et les deltas
|
||||
swap-in/swap-out sont des signaux actifs ; l'occupation totale du swap reste
|
||||
historique. Admission CPU et lot sont indépendantes. Sur la 780M, Hardware
|
||||
Profile classe conservativement comme UMA le petit aperture VRAM amdgpu de
|
||||
512 Mio accompagné d'environ 7,99 Go de GTT système ; la capacité rapportée
|
||||
reste observable mais ne devient pas un budget séparé. Les coûts GPU sont
|
||||
débités exactement une fois de la RAM hôte.
|
||||
|
||||
L'audit Phase 1 couvre les 14 kinds de production et sépare leurs dimensions
|
||||
fixes des dimensions réellement adaptables. Il confirme que tous passent par
|
||||
l'unique Governor, même les formes fixes, et que le contrat reste immutable
|
||||
pendant une séquence. Cette tranche ferme l'enveloppe privée, la sélection AUTO,
|
||||
les diagnostics bornés, l'enforcement OpenCV borné au compute-pool, la politique
|
||||
d'affinité privée et la réconciliation du contexte retenu des campagnes
|
||||
nouvelles ou restaurées. La création/reprise AUTO ne touche plus Vulkan sur le
|
||||
main ; le premier begin appartient au worker contraint. Inflight ORB normal est
|
||||
maintenant fixé à 1, helpers reste 0; depth 2 reste une capacité privée de
|
||||
sûreté/benchmark. La clôture v2 est acquise. Ces choix ne créent aucune limite
|
||||
de dataset, identité scientifique ou version Project DB.
|
||||
La nouvelle autorité humaine `SERIALISM_REQUIRES_PROOF` **supersède toute
|
||||
interprétation** de l'inventaire de maintenance qui ferait d'un ancien
|
||||
`CPU1`/`batch1` un plafond permanent. Les preuves scientifiques et résultats de
|
||||
la maintenance restent FROZEN ; seules les enveloppes opérationnelles peuvent
|
||||
être rouvertes lorsqu'un travail indépendant est inutilement sérialisé. Une
|
||||
Task fixe demeure fixe seulement si sa sérialité ou son genou de scaling est
|
||||
prouvé. Cette règle doit être appliquée à toutes les Tasks de production avant
|
||||
d'engager de nouvelles preuves réelles longues.
|
||||
|
||||
La signature durable des nouvelles Tasks ORB normales est maintenant la classe
|
||||
`MIXED`, sémantiquement réelle pour une politique susceptible d'exécuter CPU ou
|
||||
Vulkan. Elle reconstruit AUTO ; toutes les signatures CPU anciennes/courantes
|
||||
restent CPU fixes et Vulkan reste fixe, sans migration DB/codec. La couture
|
||||
asynchrone est privée, request-bound et nettoie son slot sur toute sortie. Une
|
||||
preuve événementielle bornée établit la soumission du successeur avant la
|
||||
publication du prédécesseur pour deux paires 769×769. La rampe ne croît plus sur
|
||||
la seule santé : l'adaptation générique exige deux observations par fenêtre,
|
||||
tandis que le lot ORB Vulkan en exige huit, avec retour au palier accepté sans
|
||||
gain et reset immédiat sous pression. Ces
|
||||
éléments sont **PASS / FROZEN** dans les limites validées.
|
||||
L'audit Phase 1 couvre les kinds de production de l'époque et sépare leurs
|
||||
dimensions fixes des dimensions réellement adaptables. Il confirme que tous
|
||||
passent par l'unique Governor, même les formes fixes, et que le contrat reste
|
||||
immutable pendant une séquence. Cette tranche ferme l'enveloppe privée, la
|
||||
sélection AUTO, les diagnostics bornés, l'enforcement OpenCV borné au
|
||||
compute-pool, la politique d'affinité privée et la réconciliation du contexte
|
||||
retenu des campagnes nouvelles ou restaurées. La création/reprise AUTO ne
|
||||
touche plus Vulkan sur le main ; le premier begin appartient au worker
|
||||
contraint. Inflight ORB normal est fixé à 1, helpers reste 0 ; depth 2 reste une
|
||||
capacité privée de sûreté/benchmark. Ces choix ne créent aucune limite de
|
||||
dataset, identité scientifique ou version Project DB.
|
||||
|
||||
Les diagnostics de séquence distinguent maintenant backend sélectionné et
|
||||
backend réel ; les participants CPU Matcher restent `cpu_threads` et
|
||||
`helpers=0`. Une paire Vulkan inéligible ou en panne est recalculée entièrement
|
||||
sur CPU sans preuve partielle. Les extractions ORB/SIFT/RootSIFT testées à
|
||||
1/2/4/8/12 consomment leur contrat OpenCV immutable avec sorties égales.
|
||||
Le rolling distingue désormais un handle soumis d'une inéligibilité locale et
|
||||
n'appelle jamais `finish` sans requête. La panne backend invalide immédiatement
|
||||
l'admission partagée sur toute sortie précoce ; seules les reprises AUTO peuvent
|
||||
établir cette disponibilité, indépendamment de l'ordre des reprises fixes ou
|
||||
historiques. Le statut est **PASS / FROZEN**.
|
||||
La signature durable des nouvelles Tasks ORB normales est la classe `MIXED`,
|
||||
sémantiquement réelle pour une politique susceptible d'exécuter CPU ou Vulkan.
|
||||
Elle reconstruit AUTO ; toutes les signatures CPU anciennes/courantes restent
|
||||
CPU fixes et Vulkan reste fixe, sans migration DB/codec. La couture asynchrone
|
||||
est privée, request-bound et nettoie son slot sur toute sortie. Une preuve
|
||||
événementielle bornée établit la soumission du successeur avant la publication
|
||||
du prédécesseur pour deux paires 769×769. La rampe ne croît plus sur la seule
|
||||
santé : l'adaptation générique exige deux observations par fenêtre, tandis que
|
||||
le lot ORB Vulkan en exige huit, avec retour au palier accepté sans gain et
|
||||
reset immédiat sous pression. Ces éléments sont **PASS / FROZEN** dans les
|
||||
limites validées.
|
||||
|
||||
La boucle privée mesure désormais l'utilisation `/proc/stat` du compute-pool,
|
||||
Les diagnostics de séquence distinguent backend sélectionné et backend réel ;
|
||||
les participants CPU Matcher restent `cpu_threads` et `helpers=0`. Une paire
|
||||
Vulkan inéligible ou en panne est recalculée entièrement sur CPU sans preuve
|
||||
partielle. Les extractions ORB/SIFT/RootSIFT testées à 1/2/4/8/12 consomment
|
||||
leur contrat OpenCV immutable avec sorties égales. Le rolling distingue un
|
||||
handle soumis d'une inéligibilité locale et n'appelle jamais `finish` sans
|
||||
requête. La panne backend invalide immédiatement l'admission partagée sur toute
|
||||
sortie précoce ; seules les reprises AUTO peuvent établir cette disponibilité,
|
||||
indépendamment de l'ordre des reprises fixes ou historiques. Le statut est
|
||||
**PASS / FROZEN**.
|
||||
|
||||
La boucle privée mesure l'utilisation `/proc/stat` du compute-pool,
|
||||
`MemAvailable`, PSI mémoire/I/O `some/full`, deltas swap actifs, RSS/HWM observé
|
||||
et GPU busy DRM, avec `unknown` sur absence ou parse non strict. Le backend
|
||||
Vulkan fournit des compteurs cumulatifs bornés de submit/complétion/fence/
|
||||
readback/GPU/starvation/panne/discard ; Matcher agrège en plus CPU et publication
|
||||
par séquence. Le diagnostic est tirable par numéro de série, sans log ncurses ni
|
||||
histoire persistée. Les CPU réductibles progressent par puissances de deux vers
|
||||
la capacité exacte du kind/compute-pool après deux observations de baseline et
|
||||
deux gains d'au moins 5 % ; CPU et lot ne changent
|
||||
jamais dans le même essai. Sous pression, une admission adaptative encore
|
||||
permise réserve immédiatement CPU1 et lot minimum. Feature/SIFT/RootSIFT ne
|
||||
comptent un item qu'après extraction et publication durable propre ; READY,
|
||||
`ALREADY_PRESENT` et publication incertaine comptent zéro, comme un segment
|
||||
Visual Index non durable.
|
||||
Candidate mesure chaque séquence. Un
|
||||
Vulkan fournit des compteurs cumulés bornés de submit/complétion/fence/readback/
|
||||
GPU/starvation/panne/discard ; Matcher agrège en plus CPU et publication par
|
||||
séquence. Le diagnostic est tirable par numéro de série, sans log ncurses ni
|
||||
histoire persistée. Les CPU réductibles progressent vers la capacité exacte du
|
||||
kind/compute-pool selon les observations de scaling ; sous pression, l'admission
|
||||
se réduit puis doit pouvoir remonter. Candidate mesure chaque séquence. Un
|
||||
fallback réel annule l'essai Vulkan au lieu d'empoisonner sa baseline. Cette
|
||||
implémentation est **PASS / FROZEN** sans ajouter de helper GPU.
|
||||
|
||||
|
|
@ -359,11 +458,11 @@ digest `7a9dbc38a23a600379167d55e24836b7acbb22eea25573e7440bdc9e4602b3b3`.
|
|||
La starvation depth 1 (53,847 s sur 74,613 s, GPU busy max 25 %) motive deux
|
||||
slots bornés mais ne constitue pas une mesure depth 2. Le backend partage
|
||||
device/pipeline/layout/cache et duplique seulement 640 Kio de payload,
|
||||
command/fence/descriptors/query par slot. Le payload mappé suit désormais
|
||||
exactement la capacité de séquence admise : zéro avant initialisation, 640 Kio
|
||||
à depth 1 et 1,25 Mio uniquement pendant un contrôle privé depth 2, avec retour
|
||||
à 640 Kio avant la prochaine admission depth 1. L'enveloppe normale n'essaie
|
||||
plus inflight 2. Les générations de requête ne bouclent pas; un slot épuisé est
|
||||
command/fence/descriptors/query par slot. Le payload mappé suit exactement la
|
||||
capacité de séquence admise : zéro avant initialisation, 640 Kio à depth 1 et
|
||||
1,25 Mio uniquement pendant un contrôle privé depth 2, avec retour à 640 Kio
|
||||
avant la prochaine admission depth 1. L'enveloppe normale n'essaie plus
|
||||
inflight 2. Les générations de requête ne bouclent pas ; un slot épuisé est
|
||||
retiré définitivement.
|
||||
|
||||
Le harness réel a exécuté le corpus 4113 paires pour le contrôle synchrone,
|
||||
|
|
@ -372,92 +471,42 @@ rolling depth 1 et l'A/B forcé depth 1/depth 2. L'ABBA forcé donne
|
|||
Chaque run porte 4113 paires durables ; le débit combiné est
|
||||
`(2 * 4113 * 1e9) / somme(wall_ns)`, pas la moyenne des débits par run. Les
|
||||
walls bruts 75326831673/75162582080 et 73662096698/73764360098 ns donnent les
|
||||
moyennes 75,244706877/73,713228398 s. Fence vaut 6,0684/3,6776 s, starvation 54,4534/50,1465 s,
|
||||
publication 29,2582/30,0548 s, submit CPU 0,2655/0,3818 s, readback
|
||||
0,0460/0,0873 s et GPU busy max 23/24 %. Les quatre exécutions conservent le
|
||||
digest `7a9dbc38a23a600379167d55e24836b7acbb22eea25573e7440bdc9e4602b3b3`, quatre
|
||||
séquences de fallback local par exécution et zéro panne/discard. Depth 2 est donc
|
||||
**REJECTED_WITH_MEASURED_REASON** pour la politique normale :
|
||||
`DEPTH_MAX_VALIDATED_SAFETY=2`, `DEPTH_MAX_USEFUL=1`. Une comparaison
|
||||
whole-corpus adaptative antérieure (+1,12 %) ne séparait pas les changements de
|
||||
contrat; l'ABBA ci-dessus la remplace comme décision de profondeur. Le
|
||||
répertoire retenu est
|
||||
`/home/fy59/Documents/Lardon/.real-pre-sfm-2026-08-30/governor-v2-evidence/`,
|
||||
avec `forced-depth1-a.stdout.jsonl`, `forced-depth1-b.stdout.jsonl`,
|
||||
`forced-depth2-a.stdout.jsonl` et `forced-depth2-b.stdout.jsonl`. Le
|
||||
mode Matcher par défaut est AUTO/rolling ; CPU et Vulkan explicites restent des
|
||||
overrides. Un contrôle synchrone fence par fence est compilé seulement dans le
|
||||
runner/test, absent de `lardon3d`, non persisté et non applicable à une reprise
|
||||
Matcher pendante. Les fixtures et ces deux exécutions établissent l'égalité
|
||||
exacte des sorties; les chiffres depth 1 sont rapportés ci-dessus. Le runner émet
|
||||
des diagnostics JSON échantillonnés sans prétendre voir chaque séquence, plus
|
||||
un agrégat Governor fixe exact pour les compteurs enregistrés. Il valide la
|
||||
bijection Candidate Pair/Match Result, les assets et le curseur, puis produit le
|
||||
digest canonique de comparaison `L3DMRD1`, qui exclut IDs Task, timestamps et
|
||||
choix opérationnels. Ces preuves ferment **PASS / FROZEN** sans promouvoir
|
||||
depth 2 dans AUTO normal.
|
||||
moyennes 75,244706877/73,713228398 s. Fence vaut 6,0684/3,6776 s, starvation
|
||||
54,4534/50,1465 s, publication 29,2582/30,0548 s, submit CPU 0,2655/0,3818 s,
|
||||
readback 0,0460/0,0873 s et GPU busy max 23/24 %. Les quatre exécutions
|
||||
conservent le digest `7a9dbc38a23a600379167d55e24836b7acbb22eea25573e7440bdc9e4602b3b3`,
|
||||
quatre séquences de fallback local par exécution et zéro panne/discard. Depth 2
|
||||
est donc **REJECTED_WITH_MEASURED_REASON** pour la politique normale :
|
||||
`DEPTH_MAX_VALIDATED_SAFETY=2`, `DEPTH_MAX_USEFUL=1`.
|
||||
|
||||
Le runner possède maintenant, dans ce seul target, `--matcher-inflight 1|2` :
|
||||
AUTO rolling fixe batch 2 et min=max inflight pour permettre la mesure du même
|
||||
binaire à profondeur 1 puis 2, en n'exposant que la capacité Vulkan forcée.
|
||||
`--matcher-batch 2|4|8|12`, valable seulement avec inflight et AUTO rolling,
|
||||
fixe aussi min=max batch pour la prochaine matrice contrôlée.
|
||||
Synchronous reste depth 1. Le Governor conserve son admission et sa charge UMA;
|
||||
GPU budget zéro ou capacité GPU/backend/mémoire indisponible échoue au lieu de
|
||||
sélectionner CPU. La valeur n'est ni persistée ni scientifique,
|
||||
est refusée sur une Task pendante, restaurée dans l'environnement à toute sortie
|
||||
et émise comme `1`, `2` ou `null` dans les résumés/agrégats. Le runner valide le
|
||||
contrat exact et zéro panne/discard/pending; seule l'inéligibilité locale peut
|
||||
produire un fallback CPU complet. La production compte désormais chaque item
|
||||
exactement une fois après publication durable dans local-ineligible,
|
||||
backend-failure ou other; une admission CPU normale reste à zéro. Le compteur
|
||||
est incrémenté immédiatement et reste visible si une paire suivante échoue ou est
|
||||
annulée, sans valider ni entraîner la séquence incomplète; son high-water mark
|
||||
de déduplication reste privé et non persisté. L'égalité
|
||||
inter-cohortes porte sur les items locaux, pas sur les séquences dont le nombre
|
||||
dépend du batch. Une panne/raison autre, ou une panne tardive, invalide la
|
||||
cohorte; la Task de benchmark échoue après checkpoint de toute preuve CPU déjà
|
||||
durable. Les fixtures
|
||||
établissent l'égalité exacte rolling1/rolling2/synchrone1 et des batches
|
||||
2/4/8/12 sur fixture, avec le même compte de 29 items locaux malgré des comptes
|
||||
de séquences différents. Les logs corpus préliminaires
|
||||
`forced-batch2-current.stdout.jsonl` et `forced-batch4.stdout.jsonl` couvrent
|
||||
les mêmes 4113 IDs/digest mais portent quatre contre trois séquences locales :
|
||||
ils sont conservés comme preuve que l'ancien comparateur était invalide et ne
|
||||
participent pas à la décision. Les huit runs item-valides
|
||||
`forced-batch{2,4,8,12}-items{,-b}.stdout.jsonl` publient chacun 4113 paires,
|
||||
six items locaux, zéro panne/autre et le même digest. Leurs débits combinés
|
||||
sont 54,180767704, 66,094373197, 74,784998723 et 76,755814095 paires/s. Batch
|
||||
4 puis 8 gagnent +21,988624373 % et +13,148812987 %; batch 12 ne gagne que
|
||||
+2,635308425 %, sous le deadband 5 %. AUTO normal suit donc
|
||||
`BATCH_MAX_USEFUL=8`; batch 12 reste une capacité privée sûre
|
||||
`REJECTED_WITH_MEASURED_REASON`.
|
||||
Le contrôle de production sans override
|
||||
`short-auto-batch8-governor-v2.stdout.jsonl` atteint réellement
|
||||
`1 → 2 → 4 → 8`, puis publie 4113/4113 résultats à 76,072 paires/s avec le
|
||||
même digest, six fallbacks locaux, zéro panne/discard, inflight 1 et helpers 0.
|
||||
La preuve de fermeture S21
|
||||
`final-s21-auto.stdout.jsonl` exécute ensuite le chemin production normal sur
|
||||
172 741 Candidate Pairs : 172 741 Match Results, zéro doublon, curseur complet,
|
||||
digest `e5128a2e599ff593c4f79850e067254b1f249d19e8480a44973306b1af250f70`
|
||||
et 73,649 résultats durables/s. AUTO choisit Vulkan sur toutes les admissions,
|
||||
Les huit runs item-valides `forced-batch{2,4,8,12}-items{,-b}.stdout.jsonl`
|
||||
publient chacun 4113 paires, six items locaux, zéro panne/autre et le même
|
||||
digest. Leurs débits combinés sont 54,180767704, 66,094373197, 74,784998723 et
|
||||
76,755814095 paires/s. Batch 4 puis 8 gagnent +21,988624373 % et
|
||||
+13,148812987 % ; batch 12 ne gagne que +2,635308425 %, sous le deadband 5 %.
|
||||
AUTO normal suit donc `BATCH_MAX_USEFUL=8`; batch 12 reste une capacité privée
|
||||
sûre `REJECTED_WITH_MEASURED_REASON`. Le contrôle de production sans override
|
||||
`short-auto-batch8-governor-v2.stdout.jsonl` atteint réellement `1 → 2 → 4 → 8`,
|
||||
puis publie 4113/4113 résultats à 76,072 paires/s avec le même digest, six
|
||||
fallbacks locaux, zéro panne/discard, inflight 1 et helpers 0.
|
||||
|
||||
La preuve de fermeture S21 `final-s21-auto.stdout.jsonl` exécute le chemin
|
||||
production normal sur 172 741 Candidate Pairs : 172 741 Match Results, zéro
|
||||
doublon, curseur complet, digest
|
||||
`e5128a2e599ff593c4f79850e067254b1f249d19e8480a44973306b1af250f70` et
|
||||
73,649 résultats durables/s. AUTO choisit Vulkan sur toutes les admissions,
|
||||
termine batch 8/inflight 1/helpers 0 et ne compte aucune panne/discard/pending.
|
||||
Une admission YELLOW réduit batch 8 à 1, puis les séquences GREEN rétablissent
|
||||
1 → 2 → 4 → 8; le gate possède donc aussi une preuve réelle de recovery.
|
||||
1 → 2 → 4 → 8 ; le gate possède donc une preuve réelle de recovery.
|
||||
|
||||
La poursuite S21 ferme ensuite le point Geometric Verifier v3 sans rejouer le
|
||||
Matcher. La Task 2832 consomme les 172 741 Match Results et publie 172 275 GVR
|
||||
v3, dont 24 065 acceptés et 148 210 rejetés, avec zéro doublon et curseur
|
||||
complet. Le digest Matcher reste
|
||||
`e5128a2e599ff593c4f79850e067254b1f249d19e8480a44973306b1af250f70`.
|
||||
La seconde reprise crée zéro ligne ; une Task interrompue sur une copie dédiée
|
||||
reprend le même ID et converge vers les mêmes lignes exactes. Feature,
|
||||
Candidate et Matcher ne sont pas rejoués. Track Builder et Sparse SfM ne sont
|
||||
pas exécutés. `REAL_S21_GV_V3=PASS/FROZEN` ; la prochaine tranche scientifique
|
||||
S21 commencera donc à Tracks maintenant que la gate de maintenance globale
|
||||
ci-dessous est fermée. Tracks n'est pas acquis.
|
||||
La poursuite S21 ferme le Geometric Verifier v3 sans rejouer le Matcher. La Task
|
||||
2832 consomme les 172 741 Match Results et publie 172 275 GVR v3, dont 24 065
|
||||
acceptés et 148 210 rejetés, avec zéro doublon et curseur complet. La seconde
|
||||
reprise crée zéro ligne ; une Task interrompue sur une copie dédiée reprend le
|
||||
même ID et converge vers les mêmes lignes exactes. Feature, Candidate et
|
||||
Matcher ne sont pas rejoués. `REAL_S21_GV_V3=PASS/FROZEN`.
|
||||
|
||||
Les gates de fermeture sont acquis :
|
||||
Les gates de fermeture acquises incluent :
|
||||
|
||||
```text
|
||||
GOVERNOR_CONTROLS_ALL_TASKS=PASS
|
||||
|
|
@ -489,39 +538,17 @@ REAL_S21_GV_V3=PASS/FROZEN
|
|||
FEATURE_REPLAY=0
|
||||
CANDIDATE_REPLAY=0
|
||||
MATCHER_REPLAY=0
|
||||
TRACKS_EXECUTED=0
|
||||
SPARSE_SFM_EXECUTED=0
|
||||
```
|
||||
|
||||
L'expérience normale est donc : l'utilisateur lance une Task ; l'unique
|
||||
Governor choisit et explique le contrat borné de sa prochaine séquence. Aucun
|
||||
réglage CPU/GPU/lot/inflight/helper n'est requis en production ordinaire.
|
||||
|
||||
L'ordre de travail autorisé à court terme est :
|
||||
|
||||
```text
|
||||
fondation scientifique pré-SfM courante
|
||||
→ INTERNAL PARALLELISM + COMPUTE RESOURCES v1
|
||||
(Candidate, Matcher, Visual Index, audit feature threading,
|
||||
correction progression/Resource Governor, audit GPU)
|
||||
→ COMPUTE GOVERNOR v2 / ORB VULKAN ASYNC EXECUTION (PASS / FROZEN)
|
||||
→ REAL S21 GV v3 (PASS / FROZEN; arrêt avant Tracks)
|
||||
→ GLOBAL MAINTENANCE AUDIT
|
||||
(PASS / FROZEN; validation et revue finale indépendante acquises)
|
||||
→ poursuite pré-SfM réelle de S21 à partir de Tracks
|
||||
→ acquisition dédiée de calibration
|
||||
→ Sparse SfM réel
|
||||
→ Dense / MVS
|
||||
```
|
||||
réglage CPU/GPU/lot/inflight/helper n'est requis en production ordinaire. Avec
|
||||
la nouvelle autorité humaine, ce choix doit désormais rechercher explicitement
|
||||
le **maximum sûr et utile**, et non conserver une sous-utilisation historique.
|
||||
|
||||
## GLOBAL MAINTENANCE AUDIT — PASS / FROZEN
|
||||
|
||||
Le jalon scientifique acquis reste arrêté après GV. Avant Tracks, Sparse SfM
|
||||
ou Dense/MVS, la gate de maintenance globale a réconcilié l'architecture, les
|
||||
ressources, la persistance et l'observation TUI. Son état et ses preuves
|
||||
consolidées sont consignés dans le
|
||||
[registre canonique de maintenance](../architecture/global_maintenance_audit.md).
|
||||
|
||||
Le checkpoint canonique de revue est le tag
|
||||
`global-maintenance-2026-09-01`, au commit
|
||||
`b84f860d868c66d9ee84b85ceb1bc6480b95aca5`. Les revues futures sont
|
||||
|
|
@ -532,44 +559,30 @@ dépendances traversées. Les systèmes PASS/FROZEN inchangés héritent de la p
|
|||
du [registre de maintenance](../architecture/global_maintenance_audit.md) et ne
|
||||
sont rouverts que sur preuve concrète ; ne pas répéter un audit global A-à-Z.
|
||||
|
||||
Les résultats déjà réglés sont :
|
||||
Les résultats acquis restent notamment :
|
||||
|
||||
- Project DB v23 ajoute neuf relations optiques sans backfill ni inférence ;
|
||||
les boîtiers, objectifs manuels/électroniques, configurations et calibrations
|
||||
restent distincts, et la sélection de calibration exige une compatibilité
|
||||
exacte et explicite ;
|
||||
- le Governor réserve d'abord l'hôte, sans plafond CPU global 12 : réserve de
|
||||
quatre CPU logiques sur hôte capable, groupes cœur/SMT complets lorsque la
|
||||
topologie est fiable, au moins une unité de calcul sur petit hôte, réserve
|
||||
RAM dure 3 GiB et prudence 3–4 GiB, PSI/swap actif et UMA comptée une fois ;
|
||||
- Candidate porte une capacité sûre 64, Visual Index 16, ORB/SIFT/RootSIFT sont
|
||||
bornés par le compute-pool, Matcher conserve sa borne intrinsèque sûre 12 et
|
||||
utile 8, et GV conserve son USAC interne sériel mais admet 16 participants
|
||||
sûrs/8 utiles et 16 parents par lot ;
|
||||
- le Governor réserve l'hôte sans plafond CPU global 12, avec réserve RAM dure
|
||||
3 GiB, PSI/swap actif et UMA comptée une fois ;
|
||||
- Candidate, Visual Index, Feature, Matcher et GV possèdent leurs capacités et
|
||||
décisions GPU mesurées ;
|
||||
- le contrôleur SSD UDisks2 optionnel est une frontière physique revue, avec
|
||||
identité Drive+labels+UUID, leases, drain sûr et latch de danger pour action
|
||||
indéterminée. Son état est enregistré auprès du Governor, seul orchestrateur
|
||||
des leases de production ; il n'est ni scheduler ni second Governor ;
|
||||
- la TUI est un observatoire/centre de contrôle validé opérationnellement :
|
||||
ncurses main-thread, modèle pur, observation coalescée et bornée, progression
|
||||
durable/ETA, pipeline et ressources honnêtes, profils optiques, layouts
|
||||
100×30/72×20/60×15, repli texte/couleur et F10 SSD asynchrone toujours
|
||||
visible ;
|
||||
identité Drive+labels+UUID, leases et drain sûr ;
|
||||
- la TUI est un observatoire/centre de contrôle validé opérationnellement ;
|
||||
- la frontière de session détruit/joint la Queue avant Project DB, puis recrée
|
||||
une seule Queue ; l'arrêt global libère les leases Task avant unregister SSD,
|
||||
contrôleur et Governor.
|
||||
une seule Queue.
|
||||
|
||||
Les validations finales exécutables sont acquises : build Clang portable
|
||||
931/931 + suite 64/64, build Clang Vulkan 939/939 + suite 65/65 sur Radeon
|
||||
réelle, ASan/UBSan 64/64 avec la limitation LSan OpenCL externe explicitement
|
||||
qualifiée, LSan loader-free 20/20, TSan 14/14 + 220 répétitions, headers publics
|
||||
76/76 sur 19 headers modifiés/nouveaux et contrôles ABI/diff/`scan3d`.
|
||||
L'unique revue finale indépendante GPT-5.6 SOL/ULTRA a conclu PASS sans finding
|
||||
bloquant après avoir indépendamment rejoué le build portable, la suite 64/64,
|
||||
15/15 tests focalisés, les 76/76 probes de headers, l'ABI, les négatifs de seams
|
||||
production, le SHA du manifest GV retenu et le diff-check. La gate de
|
||||
maintenance est donc fermée ; la poursuite réelle depuis Tracks devient la
|
||||
prochaine tranche séparée, sans avoir été exécutée par cette synchronisation.
|
||||
Le nouvel impératif `SERIALISM_REQUIRES_PROOF` ne nie aucune de ces preuves. Il
|
||||
supersède uniquement l'idée qu'un ancien profil opérationnel conservateur serait
|
||||
un plafond permanent lorsque de nouvelles preuves réelles montrent du travail
|
||||
indépendant et des ressources sûres inutilisées.
|
||||
|
||||
Les validations finales exécutables de la maintenance sont acquises : build
|
||||
Clang portable 931/931 + suite 64/64, build Clang Vulkan 939/939 + suite 65/65
|
||||
sur Radeon réelle, ASan/UBSan 64/64 avec la limitation LSan OpenCL externe
|
||||
qualifiée, LSan loader-free 20/20, TSan, headers publics 76/76 et contrôles
|
||||
ABI/diff/`scan3d`. La gate de maintenance reste fermée ; les nouvelles
|
||||
corrections de ressources sont revues en delta et ne réouvrent pas l'audit A→Z.
|
||||
|
||||
## REAL S21 TRACKS — PASS / FROZEN
|
||||
|
||||
|
|
@ -606,8 +619,175 @@ runner corrigé. Elle termine `COMPLETE` à 100 % et retrouve le même Track Set
|
|||
zéro Track Set. Cette preuve valide la reprise opérationnelle sans rouvrir le
|
||||
contrat scientifique Tracks gelé.
|
||||
|
||||
## PROJECT DB v24 / RAW BATCH A6000 — IMPLEMENTED / VALIDATION IN PROGRESS
|
||||
|
||||
L'autorisation humaine additive v24 est limitée au parallélisme opérationnel du
|
||||
développement RAW. Project DB v24 conserve v22/v23 et ajoute uniquement la
|
||||
persistance typée `raw_development_batch_tasks` nécessaire au nouveau kind
|
||||
`raw.develop.batch/1`. Cette version ne change aucune identité scientifique,
|
||||
aucune calibration ni `L3DRAWD1`.
|
||||
|
||||
Le pattern de production est :
|
||||
|
||||
```text
|
||||
une Task owner admise par le Governor
|
||||
→ participants RAW indépendants et bornés
|
||||
→ join de tous les participants
|
||||
→ publication owner-only dans l'ordre des selected_item_index
|
||||
→ avance durable du curseur selected_execution
|
||||
```
|
||||
|
||||
L'atomicité par Capture est conservée ; la sérialisation entre Captures
|
||||
indépendantes ne l'est plus. La transition historique `raw.develop/1` →
|
||||
`raw.develop.batch/1` reste légitime et ne réécrit aucune ancienne Task.
|
||||
|
||||
État durable A6000 au dernier arrêt sûr avant l'audit global d'utilisation des
|
||||
ressources :
|
||||
|
||||
```text
|
||||
Projet:
|
||||
/home/fy59/Documents/Lardon/.real-pre-sfm-2026-09-01/a6000-pre-sfm-v23-final
|
||||
|
||||
Schema v24
|
||||
Captures 953
|
||||
GOOD 689
|
||||
SUSPECT 206
|
||||
REJECT 58
|
||||
selected_execution 1
|
||||
selected items 689 RAW_ASSET
|
||||
published representations 259
|
||||
next_item_index 259
|
||||
remaining RAW representations 430
|
||||
Features 0
|
||||
Visual Index 0
|
||||
Candidate 0
|
||||
Matcher 0
|
||||
GV 0
|
||||
Tracks 0
|
||||
Sparse SfM 0
|
||||
Dense/MVS 0
|
||||
```
|
||||
|
||||
Le digest d'inventaire source pré/post est
|
||||
`fa725d8a82b529521134dd600b5cf42ed58ad523108636741a1431441e17b029`.
|
||||
Les JPEG caméra et RAW sont tous deux des SOURCE assets explicites par Capture,
|
||||
mais cette exécution A6000 FROZEN utilise les PNG déterministes dérivés RAW pour
|
||||
la géométrie. Les JPEG restent les proxies de triage ; ils ne remplacent pas
|
||||
cette représentation scientifique sans nouvelle tranche d'équivalence
|
||||
scientifique/calibration.
|
||||
|
||||
La v24 et la Task batch ont déjà passé build normal, matrice ciblée, contrôles
|
||||
C17, `git diff --check`, ASan/UBSan ciblé et TSan sur la couture concurrente.
|
||||
La preuve réelle complète, l'équivalence finale, l'audit d'utilisation de toutes
|
||||
les Tasks affectées et la fermeture documentaire restent à terminer avant de
|
||||
marquer cette tranche `PASS/FROZEN`.
|
||||
|
||||
## PROJECT DB v25 / FEATURE BATCH — IMPLEMENTED / VALIDATION IN PROGRESS
|
||||
|
||||
L'overlay additif v25 ajoute uniquement la persistance typée de
|
||||
`features.extract.batch/1`. Il ancre un propriétaire durable à l'ordre immutable
|
||||
de l'exécution sélectionnée, au domaine ORB exact et à un curseur monotone. Les
|
||||
Tasks `features.extract/1` historiques restent inchangées et sont reprises avant
|
||||
le suffixe batch. La préparation inter-images est bornée par le Governor,
|
||||
OpenCV reste CPU1 dans chaque participant, et publication/cursor/checkpoint
|
||||
restent owner-only et ordonnés. Le build et les tests ciblés sont acquis ; le
|
||||
benchmark réel du palier utile et la preuve A6000 restent à exécuter avant toute
|
||||
revendication PASS/FROZEN.
|
||||
|
||||
## REAL A6000 PRE-SFM — PASS / FROZEN
|
||||
|
||||
`REAL_A6000_PRE_SFM=PASS/FROZEN`. La continuation du 2 septembre 2026 utilise
|
||||
le projet durable
|
||||
`/home/fy59/Documents/Lardon/.real-pre-sfm-2026-09-01/a6000-pre-sfm-v23-final`
|
||||
directement à la frontière Match Result acquise. Les 689 Feature Sets, le Visual
|
||||
Index complet, les 38 420 Candidate Pairs et les 38 420 Match Results de la
|
||||
Task Matcher 170 `COMPLETE/100` sont conservés :
|
||||
`FEATURE_REPLAY=0`, `CANDIDATE_REPLAY=0` et `MATCHER_REPLAY=0`. Ni l'acquisition,
|
||||
ni RAW, ni les assets/identités amont ne sont réécrits.
|
||||
|
||||
La Task GV v3 171 termine `COMPLETE/100`, consomme le curseur Match Result
|
||||
jusqu'à 38 420 et publie exactement 37 805 GVR applicables : 10 952
|
||||
`GEOMETRIC_VERIFIED` et 26 853 `GEOMETRIC_REJECTED`. Son fingerprint reste
|
||||
`6944a471d611d8ffc59dac7cf15a5b79b97e2371d4c51785c477d68c1577f74c`;
|
||||
zéro mapping GVR dupliqué est observé. Les diagnostics Governor coalescés
|
||||
retiennent 2 714 admissions, le compute-pool de 12 CPU avec 4 CPU réservés,
|
||||
une fenêtre finale de 16, RSS/HWM maximal 41 877 504/42 319 872 octets,
|
||||
`MemAvailable` minimal 10 555 641 856 octets et des deltas swap-in/out nuls.
|
||||
La préparation GV reste dans la forme validée Governor-admise et bornée ; aucune
|
||||
nouvelle policy CPU/GPU n'est introduite par ce run.
|
||||
|
||||
La Task Track Builder 172 termine `COMPLETE/100` et publie atomiquement le seul
|
||||
Track Set 1, de scope 10 952 GVR, avec 130 714 Tracks et 318 944 observations.
|
||||
L'audit SQL confirme zéro observation de Track dupliquée, zéro Track contenant
|
||||
deux observations de la même image et zéro observation orpheline. La seconde
|
||||
continuation crée seulement la Task GV 173, qui traverse le même curseur,
|
||||
publie zéro GVR et réutilise Track Set 1 sans créer de Task Track Builder ni
|
||||
modifier les 130 714 Tracks. `RESTART_IDEMPOTENCE`, `DETERMINISM`,
|
||||
`TRACK_CONFLICT_INVARIANTS` et `RESOURCE_POLICY_ENFORCED` sont donc PASS pour
|
||||
cette frontière réelle.
|
||||
|
||||
Sparse SfM, reconstruction sparse, Dense/MVS et fusion inter-campagnes restent
|
||||
à zéro. Ce gel atteste uniquement le pré-SfM A6000 à travers GV puis Tracks ; il
|
||||
ne contourne pas le blocage de calibration réel et ne promeut aucune étape aval.
|
||||
|
||||
## RESOURCE UTILIZATION ENFORCEMENT — CURRENT NEXT
|
||||
|
||||
Avant de reprendre les 430 RAW restants ou toute autre preuve réelle longue,
|
||||
la prochaine tranche opérationnelle doit appliquer la politique canonique à
|
||||
l'ensemble des Tasks de production, en revue delta uniquement.
|
||||
|
||||
Objectifs :
|
||||
|
||||
```text
|
||||
RESOURCE_UTILIZATION_POLICY=IMPLEMENTED
|
||||
SERIALISM_REQUIRES_PROOF=ENFORCED
|
||||
ACCIDENTAL_CPU1_PATHS=0_OR_MEASURED_JUSTIFICATION
|
||||
VALIDATED_GPU_BACKENDS=PREFERRED_WHEN_ELIGIBLE
|
||||
BUILD_TEST_PARALLELISM=HOST_RESERVE_AWARE
|
||||
```
|
||||
|
||||
Pour chaque Task, auditer uniquement les dimensions opérationnelles :
|
||||
minimum/useful/safe CPU et batch, indépendance des items, séparation
|
||||
préparation/publication, coûts RAM par participant, saturation I/O et backend
|
||||
GPU validé. Une valeur CPU1/batch1 peut rester seulement avec preuve concrète.
|
||||
Il ne faut pas inventer de second scheduler, pool global ou parallélisme
|
||||
inter-Tasks pour satisfaire cette politique : le parallélisme interne borné et
|
||||
l'unique Queue/Governor restent la voie normale.
|
||||
|
||||
Les tests Codex doivent suivre la même règle : build et tests indépendants
|
||||
parallèles jusqu'à la réserve interactive ; serialisation uniquement pour
|
||||
fixtures partagées, déterminisme scientifique, mémoire/I/O/GPU exclusif,
|
||||
sanitizer ou pression mesurée. Les validations acquises et non affectées ne
|
||||
sont pas répétées. Les modèles/agents coûteux sont réservés aux vraies
|
||||
frontières scientifiques, concurrence/persistance délicate et revue finale
|
||||
nécessaire.
|
||||
|
||||
## NEAR TERM
|
||||
|
||||
Ordre court terme désormais canonique :
|
||||
|
||||
```text
|
||||
GLOBAL MAINTENANCE AUDIT PASS / FROZEN
|
||||
→ REAL S21 TRACKS PASS / FROZEN
|
||||
→ RESOURCE UTILIZATION ENFORCEMENT CURRENT NEXT
|
||||
(SERIALISM_REQUIRES_PROOF, max safe/useful throughput,
|
||||
politique tests/builds Codex comprise)
|
||||
→ reprise A6000 v24 au curseur 259
|
||||
→ 430 représentations RAW restantes
|
||||
→ Features
|
||||
→ Visual Index
|
||||
→ Candidate
|
||||
→ Matcher AUTO / ORB Vulkan si éligible
|
||||
→ GV
|
||||
→ Tracks
|
||||
→ STOP avant Sparse SfM
|
||||
→ acquisition physique dédiée de calibration
|
||||
→ calibration connue validée
|
||||
→ Sparse SfM réel par campagne
|
||||
→ fusion / Phase H selon dépendances
|
||||
→ Dense / MVS
|
||||
```
|
||||
|
||||
### Scratch SSD externe et swap optionnel
|
||||
|
||||
Le contrôleur physique actuel découvre par UDisks2 une paire exacte de labels
|
||||
|
|
@ -621,9 +801,8 @@ Les usages futurs possibles sont workspace/scratch et intermédiaires
|
|||
dense/mesh/texturing. Leur consommation devra être explicitement possédée par
|
||||
les Tasks et passer par les wrappers de lease du Resource Governor ; le
|
||||
contrôleur physique et le registre actuel n'inventent aucune éligibilité Task.
|
||||
Les quatorze kinds courants ne consomment aucun scratch. Le swap de sécurité,
|
||||
lui, reste une fonction du drain physique explicite et jamais un budget de
|
||||
travail.
|
||||
Le swap de sécurité reste une fonction du drain physique explicite et jamais un
|
||||
budget de travail.
|
||||
|
||||
- les leases scratch ont une ownership explicite ; `DRAINING` refuse les
|
||||
nouveaux leases et attend leur libération exacte ;
|
||||
|
|
@ -636,14 +815,11 @@ travail.
|
|||
- une action UDisks potentiellement appliquée mais non vérifiable verrouille le
|
||||
tuple physique original ; un remplacement n'obtient aucune autorité ;
|
||||
- aucun formatage, partitionnement, fsck, réparation, arrêt forcé ou suppression
|
||||
n'est permis. La production n'appelle pas `statvfs`; espace total/libre reste
|
||||
`UNKNOWN` quand UDisks ne le fournit pas.
|
||||
n'est permis.
|
||||
|
||||
Le contrôleur, son registre Governor, le worker joinable unique et sa
|
||||
présentation/actions F10 sont **CURRENT / VALIDATED OPERATIONAL**. Les tests
|
||||
emploient un provider factice et ne montent, n'arrêtent ni ne modifient un vrai
|
||||
disque. Ce statut ne crée aucun consommateur scratch ; c'est la validation
|
||||
consolidée et la revue indépendante ci-dessus qui ont fermé la gate globale.
|
||||
présentation/actions F10 sont **CURRENT / VALIDATED OPERATIONAL**. Ce statut ne
|
||||
crée aucun consommateur scratch par lui-même.
|
||||
|
||||
### Publication durable dense / mesh
|
||||
|
||||
|
|
@ -715,8 +891,7 @@ Le but à long terme n'est pas seulement de reconstruire ce qui a été
|
|||
photographié, mais de guider activement l'opérateur vers les photographies qui
|
||||
manquent encore pour obtenir une reconstruction fiable. Cette capacité est
|
||||
postérieure à une reconstruction suffisamment mature ; elle ne fait pas partie
|
||||
de S3 et ne remplace pas le prochain milestone d'intégration multi-campagne
|
||||
A6000 + S21.
|
||||
de S3.
|
||||
|
||||
```text
|
||||
photographies existantes
|
||||
|
|
@ -933,20 +1108,25 @@ L'ordre demeure sans ambiguïté :
|
|||
|
||||
```text
|
||||
CURRENT NEXT
|
||||
fondation scientifique pré-SfM courante
|
||||
→ INTERNAL PARALLELISM + COMPUTE RESOURCES v1
|
||||
→ COMPUTE GOVERNOR v2 / ORB VULKAN ASYNC EXECUTION
|
||||
(PASS / FROZEN; AUTO GPU-first ORB et preuve S21 Matcher complète)
|
||||
→ REAL S21 GV v3 (PASS / FROZEN; Tracks/Sparse SfM non exécutés)
|
||||
→ GLOBAL MAINTENANCE AUDIT
|
||||
(PASS / FROZEN; gate fermée avant la tranche Tracks séparée)
|
||||
→ poursuite pré-SfM réelle de S21 à partir de Tracks
|
||||
→ acquisition physique dédiée de calibration
|
||||
→ calibration connue validée → Sparse SfM réel multi-campagne
|
||||
GLOBAL MAINTENANCE AUDIT PASS / FROZEN
|
||||
→ REAL S21 TRACKS PASS / FROZEN
|
||||
→ RESOURCE UTILIZATION ENFORCEMENT CURRENT NEXT
|
||||
→ A6000 PRE-SFM v24 resume from cursor 259
|
||||
→ RAW batch
|
||||
→ Features
|
||||
→ Visual Index
|
||||
→ Candidate
|
||||
→ Matcher AUTO / validated Vulkan when eligible
|
||||
→ GV
|
||||
→ Tracks
|
||||
→ STOP before Sparse SfM
|
||||
→ dedicated physical calibration acquisition
|
||||
→ known calibration → real Sparse SfM
|
||||
→ multi-campaign registration/fusion
|
||||
→ dense / MVS → publication
|
||||
→ LATER
|
||||
Coverage Analysis → Coverage Viewer → suggestions de points de vue
|
||||
→ localisation live → intégration HDMI/USB → Capture Guidance / Live Coverage
|
||||
Coverage Analysis → Coverage Viewer → suggested viewpoints
|
||||
→ live localization → HDMI/USB integration → Capture Guidance / Live Coverage
|
||||
```
|
||||
|
||||
Vidéo/keyframes pourra progresser en parallèle lors d'une phase ultérieure,
|
||||
|
|
@ -961,21 +1141,34 @@ scientifique.
|
|||
|
||||
## DEFERRED / OPTIONAL
|
||||
|
||||
- pools multiples CPU/GPU/IO, parallélisme inter-tâches et multi-GPU ;
|
||||
- pools multiples CPU/GPU/IO, parallélisme inter-Tasks et multi-GPU ;
|
||||
- DAG général de dépendances et priorités complexes ;
|
||||
- infrastructure backend générique et distribution de calcul ;
|
||||
- ALIKED tant que provenance modèle, export ONNX et oracle upstream ne sont pas
|
||||
reproductibles.
|
||||
|
||||
Le report du parallélisme **inter-Tasks** ne constitue jamais une permission de
|
||||
sérialiser accidentellement les unités indépendantes **à l'intérieur d'une Task
|
||||
active**. L'unique Queue/Governor et le fanout interne borné suffisent pour
|
||||
appliquer `SERIALISM_REQUIRES_PROOF` sans créer un second runtime.
|
||||
|
||||
Ces idées ne doivent pas devancer l'intégration réelle multi-campagne ni créer
|
||||
un second runtime, Governor ou système de persistance.
|
||||
|
||||
## Principes de séquencement
|
||||
|
||||
1. stabilité et intégrité scientifique avant débit ;
|
||||
2. résultats atomiques et durables avant parallélisme ;
|
||||
3. lots bornés et reprise avant taille de campagne ;
|
||||
4. un Task Runtime/Queue et un Resource Governor, sans second scheduler ;
|
||||
5. scratch optionnel sans élargissement implicite des budgets RAM ;
|
||||
6. TUI de contrôle avant visualisation riche ;
|
||||
7. documentation canonique alignée sur le code validé.
|
||||
1. intégrité scientifique et réserve interactive hôte sont inviolables ; à
|
||||
l'intérieur de ces limites, le maximum de débit sûr et utile est obligatoire ;
|
||||
2. `SERIALISM_REQUIRES_PROOF` : l'atomicité d'un item ne sérialise pas les items
|
||||
indépendants et toute sous-utilisation durable doit avoir une raison mesurée ;
|
||||
3. résultats atomiques et durables avant publication concurrente ; préparation
|
||||
parallèle bornée autorisée lorsque l'égalité exacte est préservée ;
|
||||
4. lots bornés, reprise et mémoire honnêtement comptée avant taille de campagne ;
|
||||
5. un Task Runtime/Queue et un Resource Governor, sans second scheduler ;
|
||||
6. backend GPU validé et utile préféré automatiquement, sans GPU artificiel ;
|
||||
7. scratch optionnel sans élargissement implicite des budgets RAM ;
|
||||
8. builds/tests/preuves Codex parallèles jusqu'à la réserve hôte lorsqu'ils sont
|
||||
indépendants ; aucune sérialisation ou revalidation lourde par habitude ;
|
||||
9. TUI de contrôle avant visualisation riche ;
|
||||
10. documentation canonique alignée sur le code validé et revue delta-based
|
||||
depuis `global-maintenance-2026-09-01`.
|
||||
|
|
|
|||
|
|
@ -10,6 +10,8 @@
|
|||
|
||||
#define LARDON3D_FEATURE_EXTRACT_TASK_KIND "features.extract"
|
||||
enum { LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION = 1 };
|
||||
#define LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND "features.extract.batch"
|
||||
enum { LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION = 1 };
|
||||
|
||||
Lardon3DTask *
|
||||
lardon3d_project_create_feature_extract_task(Lardon3DAppState *state, uint64_t image_id,
|
||||
|
|
@ -20,5 +22,18 @@ bool lardon3d_project_enqueue_feature_extract(Lardon3DAppState *state, uint64_t
|
|||
uint64_t *task_id);
|
||||
bool lardon3d_feature_extract_reconstruct(const Lardon3DTaskDurableSnapshot *snapshot,
|
||||
void *context, Lardon3DTaskKindBinding *binding);
|
||||
/* Create/enqueue one durable owner Task over the immutable selected-execution
|
||||
* image order. ORB parameters are copied into typed v25 state; participants are
|
||||
* Governor-bounded, cancellable at window boundaries, and joined before the
|
||||
* owner publishes or advances its cursor. */
|
||||
Lardon3DTask *lardon3d_project_create_feature_extract_batch_task(
|
||||
Lardon3DAppState *state, uint64_t selected_execution_id,
|
||||
const Lardon3DFeatureExtractorParameters *parameters, uint64_t *task_id);
|
||||
bool lardon3d_project_enqueue_feature_extract_batch(
|
||||
Lardon3DAppState *state, uint64_t selected_execution_id,
|
||||
const Lardon3DFeatureExtractorParameters *parameters, uint64_t *task_id);
|
||||
bool lardon3d_feature_extract_batch_reconstruct(
|
||||
const Lardon3DTaskDurableSnapshot *snapshot, void *context,
|
||||
Lardon3DTaskKindBinding *binding);
|
||||
|
||||
#endif
|
||||
|
|
|
|||
|
|
@ -73,9 +73,21 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_photo_quality_ta
|
|||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_raw_development_task(
|
||||
Lardon3DAppState *state, const Lardon3DTask *task,
|
||||
const Lardon3DProjectDbRawDevelopmentTask *parameters);
|
||||
/* Persist generic parent Task state atomically with its immutable selected
|
||||
* execution association. Call only after the selected execution cursor commit
|
||||
* so generic progress can lag but never lead scientific publication. */
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_raw_development_batch_task(
|
||||
Lardon3DAppState *state, const Lardon3DTask *task,
|
||||
const Lardon3DProjectDbRawDevelopmentBatchTask *parameters);
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_task(
|
||||
Lardon3DAppState *state, const Lardon3DTask *task,
|
||||
const Lardon3DProjectDbFeatureExtractTask *parameters);
|
||||
/* Persist the generic checkpoint and typed batch domain/cursor in one SQLite
|
||||
* transaction. Call only after owner-only Feature publication and cursor
|
||||
* advancement, so generic progress may lag but never lead durable work. */
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_batch_task(
|
||||
Lardon3DAppState *state, const Lardon3DTask *task,
|
||||
const Lardon3DProjectDbFeatureExtractBatchTask *parameters);
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_sift_extract_task(
|
||||
Lardon3DAppState *state, const Lardon3DTask *task,
|
||||
const Lardon3DProjectDbSiftExtractTask *parameters);
|
||||
|
|
|
|||
|
|
@ -11,10 +11,10 @@
|
|||
#include <lardon3d/sparse_sfm_incremental.h>
|
||||
|
||||
enum {
|
||||
/* v23 is an additive optical-context layer. Historical v22 rows and sparse
|
||||
* calibration identities remain untouched; migration never guesses profiles
|
||||
* or assignments for existing Captures. */
|
||||
LARDON3D_PROJECT_DB_SCHEMA_VERSION = 23,
|
||||
/* v25 adds only durable features.extract.batch/1 operational state. The
|
||||
* selected execution remains the immutable ordered image domain; migration
|
||||
* creates no batch association, cursor, Feature Set, or inferred identity. */
|
||||
LARDON3D_PROJECT_DB_SCHEMA_VERSION = 25,
|
||||
LARDON3D_PROJECT_DB_ID_CAPACITY = 65,
|
||||
LARDON3D_PROJECT_DB_KIND_CAPACITY = 65,
|
||||
LARDON3D_PROJECT_DB_PATH_CAPACITY = 4096,
|
||||
|
|
@ -145,6 +145,13 @@ typedef struct {
|
|||
uint64_t image_id;
|
||||
} Lardon3DProjectDbRawDevelopmentTask;
|
||||
|
||||
typedef struct {
|
||||
uint64_t task_id;
|
||||
/* Selected execution owns the ordered scientific inputs and durable cursor.
|
||||
* This typed Task row deliberately copies none of those identities. */
|
||||
uint64_t selected_execution_id;
|
||||
} Lardon3DProjectDbRawDevelopmentBatchTask;
|
||||
|
||||
typedef struct {
|
||||
uint64_t task_id;
|
||||
uint32_t group_id;
|
||||
|
|
@ -406,6 +413,19 @@ typedef struct {
|
|||
unsigned char parameter_fingerprint[32];
|
||||
} Lardon3DProjectDbFeatureExtractTask;
|
||||
|
||||
typedef struct {
|
||||
uint64_t task_id;
|
||||
uint64_t selected_execution_id;
|
||||
/* Zero-based durable prefix in the immutable selected-execution order. */
|
||||
uint32_t next_item_index;
|
||||
char extractor_kind[LARDON3D_PROJECT_DB_KIND_CAPACITY];
|
||||
uint32_t extractor_version;
|
||||
uint32_t max_features;
|
||||
uint32_t pyramid_levels;
|
||||
uint32_t fast_threshold;
|
||||
unsigned char parameter_fingerprint[LARDON3D_PROJECT_DB_SHA256_SIZE];
|
||||
} Lardon3DProjectDbFeatureExtractBatchTask;
|
||||
|
||||
typedef struct {
|
||||
uint64_t task_id;
|
||||
uint64_t image_id;
|
||||
|
|
@ -785,6 +805,21 @@ Lardon3DProjectDbResult lardon3d_project_db_record_raw_development_task(
|
|||
Lardon3DProjectDbResult lardon3d_project_db_load_raw_development_task(
|
||||
Lardon3DProjectDb *database, uint64_t task_id,
|
||||
Lardon3DProjectDbRawDevelopmentTask *parameters);
|
||||
/* Atomically record generic raw.develop.batch/1 state and its immutable link
|
||||
* to one selected execution. Exact retry is idempotent; rebinding either a
|
||||
* Task or selected execution conflicts. The selected execution cursor remains
|
||||
* the sole durable item-progress authority. */
|
||||
Lardon3DProjectDbResult lardon3d_project_db_record_raw_development_batch_task(
|
||||
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot,
|
||||
const char *task_kind, uint32_t task_kind_version,
|
||||
const Lardon3DProjectDbCheckpoint *checkpoint,
|
||||
const Lardon3DProjectDbRawDevelopmentBatchTask *parameters, int64_t updated_at);
|
||||
/* Load the caller-owned immutable Task -> selected-execution association.
|
||||
* Malformed integer storage or a generic kind/version mismatch is corruption;
|
||||
* no item identity or cursor is inferred from other durable state. */
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_raw_development_batch_task(
|
||||
Lardon3DProjectDb *database, uint64_t task_id,
|
||||
Lardon3DProjectDbRawDevelopmentBatchTask *parameters);
|
||||
/* Result publication and next_group_id advance are atomic. result->group_id is
|
||||
* the canonical plan ID in 1..N and must equal the task's current one-based
|
||||
* cursor; next_group_id must equal result->group_id+1 (N+1 after the last
|
||||
|
|
@ -869,6 +904,26 @@ Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_task(
|
|||
Lardon3DProjectDbResult
|
||||
lardon3d_project_db_load_feature_extract_task(Lardon3DProjectDb *database, uint64_t task_id,
|
||||
Lardon3DProjectDbFeatureExtractTask *parameters);
|
||||
/* Atomically persist generic features.extract.batch/1 state and its typed
|
||||
* immutable selected-execution/ORB domain plus monotone cursor. A stale
|
||||
* generic checkpoint may repeat an older cursor but can never move it back. */
|
||||
Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_batch_task(
|
||||
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot,
|
||||
const char *task_kind, uint32_t task_kind_version,
|
||||
const Lardon3DProjectDbCheckpoint *checkpoint,
|
||||
const Lardon3DProjectDbFeatureExtractBatchTask *parameters, int64_t updated_at);
|
||||
/* Load and validate the exact typed batch domain and generic dispatch. The
|
||||
* selected execution must be representation-complete and the cursor bounded by
|
||||
* its immutable item count; malformed durable state is CORRUPT. */
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_feature_extract_batch_task(
|
||||
Lardon3DProjectDb *database, uint64_t task_id,
|
||||
Lardon3DProjectDbFeatureExtractBatchTask *parameters);
|
||||
/* Advance exactly one ordered item only after its selected image has a READY
|
||||
* Feature Set for the task's immutable ORB fingerprint. Exact retry after an
|
||||
* already committed advance converges; no image identity is inferred. */
|
||||
Lardon3DProjectDbResult lardon3d_project_db_advance_feature_extract_batch_task(
|
||||
Lardon3DProjectDb *database, uint64_t task_id, uint32_t item_index,
|
||||
uint32_t next_item_index);
|
||||
Lardon3DProjectDbResult lardon3d_project_db_record_sift_extract_task(
|
||||
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot,
|
||||
const char *task_kind, uint32_t task_kind_version,
|
||||
|
|
|
|||
|
|
@ -13,6 +13,8 @@ extern "C" {
|
|||
|
||||
#define LARDON3D_RAW_DEVELOPMENT_TASK_KIND "raw.develop"
|
||||
enum { LARDON3D_RAW_DEVELOPMENT_TASK_KIND_VERSION = 1 };
|
||||
#define LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND "raw.develop.batch"
|
||||
enum { LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION = 1 };
|
||||
|
||||
/* Create one durable, initially unqueued S3-B1 Task for the exact existing
|
||||
* Capture-owned SOURCE RAW asset. The two IDs are immutable and are never
|
||||
|
|
@ -37,6 +39,25 @@ bool lardon3d_raw_development_task_reconstruct(
|
|||
const Lardon3DTaskDurableSnapshot *snapshot, void *context,
|
||||
Lardon3DTaskKindBinding *binding);
|
||||
|
||||
/* Create one durable, initially unqueued parent Task for the exact immutable
|
||||
* selected execution. The selected execution, not Task identity, owns item
|
||||
* order, explicit Capture/SOURCE RAW identity, and the durable cursor. Queue
|
||||
* admission bounds each parallel window; the caller owns the returned Task. */
|
||||
Lardon3DTask *lardon3d_project_create_raw_development_batch_task(
|
||||
Lardon3DAppState *state, uint64_t selected_execution_id, uint64_t *task_id);
|
||||
|
||||
/* Create and transfer the parent Task to state's existing sole Queue. The
|
||||
* Queue/Governor reservation covers only one bounded window at a time and is
|
||||
* released/re-established at each sequence boundary. */
|
||||
bool lardon3d_project_enqueue_raw_development_batch(
|
||||
Lardon3DAppState *state, uint64_t selected_execution_id, uint64_t *task_id);
|
||||
|
||||
/* Reconstruct only from the validated v24 Task -> selected execution row.
|
||||
* binding owns its returned context until the runtime invokes its destroyer. */
|
||||
bool lardon3d_raw_development_batch_task_reconstruct(
|
||||
const Lardon3DTaskDurableSnapshot *snapshot, void *context,
|
||||
Lardon3DTaskKindBinding *binding);
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
|
|
|||
|
|
@ -26,6 +26,13 @@ extern "C" bool lardon3d_feature_opencv_configure_threads(unsigned int threads)
|
|||
const char *forced = std::getenv(
|
||||
"LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS");
|
||||
if (forced) {
|
||||
if (std::strcmp(forced, "once") == 0) {
|
||||
/* The Task's admitted count is deliberately host-derived. Consume the
|
||||
* one-shot before failing so the rollback can restore the captured
|
||||
* process-wide value without a test-side guess of that count. */
|
||||
(void)unsetenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS");
|
||||
return false;
|
||||
}
|
||||
char *end = nullptr;
|
||||
unsigned long parsed = std::strtoul(forced, &end, 10);
|
||||
if (end && *end == '\0' && parsed == threads) {
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@
|
|||
#include <fcntl.h>
|
||||
#include <limits.h>
|
||||
#include <openssl/evp.h>
|
||||
#include <pthread.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdio.h>
|
||||
|
|
@ -26,6 +27,18 @@ typedef struct {
|
|||
Lardon3DProjectDbFeatureExtractTask parameters;
|
||||
} Lardon3DFeatureTaskContext;
|
||||
|
||||
enum {
|
||||
FEATURE_BATCH_MAX = 12,
|
||||
FEATURE_BATCH_CHILD_STACK = 1024 * 1024,
|
||||
};
|
||||
|
||||
typedef struct {
|
||||
char project_path[PATH_MAX];
|
||||
Lardon3DProjectDb *database;
|
||||
Lardon3DResourceGovernor *governor;
|
||||
Lardon3DProjectDbFeatureExtractBatchTask parameters;
|
||||
} Lardon3DFeatureBatchTaskContext;
|
||||
|
||||
static void destroy_context(void *userdata) { free(userdata); }
|
||||
|
||||
static void runtime_state(const Lardon3DFeatureTaskContext *context, Lardon3DAppState *state) {
|
||||
|
|
@ -129,6 +142,247 @@ static bool load_validated_source(const Lardon3DFeatureTaskContext *context, cha
|
|||
return file_hash(path, actual) && memcmp(actual, asset.sha256, 32) == 0;
|
||||
}
|
||||
|
||||
static void batch_runtime_state(const Lardon3DFeatureBatchTaskContext *context,
|
||||
Lardon3DAppState *state) {
|
||||
lardon3d_app_state_init(state);
|
||||
state->project_loaded = true;
|
||||
state->project_db = context->database;
|
||||
state->resource_governor = context->governor;
|
||||
(void)snprintf(state->project_path, sizeof(state->project_path), "%s", context->project_path);
|
||||
}
|
||||
|
||||
static bool batch_checkpoint(Lardon3DFeatureBatchTaskContext *context, Lardon3DTask *task,
|
||||
uint32_t cursor) {
|
||||
Lardon3DAppState state;
|
||||
batch_runtime_state(context, &state);
|
||||
Lardon3DProjectDbFeatureExtractBatchTask parameters = context->parameters;
|
||||
parameters.task_id = lardon3d_task_id(task);
|
||||
parameters.next_item_index = cursor;
|
||||
return lardon3d_project_checkpoint_feature_extract_batch_task(&state, task, ¶meters) ==
|
||||
LARDON3D_PROJECT_TASK_CHECKPOINT_OK;
|
||||
}
|
||||
|
||||
typedef struct {
|
||||
Lardon3DFeatureBatchTaskContext *context;
|
||||
uint32_t item_index;
|
||||
uint64_t image_id;
|
||||
char source_path[PATH_MAX];
|
||||
bool already_ready;
|
||||
Lardon3DFeatureExtractResult result;
|
||||
Lardon3DExtractedFeatures features;
|
||||
} FeatureBatchWorker;
|
||||
|
||||
static void *extract_batch_item(void *value) {
|
||||
FeatureBatchWorker *worker = value;
|
||||
if (worker->already_ready) return NULL;
|
||||
Lardon3DFeatureExtractorParameters parameters = {
|
||||
worker->context->parameters.max_features,
|
||||
worker->context->parameters.pyramid_levels,
|
||||
worker->context->parameters.fast_threshold};
|
||||
worker->result = lardon3d_feature_extract_orb(worker->source_path, ¶meters,
|
||||
&worker->features);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
typedef struct {
|
||||
FeatureBatchWorker *workers;
|
||||
size_t count;
|
||||
size_t first;
|
||||
size_t stride;
|
||||
} FeatureBatchParticipant;
|
||||
|
||||
static void *extract_batch_participant(void *value) {
|
||||
FeatureBatchParticipant *participant = value;
|
||||
for (size_t index = participant->first; index < participant->count;
|
||||
index += participant->stride)
|
||||
(void)extract_batch_item(&participant->workers[index]);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
static bool extract_batch_window(FeatureBatchWorker *workers, size_t count,
|
||||
unsigned int admitted_threads) {
|
||||
if (!workers || count == 0 || count > FEATURE_BATCH_MAX || admitted_threads == 0) return false;
|
||||
size_t participants = count < admitted_threads ? count : admitted_threads;
|
||||
pthread_t children[FEATURE_BATCH_MAX - 1];
|
||||
FeatureBatchParticipant work[FEATURE_BATCH_MAX];
|
||||
pthread_attr_t attributes;
|
||||
if (pthread_attr_init(&attributes) != 0) return false;
|
||||
if (pthread_attr_setstacksize(&attributes, FEATURE_BATCH_CHILD_STACK) != 0) {
|
||||
(void)pthread_attr_destroy(&attributes);
|
||||
return false;
|
||||
}
|
||||
size_t created = 0;
|
||||
bool started = true;
|
||||
for (size_t participant = 1; participant < participants; ++participant) {
|
||||
work[participant] = (FeatureBatchParticipant){workers, count, participant, participants};
|
||||
/* Participants own only disjoint extraction outputs. SQLite, Feature Store
|
||||
publication, cursor advancement, and generic checkpoints remain owner-only. */
|
||||
if (pthread_create(&children[created], &attributes, extract_batch_participant,
|
||||
&work[participant]) != 0) {
|
||||
started = false;
|
||||
break;
|
||||
}
|
||||
++created;
|
||||
}
|
||||
if (pthread_attr_destroy(&attributes) != 0) started = false;
|
||||
work[0] = (FeatureBatchParticipant){workers, count, 0, participants};
|
||||
if (started) (void)extract_batch_participant(&work[0]);
|
||||
for (size_t index = 0; index < created; ++index)
|
||||
if (pthread_join(children[index], NULL) != 0) started = false;
|
||||
return started;
|
||||
}
|
||||
|
||||
static void destroy_batch_outputs(FeatureBatchWorker *workers, size_t count) {
|
||||
if (!workers) return;
|
||||
/* A joined window owns every participant output until ordered owner
|
||||
publication consumes it. Any boundary failure must release the whole
|
||||
window; the Governor reservation must not outlive abandoned ORB buffers. */
|
||||
for (size_t index = 0; index < count; ++index)
|
||||
lardon3d_extracted_features_destroy(&workers[index].features);
|
||||
}
|
||||
|
||||
static bool run_batch_body(Lardon3DTask *task, Lardon3DFeatureBatchTaskContext *context) {
|
||||
for (;;) {
|
||||
if (!lardon3d_task_checkpoint(task)) return false;
|
||||
Lardon3DProjectDbFeatureExtractBatchTask persisted;
|
||||
Lardon3DProjectDbSelectedExecution execution;
|
||||
if (lardon3d_project_db_load_feature_extract_batch_task(
|
||||
context->database, lardon3d_task_id(task), &persisted) != LARDON3D_PROJECT_DB_OK ||
|
||||
memcmp(&persisted.parameter_fingerprint, &context->parameters.parameter_fingerprint,
|
||||
32) != 0 ||
|
||||
lardon3d_project_db_load_selected_execution(context->database,
|
||||
context->parameters.selected_execution_id, &execution) != LARDON3D_PROJECT_DB_OK)
|
||||
return lardon3d_task_fail(task, "État durable Feature batch invalide.");
|
||||
if (persisted.next_item_index == execution.item_count)
|
||||
return lardon3d_task_set_durable_progress(task, execution.item_count, execution.item_count,
|
||||
"Extraction Feature batch terminée.") &&
|
||||
batch_checkpoint(context, task, execution.item_count);
|
||||
|
||||
Lardon3DTaskExecutionContract contract;
|
||||
if (!lardon3d_task_execution_contract(task, &contract) || contract.cpu_threads == 0 ||
|
||||
contract.batch_size == 0 || contract.batch_size > FEATURE_BATCH_MAX)
|
||||
return lardon3d_task_fail(task, "Contrat Governor Feature batch invalide.");
|
||||
size_t count = execution.item_count - persisted.next_item_index;
|
||||
if (count > contract.batch_size) count = contract.batch_size;
|
||||
FeatureBatchWorker workers[FEATURE_BATCH_MAX] = {{0}};
|
||||
for (size_t offset = 0; offset < count; ++offset) {
|
||||
FeatureBatchWorker *worker = &workers[offset];
|
||||
Lardon3DProjectDbSelectedExecutionItem item;
|
||||
if (lardon3d_project_db_load_selected_execution_item(
|
||||
context->database, context->parameters.selected_execution_id,
|
||||
persisted.next_item_index + (uint32_t)offset, &item) != LARDON3D_PROJECT_DB_OK ||
|
||||
!item.has_image)
|
||||
return lardon3d_task_fail(task, "Item sélectionné Feature batch invalide.");
|
||||
worker->context = context;
|
||||
worker->item_index = item.item_index;
|
||||
worker->image_id = item.image_id;
|
||||
Lardon3DFeatureTaskContext single = {0};
|
||||
single.database = context->database;
|
||||
single.parameters.image_id = item.image_id;
|
||||
(void)snprintf(single.project_path, sizeof(single.project_path), "%s", context->project_path);
|
||||
Lardon3DProjectDbResult found = find_valid_feature_set(
|
||||
&single, context->parameters.parameter_fingerprint);
|
||||
worker->already_ready = found == LARDON3D_PROJECT_DB_OK;
|
||||
if (found == LARDON3D_PROJECT_DB_CORRUPT ||
|
||||
(found != LARDON3D_PROJECT_DB_OK && found != LARDON3D_PROJECT_DB_NOT_FOUND) ||
|
||||
(!worker->already_ready && !load_validated_source(&single, worker->source_path)))
|
||||
return lardon3d_task_fail(task, "Source ou Feature Set batch invalide.");
|
||||
}
|
||||
|
||||
struct timespec begin = {0}, end = {0};
|
||||
bool timing_known = clock_gettime(CLOCK_MONOTONIC, &begin) == 0;
|
||||
if (!extract_batch_window(workers, count, contract.cpu_threads)) {
|
||||
destroy_batch_outputs(workers, count);
|
||||
return lardon3d_task_fail(task, "Participants Feature batch impossibles.");
|
||||
}
|
||||
if (!lardon3d_task_checkpoint(task)) {
|
||||
destroy_batch_outputs(workers, count);
|
||||
return false;
|
||||
}
|
||||
size_t durable_items = 0;
|
||||
Lardon3DAppState state;
|
||||
batch_runtime_state(context, &state);
|
||||
Lardon3DFeatureExtractorParameters parameters = {
|
||||
context->parameters.max_features, context->parameters.pyramid_levels,
|
||||
context->parameters.fast_threshold};
|
||||
for (size_t offset = 0; offset < count; ++offset) {
|
||||
FeatureBatchWorker *worker = &workers[offset];
|
||||
if (!worker->already_ready) {
|
||||
if (worker->result != LARDON3D_FEATURE_EXTRACT_OK) {
|
||||
destroy_batch_outputs(workers, count);
|
||||
return lardon3d_task_fail(task, "Extraction ORB batch impossible.");
|
||||
}
|
||||
Lardon3DProjectDbFeatureSet set;
|
||||
Lardon3DFeatureStoreResult published = lardon3d_feature_store_publish(
|
||||
&state, worker->image_id, lardon3d_task_id(task), ¶meters,
|
||||
&worker->features, &set);
|
||||
lardon3d_extracted_features_destroy(&worker->features);
|
||||
if (published != LARDON3D_FEATURE_STORE_OK &&
|
||||
published != LARDON3D_FEATURE_STORE_ALREADY_PRESENT) {
|
||||
destroy_batch_outputs(workers, count);
|
||||
return lardon3d_task_fail(task, "Publication Feature batch impossible.");
|
||||
}
|
||||
if (published == LARDON3D_FEATURE_STORE_OK) ++durable_items;
|
||||
}
|
||||
if (lardon3d_project_db_advance_feature_extract_batch_task(
|
||||
context->database, lardon3d_task_id(task), worker->item_index,
|
||||
worker->item_index + 1u) != LARDON3D_PROJECT_DB_OK ||
|
||||
!lardon3d_task_set_durable_progress(task, worker->item_index + 1u,
|
||||
execution.item_count,
|
||||
"Feature batch publié.") ||
|
||||
!batch_checkpoint(context, task, worker->item_index + 1u)) {
|
||||
destroy_batch_outputs(workers, count);
|
||||
return lardon3d_task_fail(task, "Curseur Feature batch impossible.");
|
||||
}
|
||||
}
|
||||
timing_known = timing_known && clock_gettime(CLOCK_MONOTONIC, &end) == 0;
|
||||
if (timing_known)
|
||||
(void)lardon3d_task_internal_record_sequence(task, elapsed_ns(begin, end), durable_items);
|
||||
if (persisted.next_item_index + count == execution.item_count) return true;
|
||||
Lardon3DResourceReservation *reservation = NULL;
|
||||
if (!lardon3d_task_sequence_break(task, context->governor, &reservation, &contract))
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
static bool run_batch(Lardon3DTask *task, void *userdata) {
|
||||
Lardon3DFeatureBatchTaskContext *context = userdata;
|
||||
Lardon3DTaskExecutionContract contract;
|
||||
if (!lardon3d_task_execution_contract(task, &contract) || contract.cpu_threads == 0)
|
||||
return lardon3d_task_fail(task, "Contrat CPU Feature batch invalide.");
|
||||
Lardon3DOpenCvTaskThreadControl control = {
|
||||
.previous = lardon3d_feature_opencv_thread_count(),
|
||||
.restore_required = true,
|
||||
};
|
||||
/* Cross-image participants are the admitted CPU dimension. OpenCV stays at
|
||||
one thread per extraction to prevent nested oversubscription; Queue owns
|
||||
the sole process-wide guard. Configuration may mutate before verification
|
||||
fails, so restoration duty exists before configure and covers every exit. */
|
||||
if (!lardon3d_feature_opencv_configure_threads(1)) {
|
||||
(void)lardon3d_opencv_task_threads_end(&control);
|
||||
return lardon3d_task_fail(task, "Configuration OpenCV Feature batch impossible.");
|
||||
}
|
||||
bool result = run_batch_body(task, context);
|
||||
if (!lardon3d_opencv_task_threads_end(&control))
|
||||
return lardon3d_task_fail(task, "Restauration OpenCV Feature batch impossible.");
|
||||
return result;
|
||||
}
|
||||
|
||||
#ifdef LARDON3D_FEATURE_TASK_TESTING
|
||||
bool lardon3d_feature_extract_batch_test_run(Lardon3DTask *task) {
|
||||
Lardon3DFeatureBatchTaskContext context = {0};
|
||||
return run_batch(task, &context);
|
||||
}
|
||||
#endif
|
||||
|
||||
static void finished_batch(const Lardon3DTask *task, void *userdata) {
|
||||
Lardon3DFeatureBatchTaskContext *context = userdata;
|
||||
Lardon3DProjectDbFeatureExtractBatchTask persisted;
|
||||
if (lardon3d_project_db_load_feature_extract_batch_task(
|
||||
context->database, lardon3d_task_id(task), &persisted) == LARDON3D_PROJECT_DB_OK)
|
||||
(void)batch_checkpoint(context, (Lardon3DTask *)task, persisted.next_item_index);
|
||||
}
|
||||
|
||||
static bool run_body(Lardon3DTask *task, void *userdata, size_t *durable_items) {
|
||||
Lardon3DFeatureTaskContext *context = userdata;
|
||||
*durable_items = 0;
|
||||
|
|
@ -376,3 +630,121 @@ bool lardon3d_project_enqueue_feature_extract(Lardon3DAppState *state, uint64_t
|
|||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
static Lardon3DFeatureBatchTaskContext *make_batch_context(
|
||||
const Lardon3DTaskReconstructionContext *runtime,
|
||||
const Lardon3DProjectDbFeatureExtractBatchTask *parameters) {
|
||||
if (!runtime || !runtime->project_path || !runtime->project_db ||
|
||||
!runtime->resource_governor || !parameters) return NULL;
|
||||
Lardon3DFeatureBatchTaskContext *context = calloc(1, sizeof(*context));
|
||||
if (!context) return NULL;
|
||||
int n = snprintf(context->project_path, sizeof(context->project_path), "%s",
|
||||
runtime->project_path);
|
||||
if (n <= 0 || (size_t)n >= sizeof(context->project_path)) {
|
||||
free(context);
|
||||
return NULL;
|
||||
}
|
||||
context->database = runtime->project_db;
|
||||
context->governor = runtime->resource_governor;
|
||||
context->parameters = *parameters;
|
||||
return context;
|
||||
}
|
||||
|
||||
bool lardon3d_feature_extract_batch_reconstruct(
|
||||
const Lardon3DTaskDurableSnapshot *snapshot, void *userdata,
|
||||
Lardon3DTaskKindBinding *binding) {
|
||||
Lardon3DTaskReconstructionContext *runtime = userdata;
|
||||
if (!snapshot || !runtime || !binding) return false;
|
||||
Lardon3DProjectDbFeatureExtractBatchTask parameters;
|
||||
if (lardon3d_project_db_load_feature_extract_batch_task(
|
||||
runtime->project_db, snapshot->id, ¶meters) != LARDON3D_PROJECT_DB_OK)
|
||||
return false;
|
||||
unsigned char expected[32];
|
||||
lardon3d_feature_extractor_parameter_fingerprint(
|
||||
&(Lardon3DFeatureExtractorParameters){parameters.max_features,
|
||||
parameters.pyramid_levels,
|
||||
parameters.fast_threshold}, expected);
|
||||
if (memcmp(expected, parameters.parameter_fingerprint, 32) != 0) return false;
|
||||
Lardon3DFeatureBatchTaskContext *context = make_batch_context(runtime, ¶meters);
|
||||
if (!context) return false;
|
||||
*binding = (Lardon3DTaskKindBinding){.callback = run_batch,
|
||||
.userdata = context,
|
||||
.userdata_destroy = destroy_context,
|
||||
.finished_callback = finished_batch,
|
||||
.finished_userdata = context};
|
||||
return true;
|
||||
}
|
||||
|
||||
Lardon3DTask *lardon3d_project_create_feature_extract_batch_task(
|
||||
Lardon3DAppState *state, uint64_t selected_execution_id,
|
||||
const Lardon3DFeatureExtractorParameters *parameters, uint64_t *task_id) {
|
||||
if (task_id) *task_id = 0;
|
||||
if (!state || !state->project_loaded || !state->project_db ||
|
||||
!state->resource_governor || !parameters || !task_id ||
|
||||
!lardon3d_feature_extractor_parameters_valid(parameters) || selected_execution_id == 0)
|
||||
return NULL;
|
||||
Lardon3DProjectDbSelectedExecution execution;
|
||||
if (lardon3d_project_db_load_selected_execution(state->project_db, selected_execution_id,
|
||||
&execution) != LARDON3D_PROJECT_DB_OK ||
|
||||
execution.stage < LARDON3D_SELECTED_EXECUTION_CALIBRATION || execution.item_count == 0)
|
||||
return NULL;
|
||||
uint64_t id = 0;
|
||||
if (lardon3d_project_db_allocate_task_id(state->project_db, &id) != LARDON3D_PROJECT_DB_OK)
|
||||
return NULL;
|
||||
Lardon3DProjectDbFeatureExtractBatchTask durable = {
|
||||
.task_id = id,
|
||||
.selected_execution_id = selected_execution_id,
|
||||
.next_item_index = 0,
|
||||
.extractor_version = LARDON3D_FEATURE_EXTRACTOR_VERSION,
|
||||
.max_features = parameters->max_features,
|
||||
.pyramid_levels = parameters->pyramid_levels,
|
||||
.fast_threshold = parameters->fast_threshold};
|
||||
(void)snprintf(durable.extractor_kind, sizeof(durable.extractor_kind), "%s",
|
||||
LARDON3D_FEATURE_EXTRACTOR_KIND);
|
||||
lardon3d_feature_extractor_parameter_fingerprint(parameters,
|
||||
durable.parameter_fingerprint);
|
||||
Lardon3DTaskReconstructionContext runtime = {
|
||||
state->project_path, state->project_db, state->resource_governor,
|
||||
state->orb_vulkan_backend};
|
||||
Lardon3DFeatureBatchTaskContext *context = make_batch_context(&runtime, &durable);
|
||||
if (!context) return NULL;
|
||||
/* One admitted participant owns decode, ORB pyramid/keypoints/descriptors,
|
||||
and publication staging bounded by the existing conservative 512 MiB/item.
|
||||
The 1..12 ceiling is operational benchmark capacity, not selected-domain
|
||||
cardinality; Governor feedback records each joined window's useful items. */
|
||||
Lardon3DResourceEstimate estimate = {
|
||||
.memory_fixed_bytes = 64ULL * 1024 * 1024,
|
||||
.memory_bytes_per_item = 512ULL * 1024 * 1024,
|
||||
.minimum_batch_size = 1,
|
||||
.maximum_batch_size = FEATURE_BATCH_MAX,
|
||||
.desired_cpu_threads = FEATURE_BATCH_MAX,
|
||||
.desired_io_slots = 1,
|
||||
.task_class = LARDON3D_RESOURCE_TASK_CPU};
|
||||
Lardon3DTask *task = lardon3d_task_create_typed(
|
||||
"Extraction Feature sélectionnée", &estimate,
|
||||
LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND,
|
||||
LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION, run_batch, context,
|
||||
destroy_context);
|
||||
if (!task || !lardon3d_task_assign_id(task, id) ||
|
||||
!lardon3d_task_set_finished_callback(task, finished_batch, context) ||
|
||||
!batch_checkpoint(context, task, 0)) {
|
||||
lardon3d_task_destroy(task);
|
||||
return NULL;
|
||||
}
|
||||
*task_id = id;
|
||||
return task;
|
||||
}
|
||||
|
||||
bool lardon3d_project_enqueue_feature_extract_batch(
|
||||
Lardon3DAppState *state, uint64_t selected_execution_id,
|
||||
const Lardon3DFeatureExtractorParameters *parameters, uint64_t *task_id) {
|
||||
if (!state || !state->task_queue) return false;
|
||||
Lardon3DTask *task = lardon3d_project_create_feature_extract_batch_task(
|
||||
state, selected_execution_id, parameters, task_id);
|
||||
if (!task) return false;
|
||||
if (!lardon3d_task_queue_add(state->task_queue, task, NULL)) {
|
||||
lardon3d_task_destroy(task);
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -717,7 +717,9 @@ checkpoint_task_internal(Lardon3DAppState *state, const Lardon3DTask *task,
|
|||
const Lardon3DProjectDbGeometricVerifierTask *geometric_parameters,
|
||||
const Lardon3DProjectDbAcquisitionCampaignTask *campaign_parameters,
|
||||
const Lardon3DProjectDbPhotoQualityTask *quality_parameters,
|
||||
const Lardon3DProjectDbRawDevelopmentTask *raw_parameters) {
|
||||
const Lardon3DProjectDbRawDevelopmentTask *raw_parameters,
|
||||
const Lardon3DProjectDbRawDevelopmentBatchTask *raw_batch_parameters,
|
||||
const Lardon3DProjectDbFeatureExtractBatchTask *feature_batch_parameters) {
|
||||
if (!state || !state->project_loaded || !state->project_db) {
|
||||
return LARDON3D_PROJECT_TASK_CHECKPOINT_NO_PROJECT;
|
||||
}
|
||||
|
|
@ -813,6 +815,14 @@ checkpoint_task_internal(Lardon3DAppState *state, const Lardon3DTask *task,
|
|||
? lardon3d_project_db_record_raw_development_task(
|
||||
state->project_db, &snapshot, task_kind, task_kind_version, &checkpoint,
|
||||
raw_parameters, now.tv_sec)
|
||||
: raw_batch_parameters
|
||||
? lardon3d_project_db_record_raw_development_batch_task(
|
||||
state->project_db, &snapshot, task_kind, task_kind_version, &checkpoint,
|
||||
raw_batch_parameters, now.tv_sec)
|
||||
: feature_batch_parameters
|
||||
? lardon3d_project_db_record_feature_extract_batch_task(
|
||||
state->project_db, &snapshot, task_kind, task_kind_version, &checkpoint,
|
||||
feature_batch_parameters, now.tv_sec)
|
||||
: lardon3d_project_db_record_task(state->project_db, &snapshot, task_kind,
|
||||
task_kind_version, &checkpoint, now.tv_sec);
|
||||
if (recorded == LARDON3D_PROJECT_DB_BUSY) {
|
||||
|
|
@ -841,7 +851,7 @@ checkpoint_task_internal(Lardon3DAppState *state, const Lardon3DTask *task,
|
|||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_task(Lardon3DAppState *state,
|
||||
const Lardon3DTask *task) {
|
||||
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
NULL, NULL);
|
||||
NULL, NULL, NULL, NULL);
|
||||
}
|
||||
|
||||
Lardon3DProjectTaskCheckpointResult
|
||||
|
|
@ -851,7 +861,8 @@ lardon3d_project_checkpoint_image_import_task(Lardon3DAppState *state, const Lar
|
|||
return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
|
||||
}
|
||||
return checkpoint_task_internal(
|
||||
state, task, source_path, scanset_id, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
|
||||
state, task, source_path, scanset_id, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
NULL, NULL);
|
||||
}
|
||||
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_acquisition_campaign_task(
|
||||
|
|
@ -859,7 +870,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_acquisition_camp
|
|||
const Lardon3DProjectDbAcquisitionCampaignTask *parameters) {
|
||||
if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
|
||||
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
parameters, NULL, NULL);
|
||||
parameters, NULL, NULL, NULL, NULL);
|
||||
}
|
||||
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_photo_quality_task(
|
||||
|
|
@ -867,7 +878,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_photo_quality_ta
|
|||
const Lardon3DProjectDbPhotoQualityTask *parameters) {
|
||||
if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
|
||||
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
parameters, NULL);
|
||||
parameters, NULL, NULL, NULL);
|
||||
}
|
||||
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_raw_development_task(
|
||||
|
|
@ -875,7 +886,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_raw_development_
|
|||
const Lardon3DProjectDbRawDevelopmentTask *parameters) {
|
||||
if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
|
||||
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
NULL, parameters);
|
||||
NULL, parameters, NULL, NULL);
|
||||
}
|
||||
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_task(
|
||||
|
|
@ -885,7 +896,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_
|
|||
return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
|
||||
}
|
||||
return checkpoint_task_internal(state, task, NULL, 0, parameters, NULL, NULL, NULL, NULL, NULL,
|
||||
NULL, NULL, NULL);
|
||||
NULL, NULL, NULL, NULL, NULL);
|
||||
}
|
||||
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_sift_extract_task(
|
||||
|
|
@ -893,7 +904,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_sift_extract_tas
|
|||
const Lardon3DProjectDbSiftExtractTask *parameters) {
|
||||
if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
|
||||
return checkpoint_task_internal(state, task, NULL, 0, NULL, parameters, NULL, NULL, NULL, NULL,
|
||||
NULL, NULL, NULL);
|
||||
NULL, NULL, NULL, NULL, NULL);
|
||||
}
|
||||
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_visual_index_update_task(
|
||||
|
|
@ -903,7 +914,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_visual_index_upd
|
|||
return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
|
||||
}
|
||||
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, parameters, NULL, NULL, NULL,
|
||||
NULL, NULL, NULL);
|
||||
NULL, NULL, NULL, NULL, NULL);
|
||||
}
|
||||
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_candidate_pair_generate_task(
|
||||
|
|
@ -913,7 +924,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_candidate_pair_g
|
|||
return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
|
||||
}
|
||||
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, parameters, NULL, NULL,
|
||||
NULL, NULL, NULL);
|
||||
NULL, NULL, NULL, NULL, NULL);
|
||||
}
|
||||
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_matcher_task(
|
||||
|
|
@ -923,7 +934,7 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_matcher_task(
|
|||
return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
|
||||
}
|
||||
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, parameters, NULL,
|
||||
NULL, NULL, NULL);
|
||||
NULL, NULL, NULL, NULL, NULL);
|
||||
}
|
||||
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_geometric_verifier_task(
|
||||
|
|
@ -933,7 +944,23 @@ Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_geometric_verifi
|
|||
return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
|
||||
}
|
||||
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL,
|
||||
parameters, NULL, NULL, NULL);
|
||||
parameters, NULL, NULL, NULL, NULL, NULL);
|
||||
}
|
||||
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_raw_development_batch_task(
|
||||
Lardon3DAppState *state, const Lardon3DTask *task,
|
||||
const Lardon3DProjectDbRawDevelopmentBatchTask *parameters) {
|
||||
if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
|
||||
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
NULL, NULL, parameters, NULL);
|
||||
}
|
||||
|
||||
Lardon3DProjectTaskCheckpointResult lardon3d_project_checkpoint_feature_extract_batch_task(
|
||||
Lardon3DAppState *state, const Lardon3DTask *task,
|
||||
const Lardon3DProjectDbFeatureExtractBatchTask *parameters) {
|
||||
if (!parameters) return LARDON3D_PROJECT_TASK_CHECKPOINT_INVALID_TASK;
|
||||
return checkpoint_task_internal(state, task, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
NULL, NULL, NULL, parameters);
|
||||
}
|
||||
|
||||
static bool coherent_recovery(const Lardon3DProjectDbTask *database_task,
|
||||
|
|
|
|||
345
src/project_db.c
345
src/project_db.c
|
|
@ -12,6 +12,7 @@
|
|||
#include <time.h>
|
||||
|
||||
#include <lardon3d/acquisition_campaign_task.h>
|
||||
#include <lardon3d/feature_task.h>
|
||||
#include <lardon3d/optical_profiles.h>
|
||||
#include <lardon3d/photo_quality_task.h>
|
||||
#include <lardon3d/project_db.h>
|
||||
|
|
@ -851,6 +852,36 @@ static const char schema_optical_calibrations_v23[] =
|
|||
"FOREIGN KEY(calibration_profile_id,optical_configuration_id) REFERENCES "
|
||||
"optical_calibration_profiles(calibration_profile_id,optical_configuration_id));";
|
||||
|
||||
/* PERSISTENCE CONTRACT v24: this additive row stores only the operational
|
||||
Task -> selected-execution association. Selected execution items/cursor stay
|
||||
authoritative, and migration creates no rows or inferred identities. */
|
||||
static const char schema_raw_development_batch_v24[] =
|
||||
"CREATE TABLE raw_development_batch_tasks("
|
||||
"task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE CASCADE,"
|
||||
"selected_execution_id INTEGER NOT NULL UNIQUE REFERENCES "
|
||||
"selected_executions(execution_id));";
|
||||
|
||||
/* PERSISTENCE CONTRACT v25: this operational row anchors one owner to the
|
||||
immutable selected-execution order and exact ORB parameter domain. Its
|
||||
monotone cursor is a durable prefix only; migration creates no rows. */
|
||||
static const char schema_feature_extract_batch_v25[] =
|
||||
"CREATE TABLE feature_extract_batch_tasks("
|
||||
"task_id INTEGER PRIMARY KEY REFERENCES tasks(task_id) ON DELETE CASCADE,"
|
||||
"selected_execution_id INTEGER NOT NULL UNIQUE REFERENCES selected_executions(execution_id),"
|
||||
"next_item_index INTEGER NOT NULL CHECK(typeof(next_item_index)='integer' AND "
|
||||
"next_item_index>=0 AND next_item_index<=4096),"
|
||||
"extractor_kind TEXT NOT NULL CHECK(typeof(extractor_kind)='text' AND extractor_kind='orb'),"
|
||||
"extractor_version INTEGER NOT NULL CHECK(typeof(extractor_version)='integer' AND "
|
||||
"extractor_version=1),"
|
||||
"max_features INTEGER NOT NULL CHECK(typeof(max_features)='integer' AND max_features>0 AND "
|
||||
"max_features<=8192),"
|
||||
"pyramid_levels INTEGER NOT NULL CHECK(typeof(pyramid_levels)='integer' AND "
|
||||
"pyramid_levels>0 AND pyramid_levels<=16),"
|
||||
"fast_threshold INTEGER NOT NULL CHECK(typeof(fast_threshold)='integer' AND "
|
||||
"fast_threshold>0 AND fast_threshold<=255),"
|
||||
"parameter_fingerprint BLOB NOT NULL CHECK(typeof(parameter_fingerprint)='blob' AND "
|
||||
"length(parameter_fingerprint)=32));";
|
||||
|
||||
static void copy_error(char destination[LARDON3D_PROJECT_DB_ERROR_CAPACITY], const char *text) {
|
||||
if (destination) {
|
||||
(void)snprintf(destination, LARDON3D_PROJECT_DB_ERROR_CAPACITY, "%s", text ? text : "");
|
||||
|
|
@ -943,7 +974,8 @@ static Lardon3DProjectDbResult migrate(Lardon3DProjectDb *database, unsigned int
|
|||
from_version != 8 && from_version != 9 && from_version != 10 && from_version != 11 &&
|
||||
from_version != 12 && from_version != 13 && from_version != 14 &&
|
||||
from_version != 15 && from_version != 16 && from_version != 17 && from_version != 18 &&
|
||||
from_version != 19 && from_version != 20 && from_version != 21 && from_version != 22) {
|
||||
from_version != 19 && from_version != 20 && from_version != 21 && from_version != 22 &&
|
||||
from_version != 23 && from_version != 24) {
|
||||
return LARDON3D_PROJECT_DB_CORRUPT;
|
||||
}
|
||||
Lardon3DProjectDbResult result = execute(database, "BEGIN IMMEDIATE", "begin migration");
|
||||
|
|
@ -1414,6 +1446,46 @@ static Lardon3DProjectDbResult migrate(Lardon3DProjectDb *database, unsigned int
|
|||
result = LARDON3D_PROJECT_DB_CORRUPT;
|
||||
}
|
||||
}
|
||||
if (result == LARDON3D_PROJECT_DB_OK && from_version < 24) {
|
||||
/* v23 contains no evidence that a batch Task existed, so migration is
|
||||
DDL-only and cannot synthesize selected execution associations. */
|
||||
result = execute(database, schema_raw_development_batch_v24,
|
||||
"migrate RAW development batch v23 to v24");
|
||||
#ifdef LARDON3D_PROJECT_DB_TESTING
|
||||
if (result == LARDON3D_PROJECT_DB_OK &&
|
||||
getenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V24")) {
|
||||
result = execute(database, "INSERT INTO missing_v24_test_table VALUES(1)",
|
||||
"forced migration v24 failure");
|
||||
}
|
||||
#endif
|
||||
if (result == LARDON3D_PROJECT_DB_OK) {
|
||||
result = execute(database,
|
||||
"UPDATE metadata SET value=24 WHERE key='schema_version' AND value=23",
|
||||
"finish schema v24 migration");
|
||||
if (result == LARDON3D_PROJECT_DB_OK &&
|
||||
sqlite3_changes(database->connection) != 1)
|
||||
result = LARDON3D_PROJECT_DB_CORRUPT;
|
||||
}
|
||||
}
|
||||
if (result == LARDON3D_PROJECT_DB_OK && from_version < 25) {
|
||||
/* No v24 row identifies a Feature batch or cursor. Keep this DDL-only so
|
||||
historical single-image Tasks remain exactly as persisted. */
|
||||
result = execute(database, schema_feature_extract_batch_v25,
|
||||
"migrate feature extraction batch v24 to v25");
|
||||
#ifdef LARDON3D_PROJECT_DB_TESTING
|
||||
if (result == LARDON3D_PROJECT_DB_OK &&
|
||||
getenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V25"))
|
||||
result = execute(database, "INSERT INTO missing_v25_test_table VALUES(1)",
|
||||
"forced migration v25 failure");
|
||||
#endif
|
||||
if (result == LARDON3D_PROJECT_DB_OK) {
|
||||
result = execute(database,
|
||||
"UPDATE metadata SET value=25 WHERE key='schema_version' AND value=24",
|
||||
"finish schema v25 migration");
|
||||
if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1)
|
||||
result = LARDON3D_PROJECT_DB_CORRUPT;
|
||||
}
|
||||
}
|
||||
if (result == LARDON3D_PROJECT_DB_OK) {
|
||||
result = execute(database, "COMMIT", "commit migration");
|
||||
}
|
||||
|
|
@ -1550,6 +1622,8 @@ Lardon3DProjectDbResult lardon3d_project_db_open(const char *path, Lardon3DProje
|
|||
"capture_assets",
|
||||
"capture_source_assets",
|
||||
"raw_development_tasks",
|
||||
"raw_development_batch_tasks",
|
||||
"feature_extract_batch_tasks",
|
||||
"capture_selections",
|
||||
"asset_derivations",
|
||||
"acquisition_campaign_tasks",
|
||||
|
|
@ -1821,6 +1895,8 @@ record_task_internal(Lardon3DProjectDb *database, const Lardon3DTaskDurableSnaps
|
|||
const Lardon3DProjectDbAcquisitionCampaignTask *campaign,
|
||||
const Lardon3DProjectDbPhotoQualityTask *photo_quality,
|
||||
const Lardon3DProjectDbRawDevelopmentTask *raw_development,
|
||||
const Lardon3DProjectDbRawDevelopmentBatchTask *raw_batch,
|
||||
const Lardon3DProjectDbFeatureExtractBatchTask *feature_batch,
|
||||
int64_t updated_at) {
|
||||
bool typed = task_kind != NULL;
|
||||
if (!database || !valid_durable_task(snapshot, updated_at) ||
|
||||
|
|
@ -1924,6 +2000,21 @@ record_task_internal(Lardon3DProjectDb *database, const Lardon3DTaskDurableSnaps
|
|||
(raw_development->has_image || raw_development->image_id != 0)) ||
|
||||
(raw_development->phase == LARDON3D_RAW_DEVELOPMENT_TASK_PUBLISHED &&
|
||||
(!raw_development->has_image || !valid_task_id(raw_development->image_id))))) ||
|
||||
(raw_batch &&
|
||||
(!task_kind || strcmp(task_kind, LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND) != 0 ||
|
||||
task_kind_version != LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION ||
|
||||
!valid_task_id(raw_batch->task_id) || raw_batch->task_id != snapshot->id ||
|
||||
!valid_task_id(raw_batch->selected_execution_id))) ||
|
||||
(feature_batch &&
|
||||
(!task_kind || strcmp(task_kind, LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND) != 0 ||
|
||||
task_kind_version != LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION ||
|
||||
!valid_task_id(feature_batch->task_id) || feature_batch->task_id != snapshot->id ||
|
||||
!valid_task_id(feature_batch->selected_execution_id) ||
|
||||
strcmp(feature_batch->extractor_kind, LARDON3D_FEATURE_EXTRACTOR_KIND) != 0 ||
|
||||
feature_batch->extractor_version != LARDON3D_FEATURE_EXTRACTOR_VERSION ||
|
||||
feature_batch->max_features == 0 || feature_batch->max_features > 8192 ||
|
||||
feature_batch->pyramid_levels == 0 || feature_batch->pyramid_levels > 16 ||
|
||||
feature_batch->fast_threshold == 0 || feature_batch->fast_threshold > 255)) ||
|
||||
(checkpoint && !valid_checkpoint(checkpoint))) {
|
||||
return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
|
||||
}
|
||||
|
|
@ -2064,6 +2155,64 @@ record_task_internal(Lardon3DProjectDb *database, const Lardon3DTaskDurableSnaps
|
|||
}
|
||||
}
|
||||
}
|
||||
if (result == LARDON3D_PROJECT_DB_OK && raw_batch) {
|
||||
/* The selected execution owns every scientific item and its cursor. This
|
||||
immutable row only makes the parent operational Task reconstructible. */
|
||||
result = prepare(database,
|
||||
"INSERT INTO raw_development_batch_tasks(task_id,selected_execution_id) "
|
||||
"VALUES(?1,?2) ON CONFLICT(task_id) DO UPDATE SET "
|
||||
"selected_execution_id=excluded.selected_execution_id WHERE "
|
||||
"raw_development_batch_tasks.selected_execution_id=excluded.selected_execution_id",
|
||||
&statement);
|
||||
if (result == LARDON3D_PROJECT_DB_OK) {
|
||||
sqlite3_bind_int64(statement, 1, (sqlite3_int64)raw_batch->task_id);
|
||||
sqlite3_bind_int64(statement, 2,
|
||||
(sqlite3_int64)raw_batch->selected_execution_id);
|
||||
result = step_done(database, statement, "upsert RAW development batch task");
|
||||
if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1) {
|
||||
copy_error(database->error, "Exécution sélectionnée RAW batch immuable.");
|
||||
result = LARDON3D_PROJECT_DB_CONSTRAINT;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (result == LARDON3D_PROJECT_DB_OK && feature_batch) {
|
||||
/* Immutable domain fields never change. Only an equal-or-newer cursor may
|
||||
be repeated by a generic checkpoint that lagged an item publication. */
|
||||
result = prepare(database,
|
||||
"INSERT INTO feature_extract_batch_tasks(task_id,selected_execution_id,"
|
||||
"next_item_index,extractor_kind,extractor_version,max_features,pyramid_levels,"
|
||||
"fast_threshold,parameter_fingerprint) SELECT ?1,?2,?3,?4,?5,?6,?7,?8,?9 "
|
||||
"FROM selected_executions e WHERE e.execution_id=?2 AND e.item_count>0 AND "
|
||||
"?3<=e.item_count AND e.stage>=?10 "
|
||||
"ON CONFLICT(task_id) DO UPDATE SET next_item_index=MAX("
|
||||
"feature_extract_batch_tasks.next_item_index,excluded.next_item_index) WHERE "
|
||||
"feature_extract_batch_tasks.selected_execution_id=excluded.selected_execution_id AND "
|
||||
"feature_extract_batch_tasks.extractor_kind=excluded.extractor_kind AND "
|
||||
"feature_extract_batch_tasks.extractor_version=excluded.extractor_version AND "
|
||||
"feature_extract_batch_tasks.max_features=excluded.max_features AND "
|
||||
"feature_extract_batch_tasks.pyramid_levels=excluded.pyramid_levels AND "
|
||||
"feature_extract_batch_tasks.fast_threshold=excluded.fast_threshold AND "
|
||||
"feature_extract_batch_tasks.parameter_fingerprint=excluded.parameter_fingerprint AND "
|
||||
"excluded.next_item_index<=(SELECT item_count FROM selected_executions WHERE "
|
||||
"execution_id=excluded.selected_execution_id)", &statement);
|
||||
if (result == LARDON3D_PROJECT_DB_OK) {
|
||||
sqlite3_bind_int64(statement, 1, (sqlite3_int64)feature_batch->task_id);
|
||||
sqlite3_bind_int64(statement, 2, (sqlite3_int64)feature_batch->selected_execution_id);
|
||||
sqlite3_bind_int64(statement, 3, feature_batch->next_item_index);
|
||||
sqlite3_bind_text(statement, 4, feature_batch->extractor_kind, -1, SQLITE_TRANSIENT);
|
||||
sqlite3_bind_int64(statement, 5, feature_batch->extractor_version);
|
||||
sqlite3_bind_int64(statement, 6, feature_batch->max_features);
|
||||
sqlite3_bind_int64(statement, 7, feature_batch->pyramid_levels);
|
||||
sqlite3_bind_int64(statement, 8, feature_batch->fast_threshold);
|
||||
sqlite3_bind_blob(statement, 9, feature_batch->parameter_fingerprint, 32, SQLITE_TRANSIENT);
|
||||
sqlite3_bind_int64(statement, 10, LARDON3D_SELECTED_EXECUTION_CALIBRATION);
|
||||
result = step_done(database, statement, "upsert feature extraction batch task");
|
||||
if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(database->connection) != 1) {
|
||||
copy_error(database->error, "Domaine ou curseur Feature batch incohérent.");
|
||||
result = LARDON3D_PROJECT_DB_CONSTRAINT;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (result == LARDON3D_PROJECT_DB_OK) {
|
||||
result =
|
||||
prepare(database,
|
||||
|
|
@ -2398,7 +2547,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_task(
|
|||
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot, const char *task_kind,
|
||||
uint32_t task_kind_version, const Lardon3DProjectDbCheckpoint *checkpoint, int64_t updated_at) {
|
||||
return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0,
|
||||
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, updated_at);
|
||||
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
NULL, updated_at);
|
||||
}
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_record_image_import_task(
|
||||
|
|
@ -2410,7 +2560,7 @@ Lardon3DProjectDbResult lardon3d_project_db_record_image_import_task(
|
|||
}
|
||||
return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint,
|
||||
source_path, scanset_id, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
NULL, NULL, NULL, NULL, updated_at);
|
||||
NULL, NULL, NULL, NULL, NULL, NULL, updated_at);
|
||||
}
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_task(
|
||||
|
|
@ -2421,8 +2571,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_task(
|
|||
return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
|
||||
}
|
||||
return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0,
|
||||
parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
updated_at);
|
||||
parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
NULL, updated_at);
|
||||
}
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_record_sift_extract_task(
|
||||
|
|
@ -2432,8 +2582,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_sift_extract_task(
|
|||
const Lardon3DProjectDbSiftExtractTask *parameters, int64_t updated_at) {
|
||||
if (!parameters) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
|
||||
return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0,
|
||||
NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
updated_at);
|
||||
NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
NULL, updated_at);
|
||||
}
|
||||
|
||||
static bool read_task(sqlite3_stmt *statement, Lardon3DProjectDbTask *task) {
|
||||
|
|
@ -2680,7 +2830,7 @@ Lardon3DProjectDbResult lardon3d_project_db_record_acquisition_campaign_task(
|
|||
never observes a valid campaign Task without its immutable request blob. */
|
||||
return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint,
|
||||
NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, parameters, NULL,
|
||||
NULL, updated_at);
|
||||
NULL, NULL, NULL, updated_at);
|
||||
}
|
||||
|
||||
static Lardon3DProjectDbResult validate_acquisition_campaign_prefix_locked(
|
||||
|
|
@ -3397,8 +3547,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_photo_quality_task(
|
|||
/* Generic runtime state and typed immutable request share the transaction;
|
||||
recovery cannot observe one without the other. */
|
||||
return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0,
|
||||
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, parameters, NULL,
|
||||
updated_at);
|
||||
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, parameters, NULL, NULL,
|
||||
NULL, updated_at);
|
||||
}
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_photo_quality_task(
|
||||
|
|
@ -3470,8 +3620,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_raw_development_task(
|
|||
const Lardon3DProjectDbRawDevelopmentTask *parameters, int64_t updated_at) {
|
||||
if (!parameters) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
|
||||
return record_task_internal(database, snapshot, task_kind, task_kind_version, checkpoint, NULL, 0,
|
||||
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, parameters,
|
||||
updated_at);
|
||||
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, parameters, NULL,
|
||||
NULL, updated_at);
|
||||
}
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_raw_development_task(
|
||||
|
|
@ -3537,6 +3687,61 @@ Lardon3DProjectDbResult lardon3d_project_db_load_raw_development_task(
|
|||
return result;
|
||||
}
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_record_raw_development_batch_task(
|
||||
Lardon3DProjectDb *database, const Lardon3DTaskDurableSnapshot *snapshot,
|
||||
const char *task_kind, uint32_t task_kind_version,
|
||||
const Lardon3DProjectDbCheckpoint *checkpoint,
|
||||
const Lardon3DProjectDbRawDevelopmentBatchTask *parameters, int64_t updated_at) {
|
||||
if (!parameters) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
|
||||
return record_task_internal(database, snapshot, task_kind, task_kind_version,
|
||||
checkpoint, NULL, 0, NULL, NULL, NULL, NULL, NULL,
|
||||
NULL, NULL, NULL, NULL, NULL, parameters, NULL, updated_at);
|
||||
}
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_raw_development_batch_task(
|
||||
Lardon3DProjectDb *database, uint64_t task_id,
|
||||
Lardon3DProjectDbRawDevelopmentBatchTask *parameters) {
|
||||
if (!database || !valid_task_id(task_id) || !parameters)
|
||||
return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
|
||||
memset(parameters, 0, sizeof(*parameters));
|
||||
(void)pthread_mutex_lock(&database->mutex);
|
||||
sqlite3_stmt *statement = NULL;
|
||||
Lardon3DProjectDbResult result = prepare(
|
||||
database,
|
||||
"SELECT b.selected_execution_id,t.task_kind,t.task_kind_version,e.execution_id "
|
||||
"FROM raw_development_batch_tasks b JOIN tasks t ON t.task_id=b.task_id "
|
||||
"LEFT JOIN selected_executions e ON e.execution_id=b.selected_execution_id "
|
||||
"WHERE b.task_id=?1", &statement);
|
||||
if (result == LARDON3D_PROJECT_DB_OK) {
|
||||
sqlite3_bind_int64(statement, 1, (sqlite3_int64)task_id);
|
||||
int code = sqlite3_step(statement);
|
||||
if (code == SQLITE_DONE) {
|
||||
result = LARDON3D_PROJECT_DB_NOT_FOUND;
|
||||
} else if (code != SQLITE_ROW) {
|
||||
result = sqlite_result(database, code, "load RAW development batch task");
|
||||
} else {
|
||||
sqlite3_int64 execution = sqlite3_column_int64(statement, 0);
|
||||
sqlite3_int64 version = sqlite3_column_int64(statement, 2);
|
||||
if (sqlite3_column_type(statement, 0) != SQLITE_INTEGER || execution <= 0 ||
|
||||
sqlite3_column_type(statement, 1) != SQLITE_TEXT ||
|
||||
sqlite3_column_type(statement, 2) != SQLITE_INTEGER ||
|
||||
sqlite3_column_type(statement, 3) != SQLITE_INTEGER ||
|
||||
sqlite3_column_int64(statement, 3) != execution ||
|
||||
!column_text_equals(statement, 1,
|
||||
LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND) ||
|
||||
version != LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION) {
|
||||
result = LARDON3D_PROJECT_DB_CORRUPT;
|
||||
} else {
|
||||
parameters->task_id = task_id;
|
||||
parameters->selected_execution_id = (uint64_t)execution;
|
||||
}
|
||||
}
|
||||
sqlite3_finalize(statement);
|
||||
}
|
||||
(void)pthread_mutex_unlock(&database->mutex);
|
||||
return result;
|
||||
}
|
||||
|
||||
static bool valid_quality_result(const Lardon3DProjectDbPhotoQualityResult *r) {
|
||||
const Lardon3DPhotoQualityMetrics *m = r ? &r->metrics : NULL;
|
||||
bool decoded_pair = m && ((m->decoded_width == 0) == (m->decoded_height == 0));
|
||||
|
|
@ -6466,6 +6671,105 @@ lardon3d_project_db_load_feature_extract_task(Lardon3DProjectDb *db, uint64_t ta
|
|||
return result;
|
||||
}
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_record_feature_extract_batch_task(
|
||||
Lardon3DProjectDb *db, const Lardon3DTaskDurableSnapshot *snapshot,
|
||||
const char *kind, uint32_t version, const Lardon3DProjectDbCheckpoint *checkpoint,
|
||||
const Lardon3DProjectDbFeatureExtractBatchTask *parameters, int64_t updated_at) {
|
||||
if (!parameters) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
|
||||
return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0,
|
||||
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
NULL, NULL, parameters, updated_at);
|
||||
}
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_feature_extract_batch_task(
|
||||
Lardon3DProjectDb *db, uint64_t task_id,
|
||||
Lardon3DProjectDbFeatureExtractBatchTask *p) {
|
||||
if (!db || !valid_task_id(task_id) || !p) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
|
||||
memset(p, 0, sizeof(*p));
|
||||
pthread_mutex_lock(&db->mutex);
|
||||
sqlite3_stmt *s = NULL;
|
||||
Lardon3DProjectDbResult result = prepare(
|
||||
db, "SELECT b.selected_execution_id,b.next_item_index,b.extractor_kind,"
|
||||
"b.extractor_version,b.max_features,b.pyramid_levels,b.fast_threshold,"
|
||||
"b.parameter_fingerprint,e.item_count,e.stage,t.task_kind,t.task_kind_version "
|
||||
"FROM feature_extract_batch_tasks b JOIN selected_executions e ON "
|
||||
"e.execution_id=b.selected_execution_id JOIN tasks t ON t.task_id=b.task_id "
|
||||
"WHERE b.task_id=?1", &s);
|
||||
if (result == LARDON3D_PROJECT_DB_OK) {
|
||||
sqlite3_bind_int64(s, 1, (sqlite3_int64)task_id);
|
||||
int code = sqlite3_step(s);
|
||||
if (code == SQLITE_DONE) result = LARDON3D_PROJECT_DB_NOT_FOUND;
|
||||
else if (code != SQLITE_ROW) result = sqlite_result(db, code, "load feature batch task");
|
||||
else {
|
||||
sqlite3_int64 execution = sqlite3_column_int64(s, 0);
|
||||
sqlite3_int64 cursor = sqlite3_column_int64(s, 1);
|
||||
sqlite3_int64 version = sqlite3_column_int64(s, 3);
|
||||
sqlite3_int64 max = sqlite3_column_int64(s, 4);
|
||||
sqlite3_int64 levels = sqlite3_column_int64(s, 5);
|
||||
sqlite3_int64 threshold = sqlite3_column_int64(s, 6);
|
||||
sqlite3_int64 count = sqlite3_column_int64(s, 8);
|
||||
sqlite3_int64 stage = sqlite3_column_int64(s, 9);
|
||||
const void *fp = sqlite3_column_blob(s, 7);
|
||||
const unsigned char *task_kind = sqlite3_column_text(s, 10);
|
||||
if (sqlite3_column_type(s, 0) != SQLITE_INTEGER || execution <= 0 ||
|
||||
sqlite3_column_type(s, 1) != SQLITE_INTEGER || cursor < 0 || cursor > count ||
|
||||
!copy_column(s, 2, p->extractor_kind, sizeof(p->extractor_kind)) ||
|
||||
strcmp(p->extractor_kind, LARDON3D_FEATURE_EXTRACTOR_KIND) != 0 || version != 1 ||
|
||||
max <= 0 || max > 8192 || levels <= 0 || levels > 16 || threshold <= 0 ||
|
||||
threshold > 255 || sqlite3_column_type(s, 7) != SQLITE_BLOB ||
|
||||
sqlite3_column_bytes(s, 7) != 32 || !fp || count <= 0 || count > 4096 ||
|
||||
stage < LARDON3D_SELECTED_EXECUTION_CALIBRATION || !task_kind ||
|
||||
strcmp((const char *)task_kind, LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND) != 0 ||
|
||||
sqlite3_column_int64(s, 11) != LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION) {
|
||||
result = LARDON3D_PROJECT_DB_CORRUPT;
|
||||
} else {
|
||||
p->task_id = task_id;
|
||||
p->selected_execution_id = (uint64_t)execution;
|
||||
p->next_item_index = (uint32_t)cursor;
|
||||
p->extractor_version = (uint32_t)version;
|
||||
p->max_features = (uint32_t)max;
|
||||
p->pyramid_levels = (uint32_t)levels;
|
||||
p->fast_threshold = (uint32_t)threshold;
|
||||
memcpy(p->parameter_fingerprint, fp, 32);
|
||||
}
|
||||
}
|
||||
sqlite3_finalize(s);
|
||||
}
|
||||
pthread_mutex_unlock(&db->mutex);
|
||||
return result;
|
||||
}
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_advance_feature_extract_batch_task(
|
||||
Lardon3DProjectDb *db, uint64_t task_id, uint32_t item_index,
|
||||
uint32_t next_item_index) {
|
||||
if (!db || !valid_task_id(task_id) || next_item_index != item_index + 1u)
|
||||
return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
|
||||
pthread_mutex_lock(&db->mutex);
|
||||
sqlite3_stmt *s = NULL;
|
||||
Lardon3DProjectDbResult result = execute(db, "BEGIN IMMEDIATE", "begin feature batch advance");
|
||||
if (result == LARDON3D_PROJECT_DB_OK)
|
||||
result = prepare(db,
|
||||
"UPDATE feature_extract_batch_tasks AS b SET next_item_index=?3 WHERE b.task_id=?1 AND "
|
||||
"(b.next_item_index=?2 OR b.next_item_index=?3) AND EXISTS(SELECT 1 FROM "
|
||||
"selected_execution_items i JOIN feature_sets f ON f.image_id=i.image_id JOIN "
|
||||
"feature_assets a ON a.feature_asset_id=f.feature_asset_id WHERE "
|
||||
"i.execution_id=b.selected_execution_id AND i.item_index=?2 AND i.image_id IS NOT NULL "
|
||||
"AND f.extractor_kind=b.extractor_kind AND f.extractor_version=b.extractor_version AND "
|
||||
"f.parameter_fingerprint=b.parameter_fingerprint AND a.durability=0)", &s);
|
||||
if (result == LARDON3D_PROJECT_DB_OK) {
|
||||
sqlite3_bind_int64(s, 1, (sqlite3_int64)task_id);
|
||||
sqlite3_bind_int64(s, 2, item_index);
|
||||
sqlite3_bind_int64(s, 3, next_item_index);
|
||||
result = step_done(db, s, "advance feature batch cursor");
|
||||
if (result == LARDON3D_PROJECT_DB_OK && sqlite3_changes(db->connection) != 1)
|
||||
result = LARDON3D_PROJECT_DB_CONSTRAINT;
|
||||
}
|
||||
if (result == LARDON3D_PROJECT_DB_OK) result = execute(db, "COMMIT", "commit feature batch advance");
|
||||
else (void)execute(db, "ROLLBACK", "rollback feature batch advance");
|
||||
pthread_mutex_unlock(&db->mutex);
|
||||
return result;
|
||||
}
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_sift_extract_task(
|
||||
Lardon3DProjectDb *db, uint64_t task_id, Lardon3DProjectDbSiftExtractTask *p) {
|
||||
if (!db || !valid_task_id(task_id) || !p) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
|
||||
|
|
@ -6965,7 +7269,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_visual_index_update_task(
|
|||
return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
|
||||
}
|
||||
return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0, NULL,
|
||||
NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, updated_at);
|
||||
NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
NULL, updated_at);
|
||||
}
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_visual_index_update_task(
|
||||
|
|
@ -7012,7 +7317,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_candidate_pair_generate_task(
|
|||
return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
|
||||
}
|
||||
return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0, NULL,
|
||||
NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, updated_at);
|
||||
NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
NULL, updated_at);
|
||||
}
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_candidate_pair_generate_task(
|
||||
|
|
@ -7068,7 +7374,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_matcher_task(
|
|||
return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
|
||||
}
|
||||
return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0, NULL,
|
||||
NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL, updated_at);
|
||||
NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
NULL, updated_at);
|
||||
}
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_matcher_task(
|
||||
|
|
@ -7126,7 +7433,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_geometric_verifier_task(
|
|||
return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
|
||||
}
|
||||
return record_task_internal(db, snapshot, kind, version, checkpoint, NULL, 0, NULL,
|
||||
NULL, NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL, updated_at);
|
||||
NULL, NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL, NULL,
|
||||
NULL, updated_at);
|
||||
}
|
||||
|
||||
Lardon3DProjectDbResult lardon3d_project_db_load_geometric_verifier_task(
|
||||
|
|
@ -7199,7 +7507,7 @@ Lardon3DProjectDbResult lardon3d_project_db_record_track_builder_task(
|
|||
}
|
||||
Lardon3DProjectDbResult result = record_task_internal(
|
||||
db, snapshot, kind, version, checkpoint, NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL,
|
||||
NULL, NULL, NULL, updated_at);
|
||||
NULL, NULL, NULL, NULL, NULL, updated_at);
|
||||
if (result != LARDON3D_PROJECT_DB_OK) return result;
|
||||
(void)pthread_mutex_lock(&db->mutex);
|
||||
result = execute(db, "BEGIN IMMEDIATE", "begin track builder task");
|
||||
|
|
@ -7310,7 +7618,8 @@ Lardon3DProjectDbResult lardon3d_project_db_record_sparse_sfm_task(
|
|||
const Lardon3DProjectDbSparseSfmTask *parameters, int64_t updated_at) {
|
||||
if (!parameters) return LARDON3D_PROJECT_DB_INVALID_ARGUMENT;
|
||||
return record_task_internal(database, snapshot, kind, version, checkpoint, NULL, 0, NULL, NULL,
|
||||
NULL, NULL, NULL, NULL, parameters, NULL, NULL, NULL, updated_at);
|
||||
NULL, NULL, NULL, NULL, parameters, NULL, NULL, NULL, NULL,
|
||||
NULL, updated_at);
|
||||
}
|
||||
|
||||
static bool read_u64_blob(sqlite3_stmt *statement, int column, uint64_t *value) {
|
||||
|
|
|
|||
|
|
@ -8,7 +8,10 @@ extern "C" {
|
|||
#include "opencv_task_thread_guard.h"
|
||||
|
||||
#include <cstdio>
|
||||
#include <algorithm>
|
||||
#include <array>
|
||||
#include <new>
|
||||
#include <pthread.h>
|
||||
|
||||
namespace {
|
||||
/* One admitted RAW execution owns a conservative allowance for the bounded
|
||||
|
|
@ -17,7 +20,12 @@ namespace {
|
|||
* scientific limit on the number of Captures. The Queue/Governor reservation
|
||||
* exists only for this one Task callback and is released by Task runtime on
|
||||
* every terminal path. */
|
||||
constexpr uint64_t kRawWorkingBytes = UINT64_C(2) * 1024u * 1024u * 1024u;
|
||||
constexpr uint64_t kRawSingleWorkingBytes = UINT64_C(2) * 1024u * 1024u * 1024u;
|
||||
constexpr size_t kRawBatchWindowMax = 8;
|
||||
constexpr size_t kRawBatchChildStackBytes = 1024u * 1024u;
|
||||
constexpr uint64_t kRawBatchParticipantBytes = UINT64_C(896) * 1024u * 1024u;
|
||||
static_assert(kRawBatchParticipantBytes <= UINT64_MAX / kRawBatchWindowMax,
|
||||
"RAW batch memory envelope must fit uint64_t");
|
||||
|
||||
struct Context {
|
||||
char project_path[LARDON3D_APP_STATE_PATH_CAPACITY]{};
|
||||
|
|
@ -27,6 +35,13 @@ struct Context {
|
|||
uint64_t source_asset_id{};
|
||||
};
|
||||
|
||||
struct BatchContext {
|
||||
char project_path[LARDON3D_APP_STATE_PATH_CAPACITY]{};
|
||||
Lardon3DProjectDb *database{};
|
||||
Lardon3DResourceGovernor *governor{};
|
||||
uint64_t selected_execution_id{};
|
||||
};
|
||||
|
||||
void destroy(void *value) { delete static_cast<Context *>(value); }
|
||||
|
||||
void runtime(const Context *context, Lardon3DAppState *state) {
|
||||
|
|
@ -129,6 +144,242 @@ Context *make_context(const Lardon3DTaskReconstructionContext *runtime_context,
|
|||
context->source_asset_id = source_asset_id;
|
||||
return context;
|
||||
}
|
||||
|
||||
void destroy_batch(void *value) { delete static_cast<BatchContext *>(value); }
|
||||
|
||||
BatchContext *make_batch_context(
|
||||
const Lardon3DTaskReconstructionContext *runtime_context,
|
||||
uint64_t selected_execution_id) {
|
||||
if (!runtime_context || !runtime_context->project_path ||
|
||||
!runtime_context->project_path[0] || !runtime_context->project_db ||
|
||||
!runtime_context->resource_governor || selected_execution_id == 0)
|
||||
return nullptr;
|
||||
auto *context = new (std::nothrow) BatchContext;
|
||||
if (!context) return nullptr;
|
||||
int written = std::snprintf(context->project_path, sizeof(context->project_path), "%s",
|
||||
runtime_context->project_path);
|
||||
if (written <= 0 || written >= static_cast<int>(sizeof(context->project_path))) {
|
||||
delete context;
|
||||
return nullptr;
|
||||
}
|
||||
context->database = runtime_context->project_db;
|
||||
context->governor = runtime_context->resource_governor;
|
||||
context->selected_execution_id = selected_execution_id;
|
||||
return context;
|
||||
}
|
||||
|
||||
void batch_runtime(const BatchContext *context, Lardon3DAppState *state) {
|
||||
lardon3d_app_state_init(state);
|
||||
state->project_loaded = true;
|
||||
state->project_db = context->database;
|
||||
state->resource_governor = context->governor;
|
||||
std::snprintf(state->project_path, sizeof(state->project_path), "%s",
|
||||
context->project_path);
|
||||
}
|
||||
|
||||
bool batch_checkpoint(BatchContext *context, Lardon3DTask *task) {
|
||||
Lardon3DAppState state;
|
||||
batch_runtime(context, &state);
|
||||
Lardon3DProjectDbRawDevelopmentBatchTask parameters{
|
||||
lardon3d_task_id(task), context->selected_execution_id};
|
||||
return lardon3d_project_checkpoint_raw_development_batch_task(
|
||||
&state, task, ¶meters) == LARDON3D_PROJECT_TASK_CHECKPOINT_OK;
|
||||
}
|
||||
|
||||
struct BatchWorker {
|
||||
BatchContext *context{};
|
||||
Lardon3DProjectDbSelectedExecutionItem item{};
|
||||
uint64_t producer_task_id{};
|
||||
Lardon3DRawDevelopmentResult raw_result{LARDON3D_RAW_DEVELOPMENT_INTERNAL_ERROR};
|
||||
Lardon3DProjectDbResult db_result{LARDON3D_PROJECT_DB_IO_ERROR};
|
||||
uint64_t image_id{};
|
||||
};
|
||||
|
||||
void *develop_batch_item(void *value) {
|
||||
auto *worker = static_cast<BatchWorker *>(value);
|
||||
if (worker->item.representation_source ==
|
||||
LARDON3D_SELECTED_REPRESENTATION_SOURCE_IMAGE) {
|
||||
worker->db_result = lardon3d_project_db_get_selected_capture_image(
|
||||
worker->context->database, worker->item.capture_id, &worker->image_id);
|
||||
return nullptr;
|
||||
}
|
||||
Lardon3DProjectDbCapture capture{};
|
||||
worker->db_result = lardon3d_project_db_load_capture(
|
||||
worker->context->database, worker->item.capture_id, &capture);
|
||||
if (worker->db_result != LARDON3D_PROJECT_DB_OK) return nullptr;
|
||||
Lardon3DAppState state;
|
||||
batch_runtime(worker->context, &state);
|
||||
Lardon3DRawDevelopmentOutput output{};
|
||||
worker->raw_result = lardon3d_raw_develop_asset_to_capture(
|
||||
&state, worker->item.capture_id, worker->item.source_asset_id,
|
||||
worker->producer_task_id, capture.created_at, &output);
|
||||
if (worker->raw_result == LARDON3D_RAW_DEVELOPMENT_OK) {
|
||||
worker->image_id = output.image.image_id;
|
||||
worker->db_result = LARDON3D_PROJECT_DB_OK;
|
||||
}
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
struct BatchParticipant {
|
||||
BatchWorker *workers{};
|
||||
size_t count{};
|
||||
size_t first{};
|
||||
size_t stride{};
|
||||
};
|
||||
|
||||
void *develop_batch_participant(void *value) {
|
||||
auto *participant = static_cast<BatchParticipant *>(value);
|
||||
for (size_t index = participant->first; index < participant->count;
|
||||
index += participant->stride)
|
||||
(void)develop_batch_item(&participant->workers[index]);
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
bool develop_batch_window(BatchContext *context, uint64_t producer_task_id,
|
||||
const Lardon3DProjectDbSelectedExecutionItem *items,
|
||||
size_t count, unsigned int admitted_threads,
|
||||
std::array<BatchWorker, kRawBatchWindowMax> *workers) {
|
||||
if (!context || !items || !workers || count == 0 || count > kRawBatchWindowMax ||
|
||||
admitted_threads == 0)
|
||||
return false;
|
||||
size_t participants = std::min(count, static_cast<size_t>(admitted_threads));
|
||||
std::array<pthread_t, kRawBatchWindowMax - 1> children{};
|
||||
std::array<BatchParticipant, kRawBatchWindowMax> participant_work{};
|
||||
pthread_attr_t attributes;
|
||||
if (pthread_attr_init(&attributes) != 0) return false;
|
||||
if (pthread_attr_setstacksize(&attributes, kRawBatchChildStackBytes) != 0) {
|
||||
(void)pthread_attr_destroy(&attributes);
|
||||
return false;
|
||||
}
|
||||
for (size_t index = 0; index < count; ++index) {
|
||||
(*workers)[index].context = context;
|
||||
(*workers)[index].item = items[index];
|
||||
(*workers)[index].producer_task_id = producer_task_id;
|
||||
}
|
||||
size_t created = 0;
|
||||
bool started = true;
|
||||
for (size_t participant = 1; participant < participants; ++participant) {
|
||||
participant_work[participant] = {
|
||||
workers->data(), count, participant, participants};
|
||||
/* Each participant processes one disjoint strided subset. No child owns
|
||||
* selected cursor/progress publication, even when it finishes first. */
|
||||
if (pthread_create(&children[created], &attributes,
|
||||
develop_batch_participant,
|
||||
&participant_work[participant]) != 0) {
|
||||
started = false;
|
||||
break;
|
||||
}
|
||||
++created;
|
||||
}
|
||||
if (pthread_attr_destroy(&attributes) != 0) started = false;
|
||||
participant_work[0] = {workers->data(), count, 0, participants};
|
||||
if (started) (void)develop_batch_participant(&participant_work[0]);
|
||||
for (size_t index = 0; index < created; ++index) {
|
||||
if (pthread_join(children[index], nullptr) != 0) started = false;
|
||||
}
|
||||
return started;
|
||||
}
|
||||
|
||||
bool run_batch_impl(Lardon3DTask *task, BatchContext *context) {
|
||||
for (;;) {
|
||||
if (!lardon3d_task_checkpoint(task)) return false;
|
||||
Lardon3DProjectDbSelectedExecution execution{};
|
||||
if (lardon3d_project_db_load_selected_execution(
|
||||
context->database, context->selected_execution_id, &execution) !=
|
||||
LARDON3D_PROJECT_DB_OK)
|
||||
return lardon3d_task_fail(task, "Exécution sélectionnée RAW batch invalide.");
|
||||
if (execution.next_item_index == execution.item_count) {
|
||||
return lardon3d_task_set_durable_progress(
|
||||
task, execution.item_count, execution.item_count,
|
||||
"Développement RAW batch terminé.") &&
|
||||
batch_checkpoint(context, task);
|
||||
}
|
||||
if (execution.stage != LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS)
|
||||
return lardon3d_task_fail(task, "Étape sélectionnée RAW batch invalide.");
|
||||
|
||||
Lardon3DTaskExecutionContract contract{};
|
||||
if (!lardon3d_task_execution_contract(task, &contract) ||
|
||||
contract.batch_size == 0 || contract.batch_size > kRawBatchWindowMax ||
|
||||
contract.cpu_threads == 0)
|
||||
return lardon3d_task_fail(task, "Contrat Governor RAW batch invalide.");
|
||||
size_t count = std::min<size_t>(contract.batch_size,
|
||||
execution.item_count - execution.next_item_index);
|
||||
std::array<Lardon3DProjectDbSelectedExecutionItem, kRawBatchWindowMax> items{};
|
||||
for (size_t offset = 0; offset < count; ++offset) {
|
||||
if (lardon3d_project_db_load_selected_execution_item(
|
||||
context->database, context->selected_execution_id,
|
||||
execution.next_item_index + static_cast<uint32_t>(offset),
|
||||
&items[offset]) != LARDON3D_PROJECT_DB_OK)
|
||||
return lardon3d_task_fail(task, "Item sélectionné RAW batch invalide.");
|
||||
}
|
||||
|
||||
std::array<BatchWorker, kRawBatchWindowMax> workers{};
|
||||
/* Each independent LibRaw/BGR/PNG participant fits the admitted 896 MiB
|
||||
* per-item envelope. Every child is joined before cursor publication or
|
||||
* reservation release. */
|
||||
if (!develop_batch_window(context, lardon3d_task_id(task), items.data(), count,
|
||||
contract.cpu_threads, &workers))
|
||||
return lardon3d_task_fail(task, "Workers RAW batch impossibles.");
|
||||
if (!lardon3d_task_checkpoint(task)) return false;
|
||||
|
||||
for (size_t offset = 0; offset < count; ++offset) {
|
||||
BatchWorker &worker = workers[offset];
|
||||
if (worker.db_result != LARDON3D_PROJECT_DB_OK || worker.image_id == 0 ||
|
||||
(worker.item.representation_source ==
|
||||
LARDON3D_SELECTED_REPRESENTATION_RAW_ASSET &&
|
||||
worker.raw_result != LARDON3D_RAW_DEVELOPMENT_OK))
|
||||
return lardon3d_task_fail(task, "Développement d'un item RAW batch impossible.");
|
||||
uint32_t item_index = worker.item.item_index;
|
||||
if (lardon3d_project_db_record_selected_representation(
|
||||
context->database, context->selected_execution_id, item_index,
|
||||
worker.image_id, item_index + 1u) != LARDON3D_PROJECT_DB_OK)
|
||||
return lardon3d_task_fail(task, "Publication ordonnée RAW batch impossible.");
|
||||
/* The selected cursor commit is the authority. Generic durable progress
|
||||
* follows it and may lag after a crash, but must never lead it. */
|
||||
if (!lardon3d_task_set_durable_progress(
|
||||
task, item_index + 1u, execution.item_count,
|
||||
"Représentation RAW batch publiée.") ||
|
||||
!batch_checkpoint(context, task))
|
||||
return lardon3d_task_fail(task, "Checkpoint RAW batch impossible.");
|
||||
}
|
||||
if (execution.next_item_index + count == execution.item_count)
|
||||
return true;
|
||||
Lardon3DResourceReservation *reservation = nullptr;
|
||||
if (!lardon3d_task_sequence_break(task, context->governor, &reservation,
|
||||
&contract))
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
bool run_batch(Lardon3DTask *task, void *value) noexcept {
|
||||
try {
|
||||
Lardon3DOpenCvTaskThreadGuard threads(task);
|
||||
if (!threads.valid())
|
||||
return lardon3d_task_fail(task, "Contrat CPU OpenCV RAW batch invalide.");
|
||||
/* External Task participants are the admitted CPU dimension. OpenCV stays
|
||||
* single-threaded so imencode/imdecode cannot create a nested fan-out. */
|
||||
cv::setNumThreads(1);
|
||||
if (cv::getNumThreads() != 1)
|
||||
return lardon3d_task_fail(task, "Configuration OpenCV RAW batch impossible.");
|
||||
bool result = run_batch_impl(task, static_cast<BatchContext *>(value));
|
||||
if (!threads.restore())
|
||||
return lardon3d_task_fail(task, "Restauration OpenCV RAW batch impossible.");
|
||||
return result;
|
||||
} catch (const std::bad_alloc &) {
|
||||
return lardon3d_task_fail(task, "Mémoire insuffisante pour le RAW batch.");
|
||||
} catch (...) {
|
||||
return lardon3d_task_fail(task, "Erreur interne du RAW batch.");
|
||||
}
|
||||
}
|
||||
|
||||
void finished_batch(const Lardon3DTask *task, void *value) noexcept {
|
||||
try {
|
||||
Lardon3DAppState state;
|
||||
batch_runtime(static_cast<BatchContext *>(value), &state);
|
||||
(void)lardon3d_project_checkpoint_task(&state, task);
|
||||
} catch (...) {
|
||||
}
|
||||
}
|
||||
} // namespace
|
||||
|
||||
extern "C" bool lardon3d_raw_development_task_reconstruct(
|
||||
|
|
@ -173,7 +424,7 @@ extern "C" Lardon3DTask *lardon3d_project_create_raw_development_task(
|
|||
delete context;
|
||||
return nullptr;
|
||||
}
|
||||
Lardon3DResourceEstimate estimate{kRawWorkingBytes + sizeof(Context), 0, 0, 0,
|
||||
Lardon3DResourceEstimate estimate{kRawSingleWorkingBytes + sizeof(Context), 0, 0, 0,
|
||||
1, 1, 1, 0, 1, LARDON3D_RESOURCE_TASK_MIXED};
|
||||
Lardon3DTask *task = lardon3d_task_create_typed(
|
||||
"Développement RAW S3-B1", &estimate, LARDON3D_RAW_DEVELOPMENT_TASK_KIND,
|
||||
|
|
@ -208,3 +459,104 @@ extern "C" bool lardon3d_project_enqueue_raw_development(
|
|||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
extern "C" bool lardon3d_raw_development_batch_task_reconstruct(
|
||||
const Lardon3DTaskDurableSnapshot *snapshot, void *value,
|
||||
Lardon3DTaskKindBinding *binding) {
|
||||
try {
|
||||
auto *runtime_context = static_cast<Lardon3DTaskReconstructionContext *>(value);
|
||||
if (!snapshot || !runtime_context || !binding) return false;
|
||||
Lardon3DProjectDbRawDevelopmentBatchTask persisted{};
|
||||
if (lardon3d_project_db_load_raw_development_batch_task(
|
||||
runtime_context->project_db, snapshot->id, &persisted) !=
|
||||
LARDON3D_PROJECT_DB_OK)
|
||||
return false;
|
||||
BatchContext *context = make_batch_context(
|
||||
runtime_context, persisted.selected_execution_id);
|
||||
if (!context) return false;
|
||||
*binding = {};
|
||||
binding->callback = run_batch;
|
||||
binding->userdata = context;
|
||||
binding->userdata_destroy = destroy_batch;
|
||||
binding->finished_callback = finished_batch;
|
||||
binding->finished_userdata = context;
|
||||
return true;
|
||||
} catch (...) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
extern "C" Lardon3DTask *lardon3d_project_create_raw_development_batch_task(
|
||||
Lardon3DAppState *state, uint64_t selected_execution_id, uint64_t *task_id) {
|
||||
if (task_id) *task_id = 0;
|
||||
try {
|
||||
if (!state || !state->project_loaded || !state->project_db ||
|
||||
!state->resource_governor || !task_id || selected_execution_id == 0)
|
||||
return nullptr;
|
||||
Lardon3DProjectDbSelectedExecution execution{};
|
||||
if (lardon3d_project_db_load_selected_execution(
|
||||
state->project_db, selected_execution_id, &execution) !=
|
||||
LARDON3D_PROJECT_DB_OK ||
|
||||
execution.stage != LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS)
|
||||
return nullptr;
|
||||
Lardon3DTaskReconstructionContext runtime_context{
|
||||
state->project_path, state->project_db, state->resource_governor, nullptr};
|
||||
BatchContext *context = make_batch_context(&runtime_context,
|
||||
selected_execution_id);
|
||||
if (!context) return nullptr;
|
||||
uint64_t id = 0;
|
||||
if (lardon3d_project_db_allocate_task_id(state->project_db, &id) !=
|
||||
LARDON3D_PROJECT_DB_OK) {
|
||||
delete context;
|
||||
return nullptr;
|
||||
}
|
||||
/* At the frozen 40 MP decoder ceiling, one participant owns LibRaw's
|
||||
* mosaic/16-bit four-channel workspace, the processed RGB/BGR copies, and
|
||||
* the later PNG/validation buffers. Their lifetime analysis plus the 1 MiB
|
||||
* child stack and allocator/codec headroom is bounded by 896 MiB. Charge
|
||||
* the tiny owner context through that per-item allowance too: a 7 GiB
|
||||
* post-reserve host budget can then admit all eight independently bounded
|
||||
* participants instead of losing one to a redundant fixed charge.
|
||||
*
|
||||
* The 1..8 capacity is operational admission only. Immutable selected
|
||||
* execution and item order remain the scientific, durable input, never a
|
||||
* dataset-size limit. */
|
||||
Lardon3DResourceEstimate estimate{
|
||||
0, 0, kRawBatchParticipantBytes, 0,
|
||||
1, kRawBatchWindowMax, static_cast<unsigned int>(kRawBatchWindowMax),
|
||||
0, 1, LARDON3D_RESOURCE_TASK_MIXED};
|
||||
Lardon3DTask *task = lardon3d_task_create_typed(
|
||||
"Développement RAW sélectionné", &estimate,
|
||||
LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND,
|
||||
LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION, run_batch, context,
|
||||
destroy_batch);
|
||||
if (!task || !lardon3d_task_assign_id(task, id) ||
|
||||
!lardon3d_task_set_finished_callback(task, finished_batch, context) ||
|
||||
!batch_checkpoint(context, task)) {
|
||||
lardon3d_task_destroy(task);
|
||||
return nullptr;
|
||||
}
|
||||
*task_id = id;
|
||||
return task;
|
||||
} catch (...) {
|
||||
return nullptr;
|
||||
}
|
||||
}
|
||||
|
||||
extern "C" bool lardon3d_project_enqueue_raw_development_batch(
|
||||
Lardon3DAppState *state, uint64_t selected_execution_id,
|
||||
uint64_t *task_id) {
|
||||
try {
|
||||
if (!state || !state->task_queue) return false;
|
||||
Lardon3DTask *task = lardon3d_project_create_raw_development_batch_task(
|
||||
state, selected_execution_id, task_id);
|
||||
if (!task) return false;
|
||||
if (!lardon3d_task_queue_add(state->task_queue, task, nullptr)) {
|
||||
lardon3d_task_destroy(task);
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
} catch (...) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1644,7 +1644,10 @@ open_next_capability_trial_locked(
|
|||
&& feedback->accepted_cpu_limit < cpu_max) {
|
||||
feedback->adaptive_cpu_limit = next_trial_cpu(
|
||||
feedback->accepted_cpu_limit, cpu_max);
|
||||
feedback->adaptive_batch_limit = feedback->accepted_batch_limit;
|
||||
feedback->adaptive_batch_limit = capability->cpu_batch_coupled
|
||||
? minimum_size(feedback->adaptive_cpu_limit,
|
||||
capability->estimate.maximum_batch_size)
|
||||
: feedback->accepted_batch_limit;
|
||||
feedback->adaptive_inflight_limit =
|
||||
feedback->accepted_inflight_limit;
|
||||
feedback->trial_dimension = LARDON3D_CAPABILITY_TRIAL_CPU;
|
||||
|
|
@ -4292,6 +4295,9 @@ lardon3d_resource_governor_internal_record_sequence_execution_metrics(
|
|||
== LARDON3D_CAPABILITY_TRIAL_CPU
|
||||
? selection->decision.cpu_threads
|
||||
== feedback->adaptive_cpu_limit
|
||||
&& (!selection->capability.cpu_batch_coupled
|
||||
|| selection->decision.batch_size
|
||||
== feedback->adaptive_batch_limit)
|
||||
: feedback->trial_dimension
|
||||
== LARDON3D_CAPABILITY_TRIAL_INFLIGHT
|
||||
? selection->inflight_limit
|
||||
|
|
@ -4350,6 +4356,13 @@ lardon3d_resource_governor_internal_record_sequence_execution_metrics(
|
|||
== LARDON3D_CAPABILITY_TRIAL_CPU) {
|
||||
feedback->accepted_cpu_limit =
|
||||
feedback->adaptive_cpu_limit;
|
||||
if (selection->capability.cpu_batch_coupled) {
|
||||
/* The measured gain belongs to the complete
|
||||
* cross-image rung, not to an unusable CPU
|
||||
* count detached from its participant window. */
|
||||
feedback->accepted_batch_limit =
|
||||
feedback->adaptive_batch_limit;
|
||||
}
|
||||
} else if (completed_dimension
|
||||
== LARDON3D_CAPABILITY_TRIAL_INFLIGHT) {
|
||||
feedback->accepted_inflight_limit =
|
||||
|
|
|
|||
|
|
@ -44,6 +44,11 @@ typedef struct {
|
|||
bool preferred;
|
||||
bool cpu_reducible;
|
||||
bool batch_adaptive;
|
||||
/* Cross-item CPU callbacks cannot exercise a CPU trial unless the same
|
||||
* immutable sequence admits at least that many independent items. The
|
||||
* Governor therefore advances and accepts these two operational limits as
|
||||
* one rung; this property is private and never changes durable science. */
|
||||
bool cpu_batch_coupled;
|
||||
/* Current ORB Vulkan batch trials use a longer observation window than
|
||||
* generic CPU adaptation. This private operational property is
|
||||
* reconstructed with the capability and is never durable/scientific. */
|
||||
|
|
|
|||
25
src/task.c
25
src/task.c
|
|
@ -112,6 +112,7 @@ kind_has_validated_cpu_range(const char *task_kind, uint32_t task_kind_version)
|
|||
return false;
|
||||
}
|
||||
return strcmp(task_kind, "features.extract") == 0
|
||||
|| strcmp(task_kind, "features.extract.batch") == 0
|
||||
|| strcmp(task_kind, "features.extract.sift") == 0
|
||||
|| strcmp(task_kind, "features.extract.rootsift") == 0
|
||||
|| strcmp(task_kind, "visual_index.update") == 0
|
||||
|
|
@ -127,7 +128,20 @@ kind_has_validated_batch_range(const char *task_kind, uint32_t version)
|
|||
* one-item batch cannot exercise a second participant; Gate G may still
|
||||
* reduce that bounded window for capacity or pressure. */
|
||||
return task_kind && version == 1
|
||||
&& strcmp(task_kind, "candidate_pair.generate") == 0;
|
||||
&& (strcmp(task_kind, "candidate_pair.generate") == 0
|
||||
|| strcmp(task_kind, "features.extract.batch") == 0);
|
||||
}
|
||||
|
||||
static bool
|
||||
kind_has_coupled_cpu_batch_range(const char *task_kind, uint32_t version)
|
||||
{
|
||||
/* Feature batch and Candidate obtain CPU parallelism only across
|
||||
* independent items. A CPU2/batch1 trial is structurally incapable of
|
||||
* measuring either path, so Governor feedback must trial the complete
|
||||
* participant rung without changing scientific ordering or publication. */
|
||||
return task_kind && version == 1
|
||||
&& (strcmp(task_kind, "features.extract.batch") == 0
|
||||
|| strcmp(task_kind, "candidate_pair.generate") == 0);
|
||||
}
|
||||
|
||||
static void
|
||||
|
|
@ -244,6 +258,9 @@ lardon3d_task_create_typed(
|
|||
&& kind_has_validated_cpu_range(task_kind, task_kind_version),
|
||||
.batch_adaptive = typed
|
||||
&& kind_has_validated_batch_range(task_kind, task_kind_version),
|
||||
.cpu_batch_coupled = typed
|
||||
&& kind_has_coupled_cpu_batch_range(
|
||||
task_kind, task_kind_version),
|
||||
}},
|
||||
};
|
||||
copy_text(task->message, sizeof(task->message), "En attente.");
|
||||
|
|
@ -276,6 +293,9 @@ lardon3d_task_internal_set_capability_envelope(
|
|||
&& (!capability->batch_adaptive
|
||||
|| capability->backend
|
||||
!= LARDON3D_RESOURCE_BACKEND_ORB_VULKAN))
|
||||
|| (capability->cpu_batch_coupled
|
||||
&& (!capability->cpu_reducible
|
||||
|| !capability->batch_adaptive))
|
||||
|| (capability->inflight_adaptive
|
||||
&& (capability->minimum_inflight_limit == 0
|
||||
|| capability->gpu_memory_bytes_per_inflight == 0))
|
||||
|
|
@ -317,6 +337,9 @@ lardon3d_task_internal_enable_known_capabilities(Lardon3DTask *task)
|
|||
task->capability_envelope.capabilities[0].batch_adaptive =
|
||||
kind_has_validated_batch_range(
|
||||
task->task_kind, task->task_kind_version);
|
||||
task->capability_envelope.capabilities[0].cpu_batch_coupled =
|
||||
kind_has_coupled_cpu_batch_range(
|
||||
task->task_kind, task->task_kind_version);
|
||||
}
|
||||
(void)pthread_mutex_unlock(&task->mutex);
|
||||
return true;
|
||||
|
|
|
|||
|
|
@ -38,6 +38,13 @@ enum {
|
|||
VISUAL_INDEX_PER_ITEM_BYTES = 2 * 1024 * 1024,
|
||||
};
|
||||
|
||||
#define RAW_BATCH_LEGACY_PER_ITEM_BYTES \
|
||||
((UINT64_C(2) * 1024u * 1024u * 1024u) + (UINT64_C(1) * 1024u * 1024u))
|
||||
#define RAW_BATCH_CURRENT_PER_ITEM_BYTES (UINT64_C(896) * 1024u * 1024u)
|
||||
/* The first v24 implementation persisted sizeof(BatchContext) as fixed
|
||||
* memory. Its public checkpoint ABI made that 4120-byte exact shape durable. */
|
||||
#define RAW_BATCH_LEGACY_FIXED_BYTES UINT64_C(4120)
|
||||
|
||||
static bool
|
||||
estimate_equals(const Lardon3DResourceEstimate *left,
|
||||
const Lardon3DResourceEstimate *right)
|
||||
|
|
@ -182,6 +189,23 @@ normalize_known_legacy_estimate(const char *kind,
|
|||
/* CPU/batch/memory admission is operational, not GVR identity. Accept
|
||||
* only the complete frozen serial envelope, normalize it in memory,
|
||||
* and never publish an estimate-only checkpoint during recovery. */
|
||||
} else if (strcmp(kind, "raw.develop.batch") == 0) {
|
||||
current = (Lardon3DResourceEstimate) {
|
||||
.memory_bytes_per_item = RAW_BATCH_CURRENT_PER_ITEM_BYTES,
|
||||
.minimum_batch_size = 1,
|
||||
.maximum_batch_size = 8,
|
||||
.desired_cpu_threads = 8,
|
||||
.desired_io_slots = 1,
|
||||
.task_class = LARDON3D_RESOURCE_TASK_MIXED,
|
||||
};
|
||||
historical = current;
|
||||
historical.memory_fixed_bytes = RAW_BATCH_LEGACY_FIXED_BYTES;
|
||||
historical.memory_bytes_per_item = RAW_BATCH_LEGACY_PER_ITEM_BYTES;
|
||||
/* v24 originally persisted this exact 4120-byte owner plus 2 GiB and
|
||||
* 1 MiB stack participant allowance. It is Governor accounting, not
|
||||
* RAW identity or the selected-execution payload. Accept only this
|
||||
* complete legacy shape and normalize it in memory, so recovery can
|
||||
* reach the corrected safe window without rewriting a checkpoint. */
|
||||
} else {
|
||||
*effective = *durable;
|
||||
return true;
|
||||
|
|
|
|||
|
|
@ -33,6 +33,11 @@ const Lardon3DTaskKindRegistry *lardon3d_task_kind_registry_production(void) {
|
|||
.kind_version = LARDON3D_RAW_DEVELOPMENT_TASK_KIND_VERSION,
|
||||
.reconstruct = lardon3d_raw_development_task_reconstruct,
|
||||
},
|
||||
{
|
||||
.kind = LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND,
|
||||
.kind_version = LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION,
|
||||
.reconstruct = lardon3d_raw_development_batch_task_reconstruct,
|
||||
},
|
||||
#endif
|
||||
#ifdef LARDON3D_PHOTO_QUALITY_TASK_AVAILABLE
|
||||
{
|
||||
|
|
@ -58,6 +63,11 @@ const Lardon3DTaskKindRegistry *lardon3d_task_kind_registry_production(void) {
|
|||
.kind_version = LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION,
|
||||
.reconstruct = lardon3d_feature_extract_reconstruct,
|
||||
},
|
||||
{
|
||||
.kind = LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND,
|
||||
.kind_version = LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION,
|
||||
.reconstruct = lardon3d_feature_extract_batch_reconstruct,
|
||||
},
|
||||
{
|
||||
.kind = LARDON3D_SIFT_EXTRACT_TASK_KIND,
|
||||
.kind_version = LARDON3D_SIFT_EXTRACT_TASK_KIND_VERSION,
|
||||
|
|
|
|||
|
|
@ -167,11 +167,13 @@ kind_to_stage(
|
|||
if (strcmp(kind, "import.images") == 0
|
||||
|| strcmp(kind, "acquisition_campaign.run") == 0) {
|
||||
*stage = LARDON3D_TUI_STAGE_ACQUISITION;
|
||||
} else if (strcmp(kind, "raw.develop") == 0) {
|
||||
} else if (strcmp(kind, "raw.develop") == 0 ||
|
||||
strcmp(kind, "raw.develop.batch") == 0) {
|
||||
*stage = LARDON3D_TUI_STAGE_RAW;
|
||||
} else if (strcmp(kind, "photo_quality.triage") == 0) {
|
||||
*stage = LARDON3D_TUI_STAGE_QUALITY;
|
||||
} else if (strcmp(kind, "features.extract") == 0
|
||||
|| strcmp(kind, "features.extract.batch") == 0
|
||||
|| strcmp(kind, "features.extract.sift") == 0
|
||||
|| strcmp(kind, "features.extract.rootsift") == 0) {
|
||||
*stage = LARDON3D_TUI_STAGE_FEATURES;
|
||||
|
|
|
|||
|
|
@ -53,6 +53,10 @@ struct Options {
|
|||
bool resume_geometry_existing{};
|
||||
bool resume_pre_gv_existing{};
|
||||
bool resume_candidate_existing{};
|
||||
bool resume_representations_existing{};
|
||||
uint64_t selected_execution_id{};
|
||||
bool has_selected_execution_id{};
|
||||
bool stop_after_representations{};
|
||||
unsigned int cpu_budget{};
|
||||
unsigned int gpu_budget{};
|
||||
bool has_gpu_budget{};
|
||||
|
|
@ -158,13 +162,30 @@ void usage(const char *program) {
|
|||
"[--matcher-batch 2|4|8|12] "
|
||||
"[--stop-after-matcher]\n"
|
||||
" %s --resume-candidate-existing --project-dir "
|
||||
"ABSOLUTE_EXISTING_DIR [--cpu-budget 1..12] [--gpu-budget 0..1]\n",
|
||||
"ABSOLUTE_EXISTING_DIR [--cpu-budget 1..12] [--gpu-budget 0..1]\n"
|
||||
" %s --resume-representations-existing --project-dir "
|
||||
"ABSOLUTE_EXISTING_DIR --selected-execution-id ID "
|
||||
"[--stop-after-representations]\n",
|
||||
program,
|
||||
program,
|
||||
program,
|
||||
program,
|
||||
program);
|
||||
}
|
||||
|
||||
bool parse_u64_id(const char *text, uint64_t &value) {
|
||||
if (!text || !*text || *text == '0') return false;
|
||||
for (const char *character = text; *character; ++character) {
|
||||
if (*character < '0' || *character > '9') return false;
|
||||
}
|
||||
char *end = nullptr;
|
||||
errno = 0;
|
||||
const unsigned long long parsed = std::strtoull(text, &end, 10);
|
||||
if (errno != 0 || end == text || *end != '\0' || parsed == 0) return false;
|
||||
value = static_cast<uint64_t>(parsed);
|
||||
return true;
|
||||
}
|
||||
|
||||
bool parse_size(const char *text, size_t &value) {
|
||||
char *end = nullptr;
|
||||
errno = 0;
|
||||
|
|
@ -228,11 +249,16 @@ bool parse_options(int argc, char **argv, Options &options) {
|
|||
options.resume_candidate_existing = true;
|
||||
continue;
|
||||
}
|
||||
if (argument == "--resume-representations-existing") {
|
||||
options.resume_representations_existing = true;
|
||||
continue;
|
||||
}
|
||||
if ((argument == "--mode" || argument == "--project-dir" || argument == "--root" ||
|
||||
argument == "--limit" || argument == "--restart-boundary" ||
|
||||
argument == "--cpu-budget" || argument == "--gpu-budget" ||
|
||||
argument == "--matcher-mode" || argument == "--matcher-pipeline" ||
|
||||
argument == "--matcher-inflight" || argument == "--matcher-batch") &&
|
||||
argument == "--matcher-inflight" || argument == "--matcher-batch" ||
|
||||
argument == "--selected-execution-id") &&
|
||||
index + 1 >= argc)
|
||||
return false;
|
||||
if (argument == "--mode") {
|
||||
|
|
@ -280,40 +306,59 @@ bool parse_options(int argc, char **argv, Options &options) {
|
|||
if (!parse_matcher_batch(argv[++index], options.matcher_batch_override))
|
||||
return false;
|
||||
options.has_matcher_batch_override = true;
|
||||
} else if (argument == "--selected-execution-id") {
|
||||
if (!parse_u64_id(argv[++index], options.selected_execution_id)) return false;
|
||||
options.has_selected_execution_id = true;
|
||||
} else if (argument == "--stop-after-matcher") {
|
||||
options.stop_after_matcher = true;
|
||||
} else if (argument == "--stop-after-gv") {
|
||||
options.stop_after_gv = true;
|
||||
} else if (argument == "--stop-after-representations") {
|
||||
options.stop_after_representations = true;
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
const unsigned int resume_mode_count = options.resume_geometry_existing +
|
||||
options.resume_pre_gv_existing +
|
||||
options.resume_candidate_existing;
|
||||
options.resume_candidate_existing +
|
||||
options.resume_representations_existing;
|
||||
if (resume_mode_count != 0) {
|
||||
if (options.has_mode || options.project_dir.empty() || !options.roots.empty() ||
|
||||
options.restart != RestartBoundary::kNone || resume_mode_count != 1)
|
||||
return false;
|
||||
if (options.resume_pre_gv_existing) {
|
||||
return !options.stop_after_gv;
|
||||
return !options.stop_after_gv && !options.has_selected_execution_id &&
|
||||
!options.stop_after_representations;
|
||||
}
|
||||
if (options.resume_candidate_existing)
|
||||
return !options.has_matcher_mode && !options.has_matcher_pipeline &&
|
||||
!options.has_matcher_inflight_override &&
|
||||
!options.has_matcher_batch_override &&
|
||||
!options.stop_after_matcher && !options.stop_after_gv;
|
||||
!options.stop_after_matcher && !options.stop_after_gv &&
|
||||
!options.has_selected_execution_id &&
|
||||
!options.stop_after_representations;
|
||||
if (options.resume_representations_existing)
|
||||
return options.has_selected_execution_id && options.cpu_budget == 0 &&
|
||||
!options.has_gpu_budget && !options.has_matcher_mode &&
|
||||
!options.has_matcher_pipeline &&
|
||||
!options.has_matcher_inflight_override &&
|
||||
!options.has_matcher_batch_override && !options.stop_after_matcher &&
|
||||
!options.stop_after_gv;
|
||||
return options.cpu_budget == 0 && !options.has_gpu_budget &&
|
||||
!options.has_matcher_mode && !options.has_matcher_pipeline &&
|
||||
!options.has_matcher_inflight_override &&
|
||||
!options.has_matcher_batch_override &&
|
||||
!options.stop_after_matcher;
|
||||
!options.stop_after_matcher && !options.has_selected_execution_id &&
|
||||
!options.stop_after_representations;
|
||||
}
|
||||
if (options.cpu_budget != 0 || options.has_gpu_budget ||
|
||||
options.has_matcher_mode || options.has_matcher_pipeline ||
|
||||
options.has_matcher_inflight_override ||
|
||||
options.has_matcher_batch_override ||
|
||||
options.stop_after_matcher || options.stop_after_gv) return false;
|
||||
options.stop_after_matcher || options.stop_after_gv ||
|
||||
options.has_selected_execution_id || options.stop_after_representations)
|
||||
return false;
|
||||
return options.has_mode && !options.project_dir.empty() && !options.roots.empty() &&
|
||||
options.roots.size() <= LARDON3D_ACQUISITION_CAMPAIGN_MAX_ROOTS;
|
||||
}
|
||||
|
|
@ -1440,46 +1485,233 @@ bool create_selection(Runtime &runtime, Mode mode, uint64_t quality_task_id,
|
|||
items.size(), std::time(nullptr), &execution) == LARDON3D_PROJECT_DB_OK;
|
||||
}
|
||||
|
||||
enum class RepresentationPendingKind { kNone, kLegacySerial, kBatch };
|
||||
enum class RepresentationAction {
|
||||
kReject,
|
||||
kCollect,
|
||||
kRecoverLegacySerial,
|
||||
kRecoverBatch,
|
||||
kEnqueueBatch,
|
||||
};
|
||||
|
||||
RepresentationAction choose_representation_action(
|
||||
const Lardon3DProjectDbSelectedExecution &execution,
|
||||
RepresentationPendingKind pending) {
|
||||
if (execution.next_item_index > execution.item_count) return RepresentationAction::kReject;
|
||||
if (execution.stage == LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS &&
|
||||
execution.next_item_index < execution.item_count) {
|
||||
if (pending == RepresentationPendingKind::kLegacySerial)
|
||||
return RepresentationAction::kRecoverLegacySerial;
|
||||
if (pending == RepresentationPendingKind::kBatch)
|
||||
return RepresentationAction::kRecoverBatch;
|
||||
return RepresentationAction::kEnqueueBatch;
|
||||
}
|
||||
if (execution.stage == LARDON3D_SELECTED_EXECUTION_CALIBRATION &&
|
||||
execution.next_item_index == execution.item_count) {
|
||||
return pending == RepresentationPendingKind::kBatch
|
||||
? RepresentationAction::kRecoverBatch
|
||||
: pending == RepresentationPendingKind::kNone
|
||||
? RepresentationAction::kCollect
|
||||
: RepresentationAction::kReject;
|
||||
}
|
||||
return RepresentationAction::kReject;
|
||||
}
|
||||
|
||||
bool representation_defers_feature_task(
|
||||
const Lardon3DProjectDbSelectedExecution &execution,
|
||||
const Lardon3DProjectRecoveryEntry &entry) {
|
||||
if (execution.stage != LARDON3D_SELECTED_EXECUTION_CALIBRATION ||
|
||||
execution.next_item_index != execution.item_count)
|
||||
return false;
|
||||
return (std::strcmp(entry.task_kind, LARDON3D_FEATURE_EXTRACT_TASK_KIND) == 0 &&
|
||||
entry.task_kind_version == LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION) ||
|
||||
(std::strcmp(entry.task_kind,
|
||||
LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND) == 0 &&
|
||||
entry.task_kind_version ==
|
||||
LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION);
|
||||
}
|
||||
|
||||
bool find_pending_representation_task(
|
||||
Runtime &runtime, const Lardon3DProjectDbSelectedExecution &execution,
|
||||
RepresentationPendingKind &kind, uint64_t &task_id) {
|
||||
kind = RepresentationPendingKind::kNone;
|
||||
task_id = 0;
|
||||
uint64_t cursor = 0;
|
||||
for (;;) {
|
||||
Lardon3DProjectRecoveryEntry entries[8]{};
|
||||
size_t count = 0;
|
||||
if (lardon3d_project_list_recoverable(
|
||||
&runtime.state, lardon3d_task_kind_registry_production(), cursor,
|
||||
entries, 8, &count) != LARDON3D_PROJECT_DB_OK)
|
||||
return false;
|
||||
for (size_t index = 0; index < count; ++index) {
|
||||
const auto &entry = entries[index];
|
||||
cursor = entry.task_id;
|
||||
if ((entry.status != LARDON3D_PROJECT_RECOVERABLE &&
|
||||
entry.status != LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE) ||
|
||||
entry.snapshot.recovery_state != TASK_PENDING) {
|
||||
std::fprintf(stderr,
|
||||
"representation recovery refuses unsafe pending task %llu of kind %s\n",
|
||||
static_cast<unsigned long long>(entry.task_id), entry.task_kind);
|
||||
return false;
|
||||
}
|
||||
/* Feature recovery owns the phase after representation collection. A
|
||||
* precise Feature kind/version may already be queued by project-open,
|
||||
* but representation recovery must neither resume nor reject it once the
|
||||
* selected representation prefix is durably complete. extract_features
|
||||
* validates its ORB fingerprint and legacy/batch ownership before wait.
|
||||
* Every other pending kind remains fail-closed here. */
|
||||
if (representation_defers_feature_task(execution, entry)) continue;
|
||||
if (task_id != 0) {
|
||||
std::fprintf(stderr,
|
||||
"representation recovery refuses multiple pending owners"
|
||||
" including task %llu of kind %s\n",
|
||||
static_cast<unsigned long long>(entry.task_id),
|
||||
entry.task_kind);
|
||||
return false;
|
||||
}
|
||||
if (std::strcmp(entry.task_kind, LARDON3D_RAW_DEVELOPMENT_TASK_KIND) == 0 &&
|
||||
entry.task_kind_version == LARDON3D_RAW_DEVELOPMENT_TASK_KIND_VERSION &&
|
||||
execution.stage == LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS &&
|
||||
execution.next_item_index < execution.item_count) {
|
||||
Lardon3DProjectDbSelectedExecutionItem item{};
|
||||
Lardon3DProjectDbRawDevelopmentTask raw{};
|
||||
if (lardon3d_project_db_load_selected_execution_item(
|
||||
runtime.state.project_db, execution.execution_id,
|
||||
execution.next_item_index, &item) != LARDON3D_PROJECT_DB_OK ||
|
||||
item.representation_source != LARDON3D_SELECTED_REPRESENTATION_RAW_ASSET ||
|
||||
lardon3d_project_db_load_raw_development_task(
|
||||
runtime.state.project_db, entry.task_id, &raw) != LARDON3D_PROJECT_DB_OK ||
|
||||
raw.capture_id != item.capture_id ||
|
||||
raw.source_asset_id != item.source_asset_id) {
|
||||
std::fprintf(stderr,
|
||||
"legacy RAW task %llu does not own selected cursor %u\n",
|
||||
static_cast<unsigned long long>(entry.task_id),
|
||||
execution.next_item_index);
|
||||
return false;
|
||||
}
|
||||
kind = RepresentationPendingKind::kLegacySerial;
|
||||
} else if (std::strcmp(entry.task_kind,
|
||||
LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND) == 0 &&
|
||||
entry.task_kind_version ==
|
||||
LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION) {
|
||||
Lardon3DProjectDbRawDevelopmentBatchTask batch{};
|
||||
if (lardon3d_project_db_load_raw_development_batch_task(
|
||||
runtime.state.project_db, entry.task_id, &batch) !=
|
||||
LARDON3D_PROJECT_DB_OK ||
|
||||
batch.selected_execution_id != execution.execution_id) {
|
||||
std::fprintf(stderr,
|
||||
"RAW batch task %llu does not own selected execution %llu\n",
|
||||
static_cast<unsigned long long>(entry.task_id),
|
||||
static_cast<unsigned long long>(execution.execution_id));
|
||||
return false;
|
||||
}
|
||||
kind = RepresentationPendingKind::kBatch;
|
||||
} else {
|
||||
std::fprintf(stderr,
|
||||
"representation recovery refuses pending task %llu of kind %s v%u\n",
|
||||
static_cast<unsigned long long>(entry.task_id), entry.task_kind,
|
||||
entry.task_kind_version);
|
||||
return false;
|
||||
}
|
||||
task_id = entry.task_id;
|
||||
}
|
||||
if (count < 8) return true;
|
||||
}
|
||||
}
|
||||
|
||||
bool recover_one_representation_task(Runtime &runtime, uint64_t task_id,
|
||||
const char *phase) {
|
||||
Lardon3DProjectRecoverySummary recovery{};
|
||||
return lardon3d_project_resume_recoverable_tasks(
|
||||
&runtime.state, lardon3d_task_kind_registry_production(),
|
||||
&recovery) == LARDON3D_PROJECT_DB_OK &&
|
||||
recovery.inspected == 1 && recovery.resumed == 1 &&
|
||||
recovery.failed == 0 && !recovery.queue_full &&
|
||||
wait_completed(runtime, task_id, phase);
|
||||
}
|
||||
|
||||
enum class FeaturePendingKind { kLegacySerial, kBatch };
|
||||
|
||||
bool feature_recovery_kind(const Lardon3DProjectRecoveryEntry &entry,
|
||||
FeaturePendingKind &kind) {
|
||||
if ((entry.status != LARDON3D_PROJECT_RECOVERABLE &&
|
||||
entry.status != LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE) ||
|
||||
entry.snapshot.recovery_state != TASK_PENDING)
|
||||
return false;
|
||||
if (std::strcmp(entry.task_kind, LARDON3D_FEATURE_EXTRACT_TASK_KIND) == 0 &&
|
||||
entry.task_kind_version == LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION) {
|
||||
kind = FeaturePendingKind::kLegacySerial;
|
||||
return true;
|
||||
}
|
||||
if (std::strcmp(entry.task_kind, LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND) == 0 &&
|
||||
entry.task_kind_version == LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION) {
|
||||
kind = FeaturePendingKind::kBatch;
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
bool publish_representations(Runtime &runtime, uint64_t execution_id,
|
||||
std::vector<uint64_t> &image_ids) {
|
||||
for (;;) {
|
||||
Lardon3DProjectDbSelectedExecution execution{};
|
||||
if (lardon3d_project_db_load_selected_execution(runtime.state.project_db, execution_id,
|
||||
&execution) != LARDON3D_PROJECT_DB_OK)
|
||||
return false;
|
||||
for (uint32_t index = execution.next_item_index; index < execution.item_count; ++index) {
|
||||
Lardon3DProjectDbSelectedExecutionItem item{};
|
||||
if (lardon3d_project_db_load_selected_execution_item(runtime.state.project_db, execution_id,
|
||||
index, &item) != LARDON3D_PROJECT_DB_OK)
|
||||
return false;
|
||||
uint64_t image_id = 0;
|
||||
if (item.representation_source == LARDON3D_SELECTED_REPRESENTATION_RAW_ASSET) {
|
||||
uint64_t task_id = 0;
|
||||
if (!lardon3d_project_enqueue_raw_development(&runtime.state, item.capture_id,
|
||||
item.source_asset_id, &task_id) ||
|
||||
!wait_completed(runtime, task_id, "raw.develop"))
|
||||
return false;
|
||||
Lardon3DProjectDbRawDevelopmentTask raw{};
|
||||
if (lardon3d_project_db_load_raw_development_task(runtime.state.project_db, task_id, &raw) !=
|
||||
LARDON3D_PROJECT_DB_OK ||
|
||||
raw.phase != LARDON3D_RAW_DEVELOPMENT_TASK_PUBLISHED || !raw.has_image ||
|
||||
raw.capture_id != item.capture_id || raw.source_asset_id != item.source_asset_id)
|
||||
return false;
|
||||
image_id = raw.image_id;
|
||||
} else {
|
||||
if (lardon3d_project_db_get_selected_capture_image(runtime.state.project_db, item.capture_id,
|
||||
&image_id) != LARDON3D_PROJECT_DB_OK)
|
||||
return false;
|
||||
}
|
||||
if (lardon3d_project_db_record_selected_representation(runtime.state.project_db, execution_id,
|
||||
index, image_id, index + 1u) !=
|
||||
if (lardon3d_project_db_load_selected_execution(
|
||||
runtime.state.project_db, execution_id, &execution) !=
|
||||
LARDON3D_PROJECT_DB_OK)
|
||||
return false;
|
||||
RepresentationPendingKind pending{};
|
||||
uint64_t pending_task_id = 0;
|
||||
if (!find_pending_representation_task(runtime, execution, pending,
|
||||
pending_task_id))
|
||||
return false;
|
||||
const RepresentationAction action = choose_representation_action(execution, pending);
|
||||
if (action == RepresentationAction::kReject) return false;
|
||||
if (action == RepresentationAction::kCollect) break;
|
||||
if (action == RepresentationAction::kRecoverLegacySerial) {
|
||||
const uint32_t item_index = execution.next_item_index;
|
||||
Lardon3DProjectDbSelectedExecutionItem item{};
|
||||
if (lardon3d_project_db_load_selected_execution_item(
|
||||
runtime.state.project_db, execution_id, item_index, &item) !=
|
||||
LARDON3D_PROJECT_DB_OK ||
|
||||
!recover_one_representation_task(runtime, pending_task_id,
|
||||
"raw.develop recovered existing"))
|
||||
return false;
|
||||
Lardon3DProjectDbRawDevelopmentTask raw{};
|
||||
if (lardon3d_project_db_load_raw_development_task(
|
||||
runtime.state.project_db, pending_task_id, &raw) !=
|
||||
LARDON3D_PROJECT_DB_OK ||
|
||||
raw.phase != LARDON3D_RAW_DEVELOPMENT_TASK_PUBLISHED || !raw.has_image ||
|
||||
raw.capture_id != item.capture_id ||
|
||||
raw.source_asset_id != item.source_asset_id ||
|
||||
lardon3d_project_db_record_selected_representation(
|
||||
runtime.state.project_db, execution_id, item_index, raw.image_id,
|
||||
item_index + 1u) != LARDON3D_PROJECT_DB_OK)
|
||||
return false;
|
||||
/* The legacy Task predates the selected-execution batch owner. Its exact
|
||||
* current item is committed once before the batch reads the advanced
|
||||
* cursor; no completed selected prefix is replayed or re-identified. */
|
||||
continue;
|
||||
}
|
||||
if (lardon3d_project_db_load_selected_execution(runtime.state.project_db, execution_id,
|
||||
&execution) != LARDON3D_PROJECT_DB_OK ||
|
||||
if (action == RepresentationAction::kRecoverBatch) {
|
||||
if (!recover_one_representation_task(runtime, pending_task_id,
|
||||
"raw.develop.batch recovered existing"))
|
||||
return false;
|
||||
continue;
|
||||
}
|
||||
uint64_t batch_task_id = 0;
|
||||
if (!lardon3d_project_enqueue_raw_development_batch(
|
||||
&runtime.state, execution_id, &batch_task_id) ||
|
||||
!wait_completed(runtime, batch_task_id, "raw.develop.batch"))
|
||||
return false;
|
||||
}
|
||||
image_ids.clear();
|
||||
Lardon3DProjectDbSelectedExecution execution{};
|
||||
if (lardon3d_project_db_load_selected_execution(runtime.state.project_db,
|
||||
execution_id, &execution) !=
|
||||
LARDON3D_PROJECT_DB_OK ||
|
||||
execution.stage != LARDON3D_SELECTED_EXECUTION_CALIBRATION)
|
||||
return false;
|
||||
image_ids.clear();
|
||||
for (uint32_t index = 0; index < execution.item_count; ++index) {
|
||||
Lardon3DProjectDbSelectedExecutionItem item{};
|
||||
if (lardon3d_project_db_load_selected_execution_item(runtime.state.project_db, execution_id,
|
||||
|
|
@ -1491,28 +1723,98 @@ bool publish_representations(Runtime &runtime, uint64_t execution_id,
|
|||
return true;
|
||||
}
|
||||
|
||||
bool extract_features(Runtime &runtime, const std::vector<uint64_t> &image_ids,
|
||||
bool extract_features(Runtime &runtime, uint64_t selected_execution_id,
|
||||
const std::vector<uint64_t> &image_ids,
|
||||
const Lardon3DFeatureExtractorParameters ¶meters,
|
||||
std::vector<Lardon3DProjectDbFeatureSet> &feature_sets) {
|
||||
unsigned char fingerprint[32]{};
|
||||
lardon3d_feature_extractor_parameter_fingerprint(¶meters, fingerprint);
|
||||
feature_sets.clear();
|
||||
bool complete = true;
|
||||
for (const uint64_t image_id : image_ids) {
|
||||
Lardon3DProjectDbFeatureSet feature{};
|
||||
if (lardon3d_project_db_find_feature_set(runtime.state.project_db, image_id,
|
||||
LARDON3D_FEATURE_EXTRACTOR_KIND,
|
||||
LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint,
|
||||
&feature) != LARDON3D_PROJECT_DB_OK) {
|
||||
uint64_t task_id = 0;
|
||||
if (!lardon3d_project_enqueue_feature_extract(&runtime.state, image_id, ¶meters,
|
||||
&task_id) ||
|
||||
!wait_completed(runtime, task_id, "features.extract") ||
|
||||
lardon3d_project_db_find_feature_set(
|
||||
complete = false;
|
||||
break;
|
||||
}
|
||||
feature_sets.push_back(feature);
|
||||
}
|
||||
if (complete) return true;
|
||||
|
||||
/* A pre-v25 single-image Task is historical durable work, not a batch
|
||||
* cursor. Reopening creates an empty Queue, so restore and finish that exact
|
||||
* Task once before the new owner starts its selected suffix. Never rewrite or
|
||||
* replace the legacy row (Task 166 in the inspected A6000 project). */
|
||||
uint64_t pending_task_id = 0;
|
||||
FeaturePendingKind pending_kind{};
|
||||
uint64_t cursor = 0;
|
||||
for (;;) {
|
||||
Lardon3DProjectRecoveryEntry tasks[8]{};
|
||||
size_t count = 0;
|
||||
if (lardon3d_project_list_recoverable(
|
||||
&runtime.state, lardon3d_task_kind_registry_production(), cursor,
|
||||
tasks, 8, &count) !=
|
||||
LARDON3D_PROJECT_DB_OK)
|
||||
return false;
|
||||
for (size_t index = 0; index < count; ++index) {
|
||||
cursor = tasks[index].task_id;
|
||||
FeaturePendingKind kind{};
|
||||
if (pending_task_id != 0 || !feature_recovery_kind(tasks[index], kind)) {
|
||||
/* Registry recovery resumes every pending Task. Refuse ambiguity or an
|
||||
* unrelated pipeline owner before transferring anything to Queue. */
|
||||
std::fprintf(stderr,
|
||||
"feature recovery refuses pending task %llu of kind %s v%u\n",
|
||||
static_cast<unsigned long long>(tasks[index].task_id),
|
||||
tasks[index].task_kind, tasks[index].task_kind_version);
|
||||
return false;
|
||||
}
|
||||
if (kind == FeaturePendingKind::kLegacySerial) {
|
||||
Lardon3DProjectDbFeatureExtractTask legacy{};
|
||||
if (lardon3d_project_db_load_feature_extract_task(
|
||||
runtime.state.project_db, tasks[index].task_id, &legacy) !=
|
||||
LARDON3D_PROJECT_DB_OK ||
|
||||
std::memcmp(legacy.parameter_fingerprint, fingerprint, 32) != 0)
|
||||
return false;
|
||||
} else {
|
||||
Lardon3DProjectDbFeatureExtractBatchTask batch{};
|
||||
if (lardon3d_project_db_load_feature_extract_batch_task(
|
||||
runtime.state.project_db, tasks[index].task_id, &batch) !=
|
||||
LARDON3D_PROJECT_DB_OK ||
|
||||
batch.selected_execution_id != selected_execution_id ||
|
||||
std::memcmp(batch.parameter_fingerprint, fingerprint, 32) != 0)
|
||||
return false;
|
||||
}
|
||||
pending_task_id = tasks[index].task_id;
|
||||
pending_kind = kind;
|
||||
}
|
||||
if (count < 8) break;
|
||||
}
|
||||
if (pending_task_id != 0) {
|
||||
const char *phase = pending_kind == FeaturePendingKind::kLegacySerial
|
||||
? "features.extract recovered legacy"
|
||||
: "features.extract.batch recovered";
|
||||
if (!recover_one_representation_task(runtime, pending_task_id, phase))
|
||||
return false;
|
||||
}
|
||||
uint64_t batch_task_id = 0;
|
||||
if (pending_task_id == 0 || pending_kind == FeaturePendingKind::kLegacySerial) {
|
||||
if (!lardon3d_project_enqueue_feature_extract_batch(
|
||||
&runtime.state, selected_execution_id, ¶meters, &batch_task_id) ||
|
||||
!wait_completed(runtime, batch_task_id, "features.extract.batch")) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
feature_sets.clear();
|
||||
for (const uint64_t image_id : image_ids) {
|
||||
Lardon3DProjectDbFeatureSet feature{};
|
||||
if (lardon3d_project_db_find_feature_set(
|
||||
runtime.state.project_db, image_id, LARDON3D_FEATURE_EXTRACTOR_KIND,
|
||||
LARDON3D_FEATURE_EXTRACTOR_VERSION, fingerprint, &feature) !=
|
||||
LARDON3D_PROJECT_DB_OK)
|
||||
return false;
|
||||
}
|
||||
feature_sets.push_back(feature);
|
||||
}
|
||||
return true;
|
||||
|
|
@ -2724,7 +3026,8 @@ int main(int argc, char **argv) {
|
|||
}
|
||||
Runtime runtime;
|
||||
if (options.resume_geometry_existing || options.resume_pre_gv_existing ||
|
||||
options.resume_candidate_existing) {
|
||||
options.resume_candidate_existing ||
|
||||
options.resume_representations_existing) {
|
||||
if (!prepare_existing_project(options.project_dir, runtime)) {
|
||||
std::fprintf(stderr, "existing project must be absolute, normalized, and durable\n");
|
||||
return 2;
|
||||
|
|
@ -2741,11 +3044,31 @@ int main(int argc, char **argv) {
|
|||
runtime.stop_after_gv = options.stop_after_gv;
|
||||
if (options.resume_candidate_existing)
|
||||
return run_existing_candidate(runtime) ? 0 : 1;
|
||||
if (options.resume_pre_gv_existing || options.resume_geometry_existing)
|
||||
return (options.resume_pre_gv_existing ? run_existing_pre_gv(runtime)
|
||||
: run_existing_geometry(runtime)) ? 0 : 1;
|
||||
}
|
||||
Lardon3DProjectDbSelectedExecution execution{};
|
||||
const char *mode_name = options.mode == Mode::kA6000 ? "a6000" : "s21";
|
||||
bool ok = true;
|
||||
if (options.resume_representations_existing) {
|
||||
runtime.matcher_needed = !options.stop_after_representations;
|
||||
if (!start_runtime(runtime) ||
|
||||
lardon3d_project_db_load_selected_execution(
|
||||
runtime.state.project_db, options.selected_execution_id,
|
||||
&execution) != LARDON3D_PROJECT_DB_OK ||
|
||||
(execution.stage != LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS &&
|
||||
execution.stage != LARDON3D_SELECTED_EXECUTION_CALIBRATION)) {
|
||||
std::fprintf(stderr,
|
||||
"existing selected execution is unavailable or past representations\n");
|
||||
stop_runtime(runtime);
|
||||
return 1;
|
||||
}
|
||||
mode_name = "existing";
|
||||
} else {
|
||||
if (!prepare_empty_project(options.project_dir, runtime.project_path)) {
|
||||
std::fprintf(stderr, "project directory must be absolute, normalized and empty\n");
|
||||
std::fprintf(stderr,
|
||||
"project directory must be absolute, normalized and empty\n");
|
||||
return 2;
|
||||
}
|
||||
runtime.database_path = (options.project_dir / "project.lardon3d").string();
|
||||
|
|
@ -2755,29 +3078,46 @@ int main(int argc, char **argv) {
|
|||
std::fprintf(stderr, "campaign discovery or runtime setup failed\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
const char *mode_name = options.mode == Mode::kA6000 ? "a6000" : "s21";
|
||||
Lardon3DProjectDbScanSet scanset{};
|
||||
if (lardon3d_project_db_create_scanset(runtime.state.project_db, mode_name, &scanset) !=
|
||||
LARDON3D_PROJECT_DB_OK) {
|
||||
if (lardon3d_project_db_create_scanset(runtime.state.project_db, mode_name,
|
||||
&scanset) != LARDON3D_PROJECT_DB_OK) {
|
||||
stop_runtime(runtime);
|
||||
return 1;
|
||||
}
|
||||
uint64_t quality_task_id = 0;
|
||||
uint64_t campaign_task_id = 0;
|
||||
Lardon3DProjectDbSelectedExecution execution{};
|
||||
bool ok = run_task_pair(runtime, scanset.scanset_id, campaign,
|
||||
options.restart != RestartBoundary::kNone, quality_task_id,
|
||||
campaign_task_id) &&
|
||||
create_selection(runtime, options.mode, quality_task_id, campaign_task_id,
|
||||
campaign.confirmations.size(), execution);
|
||||
ok = run_task_pair(runtime, scanset.scanset_id, campaign,
|
||||
options.restart != RestartBoundary::kNone,
|
||||
quality_task_id, campaign_task_id) &&
|
||||
create_selection(runtime, options.mode, quality_task_id,
|
||||
campaign_task_id, campaign.confirmations.size(),
|
||||
execution);
|
||||
if (!ok || execution.execution_id == 0) {
|
||||
stop_runtime(runtime);
|
||||
return ok ? 0 : 1;
|
||||
}
|
||||
}
|
||||
|
||||
std::vector<uint64_t> image_ids;
|
||||
ok = publish_representations(runtime, execution.execution_id, image_ids);
|
||||
if (ok && options.stop_after_representations) {
|
||||
ok = lardon3d_project_db_load_selected_execution(
|
||||
runtime.state.project_db, execution.execution_id, &execution) ==
|
||||
LARDON3D_PROJECT_DB_OK &&
|
||||
execution.stage == LARDON3D_SELECTED_EXECUTION_CALIBRATION &&
|
||||
execution.next_item_index == execution.item_count;
|
||||
}
|
||||
if (ok && options.stop_after_representations) {
|
||||
std::printf(
|
||||
"{\"record\":\"existing_representations_summary\",\"ok\":true,"
|
||||
"\"execution_id\":%llu,\"next_item_index\":%u,\"item_count\":%u,"
|
||||
"\"stage\":\"CALIBRATION\",\"selected_prefix_replayed\":false,"
|
||||
"\"features_enqueued\":false,\"sparse_sfm_run\":false}\n",
|
||||
static_cast<unsigned long long>(execution.execution_id),
|
||||
execution.next_item_index, execution.item_count);
|
||||
stop_runtime(runtime);
|
||||
return 0;
|
||||
}
|
||||
if (ok && options.restart == RestartBoundary::kRepresentations)
|
||||
ok = restart_runtime(runtime, execution.execution_id, "representations") &&
|
||||
publish_representations(runtime, execution.execution_id, image_ids);
|
||||
|
|
@ -2787,11 +3127,11 @@ int main(int argc, char **argv) {
|
|||
* campaign-size rule; callers use one empty project per campaign. */
|
||||
const Lardon3DFeatureExtractorParameters orb{8192, 8, 20};
|
||||
std::vector<Lardon3DProjectDbFeatureSet> features;
|
||||
if (ok) ok = extract_features(runtime, image_ids, orb, features);
|
||||
if (ok) ok = extract_features(runtime, execution.execution_id, image_ids, orb, features);
|
||||
if (ok && options.restart == RestartBoundary::kFeatures) {
|
||||
ok = restart_runtime(runtime, execution.execution_id, "features") &&
|
||||
publish_representations(runtime, execution.execution_id, image_ids) &&
|
||||
extract_features(runtime, image_ids, orb, features);
|
||||
extract_features(runtime, execution.execution_id, image_ids, orb, features);
|
||||
}
|
||||
|
||||
uint64_t geometry_task_id = 0;
|
||||
|
|
|
|||
|
|
@ -151,6 +151,13 @@ typedef struct {
|
|||
Lardon3DExtractedFeatures output;
|
||||
} AdaptiveExtractWork;
|
||||
|
||||
bool lardon3d_feature_extract_batch_test_run(Lardon3DTask *task);
|
||||
|
||||
static bool batch_test_callback(Lardon3DTask *task, void *userdata) {
|
||||
(void)userdata;
|
||||
return lardon3d_feature_extract_batch_test_run(task);
|
||||
}
|
||||
|
||||
static bool adaptive_extract_callback(Lardon3DTask *task, void *userdata) {
|
||||
AdaptiveExtractWork *work = userdata;
|
||||
Lardon3DOpenCvTaskThreadControl control;
|
||||
|
|
@ -265,6 +272,92 @@ static bool deterministic_capture_preserves_load_gate(void) {
|
|||
return ok;
|
||||
}
|
||||
|
||||
static bool feature_batch_trials_exercise_participants(void) {
|
||||
Lardon3DHardwareProfile profile = {
|
||||
.logical_cpu_count = 8,
|
||||
.page_size_bytes = 4096,
|
||||
.memory_total_bytes = UINT64_C(16) * 1024 * 1024 * 1024,
|
||||
.cpu_architecture = "test",
|
||||
};
|
||||
Lardon3DResourcePolicy policy = {
|
||||
.system_cpu_reserve = 4,
|
||||
.maximum_cpu_load_ratio = 1.0,
|
||||
.maximum_io_pressure_avg10 = 100.0,
|
||||
.io_slot_capacity = 1,
|
||||
};
|
||||
Lardon3DResourceGovernor *governor =
|
||||
lardon3d_resource_governor_create(&profile, &policy);
|
||||
Lardon3DResourceSnapshot resources = deterministic_resource_snapshot(0.0);
|
||||
Lardon3DTaskCapabilityEnvelope envelope = {
|
||||
.count = 1,
|
||||
.capabilities = {{
|
||||
.estimate = {
|
||||
.memory_fixed_bytes = 64ULL * 1024 * 1024,
|
||||
.memory_bytes_per_item = 512ULL * 1024 * 1024,
|
||||
.minimum_batch_size = 1,
|
||||
.maximum_batch_size = 12,
|
||||
.desired_cpu_threads = 12,
|
||||
.desired_io_slots = 1,
|
||||
.task_class = LARDON3D_RESOURCE_TASK_CPU,
|
||||
},
|
||||
.backend = LARDON3D_RESOURCE_BACKEND_FIXED,
|
||||
.inflight_limit = 1,
|
||||
.cpu_reducible = true,
|
||||
.batch_adaptive = true,
|
||||
.cpu_batch_coupled = true,
|
||||
}},
|
||||
};
|
||||
bool ok = governor &&
|
||||
lardon3d_resource_governor_internal_set_capture_snapshot(governor, &resources);
|
||||
for (unsigned int sample = 0; sample < 2 && ok; ++sample) {
|
||||
Lardon3DResourceCapabilitySelection selection;
|
||||
Lardon3DResourceReservation *reservation = NULL;
|
||||
ok = lardon3d_resource_governor_internal_reserve_capability_available(
|
||||
governor, LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND, 1,
|
||||
&envelope, &selection, &reservation) && reservation &&
|
||||
selection.decision.cpu_threads == 1 && selection.decision.batch_size == 1 &&
|
||||
lardon3d_resource_governor_internal_record_sequence(
|
||||
governor, LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND, 1,
|
||||
&selection, UINT64_C(1000000000), 1) &&
|
||||
lardon3d_resource_governor_release(governor, reservation);
|
||||
}
|
||||
Lardon3DResourceCapabilitySelection trial;
|
||||
Lardon3DResourceReservation *trial_reservation = NULL;
|
||||
ok = ok && lardon3d_resource_governor_internal_reserve_capability_available(
|
||||
governor, LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND, 1,
|
||||
&envelope, &trial, &trial_reservation) && trial_reservation &&
|
||||
trial.decision.cpu_threads == 2 && trial.decision.batch_size == 2;
|
||||
if (trial_reservation)
|
||||
ok = lardon3d_resource_governor_release(governor, trial_reservation) && ok;
|
||||
|
||||
/* Candidate uses the same cross-item coupling: CPU2 with batch1 still has
|
||||
* only one membership to query. Its independent feedback key must therefore
|
||||
* open a measurable CPU2/batch2 rung as well. */
|
||||
for (unsigned int sample = 0; sample < 2 && ok; ++sample) {
|
||||
Lardon3DResourceCapabilitySelection selection;
|
||||
Lardon3DResourceReservation *reservation = NULL;
|
||||
ok = lardon3d_resource_governor_internal_reserve_capability_available(
|
||||
governor, "candidate_pair.generate", 1, &envelope, &selection,
|
||||
&reservation) && reservation &&
|
||||
selection.decision.cpu_threads == 1 && selection.decision.batch_size == 1 &&
|
||||
lardon3d_resource_governor_internal_record_sequence(
|
||||
governor, "candidate_pair.generate", 1, &selection,
|
||||
UINT64_C(1000000000), 1) &&
|
||||
lardon3d_resource_governor_release(governor, reservation);
|
||||
}
|
||||
trial_reservation = NULL;
|
||||
ok = ok && lardon3d_resource_governor_internal_reserve_capability_available(
|
||||
governor, "candidate_pair.generate", 1, &envelope, &trial,
|
||||
&trial_reservation) && trial_reservation &&
|
||||
trial.decision.cpu_threads == 2 && trial.decision.batch_size == 2;
|
||||
if (trial_reservation)
|
||||
ok = lardon3d_resource_governor_release(governor, trial_reservation) && ok;
|
||||
if (governor)
|
||||
ok = lardon3d_resource_governor_internal_set_capture_snapshot(governor, NULL) && ok;
|
||||
lardon3d_resource_governor_destroy(governor);
|
||||
return ok;
|
||||
}
|
||||
|
||||
static bool run_adaptive_output_equivalence(const char *path) {
|
||||
cpu_set_t allowed_mask;
|
||||
CPU_ZERO(&allowed_mask);
|
||||
|
|
@ -466,6 +559,7 @@ static void *consolidate_thread(void *userdata) {
|
|||
}
|
||||
|
||||
static bool run_test(void) {
|
||||
CHECK(feature_batch_trials_exercise_participants());
|
||||
char root[] = "/tmp/lardon3d-feature-task-XXXXXX";
|
||||
CHECK(deterministic_capture_preserves_load_gate());
|
||||
CHECK(mkdtemp(root) && setenv("LARDON3D_PROJECTS_ROOT", root, 1) == 0);
|
||||
|
|
@ -557,31 +651,61 @@ static bool run_test(void) {
|
|||
* pool. Feature, SIFT, and RootSIFT must each restore the captured value. */
|
||||
CHECK(lardon3d_feature_opencv_configure_threads(3));
|
||||
CHECK(setenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS", "1", 1) == 0);
|
||||
/* Batch CPU1 configuration deliberately fails after mutating OpenCV. The
|
||||
* callback must restore the prior process-global value before Queue can run
|
||||
* any later Task. */
|
||||
Lardon3DResourceEstimate failed_batch_estimate = {
|
||||
.minimum_batch_size = 1,
|
||||
.maximum_batch_size = 1,
|
||||
.desired_cpu_threads = 1,
|
||||
.task_class = LARDON3D_RESOURCE_TASK_CPU,
|
||||
};
|
||||
Lardon3DTask *failed_batch = lardon3d_task_create_typed(
|
||||
"Feature batch restoration", &failed_batch_estimate,
|
||||
"features.extract.batch.restore-test",
|
||||
LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION, batch_test_callback,
|
||||
NULL, NULL);
|
||||
uint64_t failed_batch_task = 0;
|
||||
/* This synthetic Task shares the project Queue with durable Tasks. Reserve
|
||||
* its ID through Project DB so the next durable enqueue cannot reuse it. */
|
||||
CHECK(failed_batch &&
|
||||
lardon3d_project_db_allocate_task_id(state.project_db,
|
||||
&failed_batch_task) ==
|
||||
LARDON3D_PROJECT_DB_OK &&
|
||||
lardon3d_task_assign_id(failed_batch, failed_batch_task) &&
|
||||
lardon3d_task_queue_add(state.task_queue, failed_batch,
|
||||
NULL) &&
|
||||
wait_state(state.task_queue, failed_batch_task, TASK_FAILED,
|
||||
&snapshot) &&
|
||||
lardon3d_feature_opencv_thread_count() == 3 &&
|
||||
unsetenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS") == 0);
|
||||
Lardon3DFeatureExtractorParameters failed_orb_parameters = {509, 4, 10};
|
||||
uint64_t failed_orb_task = 0;
|
||||
CHECK(setenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS", "once", 1) == 0);
|
||||
CHECK(lardon3d_project_enqueue_feature_extract(
|
||||
&state, image.image_id, &failed_orb_parameters, &failed_orb_task) &&
|
||||
wait_state(state.task_queue, failed_orb_task, TASK_FAILED, &snapshot) &&
|
||||
lardon3d_feature_opencv_thread_count() == 3);
|
||||
&state, image.image_id, &failed_orb_parameters, &failed_orb_task));
|
||||
CHECK(wait_state(state.task_queue, failed_orb_task, TASK_FAILED, &snapshot));
|
||||
CHECK(lardon3d_feature_opencv_thread_count() == 3);
|
||||
|
||||
Lardon3DSiftExtractorParameters failed_sift_parameters =
|
||||
lardon3d_sift_precision_classic_v1(false);
|
||||
failed_sift_parameters.max_features = 511;
|
||||
uint64_t failed_sift_task = 0;
|
||||
CHECK(setenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS", "once", 1) == 0);
|
||||
CHECK(lardon3d_project_enqueue_sift_extract(
|
||||
&state, image.image_id, &failed_sift_parameters,
|
||||
&failed_sift_task) &&
|
||||
wait_state(state.task_queue, failed_sift_task, TASK_FAILED, &snapshot) &&
|
||||
lardon3d_feature_opencv_thread_count() == 3);
|
||||
&failed_sift_task));
|
||||
CHECK(wait_state(state.task_queue, failed_sift_task, TASK_FAILED, &snapshot));
|
||||
CHECK(lardon3d_feature_opencv_thread_count() == 3);
|
||||
failed_sift_parameters.rootsift = true;
|
||||
failed_sift_parameters.max_features = 510;
|
||||
uint64_t failed_rootsift_task = 0;
|
||||
CHECK(setenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS", "once", 1) == 0);
|
||||
CHECK(lardon3d_project_enqueue_sift_extract(
|
||||
&state, image.image_id, &failed_sift_parameters,
|
||||
&failed_rootsift_task) &&
|
||||
wait_state(state.task_queue, failed_rootsift_task, TASK_FAILED, &snapshot) &&
|
||||
lardon3d_feature_opencv_thread_count() == 3 &&
|
||||
unsetenv("LARDON3D_TEST_OPENCV_CONFIGURE_FAILURE_THREADS") == 0);
|
||||
&failed_rootsift_task));
|
||||
CHECK(wait_state(state.task_queue, failed_rootsift_task, TASK_FAILED, &snapshot));
|
||||
CHECK(lardon3d_feature_opencv_thread_count() == 3);
|
||||
|
||||
Lardon3DSiftExtractorParameters sift_parameters = lardon3d_sift_precision_classic_v1(false);
|
||||
sift_parameters.max_features = 512;
|
||||
|
|
|
|||
|
|
@ -172,6 +172,8 @@ static bool create_v11_database(const char *path) {
|
|||
lardon3d_project_db_close(database);
|
||||
return execute_sql(path,
|
||||
"PRAGMA foreign_keys=OFF;BEGIN IMMEDIATE;"
|
||||
"DROP TABLE IF EXISTS feature_extract_batch_tasks;"
|
||||
"DROP TABLE IF EXISTS raw_development_batch_tasks;"
|
||||
"DROP TABLE IF EXISTS capture_calibration_selections;"
|
||||
"DROP TABLE IF EXISTS optical_calibration_profiles;"
|
||||
"DROP TABLE IF EXISTS capture_optical_configurations;"
|
||||
|
|
|
|||
|
|
@ -878,6 +878,8 @@ static bool run_task_test(void) {
|
|||
CHECK(exec_sql(
|
||||
database_path,
|
||||
"PRAGMA foreign_keys=OFF;BEGIN IMMEDIATE;"
|
||||
"DROP TABLE IF EXISTS feature_extract_batch_tasks;"
|
||||
"DROP TABLE IF EXISTS raw_development_batch_tasks;"
|
||||
"DROP TABLE IF EXISTS capture_calibration_selections;"
|
||||
"DROP TABLE IF EXISTS optical_calibration_profiles;"
|
||||
"DROP TABLE IF EXISTS capture_optical_configurations;"
|
||||
|
|
|
|||
|
|
@ -252,7 +252,8 @@ static bool test_profiles_assignments_and_calibrations(void) {
|
|||
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY];
|
||||
CHECK(lardon3d_project_db_open(path, &database, error) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_schema_version(database) == 23);
|
||||
CHECK(lardon3d_project_db_schema_version(database) ==
|
||||
LARDON3D_PROJECT_DB_SCHEMA_VERSION);
|
||||
|
||||
Lardon3DProjectDbScanSet scanset;
|
||||
CHECK(lardon3d_project_db_create_scanset(database, "mixed optical campaign",
|
||||
|
|
@ -1146,6 +1147,7 @@ static bool downgrade_to_v22_fixture(const char *path) {
|
|||
return raw_sql(
|
||||
path,
|
||||
"PRAGMA foreign_keys=OFF;BEGIN IMMEDIATE;"
|
||||
"DROP TABLE raw_development_batch_tasks;"
|
||||
"DROP TABLE capture_calibration_selections;"
|
||||
"DROP TABLE optical_calibration_profiles;"
|
||||
"DROP TABLE capture_optical_configurations;"
|
||||
|
|
@ -1195,7 +1197,8 @@ static bool test_migration_rollback_retry_and_equivalence(void) {
|
|||
|
||||
CHECK(lardon3d_project_db_open(migrated_path, &database, error) ==
|
||||
LARDON3D_PROJECT_DB_OK &&
|
||||
lardon3d_project_db_schema_version(database) == 23);
|
||||
lardon3d_project_db_schema_version(database) ==
|
||||
LARDON3D_PROJECT_DB_SCHEMA_VERSION);
|
||||
Lardon3DOpticalCaptureAssignment migrated_unresolved;
|
||||
CHECK(lardon3d_optical_capture_assignment_load(database, 1,
|
||||
&migrated_unresolved) ==
|
||||
|
|
|
|||
|
|
@ -365,7 +365,89 @@ int main() {
|
|||
recovery_entry.task_kind_version = 1;
|
||||
CHECK(!geometry_recovery_kind(recovery_entry, recovery_kind));
|
||||
|
||||
Lardon3DProjectDbSelectedExecution selected{};
|
||||
selected.execution_id = 9;
|
||||
selected.stage = LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS;
|
||||
selected.next_item_index = 72;
|
||||
selected.item_count = 100;
|
||||
/* A legacy serial owner is completed only at the exact durable cursor. The
|
||||
* subsequent no-owner decision hands the untouched suffix to one batch. */
|
||||
CHECK(choose_representation_action(
|
||||
selected, RepresentationPendingKind::kLegacySerial) ==
|
||||
RepresentationAction::kRecoverLegacySerial);
|
||||
selected.next_item_index = 73;
|
||||
CHECK(choose_representation_action(selected, RepresentationPendingKind::kNone) ==
|
||||
RepresentationAction::kEnqueueBatch);
|
||||
selected.next_item_index = 0;
|
||||
CHECK(choose_representation_action(selected, RepresentationPendingKind::kNone) ==
|
||||
RepresentationAction::kEnqueueBatch);
|
||||
selected.next_item_index = selected.item_count;
|
||||
selected.stage = LARDON3D_SELECTED_EXECUTION_CALIBRATION;
|
||||
CHECK(choose_representation_action(selected, RepresentationPendingKind::kNone) ==
|
||||
RepresentationAction::kCollect);
|
||||
CHECK(choose_representation_action(selected, RepresentationPendingKind::kBatch) ==
|
||||
RepresentationAction::kRecoverBatch);
|
||||
|
||||
Lardon3DProjectRecoveryEntry feature_pending{};
|
||||
feature_pending.status = LARDON3D_PROJECT_RECOVERABLE;
|
||||
feature_pending.snapshot.recovery_state = TASK_PENDING;
|
||||
std::snprintf(feature_pending.task_kind, sizeof(feature_pending.task_kind),
|
||||
"%s", LARDON3D_FEATURE_EXTRACT_TASK_KIND);
|
||||
feature_pending.task_kind_version = LARDON3D_FEATURE_EXTRACT_TASK_KIND_VERSION;
|
||||
FeaturePendingKind feature_kind{};
|
||||
CHECK(feature_recovery_kind(feature_pending, feature_kind));
|
||||
CHECK(feature_kind == FeaturePendingKind::kLegacySerial);
|
||||
/* Completed representations defer exact Feature owners to extract_features,
|
||||
* which performs the scientific-domain and selected-execution checks. */
|
||||
CHECK(representation_defers_feature_task(selected, feature_pending));
|
||||
selected.stage = LARDON3D_SELECTED_EXECUTION_REPRESENTATIONS;
|
||||
CHECK(!representation_defers_feature_task(selected, feature_pending));
|
||||
selected.stage = LARDON3D_SELECTED_EXECUTION_CALIBRATION;
|
||||
--selected.next_item_index;
|
||||
CHECK(!representation_defers_feature_task(selected, feature_pending));
|
||||
selected.next_item_index = selected.item_count;
|
||||
++feature_pending.task_kind_version;
|
||||
CHECK(!representation_defers_feature_task(selected, feature_pending));
|
||||
std::snprintf(feature_pending.task_kind, sizeof(feature_pending.task_kind),
|
||||
"%s", LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND);
|
||||
feature_pending.task_kind_version =
|
||||
LARDON3D_FEATURE_EXTRACT_BATCH_TASK_KIND_VERSION;
|
||||
CHECK(representation_defers_feature_task(selected, feature_pending));
|
||||
CHECK(feature_recovery_kind(feature_pending, feature_kind));
|
||||
CHECK(feature_kind == FeaturePendingKind::kBatch);
|
||||
feature_pending.status = LARDON3D_PROJECT_RECOVERY_UNKNOWN_TASK_KIND;
|
||||
CHECK(!feature_recovery_kind(feature_pending, feature_kind));
|
||||
feature_pending.status = LARDON3D_PROJECT_RECOVERABLE;
|
||||
std::snprintf(feature_pending.task_kind, sizeof(feature_pending.task_kind),
|
||||
"%s", LARDON3D_TRACK_BUILDER_TASK_KIND);
|
||||
feature_pending.task_kind_version = LARDON3D_TRACK_BUILDER_TASK_KIND_VERSION;
|
||||
CHECK(!representation_defers_feature_task(selected, feature_pending));
|
||||
CHECK(!feature_recovery_kind(feature_pending, feature_kind));
|
||||
|
||||
Options options;
|
||||
CHECK(parse_case({"runner", "--resume-representations-existing",
|
||||
"--project-dir", "/tmp/not-opened",
|
||||
"--selected-execution-id", "9",
|
||||
"--stop-after-representations"}, options));
|
||||
CHECK(options.resume_representations_existing &&
|
||||
options.selected_execution_id == 9 &&
|
||||
options.has_selected_execution_id && options.stop_after_representations);
|
||||
for (const char *invalid_id : {"0", "-1", "+1", "01", "1x"}) {
|
||||
options = {};
|
||||
CHECK(!parse_case({"runner", "--resume-representations-existing",
|
||||
"--project-dir", "/tmp/not-opened",
|
||||
"--selected-execution-id", invalid_id}, options));
|
||||
}
|
||||
options = {};
|
||||
CHECK(!parse_case({"runner", "--resume-representations-existing",
|
||||
"--project-dir", "/tmp/not-opened"}, options));
|
||||
options = {};
|
||||
CHECK(!parse_case({"runner", "--resume-representations-existing",
|
||||
"--project-dir", "/tmp/not-opened",
|
||||
"--selected-execution-id", "9", "--cpu-budget", "2"},
|
||||
options));
|
||||
|
||||
options = {};
|
||||
CHECK(parse_case({"runner", "--resume-pre-gv-existing", "--project-dir",
|
||||
"/tmp/not-opened"}, options));
|
||||
CHECK(options.matcher_mode == MatcherMode::kAuto &&
|
||||
|
|
|
|||
|
|
@ -22,9 +22,11 @@
|
|||
} while (0)
|
||||
|
||||
/* Historical migration fixtures are produced from a temporary current DB.
|
||||
* Remove every additive v23 object first so the fixture really is historical
|
||||
* Remove every additive v23/v24/v25 object first so the fixture really is historical
|
||||
* rather than a lower version number with future tables left behind. */
|
||||
#define DROP_OPTICAL_V23_SQL \
|
||||
"DROP TABLE IF EXISTS feature_extract_batch_tasks;" \
|
||||
"DROP TABLE IF EXISTS raw_development_batch_tasks;" \
|
||||
"DROP TABLE IF EXISTS capture_calibration_selections;" \
|
||||
"DROP TABLE IF EXISTS optical_calibration_profiles;" \
|
||||
"DROP TABLE IF EXISTS capture_optical_configurations;" \
|
||||
|
|
@ -131,7 +133,7 @@ static bool create_future_database(const char *path) {
|
|||
sqlite3_exec(
|
||||
connection,
|
||||
"CREATE TABLE metadata(key TEXT PRIMARY KEY,value INTEGER NOT NULL);"
|
||||
"INSERT INTO metadata VALUES('schema_version',24);",
|
||||
"INSERT INTO metadata VALUES('schema_version',26);",
|
||||
NULL, NULL, NULL) == SQLITE_OK;
|
||||
return sqlite3_close(connection) == SQLITE_OK && ok;
|
||||
}
|
||||
|
|
@ -907,6 +909,7 @@ static bool run_test(void) {
|
|||
char v13_path[512], true_v14_path[512], failed_v14_path[512],
|
||||
failed_v15_path[512], true_v15_path[512], v17_path[512],
|
||||
failed_v18_path[512], failed_v19_path[512], raw_missing_table_path[512];
|
||||
char v24_path[512];
|
||||
CHECK(snprintf(database_path, sizeof(database_path), "%s/project.db",
|
||||
directory) > 0);
|
||||
CHECK(snprintf(artifact_path, sizeof(artifact_path), "%s/artifact.bin",
|
||||
|
|
@ -956,9 +959,34 @@ static bool run_test(void) {
|
|||
"%s/failed-v19.db", directory) > 0);
|
||||
CHECK(snprintf(raw_missing_table_path, sizeof(raw_missing_table_path),
|
||||
"%s/raw-missing-table.db", directory) > 0);
|
||||
CHECK(snprintf(v24_path, sizeof(v24_path), "%s/v24.db", directory) > 0);
|
||||
|
||||
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY];
|
||||
Lardon3DProjectDb *database = NULL;
|
||||
/* v25 is DDL-only: an injected failure rolls back table and marker, and a
|
||||
* retry creates no fabricated association or historical cursor. */
|
||||
CHECK(lardon3d_project_db_open(v24_path, &database, error) == LARDON3D_PROJECT_DB_OK);
|
||||
lardon3d_project_db_close(database);
|
||||
database = NULL;
|
||||
CHECK(execute_test_sql(v24_path,
|
||||
"DROP TABLE feature_extract_batch_tasks;"
|
||||
"UPDATE metadata SET value=24 WHERE key='schema_version';"));
|
||||
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V25", "1", 1) == 0);
|
||||
CHECK(lardon3d_project_db_open(v24_path, &database, error) != LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V25") == 0);
|
||||
CHECK(query_integer(v24_path, "SELECT value FROM metadata WHERE key='schema_version'", 24));
|
||||
CHECK(query_integer(v24_path,
|
||||
"SELECT count(*) FROM sqlite_master WHERE type='table' AND "
|
||||
"name='feature_extract_batch_tasks'", 0));
|
||||
database = NULL;
|
||||
Lardon3DProjectDbResult v25_retry = lardon3d_project_db_open(v24_path, &database, error);
|
||||
if (v25_retry != LARDON3D_PROJECT_DB_OK)
|
||||
fprintf(stderr, "v25 retry: %d %s\n", v25_retry, error);
|
||||
CHECK(v25_retry == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_schema_version(database) == LARDON3D_PROJECT_DB_SCHEMA_VERSION);
|
||||
lardon3d_project_db_close(database);
|
||||
database = NULL;
|
||||
CHECK(query_integer(v24_path, "SELECT count(*) FROM feature_extract_batch_tasks", 0));
|
||||
CHECK(lardon3d_project_db_open(database_path, &database, error) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(database && lardon3d_project_db_schema_version(database) ==
|
||||
|
|
@ -2676,6 +2704,7 @@ static bool run_test(void) {
|
|||
CHECK(unlink(failed_v18_path) == 0);
|
||||
CHECK(unlink(failed_v19_path) == 0);
|
||||
CHECK(unlink(raw_missing_table_path) == 0);
|
||||
CHECK(unlink(v24_path) == 0);
|
||||
CHECK(rmdir(directory) == 0);
|
||||
return true;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -89,6 +89,13 @@ int main() {
|
|||
Lardon3DTask *task = lardon3d_project_create_raw_development_task(
|
||||
&state, capture.capture_id, raw_asset.asset_id, &task_id);
|
||||
CHECK(task != nullptr && task_id != 0);
|
||||
Lardon3DResourceEstimate single_estimate{};
|
||||
CHECK(lardon3d_task_resource_estimate(task, &single_estimate));
|
||||
CHECK(single_estimate.memory_fixed_bytes >= (UINT64_C(2) << 30) &&
|
||||
single_estimate.memory_bytes_per_item == 0 &&
|
||||
single_estimate.minimum_batch_size == 1 &&
|
||||
single_estimate.maximum_batch_size == 1 &&
|
||||
single_estimate.desired_cpu_threads == 1);
|
||||
Lardon3DProjectDbTask generic{};
|
||||
Lardon3DProjectDbRawDevelopmentTask typed{};
|
||||
CHECK(lardon3d_project_db_load_task(database, task_id, &generic) == LARDON3D_PROJECT_DB_OK);
|
||||
|
|
@ -154,6 +161,64 @@ int main() {
|
|||
database, &snapshot, LARDON3D_RAW_DEVELOPMENT_TASK_KIND, 2, nullptr, &typed,
|
||||
7) == LARDON3D_PROJECT_DB_INVALID_ARGUMENT);
|
||||
|
||||
/* These minimal selected rows isolate the v24 parent association contract;
|
||||
* item science remains covered by selected-execution tests and the frozen
|
||||
* per-Capture developer tests. */
|
||||
sqlite3 *batch_seed = nullptr;
|
||||
CHECK(sqlite3_open(database_path.c_str(), &batch_seed) == SQLITE_OK);
|
||||
CHECK(sqlite3_exec(batch_seed, "PRAGMA foreign_keys=OFF;"
|
||||
"INSERT INTO selected_executions(execution_id,quality_task_id,"
|
||||
"campaign_task_id,stage,next_item_index,item_count,created_at)"
|
||||
"VALUES(9001,7001,7002,1,0,1,7),(9002,7003,7004,1,0,1,7)",
|
||||
nullptr, nullptr, nullptr) == SQLITE_OK);
|
||||
CHECK(sqlite3_close(batch_seed) == SQLITE_OK);
|
||||
uint64_t batch_task_id = 0;
|
||||
Lardon3DTask *batch_task = lardon3d_project_create_raw_development_batch_task(
|
||||
&state, 9001, &batch_task_id);
|
||||
CHECK(batch_task != nullptr && batch_task_id != 0);
|
||||
Lardon3DResourceEstimate batch_estimate{};
|
||||
CHECK(lardon3d_task_resource_estimate(batch_task, &batch_estimate));
|
||||
/* The pixel-bounded participant allowance must permit the full safe window
|
||||
* from a 7 GiB post-reserve budget; a fixed context charge would incorrectly
|
||||
* reduce this exact boundary to seven participants. */
|
||||
CHECK(batch_estimate.memory_fixed_bytes == 0 &&
|
||||
batch_estimate.memory_bytes_per_item == (UINT64_C(896) << 20) &&
|
||||
batch_estimate.minimum_batch_size == 1 &&
|
||||
batch_estimate.maximum_batch_size == 8 &&
|
||||
batch_estimate.desired_cpu_threads == 8 &&
|
||||
batch_estimate.memory_bytes_per_item * batch_estimate.maximum_batch_size ==
|
||||
(UINT64_C(7) << 30) &&
|
||||
batch_estimate.desired_gpu_slots == 0 &&
|
||||
batch_estimate.desired_io_slots == 1 &&
|
||||
batch_estimate.task_class == LARDON3D_RESOURCE_TASK_MIXED);
|
||||
Lardon3DProjectDbRawDevelopmentBatchTask batch_typed{};
|
||||
CHECK(lardon3d_project_db_load_raw_development_batch_task(
|
||||
database, batch_task_id, &batch_typed) == LARDON3D_PROJECT_DB_OK &&
|
||||
batch_typed.selected_execution_id == 9001);
|
||||
Lardon3DTaskDurableSnapshot batch_snapshot{};
|
||||
CHECK(lardon3d_task_durable_snapshot(batch_task, &batch_snapshot));
|
||||
Lardon3DTaskKindBinding batch_binding{};
|
||||
CHECK(lardon3d_raw_development_batch_task_reconstruct(
|
||||
&batch_snapshot, &reconstruction, &batch_binding));
|
||||
CHECK(batch_binding.callback != nullptr && batch_binding.userdata != nullptr &&
|
||||
batch_binding.userdata_destroy != nullptr);
|
||||
batch_binding.userdata_destroy(batch_binding.userdata);
|
||||
CHECK(lardon3d_project_db_record_raw_development_batch_task(
|
||||
database, &batch_snapshot,
|
||||
LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND,
|
||||
LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION, nullptr,
|
||||
&batch_typed, 7) == LARDON3D_PROJECT_DB_OK);
|
||||
batch_typed.selected_execution_id = 9002;
|
||||
CHECK(lardon3d_project_db_record_raw_development_batch_task(
|
||||
database, &batch_snapshot,
|
||||
LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND,
|
||||
LARDON3D_RAW_DEVELOPMENT_BATCH_TASK_KIND_VERSION, nullptr,
|
||||
&batch_typed, 7) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||
CHECK(lardon3d_project_db_load_raw_development_batch_task(
|
||||
database, batch_task_id, &batch_typed) == LARDON3D_PROJECT_DB_OK &&
|
||||
batch_typed.selected_execution_id == 9001);
|
||||
lardon3d_task_destroy(batch_task);
|
||||
|
||||
sqlite3 *raw_database = nullptr;
|
||||
char update_mismatch_kind[128];
|
||||
std::snprintf(update_mismatch_kind, sizeof(update_mismatch_kind),
|
||||
|
|
|
|||
|
|
@ -18,6 +18,20 @@ static bool sql(const char *path, const char *text) {
|
|||
return sqlite3_close(connection) == SQLITE_OK && code == SQLITE_OK;
|
||||
}
|
||||
|
||||
static void feature_asset_path(
|
||||
const unsigned char hash[LARDON3D_PROJECT_DB_SHA256_SIZE],
|
||||
char path[LARDON3D_PROJECT_DB_PATH_CAPACITY]) {
|
||||
static const char digits[] = "0123456789abcdef";
|
||||
char hex[LARDON3D_PROJECT_DB_SHA256_SIZE * 2 + 1];
|
||||
for (size_t index = 0; index < LARDON3D_PROJECT_DB_SHA256_SIZE; ++index) {
|
||||
hex[index * 2] = digits[hash[index] >> 4];
|
||||
hex[index * 2 + 1] = digits[hash[index] & 15U];
|
||||
}
|
||||
hex[sizeof(hex) - 1] = '\0';
|
||||
(void)snprintf(path, LARDON3D_PROJECT_DB_PATH_CAPACITY,
|
||||
"assets/features/%c%c/%s", hex[0], hex[1], hex);
|
||||
}
|
||||
|
||||
static bool seed_v21(const char *path) {
|
||||
Lardon3DProjectDb *database = NULL;
|
||||
char error[LARDON3D_PROJECT_DB_ERROR_CAPACITY];
|
||||
|
|
@ -50,6 +64,8 @@ static bool seed_v21(const char *path) {
|
|||
"INSERT INTO sparse_calibration_scope_images VALUES(1,1,1);"
|
||||
/* This fixture is a true v21 database; future additive objects must not
|
||||
remain merely because it was generated from a temporary current DB. */
|
||||
"DROP TABLE feature_extract_batch_tasks;"
|
||||
"DROP TABLE raw_development_batch_tasks;"
|
||||
"DROP TABLE capture_calibration_selections;"
|
||||
"DROP TABLE optical_calibration_profiles;"
|
||||
"DROP TABLE capture_optical_configurations;"
|
||||
|
|
@ -89,6 +105,29 @@ static bool run(void) {
|
|||
if (migration_result != LARDON3D_PROJECT_DB_OK)
|
||||
fprintf(stderr, "retry migration result=%d error=%s\n", (int)migration_result, error);
|
||||
CHECK(migration_result == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_schema_version(database) ==
|
||||
LARDON3D_PROJECT_DB_SCHEMA_VERSION);
|
||||
CHECK(sql(path, "SELECT 1 FROM raw_development_batch_tasks WHERE 0;"));
|
||||
lardon3d_project_db_close(database);
|
||||
database = NULL;
|
||||
CHECK(sql(path, "DROP TABLE feature_extract_batch_tasks;"
|
||||
"DROP TABLE raw_development_batch_tasks;"
|
||||
"UPDATE metadata SET value=23 WHERE key='schema_version';"));
|
||||
CHECK(setenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V24", "1", 1) == 0);
|
||||
CHECK(lardon3d_project_db_open(path, &database, error) != LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(database == NULL);
|
||||
CHECK(unsetenv("LARDON3D_TEST_PROJECT_DB_FAIL_MIGRATION_V24") == 0);
|
||||
CHECK(sqlite3_open(path, &raw) == SQLITE_OK);
|
||||
CHECK(sqlite3_prepare_v2(raw,
|
||||
"SELECT value FROM metadata WHERE key='schema_version'", -1,
|
||||
&query, NULL) == SQLITE_OK);
|
||||
CHECK(sqlite3_step(query) == SQLITE_ROW && sqlite3_column_int(query, 0) == 23);
|
||||
sqlite3_finalize(query);
|
||||
CHECK(sqlite3_close(raw) == SQLITE_OK);
|
||||
migration_result = lardon3d_project_db_open(path, &database, error);
|
||||
if (migration_result != LARDON3D_PROJECT_DB_OK)
|
||||
fprintf(stderr, "v24 retry result=%d error=%s\n", (int)migration_result, error);
|
||||
CHECK(migration_result == LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_schema_version(database) ==
|
||||
LARDON3D_PROJECT_DB_SCHEMA_VERSION);
|
||||
Lardon3DProjectDbSelectedExecutionItem item = {
|
||||
|
|
@ -148,6 +187,85 @@ static bool run(void) {
|
|||
&execution) == LARDON3D_PROJECT_DB_OK &&
|
||||
execution.execution_id == execution_id &&
|
||||
execution.stage == LARDON3D_SELECTED_EXECUTION_READY);
|
||||
|
||||
Lardon3DTaskDurableSnapshot batch_snapshot = {
|
||||
.id = 3,
|
||||
.estimate = {.memory_fixed_bytes = 64U * 1024U * 1024U,
|
||||
.memory_bytes_per_item = 512U * 1024U * 1024U,
|
||||
.minimum_batch_size = 1,
|
||||
.maximum_batch_size = 12,
|
||||
.desired_cpu_threads = 12,
|
||||
.desired_io_slots = 1,
|
||||
.task_class = LARDON3D_RESOURCE_TASK_CPU},
|
||||
.saved_state = TASK_PENDING,
|
||||
.recovery_state = TASK_PENDING,
|
||||
};
|
||||
(void)snprintf(batch_snapshot.name, sizeof(batch_snapshot.name), "Feature batch");
|
||||
Lardon3DProjectDbFeatureExtractBatchTask batch = {
|
||||
.task_id = 3,
|
||||
.selected_execution_id = execution_id,
|
||||
.next_item_index = 2,
|
||||
.extractor_version = 1,
|
||||
.max_features = 512,
|
||||
.pyramid_levels = 4,
|
||||
.fast_threshold = 10,
|
||||
.parameter_fingerprint = {7},
|
||||
};
|
||||
(void)snprintf(batch.extractor_kind, sizeof(batch.extractor_kind), "orb");
|
||||
/* An invalid new cursor must roll back both generic and typed state; v25
|
||||
cannot publish a Task whose durable prefix exceeds its immutable domain. */
|
||||
CHECK(lardon3d_project_db_record_feature_extract_batch_task(
|
||||
database, &batch_snapshot, "features.extract.batch", 1, NULL,
|
||||
&batch, 20) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||
Lardon3DProjectDbTask absent_batch_task;
|
||||
CHECK(lardon3d_project_db_load_task(database, 3, &absent_batch_task) ==
|
||||
LARDON3D_PROJECT_DB_NOT_FOUND);
|
||||
|
||||
batch.next_item_index = 0;
|
||||
CHECK(lardon3d_project_db_record_feature_extract_batch_task(
|
||||
database, &batch_snapshot, "features.extract.batch", 1, NULL,
|
||||
&batch, 21) == LARDON3D_PROJECT_DB_OK);
|
||||
Lardon3DProjectDbFeatureExtractBatchTask loaded_batch;
|
||||
CHECK(lardon3d_project_db_load_feature_extract_batch_task(database, 3,
|
||||
&loaded_batch) ==
|
||||
LARDON3D_PROJECT_DB_OK &&
|
||||
loaded_batch.selected_execution_id == execution_id &&
|
||||
loaded_batch.next_item_index == 0 &&
|
||||
memcmp(loaded_batch.parameter_fingerprint,
|
||||
batch.parameter_fingerprint, 32) == 0);
|
||||
CHECK(lardon3d_project_db_advance_feature_extract_batch_task(database, 3, 0,
|
||||
1) ==
|
||||
LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||
|
||||
unsigned char source_hash[32] = {0};
|
||||
unsigned char feature_hash[32] = {9};
|
||||
char feature_path[LARDON3D_PROJECT_DB_PATH_CAPACITY];
|
||||
feature_asset_path(feature_hash, feature_path);
|
||||
Lardon3DProjectDbFeatureSet feature_set;
|
||||
CHECK(lardon3d_project_db_register_feature_set(
|
||||
database, 1, "orb", 1, batch.parameter_fingerprint, source_hash,
|
||||
1, 1, 32, feature_hash, feature_path, 32,
|
||||
LARDON3D_DB_FEATURE_ASSET_DURABLE, 3, 22, &feature_set) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
CHECK(lardon3d_project_db_advance_feature_extract_batch_task(database, 3, 0,
|
||||
1) ==
|
||||
LARDON3D_PROJECT_DB_OK &&
|
||||
lardon3d_project_db_advance_feature_extract_batch_task(database, 3, 0,
|
||||
1) ==
|
||||
LARDON3D_PROJECT_DB_OK);
|
||||
/* A lagging generic checkpoint may repeat cursor zero, but the typed prefix
|
||||
remains at one. Changing any immutable ORB field is a durable conflict. */
|
||||
CHECK(lardon3d_project_db_record_feature_extract_batch_task(
|
||||
database, &batch_snapshot, "features.extract.batch", 1, NULL,
|
||||
&batch, 23) == LARDON3D_PROJECT_DB_OK &&
|
||||
lardon3d_project_db_load_feature_extract_batch_task(database, 3,
|
||||
&loaded_batch) ==
|
||||
LARDON3D_PROJECT_DB_OK &&
|
||||
loaded_batch.next_item_index == 1);
|
||||
batch.max_features++;
|
||||
CHECK(lardon3d_project_db_record_feature_extract_batch_task(
|
||||
database, &batch_snapshot, "features.extract.batch", 1, NULL,
|
||||
&batch, 24) == LARDON3D_PROJECT_DB_CONSTRAINT);
|
||||
lardon3d_project_db_close(database);
|
||||
database = NULL;
|
||||
|
||||
|
|
|
|||
|
|
@ -87,8 +87,10 @@ test_stage_matrix(void)
|
|||
} cases[] = {
|
||||
{"acquisition_campaign.run", LARDON3D_TUI_STAGE_ACQUISITION},
|
||||
{"raw.develop", LARDON3D_TUI_STAGE_RAW},
|
||||
{"raw.develop.batch", LARDON3D_TUI_STAGE_RAW},
|
||||
{"photo_quality.triage", LARDON3D_TUI_STAGE_QUALITY},
|
||||
{"features.extract", LARDON3D_TUI_STAGE_FEATURES},
|
||||
{"features.extract.batch", LARDON3D_TUI_STAGE_FEATURES},
|
||||
{"features.extract.sift", LARDON3D_TUI_STAGE_FEATURES},
|
||||
{"features.extract.rootsift", LARDON3D_TUI_STAGE_FEATURES},
|
||||
{"visual_index.update", LARDON3D_TUI_STAGE_VISUAL_INDEX},
|
||||
|
|
|
|||
Loading…
Reference in a new issue