tracks: freeze real S21 track builder proof
This commit is contained in:
parent
b84f860d86
commit
f13512ad04
13 changed files with 1169 additions and 337 deletions
|
|
@ -63,6 +63,15 @@ probes across 19 modified/new public headers, ABI and production-seam checks,
|
||||||
retained-manifest verification and diff validation, with zero blocking findings.
|
retained-manifest verification and diff validation, with zero blocking findings.
|
||||||
Do not reopen this boundary or infer a new scientific policy from the freeze.
|
Do not reopen this boundary or infer a new scientific policy from the freeze.
|
||||||
|
|
||||||
|
The canonical review checkpoint is tag `global-maintenance-2026-09-01` at
|
||||||
|
commit `b84f860d868c66d9ee84b85ceb1bc6480b95aca5`; its detailed evidence is
|
||||||
|
[`docs/architecture/global_maintenance_audit.md`](docs/architecture/global_maintenance_audit.md).
|
||||||
|
Future reviews are delta-based from this checkpoint: begin with
|
||||||
|
`git diff global-maintenance-2026-09-01...HEAD` and examine changed files,
|
||||||
|
their directly affected contracts, tests, documentation, and crossed dependency
|
||||||
|
boundaries. Unchanged PASS/FROZEN systems inherit this evidence and are reopened
|
||||||
|
only by concrete evidence; do not repeat a global A-to-Z audit.
|
||||||
|
|
||||||
When a ticket declares `NO_NEW_SUBSYSTEM`, do not introduce an unrelated:
|
When a ticket declares `NO_NEW_SUBSYSTEM`, do not introduce an unrelated:
|
||||||
|
|
||||||
- Task Runtime;
|
- Task Runtime;
|
||||||
|
|
|
||||||
|
|
@ -295,14 +295,56 @@ a complete Track Set with the exact non-empty scope.
|
||||||
|
|
||||||
## Resource model
|
## Resource model
|
||||||
|
|
||||||
**DECISION:** use a full sparse in-memory graph with deterministic edge sort,
|
The scientific choice remains **FROZEN**: a full sparse graph and deterministic
|
||||||
not a dense matrix, SQLite temporary graph or external merge in v1. This is
|
edge sort, never a dense matrix. The operational representation was reopened
|
||||||
the minimum complexity credible for the 16 GiB target and permits exhaustive
|
after the S21 admission failure without changing observation identity, edges,
|
||||||
canonical finalization. Input Match Files are read one at a time; they are
|
DSU components, conflict rejection, canonicalization, fingerprint or version.
|
||||||
never all resident.
|
Input Match Files are still read one at a time and are never all resident.
|
||||||
|
|
||||||
Approximate implementation-neutral accounting, including allocator/alignment
|
The Project path now constructs one private compact graph directly. It owns a
|
||||||
headroom:
|
Feature Set metadata projection, 16-byte nodes, 8-byte indexed raw edges and
|
||||||
|
one 16-byte-slot open-addressed identity index. The public C core remains an
|
||||||
|
adapter; Project no longer creates a `deque<Observation>`, pointer-edge vector,
|
||||||
|
second observation/map/edge transport and then another public-core table. DSU
|
||||||
|
uses a 32-bit parent array and byte ranks. Component grouping and accepted
|
||||||
|
Track ranges are flat arrays, not `vector<vector<...>>` ownership forests.
|
||||||
|
|
||||||
|
The RAM-only path is valid and is the current production path. No external
|
||||||
|
merge or scratch lease is used, no temporary graph state is authoritative or
|
||||||
|
stored in the DB, and no user flag selects a representation. Consequently no
|
||||||
|
new Governor/SSD wrapper is justified by this change.
|
||||||
|
|
||||||
|
The Task admission estimator is checked integer arithmetic over retained
|
||||||
|
capacities. For `N = 2 × E_raw` (the safe endpoint upper bound), `S` the next
|
||||||
|
power of two at least `1.5 × N + 1`, and `F` the number of distinct Feature
|
||||||
|
Sets resolved from the exact GVR scope, and `M_max` the largest parent Match
|
||||||
|
Result `match_count` in that scope, it reserves:
|
||||||
|
|
||||||
|
`4 MiB + 101 × N + 8 × E_raw + 16 × S + 640 × F + 12 × M_max` bytes.
|
||||||
|
|
||||||
|
The `101 × N` term is documented code accounting: retained compact nodes (16),
|
||||||
|
DSU/group/conflict scratch (17), flat canonical output (24), and worst-case
|
||||||
|
simultaneous per-Track publication serialization (44): stored observation
|
||||||
|
capacity (16), vector objects/capacity (up to 12), and publication rows (16).
|
||||||
|
The other terms are actual reserved
|
||||||
|
edge capacity, actual power-of-two identity slot capacity, and a conservative
|
||||||
|
Feature Set projection/cache/hash allowance. The final term is the largest
|
||||||
|
live `Lardon3DMatchFileEntry` vector: Match Files are resolved sequentially,
|
||||||
|
so this is a peak rather than a sum, and `match_count` is intentionally not
|
||||||
|
approximated by the potentially much smaller `inlier_count`. Overflow or more than
|
||||||
|
`UINT32_MAX` possible nodes fails closed before Task creation.
|
||||||
|
|
||||||
|
The former Task formula
|
||||||
|
`(4 MiB + E×48 + 2E×160) × (2 or 8)` was a historical envelope whose
|
||||||
|
coefficients and multiplier did not name live allocations. It is not the
|
||||||
|
current estimator and must not be reinterpreted as such. At S21
|
||||||
|
`E_raw = 6,628,174`, it produced `19,546,898,688` bytes (18.204 GiB), exceeding
|
||||||
|
the `12,750,811,136`-byte host capacity after canonical reserve. The compact
|
||||||
|
formula with `F=0, M_max=0` produces `1,932,981,756` bytes; real admission adds
|
||||||
|
exactly `640 × F + 12 × M_max` and therefore remains derived from the exact scope rather than from
|
||||||
|
an invented fixed 10.48 GiB target.
|
||||||
|
|
||||||
|
Historical implementation-neutral guidance, retained only for comparison:
|
||||||
|
|
||||||
- observation key: 16–24 bytes;
|
- observation key: 16–24 bytes;
|
||||||
- unique edge: 16–32 bytes for two compact node references and sort metadata;
|
- unique edge: 16–32 bytes for two compact node references and sort metadata;
|
||||||
|
|
@ -380,6 +422,13 @@ unit. It replays the explicit scope after recovery and pauses/cancels only at
|
||||||
safe boundaries; publication occurs only after complete canonical output and
|
safe boundaries; publication occurs only after complete canonical output and
|
||||||
final scope validation. A cancelled or failed run publishes no incomplete set.
|
final scope validation. A cancelled or failed run publishes no incomplete set.
|
||||||
|
|
||||||
|
The Task supplies an internal checkpoint callback to the Project adapter after
|
||||||
|
each complete GVR has been read and incorporated. It never preempts Match File
|
||||||
|
reading, DSU/canonicalization, final revalidation or atomic publication. A
|
||||||
|
pause/cancel therefore discards only reconstructible, non-authoritative RAM;
|
||||||
|
restart replays the same explicit durable scope and exact-identity lookup
|
||||||
|
reuses a publication that committed before a crash.
|
||||||
|
|
||||||
## Resource Governor integration (Gate D)
|
## Resource Governor integration (Gate D)
|
||||||
|
|
||||||
The Task estimates bounded graph memory and CPU, obtains reservations before
|
The Task estimates bounded graph memory and CPU, obtains reservations before
|
||||||
|
|
@ -511,14 +560,17 @@ internal SHA-256 helper. Its fingerprint vector remains
|
||||||
|
|
||||||
The historical isolated Gate B 1M benchmark's `973752 KiB` is a process
|
The historical isolated Gate B 1M benchmark's `973752 KiB` is a process
|
||||||
high-water RSS measurement; it is not the Gate C/D project integration
|
high-water RSS measurement; it is not the Gate C/D project integration
|
||||||
measurement. The
|
measurement and predates the compact Project ownership change. The
|
||||||
synthetic caller retains 136,000,136 bytes of observations and 16,000,000
|
synthetic caller retains 136,000,136 bytes of observations and 16,000,000
|
||||||
bytes of pointer edges while the core retains its 136,000,136-byte canonical
|
bytes of pointer edges while the core retains its 136,000,136-byte canonical
|
||||||
metadata table, 16,000,000-byte normalized edges, DSU/component temporaries
|
metadata table, 16,000,000-byte normalized edges, DSU/component temporaries
|
||||||
and compact output. Compact edges contain node indices only; they do not copy
|
and compact output. Compact edges contain node indices only; they do not copy
|
||||||
136-byte metadata at either endpoint. The remaining high-water gap is
|
136-byte metadata at either endpoint. The remaining high-water gap is
|
||||||
primarily allocator retention plus hash-table buckets/nodes and vector
|
primarily allocator retention plus hash-table buckets/nodes and vector
|
||||||
capacity. No core duplication bug was found or changed in the closure audit.
|
capacity. Those figures characterize the historical public-adapter benchmark,
|
||||||
|
not the current direct Project path; the operational reopening removed the
|
||||||
|
identified Project-to-core duplicate transport while preserving that public
|
||||||
|
adapter for ABI compatibility.
|
||||||
|
|
||||||
The targeted corpus explicitly covers reversed endpoints, 100 identical
|
The targeted corpus explicitly covers reversed endpoints, 100 identical
|
||||||
edges, contradictory metadata, fingerprint/dimension/version conflicts,
|
edges, contradictory metadata, fingerprint/dimension/version conflicts,
|
||||||
|
|
@ -542,7 +594,8 @@ Only Feature Set metadata is loaded—descriptor payloads are never read. A
|
||||||
small per-build Feature Set cache is permitted, while Match File buffers are
|
small per-build Feature Set cache is permitted, while Match File buffers are
|
||||||
released immediately after edge extraction.
|
released immediately after edge extraction.
|
||||||
|
|
||||||
The complete resolved graph is passed unchanged to the Gate B core. Before the
|
The complete resolved evidence is inserted directly into the shared private
|
||||||
|
compact core representation. Before the
|
||||||
frozen `lardon3d_project_db_create_track_set()` publication transaction, the
|
frozen `lardon3d_project_db_create_track_set()` publication transaction, the
|
||||||
same explicit GVR IDs, status, selector, parent chain and scope count/hash are
|
same explicit GVR IDs, status, selector, parent chain and scope count/hash are
|
||||||
revalidated. Any missing, rejected, mismatched or corrupt selected input aborts
|
revalidated. Any missing, rejected, mismatched or corrupt selected input aborts
|
||||||
|
|
@ -653,3 +706,21 @@ persistent shell that launched direct runs; it was not Track Builder memory.
|
||||||
A normal 250k hardware run completed in 0.368 seconds with 33,228 KiB peak RSS,
|
A normal 250k hardware run completed in 0.368 seconds with 33,228 KiB peak RSS,
|
||||||
MemAvailable loss of 14,108 KiB, no swap delta and zero PSI averages. Gate D
|
MemAvailable loss of 14,108 KiB, no swap delta and zero PSI averages. Gate D
|
||||||
is **PASS**; Gate E final validation and freeze are complete.
|
is **PASS**; Gate E final validation and freeze are complete.
|
||||||
|
|
||||||
|
The final real S21 proof validates the compact operational path on the immutable
|
||||||
|
GV v3 source (Project DB SHA-256 unchanged before and after:
|
||||||
|
`56aa5ec37624b322e9f77a90b138cc7390ef817a9cec3bef7e4c87609fd2eeed`). The
|
||||||
|
fresh primary task 2837 completed at 100% and published Track Set 1 with
|
||||||
|
912,447 Tracks and 2,495,768 observations (length min/max/mean
|
||||||
|
2/42/2.7352470883240341), zero duplicates, zero conflicting-image Tracks and
|
||||||
|
the persisted canonical digest
|
||||||
|
`c30eba192627bf73eaf21ff30d81038d8cc6bbf36a69226f88cdc8c37f7d74a1`. Its
|
||||||
|
wall time was 3,316 seconds (1788262322 to 1788265638). Exact resume reused
|
||||||
|
that Track Set unchanged. In a separate reflink, pending Task 2835 was
|
||||||
|
terminated with `SIGTERM` before publication (zero published Track Sets) and
|
||||||
|
then completed at 100% through the corrected runner, producing the identical
|
||||||
|
Track Set/counts/digest. RSS samples remained stable at approximately 845 MiB
|
||||||
|
and 862 MiB, minimum observed MemAvailable was about 8 GiB, no swap delta was
|
||||||
|
observed, and Governor admission passed. No scratch or scratch lease was used.
|
||||||
|
No Feature, Candidate Pair, Matcher or GVR work was replayed or created, and
|
||||||
|
Sparse SfM/Dense remained at zero.
|
||||||
|
|
|
||||||
|
|
@ -504,6 +504,16 @@ ressources, la persistance et l'observation TUI. Son état et ses preuves
|
||||||
consolidées sont consignés dans le
|
consolidées sont consignés dans le
|
||||||
[registre canonique de maintenance](../architecture/global_maintenance_audit.md).
|
[registre canonique de maintenance](../architecture/global_maintenance_audit.md).
|
||||||
|
|
||||||
|
Le checkpoint canonique de revue est le tag
|
||||||
|
`global-maintenance-2026-09-01`, au commit
|
||||||
|
`b84f860d868c66d9ee84b85ceb1bc6480b95aca5`. Les revues futures sont
|
||||||
|
strictement delta-based depuis ce point : `git diff
|
||||||
|
global-maintenance-2026-09-01...HEAD`, puis examen des fichiers modifiés, des
|
||||||
|
contrats, tests et documents directement affectés, et des frontières de
|
||||||
|
dépendances traversées. Les systèmes PASS/FROZEN inchangés héritent de la preuve
|
||||||
|
du [registre de maintenance](../architecture/global_maintenance_audit.md) et ne
|
||||||
|
sont rouverts que sur preuve concrète ; ne pas répéter un audit global A-à-Z.
|
||||||
|
|
||||||
Les résultats déjà réglés sont :
|
Les résultats déjà réglés sont :
|
||||||
|
|
||||||
- Project DB v23 ajoute neuf relations optiques sans backfill ni inférence ;
|
- Project DB v23 ajoute neuf relations optiques sans backfill ni inférence ;
|
||||||
|
|
@ -543,6 +553,41 @@ production, le SHA du manifest GV retenu et le diff-check. La gate de
|
||||||
maintenance est donc fermée ; la poursuite réelle depuis Tracks devient la
|
maintenance est donc fermée ; la poursuite réelle depuis Tracks devient la
|
||||||
prochaine tranche séparée, sans avoir été exécutée par cette synchronisation.
|
prochaine tranche séparée, sans avoir été exécutée par cette synchronisation.
|
||||||
|
|
||||||
|
## REAL S21 TRACKS — PASS / FROZEN
|
||||||
|
|
||||||
|
`REAL_S21_TRACKS=PASS/FROZEN`. La preuve part du projet GV v3 immuable
|
||||||
|
`/home/fy59/Documents/Lardon/.real-pre-sfm-2026-08-31/s21-gv-v3`, dont le
|
||||||
|
SHA-256 de Project DB est, avant et après les exécutions,
|
||||||
|
`56aa5ec37624b322e9f77a90b138cc7390ef817a9cec3bef7e4c87609fd2eeed`.
|
||||||
|
Les reflinks de preuve conservent exactement 2 826 Feature Sets, 172 741
|
||||||
|
Candidate Pairs/Match Results, 172 275 parents GVR (24 065
|
||||||
|
`GEOMETRIC_VERIFIED`, 148 210 `GEOMETRIC_REJECTED`). Aucun Feature, Candidate
|
||||||
|
Pair, Matcher ou GVR n'a été rejoué ni créé ; le source reste intègre. Sparse
|
||||||
|
SfM et Dense restent à zéro.
|
||||||
|
|
||||||
|
Le rejet initial de la Task `track_builder.run` 2835 à 0 % est conservé comme
|
||||||
|
constat historique : son enveloppe alors utilisée valait 19 546 898 688 octets
|
||||||
|
(18,204 Gio), au-delà des 12 750 811 136 octets (11,875 Gio) admis après
|
||||||
|
réserve. Ce n'est pas le modèle opérationnel validé. Le modèle compact actuel,
|
||||||
|
qui réserve les capacités vivantes du scope et le pic du Match File parent,
|
||||||
|
est admis par le Governor ; aucun scratch ni lease scratch n'a été utilisé.
|
||||||
|
|
||||||
|
La Task fraîche primaire 2837 termine `COMPLETE` à 100 % et publie le seul
|
||||||
|
Track Set 1 : 912 447 Tracks, 2 495 768 observations, longueurs min/max/moyenne
|
||||||
|
2/42/2,7352470883240341, zéro doublon et zéro Track à images conflictuelles.
|
||||||
|
Son digest canonique persistant est
|
||||||
|
`c30eba192627bf73eaf21ff30d81038d8cc6bbf36a69226f88cdc8c37f7d74a1`.
|
||||||
|
La reprise exacte suivante réutilise Track Set 1 (`task_track_id=0`) sans
|
||||||
|
modifier ces Tracks. Le run complet propre de la Task 2837 dure 3 316 s
|
||||||
|
(1788262322 → 1788265638).
|
||||||
|
|
||||||
|
La preuve de récupération est distincte : la Task 2835 est créée pendante sur
|
||||||
|
un reflink, interrompue par `SIGTERM` sans publication, puis reprise par le
|
||||||
|
runner corrigé. Elle termine `COMPLETE` à 100 % et retrouve le même Track Set
|
||||||
|
1, les mêmes comptes et le même digest ; le scénario interrompu avait publié
|
||||||
|
zéro Track Set. Cette preuve valide la reprise opérationnelle sans rouvrir le
|
||||||
|
contrat scientifique Tracks gelé.
|
||||||
|
|
||||||
## NEAR TERM
|
## NEAR TERM
|
||||||
|
|
||||||
### Scratch SSD externe et swap optionnel
|
### Scratch SSD externe et swap optionnel
|
||||||
|
|
|
||||||
|
|
@ -38,6 +38,7 @@ typedef enum {
|
||||||
LARDON3D_TRACK_BUILDER_PROJECT_DATABASE_ERROR,
|
LARDON3D_TRACK_BUILDER_PROJECT_DATABASE_ERROR,
|
||||||
LARDON3D_TRACK_BUILDER_PROJECT_OUT_OF_MEMORY,
|
LARDON3D_TRACK_BUILDER_PROJECT_OUT_OF_MEMORY,
|
||||||
LARDON3D_TRACK_BUILDER_PROJECT_CORE_ERROR,
|
LARDON3D_TRACK_BUILDER_PROJECT_CORE_ERROR,
|
||||||
|
LARDON3D_TRACK_BUILDER_PROJECT_INTERRUPTED,
|
||||||
} Lardon3DTrackBuilderProjectStatus;
|
} Lardon3DTrackBuilderProjectStatus;
|
||||||
|
|
||||||
Lardon3DTrackBuilderProjectStatus lardon3d_track_builder_build_project(
|
Lardon3DTrackBuilderProjectStatus lardon3d_track_builder_build_project(
|
||||||
|
|
|
||||||
|
|
@ -44,6 +44,18 @@ bool lardon3d_track_builder_task_reconstruct(
|
||||||
const Lardon3DTaskDurableSnapshot *snapshot, void *context,
|
const Lardon3DTaskDurableSnapshot *snapshot, void *context,
|
||||||
Lardon3DTaskKindBinding *binding);
|
Lardon3DTaskKindBinding *binding);
|
||||||
|
|
||||||
|
/* Checked operational envelope for the compact RAM-only implementation.
|
||||||
|
* This is admission accounting, not Builder identity or scientific policy. */
|
||||||
|
bool lardon3d_track_builder_task_memory_estimate(
|
||||||
|
uint64_t raw_edge_count, uint64_t feature_set_count,
|
||||||
|
uint64_t *memory_bytes);
|
||||||
|
|
||||||
|
/* Additive operational estimator used when the exact scope scan has resolved
|
||||||
|
* the largest parent Match Result. The legacy E/F estimator remains ABI-stable. */
|
||||||
|
bool lardon3d_track_builder_task_memory_estimate_with_match_peak(
|
||||||
|
uint64_t raw_edge_count, uint64_t feature_set_count,
|
||||||
|
uint64_t max_match_count, uint64_t *memory_bytes);
|
||||||
|
|
||||||
#ifdef __cplusplus
|
#ifdef __cplusplus
|
||||||
}
|
}
|
||||||
#endif
|
#endif
|
||||||
|
|
|
||||||
|
|
@ -1,37 +1,26 @@
|
||||||
|
#include "track_builder_internal.hpp"
|
||||||
|
|
||||||
#include <algorithm>
|
#include <algorithm>
|
||||||
#include <cstring>
|
#include <cstring>
|
||||||
#include <limits>
|
#include <limits>
|
||||||
#include <new>
|
#include <new>
|
||||||
#include <openssl/evp.h>
|
#include <openssl/evp.h>
|
||||||
#include <unordered_map>
|
#include <stdexcept>
|
||||||
#include <vector>
|
#include <vector>
|
||||||
|
|
||||||
extern "C" {
|
namespace tb = lardon3d::track_builder_internal;
|
||||||
#include <lardon3d/track_builder.h>
|
using Observation = Lardon3DTrackBuilderObservation;
|
||||||
}
|
|
||||||
|
static_assert(sizeof(tb::Node) == 16);
|
||||||
|
static_assert(sizeof(tb::Edge) == 8);
|
||||||
|
static_assert(sizeof(tb::IdentitySlot) == 16);
|
||||||
|
static_assert(sizeof(tb::Membership) == 16);
|
||||||
|
|
||||||
namespace {
|
namespace {
|
||||||
using Observation = Lardon3DTrackBuilderObservation;
|
bool meta_equal(const tb::FeatureMetadata &a, const tb::FeatureMetadata &b,
|
||||||
using Edge = Lardon3DTrackBuilderEdge;
|
bool include_identity) {
|
||||||
|
return (!include_identity || (a.feature_set_id == b.feature_set_id &&
|
||||||
struct CompactEdge {
|
a.image_id == b.image_id)) &&
|
||||||
size_t low;
|
|
||||||
size_t high;
|
|
||||||
};
|
|
||||||
|
|
||||||
bool identity_less(const Observation &a, const Observation &b) {
|
|
||||||
return a.feature_set_id < b.feature_set_id ||
|
|
||||||
(a.feature_set_id == b.feature_set_id &&
|
|
||||||
a.feature_index < b.feature_index);
|
|
||||||
}
|
|
||||||
|
|
||||||
bool identity_equal(const Observation &a, const Observation &b) {
|
|
||||||
return a.feature_set_id == b.feature_set_id &&
|
|
||||||
a.feature_index == b.feature_index;
|
|
||||||
}
|
|
||||||
|
|
||||||
bool metadata_equal(const Observation &a, const Observation &b) {
|
|
||||||
return a.image_id == b.image_id &&
|
|
||||||
a.extractor_version == b.extractor_version &&
|
a.extractor_version == b.extractor_version &&
|
||||||
a.descriptor_type == b.descriptor_type &&
|
a.descriptor_type == b.descriptor_type &&
|
||||||
a.descriptor_dimension == b.descriptor_dimension &&
|
a.descriptor_dimension == b.descriptor_dimension &&
|
||||||
|
|
@ -39,77 +28,210 @@ bool metadata_equal(const Observation &a, const Observation &b) {
|
||||||
LARDON3D_TRACK_BUILDER_KIND_CAPACITY) == 0 &&
|
LARDON3D_TRACK_BUILDER_KIND_CAPACITY) == 0 &&
|
||||||
std::memcmp(a.parameter_fingerprint, b.parameter_fingerprint, 32) == 0;
|
std::memcmp(a.parameter_fingerprint, b.parameter_fingerprint, 32) == 0;
|
||||||
}
|
}
|
||||||
|
bool meta_valid(const tb::FeatureMetadata &m) {
|
||||||
bool observation_valid(const Observation &observation) {
|
return m.feature_set_id != 0 && m.image_id != 0 &&
|
||||||
return observation.feature_set_id != 0 && observation.image_id != 0 &&
|
std::memchr(m.extractor_kind, '\0', sizeof(m.extractor_kind));
|
||||||
std::memchr(observation.extractor_kind, '\0',
|
}
|
||||||
LARDON3D_TRACK_BUILDER_KIND_CAPACITY) != nullptr;
|
uint64_t hash_key(uint64_t set, uint32_t index) {
|
||||||
|
uint64_t v = set ^ (static_cast<uint64_t>(index) + 0x9e3779b97f4a7c15ULL +
|
||||||
|
(set << 6U) + (set >> 2U));
|
||||||
|
v ^= v >> 30U; v *= 0xbf58476d1ce4e5b9ULL;
|
||||||
|
v ^= v >> 27U; v *= 0x94d049bb133111ebULL;
|
||||||
|
return v ^ (v >> 31U);
|
||||||
|
}
|
||||||
|
size_t table_capacity(uint64_t nodes) {
|
||||||
|
if (nodes == 0) return 0;
|
||||||
|
if (nodes > std::numeric_limits<size_t>::max() / 2U) throw std::bad_alloc();
|
||||||
|
const size_t required = static_cast<size_t>(nodes + nodes / 2U + 1U);
|
||||||
|
size_t capacity = 8;
|
||||||
|
while (capacity < required) {
|
||||||
|
if (capacity > std::numeric_limits<size_t>::max() / 2U) throw std::bad_alloc();
|
||||||
|
capacity *= 2U;
|
||||||
|
}
|
||||||
|
return capacity;
|
||||||
}
|
}
|
||||||
|
|
||||||
struct Dsu {
|
struct Dsu {
|
||||||
std::vector<size_t> parent;
|
std::vector<uint32_t> parent;
|
||||||
std::vector<unsigned char> rank;
|
std::vector<unsigned char> rank;
|
||||||
explicit Dsu(size_t size) : parent(size), rank(size, 0) {
|
explicit Dsu(size_t n) : parent(n), rank(n, 0) {
|
||||||
for (size_t i = 0; i < size; ++i) parent[i] = i;
|
for (size_t i = 0; i < n; ++i) parent[i] = static_cast<uint32_t>(i);
|
||||||
}
|
}
|
||||||
size_t root(size_t value) {
|
uint32_t root(uint32_t v) {
|
||||||
while (parent[value] != value) {
|
while (parent[v] != v) { parent[v] = parent[parent[v]]; v = parent[v]; }
|
||||||
parent[value] = parent[parent[value]];
|
return v;
|
||||||
value = parent[value];
|
|
||||||
}
|
}
|
||||||
return value;
|
void unite(uint32_t a, uint32_t b) {
|
||||||
}
|
a = root(a); b = root(b); if (a == b) return;
|
||||||
void unite(size_t a, size_t b) {
|
|
||||||
a = root(a); b = root(b);
|
|
||||||
if (a == b) return;
|
|
||||||
if (rank[a] < rank[b]) std::swap(a, b);
|
if (rank[a] < rank[b]) std::swap(a, b);
|
||||||
parent[b] = a;
|
parent[b] = a; if (rank[a] == rank[b]) ++rank[a];
|
||||||
if (rank[a] == rank[b]) ++rank[a];
|
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
tb::FeatureMetadata project(const Observation &o) {
|
||||||
struct IdentityHash {
|
tb::FeatureMetadata m{};
|
||||||
size_t operator()(const std::pair<uint64_t, uint32_t> &key) const noexcept {
|
m.feature_set_id = o.feature_set_id; m.image_id = o.image_id;
|
||||||
uint64_t value = key.first ^ (static_cast<uint64_t>(key.second) +
|
std::memcpy(m.extractor_kind, o.extractor_kind, sizeof(m.extractor_kind));
|
||||||
0x9e3779b97f4a7c15ULL + (key.first << 6U) +
|
m.extractor_version = o.extractor_version;
|
||||||
(key.first >> 2U));
|
std::memcpy(m.parameter_fingerprint, o.parameter_fingerprint, 32);
|
||||||
value ^= value >> 30U;
|
m.descriptor_type = o.descriptor_type; m.descriptor_dimension = o.descriptor_dimension;
|
||||||
value *= 0xbf58476d1ce4e5b9ULL;
|
return m;
|
||||||
value ^= value >> 27U;
|
|
||||||
return static_cast<size_t>(value ^ (value >> 31U));
|
|
||||||
}
|
}
|
||||||
};
|
|
||||||
|
|
||||||
std::pair<uint64_t, uint32_t> identity_key(const Observation &observation) {
|
|
||||||
return {observation.feature_set_id, observation.feature_index};
|
|
||||||
}
|
|
||||||
|
|
||||||
bool digest(const unsigned char *input, size_t size, unsigned char output[32]) {
|
bool digest(const unsigned char *input, size_t size, unsigned char output[32]) {
|
||||||
unsigned int output_size = 0;
|
unsigned int length = 0;
|
||||||
return EVP_Digest(input, size, output, &output_size, EVP_sha256(), nullptr) == 1 &&
|
return EVP_Digest(input, size, output, &length, EVP_sha256(), nullptr) == 1 &&
|
||||||
output_size == 32;
|
length == 32;
|
||||||
|
}
|
||||||
|
} // namespace
|
||||||
|
|
||||||
|
tb::CompactGraph::CompactGraph(uint64_t edge_hint) {
|
||||||
|
if (edge_hint > UINT32_MAX / 2U || edge_hint > std::numeric_limits<size_t>::max())
|
||||||
|
throw std::bad_alloc();
|
||||||
|
const uint64_t node_hint = edge_hint * 2U;
|
||||||
|
nodes_.reserve(static_cast<size_t>(node_hint));
|
||||||
|
edges_.reserve(static_cast<size_t>(edge_hint));
|
||||||
|
identity_.resize(table_capacity(node_hint));
|
||||||
}
|
}
|
||||||
|
|
||||||
} // namespace
|
uint32_t tb::CompactGraph::register_feature(const FeatureMetadata &m) {
|
||||||
|
if (!meta_valid(m)) throw std::invalid_argument("invalid feature metadata");
|
||||||
|
/* CONTRACT: CompactGraph validates metadata inside each connected component;
|
||||||
|
* it must not globally bind a feature_set_id. The public C ABI historically
|
||||||
|
* permits unrelated observations from that set to carry divergent metadata,
|
||||||
|
* while the Project adapter enforces immutable Feature Set metadata in its
|
||||||
|
* own cache before registering exactly one entry per set. */
|
||||||
|
if (metadata_.size() == UINT32_MAX) throw std::bad_alloc();
|
||||||
|
metadata_.push_back(m);
|
||||||
|
return static_cast<uint32_t>(metadata_.size() - 1U);
|
||||||
|
}
|
||||||
|
|
||||||
|
void tb::CompactGraph::insert_identity(const Node &node, uint32_t index) {
|
||||||
|
const size_t mask = identity_.size() - 1U;
|
||||||
|
size_t slot = static_cast<size_t>(hash_key(node.feature_set_id, node.feature_index)) & mask;
|
||||||
|
while (identity_[slot].node_plus_one) slot = (slot + 1U) & mask;
|
||||||
|
identity_[slot] = {node.feature_set_id, node.feature_index, index + 1U};
|
||||||
|
++identity_size_;
|
||||||
|
}
|
||||||
|
|
||||||
|
uint32_t tb::CompactGraph::resolve_node(uint32_t metadata_index,
|
||||||
|
uint32_t feature_index) {
|
||||||
|
if (metadata_index >= metadata_.size() || identity_.empty())
|
||||||
|
throw std::invalid_argument("invalid metadata index");
|
||||||
|
const uint64_t set = metadata_[metadata_index].feature_set_id;
|
||||||
|
const size_t mask = identity_.size() - 1U;
|
||||||
|
size_t slot = static_cast<size_t>(hash_key(set, feature_index)) & mask;
|
||||||
|
while (identity_[slot].node_plus_one) {
|
||||||
|
const IdentitySlot &found = identity_[slot];
|
||||||
|
if (found.feature_set_id == set && found.feature_index == feature_index) {
|
||||||
|
const uint32_t index = found.node_plus_one - 1U;
|
||||||
|
if (nodes_[index].metadata_index != metadata_index)
|
||||||
|
throw std::invalid_argument("contradictory observation");
|
||||||
|
return index;
|
||||||
|
}
|
||||||
|
slot = (slot + 1U) & mask;
|
||||||
|
}
|
||||||
|
if (nodes_.size() == UINT32_MAX || identity_size_ == identity_.size())
|
||||||
|
throw std::bad_alloc();
|
||||||
|
nodes_.push_back({set, feature_index, metadata_index});
|
||||||
|
const uint32_t index = static_cast<uint32_t>(nodes_.size() - 1U);
|
||||||
|
insert_identity(nodes_.back(), index);
|
||||||
|
return index;
|
||||||
|
}
|
||||||
|
|
||||||
|
bool tb::CompactGraph::add_edge(uint32_t am, uint32_t ai, uint32_t bm,
|
||||||
|
uint32_t bi) {
|
||||||
|
const uint32_t a = resolve_node(am, ai), b = resolve_node(bm, bi);
|
||||||
|
if (a == b) return false;
|
||||||
|
edges_.push_back(a < b ? Edge{a, b} : Edge{b, a});
|
||||||
|
++raw_edge_count_;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
const tb::FeatureMetadata &tb::CompactGraph::metadata(uint32_t i) const {
|
||||||
|
return metadata_.at(i);
|
||||||
|
}
|
||||||
|
uint64_t tb::CompactGraph::node_image(uint32_t i) const {
|
||||||
|
return metadata_.at(nodes_.at(i).metadata_index).image_id;
|
||||||
|
}
|
||||||
|
|
||||||
|
Lardon3DTrackBuilderResult tb::CompactGraph::build(Output *output) {
|
||||||
|
if (!output) return LARDON3D_TRACK_BUILDER_INVALID_ARGUMENT;
|
||||||
|
output->memberships.clear(); output->tracks.clear();
|
||||||
|
try {
|
||||||
|
std::sort(edges_.begin(), edges_.end(), [](const Edge &a, const Edge &b) {
|
||||||
|
return a.low < b.low || (a.low == b.low && a.high < b.high);
|
||||||
|
});
|
||||||
|
edges_.erase(std::unique(edges_.begin(), edges_.end(), [](const Edge &a, const Edge &b) {
|
||||||
|
return a.low == b.low && a.high == b.high;
|
||||||
|
}), edges_.end());
|
||||||
|
Dsu dsu(nodes_.size());
|
||||||
|
for (const Edge &edge : edges_) dsu.unite(edge.low, edge.high);
|
||||||
|
|
||||||
|
/* INVARIANT: one flat grouping owns every node exactly once. DSU roots
|
||||||
|
* delimit complete components but never determine persisted order. */
|
||||||
|
std::vector<uint32_t> grouped(nodes_.size());
|
||||||
|
for (size_t i = 0; i < grouped.size(); ++i) grouped[i] = static_cast<uint32_t>(i);
|
||||||
|
std::sort(grouped.begin(), grouped.end(), [&](uint32_t a, uint32_t b) {
|
||||||
|
const uint32_t ra = dsu.root(a), rb = dsu.root(b);
|
||||||
|
if (ra != rb) return ra < rb;
|
||||||
|
const Node &x = nodes_[a], &y = nodes_[b];
|
||||||
|
return x.feature_set_id < y.feature_set_id ||
|
||||||
|
(x.feature_set_id == y.feature_set_id && x.feature_index < y.feature_index);
|
||||||
|
});
|
||||||
|
std::vector<TrackRange> accepted;
|
||||||
|
for (size_t begin = 0; begin < grouped.size();) {
|
||||||
|
size_t end = begin + 1U; const uint32_t root = dsu.root(grouped[begin]);
|
||||||
|
while (end < grouped.size() && dsu.root(grouped[end]) == root) ++end;
|
||||||
|
bool valid = end - begin >= 2U;
|
||||||
|
const FeatureMetadata &first = metadata_[nodes_[grouped[begin]].metadata_index];
|
||||||
|
std::vector<uint64_t> images; images.reserve(end - begin);
|
||||||
|
for (size_t i = begin; i < end; ++i) {
|
||||||
|
const FeatureMetadata &m = metadata_[nodes_[grouped[i]].metadata_index];
|
||||||
|
valid = valid && meta_equal(first, m, false); images.push_back(m.image_id);
|
||||||
|
}
|
||||||
|
std::sort(images.begin(), images.end());
|
||||||
|
valid = valid && std::adjacent_find(images.begin(), images.end()) == images.end();
|
||||||
|
if (valid) accepted.push_back({begin, end - begin});
|
||||||
|
begin = end;
|
||||||
|
}
|
||||||
|
auto less = [&](const TrackRange &a, const TrackRange &b) {
|
||||||
|
const size_t count = std::min(a.count, b.count);
|
||||||
|
for (size_t i = 0; i < count; ++i) {
|
||||||
|
const Node &x = nodes_[grouped[a.begin + i]], &y = nodes_[grouped[b.begin + i]];
|
||||||
|
if (x.feature_set_id != y.feature_set_id) return x.feature_set_id < y.feature_set_id;
|
||||||
|
if (x.feature_index != y.feature_index) return x.feature_index < y.feature_index;
|
||||||
|
}
|
||||||
|
return a.count < b.count;
|
||||||
|
};
|
||||||
|
std::sort(accepted.begin(), accepted.end(), less);
|
||||||
|
size_t count = 0; for (const TrackRange &range : accepted) count += range.count;
|
||||||
|
output->memberships.reserve(count); output->tracks.reserve(accepted.size());
|
||||||
|
for (const TrackRange &range : accepted) {
|
||||||
|
const size_t begin = output->memberships.size();
|
||||||
|
for (size_t i = 0; i < range.count; ++i) {
|
||||||
|
const uint32_t ni = grouped[range.begin + i]; const Node &n = nodes_[ni];
|
||||||
|
output->memberships.push_back({n.feature_set_id, n.feature_index, ni});
|
||||||
|
}
|
||||||
|
output->tracks.push_back({begin, range.count});
|
||||||
|
}
|
||||||
|
return LARDON3D_TRACK_BUILDER_OK;
|
||||||
|
} catch (const std::bad_alloc &) {
|
||||||
|
output->memberships.clear(); output->tracks.clear();
|
||||||
|
return LARDON3D_TRACK_BUILDER_OUT_OF_MEMORY;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
extern "C" bool lardon3d_track_builder_fingerprint_bytes(unsigned char bytes[48]) {
|
extern "C" bool lardon3d_track_builder_fingerprint_bytes(unsigned char bytes[48]) {
|
||||||
if (!bytes) return false;
|
if (!bytes) return false;
|
||||||
std::memset(bytes, 0, 48);
|
std::memset(bytes, 0, 48);
|
||||||
std::memcpy(bytes, "L3DTBFP1", 8);
|
std::memcpy(bytes, "L3DTBFP1", 8);
|
||||||
const uint32_t fields[] = {1,1,1,1,1,1,1,1,2,0};
|
const uint32_t fields[] = {1,1,1,1,1,1,1,1,2,0};
|
||||||
for (size_t field = 0; field < 10; ++field)
|
for (size_t f = 0; f < 10; ++f) for (size_t b = 0; b < 4; ++b)
|
||||||
for (size_t byte = 0; byte < 4; ++byte)
|
bytes[8 + f * 4 + b] = static_cast<unsigned char>(fields[f] >> (b * 8));
|
||||||
bytes[8 + field * 4 + byte] =
|
|
||||||
static_cast<unsigned char>(fields[field] >> (byte * 8));
|
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
extern "C" bool lardon3d_track_builder_fingerprint(unsigned char output[32]) {
|
||||||
extern "C" bool lardon3d_track_builder_fingerprint(unsigned char fingerprint[32]) {
|
unsigned char bytes[48]; return output && lardon3d_track_builder_fingerprint_bytes(bytes) &&
|
||||||
if (!fingerprint) return false;
|
digest(bytes, sizeof(bytes), output);
|
||||||
unsigned char bytes[48];
|
|
||||||
return lardon3d_track_builder_fingerprint_bytes(bytes) && digest(bytes, 48, fingerprint);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
extern "C" void lardon3d_track_builder_result_free(Lardon3DTrackBuilderResultSet *result) {
|
extern "C" void lardon3d_track_builder_result_free(Lardon3DTrackBuilderResultSet *result) {
|
||||||
if (!result) return;
|
if (!result) return;
|
||||||
if (result->tracks) {
|
if (result->tracks) {
|
||||||
|
|
@ -117,123 +239,96 @@ extern "C" void lardon3d_track_builder_result_free(Lardon3DTrackBuilderResultSet
|
||||||
delete[] result->tracks[i].observations;
|
delete[] result->tracks[i].observations;
|
||||||
delete[] result->tracks;
|
delete[] result->tracks;
|
||||||
}
|
}
|
||||||
result->tracks = nullptr;
|
result->tracks = nullptr; result->track_count = 0;
|
||||||
result->track_count = 0;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
extern "C" Lardon3DTrackBuilderResult lardon3d_track_builder_build(
|
extern "C" Lardon3DTrackBuilderResult lardon3d_track_builder_build(
|
||||||
const Observation *input_observations, size_t observation_count, const Edge *input_edges,
|
const Observation *observations, size_t observation_count,
|
||||||
size_t edge_count, Lardon3DTrackBuilderResultSet *result) {
|
const Lardon3DTrackBuilderEdge *edges, size_t edge_count,
|
||||||
if (!result || (observation_count != 0 && !input_observations) ||
|
Lardon3DTrackBuilderResultSet *result) {
|
||||||
(edge_count != 0 && !input_edges))
|
if (!result || (observation_count && !observations) || (edge_count && !edges))
|
||||||
return LARDON3D_TRACK_BUILDER_INVALID_ARGUMENT;
|
return LARDON3D_TRACK_BUILDER_INVALID_ARGUMENT;
|
||||||
lardon3d_track_builder_result_free(result);
|
lardon3d_track_builder_result_free(result);
|
||||||
try {
|
try {
|
||||||
if (edge_count > std::numeric_limits<size_t>::max() / 2)
|
tb::CompactGraph graph(edge_count);
|
||||||
return LARDON3D_TRACK_BUILDER_INVALID_ARGUMENT;
|
struct PublicIdentity {
|
||||||
std::vector<Observation> table;
|
uint64_t feature_set_id;
|
||||||
if (observation_count != 0)
|
uint32_t feature_index;
|
||||||
table.assign(input_observations, input_observations + observation_count);
|
tb::FeatureMetadata metadata;
|
||||||
for (const Observation &observation : table)
|
uint32_t metadata_index = 0;
|
||||||
if (!observation_valid(observation)) return LARDON3D_TRACK_BUILDER_CORRUPT_INPUT;
|
bool registered = false;
|
||||||
std::sort(table.begin(), table.end(), identity_less);
|
|
||||||
for (size_t i = 1; i < table.size(); ++i)
|
|
||||||
if (identity_equal(table[i - 1], table[i]) &&
|
|
||||||
!metadata_equal(table[i - 1], table[i]))
|
|
||||||
return LARDON3D_TRACK_BUILDER_CORRUPT_INPUT;
|
|
||||||
table.erase(std::unique(table.begin(), table.end(), identity_equal), table.end());
|
|
||||||
std::unordered_map<std::pair<uint64_t, uint32_t>, size_t, IdentityHash> table_indices;
|
|
||||||
table_indices.reserve(table.size());
|
|
||||||
for (size_t i = 0; i < table.size(); ++i) table_indices.emplace(identity_key(table[i]), i);
|
|
||||||
|
|
||||||
std::vector<CompactEdge> normalized;
|
|
||||||
normalized.reserve(edge_count);
|
|
||||||
for (size_t i = 0; i < edge_count; ++i) {
|
|
||||||
const Edge &edge = input_edges[i];
|
|
||||||
if (!edge.first || !edge.second || !observation_valid(*edge.first) ||
|
|
||||||
!observation_valid(*edge.second) || identity_equal(*edge.first, *edge.second))
|
|
||||||
return LARDON3D_TRACK_BUILDER_CORRUPT_INPUT;
|
|
||||||
auto first_it = table_indices.find(identity_key(*edge.first));
|
|
||||||
auto second_it = table_indices.find(identity_key(*edge.second));
|
|
||||||
if (first_it == table_indices.end() || second_it == table_indices.end() ||
|
|
||||||
first_it->second >= table.size() || second_it->second >= table.size() ||
|
|
||||||
!metadata_equal(table[first_it->second], *edge.first) ||
|
|
||||||
!metadata_equal(table[second_it->second], *edge.second))
|
|
||||||
return LARDON3D_TRACK_BUILDER_CORRUPT_INPUT;
|
|
||||||
if (first_it->second < second_it->second)
|
|
||||||
normalized.push_back({first_it->second, second_it->second});
|
|
||||||
else
|
|
||||||
normalized.push_back({second_it->second, first_it->second});
|
|
||||||
}
|
|
||||||
std::sort(normalized.begin(), normalized.end(), [](const CompactEdge &a,
|
|
||||||
const CompactEdge &b) {
|
|
||||||
return a.low < b.low || (a.low == b.low && a.high < b.high);
|
|
||||||
});
|
|
||||||
normalized.erase(std::unique(normalized.begin(), normalized.end(),
|
|
||||||
[](const CompactEdge &a, const CompactEdge &b) {
|
|
||||||
return a.low == b.low && a.high == b.high;
|
|
||||||
}),
|
|
||||||
normalized.end());
|
|
||||||
Dsu dsu(table.size());
|
|
||||||
for (const CompactEdge &edge : normalized) dsu.unite(edge.low, edge.high);
|
|
||||||
std::vector<std::vector<size_t>> components(table.size());
|
|
||||||
for (size_t i = 0; i < table.size(); ++i)
|
|
||||||
if (!normalized.empty()) components[dsu.root(i)].push_back(i);
|
|
||||||
std::vector<std::vector<size_t>> accepted;
|
|
||||||
for (auto &component : components) {
|
|
||||||
if (component.size() < 2) continue;
|
|
||||||
std::sort(component.begin(), component.end());
|
|
||||||
bool valid = true;
|
|
||||||
std::vector<uint64_t> images;
|
|
||||||
images.reserve(component.size());
|
|
||||||
const Observation &first = table[component.front()];
|
|
||||||
for (size_t node : component) {
|
|
||||||
const Observation &observation = table[node];
|
|
||||||
if (observation.image_id == 0 ||
|
|
||||||
(observation.extractor_version != first.extractor_version) ||
|
|
||||||
(observation.descriptor_type != first.descriptor_type) ||
|
|
||||||
(observation.descriptor_dimension != first.descriptor_dimension) ||
|
|
||||||
std::strncmp(observation.extractor_kind, first.extractor_kind,
|
|
||||||
LARDON3D_TRACK_BUILDER_KIND_CAPACITY) != 0 ||
|
|
||||||
std::memcmp(observation.parameter_fingerprint, first.parameter_fingerprint, 32) != 0)
|
|
||||||
valid = false;
|
|
||||||
images.push_back(observation.image_id);
|
|
||||||
}
|
|
||||||
std::sort(images.begin(), images.end());
|
|
||||||
for (size_t i = 1; i < images.size(); ++i)
|
|
||||||
if (images[i - 1] == images[i]) valid = false;
|
|
||||||
if (valid) accepted.push_back(component);
|
|
||||||
}
|
|
||||||
auto component_less = [&table](const std::vector<size_t> &a,
|
|
||||||
const std::vector<size_t> &b) {
|
|
||||||
size_t count = std::min(a.size(), b.size());
|
|
||||||
for (size_t i = 0; i < count; ++i) {
|
|
||||||
if (identity_less(table[a[i]], table[b[i]])) return true;
|
|
||||||
if (identity_less(table[b[i]], table[a[i]])) return false;
|
|
||||||
}
|
|
||||||
return a.size() < b.size();
|
|
||||||
};
|
};
|
||||||
std::sort(accepted.begin(), accepted.end(), component_less);
|
std::vector<PublicIdentity> input_identities;
|
||||||
if (!accepted.empty()) {
|
input_identities.reserve(observation_count);
|
||||||
result->tracks = new Lardon3DTrackBuilderTrack[accepted.size()]();
|
for (size_t i = 0; i < observation_count; ++i) {
|
||||||
result->track_count = accepted.size();
|
const tb::FeatureMetadata metadata = project(observations[i]);
|
||||||
for (size_t i = 0; i < accepted.size(); ++i) {
|
if (!meta_valid(metadata)) return LARDON3D_TRACK_BUILDER_CORRUPT_INPUT;
|
||||||
result->tracks[i].observation_count = accepted[i].size();
|
input_identities.push_back({observations[i].feature_set_id,
|
||||||
result->tracks[i].observations =
|
observations[i].feature_index, metadata});
|
||||||
new Lardon3DTrackBuilderTrackObservation[accepted[i].size()];
|
}
|
||||||
for (size_t j = 0; j < accepted[i].size(); ++j) {
|
std::sort(input_identities.begin(), input_identities.end(),
|
||||||
const Observation &source = table[accepted[i][j]];
|
[](const PublicIdentity &a, const PublicIdentity &b) {
|
||||||
result->tracks[i].observations[j] = {
|
return std::make_pair(a.feature_set_id, a.feature_index) <
|
||||||
source.feature_set_id, source.feature_index, source.image_id};
|
std::make_pair(b.feature_set_id, b.feature_index);
|
||||||
|
});
|
||||||
|
for (size_t i = 1; i < input_identities.size(); ++i) {
|
||||||
|
if (input_identities[i - 1].feature_set_id == input_identities[i].feature_set_id &&
|
||||||
|
input_identities[i - 1].feature_index == input_identities[i].feature_index &&
|
||||||
|
!meta_equal(input_identities[i - 1].metadata, input_identities[i].metadata, true))
|
||||||
|
return LARDON3D_TRACK_BUILDER_CORRUPT_INPUT;
|
||||||
|
}
|
||||||
|
auto find_identity = [&](uint64_t set, uint32_t index) -> PublicIdentity * {
|
||||||
|
auto found = std::lower_bound(
|
||||||
|
input_identities.begin(), input_identities.end(), std::make_pair(set, index),
|
||||||
|
[](const PublicIdentity &value, const std::pair<uint64_t, uint32_t> &key) {
|
||||||
|
return std::make_pair(value.feature_set_id, value.feature_index) < key;
|
||||||
|
});
|
||||||
|
return found != input_identities.end() && found->feature_set_id == set &&
|
||||||
|
found->feature_index == index ? &*found : nullptr;
|
||||||
|
};
|
||||||
|
for (size_t i = 0; i < edge_count; ++i) {
|
||||||
|
if (!edges[i].first || !edges[i].second)
|
||||||
|
return LARDON3D_TRACK_BUILDER_CORRUPT_INPUT;
|
||||||
|
PublicIdentity *first = find_identity(edges[i].first->feature_set_id,
|
||||||
|
edges[i].first->feature_index);
|
||||||
|
PublicIdentity *second = find_identity(edges[i].second->feature_set_id,
|
||||||
|
edges[i].second->feature_index);
|
||||||
|
if (!first || !second || !meta_equal(first->metadata, project(*edges[i].first), true) ||
|
||||||
|
!meta_equal(second->metadata, project(*edges[i].second), true))
|
||||||
|
return LARDON3D_TRACK_BUILDER_CORRUPT_INPUT;
|
||||||
|
/* WHY: only observations participating in an edge enter CompactGraph.
|
||||||
|
* Isolated public inputs cannot invalidate an unrelated component. */
|
||||||
|
if (!first->registered) {
|
||||||
|
first->metadata_index = graph.register_feature(first->metadata);
|
||||||
|
first->registered = true;
|
||||||
|
}
|
||||||
|
if (!second->registered) {
|
||||||
|
second->metadata_index = graph.register_feature(second->metadata);
|
||||||
|
second->registered = true;
|
||||||
|
}
|
||||||
|
if (!graph.add_edge(first->metadata_index, first->feature_index,
|
||||||
|
second->metadata_index, second->feature_index))
|
||||||
|
return LARDON3D_TRACK_BUILDER_CORRUPT_INPUT;
|
||||||
|
}
|
||||||
|
tb::Output output; const auto status = graph.build(&output); if (status) return status;
|
||||||
|
if (!output.tracks.empty()) {
|
||||||
|
result->tracks = new Lardon3DTrackBuilderTrack[output.tracks.size()]();
|
||||||
|
result->track_count = output.tracks.size();
|
||||||
|
for (size_t i = 0; i < output.tracks.size(); ++i) {
|
||||||
|
const tb::TrackRange &range = output.tracks[i];
|
||||||
|
result->tracks[i].observation_count = range.count;
|
||||||
|
result->tracks[i].observations = new Lardon3DTrackBuilderTrackObservation[range.count];
|
||||||
|
for (size_t j = 0; j < range.count; ++j) {
|
||||||
|
const tb::Membership &m = output.memberships[range.begin + j];
|
||||||
|
result->tracks[i].observations[j] =
|
||||||
|
{m.feature_set_id, m.feature_index, graph.node_image(m.node_index)};
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return LARDON3D_TRACK_BUILDER_OK;
|
return LARDON3D_TRACK_BUILDER_OK;
|
||||||
} catch (const std::bad_alloc &) {
|
} catch (const std::invalid_argument &) {
|
||||||
lardon3d_track_builder_result_free(result);
|
lardon3d_track_builder_result_free(result); return LARDON3D_TRACK_BUILDER_CORRUPT_INPUT;
|
||||||
return LARDON3D_TRACK_BUILDER_OUT_OF_MEMORY;
|
|
||||||
} catch (...) {
|
} catch (...) {
|
||||||
lardon3d_track_builder_result_free(result);
|
lardon3d_track_builder_result_free(result); return LARDON3D_TRACK_BUILDER_OUT_OF_MEMORY;
|
||||||
return LARDON3D_TRACK_BUILDER_OUT_OF_MEMORY;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
95
src/track_builder_internal.hpp
Normal file
95
src/track_builder_internal.hpp
Normal file
|
|
@ -0,0 +1,95 @@
|
||||||
|
#ifndef LARDON3D_TRACK_BUILDER_INTERNAL_HPP
|
||||||
|
#define LARDON3D_TRACK_BUILDER_INTERNAL_HPP
|
||||||
|
|
||||||
|
#include <cstddef>
|
||||||
|
#include <cstdint>
|
||||||
|
#include <vector>
|
||||||
|
|
||||||
|
extern "C" {
|
||||||
|
#include <lardon3d/track_builder.h>
|
||||||
|
#include <lardon3d/track_builder_project.h>
|
||||||
|
}
|
||||||
|
|
||||||
|
namespace lardon3d::track_builder_internal {
|
||||||
|
|
||||||
|
struct FeatureMetadata {
|
||||||
|
uint64_t feature_set_id;
|
||||||
|
uint64_t image_id;
|
||||||
|
char extractor_kind[LARDON3D_TRACK_BUILDER_KIND_CAPACITY];
|
||||||
|
uint32_t extractor_version;
|
||||||
|
unsigned char parameter_fingerprint[32];
|
||||||
|
uint32_t descriptor_type;
|
||||||
|
uint32_t descriptor_dimension;
|
||||||
|
};
|
||||||
|
|
||||||
|
struct Node {
|
||||||
|
uint64_t feature_set_id;
|
||||||
|
uint32_t feature_index;
|
||||||
|
uint32_t metadata_index;
|
||||||
|
};
|
||||||
|
|
||||||
|
struct Edge {
|
||||||
|
uint32_t low;
|
||||||
|
uint32_t high;
|
||||||
|
};
|
||||||
|
|
||||||
|
struct IdentitySlot {
|
||||||
|
uint64_t feature_set_id = 0;
|
||||||
|
uint32_t feature_index = 0;
|
||||||
|
uint32_t node_plus_one = 0;
|
||||||
|
};
|
||||||
|
|
||||||
|
struct Membership {
|
||||||
|
uint64_t feature_set_id;
|
||||||
|
uint32_t feature_index;
|
||||||
|
uint32_t node_index;
|
||||||
|
};
|
||||||
|
|
||||||
|
struct TrackRange {
|
||||||
|
size_t begin;
|
||||||
|
size_t count;
|
||||||
|
};
|
||||||
|
|
||||||
|
struct Output {
|
||||||
|
std::vector<Membership> memberships;
|
||||||
|
std::vector<TrackRange> tracks;
|
||||||
|
};
|
||||||
|
|
||||||
|
/* WHY: Project construction must have one owner for nodes, identity lookup and
|
||||||
|
* indexed edges. Transporting pointer edges through the public ABI recreated
|
||||||
|
* the complete graph twice before DSU. This private type is deliberately not
|
||||||
|
* persistent and carries no scientific policy. */
|
||||||
|
class CompactGraph {
|
||||||
|
public:
|
||||||
|
explicit CompactGraph(uint64_t raw_edge_hint);
|
||||||
|
uint32_t register_feature(const FeatureMetadata &metadata);
|
||||||
|
bool add_edge(uint32_t first_metadata, uint32_t first_feature_index,
|
||||||
|
uint32_t second_metadata, uint32_t second_feature_index);
|
||||||
|
Lardon3DTrackBuilderResult build(Output *output);
|
||||||
|
const FeatureMetadata &metadata(uint32_t index) const;
|
||||||
|
uint64_t node_image(uint32_t node_index) const;
|
||||||
|
size_t node_count() const { return nodes_.size(); }
|
||||||
|
uint64_t raw_edge_count() const { return raw_edge_count_; }
|
||||||
|
|
||||||
|
private:
|
||||||
|
uint32_t resolve_node(uint32_t metadata_index, uint32_t feature_index);
|
||||||
|
void insert_identity(const Node &node, uint32_t node_index);
|
||||||
|
|
||||||
|
std::vector<FeatureMetadata> metadata_;
|
||||||
|
std::vector<Node> nodes_;
|
||||||
|
std::vector<Edge> edges_;
|
||||||
|
std::vector<IdentitySlot> identity_;
|
||||||
|
size_t identity_size_ = 0;
|
||||||
|
uint64_t raw_edge_count_ = 0;
|
||||||
|
};
|
||||||
|
|
||||||
|
using Checkpoint = bool (*)(void *userdata);
|
||||||
|
|
||||||
|
Lardon3DTrackBuilderProjectStatus build_project(
|
||||||
|
const Lardon3DTrackBuilderProjectRequest *request,
|
||||||
|
Lardon3DTrackBuilderProjectResult *result, Checkpoint checkpoint,
|
||||||
|
void *checkpoint_userdata);
|
||||||
|
|
||||||
|
} // namespace lardon3d::track_builder_internal
|
||||||
|
|
||||||
|
#endif
|
||||||
|
|
@ -2,18 +2,20 @@
|
||||||
#include <array>
|
#include <array>
|
||||||
#include <cstring>
|
#include <cstring>
|
||||||
#include <cstdio>
|
#include <cstdio>
|
||||||
#include <deque>
|
|
||||||
#include <fcntl.h>
|
#include <fcntl.h>
|
||||||
#include <limits>
|
#include <limits>
|
||||||
#include <new>
|
#include <new>
|
||||||
#include <openssl/evp.h>
|
#include <openssl/evp.h>
|
||||||
#include <string>
|
#include <string>
|
||||||
|
#include <stdexcept>
|
||||||
#include <sys/stat.h>
|
#include <sys/stat.h>
|
||||||
#include <unistd.h>
|
#include <unistd.h>
|
||||||
#include <unordered_map>
|
#include <unordered_map>
|
||||||
#include <utility>
|
#include <utility>
|
||||||
#include <vector>
|
#include <vector>
|
||||||
|
|
||||||
|
#include "track_builder_internal.hpp"
|
||||||
|
|
||||||
extern "C" {
|
extern "C" {
|
||||||
#include <lardon3d/match_file.h>
|
#include <lardon3d/match_file.h>
|
||||||
#include <lardon3d/track_builder.h>
|
#include <lardon3d/track_builder.h>
|
||||||
|
|
@ -21,25 +23,32 @@ extern "C" {
|
||||||
}
|
}
|
||||||
|
|
||||||
namespace {
|
namespace {
|
||||||
using Observation = Lardon3DTrackBuilderObservation;
|
namespace internal = lardon3d::track_builder_internal;
|
||||||
using Edge = Lardon3DTrackBuilderEdge;
|
|
||||||
|
|
||||||
struct Key {
|
#ifdef LARDON3D_TRACK_BUILDER_PROJECT_TESTING
|
||||||
uint64_t set;
|
extern "C" void lardon3d_track_builder_project_test_publication_capacities(
|
||||||
uint32_t index;
|
uint64_t node_count, uint64_t serialization_capacity_bytes);
|
||||||
bool operator==(const Key &other) const { return set == other.set && index == other.index; }
|
#endif
|
||||||
};
|
|
||||||
|
|
||||||
struct KeyHash {
|
|
||||||
size_t operator()(const Key &key) const noexcept {
|
|
||||||
return static_cast<size_t>(key.set ^ (static_cast<uint64_t>(key.index) * 0x9e3779b97f4a7c15ULL));
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
struct FeatureCacheEntry {
|
struct FeatureCacheEntry {
|
||||||
Lardon3DProjectDbFeatureSet value{};
|
uint64_t feature_set_id = 0;
|
||||||
|
uint64_t image_id = 0;
|
||||||
|
uint32_t feature_count = 0;
|
||||||
|
uint32_t metadata_index = 0;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
internal::FeatureMetadata project_metadata(const Lardon3DProjectDbFeatureSet &set) {
|
||||||
|
internal::FeatureMetadata value{};
|
||||||
|
value.feature_set_id = set.feature_set_id;
|
||||||
|
value.image_id = set.image_id;
|
||||||
|
std::memcpy(value.extractor_kind, set.extractor_kind, sizeof(value.extractor_kind));
|
||||||
|
value.extractor_version = set.extractor_version;
|
||||||
|
std::memcpy(value.parameter_fingerprint, set.parameter_fingerprint, 32);
|
||||||
|
value.descriptor_type = set.descriptor_type;
|
||||||
|
value.descriptor_dimension = set.descriptor_dimension;
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
Lardon3DTrackBuilderProjectStatus map_db(Lardon3DProjectDbResult value) {
|
Lardon3DTrackBuilderProjectStatus map_db(Lardon3DProjectDbResult value) {
|
||||||
if (value == LARDON3D_PROJECT_DB_INVALID_ARGUMENT || value == LARDON3D_PROJECT_DB_CONSTRAINT)
|
if (value == LARDON3D_PROJECT_DB_INVALID_ARGUMENT || value == LARDON3D_PROJECT_DB_CONSTRAINT)
|
||||||
return LARDON3D_TRACK_BUILDER_PROJECT_INVALID_ARGUMENT;
|
return LARDON3D_TRACK_BUILDER_PROJECT_INVALID_ARGUMENT;
|
||||||
|
|
@ -120,9 +129,7 @@ Lardon3DProjectDbResult validate_gvr_snapshot(
|
||||||
Lardon3DTrackBuilderProjectStatus resolve_gvr(
|
Lardon3DTrackBuilderProjectStatus resolve_gvr(
|
||||||
const Lardon3DTrackBuilderProjectRequest &request, uint64_t gvr_id,
|
const Lardon3DTrackBuilderProjectRequest &request, uint64_t gvr_id,
|
||||||
std::unordered_map<uint64_t, FeatureCacheEntry> &cache,
|
std::unordered_map<uint64_t, FeatureCacheEntry> &cache,
|
||||||
std::deque<Observation> &observations,
|
internal::CompactGraph &graph) {
|
||||||
std::unordered_map<Key, Observation *, KeyHash> &observation_map,
|
|
||||||
std::vector<Edge> &edges) {
|
|
||||||
Lardon3DProjectDbGeometricVerificationResult gvr{};
|
Lardon3DProjectDbGeometricVerificationResult gvr{};
|
||||||
Lardon3DProjectDbResult db = lardon3d_project_db_load_geometric_verification_result(
|
Lardon3DProjectDbResult db = lardon3d_project_db_load_geometric_verification_result(
|
||||||
request.database, gvr_id, &gvr);
|
request.database, gvr_id, &gvr);
|
||||||
|
|
@ -137,23 +144,28 @@ Lardon3DTrackBuilderProjectStatus resolve_gvr(
|
||||||
Lardon3DProjectDbCandidatePair pair{};
|
Lardon3DProjectDbCandidatePair pair{};
|
||||||
db = lardon3d_project_db_load_candidate_pair(request.database, match.candidate_pair_id, &pair);
|
db = lardon3d_project_db_load_candidate_pair(request.database, match.candidate_pair_id, &pair);
|
||||||
if (db != LARDON3D_PROJECT_DB_OK) return map_db(db);
|
if (db != LARDON3D_PROJECT_DB_OK) return map_db(db);
|
||||||
auto load_set = [&](uint64_t id, Lardon3DProjectDbFeatureSet **out) {
|
auto load_set = [&](uint64_t id, FeatureCacheEntry **out) {
|
||||||
auto found = cache.find(id);
|
auto found = cache.find(id);
|
||||||
if (found != cache.end()) {
|
if (found != cache.end()) {
|
||||||
*out = &found->second.value;
|
*out = &found->second;
|
||||||
return LARDON3D_PROJECT_DB_OK;
|
return LARDON3D_PROJECT_DB_OK;
|
||||||
}
|
}
|
||||||
FeatureCacheEntry entry{};
|
FeatureCacheEntry entry{};
|
||||||
|
Lardon3DProjectDbFeatureSet loaded{};
|
||||||
Lardon3DProjectDbResult value = lardon3d_project_db_load_feature_set(
|
Lardon3DProjectDbResult value = lardon3d_project_db_load_feature_set(
|
||||||
request.database, id, &entry.value);
|
request.database, id, &loaded);
|
||||||
if (value == LARDON3D_PROJECT_DB_OK) {
|
if (value == LARDON3D_PROJECT_DB_OK) {
|
||||||
|
entry.feature_set_id = loaded.feature_set_id;
|
||||||
|
entry.image_id = loaded.image_id;
|
||||||
|
entry.feature_count = loaded.feature_count;
|
||||||
|
entry.metadata_index = graph.register_feature(project_metadata(loaded));
|
||||||
auto inserted = cache.emplace(id, entry);
|
auto inserted = cache.emplace(id, entry);
|
||||||
*out = &inserted.first->second.value;
|
*out = &inserted.first->second;
|
||||||
}
|
}
|
||||||
return value;
|
return value;
|
||||||
};
|
};
|
||||||
Lardon3DProjectDbFeatureSet *set_a = nullptr;
|
FeatureCacheEntry *set_a = nullptr;
|
||||||
Lardon3DProjectDbFeatureSet *set_b = nullptr;
|
FeatureCacheEntry *set_b = nullptr;
|
||||||
db = load_set(match.feature_set_id_a, &set_a);
|
db = load_set(match.feature_set_id_a, &set_a);
|
||||||
if (db != LARDON3D_PROJECT_DB_OK) return map_db(db);
|
if (db != LARDON3D_PROJECT_DB_OK) return map_db(db);
|
||||||
db = load_set(match.feature_set_id_b, &set_b);
|
db = load_set(match.feature_set_id_b, &set_b);
|
||||||
|
|
@ -185,31 +197,12 @@ Lardon3DTrackBuilderProjectStatus resolve_gvr(
|
||||||
if ((gvr.inlier_mask[i / 8U] & static_cast<unsigned char>(1U << (i % 8U))) == 0) continue;
|
if ((gvr.inlier_mask[i / 8U] & static_cast<unsigned char>(1U << (i % 8U))) == 0) continue;
|
||||||
++selected;
|
++selected;
|
||||||
const auto &entry = entries[i];
|
const auto &entry = entries[i];
|
||||||
Key keys[2] = {{set_a->feature_set_id, entry.feature_index_a},
|
const auto cache_a = cache.find(set_a->feature_set_id);
|
||||||
{set_b->feature_set_id, entry.feature_index_b}};
|
const auto cache_b = cache.find(set_b->feature_set_id);
|
||||||
Observation *resolved[2] = {};
|
if (cache_a == cache.end() || cache_b == cache.end() ||
|
||||||
Lardon3DProjectDbFeatureSet *sets[2] = {set_a, set_b};
|
!graph.add_edge(cache_a->second.metadata_index, entry.feature_index_a,
|
||||||
for (size_t endpoint = 0; endpoint < 2; ++endpoint) {
|
cache_b->second.metadata_index, entry.feature_index_b))
|
||||||
auto found = observation_map.find(keys[endpoint]);
|
return LARDON3D_TRACK_BUILDER_PROJECT_INPUT_CORRUPT;
|
||||||
if (found == observation_map.end()) {
|
|
||||||
Observation value{};
|
|
||||||
value.feature_set_id = keys[endpoint].set;
|
|
||||||
value.feature_index = keys[endpoint].index;
|
|
||||||
value.image_id = sets[endpoint]->image_id;
|
|
||||||
std::memcpy(value.extractor_kind, sets[endpoint]->extractor_kind,
|
|
||||||
sizeof(value.extractor_kind));
|
|
||||||
value.extractor_version = sets[endpoint]->extractor_version;
|
|
||||||
std::memcpy(value.parameter_fingerprint, sets[endpoint]->parameter_fingerprint, 32);
|
|
||||||
value.descriptor_type = sets[endpoint]->descriptor_type;
|
|
||||||
value.descriptor_dimension = sets[endpoint]->descriptor_dimension;
|
|
||||||
observations.push_back(value);
|
|
||||||
resolved[endpoint] = &observations.back();
|
|
||||||
observation_map.emplace(keys[endpoint], resolved[endpoint]);
|
|
||||||
} else {
|
|
||||||
resolved[endpoint] = found->second;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
edges.push_back({resolved[0], resolved[1]});
|
|
||||||
}
|
}
|
||||||
if (selected != gvr.inlier_count) return LARDON3D_TRACK_BUILDER_PROJECT_INPUT_CORRUPT;
|
if (selected != gvr.inlier_count) return LARDON3D_TRACK_BUILDER_PROJECT_INPUT_CORRUPT;
|
||||||
return LARDON3D_TRACK_BUILDER_PROJECT_OK;
|
return LARDON3D_TRACK_BUILDER_PROJECT_OK;
|
||||||
|
|
@ -217,9 +210,10 @@ Lardon3DTrackBuilderProjectStatus resolve_gvr(
|
||||||
|
|
||||||
} // namespace
|
} // namespace
|
||||||
|
|
||||||
extern "C" Lardon3DTrackBuilderProjectStatus lardon3d_track_builder_build_project(
|
Lardon3DTrackBuilderProjectStatus internal::build_project(
|
||||||
const Lardon3DTrackBuilderProjectRequest *request,
|
const Lardon3DTrackBuilderProjectRequest *request,
|
||||||
Lardon3DTrackBuilderProjectResult *result) {
|
Lardon3DTrackBuilderProjectResult *result, Checkpoint checkpoint,
|
||||||
|
void *checkpoint_userdata) {
|
||||||
if (!request || !result || !request->project_path || !request->database ||
|
if (!request || !result || !request->project_path || !request->database ||
|
||||||
!request->verifier_fingerprint || !request->gvr_ids || request->gvr_count == 0 ||
|
!request->verifier_fingerprint || !request->gvr_ids || request->gvr_count == 0 ||
|
||||||
request->verifier_kind <= 0 || request->verifier_version == 0) {
|
request->verifier_kind <= 0 || request->verifier_version == 0) {
|
||||||
|
|
@ -265,79 +259,97 @@ extern "C" Lardon3DTrackBuilderProjectStatus lardon3d_track_builder_build_projec
|
||||||
}
|
}
|
||||||
if (db != LARDON3D_PROJECT_DB_NOT_FOUND) return map_db(db);
|
if (db != LARDON3D_PROJECT_DB_NOT_FOUND) return map_db(db);
|
||||||
|
|
||||||
std::unordered_map<uint64_t, FeatureCacheEntry> cache;
|
uint64_t raw_edge_hint = 0;
|
||||||
std::deque<Observation> observations;
|
|
||||||
std::unordered_map<Key, Observation *, KeyHash> observation_map;
|
|
||||||
std::vector<Edge> edges;
|
|
||||||
for (size_t i = 0; i < request->gvr_count; ++i) {
|
for (size_t i = 0; i < request->gvr_count; ++i) {
|
||||||
auto status = resolve_gvr(*request, request->gvr_ids[i], cache, observations,
|
Lardon3DProjectDbGeometricVerificationResult gvr{};
|
||||||
observation_map, edges);
|
db = lardon3d_project_db_load_geometric_verification_result(
|
||||||
|
request->database, request->gvr_ids[i], &gvr);
|
||||||
|
if (db != LARDON3D_PROJECT_DB_OK) return map_db(db);
|
||||||
|
if (gvr.inlier_count > UINT64_MAX - raw_edge_hint)
|
||||||
|
return LARDON3D_TRACK_BUILDER_PROJECT_OUT_OF_MEMORY;
|
||||||
|
raw_edge_hint += gvr.inlier_count;
|
||||||
|
}
|
||||||
|
internal::CompactGraph graph(raw_edge_hint);
|
||||||
|
std::unordered_map<uint64_t, FeatureCacheEntry> cache;
|
||||||
|
for (size_t i = 0; i < request->gvr_count; ++i) {
|
||||||
|
auto status = resolve_gvr(*request, request->gvr_ids[i], cache, graph);
|
||||||
if (status != LARDON3D_TRACK_BUILDER_PROJECT_OK) return status;
|
if (status != LARDON3D_TRACK_BUILDER_PROJECT_OK) return status;
|
||||||
|
/* CONTRACT: Task pause/cancel is observed only between complete GVR
|
||||||
|
* reads. DSU, canonicalization and atomic publication are never
|
||||||
|
* preempted, so interruption cannot expose a partial Track Set. */
|
||||||
|
if (checkpoint && !checkpoint(checkpoint_userdata))
|
||||||
|
return LARDON3D_TRACK_BUILDER_PROJECT_INTERRUPTED;
|
||||||
}
|
}
|
||||||
std::vector<Observation> core_observations(observations.begin(), observations.end());
|
internal::Output core;
|
||||||
std::unordered_map<Key, Observation *, KeyHash> core_map;
|
auto core_status = graph.build(&core);
|
||||||
core_map.reserve(core_observations.size());
|
|
||||||
for (auto &observation : core_observations)
|
|
||||||
core_map.emplace(Key{observation.feature_set_id, observation.feature_index}, &observation);
|
|
||||||
std::vector<Edge> core_edges;
|
|
||||||
core_edges.reserve(edges.size());
|
|
||||||
for (const Edge &edge : edges) {
|
|
||||||
Key first{edge.first->feature_set_id, edge.first->feature_index};
|
|
||||||
Key second{edge.second->feature_set_id, edge.second->feature_index};
|
|
||||||
core_edges.push_back({core_map.at(first), core_map.at(second)});
|
|
||||||
}
|
|
||||||
Lardon3DTrackBuilderResultSet core{};
|
|
||||||
auto core_status = lardon3d_track_builder_build(
|
|
||||||
core_observations.empty() ? nullptr : core_observations.data(), core_observations.size(),
|
|
||||||
core_edges.empty() ? nullptr : core_edges.data(), core_edges.size(), &core);
|
|
||||||
if (core_status != LARDON3D_TRACK_BUILDER_OK) {
|
if (core_status != LARDON3D_TRACK_BUILDER_OK) {
|
||||||
lardon3d_track_builder_result_free(&core);
|
|
||||||
return core_status == LARDON3D_TRACK_BUILDER_OUT_OF_MEMORY
|
return core_status == LARDON3D_TRACK_BUILDER_OUT_OF_MEMORY
|
||||||
? LARDON3D_TRACK_BUILDER_PROJECT_OUT_OF_MEMORY
|
? LARDON3D_TRACK_BUILDER_PROJECT_OUT_OF_MEMORY
|
||||||
: LARDON3D_TRACK_BUILDER_PROJECT_CORE_ERROR;
|
: LARDON3D_TRACK_BUILDER_PROJECT_CORE_ERROR;
|
||||||
}
|
}
|
||||||
std::vector<std::vector<Lardon3DProjectDbTrackObservation>> stored(core.track_count);
|
/* GATE D CONTRACT: Gate B (DSU/canonicalization) is deliberately
|
||||||
std::vector<Lardon3DProjectDbTrack> publish(core.track_count);
|
* non-preemptible, but a pause/cancel raised during it must be observed
|
||||||
for (size_t i = 0; i < core.track_count; ++i) {
|
* before any publication preparation can become durable. */
|
||||||
stored[i].resize(core.tracks[i].observation_count);
|
if (checkpoint && !checkpoint(checkpoint_userdata))
|
||||||
|
return LARDON3D_TRACK_BUILDER_PROJECT_INTERRUPTED;
|
||||||
|
/* CONTRACT/SERIALIZATION: this is the sole transient conversion from
|
||||||
|
* canonical compact membership to Track Model rows. position_in_track is
|
||||||
|
* the already-canonical flat order; DB publication below remains the sole
|
||||||
|
* atomic, owner-only persistence point. */
|
||||||
|
std::vector<std::vector<Lardon3DProjectDbTrackObservation>> stored(core.tracks.size());
|
||||||
|
std::vector<Lardon3DProjectDbTrack> publish(core.tracks.size());
|
||||||
|
for (size_t i = 0; i < core.tracks.size(); ++i) {
|
||||||
|
const internal::TrackRange &range = core.tracks[i];
|
||||||
|
stored[i].resize(range.count);
|
||||||
for (size_t j = 0; j < stored[i].size(); ++j) {
|
for (size_t j = 0; j < stored[i].size(); ++j) {
|
||||||
const auto &source = core.tracks[i].observations[j];
|
const auto &source = core.memberships[range.begin + j];
|
||||||
stored[i][j] = {source.feature_set_id, source.feature_index,
|
stored[i][j] = {source.feature_set_id, source.feature_index,
|
||||||
static_cast<uint32_t>(j)};
|
static_cast<uint32_t>(j)};
|
||||||
}
|
}
|
||||||
publish[i] = {0, 0, static_cast<uint32_t>(stored[i].size()), stored[i].data()};
|
publish[i] = {0, 0, static_cast<uint32_t>(stored[i].size()), stored[i].data()};
|
||||||
}
|
}
|
||||||
#ifdef LARDON3D_TRACK_BUILDER_PROJECT_TESTING
|
#ifdef LARDON3D_TRACK_BUILDER_PROJECT_TESTING
|
||||||
|
uint64_t stored_observation_capacity = 0;
|
||||||
|
for (const auto &observations : stored)
|
||||||
|
stored_observation_capacity += observations.capacity();
|
||||||
|
const uint64_t serialization_capacity_bytes =
|
||||||
|
stored_observation_capacity * sizeof(Lardon3DProjectDbTrackObservation) +
|
||||||
|
stored.capacity() * sizeof(std::vector<Lardon3DProjectDbTrackObservation>) +
|
||||||
|
publish.capacity() * sizeof(Lardon3DProjectDbTrack);
|
||||||
|
lardon3d_track_builder_project_test_publication_capacities(
|
||||||
|
graph.node_count(), serialization_capacity_bytes);
|
||||||
lardon3d_track_builder_project_test_before_revalidation(
|
lardon3d_track_builder_project_test_before_revalidation(
|
||||||
request->database, request->gvr_ids, request->gvr_count);
|
request->database, request->gvr_ids, request->gvr_count);
|
||||||
#endif
|
#endif
|
||||||
for (size_t i = 0; i < request->gvr_count; ++i) {
|
for (size_t i = 0; i < request->gvr_count; ++i) {
|
||||||
db = validate_gvr_snapshot(*request, request->gvr_ids[i]);
|
db = validate_gvr_snapshot(*request, request->gvr_ids[i]);
|
||||||
if (db != LARDON3D_PROJECT_DB_OK) {
|
if (db != LARDON3D_PROJECT_DB_OK) {
|
||||||
lardon3d_track_builder_result_free(&core);
|
|
||||||
return map_db(db);
|
return map_db(db);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
/* GATE D CONTRACT: this is the final interruptible boundary. Refusal
|
||||||
|
* guarantees zero publication; create_track_set below remains one
|
||||||
|
* non-preemptible atomic publication. */
|
||||||
|
if (checkpoint && !checkpoint(checkpoint_userdata))
|
||||||
|
return LARDON3D_TRACK_BUILDER_PROJECT_INTERRUPTED;
|
||||||
Lardon3DProjectDbTrackSet published{};
|
Lardon3DProjectDbTrackSet published{};
|
||||||
identity.track_count = core.track_count;
|
identity.track_count = core.tracks.size();
|
||||||
#ifdef LARDON3D_TRACK_BUILDER_PROJECT_TESTING
|
#ifdef LARDON3D_TRACK_BUILDER_PROJECT_TESTING
|
||||||
lardon3d_track_builder_project_test_before_publication(request);
|
lardon3d_track_builder_project_test_before_publication(request);
|
||||||
Lardon3DProjectDbTrackSet raced_before_publish{};
|
Lardon3DProjectDbTrackSet raced_before_publish{};
|
||||||
if (lardon3d_project_db_find_track_set(request->database, &identity,
|
if (lardon3d_project_db_find_track_set(request->database, &identity,
|
||||||
&raced_before_publish) == LARDON3D_PROJECT_DB_OK) {
|
&raced_before_publish) == LARDON3D_PROJECT_DB_OK) {
|
||||||
lardon3d_track_builder_result_free(&core);
|
|
||||||
result->track_set_id = raced_before_publish.track_set_id;
|
result->track_set_id = raced_before_publish.track_set_id;
|
||||||
result->gvr_count = raced_before_publish.gvr_count;
|
result->gvr_count = raced_before_publish.gvr_count;
|
||||||
result->track_count = raced_before_publish.track_count;
|
result->track_count = raced_before_publish.track_count;
|
||||||
result->raw_inlier_edge_count = edges.size();
|
result->raw_inlier_edge_count = graph.raw_edge_count();
|
||||||
result->core_observation_count = observations.size();
|
result->core_observation_count = graph.node_count();
|
||||||
result->reused = true;
|
result->reused = true;
|
||||||
return LARDON3D_TRACK_BUILDER_PROJECT_OK;
|
return LARDON3D_TRACK_BUILDER_PROJECT_OK;
|
||||||
}
|
}
|
||||||
#endif
|
#endif
|
||||||
db = lardon3d_project_db_create_track_set(request->database, &identity,
|
db = lardon3d_project_db_create_track_set(request->database, &identity,
|
||||||
publish.data(), publish.size(), &published);
|
publish.data(), publish.size(), &published);
|
||||||
lardon3d_track_builder_result_free(&core);
|
|
||||||
if (db == LARDON3D_PROJECT_DB_CONSTRAINT) {
|
if (db == LARDON3D_PROJECT_DB_CONSTRAINT) {
|
||||||
Lardon3DProjectDbTrackSet raced{};
|
Lardon3DProjectDbTrackSet raced{};
|
||||||
db = lardon3d_project_db_find_track_set(request->database, &identity, &raced);
|
db = lardon3d_project_db_find_track_set(request->database, &identity, &raced);
|
||||||
|
|
@ -345,8 +357,8 @@ extern "C" Lardon3DTrackBuilderProjectStatus lardon3d_track_builder_build_projec
|
||||||
result->track_set_id = raced.track_set_id;
|
result->track_set_id = raced.track_set_id;
|
||||||
result->gvr_count = raced.gvr_count;
|
result->gvr_count = raced.gvr_count;
|
||||||
result->track_count = raced.track_count;
|
result->track_count = raced.track_count;
|
||||||
result->raw_inlier_edge_count = edges.size();
|
result->raw_inlier_edge_count = graph.raw_edge_count();
|
||||||
result->core_observation_count = observations.size();
|
result->core_observation_count = graph.node_count();
|
||||||
result->reused = true;
|
result->reused = true;
|
||||||
return LARDON3D_TRACK_BUILDER_PROJECT_OK;
|
return LARDON3D_TRACK_BUILDER_PROJECT_OK;
|
||||||
}
|
}
|
||||||
|
|
@ -355,13 +367,21 @@ extern "C" Lardon3DTrackBuilderProjectStatus lardon3d_track_builder_build_projec
|
||||||
result->track_set_id = published.track_set_id;
|
result->track_set_id = published.track_set_id;
|
||||||
result->gvr_count = published.gvr_count;
|
result->gvr_count = published.gvr_count;
|
||||||
result->track_count = published.track_count;
|
result->track_count = published.track_count;
|
||||||
result->raw_inlier_edge_count = edges.size();
|
result->raw_inlier_edge_count = graph.raw_edge_count();
|
||||||
result->core_observation_count = observations.size();
|
result->core_observation_count = graph.node_count();
|
||||||
result->reused = false;
|
result->reused = false;
|
||||||
return LARDON3D_TRACK_BUILDER_PROJECT_OK;
|
return LARDON3D_TRACK_BUILDER_PROJECT_OK;
|
||||||
} catch (const std::bad_alloc &) {
|
} catch (const std::bad_alloc &) {
|
||||||
return LARDON3D_TRACK_BUILDER_PROJECT_OUT_OF_MEMORY;
|
return LARDON3D_TRACK_BUILDER_PROJECT_OUT_OF_MEMORY;
|
||||||
|
} catch (const std::invalid_argument &) {
|
||||||
|
return LARDON3D_TRACK_BUILDER_PROJECT_INPUT_CORRUPT;
|
||||||
} catch (...) {
|
} catch (...) {
|
||||||
return LARDON3D_TRACK_BUILDER_PROJECT_DATABASE_ERROR;
|
return LARDON3D_TRACK_BUILDER_PROJECT_DATABASE_ERROR;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
extern "C" Lardon3DTrackBuilderProjectStatus lardon3d_track_builder_build_project(
|
||||||
|
const Lardon3DTrackBuilderProjectRequest *request,
|
||||||
|
Lardon3DTrackBuilderProjectResult *result) {
|
||||||
|
return internal::build_project(request, result, nullptr, nullptr);
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -12,7 +12,10 @@
|
||||||
#include <unistd.h>
|
#include <unistd.h>
|
||||||
#include <vector>
|
#include <vector>
|
||||||
|
|
||||||
|
#include "track_builder_internal.hpp"
|
||||||
|
|
||||||
extern "C" {
|
extern "C" {
|
||||||
|
#include <lardon3d/match_file.h>
|
||||||
#include <lardon3d/project.h>
|
#include <lardon3d/project.h>
|
||||||
#include <lardon3d/task_checkpoint.h>
|
#include <lardon3d/task_checkpoint.h>
|
||||||
#include <lardon3d/task_queue.h>
|
#include <lardon3d/task_queue.h>
|
||||||
|
|
@ -25,11 +28,6 @@ constexpr std::array<unsigned char, 8> kScopeMagic = {
|
||||||
'L', '3', 'D', 'T', 'S', 'C', 'P', '1'};
|
'L', '3', 'D', 'T', 'S', 'C', 'P', '1'};
|
||||||
constexpr uint32_t kScopeVersion = 1;
|
constexpr uint32_t kScopeVersion = 1;
|
||||||
constexpr uint64_t kFixedMemory = 4ULL * 1024ULL * 1024ULL;
|
constexpr uint64_t kFixedMemory = 4ULL * 1024ULL * 1024ULL;
|
||||||
constexpr uint64_t kBytesPerEdge = 48;
|
|
||||||
constexpr uint64_t kBytesPerObservation = 160;
|
|
||||||
constexpr uint64_t kNormalSafetyMultiplier = 2;
|
|
||||||
constexpr uint64_t kHighScaleSafetyMultiplier = 8;
|
|
||||||
constexpr uint64_t kHighScaleEdgeThreshold = 400000;
|
|
||||||
constexpr uint64_t kMaximumScopeBytes = 64ULL * 1024ULL * 1024ULL;
|
constexpr uint64_t kMaximumScopeBytes = 64ULL * 1024ULL * 1024ULL;
|
||||||
|
|
||||||
struct Context {
|
struct Context {
|
||||||
|
|
@ -54,6 +52,45 @@ bool checked_mul(uint64_t left, uint64_t right, uint64_t *result) {
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
bool compact_memory_estimate(uint64_t raw_edges, uint64_t feature_sets,
|
||||||
|
uint64_t max_match_count,
|
||||||
|
uint64_t *memory) {
|
||||||
|
if (!memory || raw_edges > UINT32_MAX / 2U) return false;
|
||||||
|
const uint64_t nodes = raw_edges * 2U;
|
||||||
|
uint64_t required_slots = 0;
|
||||||
|
if (!checked_add(nodes, nodes / 2U, &required_slots) ||
|
||||||
|
!checked_add(required_slots, 1U, &required_slots)) return false;
|
||||||
|
uint64_t slots = required_slots == 0 ? 0 : 8;
|
||||||
|
while (slots < required_slots) {
|
||||||
|
if (!checked_mul(slots, 2U, &slots)) return false;
|
||||||
|
}
|
||||||
|
uint64_t graph_and_peak_node_bytes = 0, edge_bytes = 0;
|
||||||
|
uint64_t identity_bytes = 0, feature_bytes = 0, match_file_peak_bytes = 0;
|
||||||
|
/* WHY/ACCOUNTING: 101 B/node is the simultaneous retained node (16),
|
||||||
|
* DSU/group/conflict scratch (17), flat canonical output (24) and worst
|
||||||
|
* per-track publication serialization (44: stored observation capacity 16,
|
||||||
|
* per-track vector capacity/objects up to 12, and publish rows 16). Identity slots and raw indexed
|
||||||
|
* edges are charged by their actual reserved capacities. 640 B/Feature Set
|
||||||
|
* bounds the metadata projection plus the adapter cache/hash allocation.
|
||||||
|
* WHY/CONTRACT: resolve_gvr materializes exactly one Match File at a time;
|
||||||
|
* charge its largest entry vector as a peak, not the sum across the scope. */
|
||||||
|
if (!checked_mul(nodes, 101U, &graph_and_peak_node_bytes) ||
|
||||||
|
!checked_mul(raw_edges, sizeof(lardon3d::track_builder_internal::Edge), &edge_bytes) ||
|
||||||
|
!checked_mul(slots, sizeof(lardon3d::track_builder_internal::IdentitySlot),
|
||||||
|
&identity_bytes) ||
|
||||||
|
!checked_mul(feature_sets, 640U, &feature_bytes) ||
|
||||||
|
!checked_mul(max_match_count, sizeof(Lardon3DMatchFileEntry),
|
||||||
|
&match_file_peak_bytes)) return false;
|
||||||
|
uint64_t total = kFixedMemory;
|
||||||
|
if (!checked_add(total, graph_and_peak_node_bytes, &total) ||
|
||||||
|
!checked_add(total, edge_bytes, &total) ||
|
||||||
|
!checked_add(total, identity_bytes, &total) ||
|
||||||
|
!checked_add(total, feature_bytes, &total) ||
|
||||||
|
!checked_add(total, match_file_peak_bytes, &total)) return false;
|
||||||
|
*memory = total;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
bool hash_bytes(const unsigned char *data, size_t size, unsigned char output[32]) {
|
bool hash_bytes(const unsigned char *data, size_t size, unsigned char output[32]) {
|
||||||
unsigned int length = 0;
|
unsigned int length = 0;
|
||||||
return EVP_Digest(data, size, output, &length, EVP_sha256(), nullptr) == 1 && length == 32;
|
return EVP_Digest(data, size, output, &length, EVP_sha256(), nullptr) == 1 && length == 32;
|
||||||
|
|
@ -101,8 +138,7 @@ bool scope_asset(const std::string &project_path, uint64_t task_id,
|
||||||
if (written <= 0 || written >= 4096) return false;
|
if (written <= 0 || written >= 4096) return false;
|
||||||
std::string final_path = project_path + "/" + relative;
|
std::string final_path = project_path + "/" + relative;
|
||||||
std::string temporary = final_path + ".tmp";
|
std::string temporary = final_path + ".tmp";
|
||||||
int fd = open(temporary.c_str(), O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC, 0600);
|
std::string directory_path = project_path + "/.lardon3d/checkpoints";
|
||||||
if (fd < 0) return false;
|
|
||||||
std::vector<unsigned char> bytes;
|
std::vector<unsigned char> bytes;
|
||||||
bytes.reserve(20U + ids.size() * 8U);
|
bytes.reserve(20U + ids.size() * 8U);
|
||||||
bytes.insert(bytes.end(), kScopeMagic.begin(), kScopeMagic.end());
|
bytes.insert(bytes.end(), kScopeMagic.begin(), kScopeMagic.end());
|
||||||
|
|
@ -114,6 +150,9 @@ bool scope_asset(const std::string &project_path, uint64_t task_id,
|
||||||
for (uint64_t id : ids)
|
for (uint64_t id : ids)
|
||||||
for (unsigned shift = 0; shift < 8; ++shift)
|
for (unsigned shift = 0; shift < 8; ++shift)
|
||||||
bytes.push_back(static_cast<unsigned char>((id >> (shift * 8U)) & 0xffU));
|
bytes.push_back(static_cast<unsigned char>((id >> (shift * 8U)) & 0xffU));
|
||||||
|
if (!hash_bytes(bytes.data(), bytes.size(), sha256)) return false;
|
||||||
|
int fd = open(temporary.c_str(), O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC, 0600);
|
||||||
|
if (fd < 0) return false;
|
||||||
bool ok = write_all(fd, bytes.data(), bytes.size()) && fsync(fd) == 0;
|
bool ok = write_all(fd, bytes.data(), bytes.size()) && fsync(fd) == 0;
|
||||||
if (close(fd) != 0) ok = false;
|
if (close(fd) != 0) ok = false;
|
||||||
if (!ok) {
|
if (!ok) {
|
||||||
|
|
@ -121,12 +160,25 @@ bool scope_asset(const std::string &project_path, uint64_t task_id,
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
ok = rename(temporary.c_str(), final_path.c_str()) == 0;
|
ok = rename(temporary.c_str(), final_path.c_str()) == 0;
|
||||||
if (ok) ok = hash_bytes(bytes.data(), bytes.size(), sha256);
|
bool final_published = ok;
|
||||||
if (ok) {
|
if (ok) {
|
||||||
int directory = open((project_path + "/.lardon3d/checkpoints").c_str(), O_RDONLY | O_DIRECTORY);
|
int directory = open(directory_path.c_str(), O_RDONLY | O_DIRECTORY | O_CLOEXEC);
|
||||||
ok = directory >= 0 && fsync(directory) == 0 && close(directory) == 0;
|
if (directory < 0) {
|
||||||
|
ok = false;
|
||||||
|
} else {
|
||||||
|
bool synced = fsync(directory) == 0;
|
||||||
|
#ifdef LARDON3D_TRACK_BUILDER_TASK_TESTING
|
||||||
|
const char *fail = std::getenv("LARDON3D_TRACK_BUILDER_TEST_FAIL_SCOPE_DIR_FSYNC");
|
||||||
|
if (fail && std::strcmp(fail, "1") == 0) synced = false;
|
||||||
|
#endif
|
||||||
|
bool closed = close(directory) == 0;
|
||||||
|
ok = synced && closed;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!ok) {
|
||||||
|
(void)unlink(temporary.c_str());
|
||||||
|
if (final_published) (void)unlink(final_path.c_str());
|
||||||
}
|
}
|
||||||
if (!ok) (void)unlink(temporary.c_str());
|
|
||||||
*size_bytes = bytes.size();
|
*size_bytes = bytes.size();
|
||||||
return ok;
|
return ok;
|
||||||
}
|
}
|
||||||
|
|
@ -184,12 +236,48 @@ bool persist(Context *context, const Lardon3DTask *task) {
|
||||||
? LARDON3D_DB_CHECKPOINT_DURABLE
|
? LARDON3D_DB_CHECKPOINT_DURABLE
|
||||||
: LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE;
|
: LARDON3D_DB_CHECKPOINT_PUBLISHED_NOT_DURABLE;
|
||||||
Lardon3DProjectDbTrackBuilderTask durable = context->durable;
|
Lardon3DProjectDbTrackBuilderTask durable = context->durable;
|
||||||
|
#ifdef LARDON3D_TRACK_BUILDER_TASK_TESTING
|
||||||
|
const char *fail = std::getenv("LARDON3D_TRACK_BUILDER_TEST_FAIL_INITIAL_PERSIST");
|
||||||
|
if (fail && std::strcmp(fail, "1") == 0) return false;
|
||||||
|
#endif
|
||||||
return lardon3d_project_db_record_track_builder_task(
|
return lardon3d_project_db_record_track_builder_task(
|
||||||
context->database, &snapshot, LARDON3D_TRACK_BUILDER_TASK_KIND,
|
context->database, &snapshot, LARDON3D_TRACK_BUILDER_TASK_KIND,
|
||||||
LARDON3D_TRACK_BUILDER_TASK_KIND_VERSION, &checkpoint, &durable, 0) ==
|
LARDON3D_TRACK_BUILDER_TASK_KIND_VERSION, &checkpoint, &durable, 0) ==
|
||||||
LARDON3D_PROJECT_DB_OK;
|
LARDON3D_PROJECT_DB_OK;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void cleanup_unowned_creation_assets(const std::string &project_path,
|
||||||
|
const std::string &scope_path,
|
||||||
|
uint64_t task_id) {
|
||||||
|
/* WHY/OWNERSHIP: task_id was freshly allocated and no durable Task row owns
|
||||||
|
* these paths until the initial persist succeeds. This helper is used only
|
||||||
|
* on that pre-record creation path; recovery/existing durable tasks must
|
||||||
|
* never have their scope or checkpoint removed here. */
|
||||||
|
std::string scope = project_path + "/" + scope_path;
|
||||||
|
std::string checkpoint = project_path + "/.lardon3d/checkpoints/" +
|
||||||
|
std::to_string(task_id) + ".chk";
|
||||||
|
(void)unlink((scope + ".tmp").c_str());
|
||||||
|
(void)unlink(scope.c_str());
|
||||||
|
(void)unlink((checkpoint + ".next").c_str());
|
||||||
|
(void)unlink((checkpoint + ".tmp").c_str());
|
||||||
|
(void)unlink(checkpoint.c_str());
|
||||||
|
}
|
||||||
|
|
||||||
|
struct UnownedCreationAssets {
|
||||||
|
std::string project_path;
|
||||||
|
std::string scope_path;
|
||||||
|
uint64_t task_id = 0;
|
||||||
|
bool active = true;
|
||||||
|
|
||||||
|
~UnownedCreationAssets() {
|
||||||
|
if (active) cleanup_unowned_creation_assets(project_path, scope_path, task_id);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
bool project_checkpoint(void *userdata) {
|
||||||
|
return lardon3d_task_checkpoint(static_cast<Lardon3DTask *>(userdata));
|
||||||
|
}
|
||||||
|
|
||||||
bool run(Lardon3DTask *task, void *userdata) {
|
bool run(Lardon3DTask *task, void *userdata) {
|
||||||
try {
|
try {
|
||||||
Context *context = static_cast<Context *>(userdata);
|
Context *context = static_cast<Context *>(userdata);
|
||||||
|
|
@ -199,7 +287,10 @@ bool run(Lardon3DTask *task, void *userdata) {
|
||||||
context->durable.verifier_version, context->durable.verifier_fingerprint,
|
context->durable.verifier_version, context->durable.verifier_fingerprint,
|
||||||
context->ids.data(), context->ids.size()};
|
context->ids.data(), context->ids.size()};
|
||||||
Lardon3DTrackBuilderProjectResult result{};
|
Lardon3DTrackBuilderProjectResult result{};
|
||||||
Lardon3DTrackBuilderProjectStatus status = lardon3d_track_builder_build_project(&request, &result);
|
Lardon3DTrackBuilderProjectStatus status =
|
||||||
|
lardon3d::track_builder_internal::build_project(
|
||||||
|
&request, &result, project_checkpoint, task);
|
||||||
|
if (status == LARDON3D_TRACK_BUILDER_PROJECT_INTERRUPTED) return false;
|
||||||
if (status != LARDON3D_TRACK_BUILDER_PROJECT_OK)
|
if (status != LARDON3D_TRACK_BUILDER_PROJECT_OK)
|
||||||
return lardon3d_task_fail(task, "Track Builder publication impossible.");
|
return lardon3d_task_fail(task, "Track Builder publication impossible.");
|
||||||
return lardon3d_task_set_progress(task, 100, result.reused ? "Track Set réutilisé."
|
return lardon3d_task_set_progress(task, 100, result.reused ? "Track Set réutilisé."
|
||||||
|
|
@ -248,6 +339,19 @@ bool make_context(const Lardon3DTaskReconstructionContext *runtime,
|
||||||
}
|
}
|
||||||
} // namespace
|
} // namespace
|
||||||
|
|
||||||
|
extern "C" bool lardon3d_track_builder_task_memory_estimate(
|
||||||
|
uint64_t raw_edge_count, uint64_t feature_set_count,
|
||||||
|
uint64_t *memory_bytes) {
|
||||||
|
return compact_memory_estimate(raw_edge_count, feature_set_count, 0, memory_bytes);
|
||||||
|
}
|
||||||
|
|
||||||
|
extern "C" bool lardon3d_track_builder_task_memory_estimate_with_match_peak(
|
||||||
|
uint64_t raw_edge_count, uint64_t feature_set_count,
|
||||||
|
uint64_t max_match_count, uint64_t *memory_bytes) {
|
||||||
|
return compact_memory_estimate(raw_edge_count, feature_set_count,
|
||||||
|
max_match_count, memory_bytes);
|
||||||
|
}
|
||||||
|
|
||||||
static bool reconstruct_impl(
|
static bool reconstruct_impl(
|
||||||
const Lardon3DTaskDurableSnapshot *snapshot, void *userdata,
|
const Lardon3DTaskDurableSnapshot *snapshot, void *userdata,
|
||||||
Lardon3DTaskKindBinding *binding) {
|
Lardon3DTaskKindBinding *binding) {
|
||||||
|
|
@ -283,18 +387,27 @@ static Lardon3DTask *create_impl(
|
||||||
for (size_t i = 0; i < ids.size(); ++i)
|
for (size_t i = 0; i < ids.size(); ++i)
|
||||||
if (ids[i] == 0 || (i != 0 && ids[i - 1] >= ids[i])) return nullptr;
|
if (ids[i] == 0 || (i != 0 && ids[i - 1] >= ids[i])) return nullptr;
|
||||||
uint64_t raw_edges = 0;
|
uint64_t raw_edges = 0;
|
||||||
|
uint64_t max_match_count = 0;
|
||||||
|
std::vector<uint64_t> feature_sets;
|
||||||
|
feature_sets.reserve(ids.size() * 2U);
|
||||||
for (uint64_t id : ids) {
|
for (uint64_t id : ids) {
|
||||||
Lardon3DProjectDbGeometricVerificationResult gvr{};
|
Lardon3DProjectDbGeometricVerificationResult gvr{};
|
||||||
if (lardon3d_project_db_load_geometric_verification_result(state->project_db, id, &gvr) !=
|
if (lardon3d_project_db_load_geometric_verification_result(state->project_db, id, &gvr) !=
|
||||||
LARDON3D_PROJECT_DB_OK || !checked_add(raw_edges, gvr.inlier_count, &raw_edges)) return nullptr;
|
LARDON3D_PROJECT_DB_OK || !checked_add(raw_edges, gvr.inlier_count, &raw_edges)) return nullptr;
|
||||||
|
Lardon3DProjectDbMatchResult match{};
|
||||||
|
if (lardon3d_project_db_load_match_result(state->project_db, gvr.match_result_id,
|
||||||
|
&match) != LARDON3D_PROJECT_DB_OK)
|
||||||
|
return nullptr;
|
||||||
|
max_match_count = std::max(max_match_count,
|
||||||
|
static_cast<uint64_t>(match.match_count));
|
||||||
|
feature_sets.push_back(match.feature_set_id_a);
|
||||||
|
feature_sets.push_back(match.feature_set_id_b);
|
||||||
}
|
}
|
||||||
uint64_t edge_bytes = 0, obs_bytes = 0, memory = kFixedMemory;
|
std::sort(feature_sets.begin(), feature_sets.end());
|
||||||
if (!checked_mul(raw_edges, kBytesPerEdge, &edge_bytes) ||
|
feature_sets.erase(std::unique(feature_sets.begin(), feature_sets.end()),
|
||||||
!checked_mul(raw_edges, 2 * kBytesPerObservation, &obs_bytes) ||
|
feature_sets.end());
|
||||||
!checked_add(memory, edge_bytes, &memory) || !checked_add(memory, obs_bytes, &memory) ||
|
uint64_t memory = 0;
|
||||||
!checked_mul(memory, raw_edges > kHighScaleEdgeThreshold
|
if (!compact_memory_estimate(raw_edges, feature_sets.size(), max_match_count,
|
||||||
? kHighScaleSafetyMultiplier
|
|
||||||
: kNormalSafetyMultiplier,
|
|
||||||
&memory)) return nullptr;
|
&memory)) return nullptr;
|
||||||
unsigned char scope[32]{}, builder[32]{};
|
unsigned char scope[32]{}, builder[32]{};
|
||||||
if (!scope_hash(ids, scope) || !lardon3d_track_builder_fingerprint(builder)) return nullptr;
|
if (!scope_hash(ids, scope) || !lardon3d_track_builder_fingerprint(builder)) return nullptr;
|
||||||
|
|
@ -316,11 +429,23 @@ static Lardon3DTask *create_impl(
|
||||||
std::memcpy(context->durable.input_scope_hash, scope, 32);
|
std::memcpy(context->durable.input_scope_hash, scope, 32);
|
||||||
context->durable.gvr_count = ids.size();
|
context->durable.gvr_count = ids.size();
|
||||||
context->durable.scope_format_version = kScopeVersion;
|
context->durable.scope_format_version = kScopeVersion;
|
||||||
|
int scope_written = std::snprintf(
|
||||||
|
context->durable.scope_path, sizeof(context->durable.scope_path),
|
||||||
|
".lardon3d/checkpoints/%llu.scope", static_cast<unsigned long long>(id));
|
||||||
|
if (scope_written <= 0 || static_cast<size_t>(scope_written) >=
|
||||||
|
sizeof(context->durable.scope_path)) {
|
||||||
|
delete context;
|
||||||
|
return nullptr;
|
||||||
|
}
|
||||||
|
/* Arm before publication so allocation exceptions cannot strand an asset. */
|
||||||
|
UnownedCreationAssets unowned{context->project_path, context->durable.scope_path, id, true};
|
||||||
if (!scope_asset(context->project_path, id, ids, context->durable.scope_path,
|
if (!scope_asset(context->project_path, id, ids, context->durable.scope_path,
|
||||||
&context->durable.scope_size_bytes, context->durable.scope_sha256)) {
|
&context->durable.scope_size_bytes, context->durable.scope_sha256)) {
|
||||||
delete context;
|
delete context;
|
||||||
return nullptr;
|
return nullptr;
|
||||||
}
|
}
|
||||||
|
/* Disarm only after the Task row transaction succeeds. Copies keep
|
||||||
|
* exception rollback independent of the Context lifetime. */
|
||||||
Lardon3DResourceEstimate estimate = {memory, 0, 0, 0, 1, 1, 1, 0, 1, LARDON3D_RESOURCE_TASK_CPU};
|
Lardon3DResourceEstimate estimate = {memory, 0, 0, 0, 1, 1, 1, 0, 1, LARDON3D_RESOURCE_TASK_CPU};
|
||||||
Lardon3DTask *task = lardon3d_task_create_typed(
|
Lardon3DTask *task = lardon3d_task_create_typed(
|
||||||
"Track Builder", &estimate, LARDON3D_TRACK_BUILDER_TASK_KIND,
|
"Track Builder", &estimate, LARDON3D_TRACK_BUILDER_TASK_KIND,
|
||||||
|
|
@ -330,7 +455,8 @@ static Lardon3DTask *create_impl(
|
||||||
if (task) lardon3d_task_destroy(task); else delete context;
|
if (task) lardon3d_task_destroy(task); else delete context;
|
||||||
return nullptr;
|
return nullptr;
|
||||||
}
|
}
|
||||||
*task_id = id;
|
unowned.active = false;
|
||||||
|
if (task_id) *task_id = id;
|
||||||
return task;
|
return task;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -2053,8 +2053,31 @@ bool match_result_frontier(Runtime &runtime, size_t &count,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
bool find_pending_geometry_task(Runtime &runtime, uint64_t &geometry_task_id) {
|
enum class GeometryRecoveryKind { kNone, kGeometry, kTrackBuilder };
|
||||||
|
|
||||||
|
bool geometry_recovery_kind(const Lardon3DProjectRecoveryEntry &entry,
|
||||||
|
GeometryRecoveryKind &kind) {
|
||||||
|
kind = GeometryRecoveryKind::kNone;
|
||||||
|
if ((entry.status != LARDON3D_PROJECT_RECOVERABLE &&
|
||||||
|
entry.status != LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE) ||
|
||||||
|
entry.snapshot.recovery_state != TASK_PENDING)
|
||||||
|
return false;
|
||||||
|
if (std::strcmp(entry.task_kind, LARDON3D_GEOMETRIC_VERIFIER_TASK_KIND) == 0) {
|
||||||
|
kind = GeometryRecoveryKind::kGeometry;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (std::strcmp(entry.task_kind, LARDON3D_TRACK_BUILDER_TASK_KIND) == 0 &&
|
||||||
|
entry.task_kind_version == LARDON3D_TRACK_BUILDER_TASK_KIND_VERSION) {
|
||||||
|
kind = GeometryRecoveryKind::kTrackBuilder;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
bool find_pending_geometry_task(Runtime &runtime, uint64_t &geometry_task_id,
|
||||||
|
uint64_t &track_task_id) {
|
||||||
geometry_task_id = 0;
|
geometry_task_id = 0;
|
||||||
|
track_task_id = 0;
|
||||||
uint64_t cursor = 0;
|
uint64_t cursor = 0;
|
||||||
for (;;) {
|
for (;;) {
|
||||||
Lardon3DProjectRecoveryEntry entries[8]{};
|
Lardon3DProjectRecoveryEntry entries[8]{};
|
||||||
|
|
@ -2066,23 +2089,22 @@ bool find_pending_geometry_task(Runtime &runtime, uint64_t &geometry_task_id) {
|
||||||
for (size_t index = 0; index < count; ++index) {
|
for (size_t index = 0; index < count; ++index) {
|
||||||
const auto &entry = entries[index];
|
const auto &entry = entries[index];
|
||||||
cursor = entry.task_id;
|
cursor = entry.task_id;
|
||||||
if ((entry.status != LARDON3D_PROJECT_RECOVERABLE &&
|
GeometryRecoveryKind kind{};
|
||||||
entry.status !=
|
if (!geometry_recovery_kind(entry, kind) || geometry_task_id != 0 ||
|
||||||
LARDON3D_PROJECT_RECOVERABLE_PUBLISHED_NOT_DURABLE) ||
|
track_task_id != 0) {
|
||||||
entry.snapshot.recovery_state != TASK_PENDING ||
|
|
||||||
std::strcmp(entry.task_kind,
|
|
||||||
LARDON3D_GEOMETRIC_VERIFIER_TASK_KIND) != 0 ||
|
|
||||||
geometry_task_id != 0) {
|
|
||||||
/* The production recovery API resumes every pending Task. This
|
/* The production recovery API resumes every pending Task. This
|
||||||
* GV-only proof route must never replay an upstream or downstream kind
|
* GV-only proof route must never replay an upstream or downstream kind
|
||||||
* merely because it shares the same project. */
|
* merely because it shares the same project. */
|
||||||
std::fprintf(stderr,
|
std::fprintf(stderr,
|
||||||
"GV-only recovery refuses unsafe pending task %llu of kind %s\n",
|
"geometry recovery refuses unsafe pending task %llu of kind %s v%u\n",
|
||||||
static_cast<unsigned long long>(entry.task_id),
|
static_cast<unsigned long long>(entry.task_id),
|
||||||
entry.task_kind);
|
entry.task_kind, entry.task_kind_version);
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
if (kind == GeometryRecoveryKind::kGeometry)
|
||||||
geometry_task_id = entry.task_id;
|
geometry_task_id = entry.task_id;
|
||||||
|
else
|
||||||
|
track_task_id = entry.task_id;
|
||||||
}
|
}
|
||||||
if (count < 8) return true;
|
if (count < 8) return true;
|
||||||
}
|
}
|
||||||
|
|
@ -2100,6 +2122,18 @@ bool recover_geometry_task(Runtime &runtime, uint64_t geometry_task_id,
|
||||||
"geometric_verifier.run recovered existing");
|
"geometric_verifier.run recovered existing");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
bool recover_track_builder_task(Runtime &runtime, uint64_t track_task_id,
|
||||||
|
Lardon3DProjectRecoverySummary &recovery) {
|
||||||
|
recovery = {};
|
||||||
|
return lardon3d_project_resume_recoverable_tasks(
|
||||||
|
&runtime.state, lardon3d_task_kind_registry_production(),
|
||||||
|
&recovery) == LARDON3D_PROJECT_DB_OK &&
|
||||||
|
recovery.inspected == 1 && recovery.resumed == 1 &&
|
||||||
|
recovery.failed == 0 && !recovery.queue_full &&
|
||||||
|
wait_completed(runtime, track_task_id,
|
||||||
|
"track_builder.run recovered existing");
|
||||||
|
}
|
||||||
|
|
||||||
bool find_pending_pre_gv_task(Runtime &runtime, uint64_t &candidate_task_id,
|
bool find_pending_pre_gv_task(Runtime &runtime, uint64_t &candidate_task_id,
|
||||||
uint64_t &matcher_task_id) {
|
uint64_t &matcher_task_id) {
|
||||||
candidate_task_id = 0;
|
candidate_task_id = 0;
|
||||||
|
|
@ -2422,11 +2456,13 @@ bool run_existing_geometry(Runtime &runtime) {
|
||||||
size_t match_results_before = 0;
|
size_t match_results_before = 0;
|
||||||
uint64_t match_frontier_before = 0;
|
uint64_t match_frontier_before = 0;
|
||||||
uint64_t pending_geometry_task_id = 0;
|
uint64_t pending_geometry_task_id = 0;
|
||||||
|
uint64_t pending_track_task_id = 0;
|
||||||
if (!collect_evidence(runtime, 0, before) ||
|
if (!collect_evidence(runtime, 0, before) ||
|
||||||
!count_exact_gvrs(runtime, fingerprint, exact_before) ||
|
!count_exact_gvrs(runtime, fingerprint, exact_before) ||
|
||||||
!match_result_frontier(runtime, match_results_before,
|
!match_result_frontier(runtime, match_results_before,
|
||||||
match_frontier_before) ||
|
match_frontier_before) ||
|
||||||
!find_pending_geometry_task(runtime, pending_geometry_task_id)) {
|
!find_pending_geometry_task(runtime, pending_geometry_task_id,
|
||||||
|
pending_track_task_id)) {
|
||||||
stop_runtime(runtime);
|
stop_runtime(runtime);
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
@ -2439,17 +2475,23 @@ bool run_existing_geometry(Runtime &runtime) {
|
||||||
* call after a completed/audited production GV Task. */
|
* call after a completed/audited production GV Task. */
|
||||||
uint64_t geometry_task_id = pending_geometry_task_id;
|
uint64_t geometry_task_id = pending_geometry_task_id;
|
||||||
Lardon3DProjectRecoverySummary recovery{};
|
Lardon3DProjectRecoverySummary recovery{};
|
||||||
const char *geometry_action = pending_geometry_task_id == 0
|
const char *geometry_action = pending_track_task_id != 0
|
||||||
|
? "reused-completed"
|
||||||
|
: pending_geometry_task_id == 0
|
||||||
? "enqueued"
|
? "enqueued"
|
||||||
: "recovered";
|
: "recovered";
|
||||||
|
bool task_completed = true;
|
||||||
|
bool governor_admission_ok = true;
|
||||||
|
if (pending_track_task_id == 0) {
|
||||||
begin_geometric_evidence(runtime);
|
begin_geometric_evidence(runtime);
|
||||||
const bool task_completed = pending_geometry_task_id != 0
|
task_completed = pending_geometry_task_id != 0
|
||||||
? recover_geometry_task(runtime, geometry_task_id, recovery)
|
? recover_geometry_task(runtime, geometry_task_id, recovery)
|
||||||
: lardon3d_project_enqueue_geometric_verifier_task(
|
: lardon3d_project_enqueue_geometric_verifier_task(
|
||||||
&runtime.state, &configuration, &geometry_task_id) &&
|
&runtime.state, &configuration, &geometry_task_id) &&
|
||||||
wait_completed(runtime, geometry_task_id,
|
wait_completed(runtime, geometry_task_id,
|
||||||
"geometric_verifier.run current existing");
|
"geometric_verifier.run current existing");
|
||||||
const bool governor_admission_ok = end_geometric_evidence(runtime);
|
governor_admission_ok = end_geometric_evidence(runtime);
|
||||||
|
}
|
||||||
if (!task_completed || !governor_admission_ok) {
|
if (!task_completed || !governor_admission_ok) {
|
||||||
std::printf("{\"record\":\"existing_geometry_summary\",\"ok\":false,"
|
std::printf("{\"record\":\"existing_geometry_summary\",\"ok\":false,"
|
||||||
"\"stage\":\"geometric_verifier.run\",\"task_id\":%llu,"
|
"\"stage\":\"geometric_verifier.run\",\"task_id\":%llu,"
|
||||||
|
|
@ -2480,11 +2522,13 @@ bool run_existing_geometry(Runtime &runtime) {
|
||||||
!match_result_frontier(runtime, match_results_after,
|
!match_result_frontier(runtime, match_results_after,
|
||||||
match_frontier_after) ||
|
match_frontier_after) ||
|
||||||
!audit_match_results(runtime, matcher_task_id, match_audit) ||
|
!audit_match_results(runtime, matcher_task_id, match_audit) ||
|
||||||
|
(pending_track_task_id == 0 &&
|
||||||
lardon3d_project_db_load_task(runtime.state.project_db, geometry_task_id,
|
lardon3d_project_db_load_task(runtime.state.project_db, geometry_task_id,
|
||||||
&durable_task) != LARDON3D_PROJECT_DB_OK ||
|
&durable_task) != LARDON3D_PROJECT_DB_OK) ||
|
||||||
|
(pending_track_task_id == 0 &&
|
||||||
lardon3d_project_db_load_geometric_verifier_task(
|
lardon3d_project_db_load_geometric_verifier_task(
|
||||||
runtime.state.project_db, geometry_task_id,
|
runtime.state.project_db, geometry_task_id,
|
||||||
&durable_geometry) != LARDON3D_PROJECT_DB_OK ||
|
&durable_geometry) != LARDON3D_PROJECT_DB_OK) ||
|
||||||
exact_after != applicable || verified + rejected != applicable ||
|
exact_after != applicable || verified + rejected != applicable ||
|
||||||
match_results_before != before.matches ||
|
match_results_before != before.matches ||
|
||||||
match_results_after != after.matches ||
|
match_results_after != after.matches ||
|
||||||
|
|
@ -2492,10 +2536,11 @@ bool run_existing_geometry(Runtime &runtime) {
|
||||||
before.features != after.features || before.pairs != after.pairs ||
|
before.features != after.features || before.pairs != after.pairs ||
|
||||||
before.matches != after.matches ||
|
before.matches != after.matches ||
|
||||||
match_frontier_before != match_frontier_after ||
|
match_frontier_before != match_frontier_after ||
|
||||||
durable_task.saved_state != TASK_COMPLETED ||
|
(pending_track_task_id == 0 &&
|
||||||
|
(durable_task.saved_state != TASK_COMPLETED ||
|
||||||
durable_task.recovery_state != TASK_COMPLETED ||
|
durable_task.recovery_state != TASK_COMPLETED ||
|
||||||
durable_task.progress != 100 ||
|
durable_task.progress != 100 ||
|
||||||
durable_geometry.after_match_result_id != match_frontier_after) {
|
durable_geometry.after_match_result_id != match_frontier_after))) {
|
||||||
std::printf("{\"record\":\"existing_geometry_summary\",\"ok\":false,"
|
std::printf("{\"record\":\"existing_geometry_summary\",\"ok\":false,"
|
||||||
"\"stage\":\"current_evidence_audit\",\"task_errors\":1}\n");
|
"\"stage\":\"current_evidence_audit\",\"task_errors\":1}\n");
|
||||||
stop_runtime(runtime);
|
stop_runtime(runtime);
|
||||||
|
|
@ -2508,6 +2553,15 @@ bool run_existing_geometry(Runtime &runtime) {
|
||||||
// reuses an existing Track Set instead of publishing duplicate lineage.
|
// reuses an existing Track Set instead of publishing duplicate lineage.
|
||||||
uint64_t track_task_id = 0;
|
uint64_t track_task_id = 0;
|
||||||
Lardon3DProjectDbTrackSet track_set{};
|
Lardon3DProjectDbTrackSet track_set{};
|
||||||
|
Lardon3DProjectRecoverySummary track_recovery{};
|
||||||
|
if (!runtime.stop_after_gv && pending_track_task_id != 0 &&
|
||||||
|
!recover_track_builder_task(runtime, pending_track_task_id,
|
||||||
|
track_recovery)) {
|
||||||
|
std::printf("{\"record\":\"existing_geometry_summary\",\"ok\":false,"
|
||||||
|
"\"stage\":\"track_builder.run recovered\",\"task_errors\":1}\n");
|
||||||
|
stop_runtime(runtime);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
if (!runtime.stop_after_gv &&
|
if (!runtime.stop_after_gv &&
|
||||||
!build_tracks(runtime, verified_ids, fingerprint, &track_task_id,
|
!build_tracks(runtime, verified_ids, fingerprint, &track_task_id,
|
||||||
&track_set)) {
|
&track_set)) {
|
||||||
|
|
@ -2516,6 +2570,7 @@ bool run_existing_geometry(Runtime &runtime) {
|
||||||
stop_runtime(runtime);
|
stop_runtime(runtime);
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
if (pending_track_task_id != 0) track_task_id = pending_track_task_id;
|
||||||
if (!runtime.stop_after_gv) {
|
if (!runtime.stop_after_gv) {
|
||||||
Evidence after_tracks{};
|
Evidence after_tracks{};
|
||||||
if (!collect_evidence(runtime, 0, after_tracks) ||
|
if (!collect_evidence(runtime, 0, after_tracks) ||
|
||||||
|
|
|
||||||
|
|
@ -349,6 +349,22 @@ int main() {
|
||||||
geometric_reset.geometric_maximum_process_rss_bytes == 0);
|
geometric_reset.geometric_maximum_process_rss_bytes == 0);
|
||||||
geometric_reset.geometric_evidence_active = false;
|
geometric_reset.geometric_evidence_active = false;
|
||||||
|
|
||||||
|
Lardon3DProjectRecoveryEntry recovery_entry{};
|
||||||
|
recovery_entry.status = LARDON3D_PROJECT_RECOVERABLE;
|
||||||
|
recovery_entry.snapshot.recovery_state = TASK_PENDING;
|
||||||
|
std::snprintf(recovery_entry.task_kind, sizeof(recovery_entry.task_kind),
|
||||||
|
"%s", LARDON3D_TRACK_BUILDER_TASK_KIND);
|
||||||
|
recovery_entry.task_kind_version = LARDON3D_TRACK_BUILDER_TASK_KIND_VERSION;
|
||||||
|
GeometryRecoveryKind recovery_kind{};
|
||||||
|
CHECK(geometry_recovery_kind(recovery_entry, recovery_kind));
|
||||||
|
CHECK(recovery_kind == GeometryRecoveryKind::kTrackBuilder);
|
||||||
|
++recovery_entry.task_kind_version;
|
||||||
|
CHECK(!geometry_recovery_kind(recovery_entry, recovery_kind));
|
||||||
|
std::snprintf(recovery_entry.task_kind, sizeof(recovery_entry.task_kind),
|
||||||
|
"unknown.task");
|
||||||
|
recovery_entry.task_kind_version = 1;
|
||||||
|
CHECK(!geometry_recovery_kind(recovery_entry, recovery_kind));
|
||||||
|
|
||||||
Options options;
|
Options options;
|
||||||
CHECK(parse_case({"runner", "--resume-pre-gv-existing", "--project-dir",
|
CHECK(parse_case({"runner", "--resume-pre-gv-existing", "--project-dir",
|
||||||
"/tmp/not-opened"}, options));
|
"/tmp/not-opened"}, options));
|
||||||
|
|
|
||||||
|
|
@ -179,6 +179,16 @@ void test_corruption_and_immutability() {
|
||||||
LARDON3D_TRACK_BUILDER_INVALID_ARGUMENT);
|
LARDON3D_TRACK_BUILDER_INVALID_ARGUMENT);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void test_isolated_divergent_metadata_regression() {
|
||||||
|
const Observation o1 = observation(1, 0, 10, 1);
|
||||||
|
const Observation o2 = observation(1, 1, 10, 2);
|
||||||
|
const Observation o3 = observation(2, 0, 11, 1);
|
||||||
|
/* Public ABI contract: metadata is tied to an observation identity and is
|
||||||
|
* checked per connected component. Divergent metadata on isolated O2 must
|
||||||
|
* not poison the valid O1--O3 component. */
|
||||||
|
expect({o1, o2, o3}, {edge(o1, o3)}, {{{1, 0}, {2, 0}}});
|
||||||
|
}
|
||||||
|
|
||||||
void test_additional_adversarial() {
|
void test_additional_adversarial() {
|
||||||
const Observation a = observation(1, 0, 10);
|
const Observation a = observation(1, 0, 10);
|
||||||
const Observation b = observation(2, 0, 11);
|
const Observation b = observation(2, 0, 11);
|
||||||
|
|
@ -290,6 +300,7 @@ int main() {
|
||||||
test_fingerprint();
|
test_fingerprint();
|
||||||
test_adversarial();
|
test_adversarial();
|
||||||
test_corruption_and_immutability();
|
test_corruption_and_immutability();
|
||||||
|
test_isolated_divergent_metadata_regression();
|
||||||
test_additional_adversarial();
|
test_additional_adversarial();
|
||||||
test_exhaustive();
|
test_exhaustive();
|
||||||
test_large_and_repeatable();
|
test_large_and_repeatable();
|
||||||
|
|
|
||||||
|
|
@ -3,6 +3,7 @@
|
||||||
#include <cstdlib>
|
#include <cstdlib>
|
||||||
#include <cstring>
|
#include <cstring>
|
||||||
#include <chrono>
|
#include <chrono>
|
||||||
|
#include <dirent.h>
|
||||||
#include <fcntl.h>
|
#include <fcntl.h>
|
||||||
#include <openssl/evp.h>
|
#include <openssl/evp.h>
|
||||||
#include <sqlite3.h>
|
#include <sqlite3.h>
|
||||||
|
|
@ -23,6 +24,14 @@ extern "C" {
|
||||||
#include <lardon3d/track_builder_project.h>
|
#include <lardon3d/track_builder_project.h>
|
||||||
}
|
}
|
||||||
|
|
||||||
|
namespace lardon3d::track_builder_internal {
|
||||||
|
using Checkpoint = bool (*)(void *userdata);
|
||||||
|
Lardon3DTrackBuilderProjectStatus build_project(
|
||||||
|
const Lardon3DTrackBuilderProjectRequest *request,
|
||||||
|
Lardon3DTrackBuilderProjectResult *result, Checkpoint checkpoint,
|
||||||
|
void *checkpoint_userdata);
|
||||||
|
}
|
||||||
|
|
||||||
namespace {
|
namespace {
|
||||||
constexpr Lardon3DGeometricVerifierKind kVerifier = LARDON3D_GEOMETRIC_VERIFIER_FUNDAMENTAL;
|
constexpr Lardon3DGeometricVerifierKind kVerifier = LARDON3D_GEOMETRIC_VERIFIER_FUNDAMENTAL;
|
||||||
constexpr uint32_t kVersion = 1;
|
constexpr uint32_t kVersion = 1;
|
||||||
|
|
@ -34,6 +43,8 @@ std::string g_insert_db;
|
||||||
uint64_t g_insert_match_id = 0;
|
uint64_t g_insert_match_id = 0;
|
||||||
bool g_insert_during_build = false;
|
bool g_insert_during_build = false;
|
||||||
bool g_publication_race = false;
|
bool g_publication_race = false;
|
||||||
|
uint64_t g_publication_nodes = 0;
|
||||||
|
uint64_t g_publication_capacity_bytes = 0;
|
||||||
|
|
||||||
void check(bool value, const char *expression, int line) {
|
void check(bool value, const char *expression, int line) {
|
||||||
if (!value) {
|
if (!value) {
|
||||||
|
|
@ -654,16 +665,25 @@ void run_resource_case(size_t gvr_count = 13) {
|
||||||
auto started = std::chrono::steady_clock::now();
|
auto started = std::chrono::steady_clock::now();
|
||||||
report_rss("before-build");
|
report_rss("before-build");
|
||||||
uint64_t expected_raw = static_cast<uint64_t>(gvr_count) * 8192ULL;
|
uint64_t expected_raw = static_cast<uint64_t>(gvr_count) * 8192ULL;
|
||||||
CHECK(lardon3d_track_builder_build_project(&request, &result) ==
|
const auto build_status = lardon3d_track_builder_build_project(&request, &result);
|
||||||
LARDON3D_TRACK_BUILDER_PROJECT_OK && result.raw_inlier_edge_count == expected_raw &&
|
if (build_status != LARDON3D_TRACK_BUILDER_PROJECT_OK ||
|
||||||
|
result.raw_inlier_edge_count != expected_raw ||
|
||||||
|
result.core_observation_count != 16384 || result.track_count != 8192)
|
||||||
|
std::fprintf(stderr, "resource build status=%d raw=%llu nodes=%llu tracks=%llu\n",
|
||||||
|
static_cast<int>(build_status),
|
||||||
|
static_cast<unsigned long long>(result.raw_inlier_edge_count),
|
||||||
|
static_cast<unsigned long long>(result.core_observation_count),
|
||||||
|
static_cast<unsigned long long>(result.track_count));
|
||||||
|
CHECK(build_status == LARDON3D_TRACK_BUILDER_PROJECT_OK &&
|
||||||
|
result.raw_inlier_edge_count == expected_raw &&
|
||||||
result.core_observation_count == 16384 && result.track_count == 8192);
|
result.core_observation_count == 16384 && result.track_count == 8192);
|
||||||
report_rss("after-build");
|
report_rss("after-build");
|
||||||
auto elapsed = std::chrono::duration<double>(std::chrono::steady_clock::now() - started);
|
auto elapsed = std::chrono::duration<double>(std::chrono::steady_clock::now() - started);
|
||||||
struct rusage usage{};
|
struct rusage usage{};
|
||||||
CHECK(getrusage(RUSAGE_SELF, &usage) == 0);
|
CHECK(getrusage(RUSAGE_SELF, &usage) == 0);
|
||||||
uint64_t predicted = 4ULL * 1024ULL * 1024ULL +
|
uint64_t predicted = 0;
|
||||||
result.raw_inlier_edge_count * (48ULL + 2ULL * 160ULL);
|
CHECK(lardon3d_track_builder_task_memory_estimate(
|
||||||
predicted *= result.raw_inlier_edge_count > 400000ULL ? 8ULL : 2ULL;
|
result.raw_inlier_edge_count, 2, &predicted));
|
||||||
std::printf("RESOURCE GVR=%zu RAW=%llu INLIERS=%llu FEATURES=3 OBS=%llu TRACKS=%llu "
|
std::printf("RESOURCE GVR=%zu RAW=%llu INLIERS=%llu FEATURES=3 OBS=%llu TRACKS=%llu "
|
||||||
"PREDICTED_RESERVATION=%llu\n",
|
"PREDICTED_RESERVATION=%llu\n",
|
||||||
ids.size(), static_cast<unsigned long long>(result.raw_inlier_edge_count),
|
ids.size(), static_cast<unsigned long long>(result.raw_inlier_edge_count),
|
||||||
|
|
@ -961,6 +981,251 @@ void run_pause_cancel_case() {
|
||||||
std::puts("PAUSE/CANCEL: PASS");
|
std::puts("PAUSE/CANCEL: PASS");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
struct LateRefusal {
|
||||||
|
size_t calls = 0;
|
||||||
|
size_t refuse_at = 0;
|
||||||
|
};
|
||||||
|
|
||||||
|
bool refuse_checkpoint(void *userdata) {
|
||||||
|
auto *state = static_cast<LateRefusal *>(userdata);
|
||||||
|
++state->calls;
|
||||||
|
return state->calls != state->refuse_at;
|
||||||
|
}
|
||||||
|
|
||||||
|
void run_late_refusal_boundaries_case() {
|
||||||
|
Fixture fixture;
|
||||||
|
const uint64_t gvr = fixture.add_gvr(0, 1, {{0, 0, 0.1F}}, {0x01});
|
||||||
|
const uint64_t ids[] = {gvr};
|
||||||
|
const auto request = fixture.request(ids, 1);
|
||||||
|
|
||||||
|
for (size_t refusal : {2U, 3U}) {
|
||||||
|
LateRefusal checkpoint{0, refusal};
|
||||||
|
Lardon3DTrackBuilderProjectResult interrupted{};
|
||||||
|
CHECK(lardon3d::track_builder_internal::build_project(
|
||||||
|
&request, &interrupted, refuse_checkpoint, &checkpoint) ==
|
||||||
|
LARDON3D_TRACK_BUILDER_PROJECT_INTERRUPTED &&
|
||||||
|
checkpoint.calls == refusal && track_set_count(fixture.db_path) == 0 &&
|
||||||
|
table_count(fixture.db_path, "tracks") == 0 &&
|
||||||
|
table_count(fixture.db_path, "track_observations") == 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
Lardon3DTrackBuilderProjectResult completed{};
|
||||||
|
CHECK(lardon3d_track_builder_build_project(&request, &completed) ==
|
||||||
|
LARDON3D_TRACK_BUILDER_PROJECT_OK &&
|
||||||
|
!completed.reused && completed.track_count == 1 &&
|
||||||
|
track_set_count(fixture.db_path) == 1 &&
|
||||||
|
table_count(fixture.db_path, "tracks") == 1 &&
|
||||||
|
table_count(fixture.db_path, "track_observations") == 2);
|
||||||
|
assert_two_observation_track(fixture.db, completed.track_set_id,
|
||||||
|
fixture.feature[0], fixture.feature[1]);
|
||||||
|
std::puts("LATE REFUSAL: PASS (post-DSU and pre-publication remain zero, retry exact)");
|
||||||
|
}
|
||||||
|
|
||||||
|
void run_s21_admission_estimate_case() {
|
||||||
|
constexpr uint64_t edges = 6628174ULL;
|
||||||
|
constexpr uint64_t capacity_after_canonical_reserve = 12750811136ULL;
|
||||||
|
constexpr uint64_t historical_envelope = 19546898688ULL;
|
||||||
|
uint64_t compact = 0;
|
||||||
|
CHECK(lardon3d_track_builder_task_memory_estimate(edges, 0, &compact));
|
||||||
|
CHECK(compact == 1932981756ULL && compact < capacity_after_canonical_reserve &&
|
||||||
|
historical_envelope > capacity_after_canonical_reserve);
|
||||||
|
CHECK(!lardon3d_track_builder_task_memory_estimate(
|
||||||
|
static_cast<uint64_t>(UINT32_MAX) / 2U + 1U, 0, &compact));
|
||||||
|
CHECK(!lardon3d_track_builder_task_memory_estimate(edges, UINT64_MAX, &compact));
|
||||||
|
uint64_t with_match_peak = 0;
|
||||||
|
CHECK(lardon3d_track_builder_task_memory_estimate_with_match_peak(
|
||||||
|
edges, 0, LARDON3D_MATCH_FILE_MAX_MATCHES, &with_match_peak));
|
||||||
|
CHECK(with_match_peak == compact +
|
||||||
|
LARDON3D_MATCH_FILE_MAX_MATCHES * sizeof(Lardon3DMatchFileEntry));
|
||||||
|
CHECK(!lardon3d_track_builder_task_memory_estimate_with_match_peak(
|
||||||
|
0, 0, UINT64_MAX, &with_match_peak));
|
||||||
|
uint64_t adversarial_compact = 0, adversarial_peak = 0;
|
||||||
|
CHECK(lardon3d_track_builder_task_memory_estimate(1, 2, &adversarial_compact));
|
||||||
|
CHECK(lardon3d_track_builder_task_memory_estimate_with_match_peak(
|
||||||
|
1, 2, LARDON3D_MATCH_FILE_MAX_MATCHES, &adversarial_peak));
|
||||||
|
CHECK(adversarial_peak == adversarial_compact +
|
||||||
|
LARDON3D_MATCH_FILE_MAX_MATCHES * sizeof(Lardon3DMatchFileEntry));
|
||||||
|
Lardon3DHardwareProfile profile{};
|
||||||
|
char error[128]{};
|
||||||
|
CHECK(lardon3d_hardware_profile_detect(&profile, error, sizeof(error)));
|
||||||
|
profile.memory_total_bytes = capacity_after_canonical_reserve;
|
||||||
|
Lardon3DResourcePolicy policy{};
|
||||||
|
CHECK(lardon3d_resource_policy_default(&profile, &policy));
|
||||||
|
policy.system_memory_reserve_bytes = 0;
|
||||||
|
policy.emergency_memory_floor_bytes = 0;
|
||||||
|
Lardon3DResourceGovernor *governor =
|
||||||
|
lardon3d_resource_governor_create(&profile, &policy);
|
||||||
|
CHECK(governor != nullptr);
|
||||||
|
Lardon3DResourceSnapshot snapshot{};
|
||||||
|
CHECK(clock_gettime(CLOCK_MONOTONIC, &snapshot.captured_at) == 0);
|
||||||
|
snapshot.memory_available_bytes = capacity_after_canonical_reserve;
|
||||||
|
snapshot.memory_free_bytes = capacity_after_canonical_reserve;
|
||||||
|
Lardon3DResourceEstimate estimate = {
|
||||||
|
compact, 0, 0, 0, 1, 1, 1, 0, 1, LARDON3D_RESOURCE_TASK_CPU};
|
||||||
|
Lardon3DResourceDecision decision{};
|
||||||
|
Lardon3DResourceReservation *reservation = nullptr;
|
||||||
|
CHECK(lardon3d_resource_governor_reserve(
|
||||||
|
governor, &snapshot, &estimate, &decision, &reservation) &&
|
||||||
|
decision.kind == LARDON3D_RESOURCE_START && reservation != nullptr &&
|
||||||
|
lardon3d_resource_governor_release(governor, reservation));
|
||||||
|
lardon3d_resource_governor_destroy(governor);
|
||||||
|
|
||||||
|
/* WHY/CONTRACT: matches can greatly exceed inliers. At the exact boundary
|
||||||
|
* the legacy E/F subtotal would start, while truthful peak accounting must
|
||||||
|
* make the sole Governor refuse admission. */
|
||||||
|
profile.memory_total_bytes = adversarial_peak - 1U;
|
||||||
|
CHECK(lardon3d_resource_policy_default(&profile, &policy));
|
||||||
|
policy.system_memory_reserve_bytes = 0;
|
||||||
|
policy.emergency_memory_floor_bytes = 0;
|
||||||
|
governor = lardon3d_resource_governor_create(&profile, &policy);
|
||||||
|
CHECK(governor != nullptr);
|
||||||
|
snapshot.memory_available_bytes = adversarial_peak - 1U;
|
||||||
|
snapshot.memory_free_bytes = adversarial_peak - 1U;
|
||||||
|
estimate.memory_fixed_bytes = adversarial_peak;
|
||||||
|
reservation = nullptr;
|
||||||
|
CHECK(lardon3d_resource_governor_reserve(
|
||||||
|
governor, &snapshot, &estimate, &decision, &reservation) &&
|
||||||
|
decision.kind != LARDON3D_RESOURCE_START && reservation == nullptr);
|
||||||
|
lardon3d_resource_governor_destroy(governor);
|
||||||
|
std::puts("S21 ADMISSION: PASS (truthful checked envelope fits canonical reserve)");
|
||||||
|
}
|
||||||
|
|
||||||
|
void run_adversarial_match_peak_task_case() {
|
||||||
|
Fixture fixture(LARDON3D_MATCH_FILE_MAX_MATCHES);
|
||||||
|
std::vector<Lardon3DMatchFileEntry> entries(LARDON3D_MATCH_FILE_MAX_MATCHES);
|
||||||
|
for (uint32_t i = 0; i < LARDON3D_MATCH_FILE_MAX_MATCHES; ++i)
|
||||||
|
entries[i] = {i, i, 0.1F};
|
||||||
|
std::vector<unsigned char> mask(LARDON3D_MATCH_FILE_MAX_MATCHES / 8U, 0);
|
||||||
|
mask[0] = 0x01;
|
||||||
|
const uint64_t gvr = fixture.add_gvr(0, 1, entries, mask);
|
||||||
|
const uint64_t ids[] = {gvr};
|
||||||
|
Lardon3DHardwareProfile profile{};
|
||||||
|
char error[128]{};
|
||||||
|
CHECK(lardon3d_hardware_profile_detect(&profile, error, sizeof(error)));
|
||||||
|
Lardon3DResourcePolicy policy{};
|
||||||
|
CHECK(lardon3d_resource_policy_default(&profile, &policy));
|
||||||
|
Lardon3DResourceGovernor *governor =
|
||||||
|
lardon3d_resource_governor_create(&profile, &policy);
|
||||||
|
CHECK(governor != nullptr);
|
||||||
|
Lardon3DAppState state{};
|
||||||
|
lardon3d_app_state_init(&state);
|
||||||
|
state.project_loaded = true;
|
||||||
|
state.project_db = fixture.db;
|
||||||
|
state.resource_governor = governor;
|
||||||
|
std::snprintf(state.project_path, sizeof(state.project_path), "%s",
|
||||||
|
fixture.directory.c_str());
|
||||||
|
Lardon3DTrackBuilderTaskConfiguration configuration = {
|
||||||
|
state.project_path, fixture.db, kVerifier, kVersion, g_verifier, ids, 1};
|
||||||
|
uint64_t task_id = 0;
|
||||||
|
Lardon3DTask *task = lardon3d_project_create_track_builder_task(
|
||||||
|
&state, &configuration, &task_id);
|
||||||
|
uint64_t expected = 0;
|
||||||
|
Lardon3DResourceEstimate estimate{};
|
||||||
|
CHECK(task != nullptr && task_id != 0 &&
|
||||||
|
lardon3d_track_builder_task_memory_estimate_with_match_peak(
|
||||||
|
1, 2, LARDON3D_MATCH_FILE_MAX_MATCHES, &expected) &&
|
||||||
|
lardon3d_task_resource_estimate(task, &estimate) &&
|
||||||
|
estimate.memory_fixed_bytes == expected);
|
||||||
|
lardon3d_task_destroy(task);
|
||||||
|
lardon3d_resource_governor_destroy(governor);
|
||||||
|
std::puts("MATCH PEAK TASK: PASS (8192 matches, 1 inlier, exact creation estimate)");
|
||||||
|
}
|
||||||
|
|
||||||
|
void run_disjoint_publication_capacity_case() {
|
||||||
|
constexpr uint32_t track_count = 256;
|
||||||
|
Fixture fixture(track_count);
|
||||||
|
std::vector<Lardon3DMatchFileEntry> entries(track_count);
|
||||||
|
for (uint32_t i = 0; i < track_count; ++i) entries[i] = {i, i, 0.1F};
|
||||||
|
std::vector<unsigned char> mask(track_count / 8U, 0xffU);
|
||||||
|
const uint64_t gvr = fixture.add_gvr(0, 1, entries, mask);
|
||||||
|
const uint64_t ids[] = {gvr};
|
||||||
|
auto request = fixture.request(ids, 1);
|
||||||
|
g_publication_nodes = 0;
|
||||||
|
g_publication_capacity_bytes = 0;
|
||||||
|
Lardon3DTrackBuilderProjectResult result{};
|
||||||
|
CHECK(lardon3d_track_builder_build_project(&request, &result) ==
|
||||||
|
LARDON3D_TRACK_BUILDER_PROJECT_OK &&
|
||||||
|
result.track_count == track_count && result.core_observation_count == 2U * track_count);
|
||||||
|
CHECK(g_publication_nodes == 2U * track_count &&
|
||||||
|
g_publication_capacity_bytes == 44U * g_publication_nodes);
|
||||||
|
uint64_t estimate = 0;
|
||||||
|
CHECK(lardon3d_track_builder_task_memory_estimate(track_count, 2, &estimate));
|
||||||
|
uint64_t required_slots = g_publication_nodes + g_publication_nodes / 2U + 1U;
|
||||||
|
uint64_t slots = 8;
|
||||||
|
while (slots < required_slots) slots *= 2U;
|
||||||
|
const uint64_t legacy_undercharge = 4ULL * 1024ULL * 1024ULL +
|
||||||
|
85ULL * g_publication_nodes + 8ULL * track_count + 16ULL * slots + 640ULL * 2U;
|
||||||
|
CHECK(estimate == legacy_undercharge + 16ULL * g_publication_nodes);
|
||||||
|
|
||||||
|
Lardon3DHardwareProfile profile{};
|
||||||
|
char error[128]{};
|
||||||
|
CHECK(lardon3d_hardware_profile_detect(&profile, error, sizeof(error)));
|
||||||
|
profile.memory_total_bytes = estimate - 1U;
|
||||||
|
Lardon3DResourcePolicy policy{};
|
||||||
|
CHECK(lardon3d_resource_policy_default(&profile, &policy));
|
||||||
|
policy.system_memory_reserve_bytes = 0;
|
||||||
|
policy.emergency_memory_floor_bytes = 0;
|
||||||
|
Lardon3DResourceGovernor *governor = lardon3d_resource_governor_create(&profile, &policy);
|
||||||
|
CHECK(governor != nullptr);
|
||||||
|
Lardon3DResourceSnapshot snapshot{};
|
||||||
|
CHECK(clock_gettime(CLOCK_MONOTONIC, &snapshot.captured_at) == 0);
|
||||||
|
snapshot.memory_available_bytes = estimate - 1U;
|
||||||
|
snapshot.memory_free_bytes = estimate - 1U;
|
||||||
|
Lardon3DResourceEstimate resource = {
|
||||||
|
estimate, 0, 0, 0, 1, 1, 1, 0, 1, LARDON3D_RESOURCE_TASK_CPU};
|
||||||
|
Lardon3DResourceDecision decision{};
|
||||||
|
Lardon3DResourceReservation *reservation = nullptr;
|
||||||
|
CHECK(lardon3d_resource_governor_reserve(
|
||||||
|
governor, &snapshot, &resource, &decision, &reservation) &&
|
||||||
|
decision.kind != LARDON3D_RESOURCE_START && reservation == nullptr);
|
||||||
|
lardon3d_resource_governor_destroy(governor);
|
||||||
|
std::puts("PUBLICATION CAPACITY: PASS (256 disjoint tracks, truthful 44 B/node peak)");
|
||||||
|
}
|
||||||
|
|
||||||
|
bool checkpoints_directory_empty(const Fixture &fixture) {
|
||||||
|
DIR *directory = opendir((fixture.directory + "/.lardon3d/checkpoints").c_str());
|
||||||
|
CHECK(directory != nullptr);
|
||||||
|
bool empty = true;
|
||||||
|
while (dirent *entry = readdir(directory))
|
||||||
|
if (std::strcmp(entry->d_name, ".") != 0 && std::strcmp(entry->d_name, "..") != 0)
|
||||||
|
empty = false;
|
||||||
|
CHECK(closedir(directory) == 0);
|
||||||
|
return empty;
|
||||||
|
}
|
||||||
|
|
||||||
|
void run_creation_ownership_failures_case() {
|
||||||
|
for (const char *fault : {"LARDON3D_TRACK_BUILDER_TEST_FAIL_SCOPE_DIR_FSYNC",
|
||||||
|
"LARDON3D_TRACK_BUILDER_TEST_FAIL_INITIAL_PERSIST"}) {
|
||||||
|
Fixture fixture;
|
||||||
|
const uint64_t gvr = fixture.add_gvr(0, 1, {{0, 0, 0.1F}}, {0x01});
|
||||||
|
const uint64_t ids[] = {gvr};
|
||||||
|
Lardon3DHardwareProfile profile{};
|
||||||
|
char error[128]{};
|
||||||
|
CHECK(lardon3d_hardware_profile_detect(&profile, error, sizeof(error)));
|
||||||
|
Lardon3DResourcePolicy policy{};
|
||||||
|
CHECK(lardon3d_resource_policy_default(&profile, &policy));
|
||||||
|
Lardon3DResourceGovernor *governor =
|
||||||
|
lardon3d_resource_governor_create(&profile, &policy);
|
||||||
|
CHECK(governor != nullptr);
|
||||||
|
Lardon3DAppState state{};
|
||||||
|
lardon3d_app_state_init(&state);
|
||||||
|
state.project_loaded = true;
|
||||||
|
state.project_db = fixture.db;
|
||||||
|
state.resource_governor = governor;
|
||||||
|
std::snprintf(state.project_path, sizeof(state.project_path), "%s", fixture.directory.c_str());
|
||||||
|
Lardon3DTrackBuilderTaskConfiguration configuration = {
|
||||||
|
state.project_path, fixture.db, kVerifier, kVersion, g_verifier, ids, 1};
|
||||||
|
uint64_t task_id = 99;
|
||||||
|
CHECK(setenv(fault, "1", 1) == 0);
|
||||||
|
CHECK(lardon3d_project_create_track_builder_task(&state, &configuration, &task_id) == nullptr);
|
||||||
|
CHECK(unsetenv(fault) == 0 && task_id == 0 && checkpoints_directory_empty(fixture) &&
|
||||||
|
table_count(fixture.db_path, "tasks") == 0 &&
|
||||||
|
table_count(fixture.db_path, "track_builder_tasks") == 0);
|
||||||
|
lardon3d_resource_governor_destroy(governor);
|
||||||
|
}
|
||||||
|
std::puts("CREATION OWNERSHIP: PASS (post-rename and post-checkpoint rollback clean)");
|
||||||
|
}
|
||||||
|
|
||||||
} // namespace
|
} // namespace
|
||||||
|
|
||||||
#ifdef LARDON3D_TRACK_BUILDER_PROJECT_TESTING
|
#ifdef LARDON3D_TRACK_BUILDER_PROJECT_TESTING
|
||||||
|
|
@ -1008,6 +1273,12 @@ extern "C" void lardon3d_track_builder_project_test_before_publication(
|
||||||
CHECK(lardon3d_track_builder_build_project(request, &result) ==
|
CHECK(lardon3d_track_builder_build_project(request, &result) ==
|
||||||
LARDON3D_TRACK_BUILDER_PROJECT_OK && !result.reused);
|
LARDON3D_TRACK_BUILDER_PROJECT_OK && !result.reused);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
extern "C" void lardon3d_track_builder_project_test_publication_capacities(
|
||||||
|
uint64_t node_count, uint64_t serialization_capacity_bytes) {
|
||||||
|
g_publication_nodes = node_count;
|
||||||
|
g_publication_capacity_bytes = serialization_capacity_bytes;
|
||||||
|
}
|
||||||
#endif
|
#endif
|
||||||
|
|
||||||
int main() {
|
int main() {
|
||||||
|
|
@ -1036,9 +1307,14 @@ int main() {
|
||||||
run_scope_snapshot_case();
|
run_scope_snapshot_case();
|
||||||
run_partial_input_failure();
|
run_partial_input_failure();
|
||||||
run_resource_case();
|
run_resource_case();
|
||||||
|
run_s21_admission_estimate_case();
|
||||||
|
run_adversarial_match_peak_task_case();
|
||||||
|
run_disjoint_publication_capacity_case();
|
||||||
|
run_creation_ownership_failures_case();
|
||||||
run_durable_task_case();
|
run_durable_task_case();
|
||||||
run_late_identity_race_case();
|
run_late_identity_race_case();
|
||||||
run_pause_cancel_case();
|
run_pause_cancel_case();
|
||||||
|
run_late_refusal_boundaries_case();
|
||||||
run_crash_recovery_case();
|
run_crash_recovery_case();
|
||||||
std::puts("C01-C27 integration harness: PASS (C26 N/A; C27 Gate D closed)");
|
std::puts("C01-C27 integration harness: PASS (C26 N/A; C27 Gate D closed)");
|
||||||
return 0;
|
return 0;
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue