Compare commits

...

4 commits

5 changed files with 1351 additions and 50 deletions

View file

@ -125,8 +125,8 @@ are inferred from focal pixels, image resolution or baseline normalization.
## Reconstruction strategy ## Reconstruction strategy
**DECISION: incremental SfM with bounded local refinement and optional final **DECISION: incremental SfM followed by final per-component refinement.** It
global refinement.** It matches the expected sequential vehicle/phone capture, matches the expected sequential vehicle/phone capture,
allows unregistered images to remain visible as a scientific result, and keeps allows unregistered images to remain visible as a scientific result, and keeps
the active problem bounded. Global-only rotation/translation averaging would the active problem bounded. Global-only rotation/translation averaging would
add a larger initialization and robustness surface without a current project add a larger initialization and robustness surface without a current project
@ -192,27 +192,382 @@ only on geometry; multi-view Tracks use all valid observations rather than a
random pair. Robust observation dropping is deferred: v1 rejects the landmark random pair. Robust observation dropping is deferred: v1 rejects the landmark
as a whole, so Track identity and observation ownership remain simple. as a whole, so Track identity and observation ownership remain simple.
## Bundle Adjustment decision ## Gate E v1 — Final Bundle Adjustment decision
**DECISION: BA is required for a useful final reconstruction but is not part of **DECISION: Gate E v1 is a synchronous, independent final per-component Bundle
the first pure-geometry gate.** The later BA gate will use a block-sparse Adjustment applied as post-processing to a copy of the immutable final Gate D
camera/landmark problem with binary64 poses and points, fixed or explicitly result.** It consumes two caller-owned immutable views that must remain coherent
fingerprinted calibration variables, and a robust loss whose kind/scale belong for the complete call: that final Gate D result, and the same resolved
to scientific identity. Dense camera×landmark Jacobians are forbidden. observation/calibration view used to construct the scientific Gate D input. It
never mutates either view, never creates constraints between disconnected
components, preserves each component's independent gauge, and produces a
distinct in-memory BA result.
Ceres is not available in the current host pkg-config environment and is not a The Gate D result alone is authoritative for final components, registered
Lardon3D production dependency. It is a **NEW_CANDIDATE**, not silently added. cameras, initial poses, landmarks and the observations associated with each
Its sparse Schur solvers and block structure make it the leading BA study landmark. The second view only resolves an observation already published by
candidate; Eigen is host-available, while SuiteSparse is not detected. A later Gate D. Its canonical key is `(feature_set_id, feature_index)`; resolution must
gate must prove license, reproducibility, thread behavior, memory scaling and return the matching `image_id`, source keypoint `x,y` and immutable calibration,
fallback before adding Ceres. OpenCV remains appropriate for small relative and must also agree with the published Track and image identities. Missing,
pose/PnP/triangulation primitives, not as an implicit BA architecture. ambiguous, duplicate or inconsistent resolution is a Gate E input error. Gate E
must not use array position, proximity or another heuristic fallback, add an
observation, restore a rejected association or camera, or rerun incremental SfM.
The first BA implementation should be local-window BA after registration, Source keypoint coordinates are the Feature File binary32 `x,y` in decoded-image
followed by at most one explicitly admitted global BA at finalization. No two pixels, with top-left origin, +x right and +y down. Gate E converts them to
heavy global BAs run concurrently. Global BA is allowed to be deferred by the binary64 for computation; it does not treat them as already undistorted or
Resource Governor. The trigger, window selection, robust loss, convergence and normalized. Given `Xc = R_cw * Xw + t_cw`, define `xn = Xc.x / Xc.z`,
thread policy are configuration fields, not runtime identity. `yn = Xc.y / Xc.z`, and `r2 = xn*xn + yn*yn`. The canonical OpenCV-compatible
forward model is:
```text
radial = 1 + k1*r2 + k2*r2*r2
xd = xn*radial + 2*p1*xn*yn + p2*(r2 + 2*xn*xn)
yd = yn*radial + p1*(r2 + 2*yn*yn) + 2*p2*xn*yn
u = fx*xd + cx
v = fy*yd + cy
residual = [u - observed_x, v - observed_y]
```
The residual is therefore binary64 in source pixels and uses the complete
canonical calibration model. A private Gate D validation helper that omits
distortion does not redefine this contract and is not a precedent for Gate E.
The second immutable view is an explicit scientific input, not persistence,
Project DB integration, a loader, resolver subsystem, cache, handle or Resource
System.
### Scientific and numerical contract
Gate E processes every reconstructed Gate D component independently. It
resolves the selected observations, copies the component poses and landmarks
into a private working set, builds and solves one BA problem, validates the
complete candidate, then either publishes that candidate in the distinct Gate
E result or preserves the original Gate D component. No component constrains or
influences another component.
Gate E v1 optimizes only camera extrinsic rotations, camera centers and
landmark positions. The known `fx`, `fy`, `cx`, `cy`, `k1`, `k2`, `p1`, `p2`,
observations, Track membership, identities and observation/landmark
associations are fixed and immutable. Future intrinsic optimization requires a
separate scientific and identity decision.
The public boundary remains solver-independent and world-to-camera. The private
C++ adapter uses a unit quaternion for `R_cw`, with an appropriate quaternion
manifold, and world camera center `Cw`:
```text
Xc = R_cw * (Xw - Cw)
t_cw = -R_cw * Cw
```
Conversion to or from public rotation matrices canonicalizes quaternion sign,
so `q` and `-q` cannot produce distinct observable representations. No Ceres
type crosses the future C17 ABI.
### Gate E gauge
Gate E derives deterministic BA anchors from the final Gate D result and does
not depend on historical seed IDs. In each component, the registered camera
with the lowest `image_id` is the pose anchor; its complete initial Gate D
rotation and camera center are fixed.
Among the other registered cameras, the scale anchor is the camera whose
binary64 Euclidean distance from the pose anchor is greatest. An exact distance
tie selects the lowest `image_id`; no hidden tolerance participates. For
`delta = C_scale - C_anchor`, the coordinate with greatest absolute value is
the scale axis, with exact ties resolved X, then Y, then Z. That one initial
Gate D coordinate of `C_scale` is fixed. Its other two center coordinates and
its rotation remain variable. The fixed pose removes the six rigid degrees of
freedom and the fixed nonzero scale coordinate removes the scale degree of
freedom without fixing a second pose.
The scale anchor is degenerate when:
```text
max(abs(delta.x), abs(delta.y), abs(delta.z)) <= 1e-9
```
Gate D fixes each valid component to a unit seed baseline, so `1e-9` world
units is a numerically negligible separation in that scientific gauge. A
component with no second valid camera or a degenerate scale anchor is not
optimized; its Gate D data is retained with a gauge/degenerate diagnostic.
### Objective and solver
Every valid observation contributes one two-dimensional source-pixel residual
block `f_i = [dx, dy]`, where `dx = predicted_x - observed_x` and
`dy = predicted_y - observed_y`. Gate E uses binary64 throughout. Define:
```text
s_i = dx*dx + dy*dy
delta = 2.0
delta2 = 4.0
rho_delta(s) = s if s <= delta2
rho_delta(s) = 2*delta*sqrt(s) - delta2 if s > delta2
robust_cost = 0.5 * sum_i(rho_delta(s_i))
```
Thus, with `delta = 2.0` source pixels, the second branch is
`4.0*sqrt(s) - 4.0`. The Huber loss applies once to the norm squared of the
complete 2D observation, never independently to `dx` and `dy`; the factor
`0.5` is contractual. Each observation must likewise be one 2D Ceres residual
block, not two scalar blocks.
Lardon3D computes initial and final robust costs independently of the solver
using exactly this formula, and those values govern acceptance. Ceres summary
costs may only diagnose or cross-check them. With the identical problem, a
disagreement beyond the applicable numerical tolerance stops implementation
for contract review; neither value silently replaces the other. Non-finite
`dx`, `dy`, `s_i`, `rho_delta(s_i)`, accumulation, pose, landmark or projection,
and camera-frame depth invalid under the frozen camera invariants, reject the
candidate. No clamp or fallback is permitted.
The Huber kind and scale are Gate E scientific policy, not Governor parameters,
Resource parameters or a Gate E fingerprint.
Gate E v1 selects the Ceres Solver 2.2.x API, CPU-only, with these explicit
options:
```text
minimizer_type = TRUST_REGION
trust_region_strategy_type = LEVENBERG_MARQUARDT
linear_solver_type = ITERATIVE_SCHUR
preconditioner_type = SCHUR_JACOBI
num_threads = 1
max_num_iterations = 50
function_tolerance = 1e-6
gradient_tolerance = 1e-10
parameter_tolerance = 1e-8
```
Landmark parameter blocks form elimination group 0 in increasing canonical
Track/landmark identity; camera blocks form group 1 in increasing `image_id`.
Components, cameras, landmarks, observations and residual blocks are all built
in canonical identity order. Automatic Ceres ordering is not used when the API
accepts an explicit ordering.
There is exactly one solver attempt per eligible component, with no automatic
retry, wall-clock timeout or `max_solver_time`. Environment variables, hardware
profiles, Tasks, schedulers and the Governor cannot change the single-thread
reference. `ITERATIVE_SCHUR` with `SCHUR_JACOBI` provides the required
block-sparse path without a functional SuiteSparse dependency; `SPARSE_SCHUR`,
CUDA and GPU execution are not Gate E v1.
### Eligibility, bounds and acceptance
An eligible component has at least two registered cameras, at least one valid
BA landmark, exactly resolved observations and calibrations, finite inputs, a
valid gauge, overflow-safe dimensions and no manifest underconstraint after
the anchors. Gate D already guarantees multi-view support for every published
landmark, so Gate E introduces no separate support threshold.
For a component with a valid non-degenerate Gate E gauge, let `C` be its
registered camera count, `P` its optimized landmark count and `O` its retained,
resolved observation count. Camera intrinsics and distortion are fixed. The
free tangent dimension is therefore:
```text
free_dof = 6*C + 3*P - 7
scalar_residual_count = 2*O
```
The completely fixed pose anchor removes six degrees of freedom, and the fixed
scale-anchor center coordinate removes one. Gate E v1 defines **manifest
underconstraint** as at least one of these exact structural conditions:
- **UC1:** `2*O < 6*C + 3*P - 7`, using overflow-checked integer arithmetic;
- **UC2:** an optimized landmark is observed by fewer than two distinct
registered cameras;
- **UC3:** an optimizable camera, including the scale anchor but excluding the
completely fixed pose anchor, observes fewer than three distinct landmarks;
- **UC4:** the bipartite camera-landmark optimization graph is not one connected
component containing the pose anchor.
E19 evaluates UC1--UC4 only after the existing structural validation and valid
anchor selection. E18 remains the existing insufficient-camera case and is not
redefined by E19. These conditions are necessary structural checks, not proof
of full numerical rank. Gate E v1 performs no numerical rank estimate, SVD,
singular-value or condition-number threshold, Jacobian/Hessian rank epsilon, or
Ceres covariance/rank heuristic for E19. Geometry that passes UC1--UC4 can
still be rejected by the existing projection, solver termination, finite-value,
cost non-regression and atomic-publication contracts.
UC1--UC4 are deterministic and solver-independent. Their implementation uses
the existing canonical flat Gate E working set and temporary storage bounded by
`O(C + P + O)` or better. It uses no hash-order dependency, dense `C * P`
storage, materialized rank matrix or new Resource subsystem.
Gate E retains the identically-scoped Gate D bounds of at most 4096 registered
cameras, 250,000 Tracks/landmarks and 1,000,000 observations. The Gate D
landmarks-per-growth-round bound is not a Gate E bound. All allocation and
dimension arithmetic is overflow-checked. The architecture is block-sparse;
no dense camera-count × landmark-count allocation or Jacobian is permitted.
Ceres `NO_CONVERGENCE` is rejection even if an intermediate candidate has
lower cost. Only a termination classified as successful convergence by the
private Ceres adapter is acceptable. The robust-cost comparison uses exactly:
```text
cost_tolerance = 1e-12 * max(1.0, abs(initial_robust_cost))
final_robust_cost <= initial_robust_cost + cost_tolerance
```
This comparison tolerance absorbs insignificant binary64 noise and is not a
Ceres convergence tolerance. A component is published only when its inputs are
coherent and eligible, termination is accepted, all candidate poses,
landmarks, required projections and robust costs are finite, both gauge anchors
are strictly preserved in their contract representations, the cost condition
holds, and no consumed frozen invariant is violated. Otherwise the original
Gate D component is preserved exactly and accompanied by a rejection
diagnostic. All optimization occurs on a private copy, so publication is atomic
per component and requires no in-place rollback.
### Result and diagnostics
The future solver-independent Gate E result has these conceptual states:
- `COMPLETE`: at least one component is eligible and every eligible component
is optimized and accepted;
- `PARTIAL`: at least one component is accepted and at least one other eligible
component is rejected or fails;
- `FAILED`: no eligible component produces an accepted BA result, including an
input with no eligible component.
`Lardon3DSparseBundleAdjustmentStatus` contains only these three scientific
result states. In particular, `FAILED` is not an invalid-argument,
out-of-memory or internal execution error.
The future synchronous execution function returns the separate,
solver-independent `Lardon3DSparseBundleAdjustmentExecutionStatus`:
```text
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_EXECUTION_OK
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_EXECUTION_INVALID_ARGUMENT
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_EXECUTION_OUT_OF_MEMORY
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_EXECUTION_INTERNAL_ERROR
```
`EXECUTION_OK` means the public input was structurally valid, Gate E reached a
complete scientific decision and produced the owned result. Its scientific
status may be `COMPLETE`, `PARTIAL` or `FAILED`; `EXECUTION_OK` with scientific
`FAILED` is valid and means that no eligible component was accepted.
`EXECUTION_INVALID_ARGUMENT` covers a violated public input contract, including
pointer/count, bounds, identity, finiteness, observation-resolution or
Gate-D/result-view coherence failures. `EXECUTION_OUT_OF_MEMORY` covers an
allocation failure, including `std::bad_alloc` caught at the C/C++ boundary,
that prevents production of a complete scientific result.
`EXECUTION_INTERNAL_ERROR` is reserved for an unexpected internal failure that
prevents safe completion; it is not a component-rejection fallback. Normal
component rejection for insufficient cameras, gauge degeneracy, manifest
underconstraint, invalid candidate projection, solver `NO_CONVERGENCE` or
`FAILURE`, a non-finite candidate or robust-cost regression contributes only to
the scientific `COMPLETE`/`PARTIAL`/`FAILED` result.
On every execution status other than `EXECUTION_OK`, the public result remains
in its canonical zero state: all counts are zero, all owned array and diagnostic
pointers are null, and destruction is safe. The execution function never
publishes a partial owned result and then returns an execution error.
Ineligible and rejected components retain their Gate D data. Each component
diagnostic contains at least component key, camera/landmark/observation counts,
pose-anchor and scale-anchor `image_id`, scale axis X/Y/Z, initial and final
robust cost, initial and final reprojection RMSE, iteration count, solver
termination class, accepted/rejected state and rejection reason. It exposes no
Ceres pointer or type.
Diagnostic reprojection RMSE is non-robust:
```text
sqrt(sum(dx*dx + dy*dy) / observation_count)
```
Acceptance remains based on robust cost and all contract invariants. Raw RMSE
is not required to improve universally in the presence of outliers.
### Reproducibility
For identical input, executable, build, dependency versions and machine with
one solver thread, component order, anchors, parameter/residual ordering,
states, accept/reject decisions and structural diagnostics are deterministic.
Comparable binary64 geometric scalars satisfy:
```text
abs(a - b) <= 1e-12 * max(1.0, abs(a), abs(b))
```
Rotations are compared geometrically rather than by raw quaternion sign. If
fresh-process tests in an identical environment cannot meet this tolerance,
implementation stops for contract review; tests must not widen it silently.
### Canonical Gate E validation matrix
| Case | Contract evidence |
|---|---|
| E01 Null/invalid input | Safe rejection; no exception crosses C |
| E02 Empty/non-eligible result | Deterministic `FAILED` with diagnostics |
| E03 Clean synthetic component | Finite accepted result, gauge held, cost non-regression |
| E04 Perturbed poses | Fixture-defined measurable improvement |
| E05 Perturbed landmarks | Fixture-defined measurable improvement |
| E06 Perturbed poses and landmarks | Convergence and fixture-defined improvement |
| E07 Noise 0.5 px | Finite accepted result or contractually justified rejection |
| E08 Noise 1.0 px | Finite accepted result or contractually justified rejection |
| E09 Noise 2.0 px | Finite accepted result or contractually justified rejection |
| E10 Outliers 10% | Huber active; finite result or clean rejection; no invariant violation |
| E11 Outliers 20% | Huber active; finite result or clean rejection; no invariant violation |
| E12 Outliers 40% | Huber active; finite result or clean rejection; no invariant violation |
| E13 Disconnected components | Independent optimization and gauges |
| E14 One success, one failure | Global `PARTIAL` |
| E15 Pose anchor | Initial rotation and center strictly preserved |
| E16 Scale anchor | Selected center coordinate strictly preserved |
| E17 Deterministic anchors | Exact distance/ID and X/Y/Z ties; `1e-9` degeneracy boundary |
| E18 Insufficient cameras | No solve; Gate D data retained |
| E19 Underconstrained geometry | No solve; Gate D data retained |
| E20 Non-finite input | Input rejection |
| E21 Non-finite projection candidate | Atomic candidate rejection |
| E22 Forced non-convergence | Private summary interpreter rejects `NO_CONVERGENCE` |
| E23 Candidate regression | Cost condition prevents publication |
| E24 Atomic rejection | Original component preserved exactly |
| E25 Canonical ordering | Explicit groups and parameter/residual order |
| E26 Same-process repeats | Structural equality and numeric tolerance |
| E27 Fresh-process repeats | At least 20 processes in one identical environment |
| E28 Ownership/destruction | Caller inputs retained; owned result safely destroyed |
| E29 Null/repeated destroy | Required only if E1 adopts the existing null-safe convention |
| E30 Allocation/overflow | Checked rejection before allocation |
| E31 Maximum boundary guards | Exact documented limits without a giant solve where isolatable |
| E32 Sparse architecture | No dense camera-count × landmark-count allocation |
| E33 Calibration immutability | Before/after identical |
| E34 Track/observation immutability | Before/after identical |
| E35 Gate D immutability | Input unchanged after success and every failure path |
E22 tests the private solver-summary-to-decision interpreter directly. It does
not expose an iteration override, add a production behavior for testing or
change `max_num_iterations = 50`. Synthetic ground-truth fixtures measure
pre/post geometric error and robust cost. Fixtures intended to improve define
their own scientifically measurable improvement; no universal pose or landmark
threshold is invented.
Local BA after registration is deferred. Gate D exposes no intermediate
scientific seam or complete registration history, and an interleaved BA could
change its subsequent growth. Introducing that policy requires a future
explicit scientific seam/version architecture decision; Gate E v1 does not
create or name such a version.
Gate E v1 remains independent of Project DB, Task Runtime, the Resource
Governor and any Resource System. It neither computes nor carries a parameter
fingerprint, defines no persistent identity, and publishes nothing. Gate F
retains project/task orchestration and persistence; Gate G retains Resource
Governor integration and final resource validation.
Ceres availability on a host must be distinguished from Lardon3D dependency
declaration. Gate E selects the Ceres Solver 2.2.x API scientifically, but
Lardon3D currently declares no production Ceres dependency in Meson. A future
dependency slice must verify the used API, licensing, CPU-only construction and
a build without required SuiteSparse or CUDA. Package discovery may use CMake;
a pkg-config miss alone does not prove host unavailability, and an installed
host package is not a declared Lardon3D dependency.
## Determinism and scientific identity ## Determinism and scientific identity
@ -267,13 +622,13 @@ working set. It forbids a dense `C×P`, `C×C` or co-visibility matrix. Track
length has no arbitrary 256 cap; long Tracks are iterated through checked length has no arbitrary 256 cap; long Tracks are iterated through checked
bounded storage. bounded storage.
Triangulation/registration are light CPU units and can be batched. Local BA is Triangulation/registration are light CPU units and can be batched. Gate E v1
bounded by an active camera/landmark window and needs a Governor reservation. uses local scientific limits for its final per-component BA and does not query
Global BA is one heavy job at a time, with explicit admission and a conservative the Governor. Future Gate G admission may use `C`, `P`, `O`, solver mode and
thread cap. Future reservation inputs are `C`, `P`, `O`, active window size, calibration-variable count without changing scientific results. The existing
solver mode and calibration-variable count. The existing Resource Governor Resource Governor owns RAM/PSI/swap policy; Sparse SfM adds no system-pressure
owns RAM/PSI/swap policy; Sparse SfM adds no thresholds. Swap is never normal thresholds. Swap is never normal working memory, and UMA RAM must preserve
working memory, and UMA RAM must preserve several GiB of desktop/iGPU headroom. several GiB of desktop/iGPU headroom.
## Hardware and probe study ## Hardware and probe study
@ -282,11 +637,10 @@ Gate A preflight measured 16 logical CPUs, `MemTotal=15597716 KiB`,
zram device, and zero current memory/IO PSI average. The host is the Ryzen 7 zram device, and zero current memory/IO PSI average. The host is the Ryzen 7
8845HS/Radeon 780M UMA target described by the performance document. 8845HS/Radeon 780M UMA target described by the performance document.
The project already links OpenCV 5.0.0. Eigen 5.0.1 and Ceres 3.12.0 are The project already links OpenCV 5.0.0. Host-installed libraries and their
available through host pkg-config but are not current production dependencies; pkg-config or CMake discovery metadata are capabilities, not Lardon3D
SuiteSparse/BLAS/LAPACK availability is host capability only. TBB 2023.1 is production dependencies. Lardon3D currently declares no Ceres dependency in
present through the existing OpenCV stack. No package, system setting, swap Meson. No package, system setting, swap device or GPU mode was changed.
device or GPU mode was changed.
Gate A probes use deterministic synthetic camera arcs, controlled noise and Gate A probes use deterministic synthetic camera arcs, controlled noise and
degenerate planar/pure-rotation cases. Every RSS probe is a separate normal degenerate planar/pure-rotation cases. Every RSS probe is a separate normal
@ -304,8 +658,9 @@ production Sparse SfM code is created by this gate.
deterministic seed, triangulation and PnP with synthetic ground truth. deterministic seed, triangulation and PnP with synthetic ground truth.
- **Gate D — Incremental core:** registration ordering, components, - **Gate D — Incremental core:** registration ordering, components,
unregistered-image policy and deterministic reconstruction output. unregistered-image policy and deterministic reconstruction output.
- **Gate E — BA integration:** sparse BA candidate, robust loss, local/global - **Gate E — Final Bundle Adjustment:** synchronous final per-component BA on a
policy, numerical reproducibility and solver dependency decision. copy of the immutable Gate D result, with its scientific and numerical
contract frozen here; interleaved local BA is deferred.
- **Gate F — Project orchestration:** explicit Track Set/calibration input, - **Gate F — Project orchestration:** explicit Track Set/calibration input,
atomic publication and durable runtime integration. atomic publication and durable runtime integration.
- **Gate G — Resource/freeze:** Governor admission, sustained hardware safety, - **Gate G — Resource/freeze:** Governor admission, sustained hardware safety,
@ -317,7 +672,7 @@ production Sparse SfM code is created by this gate.
Seed/order risk is controlled by deterministic policy. It is robust for Seed/order risk is controlled by deterministic policy. It is robust for
sequential capture, has canonical queues and seeds, moderate complexity, and sequential capture, has canonical queues and seeds, moderate complexity, and
sparse `C,T,O` scaling with local BA. **SELECTED v1.** sparse `C,T,O` scaling followed by final per-component BA. **SELECTED v1.**
### Global SfM ### Global SfM
@ -342,7 +697,7 @@ Triangulation candidates:
|---|---|---|---| |---|---|---|---|
| Dense normal equations | Prohibited for serious `C×P` problems | No | Rejected | | Dense normal equations | Prohibited for serious `C×P` problems | No | Rejected |
| OpenCV generic optimization | Not a sparse BA contract | Present, wrong abstraction | Rejected | | OpenCV generic optimization | Not a sparse BA contract | Present, wrong abstraction | Rejected |
| Ceres sparse Schur | Appropriate block structure | New candidate dependency | **Later-gate candidate** | | Ceres 2.2.x iterative Schur | Block-sparse | Scientific selection; not in Meson | **Selected Gate E v1** |
### Synthetic geometry probe ### Synthetic geometry probe
@ -365,23 +720,23 @@ checks before accepting a component.
The project already links OpenCV 5.0.0. Host probes found Eigen 5.0.1, BLAS The project already links OpenCV 5.0.0. Host probes found Eigen 5.0.1, BLAS
3.12.0, LAPACK 3.12.0 and TBB 2023.1 as host capabilities or transitive 3.12.0, LAPACK 3.12.0 and TBB 2023.1 as host capabilities or transitive
facilities rather than current Lardon3D production dependencies. Ceres and facilities rather than current Lardon3D production dependencies. Ceres may use
SuiteSparse are not available through the current host pkg-config environment; CMake discovery, so pkg-config alone does not establish host availability.
Ceres remains a new dependency candidate, not an installed fact. No new Ceres 2.2.x is the selected Gate E scientific API, but Lardon3D declares no
dependency is added by Gate A. The measured machine has 16 logical CPUs, production Ceres dependency yet. No new dependency is added by this contract
slice. The measured machine has 16 logical CPUs,
`MemTotal=15597716 KiB`, `MemAvailable=8245288 KiB` at preflight, an 8 GiB `MemTotal=15597716 KiB`, `MemAvailable=8245288 KiB` at preflight, an 8 GiB
swapfile, 6 GiB zram and zero memory/IO PSI averages at the probe start. A swapfile, 6 GiB zram and zero memory/IO PSI averages at the probe start. Gate E
single heavy BA and a future solver thread cap of 4 are the conservative uses one solver thread; future Gate G resource admission cannot change that
resource candidates; these are not yet Governor settings. scientific setting.
## Gate A unresolved boundaries ## Gate A unresolved boundaries
The following remain deliberately deferred to later gates rather than hidden: The following remain deliberately deferred rather than hidden: Ceres
exact numeric parallax/reprojection thresholds, Ceres licensing/dependency licensing/dependency integration, metric alignment, persistent orchestration
adoption, robust-loss scale, local-window selection, BA convergence criteria, and durable SfM checkpoints. Gate E freezes its own robust loss, convergence,
metric alignment, persistent reconstruction schema and durable SfM checkpoints. ordering and acceptance policy here without introducing persistence or a
Their semantic ownership is decided here; their final numeric values require fingerprint.
the synthetic ground-truth and sparse-solver gates.
## Gate C — pure calibrated geometry ## Gate C — pure calibrated geometry
@ -439,7 +794,7 @@ parameters and do not alter Project DB identity.
## Gate D — incremental Sparse SfM core ## Gate D — incremental Sparse SfM core
**GATE D — PASS.** Gate D is the first executable link **GATE D — PASS / FROZEN.** Gate D is the first executable link
between the immutable Track/Calibration contracts and the Gate C primitives. between the immutable Track/Calibration contracts and the Gate C primitives.
The reference implementation is synchronous, deterministic, CPU-only, The reference implementation is synchronous, deterministic, CPU-only,
in-memory, bounded and independent of Project DB, Task Runtime, Resource in-memory, bounded and independent of Project DB, Task Runtime, Resource

View file

@ -0,0 +1,113 @@
#ifndef LARDON3D_SPARSE_SFM_BUNDLE_ADJUSTMENT_H
#define LARDON3D_SPARSE_SFM_BUNDLE_ADJUSTMENT_H
#include <stdbool.h>
#include <stddef.h>
#include <stdint.h>
#include <lardon3d/sparse_sfm_incremental.h>
#ifdef __cplusplus
extern "C" {
#endif
/* COMPLETE means every eligible component was accepted, PARTIAL means accepted
* and rejected eligible components coexist, and FAILED means none was
* accepted. E1 defines the ABI but produces no final status before E2. */
typedef enum {
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_COMPLETE = 0,
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_PARTIAL,
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_FAILED
} Lardon3DSparseBundleAdjustmentStatus;
typedef enum {
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_AXIS_NONE = 0,
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_AXIS_X,
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_AXIS_Y,
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_AXIS_Z
} Lardon3DSparseBundleAdjustmentScaleAxis;
typedef enum {
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_TERMINATION_NOT_RUN = 0,
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_TERMINATION_CONVERGED,
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_TERMINATION_NO_CONVERGENCE,
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_TERMINATION_FAILURE
} Lardon3DSparseBundleAdjustmentTermination;
typedef enum {
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_REJECTION_NONE = 0,
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_REJECTION_INELIGIBLE,
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_REJECTION_GAUGE_DEGENERATE,
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_REJECTION_INPUT,
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_REJECTION_NONFINITE,
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_REJECTION_NO_CONVERGENCE,
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_REJECTION_SOLVER_FAILURE,
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_REJECTION_COST_REGRESSION
} Lardon3DSparseBundleAdjustmentRejectionReason;
/* Both views are caller-owned and immutable for the future synchronous Gate E
* call. incremental_result is authoritative for components, registered
* image_id values, world-to-camera poses, landmarks and final associations.
* images/observations are the same resolved scientific view used for Gate D.
* An observation is identified by (feature_set_id, feature_index); its image_id
* must agree with the final association. Source binary32 x/y become binary64
* source-image pixels, origin top-left, +x right and +y down. Calibrations are
* known, fixed and immutable. Each pointer is paired with its size_t count. */
typedef struct {
const Lardon3DSparseIncrementalResult *incremental_result;
const Lardon3DSparseIncrementalImage *images;
size_t image_count;
const Lardon3DSparseIncrementalObservation *observations;
size_t observation_count;
} Lardon3DSparseBundleAdjustmentInput;
/* Counts describe one component. pose_anchor_image_id fixes the complete pose;
* scale_anchor_image_id and scale_axis identify the fixed camera-center
* coordinate. Costs are 0.5*sum(Huber(dx*dx+dy*dy)) with delta 2 source pixels. RMSE is
* sqrt(sum(dx*dx+dy*dy)/observation_count) in source pixels. has_costs,
* has_rmse and has_anchors determine whether the corresponding fields are
* available; unavailable doubles are zero, never NaN sentinels. */
typedef struct {
uint64_t component_key;
uint64_t camera_count;
uint64_t landmark_count;
uint64_t observation_count;
bool eligible;
bool has_anchors;
uint64_t pose_anchor_image_id;
uint64_t scale_anchor_image_id;
Lardon3DSparseBundleAdjustmentScaleAxis scale_axis;
bool has_costs;
double initial_robust_cost;
double final_robust_cost;
bool has_rmse;
double initial_reprojection_rmse_px;
double final_reprojection_rmse_px;
uint32_t iteration_count;
Lardon3DSparseBundleAdjustmentTermination termination;
bool accepted;
Lardon3DSparseBundleAdjustmentRejectionReason rejection_reason;
} Lardon3DSparseBundleAdjustmentComponentDiagnostic;
/* Future E2 output. All arrays are owned by the result and ordered by the
* canonical component/image/Track/observation identities. Poses remain
* world-to-camera binary64. A destruction function is added with the E2 run;
* E1 intentionally exposes no fake execution function. */
typedef struct {
Lardon3DSparseBundleAdjustmentStatus status;
Lardon3DSparseIncrementalComponent *components;
size_t component_count;
Lardon3DSparseIncrementalCamera *cameras;
size_t camera_count;
Lardon3DSparseIncrementalLandmark *landmarks;
size_t landmark_count;
Lardon3DSparseIncrementalLandmarkObservation *observations;
size_t observation_count;
Lardon3DSparseBundleAdjustmentComponentDiagnostic *diagnostics;
} Lardon3DSparseBundleAdjustmentResult;
#ifdef __cplusplus
}
#endif
#endif

View file

@ -146,6 +146,7 @@ executable(
'src/project_db.c', 'src/project_db_sparse_sfm.c', 'src/project_db.c', 'src/project_db_sparse_sfm.c',
'src/sparse_sfm_geometry.cpp', 'src/sparse_sfm_geometry.cpp',
'src/sparse_sfm_incremental.cpp', 'src/sparse_sfm_incremental.cpp',
'src/sparse_sfm_bundle_adjustment.cpp',
'src/task.c', 'src/task.c',
'src/task_checkpoint.c', 'src/task_checkpoint.c',
'src/task_kind_registry.c', 'src/task_kind_registry.c',
@ -546,6 +547,19 @@ sparse_sfm_incremental_test = executable(
test('sparse-sfm-incremental', sparse_sfm_incremental_test, timeout: 60) test('sparse-sfm-incremental', sparse_sfm_incremental_test, timeout: 60)
sparse_sfm_bundle_adjustment_test = executable(
'test-sparse-sfm-bundle-adjustment',
sources: [
'tests/test_sparse_sfm_bundle_adjustment.cpp',
'src/sparse_sfm_bundle_adjustment.cpp',
],
cpp_args: ['-DLARDON3D_SPARSE_BUNDLE_ADJUSTMENT_TESTING'],
include_directories: include_directories('include'),
)
test('sparse-sfm-bundle-adjustment', sparse_sfm_bundle_adjustment_test,
timeout: 30)
executable( executable(
'benchmark-sparse-sfm-incremental', 'benchmark-sparse-sfm-incremental',
sources: [ sources: [

View file

@ -0,0 +1,463 @@
#include <lardon3d/sparse_sfm_bundle_adjustment.h>
#include <algorithm>
#include <cmath>
#include <new>
#include <utility>
#include <vector>
namespace lardon3d::sparse_bundle_adjustment {
constexpr size_t maximum_images = 4096;
constexpr size_t maximum_landmarks = 250000;
constexpr size_t maximum_observations = 1000000;
constexpr double depth_epsilon = 1e-9;
constexpr double gauge_epsilon = 1e-9;
enum class PreparationStatus { prepared, invalid_argument, out_of_memory };
enum class PrivateTermination { converged, no_convergence, failure };
struct ResolvedObservation {
Lardon3DSparseIncrementalObservation source;
size_t image_index;
};
struct Preparation {
std::vector<Lardon3DSparseIncrementalImage> images;
std::vector<Lardon3DSparseIncrementalComponent> components;
std::vector<Lardon3DSparseIncrementalCamera> cameras;
std::vector<Lardon3DSparseIncrementalLandmark> landmarks;
std::vector<Lardon3DSparseIncrementalLandmarkObservation> observations;
std::vector<ResolvedObservation> resolved_observations;
std::vector<Lardon3DSparseBundleAdjustmentComponentDiagnostic> diagnostics;
};
struct ImageIndex {
uint64_t image_id;
size_t index;
};
struct ObservationIndex {
uint64_t feature_set_id;
uint32_t feature_index;
size_t index;
};
bool finite_calibration(const Lardon3DSparseGeometryCalibration &value) {
return value.width > 0 && value.height > 0 && std::isfinite(value.fx) &&
std::isfinite(value.fy) && value.fx > 0.0 && value.fy > 0.0 &&
std::isfinite(value.cx) && std::isfinite(value.cy) && value.cx >= 0.0 &&
value.cy >= 0.0 && value.cx < value.width && value.cy < value.height &&
std::isfinite(value.k1) && std::isfinite(value.k2) &&
std::isfinite(value.p1) && std::isfinite(value.p2);
}
bool finite_pose(const Lardon3DSparseGeometryPose &value) {
for (double item : value.rotation_cw)
if (!std::isfinite(item)) return false;
for (double item : value.translation_cw)
if (!std::isfinite(item)) return false;
return true;
}
bool valid_rotation(const Lardon3DSparseGeometryPose &value) {
const double *r = value.rotation_cw;
for (size_t row = 0; row < 3; ++row) {
for (size_t column = 0; column < 3; ++column) {
double dot = 0.0;
for (size_t item = 0; item < 3; ++item)
dot += r[row * 3 + item] * r[column * 3 + item];
if (!std::isfinite(dot) ||
std::abs(dot - (row == column ? 1.0 : 0.0)) >= 1e-6)
return false;
}
}
const double determinant =
r[0] * (r[4] * r[8] - r[5] * r[7]) -
r[1] * (r[3] * r[8] - r[5] * r[6]) +
r[2] * (r[3] * r[7] - r[4] * r[6]);
return std::isfinite(determinant) && std::abs(determinant - 1.0) < 1e-6;
}
bool finite_point(const Lardon3DSparseGeometryPoint3 &value) {
return std::isfinite(value.x) && std::isfinite(value.y) && std::isfinite(value.z);
}
bool camera_center(const Lardon3DSparseGeometryPose &pose, double center[3]) {
if (!finite_pose(pose) || !valid_rotation(pose)) return false;
for (size_t column = 0; column < 3; ++column) {
center[column] = -(pose.rotation_cw[column] * pose.translation_cw[0] +
pose.rotation_cw[3 + column] * pose.translation_cw[1] +
pose.rotation_cw[6 + column] * pose.translation_cw[2]);
if (!std::isfinite(center[column])) return false;
}
return true;
}
bool select_anchors(const std::vector<Lardon3DSparseIncrementalCamera> &cameras,
uint64_t component_key,
Lardon3DSparseBundleAdjustmentComponentDiagnostic *diagnostic) {
const Lardon3DSparseIncrementalCamera *pose_anchor = nullptr;
for (const auto &camera : cameras) {
if (camera.component_key == component_key &&
(!pose_anchor || camera.image_id < pose_anchor->image_id))
pose_anchor = &camera;
}
if (!pose_anchor) return false;
double anchor_center[3];
if (!camera_center(pose_anchor->pose_cw, anchor_center)) return false;
const Lardon3DSparseIncrementalCamera *scale_anchor = nullptr;
double best_distance = -1.0;
double best_delta[3] = {};
for (const auto &camera : cameras) {
if (camera.component_key != component_key || camera.image_id == pose_anchor->image_id)
continue;
double center[3];
if (!camera_center(camera.pose_cw, center)) return false;
double delta[3] = {center[0] - anchor_center[0], center[1] - anchor_center[1],
center[2] - anchor_center[2]};
const double distance = std::hypot(delta[0], delta[1], delta[2]);
if (!std::isfinite(distance)) return false;
if (!scale_anchor || distance > best_distance ||
(distance == best_distance && camera.image_id < scale_anchor->image_id)) {
scale_anchor = &camera;
best_distance = distance;
std::copy(delta, delta + 3, best_delta);
}
}
if (!scale_anchor) return false;
size_t axis = 0;
if (std::abs(best_delta[1]) > std::abs(best_delta[axis])) axis = 1;
if (std::abs(best_delta[2]) > std::abs(best_delta[axis])) axis = 2;
diagnostic->pose_anchor_image_id = pose_anchor->image_id;
diagnostic->scale_anchor_image_id = scale_anchor->image_id;
diagnostic->scale_axis =
static_cast<Lardon3DSparseBundleAdjustmentScaleAxis>(axis + 1);
diagnostic->has_anchors = std::abs(best_delta[axis]) > gauge_epsilon;
return diagnostic->has_anchors;
}
bool termination_accepted(PrivateTermination value) {
return value == PrivateTermination::converged;
}
bool project(const Lardon3DSparseGeometryCalibration &calibration,
const Lardon3DSparseGeometryPose &pose,
const Lardon3DSparseGeometryPoint3 &point,
Lardon3DSparseGeometryPoint2 *pixel) {
if (!pixel || !finite_calibration(calibration) || !finite_pose(pose) ||
!valid_rotation(pose) || !finite_point(point))
return false;
const double *r = pose.rotation_cw;
const double *t = pose.translation_cw;
const double x = r[0] * point.x + r[1] * point.y + r[2] * point.z + t[0];
const double y = r[3] * point.x + r[4] * point.y + r[5] * point.z + t[1];
const double z = r[6] * point.x + r[7] * point.y + r[8] * point.z + t[2];
if (!std::isfinite(x) || !std::isfinite(y) || !std::isfinite(z) ||
z <= depth_epsilon)
return false;
const double xn = x / z;
const double yn = y / z;
const double r2 = xn * xn + yn * yn;
const double radial = 1.0 + calibration.k1 * r2 + calibration.k2 * r2 * r2;
const double xd = xn * radial + 2.0 * calibration.p1 * xn * yn +
calibration.p2 * (r2 + 2.0 * xn * xn);
const double yd = yn * radial + calibration.p1 * (r2 + 2.0 * yn * yn) +
2.0 * calibration.p2 * xn * yn;
pixel->x = calibration.fx * xd + calibration.cx;
pixel->y = calibration.fy * yd + calibration.cy;
return std::isfinite(pixel->x) && std::isfinite(pixel->y);
}
bool residual_metrics(const double *residuals, size_t count, double *rmse,
double *huber_cost) {
if (!residuals || count == 0 || !rmse || !huber_cost || count > SIZE_MAX / 2)
return false;
double squared_sum = 0.0;
double rho_sum = 0.0;
for (size_t index = 0; index < count; ++index) {
const double dx = residuals[index * 2];
const double dy = residuals[index * 2 + 1];
const double squared = dx * dx + dy * dy;
if (!std::isfinite(dx) || !std::isfinite(dy) || !std::isfinite(squared))
return false;
const double rho = squared <= 4.0 ? squared : 4.0 * std::sqrt(squared) - 4.0;
squared_sum += squared;
rho_sum += rho;
if (!std::isfinite(rho) || !std::isfinite(squared_sum) || !std::isfinite(rho_sum))
return false;
}
*rmse = std::sqrt(squared_sum / static_cast<double>(count));
*huber_cost = 0.5 * rho_sum;
return std::isfinite(*rmse) && std::isfinite(*huber_cost);
}
bool cost_acceptable(double initial_cost, double final_cost) {
if (!std::isfinite(initial_cost) || !std::isfinite(final_cost)) return false;
const double tolerance = 1e-12 * std::max(1.0, std::abs(initial_cost));
return std::isfinite(tolerance) && final_cost <= initial_cost + tolerance;
}
template <typename T>
void copy_view(std::vector<T> *destination, const T *source, size_t count) {
destination->clear();
if (count != 0) destination->assign(source, source + count);
}
PreparationStatus prepare(const Lardon3DSparseBundleAdjustmentInput &input,
Preparation *preparation) {
if (!preparation || !input.incremental_result)
return PreparationStatus::invalid_argument;
const auto &result = *input.incremental_result;
if (result.status < LARDON3D_SPARSE_INCREMENTAL_COMPLETE ||
result.status > LARDON3D_SPARSE_INCREMENTAL_FAILED ||
input.image_count > maximum_images || result.camera_count > maximum_images ||
result.landmark_count > maximum_landmarks ||
result.observation_count > maximum_observations ||
input.observation_count > maximum_observations ||
(input.image_count && !input.images) ||
(input.observation_count && !input.observations) ||
(result.component_count && !result.components) ||
(result.camera_count && !result.cameras) ||
(result.landmark_count && !result.landmarks) ||
(result.observation_count && !result.observations))
return PreparationStatus::invalid_argument;
try {
Preparation candidate;
copy_view(&candidate.images, input.images, input.image_count);
copy_view(&candidate.components, result.components, result.component_count);
copy_view(&candidate.cameras, result.cameras, result.camera_count);
copy_view(&candidate.landmarks, result.landmarks, result.landmark_count);
copy_view(&candidate.observations, result.observations, result.observation_count);
std::sort(candidate.images.begin(), candidate.images.end(),
[](const auto &a, const auto &b) { return a.image_id < b.image_id; });
std::vector<ImageIndex> images;
images.reserve(input.image_count);
for (size_t index = 0; index < input.image_count; ++index) {
const auto &image = candidate.images[index];
if (image.image_id == 0 || !finite_calibration(image.calibration) ||
(index && candidate.images[index - 1].image_id == image.image_id))
return PreparationStatus::invalid_argument;
images.push_back({image.image_id, index});
}
std::vector<ObservationIndex> observation_index;
observation_index.reserve(input.observation_count);
for (size_t index = 0; index < input.observation_count; ++index) {
const auto &observation = input.observations[index];
if (observation.track_id == 0 || observation.image_id == 0 ||
observation.feature_set_id == 0 || observation.feature_count == 0 ||
observation.feature_index >= observation.feature_count ||
!std::isfinite(observation.x) || !std::isfinite(observation.y))
return PreparationStatus::invalid_argument;
auto image = std::lower_bound(
images.begin(), images.end(), observation.image_id,
[](const auto &item, uint64_t key) { return item.image_id < key; });
if (image == images.end() || image->image_id != observation.image_id)
return PreparationStatus::invalid_argument;
observation_index.push_back(
{observation.feature_set_id, observation.feature_index, index});
}
std::sort(observation_index.begin(), observation_index.end(),
[](const auto &a, const auto &b) {
return std::pair{a.feature_set_id, a.feature_index} <
std::pair{b.feature_set_id, b.feature_index};
});
for (size_t index = 1; index < observation_index.size(); ++index) {
if (observation_index[index - 1].feature_set_id ==
observation_index[index].feature_set_id &&
observation_index[index - 1].feature_index ==
observation_index[index].feature_index)
return PreparationStatus::invalid_argument;
}
std::vector<uint64_t> component_camera_counts(result.component_count, 0);
std::vector<uint64_t> component_landmark_counts(result.component_count, 0);
for (size_t index = 0; index < result.component_count; ++index) {
const auto &component = candidate.components[index];
if (component.component_key == 0 ||
(index && candidate.components[index - 1].component_key >=
component.component_key) ||
component.registered_image_count > component.image_count)
return PreparationStatus::invalid_argument;
}
for (size_t index = 0; index < result.camera_count; ++index) {
const auto &camera = candidate.cameras[index];
if (camera.image_id == 0 ||
(index && candidate.cameras[index - 1].image_id >= camera.image_id) ||
!finite_pose(camera.pose_cw) || !valid_rotation(camera.pose_cw))
return PreparationStatus::invalid_argument;
auto component = std::lower_bound(
candidate.components.begin(), candidate.components.end(), camera.component_key,
[](const auto &item, uint64_t key) { return item.component_key < key; });
auto image = std::lower_bound(
images.begin(), images.end(), camera.image_id,
[](const auto &item, uint64_t key) { return item.image_id < key; });
if (component == candidate.components.end() ||
component->component_key != camera.component_key || image == images.end() ||
image->image_id != camera.image_id)
return PreparationStatus::invalid_argument;
++component_camera_counts[static_cast<size_t>(component - candidate.components.begin())];
}
for (size_t index = 0; index < result.landmark_count; ++index) {
const auto &landmark = candidate.landmarks[index];
if (landmark.landmark_id == 0 || landmark.track_id == 0 ||
landmark.observation_count < 2 || !finite_point(landmark.point) ||
(index && candidate.landmarks[index - 1].track_id >= landmark.track_id))
return PreparationStatus::invalid_argument;
auto component = std::lower_bound(
candidate.components.begin(), candidate.components.end(), landmark.component_key,
[](const auto &item, uint64_t key) { return item.component_key < key; });
if (component == candidate.components.end() ||
component->component_key != landmark.component_key)
return PreparationStatus::invalid_argument;
++component_landmark_counts[static_cast<size_t>(component - candidate.components.begin())];
}
for (size_t index = 0; index < result.component_count; ++index) {
if (candidate.components[index].registered_image_count !=
component_camera_counts[index] ||
candidate.components[index].landmark_count != component_landmark_counts[index])
return PreparationStatus::invalid_argument;
}
candidate.diagnostics.resize(result.component_count);
std::vector<uint64_t> landmark_observation_counts(result.landmark_count, 0);
std::vector<uint8_t> published_observations(input.observation_count, 0);
candidate.resolved_observations.reserve(result.observation_count);
uint64_t previous_landmark = 0;
uint32_t previous_position = 0;
for (size_t index = 0; index < result.observation_count; ++index) {
const auto &published = candidate.observations[index];
const bool ordered = index == 0 || published.landmark_id > previous_landmark ||
(published.landmark_id == previous_landmark &&
published.position_in_track > previous_position);
auto landmark = std::lower_bound(
candidate.landmarks.begin(), candidate.landmarks.end(), published.track_id,
[](const auto &item, uint64_t key) { return item.track_id < key; });
auto observation = std::lower_bound(
observation_index.begin(), observation_index.end(),
std::pair{published.feature_set_id, published.feature_index},
[](const auto &item, const auto &key) {
return std::pair{item.feature_set_id, item.feature_index} < key;
});
auto image = std::lower_bound(
images.begin(), images.end(), published.image_id,
[](const auto &item, uint64_t key) { return item.image_id < key; });
if (!ordered || landmark == candidate.landmarks.end() ||
landmark->landmark_id != published.landmark_id ||
landmark->track_id != published.track_id || observation == observation_index.end() ||
observation->feature_set_id != published.feature_set_id ||
observation->feature_index != published.feature_index || image == images.end() ||
image->image_id != published.image_id)
return PreparationStatus::invalid_argument;
const auto &source = input.observations[observation->index];
if (published_observations[observation->index])
return PreparationStatus::invalid_argument;
published_observations[observation->index] = 1;
auto camera = std::lower_bound(
candidate.cameras.begin(), candidate.cameras.end(), published.image_id,
[](const auto &item, uint64_t key) { return item.image_id < key; });
if (source.track_id != published.track_id || source.image_id != published.image_id ||
camera == candidate.cameras.end() || camera->image_id != published.image_id ||
camera->component_key != landmark->component_key)
return PreparationStatus::invalid_argument;
const size_t landmark_index = static_cast<size_t>(landmark - candidate.landmarks.begin());
auto component = std::lower_bound(
candidate.components.begin(), candidate.components.end(), landmark->component_key,
[](const auto &item, uint64_t key) { return item.component_key < key; });
const size_t component_index =
static_cast<size_t>(component - candidate.components.begin());
++landmark_observation_counts[landmark_index];
++candidate.diagnostics[component_index].observation_count;
candidate.resolved_observations.push_back({source, image->index});
previous_landmark = published.landmark_id;
previous_position = published.position_in_track;
}
for (size_t index = 0; index < candidate.landmarks.size(); ++index) {
if (candidate.landmarks[index].observation_count !=
landmark_observation_counts[index])
return PreparationStatus::invalid_argument;
}
for (size_t index = 0; index < candidate.components.size(); ++index) {
auto &diagnostic = candidate.diagnostics[index];
const auto &component = candidate.components[index];
diagnostic.component_key = component.component_key;
diagnostic.camera_count = component.registered_image_count;
diagnostic.landmark_count = component.landmark_count;
diagnostic.termination = LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_TERMINATION_NOT_RUN;
diagnostic.rejection_reason =
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_REJECTION_INELIGIBLE;
diagnostic.eligible = component.registered_image_count >= 2 &&
component.landmark_count >= 1 &&
select_anchors(candidate.cameras, component.component_key,
&diagnostic);
if (diagnostic.eligible)
diagnostic.rejection_reason = LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_REJECTION_NONE;
else if (!diagnostic.has_anchors && component.registered_image_count >= 2)
diagnostic.rejection_reason =
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_REJECTION_GAUGE_DEGENERATE;
}
*preparation = std::move(candidate);
return PreparationStatus::prepared;
} catch (const std::bad_alloc &) {
return PreparationStatus::out_of_memory;
} catch (...) {
return PreparationStatus::invalid_argument;
}
}
} // namespace lardon3d::sparse_bundle_adjustment
#ifdef LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_TESTING
extern "C" int lardon3d_sparse_bundle_adjustment_test_prepare(
const Lardon3DSparseBundleAdjustmentInput *input,
Lardon3DSparseBundleAdjustmentComponentDiagnostic *diagnostics,
size_t diagnostic_capacity, Lardon3DSparseIncrementalObservation *resolved,
size_t resolved_capacity, size_t *diagnostic_count, size_t *resolved_count) {
using namespace lardon3d::sparse_bundle_adjustment;
if (!input || !diagnostic_count || !resolved_count) return 1;
Preparation preparation;
const auto status = prepare(*input, &preparation);
if (status != PreparationStatus::prepared)
return status == PreparationStatus::out_of_memory ? 2 : 1;
*diagnostic_count = preparation.diagnostics.size();
*resolved_count = preparation.resolved_observations.size();
if ((preparation.diagnostics.size() > diagnostic_capacity) ||
(preparation.resolved_observations.size() > resolved_capacity) ||
(!preparation.diagnostics.empty() && !diagnostics) ||
(!preparation.resolved_observations.empty() && !resolved))
return 1;
std::copy(preparation.diagnostics.begin(), preparation.diagnostics.end(), diagnostics);
for (size_t index = 0; index < preparation.resolved_observations.size(); ++index)
resolved[index] = preparation.resolved_observations[index].source;
return 0;
}
extern "C" bool lardon3d_sparse_bundle_adjustment_test_project(
const Lardon3DSparseGeometryCalibration *calibration,
const Lardon3DSparseGeometryPose *pose,
const Lardon3DSparseGeometryPoint3 *point,
Lardon3DSparseGeometryPoint2 *pixel) {
return calibration && pose && point &&
lardon3d::sparse_bundle_adjustment::project(*calibration, *pose, *point, pixel);
}
extern "C" bool lardon3d_sparse_bundle_adjustment_test_metrics(
const double *residuals, size_t count, double *rmse, double *huber_cost) {
return lardon3d::sparse_bundle_adjustment::residual_metrics(
residuals, count, rmse, huber_cost);
}
extern "C" bool lardon3d_sparse_bundle_adjustment_test_cost_acceptable(
double initial_cost, double final_cost) {
return lardon3d::sparse_bundle_adjustment::cost_acceptable(initial_cost, final_cost);
}
extern "C" bool lardon3d_sparse_bundle_adjustment_test_termination_accepted(
int value) {
using namespace lardon3d::sparse_bundle_adjustment;
if (value < 0 || value > 2) return false;
return termination_accepted(static_cast<PrivateTermination>(value));
}
#endif

View file

@ -0,0 +1,356 @@
#include <lardon3d/sparse_sfm_bundle_adjustment.h>
#include <cmath>
#include <cstdio>
#include <cstring>
#include <limits>
#define CHECK(value) \
do { \
if (!(value)) { \
std::fprintf(stderr, "bundle adjustment check failed at line %d: %s\n", \
__LINE__, #value); \
return 1; \
} \
} while (0)
extern "C" bool lardon3d_sparse_bundle_adjustment_test_termination_accepted(
int value);
extern "C" int lardon3d_sparse_bundle_adjustment_test_prepare(
const Lardon3DSparseBundleAdjustmentInput *input,
Lardon3DSparseBundleAdjustmentComponentDiagnostic *diagnostics,
size_t diagnostic_capacity, Lardon3DSparseIncrementalObservation *resolved,
size_t resolved_capacity, size_t *diagnostic_count, size_t *resolved_count);
extern "C" bool lardon3d_sparse_bundle_adjustment_test_project(
const Lardon3DSparseGeometryCalibration *calibration,
const Lardon3DSparseGeometryPose *pose,
const Lardon3DSparseGeometryPoint3 *point,
Lardon3DSparseGeometryPoint2 *pixel);
extern "C" bool lardon3d_sparse_bundle_adjustment_test_metrics(
const double *residuals, size_t count, double *rmse, double *huber_cost);
extern "C" bool lardon3d_sparse_bundle_adjustment_test_cost_acceptable(
double initial_cost, double final_cost);
static Lardon3DSparseGeometryCalibration calibration() {
return {1280, 960, 800.0, 810.0, 640.0, 480.0, 0.0, 0.0, 0.0, 0.0};
}
static int test_helpers() {
const Lardon3DSparseGeometryPose pose = {
{1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0}, {0.0, 0.0, 0.0}};
const Lardon3DSparseGeometryPoint3 point = {0.2, -0.1, 2.0};
auto camera = calibration();
Lardon3DSparseGeometryPoint2 pixel = {};
CHECK(lardon3d_sparse_bundle_adjustment_test_project(&camera, &pose, &point, &pixel));
CHECK(std::abs(pixel.x - 720.0) < 1e-12);
CHECK(std::abs(pixel.y - 439.5) < 1e-12);
camera.k1 = 0.1;
camera.k2 = -0.02;
camera.p1 = 0.003;
camera.p2 = -0.004;
CHECK(lardon3d_sparse_bundle_adjustment_test_project(&camera, &pose, &point, &pixel));
const double xn = 0.1;
const double yn = -0.05;
const double r2 = xn * xn + yn * yn;
const double radial = 1.0 + camera.k1 * r2 + camera.k2 * r2 * r2;
const double xd = xn * radial + 2.0 * camera.p1 * xn * yn +
camera.p2 * (r2 + 2.0 * xn * xn);
const double yd = yn * radial + camera.p1 * (r2 + 2.0 * yn * yn) +
2.0 * camera.p2 * xn * yn;
CHECK(std::abs(pixel.x - (camera.fx * xd + camera.cx)) < 1e-12);
CHECK(std::abs(pixel.y - (camera.fy * yd + camera.cy)) < 1e-12);
double residuals[4] = {3.0, 4.0, 0.0, 0.0};
double rmse = 0.0;
double cost = 0.0;
CHECK(lardon3d_sparse_bundle_adjustment_test_metrics(residuals, 2, &rmse, &cost));
CHECK(std::abs(rmse - std::sqrt(12.5)) < 1e-12);
CHECK(cost == 8.0);
double boundary[2] = {2.0, 0.0};
CHECK(lardon3d_sparse_bundle_adjustment_test_metrics(boundary, 1, &rmse, &cost));
CHECK(rmse == 2.0 && cost == 2.0);
CHECK(lardon3d_sparse_bundle_adjustment_test_cost_acceptable(10.0, 10.0 + 1e-11));
CHECK(!lardon3d_sparse_bundle_adjustment_test_cost_acceptable(10.0,
10.0 + 2e-11));
CHECK(!lardon3d_sparse_bundle_adjustment_test_cost_acceptable(NAN, 0.0));
residuals[0] = std::numeric_limits<double>::infinity();
CHECK(!lardon3d_sparse_bundle_adjustment_test_metrics(residuals, 2, &rmse, &cost));
CHECK(!lardon3d_sparse_bundle_adjustment_test_project(nullptr, &pose, &point, &pixel));
Lardon3DSparseGeometryPoint3 behind = {0.0, 0.0, -1.0};
CHECK(!lardon3d_sparse_bundle_adjustment_test_project(&camera, &pose, &behind,
&pixel));
Lardon3DSparseGeometryPoint3 depth_boundary = {0.0, 0.0, 1e-9};
CHECK(!lardon3d_sparse_bundle_adjustment_test_project(
&camera, &pose, &depth_boundary, &pixel));
depth_boundary.z = std::nextafter(1e-9, 1.0);
CHECK(lardon3d_sparse_bundle_adjustment_test_project(
&camera, &pose, &depth_boundary, &pixel));
auto invalid_pose = pose;
invalid_pose.rotation_cw[0] += 1e-6;
CHECK(!lardon3d_sparse_bundle_adjustment_test_project(
&camera, &invalid_pose, &point, &pixel));
auto invalid_calibration = camera;
invalid_calibration.cx = -1.0;
CHECK(!lardon3d_sparse_bundle_adjustment_test_project(
&invalid_calibration, &pose, &point, &pixel));
CHECK(lardon3d_sparse_bundle_adjustment_test_termination_accepted(0));
CHECK(!lardon3d_sparse_bundle_adjustment_test_termination_accepted(1));
CHECK(!lardon3d_sparse_bundle_adjustment_test_termination_accepted(2));
CHECK(!lardon3d_sparse_bundle_adjustment_test_termination_accepted(3));
return 0;
}
struct Fixture {
Lardon3DSparseIncrementalImage images[3];
Lardon3DSparseIncrementalObservation input_observations[3];
Lardon3DSparseIncrementalComponent components[1];
Lardon3DSparseIncrementalCamera cameras[3];
Lardon3DSparseIncrementalLandmark landmarks[1];
Lardon3DSparseIncrementalLandmarkObservation result_observations[3];
Lardon3DSparseIncrementalUnregisteredImage unregistered_images[1];
Lardon3DSparseIncrementalResult result;
Lardon3DSparseBundleAdjustmentInput input;
};
static Fixture fixture() {
Fixture value = {};
const auto intrinsic = calibration();
value.images[0] = {10, intrinsic};
value.images[1] = {20, intrinsic};
value.images[2] = {30, intrinsic};
value.input_observations[0] = {7, 10, 100, 1, 2, 650.0, 480.0};
value.input_observations[1] = {7, 20, 200, 1, 2, 500.0, 480.0};
value.input_observations[2] = {7, 30, 300, 1, 2, 700.0, 480.0};
value.components[0] = {10, 3, 3, 1};
const double identity[9] = {1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0};
std::memcpy(value.cameras[0].pose_cw.rotation_cw, identity, sizeof(identity));
std::memcpy(value.cameras[1].pose_cw.rotation_cw, identity, sizeof(identity));
std::memcpy(value.cameras[2].pose_cw.rotation_cw, identity, sizeof(identity));
value.cameras[0].image_id = 10;
value.cameras[1].image_id = 20;
value.cameras[2].image_id = 30;
for (auto &camera : value.cameras) camera.component_key = 10;
value.cameras[1].pose_cw.translation_cw[0] = -2.0;
value.cameras[1].pose_cw.translation_cw[1] = -2.0;
value.cameras[2].pose_cw.translation_cw[0] = 2.0;
value.cameras[2].pose_cw.translation_cw[1] = 2.0;
value.landmarks[0] = {1, 7, 10, {0.0, 0.0, 5.0}, 0.5, 0.4, 3};
for (size_t index = 0; index < 3; ++index) {
value.result_observations[index] = {
1, 7, value.images[index].image_id,
value.input_observations[index].feature_set_id,
value.input_observations[index].feature_index,
static_cast<uint32_t>(index)};
}
value.result.status = LARDON3D_SPARSE_INCREMENTAL_COMPLETE;
value.result.track_set_id = 77;
value.result.calibration_scope_id = 88;
value.result.component_count = 1;
value.result.camera_count = 3;
value.result.landmark_count = 1;
value.result.observation_count = 3;
value.unregistered_images[0] = {99, 10};
value.result.unregistered_image_count = 1;
value.result.seed_candidates_considered = 3;
value.result.seed_candidates_available = 4;
value.result.seed_image_a = 10;
value.result.seed_image_b = 20;
value.result.last_seed_geometry_status = 2;
value.result.last_seed_parallax_rad = 0.25;
value.result.registration_rounds = 5;
value.result.registration_attempts = 6;
value.result.registration_successes = 3;
value.result.registration_failures = 3;
value.result.last_pnp_inlier_count = 8;
value.result.triangulation_attempts = 9;
value.result.triangulation_failures = 1;
value.result.rejected_behind_camera = 2;
value.result.rejected_reprojection = 3;
value.result.rejected_landmarks = 4;
value.result.last_triangulation_status = 5;
value.result.landmark_update_attempts = 6;
value.result.landmark_update_successes = 7;
value.result.landmark_update_failures = 8;
value.result.no_growth_terminations = 9;
value.result.round_limit_terminations = 10;
value.result.point_refinement_attempts = 11;
value.result.point_refinement_successes = 12;
return value;
}
static void bind_fixture(Fixture *value) {
value->result.components = value->components;
value->result.cameras = value->cameras;
value->result.landmarks = value->landmarks;
value->result.observations = value->result_observations;
value->result.unregistered_images = value->unregistered_images;
value->input = {&value->result, value->images, 3, value->input_observations, 3};
}
static int test_preparation() {
Fixture value = fixture();
bind_fixture(&value);
const Fixture original = value;
Lardon3DSparseBundleAdjustmentComponentDiagnostic diagnostics[1] = {};
Lardon3DSparseIncrementalObservation resolved[3] = {};
size_t diagnostic_count = 0;
size_t resolved_count = 0;
CHECK(lardon3d_sparse_bundle_adjustment_test_prepare(
&value.input, diagnostics, 1, resolved, 3, &diagnostic_count,
&resolved_count) == 0);
CHECK(diagnostic_count == 1 && resolved_count == 3);
CHECK(resolved[1].image_id == 20 && resolved[1].x == 500.0);
CHECK(diagnostics[0].eligible);
CHECK(diagnostics[0].pose_anchor_image_id == 10);
CHECK(diagnostics[0].scale_anchor_image_id == 20);
CHECK(diagnostics[0].scale_axis ==
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_AXIS_X);
CHECK(diagnostics[0].termination ==
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_TERMINATION_NOT_RUN);
CHECK(!diagnostics[0].has_costs && !diagnostics[0].has_rmse);
CHECK(!diagnostics[0].accepted);
CHECK(std::memcmp(&value.result, &original.result, sizeof(value.result)) == 0);
CHECK(std::memcmp(value.images, original.images, sizeof(value.images)) == 0);
CHECK(std::memcmp(value.input_observations, original.input_observations,
sizeof(value.input_observations)) == 0);
CHECK(std::memcmp(value.cameras, original.cameras, sizeof(value.cameras)) == 0);
CHECK(std::memcmp(value.landmarks, original.landmarks, sizeof(value.landmarks)) == 0);
CHECK(std::memcmp(value.components, original.components, sizeof(value.components)) == 0);
CHECK(std::memcmp(value.result_observations, original.result_observations,
sizeof(value.result_observations)) == 0);
CHECK(std::memcmp(value.unregistered_images, original.unregistered_images,
sizeof(value.unregistered_images)) == 0);
value = fixture();
bind_fixture(&value);
value.cameras[1].pose_cw.translation_cw[0] = -1e-9;
value.cameras[1].pose_cw.translation_cw[1] = 0.0;
value.cameras[2].pose_cw.translation_cw[0] = 1e-9;
value.cameras[2].pose_cw.translation_cw[1] = 0.0;
CHECK(lardon3d_sparse_bundle_adjustment_test_prepare(
&value.input, diagnostics, 1, resolved, 3, &diagnostic_count,
&resolved_count) == 0);
CHECK(!diagnostics[0].eligible);
CHECK(diagnostics[0].rejection_reason ==
LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_REJECTION_GAUGE_DEGENERATE);
return 0;
}
static int test_invalid() {
Lardon3DSparseBundleAdjustmentComponentDiagnostic diagnostics[1] = {};
Lardon3DSparseIncrementalObservation resolved[3] = {};
size_t diagnostic_count = 0;
size_t resolved_count = 0;
CHECK(lardon3d_sparse_bundle_adjustment_test_prepare(
nullptr, diagnostics, 1, resolved, 3, &diagnostic_count,
&resolved_count) == 1);
Fixture value = fixture();
bind_fixture(&value);
value.input_observations[1].feature_set_id = 999;
CHECK(lardon3d_sparse_bundle_adjustment_test_prepare(
&value.input, diagnostics, 1, resolved, 3, &diagnostic_count,
&resolved_count) == 1);
value = fixture();
bind_fixture(&value);
value.input_observations[1].image_id = 30;
CHECK(lardon3d_sparse_bundle_adjustment_test_prepare(
&value.input, diagnostics, 1, resolved, 3, &diagnostic_count,
&resolved_count) == 1);
value = fixture();
bind_fixture(&value);
value.input_observations[1].x = NAN;
CHECK(lardon3d_sparse_bundle_adjustment_test_prepare(
&value.input, diagnostics, 1, resolved, 3, &diagnostic_count,
&resolved_count) == 1);
value = fixture();
bind_fixture(&value);
value.input_observations[1].feature_set_id = value.input_observations[0].feature_set_id;
value.input_observations[1].feature_index = value.input_observations[0].feature_index;
CHECK(lardon3d_sparse_bundle_adjustment_test_prepare(
&value.input, diagnostics, 1, resolved, 3, &diagnostic_count,
&resolved_count) == 1);
value = fixture();
bind_fixture(&value);
std::swap(value.cameras[0], value.cameras[1]);
CHECK(lardon3d_sparse_bundle_adjustment_test_prepare(
&value.input, diagnostics, 1, resolved, 3, &diagnostic_count,
&resolved_count) == 1);
value = fixture();
bind_fixture(&value);
value.input.image_count = 4097;
CHECK(lardon3d_sparse_bundle_adjustment_test_prepare(
&value.input, diagnostics, 1, resolved, 3, &diagnostic_count,
&resolved_count) == 1);
Lardon3DSparseIncrementalResult empty = {};
empty.status = LARDON3D_SPARSE_INCREMENTAL_FAILED;
Lardon3DSparseBundleAdjustmentInput empty_input = {&empty, nullptr, 0, nullptr, 0};
CHECK(lardon3d_sparse_bundle_adjustment_test_prepare(
&empty_input, nullptr, 0, nullptr, 0, &diagnostic_count,
&resolved_count) == 0);
CHECK(diagnostic_count == 0 && resolved_count == 0);
return 0;
}
static int test_multiple_components() {
const auto intrinsic = calibration();
Lardon3DSparseIncrementalImage images[4] = {
{10, intrinsic}, {20, intrinsic}, {30, intrinsic}, {40, intrinsic}};
Lardon3DSparseIncrementalObservation source_observations[4] = {
{7, 10, 100, 1, 2, 640.0, 480.0},
{7, 20, 200, 1, 2, 600.0, 480.0},
{8, 30, 300, 1, 2, 640.0, 480.0},
{8, 40, 400, 1, 2, 600.0, 480.0}};
Lardon3DSparseIncrementalComponent components[2] = {
{10, 2, 2, 1}, {30, 2, 2, 1}};
Lardon3DSparseIncrementalCamera cameras[4] = {};
const double identity[9] = {1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0};
for (size_t index = 0; index < 4; ++index) {
cameras[index].image_id = images[index].image_id;
cameras[index].component_key = index < 2 ? 10 : 30;
std::memcpy(cameras[index].pose_cw.rotation_cw, identity, sizeof(identity));
}
cameras[1].pose_cw.translation_cw[0] = -1.0;
cameras[3].pose_cw.translation_cw[1] = -1.0;
Lardon3DSparseIncrementalLandmark landmarks[2] = {
{1, 7, 10, {0.0, 0.0, 5.0}, 0.5, 0.4, 2},
{2, 8, 30, {0.0, 0.0, 6.0}, 0.5, 0.4, 2}};
Lardon3DSparseIncrementalLandmarkObservation observations[4] = {
{1, 7, 10, 100, 1, 0}, {1, 7, 20, 200, 1, 1},
{2, 8, 30, 300, 1, 0}, {2, 8, 40, 400, 1, 1}};
Lardon3DSparseIncrementalResult result = {};
result.status = LARDON3D_SPARSE_INCREMENTAL_COMPLETE;
result.components = components;
result.component_count = 2;
result.cameras = cameras;
result.camera_count = 4;
result.landmarks = landmarks;
result.landmark_count = 2;
result.observations = observations;
result.observation_count = 4;
Lardon3DSparseBundleAdjustmentInput input = {
&result, images, 4, source_observations, 4};
Lardon3DSparseBundleAdjustmentComponentDiagnostic diagnostics[2] = {};
Lardon3DSparseIncrementalObservation resolved[4] = {};
size_t diagnostic_count = 0;
size_t resolved_count = 0;
CHECK(lardon3d_sparse_bundle_adjustment_test_prepare(
&input, diagnostics, 2, resolved, 4, &diagnostic_count,
&resolved_count) == 0);
CHECK(diagnostic_count == 2 && resolved_count == 4);
CHECK(diagnostics[0].component_key == 10 && diagnostics[0].observation_count == 2);
CHECK(diagnostics[1].component_key == 30 && diagnostics[1].observation_count == 2);
CHECK(diagnostics[0].scale_axis == LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_AXIS_X);
CHECK(diagnostics[1].scale_axis == LARDON3D_SPARSE_BUNDLE_ADJUSTMENT_AXIS_Y);
return 0;
}
int main() {
if (test_helpers() != 0) return 1;
if (test_preparation() != 0) return 1;
if (test_invalid() != 0) return 1;
return test_multiple_components();
}