diff --git a/docs/development/build.md b/docs/development/build.md index 7cfe867..d3e075f 100644 --- a/docs/development/build.md +++ b/docs/development/build.md @@ -1,193 +1,293 @@ -# Instructions de build +# Build -## Prérequis +## Status -- **OS** : Linux (testé sur distributions récentes) -- **Compilateur** : Clang (recommandé) ou GCC -- **Système de build** : Meson + Ninja -- **Dépendances principales** : ncursesw, SQLite, OpenSSL, GIO/GLib, OpenCV, - LibRaw, libexif, libpng, libdeflate, Ceres ; Vulkan reste optionnel -- **Langages** : API publiques C17 et implémentation mixte C17/C++17 +```text +BUILD_SYSTEM=MESON_NINJA +PUBLIC_API_LANGUAGE=C17 +IMPLEMENTATION_LANGUAGES=C17_CXX17 +BUILD_PARALLELISM=HOST_AWARE +RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT +SERIALISM_REQUIRES_PROOF=CANONICAL +``` -## Bootstrap des outils +Meson is the build-system authority. Do not duplicate dependency-version truth +in this document when `meson.build` already enforces it. -Les commandes ci-dessous installent seulement le compilateur, Meson/Ninja, -`pkg-config` et ncurses. Les bibliothèques listées plus haut doivent aussi être -disponibles dans les versions acceptées par `meson.build`; Meson reste la source -de vérité et refuse explicitement une dépendance absente ou incompatible. +## Requirements + +Lardon3D targets Linux. + +Primary toolchain: + +```text +Clang or GCC +Meson +Ninja +pkg-config +``` + +Major dependencies currently include ncursesw, SQLite, OpenSSL, GIO/GLib, +OpenCV, LibRaw, libexif, libpng, libdeflate and Ceres. Vulkan remains optional +at configuration level. + +Public APIs are C17. Implementation is mixed C17/C++17. + +## Bootstrap examples + +These commands install only the basic compiler/build front end; Meson remains +authoritative for the complete dependency set. + +Debian/Ubuntu: ```sh -# Debian / Ubuntu sudo apt install clang meson ninja-build libncursesw5-dev pkg-config +``` -# Fedora +Fedora: + +```sh sudo dnf install clang meson ninja-build ncurses-devel pkg-config +``` -# Arch +Arch Linux: + +```sh sudo pacman -S clang meson ninja ncurses pkgconf ``` -## Build standard +## Standard build + +First configuration: ```sh -# Première configuration -CC=clang meson setup build +CC=clang CXX=clang++ meson setup build +``` -# Arbre existant +Existing tree: + +```sh meson setup --reconfigure build -meson compile -C build -j8 +meson compile -C build ``` -### Options utiles +Do not hard-code `-j8` as project policy. + +Meson/Ninja should use host-appropriate parallelism unless a specific +validation has a reason to constrain it. + +## Build parallelism policy + +The build is not governed by a portable fixed job count. + +Canonical policy: + +```text +preserve the interactive host reserve +then use maximum safe useful throughput +``` + +A reference host measurement such as 8 or 12 useful jobs is evidence for that +host at that time, not a repository constant. + +If memory-heavy compilation or another active workload creates pressure, +reduce build width for that run. Do not convert the temporary reduction into a +global documentation rule. + +## Reconfigure versus wipe + +Prefer incremental reuse: ```sh -# Build de debug (défaut) -meson setup build --wipe - -# Build de release -meson setup build --wipe --buildtype=release - -# Build avec optimisations aggressive -meson setup build --wipe --buildtype=release -Db_lto=true +meson setup --reconfigure build +meson compile -C build ``` +Use `--wipe` only when a fresh configuration is actually required, such as: + +- switching sanitizer configuration in the same directory; +- changing compiler family; +- changing a configuration whose cached state cannot be reused safely; +- reproducing a clean release/global-maintenance proof; +- recovering from a stale or corrupt build directory. + +A normal edit/test loop should not wipe the build tree repeatedly. + +## Release build + +Use an explicit release directory or deliberate reconfiguration. + +Example: + +```sh +CC=clang CXX=clang++ meson setup build-release --buildtype=release +meson compile -C build-release +``` + +For LTO: + +```sh +CC=clang CXX=clang++ meson setup build-release-lto --buildtype=release -Db_lto=true +meson compile -C build-release-lto +``` + +Separate directories avoid destroying a useful incremental debug tree. + +## Vulkan configuration + +Portable CPU-only proof: + +```sh +CC=clang CXX=clang++ meson setup build-portable -Dvulkan_orb=disabled +meson compile -C build-portable +``` + +Vulkan-enabled proof: + +```sh +CC=clang CXX=clang++ meson setup build-vulkan -Dvulkan_orb=enabled +meson compile -C build-vulkan +``` + +A Vulkan-on build is not automatically a proof that every scientific path uses +or should use the GPU. + +Current production GPU promotion remains limited by each subsystem's validated +backend contract. + ## Validation +Normal configured tests: + ```sh -# Tests unitaires meson test -C build --print-errorlogs - -# Vérification du style (whitespace) -git diff --check - -# Vérification autonome d'un header C public modifié -cc -x c -std=c17 -fsyntax-only -Iinclude \ - -include lardon3d/
.h /dev/null ``` -### Preuve fraîche de maintenance globale — 1er septembre 2026 - -Le [registre canonique](../architecture/global_maintenance_audit.md) conserve -le détail et les qualifications. Les résultats reproductibles acquis sont : - -| Configuration fraîche | Compilateurs/options | Build | Suite | -| --- | --- | ---: | ---: | -| portable | Clang/Clang++ 22.1.8, C17/C++17, `-Dvulkan_orb=disabled` | 931/931 | 64/64 sériel | -| Vulkan | Clang/Clang++ 22.1.8, C17/C++17, `-Dvulkan_orb=enabled` | 939/939 | 65/65 sériel | -| ASan/UBSan portable | Clang/Clang++ 22.1.8, `address,undefined` | graphe complet | 64/64 avec LSan désactivé après attribution externe | -| TSan portable | GCC/G++ 16.2.1, Vulkan désactivé | cibles concurrentes | 14/14 + 220 répétitions | - -La suite Vulkan comprend `orb-vulkan-backend` sur la Radeon 780M RADV PHOENIX -réelle. La cible de feasibility SIFT/RootSIFT, non enregistrée dans la suite, -a été compilée/exécutée séparément : zéro divergence de décision Lowe mais des -divergences d'index et de bits de distance, donc aucune promotion en backend -production. Les probes stricts GCC/Clang C17+C++17 passent 76/76 sur les -19 headers publics modifiés ou nouveaux, ainsi que le fixture ABI, le lien -application et `git diff --check`; `scan3d/` reste intact. - -L'unique revue finale indépendante GPT-5.6 SOL/ULTRA a conclu PASS sans finding -bloquant. Elle a indépendamment rejoué le build portable, la suite complète -64/64, une matrice focalisée 15/15, les 76/76 probes de headers, l'ABI, les -négatifs de seams production, le SHA-256 du manifest GV retenu et le diff-check. -Le statut canonique est donc `GLOBAL_MAINTENANCE_AUDIT=PASS/FROZEN` ; les -qualifications sanitizer ci-dessous restent néanmoins partie de la preuve. - -Une validation post-freeze a ensuite attribué le délai intermittent de -`test-feature-task` à la capture de la télémétrie hôte réelle par ses Governors -synthétiques. Le fixture utilise maintenant un `ResourceSnapshot` complet, -privé, par Governor et compilé pour cette seule cible ; production continue de -lire la télémétrie réelle et n'exporte aucun seam. Après correction du second -Governor relevé en revue, Feature passe 100/100, la matrice ordonnée 4/4 et les -suites finales portable/Vulkan 64/64 et 65/65 ; ASan/UBSan ciblé avec -`detect_leaks=0` et TSan passent. Le registre canonique conserve la régression -charge 5 `WAIT`/charge 0 `START` et la qualification exacte. Un timeout `task` -isolé dans une suite normale mixte après reconstruction large reste -non reproductible : le ciblé immédiat et sa matrice de revue 100/100 passent, -sans modification de Task ni de son timeout. - -La première suite LSan complète est volontairement conservée comme non-PASS : -57 OK, 6 FAIL et 1 timeout. Cinq échecs partagent exactement la fuite externe -OpenCL de 3 808 octets/68 allocations ; les deux anomalies de 30 s n'ont aucun -diagnostic sanitizer. Le délai Feature, absent du suivi initial, a ensuite été -reproduit et corrigé comme décrit ci-dessus ; le délai Task reste non -reproductible. La suite entière passe 64/64 avec ASan/UBSan actifs et -`detect_leaks=0`, tandis qu'un sous-ensemble prouvé sans loader OpenCV/OpenCL -passe 20/20 avec LSan actif. Il est donc incorrect de résumer cette preuve par -« LSan 64/64 ». - -Le log Clang complet a aussi été audité. La conversion publique Sparse SfM -`uint32_t → int` était matérielle et a été corrigée avec validation ciblée ; les -autres émissions sont soit des conversions baseline déjà bornées, soit des -tests/benchmarks, soit des headers OpenCV/Ceres externes. Les emplacements et -justifications exacts restent centralisés dans le registre afin de ne pas -dupliquer une seconde liste normative ici. - -## Build ASan/UBSan (debug mémoire) - -À exécuter pour tout ticket touchant la mémoire, les durées de vie ou les -allocations : +Whitespace/style boundary: ```sh -CC=clang meson setup build-asan --wipe \ - -Db_sanitize=address,undefined -meson compile -C build-asan -j8 +git diff --check +``` + +Public C header probe: + +```sh +cc -x c -std=c17 -fsyntax-only -Iinclude -include lardon3d/
.h /dev/null +``` + +Use `docs/development/testing.md` for sanitizer and validation policy. + +## ASan / UBSan build + +Example dedicated directory: + +```sh +CC=clang CXX=clang++ meson setup build-asan -Db_sanitize=address,undefined +meson compile -C build-asan meson test -C build-asan --print-errorlogs ``` -## Build TSan (concurrence) +Do not claim an unqualified full LeakSanitizer pass from the retained global +maintenance checkpoint. The external OpenCL loader qualification documented in +the canonical audit remains part of that evidence. -À exécuter pour tout ticket touchant la concurrence (pthread, mutex, -variables de condition, états partagés) : +## TSan build + +Use TSan only with the configuration that matches the intended proof. + +The retained global maintenance concurrency proof used GCC/G++ with Vulkan +disabled, because the project TSan matrix and the Vulkan runtime validation are +separate evidence boundaries. + +Example: ```sh -CC=clang meson setup build-tsan --wipe \ - -Db_sanitize=thread \ - -Db_lundef=false -meson compile -C build-tsan -j8 +CC=gcc CXX=g++ meson setup build-tsan -Db_sanitize=thread -Db_lundef=false -Dvulkan_orb=disabled +meson compile -C build-tsan meson test -C build-tsan --print-errorlogs ``` -## Variables d'environnement +The exact target subset, suppression qualification and repetition evidence are +documented in `docs/development/concurrency.md` and the global maintenance +audit. -| Variable | Description | -|---|---| -| `CC` | Compilateur C (défaut : gcc) | -| `CFLAGS` | Drapeaux de compilation supplémentaires | -| `LDFLAGS` | Drapeaux de liaison supplémentaires | +## Current retained maintenance checkpoint -## Structure du build +The canonical detailed evidence is: + +```text +docs/architecture/global_maintenance_audit.md +GLOBAL_MAINTENANCE_AUDIT=PASS/FROZEN +``` + +The 2026-09-01 checkpoint retained: + +```text +portable Clang/Clang++ build and suite +Vulkan-on Clang/Clang++ build and suite +ASan/UBSan qualified run +portable GCC/G++ TSan matrix +public-header C17/C++17 probes +ABI and application-link checks +independent review +``` + +Those exact historical counts belong to the audit and should not be duplicated +as a new current build contract. + +## Build directory layout + +A configured Meson tree typically contains: ```text build/ -├── src/ # objets et binaires -├── tests/ # binaires de tests -└── compile_commands.json # pour LSP / clangd + src/ + tests/ + compile_commands.json ``` -## Dépannage +Exact generated layout is Meson/Ninja output and may evolve. -### Erreur : ncursesw introuvable +## Environment + +Common variables include: + +| Variable | Purpose | +| --- | --- | +| `CC` | C compiler | +| `CXX` | C++ compiler | +| `CFLAGS` | additional C flags | +| `CXXFLAGS` | additional C++ flags | +| `LDFLAGS` | additional linker flags | + +Prefer Meson options for project features rather than ad-hoc environment flags +that make builds difficult to reproduce. + +## Troubleshooting + +Check ncursesw discovery: ```sh -# Vérifier l'installation pkg-config --libs ncursesw -# Si absent, installer le paquet de développement ncursesw ``` -### Erreur : clang introuvable +If Clang is unavailable, GCC is supported where the current Meson checks allow +it. -```sh -# Utiliser gcc en alternative -meson setup build --wipe -# ou installer clang -sudo apt install clang +For a slow build, first preserve the existing build tree and let Ninja use +normal host-aware scheduling. Reduce concurrency only when actual host pressure +or another active workload justifies it. + +`ccache` may be used when available, but it is optional operational tooling and +not part of scientific identity. + +## Rules + +```text +NO_FIXED_GLOBAL_J8=YES +NO_REPEATED_UNCHANGED_WIPE=YES +HOST_AWARE_BUILD_PARALLELISM=YES ``` -### Build lent - -```sh -# Réduire la parallélisation -meson compile -C build -j4 -# ou utiliser ccache -CC="ccache clang" meson setup build --wipe -``` +Build configuration is operational state. It must not silently redefine +scientific formats, fingerprints or persistence contracts. diff --git a/docs/development/concurrency.md b/docs/development/concurrency.md index 98d808b..e882ef3 100644 --- a/docs/development/concurrency.md +++ b/docs/development/concurrency.md @@ -1,255 +1,412 @@ -# Règles de concurrence +# Concurrency -## Vue d'ensemble - -Lardon3D utilise un modèle de concurrence à thread unique pour ncurses -et un modèle multi-thread pour le traitement. La séparation est stricte : -le thread ncurses ne fait jamais de travail métier, et les workers ne -touchent jamais ncurses. - -## Modèle de concurrence +## Status ```text -Thread principal (ncurses) -├── Gestion des entrées -├── Affichage TUI -└── Orchestration +NCURSES_OWNER=MAIN_THREAD_ONLY +ACTIVE_HEAVY_QUEUE_CALLBACKS=1 +TASK_CANCELLATION=COOPERATIVE +INTERNAL_PARALLELISM=BOUNDED +OWNER_ONLY_PUBLICATION=CANONICAL_WHERE_REQUIRED -Worker thread -├── Exécution des tâches -├── Calculs métier -└── Écritures de résultats +TSAN_PORTABLE_PROJECT_MATRIX=QUALIFIED_PASS +TSAN_EXTERNAL_OPENCV_TBB=QUALIFIED +VULKAN_CONCURRENCY_VALIDATION=SEPARATE -SSD operation thread (0 ou 1, joinable) -└── Un poll ou contrôle UDisks synchrone borné, sans ncurses ni Task +RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT +SERIALISM_REQUIRES_PROOF=CANONICAL ``` -## Règles fondamentales +Lardon3D separates UI ownership from heavy processing. -### 1. ncurses appartient au thread principal +The main thread owns ncurses. The Task Queue owns one active heavy callback. +Individual validated Task Kinds may create bounded internal participants inside +that callback. -```c -// ✅ Correct : appel depuis le thread principal -mvprintw(0, 0, "Progression: %d%%", progress); +An SSD controller operation may also use at most one bounded joinable operation +thread under its own ownership contract. -// ❌ Interdit : appel depuis un worker -// mvprintw() dans un thread secondaire +## Execution model + +```text +main thread + input + ncurses + TUI orchestration + +Task Queue worker + one active heavy callback + admitted Task sequence + optional bounded internal participants + deterministic owner publication + +SSD operation thread + zero or one bounded joinable controller operation + no ncurses + no Task callback ``` -### 2. Variables partagées protégées par mutex +Internal participants are not a second global scheduler or Queue. -```c -// ✅ Correct -pthread_mutex_lock(&queue->mutex); -queue->count++; -pthread_mutex_unlock(&queue->mutex); +## Fundamental rules -// ❌ Interdit -// queue->count++; sans protection +### ncurses ownership + +Only the main thread calls ncurses. + +Workers publish observable state through protected data. They never call +`mvprintw`, `wrefresh`, or other ncurses APIs. + +### Shared mutable state + +Shared mutable state must have an explicit synchronization owner: + +```text +mutex +condition variable +atomic primitive where the contract explicitly permits it +single-thread ownership +immutable-after-publication ``` -### 3. Variables de condition pour la synchronisation +Do not rely on timing or "normally only one caller". -```c -// Producteur (caller de la Task Queue) -pthread_mutex_lock(&queue->mutex); -queue->ready = true; -pthread_cond_signal(&queue->cond); -pthread_mutex_unlock(&queue->mutex); +### Condition variables -// Consommateur (worker) -pthread_mutex_lock(&queue->mutex); -while (!queue->ready) { - pthread_cond_wait(&queue->cond, &queue->mutex); -} -// traitement -pthread_mutex_unlock(&queue->mutex); +Always test the predicate in a loop around `pthread_cond_wait()`. + +A signal is not durable state; the protected predicate is. + +### Cooperative cancellation + +Production Task cancellation is cooperative. + +Do not use `pthread_cancel()` to stop a Task. + +Task-specific non-preemptible operations finish their current atomic boundary +before pause/cancel is observed. + +### Reservation before callback + +No Task callback runs without the Resource Governor admission/reservation +required by its installed sequence contract. + +Fixed-resource Tasks do not bypass the Governor. + +### Terminal lifetime + +Terminal callback completion precedes destruction of Task userdata. + +Queue/Task ownership must ensure no observer dereferences freed userdata. + +## Lock ordering + +When multiple locks are required, the owning subsystem must define and preserve +one order. + +Never add a reverse-order path to solve a local problem. + +Avoid holding one subsystem mutex while calling into another subsystem that may +call back. + +Where practical: + +```text +copy bounded state under lock +release lock +perform I/O / expensive work +reacquire only for publication ``` -### 4. Pas de callback ncurses depuis un worker +## Queue ingress lifetime -```c -// ✅ Correct : le worker signale au thread principal -void worker_callback(task_t *task, void *userdata) { - shared_state_t *state = userdata; - pthread_mutex_lock(&state->mutex); - state->result_ready = true; - pthread_cond_signal(&state->cond); - pthread_mutex_unlock(&state->mutex); -} +The Queue owner closes ingress before destruction. -// ❌ Interdit : appel ncurses depuis le worker -// void worker_callback(...) { -// mvprintw(...); -// } +Shutdown waits for: + +- active worker completion; +- registered in-flight API calls covered by the ownership contract; +- terminal callbacks. + +This cannot make a raw C pointer safe if a caller begins a new call after the +object has already been freed. Callers must obey lifetime ownership. + +## Bounded internal parallelism + +A validated Task Kind may use internal participants while the Queue callback +remains the sole Task owner. + +Required shape: + +```text +one admitted Task owner +-> bounded participant count +-> bounded private work +-> join all participants +-> owner-only deterministic publication when required +-> Task-specific durable cursor +-> generic checkpoint +-> sequence_break ``` -## Primitives utilisées +Participant count and memory must fit the admitted Resource Governor contract. -| Primitive | Usage | -|---|---| -| `pthread_mutex_t` | Protection des données partagées | -| `pthread_cond_t` | Synchronisation producteur/consommateur | -| `pthread_create()` | Création des workers | -| `pthread_join()` | Attente de fin des workers | -| `pthread_cancel()` | Non utilisé pour interrompre une Task ; annulation coopérative | +No participant may silently exceed the installed sequence contract. -## Invariants de concurrence +## Atomicity does not imply serialism -1. **Un seul thread ncurses** : ncurses n'est jamais appelé depuis un - worker. Toute mise à jour de l'UI passe par des variables partagées - protégées. +Per-item scientific atomicity and cross-item execution width are separate. -2. **Mutex hiérarchique** : si plusieurs mutex sont acquis, toujours dans - le même ordre pour éviter les deadlocks. - -3. **Annulation coopérative** : les workers vérifient périodiquement un - drapeau d'annulation. Une Task n'est pas interrompue brutalement. - -4. **Réservation atomique** : la réservation du gouverneur est atomique. - Deux threads ne peuvent pas obtenir la même réservation. - -5. **Pas de callback sans réservation** : aucun callback de tâche n'est - invoqué sans réservation active. Cet invariant est maintenu même en - présence d'erreurs. - -6. **Retraite après callback** : la notification terminale finit avant la - destruction du userdata. Queue détruit la Task hors de son mutex et ne - conserve ensuite qu'un snapshot borné. - -7. **Fermeture d'ingress** : le propriétaire empêche les nouveaux appels Queue - avant `destroy()`. La fermeture interne attend le worker et chaque appel - enregistré avant le close ; elle ne peut rendre sûr un appel démarré après - la libération d'un pointeur C brut. - -8. **Parallélisme scientifique propriétaire** : lorsqu'un kind emploie des - participants internes, le callback Queue demeure l'unique propriétaire. Le - nombre de participants et leur mémoire sont admis par le Governor ; seul le - propriétaire publie le préfixe durable ordonné et joint tous les enfants. - -9. **Lease SSD par objet** : un lease scratch appartient à l'adresse exacte de - l'objet fourni par le caller. Tous ses champs sont lus/écrits sous le mutex - du contrôleur. Le caller lui garantit un accès exclusif et ne le copie, ne le - déplace ni ne le présente simultanément à deux contrôleurs. En production, - acquire/release passent par les wrappers Governor ; le Governor relâche son - mutex avant l'appel contrôleur, et le contrôleur ne rappelle jamais le - Governor. À la saturation légale `generation == UINT64_MAX`, seule la fin - du wrapper exact déjà sérialisé peut réconcilier sa propre opération et le - compte fondé sur les adresses ; une update publique au même watermark ne - peut pas rendre une autorité stale. - -10. **Owner SSD unique** : la TUI/main demande et poll l'opération ; au plus un - thread joinable exécute une opération bornée et ne touche jamais ncurses. - Le destroy le joint avant unregister. Une observation malformée enregistre - `ERROR` et ne confère aucune autorité de contrôle ou de lease. - -11. **Frontière projet** : les vues libèrent leurs borrows, puis la Queue est - annulée/jointe/détruite avant Project DB. Une Queue vide est créée ensuite. - Aucun callback terminal ne peut donc déréférencer une DB déjà fermée et - l'histoire d'un projet ne fuit pas dans le suivant. - -12. **Ordre d'arrêt global** : Queue et leases Task, puis fermeture projet, - join/unregister du binding SSD, contrôleur SSD, et enfin Governor. Un - unregister encore bloqué par un lease est un échec observable, jamais un - pointeur abandonné. - -## Anti-patterns - -### Deadlock - -```c -// ❌ Risque de deadlock -pthread_mutex_lock(&mutex_a); -pthread_mutex_lock(&mutex_b); // attend mutex_b - -// Dans un autre thread : -pthread_mutex_lock(&mutex_b); -pthread_mutex_lock(&mutex_a); // attend mutex_a → DEADLOCK +```text +PER_ITEM_ATOMICITY_REQUIRES_CROSS_ITEM_SERIALISM=NO +OWNER_ONLY_PUBLICATION_REQUIRES_SERIAL_PREPARATION=NO ``` -**Solution** : toujours acquérir les mutex dans le même ordre. +Current examples include selected RAW, selected Feature extraction, Candidate +Pair source work and outer Geometric Verification preparation. -### Race condition +Serialization is valid only where the subsystem's scientific, persistence, +library or measured-throughput contract proves it necessary. -```c -// ❌ Race condition -if (task->state == TASK_STATE_QUEUED) { - task->state = TASK_STATE_RUNNING; -} +## CPU/batch coupling -// ✅ Correct -pthread_mutex_lock(&task->mutex); -if (task->state == TASK_STATE_QUEUED) { - task->state = TASK_STATE_RUNNING; -} -pthread_mutex_unlock(&task->mutex); +CPU and batch/window are not globally independent dimensions. + +For a Task whose additional participants cannot do useful work while the +admitted item window remains one, a Task-specific capability may couple those +dimensions. + +Current validated examples include: + +```text +candidate_pair.generate/1 +features.extract.batch/1 ``` -### Use-after-free +This is not a universal rule for all Task Kinds. -```c -// ❌ Use-after-free -task_destroy(task); -task_callback(task); // task est libéré +## Project lifetime boundary -// ✅ Correct : le callback est entièrement revenu avant la destruction -task_callback(task); -task_destroy(task); +Before closing a project: + +```text +views release Project DB borrows +-> Queue is cancelled/joined/destroyed +-> Project DB closes +-> fresh empty Queue may be created for the next project ``` -## Validation +A terminal callback must never observe a Project DB already destroyed. -Les readers Visual Index sont sans état partagé mutable. Une query copie la -liste bornée des segments sous le mutex DB, puis effectue hash, lectures et -accumulation après déverrouillage. Un update ne rend le nouveau segment visible -qu'au commit memberships+segment ; une query en cours garde son snapshot. +Project-specific runtime history must not leak into the next project. -Pour tout ticket touchant la concurrence, exécuter : +## Global shutdown boundary + +The current shutdown order preserves ownership across: + +```text +Task Queue and Task leases +-> project close +-> join/unregister SSD binding +-> SSD controller +-> Resource Governor +``` + +A scratch unregister blocked by a real outstanding lease is an observable +failure, not permission to abandon a live pointer. + +## SSD lease ownership + +A scratch lease belongs to the exact caller-owned lease object used for the +operation. + +Its mutable fields are controlled under the SSD controller mutex. + +The caller must not: + +- copy a live lease; +- move a live lease; +- present the same lease object to two controllers; +- release through a different ownership path. + +Production acquire/release uses the Resource Governor wrappers. + +The Governor releases its own mutex before entering the controller, and the +controller does not callback into the Governor while holding its mutex. + +Scratch remains storage capacity, never RAM admission. + +## Visual Index readers + +Visual Index query readers do not share mutable query state. + +A query obtains a bounded segment snapshot under the Project DB boundary, then +performs hash/read/accumulation after release. + +A concurrent update makes a new segment visible only at its canonical commit +boundary. An already running query continues with its retained snapshot. + +## OpenCV process-wide state + +OpenCV thread configuration is process-wide. + +The active heavy Queue callback owns temporary mutation of that setting where a +Task contract requires it and restores the previous/baseline value on all exit +paths. + +Internal participants must not independently race `cv::setNumThreads()`. + +For Feature batch, cross-image participants are used while internal OpenCV +threading is controlled explicitly. + +## Vulkan boundary + +ORB Vulkan concurrency is validated separately from portable TSan. + +The production AUTO contract currently uses: + +```text +normal inflight depth = 1 +private validated safety depth = 2 +helpers = 0 +``` + +Depth 2 is a private safety/benchmark capability and was rejected as the normal +useful setting by measured throughput. + +A Vulkan backend failure produces complete CPU fallback before publication. +Partial GPU scientific output is never published. + +## TSan policy + +For project concurrency changes, use a dedicated TSan build that matches the +supported proof boundary. + +The retained global maintenance matrix used: + +```text +GCC/G++ +Vulkan disabled +selected concurrent targets +deterministic repetitions +``` + +and completed the retained 14/14 target matrix plus 220 repetitions. + +The narrow suppression file is: + +```text +tests/tsan-opencv.supp +``` + +It covers external non-instrumented OpenCV/TBB objects only. + +It must not suppress Lardon3D frames. + +Therefore the correct retained claim is not "TSan proves all concurrency". +It is: + +```text +portable project concurrency matrix passed under the documented qualification +external OpenCV/TBB reports are narrowly qualified +Vulkan concurrency has a separate validation boundary +``` + +## What TSan does not prove + +TSan is useful for instrumented conflicting memory access and some +synchronization misuse. + +It does not prove absence of: + +- deadlock; +- lost wakeup caused by incorrect predicate design; +- lifetime bugs outside the exercised paths; +- races hidden inside non-instrumented external libraries; +- Vulkan driver/runtime correctness; +- scientific determinism. + +Lock-order review, ownership reasoning and deterministic tests remain required. + +## Sanitizer command policy + +Do not encode fixed `-j8` as canonical validation. + +Example configuration: ```sh -# Build TSan -CC=clang meson setup build-tsan --wipe -Db_sanitize=thread -Db_lundef=false -meson compile -C build-tsan -j8 +CC=gcc CXX=g++ meson setup build-tsan -Db_sanitize=thread -Db_lundef=false -Dvulkan_orb=disabled +meson compile -C build-tsan meson test -C build-tsan --print-errorlogs ``` -TSan détecte automatiquement : +Use host-aware compile/test parallelism unless the proof itself requires +serialization. -- les accès concurrents conflictuels instrumentés ; -- certaines utilisations incohérentes des primitives de synchronisation. +Do not repeatedly wipe an unchanged TSan tree. -Il ne prouve pas l'absence de deadlock, de signal perdu ou de bug dans une -bibliothèque non instrumentée. Les invariants de lifetime et d'ordre de locks -restent donc soumis aux tests déterministes et à la revue. +## Concurrency review checklist -### Preuve TSan globale courante +Before closing a concurrency-sensitive change, verify: -La matrice fraîche emploie GCC/G++ 16.2.1 et désactive explicitement Vulkan. -Elle passe 14/14 cibles couvrant Task, Project, Queue, Governor, registre/leases -SSD, contrôleur SSD, observateur/TUI async, Candidate, Visual Index, Feature, -Matcher et GV, puis 220/220 répétitions déterministes : **234/234** au total. +- ncurses remains main-thread-only; +- every shared mutable field has an explicit synchronization owner; +- condition predicates are checked in loops; +- lock order remains consistent; +- no Task uses forced asynchronous cancellation; +- Queue callbacks have an active reservation; +- internal participants stay within the admitted contract; +- all children join on every exit path; +- owner-only publication remains ordered where required; +- project-close ordering prevents DB use-after-close; +- SSD lease ownership remains exact; +- Task userdata outlives terminal notification; +- appropriate deterministic concurrency tests pass; +- portable TSan qualification is preserved; +- Vulkan validation is reported separately; +- ASan/UBSan is run when the change also affects lifetime/memory. -La seule liste de suppressions est `tests/tsan-opencv.supp`, limitée aux objets -partagés externes non instrumentés `libopencv_features.so`, -`libopencv_core.so` et `libtbb.so`. Elle ne masque aucune frame Lardon3D. Les -warnings GCC `-Wmaybe-uninitialized` des contrôles OpenCV Feature/SIFT sont -classés non matériels : le callback fournit une Task non nulle et le helper -initialise la structure avant toute autre sortie d'échec. Les warnings OpenCV -du build GV appartiennent aux headers externes. +## Current retained evidence -Cette preuve TSan ne vaut pas validation de concurrence Vulkan. Le backend -ORB Vulkan réel est couvert séparément par le build Clang Vulkan-on 939/939, -la suite 65/65 et ses tests de backend/handle/publication ; cette séparation -doit rester explicite dans tout rapport. +The canonical global-maintenance record is: -## Checklist de concurrence +```text +docs/architecture/global_maintenance_audit.md +GLOBAL_MAINTENANCE_AUDIT=PASS/FROZEN +``` -Avant de livrer un ticket touchant la concurrence : +The current A6000 checkpoint is later: -- [ ] Toutes les variables partagées sont protégées par un mutex -- [ ] Les mutex sont toujours libérés (même en cas d'erreur) -- [ ] Les variables de condition sont vérifiées dans une boucle `while` -- [ ] Aucun appel ncurses depuis un worker -- [ ] L'annulation des Tasks est coopérative (pas de `pthread_cancel`) -- [ ] TSan ne signale aucune erreur -- [ ] Le build ASan ne signale aucune fuite mémoire liée aux threads +```text +real-a6000-pre-sfm-2026-09-02 +REAL_A6000_PRE_SFM=PASS/FROZEN +``` + +The later A6000 proof exercised current bounded parallel paths through selected +Feature batch, Candidate, Matcher, Geometric Verifier v3 and Tracks without +changing the historical TSan qualification. + +Historical evidence remains historical; new changes require validation scoped +to their actual concurrency surface. + +## Summary + +```text +NCURSES_OWNER=MAIN_THREAD_ONLY +ACTIVE_HEAVY_QUEUE_CALLBACKS=1 +TASK_CANCELLATION=COOPERATIVE +INTERNAL_PARALLELISM=BOUNDED + +PER_ITEM_ATOMICITY_REQUIRES_CROSS_ITEM_SERIALISM=NO +OWNER_ONLY_PUBLICATION_REQUIRES_SERIAL_PREPARATION=NO + +TSAN_PORTABLE_PROJECT_MATRIX=QUALIFIED_PASS +TSAN_EXTERNAL_OPENCV_TBB=QUALIFIED +VULKAN_CONCURRENCY_VALIDATION=SEPARATE + +NO_FIXED_GLOBAL_J8=YES +NO_REPEATED_UNCHANGED_WIPE=YES + +RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT +SERIALISM_REQUIRES_PROOF=CANONICAL +``` diff --git a/docs/development/testing.md b/docs/development/testing.md index b2dba9b..70786f9 100644 --- a/docs/development/testing.md +++ b/docs/development/testing.md @@ -1,136 +1,298 @@ -# Procédures de test +# Testing -## Vue d'ensemble +## Status -Lardon3D utilise le framework de test intégré à Meson. Chaque module possède -un fichier de test dans `tests/` correspondant au module testé. +```text +DOCUMENTATION_LANGUAGE=ENGLISH +TEST_POLICY=HOST_AWARE +REPEATED_UNCHANGED_EXPENSIVE_VALIDATION=AVOID +TSAN_OPEN_CV_TBB_QUALIFICATION=REQUIRED +VULKAN_CONCURRENCY_VALIDATION=SEPARATE +``` -## Lancer les tests +Lardon3D uses Meson's test runner. Tests live under `tests/` and combine unit, +integration, persistence, restart, resource and real-path validation. + +## Normal commands + +Run the configured suite: ```sh -# Tous les tests meson test -C build --print-errorlogs +``` -# Un test spécifique -meson test -C build test_task_queue --print-errorlogs +Run one named test: -# Tests avec verbose +```sh +meson test -C build --print-errorlogs +``` + +Verbose execution: + +```sh meson test -C build -v --print-errorlogs +``` -# Réexécuter uniquement les tests échoués +Re-run failures only: + +```sh meson test -C build --reprint=failed ``` -## Structure des tests +Use the names registered by the current `meson.build`; this document does not +maintain a second authoritative list of every test target. + +## Validation policy + +Validation must match the change. + +A documentation-only change normally requires: ```text -tests/ -├── test_task_queue.c # tests de la file de tâches -├── test_task.c # tests du module task -├── test_resource_governor.c # tests du gouverneur -├── test_hardware_profile.c # tests du profil matériel -├── test_import.c # tests de l'import -├── test_project.c # tests des projets -└── test_*.c # autres modules +git diff --check +targeted content checks +targeted link/authority review ``` -## Écrire un test +It does not justify wiping and rebuilding unchanged code. -```c -#include -#include "lardon3d/task.h" +A code change normally requires, in increasing scope: -void test_task_create(void) { - task_estimate_t est = { - .ram_bytes = 1024 * 1024, - .gpu_bytes = 0, - .cpu_weight = 1, - .io_weight = 0, - .batch_size = 10, - .batch_max = 100 - }; - task_t *t = task_create("test", &est, NULL, NULL); - g_assert_nonnull(t); - g_assert_cmpint(task_get_state(t), ==, TASK_STATE_IDLE); - task_destroy(t); -} - -int main(int argc, char **argv) { - g_test_init(&argc, &argv, NULL); - g_test_add_func("/task/create", test_task_create); - return g_test_run(); -} +```text +targeted build +targeted tests +broader affected suite +sanitizer or concurrency validation when relevant +full suite when the change or release boundary justifies it ``` -## Conventions +Do not repeatedly rerun an unchanged expensive suite between documentation +edits merely to create activity. -1. **Préfixe `test_`** : chaque fonction de test porte le préfixe `test_`. -2. **Chemin hiérarchique** : le nom du test suit le pattern `/module/action`. -3. **Asserts GLib** : utiliser `g_assert_*` pour les vérifications. -4. **Nettoyage** : chaque test libère toutes ses ressources. -5. **Isolation** : un test ne dépend pas de l'état d'un autre test. -6. **Déterminisme** : les tests ne dépendent pas de l'heure, du filesystem - ou de l'état réseau (sauf test d'import). +## Host-aware parallelism -## Commentaires source +Build and test parallelism are host-aware. -Les commentaires documentent le pourquoi et les contrats non évidents : -invariants, propriété et durée de vie, persistance, ainsi que limites et -frontières de ressources. Les API publiques documentent leurs contrats non -évidents. Ils ne paraphrasent pas le code ligne par ligne et sont mis à jour -avec tout changement de comportement. +Do not encode a project-wide fixed `-j8`, `--num-processes 1`, or equivalent +constant as canonical policy. -## Tests unitaires vs tests d'intégration +The correct width depends on the current machine, interactive reserve, memory, +toolchain and workload. Use all safe useful host capacity while preserving the +defined interactive reserve. -`test-visual-index` couvre les descriptors synthétiques, le retrieval ORB réel, -les filtres inter-ScanSets, quatre queries concurrentes, la corruption/absence/ -troncature d'un segment et 4 000 Feature Sets synthétiques. Le scénario de -reprise `visual_index.update` est exercé dans `test-feature-task`. +```text +RESOURCE_UTILIZATION_POLICY=MAXIMUM_SAFE_USEFUL_THROUGHPUT +SERIALISM_REQUIRES_PROOF=CANONICAL +``` -| Type | Portée | Fichier | -|---|---|---| -| Unitaire | Un module isolé | `tests/test_.c` | -| Intégration | Interaction entre modules | `tests/test_.c` avec dépendances réelles | +If a temporary validation must be serialized for determinism, diagnosis or a +known tool limitation, label that serialization as test-specific evidence +rather than a global default. -## Validation par ticket +## Fresh build policy -Avant de livrer un ticket, exécuter la séquence complète : +Do not use `meson setup --wipe` by default. + +Prefer: ```sh -# 1. Build clean -CC=clang meson setup build --wipe -meson compile -C build -j8 - -# 2. Tests -meson test -C build --print-errorlogs - -# 3. Style -git diff --check - -# 4. Si mémoire/concurrence touchés -CC=clang meson setup build-asan --wipe -Db_sanitize=address,undefined -meson compile -C build-asan -j8 -meson test -C build-asan --print-errorlogs - -# 5. Si concurrence touchée -CC=clang meson setup build-tsan --wipe -Db_sanitize=thread -Db_lundef=false -meson compile -C build-tsan -j8 -meson test -C build-tsan --print-errorlogs +meson setup --reconfigure build +meson compile -C build ``` -## Dépannage +Create or wipe a build directory when the configuration genuinely needs a +fresh environment, for example: -### Test qui échoue en ASan +```text +different sanitizer set +portable Vulkan-off proof +Vulkan-on proof +compiler-family change +known stale/corrupt build directory +release-grade clean proof +``` -Vérifier les durées de vie des allocations. Ne jamais libérer un objet puis -y accéder. Vérifier que chaque `task_destroy()` est appelée. +Repeated wipes of the same unchanged configuration waste time and invalidate +incremental-build advantages. -### Test qui échoue en TSan +## Sanitizers -Vérifier que toutes les variables partagées sont protégées par un mutex. -Vérifier que ncurses est utilisé uniquement depuis le thread principal. +### ASan / UBSan -### Test qui échoue uniquement en release +For memory, lifetime, ownership or undefined-behavior changes, use a dedicated +sanitizer build. -Vérifier les assertions et les overflow arithmétiques. Compiler avec -`-fsanitize=undefined` pour détecter les comportements indéfinis. +Example configuration: + +```sh +CC=clang CXX=clang++ meson setup build-asan -Db_sanitize=address,undefined +meson compile -C build-asan +meson test -C build-asan --print-errorlogs +``` + +Reconfigure or wipe only when the existing sanitizer directory does not match +the requested configuration. + +### LeakSanitizer qualification + +The retained global maintenance evidence must not be summarized as +`LSan 64/64`. + +The full first leak-enabled run exposed an externally attributed OpenCL loader +leak and two timeout anomalies. The retained qualified result is: + +```text +ASan/UBSan full suite: PASS with detect_leaks=0 +proved subset without the external loader: LSan PASS +full leak-enabled suite: not an unqualified PASS +``` + +Preserve that distinction in future reports unless new evidence supersedes it. + +## ThreadSanitizer + +Concurrency changes require TSan where the instrumented boundary is meaningful. + +The retained global maintenance TSan proof used GCC/G++ with Vulkan disabled +and covered the selected concurrent targets plus deterministic repetitions. + +The only retained suppression file is: + +```text +tests/tsan-opencv.supp +``` + +Its purpose is limited to external OpenCV/TBB objects. It must not suppress +Lardon3D frames. + +Therefore never report a blanket statement such as: + +```text +TSan proves the entire Vulkan build race-free +``` + +The valid qualification is: + +```text +project concurrent paths covered by the retained portable TSan matrix +external OpenCV/TBB reports qualified by the narrow suppression boundary +Vulkan concurrency validated separately +``` + +## Vulkan validation + +ORB Vulkan uses a separate validation boundary. + +The retained global maintenance evidence includes a Vulkan-on build and suite +on the real Radeon 780M/RADV host, plus dedicated backend/handle/publication +tests. + +That evidence is not interchangeable with TSan. + +SIFT/RootSIFT feasibility results did not establish a production GPU backend; +do not turn feasibility checks into production validation claims. + +## Determinism and repetition + +Repeat tests when repetition proves something specific: + +```text +deterministic restart +race sensitivity +resource adaptation +ordering stability +flaky regression reproduction +``` + +Do not repeat unchanged tests without a stated purpose. + +When repetition is the evidence, record: + +```text +exact test/corpus +run count +relevant configuration +success/failure count +digest or invariant when applicable +``` + +## Test isolation + +Tests should: + +- own and clean up their temporary resources; +- avoid depending on another test's execution order; +- avoid network state unless the test explicitly owns that dependency; +- use synthetic/private Resource snapshots where the test is about deterministic + policy rather than live host telemetry; +- avoid changing global process state without restoring it. + +OpenCV thread configuration is process-wide and must be restored on every exit +path in tests that change it. + +## Public-header validation + +When a public C header changes, run a standalone C17 syntax probe in addition to +normal build coverage. + +Conceptually: + +```sh +cc -x c -std=c17 -fsyntax-only -Iinclude -include lardon3d/
.h /dev/null +``` + +Use the current supported compiler matrix when the change affects ABI or +C/C++ interoperability. + +## Source comments + +Source comments explain non-obvious contracts: + +```text +invariants +ownership and lifetime +persistence ordering +resource boundaries +recovery behavior +scientific constraints +``` + +They should not paraphrase obvious code line by line. + +Repository source comments are English. + +## Current retained global maintenance evidence + +The canonical detailed record is: + +```text +docs/architecture/global_maintenance_audit.md +GLOBAL_MAINTENANCE_AUDIT=PASS/FROZEN +``` + +That historical checkpoint includes fresh portable/Vulkan builds, full suites, +sanitizer work, TSan work, public-header probes, ABI/application-link checks and +independent review. + +It remains historical evidence. New changes require only the validation +appropriate to the changed surface unless a new global checkpoint is being +created. + +## Ticket closure checklist + +Before closing a code ticket: + +- confirm the requested scope only was changed; +- run `git diff --check`; +- run targeted tests for changed behavior; +- run the affected broader suite when justified; +- run ASan/UBSan for memory/lifetime-sensitive changes; +- run TSan for concurrency-sensitive project code when applicable; +- keep Vulkan validation separate from portable TSan claims; +- preserve exact external-library qualifications; +- avoid fixed host-parallelism constants; +- avoid repeated unchanged clean builds or suites; +- report what actually ran, not a stronger claim. + +For documentation-only remediation, use documentation checks rather than +rebuilding unchanged production code.