From aa8ab7ae695601c837f45d439d2ec6f620f0854d Mon Sep 17 00:00:00 2001 From: fy59 Date: Thu, 17 Sep 2026 13:07:06 +0200 Subject: [PATCH] docs(sync): close web end-to-end validation --- AGENTS.md | 4 + CHANGELOG.md | 6 ++ README.md | 5 +- docs/architecture.md | 9 +++ docs/current_state.md | 4 + .../design/sync_web_end_to_end_v1_evidence.md | 75 +++++++++++++++++++ docs/exchange_format.md | 10 +++ docs/roadmap.md | 7 +- docs/sync_exchange.md | 12 +++ docs/sync_rollout_readiness.md | 38 ++++++++++ docs/tests.md | 8 ++ 11 files changed, 173 insertions(+), 5 deletions(-) create mode 100644 docs/design/sync_web_end_to_end_v1_evidence.md create mode 100644 docs/sync_rollout_readiness.md diff --git a/AGENTS.md b/AGENTS.md index 4a24275..b2022d5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,6 +17,10 @@ Trainlog consists of: The desktop SQLite database is the canonical long-term history. +The full-generation orchestrator and Web sync surface are implemented behind a +trusted local opt-in. Default deployed automatic exchange remains V3 until the +separate controlled-rollout decision. + Android has an independent local SQLite store for offline capture. SQLite database files are never synchronized directly. diff --git a/CHANGELOG.md b/CHANGELOG.md index b89d9c2..524f110 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,12 @@ ## Unreleased — 0.1.2 +- Completed the grouped `TRAINLOG_SYNC_WEB_END_TO_END_V1` mission: the opt-in + full-generation orchestrator, protected asynchronous Web API, global sync + control, durable reconnect reporting, Dashboard refresh and read-only draft + summary pass isolated C, Python, Android, frontend and Firefox validation. + Default V3, physical MTP, Drive and deployment remain unchanged. + - Completed `TRAINLOG_SYNC_GENERATION_ACK_V1` with desktop schema v21 and Android schema v20. Added coherent per-platform capture, strict bounded manifest V1, immutable manifest-last publication, whole-generation SQLite diff --git a/README.md b/README.md index 1871a37..29830e2 100644 --- a/README.md +++ b/README.md @@ -323,8 +323,9 @@ As of 2026-09-17: `WEB_DASHBOARD_V1=PASS/FROZEN`; the other four Web routes remain placeholders; - `TRAINLOG_WEB_V1=CONTRACT_FROZEN / IMPLEMENTATION_STARTED`; - `TRAINLOG_SYNC_GENERATION_ACK_V1=PASS/FROZEN` through explicit staged - services; automatic synchronization still selects V3. The next cursor is - `TRAINLOG_SYNC_ORCHESTRATOR_REPORT_V1`. + services; automatic synchronization still selects V3. The opt-in grouped Web + path is validated in isolation; the next boundary is + `TRAINLOG_SYNC_DEPLOYMENT_READINESS_V1`, not an authorized deployment. The latest executable result belongs in [current state](docs/current_state.md), not in multiple README narratives. diff --git a/docs/architecture.md b/docs/architecture.md index fab92aa..0daf34f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -414,6 +414,15 @@ carrier. `trainlog-syncd` is a small user-session agent. +### Full-generation application orchestrator + +The opt-in full-generation path is owned by `sync_orchestrator.py`, launched as +an owned process by the C Web adapter. It uses the same XDG `sync.lock` as the +legacy engine, never borrows the Web process's SQLite handle, and accepts peer +transport only from a trusted local configuration. The HTTP event loop only +admits work and reads a bounded durable report. V3 remains the default when the +trusted generation configuration is absent. + It polls for a new Android request and invokes the same shared C synchronization engine used by the TUI. diff --git a/docs/current_state.md b/docs/current_state.md index 1feee93..8a984c7 100644 --- a/docs/current_state.md +++ b/docs/current_state.md @@ -47,6 +47,10 @@ parallel implementation of its rules. | Web Dashboard tiles | `WEB_DASHBOARD_TILES_V1=PASS/FROZEN` | | Web Dashboard visualizations | `WEB_DASHBOARD_VISUALIZATIONS_V1=PASS/FROZEN` | | Web Dashboard V1 | `WEB_DASHBOARD_V1=PASS/FROZEN` | +| Sync orchestrator/report V1 | `TRAINLOG_SYNC_ORCHESTRATOR_REPORT_V1=PASS/FROZEN` | +| Web sync API V1 | `TRAINLOG_WEB_SYNC_API_V1=PASS/FROZEN` | +| Web sync button V1 | `TRAINLOG_WEB_SYNC_BUTTON_V1=PASS/FROZEN` | +| Sync Web end-to-end wrapper | `TRAINLOG_SYNC_WEB_END_TO_END_V1=PASS/FROZEN` (isolated) | | Local Web | `TRAINLOG_WEB_V1=CONTRACT_FROZEN / IMPLEMENTATION_STARTED` | | Complete synchronization gap contract | `TRAINLOG_SYNC_GAP_CONTRACT_V1=CONTRACT_FROZEN / IMPLEMENTATION_IN_PROGRESS` | | Isolated synchronization test environment | `TRAINLOG_SYNC_TEST_ENV_V1=PASS/FROZEN` | diff --git a/docs/design/sync_web_end_to_end_v1_evidence.md b/docs/design/sync_web_end_to_end_v1_evidence.md new file mode 100644 index 0000000..2145a47 --- /dev/null +++ b/docs/design/sync_web_end_to_end_v1_evidence.md @@ -0,0 +1,75 @@ +# Sync Web end-to-end V1 evidence + +Status: `TRAINLOG_SYNC_WEB_END_TO_END_V1=PASS/FROZEN` on the development +branch. This status means implemented and validated with isolated stores and a +directory transport adapter. It does not mean physical MTP, Drive, deployment, +or a merge to `main`. + +## Production call graph + +```text +React SyncControl + -> POST /api/v1/sync (Host + Origin + ephemeral CSRF) + -> C Web admission (trusted opt-in config, one request identity) + -> owned sync_orchestrator.py process + -> common XDG sync.lock + -> trusted peer adapter + -> Android SyncGenerationService / desktop sync_generation_exchange.py + -> durable generation and ACK evidence + -> bounded run report + -> GET /api/v1/sync/status polling + -> Dashboard Core snapshot refresh after local commit +``` + +The deployed/default path remains V3. Full generation mode is unavailable +unless `TRAINLOG_SYNC_GENERATION_CONFIG` names an absolute, trusted local +configuration. HTTP input cannot select a database, executable, transport, +capability, path, timeout, or causal bypass. + +## Requirement-to-proof matrix + +| Requirement | Production entry point | Exact proof | Observed result | +|---|---|---|---| +| Web admission and passive status | `web_server.c` `/api/v1/sync`, `/api/v1/sync/status` | `web_server`, `sync API` | 202/409/403/405, passive GET and CSRF header pass | +| Async owned worker | `sync_orchestrator.py` | `sync_orchestrator`, Firefox scenario | HTTP remains responsive and child is reaped | +| Durable typed phases | atomic `.sync-run.json` replacement | `test_sync_orchestrator.py` | correlated completion survives reopen | +| Common contention | XDG `sync.lock` | orchestrator conflict test and legacy lock contract | second owner fails without a second import | +| Strict compatibility | orchestrator capability set | missing-capability test | `explicitly_degraded`, missing capabilities retained | +| Generation semantics | desktop and Android generation services | `sync_generation_exchange`, `SyncGenerationServiceTest` | coherent capture, whole transaction, ACK and replay pass | +| Independent producers | public C fixture and Android repository fixture | `androidGenerationCapturePublishConsumeAndAckSurviveReopen` | both direction chains pass after reopen | +| All mandatory domains | generation descriptors and apply order | generation suites and causal seven-kind test | required empty artifacts included and validated | +| Causal immutability | `sync_causal_publications` | causal multi-generation tests | operation bytes/digests unchanged | +| Late failure | transaction-neutral apply helpers | desktop and Android late-domain tests | prior writes roll back; no consumed ACK | +| Lost/late ACK | durable ACK tables | generation restart tests | exact ACK reconstructed; replay unchanged | +| UI click/reload | `SyncControl` | `test_web_sync_browser.py` | real Firefox click, completion, draft details and reload pass | +| Draft visibility | status `drafts` projection | `SyncControl.test.tsx`, Firefox scenario | stable ID/state/type/count shown read-only | +| Dashboard freshness | `onCommitted` callback | component test and browser flow | Core snapshot refetched after committed phase | +| Mutation security | existing Host/Origin/CSRF plus strict body | `web_server`, `sync API` | proxy origin retained; no CORS or client paths | +| Optional AI | separate `ai_midpoint` and `ai_post_sync` | orchestrator report validation | not-configured remains explicit, never main success | +| Default isolation | missing trusted config | C HTTP test | button disabled and POST returns explicit conflict | + +The Firefox peer command in the browser scenario is the bounded directory +transport fixture. Cross-implementation Android/desktop behavior is proven by +the production generation suites in the same final validation run. The test +does not claim physical-device timing, MTP object behavior, or Drive access. + +## Bounds + +- request body: existing 4 KiB HTTP limit; +- headers: existing 8 KiB limit; +- durable report and each worker output stream: 64 KiB; +- trusted configuration: 16 KiB; +- diagnostic: 1024 UTF-8 bytes; +- draft summaries: 32; +- peer command: 16 fixed arguments, each at most 4096 bytes; +- trusted timeout: 1 to 900 seconds; +- frontend polling: one request at a time, 750 ms between completions; +- generation retention: existing eight outgoing generations per consumer. + +## Real limitations + +Android must participate through its foreground synchronization flow to create +a publication correlated to the accepted run. No always-on Android background +service was added. A waiting phase is therefore truthful and may require user +action. Physical MTP and Drive remain rollout gates. A browser disconnect does +not cancel the worker; server shutdown terminates only its owned process group. diff --git a/docs/exchange_format.md b/docs/exchange_format.md index f497066..3afdf7f 100644 --- a/docs/exchange_format.md +++ b/docs/exchange_format.md @@ -701,3 +701,13 @@ Generation membership never changes a causal V1 operation. The immutable operation retains null `publication_context`, its original ID and digest; `sync_causal_publications` records first emission and retransmission by operation/generation identity. +## Operational full-generation run report + +The Web/application boundary uses a separate bounded operational report; it +does not extend any frozen business artifact. It carries the stable run and +request IDs, trigger, requested/effective mode, peer and generation IDs, +canonical phase, monotonic progress revision, real or unavailable counters, +per-domain coverage, bounded draft summaries, diagnostics and separate AI +results. `completed` is accepted only after the worker supplies the correlated +manifest/ACK evidence. The report is at most 64 KiB and is replaced atomically +with private permissions. diff --git a/docs/roadmap.md b/docs/roadmap.md index b2e642e..77cbd92 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -33,7 +33,7 @@ visualizations are implemented; the other four Web routes remain placeholders. ## Current cursor ```text -CURRENT_OPERATIONAL_CURSOR=TRAINLOG_SYNC_ORCHESTRATOR_REPORT_V1 +CURRENT_OPERATIONAL_CURSOR=TRAINLOG_SYNC_DEPLOYMENT_READINESS_V1 ``` The focused `TRAINLOG_SYNC_CAUSAL_DELETE_V1_CLOSEOUT` is complete before this @@ -42,8 +42,9 @@ cross-platform draft revision identity and resource-bound admission are closed. `TRAINLOG_SYNC_GENERATION_ACK_V1=PASS/FROZEN`: both platforms expose explicit coherent capture, immutable manifest publication, whole-generation transactional consumption, durable correlated ACK and restart/replay entry -points. Automatic V3 selection remains unchanged. The next cursor is only -`TRAINLOG_SYNC_ORCHESTRATOR_REPORT_V1`; no orchestrator work starts here. +points. The orchestrator, Web API and Web control are now complete behind an +explicit trusted opt-in. Automatic V3 selection remains unchanged. The next +boundary is deployment readiness; no rollout starts here. `TRAINLOG_SYNC_GAP_CONTRACT_V1=CONTRACT_FROZEN / IMPLEMENTATION_IN_PROGRESS` defines the complete-sync target and its bounded dependency order. diff --git a/docs/sync_exchange.md b/docs/sync_exchange.md index e638452..3031110 100644 --- a/docs/sync_exchange.md +++ b/docs/sync_exchange.md @@ -777,6 +777,18 @@ and retransmission because generation membership lives only in ## 19. Legacy active-protocol limitations +The Web end-to-end tranche adds an explicit opt-in application orchestrator. +`POST /api/v1/sync` admits one correlated run and `GET /api/v1/sync/status` is +passive. The worker owns blocking peer/transport work, shares `sync.lock` with +the TUI and daemon, and publishes bounded durable phases. Android foreground +participation is required for a fresh run-correlated publication. No filesystem +mtime or old generation can satisfy that freshness boundary. + +The opt-in is `TRAINLOG_SYNC_GENERATION_CONFIG`, an absolute trusted local +configuration path. It is launcher-owned and cannot be supplied through HTTP. +Without it, the staged path reports disabled and the deployed V3 path remains +unchanged. + The complete-sync semantics remain frozen separately in [`TRAINLOG_SYNC_GAP_CONTRACT_V1`](design/sync_gap_contract_v1.md). The staged generation slice implements its batch boundary but does not activate it. diff --git a/docs/sync_rollout_readiness.md b/docs/sync_rollout_readiness.md new file mode 100644 index 0000000..91e7b57 --- /dev/null +++ b/docs/sync_rollout_readiness.md @@ -0,0 +1,38 @@ +# Full-generation sync rollout readiness + +This is a non-executed rollout checklist. The development branch implements the +path; the daily installation still uses V3. + +## Compatible set + +- desktop binary containing schema v21, the Web API and the generation worker; +- `sync_generation_exchange.py` and all matching transaction-neutral codecs; +- Android schema v20 with `SyncGenerationService`; +- peers advertising manifest V1, ACK V1, causal deletion V1, history V4 and + execution draft V1 under their stable `peer_` identities; +- a trusted local orchestrator configuration referencing the packaged peer/MTP + helper with fixed arguments. + +## Controlled sequence + +1. Back up both SQLite stores independently and record their schema versions. +2. Stop the user services only during the separately authorized deployment. +3. Install the mutually compatible desktop helper, binary and Android APK. +4. Migrate synthetic copies first; migration rollback means restoring a backup, + not merely replacing the binary. +5. Verify peer identities and advertised capabilities without pairing by name. +6. Perform one non-destructive foreground Android publication and desktop run. +7. Verify both manifest digests, both durable ACK records, lineage and readers. +8. Keep V3 selectable until the controlled full-generation smoke is accepted. + +## Recovery limits + +Do not delete an unacknowledged generation, the last acknowledged generation, +its previous valid generation, or causal evidence. An ACK timeout is unresolved, +not rollback. Eight retained outgoing generations exhaust admission and require +operator diagnosis; this tranche adds no tombstone or evidence garbage +collection. A migrated database is recovered from its verified backup, never by +assuming an older binary reverses its migration. + +Physical MTP, actual Drive configuration, service coordination, installed APK, +main merge and deployment are intentionally not performed here. diff --git a/docs/tests.md b/docs/tests.md index 8e65c64..512d48c 100644 --- a/docs/tests.md +++ b/docs/tests.md @@ -1216,6 +1216,14 @@ does not author expected business JSON or repair artifacts. After these additions the direct normal suites report 78/78 Meson tests and 216 Android tests: 212 passed, four historical-fixture skips, zero failures. + +The grouped Web end-to-end tranche raises the direct normal and sanitizer +inventories to 79/79 and adds `sync_orchestrator`, protected sync API +coverage, frontend sync assertions and a real headless Firefox scenario in +`tests/test_web_sync_browser.py`. Its detailed matrix is +[`sync_web_end_to_end_v1_evidence.md`](design/sync_web_end_to_end_v1_evidence.md). +The browser uses the production C server and worker with an isolated directory +transport fixture; physical MTP and Drive are not claimed. The publication adapter is an isolated directory/object-store double. No physical MTP, phone, Drive, active service, or user database participates. ## Causal deletion V1