Tracksolid deployment with timescale & grafana with backup
Find a file
david kiania b11294009b
Some checks are pending
Static Analysis / static (push) Waiting to run
Tests / test (push) Waiting to run
Static Analysis / static (pull_request) Waiting to run
Tests / test (pull_request) Waiting to run
fix(security,ingest): 260702 audit — secure the stack, correct poller counters
Security:
- .dockerignore + Dockerfile: stop baking .env / the 346MB OSM pbf into image
  layers; install pinned from uv.lock (reproducible builds) (SEC-04/05).
- docker-compose: DB port binds ${DB_BIND_ADDR:-127.0.0.1} — loopback-only by
  default; remote tooling moves to an SSH tunnel (SEC-01).
- webhook_receiver: CRITICAL startup warning + WEBHOOK_REQUIRE_TOKEN=1 fail-closed
  when JIMI_WEBHOOK_TOKEN is empty (SEC-02 / FIX-W01).

Correctness:
- FIX-M22/E07: capture cur.rowcount BEFORE RELEASE SAVEPOINT in poll_alarms/
  poll_trips/poll_parking — the RELEASE reported -1, producing "Alarms: -4 new
  events inserted" logs and negative ingestion_log.rows_inserted.
- FIX-W02: parse application/json push bodies (were silently dropped).
- FIX-W03: move webhook DB work off the event loop via asyncio.to_thread.
- FIX-M23: poll_trips phased so no txn/connection is held across Tracksolid +
  Nominatim (1 req/s) network calls.
- FIX-M24: sync_devices disables devices absent from every target (guarded).
- FIX-W04: reject device-clock-garbage alarm_time (2019 timestamps observed).
- get_token(): don't relabel already-aware timestamptz expiries (BUG-P9).

Observability/lifecycle:
- migration 21: v_ingest_health restricted to active pipeline endpoints so
  one-shot tools stop wedging /health/ingest at 'stale' (dry-run verified).
- FIX-M25: daily purge_audit_logs() trims ingestion_log (90d) + refresh_log (180d).
- remove orphaned duplicate migrations/10_driver_clock_views.sql; ruff lint config.

+5 webhook tests (82 pass). Report/plan/work-log in docs/reports/260702_*.
Local only; not deployed. CLAUDE.md fix-history edits left uncommitted (that file
also carries unrelated in-progress edits).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-02 09:51:02 +03:00
.claude feat(dwh): bronze pipeline migrations, runbook, and execution manual 2026-04-25 01:07:53 +03:00
.forgejo/workflows feat: add db_audit health checks, runner, and scheduled Forgejo workflow 2026-04-12 21:40:29 +03:00
administration Add DB connection string to ops manual, add administration notes, remove stale deploy guide 2026-04-10 22:34:56 +03:00
backup feat(dwh): bronze pipeline migrations, runbook, and execution manual 2026-04-25 01:07:53 +03:00
data chore(repo): reorganize tree into migrations/ data/ legacy/ docs/ 2026-06-01 02:27:30 +03:00
db_audit chore(cleanup): purge n8n, Grafana, and DWH references + dead artifacts 2026-06-10 21:41:27 +03:00
docs fix(security,ingest): 260702 audit — secure the stack, correct poller counters 2026-07-02 09:51:02 +03:00
legacy chore(repo): reorganize tree into migrations/ data/ legacy/ docs/ 2026-06-01 02:27:30 +03:00
migrations fix(security,ingest): 260702 audit — secure the stack, correct poller counters 2026-07-02 09:51:02 +03:00
scripts feat(db): dedicated read-only dashboard_ro role + repoint staging 2026-06-10 12:33:53 +03:00
tests fix(security,ingest): 260702 audit — secure the stack, correct poller counters 2026-07-02 09:51:02 +03:00
tools feat(tools): import_drivers_csv reads raw Tracksolid Pro export format 2026-06-25 14:52:11 +03:00
.dockerignore fix(security,ingest): 260702 audit — secure the stack, correct poller counters 2026-07-02 09:51:02 +03:00
.env chore(cleanup): purge n8n, Grafana, and DWH references + dead artifacts 2026-06-10 21:41:27 +03:00
.gitignore docs+tooling: OSM POI export workflow + Shell stations data; graded DQ report 2026-06-08 22:04:49 +03:00
.python-version chore: align .python-version to 3.12.0 (matches Docker image and pyproject.toml) 2026-04-12 21:41:43 +03:00
55_ts_coolify_gemini_prod.code-workspace feat(dwh): bronze pipeline migrations, runbook, and execution manual 2026-04-25 01:07:53 +03:00
audit_device_reconciliation.py feat(analytics): Phase 0 — analytics-config migration and CSV importer rewrite 2026-04-27 23:42:37 +03:00
backfill_trips_enrichment.py feat(trips): add --skip-geocode flag to backfill script 2026-05-01 22:12:07 +03:00
CLAUDE.md fix(reporting): restore live-feed cost-centre exclusion + vehicle_type (migration 20) 2026-06-10 22:36:45 +03:00
dashboard_api_rev.py feat(dashboard_api): CRQ tab routes (crq-dashboard/search/filter-options) 2026-06-26 00:20:10 +03:00
deploy_dashboard_api.sh feat(db): split refresher onto REFRESH_DATABASE_URL; prod reads via dashboard_ro 2026-06-10 20:19:40 +03:00
deploy_dashboard_api_staging.sh chore(cleanup): purge n8n, Grafana, and DWH references + dead artifacts 2026-06-10 21:41:27 +03:00
docker-compose.yaml fix(security,ingest): 260702 audit — secure the stack, correct poller counters 2026-07-02 09:51:02 +03:00
Dockerfile fix(security,ingest): 260702 audit — secure the stack, correct poller counters 2026-07-02 09:51:02 +03:00
import_drivers_csv.py chore(repo): reorganize tree into migrations/ data/ legacy/ docs/ 2026-06-01 02:27:30 +03:00
ingest_events_rev.py fix(security,ingest): 260702 audit — secure the stack, correct poller counters 2026-07-02 09:51:02 +03:00
ingest_movement_rev.py fix(security,ingest): 260702 audit — secure the stack, correct poller counters 2026-07-02 09:51:02 +03:00
ingest_worker_rev.py feat(stack): consolidate 7→4 services (merge pollers, drop pgbouncer/grafana) 2026-06-10 21:41:05 +03:00
pyproject.toml fix(security,ingest): 260702 audit — secure the stack, correct poller counters 2026-07-02 09:51:02 +03:00
README.md first commit 2026-04-07 20:41:16 +03:00
run_migrations.py fix(security,ingest): 260702 audit — secure the stack, correct poller counters 2026-07-02 09:51:02 +03:00
run_migrations.sh chore(repo): reorganize tree into migrations/ data/ legacy/ docs/ 2026-06-01 02:27:30 +03:00
shell_stations.csv docs+tooling: OSM POI export workflow + Shell stations data; graded DQ report 2026-06-08 22:04:49 +03:00
shell_stations.geojson docs+tooling: OSM POI export workflow + Shell stations data; graded DQ report 2026-06-08 22:04:49 +03:00
sync_driver_audit.py perf+fix: SAVEPOINT-per-item pollers, batched GPS inserts, parallel detail fetch 2026-04-18 00:33:55 +03:00
ts_shared_rev.py fix(security,ingest): 260702 audit — secure the stack, correct poller counters 2026-07-02 09:51:02 +03:00
uv.lock fix(security,ingest): 260702 audit — secure the stack, correct poller counters 2026-07-02 09:51:02 +03:00
webhook_receiver_rev.py fix(security,ingest): 260702 audit — secure the stack, correct poller counters 2026-07-02 09:51:02 +03:00