Map read-API, reporting schema, fleet segmentation + FleetNow data plumbing #16
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/dashboard-read-api"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Brings the dashboard read-API line of work onto
main(30 commits). Highlights:This session (migrations 14-16):
reporting.fn_fleet_segment+reporting.v_vehicles— field-service vs specialist segmentation, deduped vehicle roster.reporting.map_excluded_cost_centres+ filteredv_live_positions— hide personal/management/MTN(Uganda) from the live map.vehicle_type+fleet_segmenton thefn_live_positionsfeed (powers FleetNow specialist icons).docs/.Earlier on the branch: dedicated FastAPI read-API (replaces n8n webhooks),
reporting.*schema as migration 11, ops/dwh_gold purge (12/13), trips enrichment (FIX-M20), pgbouncer, DWH bronze pipeline, FIX-D01/D02/D03.All three migrations are already applied + verified on prod and recorded in
tracksolid.schema_migrations.Known conflict to resolve:
grafana/provisioning/dashboards-json/daily_operations_dashboard.json(diverged between this branch and main's grafana fixes).Generated with Claude Code.
DWH pipeline (new): - dwh/261001_dwh_control.sql — watermarks + per-run audit log schema - dwh/261002_bronze_constraints_audit.sql — ON CONFLICT key assertion - dwh/261003_dwh_roles.sql — dwh_owner / grafana_ro contract assertion - dwh/261004_dwh_observability_views.sql — v_table_freshness, v_recent_failures, v_watermark_lag (readable by grafana_ro) - docs/DWH_PIPELINE.md — operations runbook (setup, troubleshooting, manual re-run, back-fill, rotation) - DWH_Execution_Manual.md — reusable playbook for future data projects (extract → blob → load pattern, 7 design principles, snapshot-vs-incremental matrix, verification gates) - docs/superpowers/{specs,plans}/2026-04-24-n8n-dwh-bronze-pipeline-* — design spec + 27-task implementation plan Security: - dwh/260423_dwh_ddl_v1.sql — redacted plaintext role passwords to 'CHANGE_ME_BEFORE_APPLY' placeholders; added SECURITY header documenting generation + rotation flow Docs: - CLAUDE.md — §3 adds tracksolid_dwh@31.97.44.246:5888 target, §4 adds dwh/ + docs/DWH_PIPELINE.md to codebase map, §5 adds bronze + dwh_control schema roll-up, §10 adds deploy task + password rotation follow-up Also includes miscellaneous in-progress files accumulated on this branch (workspace, analytics notes, vehicle CSVs, extract helpers, renamed markdown archives). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>Polling jimi.device.track.mileage does not return start/end coordinates, fuel, idle, or trip sequence — leaving most trip columns NULL. This change closes those gaps using data we already have in position_history plus a best-effort Nominatim lookup. Migration 09_trips_enrichment.sql adds: • route_geom (LineString), start_address, end_address, vehicle_plate, waypoints_count on tracksolid.trips • GIST indexes on the three geometry columns • view tracksolid.v_trips_enriched exposing daily_seq + trip_date_eat (replaces reliance on the device-supplied trip_seq, which is only populated when /pushtripreport fires) ingest_movement_rev.py::poll_trips now: • extracts idleSecond from the poll response (was previously dropped) • per-trip: SELECTs start fix, end fix, ST_MakeLine route, and waypoint count from position_history within (start_time, end_time) • reverse-geocodes start/end via the new ts_shared_rev.reverse_geocode helper (Nominatim, LRU-cached at ~11m precision, 1 req/sec, never raises) • caches vehicle_plate from a per-cycle plates dict • ON CONFLICT preserves webhook-supplied data when /pushtripreport later delivers native coords/fuel/trip_seq backfill_trips_enrichment.py is a one-shot script (dry-run by default, --apply to commit, --imei / --since flags) that runs the same enrichment against historical NULL rows and COALESCEs only — never overwrites. DWH bronze mirrors and Grafana panels intentionally not touched (frozen on this branch until the schema work lands). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>Group root-level files (accreted from incremental changes) by purpose without moving any deployment entrypoint or breaking imports: - migrations/ : numbered SQL 02-10 - data/ : source CSVs - legacy/ : superseded pre-_rev scripts + old pipeline notes (not deployed) - docs/{manuals,reference,reports}/ : loose manuals, references, reports - strip stray ** / *** prefixes from 5 doc filenames - delete empty documents.txt / push_webhook.md Reference updates so nothing breaks: - run_migrations.py -> /app/migrations/<file> - run_migrations.sh -> $SCRIPT_DIR/migrations - import_drivers_csv.py -> data/<csv> - docker-compose.yaml -> runbook path comment - CLAUDE.md -> codebase map + inline doc references Deployed Python (3 services + ts_shared_rev + run_migrations) and the documented ops one-shots stay at root, preserving the flat-import layout and all documented commands. Verified: py_compile clean across all modules, every MIGRATIONS entry resolves under migrations/, CI-referenced paths (tests/, mypy targets, db_audit) and the grafana build context intact. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>n8n was a thin HTTP->SQL proxy for the Live Position and Fleet Trips maps and proved fragile (credential reloads, :latest drift, shared connection limits). This service calls the same proven reporting.* functions directly, reusing the existing psycopg2 pool / Docker image / Coolify deploy. Endpoints mirror the n8n webhook paths so the only frontend change is N8N_BASE: GET /webhook/live-positions -> {summary, geojson} (fn_live_positions) GET /webhook/vehicle-track -> GeoJSON Feature (fn_vehicle_track) GET /webhook/fleet-dashboard -> filter options POST /webhook/fleet-dashboard -> trips payload (fn_trips_for_map) Response shapes replicate the n8n "Build response JSON" nodes exactly; empty filters/sentinels ('', null, undefined) normalize to SQL wildcards. CORS limited to the dashboard origins. Added dashboard_api service to docker-compose (port 8890, Coolify-routed). SQL contracts validated against prod. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>The Fleet Trips SPA posts application/x-www-form-urlencoded, but the POST /webhook/fleet-dashboard handler read the body with request.json(). That threw on every request, the except swallowed it to body={}, and all filters (vehicle_numbers, cost_centre, assigned_city) plus period/dates were dropped — so every query returned the full unfiltered fleet (1,266 trips) regardless of the dropdowns. The map/KPIs/trips never changed, which read as "the dropdowns don't work." Parse by Content-Type: urllib.parse.parse_qs for form bodies (no new dependency — avoids python-multipart), JSON still accepted defensively for n8n-compat callers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>Pull request closed