# Data Flow — Ingestion → Aggregation → Views → Functions → Consumers **Scope:** the *live* fleet pipeline only (`tracksolid` + `reporting`). The `ops` and `dwh_gold` schemas were **purged on 2026-06-05** (migrations 12/13) — those workshop / dispatch / SLA / utilisation features were never implemented. They are shown below only as a struck-out "removed" footnote for historical context. > **Update 2026-06-10:** Grafana, pgbouncer, and the separate DWH pipeline were removed. > The `tracksolid.v_*` analytics views still exist but no longer have an active consumer > (Grafana was their only reader). The two pollers were merged into one `ingest_worker`. **Verified against prod 2026-06-05** (TimescaleDB hypertable + continuous-aggregate catalog, `pg_depend` view graph, ingestion `INSERT` targets, `dashboard_api` queries). Key facts that surprised the docs: - Only `position_history` (and the empty/planned `heartbeats`, `fuel_readings`, `temperature_readings`) are **hypertables**. `trips` and `alarms` are **plain tables**. - `tracksolid.v_mileage_daily_cagg` is a **real TimescaleDB continuous aggregate**, not a plain view — and it currently has **no downstream consumer**. - `reporting.v_trips` is a **matview**, refreshed every 5 min by the in-process `dashboard_api` background loop (FIX-D02), pg advisory-lock `920145`. --- ## Mermaid ```mermaid flowchart TD API["Tracksolid / Jimi API
poll + push webhooks · OAuth2"] subgraph ING["Ingestion — ts_shared_rev.py: get_conn() · api_post() · clean*()"] IM["ingest_movement_rev.py"] IE["ingest_events_rev.py"] WR["webhook_receiver_rev.py"] end API --> IM & IE & WR subgraph L1["L1 · Base tables + hypertables — schema: tracksolid (single source of truth)"] LP["live_positions
current fix / IMEI"] PH[("position_history
HYPERTABLE · high-res GPS trail")] TR["trips
plain table — NOT a hypertable"] DV["devices
vehicle / driver registry"] AL["alarms
plain table"] ILOG["ingestion_log · api_token_cache"] EMPTY["heartbeats* · fuel_readings* · temperature_readings* (HYPERTABLES, empty)
device_events · fault_codes · obd_readings · parking_events · lbs_readings · geofences (empty)"] end IM --> LP & PH & TR & DV IM --> ILOG IE --> AL WR --> EMPTY subgraph L2["L2 · Aggregation"] CAGG[("v_mileage_daily_cagg
CONTINUOUS AGGREGATE")] VT[("reporting.v_trips
MATVIEW · ix_v_trips_trip_id")] RLOG["reporting.refresh_log"] end PH -->|Timescale cont-agg policy| CAGG TR --> VT DV --> VT VT -->|"REFRESH CONCURRENTLY every 300s
(dashboard_api loop, adv-lock 920145)"| RLOG subgraph L3L["L3 · Reporting views — fed by v_trips matview"] FILT["v_filter_vehicles · v_filter_drivers
v_filter_cost_centres · v_filter_cities"] SUMM["v_daily/weekly/monthly_summary
v_*_cost_centre · v_trips_today"] VLP["v_live_positions"] end VT --> FILT & SUMM LP --> VLP subgraph L3R["L3 · Analytics views — tracksolid.* (read base tables directly · UNCONSUMED since Grafana removed 2026-06-10)"] GV["v_fleet_today · v_fleet_status · v_active_dispatch_map
v_currently_idle · v_alarms_daily · v_fleet_km_daily
v_ingestion_health · v_vehicles_not_moved_today
v_driver_aggregates_daily · v_fleet_trace · v_driver_clock_*"] end LP --> GV TR --> GV DV --> GV AL --> GV PH --> GV subgraph L4["L4 · Functions — reporting.* (the only API entrypoints)"] FLP["fn_live_positions(cost_centre, acc_status)"] FVT["fn_vehicle_track(vehicle_number, hours)"] FTM["fn_trips_for_map(veh[], driver, cc, city, start, end)"] NP["normalize_plate(p) · helper"] end LP --> FLP DV --> FLP PH --> FVT VT --> FTM NP -.-> FTM subgraph L5["L5 · Consumers"] DAPI["dashboard_api_rev.py
FastAPI :8890 · 2 workers"] SPA["SPAs: liveposition.* · fleetintelligence.*
(rustfs / S3 single-file maps)"] end FLP --> DAPI FVT --> DAPI FTM --> DAPI FILT --> DAPI DAPI -->|"HTTPS · fleetapi.rahamafresh.com"| SPA CAGG -.->|no consumer yet| NONE(["⚠ unconsumed — no panel / API reads it"]) GV -.->|Grafana removed 2026-06-10| NONE subgraph PARK["REMOVED 2026-06-05 — purged via migrations 12 / 13 (never implemented)"] GONE["ops.* (tickets · dispatch_log · service_log · odometer_readings · cost_rates · kpi_targets · vw_service_forecast)
dwh_gold.* (dim_vehicles · fact_daily_fleet_metrics · refresh_daily_metrics)
tracksolid.v_sla_inflight · tracksolid.v_utilisation_daily + their Grafana panels"] end classDef hyper fill:#e1f0ff,stroke:#3b82f6,color:#0b3d91; classDef mat fill:#fff3cd,stroke:#d4a017,color:#664d03; classDef cagg fill:#e7f7e7,stroke:#2e9e2e,color:#1e5e1e; classDef parked fill:#f0f0f0,stroke:#999,stroke-dasharray:5 5,color:#555; classDef warn fill:#fdecea,stroke:#d93025,color:#a52714; class PH hyper; class VT mat; class CAGG cagg; class PARK,GONE parked; class NONE warn; ``` --- ## ASCII ``` ╔══════════════════════════════════════════════╗ ║ TRACKSOLID / JIMI API ║ ║ (poll + push webhooks, OAuth2) ║ ╚════════════════════╤═════════════════════════╝ │ ┌──────────────────────────┬───────────┴───────────┬────────────────────────────┐ ▼ ▼ ▼ ▼ ┌─────────────────┐ ┌────────────────────┐ ┌──────────────────────┐ │ ingest_movement │ │ ingest_events │ │ webhook_receiver │ (all via ts_shared_rev.py: │ _rev.py │ │ _rev.py │ │ _rev.py │ get_conn() pool, api_post(), └────────┬────────┘ └─────────┬──────────┘ └──────────┬───────────┘ clean*() , token cache) │ │ │ │ writes │ writes │ writes (push) ▼ ▼ ▼ ╔════════════════════════════════════════════════════════════════════════════════════════════╗ ║ L1 · BASE TABLES + HYPERTABLES schema: tracksolid (single source of truth) ║ ║ ║ ║ live_positions ── current fix / IMEI (plain) api_token_cache (auth) ║ ║ position_history ◀═══ HYPERTABLE (high-res GPS trail, partitioned by gps_time) ║ ║ trips ── trip summaries (plain table — NOT a hypertable) ║ ║ devices ── vehicle/driver registry (plain) ║ ║ alarms ── alarm events (plain table) ║ ║ ingestion_log ── API audit trail (plain) ║ ║ heartbeats*, fuel_readings*, temperature_readings* ◀═ HYPERTABLES (empty / planned push) ║ ║ device_events, fault_codes, obd_readings, parking_events, lbs_readings, geofences (empty) ║ ╚══════╤══════════════════════════╤══════════════════════════════════╤════════════════════════╝ │ │ │ │ TimescaleDB │ matview refresh │ direct reads │ cont-agg policy │ (in-app loop) │ (ad-hoc SQL) ▼ ▼ ▼ ╔══════════════════╗ ╔════════════════════════════════╗ ╔══════════════════════════════════╗ ║ L2 · AGGREGATION ║ ║ L2 · AGGREGATION (matview) ║ ║ L3 · ANALYTICS VIEWS — UNCONSUMED ║ ║ ║ ║ ║ ║ read base tables directly ║ ║ v_mileage_daily ║ ║ reporting.v_trips [MATVIEW] ║ ║ ║ ║ _cagg ║ ║ ◀── trips + devices ║ ║ v_fleet_today v_fleet_status ║ ║ CONTINUOUS AGG ║ ║ unique ix_v_trips_trip_id ║ ║ v_active_dispatch_map ║ ║ ◀═ position_ ║ ║ ║ ║ v_currently_idle v_alarms_daily ║ ║ history ║ ║ REFRESH … CONCURRENTLY every ║ ║ v_fleet_km_daily v_ingestion_health║ ║ (auto refresh ║ ║ VTRIPS_REFRESH_INTERVAL_S=300 ║ ║ v_vehicles_not_moved_today ║ ║ policy) ║ ║ by dashboard_api bg asyncio ║ ║ v_driver_aggregates_daily ║ ╚════════╤═════════╝ ║ loop, pg advisory-lock 920145 ║ ║ v_fleet_trace ║ │ ║ │ ║ ║ v_driver_clock_daily/_today/_attnd ║ │ ║ └──▶ reporting.refresh_log ║ ╚══════════════╤═════════════════════╝ │ ╚═══════════════╤════════════════╝ │ │ │ │ │ ▼ │ │ ╔════════════════════════════════════════════╗ │ │ ║ L3 · REPORTING VIEWS (← v_trips matview) ║ │ │ ║ v_filter_vehicles v_filter_drivers ║ │ │ ║ v_filter_cost_centres v_filter_cities ║ │ │ ║ v_daily/weekly/monthly_summary ║ │ │ ║ v_*_cost_centre v_trips_today ║ │ │ ║ v_live_positions (view, ← live_positions) ║ │ │ ╚════════════════════════╤═══════════════════╝ │ │ │ │ │ ╔══════════════════════════════════╧═══════════════════╗ │ │ ║ L4 · FUNCTIONS (schema reporting — API entrypoints) ║ │ │ ║ ║ │ │ ║ fn_live_positions(cost_centre, acc_status) ║ │ │ ║ ◀── live_positions + devices ║ │ │ ║ fn_vehicle_track(vehicle_number, hours) ║ │ │ ║ ◀── position_history (hypertable trail) ────────╫─────┘ (also reads L1) │ ║ fn_trips_for_map(veh[],driver,cc,city,start,end) ║ │ ║ ◀── reporting.v_trips (matview) ║ │ ║ normalize_plate(p) ── helper used by the above ║ │ ╚════════════════════════════╤══════════════════════════╝ │ │ ▼ ▼ (cagg currently ╔═══════════════════════════════════════════════╗ unconsumed — ║ L5 · CONSUMERS ║ no panel/API yet) ║ ║ ║ dashboard_api_rev.py (FastAPI :8890, ×2 wkrs) ║ ║ POST /webhook/fleet-dashboard → fn_trips_for_map + v_filter_* ║ GET /webhook/live-positions → fn_live_positions ║ GET /webhook/live-positions/track → fn_vehicle_track ║ │ ║ ║ ▼ HTTPS (fleetapi.rahamafresh.com) ║ ║ SPAs: liveposition.* · fleetintelligence.* ║ ║ (rustfs/S3 single-file maps) ║ ║ ║ ║Grafana removed 2026-06-10 — v_* now unconsumed║ ╚═══════════════════════════════════════════════╝ ┌─ REMOVED 2026-06-05 · purged via migrations 12 / 13 (never implemented) ───────────────┐ │ ops.* : tickets, dispatch_log, service_log, odometer_readings, │ │ cost_rates, kpi_targets, vw_service_forecast │ │ dwh_gold.* : dim_vehicles, fact_daily_fleet_metrics, refresh_daily_metrics() │ │ tracksolid : v_sla_inflight, v_utilisation_daily (+ their Grafana panels removed) │ │ → Schemas dropped from prod. The separate tracksolid_dwh server is unrelated/untouched.│ └────────────────────────────────────────────────────────────────────────────────────────┘ ``` --- ## How to read it - **L1** is where ingestion lands. Only `position_history` (and three empty/planned hypertables) are TimescaleDB hypertables; `trips`/`alarms` are ordinary tables. - **Two aggregation mechanisms** run in parallel: the TimescaleDB **continuous aggregate** `v_mileage_daily_cagg` (refreshed by a Timescale policy off `position_history`), and the **`reporting.v_trips` matview** (refreshed every 5 min by the in-process `dashboard_api` loop — the FIX-D02 self-refresher that replaced the dead n8n job). - **L4 functions are the only thing the API calls.** SPAs never touch tables directly: SPA → `dashboard_api` → `fn_*` → (matview / tables). - **The `tracksolid.v_*` analytics views (L3 right) are now unconsumed.** Grafana was their only reader and was removed 2026-06-10; the views still exist (migration 07) but nothing reads them. Candidates for removal or for wiring into a FleetOps panel. ## Caveats / housekeeping notes 1. **`v_mileage_daily_cagg` has no consumer** — nothing in the API reads it. It is a live continuous aggregate maintaining itself for nobody. Candidate for removal or wiring into a panel. 2. **The `tracksolid.v_*` analytics views are unconsumed** since Grafana's removal (2026-06-10) — same housekeeping candidate as the cagg above. 3. **`ops` and `dwh_gold` were purged** (2026-06-05, migrations 12/13) along with `v_utilisation_daily`, `v_sla_inflight`, and their (then-live) Grafana panels — the features were never implemented. 4. See the companion housekeeping audit (2026-06-05) for the full unused-object list; the only clean ad-hoc drop is `public.trips_viz_v1`.