tracksolid_timescale_grafana_prod/docs/DATA_FLOW.md

# Data Flow — Ingestion → Aggregation → Views → Functions → Consumers

**Scope:** the *live* fleet pipeline only (`tracksolid` + `reporting`). The `ops` and
`dwh_gold` schemas were **purged on 2026-06-05** (migrations 12/13) — those workshop /
dispatch / SLA / utilisation features were never implemented. They are shown below only as
a struck-out "removed" footnote for historical context.

> **Update 2026-06-10:** Grafana, pgbouncer, and the separate DWH pipeline were removed.
> The `tracksolid.v_*` analytics views still exist but no longer have an active consumer
> (Grafana was their only reader). The two pollers were merged into one `ingest_worker`.

**Verified against prod 2026-06-05** (TimescaleDB hypertable + continuous-aggregate
catalog, `pg_depend` view graph, ingestion `INSERT` targets, `dashboard_api` queries).
Key facts that surprised the docs:

- Only `position_history` (and the empty/planned `heartbeats`, `fuel_readings`,
  `temperature_readings`) are **hypertables**. `trips` and `alarms` are **plain tables**.
- `tracksolid.v_mileage_daily_cagg` is a **real TimescaleDB continuous aggregate**, not a
  plain view — and it currently has **no downstream consumer**.
- `reporting.v_trips` is a **matview**, refreshed every 5 min by the in-process
  `dashboard_api` background loop (FIX-D02), pg advisory-lock `920145`.

---

## Mermaid

```mermaid
flowchart TD
    API["Tracksolid / Jimi API<br/>poll + push webhooks · OAuth2"]

    subgraph ING["Ingestion — ts_shared_rev.py: get_conn() · api_post() · clean*()"]
        IM["ingest_movement_rev.py"]
        IE["ingest_events_rev.py"]
        WR["webhook_receiver_rev.py"]
    end
    API --> IM & IE & WR

    subgraph L1["L1 · Base tables + hypertables — schema: tracksolid (single source of truth)"]
        LP["live_positions<br/>current fix / IMEI"]
        PH[("position_history<br/>HYPERTABLE · high-res GPS trail")]
        TR["trips<br/>plain table — NOT a hypertable"]
        DV["devices<br/>vehicle / driver registry"]
        AL["alarms<br/>plain table"]
        ILOG["ingestion_log · api_token_cache"]
        EMPTY["heartbeats* · fuel_readings* · temperature_readings*  (HYPERTABLES, empty)<br/>device_events · fault_codes · obd_readings · parking_events · lbs_readings · geofences  (empty)"]
    end
    IM --> LP & PH & TR & DV
    IM --> ILOG
    IE --> AL
    WR --> EMPTY

    subgraph L2["L2 · Aggregation"]
        CAGG[("v_mileage_daily_cagg<br/>CONTINUOUS AGGREGATE")]
        VT[("reporting.v_trips<br/>MATVIEW · ix_v_trips_trip_id")]
        RLOG["reporting.refresh_log"]
    end
    PH -->|Timescale cont-agg policy| CAGG
    TR --> VT
    DV --> VT
    VT -->|"REFRESH CONCURRENTLY every 300s<br/>(dashboard_api loop, adv-lock 920145)"| RLOG

    subgraph L3L["L3 · Reporting views — fed by v_trips matview"]
        FILT["v_filter_vehicles · v_filter_drivers<br/>v_filter_cost_centres · v_filter_cities"]
        SUMM["v_daily/weekly/monthly_summary<br/>v_*_cost_centre · v_trips_today"]
        VLP["v_live_positions"]
    end
    VT --> FILT & SUMM
    LP --> VLP

    subgraph L3R["L3 · Analytics views — tracksolid.* (read base tables directly · UNCONSUMED since Grafana removed 2026-06-10)"]
        GV["v_fleet_today · v_fleet_status · v_active_dispatch_map<br/>v_currently_idle · v_alarms_daily · v_fleet_km_daily<br/>v_ingestion_health · v_vehicles_not_moved_today<br/>v_driver_aggregates_daily · v_fleet_trace · v_driver_clock_*"]
    end
    LP --> GV
    TR --> GV
    DV --> GV
    AL --> GV
    PH --> GV

    subgraph L4["L4 · Functions — reporting.* (the only API entrypoints)"]
        FLP["fn_live_positions(cost_centre, acc_status)"]
        FVT["fn_vehicle_track(vehicle_number, hours)"]
        FTM["fn_trips_for_map(veh[], driver, cc, city, start, end)"]
        NP["normalize_plate(p) · helper"]
    end
    LP --> FLP
    DV --> FLP
    PH --> FVT
    VT --> FTM
    NP -.-> FTM

    subgraph L5["L5 · Consumers"]
        DAPI["dashboard_api_rev.py<br/>FastAPI :8890 · 2 workers"]
        SPA["SPAs: liveposition.* · fleetintelligence.*<br/>(rustfs / S3 single-file maps)"]
    end
    FLP --> DAPI
    FVT --> DAPI
    FTM --> DAPI
    FILT --> DAPI
    DAPI -->|"HTTPS · fleetapi.rahamafresh.com"| SPA

    CAGG -.->|no consumer yet| NONE(["⚠ unconsumed — no panel / API reads it"])
    GV  -.->|Grafana removed 2026-06-10| NONE

    subgraph PARK["REMOVED 2026-06-05 — purged via migrations 12 / 13 (never implemented)"]
        GONE["ops.*  (tickets · dispatch_log · service_log · odometer_readings · cost_rates · kpi_targets · vw_service_forecast)<br/>dwh_gold.*  (dim_vehicles · fact_daily_fleet_metrics · refresh_daily_metrics)<br/>tracksolid.v_sla_inflight · tracksolid.v_utilisation_daily  + their Grafana panels"]
    end

    classDef hyper fill:#e1f0ff,stroke:#3b82f6,color:#0b3d91;
    classDef mat fill:#fff3cd,stroke:#d4a017,color:#664d03;
    classDef cagg fill:#e7f7e7,stroke:#2e9e2e,color:#1e5e1e;
    classDef parked fill:#f0f0f0,stroke:#999,stroke-dasharray:5 5,color:#555;
    classDef warn fill:#fdecea,stroke:#d93025,color:#a52714;
    class PH hyper;
    class VT mat;
    class CAGG cagg;
    class PARK,GONE parked;
    class NONE warn;
```

---

## ASCII

```
                          ╔══════════════════════════════════════════════╗
                          ║              TRACKSOLID / JIMI API            ║
                          ║         (poll + push webhooks, OAuth2)        ║
                          ╚════════════════════╤═════════════════════════╝
                                               │
        ┌──────────────────────────┬───────────┴───────────┬────────────────────────────┐
        ▼                          ▼                        ▼                            ▼
┌─────────────────┐   ┌────────────────────┐   ┌──────────────────────┐
│ ingest_movement │   │   ingest_events    │   │  webhook_receiver    │   (all via ts_shared_rev.py:
│   _rev.py       │   │     _rev.py        │   │     _rev.py          │    get_conn() pool, api_post(),
└────────┬────────┘   └─────────┬──────────┘   └──────────┬───────────┘    clean*() , token cache)
         │                      │                         │
         │ writes               │ writes                  │ writes (push)
         ▼                      ▼                         ▼
╔════════════════════════════════════════════════════════════════════════════════════════════╗
║  L1 · BASE TABLES + HYPERTABLES        schema: tracksolid  (single source of truth)          ║
║                                                                                              ║
║   live_positions      ── current fix / IMEI  (plain)         api_token_cache  (auth)         ║
║   position_history ◀═══ HYPERTABLE  (high-res GPS trail, partitioned by gps_time)            ║
║   trips               ── trip summaries        (plain table — NOT a hypertable)              ║
║   devices             ── vehicle/driver registry (plain)                                     ║
║   alarms              ── alarm events          (plain table)                                 ║
║   ingestion_log       ── API audit trail       (plain)                                       ║
║   heartbeats*, fuel_readings*, temperature_readings*  ◀═ HYPERTABLES (empty / planned push)  ║
║   device_events, fault_codes, obd_readings, parking_events, lbs_readings, geofences (empty)  ║
╚══════╤══════════════════════════╤══════════════════════════════════╤════════════════════════╝
       │                          │                                  │
       │ TimescaleDB              │ matview refresh                  │ direct reads
       │ cont-agg policy          │ (in-app loop)                    │ (ad-hoc SQL)
       ▼                          ▼                                  ▼
╔══════════════════╗   ╔════════════════════════════════╗   ╔══════════════════════════════════╗
║ L2 · AGGREGATION ║   ║ L2 · AGGREGATION (matview)     ║   ║ L3 · ANALYTICS VIEWS — UNCONSUMED ║
║                  ║   ║                                ║   ║   read base tables directly        ║
║ v_mileage_daily  ║   ║ reporting.v_trips  [MATVIEW]   ║   ║                                    ║
║   _cagg          ║   ║   ◀── trips + devices          ║   ║ v_fleet_today    v_fleet_status    ║
║  CONTINUOUS AGG  ║   ║   unique ix_v_trips_trip_id    ║   ║ v_active_dispatch_map              ║
║  ◀═ position_    ║   ║                                ║   ║ v_currently_idle v_alarms_daily    ║
║     history      ║   ║  REFRESH … CONCURRENTLY every  ║   ║ v_fleet_km_daily v_ingestion_health║
║  (auto refresh   ║   ║  VTRIPS_REFRESH_INTERVAL_S=300 ║   ║ v_vehicles_not_moved_today         ║
║   policy)        ║   ║  by dashboard_api bg asyncio   ║   ║ v_driver_aggregates_daily          ║
╚════════╤═════════╝   ║  loop, pg advisory-lock 920145 ║   ║ v_fleet_trace                      ║
         │             ║      │                         ║   ║ v_driver_clock_daily/_today/_attnd ║
         │             ║      └──▶ reporting.refresh_log ║   ╚══════════════╤═════════════════════╝
         │             ╚═══════════════╤════════════════╝                  │
         │                             │                                   │
         │                             ▼                                   │
         │             ╔════════════════════════════════════════════╗      │
         │             ║ L3 · REPORTING VIEWS  (← v_trips matview)   ║      │
         │             ║  v_filter_vehicles  v_filter_drivers        ║      │
         │             ║  v_filter_cost_centres  v_filter_cities     ║      │
         │             ║  v_daily/weekly/monthly_summary             ║      │
         │             ║  v_*_cost_centre   v_trips_today            ║      │
         │             ║  v_live_positions  (view, ← live_positions) ║      │
         │             ╚════════════════════════╤═══════════════════╝      │
         │                                      │                          │
         │   ╔══════════════════════════════════╧═══════════════════╗      │
         │   ║ L4 · FUNCTIONS  (schema reporting — API entrypoints)  ║      │
         │   ║                                                        ║     │
         │   ║  fn_live_positions(cost_centre, acc_status)            ║     │
         │   ║       ◀── live_positions + devices                     ║     │
         │   ║  fn_vehicle_track(vehicle_number, hours)               ║     │
         │   ║       ◀── position_history  (hypertable trail) ────────╫─────┘ (also reads L1)
         │   ║  fn_trips_for_map(veh[],driver,cc,city,start,end)      ║
         │   ║       ◀── reporting.v_trips  (matview)                 ║
         │   ║  normalize_plate(p)  ── helper used by the above       ║
         │   ╚════════════════════════════╤══════════════════════════╝
         │                                │
         ▼                                ▼
  (cagg currently         ╔═══════════════════════════════════════════════╗
   unconsumed —           ║ L5 · CONSUMERS                                ║
   no panel/API yet)      ║                                               ║
                          ║  dashboard_api_rev.py (FastAPI :8890, ×2 wkrs) ║
                          ║    POST /webhook/fleet-dashboard → fn_trips_for_map + v_filter_*
                          ║    GET  /webhook/live-positions       → fn_live_positions
                          ║    GET  /webhook/live-positions/track → fn_vehicle_track
                          ║         │                                      ║
                          ║         ▼  HTTPS (fleetapi.rahamafresh.com)    ║
                          ║  SPAs:  liveposition.* · fleetintelligence.*   ║
                          ║         (rustfs/S3 single-file maps)           ║
                          ║                                               ║
                          ║Grafana removed 2026-06-10 — v_* now unconsumed║
                          ╚═══════════════════════════════════════════════╝


  ┌─ REMOVED 2026-06-05 · purged via migrations 12 / 13 (never implemented) ───────────────┐
  │  ops.*        : tickets, dispatch_log, service_log, odometer_readings,                  │
  │                 cost_rates, kpi_targets, vw_service_forecast                            │
  │  dwh_gold.*   : dim_vehicles, fact_daily_fleet_metrics, refresh_daily_metrics()         │
  │  tracksolid   : v_sla_inflight, v_utilisation_daily  (+ their Grafana panels removed)   │
  │  → Schemas dropped from prod. The separate tracksolid_dwh server is unrelated/untouched.│
  └────────────────────────────────────────────────────────────────────────────────────────┘
```

---

## How to read it

- **L1** is where ingestion lands. Only `position_history` (and three empty/planned
  hypertables) are TimescaleDB hypertables; `trips`/`alarms` are ordinary tables.
- **Two aggregation mechanisms** run in parallel: the TimescaleDB **continuous aggregate**
  `v_mileage_daily_cagg` (refreshed by a Timescale policy off `position_history`), and the
  **`reporting.v_trips` matview** (refreshed every 5 min by the in-process `dashboard_api`
  loop — the FIX-D02 self-refresher that replaced the dead n8n job).
- **L4 functions are the only thing the API calls.** SPAs never touch tables directly:
  SPA → `dashboard_api` → `fn_*` → (matview / tables).
- **The `tracksolid.v_*` analytics views (L3 right) are now unconsumed.** Grafana was their
  only reader and was removed 2026-06-10; the views still exist (migration 07) but nothing
  reads them. Candidates for removal or for wiring into a FleetOps panel.

## Caveats / housekeeping notes

1. **`v_mileage_daily_cagg` has no consumer** — nothing in the API reads it. It
   is a live continuous aggregate maintaining itself for nobody. Candidate for removal or
   wiring into a panel.
2. **The `tracksolid.v_*` analytics views are unconsumed** since Grafana's removal
   (2026-06-10) — same housekeeping candidate as the cagg above.
3. **`ops` and `dwh_gold` were purged** (2026-06-05, migrations 12/13) along with
   `v_utilisation_daily`, `v_sla_inflight`, and their (then-live) Grafana panels — the
   features were never implemented.
4. See the companion housekeeping audit (2026-06-05) for the full unused-object list; the
   only clean ad-hoc drop is `public.trips_viz_v1`.