d5093f0a1c
These subsystems are retired and replaced by better alternatives (FleetNow / FleetOps SPAs via dashboard_api; in-process pooling; reporting.v_ingest_health). Remove them so the repo reflects the live stack only. Nothing running depends on the deleted artifacts. Deleted (dead artifacts): - n8n-workflows/ (retired webhook exports), grafana/ (provisioning for the removed service), dwh/ (migrations for the decommissioned external warehouse) - runbooks: DWH_PIPELINE.md, DWH_Execution_Manual.md, grafanaDeployment.md, grafanaOperationalManual.md Code/config: - run_migrations.py: drop sync_role_passwords() (its only entries were the now -dead grafana_ro + pgbouncer syncs; the guard already made it inert) - .env: remove the two unused GRAFANA_* vars - ingest_movement_rev.py / db_audit / deploy_dashboard_api_staging.sh: reword stale Grafana/grafana_ro comments Docs: scrub n8n/Grafana/DWH from CLAUDE.md, CONNECTIONS, DATA_FLOW, OPERATIONS_MANUAL, docker_commands, KPI_FRAMEWORK, PLATFORM_OVERVIEW, STAGING_FLEETOPS, and deprecation-banner the two large SQL libraries (dwh_gold was already dropped 2026-06-05). Kept deliberately: the grafana_ro DB role (now an unused read-only login), applied migration history, dated docs/reports/*, and docs/superpowers/* specs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
245 lines
17 KiB
Markdown
245 lines
17 KiB
Markdown
# Data Flow — Ingestion → Aggregation → Views → Functions → Consumers
|
||
|
||
**Scope:** the *live* fleet pipeline only (`tracksolid` + `reporting`). The `ops` and
|
||
`dwh_gold` schemas were **purged on 2026-06-05** (migrations 12/13) — those workshop /
|
||
dispatch / SLA / utilisation features were never implemented. They are shown below only as
|
||
a struck-out "removed" footnote for historical context.
|
||
|
||
> **Update 2026-06-10:** Grafana, pgbouncer, and the separate DWH pipeline were removed.
|
||
> The `tracksolid.v_*` analytics views still exist but no longer have an active consumer
|
||
> (Grafana was their only reader). The two pollers were merged into one `ingest_worker`.
|
||
|
||
**Verified against prod 2026-06-05** (TimescaleDB hypertable + continuous-aggregate
|
||
catalog, `pg_depend` view graph, ingestion `INSERT` targets, `dashboard_api` queries).
|
||
Key facts that surprised the docs:
|
||
|
||
- Only `position_history` (and the empty/planned `heartbeats`, `fuel_readings`,
|
||
`temperature_readings`) are **hypertables**. `trips` and `alarms` are **plain tables**.
|
||
- `tracksolid.v_mileage_daily_cagg` is a **real TimescaleDB continuous aggregate**, not a
|
||
plain view — and it currently has **no downstream consumer**.
|
||
- `reporting.v_trips` is a **matview**, refreshed every 5 min by the in-process
|
||
`dashboard_api` background loop (FIX-D02), pg advisory-lock `920145`.
|
||
|
||
---
|
||
|
||
## Mermaid
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
API["Tracksolid / Jimi API<br/>poll + push webhooks · OAuth2"]
|
||
|
||
subgraph ING["Ingestion — ts_shared_rev.py: get_conn() · api_post() · clean*()"]
|
||
IM["ingest_movement_rev.py"]
|
||
IE["ingest_events_rev.py"]
|
||
WR["webhook_receiver_rev.py"]
|
||
end
|
||
API --> IM & IE & WR
|
||
|
||
subgraph L1["L1 · Base tables + hypertables — schema: tracksolid (single source of truth)"]
|
||
LP["live_positions<br/>current fix / IMEI"]
|
||
PH[("position_history<br/>HYPERTABLE · high-res GPS trail")]
|
||
TR["trips<br/>plain table — NOT a hypertable"]
|
||
DV["devices<br/>vehicle / driver registry"]
|
||
AL["alarms<br/>plain table"]
|
||
ILOG["ingestion_log · api_token_cache"]
|
||
EMPTY["heartbeats* · fuel_readings* · temperature_readings* (HYPERTABLES, empty)<br/>device_events · fault_codes · obd_readings · parking_events · lbs_readings · geofences (empty)"]
|
||
end
|
||
IM --> LP & PH & TR & DV
|
||
IM --> ILOG
|
||
IE --> AL
|
||
WR --> EMPTY
|
||
|
||
subgraph L2["L2 · Aggregation"]
|
||
CAGG[("v_mileage_daily_cagg<br/>CONTINUOUS AGGREGATE")]
|
||
VT[("reporting.v_trips<br/>MATVIEW · ix_v_trips_trip_id")]
|
||
RLOG["reporting.refresh_log"]
|
||
end
|
||
PH -->|Timescale cont-agg policy| CAGG
|
||
TR --> VT
|
||
DV --> VT
|
||
VT -->|"REFRESH CONCURRENTLY every 300s<br/>(dashboard_api loop, adv-lock 920145)"| RLOG
|
||
|
||
subgraph L3L["L3 · Reporting views — fed by v_trips matview"]
|
||
FILT["v_filter_vehicles · v_filter_drivers<br/>v_filter_cost_centres · v_filter_cities"]
|
||
SUMM["v_daily/weekly/monthly_summary<br/>v_*_cost_centre · v_trips_today"]
|
||
VLP["v_live_positions"]
|
||
end
|
||
VT --> FILT & SUMM
|
||
LP --> VLP
|
||
|
||
subgraph L3R["L3 · Analytics views — tracksolid.* (read base tables directly · UNCONSUMED since Grafana removed 2026-06-10)"]
|
||
GV["v_fleet_today · v_fleet_status · v_active_dispatch_map<br/>v_currently_idle · v_alarms_daily · v_fleet_km_daily<br/>v_ingestion_health · v_vehicles_not_moved_today<br/>v_driver_aggregates_daily · v_fleet_trace · v_driver_clock_*"]
|
||
end
|
||
LP --> GV
|
||
TR --> GV
|
||
DV --> GV
|
||
AL --> GV
|
||
PH --> GV
|
||
|
||
subgraph L4["L4 · Functions — reporting.* (the only API entrypoints)"]
|
||
FLP["fn_live_positions(cost_centre, acc_status)"]
|
||
FVT["fn_vehicle_track(vehicle_number, hours)"]
|
||
FTM["fn_trips_for_map(veh[], driver, cc, city, start, end)"]
|
||
NP["normalize_plate(p) · helper"]
|
||
end
|
||
LP --> FLP
|
||
DV --> FLP
|
||
PH --> FVT
|
||
VT --> FTM
|
||
NP -.-> FTM
|
||
|
||
subgraph L5["L5 · Consumers"]
|
||
DAPI["dashboard_api_rev.py<br/>FastAPI :8890 · 2 workers"]
|
||
SPA["SPAs: liveposition.* · fleetintelligence.*<br/>(rustfs / S3 single-file maps)"]
|
||
end
|
||
FLP --> DAPI
|
||
FVT --> DAPI
|
||
FTM --> DAPI
|
||
FILT --> DAPI
|
||
DAPI -->|"HTTPS · fleetapi.rahamafresh.com"| SPA
|
||
|
||
CAGG -.->|no consumer yet| NONE(["⚠ unconsumed — no panel / API reads it"])
|
||
GV -.->|Grafana removed 2026-06-10| NONE
|
||
|
||
subgraph PARK["REMOVED 2026-06-05 — purged via migrations 12 / 13 (never implemented)"]
|
||
GONE["ops.* (tickets · dispatch_log · service_log · odometer_readings · cost_rates · kpi_targets · vw_service_forecast)<br/>dwh_gold.* (dim_vehicles · fact_daily_fleet_metrics · refresh_daily_metrics)<br/>tracksolid.v_sla_inflight · tracksolid.v_utilisation_daily + their Grafana panels"]
|
||
end
|
||
|
||
classDef hyper fill:#e1f0ff,stroke:#3b82f6,color:#0b3d91;
|
||
classDef mat fill:#fff3cd,stroke:#d4a017,color:#664d03;
|
||
classDef cagg fill:#e7f7e7,stroke:#2e9e2e,color:#1e5e1e;
|
||
classDef parked fill:#f0f0f0,stroke:#999,stroke-dasharray:5 5,color:#555;
|
||
classDef warn fill:#fdecea,stroke:#d93025,color:#a52714;
|
||
class PH hyper;
|
||
class VT mat;
|
||
class CAGG cagg;
|
||
class PARK,GONE parked;
|
||
class NONE warn;
|
||
```
|
||
|
||
---
|
||
|
||
## ASCII
|
||
|
||
```
|
||
╔══════════════════════════════════════════════╗
|
||
║ TRACKSOLID / JIMI API ║
|
||
║ (poll + push webhooks, OAuth2) ║
|
||
╚════════════════════╤═════════════════════════╝
|
||
│
|
||
┌──────────────────────────┬───────────┴───────────┬────────────────────────────┐
|
||
▼ ▼ ▼ ▼
|
||
┌─────────────────┐ ┌────────────────────┐ ┌──────────────────────┐
|
||
│ ingest_movement │ │ ingest_events │ │ webhook_receiver │ (all via ts_shared_rev.py:
|
||
│ _rev.py │ │ _rev.py │ │ _rev.py │ get_conn() pool, api_post(),
|
||
└────────┬────────┘ └─────────┬──────────┘ └──────────┬───────────┘ clean*() , token cache)
|
||
│ │ │
|
||
│ writes │ writes │ writes (push)
|
||
▼ ▼ ▼
|
||
╔════════════════════════════════════════════════════════════════════════════════════════════╗
|
||
║ L1 · BASE TABLES + HYPERTABLES schema: tracksolid (single source of truth) ║
|
||
║ ║
|
||
║ live_positions ── current fix / IMEI (plain) api_token_cache (auth) ║
|
||
║ position_history ◀═══ HYPERTABLE (high-res GPS trail, partitioned by gps_time) ║
|
||
║ trips ── trip summaries (plain table — NOT a hypertable) ║
|
||
║ devices ── vehicle/driver registry (plain) ║
|
||
║ alarms ── alarm events (plain table) ║
|
||
║ ingestion_log ── API audit trail (plain) ║
|
||
║ heartbeats*, fuel_readings*, temperature_readings* ◀═ HYPERTABLES (empty / planned push) ║
|
||
║ device_events, fault_codes, obd_readings, parking_events, lbs_readings, geofences (empty) ║
|
||
╚══════╤══════════════════════════╤══════════════════════════════════╤════════════════════════╝
|
||
│ │ │
|
||
│ TimescaleDB │ matview refresh │ direct reads
|
||
│ cont-agg policy │ (in-app loop) │ (ad-hoc SQL)
|
||
▼ ▼ ▼
|
||
╔══════════════════╗ ╔════════════════════════════════╗ ╔══════════════════════════════════╗
|
||
║ L2 · AGGREGATION ║ ║ L2 · AGGREGATION (matview) ║ ║ L3 · ANALYTICS VIEWS — UNCONSUMED ║
|
||
║ ║ ║ ║ ║ read base tables directly ║
|
||
║ v_mileage_daily ║ ║ reporting.v_trips [MATVIEW] ║ ║ ║
|
||
║ _cagg ║ ║ ◀── trips + devices ║ ║ v_fleet_today v_fleet_status ║
|
||
║ CONTINUOUS AGG ║ ║ unique ix_v_trips_trip_id ║ ║ v_active_dispatch_map ║
|
||
║ ◀═ position_ ║ ║ ║ ║ v_currently_idle v_alarms_daily ║
|
||
║ history ║ ║ REFRESH … CONCURRENTLY every ║ ║ v_fleet_km_daily v_ingestion_health║
|
||
║ (auto refresh ║ ║ VTRIPS_REFRESH_INTERVAL_S=300 ║ ║ v_vehicles_not_moved_today ║
|
||
║ policy) ║ ║ by dashboard_api bg asyncio ║ ║ v_driver_aggregates_daily ║
|
||
╚════════╤═════════╝ ║ loop, pg advisory-lock 920145 ║ ║ v_fleet_trace ║
|
||
│ ║ │ ║ ║ v_driver_clock_daily/_today/_attnd ║
|
||
│ ║ └──▶ reporting.refresh_log ║ ╚══════════════╤═════════════════════╝
|
||
│ ╚═══════════════╤════════════════╝ │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ ╔════════════════════════════════════════════╗ │
|
||
│ ║ L3 · REPORTING VIEWS (← v_trips matview) ║ │
|
||
│ ║ v_filter_vehicles v_filter_drivers ║ │
|
||
│ ║ v_filter_cost_centres v_filter_cities ║ │
|
||
│ ║ v_daily/weekly/monthly_summary ║ │
|
||
│ ║ v_*_cost_centre v_trips_today ║ │
|
||
│ ║ v_live_positions (view, ← live_positions) ║ │
|
||
│ ╚════════════════════════╤═══════════════════╝ │
|
||
│ │ │
|
||
│ ╔══════════════════════════════════╧═══════════════════╗ │
|
||
│ ║ L4 · FUNCTIONS (schema reporting — API entrypoints) ║ │
|
||
│ ║ ║ │
|
||
│ ║ fn_live_positions(cost_centre, acc_status) ║ │
|
||
│ ║ ◀── live_positions + devices ║ │
|
||
│ ║ fn_vehicle_track(vehicle_number, hours) ║ │
|
||
│ ║ ◀── position_history (hypertable trail) ────────╫─────┘ (also reads L1)
|
||
│ ║ fn_trips_for_map(veh[],driver,cc,city,start,end) ║
|
||
│ ║ ◀── reporting.v_trips (matview) ║
|
||
│ ║ normalize_plate(p) ── helper used by the above ║
|
||
│ ╚════════════════════════════╤══════════════════════════╝
|
||
│ │
|
||
▼ ▼
|
||
(cagg currently ╔═══════════════════════════════════════════════╗
|
||
unconsumed — ║ L5 · CONSUMERS ║
|
||
no panel/API yet) ║ ║
|
||
║ dashboard_api_rev.py (FastAPI :8890, ×2 wkrs) ║
|
||
║ POST /webhook/fleet-dashboard → fn_trips_for_map + v_filter_*
|
||
║ GET /webhook/live-positions → fn_live_positions
|
||
║ GET /webhook/live-positions/track → fn_vehicle_track
|
||
║ │ ║
|
||
║ ▼ HTTPS (fleetapi.rahamafresh.com) ║
|
||
║ SPAs: liveposition.* · fleetintelligence.* ║
|
||
║ (rustfs/S3 single-file maps) ║
|
||
║ ║
|
||
║Grafana removed 2026-06-10 — v_* now unconsumed║
|
||
╚═══════════════════════════════════════════════╝
|
||
|
||
|
||
┌─ REMOVED 2026-06-05 · purged via migrations 12 / 13 (never implemented) ───────────────┐
|
||
│ ops.* : tickets, dispatch_log, service_log, odometer_readings, │
|
||
│ cost_rates, kpi_targets, vw_service_forecast │
|
||
│ dwh_gold.* : dim_vehicles, fact_daily_fleet_metrics, refresh_daily_metrics() │
|
||
│ tracksolid : v_sla_inflight, v_utilisation_daily (+ their Grafana panels removed) │
|
||
│ → Schemas dropped from prod. The separate tracksolid_dwh server is unrelated/untouched.│
|
||
└────────────────────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## How to read it
|
||
|
||
- **L1** is where ingestion lands. Only `position_history` (and three empty/planned
|
||
hypertables) are TimescaleDB hypertables; `trips`/`alarms` are ordinary tables.
|
||
- **Two aggregation mechanisms** run in parallel: the TimescaleDB **continuous aggregate**
|
||
`v_mileage_daily_cagg` (refreshed by a Timescale policy off `position_history`), and the
|
||
**`reporting.v_trips` matview** (refreshed every 5 min by the in-process `dashboard_api`
|
||
loop — the FIX-D02 self-refresher that replaced the dead n8n job).
|
||
- **L4 functions are the only thing the API calls.** SPAs never touch tables directly:
|
||
SPA → `dashboard_api` → `fn_*` → (matview / tables).
|
||
- **The `tracksolid.v_*` analytics views (L3 right) are now unconsumed.** Grafana was their
|
||
only reader and was removed 2026-06-10; the views still exist (migration 07) but nothing
|
||
reads them. Candidates for removal or for wiring into a FleetOps panel.
|
||
|
||
## Caveats / housekeeping notes
|
||
|
||
1. **`v_mileage_daily_cagg` has no consumer** — nothing in the API reads it. It
|
||
is a live continuous aggregate maintaining itself for nobody. Candidate for removal or
|
||
wiring into a panel.
|
||
2. **The `tracksolid.v_*` analytics views are unconsumed** since Grafana's removal
|
||
(2026-06-10) — same housekeeping candidate as the cagg above.
|
||
3. **`ops` and `dwh_gold` were purged** (2026-06-05, migrations 12/13) along with
|
||
`v_utilisation_daily`, `v_sla_inflight`, and their (then-live) Grafana panels — the
|
||
features were never implemented.
|
||
4. See the companion housekeeping audit (2026-06-05) for the full unused-object list; the
|
||
only clean ad-hoc drop is `public.trips_viz_v1`.
|