9943932200
Self-contained ingestion module (mirrors fleettickets) for the WhatsApp fuel-record feed in the rustfs `fuel` bucket: - import_fuel.py — snapshot/changes/file modes, raw-jsonb upsert on id - migrations/01_fuel_schema.sql — fuel schema, plate/fuel-type/department normalizers, trigger-derived columns, reporting.v_fuel_fills + v_fuel_efficiency, grafana_ro grants - s3util.py / shared.py / run_migrations.py — rustfs client + DB helpers - docs/plan.html — implementation plan Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
75 lines
3.2 KiB
Markdown
75 lines
3.2 KiB
Markdown
# fleetfuel
|
|
|
|
Fleet **fuel-spend** ingestion and read-schema that powers the **Fuel Log** tab in
|
|
FleetOps. Sibling of **fleettickets** (same self-contained module shape).
|
|
|
|
The feed is WhatsApp fuel-update messages, extracted by an n8n CDC job from the client's
|
|
`logistics_department.fuel_records` table and dropped in the rustfs `fuel` bucket. Each
|
|
record is an *actual* fill: litres, KES amount, odometer, fuel type, driver, department,
|
|
keyed by number plate (`car`).
|
|
|
|
## What this owns
|
|
|
|
| Piece | What |
|
|
|---|---|
|
|
| `migrations/01_fuel_schema.sql` | The `fuel` schema: `fuel.records` (raw-jsonb-first + trigger-derived columns), `fuel.norm_plate` / `fuel.canon_fuel_type` / `fuel.canon_department` normalizers, `fuel.ingest_state` (CDC watermark), and `reporting.v_fuel_fills` / `reporting.v_fuel_efficiency` (the read views) |
|
|
| `import_fuel.py` | Pulls fuel records from the rustfs `fuel` bucket and upserts them on `id` |
|
|
| `s3util.py` | rustfs (S3-compatible) client factory — path-style, custom endpoint |
|
|
| `run_migrations.py` | Applies `migrations/*.sql` in order (ledger: `fuel.schema_migrations`) |
|
|
| `shared.py` | Minimal DB/logging helpers (self-contained — no tracksolid dependency) |
|
|
|
|
## What this does NOT own (stays where it is)
|
|
|
|
- **The DB** — the `fuel` schema lives in the shared `tracksolid_db`.
|
|
- **The read-API** — `dashboard_api` (in the tracksolid stack) serves
|
|
`GET /analytics/fuel-fills`, which reads `reporting.v_fuel_fills` (defined here).
|
|
- **The frontend** — the Fuel Log map/panel is a tab in the **FleetOps** SPA (`fleetops` repo).
|
|
|
|
## Data model (raw-first)
|
|
|
|
Each row is `id` (the source PK) + `raw` (the full source record as `jsonb`) + derived,
|
|
*normalized* columns the DB trigger fills from `raw`. The WhatsApp feed is messy
|
|
(`KCA 542Q` vs `KCA542Q`, fuel-type typos like `DISIEL`, ~30 department spellings), so the
|
|
normalizers (`fuel.norm_plate`, `fuel.canon_fuel_type`, `fuel.canon_department`) are the single
|
|
source of truth. Soft-deletes (`deleted_at`) are kept on the row and excluded by the read views.
|
|
|
|
The fuel record links to the fleet by plate: `reporting.v_fuel_fills` joins
|
|
`fuel.norm_plate(car)` to `fuel.norm_plate(devices.vehicle_number)` to pick up `cost_centre` /
|
|
`assigned_city` / `imei`.
|
|
|
|
## Bucket layout
|
|
|
|
```
|
|
fuel_records/latest.json full snapshot { metadata, records[] }
|
|
fuel_records/changes/<ISO-ts>.json hourly CDC delta (incl. soft-deletes)
|
|
```
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
uv sync
|
|
cp .env.example .env # fill in DATABASE_URL, RUSTFS_*
|
|
python run_migrations.py # apply the schema (idempotent)
|
|
```
|
|
|
|
## Run
|
|
|
|
```bash
|
|
# full reconcile from the snapshot (self-healing; the default cron job)
|
|
python import_fuel.py --snapshot --apply
|
|
|
|
# incremental, since the stored watermark (lower latency)
|
|
python import_fuel.py --changes --apply
|
|
|
|
# from a local file instead of the bucket
|
|
python import_fuel.py --file latest.json --apply
|
|
```
|
|
|
|
Drop `--apply` for a dry-run (parses + logs counts, writes nothing).
|
|
|
|
## Deploy
|
|
|
|
Like fleettickets: a Coolify container/cron in the tracksolid stack runs
|
|
`python run_migrations.py` then `python import_fuel.py --snapshot --apply` hourly (matching the
|
|
n8n export cadence). Env from the Coolify `.env`. The `dashboard_api` and `fleetops` apps ride
|
|
their own existing pipelines.
|