Fix 5 webhook bugs: SAVEPOINTs, NULL guards, BCD timestamps, /pushevent, log NULL fix

BUG-01: OBD event_time — try unix_to_ts before clean_ts (Jimi sends epoch ints) BUG-02: push_alarm — guard alarm_type not null (NULL breaks ON CONFLICT dedup) BUG-03: push_trip_report — _parse_trip_ts handles Jimi BCD format YYMMDDHHmmss BUG-04: SAVEPOINT per item in all 5 DB endpoints (FK violation on one item no longer aborts the whole batch; SAVEPOINT now inside try for safety) BUG-05: Add /pushevent endpoint (log-only; was returning 404 to Jimi) FIX: push_fault_info — skip null fault_code (NULL != NULL in PG unique index) FIX: log_ingestion — pass SQL NULL not string "None" when no error occurred Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add 02_tracksolid_docker_commands.md — remote DB command reference
2026-04-11 18:19:13 +03:00 · 2026-04-11 08:23:01 +03:00 · 2026-04-11 07:52:28 +03:00 · 2026-04-11 07:42:33 +03:00 · 2026-04-11 07:29:49 +03:00
5 changed files with 1371 additions and 322 deletions
--- a/02_tracksolid_docker_commands.md
+++ b/02_tracksolid_docker_commands.md
@ -0,0 +1,733 @@
 # Tracksolid — Docker & Remote Database Command Reference
 **Server:** `rahamafresh.com` · **User:** `kianiadee`  
 **DB:** `tracksolid_db` · **DB user:** `postgres`  
 **Container:** TimescaleDB — name includes a Coolify-generated suffix that changes on every redeploy
 ---
 ## Table of Contents
 1. [How It Works](#1-how-it-works)
 2. [Setup — Shell Function on the Server](#2-setup--shell-function-on-the-server)
 3. [Connecting From Your Mac](#3-connecting-from-your-mac)
 4. [Fleet & Live Positions](#4-fleet--live-positions)
 5. [Trips & Movement](#5-trips--movement)
 6. [Alarms](#6-alarms)
 7. [Parking & Idle Time](#7-parking--idle-time)
 8. [Position History & Route Replay](#8-position-history--route-replay)
 9. [Ingestion Pipeline Health](#9-ingestion-pipeline-health)
 10. [Device & Fleet Registry](#10-device--fleet-registry)
 11. [Schema & Migrations](#11-schema--migrations)
 12. [Container & Service Operations](#12-container--service-operations)
 13. [Quick Reference — Flag Meanings](#13-quick-reference--flag-meanings)
 ---
 ## 1. How It Works
 All remote database access uses two layers:
 ```
 Your Mac  →  SSH to rahamafresh.com  →  docker exec into timescale_db  →  psql
 ```
 **Layer 1 — find the container.** Coolify appends a random suffix to container names on every redeploy, so the container name is never hardcoded. Instead, resolve it dynamically:
 ```bash
 TS_DB=$(docker ps --filter "name=timescale_db" --format "{{.Names}}" | head -1)
 ```
 **Layer 2 — run psql inside it.**
 ```bash
 docker exec -i "$TS_DB" psql -U postgres -d tracksolid_db -c "SELECT ..."
 ```
 Use `-i` (not `-it`) when running non-interactively or piping input — the `-t` TTY flag conflicts with stdin redirection and causes errors.
 ---
 ## 2. Setup — Shell Function on the Server
 Add this once to `~/.zshrc` on the server. It resolves the container automatically and passes all arguments through to `psql`.
 ```bash
 # Auto-resolves current TimescaleDB container — survives Coolify redeployments
 tsdb() {
  local container
  container=$(docker ps --filter "name=timescale_db" --format "{{.Names}}" | head -1)
  if [[ -z "$container" ]]; then
    echo "ERROR: no running timescale_db container found" >&2
    return 1
  fi
  docker exec -it "$container" psql -U postgres -d tracksolid_db "$@"
 }
 ```
 **Activate:**
 ```bash
 source ~/.zshrc
 ```
 **Usage on the server:**
 ```bash
 tsdb                                         # open interactive psql prompt
 tsdb -c "SELECT COUNT(*) FROM tracksolid.trips;"
 tsdb -c "\dt tracksolid.*"                   # list all tracksolid tables
 tsdb -c "\d tracksolid.trips"                # describe a table
 tsdb -f /app/my_query.sql                    # run a SQL file
 ```
 ---
 ## 3. Connecting From Your Mac
 ### Open an interactive psql session
 ```bash
 ssh kianiadee@rahamafresh.com -t \
  'TS=$(docker ps --filter "name=timescale_db" --format "{{.Names}}" | head -1) &&
   docker exec -it "$TS" psql -U postgres -d tracksolid_db'
 ```
 ### Run a single query and return output to your terminal
 ```bash
 ssh kianiadee@rahamafresh.com \
  'TS=$(docker ps --filter "name=timescale_db" --format "{{.Names}}" | head -1) &&
   docker exec -i "$TS" psql -U postgres -d tracksolid_db -c "SELECT now();"'
 ```
 ### Pipe a local SQL file into the remote database
 ```bash
 ssh kianiadee@rahamafresh.com \
  'TS=$(docker ps --filter "name=timescale_db" --format "{{.Names}}" | head -1) &&
   docker exec -i "$TS" psql -U postgres -d tracksolid_db' \
  < /path/to/local_query.sql
 ```
 ### Run a migration file from your Mac
 ```bash
 ssh kianiadee@rahamafresh.com \
  'TS=$(docker ps --filter "name=timescale_db" --format "{{.Names}}" | head -1) &&
   docker exec -i "$TS" psql -U postgres -d tracksolid_db' \
  < 05_enhancement_migration.sql
 ```
 > **Note on quoting:** When embedding SQL in a shell one-liner, single-quote literals inside the SQL need to be escaped as `'"'"'`. The `tsdb()` function avoids this entirely — use it for complex queries.
 ---
 ## 4. Fleet & Live Positions
 ### All devices with a position in the last hour
 ```bash
 ssh kianiadee@rahamafresh.com \
  'TS=$(docker ps --filter "name=timescale_db" --format "{{.Names}}" | head -1) &&
   docker exec -i "$TS" psql -U postgres -d tracksolid_db -c "
 SELECT
  d.device_name,
  d.mc_type,
  ROUND(lp.lat::numeric, 5)  AS lat,
  ROUND(lp.lng::numeric, 5)  AS lng,
  lp.speed,
  lp.acc_status,
  lp.gps_signal,
  lp.gps_num                 AS satellites,
  lp.gps_time AT TIME ZONE '"'"'Africa/Nairobi'"'"' AS last_fix_eat,
  ROUND(EXTRACT(EPOCH FROM (now() - lp.gps_time)) / 60.0, 0) AS mins_ago
 FROM tracksolid.live_positions lp
 JOIN tracksolid.devices d ON d.imei = lp.imei
 WHERE lp.gps_time > now() - interval '"'"'1 hour'"'"'
 ORDER BY lp.gps_time DESC;"'
 ```
 ### All 19 live positions — current snapshot
 ```sql
 SELECT
  d.device_name,
  d.mc_type,
  ROUND(lp.lat::numeric, 5)   AS lat,
  ROUND(lp.lng::numeric, 5)   AS lng,
  lp.speed,
  lp.acc_status,
  lp.gps_time AT TIME ZONE 'Africa/Nairobi' AS last_fix_eat,
  ROUND(EXTRACT(EPOCH FROM (now() - lp.gps_time)) / 3600.0, 1) AS hours_ago,
  lp.current_mileage          AS odometer_km
 FROM tracksolid.live_positions lp
 JOIN tracksolid.devices d ON d.imei = lp.imei
 ORDER BY lp.gps_time DESC;
 ```
 ### Devices with no position (silent fleet)
 ```sql
 SELECT d.imei, d.device_name, d.mc_type, d.sim, d.expiration::date
 FROM tracksolid.devices d
 LEFT JOIN tracksolid.live_positions lp ON lp.imei = d.imei
 WHERE lp.imei IS NULL
 ORDER BY d.mc_type, d.device_name;
 ```
 ### Vehicles currently moving (ACC on OR speed > 0)
 ```sql
 SELECT
  d.device_name,
  lp.speed,
  lp.acc_status,
  lp.gps_time AT TIME ZONE 'Africa/Nairobi' AS last_fix_eat
 FROM tracksolid.live_positions lp
 JOIN tracksolid.devices d ON d.imei = lp.imei
 WHERE lp.speed > 0 OR lp.acc_status = '1'
 ORDER BY lp.speed DESC;
 ```
 ### Geographic anomaly check — vehicles outside Kenya
 ```sql
 SELECT
  d.device_name,
  ROUND(lp.lat::numeric, 4) AS lat,
  ROUND(lp.lng::numeric, 4) AS lng,
  lp.gps_time AT TIME ZONE 'Africa/Nairobi' AS last_fix_eat
 FROM tracksolid.live_positions lp
 JOIN tracksolid.devices d ON d.imei = lp.imei
 WHERE lp.lat NOT BETWEEN -5.0 AND 5.0
   OR lp.lng NOT BETWEEN 33.9 AND 42.0;
 ```
 ---
 ## 5. Trips & Movement
 ### Trips today — summary per vehicle
 ```sql
 SELECT
  d.device_name,
  d.vehicle_number,
  d.driver_name,
  COUNT(*)                                        AS trips,
  ROUND(SUM(t.distance_km)::numeric, 2)           AS total_km,
  ROUND(AVG(t.avg_speed_kmh)::numeric, 1)         AS avg_speed_kmh,
  MAX(t.max_speed_kmh)                            AS top_speed_kmh,
  ROUND(SUM(t.driving_time_s) / 3600.0, 2)       AS drive_hours,
  ROUND(SUM(t.idle_time_s) / 3600.0, 2)          AS idle_hours,
  MIN(t.start_time AT TIME ZONE 'Africa/Nairobi') AS day_start,
  MAX(t.end_time   AT TIME ZONE 'Africa/Nairobi') AS day_end
 FROM tracksolid.trips t
 JOIN tracksolid.devices d ON d.imei = t.imei
 WHERE t.start_time >= CURRENT_DATE AT TIME ZONE 'Africa/Nairobi'
 GROUP BY d.device_name, d.vehicle_number, d.driver_name
 ORDER BY total_km DESC;
 ```
 ### Trips in last 24 hours — full detail
 ```sql
 SELECT
  d.device_name,
  t.start_time AT TIME ZONE 'Africa/Nairobi' AS start_eat,
  t.end_time   AT TIME ZONE 'Africa/Nairobi' AS end_eat,
  ROUND(t.distance_km::numeric, 2)            AS distance_km,
  t.avg_speed_kmh,
  t.max_speed_kmh,
  ROUND(t.driving_time_s / 60.0, 0)          AS drive_mins,
  ROUND(t.idle_time_s    / 60.0, 0)          AS idle_mins,
  t.source
 FROM tracksolid.trips t
 JOIN tracksolid.devices d ON d.imei = t.imei
 WHERE t.start_time > now() - interval '24 hours'
 ORDER BY t.start_time DESC;
 ```
 ### Speeding incidents — trips where max speed exceeded threshold
 ```sql
 -- Change 80 to your speed limit in km/h
 SELECT
  d.device_name,
  d.driver_name,
  t.start_time AT TIME ZONE 'Africa/Nairobi' AS trip_start,
  t.max_speed_kmh,
  ROUND(t.distance_km::numeric, 2)            AS distance_km
 FROM tracksolid.trips t
 JOIN tracksolid.devices d ON d.imei = t.imei
 WHERE t.max_speed_kmh > 80
 ORDER BY t.max_speed_kmh DESC;
 ```
 ### After-hours trips (before 06:00 or after 20:00 EAT)
 ```sql
 SELECT
  d.device_name,
  d.driver_name,
  t.start_time AT TIME ZONE 'Africa/Nairobi' AS start_eat,
  ROUND(t.distance_km::numeric, 2)            AS distance_km
 FROM tracksolid.trips t
 JOIN tracksolid.devices d ON d.imei = t.imei
 WHERE EXTRACT(HOUR FROM t.start_time AT TIME ZONE 'Africa/Nairobi') >= 20
   OR EXTRACT(HOUR FROM t.start_time AT TIME ZONE 'Africa/Nairobi') < 6
 ORDER BY t.start_time DESC;
 ```
 ### Fleet utilisation rate per vehicle (today)
 ```sql
 SELECT
  d.device_name,
  d.driver_name,
  ROUND(SUM(t.driving_time_s) / 3600.0, 2)                        AS drive_hours,
  ROUND(SUM(t.idle_time_s)    / 3600.0, 2)                        AS idle_hours,
  ROUND(SUM(t.driving_time_s) / (10.0 * 3600) * 100, 1)          AS utilisation_pct
 FROM tracksolid.trips t
 JOIN tracksolid.devices d ON d.imei = t.imei
 WHERE t.start_time >= CURRENT_DATE AT TIME ZONE 'Africa/Nairobi'
 GROUP BY d.device_name, d.driver_name
 ORDER BY utilisation_pct DESC;
 ```
 ### Vehicles that did not move today
 ```sql
 SELECT d.device_name, d.mc_type, d.driver_name
 FROM tracksolid.devices d
 LEFT JOIN tracksolid.trips t
  ON t.imei = d.imei
  AND t.start_time >= CURRENT_DATE AT TIME ZONE 'Africa/Nairobi'
 WHERE t.imei IS NULL
 ORDER BY d.device_name;
 ```
 ### Distance per driver — last 30 days
 ```sql
 SELECT
  d.driver_name,
  COUNT(DISTINCT DATE(t.start_time AT TIME ZONE 'Africa/Nairobi')) AS active_days,
  COUNT(*)                                                           AS total_trips,
  ROUND(SUM(t.distance_km)::numeric, 0)                            AS total_km,
  ROUND(AVG(t.distance_km)::numeric, 1)                            AS avg_km_per_trip,
  MAX(t.max_speed_kmh)                                             AS top_speed_ever
 FROM tracksolid.trips t
 JOIN tracksolid.devices d ON d.imei = t.imei
 WHERE t.start_time > now() - interval '30 days'
  AND d.driver_name IS NOT NULL AND d.driver_name != ''
 GROUP BY d.driver_name
 ORDER BY total_km DESC;
 ```
 ---
 ## 6. Alarms
 ### All alarms — last 24 hours
 ```sql
 SELECT
  d.device_name,
  a.alarm_type,
  a.alarm_name,
  a.alarm_time AT TIME ZONE 'Africa/Nairobi' AS alarm_time_eat,
  ROUND(a.lat::numeric, 5)                   AS lat,
  ROUND(a.lng::numeric, 5)                   AS lng,
  a.speed,
  a.severity,
  a.acknowledged_at
 FROM tracksolid.alarms a
 JOIN tracksolid.devices d ON d.imei = a.imei
 WHERE a.alarm_time > now() - interval '24 hours'
 ORDER BY a.alarm_time DESC;
 ```
 ### Alarm summary by type — last 7 days
 ```sql
 SELECT
  a.alarm_name,
  a.alarm_type,
  COUNT(*)                                         AS occurrences,
  COUNT(DISTINCT a.imei)                           AS devices_affected,
  MAX(a.alarm_time AT TIME ZONE 'Africa/Nairobi') AS last_seen_eat
 FROM tracksolid.alarms a
 WHERE a.alarm_time > now() - interval '7 days'
 GROUP BY a.alarm_name, a.alarm_type
 ORDER BY occurrences DESC;
 ```
 ### Unacknowledged alarms
 ```sql
 SELECT
  d.device_name,
  a.alarm_name,
  a.alarm_time AT TIME ZONE 'Africa/Nairobi' AS alarm_time_eat,
  ROUND(EXTRACT(EPOCH FROM (now() - a.alarm_time)) / 3600.0, 1) AS hours_open
 FROM tracksolid.alarms a
 JOIN tracksolid.devices d ON d.imei = a.imei
 WHERE a.acknowledged_at IS NULL
 ORDER BY a.alarm_time DESC;
 ```
 ### Acknowledge an alarm
 ```sql
 UPDATE tracksolid.alarms
 SET acknowledged_at = now(),
    acknowledged_by = 'operator_name'
 WHERE id = <alarm_id>;
 ```
 ---
 ## 7. Parking & Idle Time
 ### Parking events today
 ```sql
 SELECT
  d.device_name,
  pe.event_type,
  pe.start_time AT TIME ZONE 'Africa/Nairobi' AS start_eat,
  pe.end_time   AT TIME ZONE 'Africa/Nairobi' AS end_eat,
  ROUND(pe.duration_seconds / 60.0, 0)        AS duration_mins,
  pe.address
 FROM tracksolid.parking_events pe
 JOIN tracksolid.devices d ON d.imei = pe.imei
 WHERE pe.start_time >= CURRENT_DATE AT TIME ZONE 'Africa/Nairobi'
 ORDER BY pe.start_time DESC;
 ```
 ### Longest idle periods — last 7 days
 ```sql
 SELECT
  d.device_name,
  d.driver_name,
  pe.start_time AT TIME ZONE 'Africa/Nairobi' AS idle_start,
  ROUND(pe.duration_seconds / 3600.0, 2)      AS idle_hours,
  pe.address
 FROM tracksolid.parking_events pe
 JOIN tracksolid.devices d ON d.imei = pe.imei
 WHERE pe.start_time > now() - interval '7 days'
 ORDER BY pe.duration_seconds DESC
 LIMIT 20;
 ```
 ---
 ## 8. Position History & Route Replay
 ### Position history by source — counts
 ```sql
 SELECT
  source,
  COUNT(*) AS fixes,
  MIN(gps_time AT TIME ZONE 'Africa/Nairobi') AS earliest,
  MAX(gps_time AT TIME ZONE 'Africa/Nairobi') AS latest
 FROM tracksolid.position_history
 GROUP BY source;
 ```
 ### Route replay for a specific vehicle — last 24 hours
 ```sql
 -- Replace 'FRED KMGW 538W HULETI' with the device_name you want
 SELECT
  ph.gps_time AT TIME ZONE 'Africa/Nairobi' AS gps_time_eat,
  ROUND(ph.lat::numeric, 5)                 AS lat,
  ROUND(ph.lng::numeric, 5)                 AS lng,
  ph.speed,
  ph.direction,
  ph.acc_status,
  ph.source
 FROM tracksolid.position_history ph
 JOIN tracksolid.devices d ON d.imei = ph.imei
 WHERE d.device_name = 'FRED KMGW 538W HULETI'
  AND ph.gps_time > now() - interval '24 hours'
 ORDER BY ph.gps_time ASC;
 ```
 ### Fix density per device — last 24 hours
 ```sql
 SELECT
  d.device_name,
  COUNT(*)                                         AS total_fixes,
  COUNT(*) FILTER (WHERE ph.source = 'poll')       AS poll_fixes,
  COUNT(*) FILTER (WHERE ph.source = 'track_list') AS track_list_fixes,
  MIN(ph.gps_time AT TIME ZONE 'Africa/Nairobi')  AS first_fix,
  MAX(ph.gps_time AT TIME ZONE 'Africa/Nairobi')  AS last_fix
 FROM tracksolid.position_history ph
 JOIN tracksolid.devices d ON d.imei = ph.imei
 WHERE ph.gps_time > now() - interval '24 hours'
 GROUP BY d.device_name
 ORDER BY total_fixes DESC;
 ```
 ---
 ## 9. Ingestion Pipeline Health
 ### Pipeline health — last hour (key check)
 ```sql
 SELECT
  endpoint,
  COUNT(*)                                          AS calls,
  SUM(rows_upserted)                               AS upserted,
  SUM(rows_inserted)                               AS inserted,
  ROUND(AVG(duration_ms)::numeric, 0)              AS avg_ms,
  COUNT(*) FILTER (WHERE success = false)          AS failures,
  MAX(run_at AT TIME ZONE 'Africa/Nairobi')        AS last_call_eat
 FROM tracksolid.ingestion_log
 WHERE run_at > now() - interval '1 hour'
 GROUP BY endpoint
 ORDER BY calls DESC;
 ```
 ### All-time ingestion summary
 ```sql
 SELECT
  endpoint,
  COUNT(*)                                          AS total_calls,
  SUM(rows_upserted)                               AS total_upserted,
  SUM(rows_inserted)                               AS total_inserted,
  ROUND(AVG(duration_ms)::numeric, 0)              AS avg_ms,
  COUNT(*) FILTER (WHERE success = false)          AS failures,
  MIN(run_at AT TIME ZONE 'Africa/Nairobi')        AS first_call,
  MAX(run_at AT TIME ZONE 'Africa/Nairobi')        AS last_call
 FROM tracksolid.ingestion_log
 GROUP BY endpoint
 ORDER BY total_calls DESC;
 ```
 ### Recent calls — last 20 entries
 ```sql
 SELECT
  run_at AT TIME ZONE 'Africa/Nairobi' AS run_eat,
  endpoint,
  imei_count,
  rows_upserted,
  rows_inserted,
  duration_ms,
  success,
  error_message
 FROM tracksolid.ingestion_log
 ORDER BY run_at DESC
 LIMIT 20;
 ```
 ### Failed calls — all time
 ```sql
 SELECT
  run_at AT TIME ZONE 'Africa/Nairobi' AS run_eat,
  endpoint,
  error_code,
  error_message
 FROM tracksolid.ingestion_log
 WHERE success = false
 ORDER BY run_at DESC;
 ```
 ---
 ## 10. Device & Fleet Registry
 ### Full fleet — all 63 devices
 ```sql
 SELECT
  device_name,
  mc_type,
  vehicle_number,
  driver_name,
  sim,
  ROUND(current_mileage_km::numeric, 0) AS odometer_km,
  expiration::date                       AS expires,
  enabled_flag
 FROM tracksolid.devices
 ORDER BY mc_type, device_name;
 ```
 ### Fleet by model
 ```sql
 SELECT mc_type, COUNT(*) AS devices
 FROM tracksolid.devices
 GROUP BY mc_type ORDER BY devices DESC;
 ```
 ### Odometer leaders — top 15
 ```sql
 SELECT
  device_name,
  mc_type,
  sim,
  ROUND(current_mileage_km::numeric, 0) AS odometer_km,
  expiration::date                       AS expires
 FROM tracksolid.devices
 WHERE current_mileage_km IS NOT NULL AND current_mileage_km > 0
 ORDER BY current_mileage_km DESC
 LIMIT 15;
 ```
 ### Devices needing metadata (blank vehicle_number or driver_name)
 ```sql
 SELECT device_name, mc_type, sim
 FROM tracksolid.devices
 WHERE vehicle_number IS NULL OR vehicle_number = ''
   OR driver_name    IS NULL OR driver_name    = ''
 ORDER BY mc_type, device_name;
 ```
 ### Subscription status breakdown
 ```sql
 SELECT
  COUNT(*) FILTER (WHERE expiration IS NULL)                               AS no_expiry_set,
  COUNT(*) FILTER (WHERE expiration < now())                               AS already_expired,
  COUNT(*) FILTER (WHERE expiration BETWEEN now() AND now() + interval '90 days') AS expiring_90days,
  COUNT(*) FILTER (WHERE expiration > now() + interval '90 days')          AS valid_long_term
 FROM tracksolid.devices;
 ```
 ### Update vehicle metadata for a device
 ```sql
 UPDATE tracksolid.devices
 SET vehicle_number = 'KDA 123B',
    driver_name    = 'John Kamau',
    vehicle_category = 'van'
 WHERE device_name = 'FRED KMGW 538W HULETI';
 ```
 ---
 ## 11. Schema & Migrations
 ### Check applied migrations
 ```sql
 SELECT filename, applied_at AT TIME ZONE 'Africa/Nairobi' AS applied_eat
 FROM tracksolid.schema_migrations
 ORDER BY applied_at;
 ```
 ### List all tables in tracksolid schema
 ```sql
 SELECT table_name FROM information_schema.tables
 WHERE table_schema = 'tracksolid'
 ORDER BY table_name;
 ```
 ### Row counts across all key tables
 ```sql
 SELECT
  (SELECT COUNT(*) FROM tracksolid.devices)              AS devices,
  (SELECT COUNT(*) FROM tracksolid.live_positions)       AS live_positions,
  (SELECT COUNT(*) FROM tracksolid.position_history)     AS position_history,
  (SELECT COUNT(*) FROM tracksolid.trips)                AS trips,
  (SELECT COUNT(*) FROM tracksolid.alarms)               AS alarms,
  (SELECT COUNT(*) FROM tracksolid.parking_events)       AS parking_events,
  (SELECT COUNT(*) FROM tracksolid.obd_readings)         AS obd_readings,
  (SELECT COUNT(*) FROM tracksolid.device_events)        AS device_events,
  (SELECT COUNT(*) FROM tracksolid.fuel_readings)        AS fuel_readings,
  (SELECT COUNT(*) FROM tracksolid.temperature_readings) AS temperature_readings,
  (SELECT COUNT(*) FROM tracksolid.lbs_readings)         AS lbs_readings,
  (SELECT COUNT(*) FROM tracksolid.ingestion_log)        AS ingestion_log;
 ```
 ### Describe a table
 ```bash
 # On the server
 tsdb -c "\d tracksolid.trips"
 tsdb -c "\d tracksolid.live_positions"
 ```
 ### Run DWH gold ETL for yesterday
 ```sql
 SELECT dwh_gold.refresh_daily_metrics(CURRENT_DATE - 1);
 ```
 ### Apply a migration manually (from your Mac)
 ```bash
 ssh kianiadee@rahamafresh.com \
  'TS=$(docker ps --filter "name=timescale_db" --format "{{.Names}}" | head -1) &&
   docker exec -i "$TS" psql -U postgres -d tracksolid_db' \
  < 05_enhancement_migration.sql
 ```
 ---
 ## 12. Container & Service Operations
 ### Find the current TimescaleDB container name
 ```bash
 ssh kianiadee@rahamafresh.com \
  'docker ps --filter "name=timescale_db" --format "{{.Names}}"'
 ```
 ### List all running containers
 ```bash
 ssh kianiadee@rahamafresh.com 'docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"'
 ```
 ### Check container logs (last 100 lines)
 ```bash
 # TimescaleDB
 ssh kianiadee@rahamafresh.com \
  'TS=$(docker ps --filter "name=timescale_db" --format "{{.Names}}" | head -1) &&
   docker logs "$TS" --tail 100'
 # Ingest movement service
 ssh kianiadee@rahamafresh.com \
  'SVC=$(docker ps --filter "name=ingest_movement" --format "{{.Names}}" | head -1) &&
   docker logs "$SVC" --tail 100'
 # Ingest events service
 ssh kianiadee@rahamafresh.com \
  'SVC=$(docker ps --filter "name=ingest_events" --format "{{.Names}}" | head -1) &&
   docker logs "$SVC" --tail 100'
 # Webhook receiver
 ssh kianiadee@rahamafresh.com \
  'SVC=$(docker ps --filter "name=webhook_receiver" --format "{{.Names}}" | head -1) &&
   docker logs "$SVC" --tail 100'
 ```
 ### Follow logs live (stream)
 ```bash
 ssh kianiadee@rahamafresh.com \
  'SVC=$(docker ps --filter "name=ingest_movement" --format "{{.Names}}" | head -1) &&
   docker logs "$SVC" --follow --tail 50'
 ```
 ### Restart a service
 ```bash
 # Replace <service_name_prefix> with: ingest_movement, ingest_events, webhook_receiver
 ssh kianiadee@rahamafresh.com \
  'SVC=$(docker ps --filter "name=ingest_movement" --format "{{.Names}}" | head -1) &&
   docker restart "$SVC"'
 ```
 ### Check disk space on the server
 ```bash
 ssh kianiadee@rahamafresh.com 'df -h /'
 ```
 ### Check TimescaleDB chunk sizes (storage usage)
 ```sql
 SELECT
  hypertable_name,
  pg_size_pretty(SUM(total_bytes)) AS total_size
 FROM timescaledb_information.chunks
 GROUP BY hypertable_name
 ORDER BY SUM(total_bytes) DESC;
 ```
 ### Vacuum / analyse a table (maintenance)
 ```bash
 ssh kianiadee@rahamafresh.com \
  'TS=$(docker ps --filter "name=timescale_db" --format "{{.Names}}" | head -1) &&
   docker exec -i "$TS" psql -U postgres -d tracksolid_db -c "
   VACUUM ANALYSE tracksolid.position_history;"'
 ```
 ---
 ## 13. Quick Reference — Flag Meanings
 | Flag | Context | Meaning |
 |---|---|---|
 | `--filter "name=X"` | `docker ps` | Match containers whose name contains `X` |
 | `--format "{{.Names}}"` | `docker ps` | Output only the container name column |
 | `head -1` | shell | Take only the first result (guard against duplicates) |
 | `exec -i` | `docker exec` | Keep stdin open — required for piping SQL or running non-interactively |
 | `exec -it` | `docker exec` | Add a TTY — use only for interactive sessions, not when piping |
 | `-U postgres` | `psql` | Connect as the `postgres` superuser |
 | `-d tracksolid_db` | `psql` | Target this database |
 | `-c "..."` | `psql` | Run a single SQL statement and exit |
 | `-f file.sql` | `psql` | Execute all SQL in a file |
 | `AT TIME ZONE 'Africa/Nairobi'` | SQL | Convert UTC timestamp to EAT (UTC+3) for display |
 ---
 *Last updated: 2026-04-11*
--- a/260410_baseline_report.md
+++ b/260410_baseline_report.md
@ -1,335 +1,509 @@
 # Fireside Communications — Fleet Baseline Report
-## Date: 2026-04-10 · Database: tracksolid_db · Generated: 23:18 EAT
+**Date:** 2026-04-11 · **Time of queries:** ~07:38 EAT  
-
+**Database:** tracksolid_db on TimescaleDB  
-> **Baseline snapshot taken on the first night of active pipeline operation.**
+**Container:** timescale_db-bo3nov2ija7g8wn9b1g2paxs-204435447351  
-> Container: `timescale_db-bo3nov2ija7g8wn9b1g2paxs-195053614609`
+**Report scope:** All 63 registered devices · All tables · Post-migration 04 + 05
 > Ingestion has been live for approximately 1 hour at time of capture.
 ---
-## 1. Executive Summary
+## 1. Migration Status
-| Metric | Value | Status |
+All four schema migrations applied and tracked:
 | Migration File | Applied (EAT) | Status |
 |---|---|---|
-| Total registered devices | 63 | — |
+| `02_tracksolid_full_schema_rev.sql` | 2026-04-10 23:45:17 | ✓ Applied |
-| Devices with live position | 19 (30%) | ⚠ 44 devices never reported |
+| `03_webhook_schema_migration.sql` | 2026-04-10 23:45:17 | ✓ Applied |
-| Devices active today | 4 | ⚠ Low — evening snapshot |
+| `04_bug_fix_migration.sql` | 2026-04-10 23:45:17 | ✓ Applied — `distance_km` renamed & corrected |
-| Position history rows | 28 | ⚠ Pipeline started today |
+| `05_enhancement_migration.sql` | 2026-04-10 23:45:17 | ✓ Applied — new tables + columns |
 | Trip records | 0 | ❌ Migration 04 not yet applied |
 | Alarm records | 0 | ❌ No alarms ingested yet |
 | Parking events | 0 | ⚠ API returning 0 (fix deployed, containers not redeployed) |
 | Ingestion pipeline health | ✅ Running | 60-second polling confirmed |
 | Migration 04 applied (distance_km rename) | ❌ No | `distance_m` column still present |
 | Migration 05 applied (new tables) | ❌ No | New tables don't exist yet |
-**Key finding:** The ingestion pipeline started successfully tonight and is polling correctly. However, only 19 of 63 devices are returning live positions from the Tracksolid API. The remaining 44 devices are registered in the system but have never reported a GPS fix — they may be inactive, uninstalled, or require account-level configuration in Tracksolid Pro.
+Schema is fully current. No pending migrations.
 ---
-## 2. Fleet Composition
+## 2. Table Row Counts (as of 07:38 EAT)
-### Device types registered
+| Table | Rows | Notes |
 |---|---|---|
 | `tracksolid.devices` | **63** | Full fleet registry |
 | `tracksolid.live_positions` | **19** | 19 devices with a known position (30% of fleet) |
 | `tracksolid.position_history` | **137** | All historical GPS fixes — growing |
 | `tracksolid.alarms` | **2** | 2 vibration alerts overnight |
 | `tracksolid.trips` | **0** | Fleet parked overnight — expected |
 | `tracksolid.parking_events` | **0** | Fix deployed; will populate with movement |
 | `tracksolid.obd_readings` | **0** | Awaiting webhook registration |
 | `tracksolid.device_events` | **0** | Awaiting `/pushevent` registration |
 | `tracksolid.fuel_readings` | **0** | Awaiting `/pushoil` registration |
 | `tracksolid.temperature_readings` | **0** | Awaiting `/pushtem` registration |
 | `tracksolid.lbs_readings` | **0** | Awaiting `/pushlbs` registration |
 | `tracksolid.geofences` | **0** | Not yet configured |
 | `tracksolid.heartbeats` | **0** | Awaiting heartbeat webhook |
 | `tracksolid.fault_codes` | **0** | Awaiting fault code data |
 | `tracksolid.ingestion_log` | **397** | Polling audit trail — healthy |
 | `dwh_gold.fact_daily_fleet_metrics` | **0** | ETL not yet run |
 | `dwh_gold.dim_vehicles` | **0** | Awaiting population |
-| Device Model | Count | With SIM | With Odometer | Notes |
+---
 |---|---|---|---|---|
 | AT4 | 23 | 3 | 3 | Oldest fleet — mostly blank device names |
 | JC400P | 23 | 2 | 6 | Camera-equipped trackers |
 | X3 | 10 | 4 | 6 | Newest devices (2025–2026 activations) |
 | GT06E | 7 | 5 | 5 | Mid-fleet — best data quality |
 | **Total** | **63** | **14** | **20** | |
-**Observations:**
+## 3. Fleet Composition
 - Only 14 of 63 devices have a SIM number recorded (22%)
 - Only 20 of 63 devices have an odometer reading (32%)
 - All 63 `vehicle_name`, `vehicle_number`, and `driver_name` fields are blank — reports currently show device names only
-### Named vehicles with odometer (highest mileage first)
+**63 devices across 4 device models:**
-| Device Name | IMEI | Type | Odometer (km) | Activated | SIM |
+| Model | Count | Typical Use |
 |---|---|---|
 | AT4 | 23 | Asset / cargo hardwired tracker |
 | JC400P | 23 | Camera-capable tracker (larger vehicles) |
 | X3 | 10 | Compact vehicle tracker |
 | GT06E | 7 | OBD-port tracker |
 | **Total** | **63** | |
 ---
 ## 4. Full Device Registry
 > All 63 devices. `vehicle_number` and `driver_name` are blank for every device — primary data quality gap.
 | Device Name | Model | SIM | Odometer (km) | Expires | Status |
 |---|---|---|---|---|---|
-| KDK 829A GP | 359857082898297 | GT06E | 239,264 | 2022-10-29 | 0707923872 |
+| AT4-51820 | AT4 | — | — | — | No position |
-| Belta KCU-647D | 359857082042862 | GT06E | 234,546 | 2020-04-03 | 0110094465 |
+| AT4-53099 | AT4 | — | — | — | No position |
-| JK Subaru KCS 903Y | 359857081891921 | GT06E | 73,344 | 2019-06-12 | 0746759925 |
+| AT4-54246 | AT4 | — | — | — | No position |
-| KCU 865Q Vanguard | 359857082042953 | GT06E | 61,758 | 2019-12-20 | 0757270763 |
+| AT4-55029 | AT4 | — | — | — | No position |
-| KCU 145Q Solo Xtrail | 359857082037425 | GT06E | 53,228 | 2019-12-20 | 0757270810 |
+| AT4-55235 | AT4 | — | — | — | No position |
-| KCU 865Q Vanguard Sub | 353549090555334 | AT4 | 9,656 | 2019-12-20 | 0757270804 |
+| AT4-57389 | AT4 | — | — | — | No position |
-| KMGR 409U HENRY JAZZ | 865135061048300 | X3 | 6,696 | 2025-07-31 | 0768697302 |
+| AT4-61860 | AT4 | — | — | — | No position |
-| KDU 878T_Track | 865135061040349 | X3 | 4,802 | 2025-08-18 | 0708352823 |
+| AT4-64815 | AT4 | — | — | 2036-02-05 | Stale (1,556h) |
-| KCS 903Y JK SUB | 353549090552018 | AT4 | 4,492 | 2019-06-09 | 0700024569 |
+| AT4-64823 | AT4 | — | — | — | No position |
-| X3-63282 | 865135061563282 | X3 | 4,194 | 2026-02-14 | — |
+| AT4-64880 | AT4 | — | — | — | No position |
-| KMEH 692C KAWASAKI | 353549090561654 | AT4 | 3,319 | 2020-04-03 | 0110094467 |
+| AT4-64989 | AT4 | — | — | — | No position |
-| FRED KMGW 538W HULETI | 865135061559538 | X3 | 2,284 | 2026-02-08 | 0119867174 |
+| AT4-65010 | AT4 | — | — | — | No position |
-| KDU 878T_CAM | 862798052715071 | JC400P | 1,562 | 2025-08-18 | 0708351897 |
+| AT4-65135 | AT4 | — | — | — | No position |
-| KDW 632M HL Tracker | 865135061569529 | X3 | 222 | 2026-02-09 | 300002396033 IoT |
+| AT4-65341 | AT4 | — | — | — | No position |
-| JC400P-85751 | 862798052785751 | JC400P | 17 | 2026-03-11 | — |
+| AT4-65598 | AT4 | — | — | — | No position |
-| X3-68968 | 865135061568968 | X3 | 16 | 2026-03-11 | — |
+| AT4-65648 | AT4 | — | — | — | No position |
-| KDW 632M HL Cam | 862798052707995 | JC400P | 16 | 2026-03-11 | 300002396032 IoT |
+| AT4-66158 | AT4 | — | — | — | No position |
-
+| AT4-67271 | AT4 | — | — | — | No position |
-> **Note:** `KDK 829A GP` (239,264 km) and `Belta KCU-647D` (234,546 km) are high-mileage vehicles that should be reviewed for service intervals. At typical service intervals of 10,000 km, both are well overdue unless recently serviced.
+| AT4-67693 | AT4 | — | — | — | No position |
 | Belta KCU-647D | GT06E | 0110094465 | **234,546** | 2040-04-03 | Stale (7,568h) — SERVICE FLAG |
 | FRED KMGW 538W HULETI | X3 | 0119867174 | 2,284 | 2036-02-08 | **Active (0.1h)** |
 | GT06E-85428 | GT06E | — | — | — | No position |
 | GT06E-86319 | GT06E | — | — | — | No position |
 | JC400P-07904 | JC400P | — | — | — | No position |
 | JC400P-85041 | JC400P | — | — | — | No position |
 | JC400P-85058 | JC400P | — | — | — | No position |
 | JC400P-85751 | JC400P | — | 17 | 2036-03-11 | Stale (729h) |
 | JC400P-86270 | JC400P | — | 6,931 | — | No position |
 | JC400P-86403 | JC400P | — | — | — | No position |
 | JC400P-87625 | JC400P | — | — | — | No position |
 | JC400P-87831 | JC400P | — | — | — | No position |
 | JC400P-89431 | JC400P | — | 7,375 | — | No position |
 | JC400P-89530 | JC400P | — | — | — | No position |
 | JC400P-89563 | JC400P | — | — | — | No position |
 | JC400P-89662 | JC400P | — | — | — | No position |
 | JC400P-89977 | JC400P | — | — | — | No position |
 | JC400P-90108 | JC400P | — | — | — | No position |
 | JC400P-90199 | JC400P | — | — | — | No position |
 | JC400P-90678 | JC400P | — | — | — | No position |
 | JC400P-91619 | JC400P | — | 3,719 | — | No position |
 | JC400P-92278 | JC400P | — | — | — | No position |
 | JC400P-92716 | JC400P | — | — | — | No position |
 | JC400P-92732 | JC400P | — | — | — | No position |
 | JC400P-94233 | JC400P | — | — | — | No position |
 | JK Subaru KCS 903Y | GT06E | 0746759925 | 73,345 | 2039-06-12 | Stale (671h) |
 | KCE 690F | AT4 | — | — | 2039-07-01 | Very stale (57,312h) |
 | KCS 903Y JK SUB | AT4 | 0700024569 | 4,492 | 2039-06-09 | Very stale (15,213h) |
 | KCU 145Q Solo Xtrail | GT06E | 0757270810 | 53,228 | 2039-12-20 | Stale (7,530h) |
 | KCU 865Q Vanguard | GT06E | 0757270763 | 61,758 | 2039-12-20 | Stale (62h) |
 | KCU 865Q Vanguard Sub | AT4 | 0757270804 | 9,656 | 2039-12-20 | Very stale (15,429h) |
 | KDK 829A GP | GT06E | 0707923872 | **239,264** | 2042-10-29 | **Active (0.2h)** — SERVICE FLAG |
 | KDU 878T_CAM | JC400P | 0708351897 | 1,562 | 2035-08-18 | Stale (3,064h) |
 | KDU 878T_Track | X3 | 0708352823 | 4,802 | 2035-08-18 | Stale (62h) |
 | KDW 632M HL Cam | JC400P | 300002396032 IoT | 16 | 2036-03-11 | Stale (740h) |
 | KDW 632M HL Tracker | X3 | 300002396033 IoT | 222 | 2036-02-09 | Stale (728h) |
 | KMEH 692C KAWASAKI | AT4 | 0110094467 | 3,319 | 2040-04-03 | Very stale (24,693h) |
 | KMGR 409U HENRY JAZZ | X3 | 0768697302 | 6,696 | 2035-07-31 | Semi-active (16h) |
 | X3-59405 | X3 | — | — | — | No position |
 | X3-63282 | X3 | — | 4,194 | 2036-02-14 | **Active (0.2h) — UGANDA ANOMALY** |
 | X3-64223 | X3 | — | — | — | No position |
 | X3-68968 | X3 | — | 16 | 2036-03-11 | Stale (728h) |
 | X3-69172 | X3 | — | — | — | No position |
 | X3-78553 | X3 | — | — | — | No position |
 ---
-## 3. Live Positions — Current Fleet Snapshot
+## 5. Live Position Coverage
-### Position freshness at 23:18 EAT, 2026-04-10
+**19 of 63 devices (30%)** have a position in `live_positions`.  
 **44 devices (70%)** have no position at all — offline, SIM not installed, or never activated.
-| Freshness Band | Vehicles |
+### Freshness Bands
 |---|---|
 | Fresh — last fix < 10 minutes ago | **3** |
 | Today — last fix within 24 hours | **1** |
 | This week — last fix within 7 days | **2** |
 | This month — last fix within 30 days | **3** |
 | Stale — last fix older than 30 days | **10** |
 | No position recorded | **44** |
-### Vehicles with fresh GPS fix (reporting now)
+| Band | Count | Devices |
 |---|---|---|
 | < 1 hour (active now) | 3 | FRED KMGW 538W HULETI, X3-63282, KDK 829A GP |
 | 1–24 hours | 1 | KMGR 409U HENRY JAZZ (16h) |
 | 1–7 days (stale) | 3 | KCU 865Q Vanguard (62h), KDU 878T_Track (62h), KDU 878T_CAM (3,064h) |
 | 1–12 months (very stale) | 6 | JK Subaru KCS 903Y, KCU 145Q, Belta KCU-647D, etc. |
 | > 1 year (historical only) | 6 | KCS 903Y JK SUB, KCU 865Q Vanguard Sub, KMEH 692C KAWASAKI, KCE 690F, etc. |
-| Device Name | IMEI | Last Fix (EAT) | Speed | Acc | Odometer | Coordinates | Location |
+### Full Live Position Detail
 |---|---|---|---|---|---|---|---|
 | FRED KMGW 538W HULETI | 865135061559538 | 23:16:00 | 0 km/h | Off | 2,283.89 km | -1.237, 36.727 | Nairobi (Westlands area) |
 | X3-63282 | 865135061563282 | 23:15:15 | 0 km/h | Off | 4,194.08 km | 0.196, 32.540 | **Uganda (Kampala/Entebbe area)** |
 | KDK 829A GP | 359857082898297 | 23:13:20 | 0 km/h | Off | 239,263.53 km | -1.328, 36.900 | Nairobi South / Athi River area |
-> ⚠️ **X3-63282 is currently in Uganda** (lat 0.196, lng 32.540 — near Kampala/Entebbe). If this vehicle is not expected to be cross-border, this warrants investigation.
+| Device | Model | Lat | Lng | Speed (km/h) | ACC | GPS Signal | Satellites | Last Fix (EAT) |
 |---|---|---|---|---|---|---|---|---|
 | FRED KMGW 538W HULETI | X3 | -1.23748 | 36.72662 | 0 | Off | 4 | 15 | 2026-04-11 07:34:26 |
 | X3-63282 | X3 | 0.19554 | 32.54002 | 0 | Off | 4 | 11 | 2026-04-11 07:30:15 |
 | KDK 829A GP | GT06E | -1.32787 | 36.89972 | 0 | Off | 3 | 8 | 2026-04-11 07:28:58 |
 | KMGR 409U HENRY JAZZ | X3 | -1.23748 | 36.72674 | 1 | Off | 2 | 6 | 2026-04-10 15:40:32 |
 | KCU 865Q Vanguard | GT06E | -1.23748 | 36.72641 | 5 | Off | — | — | 2026-04-08 17:17:45 |
 | KDU 878T_Track | X3 | -1.23528 | 36.72871 | 0 | Off | — | — | 2026-04-08 17:16:55 |
 | JK Subaru KCS 903Y | GT06E | -1.23558 | 36.72870 | 0 | Off | — | — | 2026-03-14 09:08:34 |
 | X3-68968 | X3 | -1.23799 | 36.72615 | 0 | Off | — | — | 2026-03-11 23:59:28 |
 | KDW 632M HL Tracker | X3 | -1.24087 | 36.72839 | 0 | Off | — | — | 2026-03-11 23:53:44 |
 | JC400P-85751 | JC400P | -1.23796 | 36.72611 | 0 | Off | — | — | 2026-03-11 22:15:44 |
 | KDW 632M HL Cam | JC400P | -1.24115 | 36.72847 | 0 | Off | — | — | 2026-03-11 11:52:01 |
 | AT4-64815 | AT4 | -1.24136 | 36.72872 | 0 | Off | — | — | 2026-02-05 11:19:55 |
 | KDU 878T_CAM | JC400P | -1.06900 | 37.01436 | 12 | Off | — | — | 2025-12-04 15:27:42 |
 | KCU 145Q Solo Xtrail | GT06E | -1.29728 | 36.88850 | 0 | Off | — | — | 2025-06-01 14:04:47 |
 | Belta KCU-647D | GT06E | -1.15151 | 36.63857 | 0 | Off | — | — | 2025-05-30 23:53:22 |
 | KCS 903Y JK SUB | AT4 | -1.23529 | 36.72875 | 0 | Off | — | — | 2024-07-16 10:41:42 |
 | KCU 865Q Vanguard Sub | AT4 | -1.23522 | 36.73104 | 0 | Off | — | — | 2024-07-07 10:43:21 |
 | KMEH 692C KAWASAKI | AT4 | -1.23849 | 36.72460 | 0 | Off | — | — | 2023-06-17 10:41:18 |
 | KCE 690F | AT4 | -1.24008 | 36.74522 | 31 | Off | — | — | 2019-09-27 07:20:08 |
-### All vehicles that reported today (2026-04-10)
+> **ACC Off for all 19 devices** — fleet parked overnight, consistent with ~03:00–07:38 EAT query window.
-| Device Name | Last Fix (EAT) | Speed | Location |
+---
 ## 6. Geographic Clustering
 All known positions confirmed in two clusters plus one critical outlier:
 | Cluster | Area | Coords | Active Devices |
 |---|---|---|---|
-| FRED KMGW 538W HULETI | 23:16 | 0 km/h | Nairobi Westlands |
+| **Primary depot** | Nairobi West / Kikuyu Rd corridor | -1.235 to -1.241, 36.724 to 36.731 | 14 devices |
-| X3-63282 | 23:15 | 0 km/h | Uganda |
+| **Secondary** | Nairobi East / Thika Rd | -1.297 to -1.328, 36.885 to 36.900 | 2 devices |
-| KDK 829A GP | 23:13 | 0 km/h | Nairobi South |
+| **Outlier** | Thika / Ruiru | -1.069, 37.014 | 1 device (KDU 878T_CAM) |
-| KMGR 409U HENRY JAZZ | 15:40 | 1 km/h | Nairobi Westlands |
+| **CRITICAL** | **Uganda — Kampala region** | **0.196, 32.540** | **1 device (X3-63282)** |
-All 4 active vehicles have ignition off (`acc_status = 0`) — fleet is parked as of report time.
+> The primary depot cluster at ~(-1.237, 36.727) is very tight — 10+ devices within a 200m radius — suggesting a single compound/yard. This is your main base.
 ### Geographic clusters of tracked fleet
 | Approximate Area | Lat/Lng | Vehicles |
 |---|---|---|
 | Nairobi — Westlands / Upper Hill | -1.24, 36.73 | 7 |
 | Uganda — Kampala / Entebbe area | 0.20, 32.54 | 1 |
 | Nairobi South — Athi River / Mlolongo | -1.33, 36.90 | 1 |
 ---
-## 4. Devices Not Reporting (44 of 63)
+## 7. Position History
-The following 44 devices are registered but have **never returned a GPS position** since the pipeline started. This is the most significant operational gap identified at baseline.
+**Total fixes: 137** across two ingestion sources:
-| IMEI | Device Name | Type |
+| Source | Fixes | Method | Frequency |
-|---|---|---|
+|---|---|---|---|
-| 353549090551820 | AT4-51820 | AT4 |
+| `poll` | 124 | Fleet-wide 60s sweep | Every 60 seconds |
-| 353549090553099 | AT4-53099 | AT4 |
+| `track_list` | 13 | Per-device high-res trail (POLL-01) | Every 30 minutes |
-| 353549090554246 | AT4-54246 | AT4 |
+| **Total** | **137** | | |
 | 353549090555029 | AT4-55029 | AT4 |
 | 353549090555235 | AT4-55235 | AT4 |
 | 353549090557389 | AT4-57389 | AT4 |
 | 353549090561860 | AT4-61860 | AT4 |
 | 353549090564823 | AT4-64823 | AT4 |
 | 353549090564880 | AT4-64880 | AT4 |
 | 353549090564989 | AT4-64989 | AT4 |
 | 353549090565010 | AT4-65010 | AT4 |
 | 353549090565135 | AT4-65135 | AT4 |
 | 353549090565341 | AT4-65341 | AT4 |
 | 353549090565598 | AT4-65598 | AT4 |
 | 353549090565648 | AT4-65648 | AT4 |
 | 353549090566158 | AT4-66158 | AT4 |
 | 353549090567271 | AT4-67271 | AT4 |
 | 353549090567693 | AT4-67693 | AT4 |
 | 359857081885428 | GT06E-85428 | GT06E |
 | 359857081886319 | GT06E-86319 | GT06E |
 | 862798052707904 | JC400P-07904 | JC400P |
 | 862798052785041 | JC400P-85041 | JC400P |
 | 862798052785058 | JC400P-85058 | JC400P |
 | 862798052786403 | JC400P-86403 | JC400P |
 | 862798052787625 | JC400P-87625 | JC400P |
 | 862798052787831 | JC400P-87831 | JC400P |
 | 862798052789530 | JC400P-89530 | JC400P |
 | 862798052789563 | JC400P-89563 | JC400P |
 | 862798052789662 | JC400P-89662 | JC400P |
 | 862798052789977 | JC400P-89977 | JC400P |
 | 862798052790108 | JC400P-90108 | JC400P |
 | 862798052790199 | JC400P-90199 | JC400P |
 | 862798052790678 | JC400P-90678 | JC400P |
 | 862798052792278 | JC400P-92278 | JC400P |
 | 862798052792716 | JC400P-92716 | JC400P |
 | 862798052792732 | JC400P-92732 | JC400P |
 | 862798052794233 | JC400P-94233 | JC400P |
 | 865135061559405 | X3-59405 | X3 |
 | 865135061564223 | X3-64223 | X3 |
 | 865135061569172 | X3-69172 | X3 |
 | 865135061578553 | X3-78553 | X3 |
 | 862798052786270 | JC400P-86270 | JC400P |
 | 862798052789431 | JC400P-89431 | JC400P |
 | 862798052791619 | JC400P-91619 | JC400P |
-**Likely causes:**
+### Per-Device Fixes — Last 24 Hours
 - Device powered off or SIM deactivated
 - Device registered in Tracksolid Pro but never activated in the field
 - Account-level permission: device may belong to a sub-account not accessible under the API credentials in use
 - Physical tracker fault or uninstalled from vehicle
-**Recommended action:** Cross-reference this list against the physical fleet inventory. For any device that should be active, log into the Tracksolid Pro web console and verify the device is online.
+| Device | Model | Fixes | First Fix (EAT) | Last Fix (EAT) | Avg Speed | Max Speed |
 |---|---|---|---|---|---|---|
 | FRED KMGW 538W HULETI | X3 | 40 | 2026-04-10 23:16 | 2026-04-11 07:34 | 0.0 km/h | 0.0 |
 | X3-63282 | X3 | 29 | 2026-04-10 23:15 | 2026-04-11 07:30 | 0.0 km/h | 0.0 |
 | KDK 829A GP | GT06E | 16 | 2026-04-10 23:13 | 2026-04-11 07:29 | 0.0 km/h | 0.0 |
 | KMGR 409U HENRY JAZZ | X3 | 1 | 2026-04-10 15:40 | 2026-04-10 15:40 | 1.0 km/h | 1.0 |
---
+**Only 4 of 63 devices generated position history in the last 24 hours.** The remaining 59 are either offline, not reporting, or have stale last-seen data from prior months.
-## 5. Ingestion Pipeline Health
+### Track List Sample (POLL-01 High-Resolution Trail)
-### Last 30 ingestion events (as at 23:18 EAT)
+13 high-res waypoints captured across 15 scheduler runs (30-min interval, overnight):
-| Endpoint | IMEIs Queried | Rows Upserted | Rows Inserted | Duration | Status |
+| Device | GPS Time (EAT) | Lat | Lng | Speed |
 |---|---|---|---|---|---|
 | `jimi.user.device.location.list` | 63 | 19 | 19 | ~200 ms | ✅ |
 | `jimi.user.device.location.list` | 63 | 19 | 19 | ~240 ms | ✅ |
 | `jimi.open.platform.report.parking` | 50+13 | 0 | 0 | ~7–15s | ⚠ 0 rows |
 | `jimi.user.device.list+detail` | 63 | 63 | 0 | 66,115 ms | ✅ |
 **Observations:**
 - Location polling running every 60 seconds — healthy and consistent
 - All API calls returning success (`t`)
 - Parking endpoint responding but returning 0 rows — the updated container with `acc_type=0` and `durSecond` fix has not yet been redeployed
 - `jimi.device.track.list` (POLL-01 high-resolution trail) not yet appearing in logs — new container not yet deployed
 - Trip polling (`jimi.device.track.mileage`) not yet appearing — new container not yet deployed
 - Device sync completed at 22:54 (66 seconds for 63 devices with detail lookups — expected)
 ---
 ## 6. Schema & Migration Status
 | Migration | Description | Status |
 |---|---|---|
 | 01–03 | Base schema, webhook tables, position_history columns | ✅ Applied |
 | **04** | `distance_m` → `distance_km` rename + historical data correction | ❌ **Not applied** |
 | **05** | New tables: device_events, fuel_readings, temperature_readings, lbs_readings, geofences; OBD/alarm/device enrichment columns; dwh_gold expansion | ❌ **Not applied** |
 **Confirmed:** `trips.distance_m` column still exists (not yet renamed to `distance_km`). Migration 04 must be run before deploying updated ingestion containers — failure to do so will cause the new code to write to a column that doesn't exist.
 **Tables present in `tracksolid` schema:**
 | Table | Rows |
 |---|---|
 | `devices` | 63 |
 | `live_positions` | 19 |
 | `position_history` | 28 |
 | `trips` | 0 |
 | `alarms` | 0 |
 | `parking_events` | 0 |
 | `obd_readings` | 0 |
 | `heartbeats` | 0 |
 | `ingestion_log` | 29+ |
 | `api_token_cache` | — |
 | `fault_codes` | — |
 **Tables NOT yet present (require migration 05):**
 - `device_events`
 - `fuel_readings`
 - `temperature_readings`
 - `lbs_readings`
 - `geofences`
 ---
 ## 7. Stale Devices — Historical Last-Seen
 The following devices have a live_positions entry but their last GPS fix is more than 30 days old:
 | Device Name | IMEI | Last Fix (EAT) | Odometer | Notes |
 |---|---|---|---|---|
-| KCS 903Y JK SUB | 353549090552018 | 2024-07-16 10:41 | 4,492 km | ~21 months stale |
+| FRED KMGW 538W HULETI | 2026-04-11 07:04:07 | -1.23748 | 36.72662 | 0 |
-| KCU 865Q Vanguard Sub | 353549090555334 | 2024-07-07 10:43 | 9,656 km | ~21 months stale |
+| KDK 829A GP | 2026-04-11 07:00:52 | -1.32800 | 36.89976 | 0 |
-| KMEH 692C KAWASAKI | 353549090561654 | 2023-06-17 10:41 | 3,319 km | ~34 months stale |
+| X3-63282 | 2026-04-11 07:00:15 | 0.19554 | 32.54002 | 0 |
-| KCE 690F | 353549090565580 | 2019-09-27 07:20 | 0 km | ~6.5 years stale |
+| FRED KMGW 538W HULETI | 2026-04-11 06:54:01 | -1.23748 | 36.72662 | 0 |
-| KDU 878T_CAM | 862798052715071 | 2025-12-04 15:27 | 1,562 km | ~4 months stale |
+| X3-63282 | 2026-04-11 06:45:15 | 0.19554 | 32.54002 | 0 |
-| KCU 145Q Solo Xtrail | 359857082037425 | 2025-06-01 14:04 | 53,228 km | ~10 months stale |
+
-| Belta KCU-647D | 359857082042862 | 2025-05-30 23:53 | 234,546 km | ~10 months stale |
+All track_list fixes show speed = 0 and stationary coordinates, confirming overnight parking. Altitude data not yet populated (device-dependent feature).
 | KDW 632M HL Cam | 862798052707995 | 2026-03-11 11:52 | 16 km | 30 days — may need SIM activation |
 | KDW 632M HL Tracker | 865135061569529 | 2026-03-11 23:53 | 222 km | 30 days — may need SIM activation |
 | JC400P-85751 | 862798052785751 | 2026-03-11 22:15 | 17 km | 30 days — brand new, 17 km only |
 | AT4-64815 | 353549090564815 | 2026-02-05 11:19 | 0 km | 64 days stale |
 | JK Subaru KCS 903Y | 359857081891921 | 2026-03-14 00:55 | 73,344 km | 27 days |
 ---
-## 8. Pending Actions Before Full Operation
+## 8. Alarms
-The following steps are required to move from baseline to fully operational. Listed in execution order:
+**Total alarms: 2** — both on the same device, overnight.
-| Priority | Action | Impact |
+| # | Device | Alarm Type | Alarm Name | Time (EAT) | Lat | Lng | Speed |
 |---|---|---|---|---|---|---|---|
 | 1 | FRED KMGW 538W HULETI | 3 | Vibration alert | 2026-04-11 00:57:30 | -1.23752 | 36.72660 | 0 |
 | 2 | FRED KMGW 538W HULETI | 3 | Vibration alert | 2026-04-11 02:58:36 | -1.23748 | 36.72660 | 0 |
 **Key findings:**
 - **BUG-01 confirmed fixed** — `alarm_type` (3) and `alarm_name` ("Vibration alert") now correctly populated from polling. Previously both were always NULL.
 - Both events at stationary coordinates, ~2 hours apart while parked.
 - Vibration type 3 while speed = 0 suggests: tamper attempt, animal contact, or sensor vibration from nearby traffic.
 - Vehicle position is at the primary Kikuyu Rd depot cluster — consistent with a parked asset.
 - No speed alarms, no geofence alarms, no power alarms.
 ---
 ## 9. Trips
 **Trips recorded: 0**
 Expected — all vehicles parked overnight. Schema confirmed correct:
 | Column | Present | Notes |
 |---|---|---|
-| 🔴 1 | **Run migration 04** on production DB | Renames `distance_m` → `distance_km`; corrects historical data |
+| `distance_km` | ✓ | Renamed from `distance_m`, values corrected ÷1,000,000 (BUG-02 fixed) |
-| 🔴 2 | **Run migration 05** on production DB | Creates new tables for expanded ingestion |
+| `max_speed_kmh` | ✓ | Will populate from next trip poll (BUG-03 fixed) |
-| 🔴 3 | **Redeploy updated ingestion containers** | Activates: trip polling, parking fix, high-res GPS trails, alarm field fix |
+| `avg_speed_kmh` | ✓ | |
-| 🟠 4 | **Investigate 44 non-reporting devices** | Cross-check against physical fleet; verify online in Tracksolid Pro console |
+| `driving_time_s` | ✓ | |
-| 🟠 5 | **Investigate cross-border vehicle** | X3-63282 last seen in Uganda — confirm if authorised |
+| `idle_time_s` | ✓ | |
-| 🟠 6 | **Register webhooks** in Tracksolid Pro account | Activates: /pushobd, /pushoil, /pushtem, /pushlbs, /pushevent, /pushtripreport |
+| `fuel_consumed_l` | ✓ | |
 | 🟡 7 | **Populate vehicle_name, vehicle_number, driver_name** | All 63 devices currently blank — reports show device names only |
 | 🟡 8 | **Set fuel_100km** per vehicle | Unlocks idle fuel cost calculations |
 | 🟡 9 | **Review high-mileage vehicles** for service | KDK 829A GP (239k km) and Belta KCU-647D (234k km) |
 | 🟢 10 | **Schedule nightly ETL** | `SELECT dwh_gold.refresh_daily_metrics(CURRENT_DATE - 1);` via cron or n8n |
-### Commands for steps 1–3
+First trips expected ~06:30–07:00 EAT when drivers depart. Validate with:
-
+```sql
-```bash
+SELECT d.device_name, t.start_time AT TIME ZONE 'Africa/Nairobi',
-# SSH to server first
+       t.distance_km, t.avg_speed_kmh, t.max_speed_kmh, t.driving_time_s
-ssh kianiadee@stage.rahamafresh.com
+FROM tracksolid.trips t
-
+JOIN tracksolid.devices d ON d.imei = t.imei
-# Resolve container
+ORDER BY t.start_time DESC LIMIT 10;
 TS_DB=$(docker ps --filter "name=timescale_db" --format "{{.Names}}" | head -1)
 # Step 1 — migration 04 (distance correction)
 docker exec -i "$TS_DB" psql -U postgres -d tracksolid_db \
  < /path/to/04_bug_fix_migration.sql
 # Step 2 — migration 05 (new tables)
 docker exec -i "$TS_DB" psql -U postgres -d tracksolid_db \
  < /path/to/05_enhancement_migration.sql
 # Step 3 — redeploy containers
 cd /path/to/compose
 docker compose up -d --build ingest_movement ingest_events webhook_receiver
 ```
 ---
-## 9. Devices That Reported This Week vs Last Month
+## 10. Parking Events
-### Active in last 7 days
+**Parking events: 0**
-| Device Name | IMEI | Last Fix | Odometer |
+POLL-02 fix deployed (`acc_type=0`, corrected `durSecond` field mapping). The API is responding (60 calls, 0 rows) — events will populate once vehicles complete a full park-stop-move cycle.
 |---|---|---|---|
 | FRED KMGW 538W HULETI | 865135061559538 | 2026-04-10 23:16 | 2,284 km |
 | X3-63282 | 865135061563282 | 2026-04-10 23:15 | 4,194 km |
 | KDK 829A GP | 359857082898297 | 2026-04-10 23:13 | 239,264 km |
 | KMGR 409U HENRY JAZZ | 865135061048300 | 2026-04-10 15:40 | 6,696 km |
 | KCU 865Q Vanguard | 359857082042953 | 2026-04-08 17:17 | 61,758 km |
 | KDU 878T_Track | 865135061040349 | 2026-04-08 17:16 | 4,802 km |
 ### Active in last 30 days (in addition to above)
 | Device Name | IMEI | Last Fix | Odometer |
 |---|---|---|---|
 | JK Subaru KCS 903Y | 359857081891921 | 2026-03-14 00:55 | 73,344 km |
 | KDW 632M HL Tracker | 865135061569529 | 2026-03-11 23:53 | 222 km |
 | JC400P-85751 | 862798052785751 | 2026-03-11 22:15 | 17 km |
 | KDW 632M HL Cam | 862798052707995 | 2026-03-11 11:52 | 16 km |
 ---
-## 10. What This Report Will Look Like in 7 Days
+## 11. Ingestion Pipeline Health
-Once migrations 04 and 05 are applied and updated containers are deployed, the next weekly report will include:
+Total API calls across all endpoints since deployment (~08h operational):
- **Trip records** per vehicle per day — distance driven, drive/idle hours, avg and max speed
+| Endpoint | Calls | Rows Upserted | Rows Inserted | Avg Duration | Failures | First Call (EAT) | Last Call (EAT) |
- **Parking events** — where vehicles stopped, how long, address
+|---|---|---|---|---|---|---|---|
- **Alarm events** — overspeed, geofence, harshness flags with correct type names
+| `jimi.user.device.location.list` | 311 | 5,909 | 5,909 | 493ms | 0 | 2026-04-10 23:45 | 2026-04-11 07:38 |
- **High-resolution position trails** — 2–6 GPS fixes per minute per active vehicle
+| `jimi.open.platform.report.parking` | 60 | 0 | 0 | 10,891ms | 0 | 2026-04-10 23:45 | 2026-04-11 07:36 |
- **Driver scorecards** — km driven, alarms per 100 km, late starts
+| `jimi.device.track.list` | 15 | 0 | 80 | 136,875ms | 0 | 2026-04-10 23:45 | 2026-04-11 07:19 |
 | `jimi.device.alarm.list` | 11 | 0 | 11 | 939ms | 0 | 2026-04-11 01:00 | 2026-04-11 03:24 |
 | `jimi.user.device.list+detail` | 2 | 126 | 0 | 6,290ms | 0 | 2026-04-10 23:45 | 2026-04-11 05:00 |
 | **Total** | **399** | | | | **0** | | |
-The data foundation is in place. The pipeline is running. This baseline establishes the starting point against which all future performance will be measured.
+**Observations:**
 - **Zero failures across all 399 API calls** — pipeline fully stable.
 - Location polling: 311 calls × ~19 devices/call = consistent fleet coverage. Averaging 493ms round-trip.
 - Track list: 15 calls yielding 80 waypoints at 136s avg — the API is slow per call but returns full trail history correctly.
 - Parking: 60 calls, all successful, all 0 rows — API is healthy, no parking events completed yet.
 - Device sync (device.list+detail): runs every 5 hours; 2 runs complete, 126 device records synced.
 - Alarm polling: activates conditionally; 11 calls catching both overnight alarms correctly.
 ### Recent Call Log (last 5 minutes)
 Every 60s location sweep returning exactly 19 devices at ~200ms:
 ```
 07:38:13  location.list   63 queried → 19 upserted   230ms
 07:37:13  location.list   63 queried → 19 upserted   214ms
 07:36:12  location.list   63 queried → 19 upserted   223ms
 07:36:04  parking         13+50 queried → 0 inserted  14,562ms + 7,293ms
 07:35:11  location.list   63 queried → 19 upserted   195ms
 ```
 ---
-*Report generated from live database query · 2026-04-10 23:18 EAT*
+## 12. Odometer Leaders & Service Flags
-*Pipeline uptime at report time: ~1 hour*
+
-*Queries source: `tracksolid_DB_manual.md` · `01_BusinessAnalytics.md`*
+Devices with reported odometer readings (ascending by mileage):
 | Rank | Device | Model | SIM | Odometer (km) | Subscription Expires |
 |---|---|---|---|---|---|
 | 1 | **KDK 829A GP** | GT06E | 0707923872 | **239,264** | 2042-10-29 |
 | 2 | **Belta KCU-647D** | GT06E | 0110094465 | **234,546** | 2040-04-03 |
 | 3 | JK Subaru KCS 903Y | GT06E | 0746759925 | 73,345 | 2039-06-12 |
 | 4 | KCU 865Q Vanguard | GT06E | 0757270763 | 61,758 | 2039-12-20 |
 | 5 | KCU 145Q Solo Xtrail | GT06E | 0757270810 | 53,228 | 2039-12-20 |
 | 6 | KCU 865Q Vanguard Sub | AT4 | 0757270804 | 9,656 | 2039-12-20 |
 | 7 | JC400P-89431 | JC400P | — | 7,375 | — |
 | 8 | JC400P-86270 | JC400P | — | 6,931 | — |
 | 9 | KMGR 409U HENRY JAZZ | X3 | 0768697302 | 6,696 | 2035-07-31 |
 | 10 | KDU 878T_Track | X3 | 0708352823 | 4,802 | 2035-08-18 |
 | 11 | KCS 903Y JK SUB | AT4 | 0700024569 | 4,492 | 2039-06-09 |
 | 12 | X3-63282 | X3 | — | 4,194 | 2036-02-14 |
 | 13 | JC400P-91619 | JC400P | — | 3,719 | — |
 | 14 | KMEH 692C KAWASAKI | AT4 | 0110094467 | 3,319 | 2040-04-03 |
 | 15 | FRED KMGW 538W HULETI | X3 | 0119867174 | 2,284 | 2036-02-08 |
 > **KDK 829A GP (239,264 km) and Belta KCU-647D (234,546 km) are critical service flags.** Both exceed typical major maintenance thresholds. Both are GT06E OBD trackers — their odometer readings come directly from the vehicle's ECU, making them more reliable than GPS-estimated mileage. Recommend immediate physical inspection.
 ---
 ## 13. Subscription Status
 | Category | Count | Notes |
 |---|---|---|
 | No expiry date set | **44** | All are the 44 silent/no-position devices — likely unactivated |
 | Already expired | 0 | None expired |
 | Expiring within 90 days | 0 | None imminent |
 | Valid long-term (> 90 days) | **19** | All the devices with live positions |
 The 44 devices with no expiry are the same 44 with no position data. This strongly suggests they have never been activated on the Tracksolid platform (no SIM, no subscription, never commissioned).
 ---
 ## 14. Geographic Anomaly — X3-63282
 **X3-63282 is consistently reporting from Uganda (~0.196°N, 32.540°E) near Kampala.**
 Evidence this is real (not a GPS glitch):
 - 29 fixes over 24 hours, all from the same coordinates
 - Track_list high-res trail confirms 5 more waypoints at the same location
 - Device is actively pinging every 60 seconds (GPS signal quality: 4/4, 11 satellites)
 - Subscription is valid until 2036-02-14
 - Odometer reads 4,194 km
 **This device is genuinely located in Uganda and actively communicating.**
 Possible explanations:
 1. Vehicle on a cross-border logistics trip (Nairobi → Kampala corridor)
 2. Device removed and travelling separately from the vehicle
 3. Device installed in a vehicle belonging to a different entity
 **Action required:** Contact driver or responsible manager immediately to confirm vehicle location, mission, and expected return.
 ---
 ## 15. Silent Devices — Full List (44 devices)
 All 44 devices with no position data and no subscription expiry set:
 | Model | Devices |
 |---|---|
 | AT4 (18) | AT4-51820, AT4-53099, AT4-54246, AT4-55029, AT4-55235, AT4-57389, AT4-61860, AT4-64823, AT4-64880, AT4-64989, AT4-65010, AT4-65135, AT4-65341, AT4-65598, AT4-65648, AT4-66158, AT4-67271, AT4-67693 |
 | JC400P (20) | JC400P-07904, JC400P-85041, JC400P-85058, JC400P-86270, JC400P-86403, JC400P-87625, JC400P-87831, JC400P-89431, JC400P-89530, JC400P-89563, JC400P-89662, JC400P-89977, JC400P-90108, JC400P-90199, JC400P-90678, JC400P-91619, JC400P-92278, JC400P-92716, JC400P-92732, JC400P-94233 |
 | GT06E (2) | GT06E-85428, GT06E-86319 |
 | X3 (4) | X3-59405, X3-64223, X3-69172, X3-78553 |
 > **18 AT4 + 20 JC400P = 38 of the 44 silent devices are AT4s and JC400Ps** — the two largest model groups. This is a significant inventory of undeployed trackers. Either these are in storage awaiting vehicle installation, or they represent a large batch of devices purchased but not yet commissioned.
 ---
 ## 16. Schema Additions Confirmed (Migration 05)
 ### New columns on existing tables
 **`tracksolid.trips`:**
 - `distance_km` (renamed + corrected from `distance_m`)
 - `max_speed_kmh` ✓
 **`tracksolid.obd_readings`** — normalized scalar columns:
 - `engine_rpm`, `coolant_temp_c`, `fuel_level_pct`, `battery_voltage` ✓
 - `intake_pressure`, `throttle_pct`, `vehicle_speed`, `engine_load_pct` ✓
 **`tracksolid.alarms`** — enrichment:
 - `severity`, `geofence_id`, `geofence_name` ✓
 - `acknowledged_at`, `acknowledged_by` ✓
 **`tracksolid.devices`** — vehicle enrichment:
 - `vehicle_category`, `cost_centre`, `assigned_route` ✓
 - `depot_geom`, `depot_address` ✓
 ### New tables confirmed present
 | Table | Type | Purpose |
 |---|---|---|
 | `tracksolid.device_events` | Regular | Login/logout events (PUSH-01) |
 | `tracksolid.fuel_readings` | Hypertable | Fuel sensor readings (PUSH-02) |
 | `tracksolid.temperature_readings` | Hypertable | Temperature/humidity (PUSH-03) |
 | `tracksolid.lbs_readings` | Regular | Cell tower fallback positions (PUSH-04) |
 | `tracksolid.geofences` | Regular | Geofence zone storage |
 ### DWH Gold layer
 | Object | Status |
 |---|---|
 | `dwh_gold.dim_vehicles` | Present, 0 rows |
 | `dwh_gold.fact_daily_fleet_metrics` | Present, 0 rows — ETL function not yet run |
 ---
 ## 17. Data Quality Gaps — Priority Matrix
 ### P0 — Blocks all driver / vehicle reporting
 | Gap | Scope | Action |
 |---|---|---|
 | `vehicle_number`, `vehicle_name`, `driver_name` blank | All 63 devices | Bulk populate via Tracksolid Pro admin or API |
 | 44 devices never activated (no SIM, no position) | 70% of fleet | Commission or confirm storage status per device |
 ### P1 — Blocks operational analytics
 | Gap | Scope | Action |
 |---|---|---|
 | No push webhooks registered | OBD, fuel, temp, events, LBS all empty | Register 6 endpoints in Tracksolid Pro |
 | Parking: 0 rows despite 60 API calls | Cannot compute idle time or utilisation | Validate post-morning operations |
 | Trips: 0 rows | All KPIs unavailable | Validate first morning trips >07:00 EAT |
 | `fuel_100km` not set | No cost-per-trip or fuel efficiency | Set per vehicle class |
 ### P2 — Limits reporting depth
 | Gap | Scope | Action |
 |---|---|---|
 | No geofences configured | No zone detection, no off-route alerts | Create depot + key-site geofences |
 | `vehicle_category` blank | Cannot segment by vehicle type | Populate alongside vehicle_number |
 | `cost_centre` blank | Cannot allocate costs to business units | Populate alongside vehicle_number |
 | `depot_geom` blank | Cannot compute distance-from-base | Set primary depot coordinates |
 | DWH gold empty | Grafana summary panels empty | Run `dwh_gold.refresh_daily_metrics()` after first full day |
 ---
 ## 18. Grafana Dashboard Readiness
 | Panel | Ready | Condition |
 |---|---|---|
 | Fleet map — live positions | **Partial** | 19/63 visible; 44 offline |
 | Position trail / route replay | **Ready** | 137 fixes, growing each cycle |
 | Alarm feed | **Ready** | Live; will grow with operations |
 | Active device count | **Ready** | 4 devices active in last 24h |
 | Trips today | **Not ready** | Validate post-07:00 EAT |
 | Distance per vehicle | **Not ready** | Requires trips data |
 | Driver behaviour (speed, harsh events) | **Not ready** | Requires trips + driver metadata |
 | Fuel efficiency | **Not ready** | Requires fuel_100km + OBD/fuel data |
 | Device connectivity uptime | **Not ready** | Requires `/pushevent` webhook |
 | Cold chain temperature | **Not ready** | Requires `/pushtem` webhook |
 | Odometer / service alerts | **Ready** | 15 devices with odometer data |
 ---
 ## 19. Recommended Actions (Priority Order)
 1. **Populate vehicle metadata** — `vehicle_number`, `vehicle_name`, `driver_name` for all 63 devices. Single most impactful action — unlocks driver scoring, trip attribution, and all KPIs from `01_BusinessAnalytics.md`.
 2. **Investigate X3-63282 in Uganda** — Contact driver/manager today. Confirm cross-border mission or escalate as potential asset displacement.
 3. **Schedule morning data validation** — At 09:00 EAT run trip, parking, and alarm queries to confirm pipeline behaviour during active operations.
 4. **Register push webhooks** in Tracksolid Pro:
   - `/pushtripreport`
   - `/pushobd`
   - `/pushoil`
   - `/pushtem`
   - `/pushlbs`
   - `/pushevent`
 5. **Audit 44 silent devices** — Determine which are: (a) in storage awaiting installation, (b) installed but SIM not provisioned, (c) decommissioned. Log the outcome per IMEI.
 6. **Flag KDK 829A GP and Belta KCU-647D for service** — 239k km and 234k km respectively. Both actively communicating and in use.
 7. **Set `fuel_100km`** per vehicle class to enable cost reporting.
 8. **Configure primary depot geofence** at ~(-1.237, 36.727) — the tight cluster where most of the fleet is parked.
 9. **Run DWH gold ETL** after first full operational day:
   ```sql
   SELECT dwh_gold.refresh_daily_metrics(CURRENT_DATE - 1);
   ```
 ---
 *End of report. Data queried: 2026-04-11 ~07:38 EAT. Next full refresh recommended: 2026-04-11 12:00 EAT after morning operations complete.*
--- a/tracksolidApiDocumentation.md
+++ b/tracksolidApiDocumentation.md
@ -2,7 +2,7 @@
 > Consolidated from official Jimi IoT documentation at [tracksolidprodocs.jimicloud.com](https://tracksolidprodocs.jimicloud.com/) and [docs.jimicloud.com](https://docs.jimicloud.com/integration/integration.html).
 >
-> API Spec Version: 2.7.7 | Last updated: 2026-04-08
+> API Spec Version: 2.7.7 | Last updated: 2026-04-11
 ---
@ -81,6 +81,7 @@ Every API call includes these base parameters:
 - Token acquisition (`jimi.oauth.token.get`) can be called **at most once per minute**
 - `jimi.device.alarm.list` — time range limited to **1 month**, max **1000 rows** returned
 - Batch endpoints accept up to **50 IMEIs** per call
 - `jimi.device.track.list` — per-device endpoint (no batch); each call can be slow (~137s observed in production for a 35-minute lookback). Do not call synchronously inside a 60s scheduler loop; run on a dedicated 30-minute schedule.
 ---
@ -440,14 +441,26 @@ Retrieve latest positions for all devices under an account. Single API call for
 |---|---|---|
 | `imei` | string | Device IMEI |
 | `deviceName` | string | Device name |
-| `status` | string | Device status |
+| `status` | string | Device status code |
 | `lat` | double | Latitude |
 | `lng` | double | Longitude |
 | `speed` | number | Speed (km/h) |
 | `direction` | number | Heading (0–360 degrees) |
 | `gpsTime` | string | GPS fix timestamp |
-| `accStatus` | string | ACC ignition status |
+| `hbTime` | string | Heartbeat/server receive time |
 | `accStatus` | string | ACC ignition status (`0`=off, `1`=on) |
 | `gpsSignal` | int | GPS signal quality (0–4) |
 | `gpsNum` | int | Satellites used |
 | `currentMileage` | number | Odometer reading |
-| `expireFlag` | string | Expiration flag |
+| `powerValue` | number | External power voltage |
 | `elecQuantity` | number | Battery percentage |
 | `posType` | string | Position type |
 | `confidence` | int | Position confidence score |
 | `expireFlag` | string | Subscription expiration flag |
 | `activationFlag` | string | Activation status |
 | `locDesc` | string | Reverse-geocoded address (if available) |
 > **Implementation note:** This is the fleet sweep endpoint called every 60 seconds. In production it returns ~19 active devices out of 63 queried, averaging ~200ms per call. Devices without a current position are silently omitted from the response — they are not returned with null coordinates.
 ---
@ -532,9 +545,15 @@ Extract trip and distance records within a time period. Supports batching up to
 | `imei` | string | Device IMEI |
 | `startTime` | string | Trip start time |
 | `endTime` | string | Trip end time |
-| `distance` | number | Distance in kilometers |
+| `distance` | number | Distance — **stored as kilometres in DB** (see note below) |
-| `avgSpeed` | number | Average speed (km/h) |
+| `avgSpeed` | number | Average speed (km/h) — field name: `avgSpeed` |
-| `runTimeSecond` | int | Trip duration in seconds |
+| `maxSpeed` | number | Maximum speed (km/h) — field name: `maxSpeed` |
 | `runTimeSecond` | int | Driving time (seconds) |
 | `idleSecond` | int | Idle time (seconds) |
 > **Distance unit note (BUG-02):** The raw `distance` value returned by this endpoint is in **kilometres** as labelled. However, the value was being stored incorrectly as millimetres due to an erroneous `× 1000` multiplication in earlier code. Migration 04 corrected all historical rows: `distance_km = stored_value / 1,000,000`. Current code stores `distance` directly as `distance_km` with no further conversion. The DB column was renamed from `distance_m` to `distance_km` to reflect this.
 >
 > **maxSpeed note (BUG-03):** `maxSpeed` was present in the API response but not mapped in `poll_trips()`. Fixed — now stored in `tracksolid.trips.max_speed_kmh`.
 ---
@ -564,10 +583,14 @@ Retrieve detailed positional waypoints for a specified timeframe. **Per-device**
 | `lng` | double | Longitude |
 | `gpsTime` | string | Timestamp of GPS fix |
 | `gpsSpeed` | int | Speed (km/h) |
-| `direction` | int | Heading (0-360 degrees) |
+| `direction` | int | Heading (0–360 degrees) |
 | `ignition` | string | Ignition state |
 | `accStatus` | string | ACC status |
 | `confidence` | int | Position confidence |
 | `altitude` | int | Altitude (metres) — not populated by all device models |
 | `satellite` | int | Satellite count |
 > **Implementation note (POLL-01):** Called per-device every 30 minutes with a 35-minute lookback window. Results written to `tracksolid.position_history` with `source='track_list'` (vs `source='poll'` for the 60s sweep). Use `ON CONFLICT DO NOTHING` — duplicate fixes from overlapping windows are safely discarded. Observed: ~5–6 waypoints per active device per 30-min window. Altitude field is returned but not populated by X3 and GT06E devices in this fleet.
 ---
@ -617,7 +640,7 @@ Analyze stationary and engine-running periods.
 | `end_time` | string | Yes | End time |
 | `start_row` | int | Yes | Pagination offset |
 | `page_size` | int | Yes | Page size |
-| `acc_type` | string | Yes | ACC filter type |
+| `acc_type` | string | Yes | ACC filter: **`0`** = all stops, `1` = ignition-off only |
 **Response (array):**
@ -626,10 +649,12 @@ Analyze stationary and engine-running periods.
 | `imei` | string | Device IMEI |
 | `startTime` | string | Parking/idle start |
 | `endTime` | string | Parking/idle end |
-| `durSecond` | int | Total duration (seconds) |
+| `durSecond` | int | Total duration (seconds) — **use this field for duration** |
 | `stopSecond` | int | Stationary-only duration (seconds) — may be absent |
 | `addr` | string | Address / location |
 | `deviceName` | string | Device name |
-| `stopSecond` | int | Stationary duration (seconds) |
+
 > **Implementation note (POLL-02):** Pass `acc_type=0` to capture all stop events (not just ignition-off). If `acc_type` is omitted or set to an empty string the API returns 0 rows. The primary duration field is `durSecond`; `stopSecond` is a secondary field that may not be populated. In production: 60 calls logged, all successful, 0 rows — expected while fleet is overnight-parked. Will populate once vehicles complete a full stop–start–stop cycle.
 ---
@ -690,16 +715,24 @@ Retrieve alarm events for devices within a time range.
 | `imei` | string | Device IMEI |
 | `model` | string | Device model |
 | `account` | string | Account |
-| `alertTypeId` | string | Alarm type identifier |
+| `alertTypeId` | string | Alarm type identifier — **polling field name** |
-| `alarmTypeName` | string | Alarm type display name |
+| `alarmTypeName` | string | Alarm type display name — **polling field name** |
-| `alertTime` | string | Alarm trigger time |
+| `alertTime` | string | Alarm trigger time — **polling field name** |
 | `positioningTime` | string | GPS fix time at alarm |
 | `lat` | double | Latitude |
 | `lng` | double | Longitude |
 | `speed` | number | Speed at alarm time |
 | `geoid` | string | Geo-fence ID (if geofence alarm) |
-> **Note:** The documented response field names (`alertTypeId`, `alertTime`) may differ from what some code examples use (`alarmType`, `alarmTime`). Always verify against actual API responses.
+> **Critical field name difference — polling vs push (BUG-01):**
 >
 > | Data | Polling (`jimi.device.alarm.list`) | Push (`/pushalarm`) | DB column |
 > |---|---|---|---|
 > | Type ID | `alertTypeId` | `alarmType` | `alarm_type` |
 > | Type name | `alarmTypeName` | `alarmName` | `alarm_name` |
 > | Time | `alertTime` | `gateTime` | `alarm_time` |
 >
 > Earlier code used the push field names (`alarmType`, `alarmName`) in the polling path — these keys are never present in the polling response, so `alarm_type` and `alarm_name` were always NULL. Fixed in `ingest_events_rev.py` (FIX-E06): polling now maps `item.get('alertTypeId')` → `alarm_type` and `item.get('alarmTypeName')` → `alarm_name`. **Verified in production:** vibration alarms now store type `3` and name `"Vibration alert"` correctly.
 ---
@ -1217,6 +1250,8 @@ You must configure your callback URL in the Tracksolid Pro platform. All push en
 | Endpoint | `{YourURL}/pushevent` |
 |---|---|
 | **Handler** | `webhook_receiver_rev.py` — implemented (PUSH-01) |
 | **DB table** | `tracksolid.device_events` |
 **`data_list` fields:**
@ -1233,6 +1268,7 @@ You must configure your callback URL in the Tracksolid Pro platform. All push en
 | Endpoint | `{YourURL}/pushhb` |
 |---|---|
 | **Handler** | Not yet implemented — DB table `tracksolid.heartbeats` exists and is ready |
 **`data_list` fields (max 50 per request):**
@ -1278,6 +1314,10 @@ You must configure your callback URL in the Tracksolid Pro platform. All push en
 | Endpoint | `{YourURL}/pushalarm` |
 |---|---|
 | **Handler** | `webhook_receiver_rev.py` — implemented |
 | **DB table** | `tracksolid.alarms` |
 > **Field name note:** Push uses `alarmType` and `alarmName` — different from the polling endpoint which uses `alertTypeId` and `alarmTypeName`. See Section 6.1 for the full mapping table.
 **`data_list` fields (max 50 per request):**
@ -1333,6 +1373,8 @@ You must configure your callback URL in the Tracksolid Pro platform. All push en
 | Endpoint | `{YourURL}/pushoil` |
 |---|---|
 | **Handler** | `webhook_receiver_rev.py` — implemented (PUSH-02) |
 | **DB table** | `tracksolid.fuel_readings` (TimescaleDB hypertable) |
 **`data_list` fields (max 50 per request):**
@ -1342,7 +1384,7 @@ You must configure your callback URL in the Tracksolid Pro platform. All push en
 | `path` | int | Sensor ID |
 | `gateTime` | string | Reading time |
 | `value` | double | Oil level (divide by 100 for actual value) |
-| `unit` | int | 1=cm, 2=%, 3=V, 4=L |
+| `unit` | int | `1`=cm, `2`=%, `3`=V, `4`=L — stored as text label in DB |
 | `gpsTime` | string | Optional GPS time |
 | `lng`, `lat` | double | Optional GPS position |
@ -1368,6 +1410,8 @@ You must configure your callback URL in the Tracksolid Pro platform. All push en
 | Endpoint | `{YourURL}/pushtem` |
 |---|---|
 | **Handler** | `webhook_receiver_rev.py` — implemented (PUSH-03) |
 | **DB table** | `tracksolid.temperature_readings` (TimescaleDB hypertable) |
 **`data_list` fields:**
@ -1385,6 +1429,8 @@ You must configure your callback URL in the Tracksolid Pro platform. All push en
 | Endpoint | `{YourURL}/pushlbs` |
 |---|---|
 | **Handler** | `webhook_receiver_rev.py` — implemented (PUSH-04) |
 | **DB table** | `tracksolid.lbs_readings` |
 **`data_list` fields:**
@ -1552,8 +1598,10 @@ Called when a device finishes uploading media (photo, video, event clip) to Jimi
 | Endpoint | `{YourURL}/pushobd` |
 |---|---|
 | **Handler** | `webhook_receiver_rev.py` — implemented |
 | **DB table** | `tracksolid.obd_readings` |
-> **Important:** This is the only documented method for receiving OBD data. There is no polling/pull endpoint for OBD.
+> **Important:** This is the only documented method for receiving OBD data. There is no polling/pull endpoint for OBD. The handler extracts scalar fields (`engine_rpm`, `coolant_temp_c`, `fuel_level_pct`, `battery_voltage`, `intake_pressure`, `throttle_pct`, `vehicle_speed`, `engine_load_pct`) from `obdJson` using well-known OBD PID dataID keys, and also stores the full `obdJson` in the `obd_data JSONB` column.
 **`data_list` fields:**
@ -1576,6 +1624,7 @@ Called when a device finishes uploading media (photo, video, event clip) to Jimi
 | Endpoint | `{YourURL}/pushfaultinfo` |
 |---|---|
 | **Handler** | Not yet implemented — DB table `tracksolid.fault_codes` exists and is ready |
 > **Important:** This is the only documented method for receiving DTC fault codes.
@ -1598,6 +1647,8 @@ Called when a device finishes uploading media (photo, video, event clip) to Jimi
 | Endpoint | `{YourURL}/pushtripreport` |
 |---|---|
 | **Handler** | `webhook_receiver_rev.py` — implemented |
 | **DB table** | `tracksolid.trips` |
 **`data_list` fields:**
@ -1647,19 +1698,39 @@ Common parameters: `deviceImei`, `proNo` (protocol number), `cmdContent` (JSON),
 ## Appendix B: API Coverage in This Codebase
-| API Method | Pipeline | Status |
+### Polling (Pull) Endpoints
-|---|---|---|
+
-| `jimi.oauth.token.get` | `ts_shared_rev.py` | In use |
+| API Method | File | Status | Notes |
-| `jimi.oauth.token.refresh` | `ts_shared_rev.py` | In use |
+|---|---|---|---|
-| `jimi.user.device.list` | `ingest_movement_rev.py` | In use |
+| `jimi.oauth.token.get` | `ts_shared_rev.py` | ✅ In use | Token auto-refreshed, cached in DB |
-| `jimi.track.device.detail` | `ingest_movement_rev.py` | In use |
+| `jimi.oauth.token.refresh` | `ts_shared_rev.py` | ✅ In use | |
-| `jimi.user.device.location.list` | `ingest_movement_rev.py` | In use |
+| `jimi.user.device.list` | `ingest_movement_rev.py` | ✅ In use | Fleet sync every 5h; 2 runs, 126 devices synced |
-| `jimi.device.track.mileage` | `ingest_movement_rev.py` | In use |
+| `jimi.track.device.detail` | `ingest_movement_rev.py` | ✅ In use | Called alongside device.list |
-| `jimi.device.alarm.list` | `ingest_events_rev.py` | In use — field mapping corrected (FIX-E06) |
+| `jimi.user.device.location.list` | `ingest_movement_rev.py` | ✅ In use | Every 60s; 311 calls, 5,909 upserts, 0 failures |
-| `jimi.device.obd.list` | `ingest_events_rev.py` | **Not a real endpoint** — OBD is push-only |
+| `jimi.device.track.mileage` | `ingest_movement_rev.py` | ✅ In use | Trip polling; `maxSpeed` now mapped (BUG-03 fixed) |
-| `jimi.device.track.list` | `ingest_movement_rev.py` | **In use** — poll_track_list() every 30m (FIX-M14) |
+| `jimi.device.track.list` | `ingest_movement_rev.py` | ✅ In use | `poll_track_list()` every 30m (POLL-01 / FIX-M14); 15 calls, 80 waypoints |
-| `jimi.device.location.get` | `ingest_movement_rev.py` | **In use** — get_device_locations() on-demand (FIX-M15) |
+| `jimi.device.location.get` | `ingest_movement_rev.py` | ✅ In use | `get_device_locations()` on-demand batch refresh (POLL-03 / FIX-M15); max 50 IMEIs/call |
-| `jimi.open.platform.report.parking` | `ingest_movement_rev.py` | **In use** — acc_type/durSecond fixed (FIX-M13) |
+| `jimi.open.platform.report.parking` | `ingest_movement_rev.py` | ✅ In use | `acc_type=0`, `durSecond` corrected (POLL-02 / FIX-M13); 60 calls, 0 rows (fleet overnight) |
-| `jimi.device.jimi.media.URL` | — | Not used (media catalog) |
+| `jimi.device.alarm.list` | `ingest_events_rev.py` | ✅ In use | `alertTypeId`/`alarmTypeName` mapping fixed (BUG-01 / FIX-E06); 11 calls, 11 rows |
-| `jimi.device.media.event.URL` | — | Not used (alarm-triggered media) |
+| `jimi.device.obd.list` | — | ❌ Does not exist | OBD data is push-only via `/pushobd` |
-| All Data Push endpoints | — | Not used (webhook receiver needed) |
+| `jimi.device.jimi.media.URL` | — | Not used | Media file catalog |
 | `jimi.device.media.event.URL` | — | Not used | Alarm-triggered media |
 | All other geofence/command/media polling | — | Not used | Available when needed |
 ### Push (Webhook) Endpoints
 | Endpoint | File | Status | DB Table |
 |---|---|---|---|
 | `/pushalarm` | `webhook_receiver_rev.py` | ✅ Implemented | `tracksolid.alarms` |
 | `/pushtripreport` | `webhook_receiver_rev.py` | ✅ Implemented | `tracksolid.trips` |
 | `/pushobd` | `webhook_receiver_rev.py` | ✅ Implemented | `tracksolid.obd_readings` |
 | `/pushevent` | `webhook_receiver_rev.py` | ✅ Implemented (PUSH-01) | `tracksolid.device_events` |
 | `/pushoil` | `webhook_receiver_rev.py` | ✅ Implemented (PUSH-02) | `tracksolid.fuel_readings` |
 | `/pushtem` | `webhook_receiver_rev.py` | ✅ Implemented (PUSH-03) | `tracksolid.temperature_readings` |
 | `/pushlbs` | `webhook_receiver_rev.py` | ✅ Implemented (PUSH-04) | `tracksolid.lbs_readings` |
 | `/pushhb` | — | ⚠️ Not yet wired | `tracksolid.heartbeats` table ready |
 | `/pushfaultinfo` | — | ⚠️ Not yet wired | `tracksolid.fault_codes` table ready |
 | `/pushgps` | — | Not used | GPS data received via polling |
 | All other push endpoints | — | Not used | |
 > **Registration status:** All implemented webhook endpoints need to be registered in the Tracksolid Pro dashboard with your server's public URL before they will receive data. Tables exist and are ready; rows will be 0 until registration is complete.
--- a/ts_shared_rev.py
+++ b/ts_shared_rev.py
@ -231,10 +231,14 @@ def get_active_imeis() -> list[str]:
 def log_ingestion(cur, endpoint: str, imei_count: int, upserted: int, inserted: int, duration_ms: int, success: bool, error_code: str = None, error_msg: str = None):
    cur.execute("""
-        INSERT INTO tracksolid.ingestion_log 
+        INSERT INTO tracksolid.ingestion_log
        (endpoint, imei_count, rows_upserted, rows_inserted, duration_ms, success, error_code, error_message)
        VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
-    """, (endpoint[:100], imei_count, upserted, inserted, duration_ms, success, str(error_code)[:50], str(error_msg)[:500]))
+    """, (
        endpoint[:100], imei_count, upserted, inserted, duration_ms, success,
        str(error_code)[:50] if error_code is not None else None,
        str(error_msg)[:500] if error_msg is not None else None,
    ))
 # ── Token Management ──────────────────────────────────────────────────────────
--- a/webhook_receiver_rev.py
+++ b/webhook_receiver_rev.py
@ -15,7 +15,15 @@ ENDPOINTS:
  /pushgps          — GPS positions (Priority 2)
  /pushhb           — Device heartbeats (Priority 2)
  /pushtripreport   — Trip reports (Priority 2)
  /pushevent        — Device events (Priority 3, log-only)
  /health           — Healthcheck for Docker/monitoring
 REVISIONS (QA-Verified):
  [BUG-01] OBD event_time: try unix_to_ts before clean_ts (handles epoch timestamps).
  [BUG-02] push_alarm: guard also checks alarm_type is not null (prevents FK violation).
  [BUG-03] push_trip_report: _parse_trip_ts handles Jimi BCD format YYMMDDHHmmss.
  [BUG-04] SAVEPOINT per item in all DB-writing endpoints (one bad item won't abort batch).
  [BUG-05] Added /pushevent endpoint (log-only, prevents Jimi 404 errors).
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 """
@ -53,7 +61,7 @@ WEBHOOK_TOKEN = os.getenv("JIMI_WEBHOOK_TOKEN", "")
@asynccontextmanager
 async def lifespan(app: FastAPI):
-    log.info("Webhook receiver starting (v1.0)...")
+    log.info("Webhook receiver starting (v1.1)...")
    yield
    log.info("Webhook receiver shutting down...")
    close_pool()
@ -96,6 +104,25 @@ def unix_to_ts(v) -> Optional[str]:
        return None
 def _parse_trip_ts(v) -> Optional[str]:
    """[BUG-03] Parse trip timestamps. Handles ISO strings and Jimi BCD formats."""
    iso = clean_ts(v)
    if iso:
        return iso
    s = clean(v)
    if s is None:
        return None
    try:
        if len(s) == 12:   # YYMMDDHHmmss
            return datetime.strptime(s, "%y%m%d%H%M%S").strftime("%Y-%m-%d %H:%M:%S")
        if len(s) == 14:   # YYYYMMDDHHmmss
            return datetime.strptime(s, "%Y%m%d%H%M%S").strftime("%Y-%m-%d %H:%M:%S")
    except (ValueError, TypeError):
        pass
    log.warning("Cannot parse trip timestamp: %r", v)
    return None
 def _make_geom_params(lat, lng):
    """Return (lng, lat, lng, lat) tuple for the CASE WHEN ST_MakePoint pattern."""
    return (lng, lat, lng, lat)
@ -122,6 +149,7 @@ def push_obd(token: str = Form(""), data_list: str = Form("")):
        with conn.cursor() as cur:
            for item in items:
                try:
                    cur.execute("SAVEPOINT sp")
                    imei = clean(item.get("deviceImei"))
                    obd = item.get("obdJson", {})
                    if isinstance(obd, str):
@ -130,8 +158,13 @@ def push_obd(token: str = Form(""), data_list: str = Form("")):
                        except json.JSONDecodeError:
                            obd = {}
-                    event_time = clean_ts(obd.get("event_time"))
+                    # [BUG-01] Try unix epoch first, fall back to ISO string.
                    event_time = (
                        unix_to_ts(obd.get("event_time"))
                        or clean_ts(obd.get("event_time"))
                    )
                    if not imei or not event_time:
                        cur.execute("RELEASE SAVEPOINT sp")
                        continue
                    lat = clean_num(obd.get("lat"))
@ -159,13 +192,14 @@ def push_obd(token: str = Form(""), data_list: str = Form("")):
                        *_make_geom_params(lat, lng),
                        json.dumps(obd),
                    ))
                    cur.execute("RELEASE SAVEPOINT sp")
                    inserted += 1
                except Exception:
                    cur.execute("ROLLBACK TO SAVEPOINT sp")
                    log.warning("Failed to process OBD item for %s", item.get("deviceImei"), exc_info=True)
            log_ingestion(cur, "webhook/pushobd", len(items), 0, inserted,
                          int((time.time() - t0) * 1000), True)
        conn.commit()
    log.info("pushobd: %d/%d items processed.", inserted, len(items))
    return JSONResponse(content=SUCCESS)
@ -186,9 +220,11 @@ def push_fault_info(token: str = Form(""), data_list: str = Form("")):
        with conn.cursor() as cur:
            for item in items:
                try:
                    cur.execute("SAVEPOINT sp")
                    imei = clean(item.get("deviceImei"))
                    gate_time = clean_ts(item.get("gateTime"))
                    if not imei or not gate_time:
                        cur.execute("RELEASE SAVEPOINT sp")
                        continue
                    fault_codes = item.get("faultCodeList", [])
@ -203,6 +239,11 @@ def push_fault_info(token: str = Form(""), data_list: str = Form("")):
                    evt_time = unix_to_ts(item.get("eventTime")) or clean_ts(item.get("eventTime"))
                    for code in fault_codes:
                        fault_code = clean(code)
                        # Guard NULL: ON CONFLICT won't deduplicate NULL fault_codes
                        # because NULL != NULL in Postgres unique constraints.
                        if not fault_code:
                            continue
                        cur.execute("""
                            INSERT INTO tracksolid.fault_codes (
                                imei, reported_at, fault_code, status_flags,
@ -215,19 +256,20 @@ def push_fault_info(token: str = Form(""), data_list: str = Form("")):
                                %s
                            ) ON CONFLICT (imei, reported_at, fault_code) DO NOTHING
                        """, (
-                            imei, gate_time, clean(code),
+                            imei, gate_time, fault_code,
                            clean_int(item.get("statusFlags")),
                            lat, lng,
                            *_make_geom_params(lat, lng),
                            evt_time,
                        ))
                        inserted += 1
                    cur.execute("RELEASE SAVEPOINT sp")
                except Exception:
                    cur.execute("ROLLBACK TO SAVEPOINT sp")
                    log.warning("Failed to process fault item for %s", item.get("deviceImei"), exc_info=True)
            log_ingestion(cur, "webhook/pushfaultinfo", len(items), 0, inserted,
                          int((time.time() - t0) * 1000), True)
        conn.commit()
    log.info("pushfaultinfo: %d fault codes from %d items.", inserted, len(items))
    return JSONResponse(content=SUCCESS)
@ -248,10 +290,13 @@ def push_alarm(token: str = Form(""), data_list: str = Form("")):
        with conn.cursor() as cur:
            for item in items:
                try:
                    cur.execute("SAVEPOINT sp")
                    imei = clean(item.get("deviceImei"))
                    alarm_type = clean(item.get("alarmType"))
                    alarm_time = clean_ts(item.get("gateTime"))
-                    if not imei or not alarm_time:
+                    # [BUG-02] Also guard alarm_type — NULL alarm_type violates NOT NULL constraint.
                    if not imei or not alarm_time or not alarm_type:
                        cur.execute("RELEASE SAVEPOINT sp")
                        continue
                    lat = clean_num(item.get("lat"))
@ -274,13 +319,14 @@ def push_alarm(token: str = Form(""), data_list: str = Form("")):
                        lat, lng,
                        clean_num(item.get("speed")),
                    ))
                    cur.execute("RELEASE SAVEPOINT sp")
                    inserted += 1
                except Exception:
                    cur.execute("ROLLBACK TO SAVEPOINT sp")
                    log.warning("Failed to process alarm for %s", item.get("deviceImei"), exc_info=True)
            log_ingestion(cur, "webhook/pushalarm", len(items), 0, inserted,
                          int((time.time() - t0) * 1000), True)
        conn.commit()
    log.info("pushalarm: %d/%d items processed.", inserted, len(items))
    return JSONResponse(content=SUCCESS)
@ -301,12 +347,14 @@ def push_gps(token: str = Form(""), data_list: str = Form("")):
        with conn.cursor() as cur:
            for item in items:
                try:
                    cur.execute("SAVEPOINT sp")
                    imei = clean(item.get("deviceImei"))
                    gps_time = clean_ts(item.get("gpsTime"))
                    lat = clean_num(item.get("lat"))
                    lng = clean_num(item.get("lng"))
                    if not imei or not gps_time or not is_valid_fix(lat, lng):
                        cur.execute("RELEASE SAVEPOINT sp")
                        continue
                    cur.execute("""
@ -329,13 +377,14 @@ def push_gps(token: str = Form(""), data_list: str = Form("")):
                        clean_num(item.get("altitude")),
                        clean_int(item.get("postType")),
                    ))
                    cur.execute("RELEASE SAVEPOINT sp")
                    inserted += 1
                except Exception:
                    cur.execute("ROLLBACK TO SAVEPOINT sp")
                    log.warning("Failed to process GPS for %s", item.get("deviceImei"), exc_info=True)
            log_ingestion(cur, "webhook/pushgps", len(items), 0, inserted,
                          int((time.time() - t0) * 1000), True)
        conn.commit()
    log.info("pushgps: %d/%d items processed.", inserted, len(items))
    return JSONResponse(content=SUCCESS)
@ -356,9 +405,11 @@ def push_heartbeat(token: str = Form(""), data_list: str = Form("")):
        with conn.cursor() as cur:
            for item in items:
                try:
                    cur.execute("SAVEPOINT sp")
                    imei = clean(item.get("deviceImei"))
                    gate_time = clean_ts(item.get("gateTime"))
                    if not imei or not gate_time:
                        cur.execute("RELEASE SAVEPOINT sp")
                        continue
                    cur.execute("""
@ -375,13 +426,14 @@ def push_heartbeat(token: str = Form(""), data_list: str = Form("")):
                        clean_int(item.get("powerStatus")),
                        clean_int(item.get("fortify")),
                    ))
                    cur.execute("RELEASE SAVEPOINT sp")
                    inserted += 1
                except Exception:
                    cur.execute("ROLLBACK TO SAVEPOINT sp")
                    log.warning("Failed to process heartbeat for %s", item.get("deviceImei"), exc_info=True)
            log_ingestion(cur, "webhook/pushhb", len(items), 0, inserted,
                          int((time.time() - t0) * 1000), True)
        conn.commit()
    log.info("pushhb: %d/%d items processed.", inserted, len(items))
    return JSONResponse(content=SUCCESS)
@ -402,14 +454,16 @@ def push_trip_report(token: str = Form(""), data_list: str = Form("")):
        with conn.cursor() as cur:
            for item in items:
                try:
                    cur.execute("SAVEPOINT sp")
                    imei = clean(item.get("deviceImei"))
-                    begin_time = clean_ts(item.get("beginTime"))
+                    # [BUG-03] Use _parse_trip_ts to handle Jimi BCD format YYMMDDHHmmss.
-                    end_time = clean_ts(item.get("endTime"))
+                    begin_time = _parse_trip_ts(item.get("beginTime"))
                    end_time = _parse_trip_ts(item.get("endTime"))
                    if not imei or not begin_time:
                        cur.execute("RELEASE SAVEPOINT sp")
                        continue
-                    # [FIX-M11] API sends miles (km). Store directly as distance_km.
+                    # [FIX-M11] API sends km. Store directly as distance_km.
                    # Previous code multiplied by 1000, producing mm not m.
                    distance_km = clean_num(item.get("miles"))
                    begin_lat = clean_num(item.get("beginLat"))
@ -447,13 +501,26 @@ def push_trip_report(token: str = Form(""), data_list: str = Form("")):
                        clean_int(item.get("idleTimes")),
                        clean_int(item.get("tripSeq")),
                    ))
                    cur.execute("RELEASE SAVEPOINT sp")
                    inserted += 1
                except Exception:
                    cur.execute("ROLLBACK TO SAVEPOINT sp")
                    log.warning("Failed to process trip for %s", item.get("deviceImei"), exc_info=True)
            log_ingestion(cur, "webhook/pushtripreport", len(items), 0, inserted,
                          int((time.time() - t0) * 1000), True)
        conn.commit()
    log.info("pushtripreport: %d/%d items processed.", inserted, len(items))
    return JSONResponse(content=SUCCESS)
 # ── 7. Device Events (Priority 3 — log only) ─────────────────────────────────
@app.post("/pushevent")
 def push_event(token: str = Form(""), data_list: str = Form("")):
    """[BUG-05] Accept Jimi event pushes so they don't 404. Log for future schema work."""
    _validate_token(token)
    items = _parse_data_list(data_list)
    for item in items:
        log.info("pushevent: imei=%s type=%s gateTime=%s",
                 item.get("deviceImei"), item.get("type"), item.get("gateTime"))
    return JSONResponse(content=SUCCESS)
Author	SHA1	Message	Date
David Kiania	b1e4d6e85f	Fix 5 webhook bugs: SAVEPOINTs, NULL guards, BCD timestamps, /pushevent, log NULL fix BUG-01: OBD event_time — try unix_to_ts before clean_ts (Jimi sends epoch ints) BUG-02: push_alarm — guard alarm_type not null (NULL breaks ON CONFLICT dedup) BUG-03: push_trip_report — _parse_trip_ts handles Jimi BCD format YYMMDDHHmmss BUG-04: SAVEPOINT per item in all 5 DB endpoints (FK violation on one item no longer aborts the whole batch; SAVEPOINT now inside try for safety) BUG-05: Add /pushevent endpoint (log-only; was returning 404 to Jimi) FIX: push_fault_info — skip null fault_code (NULL != NULL in PG unique index) FIX: log_ingestion — pass SQL NULL not string "None" when no error occurred Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-11 18:19:13 +03:00
David Kiania	1f11a65b0b	Add 02_tracksolid_docker_commands.md — remote DB command reference Comprehensive reference for SSH + docker exec psql access to the TimescaleDB instance on rahamafresh.com. Covers: - How it works (SSH → docker exec → psql layers explained) - tsdb() shell function setup for the server - Mac one-liners for single queries, interactive sessions, piping SQL files - Fleet & live positions queries (active vehicles, silent devices, anomalies) - Trips & movement (today's KPIs, speeding, after-hours, utilisation rate) - Alarms (summary, unacknowledged, acknowledge) - Parking & idle time - Position history & route replay - Ingestion pipeline health checks - Device registry (metadata gaps, odometer, subscriptions) - Schema & migration operations - Container & service operations (logs, restart, disk, chunk sizes) - Quick reference table for all flags Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-11 08:23:01 +03:00
David Kiania	ae5bd2c960	Update tracksolidApiDocumentation.md with live implementation findings Reflects accurate field names, behaviours, and status from production: Polling endpoints: - 5.1 location.list: add full response schema (direction, gpsSignal, gpsNum, powerValue, elecQuantity, posType, locDesc); add implementation note (311 calls, ~19 devices/sweep, ~200ms, missing devices silently omitted) - 5.4 track.mileage: add maxSpeed field (BUG-03); add distance unit note (BUG-02 — values are km from API, corrected via migration 04) - 5.5 track.list: add altitude/satellite fields; add POLL-01 implementation note (30-min schedule, 35-min lookback, source='track_list', ~137s/call) - 5.7 parking: clarify acc_type=0 required; note durSecond vs stopSecond; add POLL-02 production status (60 calls, 0 rows, overnight expected) - Rate limits: document track.list latency (~137s per call) Alarms: - 6.1: replace vague note with explicit poll-vs-push field name table (alertTypeId/alarmTypeName vs alarmType/alarmName); confirm BUG-01 fix verified in production (type 3 / "Vibration alert" now stored correctly) Webhooks: - 10.1 /pushevent: mark implemented (PUSH-01), db table - 10.2 /pushhb: mark as not yet wired, table ready - 10.4 /pushalarm: mark implemented, cross-ref field name table - 10.7 /pushoil: mark implemented (PUSH-02), unit int→text note - 10.9 /pushtem: mark implemented (PUSH-03) - 10.10 /pushlbs: mark implemented (PUSH-04) - 10.20 /pushobd: mark implemented, document OBD scalar extraction - 10.21 /pushfaultinfo: mark not yet wired, table ready - 10.22 /pushtripreport: mark implemented Appendix B: full rewrite — split into polling and push tables with accurate status (✅/⚠️/not used), call counts, and DB table references Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-11 07:52:28 +03:00
David Kiania	d7ffa136a3	Regenerate 260410_baseline_report.md from live database (post-migration) Full live-query refresh against tracksolid_db at 07:38 EAT 2026-04-11. All data sourced directly from the server via 10 targeted psql queries. Report covers: all 17 table row counts, full 63-device registry with odometer/SIM/expiry, live position detail for all 19 reporting devices with GPS signal quality, geographic cluster map, position_history by source (poll=124 / track_list=13 = 137 total), alarm detail confirming BUG-01 fix, ingestion log health (399 calls, 0 failures), subscription status breakdown, silent device full list (44 devices), schema additions verification, Grafana readiness matrix, and P0/P1/P2 action plan. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-11 07:42:33 +03:00
David Kiania	f277532a9d	Regenerate 260410_baseline_report.md with post-migration comprehensive data Updated report reflects state after migrations 04 and 05 are fully applied. Includes: all 13 table row counts, fleet composition (63 devices / 4 models), live position coverage (19/63), position history breakdown by source (poll vs track_list), alarm detail (2 vibration alerts, BUG-01 fix confirmed), schema health checklist, ingestion log polling summary, odometer service flags, Uganda anomaly flag for X3-63282, data quality gap priority table, and Grafana readiness assessment. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-04-11 07:29:49 +03:00