Compare commits
11 commits
2f3879aa2a
...
20a98074a6
| Author | SHA1 | Date | |
|---|---|---|---|
|
20a98074a6 | ||
|
|
97b19eb968 | ||
|
|
40e452e156 | ||
|
|
09b3860706 | ||
|
|
3797a4e2ca | ||
|
|
d534aceadc | ||
|
|
05993100e9 | ||
|
|
c05b47abe2 | ||
|
|
791bf2700c | ||
|
|
82761e1e3f | ||
|
|
cd6b2ca81a |
18 changed files with 4932 additions and 24 deletions
772
01_BusinessAnalytics.md
Normal file
772
01_BusinessAnalytics.md
Normal file
|
|
@ -0,0 +1,772 @@
|
|||
# Fireside Communications — Fleet Business Analytics
|
||||
## Tracksolid Pro · Field Operations & Logistics Intelligence Assessment
|
||||
### April 2026
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Data Foundation Summary](#1-data-foundation-summary)
|
||||
2. [Fleet Utilisation](#2-fleet-utilisation)
|
||||
3. [Driver Behaviour](#3-driver-behaviour)
|
||||
4. [Real-Time Dispatch — Nearest Vehicle to Job](#4-real-time-dispatch--nearest-vehicle-to-job)
|
||||
5. [Distance per Driver per Day](#5-distance-per-driver-per-day)
|
||||
6. [Business Questions Now Answerable](#6-business-questions-now-answerable)
|
||||
7. [Grafana Dashboard Blueprint](#7-grafana-dashboard-blueprint)
|
||||
8. [What Unlocks the Remaining 30%](#8-what-unlocks-the-remaining-30)
|
||||
|
||||
---
|
||||
|
||||
## 1. Data Foundation Summary
|
||||
|
||||
The ingestion stack currently populates the following data sources, each feeding the analytics layer:
|
||||
|
||||
| Table | Content | Frequency |
|
||||
|---|---|---|
|
||||
| `tracksolid.live_positions` | Current position of every vehicle | Every 60 seconds |
|
||||
| `tracksolid.position_history` (source: `poll`) | Fleet position snapshot | Every 60 seconds |
|
||||
| `tracksolid.position_history` (source: `track_list`) | Every GPS waypoint per device | Every 30 minutes (35-min window) |
|
||||
| `tracksolid.trips` | Trip summaries: distance, speed, duration, idle | Every 15 minutes |
|
||||
| `tracksolid.parking_events` | Stop/idle events with address and duration | Every 15 minutes |
|
||||
| `tracksolid.alarms` | Alarm events with type, severity, location | Every 5 minutes |
|
||||
| `tracksolid.devices` | Vehicle and driver registry | Daily at 02:00 |
|
||||
| `dwh_gold.fact_daily_fleet_metrics` | Daily KPI aggregates per vehicle | Nightly ETL |
|
||||
|
||||
**Position history density** increased significantly with the addition of `poll_track_list` (POLL-01):
|
||||
|
||||
| Before | After |
|
||||
|---|---|
|
||||
| ~1 fix per minute per vehicle | 2–6 fixes per minute per active vehicle |
|
||||
| Route gaps of 1–2 km between points | Continuous accurate path traces |
|
||||
| Speed deltas invisible at 60s intervals | Harsh driving events detectable at 10–30s intervals |
|
||||
|
||||
All timestamps are stored in UTC and displayed in `Africa/Nairobi` (EAT = UTC+3) throughout this document.
|
||||
|
||||
---
|
||||
|
||||
## 2. Fleet Utilisation
|
||||
|
||||
### 2.1 Utilisation Rate
|
||||
|
||||
The percentage of working hours a vehicle is actively driving versus sitting idle or unused. Calculated per vehicle per day:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
t.imei,
|
||||
d.driver_name,
|
||||
d.vehicle_number,
|
||||
DATE(t.start_time AT TIME ZONE 'Africa/Nairobi') AS working_day,
|
||||
ROUND(SUM(t.driving_time_s) / 3600.0, 2) AS drive_hours,
|
||||
ROUND(SUM(t.idle_time_s) / 3600.0, 2) AS idle_hours,
|
||||
ROUND(
|
||||
SUM(t.driving_time_s) / (10.0 * 3600) * 100, 1
|
||||
) AS utilisation_pct
|
||||
FROM tracksolid.trips t
|
||||
JOIN tracksolid.devices d ON d.imei = t.imei
|
||||
WHERE t.start_time >= CURRENT_DATE AT TIME ZONE 'Africa/Nairobi'
|
||||
AND t.end_time IS NOT NULL
|
||||
GROUP BY t.imei, d.driver_name, d.vehicle_number, working_day
|
||||
ORDER BY utilisation_pct DESC;
|
||||
```
|
||||
|
||||
**Benchmark targets:**
|
||||
|
||||
| Rate | Interpretation | Action |
|
||||
|---|---|---|
|
||||
| > 70% | Excellent — asset working hard | Monitor driver fatigue |
|
||||
| 55–70% | Good — healthy operational range | No action required |
|
||||
| 40–55% | Below average — investigate stops | Review route planning |
|
||||
| < 40% | Poor — asset underutilised | Reassign or investigate |
|
||||
| 0% | Vehicle did not move today | Verify not broken down or abandoned |
|
||||
|
||||
> **Note:** The denominator (10 hours) should be adjusted to match your actual contractual shift length.
|
||||
|
||||
---
|
||||
|
||||
### 2.2 Daily Revenue-Generating Hours vs Fuel-Wasting Idle
|
||||
|
||||
Engine-on-but-stationary time is direct cost with no output. At Kenya diesel prices (~KES 180/litre) and typical 8 L/100 km consumption, a stationary diesel engine burns approximately **0.8 L/hour** at idle.
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
imei,
|
||||
SUM(total_drive_hours) AS drive_hours,
|
||||
SUM(total_idle_hours) AS idle_hours,
|
||||
ROUND(
|
||||
SUM(total_idle_hours) * 0.8 * 180, 0
|
||||
) AS idle_fuel_cost_kes,
|
||||
ROUND(
|
||||
SUM(total_idle_hours) /
|
||||
NULLIF(SUM(total_drive_hours + total_idle_hours), 0) * 100, 1
|
||||
) AS idle_pct
|
||||
FROM dwh_gold.fact_daily_fleet_metrics
|
||||
WHERE day >= CURRENT_DATE - INTERVAL '7 days'
|
||||
GROUP BY imei
|
||||
ORDER BY idle_fuel_cost_kes DESC;
|
||||
```
|
||||
|
||||
**Fleet-wide idle cost this month:**
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
ROUND(SUM(total_idle_hours), 1) AS fleet_idle_hours,
|
||||
ROUND(SUM(total_idle_hours) * 0.8 * 180) AS estimated_wasted_kes
|
||||
FROM dwh_gold.fact_daily_fleet_metrics
|
||||
WHERE day >= DATE_TRUNC('month', CURRENT_DATE);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2.3 Vehicles That Did Not Move Today
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
d.imei,
|
||||
d.vehicle_name,
|
||||
d.vehicle_number,
|
||||
d.driver_name,
|
||||
lp.gps_time AT TIME ZONE 'Africa/Nairobi' AS last_seen,
|
||||
lp.speed
|
||||
FROM tracksolid.devices d
|
||||
LEFT JOIN tracksolid.live_positions lp ON lp.imei = d.imei
|
||||
LEFT JOIN tracksolid.trips t
|
||||
ON t.imei = d.imei
|
||||
AND DATE(t.start_time AT TIME ZONE 'Africa/Nairobi') = CURRENT_DATE
|
||||
WHERE d.enabled_flag = 1
|
||||
AND t.imei IS NULL
|
||||
ORDER BY d.imei;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Driver Behaviour
|
||||
|
||||
### 3.1 Speeding
|
||||
|
||||
Counts position fixes where speed exceeded threshold, normalised per 100 km to avoid penalising drivers who simply drive more.
|
||||
|
||||
```sql
|
||||
WITH driver_speed AS (
|
||||
SELECT
|
||||
ph.imei,
|
||||
COUNT(*) FILTER (WHERE ph.speed > 80) AS fixes_over_80,
|
||||
COUNT(*) FILTER (WHERE ph.speed > 100) AS fixes_over_100,
|
||||
COUNT(*) FILTER (WHERE ph.speed > 120) AS fixes_over_120,
|
||||
COUNT(*) AS total_fixes
|
||||
FROM tracksolid.position_history ph
|
||||
WHERE ph.gps_time > NOW() - INTERVAL '7 days'
|
||||
AND ph.gps_time < NOW()
|
||||
AND ph.speed IS NOT NULL
|
||||
GROUP BY ph.imei
|
||||
),
|
||||
driver_km AS (
|
||||
SELECT imei, SUM(distance_km) AS total_km
|
||||
FROM tracksolid.trips
|
||||
WHERE start_time > NOW() - INTERVAL '7 days'
|
||||
GROUP BY imei
|
||||
)
|
||||
SELECT
|
||||
ds.imei,
|
||||
d.driver_name,
|
||||
d.vehicle_number,
|
||||
ROUND(dk.total_km, 1) AS km_driven,
|
||||
ds.fixes_over_80 AS events_80_kmh,
|
||||
ds.fixes_over_100 AS events_100_kmh,
|
||||
ds.fixes_over_120 AS events_120_kmh,
|
||||
ROUND(ds.fixes_over_80 / NULLIF(dk.total_km, 0) * 100, 2) AS rate_per_100km
|
||||
FROM driver_speed ds
|
||||
JOIN driver_km dk ON dk.imei = ds.imei
|
||||
JOIN tracksolid.devices d ON d.imei = ds.imei
|
||||
ORDER BY rate_per_100km DESC;
|
||||
```
|
||||
|
||||
**Severity banding:**
|
||||
|
||||
| Speed | Classification | Response |
|
||||
|---|---|---|
|
||||
| 80–100 km/h | Warning | Log, notify supervisor if persistent |
|
||||
| 100–120 km/h | Serious | Formal driver warning |
|
||||
| > 120 km/h | Critical | Immediate management escalation |
|
||||
|
||||
---
|
||||
|
||||
### 3.2 Harsh Driving — Hard Braking and Sudden Acceleration
|
||||
|
||||
Requires `track_list` data (POLL-01). Identifies speed changes greater than 30 km/h within a 60-second window — the signature of hard braking or sudden acceleration. Both events cause tyre wear, brake wear, fuel spikes, and increase accident probability.
|
||||
|
||||
```sql
|
||||
WITH ordered AS (
|
||||
SELECT
|
||||
imei,
|
||||
gps_time,
|
||||
speed,
|
||||
LAG(speed) OVER (PARTITION BY imei ORDER BY gps_time) AS prev_speed,
|
||||
LAG(gps_time) OVER (PARTITION BY imei ORDER BY gps_time) AS prev_time
|
||||
FROM tracksolid.position_history
|
||||
WHERE source = 'track_list'
|
||||
AND gps_time > NOW() - INTERVAL '7 days'
|
||||
AND gps_time < NOW()
|
||||
)
|
||||
SELECT
|
||||
imei,
|
||||
gps_time AT TIME ZONE 'Africa/Nairobi' AS event_time,
|
||||
prev_speed AS speed_before,
|
||||
speed AS speed_after,
|
||||
ABS(speed - prev_speed) AS delta_kmh,
|
||||
CASE
|
||||
WHEN speed > prev_speed THEN 'hard_acceleration'
|
||||
ELSE 'hard_braking'
|
||||
END AS event_type
|
||||
FROM ordered
|
||||
WHERE ABS(speed - prev_speed) > 30
|
||||
AND EXTRACT(EPOCH FROM (gps_time - prev_time)) BETWEEN 5 AND 60
|
||||
ORDER BY event_time DESC;
|
||||
```
|
||||
|
||||
**Driver aggression index** — normalised harsh events per 100 km:
|
||||
|
||||
```sql
|
||||
WITH harsh AS (
|
||||
SELECT
|
||||
imei,
|
||||
COUNT(*) AS harsh_events
|
||||
FROM (
|
||||
SELECT
|
||||
imei,
|
||||
speed,
|
||||
LAG(speed) OVER (PARTITION BY imei ORDER BY gps_time) AS prev_speed,
|
||||
LAG(gps_time) OVER (PARTITION BY imei ORDER BY gps_time) AS prev_time,
|
||||
gps_time
|
||||
FROM tracksolid.position_history
|
||||
WHERE source = 'track_list'
|
||||
AND gps_time > NOW() - INTERVAL '30 days'
|
||||
) sub
|
||||
WHERE ABS(speed - prev_speed) > 30
|
||||
AND EXTRACT(EPOCH FROM (gps_time - prev_time)) BETWEEN 5 AND 60
|
||||
GROUP BY imei
|
||||
),
|
||||
km AS (
|
||||
SELECT imei, SUM(distance_km) AS total_km
|
||||
FROM tracksolid.trips
|
||||
WHERE start_time > NOW() - INTERVAL '30 days'
|
||||
GROUP BY imei
|
||||
)
|
||||
SELECT
|
||||
h.imei,
|
||||
d.driver_name,
|
||||
d.vehicle_number,
|
||||
h.harsh_events,
|
||||
ROUND(k.total_km, 0) AS km_driven,
|
||||
ROUND(h.harsh_events / NULLIF(k.total_km, 0) * 100, 2) AS aggression_index
|
||||
FROM harsh h
|
||||
JOIN km k ON k.imei = h.imei
|
||||
JOIN tracksolid.devices d ON d.imei = h.imei
|
||||
ORDER BY aggression_index DESC;
|
||||
```
|
||||
|
||||
> An aggression index below **0.5** is good. Above **2.0** warrants a driver coaching conversation. Above **5.0** is a safety concern.
|
||||
|
||||
---
|
||||
|
||||
### 3.3 Tardiness — Late Starts and Early Knock-Off
|
||||
|
||||
**Late starts** (first ignition-on after scheduled shift start):
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
f.vehicle_key AS imei,
|
||||
d.driver_name,
|
||||
d.vehicle_number,
|
||||
f.day,
|
||||
f.day_start_time,
|
||||
CASE
|
||||
WHEN f.day_start_time > '07:45:00' THEN
|
||||
EXTRACT(EPOCH FROM (f.day_start_time - '07:30:00'::TIME)) / 60
|
||||
ELSE 0
|
||||
END::INT AS minutes_late
|
||||
FROM dwh_gold.fact_daily_fleet_metrics f
|
||||
JOIN tracksolid.devices d ON d.imei = f.vehicle_key
|
||||
WHERE f.day >= CURRENT_DATE - INTERVAL '30 days'
|
||||
AND f.day_start_time > '07:45:00'
|
||||
ORDER BY minutes_late DESC;
|
||||
```
|
||||
|
||||
**Early knock-off** (last trip ended before scheduled shift end):
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
f.vehicle_key AS imei,
|
||||
d.driver_name,
|
||||
f.day,
|
||||
f.day_end_time,
|
||||
CASE
|
||||
WHEN f.day_end_time < '17:00:00' THEN
|
||||
EXTRACT(EPOCH FROM ('17:00:00'::TIME - f.day_end_time)) / 60
|
||||
ELSE 0
|
||||
END::INT AS minutes_early
|
||||
FROM dwh_gold.fact_daily_fleet_metrics f
|
||||
JOIN tracksolid.devices d ON d.imei = f.vehicle_key
|
||||
WHERE f.day >= CURRENT_DATE - INTERVAL '30 days'
|
||||
AND f.day_end_time < '17:00:00'
|
||||
AND f.total_trips > 0
|
||||
ORDER BY minutes_early DESC;
|
||||
```
|
||||
|
||||
> Adjust `'07:30:00'` and `'17:00:00'` to match your actual contracted shift times.
|
||||
|
||||
**Chronic late starters — monthly pattern:**
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
f.vehicle_key AS imei,
|
||||
d.driver_name,
|
||||
COUNT(*) AS late_days,
|
||||
ROUND(AVG(
|
||||
EXTRACT(EPOCH FROM (f.day_start_time - '07:30:00'::TIME)) / 60
|
||||
), 0) AS avg_minutes_late
|
||||
FROM dwh_gold.fact_daily_fleet_metrics f
|
||||
JOIN tracksolid.devices d ON d.imei = f.vehicle_key
|
||||
WHERE f.day >= DATE_TRUNC('month', CURRENT_DATE)
|
||||
AND f.day_start_time > '07:45:00'
|
||||
GROUP BY f.vehicle_key, d.driver_name
|
||||
HAVING COUNT(*) >= 3
|
||||
ORDER BY late_days DESC, avg_minutes_late DESC;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3.4 After-Hours Movement
|
||||
|
||||
Any trip starting or ending outside contracted hours. Flags unauthorised vehicle use, night deliveries not on schedule, or potential vehicle theft.
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
t.imei,
|
||||
d.driver_name,
|
||||
d.vehicle_number,
|
||||
t.start_time AT TIME ZONE 'Africa/Nairobi' AS departure_nairobi,
|
||||
t.end_time AT TIME ZONE 'Africa/Nairobi' AS arrival_nairobi,
|
||||
ROUND(t.distance_km::numeric, 1) AS distance_km,
|
||||
CASE
|
||||
WHEN EXTRACT(HOUR FROM t.start_time AT TIME ZONE 'Africa/Nairobi') < 6 THEN 'pre-dawn departure'
|
||||
WHEN EXTRACT(HOUR FROM t.start_time AT TIME ZONE 'Africa/Nairobi') >= 20 THEN 'night departure'
|
||||
ELSE 'after-hours return'
|
||||
END AS flag
|
||||
FROM tracksolid.trips t
|
||||
JOIN tracksolid.devices d ON d.imei = t.imei
|
||||
WHERE (
|
||||
EXTRACT(HOUR FROM t.start_time AT TIME ZONE 'Africa/Nairobi') < 6
|
||||
OR EXTRACT(HOUR FROM t.start_time AT TIME ZONE 'Africa/Nairobi') >= 20
|
||||
OR EXTRACT(HOUR FROM t.end_time AT TIME ZONE 'Africa/Nairobi') >= 21
|
||||
)
|
||||
AND t.start_time > NOW() - INTERVAL '30 days'
|
||||
ORDER BY t.start_time DESC;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3.5 Km Covered per Driver per Day
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
t.imei,
|
||||
d.driver_name,
|
||||
d.vehicle_number,
|
||||
DATE(t.start_time AT TIME ZONE 'Africa/Nairobi') AS working_day,
|
||||
ROUND(SUM(t.distance_km)::numeric, 1) AS km_driven,
|
||||
COUNT(*) AS trips,
|
||||
ROUND(SUM(t.driving_time_s) / 3600.0, 2) AS drive_hours,
|
||||
ROUND(SUM(t.idle_time_s) / 3600.0, 2) AS idle_hours,
|
||||
MAX(t.max_speed_kmh) AS peak_speed_kmh,
|
||||
MIN(t.start_time AT TIME ZONE 'Africa/Nairobi')::TIME AS first_departure,
|
||||
MAX(t.end_time AT TIME ZONE 'Africa/Nairobi')::TIME AS last_return
|
||||
FROM tracksolid.trips t
|
||||
JOIN tracksolid.devices d ON d.imei = t.imei
|
||||
WHERE t.start_time >= CURRENT_DATE AT TIME ZONE 'Africa/Nairobi'
|
||||
AND t.end_time IS NOT NULL
|
||||
GROUP BY t.imei, d.driver_name, d.vehicle_number, working_day
|
||||
ORDER BY km_driven DESC;
|
||||
```
|
||||
|
||||
**Expected daily km benchmarks by vehicle type:**
|
||||
|
||||
| Vehicle Type | Expected Daily km | Flag: Below | Flag: Above |
|
||||
|---|---|---|---|
|
||||
| Urban delivery van | 80–150 km | < 40 km | > 300 km |
|
||||
| Long-haul truck | 300–500 km | < 150 km | > 700 km |
|
||||
| Field/supervisor vehicle | 50–120 km | < 20 km | > 250 km |
|
||||
| Motorcycle courier | 60–120 km | < 30 km | > 200 km |
|
||||
|
||||
A driver consistently covering 250 km/day in an urban van either has a legitimately large route or is running personal errands between jobs. Both scenarios need different responses.
|
||||
|
||||
**Weekly km trend per driver:**
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
t.imei,
|
||||
d.driver_name,
|
||||
DATE_TRUNC('week', t.start_time AT TIME ZONE 'Africa/Nairobi') AS week_start,
|
||||
ROUND(SUM(t.distance_km)::numeric, 1) AS total_km,
|
||||
COUNT(DISTINCT DATE(t.start_time AT TIME ZONE 'Africa/Nairobi')) AS days_active,
|
||||
ROUND(SUM(t.distance_km)::numeric /
|
||||
NULLIF(COUNT(DISTINCT DATE(t.start_time AT TIME ZONE 'Africa/Nairobi')), 0), 1
|
||||
) AS avg_km_per_day
|
||||
FROM tracksolid.trips t
|
||||
JOIN tracksolid.devices d ON d.imei = t.imei
|
||||
WHERE t.start_time > NOW() - INTERVAL '90 days'
|
||||
AND t.end_time IS NOT NULL
|
||||
GROUP BY t.imei, d.driver_name, week_start
|
||||
ORDER BY t.imei, week_start;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Real-Time Dispatch — Nearest Vehicle to Job
|
||||
|
||||
### 4.1 Find the 5 Closest Available Vehicles
|
||||
|
||||
Given a new job at a known location, this query returns the nearest active vehicles with a fresh GPS fix. Runs in milliseconds against the `live_positions` table with the PostGIS spatial index.
|
||||
|
||||
```sql
|
||||
-- Replace :job_lat and :job_lng with the job coordinates
|
||||
SELECT
|
||||
lp.imei,
|
||||
d.vehicle_name,
|
||||
d.vehicle_number,
|
||||
d.driver_name,
|
||||
d.driver_phone,
|
||||
d.vehicle_category,
|
||||
lp.acc_status,
|
||||
lp.speed,
|
||||
ROUND(
|
||||
ST_Distance(
|
||||
lp.geom::geography,
|
||||
ST_SetSRID(ST_MakePoint(:job_lng, :job_lat), 4326)::geography
|
||||
) / 1000.0, 2
|
||||
) AS distance_km,
|
||||
ROUND(
|
||||
ST_Distance(
|
||||
lp.geom::geography,
|
||||
ST_SetSRID(ST_MakePoint(:job_lng, :job_lat), 4326)::geography
|
||||
) / 1000.0 / 30.0 * 60, 0
|
||||
) AS eta_minutes_urban,
|
||||
lp.gps_time AT TIME ZONE 'Africa/Nairobi' AS last_seen
|
||||
FROM tracksolid.live_positions lp
|
||||
JOIN tracksolid.devices d ON d.imei = lp.imei
|
||||
WHERE lp.acc_status = '1'
|
||||
AND lp.speed < 5
|
||||
AND lp.gps_time > NOW() - INTERVAL '5 minutes'
|
||||
ORDER BY distance_km ASC
|
||||
LIMIT 5;
|
||||
```
|
||||
|
||||
**ETA speed assumptions** — adjust the divisor to match the route type:
|
||||
|
||||
| Route type | Speed (km/h) | Formula |
|
||||
|---|---|---|
|
||||
| Nairobi CBD | 20 km/h | `/ 20.0 * 60` |
|
||||
| Nairobi urban | 30 km/h | `/ 30.0 * 60` |
|
||||
| Peri-urban | 50 km/h | `/ 50.0 * 60` |
|
||||
| Highway | 80 km/h | `/ 80.0 * 60` |
|
||||
|
||||
---
|
||||
|
||||
### 4.2 Dispatch Logic for n8n or API Integration
|
||||
|
||||
The recommended workflow when a new job/ticket arrives:
|
||||
|
||||
1. **Trigger:** New job created (webhook from job management system or n8n)
|
||||
2. **Force-refresh positions:** Call `get_device_locations()` for the top 10 candidate IMEIs to get sub-second fresh positions before committing
|
||||
3. **Run dispatch query** above with job coordinates
|
||||
4. **Filter by vehicle type** if the job requires specific capacity (`AND d.vehicle_category = 'van'`)
|
||||
5. **Exclude vehicles with open alarms:** `AND NOT EXISTS (SELECT 1 FROM tracksolid.alarms a WHERE a.imei = lp.imei AND a.alarm_time > NOW() - INTERVAL '1 hour')`
|
||||
6. **Present top 3 candidates** to dispatcher (or auto-assign #1 if fully automated)
|
||||
7. **Log dispatch decision** to a separate `dispatch_log` table for SLA tracking
|
||||
|
||||
---
|
||||
|
||||
### 4.3 All Active Vehicles Map — Live Fleet View
|
||||
|
||||
Returns all vehicles with a position fix in the last 10 minutes, suitable for a Grafana Geomap panel with auto-refresh at 30 seconds.
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
lp.imei,
|
||||
COALESCE(d.vehicle_name, d.vehicle_number, lp.imei) AS label,
|
||||
d.driver_name,
|
||||
lp.lat,
|
||||
lp.lng,
|
||||
lp.speed,
|
||||
lp.acc_status,
|
||||
CASE
|
||||
WHEN lp.speed > 5 THEN 'moving'
|
||||
WHEN lp.acc_status = '1' THEN 'idle'
|
||||
ELSE 'parked'
|
||||
END AS vehicle_state,
|
||||
lp.gps_time AT TIME ZONE 'Africa/Nairobi' AS last_seen
|
||||
FROM tracksolid.live_positions lp
|
||||
JOIN tracksolid.devices d ON d.imei = lp.imei
|
||||
WHERE lp.gps_time > NOW() - INTERVAL '10 minutes'
|
||||
ORDER BY lp.imei;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Distance per Driver per Day
|
||||
|
||||
### 5.1 Today's Summary
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
t.imei,
|
||||
COALESCE(d.driver_name, 'Unassigned') AS driver,
|
||||
COALESCE(d.vehicle_number, t.imei) AS vehicle,
|
||||
ROUND(SUM(t.distance_km)::numeric, 1) AS km_today,
|
||||
COUNT(*) AS trips_today,
|
||||
ROUND(SUM(t.driving_time_s) / 3600.0, 2) AS drive_hours,
|
||||
ROUND(SUM(t.idle_time_s) / 3600.0, 2) AS idle_hours,
|
||||
MIN(t.start_time AT TIME ZONE 'Africa/Nairobi')::TIME AS first_departure,
|
||||
MAX(t.end_time AT TIME ZONE 'Africa/Nairobi')::TIME AS last_return
|
||||
FROM tracksolid.trips t
|
||||
JOIN tracksolid.devices d ON d.imei = t.imei
|
||||
WHERE t.start_time >= CURRENT_DATE AT TIME ZONE 'Africa/Nairobi'
|
||||
AND t.end_time IS NOT NULL
|
||||
GROUP BY t.imei, d.driver_name, d.vehicle_number
|
||||
ORDER BY km_today DESC;
|
||||
```
|
||||
|
||||
### 5.2 30-Day Driver Performance Scorecard
|
||||
|
||||
Combines distance, behaviour, and punctuality into a single view per driver.
|
||||
|
||||
```sql
|
||||
WITH km_summary AS (
|
||||
SELECT
|
||||
imei,
|
||||
COUNT(DISTINCT DATE(start_time AT TIME ZONE 'Africa/Nairobi')) AS days_active,
|
||||
ROUND(SUM(distance_km)::numeric, 1) AS total_km,
|
||||
ROUND(AVG(distance_km)::numeric, 1) AS avg_km_per_trip,
|
||||
MAX(max_speed_kmh) AS peak_speed
|
||||
FROM tracksolid.trips
|
||||
WHERE start_time > NOW() - INTERVAL '30 days'
|
||||
AND end_time IS NOT NULL
|
||||
GROUP BY imei
|
||||
),
|
||||
alarm_summary AS (
|
||||
SELECT imei, COUNT(*) AS alarm_count
|
||||
FROM tracksolid.alarms
|
||||
WHERE alarm_time > NOW() - INTERVAL '30 days'
|
||||
GROUP BY imei
|
||||
),
|
||||
late_summary AS (
|
||||
SELECT vehicle_key AS imei, COUNT(*) AS late_days
|
||||
FROM dwh_gold.fact_daily_fleet_metrics
|
||||
WHERE day > CURRENT_DATE - 30
|
||||
AND day_start_time > '07:45:00'
|
||||
GROUP BY vehicle_key
|
||||
)
|
||||
SELECT
|
||||
k.imei,
|
||||
d.driver_name,
|
||||
d.vehicle_number,
|
||||
k.days_active,
|
||||
k.total_km,
|
||||
ROUND(k.total_km / NULLIF(k.days_active, 0), 1) AS avg_km_per_day,
|
||||
k.peak_speed AS peak_speed_kmh,
|
||||
COALESCE(a.alarm_count, 0) AS alarms_30d,
|
||||
COALESCE(l.late_days, 0) AS late_starts_30d
|
||||
FROM km_summary k
|
||||
JOIN tracksolid.devices d ON d.imei = k.imei
|
||||
LEFT JOIN alarm_summary a ON a.imei = k.imei
|
||||
LEFT JOIN late_summary l ON l.imei = k.imei
|
||||
ORDER BY k.total_km DESC;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Business Questions Now Answerable
|
||||
|
||||
| Business Question | Primary Data Source | Confidence |
|
||||
|---|---|---|
|
||||
| Which vehicles are moving right now? | `live_positions` | High |
|
||||
| Who started work latest today? | `fact_daily_fleet_metrics.day_start_time` | High |
|
||||
| Who drove the most km this week? | `trips` + `devices` | High |
|
||||
| Which vehicle spent the most time idling? | `trips.idle_time_s` | High |
|
||||
| How much fuel was wasted on idle today? | `trips.idle_time_s` × est. rate | Medium (needs `fuel_100km` set) |
|
||||
| Which driver triggered the most alarms this month? | `alarms` + `devices` | High |
|
||||
| What is total fleet distance this month? | `trips` | High |
|
||||
| Which vehicles did not move at all today? | `trips` LEFT JOIN `devices` | High |
|
||||
| Who is nearest to a new job right now? | `live_positions` + PostGIS | High |
|
||||
| Did any vehicle leave depot after hours? | `trips` time filter | High |
|
||||
| What is the speeding rate per driver per week? | `position_history` speed filter | High |
|
||||
| Which driver has the harshest driving style? | `position_history` delta query | High (needs 1–2 weeks of `track_list` data to accumulate) |
|
||||
| Are vehicles on approved routes? | `position_history` + `geofences` | Low (pending geofence population) |
|
||||
| Is cold chain in temperature range? | `temperature_readings` | Low (pending webhook registration) |
|
||||
| How much fuel is consumed per route? | `fuel_readings` + `trips` | Low (pending fuel sensor webhook) |
|
||||
| What is the real odometer per vehicle? | `live_positions.current_mileage` | Medium (depends on tracker calibration) |
|
||||
| How many km to next service interval? | `live_positions.current_mileage` - last service | Open (requires service log) |
|
||||
| Did any vehicle enter a restricted zone? | `alarms` (geofence type) + `geofences` | Low (pending geofence setup) |
|
||||
| Which drivers are consistently late on Mondays? | `fact_daily_fleet_metrics` day-of-week filter | High |
|
||||
| What percentage of the fleet was utilised today? | `trips` + `devices` count | High |
|
||||
|
||||
---
|
||||
|
||||
## 7. Grafana Dashboard Blueprint
|
||||
|
||||
### Panel 1 — Real-Time Fleet Map (auto-refresh: 30s)
|
||||
- **Type:** Geomap
|
||||
- **Source:** `live_positions` joined to `devices`
|
||||
- **Colour coding:**
|
||||
- Green = moving (speed > 5 km/h)
|
||||
- Amber = ignition on, stationary (acc_status = '1', speed ≤ 5)
|
||||
- Red = offline (last fix > 10 minutes ago)
|
||||
- **Tooltip:** driver name, vehicle number, speed, last seen
|
||||
|
||||
### Panel 2 — Fleet Status Summary Row (auto-refresh: 1m)
|
||||
| Stat | Query |
|
||||
|---|---|
|
||||
| Vehicles active now | COUNT WHERE acc_status = '1' AND gps_time > NOW() - 5m |
|
||||
| Vehicles moving | COUNT WHERE speed > 5 AND gps_time > NOW() - 5m |
|
||||
| Vehicles offline | COUNT WHERE gps_time < NOW() - 10m |
|
||||
| Open alarms | COUNT FROM alarms WHERE alarm_time > NOW() - 1h |
|
||||
| Fleet km today | SUM(distance_km) WHERE start_time >= today |
|
||||
|
||||
### Panel 3 — Daily KPI Table (refresh: 1h)
|
||||
Columns: Vehicle · Driver · Km Today · Trips · Drive Hours · Idle Hours · First Departure · Last Return · Alarms
|
||||
|
||||
### Panel 4 — Driver Behaviour Leaderboard (refresh: 1h)
|
||||
Ranked by aggression index (harsh events per 100 km), speeding events, and late starts. Colour-coded red/amber/green per threshold.
|
||||
|
||||
### Panel 5 — Distance Trend (7-day bar chart)
|
||||
- X-axis: Date
|
||||
- Y-axis: Total km
|
||||
- Series: one bar per vehicle or fleet total with daily breakdown
|
||||
|
||||
### Panel 6 — Idle Cost Tracker (refresh: 1h)
|
||||
- Running total of idle hours and estimated KES wasted this month
|
||||
- Trend line showing improvement or deterioration week-over-week
|
||||
|
||||
### Panel 7 — Alarm Frequency (30-day time series)
|
||||
- Line chart: alarm count per day
|
||||
- Breakdown by alarm type (overspeed, geofence, harsh braking)
|
||||
|
||||
### Panel 8 — Utilisation Heatmap (weekly)
|
||||
- Y-axis: Vehicle/driver
|
||||
- X-axis: Day of week
|
||||
- Colour: utilisation % (green > 60%, amber 40–60%, red < 40%)
|
||||
|
||||
---
|
||||
|
||||
## 8. What Unlocks the Remaining 30%
|
||||
|
||||
The data foundation is in place. The following five steps activate the remaining analytics capabilities:
|
||||
|
||||
### Step 1 — Register Webhooks in Tracksolid Pro Account *(Blocker)*
|
||||
Without registration, the following tables remain empty regardless of code:
|
||||
|
||||
| Webhook | Table | Unlocks |
|
||||
|---|---|---|
|
||||
| `/pushobd` | `obd_readings` | Engine health, fuel level per fix, RPM |
|
||||
| `/pushoil` | `fuel_readings` | Fuel theft detection, tank level trend |
|
||||
| `/pushtem` | `temperature_readings` | Cold chain compliance alerts |
|
||||
| `/pushlbs` | `lbs_readings` | Positions when GPS signal lost |
|
||||
| `/pushevent` | `device_events` | Device powered off/on events (tamper detection) |
|
||||
| `/pushtripreport` | `trips` (push source) | Real-time trip completion events |
|
||||
|
||||
**Action:** Log into Tracksolid Pro → Account Settings → Webhook Configuration → add server URL for each endpoint.
|
||||
|
||||
---
|
||||
|
||||
### Step 2 — Set `fuel_100km` per Vehicle Type
|
||||
|
||||
Currently null for all 63 devices. Once set, all fuel cost calculations activate automatically.
|
||||
|
||||
```sql
|
||||
-- Example: set consumption rates by vehicle category
|
||||
UPDATE tracksolid.devices SET fuel_100km = 8.5 WHERE vehicle_category = 'truck';
|
||||
UPDATE tracksolid.devices SET fuel_100km = 7.0 WHERE vehicle_category = 'van';
|
||||
UPDATE tracksolid.devices SET fuel_100km = 4.5 WHERE vehicle_category = 'motorcycle';
|
||||
UPDATE tracksolid.devices SET fuel_100km = 9.0 WHERE vehicle_category = 'car';
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 3 — Populate Vehicle Names and Driver Names
|
||||
|
||||
Currently all 63 devices show blank fields. Reports display IMEI numbers instead of human-readable identities.
|
||||
|
||||
```sql
|
||||
-- Update individually or import from CSV via COPY
|
||||
UPDATE tracksolid.devices
|
||||
SET vehicle_name = 'KBZ 123A',
|
||||
vehicle_number = 'KBZ 123A',
|
||||
driver_name = 'John Kamau',
|
||||
driver_phone = '+254700000001',
|
||||
vehicle_category = 'van'
|
||||
WHERE imei = '352093080000001';
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 4 — Define Geofences
|
||||
|
||||
Populate `tracksolid.geofences` with:
|
||||
- **Depot boundaries** — alert when vehicles leave outside working hours
|
||||
- **Approved route corridors** — alert when vehicles deviate from assigned routes
|
||||
- **Restricted zones** — alert when vehicles enter prohibited areas (e.g. competitor premises, residential zones during noise hours)
|
||||
|
||||
```sql
|
||||
-- Example: circular depot geofence
|
||||
INSERT INTO tracksolid.geofences (fence_id, fence_name, fence_type, geom, radius_m)
|
||||
VALUES (
|
||||
'depot_nairobi_main',
|
||||
'Main Nairobi Depot',
|
||||
'circle',
|
||||
ST_SetSRID(ST_MakePoint(36.8219, -1.2921), 4326),
|
||||
200
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 5 — Run Migrations and Deploy Updated Containers
|
||||
|
||||
```bash
|
||||
# Resolve container name dynamically (survives Coolify redeployments)
|
||||
TS_DB=$(docker ps --filter "name=timescale_db" --format "{{.Names}}" | head -1)
|
||||
|
||||
# 1. Run distance correction migration (fixes historical data)
|
||||
docker exec -i "$TS_DB" psql -U postgres -d tracksolid_db \
|
||||
< /migrations/04_bug_fix_migration.sql
|
||||
|
||||
# 2. Run schema enhancement migration (new tables + columns)
|
||||
docker exec -i "$TS_DB" psql -U postgres -d tracksolid_db \
|
||||
< /migrations/05_enhancement_migration.sql
|
||||
|
||||
# 3. Rebuild and restart ingestion containers with updated code
|
||||
docker compose up -d --build ingest_movement ingest_events webhook_receiver
|
||||
|
||||
# 4. Schedule nightly ETL
|
||||
# Add to cron or n8n:
|
||||
# SELECT dwh_gold.refresh_daily_metrics(CURRENT_DATE - 1);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Appendix — Key Metric Thresholds Reference
|
||||
|
||||
| Metric | Green | Amber | Red |
|
||||
|---|---|---|---|
|
||||
| Fleet utilisation rate | > 60% | 40–60% | < 40% |
|
||||
| Idle time as % of shift | < 15% | 15–30% | > 30% |
|
||||
| Speeding events per 100 km | < 0.5 | 0.5–2.0 | > 2.0 |
|
||||
| Harsh driving index per 100 km | < 0.5 | 0.5–2.0 | > 2.0 |
|
||||
| Late starts per month (per driver) | 0–1 | 2–4 | ≥ 5 |
|
||||
| Days vehicle not used (per month) | 0–2 | 3–5 | > 5 |
|
||||
| GPS fix age (live_positions) | < 2 min | 2–10 min | > 10 min |
|
||||
| Alarm rate per vehicle per week | 0–2 | 3–7 | > 7 |
|
||||
|
||||
---
|
||||
|
||||
*Document generated: 2026-04-10 · Stack: TimescaleDB 2.15 + PostGIS + Tracksolid Pro Open Platform API*
|
||||
*Ingestion pipeline: `ingest_movement_rev.py` v2.2 · `ingest_events_rev.py` · `webhook_receiver_rev.py`*
|
||||
36
04_bug_fix_migration.sql
Normal file
36
04_bug_fix_migration.sql
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
-- ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
-- Migration 04 — Bug Fix: distance unit correction + column rename
|
||||
-- ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
-- BUG-02: tracksolid.trips.distance_m was storing millimetres not metres.
|
||||
--
|
||||
-- Root cause: ingestion code applied `km * 1000` but the API already returns
|
||||
-- values in km, producing mm. Confirmed by cross-checking stored values against
|
||||
-- avg_speed_kmh × driving_time_s — e.g. 4,203,000 stored for a 4.203 km trip.
|
||||
--
|
||||
-- Fix applied here:
|
||||
-- 1. Divide all existing rows by 1,000,000 to convert mm → km.
|
||||
-- 2. Rename the column to distance_km to eliminate future ambiguity.
|
||||
--
|
||||
-- Corresponding code fix: removed * 1000 from poll_trips() in
|
||||
-- ingest_movement_rev.py and push_trip_report() in webhook_receiver_rev.py.
|
||||
-- Both now store the raw API value directly as km.
|
||||
--
|
||||
-- Run once against tracksolid_db before deploying updated ingestion containers.
|
||||
-- ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
BEGIN;
|
||||
|
||||
-- Step 1: Correct all existing stored values (mm → km)
|
||||
UPDATE tracksolid.trips
|
||||
SET distance_m = distance_m / 1000000.0
|
||||
WHERE distance_m IS NOT NULL;
|
||||
|
||||
-- Step 2: Rename column
|
||||
ALTER TABLE tracksolid.trips
|
||||
RENAME COLUMN distance_m TO distance_km;
|
||||
|
||||
-- Step 3: Update column comment
|
||||
COMMENT ON COLUMN tracksolid.trips.distance_km
|
||||
IS 'Trip distance in kilometres. Corrected from mm storage on migration 04 (2026-04-10).';
|
||||
|
||||
COMMIT;
|
||||
270
05_enhancement_migration.sql
Normal file
270
05_enhancement_migration.sql
Normal file
|
|
@ -0,0 +1,270 @@
|
|||
-- ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
-- Migration 05 — Schema Enhancements for Expanded Ingestion
|
||||
-- ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
-- Adds columns and tables to support:
|
||||
-- • Normalized OBD scalar fields (from /pushobd JSONB payload)
|
||||
-- • Alarm enrichment (severity, geofence context, acknowledgement)
|
||||
-- • Vehicle enrichment (category, cost centre, depot location)
|
||||
-- • New webhook endpoints: /pushevent, /pushoil, /pushtem, /pushlbs
|
||||
-- • Geofence definition storage
|
||||
-- • dwh_gold fact table expansion for full daily KPI reporting
|
||||
--
|
||||
-- Run after migration 04. Safe to re-run (uses IF NOT EXISTS / DO NOTHING).
|
||||
-- ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
BEGIN;
|
||||
|
||||
-- ── 1. Normalize OBD scalar fields ───────────────────────────────────────────
|
||||
-- These are extracted from the obd_data JSONB column during /pushobd ingestion.
|
||||
-- Raw JSONB is retained for full fidelity. Common OBD PID values only.
|
||||
|
||||
ALTER TABLE tracksolid.obd_readings
|
||||
ADD COLUMN IF NOT EXISTS engine_rpm INTEGER,
|
||||
ADD COLUMN IF NOT EXISTS coolant_temp_c NUMERIC(6,2),
|
||||
ADD COLUMN IF NOT EXISTS fuel_level_pct NUMERIC(5,2),
|
||||
ADD COLUMN IF NOT EXISTS battery_voltage NUMERIC(5,2),
|
||||
ADD COLUMN IF NOT EXISTS intake_pressure NUMERIC(6,2),
|
||||
ADD COLUMN IF NOT EXISTS throttle_pct NUMERIC(5,2),
|
||||
ADD COLUMN IF NOT EXISTS vehicle_speed NUMERIC(7,2),
|
||||
ADD COLUMN IF NOT EXISTS engine_load_pct NUMERIC(5,2);
|
||||
|
||||
COMMENT ON COLUMN tracksolid.obd_readings.engine_rpm IS 'Engine RPM from OBD PID 0x0C';
|
||||
COMMENT ON COLUMN tracksolid.obd_readings.coolant_temp_c IS 'Coolant temperature °C from OBD PID 0x05';
|
||||
COMMENT ON COLUMN tracksolid.obd_readings.fuel_level_pct IS 'Fuel tank level % from OBD PID 0x2F';
|
||||
COMMENT ON COLUMN tracksolid.obd_readings.battery_voltage IS 'Battery voltage (V) from OBD PID 0x42';
|
||||
COMMENT ON COLUMN tracksolid.obd_readings.intake_pressure IS 'Intake manifold pressure kPa from OBD PID 0x0B';
|
||||
COMMENT ON COLUMN tracksolid.obd_readings.throttle_pct IS 'Throttle position % from OBD PID 0x11';
|
||||
COMMENT ON COLUMN tracksolid.obd_readings.vehicle_speed IS 'Vehicle speed km/h from OBD PID 0x0D';
|
||||
COMMENT ON COLUMN tracksolid.obd_readings.engine_load_pct IS 'Calculated engine load % from OBD PID 0x04';
|
||||
|
||||
-- ── 2. Alarm enrichment ───────────────────────────────────────────────────────
|
||||
|
||||
ALTER TABLE tracksolid.alarms
|
||||
ADD COLUMN IF NOT EXISTS severity TEXT,
|
||||
ADD COLUMN IF NOT EXISTS geofence_id TEXT,
|
||||
ADD COLUMN IF NOT EXISTS geofence_name TEXT,
|
||||
ADD COLUMN IF NOT EXISTS acknowledged_at TIMESTAMPTZ,
|
||||
ADD COLUMN IF NOT EXISTS acknowledged_by TEXT;
|
||||
|
||||
COMMENT ON COLUMN tracksolid.alarms.severity IS 'Alarm severity level: critical | warning | info';
|
||||
COMMENT ON COLUMN tracksolid.alarms.geofence_id IS 'Tracksolid geofence ID if this is a geofence alarm';
|
||||
COMMENT ON COLUMN tracksolid.alarms.geofence_name IS 'Human-readable geofence name';
|
||||
COMMENT ON COLUMN tracksolid.alarms.acknowledged_at IS 'Timestamp when alarm was acknowledged by an operator';
|
||||
COMMENT ON COLUMN tracksolid.alarms.acknowledged_by IS 'Username or ID of operator who acknowledged the alarm';
|
||||
|
||||
-- ── 3. Vehicle enrichment ─────────────────────────────────────────────────────
|
||||
|
||||
ALTER TABLE tracksolid.devices
|
||||
ADD COLUMN IF NOT EXISTS vehicle_category TEXT,
|
||||
ADD COLUMN IF NOT EXISTS cost_centre TEXT,
|
||||
ADD COLUMN IF NOT EXISTS assigned_route TEXT,
|
||||
ADD COLUMN IF NOT EXISTS depot_geom geometry(Point,4326),
|
||||
ADD COLUMN IF NOT EXISTS depot_address TEXT;
|
||||
|
||||
COMMENT ON COLUMN tracksolid.devices.vehicle_category IS 'Vehicle type: truck | van | motorcycle | car | other';
|
||||
COMMENT ON COLUMN tracksolid.devices.cost_centre IS 'Business unit or department this vehicle belongs to';
|
||||
COMMENT ON COLUMN tracksolid.devices.assigned_route IS 'Regular route name or ID for route-based reporting';
|
||||
COMMENT ON COLUMN tracksolid.devices.depot_geom IS 'Home base/depot coordinates (WGS84)';
|
||||
COMMENT ON COLUMN tracksolid.devices.depot_address IS 'Human-readable depot address';
|
||||
|
||||
-- ── 4. Device login/logout events (webhook /pushevent) ────────────────────────
|
||||
|
||||
CREATE TABLE IF NOT EXISTS tracksolid.device_events (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
imei TEXT NOT NULL REFERENCES tracksolid.devices(imei),
|
||||
event_type TEXT NOT NULL, -- 'LOGIN' | 'LOGOUT'
|
||||
event_time TIMESTAMPTZ NOT NULL,
|
||||
timezone TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
UNIQUE (imei, event_type, event_time)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_device_events_imei_time
|
||||
ON tracksolid.device_events (imei, event_time DESC);
|
||||
|
||||
COMMENT ON TABLE tracksolid.device_events
|
||||
IS 'Device network connection and disconnection events from /pushevent webhook.';
|
||||
COMMENT ON COLUMN tracksolid.device_events.event_type
|
||||
IS 'LOGIN = device connected to network; LOGOUT = device disconnected';
|
||||
|
||||
-- ── 5. Fuel sensor readings (webhook /pushoil) — hypertable ──────────────────
|
||||
|
||||
CREATE TABLE IF NOT EXISTS tracksolid.fuel_readings (
|
||||
imei TEXT NOT NULL REFERENCES tracksolid.devices(imei),
|
||||
reading_time TIMESTAMPTZ NOT NULL,
|
||||
sensor_path TEXT,
|
||||
value NUMERIC(10,3),
|
||||
unit TEXT,
|
||||
lat DOUBLE PRECISION,
|
||||
lng DOUBLE PRECISION,
|
||||
geom geometry(Point,4326),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (imei, reading_time)
|
||||
);
|
||||
|
||||
COMMENT ON TABLE tracksolid.fuel_readings
|
||||
IS 'Fuel/oil sensor readings from /pushoil webhook. Unit varies per sensor: cm | % | V | L.';
|
||||
COMMENT ON COLUMN tracksolid.fuel_readings.sensor_path
|
||||
IS 'Sensor channel identifier from the device (path field in API payload)';
|
||||
COMMENT ON COLUMN tracksolid.fuel_readings.unit
|
||||
IS 'Measurement unit: cm (tank depth), % (percentage), V (voltage), L (litres)';
|
||||
|
||||
SELECT create_hypertable(
|
||||
'tracksolid.fuel_readings', 'reading_time',
|
||||
chunk_time_interval => INTERVAL '7 days',
|
||||
if_not_exists => TRUE
|
||||
);
|
||||
|
||||
-- ── 6. Temperature & humidity readings (webhook /pushtem) — hypertable ────────
|
||||
|
||||
CREATE TABLE IF NOT EXISTS tracksolid.temperature_readings (
|
||||
imei TEXT NOT NULL REFERENCES tracksolid.devices(imei),
|
||||
reading_time TIMESTAMPTZ NOT NULL,
|
||||
temperature NUMERIC(6,2),
|
||||
humidity_pct NUMERIC(5,2),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (imei, reading_time)
|
||||
);
|
||||
|
||||
COMMENT ON TABLE tracksolid.temperature_readings
|
||||
IS 'Temperature and humidity sensor readings from /pushtem webhook. For cold-chain / refrigerated cargo monitoring.';
|
||||
|
||||
SELECT create_hypertable(
|
||||
'tracksolid.temperature_readings', 'reading_time',
|
||||
chunk_time_interval => INTERVAL '7 days',
|
||||
if_not_exists => TRUE
|
||||
);
|
||||
|
||||
-- ── 7. LBS / cell-tower fallback positions (webhook /pushlbs) ────────────────
|
||||
|
||||
CREATE TABLE IF NOT EXISTS tracksolid.lbs_readings (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
imei TEXT NOT NULL REFERENCES tracksolid.devices(imei),
|
||||
gate_time TIMESTAMPTZ NOT NULL,
|
||||
post_type TEXT,
|
||||
lbs_data JSONB,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
UNIQUE (imei, gate_time)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_lbs_readings_imei_time
|
||||
ON tracksolid.lbs_readings (imei, gate_time DESC);
|
||||
|
||||
COMMENT ON TABLE tracksolid.lbs_readings
|
||||
IS 'Cell tower / WiFi positioning fallback data from /pushlbs webhook. Used when GPS signal is unavailable.';
|
||||
COMMENT ON COLUMN tracksolid.lbs_readings.post_type
|
||||
IS 'Positioning technology: WIFI | LBS (cell tower)';
|
||||
COMMENT ON COLUMN tracksolid.lbs_readings.lbs_data
|
||||
IS 'Raw JSON payload containing MCC, MNC, and cell tower list for approximate geocoding.';
|
||||
|
||||
-- ── 8. Geofence definitions ───────────────────────────────────────────────────
|
||||
|
||||
CREATE TABLE IF NOT EXISTS tracksolid.geofences (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
fence_id TEXT UNIQUE,
|
||||
fence_name TEXT NOT NULL,
|
||||
fence_type TEXT,
|
||||
geom geometry(Geometry,4326),
|
||||
radius_m NUMERIC(10,2),
|
||||
description TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
COMMENT ON TABLE tracksolid.geofences
|
||||
IS 'Geofence boundary definitions synced from the Tracksolid platform.';
|
||||
COMMENT ON COLUMN tracksolid.geofences.fence_type
|
||||
IS 'circle | polygon';
|
||||
COMMENT ON COLUMN tracksolid.geofences.radius_m
|
||||
IS 'Radius in metres — only applicable for circle type geofences';
|
||||
|
||||
-- ── 9. Expand dwh_gold.fact_daily_fleet_metrics ───────────────────────────────
|
||||
|
||||
ALTER TABLE dwh_gold.fact_daily_fleet_metrics
|
||||
ADD COLUMN IF NOT EXISTS total_distance_km NUMERIC(12,3),
|
||||
ADD COLUMN IF NOT EXISTS total_trips INTEGER,
|
||||
ADD COLUMN IF NOT EXISTS total_drive_hours NUMERIC(8,2),
|
||||
ADD COLUMN IF NOT EXISTS total_idle_hours NUMERIC(8,2),
|
||||
ADD COLUMN IF NOT EXISTS fuel_consumed_l NUMERIC(10,3),
|
||||
ADD COLUMN IF NOT EXISTS alarm_count INTEGER,
|
||||
ADD COLUMN IF NOT EXISTS overspeed_count INTEGER,
|
||||
ADD COLUMN IF NOT EXISTS day_start_time TIME,
|
||||
ADD COLUMN IF NOT EXISTS day_end_time TIME,
|
||||
ADD COLUMN IF NOT EXISTS avg_speed_kmh NUMERIC(7,2),
|
||||
ADD COLUMN IF NOT EXISTS peak_speed_kmh NUMERIC(7,2);
|
||||
|
||||
COMMENT ON COLUMN dwh_gold.fact_daily_fleet_metrics.total_distance_km IS 'Total km driven that day across all trips';
|
||||
COMMENT ON COLUMN dwh_gold.fact_daily_fleet_metrics.total_trips IS 'Number of completed trips';
|
||||
COMMENT ON COLUMN dwh_gold.fact_daily_fleet_metrics.total_drive_hours IS 'Total hours of active driving (engine on + moving)';
|
||||
COMMENT ON COLUMN dwh_gold.fact_daily_fleet_metrics.total_idle_hours IS 'Total hours engine on but stationary';
|
||||
COMMENT ON COLUMN dwh_gold.fact_daily_fleet_metrics.fuel_consumed_l IS 'Total fuel consumed in litres (from webhook trip reports)';
|
||||
COMMENT ON COLUMN dwh_gold.fact_daily_fleet_metrics.alarm_count IS 'Total alarm events triggered that day';
|
||||
COMMENT ON COLUMN dwh_gold.fact_daily_fleet_metrics.overspeed_count IS 'Number of overspeed alarm events';
|
||||
COMMENT ON COLUMN dwh_gold.fact_daily_fleet_metrics.day_start_time IS 'Time of first trip start (Africa/Nairobi)';
|
||||
COMMENT ON COLUMN dwh_gold.fact_daily_fleet_metrics.day_end_time IS 'Time of last trip end (Africa/Nairobi)';
|
||||
COMMENT ON COLUMN dwh_gold.fact_daily_fleet_metrics.avg_speed_kmh IS 'Fleet average speed across all trips that day';
|
||||
COMMENT ON COLUMN dwh_gold.fact_daily_fleet_metrics.peak_speed_kmh IS 'Highest max_speed_kmh recorded across all trips';
|
||||
|
||||
-- ── 10. ETL function — refresh daily metrics ──────────────────────────────────
|
||||
-- Populates dwh_gold.fact_daily_fleet_metrics for a given date.
|
||||
-- Call nightly: SELECT dwh_gold.refresh_daily_metrics(CURRENT_DATE - 1);
|
||||
|
||||
CREATE OR REPLACE FUNCTION dwh_gold.refresh_daily_metrics(target_date DATE)
|
||||
RETURNS void LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
INSERT INTO dwh_gold.fact_daily_fleet_metrics (
|
||||
day,
|
||||
vehicle_key,
|
||||
total_distance_km,
|
||||
total_trips,
|
||||
total_drive_hours,
|
||||
total_idle_hours,
|
||||
fuel_consumed_l,
|
||||
alarm_count,
|
||||
overspeed_count,
|
||||
day_start_time,
|
||||
day_end_time,
|
||||
avg_speed_kmh,
|
||||
peak_speed_kmh
|
||||
)
|
||||
SELECT
|
||||
target_date AS day,
|
||||
t.imei AS vehicle_key,
|
||||
ROUND(SUM(t.distance_km)::numeric, 3) AS total_distance_km,
|
||||
COUNT(*) AS total_trips,
|
||||
ROUND((SUM(t.driving_time_s) / 3600.0)::numeric, 2) AS total_drive_hours,
|
||||
ROUND((SUM(t.idle_time_s) / 3600.0)::numeric, 2) AS total_idle_hours,
|
||||
ROUND(SUM(t.fuel_consumed_l)::numeric, 3) AS fuel_consumed_l,
|
||||
COUNT(a.id) AS alarm_count,
|
||||
COUNT(a.id) FILTER (WHERE a.alarm_type ILIKE '%speed%') AS overspeed_count,
|
||||
MIN(t.start_time AT TIME ZONE 'Africa/Nairobi')::TIME AS day_start_time,
|
||||
MAX(t.end_time AT TIME ZONE 'Africa/Nairobi')::TIME AS day_end_time,
|
||||
ROUND(AVG(t.avg_speed_kmh)::numeric, 2) AS avg_speed_kmh,
|
||||
MAX(t.max_speed_kmh) AS peak_speed_kmh
|
||||
FROM tracksolid.trips t
|
||||
LEFT JOIN tracksolid.alarms a
|
||||
ON a.imei = t.imei
|
||||
AND DATE(a.alarm_time AT TIME ZONE 'Africa/Nairobi') = target_date
|
||||
WHERE DATE(t.start_time AT TIME ZONE 'Africa/Nairobi') = target_date
|
||||
AND t.end_time IS NOT NULL
|
||||
GROUP BY t.imei
|
||||
ON CONFLICT (day, vehicle_key) DO UPDATE SET
|
||||
total_distance_km = EXCLUDED.total_distance_km,
|
||||
total_trips = EXCLUDED.total_trips,
|
||||
total_drive_hours = EXCLUDED.total_drive_hours,
|
||||
total_idle_hours = EXCLUDED.total_idle_hours,
|
||||
fuel_consumed_l = EXCLUDED.fuel_consumed_l,
|
||||
alarm_count = EXCLUDED.alarm_count,
|
||||
overspeed_count = EXCLUDED.overspeed_count,
|
||||
day_start_time = EXCLUDED.day_start_time,
|
||||
day_end_time = EXCLUDED.day_end_time,
|
||||
avg_speed_kmh = EXCLUDED.avg_speed_kmh,
|
||||
peak_speed_kmh = EXCLUDED.peak_speed_kmh;
|
||||
END;
|
||||
$$;
|
||||
|
||||
COMMENT ON FUNCTION dwh_gold.refresh_daily_metrics(DATE)
|
||||
IS 'Populates or refreshes fact_daily_fleet_metrics for the given date. '
|
||||
'Call nightly: SELECT dwh_gold.refresh_daily_metrics(CURRENT_DATE - 1);';
|
||||
|
||||
COMMIT;
|
||||
335
260410_baseline_report.md
Normal file
335
260410_baseline_report.md
Normal file
|
|
@ -0,0 +1,335 @@
|
|||
# Fireside Communications — Fleet Baseline Report
|
||||
## Date: 2026-04-10 · Database: tracksolid_db · Generated: 23:18 EAT
|
||||
|
||||
> **Baseline snapshot taken on the first night of active pipeline operation.**
|
||||
> Container: `timescale_db-bo3nov2ija7g8wn9b1g2paxs-195053614609`
|
||||
> Ingestion has been live for approximately 1 hour at time of capture.
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Summary
|
||||
|
||||
| Metric | Value | Status |
|
||||
|---|---|---|
|
||||
| Total registered devices | 63 | — |
|
||||
| Devices with live position | 19 (30%) | ⚠ 44 devices never reported |
|
||||
| Devices active today | 4 | ⚠ Low — evening snapshot |
|
||||
| Position history rows | 28 | ⚠ Pipeline started today |
|
||||
| Trip records | 0 | ❌ Migration 04 not yet applied |
|
||||
| Alarm records | 0 | ❌ No alarms ingested yet |
|
||||
| Parking events | 0 | ⚠ API returning 0 (fix deployed, containers not redeployed) |
|
||||
| Ingestion pipeline health | ✅ Running | 60-second polling confirmed |
|
||||
| Migration 04 applied (distance_km rename) | ❌ No | `distance_m` column still present |
|
||||
| Migration 05 applied (new tables) | ❌ No | New tables don't exist yet |
|
||||
|
||||
**Key finding:** The ingestion pipeline started successfully tonight and is polling correctly. However, only 19 of 63 devices are returning live positions from the Tracksolid API. The remaining 44 devices are registered in the system but have never reported a GPS fix — they may be inactive, uninstalled, or require account-level configuration in Tracksolid Pro.
|
||||
|
||||
---
|
||||
|
||||
## 2. Fleet Composition
|
||||
|
||||
### Device types registered
|
||||
|
||||
| Device Model | Count | With SIM | With Odometer | Notes |
|
||||
|---|---|---|---|---|
|
||||
| AT4 | 23 | 3 | 3 | Oldest fleet — mostly blank device names |
|
||||
| JC400P | 23 | 2 | 6 | Camera-equipped trackers |
|
||||
| X3 | 10 | 4 | 6 | Newest devices (2025–2026 activations) |
|
||||
| GT06E | 7 | 5 | 5 | Mid-fleet — best data quality |
|
||||
| **Total** | **63** | **14** | **20** | |
|
||||
|
||||
**Observations:**
|
||||
- Only 14 of 63 devices have a SIM number recorded (22%)
|
||||
- Only 20 of 63 devices have an odometer reading (32%)
|
||||
- All 63 `vehicle_name`, `vehicle_number`, and `driver_name` fields are blank — reports currently show device names only
|
||||
|
||||
### Named vehicles with odometer (highest mileage first)
|
||||
|
||||
| Device Name | IMEI | Type | Odometer (km) | Activated | SIM |
|
||||
|---|---|---|---|---|---|
|
||||
| KDK 829A GP | 359857082898297 | GT06E | 239,264 | 2022-10-29 | 0707923872 |
|
||||
| Belta KCU-647D | 359857082042862 | GT06E | 234,546 | 2020-04-03 | 0110094465 |
|
||||
| JK Subaru KCS 903Y | 359857081891921 | GT06E | 73,344 | 2019-06-12 | 0746759925 |
|
||||
| KCU 865Q Vanguard | 359857082042953 | GT06E | 61,758 | 2019-12-20 | 0757270763 |
|
||||
| KCU 145Q Solo Xtrail | 359857082037425 | GT06E | 53,228 | 2019-12-20 | 0757270810 |
|
||||
| KCU 865Q Vanguard Sub | 353549090555334 | AT4 | 9,656 | 2019-12-20 | 0757270804 |
|
||||
| KMGR 409U HENRY JAZZ | 865135061048300 | X3 | 6,696 | 2025-07-31 | 0768697302 |
|
||||
| KDU 878T_Track | 865135061040349 | X3 | 4,802 | 2025-08-18 | 0708352823 |
|
||||
| KCS 903Y JK SUB | 353549090552018 | AT4 | 4,492 | 2019-06-09 | 0700024569 |
|
||||
| X3-63282 | 865135061563282 | X3 | 4,194 | 2026-02-14 | — |
|
||||
| KMEH 692C KAWASAKI | 353549090561654 | AT4 | 3,319 | 2020-04-03 | 0110094467 |
|
||||
| FRED KMGW 538W HULETI | 865135061559538 | X3 | 2,284 | 2026-02-08 | 0119867174 |
|
||||
| KDU 878T_CAM | 862798052715071 | JC400P | 1,562 | 2025-08-18 | 0708351897 |
|
||||
| KDW 632M HL Tracker | 865135061569529 | X3 | 222 | 2026-02-09 | 300002396033 IoT |
|
||||
| JC400P-85751 | 862798052785751 | JC400P | 17 | 2026-03-11 | — |
|
||||
| X3-68968 | 865135061568968 | X3 | 16 | 2026-03-11 | — |
|
||||
| KDW 632M HL Cam | 862798052707995 | JC400P | 16 | 2026-03-11 | 300002396032 IoT |
|
||||
|
||||
> **Note:** `KDK 829A GP` (239,264 km) and `Belta KCU-647D` (234,546 km) are high-mileage vehicles that should be reviewed for service intervals. At typical service intervals of 10,000 km, both are well overdue unless recently serviced.
|
||||
|
||||
---
|
||||
|
||||
## 3. Live Positions — Current Fleet Snapshot
|
||||
|
||||
### Position freshness at 23:18 EAT, 2026-04-10
|
||||
|
||||
| Freshness Band | Vehicles |
|
||||
|---|---|
|
||||
| Fresh — last fix < 10 minutes ago | **3** |
|
||||
| Today — last fix within 24 hours | **1** |
|
||||
| This week — last fix within 7 days | **2** |
|
||||
| This month — last fix within 30 days | **3** |
|
||||
| Stale — last fix older than 30 days | **10** |
|
||||
| No position recorded | **44** |
|
||||
|
||||
### Vehicles with fresh GPS fix (reporting now)
|
||||
|
||||
| Device Name | IMEI | Last Fix (EAT) | Speed | Acc | Odometer | Coordinates | Location |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| FRED KMGW 538W HULETI | 865135061559538 | 23:16:00 | 0 km/h | Off | 2,283.89 km | -1.237, 36.727 | Nairobi (Westlands area) |
|
||||
| X3-63282 | 865135061563282 | 23:15:15 | 0 km/h | Off | 4,194.08 km | 0.196, 32.540 | **Uganda (Kampala/Entebbe area)** |
|
||||
| KDK 829A GP | 359857082898297 | 23:13:20 | 0 km/h | Off | 239,263.53 km | -1.328, 36.900 | Nairobi South / Athi River area |
|
||||
|
||||
> ⚠️ **X3-63282 is currently in Uganda** (lat 0.196, lng 32.540 — near Kampala/Entebbe). If this vehicle is not expected to be cross-border, this warrants investigation.
|
||||
|
||||
### All vehicles that reported today (2026-04-10)
|
||||
|
||||
| Device Name | Last Fix (EAT) | Speed | Location |
|
||||
|---|---|---|---|
|
||||
| FRED KMGW 538W HULETI | 23:16 | 0 km/h | Nairobi Westlands |
|
||||
| X3-63282 | 23:15 | 0 km/h | Uganda |
|
||||
| KDK 829A GP | 23:13 | 0 km/h | Nairobi South |
|
||||
| KMGR 409U HENRY JAZZ | 15:40 | 1 km/h | Nairobi Westlands |
|
||||
|
||||
All 4 active vehicles have ignition off (`acc_status = 0`) — fleet is parked as of report time.
|
||||
|
||||
### Geographic clusters of tracked fleet
|
||||
|
||||
| Approximate Area | Lat/Lng | Vehicles |
|
||||
|---|---|---|
|
||||
| Nairobi — Westlands / Upper Hill | -1.24, 36.73 | 7 |
|
||||
| Uganda — Kampala / Entebbe area | 0.20, 32.54 | 1 |
|
||||
| Nairobi South — Athi River / Mlolongo | -1.33, 36.90 | 1 |
|
||||
|
||||
---
|
||||
|
||||
## 4. Devices Not Reporting (44 of 63)
|
||||
|
||||
The following 44 devices are registered but have **never returned a GPS position** since the pipeline started. This is the most significant operational gap identified at baseline.
|
||||
|
||||
| IMEI | Device Name | Type |
|
||||
|---|---|---|
|
||||
| 353549090551820 | AT4-51820 | AT4 |
|
||||
| 353549090553099 | AT4-53099 | AT4 |
|
||||
| 353549090554246 | AT4-54246 | AT4 |
|
||||
| 353549090555029 | AT4-55029 | AT4 |
|
||||
| 353549090555235 | AT4-55235 | AT4 |
|
||||
| 353549090557389 | AT4-57389 | AT4 |
|
||||
| 353549090561860 | AT4-61860 | AT4 |
|
||||
| 353549090564823 | AT4-64823 | AT4 |
|
||||
| 353549090564880 | AT4-64880 | AT4 |
|
||||
| 353549090564989 | AT4-64989 | AT4 |
|
||||
| 353549090565010 | AT4-65010 | AT4 |
|
||||
| 353549090565135 | AT4-65135 | AT4 |
|
||||
| 353549090565341 | AT4-65341 | AT4 |
|
||||
| 353549090565598 | AT4-65598 | AT4 |
|
||||
| 353549090565648 | AT4-65648 | AT4 |
|
||||
| 353549090566158 | AT4-66158 | AT4 |
|
||||
| 353549090567271 | AT4-67271 | AT4 |
|
||||
| 353549090567693 | AT4-67693 | AT4 |
|
||||
| 359857081885428 | GT06E-85428 | GT06E |
|
||||
| 359857081886319 | GT06E-86319 | GT06E |
|
||||
| 862798052707904 | JC400P-07904 | JC400P |
|
||||
| 862798052785041 | JC400P-85041 | JC400P |
|
||||
| 862798052785058 | JC400P-85058 | JC400P |
|
||||
| 862798052786403 | JC400P-86403 | JC400P |
|
||||
| 862798052787625 | JC400P-87625 | JC400P |
|
||||
| 862798052787831 | JC400P-87831 | JC400P |
|
||||
| 862798052789530 | JC400P-89530 | JC400P |
|
||||
| 862798052789563 | JC400P-89563 | JC400P |
|
||||
| 862798052789662 | JC400P-89662 | JC400P |
|
||||
| 862798052789977 | JC400P-89977 | JC400P |
|
||||
| 862798052790108 | JC400P-90108 | JC400P |
|
||||
| 862798052790199 | JC400P-90199 | JC400P |
|
||||
| 862798052790678 | JC400P-90678 | JC400P |
|
||||
| 862798052792278 | JC400P-92278 | JC400P |
|
||||
| 862798052792716 | JC400P-92716 | JC400P |
|
||||
| 862798052792732 | JC400P-92732 | JC400P |
|
||||
| 862798052794233 | JC400P-94233 | JC400P |
|
||||
| 865135061559405 | X3-59405 | X3 |
|
||||
| 865135061564223 | X3-64223 | X3 |
|
||||
| 865135061569172 | X3-69172 | X3 |
|
||||
| 865135061578553 | X3-78553 | X3 |
|
||||
| 862798052786270 | JC400P-86270 | JC400P |
|
||||
| 862798052789431 | JC400P-89431 | JC400P |
|
||||
| 862798052791619 | JC400P-91619 | JC400P |
|
||||
|
||||
**Likely causes:**
|
||||
- Device powered off or SIM deactivated
|
||||
- Device registered in Tracksolid Pro but never activated in the field
|
||||
- Account-level permission: device may belong to a sub-account not accessible under the API credentials in use
|
||||
- Physical tracker fault or uninstalled from vehicle
|
||||
|
||||
**Recommended action:** Cross-reference this list against the physical fleet inventory. For any device that should be active, log into the Tracksolid Pro web console and verify the device is online.
|
||||
|
||||
---
|
||||
|
||||
## 5. Ingestion Pipeline Health
|
||||
|
||||
### Last 30 ingestion events (as at 23:18 EAT)
|
||||
|
||||
| Endpoint | IMEIs Queried | Rows Upserted | Rows Inserted | Duration | Status |
|
||||
|---|---|---|---|---|---|
|
||||
| `jimi.user.device.location.list` | 63 | 19 | 19 | ~200 ms | ✅ |
|
||||
| `jimi.user.device.location.list` | 63 | 19 | 19 | ~240 ms | ✅ |
|
||||
| `jimi.open.platform.report.parking` | 50+13 | 0 | 0 | ~7–15s | ⚠ 0 rows |
|
||||
| `jimi.user.device.list+detail` | 63 | 63 | 0 | 66,115 ms | ✅ |
|
||||
|
||||
**Observations:**
|
||||
- Location polling running every 60 seconds — healthy and consistent
|
||||
- All API calls returning success (`t`)
|
||||
- Parking endpoint responding but returning 0 rows — the updated container with `acc_type=0` and `durSecond` fix has not yet been redeployed
|
||||
- `jimi.device.track.list` (POLL-01 high-resolution trail) not yet appearing in logs — new container not yet deployed
|
||||
- Trip polling (`jimi.device.track.mileage`) not yet appearing — new container not yet deployed
|
||||
- Device sync completed at 22:54 (66 seconds for 63 devices with detail lookups — expected)
|
||||
|
||||
---
|
||||
|
||||
## 6. Schema & Migration Status
|
||||
|
||||
| Migration | Description | Status |
|
||||
|---|---|---|
|
||||
| 01–03 | Base schema, webhook tables, position_history columns | ✅ Applied |
|
||||
| **04** | `distance_m` → `distance_km` rename + historical data correction | ❌ **Not applied** |
|
||||
| **05** | New tables: device_events, fuel_readings, temperature_readings, lbs_readings, geofences; OBD/alarm/device enrichment columns; dwh_gold expansion | ❌ **Not applied** |
|
||||
|
||||
**Confirmed:** `trips.distance_m` column still exists (not yet renamed to `distance_km`). Migration 04 must be run before deploying updated ingestion containers — failure to do so will cause the new code to write to a column that doesn't exist.
|
||||
|
||||
**Tables present in `tracksolid` schema:**
|
||||
|
||||
| Table | Rows |
|
||||
|---|---|
|
||||
| `devices` | 63 |
|
||||
| `live_positions` | 19 |
|
||||
| `position_history` | 28 |
|
||||
| `trips` | 0 |
|
||||
| `alarms` | 0 |
|
||||
| `parking_events` | 0 |
|
||||
| `obd_readings` | 0 |
|
||||
| `heartbeats` | 0 |
|
||||
| `ingestion_log` | 29+ |
|
||||
| `api_token_cache` | — |
|
||||
| `fault_codes` | — |
|
||||
|
||||
**Tables NOT yet present (require migration 05):**
|
||||
- `device_events`
|
||||
- `fuel_readings`
|
||||
- `temperature_readings`
|
||||
- `lbs_readings`
|
||||
- `geofences`
|
||||
|
||||
---
|
||||
|
||||
## 7. Stale Devices — Historical Last-Seen
|
||||
|
||||
The following devices have a live_positions entry but their last GPS fix is more than 30 days old:
|
||||
|
||||
| Device Name | IMEI | Last Fix (EAT) | Odometer | Notes |
|
||||
|---|---|---|---|---|
|
||||
| KCS 903Y JK SUB | 353549090552018 | 2024-07-16 10:41 | 4,492 km | ~21 months stale |
|
||||
| KCU 865Q Vanguard Sub | 353549090555334 | 2024-07-07 10:43 | 9,656 km | ~21 months stale |
|
||||
| KMEH 692C KAWASAKI | 353549090561654 | 2023-06-17 10:41 | 3,319 km | ~34 months stale |
|
||||
| KCE 690F | 353549090565580 | 2019-09-27 07:20 | 0 km | ~6.5 years stale |
|
||||
| KDU 878T_CAM | 862798052715071 | 2025-12-04 15:27 | 1,562 km | ~4 months stale |
|
||||
| KCU 145Q Solo Xtrail | 359857082037425 | 2025-06-01 14:04 | 53,228 km | ~10 months stale |
|
||||
| Belta KCU-647D | 359857082042862 | 2025-05-30 23:53 | 234,546 km | ~10 months stale |
|
||||
| KDW 632M HL Cam | 862798052707995 | 2026-03-11 11:52 | 16 km | 30 days — may need SIM activation |
|
||||
| KDW 632M HL Tracker | 865135061569529 | 2026-03-11 23:53 | 222 km | 30 days — may need SIM activation |
|
||||
| JC400P-85751 | 862798052785751 | 2026-03-11 22:15 | 17 km | 30 days — brand new, 17 km only |
|
||||
| AT4-64815 | 353549090564815 | 2026-02-05 11:19 | 0 km | 64 days stale |
|
||||
| JK Subaru KCS 903Y | 359857081891921 | 2026-03-14 00:55 | 73,344 km | 27 days |
|
||||
|
||||
---
|
||||
|
||||
## 8. Pending Actions Before Full Operation
|
||||
|
||||
The following steps are required to move from baseline to fully operational. Listed in execution order:
|
||||
|
||||
| Priority | Action | Impact |
|
||||
|---|---|---|
|
||||
| 🔴 1 | **Run migration 04** on production DB | Renames `distance_m` → `distance_km`; corrects historical data |
|
||||
| 🔴 2 | **Run migration 05** on production DB | Creates new tables for expanded ingestion |
|
||||
| 🔴 3 | **Redeploy updated ingestion containers** | Activates: trip polling, parking fix, high-res GPS trails, alarm field fix |
|
||||
| 🟠 4 | **Investigate 44 non-reporting devices** | Cross-check against physical fleet; verify online in Tracksolid Pro console |
|
||||
| 🟠 5 | **Investigate cross-border vehicle** | X3-63282 last seen in Uganda — confirm if authorised |
|
||||
| 🟠 6 | **Register webhooks** in Tracksolid Pro account | Activates: /pushobd, /pushoil, /pushtem, /pushlbs, /pushevent, /pushtripreport |
|
||||
| 🟡 7 | **Populate vehicle_name, vehicle_number, driver_name** | All 63 devices currently blank — reports show device names only |
|
||||
| 🟡 8 | **Set fuel_100km** per vehicle | Unlocks idle fuel cost calculations |
|
||||
| 🟡 9 | **Review high-mileage vehicles** for service | KDK 829A GP (239k km) and Belta KCU-647D (234k km) |
|
||||
| 🟢 10 | **Schedule nightly ETL** | `SELECT dwh_gold.refresh_daily_metrics(CURRENT_DATE - 1);` via cron or n8n |
|
||||
|
||||
### Commands for steps 1–3
|
||||
|
||||
```bash
|
||||
# SSH to server first
|
||||
ssh kianiadee@stage.rahamafresh.com
|
||||
|
||||
# Resolve container
|
||||
TS_DB=$(docker ps --filter "name=timescale_db" --format "{{.Names}}" | head -1)
|
||||
|
||||
# Step 1 — migration 04 (distance correction)
|
||||
docker exec -i "$TS_DB" psql -U postgres -d tracksolid_db \
|
||||
< /path/to/04_bug_fix_migration.sql
|
||||
|
||||
# Step 2 — migration 05 (new tables)
|
||||
docker exec -i "$TS_DB" psql -U postgres -d tracksolid_db \
|
||||
< /path/to/05_enhancement_migration.sql
|
||||
|
||||
# Step 3 — redeploy containers
|
||||
cd /path/to/compose
|
||||
docker compose up -d --build ingest_movement ingest_events webhook_receiver
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Devices That Reported This Week vs Last Month
|
||||
|
||||
### Active in last 7 days
|
||||
|
||||
| Device Name | IMEI | Last Fix | Odometer |
|
||||
|---|---|---|---|
|
||||
| FRED KMGW 538W HULETI | 865135061559538 | 2026-04-10 23:16 | 2,284 km |
|
||||
| X3-63282 | 865135061563282 | 2026-04-10 23:15 | 4,194 km |
|
||||
| KDK 829A GP | 359857082898297 | 2026-04-10 23:13 | 239,264 km |
|
||||
| KMGR 409U HENRY JAZZ | 865135061048300 | 2026-04-10 15:40 | 6,696 km |
|
||||
| KCU 865Q Vanguard | 359857082042953 | 2026-04-08 17:17 | 61,758 km |
|
||||
| KDU 878T_Track | 865135061040349 | 2026-04-08 17:16 | 4,802 km |
|
||||
|
||||
### Active in last 30 days (in addition to above)
|
||||
|
||||
| Device Name | IMEI | Last Fix | Odometer |
|
||||
|---|---|---|---|
|
||||
| JK Subaru KCS 903Y | 359857081891921 | 2026-03-14 00:55 | 73,344 km |
|
||||
| KDW 632M HL Tracker | 865135061569529 | 2026-03-11 23:53 | 222 km |
|
||||
| JC400P-85751 | 862798052785751 | 2026-03-11 22:15 | 17 km |
|
||||
| KDW 632M HL Cam | 862798052707995 | 2026-03-11 11:52 | 16 km |
|
||||
|
||||
---
|
||||
|
||||
## 10. What This Report Will Look Like in 7 Days
|
||||
|
||||
Once migrations 04 and 05 are applied and updated containers are deployed, the next weekly report will include:
|
||||
|
||||
- **Trip records** per vehicle per day — distance driven, drive/idle hours, avg and max speed
|
||||
- **Parking events** — where vehicles stopped, how long, address
|
||||
- **Alarm events** — overspeed, geofence, harshness flags with correct type names
|
||||
- **High-resolution position trails** — 2–6 GPS fixes per minute per active vehicle
|
||||
- **Driver scorecards** — km driven, alarms per 100 km, late starts
|
||||
|
||||
The data foundation is in place. The pipeline is running. This baseline establishes the starting point against which all future performance will be measured.
|
||||
|
||||
---
|
||||
|
||||
*Report generated from live database query · 2026-04-10 23:18 EAT*
|
||||
*Pipeline uptime at report time: ~1 hour*
|
||||
*Queries source: `tracksolid_DB_manual.md` · `01_BusinessAnalytics.md`*
|
||||
|
|
@ -3,6 +3,60 @@
|
|||
|
||||
---
|
||||
|
||||
## Database Access
|
||||
|
||||
The TimescaleDB container name includes a Coolify-generated suffix that changes on every redeploy. Use the shell function below instead of hardcoding the container name.
|
||||
|
||||
### Recommended: add to `~/.zshrc` on the server
|
||||
|
||||
```bash
|
||||
# Auto-resolves current TimescaleDB container — survives Coolify redeployments
|
||||
tsdb() {
|
||||
local container
|
||||
container=$(docker ps --filter "name=timescale_db" --format "{{.Names}}" | head -1)
|
||||
if [[ -z "$container" ]]; then
|
||||
echo "ERROR: no running timescale_db container found" >&2
|
||||
return 1
|
||||
fi
|
||||
docker exec -it "$container" psql -U postgres -d tracksolid_db "$@"
|
||||
}
|
||||
```
|
||||
|
||||
After adding it: `source ~/.zshrc`
|
||||
|
||||
**Usage:**
|
||||
|
||||
```bash
|
||||
tsdb # open interactive psql prompt
|
||||
tsdb -c "\dt tracksolid.*" # list all tracksolid tables
|
||||
tsdb -c "SELECT COUNT(*) FROM tracksolid.trips;" # run a single query
|
||||
tsdb -c "SELECT dwh_gold.refresh_daily_metrics(CURRENT_DATE - 1);" # run nightly ETL
|
||||
```
|
||||
|
||||
### One-liner (for scripts and migrations)
|
||||
|
||||
```bash
|
||||
TS_DB=$(docker ps --filter "name=timescale_db" --format "{{.Names}}" | head -1)
|
||||
docker exec -i "$TS_DB" psql -U postgres -d tracksolid_db < migration.sql
|
||||
```
|
||||
|
||||
> Use `-i` (not `-it`) when piping SQL files — the `-t` TTY flag conflicts with stdin redirection.
|
||||
|
||||
### If multiple TimescaleDB containers are running
|
||||
|
||||
Filter by Coolify label instead of name to target a specific instance:
|
||||
|
||||
```bash
|
||||
# Inspect available labels first
|
||||
docker inspect $(docker ps --filter "name=timescale_db" --format "{{.Names}}" | head -1) \
|
||||
--format '{{json .Config.Labels}}' | jq .
|
||||
|
||||
# Then filter by a stable label, e.g.:
|
||||
TS_DB=$(docker ps --filter "label=coolify.name=timescale_db" --format "{{.Names}}" | head -1)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. Service Architecture
|
||||
|
||||
```
|
||||
|
|
|
|||
14
administration/ingest_movement_dockerps.md
Normal file
14
administration/ingest_movement_dockerps.md
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
docker logs -f ingest_events-bo3nov2ija7g8wn9b1g2paxs-162026778012
|
||||
docker logs -f ingest_movement-bo3nov2ija7g8wn9b1g2paxs-162026773516
|
||||
|
||||
Or to see both at once with labels, use --tail to get the last N lines:
|
||||
|
||||
# Last 50 lines from each
|
||||
docker logs --tail 50 ingest_events-bo3nov2ija7g8wn9b1g2paxs-162026778012
|
||||
docker logs --tail 50 ingest_movement-bo3nov2ija7g8wn9b1g2paxs-162026773516
|
||||
|
||||
To follow both simultaneously in split view:
|
||||
|
||||
# Option 1: Follow both in one terminal (prefixed)
|
||||
docker logs -f ingest_events-bo3nov2ija7g8wn9b1g2paxs-162026778012 2>&1 | sed 's/^/[EVENTS] /' &
|
||||
docker logs -f ingest_movement-bo3nov2ija7g8wn9b1g2paxs-162026773516 2>&1 | sed 's/^/[MOVEMENT] /'
|
||||
|
|
@ -60,10 +60,14 @@ services:
|
|||
depends_on:
|
||||
timescale_db:
|
||||
condition: service_healthy
|
||||
env_file: .env
|
||||
environment:
|
||||
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}
|
||||
- GF_USERS_DEFAULT_THEME=dark
|
||||
- GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/etc/grafana/provisioning/dashboards-json/noc_fleet_dashboard.json
|
||||
volumes:
|
||||
- grafana-data:/var/lib/grafana
|
||||
- ./grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
# COOLIFY DOMAIN LOGIC:
|
||||
# You will set the actual URL in the Coolify UI,
|
||||
# but the service needs to expose port 3000 internally.
|
||||
|
|
|
|||
589
grafana/provisioning/dashboards-json/noc_fleet_dashboard.json
Normal file
589
grafana/provisioning/dashboards-json/noc_fleet_dashboard.json
Normal file
|
|
@ -0,0 +1,589 @@
|
|||
{
|
||||
"title": "NOC Fleet Operations — Live",
|
||||
"uid": "noc-fleet-live",
|
||||
"schemaVersion": 39,
|
||||
"version": 1,
|
||||
"refresh": "30s",
|
||||
"time": { "from": "now-1h", "to": "now" },
|
||||
"timepicker": {
|
||||
"refresh_intervals": ["10s", "30s", "1m", "5m"]
|
||||
},
|
||||
"editable": false,
|
||||
"tags": ["noc", "fleet", "live"],
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 0,
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"type": "stat",
|
||||
"title": "Total Vehicles",
|
||||
"gridPos": { "x": 0, "y": 0, "w": 4, "h": 3 },
|
||||
"datasource": { "type": "postgres", "uid": "tracksolid_pg" },
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
|
||||
"textMode": "auto"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": { "mode": "thresholds" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [{ "color": "text", "value": null }]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": { "type": "postgres", "uid": "tracksolid_pg" },
|
||||
"rawSql": "SELECT COUNT(*) AS \"Total Vehicles\"\nFROM tracksolid.devices WHERE enabled_flag = 1;",
|
||||
"format": "table",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "stat",
|
||||
"title": "Online Now",
|
||||
"description": "GPS fix within last 5 minutes",
|
||||
"gridPos": { "x": 4, "y": 0, "w": 4, "h": 3 },
|
||||
"datasource": { "type": "postgres", "uid": "tracksolid_pg" },
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
|
||||
"textMode": "auto"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": { "mode": "thresholds" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "red", "value": null },
|
||||
{ "color": "green", "value": 1 }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": { "type": "postgres", "uid": "tracksolid_pg" },
|
||||
"rawSql": "SELECT COUNT(*) AS \"Online\"\nFROM tracksolid.v_fleet_status\nWHERE connectivity_status = 'online';",
|
||||
"format": "table",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "stat",
|
||||
"title": "Recent (5-30 min)",
|
||||
"description": "GPS fix between 5 and 30 minutes ago",
|
||||
"gridPos": { "x": 8, "y": 0, "w": 4, "h": 3 },
|
||||
"datasource": { "type": "postgres", "uid": "tracksolid_pg" },
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
|
||||
"textMode": "auto"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": { "mode": "thresholds" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "text", "value": null },
|
||||
{ "color": "yellow", "value": 1 }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": { "type": "postgres", "uid": "tracksolid_pg" },
|
||||
"rawSql": "SELECT COUNT(*) AS \"Recent\"\nFROM tracksolid.v_fleet_status\nWHERE connectivity_status = 'recent';",
|
||||
"format": "table",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "stat",
|
||||
"title": "Offline",
|
||||
"description": "GPS fix older than 30 minutes",
|
||||
"gridPos": { "x": 12, "y": 0, "w": 4, "h": 3 },
|
||||
"datasource": { "type": "postgres", "uid": "tracksolid_pg" },
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
|
||||
"textMode": "auto"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": { "mode": "thresholds" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "red", "value": 1 }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": { "type": "postgres", "uid": "tracksolid_pg" },
|
||||
"rawSql": "SELECT COUNT(*) AS \"Offline\"\nFROM tracksolid.v_fleet_status\nWHERE connectivity_status = 'offline';",
|
||||
"format": "table",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "stat",
|
||||
"title": "Moving Now",
|
||||
"description": "Vehicles with speed > 0 and engine on",
|
||||
"gridPos": { "x": 16, "y": 0, "w": 4, "h": 3 },
|
||||
"datasource": { "type": "postgres", "uid": "tracksolid_pg" },
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
|
||||
"textMode": "auto"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": { "mode": "thresholds" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "text", "value": null },
|
||||
{ "color": "green", "value": 1 }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": { "type": "postgres", "uid": "tracksolid_pg" },
|
||||
"rawSql": "SELECT COUNT(*) AS \"Moving\"\nFROM tracksolid.v_fleet_status\nWHERE speed > 0 AND acc_status = '1' AND connectivity_status = 'online';",
|
||||
"format": "table",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "stat",
|
||||
"title": "Avg Speed (km/h)",
|
||||
"description": "Average speed of currently moving vehicles",
|
||||
"gridPos": { "x": 20, "y": 0, "w": 4, "h": 3 },
|
||||
"datasource": { "type": "postgres", "uid": "tracksolid_pg" },
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
|
||||
"textMode": "auto"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": { "mode": "thresholds" },
|
||||
"unit": "velocitykmh",
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "yellow", "value": 80 },
|
||||
{ "color": "red", "value": 120 }
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": { "type": "postgres", "uid": "tracksolid_pg" },
|
||||
"rawSql": "SELECT ROUND(AVG(speed)::numeric, 1) AS \"Avg Speed km/h\"\nFROM tracksolid.v_fleet_status\nWHERE speed > 0 AND acc_status = '1' AND connectivity_status = 'online';",
|
||||
"format": "table",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "geomap",
|
||||
"title": "Live Vehicle Locations",
|
||||
"gridPos": { "x": 0, "y": 3, "w": 24, "h": 16 },
|
||||
"datasource": { "type": "postgres", "uid": "tracksolid_pg" },
|
||||
"options": {
|
||||
"basemap": {
|
||||
"config": { "theme": "dark" },
|
||||
"name": "Basemap",
|
||||
"type": "carto"
|
||||
},
|
||||
"controls": {
|
||||
"mouseWheelZoom": true,
|
||||
"showAttribution": true,
|
||||
"showDebug": false,
|
||||
"showScale": true,
|
||||
"showZoom": true
|
||||
},
|
||||
"layers": [
|
||||
{
|
||||
"config": {
|
||||
"showLegend": false,
|
||||
"style": {
|
||||
"color": {
|
||||
"field": "vehicle_number",
|
||||
"fixed": "dark-green",
|
||||
"mode": "field"
|
||||
},
|
||||
"opacity": 1,
|
||||
"rotation": {
|
||||
"field": "direction",
|
||||
"fixed": 0,
|
||||
"max": 360,
|
||||
"min": -360,
|
||||
"mode": "field"
|
||||
},
|
||||
"size": {
|
||||
"fixed": 18,
|
||||
"max": 15,
|
||||
"min": 2,
|
||||
"mode": "fixed"
|
||||
},
|
||||
"symbol": {
|
||||
"fixed": "img/icons/marker/arrow-up.svg",
|
||||
"mode": "fixed"
|
||||
},
|
||||
"symbolAlign": {
|
||||
"h": "center",
|
||||
"v": "center"
|
||||
}
|
||||
}
|
||||
},
|
||||
"filterData": { "id": "byRefId", "options": "A" },
|
||||
"location": {
|
||||
"latitude": "lat",
|
||||
"longitude": "lng",
|
||||
"mode": "coords"
|
||||
},
|
||||
"name": "Vehicles",
|
||||
"tooltip": true,
|
||||
"type": "markers"
|
||||
}
|
||||
],
|
||||
"tooltip": { "mode": "details" },
|
||||
"view": {
|
||||
"allLayers": true,
|
||||
"id": "coords",
|
||||
"lat": -1.5,
|
||||
"lon": 36.5,
|
||||
"zoom": 6
|
||||
}
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": { "mode": "palette-classic-by-name" }
|
||||
},
|
||||
"overrides": [
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "lat" },
|
||||
"properties": [
|
||||
{ "id": "custom.hideFrom", "value": { "legend": true, "tooltip": true, "viz": false } }
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "lng" },
|
||||
"properties": [
|
||||
{ "id": "custom.hideFrom", "value": { "legend": true, "tooltip": true, "viz": false } }
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "imei" },
|
||||
"properties": [
|
||||
{ "id": "custom.hideFrom", "value": { "legend": true, "tooltip": true, "viz": false } }
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "direction" },
|
||||
"properties": [
|
||||
{ "id": "unit", "value": "degree" },
|
||||
{ "id": "displayName", "value": "Heading" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "speed" },
|
||||
"properties": [
|
||||
{ "id": "unit", "value": "velocitykmh" },
|
||||
{ "id": "displayName", "value": "Speed" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "vehicle_number" },
|
||||
"properties": [
|
||||
{ "id": "displayName", "value": "Plate" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "driver_name" },
|
||||
"properties": [
|
||||
{ "id": "displayName", "value": "Driver" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "driver_phone" },
|
||||
"properties": [
|
||||
{ "id": "displayName", "value": "Phone" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "vehicle_name" },
|
||||
"properties": [
|
||||
{ "id": "displayName", "value": "Vehicle" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "connectivity_status" },
|
||||
"properties": [
|
||||
{ "id": "displayName", "value": "Status" },
|
||||
{
|
||||
"id": "mappings",
|
||||
"value": [
|
||||
{
|
||||
"type": "value",
|
||||
"options": {
|
||||
"online": { "color": "green", "index": 0, "text": "Online" },
|
||||
"recent": { "color": "yellow", "index": 1, "text": "Recent" },
|
||||
"offline": { "color": "red", "index": 2, "text": "Offline" }
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "acc_status" },
|
||||
"properties": [
|
||||
{ "id": "displayName", "value": "ACC" },
|
||||
{
|
||||
"id": "mappings",
|
||||
"value": [
|
||||
{
|
||||
"type": "value",
|
||||
"options": {
|
||||
"1": { "text": "On", "color": "green", "index": 0 },
|
||||
"0": { "text": "Off", "color": "red", "index": 1 }
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "gps_time" },
|
||||
"properties": [
|
||||
{ "id": "displayName", "value": "Last Fix" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "loc_desc" },
|
||||
"properties": [
|
||||
{ "id": "displayName", "value": "Location" }
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": { "type": "postgres", "uid": "tracksolid_pg" },
|
||||
"rawSql": "SELECT\n d.imei,\n d.vehicle_number,\n d.vehicle_name,\n d.driver_name,\n d.driver_phone,\n d.city,\n d.device_group,\n lp.lat,\n lp.lng,\n lp.speed,\n lp.direction,\n lp.acc_status,\n lp.loc_desc,\n lp.gps_time,\n CASE\n WHEN lp.gps_time >= NOW() - INTERVAL '5 minutes' THEN 'online'\n WHEN lp.gps_time >= NOW() - INTERVAL '30 minutes' THEN 'recent'\n ELSE 'offline'\n END AS connectivity_status\nFROM tracksolid.devices d\nINNER JOIN tracksolid.live_positions lp USING (imei)\nWHERE d.enabled_flag = 1\n AND lp.lat IS NOT NULL\n AND lp.lng IS NOT NULL\nORDER BY d.vehicle_number;",
|
||||
"format": "table",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "table",
|
||||
"title": "Vehicle Status",
|
||||
"gridPos": { "x": 0, "y": 19, "w": 24, "h": 10 },
|
||||
"datasource": { "type": "postgres", "uid": "tracksolid_pg" },
|
||||
"options": {
|
||||
"cellHeight": "sm",
|
||||
"footer": { "countRows": false, "fields": "", "reducer": ["sum"], "show": false },
|
||||
"showHeader": true,
|
||||
"sortBy": []
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"align": "auto",
|
||||
"cellOptions": { "type": "auto" },
|
||||
"filterable": true,
|
||||
"inspect": false
|
||||
}
|
||||
},
|
||||
"overrides": [
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "Status" },
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.cellOptions",
|
||||
"value": { "type": "color-background", "mode": "basic" }
|
||||
},
|
||||
{
|
||||
"id": "mappings",
|
||||
"value": [
|
||||
{
|
||||
"type": "value",
|
||||
"options": {
|
||||
"Online": { "color": "green", "index": 0 },
|
||||
"Recent": { "color": "yellow", "index": 1 },
|
||||
"Offline": { "color": "red", "index": 2 }
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "Speed (km/h)" },
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.cellOptions",
|
||||
"value": { "type": "color-text" }
|
||||
},
|
||||
{
|
||||
"id": "color",
|
||||
"value": { "mode": "thresholds" }
|
||||
},
|
||||
{
|
||||
"id": "thresholds",
|
||||
"value": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "yellow", "value": 80 },
|
||||
{ "color": "red", "value": 120 }
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "Last Fix" },
|
||||
"properties": [
|
||||
{ "id": "unit", "value": "dateTimeAsLocal" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "Min Since Fix" },
|
||||
"properties": [
|
||||
{ "id": "custom.width", "value": 110 },
|
||||
{ "id": "displayName", "value": "Min Ago" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "Driver Phone" },
|
||||
"properties": [
|
||||
{ "id": "custom.width", "value": 130 }
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "Plate" },
|
||||
"properties": [
|
||||
{ "id": "custom.width", "value": 110 }
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": { "type": "postgres", "uid": "tracksolid_pg" },
|
||||
"rawSql": "SELECT\n d.vehicle_number AS \"Plate\",\n d.vehicle_name AS \"Vehicle\",\n d.driver_name AS \"Driver\",\n d.driver_phone AS \"Driver Phone\",\n d.city AS \"City\",\n ROUND(lp.speed::numeric, 0) AS \"Speed (km/h)\",\n lp.loc_desc AS \"Last Location\",\n lp.gps_time AS \"Last Fix\",\n CASE\n WHEN lp.gps_time >= NOW() - INTERVAL '5 minutes' THEN 'Online'\n WHEN lp.gps_time >= NOW() - INTERVAL '30 minutes' THEN 'Recent'\n ELSE 'Offline'\n END AS \"Status\",\n EXTRACT(EPOCH FROM (NOW() - lp.gps_time))::int / 60 AS \"Min Since Fix\"\nFROM tracksolid.devices d\nLEFT JOIN tracksolid.live_positions lp USING (imei)\nWHERE d.enabled_flag = 1\nORDER BY\n CASE\n WHEN lp.gps_time >= NOW() - INTERVAL '5 minutes' THEN 0\n WHEN lp.gps_time >= NOW() - INTERVAL '30 minutes' THEN 1\n ELSE 2\n END,\n d.vehicle_number;",
|
||||
"format": "table",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"type": "table",
|
||||
"title": "Ingestion Health",
|
||||
"collapsed": true,
|
||||
"gridPos": { "x": 0, "y": 29, "w": 24, "h": 8 },
|
||||
"datasource": { "type": "postgres", "uid": "tracksolid_pg" },
|
||||
"options": {
|
||||
"cellHeight": "sm",
|
||||
"footer": { "countRows": false, "fields": "", "reducer": ["sum"], "show": false },
|
||||
"showHeader": true
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"align": "auto",
|
||||
"cellOptions": { "type": "auto" },
|
||||
"filterable": false,
|
||||
"inspect": false
|
||||
}
|
||||
},
|
||||
"overrides": [
|
||||
{
|
||||
"matcher": { "id": "byName", "options": "Result" },
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.cellOptions",
|
||||
"value": { "type": "color-background", "mode": "basic" }
|
||||
},
|
||||
{
|
||||
"id": "mappings",
|
||||
"value": [
|
||||
{
|
||||
"type": "value",
|
||||
"options": {
|
||||
"OK": { "color": "green", "index": 0 },
|
||||
"FAIL": { "color": "red", "index": 1 }
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": { "type": "postgres", "uid": "tracksolid_pg" },
|
||||
"rawSql": "SELECT\n endpoint AS \"Endpoint\",\n TO_CHAR(run_at, 'HH24:MI DD-Mon') AS \"Last Run\",\n CASE WHEN success THEN 'OK' ELSE 'FAIL' END AS \"Result\",\n error_message AS \"Error\",\n seconds_ago AS \"Lag (s)\"\nFROM tracksolid.v_ingestion_health\nORDER BY endpoint;",
|
||||
"format": "table",
|
||||
"refId": "A"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
12
grafana/provisioning/dashboards/noc_fleet.yaml
Normal file
12
grafana/provisioning/dashboards/noc_fleet.yaml
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
apiVersion: 1
|
||||
|
||||
providers:
|
||||
- name: NOC Fleet Dashboards
|
||||
orgId: 1
|
||||
folder: NOC
|
||||
type: file
|
||||
disableDeletion: true
|
||||
updateIntervalSeconds: 30
|
||||
allowUiUpdates: false
|
||||
options:
|
||||
path: /etc/grafana/provisioning/dashboards-json
|
||||
20
grafana/provisioning/datasources/tracksolid_postgres.yaml
Normal file
20
grafana/provisioning/datasources/tracksolid_postgres.yaml
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: TracksolidDB
|
||||
type: postgres
|
||||
uid: tracksolid_pg
|
||||
url: timescale_db:5432
|
||||
database: tracksolid_db
|
||||
user: grafana_ro
|
||||
secureJsonData:
|
||||
password: ${GRAFANA_DB_RO_PASSWORD}
|
||||
jsonData:
|
||||
sslmode: disable
|
||||
maxOpenConns: 5
|
||||
maxIdleConns: 2
|
||||
connMaxLifetime: 14400
|
||||
postgresVersion: 1600
|
||||
timescaledb: true
|
||||
editable: false
|
||||
isDefault: true
|
||||
324
grafanaDeployment.md
Normal file
324
grafanaDeployment.md
Normal file
|
|
@ -0,0 +1,324 @@
|
|||
# Grafana NOC Fleet Dashboard — Deployment Guide
|
||||
|
||||
## Overview
|
||||
|
||||
Single-region NOC dashboard monitoring 80 vehicles (Nairobi, Mombasa, Kampala) via Tracksolid/Jimi GPS.
|
||||
Live location, direction arrows, speed, and driver info. Auto-refreshes every 30 seconds.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker Compose stack running (`timescale_db`, `grafana`)
|
||||
- `.env` file with all existing variables
|
||||
- Add one new variable to `.env`:
|
||||
```
|
||||
GRAFANA_DB_RO_PASSWORD=<password for grafana_ro postgres user>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
grafana/
|
||||
└── provisioning/
|
||||
├── datasources/
|
||||
│ └── tracksolid_postgres.yaml
|
||||
├── dashboards/
|
||||
│ └── noc_fleet.yaml
|
||||
└── dashboards-json/
|
||||
└── noc_fleet_dashboard.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 1 — docker-compose.yaml (grafana service)
|
||||
|
||||
Replace the existing `grafana` service block with:
|
||||
|
||||
```yaml
|
||||
grafana:
|
||||
image: grafana/grafana:11.0.0
|
||||
restart: always
|
||||
depends_on:
|
||||
timescale_db:
|
||||
condition: service_healthy
|
||||
env_file: .env
|
||||
environment:
|
||||
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}
|
||||
- GF_USERS_DEFAULT_THEME=dark
|
||||
- GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/etc/grafana/provisioning/dashboards-json/noc_fleet_dashboard.json
|
||||
volumes:
|
||||
- grafana-data:/var/lib/grafana
|
||||
- ./grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2 — Datasource: `grafana/provisioning/datasources/tracksolid_postgres.yaml`
|
||||
|
||||
```yaml
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: TracksolidDB
|
||||
type: postgres
|
||||
uid: tracksolid_pg
|
||||
url: timescale_db:5432
|
||||
database: tracksolid_db
|
||||
user: grafana_ro
|
||||
secureJsonData:
|
||||
password: ${GRAFANA_DB_RO_PASSWORD}
|
||||
jsonData:
|
||||
sslmode: disable
|
||||
maxOpenConns: 5
|
||||
maxIdleConns: 2
|
||||
connMaxLifetime: 14400
|
||||
postgresVersion: 1600
|
||||
timescaledb: true
|
||||
editable: false
|
||||
isDefault: true
|
||||
```
|
||||
|
||||
> **Note:** `uid: tracksolid_pg` is referenced by the dashboard JSON. Never rename it after deployment.
|
||||
|
||||
---
|
||||
|
||||
## Step 3 — Dashboard Provider: `grafana/provisioning/dashboards/noc_fleet.yaml`
|
||||
|
||||
```yaml
|
||||
apiVersion: 1
|
||||
|
||||
providers:
|
||||
- name: NOC Fleet Dashboards
|
||||
orgId: 1
|
||||
folder: NOC
|
||||
type: file
|
||||
disableDeletion: true
|
||||
updateIntervalSeconds: 30
|
||||
allowUiUpdates: false
|
||||
options:
|
||||
path: /etc/grafana/provisioning/dashboards-json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 4 — Dashboard JSON: `grafana/provisioning/dashboards-json/noc_fleet_dashboard.json`
|
||||
|
||||
### Dashboard Settings
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| UID | `noc-fleet-live` |
|
||||
| Auto-refresh | 30s |
|
||||
| Theme | Dark |
|
||||
| Schema version | 39 (Grafana 11.0.0) |
|
||||
| Editable | false |
|
||||
| Bookmark URL | `/d/noc-fleet-live` |
|
||||
|
||||
---
|
||||
|
||||
### Panel Layout
|
||||
|
||||
| Row | Panel | Type | Notes |
|
||||
|-----|-------|------|-------|
|
||||
| 1 | Total Vehicles | Stat | Count all enabled devices |
|
||||
| 1 | Online Now | Stat | GPS fix < 5 min — green |
|
||||
| 1 | Recent (5-30m) | Stat | GPS fix 5-30 min — amber |
|
||||
| 1 | Offline | Stat | GPS fix > 30 min — red |
|
||||
| 1 | Moving Now | Stat | speed > 0 AND acc on |
|
||||
| 1 | Avg Speed (km/h) | Stat | Moving vehicles only |
|
||||
| 2 | Live Vehicle Locations | Geomap | Direction arrows, color by plate |
|
||||
| 3 | Vehicle Status Table | Table | All vehicles, sorted Online first |
|
||||
| 4 | Ingestion Health | Table | Collapsed by default |
|
||||
|
||||
---
|
||||
|
||||
### SQL Queries
|
||||
|
||||
**Total Vehicles**
|
||||
```sql
|
||||
SELECT COUNT(*) AS "Total Vehicles"
|
||||
FROM tracksolid.devices WHERE enabled_flag = 1;
|
||||
```
|
||||
|
||||
**Online Now**
|
||||
```sql
|
||||
SELECT COUNT(*) AS "Online"
|
||||
FROM tracksolid.v_fleet_status
|
||||
WHERE connectivity_status = 'online';
|
||||
```
|
||||
|
||||
**Recent (5-30 min)**
|
||||
```sql
|
||||
SELECT COUNT(*) AS "Recent"
|
||||
FROM tracksolid.v_fleet_status
|
||||
WHERE connectivity_status = 'recent';
|
||||
```
|
||||
|
||||
**Offline**
|
||||
```sql
|
||||
SELECT COUNT(*) AS "Offline"
|
||||
FROM tracksolid.v_fleet_status
|
||||
WHERE connectivity_status = 'offline';
|
||||
```
|
||||
|
||||
**Moving Now**
|
||||
```sql
|
||||
SELECT COUNT(*) AS "Moving"
|
||||
FROM tracksolid.v_fleet_status
|
||||
WHERE speed > 0 AND acc_status = '1' AND connectivity_status = 'online';
|
||||
```
|
||||
|
||||
**Avg Speed**
|
||||
```sql
|
||||
SELECT ROUND(AVG(speed)::numeric, 1) AS "Avg Speed km/h"
|
||||
FROM tracksolid.v_fleet_status
|
||||
WHERE speed > 0 AND acc_status = '1' AND connectivity_status = 'online';
|
||||
```
|
||||
|
||||
**Geomap — Live Vehicle Locations**
|
||||
```sql
|
||||
SELECT
|
||||
d.imei,
|
||||
d.vehicle_number,
|
||||
d.vehicle_name,
|
||||
d.driver_name,
|
||||
d.driver_phone,
|
||||
d.city,
|
||||
d.device_group,
|
||||
lp.lat,
|
||||
lp.lng,
|
||||
lp.speed,
|
||||
lp.direction,
|
||||
lp.acc_status,
|
||||
lp.loc_desc,
|
||||
lp.gps_time,
|
||||
CASE
|
||||
WHEN lp.gps_time >= NOW() - INTERVAL '5 minutes' THEN 'online'
|
||||
WHEN lp.gps_time >= NOW() - INTERVAL '30 minutes' THEN 'recent'
|
||||
ELSE 'offline'
|
||||
END AS connectivity_status
|
||||
FROM tracksolid.devices d
|
||||
INNER JOIN tracksolid.live_positions lp USING (imei)
|
||||
WHERE d.enabled_flag = 1
|
||||
AND lp.lat IS NOT NULL
|
||||
AND lp.lng IS NOT NULL
|
||||
ORDER BY d.vehicle_number;
|
||||
```
|
||||
|
||||
> INNER JOIN — only vehicles with a valid GPS fix appear on the map.
|
||||
> `acc_status` is TEXT ('0'/'1') in the schema — all queries use string comparisons.
|
||||
|
||||
**Vehicle Status Table**
|
||||
```sql
|
||||
SELECT
|
||||
d.vehicle_number AS "Plate",
|
||||
d.vehicle_name AS "Vehicle",
|
||||
d.driver_name AS "Driver",
|
||||
d.driver_phone AS "Driver Phone",
|
||||
d.city AS "City",
|
||||
ROUND(lp.speed::numeric, 0) AS "Speed (km/h)",
|
||||
lp.loc_desc AS "Last Location",
|
||||
lp.gps_time AS "Last Fix",
|
||||
CASE
|
||||
WHEN lp.gps_time >= NOW() - INTERVAL '5 minutes' THEN 'Online'
|
||||
WHEN lp.gps_time >= NOW() - INTERVAL '30 minutes' THEN 'Recent'
|
||||
ELSE 'Offline'
|
||||
END AS "Status",
|
||||
EXTRACT(EPOCH FROM (NOW() - lp.gps_time))::int / 60 AS "Min Since Fix"
|
||||
FROM tracksolid.devices d
|
||||
LEFT JOIN tracksolid.live_positions lp USING (imei)
|
||||
WHERE d.enabled_flag = 1
|
||||
ORDER BY
|
||||
CASE
|
||||
WHEN lp.gps_time >= NOW() - INTERVAL '5 minutes' THEN 0
|
||||
WHEN lp.gps_time >= NOW() - INTERVAL '30 minutes' THEN 1
|
||||
ELSE 2
|
||||
END,
|
||||
d.vehicle_number;
|
||||
```
|
||||
|
||||
> LEFT JOIN — offline vehicles (no GPS fix) still appear in the table showing their null status.
|
||||
|
||||
**Ingestion Health**
|
||||
```sql
|
||||
SELECT
|
||||
endpoint AS "Endpoint",
|
||||
TO_CHAR(run_at, 'HH24:MI DD-Mon') AS "Last Run",
|
||||
CASE WHEN success THEN 'OK' ELSE 'FAIL' END AS "Result",
|
||||
error_message AS "Error",
|
||||
seconds_ago AS "Lag (s)"
|
||||
FROM tracksolid.v_ingestion_health
|
||||
ORDER BY endpoint;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Geomap Configuration
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Basemap | Carto Dark |
|
||||
| Location mode | `coords` — fields `lat` / `lng` |
|
||||
| Marker symbol | `img/icons/marker/arrow-up.svg` (built-in Grafana 11) |
|
||||
| Rotation | field: `direction` (0° = North, clockwise) |
|
||||
| Color | field: `vehicle_number`, scheme: `palette-classic-by-name` |
|
||||
| Tooltip hidden fields | `lat`, `lng`, `imei` |
|
||||
| Direction unit override | `degree` (shows `245°` in tooltip) |
|
||||
| Initial view | East Africa center — auto-fits to vehicle positions |
|
||||
|
||||
### Connectivity Status Color Mapping
|
||||
|
||||
| Value | Color |
|
||||
|-------|-------|
|
||||
| Online / online | Green |
|
||||
| Recent / recent | Amber |
|
||||
| Offline / offline | Red |
|
||||
|
||||
### Stat Panel Thresholds
|
||||
|
||||
| Panel | Green | Amber | Red |
|
||||
|-------|-------|-------|-----|
|
||||
| Online Now | > 0 | — | = 0 |
|
||||
| Recent | — | > 0 | — |
|
||||
| Offline | = 0 | — | > 0 |
|
||||
| Moving Now | > 0 | — | — |
|
||||
| Avg Speed | < 80 km/h | 80-120 | > 120 |
|
||||
|
||||
All stat panels: `colorMode: background`, `graphMode: none`.
|
||||
|
||||
---
|
||||
|
||||
## Step 5 — Deploy
|
||||
|
||||
```bash
|
||||
docker compose up -d grafana
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 6 — Verify
|
||||
|
||||
1. Open `http://localhost:3000/d/noc-fleet-live`
|
||||
2. Map renders with arrow markers on vehicle positions
|
||||
3. Arrows rotate per heading (northbound vehicle = arrow pointing up)
|
||||
4. Tooltip shows: plate, driver name, speed, city, connectivity status
|
||||
5. Table sorts: Online → Recent → Offline
|
||||
6. 30s auto-refresh fires (watch "Last Fix" timestamps)
|
||||
7. Check provisioning logs:
|
||||
```bash
|
||||
docker compose logs grafana | grep -i provision
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- The `grafana/` directory must be committed to git — Coolify's clone must include it
|
||||
- `:ro` mounts prevent Grafana from writing back into the repo
|
||||
- `allowUiUpdates: false` in the provider YAML means UI edits are not persisted — the JSON file is the source of truth
|
||||
- `acc_status` in `live_positions` is TEXT (`'0'`/`'1'`), not integer — all queries use string comparisons
|
||||
- `uid: tracksolid_pg` in the datasource YAML is referenced by the dashboard JSON — never rename it after deployment
|
||||
- ServiceNow ticket integration is deferred to a future phase
|
||||
647
grafanaOperationalManual.md
Normal file
647
grafanaOperationalManual.md
Normal file
|
|
@ -0,0 +1,647 @@
|
|||
# Grafana NOC Fleet Dashboard — Operational Manual
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Pre-Deployment Checklist](#1-pre-deployment-checklist)
|
||||
2. [Deploy](#2-deploy)
|
||||
3. [Post-Deployment Verification](#3-post-deployment-verification)
|
||||
4. [Dashboard Panel Verification](#4-dashboard-panel-verification)
|
||||
5. [Database Verification Queries](#5-database-verification-queries)
|
||||
6. [Troubleshooting](#6-troubleshooting)
|
||||
7. [Day-to-Day NOC Operations](#7-day-to-day-noc-operations)
|
||||
8. [Maintenance](#8-maintenance)
|
||||
|
||||
---
|
||||
|
||||
## 1. Pre-Deployment Checklist
|
||||
|
||||
Run these checks before starting Grafana for the first time.
|
||||
|
||||
### 1.1 Environment Variables
|
||||
|
||||
Confirm `.env` contains all required Grafana variables:
|
||||
|
||||
```bash
|
||||
grep -E 'GRAFANA_ADMIN_PASSWORD|GRAFANA_DB_RO_PASSWORD' .env
|
||||
```
|
||||
|
||||
Expected output — both lines must be present and non-empty:
|
||||
```
|
||||
GRAFANA_ADMIN_PASSWORD=<your admin password>
|
||||
GRAFANA_DB_RO_PASSWORD=<grafana_ro postgres password>
|
||||
```
|
||||
|
||||
If `GRAFANA_DB_RO_PASSWORD` is missing, add it before continuing:
|
||||
```bash
|
||||
echo "GRAFANA_DB_RO_PASSWORD=<password>" >> .env
|
||||
```
|
||||
|
||||
### 1.2 Verify grafana_ro User Exists in Postgres
|
||||
|
||||
```bash
|
||||
docker compose exec timescale_db psql -U $POSTGRES_USER -d tracksolid_db -c "\du grafana_ro"
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
Role name | Attributes
|
||||
------------+---------------------------
|
||||
grafana_ro | Cannot login, ...
|
||||
```
|
||||
|
||||
If the role is missing, create it:
|
||||
```bash
|
||||
docker compose exec timescale_db psql -U $POSTGRES_USER -d tracksolid_db -c "
|
||||
CREATE ROLE grafana_ro WITH LOGIN PASSWORD '<password>' NOSUPERUSER NOCREATEDB NOCREATEROLE;
|
||||
GRANT USAGE ON SCHEMA tracksolid TO grafana_ro;
|
||||
GRANT SELECT ON ALL TABLES IN SCHEMA tracksolid TO grafana_ro;
|
||||
ALTER DEFAULT PRIVILEGES IN SCHEMA tracksolid GRANT SELECT ON TABLES TO grafana_ro;
|
||||
"
|
||||
```
|
||||
|
||||
### 1.3 Verify Provisioning Files Are Present
|
||||
|
||||
```bash
|
||||
ls -R grafana/provisioning/
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
grafana/provisioning/:
|
||||
dashboards dashboards-json datasources
|
||||
|
||||
grafana/provisioning/datasources:
|
||||
tracksolid_postgres.yaml
|
||||
|
||||
grafana/provisioning/dashboards:
|
||||
noc_fleet.yaml
|
||||
|
||||
grafana/provisioning/dashboards-json:
|
||||
noc_fleet_dashboard.json
|
||||
```
|
||||
|
||||
### 1.4 Verify Database Has Live Data
|
||||
|
||||
```bash
|
||||
docker compose exec timescale_db psql -U $POSTGRES_USER -d tracksolid_db -c "
|
||||
SELECT COUNT(*) AS devices,
|
||||
COUNT(lp.imei) AS with_position
|
||||
FROM tracksolid.devices d
|
||||
LEFT JOIN tracksolid.live_positions lp USING (imei)
|
||||
WHERE d.enabled_flag = 1;
|
||||
"
|
||||
```
|
||||
|
||||
Expected: `devices` = total enabled vehicles, `with_position` > 0 (at least some vehicles have GPS fixes).
|
||||
|
||||
---
|
||||
|
||||
## 2. Deploy
|
||||
|
||||
### Start Grafana
|
||||
|
||||
```bash
|
||||
docker compose up -d grafana
|
||||
```
|
||||
|
||||
### Confirm Container Is Running
|
||||
|
||||
```bash
|
||||
docker compose ps grafana
|
||||
```
|
||||
|
||||
Expected `STATUS`: `Up` (not `Restarting` or `Exit`).
|
||||
|
||||
### Tail Startup Logs
|
||||
|
||||
```bash
|
||||
docker compose logs --follow grafana
|
||||
```
|
||||
|
||||
Watch for these lines — they confirm provisioning loaded successfully:
|
||||
|
||||
```
|
||||
msg="Starting Grafana"
|
||||
msg="Provisioning datasource" name=TracksolidDB
|
||||
msg="Finished provisioning data sources"
|
||||
msg="Inserting/updating dashboard" name="NOC Fleet Operations — Live"
|
||||
msg="HTTP Server Listen" address=0.0.0.0:3000
|
||||
```
|
||||
|
||||
Press `Ctrl+C` to stop following once you see the server listening.
|
||||
|
||||
---
|
||||
|
||||
## 3. Post-Deployment Verification
|
||||
|
||||
### 3.1 Check Provisioning Loaded
|
||||
|
||||
```bash
|
||||
docker compose logs grafana | grep -E "provision|Inserting|dashboard|datasource"
|
||||
```
|
||||
|
||||
| What to look for | Means |
|
||||
|---|---|
|
||||
| `Provisioning datasource` with `name=TracksolidDB` | Datasource YAML was read |
|
||||
| `Finished provisioning data sources` | Datasource created successfully |
|
||||
| `Inserting/updating dashboard` with `noc-fleet-live` | Dashboard JSON was loaded |
|
||||
| `Failed to provision` or `Error` | See Troubleshooting section |
|
||||
|
||||
### 3.2 Verify Datasource via API
|
||||
|
||||
```bash
|
||||
curl -s -u admin:${GRAFANA_ADMIN_PASSWORD} \
|
||||
http://localhost:3000/api/datasources/name/TracksolidDB \
|
||||
| python3 -m json.tool | grep -E '"name"|"uid"|"type"|"url"'
|
||||
```
|
||||
|
||||
Expected:
|
||||
```json
|
||||
"name": "TracksolidDB",
|
||||
"type": "postgres",
|
||||
"uid": "tracksolid_pg",
|
||||
"url": "timescale_db:5432",
|
||||
```
|
||||
|
||||
### 3.3 Test Datasource Connection via API
|
||||
|
||||
```bash
|
||||
curl -s -u admin:${GRAFANA_ADMIN_PASSWORD} \
|
||||
-X POST http://localhost:3000/api/datasources/uid/tracksolid_pg/health \
|
||||
| python3 -m json.tool
|
||||
```
|
||||
|
||||
Expected:
|
||||
```json
|
||||
{
|
||||
"message": "Database Connection OK",
|
||||
"status": "OK"
|
||||
}
|
||||
```
|
||||
|
||||
If status is `ERROR` — see [Troubleshooting: Datasource Connection Fails](#datasource-connection-fails).
|
||||
|
||||
### 3.4 Verify Dashboard Is Registered
|
||||
|
||||
```bash
|
||||
curl -s -u admin:${GRAFANA_ADMIN_PASSWORD} \
|
||||
http://localhost:3000/api/dashboards/uid/noc-fleet-live \
|
||||
| python3 -m json.tool | grep -E '"title"|"uid"|"version"'
|
||||
```
|
||||
|
||||
Expected:
|
||||
```json
|
||||
"title": "NOC Fleet Operations — Live",
|
||||
"uid": "noc-fleet-live",
|
||||
"version": 1,
|
||||
```
|
||||
|
||||
### 3.5 Open the Dashboard in a Browser
|
||||
|
||||
Navigate to: `http://localhost:3000/d/noc-fleet-live`
|
||||
|
||||
Login with:
|
||||
- **Username:** `admin`
|
||||
- **Password:** value of `GRAFANA_ADMIN_PASSWORD` from `.env`
|
||||
|
||||
The NOC dashboard should load as the home page automatically.
|
||||
|
||||
---
|
||||
|
||||
## 4. Dashboard Panel Verification
|
||||
|
||||
Work through each panel to confirm it renders correctly.
|
||||
|
||||
### 4.1 Stat Panels (Row 1)
|
||||
|
||||
| Panel | Expected | Red Flag |
|
||||
|---|---|---|
|
||||
| Total Vehicles | Integer matching enabled device count (should be ~80) | Shows 0 or `-` |
|
||||
| Online Now | Integer ≤ Total Vehicles | Shows `-` |
|
||||
| Recent (5-30m) | Integer ≤ Total Vehicles | Shows `-` |
|
||||
| Offline | Integer ≤ Total Vehicles | Shows `-` |
|
||||
| Moving Now | Integer ≤ Online Now | Shows `-` |
|
||||
| Avg Speed (km/h) | Numeric value, green < 80 / amber 80-120 / red > 120 | Shows `No data` |
|
||||
|
||||
**Online + Recent + Offline should sum to Total Vehicles.**
|
||||
|
||||
Check the sum:
|
||||
```bash
|
||||
docker compose exec timescale_db psql -U $POSTGRES_USER -d tracksolid_db -c "
|
||||
SELECT
|
||||
connectivity_status,
|
||||
COUNT(*) AS vehicles
|
||||
FROM tracksolid.v_fleet_status
|
||||
GROUP BY connectivity_status
|
||||
ORDER BY connectivity_status;
|
||||
"
|
||||
```
|
||||
|
||||
### 4.2 Geomap Panel
|
||||
|
||||
Work through this checklist visually:
|
||||
|
||||
- [ ] Map loads with dark Carto basemap (not a grey blank tile)
|
||||
- [ ] Arrow markers appear on the map (not dots or circles)
|
||||
- [ ] Markers are clustered around East Africa (Nairobi / Mombasa / Kampala area)
|
||||
- [ ] Arrows point in different directions — not all the same
|
||||
- [ ] Clicking a marker opens a tooltip showing: Plate, Driver, Speed, Heading, Status, Location
|
||||
- [ ] `lat` and `lng` fields are NOT visible in the tooltip (hidden by field overrides)
|
||||
- [ ] `imei` field is NOT visible in the tooltip
|
||||
|
||||
**Verify direction data exists:**
|
||||
```bash
|
||||
docker compose exec timescale_db psql -U $POSTGRES_USER -d tracksolid_db -c "
|
||||
SELECT
|
||||
MIN(direction) AS min_dir,
|
||||
MAX(direction) AS max_dir,
|
||||
COUNT(*) FILTER (WHERE direction IS NOT NULL) AS with_direction,
|
||||
COUNT(*) AS total
|
||||
FROM tracksolid.live_positions;
|
||||
"
|
||||
```
|
||||
|
||||
If `with_direction` = 0, all arrows will point North (0°) — this means the GPS devices haven't sent bearing data yet, which is normal for parked vehicles.
|
||||
|
||||
### 4.3 Vehicle Status Table
|
||||
|
||||
- [ ] Table shows all enabled vehicles (row count matches Total Vehicles stat)
|
||||
- [ ] Rows sort: Online (green) → Recent (amber) → Offline (red)
|
||||
- [ ] "Status" column has colour-coded background
|
||||
- [ ] "Speed (km/h)" column shows colour-coded text (green/amber/red)
|
||||
- [ ] "Last Fix" column shows a readable timestamp (not a raw epoch number)
|
||||
- [ ] "Min Ago" column shows integers (minutes since last GPS fix)
|
||||
- [ ] Vehicles with no GPS fix show `null` in speed/location columns but still appear
|
||||
|
||||
**Spot-check a specific vehicle:**
|
||||
```bash
|
||||
docker compose exec timescale_db psql -U $POSTGRES_USER -d tracksolid_db -c "
|
||||
SELECT
|
||||
d.vehicle_number, d.driver_name, lp.speed,
|
||||
lp.gps_time, lp.lat, lp.lng
|
||||
FROM tracksolid.devices d
|
||||
LEFT JOIN tracksolid.live_positions lp USING (imei)
|
||||
WHERE d.enabled_flag = 1
|
||||
LIMIT 5;
|
||||
"
|
||||
```
|
||||
|
||||
### 4.4 Ingestion Health Panel (Collapsed)
|
||||
|
||||
Expand the panel by clicking its title row.
|
||||
|
||||
- [ ] Rows appear for each polling endpoint (`live_positions`, `trips`, `alarms`, etc.)
|
||||
- [ ] "Result" column shows `OK` (green) for all active endpoints
|
||||
- [ ] "Last Run" timestamps are recent (within the last few minutes for live_positions)
|
||||
- [ ] No `FAIL` (red) entries
|
||||
|
||||
```bash
|
||||
docker compose exec timescale_db psql -U $POSTGRES_USER -d tracksolid_db -c "
|
||||
SELECT endpoint, success, seconds_ago, error_message
|
||||
FROM tracksolid.v_ingestion_health
|
||||
ORDER BY endpoint;
|
||||
"
|
||||
```
|
||||
|
||||
### 4.5 Auto-Refresh Verification
|
||||
|
||||
1. Note the "Last Fix" timestamp for any vehicle in the table
|
||||
2. Wait 30 seconds
|
||||
3. Confirm the page auto-refreshes (watch the Grafana spinner in the top-right)
|
||||
4. Confirm "Last Fix" values have updated for Online vehicles
|
||||
|
||||
---
|
||||
|
||||
## 5. Database Verification Queries
|
||||
|
||||
Run these directly against the database to validate the data powering each panel.
|
||||
|
||||
### Fleet Status Summary
|
||||
```sql
|
||||
SELECT
|
||||
connectivity_status,
|
||||
COUNT(*) AS vehicles,
|
||||
ROUND(AVG(speed)::numeric, 1) AS avg_speed_kmh
|
||||
FROM tracksolid.v_fleet_status
|
||||
GROUP BY connectivity_status
|
||||
ORDER BY connectivity_status;
|
||||
```
|
||||
|
||||
### Live Position Freshness
|
||||
```sql
|
||||
SELECT
|
||||
COUNT(*) AS total_positions,
|
||||
COUNT(*) FILTER (WHERE gps_time >= NOW() - INTERVAL '5 minutes') AS fresh_5m,
|
||||
COUNT(*) FILTER (WHERE gps_time >= NOW() - INTERVAL '30 minutes') AS fresh_30m,
|
||||
MAX(gps_time) AS newest_fix,
|
||||
MIN(gps_time) AS oldest_fix
|
||||
FROM tracksolid.live_positions;
|
||||
```
|
||||
|
||||
### Moving Vehicles
|
||||
```sql
|
||||
SELECT
|
||||
d.vehicle_number, d.driver_name,
|
||||
lp.speed, lp.direction, lp.loc_desc, lp.gps_time
|
||||
FROM tracksolid.devices d
|
||||
INNER JOIN tracksolid.live_positions lp USING (imei)
|
||||
WHERE d.enabled_flag = 1
|
||||
AND lp.speed > 0
|
||||
AND lp.acc_status = '1'
|
||||
ORDER BY lp.speed DESC;
|
||||
```
|
||||
|
||||
### Vehicles With No GPS Fix
|
||||
```sql
|
||||
SELECT
|
||||
d.vehicle_number, d.vehicle_name, d.driver_name, d.city
|
||||
FROM tracksolid.devices d
|
||||
LEFT JOIN tracksolid.live_positions lp USING (imei)
|
||||
WHERE d.enabled_flag = 1
|
||||
AND lp.imei IS NULL
|
||||
ORDER BY d.vehicle_number;
|
||||
```
|
||||
|
||||
### Ingestion Pipeline Health
|
||||
```sql
|
||||
SELECT
|
||||
endpoint,
|
||||
run_at,
|
||||
success,
|
||||
imei_count,
|
||||
rows_upserted,
|
||||
duration_ms,
|
||||
seconds_ago,
|
||||
error_message
|
||||
FROM tracksolid.v_ingestion_health
|
||||
ORDER BY endpoint;
|
||||
```
|
||||
|
||||
### Stale Positions (no update in > 1 hour)
|
||||
```sql
|
||||
SELECT
|
||||
d.vehicle_number, d.driver_name, d.city,
|
||||
lp.gps_time,
|
||||
EXTRACT(EPOCH FROM (NOW() - lp.gps_time))::int / 60 AS minutes_since_fix
|
||||
FROM tracksolid.devices d
|
||||
INNER JOIN tracksolid.live_positions lp USING (imei)
|
||||
WHERE d.enabled_flag = 1
|
||||
AND lp.gps_time < NOW() - INTERVAL '1 hour'
|
||||
ORDER BY lp.gps_time ASC;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Troubleshooting
|
||||
|
||||
### Grafana Container Keeps Restarting
|
||||
|
||||
```bash
|
||||
docker compose logs grafana --tail=50
|
||||
```
|
||||
|
||||
Common causes:
|
||||
|
||||
| Error in logs | Fix |
|
||||
|---|---|
|
||||
| `failed to connect to server` | Database not healthy yet — wait 30s and retry |
|
||||
| `permission denied` on provisioning path | Check the `./grafana/provisioning` directory exists and is readable |
|
||||
| `GF_SECURITY_ADMIN_PASSWORD not set` | Add `GRAFANA_ADMIN_PASSWORD` to `.env` |
|
||||
|
||||
### Datasource Connection Fails
|
||||
|
||||
```bash
|
||||
docker compose logs grafana | grep -i "datasource\|postgres\|connect"
|
||||
```
|
||||
|
||||
Check the password is correct:
|
||||
```bash
|
||||
docker compose exec timescale_db psql -U grafana_ro -d tracksolid_db -c "SELECT 1;"
|
||||
```
|
||||
|
||||
If this fails with `authentication failed`, reset the password:
|
||||
```bash
|
||||
docker compose exec timescale_db psql -U $POSTGRES_USER -d tracksolid_db -c \
|
||||
"ALTER ROLE grafana_ro WITH PASSWORD '<new_password>';"
|
||||
```
|
||||
|
||||
Then update `GRAFANA_DB_RO_PASSWORD` in `.env` and restart:
|
||||
```bash
|
||||
docker compose restart grafana
|
||||
```
|
||||
|
||||
### Dashboard Shows "No Data"
|
||||
|
||||
**Step 1 — confirm the datasource UID matches:**
|
||||
```bash
|
||||
curl -s -u admin:${GRAFANA_ADMIN_PASSWORD} \
|
||||
http://localhost:3000/api/datasources \
|
||||
| python3 -m json.tool | grep uid
|
||||
```
|
||||
|
||||
Must show `"uid": "tracksolid_pg"`. If the UID is different, the dashboard JSON references the wrong datasource.
|
||||
|
||||
**Step 2 — test the query directly in Grafana:**
|
||||
1. Open the dashboard
|
||||
2. Click the panel title → Edit
|
||||
3. Switch to the Query tab
|
||||
4. Click "Run query"
|
||||
5. If it errors, the SQL or connection is the issue
|
||||
|
||||
**Step 3 — verify data exists:**
|
||||
```bash
|
||||
docker compose exec timescale_db psql -U $POSTGRES_USER -d tracksolid_db -c \
|
||||
"SELECT COUNT(*) FROM tracksolid.live_positions;"
|
||||
```
|
||||
|
||||
If this returns 0, the ingestion pipeline has not populated data yet. Check ingestion service logs:
|
||||
```bash
|
||||
docker compose logs ingest_movement --tail=30
|
||||
docker compose logs ingest_events --tail=30
|
||||
```
|
||||
|
||||
### Geomap Shows No Markers
|
||||
|
||||
**Check coordinates are valid:**
|
||||
```bash
|
||||
docker compose exec timescale_db psql -U $POSTGRES_USER -d tracksolid_db -c "
|
||||
SELECT COUNT(*) AS total,
|
||||
COUNT(*) FILTER (WHERE lat IS NOT NULL AND lng IS NOT NULL) AS with_coords,
|
||||
COUNT(*) FILTER (WHERE lat BETWEEN -90 AND 90 AND lng BETWEEN -180 AND 180) AS valid_coords
|
||||
FROM tracksolid.live_positions;
|
||||
"
|
||||
```
|
||||
|
||||
If `with_coords` = 0: GPS data has not arrived yet. Wait for the next ingest cycle (60 seconds).
|
||||
|
||||
If `valid_coords` < `with_coords`: some coordinates are out of range — data quality issue on the device side.
|
||||
|
||||
**Check the basemap is loading:**
|
||||
|
||||
Open browser DevTools (F12) → Network tab → filter for `tile` or `carto`. If Carto tile requests are failing, there may be no internet access from the browser. Try switching the basemap to OpenStreetMap in the Grafana panel editor temporarily.
|
||||
|
||||
### Arrows All Point North (No Direction Data)
|
||||
|
||||
This is expected for parked vehicles — direction is only meaningful when the vehicle is moving. Confirm:
|
||||
|
||||
```bash
|
||||
docker compose exec timescale_db psql -U $POSTGRES_USER -d tracksolid_db -c "
|
||||
SELECT COUNT(*) FILTER (WHERE direction > 0) AS with_bearing,
|
||||
COUNT(*) AS total
|
||||
FROM tracksolid.live_positions;
|
||||
"
|
||||
```
|
||||
|
||||
If `with_bearing` = 0 even for moving vehicles, the GPS device firmware may not be sending bearing in the position payload. Check with the Tracksolid account settings.
|
||||
|
||||
### Dashboard Not Loading as Home Page
|
||||
|
||||
Confirm the environment variable is set:
|
||||
```bash
|
||||
docker compose exec grafana env | grep DASHBOARDS
|
||||
```
|
||||
|
||||
Expected:
|
||||
```
|
||||
GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/etc/grafana/provisioning/dashboards-json/noc_fleet_dashboard.json
|
||||
```
|
||||
|
||||
If missing, confirm `env_file: .env` is present in the grafana service block of `docker-compose.yaml` and restart:
|
||||
```bash
|
||||
docker compose restart grafana
|
||||
```
|
||||
|
||||
### Provisioning Changes Not Reflecting
|
||||
|
||||
Grafana polls the provisioning directory every 30 seconds. If changes to the dashboard JSON are not appearing:
|
||||
|
||||
```bash
|
||||
docker compose restart grafana
|
||||
docker compose logs grafana | grep -i "provision\|dashboard"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Day-to-Day NOC Operations
|
||||
|
||||
### Morning Health Check (run at shift start)
|
||||
|
||||
```bash
|
||||
# 1. Confirm all services are up
|
||||
docker compose ps
|
||||
|
||||
# 2. Check ingestion pipeline ran recently
|
||||
docker compose exec timescale_db psql -U $POSTGRES_USER -d tracksolid_db -c \
|
||||
"SELECT endpoint, run_at, success, seconds_ago FROM tracksolid.v_ingestion_health ORDER BY endpoint;"
|
||||
|
||||
# 3. Check live position freshness
|
||||
docker compose exec timescale_db psql -U $POSTGRES_USER -d tracksolid_db -c \
|
||||
"SELECT connectivity_status, COUNT(*) FROM tracksolid.v_fleet_status GROUP BY 1 ORDER BY 1;"
|
||||
```
|
||||
|
||||
### Investigating a Specific Vehicle
|
||||
|
||||
Replace `KAA 123A` with the actual plate:
|
||||
|
||||
```bash
|
||||
docker compose exec timescale_db psql -U $POSTGRES_USER -d tracksolid_db -c "
|
||||
SELECT
|
||||
d.vehicle_number, d.vehicle_name, d.driver_name, d.driver_phone,
|
||||
lp.lat, lp.lng, lp.speed, lp.direction,
|
||||
lp.acc_status, lp.gps_time, lp.loc_desc,
|
||||
EXTRACT(EPOCH FROM (NOW() - lp.gps_time))::int / 60 AS minutes_ago
|
||||
FROM tracksolid.devices d
|
||||
LEFT JOIN tracksolid.live_positions lp USING (imei)
|
||||
WHERE d.vehicle_number = 'KAA 123A';
|
||||
"
|
||||
```
|
||||
|
||||
### Checking Recent Trips for a Vehicle
|
||||
|
||||
```bash
|
||||
docker compose exec timescale_db psql -U $POSTGRES_USER -d tracksolid_db -c "
|
||||
SELECT
|
||||
t.start_time, t.end_time,
|
||||
ROUND(t.distance_m / 1000.0, 2) AS distance_km,
|
||||
t.avg_speed_kmh, t.max_speed_kmh,
|
||||
ROUND(t.driving_time_s / 3600.0, 2) AS driving_hours
|
||||
FROM tracksolid.trips t
|
||||
INNER JOIN tracksolid.devices d USING (imei)
|
||||
WHERE d.vehicle_number = 'KAA 123A'
|
||||
AND t.start_time >= NOW() - INTERVAL '24 hours'
|
||||
ORDER BY t.start_time DESC;
|
||||
"
|
||||
```
|
||||
|
||||
### Checking Recent Alarms
|
||||
|
||||
```bash
|
||||
docker compose exec timescale_db psql -U $POSTGRES_USER -d tracksolid_db -c "
|
||||
SELECT
|
||||
d.vehicle_number, d.driver_name,
|
||||
a.alarm_type, a.alarm_name, a.alarm_time, a.speed
|
||||
FROM tracksolid.alarms a
|
||||
INNER JOIN tracksolid.devices d USING (imei)
|
||||
WHERE a.alarm_time >= NOW() - INTERVAL '24 hours'
|
||||
ORDER BY a.alarm_time DESC
|
||||
LIMIT 20;
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Maintenance
|
||||
|
||||
### Restart Grafana Only
|
||||
|
||||
```bash
|
||||
docker compose restart grafana
|
||||
```
|
||||
|
||||
### Restart Full Stack
|
||||
|
||||
```bash
|
||||
docker compose down && docker compose up -d
|
||||
```
|
||||
|
||||
### Update the Dashboard JSON
|
||||
|
||||
1. Edit `grafana/provisioning/dashboards-json/noc_fleet_dashboard.json`
|
||||
2. Grafana auto-reloads within 30 seconds (no restart needed)
|
||||
3. Commit the change: `git add grafana/ && git commit -m "Update NOC dashboard"`
|
||||
|
||||
### Check Container Resource Usage
|
||||
|
||||
```bash
|
||||
docker stats grafana timescale_db --no-stream
|
||||
```
|
||||
|
||||
### Grafana Data Volume Size
|
||||
|
||||
```bash
|
||||
docker system df -v | grep grafana-data
|
||||
```
|
||||
|
||||
### View Grafana Version
|
||||
|
||||
```bash
|
||||
docker compose exec grafana grafana-server -v
|
||||
```
|
||||
|
||||
Expected: `Version 11.0.0`
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Task | Command |
|
||||
|---|---|
|
||||
| Start Grafana | `docker compose up -d grafana` |
|
||||
| Stop Grafana | `docker compose stop grafana` |
|
||||
| Restart Grafana | `docker compose restart grafana` |
|
||||
| View logs | `docker compose logs grafana --tail=50` |
|
||||
| Check provisioning | `docker compose logs grafana \| grep -i provision` |
|
||||
| Test datasource | `curl -s -u admin:$GRAFANA_ADMIN_PASSWORD http://localhost:3000/api/datasources/uid/tracksolid_pg/health` |
|
||||
| Open dashboard | `http://localhost:3000/d/noc-fleet-live` |
|
||||
| Fleet summary | `SELECT connectivity_status, COUNT(*) FROM tracksolid.v_fleet_status GROUP BY 1;` |
|
||||
| Ingestion health | `SELECT * FROM tracksolid.v_ingestion_health ORDER BY endpoint;` |
|
||||
|
|
@ -12,6 +12,10 @@ REVISIONS (QA-Verified):
|
|||
[FIX-E04] Signal Handling: Clean pool closure on SIGTERM/SIGINT.
|
||||
[FIX-E05] Removed poll_obd: OBD data is push-only via /pushobd webhook.
|
||||
[FIX-11] Uses shared safe_task/setup_shutdown from ts_shared_rev (DRY).
|
||||
[FIX-E06] BUG-01: jimi.device.alarm.list returns alertTypeId/alarmTypeName/
|
||||
alertTime — not alarmType/alarmName/alarmTime (those are webhook
|
||||
field names). Corrected field mapping so alarm_type and alarm_name
|
||||
are no longer silently stored as NULL.
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
"""
|
||||
|
||||
|
|
@ -64,20 +68,25 @@ def poll_alarms():
|
|||
with conn.cursor() as cur:
|
||||
for a in alarms:
|
||||
lat, lng = clean_num(a.get("lat")), clean_num(a.get("lng"))
|
||||
# [FIX-E06] Poll response uses alertTypeId/alarmTypeName/alertTime,
|
||||
# not alarmType/alarmName/alarmTime (those are webhook push field names).
|
||||
alarm_type = clean(a.get("alertTypeId"))
|
||||
alarm_name = clean(a.get("alarmTypeName"))
|
||||
alarm_time = clean_ts(a.get("alertTime"))
|
||||
|
||||
cur.execute("""
|
||||
INSERT INTO tracksolid.alarms (
|
||||
imei, alarm_type, alarm_time, geom, lat, lng,
|
||||
speed, acc_status, updated_at
|
||||
imei, alarm_type, alarm_name, alarm_time, geom, lat, lng,
|
||||
speed, acc_status, source, updated_at
|
||||
) VALUES (
|
||||
%s, %s, %s,
|
||||
%s, %s, %s, %s,
|
||||
CASE WHEN %s IS NOT NULL AND %s IS NOT NULL
|
||||
THEN ST_SetSRID(ST_MakePoint(%s, %s), 4326)
|
||||
ELSE NULL END,
|
||||
%s, %s, %s, %s, NOW()
|
||||
%s, %s, %s, %s, 'poll', NOW()
|
||||
) ON CONFLICT (imei, alarm_type, alarm_time) DO NOTHING
|
||||
""", (
|
||||
a.get("imei"), clean(a.get("alarmType")), clean_ts(a.get("alarmTime")),
|
||||
a.get("imei"), alarm_type, alarm_name, alarm_time,
|
||||
lng, lat, lng, lat, lat, lng,
|
||||
clean_num(a.get("speed")), clean(a.get("accStatus"))
|
||||
))
|
||||
|
|
|
|||
|
|
@ -9,10 +9,27 @@ REVISIONS (QA-Verified):
|
|||
[FIX-M05] Batching: Groups 50 IMEIs per API call (API Limit Compliance).
|
||||
[FIX-M07] Signal Handling: Clean DB pool closure on SIGTERM/SIGINT.
|
||||
[FIX-M08] Atomic Logging: log_ingestion happens within the data transaction.
|
||||
[FIX-QA-01] Distance: Explicit km to meters conversion (* 1000).
|
||||
[FIX-11] Uses shared safe_task/setup_shutdown from ts_shared_rev (DRY).
|
||||
[FIX-M09] Trips: Captures runTimeSecond and maxSpeed from API.
|
||||
[FIX-M10] Parking: New poll_parking via jimi.open.platform.report.parking.
|
||||
[FIX-M11] BUG-02: distance_m was stored in millimetres due to erroneous
|
||||
* 1000 on an already-metric API value. Removed multiplication;
|
||||
column renamed to distance_km in migration 04. Both poll_trips
|
||||
and push_trip_report (webhook) corrected.
|
||||
[FIX-M12] BUG-02: Renamed distance_m → distance_km in all SQL to match
|
||||
migration 04 schema change.
|
||||
[FIX-M13] POLL-02: Parking poll was returning 0 rows — added missing
|
||||
acc_type=0 and account params; fixed response field durSecond
|
||||
(was mapped as 'seconds').
|
||||
[FIX-M14] POLL-01: New poll_track_list() — calls jimi.device.track.list
|
||||
per device every 30 minutes to capture high-resolution GPS
|
||||
waypoints between the 60-second fleet sweep snapshots. Writes to
|
||||
position_history with source='track_list'. Fills gaps in route
|
||||
reconstruction and enables accurate path drawing on maps.
|
||||
[FIX-M15] POLL-03: New get_device_locations() utility — calls
|
||||
jimi.device.location.get for up to 50 specific IMEIs on demand.
|
||||
Used for precision refreshes (alarm enrichment, stale device
|
||||
recovery) without waiting for the next full fleet sweep.
|
||||
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
"""
|
||||
|
||||
|
|
@ -174,21 +191,22 @@ def poll_trips():
|
|||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
for t in trips:
|
||||
# [FIX-M11] API returns distance in km. Store directly as distance_km.
|
||||
# Previous code multiplied by 1000 (→ mm), which was wrong.
|
||||
dist_km = clean_num(t.get("distance"))
|
||||
dist_m = dist_km * 1000 if dist_km is not None else 0 # [QA-01] Conversion
|
||||
cur.execute("""
|
||||
INSERT INTO tracksolid.trips (
|
||||
imei, start_time, end_time, distance_m,
|
||||
imei, start_time, end_time, distance_km,
|
||||
avg_speed_kmh, max_speed_kmh, driving_time_s, source
|
||||
) VALUES (%s, %s, %s, %s, %s, %s, %s, 'poll')
|
||||
ON CONFLICT (imei, start_time) DO UPDATE SET
|
||||
end_time = EXCLUDED.end_time,
|
||||
distance_m = EXCLUDED.distance_m,
|
||||
distance_km = EXCLUDED.distance_km,
|
||||
max_speed_kmh = COALESCE(EXCLUDED.max_speed_kmh, tracksolid.trips.max_speed_kmh),
|
||||
driving_time_s = COALESCE(EXCLUDED.driving_time_s, tracksolid.trips.driving_time_s)
|
||||
""", (
|
||||
t.get("imei"), clean_ts(t.get("startTime")), clean_ts(t.get("endTime")),
|
||||
dist_m, clean_num(t.get("avgSpeed")),
|
||||
dist_km, clean_num(t.get("avgSpeed")),
|
||||
clean_num(t.get("maxSpeed")), clean_int(t.get("runTimeSecond"))
|
||||
))
|
||||
inserted += 1
|
||||
|
|
@ -208,10 +226,14 @@ def poll_parking():
|
|||
|
||||
for i in range(0, len(imeis), 50):
|
||||
batch = imeis[i:i+50]
|
||||
# [FIX-M13] Added account + acc_type=0 (all stop types). Without these
|
||||
# the API returns empty results even when parking events exist.
|
||||
resp = api_post("jimi.open.platform.report.parking", {
|
||||
"imeis": ",".join(batch),
|
||||
"account": TARGET_ACCOUNT,
|
||||
"imeis": ",".join(batch),
|
||||
"begin_time": start_ts.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"end_time": end_ts.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"end_time": end_ts.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"acc_type": 0,
|
||||
}, token)
|
||||
|
||||
events = resp.get("result") or []
|
||||
|
|
@ -236,7 +258,7 @@ def poll_parking():
|
|||
) ON CONFLICT (imei, start_time, event_type) DO NOTHING
|
||||
""", (
|
||||
imei, start_time, clean_ts(p.get("endTime")),
|
||||
clean_int(p.get("seconds")),
|
||||
clean_int(p.get("durSecond")), # [FIX-M13] API returns durSecond, not seconds
|
||||
lng, lat, lng, lat,
|
||||
clean(p.get("address"))
|
||||
))
|
||||
|
|
@ -245,21 +267,173 @@ def poll_parking():
|
|||
int((time.time() - t0) * 1000), True)
|
||||
log.info("Parking: %d events processed.", inserted)
|
||||
|
||||
# ── 5. High-Resolution GPS Trail (Every 30m) — POLL-01 ───────────────────────
|
||||
|
||||
def poll_track_list():
|
||||
"""[FIX-M14] Fetch per-device GPS waypoint trails via jimi.device.track.list.
|
||||
|
||||
The 60-second fleet sweep (poll_live_positions) captures only the most
|
||||
recent fix per vehicle — all motion between sweeps is invisible. This
|
||||
function retrieves every waypoint the device logged in the last 35 minutes
|
||||
(5-min overlap ensures no gaps at scheduling boundaries) and inserts them
|
||||
into position_history with source='track_list'.
|
||||
|
||||
Impact on reporting:
|
||||
- position_history row density increases from ~1/min to ~1–4/min per device
|
||||
- Route traces in Grafana become accurate continuous paths
|
||||
- Speed profile queries gain meaningful resolution (avg over 10s intervals
|
||||
vs 60s intervals) — enables hard-braking / harsh-acceleration detection
|
||||
- v_mileage_daily_cagg continuous aggregate gains finer odometer deltas
|
||||
"""
|
||||
t0 = time.time()
|
||||
token, imeis = get_token(), get_active_imeis()
|
||||
if not token or not imeis:
|
||||
return
|
||||
|
||||
end_ts = datetime.now(timezone.utc)
|
||||
start_ts = end_ts - timedelta(minutes=35) # 5-min overlap avoids boundary gaps
|
||||
total_inserted = 0
|
||||
devices_with_data = 0
|
||||
|
||||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
for imei in imeis:
|
||||
resp = api_post("jimi.device.track.list", {
|
||||
"imei": imei,
|
||||
"begin_time": start_ts.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"end_time": end_ts.strftime("%Y-%m-%d %H:%M:%S"),
|
||||
"map_type": "GOOGLE",
|
||||
}, token)
|
||||
|
||||
waypoints = resp.get("result") or []
|
||||
if not waypoints:
|
||||
continue
|
||||
|
||||
inserted = 0
|
||||
for wp in waypoints:
|
||||
lat = clean_num(wp.get("lat"))
|
||||
lng = clean_num(wp.get("lng"))
|
||||
gps_time = clean_ts(wp.get("gpsTime"))
|
||||
if not is_valid_fix(lat, lng) or not gps_time:
|
||||
continue
|
||||
|
||||
cur.execute("""
|
||||
INSERT INTO tracksolid.position_history (
|
||||
imei, gps_time, geom, lat, lng,
|
||||
speed, direction, acc_status, source
|
||||
) VALUES (
|
||||
%s, %s,
|
||||
ST_SetSRID(ST_MakePoint(%s, %s), 4326),
|
||||
%s, %s, %s, %s, %s, 'track_list'
|
||||
)
|
||||
ON CONFLICT (imei, gps_time) DO NOTHING
|
||||
""", (
|
||||
imei, gps_time,
|
||||
lng, lat, # ST_MakePoint(lng, lat)
|
||||
lat, lng, # lat, lng columns
|
||||
clean_num(wp.get("gpsSpeed")),
|
||||
clean_num(wp.get("direction")),
|
||||
clean(wp.get("accStatus")),
|
||||
))
|
||||
inserted += 1
|
||||
|
||||
if inserted:
|
||||
total_inserted += inserted
|
||||
devices_with_data += 1
|
||||
|
||||
log_ingestion(cur, "jimi.device.track.list", len(imeis),
|
||||
0, total_inserted, int((time.time() - t0) * 1000), True)
|
||||
conn.commit()
|
||||
|
||||
log.info("Track list: %d waypoints inserted across %d/%d devices.",
|
||||
total_inserted, devices_with_data, len(imeis))
|
||||
|
||||
|
||||
# ── 6. On-Demand Device Location Refresh — POLL-03 ───────────────────────────
|
||||
|
||||
def get_device_locations(imeis: list) -> int:
|
||||
"""[FIX-M15] Precision position refresh for a specific list of IMEIs.
|
||||
|
||||
Calls jimi.device.location.get (up to 50 IMEIs per call) and upserts
|
||||
results into live_positions. Use this for:
|
||||
- Alarm enrichment: get exact position immediately after an alarm fires
|
||||
- Stale device recovery: force-refresh a vehicle that has been offline
|
||||
- Dashboard on-demand refresh without waiting for the 60s fleet sweep
|
||||
|
||||
Returns the number of positions successfully upserted.
|
||||
"""
|
||||
token = get_token()
|
||||
if not token or not imeis:
|
||||
return 0
|
||||
|
||||
upserted = 0
|
||||
with get_conn() as conn:
|
||||
with conn.cursor() as cur:
|
||||
for i in range(0, len(imeis), 50):
|
||||
batch = imeis[i:i + 50]
|
||||
resp = api_post("jimi.device.location.get", {
|
||||
"imeis": ",".join(batch),
|
||||
"map_type": "GOOGLE",
|
||||
}, token)
|
||||
|
||||
positions = resp.get("result") or []
|
||||
for p in positions:
|
||||
imei = p.get("imei")
|
||||
lat = clean_num(p.get("lat"))
|
||||
lng = clean_num(p.get("lng"))
|
||||
if not imei or not is_valid_fix(lat, lng):
|
||||
continue
|
||||
|
||||
cur.execute("""
|
||||
INSERT INTO tracksolid.live_positions (
|
||||
imei, geom, lat, lng, speed, direction,
|
||||
gps_time, acc_status, current_mileage, recorded_at
|
||||
) VALUES (
|
||||
%s, ST_SetSRID(ST_MakePoint(%s, %s), 4326),
|
||||
%s, %s, %s, %s, %s, %s, %s, NOW()
|
||||
)
|
||||
ON CONFLICT (imei) DO UPDATE SET
|
||||
geom = EXCLUDED.geom,
|
||||
lat = EXCLUDED.lat,
|
||||
lng = EXCLUDED.lng,
|
||||
speed = EXCLUDED.speed,
|
||||
direction = EXCLUDED.direction,
|
||||
gps_time = EXCLUDED.gps_time,
|
||||
acc_status = EXCLUDED.acc_status,
|
||||
current_mileage = EXCLUDED.current_mileage,
|
||||
updated_at = NOW()
|
||||
""", (
|
||||
imei, lng, lat, lat, lng,
|
||||
clean_num(p.get("speed")),
|
||||
clean_num(p.get("direction")),
|
||||
clean_ts(p.get("gpsTime")),
|
||||
clean(p.get("accStatus")),
|
||||
clean_num(p.get("currentMileage")),
|
||||
))
|
||||
upserted += 1
|
||||
|
||||
conn.commit()
|
||||
log.info("get_device_locations: %d positions refreshed.", upserted)
|
||||
return upserted
|
||||
|
||||
|
||||
# ── Main Loop ─────────────────────────────────────────────────────────────────
|
||||
|
||||
def main():
|
||||
log.info("Starting MOVEMENT PIPELINE (v2.1)...")
|
||||
log.info("Starting MOVEMENT PIPELINE (v2.2)...")
|
||||
|
||||
# Startup catch-up
|
||||
safe_task(sync_devices, log)()
|
||||
safe_task(poll_live_positions, log)()
|
||||
safe_task(poll_trips, log)()
|
||||
safe_task(poll_parking, log)()
|
||||
safe_task(poll_track_list, log)()
|
||||
|
||||
# Schedule
|
||||
schedule.every(60).seconds.do(safe_task(poll_live_positions, log))
|
||||
schedule.every(15).minutes.do(safe_task(poll_trips, log))
|
||||
schedule.every(15).minutes.do(safe_task(poll_parking, log))
|
||||
schedule.every(30).minutes.do(safe_task(poll_track_list, log)) # [FIX-M14]
|
||||
schedule.every().day.at("02:00").do(safe_task(sync_devices, log))
|
||||
|
||||
while True:
|
||||
|
|
|
|||
102
run_migrations.sh
Executable file
102
run_migrations.sh
Executable file
|
|
@ -0,0 +1,102 @@
|
|||
#!/usr/bin/env bash
|
||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
# run_migrations.sh — Tracksolid DB Migration Runner
|
||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
# Applies any pending numbered SQL migration files in order.
|
||||
# Safe to run on every deployment — skips already-applied migrations.
|
||||
#
|
||||
# Usage:
|
||||
# ./run_migrations.sh # auto-detect container + DB
|
||||
# ./run_migrations.sh --dry-run # show what would run, don't apply
|
||||
#
|
||||
# Called by Coolify post-deployment command:
|
||||
# bash /app/run_migrations.sh
|
||||
#
|
||||
# How it tracks applied migrations:
|
||||
# Creates tracksolid.schema_migrations table on first run.
|
||||
# Records the filename of every successfully applied migration.
|
||||
# Skips any file already recorded in that table.
|
||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# ── Config ────────────────────────────────────────────────────────────────────
|
||||
DB_NAME="${DB_NAME:-tracksolid_db}"
|
||||
DB_USER="${DB_USER:-postgres}"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
DRY_RUN=false
|
||||
|
||||
# ── Argument parsing ──────────────────────────────────────────────────────────
|
||||
for arg in "$@"; do
|
||||
case $arg in
|
||||
--dry-run) DRY_RUN=true ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# ── Resolve TimescaleDB container ─────────────────────────────────────────────
|
||||
TS_DB=$(docker ps --filter "name=timescale_db" --format "{{.Names}}" | head -1)
|
||||
if [[ -z "$TS_DB" ]]; then
|
||||
echo "ERROR: no running timescale_db container found." >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "Using container: $TS_DB"
|
||||
|
||||
# ── Helper: run SQL against the DB ───────────────────────────────────────────
|
||||
run_sql() {
|
||||
docker exec -i "$TS_DB" psql -U "$DB_USER" -d "$DB_NAME" "$@"
|
||||
}
|
||||
|
||||
# ── Ensure migration tracking table exists ───────────────────────────────────
|
||||
run_sql -c "
|
||||
CREATE TABLE IF NOT EXISTS tracksolid.schema_migrations (
|
||||
filename TEXT PRIMARY KEY,
|
||||
applied_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
" > /dev/null
|
||||
|
||||
# ── Find and apply pending migrations ────────────────────────────────────────
|
||||
MIGRATION_FILES=$(find "$SCRIPT_DIR" -maxdepth 1 -name '[0-9][0-9]_*.sql' | sort)
|
||||
|
||||
if [[ -z "$MIGRATION_FILES" ]]; then
|
||||
echo "No migration files found in $SCRIPT_DIR"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
APPLIED=0
|
||||
SKIPPED=0
|
||||
|
||||
for filepath in $MIGRATION_FILES; do
|
||||
filename=$(basename "$filepath")
|
||||
|
||||
# Check if already applied
|
||||
already_applied=$(run_sql -t -c \
|
||||
"SELECT COUNT(*) FROM tracksolid.schema_migrations WHERE filename = '$filename';" \
|
||||
2>/dev/null | tr -d '[:space:]')
|
||||
|
||||
if [[ "$already_applied" == "1" ]]; then
|
||||
echo " SKIP $filename (already applied)"
|
||||
((SKIPPED++)) || true
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo " PENDING $filename (would apply)"
|
||||
continue
|
||||
fi
|
||||
|
||||
echo " APPLY $filename ..."
|
||||
if run_sql < "$filepath"; then
|
||||
# Record successful application
|
||||
run_sql -c \
|
||||
"INSERT INTO tracksolid.schema_migrations (filename) VALUES ('$filename');" \
|
||||
> /dev/null
|
||||
echo " OK $filename"
|
||||
((APPLIED++)) || true
|
||||
else
|
||||
echo " FAIL $filename — aborting migration run" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "Migrations complete: $APPLIED applied, $SKIPPED skipped."
|
||||
|
|
@ -1655,10 +1655,11 @@ Common parameters: `deviceImei`, `proNo` (protocol number), `cmdContent` (JSON),
|
|||
| `jimi.track.device.detail` | `ingest_movement_rev.py` | In use |
|
||||
| `jimi.user.device.location.list` | `ingest_movement_rev.py` | In use |
|
||||
| `jimi.device.track.mileage` | `ingest_movement_rev.py` | In use |
|
||||
| `jimi.device.alarm.list` | `ingest_events_rev.py` | In use (verify field names) |
|
||||
| `jimi.device.alarm.list` | `ingest_events_rev.py` | In use — field mapping corrected (FIX-E06) |
|
||||
| `jimi.device.obd.list` | `ingest_events_rev.py` | **Not a real endpoint** — OBD is push-only |
|
||||
| `jimi.device.track.list` | — | Not used (high-res GPS trails) |
|
||||
| `jimi.open.platform.report.parking` | — | Not used (parking_events table exists) |
|
||||
| `jimi.device.track.list` | `ingest_movement_rev.py` | **In use** — poll_track_list() every 30m (FIX-M14) |
|
||||
| `jimi.device.location.get` | `ingest_movement_rev.py` | **In use** — get_device_locations() on-demand (FIX-M15) |
|
||||
| `jimi.open.platform.report.parking` | `ingest_movement_rev.py` | **In use** — acc_type/durSecond fixed (FIX-M13) |
|
||||
| `jimi.device.jimi.media.URL` | — | Not used (media catalog) |
|
||||
| `jimi.device.media.event.URL` | — | Not used (alarm-triggered media) |
|
||||
| All Data Push endpoints | — | Not used (webhook receiver needed) |
|
||||
|
|
|
|||
1544
tracksolid_DB_manual.md
Normal file
1544
tracksolid_DB_manual.md
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -408,8 +408,9 @@ def push_trip_report(token: str = Form(""), data_list: str = Form("")):
|
|||
if not imei or not begin_time:
|
||||
continue
|
||||
|
||||
miles_km = clean_num(item.get("miles"))
|
||||
distance_m = miles_km * 1000 if miles_km is not None else None
|
||||
# [FIX-M11] API sends miles (km). Store directly as distance_km.
|
||||
# Previous code multiplied by 1000, producing mm not m.
|
||||
distance_km = clean_num(item.get("miles"))
|
||||
|
||||
begin_lat = clean_num(item.get("beginLat"))
|
||||
begin_lng = clean_num(item.get("beginLng"))
|
||||
|
|
@ -418,7 +419,7 @@ def push_trip_report(token: str = Form(""), data_list: str = Form("")):
|
|||
|
||||
cur.execute("""
|
||||
INSERT INTO tracksolid.trips (
|
||||
imei, start_time, end_time, distance_m,
|
||||
imei, start_time, end_time, distance_km,
|
||||
start_geom, end_geom,
|
||||
fuel_consumed_l, idle_time_s, trip_seq, source,
|
||||
updated_at
|
||||
|
|
@ -433,13 +434,13 @@ def push_trip_report(token: str = Form(""), data_list: str = Form("")):
|
|||
%s, %s, %s, 'push', NOW()
|
||||
) ON CONFLICT (imei, start_time) DO UPDATE SET
|
||||
end_time = EXCLUDED.end_time,
|
||||
distance_m = EXCLUDED.distance_m,
|
||||
distance_km = EXCLUDED.distance_km,
|
||||
end_geom = EXCLUDED.end_geom,
|
||||
fuel_consumed_l = EXCLUDED.fuel_consumed_l,
|
||||
idle_time_s = EXCLUDED.idle_time_s,
|
||||
updated_at = NOW()
|
||||
""", (
|
||||
imei, begin_time, end_time, distance_m,
|
||||
imei, begin_time, end_time, distance_km,
|
||||
begin_lng, begin_lat, begin_lng, begin_lat,
|
||||
end_lng, end_lat, end_lng, end_lat,
|
||||
clean_num(item.get("oils")),
|
||||
|
|
|
|||
Loading…
Reference in a new issue