feat(crq): add CRQ ingestion via shared engine + thin inc/crq entrypoints
Split the INC-only loader into a dataset-agnostic engine (pipeline.py, renamed
from import_tickets.py) parameterized by a Dataset config, with thin per-type
entrypoints inc/import_inc.py and crq/import_crq.py. CRQ shares INC's identical
32-column source schema and CDC change stream, so the engine is fully shared.
- pipeline.py: Dataset config (name/table/prefixes/key_regex/post_apply); INC
keeps the capture_history post-apply hook, CRQ has none yet. geocode_locations
now unions tickets.crq (geocoding is cross-dataset: one gazetteer/budget).
- crq/import_crq.py: drains automations/crq/changes/ from isptickets into
tickets.crq (data layer + map; SLA/dashboard/history deferred).
- migrations/13_crq_columns.sql: CRQ mirror of 03 — typed STORED generated
columns + indexes on tickets.crq (reuses tickets.eat_ts()).
- Deployment: Dockerfile/run_ingest.sh run both via `python -m`; pyproject
packages inc/crq. Docs (README, implementation, deployment-and-operations,
n8n export ref, phase-1) updated for the split + the one-time CRQ seed runbook.
tickets.crq already exists (mig 01, LIKE tickets.inc) and is unioned into
reporting.fn_tickets_for_map + resolve_ticket_geoms, so CRQ appears on the
existing Tickets map once seeded. Verified locally: ruff-clean new files, engine
lists/parses both streams against live S3 (crq=52 files, inc unaffected).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 20:16:38 +00:00
|
|
|
# fleettickets — INC + CRQ ticket ingestion image (Coolify-deployable).
|
2026-06-15 17:08:05 +00:00
|
|
|
# A small batch/cron worker: it has no web server. Coolify keeps the container
|
feat(crq): add CRQ ingestion via shared engine + thin inc/crq entrypoints
Split the INC-only loader into a dataset-agnostic engine (pipeline.py, renamed
from import_tickets.py) parameterized by a Dataset config, with thin per-type
entrypoints inc/import_inc.py and crq/import_crq.py. CRQ shares INC's identical
32-column source schema and CDC change stream, so the engine is fully shared.
- pipeline.py: Dataset config (name/table/prefixes/key_regex/post_apply); INC
keeps the capture_history post-apply hook, CRQ has none yet. geocode_locations
now unions tickets.crq (geocoding is cross-dataset: one gazetteer/budget).
- crq/import_crq.py: drains automations/crq/changes/ from isptickets into
tickets.crq (data layer + map; SLA/dashboard/history deferred).
- migrations/13_crq_columns.sql: CRQ mirror of 03 — typed STORED generated
columns + indexes on tickets.crq (reuses tickets.eat_ts()).
- Deployment: Dockerfile/run_ingest.sh run both via `python -m`; pyproject
packages inc/crq. Docs (README, implementation, deployment-and-operations,
n8n export ref, phase-1) updated for the split + the one-time CRQ seed runbook.
tickets.crq already exists (mig 01, LIKE tickets.inc) and is unioned into
reporting.fn_tickets_for_map + resolve_ticket_geoms, so CRQ appears on the
existing Tickets map once seeded. Verified locally: ruff-clean new files, engine
lists/parses both streams against live S3 (crq=52 files, inc unaffected).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 20:16:38 +00:00
|
|
|
# running (CMD below) and fires the ingests via two Scheduled Tasks:
|
|
|
|
|
# python -m inc.import_inc --from-bucket --apply (cron: */20 6-20 * * *)
|
|
|
|
|
# python -m crq.import_crq --from-bucket --apply (cron: */20 6-20 * * *)
|
|
|
|
|
# (run from /app so the inc/ and crq/ packages + pipeline.py/shared.py import.)
|
2026-06-15 17:08:05 +00:00
|
|
|
# Env (set in Coolify): DATABASE_URL, RUSTFS_*, GEOCODER_*. S3 is via boto3 — no
|
|
|
|
|
# aws CLI needed. psycopg2-binary ships its own libpq, so no build toolchain.
|
|
|
|
|
FROM python:3.12-slim
|
|
|
|
|
|
|
|
|
|
ENV PYTHONUNBUFFERED=1 \
|
|
|
|
|
PIP_NO_CACHE_DIR=1 \
|
|
|
|
|
TZ=Africa/Nairobi
|
|
|
|
|
|
|
|
|
|
RUN apt-get update \
|
|
|
|
|
&& apt-get install -y --no-install-recommends tzdata \
|
|
|
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
|
|
|
|
|
|
WORKDIR /app
|
|
|
|
|
|
2026-07-02 06:47:15 +00:00
|
|
|
# Pinned, reproducible installs from uv.lock (FT-SEC-02): uv export --frozen fails
|
|
|
|
|
# the build if the lockfile drifts from pyproject.toml. Runtime imports straight
|
|
|
|
|
# from /app via `python -m inc.import_inc` — the project itself needs no install.
|
|
|
|
|
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv
|
|
|
|
|
COPY pyproject.toml uv.lock ./
|
|
|
|
|
RUN uv export --frozen --no-dev --no-emit-project --format requirements-txt -o /tmp/requirements.txt \
|
|
|
|
|
&& uv pip install --system -r /tmp/requirements.txt \
|
|
|
|
|
&& rm /tmp/requirements.txt
|
|
|
|
|
|
2026-06-15 17:08:05 +00:00
|
|
|
COPY . .
|
2026-07-02 06:47:15 +00:00
|
|
|
|
|
|
|
|
# Non-privileged runtime user (Coolify Scheduled Tasks exec as this user too).
|
|
|
|
|
RUN useradd -m tickets-user
|
|
|
|
|
USER tickets-user
|
2026-06-15 17:08:05 +00:00
|
|
|
|
|
|
|
|
# Keep the container alive so Coolify Scheduled Tasks can exec into it.
|
|
|
|
|
CMD ["tail", "-f", "/dev/null"]
|