Data Engineering
Vehicle Field Reliability Intelligence System
An end-to-end reliability intelligence platform for vehicle fleet data. Repair records land in PostgreSQL, move through dbt medallion models, and feed PySpark analytics computing Mean Time to Failure, rolling failure-rate windows, and regional failure clustering. A HuggingFace NLP pipeline classifies free-text repair descriptions into structured failure modes, and Prophet forecasts component failure rates 30, 60, and 90 days out.
Problem Statement
Fleet reliability problems announce themselves late. By the time a component failure shows up in aggregate warranty reporting, it has already propagated across a region or a model year. Repair records also arrive largely as unstructured free text, which makes systematic failure-mode analysis impossible without manual labeling. This system was built to detect emerging reliability risks from raw repair data before they escalate.
Tech Stack
Key Contributions
- Designed and orchestrated an 8-task Dockerized Airflow DAG covering data generation, ingestion, dbt transformation, and PySpark analytics, with automated retries and exponential backoff
- Built an NLP text-mining pipeline using HuggingFace Transformers (BART-MNLI zero-shot classification) to auto-classify free-text repair descriptions into structured failure categories without manual labeling
- Implemented PySpark analytics computing Mean Time to Failure per component and vehicle model, 30/60/90-day rolling failure-rate windows, and regional failure clustering
- Built a Prophet forecasting model projecting component failure rates 30, 60, and 90 days out to support proactive maintenance prioritization
- Implemented SHA-256 hash-based deduplication for idempotent re-runs, per-run audit logging, and dbt data-quality tests covering uniqueness, null validation, and accepted values
- Developed a Streamlit reliability dashboard with component health scores, ranked failure-mode tables, trend charts, and threshold alerts
Results
- Regional failure clustering surfaced hotspots running at 2.1x the baseline failure rate
- Free-text repair descriptions classified into structured failure modes automatically, making failure-mode analysis possible across tens of thousands of records
- Hash-based deduplication makes the pipeline idempotent, so re-runs are safe and produce no duplicate records
- Forecasted failure rates give a 30/60/90-day window for prioritizing maintenance before failures cluster
Engineering Decisions
- Chose zero-shot classification over a supervised classifier because the failure taxonomy needed to be extensible without relabeling the corpus every time a category was added
- Ran heavy aggregations in PySpark rather than SQL so rolling-window and clustering logic could scale past what the warehouse handles comfortably
- Made every DAG task idempotent via content hashing, so a partial failure can be re-run without corrupting downstream marts
- Kept dbt tests inside the DAG so a data-quality regression fails the pipeline instead of silently reaching the dashboard
Challenges Faced
- Prophet's Stan backend resolves its binary paths incorrectly on Apple Silicon, which required pinning and re-pathing the model backend before the forecasting task would run
- Airflow's containerized UID handling conflicted with local file ownership, breaking DAG writes until the container user was aligned with the host
- Protobuf version conflicts between the HuggingFace stack and the Spark and Airflow stack required careful dependency isolation