From 48M Records to Top 8 Finalists: How We Built an End-to-End Urban Flow AI

We have some exciting news to share! 🎉 Our team, DataMinds, competed in the prestigious SLIIT CodeFest Datathon 2026 and proudly secured a spot as Finalists (Top 8 teams) in the Urban Flow Analytics Data Challenge! In this competition, we tackled a massive, real-world urban transit dataset containing 48.6 million trip records across 260+ urban zones. The challenge demanded much more than just training a model in a Jupyter Notebook: it required building a production-ready, mathematically sound, and business-viable end-to-end data platform covering everything from chunked streaming data pipelines and spatio-temporal forecasting to an AI-powered executive platform. Here is the complete behind-the-scenes breakdown of our architectural strategy, the data engineering traps we avoided, our modeling breakthroughs, and the engineering principles that helped us reach the Top 8. 🎥 The 4-Minute Walkthrough If you prefer a visual walkthrough, check out our official demo video demonstrating the full pipeline, interactive dashboard, and conversational AI mobility assistant: (Link:

) 🏗️ High-Level System Architecture Rather than treating the challenge as isolated competition tasks, we engineered a cohesive 4-Tier Architecture that bridges raw streaming telemetry to executive decision-making: 1. Taming 48.6 Million Records Without Running Out of Memory Working with tens of millions of raw geospatial records presents two massive hurdles: - Memory limits: A naive pd.read_csv() on 48.6 million rows with 20+ columns will immediately crash standard workstations or cloud instances with Out-Of-Memory (OOM) errors. - Real-world data anomalies: Real mobility data is filled with sensor noise, GPS dropouts, fare metering glitches, and data logging bugs. Chunked Streaming Ingestion To maintain sub-gigabyte RAM footprints during data auditing, we used Python generators with chunked processing (chunksize=100_000 ): def stream_audit(csv_path, chunk_size=100_000): total_records = 0 anomalies = {"negative_fare": 0, "excessive_speed": 0, "zero_duration": 0} for chunk in pd.read_csv(csv_path, chunksize=chunk_size): # Calculate trip duration and implied speed duration_hours = (chunk['dropoff_datetime'] - chunk['pickup_datetime']).dt.total_seconds() / 3600.0 implied_mph = chunk['trip_distance'] / duration_hours.replace(0, np.nan) # Track edge cases anomalies["negative_fare"] += (chunk['fare_amount'] 100).sum() anomalies["zero_duration"] += (duration_hours <= 0).sum() total_records += len(chunk) return total_records, anomalies The "Speed Trap" and Temporal Anomaly Cleaning Through our systematic audit across all 48.6M rows, we discovered and programmatically filtered out critical real-world edge cases: - Teleportation glitches: Trips logging positive distance but zero duration. - Physical speed violations: Urban taxi trips logging implied velocities exceeding 100 mph (often GPS drift or corrupt timestamps). - Billing corrections & disputes: Trips with negative fares or negative tip amounts. - Zone boundary mismatches: Trips with missing or non-existent zone IDs. 2. Preventing Data Leakage (The Competition Trap) A common mistake in ML hackathons is data leakage. If you leak post-trip information into an upfront prediction model, your leaderboard score looks amazing, but your model is completely useless in production. Chronological Holdouts vs. Random Splits Random train_test_split on time-series mobility data is fatal-it allows models to learn from the future to predict the past. We enforced strict chronological partitioning: - Training Period: 9 months of historical data - Validation Period: 2 months of subsequent data - Testing Holdout: Final future month (never seen during feature extraction or tuning) Pre-Trip Only Features For our Upfront Fare and Duration models, we restricted feature engineering strictly to information available before the passenger steps into the vehicle: - Cyclical Temporal Encodings: sin(2π * hour / 24) andcos(2π * hour / 24) to preserve diurnal continuity (midnight connects smoothly to 1 AM). - Historical Zone Congestion: Rolling 4-week historical pickup velocity per zone pair. - Surge Multipliers: Real-time demand-to-supply ratio proxy computed dynamically. 3. The Supervised Modeling Tournament We benchmarked multiple architectures across our holdout test set to select the optimal production model: | Model Architecture | Fare R² | Fare RMSE ($) | Duration R² | Duration RMSE (min) | Inference Latency | |---|---|---|---|---|---| | Ordinary Least Squares (OLS) | 0.812 | $7.14 | 0.621 | 9.85 min | 0.8 ms | | Random Forest Regressor | 0.941 | $4.12 | 0.789 | 6.94 min | 45.2 ms | | LightGBM Regressor (Winner - Fare) | 0.9657 | $3.25 | 0.8142 | 6.55 min | 2.1 ms | | XGBoost Regressor (Winner - Duration) | 0.9612 | $3.41 | 0.8268 | 6.39 min | 3.4 ms | Why Gradient Boosting Triumphed Gradient boosting trees handled the non-linear relationship between Manhattan distance, toll zones, and peak-hour traffic multipliers effortlessly. LightGBM provided lightning-fast inference with sub-cent precision, while XGBoost effectively captured the heavy-tailed variance in urban traffic delays. 4. Spatio-Temporal Forecasting & Origin-Destination Corridors Urban mobility is heavily spatial. Having accurate pricing is only half the battle; fleet operators must know where demand will surge 24, 48, and 72 hours in advance. Autoregressive Lag-Based Forecasting We aggregated zone-level trip demand into hourly buckets and engineered multi-scale temporal lag features: - Immediate Autoregressive Lags: t-1 ,t-2 ,t-3 hours. - Diurnal Seasonality Lags: t-24 ,t-48 ,t-72 hours (same hour over preceding days). - Weekly Seasonality Lags: t-168 hours (same day and hour of the previous week). Our forecaster achieved a test RMSE of ~1.00 to 1.22 trips/hour across all major urban zones, allowing dispatchers to pre-position fleet vehicles before surges materialized. 4 Time-Slice OD Flow Clustering By clustering pickup-to-dropoff vectors across Morning Rush (07:00-10:00), Midday (11:00-14:00), Evening Rush (16:00-19:00), and Late Night (22:00-02:00), we revealed major commercial arterial corridors and airport shuttle dynamics, illuminating severe deadheading (empty return) imbalances. 5. From Raw Code to Executive Decision-Making Judges at modern hackathons don't just want .ipynb files; they want to see how engineering impacts business. We translated our models into an Executive Command Center built with Streamlit: 1. The Conversational AI Mobility Assistant We integrated a natural language interface that allows city planners and dispatch managers to ask plain-English questions: "What are the top 5 revenue-generating pickup zones during the Friday evening rush?" The Innovation - Ambiguity Guardrails: Real users often ask vague questions like "Show me the best zones". Rather than letting the AI hallucinate or make unsafe assumptions, our engine implements strict guardrails that detect ambiguity and clarify whether the user intends "highest volume", "highest fare margin", or "fastest turnaround time". 2. The $51.8M ROI Financial Model We mapped model improvements directly to bottom-line business metrics: - Deadhead Reduction: Pre-dispatching idle vehicles based on our 24h forecaster reduces empty miles by 14.2%. - Airport Reverse-Trips: Pairing drop-offs at airport terminals with immediate outbound demand captures substantial hidden margins. - Estimated Annual Network Gain: Over $51.8 Million across the city transit ecosystem. 6. Four Lessons That Helped Us Secure the Top 8 If you are competing in data science competitions or datathons, here are four principles that made the difference for Team DataMinds: - Clean Data Beats Fancy Ensembles: Spending 40% of our time auditing anomalies across the 48.6M records yielded vastly higher accuracy gains than endless hyperparameter tuning on noisy data. - Respect the Temporal Dimension: Never use standard K-fold cross-validation or random splits on temporal data. A leak-free validation strategy ensures that your local score matches real-world performance. - Build Modular, Reusable Code: We moved all logic out of messy notebooks into modular Python packages ( src/data_cleaner.py ,src/supervised_models.py ,src/demand_forecaster.py ). This allowed our notebook, test scripts, and UI to share the exact same underlying logic. - Tell a Clear Story: A technical report or dashboard should not just be a collection of charts. Frame your findings as a business narrative: Problem → Solution → Measurable Economic Value. 🏁 Looking Forward to the Finals! Reaching the Top 8 Finalist stage among brilliant teams across the country is an incredible milestone for Team DataMinds. A huge thank you to the SLIIT CodeFest Datathon 2026 organizers and judges for organizing such an inspiring, high-impact data challenge. Are you building with large-scale mobility data or competing in data challenges? Drop your thoughts or questions in the comments below! Top comments (0)