Scalable Real-Time Data Infrastructure for Urban Planning Insights

As a scale-up in the urban planning industry, we are committed to leveraging data to drive smarter, more efficient urban development strategies. Our current data infrastructure faces limitations in handling real-time data and providing actionable insights, which hinders our ability to promptly address urban challenges. This project seeks a skilled data engineer to design and implement a state-of-the-art data infrastructure that supports real-time analytics. Key components include deploying Apache Kafka for event streaming, Apache Spark for big data processing, and Airflow for workflow automation. The use of dbt and Snowflake will ensure robust data modeling and storage solutions, while BigQuery provides powerful analytics capabilities. Additionally, incorporating data mesh principles and MLOps will enhance scalability and model deployment processes. The solution must be robust, efficient, and capable of integrating with our existing systems to provide seamless data observability and continuous improvement in urban planning operations.