Real-Time Data Engineering for Building Design Optimization

As a scale-up architecture and design firm, we are committed to leveraging data-driven insights to optimize our building design process. Currently, our data is siloed across different platforms, leading to inefficiencies and delayed project timelines. We are seeking a skilled data engineer to develop a real-time data infrastructure that consolidates all our data streams into a single, accessible platform. The project involves setting up a data mesh architecture, where Apache Kafka will facilitate event streaming, while Spark and Databricks will handle big-data processing. We aim to use Airflow for orchestrating workflows and dbt for transforming and modeling data. Our goal is to implement real-time analytics that provide actionable insights, improving decision-making processes and enhancing client engagement. The successful implementation will reduce design iteration times, enhance collaboration among stakeholders, and ultimately increase client satisfaction.