Real-time Data Pipeline Implementation for Enhanced Design Insights

As an ambitious architecture and design startup, we are committed to transforming how design projects manage data. We are looking for a skilled data engineer to establish a robust real-time data pipeline that integrates architectural design data from various sources. The project involves configuring and deploying a system utilizing tools like Apache Kafka for data streaming, Spark for data processing, Airflow for orchestrating data workflows, and Snowflake or BigQuery for data warehousing. The goal is to enable our designers and architects to access and analyze data in real-time, thus significantly improving the decision-making process and project outcomes. This system will support a data mesh approach, ensuring that data is easily accessible, interoperable, and remains the single source of truth across the organization. Additionally, implementing data observability practices will ensure data quality and compliance, minimizing project risks and delays. We are looking for a solution that can be set up within a 4-6 week timeframe with a focus on scalability and future integration with machine learning models through MLOps.