Real-time Data Pipeline Development for Circular Economy Insights

As a startup in the Circular Economy sector, we aim to provide cutting-edge solutions that promote sustainable business practices. We are looking for an experienced data engineer to design and implement a real-time data pipeline that can ingest, process, and analyze data from various sources such as IoT sensors, CRM systems, and external sustainability databases. The pipeline must facilitate real-time analytics to help clients optimize their resource usage and minimize waste. Key components of the project include setting up event streaming with Apache Kafka to handle data ingestion, utilizing Apache Spark for data processing, and integrating with a data warehouse like Snowflake or BigQuery for storage and analysis. Additionally, workflow orchestration should be managed using Apache Airflow, while dbt will be employed for data transformation. The solution must incorporate data observability practices to ensure pipeline reliability and performance. The ideal candidate should possess expertise in the aforementioned technologies and demonstrate a strong understanding of MLOps practices to eventually incorporate machine learning models into the analytics pipeline. By completing this project, we aim to empower businesses to make informed decisions that enhance their sustainability efforts and contribute positively to the circular economy.