Real-Time Data Pipeline Implementation for Predictive Maintenance

As a rapidly growing company in the manufacturing and production industry, we recognize the critical need for efficient operations and minimized downtime. Currently, our systems rely heavily on outdated batch processing methods which delay insights and decision-making. To overcome this, we aim to develop a real-time data pipeline capable of handling large volumes of sensor data across multiple production lines. This project will involve setting up event streaming using Apache Kafka and implementing data processing with Apache Spark. Airflow will orchestrate data workflows, while dbt will be used for data transformation. The processed data will be stored in Snowflake and BigQuery to facilitate real-time analytics. This pipeline will empower our predictive maintenance strategy, allowing us to preemptively identify and address machinery issues, thereby optimizing production line efficiency. We seek a freelancer with a strong background in data engineering and familiarity with the latest technologies in real-time analytics and data mesh architectures.