Real-time Data Pipeline Implementation for Predictive Maintenance in Steel Production

As a leading scale-up in the steel and metals industry, we recognize the strategic importance of predictive maintenance to maintain operational efficiency and reduce unexpected equipment failures. We aim to implement a robust data engineering solution that integrates real-time data streaming from our production line sensors with advanced analytics capabilities. This will enable us to predict maintenance needs before a breakdown occurs, thus reducing downtime and extending equipment life. The project will involve setting up a data mesh architecture using Apache Kafka for event streaming, Spark for real-time processing, and orchestrating workflows with Airflow. We plan to store and process data in Snowflake, utilizing dbt for data transformations and Databricks for advanced machine learning model deployment. Key objectives include improving data observability and implementing MLOps practices to ensure continuous model improvement. We seek a freelancer who can effectively liaise with our in-house IT and operations teams to deliver a scalable, high-performance solution that aligns with our business objectives. The successful candidate will demonstrate expertise in the latest data engineering technologies, with a proven track record of implementing similar projects.