Real-time Data Mesh Implementation for Optimizing Steel Production Processes

As a leading enterprise in the Steel & Metals industry, we are seeking to transform our production processes through advanced data engineering. The goal of this project is to design and implement a real-time data mesh architecture that facilitates dynamic and decentralized data management across our production facilities. The project will integrate cutting-edge technologies, including Apache Kafka for event streaming and Spark for real-time analytics, alongside dbt and Airflow for seamless data pipeline orchestration. By leveraging Snowflake and BigQuery, we will ensure robust data storage and access, while Databricks will be utilized for machine learning operations (MLOps). This initiative aims to provide our teams with real-time insights into production metrics, enabling proactive decision-making and fostering an environment of continuous improvement. The successful implementation of this data mesh will result in reduced operational downtimes, optimized production schedules, and cost-effective resource management. Our timeline spans approximately 16-24 weeks, with a flexible budget range of $50,000 to $150,000.