Real-Time Data Pipeline for Enhanced Import & Export Operations

In the fast-paced world of import and export, timely and accurate data insights are critical for operational success. Our company is experiencing rapid growth, and we need a robust data engineering solution to streamline our operations. We are looking to build a real-time data pipeline leveraging Apache Kafka for event streaming, integrated with Snowflake for data storage and processing. The solution should include data observability and monitoring functionalities using tools like Datadog or Prometheus. Our goal is to ensure that our data is not only accessible in real-time but also reliable and actionable, supporting advanced analytics and decision-making processes. The ideal candidate will have experience with MLOps to facilitate model deployment and with data mesh architecture principles to ensure scalability and flexibility. Additionally, we aim to automate data workflows using Apache Airflow and dbt to enhance data transformation and loading processes. This project is crucial for maintaining our competitive edge in the market by reducing information latency and providing our operations team with the most up-to-date insights.