Advanced Data Pipeline Development for Real-Time Medical Research Insights

As a growing SME in the Medical Research industry, we face challenges in efficiently integrating and analyzing large volumes of data from multiple sources, hindering our ability to derive timely insights crucial for our research studies. We are seeking a data engineering expert to design and implement a scalable data pipeline utilizing technologies such as Apache Kafka, Spark, and Airflow. The goal is to establish a data mesh architecture that ensures data observability, real-time data streaming, and improved data quality management. The solution should integrate seamlessly with our existing infrastructure, which includes Snowflake and BigQuery for data warehousing, and leverage MLOps practices to facilitate machine learning model deployment and monitoring. The project will significantly enhance our ability to conduct complex analyses and drive innovation in medical research by enabling more dynamic and responsive data handling.