Real-Time Data Pipeline for Public Health Insights

Our startup is at the forefront of public health innovation, aiming to leverage data engineering to address pressing public health challenges. We are seeking an experienced data engineer to build a real-time data pipeline capable of handling vast amounts of health-related data. The project will employ technologies such as Apache Kafka for event streaming, Spark for real-time processing, and Airflow for orchestrating data workflows. This pipeline will allow us to analyze trends rapidly and respond promptly to emerging health threats, thereby improving public health outcomes. The solution should integrate with our existing data platforms, including Snowflake and BigQuery, to ensure seamless data storage and retrieval. Additionally, we require robust data observability features to maintain data quality and governance. The ideal candidate will have a strong background in building scalable data architectures and possess a keen understanding of public health data dynamics.