Real-Time Public Health Data Infrastructure Development

In the dynamic field of public health, timely data is critical for effective intervention and policy development. Our company is scaling rapidly, and we aim to develop a robust data infrastructure that can handle real-time data ingestion, processing, and analysis. The ideal candidate will build an end-to-end data pipeline using Apache Kafka for real-time event streaming and Apache Spark for batch processing. This infrastructure must be capable of integrating with cloud-based data warehouses like Snowflake or BigQuery for scalable storage and analytics. We plan to implement a data mesh architecture to promote decentralized data ownership and enable domain-specific data insights. Additionally, incorporating MLOps practices will facilitate the deployment of machine learning models on the fly, enhancing our predictive analysis capabilities. Data observability will ensure the reliability and quality of our data pipeline, allowing us to act swiftly on accurate information. This project will significantly enhance our ability to address public health challenges efficiently and effectively.