Real-time Genomic Data Processing and Analytics Pipeline

As a rapidly growing biotechnology company, we are at the forefront of applying genetic engineering to develop innovative solutions for complex biological challenges. Our current data processing infrastructure is unable to keep up with the increasing volume and velocity of genomic data generated by high-throughput sequencing technologies. This bottleneck delays critical insights and diminishes our competitive edge. We are looking for a data engineering expert to architect and deploy a scalable, real-time data processing pipeline using technologies like Apache Kafka for event streaming, Databricks for real-time analytics, and Snowflake for data warehousing. The pipeline must support seamless integration with our existing MLOps framework to facilitate machine learning model deployment and monitoring, ensuring data observability and consistency. Key deliverables include setting up a data mesh architecture to decentralize data ownership and improve collaboration across teams, implementing data quality frameworks with Airflow for ETL orchestration, and integrating dbt for data transformation and documentation. This solution will drastically reduce time-to-insight, enhance data reliability, and provide a foundation for future data-driven innovations.