Real-time Data Pipeline for Nano-Material Prototyping

We are a nascent startup in the nanotechnology industry focused on developing innovative nano-materials for environmental applications. Our R&D team is experiencing bottlenecks due to the lack of efficient data handling systems, which hampers our ability to process and analyze experimental data in real time. To address this, we require a data engineer to build a robust data pipeline that integrates with our existing systems and supports real-time analytics. Key tasks include setting up an event streaming architecture using Apache Kafka, implementing data processing with Spark, orchestrating workflows with Airflow, and ensuring efficient data storage and querying via Snowflake or BigQuery. The engineer will also incorporate data observability to monitor pipeline health and performance. This project is critical to enhancing our R&D capabilities, reducing time-to-market for our products, and maintaining a competitive edge.