Lead AI Data Engineer
Lead AI Data Engineer
Location: Bengaluru, Karnataka, India
About the Role:
We’re looking for an experienced AI Data Engineer (4-8 years) to join our data team. In this role, you’ll build and maintain our data infrastructure on AWS, enabling analytics and AI teams to extract actionable insights. You’ll design and manage end-to-end data pipelines, ensuring high-quality, reliable, and real-time data, while also contributing to ML/GenAI workflows and model deployment pipelines.
What You'll Do:
Design and build scalable data pipelines/transformations using Spark / PySpark / Scala.
Manage and optimize Airflow DAGs for complex data workflows.
Clean, transform, and prepare data for analytics, AI, and ML use cases.
Use Python for automation, data processing, and internal tooling.
Work with AWS services (S3, Redshift, EMR, Glue, Athena) to maintain robust data infrastructure.
Collaborate with Analytics and AI teams to design pipelines for ML/GenAI projects.
Contribute to Node.js (TypeScript) backend development for data services.
Automate deployments using CI/CD pipelines (GitHub Actions).
Monitor, troubleshoot, and ensure data quality, consistency, and reliability across systems.
Build and maintain data warehouses/lakes and handle real-time streaming data using Kafka or similar technologies.
What You'll Need:
Bachelor’s or Master’s in Computer Science, Engineering, or related field.
4-8 years of hands-on experience in data engineering.
Strong expertise in Spark / Scala for large-scale data processing.
Proficient in Airflow for managing and optimizing complex DAGs.
Advanced Python skills for data manipulation, automation, and tool development.
Proven experience with AWS related cloud services (S3, Redshift, EMR, Glue, Athena, IAM, EC2).
Solid understanding of ETL/ELT, data preparation, and analytics workflows.
Familiar with Node.js and TypeScript for backend data services.
Experience with automated CI/CD (GitHub Actions).
Familiarity with CDC Tools like Debezium.
Strong SQL, knowledge of data warehousing and streaming (Kafka, Flink, Kinesis), and excellent communication skills.
Bonus Points:
Experience with data lake technologies (Delta Lake, Apache Iceberg).
Knowledge of ML/GenAI model deployment pipelines.
Familiarity with data governance, quality frameworks, and statistics.
Experience with infrastructure as code (Terraform).
Familiarity with containers (Docker, Kubernetes).
Experience with monitoring and logging tools (Prometheus, Grafana).
