Senior Databricks AI Engineer

Role Overview

We are seeking a Senior Databricks AI Engineer to design, build, and operate AI platforms natively on the Databricks Lakehouse. This role is Databricks-first: ownership of large-scale data, feature, and AI pipelines using Spark, Delta Lake, MLflow, Feature Store, and Unity Catalog. The focus is on engineering reliable AI systems at scale, not standalone model experimentation.

Key Responsibilities (Databricks-Centric)

• Architect and implement end-to-end AI pipelines on Databricks Lakehouse
• Build scalable data ingestion, transformation, and feature pipelines using Spark
• Design AI-ready Delta Lake architectures (Bronze/Silver/Gold)
• Implement AI lifecycle management using MLflow (tracking, registry, serving)
• Develop reusable Feature Store assets to support enterprise AI use cases
• Operationalize AI workloads using Databricks Workflows and Jobs
• Enable batch and real-time inference using Databricks-native serving
• Enforce data governance, lineage, and access control via Unity Catalog
• Optimize Spark clusters for performance, reliability, and cost
• Establish Databricks AI engineering standards and best practices
• Partner with data science teams to productionize AI solutions

Required Technical Skills (Databricks-First)

Databricks Platform

• Deep hands-on experience with Databricks Lakehouse
• Advanced expertise in Apache Spark (PySpark & Spark SQL)
• Strong command of Delta Lake (ACID, OPTIMIZE, Z-ORDER, VACUUM)
• Production experience with MLflow on Databricks
• Hands-on with Databricks Feature Store
• Strong experience with Unity Catalog for governance and security
• Databricks Jobs, Workflows, Repos, and SQL Warehouses

AI Engineering (Platform-Oriented)

• Feature engineering at scale using Spark
• Model packaging, versioning, and promotion using MLflow
• AI pipeline orchestration and automation
• Batch and near-real-time inference pipelines
• Model monitoring and retraining workflows on Databricks

Nice-to-Have (Still Databricks-Aligned)

• Databricks AutoML
• Databricks Model Serving APIs
• Photon performance tuning
• Multi-workspace Databricks deployments
• AI enablement in regulated environments (banking / finance)

Experience & Qualifications

• 10+ years in data engineering, AI engineering, or platform engineering
• 5+ years building enterprise solutions on Databricks
• Proven experience delivering production-grade AI platforms
• Strong understanding of distributed systems and Spark internals
• Ability to lead architecture decisions and mentor engineers

What Success Looks Like

• AI workloads run reliably, securely, and cost-effectively on Databricks
• Data scientists deploy models without friction using platform tooling
• Feature reuse and governance are enforced through Databricks-native services
• AI delivery cycles are shortened through automation and standardization

Preferred Certifications

• Databricks Certified AI Engineer
• Databricks Certified Data Engineer Professional
• Databricks Certified Machine Learning Professional