Senior Data Engineer
About the Company
At Delta, we are reimagining and rebuilding the financial system. Join our team to make a positive impact on the future of finance.
🎯 Mission Driven: Re-imagine and rebuild the future of finance.
💡 Most innovative cryptocurrency derivatives exchange. With a daily traded volume of ~$ 3.5 billion, and increasing. Delta is bigger than all the Indian crypto exchanges combined.
📈 Offer the widest range of derivative products and have been serving traders all over the globe since 2018 and growing fast.
💪🏻 The founding team is comprised of IIT and ISB graduates. Business co-founders have previously worked with Citibank, UBS and GIC; and our tech co-founder is a serial entrepreneur who previously co-founded TinyOwl and Housing.com.
💰 Funded by top crypto funds (Sino Global Capital, CoinFund, Gumi Cryptos) and crypto projects (Aave and Kyber Network).
Role Summary:
Support our analytics team by owning the full ETL lifecycle—from master data to analytics-ready datasets. You will build and maintain daily batch pipelines that process 1–10 million master-data rows per run (and scale up to tens or hundreds of millions of rows), all within sub-hourly SLAs. Extract from OLTP and time-series sources, apply SQL/stored-procedure logic or Python transformations, then load into partitioned, indexed analytics tables. Reads run exclusively on read-only replicas to guarantee zero impact on the production master DB. You’ll also implement monitoring, alerting, retries, and robust error handling to ensure near-real-time dashboard refreshes.
Requirements
Required Skills & Experience:
* 4+ years in data engineering or analytics roles, building daily batch ETL pipelines at 1–10 M rows/run scale (and up to 100 M+).
* Expert SQL skills, including stored procedures and query optimisation on Postgres, Mysql, or similar RDBMS.
* Proficient in Python for data transformation (pandas, NumPy, SQLAlchemy, psycopg2).
* Hands-on with CDC/incremental load patterns and batch schedulers (Airflow, cron).
* Deep understanding of replicas, partitioning, and indexing strategies.
* Strong computer-science fundamentals and deep knowledge of database internals—including storage engines, indexing mechanisms, query execution plans and optimisers for MySQL and time-series DBs like TimescaleDB.
* Experience setting up monitoring and alerting (Prometheus, Grafana, etc.).
Key Responsibilities:
1. Nightly Batch Jobs: Schedule and execute ETL runs.
2. In-Database Transformations: Write optimised SQL and stored procedures.
3. Python Orchestration: Develop Python scripts for more complex analytics transformations.
4. Data Loading & Modelling: Load cleansed data into partitioned, indexed analytics schemas designed for fast querying.
5. Performance SLAs: Deliver end-to-end sub-hourly runtimes.
6. Monitoring & Resilience:Implement pipeline health checks, metrics, alerting, automatic retries, and robust error handling.
7. Stakeholder Collaboration: Work closely with analysts to validate data quality and ensure timely delivery of analytics-ready datasets.