Data Engineer

Description:

Primary Responsibilities
• Establish technical designs to meet requirements aligned with architectural and data standards
• Optimize ETL/data pipelines balancing performance, functionality, and operational requirements
• Fine-tune and optimize queries using Snowflake platform and database techniques
• Manage data ingestion, transformation, processing, and orchestration of pipelines
• Develop and maintain API, ETL Pipeline, and CI/CD integration processes
• Assist with technical solution discovery to ensure feasibility
• Set up and manage CI/CD pipelines and develop automated tests
• Conduct peer reviews for quality, consistency, and rigor for production-level solutions
• Promote best practices for code management, automated testing, and deployments
• Create design and development documentation based on standards
• Actively contribute to Data Engineering community and define leading practices and frameworks
Qualifications
Required
• Bachelor's degree in computer science, engineering, or similar quantitative field
• 5+ years of relevant experience developing backend, integration, data pipelining, and infrastructure
• Strong expertise in Python, PySpark, and Snowpark
• Proven experience with Snowflake and AWS cloud platforms
• Experience with Informatica/IICS for data integration
• Expertise in database optimization and performance improvement
• Experience with data warehousing and writing efficient SQL queries
• Understanding of data structures and algorithms
Preferred
• Knowledge of DevOps best practices and associated tools:
o Containers and containerization technologies (Kubernetes, Argo, Red Hat OpenShift)
o Infrastructure as code (Terraform)
o CI/CD Pipelines (JFrog Artifactory)
o Scripting and automation (Python, GitHub, GitHub actions)