Data Engineer

• Oversee day-to-day operations of big data platform to ensure high availability, reliability and performance. 

• Proactively monitor big data platform services, components and clusters to identify potential issues. Take corrective actions as needed to maintain platform health 

• Manage configuration, upgrades, and patching of big data platform, ensuring all services are up to date 

• Work with the Authority’s technical teams to ensure smooth deployment and adoption of new solution to support data ingestions, process and workflows 

• Maintain clear and detailed documentation of platform configuration, troubleshooting steps and incident resolution. 

• Continuously monitor for and address platform security vulnerabilities. Implement patching strategies to resolve identified vulnerabilities and maintain a secure environment. 

• Develop automation script to streamline administrative tasks, platform health and ensure operational consistency. 

• Ensure the smooth operations and service level of IT solutions. 

• Support production issues 

Requirements

• Hands-on experience, knowledge and troubleshooting of Cloudera Data Platform such as HDFS, YARN, HIVE, Spark, Impala, Ranger, operating systems, security and network. 

• Hands on experience with monitoring tools like Cloudera Manager, Zabbix, Grafana, Splunk, SyslogNG 

• Familiarity with middleware applications i.e. Informatica, Denodo and scripting languages like bash, python, or shell scripting for automation 

• Experience with cloud technology i.e. AWS, Azure is a plus 

• Ability to troubleshoot complex issues ranging from system resource to application stack traces. 

• Track record in implementing systems with high availability, high performance, high security hosted at various data centres or hybrid cloud environments will be an added advantage. 

• Cloudera Certified Administrator or similar certification are a plus. 

• Excellent communication skill to work with cross-functional teams 

• Ability to handle high-pressure situations and manage critical incidents