Data Engineer
The primary focus of Data Engineer will be on choosing optimal solutions to use for collecting, storing, processing, and analyzing of huge sets of data, then maintaining, implementing, and monitoring them. Data Engineer will also be responsible for integrating them with the architecture used across the company.
What you will be doing :
- Designing, constructing, maintaining, and scaling our data infrastructure. Our data infrastructure is data lake for all industry under Kompas Gramedia.
- Designing and constructingETL process as data pipeline for each data source inside Kompas Gramedia
- As a person who solve business case using data technology, such as Kafka, Spark, DataFlow, BigQuery, Airflow, etc.
- Ensure systems meet business requirements and industry practices
- Employ a variety of languages and tools (e.g. scripting languages) to marry systems together
- Collaborate with other Data Team members on project goals
Qualifications
- Passion in big data, software engineering, and systems.
- Ability to design and architect distributed data systems
- Knowledge of data warehousing and data pipeline concepts, such as ETL, data integration, data streaming
- In-depth knowledge of SQL and other database solutions (No SQL), such as MongoDB, Elastic Search, and so on
- Knowledgeable in Linux-based system administrating
- Practical experience in planning, sizing & migrating large enterprise workloads from on-premises to cloud platform.
- Working knowledge of big data components like Hadoop v2, MapReduce, HDFS, Hive, MapReduce, Spark, Apache Beam, Storm is preferred
- Experience with various messaging systems, such as Kafka, RabbitMQ, NSQ is preferred
- Experience with Big Data ML toolkits, such as Mahout, SparkML, or H2O is preferred
- Good knowledge on big data platform such as Cloudera, MapR, Hortonworks, or ELK (Elastic Search – Log stash – Kibana) is an added advantage
- Experience in DevOps tools such as Ansible, Docker, Kubernetes, Chef, Vagrant is a plus
- Knowledge of other open source database technologies is an added advantage.
- A bachelor degree in technical majors is preferred (e.g. Computer Science, Information Technology)
Additional Information
Require at least 2 years of work experience