Senior Data Engineer

Description:

RESPONSIBILITIES:

• Help build large-scale distributed automated data processing systems and data lakes, while optimizing for both computational and storage efficiency on AWS Databricks.
• Create data pipelines, infrastructure, and overall workflow orchestration to pull data from a diverse set of data sources into the Kharon Data Lake.
• Build systems to track, monitor and validate the quality of data coming into or going out of Kharon Data Lake.
• Collaborate with engineers, architects, and data scientists to implement scalable solutions to solve complex problems.
• Interact and develop with internal and external APIs.
• Work with product and business teams to define novel and critical metrics for the business.
• Follow good engineering practices like documentation, diagrams, and unit/validation tests
• Take full responsibilities for the development, deployment, adoption monitoring and maintenance of the data systems and services you build.

QUALIFICATIONS:

• 5+ years of experience, not including internships as a software engineer or a data engineer and a strong passion to learn.
• BS/MS in Computer Science or equivalent experience in related fields.
• Experience in Python, Pandas, PySpark, and Notebooks.
• SQL knowledge and experience working with relational databases including schema design, access patterns, query performance optimization, etc.
• Experience with data pipeline technologies like AWS Glue, Airflow, Kafka, or other cloud based equivalence.
• Experience with ETL and data warehousing like Databricks, Snowflake, or equivalent.
• Container-based deployment experience using Docker and Kubernetes.
• Strong verbal and written communication skills.