Data Engineer II

Hagerty is hiring a Data Engineer II to build and maintain data pipelines, services, and cloud infrastructure for its Enterprise Data Hub. The role supports Hagerty’s Insurance business and automotive lifestyle brand, enabling analytics, data products, and real-time reporting.

Responsibilities

• Implement best practices around software development and big data engineering
• Develop and implement robust and scalable data pipelines using Python, SQL, parallel processing frameworks, and other AWS/Salesforce cloud solutions
• Develop and implement batch data pipelines using tools such as Apache Airflow, Snowflake, and numerous AWS products (EC2, Fargate, ECS, Lambda, and RDS)
• Develop streaming data integrations to support products across the Hagerty portfolio and support real-time reporting
• Develop Enterprise Data Hub platform infrastructure using Terraform infrastructure-as-code
• Develop and support Hagerty’s cloud-based data warehouse to enable analytics and product reporting
• Partner with internal and external stakeholders to collect requirements, recommend best practice solutions, and productionize new data ingestions/analytic workloads
• Develop solutions to catalog and manage metadata to support data governance and data democratization
• Partner with Data Quality Engineers to define and implement automated test cases and data reconciliation to validate ETL processes and data quality & integrity
• Mentor junior team members in software and big data engineering best practices
• Partner with Data Scientists to design, code, train, test, deploy and iterate machine learning algorithms and systems at scale

Preferred

• Associates degree, preferably in a technical/analytical field, or relevant work experience
• Additional 3+ years working in another role within an IT delivery team, such as a developer, engineer, data analyst, quality assurance analyst, ETL developer or DBA
• Developing infrastructure as code in a cloud-based environment (Terraform experience preferred)
• Experience cataloging and processing non-relational data
• Experience with open-source data processing technologies such as streaming services (Kafka / SQS), big data processing frameworks (MapReduce/Spark), big data file stores (EMRFS / HDFS)
• Experience working with evaluating different data containers based on workload needs (JSON, delimited files, Avro, Parquet)
• Experience with container-based development
• IT Delivery
• Software Development
• Data Analysis
• Quality Assurance
• ETL Development
• Database Administration
• Infrastructure as Code
• Cloud Computing
• Terraform
• Non-Relational Data
• Data Cataloging
• Data Processing
• Kafka
• Amazon SQS
• MapReduce
• Apache Spark
• EMRFS
• HDFS
• Data Container Evaluation
• JSON
• Delimited Files
• Avro
• Parquet
• Container-Based Development

Benefits

• Open to U.S. remote work; team members who reside within 20 miles of the Traverse City headquarters work in a hybrid schedule, working from the office three days per week.

Company

• Classic car insurance and marketplace for enthusiast drivers
• Traverse City, Michigan, USA
• http://www.hagerty.com/
• Founded in 1983
• 1001-5000 employees

Qualifications

• You have strong problem-solving abilities and attention to detail
• You can authentically and effectively communicate (written and verbally) with various stakeholders
• You create and share technical artifacts and documentation to support development and maintenance of data products
• You have experience in successful delivery of data products as productionizable software solutions
• You ensure quality through rigorous code development, testing, automation, and other software engineering best practices
• You have experience developing solutions using Python and cloud-based infrastructure (AWS, Azure, or GCP)
• You demonstrate experience in imperative (e.g., Apache Airflow / NiFi) or declarative (e.g., Informatica/Talend/Pentaho) ETL design, implementation, and maintenance
• You have functional knowledge of relational databases and query authoring (SQL)
• Associates degree, preferably in a technical/analytical field, or relevant work experience
• Additional 3+ years working in another role within an IT delivery team, such as a developer, engineer, data analyst, quality assurance analyst, ETL developer or DBA
• Developing infrastructure as code in a cloud-based environment (Terraform experience preferred)
• Experience cataloging and processing non-relational data
• Experience with open-source data processing technologies such as streaming services (Kafka / SQS), big data processing frameworks (MapReduce/Spark), big data file stores (EMRFS / HDFS)
• Experience working with evaluating different data containers based on workload needs (JSON, delimited files, Avro, Parquet)
• Experience with container-based development

Required skills

• Apache Airflow, Cloud Infrastructure, ETL Design and Implementation, Informatica, Pentaho, Python, SQL, Talend

Preferred skills

• Experience cataloging and processing non-relational data, Experience with container-based development

Experience

• 3+ years