Data Engineer (Temporal & Apache Kafka required)

Infinitive is a data and AI consultancy that helps clients modernize, monetize, and operationalize their data. The company is seeking an experienced Data Engineer to design, build, and scale event-driven data platforms using Apache Kafka, Temporal, Python, Apache Spark, and cloud data warehouses. The role focuses on distributed streaming, durable workflow orchestration, data schema governance, pipeline development, reliability, and cross-functional collaboration.

Responsibilities

Architect, deploy, and maintain high-volume distributed data streams using Apache Kafka (producers, consumers, Kafka Connect, Schema Registry)

Establish and enforce schema design standards, versioning strategies, and automated schema validation (e.g., Avro, Protobuf, JSON Schema) to maintain strict data contracts across microservices, streaming consumers, and Lakehouse storage

Design and implement durable execution workflows using Temporal to coordinate long-running distributed pipelines, compensate transactions (Saga pattern), and manage cross-system ETL tasks

Build end-to-end batch and near-real-time pipelines using Python, SQL, and Apache Spark / PySpark

Design and optimize analytical data models (dimensional/star schema) in modern cloud data warehouses / Lakehouse (e.g., Snowflake, BigQuery, Databricks, Redshift)

Implement automated testing, continuous schema validation, data drift detection, and observability across streaming and batch workflows

Partner with software engineers, machine learning engineers, and analysts to define standard schema definitions, data contracts, and production-grade CI/CD release patterns

Preferred

Go or Java is a plus

Company

Data and AI consultancy for enterprise digital transformation

Ashburn, Virginia, USA

http://www.infinitive.com/

Founded in 2003

51-200 employees

Qualifications

4+ years of professional experience in data engineering, backend distributed systems, or software engineering

Hands-on experience with **Temporal (or Cadence):** Proven understanding of durable workflows, activities, retries, signals, queries, and long-running distributed task orchestration

Deep expertise with Apache Kafka: Practical experience with message partitioning, consumer groups, offset management, and topic design

Demonstrated proficiency with schema definition frameworks (Apache Avro, Protocol Buffers/gRPC, or JSON Schema)

Practical experience managing schema evolution, compatibility modes (backward/forward/full), and schema registries (e.g., Confluent Schema Registry, AWS Glue Schema Registry)

Experience enforcing data validation rules, contract testing, and data quality checks (e.g., Great Expectations, Pandera, Pydantic, dbt tests)

Strong programming proficiency in Python (Go or Java is a plus) with clean code, design patterns, and unit/integration testing standards

Distributed computing experience: Hands-on development with Apache Spark (PySpark/Spark SQL) processing large-scale datasets

Advanced SQL & Data Modeling: Strong experience with relational databases, dimensional data modeling, and query performance tuning

Go or Java is a plus

4+ years of professional experience in data engineering, backend distributed systems, or software engineering.

Required skills

Apache Kafka, Data Schema Design and Validation, Apache Avro, Protocol Buffers/gRPC, JSON Schema, Schema Registries, Python, Apache Spark/PySpark, SQL, Data Modeling, Relational Databases

Preferred skills

Go or Java is a plus, Ashburn, Virginia, USA, http://www.infinitive.com, Growth Stage, Leadership Team, Denis McFarlane, Founder & Executive Chairman, Company data provided by, AI Tools, Customize Your Resume, Maximize your interview chances, Build Cover Letter, Make your application stand out, Analyze How Well You Fit, Understand your strength & weakness

Experience

4+ years