← Back to jobsJob ID col-cmv8ddbmp03vloz01w0e5vpnh

SwoonPosted 30+ days ago

Data Engineer

Full-time
Hybrid
Mid level
Competitive
Posted 30+ days ago

Swoon is seeking a Data Engineer to design, develop, and maintain scalable data pipelines and distributed data processing solutions. The role focuses on AWS-based, event-driven data architectures, including pipeline orchestration, data quality, schema management, troubleshooting, and collaboration on reliable data platform improvements.

Posted
30+ Days Ago
Source
Ethic Leaf recruiter
Sector
Information Technology & AI

No employer link yet — an Ethic Leaf recruiter forwards your application

Responsibilities

  • Design, develop, and maintain scalable data pipelines using Java, Scala, and Apache Spark
  • Build and support multi-stage AWS data pipelines using S3, AWS Glue, Step Functions, and Lambda
  • Develop and maintain event-driven data architectures using Amazon EventBridge and S3 event notifications
  • Design pipeline workflows that appropriately handle event-triggered reruns, retries, failures, and recovery scenarios
  • Configure and manage AWS Step Functions state machines, including scheduling, orchestration, and synchronization of downstream data processing jobs
  • Develop and optimize AWS Glue jobs, including configuration, resource allocation, and scaling for large-volume data workloads
  • Work with modern data formats and technologies such as Apache Iceberg, including troubleshooting platform and version-specific limitations
  • Implement data quality and validation components to ensure data security, integrity, and consistency across pipelines
  • Identify and address schema drift, data changes, and backfill requirements within data pipelines
  • Troubleshoot pipeline failures, performance issues, and data inconsistencies across distributed AWS environments
  • Collaborate with data engineers, architects, and other technical stakeholders to develop reliable and maintainable data solutions
  • Contribute to technical design decisions, documentation, testing, and continuous improvement of the data platform

Qualifications

  • Strong hands-on experience with Java and/or Scala
  • Strong experience with Apache Spark and distributed data processing
  • Hands-on experience building AWS data pipelines using:
  • Experience with event-driven architectures , particularly EventBridge rules and S3 event notifications
  • Understanding of Step Functions state machines, scheduling, orchestration, and downstream job synchronization
  • Experience configuring and scaling AWS Glue jobs
  • Strong understanding of data quality, schema validation, schema drift, and backfill strategies
  • Experience troubleshooting and supporting production data pipelines

Preferred

  • Experience with Apache Iceberg , including Glue/Iceberg integration and troubleshooting
  • Experience designing resilient pipelines with rerun, retry, and failure-recovery mechanisms
  • Experience implementing data validation or security-focused data quality components
  • Experience working with large-scale cloud data platforms and distributed processing environments
  • Strong problem-solving skills and ability to independently research and resolve technical limitations

Required skills

JavaScalaApache SparkAmazon S3AWS GlueAWS Step FunctionsAWS LambdaAmazon EventBridgeEvent-Driven ArchitectureApache IcebergData QualitySchema ValidationSchema DriftData Backfill

Full job description

Swoon is seeking a Data Engineer to design, develop, and maintain scalable data pipelines and distributed data processing solutions. The role focuses on AWS-based, event-driven data architectures, including pipeline orchestration, data quality, schema management, troubleshooting, and collaboration on reliable data platform improvements.

Responsibilities

  • Design, develop, and maintain scalable data pipelines using Java, Scala, and Apache Spark
  • Build and support multi-stage AWS data pipelines using S3, AWS Glue, Step Functions, and Lambda
  • Develop and maintain event-driven data architectures using Amazon EventBridge and S3 event notifications
  • Design pipeline workflows that appropriately handle event-triggered reruns, retries, failures, and recovery scenarios
  • Configure and manage AWS Step Functions state machines, including scheduling, orchestration, and synchronization of downstream data processing jobs
  • Develop and optimize AWS Glue jobs, including configuration, resource allocation, and scaling for large-volume data workloads
  • Work with modern data formats and technologies such as Apache Iceberg, including troubleshooting platform and version-specific limitations
  • Implement data quality and validation components to ensure data security, integrity, and consistency across pipelines
  • Identify and address schema drift, data changes, and backfill requirements within data pipelines
  • Troubleshoot pipeline failures, performance issues, and data inconsistencies across distributed AWS environments
  • Collaborate with data engineers, architects, and other technical stakeholders to develop reliable and maintainable data solutions
  • Contribute to technical design decisions, documentation, testing, and continuous improvement of the data platform

Preferred

  • Experience with Apache Iceberg , including Glue/Iceberg integration and troubleshooting
  • Experience designing resilient pipelines with rerun, retry, and failure-recovery mechanisms
  • Experience implementing data validation or security-focused data quality components
  • Experience working with large-scale cloud data platforms and distributed processing environments
  • Strong problem-solving skills and ability to independently research and resolve technical limitations
  • Apache Iceberg
  • Glue/Iceberg Integration
  • Technical Troubleshooting
  • Resilient Pipeline Design
  • Pipeline Reruns
  • Retry Mechanisms
  • Failure Recovery
  • Data Validation
  • Data Security
  • Cloud Data Platforms
  • Distributed Processing
  • Problem-Solving
  • Independent Research
  • Self-Direction

Company

  • Tailored staffing solutions connecting talent with companies
  • Chicago, Illinois, USA
  • http://swoonstaffing.com/
  • Founded in 2009
  • 501-1000 employees

Qualifications

  • Strong hands-on experience with Java and/or Scala
  • Strong experience with Apache Spark and distributed data processing
  • Hands-on experience building AWS data pipelines using:
  • Amazon S3
  • AWS Glue
  • AWS Step Functions
  • AWS Lambda
  • Experience with event-driven architectures , particularly EventBridge rules and S3 event notifications
  • Understanding of Step Functions state machines, scheduling, orchestration, and downstream job synchronization
  • Experience configuring and scaling AWS Glue jobs
  • Strong understanding of data quality, schema validation, schema drift, and backfill strategies
  • Experience troubleshooting and supporting production data pipelines
  • Experience with Apache Iceberg , including Glue/Iceberg integration and troubleshooting
  • Experience designing resilient pipelines with rerun, retry, and failure-recovery mechanisms
  • Experience implementing data validation or security-focused data quality components
  • Experience working with large-scale cloud data platforms and distributed processing environments
  • Strong problem-solving skills and ability to independently research and resolve technical limitations

Required skills

  • Java, Scala, Apache Spark, Amazon S3, AWS Glue, AWS Step Functions, AWS Lambda, Amazon EventBridge, Event-Driven Architecture, Apache Iceberg, Data Quality, Schema Validation, Schema Drift, Data Backfill, Distributed Data Processing

Experience

  • Mid level

Job details

Employment type

Full-time

Seniority level

Mid level

Date posted

Oct 13, 2026

Work mode

Hybrid

Job source

Ethic Leaf recruiter

Company

Swoon

Information Technology & AI

Tailored staffing solutions connecting talent with companies

See all 2 jobs from Swoon →
  • Chicago, Illinois, USA
  • Information Technology & AI
  • 501-1000 employees
  • Founded 2009
  • swoonstaffing.com

Similar jobs

Matched on role, skills and location · live jobs only

See more Data Engineer jobs in Wilmington

EXL

1d ago

New

Senior Data Engineer

Full-timeRemoteUnited StatesCompetitive

EXL is seeking a seasoned Data Engineer to join their Sports Analytics & Engineering Practice. This role is pivotal in shaping and implementing a cloud-native data ecosystem, where you will architect and build scalable data infrastructure that transforms raw…

Why similarSame roleAWS GlueRemote

Airbnb

1d ago

New

Data Engineer Tech Foundations

Full-timeRemoteUnited States$162K – $186K/yr

Airbnb is hiring a Data Engineer to build and operate reliable, reusable datasets and pipelines that support AI usage and cost measurement, Unit Economics, enterprise application data, and critical financial metrics.

Why similarSame roleApache SparkRemote

Dice

1d ago

New

Senior AWS Data Engineer

TemporaryHybridIndianapolis, INCompetitive

Alchemy is recruiting for an unnamed client seeking a Senior AWS Data Engineer with healthcare data project experience.

Why similarSame roleAWS GlueAWS Lambda

Get new Data Engineer jobs in Wilmington by email

Create job alert

Data Engineer

Swoon · Competitive

Apply via Ethic Leaf