← Back to jobsJob ID col-cmv8iuog70ajioz01dz8u9cba

Cyber SecurityPosted 30+ days ago

Data Engineer - AWS/Databricks - Mid Level

Full-time
On-site
4+ years
Competitive
Posted 30+ days ago

Responsibilities Build and maintain scalable PySpark-based data pipelines in Databricks notebooks to support ingestion, transformation, and enrichment of structured and semi-structured data Design and implement Delta Lake tables optimized for ACID compliance, partition pruning, schema enforcement, and query performance across large datasets Develop ETL and ELT workflows that integrate multiple source systems into a c

Posted
30+ Days Ago
Source
Ethic Leaf recruiter
Sector
Information Technology & AI

No employer link yet — an Ethic Leaf recruiter forwards your application

Responsibilities

  • Build and maintain scalable PySpark-based data pipelines in Databricks notebooks to support ingestion, transformation, and enrichment of structured and semi-structured data
  • Design and implement Delta Lake tables optimized for ACID compliance, partition pruning, schema enforcement, and query performance across large datasets
  • Develop ETL and ELT workflows that integrate multiple source systems into a centralized, query-optimized data warehouse architecture
  • Leverage Spark SQL and DataFrame APIs to implement business rules, dimensional joins, and aggregation logic aligned to warehouse modeling best practices
  • Collaborate with data architects and engineers to implement cloud-native data solutions on AWS using S3, Glue, RDS, and IAM for secure, scalable storage and access control
  • Optimize pipeline performance through intelligent partitioning, caching, broadcast joins, and adaptive query tuning
  • Deploy and version data engineering assets using Git-integrated development workflows and automate deployment with CI/CD tools such as GitLab or Jenkins
  • Monitor pipeline health, job execution, and cluster utilization using native Databricks tools and AWS CloudWatch, identifying bottlenecks and optimizing cost-performance tradeoffs
  • Conduct technical discovery and mapping of legacy source systems, identifying required transformations and designing end-to-end data flows
  • Implement governance practices including metadata tagging, data quality validation, audit logging, and lineage tracking using platform-native features and custom logic
  • Support ad hoc data access requests, develop reusable data assets, and maintain shared notebooks that meet operational reporting and analytics needs across teams

Qualifications

  • 4+ years of experience in data engineering and Agile analytics
  • 4+ years of experience creating software for retrieving, parsing and processing structured and unstructured data
  • 2+ years of experience building scalable ETL and ELT workflows for reporting and analytics
  • 2+ years experience building enterprise data engineering solutions in the cloud, with preferred experience with cloud native technologies from AWS and Databricks
  • Experience with data quality, validation frameworks, and storage optimization strategies
  • BA or BS degree
  • Must be US Citizen with an ability to obtain and maintain US Suitability

Required skills

AWSDatabricksSparkDelta LakePySparkETL workflowsELT workflowsSpark SQLDataFrame APIsAWS S3AWS GlueAWS RDSAWS IAMGit

Full job description

Responsibilities

  • Build and maintain scalable PySpark-based data pipelines in Databricks notebooks to support ingestion, transformation, and enrichment of structured and semi-structured data
  • Design and implement Delta Lake tables optimized for ACID compliance, partition pruning, schema enforcement, and query performance across large datasets
  • Develop ETL and ELT workflows that integrate multiple source systems into a centralized, query-optimized data warehouse architecture
  • Leverage Spark SQL and DataFrame APIs to implement business rules, dimensional joins, and aggregation logic aligned to warehouse modeling best practices
  • Collaborate with data architects and engineers to implement cloud-native data solutions on AWS using S3, Glue, RDS, and IAM for secure, scalable storage and access control
  • Optimize pipeline performance through intelligent partitioning, caching, broadcast joins, and adaptive query tuning
  • Deploy and version data engineering assets using Git-integrated development workflows and automate deployment with CI/CD tools such as GitLab or Jenkins
  • Monitor pipeline health, job execution, and cluster utilization using native Databricks tools and AWS CloudWatch, identifying bottlenecks and optimizing cost-performance tradeoffs
  • Conduct technical discovery and mapping of legacy source systems, identifying required transformations and designing end-to-end data flows
  • Implement governance practices including metadata tagging, data quality validation, audit logging, and lineage tracking using platform-native features and custom logic
  • Support ad hoc data access requests, develop reusable data assets, and maintain shared notebooks that meet operational reporting and analytics needs across teams

Benefits

  • Personalized development plans
  • Mentorship
  • Up to $3,000 annually for training and certifications
  • Up to $3,000 for degree seeking programs
  • Competitive compensation
  • Comprehensive benefits
  • A strong focus on work-life balance
  • Work remotely (#LI-Remote)

Qualifications

  • 4+ years of experience in data engineering and Agile analytics
  • 4+ years of experience creating software for retrieving, parsing and processing structured and unstructured data
  • 2+ years of experience building scalable ETL and ELT workflows for reporting and analytics
  • 2+ years experience building enterprise data engineering solutions in the cloud, with preferred experience with cloud native technologies from AWS and Databricks
  • Experience with data quality, validation frameworks, and storage optimization strategies
  • BA or BS degree
  • Must be US Citizen with an ability to obtain and maintain US Suitability

Required skills

  • AWS, Databricks, Spark, Delta Lake, PySpark, ETL workflows, ELT workflows, Spark SQL, DataFrame APIs, AWS S3, AWS Glue, AWS RDS, AWS IAM, Git, GitLab, Jenkins, AWS CloudWatch, Data quality validation

Experience

  • 4+ years

Company

  • Acuity is an information technology company that offers cybersecurity and project management services.
  • Reston, Virginia, USA
  • http://myacuity.com

Job details

Employment type

Full-time

Seniority level

4+ years

Date posted

Oct 11, 2026

Work mode

On-site

Job source

Ethic Leaf recruiter

Company

Cyber Security

Information Technology & AI

Acuity is an information technology company that offers cybersecurity and project management services.

See all 5 jobs from Cyber Security →
  • Reston, Virginia, USA
  • Information Technology & AI
  • myacuity.com

Similar jobs

Matched on role, skills and location · live jobs only

See more Data Engineer jobs in United States

Meritore Technologies

1d ago

New

AWS Data Engineer

Full-timeOn-siteTorrance, CACompetitive

Meritore Technologies is hiring an AWS Data Engineer to build and operate scalable AWS data pipelines and cloud-based data lake and warehouse solutions.

Why similarSame roleAWS GlueAWS IAM

Capgemini

1d ago

New

Senior Data Engineer - Databricks

Full-timeOn-siteNew York, NY$84K – $156K/yr

Capgemini is seeking a Senior Databricks Data Engineer to design, build, and optimize scalable data platforms and pipelines in cloud-based environments.

Why similarSame roleAWSDatabricks

Dice

1d ago

New

Senior Data Engineer - Databricks

Full-timeOn-siteNew York, NY$84K – $156K/yr

Capgemini is seeking a Senior Databricks Data Engineer to design, build, and optimize scalable cloud-based data platforms and pipelines.

Why similarSame roleAWSDatabricks

Get new Data Engineer jobs in United States by email

Create job alert

Data Engineer - AWS/Databricks - Mid Level

Cyber Security · Competitive

Apply via Ethic Leaf