Big Data Engineer

BrothersTech is seeking a Big Data Engineer to design, develop, and maintain scalable data pipelines and distributed data platforms. The role builds reliable data solutions for analytics, reporting, and data-driven decision-making in collaboration with data scientists, analysts, and engineering teams.

Responsibilities

Design, develop, and maintain scalable data pipelines for processing large volumes of structured and unstructured data

Build batch and real-time data processing solutions using Apache Spark, PySpark, and related technologies

Develop data ingestion, transformation, and integration workflows from multiple data sources

Work with distributed computing frameworks such as Hadoop, Hive, and HDFS

Design and optimize data storage solutions, including data lakes and lakehouse architectures

Implement streaming data pipelines using Apache Kafka or similar technologies

Develop efficient SQL queries and data transformations for analytics and reporting

Integrate data from databases, APIs, enterprise applications, and cloud storage systems

Monitor data pipeline performance, troubleshoot failures, and improve processing efficiency

Implement data quality checks, validation, error handling, and monitoring

Collaborate with data scientists and machine learning engineers to support advanced analytics and AI workloads

Apply data security, governance, access control, and compliance standards

Maintain technical documentation and follow version control and CI/CD best practices

Preferred

Experience with Hadoop, HDFS, Hive, or HBase

Familiarity with Apache Kafka, Spark Structured Streaming, or other streaming technologies

Experience with cloud platforms such as AWS, Microsoft Azure, or Google Cloud

Knowledge of cloud data services such as Amazon EMR, AWS Glue, Azure Databricks, or Google Cloud Dataproc

Familiarity with Databricks, Delta Lake, or Apache Iceberg

Experience with workflow orchestration tools such as Apache Airflow

Knowledge of Docker, Kubernetes, CI/CD, and infrastructure automation

Understanding of data governance, metadata management, and data security practices

Hadoop

HDFS

Hive

HBase

Apache Kafka

Spark Structured Streaming

Streaming Technologies

AWS

Microsoft Azure

Google Cloud

Amazon EMR

AWS Glue

Azure Databricks

Google Cloud Dataproc

Databricks

Delta Lake

Apache Iceberg

Workflow Orchestration

Apache Airflow

Docker

Kubernetes

CI/CD

Infrastructure Automation

Data Governance

Metadata Management

Data Security

Benefits

Remote – United States

Company

Intelligent technology solutions for business operations and growth

Dover, Delaware, USA

https://www.brotherstechinc.com/

Founded in 2017

51-200 employees

Qualifications

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent practical experience

3–6 years of experience in big data engineering, data engineering, or distributed data processing

Strong programming skills in Python, Scala, or Java

Hands-on experience with Apache Spark and PySpark

Strong SQL skills and experience working with large datasets

Understanding of distributed computing, parallel processing, and data partitioning

Experience building ETL/ELT pipelines and data integration workflows

Knowledge of data lakes, data warehouses, and common data storage formats such as Parquet, Avro, and ORC

Familiarity with relational and NoSQL databases

Experience with Git, debugging, testing, and production support

Experience with Hadoop, HDFS, Hive, or HBase

Familiarity with Apache Kafka, Spark Structured Streaming, or other streaming technologies

Experience with cloud platforms such as AWS, Microsoft Azure, or Google Cloud

Knowledge of cloud data services such as Amazon EMR, AWS Glue, Azure Databricks, or Google Cloud Dataproc

Familiarity with Databricks, Delta Lake, or Apache Iceberg

Experience with workflow orchestration tools such as Apache Airflow

Knowledge of Docker, Kubernetes, CI/CD, and infrastructure automation

Understanding of data governance, metadata management, and data security practices

3–6 years of experience in big data engineering, data engineering, or distributed data processing.

Required skills

Python, Scala, Java, Apache Spark, PySpark, SQL, ETL/ELT Pipelines, Hadoop, Apache Kafka, AWS, Microsoft Azure, Google Cloud

Preferred skills

Experience with Hadoop, HDFS, Hive, or HBase, Familiarity with Databricks, Delta Lake, or Apache Iceberg

Experience

3+ years