Big Data Engineer

TP-Link Systems Inc. is a global provider of networking devices and smart home products. The Big Data Engineer will build, operate, and maintain production ETL pipelines, data models, and data quality processes using technologies including Spark, Airflow, SQL, Python, and AWS. The role also partners with analysts and business stakeholders to deliver efficient, reliable datasets.

Responsibilities

• Build, run, and own ETL pipelines on EMR (Spark) orchestrated in Airflow — including their monitoring, alerting, and recovery
• Write SQL and Python for data ingestion, transformation, and the datasets that analysts and dashboards depend on
• Own data quality for what you build — run the checks before you ship and add new ones where they're missing; confirm the results are correct, not only that the job completed
• Build and extend the data models for your area: fact and dimension tables following the team's layering conventions
• Keep your jobs efficient — watch runtime and cost, and raise slow or expensive jobs rather than living with them
• Use and extend the team's shared patterns and templates, and turn work you find yourself repeating into something automated or reusable
• Work with analysts and business stakeholders to turn requests into datasets that actually get used

Preferred

• Data ingestion or CDC tooling: DataX, Sqoop, Debezium, Fivetran, or similar
• An OLAP / MPP engine: StarRocks, Doris, ClickHouse, Redshift, or similar
• Spark performance work: partitioning, shuffle, skew, and memory tuning
• AWS cost optimization: EMR instance sizing and Spot strategy, S3 lifecycle policies
• Lakehouse formats (Iceberg, Hudi, Delta), streaming (Kafka, Flink), dbt, or a data quality framework
• QuickSight or another BI tool: dataset and permission design
• Self-directed learning: a project, an open-source contribution, or a tool you picked up on your own and put to real use
• Data Ingestion
• Change Data Capture
• DataX
• Sqoop
• Debezium
• Fivetran
• OLAP Engines
• MPP Engines
• StarRocks
• Doris
• ClickHouse
• Amazon Redshift
• Spark Performance Tuning
• Data Partitioning
• Shuffle Tuning
• Data Skew Handling
• Memory Tuning
• AWS Cost Optimization
• EMR Instance Sizing
• AWS Spot Strategy
• S3 Lifecycle Policies
• Lakehouse Formats
• Apache Iceberg
• Apache Hudi
• Delta Lake
• Stream Processing
• Apache Kafka
• Apache Flink
• dbt
• Data Quality Frameworks
• Amazon QuickSight
• BI Tools
• Dataset Design
• Permission Design
• Open-Source Contribution
• Project Development
• Self-Directed Learning
• Initiative

Benefits

• Free snacks and drinks
• Fully paid medical, dental, and vision insurance (partial coverage for dependents)
• Contributions to 401K funds
• Bi-annual reviews, and annual pay increases
• Health and wellness benefits, including free gym membership
• Quarterly team-building events

Company

• Wi-Fi networking devices and smart home products
• Irvine, California, USA
• https://www.tp-link.com/
• Founded in 1996
• 51-200 employees

Qualifications

• 2–4 years of hands-on data development in a production environment — pipelines that run on a schedule with real downstream consumers, that you were responsible for including when they broke
• Strong SQL: window functions, complex multi-table joins, incremental loads, and the ability to work out why a query is slow
• Python for production ETL and tooling (PySpark, pandas, boto3) — code that runs on a schedule, not only notebooks
• Hands-on experience with Spark on a cloud platform: AWS EMR, Databricks, Glue, or equivalent
• Production experience with a scheduler, Airflow preferred: DAG design, dependencies, retries, and reruns that are safe to repeat
• Working understanding of dimensional modeling: fact and dimension tables, star schema, and warehouse layering
• Comfortable working on AWS (S3 with Parquet / ORC), Linux, and Git — and comfortable picking up new tools as the platform evolves
• Effective use of AI to solve data problems — using it to move faster on SQL, debugging, and unfamiliar schemas, with the judgment to catch output that looks right but isn't
• Bachelor's degree in Computer Science, Information Systems, or a related field, or equivalent practical experience
• Data ingestion or CDC tooling: DataX, Sqoop, Debezium, Fivetran, or similar
• An OLAP / MPP engine: StarRocks, Doris, ClickHouse, Redshift, or similar
• Spark performance work: partitioning, shuffle, skew, and memory tuning
• AWS cost optimization: EMR instance sizing and Spot strategy, S3 lifecycle policies
• Lakehouse formats (Iceberg, Hudi, Delta), streaming (Kafka, Flink), dbt, or a data quality framework
• QuickSight or another BI tool: dataset and permission design
• Self-directed learning: a project, an open-source contribution, or a tool you picked up on your own and put to real use
• 2–4 years of hands-on data development in a production environment — pipelines that run on a schedule with real downstream consumers, that you were responsible for including when they broke.

Required skills

• AWS EMR, AWS S3, Apache Airflow, Apache Spark, Dimensional Modeling, Git, Linux, ORC, Parquet, Python, SQL

Preferred skills

• QuickSight or another BI tool: dataset and permission design

Experience

• 2-4 years