Staff Software Engineer, Data Infrastructure

Your Impact on our Mission:

As a Staff Software Engineer (Data Infrastructure), you'll lead the design and development of the software that underpins our data platform: ingestion frameworks, execution services, orchestration, metadata, data governance, and developer experience. Your focus is building APIs, libraries, and services that make data producers/consumers effective, while optimizing reliability, performance, and spend on AWS.

You'll enjoy this role if you are…

• A builder of platforms and frameworks (not just point pipelines).
• Comfortable with distributed systems abstractions (compute scheduling, storage layout, back‑pressure, retries, idempotency).
• Excited by lakehouse tech and modern data contracts, and you want to create self‑service for hundreds of use cases.

Your day to day is…

• Design and ship platform services for ingestion, transformation, orchestration, and metadata (e.g., service‑backed interfaces for Dagster/Airflow, lineage, quality, and data contracts).
• Build execution & scheduling capabilities for Spark/SQL jobs (queuing, prioritization, retries, resource isolation on EMR/EKS/Databricks), focusing on throughput and developer experience.
• Implement lakehouse features (Delta/Iceberg): schema evolution, partitioning, compaction, vacuum, snapshotting, ACID guarantees, and table‑format governance
• Optimize Snowflake and other warehouses: cost controls, query profiling/pruning, workload isolation, RBAC; expose safe self‑service patterns.
• Deliver SDKs, CLIs, and templates that standardize how teams build reliable data products; enable CI/CD for data and contract testing.
• Work across AWS (S3, EMR/EKS, Glue/Athena, Lambda, Kinesis/MSK) with IaC (Terraform) and strong observability (Datadog/CloudWatch).

You'll be successful in this role if you have…

• 8+ years building backend/platform software with Python/Scala/Java and strong SQL; proven track record designing distributed systems.
• Deep experience with Spark (Databricks or EMR/EKS) and AWS data services; solid grasp of scheduler/executor behavior and performance tradeoffs.
• Hands‑on data warehouse optimization (Snowflake ideal; others welcome).
• Experience building platform APIs/SDKs that other engineers adopt; excellent collaboration and technical leadership.

Bonus if you have…

• Experience at petabyte‑scale data platforms, distributed big data compute, or lakehouse engines (Delta/Iceberg).
• Familiarity with metadata/governance tech (Unity Catalog, Collibra, Lake Formation).

Benefits:

• Flexible, hybrid work environment at our convenient Soho location
• Unlimited Vacation
• 100% paid employee health benefit options (including medical, dental, and vision)
• Commuter Benefits
• 401(k) with employer funded match
• Corporate wellness program with Wellhub
• Sabbatical leave (for employees with 5+ years of service)
• Competitive paid parental leave and fertility/family planning reimbursement
• Cell phone reimbursement
• Catered lunch everyday along with beverages and snacks
• Employee Resource Groups and ZocClubs to promote shared community and belonging
• Great Place to Work Certified