Data Platform Engineer

At Adyen, Data Platform Engineers sit at the intersection of Data Engineering, Backend Engineering, and Systems Engineering, building the foundational layer of tooling and processes for on-premise Big Data Platforms. These tools support tens of products, hundreds of developers, and thousands of daily jobs that add to Adyen’s portfolio of capabilities.

You’ll collaborate with data engineers and ML scientists and engineers to build and roll out these tools. Along with DataOps Engineers, you’ll develop and operate multiple big data platforms, of which the biggest is 500+ nodes, with 25k+ CPU cores, 240+ TB of RAM, and 23+ PB of storage capacity, running tens of thousands of Spark jobs a day.

What you’ll do

You’ll be asked to both build new tools and improve the existing tooling. This role is perfect for you if you're passionate about a few of the following:

• Develop and maintain scalable and high-performance big data platforms.
• Work with distributed systems in all shapes and flavors (databases, filesystems, compute, etc.).
• Identify opportunities to improve continuous release and deployment environments.
• Build or deploy tools to enhance data discoverability through the collection and presentation of metadata.
• Introduce and extend tools to enhance the quality of data platform-wide.
• Explore and introduce technologies and practices to reduce the time to insight for analysts and data scientists.
• Develop streaming processing applications and frameworks.
• Work with infrastructure and large-scale private cloud systems.

For the currently open positions, the immediate focus will be the OLAP ecosystem (Druid, Clickhouse, Pinot), or the Data Engineering ecosystem (data contracts, validations, optimization tooling, Kafka ingestion tooling).

Who you are

• Fluent in Python. Java, Golang, or Rust is also appreciated.
• A team player with strong communication skills.
• Able to work closely with diverse stakeholders you enable (analysts, data scientists, data engineers, etc.) and depend upon (infrastructure, security, etc.).
• Experience developing and maintaining distributed data and compute systems like Spark, Trino, Druid, etc.
• Experience with data modelling tooling.
• Experience with DevOps pipelines and development ecosystems.
• Experience with real-time and batch data pipelines (via Kafka, Spark streaming) with an eye for frameworks, user friendliness, and quality.
• Experience with the Kubernetes ecosystem (k8s, docker), and/or Hadoop ecosystems (Hive, Yarn, HDFS, Kerberos).

This role is based out of our Amsterdam office. We are an office-first company and value in-person collaboration; we do not offer remote-only roles.