Pinterest's Moka: How Kubernetes Is Revolutionizing Big Data Processing
In a groundbreaking move, Pinterest has unveiled its innovative approach to large-scale data processing with the introduction of Moka, a cutting-edge platform that promises to redefine the landscape of big data management. The company's blog post (https://medium.com/pinterest-engineering/next-gen-data-processing-at-massive-scale-at-pinterest-with-moka-part-2-of-2-d0210ded34e0) provides an in-depth look at how Pinterest is transforming its data infrastructure, shifting away from traditional Hadoop systems towards a Kubernetes-based architecture on Amazon EKS.
The Pinterest Big Data Platform team, comprising Soam Acharya, Rainie Li, William Tom, and Ang Zhang, embarked on a journey to create a next-generation data processing platform as the limitations of their existing Hadoop-based system, Monarch, became apparent. Moka emerged as the solution, a cloud-native data processing platform designed to handle Pinterest's massive-scale operations. The two-part blog series offers a comprehensive insight into this transformation.
Part one (https://www.infoq.com/news/2025/07/pinterest-spark-kubernetes/) focused on the overall design and application layer, while part two delves into the infrastructure-focused aspects of Moka, including key learnings and future directions. The authors emphasize the practical benefits of adopting Kubernetes, highlighting a shift in the industry where big tech companies are increasingly treating Kubernetes as a control plane for data, rather than just a stateless service platform.
The team's exploration of Kubernetes-based systems was driven by the growing popularity and adoption in the Big Data community, aiming to replace Hadoop 2.x. Moka exemplifies how a Hadoop-era data platform can be modernized without compromising existing Spark investments. A central theme in part two is the efficient operation of Spark at a large scale on Kubernetes, with the authors detailing how they enhanced Moka with logging, metrics, and job history services, enabling engineers to debug and optimize jobs without deep knowledge of cluster topology.
Pinterest's commitment to reproducibility is evident through infrastructure-as-code practices. The team utilizes Terraform and Helm to create EKS clusters, configure networking and security, and deploy essential components like the Spark History Server. This approach ensures consistency and control over the data processing environment.
One of the critical aspects of Moka is its ability to support different hardware architectures. Pinterest's engineers developed multi-architecture images to ensure optimal performance on Intel and ARM-based instances, including AWS Graviton. This strategy aligns with cost and efficiency goals, as noted by InfoQ editor Eran Stiller in a LinkedIn summary (https://www.linkedin.com/posts/estiller_from-hadoop-to-kubernetes-pinterests-scalable-activity-7356213210661744640-17Ee/), emphasizing container-level isolation, ARM support, YuniKorn scheduling, and significant cost savings through workload consolidation and auto-scaling.
The broader industry conversation on processing engines adds depth to Pinterest's story. Acharya's LinkedIn post (https://www.linkedin.com/posts/soamacharya_next-gen-data-processing-at-massive-scale-activity-7350931757694640128-vCg0/) reveals that while Spark remains the primary engine, Moka's success has led to the adoption of other frameworks. Flink Batch is now in production, with Apache Ray following suit, and Flink Streaming scheduled for deployment later this year. This diversification underscores the flexibility of Moka as a platform that can accommodate various processing engines based on specific workload requirements.
External observers have drawn valuable lessons from Pinterest's Moka implementation. The ML Engineer newsletter (https://ml.blaze.email/archive/ml-engineering-newsletter-5558/) highlights Moka as a reference architecture for modern data infrastructure, showcasing its capabilities in deploying EKS clusters, logging with Fluent Bit, metrics pipelines with OTEL, image management, and a custom Moka UI for Spark on Kubernetes. These observations suggest that Moka is becoming a benchmark for cloud-native data systems.
However, the team emphasizes that their migration journey is ongoing, not a completed project. In the blog and a LinkedIn post (https://www.linkedin.com/posts/pinterest-engineering_next-gen-data-processing-at-massive-scale-activity-7373833079498371072-EREu/), they discuss the 'learnings and future direction,' detailing how early proof-of-concepts led to a phased transition away from Hadoop as confidence in the new stack grew. Acharya's insight that 'the best problems show up at scale' highlights the importance of addressing challenges as real workloads are migrated.
For other organizations, this experience may be the most crucial lesson. While adopting Kubernetes, EKS, and Spark is relatively straightforward, the real challenge lies in decoupling from legacy systems and investing in observability, automation, and multi-engine support. This process is likely to be the ongoing work that lies ahead for Pinterest and other companies embarking on similar data infrastructure transformations.
