Shehab Amin, LakeSail, Spark Rebuilt In Rust: Making The Data Engine AI-Native
This talk video is a node in devreal.ai, the developer community graph.
It connects to:
Talk: Spark Rebuilt In Rust: Making The Data Engine AI-Native — https://devreal.ai/t/260826-spark-reb...
Speaker: Shehab Amin — https://devreal.ai/n/shehab-amin
Company: LakeSail — https://devreal.ai/c/lakesail
Project: Apache Spark — https://devreal.ai/p/apache-spark
Project: Sail — https://devreal.ai/p/sail
Project: Apache DataFusion — https://devreal.ai/p/apache-datafusion
Project: Apache Arrow — https://devreal.ai/p/apache-arrow
Project: Delta Lake — https://devreal.ai/p/delta-lake
Project: Apache Iceberg — https://devreal.ai/p/apache-iceberg
Project: DuckLake — https://devreal.ai/p/ducklake
Event: Apache Spark NYC Meetup August 2026 — https://devreal.ai/e/260826-spark-nyc...
Explore these connections and related videos, photos, speakers’ blogs and podcasts, and other By the Bay networks.
-----
What if Spark were rebuilt from scratch for the AI era?
Shehab Amin of LakeSail presents Sail—a Spark Connect-compatible data engine written entirely in Rust. Sail keeps the familiar Spark SQL and DataFrame interfaces while replacing the JVM-based runtime with an Arrow-native architecture powered by Apache DataFusion.
In this technical deep dive, Shehab explores:
• Why the JVM and out-of-process Python create fundamental barriers for AI workloads
• How Sail runs Python UDFs in-process using PyO3 and shared Arrow memory
• Sail’s actor-based control plane and stateless worker architecture
• Distributed query planning, task regions, and resource management
• Pipelined versus blocking shuffle execution
• Native Delta Lake, Iceberg, and DuckLake integration
• MERGE planning and execution inside a distributed engine
• SQL-based observability through Sail’s system catalog
• TPC-H and ClickBench results comparing Sail with Spark, Spark accelerators, and Databricks
The result is a unified engine for batch, streaming, ad hoc analytics, and AI workloads—with Spark compatibility at the protocol boundary and a completely new execution model underneath.
Sail’s message is simple: stop tuning the JVM. Replace it.
Watch to learn how Rust, Arrow, DataFusion, embedded Python, and distributed systems design come together to make the data engine AI-native.
#ApacheSpark #Rust #DataEngineering #Lakehouse #ApacheArrow #DataFusion #DistributedSystems #ArtificialIntelligence #Python #LakeSail