One engine for
data and AI.
Run the Spark workloads you already have. Build the AI data pipelines you need next.
# Daily active userschart Example dataevents = spark.table("events")daily = events.groupBy("event_date")users = daily.agg( F.count_distinct("user_id").alias("dau"))What changes when the runtime is Rust
Sail is the open-source Rust-native multimodal compute engine at LakeSail’s core. Built on Apache Arrow and DataFusion, it runs SQL, batch data pipelines, and Python processing without a JVM.
Rust-native runtime
No JVM startup. No garbage collection pauses. No JVM memory tuning. Sail boots instantly and scales to zero between jobs, you only pay for compute you use.
Vectorized query execution
Built on Apache Arrow's columnar format and DataFusion's vectorized query engine. Processes data with SIMD acceleration where available.
SQL, Python, and AI data pipelines
One engine, one API, one cost model. Run batch ETL, Python workloads, and interactive SQL queries without switching tools, re-learning APIs, or managing separate clusters.
Native Python at engine speed
Python UDFs and workloads execute natively in-process, no inter-process serialization, no JVM-to-Python IPC overhead. AI/ML pipelines that previously paid a heavy tax run at native speed.
Stateless, secure workers
Workers are fully stateless, no shuffle data on disk between runs, no leftover JVM processes consuming memory. Each job gets a clean, isolated execution environment. Easier security audits, simpler ops.
Transparent cost model
Charged for actual compute hours, fully transparent, predictable, with no opaque credits. Autoscales to zero between jobs. No minimum spend. No contract lock-in. You see exactly what you're paying for.
Plug in. Nothing else changes.
LakeSail implements the Spark Connect protocol. Point your existing Spark code at a new endpoint and that's it. Same DataFrame API, same libraries, same pipelines. The engine upgrades; your code doesn't.
Update one config line
Swap the .remote() endpoint string. That's the migration. No code changes, no schema conversions,
no library updates.
Run your existing jobs
Run Spark SQL and DataFrame code, including Python UDFs, on Sail’s Rust-native engine. The compatibility guide covers supported features and migration details.
Build your next AI pipeline →
Extend your SQL and DataFrame workflows with Python libraries for multimodal processing, embeddings, and model scoring.
From raw data to AI datasets.
Prepare data for training, evaluation, and retrieval applications. Sail combines SQL transformations with Python libraries and model code in the same engine.
Training and evaluation data
Filter, deduplicate, join, and transform datasets for LLM training and evaluation using SQL and Python.
Physical AI data preparation
Align video frames with state and action records. Validate episode structure and create training and evaluation splits.
Embeddings and model scoring
Run Python model code alongside data transformations. Write results to Parquet or Delta Lake for downstream applications.
From one node to a cluster
Scale data pipelines from a single node to distributed execution as your workload grows.
File contents, text, and vectors in one data pipeline.
- ReadPythonPDF bytesFile reader
- ExtractPythonText + metadataDocument library
- ChunkPythonText passagesText splitter
- EmbedPythonVector arraysYour model
- WriteSailParquet datasetDataFrame writer
sourcepagechunkembeddingExtraction, chunking, and embeddings use your Python libraries and model.
No lock-in. Your formats, your cloud.
LakeSail is built on open standards end to end. Native Apache Iceberg and Delta Lake support means your tables stay exactly where they are, no conversion, no copying. Your data stays in open formats in your own AWS account. Switching engines should never require migrating terabytes of data.
Apache Iceberg
Native read and write support. Time travel, schema evolution, partition evolution, all supported. LakeSail does not require you to convert or copy Iceberg tables.
Delta Lake
Native read and write support for Delta tables. Keep Delta indefinitely or migrate to another format at your own pace, your choice, not ours.
Spark Connect protocol
Sail implements the Spark Connect protocol for Spark SQL and DataFrame code. Keep your existing client tools and check Sail’s supported features for your workload.
Apache Arrow & DataFusion
The Sail engine is built on Apache Arrow and DataFusion, both Apache Software Foundation projects with large, independent ecosystems.
Built for AI agents from day one.
Competitors retrofit agent support onto a JVM platform that was never designed for it. LakeSail ships an MCP server, lakehouse branching, and full audit trails as core engine features, not add-ons.
Native MCP server
LakeSail exposes a Model Context Protocol server out of the box. Connect any MCP-compatible AI agent, Claude, GPT, custom agents, to your lakehouse directly. Query, transform, and write data without building a custom tool layer.
Lakehouse branching
Agents can branch your lakehouse like a git repo. Create an isolated sandbox for a transformation or analysis, run it, review the diff, and commit or discard, all without touching production data.
Elastic agent compute
Compute provisions per agent workload, scales with execution, and releases when work is done. Sub-second cold starts on the Rust-native engine mean short-lived agent loops never pay JVM warm-up, and there are no idle clusters between calls.
Dynamic Python tooling
Agents can define Python tools and data sources at runtime, custom logic that runs at engine speed against lakehouse data. No pre-registration, no redeployment. The agent writes it, Sail executes it.
Agent receives task
LLM agent gets context via MCP server tools
Branch created
Isolated sandbox branched from production lakehouse
Actions executed
Queries, transforms, writes, all audited in real time
Human reviews diff
Exact changes surfaced for approval before any commit
Commit or discard
One-click merge to production, or clean rollback
From open source to fully managed.
Three ways to run Sail. OSS if you want to self-host, LakeSail Platform if you want managed, and Enterprise for organizations with procurement requirements. All three use the same engine.
Sail OSS
The Rust engine, open source on GitHub under Apache 2.0. Self-host on your own infrastructure. No managed services, no support SLA, but full access to the engine.
- Apache 2.0 license
- Full Spark Connect compatibility
- Community support via GitHub & Slack
- No contracts, no commitments
LakeSail Platform
Fully managed Sail, deployed into your AWS account (BYOC). We handle the infrastructure; you keep data sovereignty and security. No cluster management, no ops overhead.
- Deploys into your AWS account (BYOC)
- Managed upgrades, patches, monitoring
- Autoscale to zero between jobs
- Per-second billing, no minimums
Enterprise
For organizations with procurement, security review, and custom SLA requirements. Includes dedicated support, on-premises deployment options, and enterprise SSO.
- Everything in Platform, plus:
- On-premises deployment available
- Dedicated support with SLA
- Enterprise SSO (OIDC)
- Custom licensing models
Bring your data
and AI workloads.
Talk to a LakeSail engineer about your pipeline, from Spark analytics to training data and embeddings.