DATA ENGINEERING

Spark pipelines, on a Rust engine.

Run the Spark workloads you already have. Build the AI data pipelines you need next.

Try it out Talk to an engineer →
platform.lakesail.com/jobs
Example data
Jobsorders_daily
orders_daily
Job VersionTriggerStatusDurationStart Time
v7Schedule for Nov 3, 02:00 AMRunning3m 52sNov 3, 02:00 AM
v7Schedule for Nov 2, 02:00 AMSucceeded4m 12sNov 2, 02:00 AM
v7Manual Succeeded4m 31sNov 1, 09:14 AM
v6Schedule for Nov 1, 02:00 AMFailed51sNov 1, 02:00 AM

Coverage

Everything your Spark work covers.

Lakehouse pipelines, scheduling, and analytics on one engine, with the Spark APIs your team already uses.

Batch ETL and ELT

Python and SQL jobs that run your Spark code as it is.

Example

Lakehouse tables

Delta Lake and Iceberg, built into the engine.

your object storageDelta LakeIceberg
  • MERGE
  • UPDATE
  • DELETE
  • Time travel

Sail reads and writes Amazon S3, Azure Data Lake Storage, Google Cloud Storage, and HDFS.

Schedules and retries

Cron schedules, automatic restarts, and alerts.

0 2 * * *daily, 02:00 UTCExample

Backfills and reruns

Run with parameters, catch up missed runs, or retry.

Airflow, Dagster, Prefect

Keep them. They can start and watch jobs through the API.

AirflowDagsterPrefectLakeSail APIstart · watch

Notebooks and ad-hoc SQL

Console notebooks, or Spark Connect from your own client.

Spark compatibility

Same Spark code. Nothing to rewrite.

Sail speaks Spark Connect. Any difference from Spark’s behavior is treated as a bug.

On LakeSail, nothing changes

The job runs as written. LakeSail points the session at Sail for you.

Example

Check your code first

Paste a file, or run spark-to-sail scan on a repo.

Scan your Spark code

From Apache Spark

Scan the job, compare the output, then schedule it.

Migrating from Spark

From Databricks

Connect Unity Catalog, recreate your jobs, and swap out dbutils.

Databricks migration guide

With LakeSail engineers

They run the comparison with you under the Spark migration guarantee.

The guarantee

Production

From notebook to production job.

Explore in a notebook, run it as a job, and let LakeSail run it on a schedule, with alerts when a run fails.

  1. 01

    Explore in a notebook

    Write Spark SQL and DataFrame code against your tables, in the console.

  2. 02

    Run it as a job

    Run the notebook, a Python file, or SQL as a job.

  3. 03

    Run it on a schedule

    Cron schedules, automatic restarts, run history, and alerts.

    Cron Expression0 2 * * *
    TimezoneUTC
    Example
    Last run Succeeded

Open tables

Your tables stay open.

Delta Lake and Iceberg are built into the Sail engine, with no extra jars. Tables stay in your object storage, registered in your catalog, in standard formats other engines read too.

All formats, catalogs and storage

On both formats

your object storageyour catalogDelta LakeIcebergSail
  1. Engine
    Sailreads and writes natively, no extra jars
  2. Table formats
    Delta LakeIceberg
  3. Catalog
    AWS GlueUnity CatalogIceberg RESTHive Metastore
  4. Where Sail reads and writes
    Amazon S3Azure Data Lake StorageGoogle Cloud StorageHDFS

Performance

Measured on one machine and on a cluster.

On derived TPC-H, Sail finished in a tenth of Spark’s time on the same machine. On derived TPC-DS, it ran the 1 TB and 3 TB suites on eight workers.

Same machine

Derived TPC-H · SF100 · r8g.4xlarge
10x faster than SparkTotal query time
About a third of the memoryPeak memory
No spill to diskShuffle spill

Sail against Spark. Methodology

Terabyte scale

Derived TPC-DS · total seconds, lower is better
1 TB
SailLakeSail · 8 r8gn.8xlarge workers 444.32 s
Native SparkAWS · 6 r8g.12xlarge nodes 1,731.67 s
3 TB
SailLakeSail · 8 r8gn.8xlarge workers 1,288.77 s
Spark + Gluten/Velox 1.6.0AWS · 12 r8gd.12xlarge nodes 2,239.93 s
Native SparkAWS · 12 r8gd.12xlarge nodes 3,650.56 s

Different hardware and methods. Not a controlled head-to-head comparison. Derived from TPC-DS, not officially audited. LakeSail blog · AWS, 1 TB · AWS, 3 TB

Sessions

Your cluster, from anywhere.

Set one URL and work from your laptop, or hand the same session to your coding agent.

$ export SPARK_REMOTE="sc://…/;token=…" How sessions work

What comes next

Build the AI data pipelines you need next.

Add model steps to the Spark jobs you already run. Python runs inside the Sail engine, so a model call works on the same batches as your SQL, with no second process.

Python and AI workloads
Exampleproducts job

The Spark job you run today

  1. Read spark.read.parquet(…) From your object storage
  2. Clean .dropDuplicates() DataFrame code
  3. Merge MERGE INTO products Delta Lake or Iceberg

Two more steps

  1. Embed .mapInArrow(embed, …) Your model, in Python
  2. Search vector_cosine_similarity In SQL

All in one Sail engine

Agents

Bring your own model.

Give your agent a personal access token. Through the LakeSail API it can create jobs, run them, and watch them.

API reference
Your agent
On any model you choose
Query tables
Sessionsc://…
LakeSail API
with your access
Define and publish jobs
POST/workloads/jobs
Start runs with parameters
POST/workloads/jobs/{job}/runs
Release a held run
POST…/runs/{jobRun}/ready
Retry a failed run
POST…/runs/{jobRun}/retry
Read runner and engine logs
POST…/runs/{jobRun}/logs/stream
React to run events
job_run.status.failed
job_run.status.succeeded

Getting started

From sign-up to your first job.

  1. Create your account

    Sign up with your email.

    Verified

  2. Connect your cloud account

    Deploy the LakeSail setup template.

    Connected

  3. Create a network and a cluster

    LakeSail provisions both.

    Ready

  4. Run your first job

    Run SQL, Python, or a notebook.

    Succeeded

Where it runs LakeSail hosts the console and API. Compute and data stay in your environment.

Run your next pipeline on LakeSail.

Start with one job you run today. Compare the output, then move the rest.

Try it out Talk to an engineer →

Privacy

Loading your saved choices…

Always on