SPARK CONNECT

Spark’s client. Sail’s engine.

Point your Spark session at Sail. The same Spark SQL and DataFrame code runs on a Rust engine, with no JVM.

Start free trial Check your code →

daily_orders.py Spark-to-Sail scan

from pyspark.sql import SparkSession, Window, functions as F

spark = SparkSession.builder \

.remote("sc://sail:50051").getOrCreate()

items = spark.sql("""SELECT * FROM sales.orders

LATERAL VIEW explode(items) t AS item""")

w = Window.partitionBy("customer").orderBy(F.desc("ts"))

latest = (items.withColumn("rn", F.row_number().over(w))

.filter("rn = 1"))

daily = latest.groupBy("day").agg(F.sum("amount"))

daily.write.format("delta").saveAsTable("sales.daily")

Ready 16 Spark features, all supported

How it works

Two ways in. One engine.

Both arrive over Spark Connect and become one Sail plan.

How we built the SQL parser

PYDataFrame API daily = (orders .groupBy("day") .agg(F.sum("amount")))

SQLSpark SQL SELECT order_id, itemFROM sales.ordersLATERAL VIEW explode(items) t AS item

Spark Connect

One Sail plan Resolved, then optimized. SQL text is read by Sail’s own parser first.

The Sail engine Written in Rust. Results return to your client as Arrow.

What runs today

Sail runs the Spark code your team depends on.

Spark SQL

SQLranked.sql

SELECT customer_id, amount,

rank() OVER (PARTITION BY day

ORDER BY amount DESC) AS r

FROM sales.orders

DataFrame API

PYjoin.py

de_orders = (orders

.join(customers, "customer_id")

.filter("country = 'DE'"))

Python UDFs

PYudfs.py

@pandas_udf("double")

def usd(a: pd.Series) -> pd.Series:

return a * 1.08

Delta Lake and Iceberg

SQLupsert.sql

MERGE INTO sales.daily t

USING updates s ON t.day = s.day

WHEN MATCHED THEN UPDATE SET *

WHEN NOT MATCHED THEN INSERT *

Catalogs

spark.read.table("sales.orders")

AWS GlueUnity CatalogIceberg RESTHive Metastore

Full feature lists

Performance

Spark SQL at terabyte scale.

Derived TPC-DS at 1 TB and 3 TB, beside AWS’s Spark results.

All benchmarks

1 TB

Sail LakeSail · 8 r8gn.8xlarge workers · 256 vCPU

444.32 s

Native Spark AWS · 6 r8g.12xlarge nodes · 288 vCPU

1,731.67 s

3 TB

Sail LakeSail · 8 r8gn.8xlarge workers · 256 vCPU

1,288.77 s

Spark + Gluten/Velox 1.6.0 AWS · 12 r8gd.12xlarge nodes · 576 vCPU

2,239.93 s

Native Spark AWS · 12 r8gd.12xlarge nodes · 576 vCPU

3,650.56 s

Total time for the query suite, in seconds. Lower is better. Different hardware and methods. Not a controlled head-to-head comparison. Derived from TPC-DS, not officially audited. Sources: LakeSail blogAWS Data on EKS, 1 TBAWS Data on EKS, 3 TB

How we test it

Tested against Spark itself.

Apache Spark’s tests, SQL, and results. Any difference from Spark’s behavior is treated as a bug.

Known differences from Spark

Same query, both engines

Run on both, rows compared.

Example

SELECT item, size(amounts) AS n, aggregate(amounts, 0D, (acc, x) -> acc + x) AS totalFROM orders ORDER BY item

Apache Spark

item	n	total
jam	1	9.25
scones	2	21.75
tea	2	16.5

Sail

item	n	total
jam	1	9.25
scones	2	21.75
tea	2	16.5

3 of 3 rows identical, same schema. Compare on your own data

Spark’s PySpark tests

On every merge to main.

Spark tests

pyspark.sql.tests.connect
doctests: functions, dataframe, column, catalog

How the tests run

SQL from Spark’s test suites

select a, b, sum(c) as c from d group by rollup(a, b) Both accept
SELECT FROM orders WHERE Both reject

How the SQL cases are built

Check your code

Know before you move anything.

Scan a file, a notebook, or a folder of jobs.

Scan your Spark code

Runs in your browser, or as a CLI.
A scan reads your code. Compare outputs before you switch.

ls jobs

PYdaily_orders.py
PYprice_orders.py
SQLupsert.sql

spark-to-sail scan ./jobs

Ready

[+] 21 supported [!] 0 partial [x] 0 unsupported

21 features detected in 3 files.

Try it locally

Run it on your laptop.

Open source, Apache 2.0.

Download Sail

$ pip install pysail $ pip install pyspark-client==4.2.0 $ sail spark server # then, in a second terminal $ python >>> from pyspark.sql import SparkSession >>> spark = SparkSession.builder \ ... .remote("sc://localhost:50051").getOrCreate()

With LakeSail, we can develop and test PySpark code locally far more efficiently – even on Windows. In our environment, it starts up in seconds and processes small data samples almost instantaneously. This helps reduce our test runtimes and streamlines our overall development workflow.

Martin Bode Data Solution Architect · Provinzial

Bring your Spark code to LakeSail.

Scan one file. Run one job.

Start free trial Scan your code →