Apache Spark vs VecRuntime vs DataFusion Comet on TPC-DS 1 TB

VecRuntime runs Spark SQL's filters, projections, aggregates, sorts and joins on Arrow-layout batches with the Java Vector API -- on the JVM, no native code -- and moves batches between executors over its own Arrow Flight shuffle. Apache DataFusion Comet offloads the same operators to a native Rust engine. This page compares both against plain Apache Spark on the TPC-DS 1 TB workload on Amazon EKS, all three on identical hardware, data and Spark settings.

Summary

Benchmark infrastructure

Methodology. Spark, VecRuntime and Comet ran one after another in one cluster session on 2026-09-30 (morning), on the same nine nodes, each alone on the cluster, over the same S3 data with the same Spark settings; only the execution engine and its own memory split differ (every engine has 50 GB per executor; where it puts them follows where it allocates). Comet Native Scan + VecRuntime ran the same afternoon in a second cluster session on the same node group, availability zone, image and settings; S3 throughput varies between sessions, so its comparison with the other three carries that uncertainty. Each query ran once after the plan was compiled; the time is the wall-clock of the query's execution as the runner measures it.

Test environment

Versions

Configuration

Common to all three engines:

spark.sql.shuffle.partitions=300
# spark.sql.adaptive.advisoryPartitionSizeInBytes left at Spark's default (64 MB); no coalescePartitions.minPartitionNum
spark.eventLog.enabled=true

spark.plugins=io.vecruntime.spark.VectorPlugin
spark.shuffle.manager=org.apache.spark.sql.vecruntime.shuffle.VectorShuffleManager
spark.vecruntime.exec.strictFloatingPoint=false   # Comet's default too
spark.vecruntime.shuffle.aqe.mapSizeScaling=true   # AQE sees Spark-scale map output sizes (#514)
spark.vecruntime.shuffle.aqe.sparkCompressionRatio=0   # the uncompressed-bytes ratio (#514)
AOT class-data cache off
--add-modules=jdk.incubator.vector --enable-native-access=ALL-UNNAMED (driver and executors)

spark.plugins=org.apache.spark.CometPlugin
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=24g
spark.comet.exec.enabled=true
spark.comet.scan.enabled=true
spark.comet.exec.shuffle.enabled=true
spark.comet.exec.shuffle.mode=auto
spark.comet.cast.allowIncompatible=true
spark.comet.explainFallback.enabled=true

Comet Native Scan + VecRuntime:

spark.plugins=org.apache.spark.CometPlugin,io.vecruntime.spark.VectorPlugin
spark.shuffle.manager=org.apache.spark.sql.vecruntime.shuffle.VectorShuffleManager
spark.vecruntime.shuffle.enabled=true
spark.comet.enabled=true
spark.comet.scan.enabled=true   # CometNativeScanExec: DataFusion's Rust Parquet reader and object_store S3 I/O
spark.comet.exec.enabled=true   # with every Comet operator switched off: filter, project, aggregate, joins, sort, window, ...
spark.comet.exec.shuffle.enabled=false
spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=10g   # executors 22 GB heap / 18 GB overhead (16 GB direct) / 10 GB off-heap = 50 GB
# plus VecRuntime's keys above (strictFloatingPoint=false, mapSizeScaling=true, sparkCompressionRatio=0)

Performance results

Per query

Seconds per query, all 103, the three engines side by side (hover for values; click a legend entry to hide an engine).

Speedup over Spark, per query

Spark's time divided by the engine's; above 1 is faster than Spark. Log scale.

Performance distribution

Top 10 improvements -- VecRuntime

Regressions -- VecRuntime

Top 10 improvements -- Comet

Regressions -- Comet

Top 10 improvements -- Comet Native Scan + VecRuntime

Regressions -- Comet Native Scan + VecRuntime

Where each engine wins

The heavy joins are VecRuntime's: q93 57.0 s against Spark's 139.3 and Comet's 85.9; q64 48.6 against 92.7 and 63.8; q50 38.4 against 66.8 and 47.4; q29 9.8 against 34.3 and 15.6; q23b 117.7 against 281.7 and 123.5. Its shuffle moves 0.47 TB where Spark's moves 0.94 and Comet's 0.66. Comet leads where the scan dominates -- q9 60.1 s against VecRuntime's 87.1, q28 84.6 against 115.6, q88 96.5 against 111.2 -- and on the heavy aggregates: q4 46.4 against 86.4, q11 27.4 against 48.2, q67 55.0 against 66.8. Comet Native Scan + VecRuntime separates the two: on Comet's Parquet reader VecRuntime's operators take q28 to 82.5 s, q9 to 65.4, q88 to 97.1 and q44 to 26.5 (Comet 26.9), and q50 to 27.9 and q93 to 52.4, so the scan-bound gap is the reader's; q4 (84.1), q11 (46.3) and q67 (68.2) barely move, so that gap is in the aggregates. VecRuntime's losses to Spark are q18 (12.4 s against 9.2, a cold query; the AOT cache is off here, #558), q99, q11, and the scan-bound q28 and q24b.

Notes

• Every engine returned Spark's row counts on every query, and Spark's checksums on every query except q65, whose result has ties that every engine orders differently.
• The three legs ran on 2026-09-30 in one cluster session on the same nine nodes, one after another (Spark, VecRuntime, Comet), each alone on the cluster. S3 throughput varies between sessions, so only legs from one session are compared.
• Comet is a source build of main with two unmerged fixes: #6268, without which q5 fails at 1 TB with native scans, and #6270, without which q64 loses rows (apache/datafusion-comet#6264; 0 rows against 12,185 in the earlier run, #6133). With both, q5 and q64 return Spark's rows and checksums.
• AQE is at its defaults for every engine. The earlier x86 run used a 128m advisory size and coalescePartitions.minPartitionNum=208 (deprecated in Spark 3.2+); its numbers are in docs/results.md.
• Comet Native Scan + VecRuntime returned Spark's row counts on all 103 queries and Spark's checksums on all but q65 (ties). It uses the same patched Comet build.

All queries

Running the benchmark

The cluster runner, the Spark-on-Kubernetes manifests, the image and the data generation are in the benchmark runner's README: run-matrix.sh renders a SparkApplication per engine configuration (spark, vector-shuffle, comet, ...) and writes one JSON-lines result file per run; this page is rendered from three of them by benchmarks/scripts/render-benchmark-page.py.

TPC-DS is a benchmark of the Transaction Processing Performance Council; these results are not audited TPC results and are not comparable to published TPC-DS results. Times are the median of one measured iteration per query on the cluster described above; run-to-run variation on the heavy queries is a few percent.