VecRuntime on TPC-DS 10 TB, AWS Graviton4 (11 x m8gd.4xlarge)

VecRuntime runs Spark SQL's filters, projections, aggregates, sorts and joins on Arrow-layout batches with the Java Vector API -- on the JVM, no native code -- and moves batches between executors over its own Arrow Flight shuffle. This page is its TPC-DS 10 TB run on eleven Graviton4 nodes, set against the totals quanton.dev/benchmarks (Onehouse), read 2026-10-21 publishes for other engines on the same instance type and count.

VecRuntime, 99 queries

2,430 s

10 TB · 11 x m8gd.4xlarge

vs OSS Apache Spark (published)

5.0x

12,200 s against 2,430 s

Second pass / first pass

2,430 / 2,449 s

0.8% apart

Total runtime

Lower is better. The grey bars are published totals, not re-run here (see the notes); VecRuntime's is this page's run.

Engine	Total (s)	Its time / VecRuntime's	Source
OSS Apache Spark	12,200	5.02x	published
DataFusion Comet (tuned)	9,122	3.75x	published
Gluten + Velox (tuned)	8,563	3.52x	published
Databricks Photon 18.2	2,550	1.05x	published
VecRuntime	2,430		this page
Quanton v0.43.0	2,034	0.84x	published

Totals as published on quanton.dev: the Quanton v0.43.0 suite total, and the other engines from the Quanton storage-aware Spark benchmark deck. Comet and Gluten are tuned configurations (their out-of-the-box runs failed with out-of-memory errors on Q67 and Q93). Not re-run here; per-query times are not published.

Where the time goes

The ten longest queries:

Query	Seconds	Share of total
q23	207.3	8.5%
q24	196.0	8.1%
q14	156.5	6.4%
q78	154.9	6.4%
q67	151.2	6.2%
q28	109.1	4.5%
q93	93.8	3.9%
q64	77.8	3.2%
q4	73.5	3.0%
q50	62.9	2.6%

Across the 99 queries:

Measure	Value
Executor time	101.9 h
GC time	0.81 h
Spill	942 GiB
Shuffle written	2.23 TiB

Per query

Seconds per query, both passes (hover for values).

Benchmark infrastructure

Methodology. Two full passes of the 99 queries, one after the other, each with the cluster to itself and all 11 executors up (checked before the first query). Each query ran once after its plan was compiled; its time is the wall-clock of its execution as the runner measures it, and the total is the sum over the queries. The page shows pass 2 and, per query, pass 1 next to it.

Test environment

Component	Configuration
Dataset	TPC-DS scale factor 10000 (10 TB), Parquet on Amazon S3; the 99 queries (q14, q23, q24 and q39 as the sums of their a and b parts)
Cluster	Amazon EKS; 11 x m8gd.4xlarge executor nodes (16 vCPU AWS Graviton4, 64 GB, local NVMe for shuffle and spill), one executor per node; the driver on a separate c8g.xlarge node
Executors	11 executors x 15 cores; 20 GB heap + 10 GB overhead + 28,928 MB off-heap (spark.memory.offHeap.size), 6 GB direct memory; G1; driver 3 cores x 4 GB
Storage	Amazon S3 through Hadoop S3A; VecRuntime's native Parquet scan (VectorParquetScanExec)

Versions

Component	Version
VecRuntime	PR #911's head 60c645c2, merged as main e74d0be5 (2026-10-21): DistinctBelowSemiJoin for semi and anti joins (#906, #907), operator arenas on the native pool (#909), pool accounting (#911); arm64 image
Spark (VecRuntime)	4.1.3, Scala 2.13, Amazon Corretto 25.0.4.1

VecRuntime configuration

spark.plugins=io.vecruntime.spark.VectorPlugin
spark.shuffle.manager=org.apache.spark.sql.vecruntime.shuffle.VectorShuffleManager
spark.sql.shuffle.partitions=1500, spark.sql.adaptive.enabled=true (coalescing on)
spark.memory.offHeap.enabled=true, spark.memory.offHeap.size=28928m
spark.vecruntime.exec.strictFloatingPoint=false, spark.vecruntime.exec.sortMergeJoin.enabled=true
spark.vecruntime.memory.poolAccounting.enabled=true (the default)
Executors: G1, -XX:MaxDirectMemorySize=6g, Arrow on its Unsafe allocation manager; glibc malloc defaults (no MALLOC_ARENA_MAX, no MALLOC_MMAP_THRESHOLD_)
--add-modules=jdk.incubator.vector --enable-native-access=ALL-UNNAMED (driver and executors)

Notes

The other engines' totals are the ones quanton.dev publishes for TPC-DS 10 TB on 11 x m8gd.4xlarge; they were not re-run here, and only their totals are published, so the comparison is of totals only.
Their setups are not documented to the level of this page (dataset generation and table format, Spark versions, executor layout, whether the driver is one of the 11 nodes). Treat the cross-engine bars as indicative, not as a controlled comparison.
VecRuntime needs Spark 4.1+ and JDK 25.
A third pass at 600 shuffle partitions took 2,520 s (1,500 partitions stays the setting); the same configuration before #909 and #911, with MALLOC_ARENA_MAX=2 and MALLOC_MMAP_THRESHOLD_=131072, took 2,618 s. See issue #869.
q18 and q68 end in LIMIT 100 over rows with tied sort keys, so their checksums can differ between runs and engines; every other query's result is the same in both passes.

All queries

Seconds per query

Query	Pass 2 (s)	Pass 1 (s)	Difference
q1	8.1	8.1	-0.0
q2	12.9	12.8	-0.1
q3	5.9	6.0	+0.1
q4	73.5	73.0	-0.5
q5	17.9	18.0	+0.1
q6	3.0	2.9	-0.0
q7	11.9	11.6	-0.3
q8	4.2	3.5	-0.7
q9	53.2	52.3	-0.9
q10	4.2	4.3	+0.1
q11	30.3	30.3	+0.0
q12	2.8	2.7	-0.0
q13	14.7	13.6	-1.1
q14	156.5	153.5	-3.1
q15	3.7	4.0	+0.3
q16	51.1	49.8	-1.4
q17	10.3	9.5	-0.8
q18	12.9	13.1	+0.1
q19	4.0	3.8	-0.1
q20	2.5	2.4	-0.1
q21	1.6	1.6	-0.0
q22	4.3	4.4	+0.1
q23	207.3	201.0	-6.4
q24	196.0	196.0	+0.0
q25	8.1	8.1	+0.0
q26	7.1	6.5	-0.6
q27	10.7	9.9	-0.8
q28	109.1	112.7	+3.6
q29	22.1	22.5	+0.3
q30	6.7	7.3	+0.7
q31	6.8	6.7	-0.1
q32	1.6	1.9	+0.3
q33	3.6	3.2	-0.4
q34	5.4	5.3	-0.1
q35	9.6	9.5	-0.1
q36	8.2	7.7	-0.5
q37	16.5	16.1	-0.4
q38	34.0	34.3	+0.3
q39	8.1	8.3	+0.1
q40	5.6	5.7	+0.1
q41	0.6	0.6	+0.0
q42	1.6	1.9	+0.3
q43	6.5	6.6	+0.1
q44	59.0	59.2	+0.2
q45	3.2	3.2	+0.0
q46	7.4	7.2	-0.2
q47	23.8	23.7	-0.0
q48	10.6	11.0	+0.4
q49	15.4	15.8	+0.4
q50	62.9	64.2	+1.3
q51	12.7	12.8	+0.1
q52	1.8	1.8	+0.0
q53	6.6	6.6	+0.1
q54	5.2	5.5	+0.3
q55	1.8	2.0	+0.2
q56	2.7	2.6	-0.1
q57	15.7	15.8	+0.0
q58	3.3	3.1	-0.3
q59	13.7	13.9	+0.2
q60	3.9	3.9	-0.0
q61	3.3	3.6	+0.3
q62	9.1	25.2	+16.2
q63	6.7	6.8	+0.1
q64	77.8	76.6	-1.2
q65	32.5	32.5	+0.0
q66	9.1	8.9	-0.2
q67	151.2	157.3	+6.1
q68	3.4	3.7	+0.3
q69	3.4	3.4	-0.1
q70	10.5	10.5	-0.1
q71	4.2	4.7	+0.5
q72	24.2	24.1	-0.0
q73	2.6	2.6	+0.0
q74	24.3	24.7	+0.3
q75	60.9	62.1	+1.1
q76	51.0	52.2	+1.2
q77	3.0	3.1	+0.0
q78	154.9	159.5	+4.6
q79	7.0	7.0	+0.0
q80	22.3	22.5	+0.2
q81	6.3	6.6	+0.3
q82	24.6	23.7	-0.9
q83	1.7	1.8	+0.1
q84	7.2	7.1	-0.1
q85	9.4	9.3	-0.1
q86	2.9	2.7	-0.2
q87	35.6	34.3	-1.3
q88	24.2	23.4	-0.8
q89	6.9	6.9	+0.0
q90	7.8	7.4	-0.4
q91	1.8	1.8	+0.0
q92	1.5	1.3	-0.2
q93	93.8	97.8	+4.0
q94	28.7	27.6	-1.2
q95	25.0	23.7	-1.3
q96	22.6	22.4	-0.2
q97	37.4	38.0	+0.7
q98	3.3	3.1	-0.2
q99	13.5	14.0	+0.5

Running the benchmark

The cluster runner, the Spark-on-Kubernetes manifests and the image are in the benchmark runner's README. run-matrix.sh writes one JSON-lines result file per run; this page is rendered from them by benchmarks/scripts/render-10tb-page.py. The runs and their memory measurements are discussed in #869.

TPC-DS is a benchmark of the Transaction Processing Performance Council; these results are not audited TPC results and are not comparable to published TPC-DS results. VecRuntime's times are one measured iteration per query. The other engines' totals are their publishers' figures.