Arrow Flight shuffle transport on Graviton4

VecRuntime's columnar shuffle serves each executor's map outputs from an Arrow Flight server; a reduce task opens one DoGet per remote executor and decodes the Arrow IPC stream as it arrives (how it works). This page is every experiment on that transport between two Graviton4 nodes: how many rows per second one executor's reduce tasks can pull from another, and which of the transport's settings move it.

Setup

r8gd.12xlarge (48 vCPU Graviton4, up to 22.5 Gbit/s, 5-core pods)

Server defaults. M rows/s per reduce-side executor; green: at least 15 % above the default (one connection, gRPC window, 5 tasks), red: at least 13 % below.

One server setting changed at a time (5 tasks):

m8g.4xlarge (16 vCPU Graviton4, up to 15 Gbit/s, 13-core pods)

Server defaults. M rows/s per reduce-side executor; green: at least 15 % above the default (one connection, gRPC window, 13 tasks), red: at least 13 % below.

One server setting changed at a time (13 tasks):

What the numbers say

• One connection per remote executor is the limit with the defaults. Every one-connection row sits at about 10 Gbit/s whatever the window, the threads or the number of tasks: EC2 caps a single flow outside a cluster placement group at 5 Gbit/s, and the client reads from two servers. A second connection per server lifts the rate on both instance types; a fourth adds nothing more.
• Past that, the client's decode is the limit on r8gd and the NIC on m8g. At 2.6-2.8 CPU-seconds per GB of zstd stream, five r8gd cores decode about 15 Gbit/s, where the two- and four-connection rows stop; m8g.4xlarge's 13 cores have room, and its rows stop at the instance's 15 Gbit/s.
• Without zstd the client spends a quarter of the CPU per byte, but the stream is about twice the bytes: r8gd reaches its 22 Gbit/s line rate and the most rows per second, m8g fills its NIC with fewer rows than with zstd. Compression stays the default; none pays only where the network has the headroom and the cores do not.
• 1 MB messages cut server CPU per byte by about a third and, on r8gd with four connections, raise the rate where the network is not the bound; with one connection they are slower (more messages through one flow-control window). 16 MB messages, four times the server threads and server backpressure change nothing measurable; backpressure on one connection is slower.
• The flow-control window matters only for a single connection on m8g, where a fixed window holds 10 Gbit/s that gRPC's own sometimes does not; with several connections every window gives the same rate.

TPC-DS 1 TB on 9 x m8g.4xlarge

9 x m8g.4xlarge (8 executors x 13 cores), image of main 51c90ed4 (arm64), no table statistics; legs alternated c1 c2 c4m c4m c2 c1, 2 iterations + 1 warm-up per leg; mean of the two legs' medians, seconds; green / red: 10 % faster / slower than the defaults. Identical results in every leg.

No setting moves the shuffle-heavy queries by 10 %: the six total 144.1, 143.9 and 142.9 s, within 1 %, and the spread between the two legs of one setting (q14a 34.8-38.2 s with the defaults) is larger than the differences between settings. In a cluster every executor serves and reads at once over many connections, so the per-connection cap the two-node benchmark hits is not what holds a stage back; the defaults stay. The two-node numbers above still say where to look on hosts with fewer, larger executors or a faster NIC.

Reproducing

benchmarks/k8s/render-flight-net.sh <image> \
  "partitions=200 maps=16 rowsPerMap=1000000 threads=5" \
  "partitions=200 maps=16 rowsPerMap=1000000 seconds=30 runs=3 channelsPerPeer=1,2,4 clientWindow=0,1m,16m concurrency=5,10" \
  | kubectl apply -f -
kubectl -n bench logs job/flight-net-client     # one JSON line per run

Copyright 2026-2027 Angel Conde and the vecruntime contributors. Measured 2026-10-10/11.