Dev Weekly · Diagram Design

Spark Shuffle 数据重分布

Spark shuffle boundaryInput partitions are processed locally, shuffled across executors by key, then aggregated in downstream partitions.WHY GROUP BY AND JOIN CAN BE EXPENSIVEInput partitionsShuffle by keyDownstream partitionsPartition 0A · B · A · CPartition 1B · C · A · BPartition 2C · A · C · BNetwork + disk exchangePartition records by keySort / hash / spillSerialize and fetch blocksStage boundaryDownstream tasks can startKey bucket AA · A · A · AKey bucket BB · B · B · BKey bucket CC · C · C · CA hot key can overload one bucket: skew creates long-tail tasks even when average utilization looks low.

Diagram is provided as standalone SVG for embedding and this HTML page for full-size viewing.