Dev Weekly · Diagram Design

Spark 集群执行架构

Spark cluster execution architectureDriver coordinates Spark jobs and schedules tasks to executors across worker nodes, which read partitions from storage and write results.APACHE SPARK · DISTRIBUTED RUNTIMEApplicationPySpark / SQLDataFrame planActions trigger jobsDriverSparkSession · DAG SchedulerPlans Jobs → Stages → TasksCoordinates executorsCluster ManagerKubernetes · YARN · StandaloneAllocates executor resourcesNot the SQL query optimizerStorage / SourcesS3 / ADLS / GCSParquet / Iceberg / DeltaKafka / JDBC / HiveData is split into partitionsWORKER NODESExecutor ATask 1Task 2Cache · Read · ComputeExecutor BTask 3Task 4Shuffle · AggregateExecutor CTask 5Task 6Write · Return statusReadOne task processes one partition at a time; executors run many tasks concurrently.

Diagram is provided as standalone SVG for embedding and this HTML page for full-size viewing.