Dev Weekly · Diagram Design

Spark DataFrame 惰性执行与任务生成

Spark lazy evaluationDataFrame transformations build a logical plan; an action triggers optimization, physical planning, and distributed tasks.FROM PYTHON EXPRESSIONS TO CLUSTER TASKSReadspark.read.parquet()Transformfilter · select · joinTransformgroupBy · aggActionshow() · count() · write()Execution starts hereCatalyst optimizationLogical → optimized → physical planJob → Stages → TasksTasks run on executor partitionsBefore an ActionSpark builds a plan; it does not eagerly materialize every intermediate DataFrame.This allows the optimizer to combine filters, prune columns, and choose operators.Repeated Actions may re-run the plan unless useful results are persisted.

Diagram is provided as standalone SVG for embedding and this HTML page for full-size viewing.