[SPARK-59023][SQL] Support none mode in spark.sql.ui.explainMode to skip plan description in SQL events

[pan3793]

What changes were proposed in this pull request?

Add a none option to spark.sql.ui.explainMode. When it is set, SparkListenerSQLExecutionStart and SparkListenerSQLAdaptiveExecutionUpdate carry a placeholder string instead of the rendered plan description:

No plan description because spark.sql.ui.explainMode=none

This skips the explain string generation for the SQL UI entirely. The change touches:

• SQLConf.UI_EXPLAIN_MODE: accepts none in addition to simple, extended, codegen, cost, formatted (doc and validation updated)
• SQLExecution: skips QueryExecution.explainString when the mode is none
• AdaptiveSparkPlanExec: same for the plan description posted with adaptive execution updates

Why are the changes needed?

Rendering the plan description is done solely for the SQL UI. For workloads with large plans, or with adaptive execution enabled (where the plan description is re-rendered on every adaptive update), this generation is non-trivial overhead that users who do not use the SQL UI cannot avoid today, since every valid mode renders something. none lets them opt out of the cost.

For example, we have a customer job that constructs a huge plan whose treeString exceeds 280,000 lines. Rendering the plan description takes more than 3 minutes per iteration, and with AQE enabled, assembling the plan tree strings takes more than 40 minutes across the query execution. A string this large also pressures driver memory (we have observed driver OOM), and the browser becomes unresponsive when the SQL UI tries to render it.

Does this PR introduce any user-facing change?

Yes. spark.sql.ui.explainMode now accepts none. When set, the SQL UI shows the placeholder text above instead of the query plan description for SQL executions. The default value (formatted) is unchanged.

How was this patch tested?

The existing control a plan explain mode in listeners via SQLConf tests are extended with a none case, asserting the placeholder is carried by the events:

• AdaptiveQueryExecSuite (covers SparkListenerSQLAdaptiveExecutionUpdate)
• SQLAppStatusListenerSuite (covers SparkListenerSQLExecutionStart)

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Qwen3.8 Max

[dongjoon-hyun]

Thanks for working on this. The direction looks reasonable to me: physicalPlanDescription is generated solely for the SQL UI, there is no way to opt out today, and with AQE it is re-rendered on every plan update, so the cost is real for large plans. The change is small and the default (formatted) is unchanged.

Not adding a NoneMode to ExplainMode and handling none only at the UI conf level looks like the right call, since it keeps df.explain("none") from becoming a valid public API. It would be nice to leave a short comment recording that intent.

I also checked the behavior: UI_EXPLAIN_MODE has .transform(_.toUpperCase(Locale.ROOT)), and createWithDefault("formatted") routes to createWithDefaultString, so the value read back is always upper-cased and the == "NONE" comparison holds. The UI renders fine as well - ExecutionPage.extractInitialAndFinalPlans finds no AQE marker and falls back to the plain <pre> branch.

A few comments, none of them blocking.

1. The placeholder string is duplicated in four places

"No plan description because spark.sql.ui.explainMode=none" is hard-coded in two production files and two test files, and the "NONE" literal appears twice. Changing the wording would require touching all of them. The two call sites also diverge in style (Option + map/getOrElse vs. if/else). How about folding both into one helper in SQLExecution?

object SQLExecution extends Logging {
  private[sql] val NONE_EXPLAIN_MODE = "none"

  private[sql] val NO_PLAN_DESCRIPTION =
    s"No plan description because ${SQLConf.UI_EXPLAIN_MODE.key}=$NONE_EXPLAIN_MODE"

  /**
   * Returns the plan description for SQL UI events, or a placeholder when the UI explain
   * mode is `none`, in which case the (potentially expensive) explain string is not generated.
   */
  private[sql] def planDescription(qe: QueryExecution, uiExplainMode: String): String = {
    if (uiExplainMode.equalsIgnoreCase(NONE_EXPLAIN_MODE)) NO_PLAN_DESCRIPTION
    else qe.explainString(ExplainMode.fromString(uiExplainMode))
  }

Each call site then becomes a single line:

// SQLExecution.scala
val planDesc = planDescription(queryExecution, sparkSession.sessionState.conf.uiExplainMode)

// AdaptiveSparkPlanExec.scala
val planDescription = SQLExecution.planDescription(context.qe, conf.uiExplainMode)

Using equalsIgnoreCase also removes the implicit dependency on the conf's transform(_.toUpperCase). As written, if that transform is ever changed or dropped, none would silently stop matching and ExplainMode.fromString("none") would throw IllegalArgumentException instead.

2. conf.uiExplainMode is read twice in AdaptiveSparkPlanExec

The else branch calls getConf once in the condition and once in the body. The helper above resolves this naturally.

3. Question on how much of the cost is actually avoided

Even with none, SparkPlanInfo.fromSparkPlan is still built, and under AQE it is rebuilt on every update. It calls simpleString(maxFields) plus metadata per node, so for the plan you describe (a treeString over 280,000 lines) it should also contribute meaningfully to driver time and memory. Do you have numbers for the customer job after setting none? Not asking to widen the scope of this PR - just trying to understand whether a follow-up is needed to fully address the OOM you saw.

Minor

• The tests hard-code the placeholder text; if the constant above is introduced, they could reference SQLExecution.NO_PLAN_DESCRIPTION instead.
• There is no SQLConf-level test asserting that none is accepted and that invalid values are still rejected. Optional, and consistent with how most other confs are tested.
• No doc change needed, since the SQL config table is generated from the conf doc, and this is a new option rather than a behavior change.

LGTM otherwise. +1 pending the constant extraction in (1).