[WIP][POC][GRAPHFRAMES] Add GraphFrames as an in-tree Spark module

[zhengruifeng]

What changes were proposed in this pull request?

This draft upstreams GraphFrames into Apache Spark as an in-tree module. It uses Spark-owned public namespaces:

• JVM: org.apache.spark.graphframes
• Python: pyspark.graphframes

The patch:

• adds the top-level spark-graphframes module and includes its JAR in Spark distributions;
• ports the GraphFrame DataFrame abstraction, motif finding, AggregateMessages, Pregel, and the current GraphFrames algorithm implementations;
• uses Spark's existing spark-graphx module rather than the private GraphFrames GraphX fork;
• removes cross-Spark-version shims because the implementation now builds against the Spark source tree;
• provides classic PySpark wrappers; and
• supports Spark Connect as built-in functionality through a typed Relation.graph_frames protocol field and direct SparkConnectPlanner handling.

Spark Connect does not use Any, a relation extension, or RelationPlugin. The server invokes the same JVM GraphFrames implementations used by classic Spark.

Algorithm coverage includes:

• BFS, motif finding, all paths, cycle detection, and triplets;
• AggregateMessages, Pregel, and multi-hop aggregate-neighbors;
• connected components (two-phase, randomized contraction, and GraphX);
• strongly connected components;
• label propagation (GraphFrames and GraphX) and neighborhood-aware community detection;
• PageRank and parallel personalized PageRank;
• shortest paths;
• exact and approximate triangle count;
• power iteration clustering;
• SVD++;
• maximal independent set;
• k-core decomposition;
• HyperANF; and
• random-walk embeddings and Hash2Vec.

The relational APIs also include graph construction and validation, degree DataFrames, typed degrees, filtering, isolated-vertex removal, reversal, and undirected conversion.

This remains an experimental draft and is not proposed for merge until the community agrees on the SPIP, package and artifact compatibility, maintainership, API stability, documentation, benchmarks, and any required IP-clearance process. The working upstreaming plan is here:
https://docs.google.com/document/d/1emar4QUbrf30jC1Iffv41AVBr-UmRoz2H32VRG3zmAs/edit

Why are the changes needed?

GraphFrames provides a widely used DataFrame graph abstraction, but its external release model requires a separate Spark/Scala compatibility matrix and separately packaged Spark Connect extensions. An in-tree module can provide release-aligned JVM, Python, and built-in Connect APIs while sharing Spark SQL, GraphX, and MLlib internals.

Does this PR introduce any user-facing change?

Yes. It proposes experimental APIs under new Spark-owned namespaces:

import org.apache.spark.graphframes.GraphFrame

val graph = GraphFrame(vertices, edges)
graph.pageRank.maxIter(10).run().vertices.show()

from pyspark.graphframes import GraphFrame

graph = GraphFrame(vertices, edges)
graph.pageRank(maxIter=10).vertices.show()

No released Spark version contains these APIs. Existing org.graphframes and graphframes packages are not changed by this patch.

How was this patch tested?

• build/sbt graphframes/compile connect/compile
• build/sbt graphframes/test (324/325 in the combined run; the sole broadcast-memory failure passed when its suite was rerun in isolation)
• build/sbt 'graphframes/testOnly org.apache.spark.graphframes.lib.MaximalIndependentSetSuite'
• build/sbt assembly/package
• python/run-tests --testnames pyspark.graphframes.tests.test_graphframe
• python/run-tests --testnames pyspark.graphframes.tests.connect.test_parity_graphframe
• python/run-tests --testnames pyspark.graphframes.tests.connect.test_all_algorithms
• ruff check python/pyspark/graphframes
• targeted GraphFrames and Connect Scalafmt
• Connect protobuf regeneration with Spark's pinned generator versions
• license-header audit of every newly added file

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-5)

[zhengruifeng]

Test coverage and GraphX-fork audit

I compared this PR with the current GraphFrames checkout (16a160b) and checked the OSS CI run.

Test coverage

• Every public GraphFrames algorithm exposed by this PR is executed by at least one test.
• The Scala GraphFrames suite passed in OSS CI: 325 succeeded, 0 failed, 0 ignored.
• The dedicated pyspark-graphframes OSS job passed all three registered modules:
  • pyspark.graphframes.tests.test_graphframe
  • pyspark.graphframes.tests.connect.test_parity_graphframe
  • pyspark.graphframes.tests.connect.test_all_algorithms
• Spark precompilation also passed.

This is broad algorithm coverage, but it is not a complete copy of the upstream test tree:

• Core Scala: 28 of 34 upstream test files have counterparts. The missing files are SparkShimsSuite, BeliefPropagationSuite, GraphsSuite, TestLDBCCases, PropertyGraphFrameTest, and KMinSamplingSuite.
• The 18 suites for GraphFrames' private GraphX fork were intentionally not copied because this PR uses Spark's built-in GraphX and its existing tests.
• Python upstream currently has 45 classic test functions plus 14 PropertyGraph tests. This PR has 4 classic test functions plus 10 new grouped Connect test functions.
• KMinSampling has indirect coverage through consumers, but its dedicated unit suite is absent. The Belief Propagation example and PropertyGraph APIs were not imported.
• Therefore, having at least one test per exposed algorithm should not be interpreted as complete backend, parameter, error-path, or upstream test parity.

GraphFrames GraphX fork versus Spark GraphX

The GraphFrames fork is a renamed snapshot of GraphX, not a separate graph execution design: it duplicates 42 production files under org.apache.spark.graphframes.graphx. Its substantive divergences are:

• Label propagation uses vector-based messages and deterministic tie-breaking.
• Several implicit GraphX caches are removed, and Pregel message checkpointing behavior differs.
• Shortest paths adds directed/undirected execution.
• The fork currently lacks Spark's July 2026 SVD++ combiner correction.

This PR deliberately uses canonical org.apache.spark.graphx instead of importing a second GraphX implementation. It preserves undirected shortest paths in the GraphFrames wrapper, does not carry the fork's label-propagation/cache changes, and benefits from Spark's newer SVD++ fix. Any generally useful GraphX changes should be proposed separately against Spark GraphX.

At the time of this check, the GraphFrames Scala and Python jobs were green. The overall workflow was still running, with separate documentation and lint failures in GraphFrames code; no fixes or reruns were made as part of this audit.

[zhengruifeng]

      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 5.0.0.dev0
      /_/

Using Python version 3.13.12 (main, Feb 24 2026 16:13:31)
Client connected to the Spark Connect server at localhost
SparkSession available as 'spark'.

In [1]: from pyspark.graphframes import GraphFrame

In [2]: v = spark.createDataFrame([
   ...:     ("a", "Alice", 34), ("b", "Bob", 36), ("c", "Charlie", 30), ("d", "David", 29),
   ...:     ("e", "Esther", 32), ("f", "Fanny", 36), ("g", "Gabby", 60)
   ...: ], ["id", "name", "age"])
   ...: e = spark.createDataFrame([
   ...:     ("a", "b", "friend"), ("b", "c", "follow"), ("c", "b", "follow"), ("f", "c", "follow"),
   ...:     ("e", "f", "follow"), ("e", "d", "friend"), ("d", "a", "friend"), ("a", "e", "friend")
   ...: ], ["src", "dst", "relationship"])
   ...: g = GraphFrame(v, e)

In [3]: results = g.pageRank(resetProbability=0.15, tol=0.01)
26/08/26 17:32:50 WARN PageRank: Returned DataFrame is persistent and materialized!

In [4]: results.vertices.show()
[*********************************************************-----------------------] 72.40% Complete (0 Tasks running, 0s, Scanned 8.4[*********************************************************-----------------------] 72.40% Complete (0 Tasks running, 0s, Scanned 8.4[*********************************************************-----------------------] 72.40% Complete (0 Tasks running, 0s, Scanned 8.4[*********************************************************-----------------------] 72.40% Complete (0 Tasks running, 0s, Scanned 8.4                                                                                                                                    +---+-------+---+-------------------+
| id|   name|age|           pagerank|
+---+-------+---+-------------------+
|  f|  Fanny| 36| 0.3283606792049851|
|  g|  Gabby| 60| 0.1799821386239711|
|  a|  Alice| 34|0.44910633706538744|
|  e| Esther| 32|0.37085233187676075|
|  d|  David| 29| 0.3283606792049851|
|  b|    Bob| 36|  2.655507832863289|
|  c|Charlie| 30| 2.6878300011606218|
+---+-------+---+-------------------+


In [5]: results.edges.show()
[************************************************************--------------------] 75.00% Complete (0 Tasks running, 0s, Scanned 0.0[************************************************************--------------------] 75.00% Complete (0 Tasks running, 0s, Scanned 0.0[************************************************************--------------------] 75.00% Complete (0 Tasks running, 0s, Scanned 0.0                                                                                                                                    +---+---+------------+------+
|src|dst|relationship|weight|
+---+---+------------+------+
|  a|  b|      friend|   0.5|
|  b|  c|      follow|   1.0|
|  c|  b|      follow|   1.0|
|  f|  c|      follow|   1.0|
|  e|  f|      follow|   0.5|
|  e|  d|      friend|   0.5|
|  d|  a|      friend|   1.0|
|  a|  e|      friend|   0.5|
+---+---+------------+------+


In [6]: spark
Out[6]: <pyspark.sql.connect.session.SparkSession at 0x7e982c56cc20>