parser: close binlog-relevant statement gaps (XA, tablespace, ALTER VIEW, SRS, admin)

.github/workflows/parser-regen.yml

@@ -0,0 +1,27 @@
+name: parser-regen
+on:
+  push:
+    tags:
+      - v*
+    branches:
+      - master
+      - main
+  pull_request:
+permissions:
+  contents: read
+jobs:
+  regen:
+    name: generated parser matches grammar
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/setup-go@v7.0.0
+        with:
+          go-version: "1.26.6"
+      - name: Checkout code
+        uses: actions/checkout@v7
+      - name: Regenerate parser from grammar
+        run: |
+          rm -f pkg/parser/parser.go pkg/parser/hintparser.go
+          make -C pkg/parser
+      - name: Verify generated files match the checked-in ones
+        run: git diff --exit-code

.gitignore

@@ -23,3 +23,7 @@ lint-error.logs
 .goose
 
 bin/
+genkeyword
+bin/
+y.output
+pkg/parser/goyacc/goyacc

AGENTS.md

@@ -165,7 +165,7 @@ pkg/
   table/      → Chunking strategies (optimistic, composite, multi)
   checksum/   → Post-copy data verification (CRC32 + BIT_XOR)
   dbconn/     → MySQL connection management, TLS, retries, locking, kill logic
-  statement/  → SQL parsing via TiDB parser (ALTER, CREATE, DROP, RENAME)
+  statement/  → SQL parsing via pkg/parser (ALTER, CREATE, DROP, RENAME)
   lint/       → Static analysis framework for schemas and DDL (built-in linters)
   fmt/        → Schema file formatter (canonicalize CREATE TABLE .sql files)
   throttler/  → Rate limiting interface (noop, mock, replica-lag based)
@@ -224,9 +224,9 @@ Three chunker implementations:
 - **MultiChunker** — wraps multiple child chunkers for multi-table operations
 
 ### `pkg/statement`
-Uses the [TiDB parser](https://github.com/pingcap/tidb/tree/master/pkg/parser) for SQL parsing. If a DDL cannot be parsed by TiDB, Spirit cannot execute it. `create_table.go` provides structured `CREATE TABLE` parsing (the `CreateTable` struct and its parse/diff methods).
+Uses [pkg/parser](pkg/parser/README.md) (Spirit's MySQL-only fork of the TiDB parser) for SQL parsing. If a DDL cannot be parsed, Spirit cannot execute it. `create_table.go` provides structured `CREATE TABLE` parsing (the `CreateTable` struct and its parse/diff methods).
 
-**Normalization pipeline:** MySQL rewrites many constructs when it stores a table (inline `PRIMARY KEY`/`UNIQUE` → table-level, column `CHECK` hoisted to table-level, `int(11)` → `int`, the legacy `BINARY` attribute → a `_bin` collation). To stop a hand-written schema from diffing spuriously against a live `SHOW CREATE TABLE`, `ParseCreateTable` runs a registry of **normalization rules** over the parsed `CreateTable` before returning it. Each rule is a `Normalizer` (`normalize.go`) that self-registers via `init()` in its own `normalize_*.go` file and rewrites the struct's fields in place (never `Raw`). Rules run after the struct is fully parsed, so they are order-independent. Consequence: `CreateTable.Diff` **assumes normalized input**. The TiDB parser already folds most type *aliases* (`BOOL`→`tinyint(1)`, `SERIAL`→`bigint unsigned … UNIQUE`, `INTEGER`→`int`), so rules only handle what the parser leaves alone. See `pkg/statement/README.md` for the full concept and rule list.
+**Normalization pipeline:** MySQL rewrites many constructs when it stores a table (inline `PRIMARY KEY`/`UNIQUE` → table-level, column `CHECK` hoisted to table-level, `int(11)` → `int`, the legacy `BINARY` attribute → a `_bin` collation). To stop a hand-written schema from diffing spuriously against a live `SHOW CREATE TABLE`, `ParseCreateTable` runs a registry of **normalization rules** over the parsed `CreateTable` before returning it. Each rule is a `Normalizer` (`normalize.go`) that self-registers via `init()` in its own `normalize_*.go` file and rewrites the struct's fields in place (never `Raw`). Rules run after the struct is fully parsed, so they are order-independent. Consequence: `CreateTable.Diff` **assumes normalized input**. The parser already folds most type *aliases* (`BOOL`→`tinyint(1)`, `SERIAL`→`bigint unsigned … UNIQUE`, `INTEGER`→`int`), so rules only handle what the parser leaves alone. See `pkg/statement/README.md` for the full concept and rule list.
 
 ### `pkg/lint`
 Built-in linters auto-register via `init()`. Each linter is in its own file (`lint_<name>.go`). To add a new linter, create a new file following the existing pattern and implement the `Linter` interface from `linter.go`.
@@ -308,8 +308,8 @@ Normalization canonicalizes a parsed `CreateTable` so a user-written schema matc
 3. Mutate the **structured** fields of `CreateTable` (`Columns`, `Indexes`, …) and return the same instance — never touch `Raw`
 4. Keep the rule order-independent (it runs after the struct is fully parsed) and follow an existing rule (e.g., `normalize_integer_display_width.go`)
 
-### Working with the TiDB parser
-All SQL parsing goes through `pkg/statement/`. Do not parse SQL manually. The `Statement` type wraps parsed DDL and provides safety analysis methods.
+### Working with the parser
+All SQL parsing goes through `pkg/statement/` (built on `pkg/parser`, Spirit's fork of the TiDB parser). Do not parse SQL manually. The `Statement` type wraps parsed DDL and provides safety analysis methods.
 
 ### Database connections
 Always use `pkg/dbconn` for MySQL connections. Never create raw `sql.Open()` calls in production code (test utilities are the exception). The `DBConn` type handles retries, TLS, and connection pooling.

README.md

@@ -147,7 +147,7 @@ Writing a new data migration tool is scary, since bugs have real consequences (d
 
 We have also tried to balance making Spirit _as fast as possible_ while still being safe to run on production systems that are running existing workloads. Sometimes this means spirit might venture into creating slow downs in application performance. If it does, please file an issue and help us make improvements.
 
-We make extensive use of the TiDB parser. If a DDL statement can not be parsed by TiDB, it will not be possible to execute it. Usually this is not a problem, but there can be [edge-cases](https://github.com/pingcap/tidb/issues/54700).
+We make extensive use of a SQL parser (see [pkg/parser](pkg/parser/README.md), a MySQL-only fork of the TiDB parser). If a DDL statement cannot be parsed, it will not be possible to execute it. Usually this is not a problem, but there can be edge-cases with unusual syntax.
 
 ## Development
 

go.mod

@@ -8,11 +8,10 @@ require (
 	github.com/go-mysql-org/go-mysql v1.16.1-0.20260731133054-6f853f178dc3
 	github.com/go-sql-driver/mysql v1.10.0
 	github.com/google/uuid v1.6.0
-	github.com/pingcap/errors v0.11.5-0.20260310054046-9c8b3586e4b2
-	github.com/pingcap/tidb/pkg/parser v0.0.0-20260504140133-511dba1dbe17
 	github.com/stretchr/testify v1.11.1
 	go.uber.org/goleak v1.3.0
 	golang.org/x/sync v0.22.0
+	golang.org/x/text v0.36.0
 )
 
 require (
@@ -21,17 +20,15 @@ require (
 	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
 	github.com/goccy/go-json v0.10.6 // indirect
 	github.com/klauspost/compress v1.18.6 // indirect
-	github.com/pingcap/failpoint v0.0.0-20260406204437-bbc9d102c19e // indirect
+	github.com/pingcap/errors v0.11.5-0.20260310054046-9c8b3586e4b2 // indirect
 	github.com/pingcap/log v1.1.1-0.20260227082333-572e590d08f1 // indirect
+	github.com/pingcap/tidb/pkg/parser v0.0.0-20260504140133-511dba1dbe17 // indirect
 	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
 	github.com/shopspring/decimal v1.4.0 // indirect
 	go.uber.org/atomic v1.11.0 // indirect
 	go.uber.org/multierr v1.11.0 // indirect
 	go.uber.org/zap v1.28.0 // indirect
-	golang.org/x/text v0.36.0 // indirect
 	gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
 	gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 )
-
-replace github.com/pingcap/tidb/pkg/parser => github.com/block/tidb/pkg/parser v0.0.0-20260506200501-e528fd979fc8

go.sum

@@ -6,8 +6,6 @@ github.com/alecthomas/kong v1.16.0 h1:g92/kUxBcdcTPOM79yE63viJgtcp5dNyrB3/O2cjYT
 github.com/alecthomas/kong v1.16.0/go.mod h1:wrlbXem1CWqUV5Vbmss5ISYhsVPkBb1Yo7YKJghju2I=
 github.com/alecthomas/repr v0.5.2 h1:SU73FTI9D1P5UNtvseffFSGmdNci/O6RsqzeXJtP0Qs=
 github.com/alecthomas/repr v0.5.2/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
-github.com/block/tidb/pkg/parser v0.0.0-20260506200501-e528fd979fc8 h1:+OfdTacrEyjlqcRUpBFX9uJ6ROBq6cUjwY4DClhnsdU=
-github.com/block/tidb/pkg/parser v0.0.0-20260506200501-e528fd979fc8/go.mod h1:zDLDsfNBU5+L6T4J9/OgWAHc/WZvMUjbpgHqQ/t3yKo=
 github.com/coreos/go-semver v0.3.1 h1:yi21YpKnrx1gt5R+la8n5WgS0kCrsPp33dmEyHReZr4=
 github.com/coreos/go-semver v0.3.1/go.mod h1:irMmmIw/7yzSRPWryHsK7EYSg09caPQL03VsM8rvUec=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
@@ -36,10 +34,10 @@ github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
 github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
 github.com/pingcap/errors v0.11.5-0.20260310054046-9c8b3586e4b2 h1:cLgCk5mwDG9lDH+dPK8TmEliTjyGJwwKN0qevWAl8IY=
 github.com/pingcap/errors v0.11.5-0.20260310054046-9c8b3586e4b2/go.mod h1:ktAJCA9lxrHHjVyVl2pKJFvzBnq2eZbb+CUOjBRPlXo=
-github.com/pingcap/failpoint v0.0.0-20260406204437-bbc9d102c19e h1:il8go9El5o10EyPmalSG6Lg3zu2rtkq7c2wbRwBmdwo=
-github.com/pingcap/failpoint v0.0.0-20260406204437-bbc9d102c19e/go.mod h1:jimwlLpI/XtwQdlZML15HS+j4rirvwZM0GLY07wwgOo=
 github.com/pingcap/log v1.1.1-0.20260227082333-572e590d08f1 h1:A2bEfgSb7hLwR9mxDszgGKweF+xY9YoTDG+8RjdFjDE=
 github.com/pingcap/log v1.1.1-0.20260227082333-572e590d08f1/go.mod h1:pxfz2oJfAuhwrb3/rcLqD//GS/5gRP4gD022iP3cEO0=
+github.com/pingcap/tidb/pkg/parser v0.0.0-20260504140133-511dba1dbe17 h1:cfAVPis6GP6lxQgm1WGaNGi4rVXTB4KDvYf96LjqRCM=
+github.com/pingcap/tidb/pkg/parser v0.0.0-20260504140133-511dba1dbe17/go.mod h1:zDLDsfNBU5+L6T4J9/OgWAHc/WZvMUjbpgHqQ/t3yKo=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=

pkg/change/utils.go

@@ -5,12 +5,11 @@ import (
 	"database/sql"
 	"fmt"
 
+	"github.com/block/spirit/pkg/parser"
+	"github.com/block/spirit/pkg/parser/ast"
 	"github.com/block/spirit/pkg/table"
 	"github.com/block/spirit/pkg/utils"
 	"github.com/go-mysql-org/go-mysql/replication"
-	"github.com/pingcap/tidb/pkg/parser"
-	"github.com/pingcap/tidb/pkg/parser/ast"
-	_ "github.com/pingcap/tidb/pkg/parser/test_driver"
 )
 
 func encodeSchemaTable(schema, table string) string {

pkg/dbconn/sqlescape/utils.go

@@ -17,13 +17,12 @@ package sqlescape
 
 import (
 	"encoding/json"
+	"fmt"
 	"io"
 	"reflect"
 	"strconv"
 	"strings"
 	"time"
-
-	"github.com/pingcap/errors"
 )
 
 func reserveBuffer(buf []byte, appendSize int) []byte {
@@ -143,20 +142,20 @@ func escapeSQL(sql string, args ...any) ([]byte, error) {
 		switch ch {
 		case 'n':
 			if argPos >= len(args) {
-				return nil, errors.Errorf("missing arguments, need %d-th arg, but only got %d args", argPos+1, len(args))
+				return nil, fmt.Errorf("missing arguments, need %d-th arg, but only got %d args", argPos+1, len(args))
 			}
 			arg := args[argPos]
 			argPos++
 
 			v, ok := arg.(string)
 			if !ok {
-				return nil, errors.Errorf("expect a string identifier, got %v", arg)
+				return nil, fmt.Errorf("expect a string identifier, got %v", arg)
 			}
 			buf = appendEscapedIdentifier(buf, v)
 			i++ // skip specifier
 		case '?':
 			if argPos >= len(args) {
-				return nil, errors.Errorf("missing arguments, need %d-th arg, but only got %d args", argPos+1, len(args))
+				return nil, fmt.Errorf("missing arguments, need %d-th arg, but only got %d args", argPos+1, len(args))
 			}
 			arg := args[argPos]
 			argPos++
@@ -254,14 +253,14 @@ func escapeSQL(sql string, args ...any) ([]byte, error) {
 					case reflect.String:
 						buf = appendSQLArgString(buf, reflect.ValueOf(arg).String())
 					default:
-						return nil, errors.Errorf("unsupported %d-th argument: %v", argPos, arg)
+						return nil, fmt.Errorf("unsupported %d-th argument: %v", argPos, arg)
 					}
 				}
 			}
 			i++ // skip specifier
 		case 'r':
 			if argPos >= len(args) {
-				return nil, errors.Errorf("missing arguments, need %d-th arg, but only got %d args", argPos+1, len(args))
+				return nil, fmt.Errorf("missing arguments, need %d-th arg, but only got %d args", argPos+1, len(args))
 			}
 			arg := args[argPos]
 			argPos++
@@ -271,7 +270,7 @@ func escapeSQL(sql string, args ...any) ([]byte, error) {
 				// A plain string is rejected on purpose: the RawSQL conversion
 				// at the call site is the explicit, greppable assertion that
 				// this text is safe to splice verbatim.
-				return nil, errors.Errorf("expect sqlescape.RawSQL for %%r, got %T", arg)
+				return nil, fmt.Errorf("expect sqlescape.RawSQL for %%r, got %T", arg)
 			}
 			// Spliced verbatim: buf is never re-scanned, so %-sequences inside
 			// v (e.g. COMMENT '100%new') are data, not format specifiers.

pkg/fmt/fmt.go

@@ -23,9 +23,9 @@ import (
 	"path/filepath"
 	"strings"
 
+	"github.com/block/spirit/pkg/parser/ast"
+	"github.com/block/spirit/pkg/parser/format"
 	_ "github.com/go-sql-driver/mysql"
-	"github.com/pingcap/tidb/pkg/parser/ast"
-	"github.com/pingcap/tidb/pkg/parser/format"
 
 	"github.com/block/spirit/pkg/dbconn/sqlescape"
 	"github.com/block/spirit/pkg/statement"

pkg/lint/lint.go

@@ -62,8 +62,8 @@ import (
 	"maps"
 	"os"
 
+	"github.com/block/spirit/pkg/parser/ast"
 	"github.com/block/spirit/pkg/statement"
-	"github.com/pingcap/tidb/pkg/parser/ast"
 )
 
 // Config holds linter configuration
@@ -332,8 +332,6 @@ func AlterTableTypeToString(tp ast.AlterTableType) string {
 		return "ALTER INDEX INVISIBLE"
 	case ast.AlterTableOrderByColumns:
 		return "ORDER BY"
-	case ast.AlterTableSetTiFlashReplica:
-		return "SET TIFLASH REPLICA"
 	default:
 		return fmt.Sprintf("ALTER TABLE (type %d)", tp)
 	}

pkg/lint/lint_allow_charset.go

@@ -5,8 +5,8 @@ import (
 	"slices"
 	"strings"
 
+	"github.com/block/spirit/pkg/parser/ast"
 	"github.com/block/spirit/pkg/statement"
-	"github.com/pingcap/tidb/pkg/parser/ast"
 )
 
 func init() {

pkg/lint/lint_auto_inc_capacity.go

@@ -6,8 +6,8 @@ import (
 	"strconv"
 	"strings"
 
+	"github.com/block/spirit/pkg/parser/mysql"
 	"github.com/block/spirit/pkg/statement"
-	"github.com/pingcap/tidb/pkg/parser/mysql"
 )
 
 func init() {

pkg/lint/lint_datetime_index_position.go

@@ -4,8 +4,8 @@ import (
 	"fmt"
 	"strings"
 
+	"github.com/block/spirit/pkg/parser/mysql"
 	"github.com/block/spirit/pkg/statement"
-	"github.com/pingcap/tidb/pkg/parser/mysql"
 )
 
 type DatetimeIndexPositionLinter struct{}

pkg/lint/lint_has_float.go

@@ -4,8 +4,8 @@ import (
 	"fmt"
 	"strings"
 
+	"github.com/block/spirit/pkg/parser/mysql"
 	"github.com/block/spirit/pkg/statement"
-	"github.com/pingcap/tidb/pkg/parser/mysql"
 )
 
 type HasFloatLinter struct{}

pkg/lint/lint_has_timestamp.go

@@ -4,8 +4,8 @@ import (
 	"fmt"
 	"strings"
 
+	"github.com/block/spirit/pkg/parser/mysql"
 	"github.com/block/spirit/pkg/statement"
-	"github.com/pingcap/tidb/pkg/parser/mysql"
 )
 
 type HasTimestampLinter struct{}

pkg/lint/lint_index_visibility.go

@@ -5,8 +5,8 @@ import (
 	"slices"
 	"strings"
 
+	"github.com/block/spirit/pkg/parser/ast"
 	"github.com/block/spirit/pkg/statement"
-	"github.com/pingcap/tidb/pkg/parser/ast"
 )
 
 func init() {

pkg/lint/lint_invisible_index.go

@@ -4,8 +4,8 @@ import (
 	"fmt"
 	"strings"
 
+	"github.com/block/spirit/pkg/parser/ast"
 	"github.com/block/spirit/pkg/statement"
-	"github.com/pingcap/tidb/pkg/parser/ast"
 )
 
 func init() {

pkg/lint/lint_primary_key_type.go

@@ -6,8 +6,8 @@ import (
 	"slices"
 	"strings"
 
+	"github.com/block/spirit/pkg/parser/mysql"
 	"github.com/block/spirit/pkg/statement"
-	"github.com/pingcap/tidb/pkg/parser/mysql"
 )
 
 func init() {

pkg/lint/lint_rename_column.go

@@ -3,8 +3,8 @@ package lint
 import (
 	"fmt"
 
+	"github.com/block/spirit/pkg/parser/ast"
 	"github.com/block/spirit/pkg/statement"
-	"github.com/pingcap/tidb/pkg/parser/ast"
 )
 
 // RenameColumnLinter detects column renames in ALTER TABLE statements.

pkg/lint/lint_unsafe.go

@@ -4,8 +4,8 @@ import (
 	"fmt"
 
 	"github.com/block/spirit/pkg/dbconn/sqlescape"
+	"github.com/block/spirit/pkg/parser/ast"
 	"github.com/block/spirit/pkg/statement"
-	"github.com/pingcap/tidb/pkg/parser/ast"
 )
 
 type UnsafeLinter struct {

pkg/lint/lint_zero_date.go

@@ -4,10 +4,10 @@ import (
 	"fmt"
 	"strings"
 
+	"github.com/block/spirit/pkg/parser/ast"
+	"github.com/block/spirit/pkg/parser/format"
+	"github.com/block/spirit/pkg/parser/mysql"
 	"github.com/block/spirit/pkg/statement"
-	"github.com/pingcap/tidb/pkg/parser/ast"
-	"github.com/pingcap/tidb/pkg/parser/format"
-	"github.com/pingcap/tidb/pkg/parser/mysql"
 )
 
 type ZeroDateLinter struct{}
@@ -64,10 +64,20 @@ func (l *ZeroDateLinter) checkColumnZeroDate(column *ast.ColumnDef, tableName st
 			nullable = true
 		case ast.ColumnOptionDefaultValue:
 			if option.Expr != nil {
+				// Unwrap the expression-default parentheses so
+				// DEFAULT ('0000-00-00') is inspected like DEFAULT '0000-00-00'.
+				expr := option.Expr
+				for {
+					paren, ok := expr.(*ast.ParenthesesExpr)
+					if !ok {
+						break
+					}
+					expr = paren.Expr
+				}
 				// Extract default value from expression using Restore
 				var sb strings.Builder
 				rCtx := format.NewRestoreCtx(format.DefaultRestoreFlags|format.RestoreStringWithoutCharset, &sb)
-				err := option.Expr.Restore(rCtx)
+				err := expr.Restore(rCtx)
 				if err == nil {
 					val := sb.String()
 					// Remove surrounding quotes if present for string literals

pkg/lint/post_state.go

@@ -5,9 +5,9 @@ import (
 	"slices"
 	"strings"
 
+	"github.com/block/spirit/pkg/parser/ast"
+	"github.com/block/spirit/pkg/parser/types"
 	"github.com/block/spirit/pkg/statement"
-	"github.com/pingcap/tidb/pkg/parser/ast"
-	"github.com/pingcap/tidb/pkg/parser/types"
 )
 
 // PostState returns a deterministic post-state view of the schema: the existing