Skip to content

feat: Add C ABI for embedded seekdb (multi‑language SDKs) - #674

Open
dengfuping wants to merge 161 commits into
masterfrom
feat/embedded-mode
Open

feat: Add C ABI for embedded seekdb (multi‑language SDKs)#674
dengfuping wants to merge 161 commits into
masterfrom
feat/embedded-mode

Conversation

@dengfuping

@dengfuping dengfuping commented Apr 26, 2026

Copy link
Copy Markdown
Collaborator

Task Description

Solution Description

  • Architecture Design:
Language Bindings (Node.js, Python, Rust, Go)
           │
           ▼
    SeekDB C ABI (seekdb.h/seekdb.cpp)
           │
           ▼
    OceanBase Core (liboceanbase.so)
  • Source Code Directory (src/include/)
src/include/
├── seekdb.h                          # Public C API header file
│                                     # - All function declarations
│                                     # - Type definitions (SeekdbHandle, SeekdbResult, etc.)
│                                     # - Data structures (SeekdbBind, SeekdbField)
│                                     # - MySQL C API compatible interface
│
├── seekdb.cpp                        # C ABI implementation (~4800 lines)
│                                     # - Global initialization (seekdb_open/close)
│                                     # - Connection management
│                                     # - Query execution & Result set operations
│                                     # - Prepared Statement API
│                                     # - Signal handling & Stack management
│
├── CMakeLists.txt                    # Build configuration for libseekdb.so

Test Directory (unittest/include/)

unittest/include/
├── test_seekdb.cpp                   # C++ unit test suite (comprehensive)
│                                     # - Tests all C API functions directly
│                                     # - Complete API coverage
│
├── nodejs/                           # Node.js FFI binding (koffi)
│   ├── seekdb.js                    # FFI wrapper
│   ├── test.js                      # Test suite
│   └── test.sh                      # Test execution script
│
├── nodejs_napi/                      # Node.js N-API binding (native addon)
│   ├── seekdb.cpp                   # N-API C++ implementation
│   ├── test.js                      # Test suite
│   └── test.sh                      # Test execution script
│
├── python/                           # Python binding (ctypes)
│   ├── seekdb.py                    # FFI wrapper
│   ├── test.py                      # Test suite
│   └── test.sh                      # Test execution script
│
├── rust/                             # Rust FFI binding
│   ├── src/
│   │   ├── lib.rs                   # FFI bindings
│   │   └── test.rs                  # Test suite
│   └── test.sh                      # Test execution script
│
└── go/                               # Go binding (cgo)
    ├── seekdb/seekdb.go              # FFI wrapper
    ├── test.go                       # Test suite
    └── test.sh                       # Test execution script

Passed Regressions

  • Unittest
  • Node.js FFI bindings
  • Node.js N-API bindings
  • Python FFI bindings
  • Go FFI bindings
  • Rust FFI bindings

Build and test:

# build libseekdb library
./build.sh release --init -DBUILD_EMBED_MODE=ON --make

# unittest
make -j$(nproc) test_seekdb
cd build_release/unittest/include && ./test_seekdb

# bindings test
cd unittest/include/nodejs && bash ./test.sh
cd unittest/include/nodejs_napi && bash ./test.sh
cd unittest/include/python && bash ./test.sh
cd unittest/include/go && bash ./test.sh
cd unittest/include/rust && bash ./test.sh

Upgrade Compatibility

  • Compatible

Other Information

Release Note

  • 🆕 Add C ABI for embedded seekdb (multi‑language SDKs).

@dengfuping
feat: Add C ABI for embedded seekdb (multi‑language SDKs)
ce1b6df
@dengfuping
refactor(seekdb): remove seekdb_row_free API and align row lifetime w…
01a9b14 
…ith MySQL C API
@dengfuping
fix(build): only create the seekdb library in BUILD_EMBED_MODE
3a47572
@dengfuping
improve(include): libseekdb support macos build
19cf0ec
@dengfuping
fix(build): specify DPYTHON_VERSION to build libseekdb
4b8ca2f
@dengfuping
fix(ci): make seekdb => make libseekdb
d80c2fc
@dengfuping
fix(ci): build cache should work
bbfd6ac
@dengfuping
improve(include): generate macOS export list from seekdb.h and fix te…
58be1aa 
…st scripts for darwin
@dengfuping
fix(embed): VARBINARY_ID param binding and fix parameterized query re…
5e5ea84 
…sult set
@dengfuping
fix(embed): return VECTOR column as JSON string without rounding (e.g…
74feed2 
…. [1.1, 2.2, 3.3])
@dengfuping
improve(ci): use build_debug for libseekdb
ac57a17
@dengfuping
fix(include): remove unimplemented API declarations from seekdb.h
68f3c2f
@dengfuping
fix(embed): suppress "successfully init log writer" by redirecting st…
a4f8d52 
…dout to log file before set_file_name
@dengfuping
feat(embedded): support 100KB document; LOB in-row via session var; f…
be07d23 
…ix MTL_SWITCH namespace; remove §7/doc refs

- C ABI: row_nulls for NULL vs empty; read_lob_data for TEXT/JSON LOB; MTL_SWITCH(OB_SYS_TENANT_ID) for ObLobManager; namespace share alias for MTL_SWITCH
- test: 100KB LONGTEXT via SET SESSION ob_default_lob_inrow_threshold=262144, no table option; special chars metadata test; remove §7 and c-abi-modification-reference references from comments and error messages
@dengfuping
c-api: improve LOB and metadata handling for embedded mode
2968178 
- Add LOB read paths for TEXT/JSON columns in result row serialization
- Use print_sql_literal for ObJsonType cells and expand buffer to 32KB
- Add col_is_metadata heuristic: write "{}" for empty metadata in fallback paths
- Add unit tests: very long document (100KB), special chars in metadata, empty JSON metadata
@dengfuping
feat(package): add libseekdb build script and zip packaging
dca48a3
@dengfuping
feat(ci): Add build-libseekdb.yml
7a53af1
@dengfuping
ci(libseekdb): multi-platform build, S3 upload, CentOS 7/macOS 11+ co…
1bd1cb2 
…mpat, install GTest on macOS
@dengfuping
ci(libseekdb): switch macOS runner from macos-13 to macos-14, update …
fee1009 
…docs
@dengfuping
ci(libseekdb): add git safe.directory in Linux container to fix dubio…
985a640 
…us ownership
@dengfuping
ci(libseekdb): fix make receiving -D options by dropping --make, run …
dae2642 
…make explicitly
@dengfuping
ci(libseekdb): add ccache cache for Linux and macOS builds
dc93cb6
@dengfuping
ci(libseekdb): detect runner Python first, install if not usable; set…
d4223c6 
… OB_CCACHE in workflow
@dengfuping
ci(libseekdb): fix macOS/Linux ccache and Linux wget for manylinux build
5f42517 
- macOS: put ccache in deps devtools/bin so Env.cmake finds it without code change
- Linux: install base deps (incl. wget) first so dep_create.sh can download RPMs
- Linux: try EPEL+ccache; use OB_USE_CCACHE=OFF when ccache not available in container
@dengfuping
ci(libseekdb): update build-libseekdb workflow
2c75910
@dengfuping
ci(libseekdb): install pybind11 via brew for macOS embed build
bd13837
@dengfuping
ci(libseekdb): checkout PR head, restore darwin-x64; Env.cmake fix ma…
512b612 
…cOS cross-build arch

- Checkout PR head ref instead of merge ref to avoid conflict markers in CI
- Restore darwin-x64 job (cross-compile on Apple Silicon for Intel Mac)
- Env.cmake: use CMAKE_OSX_ARCHITECTURES for ARCHITECTURE on macOS x86_64 cross-build
@dengfuping
ci(libseekdb): save deps/ccache cache on failure (if: always())
3be118d 
So next push can restore cache and avoid full rebuild from scratch.
@dengfuping
fix(libseekdb): Linux pack path; disable QPL on macOS
dad7d50 
- workflow: cd to GITHUB_WORKSPACE/package/libseekdb for Linux pack (was relative to build_release)
- zlib_lite: build QPL only on x86_64 Linux (NOT APPLE); macOS deps do not ship qpl
@dengfuping
ci(libseekdb): add macOS Homebrew deps (utf8proc, thrift, re2, brotli…
2288133 
…, bzip2) for darwin-arm64 link
@dengfuping
dengfuping force-pushed the feat/embedded-mode branch from 24dbf76 to a81137f Compare July 28, 2026 06:25
@dengfuping
dengfuping force-pushed the feat/embedded-mode branch from e0defb4 to bd6648c Compare July 31, 2026 03:02
@dengfuping
Merge master into feat/embedded-mode
67bc090 
Resolve conflicts per agreed policy:
- master wins: CMake bazel-inventory mechanism, DML service refactor,
  pretest unittest system, deps/oblib -> src/oblib migration
- HEAD wins: Android/embedded platform support, wix packaging, build.ps1
@dengfuping
fix(rust): commit Cargo.lock so --locked CI passes
587f5bb 
The rust-checks clippy job runs `cargo clippy --locked`, which requires an
existing lockfile; rust/Cargo.lock was never committed, so the job failed
at the resolution step. Commit the generated lockfile (36 packages, cargo
1.97.1) and gitignore rust/target/ build output.
@dengfuping
fix(build): restore seekdb embedded-mode build and public CI
d086188
@dengfuping
Merge remote-tracking branch 'origin/master' into feat/embedded-mode
e4f221f 
# Conflicts:
#	CMakeLists.txt
#	src/oblib/lib/alloc/alloc_func.h
#	src/observer/ob_server.cpp
#	src/share/config/ob_server_config.cpp
#	src/share/parameter/ob_parameter_seed.ipp
@dengfuping
Merge remote-tracking branch 'origin/master' into feat/embedded-mode
bb34552
@dengfuping
fix(log): release leaked MPSC ring entries and self-heal stale busy head
f48b48e 
do_log_message dropped ring entries on formatting failure without
commit/rollback, leaving the busy bit set forever and wedging the flush
thread at the head (continuous alloc_log_item -4013, possible OB_ABORT
from the tail-guard abort). Release the entry on every failure path and
downgrade the tail-guard check from abort to drop. Also let the flush
thread force-release a head entry that stays busy past a timeout, so a
leaked entry can no longer permanently fill the 2MB ring.
@dengfuping
fix(embedded): materialize out-row LOB cells via LOB manager
04f558f 
Long documents (>75KB, out-row LOB) round-tripped as empty strings:
ObObj::get_string() returns the LOB locator binary (leading NUL), and the
embedded build ships a NOT_SUPPORTED stub for ob_obj_read_lob_data, so the
materializer fell through to an empty cell.

Restructure seekdb_materialize_string_cell:
- cells with a LOB header skip the get_string() branch entirely (locator
  binary is not the document text);
- in-row LOB goes through ObObj::read_lob_data (payload in locator);
- out-row LOB falls back to data_plane::read_lob_to_buffer via the LOB
  manager, reading from the read snapshot embedded in the locator.
@dengfuping
fix(embedded): materialize LOB cells whose header flag is unset
4d9e857 
datum2obj<OBJ_DATUM_STRING> copies a storage cell payload into ObObj
without setting has_lob_header, so out-row LOB cells (vector-index
tables build MEM persist locators) reached get_string() and returned
the raw locator binary (leading NUL), truncating the document to an
empty string in the Napi C-string layer.

Detect LOB locator payloads by content (ObLobLocatorV2::is_valid) for
text_tc cells and materialize through read_lob_to_buffer, constructing
the locator with has_lob_header=true since the payload is the full
locator binary. Keep plain in-row text on the get_string path.
@dengfuping
fix(embedded): pre-size output buffer before read_lob_to_buffer
590d9f9 
read_lob_to_buffer writes into a caller-supplied ObString and fails with
OB_ERR_INTERVAL_INVALID when its buffer_size_ is 0 (empty ObString).
Allocate the full LOB byte size from the row allocator and assign_buffer
before reading, matching ObLobManager::get_outrow_lob_full_data.
@dengfuping
fix(embedded): avoid OB_FAIL macro in seekdb_read_outrow_lob
59fcdc3 
OB_FAIL expands to (ret = statement) and this function has no local ret
variable, breaking the remote build. Check get_lob_data_byte_len's return
value directly instead.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dengfuping @hnwyllmm