Skip to content
Open
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
c5807c8
feat(benchmark): add starcoin-execute-bench with pipeline timing and …
lushengguo Mar 31, 2026
fa2b2ce
feat(timing): add starcoin-pipeline-timing crate for TPS analysis
lushengguo Apr 2, 2026
7e67e9e
feat(bench): add TPS optimization agent loop
lushengguo Apr 2, 2026
9aa73eb
feat(bench): add automated iteration and experimentation modules
lushengguo Apr 2, 2026
dab8194
docs: add TPS optimization knowledge sharing framework
lushengguo Apr 2, 2026
1b2d346
chore: ignore benchmark output files
lushengguo Apr 2, 2026
8ee9f19
refactor: simplify benchmark tool for CI integration
lushengguo Apr 6, 2026
3a263c1
refactor: remove history storage, make benchmark stateless for CI
lushengguo Apr 6, 2026
c56f84c
feat: improve TPS measurement stability for CI
lushengguo Apr 6, 2026
22c3372
fix: reduce default account-count to 4000 and add serde default for b…
lushengguo Apr 6, 2026
0d584fa
feat: implement stable TPS metric for CI
lushengguo Apr 6, 2026
f9831a6
feat: add --rounds parameter for more stable TPS measurement
lushengguo Apr 6, 2026
fcc2427
feat: add STARCOIN_FIXED_BLOCK_TIME env var for deterministic block i…
lushengguo Apr 6, 2026
6c6821a
feat: add --fixed-block-time CLI flag for benchmark
lushengguo Apr 6, 2026
e0ec9e9
fix: use block timestamps for stable TPS calculation
lushengguo Apr 7, 2026
795c041
feat(bench): add --prepare-bench/--load-bench for pre-signed txn pers…
lushengguo Apr 7, 2026
4266644
fix: resolve clippy warnings and fmt issues for CI compliance
lushengguo Apr 9, 2026
6bcff1a
Merge branch 'dual-verse-dag' into dio/tps_optimization_agent_loop
lushengguo Apr 9, 2026
17f020e
fix(bench): correct test data in test_bottleneck_detection
lushengguo Apr 9, 2026
57dee08
fix: address CodeRabbit review comments
lushengguo Apr 9, 2026
70834b7
fix: handle base branch without new bench flags in CI workflow
lushengguo Apr 10, 2026
d6f8a82
add configuration for tps benchmark
lushengguo Apr 13, 2026
6137c53
fix: cargo fmt for pipeline-timing gating code
lushengguo Apr 13, 2026
0454f6b
Merge remote-tracking branch 'origin/dual-verse-dag' into dio/tps_opt…
lushengguo Apr 14, 2026
afd0fb0
fix clippy
lushengguo Apr 15, 2026
cd23005
loose tps benchmark ci
lushengguo Apr 16, 2026
d9d1eaa
Merge remote-tracking branch 'origin/dual-verse-dag' into dio/tps_opt…
lushengguo Apr 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 41 additions & 34 deletions .github/workflows/tps_benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ jobs:
name: TPS Benchmark
needs: choose-runner
runs-on: ${{ fromJSON(needs.choose-runner.outputs.runs_on_json) }}
timeout-minutes: 30
timeout-minutes: 60
steps:
# ── 1. Checkout current branch ──
- name: Checkout current branch
Expand All @@ -83,34 +83,21 @@ jobs:
- name: Setup build environment
run: bash ./scripts/dev_setup.sh -b -t -m

# ── 2. Build & run benchmark on current branch ──
# ── 2. Build benchmark binary on current branch ──
- name: Build benchmark binary (current branch)
run: cargo build --release -p starcoin-execute-bench

- name: Run benchmark (current branch)
run: |
rm -f benchmark_results.json
STARCOIN_FIXED_BLOCK_TIME=1 ./target/release/starcoin-execute-bench \
--simple-transfer \
--fixed-block-time \
--rounds 10 \
--agent-mode \
--pipeline-timing \
2>&1 | tail -20
cp benchmark_results.json bench_current.json
env:
RUST_LOG: error
RUST_BACKTRACE: full

# ── 3. Checkout base branch & run benchmark ──
- name: Run benchmark on base branch (dual-verse-dag)
id: base_bench
# ── 3. Save current branch benchmark artifacts, then switch to base ──
- name: Save current binary and switch to base branch
id: prepare_base
run: |
set -euo pipefail
# Preserve the current-branch binary so the base build can overwrite target/
cp ./target/release/starcoin-execute-bench ./bench_current_bin

BASE_BRANCH="dual-verse-dag"
CURRENT_SHA=$(git rev-parse HEAD)

# Stash any untracked files from the current-branch run
# Stash any untracked files from the current-branch build
git stash --include-untracked || true
Comment on lines +95 to 101

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

bench_current_bin is stashed away before it is used.

At Line 96 you save ./bench_current_bin, then at Line 101 git stash --include-untracked removes it from the working tree. The later run at Line 171 can fail with “No such file or directory”.

Proposed fix
-          cp ./target/release/starcoin-execute-bench ./bench_current_bin
+          cp ./target/release/starcoin-execute-bench "${RUNNER_TEMP}/bench_current_bin"
@@
-          STARCOIN_FIXED_BLOCK_TIME=1 ./bench_current_bin \
+          STARCOIN_FIXED_BLOCK_TIME=1 "${RUNNER_TEMP}/bench_current_bin" \

Also applies to: 171-172

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/tps_benchmark.yml around lines 95 - 101, The script copies
./target/release/starcoin-execute-bench to ./bench_current_bin before running
git stash --include-untracked, which causes bench_current_bin to be removed and
later steps to fail; move the cp ./target/release/starcoin-execute-bench
./bench_current_bin line to after the git stash --include-untracked call (and
apply the same reordering to the second occurrence around lines 171-172) so
bench_current_bin is created after stashing and remains available for later use.


# Checkout base branch
Expand All @@ -121,15 +108,23 @@ jobs:
if [ ! -f sync/starcoin-execute-bench/Cargo.toml ]; then
echo "ℹ️ starcoin-execute-bench not found on $BASE_BRANCH — skipping base benchmark."
echo "has_base=false" >> "$GITHUB_OUTPUT"
git checkout "$CURRENT_SHA"
git stash pop || true
exit 0
fi

echo "has_base=true" >> "$GITHUB_OUTPUT"

# Build on base
cargo build --release -p starcoin-execute-bench
# ── 4. Build benchmark binary on base branch (reuses cargo cache) ──
- name: Build benchmark binary (base branch)
if: steps.prepare_base.outputs.has_base == 'true'
run: cargo build --release -p starcoin-execute-bench

# ── 5. Run benchmark on base branch first (we are already on it) ──
- name: Run benchmark (base branch)
id: base_bench
if: steps.prepare_base.outputs.has_base == 'true'
timeout-minutes: 10
run: |
set -euo pipefail

# Detect which flags the base binary supports
HELP_TEXT=$(./target/release/starcoin-execute-bench --help 2>&1 || true)
Expand All @@ -152,28 +147,40 @@ jobs:

echo "ℹ️ Base bench args: $BASE_ARGS"

# Run on base
rm -f benchmark_results.json
STARCOIN_FIXED_BLOCK_TIME=1 ./target/release/starcoin-execute-bench \
$BASE_ARGS \
2>&1 | tail -20

# The base bench may not produce benchmark_results.json (older versions)
if [ -f benchmark_results.json ]; then
cp benchmark_results.json bench_base.json
echo "has_base=true" >> "$GITHUB_OUTPUT"
else
echo "⚠️ Base bench did not produce benchmark_results.json — skipping comparison."
echo "has_base=false" >> "$GITHUB_OUTPUT"
fi
env:
RUST_LOG: error
RUST_BACKTRACE: full

# Return to current branch
git checkout "$CURRENT_SHA"
git stash pop || true
# ── 6. Run benchmark on current branch (using saved binary) ──
- name: Run benchmark (current branch)
timeout-minutes: 10
run: |
rm -f benchmark_results.json
STARCOIN_FIXED_BLOCK_TIME=1 ./bench_current_bin \
--simple-transfer \
--fixed-block-time \
--rounds 10 \
--agent-mode \
--pipeline-timing \
2>&1 | tail -20
cp benchmark_results.json bench_current.json
env:
RUST_LOG: error
RUST_BACKTRACE: full

# ── 4. Compare results ──
# ── 7. Compare results ──
- name: Compare benchmark results
id: compare
run: |
Expand Down Expand Up @@ -206,7 +213,7 @@ jobs:

echo "has_report=true" >> "$GITHUB_OUTPUT"

# ── 5. Upload artifacts ──
# ── 8. Upload artifacts ──
- name: Upload benchmark artifacts
if: always()
uses: actions/upload-artifact@v4
Expand All @@ -218,7 +225,7 @@ jobs:
bench_report.md
retention-days: 90

# ── 6. Post report as PR comment ──
# ── 9. Post report as PR comment ──
- name: Post benchmark report to PR
if: always() && github.event_name == 'pull_request' && steps.compare.outputs.has_report == 'true'
uses: marocchino/sticky-pull-request-comment@v2
Expand Down
Loading