Skip to content
Open
Show file tree
Hide file tree
Changes from 65 commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
e811689
Capture kubectl describe pod for stuck/timed-out FIO pods
RaunakJalan Aug 1, 2026
240cb4c
Fix checksum pod race and stale VolumeSnapshot between tests
RaunakJalan Aug 1, 2026
fbe9743
Improve API parity audit: fix pool ordering, severity, and diagnostics
RaunakJalan Aug 1, 2026
ed14b7e
Add complete K8s upgrade operations guide (R25 Helm to R26+ Operator)
RaunakJalan Aug 1, 2026
62176b9
Fix backup test bugs and exclude topology tests from backup pipeline
RaunakJalan Aug 2, 2026
7d72054
Fix new_worker_nodes description: at least 1, not 2
RaunakJalan Aug 2, 2026
73c2417
Align topology suite defaults with k8s-native workflows
RaunakJalan Aug 2, 2026
dde590e
Add backup config support to K8s topology workflows
RaunakJalan Aug 2, 2026
e7f14fa
Fix 25-input limit for workflow_dispatch in migration workflows
RaunakJalan Aug 2, 2026
5009897
Fix TLS propagation: use truthy check instead of string comparison
RaunakJalan Aug 2, 2026
ea4c608
Don't label new_worker_nodes at bootstrap time
RaunakJalan Aug 2, 2026
c2dab1f
Remove simplyblock label from new_worker_nodes during cleanup
RaunakJalan Aug 2, 2026
860ab0e
Clean stale /etc/simplyblock config, add sequential node expansion, d…
RaunakJalan Aug 2, 2026
90f05d0
Pass driveSizeRange and pcieModel from StorageNodeSet to StorageNode CR
RaunakJalan Aug 2, 2026
527582a
Wait for per-node-config ConfigMap before deleting stale pods
RaunakJalan Aug 2, 2026
703a271
Fix topology suite summary parsing and stale pod handling
RaunakJalan Aug 3, 2026
35a9034
Move ConfigMap wait and stale pod deletion before StorageNodeOps crea…
RaunakJalan Aug 3, 2026
b6832db
Add MinIO trace logging to backup test workflows, align upgrade pipel…
RaunakJalan Aug 3, 2026
7add04a
Remove stale storagenodeset labels during cleanup in all K8s workflows
RaunakJalan Aug 3, 2026
a8d27d4
Fix TARGET_DOCKER_IMAGE default from main-latest to main
RaunakJalan Aug 3, 2026
1f588ea
Add target worker to StorageNodeSet before migration
RaunakJalan Aug 3, 2026
36d8716
Revert "Add target worker to StorageNodeSet before migration"
RaunakJalan Aug 3, 2026
7d4a0ad
Fix MinIO trace setup: use secret keys and resilient mc install
RaunakJalan Aug 4, 2026
2f6bfc2
Add preserve_resources_on_failure to K8s topology pipelines
RaunakJalan Aug 4, 2026
4295d31
Fix 25-input limit for node-migration workflow_dispatch
RaunakJalan Aug 4, 2026
6452ec0
Add cleanup-simplyblock.sh to all K8s workflow cleanup steps
RaunakJalan Aug 4, 2026
bd367d3
Remove hardcoded pcieModel from K8s StorageNodeSet specs
RaunakJalan Aug 4, 2026
f7b23be
Make pcieModel conditional: skip for openshift-local and openshift-ba…
RaunakJalan Aug 4, 2026
393764f
Stop upgrade test runner after first test failure
RaunakJalan Aug 4, 2026
fd58692
Pass --spdk-proxy-image on sn restart during upgrade tests
RaunakJalan Aug 4, 2026
ff7ad26
Add dual-node-per-host test classes and fix K8s upgrade workflow
RaunakJalan Aug 5, 2026
3776dd3
Rename operator_repo_branches to helm_repo_branches in k8s upgrade wo…
RaunakJalan Aug 5, 2026
4fb9859
Fix KUBECONFIG setup: try existing kubeconfig before falling back to …
RaunakJalan Aug 5, 2026
4a3bb82
Use per-environment kubeconfig secrets matching other K8s native pipe…
RaunakJalan Aug 5, 2026
e3ed8dc
Add helm dependency build before R25 chart installs
RaunakJalan Aug 5, 2026
713b076
Add old deployment cleanup to k8s upgrade workflow
RaunakJalan Aug 5, 2026
5776414
Add set -euxo pipefail to all K8s helm install steps for debug output
RaunakJalan Aug 5, 2026
0390e79
Fix R25 admin pod detection: exclude ingress controller from grep
RaunakJalan Aug 5, 2026
51f7559
Parallelize hugepages reset with 120s timeout to prevent cleanup hang
RaunakJalan Aug 5, 2026
bc1a64c
Rewrite cleanup to force-delete everything with no graceful waits
RaunakJalan Aug 5, 2026
8b92601
Fix cleanup: strip PV/PVC finalizers before namespace delete, reduce …
RaunakJalan Aug 5, 2026
cdb8d02
Disable ingress-nginx admission webhook for R25 sbcli chart install
RaunakJalan Aug 5, 2026
a50adbb
Fix cleanup Phase 8: use namespace finalize API instead of api-resour…
RaunakJalan Aug 5, 2026
01910f3
Fix cluster secret parsing: use 'cluster get-secret' instead of awk o…
RaunakJalan Aug 5, 2026
f168066
Fix R25 spdk-csi chart path: v0.2.4 uses charts/ not csi-driver/charts/
RaunakJalan Aug 5, 2026
b646657
Improve cleanup: force-delete resources in Phase 4, finalize namespac…
RaunakJalan Aug 5, 2026
9ca4bcf
Set storagenode.coresPercentage=50 for R25 spdk-csi chart install
RaunakJalan Aug 5, 2026
99bb1d3
Fix R25 pool creation: use direct CLI instead of Pool CRD
RaunakJalan Aug 5, 2026
1bc5b76
Pass cluster params to R25 cluster create and add cluster activate
RaunakJalan Aug 5, 2026
fc873c5
Delete stale StorageClasses and VolumeSnapshotClasses during cleanup
RaunakJalan Aug 5, 2026
2c079c5
Fix R25 pre-upgrade: use chart-created StorageClass, align pool name,…
RaunakJalan Aug 5, 2026
b36ec91
Fix R25 spdk-csi install and test to match actual R25 deployment flow
RaunakJalan Aug 5, 2026
dabad4c
Mask cluster secret in logs and set numPartitions=1
RaunakJalan Aug 5, 2026
645a589
Redact cluster secret from pre-upgrade state log
RaunakJalan Aug 5, 2026
c436f21
Remove unsupported cluster params from R25 cluster create
RaunakJalan Aug 5, 2026
854dec5
Enable preserve_resources_on_failure by default in upgrade tests
RaunakJalan Aug 5, 2026
3944793
Add nvme disconnect-all to cleanup phase in all K8s workflows
RaunakJalan Aug 5, 2026
3cf35f5
Make pre-upgrade FIO non-fatal in R25 maintenance upgrade test
RaunakJalan Aug 5, 2026
17e9436
Add MD5 checksum verification and clone FIO to R25 maintenance upgrade
RaunakJalan Aug 5, 2026
bf4b121
Fix FIO cleanup to preserve PVCs/snapshots/clones for MD5 checksums
RaunakJalan Aug 5, 2026
b2399c6
Use force flag on node shutdown during maintenance upgrade
RaunakJalan Aug 6, 2026
e713d5d
Skip suspend, use shutdown --force directly in maintenance upgrade
RaunakJalan Aug 6, 2026
5855de9
Wait for all SPDK pods ready and nodes online after sequential restart
RaunakJalan Aug 6, 2026
bafa67f
Wait for all storage nodes online before R25 cluster activate
RaunakJalan Aug 6, 2026
40109d7
Install cert-manager in test before operator helm install
RaunakJalan Aug 6, 2026
6c7378f
Add retry and stale cleanup to cert-manager install, revert helm retry
RaunakJalan Aug 6, 2026
e316235
Add rapid-restart stress test: 6000 objects, 30x stop/restart cycles
RaunakJalan Aug 6, 2026
156d528
Support NEW_NODE_IPS for cross-cluster restore auto-bootstrap
RaunakJalan Aug 6, 2026
9787af1
Register MassCreateRapidRestart tests in stress test discovery
RaunakJalan Aug 6, 2026
7d4ee85
Fix NS_PER_SUBSYSTEM exceeding 50 hard limit in rapid restart tests
RaunakJalan Aug 6, 2026
ecd43cc
Fix spdk-csi snapshot-controller blocking operator helm install
RaunakJalan Aug 6, 2026
8595735
Enforce max 50 ns/subsystem API limit across all mass create tests
RaunakJalan Aug 6, 2026
3ccc077
Fix FDB cluster-config ConfigMap lost during helm uninstall sbcli
RaunakJalan Aug 6, 2026
e5f4a57
Document new upgrade steps in UPGRADE.md
RaunakJalan Aug 6, 2026
712cc55
Use shared cleanup scripts in k8s-native-upgrade workflow
RaunakJalan Aug 7, 2026
082df55
Add R25 helm uninstall and kube-system cleanup to upgrade workflow
RaunakJalan Aug 7, 2026
1cf5e73
Change default cluster_params in k8s-native-upgrade workflow
RaunakJalan Aug 7, 2026
5e4c73e
Add FDB readiness wait and secret validation in upgrade workflow
RaunakJalan Aug 7, 2026
16e7d76
Fix wrong label selector for admin-control pods in upgrade test
RaunakJalan Aug 7, 2026
2450de0
Add FDB CRD protection step and comprehensive FDB verification to UPG…
RaunakJalan Aug 7, 2026
177369a
Fix FDB keep annotations: patch Helm release secret instead of kubect…
RaunakJalan Aug 7, 2026
dd63975
Add comprehensive cleanup script for R25 + R26 upgrade test setups
RaunakJalan Aug 7, 2026
4793536
Fix FDB keep annotations: patch Helm release secret instead of kubect…
RaunakJalan Aug 10, 2026
8204e6c
Change default SBCLI_BRANCH from R25.10-Hotfix to main in e2e and str…
RaunakJalan Aug 10, 2026
913194b
Add Graylog/OpenSearch log collection to Docker upgrade pipelines
RaunakJalan Aug 10, 2026
3dcdfe6
Add defensive cancel-task and disable-auto-restart methods for upgrade
RaunakJalan Aug 10, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions .github/workflows/e2e-bootstrap.yml
Original file line number Diff line number Diff line change
Expand Up @@ -844,6 +844,35 @@ jobs:
echo "TEST_START_EPOCH=$(date +%s)" >> "$GITHUB_ENV"
echo "TEST_START_HUMAN=$(date -u +'%Y-%m-%d %H:%M:%S UTC')" >> "$GITHUB_ENV"

- name: Start MinIO trace logging
if: ${{ env.CLUSTER_SECURITY == 'backup' }}
shell: bash
run: |
set -euxo pipefail

# 1. Install mc (MinIO Client)
if ! command -v mc &>/dev/null; then
if curl -fsSL https://dl.min.io/client/mc/release/linux-amd64/mc -o /usr/local/bin/mc 2>/dev/null; then
chmod +x /usr/local/bin/mc
else
curl -fsSL https://dl.min.io/client/mc/release/linux-amd64/mc -o /tmp/mc
chmod +x /tmp/mc
export PATH="/tmp:$PATH"
echo "/tmp" >> "$GITHUB_PATH"
fi
fi

# 2. Configure mc alias to external MinIO
mc alias set myminio http://192.168.10.164:9000 "${MINIO_ACCESS_KEY}" "${MINIO_SECRET_KEY}"

# 3. Start admin trace in background
MINIO_TRACE_LOG="/tmp/minio-trace-${GITHUB_RUN_ID}.log"
echo "MINIO_TRACE_LOG=${MINIO_TRACE_LOG}" >> "$GITHUB_ENV"
nohup mc admin trace -v --all myminio > "${MINIO_TRACE_LOG}" 2>&1 &
echo "MC_TRACE_PID=$!" >> "$GITHUB_ENV"

echo "=== MinIO trace logging started (PID=$!, log=${MINIO_TRACE_LOG}) ==="

- name: Run e2e tests
shell: bash
working-directory: sbcli/e2e
Expand Down Expand Up @@ -1754,6 +1783,25 @@ jobs:
ssh "${SSH_OPTS[@]}" "${SSH_USER}@${MGMT_IP}" "rm -rf '${REMOTE_OUTPUT_DIR}'" 2>/dev/null || true
done
echo "=== Log collection complete (${CHUNK} chunks, newest-first): ${OUTPUT_DIR} ==="
- name: Stop MinIO trace logging and save
if: always()
shell: bash
run: |
set +e
if [ -n "${MC_TRACE_PID:-}" ]; then
echo "Stopping MinIO trace (PID=${MC_TRACE_PID})..."
kill "${MC_TRACE_PID}" 2>/dev/null || true
wait "${MC_TRACE_PID}" 2>/dev/null || true
fi
if [ -f "${MINIO_TRACE_LOG:-}" ]; then
LINES=$(wc -l < "${MINIO_TRACE_LOG}")
SIZE=$(du -h "${MINIO_TRACE_LOG}" | cut -f1)
echo "=== MinIO trace log: ${LINES} lines, ${SIZE} ==="
cp "${MINIO_TRACE_LOG}" sbcli/e2e/minio-trace.log 2>/dev/null || true
else
echo "No MinIO trace log found (backup tests may not have run)"
fi

- name: Upload logs (always)
if: always()
uses: actions/upload-artifact@v4
Expand All @@ -1762,6 +1810,7 @@ jobs:
path: |
simplyBlockDeploy/bare-metal/bootstrap.log
sbcli/e2e/output.log
sbcli/e2e/minio-trace.log
sbcli/e2e/logs/**
if-no-files-found: warn

Expand Down
Loading
Loading