Add option to shuffle test execution order - #1041
Merged
mkarlesky merged 3 commits intoAug 18, 2026
Merged
Conversation
This commit adds the option to pass the shuffle_tests Unity option to the test runner configuration. Signed-off-by: James Raphael Tiovalen <jamestiotio@gmail.com>
Test suites can sometimes pass or fail depending on the order test cases happen to run in — usually a sign of a hidden dependency between tests. Enabling :unity ↳ :shuffle_tests randomizes execution order for a test executable each run, making that kind of bug easier to catch. The seed can be pinned with :test_runner ↳ :rng_seed for a reproducible order when tracking down a specific failure. Adds the configuration default, unit coverage for the settings' path through Configurator, a system test exercising randomized ordering end to end, and reference documentation for both options.
mkarlesky
force-pushed
the
feature/shuffle_tests
branch
from
August 18, 2026 17:33
b6242be to
d1afaab
Compare
Member
|
Thanks @jamestiotio. Pulling this into the 1.2.0 work now with some additional complementary changes. Once CI passes, we'll merge this. After the merge, I am also going to mesh this together with Ceedling's newly restored (in 1.2.0) delta builds. Ceedling is once again smart enough to skip unnecessary file regeneration, compiling, linking, and test suite running. That poses a small challenge for randomly ordered tests. In the current scheme, if no files have changed, there's no need to re-run a test suite. I am going to add an option that allows a delta build to still force a test suite re-run and to force that option when test shuffling is enabled so that every test build exercises randomly ordered test cases. The first 1.2.0 prerelease should be coming out maybe this week. |
A fixed random seed's exact shuffle output depends on the C library providing rand(), which varies by platform. With only a handful of test cases, a given seed can coincidentally reproduce the original declared order on some platforms even though shuffling is working correctly, which is exactly what happened on macOS. Raises the test count so the odds of that coincidence are negligible, and checks several distinct seeds together rather than one, asserting that shuffling produces variation somewhere among them rather than requiring every pair to differ.
mkarlesky
force-pushed
the
feature/shuffle_tests
branch
from
August 18, 2026 19:38
6424b4f to
8f3bec6
Compare
mkarlesky
merged commit Aug 18, 2026
0836d57
into
ThrowTheSwitch:pr-1041-shuffle-tests
20 checks passed
mkarlesky
added a commit
that referenced
this pull request
Aug 19, 2026
Brings in the actual shuffle_tests feature content that a stale local branch reference caused to be missed in the prior next_version merge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit adds the option to pass the
shuffle_testsUnity option to the test runner configuration.This depends on ThrowTheSwitch/Unity#714.