Repository navigation
fix(analyzer): bound file analysis workers - #8058
Conversation
|
I have tested this fix against my repo where the issue was occurring - the issue is no longer reproducible |
|
Hi @omribz156, I've also verified this resolves the thread-exhaustion crash cleanly and the bounded-pool wiring (channel-fed workers, wg.Add/wg.Done moved to the pool level) is correct, including preserving the existing unwanted-channel drain race handling in computeValues that I've introduced earlier this year. One addition I would propose: the added test to analyzer_test only covers Let me know what you think and thanks again for your help improving kics! P.S: I can address the changes on my side and close this PR once we merge the changes, mentioning both the issue and the original pull request. |
|
Commits do need to be signed in order for any pull request to be merged. If this is something that you cannot achieve on your side, I will open the PR on my side and tag you, as well as the pull request and related issue as well. Let me know if you could sign the commits on your side @omribz156. |
|
Hi @omribz156, The bounded pool implementation looks good. Let me know what you think and thanks again for your help improving KICS! |
2880735 to
8724dcd
Compare
… library embedders
* fix(analyzer): bound file analysis workers * test: cover bounded analyzer worker concurrency * update: add MaxAnalyzerWorkers to Analyzer providing better usage for library embedders --------- Co-authored-by: Artur Ribeiro <153724638+cx-artur-ribeiro@users.noreply.github.com>
kics 2.2.0 Created-by: HarmonybrewBot Commit-by: HarmonybrewBot Merged-by: HarmonybrewBot Description: Created by `brew bump` --- Created with `brew bump-formula-pr`.<details> <summary>release notes</summary> <pre>## What's Changed * docs(release): update queries catalog, index and dockerfile for upcoming release by @cx-artur-ribeiro in Checkmarx/kics#8094 * fix(query): added missing case to "Last User is Root" Dockerfile query. by @cx-andre-pereira in Checkmarx/kics#8095 * feat(queries): new queries to check if synapse workspace managed virtual network is enabled by @cx-ricardo-jesus in Checkmarx/kics#8097 * refactor(descriptions): remove pkg/descriptions and related CLI flag by @cx-ricardo-jesus in Checkmarx/kics#8092 * CISO-1264 - Update GitHub Actions runner labels by @cx-jonathan-hartman in Checkmarx/kics#8093 * fix(query): Fix for Missing Backslash Support on Copy_With_More_Than_Two_Arguments_Not_Ending_With_Slash query by @cx-andre-pereira in Checkmarx/kics#8099 * fix(actions): refactor gh actions to fix CI by @cx-artur-ribeiro in Checkmarx/kics#8098 * fix(query): change to keyExpectedValue on ARM 'default_azure_storage_account_network_access_is_too_permissive' query by @cx-andre-pereira in Checkmarx/kics#8101 * update(version): rename VERSION build-arg to ENGINE_VERSION by @cx-artur-ribeiro in Checkmarx/kics#8102 * fix(query): correct keyExpectedValue and keyActualValue in redshift_not_encrypted by @cx-ricardo-jesus in Checkmarx/kics#8105 * fix(analyzer): bound file analysis workers by @omribz156 in Checkmarx/kics#8058 * fix(actions): remove outdated actions and update documentation accordingly by @cx-artur-ribeiro in Checkmarx/kics#8110 * fix(action): remove concurrent group from run projects github action by @cx-artur-ribeiro in Checkmarx/kics#8111 * fix(test): normalize timestamp comparison in TestInitCycloneDxReport by @cx-artur-ribeiro in Checkmarx/kics#8113 * fix(actions): fix security vulnerabilities and update ci with new enforced rules by @cx-artur-ribeiro in Checkmarx/kics#8118 * fix(release): gate kics release workflows behind release environment by @cx-lior-poterman in Checkmarx/kics#8108 * fix(validator): update queries validator for cwe and risk score fields by @cx-artur-ribeiro in Checkmarx/kics#8028 * fix(version): new available version with additional v prefix by @cx-artur-ribeiro in Checkmarx/kics#8119 * fix(analyzer): improvement to dockerfile scanning by @cx-andre-pereira in Checkmarx/kics#8114 * chore(release): removed unused goreleaser configuration files by @cx-ricardo-jesus in Checkmarx/kics#8122 * fix(filesystem): skip cache files by extension by @cx-laura-rodrigues in Checkmarx/kics#8112 * fix(query): changed all dockerfile queries for case insensitive support of dockerfile commands by @cx-andre-pereira in Checkmarx/kics#8115 * docs(release): update queries catalog, index and dockerfile for 2.2.0 by @cx-artur-ribeiro in Checkmarx/kics#8126 ## New Contributors * @cx-jonathan-hartman made their first contribution in Checkmarx/kics#8093 * @omribz156 made their first contribution in Checkmarx/kics#8058 * @cx-lior-poterman made their first contribution in Checkmarx/kics#8108 **Full Changelog**: https://github.com/Checkmarx/kics/compare/v2.1.21...v2.2.0</pre> <p>View the full release notes at <a href="https://github.com/Checkmarx/kics/releases/tag/v2.2.0">https://github.com/Checkmarx/kics/releases/tag/v2.2.0</a>.</p> </details> <hr> See merge request: Harmonybrew/homebrew-core!20473
* fix(analyzer): bound file analysis workers * test: cover bounded analyzer worker concurrency * update: add MaxAnalyzerWorkers to Analyzer providing better usage for library embedders --------- Co-authored-by: Artur Ribeiro <153724638+cx-artur-ribeiro@users.noreply.github.com>
* fix(analyzer): bound file analysis workers * test: cover bounded analyzer worker concurrency * update: add MaxAnalyzerWorkers to Analyzer providing better usage for library embedders --------- Co-authored-by: Artur Ribeiro <153724638+cx-artur-ribeiro@users.noreply.github.com>
Closes #8046
Reason for Proposed Changes
Proposed Changes
GOMAXPROCS, capped at 128 workers and never larger than the candidate file count.Verification
go test ./pkg/analyzer -count=1go test ./pkg/scan -run "Test.*Analyze|Test.*Init|Test.*Prepare" -count=1git diff --checkThis was implemented with Codex assistance, with the final patch kept focused and manually reviewed.
I submit this contribution under the Apache-2.0 license.