Add maxissues threshold support for CI/CD rule enforcement by B1naryVision · Pull Request #15 · certinia/asist

B1naryVision · 2026-02-19T12:49:44Z

Currently, asist fails if any issues are detected (equivalent to a hard cap of 0). This change allows users to define a custom limit per rule.

💡 Why is this needed?
On large or legacy repositories, "perfect" compliance is often impossible to achieve overnight. By allowing a maxIssues threshold, teams can:

Grandfather in existing issues: Set the cap to the current issue count to prevent new regressions.

Incremental Improvement: Gradually lower the cap over time as technical debt is cleared.

CI Adoption: Enable the analyzer in CI/CD pipelines without immediately breaking the build for existing debt.

🛠 Proposed Changes
Added a maxIssues configuration option per rule.

Default Behavior: If no cap is defined, it defaults to 0, maintaining the current strict enforcement behavior.

Logic: The build fails only if actualIssues > maxIssues.

Why only CI/CD?: It only exits with error codes if it's in CI/CD mode, otherwise it simply prints out the issues.
Perhaps we would also like to Print a warning outside of CI/CD? Would that mess with baseline or anything though?

🧪 WIP Notes (Seeking Early Input)
I am raising this as a Draft PR to get feedback on the implementation logic before I finalize the test suite. Specifically:

Does the flag naming (maxIssues) align with the project's naming conventions?
Here's some other options I considered.

issuethreshold
maxallowedissues
cicdmaxissues
failon

Are there any edge cases in the threshold logic I should account for before I write the unit tests?

I added a printing log in CI/CD mode, is this a breaking change as it now no longer ONLY prints the issues?

- Add MaxIssues field to RuleMetadataOverride (Rule.go) - Add GetRuleMaxIssues method to Config to read the configured threshold (config.go) - Add checkThresholdViolations in output.go to compare findings against thresholds and exit non-zero when a rule exceeds its configured maxissues in CI/CD mode

- Add maxissues property description and usage to README and index.md - Update .asist.example.yaml with maxissues example and explanatory comments

B1naryVision · 2026-02-19T12:55:02Z

+	for ruleId, count := range findingsPerRule {
+		maxIssuesAllowed := configFile.GetRuleMaxIssues(ruleId)
+
+		if count > maxIssuesAllowed {


This may be considered a breaking change if we really want CI/CD to only print the issues.

This just prints when you go over, default behaviour is 0 allowed in CI/CD

Could be prettified too, it shows just after printing issues, so probably need to add a linebreak before if we do decide to keep it.

Eg. example

{ "ID": "XSSMergeField", "Name": "Potential XSS with mergefield", "Description": "Using a merge field inside a script or style tag may introduce an XSS issue. Consider adding an encode method for the merge field; otherwise, ensure that all user inputs are sanitized properly.", "Severity": "High", "RuleCategory": "Security", "Occurrence": { "File": "<redacted>", "Line": " <redacted>", "LineNumber": 13, "ColumnRange": [ 52, 83 ] } } ] } Threshold violation: Rule XSSMergeField has 50 issues (max allowed: 5)

Prettified it a little bit more, I added a space

... JSON } Threshold violations: Rule XSSDomHtml has 40 issues (max allowed: 0) Rule XSSMergeField has 36 issues (max allowed: 5) 2 rule(s) exceed their maxissues threshold.

B1naryVision · 2026-02-19T14:31:00Z

-2. Enable the `Custom Binary Enabled` setting
-3. Provide the path to your ASIST binary in `Custom Binary Path`
+1. Enable the `Custom Binary Enabled` setting
+1. Provide the path to your ASIST binary in `Custom Binary Path`


Context: 1. 1. 1 etc is valid for MD, it auto-increments.

B1naryVision · 2026-02-19T14:32:20Z


-
-### 🫣 .gitignore and .forceignore files
+## 🫣 .gitignore and .forceignore files


I did some nitpicking, a H1 -> H3.

Arguably this should have stayed a H3 and CI/CD should be a H2 but...
Happy to change it up 😅

B1naryVision · 2026-02-19T14:34:19Z

Future enhancement I might do, is make the index/readme copy from a central file with these details so I don't need to copy-paste across except for the headers/footers. Maybe it does and I just didn't see it?

ajaincertinia · 2026-02-25T07:17:12Z

-* Exclude files or directories for specific rules
-* Add custom regex rules
-* Define rules to be run in CI/CD mode
+- Enable/disable all the standard rules  


Suggested change

- Enable/disable all the standard rules

- Enable/disable all the standard rules

jwoodwardcertinia · 2026-02-25T15:14:43Z

 cicdrules:
  - "XSSLabel"
  - "XSSMergeField"
+  - "XSSDomHtml"


Should this list be alphabetical?

jwoodwardcertinia · 2026-02-25T15:17:47Z


+/**
+ * Checks if any rule exceeds its configured maxissues.
+ * Default maxissues is 0 (no issues allowed) if not explicitly set.


Default and 'not explicitly set' mean the same. So there is redundancy in this sentence.

jwoodwardcertinia

LGTM

- Alphabetising cicd rules - Removing rogue spaces at the end of list

- Ordering the rules to ensure they are deterministic for testing - Removed unintentional repetition in comment - Added padding between the violations - Added a message for saying how many violated rules there are

Added: - Integration tests - Output tests - config tests

ajaincertinia

Hi @B1naryVision, we had internal discussion regarding the --list-rule case, and decided that maxissues should be renamed to cicdmaxissues (as per your earlier suggestion), and --list-rule should print this property.

- Rename RuleMetadataOverride.MaxIssues → CicdMaxIssues; update YAML/JSON deserialization key from `maxissues` to `cicdmaxissues` accordingly - Add CicdMaxIssues int field to RuleMetadata so --list-rules output includes the configured threshold per rule (default 0) - Populate CicdMaxIssues on RuleMetadata via Override() at rule load time - Update all tests, fixtures, docs, example config, and changelog

ajaincertinia · 2026-03-03T10:50:31Z

+	}
+}
+
+func TestCheckThresholdViolations_WhenMaxIssuesNotSet_DefaultsToZero(t *testing.T) {


Suggested change

func TestCheckThresholdViolations_WhenMaxIssuesNotSet_DefaultsToZero(t *testing.T) {

func TestCheckThresholdViolations_WhenCicdMaxIssuesNotSet_DefaultsToZero(t *testing.T) {

ajaincertinia · 2026-03-03T10:51:12Z

+
+	//Then
+	if !result {
+		t.Errorf("Expected violation when maxissues defaults to 0 and rule has 1 finding")


Suggested change

t.Errorf("Expected violation when maxissues defaults to 0 and rule has 1 finding")

t.Errorf("Expected violation when cicdmaxissues defaults to 0 and rule has 1 finding")

ajaincertinia · 2026-03-03T11:00:54Z

+ * Returns the max allowed issues for a specific rule ID.
+ * Returns ruleoverrides.<rule>.cicdmaxissues if set, otherwise 0.
+ */
+func (c *Config) GetRuleMaxIssues(ruleId rules.RuleID) int {


Not sure but should GetRuleMaxIssues also be renamed to GetRuleCicdMaxIssues?

I don't mind it, I think the method is clear. But happy to change it if you feel like it should be renamed

It's okay to me.

Co-authored-by: Govardhan <13658588+gkumarcertinia@users.noreply.github.com>

- Add CicdMaxIssues *int field to CustomRegexRule config struct - Propagate it through NewCustomRule into RuleMetadata.CicdMaxIssues - Extend GetRuleCicdMaxIssues to fall back to CustomRegexRules after checking RuleOverrides, so custom rule thresholds are respected in CI/CD mode (-j) - Update config testdata fixtures (YAML + JSON) with cicdmaxissues - Update CustomRule_test.go and config_test.go with new tests - Update README, docs/index.md, .asist.example.yaml, and changelog

B1naryVision · 2026-03-05T13:30:52Z

I meant to merge, accidentally closed. I just realised I don't see a merge button

Harold Albertsson added 2 commits February 19, 2026 12:07

docs: document maxissues threshold configuration

2c71736

- Add maxissues property description and usage to README and index.md - Update .asist.example.yaml with maxissues example and explanatory comments

B1naryVision commented Feb 19, 2026

View reviewed changes

pjaincertinia2025 requested a review from gkumarcertinia February 23, 2026 05:28

gkumarcertinia previously approved these changes Feb 23, 2026

View reviewed changes

pjaincertinia2025 requested a review from ajaincertinia February 24, 2026 05:29

ajaincertinia reviewed Feb 25, 2026

View reviewed changes

jwoodwardcertinia reviewed Feb 25, 2026

View reviewed changes

jwoodwardcertinia previously approved these changes Feb 25, 2026

View reviewed changes

Harold Albertsson added 3 commits February 27, 2026 10:10

docs: Updating README with requested changes

bbe8165

- Alphabetising cicd rules - Removing rogue spaces at the end of list

feat: Improving the violation messages

3681423

- Ordering the rules to ensure they are deterministic for testing - Removed unintentional repetition in comment - Added padding between the violations - Added a message for saying how many violated rules there are

test: writing tests for maxissues

15621c4

Added: - Integration tests - Output tests - config tests

B1naryVision dismissed stale reviews from jwoodwardcertinia and gkumarcertinia via 15621c4 February 27, 2026 10:56

docs: add maxissues to changelog

7ce279b

B1naryVision marked this pull request as ready for review February 27, 2026 11:21

ajaincertinia reviewed Mar 3, 2026

View reviewed changes

fix: renaming remaining instances of maxissues to cicdmaxissues

4e58b79

ajaincertinia previously approved these changes Mar 3, 2026

View reviewed changes

B1naryVision requested a review from gkumarcertinia March 3, 2026 11:50

gkumarcertinia reviewed Mar 3, 2026

View reviewed changes

Comment thread README.md Resolved

Update README.md

598f280

Co-authored-by: Govardhan <13658588+gkumarcertinia@users.noreply.github.com>

B1naryVision dismissed ajaincertinia’s stale review via 598f280 March 3, 2026 12:18

docs: updating index.md with clarification on cicdmaxissues

6728621

gkumarcertinia previously approved these changes Mar 3, 2026

View reviewed changes

B1naryVision requested a review from ajaincertinia March 3, 2026 12:27

B1naryVision dismissed gkumarcertinia’s stale review via 5b41966 March 3, 2026 13:14

B1naryVision requested a review from gkumarcertinia March 3, 2026 13:16

gkumarcertinia approved these changes Mar 3, 2026

View reviewed changes

ajaincertinia approved these changes Mar 5, 2026

View reviewed changes

B1naryVision closed this Mar 5, 2026

B1naryVision reopened this Mar 5, 2026

pjaincertinia2025 closed this Mar 5, 2026

pjaincertinia2025 reopened this Mar 5, 2026

pjaincertinia2025 merged commit 4549282 into certinia:main Mar 5, 2026
16 checks passed

B1naryVision deleted the add-support-for-issue-cap branch March 5, 2026 14:45



		### 🫣 .gitignore and .forceignore files
		## 🫣 .gitignore and .forceignore files

	- Enable/disable all the standard rules
	- Enable/disable all the standard rules

	func TestCheckThresholdViolations_WhenMaxIssuesNotSet_DefaultsToZero(t *testing.T) {
	func TestCheckThresholdViolations_WhenCicdMaxIssuesNotSet_DefaultsToZero(t *testing.T) {

	t.Errorf("Expected violation when maxissues defaults to 0 and rule has 1 finding")
	t.Errorf("Expected violation when cicdmaxissues defaults to 0 and rule has 1 finding")

Conversation

B1naryVision commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

B1naryVision Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jwoodwardcertinia left a comment

Choose a reason for hiding this comment

Uh oh!

ajaincertinia left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

B1naryVision Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

B1naryVision commented Mar 5, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

B1naryVision commented Feb 19, 2026 •

edited

Loading

B1naryVision Feb 27, 2026 •

edited

Loading

B1naryVision Mar 3, 2026 •

edited

Loading