Surface per-task status summary in Helix JobContext by LZD-PratyushBhatt · Pull Request #214 · linkedin/helix

LZD-PratyushBhatt · 2026-07-21T04:17:15Z

When a Task Framework job is configured with a high FailureThreshold so that every partition task is allowed to run to completion, the job's own status flag stays COMPLETED even if individual tasks fail. That masks partial failures: operators watching only the job state cannot tell that some partitions errored, aborted, or timed out.

This adds an aggregated, job-level task status summary that the controller computes and writes into the JobContext when a job reaches a terminal state (completed, failed, or timed out). The summary carries completed/failed counts, a per-state breakdown, and the list of failed partitions, so partial failures stay visible even when the job flag is COMPLETED. Reading the summary is a clean alternative to the Integer.MAX_VALUE FailureThreshold workaround.

helix-core:

JobContext#updateTaskStatusSummary computes the summary from per-partition states and stores it as the TASK_STATUS_SUMMARY simple field (JSON); JobContext#getTaskStatusSummary reads it back.
AbstractTaskDispatcher invokes it at the three terminal choke points (markJobComplete, failJob, handleJobTimeout).

helix-front:

Job detail view gains a "Task Summary" tab that parses TASK_STATUS_SUMMARY and highlights failures.

Tests:

TestJobTaskStatusSummary integration test (job COMPLETED, summary reports the failures).
JobTaskSummaryDriver standalone end-to-end driver (runs against a live ZK).
job-detail component specs for the new getters.

Locally tested in real helix cluster:

Issues

My PR addresses the following Helix issues and references them in the PR description:

(#200 - Link your issue number here: You can write "Fixes #XXX". Please use the proper keyword so that the issue gets closed automatically. See https://docs.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue
Any of the following keywords can be used: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved)

Description

Here are some details about my PR, including screenshots of any UI changes:

(Write a concise description including what, why, how)

Tests

The following tests are written for this issue:

(List the names of added unit/integration tests)

The following is the result of the "mvn test" command on the appropriate module:

(If CI test fails due to known issue, please specify the issue and test PR locally. Then copy & paste the result of "mvn test" to here.)

Changes that Break Backward Compatibility (Optional)

My PR contains changes that break backward compatibility or previous assumptions for certain methods or API. They include:

(Consider including all behavior changes for public methods or API. Also include these changes in merge description so that other developers are aware of these changes. This allows them to make relevant code changes in feature branches accounting for the new method/API behavior.)

Documentation (Optional)

In case of new functionality, my PR adds documentation in the following wiki page:

(Link the GitHub wiki you added)

Commits

My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters (not including Jira issue reference)
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"

Code Quality

My diff has been formatted using helix-style.xml
(helix-style-intellij.xml if IntelliJ IDE is used)

When a Task Framework job is configured with a high FailureThreshold so that every partition task is allowed to run to completion, the job's own status flag stays COMPLETED even if individual tasks fail. That masks partial failures: operators watching only the job state cannot tell that some partitions errored, aborted, or timed out. This adds an aggregated, job-level task status summary that the controller computes and writes into the JobContext when a job reaches a terminal state (completed, failed, or timed out). The summary carries completed/failed counts, a per-state breakdown, and the list of failed partitions, so partial failures stay visible even when the job flag is COMPLETED. Reading the summary is a clean alternative to the Integer.MAX_VALUE FailureThreshold workaround. helix-core: - JobContext#updateTaskStatusSummary computes the summary from per-partition states and stores it as the TASK_STATUS_SUMMARY simple field (JSON); JobContext#getTaskStatusSummary reads it back. - AbstractTaskDispatcher invokes it at the three terminal choke points (markJobComplete, failJob, handleJobTimeout). helix-front: - Job detail view gains a "Task Summary" tab that parses TASK_STATUS_SUMMARY and highlights failures. Tests: - TestJobTaskStatusSummary integration test (job COMPLETED, summary reports the failures). - JobTaskSummaryDriver standalone end-to-end driver (runs against a live ZK). - job-detail component specs for the new getters. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

sjainit · 2026-07-30T04:30:29Z

+    for (int p : partitions) {
+      TaskPartitionState state = getPartitionState(p);
+      String key = (state == null) ? "UNSCHEDULED" : state.name();
+      byState.merge(key, 1, Integer::sum);


what is byState ?

good one, byState is just the raw per-state histogram behind the coarse counts. each TaskPartitionState name (plus UNSCHEDULED when a partition has no state yet) maps to how many tasks are in that state. added a javadoc para on updateTaskStatusSummary() spelling it out.

sjainit · 2026-07-30T04:30:52Z

+    failedTasks.sort(null);
+
+    Map<String, Object> summary = new LinkedHashMap<>();
+    summary.put("total", partitions.size());


I'd recommend using string constants instead of hard coding these keys in the map

yeah makes sense. pulled all the summary keys (total, completed, failed, timedOut and the Tasks arrays) plus UNSCHEDULED into named SUMMARY_KEY_ and SUMMARY_STATE_UNSCHEDULED constants, and the map builds off those now instead of inline literals.

sjainit · 2026-07-30T04:34:21Z

+
+      Workflow flow =
+          WorkflowGenerator.generateSingleJobWorkflowBuilder(jobResource, jobBuilder).build();
+      System.out.println("Submitting workflow '" + jobResource + "' with " + NUM_TASKS


do we know need these print statements in tests anymore?

nice catch, not needed. that was a standalone main() driver, not a TestNG test, and it needed an external ZK plus it just duplicated TestJobTaskStatusSummary. dropped the whole file so the prints go with it. coverage stays via that integration test plus a new deterministic TestJobContextTaskStatusSummary unit test.

sjainit

Just wondering, do we also have workflow level summary like we are aggregating at job level ?

Reviewer feedback on the per-task status summary asked to also surface in-progress and timed-out tasks as first-class counts, rather than having timed-out tasks folded into the failed count and in-progress tasks hidden inside "other". helix-core: - JobContext#updateTaskStatusSummary now emits explicit "timedOut" and "inProgress" counts plus "timedOutTasks" and "inProgressTasks" partition lists. The top-level counts partition the tasks as total = completed + failed + inProgress + other; timedOut is called out as the subset of failed that timed out, and inProgress (INIT or RUNNING) is pulled out of "other" so still-running tasks are visible when the summary is written at the job-timeout choke point. - failed and failedTasks keep their existing meaning (a timed-out task is still counted as failed), so existing consumers are unaffected. helix-front: - Job detail Task Summary tab shows Timed Out and In Progress chips and the corresponding partition lists, and flags timed-out tasks as a problem. Tests: - Integration test asserts the new counts are present and that the top-level counts partition the total. - Standalone driver verifies the timedOut and inProgress keys are always emitted. - Job detail component spec covers the new getters and a scenario with timed-out and in-progress tasks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ew feedback - Split INIT tasks into a separate "pending" bucket, distinct from "inProgress" (RUNNING), in the JobContext task status summary. - Recompute the summary on the job detail page from per-partition states each time the page is opened, falling back to the stored terminal snapshot when needed. - Extract the summary JSON field names into named SUMMARY_KEY_* constants instead of inline string literals. - Document what the byState histogram represents in updateTaskStatusSummary(). - Remove the standalone JobTaskSummaryDriver; the TestNG integration test and a new deterministic JobContext unit test already cover the same scenario. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

LZD-PratyushBhatt · 2026-08-14T09:16:37Z

hmmm not yet, this aggregates at the job level only. WorkflowContext already tracks per-job states via getJobStates() so a workflow-level roll-up is very doable, but I think we should keep this PR job-scoped and do the workflow-level one as a follow-up so it gets its own UI and tests. can pick it up next.

LZD-PratyushBhatt requested review from PranaviAncha, arkmish, kabragaurav, laxman-ch, ngngwr, sjainit and thestreak101 as code owners July 21, 2026 04:17

LZD-PratyushBhatt force-pushed the lzd/job-task-status-summary branch from 7edef10 to 5d3303d Compare July 21, 2026 04:45

sjainit reviewed Jul 30, 2026

View reviewed changes

sjainit suggested changes Jul 30, 2026

View reviewed changes

LZD-PratyushBhatt and others added 2 commits July 30, 2026 22:08

Navigation Menu

Uh oh!

Surface per-task status summary in Helix JobContext - #214

Surface per-task status summary in Helix JobContext#214
LZD-PratyushBhatt wants to merge 3 commits into
devfrom
lzd/job-task-status-summary

LZD-PratyushBhatt commented Jul 21, 2026 •

edited

Loading

Uh oh!

sjainit Jul 30, 2026

Uh oh!

LZD-PratyushBhatt Aug 14, 2026

Uh oh!

sjainit Jul 30, 2026

Uh oh!

LZD-PratyushBhatt Aug 14, 2026

Uh oh!

sjainit Jul 30, 2026

Uh oh!

LZD-PratyushBhatt Aug 14, 2026

Uh oh!

sjainit left a comment

Uh oh!

LZD-PratyushBhatt commented Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

LZD-PratyushBhatt commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issues

Description

Tests

Changes that Break Backward Compatibility (Optional)

Documentation (Optional)

Commits

Code Quality

Uh oh!

sjainit Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

LZD-PratyushBhatt Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

sjainit Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

LZD-PratyushBhatt Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

sjainit Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

LZD-PratyushBhatt Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

sjainit left a comment

Choose a reason for hiding this comment

Uh oh!

LZD-PratyushBhatt commented Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

LZD-PratyushBhatt commented Jul 21, 2026 •

edited

Loading