From 4647f3af817a38f72bd255526a0387bd499c5c6f Mon Sep 17 00:00:00 2001 From: Mohd Hamza Shaikh Date: Tue, 4 Aug 2026 23:50:34 -0500 Subject: [PATCH 1/3] docs(academy): correct Instructor Console guide against shipped Overview The Instructor Console's Overview tab was reworked in meshery-cloud v1.0.225 (layer5io/meshery-cloud#5889) and several statements in this guide describe metrics that no longer exist. - Total Test Taken was replaced by Quiz Pass Rate. The old card reported a raw attempt count with no denominator, which never said whether learners were succeeding. Updated the card description, the Strategic Uses bullet, and the FAQ entry. - Active Learners no longer counts registrations in `Registered` status. It counts distinct learners active in the current calendar month, via a registration touch or a quiz attempt, against live content only. Rewrote the definition alert to match. - The Core Metrics row ships five cards, not three. Added the missing Registrations and Completion Rate cards, plus the two deliberate product behaviours a reader will otherwise misread: a rate with no denominator renders as `-` rather than `0%`, and the Curricula tab's narrower "Registrations Here" card is smaller on purpose. Also corrected in the same file, since the same release renamed them and leaving them would have documented cards that no longer exist: the Test Stats insight cards are now Lowest Pass Rate and Highest Pass Rate, and they rank on pass rate behind a five-attempt floor rather than on raw pass/fail counts. The core-metrics screenshot still shows the old three-card row and needs recapturing; flagged inline for whoever has console access. Verified against the shipped sources: overview.tsx, metrics.ts, curricula-insights.tsx, and docs/reference/academy-instructor-console-metrics.md. Closes #1184 Signed-off-by: Mohd Hamza Shaikh --- .../instructor-console-guide/index.md | 37 +++++++++++++------ 1 file changed, 26 insertions(+), 11 deletions(-) diff --git a/content/en/cloud/academy/creating-content/instructor-console-guide/index.md b/content/en/cloud/academy/creating-content/instructor-console-guide/index.md index 86c5b9eebc9..203c35c0eea 100644 --- a/content/en/cloud/academy/creating-content/instructor-console-guide/index.md +++ b/content/en/cloud/academy/creating-content/instructor-console-guide/index.md @@ -53,20 +53,33 @@ Below the creation tool is the main dashboard, which provides an overview of you These charts display your most important, high-level statistics, giving you a snapshot of your academy's health: + ![Core Metrics Dashboard](./images/core-metrics.png) - **Total Learners**: Represents the overall reach of your academy. This is the total number of people who have ever shown interest in your content. -- **Active Learners**: Measures current engagement. This is the number of learners currently working through content, giving you an idea of your active student body. -- **Total Test Taken**: Indicates the level of interaction with your assessments. +- **Active Learners**: Measures current engagement. This is the number of distinct learners who have worked with your content this calendar month, giving you an idea of your active student body. +- **Registrations**: Every registration in the organization, whatever the state of the content it points at. This is the same total the [Learner Registration](#learner-registration) chart breaks down, so the two always agree. +- **Completion Rate**: Registrations marked completed as a share of all registrations. +- **Quiz Pass Rate**: The share of quiz attempts that passed, across every test in your organization. The card's subtitle carries the underlying counts (for example "762 of 1,036 attempts passed"), so the raw attempt total is still visible. + +A rate card with nothing to divide renders as `-`, not `0%`. "Nobody has attempted this quiz yet" and "everybody failed it" are different facts, and reporting the first as zero would send you looking for a problem that isn't there. **Strategic Uses** - Monitor Total Learners growth - steady increases indicate growing academy reach -- Analyze Total Test Taken - high counts suggest challenging material requiring retakes, low counts may indicate learners avoiding assessments, assessments not being mandatory, or content being too easy +- Analyze Quiz Pass Rate - a low rate suggests challenging material or unclear content, a very high rate may indicate assessments that are too easy. The attempt count in the card's subtitle distinguishes a low rate over many attempts (a real difficulty signal) from one over a handful (not yet meaningful). {{< alert type="info" title="What Defines an 'Active Learner'?" >}} -The **Active Learners** metric is a key indicator for engagement and billing, calculated based on the current status of content registrations. +The **Active Learners** metric is a key indicator for engagement and billing. It counts *learners*, not registrations, and it is scoped to the current calendar month. + +A learner counts as active this month if, against academy content that is still live, they either created or touched a registration during the month, or submitted a quiz attempt during the month. A learner who does both counts once. A learner who enrolled months ago and sits a quiz this month is active this month, even though their registration row has not changed. + +Because it counts distinct people rather than open registrations, it is smaller than the number of registrations in progress, and it resets its window each month rather than accumulating. +{{< /alert >}} -Specifically, it counts the number of registrations that are currently in the `Registered` status. This represents learners who have enrolled in your content but have not yet `Completed` it. It gives you a clear picture of your "in-progress" student body. +{{< alert type="info" title="'Registrations' vs. 'Registrations Here'" >}} +The **Curricula** tab has its own registrations card, named **Registrations Here**, which deliberately counts something narrower: only registrations against the published curricula listed on that tab. It is normally a smaller number than the Overview's **Registrations** card, which counts every registration in the organization including those against draft or retired content. That gap is expected, not a discrepancy - the card carries a tooltip explaining the difference. {{< /alert >}} ### Content Details @@ -166,10 +179,12 @@ These statistics provide a high-level summary of all test activities in your aca - **Pass/Fail Summary Bar:** This bar chart gives you an at-a-glance comparison of the total number of passed attempts (green) versus failed attempts (red) across all tests. - **Insight Cards:** These three cards automatically surface key trends and outliers from your data: - - **Most Difficult Test:** The quiz with the lowest pass-to-fail ratio, helping you identify challenging content. - - **Easiest Test:** The quiz with the highest pass-to-fail ratio. + - **Lowest Pass Rate:** The quiz with the smallest share of passing attempts, helping you identify challenging content. + - **Highest Pass Rate:** The quiz with the largest share of passing attempts. - **Most Attempted Test:** The quiz that learners have attempted the most times, regardless of the outcome. + The two rate cards rank on pass rate rather than on raw pass and fail counts, and only tests with at least five attempts are ranked. Ranking on counts would measure popularity instead of difficulty - the test the most people sit holds both the most passes and the most failures - and the attempts floor stops a single learner failing a brand-new quiz from crowning it the hardest test. In a young academy where no test has reached five attempts yet, the ranking falls back to the full set. + #### Test Metrics The **Test Metrics** list provides a detailed performance breakdown for each individual test. Each test title in this list is a clickable link that will take you directly to that assessment. @@ -181,7 +196,7 @@ Use this list to quickly identify which specific assessments are causing the mos {{< /alert >}} **Strategic Uses** -- Investigate difficult tests - when a quiz appears in the **Most Difficult Test** card with low pass rates, review the content and questions for clarity +- Investigate difficult tests - when a quiz appears in the **Lowest Pass Rate** card, review the content and questions for clarity - Monitor test attempt patterns - if **Most Attempted Test** has low pass rates, learners may be struggling with fundamental concepts - Use pass rate trends - improving pass rates suggest content optimization success, declining rates may indicate new content is too challenging @@ -199,7 +214,7 @@ Currently, the Instructor Console does not have a built-in feature to export the
-Does "Total Test Taken" show the number of unique learners who took tests? +Does "Quiz Pass Rate" count unique learners? -No. The "Total Test Taken" metric is a raw count of all attempts, including retakes by the same user. It is a measure of overall testing activity, not the number of unique learners who have been tested. -
\ No newline at end of file +No. It is computed over attempts, not people: retakes by the same learner each count, so a learner who fails twice and then passes contributes two failures and one pass. It measures assessment outcomes, not how many distinct learners have been tested. For a headcount, use Total Learners or Active Learners. + From b2576f339b1986a59218218a3ceafa4943235df5 Mon Sep 17 00:00:00 2001 From: Mohd Hamza Shaikh Date: Tue, 4 Aug 2026 23:54:04 -0500 Subject: [PATCH 2/3] docs(academy): document all five learner registration statuses The Learner Registration section claimed to list the definitions the system recognizes and then named three of them. The `public.academy_registration_status` enum carries five, and the same release that reworked the Overview extended the chart's colour map to cover all of them - before that, two statuses shared a colour, which is why the gap was invisible. Adds `In Progress` (amber) and `Failed` (crimson), and narrows the `Registered` description, which previously read "has enrolled but has not yet completed the content" and so covered `In Progress` and `Failed` as well. Colours and labels taken from REGISTRATION_STYLES and CurriculaRegistrationStatus in meshery-cloud ui/components/academy/utils.ts. Signed-off-by: Mohd Hamza Shaikh --- .../creating-content/instructor-console-guide/index.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/content/en/cloud/academy/creating-content/instructor-console-guide/index.md b/content/en/cloud/academy/creating-content/instructor-console-guide/index.md index 203c35c0eea..6653324b481 100644 --- a/content/en/cloud/academy/creating-content/instructor-console-guide/index.md +++ b/content/en/cloud/academy/creating-content/instructor-console-guide/index.md @@ -107,9 +107,11 @@ This chart provides an immediate visual summary of your learner base. ![Learner Registration](./images/learner-registration.png) -While the chart will only display statuses that currently have data, the system recognizes the following definitions, each represented by a specific color: - - `Registered` (Blue): The learner has enrolled but has not yet completed the content. +While the chart will only display statuses that currently have data, the system recognizes the following five definitions, each represented by a specific color: + - `Registered` (Blue): The learner has enrolled but has not yet started the content. + - `In Progress` (Amber): The learner has started the content but has not yet finished it. - `Completed` (Green): The learner has successfully finished all required parts of the content. + - `Failed` (Crimson): The learner did not pass the content. - `Withdrawn` (Red): The learner has unenrolled from the content. **Strategic Uses** From 171063bdae0f09649f96b13e9ff38d9924b1430c Mon Sep 17 00:00:00 2001 From: Mohd Hamza Shaikh Date: Tue, 4 Aug 2026 23:58:45 -0500 Subject: [PATCH 3/3] docs(academy): address review on the Instructor Console corrections Three fixes from automated review, all valid: - Active Learners is not bounded by in-progress registrations. The previous wording said it is "smaller than the number of registrations in progress", which is false: a learner whose registration is already completed becomes active again by taking a quiz, so an academy with no in-progress registrations and ten quiz-takers this month reports ten active learners against zero. Replaced the false comparison with the point it was reaching for - the two are not directly comparable. This error came in verbatim from the wording proposed in #1184. - US English: "sits a quiz" / "people sit" -> "takes" / "take". - The core-metrics screenshot still shows the old three-card row, and the recapture note flagging that was an HTML comment, so it was invisible to readers - who would see three cards above a list of five. Added a short reader-visible caption until the image is recaptured. Signed-off-by: Mohd Hamza Shaikh --- .../creating-content/instructor-console-guide/index.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/content/en/cloud/academy/creating-content/instructor-console-guide/index.md b/content/en/cloud/academy/creating-content/instructor-console-guide/index.md index 6653324b481..0fe2178964f 100644 --- a/content/en/cloud/academy/creating-content/instructor-console-guide/index.md +++ b/content/en/cloud/academy/creating-content/instructor-console-guide/index.md @@ -58,6 +58,8 @@ These charts display your most important, high-level statistics, giving you a sn current console. Tracked in layer5io/docs#1184. --> ![Core Metrics Dashboard](./images/core-metrics.png) +*This screenshot predates the current layout and is pending recapture. The console now shows the five cards described below.* + - **Total Learners**: Represents the overall reach of your academy. This is the total number of people who have ever shown interest in your content. - **Active Learners**: Measures current engagement. This is the number of distinct learners who have worked with your content this calendar month, giving you an idea of your active student body. - **Registrations**: Every registration in the organization, whatever the state of the content it points at. This is the same total the [Learner Registration](#learner-registration) chart breaks down, so the two always agree. @@ -73,9 +75,9 @@ A rate card with nothing to divide renders as `-`, not `0%`. "Nobody has attempt {{< alert type="info" title="What Defines an 'Active Learner'?" >}} The **Active Learners** metric is a key indicator for engagement and billing. It counts *learners*, not registrations, and it is scoped to the current calendar month. -A learner counts as active this month if, against academy content that is still live, they either created or touched a registration during the month, or submitted a quiz attempt during the month. A learner who does both counts once. A learner who enrolled months ago and sits a quiz this month is active this month, even though their registration row has not changed. +A learner counts as active this month if, against academy content that is still live, they either created or touched a registration during the month, or submitted a quiz attempt during the month. A learner who does both counts once. A learner who enrolled months ago and takes a quiz this month is active this month, even though their registration row has not changed. -Because it counts distinct people rather than open registrations, it is smaller than the number of registrations in progress, and it resets its window each month rather than accumulating. +Because it counts distinct people over a monthly window rather than counting registrations, it is not directly comparable to the registration totals beside it, and it resets each month rather than accumulating. A learner who finished a curriculum long ago still counts as active in any month they take a quiz. {{< /alert >}} {{< alert type="info" title="'Registrations' vs. 'Registrations Here'" >}} @@ -185,7 +187,7 @@ These statistics provide a high-level summary of all test activities in your aca - **Highest Pass Rate:** The quiz with the largest share of passing attempts. - **Most Attempted Test:** The quiz that learners have attempted the most times, regardless of the outcome. - The two rate cards rank on pass rate rather than on raw pass and fail counts, and only tests with at least five attempts are ranked. Ranking on counts would measure popularity instead of difficulty - the test the most people sit holds both the most passes and the most failures - and the attempts floor stops a single learner failing a brand-new quiz from crowning it the hardest test. In a young academy where no test has reached five attempts yet, the ranking falls back to the full set. + The two rate cards rank on pass rate rather than on raw pass and fail counts, and only tests with at least five attempts are ranked. Ranking on counts would measure popularity instead of difficulty - the test the most people take holds both the most passes and the most failures - and the attempts floor stops a single learner failing a brand-new quiz from crowning it the hardest test. In a young academy where no test has reached five attempts yet, the ranking falls back to the full set. #### Test Metrics