Community Health: monthly community health report system

documentation/docs/admin/insights.md

@@ -1,35 +1,57 @@
 # Insights
 
+Pontoon provides 2 major systems for admins to monitor locale & project health. The tools
+available are the `Insights` dashboard and the `Monthly Health Report`, which both provide
+information on locale growth & decline in easily ingestible formats.
+
+## Insights dashboard
+
 Pontoon provides a global **Insights** dashboard for analyzing the health of localization communities and the quality of pretranslations across all locales and projects.
 
 The dashboard is restricted to staff and is available at `/insights` when logged in as an administrator (e.g. [https://pontoon.mozilla.org/insights/](https://pontoon.mozilla.org/insights/) for the instance hosted by Mozilla).
 
-## Community health activity
+### Community health score
 
 This section presents a table of the monthly `Community health score` and its underlying metrics for each selected locale. For every team it shows the number of users in different roles — managers, translators, contributors — above a certain threshold of submitted translations, all contributors independently of the number of approved translations, and new signups, along with the number of enabled projects, project completion, and the resulting `Community health score`.
 
 Each cell also displays the month-over-month change, making it possible to compare the current values against the previous month. Use the `Filter teams` search box to narrow the table down to specific locales.
 
-### Edit locales
+#### Edit locales
 
 The `Edit Locales` button toggles a view where the user selects which locales are displayed in the dashboard and its charts. Move one or more locales into the target list to save preferred locales for display. If no locales are selected, the dashboard prompts the user to choose at least one before any data is shown. Press `Back` to toggle back to the dashboard view.
 
-### Show scores
+#### Show scores
 
 By default the table shows the raw value of each metric. Click the `Show scores` button to switch to the score view, which instead displays the individual component scores that add up to the `Community health score`. Click the button again (now labeled `Show default`) to return to the raw values.
 
-## Community health score chart
+#### Key projects
+
+Two of the metrics above — number of enabled projects and project completion — are not measured across everything a team works on, but only across the projects marked as **key projects**. Enabled projects is scored out of a share of total projects, while project completion is the share of translated strings within the key projects the team is enabled for. Disabled projects never count toward either metric, and project completion ignores system projects and projects that are not visible.
+
+Marking a project as a key project is available in the Django admin (`/a/base/project/`) using the `Is chs project` flag. A team then counts as enabled for that project as soon as the team is added to it, the same way as for any other project (see [Adding a new project](adding-new-project.md)).
+
+If no project is marked as a key project, both metrics are zero for every team, the `Community health score` loses the points they carry, and no team qualifies for the [monthly health report](#monthly-health-report).
+
+#### Community health score chart
 
 This chart plots the monthly `Community health scores` of each selected locale for the most recent 12 months, along with the average of all selected locales. Hover over a data point in the graph to see each selected locale's score for that month along with the average.
 
-## Pretranslation quality
+### Pretranslation quality
 
 Two charts track the quality of [pretranslations](../localizer/glossary.md#pretranslation) over time, measured as the approval rate of pretranslated strings (the share of pretranslations that reviewers approve rather than reject).
 
-### Team pretranslation quality
+#### Team pretranslation quality
 
 Plots the approval rate of pretranslations for each team, making it possible to compare pretranslation quality across locales and spot teams whose custom machine translation models may need attention.
 
-### Project pretranslation quality
+#### Project pretranslation quality
 
 Plots the approval rate of pretranslations for each project, highlighting which projects produce the most and least reliable pretranslations.
+
+## Monthly health report
+
+Once a month, after new locale health snapshots are collected, Pontoon compares the two most recent `Community health scores` of each team and reports the ones that moved the most. The report lists every affected team with its score from the previous month, its current score, and the change between them, expressed as a percentage of the previous score. Teams are ordered by the size of that change, largest first, regardless of whether they went up or down.
+
+A team is included when it is enabled for at least one [key project](#key-projects), has a snapshot in both months, and its score changed by at least 2%. The threshold can be changed through the `MONTHLY_HEALTH_REPORT_CHS_THRESHOLD` environment variable.
+
+Staff users receive the report as a notification in Pontoon. To also receive it by email, enable `Monthly health report` in the `Email` section of your [settings](../localizer/users.md).

documentation/docs/dev/deployment.md

@@ -461,6 +461,20 @@ at any time using the `--force` option.
 ./manage.py send_monthly_activity_emails
 ```
 
+### Send Monthly Health Report Notifications & Emails
+
+This job sends a summary of monthly health report to admin users via
+notifications & email. It is designed to run on a specific day of the
+month but can be forced to run at any time using the `--force` option.
+
+Note: this command should only be run after the `collect_chs_snapshots`
+command, as community health reports depend on the most recent locale
+health snapshots available.
+
+``` bash
+./manage.py send_monthly_health_report_notifications_and_emails
+```
+
 ### Send Onboarding Emails
 
 Pontoon sends onboarding emails to new users. The first one is sent upon

pontoon/base/migrations/0128_userprofile_monthly_health_report.py

@@ -0,0 +1,17 @@
+# Generated by Django 5.2.15 on 2026-08-07 18:25
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("base", "0127_add_missing_sections"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="userprofile",
+            name="monthly_health_report",
+            field=models.BooleanField(default=False),
+        ),
+    ]

pontoon/base/models/user_profile.py

@@ -26,6 +26,7 @@ class UserProfile(models.Model):
     email_communications_enabled = models.BooleanField(default=False)
     email_consent_dismissed_at = models.DateTimeField(null=True, blank=True)
     monthly_activity_summary = models.BooleanField(default=False)
+    monthly_health_report = models.BooleanField(default=False)
 
     # Theme
     class Themes(models.TextChoices):

pontoon/base/static/css/style.css

@@ -787,11 +787,6 @@ header .right .select .menu {
   color: var(--white-1);
 }
 
-.notifications .menu li.notification-item .description ul {
-  list-style: disc;
-  padding: 10px 0 10px 20px;
-}
-
 .notifications .menu li.notification-item .message {
   padding: 10px;
 }
@@ -819,13 +814,31 @@ header .right .select .menu {
   padding: 0;
 }
 
-/* Users can include HTML tags in their messages */
+.notifications .menu li.notification-item .description {
+  color: var(--translation-color);
+}
+
+.notifications .menu li.notification-item .description .positive {
+  color: var(--status-translated);
+}
+
+.notifications .menu li.notification-item .description .negative {
+  color: var(--status-error);
+}
+
+/* Notifications can include HTML tags in their body */
 .notifications .menu li.notification-item .message h1,
 .notifications .menu li.notification-item .message h2,
 .notifications .menu li.notification-item .message h3,
 .notifications .menu li.notification-item .message h4,
 .notifications .menu li.notification-item .message h5,
-.notifications .menu li.notification-item .message h6 {
+.notifications .menu li.notification-item .message h6,
+.notifications .menu li.notification-item .description h1,
+.notifications .menu li.notification-item .description h2,
+.notifications .menu li.notification-item .description h3,
+.notifications .menu li.notification-item .description h4,
+.notifications .menu li.notification-item .description h5,
+.notifications .menu li.notification-item .description h6 {
   color: var(--white-1);
   font-size: 14px;
   font-style: normal;
@@ -834,23 +847,28 @@ header .right .select .menu {
   text-transform: none;
 }
 
-.notifications .menu li.notification-item .message h1 {
+.notifications .menu li.notification-item .message h1,
+.notifications .menu li.notification-item .description h1 {
   font-size: 18px;
 }
 
-.notifications .menu li.notification-item .message h2 {
+.notifications .menu li.notification-item .message h2,
+.notifications .menu li.notification-item .description h2 {
   font-size: 16px;
 }
 
-.notifications .menu li.notification-item .message p {
+.notifications .menu li.notification-item .message p,
+.notifications .menu li.notification-item .description p {
   padding: 10px 0;
 }
 
-.notifications .menu li.notification-item .message ol {
+.notifications .menu li.notification-item .message ol,
+.notifications .menu li.notification-item .description ol {
   padding-left: 20px;
 }
 
-.notifications .menu li.notification-item .message ul {
+.notifications .menu li.notification-item .message ul,
+.notifications .menu li.notification-item .description ul {
   list-style: disc;
   padding: 10px 0 10px 20px;
 }

pontoon/contributors/templates/contributors/settings.html

@@ -110,6 +110,18 @@ <h3>Email</h3>
             title='Receive monthly activity summary email',
             help="Get an email summary of your personal activity and the activity within your teams in the last month.")
           }}
+
+          {% if user.is_superuser %}
+            {{
+              Checkbox.checkbox(
+              'Monthly health report',
+              class='field monthly-health-report',
+              attribute='monthly_health_report',
+              is_enabled=user.profile.monthly_health_report,
+              title='Receive monthly health report email',
+              help="Get an email report of community health score metrics in the last month.")
+            }}
+          {% endif %}
         </div>
       </section>
 

pontoon/contributors/templates/contributors/widgets/notifications_menu.html

@@ -87,7 +87,7 @@
           {% endif %}
 
           {% if description and (description.startswith("Unreviewed suggestions") or notification.verb == "has reviewed suggestions" or notification.verb == "ignore") %}
-            <span class="description">{{ description|safe }}</span>
+            <div class="description">{{ description|safe }}</div>
 
             {{ notification_time(notification) }}
           {% elif target and target.resource %}

pontoon/contributors/views.py

@@ -294,6 +294,7 @@ def toggle_user_profile_attribute(request):
         # Email settings
         "email_communications_enabled",
         "monthly_activity_summary",
+        "monthly_health_report",
         # Editor settings
         "quality_checks",
         "force_suggestions",

pontoon/insights/tests/test_utils.py

@@ -0,0 +1,169 @@
+import pytest
+
+from dateutil.relativedelta import relativedelta
+
+from django.utils import timezone
+
+from pontoon.insights.utils import get_monthly_health_report
+from pontoon.settings.base import MONTHLY_HEALTH_REPORT_CHS_THRESHOLD
+from pontoon.test.factories import (
+    LocaleFactory,
+    LocaleHealthSnapshotFactory,
+    ProjectFactory,
+    ProjectLocaleFactory,
+    ResourceFactory,
+    TranslatedResourceFactory,
+)
+
+
+def anchors():
+    current = timezone.now().date()
+    previous = current.replace(day=1) - relativedelta(days=1)
+    return current, previous
+
+
+@pytest.mark.django_db
+def test_get_monthly_health_report_reports_locales_above_threshold():
+    current_anchor, previous_anchor = anchors()
+
+    locale_a = LocaleFactory.create(code="kg", name="Klingon")
+    locale_b = LocaleFactory.create(code="gs", name="Geonosian")
+    locale_c = LocaleFactory.create(code="vu", name="Vulcan")
+
+    project = ProjectFactory.create(slug="project", name="Project", repositories=[])
+    resource = ResourceFactory.create(project=project, path="resource.po")
+    for locale in (locale_a, locale_b, locale_c):
+        ProjectLocaleFactory.create(project=project, locale=locale)
+        TranslatedResourceFactory.create(resource=resource, locale=locale)
+
+    LocaleHealthSnapshotFactory.create(
+        locale=locale_a, created_at=previous_anchor, chs=50
+    )
+    LocaleHealthSnapshotFactory.create(
+        locale=locale_a, created_at=current_anchor, chs=50.5
+    )
+    LocaleHealthSnapshotFactory.create(
+        locale=locale_b, created_at=previous_anchor, chs=50
+    )
+    LocaleHealthSnapshotFactory.create(
+        locale=locale_b, created_at=current_anchor, chs=60
+    )
+    LocaleHealthSnapshotFactory.create(
+        locale=locale_c, created_at=previous_anchor, chs=40
+    )
+    LocaleHealthSnapshotFactory.create(
+        locale=locale_c, created_at=current_anchor, chs=35
+    )
+
+    report = get_monthly_health_report()
+
+    assert report["locale_rows"] == [
+        {
+            "locale": locale_b,
+            "previous_chs": 50,
+            "current_chs": 60,
+            "delta": 10,
+            "percentage": 20,
+        },
+        {
+            "locale": locale_c,
+            "previous_chs": 40,
+            "current_chs": 35,
+            "delta": -5,
+            "percentage": -12.5,
+        },
+    ]
+
+
+@pytest.mark.django_db
+def test_get_monthly_health_report_excludes_locales_without_key_projects_and_hidden_projects():
+    current_anchor, previous_anchor = anchors()
+
+    locale_a = LocaleFactory.create(code="kg", name="Klingon")
+    project = ProjectFactory.create(slug="project", name="Project", repositories=[])
+    resource = ResourceFactory.create(project=project, path="resource.po")
+    ProjectLocaleFactory.create(project=project, locale=locale_a)
+    TranslatedResourceFactory.create(resource=resource, locale=locale_a)
+
+    LocaleHealthSnapshotFactory.create(
+        locale=locale_a, created_at=previous_anchor, chs=50, key_projects_enabled=0
+    )
+    LocaleHealthSnapshotFactory.create(
+        locale=locale_a, created_at=current_anchor, chs=60, key_projects_enabled=0
+    )
+
+    assert get_monthly_health_report()["locale_rows"] == []
+
+
+@pytest.mark.django_db
+def test_get_monthly_health_report_excludes_locales_without_previous_snapshot():
+    current_anchor, _ = anchors()
+
+    locale_a = LocaleFactory.create(code="kg", name="Klingon")
+    project = ProjectFactory.create(slug="project", name="Project", repositories=[])
+    resource = ResourceFactory.create(project=project, path="resource.po")
+    ProjectLocaleFactory.create(project=project, locale=locale_a)
+    TranslatedResourceFactory.create(resource=resource, locale=locale_a)
+
+    LocaleHealthSnapshotFactory.create(
+        locale=locale_a, created_at=current_anchor, chs=60
+    )
+
+    assert get_monthly_health_report()["locale_rows"] == []
+
+
+@pytest.mark.django_db
+def test_get_monthly_health_report_reports_locales_without_previous_chs_as_full_gain():
+    current_anchor, previous_anchor = anchors()
+
+    locale_a = LocaleFactory.create(code="kg", name="Klingon")
+    project = ProjectFactory.create(slug="project", name="Project", repositories=[])
+    resource = ResourceFactory.create(project=project, path="resource.po")
+    ProjectLocaleFactory.create(project=project, locale=locale_a)
+    TranslatedResourceFactory.create(resource=resource, locale=locale_a)
+
+    LocaleHealthSnapshotFactory.create(
+        locale=locale_a, created_at=previous_anchor, chs=0
+    )
+    LocaleHealthSnapshotFactory.create(
+        locale=locale_a, created_at=current_anchor, chs=60
+    )
+
+    (locale_row,) = get_monthly_health_report()["locale_rows"]
+    assert locale_row["delta"] == 60
+    assert locale_row["percentage"] == 100
+
+
+@pytest.mark.django_db
+def test_get_monthly_health_report_uses_latest_snapshot_of_the_month():
+    current_anchor, previous_anchor = anchors()
+
+    locale_a = LocaleFactory.create(code="kg", name="Klingon")
+    project = ProjectFactory.create(slug="project", name="Project", repositories=[])
+    resource = ResourceFactory.create(project=project, path="resource.po")
+    ProjectLocaleFactory.create(project=project, locale=locale_a)
+    TranslatedResourceFactory.create(resource=resource, locale=locale_a)
+
+    LocaleHealthSnapshotFactory.create(
+        locale=locale_a, created_at=previous_anchor.replace(day=1), chs=10
+    )
+    LocaleHealthSnapshotFactory.create(
+        locale=locale_a, created_at=previous_anchor, chs=50
+    )
+    LocaleHealthSnapshotFactory.create(
+        locale=locale_a, created_at=current_anchor, chs=60
+    )
+
+    (locale_row,) = get_monthly_health_report()["locale_rows"]
+    assert locale_row["previous_chs"] == 50
+    assert locale_row["delta"] == 10
+
+
+@pytest.mark.django_db
+def test_get_monthly_health_report_reports_the_previous_month():
+    reported_month = timezone.now().date().replace(day=1) - relativedelta(months=1)
+    report = get_monthly_health_report()
+
+    assert report["month"] == reported_month.strftime("%B")
+    assert report["year"] == reported_month.year
+    assert report["threshold"] == MONTHLY_HEALTH_REPORT_CHS_THRESHOLD