Document how the dashboard scoring works #667
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -14,4 +14,38 @@ Scores for each focus area are computed in [results-analysis](https://github.com | |||||
|
|
||||||
| The "pipeline" for updating scores is as follows: | ||||||
|
|
||||||
| - All of WPT is run on Chrome, Edge, Firefox, and Safari across two different CI systems: Taskcluster and Azure Pipelines. The cadence varies: | ||||||
|
|
||||||
| - Chrome Canary: [every commit](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/tools/ci/tc/tasks/test.yml#L188-L193) | ||||||
|
foolip marked this conversation as resolved.
|
||||||
| - Chrome Stable: [daily](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/tools/ci/tc/tasks/test.yml#L206-L211) | ||||||
| - Edge Dev: [every 3 hours](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/.azure-pipelines.yml#L380) | ||||||
| - Edge Stable: [daily](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/.azure-pipelines.yml#L344) | ||||||
| - Firefox Nightly: [every commit](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/tools/ci/tc/tasks/test.yml#L163-L168) | ||||||
| - Firefox Stable: [daily](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/tools/ci/tc/tasks/test.yml#L175-L180) | ||||||
| - Safari Technology Preview: [every 3 hours](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/.azure-pipelines.yml#L491) | ||||||
| - Safari Stable: [daily](https://github.com/web-platform-tests/wpt/blob/6f491c45bc5182275419be89a0820c2deaddc6b0/.azure-pipelines.yml#L451) | ||||||
|
Comment on lines
+25
to
+32
Member
There was a problem hiding this comment. web-platform-tests/wpt@6f491c4 is over two years old — why are we linking to that commit?
Comment on lines
+31
to
+32
Member
There was a problem hiding this comment. Safari hasn't run on Azure pipelines in a year and a half: web-platform-tests/wpt@8870a21. https://github.com/web-platform-tests/wpt/blob/ec31a461d352a7a67ae972b347aea77d5ca9160b/.github/workflows/safari_technology_preview.yml#L24 and https://github.com/web-platform-tests/wpt/blob/ec31a461d352a7a67ae972b347aea77d5ca9160b/.github/workflows/safari_stable.yml#L24 are the relevant links now. |
||||||
|
|
||||||
| Runs take 1-3 hours to complete. Results are uploaded to wpt.fyi and are made available within minutes of completion. | ||||||
|
foolip marked this conversation as resolved.
Outdated
|
||||||
|
|
||||||
| Delay: 1-6 hours for experimental, 1-27 hours for stable | ||||||
|
|
||||||
| - [results-analysis-cache](https://github.com/web-platform-tests/results-analysis-cache) downloads all runs from wpt.fyi and coverts them into a more efficient format to speed up results scoring. This runs [hourly](https://github.com/web-platform-tests/results-analysis-cache/blob/683fe10f5f58c096e2a0dc3214c53ce949192c39/.github/workflows/results.yml#L7-L9) and takes less than 10 minutes. | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| Delay: 10-70 minutes | ||||||
|
|
||||||
| - [results-analysis](https://github.com/web-platform-tests/results-analysis) fetches results from `results-analysis-cache` and produces the CSV files used by the dashboard. This runs [every 3 hours](https://github.com/web-platform-tests/results-analysis/blob/67c603a9b306c71da37c4798301935318949c45e/.github/workflows/update_gh_pages.yml#L7-L9) and takes a few minutes. | ||||||
|
Member
There was a problem hiding this comment. https://github.com/web-platform-tests/results-analysis/blob/da8fe496b093e0c81cd66274594e2febe6b515fe/build.sh#L20 starts off by running the |
||||||
|
|
||||||
| Delay: 0-3 hours | ||||||
|
foolip marked this conversation as resolved.
|
||||||
|
|
||||||
| - The wpt.fyi frontend always fetches the latest CSV files, so there is no additional delay before the changes appear on the dashboard. | ||||||
|
|
||||||
| ## Troubleshooting | ||||||
|
|
||||||
| It's normal for experimental results to take up 10 hours to update and stable results over 30 hours. If things take longer than this it's time to troubleshoot. | ||||||
|
|
||||||
| First check if new runs are appearing in [wpt.fyi/runs](https://wpt.fyi/runs)? If not, the runs are probably not completing in Taskcluster or Azure Pipelines. Diagnosing why this happens is not always easy, so please [file an issue](https://github.com/web-platform-tests/wpt/issues/new) and poke someone for help. | ||||||
|
|
||||||
| Once wpt.fyi has the results it's rare for anything to go wrong, but it could be in one of the GitHub Actions workflows: | ||||||
|
|
||||||
| - [results-analysis-cache](https://github.com/web-platform-tests/results-analysis-cache/actions) | ||||||
| - [results-analysis](https://github.com/web-platform-tests/results-analysis/actions) | ||||||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.