Add errr-handling.md discussion - #35
Conversation
|
I think this is a metric we want to be tracking using prometheus/grafana. We have a workstream starting soon to improve our observability. But this looks a like a good fit, we should have monitoring dashboards and get pinged on teams when it happens |
| - **Option B**: Pipeline is terminated. All data that was processed without failing is saved. | ||
| - **Pros:** Easy to implement, minimal use of resources. | ||
| - **Cons:** Does not tell us if independent branches were passing or failing processing. | ||
| - **Option C**: Pipeline continues, but only datasets that are independent of the failing dataset are processed and saved. No processing of parent classes with dependencies failing processing. |
There was a problem hiding this comment.
I think option C is basically what we already have - apart from the logic of "don't process parent datasets that depend on this failed dataset".
There is a try/except around the self.process_dataset(dataset_id) call, which logs to the prometheus metrics within the except block.
I think I'd continue with this option and refine.
There is merit in saving out any intermediate datasets that were successful, particularly when the most common use case of the pipeline is to "process everything". All of the intermediate datasets are viewing on the UI and are of interest to end users.
3 participants
The file 20260218-error-handling.md contains discussion points on how the time-series processing pipeline should be managed if one or more dataset dependencies fail to process.