How to approach callbacks?

[stan-dot]

With just bluesky the callbacks are easy to use - it's unclear how to use them with blueapi.
So far the possible implementations would be:

1. a server listening to the event-model events and manually doing the logic
2. connection to the callbacks from inside RunEngine

For 1 this deprives us of much of the core-bluesky developments which is good and bad as best_effort_callback.py has a good runtime track record but the code is hard to read and contribute to.

For 2 I have no idea how to do this, and how to render plots for instance.

Application of those is also for adaptive scans. If that is in a separate server, that'd need another adapter to work with the plan context.

I have committed some time last year towards exploring this question and made work towards number 2 but haven't made it feature complete.

[callumforrester]

In the interest of upholding saas principles, 1. would be my preference. What do you have to do manually for 1. that 2. does automatically for you? We can provide library code to replace boilerplate. See use of BestEffortCallback with a network in between RunEngine and the callback here.

[stan-dot]

why would saas principles apply to DLS? are they mentioned anywhere in the D-II designs? Or do you just mean 'microservices architectre'?

[callumforrester]

Apologies, wrong acronym, I mean SoC, should actually read what I paste. Although as a matter of interest I think saas does apply to many areas of DLS, such as data analysis and LIMS.

[DiamondJoseph]

I think this discussion may make more sense in the blueapi repository?

I would like us to provide some standard callbacks:

• Insert documents into the document store: always enabled
• Initialise specific workflow(s)
• Wait for a specific workflow to return data
• [Reshape and] forward data to a GUI service

[callumforrester]

By doing DiamondLightSource/blueapi#723 we are basically saying that the blueapi codebase is immutable in prod and dodal et al. are (potentially) mutable in prod. So which do we think those callbacks should live in?

[DiamondJoseph]

Of the ones I mentioned, all immutable so blueapi.

[callumforrester]

Really? Would agree for the first one, not sure about the rest

[stan-dot]

presumably some callbacks will be specific to a given format - that hints they might need to live where the plans do

[DiamondJoseph]

If those things can be extracted into processing workflows then the call to trigger and read back from the workflows remains generic

[stan-dot]

and that generic call it would be inside a plan?

so you mean if it's generic then it wouldn't live where the plans do?