Plotting differences between

[DavidSagan]

In Tao there are three lattices (for a given universe but I am not worrying about multiple universes here): "model", "design", and "base". There needs to be a way to plot things like orbits, Twiss, etc. for the different lattices and there needs to be a way to plot differences. EG: model - design.

Additionally, when data measurements thrown in, there can be the "measured" data and the "reference" data (data taken with different conditions like say a kicker turned on). Then a user
might want to plot something like:
(measured - reference) - (model - design)

[coldfix]

I think I'd like to handle different saved models on the GUI end and ignore tao's internal notion of model/design/reference.

This seems to be more flexible and generic by just using a dict instead of three special cases - and it would be applicable to madx as well.

[DavidSagan]

This is a design decision that needs some thought. The model/design/reference construct is tied in with optimization and is also tied in with how "data" and "variables" are organized in tao. For example, the data and variable structures have "model", "design", and "reference" components. And the computation of the merit function is affected by whether the user, for example, is trying to optimize just the "design" lattice or trying to fit to some orbit difference ("measured" - "reference").

One option would be to implement on the python side the ability to define an array of models (using a dict or whatever): model1, model2, model3, etc. and then, if tao is used, make the assoiation model1 -> model, model2 -> design, model3 -> base. The same thing can with measurements. On the python side you would have measured1, measured2, etc and if tao is used, make the association measured1 -> meas, measured2 -> ref.

What do you think?

[coldfix]

Okay, we will have to consider this more carefully.

My concern is that if we push into this too fast, we might end up with a lot of code managing the interaction with tao - while similar results that could be achieved on the python side (except for the performance overhead of the communication + python itself) in a more generic way that also benefits MAD-X users.

On the other hand, I don't know enough about how this is tied in with tao. So far, I've seen it as a way to let every element/data/variable hold three sets of values associated with a specific state of the lattice (current/original/saved). The main benefit that I know of over doing it in python is that setup cost of certain optimizations will be a lot cheaper.

Anyway, this is something that I won't be tackling until in about 2-3 months when I'm back in Heidelberg unless you want to give it a try.

[DavidSagan]

This is a good point. You will definitely want to be able to do optimizations on the python side. And you will want to have, on the python side, an array of models and an array of measurement structures. For a user who is doing something like an orbit response matrix (ORM) analysis the number of models and measurements will be approximately equal to the number of correctors in the machine. That is, there could be many models and measurements used.

What I suggest is to have a user settable "name" component for the model and measurement structures. The default, if using tao, would be to be to have three models named "design", "base", and "theory", and two measurements named "measured" and "reference".

And no matter what a user is using, be it tao or mad, the ability to plot differences (EG model - design) will be needed.

Thoughts?

[coldfix]

I came across this problem when implementing matching dialogs (#18, WIP), and so I had some thoughts, but I'm not entirely sure.

I'm now pretty sure needs to be solved on the python side and it concerns pretty much every dialog that lets you change settings. There is always a question of whether and how the actions should be cancellable. I'm thinking that maybe we could have something like a revision based model, where when you press "Save" in a dialog, you "commit", i.e. create a discrete restore point as a base model that you can go back to later on.

Of course when you go back to a previous revision, you should be able to create a new revision with this same parent, so the structure wouldn't be quite as linear as e.g. SVN, but also not quite as flexible as git (don't need revisions with multiple parents for now).

So basically, we have a list of objects (Name, Time, Data, Parent) from which the user can select by name. When you press Save in a dialog, you create a new revision, when you press Revert you revert data to the current base point.

When you plot a curve / show data, you can select whether the data should come from a previous revision or from the active (changing) model. When you ask for data that is not yet cached (it's not possible to pre-emptively save all the curve data and element data for all revisions), the application should seemlessly switch back to the old configuration to retrieve the data and forth again.

It may be overengineering - but in the end, I find it pretty clean and intuitive. However, it will be a lot of work, in particular the last part about switching back and forth.

What do you think?

[DavidSagan]

There are several issues here. First, in my experience, having a tree of models is not useful. The majority of the time a user is using only one or two models. The cases where more models are needed are during optimization where the user wants to have a backup list of intermediate states and when the user has multiple data sets (such as orbit response matrix (ORM) analysis) where each model corresponds to the conditions under which a given measurement is made. So just having an array of models should be fine.

There is a deeper problem: If you are going to save model states on the python side that potentially means saving all the information about the entire lattice. This is a huge undertaking which has to be repeated for each backend simulation code accommodated. And with MAD, remember that the global variables are part of the lattice and will have to be saved along with everything else. My thought is that it is too complicated to try to save models on the python side. If the backend can do it (and MAD-NG will be able to do this), then let the backend save the model and if the backend cannot, the user will be limited to just one model.

When you plot a curve / show data, you can select whether the data should come from a previous revision or from the active (changing) model. When you ask for data that is not yet cached (it's not possible to pre-emptively save all the curve data and element data for all revisions), the application should seemlessly switch back to the old configuration to retrieve the data and forth again.

But a user will want to be able to plot linear combinations like:

  (measured - ref_measured) - (model - design)

Notice that here "measure" and "ref_measured" are derived from measurement. So you don't want to restrict plotting to just one model.