diff --git a/docs/source/_static/napari_edit_overview.png b/docs/source/_static/napari_edit_overview.png new file mode 100644 index 0000000000..0956f13808 Binary files /dev/null and b/docs/source/_static/napari_edit_overview.png differ diff --git a/docs/source/_static/napari_edit_timeline.mp4 b/docs/source/_static/napari_edit_timeline.mp4 new file mode 100644 index 0000000000..92162375a5 Binary files /dev/null and b/docs/source/_static/napari_edit_timeline.mp4 differ diff --git a/docs/source/_static/napari_edit_timeline_individuals.mp4 b/docs/source/_static/napari_edit_timeline_individuals.mp4 new file mode 100644 index 0000000000..e5f73e24d6 Binary files /dev/null and b/docs/source/_static/napari_edit_timeline_individuals.mp4 differ diff --git a/docs/source/_static/napari_move_keypoints.mp4 b/docs/source/_static/napari_move_keypoints.mp4 new file mode 100644 index 0000000000..6c179e0140 Binary files /dev/null and b/docs/source/_static/napari_move_keypoints.mp4 differ diff --git a/docs/source/_static/napari_remove_keypoints.mp4 b/docs/source/_static/napari_remove_keypoints.mp4 new file mode 100644 index 0000000000..a6db73715c Binary files /dev/null and b/docs/source/_static/napari_remove_keypoints.mp4 differ diff --git a/docs/source/_static/napari_select_keypoints.mp4 b/docs/source/_static/napari_select_keypoints.mp4 new file mode 100644 index 0000000000..e782b9ac99 Binary files /dev/null and b/docs/source/_static/napari_select_keypoints.mp4 differ diff --git a/docs/source/user_guide/gui.md b/docs/source/user_guide/gui.md index 022ad3f821..6cd848555f 100644 --- a/docs/source/user_guide/gui.md +++ b/docs/source/user_guide/gui.md @@ -6,7 +6,8 @@ The `movement` graphical user interface (GUI), powered by our custom plugin for motion tracks. Currently, you can use it to visualise 2D [movement datasets](target-poses-and-bboxes-dataset) as points, tracks, and rectangular bounding boxes (if defined) overlaid on -video frames, as well as to define regions of interest (RoIs). +video frames, to manually [correct tracking errors](target-edit-tracked-data), +and to define regions of interest (RoIs). The `napari` plugin is shipped with the `movement` package starting from version `0.1.0`. To use it, you need to @@ -119,7 +120,8 @@ Dragging and dropping the image file onto the `napari` window (or opening it via the `File` menu) will load the image as a single 2D frame without a slider. -## Load the tracked dataset +(target-load-tracked-data)= +## Load tracked data Now you are ready to load some motion tracks over your chosen background layer. @@ -202,7 +204,7 @@ And for a bounding boxes dataset, you will see a view more like the one below: Note the additional bounding boxes layer that is loaded for bounding boxes datasets. For both poses and bounding boxes datasets, you can toggle the visibility of any of these layers by clicking on the eye icon. - +(target-points-layer)= ### The points layer The points layer shows the data for the current frame. @@ -248,7 +250,7 @@ You can find all the [keyboard shortcuts](napari:guides/preferences.html#shortcu ::: - +(target-tracks-layer)= ### The tracks layer The tracks layer allows us to visualise data before and after the current frame. @@ -312,6 +314,163 @@ You can find all the [keyboard shortcuts](napari:guides/preferences.html#shortcu ::: +(target-edit-tracked-data)= +## Edit tracked data + +Pose estimation models occasionally produce errors such as mis-localised +keypoints and false-positive detections in the background, which automated +post-processing (e.g. {ref}`filtering or interpolation `) +cannot fully resolve. +The `movement` GUI lets you correct these prediction errors by interactively +[editing the keypoints](target-edit-types) on the +[points layer](target-points-layer) created when you +[load the tracked dataset](target-load-tracked-data). + +Editing involves the following parts of the GUI: + +- **The viewer**, where you select and edit keypoints on the points layer. +- **The `Edit tracked data` menu** on the right-hand side of the window, + which holds the [editing options](target-edit-types). Expanding or collapsing it shows or + hides the edit timeline. +- **The [edit timeline](target-edit-timeline)** at the bottom of the + window, which provides a visual summary of all edited frames and lets + you navigate to them. +- **The `Save tracked data` menu**, which lets you + [save your changes](target-save-edits) to a file. + +![napari GUI elements involved in editing tracked data](../_static/napari_edit_overview.png) + +:::{warning} +Editing is currently supported for **poses datasets** only. +While the keypoints in bounding boxes datasets are editable, saving any +changes is not yet supported and will result in an error. +::: + +### Select keypoints + +To start editing: + +1. Select the points layer in the layer list. +2. Use the frame slider to go to the frame you want to correct. +3. In the layer controls panel, activate the *Select points* tool + (the arrowhead icon, or press `S` or `3`). + +With the *Select points* tool active, you can select keypoints directly in the +viewer using one of the following methods: + +- Click a keypoint to select it. +- Hold `Shift` while clicking to select multiple keypoints. +- Drag a selection box to select a group of keypoints. + + + +:::{note} +Editing is disabled whenever the frame slider is controlling a non-time +dimension, e.g. after changing the order of the viewer axes or switching to 3D +view. Return to the 2D view with time as the leading axis to re-enable editing. +::: + +(target-edit-types)= +### Edit types + +When you edit a keypoint, the `Edit tracked data` menu expands and the +[edit timeline](target-edit-timeline) appears at the bottom of the window, +showing a bar for the current frame. The [tracks layer](target-tracks-layer) +updates automatically to stay in sync with the points layer, so trajectories +always reflect the current keypoint positions. + +#### Move keypoints + +To **move** the selected keypoint(s), drag them to the correct position. +Moved keypoints are shown as rings. Their confidence is set to `NaN` +(hover over a point to check), because the model's score no longer applies. + + + +#### Remove keypoints + +To **remove** the selected keypoint(s), click the ✕ icon in the layer controls +panel, or press `Delete`, `Backspace`, or `1`. +Removed keypoints disappear from the viewer, and their confidence scores are +set to `NaN`. + + + +(target-edit-timeline)= +### Edit timeline + +The edit timeline spans all frames of the dataset in the currently selected +`movement` points layer, and shows a vertical bar for every frame +containing at least one edited keypoint. +A dashed line marks the current frame displayed in the viewer and +moves with the frame slider as you navigate through the frames. + +You can interact with the timeline as follows: + +| Action | Effect | +|---|---| +| Click a bar | Go to the corresponding edited frame | +| Scroll up / down | Zoom in / out, centred on the cursor (e.g. to distinguish between bars of adjacent frames) | +| Click and drag | Pan along the timeline (when zoomed in) | +| Double-click | Reset the view to show all frames | + + + +For datasets with multiple individuals, tick the `Display individuals` +checkbox in the `Edit tracked data` menu to split the timeline into one +row per individual. Each row's bars match the colour of that individual's +points, showing which individuals were corrected in which frames. +The checkbox is disabled for single-individual datasets. + + + +(target-save-edits)= +### Save edits + +To save your edits to a file: + +1. Select the `movement` points layer that contains the edits in the layer list. +2. Expand the `Save tracked data` menu and click `Save` + (enabled only when a `movement` points layer is selected). +3. In the dialog that opens, choose a location and enter a file name. + The data is currently saved in `movement`'s native + [netCDF](target-netcdf) format. + +The saved file contains a valid +[movement poses dataset](target-poses-and-bboxes-dataset) with: + +- the corrected `position` values; removed keypoints are stored as `NaN`; +- the `confidence` values, with moved and removed keypoints stored as `NaN`; +- a boolean `edited` data variable, with dimensions + `(time, keypoint, individual)`, storing `True` for every edited keypoint and `False` for all others. + +The `edited` variable is included only if at least one keypoint was edited. +Keypoints or individuals that end up with no data in any frame +(e.g. because all their points were removed) are dropped from the saved +dataset. Frames are never dropped: a frame whose points were all removed +is kept, with `NaN` values. + +To resume editing later, [load](target-load-tracked-data) the saved +`.nc` file in the GUI: all previous edits are restored, so you can pick +up exactly where you left off. + +:::{tip} +Because the edits are saved in the `edited` variable, you can use them in +your analysis, for example to report how many keypoints were corrected by +hand, or to compare results with and without the corrected frames. +See [](target-netcdf) for how to load the file in Python. +::: + (target-define-rois)= ## Define regions of interest