Correction to resolve the event tag once per exported event #5185 - #5187
Conversation
…ine#5185 The output and formatting engine resolved the event tag twice for every exported event, once in _ExportEvents for the event filter and again in _FlushExportBuffer for the output module. The event tag is now resolved once and carried through PsortEventHeap, or resolved when the export buffer is flushed when no event filter is used. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Haven't dug into it much, but the failing check here ( A quick look points at |
|
I opt to change the path spec type form TSK to EXT for now. Will take a closer look at the TSK timestamp when time permits #5189 |
|
Changes in #5190 to address the pre-release compatibility issue |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5187 +/- ##
==========================================
- Coverage 85.10% 85.09% -0.01%
==========================================
Files 456 456
Lines 41089 41103 +14
==========================================
+ Hits 34967 34977 +10
- Misses 6122 6126 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| self._number_of_consumed_events = 0 | ||
| self._output_mediator = None | ||
| self._processing_configuration = None | ||
| self._resolve_event_tag_at_flush = True |
There was a problem hiding this comment.
Instead of tracking this state at the object level, I opt to pass this as an argument to _FlushExportBuffer e.g. resolve_event_tag = True/False
There was a problem hiding this comment.
Done.
One choice worth flagging: resolve_event_tag defaults to True on both _ExportEvent and _FlushExportBuffer. _ExportEvents always passes it explicitly so the default only affects other callers, but True costs at most a redundant lookup where False would leave the event tag unresolved. Happy to flip it if you prefer False.
psort -o dynamic output is byte identical before and after, without a filter and with filters on timestamp_desc and tag.
There was a problem hiding this comment.
Thanks,. It does the safe thing by default, that is fine.
|
Thanks for the proposed changes, one suggestion |
…timeline#5185 Addresses review feedback: replaces the _resolve_event_tag_at_flush attribute with a resolve_event_tag argument passed to _ExportEvent and _FlushExportBuffer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixes #5185.
Resolves the event tag once per exported event instead of twice.
Where to look
The two call sites on
main, bothstorage_reader.GetEventTagByEventIdentifer:plaso/multi_process/output_engine.py:256, in_ExportEvents, used for the eventfilter and only consumed when one is set;
plaso/multi_process/output_engine.py:371, in_FlushExportBuffer, used for theoutput module.
Both were introduced together in 65fb698, "Removed event tag index #3714 (#4154)",
which replaced
EventTagIndex.GetEventTagByIdentifierwithreader.GetEventTagByEventIdentiferat each site. In the removedplaso/storage/event_tag_index.pya lookup was a dict hit followed byGetAttributeContainerByIdentifier, and returnedNonewithout touching the databasewhen the storage file had no event tags, so calling it twice was close to free.
Change
The event tag is now resolved in one of two places, depending on whether the event
filter needs it:
carried through
PsortEventHeapto the flush the same wayevent_dataandevent_data_streamalready are;discarded, so deduplicated events no longer resolve a tag at all.
Events released from the time slice buffer are resolved at the point they are
exported, since they do not pass through the read loop.
Output is unchanged in both cases. The second point is a behaviour change in that a
deduplicated event no longer causes a lookup, but such events are never written, so
nothing reaches an output module that did not before.
Effect
Measured against a storage file of 1,296,316 events extracted from a disk image, of
which 1,293,801 are exported and 2,515 are discarded as duplicates. Both versions were
run alternately on the same idle machine, twice each, so that run to run variation is
visible rather than assumed:
That is 342.0 s against 290.4 s on average, or about 15 percent of export, for 1.30
million lookups removed.
The proportion depends on what is being exported. The measurement in the issue, over
134,499 events extracted from
/var/logwhere the event data is cheaper todeserialise and the lookup is therefore a larger share of the total, came out closer
to 10 percent. The number of lookups removed per exported event is one in either case.
Note that these storage files contain no event tags, so each removed lookup is the
inexpensive path: the filter expression is still built and parsed, but
_GetAttributeContainersWithFilterreturns without querying when there are nocontainers of that type. On a storage file that has been tagged the removed lookup
also avoids a query and the deserialisation of a container.
Tests
tests/multi_process/output_engine.pypasses unmodified. GivingPushEventanevent_tag=Nonedefault keeps existing callers working.psort.py -o dynamicoutput is byte identical before and after againsttest_data/psort_test.plaso, with no filter, with a filter ontimestamp_desc, andwith a filter on
tag, the last of which exercises resolving the tag before thefilter is applied.
Full end-to-end suite on Ubuntu 26.04, exit 0.