feat(explore): add PPL and SQL as search modes - #3888
Open
stsawyer1 wants to merge 4 commits into
Open
Conversation
Lucene answers "which events match", but not "how many per host" or "which accounts appear in both timelines". OpenSearch already ships the SQL plugin that can answer those; this exposes it to a sketch. Each language gets its own routes under /explore/ppl/ and /explore/sql/, mirroring how wildcard search is separated from query string search, so the language is fixed by the route and can never be steered by the request body. A dialect object owns what differs between the two (read-only validation, sketch scoping, payload shape, export paging) and one shared resource shell owns what does not. Scoping is the security boundary: every query has the sketch's own indices injected as its source, and any query naming an index outside the sketch is refused rather than rewritten. Read-only enforcement runs before that, so DELETE and DROP never reach the cluster. The languages come from the SQL plugin rather than the search API, so availability is a property of the cluster. The sketch metadata now reports supports_direct_query alongside supports_wildcard, letting a client drop them from its menu instead of offering a query that will fail. Queries reach the cluster through an opensearch-py client, so they fail over between nodes like every other OpenSearch call in Timesketch. The connection setup that OpenSearchDataStore builds inline moves into build_opensearch_client() so both callers derive the cluster the same way. It sends nothing to the cluster, which lets the client be built at startup without a cluster that is still coming up blocking the app. It also asks urllib3 for a connection pool sized to what a process can have in flight, since the default is a single connection per node.
PPL and SQL join Query String and Wildcard in the existing search-mode selector rather than arriving as a separate control, so there is one place to choose how a search is written. They appear only when the sketch metadata reports the cluster can serve them. A direct query is a multi-line pipeline or statement, so it gets a monospace auto-growing editor instead of the one-line Lucene input, and returns a table rather than an event list. The Lucene-only chrome (filter chips, the time filter menu, saved searches) is hidden while one is selected, because a direct query carries only its query string and the enabled timeline ids; leaving those on screen would imply a narrowing that never reaches OpenSearch. The time range therefore moves into the editor itself. Aggregates are a starting point, not an answer, so a cell can be pivoted into the events behind it: the pivot hands the language back to Lucene, raises the chip, and carries the editor's time range over as a datetime_range chip so the investigation does not widen back out. Queries are linted in the browser before they are sent. The rules catch the failure that costs the most time, a query that succeeds while quietly meaning something else, and never block a run, since OpenSearch is the authority on its own syntax.
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
stsawyer1
changed the title
Collaborator
|
Thx for the Pull request, this is a quite a lot of code, we need to discuss this internally how to approach it and ensure maintenance from our side. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #3887
What existing problem does this PR solve?
Lucene (query search/wildcard) returns the list of events which contain the search query. But, it does not allow analysts to perform extraction, transformation, or aggregation as part of the investigation process. An analyst who needs that today leaves Timesketch and queries OpenSearch directly, losing sketch scoping, the ACL check, and the timeline filter. OpenSearch already ships a plugin that can answer those questions; this exposes it to a sketch.
What new feature is being introduced?
PPL and SQL as two additional search modes with dedicated API routes, mirroring how
explore_wildcardis separated fromexplore, so the language is fixed by the route and can never be steered by the request body:On query execution, the sketch's own indices are injected as its source, and a query naming an index outside the sketch is refused rather than rewritten, so a rewrite bug can't silently widen a read.
In the frontend, PPL and SQL have been added to the search mode selector. Selecting these modes swaps the Lucene input for a monospace editor and returns a table instead of an event list. Queries are linted in the browser first, catching the query that succeeds while quietly meaning something else, but never blocking a run. In the results, a cell can be pivoted into the events behind it, handing the language back to Lucene and carrying the time range across.
Requires OpenSearch 3.7.0
Scoping injects a filter ahead of the rest of the query, and the behavior that depends on settled in 3.7.0, when Calcite became the default PPL engine. Older clusters accept the same queries but differ in how a filter ahead of a stats stage is pushed down. Rather than scope correctly on some versions and subtly not on others, the capability probe refuses below 3.7.0 and the UI drops the modes—the same way it drops wildcard on a sketch without wildcard mappings. Sketch metadata reports
supports_direct_queryalongsidesupports_wildcard, cached for five minutes, so an upgraded cluster lights the feature up without a restart.docker/dev/docker-compose.ymlalready runs 3.7.0.docker/release/config.envand the e2e harness pin 2.19.5, so a stock release deployment won't see the feature until the operator upgrades.Overview of changes to existing functions
Three are trivial: route registration, the
supports_direct_querymetadata field, and a call in the app factory that builds the shared client at startup.The substantive one is
timesketch/lib/datastores/opensearch.py. The connection setup thatOpenSearchDataStore.__init__builds inline moves into a newbuild_opensearch_client()so the direct-query package derives the cluster from the same configuration rather than parsingOPENSEARCH_HOSTStwice. That body is existing code moved, not rewritten. Two things are new: the builder sends nothing to the cluster, so a client can be built at startup without a cluster that is still coming up blocking the app; and it sizes the urllib3 pool to what one process can have in flight (OPENSEARCH_POOL_MAXSIZE, default 20), where the opensearch-py default of one keep-alive connection per node makes any two overlapping requests open and discard a throwaway. That second one helps all OpenSearch traffic.If you'd rather not touch the datastore, the package can build its own client at the cost of duplicating the config parsing, and the pooling fix can be split out.
Why I think this can go in the main branch
plugins.pplandplugins.sqlnamespaces, so queries fail over between nodes like every other OpenSearch call. These plugins have identical methods and parameters in the pinned 2.8.0 version of opensearch-py through current (3.2.0).Checks
OPENSEARCH_VERSIONraised for that job.docs/guides/user/search-query-guide.md