[aguest-kc]

Description:

Set toolChoice to any and explicitly call execute_filter on the last loop.

Technical Details:

Requirements for PR Merge:

1. Unit & integration tests updated
2. Data validation completed (examples listed below)
   1. Does this work well with the current frontend? Or is the frontend aware of a needed change?
   2. Is performance impacted in the changes (e.g., API, pipeline, downloads, etc.)?
   3. Is the expected data returned with the expected format?
3. Jira Ticket(s)
   1. DEV-16408

Explain N/A in above checklist:

[zachflanders-frb]

Evals

nova pro

• cases: 61/177 -> 48/177
• score: 0.928 -> 0.803

This is unexpectantly resulting in a pretty significant regression. A bunch of the cases in the eval framework are failing with:

13:43:03 - Filter Search execution failed for case '188': An error occurred (ModelErrorException) when calling the Converse operation: Model produced invalid sequence as part of ToolUse. Please refer to the model tool use troubleshooting guide.
Traceback (most recent call last):
  File "/usaspending-api/usaspending_api/llm/evals/assistants/filter_search.py", line 110, in run_eval_case
    events = [event async for event in assistant.search(case.input["query"])]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usaspending-api/usaspending_api/llm/assistants/filter_search.py", line 231, in search
    response = await client.converse(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/.venv/lib/python3.12/site-packages/aiobotocore/client.py", line 412, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ModelErrorException: An error occurred (ModelErrorException) when calling the Converse operation: Model produced invalid sequence as part of ToolUse. Please refer to the model tool use troubleshooting guide.

We'll want to see what is going on with this and see if can be resolved before we merge this.

[aguest-kc]

Using Nova Pro I'm seeing the following results in the eval framework:

QAT

• Pass count: 22/177
• Score: 0.60

Tool choice branch:

• Pass count: 23/177
• Score: 0.89

[zachflanders-frb]

Evals

Nova Pro

• 74/177 cases (Up from 72)
• 0.954 (down from 0.960)

This had one error related to the DEFC filter, but this one has been failing and should be addressed in a subsequent effort.

[zachflanders-frb]

Auto is the default, so we don't need to explicitly set it here (although it doesn't hurt)