feat(otel): instrument LLM providers for latency and prompt visibility

[jaegeral]

"Note: This PR is stacked on top of PR #3797 (Phase 2). It contains those changes to ensure the
telemetry stack is fully initialized."

[jaegeral]

/gemini review

[gemini-code-assist]

Code Review

This pull request implements OpenTelemetry instrumentation for LLM providers, OpenSearch, and SQLAlchemy, while introducing a privacy-focused span processor for PII redaction and a logging filter for trace correlation. Feedback focused on ensuring that exceptions are properly recorded within the new LLM spans and addressing a potential KeyError in the logging filter when telemetry is disabled.

[gemini-code-assist]

When an exception occurs during the LLM request, it is important to record the exception and set the span status to ERROR to ensure visibility in the trace waterfall. Currently, this exception block only raises a ValueError without updating the OpenTelemetry span.

Suggested change

	except (KeyError, IndexError, requests.RequestException) as e:
	raise ValueError(
	f"Error JSON parsing text: {formatted_data}: {error}"
	) from error
	f"Error generating text with Azure API: {e}"
	) from e
	except (KeyError, IndexError, requests.RequestException) as e:
	span.record_exception(e)
	span.set_status(telemetry.get_status_code("ERROR"), description=str(e))
	raise ValueError(
	f"Error generating text with Azure API: {e}"
	) from e