Prometheus query failures with Google Managed Prometheus

[clearclaw]

I've installed Pyrra in GKE using @rlex' Helm chart, atop Google's Managed Prometheus offering. Thus: --config-map-mode=true (see other thread on that and request for support of Google's custom CRDs -- pretty please!).

However, it looks like Pyrra is never able to successfully query Prometheus.

All the Query and QueryRange calls in the UI return 500s (so the Availability and ErrorBudget elements never populate), and I get log entries like this for Pyrra:

level=warn service=objective ts=2024-02-12T20:25:05.440029807Z caller=main.go:1017 msg="failed to query alerts" query="ALERTS{slo=~\".+\"}" err="prometheus query: bad_response: v1.apiResponse.Error: ReadString: expects \" or n, but found {, error found in #10 byte of ...|\"error\": {\n    \"code|..., bigger context ...|{\n  \"error\": {\n    \"code\": 400,\n    \"message\": \"Invalid JSON pay|...

The internal Prometheus service is being passed to the Pyrra container, --prometheus-url=http://frontend.gmp-system.svc:9090, which is the exact value Grafana is happy with etc. Testing via cURL is also happy:

$ curl -g --data-urlencode 'query=ALERTS{slo=~".+"}' http://localhost:9090/api/v1/query -X POST
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"ALERTS","alertname":"SLOMetricAbsent",...

(Using POST simply because the UI does -- same result with GET and a query string)

Or running directly on the Pyrra pod:

$ wget -O - 'http://frontend.gmp-system.svc:9090/api/v1/query?query=ALERTS%7Bslo%3D~%22.%2B%22%7D'
Connecting to frontend.gmp-system.svc.cluster.local:9090 (10.72.142.54:9090)
writing to stdout
{"status":"success","data":{"result...

PEBKAC? I'm scratching my head here.

[clearclaw]

Oh, GMP doesn't allow partial_response in queries.

[clearclaw]

Bug filed: #1063