Add tracing to Nginx Ingresses

[dachrillz]

Nginx can support open telemetry style distributed tracing. We would like to add this functionality as this would increase the observability of the platform.

After some research it turns out that there might be some bugs with ingress-nginx and the open telemetry stack. This work can probably not continue until those issues are resolved.

Some things to consider,

1. make sure that this does not add too much load to the system. Both performance and storage wise.

[nissessenap]

There is currently issues with running otel in ingress-nginx, we can follow the issue here: open-telemetry/opentelemetry-cpp-contrib#141
kubernetes/ingress-nginx#8437

[nissessenap]

Just an update, there is still work ongoing in this area: kubernetes/ingress-nginx#8735
But it looks like it's getting closer and closer and hopefully we will be able to try this out in a month or two.

[landerss1]

Looks like it's only a matter of adding the correct configuration to the opentelemetry attribute in the chart: https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml#L709C3-L709C16. This should only be done for unbox clusters where we are actually using tracing, so the ingress-nginx module should be extended with the extra config variables.