Questions about Receiver using HTTP

[kuro337]

Hey all!

I was using a simple app to test and get a feel of OpenTelemetry - and I was running into an issue with the HTTP Receiver. I wanted to use an HTTP Receiver to confirm whether my Sidecar deployment is correct - but I see this error

2023/09/17 04:51:18 traces export: Post "https://0.0.0.0:4318/v1/traces": http: server gave HTTP response to HTTPS client

I have a Go application and using this - (tried both WithInsecure() and without it

func CreateOTELTracing(ctx context.Context, l *structuredlogger.CustomLogger) (cleanup func(), err error) {
	exp, err := otlptracehttp.New(context.Background(),
		//otlptracehttp.WithInsecure(),
		otlptracehttp.WithEndpoint("0.0.0.0:4318"))
	if err != nil {
		return nil, fmt.Errorf("failed to initialize stdouttrace exporter: %w", err)
	}

	bsp := sdktrace.NewBatchSpanProcessor(exp)
	tp := sdktrace.NewTracerProvider(
		sdktrace.WithSampler(sdktrace.AlwaysSample()),
		sdktrace.WithSpanProcessor(bsp),
	)
	otel.SetTracerProvider(tp)

	cleanup = func() {
		if err := tp.Shutdown(context.Background()); err != nil {
			l.Fatal(err)
		}
	}

	return cleanup, nil
}

When the insecure() is not there - I get above error log

2023/09/17 04:51:18 traces export: Post "https://0.0.0.0:4318/v1/traces": http: server gave HTTP response to HTTPS client

When I add it - there are no logs at all.

If I just have a collector setup and an app in a sidecar , is there a way to confirm whether my image is sending traces to the colleector?

I do not see any logs in the otel-collector or my go app container for successful data being sent to the collector.

I tried adding insecure , insecureSkip , etc. to my collector yaml - but have not found the correct setting yet.

Would appreciate any pointers or recommendations - I am basically trying to setup something minimal where I can have collectors running as sidecars with my app deployments - send that data to a tempo backend , and visualize the traces - and approaching it piece by piece. Thank you!

This is my otel-collector-config.yaml. (have not configured the backend yet for tempo)

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-conf
  labels:
    app: opentelemetry
    component: otel-collector-conf
data:
  otel-collector-config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
            endpoint: 0.0.0.0:4318
    processors:
    extensions:
      health_check: {}
    exporters:
      otlp:
        endpoint: localhost:4317
        tls:
          insecure: true
    service:
      extensions: [health_check]
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [otlp]

This is my deployment with the sidecar go-otel-dep.yaml 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-app-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-app
  template:
    metadata:
      labels:
        app: go-app
    spec:
      containers:
        - name: go-app
          image: ghcr.io/<user>/<image>:latest
          ports:
            - containerPort: 8080

        - name: otel-collector
          image: otel/opentelemetry-collector:latest
          args:
            - "--config=/etc/otel/otel-collector-config.yaml"
          volumeMounts:
            - name: otel-config-vol
              mountPath: /etc/otel

      imagePullSecrets:
        - name: ghcr-secret

      volumes:
        - name: otel-config-vol
          configMap:
            name: otel-collector-conf
            items:
              - key: otel-collector-config.yaml
                path: otel-collector-config.yaml

[bhushanasati25]

Hi! There are two separate things I would change here.

First, 0.0.0.0 is a bind/listen address, not a good client target. It makes sense in the Collector config because the Collector is listening on all interfaces:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

But from your Go app, if the Collector is running as a sidecar in the same Kubernetes pod, point the exporter to localhost:4318 or 127.0.0.1:4318, not 0.0.0.0:4318.

Second, this error means the Go exporter is using HTTPS while the Collector receiver is plain HTTP:

http: server gave HTTP response to HTTPS client

So your Go exporter should be configured like this:

exp, err := otlptracehttp.New(
    context.Background(),
    otlptracehttp.WithEndpoint("localhost:4318"),
    otlptracehttp.WithInsecure(),
)

Or, if your OpenTelemetry Go version supports it, use the URL form explicitly:

exp, err := otlptracehttp.New(
    context.Background(),
    otlptracehttp.WithEndpointURL("http://localhost:4318/v1/traces"),
)

For verifying the sidecar before adding Tempo, I would replace the otlp exporter with the debug exporter. Right now this part is suspicious:

exporters:
  otlp:
    endpoint: localhost:4317
    tls:
      insecure: true

That points the Collector exporter back to localhost:4317, which is usually not what you want unless another backend is running inside the same container/pod. For a minimal test, use:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]

Then trigger one request in your app and check the Collector sidecar logs:

kubectl logs <pod-name> -c otel-collector

If the app is sending traces correctly, you should see trace/span output from the debug exporter.

After that works, replace the debug exporter with your Tempo exporter, using the actual Tempo service DNS name, for example something like:

exporters:
  otlp:
    endpoint: tempo.monitoring.svc.cluster.local:4317
    tls:
      insecure: true

So the short version is: use localhost:4318 from the app container, add WithInsecure() for OTLP/HTTP, use debug exporter first to confirm traces are arriving, and only then wire the Collector exporter to Tempo.