Title
b
Brian Richardson
10/04/2022, 1:31 AMgood news, the signoz cluster hasn't crashed due to low ephemeral storage in 12h, bad news nothing is getting into Signoz. The troubleshoot tool is getting a 404 back. It is entirely possible this is my fault, because I wanted to SSL-terminate those collector endpoints and make them available outside the cluster. To do so, I added a Traefik IngressRoute (a reverse proxy with SSL-termination). The configuration looks correct in the dashboard, but I also get a 404 when curling those endpoints. Is it possible the reverse proxy is interfering with collection? Is there any way to accomplish what I'd like?
p
Pranay
10/04/2022, 1:55 AMhey @Brian Richardson the troubleshooting tool returning returning 404 means that your SigNoz cluster is not reachable by the application - and hence you dont see any data in SigNoz
just wanted to check - you have set address as suggseted in this grid right? - https://signoz.io/docs/install/troubleshooting/#signoz-otel-collector-address-grid
b
Brian Richardson
10/04/2022, 2:00 AMwell no, that's what I am getting at. I am proxying the request into that port via a reverse proxy
I have tested via kubectl port-forward and it works, so sadly it seems that Traefik can't forward either Protobuf or gRPC traffic correctly, over http or https.
I don't suppose you would know of a reverse proxy configuration that would work?
p
Pranay
10/04/2022, 4:24 AMI don't but paging @Prashant Shahi if he has any insights on this
a
Ankit Nayan
10/04/2022, 4:40 AM@Brian Richardson I worked out nginx config to terminate ssl and do auth before otel-collectors. One difference, I am routing grpc request to relevant namespace based on response from auth. Attaching the nginx config if it helps
you can remove
access_by_lua_blockgrpc_pass <grpc://my-release-signoz-otel-collector.platform.svc.cluster.local:4317>;you can also remove the auth section
Or you can use keycloak as mentioned in the blog, though a bit harder to setup https://medium.com/opentelemetry/securing-your-opentelemetry-collector-1a4f9fa5bd6f
b
Brian Richardson
10/04/2022, 8:57 PMthat's excellent, tyvm