good news, the signoz cluster hasn't crashed due t...
# support
good news, the signoz cluster hasn't crashed due to low ephemeral storage in 12h, bad news nothing is getting into Signoz. The troubleshoot tool is getting a 404 back. It is entirely possible this is my fault, because I wanted to SSL-terminate those collector endpoints and make them available outside the cluster. To do so, I added a Traefik IngressRoute (a reverse proxy with SSL-termination). The configuration looks correct in the dashboard, but I also get a 404 when curling those endpoints. Is it possible the reverse proxy is interfering with collection? Is there any way to accomplish what I'd like?
hey @Brian Richardson the troubleshooting tool returning returning 404 means that your SigNoz cluster is not reachable by the application - and hence you dont see any data in SigNoz just wanted to check - you have set address as suggseted in this grid right? -
well no, that's what I am getting at. I am proxying the request into that port via a reverse proxy
I have tested via kubectl port-forward and it works, so sadly it seems that Traefik can't forward either Protobuf or gRPC traffic correctly, over http or https.
I don't suppose you would know of a reverse proxy configuration that would work?
I don't but paging @Prashant Shahi if he has any insights on this
@Brian Richardson I worked out nginx config to terminate ssl and do auth before otel-collectors. One difference, I am routing grpc request to relevant namespace based on response from auth. Attaching the nginx config if it helps
you can remove
section and directly use
Copy code
grpc_pass <grpc://my-release-signoz-otel-collector.platform.svc.cluster.local:4317>;
you can also remove the auth section
Or you can use keycloak as mentioned in the blog, though a bit harder to setup
that's excellent, tyvm