https://signoz.io logo
Powered by
Title
t

Timothy Wigginton

11/15/2022, 5:15 PM
#support I am setting up Signoz in an EKS on a private vpc and I am getting this error in the signoz-clickhouse-operator 
E1115 16:56:11.312079       1 connection.go:105] connect():FAILED Ping(http://***:***@chi-signoz-chart-clickhouse-cluster-0-0.default.svc.cluster.local:8123/). Err: dial tcp: lookup chi-signoz-chart-clickhouse-cluster-0-0.default.svc.cluster.local on 10.100.0.10:53: no such host
p

Pranay

11/15/2022, 5:22 PM
@Timothy Wigginton What version of EKS are you running? Also, here are some older threads on similar issues - https://community-chat.signoz.io/t/442846/hi-all-i-am-trying-to-run-signoz-with-the-help-of-helm-chart Can you check if they are helpful
t

Timothy Wigginton

11/15/2022, 5:32 PM
Thanks for your help. My version is 1.21. My chi signoz cluster does not seem to have issues with it’s liveness probe. I did notice that it pings chi-signoz-chart-clickhouse-cluster-0-0.default.svc.cluster.local:8123, as I understand it default is the namespace but that service is in the platform namespace
p

Pranay

11/15/2022, 5:41 PM
I see, I will let @Prashant Shahi get back if he has any insights on this
p

Prashant Shahi

11/16/2022, 9:31 AM
Hey @Timothy Wigginton 👋 could you please share more details regarding the EKS cluster and the state of the SigNoz components?
you are right. It should not point to 
default
namespace but 
platform
Also, which chart version are you using? Is it the latest one?
t

Timothy Wigginton

11/16/2022, 8:05 PM
Hey @Prashant Shahi yes it’s the latest. The state of the other pods seem to be working correctly I can connect to the frontend and if I use hot rod I can see the traces. Can not see my own traces that I send to the otel collector. The collector and the frontend are exposed using nodeport and we are using an external load balancer.
p

Prashant Shahi

11/17/2022, 10:47 AM
Oh, if that's the case, I don't think the errors from 
clickhouse-operator
should affect anything.
you could try troubleshooting using the docs here: https://signoz.io/docs/install/troubleshooting/
That way you can figure out two things: • whether the SigNoz OtelCollector pod is accessible • whether the SigNoz OtelCollector is healthy and accepting telemetry data
t

Timothy Wigginton

11/17/2022, 4:10 PM
I think we are having firewall issues for grpc requests but I am able to send http traces through postman
{
  "resourceSpans": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {
              "stringValue": "test-with-curl"
            }
          }
        ]
      },
      "instrumentationLibrarySpans": [
        {
          "instrumentationLibrary": {
            "name": "instrumentatron"
          },
          "spans": [
            {
              "traceId": "71699b6fe85982c741292ea3d9c32dd1",
              "spanId": "3c191d03fa8be065",
              "name": "spanitron",
              "kind": 3,
              "droppedAttributesCount": 0,
              "events": [],
              "droppedEventsCount": 0,
              "status": {
                "code": 1
              }
            }
          ]
        }
      ]
    }
  ]
}
Does this look like a correct trace?
I would like to add the gRPC calls do work within the cluster but not when using our loadbalancer. So it seems the collector is healthy
p

Prashant Shahi

11/17/2022, 7:19 PM
Oh, I see. It's likely caused by the LoadBalancer not being configured for gRPC. Could you please share which cloud vendor are you using? Are you configuring custom LoadBalancer or the one provided by default with the K8s cluster by the vendor?
You would likely have to use Ingress with gRPC support.
Here is an example 
override-values.yaml
if you use Nginx Ingress Controller: 
otelCollector:
  ingress:
    enabled: true
    className: nginx
    annotations:
      <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
      <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "GRPC"
    hosts:
      - host: <http://otelcollector.domain.com|otelcollector.domain.com>
        paths:
          - path: /
            pathType: ImplementationSpecific
            port: 4317
t

Timothy Wigginton

11/17/2022, 8:07 PM
We are currently using AWS. We created an application load balancer on aws and setup target groups for the frontend and the collector. I had to expose those services using NodePorts. Would an ingress controller be necessary? AWS supports gRPC calls. Also there is still the issue that the http traces don’t show even though they reach the collector( I enabled debug logging in the collector and can see it receiving the http traces). They reach Prometheus though since I can create a dashboard showing number signoz calls and see it update. Just nothing showing in the traces tab. I really appreciate your help.
Update when looking through the database I found my traces. Why might it not show in the traces tab even though it’s in clickhouse? The hotrod traces did display
p

Pranay

11/18/2022, 12:37 AM
@Timothy Wigginton you should check the Global time filter on the right top corner. The traces may have come for a few mins, hrs - you should ensure that time interval is selected in the 
Global Time filter
on the traces page
3 Views
#supportJoin Slack
Powered by