SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.

SigNoz Community

Hii eveyone,

There's a problem happening every now and then with me,
Whenever I use docker compose to run the signoz service, one of the containers is flagged as unhealthy and the service doesn't start.

Sometimes it is the alert manager, sometimes it is otel metrics collector.

I am not able to understand the problem with it.

I even reinstalled docker and signoz both, it worked once and then the problem started to occur again.

Please help

Screenshot from 2022-07-25 16-12-29.png

Hi <@U03J9JBPXS7>!

Can you share logs of unhealthy containers?

Screenshot from 2022-07-25 16-22-23.png

This time it is for the alert manager container

query-service, otel-collector, otel-collector-metrics are dependent on clickhouse

alertmanager is dependent on query-service

Sure I will just search how to do that ! and then I will share.

```docker logs --tail=100 clickhouse-setup_clickhouse_1```

Screenshot from 2022-07-25 16-28-07.png

Screenshot from 2022-07-25 16-28-29.png

OpenSSL `crt` issues.. Those config should have been ignore if not enabled.

anyways, can you comment out the following lines from your local cloned repo and retry?

<https://github.com/SigNoz/signoz/blob/develop/deploy/docker/clickhouse-setup/clickhouse-config.xml#L239-L240>

Did you bring down the cluster as the config won't be updated if same container is used?

I killed all the containers and then restarted.

okay.. can you share clickhouse logs now?

Screenshot from 2022-07-25 16-42-46.png

Screenshot from 2022-07-25 16-43-06.png

<https://signoz.io/docs/operate/docker-standalone/#uninstall|https://signoz.io/docs/operate/docker-standalone/#uninstall>

Screenshot from 2022-07-25 16-51-04.png

Here's the ss of the config file and I have took down the cluster as well but still the same error is there for the alert manager container

image.png

okay, lastly clickhouse and alertmanager logs?

Screenshot from 2022-07-25 17-08-29.png

Screenshot from 2022-07-25 17-08-03.png

If you're looking for error
It is not there this time

I confirm these are the complete logs and no error message is there in the clickhouse logs

Let's catch up over DM to get this resolved 

some one share me the complete list of commands to deploy on eks cluster

Hi <@U03MVJY3HV5> :wave: 

You can find it here:

<https://signoz.io/docs/install/kubernetes/aws/|https://signoz.io/docs/install/kubernetes/aws/>

```clickhouse-operator-74b4b658fc-mmm8n                       2/2     Running    0          33m
my-release-signoz-alertmanager-0                           0/1     Init:0/1   0          33m
my-release-signoz-frontend-7bfb5dc74f-qj7gq                0/1     Init:0/1   0          33m
my-release-signoz-otel-collector-67576d7cf7-pjwb2          0/1     Init:0/1   0          33m
my-release-signoz-otel-collector-metrics-d5ddfbf75-hhzdq   0/1     Init:0/1   0          33m
my-release-signoz-query-service-0                          0/1     Init:0/1   0          33m
my-release-zookeeper-0                                     1/1     Running    0          33m```

<@U03MVJY3HV5> can you share EKS cluster node size and type?

I see `chi`  pods not spinning up, blocking other dependent pods.

<http://kubernetes.io/arch=amd64|kubernetes.io/arch=amd64>

we do have enough cpu and memory
```ip-11-1-64-234.us-east-2.compute.internal   318m         2%     13652Mi         22%
ip-11-1-65-150.us-east-2.compute.internal   1514m        19%    18105Mi         28%
ip-11-1-66-16.us-east-2.compute.internal    297m         1%     12408Mi         20%
ip-11-1-66-238.us-east-2.compute.internal   2460m        15%    19802Mi         15%
ip-11-1-68-107.us-east-2.compute.internal   1074m        6%     9032Mi          7%
ip-11-1-68-242.us-east-2.compute.internal   316m         1%     10678Mi         17%
ip-11-1-69-135.us-east-2.compute.internal   448m         2%     9860Mi          16%
ip-11-1-69-183.us-east-2.compute.internal   328m         2%     8647Mi          6%
ip-11-1-70-133.us-east-2.compute.internal   1918m        12%    16146Mi         12%
ip-11-1-70-39.us-east-2.compute.internal    314m         1%     15469Mi         25%
ip-11-1-70-77.us-east-2.compute.internal    340m         2%     12741Mi         10%
ip-11-1-71-71.us-east-2.compute.internal    180m         2%     21299Mi         34%
ip-11-1-72-204.us-east-2.compute.internal   382m         2%     11212Mi         18%
ip-11-1-72-241.us-east-2.compute.internal   290m         3%     16032Mi         25%
ip-11-1-73-140.us-east-2.compute.internal   873m         5%     10821Mi         17%
ip-11-1-74-130.us-east-2.compute.internal   341m         2%     11476Mi         18%
ip-11-1-74-173.us-east-2.compute.internal   494m         3%     13862Mi         11%
ip-11-1-75-224.us-east-2.compute.internal   537m         3%     10391Mi         8%
ip-11-1-75-37.us-east-2.compute.internal    365m         2%     13888Mi         11%
ip-11-1-76-118.us-east-2.compute.internal   416m         2%     15367Mi         25%
ip-11-1-76-119.us-east-2.compute.internal   616m         3%     21045Mi         16%
ip-11-1-76-20.us-east-2.compute.internal    274m         1%     13665Mi         22%
ip-11-1-76-255.us-east-2.compute.internal   242m         3%     10066Mi         16%
ip-11-1-79-94.us-east-2.compute.internal    466m         2%     9925Mi          7%
ip-11-1-80-200.us-east-2.compute.internal   651m         4%     11963Mi         9%
ip-11-1-80-60.us-east-2.compute.internal    1116m        7%     13236Mi         10%
ip-11-1-82-80.us-east-2.compute.internal    586m         3%     25651Mi         20%
ip-11-1-83-154.us-east-2.compute.internal   429m         2%     12201Mi         20%
ip-11-1-86-221.us-east-2.compute.internal   1084m        6%     16720Mi         13%
ip-11-1-87-11.us-east-2.compute.internal    763m         4%     16127Mi         26%
ip-11-1-87-175.us-east-2.compute.internal   407m         2%     13012Mi         21%
ip-11-1-87-38.us-east-2.compute.internal    381m         2%     13322Mi         10%
ip-11-1-88-18.us-east-2.compute.internal    487m         3%     9539Mi          7%
ip-11-1-88-187.us-east-2.compute.internal   274m         1%     8748Mi          7%
ip-11-1-89-31.us-east-2.compute.internal    872m         5%     8698Mi          6%
ip-11-1-90-83.us-east-2.compute.internal    231m         2%     25157Mi         40%
ip-11-1-92-161.us-east-2.compute.internal   1099m        6%     13193Mi         10%
ip-11-1-93-138.us-east-2.compute.internal   840m         10%    13519Mi         21%
ip-11-1-93-227.us-east-2.compute.internal   1350m        8%     16735Mi         13%
ip-11-1-95-177.us-east-2.compute.internal   919m         5%     15284Mi         25%```

can you try helm install command again but with upgrade?

i have delete everything and tried as well

```aws-storage-class        <http://kubernetes.io/aws-ebs|kubernetes.io/aws-ebs>   Delete          Immediate           true                   5h1m
csdn-aws-storage-class   <http://kubernetes.io/aws-ebs|kubernetes.io/aws-ebs>   Delete          Immediate           true                   55d```

it seems like custom storage class was not installed.

can you use this `override-values.yaml` instead?
```clickhouse:
  cloud: aws
  persistence:
    storageClass: aws-storage-class```

<@U03MVJY3HV5> did that resolve your issue?

Let's get over call and resolve this issue.

OpenTelemetry Operator now supports dotnet :star-struck:
<https://github.com/open-telemetry/opentelemetry-operator/pull/989>

something <@U03KKRXNV9T> had been looking forward :smile:

Just waiting on the CRDs to support it then it’ll be hot

I was surprised they’d add the docker image without adding the support in the CRDs

oh, right.. The other PR is still pending.

<https://github.com/open-telemetry/opentelemetry-operator/pull/976>

^ After this is merged, we should be able to use it.

Sweet didn’t even know this PR existed already

<@U03KKRXNV9T> wanna give the beta release a try with SigNoz :wink:

<!channel>
As we are getting closer to releasing support for logs, wanted to know what is the stack that is being used by people here so that we can provide a seamless transition from your existing logging systems.
:one: Elastic + Logstash
:two: Elastic + any other agent
:three: Promtail + Loki
:four: Splunk
:five: DataDog
:six: Others (please mention in the thread)

<https://github.com/SigNoz/signoz/issues/1139>

So for some of your services you use Graylog and for others you use elasticsearch. Is that correct  <@U03S4CCJPBQ>? Also what made you choose both Graylog and Elasticsearch?

One more point wants to add regarding cost

U should come with support of multiple backend support

So you can store logs for certain days in hot storage i.e clickhouse and then it can be moved to S3. Is that what you are asking?

Can be like loki , entire backend can be on s3.

Anything on ebs or elastic search will be expensive

We are using clickhouse for fast query performance and good compression. We don’t support s3 directly, but you can move data to s3 as soon as you want from clickhouse while compromising on query performance. Though recommendation will be to store the most frequent data on clickhouse for example 1 day and then move to s3.

But it can query data present in S3, but as mentioned it will be slower.

It upto the user to configure retention period.

Means clickhouse can query data presented in s3

yeah, ebs is better than elasticsearch :)

Hey <@U03R0663Q6S>!

Welcome to SigNoz community :tada:

Hey all! When deploying SigNoz to an k8s cluster and exposing it using an ingress

hey <@U03JGSWRM3P> Welcome to SigNoz community:wave:

<@U02SS3ZAMKQ> may have more insights on this

<@U02SS3ZAMKQ> do you have any ideas regarding this?

Hey <@U03JGSWRM3P> :wave:
It is out of scope of SigNoz.

However, you should be able to follow the guide below:
<https://medium.com/flant-com/cert-manager-lets-encrypt-ssl-certs-for-kubernetes-7642e463bbce>

I could solve the problem on my own by creating a second Service/NodePort:

```---
apiVersion: v1
kind: Service
metadata:
  labels:
    <http://app.kubernetes.io/component|app.kubernetes.io/component>: frontend
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: signoz-telemetry
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: signoz
  name: signoz-telemetry-frontend-public
spec:
  type: NodePort
  selector:
    <http://app.kubernetes.io/component|app.kubernetes.io/component>: frontend
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: signoz-telemetry
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: signoz
  ports:
    - name: client
      protocol: TCP
      port: 80
      targetPort: 3301```

I was able to attach this to the AWS Ingress and publish it

```---
apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  name: staging-ingress-monitoring
  namespace: default
  annotations:
    <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: arn:aws:acm:*****************************
    <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: '[{"HTTP": 80}, {"HTTPS":443}]'
    <http://alb.ingress.kubernetes.io/ssl-redirect|alb.ingress.kubernetes.io/ssl-redirect>: '443'
    <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
    <http://alb.ingress.kubernetes.io/subnets|alb.ingress.kubernetes.io/subnets>: subnet-***, subnet-***
spec:
  ingressClassName: alb
  rules:
    - host: dev-monitoring.*******.**
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: signoz-telemetry-frontend-public
                port:
                  number: 80```

Attaching directly internally to port 3301 was not woring for me

it wouldn't require another service, you can use frontend service and ingress alongside Ingress-Nginx Controller.

I see you are using ALB instead.

Did you check out the guide below?
<https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html>

I will take a look, but it is already working :wink:

(As I've written above, just wanted to share my solution for the problem)

yup.. familiar with old wisdom "if it works, do not touch it" :stuck_out_tongue:

&gt; (As I've written above, just wanted to share my solution for the problem)
Yeah, really appreciate you sharing it.

do have a look at the following example:

<https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.4.2/docs/examples/2048/2048_full.yaml>

you could include those similar annotation and class name in Frontend Ingress

Good point, I will check that. Are there any advantages in that solution?

you wouldn't have another service in between so yeah, one less hop.

As far as I understood, my service is not communicating to the signoz service, rather has its on node selector. So shouldn't it be the same?

The selector is picking the pods as target

you are right.. It is using the selector.

Because I've copied some parts of the original svc

now, only advantage would be single service and ingress for frontend as provided from Helm chart.

Yes, that is true. I've not that much experiance with helm and don't know what happens if I change the original service

no worries.. you can keep using the way you are.

Or, you can refer to the following equivalent:
```clickhouse:
  cloud: aws
  installCustomStorageClass: true
  persistence:
    storageClass: gp2-resizable

frontend:
  service:
    type: NodePort
  ingress:
    enabled: true
    classname: alb
    hosts:
      - host: dev-monitoring.*******.**
        paths:
          - path: /
            pathType: ImplementationSpecific
    # tls:
    #   - secretName: dev-monitoring.*******.**
    #     hosts:
    #       - dev-monitoring.*******.**
    annotations:
      <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: arn:aws:acm:*****************************
      <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: '[{"HTTP": 80}, {"HTTPS":443}]'
      <http://alb.ingress.kubernetes.io/ssl-redirect|alb.ingress.kubernetes.io/ssl-redirect>: '443'
      <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: internet-facing
      <http://alb.ingress.kubernetes.io/subnets|alb.ingress.kubernetes.io/subnets>: subnet-***, subnet-***```

because only forwarding from localhost (kubectl) is working