This message was deleted.

when the agents count is 200

Why do you run this many agents?

we are using in signoz in EKS our cluster will scale upto 400 nodes at peak Signoz we are mainly using to collect logs from pods

You don't need hundreds of signoz-otel-collector. What is your typical ingest volume at peak? and what are the resources given to ClickHouse?

i am using below overwrite file

global:
  storageClass: gp2-resizable
  cloud: aws

clickhouse:
   cloud: aws
   resources:
     limits:
       memory: 16Gi
       cpu: 6000m
   zookeeper:
     replicaCount: 3
     resources:
       limits:
         memory: 1Gi
         cpu: 1000m
   layout:
     shardsCount: 2
     replicasCount: 1
   persistence:
     size: 150Gi
   podAnnotations:
     prometheus/endpoint: /metrics
     prometheus/port: 9363
     prometheus/metrics: true
     prometheus/events: true
     prometheus/asynchronous_metrics: true
   settings:
     prometheus/endpoint: /metrics
     prometheus/port: 9363
     prometheus/metrics: true
     prometheus/events: true
     prometheus/asynchronous_metrics: true
     # Cold storage configuration

    # Clickhouse logging config
   logger:
      # -- Logging level. Acceptable values: trace, debug, information, warning, error.
      level: debug


k8s-infra:
  # -- Whether to enable K8s infra monitoring
  enabled: true
  otelAgent:
    resources:
      requests:
        cpu: 100m
        memory: 100Mi
      limits:
        cpu: 250m
        memory: 500Mi

otelCollector:
  replicaCount: 15
  resources:
    requests:
      cpu: 1000m
      memory: 4000Mi
    limits:
      cpu: 2000m
      memory: 8000Mi

What is your ClickHouse CPU usage when the pods get OOM Killed?

it hits the maximum CPU of 12 (6 each in clickhouse instance) which is the limit we defined do we need to modify the resource sizes of clickhouse ? @Srikanth Chekuri

I am asking for the Usage stats of both the CPU and memory of ClickHouse pods when the issue occurred not the limits.

i am saying it hit the maximum CPU , maximum memory at that time

It wasn't clear before you edited the message

i can increase clickhouse resources to 16 CPU, 32 GB RAM what are the ideal number of collectors and their resource limits for my case ?

It's not the number of collectors. 15 collectors are more than enough. You need to adjust the resources. I'd suggest 3 shards and 8 CPUs each. After some point you may want to bring in kafka to queue your data.

Thanks, will try with this overwrite file can you give me a link or resource to use kafka queueing in signoz

global:
  storageClass: gp2-resizable
  cloud: aws

clickhouse:
   cloud: aws
   resources:
     limits:
       memory: 16Gi
       cpu: 8000m
   zookeeper:
     replicaCount: 3
     resources:
       limits:
         memory: 1Gi
         cpu: 1000m
   layout:
     shardsCount: 3
     replicasCount: 1
   persistence:
     size: 150Gi
   podAnnotations:
     prometheus/endpoint: /metrics
     prometheus/port: 9363
     prometheus/metrics: true
     prometheus/events: true
     prometheus/asynchronous_metrics: true
   settings:
     prometheus/endpoint: /metrics
     prometheus/port: 9363
     prometheus/metrics: true
     prometheus/events: true
     prometheus/asynchronous_metrics: true
     # Cold storage configuration

    # Clickhouse logging config
   logger:
      # -- Logging level. Acceptable values: trace, debug, information, warning, error.
      level: debug


k8s-infra:
  # -- Whether to enable K8s infra monitoring
  enabled: true
  otelAgent:
    resources:
      requests:
        cpu: 100m
        memory: 100Mi
      limits:
        cpu: 250m
        memory: 500Mi

otelCollector:
  replicaCount: 15
  resources:
    requests:
      cpu: 1000m
      memory: 2000Mi
    limits:
      cpu: 2000m
      memory: 4000Mi