https://signoz.io logo
Join Slack
Powered by
Hello, what are the possible causes of having so m...
# support
m
Hello, what are the possible causes of having so many parts in the time_series_v4_1week table?
s
look at the logs of clickhouse. do you see any errors?
m
{"date_time":"1750283204.747538","thread_name":"TCPServerConnection ([#33])","thread_id":"868","level":"Error","query_id":"1866fcd2-5e75-4803-b5a5-6a648a62564c","logger_name":"TCPHandler","message":"Code: 252. DB:Exception Too many parts (3501 with average size of 140.25 KiB) in table 'signoz_metrics.time_series_v4_1week (2f866987-d1a9-4cf0-913b-da5e18e20eb3)'. Merges are processing significantly slower than inserts:
s
the time_series_v4_1week can't have more parts than the v4 table. is there any addl context what happened here?
m
I don't think so. The ingestion of metrics is affected, but I don't see any damage to other tables. I have k8s-infra for the SigNoz cluster and another one, some other application instrumentation, secondary storage with s3, but nothing out of the ordinary. I don't understand why this table has this increase in parts, I appreciate any help.
s
Did something change around 11th or 12th that cause the partition to have so many parts?
m
No new instrumentation this month, I've been seeing the error message for a while now. But now the gap in the metrics has become a problem, is there anything I can do to fix or avoid so many parts in this table?
s
What is you collector config?
Please share your collector config
m
Copy code
config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
            max_recv_msg_size_mib: 16
          http:
            endpoint: 0.0.0.0:4318
      jaeger:
        protocols:
          grpc:
            endpoint: 0.0.0.0:14250
          thrift_http:
            endpoint: 0.0.0.0:14268
            # Uncomment to enable thift_company receiver.
            # You will also have set set enable it in `otelCollector.ports
            # thrift_compact:
            #   endpoint: 0.0.0.0:6831
      httplogreceiver/heroku:
        # endpoint specifies the network interface and port which will receive data
        endpoint: 0.0.0.0:8081
        source: heroku
      httplogreceiver/json:
        # endpoint specifies the network interface and port which will receive data
        endpoint: 0.0.0.0:8082
        source: json
    processors:
      # Batch processor config.
      # ref: <https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/README.md>
      memory_limiter:
        check_interval: 1s
        limit_mib: 9500
        spike_limit_mib: 2000
      batch:
        send_batch_size: 200000
        timeout: 30s
        send_batch_max_size: 250000
      # Memory Limiter processor.
      # If not set, will be overridden with values based on k8s resource limits.
      # ref: <https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/README.md>
      # memory_limiter: null
      signozspanmetrics/delta:
        metrics_exporter: clickhousemetricswrite, signozclickhousemetrics
        latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 50ms, 100ms, 250ms, 500ms, 1000ms, 1400ms, 2000ms, 5s, 10s, 20s, 40s, 60s]
        dimensions_cache_size: 100000
        dimensions:
          - name: service.namespace
            default: default
          - name: deployment.environment
            default: default
          - name: signoz.collector.id
        aggregation_temporality: AGGREGATION_TEMPORALITY_DELTA
    extensions:
      health_check:
        endpoint: 0.0.0.0:13133
      zpages:
        endpoint: localhost:55679
      pprof:
        endpoint: localhost:1777
    exporters:
      clickhousetraces:
        datasource: tcp://${env:CLICKHOUSE_USER}:${env:CLICKHOUSE_PASSWORD}@${env:CLICKHOUSE_HOST}:${env:CLICKHOUSE_PORT}/${env:CLICKHOUSE_TRACE_DATABASE}
        low_cardinal_exception_grouping: ${env:LOW_CARDINAL_EXCEPTION_GROUPING}
        use_new_schema: true
        timeout: 30s
      clickhousemetricswrite:
        endpoint: tcp://${env:CLICKHOUSE_USER}:${env:CLICKHOUSE_PASSWORD}@${env:CLICKHOUSE_HOST}:${env:CLICKHOUSE_PORT}/${env:CLICKHOUSE_DATABASE}
        timeout: 30s
        resource_to_telemetry_conversion:
          enabled: true
        disable_v2: true
      signozclickhousemetrics:
        dsn: tcp://${env:CLICKHOUSE_USER}:${env:CLICKHOUSE_PASSWORD}@${env:CLICKHOUSE_HOST}:${env:CLICKHOUSE_PORT}/${env:CLICKHOUSE_DATABASE}
        timeout: 50s
      clickhouselogsexporter:
        dsn: tcp://${env:CLICKHOUSE_USER}:${env:CLICKHOUSE_PASSWORD}@${env:CLICKHOUSE_HOST}:${env:CLICKHOUSE_PORT}/${env:CLICKHOUSE_LOG_DATABASE}
        timeout: 10s
        use_new_schema: true
      metadataexporter:
        dsn: tcp://${env:CLICKHOUSE_USER}:${env:CLICKHOUSE_PASSWORD}@${env:CLICKHOUSE_HOST}:${env:CLICKHOUSE_PORT}/signoz_metadata
        timeout: 10s
        tenant_id: ${env:TENANT_ID}
        cache:
          provider: in_memory
    service:
      telemetry:
        logs:
          encoding: json
        metrics:
          address: 0.0.0.0:8888
      extensions: [health_check, zpages, pprof]
      pipelines:
        traces:
          receivers: [otlp, jaeger]
          processors: [signozspanmetrics/delta, batch, memory_limiter]
          exporters: [clickhousetraces, metadataexporter]
        metrics:
          receivers: [otlp]
          processors: [batch, memory_limiter]
          exporters: [clickhousemetricswrite, metadataexporter, signozclickhousemetrics]
        logs:
          receivers: [otlp, httplogreceiver/heroku, httplogreceiver/json]
          processors: [batch, memory_limiter]
          exporters: [clickhouselogsexporter, metadataexporter]
@Srikanth Chekuri, any suggestions?
s
Hi @Matheus Henrique, sorry, missed the last message. The config looks fine to me. Has this been the config you have been following or did it get any updates before 12th?
m
Hello, this is the current configuration.
Open in Slack
PreviousNext
Powered by