Hello Everyone, I'm currently using signoz on my K...
# support
n
Hello Everyone, I'm currently using signoz on my Kubernetes cluster and deployed using helm chart and using 0.67.1 version. the logs are gathering but not the matrics. when i checked the logs on matrics-collector pod i'm getting error replated to clickhouse sharding, kindly check following
Copy code
{"level":"info","ts":1737047123.7967355,"caller":"service@v0.111.0/service.go:234","msg":"Everything is ready. Begin running and processing data."}
{"level":"info","ts":1737047123.7968023,"caller":"prometheusreceiver@v0.111.0/metrics_receiver.go:187","msg":"Starting scrape manager","kind":"receiver","name":"prometheus","data_type":"metrics"}
{"level":"info","timestamp":"2025-01-16T17:05:23.956Z","caller":"signozcol/collector.go:121","msg":"Collector service is running"}
{"level":"info","timestamp":"2025-01-16T17:05:23.956Z","caller":"service/service.go:73","msg":"Client started successfully"}
{"level":"info","timestamp":"2025-01-16T17:05:23.956Z","caller":"opamp/client.go:49","msg":"Ensuring collector is running","component":"simple-client"}
{"level":"info","ts":1737047135.8053339,"caller":"internal/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"clickhousemetricswrite","error":"dial tcp 10.0.223.240:9000: connect: connection refused","interval":"6.369407404s"}
{"level":"warn","ts":1737047138.189477,"caller":"internal/transaction.go:128","msg":"Failed to scrape Prometheus endpoint","kind":"receiver","name":"prometheus","data_type":"metrics","scrape_timestamp":1737047138188,"target_labels":"{__name__=\"up\", instance=\"10.244.3.45:8888\", job=\"generic-collector\", job_name=\"generic-collector\", k8s_container_name=\"collector\", k8s_namespace_name=\"signoz\", k8s_node_name=\"aks-ondemandv2-37539849-vmss00000a\", k8s_pod_name=\"signoz-otel-collector-6bf576d988-n7gvv\", k8s_pod_phase=\"Running\", k8s_pod_ready=\"false\", k8s_pod_uid=\"fb1f6163-b423-4d77-b199-47f51faad262\", signoz_k8s_component=\"otel-collector\", signoz_k8s_instance=\"signoz\", signoz_k8s_name=\"signoz\"}"}
{"level":"info","ts":1737047142.177735,"caller":"internal/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"clickhousemetricswrite","error":"dial tcp 10.0.223.240:9000: connect: connection refused","interval":"6.667483926s"}
{"level":"info","ts":1737047148.8479598,"caller":"internal/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"clickhousemetricswrite","error":"dial tcp 10.0.223.240:9000: connect: connection refused","interval":"10.963357684s"}
{"level":"error","ts":1737047153.7657096,"caller":"clickhousemetricsexporter/clickhouse.go:206","msg":"error getting shard count","kind":"exporter","data_type":"metrics","name":"clickhousemetricswrite","error":"dial tcp 10.0.223.240:9000: connect: connection refused","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.(*clickHouse).shardCountWatcher\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousemetricsexporter/clickhouse.go:206\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.NewClickHouse.func1\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousemetricsexporter/clickhouse.go:170"}
{"level":"info","ts":1737047159.8231156,"caller":"internal/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"clickhousemetricswrite","error":"dial tcp 10.0.223.240:9000: connect: connection refused","interval":"18.621903266s"}
{"level":"info","ts":1737047178.4471312,"caller":"internal/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"clickhousemetricswrite","error":"dial tcp 10.0.223.240:9000: connect: connection refused","interval":"16.746201209s"}
{"level":"error","ts":1737047183.7659037,"caller":"clickhousemetricsexporter/clickhouse.go:206","msg":"error getting shard count","kind":"exporter","data_type":"metrics","name":"clickhousemetricswrite","error":"dial tcp 10.0.223.240:9000: connect: connection refused","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.(*clickHouse).shardCountWatcher\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousemetricsexporter/clickhouse.go:206\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.NewClickHouse.func1\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousemetricsexporter/clickhouse.go:170"}
{"level":"info","ts":1737047195.1962469,"caller":"internal/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"clickhousemetricswrite","error":"dial tcp 10.0.223.240:9000: connect: connection refused","interval":"32.232409862s"}
{"level":"warn","ts":1737047198.190159,"caller":"internal/transaction.go:128","msg":"Failed to scrape Prometheus endpoint","kind":"receiver","name":"prometheus","data_type":"metrics","scrape_timestamp":1737047198188,"target_labels":"{__name__=\"up\", instance=\"10.244.3.45:8888\", job=\"generic-collector\", job_name=\"generic-collector\", k8s_container_name=\"collector\", k8s_namespace_name=\"signoz\", k8s_node_name=\"aks-ondemandv2-37539849-vmss00000a\", k8s_pod_name=\"signoz-otel-collector-6bf576d988-n7gvv\", k8s_pod_phase=\"Running\", k8s_pod_ready=\"false\", k8s_pod_uid=\"fb1f6163-b423-4d77-b199-47f51faad262\", signoz_k8s_component=\"otel-collector\", signoz_k8s_instance=\"signoz\", signoz_k8s_name=\"signoz\"}"}
{"level":"error","ts":1737047213.7648773,"caller":"clickhousemetricsexporter/clickhouse.go:206","msg":"error getting shard count","kind":"exporter","data_type":"metrics","name":"clickhousemetricswrite","error":"dial tcp 10.0.223.240:9000: connect: connection refused","stacktrace":"github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.(*clickHouse).shardCountWatcher\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousemetricsexporter/clickhouse.go:206\ngithub.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.NewClickHouse.func1\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousemetricsexporter/clickhouse.go:170"}
{"level":"info","ts":1737047243.766175,"caller":"clickhousemetricsexporter/clickhouse.go:198","msg":"Shard count changed. Resetting time series map.","kind":"exporter","data_type":"metrics","name":"clickhousemetricswrite","prev":0,"current":4}
can any one please help me with this?
@Vibhu Pandey pls help me here
@Srikanth Chekuri
s
It says collector is unable to reach clickhouse, please check clickhouse health