Logs for both pods which are crashing

kubectl logs signoz-playground-dev-otel-collector-metrics-6697cf9d44-75jbs -n playground-dev 2022-09-25T121621.332Z info service/telemetry.go:103 Setting up own telemetry... 2022-09-25T121621.332Z info service/telemetry.go:138 Serving Prometheus metrics {"address": "0.0.0.0:8888", "level": "basic"} time="2022-09-25T121621Z" level=info msg="Executing:\nCREATE DATABASE IF NOT EXISTS signoz_metrics\n" component=clickhouse 2022-09-25T121622.354Z info extensions/extensions.go:42 Starting extensions... 2022-09-25T121622.355Z info extensions/extensions.go:45 Extension is starting... {"kind": "extension", "name": "health_check"} 2022-09-25T121622.355Z info healthcheckextension@v0.55.0/healthcheckextension.go:44 Starting health_check extension {"kind": "extension", "name": "health_check", "config": {"Port":0,"TCPAddr":{"Endpoint":"0.0.0.0:13133"},"Path":"/","CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}} 2022-09-25T121622.355Z info extensions/extensions.go:49 Extension started. {"kind": "extension", "name": "health_check"} 2022-09-25T121622.355Z info extensions/extensions.go:45 Extension is starting... {"kind": "extension", "name": "zpages"} 2022-09-25T121622.355Z info zpagesextension/zpagesextension.go:64 Registered zPages span processor on tracer provider {"kind": "extension", "name": "zpages"} 2022-09-25T121622.355Z info zpagesextension/zpagesextension.go:74 Registered Host's zPages {"kind": "extension", "name": "zpages"} 2022-09-25T121622.355Z info zpagesextension/zpagesextension.go:86 Starting zPages extension {"kind": "extension", "name": "zpages", "config": {"TCPAddr":{"Endpoint":"localhost:55679"}}} 2022-09-25T121622.356Z info extensions/extensions.go:49 Extension started. {"kind": "extension", "name": "zpages"} 2022-09-25T121622.356Z info extensions/extensions.go:45 Extension is starting... {"kind": "extension", "name": "pprof"} 2022-09-25T121622.356Z info pprofextension@v0.55.0/pprofextension.go:71 Starting net/http/pprof server {"kind": "extension", "name": "pprof", "config": {"TCPAddr":{"Endpoint":"localhost:1777"},"BlockProfileFraction":0,"MutexProfileFraction":0,"SaveToFile":""}} 2022-09-25T121622.356Z info extensions/extensions.go:49 Extension started. {"kind": "extension", "name": "pprof"} 2022-09-25T121622.356Z info pipelines/pipelines.go:74 Starting exporters... 2022-09-25T121622.357Z info pipelines/pipelines.go:78 Exporter is starting... {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite"} 2022-09-25T121622.357Z info pipelines/pipelines.go:82 Exporter started. {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite"} 2022-09-25T121622.357Z info pipelines/pipelines.go:86 Starting processors... 2022-09-25T121622.357Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "batch", "pipeline": "metrics"} 2022-09-25T121622.357Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "batch", "pipeline": "metrics"} 2022-09-25T121622.357Z info pipelines/pipelines.go:98 Starting receivers... 2022-09-25T121622.357Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "prometheus", "pipeline": "metrics"} 2022-09-25T121622.357Z info kubernetes/kubernetes.go:325 Using pod service account via in-cluster config {"kind": "receiver", "name": "prometheus", "pipeline": "metrics", "discovery": "kubernetes"} 2022-09-25T121622.358Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "prometheus", "pipeline": "metrics"} 2022-09-25T121622.358Z info healthcheck/handler.go:129 Health Check state change {"kind": "extension", "name": "health_check", "status": "ready"} 2022-09-25T121622.358Z info service/collector.go:215 Starting signoz-otel-collector... {"Version": "latest", "NumCPU": 4} 2022-09-25T121622.358Z info service/collector.go:128 Everything is ready. Begin running and processing data. panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x343dd12] goroutine 155 [running]: github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.(*PrwExporter).export.func1() /src/exporter/clickhousemetricsexporter/exporter.go:280 +0xf2 created by github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.(*PrwExporter).export /src/exporter/clickhousemetricsexporter/exporter.go:276 +0x256 dev-playstation@ip-18-0-2-129:*~*$ kubectl logs signoz-playground-dev-otel-collector-694d5dd67d-24wxf -n playground-dev 2022-09-25T121350.268Z info service/telemetry.go:103 Setting up own telemetry... 2022-09-25T121350.268Z info service/telemetry.go:138 Serving Prometheus metrics {"address": "0.0.0.0:8888", "level": "basic"} Error: cannot build pipelines: failed to create "clickhousetraces" exporter, in pipeline "traces": error connecting to primary db: dial tcp 10.100.169.1919000 i/o timeout 2022/09/25 121351 application run finished with error: cannot build pipelines: failed to create "clickhousetraces" exporter, in pipeline "traces": error connecting to primary db: dial tcp 10.100.169.1919000 i/o timeout

will try and share the result

almost all error fixed, one is still pending will try to delete and re-deploy will update you

thanks

not sure, what it could be

2022-09-25T132041.418Z info service/collector.go:128 Everything is ready. Begin running and processing data. 2022-09-25T132111.422Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 117, message: Object has ambiguous path: service_instance_id: while executing 'FUNCTION _CAST(labels :: 3, 'Object(\\'json\\')' :: 4) -> _CAST(labels, 'Object(\\'json\\')') Object('json') : 6'", "interval": "55.233015ms"} 2022-09-25T132141.425Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 117, message: Object has ambiguous path: service_instance_id: while executing 'FUNCTION _CAST(labels :: 3, 'Object(\\'json\\')' :: 4) -> _CAST(labels, 'Object(\\'json\\')') Object('json') : 6'", "interval": "28.281851ms"} 2022-09-25T132241.430Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 117, message: Object has ambiguous path: service_instance_id: while executing 'FUNCTION _CAST(labels :: 3, 'Object(\\'json\\')' :: 4) -> _CAST(labels, 'Object(\\'json\\')') Object('json') : 6'", "interval": "35.713193ms"} 2022-09-25T132311.431Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 117, message: Object has ambiguous path: service_instance_id: while executing 'FUNCTION _CAST(labels :: 3, 'Object(\\'json\\')' :: 4) -> _CAST(labels, 'Object(\\'json\\')') Object('json') : 6'", "interval": "44.032859ms"}

@Kapil Yadav From what I recall, that's info level message and shouldn't affect anything significant.