I am getting errors in the otel-collector, when I enable a specific service to send to it using OTLP. My three other similar services work fine, but once I enable this one, I am getting these errors from the pod, and nothing gets stored. 2022-10-31T105610.924Z error clickhousetracesexporter/writer.go:101 Could not write a batch of spans {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "error": "clickhouse [Encode]: mismatched len of columns"} github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter /src/exporter/clickhousetracesexporter/writer.go:101 Any idea how I can hunt down the cause for this? There seem to be no further info logged from the pod than this

@Henrik which version of otel-collector are you using?

that's a correct release. It should not give errors. This can come if you introduced any extra column in clickhouse traces data

I installed SigNoz from K8S helm scripts and have not made any changes to it manually. I have no idea how to connect to clickhouse directly.

kubectl -n platform exec -it chi-my-release-clickhouse-cluster-0-0-0 /bin/bash

clickhouse-client

describe table signoz_traces.signoz_index_v2;

┌─name───────────────┬─type────────────────────────────────┬─default_type─┬─default_expression─┬─comment─┬─codec_expression─┬─ttl_expression─┐ │ timestamp │ DateTime64(9) │ │ │ │ DoubleDelta, LZ4 │ │ │ traceID │ FixedString(32) │ │ │ │ ZSTD(1) │ │ │ spanID │ String │ │ │ │ ZSTD(1) │ │ │ parentSpanID │ String │ │ │ │ ZSTD(1) │ │ │ serviceName │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ name │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ kind │ Int8 │ │ │ │ T64, ZSTD(1) │ │ │ durationNano │ UInt64 │ │ │ │ T64, ZSTD(1) │ │ │ statusCode │ Int16 │ │ │ │ T64, ZSTD(1) │ │ │ externalHttpMethod │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ externalHttpUrl │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ component │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ dbSystem │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ dbName │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ dbOperation │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ peerService │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ events │ Array(String) │ │ │ │ ZSTD(2) │ │ │ httpMethod │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ httpUrl │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ httpCode │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ httpRoute │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ httpHost │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ msgSystem │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ msgOperation │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ hasError │ Bool │ │ │ │ T64, ZSTD(1) │ │ │ tagMap │ Map(LowCardinality(String), String) │ │ │ │ ZSTD(1) │ │ │ gRPCMethod │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ gRPCCode │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ rpcSystem │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ rpcService │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ rpcMethod │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ responseStatusCode │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ └────────────────────┴─────────────────────────────────────┴──────────────┴────────────────────┴─────────┴──────────────────┴────────────────┘

this looks good

That's all there is, sorry 😞 It gets reperated roughhly every 10 seconds

yeah..some data incompatibility in writing to tables..we would have to enable debug statements there and build a new image to test it out. Can you raise an issue for this? We will look at this soon

It seems to be preventing saving other data as well. I assume because it saves them in batches, so I need to disable collection on this service.

You can ask enabling debug logs there and we will reach out to you when that is done 🙂

Okay, I will raise a GitHub issue