https://signoz.io logo
#support
Title
# support
h

Henrik

10/31/2022, 11:00 AM
I am getting errors in the otel-collector, when I enable a specific service to send to it using OTLP. My three other similar services work fine, but once I enable this one, I am getting these errors from the pod, and nothing gets stored. 2022-10-31T105610.924Z error clickhousetracesexporter/writer.go:101 Could not write a batch of spans {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "error": "clickhouse [Encode]: mismatched len of columns"} github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter /src/exporter/clickhousetracesexporter/writer.go:101 Any idea how I can hunt down the cause for this? There seem to be no further info logged from the pod than this
a

Ankit Nayan

10/31/2022, 11:13 AM
@Henrik which version of otel-collector are you using?
a

Ankit Nayan

10/31/2022, 11:50 AM
that's a correct release. It should not give errors. This can come if you introduced any extra column in clickhouse traces data
can you paste below output after connecting to clickhouse
Copy code
describe table signoz_traces.signoz_index_v2;
h

Henrik

10/31/2022, 11:53 AM
I installed SigNoz from K8S helm scripts and have not made any changes to it manually. I have no idea how to connect to clickhouse directly.
a

Ankit Nayan

10/31/2022, 11:55 AM
Copy code
kubectl -n platform exec -it chi-my-release-clickhouse-cluster-0-0-0 /bin/bash
Copy code
clickhouse-client
Copy code
describe table signoz_traces.signoz_index_v2;
h

Henrik

10/31/2022, 11:56 AM
┌─name───────────────┬─type────────────────────────────────┬─default_type─┬─default_expression─┬─comment─┬─codec_expression─┬─ttl_expression─┐ │ timestamp │ DateTime64(9) │ │ │ │ DoubleDelta, LZ4 │ │ │ traceID │ FixedString(32) │ │ │ │ ZSTD(1) │ │ │ spanID │ String │ │ │ │ ZSTD(1) │ │ │ parentSpanID │ String │ │ │ │ ZSTD(1) │ │ │ serviceName │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ name │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ kind │ Int8 │ │ │ │ T64, ZSTD(1) │ │ │ durationNano │ UInt64 │ │ │ │ T64, ZSTD(1) │ │ │ statusCode │ Int16 │ │ │ │ T64, ZSTD(1) │ │ │ externalHttpMethod │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ externalHttpUrl │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ component │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ dbSystem │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ dbName │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ dbOperation │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ peerService │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ events │ Array(String) │ │ │ │ ZSTD(2) │ │ │ httpMethod │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ httpUrl │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ httpCode │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ httpRoute │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ httpHost │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ msgSystem │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ msgOperation │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ hasError │ Bool │ │ │ │ T64, ZSTD(1) │ │ │ tagMap │ Map(LowCardinality(String), String) │ │ │ │ ZSTD(1) │ │ │ gRPCMethod │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ gRPCCode │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ rpcSystem │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ rpcService │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ rpcMethod │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ │ responseStatusCode │ LowCardinality(String) │ │ │ │ ZSTD(1) │ │ └────────────────────┴─────────────────────────────────────┴──────────────┴────────────────────┴─────────┴──────────────────┴────────────────┘
Sorry about the formatting
a

Ankit Nayan

10/31/2022, 12:06 PM
this looks good
Is that the complete stack trace? maybe a couple of lines more would be there?
h

Henrik

10/31/2022, 12:08 PM
That's all there is, sorry 😞 It gets reperated roughhly every 10 seconds
a

Ankit Nayan

10/31/2022, 12:09 PM
yeah..some data incompatibility in writing to tables..we would have to enable debug statements there and build a new image to test it out. Can you raise an issue for this? We will look at this soon
I am afraid the data from that service won't appear till this is fixed
though it won't block other data
h

Henrik

10/31/2022, 12:14 PM
It seems to be preventing saving other data as well. I assume because it saves them in batches, so I need to disable collection on this service.
I can raise an issue, but I guess without a specific example you won't be able to find the issue anyway, without knowing what data causes the failure. So for you to have any use of it, I suspect it requires me to find out which trace(s) cause this in my application first?
a

Ankit Nayan

10/31/2022, 12:17 PM
You can ask enabling debug logs there and we will reach out to you when that is done 🙂
h

Henrik

10/31/2022, 12:23 PM
Okay, I will raise a GitHub issue
Hope I did it right, and included enough information? 🙂 https://github.com/SigNoz/signoz/issues/1671
Thanks for your help so far !
4 Views