This message was deleted.
# supports
Slackbot
07/28/2023, 4:02 PMThis message was deleted.
p
Prashant Shahi
07/28/2023, 4:54 PMThis happens when migration files are not executed properly.
OtelCollector and OtelCollector Metrics are responsible for running those to create the tables.
Prashant Shahi
07/28/2023, 4:55 PMInit containers make sure that those pods run only after clickhouse cluster is healthy.
But it seems like something is not right with the cluster health or order.
Prashant Shahi
07/28/2023, 4:56 PMCould you try restarting SigNoz OtelCollector and OtelCollector Metrics pods?
r
Romil Khanna
07/28/2023, 4:56 PMi did ... it ended up restarting over a 100x by now
Romil Khanna
07/28/2023, 4:57 PMi just fixed it 2s ago
Romil Khanna
07/28/2023, 4:57 PMi found the last migration, in
schema_migrationsRomil Khanna
07/28/2023, 4:58 PMthere are 2 problems that i see:
• the chart doesn't respect
clickhouse.clusterp
Prashant Shahi
07/28/2023, 5:06 PMYes,
clickhouse.clusterclusterPrashant Shahi
07/28/2023, 5:08 PMthe pod does not query to see how many shards or replicas are present so it marks 1 as good and other as dirtyCould you please elaborate on this?
r
Romil Khanna
07/28/2023, 5:09 PMif i have 2 replicas (default behavior) then the other replica still has migration marked as dirty
p
Prashant Shahi
07/29/2023, 7:16 AMI see. Dirty migration issue. TMK, this is usually resolved in later pod restarts.
@Srikanth Chekuri could you please look into this?
r
Romil Khanna
08/01/2023, 5:02 PM2 more issues @Prashant Shahi
Copy code
signoz-otel-collector-7b86585cd5-nrnnd signoz-otel-collector 2023-08-01T16:57:09.381Z info clickhousetracesexporter/clickhouse_factory.go:127 Clickhouse Migrate finished {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "error": "migration failed in line 0: \n\n\n\nCREATE MATERIALIZED VIEW IF NOT EXISTS signoz_traces.dependency_graph_minutes_service_calls_mv ON CLUSTER cluster\nTO signoz_traces.dependency_graph_minutes AS\nSELECT\n A.serviceName as src,\n B.serviceName as dest,\n quantilesState(0.5, 0.75, 0.9, 0.95, 0.99)(toFloat64(B.durationNano)) as duration_quantiles_state,\n countIf(B.statusCode=2) as error_count,\n count(*) as total_count,\n toStartOfMinute(B.timestamp) as timestamp\nFROM signoz_traces.signoz_index_v2 AS A, signoz_traces.signoz_index_v2 AS B\nWHERE (A.serviceName != B.serviceName) AND (A.spanID = B.parentSpanID)\nGROUP BY timestamp, src, dest; (details: code: 47, message: Unknown identifier: B.timestamp; there are columns: timestamp, serviceName, B.serviceName, quantilesState(0.5, 0.75, 0.9, 0.95, 0.99)(toFloat64(B.durationNano)), countIf(equals(B.statusCode, 2)), count(): While processing serviceName AS src, B.serviceName AS dest, quantilesState(0.5, 0.75, 0.9, 0.95, 0.99)(toFloat64(B.durationNano)) AS duration_quantiles_state, countIf(B.statusCode = 2) AS error_count, count() AS total_count, toStartOfMinute(B.timestamp) AS timestamp)"} Copy code
signoz-otel-collector-7b86585cd5-nrnnd signoz-otel-collector 2023-08-01T17:00:01.906Z error clickhousetracesexporter/writer.go:128 Could not write a batch of spans {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "error": "code: 60, message: Table signoz_traces.distributed_span_attributes doesn't exist"}p
Prashant Shahi
08/01/2023, 5:15 PMcc @Ankit Nayan @Vishal Sharma
r
Romil Khanna
08/01/2023, 5:15 PMit seems like the problem is related to the dirty flag that i pointed out earlier
Romil Khanna
08/01/2023, 5:15 PMconstantly changing it to 0 and restarting the pod resolves
Romil Khanna
08/01/2023, 5:16 PMthe issue is that if you don't know how many total migrations there are, you don't know when to stop
Romil Khanna
08/01/2023, 5:16 PMapart from the issue that it shouldn't be required 😕
s
Srikanth Chekuri
08/02/2023, 2:29 AMPlease subscribe to this issue if you want to keep an eye on when it gets fixed https://github.com/SigNoz/signoz-otel-collector/issues/156. The root cause is a lack of atomicity.
51 Views