Hima Vyas
05/09/2022, 7:15 AMkubectl -n signoz logs -f signoz_migrate
is returning pod not found for me thoughPrashant Shahi
05/09/2022, 7:25 AMkubectl -n platform logs -f signoz-migrate
Hima Vyas
05/09/2022, 7:26 AM127.0.0.1 9000 default
2022/05/09 07:25:48 dial tcp 127.0.0.1:9000: connect: connection refused
Prashant Shahi
05/09/2022, 7:27 AMadmin
with default password 27ff0399-0d3a-4bd8-919d-17c2181e6fb9
Hima Vyas
05/09/2022, 7:27 AMPrashant Shahi
05/09/2022, 7:39 AMHima Vyas
05/09/2022, 7:39 AMPrashant Shahi
05/09/2022, 7:40 AMHima Vyas
05/09/2022, 7:40 AMPrashant Shahi
05/09/2022, 8:14 AMHima Vyas
05/09/2022, 8:22 AMPrashant Shahi
05/09/2022, 8:23 AMHima Vyas
05/09/2022, 11:57 AMVishal Sharma
05/09/2022, 12:18 PM-service=[yourServiceName]
: If you want to restart the migration starting with the service yourServiceName
after it has failed.
β’ -timeNano=[timeStampinNano]
:Timestamp in nano after which the migration needs to be restarted.
Can you please tell me if you passed these params?Hima Vyas
05/09/2022, 12:19 PMVishal Sharma
05/09/2022, 12:20 PMHima Vyas
05/09/2022, 12:20 PMVishal Sharma
05/09/2022, 12:25 PMAnd when does the old data get removed? After the script has finished for all the services?
How can I check old and new table in clickhouse?Follow below commands to connect to clickhouse:
kubectl -n platform exec -i --tty pod/chi-signoz-cluster-0-0-0 -- bash
clickhouse-client
Old table name: default.signoz_index
New table names: signoz_traces.signoz_index_v2
and signoz_traces.signoz_spans
Hima Vyas
05/09/2022, 12:30 PMdefault.signoz_index
would not be required at all right?Vishal Sharma
05/09/2022, 12:41 PMHima Vyas
05/09/2022, 12:43 PMVishal Sharma
05/09/2022, 12:47 PMD
then you can safely assume that services A-C
were successful.Hima Vyas
05/09/2022, 12:48 PMVishal Sharma
05/09/2022, 1:11 PM-dropOldTable=false
to prevent the script delete old tables automatically.
Also you need to delete default.signoz_error_index
too.Hima Vyas
05/09/2022, 1:12 PMVishal Sharma
05/11/2022, 7:19 AMOPTIMIZE TABLE signoz_traces.signoz_index_v2 DEDUPLICATE;
and OPTIMIZE TABLE signoz_traces.durationSort DEDUPLICATE;
and OPTIMIZE TABLE signoz_traces.signoz_spans DEDUPLICATE;
Hima Vyas
05/11/2022, 7:19 AMSELECT
*,
count() AS cnt
FROM signoz_traces.signoz_index_v2
WHERE serviceName = 'vm' and (timestamp >= '2022-05-09 14:14:14.000000000') AND (timestamp < '2022-05-09 23:23:23.000000000')
GROUP BY *
HAVING cnt > 1
ORDER BY timestamp ASC limit 5
I am using this query to check duplicates. I am seeing duplicates for the time duration of older version only [0.7.5] There aren't duplicates after new version.Vishal Sharma
05/11/2022, 7:53 AMHima Vyas
05/11/2022, 8:35 AMVishal Sharma
05/11/2022, 9:10 AMOPTIMIZE TABLE signoz_traces.signoz_index_v2 FINAL DEDUPLICATE;
works well but the issue is that while migrating the key value pairs of tags are inserted in different order each time which causes the row to be different.
We could try OPTIMIZE TABLE signoz_traces.signoz_index_v2 FINAL DEDUPLICATE BY spanID;
but itβs wants spanID to be the only column in ORDER BY, thereβs an issue created on this: https://github.com/ClickHouse/ClickHouse/issues/34032OPTIMIZE TABLE signoz_traces.signoz_index_v2 FINAL DEDUPLICATE BY spanID, timestamp, serviceName, name, hasError;
worked for me.Hima Vyas
05/11/2022, 9:12 AMVishal Sharma
05/11/2022, 9:17 AMOPTIMIZE TABLE signoz_traces.durationSort FINAL DEDUPLICATE BY spanID, timestamp, durationNano;
to remove duplicates from durationSort
table.
Use OPTIMIZE TABLE signoz_traces.signoz_spans FINAL DEDUPLICATE;
to remove duplicates from signoz_spans
table.Hima Vyas
05/11/2022, 9:50 AMOPTIMIZE TABLE signoz_traces.signoz_index_v2 FINAL DEDUPLICATE BY spanID, timestamp, serviceName, name, hasError;
This also returning 0 rows for me.
Sample query to find duplicates -
SELECT
*,
count() AS cnt
FROM signoz_traces.signoz_index_v2
WHERE (serviceName = 'at') AND (timestamp >= '2022-05-09 00:00:00.000000000') AND (timestamp < '2022-05-09 01:01:01.000000000')
GROUP BY *
HAVING cnt > 1
ORDER BY timestamp ASC
LIMIT 5
Response has all the below columns as duplicates:
ββββββββββββββββββββββtimestampββ¬βtraceIDβββββββββββββββββββββββββββ¬βspanIDββββββββββββ¬βparentSpanIDββββββ¬βserviceNameββ¬βnameββββββββββββββββββββββββββββββββββββββ¬βkindββ¬βdurationNanoββ¬βstatusCodeββ¬βexternalHttpMethodββ¬βexternalHttpUrlββ¬βcomponentββ¬βdbSystemββ¬βdbNameββ¬βdbOperationββ¬βpeerServiceββ¬βeventsββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ¬βhttpMethodββ¬βhttpUrlββ¬βhttpCodeββ¬βhttpRouteββ¬βhttpHostββ¬βmsgSystemββ¬βmsgOperationββ¬βhasErrorββ¬βtagMapβββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ¬βgRPCMethodββ¬βgRPCCodeββ¬βcntββ
Vishal Sharma
05/11/2022, 9:57 AMOPTIMIZE TABLE signoz_traces.signoz_index_v2 FINAL DEDUPLICATE BY spanID, timestamp, serviceName, name, hasError;
wonβt return any rows.
How many rows are duplicates, as above optimize query worked well for me?Hima Vyas
05/11/2022, 10:08 AMVishal Sharma
05/11/2022, 10:10 AMsignoz_spans
duplicates get removed with below query?
OPTIMIZE TABLE signoz_traces.signoz_spans FINAL DEDUPLICATE;
Hima Vyas
05/11/2022, 10:24 AMVishal Sharma
05/11/2022, 10:25 AMHima Vyas
05/11/2022, 10:26 AMVishal Sharma
05/12/2022, 4:27 AMHima Vyas
05/12/2022, 4:47 AM