After upgrade to 0 8 0 in kubernetes I am following

Question

After upgrade to 0 8 0 in kubernetes I am following <https signoz io docs operate upgrade kubernetes> My namespace is signoz I have run the command kubectl n signoz run i t signoz migrate image=signoz migrate 0 8 host=my release clickhouse port=9000 userName=admin password=27ff0399 0d3a 4bd8 919d 17c2181e6fb9 signoz migrate is created with error and status CrashLoopBackOff Commnad ```kubectl n signoz logs f signoz migrate``` is returning pod not found for me though

Prashant Shahi · Answer

Oh there seems to be minor typo Can you run the following ```kubectl n platform logs f signoz migrate```

Hima Vyas · Answer

getting error ```127 0 0 1 9000 default 2022 05 09 07 25 48 dial tcp 127 0 0 1 9000 connect connection refused```

Hima Vyas · Answer

I removed clickhouse config from the command as I haven t changed any config so assuming it would be default

Prashant Shahi · Answer

That is needed as we set default user `admin` with default password `27ff0399 0d3a 4bd8 919d 17c2181e6fb9`

Hima Vyas · Answer

Okay

Hima Vyas · Answer

but host and port could be removed right

Prashant Shahi · Answer

port yes

Prashant Shahi · Answer

host nope

Hima Vyas · Answer

Yes only port my bad I have changed the host and kept admin and password as it is Migration seems to have started

Prashant Shahi · Answer

that s great do let us know when it completes

Hima Vyas · Answer

Sure Thanks Prashant

Hima Vyas · Answer

QQ If migration gets interrupted in middle will it be able to continue for the rest from the middle

Prashant Shahi · Answer

it ideally should run without any problem In case of any interruption you can run the script again with some additional flag if required cc < Vishal Sharma> should be able to answer that query

Hima Vyas · Answer

Once migration is done Will there be 2 replica of the same data on the persistent volume

Prashant Shahi · Answer

No it will drop old data

Hima Vyas · Answer

Writing 128512 rows ServiceName xxxx Migrated till 2022 05 09 02 55 46 2034198 +0000 UTC TimeNano 1652064946203419800 Writing 172796 rows ServiceName xxxx Migrated till 2022 05 09 02 51 56 695223079 +0000 UTC TimeNano 1652064716695223079 So This is how the migration is happening in chunks Now my migration job was interrupted and I had to create a new signoz migrate pod and restart the process It got restarted from the beginning itself How is data managed here Like how can I check how much data is yet to be migrated

Vishal Sharma · Answer

You need to pass two flags to specify start point ` service= yourServiceName ` If you want to restart the migration starting with the service `yourServiceName` after it has failed ` timeNano= timeStampinNano ` Timestamp in nano after which the migration needs to be restarted Can you please tell me if you passed these params

Hima Vyas · Answer

I haven t passed these params I have run the below command kubectl n signoz run i t signoz migrate image=signoz migrate 0 8 host=x x x x port=9000 userName=admin password=27ff0399 0d3a 4bd8 919d 17c2181e6fb9

Vishal Sharma · Answer

If you don t pass these params then the migration script will restart resulting in copy of data in new tables

Hima Vyas · Answer

Okay got it I will pass these params How can I check old and new table in clickhouse

Hima Vyas · Answer

And when does the old data get removed After the script has finished for all the services

Vishal Sharma · Answer

Data only gets removed after the script has completed running for all the services starting with the service and timestamp if you have passed any gt And when does the old data get removed After the script has finished for all the services

Vishal Sharma · Answer

gt How can I check old and new table in clickhouse Follow below commands to connect to clickhouse ```kubectl n platform exec i tty pod chi signoz cluster 0 0 0 bash``` ```clickhouse client``` Old table name `default signoz index` New table names `signoz traces signoz index v2` and `signoz traces signoz spans`

Hima Vyas · Answer

Okay Thanks

Hima Vyas · Answer

So After the migration `default signoz index` would not be required at all right

Vishal Sharma · Answer

Yes that table will be dropped

Vishal Sharma · Answer

Also can you please share the reason why migration failed Was it due to some issue with script

Hima Vyas · Answer

It got terminated because of network issue at my front

Hima Vyas · Answer

Plus I have 30+ services and 80GB of data with limited RAM so I may not be able to run the migration at one go service and timeNano is helpful in that case for me

Vishal Sharma · Answer

Also the migration is sorted by services alphabetically and timestamp descending So let s say the script failed at service `D` then you can safely assume that services `A C` were successful

Hima Vyas · Answer

Got it This was helpful Thanks

Hima Vyas · Answer

QQ Is it okay If I delete records manually from the table after a service is completed I mean would it affect the script execution

Vishal Sharma · Answer

Yes you can delete tables manually You can pass ` dropOldTable=false` to prevent the script delete old tables automatically Also you need to delete `default signoz error index` too

Hima Vyas · Answer

Okay understood raised hands

Hima Vyas · Answer

< Vishal Sharma> Is there a way to remove duplicates after migration I have duplicates in new table

Vishal Sharma · Answer

< Hima Vyas> You can run `OPTIMIZE TABLE signoz traces signoz index v2 DEDUPLICATE ` and `OPTIMIZE TABLE signoz traces durationSort DEDUPLICATE ` and `OPTIMIZE TABLE signoz traces signoz spans DEDUPLICATE `

Hima Vyas · Answer

I have tried this but this doesn t seem to be working I can see duplicates but deduplicate query isn t working directly

Hima Vyas · Answer

it gives 0 rows affected

Hima Vyas · Answer

```SELECT count AS cnt FROM signoz traces signoz index v2 WHERE serviceName = vm and timestamp gt = 2022 05 09 14 14 14 000000000 AND timestamp lt 2022 05 09 23 23 23 000000000 GROUP BY HAVING cnt gt 1 ORDER BY timestamp ASC limit 5``` I am using this query to check duplicates I am seeing duplicates for the time duration of older version only 0 7 5 There aren t duplicates after new version

Vishal Sharma · Answer

Couldn t find solution on google search I am checking clickhouse github issues to find something will create an issue if can t find solution

Hima Vyas · Answer

Thank you I ll let you know if I have any findings

Vishal Sharma · Answer

The `OPTIMIZE TABLE signoz traces signoz index v2 FINAL DEDUPLICATE ` works well but the issue is that while migrating the key value pairs of tags are inserted in different order each time which causes the row to be different We could try `OPTIMIZE TABLE signoz traces signoz index v2 FINAL DEDUPLICATE BY spanID ` but it s wants spanID to be the only column in ORDER BY there s an issue created on this <https github com ClickHouse ClickHouse issues 34032>

Vishal Sharma · Answer

Using `OPTIMIZE TABLE signoz traces signoz index v2 FINAL DEDUPLICATE BY spanID timestamp serviceName name hasError ` worked for me

Hima Vyas · Answer

Okay let me try this

Vishal Sharma · Answer

Use `OPTIMIZE TABLE signoz traces durationSort FINAL DEDUPLICATE BY spanID timestamp durationNano ` to remove duplicates from `durationSort` table Use `OPTIMIZE TABLE signoz traces signoz spans FINAL DEDUPLICATE ` to remove duplicates from `signoz spans` table

Hima Vyas · Answer

`OPTIMIZE TABLE signoz traces signoz index v2 FINAL DEDUPLICATE BY spanID timestamp serviceName name hasError ` This also returning 0 rows for me Sample query to find duplicates ```SELECT count AS cnt FROM signoz traces signoz index v2 WHERE serviceName = at AND timestamp gt = 2022 05 09 00 00 00 000000000 AND timestamp lt 2022 05 09 01 01 01 000000000 GROUP BY HAVING cnt gt 1 ORDER BY timestamp ASC LIMIT 5``` Response has all the below columns as duplicates ``` timestamp traceID spanID parentSpanID serviceName name kind durationNano statusCode externalHttpMethod externalHttpUrl component dbSystem dbName dbOperation peerService events httpMethod httpUrl httpCode httpRoute httpHost msgSystem msgOperation hasError tagMap gRPCMethod gRPCCode cnt ```

Vishal Sharma · Answer

< Hima Vyas> `OPTIMIZE TABLE signoz traces signoz index v2 FINAL DEDUPLICATE BY spanID timestamp serviceName name hasError ` won t return any rows How many rows are duplicates as above optimize query worked well for me

Hima Vyas · Answer

around 170083776

Vishal Sharma · Answer

Did the `signoz spans` duplicates get removed with below query `OPTIMIZE TABLE signoz traces signoz spans FINAL DEDUPLICATE `

Hima Vyas · Answer

No that also didn t work but I think clickhouse might be working in background after this query for such large data I will check after some time if duplicates are removed

Vishal Sharma · Answer

Yes optimize works in background Maybe you can monitor the process the by checking if number of duplicates are decreasing

Hima Vyas · Answer

Yes I am checking that for signoz index v2 I can see some decrement will monitor this and let you know Thanks < Vishal Sharma> for the help appreciate it

Vishal Sharma · Answer

< Hima Vyas> Were you successful to remove duplicates

Hima Vyas · Answer

Yes duplicates were removed after few hours for large table the query was timing out So I have updated receive timeout parameter as well

Hima Vyas · Answer

Thanks again for the help Vishal