https://signoz.io logo
Powered by
#support
Title
# support
a

Anil Kumar Bandrapalli

07/26/2023, 2:15 PM
@Srikanth Chekuri, i have one question regarding migration. we have version 10.2 running in our environments, we need to upgrade to 21.0 . how the migration of data takes places ? could you please help me on this
s

Srikanth Chekuri

07/27/2023, 3:00 AM
a

Anil Kumar Bandrapalli

07/27/2023, 5:26 AM
oh , so its like series of steps we need to consider
@Srikanth Chekuri, we have seen that migration to 0.12 , need to change the table names prefixed with distributed_ from 0.12 to 0.19 version we need to run the migration script which provided in the document right
please correct me if my understanding is wrong
s

Srikanth Chekuri

07/27/2023, 1:13 PM
Yes
a

Anil Kumar Bandrapalli

07/27/2023, 1:17 PM
@Srikanth Chekuri, i executed migration script and upgraded from 10.2 to 0.12.0. here is my script RENAME TABLE samples_v2 TO distributed_samples_v2, time_series_v2 TO distributed_time_series_v2, usage TO distributed_usage; RENAME TABLE dependency_graph_minutes TO distributed_dependency_graph_minutes, dependency_graph_minutes_db_calls_mv TO distributed_dependency_graph_minutes_db_calls_mv, dependency_graph_minutes_messaging_calls_mv TO distributed_dependency_graph_minutes_messaging_calls_mv, durationSort TO distributed_durationSort, signoz_error_index_v2 TO distributed_signoz_error_index_v2, signoz_spans TO distributed_signoz_spans, top_level_operations TO distributed_top_level_operations, signoz_index_v2 TO distributed_signoz_index_v2, usage TO distributed_usage, usage_explorer TO distributed_usage_explorer;
but getting my-release-signoz-otel-collector-6f47d74d65-nklqp 0/1 Init:0/1 0 8m15s my-release-signoz-otel-collector-ddb76f957-dgjm6 0/1 Init:0/1 0 6m20s
s

Srikanth Chekuri

07/27/2023, 1:17 PM
You don’t have to change the table names your self
a

Anil Kumar Bandrapalli

07/27/2023, 1:18 PM
then how data will be migrated ?
s

Srikanth Chekuri

07/27/2023, 1:20 PM
It will be taken care during the upgrade. The 
distributed_
tables are virtual tables which will use the existing tables. The migration only says if you use ClickHouse queries in dashboards you need to change queries.
a

Anil Kumar Bandrapalli

07/27/2023, 1:20 PM
oh thanks buddy
how to take back up of data
?
just in case i am asking
s

Srikanth Chekuri

07/27/2023, 1:26 PM
That’s a non-trivial process.
a

Anil Kumar Bandrapalli

07/27/2023, 1:37 PM
@Srikanth Chekuri, i just followed what you said but still we are seeing this my-release-signoz-alertmanager-0 0/1 Init:0/1 0 116s my-release-signoz-frontend-57bc68fb7d-946tf 0/1 Init:0/1 0 117s my-release-signoz-frontend-cf9b8d8c9-dg9b6 1/1 Running 0 10m my-release-signoz-otel-collector-6f47d74d65-5kj5j 0/1 Init:0/1 0 117s my-release-signoz-otel-collector-ddb76f957-6nr52 1/1 Running 0 10m my-release-signoz-otel-collector-metrics-5dd74686c-5cpwb 1/1 Running 0 10m my-release-signoz-otel-collector-metrics-84bbbddc6c-jfgkq 0/1 Init:0/1 0 117s my-release-signoz-query-service-0 0/1 Init:0/1 0 116s
pods are in init stage only
chat version is 0.2.5 to 0.6.0
s

Srikanth Chekuri

07/27/2023, 1:38 PM
Where are ClickHouse pods?
a

Anil Kumar Bandrapalli

07/27/2023, 1:39 PM
chi-my-release-clickhouse-cluster-0-0-0 1/1 Running 0 100s my-release-clickhouse-operator-b4c9cb776-jqxrm 2/2 Running 0 117s my-release-k8s-infra-otel-agent-2gt5n 1/1 Running 0 117s my-release-k8s-infra-otel-agent-4244z 1/1 Running 0 117s my-release-k8s-infra-otel-agent-4989f 1/1 Running 0 117s my-release-k8s-infra-otel-agent-4g59w 1/1 Running 0 117s my-release-k8s-infra-otel-agent-948gs 1/1 Running 0 117s my-release-k8s-infra-otel-agent-bjbtp 1/1 Running 0 117s my-release-k8s-infra-otel-agent-jl4bt 1/1 Running 0 117s my-release-k8s-infra-otel-agent-jql8z 1/1 Running 0 117s my-release-k8s-infra-otel-agent-kz86h 1/1 Running 0 117s my-release-k8s-infra-otel-agent-lxrg7 1/1 Running 0 117s my-release-k8s-infra-otel-agent-m2wzw 1/1 Running 0 117s my-release-k8s-infra-otel-agent-p2ht4 1/1 Running 0 117s my-release-k8s-infra-otel-agent-qpqcp 1/1 Running 0 117s my-release-k8s-infra-otel-agent-rr6j6 1/1 Running 0 117s my-release-k8s-infra-otel-agent-xt67c 1/1 Running 0 117s my-release-k8s-infra-otel-agent-z28nm 1/1 Running 0 117s my-release-k8s-infra-otel-deployment-5b8566bc46-bfzcf 1/1 Running 0 117s my-release-signoz-alertmanager-0 0/1 Init:0/1 0 116s my-release-signoz-frontend-57bc68fb7d-946tf 0/1 Init:0/1 0 117s my-release-signoz-frontend-cf9b8d8c9-dg9b6 1/1 Running 0 10m my-release-signoz-otel-collector-6f47d74d65-5kj5j 0/1 Init:0/1 0 117s my-release-signoz-otel-collector-ddb76f957-6nr52 1/1 Running 0 10m my-release-signoz-otel-collector-metrics-5dd74686c-5cpwb 1/1 Running 0 10m my-release-signoz-otel-collector-metrics-84bbbddc6c-jfgkq 0/1 Init:0/1 0 117s my-release-signoz-query-service-0 0/1 Init:0/1 0 116s my-release-zookeeper-0 1/1 Running 0 10m
s

Srikanth Chekuri

07/27/2023, 1:39 PM
Check the logs of ClickHouse if there are any issues
Or it may be resources issue.
a

Anil Kumar Bandrapalli

07/27/2023, 1:41 PM
helm --namespace platform install my-release signoz/signoz --set clickhouse.persistence.size=125Gi --version 0.2.5 helm -n platform upgrade my-release signoz/signoz --version 0.6.0
so first we installed 10.2 and then upgraded
logs 2023.07.27 133447.755403 [ 7 ] {} <Warning> Application: Listen [:]8123 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host> 2023.07.27 133447.755642 [ 7 ] {} <Warning> Application: Listen [:]9000 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host> 2023.07.27 133447.755952 [ 7 ] {} <Warning> Application: Listen [:]9004 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host> 2023.07.27 133447.756158 [ 7 ] {} <Warning> Application: Listen [:]9005 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host> 2023.07.27 133447.756370 [ 7 ] {} <Warning> Application: Listen [:]9363 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host> 2023.07.27 133447.845312 [ 7 ] {} <Warning> Application: Listen [:]9009 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
s

Srikanth Chekuri

07/27/2023, 1:45 PM
What does pod describe show for pending pods?
a

Anil Kumar Bandrapalli

07/27/2023, 1:46 PM
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 11m default-scheduler Successfully assigned platform/my-release-signoz-alertmanager-0 to ip-10-0-1-36.us-west-2.compute.internal Normal Pulled 11m kubelet Container image "docker.io/busybox:1.35" already present on machine Normal Created 11m kubelet Created container my-release-signoz-alertmanager-init Normal Started 11m kubelet Started container my-release-signoz-alertmanager-init
there is no error in describe pods, 5 pods are in init stage only
if you want we can huddle
s

Srikanth Chekuri

07/27/2023, 1:48 PM
If there is no error I am not sure how I can help. This is probably generic k8s pending pods issue.
a

Anil Kumar Bandrapalli

07/27/2023, 1:51 PM
2023.07.27 133447.756158 [ 7 ] {} <Warning> Application: Listen [:]9005 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
s

Srikanth Chekuri

07/27/2023, 1:55 PM
That is a warning.
a

Anil Kumar Bandrapalli

07/27/2023, 1:58 PM
i could observe that previous pods like front end, otel collector and otel metric pods of previous version are still running only. they are not deleting
when upgrading
s

Srikanth Chekuri

07/27/2023, 2:00 PM
That’s because your new pods are stuck in init. You should resolve why they are pending init. There is no error logs from them and nor the pod describe show any errors.
a

Anil Kumar Bandrapalli

07/27/2023, 2:01 PM
when we are installing version 10.2 , we are setting pvc to that. is that causing issue ?
this is the install command we used helm --namespace platform install my-release signoz/signoz --set clickhouse.persistence.size=125Gi --version 0.2.5
?
s

Srikanth Chekuri

07/27/2023, 2:10 PM
I don’t think that is related to why pods are getting stuck.
a

Anil Kumar Bandrapalli

07/27/2023, 4:48 PM
Error: cannot build pipelines: failed to create "clickhouselogsexporter" exporter
s

Srikanth Chekuri

07/27/2023, 4:52 PM
Share the full error when you share any error
2 Views
Powered by