<@U01NT95GV8B>, i have one question regarding migr...
# supporta
Anil Kumar Bandrapalli
07/26/2023, 2:15 PM@Srikanth Chekuri, i have one question regarding migration.
we have version 10.2 running in our environments, we need to upgrade to 21.0 . how the migration of data takes places ? could you please help me on this
s
Srikanth Chekuri
07/27/2023, 3:00 AMa
Anil Kumar Bandrapalli
07/27/2023, 5:26 AMoh , so its like series of steps we need to consider
Anil Kumar Bandrapalli
07/27/2023, 9:24 AM@Srikanth Chekuri, we have seen that migration to 0.12 , need to change the table names prefixed with distributed_
from 0.12 to 0.19 version we need to run the migration script which provided in the document right
Anil Kumar Bandrapalli
07/27/2023, 9:24 AMplease correct me if my understanding is wrong
s
Srikanth Chekuri
07/27/2023, 1:13 PMYes
a
Anil Kumar Bandrapalli
07/27/2023, 1:17 PM@Srikanth Chekuri, i executed migration script and upgraded from 10.2 to 0.12.0.
here is my script
RENAME TABLE samples_v2 TO distributed_samples_v2, time_series_v2 TO distributed_time_series_v2, usage TO distributed_usage;
RENAME TABLE dependency_graph_minutes TO distributed_dependency_graph_minutes, dependency_graph_minutes_db_calls_mv TO distributed_dependency_graph_minutes_db_calls_mv, dependency_graph_minutes_messaging_calls_mv TO distributed_dependency_graph_minutes_messaging_calls_mv, durationSort TO distributed_durationSort, signoz_error_index_v2 TO distributed_signoz_error_index_v2, signoz_spans TO distributed_signoz_spans, top_level_operations TO distributed_top_level_operations, signoz_index_v2 TO distributed_signoz_index_v2, usage TO distributed_usage, usage_explorer TO distributed_usage_explorer;
Anil Kumar Bandrapalli
07/27/2023, 1:17 PMbut getting
my-release-signoz-otel-collector-6f47d74d65-nklqp 0/1 Init:0/1 0 8m15s
my-release-signoz-otel-collector-ddb76f957-dgjm6 0/1 Init:0/1 0 6m20s
s
Srikanth Chekuri
07/27/2023, 1:17 PMYou don’t have to change the table names your self
a
Anil Kumar Bandrapalli
07/27/2023, 1:18 PMthen how data will be migrated ?
s
Srikanth Chekuri
07/27/2023, 1:20 PMIt will be taken care during the upgrade. The
distributed_a
Anil Kumar Bandrapalli
07/27/2023, 1:20 PMoh thanks buddy
Anil Kumar Bandrapalli
07/27/2023, 1:21 PMhow to take back up of data
Anil Kumar Bandrapalli
07/27/2023, 1:21 PM?
Anil Kumar Bandrapalli
07/27/2023, 1:21 PMjust in case i am asking
s
Srikanth Chekuri
07/27/2023, 1:26 PMThat’s a non-trivial process.
a
Anil Kumar Bandrapalli
07/27/2023, 1:37 PM@Srikanth Chekuri, i just followed what you said
but still we are seeing this
my-release-signoz-alertmanager-0 0/1 Init:0/1 0 116s
my-release-signoz-frontend-57bc68fb7d-946tf 0/1 Init:0/1 0 117s
my-release-signoz-frontend-cf9b8d8c9-dg9b6 1/1 Running 0 10m
my-release-signoz-otel-collector-6f47d74d65-5kj5j 0/1 Init:0/1 0 117s
my-release-signoz-otel-collector-ddb76f957-6nr52 1/1 Running 0 10m
my-release-signoz-otel-collector-metrics-5dd74686c-5cpwb 1/1 Running 0 10m
my-release-signoz-otel-collector-metrics-84bbbddc6c-jfgkq 0/1 Init:0/1 0 117s
my-release-signoz-query-service-0 0/1 Init:0/1 0 116s
Anil Kumar Bandrapalli
07/27/2023, 1:37 PMpods are in init stage only
Anil Kumar Bandrapalli
07/27/2023, 1:38 PMchat version is 0.2.5 to 0.6.0
s
Srikanth Chekuri
07/27/2023, 1:38 PMWhere are ClickHouse pods?
a
Anil Kumar Bandrapalli
07/27/2023, 1:39 PMchi-my-release-clickhouse-cluster-0-0-0 1/1 Running 0 100s
my-release-clickhouse-operator-b4c9cb776-jqxrm 2/2 Running 0 117s
my-release-k8s-infra-otel-agent-2gt5n 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-4244z 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-4989f 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-4g59w 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-948gs 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-bjbtp 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-jl4bt 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-jql8z 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-kz86h 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-lxrg7 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-m2wzw 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-p2ht4 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-qpqcp 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-rr6j6 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-xt67c 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-z28nm 1/1 Running 0 117s
my-release-k8s-infra-otel-deployment-5b8566bc46-bfzcf 1/1 Running 0 117s
my-release-signoz-alertmanager-0 0/1 Init:0/1 0 116s
my-release-signoz-frontend-57bc68fb7d-946tf 0/1 Init:0/1 0 117s
my-release-signoz-frontend-cf9b8d8c9-dg9b6 1/1 Running 0 10m
my-release-signoz-otel-collector-6f47d74d65-5kj5j 0/1 Init:0/1 0 117s
my-release-signoz-otel-collector-ddb76f957-6nr52 1/1 Running 0 10m
my-release-signoz-otel-collector-metrics-5dd74686c-5cpwb 1/1 Running 0 10m
my-release-signoz-otel-collector-metrics-84bbbddc6c-jfgkq 0/1 Init:0/1 0 117s
my-release-signoz-query-service-0 0/1 Init:0/1 0 116s
my-release-zookeeper-0 1/1 Running 0 10m
s
Srikanth Chekuri
07/27/2023, 1:39 PMCheck the logs of ClickHouse if there are any issues
Srikanth Chekuri
07/27/2023, 1:40 PMOr it may be resources issue.
a
Anil Kumar Bandrapalli
07/27/2023, 1:41 PMhelm --namespace platform install my-release signoz/signoz --set clickhouse.persistence.size=125Gi --version 0.2.5
helm -n platform upgrade my-release signoz/signoz --version 0.6.0
Anil Kumar Bandrapalli
07/27/2023, 1:41 PMso first we installed 10.2 and then upgraded
Anil Kumar Bandrapalli
07/27/2023, 1:44 PMlogs
2023.07.27 133447.755403 [ 7 ] {} <Warning> Application: Listen [:]8123 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 133447.755642 [ 7 ] {} <Warning> Application: Listen [:]9000 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 133447.755952 [ 7 ] {} <Warning> Application: Listen [:]9004 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 133447.756158 [ 7 ] {} <Warning> Application: Listen [:]9005 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 133447.756370 [ 7 ] {} <Warning> Application: Listen [:]9363 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 133447.845312 [ 7 ] {} <Warning> Application: Listen [:]9009 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
s
Srikanth Chekuri
07/27/2023, 1:45 PMWhat does pod describe show for pending pods?
a
Anil Kumar Bandrapalli
07/27/2023, 1:46 PMEvents:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 11m default-scheduler Successfully assigned platform/my-release-signoz-alertmanager-0 to ip-10-0-1-36.us-west-2.compute.internal
Normal Pulled 11m kubelet Container image "docker.io/busybox:1.35" already present on machine
Normal Created 11m kubelet Created container my-release-signoz-alertmanager-init
Normal Started 11m kubelet Started container my-release-signoz-alertmanager-init
Anil Kumar Bandrapalli
07/27/2023, 1:47 PMthere is no error in describe pods, 5 pods are in init stage only
Anil Kumar Bandrapalli
07/27/2023, 1:47 PMif you want we can huddle
s
Srikanth Chekuri
07/27/2023, 1:48 PMIf there is no error I am not sure how I can help. This is probably generic k8s pending pods issue.
a
Anil Kumar Bandrapalli
07/27/2023, 1:51 PM2023.07.27 133447.756158 [ 7 ] {} <Warning> Application: Listen [:]9005 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
s
Srikanth Chekuri
07/27/2023, 1:55 PMThat is a warning.
a
Anil Kumar Bandrapalli
07/27/2023, 1:58 PMi could observe that previous pods like front end, otel collector and otel metric pods of previous version are still running only. they are not deleting
Anil Kumar Bandrapalli
07/27/2023, 1:58 PMwhen upgrading
s
Srikanth Chekuri
07/27/2023, 2:00 PMThat’s because your new pods are stuck in init. You should resolve why they are pending init. There is no error logs from them and nor the pod describe show any errors.
a
Anil Kumar Bandrapalli
07/27/2023, 2:01 PMwhen we are installing version 10.2 , we are setting pvc to that. is that causing issue ?
Anil Kumar Bandrapalli
07/27/2023, 2:03 PMthis is the install command we used
helm --namespace platform install my-release signoz/signoz --set clickhouse.persistence.size=125Gi --version 0.2.5
Anil Kumar Bandrapalli
07/27/2023, 2:06 PM?
s
Srikanth Chekuri
07/27/2023, 2:10 PMI don’t think that is related to why pods are getting stuck.
a
Anil Kumar Bandrapalli
07/27/2023, 4:48 PMError: cannot build pipelines: failed to create "clickhouselogsexporter" exporter
s
Srikanth Chekuri
07/27/2023, 4:52 PMShare the full error when you share any error
42 Views