I am just upgrading from 0.41.0 to 0.42.0. For one...
# support
d
I am just upgrading from 0.41.0 to 0.42.0. For one cluster it did go without any issue but now have one cluster where
monitor-signoz-otel-collector-cc57b8f4-gffkk
and
monitor-signoz-otel-collector-metrics-799fd94b69-fz9xr
get in a
Init:CrashLoopBackOff
because
Error from server (NotFound): jobs.batch "monitor-signoz-schema-migrator-upgrade" not found
; any guidenance how to fix this?
I solved it (well with some help of @Danik Raikhlin)! But to provide some feedback: It turns out the
monitor-signoz-otel-collector
and
monitor-signoz-otel-collector-metrics
where started before the
monitor-signoz-schema-migrator-upgrade
is created. And since the ttwo collectors didnt start the migrator wasnt created at all. causing the deployment too fail
Scaling the two collectors down to 0 replicas makes help thing its all good and will start the schema-migrator-update. After that its possible to run the collectors
a
@Dave Smits please i dont get what you did here, i am using terraform helm deployment
Copy code
resource "helm_release" "signoz" {
  name         = "signoz"
  chart        = "signoz"
  repository   = "<https://charts.signoz.io>"
  version      = "0.39.1"
  namespace    = kubernetes_namespace.platform.id
  timeout      = "800"
  values       = [data.template_file.signoz-values.rendered]
  force_update = true
}
d
I manually scaled down the monitor-signoz-otel-collector and monitor-signoz-otel-collector-metrics deployment to get the schema migrator getting started. Helm was waiting for them those pods i scaled down to become ready, which they never going to get because they crashed as the schema migrator was missing
a
i just did the same now, the schema migrator didnt start
Copy code
kubectl scale deploy signoz-otel-collector-metrics -n platform --replicas=1
Copy code
kubectl scale deploy signoz-otel-collector -n platform --replicas=1
d
I scaled them to 0
a
okay
tried this, it didnt work for me
thanks adding
Copy code
wait = false
to my helm while scaling down to 0 worked
e
hi i have tried scaling down to 0 and back up but it did not trigger the signoz-schema-migrator-upgrade, how can i trigger that manually?
a
What I think I did was editing the configuration to 0 instead of scaling it down
e
im looking at the job/signoz-schema-migrator-upgrade that is what creates the migrator but it has exceeded the retries
d
The whole point of scaling down is so you have 0 pods that are in not ready state. So maybe you have other pods as well that are not yet ready where helm is waiting on before creating the job.
e
scaling both collector/collector-metric only make it wait for "schema-migrator-upgrade"
anyway recreating the job/signoz-schema-migrator-upgrade fixed it
cheers