https://signoz.io logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
✅ 1
s
@Prashant Shahi
p
@Joakim Recht can you share output for 
kubectl get pods,jobs
?
j
As mentioned, there are no jobs... these are the pods (but with 
schemaMigrator.enabled=false
right now, otherwise collectors weren't running at all:
Copy code
NAME                                                READY   STATUS    RESTARTS      AGE
chi-signoz-clickhouse-cluster-0-0-0                 1/1     Running   0             23h
signoz-alertmanager-0                               1/1     Running   0             3d23h
signoz-clickhouse-operator-55c56ccb9c-2nvwb         2/2     Running   0             23h
signoz-frontend-768dc5ccf6-jhcgb                    1/1     Running   0             23h
signoz-k8s-infra-otel-agent-2hw4q                   1/1     Running   0             23h
signoz-k8s-infra-otel-agent-7fklv                   1/1     Running   0             23h
signoz-k8s-infra-otel-agent-b9rnq                   1/1     Running   0             23h
signoz-k8s-infra-otel-agent-hpn8w                   1/1     Running   0             23h
signoz-k8s-infra-otel-agent-ljpvx                   1/1     Running   0             23h
signoz-k8s-infra-otel-agent-md84v                   1/1     Running   0             23h
signoz-k8s-infra-otel-agent-nkctv                   1/1     Running   0             23h
signoz-k8s-infra-otel-agent-qrkg4                   1/1     Running   0             23h
signoz-k8s-infra-otel-agent-twqxd                   1/1     Running   0             23h
signoz-k8s-infra-otel-deployment-846b568667-87gvx   1/1     Running   0             23h
signoz-otel-collector-7dbddb448d-fs8zz              1/1     Running   0             14h
signoz-otel-collector-7dbddb448d-g9ptb              1/1     Running   0             14h
signoz-otel-collector-metrics-66d89d5485-qbwtz      1/1     Running   0             23h
signoz-query-service-0                              1/1     Running   4 (23h ago)   23h
signoz-zookeeper-0                                  1/1     Running   0             4d
signoz-zookeeper-1                                  1/1     Running   0             3d20h
signoz-zookeeper-2                                  1/1     Running   0             3d23h
p
schemaMigrator.enabled=true
is the default value and it is a must in case of schema changes.
also, are you running latest helm chart?
j
0.31.0, as far as I can tell that's the newest?
And yes I know I need to set it to true, but helm fails when I do so because no schema migrator job appears
And collectors start crashlooping
p
0.31.0, as far as I can tell that's the newest?
yes, that is the latest one.
The latest one is working fine for me for both fresh installation and upgrading from old one.
Perhaps, we could get on quick call to take a look at it, if that works for you.
j
I just tried again, and running helm (via terraform, but that shouldn't matter) doesn't work, and no job is created when I set schemaMigrator.enabled=true... I don't really see what would prevent it from being created, though
p
which k8s version? and cloud vendor/platform?
j
It's on AWS EKS, k8s version 1.27
I tried to manually set up the job, and then things seem to work. I have no idea why helm skips it...
By the way there's a small bug in the template for the job, the image property doesn't use 
.Values.schemaMigrator.image.registry
p
@Joakim Recht right, the registry seems to be not in use. Let me raise a PR fix for that.
however, that shouldn't cause any issue with migrator job creation.
can you try with rendering the helm template and see if job is created? Also, do try to apply the job manifest with the before-creation hooks?
j
I think it was related to my helm version - I upgraded it, and then it seems to work
🙌 1
p
can you share from which version to which one did you upgrade your helm?
j
I don't quite know, we use helm through terraform, and I upgraded the terraform 
hashicorp/helm
provider from 2.11.0 to 2.12.1
🙌 1
p
Thanks for sharing it. I will look into it.
g
Hey guys, I'm facing the same problem when I'm installing using helm terraform provider. @Joakim Recht do you mind to share your helm terraform file used to install signoz to compare to mine? also the backend configuration for helm?
j
@Gabriel Ferreira it doesn't really have anything special, what made a difference for me was
Copy code
helm = {
      source  = "hashicorp/helm"
      version = "= 2.12.1"
    }
Previously I had 
version >= 2.5
g
Hey @Joakim Recht one thing that make it works for me was add 
wait = false
in helm_release resource block in Terraform, I was having problem to start the migrate job with wait parameter default value equal true.
before that changing the helm provider version did not work for me, It started working only after changing the 
wait = false
5 Views
Open in Slack
PreviousNext
Powered by