This message was deleted.

Slackbot

12/07/2023, 6:57 PM

This message was deleted.

✅ 1

Srikanth Chekuri

12/08/2023, 6:09 AM

@Prashant Shahi

Prashant Shahi

12/08/2023, 8:37 AM

@Joakim Recht can you share output for

kubectl get pods,jobs

Joakim Recht

12/08/2023, 9:48 AM

As mentioned, there are no jobs... these are the pods (but with

schemaMigrator.enabled=false

right now, otherwise collectors weren't running at all:

Copy code

NAME                                                READY   STATUS    RESTARTS      AGE
chi-signoz-clickhouse-cluster-0-0-0                 1/1     Running   0             23h
signoz-alertmanager-0                               1/1     Running   0             3d23h
signoz-clickhouse-operator-55c56ccb9c-2nvwb         2/2     Running   0             23h
signoz-frontend-768dc5ccf6-jhcgb                    1/1     Running   0             23h
signoz-k8s-infra-otel-agent-2hw4q                   1/1     Running   0             23h
signoz-k8s-infra-otel-agent-7fklv                   1/1     Running   0             23h
signoz-k8s-infra-otel-agent-b9rnq                   1/1     Running   0             23h
signoz-k8s-infra-otel-agent-hpn8w                   1/1     Running   0             23h
signoz-k8s-infra-otel-agent-ljpvx                   1/1     Running   0             23h
signoz-k8s-infra-otel-agent-md84v                   1/1     Running   0             23h
signoz-k8s-infra-otel-agent-nkctv                   1/1     Running   0             23h
signoz-k8s-infra-otel-agent-qrkg4                   1/1     Running   0             23h
signoz-k8s-infra-otel-agent-twqxd                   1/1     Running   0             23h
signoz-k8s-infra-otel-deployment-846b568667-87gvx   1/1     Running   0             23h
signoz-otel-collector-7dbddb448d-fs8zz              1/1     Running   0             14h
signoz-otel-collector-7dbddb448d-g9ptb              1/1     Running   0             14h
signoz-otel-collector-metrics-66d89d5485-qbwtz      1/1     Running   0             23h
signoz-query-service-0                              1/1     Running   4 (23h ago)   23h
signoz-zookeeper-0                                  1/1     Running   0             4d
signoz-zookeeper-1                                  1/1     Running   0             3d20h
signoz-zookeeper-2                                  1/1     Running   0             3d23h

Prashant Shahi

12/08/2023, 10:04 AM

schemaMigrator.enabled=true

is the default value and it is a must in case of schema changes.

Prashant Shahi

12/08/2023, 10:06 AM

also, are you running latest helm chart?

Joakim Recht

12/08/2023, 10:06 AM

0.31.0, as far as I can tell that's the newest?

Joakim Recht

12/08/2023, 10:06 AM

And yes I know I need to set it to true, but helm fails when I do so because no schema migrator job appears

Joakim Recht

12/08/2023, 10:07 AM

And collectors start crashlooping

Prashant Shahi

12/08/2023, 10:12 AM

0.31.0, as far as I can tell that's the newest?

yes, that is the latest one.

Prashant Shahi

12/08/2023, 10:16 AM

The latest one is working fine for me for both fresh installation and upgrading from old one.

Prashant Shahi

12/08/2023, 10:16 AM

Perhaps, we could get on quick call to take a look at it, if that works for you.

Joakim Recht

12/08/2023, 10:40 AM

I just tried again, and running helm (via terraform, but that shouldn't matter) doesn't work, and no job is created when I set schemaMigrator.enabled=true... I don't really see what would prevent it from being created, though

Prashant Shahi

12/08/2023, 11:09 AM

which k8s version? and cloud vendor/platform?

Joakim Recht

12/08/2023, 11:10 AM

It's on AWS EKS, k8s version 1.27

Joakim Recht

12/08/2023, 12:21 PM

I tried to manually set up the job, and then things seem to work. I have no idea why helm skips it...

Joakim Recht

12/08/2023, 12:33 PM

By the way there's a small bug in the template for the job, the image property doesn't use

.Values.schemaMigrator.image.registry

Prashant Shahi

12/09/2023, 7:14 AM

@Joakim Recht right, the registry seems to be not in use. Let me raise a PR fix for that.

Prashant Shahi

12/09/2023, 7:15 AM

however, that shouldn't cause any issue with migrator job creation.

Prashant Shahi

12/09/2023, 7:16 AM

can you try with rendering the helm template and see if job is created? Also, do try to apply the job manifest with the before-creation hooks?

Joakim Recht

12/09/2023, 7:06 PM

I think it was related to my helm version - I upgraded it, and then it seems to work

🙌 1

Prashant Shahi

12/10/2023, 5:14 PM

can you share from which version to which one did you upgrade your helm?

Joakim Recht

12/11/2023, 9:50 AM

I don't quite know, we use helm through terraform, and I upgraded the terraform

hashicorp/helm

provider from 2.11.0 to 2.12.1

🙌 1

Prashant Shahi

12/13/2023, 4:40 PM

Thanks for sharing it. I will look into it.

Gabriel Ferreira

01/13/2024, 4:16 PM

Hey guys, I'm facing the same problem when I'm installing using helm terraform provider. @Joakim Recht do you mind to share your helm terraform file used to install signoz to compare to mine? also the backend configuration for helm?

Joakim Recht

01/14/2024, 8:19 PM

@Gabriel Ferreira it doesn't really have anything special, what made a difference for me was

Copy code

helm = {
      source  = "hashicorp/helm"
      version = "= 2.12.1"
    }

Previously I had

version >= 2.5

Gabriel Ferreira

01/14/2024, 11:46 PM

Hey @Joakim Recht one thing that make it works for me was add

wait = false

in helm_release resource block in Terraform, I was having problem to start the migrate job with wait parameter default value equal true.

Gabriel Ferreira

01/14/2024, 11:51 PM

before that changing the helm provider version did not work for me, It started working only after changing the

wait = false

4 Views

Open in Slack

Previous Next

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.