SigNoz Community

This message was deleted.

s

Slackbot

about 2 years ago

This message was deleted.

s

h

+3

5
207
113

Hi Team, I am trying to do a fresh install of signoz (self-hosted) using helm chart version 0.57.0 (...

k

kankan ghosh

9 months ago

Hi Team, I am trying to do a fresh install of signoz (self-hosted) using helm chart version 0.57.0 (signoz appVersion 0.59.0). but schema-migrator sync and async init pods are failing with the same error as below. Any idea what this error means? also, another question that I have - why is schema-migrator needed for a fresh setup of signoz? Error message:

{"L":"info","timestamp":"2024-12-12T07:36:09.723Z","C":"signozschemamigrator/main.go:162","M":"Migrating up"}

{"L":"info","timestamp":"2024-12-12T07:36:09.723Z","C":"schema_migrator/manager.go:589","M":"Running migrations up sync"}

{"L":"info","timestamp":"2024-12-12T07:36:09.723Z","C":"schema_migrator/manager.go:551","M":"Checking if migration should run","db":"signoz_metrics","migration_id":1000,"versions":[]}

{"L":"info","timestamp":"2024-12-12T07:36:09.723Z","C":"schema_migrator/manager.go:568","M":"Fetching migration status","query":"SELECT * FROM signoz_metrics.schema_migrations_v2 WHERE migration_id = 1000 SETTINGS final = 1;"}

*{"L":"error","timestamp":"2024-12-12T07:36:09.724Z","C":"schema_migrator/manager.go:576","M":"Failed to fetch migration status","error":"code: 115, message: Unknown setting final","S":"<http://github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator.(*MigrationManager).shouldRunMigration|github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator.(*MigrationManager).shouldRunMigration>*\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator/manager.go:576\<http://ngithub.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator.(*MigrationManager).MigrateUpSync|ngithub.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator.(*MigrationManager).MigrateUpSync>\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator/manager.go:617\nmain.registerSyncMigrate.func1\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/main.go:163\<http://ngithub.com/spf13/cobra.(*Command).execute|ngithub.com/spf13/cobra.(*Command).execute>\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985\<http://ngithub.com/spf13/cobra.(*Command).ExecuteC|ngithub.com/spf13/cobra.(*Command).ExecuteC>\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117\<http://ngithub.com/spf13/cobra.(*Command).Execute|ngithub.com/spf13/cobra.(*Command).Execute>\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041\nmain.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/main.go:70\nruntime.main\n\t/opt/hostedtoolcache/go/1.23.3/x64/src/runtime/proc.go:272"}

panic: code: 115, message: Unknown setting final

goroutine 1 [running]:

<http://github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator.(*MigrationManager).shouldRunMigration(0xc0000e7520|github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator.(*MigrationManager).shouldRunMigration(0xc0000e7520>, {0xcb5fa1, 0xe}, 0x3e8, {0x13b6a40, 0x0, 0x0})

/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator/manager.go:577 +0xb85

<http://github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator.(*MigrationManager).MigrateUpSync(0xc0000e7520|github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator.(*MigrationManager).MigrateUpSync(0xc0000e7520>, {0xe21c60, 0x13b6a40}, {0x13b6a40, 0x0, 0x0})

/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator/manager.go:617 +0x27a

main.registerSyncMigrate.func1(0xc0001ca608, {0xcb1c0b?, 0x4?, 0xcb1c0f?})

/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/main.go:163 +0xfe9

<http://github.com/spf13/cobra.(*Command).execute(0xc0001ca608|github.com/spf13/cobra.(*Command).execute(0xc0001ca608>, {0xc0001ce8a0, 0x3, 0x3})

/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985 +0xaaa

<http://github.com/spf13/cobra.(*Command).ExecuteC(0xc0001ca308)|github.com/spf13/cobra.(*Command).ExecuteC(0xc0001ca308)>

/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff

<http://github.com/spf13/cobra.(*Command).Execute(...)|github.com/spf13/cobra.(*Command).Execute(...)>

/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041

main.main()

/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/main.go:70 +0x1f9

k

n

2
5
112

Hi Team Today I upgraded signoz from 0.44.0 to 0.50.0 and getting below error, Defaulted container ...

v

Vishal Kanani

about 1 year ago

Hi Team Today I upgraded signoz from 0.44.0 to 0.50.0 and getting below error, Defaulted container "schema-migrator" out of: schema-migrator, my-release-signoz-schema-migrator-init (init), my-release-signoz-schema-migrator-ch-ready (init) {"level":"info","timestamp":"2024-07-22T190403.105Z","caller":"signozschemamigrator/migrate.go:89","msg":"Setting env var SIGNOZ_CLUSTER","component":"migrate cli","cluster-name":"cluster"} {"level":"info","timestamp":"2024-07-22T190403.105Z","caller":"signozschemamigrator/migrate.go:106","msg":"Successfully set env var SIGNOZ_CLUSTER ","component":"migrate cli","cluster-name":"cluster"} {"level":"info","timestamp":"2024-07-22T190403.105Z","caller":"signozschemamigrator/migrate.go:111","msg":"Setting env var SIGNOZ_REPLICATED","component":"migrate cli","replication":false} {"level":"info","timestamp":"2024-07-22T190403.112Z","caller":"migrationmanager/manager.go:76","msg":"Running migrations for all migrators","component":"migrationmanager"} {"level":"info","timestamp":"2024-07-22T190403.112Z","caller":"migrationmanager/manager.go:78","msg":"Running migrations for logs","component":"migrationmanager","migrator":"logs"} {"level":"error","timestamp":"2024-07-22T190403.187Z","caller":"migrationmanager/manager.go:81","msg":"Failed to run migrations for migrator","component":"migrationmanager","migrator":"logs","error":"clickhouse migrate failed to run, error: Dirty database version 12. Fix and force version.","stacktrace":"github.com/SigNoz/signoz-otel-collector/migrationmanager.(*MigrationManager).Migrate\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/migrationmanager/manager.go81\nmain.main\n\t/home/runner/work/signoz otel collector/signoz otel collector/cmd/signozschemamigrator/migrate.go126\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.11/x64/src/runtime/proc.go:267"} {"level":"fatal","timestamp":"2024-07-22T190403.187Z","caller":"signozschemamigrator/migrate.go:128","msg":"Failed to run migrations","component":"migrate cli","error":"clickhouse migrate failed to run, error: Dirty database version 12. Fix and force version.","stacktrace":"main.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go128\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.11/x64/src/runtime/proc.go267"}

4
28
112

Signoz pods not Running

m

Maitryy

almost 2 years ago

Hi Team !! I'm trying to deploy signoz on a minikube using helm on a proxy setup. The status is

platform        chi-my-release-clickhouse-cluster-0-0-0                     1/1     Running                 0                 10h
platform        my-release-clickhouse-operator-657986696-mtgdq              2/2     Running                 0                 12h
platform        my-release-k8s-infra-otel-agent-pvf29                       1/1     Running                 0                 12h
platform        my-release-k8s-infra-otel-deployment-65767679c6-llgmg       1/1     Running                 0                 12h
platform        my-release-signoz-alertmanager-0                            0/1     Init:0/1                0                 9h
platform        my-release-signoz-frontend-5fc8679d4b-zd5c9                 0/1     Init:0/1                0                 15h
platform        my-release-signoz-frontend-775b95894-rl5pm                  0/1     Init:0/1                0                 11h
platform        my-release-signoz-otel-collector-577f7cc9c6-jswbm           0/1     Init:0/1                0                 12h
platform        my-release-signoz-otel-collector-7b7784c866-hr754           0/1     Init:0/1                0                 15h
platform        my-release-signoz-otel-collector-metrics-54d75b67c7-5ccx9   0/1     Init:0/1                0                 12h
platform        my-release-signoz-otel-collector-metrics-7f9fcd767-tqqxv    0/1     Init:0/1                0                 15h
platform        my-release-signoz-query-service-0                           0/1     Init:0/1                0                 10h
platform        my-release-signoz-schema-migrator-56769c434706-mzm2s        0/1     Init:0/1                0                 12h
platform        my-release-zookeeper-0                                      1/1     Running                 0                 15h

In the init container logs i see

wget: bad address 'my-release-signoz-query-service:8080'
waiting for query-service
wget: bad address 'my-release-signoz-query-service:8080'
waiting for query-service

---> init queryservice logs
wget: bad address 'my-release-clickhouse:8123'
waiting for clickhouseDB
wget: bad address 'my-release-clickhouse:8123'
waiting for clickhouseDB

I was thinking it to be a coredns issue coredns logs:

[INFO] 10.244.0.46:45441 - 56699 "AAAA IN my-release-signoz-query-service. udp 49 false 512" SERVFAIL qr,aa,rd,ra 49 0.000118854s
[INFO] 10.244.0.46:45441 - 31615 "A IN my-release-signoz-query-service. udp 49 false 512" SERVFAIL qr,aa,rd,ra 49 0.000033334s
[INFO] 10.244.0.56:54189 - 49629 "A IN my-release-clickhouse. udp 39 false 512" SERVFAIL qr,rd,ra 39 0.025739722s
[INFO] 10.244.0.56:54189 - 54743 "AAAA IN my-release-clickhouse. udp 39 false 512" SERVFAIL qr,rd,ra 39 0.025829025s
[INFO] 10.244.0.48:49914 - 36437 "AAAA IN my-release-clickhouse.platform-1.svc.cluster.local. udp 68 false 512" NOERROR qr,aa,rd 161 0.000295586s
[INFO] 10.244.0.48:49914 - 43859 "A IN my-release-clickhouse.platform-1.svc.cluster.local. udp 68 false 512" NOERROR qr,aa,rd 134 0.000323042s
[INFO] 10.244.0.43:34472 - 30113 "AAAA IN my-release-signoz-otel-collector.platform-1.svc.cluster.local. udp 90 false 1232" NOERROR qr,aa,rd 172 0.000309202s
[INFO] 10.244.0.43:43698 - 24992 "A IN my-release-signoz-otel-collector.platform-1.svc.cluster.local. udp 90 false 1232" NOERROR qr,aa,rd 156 0.000315525s

Please let me know what the problem is and how to resolve it.

m

1
1
112

Hi. We are playing around with external clickhouse setup (every other part of signoz runs on k8s), l...

w

WD

10 months ago

Hi. We are playing around with external clickhouse setup (every other part of signoz runs on k8s), latest version of chart (0.56.2, signoz 0.58.2). When we use the default clickhouse user with empty password: externalClickhouse: host: 10.10.10.10 user: default password: "" everything works correctly, signoz deployment finishes, schema-migrator completes. But when we setup a password for the default user: user: default password: yyyy or password: "yyyy" or password: 'yyyy' schema migrator seems to complete part of migrations but then fails and goes into CrashLoopBack and timeouts with this error:

Error: failed to run squashed migrations: failed to run squashed migrations

code: 516, message: Received from 10.10.10.10:9000. DB::Exception: default: Authentication failed: password is incorrect, or there is no user with such name.

If we change the user, delete the deployment and helm install fresh - same situation, same error. We use the default clickhouse ports (8123, 9000) so we don't setup them in the yaml file. Connecting locally or externally to clickhouse instance on 10.10.10.10 with the default user or the other user and providing yyyy password - works without any issues. Any idea what we are doing incorrectly?

w

s

2
5
111

Trying to set up a webhook for Alerting Channels with: <https://signoz.io/docs/alerts-management/not...

i

Ileo

about 1 year ago

Trying to set up a webhook for Alerting Channels with: https://signoz.io/docs/alerts-management/notification-channel/webhook/ What did I do wrong? I went to this website: https://webhook.site/ I pasted the given webhook URL in the right place. But afterwards, I'm not able to test or save my configuration. I'm on Kubernetes with Signoz installed using this manifest with a custom namespace.

signoz-install.yaml

i

s

2
4
111

This message was deleted.

s

Slackbot

almost 2 years ago

This message was deleted.

s

+2

4
3
111

This message was deleted.

s

Slackbot

over 2 years ago

This message was deleted.

s

r

p

3
13
111

Hi Team,, I recently started noticing Page not found in traces, When i checked the collector pod log...

s

Samuel Olowoyeye

about 1 year ago

Hi Team,, I recently started noticing Page not found in traces, When i checked the collector pod logs, i get

{"level":"warn","ts":1724310180.336868,"caller":"signozspanmetricsprocessor/processor.go:1176","msg":"Too many operations to track, using overflow operation name","kind":"processor","name":"signozspanmetrics/delta","pipeline":"traces","maxNumberOfOperationsToTrackPerService":2048}
{"level":"warn","ts":1724310180.5888464,"caller":"signozspanmetricsprocessor/processor.go:1176","msg":"Too many operations to track, using overflow operation name","kind":"processor","name":"signozspanmetrics/cumulative","pipeline":"traces","maxNumberOfOperationsToTrackPerService":2048}

What can I do to fix this issue.

s

m

3
4
110

Hello, I'm new here. I need help. Why I get this message?

u

최현진

over 1 year ago

Hello, I'm new here. I need help. Why I get this message?

4
17
110