Hello, I have observed a significant increase in CPU utilization after the Signoz version upgrade to...
s

saurabh biramwar

over 1 year ago
Hello, I have observed a significant increase in CPU utilization after the Signoz version upgrade to v0.41.1. Additionally, we're encountering the following error in the ClickHouse pod. 2024.03.21 130442.644576 [ 1011 ] {bc235080-bd8e-4a26-99d6-69e10ec9bee9} <Error> TCPHandler: Code: 210. DB:NetException I/O error: Broken pipe, while writing to socket ([:ffff10.0.145.23]:9000 -> [:ffff10.0.129.149]:33502). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): 0. DB:Exception:Exception(DB:Exception:MessageMasked&&, int, bool) @ 0x000000000c800f1b in /usr/bin/clickhouse 1. DB:NetException:NetException<String, String, String>(int, FormatStringHelperImpl<std:type identity&lt;String&gt;:type, std:type identity&lt;String&gt;:type, std:type identity&lt;String&gt;:type>, String&&, String&&, String&&) @ 0x000000000caa69a1 in /usr/bin/clickhouse 2. DB:WriteBufferFromPocoSocket:nextImpl() @ 0x000000000caa733e in /usr/bin/clickhouse 3. DB:TCPHandler:runImpl() @ 0x000000001292120f in /usr/bin/clickhouse 4. DB:TCPHandler:run() @ 0x0000000012933eb9 in /usr/bin/clickhouse 5. Poco:NetTCPServerConnection:start() @ 0x00000000153a5a72 in /usr/bin/clickhouse 6. Poco:NetTCPServerDispatcher:run() @ 0x00000000153a6871 in /usr/bin/clickhouse 7. Poco:PooledThread:run() @ 0x000000001549f047 in /usr/bin/clickhouse 8. Poco:ThreadImpl:runnableEntry(void*) @ 0x000000001549d67d in /usr/bin/clickhouse
Hi Team, I am trying to do a fresh install of signoz (self-hosted) using helm chart version 0.57.0 (...
k

kankan ghosh

12 months ago
Hi Team, I am trying to do a fresh install of signoz (self-hosted) using helm chart version 0.57.0 (signoz appVersion 0.59.0). but schema-migrator sync and async init pods are failing with the same error as below. Any idea what this error means? also, another question that I have - why is schema-migrator needed for a fresh setup of signoz? Error message:
{"L":"info","timestamp":"2024-12-12T07:36:09.723Z","C":"signozschemamigrator/main.go:162","M":"Migrating up"}
{"L":"info","timestamp":"2024-12-12T07:36:09.723Z","C":"schema_migrator/manager.go:589","M":"Running migrations up sync"}
{"L":"info","timestamp":"2024-12-12T07:36:09.723Z","C":"schema_migrator/manager.go:551","M":"Checking if migration should run","db":"signoz_metrics","migration_id":1000,"versions":[]}
{"L":"info","timestamp":"2024-12-12T07:36:09.723Z","C":"schema_migrator/manager.go:568","M":"Fetching migration status","query":"SELECT * FROM signoz_metrics.schema_migrations_v2 WHERE migration_id = 1000 SETTINGS final = 1;"}
*{"L":"error","timestamp":"2024-12-12T07:36:09.724Z","C":"schema_migrator/manager.go:576","M":"Failed to fetch migration status","error":"code: 115, message: Unknown setting final","S":"<http://github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator.(*MigrationManager).shouldRunMigration|github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator.(*MigrationManager).shouldRunMigration>*\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator/manager.go:576\<http://ngithub.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator.(*MigrationManager).MigrateUpSync|ngithub.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator.(*MigrationManager).MigrateUpSync>\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator/manager.go:617\nmain.registerSyncMigrate.func1\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/main.go:163\<http://ngithub.com/spf13/cobra.(*Command).execute|ngithub.com/spf13/cobra.(*Command).execute>\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985\<http://ngithub.com/spf13/cobra.(*Command).ExecuteC|ngithub.com/spf13/cobra.(*Command).ExecuteC>\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117\<http://ngithub.com/spf13/cobra.(*Command).Execute|ngithub.com/spf13/cobra.(*Command).Execute>\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041\nmain.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/main.go:70\nruntime.main\n\t/opt/hostedtoolcache/go/1.23.3/x64/src/runtime/proc.go:272"}
panic: code: 115, message: Unknown setting final
goroutine 1 [running]:
<http://github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator.(*MigrationManager).shouldRunMigration(0xc0000e7520|github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator.(*MigrationManager).shouldRunMigration(0xc0000e7520>, {0xcb5fa1, 0xe}, 0x3e8, {0x13b6a40, 0x0, 0x0})
/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator/manager.go:577 +0xb85
<http://github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator.(*MigrationManager).MigrateUpSync(0xc0000e7520|github.com/SigNoz/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator.(*MigrationManager).MigrateUpSync(0xc0000e7520>, {0xe21c60, 0x13b6a40}, {0x13b6a40, 0x0, 0x0})
/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/schema_migrator/manager.go:617 +0x27a
main.registerSyncMigrate.func1(0xc0001ca608, {0xcb1c0b?, 0x4?, 0xcb1c0f?})
/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/main.go:163 +0xfe9
<http://github.com/spf13/cobra.(*Command).execute(0xc0001ca608|github.com/spf13/cobra.(*Command).execute(0xc0001ca608>, {0xc0001ce8a0, 0x3, 0x3})
/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985 +0xaaa
<http://github.com/spf13/cobra.(*Command).ExecuteC(0xc0001ca308)|github.com/spf13/cobra.(*Command).ExecuteC(0xc0001ca308)>
/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff
<http://github.com/spf13/cobra.(*Command).Execute(...)|github.com/spf13/cobra.(*Command).Execute(...)>
/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
main.main()
/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/main.go:70 +0x1f9