This message was deleted.
s
This message was deleted.
p
Hi @Thomas Hunter 👋 pods states as well as logs of failing pods would be helpful.
Also, did you run the migration script after upgrading to v0.19? https://signoz.io/docs/operate/migration/upgrade-0.19/
t
I can't run the migration script since the service it targets is dead:
Copy code
yakuhito@catstation:/tmp$ kubectl -n platform logs pod/my-release-signoz-query-service-0
2023-05-23T19:50:24.790Z	INFO	version/version.go:43	

SigNoz version   : v0.19.0
Commit SHA-1     : 6e8be3f
Commit timestamp : 2023-05-20T18:20:50Z
Branch           : HEAD
Go version       : go1.18.10

For SigNoz Official Documentation,  visit <https://signoz.io/docs>
For SigNoz Community Slack,         visit <http://signoz.io/slack>
For discussions about SigNoz,       visit <https://community.signoz.io>

Check SigNoz Github repo for license details.
Copyright 2022 SigNoz


2023-05-23T19:50:24.790Z	WARN	query-service/main.go:61	No JWT secret key is specified.
main.main
	/go/src/github.com/signoz/signoz/ee/query-service/main.go:61
runtime.main
	/usr/local/go/src/runtime/proc.go:250
2023-05-23T19:50:26.023Z	INFO	license/manager.go:127	No active license found, defaulting to basic plan
2023-05-23T19:50:26.025Z	INFO	app/server.go:117	Using ClickHouse as datastore ...
ts=2023-05-23T19:50:26.032Z caller=query_logger.go:113 level=error component=activeQueryTracker msg="Failed to create directory for logging active queries"
ts=2023-05-23T19:50:26.034Z caller=engine.go:349 level=debug component="query engine" msg="Lookback delta is zero, setting to default value" value=5m0s
ts=2023-05-23T19:50:26.035Z caller=reader.go:364 level=info msg="Loading configuration file" filename=/root/config/prometheus.yml
ts=2023-05-23T19:50:26.036Z caller=query_logger.go:113 level=error msg="Failed to create directory for logging active queries"
ts=2023-05-23T19:50:26.039Z caller=engine.go:349 level=debug component="promql evaluator" msg="Lookback delta is zero, setting to default value" value=5m0s
2023-05-23T19:50:26.052Z	INFO	alertManager/notifier.go:94	Starting notifier with alert manager:[<http://my-release-signoz-alertmanager:9093/api/>]
2023-05-23T19:50:26.052Z	INFO	app/server.go:587	rules manager is ready
ts=2023-05-23T19:50:26.053Z caller=reader.go:381 level=info msg="Completed loading of configuration file" filename=/root/config/prometheus.yml
2023-05-23T19:50:26.075Z	DEBUG	rules/apiParams.go:86	postable rule(parsed):%!(EXTRA *rules.PostableRule=&{Exception EXCEPTIONS_BASED_ALERT  threshold_rule 300000000000 0 {"op":"1","target":5,"matchType":"4"} map[details:<https://monitoring.fireacademy.io/exceptions> severity:warning] map[description:This alert is fired when the defined metric (current value: {{$value}}) crosses the threshold ({{$threshold}}) summary:The rule threshold is set to {{$threshold}}, and the observed metric value is {{$value}}] false <https://monitoring.fireacademy.io/alerts/new> []  })
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1046d20]

goroutine 1 [running]:
<http://go.signoz.io/signoz/pkg/query-service/rules.parseIntoRule({{0x0|go.signoz.io/signoz/pkg/query-service/rules.parseIntoRule({{0x0>, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0, ...}, ...)
	/go/src/github.com/signoz/signoz/pkg/query-service/rules/apiParams.go:114 +0x4e0
<http://go.signoz.io/signoz/pkg/query-service/rules.parsePostableRule(...)|go.signoz.io/signoz/pkg/query-service/rules.parsePostableRule(...)>
	/go/src/github.com/signoz/signoz/pkg/query-service/rules/apiParams.go:63
<http://go.signoz.io/signoz/pkg/query-service/rules.ParsePostableRule({0xc000209800|go.signoz.io/signoz/pkg/query-service/rules.ParsePostableRule({0xc000209800>?, 0xc000209100?, 0x6f1?})
	/go/src/github.com/signoz/signoz/pkg/query-service/rules/apiParams.go:59 +0x65
<http://go.signoz.io/signoz/pkg/query-service/rules.(*Manager).initiate(0xc00019fc00)|go.signoz.io/signoz/pkg/query-service/rules.(*Manager).initiate(0xc00019fc00)>
	/go/src/github.com/signoz/signoz/pkg/query-service/rules/manager.go:156 +0x16a
<http://go.signoz.io/signoz/pkg/query-service/rules.(*Manager).Start(0xc0006219e0?)|go.signoz.io/signoz/pkg/query-service/rules.(*Manager).Start(0xc0006219e0?)>
	/go/src/github.com/signoz/signoz/pkg/query-service/rules/manager.go:130 +0x25
<http://go.signoz.io/signoz/ee/query-service/app.(*Server).Start(0xc000112e00)|go.signoz.io/signoz/ee/query-service/app.(*Server).Start(0xc000112e00)>
	/go/src/github.com/signoz/signoz/ee/query-service/app/server.go:454 +0x93
main.main()
	/go/src/github.com/signoz/signoz/ee/query-service/main.go:71 +0x574
p
@Srikanth Chekuri can you please look into this?
should the migration script have been executed prior to the upgrade for K8s?
t
The link you provided suggests the order is upgrade -> migration: https://signoz.io/docs/operate/migration/upgrade-0.19/#first-upgrade-to-v019
p
@Thomas Hunter This is addressed by this PR: https://github.com/SigNoz/charts/pull/233
You can either wait for the patch release after the PR above gets merged.
Or you can clone and upgrade locally from the specific git branch.
t
Ok, thank you!
p
@Thomas Hunter Do let us know if the issue has been resolved.
361 Views