HI guys I am using SigNoz `v0 11 4` with Slack I have an iss SigNoz Community #support

HI guys, I am using SigNoz `v0.11.4` with Slack. I...

Alejandro Decchi

01/04/2023, 3:32 PM

HI guys, I am using SigNoz

v0.11.4

with Slack. I have an issue with alert manager, sometimes the alert is triggering with status

Firing

, but the alert never trigger a message to slack. On the other hand, another issue is when I create the alarm and I test it, it does not work, but if I saved and edit it and change the name for example and test it again the alarm send the message to the slack channel.

Ankit Nayan

01/04/2023, 3:35 PM

cc: @Amol Umbark

Amol Umbark

01/05/2023, 7:19 AM

Hi @Alejandro Decchi are you able to consistently create the issue? if so please share exact steps, log of query service and alert manager and also see jf the alert shows up in triggered notifications

Amol Umbark

01/05/2023, 7:20 AM

i will try to reproduce the test alert issue. and get back.

Alejandro Decchi

01/05/2023, 1:05 PM

@Amol Umbark thanks for your reply. @Pranay create the issue https://github.com/SigNoz/signoz/issues/1986 I will try to add more details to it

Alejandro Decchi

01/05/2023, 1:23 PM

I updated the issue

Amol Umbark

01/05/2023, 1:26 PM

@Alejandro Decchi there was an issue in 0.11 where the channels were not getting into alert manager. in your log of alert manager i don't see channels.

Amol Umbark

01/05/2023, 1:27 PM

let me check that issue exists in .11.4 can you check your channel configuration and test it

Alejandro Decchi

01/05/2023, 1:38 PM

I have many channel added at signoz

Alejandro Decchi

01/05/2023, 1:39 PM

Sometimes, to resolve the issue I have to recretae the channel

Alejandro Decchi

01/05/2023, 1:40 PM

I am using

<http://docker.io/signoz/alertmanager:0.23.0-0.2|docker.io/signoz/alertmanager:0.23.0-0.2>

Amol Umbark

01/06/2023, 9:03 AM

@Alejandro Decchi I have posted an update in the issue. please take a look https://github.com/SigNoz/signoz/issues/1986

Amol Umbark

01/06/2023, 9:04 AM

I am unable to reproduce the issue. hence, need further inputs from you.

Alejandro Decchi

01/11/2023, 1:28 PM

@Amol Umbark thank you for your feedback, I will review the git hub issue to give more details

Alejandro Decchi

01/11/2023, 4:18 PM

I update the tickets 🙂

Amol Umbark

01/12/2023, 3:45 AM

Hi @Alejandro Decchi I looked at your response. the wget result is unexpected. would you be able to get on a call to resolve this. I am in IST time zone. please share a suitable time for huddle

Amol Umbark

01/12/2023, 4:49 AM

@Alejandro Decchi Also, is it possible to upgrade to v12 and try?

Amol Umbark

01/12/2023, 4:53 AM

@Prashant Shahi will it be safe to delete query-service pod to re-create it? I am supsecting the new deployment did not re-create the pod. and an older version of query-service is active

Prashant Shahi

01/12/2023, 1:07 PM

@Amol Umbark @Alejandro Decchi yes, I have not seen any issues with restarting query-service pods.

Amol Umbark

01/12/2023, 1:23 PM

@Alejandro Decchi can you please try deleting query service pod and capturing log as well after it starts. please share the log as well

Alejandro Decchi

01/18/2023, 2:42 PM

@Amol Umbark I will try to delete it and I will sahre the Logs

Alejandro Decchi

01/18/2023, 4:27 PM

Here some part of the Log output:

Copy code

2023-01-18T16:20:30.226Z	INFO	version/version.go:43	

SigNoz version   : v0.11.4
Commit SHA-1     : 8e55228
Commit timestamp : 2022-11-29T11:43:47Z
Branch           : HEAD
Go version       : go1.17.13

For SigNoz Official Documentation,  visit <https://signoz.io/docs>
For SigNoz Community Slack,         visit <http://signoz.io/slack>
For discussions about SigNoz,       visit <https://community.signoz.io>

Check SigNoz Github repo for license details.
Copyright 2022 SigNoz
2023-01-18T16:20:30.227Z	WARN	query-service/main.go:61	No JWT secret key is specified.
main.main
	/go/src/github.com/signoz/signoz/ee/query-service/main.go:61
runtime.main
	/usr/local/go/src/runtime/proc.go:255
2023-01-18T16:20:30.452Z	INFO	license/manager.go:124	No active license found, defaulting to basic plan
2023-01-18T16:20:30.452Z	INFO	app/server.go:100	Using ClickHouse as datastore ...
ts=2023-01-18T16:20:30.460035085Z caller=log.go:168 level=info msg="Loading configuration file" filename=/root/config/prometheus.yml
ts=2023-01-18T16:20:30.462181835Z caller=log.go:168 level=info msg="Completed loading of configuration file" filename=/root/config/prometheus.yml
2023-01-18T16:20:30.466Z	INFO	alertManager/notifier.go:94	Starting notifier with alert manager:[<http://signoz-alertmanager:9093/api/>]
2023-01-18T16:20:30.466Z	INFO	app/server.go:428	rules manager is ready
2023-01-18T16:20:30.468Z	DEBUG	rules/apiParams.go:85	postable rule(parsed):%!(EXTRA *rules.PostableRule=&{testing-error-rate   promql_rule 300000000000 0 {"compositeMetricQuery":{"builderQueries":{"A":{"queryName":"A","metricName":"","tagFilters":{"op":"AND","items":[]},"aggregateOperator":1,"expression":"A","disabled":false}},"promQueries":{"A":{"query":"(max(sum(rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`, status_code=\"STATUS_CODE_ERROR\"}[5m]) OR rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`, http_status_code=~\"5..\"}[5m]))*100/sum(rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`}[5m]))) \u003c 1000 OR vector(0))","disabled":false}},"panelType":0,"queryType":3},"op":"1","target":5,"matchType":"1"} map[severity:critical] map[description:A new alert] false <https://signoz.stg.travelx.it/alerts/edit?ruleId=2> [testing-alarms]  })
2023-01-18T16:20:30.468Z	DEBUG	rules/apiParams.go:126	postable rule:%!(EXTRA *rules.PostableRule=&{testing-error-rate   promql_rule 300000000000 60000000000 {"compositeMetricQuery":{"builderQueries":{"A":{"queryName":"A","metricName":"","tagFilters":{"op":"AND","items":[]},"aggregateOperator":1,"expression":"A","disabled":false}},"promQueries":{"A":{"query":"(max(sum(rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`, status_code=\"STATUS_CODE_ERROR\"}[5m]) OR rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`, http_status_code=~\"5..\"}[5m]))*100/sum(rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`}[5m]))) \u003c 1000 OR vector(0))","disabled":false}},"panelType":0,"queryType":3},"op":"1","target":5,"matchType":"1"} map[severity:critical] map[description:A new alert] false <https://signoz.stg.travelx.it/alerts/edit?ruleId=2> [testing-alarms]  }, string=	 condition, string={"compositeMetricQuery":{"builderQueries":{"A":{"queryName":"A","metricName":"","tagFilters":{"op":"AND","items":[]},"aggregateOperator":1,"expression":"A","disabled":false}},"promQueries":{"A":{"query":"(max(sum(rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`, status_code=\"STATUS_CODE_ERROR\"}[5m]) OR rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`, http_status_code=~\"5..\"}[5m]))*100/sum(rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`}[5m]))) \u003c 1000 OR vector(0))","disabled":false}},"panelType":0,"queryType":3},"op":"1","target":5,"matchType":"1"})
2023-01-18T16:20:30.468Z	DEBUG	rules/manager.go:345	msg:%!(EXTRA string=adding a new rule task, string=	 task name:, string=2-groupname)
2023-01-18T16:20:30.468Z	INFO	rules/promRule.go:94	msg:creating new alerting rule	 name:testing-error-rate	 condition:{"compositeMetricQuery":{"builderQueries":{"A":{"queryName":"A","metricName":"","tagFilters":{"op":"AND","items":[]},"aggregateOperator":1,"expression":"A","disabled":false}},"promQueries":{"A":{"query":"(max(sum(rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`, status_code=\"STATUS_CODE_ERROR\"}[5m]) OR rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`, http_status_code=~\"5..\"}[5m]))*100/sum(rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`}[5m]))) \u003c 1000 OR vector(0))","disabled":false}},"panelType":0,"queryType":3},"op":"1","target":5,"matchType":"1"}	 query:(max(sum(rate(signoz_calls_total{service_name="api-ktor-template", operation=~`HTTP GET|HTTP POST`, status_code="STATUS_CODE_ERROR"}[5m]) OR rate(signoz_calls_total{service_name="api-ktor-template", operation=~`HTTP GET|HTTP POST`, http_status_code=~"5.."}[5m]))*100/sum(rate(signoz_calls_total{service_name="api-ktor-template", operation=~`HTTP GET|HTTP POST`}[5m]))) < 1000 OR vector(0)) > 5.000000
2023-01-18T16:20:30.468Z	INFO	rules/promRuleTask.go:42	Initiating a new rule group:2-groupname	 frequency:1m0s

Amol Umbark

01/19/2023, 5:01 AM

@Alejandro Decchi thanks for sharing the log. did the recreating pod fix the issue?

Amol Umbark

01/19/2023, 5:01 AM

If not is there a good time we can get on a huddle. I am in IST

Alejandro Decchi

01/19/2023, 6:20 PM

It did not work. I am at GMT-300. When you are available ?

Alejandro Decchi

01/26/2023, 1:29 PM

any feedback ?

Amol Umbark

01/26/2023, 2:27 PM

hey .. can you try a fresh install of 0.11.4 in a separate environment. i suspect something from earlier versions is running and causing the issue

Amol Umbark

01/26/2023, 2:27 PM

i will confirm if i can connect tomorrow (friday) at 7pm ist (same time as you sent your last message)

Alejandro Decchi

01/28/2023, 4:23 PM

@Amol Umbark this issue happened in 2 different environment . If you want I am available nest Monday 30 at at 7pm (GMT-300)

Amol Umbark

01/31/2023, 6:09 AM

hey sorry I was on leave. can we connect today?

Amol Umbark

01/31/2023, 6:19 AM

Please ping me when you are online

Amol Umbark

01/31/2023, 6:24 AM

btw, i am available in IST timezone (GMT+5:30) 7pm (GMT-3) is like 3.30am here. can you take a look at this timeline and share a suitable time for you.

Amol Umbark

01/31/2023, 12:47 PM

I am available for next couple of hours. if we dont get to connect, please book a meeting from here. https://calendly.com/amol-umbark/30min?month=2023-02

Amol Umbark

01/31/2023, 2:26 PM

@Alejandro Decchi I am able to reproduce the error in one of the help installations. will have more update tomorrow

Alejandro Decchi

01/31/2023, 2:35 PM

@Amol Umbark perfect! It great to be able to reproduce this random error. I keep waiting your update

Amol Umbark

01/31/2023, 3:01 PM

@Prashant Shahi has resolved the issue. we will be publishing a PR soon. If possible, we will suggest a point fix so you can do it in your env and carry on

Alejandro Decchi

01/31/2023, 3:40 PM

@Amol Umbark can you share the image/tag to deploy in my enviornment ?

Amol Umbark

02/01/2023, 4:58 AM

@Alejandro Decchi the fix is in helm chart. so you would have to update helm chart and reinstall @Prashant Shahi can you please post here once the chart is updated.

Amol Umbark

02/01/2023, 9:45 AM

PR: https://github.com/SigNoz/charts/pull/157

Prashant Shahi

02/01/2023, 10:41 AM

@Alejandro Decchi The fix is merged and out. Follow our docs to upgrade to latest chart release: https://signoz.io/docs/operate/kubernetes/#upgrade-signoz-cluster Be sure to include

-f override-values.yaml

if you had passed custom values during installation.

Alejandro Decchi

02/01/2023, 1:17 PM

@Amol Umbark @Prashant Shahi so it is fixed at chart version 0.10.2 that was released 2 hours ago ?

Prashant Shahi

02/01/2023, 1:23 PM

@Alejandro Decchi yes, it is

Alejandro Decchi

02/01/2023, 1:23 PM

Perfect I will try it at Dev

Prashant Shahi

02/01/2023, 3:36 PM

@Alejandro Decchi Okay, do let us know if the issue persists or you face any issues upgrading.

Alejandro Decchi

02/01/2023, 3:56 PM

thanks @Prashant Shahi

Prashant Shahi

02/01/2023, 4:08 PM

happy to help 🙂

7 Views

Open in Slack

Previous Next