This message was deleted.
s
Slackbot
01/04/2023, 3:32 PMThis message was deleted.
a
Ankit Nayan
01/04/2023, 3:35 PM
cc: @Amol Umbark
a
Amol Umbark
01/05/2023, 7:19 AMHi @Alejandro Decchi are you able to consistently create the issue #1? if so please share exact steps, log of query service and alert manager and also see jf the alert shows up in triggered notifications
Amol Umbark
01/05/2023, 7:20 AMi will try to reproduce the test alert second issue issue. and get back.
a
Alejandro Decchi
01/05/2023, 1:05 PM@Amol Umbark thanks for your reply. @Pranay create the issue https://github.com/SigNoz/signoz/issues/1986 I will try to add more details to it
Alejandro Decchi
01/05/2023, 1:23 PMI updated the issue
a
Amol Umbark
01/05/2023, 1:26 PM@Alejandro Decchi there was an issue in 0.11 where the channels were not getting into alert manager. in your log of alert manager i don't see channels.
Amol Umbark
01/05/2023, 1:27 PMlet me check that issue exists in .11.4
can you check your channel configuration and test it
🙌 1
a
Alejandro Decchi
01/05/2023, 1:38 PMI have many channel added at signoz
Alejandro Decchi
01/05/2023, 1:39 PMSometimes, to resolve the issue I have to recretae the channel
Alejandro Decchi
01/05/2023, 1:40 PMI am using
<http://docker.io/signoz/alertmanager:0.23.0-0.2|docker.io/signoz/alertmanager:0.23.0-0.2>a
Amol Umbark
01/06/2023, 9:03 AM@Alejandro Decchi I have posted an update in the issue. please take a look
https://github.com/SigNoz/signoz/issues/1986
Amol Umbark
01/06/2023, 9:04 AMI am unable to reproduce the issue. hence, need further inputs from you.
a
Alejandro Decchi
01/11/2023, 1:28 PM@Amol Umbark thank you for your feedback, I will review the git hub issue to give more details
Alejandro Decchi
01/11/2023, 4:18 PMI update the tickets 🙂
a
Amol Umbark
01/12/2023, 3:45 AMHi @Alejandro Decchi I looked at your response. the wget result is unexpected. would you be able to get on a call to resolve this.
I am in IST time zone. please share a suitable time for huddle
Amol Umbark
01/12/2023, 4:49 AM@Alejandro Decchi Also, is it possible to upgrade to v12 and try?
Amol Umbark
01/12/2023, 4:53 AM@Prashant Shahi will it be safe to delete query-service pod to re-create it? I am supsecting the new deployment did not re-create the pod. and an older version of query-service is active
p
Prashant Shahi
01/12/2023, 1:07 PM@Amol Umbark @Alejandro Decchi yes, I have not seen any issues with restarting query-service pods.
a
Amol Umbark
01/12/2023, 1:23 PM@Alejandro Decchi can you please try deleting query service pod and capturing log as well after it starts. please share the log as well
a
Alejandro Decchi
01/18/2023, 2:42 PM@Amol Umbark I will try to delete it and I will sahre the Logs
Alejandro Decchi
01/18/2023, 4:27 PMHere some part of the Log output:
Copy code
2023-01-18T16:20:30.226Z INFO version/version.go:43
SigNoz version : v0.11.4
Commit SHA-1 : 8e55228
Commit timestamp : 2022-11-29T11:43:47Z
Branch : HEAD
Go version : go1.17.13
For SigNoz Official Documentation, visit <https://signoz.io/docs>
For SigNoz Community Slack, visit <http://signoz.io/slack>
For discussions about SigNoz, visit <https://community.signoz.io>
Check SigNoz Github repo for license details.
Copyright 2022 SigNoz
2023-01-18T16:20:30.227Z WARN query-service/main.go:61 No JWT secret key is specified.
main.main
/go/src/github.com/signoz/signoz/ee/query-service/main.go:61
runtime.main
/usr/local/go/src/runtime/proc.go:255
2023-01-18T16:20:30.452Z INFO license/manager.go:124 No active license found, defaulting to basic plan
2023-01-18T16:20:30.452Z INFO app/server.go:100 Using ClickHouse as datastore ...
ts=2023-01-18T16:20:30.460035085Z caller=log.go:168 level=info msg="Loading configuration file" filename=/root/config/prometheus.yml
ts=2023-01-18T16:20:30.462181835Z caller=log.go:168 level=info msg="Completed loading of configuration file" filename=/root/config/prometheus.yml
2023-01-18T16:20:30.466Z INFO alertManager/notifier.go:94 Starting notifier with alert manager:[<http://signoz-alertmanager:9093/api/>]
2023-01-18T16:20:30.466Z INFO app/server.go:428 rules manager is ready
2023-01-18T16:20:30.468Z DEBUG rules/apiParams.go:85 postable rule(parsed):%!(EXTRA *rules.PostableRule=&{testing-error-rate promql_rule 300000000000 0 {"compositeMetricQuery":{"builderQueries":{"A":{"queryName":"A","metricName":"","tagFilters":{"op":"AND","items":[]},"aggregateOperator":1,"expression":"A","disabled":false}},"promQueries":{"A":{"query":"(max(sum(rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`, status_code=\"STATUS_CODE_ERROR\"}[5m]) OR rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`, http_status_code=~\"5..\"}[5m]))*100/sum(rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`}[5m]))) \u003c 1000 OR vector(0))","disabled":false}},"panelType":0,"queryType":3},"op":"1","target":5,"matchType":"1"} map[severity:critical] map[description:A new alert] false <https://signoz.stg.travelx.it/alerts/edit?ruleId=2> [testing-alarms] })
2023-01-18T16:20:30.468Z DEBUG rules/apiParams.go:126 postable rule:%!(EXTRA *rules.PostableRule=&{testing-error-rate promql_rule 300000000000 60000000000 {"compositeMetricQuery":{"builderQueries":{"A":{"queryName":"A","metricName":"","tagFilters":{"op":"AND","items":[]},"aggregateOperator":1,"expression":"A","disabled":false}},"promQueries":{"A":{"query":"(max(sum(rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`, status_code=\"STATUS_CODE_ERROR\"}[5m]) OR rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`, http_status_code=~\"5..\"}[5m]))*100/sum(rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`}[5m]))) \u003c 1000 OR vector(0))","disabled":false}},"panelType":0,"queryType":3},"op":"1","target":5,"matchType":"1"} map[severity:critical] map[description:A new alert] false <https://signoz.stg.travelx.it/alerts/edit?ruleId=2> [testing-alarms] }, string= condition, string={"compositeMetricQuery":{"builderQueries":{"A":{"queryName":"A","metricName":"","tagFilters":{"op":"AND","items":[]},"aggregateOperator":1,"expression":"A","disabled":false}},"promQueries":{"A":{"query":"(max(sum(rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`, status_code=\"STATUS_CODE_ERROR\"}[5m]) OR rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`, http_status_code=~\"5..\"}[5m]))*100/sum(rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`}[5m]))) \u003c 1000 OR vector(0))","disabled":false}},"panelType":0,"queryType":3},"op":"1","target":5,"matchType":"1"})
2023-01-18T16:20:30.468Z DEBUG rules/manager.go:345 msg:%!(EXTRA string=adding a new rule task, string= task name:, string=2-groupname)
2023-01-18T16:20:30.468Z INFO rules/promRule.go:94 msg:creating new alerting rule name:testing-error-rate condition:{"compositeMetricQuery":{"builderQueries":{"A":{"queryName":"A","metricName":"","tagFilters":{"op":"AND","items":[]},"aggregateOperator":1,"expression":"A","disabled":false}},"promQueries":{"A":{"query":"(max(sum(rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`, status_code=\"STATUS_CODE_ERROR\"}[5m]) OR rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`, http_status_code=~\"5..\"}[5m]))*100/sum(rate(signoz_calls_total{service_name=\"api-ktor-template\", operation=~`HTTP GET|HTTP POST`}[5m]))) \u003c 1000 OR vector(0))","disabled":false}},"panelType":0,"queryType":3},"op":"1","target":5,"matchType":"1"} query:(max(sum(rate(signoz_calls_total{service_name="api-ktor-template", operation=~`HTTP GET|HTTP POST`, status_code="STATUS_CODE_ERROR"}[5m]) OR rate(signoz_calls_total{service_name="api-ktor-template", operation=~`HTTP GET|HTTP POST`, http_status_code=~"5.."}[5m]))*100/sum(rate(signoz_calls_total{service_name="api-ktor-template", operation=~`HTTP GET|HTTP POST`}[5m]))) < 1000 OR vector(0)) > 5.000000
2023-01-18T16:20:30.468Z INFO rules/promRuleTask.go:42 Initiating a new rule group:2-groupname frequency:1m0s👀 1
a
Amol Umbark
01/19/2023, 5:01 AM@Alejandro Decchi thanks for sharing the log. did the recreating pod fix the issue?
Amol Umbark
01/19/2023, 5:01 AMIf not is there a good time we can get on a huddle. I am in IST
a
Alejandro Decchi
01/19/2023, 6:20 PMIt did not work. I am at GMT-300. When you are available ?
Alejandro Decchi
01/26/2023, 1:29 PMany feedback ?
a
Amol Umbark
01/26/2023, 2:27 PMhey .. can you try a fresh install of 0.11.4 in a separate environment. i suspect something from earlier versions is running and causing the issue
Amol Umbark
01/26/2023, 2:27 PMi will confirm if i can connect tomorrow (friday) at 7pm ist (same time as you sent your last message)
a
Alejandro Decchi
01/28/2023, 4:23 PM@Amol Umbark this issue happened in 2 different environment . If you want I am available nest Monday 30 at at 7pm (GMT-300)
a
Amol Umbark
01/31/2023, 6:09 AMhey sorry I was on leave. can we connect today?
Amol Umbark
01/31/2023, 6:19 AMPlease ping me when you are online
Amol Umbark
01/31/2023, 6:24 AMbtw, i am available in IST timezone (GMT+5:30)
7pm (GMT-3) is like 3.30am here.
can you take a look at this timeline and share a suitable time for you.
Amol Umbark
01/31/2023, 12:47 PMI am available for next couple of hours. if we dont get to connect, please book a meeting from here.
https://calendly.com/amol-umbark/30min?month=2023-02
Amol Umbark
01/31/2023, 2:26 PM@Alejandro Decchi I am able to reproduce the error in one of the help installations. will have more update tomorrow
🙌 1
a
Alejandro Decchi
01/31/2023, 2:35 PM@Amol Umbark perfect! It great to be able to reproduce this random error. I keep waiting your update
a
Amol Umbark
01/31/2023, 3:01 PM@Prashant Shahi has resolved the issue. we will be publishing a PR soon. If possible, we will suggest a point fix so you can do it in your env and carry on
a
Alejandro Decchi
01/31/2023, 3:40 PM@Amol Umbark can you share the image/tag to deploy in my enviornment ?
a
Amol Umbark
02/01/2023, 4:58 AM@Alejandro Decchi the fix is in helm chart. so you would have to update helm chart and reinstall
@Prashant Shahi can you please post here once the chart is updated.
Amol Umbark
02/01/2023, 9:45 AMp
Prashant Shahi
02/01/2023, 10:41 AM@Alejandro Decchi The fix is merged and out.
Follow our docs to upgrade to latest chart release:
https://signoz.io/docs/operate/kubernetes/#upgrade-signoz-cluster
Be sure to include
-f override-values.yamla
Alejandro Decchi
02/01/2023, 1:17 PM@Amol Umbark @Prashant Shahi so it is fixed at chart version 0.10.2 that was released 2 hours ago ?
p
Prashant Shahi
02/01/2023, 1:23 PM@Alejandro Decchi yes, it is
a
Alejandro Decchi
02/01/2023, 1:23 PMPerfect I will try it at Dev
p
Prashant Shahi
02/01/2023, 3:36 PM@Alejandro Decchi Okay, do let us know if the issue persists or you face any issues upgrading.
a
Alejandro Decchi
02/01/2023, 3:56 PMthanks @Prashant Shahi
p
Prashant Shahi
02/01/2023, 4:08 PMhappy to help 🙂
13 Views
SigNoz Community
SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.
Powered by