This message was deleted.
# supports
Slackbot
10/26/2023, 7:34 PMThis message was deleted.
s
Srikanth Chekuri
10/27/2023, 11:41 AMYou can get rid of body EXISTS because each log will have body. What are you trying with group by
bodytimestamp_fieldn
Noor Ali
10/27/2023, 12:23 PMYes body and timestamp-field will work to get alert triggered
Noor Ali
10/27/2023, 12:28 PMHello Srikanth I am testing the following to get alert triggered
Noor Ali
10/27/2023, 12:29 PMlet me know which will work
s
Srikanth Chekuri
10/30/2023, 4:55 PMThe alert runs every 1 minute and only considers the last 5 minutes of data (with a max configurable time of 24 hours) by default. It will only trigger if the condition evaluates to true for evaluation period.
n
Noor Ali
10/30/2023, 4:57 PMHello Srikanth how do you write query for it
Noor Ali
10/30/2023, 4:58 PMdo make changes here for it elect
toStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL 30 MINUTE) AS interval,
toFloat64(count()) as value
FROM signoz_logs.distributed_logs
WHERE timestamp BETWEEN {{.start_timestamp_nano}} AND {{.end_timestamp_nano}}
GROUP BY interval;
-- available variables:
-- {{.start_timestamp_nano}}
-- {{.end_timestamp_nano}}
-- required columns (or alias):
-- value
-- interval
Noor Ali
10/30/2023, 5:02 PMdo we need to make any change in alerts.yml groups:
- name: ExampleCPULoadGroup
rules:
- alert: HighCpuLoad
expr: system_cpu_load_average_1m > 0.1
for: 0m
labels:
severity: warning
annotations:
summary: High CPU load
description: "CPU load is > 0.1\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
Noor Ali
10/30/2023, 5:03 PMor do we need to make any changes in alertmanger.yml I just want to make sure
s
Srikanth Chekuri
10/30/2023, 5:04 PMYou don't need to change anything. Use the query builder to set up alert.
n
Noor Ali
10/30/2023, 5:05 PMI did but my alerts don't trigger at all
s
Srikanth Chekuri
10/30/2023, 5:07 PMShow me the how you created/updated it and show me some samples of recent logs when you expected the alert but it didn't trigger.
n
Noor Ali
10/30/2023, 5:08 PMhere are the logs 1695514384 00000001 WorkSpaceMana A WKSP0500I: Workspace configuration consistency check is disabled.
1695514384 00000001 AdminTool A ADMU3200I: Server launched. Waiting for initialization status.
1695514492 00000001 AdminTool A ADMU3000I: Server igawas02 open for e-business; process id is 15991150
1698342160 00000001 AdminTool A ADMU3000I: Server igawas02 open for e-business; process id is 15991151
1698342160 00000001 AdminTool A ADMU3000I: Server igawas02 open for e-business; process id is 15991152
1698354023 00000001 AdminTool A ADMU3000I: Server igawas02 open for e-business; process id is 15991153
1698354023 00000001 AdminTool A ADMU3000I: Server igawas02 open for e-business; process id is 15991154
Noor Ali
10/30/2023, 5:09 PMlog:
Java Home = /opt/IBM/WebSphere/Ap1644436278 00000001 WorkSpaceMana A WKSP0500I: Workspace configuration consistency check is disabled.
timestamp:
2023-10-27T092211-05:00
log:
Java Home = /opt/IBM/WebSphere/Ap1644436278 00000001 WorkSpaceMana A WKSP0500I: Workspace configuration consistency check is disabled.
timestamp:
2023-10-27T091701-05:00
log:
Java Home = /opt/1638963361 00000001 AdminTool A ADMU3000I: Server igawas02 open for e-business; process id is 8192488
timestamp:
2023-10-27T091052-05:00
log:
Java Home = /opt/IBM/WebSphere/Ap1644436278 00000001 WorkSpaceMana A WKSP0500I: Workspace configuration consistency check is disabled.
Noor Ali
10/30/2023, 5:11 PMthis is what is see now
s
Srikanth Chekuri
10/30/2023, 5:12 PMAre there any logs that match the condition in the last five minutes otherwise it is correct.
n
Noor Ali
10/30/2023, 5:12 PMwe are using this eceivers:
filelog:
include: [ /tmp/NewLogTest1.log]
start_at: beginning
operators:
- type: regex_parser
regex: '^(?P<timestamp_field>(\d+))(?P<body>.*)'
timestamp:
layout_type: epoch
layout: s
parse_from: attributes.timestamp_field
exporters:
clickhouselogsexporter:
dsn: tcp://clickhouse:9000/
docker_multi_node_cluster: ${DOCKER_MULTI_NODE_CLUSTER}
timeout: 5s
sending_queue:
queue_size: 100
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
service:
pipelines:
logs:
receivers: [filelog]
exporters: [clickhouselogsexporter]
s
Srikanth Chekuri
10/30/2023, 5:13 PMAre there any logs that match the condition in the last five minutes?
n
Noor Ali
10/30/2023, 5:15 PMYes I just made change in the log
Noor Ali
10/30/2023, 5:16 PMHere is my change in the log 1698685986 00000001 AdminTool A ADMU3000I: Server igawas02 open for e-business; process id is 15991155
s
Srikanth Chekuri
10/30/2023, 5:16 PMWhat is the alert query?
n
Noor Ali
10/30/2023, 5:18 PMare you referring to this alert: LogMessageAlert
expr: count_over_time({logbody="ADMU3000I: Server igawas02 open for e-business"} == 1) by (timestamp_field) > 0
for: 1m
labels:
severity: critical
annotations:
summary: Log Message Detected
description: "The log message was detected: 'ADMU3000I: Server igawas02 open for e-business'"
s
Srikanth Chekuri
10/30/2023, 5:19 PMWhere is this count_over_time coming from? These are logs
Srikanth Chekuri
10/30/2023, 5:20 PMDon't create a metric based alert for logs
n
Noor Ali
10/30/2023, 5:20 PMI am trying to do log based alert now as test
Noor Ali
10/30/2023, 5:22 PMI get the above query when I press this button
Noor Ali
10/30/2023, 5:23 PMIf you can provide me the answer I can forward it my architect in their team my last on this project is Tuesday I am doing my final documentation thanks
s
Srikanth Chekuri
10/30/2023, 5:24 PM1. Use query builder
2. Upgrade signoz version so the default is query builder
n
Noor Ali
10/30/2023, 5:24 PMAlso my local host is over 30 days usage now is that cause of it i just want to make sure
s
Srikanth Chekuri
10/30/2023, 5:25 PMNo
n
Noor Ali
10/30/2023, 5:25 PMGot I will forward your answer to my Architect
30 Views