Hello Support what can cause Alert in signoz not t...
# support
n
Hello Support what can cause Alert in signoz not to trigger we are almost getting done with our POC and Demo just please answer my last question thanks for answering my questions before .
s
You can get rid of body EXISTS because each log will have body. What are you trying with group by
body
,
timestamp_field
?
n
Yes body and timestamp-field will work to get alert triggered
Hello Srikanth I am testing the following to get alert triggered
let me know which will work
s
The alert runs every 1 minute and only considers the last 5 minutes of data (with a max configurable time of 24 hours) by default. It will only trigger if the condition evaluates to true for evaluation period.
n
Hello Srikanth how do you write query for it
do make changes here for it elect toStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL 30 MINUTE) AS interval, toFloat64(count()) as value FROM signoz_logs.distributed_logs WHERE timestamp BETWEEN {{.start_timestamp_nano}} AND {{.end_timestamp_nano}} GROUP BY interval; -- available variables: -- {{.start_timestamp_nano}} -- {{.end_timestamp_nano}} -- required columns (or alias): -- value -- interval
do we need to make any change in alerts.yml groups: - name: ExampleCPULoadGroup rules: - alert: HighCpuLoad expr: system_cpu_load_average_1m > 0.1 for: 0m labels: severity: warning annotations: summary: High CPU load description: "CPU load is > 0.1\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
or do we need to make any changes in alertmanger.yml I just want to make sure
s
You don't need to change anything. Use the query builder to set up alert.
n
I did but my alerts don't trigger at all
s
Show me the how you created/updated it and show me some samples of recent logs when you expected the alert but it didn't trigger.
n
here are the logs 1695514384 00000001 WorkSpaceMana A WKSP0500I: Workspace configuration consistency check is disabled. 1695514384 00000001 AdminTool A ADMU3200I: Server launched. Waiting for initialization status. 1695514492 00000001 AdminTool A ADMU3000I: Server igawas02 open for e-business; process id is 15991150 1698342160 00000001 AdminTool A ADMU3000I: Server igawas02 open for e-business; process id is 15991151 1698342160 00000001 AdminTool A ADMU3000I: Server igawas02 open for e-business; process id is 15991152 1698354023 00000001 AdminTool A ADMU3000I: Server igawas02 open for e-business; process id is 15991153 1698354023 00000001 AdminTool A ADMU3000I: Server igawas02 open for e-business; process id is 15991154
log: Java Home = /opt/IBM/WebSphere/Ap1644436278 00000001 WorkSpaceMana A WKSP0500I: Workspace configuration consistency check is disabled. timestamp: 2023-10-27T092211-05:00 log: Java Home = /opt/IBM/WebSphere/Ap1644436278 00000001 WorkSpaceMana A WKSP0500I: Workspace configuration consistency check is disabled. timestamp: 2023-10-27T091701-05:00 log: Java Home = /opt/1638963361 00000001 AdminTool A ADMU3000I: Server igawas02 open for e-business; process id is 8192488 timestamp: 2023-10-27T091052-05:00 log: Java Home = /opt/IBM/WebSphere/Ap1644436278 00000001 WorkSpaceMana A WKSP0500I: Workspace configuration consistency check is disabled.
Screenshot 2023-10-30 at 12.10.00 PM.png
this is what is see now
s
Are there any logs that match the condition in the last five minutes otherwise it is correct.
n
we are using this eceivers: filelog: include: [ /tmp/NewLogTest1.log] start_at: beginning operators: - type: regex_parser regex: '^(?P<timestamp_field>(\d+))(?P<body>.*)' timestamp: layout_type: epoch layout: s parse_from: attributes.timestamp_field exporters: clickhouselogsexporter: dsn: tcp://clickhouse:9000/ docker_multi_node_cluster: ${DOCKER_MULTI_NODE_CLUSTER} timeout: 5s sending_queue: queue_size: 100 retry_on_failure: enabled: true initial_interval: 5s max_interval: 30s max_elapsed_time: 300s service: pipelines: logs: receivers: [filelog] exporters: [clickhouselogsexporter]
s
Are there any logs that match the condition in the last five minutes?
n
Yes I just made change in the log
Here is my change in the log 1698685986 00000001 AdminTool A ADMU3000I: Server igawas02 open for e-business; process id is 15991155
s
What is the alert query?
n
are you referring to this alert: LogMessageAlert expr: count_over_time({logbody="ADMU3000I: Server igawas02 open for e-business"} == 1) by (timestamp_field) > 0 for: 1m labels: severity: critical annotations: summary: Log Message Detected description: "The log message was detected: 'ADMU3000I: Server igawas02 open for e-business'"
s
Where is this count_over_time coming from? These are logs
Don't create a metric based alert for logs
n
I am trying to do log based alert now as test
I get the above query when I press this button
If you can provide me the answer I can forward it my architect in their team my last on this project is Tuesday I am doing my final documentation thanks
s
1. Use query builder 2. Upgrade signoz version so the default is query builder
n
Also my local host is over 30 days usage now is that cause of it i just want to make sure
s
No
n
Got I will forward your answer to my Architect