https://signoz.io logo
Title
a

Alexei Zenin

01/12/2023, 10:14 PM
Does anybody know which tables these metrics are stored in? https://github.com/SigNoz/signoz-otel-collector/tree/main/processor/signozspanmetricsprocessor. I am trying to query them outside of SigNoz and create some alerts based on error rates, http latency etc. Any example queries that I could use to do these in Clickhouse SQL?
p

Pranay

01/13/2023, 12:27 AM
@Alexei Zenin Curious, why do you want to query them outside SigNoz and create alerts? What was not supported in the SigNoz alerting capabilities?
a

Ankit Nayan

01/14/2023, 2:40 PM
@Alexei Zenin querying metrics from clickhouse without our query interface might be difficult because of query complexity. Can you share what you are trying to achieve?
tables ->
signoz_metrics.time_series_v2
and
signoz_metrics.samples_v2
a

Alexei Zenin

01/16/2023, 8:03 PM
Our setup is to have alerts in Grafana since all of them will be there for alerting on all datasources (cloudwatch, prometheus, clickhouse and any future ones that come up). I’ve played around with the dashboard viewer and tried out plotting average latency for a service but am getting negative numbers as seen below:
hmm, but that PR seems more to be focused on the service page
would upgrading to 0.13 fix this in the dashboard page?
p

Pranay

01/18/2023, 4:40 PM
@Srikanth Chekuri do you have ideas on this?
a

Ankit Nayan

01/18/2023, 5:03 PM
@Alexei Zenin you are choosing incorrect
metric_name
. There are 8 steps to the tutorial using a formula
A/B
. The 2 metrics are
signoz_latency_sum
and
signoz_latency_count
. Make sure you have followed all the steps correctly
a

Alexei Zenin

01/18/2023, 5:10 PM
ah right, did it fully this time, seems to be the same issue:
a

Ankit Nayan

01/18/2023, 5:19 PM
@Srikanth Chekuri will be able to dig deeper 🙂
s

Srikanth Chekuri

01/18/2023, 5:20 PM
Use the legend {{service_name}} in all sections. That aside, Can you share the graphs for A, B, C separately with other two disabled in each?
a

Alexei Zenin

01/18/2023, 6:46 PM
Only B:
Only A:
Signoz version 0.11.2, collector version signoz/signoz-otel-collector:0.55.3
s

Srikanth Chekuri

01/19/2023, 12:42 AM
The chart displays the negatives when the counters reset. Was there any incident of otel-collector-metrics restarting?
a

Alexei Zenin

01/26/2023, 5:56 PM
we have our collectors restart often with ec2 spot instances
s

Srikanth Chekuri

01/27/2023, 12:04 AM
That resets the counters, and the frequent resets lead to this graph where the value becomes negative. But otherwise if the y-axis is selected with appropriate scale (millis in this instance I believe) do you see the correct value for positive numbers?