Does anybody know which tables these metrics are stored in? https://github.com/SigNoz/signoz-otel-collector/tree/main/processor/signozspanmetricsprocessor. I am trying to query them outside of SigNoz and create some alerts based on error rates, http latency etc. Any example queries that I could use to do these in Clickhouse SQL?

@Alexei Zenin Curious, why do you want to query them outside SigNoz and create alerts? What was not supported in the SigNoz alerting capabilities?

@Alexei Zenin querying metrics from clickhouse without our query interface might be difficult because of query complexity. Can you share what you are trying to achieve?

Our setup is to have alerts in Grafana since all of them will be there for alerting on all datasources (cloudwatch, prometheus, clickhouse and any future ones that come up). I’ve played around with the dashboard viewer and tried out plotting average latency for a service but am getting negative numbers as seen below:

@Srikanth Chekuri do you have ideas on this?

@Alexei Zenin you are choosing incorrect

metric_name

. There are 8 steps to the tutorial using a formula

A/B

. The 2 metrics are

signoz_latency_sum

and

signoz_latency_count

. Make sure you have followed all the steps correctly

ah right, did it fully this time, seems to be the same issue:

@Srikanth Chekuri will be able to dig deeper 🙂

Use the legend {{service_name}} in all sections. That aside, Can you share the graphs for A, B, C separately with other two disabled in each?

Only B:

The chart displays the negatives when the counters reset. Was there any incident of otel-collector-metrics restarting?

we have our collectors restart often with ec2 spot instances

That resets the counters, and the frequent resets lead to this graph where the value becomes negative. But otherwise if the y-axis is selected with appropriate scale (millis in this instance I believe) do you see the correct value for positive numbers?