https://signoz.io logo
Join Slack
Powered by
Hi all, I need some help I have enabled Prometheu...
# support
m
Hi all, I need some help I have enabled Prometheus integration form TimeseriesDB, as stated here: https://docs.timescale.com/mst/latest/integrations/prometheus-mst/ then I have edited file: deploy/docker/clickhouse-setup/otel-collector-metrics-config.yaml and added this section:
Copy code
# Timescale Dev
        - job_name: timescale-dev-collector
          scrape_interval: 10s
          evaluation_interval: 10s
          scheme: https
          static_configs:
                  - targets: ['my-url:9273']
          tls_config:
            insecure_skip_verify: true
          basic_auth:
            username: milan
            password: password
But I don’t see anything in SigNoz. I have tried to add new dashboard with system_filesystem_usage_total, system_cpu_time_total, as PromQL, but nothing is displayed. How I can check Prometheus data from my TimeseriesDB on SigNoz instance? Thanks
s
But I don’t see anything in SigNoz. I have tried to add new dashboard with system_filesystem_usage_total, system_cpu_time_total, as PromQL, but nothing is displayed.
How are these related to the Timescale Prometheus metric data?
m
On Timescale TB I have Enabled Prometheus service integration On SigNoz I have set up otel-collector-metrics-config.yaml, as stated above. Now I am not sure how check are data flowing into SigNoz?
I want to see Timescale Prometheus data in SigNoz. Can someone help please?
p
@Milan Perovic can you verify that the 
my-url:9273
is accessible from VM/Cluster where signoz is deployed? After that, you can use suggestions from Query Builder from dashboard page to verify if any timescale related metrics have been collected.
Also, did you stop->remove->re-create otel-collector-metrics container?
since, we use volume mounting for the configuration in Docker. We often have to cleanup properly for any changes to show up.
m
1. I am able to see my-url:9273 from my signoz instance 2. I did stop, removed containers clickhouse-setup-otel-collector-1 and clickhouse-setup-otel-collector-metrics-1 and re-run install script 3. however, in dashboard when I try to save panel with e.g system_filesystem_usage_total I get response: “No Data” I am not sure anything is coming to my SigNoz instace from TimeseriesDB?
s
however, in dashboard when I try to save panel with e.g system_filesystem_usage_total I get response: “No Data”
Is the endpoint 
my-url:9273
expose this metric?
m
when I use GET my-url:9273/metrics, I am able to see huge response like cpu_usage_guest, diskio_merged_reads,… and when I try to use them in dashboard I get response: No Data
s
Are you sure the 
system_filesystem_usage_total
exists in response from my-url:9273/metrics
m
you are right, that does not exist. but I can’t display any of metrics that do exists
s
Did you mean you can’t display or you can display?
m
can’t
sorry
s
Can you open the network tab and share the response when you try to run query?
m
image.png
another metrics
image.png
I get 200 OK
s
What is the metric type and temporality for them?
m
# HELP disk_used_percent Telegraf collected metric # TYPE disk_used_percent gauge
s
Can you confirm if it was really scraped by otel-collector-metrics? Do logs show any errors? Make sure it is reachable.
m
That is the problem, I did prepare it, as stated above, but not sure how to see on Signoz side :-(\
s
Do logs show any errors?
m
Here is output
Copy code
when I do this:


docker logs clickhouse-setup-otel-collector-metrics-1 | grep error

* error decoding 'receivers': error reading receivers configuration for "prometheus": prometheus receiver failed to unmarshal yaml to prometheus config: yaml: unmarshal errors:
  line 17: field evaluation_interval not found in type config.ScrapeConfig
  line 29: field evaluation_interval not found in type config.ScrapeConfig
2023/06/07 20:09:50 application run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'receivers': error reading receivers configuration for "prometheus": prometheus receiver failed to unmarshal yaml to prometheus config: yaml: unmarshal errors:
  line 17: field evaluation_interval not found in type config.ScrapeConfig
  line 29: field evaluation_interval not found in type config.ScrapeConfig
Error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'receivers': error reading receivers configuration for "prometheus": prometheus receiver failed to unmarshal yaml to prometheus config: yaml: unmarshal errors:
  line 17: field evaluation_interval not found in type config.ScrapeConfig
  line 29: field evaluation_interval not found in type config.ScrapeConfig
2023/06/07 20:09:50 application run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

and this

docker logs clickhouse-setup-otel-collector-metrics-1 | grep error

7->10.33.105.29:63788: read: connection reset by peer	{"grpc_log": true}
2023-06-07T20:14:54.465Z	warn	zapgrpc/zapgrpc.go:191	[transport] transport: http2Server.HandleStreams failed to read frame: read tcp 173.19.0.7:4317->10.33.105.29:63822: read: connection reset by peer	{"grpc_log": true}
2023-06-07T20:14:55.280Z	warn	zapgrpc/zapgrpc.go:191	[transport] transport: http2Server.HandleStreams failed to read frame: read tcp 173.19.0.7:4317->10.33.144.245:57520: read: connection reset by peer	{"grpc_log": true}
2023-06-07T20:14:59.327Z	warn	zapgrpc/zapgrpc.go:191	[transport] transport: http2Server.HandleStreams failed to read frame: read tcp 173.19.0.7:4317->10.33.144.245:57546: read: connection reset by peer	{"grpc_log": true}
2023-06-07T20:15:06.169Z	warn	zapgrpc/zapgrpc.go:191	[transport] transport: http2Server.HandleStreams failed to read frame: read tcp 173.19.0.7:4317->10.33.105.29:29906: read: connection reset by peer	{"grpc_log": true}
2023-06-07T20:15:06.821Z	warn	zapgrpc/zapgrpc.go:191	[transport] transport: http2Server.HandleStreams failed to read frame: read tcp 173.19.0.7:4317->10.33.144.245:57492: read: connection reset by peer	{"grpc_log": true}
s
It says config is invalid.
Copy code
field evaluation_interval not found in type config.ScrapeConfig
m
I managed to set all up. Issue was that env variable evaluation_interval: 10s was not needed.
Thank for your support!
13 Views
Open in Slack
PreviousNext
Powered by