https://signoz.io logo
Join Slack
Powered by
This message was deleted.
# support
s
This message was deleted.
v
You might want follow our troubleshooting docs: https://signoz.io/docs/install/troubleshooting/
d
OTEL_LOG_LEVEL=debug ---> this is otel debug. What I want is to debug SigNoz. My prometheus endpoints are failing to scrape data clickhouse-setup-clickhouse-1 | 2022.05.19 080449.202797 [ 239 ] {77a168c3-2c0c-4376-be89-4097198d6409} <Information> executeQuery: Read 36841 rows, 9.32 MiB in 0.12079366 sec., 304991 rows/sec., 77.18 MiB/sec. clickhouse-setup-otel-collector-metrics-1 | 2022-05-19T080449.690Z warn internal/otlp_metricsbuilder.go:159 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus", "scrape_timestamp": 1652947489677, "target_labels": "map[__name__:up instance x.x.x.x:3006 job:my-ms]"} clickhouse-setup-clickhouse-1 | 2022.05.19 080449.930678 [ 88 ] {fb16ab0c-cada-4cf0-9548-81bb07a27072} <Information> executeQuery: Read 36841 rows, 9.32 MiB in 0.332335889 sec., 110854 rows/sec., 28.05 MiB/sec. clickhouse-setup-clickhouse-1 | 2022.05.19 080449.936330 [ 234 ] {ba1d719c-6a90-4d15-905f-a894f933ed4e} <Information> executeQuery: Read 36841 rows, 9.32 MiB in 0.353564869 sec., 104198 rows/sec., 26.37 MiB/sec.
v
@Srikanth Chekuri
s
@Danish Ahmad Ansari Are the scrape configs reacheable?
d
@Srikanth Chekuri did not get . I am running few microservices in server 1 + signoz server 2 microservices are able to pick prometheus metrics and show metrics in dashboard. server 1 is trying to call some local IP address for prometheus. server 2 is calling the IP address of server 1 properly and then picking metrics. problems are occuring when signoz and microservices are running on same server . I am attaching log files
This log from microservices which is running in other server2. Prometheus is working fine in this. 19-05-2022 102636.119 [http-nio-3005-exec-9] DEBUG o.s.s.w.a.AnonymousAuthenticationFilter.doFilter - Populated SecurityContextHolder with anonymous token: 'org.springframework.security.authentication.AnonymousAuthenticationToken@abcdef: Principal: anonymousUser; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@abcdes: _*RemoteIpAddress: X1.X1.X1.X1*_; SessionId: null; Granted Authorities: ROLE_ANONYMOUS' X1.X1.X1.X1 ---> server1 IP Address ---------------------------------------------------------------------------------------------------------------- This log from microservices which is running in other server1. Prometheus is NOT working . Server1 has Signoz running. org.springframework.security.authentication.AnonymousAuthenticationToken@abcdef: Principal: anonymousUser; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@abcdes: RemoteIpAddress:somelocalhost; SessionId: null; Granted Authorities: ROLE_ANONYMOUS'
s
So you are not able to see metics from server 1? Is it the same application run on both server as different instances or are they two different applications?
d
Different applications
s
Does the application in server 1 configured to export metrics and works when run standalone?
d
Yes yes. It's show signoz metrics for signoz. I am only stuck with Prometheus metrics
s
So you are saying the application (running in server1) is exporting metrics?
d
When and signoz and application run in same server this Prometheus metrics are not coming while doing query on promql box
When I use curl command the metrics are coming in command line server 1
Only the problem is front end
s
What is the curl command used? Did you configure the scrape job correctly in yaml?
d
s
But from the logs, the scrape job failed for instance x.x.x.x:3006?
d
in server1 it failed
s
What is the scrape target for which it failed?
d
can you elaborate ?
s
See you have two different applications running in different servers, right? These applications have instrumentation in place that will export the data when Prometheus scrapes for it? What is the scrape target for which it failed?
d
It's the server1_IP/actuator/prometheus
signoz on same server + microservice microservice fail to scrape
Am I able to make its clear ?
s
No, it's the prometheus which scrapes not any microservice. What are all scrape targets written in yaml config file? If you curl the <scrape_target>/metrics are you able to see the metrics?
d
yes @Srikanth Chekuri. When I use curl command with the scrape target the prometheus queries are coming in the terminal. Problem is it display no data on signoz for same server.
curl commnad queries have successful response. that prometheus is working in backend but in signoz front end its the issue.
s
the prometheus queries are coming in the terminal
What does this mean? Did you mean to say that you could see metrics? I want to know one thing first if the application in server1 has some target where the metrics are available. And then if that target is emitting metrics by request <scrape_target>/metrics (here i am not talking about queries or prom ui, I am talking about the application is being able to export metrics)?
d
how can we check that ? curl ?
s
yes
d
every ms is able to. yes its able to export metrics
s
What's the command you used and can you share the sample result?
d
let me know if you have checked
s
This is within the same instance machine . right?
d
yes
s
And these are not coming in UI?
d
exactly. no data to display
s
Is this application reachable from the signoz? There is chance to n/w issues? You could try run the same command from signoz and check the same
d
Signoz running on same server with instances running is causing issue. Its definitley network issue.
s
signoz must able to reach it, then only it can scrape the data and show any data in UI. you should be able to see data if you can get your n/w issues sorted out
d
will let you know
@Srikanth Chekuri thanks
Hi @Srikanth Chekuri , I am trying to run Signoz in different network in same server. Added network_mode : new_network. Any solution ? 2022-05-20T091554.968Z INFO app/server.go:108 Using ClickHouse as datastore ... 2022-05-20T091554.981Z ERROR clickhouseReader/reader.go:90 error connecting to primary db: dial tcp: lookup clickhouse on 127.0.0.1153 server misbehaving go.signoz.io/query-service/app/clickhouseReader.NewReader /go/src/github.com/signoz/signoz/pkg/query-service/app/clickhouseReader/reader.go:90 go.signoz.io/query-service/app.(*Server).createHTTPServer /go/src/github.com/signoz/signoz/pkg/query-service/app/server.go:109 go.signoz.io/query-service/app.NewServer /go/src/github.com/signoz/signoz/pkg/query-service/app/server.go:83 main.main /go/src/github.com/signoz/signoz/pkg/query-service/main.go:45 runtime.main /usr/local/go/src/runtime/proc.go:203
s
I am not sure I understood this. There is nothing signoz specific I could help you with here. And it's hard for me to say anything because i won't have context of your setup.
d
Can you try running signoz and microservice on one server. and some other microservice on other server. see if you get prometheus metrics.
s
It's not practical for me to simulate your environment to test this.
👍 1
16 Views
Open in Slack
PreviousNext
Powered by