Alejandro Decchi
09/16/2022, 12:49 PMsignoz-query-services
, it constantly restarting. Logs looks ok:
2022-09-16T12:44:52.645Z INFO app/server.go:84 Using ClickHouse as datastore ...
ts=2022-09-16T12:44:52.652594487Z caller=log.go:168 level=info msg="Loading configuration file" filename=/root/config/prometheus.yml
2022-09-16T12:44:52.654Z INFO alertManager/notifier.go:94 Starting notifier with alert manager:[<http://signoz-alertmanager:9093/api/>]
2022-09-16T12:44:52.654Z INFO app/server.go:396 rules manager is ready
ts=2022-09-16T12:44:52.656830443Z caller=log.go:168 level=info msg="Completed loading of configuration file" filename=/root/config/prometheus.yml
2022-09-16T12:44:52.657Z INFO alertManager/notifier.go:126 msg: Initiating alert notifier...
2022-09-16T12:44:52.658Z INFO app/server.go:273 Query server started listening on 0.0.0.0:8080...
2022-09-16T12:44:52.658Z INFO app/server.go:286 Query server started listening on private port 0.0.0.0:8085...
starting private http
2022-09-16T12:44:52.658Z INFO app/server.go:312 Starting HTTP server{port 11 8080 <nil>} {addr 15 0 0.0.0.0:8080 <nil>}
2022-09-16T12:44:52.658Z INFO app/server.go:324 Starting pprof server{addr 15 0 0.0.0.0:6060 <nil>}
2022-09-16T12:44:52.658Z INFO app/server.go:338 Starting Private HTTP server{port 11 8085 <nil>} {addr 15 0 0.0.0.0:8085 <nil>}
2022-09-16T12:44:52.700Z INFO app/server.go:189 /api/v1/version timeTaken: 27.736µs
2022-09-16T12:44:55.562Z INFO app/server.go:189 /api/v1/version timeTaken: 16.782µs
2022-09-16T12:44:55.562Z INFO app/server.go:189 /api/v1/version timeTaken: 14.205µs
Is a way to troubleshoot it ? to put on Debug ?
Second, on the other hand, is a way to set signoz-otel-collector
as deployment in place of daemonset ?Srikanth Chekuri
09/16/2022, 12:52 PMAlejandro Decchi
09/16/2022, 1:03 PMkubectl get pod signoz-query-service-0 -n platform -o jsonpath='{.status.containerStatuses[?(@.name=="signoz-query-service")].state.terminated.exitCode}'
Srikanth Chekuri
09/16/2022, 1:17 PMlastState.terminated.exitCode
? The most common and known one is OOM with 137 when the query service doesn’t have enough memory. What are the resource limits for this?Alejandro Decchi
09/16/2022, 1:20 PMresources:
limits:
memory: 2000Mi
requests:
cpu: 500m
memory: 2000Mi
Srikanth Chekuri
09/16/2022, 1:23 PMselect count() from signoz_metrics.time_series_v2;
from clickhouse client)? Right now the query service needs resources adjusted for the volume of time series. So if you can share the what’s the avg number of time series you expect in total a resource limit can be suggested.Alejandro Decchi
09/16/2022, 1:31 PMSrikanth Chekuri
09/16/2022, 1:32 PM0.11.1
? We did some improvement and it should run fine for the above number.Alejandro Decchi
09/16/2022, 1:42 PMSrikanth Chekuri
09/16/2022, 1:46 PMAlejandro Decchi
09/16/2022, 1:47 PMSrikanth Chekuri
09/16/2022, 2:06 PMAlejandro Decchi
09/16/2022, 2:20 PMSrikanth Chekuri
09/16/2022, 2:25 PMAlejandro Decchi
09/16/2022, 7:59 PMPrashant Shahi
09/19/2022, 5:33 AMk8s-infra
chart is introduced to handle the logs and metrics collection from K8s cluster.