This message was deleted.
# supports
Slackbot
02/26/2024, 10:47 AMThis message was deleted.
s
Srikanth Chekuri
02/26/2024, 4:27 PMWhich version of SigNoz deployment is this? Do you see any error logs in query-service?
v
Vibhav Parameswara Chary
02/27/2024, 3:07 AMDont see any error in query service
s
Srikanth Chekuri
02/27/2024, 3:44 AMHow long does it keep spinning? How many services do you have? Would you be able to exec into ClickHouse and run some query?
v
Vibhav Parameswara Chary
02/27/2024, 4:43 AMYeah I can login to clickhouse db
Vibhav Parameswara Chary
02/27/2024, 4:43 AMIt just keeps spinning
Vibhav Parameswara Chary
02/27/2024, 4:46 AM2024-02-26T132253.933Z ERROR clickhouseReader/reader.go:4609 error while reading time series result write: write tcp 10.107.86.25357200 >172.20.141.219000: i/o timeout
2024-02-26T132310.797Z ERROR clickhouseReader/reader.go:4609 error while reading time series result write: write tcp 10.107.86.25357260 >172.20.141.219000: i/o timeout
2024-02-26T132310.801Z ERROR clickhouseReader/reader.go:4609 error while reading time series result write: write tcp 10.107.86.25360758 >172.20.141.219000: i/o timeout
2024-02-26T132310.801Z INFO utils/time.go:12 func GetTimeSeriesResultV3 took 1m0.002184427s with args [SELECT A.
addressaddresststsaddressaddresststststststststsVibhav Parameswara Chary
02/27/2024, 4:47 AMSee some error in query service
s
Srikanth Chekuri
02/27/2024, 6:00 AMIt says i/o timeout. What are the resources given to ClickHouse?
v
Vibhav Parameswara Chary
02/27/2024, 6:03 AMDefault values but it can go up to 5-6 cores
s
Srikanth Chekuri
02/27/2024, 6:04 AMHow much data are you ingesting? And what is the current CPU and memory usage of clickhouse pods?
v
Vibhav Parameswara Chary
02/27/2024, 6:06 AMIst consuming arround 700 cores as of now
Vibhav Parameswara Chary
02/27/2024, 6:07 AMMem is arround 6 gb its using
Vibhav Parameswara Chary
02/27/2024, 6:12 AMhow do we dsiable s3 once its deployed?
Vibhav Parameswara Chary
02/27/2024, 6:12 AMPods go in to crash if I disable s3
Vibhav Parameswara Chary
02/27/2024, 6:12 AMLast optin is to uninistall and reinstall
Vibhav Parameswara Chary
02/27/2024, 6:13 AMDeleting clickhosue tables and running helm fixes issies for some time and again the services goes in to spinning mode
s
Srikanth Chekuri
02/27/2024, 6:15 AMThat's not a way to solve the issue. We first need to understand what's the reason behind this. How much data are you ingesting?
Srikanth Chekuri
02/27/2024, 6:17 AMShare the output of this
Copy code
SELECT
serviceName,
count()
FROM signoz_traces.distributed_top_level_operations
GROUP BY serviceNameSrikanth Chekuri
02/27/2024, 6:25 AMWhat is your table TTL policies?
Copy code
SHOW CREATE TABLE signoz_traces.signoz_index_v2v
Vibhav Parameswara Chary
02/27/2024, 6:41 AMSHOW CREATE TABLE signoz_traces.signoz_index_v2
Query id: 86614b46-690a-4e88-b8ba-827046d9a858
┌─statement──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ CREATE TABLE signoz_traces.signoz_index_v2
(
timestamptraceIDspanIDparentSpanIDserviceNamenamekinddurationNanostatusCodeexternalHttpMethodexternalHttpUrlcomponentdbSystemdbNamedbOperationpeerServiceeventshttpMethodhttpUrlhttpCodehttpRoutehttpHostmsgSystemmsgOperationhasErrortagMapgRPCMethodgRPCCoderpcSystemrpcServicerpcMethodresponseStatusCodestringTagMapnumberTagMapboolTagMapresourceTagsMaps
Srikanth Chekuri
02/27/2024, 6:44 AMRun this and see how long does it take to complete the query
Copy code
SELECT
serviceName,
toStartOfInterval(timestamp, toIntervalSecond(60)) AS ts,
quantile(0.5)(durationNano) AS value
FROM signoz_traces.distributed_signoz_index_v2
WHERE ((timestamp >= '1709013204000000000') AND (timestamp <= '1709015036000000000'))
GROUP BY serviceName, ts
ORDER BY serviceName, tsv
Vibhav Parameswara Chary
02/27/2024, 6:45 AM558 rows in set. Elapsed: 0.025 sec. Processed 59.00 thousand rows, 1.00 MB (2.33 million rows/s., 39.63 MB/s.)
Peak memory usage: 4.12 MiB.
s
Srikanth Chekuri
02/27/2024, 6:46 AMDid you purge clickhouse in the last 1-2 hours?
v
Vibhav Parameswara Chary
02/27/2024, 6:46 AMMay be yesterday
Vibhav Parameswara Chary
02/27/2024, 6:46 AMTrying all steps have purged couple of times
Vibhav Parameswara Chary
02/27/2024, 6:47 AMOnce purged it works for some time and then stops
s
Srikanth Chekuri
02/27/2024, 6:47 AMI don't see anything wrong based on the outputs you shared.
Srikanth Chekuri
02/27/2024, 6:49 AMWhen this happens there might be some frontend error which makes the spinner no go away. Please check if that's the case
v
Vibhav Parameswara Chary
02/27/2024, 6:53 AMrestarted forntend pods, no luck
s
Srikanth Chekuri
02/27/2024, 6:55 AMNo, I meant the javascript error. Do you see the spinner now?
v
Vibhav Parameswara Chary
02/27/2024, 6:55 AMyeah spinner stll ther e
Vibhav Parameswara Chary
02/27/2024, 6:55 AMdifferent browsers is the same
s
Srikanth Chekuri
02/27/2024, 6:56 AMWould you be able to join huddle now?
v
Vibhav Parameswara Chary
02/27/2024, 6:58 AMsure
s
Srikanth Chekuri
02/27/2024, 6:59 AMOpen the network tab and share which requests are these
Srikanth Chekuri
02/27/2024, 7:02 AMThe queries were running quick enough to not timeout. What does your setup look like? Did you make any custom changes to SigNoz deployment? I am also waiting on huddle if you would perfer that way.
v
Vibhav Parameswara Chary
02/27/2024, 7:06 AMNot sure how huddle works, do you have to send an invite?
s
Srikanth Chekuri
02/27/2024, 7:07 AMI did send an invite. You joined for a moment and then left.
Srikanth Chekuri
02/27/2024, 7:26 AM Copy code
SHOW CREATE TABLE signoz_traces.top_level_operations16 Views