https://signoz.io logo
Join Slack
Powered by
This message was deleted.
# support
s
This message was deleted.
v
@nitya-signoz Please check
👍 1
n
please check the logs of query service.
p
@nitya-signoz please find query service logs
Copy code
2024-01-19T11:48:51.027Z        INFO    app/server.go:356       /api/v1/login   timeTaken:491.366µs     {"timeTaken": "492.607µs", "path": "/api/v1/login"}
2024-01-19T11:48:51.027Z        INFO    app/server.go:356       /api/v1/featureFlags    timeTaken:592.998µs     {"timeTaken": "593.678µs", "path": "/api/v1/featureFlags"}
2024-01-19T11:48:51.110Z        INFO    app/server.go:356       /api/v1/user/{id}       timeTaken:622.858µs     {"timeTaken": "624.838µs", "path": "/api/v1/user/{id}"}
2024-01-19T11:48:51.378Z        INFO    app/server.go:356       /api/v2/licenses        timeTaken:476.246µs     {"timeTaken": "477.246µs", "path": "/api/v2/licenses"}
2024-01-19T11:48:51.399Z        INFO    app/server.go:356       /api/v1/version timeTaken:362.904µs     {"timeTaken": "363.804µs", "path": "/api/v1/version"}
2024-01-19T11:48:51.400Z        INFO    app/server.go:356       /api/v1/featureFlags    timeTaken:512.876µs     {"timeTaken": "513.466µs", "path": "/api/v1/featureFlags"}
2024-01-19T11:48:51.608Z        INFO    app/server.go:356       /api/v1/configs timeTaken:209.073391ms  {"timeTaken": "209.074781ms", "path": "/api/v1/configs"}
2024-01-19T11:48:51.681Z        INFO    utils/time.go:12        func GetTimeSeriesResultV3 took 4.252374671s with args [SELECT toStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL 60 SECOND) AS ts, toFloat64(count(*)) as value from signoz_logs.distributed_logs where (timestamp >= 1705578480000000000 AND timestamp <= 1705664880000000000) group by ts order by value DESC]
2024-01-19T11:48:51.681Z        INFO    utils/time.go:12        func GetTimeSeriesResultV3 took 4.251781983s with args [SELECT toStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL 60 SECOND) AS ts, toFloat64(count(*)) as value from signoz_logs.distributed_logs where (timestamp >= 1705578480000000000 AND timestamp <= 1705664880000000000) group by ts order by value DESC]
2024-01-19T11:48:51.682Z        INFO    app/server.go:356       /api/v3/query_range     timeTaken:4.253564786s  {"timeTaken": "4.253565916s", "path": "/api/v3/query_range"}
2024-01-19T11:48:51.682Z        INFO    app/server.go:356       /api/v3/query_range     timeTaken:4.252456151s  {"timeTaken": "4.252456721s", "path": "/api/v3/query_range"}
2024-01-19T11:48:51.683Z        INFO    app/server.go:356       /api/v3/autocomplete/aggregate_attributes       timeTaken:4.356290931s  {"timeTaken": "4.356291611s", "path": "/api/v3/autocomplete/aggregate_attributes"}
2024-01-19T11:48:51.683Z        INFO    app/server.go:356       /api/v3/autocomplete/attribute_keys     timeTaken:4.356520174s  {"timeTaken": "4.356521134s", "path": "/api/v3/autocomplete/attribute_keys"}
2024-01-19T11:48:51.909Z        INFO    app/server.go:356       /api/v1/explorer/views  timeTaken:455.536µs     {"timeTaken": "456.456µs", "path": "/api/v1/explorer/views"}
2024-01-19T11:48:52.384Z        INFO    utils/time.go:12        func GetTimeSeriesResultV3 took 329.623587ms with args [SELECT  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, toFloat64(count(*)) as value FROM signoz_metrics.distributed_samples_v2 INNER JOIN (SELECT  fingerprint FROM signoz_metrics.time_series_v2 WHERE metric_name = '' AND temporality IN ['Cumulative', 'Unspecified']) as filtered_time_series USING fingerprint WHERE metric_name = '' AND timestamp_ms >= 1705578480000 AND timestamp_ms <= 1705664880000 GROUP BY GROUPING SETS ( (ts), () ) ORDER BY  ts]
2024-01-19T11:48:52.384Z        INFO    app/server.go:356       /api/v3/query_range     timeTaken:330.428237ms  {"timeTaken": "330.428757ms", "path": "/api/v3/query_range"}
2024-01-19T11:48:53.681Z        INFO    app/server.go:356       /api/v1/services        timeTaken:22.32162ms    {"timeTaken": "22.32251ms", "path": "/api/v1/services"}
2024-01-19T11:48:55.384Z        INFO    clickhouseReader/reader.go:1268 SELECT durationNano as numTotal FROM signoz_traces.distributed_durationSort WHERE timestamp >= @timestampL AND timestamp <= @timestampU ORDER BY durationNano LIMIT 1
2024-01-19T11:48:55.482Z        INFO    clickhouseReader/reader.go:1280 SELECT durationNano as numTotal FROM signoz_traces.distributed_durationSort WHERE timestamp >= @timestampL AND timestamp <= @timestampU ORDER BY durationNano DESC LIMIT 1
2024-01-19T11:48:55.582Z        INFO    clickhouseReader/reader.go:1217 SELECT COUNT(*) as numTotal FROM signoz_traces.distributed_signoz_index_v2 WHERE timestamp >= @timestampL AND timestamp <= @timestampU AND hasError = true
2024-01-19T11:48:55.684Z        INFO    clickhouseReader/reader.go:1228 SELECT COUNT(*) as numTotal FROM signoz_traces.distributed_signoz_index_v2 WHERE timestamp >= @timestampL AND timestamp <= @timestampU AND hasError = false
2024-01-19T11:48:55.784Z        INFO    clickhouseReader/reader.go:1082 SELECT serviceName, count() as count FROM signoz_traces.distributed_signoz_index_v2 WHERE timestamp >= @timestampL AND timestamp <= @timestampU GROUP BY serviceName
2024-01-19T11:48:55.784Z        INFO    app/server.go:356       /api/v1/getSpanFilters  timeTaken:489.831318ms  {"timeTaken": "489.832668ms", "path": "/api/v1/getSpanFilters"}
2024-01-19T11:48:56.088Z        INFO    clickhouseReader/reader.go:1723
2024-01-19T11:48:56.089Z        INFO    app/server.go:356       /api/v1/getTagFilters   timeTaken:42.579103ms   {"timeTaken": "42.579883ms", "path": "/api/v1/getTagFilters"}
2024-01-19T11:48:56.181Z        INFO    clickhouseReader/reader.go:1483 SELECT timestamp, spanID, traceID, serviceName, name, durationNano, httpMethod, rpcMethod, responseStatusCode FROM signoz_traces.distributed_signoz_index_v2 WHERE timestamp >= @timestampL AND timestamp <= @timestampU LIMIT @limit
2024-01-19T11:48:56.182Z        INFO    app/server.go:356       /api/v1/getFilteredSpans        timeTaken:135.557594ms  {"timeTaken": "135.558534ms", "path": "/api/v1/getFilteredSpans"}
2024-01-19T11:48:56.185Z        INFO    clickhouseReader/reader.go:2201 SELECT toStartOfInterval(timestamp, INTERVAL 1 minute) as time,  count(*) as value  FROM signoz_traces.distributed_signoz_index_v2 WHERE timestamp >= @timestampL AND timestamp <= @timestampU GROUP BY time ORDER BY time
2024-01-19T11:48:56.186Z        INFO    app/server.go:356       /api/v1/getFilteredSpans/aggregates     timeTaken:90.859796ms   {"timeTaken": "91.084789ms", "path": "/api/v1/getFilteredSpans/aggregates"}
2024-01-19T11:48:57.119Z        INFO    app/server.go:356       /api/v1/explorer/views  timeTaken:566.698µs     {"timeTaken": "567.958µs", "path": "/api/v1/explorer/views"}
2024-01-19T11:48:57.354Z        INFO    app/server.go:356       /api/v3/autocomplete/aggregate_attributes       timeTaken:411.365µs     {"timeTaken": "412.205µs", "path": "/api/v3/autocomplete/aggregate_attributes"}
2024-01-19T11:48:57.888Z        INFO    utils/time.go:12        func GetListResultV3 took 308.97812ms with args [SELECT  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, toFloat64(count(*)) as value FROM signoz_metrics.distributed_samples_v2 INNER JOIN (SELECT  fingerprint FROM signoz_metrics.time_series_v2 WHERE metric_name = '' AND temporality IN ['Cumulative', 'Unspecified']) as filtered_time_series USING fingerprint WHERE metric_name = '' AND timestamp_ms >= 1705578480000 AND timestamp_ms <= 1705664880000 GROUP BY GROUPING SETS ( (ts), () ) ORDER BY  ts]
2024-01-19T11:48:57.889Z        INFO    app/server.go:356       /api/v3/query_range     timeTaken:309.79388ms   {"timeTaken": "309.79445ms", "path": "/api/v3/query_range"}
2024-01-19T11:48:59.790Z        INFO    app/server.go:356       /api/v1/health  timeTaken:23.87µs       {"timeTaken": "24.7µs", "path": "/api/v1/health"}
2024-01-19T11:48:59.884Z        INFO    app/server.go:356       /api/v1/health  timeTaken:93.811882ms   {"timeTaken": "93.812742ms", "path": "/api/v1/health"}
2024-01-19T11:49:09.790Z        INFO    app/server.go:356       /api/v1/health  timeTaken:29.021µs      {"timeTaken": "29.811µs", "path": "/api/v1/health"}
2024-01-19T11:49:09.887Z        INFO    app/server.go:356       /api/v1/health  timeTaken:96.629398ms   {"timeTaken": "96.630268ms", "path": "/api/v1/health"}
2024-01-19T11:49:11.684Z        INFO    app/server.go:356       /api/v3/autocomplete/attribute_keys     timeTaken:19.784005813s {"timeTaken": "19.784007683s", "path": "/api/v3/autocomplete/attribute_keys"}
2024-01-19T11:49:14.616Z        DEBUG   server/serverimpl.go:208        Agent disconnected: websocket: close 1006 (abnormal closure): unexpected EOF
2024-01-19T11:49:14.717Z        DEBUG   server/serverimpl.go:208        Agent disconnected: websocket: close 1006 (abnormal closure): unexpected EOF
2024-01-19T11:49:14.890Z        DEBUG   opamp/opamp_server.go:94        New agent added:%!(EXTRA zapcore.Field={canLb 4 1  <nil>}, zapcore.Field={ID 15 0 01HMGRVGFT06AFVGQT5ZYC0GEK <nil>}, zapcore.Field={status 23 0  1})
2024-01-19T11:49:15.557Z        DEBUG   opamp/opamp_server.go:94        New agent added:%!(EXTRA zapcore.Field={canLb 4 1  <nil>}, zapcore.Field={ID 15 0 01HMGRVH4Q06AFVGQT5ZYC0GEK <nil>}, zapcore.Field={status 23 0  1})
2024-01-19T11:49:18.689Z        INFO    app/server.go:356       /api/v3/autocomplete/attribute_keys     timeTaken:20.794118173s {"timeTaken": "20.794119833s", "path": "/api/v3/autocomplete/attribute_keys"}
2024-01-19T11:49:19.790Z        INFO    app/server.go:356       /api/v1/health  timeTaken:45.761µs      {"timeTaken": "46.551µs", "path": "/api/v1/health"}
2024-01-19T11:49:19.881Z        INFO    app/server.go:356       /api/v1/health  timeTaken:91.132932ms   {"timeTaken": "91.133902ms", "path": "/api/v1/health"}
2024-01-19T11:49:20.881Z        INFO    utils/time.go:12        func GetListResultV3 took 9.187391745s with args [SELECT timestamp, id, trace_id, span_id, trace_flags, severity_text, severity_number, body,CAST((attributes_string_key, attributes_string_value), 'Map(String, String)') as  attributes_string,CAST((attributes_int64_key, attributes_int64_value), 'Map(String, Int64)') as  attributes_int64,CAST((attributes_float64_key, attributes_float64_value), 'Map(String, Float64)') as  attributes_float64,CAST((resources_string_key, resources_string_value), 'Map(String, String)') as resources_string from signoz_logs.distributed_logs where (timestamp >= 1705578480000000000 AND timestamp <= 1705664880000000000) order by timestamp desc LIMIT 100 OFFSET 0]
2024-01-19T11:49:20.886Z        INFO    app/server.go:356       /api/v3/query_range     timeTaken:9.192488526s  {"timeTaken": "9.192489536s", "path": "/api/v3/query_range"}
2024-01-19T11:49:24.687Z        INFO    app/server.go:356       /api/v3/autocomplete/aggregate_attributes       timeTaken:32.781532727s {"timeTaken": "32.781535227s", "path": "/api/v3/autocomplete/aggregate_attributes"}
2024-01-19T11:49:29.791Z        INFO    app/server.go:356       /api/v1/health  timeTaken:23.97µs       {"timeTaken": "25.36µs", "path": "/api/v1/health"}
2024-01-19T11:49:29.793Z        INFO    app/server.go:356       /api/v1/health  timeTaken:2.331849ms    {"timeTaken": "2.332309ms", "path": "/api/v1/health"}
2024-01-19T11:49:39.790Z        INFO    app/server.go:356       /api/v1/health  timeTaken:18.901µs      {"timeTaken": "19.511µs", "path": "/api/v1/health"}
2024-01-19T11:49:39.792Z        INFO    app/server.go:356       /api/v1/health  timeTaken:2.733993ms    {"timeTaken": "2.734633ms", "path": "/api/v1/health"}
2024-01-19T11:49:49.790Z        INFO    app/server.go:356       /api/v1/health  timeTaken:24.99µs       {"timeTaken": "25.89µs", "path": "/api/v1/health"}
2024-01-19T11:49:49.792Z        INFO    app/server.go:356       /api/v1/health  timeTaken:2.229096ms    {"timeTaken": "2.229806ms", "path": "/api/v1/health"}
clickhouse logs are below:
Copy code
2024.01.19 11:49:52.182578 [ 346 ] {688920d7-0283-4c98-acc2-5124b9af394d} <Information> TCPHandler: Change cancellation status from NOT_CANCELLED to FULLY_CANCELLED. Log message: Query was cancelled.
2024.01.19 11:49:52.188525 [ 346 ] {688920d7-0283-4c98-acc2-5124b9af394d} <Error> executeQuery: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:192.168.146.242]:9000 -> [::ffff:192.168.100.250]:43550). (NETWORK_ERROR) (version 23.7.3.14 (official build)) (from [::ffff:192.168.100.250]:43550) (in query: SELECT toStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL 60 SECOND) AS ts, toFloat64(count(*)) as value from signoz_logs.distributed_logs where (timestamp >= 1705578480000000000 AND timestamp <= 1705664880000000000) group by ts order by value DESC), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000e91fbb7 in /usr/bin/clickhouse
1. ? @ 0x000000000eb98c70 in /usr/bin/clickhouse
2. DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000eb987d1 in /usr/bin/clickhouse
3. DB::TCPHandler::sendData(DB::Block const&) @ 0x0000000015284bba in /usr/bin/clickhouse
4. DB::TCPHandler::processOrdinaryQueryWithProcessors() @ 0x00000000152828c6 in /usr/bin/clickhouse
5. DB::TCPHandler::runImpl() @ 0x0000000015278d10 in /usr/bin/clickhouse
6. DB::TCPHandler::run() @ 0x000000001528f7f9 in /usr/bin/clickhouse
7. Poco::Net::TCPServerConnection::start() @ 0x0000000018293df4 in /usr/bin/clickhouse
8. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018295011 in /usr/bin/clickhouse
9. Poco::PooledThread::run() @ 0x000000001841e227 in /usr/bin/clickhouse
10. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001841bc5c in /usr/bin/clickhouse
11. ? @ 0x00007f2082918609 in ?
12. __clone @ 0x00007f208283d133 in ?

2024.01.19 11:49:52.280690 [ 346 ] {688920d7-0283-4c98-acc2-5124b9af394d} <Error> TCPHandler: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:192.168.146.242]:9000 -> [::ffff:192.168.100.250]:43550). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000e91fbb7 in /usr/bin/clickhouse
1. ? @ 0x000000000eb98c70 in /usr/bin/clickhouse
2. DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000eb987d1 in /usr/bin/clickhouse
3. DB::TCPHandler::sendData(DB::Block const&) @ 0x0000000015284bba in /usr/bin/clickhouse
4. DB::TCPHandler::processOrdinaryQueryWithProcessors() @ 0x00000000152828c6 in /usr/bin/clickhouse
5. DB::TCPHandler::runImpl() @ 0x0000000015278d10 in /usr/bin/clickhouse
6. DB::TCPHandler::run() @ 0x000000001528f7f9 in /usr/bin/clickhouse
7. Poco::Net::TCPServerConnection::start() @ 0x0000000018293df4 in /usr/bin/clickhouse
8. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018295011 in /usr/bin/clickhouse
9. Poco::PooledThread::run() @ 0x000000001841e227 in /usr/bin/clickhouse
10. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001841bc5c in /usr/bin/clickhouse
11. ? @ 0x00007f2082918609 in ?
12. __clone @ 0x00007f208283d133 in ?

2024.01.19 11:49:52.280992 [ 346 ] {688920d7-0283-4c98-acc2-5124b9af394d} <Warning> TCPHandler: Client has gone away.
2024.01.19 11:49:52.383314 [ 9 ] {cd8cc84d-33e3-4929-be55-48d6c4ee549b} <Information> TCPHandler: Change cancellation status from NOT_CANCELLED to FULLY_CANCELLED. Log message: Query was cancelled.
2024.01.19 11:49:52.385134 [ 9 ] {cd8cc84d-33e3-4929-be55-48d6c4ee549b} <Error> executeQuery: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:192.168.146.242]:9000 -> [::ffff:192.168.100.250]:43540). (NETWORK_ERROR) (version 23.7.3.14 (official build)) (from [::ffff:192.168.100.250]:43540) (in query: SELECT toStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL 60 SECOND) AS ts, toFloat64(count(*)) as value from signoz_logs.distributed_logs where (timestamp >= 1705578480000000000 AND timestamp <= 1705664880000000000) group by ts order by value DESC), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000e91fbb7 in /usr/bin/clickhouse
1. ? @ 0x000000000eb98c70 in /usr/bin/clickhouse
2. DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000eb987d1 in /usr/bin/clickhouse
3. DB::TCPHandler::sendData(DB::Block const&) @ 0x0000000015284bba in /usr/bin/clickhouse
4. DB::TCPHandler::processOrdinaryQueryWithProcessors() @ 0x00000000152828c6 in /usr/bin/clickhouse
5. DB::TCPHandler::runImpl() @ 0x0000000015278d10 in /usr/bin/clickhouse
6. DB::TCPHandler::run() @ 0x000000001528f7f9 in /usr/bin/clickhouse
7. Poco::Net::TCPServerConnection::start() @ 0x0000000018293df4 in /usr/bin/clickhouse
8. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018295011 in /usr/bin/clickhouse
9. Poco::PooledThread::run() @ 0x000000001841e227 in /usr/bin/clickhouse
10. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001841bc5c in /usr/bin/clickhouse
11. ? @ 0x00007f2082918609 in ?
12. __clone @ 0x00007f208283d133 in ?

2024.01.19 11:49:52.386632 [ 9 ] {cd8cc84d-33e3-4929-be55-48d6c4ee549b} <Error> TCPHandler: Code: 210. DB::NetException: Connection reset by peer, while writing to socket ([::ffff:192.168.146.242]:9000 -> [::ffff:192.168.100.250]:43540). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000e91fbb7 in /usr/bin/clickhouse
1. ? @ 0x000000000eb98c70 in /usr/bin/clickhouse
2. DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000eb987d1 in /usr/bin/clickhouse
3. DB::TCPHandler::sendData(DB::Block const&) @ 0x0000000015284bba in /usr/bin/clickhouse
4. DB::TCPHandler::processOrdinaryQueryWithProcessors() @ 0x00000000152828c6 in /usr/bin/clickhouse
5. DB::TCPHandler::runImpl() @ 0x0000000015278d10 in /usr/bin/clickhouse
6. DB::TCPHandler::run() @ 0x000000001528f7f9 in /usr/bin/clickhouse
7. Poco::Net::TCPServerConnection::start() @ 0x0000000018293df4 in /usr/bin/clickhouse
8. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018295011 in /usr/bin/clickhouse
9. Poco::PooledThread::run() @ 0x000000001841e227 in /usr/bin/clickhouse
10. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001841bc5c in /usr/bin/clickhouse
11. ? @ 0x00007f2082918609 in ?
12. __clone @ 0x00007f208283d133 in ?

2024.01.19 11:49:52.386896 [ 9 ] {cd8cc84d-33e3-4929-be55-48d6c4ee549b} <Warning> TCPHandler: Client has gone away.
2024.01.19 11:49:53.507516 [ 88 ] {} <Information> DNSCacheUpdater: IPs of some hosts have been changed. Will reload cluster config.
2024.01.19 11:49:57.695538 [ 465 ] {4b498d2a-805a-4744-8a46-d3bdc87fc3e4} <Information> TCPHandler: Change cancellation status from NOT_CANCELLED to FULLY_CANCELLED. Log message: Query was cancelled.
2024.01.19 11:49:57.696629 [ 465 ] {4b498d2a-805a-4744-8a46-d3bdc87fc3e4} <Information> executeQuery: Code: 394. DB::Exception: Query was cancelled or a client has unexpectedly dropped the connection. (QUERY_WAS_CANCELLED) (version 23.7.3.14 (official build)) (from [::ffff:192.168.100.250]:43566) (in query: SELECT toStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL 60 SECOND) AS ts, toFloat64(count(*)) as value from signoz_logs.distributed_logs where (timestamp >= 1705578480000000000 AND timestamp <= 1705664880000000000) group by ts order by value DESC)
2024.01.19 11:49:57.702088 [ 465 ] {4b498d2a-805a-4744-8a46-d3bdc87fc3e4} <Error> executeQuery: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:192.168.146.242]:9000 -> [::ffff:192.168.100.250]:43566). (NETWORK_ERROR) (version 23.7.3.14 (official build)) (from [::ffff:192.168.100.250]:43566) (in query: SELECT toStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL 60 SECOND) AS ts, toFloat64(count(*)) as value from signoz_logs.distributed_logs where (timestamp >= 1705578480000000000 AND timestamp <= 1705664880000000000) group by ts order by value DESC), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000e91fbb7 in /usr/bin/clickhouse
1. ? @ 0x000000000eb98c70 in /usr/bin/clickhouse
2. DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000eb9897b in /usr/bin/clickhouse
3. DB::TCPHandler::sendEndOfStream() @ 0x0000000015283747 in /usr/bin/clickhouse
4. DB::TCPHandler::runImpl() @ 0x00000000152795ae in /usr/bin/clickhouse
5. DB::TCPHandler::run() @ 0x000000001528f7f9 in /usr/bin/clickhouse
6. Poco::Net::TCPServerConnection::start() @ 0x0000000018293df4 in /usr/bin/clickhouse
7. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018295011 in /usr/bin/clickhouse
8. Poco::PooledThread::run() @ 0x000000001841e227 in /usr/bin/clickhouse
9. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001841bc5c in /usr/bin/clickhouse
10. ? @ 0x00007f2082918609 in ?
11. __clone @ 0x00007f208283d133 in ?

2024.01.19 11:49:57.702194 [ 465 ] {4b498d2a-805a-4744-8a46-d3bdc87fc3e4} <Error> TCPHandler: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:192.168.146.242]:9000 -> [::ffff:192.168.100.250]:43566). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000e91fbb7 in /usr/bin/clickhouse
1. ? @ 0x000000000eb98c70 in /usr/bin/clickhouse
2. DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000eb9897b in /usr/bin/clickhouse
3. DB::TCPHandler::sendEndOfStream() @ 0x0000000015283747 in /usr/bin/clickhouse
4. DB::TCPHandler::runImpl() @ 0x00000000152795ae in /usr/bin/clickhouse
5. DB::TCPHandler::run() @ 0x000000001528f7f9 in /usr/bin/clickhouse
6. Poco::Net::TCPServerConnection::start() @ 0x0000000018293df4 in /usr/bin/clickhouse
7. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018295011 in /usr/bin/clickhouse
8. Poco::PooledThread::run() @ 0x000000001841e227 in /usr/bin/clickhouse
9. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001841bc5c in /usr/bin/clickhouse
10. ? @ 0x00007f2082918609 in ?
11. __clone @ 0x00007f208283d133 in ?

2024.01.19 11:49:57.702335 [ 465 ] {4b498d2a-805a-4744-8a46-d3bdc87fc3e4} <Warning> TCPHandler: Client has gone away.
2024.01.19 11:50:08.528891 [ 88 ] {} <Information> DNSCacheUpdater: IPs of some hosts have been changed. Will reload cluster
just an obervation whenever this happend OTEL COLLECTOR Is getting restarted
@nitya-signoz could you please take a look at this
n
Are you noticing any issues while using the product ? These erros mostly happens when the clients cancels the query. This happens when you move bettween pages in signoz and before eveything is loaded you change pages.
p
i am trying to get the logs from log explorer when i try to search for last one week , i am getting this error
n
This means your query is taking log time and is getting cancelled, to verify can you check the network tab ? • You can do two things here extract attributes and index them for faster query results. ◦ https://signoz.io/docs/logs-pipelines/introduction/ ◦ https://signoz.io/docs/userguide/logs_fields/#selected-log-fields • Incase that doesn’t help you will have to create projection, https://signoz-community.slack.com/archives/C01HWQ1R0BC/p1704256189374369?thread_ts=1703758888.438039&amp;cid=C01HWQ1R0BC Extracting and indexing does the job in most of the cases.
p
I did the selected log fields with old explorer the query changed to below
Copy code
SELECT
    toStartOfInterval(fromUnixTimestamp64Nano(timestamp), toIntervalSecond(1980)) AS ts,
    toFloat64(count(*)) AS value
FROM signoz_logs.distributed_logs
WHERE ((timestamp >= 1705562100000000000) AND (timestamp <= 1706167980000000000)) AND (resource_string_k8s_namespace_name IN ['namespace'])
GROUP BY ts
ORDER BY value DESC
The query took more time
Copy code
┌──────────────────ts─┬─value─┐
│ 2024-01-24 10:06:00 │  1863 │
│ 2024-01-24 06:48:00 │   891 │
│ 2024-01-24 06:15:00 │   891 │
│ 2024-01-24 09:33:00 │   608 │
│ 2024-01-24 07:21:00 │   542 │
│ 2024-01-24 05:42:00 │    54 │
│ 2024-01-24 10:39:00 │    18 │
└─────────────────────┴───────┘
7 rows in set. Elapsed: 473.958 sec. Processed 514.34 million rows, 285.94 GB (1.09 million rows/s., 603.30 MB/s.) it took almost 8 minutes do we need to go with projections to have better perf
n
interesting, you try to get explain of the query just add 
explain indexes=1
brefore the query and run it on clickhouse. this will give you some idea about how much the skip index was helpful and then you can create a projection to see if something gets better.
1212 Views
Open in Slack
PreviousNext
Powered by