Hi team. We met this error when try to query a longer time 1 day logs. Not sure what’s the best way to solve it

{
  "status": "error",
  "data": {},
  "errorType": "bad_data",
  "error": "encountered multiple errors: error in query-A: context deadline exceeded"
}

Can you share the query that you are trying to run, also how much data are you ingesting any idea ?

the query is just a simple log query from 1 day range. The query stopped at exactly 1 min which seems like a timeout cutoff somewhere

Yeah you can add these env’s as well for query service

ClickHouseMaxExecutionTimeLeaf: "60"
    ClickHouseTimeoutBeforeCheckingExecutionSpeed: "5"
    ClickHouseMaxBytesToRead: "200000000000"

On a side note, we are working on a new schema release for logs which will boost perf in general, it will be released soon you can track it here https://github.com/SigNoz/signoz/issues/5555

But I still got the same error and here is the error log from Clickhouse:

clickhouse 2024.08.23 05:31:38.849771 [ 3115 ] {1c56cf6f-745e-47db-ba6b-2d57ee6e0708} <Error> executeQuery: Code: 210. DB::NetException: Connection reset by peer, whi │
│ le writing to socket ([::ffff:10.2.61.164]:9000 -> [::ffff:10.2.63.230]:48494). (NETWORK_ERROR) (version 24.1.2.5 (official build)) (from [::ffff:10.2.63.230]:48494)  │
│ (comment: {"alertID":"","client":"browser","dashboardID":"","path":"/logs/logs-explorer","servicesTab":"","source":"logs-explorer","viewName":"\"order-srv-prod\""}) ( │
│ in query: SELECT timestamp, id, trace_id, span_id, trace_flags, severity_text, severity_number, body,CAST((attributes_string_key, attributes_string_value), 'Map(Strin │
│ g, String)') as attributes_string,CAST((attributes_int64_key, attributes_int64_value), 'Map(String, Int64)') as attributes_int64,CAST((attributes_float64_key, attribu │
│ tes_float64_value), 'Map(String, Float64)') as attributes_float64,CAST((attributes_bool_key, attributes_bool_value), 'Map(String, Bool)') as attributes_bool,CAST((res │
│ ources_string_key, resources_string_value), 'Map(String, String)') as resources_string from signoz_logs.distributed_logs where (timestamp >= 1724304636000000000 AND t │
│ imestamp <= 1724391036000000000) AND attributes_string_value[indexOf(attributes_string_key, '@message')] = 'Invalid VIN' AND `resource_string_k8s$$namespace$$name` =  │
│ 'signup-service' AND resources_string_value[indexOf(resources_string_key, 'k8s.cluster.name')] = 'prodCluster' order by timestamp desc LIMIT 100), Stack trace (when c │
│ opying this message, always include the lines below):                                                                                                                  │
│ clickhouse                                                                                                                                                             │
│ clickhouse 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in /usr/bin/clickhouse                                          │
│ clickhouse 1. DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, s │
│ td::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000caa69a1 in /usr/bin/clickhouse                                                            │
│ clickhouse 2. DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000caa7192 in /usr/bin/clickhouse                                                                    │
│ clickhouse 3. DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001292ca1a in /usr/bin/clickhouse                                                                   │
│ clickhouse 4. DB::TCPHandler::runImpl() @ 0x000000001291eab2 in /usr/bin/clickhouse                                                                                    │
│ clickhouse 5. DB::TCPHandler::run() @ 0x0000000012933eb9 in /usr/bin/clickhouse                                                                                        │
│ clickhouse 6. Poco::Net::TCPServerConnection::start() @ 0x00000000153a5a72 in /usr/bin/clickhouse                                                                      │
│ clickhouse 7. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153a6871 in /usr/bin/clickhouse                                                                        │
│ clickhouse 8. Poco::PooledThread::run() @ 0x000000001549f047 in /usr/bin/clickhouse                                                                                    │
│ clickhouse 9. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001549d67d in /usr/bin/clickhouse                                                                       │
│ clickhouse 10. ? @ 0x00007f730701e609                                                                                                                                  │
│ clickhouse 11. ? @ 0x00007f7306f43353                                                                                                                                  │
│ clickhouse                                                                                                                                                             │
│ clickhouse 2024.08.23 05:31:38.856358 [ 3115 ] {1c56cf6f-745e-47db-ba6b-2d57ee6e0708} <Error> TCPHandler: Code: 210. DB::NetException: Connection reset by peer, while │
│  writing to socket ([::ffff:10.2.61.164]:9000 -> [::ffff:10.2.63.230]:48494). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below) │
│ :
clickhouse                                                                                                                                                             │
│ clickhouse 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in /usr/bin/clickhouse                                          │
│ clickhouse 1. DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, s │
│ td::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000caa69a1 in /usr/bin/clickhouse                                                            │
│ clickhouse 2. DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000caa7192 in /usr/bin/clickhouse                                                                    │
│ clickhouse 3. DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001292ca1a in /usr/bin/clickhouse                                                                   │
│ clickhouse 4. DB::TCPHandler::runImpl() @ 0x000000001291eab2 in /usr/bin/clickhouse                                                                                    │
│ clickhouse 5. DB::TCPHandler::run() @ 0x0000000012933eb9 in /usr/bin/clickhouse                                                                                        │
│ clickhouse 6. Poco::Net::TCPServerConnection::start() @ 0x00000000153a5a72 in /usr/bin/clickhouse                                                                      │
│ clickhouse 7. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153a6871 in /usr/bin/clickhouse                                                                        │
│ clickhouse 8. Poco::PooledThread::run() @ 0x000000001549f047 in /usr/bin/clickhouse                                                                                    │
│ clickhouse 9. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001549d67d in /usr/bin/clickhouse                                                                       │
│ clickhouse 10. ? @ 0x00007f730701e609                                                                                                                                  │
│ clickhouse 11. ? @ 0x00007f7306f43353                                                                                                                                  │
│ clickhouse