Hey guys - we've been running our signoz from kubernetes for more than a month now - suddenly there's a lot of crashes happening on the clickhouse database, below is the error I can see. Is this something other are aware of?

2024.06.10 05:51:17.534245 [ 762 ] {8ab7b00c-87c2-4cd8-b411-3bdd53124550} <Warning> TCPHandler: Client has gone away.
2024.06.10 05:51:21.316932 [ 763 ] {d941051c-6eaa-4fc7-ac07-3fb879023a05} <Error> executeQuery: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::ffff:172.31.4.207]:9000 -> [::ffff:172.31.5.26]:49416). (NETWORK_ERROR) (version 24.1.2.5 (official build)) (from [::ffff:172.31.5.26]:49416) (in query: INSERT INTO signoz_metrics.distributed_time_series_v4 (env, temporality, metric_name, description, unit, type, is_monotonic, fingerprint, unix_milli, labels) VALUES), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in /usr/bin/clickhouse
1. DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000caa69a1 in /usr/bin/clickhouse
2. DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000caa733e in /usr/bin/clickhouse
3. DB::TCPHandler::runImpl() @ 0x000000001292120f in /usr/bin/clickhouse
4. DB::TCPHandler::run() @ 0x0000000012933eb9 in /usr/bin/clickhouse
5. Poco::Net::TCPServerConnection::start() @ 0x00000000153a5a72 in /usr/bin/clickhouse
6. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153a6871 in /usr/bin/clickhouse
7. Poco::PooledThread::run() @ 0x000000001549f047 in /usr/bin/clickhouse
8. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001549d67d in /usr/bin/clickhouse
9. ? @ 0x00007f46a3ff4609
10. ? @ 0x00007f46a3f19353

This could be due to various reasons such as network issues, client-side errors, or deliberate disconnection.