This message was deleted.
# supports
Slackbot
01/31/2024, 3:39 PMThis message was deleted.
s
Srikanth Chekuri
02/02/2024, 1:14 AMDid anything else change from your side? Like change in TTL or something? There is nothing specific in 0.38 that should suddenly introduce high cpu.
f
Fernando Pimenta
02/06/2024, 7:53 PMThe same is happening here, after update to v.038. More than 100% CPU for clickhouse-serv
u
Ulisses Silva
02/15/2024, 1:57 PMNo. Just did the upgrade.
I have wiped all data from signoz, except the sqlite database, and started again.
It runs ok for some days, then it starts to throw a lot of errors like bellow. I need to restart the containers every time this happens.
signoz-clickhouse | 2024.02.15 131748.789532 [ 124850 ] {5e32ef89-8bf6-4855-9da8-2d4ea027ed26} <Error> TCPHandler: Code: 210. DB:NetException I/O error: Broken pipe, while writing to socket (192.168.48.3:9000 -> 192.168.48.5:60236). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):
signoz-clickhouse |
signoz-clickhouse | 0. DB:Exception:Exception(DB:Exception:MessageMasked&&, int, bool) @ 0x000000000c4fd017 in /usr/bin/clickhouse
signoz-clickhouse | 1. DB:NetException:NetException<String, String, String>(int, FormatStringHelperImpl<std:type identity<String>:type, std:type identity<String>:type, std:type identity<String>:type>, String&&, String&&, String&&) @ 0x000000000c793291 in /usr/bin/clickhouse
signoz-clickhouse | 2. DB:WriteBufferFromPocoSocket:nextImpl() @ 0x000000000c792f96 in /usr/bin/clickhouse
signoz-clickhouse | 3. DB:TCPHandler:runImpl() @ 0x000000001223752f in /usr/bin/clickhouse
signoz-clickhouse | 4. DB:TCPHandler:run() @ 0x0000000012246a79 in /usr/bin/clickhouse
signoz-clickhouse | 5. Poco:NetTCPServerConnection:start() @ 0x0000000014c6fc52 in /usr/bin/clickhouse
signoz-clickhouse | 6. Poco:NetTCPServerDispatcher:run() @ 0x0000000014c70a51 in /usr/bin/clickhouse
signoz-clickhouse | 7. Poco:PooledThread:run() @ 0x0000000014d678e7 in /usr/bin/clickhouse
signoz-clickhouse | 8. Poco:ThreadImpl:runnableEntry(void*) @ 0x0000000014d65edc in /usr/bin/clickhouse
signoz-clickhouse | 9. ? @ 0x00007f67951a9609 in ?
signoz-clickhouse | 10. ? @ 0x00007f67950ce133 in ?
signoz-clickhouse |
signoz-clickhouse | 2024.02.15 131748.789675 [ 124850 ] {5e32ef89-8bf6-4855-9da8-2d4ea027ed26} <Warning> TCPHandler: Client has gone away.
Ulisses Silva
02/15/2024, 3:43 PMI just updated to 0.39. Signoz is working but now there is a constant error (every 1~2 minutes) in the clickhouse container log:
signoz-clickhouse | 2024.02.15 154049.633749 [ 835 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 32, I/O error: Broken pipe, Stack trace (when copying this message, always include the lines below):
signoz-clickhouse |
signoz-clickhouse | 0. Poco:NetSocketImpl:error(int, String const&) @ 0x00000000153a1b5f in /usr/bin/clickhouse
signoz-clickhouse | 1. Poco:NetSocketImpl:sendBytes(void const*, int, int) @ 0x00000000153a2bbd in /usr/bin/clickhouse
signoz-clickhouse | 2. Poco:NetStreamSocketImpl:sendBytes(void const*, int, int) @ 0x00000000153a5296 in /usr/bin/clickhouse
signoz-clickhouse | 3. Poco:NetHTTPSession:write(char const*, long) @ 0x00000000153908b3 in /usr/bin/clickhouse
signoz-clickhouse | 4. Poco:NetHTTPHeaderIOS:~HTTPHeaderIOS() @ 0x000000001538bbdb in /usr/bin/clickhouse
signoz-clickhouse | 5. Poco:NetHTTPHeaderOutputStream:~HTTPHeaderOutputStream() @ 0x000000001538bf1f in /usr/bin/clickhouse
signoz-clickhouse | 6. DB:HTTPServerResponse:send() @ 0x0000000012942988 in /usr/bin/clickhouse
signoz-clickhouse | 7. DB:HTTPServerConnection:sendErrorResponse(Poco:Net:HTTPServerSession&, Poco:NetHTTPResponse:HTTPStatus) @ 0x000000001293ecda in /usr/bin/clickhouse
signoz-clickhouse | 8. DB:HTTPServerConnection:run() @ 0x000000001293e97b in /usr/bin/clickhouse
signoz-clickhouse | 9. Poco:NetTCPServerConnection:start() @ 0x00000000153a5a72 in /usr/bin/clickhouse
signoz-clickhouse | 10. Poco:NetTCPServerDispatcher:run() @ 0x00000000153a6871 in /usr/bin/clickhouse
signoz-clickhouse | 11. Poco:PooledThread:run() @ 0x000000001549f047 in /usr/bin/clickhouse
signoz-clickhouse | 12. Poco:ThreadImpl:runnableEntry(void*) @ 0x000000001549d67d in /usr/bin/clickhouse
signoz-clickhouse | 13. ? @ 0x00007f3668baf609
signoz-clickhouse | 14. ? @ 0x00007f3668ad4353
signoz-clickhouse | (version 24.1.2.5 (official build))
s
Srikanth Chekuri
02/15/2024, 4:17 PMThe broken pipe error usually happens when there is resource usage is high specifically CPU. What is the CPU usage when you see these errors?
u
Ulisses Silva
02/15/2024, 4:51 PM100~200%
32 Views