https://signoz.io
Join Slack
This message was deleted.
# support
s
This message was deleted.
s
Did anything else change from your side? Like change in TTL or something? There is nothing specific in 0.38 that should suddenly introduce high cpu.
f
The same is happening here, after update to v.038. More than 100% CPU for clickhouse-serv
u
No. Just did the upgrade. I have wiped all data from signoz, except the sqlite database, and started again. It runs ok for some days, then it starts to throw a lot of errors like bellow. I need to restart the containers every time this happens. signoz-clickhouse | 2024.02.15 131748.789532 [ 124850 ] {5e32ef89-8bf6-4855-9da8-2d4ea027ed26} <Error> TCPHandler: Code: 210. DB:NetException I/O error: Broken pipe, while writing to socket (192.168.48.3:9000 -> 192.168.48.5:60236). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below): signoz-clickhouse | signoz-clickhouse | 0. DB:Exception:Exception(DB:Exception:MessageMasked&&, int, bool) @ 0x000000000c4fd017 in /usr/bin/clickhouse signoz-clickhouse | 1. DB:NetException:NetException<String, String, String>(int, FormatStringHelperImpl<std:type identity&lt;String&gt;:type, std:type identity&lt;String&gt;:type, std:type identity&lt;String&gt;:type>, String&&, String&&, String&&) @ 0x000000000c793291 in /usr/bin/clickhouse signoz-clickhouse | 2. DB:WriteBufferFromPocoSocket:nextImpl() @ 0x000000000c792f96 in /usr/bin/clickhouse signoz-clickhouse | 3. DB:TCPHandler:runImpl() @ 0x000000001223752f in /usr/bin/clickhouse signoz-clickhouse | 4. DB:TCPHandler:run() @ 0x0000000012246a79 in /usr/bin/clickhouse signoz-clickhouse | 5. Poco:NetTCPServerConnection:start() @ 0x0000000014c6fc52 in /usr/bin/clickhouse signoz-clickhouse | 6. Poco:NetTCPServerDispatcher:run() @ 0x0000000014c70a51 in /usr/bin/clickhouse signoz-clickhouse | 7. Poco:PooledThread:run() @ 0x0000000014d678e7 in /usr/bin/clickhouse signoz-clickhouse | 8. Poco:ThreadImpl:runnableEntry(void*) @ 0x0000000014d65edc in /usr/bin/clickhouse signoz-clickhouse | 9. ? @ 0x00007f67951a9609 in ? signoz-clickhouse | 10. ? @ 0x00007f67950ce133 in ? signoz-clickhouse | signoz-clickhouse | 2024.02.15 131748.789675 [ 124850 ] {5e32ef89-8bf6-4855-9da8-2d4ea027ed26} <Warning> TCPHandler: Client has gone away.
I just updated to 0.39. Signoz is working but now there is a constant error (every 1~2 minutes) in the clickhouse container log: signoz-clickhouse | 2024.02.15 154049.633749 [ 835 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 32, I/O error: Broken pipe, Stack trace (when copying this message, always include the lines below): signoz-clickhouse | signoz-clickhouse | 0. Poco:NetSocketImpl:error(int, String const&) @ 0x00000000153a1b5f in /usr/bin/clickhouse signoz-clickhouse | 1. Poco:NetSocketImpl:sendBytes(void const*, int, int) @ 0x00000000153a2bbd in /usr/bin/clickhouse signoz-clickhouse | 2. Poco:NetStreamSocketImpl:sendBytes(void const*, int, int) @ 0x00000000153a5296 in /usr/bin/clickhouse signoz-clickhouse | 3. Poco:NetHTTPSession:write(char const*, long) @ 0x00000000153908b3 in /usr/bin/clickhouse signoz-clickhouse | 4. Poco:NetHTTPHeaderIOS:~HTTPHeaderIOS() @ 0x000000001538bbdb in /usr/bin/clickhouse signoz-clickhouse | 5. Poco:NetHTTPHeaderOutputStream:~HTTPHeaderOutputStream() @ 0x000000001538bf1f in /usr/bin/clickhouse signoz-clickhouse | 6. DB:HTTPServerResponse:send() @ 0x0000000012942988 in /usr/bin/clickhouse signoz-clickhouse | 7. DB:HTTPServerConnection:sendErrorResponse(Poco:Net:HTTPServerSession&, Poco:NetHTTPResponse:HTTPStatus) @ 0x000000001293ecda in /usr/bin/clickhouse signoz-clickhouse | 8. DB:HTTPServerConnection:run() @ 0x000000001293e97b in /usr/bin/clickhouse signoz-clickhouse | 9. Poco:NetTCPServerConnection:start() @ 0x00000000153a5a72 in /usr/bin/clickhouse signoz-clickhouse | 10. Poco:NetTCPServerDispatcher:run() @ 0x00000000153a6871 in /usr/bin/clickhouse signoz-clickhouse | 11. Poco:PooledThread:run() @ 0x000000001549f047 in /usr/bin/clickhouse signoz-clickhouse | 12. Poco:ThreadImpl:runnableEntry(void*) @ 0x000000001549d67d in /usr/bin/clickhouse signoz-clickhouse | 13. ? @ 0x00007f3668baf609 signoz-clickhouse | 14. ? @ 0x00007f3668ad4353 signoz-clickhouse | (version 24.1.2.5 (official build))
s
The broken pipe error usually happens when there is resource usage is high specifically CPU. What is the CPU usage when you see these errors?
u
100~200%
37 Views
Open in Slack
PreviousNext
Powered by

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.

Powered by