https://signoz.io logo
Join Slack
Powered by
Hi team our team got self- hosted instalation and...
# support
a
Hi team our team got self- hosted instalation and yesterday we update signoz version to 0.57.0 and logs stopped receiving, we are using K8s-Infra chart to collect logs, if someone got this problem i apprecite to help me with a solution
s
What was the previous version? We split the chart into two independent charts. Now there is a 
k8s-infra
chart which needs to be installed separately.
a
we start with signoz version 0.55.0
s
Are logs not getting ingested or are they not showing up in UI?
a
in this moment we dont see on UI
k8s-infra-0.11.18
s
Share the full logs of query-service when you use UI
a
query-service.log
query-service.log
s
Exec into ClickHouse and share the output of following query.
Copy code
SELECT count()
FROM signoz_logs.distributed_logs_v2
WHERE timestamp > toUnixTimestamp64Nano(now64() - toIntervalMinute(5))
a
no ingest
s
Do you see any error logs in signoz-otel-collector or ClickHouse pods?
a
in signoz-otel-collector dont see errors
s
Then the logs are not getting sent to SigNoz. Check what's going on with the otel-agent pods from the k8s-infra pods.
a
Copy code
i see son errors in click house 2024.11.08 16:07:48.155270 [ 569 ] {} <Error> signoz_logs.distributed_logs_v2_resource.DistributedInsertQueue.default: Code: 252. DB::Exception: Received from chi-signoz-clickhouse-cluster-0-0:9000. DB::Exception: Too many parts (3000 with average size of 26.50 KiB) in table 'signoz_logs.logs_v2_resource (a8150343-8461-489f-923f-941e6234837c)'. Merges are processing significantly slower than inserts. Stack trac
clickhouse.log,query-service-otel.log
query-service-otel.logclickhouse.log
s
Too many parts (3000 with average size of 26.50 KiB) in table 'signoz_logs.logs_v2_resource (a8150343-8461-489f-923f-941e6234837c)'. Merges are processing significantly slower than inserts
Your ClickHouse probably doesn't have enough CPU resources.
a
i gona put some more
hi i found this error too
Copy code
{"date_time":"1731085572.256652","thread_name":"TCPServerConnection ([#13])","thread_id":"782","level":"Error","query_id":"","logger_name":"Access(user directories)","message":"from: ::ffff:10.100.70.82, user: clickhouse_operator: Authentication failed: Code: 195. DB::Exception: Connections from ::ffff:10.100.70.82 are not allowed. (IP_ADDRESS_NOT_ALLOWED), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in \/usr\/bin\/clickhouse\n1. DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x0000000007219a43 in \/usr\/bin\/clickhouse\n2. DB::IAccessStorage::throwAddressNotAllowed(Poco::Net::IPAddress const&) @ 0x000000001034b691 in \/usr\/bin\/clickhouse\n3. DB::IAccessStorage::authenticateImpl(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&, bool, bool, bool) const @ 0x000000001034b5b7 in \/usr\/bin\/clickhouse\n4. DB::MultipleAccessStorage::authenticateImpl(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&, bool, bool, bool) const @ 0x000000001038244a in \/usr\/bin\/clickhouse\n5. DB::AccessControl::authenticate(DB::Credentials const&, Poco::Net::IPAddress const&) const @ 0x00000000102bd560 in \/usr\/bin\/clickhouse\n6. DB::Session::authenticate(DB::Credentials const&, Poco::Net::SocketAddress const&) @ 0x00000000116a5323 in \/usr\/bin\/clickhouse\n7. DB::HTTPHandler::authenticateUser(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&) @ 0x00000000128b6461 in \/usr\/bin\/clickhouse\n8. DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x00000000128b7ce1 in \/usr\/bin\/clickhouse\n9. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x00000000128c02d0 in \/usr\/bin\/clickhouse\n10. DB::HTTPServerConnection::run() @ 0x000000001293e47a in \/usr\/bin\/clickhouse\n11. Poco::Net::TCPServerConnection::start() @ 0x00000000153a5a72 in \/usr\/bin\/clickhouse\n12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153a6871 in \/usr\/bin\/clickhouse\n13. Poco::PooledThread::run() @ 0x000000001549f047 in \/usr\/bin\/clickhouse\n14. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001549d67d in \/usr\/bin\/clickhouse\n15. ? @ 0x00007f2d12838609\n16. ? @ 0x00007f2d1275d353\n (version 24.1.2.5 (official build))","source_file":"src\/Common\/Exception.cpp; void DB::tryLogCurrentExceptionImpl(Poco::Logger *, const std::string &)","source_line":"222"}
{"date_time":"1731085572.256931","thread_name":"TCPServerConnection ([#13])","thread_id":"782","level":"Error","query_id":"","logger_name":"DynamicQueryHandler","message":"Code: 516. DB::Exception: clickhouse_operator: Authentication failed: password is incorrect, or there is no user with such name. (AUTHENTICATION_FAILED), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in \/usr\/bin\/clickhouse\n1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000007305f8c in \/usr\/bin\/clickhouse\n2. DB::AccessControl::authenticate(DB::Credentials const&, Poco::Net::IPAddress const&) const @ 0x00000000102bd834 in \/usr\/bin\/clickhouse\n3. DB::Session::authenticate(DB::Credentials const&, Poco::Net::SocketAddress const&) @ 0x00000000116a5323 in \/usr\/bin\/clickhouse\n4. DB::HTTPHandler::authenticateUser(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&) @ 0x00000000128b6461 in \/usr\/bin\/clickhouse\n5. DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x00000000128b7ce1 in \/usr\/bin\/clickhouse\n6. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x00000000128c02d0 in \/usr\/bin\/clickhouse\n7. DB::HTTPServerConnection::run() @ 0x000000001293e47a in \/usr\/bin\/clickhouse\n8. Poco::Net::TCPServerConnection::start() @ 0x00000000153a5a72 in \/usr\/bin\/clickhouse\n9. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153a6871 in \/usr\/bin\/clickhouse\n10. Poco::PooledThread::run() @ 0x000000001549f047 in \/usr\/bin\/clickhouse\n11. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001549d67d in \/usr\/bin\/clickhouse\n12. ? @ 0x00007f2d12838609\n13. ? @ 0x00007f2d1275d353\n (version 24.1.2.5 (official build))","source_file":"src\/Common\/Exception.cpp; void DB::tryLogCurrentExceptionImpl(Poco::Logger *, const std::string &)","source_line":"222"}
{"date_time":"1731085572.258188","thread_name":"TCPServerConnection ([#13])","thread_id":"782","level":"Error","query_id":"","logger_name":"Access(user directories)","message":"from: ::ffff:10.100.70.82, user: clickhouse_operator: Authentication failed: Code: 195. DB::Exception: Connections from ::ffff:10.100.70.82 are not allowed. (IP_ADDRESS_NOT_ALLOWED), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in \/usr\/bin\/clickhouse\n1. DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x0000000007219a43 in \/usr\/bin\/clickhouse\n2. DB::IAccessStorage::throwAddressNotAllowed(Poco::Net::IPAddress const&) @ 0x000000001034b691 in \/usr\/bin\/clickhouse\n3. DB::IAccessStorage::authenticateImpl(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&, bool, bool, bool) const @ 0x000000001034b5b7 in \/usr\/bin\/clickhouse\n4. DB::MultipleAccessStorage::authenticateImpl(DB::Credentials const&, Poco::Net::IPAddress const&, DB::ExternalAuthenticators const&, bool, bool, bool) const @ 0x000000001038244a in \/usr\/bin\/clickhouse\n5. DB::AccessControl::authenticate(DB::Credentials const&, Poco::Net::IPAddress const&) const @ 0x00000000102bd560 in \/usr\/bin\/clickhouse\n6. DB::Session::authenticate(DB::Credentials const&, Poco::Net::SocketAddress const&) @ 0x00000000116a5323 in \/usr\/bin\/clickhouse\n7. DB::HTTPHandler::authenticateUser(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&) @ 0x00000000128b6461 in \/usr\/bin\/clickhouse\n8. DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x00000000128b7ce1 in \/usr\/bin\/clickhouse\n9. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x00000000128c02d0 in \/usr\/bin\/clickhouse\n10. DB::HTTPServerConnection::run() @ 0x000000001293e47a in \/usr\/bin\/clickhouse\n11. Poco::Net::TCPServerConnection::start() @ 0x00000000153a5a72 in \/usr\/bin\/clickhouse\n12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153a6871 in \/usr\/bin\/clickhouse\n13. Poco::PooledThread::run() @ 0x000000001549f047 in \/usr\/bin\/clickhouse\n14. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001549d67d in \/usr\/bin\/clickhouse\n15. ? @ 0x00007f2d12838609\n16. ? @ 0x00007f2d1275d353\n (version 24.1.2.5 (official build))","source_file":"src\/Common\/Exception.cpp; void DB::tryLogCurrentExceptionImpl(Poco::Logger *, const std::string &)","source_line":"222"}
{"date_time":"1731085572.258364","thread_name":"TCPServerConnection ([#13])","thread_id":"782","level":"Error","query_id":"","logger_name":"DynamicQueryHandler","message":"Code: 516. DB::Exception: clickhouse_operator: Authentication failed: password is incorrect, or there is no user with such name. (AUTHENTICATION_FAILED), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in \/usr\/bin\/clickhouse\n1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000007305f8c in \/usr\/bin\/clickhouse\n2. DB::AccessControl::authenticate(DB::Credentials const&, Poco::Net::IPAddress const&) const @ 0x00000000102bd834 in \/usr\/bin\/clickhouse\n3. DB::Session::authenticate(DB::Credentials const&, Poco::Net::SocketAddress const&) @ 0x00000000116a5323 in \/usr\/bin\/clickhouse\n4. DB::HTTPHandler::authenticateUser(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&) @ 0x00000000128b6461 in \/usr\/bin\/clickhouse\n5. DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x00000000128b7ce1 in \/usr\/bin\/clickhouse\n6. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x00000000128c02d0 in \/usr\/bin\/clickhouse\n7. DB::HTTPServerConnection::run() @ 0x000000001293e47a in \/usr\/bin\/clickhouse\n8. Poco::Net::TCPServerConnection::start() @ 0x00000000153a5a72 in \/usr\/bin\/clickhouse\n9. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153a6871 in \/usr\/bin\/clickhouse\n10. Poco::PooledThread::run() @ 0x000000001549f047 in \/usr\/bin\/clickhouse\n11. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001549d67d in \/usr\/bin\/clickhouse\n12. ? @ 0x00007f2d12838609\n13. ? @ 0x00007f2d1275d353\n (version 24.1.2.5 (official build))","source_file":"src\/Common\/Exception.cpp; void DB::tryLogCurrentExceptionImpl(Poco::Logger *, const std::string &)","source_line":"222"}
@Srikanth Chekuri went ypu have a time we give more resource to clikhouse and put HA but we continue dont reciving logs, we se this new logs
Copy code
{"date_time":"1731093428.440745","thread_name":"","thread_id":"474","level":"Information","query_id":"","logger_name":"signoz_logs.logs_v2_resource (ReplicatedMergeTreePartCheckThread)","message":"Found the missing part 19700121_956072_956110_5 at 19700121_956072_956110_5 on chi-signoz-clickhouse-cluster-0-1","source_file":"src\/Storages\/MergeTree\/ReplicatedMergeTreePartCheckThread.cpp; bool DB::ReplicatedMergeTreePartCheckThread::searchForMissingPartOnOtherReplicas(const String &) const","source_line":"201"}
{"date_time":"1731093428.797659","thread_name":"","thread_id":"753","level":"Information","query_id":"7b123994-af67-4bb9-88d6-e9dd8269ec64","logger_name":"signoz_logs.logs_v2 (b941206c-4bcb-48f1-8679-0d4356ac4cac)","message":"Delaying inserting block by 147 ms. because there are 1293 parts and their average size is 16.32 MiB","source_file":"src\/Storages\/MergeTree\/MergeTreeData.cpp; void DB::MergeTreeData::delayInsertOrThrowIfNeeded(Poco::Event *, const ContextPtr &, bool) const","source_line":"4558"}
{"date_time":"1731093429.329284","thread_name":"","thread_id":"913","level":"Information","query_id":"d2e89678-ff65-49ec-b248-91c223b1aff9","logger_name":"signoz_logs.logs_v2 (b941206c-4bcb-48f1-8679-0d4356ac4cac)","message":"Delaying inserting block by 147 ms. because there are 1293 parts and their average size is 16.32 MiB","source_file":"src\/Storages\/MergeTree\/MergeTreeData.cpp; void DB::MergeTreeData::delayInsertOrThrowIfNeeded(Poco::Event *, const ContextPtr &, bool) const","source_line":"4558"}
s
It had a lot of pending things. It should have been resolved by now.
31 Views
Open in Slack
PreviousNext
Powered by