Hello, I'm facing an issue when deploying signoz i...
# support
j
Hello, I'm facing an issue when deploying signoz in deattached mode. that is running it as
docker compose -f docker-compose.yaml up
Note: I have almost not changed the compose file at all, just commented the hotrop example. Every some seconds, the following error appears in clickhouse
Copy code
signoz-clickhouse      | 2024.05.13 07:45:46.188830 [ 47 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 32, I/O error: Broken pipe, Stack trace (when copying this message, always include the lines below):
signoz-clickhouse      | 
signoz-clickhouse      | 0. Poco::Net::SocketImpl::error(int, String const&) @ 0x00000000153a1b5f in /usr/bin/clickhouse
signoz-clickhouse      | 1. Poco::Net::SocketImpl::sendBytes(void const*, int, int) @ 0x00000000153a2bbd in /usr/bin/clickhouse
signoz-clickhouse      | 2. Poco::Net::StreamSocketImpl::sendBytes(void const*, int, int) @ 0x00000000153a5296 in /usr/bin/clickhouse
signoz-clickhouse      | 3. Poco::Net::HTTPSession::write(char const*, long) @ 0x00000000153908b3 in /usr/bin/clickhouse
signoz-clickhouse      | 4. Poco::Net::HTTPHeaderIOS::~HTTPHeaderIOS() @ 0x000000001538bbdb in /usr/bin/clickhouse
signoz-clickhouse      | 5. Poco::Net::HTTPHeaderOutputStream::~HTTPHeaderOutputStream() @ 0x000000001538bf1f in /usr/bin/clickhouse
signoz-clickhouse      | 6. DB::HTTPServerResponse::send() @ 0x0000000012942988 in /usr/bin/clickhouse
signoz-clickhouse      | 7. DB::HTTPServerConnection::sendErrorResponse(Poco::Net::HTTPServerSession&, Poco::Net::HTTPResponse::HTTPStatus) @ 0x000000001293ecda in /usr/bin/clickhouse
signoz-clickhouse      | 8. DB::HTTPServerConnection::run() @ 0x000000001293e97b in /usr/bin/clickhouse
signoz-clickhouse      | 9. Poco::Net::TCPServerConnection::start() @ 0x00000000153a5a72 in /usr/bin/clickhouse
signoz-clickhouse      | 10. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153a6871 in /usr/bin/clickhouse
signoz-clickhouse      | 11. Poco::PooledThread::run() @ 0x000000001549f047 in /usr/bin/clickhouse
signoz-clickhouse      | 12. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001549d67d in /usr/bin/clickhouse
signoz-clickhouse      | 13. ? @ 0x0000721d932f8609
signoz-clickhouse      | 14. ? @ 0x0000721d9321d353
signoz-clickhouse      |  (version 24.1.2.5 (official build))
My problem is that I can get trace information in clickhouse but cannot get any hostmetrics. I have a local collector running in a different machine, which I can assure it is collecting data each 10 seconds (as I can see it in the logger), but this data does not seem to reach clickhouse DB. I have no idea if it is related to the error shown above. Any help or guidance is appreciated.
n
You can ignore these clickhouse errors, these are generated by query-service when the queries are cancelled.
j
Hello Thanks for the response. Now, if that is not the issue, then I have another problem 😉 How can I make sure that the signoz collector is receiving the data sent by the other collector running in a different VM and that this data is properly stored in clickhouse? When using the clickhouse client to select all rows of table time_series_v2 (for example), I just get a few rows. However, If Im constantly sending the data, i should have more, no? Also the time stamp i see there is from 3 days ago
Any idea?
n
You can start by checking the logs of the collectors.
j
Where exactly do i find them?
n
if you are running it on docker you can just run
docker logs signoz-otel-collector -f
j
Thanks for the reply. Now, I can see that the messages for CPU arrive to my signoz_collector like>
Copy code
Resource SchemaURL: <https://opentelemetry.io/schemas/1.9.0>
Resource attributes:
     -> host.name: Str(UEcollector)
     -> os.type: Str(linux)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope otelcol/hostmetricsreceiver/cpu 0.98.0
Metric #0
Descriptor:
     -> Name: system.cpu.time
     -> Description: Total seconds each logical CPU spent on each mode.
     -> Unit: s
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> cpu: Str(cpu0)
     -> state: Str(user)
StartTimestamp: 2024-05-07 11:29:56 +0000 UTC
Timestamp: 2024-05-14 09:00:13.646666601 +0000 UTC
Value: 2580.600000
I have configured the variables with a slightly different query as when running the given one run into a lack of memory issue. According to my understanding, this is only used to get the host name and then should not make a big difference
However, none of the metrics is retrieved Data is being collected, but it seems to be a problem with the query performed. Eventhough cannot tell much from inspect mode
image.png
How could I perform the same query from clickhouse client to make sure data is there?
n