https://signoz.io logo
Join Slack
Powered by
Hi Team, We have deployed the signoz on eks cluste...
# support
a
Hi Team, We have deployed the signoz on eks cluster & we are tring to push logs from our application but not getting any logs on signoz
n
Hey @Ashok, please follow this doc to deploy signoz on eks and let me know if you're still facing any issues: https://signoz.io/docs/install/kubernetes/aws/
a
Hi Nages, I have followed the same document for installation but with this also we are not able push the logs to signoz from my application, however i have downgrade the version on 0.70 also.
n
Hey @Ashok what version were you using earlier?
a
Latest version
0.76
n
okayy, are you able to resolve the issue by degrading the version?
a
@Nagesh Bansal No, Actually we are not getting alerts on signoz yet, Our application & signoz both are on EKS cluster, we are getting the timeout issue even we have opened the connectivity on 0.0.0.0/0, But when we are trying to push the logs on same signoz from my EC2 through then we are able to push the logs.
@Work
w
Hey @Nagesh Bansal To elaborate the above issue we have setup the signoz on eks. Also we have our application deployed on same EKS just a different namespace. We are unable to send the metric/traces & logs from this eks application to signoz. Error :
Copy code
APP_LOG ERROR __init__ Exception while exporting metrics an integer is required (got type NoneType)
Traceback (most recent call last):
File "/home/feature-flow-user/.local/lib/python3.9/site-packages/opentelemetry/exporter/otlp/proto/grpc/exporter.py", line 305, in _export
self._client.Export(
File "/home/feature-flow-user/.local/lib/python3.9/site-packages/grpc/_channel.py", line 1181, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/feature-flow-user/.local/lib/python3.9/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.24.86.232:4317: Failed to connect to remote host: Timeout occurred: FD Shutdown"
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2025-04-01T09:14:17.848646459+00:00", grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.24.86.232:4317: Failed to connect to remote host: Timeout occurred: FD Shutdown"}"
>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/feature-flow-user/.local/lib/python3.9/site-packages/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 482, in _receive_metrics
self._exporter.export(metrics_data, timeout_millis=timeout_millis)
File "/home/feature-flow-user/.local/lib/python3.9/site-packages/opentelemetry/exporter/otlp/proto/grpc/metric_exporter/__init__.py", line 248, in export
return self._export(metrics_data)
File "/home/feature-flow-user/.local/lib/python3.9/site-packages/opentelemetry/exporter/otlp/proto/grpc/exporter.py", line 345, in _export
sleep(delay)
TypeError: an integer is required (got type NoneType)
Where as when the same application I deployed on one of the EC2 from this we are able to push everything and this ec2 service is visible on Signoz UI. Could you please help what might be the issue with the EKS application ? Also as mentioned by @Ashok he has opened the connectivity on 0.0.0.0/0.
Hey @Nagesh Bansal @Srikanth Chekuri These are the extra grpc debug logs here i see its writing to the signoz otel collector right but its not seen on Signoz UI
Copy code
I0000 00:00:1743575019.713487 78 <http://timer_manager.cc:188]|timer_manager.cc:188]> wait ended: was_timed:true kicked:false
I0000 00:00:1743575019.713516 78 <http://timer_manager.cc:174]|timer_manager.cc:174]> sleep for a 62 milliseconds
I0000 00:00:1743575019.756790 152 <http://posix_engine.cc:442]|posix_engine.cc:442]> PosixEventEngine:0x272b4270 executing callback:{00007f78a8003ee0,0000000000000057}
I0000 00:00:1743575019.756836 152 bdp_estimator.h:50] bdp[ipv4:10.24.50.127:4317]:sched acc=7 est=65536
I0000 00:00:1743575019.756844 152 <http://chttp2_transport.cc:970|chttp2_transport.cc:970>] W:0x7f78c4002260 CLIENT [ipv4:10.24.50.127:4317] state IDLE -> WRITING [BDP_PING]
I0000 00:00:1743575019.756856 152 <http://writing.cc:138|writing.cc:138>] CLIENT[0x7f78c4002260]: Ping 13641899506578541773 sent [ipv4:10.24.50.127:4317]: max_pings_without_data: 2, pings_before_data_required: 1, last_ping_sent_time_: @2524487ms
I0000 00:00:1743575019.756863 152 <http://chttp2_transport.cc:970]|chttp2_transport.cc:970]> W:0x7f78c4002260 CLIENT [ipv4:10.24.50.127:4317] state WRITING -> WRITING [begin write in current thread]
I0000 00:00:1743575019.756866 152 <http://chttp2_transport.cc:1100|chttp2_transport.cc:1100>] CLIENT[0x7f78c4002260]: Write 17 bytes
I0000 00:00:1743575019.756870 152 <http://tcp_posix.cc:1811|tcp_posix.cc:1811>] WRITE 0x7f78a80073a0 (peer=ipv4:10.24.50.127:4317)
I0000 00:00:1743575019.756933 152 <http://tcp_posix.cc:1856]|tcp_posix.cc:1856]> write: OK
I0000 00:00:1743575019.756942 152 <http://chttp2_transport.cc:1113]|chttp2_transport.cc:1113]> CLIENT[0x7f78c4002260]: Finish write
I0000 00:00:1743575019.756946 152 <http://chttp2_transport.cc:970|chttp2_transport.cc:970>] W:0x7f78c4002260 CLIENT [ipv4:10.24.50.127:4317] state WRITING -> IDLE [finish writing]
I0000 00:00:1743575019.756951 152 <http://chttp2_transport.cc:2903|chttp2_transport.cc:2903>] ipv4:10.24.50.127:4317: Start BDP ping err=OK
I0000 00:00:1743575019.756954 152 bdp_estimator.h:62] bdp[ipv4:10.24.50.127:4317]:start acc=0 est=65536
I0000 00:00:1743575049.715224 141 <http://promise_based_filter.cc:1555|promise_based_filter.cc:1555>] CLI[http-client:0x7f088c00f048] ClientCallData.RecvInitialMetadataReady has_promise=true sent_initial_state=FORWARDED recv_trailing_state=FORWARDED captured={} recv_initial_metadata=HOOKED_AND_GOT_PIPE error:OK md:PeerString: ipv4:10.24.86.232:4317, :status: 200, content-type: application/grpc, GrpcStatusFromWire: true
I0000 00:00:1743575049.715227 141 <http://promise_based_filter.cc:993]|promise_based_filter.cc:993]> CLI[http-client:0x7f088c00f048] ClientCallData.PollContext.Run has_promise=true sent_initial_state=FORWARDED recv_trailing_state=FORWARDED captured={} recv_initial_metadata=COMPLETE_AND_GOT_PIPE
I0000 00:00:1743575049.715230 141 <http://promise_based_filter.cc:1699]|promise_based_filter.cc:1699]> CLI[http-client:0x7f088c00f048] ClientCallData.PollTrailingMetadata has_promise=true sent_initial_state=FORWARDED recv_trailing_state=FORWARDED captured={} recv_initial_metadata=COMPLETE_AND_PUSHED_TO_PIPE
I0000 00:00:1743575049.715232 141 <http://promise_based_filter.cc:1076]|promise_based_filter.cc:1076]> CLI[http-client:0x7f088c00f048] ClientCallData.PollContext.Run: poll=<<pending>>; has_promise=true sent_initial_state=FORWARDED recv_trailing_state=FORWARDED captured={} recv_initial_metadata=COMPLETE_AND_PUSHED_TO_PIPE
I0000 00:00:1743575049.715234 141 <http://promise_based_filter.cc:993]|promise_based_filter.cc:993]> CLI[http-client:0x7f088c00f048] ClientCallData.PollContext.Run has_promise=true sent_initial_state=FORWARDED recv_trailing_state=FORWARDED captured={} recv_initial_metadata=COMPLETE_AND_PUSHED_TO_PIPE
I0000 00:00:1743575049.715236 141 <http://promise_based_filter.cc:1699]|promise_based_filter.cc:1699]> CLI[http-client:0x7f088c00f048] ClientCallData.PollTrailingMetadata has_promise=true sent_initial_state=FORWARDED recv_trailing_state=FORWARDED captured={} recv_initial_metadata=RESPONDED
I0000 00:00:1743575049.715238 141 <http://promise_based_filter.cc:1076]|promise_based_filter.cc:1076]> CLI[http-client:0x7f088c00f048] ClientCallData.PollContext.Run: poll=<<pending>>; has_promise=true sent_initial_state=FORWARDED recv_trailing_state=FORWARDED captured={} recv_initial_metadata=RESPONDED
I0000 00:00:1743575049.715242 141 <http://retry_filter_legacy_call_data.cc:793]|retry_filter_legacy_call_data.cc:793]> chand=0x7f0894004140 calld=0x7f088c00e540 attempt=0x7f088c010b10 batch_data=0x7f088c00ee10: got recv_initial_metadata_ready, error=OK
I0000 00:00:1743575049.715250 141 <http://retry_filter_legacy_call_data.cc:1861]|retry_filter_legacy_call_data.cc:1861]> chand=0x7f0894004140 calld=0x7f088c00e540: committing retries
I0000 00:00:1743575049.715253 141 <http://retry_filter_legacy_call_data.cc:1678]|retry_filter_legacy_call_data.cc:1678]> chand=0x7f0894004140 calld=0x7f088c00e540: destroying send_initial_metadata
I0000 00:00:1743575049.715256 141 <http://retry_filter_legacy_call_data.cc:1685]|retry_filter_legacy_call_data.cc:1685]> chand=0x7f0894004140 calld=0x7f088c00e540: destroying send_messages[0]
I0000 00:00:1743575049.715260 141 <http://retry_filter_legacy_call_data.cc:1692|retry_filter_legacy_call_data.cc:1692>] chand=0x7f0894004140 calld=0x7f088c00e540: destroying send_trailing_metadata
I0000 00:00:1743575049.715263 141 <http://retry_filter_legacy_call_data.cc:230]|retry_filter_legacy_call_data.cc:230]> chand=0x7f0894004140 calld=0x7f088c00e540 attempt=0x7f088c010b10: retry state no longer needed; moving LB call to parent and unreffing the call attempt
I0000 00:00:1743575049.715267 141 <http://retry_filter_legacy_call_data.cc:1845]|retry_filter_legacy_call_data.cc:1845]> chand=0x7f0894004140 calld=0x7f088c00e540: invoking recv_initial_metadata_ready for pending batch at index 0
I0000 00:00:1743575049.715274 141 <http://promise_based_filter.cc:675]|promise_based_filter.cc:675]> CLI[compression:0x7f088c00f060] ReceiveMessage.OnComplete st=FORWARDED_BATCH status=OK
I0000 00:00:1743575049.715279 141 <http://promise_based_filter.cc:993]|promise_based_filter.cc:993]> CLI[compression:0x7f088c00f060] ClientCallData.PollContext.Run has_promise=true sent_initial_state=FORWARDED recv_trailing_state=FORWARDED captured={} recv_initial_metadata=RESPONDED
I0000 00:00:1743575049.715283 141 <http://promise_based_filter.cc:469]|promise_based_filter.cc:469]> CLI[compression:0x7f088c00f060] SendMessage.WakeInsideCombiner st=IDLE
I0000 00:00:1743575049.715285 141 <http://promise_based_filter.cc:773]|promise_based_filter.cc:773]> CLI[compression:0x7f088c00f060] ReceiveMessage.WakeInsideCombiner st=BATCH_COMPLETED push?=no next?=no allow_push_to_pipe=yes
I0000 00:00:1743575049.715289 141 <http://compression_filter.cc:160]|compression_filter.cc:160]> DecompressMessage: len=2 max=4194304 alg=0
I0000 00:00:1743575049.715292 141 <http://promise_based_filter.cc:863]|promise_based_filter.cc:863]> CLI[compression:0x7f088c00f060] ReceiveMessage.WakeInsideCombiner next complete: got message new_state=PULLED_FROM_PIPE
I0000 00:00:1743575049.715294 141 <http://promise_based_filter.cc:879]|promise_based_filter.cc:879]> CLI[compression:0x7f088c00f060] ReceiveMessage.WakeInsideCombiner push complete
I0000 00:00:1743575049.715296 141 <http://promise_based_filter.cc:1699]|promise_based_filter.cc:1699]> CLI[compression:0x7f088c00f060] ClientCallData.PollTrailingMetadata has_promise=true sent_initial_state=FORWARDED recv_trailing_state=FORWARDED captured={} recv_initial_metadata=RESPONDED
I0000 00:00:1743575049.715299 141 <http://promise_based_filter.cc:1076]|promise_based_filter.cc:1076]> CLI[compression:0x7f088c00f060] ClientCallData.PollContext.Run: poll=<<pending>>; has_promise=true sent_initial_state=FORWARDED recv_trailing_state=FORWARDED captured={} recv_initial_metadata=RESPONDED
I0000 00:00:1743575049.715301 141 <http://promise_based_filter.cc:675]|promise_based_filter.cc:675]> CLI[message_size:0x7f088c00f030] ReceiveMessage.OnComplete st=FORWARDED_BATCH status=OK
I0000 00:00:1743575049.715303 141 <http://promise_based_filter.cc:993]|promise_based_filter.cc:993]> CLI[message_size:0x7f088c00f030] ClientCallData.PollContext.Run has_promise=true sent_initial_state=FORWARDED recv_trailing_state=FORWARDED captured={}
I0000 00:00:1743575049.715305 141 <http://promise_based_filter.cc:469]|promise_based_filter.cc:469]> CLI[message_size:0x7f088c00f030] SendMessage.WakeInsideCombiner st=IDLE
I0000 00:00:1743575049.715307 141 <http://promise_based_filter.cc:773]|promise_based_filter.cc:773]> CLI[message_size:0x7f088c00f030] ReceiveMessage.WakeInsideCombiner st=BATCH_COMPLETED push?=no next?=no allow_push_to_pipe=yes
I0000 00:00:1743575049.715310 141 <http://message_size_filter.cc:147]|message_size_filter.cc:147]> PBF_CLIENT[0x7f088c00f270]: [message_size] [message_size] recv len:2 max:4194304
I0000 00:00:1743575049.715313 141 <http://promise_based_filter.cc:863|promise_based_filter.cc:863>] CLI[message_size:0x7f088c00f030] ReceiveMessage.WakeInsideCombiner next complete: got message new_state=PULLED_FROM_PIPE
I0000 00:00:1743575049.715315 141 <http://promise_based_filter.cc:879]|promise_based_filter.cc:879]> CLI[message_size:0x7f088c00f030] ReceiveMessage.WakeInsideCombiner push complete
I0000 00:00:1743575049.715317 141 <http://promise_based_filter.cc:1699]|promise_based_filter.cc:1699]> CLI[message_size:0x7f088c00f030] ClientCallData.PollTrailingMetadata has_promise=true sent_initial_state=FORWARDED recv_trailing_state=FORWARDED captured={}
I0000 00:00:1743575049.715319 141 <http://promise_based_filter.cc:1076]|promise_based_filter.cc:1076]> CLI[message_size:0x7f088c00f030] ClientCallData.PollContext.Run: poll=<<pending>>; has_promise=true sent_initial_state=FORWARDED recv_trailing_state=FORWARDED captured={}
I0000 00:00:1743575049.715321 141 <http://retry_filter_legacy_call_data.cc:887]|retry_filter_legacy_call_data.cc:887]> chand=0x7f0894004140 calld=0x7f088c00e540 attempt=0x7f088c010b10 batch_data=0x7f088c00ee10: got recv_message_ready, error=OK
I0000 00:00:1743575049.715323 141 <http://retry_filter_legacy_call_data.cc:1845]|retry_filter_legacy_call_data.cc:1845]> chand=0x7f0894004140 calld=0x7f088c00e540: invoking recv_message_ready for pending batch at index 0
I0000 00:00:1743575049.715327 141 <http://promise_based_filter.cc:1749]|promise_based_filter.cc:1749]> CLI[compression:0x7f088c00f060] ClientCallData.RecvTrailingMetadataReady recv_trailing_state=FORWARDED error=OK md=grpc-message: , grpc-status: OK, GrpcStatusFromWire: true
I0000 00:00:1743575049.715332 141 <http://promise_based_filter.cc:716]|promise_based_filter.cc:716]> CLI[compression:0x7f088c00f060] ReceiveMessage.Done st=IDLE md=grpc-message: , grpc-status: OK, GrpcStatusFromWire: true
I0000 00:00:1743575049.715335 141 <http://promise_based_filter.cc:428]|promise_based_filter.cc:428]> CLI[compression:0x7f088c00f060] SendMessage.Done st=IDLE md=grpc-message: , grpc-status: OK, GrpcStatusFromWire: true
I0000 00:00:1743575049.715338 141 <http://promise_based_filter.cc:993]|promise_based_filter.cc:993]> CLI[compression:0x7f088c00f060] ClientCallData.PollContext.Run has_promise=true sent_initial_state=FORWARDED recv_trailing_state=COMPLETE captured={} recv_initial_metadata=RESPONDED
I0000 00:00:1743575049.715340 141 <http://promise_based_filter.cc:469]|promise_based_filter.cc:469]> CLI[compression:0x7f088c00f060] SendMessage.WakeInsideCombiner st=CANCELLED_BUT_NOT_YET_POLLED
I0000 00:00:1743575049.715343 141 <http://promise_based_filter.cc:773]|promise_based_filter.cc:773]> CLI[compression:0x7f088c00f060] ReceiveMessage.WakeInsideCombiner st=CANCELLED_WHILST_IDLE push?=no next?=no allow_push_to_pipe=yes
I0000 00:00:1743575049.715347 141 <http://promise_based_filter.cc:1699]|promise_based_filter.cc:1699]> CLI[compression:0x7f088c00f060] ClientCallData.PollTrailingMetadata has_promise=true sent_initial_state=FORWARDED recv_trailing_state=COMPLETE captured={} recv_initial_metadata=RESPONDED
I0000 00:00:1743575049.715351 141 <http://promise_based_filter.cc:1076]|promise_based_filter.cc:1076]> CLI[compression:0x7f088c00f060] ClientCallData.PollContext.Run: poll=grpc-message: , grpc-status: OK, GrpcStatusFromWire: true; has_promise=true sent_initial_state=FORWARDED recv_trailing_state=COMPLETE captured={} recv_initial_metadata=RESPONDED
I0000 00:00:1743575049.715359 141 <http://promise_based_filter.cc:428]|promise_based_filter.cc:428]> CLI[compression:0x7f088c00f060] SendMessage.Done st=CANCELLED md=grpc-message: , grpc-status: OK, GrpcStatusFromWire: true
I0000 00:00:1743575049.715364 141 <http://promise_based_filter.cc:716]|promise_based_filter.cc:716]> CLI[compression:0x7f088c00f060] ReceiveMessage.Done st=CANCELLED md=grpc-message: , grpc-status: OK, GrpcStatusFromWire: true
I0000 00:00:1743575049.715369 141 <http://promise_based_filter.cc:1749]|promise_based_filter.cc:1749]> CLI[http-client:0x7f088c00f048] ClientCallData.RecvTrailingMetadataReady recv_trailing_state=FORWARDED error=OK md=grpc-message: , grpc-status: OK, GrpcStatusFromWire: true
I0000 00:00:1743575049.715374 141 <http://promise_based_filter.cc:993]|promise_based_filter.cc:993]> CLI[http-client:0x7f088c00f048] ClientCallData.PollContext.Run has_promise=true sent_initial_state=FORWARDED recv_trailing_state=COMPLETE captured={} recv_initial_metadata=RESPONDED
I0000 00:00:1743575049.715378 141 <http://promise_based_filter.cc:1699]|promise_based_filter.cc:1699]> CLI[http-client:0x7f088c00f048] ClientCallData.PollTrailingMetadata has_promise=true sent_initial_state=FORWARDED recv_trailing_state=COMPLETE captured={} recv_initial_metadata=RESPONDED
I0000 00:00:1743575049.715382 141 <http://promise_based_filter.cc:1076]|promise_based_filter.cc:1076]> CLI[http-client:0x7f088c00f048] ClientCallData.PollContext.Run: poll=grpc-message: , grpc-status: OK, GrpcStatusFromWire: true; has_promise=true sent_initial_state=FORWARDED recv_trailing_state=COMPLETE captured={} recv_initial_metadata=RESPONDED
3 Views
Open in Slack
PreviousNext
Powered by