Hello, i'm having the following problem:
I started...
# supportu
Ulisses Silva
08/28/2023, 1:02 PMHello, i'm having the following problem:
I started to test signoz and have setup 3 java applications that are using the otel java agent on tomcat to send data directly to signoz collector on port 4317.
Sometimes i need to restart the signoz server and one of the applications just can't send data until i do a a restart of this application.
[otel.javaagent 2023-08-28 08:09:36:463 -0400] [OkHttp <http://signoz.server:4317/...>] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export spans. The request could not be executed. Full error message: FRAME_SIZE_ERROR: 4740180[otel.javaagent 2023-08-28 08:09:41:467 -0400] [OkHttp <http://signoz.server:4317/...>] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export spans. The request could not be executed. Full error message: FRAME_SIZE_ERROR: 4740180[otel.javaagent 2023-08-28 08:09:42:582 -0400] [OkHttp <http://signoz.server:4317/...>] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export logs. The request could not be executed. Full error message: FRAME_SIZE_ERROR: 4740180s
Srikanth Chekuri
08/28/2023, 1:06 PM``max_recv_msg_size_mib`` of otlp receiver https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configgrpc/README.md#server-configuration
u
Ulisses Silva
08/28/2023, 1:46 PMI have updated the otel-collector-config.yaml with
receivers.otlp.protocols.grpc.max_recv_msg_size_mib: 100, restarted the containers with docker compose down/up, but the problem continues.
Ulisses Silva
08/28/2023, 2:37 PMTo add some context, all 3 applications are running in the same VM server, but on different tomcat instances.
s
Srikanth Chekuri
08/29/2023, 3:21 AMLower the agent batch size to export most frequently and check.
u
Ulisses Silva
08/29/2023, 6:52 PMotel.bsp.max.export.batch.size or otel.blrp.max.export.batch.size? or both?
s
Srikanth Chekuri
08/30/2023, 12:11 PMYou might have to tweak both. How many spans does your app roughly produce? You need to adjust the settings, or you could drop the spans with head-based sampling if the volume and size are big.
u
Ulisses Silva
08/30/2023, 12:42 PMThis app is generating like 2220 spans per minute. There are some operations that generates 28000+ spans. I can't even open the Trace Detail page because the browser can't handle that number of elements and will freeze.
24 Views