https://signoz.io logo
#support
Title
# support
z

Zaid Khan

03/02/2022, 2:22 PM
Hi Team, I have installed OpenTelemetry with SigNoz and it was working fine previously. Today, I am getting exceptions in my tomcat (catalina logs) but Signoz is running perfectly. I am attaching the exception. Can someone help me? It's printing on a short interval (may be due to corn jobs running) which causes my server to slow down. Exception: ERROR io.opentelemetry.exporter.otlp.internal.grpc.OkHttpGrpcExporter - Failed to export spans. The request could not be executed. Full error message: timeout
a

Ankit Nayan

03/02/2022, 2:24 PM
are you seeing dataloss? data might have been successfully sent after automatic retry
also can you show output of
docker stats --no-stream
z

Zaid Khan

03/02/2022, 2:30 PM
@User Data is their in the traces. i am sending the stats output
@User Please find this attached stats of docker
a

Ankit Nayan

03/02/2022, 2:39 PM
this error will be intermittent and dataloss should not be there
z

Zaid Khan

03/02/2022, 2:40 PM
@User No, it's occurring continuously.
even after server restart, it's giving me this exception
a

Ankit Nayan

03/02/2022, 2:44 PM
the otel-collector has maxed out on the memory allocated
that is the RAM available at the machine
and which version of SigNoz are you using v0.6.2 or earlier?
z

Zaid Khan

03/02/2022, 2:56 PM
I have installed it on 10th February, 2022. Can you please guide me how to verify the version? I am unable to find it
I think I will be on v0.6.0 or v0.6.1
Also, can you please tell me how to check the otel-collector memory?
a

Ankit Nayan

03/02/2022, 3:06 PM
docker stats --no-stream
shows memory of otel-collector. Check one with image name
clickhouse-setup_otel-collector_1
how urgent is the issue...we are making a new release tomorrow with new otel-collector image...currently checking the memory issue
if you can't wait till tomorrow, I can help you to do manual changes after updating to v0.6.2
z

Zaid Khan

03/02/2022, 3:09 PM
ok sure I can wait it till tomorrow. But can I pause the signoz (open telemetry)? As it's running on our internal server and I can get back to you by tomorrow for the update.
a

Ankit Nayan

03/02/2022, 3:12 PM
You don't need to stop signoz. The timeout shouldn't affect your application latency. Sending telemetry data is async and maintains a buffer after which it drops telemetry data if the application is not able to reach otel-collector and hence RAM usage also should not increase.
if you must, you would need to change the run command of your application by removing the
opentelemetry-instrument
part in your command to run application
I can get back to you by tomorrow for the update.
that would be awesome
z

Zaid Khan

03/02/2022, 3:15 PM
No, actually it's printing exception in tomcat catalina logs as my application uses cron jobs in regular interval. That's why sometimes it's giving 504 even server is not used up. Also, in our catalina file, there's no exception except this one.
Can it will be paused by removing the settings from setenv.sh file?
a

Ankit Nayan

03/02/2022, 3:17 PM
yes..
removing
export CATALINA_OPTS="$CATALINA_OPTS -javaagent:/path/opentelemetry-javaagent.jar"
from setenv.sh file should work
or you can just add
OTEL_JAVAAGENT_ENABLED=false
env at setenv.sh file
z

Zaid Khan

03/02/2022, 6:22 PM
I have currently disabled it. Thanks for your help 🙂
a

Ankit Nayan

03/02/2022, 6:51 PM
Cool
hey @User you can install the latest version
v0.7.1
by
Copy code
git checkout main
git pull origin main
and from
signoz/deploy
run
Copy code
./install.sh
checkout the release features at https://github.com/SigNoz/signoz/releases
z

Zaid Khan

03/09/2022, 6:19 AM
Hey @User Thanks for the update. I will update it soon and will let you know
13 Views