https://signoz.io logo
Join Slack
Powered by
Hi Team, I have installed OpenTelemetry with SigNo...
# support
z
Hi Team, I have installed OpenTelemetry with SigNoz and it was working fine previously. Today, I am getting exceptions in my tomcat (catalina logs) but Signoz is running perfectly. I am attaching the exception. Can someone help me? It's printing on a short interval (may be due to corn jobs running) which causes my server to slow down. Exception: ERROR io.opentelemetry.exporter.otlp.internal.grpc.OkHttpGrpcExporter - Failed to export spans. The request could not be executed. Full error message: timeout
a
are you seeing dataloss? data might have been successfully sent after automatic retry
also can you show output of 
docker stats --no-stream
z
@User Data is their in the traces. i am sending the stats output
@User Please find this attached stats of docker
stats-signoz
a
this error will be intermittent and dataloss should not be there
z
@User No, it's occurring continuously.
even after server restart, it's giving me this exception
a
the otel-collector has maxed out on the memory allocated
that is the RAM available at the machine
and which version of SigNoz are you using v0.6.2 or earlier?
z
I have installed it on 10th February, 2022. Can you please guide me how to verify the version? I am unable to find it
I think I will be on v0.6.0 or v0.6.1
Also, can you please tell me how to check the otel-collector memory?
a
docker stats --no-stream
shows memory of otel-collector. Check one with image name 
clickhouse-setup_otel-collector_1
how urgent is the issue...we are making a new release tomorrow with new otel-collector image...currently checking the memory issue
if you can't wait till tomorrow, I can help you to do manual changes after updating to v0.6.2
z
ok sure I can wait it till tomorrow. But can I pause the signoz (open telemetry)? As it's running on our internal server and I can get back to you by tomorrow for the update.
a
You don't need to stop signoz. The timeout shouldn't affect your application latency. Sending telemetry data is async and maintains a buffer after which it drops telemetry data if the application is not able to reach otel-collector and hence RAM usage also should not increase.
if you must, you would need to change the run command of your application by removing the 
opentelemetry-instrument
part in your command to run application
I can get back to you by tomorrow for the update.
that would be awesome
z
No, actually it's printing exception in tomcat catalina logs as my application uses cron jobs in regular interval. That's why sometimes it's giving 504 even server is not used up. Also, in our catalina file, there's no exception except this one.
Can it will be paused by removing the settings from setenv.sh file?
a
yes..
removing 
export CATALINA_OPTS="$CATALINA_OPTS -javaagent:/path/opentelemetry-javaagent.jar"
from setenv.sh file should work
or you can just add 
OTEL_JAVAAGENT_ENABLED=false
env at setenv.sh file
z
I have currently disabled it. Thanks for your help 🙂
a
Cool
hey @User you can install the latest version 
v0.7.1
by
Copy code
git checkout main
git pull origin main
and from 
signoz/deploy
run
Copy code
./install.sh
checkout the release features at https://github.com/SigNoz/signoz/releases
z
Hey @User Thanks for the update. I will update it soon and will let you know
151 Views
Open in Slack
PreviousNext
Powered by