I'm having a few problems with the otel collector,...
# supporto
Onur Yikilmazoglu
02/03/2022, 6:06 PMI'm having a few problems with the otel collector, not sure if I should ask here or somewhere else since its not exactly what you guys do, however I keep having this problem where there the otel collector will balloon in ram usage and will eat most of the ram and almost all of my swap. I just check the my server and it using 6.5 gb of swap alone. I killed it with htop and swap usage dropped back down to 190 mb. However this is at best temporary, has anyone else ran into this problem?
Onur Yikilmazoglu
02/03/2022, 6:08 PMall have to show is this for now, I'll post a more relevant screenshot when it gets worse again
p
Pranay
02/03/2022, 6:09 PM
@User We have seen some users reporting this issue. How long have you been running this otel collector for?
Pranay
02/03/2022, 6:10 PM
@User may also have more insights on this
a
Ankit Nayan
02/03/2022, 6:58 PM
@User please change the memory limit of otel-collector from https://github.com/SigNoz/signoz/blob/main/deploy/docker/clickhouse-setup/docker-compose.yaml#L86
Ankit Nayan
02/03/2022, 6:59 PM
and you can add similar limit to this instance too https://github.com/SigNoz/signoz/blob/main/deploy/docker/clickhouse-setup/docker-compose.yaml#L92
o
Onur Yikilmazoglu
02/03/2022, 7:27 PM@User I run it constantly, about 15-20 at least basically
Onur Yikilmazoglu
02/03/2022, 7:28 PM@User will try that
👍 1
Onur Yikilmazoglu
02/14/2022, 2:19 PMI've been more meticulously logging my ram usage and this is still happening, the otel collector and the metrics collector are ballooning in size over the span of hours, you guys got any idea why?
p
Pranay
02/16/2022, 5:05 AM
thanks for the detailed report @User We will need to investigate this further. Can you also create an issue for this so that this doesn't get lost in slack threads? Would be very helpful
a
Ankit Nayan
02/16/2022, 6:16 AM
@User how many spans/hr or spans/day are you ingesting? You can see this at
Usage ExplorerAnkit Nayan
02/16/2022, 6:16 AM
Also, are you using metrics too?
o
Onur Yikilmazoglu
02/16/2022, 11:13 AM@User we peak at about 170k/h on an average day and yes we also use metrics
Onur Yikilmazoglu
02/16/2022, 11:13 AM@User will do
Onur Yikilmazoglu
02/16/2022, 11:14 AMI'm thinking of putting a limit on swap so it doesn't go out of control
Onur Yikilmazoglu
02/16/2022, 12:13 PMoh man you guys are going to make fun of me for this one, turns out the machine Im using didn't have swap memory limits enabled for cgroups.
Onur Yikilmazoglu
02/16/2022, 12:14 PMI'm crossing my fingers this sorts its out if not Im going to have to open an issue on github
p
Pranay
02/16/2022, 12:17 PM
😛
Pranay
02/16/2022, 12:18 PM
Irrespective - please open an issue 😉 Will be good reference for users in future
o
Onur Yikilmazoglu
02/17/2022, 11:32 AMsure will do
3 Views