This message was deleted.

What is the system resources usage (CPU and memory) when you run into this problem?

I haven't checked CPU, but I increased mem from the default 100m to 500m, that didn't make much of a difference - I assume because it's just buffering data until it can send it, but it never gets sent

Is is 500mb? That's too small -- at least give a 2GB.

Well, it was running with 100m just fine until I upgraded

Both issues have common underlying issues i.e resource usage. If you could share the heap profile of the collector process that would help us better understand the issue

But isn't it strange that the increase in memory usage coincides with the io error?

No, I don't think it's really that strange. The i/o error occurred because of the broken connection. The connection might have been terminated by ClickHouse under the system resource stress.

Isn't it then strange that it doesn't reconnect again?

It would reconnect. How did you confirm it didn't reconnect?

Well, it stops sending anything, then after a while it's OOM killed, then it's restarted and it works again. You see the pattern pretty clearly on the last screenshot I sent where the memory usage is flat initially, then it's interrupted and doesn't send anything until it's killed