Hello! We've been running signoz for a week now an...
# support
a
Hello! We've been running signoz for a week now and today it stopped working,
clickhousetraceexporter/writer.go
throws
Cannot reserve 1.00 MiB, not enough space
. Signoz is deployed to
Kubernetes
and following this guide https://signoz.io/docs/operate/clickhouse/increase-clickhouse-pv/ I also increased the space to
25Gi
. The error persists. Can anyone help?
p
Abel - were you able to check if the PV increase was successful. Can you share what logs are you seeing now after PV increase?
a
Sure, I'm getting:
Copy code
exporterhelper/retry_sender.go:177    Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 243, message: Cannot reserve 1.00 MiB, not enough space", "interval": "0s"}`
In the
signoz-otel-collector
pod and also this one:
Copy code
Could not write a batch of spans to model table:     {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "error": "code: 243, m │
│ essage: Cannot reserve 2.76 MiB, not enough space"}                                                                                                                                                                                │
│ <http://github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans|github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans>                                                                                                                          │
│     /home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:415                                                                                                                  │
│ <http://github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData|github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData>                                                                                                                                 │
│     /home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:409                                                                                                     │
│ <http://go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export|go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export>                                                                                                                                                      │
│     /home/runner/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.88.0/exporterhelper/traces.go:60                                                                                                                             │
│ <http://go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send|go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send>                                                                                                                                                        │
│     /home/runner/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.88.0/exporterhelper/timeout_sender.go:41                                                                                                                     │
│ <http://go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send|go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send>                                                                                                                                                          │
│     /home/runner/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.88.0/exporterhelper/retry_sender.go:138                                                                                                                      │
│ <http://go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send|go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send>                                                                                                                                      │
│     /home/runner/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.88.0/exporterhelper/traces.go:177                                                                                                                            │
│ <http://go.opentelemetry.io/collector/exporter/exporterhelper.(*queueSender).start.func1|go.opentelemetry.io/collector/exporter/exporterhelper.(*queueSender).start.func1>                                                                                                                                                   │
│     /home/runner/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.88.0/exporterhelper/queue_sender.go:126                                                                                                                      │
│ <http://go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).Start.func1|go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).Start.func1>                                                                                                                                   │
│     /home/runner/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.88.0/exporterhelper/internal/bounded_memory_queue.go:52
I confirm
pv
was successfully increased. I also see an increased memory usage in this pod
~5Gi
There's the pv
Copy code
pvc-de9f5d01-115b-4fc0-b534-aaa106932e25   1Gi        RWO            Retain           Bound    sig-platform/signoz-db-signoz-query-service-0                                gce-resizable            13d

pvc-f5bdced1-3e79-42c6-9fb1-98cea2dc3e94   1Gi        RWO            Retain           Bound    sig-platform/storage-signoz-alertmanager-0                                   gce-resizable            13d

pvc-f5d2bf34-ff44-4406-803a-f6a79ccda066   20Gi       RWO            Retain           Bound    sig-platform/data-volumeclaim-template-chi-signoz-clickhouse-cluster-0-0-0   gce-resizable            13d
I changed pv size to
50Gi
and it started working. Is there a way to clean the pv in order to prevent it from filling it up?
a
reduce the retention period