https://signoz.io
Join Slack
Hello we are seeing a lot of the following logs in our `signoz-collector` running in K8s. We gave th...
# support
a
Hello we are seeing a lot of the following logs in our 
signoz-collector
running in K8s. We gave the collector instance more memory, but we continued to see these errors
Copy code
{"caller":"exporterhelper/queue_sender.go:92", "data_type":"metrics", "dropped_items":14317, "error":"not retryable error: Permanent error: rpc error: code = ResourceExhausted desc = limit exceeded", "kind":"exporter", "level":"error", "msg":"Exporting failed. Dropping data.", "name":"otlp", "stacktrace":"<http://go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1|go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1>
	<http://go.opentelemetry.io/collector/exporter@v0.109.0/exporterhelper/queue_sender.go:92|go.opentelemetry.io/collector/exporter@v0.109.0/exporterhelper/queue_sender.go:92>
<http://go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume|go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume>
	<http://go.opentelemetry.io/collector/exporter@v0.109.0/internal/queue/bounded_memory_queue.go:52|go.opentelemetry.io/collector/exporter@v0.109.0/internal/queue/bounded_memory_queue.go:52>
<http://go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1|go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1>
	<http://go.opentelemetry.io/collector/exporter@v0.109.0/internal/queue/consumers.go:43|go.opentelemetry.io/collector/exporter@v0.109.0/internal/queue/consumers.go:43>", "ts":1.7326457593054848E9}
I can share our configuration as well in thread
Our Helm chart:
Copy code
apiVersion: v1
kind: Namespace
metadata:
  name: signoz-exporter
---
apiVersion: <http://source.toolkit.fluxcd.io/v1beta2|source.toolkit.fluxcd.io/v1beta2>
kind: HelmRepository
metadata:
  name: signoz-repo
  namespace: signoz-exporter
spec:
  interval: 12h0m0s
  url: <https://charts.signoz.io>
---
apiVersion: <http://external-secrets.io/v1beta1|external-secrets.io/v1beta1>
kind: ExternalSecret
metadata:
  name: signoz-config
  namespace: signoz-exporter
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: google-secret-manager
  target:
    name: signoz-config
    creationPolicy: Owner
    template:
      type: Opaque
      engineVersion: v2
      data:
        values-override.yaml: |
          global.cloud: others
          global.clusterName: PROD
          global.deploymentEnvironment: PROD
          otelCollectorEndpoint: ingest.us.signoz.cloud:443
          otelInsecure: false
          signozApiKey: {{ .signoz_api_key | toString }}
          presets.otlpExporter.enabled: true
          presets.loggingExporter.enabled: false
          resources.requests.memory: 500Mi
          resources.requests.cpu: 500m
  data:
    - secretKey: signoz_api_key # name of value in k8s secret
      remoteRef:
        key: SIGNOZ_API_KEY # name of secret in google secret manager
---
apiVersion: <http://helm.toolkit.fluxcd.io/v2beta1|helm.toolkit.fluxcd.io/v2beta1>
kind: HelmRelease
metadata:
  name: signoz
  namespace: signoz-exporter
spec:
  releaseName: signoz-otel-collector
  chart:
    spec:
      chart: "k8s-infra"
      sourceRef:
        kind: HelmRepository
        name: signoz-repo
        namespace: signoz-exporter
  interval: 12h0m0s
  valuesFrom:
    - kind: Secret
      name: signoz-config
      valuesKey: values-override.yaml
For context, we are trying to get started here with SigNoz as our observability platform and tried to use 
opentelemetry-instrument
on one of our services to start pushing metrics to our collector.
b
I had this problem recently and my space on clickhouse needed upped
g
@Bryan DeLong clickhouse storage space or memory? I'm seeing something similar in my deployment (will have to check if it is the same error)
n
@Prashant Shahi ^
b
It was clickhouse storage space
7 Views
Open in Slack
PreviousNext
Powered by

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.

Powered by