s mohamed
04/04/2024, 11:41 AMmy-release-signoz-otel-collector.yaml
config file by adding below snippet
prometheus:
config:
scrape_configs:
- job_name: "otel-collector"
scrape_interval: 30s
static_configs:
- targets: ["otel-collector:8889", "prometheus-operated:9090"]
my prometheus is running in prometheus-operated:9090
i have port-forward locally and confirmed on the service it is fine
and restarted otel-deployment and otel-collector deployment pods and getting below error in collector pods
{"level":"error","timestamp":"2024-04-04T11:36:23.737Z","caller":"opamp/server_client.go:268","msg":"Collector failed for restart during rollback","component":"opamp-server-client","error":"failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:\n\n* error decoding 'exporters': error reading configuration for \"prometheus\": 1 error(s) decoding:\n\n* '' has invalid keys: config","stacktrace":"<http://github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).reload|github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).reload>\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:268\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*agentConfigManager).applyRemoteConfig\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/config_manager.go:173\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*agentConfigManager).Apply\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/config_manager.go:159\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:209\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onMessageFuncHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:199\ngithub.com/open-telemetry/opamp-go/client/types.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/types/callbacks.go:162\ngithub.com/open-telemetry/opamp-go/client/internal.(*receivedProcessor).ProcessReceivedMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/receivedprocessor.go:131\ngithub.com/open-telemetry/opamp-go/client/internal.(*wsReceiver).ReceiverLoop\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/wsreceiver.go:57\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runOneCycle\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:243\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runUntilStopped\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:265\ngithub.com/open-telemetry/opamp-go/client/internal.(*ClientCommon).StartConnectAndRun.func1\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/clientcommon.go:197"}
{"level":"error","timestamp":"2024-04-04T11:36:23.737Z","caller":"opamp/server_client.go:216","msg":"failed to apply config","component":"opamp-server-client","error":"failed to reload config: /var/tmp/collector-config.yaml: collector failed to restart: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:\n\n* error decoding 'exporters': error reading configuration for \"prometheus\": 1 error(s) decoding:\n\n* '' has invalid keys: config","stacktrace":"<http://github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler|github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler>\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:216\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onMessageFuncHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:199\ngithub.com/open-telemetry/opamp-go/client/types.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/types/callbacks.go:162\ngithub.com/open-telemetry/opamp-go/client/internal.(*receivedProcessor).ProcessReceivedMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/receivedprocessor.go:131\ngithub.com/open-telemetry/opamp-go/client/internal.(*wsReceiver).ReceiverLoop\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/wsreceiver.go:57\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runOneCycle\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:243\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runUntilStopped\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:265\ngithub.com/open-telemetry/opamp-go/client/internal.(*ClientCommon).StartConnectAndRun.func1\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/clientcommon.go:197"}
s mohamed
04/04/2024, 11:44 AMPrashant Shahi
04/04/2024, 11:52 AMs mohamed
04/04/2024, 12:01 PMapiVersion: v1
data:
otel-collector-config.yaml: "exporters:\n clickhouselogsexporter:\n dsn: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}\n
\ timeout: 10s\n clickhousemetricswrite:\n endpoint: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}\n
\ resource_to_telemetry_conversion:\n enabled: true\n timeout: 15s\n
\ clickhousetraces:\n datasource: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_TRACE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}\n
\ low_cardinal_exception_grouping: ${LOW_CARDINAL_EXCEPTION_GROUPING}\n prometheus:\n
\ config:\n scrape_configs:\n - job_name: \"otel-collector\"\n scrape_interval:
30s\n static_configs:\n - targets: [\"otel-collector:8889\",
\"prometheus-operated:9090\"]\n\nextensions:\n health_check:\n endpoint: 0.0.0.0:13133\n
\ pprof:\n endpoint: localhost:1777\n zpages:\n endpoint: localhost:55679\nprocessors:\n
\ batch:\n send_batch_size: 50000\n timeout: 1s\n k8sattributes:\n extract:\n
\ metadata:\n - k8s.namespace.name\n - k8s.pod.name\n - k8s.pod.uid\n
\ - k8s.pod.start_time\n - k8s.deployment.name\n - k8s.node.name\n
\ filter:\n node_from_env_var: K8S_NODE_NAME\n passthrough: false\n
\ pod_association:\n - sources:\n - from: resource_attribute\n name:
k8s.pod.ip\n - sources:\n - from: resource_attribute\n name: k8s.pod.uid\n
\ - sources:\n - from: connection\n memory_limiter: null\n resourcedetection:\n
\ detectors:\n - env\n - system\n system:\n hostname_sources:\n
\ - dns\n - os\n timeout: 2s\n signozspanmetrics/cumulative:\n dimensions:\n
\ - default: default\n name: service.namespace\n - default: default\n
\ name: deployment.environment\n - name: signoz.collector.id\n dimensions_cache_size:
100000\n latency_histogram_buckets:\n - 100us\n - 1ms\n - 2ms\n -
6ms\n - 10ms\n - 50ms\n - 100ms\n - 250ms\n - 500ms\n - 1000ms\n
\ - 1400ms\n - 2000ms\n - 5s\n - 10s\n - 20s\n - 40s\n - 60s\n
\ metrics_exporter: clickhousemetricswrite\n signozspanmetrics/delta:\n aggregation_temporality:
AGGREGATION_TEMPORALITY_DELTA\n dimensions:\n - default: default\n name:
service.namespace\n - default: default\n name: deployment.environment\n
\ - name: signoz.collector.id\n dimensions_cache_size: 100000\n latency_histogram_buckets:\n
\ - 100us\n - 1ms\n - 2ms\n - 6ms\n - 10ms\n - 50ms\n - 100ms\n
\ - 250ms\n - 500ms\n - 1000ms\n - 1400ms\n - 2000ms\n - 5s\n
\ - 10s\n - 20s\n - 40s\n - 60s\n metrics_exporter: clickhousemetricswrite\nreceivers:\n
\ hostmetrics:\n collection_interval: 30s\n scrapers:\n cpu: {}\n disk:
{}\n filesystem: {}\n load: {}\n memory: {}\n network: {}\n
\ httplogreceiver/heroku:\n endpoint: 0.0.0.0:8081\n source: heroku\n httplogreceiver/json:\n
\ endpoint: 0.0.0.0:8082\n source: json\n jaeger:\n protocols:\n grpc:\n
\ endpoint: 0.0.0.0:14250\n thrift_http:\n endpoint: 0.0.0.0:14268\n
\ otlp:\n protocols:\n grpc:\n endpoint: 0.0.0.0:4317\n max_recv_msg_size_mib:
16\n http:\n endpoint: 0.0.0.0:4318\n otlp/spanmetrics:\n protocols:\n
\ grpc:\n endpoint: localhost:12345\nservice:\n extensions:\n - health_check\n
\ - zpages\n - pprof\n pipelines:\n logs:\n exporters:\n - clickhouselogsexporter\n
\ processors:\n - batch\n receivers:\n - otlp\n - httplogreceiver/heroku\n
\ - httplogreceiver/json\n metrics:\n exporters:\n - clickhousemetricswrite\n
\ processors:\n - batch\n receivers:\n - otlp\n - prometheus\n
\ metrics/internal:\n exporters:\n - clickhousemetricswrite\n processors:\n
\ - resourcedetection\n - k8sattributes\n - batch\n receivers:
\n - hostmetrics\n traces:\n exporters:\n - clickhousetraces\n
\ processors:\n - signozspanmetrics/cumulative\n - signozspanmetrics/delta\n
\ - batch\n receivers:\n - otlp\n - jaeger\n telemetry:\n
\ metrics:\n address: 0.0.0.0:8888"
otel-collector-opamp-config.yaml: 'server_endpoint: "<ws://my-release-signoz-query-service:4320/v1/opamp>"'
kind: ConfigMap
metadata:
annotations:
<http://meta.helm.sh/release-name|meta.helm.sh/release-name>: my-release
<http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: platform
labels:
<http://app.kubernetes.io/component|app.kubernetes.io/component>: otel-collector
<http://app.kubernetes.io/instance|app.kubernetes.io/instance>: my-release
<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
<http://app.kubernetes.io/name|app.kubernetes.io/name>: signoz
<http://app.kubernetes.io/version|app.kubernetes.io/version>: 0.41.1
<http://helm.sh/chart|helm.sh/chart>: signoz-0.37.1
k8slens-edit-resource-version: v1
name: my-release-signoz-otel-collector
namespace: platform
s mohamed
04/04/2024, 12:01 PMPrashant Shahi
04/04/2024, 12:06 PMprometheus: ...
should be under receivers
and not exporters
s mohamed
04/04/2024, 12:21 PMs mohamed
04/04/2024, 12:21 PMs mohamed
04/04/2024, 1:32 PMwarn internal/transaction.go:123 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1712237452481, "target_labels": "{__name__=\"up\", instance=\"prometheus-operated:9090\", job=\"otel-collector\"}"}
s mohamed
04/04/2024, 1:32 PMs mohamed
04/04/2024, 2:30 PMs mohamed
04/04/2024, 5:04 PMPrashant Shahi
04/04/2024, 5:08 PMPrashant Shahi
04/04/2024, 5:08 PMs mohamed
04/04/2024, 5:22 PMs mohamed
04/04/2024, 5:23 PMs mohamed
04/08/2024, 6:50 AMs mohamed
04/08/2024, 7:31 AMs mohamed
04/08/2024, 7:31 AMPrashant Shahi
04/08/2024, 8:25 AMotelCollectorMetrics:
config:
receivers:
prometheus/custom:
config:
scrape_configs:
- job_name: "custom-collector"
scrape_interval: 30s
static_configs:
- targets:
- prometheus-operated.monitoring.svc.cluster.local:9090
service:
pipelines:
metrics:
receivers: [prometheus, prometheus/custom]
Followed by helm upgrade:
helm upgrade my-release -n platform signoz/signoz -f override-config.yaml
s mohamed
04/08/2024, 8:26 AMPrashant Shahi
04/08/2024, 8:26 AMotelCollectorMetrics
instead of otelCollector
as multiple instances of otelCollector will cause redundant data.s mohamed
04/08/2024, 8:26 AM