https://signoz.io logo
Join Slack
Powered by
Currently I'm trying to set mTLS between two OTEL ...
# support
a
Currently I'm trying to set mTLS between two OTEL collectors one is running on docker and another one is running on Kubernetes. While setting up K8s I'm getting this following error @Prashant Shahi
Copy code
2024-06-18T06:49:16.787Z	info	exporterhelper/retry_sender.go:177	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\"", "interval": "13.402682275s"}
2024-06-18T06:49:30.191Z	info	exporterhelper/retry_sender.go:177	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\"", "interval": "20.649604662s"}
But it works well with the other Otel collector. Here is the command which i'm using to setup K8s-Infra. Also I'm using Secrets for TLS certs in k8s.
Copy code
helm --namespace k8s-infra install my-release signoz/k8s-infra \
  --set otelCollectorEndpoint=<https://example.dev:4317> \
  --set otelInsecure=false \
  --set global.clusterName=k04-admin@kubernetes \
  --set tls.enabled=true \
  --set tls.secretName=ss-dev-new \
  --set tls.caCert=rootCA.crt \
  --set tls.tlsCert=monitor.crt \
  --set tls.tlsKey=monitor.key
@Prashant Shahi @nitya-signoz I'm getting this error.
Copy code
": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\"", "interval": "32.884282746s"}
2024-06-19T10:29:15.804Z	info	exporterhelper/retry_sender.go:177	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\"", "interval": "12.601209538s"}
2024-06-19T10:29:23.637Z	error	exporterhelper/queue_sender.go:93	Exporting failed. No more retries left. Dropping data.	{"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "max elapsed time expired rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\"", "dropped_items": 818}
The certs works well if i use it with other otel collectors to send data to signoz.
@Prashant Shahi Waiting for your reply!
p
Copy code
--set tls.enabled=true \
  --set tls.secretName=ss-dev-new \
  --set tls.caCert=rootCA.crt \
  --set tls.tlsCert=monitor.crt \
  --set tls.tlsKey=monitor.key
^ I don't think these are valid configuration.
a
ok any guide on this ? what should i do next ?
I tried via helm chart values too got same error.
p
Try using these instead:
Copy code
global:
  clusterName: k04-admin
otelCollectorEndpoint: <https://example.dev:4317>
otelInsecure: false
otelTlsSecrets:
  enabled: true
  existingSecretName: ss-dev-new
  certificate: |
    <INCLUDE_CERTIFICATE_HERE>
  key: |
    <INCLUDE_PRIVATE_KEY_HERE>
  ca: |
    <INCLUDE_CA_HERE>
For existing secret, you will need to add certificate data with the following keys.
Copy code
cert.pem: ...
key.pem: ...
ca.pem: ... # optional
a
Now I tried to convert 
crt
& 
key
in 
pem
format and i passed this .... same issue 
transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority
Copy code
helm --namespace otel install my-release signoz/k8s-infra \
  --set otelCollectorEndpoint=example:4317 \
  --set otelInsecure=false \
  --set global.clusterName=k04-admin \
  --set otelTlsSecrets.enabled=true \
  --set otelTlsSecrets.existingSecretName=otel-key
Screenshot 2024-06-20 at 1.15.10 PM.png
p
sorry, I am caught up with things at hand right now. Will try this out and get back.
also, how did you generate the certificates?
a
openssl
The certificates works fine as i said earlier with other otel collectors
p
@Anurag Vishwakarma The CA might not mounted at all. Can you try setting 
<http://otelTlsSecrets.ca|otelTlsSecrets.ca>
to non-empty value i.e. set it to any strings. Try the updated values (include all of these): https://signoz-community.slack.com/archives/C01HWQ1R0BC/p1718863800775029?thread_ts=1718694490.175009&amp;cid=C01HWQ1R0BC
152 Views
Open in Slack
PreviousNext
Powered by