Currently I'm trying to set mTLS between two OTEL collectors one is running on docker and another on...
a

Anurag Vishwakarma

over 1 year ago
Currently I'm trying to set mTLS between two OTEL collectors one is running on docker and another one is running on Kubernetes. While setting up K8s I'm getting this following error @Prashant Shahi
2024-06-18T06:49:16.787Z	info	exporterhelper/retry_sender.go:177	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\"", "interval": "13.402682275s"}
2024-06-18T06:49:30.191Z	info	exporterhelper/retry_sender.go:177	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority\"", "interval": "20.649604662s"}
But it works well with the other Otel collector. Here is the command which i'm using to setup K8s-Infra. Also I'm using Secrets for TLS certs in k8s.
helm --namespace k8s-infra install my-release signoz/k8s-infra \
  --set otelCollectorEndpoint=<https://example.dev:4317> \
  --set otelInsecure=false \
  --set global.clusterName=k04-admin@kubernetes \
  --set tls.enabled=true \
  --set tls.secretName=ss-dev-new \
  --set tls.caCert=rootCA.crt \
  --set tls.tlsCert=monitor.crt \
  --set tls.tlsKey=monitor.key
Hi everyone. I'm evaluating SigNoz for observability and APM on a on-premise, no interne, infrastru...
g

Gil

8 months ago
Hi everyone. I'm evaluating SigNoz for observability and APM on a on-premise, no interne, infrastructure in production. Now I want to implement everything on a sandbox VM. After clickhouse install I did clickhouse-client -q "SELECT version()" -> 25.3.2.39 Then
clickhouse-client -q "CREATE USER signoz IDENTIFIED BY 'signoz';" 
clickhouse-client -q "GRANT ALL ON signoz_traces.* TO signoz;"
clickhouse-client -q "GRANT ALL ON signoz_logs.* TO signoz;"
clickhouse-client -q "GRANT ALL ON signoz_metrics.* TO signoz;"

clickhouse-client -q "CREATE DATABASE IF NOT EXISTS signoz_traces;"
clickhouse-client -q "CREATE DATABASE IF NOT EXISTS signoz_logs;"
clickhouse-client -q "CREATE DATABASE IF NOT EXISTS signoz_metrics;"

clickhouse-client --user signoz --password signoz
Connection ok FOr SigNoz itself, once installed (following the documentation)
sudo bash -c 'cat <<EOF > /opt/signoz/conf/systemd.env
SIGNOZ_INSTRUMENTATION_LOGS_LEVEL=info
INVITE_EMAIL_TEMPLATE=/opt/signoz/templates/invitation_email_template.html
SIGNOZ_SQLSTORE_SQLITE_PATH=/var/lib/signoz/signoz.db
SIGNOZ_WEB_ENABLED=true
SIGNOZ_WEB_DIRECTORY=/opt/signoz/web
SIGNOZ_JWT_SECRET=secret
SIGNOZ_ALERTMANAGER_PROVIDER=signoz
SIGNOZ_TELEMETRYSTORE_PROVIDER=clickhouse
SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=<tcp://localhost:9000?password=signoz>
EOF'
sudo getent passwd signoz >/dev/null || sudo useradd --system --home /opt/signoz --no-create-home --user-group --shell /sbin/nologin signoz
sudo chown -R signoz:signoz /var/lib/signoz
sudo chown -R signoz:signoz /opt/signoz
sudo bash -c 'cat <<EOF > /etc/systemd/system/signoz.service
[Unit]
Description=SigNoz
Documentation=<https://signoz.io/docs>
After=clickhouse-server.service

[Service]
User=signoz
Group=signoz
Type=simple
KillMode=mixed
Restart=on-failure
WorkingDirectory=/opt/signoz
EnvironmentFile=/opt/signoz/conf/systemd.env
ExecStart=/opt/signoz/bin/signoz --config=/opt/signoz/conf/prometheus.yml --use-logs-new-schema=true --use-trace-new-schema=true

[Install]
WantedBy=multi-user.target
EOF'
I have copy pasted on the dashboard the template for hostmetrics but I got.
code: 516, message: default: Authentication failed: password is incorrect, or there is no user with such name. If you use ClickHouse Cloud, the password can be reset at <https://clickhouse.cloud/> on the settings page for the corresponding service. If you have installed ClickHouse and forgot password you can reset it in the configuration file. The password for default user is typically located at /etc/clickhouse-server/users.d/default-password.xml and deleting this file will reset the password. See also /etc/clickhouse-server/users.xml on the server where ClickHouse is installed.
I may made a typo somewhere, but I think i have followed the how-to. ANy idea here ?
(*Signoz Docker Standalone deployment*) Hi All The `signoz-otel-collector` keeps restarting. Does an...
m

Murat Kuru

over 1 year ago
(Signoz Docker Standalone deployment) Hi All The
signoz-otel-collector
keeps restarting. Does anyone know what "*Error creating ClickHouse client: database should be set in ClickHouse DSN*" means? The ClickHouse database is up and running, and I can connect and run queries from the command line. Does anyone have insights into what might be causing this error or how to resolve it? Any help would be greatly appreciated.
2024-06-07T18:14:04.037Z        info    service@v0.88.0/telemetry.go:84 Setting up own telemetry...
2024-06-07T18:14:04.037Z        info    service@v0.88.0/telemetry.go:201        Serving Prometheus metrics      {"address": "0.0.0.0:8888", "level": "Basic"}
2024-06-07T18:14:04.038Z        info    exporter@v0.88.0/exporter.go:275        Stability level of component is undefined       {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite"}
2024/06/07 18:14:04 Error creating clickhouse client: database should be set in ClickHouse DSN
{"level":"info","timestamp":"2024-06-07T18:15:05.380Z","logger":"dynamic-config","caller":"opamp/config_manager.go:89","msg":"Added instance id to config file","component":"opamp-server-client","instance_id":"a2430171-01f1-4b94-b4b8-9ac406e7a368"}
{"level":"info","timestamp":"2024-06-07T18:15:05.380Z","caller":"service/service.go:69","msg":"Starting service"}
{"level":"info","timestamp":"2024-06-07T18:15:05.380Z","caller":"opamp/server_client.go:171","msg":"Waiting for initial remote config","component":"opamp-server-client"}
{"level":"info","timestamp":"2024-06-07T18:15:05.382Z","caller":"opamp/server_client.go:127","msg":"Connected to the server.","component":"opamp-server-client"}
{"level":"info","timestamp":"2024-06-07T18:15:05.391Z","logger":"agent-config-manager","caller":"opamp/config_manager.go:172","msg":"Config has changed, reloading","path":"/var/tmp/collector-config.yaml"}
{"level":"info","timestamp":"2024-06-07T18:15:05.391Z","caller":"signozcol/collector.go:168","msg":"Restarting collector service"}
{"level":"info","timestamp":"2024-06-07T18:15:05.391Z","caller":"signozcol/collector.go:144","msg":"Shutting down collector service"}
{"level":"info","timestamp":"2024-06-07T18:15:05.391Z","caller":"signozcol/collector.go:154","msg":"Collector service is not running"}
{"level":"info","timestamp":"2024-06-07T18:15:05.392Z","caller":"signozcol/collector.go:103","msg":"Starting collector service"}
2024-06-07T18:15:05.403Z        info    service@v0.88.0/telemetry.go:84 Setting up own telemetry...
2024-06-07T18:15:05.403Z        info    service@v0.88.0/telemetry.go:201        Serving Prometheus metrics      {"address": "0.0.0.0:8888", "level": "Basic"}
2024-06-07T18:15:05.405Z        info    exporter@v0.88.0/exporter.go:275        Stability level of component is undefined       {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite"}
2024/06/07 18:15:05 Error creating clickhouse client: database should be set in ClickHouse DSN