Hello, I'm trying to configure alertmanager to send notification via mail. As I'm using an internal service in the cluster I dont need to user TLS. Is there any environment variable to disable starttls/ssl?

level=warn ts=2024-07-22T11:30:53.815Z caller=notify.go:718 component=dispatcher receiver=Test integration=email[0] msg="Notify attempt failed, will retry later" attempts=1 err="send STARTTLS command: x509: certificate is valid for localhost, not email-relay-service.my-namespace.svc.cluster.local"

Hey guys we've been using the self hosted docker version of signoz for sometime now and are extremely happy with it! Recently we wanted to move our logs to signoz and migrating the signoz clusters to kubernetes but have been facing several memory issues with the otel collector pods and the cluster running out of memory we have setup 2 instances with 4vCPU and 8GiB of ram as the nodes. The collector is restarting once in a while causing disruption. but facing the below error when it's restarting: [otel.javaagent 2024-05-02 053841:914 +0000] [OkHttp https://signoz-ingest.sentisum.com/...] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export spans. The request could not be executed. Full error message: Remote host terminated the handshake And the pod memory usage:

Hello everyone, I'm facing issue replate to Otel collector it's stuck in crashloopbackoff, when i checked the logs got the following.

{"level":"info","timestamp":"2025-01-08T05:19:57.460Z","caller":"signozcol/collector.go:168","msg":"Restarting collector service"}
{"level":"info","timestamp":"2025-01-08T05:19:57.460Z","caller":"signozcol/collector.go:144","msg":"Shutting down collector service"}
{"level":"info","timestamp":"2025-01-08T05:19:57.460Z","caller":"signozcol/collector.go:154","msg":"Collector service is not running"}
{"level":"info","timestamp":"2025-01-08T05:19:57.461Z","caller":"signozcol/collector.go:103","msg":"Starting collector service"}
{"level":"info","timestamp":"2025-01-08T05:19:57.470Z","caller":"signozcol/collector.go:168","msg":"Restarting collector service"}
{"level":"info","timestamp":"2025-01-08T05:19:57.470Z","caller":"signozcol/collector.go:144","msg":"Shutting down collector service"}
{"level":"info","timestamp":"2025-01-08T05:19:57.470Z","caller":"signozcol/collector.go:152","msg":"Collector service is shut down"}
{"level":"info","timestamp":"2025-01-08T05:19:57.471Z","caller":"signozcol/collector.go:103","msg":"Starting collector service"}
{"level":"error","timestamp":"2025-01-08T05:19:57.478Z","caller":"opamp/server_client.go:268","msg":"Collector failed for restart during rollback","component":"opamp-server-client","error":"failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:\n\n* error decoding 'exporters': error reading configuration for \"clickhousetraces\": 1 error(s) decoding:\n\n* '' has invalid keys: use_new_schema","stacktrace":"github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).reload\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:268\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*agentConfigManager).applyRemoteConfig\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/config_manager.go:173\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*agentConfigManager).Apply\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/config_manager.go:159\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:209\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onMessageFuncHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:199\ngithub.com/open-telemetry/opamp-go/client/types.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/types/callbacks.go:162\ngithub.com/open-telemetry/opamp-go/client/internal.(*receivedProcessor).ProcessReceivedMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/receivedprocessor.go:131\ngithub.com/open-telemetry/opamp-go/client/internal.(*wsReceiver).ReceiverLoop\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/wsreceiver.go:57\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runOneCycle\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:243\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runUntilStopped\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:265\ngithub.com/open-telemetry/opamp-go/client/internal.(*ClientCommon).StartConnectAndRun.func1\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/clientcommon.go:197"}
{"level":"error","timestamp":"2025-01-08T05:19:57.478Z","caller":"opamp/server_client.go:216","msg":"failed to apply config","component":"opamp-server-client","error":"failed to reload config: /var/tmp/collector-config.yaml: collector failed to restart: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:\n\n* error decoding 'exporters': error reading configuration for \"clickhousetraces\": 1 error(s) decoding:\n\n* '' has invalid keys: use_new_schema","stacktrace":"github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:216\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onMessageFuncHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:199\ngithub.com/open-telemetry/opamp-go/client/types.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/types/callbacks.go:162\ngithub.com/open-telemetry/opamp-go/client/internal.(*receivedProcessor).ProcessReceivedMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/receivedprocessor.go:131\ngithub.com/open-telemetry/opamp-go/client/internal.(*wsReceiver).ReceiverLoop\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/wsreceiver.go:57\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runOneCycle\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:243\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runUntilStopped\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:265\ngithub.com/open-telemetry/opamp-go/client/internal.(*ClientCommon).StartConnectAndRun.func1\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/clientcommon.go:197"}

when i checked with GPT got the following analysis Invalid Key in Configuration: • The configuration file (

collector-config.yaml

) contains an invalid key

use_new_schema

under the

clickhousetraces

exporter. This key is not supported by the exporter. FYI using SigNoz 0.56, deployed in Kubernetes AKS cluster with helm chart. also tried to changed the value use_new_schema to "true" but still it's not shipping the logs

clickhouselogsexporter:
        dsn: tcp://${CLICKHOUSE_USER}:${CLICKHOUSE_PASSWORD}@${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/${CLICKHOUSE_LOG_DATABASE}
        timeout: 10s
        use_new_schema: true

Please any one help me here to troubleshoot the otelcollector, and take it back to working condition.

Hi, im just trying to upgrade to SigNoz 0.47 from 0.46.1 and migration is crashing with the following error: {"level":"error","timestamp":"2024-06-05T144740.007Z","caller":"migrationmanager/manager.go:81","msg":"Failed to run migrations for migrator","component":"migrationmanager","migrator":"logs","error":"failed to create database, err: code: 159, message: Distributed DDL task /clickhouse/task_queue/ddl/query-0000000195 is not finished on 1 of 1 hosts (0 of them are currently executing the task, 0 are inactive). They are going to execute the query in background. Was waiting for 180.882823243 seconds, which is longer than distributed_ddl_task_timeout","stacktrace":"github.com/SigNoz/signoz-otel-collector/migrationmanager.(*MigrationManager).Migrate\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/migrationmanager/manager.go81\nmain.main\n\t/home/runner/work/signoz otel collector/signoz otel collector/cmd/signozschemamigrator/migrate.go126\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.10/x64/src/runtime/proc.go:267"} {"level":"fatal","timestamp":"2024-06-05T144740.007Z","caller":"signozschemamigrator/migrate.go:128","msg":"Failed to run migrations","component":"migrate cli","error":"failed to create database, err: code: 159, message: Distributed DDL task /clickhouse/task_queue/ddl/query-0000000195 is not finished on 1 of 1 hosts (0 of them are currently executing the task, 0 are inactive). They are going to execute the query in background. Was waiting for 180.882823243 seconds, which is longer than distributed_ddl_task_timeout","stacktrace":"main.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go128\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.10/x64/src/runtime/proc.go267"}

I am trying to use variable in Signoz Frontend Dashboard and while making a panel, I want to append

%

to the variable name in the where clause. I have tried

Pod like {{deployment}}%

but it’s not working. Any suggestions

This message was deleted.

Hi can someone guide me on how to make this environment selection work? https://signoz.io/blog/community-update-23/#ability-to-filter-by-deployment-environment-in-service-maps

This message was deleted.

This message was deleted.

Hi team I tried sending kubernetes pods logs using promtail to my signoz hosted in a vm but getting the below logs