This message was deleted.

This message was deleted.

This message was deleted.

This message was deleted.

This message was deleted.

hello one more im really look for some quick help on I am trying to run signoz on an ec2 instance in an aws account. setting up the service on the instance with isntall script has been working well enough. I am trying to monitor my apps/services in some ecs clusters. There are two clusters 1. is keycloak, and their new v26 build has an otel preview feature that works pretty well for tracing/metrics and uses grpc by default 2. our app stack has some sorting out to do before grpc, but we can get it to work with http endpoint. when i combine the collector to an ecs sidecar I can get it working with keycloak and grpc but I can't do the same for our app with http. Im using separate sidecar collector files swapping to port 4318 etc. seeing errors like

2024-12-04T20:32:09.836Z info exporterhelper/retry_sender.go:177 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\"", "interval": "6.306292489s"}

and can't track down why in thread is my yaml config file for the signoz sidecar for that app and below that are the variables for the cont def for our app I am trying to point it to. What am I doing wrong?

Hello. How can I add an Environment and view services based on the environment selected later on?

Hi team , I am setting up kafka reciever and clickhouse exporter via opentelemetry collector taking reference through https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main. Everything set up but while using kafka reciever it stucked at logs showing that —-> 2024-08-25T041734.167+0530  info  kafkareceiver/kafka_receiver.go:474  Starting consumer group  {"kind": "receiver", "name": "kafka", "data_type": "traces", "partition": 0} Below is my config yaml receivers: kafka: protocol_version: 2.0.0 topic: mykafka brokers:"localhost:9092" resolve_canonical_bootstrap_servers_only: true metadata: retry: max: 10 backoff: 5s otlp: protocols: http: endpoint: 0.0.0.0:4318 grpc: endpoint: 0.0.0.0:4317 kafka/logs: topic: mykafka session_timeout: 45s heartbeat_interval: 15s encoding: direct brokers:"localhost:9092" initial_offset: earliest metadata: retry: max: 10 backoff: 5s processors: batch: timeout: 5s send_batch_size: 100000 resource: attributes🔑 test.key value: "test-value" action: insert exporters: debug: verbosity: normal clickhouse: endpoint: tcp://127.0.0.1:9000?dial_timeout=10s database: otel async_insert: true ttl: 72h compress: lz4 create_schema: true logs_table_name: otel_logs traces_table_name: otel_traces metrics_table_name: otel_metrics timeout: 5s retry_on_failure: enabled: true initial_interval: 5s max_interval: 30s max_elapsed_time: 300s extensions: health_check: {} zpages: {} service: extensions: [zpages, health_check] pipelines: logs: receivers: [ kafka ] processors: [ batch ] exporters: [ clickhouse ] traces: receivers: [ kafka ] processors: [ batch ] exporters: [ clickhouse ] metrics: receivers: [ kafka ] processors: [ batch ] exporters: [ clickhouse ]

Hi. We tried to upgrade signoz on k8s from chart 0.43.0 to 0.45.2 using helm and the schema-migrator-upgrade pod failed to execute migrations on the database (the clickhouse database pod was recreated while the migrations were in progress). The helm upgrade:

Error: UPGRADE FAILED: post-upgrade hooks failed: 1 error occurred:
        * timed out waiting for the condition

While the schema-migrator-upgrade pod log:

2024-07-09T07:53:36.846408300Z {"level":"error","timestamp":"2024-07-09T07:53:36.846Z","caller":"migrationmanager/manager.go:81","msg":"Failed to run migrations for migrator","component":"migrationmanager","migrator":"logs","error":"clickhouse migrate failed to run, error: Dirty database version 12. Fix and force version.","stacktrace":"<http://github.com/SigNoz/signoz-otel-collector/migrationmanager.(*MigrationManager).Migrate|github.com/SigNoz/signoz-otel-collector/migrationmanager.(*MigrationManager).Migrate>\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/migrationmanager/manager.go:81\nmain.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:126\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.11/x64/src/runtime/proc.go:267"}
2024-07-09T07:53:36.846517200Z {"level":"fatal","timestamp":"2024-07-09T07:53:36.846Z","caller":"signozschemamigrator/migrate.go:128","msg":"Failed to run migrations","component":"migrate cli","error":"clickhouse migrate failed to run, error: Dirty database version 12. Fix and force version.","stacktrace":"main.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:128\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.11/x64/src/runtime/proc.go:267"}

Trying to delete the signoz_traces.schema_migrations/signoz_metrics.schema_migrations/signoz_logs.schema_migrations (as suggested in https://community-chat.signoz.io/t/16422187/i-m-trying-to-upgrade-a-cluster-i-installed-yesterday-from-v ) and upgrading again didn't help and we are stuck that both new versions of signoz otel-collector and otel-collector-metrics pods didn't start (v102.2; we still have the previous version 0.88.26 running). The query:

chi-signoz13-clickhouse-cluster-0-0-0.chi-signoz13-clickhouse-cluster-0-0.platform13.svc.cluster.local :) select * from schema_migrations

SELECT *
FROM schema_migrations

Query id: 71233133-bedd-4e08-916e-c640d432c325

┌─version─┬─dirty─┬────────────sequence─┐
│      12 │     1 │ 1720511516687366500 │
└─────────┴───────┴─────────────────────┘
┌─version─┬─dirty─┬────────────sequence─┐
│       1 │     1 │ 1720511509536275800 │
│       1 │     0 │ 1720511509925968100 │
│       2 │     1 │ 1720511509927867300 │
│       2 │     0 │ 1720511509984443400 │
│       3 │     1 │ 1720511509986195000 │
│       3 │     0 │ 1720511510157772100 │
│       4 │     1 │ 1720511510159524100 │
│       4 │     0 │ 1720511510216625700 │
│       5 │     1 │ 1720511510218281300 │
│       5 │     0 │ 1720511510769604900 │
│       6 │     1 │ 1720511510771235700 │
│       6 │     0 │ 1720511510885832700 │
│       7 │     1 │ 1720511510887398400 │
│       7 │     0 │ 1720511511056791700 │
│       8 │     1 │ 1720511511058826200 │
│       8 │     0 │ 1720511511334694000 │
│       9 │     1 │ 1720511511336217600 │
│       9 │     0 │ 1720511511447635300 │
│      10 │     1 │ 1720511511449157700 │
│      10 │     0 │ 1720511516075765100 │
│      11 │     1 │ 1720511516077423500 │
└─────────┴───────┴─────────────────────┘
┌─version─┬─dirty─┬────────────sequence─┐
│      11 │     0 │ 1720511516685619200 │
└─────────┴───────┴─────────────────────┘

23 rows in set. Elapsed: 0.002 sec. 

chi-signoz13-clickhouse-cluster-0-0-0.chi-signoz13-clickhouse-cluster-0-0.platform13.svc.cluster.local :

Please help.

How to fix this @nitya-signoz

2024-06-24T09:46:10.735Z	error	exporterhelper/queue_sender.go:101	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "not retryable error: Permanent error: rpc error: code = ResourceExhausted desc = grpc: received message after decompression larger than max (4350965 vs. 4194304)", "dropped_items": 11000}