Hello All, After sending opentelemetry logs to signoz, TPS has reduced significantly (around 30-50%) how can I keep the TPS same?

Hello signoz team, Recently upgraded from v0.15 -> v0.20.2. Running via docker-compose. After trying to remove alerts that were behaving weirdly post update via the alert dashboard, signoz crashed and on reboot alertmanager service fails to pass startup with docker-compose up. On the rare chance it does pass, the frontend is mostly broken and displays not found on all pages after trying to mess around with alerts again. I believe this is a problem with how some alerts were updated via v0.19 upgrade script. (Running this script again gives a seg fault now). I am trying to find a way to make the service usable again, I am ok with manually deleting the current alerts if thats the problem but am unsure where they are stored and how to do so. Thanks in advance :). Current version info:

SigNoz version   : v0.20.2
Commit SHA-1     : 84c4668
Commit timestamp : 2023-06-09T08:52:13Z
Branch           : HEAD
Go version       : go1.18.10

Launch logs from alertmanager service:

2023-06-21T05:34:22.453Z        INFO    rules/thresholdRule.go:754      rule:Runner Error   alerts found: 0
2023-06-21T05:34:22.453Z        INFO    rules/thresholdRule.go:309      msg:sending alerts       rule:Runner Error
2023-06-21T05:34:23.629Z        DEBUG   rules/ruleTask.go:297   msg:%!(EXTRA string=rule task eval started, string=      name:, string=18-groupname, string=     start time:, time.Time=2023-06-21 05:34:23.629831765 +0000 UTC)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xfa9cc1]

goroutine 429 [running]:
<http://go.signoz.io/signoz/pkg/query-service/app/queryBuilder.expressionToQuery(0xc000fbe630|go.signoz.io/signoz/pkg/query-service/app/queryBuilder.expressionToQuery(0xc000fbe630>, 0x5?, 0x0)
        /go/src/github.com/signoz/signoz/pkg/query-service/app/queryBuilder/query_builder.go:82 +0x41
<http://go.signoz.io/signoz/pkg/query-service/app/queryBuilder.(*QueryBuilder).PrepareQueries(0xc0004e4dc8|go.signoz.io/signoz/pkg/query-service/app/queryBuilder.(*QueryBuilder).PrepareQueries(0xc0004e4dc8>, 0xc000fbe630, {0x0, 0x0, 0x0?})
        /go/src/github.com/signoz/signoz/pkg/query-service/app/queryBuilder/query_builder.go:178 +0x5d3
<http://go.signoz.io/signoz/pkg/query-service/rules.(*ThresholdRule).prepareBuilderQueries(0xc0004402d0|go.signoz.io/signoz/pkg/query-service/rules.(*ThresholdRule).prepareBuilderQueries(0xc0004402d0>, {0x1916378?, 0xc000fbe2d0?, 0x0?})
        /go/src/github.com/signoz/signoz/pkg/query-service/rules/thresholdRule.go:554 +0x45
<http://go.signoz.io/signoz/pkg/query-service/rules.(*ThresholdRule).buildAndRunQuery(0xc0004402d0|go.signoz.io/signoz/pkg/query-service/rules.(*ThresholdRule).buildAndRunQuery(0xc0004402d0>, {0x1916378, 0xc000fbe5d0}, {0xc0008e75f0?, 0x4588e9?, 0x0?}, {0x191f050, 0xc000ffe2c0})
        /go/src/github.com/signoz/signoz/pkg/query-service/rules/thresholdRule.go:615 +0x11f
<http://go.signoz.io/signoz/pkg/query-service/rules.(*ThresholdRule).Eval(0xc0004402d0|go.signoz.io/signoz/pkg/query-service/rules.(*ThresholdRule).Eval(0xc0004402d0>, {0x1916378, 0xc000fbe5d0}, {0x1318460?, 0xc00079ec90?, 0x0?}, 0x0?)
        /go/src/github.com/signoz/signoz/pkg/query-service/rules/thresholdRule.go:669 +0x97
<http://go.signoz.io/signoz/pkg/query-service/rules.(*RuleTask).Eval.func1({0x1916378|go.signoz.io/signoz/pkg/query-service/rules.(*RuleTask).Eval.func1({0x1916378>?, 0xc000fbe5a0?}, {0x0?, 0xc00069e3d0?, 0x0?}, 0xc0000418c0, 0x4?, {0x1923620, 0xc0004402d0})
        /go/src/github.com/signoz/signoz/pkg/query-service/rules/ruleTask.go:321 +0x22f
<http://go.signoz.io/signoz/pkg/query-service/rules.(*RuleTask).Eval(0xc0000418c0|go.signoz.io/signoz/pkg/query-service/rules.(*RuleTask).Eval(0xc0000418c0>, {0x1916378, 0xc000fbe5a0}, {0xc000794da0?, 0x46a6b9?, 0x0?})
        /go/src/github.com/signoz/signoz/pkg/query-service/rules/ruleTask.go:338 +0x24a
<http://go.signoz.io/signoz/pkg/query-service/rules.(*RuleTask).Run.func1()|go.signoz.io/signoz/pkg/query-service/rules.(*RuleTask).Run.func1()>
        /go/src/github.com/signoz/signoz/pkg/query-service/rules/ruleTask.go:114 +0x87
<http://go.signoz.io/signoz/pkg/query-service/rules.(*RuleTask).Run|go.signoz.io/signoz/pkg/query-service/rules.(*RuleTask).Run>(0xc0000418c0, {0x1916308, 0xc000048018})
        /go/src/github.com/signoz/signoz/pkg/query-service/rules/ruleTask.go:127 +0x4a9
<http://go.signoz.io/signoz/pkg/query-service/rules.(*Manager).addTask.func1()|go.signoz.io/signoz/pkg/query-service/rules.(*Manager).addTask.func1()>
        /go/src/github.com/signoz/signoz/pkg/query-service/rules/manager.go:482 +0x5a
created by <http://go.signoz.io/signoz/pkg/query-service/rules.(*Manager).addTask|go.signoz.io/signoz/pkg/query-service/rules.(*Manager).addTask>
        /go/src/github.com/signoz/signoz/pkg/query-service/rules/manager.go:477 +0x425
2023-06-21T05:34:24.438Z        INFO    version/version.go:43

Hello. How can I filter services in the services tab by tag? I see only

signoz.collector.id

Hello everybody. I still have big trouble with slow log messages in signoz (k8s cluster, go application, logrus to console). Is there anything that I can do to speed up things? Should i send the data from logrus directly to open telemetry? Is there any example for this? Where/how do I get the data in the website?

I am using signoz on k8s. This is a log message that arrives at the web ui. "Log Details -> Json". Q: How and where do I extract the fields

level

and

something

from the body?

{
  "timestamp": 1687410330621216300,
...
  "severity_text": "",
  "severity_number": 0,
  "body": "{\"level\":\"info\",\"message\":\"cleaned 0 job(s)\",\"something\":\"foobar\"}",
  "k8s_cluster_name": "",
  "k8s_container_name": "chart",
...
  "log_file_path": "/var/log/pods/foobar/chart/0.log",
  "log_iostream": "stdout",
  "logtag": "F",
  "time": "2023-06-22T07:05:30.621216224+02:00"
}

Hi Team, Getting the following error while trying to upgrade helm

Error: UPGRADE FAILED: unable to build kubernetes objects from current release manifest: resource mapping not found for name: "my-release-signoz-frontend" namespace: "platform" from "": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta2"
ensure CRDs are installed first

Current K8s version is 1.26. I see that autoscaling/v2beta2 is not supported in 1.26, but how can i override this.

Hello - I'm looking to evaluate Signoz with the OpenTelemetry collector, but I'm a bit confused. I'm running

otelcol

locally on my host, SigNoz is running via Docker, and I have applications running locally on the host as well. My applications are able to collect and send data to otelcol successfully, I can see the data being written when a file exporter is configured. Now I'd like otelcol to send data to Signoz. However, I've seen a lot of conflicting guides, and most produces an error. One example I've been trying to work with is here: https://signoz.io/blog/opentelemetry-collector-complete-guide/ However, attempting to configure these

clickhousetraces

exporters always results in an error from otelcol when attempting to start of:

error decoding 'exporters': unknown type: "clickhousetraces" for id: "clickhousetraces" (valid values: [opencensus zipkin file jaeger otlphttp kafka prometheus prometheusremotewrite logging otlp])

Interestingly, with that SAME URL, we see two other exporters mentioned that are not included later on,

kafka/traces

and

kafka/metrics

. When I attempt to configure these I receive the error that effectively nothing is running on port 9092, and I can confirm the local docker configuration doesn't expose anything on port 9092 and I'm not entirely sure if that configuration even includes kafka anything: https://github.com/SigNoz/signoz/blob/develop/deploy/docker/clickhouse-setup/docker-compose.yaml I feel like this situation should be fairly straightforward - what should my exporter(s) be configured as when using otelcol locally and the Signoz local docker instance?

hey folks quick q, how do i parse out the logging level of docker container logs as severitytext? Tried to look online but unfortunately didnt find much

Hey team, running into some issues with a new install of SigNoz (helm chart v0.16.2) on a self-hosted K8 cluster (running on EC2 in AWS and managed by Rancher). I'm able to get the following pods up in the

platform

name space:

kubectl get pods
                                            
NAME                                               READY   STATUS     RESTARTS        AGE
chi-signoz-clickhouse-cluster-0-0-0                2/2     Running    0               4h17m
signoz-alertmanager-0                              0/2     Init:0/2   0               4h18m
signoz-clickhouse-operator-7b55d5666-2jh65         3/3     Running    0               4h18m
signoz-frontend-8644f95b4c-hbs2g                   0/2     Init:0/2   0               4h18m
signoz-k8s-infra-otel-agent-87h28                  2/2     Running    0               4h18m
signoz-k8s-infra-otel-agent-fdkpd                  2/2     Running    0               4h18m
signoz-k8s-infra-otel-agent-mpn8s                  2/2     Running    0               4h18m
signoz-k8s-infra-otel-agent-rfc4x                  2/2     Running    0               4h18m
signoz-k8s-infra-otel-deployment-5f4b455cd-qjp9m   2/2     Running    2 (4h17m ago)   4h18m
signoz-otel-collector-66fb65544c-5cl6j             0/1     Init:0/1   0               24m
signoz-otel-collector-bdc5b7c66-j84md              0/2     Init:0/2   0               4h18m
signoz-otel-collector-metrics-5657769c49-b7w5l     0/1     Init:0/1   0               14m
signoz-query-service-0                             0/2     Init:0/2   0               4h18m
signoz-zookeeper-0                                 2/2     Running    0               4h18m

Peaking into the init containers of

signoz-otel-collector-bdc5b7c66-j84md

, I see the following error:

kubectl logs signoz-otel-collector-metrics-5657769c49-vbl65 -c signoz-otel-collector-metrics-init

wget: error getting response: Connection reset by peer
waiting for clickhouseDB

Same for

signoz-otel-collector-metrics-5657769c49-b7w5l

:

kubectl logs signoz-otel-collector-metrics-5657769c49-b7w5l -c signoz-otel-collector-metrics-init

wget: error getting response: Connection reset by peer
waiting for clickhouseDB

And for the frontend pod `signoz-frontend-8644f95b4c-hbs2g`:

kubectl logs signoz-frontend-8644f95b4c-hbs2g -c signoz-frontend-init

wget: can't connect to remote host (<IP_ADDRESS_HERE>): Connection refused
waiting for query-service

Query service pod

signoz-query-service-0

has the same error as otel collector:

kubectl logs signoz-query-service-0 -c signoz-query-service-init

wget: error getting response: Connection reset by peer
waiting for clickhouseDB

Any insight is appreciated, thank you!

Hello Team, Unable to filter application under services. "Getting no resource attribute to filter please refer docs to send attribute". As per my understanding should be able to filter based on service.namespace and deployment.environment dimensions by default. Any insights?

According to this post, after implementing OpenTelemetry, I can spin up the service with:

node tracing.js

From this post, the custom-server method will remove important optimizations, that's why I am wondering if there is a way to implement open telemetry without this custom-server? Because for production, the app is running with:

next build
next start -p $PORT

How to build the app and use tracing.js as an entrypoint with the build version? For node-js, I can use the following code in node.js

node -r tracing.js index.js

can I achieve a similar goal with NextJS?

Hi, deployed signoz on openshift 4.10 with the kubernetes helm chart, can't seem to get logs, getting errors on the otel agent: 2023-06-25T085432.640Z warn fileconsumer/file.go:63 no files match the configured include patterns {"kind": "receiver", "name": "filelog/k8s", "data_type": "logs", "component": "fileconsumer", "include": ["/var/log/pods/*/*/*.log", "/var/log/pods/*/*.log"], "exclude": []}

Hey how do I combine two clickhouse queries

Can signoz be setup to external clickhouse without cluster mode ?

how to configure istio metrics and trancing for SigNoz, today I use prometheus, kiali and jaeger

Hi friends,

By chance, has anyone managed to resolve this type of problem (when the cluster is shut down suddenly)?

image.png

hello, is anyone able to help me troubleshoot missing Otel data? here is the output from the troubleshoot pod:

~ » kubectl -n telemetry run troubleshoot --image=signoz/troubleshoot \                                                         ~
  --restart='Never' -i --tty --rm --command -- ./troubleshoot checkEndpoint \
  --endpoint=signoz-otel-collector.telemetry.svc.cluster.local:4317
2023-06-26T20:12:26.069Z	INFO	troubleshoot/main.go:28	STARTING!
2023-06-26T20:12:26.070Z	INFO	checkEndpoint/checkEndpoint.go:41	checking reachability of SigNoz endpoint
2023-06-26T20:12:26.093Z	INFO	troubleshoot/main.go:46	Successfully sent sample data to signoz ...
pod "troubleshoot" deleted
~ » kubectl -n telemetry run troubleshoot --image=signoz/troubleshoot \                                                         ~
  --restart='Never' -i --tty --rm --command -- ./troubleshoot checkEndpoint \
  --endpoint=<http://dev-otel.olympiafinancial.com:4317|dev-otel.olympiafinancial.com:4317>
2023-06-26T20:12:32.731Z	INFO	troubleshoot/main.go:28	STARTING!
2023-06-26T20:12:32.731Z	INFO	checkEndpoint/checkEndpoint.go:41	checking reachability of SigNoz endpoint
2023-06-26T20:12:32.739Z	INFO	troubleshoot/main.go:46	Successfully sent sample data to signoz ...
pod "troubleshoot" deleted

As you can see there are no errors with either the internal or external DNS names, yet nothing is going into Signoz

Hi guys, I have a local hosted set up that's reading a log file with Otel Collector, the file is being read, I actually was able to see data in the Logs section of the UI, but it disappeared from the Logs Table, I tried to run a query naming the condition and the fields, but nothing comes up, the log file is being refreshed every 24 hoursm, my search filter is like:

{survey!=""} | {checkouts, orders}

The version I'm using is: v0.21.0 am I missing something or doing something wrong with it?

can someone point me where to get the API access token?

continuing to struggle getting all of the telemetry to come across. in fact, there is only one method that does produce telemetry in the backend.

I am unable to run the signoz docker setup after cleaning old version images, container, network and volumes for docker via docker prune command Commit on git

commit 2ee7817685cc921ffe295db3388bc1e370ee3d15 (HEAD -> main, origin/main)
Merge: 4a467435 803cfd1a
Author: Ankit Nayan <ankit@signoz.io>
Date:   Thu Jun 22 00:56:11 2023 +0530

    Merge pull request #2955 from SigNoz/release/v0.21.0
    
    Release/v0.21.0

Error while running

docker-compose up -d

Creating network "clickhouse-setup_default" with the default driver
Creating zookeeper-1 ... done
Creating clickhouse  ... done
Creating query-service                             ... done
Creating clickhouse-setup_otel-collector_1         ... done
Creating clickhouse-setup_otel-collector-metrics_1 ... done

ERROR: for alertmanager  Container "e13908186a47" is unhealthy.
ERROR: Encountered errors while bringing up the project.

I have only commented out

hotrod

and

load-hotrod

in docker-compose.yaml

I've setup signoz on an EC2 instance - t2.medium, 40-50 GB EBS. My issue stems from the fact that the system enters into a frozen state and I'm required to reboot the instance in order to ssh and restart Signoz. Signoz is the only "heavyweight" process on that instance. Any body else encounter this issue?

Hi. Is running a vanilla alertmanager supported with signoz ? I need use some integrations and I would like to manage alertmanager via config.

Hey, I just upgraded the helm deployment on one of our signoz environments and for some reason the otel collector is failing to start with this error:

Error: failed to build pipelines: failed to create "clickhousetraces" exporter for data type "traces": code: 60, message: Table signoz_traces.distributed_signoz_index_v2 doesn't exist
2023/06/29 11:18:38 application run finished with error: failed to build pipelines: failed to create "clickhousetraces" exporter for data type "traces": code: 60, message: Table signoz_traces.distributed_signoz_index_v2 doesn't exist

this was an upgrade from 0.11.1 to 0.21.0, the migration script was run beforehand

full log

2023-06-29T11:30:20.831Z	info	service/telemetry.go:104	Setting up own telemetry...
2023-06-29T11:30:20.832Z	info	service/telemetry.go:127	Serving Prometheus metrics	{"address": "0.0.0.0:8888", "level": "Basic"}
2023-06-29T11:30:20.953Z	info	clickhouselogsexporter/exporter.go:455	Running migrations from path: 	{"kind": "exporter", "data_type": "logs", "name": "clickhouselogsexporter", "test": "/logsmigrations"}
2023-06-29T11:30:20.968Z	info	clickhouselogsexporter/exporter.go:469	Clickhouse Migrate finished	{"kind": "exporter", "data_type": "logs", "name": "clickhouselogsexporter"}
2023-06-29T11:30:21.198Z	info	processor/processor.go:289	Development component. May change in the future.	{"kind": "processor", "name": "logstransform/internal", "pipeline": "logs"}
2023-06-29T11:30:21.198Z	info	exporter@v0.79.0/exporter.go:275	Stability level of component is undefined	{"kind": "exporter", "data_type": "traces", "name": "clickhousetraces"}
2023-06-29T11:30:21.328Z	info	clickhousetracesexporter/clickhouse_factory.go:141	View does not exist, skipping patch	{"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "table": "dependency_graph_minutes_db_calls_mv"}
2023-06-29T11:30:21.328Z	info	clickhousetracesexporter/clickhouse_factory.go:115	Running migrations from path: 	{"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "test": "/migrations"}
2023-06-29T11:30:21.338Z	info	clickhousetracesexporter/clickhouse_factory.go:127	Clickhouse Migrate finished	{"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "error": "Dirty database version 10. Fix and force version."}
Error: failed to build pipelines: failed to create "clickhousetraces" exporter for data type "traces": code: 60, message: Table signoz_traces.distributed_signoz_index_v2 doesn't exist
2023/06/29 11:30:21 application run finished with error: failed to build pipelines: failed to create "clickhousetraces" exporter for data type "traces": code: 60, message: Table signoz_traces.distributed_signoz_index_v2 doesn't exist

Hello, I am trying to install sample-application and generate tracing data https://signoz.io/docs/install/kubernetes/others/#optional-install-a-sample-application-and-generate-tracing-data But pods cannot start:

Normal   Pulling    6s (x3 over 48s)   kubelet            Pulling image "signoz/locust:0.8.1-py3.6"                                                                                                                   ││   Warning  Failed     5s (x3 over 47s)   kubelet            Failed to pull image "signoz/locust:0.8.1-py3.6": rpc error: code = NotFound desc = failed to pull and unpack image "<http://docker.io/signoz/locust:0.8.1-py3.6|docker.io/signoz/locust:0.8.1-py3.6>":  ││ failed to resolve reference "<http://docker.io/signoz/locust:0.8.1-py3.6|docker.io/signoz/locust:0.8.1-py3.6>": <http://docker.io/signoz/locust:0.8.1-py3.6|docker.io/signoz/locust:0.8.1-py3.6>: not found                                                                                                       ││   Warning  Failed     5s (x3 over 47s)   kubelet            Error: ErrImagePull

When I try to pull the image on my local computer I receive the following error:

docker pull signoz/locust:0.8.1-py3.6
Error response from daemon: manifest for signoz/locust:0.8.1-py3.6 not found: manifest unknown: manifest unknown

Could you please advise how the issue can be solved?