Hello! I was setting up signoz for monitoring my rust services. Need some help in integrating opentelemetry into warp(rust server) https://github.com/seanmonstar/warp.

Hi everybody, has anybody created a Discord webhook alert channel yet? I tried to create one and when I tested it, I got a 500 error. Some information about my setup: I hosted SigNoz on my private server, which is running Linux. The only open ports are for http, https and ssh (typical webserver setup). SigNoz is running on a Docker container, which is accessible via proxy behind a nginx webserver. The exposed ports of Docker are only accessible on localhost (if that matters). I would be glad, if anybody has an idea, what could cause this behavior. Best regards, Patrick

Hi! I just deployed to GCP, followed the docs exactly, everything seems to be running fine but I'm not allowed to register for the first time. I always get an authetication error when registering. Any ideas?

Hi @Ankit Nayan, we are trying to fetch the mongodb traces from a node js application request. but we are unable to get that info from node js http request trace. is there any way to get traces of mongodb ? We are using mongoose for the connecting with db. is it causing traces not to visible on signoz ?

I upgraded from 0.10 query service and frontend to 0.10.1 and now I get this in the query service logs:

Table signoz_traces.top_level_operations doesn't exist

Did I miss updating the collector, didn’t see anything mentioned in the release notes, wish the query service would be the component that handles schema creation instead of the collectors. Seems like the minor version bump might have had a breaking change if not aligned with the right collector?

I need to reset password for signoz, who can help me to process, my email is kspine@163.com Thanks

Hey everyone! I'm new to SigNoz and set it up today morning only.

It's set up on FastAPI, Python 3.10 with postgres. I was able to see DB calls in the APIs this morning but now they have all disappeared.

I read in the docs that psycopg2-binary is not supported by opentelemetry so need to use psycopg2. I did that but still no SQLs

Answered it myself. Had to reinstall things and forgot

opentelemetry-bootstrap --action=install

🤦‍♂️

Same is for Error percentage

Can you share the request payload for this?

Has anyone faced the following exception:

Traceback (most recent call last):
  File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/opentelemetry/sdk/metrics/_internal/instrument.py", line 132, in callback
    for api_measurement in callback(callback_options):
  File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/opentelemetry/instrumentation/system_metrics/__init__.py", line 563, in _get_system_network_connections
    for net_connection in psutil.net_connections():
  File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/psutil/__init__.py", line 2158, in net_connections
    return _psplatform.net_connections(kind)
  File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/psutil/_psosx.py", line 248, in net_connections
    cons = Process(pid).connections(kind)
  File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/psutil/_psosx.py", line 350, in wrapper
    raise AccessDenied(self.pid, self._name)
psutil.AccessDenied: (pid=27843)
Callback failed for instrument system.network.connections.
Traceback (most recent call last):
  File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/psutil/_psosx.py", line 343, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/psutil/_psosx.py", line 500, in connections
    rawlist = cext.proc_connections(self.pid, families, types)
PermissionError: [Errno 1] Operation not permitted (originated from proc_pidinfo(PROC_PIDLISTFDS) 1/2)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/opentelemetry/sdk/metrics/_internal/instrument.py", line 132, in callback
    for api_measurement in callback(callback_options):
  File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/opentelemetry/instrumentation/system_metrics/__init__.py", line 563, in _get_system_network_connections
    for net_connection in psutil.net_connections():
  File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/psutil/__init__.py", line 2158, in net_connections
    return _psplatform.net_connections(kind)
  File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/psutil/_psosx.py", line 248, in net_connections
    cons = Process(pid).connections(kind)
  File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/psutil/_psosx.py", line 350, in wrapper
    raise AccessDenied(self.pid, self._name)
psutil.AccessDenied: (pid=27843)

Searched for this in issues but didn't find any.

👋 Hello, team!

might be too basic for a support channel, but I remember reading it somewhere, but can't find it now. How do I login to signoz UI thats installed in a k8s cluster . Thanks

Hi, is there any way to ignore certain endpoints?

Is there a way to auto refresh the dashboard and also to keep the sorting fixed by a certain metric?

s3 storage is a cold storage, does not pull data from the clickhouse -> s3. Does any one has idea?

Specifically - we slowly increase the load of micro-service data - but we are facing challenges - with the query builder and unable to load. - any idea or suggestion - or any pointer talks about the scalability of the system.

Can anyone direct me to a gallery of custom dashboards, if any

Hi, Is version 0.10.2 or 0.10.1 available on the main branch yet? I followed the steps to upgrade but it is still on 0.10.0

Hi, Signoz (in kubernetes) has been working well but has recently started giving OOM (out of memory) errors in the signoz-query-service. I've restarted the pod several times but has not helped. Here are the pods:

kubectl -n platform get pods
NAME                                                        READY   STATUS             RESTARTS   AGE
my-release-zookeeper-0                                      1/1     Running            0          77d
chi-signoz-cluster-0-0-0                                    1/1     Running            0          77d
my-release-signoz-alertmanager-0                            1/1     Running            0          77d
my-release-signoz-frontend-6b7dbccbc7-fgbnv                 1/1     Running            0          77d
my-release-signoz-otel-collector-metrics-68bcfd5556-7tjks   1/1     Running            0          77d
my-release-signoz-otel-collector-66c8c7dc9d-xqxbd           1/1     Running            0          77d
clickhouse-operator-7f7f84b899-dcgs4                        2/2     Running            0          21h
my-release-signoz-query-service-0                           0/1     CrashLoopBackOff   201        18h

In the description of the my-release-signoz-query-service-0 pod, it shows:

Containers:
  signoz:
    Container ID:  <containerd://aebb11f5525602fd8205e1aa49414724c6afc7c5b1784993bfe58d3e5a7e931>5
    Image:         <http://docker.io/signoz/query-service:0.8.0|docker.io/signoz/query-service:0.8.0>
    Image ID:      <http://docker.io/signoz/query-service@sha256:2febce16a8b8feb6bf96439eccccba4e6ab8e7ef401de96ea38fd1d42c3d9353|docker.io/signoz/query-service@sha256:2febce16a8b8feb6bf96439eccccba4e6ab8e7ef401de96ea38fd1d42c3d9353>
    Port:          8080/TCP
    Host Port:     0/TCP
    Args:
      -config=/root/config/prometheus.yml
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Thu, 11 Aug 2022 10:54:03 -0700
      Finished:     Thu, 11 Aug 2022 10:54:42 -0700
    Ready:          False
    Restart Count:  201
    Limits:
      cpu:     750m
      memory:  1000Mi
    Requests:
      cpu:      200m
      memory:   300Mi

I would increase the memory limit but don't see a deployment for the my-release-signoz-query-service-0 pod:

kubectl -n platform get deploy
NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
my-release-signoz-frontend                 1/1     1            1           77d
my-release-signoz-otel-collector-metrics   1/1     1            1           77d
my-release-signoz-otel-collector           1/1     1            1           77d
clickhouse-operator                        1/1     1            1           77d

How do I fix this?

Hey, trying to setup NestJS with OpenTel + Signoz but after following the sample repository, I encounter this error. Any clue what's going on there?

credentials: grpc.credentials.createInsecure(),
                    ^
TypeError: Cannot read properties of undefined (reading 'credentials')
    at Object.<anonymous> (/home/xxx/src/tracer.ts:17:21)
    at Module._compile (node:internal/modules/cjs/loader:1103:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/home/xxx/src/main.ts:13:1)
    at Module._compile (node:internal/modules/cjs/loader:1103:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)

@Jatin Sharma you can just start the app again with appropriate steps to start sending data again

👋 Hello, team! I am currently using jaeger with open telemetry; I want to move to Signoz. Can someone help me out with how to do it?

I added service.namespace and deployment.environment to my process but don’t see the ability to filter by it in the UI (using 0.10.2). The process was running without those parameters before and sending to SigNoz. Is there something I need to do to enable it? Config for collector:

receivers:
  otlp/spanmetrics:
    protocols:
      grpc:
        endpoint: localhost:12345
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    send_batch_size: 10000
    send_batch_max_size: 11000
    timeout: 10s
  signozspanmetrics/prometheus:
    metrics_exporter: prometheus
    latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 50ms, 100ms, 250ms, 500ms, 1000ms, 1400ms, 2000ms, 5s, 10s, 20s, 40s, 60s ]
    dimensions_cache_size: 10000
    dimensions:
      - name: service.namespace
        default: default
      - name: deployment.environment
        default: default

exporters:
  clickhousetraces:
    datasource: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=signoz_traces&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PWD}
  clickhousemetricswrite:
    endpoint: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=signoz_metrics&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PWD}
    resource_to_telemetry_conversion:
      enabled: true
  prometheus:
    endpoint: 0.0.0.0:8889

extensions:
  health_check:
    endpoint: 0.0.0.0:13133
service:
  telemetry:
    metrics:
      address: 0.0.0.0:8888
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [signozspanmetrics/prometheus, batch]
      exporters: [clickhousetraces]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhousemetricswrite]
    metrics/spanmetrics:
      receivers: [otlp/spanmetrics]
      exporters: [prometheus]

Note: I am not sending directly to this collector but via an agent collector on the instance which forwards to this one via OTLP

Hello, I followed the instruction from docs to install signoz in kuberntes. I used different namespace (signoz) and different release name (signoz-dev). I installed the example also using this singoz example. Signoz has been installed properly. I just change values of ingress only in values.yaml file. But I am facing error

something went wrong

. The log of otel is below:

2022-08-19T04:48:22.427Z	error	clickhousetracesexporter/writer.go:101	Could not write a batch of spans	{"kind": "exporter", "name": "clickhousetraces", "error": "code: 81, message: Database signoz_traces doesn't exist"}
<http://github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter|github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter>
	/src/exporter/clickhousetracesexporter/writer.go:101
2022-08-19T04:48:32.432Z	error	clickhousetracesexporter/writer.go:101	Could not write a batch of spans	{"kind": "exporter", "name": "clickhousetraces", "error": "code: 81, message: Database signoz_traces doesn't exist"}
<http://github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter|github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter>
	/src/exporter/clickhousetracesexporter/writer.go:101
2022-08-19T04:48:42.430Z	error	clickhousetracesexporter/writer.go:101	Could not write a batch of spans	{"kind": "exporter", "name": "clickhousetraces", "error": "code: 81, message: Database signoz_traces doesn't exist"}
<http://github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter|github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter>
	/src/exporter/clickhousetracesexporter/writer.go:101
2022-08-19T04:48:52.444Z	error	clickhousetracesexporter/writer.go:101	Could not write a batch of spans	{"kind": "exporter", "name": "clickhousetraces", "error": "code: 81, message: Database signoz_traces doesn't exist"}
<http://github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter|github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter>
	/src/exporter/clickhousetracesexporter/writer.go:101
2022-08-19T04:49:02.443Z	error	clickhousetracesexporter/writer.go:101	Could not write a batch of spans	{"kind": "exporter", "name": "clickhousetraces", "error": "code: 81, message: Database signoz_traces doesn't exist"}
<http://github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter|github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter>
	/src/exporter/clickhousetracesexporter/writer.go:101
2022-08-19T04:49:12.437Z	error	clickhousetracesexporter/writer.go:101	Could not write a batch of spans	{"kind": "exporter", "name": "clickhousetraces", "error": "code: 81, message: Database signoz_traces doesn't exist"}
<http://github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter|github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter>
	/src/exporter/clickhousetracesexporter/writer.go:101
2022-08-19T04:49:22.438Z	error	clickhousetracesexporter/writer.go:101	Could not write a batch of spans	{"kind": "exporter", "name": "clickhousetraces", "error": "code: 81, message: Database signoz_traces doesn't exist"}
<http://github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter|github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter>
	/src/exporter/clickhousetracesexporter/writer.go:101
Logs

Also in UI there is no metrics section.

hi, is there any way I can see the infrastructure? For instance, how many API servers I have running

I’d like to update my collector to 0.58.0, how badly will clickhouse break? https://github.com/open-telemetry/opentelemetry-collector-contrib/releases/tag/v0.58.0