signoz-community #support

Join the conversation Join Slack

Channels

contributing

contributing-frontend

support

Mathias Wegner

01/10/2022, 4:31 PM

I've set up signoz on my macbook to run a small poc. I took a basic django app (essentially the polls app in the django tutorial), and while I can successfully get all of the api calls, I'm unable to collect any database calls. I'm using the most recent commit to the signoz repo with docker and clickhouse, and I started my django app with

~/src/apm-poc/example> OTEL_RESOURCE_ATTRIBUTES=service.name=example OTEL_EXPORTER_OTLP_ENDPOINT="<http://127.0.0.1:4317>" OTEL_LOG_LEVEL=debug opentelemetry-instrument python3 manage.py runserver --noreload
Performing system checks...

System check identified no issues (0 silenced).
January 10, 2022 - 11:22:20
Django version 4.0.1, using settings 'example.settings'
Starting development server at <http://127.0.0.1:8000/>
Quit the server with CONTROL-C.
[10/Jan/2022 11:22:21] "GET /admin/ HTTP/1.1" 302 0
[10/Jan/2022 11:22:21] "GET /admin/login/?next=/admin/ HTTP/1.1" 200 2211
[10/Jan/2022 11:22:26] "POST /admin/login/?next=/admin/ HTTP/1.1" 302 0
[10/Jan/2022 11:22:26] "GET /admin/ HTTP/1.1" 200 3974
[10/Jan/2022 11:22:26] "GET /static/admin/css/dashboard.css HTTP/1.1" 304 0
[10/Jan/2022 11:22:26] "GET /static/admin/img/icon-addlink.svg HTTP/1.1" 304 0
[10/Jan/2022 11:22:26] "GET /static/admin/img/icon-changelink.svg HTTP/1.1" 304 0

edr

01/13/2022, 7:22 AM

Hi all, is there an easy way to configure a different network for the docker compose tied to signoz?

Vinícius Costa

01/13/2022, 2:01 PM

Are there any guides or use cases of using SignNoz with AWS? I would like to collect some logs from cloudwatch there

Sagar Dutta

01/17/2022, 10:13 AM

2022/01/17 07:29:50 [warn] 1#1: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:2 nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:2 2022/01/17 07:29:50 [emerg] 1#1: mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied) nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)

Sagar Dutta

01/17/2022, 10:14 AM

2022-01-17T10:12:24.871Z [35mDEBUG[0m query-service/main.go:31 START! 2022-01-17T10:12:24.872Z [34mINFO[0m app/server.go:109 Using ClickHouse as datastore ... 2022-01-17T10:12:29.872Z [31mERROR[0m clickhouseReader/reader.go:90 error connecting to primary db: dial tcp 172.30.57.236:9000: i/o timeout go.signoz.io/query-service/app/clickhouseReader.NewReader /go/src/github.com/signoz/signoz/pkg/query-service/app/clickhouseReader/reader.go:90 go.signoz.io/query-service/app.createHTTPServer /go/src/github.com/signoz/signoz/pkg/query-service/app/server.go:110 go.signoz.io/query-service/app.NewServer /go/src/github.com/signoz/signoz/pkg/query-service/app/server.go:69 main.main /go/src/github.com/signoz/signoz/pkg/query-service/main.go:41 runtime.main /usr/local/go/src/runtime/proc.go:203

Sagar Dutta

01/17/2022, 10:54 AM

securityContext: capabilities: drop: - KILL - MKNOD - SETGID - SETUID runAsUser: 1000850000

Ditty K.M

01/17/2022, 5:06 PM

We are sending telemetry data to SigNoz using PL/Python function inside PostgreSQL. But sometimes we can't get the data to SigNoz. Even if this data reaches SigNoz, there will be a terrible delay. If we send data from outside PostgreSQL its working perfectly. Is this due to any specific python version requirement? Please advise me for the same, we can't figure out the issue. We are using 4317 port for sending data

Ranjit Barsa

01/24/2022, 7:31 AM

hello team can anyone help me on there's no data showing in metrics I have deployed signoz in local docker but when I am accessing http://localhost:3000 it's showing No Data

Ranjit Barsa

01/24/2022, 7:32 AM

and when I deployed signoz on server it shows below data

Martin Pola

01/26/2022, 5:56 PM

Hello! I just installed Signoz and I'm trying to figure out what's the best way to monitor CPU usage (%), disk usage, and other system-wide metrics for my servers. Is that possible with Signoz, and if so, how would I go about? I'm thinking that I'll need to install some sort of agent on each server, and point them to my Signoz backend/instance.

Onur Yikilmazoglu

01/27/2022, 1:06 PM

I'm trying to run the signoz without containerization, I started out with trying to run the frontend outside the container and the rest in the container, however even when point baseURL to localhost:8080 I'm getting just a blank screen. what am I doing wrong?

Prabhu Chawandi

01/30/2022, 6:19 AM

Hello, currently, we our server is run with docker-compose, and we are using fluentd to collect all the logs from each server instance and push it to the log management server that we have set up. So, I was thinking to use sigNoz here, so what is the ideal way that we start using the SignNoz for our setup.

Onur Yikilmazoglu

01/31/2022, 2:35 PM

I've got everything working since my last post here, however I have one big issue, has anyone figured out some sort of login, authentication or anything that wont let anyone use the frontend

Jingqi Huang

02/02/2022, 7:08 PM

I run

kubectl describe pvc

for all the pvc and I got the same message

FailedBinding  23s (x903 over 3h45m)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

Jingqi Huang

02/02/2022, 7:09 PM

Besides install signoz/signoz as suggested in https://signoz.io/docs/deployment/helm_chart, I am wondering what extra steps I need to take to make all the pods run. Thanks!

Thiago Alexandria

02/02/2022, 8:44 PM

Hi team, I'm trying to run signoz locally on my Mac (intel chip), using dockerd through rancher desktop. I'm having trouble because my clickhouse-server doesn't start, I get the following error from the container:

Poco::Exception. Code: 1000, e.code() = 0, Not found: user_files_path (version 21.10.5.3 (official build))
Processing configuration file '/etc/clickhouse-server/config.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
Include not found: clickhouse_remote_servers
Include not found: clickhouse_compression
Logging trace to /var/log/clickhouse-server/clickhouse-server.log
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
Processing configuration file '/etc/clickhouse-server/config.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
Include not found: clickhouse_remote_servers
Include not found: clickhouse_compression
Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/config.xml'.
Processing configuration file '/etc/clickhouse-server/users.xml'.
Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/users.xml'.
ClickHouse init process failed.

I looked for the chat and I couldn't find something that helps me, has anyone here gone through this?

Onur Yikilmazoglu

02/03/2022, 6:06 PM

I'm having a few problems with the otel collector, not sure if I should ask here or somewhere else since its not exactly what you guys do, however I keep having this problem where there the otel collector will balloon in ram usage and will eat most of the ram and almost all of my swap. I just check the my server and it using 6.5 gb of swap alone. I killed it with htop and swap usage dropped back down to 190 mb. However this is at best temporary, has anyone else ran into this problem?

Prabhu Chawandi

02/07/2022, 5:11 AM

Where I'm doing wrong? Could you guide me please?

Patrik Potocki

02/09/2022, 12:56 PM

Is there any documentation around how the Legend Format works in Metrics dashboard? 🙂

Patrik Potocki

02/09/2022, 1:36 PM

Questions , is there a reason why this is hardcoded? https://github.com/SigNoz/signoz/blob/28bf2fe3f7c0b013685921c9d4bfdb254ac42e81/pkg/query-service/constants/constants.go#L15 ? It's causes probems when I used the helm chart to install cause then my alertmanager service will be on

<http://signoz-alertmanager:9093/api/>

? 🙂

Patrik Potocki

02/10/2022, 9:10 AM

Is there a way to see what "metrics" you have available? cause quite hard to build those dashboard without knowing all metrics available. For example I tried to find what metrics you get from the

kubeletstats

collector

✅ 1

Daniel Lima

02/10/2022, 2:41 PM

Selva

02/12/2022, 3:34 PM

Hi I’m trying to create node js micro service with mysql db call, when i try do an api call to my endpoints able to see those in metrics, whereas the mysql db calls are not getting populated in Database Calls tab, can someone help?

Selva

02/14/2022, 11:10 AM

why console.errors are not captured here? on the right end Details for selected Span which type of error can we capture here can someone help?

praddy

02/15/2022, 5:35 AM

Hello folks On executing command

go get <http://go.signoz.io/query-service/version|go.signoz.io/query-service/version>

I am getting following error

go get: unrecognized import path "<http://go.signoz.io/query-service/version|go.signoz.io/query-service/version>": https fetch: Get "<https://go.signoz.io/query-service/version?go-get=1>": x509: certificate has expired or is not yet valid: current time 2022-02-15T11:01:04+05:30 is after 2021-06-12T15:18:58Z

Can someone please provide inputs here

Nico Van Wyk

02/16/2022, 12:16 PM

apiVersion: <http://traefik.containo.us/v1alpha1|traefik.containo.us/v1alpha1>
kind: IngressRoute
metadata:
  annotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: traefik
  name: signoz
spec:
  entryPoints:
    - web
  routes:
    - kind: Rule
      match: Host(`tools.localhost`) && PathPrefix(`/signoz`)
      middlewares:
        - name: signoz
      services:
        - name: signoz-frontend
          port: 3301

---

apiVersion: <http://traefik.containo.us/v1alpha1|traefik.containo.us/v1alpha1>
kind: Middleware
metadata:
  name: signoz
spec:
  headers:
    customRequestHeaders:
      X-Script-Name: "signoz"

Axay Sagathiya

02/18/2022, 5:42 AM

Hello folks, I'm new to the open-source world and want to contribute to the backend. all the steps given in this documentation are necessary to set up a local development environment?

Karam Ashqar

02/18/2022, 11:44 PM

hi, I installed everything on EKS, attached a Loadbalancer nginx to my-release-signoz-frontend svc, created account, logged in, then I attached another Loadbalancer nginx to my-release-signoz-otel-collector mapped to port 55681 I can see this response when I call <loadbalancer_externalip>/v1/trace

{
code: 12,
message: "Not Implemented",
details: [ ]
}

and when I put this link to my nestjs app on my local machine or cluster, it doesn't show any metrics/applications at all, anyone knows why?

paz.lucky

02/21/2022, 1:10 AM

The signoz docker installation fails. panic: runtime error: invalid memory address or nil pointer reference problem occurs.

Rajneesh Mehta

02/22/2022, 9:30 PM

getting

Overriding of current TracerProvider is not allowed

and application is not sending any data to signoz backend. Here's what configuration I've done. installed on:

minikube

installed with:

helm

release name:

signoz-uat

backend framework:

django

running server with :

gunicorn

command:

OTEL_METRICS_EXPORTER=none DJANGO_SETTINGS_MODULE=shopfloor.settings  OTEL_RESOURCE_ATTRIBUTES=service.name=stitch OTEL_EXPORTER_OTLP_ENDPOINT="signoz-uat-otel-collector.platform.svc.cluster.local:4317" opentelemetry-instrument gunicorn shopfloor.wsgi -c gunicorn.config.py --bind 0.0.0.0:8000

content of

gunicorn.config.py

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor


def post_fork(server, worker):
    <http://server.log.info|server.log.info>("Worker spawned (pid: %s)", worker.pid)

    resource = Resource.create(attributes={
        "service.name": "stitch"
    })

    trace.set_tracer_provider(TracerProvider(resource=resource))
    span_processor = BatchSpanProcessor(
        OTLPSpanExporter(endpoint="signoz-uat-otel-collector.platform.svc.cluster.local:4317")
    )
    trace.get_tracer_provider().add_span_processor(span_processor)

result from running Troubleshooting binary:-

root@django-backend-deployment-bb8875d7c-h9v4l:/app# ./troubleshoot checkEndpoint --endpoint=signoz-uat-otel-collector.platform.svc.cluster.local:4317
2022-02-22T21:27:43.780Z        INFO    workspace/main.go:28    STARTING!
2022-02-22T21:27:43.780Z        INFO    checkEndpoint/checkEndpoint.go:41       checking reachability of SigNoz endpoint
2022-02-22T21:27:43.796Z        INFO    workspace/main.go:46    Successfully sent sample data to signoz ...

this is what

<http://localhost:8888/metrics> contains

# HELP otelcol_exporter_queue_size Current size of the retry queue (in batches)
# TYPE otelcol_exporter_queue_size gauge
otelcol_exporter_queue_size{exporter="clickhousemetricswrite",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 0
# HELP otelcol_exporter_send_failed_metric_points Number of metric points in failed attempts to send to destination.
# TYPE otelcol_exporter_send_failed_metric_points counter
otelcol_exporter_send_failed_metric_points{exporter="prometheus",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 0
# HELP otelcol_exporter_send_failed_spans Number of spans in failed attempts to send to destination.
# TYPE otelcol_exporter_send_failed_spans counter
otelcol_exporter_send_failed_spans{exporter="clickhouse",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 0
# HELP otelcol_exporter_sent_metric_points Number of metric points successfully sent to destination.
# TYPE otelcol_exporter_sent_metric_points counter
otelcol_exporter_sent_metric_points{exporter="prometheus",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 0
# HELP otelcol_exporter_sent_spans Number of spans successfully sent to destination.
# TYPE otelcol_exporter_sent_spans counter
otelcol_exporter_sent_spans{exporter="clickhouse",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 3
# HELP otelcol_process_cpu_seconds Total CPU user and system time in seconds
# TYPE otelcol_process_cpu_seconds gauge
otelcol_process_cpu_seconds{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 15.25
# HELP otelcol_process_memory_rss Total physical memory (resident set size)
# TYPE otelcol_process_memory_rss gauge
otelcol_process_memory_rss{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 1.3193216e+08
# HELP otelcol_process_runtime_heap_alloc_bytes Bytes of allocated heap objects (see 'go doc runtime.MemStats.HeapAlloc')
# TYPE otelcol_process_runtime_heap_alloc_bytes gauge
otelcol_process_runtime_heap_alloc_bytes{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 4.912016e+07
# HELP otelcol_process_runtime_total_alloc_bytes Cumulative bytes allocated for heap objects (see 'go doc runtime.MemStats.TotalAlloc')
# TYPE otelcol_process_runtime_total_alloc_bytes gauge
otelcol_process_runtime_total_alloc_bytes{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 2.645922376e+09
# HELP otelcol_process_runtime_total_sys_memory_bytes Total bytes of memory obtained from the OS (see 'go doc runtime.MemStats.Sys')
# TYPE otelcol_process_runtime_total_sys_memory_bytes gauge
otelcol_process_runtime_total_sys_memory_bytes{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 8.30064824e+08
# HELP otelcol_process_uptime Uptime of the process
# TYPE otelcol_process_uptime counter
otelcol_process_uptime{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 7485.002079331997
# HELP otelcol_processor_batch_batch_send_size Number of units in the batch
# TYPE otelcol_processor_batch_batch_send_size histogram
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="10"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="25"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="50"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="75"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="100"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="250"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="500"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="750"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="1000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="2000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="3000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="4000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="5000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="6000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="7000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="8000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="9000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="10000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="20000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="30000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="50000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="100000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="+Inf"} 3
otelcol_processor_batch_batch_send_size_sum{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 3
otelcol_processor_batch_batch_send_size_count{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 3
# HELP otelcol_processor_batch_timeout_trigger_send Number of times the batch was sent due to a timeout trigger
# TYPE otelcol_processor_batch_timeout_trigger_send counter
otelcol_processor_batch_timeout_trigger_send{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 3
# HELP otelcol_receiver_accepted_spans Number of spans successfully pushed into the pipeline.
# TYPE otelcol_receiver_accepted_spans counter
otelcol_receiver_accepted_spans{receiver="otlp",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",transport="grpc"} 3
# HELP otelcol_receiver_refused_spans Number of spans that could not be pushed into the pipeline.
# TYPE otelcol_receiver_refused_spans counter
otelcol_receiver_refused_spans{receiver="otlp",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",transport="grpc"} 0

Can you please help what should I do?

Title

Rajneesh Mehta

02/22/2022, 9:30 PM

getting

Overriding of current TracerProvider is not allowed

and application is not sending any data to signoz backend. Here's what configuration I've done. installed on:

minikube

installed with:

helm

release name:

signoz-uat

backend framework:

django

running server with :

gunicorn

command:

OTEL_METRICS_EXPORTER=none DJANGO_SETTINGS_MODULE=shopfloor.settings  OTEL_RESOURCE_ATTRIBUTES=service.name=stitch OTEL_EXPORTER_OTLP_ENDPOINT="signoz-uat-otel-collector.platform.svc.cluster.local:4317" opentelemetry-instrument gunicorn shopfloor.wsgi -c gunicorn.config.py --bind 0.0.0.0:8000

content of

gunicorn.config.py

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor


def post_fork(server, worker):
    <http://server.log.info|server.log.info>("Worker spawned (pid: %s)", worker.pid)

    resource = Resource.create(attributes={
        "service.name": "stitch"
    })

    trace.set_tracer_provider(TracerProvider(resource=resource))
    span_processor = BatchSpanProcessor(
        OTLPSpanExporter(endpoint="signoz-uat-otel-collector.platform.svc.cluster.local:4317")
    )
    trace.get_tracer_provider().add_span_processor(span_processor)

result from running Troubleshooting binary:-

root@django-backend-deployment-bb8875d7c-h9v4l:/app# ./troubleshoot checkEndpoint --endpoint=signoz-uat-otel-collector.platform.svc.cluster.local:4317
2022-02-22T21:27:43.780Z        INFO    workspace/main.go:28    STARTING!
2022-02-22T21:27:43.780Z        INFO    checkEndpoint/checkEndpoint.go:41       checking reachability of SigNoz endpoint
2022-02-22T21:27:43.796Z        INFO    workspace/main.go:46    Successfully sent sample data to signoz ...

this is what

<http://localhost:8888/metrics> contains

# HELP otelcol_exporter_queue_size Current size of the retry queue (in batches)
# TYPE otelcol_exporter_queue_size gauge
otelcol_exporter_queue_size{exporter="clickhousemetricswrite",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 0
# HELP otelcol_exporter_send_failed_metric_points Number of metric points in failed attempts to send to destination.
# TYPE otelcol_exporter_send_failed_metric_points counter
otelcol_exporter_send_failed_metric_points{exporter="prometheus",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 0
# HELP otelcol_exporter_send_failed_spans Number of spans in failed attempts to send to destination.
# TYPE otelcol_exporter_send_failed_spans counter
otelcol_exporter_send_failed_spans{exporter="clickhouse",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 0
# HELP otelcol_exporter_sent_metric_points Number of metric points successfully sent to destination.
# TYPE otelcol_exporter_sent_metric_points counter
otelcol_exporter_sent_metric_points{exporter="prometheus",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 0
# HELP otelcol_exporter_sent_spans Number of spans successfully sent to destination.
# TYPE otelcol_exporter_sent_spans counter
otelcol_exporter_sent_spans{exporter="clickhouse",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 3
# HELP otelcol_process_cpu_seconds Total CPU user and system time in seconds
# TYPE otelcol_process_cpu_seconds gauge
otelcol_process_cpu_seconds{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 15.25
# HELP otelcol_process_memory_rss Total physical memory (resident set size)
# TYPE otelcol_process_memory_rss gauge
otelcol_process_memory_rss{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 1.3193216e+08
# HELP otelcol_process_runtime_heap_alloc_bytes Bytes of allocated heap objects (see 'go doc runtime.MemStats.HeapAlloc')
# TYPE otelcol_process_runtime_heap_alloc_bytes gauge
otelcol_process_runtime_heap_alloc_bytes{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 4.912016e+07
# HELP otelcol_process_runtime_total_alloc_bytes Cumulative bytes allocated for heap objects (see 'go doc runtime.MemStats.TotalAlloc')
# TYPE otelcol_process_runtime_total_alloc_bytes gauge
otelcol_process_runtime_total_alloc_bytes{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 2.645922376e+09
# HELP otelcol_process_runtime_total_sys_memory_bytes Total bytes of memory obtained from the OS (see 'go doc runtime.MemStats.Sys')
# TYPE otelcol_process_runtime_total_sys_memory_bytes gauge
otelcol_process_runtime_total_sys_memory_bytes{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 8.30064824e+08
# HELP otelcol_process_uptime Uptime of the process
# TYPE otelcol_process_uptime counter
otelcol_process_uptime{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 7485.002079331997
# HELP otelcol_processor_batch_batch_send_size Number of units in the batch
# TYPE otelcol_processor_batch_batch_send_size histogram
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="10"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="25"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="50"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="75"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="100"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="250"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="500"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="750"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="1000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="2000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="3000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="4000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="5000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="6000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="7000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="8000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="9000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="10000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="20000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="30000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="50000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="100000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="+Inf"} 3
otelcol_processor_batch_batch_send_size_sum{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 3
otelcol_processor_batch_batch_send_size_count{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 3
# HELP otelcol_processor_batch_timeout_trigger_send Number of times the batch was sent due to a timeout trigger
# TYPE otelcol_processor_batch_timeout_trigger_send counter
otelcol_processor_batch_timeout_trigger_send{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 3
# HELP otelcol_receiver_accepted_spans Number of spans successfully pushed into the pipeline.
# TYPE otelcol_receiver_accepted_spans counter
otelcol_receiver_accepted_spans{receiver="otlp",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",transport="grpc"} 3
# HELP otelcol_receiver_refused_spans Number of spans that could not be pushed into the pipeline.
# TYPE otelcol_receiver_refused_spans counter
otelcol_receiver_refused_spans{receiver="otlp",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",transport="grpc"} 0

Can you please help what should I do?

3 in

otelcol_exporter_sent_spans

is from Troubleshooting binary.

Ankit Nayan

02/23/2022, 4:58 AM

@User 1. Did you hit any apis of the application? Only then trace data will be generated. 2. Did you compare to or tried running https://github.com/SigNoz/sample-django?

@User might be the best person to help here 🙂

Rajneesh Mehta

02/23/2022, 5:05 AM

Thanks @User for your quick response. 1. Yes I did hit the api via browser. 2. I did ran the sample-django application but when I first tried SigNoz. And I did it with standalone docker image(all in one).

Ankit Nayan

02/23/2022, 6:26 AM

ok...do the application logs show some error other than

Overriding of current TracerProvider is not allowed

Rajneesh Mehta

02/23/2022, 9:09 AM

I was able to get rid of

Overriding of current TracerProvider is not allowed

by changing _`OTEL_METRICS_EXPORTER=none DJANGO_SETTINGS_MODULE=shopfloor.settings OTEL_RESOURCE_ATTRIBUTES=service.name=stitch OTEL_EXPORTER_OTLP_ENDPOINT="http://signoz-uat-otel-collector.platform.svc.cluster.local:4317" opentelemetry-instrument gunicorn shopfloor.wsgi -c gunicorn.config.py --bind 0.0.0.0:8000`_ to

gunicorn shopfloor.wsgi -c gunicorn.config.py --bind 0.0.0.0:8000

It seems Django specific configuration is not allowed with gunicorn specific configuration. Documentaion needs to clarify upon it. Two more thing that is still unclear :- 1. use of _`OTEL_METRICS_EXPORTER=none`_ 2. should we use

http://

with

OTEL_ENDPOINT

or not as troubleshooting binary doesn't allow it but application doesn't throw any error. Still won't able to get anything to backend.

root@django-backend-deployment-bb8875d7c-mmbpc:/app# gunicorn shopfloor.wsgi -c gunicorn.config.py --bind 0.0.0.0:8000
[2022-02-23 08:53:24 +0000] [31] [INFO] Starting gunicorn 19.9.0
[2022-02-23 08:53:24 +0000] [31] [INFO] Listening at: <http://0.0.0.0:8000> (31)
[2022-02-23 08:53:24 +0000] [31] [INFO] Using worker: sync
/usr/local/lib/python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  return io.open(fd, *args, **kwargs)
[2022-02-23 08:53:24 +0000] [33] [INFO] Booting worker with pid: 33
[2022-02-23 08:53:24 +0000] [33] [INFO] Worker spawned (pid: 33)
[2022-02-23 08:56:11 +0000] [31] [CRITICAL] WORKER TIMEOUT (pid:33)
[2022-02-23 08:56:11 +0000] [33] [INFO] Worker exiting (pid: 33)
[2022-02-23 08:56:12 +0000] [42] [INFO] Booting worker with pid: 42
[2022-02-23 08:56:12 +0000] [42] [INFO] Worker spawned (pid: 42)
[2022-02-23 08:56:52 +0000] [31] [CRITICAL] WORKER TIMEOUT (pid:42)
[2022-02-23 08:56:52 +0000] [42] [INFO] Worker exiting (pid: 42)
[2022-02-23 08:56:52 +0000] [64] [INFO] Booting worker with pid: 64
[2022-02-23 08:56:52 +0000] [64] [INFO] Worker spawned (pid: 64)
[2022-02-23 09:00:27 +0000] [31] [CRITICAL] WORKER TIMEOUT (pid:64)
[2022-02-23 09:00:27 +0000] [64] [INFO] Worker exiting (pid: 64)
[2022-02-23 09:00:28 +0000] [93] [INFO] Booting worker with pid: 93
[2022-02-23 09:00:28 +0000] [93] [INFO] Worker spawned (pid: 93)

getting this.

Ankit Nayan

02/23/2022, 10:35 AM

I see you removed all env vars, that won't instrument the applications now. Earlier was correct.

1. use of _`OTEL_METRICS_EXPORTER=none`_

It tells not to export metrics. It is not needed now as SigNoz can ingest metrics too now. Anyways, I don't think it will affect anything.

@User Since you are running your applications + signoz on k8s, it would be good to verify a few steps: 1. The sample django app also has the docker image which you can run in your k8s and test whether it sends data or not. Taking help from this link you can make a k8s manifest for the sample app. 2. I hope you have added this step in your dockerfile to auto-install and instrument various libs and frameworks used 3. Better to use the otel-lib versions specified in the requirements.txt of sample-django-app

View count: 49