https://signoz.io logo
Join the conversationJoin Slack
Channels
contributing
contributing-frontend
general
github-activities
incidents
k8s_operator_helm
reactjs-config
signoz-alert-testing
support
testsupport
watercooler
webhook-dev
write-for-signoz
Powered by Linen
support
  • m

    Mathias Wegner

    01/10/2022, 4:31 PM
    I've set up signoz on my macbook to run a small poc. I took a basic django app (essentially the polls app in the django tutorial), and while I can successfully get all of the api calls, I'm unable to collect any database calls. I'm using the most recent commit to the signoz repo with docker and clickhouse, and I started my django app with
    ~/src/apm-poc/example> OTEL_RESOURCE_ATTRIBUTES=service.name=example OTEL_EXPORTER_OTLP_ENDPOINT="<http://127.0.0.1:4317>" OTEL_LOG_LEVEL=debug opentelemetry-instrument python3 manage.py runserver --noreload
    Performing system checks...
    
    System check identified no issues (0 silenced).
    January 10, 2022 - 11:22:20
    Django version 4.0.1, using settings 'example.settings'
    Starting development server at <http://127.0.0.1:8000/>
    Quit the server with CONTROL-C.
    [10/Jan/2022 11:22:21] "GET /admin/ HTTP/1.1" 302 0
    [10/Jan/2022 11:22:21] "GET /admin/login/?next=/admin/ HTTP/1.1" 200 2211
    [10/Jan/2022 11:22:26] "POST /admin/login/?next=/admin/ HTTP/1.1" 302 0
    [10/Jan/2022 11:22:26] "GET /admin/ HTTP/1.1" 200 3974
    [10/Jan/2022 11:22:26] "GET /static/admin/css/dashboard.css HTTP/1.1" 304 0
    [10/Jan/2022 11:22:26] "GET /static/admin/img/icon-addlink.svg HTTP/1.1" 304 0
    [10/Jan/2022 11:22:26] "GET /static/admin/img/icon-changelink.svg HTTP/1.1" 304 0
    p
    a
    • 3
    • 16
  • e

    edr

    01/13/2022, 7:22 AM
    Hi all, is there an easy way to configure a different network for the docker compose tied to signoz?
    a
    p
    • 3
    • 8
  • v

    Vinícius Costa

    01/13/2022, 2:01 PM
    Are there any guides or use cases of using SignNoz with AWS? I would like to collect some logs from cloudwatch there
    p
    a
    • 3
    • 4
  • s

    Sagar Dutta

    01/17/2022, 10:13 AM
    2022/01/17 07:29:50 [warn] 1#1: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:2 nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:2 2022/01/17 07:29:50 [emerg] 1#1: mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied) nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)
    a
    p
    • 3
    • 14
  • s

    Sagar Dutta

    01/17/2022, 10:14 AM
    2022-01-17T10:12:24.871Z [35mDEBUG[0m query-service/main.go:31 START! 2022-01-17T10:12:24.872Z [34mINFO[0m app/server.go:109 Using ClickHouse as datastore ... 2022-01-17T10:12:29.872Z [31mERROR[0m clickhouseReader/reader.go:90 error connecting to primary db: dial tcp 172.30.57.236:9000: i/o timeout go.signoz.io/query-service/app/clickhouseReader.NewReader /go/src/github.com/signoz/signoz/pkg/query-service/app/clickhouseReader/reader.go:90 go.signoz.io/query-service/app.createHTTPServer /go/src/github.com/signoz/signoz/pkg/query-service/app/server.go:110 go.signoz.io/query-service/app.NewServer /go/src/github.com/signoz/signoz/pkg/query-service/app/server.go:69 main.main /go/src/github.com/signoz/signoz/pkg/query-service/main.go:41 runtime.main /usr/local/go/src/runtime/proc.go:203
    a
    p
    • 3
    • 4
  • s

    Sagar Dutta

    01/17/2022, 10:54 AM
    securityContext:         capabilities:           drop:             - KILL             - MKNOD             - SETGID             - SETUID         runAsUser: 1000850000
    a
    • 2
    • 1
  • d

    Ditty K.M

    01/17/2022, 5:06 PM
    We are sending telemetry data to SigNoz using PL/Python function inside PostgreSQL. But sometimes we can't get the data to SigNoz. Even if this data reaches SigNoz, there will be a terrible delay. If we send data from outside PostgreSQL its working perfectly. Is this due to any specific python version requirement? Please advise me for the same, we can't figure out the issue. We are using 4317 port for sending data
    p
    • 2
    • 2
  • r

    Ranjit Barsa

    01/24/2022, 7:31 AM
    hello team can anyone help me on there's no data showing in metrics I have deployed signoz in local docker but when I am accessing http://localhost:3000 it's showing No Data
    p
    • 2
    • 8
  • r

    Ranjit Barsa

    01/24/2022, 7:32 AM
    and when I deployed signoz on server it shows below data
    a
    • 2
    • 8
  • m

    Martin Pola

    01/26/2022, 5:56 PM
    Hello! I just installed Signoz and I'm trying to figure out what's the best way to monitor CPU usage (%), disk usage, and other system-wide metrics for my servers. Is that possible with Signoz, and if so, how would I go about? I'm thinking that I'll need to install some sort of agent on each server, and point them to my Signoz backend/instance.
    p
    • 2
    • 2
  • o

    Onur Yikilmazoglu

    01/27/2022, 1:06 PM
    I'm trying to run the signoz without containerization, I started out with trying to run the frontend outside the container and the rest in the container, however even when point baseURL to localhost:8080 I'm getting just a blank screen. what am I doing wrong?
    p
    p
    a
    • 4
    • 19
  • p

    Prabhu Chawandi

    01/30/2022, 6:19 AM
    Hello, currently, we our server is run with docker-compose, and we are using fluentd to collect all the logs from each server instance and push it to the log management server that we have set up. So, I was thinking to use sigNoz here, so what is the ideal way that we start using the SignNoz for our setup.
    p
    • 2
    • 4
  • o

    Onur Yikilmazoglu

    01/31/2022, 2:35 PM
    I've got everything working since my last post here, however I have one big issue, has anyone figured out some sort of login, authentication or anything that wont let anyone use the frontend
    p
    • 2
    • 9
  • j

    Jingqi Huang

    02/02/2022, 7:08 PM
    I run
    kubectl describe pvc
    for all the pvc and I got the same message
    FailedBinding  23s (x903 over 3h45m)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set
    p
    • 2
    • 7
  • j

    Jingqi Huang

    02/02/2022, 7:09 PM
    Besides install signoz/signoz as suggested in https://signoz.io/docs/deployment/helm_chart, I am wondering what extra steps I need to take to make all the pods run. Thanks!
    a
    • 2
    • 1
  • t

    Thiago Alexandria

    02/02/2022, 8:44 PM
    Hi team, I'm trying to run signoz locally on my Mac (intel chip), using dockerd through rancher desktop. I'm having trouble because my clickhouse-server doesn't start, I get the following error from the container:
    Poco::Exception. Code: 1000, e.code() = 0, Not found: user_files_path (version 21.10.5.3 (official build))
    Processing configuration file '/etc/clickhouse-server/config.xml'.
    Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
    Include not found: clickhouse_remote_servers
    Include not found: clickhouse_compression
    Logging trace to /var/log/clickhouse-server/clickhouse-server.log
    Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
    Processing configuration file '/etc/clickhouse-server/config.xml'.
    Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
    Include not found: clickhouse_remote_servers
    Include not found: clickhouse_compression
    Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/config.xml'.
    Processing configuration file '/etc/clickhouse-server/users.xml'.
    Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/users.xml'.
    ClickHouse init process failed.
    I looked for the chat and I couldn't find something that helps me, has anyone here gone through this?
    p
    p
    • 3
    • 5
  • o

    Onur Yikilmazoglu

    02/03/2022, 6:06 PM
    I'm having a few problems with the otel collector, not sure if I should ask here or somewhere else since its not exactly what you guys do, however I keep having this problem where there the otel collector will balloon in ram usage and will eat most of the ram and almost all of my swap. I just check the my server and it using 6.5 gb of swap alone. I killed it with htop and swap usage dropped back down to 190 mb. However this is at best temporary, has anyone else ran into this problem?
    p
    a
    • 3
    • 19
  • p

    Prabhu Chawandi

    02/07/2022, 5:11 AM
    Where I'm doing wrong? Could you guide me please?
    p
    a
    • 3
    • 4
  • p

    Patrik Potocki

    02/09/2022, 12:56 PM
    Is there any documentation around how the Legend Format works in Metrics dashboard? 🙂
    a
    • 2
    • 2
  • p

    Patrik Potocki

    02/09/2022, 1:36 PM
    Questions , is there a reason why this is hardcoded? https://github.com/SigNoz/signoz/blob/28bf2fe3f7c0b013685921c9d4bfdb254ac42e81/pkg/query-service/constants/constants.go#L15 ? It's causes probems when I used the helm chart to install cause then my alertmanager service will be on
    <http://signoz-alertmanager:9093/api/>
    ? 🙂
    a
    • 2
    • 4
  • p

    Patrik Potocki

    02/10/2022, 9:10 AM
    Is there a way to see what "metrics" you have available? cause quite hard to build those dashboard without knowing all metrics available. For example I tried to find what metrics you get from the
    kubeletstats
    collector
    ✅ 1
    p
    a
    • 3
    • 9
  • d

    Daniel Lima

    02/10/2022, 2:41 PM
    p
    a
    • 3
    • 99
  • s

    Selva

    02/12/2022, 3:34 PM
    Hi I’m trying to create node js micro service with mysql db call, when i try do an api call to my endpoints able to see those in metrics, whereas the mysql db calls are not getting populated in Database Calls tab, can someone help?
    p
    a
    • 3
    • 14
  • s

    Selva

    02/14/2022, 11:10 AM
    why console.errors are not captured here? on the right end Details for selected Span which type of error can we capture here can someone help?
    p
    • 2
    • 1
  • p

    praddy

    02/15/2022, 5:35 AM
    Hello folks On executing command
    go get <http://go.signoz.io/query-service/version|go.signoz.io/query-service/version>
    I am getting following error
    go get: unrecognized import path "<http://go.signoz.io/query-service/version|go.signoz.io/query-service/version>": https fetch: Get "<https://go.signoz.io/query-service/version?go-get=1>": x509: certificate has expired or is not yet valid: current time 2022-02-15T11:01:04+05:30 is after 2021-06-12T15:18:58Z
    Can someone please provide inputs here
    p
    v
    • 3
    • 2
  • n

    Nico Van Wyk

    02/16/2022, 12:16 PM
    apiVersion: <http://traefik.containo.us/v1alpha1|traefik.containo.us/v1alpha1>
    kind: IngressRoute
    metadata:
      annotations:
        <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: traefik
      name: signoz
    spec:
      entryPoints:
        - web
      routes:
        - kind: Rule
          match: Host(`tools.localhost`) && PathPrefix(`/signoz`)
          middlewares:
            - name: signoz
          services:
            - name: signoz-frontend
              port: 3301
    
    ---
    
    apiVersion: <http://traefik.containo.us/v1alpha1|traefik.containo.us/v1alpha1>
    kind: Middleware
    metadata:
      name: signoz
    spec:
      headers:
        customRequestHeaders:
          X-Script-Name: "signoz"
    p
    p
    • 3
    • 17
  • a

    Axay Sagathiya

    02/18/2022, 5:42 AM
    Hello folks, I'm new to the open-source world and want to contribute to the backend. all the steps given in this documentation are necessary to set up a local development environment?
    p
    v
    • 3
    • 46
  • k

    Karam Ashqar

    02/18/2022, 11:44 PM
    hi, I installed everything on EKS, attached a Loadbalancer nginx to my-release-signoz-frontend svc, created account, logged in, then I attached another Loadbalancer nginx to my-release-signoz-otel-collector mapped to port 55681 I can see this response when I call <loadbalancer_externalip>/v1/trace
    {
    code: 12,
    message: "Not Implemented",
    details: [ ]
    }
    and when I put this link to my nestjs app on my local machine or cluster, it doesn't show any metrics/applications at all, anyone knows why?
    p
    • 2
    • 11
  • p

    paz.lucky

    02/21/2022, 1:10 AM
    The signoz docker installation fails. panic: runtime error: invalid memory address or nil pointer reference problem occurs.
    p
    • 2
    • 16
  • r

    Rajneesh Mehta

    02/22/2022, 9:30 PM
    getting
    Overriding of current TracerProvider is not allowed
    and application is not sending any data to signoz backend. Here's what configuration I've done. installed on:
    minikube
    installed with:
    helm
    release name:
    signoz-uat
    backend framework:
    django
    running server with :
    gunicorn
    command:
    OTEL_METRICS_EXPORTER=none DJANGO_SETTINGS_MODULE=shopfloor.settings  OTEL_RESOURCE_ATTRIBUTES=service.name=stitch OTEL_EXPORTER_OTLP_ENDPOINT="signoz-uat-otel-collector.platform.svc.cluster.local:4317" opentelemetry-instrument gunicorn shopfloor.wsgi -c gunicorn.config.py --bind 0.0.0.0:8000
    content of
    gunicorn.config.py
    from opentelemetry import trace
    from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
    from opentelemetry.sdk.resources import Resource
    from opentelemetry.sdk.trace import TracerProvider
    from opentelemetry.sdk.trace.export import BatchSpanProcessor
    
    
    def post_fork(server, worker):
        <http://server.log.info|server.log.info>("Worker spawned (pid: %s)", worker.pid)
    
        resource = Resource.create(attributes={
            "service.name": "stitch"
        })
    
        trace.set_tracer_provider(TracerProvider(resource=resource))
        span_processor = BatchSpanProcessor(
            OTLPSpanExporter(endpoint="signoz-uat-otel-collector.platform.svc.cluster.local:4317")
        )
        trace.get_tracer_provider().add_span_processor(span_processor)
    result from running Troubleshooting binary:-
    root@django-backend-deployment-bb8875d7c-h9v4l:/app# ./troubleshoot checkEndpoint --endpoint=signoz-uat-otel-collector.platform.svc.cluster.local:4317
    2022-02-22T21:27:43.780Z        INFO    workspace/main.go:28    STARTING!
    2022-02-22T21:27:43.780Z        INFO    checkEndpoint/checkEndpoint.go:41       checking reachability of SigNoz endpoint
    2022-02-22T21:27:43.796Z        INFO    workspace/main.go:46    Successfully sent sample data to signoz ...
    this is what
    <http://localhost:8888/metrics> contains
    # HELP otelcol_exporter_queue_size Current size of the retry queue (in batches)
    # TYPE otelcol_exporter_queue_size gauge
    otelcol_exporter_queue_size{exporter="clickhousemetricswrite",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 0
    # HELP otelcol_exporter_send_failed_metric_points Number of metric points in failed attempts to send to destination.
    # TYPE otelcol_exporter_send_failed_metric_points counter
    otelcol_exporter_send_failed_metric_points{exporter="prometheus",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 0
    # HELP otelcol_exporter_send_failed_spans Number of spans in failed attempts to send to destination.
    # TYPE otelcol_exporter_send_failed_spans counter
    otelcol_exporter_send_failed_spans{exporter="clickhouse",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 0
    # HELP otelcol_exporter_sent_metric_points Number of metric points successfully sent to destination.
    # TYPE otelcol_exporter_sent_metric_points counter
    otelcol_exporter_sent_metric_points{exporter="prometheus",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 0
    # HELP otelcol_exporter_sent_spans Number of spans successfully sent to destination.
    # TYPE otelcol_exporter_sent_spans counter
    otelcol_exporter_sent_spans{exporter="clickhouse",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 3
    # HELP otelcol_process_cpu_seconds Total CPU user and system time in seconds
    # TYPE otelcol_process_cpu_seconds gauge
    otelcol_process_cpu_seconds{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 15.25
    # HELP otelcol_process_memory_rss Total physical memory (resident set size)
    # TYPE otelcol_process_memory_rss gauge
    otelcol_process_memory_rss{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 1.3193216e+08
    # HELP otelcol_process_runtime_heap_alloc_bytes Bytes of allocated heap objects (see 'go doc runtime.MemStats.HeapAlloc')
    # TYPE otelcol_process_runtime_heap_alloc_bytes gauge
    otelcol_process_runtime_heap_alloc_bytes{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 4.912016e+07
    # HELP otelcol_process_runtime_total_alloc_bytes Cumulative bytes allocated for heap objects (see 'go doc runtime.MemStats.TotalAlloc')
    # TYPE otelcol_process_runtime_total_alloc_bytes gauge
    otelcol_process_runtime_total_alloc_bytes{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 2.645922376e+09
    # HELP otelcol_process_runtime_total_sys_memory_bytes Total bytes of memory obtained from the OS (see 'go doc runtime.MemStats.Sys')
    # TYPE otelcol_process_runtime_total_sys_memory_bytes gauge
    otelcol_process_runtime_total_sys_memory_bytes{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 8.30064824e+08
    # HELP otelcol_process_uptime Uptime of the process
    # TYPE otelcol_process_uptime counter
    otelcol_process_uptime{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 7485.002079331997
    # HELP otelcol_processor_batch_batch_send_size Number of units in the batch
    # TYPE otelcol_processor_batch_batch_send_size histogram
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="10"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="25"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="50"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="75"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="100"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="250"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="500"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="750"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="1000"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="2000"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="3000"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="4000"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="5000"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="6000"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="7000"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="8000"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="9000"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="10000"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="20000"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="30000"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="50000"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="100000"} 3
    otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="+Inf"} 3
    otelcol_processor_batch_batch_send_size_sum{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 3
    otelcol_processor_batch_batch_send_size_count{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 3
    # HELP otelcol_processor_batch_timeout_trigger_send Number of times the batch was sent due to a timeout trigger
    # TYPE otelcol_processor_batch_timeout_trigger_send counter
    otelcol_processor_batch_timeout_trigger_send{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 3
    # HELP otelcol_receiver_accepted_spans Number of spans successfully pushed into the pipeline.
    # TYPE otelcol_receiver_accepted_spans counter
    otelcol_receiver_accepted_spans{receiver="otlp",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",transport="grpc"} 3
    # HELP otelcol_receiver_refused_spans Number of spans that could not be pushed into the pipeline.
    # TYPE otelcol_receiver_refused_spans counter
    otelcol_receiver_refused_spans{receiver="otlp",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",transport="grpc"} 0
    Can you please help what should I do?
    a
    • 2
    • 8
Powered by Linen
Title
r

Rajneesh Mehta

02/22/2022, 9:30 PM
getting
Overriding of current TracerProvider is not allowed
and application is not sending any data to signoz backend. Here's what configuration I've done. installed on:
minikube
installed with:
helm
release name:
signoz-uat
backend framework:
django
running server with :
gunicorn
command:
OTEL_METRICS_EXPORTER=none DJANGO_SETTINGS_MODULE=shopfloor.settings  OTEL_RESOURCE_ATTRIBUTES=service.name=stitch OTEL_EXPORTER_OTLP_ENDPOINT="signoz-uat-otel-collector.platform.svc.cluster.local:4317" opentelemetry-instrument gunicorn shopfloor.wsgi -c gunicorn.config.py --bind 0.0.0.0:8000
content of
gunicorn.config.py
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor


def post_fork(server, worker):
    <http://server.log.info|server.log.info>("Worker spawned (pid: %s)", worker.pid)

    resource = Resource.create(attributes={
        "service.name": "stitch"
    })

    trace.set_tracer_provider(TracerProvider(resource=resource))
    span_processor = BatchSpanProcessor(
        OTLPSpanExporter(endpoint="signoz-uat-otel-collector.platform.svc.cluster.local:4317")
    )
    trace.get_tracer_provider().add_span_processor(span_processor)
result from running Troubleshooting binary:-
root@django-backend-deployment-bb8875d7c-h9v4l:/app# ./troubleshoot checkEndpoint --endpoint=signoz-uat-otel-collector.platform.svc.cluster.local:4317
2022-02-22T21:27:43.780Z        INFO    workspace/main.go:28    STARTING!
2022-02-22T21:27:43.780Z        INFO    checkEndpoint/checkEndpoint.go:41       checking reachability of SigNoz endpoint
2022-02-22T21:27:43.796Z        INFO    workspace/main.go:46    Successfully sent sample data to signoz ...
this is what
<http://localhost:8888/metrics> contains
# HELP otelcol_exporter_queue_size Current size of the retry queue (in batches)
# TYPE otelcol_exporter_queue_size gauge
otelcol_exporter_queue_size{exporter="clickhousemetricswrite",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 0
# HELP otelcol_exporter_send_failed_metric_points Number of metric points in failed attempts to send to destination.
# TYPE otelcol_exporter_send_failed_metric_points counter
otelcol_exporter_send_failed_metric_points{exporter="prometheus",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 0
# HELP otelcol_exporter_send_failed_spans Number of spans in failed attempts to send to destination.
# TYPE otelcol_exporter_send_failed_spans counter
otelcol_exporter_send_failed_spans{exporter="clickhouse",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 0
# HELP otelcol_exporter_sent_metric_points Number of metric points successfully sent to destination.
# TYPE otelcol_exporter_sent_metric_points counter
otelcol_exporter_sent_metric_points{exporter="prometheus",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 0
# HELP otelcol_exporter_sent_spans Number of spans successfully sent to destination.
# TYPE otelcol_exporter_sent_spans counter
otelcol_exporter_sent_spans{exporter="clickhouse",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 3
# HELP otelcol_process_cpu_seconds Total CPU user and system time in seconds
# TYPE otelcol_process_cpu_seconds gauge
otelcol_process_cpu_seconds{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 15.25
# HELP otelcol_process_memory_rss Total physical memory (resident set size)
# TYPE otelcol_process_memory_rss gauge
otelcol_process_memory_rss{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 1.3193216e+08
# HELP otelcol_process_runtime_heap_alloc_bytes Bytes of allocated heap objects (see 'go doc runtime.MemStats.HeapAlloc')
# TYPE otelcol_process_runtime_heap_alloc_bytes gauge
otelcol_process_runtime_heap_alloc_bytes{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 4.912016e+07
# HELP otelcol_process_runtime_total_alloc_bytes Cumulative bytes allocated for heap objects (see 'go doc runtime.MemStats.TotalAlloc')
# TYPE otelcol_process_runtime_total_alloc_bytes gauge
otelcol_process_runtime_total_alloc_bytes{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 2.645922376e+09
# HELP otelcol_process_runtime_total_sys_memory_bytes Total bytes of memory obtained from the OS (see 'go doc runtime.MemStats.Sys')
# TYPE otelcol_process_runtime_total_sys_memory_bytes gauge
otelcol_process_runtime_total_sys_memory_bytes{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 8.30064824e+08
# HELP otelcol_process_uptime Uptime of the process
# TYPE otelcol_process_uptime counter
otelcol_process_uptime{service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 7485.002079331997
# HELP otelcol_processor_batch_batch_send_size Number of units in the batch
# TYPE otelcol_processor_batch_batch_send_size histogram
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="10"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="25"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="50"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="75"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="100"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="250"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="500"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="750"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="1000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="2000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="3000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="4000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="5000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="6000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="7000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="8000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="9000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="10000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="20000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="30000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="50000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="100000"} 3
otelcol_processor_batch_batch_send_size_bucket{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",le="+Inf"} 3
otelcol_processor_batch_batch_send_size_sum{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 3
otelcol_processor_batch_batch_send_size_count{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 3
# HELP otelcol_processor_batch_timeout_trigger_send Number of times the batch was sent due to a timeout trigger
# TYPE otelcol_processor_batch_timeout_trigger_send counter
otelcol_processor_batch_timeout_trigger_send{processor="batch",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c"} 3
# HELP otelcol_receiver_accepted_spans Number of spans successfully pushed into the pipeline.
# TYPE otelcol_receiver_accepted_spans counter
otelcol_receiver_accepted_spans{receiver="otlp",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",transport="grpc"} 3
# HELP otelcol_receiver_refused_spans Number of spans that could not be pushed into the pipeline.
# TYPE otelcol_receiver_refused_spans counter
otelcol_receiver_refused_spans{receiver="otlp",service_instance_id="0d36facb-49c4-4315-af49-10bea6e2c55c",transport="grpc"} 0
Can you please help what should I do?
3 in
otelcol_exporter_sent_spans
is from Troubleshooting binary.
a

Ankit Nayan

02/23/2022, 4:58 AM
@User 1. Did you hit any apis of the application? Only then trace data will be generated. 2. Did you compare to or tried running https://github.com/SigNoz/sample-django?
@User might be the best person to help here 🙂
r

Rajneesh Mehta

02/23/2022, 5:05 AM
Thanks @User for your quick response. 1. Yes I did hit the api via browser. 2. I did ran the sample-django application but when I first tried SigNoz. And I did it with standalone docker image(all in one).
a

Ankit Nayan

02/23/2022, 6:26 AM
ok...do the application logs show some error other than
Overriding of current TracerProvider is not allowed
?
r

Rajneesh Mehta

02/23/2022, 9:09 AM
I was able to get rid of
Overriding of current TracerProvider is not allowed
by changing _`OTEL_METRICS_EXPORTER=none DJANGO_SETTINGS_MODULE=shopfloor.settings OTEL_RESOURCE_ATTRIBUTES=service.name=stitch OTEL_EXPORTER_OTLP_ENDPOINT="http://signoz-uat-otel-collector.platform.svc.cluster.local:4317" opentelemetry-instrument gunicorn shopfloor.wsgi -c gunicorn.config.py --bind 0.0.0.0:8000`_ to
gunicorn shopfloor.wsgi -c gunicorn.config.py --bind 0.0.0.0:8000
It seems Django specific configuration is not allowed with gunicorn specific configuration. Documentaion needs to clarify upon it. Two more thing that is still unclear :- 1. use of _`OTEL_METRICS_EXPORTER=none`_ 2. should we use
http://
with
OTEL_ENDPOINT
or not as troubleshooting binary doesn't allow it but application doesn't throw any error. Still won't able to get anything to backend.
root@django-backend-deployment-bb8875d7c-mmbpc:/app# gunicorn shopfloor.wsgi -c gunicorn.config.py --bind 0.0.0.0:8000
[2022-02-23 08:53:24 +0000] [31] [INFO] Starting gunicorn 19.9.0
[2022-02-23 08:53:24 +0000] [31] [INFO] Listening at: <http://0.0.0.0:8000> (31)
[2022-02-23 08:53:24 +0000] [31] [INFO] Using worker: sync
/usr/local/lib/python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  return io.open(fd, *args, **kwargs)
[2022-02-23 08:53:24 +0000] [33] [INFO] Booting worker with pid: 33
[2022-02-23 08:53:24 +0000] [33] [INFO] Worker spawned (pid: 33)
[2022-02-23 08:56:11 +0000] [31] [CRITICAL] WORKER TIMEOUT (pid:33)
[2022-02-23 08:56:11 +0000] [33] [INFO] Worker exiting (pid: 33)
[2022-02-23 08:56:12 +0000] [42] [INFO] Booting worker with pid: 42
[2022-02-23 08:56:12 +0000] [42] [INFO] Worker spawned (pid: 42)
[2022-02-23 08:56:52 +0000] [31] [CRITICAL] WORKER TIMEOUT (pid:42)
[2022-02-23 08:56:52 +0000] [42] [INFO] Worker exiting (pid: 42)
[2022-02-23 08:56:52 +0000] [64] [INFO] Booting worker with pid: 64
[2022-02-23 08:56:52 +0000] [64] [INFO] Worker spawned (pid: 64)
[2022-02-23 09:00:27 +0000] [31] [CRITICAL] WORKER TIMEOUT (pid:64)
[2022-02-23 09:00:27 +0000] [64] [INFO] Worker exiting (pid: 64)
[2022-02-23 09:00:28 +0000] [93] [INFO] Booting worker with pid: 93
[2022-02-23 09:00:28 +0000] [93] [INFO] Worker spawned (pid: 93)
getting this.
a

Ankit Nayan

02/23/2022, 10:35 AM
I see you removed all env vars, that won't instrument the applications now. Earlier was correct.
1. use of _`OTEL_METRICS_EXPORTER=none`_
It tells not to export metrics. It is not needed now as SigNoz can ingest metrics too now. Anyways, I don't think it will affect anything.
@User Since you are running your applications + signoz on k8s, it would be good to verify a few steps: 1. The sample django app also has the docker image which you can run in your k8s and test whether it sends data or not. Taking help from this link you can make a k8s manifest for the sample app. 2. I hope you have added this step in your dockerfile to auto-install and instrument various libs and frameworks used 3. Better to use the otel-lib versions specified in the requirements.txt of sample-django-app
View count: 49