Hi. I have trouble with auto refresh UI. How can I enable auto refresh? Many thanks!

Hi team, Happy morning I'm currently implementing signoz in GCP via helm but my pod getting "CrashLoopBackOff" FYKI: chi-signoz-cluster-0-0-0 1/1 Running 0 21h clickhouse-operator-57959d7fd8-tpl5g 2/2 Running 0 21h my-release-signoz-alertmanager-0 0/1 Init:0/1 0 23h my-release-signoz-frontend-6dd9f97bc4-jnkkb 0/1 Init:0/1 0 23h my-release-signoz-frontend-786b84f4fd-lqd2f 0/1 Init:0/1 0 21h my-release-signoz-otel-collector-6b47dd5b45-m9v7d 0/1 CrashLoopBackOff 282 (2m10s ago) 23h my-release-signoz-otel-collector-7587cb94c5-s46jg 0/1 CrashLoopBackOff 261 (117s ago) 21h my-release-signoz-otel-collector-metrics-55644664d9-qdcpj 0/1 CrashLoopBackOff 243 (4m40s ago) 21h my-release-signoz-query-service-0 0/1 CrashLoopBackOff 293 (3m29s ago) 23h my-release-zookeeper-0 1/1 Running 0 23h This is error logs : Logs: runtime.main /usr/local/go/src/runtime/proc.go:255 2022-10-18T06:47:47.649Z INFO app/server.go:112 Using ClickHouse as datastore ... 2022-10-18T06:47:47.654Z ERROR clickhouseReader/reader.go:108 error connecting to primary db: code: 516, message: admin: Authentication failed: password is incorrect or there is no user with such name go.signoz.io/query-service/app/clickhouseReader.NewReader /go/src/github.com/signoz/signoz/pkg/query-service/app/clickhouseReader/reader.go:108 go.signoz.io/query-service/app.(*Server).createHTTPServer /go/src/github.com/signoz/signoz/pkg/query-service/app/server.go:113 go.signoz.io/query-service/app.NewServer /go/src/github.com/signoz/signoz/pkg/query-service/app/server.go:87 main.main /go/src/github.com/signoz/signoz/pkg/query-service/main.go:53 runtime.main /usr/local/go/src/runtime/proc.go:255

<!here> can anyone help me on this please

Hi I'm trying to use Opentelemetry-python to auto-instrument my Django app and I'm using signoz as backend. But on the signoz UI there are only parent traces shown and there are no child spans and I think there should be one. Please help me with that. If I need to share more information then you can ask me.

Hi Team, I am facing issue to setup Signoz on local k8s system. Should we create pv and pvc before deploy it on local k8s system?

Hi I'm trying to send open telemetry data to signoz. I have added host details for outputs.opentelemetry and I can from telegraf side messages are being pushed. But I can't see Serive/messages in Signoz . Can any one help here.

Hi Team, I'm trying to setup Signoz using the official docker installation method. I was able to get the service up and running but failing to instrument my

GraphQL

application. I believe this is happening because of the way my host is set up. It's a VM with docker + nginx proxy manager. What should be the value for

OTEL_EXPORTER_OTLP_ENDPOINT

in this case?

Hi, I have another issue with sending metrics via FastAPI python application

TypeError: an integer is required (got type NoneType)
Transient error StatusCode.UNAVAILABLE encountered while exporting metrics, retrying in Nones.
Exception while exporting metrics an integer is required (got type NoneType)
Traceback (most recent call last):
  File "/opt/venv/lib/python3.9/site-packages/opentelemetry/exporter/otlp/proto/grpc/exporter.py", line 305, in _export
    self._client.Export(
  File "/opt/venv/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/opt/venv/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:Failed to pick subchannel {created_time:"2022-10-19T17:51:34.004255785+00:00", children:[UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: Failed to connect to remote host: Connection refused {created_time:"2022-10-19T17:51:34.004247397+00:00", grpc_status:14}]}"
>

It send the request successfully however doesn't send to signoz due to this exception

Hi, I wanted to filter out some log entries since Longhorn is running in debug mode it fills up the logs very quick. But I found no good way of doing it. How should I configure the signoz opentel collector pipe for it? If I try to reges the body via filter I get an error message. Here is what I want to filter out and the error message:

{
  "timestamp": 1666209977113678300,
  "id": "2GMrb6aQr7vufL4DoehXo21U7Dl",
  "trace_id": "",
  "span_id": "",
  "trace_flags": 0,
  "severity_text": "",
  "severity_number": 0,
  "body": "[longhorn-instance-manager] time=\"2022-10-19T20:06:17Z\" level=debug msg=\"Getting snapshot clone status\" serviceURL=\"10.42.0.236:10007\"",
  "resources_string": {},
  "attributes_string": {
    "k8s_container_name": "engine-manager",
    "k8s_container_restart_count": "0",
    "k8s_namespace_name": "longhorn-system",
    "k8s_pod_name": "instance-manager-e-0df031cc",
    "k8s_pod_uid": "9f5aeea7-52ac-4fa6-83de-ad957792842c",
    "log_file_path": "/var/log/pods/longhorn-system_instance-manager-e-0df031cc_9f5aeea7-52ac-4fa6-83de-ad957792842c/engine-manager/0.log",
    "log_iostream": "stderr",
    "logtag": "F",
    "time": "2022-10-19T20:06:17.113678307Z"
  },
  "attributes_int": {},
  "attributes_float": {}
}

error:

* 'logs.exclude' has invalid keys: bodies
2022/10/19 20:12:25 application run finished with error: failed to get config: cannot unmarshal the configuration: error reading processors configuration for "filter/1": 1 error(s) decoding:
* 'logs.exclude' has invalid keys: bodies

My configuration:

processors:
      filter/1:
        logs/bodies:
          exclude:
            match_type: regexp
            bodies:
            - "longhorn.*level=debug"

Thank you for your help. I also tried filtering the namespace (longhorn-system) but it didn't work either....

@nitya-signoz Hi When i run my application in local, it should be work properly. but when i run my application in docker signoz front-end can not give any service name or any output. Please help me!

How can I send nginx access and error logs to Signoz directly?

Hello everyone, I tried to install the Signoz into my Go application, already follow the instructions from the doc but when I try to hit the request to generate the service, there is no service generated into my dashboard, anyone know what happen?

Maybe there's something that I miss prolly?

Hi team I'm facing the same issues can any one help me on this :https://community-chat.signoz.io/t/428616/hi-c02271gfa4f-k8s-operator-helm-facing-issue-with-signoz-qu

Hope issue resolved by @Prashant Shahi please help me to resolve this issue

Hi team, I have just started trying out SigNoz and unable to send file logs from another host to SigNoz. Any pointers/direction appreciated! These are the steps I've tried so far SigNoz (v0.11.2) [192.168.200.20] SigNoz Setup: installed via ./install.sh SigNoz changes: removed hotrod from docker-compose NestJS app, dev OSX [192.168.200.187] NestJS using auto-node tracing SigNoz able to receive NestJS traces ✅ NestJS Logging / Winston Logs from backend.log, send to SigNoz Logs via otel-collector ❌ Example log entry in backend.log:

{"level":"info","message":"PassportModule dependencies initialized","service":"logger-service","timestamp":"2022-10-21T10:51:31.050Z"}

Following this doc: https://signoz.io/docs/userguide/collect_logs_from_file/ Docker run command:

docker run -d --name signoz-otel-collector --user root -v $(pwd)/backend.log:/tmp/backend.log:ro -v $(pwd)/signoz-winston-collector.yaml:/etc/otel/config.yaml signoz/signoz-otel-collector:0.55.3

signoz-winston-collector.yaml

receivers:
  filelog:
    include: [/tmp/backend.log]
    start_at: beginning
    operators:
      - type: json_parser
        id: nest-winston-json
        timestamp:
          parse_from: attributes.timestamp
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
      - type: move
        from: attributes.message
        to: body
      - type: remove
        field: attributes.timestamp
processors:
  batch:
    send_batch_size: 10 #low num for testing
    send_batch_max_size: 10 #low num for testing
    timeout: 10s
exporters:
  otlp/log:
    endpoint: <http://192.168.200.20:4317>
    tls:
      insecure: true
service:
  telemetry:
    logs:
      level: 'debug'
  pipelines:
    logs:
      receivers: [filelog]
      processors: [batch]
      exporters: [otlp/log]

Docker collector container logs

2022-10-22T02:31:52.529Z        info    service/telemetry.go:103        Setting up own telemetry...
2022-10-22T02:31:52.530Z        info    service/telemetry.go:138        Serving Prometheus metrics      {"address": ":8888", "level": "basic"}
2022-10-22T02:31:52.530Z        debug   pipelines/pipelines.go:343      Stability level {"kind": "exporter", "data_type": "logs", "name": "otlp/log", "stability": "beta"}
2022-10-22T02:31:52.530Z        debug   pipelines/pipelines.go:343      Stability level {"kind": "processor", "name": "batch", "pipeline": "logs", "stability": "stable"}
2022-10-22T02:31:52.530Z        debug   pipelines/pipelines.go:345      Stability level of component undefined  {"kind": "receiver", "name": "filelog", "pipeline": "logs", "stability": "undefined"}
2022-10-22T02:31:52.532Z        info    extensions/extensions.go:42     Starting extensions...
2022-10-22T02:31:52.532Z        info    pipelines/pipelines.go:74       Starting exporters...
2022-10-22T02:31:52.532Z        info    pipelines/pipelines.go:78       Exporter is starting... {"kind": "exporter", "data_type": "logs", "name": "otlp/log"}
2022-10-22T02:31:52.532Z        info    zapgrpc/zapgrpc.go:174  [core] [Channel #1] Channel created     {"grpc_log": true}
2022-10-22T02:31:52.532Z        info    zapgrpc/zapgrpc.go:174  [core] [Channel #1] original dial target is: "192.168.200.20:4317"      {"grpc_log": true}
2022-10-22T02:31:52.532Z        info    zapgrpc/zapgrpc.go:174  [core] [Channel #1] dial target "192.168.200.20:4317" parse failed: parse "192.168.200.20:4317": first path segment in URL cannot contain colon       {"grpc_log": true}
2022-10-22T02:31:52.532Z        info    zapgrpc/zapgrpc.go:174  [core] [Channel #1] fallback to scheme "passthrough"    {"grpc_log": true}
2022-10-22T02:31:52.532Z        info    zapgrpc/zapgrpc.go:174  [core] [Channel #1] parsed dial target is: {Scheme:passthrough Authority: Endpoint:192.168.200.20:4317 URL:{Scheme:passthrough Opaque: User: Host: Path:/192.168.200.20:4317 RawPath: ForceQuery:false RawQuery: Fragment: RawFragment:}}   {"grpc_log": true}
2022-10-22T02:31:52.532Z        info    zapgrpc/zapgrpc.go:174  [core] [Channel #1] Channel authority set to "192.168.200.20:4317"      {"grpc_log": true}
2022-10-22T02:31:52.532Z        info    zapgrpc/zapgrpc.go:174  [core] [Channel #1] Resolver state updated: {
  "Addresses": [
    {
      "Addr": "192.168.200.20:4317",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Type": 0,
      "Metadata": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
} (resolver returned new addresses)     {"grpc_log": true}
2022-10-22T02:31:52.533Z        info    zapgrpc/zapgrpc.go:174  [core] [Channel #1] Channel switches to new LB policy "pick_first"      {"grpc_log": true}
2022-10-22T02:31:52.533Z        info    zapgrpc/zapgrpc.go:174  [core] [Channel #1 SubChannel #2] Subchannel created    {"grpc_log": true}
2022-10-22T02:31:52.533Z        info    zapgrpc/zapgrpc.go:174  [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING{"grpc_log": true}
2022-10-22T02:31:52.533Z        info    zapgrpc/zapgrpc.go:174  [core] [Channel #1 SubChannel #2] Subchannel picks a new address "192.168.200.20:4317" to connect     {"grpc_log": true}
2022-10-22T02:31:52.533Z        info    pipelines/pipelines.go:82       Exporter started.       {"kind": "exporter", "data_type": "logs", "name": "otlp/log"}
2022-10-22T02:31:52.533Z        info    pipelines/pipelines.go:86       Starting processors...
2022-10-22T02:31:52.533Z        info    pipelines/pipelines.go:90       Processor is starting...        {"kind": "processor", "name": "batch", "pipeline": "logs"}
2022-10-22T02:31:52.534Z        info    zapgrpc/zapgrpc.go:174  [core] pickfirstBalancer: UpdateSubConnState: 0xc0008d10d0, {CONNECTING <nil>}{"grpc_log": true}
2022-10-22T02:31:52.534Z        info    zapgrpc/zapgrpc.go:174  [core] [Channel #1] Channel Connectivity change to CONNECTING   {"grpc_log": true}
2022-10-22T02:31:52.534Z        info    pipelines/pipelines.go:94       Processor started.      {"kind": "processor", "name": "batch", "pipeline": "logs"}
2022-10-22T02:31:52.534Z        info    pipelines/pipelines.go:98       Starting receivers...
2022-10-22T02:31:52.534Z        info    pipelines/pipelines.go:102      Receiver is starting... {"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-10-22T02:31:52.534Z        info    adapter/receiver.go:54  Starting stanza receiver        {"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-10-22T02:31:52.534Z        debug   pipeline/directed.go:70 Starting operator       {"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-10-22T02:31:52.534Z        debug   pipeline/directed.go:74 Started operator        {"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-10-22T02:31:52.534Z        debug   pipeline/directed.go:70 Starting operator       {"kind": "receiver", "name": "filelog", "pipeline": "logs", "operator_id": "remove", "operator_type": "remove"}
2022-10-22T02:31:52.534Z        debug   pipeline/directed.go:74 Started operator        {"kind": "receiver", "name": "filelog", "pipeline": "logs", "operator_id": "remove", "operator_type": "remove"}
2022-10-22T02:31:52.534Z        debug   adapter/converter.go:143        Starting log converter  {"kind": "receiver", "name": "filelog", "pipeline": "logs", "worker_count": 1}
2022-10-22T02:31:52.534Z        info    pipelines/pipelines.go:106      Receiver started.       {"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-10-22T02:31:52.535Z        info    service/collector.go:215        Starting signoz-otel-collector...       {"Version": "latest", "NumCPU": 6}
2022-10-22T02:31:52.535Z        info    service/collector.go:128        Everything is ready. Begin running and processing data.
2022-10-22T02:31:52.738Z        info    fileconsumer/file.go:178        Started watching file   {"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "path": "/tmp/backend.log"}
2022-10-22T02:31:52.744Z        info    zapgrpc/zapgrpc.go:174  [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to READY    {"grpc_log": true}
2022-10-22T02:31:52.744Z        info    zapgrpc/zapgrpc.go:174  [core] pickfirstBalancer: UpdateSubConnState: 0xc0008d10d0, {READY <nil>}    {"grpc_log": true}
2022-10-22T02:31:52.744Z        info    zapgrpc/zapgrpc.go:174  [core] [Channel #1] Channel Connectivity change to READY        {"grpc_log": true}

Tried the follow troubleshoot:

docker run -it --rm signoz/troubleshoot checkEndpoint --endpoint=192.168.200.20:4317
2022-10-22T01:40:57.763Z	INFO	troubleshoot/main.go:28	STARTING!
2022-10-22T01:40:57.764Z	INFO	checkEndpoint/checkEndpoint.go:41	checking reachability of SigNoz endpoint
2022-10-22T01:40:57.776Z	INFO	troubleshoot/main.go:46	Successfully sent sample data to signoz ...

2022-10-24T09:49:29.995Z	error	helper/transformer.go:110	Failed to process entry	{"kind": "receiver", "name": "filelog/k8s", "pipeline": "logs", "operator_id": "parser-nucash-regex", "operator_type": "regex_parser", "error": "regex pattern does not match", "action": "send", "entry": {"observed_timestamp":"2022-10-24T09:49:29.995336396Z","timestamp":"2022-10-24T09:49:29.908076398Z","body":"{\"component\":\"aaa\",\"env\":\"dev\",\"file\":\"/go/src/bitbucket.org/nucashin/sample/main.go:73\",\"func\":\"main.main\",\"level\":\"info\",\"msg\":\"Something\",\"service\":\"aaa\",\"time\":\"2022-10-24T09:49:29Z\"}\n","attributes":{"k8s.container.name":"sample","k8s.container.restart_count":"0","k8s.namespace.name":"services","k8s.pod.name":"sample-6c879cc484-6vmbr","k8s.pod.uid":"94b40e31-b854-4af4-9a83-f4daadadd951","log.file.path":"/var/log/pods/services_sample-6c879cc484-6vmbr_94b40e31-b854-4af4-9a83-f4daadadd951/sample/0.log","log.iostream":"stderr","time":"2022-10-24T09:49:29.908076398Z"},"severity":0,"scope_name":""}}

Getting this error in k8s-agent below is the updated configmap for k8s-agent, any idea why ..

- from: attributes.restart_count
          to: attributes["k8s.container.restart_count"]
          type: move
        - from: attributes.uid
          to: attributes["k8s.pod.uid"]
          type: move
        - from: attributes.log
          to: body
          type: move
        - id: parser-nucash-regex
          parse_from: body
          regex: ^"(?P<nucash_json>{.*})\\n"
          type: regex_parser
        - id: parser-nucash-json
          parse_from: attributes.nucash_json
          type: json_parser

is anyone try logs in Kubernetes?

@Apoorva can i see how you try it?

I am trying to install SigNoz on Kubernetes (self-hosted cluster). I see that the Helm charts have created 4 PersistentVolumeClaims which are all in Pending state. The documentation does not seem to provide any information about if (and how) I should create the volumes it needs. Can anyone point me in the right direction?

@Srikanth Chekuri

WARNING: Published ports are discarded when using host network mode
ERROR:opentelemetry.launcher.configuration:Invalid configuration: token missing. Must be set to send data to <https://ingest.lightstep.com:443>.Set environment variable LS_ACCESS_TOKEN
ERROR:opentelemetry.launcher.configuration:application instrumented via opentelemetry-instrument. all required configuration must be set via environment variables
Traceback (most recent call last):
  File "/opt/venv/lib/python3.9/site-packages/opentelemetry/launcher/configuration.py", line 383, in _configure
    configure_opentelemetry(_auto_instrumented=True)
  File "/opt/venv/lib/python3.9/site-packages/opentelemetry/launcher/configuration.py", line 237, in configure_opentelemetry
    raise InvalidConfigurationError(message)
opentelemetry.launcher.configuration.InvalidConfigurationError: Invalid configuration: token missing. Must be set to send data to <https://ingest.lightstep.com:443>.Set environment variable LS_ACCESS_TOKEN

This is a harmless log but I wanted to know why is it happening? I have not used configure_opentelemetry in my code anywhere

Now I am instrumenting another KIND of python application which runs from celery as bootable. The only documentation i could find was https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/celery/celery.html So I did the following Dockerfile:

opentelemetry-distro==0.34b0 \
    opentelemetry-exporter-otlp==1.13.0 \
    opentelemetry-launcher==1.9.0 \
    opentelemetry-instrumentation-celery==0.34b0 \
    

RUN opentelemetry-bootstrap --action=install



CMD opentelemetry-instrument --traces_exporter otlp_proto_grpc celery -A wombo.celery_paint.celeryapp worker --loglevel=info --pool=threads

code:

@worker_process_init.connect(weak=False)
def init_celery_tracing(*args, **kwargs):
    """
    When tracing a celery worker process, tracing and instrumention both must be initialized after the celery worker
    process is initialized. This is required for any tracing components that might use threading to work correctly
    such as the BatchSpanProcessor. Celery provides a signal called worker_process_init that can be used to
    accomplish this
    """
    CeleryInstrumentor().instrument()

celeryapp = Celery('paints')
celeryapp.conf.task_default_queue = sqsurl

Not sure what else to do?

Hi, I am trying to setup Signoz on a clean Ubuntu 22.04 VM, 6gb of ram, 50gb storage, runniing sudo ./install gets the following error:

Creating network "clickhouse-setup_default" with the default driver
Creating hotrod                        ... done
Creating clickhouse-setup_clickhouse_1 ... done
Creating load-hotrod                   ... done

ERROR: for otel-collector-metrics  Container "f5fa3ef5a7a3" is unhealthy.

ERROR: for otel-collector  Container "f5fa3ef5a7a3" is unhealthy.

ERROR: for query-service  Container "f5fa3ef5a7a3" is unhealthy.
ERROR: Encountered errors while bringing up the project.
Waiting for all containers to start. This check will timeout in 1 seconds ....

+++++++++++ ERROR ++++++++++++++++++++++
🔴 The containers didn't seem to start correctly. Please run the following command to check containers that may have errored out:

sudo docker-compose -f ./docker/clickhouse-setup/docker-compose.yaml ps -a
Please read our troubleshooting guide <https://signoz.io/docs/deployment/docker/#troubleshooting-of-common-issues>
or reach us on SigNoz for support <https://signoz.io/slack>
++++++++++++++++++++++++++++++++++++++++
🔴 The containers didn't seem to start correctly. Please run the following command to check containers that may have errored out:

sudo docker-compose -f ./docker/clickhouse-setup/docker-compose.yaml ps -a
or reach us for support in #help channel in our Slack Community <https://signoz.io/slack>
++++++++++++++++++++++++++++++++++++++++

I figure this must be something really basic, I tried a debian VM to see if it made a difference but getting the same result. Any help much appreciated.

hi, I am new in the community, and thanks for the nice APM, I have a question, do you know the reason the names of apps at Map Services is not display completly? Signoz v v0.11.2 Regards.

Hi SigNoz, we’re excited to try out your platform! We have a FastAPI service where we are logging with structlog. Is there an easy way to connect these logs to SigNoz?

Out of the boz the celery instrumentation doesn't seem to give number of tasks picked up etc. How to add those?

Hi, regarding the environment variables

OTEL_EXPORTER_OTLP_ENDPOINT=127.0.0.1:4317 \

OTEL_RESOURCE_ATTRIBUTES=service.name=graphql-service

Can we set these inside of the code somehow instead of in the terminal? For example inside the config object of

OTLPTraceExporter

?

Hi, I was testing how the

OTEL_EXPORTER_OTLP_ENDPOINT=127.0.0.1:4317

variable worked, in the guide it says we are suppose to set it locally in the terminal. I tried to set it as a different port and see if signoz can still detect data coming from my graphql server. I was expecting it to break (not detect the service) but it was still able to detect, why is this? Is it overriding the incorrect port I set in the terminal and using some kind of default value? Thanks.

I am facing an issue while installing signoz... My container alert manager is not running properly....I have installed using install script ./install.sh....And for this reason am not able to view signoz UI.Pls help me in solving out this issue.I have tried even after uninstalling signoz and reinstalled..But still shows me the same issue