signoz-community #support

@Juha Patrikainen nobody reported this till now. This works fine in the latest version. Possible to switch to the latest version? Meanwhile we will test

v0.12.0

David Bronke

01/09/2023, 12:08 PM

I'm trying to get started using the

Deploying to AWS

installation instructions with no modifications aside from naming my release

signoz-stack

, but I'm getting stuck at the Verify the Installation step; the

frontend

otel-collector

, and

otel-collector-metrics

pods have been stuck at

Init:0/1

for almost an hour now. Looking at the logs, it seems like

otel-collector

is having issues connecting to clickhouse, even though

clickhouse-operator

says

Running

and seems OK looking at its logs. There is a

chi-signoz-stack-clickhouse-cluster-0-0-0

pod, but it's still in

Pending

. Logs from the

signoz-stack-otel-collector-init

container:

Copy code

wget: can't connect to remote host (172.20.85.116): Connection refused
waiting for clickhouseDB
wget: can't connect to remote host (172.20.85.116): Connection refused
waiting for clickhouseDB

Running

signoz/troubleshoot

in the

platform

namespace gives:

Copy code

Error: not able to send data to SigNoz endpoint ...
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.20.58.69:4317: i/o timeout"

which I think is just because

otel-collector

isn't running yet.

Fabbio Borges

01/09/2023, 8:10 PM

how to start the tracing.js using pm2?

Paulo Henrique de Morais Santiago

01/10/2023, 9:53 PM

Hello! I have some doubt about Alert Channels to Slack. I am running signoz on K8s installed with helm chart. 1 - Who are the pod that send the request to hooks.slack.com ? Is the front end or query-service? 2 - I think my problem is because we need to use proxy here to request outside internet. I configured this envrionments on my front end - name: HTTPS_PROXY value: “http://myproxy” - name: HTTP_PROXY value: “http://myproxy”

Paulo Henrique de Morais Santiago

01/10/2023, 10:02 PM

Hello! I have some doubt about Alert Channels to Slack. I am running signoz on K8s installed with helm chart. 1 - Who are the pod that send the request to hooks.slack.com ? Is the front end or query-service? 2 - I think my problem is because we need to use proxy here to request outside internet. I configured this envrionments on my front-end pod . And after i make a curl post direct from frontend pod shell and works well on hooks.slack.com.

Copy code

- name: HTTPS_PROXY
              value: "<http://myproxy>"
            - name: HTTP_PROXY
              value: "<http://myproxy>"

But using the signoz UI the slack test still dont working. Looking for logs i see a request sended to query service. So after, i put the same envs HTTPS_PROXY and HTTP_PROXY on query-service pod. And strange, after this the front-end UI page alert channels become give only 500 error. I cant save anymore the configs. See the images on below the error after i configure proxys envs on query service pod. Resume: The notification using slack channel dont works here.

Arvin Corpuz

01/11/2023, 2:37 PM

hi. im trying to disable automatic log collection on k8s following this post - https://signoz.io/docs/userguide/collect_kubernetes_pod_logs/ but still im receiving logs from kubernetes pods. here is my otel collector config

Copy code

$ kubectl exec -it signoz-otel-collector-54889ffcc8-rz86z sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
W0111 22:29:10.199861   77630 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult <https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke>
Defaulted container "signoz-otel-collector" out of: signoz-otel-collector, signoz-otel-collector-init (init)
/ $ cat /conf/otel-collector-config.yaml 
exporters:
  clickhouselogsexporter:
    dsn: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_elapsed_time: 300s
      max_interval: 30s
    sending_queue:
      queue_size: 100
    timeout: 10s
  clickhousemetricswrite:
    endpoint: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}
    resource_to_telemetry_conversion:
      enabled: true
  clickhousetraces:
    datasource: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_TRACE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}
  prometheus:
    endpoint: 0.0.0.0:8889
extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  pprof:
    endpoint: localhost:1777
  zpages:
    endpoint: localhost:55679
processors:
  batch:
    send_batch_size: 50000
    timeout: 1s
  memory_limiter: null
  resourcedetection:
    detectors:
    - env
    - system
    system:
      hostname_sources:
      - os
    timeout: 2s
  signozspanmetrics/prometheus:
    dimensions:
    - default: default
      name: service.namespace
    - default: default
      name: deployment.environment
    dimensions_cache_size: 100000
    latency_histogram_buckets:
    - 100us
    - 1ms
    - 2ms
    - 6ms
    - 10ms
    - 50ms
    - 100ms
    - 250ms
    - 500ms
    - 1000ms
    - 1400ms
    - 2000ms
    - 5s
    - 10s
    - 20s
    - 40s
    - 60s
    metrics_exporter: prometheus
receivers:
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu: {}
      disk: {}
      filesystem: {}
      load: {}
      memory: {}
      network: {}
  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
      thrift_http:
        endpoint: 0.0.0.0:14268
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 16
      http:
        endpoint: 0.0.0.0:4318
  otlp/spanmetrics:
    protocols:
      grpc:
        endpoint: localhost:12345
service:
  extensions:
  - health_check
  - zpages
  pipelines:
    logs:
      exporters:
      - clickhouselogsexporter
      processors:
      - batch
      receivers:
      - otlp
    metrics:
      exporters:
      - clickhousemetricswrite
      processors:
      - batch
      receivers:
      - otlp
    metrics/generic:
      exporters:
      - clickhousemetricswrite
      processors:
      - resourcedetection
      - batch
      receivers:
      - hostmetrics
    metrics/spanmetrics:
      exporters:
      - prometheus
      receivers:
      - otlp/spanmetrics
    traces:
      exporters:
      - clickhousetraces
      processors:
      - signozspanmetrics/prometheus
      - batch
      receivers:
      - jaeger
      - otlp
  telemetry:
    metrics:
      address: 0.0.0.0:8888

Nate Brunette

01/11/2023, 7:02 PM

I have basically a clean setup and I'm trying to use the troubleshooter and getting a timeout

Copy code

docker run -it --rm signoz/troubleshoot checkEndpoint --endpoint=172.17.0.1:4317

Nate Brunette

01/11/2023, 7:02 PM

Copy code

Error: not able to send data to SigNoz endpoint ...
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 172.17.0.1:4317: i/o timeout"

Nate Brunette

01/11/2023, 7:03 PM

I'm using the docker standalone locally

Srikanth Chekuri

01/11/2023, 7:37 PM

Can I access the time variable in Signoz for dashboard queries?

Yes, you could use

{{.SIGNOZ_START_TIME}}

and

{{.SIGNOZ_END_TIME}}

If not then what is it’s purpose for in the dashboard tab?

The configure tab allows you to define a variable whose values are queryable from ClickHouse.

Prim Mayora

01/11/2023, 7:44 PM

Hi anyone have face issues with more than 1 query service pod?

Alexei Zenin

01/12/2023, 10:14 PM

Does anybody know which tables these metrics are stored in? https://github.com/SigNoz/signoz-otel-collector/tree/main/processor/signozspanmetricsprocessor. I am trying to query them outside of SigNoz and create some alerts based on error rates, http latency etc. Any example queries that I could use to do these in Clickhouse SQL?

Travis Chambers

01/13/2023, 1:07 AM

Hello! I'm trying to set up a simple test of Prometheus/Node Exporter to demo internally to our company (so we can decide to start using SigNoz more generally). I'm running Node Exporter on a host and can confirm that I see Prometheus successfully getting those metrics. However, I can't seem to get SigNoz to receive those same metrics. I've followed this (https://signoz.io/docs/userguide/send-metrics/#enable-a-prometheus-receiver) and added the endpoint where Node Exporter is exposing these metrics to my

deploy/docker/clickhouse-setup/otel-collector-metrics-config.yaml

. However, I can't seem to find any of these metrics showing up in SigNoz. Could I get some help troubleshooting? more info in 🧵

Vinayak Singh

01/13/2023, 3:26 AM

Hi Team, i believe this is easy but I didn't find anything in documentation. I changed my network cidr and tried to install signoz in EKS, but it showing me the clickhouse error user: admin: authentication failed code: 195. DB:Exception Connection from xxxxx is not allowed ( IP address not allowed ) . How can I add my network cidr here also didn't find anything online ) Can somebody help me here?

oluchi orji

01/13/2023, 10:21 AM

Hi Team, my team set up auto-instrumentation for our applications on EKS, but we are not seeing any tracing data for python apps on the Signoz frontend UI except the

node.js

and

java

application. Is anything wrong, we are using the following guide here

Vinayak Singh

01/13/2023, 3:35 PM

Hi team, just one quick question is invite member feature working. I tried to invite my teammates into my implementation but the invite link not working?

Paulo Henrique de Morais Santiago

01/13/2023, 3:56 PM

Hello! Im stalling signoz with helm chart version 0.8.0 at GCP. But my queryservice pod dont become Running.

Copy code

Erros:
   Warning  Unhealthy  15s (x4 over 45s)  kubelet            Liveness probe failed: Get "<http://100.100.24.178:8080/api/v1/version>": dial tcp 100.100.24.178:8080: connect: connection refused      ││   Warning  Unhealthy  5s (x11 over 53s)  kubelet            Readiness probe failed: Get "<http://100.100.24.178:8080/api/v1/version>": dial tcp 100.100.24.178:8080: connect: connection refused

Apoorva

01/14/2023, 10:18 AM

Hi There, I was trying to create monitors based on logs, few observations 1. My signoz version is 14 but still distributed_logs table is not being used, all the data is being pushed in logs 2. Was trying to write query based on container name and namespace to create alerts, not able to do it. Can you please help

Apoorva

01/14/2023, 11:56 AM

Regarding version 14.0 in signoz, logs page seems really buggy Problems I am facing 1. Getting repeated attribute keys everywhere 2. Search not giving proper result

abbot

01/15/2023, 2:12 AM

@abbot has left the channel

akshay awate

01/15/2023, 6:45 AM

Hello folks, I have build the query-service, using

Copy code

cd pkg/query-service
go build -o build/query-service main.go
ClickHouseUrl=<tcp://localhost:9001> STORAGE=clickhouse build/query-service

when i run

./query-service

i get,

Copy code

/opt/homebrew/Cellar/go/1.19.4/libexec/src/runtime/proc.go:250
2023-01-15T12:12:16.139+0530    FATAL   query-service/main.go:68        Failed to create server{error 26 0  Storage type:  is not supported in query service}

Any ideas/suggestion would be helpful.

vipul vishwakarma

01/16/2023, 4:26 PM

can someone guide me on how to set up signoz

Thomas Wiradikusuma

01/16/2023, 5:07 PM

Hi guys, I already understand how to get my

/metrics

pulled by Signoz from reading https://signoz.io/docs/userguide/send-metrics/#enable-a-prometheus-receiver. Now, I'm using Fly.io, and they already have their own Prometheus (https://fly.io/docs/reference/metrics/#prometheus-on-fly). How can I make my Signoz instance query from Fly.io's Prometheus? Without resorting to federation.

Prim Mayora

01/16/2023, 8:40 PM

Hi team, there is some kind of api documentation or someone already automated the user management?

Sebastião Pamplona

01/16/2023, 11:13 PM

hey, great product guys 🙂 I’d like to ask something about setting up signoz on different clusters I’m using terraform and helm to deploy signoz I have a cluster

cluster-apm

to host signoz and 2 other clusters

cluster-1

and

cluster-2

I want to deploy only

signoz-k8s-infra-otel-agent

cluster-1

and

cluster-2

to collect metrics and push it to

cluster-apm

is this possible using the

values.yaml

? thanks

Jason Brewer

01/17/2023, 10:52 AM

Hi everyone! I'm trying to push a message to Slack whenever there are > N log messages of

level

===

error

. My JSON structured log looks like this:

Copy code

{
  "timestamp": 1673948790110000000,
  "id": "2KPcDvuJzmCPDve2CDoFaLib3lM",
  "trace_id": "",
  "span_id": "",
  "trace_flags": 0,
  "severity_text": "",
  "severity_number": 0,
  "body": "⚠️ Auth error: Missing Authorization header",
  "resources_string": {},
  "attributes_string": {
    "env": "dev",
    "level": "error",
    "log_file_path": "/tmp/logs/udx-services-ingress-out.log",
    "service": "ingress"
  },
  "attributes_int": {},
  "attributes_float": {}
}

These docs are a great start but it's not at all obvious to me how I write a query that rips out

attributes_string.level

and

attributes_string.env

I'm just playing around to see what is available in the query and so far (but no so good!) I've got this:

Copy code

select 
toStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL 35 MINUTE) AS interval
, toFloat64(count()) as value
, attributes_string_key[1]
FROM signoz_logs.distributed_logs  
WHERE timestamp BETWEEN {{.start_timestamp_nano}} AND {{.end_timestamp_nano}}
GROUP BY interval, attributes_string_key;

Which produces this Chart preview (see thread)

Vinayak Singh

01/17/2023, 1:23 PM

Hi Team, is there any issue with jedis and opentelemetry jar with auto instrumentation. My service running fine normally but when I'm trying run the application jar with opentelemetry jar it's getting cannot get jedis connection. Any known issues related to this?

Teymour

01/17/2023, 3:11 PM

Hey all, I am having trouble setting up the signoz otel collector (receiver side) with TLS. I'll add the config and error messages I am getting as responses to this message.

Timothy Wigginton

01/17/2023, 3:31 PM

Hey Team is there a way to load in dashboard config files automatically?