If I tried replacing signoz alert manager with native prometheus alert manager, would that work?

okay anyone struggling with the above issue, the following setting works on k8s signoz version: 0.12.1 helm override:

global:

storageClass: gp2

clickhouse:

enabled: true

cloud: aws

installCustomStorageClass: true

allowedNetworkIps:

- "10.0.0.0/8"

- "0.0.0.0/0"

- "100.64.0.0/10"

- "172.16.0.0/12"

- "192.0.0.0/24"

- "198.18.0.0/15"

- "192.168.0.0/16"

alertmanager:

name: "alertmanager"

enabled: true

replicaCount: 1

extraArgs:

cluster.advertise-address: 0.0.0.0:9093

Im trying to setup alert rules in signoz but there isnt showing any y axis i have to set to miliseconds for latency

Where an application runs on Kubernetes but does not have OTEL support for emitting logs (like anything written in Rust), it seems to me that the way to get the log data into SigNoz is via the k8s-infra collectors. If I choose to make Rust apps output in JSON and generally make all apps output in JSON, then I have a consistent logging method to stdout. However the filtering and extraction of attributes from JSON objects, like can be done here https://github.com/SigNoz/charts/blob/main/charts/k8s-infra/values.yaml#L94-L156 would only be successful if the same rule can be generally applied to all received logs. There is no application-specific interpretation of attributes in JSON methods and the timestamp extraction for parser-crio and parser-containerd demonstrates that because the regexes appear to be tried in order until something succeeds seems to work by luck rather than targeted at a specific application. The rules for renaming attributes would only work if the attribute names are semantically consistent across all applications running on a cluster. Even the routing choice here is super-vague by assuming (incorrectly) that any line beginning with a open-curly-brace is in some parse-able JSON format: https://github.com/SigNoz/charts/blob/main/charts/k8s-infra/values.yaml#L96-L104 It seems that the most reasonable solution is to avoid using k8s-infra and only use FluentBit, since one can then be explicit about the parser in the Pod configs and then pair accurate parsing with the expected Pod output: https://docs.fluentbit.io/manual/pipeline/filters/kubernetes#suggest-a-parser

Hi, We're trying this stack to have logs, traces, metrics in one ui. Currently we use kube prometheus stack, I see the defaults are good, shipping metrics and logs is very easy from otel but we're missing some metrics of Prometheus service monitors and pod monitors. Is there any way to push service monitors to signoz? Or is there any alternative which signoz provides?

Hi team, we have recently enabled host metrics for our service containers, and started receiving the CPU data like this, I am not sure what does F1 represent in this graph, can someone guide me to righht resources

CleanShot 2023-04-05 at 13.56.10.png

Is there a guide somewhere on setting up alerts for exceptions? I have exception logging working and I created an alert for exceptions, but the alert that I receive in Slack is more of a metric than the specific exception.

Does anyone deployed SigNoz on to OpenShift?

Hi Team, I want to build a proper monitoring and logging system in Go for all the microservices. Anyone, please refer me to the blog or tutorial so that I can get started?

Hii team , want to alert for a https endpoint in signoz Any referral or blog ?

Hello Team, I am trying to collection logs are metrics from my eks cluster to SigNoz instance on a VM. When i follow doc from this url https://signoz.io/docs/tutorial/kubernetes-infra-metrics/ I get the error “Error: INSTALLATION FAILED: repo signoz not found”. I will be glad if you could help

Hi Team, I am installing signoz with external clickhouse server. I am facing an issue. Getting below error:

kubectl logs -f my-release-signoz-otel-collector-54bf55dbb9-ckqmz

Defaulted container "my-release-signoz-otel-collector" out of: my-release-signoz-otel-collector, my-release-signoz-otel-collector-init (init)
2023-04-07T02:08:48.916Z	info	service/telemetry.go:111	Setting up own telemetry...
2023-04-07T02:08:48.916Z	info	service/telemetry.go:141	Serving Prometheus metrics	{"address": "0.0.0.0:8888", "level": "Basic"}
2023-04-07T02:08:48.916Z	info	components/components.go:30	Stability level of component is undefined	{"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "stability": "Undefined"}
Error: cannot build pipelines: failed to create "clickhousetraces" exporter, in pipeline "traces": error connecting to primary db: code: 170, message: Requested cluster 'cluster' not found
2023/04/07 02:08:48 application run finished with error: cannot build pipelines: failed to create "clickhousetraces" exporter, in pipeline "traces": error connecting to primary db: code: 170, message: Requested cluster 'cluster' not found

My override file:

global:
  storageClass: gp2
  cloud: aws

clickhouse:
  enabled: false
  cloud: other


externalClickhouse:
  host: 54.75.80.150
  cluster: events 
  user: xxxxxx
  password: xxxxxxxxxxxxxxx
  secure: false
  verify: false
  httpPort: 8123
  tcpPort: 9000

allowedNetworkIps:
    - "0.0.0.0/0"

I really appreciate your kind response.

Hello! I’m translating a grafana dashboard into signoz. I want to import PromEx dashboards but variables aren’t working as expected. How do I translate this promql query to clickhouse query?

label_values(gitstart_engine_prom_ex_prom_ex_status_info, job)

SigNoz frontend is leaking... container_id is multiplied more and more....

how do I actually filter the logs of the very SigNoz out?

Ok. I need a help to setup my logs to be collected into SigNoz. I've tried https://signoz.io/docs/userguide/collect_docker_logs/#steps-for-collecting-logs-if-signoz-is-running-on-a-different-host and it seemed doesn't collect logs. How can I make sure it works?

Hi Team, our Signoz server went down today morning, I tried restarting with the

.install.sh

script, but getting error on restarting the services

#support Hi Team, iam doing signoz setup through kubernetes as initial setup i tried to bring up the following service 1. query-service 2. otel-collector-metrics 3. otel-collector 4. load-hotrod 5. frontend 6. clickhouse 7. alert-manager in this otel-collector-metrics , otel-collector and load-hotrod were not running i am facing issue , requested cluster not found logs from the otel-collector service 2023-04-06T104316.441Z info service/telemetry.go:111 Setting up own telemetry... 2023-04-06T104316.441Z info service/telemetry.go:141 Serving Prometheus metrics {"address": "0.0.0.0:8888", "level": "Basic"} 2023-04-06T104316.441Z info components/components.go:30 Stability level of component is undefined {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "stability": "Undefined"} time="2023-04-06T104316Z" level=info msg="Executing:\nCREATE DATABASE IF NOT EXISTS signoz_metrics ON CLUSTER cluster\n" component=clickhouse 2023/04/06 104316 Error creating clickhouse client: code: 170, message: Requested cluster 'cluster' not found alert-manager-8b4799c48-xtgfj 1/1 Running 0 3d19h clickhouse-statefulset-0 1/1 Running 0 3d17h clickhouse-statefulset-1 1/1 Running 0 3d17h clickhouse-statefulset-2 1/1 Running 0 3d17h frontend-688999b84f-dx6nh 1/1 Running 0 3d19h hotrod-56bff776c-plwlp 1/1 Running 0 4d load-hotrod-55cdc96d86-gbqwx 0/1 CrashLoopBackOff 686 (3m12s ago) 3d18h otel-collector-5bbbf57ff9-tcsc9 0/1 CrashLoopBackOff 671 (3m31s ago) 3d17h otel-collector-metrics-6b6547d5c9-m2s8w 0/1 CrashLoopBackOff 689 (2m37s ago) 3d18h signoz-query-service-85b85c98c6-vb6rq 1/1 Running 11 (7h47m ago) 3d19h

Hii team Need to do exceptions based alerting Can you share some clickhouse query for status code 400

Hi team, I am using community edition signoz updated to latest 0.18.1 and log, metrics, traces (7 day retention) set with s3 bucket credentials from past 2 months in the file

signoz/deploy/docker/clickhouse-setup/clickhouse-storage.xml

<s3>
<type>s3</type>
<endpoint><https://BUCKET-NAME.s3.amazonaws.com/data/></endpoint>
<access_key_id>ACCESS-KEY-ID</access_key_id>
<secret_access_key>SECRET-ACCESS-KEY</secret_access_key>
</s3>

however I dont see any files created in my s3 bucket. Is there anything I am doing wrong here? Running signoz with docker on aws t3a.2xlarge instance

Is anyone using any query to monitor memory leakage in an application through dashboards?

or maybe any tuning

let me know please

hey team, I am using redis_exporter for exposing redis metrics and collecting them with signoz with the below config defined in

signoz/deploy/docker/clickhouse-setup/otel-collector-metrics-config.yaml

# Redis server metrics
        - job_name: redis_exporter
          scrape_interval: 5s
          scheme: http
          static_configs:
            - targets:
                - 10.0.0.105:9121
                # - 10.0.1.170:9121

Using Signoz with docker on AWS t3a.2xlarge and I am able to collect all the metrics and have a visualisation dashboard for metrics data from instance on ip

10.0.0.105

however when I uncomment the ip below it (

10.0.1.170

) then the perf metrics of both the redis instance mixing in the clickhouse (im assuming this is happening because both uses same exporter) and there’s no distinction tag which I can use to separate the metrics stored in clickhouse to create a separate visualisation or pass the ip as parameter in existing dashboard. dashboard was made using query builder and where clause only giving

__name__

as the tag to create distinction which doesn’t provide an appropriate value at all. Expected behaviour : I should be able to separate the metrics data for visualisation based on ip/ job_name defined in config

Hello, can I change the front-end port to run on port 80 and have cloudflare DNS work with a subdomain? example: romario.mydomain.com -> signoz frontend I’m want to use cloudflare proxying to have TLS configured automatically

My frontend dashboard stopped displaying all metric graphs / trace graphs after a certain point in time. No issues with collector / query service as far as im concerned as logs look good and I am getting alert notifications on metrics I am emitting. Restarting the service doesnt help, any idea on how to troubleshoot this? Some info: signoz version: v0.15.0 running via docker-compose

Hello Team, I have installed signoz on kubernetes eks, but I am facing an issue, I am not getting all the logs from pods. Some logs are missing. I am using external clickhouse. All signoz components are running fine. signoz agent is running on every k8s node. I have verified by logging into node. I also tried to add logs in below files: 1. echo "2023-04-11T083921.052769633Z stdout F ---2023-04-11 083921.052 - INFO 7 --- [trap-executor-0] [] c.n.d.s.r.aws.ConfigClusterResolver : : Resolving otel testing via lolo otel-agent hassan_2" >> /var/log/containers/ems-func-cfg-api-call-svc-deployment-546df9d56c-qbxgs_ems_ems-func-cfg-api-call-svc-9533ac728710a67addc435de77a62414996911383d0faa6bd8b509ba1707e326.log 2. echo "2023-04-11T083921.052769633Z stdout F ---2023-04-11 083921.052 - INFO 7 --- [trap-executor-0] [] c.n.d.s.r.aws.ConfigClusterResolver : : Resolving otel testing via lolo otel-agent hassan_3" >> /var/log/pods/ems_ems-func-cfg-api-call-svc-deployment-546df9d56c-qbxgs_af9c33b0-c30e-4ff6-b89c-02a3fe84dc6b/ems-func-cfg-api-call-svc/0.log but nothing is reflected

Hi Team, i am using external clickhouse.. Signoz loads the logs with a lot of delay. Is there something with clickhouse server? the page shows the data with a lot of delay. Is there any recommended specs for clickhouse server?