https://signoz.io logo
Powered by
#support
Title
# support
v

Vinayak Singh

01/19/2023, 3:39 AM
Hi Team, I've installed signoz via helm chart. Initially there is a problem in the alertmanager pod because its not able to get private IP. I added pod_ip to extraargs and alertmanager pod starts running again. Now the problem is when I'm trying to create alert I'm not able to see any metrics graph. It says no data to display, any help on this. For system metrics, data is coming but for cloudwatch metrices it's not coming. While creating the dashboard I'm able to create dashboard for cloudwatch metrices
a

Ankit Nayan

01/19/2023, 4:29 AM
this should not be the case. When creating alerts on cloudwatch metrics, can you share the payload of the request and check if it shows failing in the network tab
cc: @Amol Umbark
v

Vinayak Singh

01/19/2023, 4:36 AM
Calls are not failing, it's 200 and the payload also looks fine
Screenshot from 2023-01-19 10-08-20.png,Screenshot from 2023-01-19 10-08-08.png
a

Ankit Nayan

01/19/2023, 4:40 AM
can you share the json payload and not the screenshot?
v

Vinayak Singh

01/19/2023, 4:42 AM
Copy code
{
  "start": 1674102564000,
  "end": 1674102864000,
  "step": 60,
  "variables": {},
  "dataSource": 1,
  "compositeMetricQuery": {
    "queryType": 1,
    "panelType": 1,
    "builderQueries": {
      "A": {
        "queryName": "A",
        "aggregateOperator": 5,
        "metricName": "aws_rds_cpuutilization_average",
        "tagFilters": {
          "op": "AND",
          "items": []
        },
        "groupBy": [
          "dbinstance_identifier"
        ],
        "expression": "A",
        "disabled": false
      }
    }
  }
}
a

Ankit Nayan

01/19/2023, 4:43 AM
I will let @Amol Umbark dive deeper into this
a

Amol Umbark

01/19/2023, 4:58 AM
@Vinayak Singh Do you see alert found message with count > 0 when you click test alert button . is this the problem only with the chart?
can you please screenshot or payload for dashboard as well. Try creating a new panel and entering same filters, in network tab you will see a call to query range api. please share the payload
v

Vinayak Singh

01/19/2023, 5:04 AM
No. when i click on test notification. it says, No alerts found on evaluation This happens when rule condition is unsatisfies
This is the request json {"start":1674102976000,"end":1674104776000,"step":60,"variables":{"SIGNOZ_START_TIME":1674102976000,"SIGNOZ_END_TIME":1674104776000},"dataSource":1,"compositeMetricQuery":{"queryType":1,"panelType":1,"builderQueries":{"A":{"queryName":"A","aggregateOperator":5,"metricName":"aws_rds_database_connections_maximum","tagFilters":{"items":[],"op":"AND"},"groupBy":[],"expression":"A","disabled":false}}}}
This is the response {"status":"success","data":{"resultType":"matrix","result":[{"queryName":"A","metric":{},"values":[[1674103080,"4.111111111111111"],[1674103140,"3.3333333333333335"],[1674103200,"3.4444444444444446"],[1674103260,"3.3333333333333335"],[1674103320,"3.4444444444444446"],[1674103380,"4"],[1674103440,"3.3333333333333335"],[1674103500,"3.6666666666666665"],[1674103560,"3.5555555555555554"],[1674103620,"3.4444444444444446"],[1674103680,"3.4444444444444446"],[1674103740,"3.3333333333333335"],[1674103800,"3.5555555555555554"],[1674103860,"3.5555555555555554"],[1674103920,"3.7777777777777777"],[1674103980,"3.3333333333333335"],[1674104040,"3.3333333333333335"],[1674104100,"3.4444444444444446"]]}]}}
one more thing for system metrics its working fine but for cloudwatch metrics its not working
a

Amol Umbark

01/19/2023, 5:25 AM
yeah, got it. Thanks. Let me review and get back.
Can you please create exact same query. The metric is different, this could also be cause of no -data. Notice the metric name in these two queries. It is different. JSON from Alerts: 
{"start":1674102976000,"end":1674104776000,"step":60,"variables":{"SIGNOZ_START_TIME":1674102976000,"SIGNOZ_END_TIME":1674104776000},"dataSource":1,"compositeMetricQuery":{"queryType":1,"panelType":1,"builderQueries":{"A":{"queryName":"A","aggregateOperator":5,"metricName":"*aws_rds_database_connections_maximum*","tagFilters":{"items":[],"op":"AND"},"groupBy":[],"expression":"A","disabled":false}}}}
JSON from Dashboard: 
{"start":1674102564000,"end":1674102864000,"step":60,"variables":{},"dataSource":1,"compositeMetricQuery":{"queryType":1,"panelType":1,"builderQueries":{"A":{"queryName":"A","aggregateOperator":5,"metricName":"*aws_rds_cpuutilization_average*","tagFilters":{"op":"AND","items":[]},"groupBy":["dbinstance_identifier"],"expression":"A","disabled":false}}}}
v

Vinayak Singh

01/19/2023, 5:34 AM
json from dashboard {"start":1674104601000,"end":1674106401000,"step":60,"variables":{"SIGNOZ_START_TIME":1674104601000,"SIGNOZ_END_TIME":1674106401000},"dataSource":1,"compositeMetricQuery":{"queryType":1,"panelType":1,"builderQueries":{"A":{"queryName":"A","aggregateOperator":5,"metricName":"aws_rds_database_connections_maximum","tagFilters":{"items":[],"op":"AND"},"groupBy":[],"expression":"A","disabled":false}}}}
@Amol Umbark any luck ?
a

Amol Umbark

01/19/2023, 12:29 PM
hey @Vinayak Singh can we get into a quick huddle ?
v

Vinayak Singh

01/19/2023, 12:29 PM
sure
a

Amol Umbark

01/19/2023, 12:34 PM
sent you a request
a

Ankit Nayan

01/19/2023, 1:33 PM
Is this fixed now?
v

Vinayak Singh

01/19/2023, 1:35 PM
yes @Ankit Nayan that problem is fixed. its because of the delay in incoming metrics. Right now im pulling cloudwatch metrics from prometheus exporter and there is a delat of 10-12 min and that is why default 5 min queries are not working
a

Ankit Nayan

01/19/2023, 1:36 PM
oh..ok
v

Vinayak Singh

01/20/2023, 3:05 AM
Actually im using the prometheus exporter for exporting cloudwatch metrices but at the otel collector metrics i'm getting this failed to scrape prometheus endpoint also sometimes its working sometime its not. Any help
This is what ive added in the values.yml
Copy code
- job_name: "CW-exporter"
              scrape_interval: 60s
              static_configs:
                - targets: ["prometheus-cloudwatch-exporter.platform.svc.cluster.local:9106"]
when i did the curl prometheus cloudwatch exporter i can see the cloudwatch metrices
@Prashant Shahi Any idea or something im not doing correcty?
One more thing how to enable a prometheus receiver in signoz helm chart.?
@Ankit Nayan can you help?
p

Prashant Shahi

01/20/2023, 6:13 AM
sometimes its working sometime its not
have you verified that metrics are scraped and accessible from Dashboard in SigNoz UI?
OtelCollector Metrics is the right place to include new scraper job. You would not need new prometheus receiver or pipeline.
@Vinayak Singh You can opt for scraping using pod annotations: 
<http://signoz.io/scrape|signoz.io/scrape>
and 
<http://signoz.io/port|signoz.io/port>
v

Vinayak Singh

01/20/2023, 6:16 AM
hi i wanted to show cloudwatch metrices, so i configured the prometheus cloudwatch exporter. I can see all the cloudwatch metrices in the prometheus but then i followed this https://signoz.io/docs/userguide/send-metrics/#enable-a-prometheus-receiver but this prometheus reciever is not able to scrape the metrices
p

Prashant Shahi

01/20/2023, 6:20 AM
@Vinayak Singh can you make sure that exporter endpoint is correct?
v

Vinayak Singh

01/20/2023, 6:22 AM
yes its correct
how to add the details in values.yaml
Copy code
config:
          scrape_configs:
            - job_name: "CWexporter"
              scrape_interval: 120s
              static_configs: - targets: ["192.20.12.111:9106"]
is this what we need to add in docker setup. What is the kubernative alternative for this
It looks like something wrong with the exporter. Something related to the syntax. Sometime its working sometime not
One more help can you guys let me know what this log means
Copy code
skipping send alert due to resend delay%!(EXTRA string=	 rule: , string=ES/Opensearch CPUUtilization, string=	 alert:, labels.Labels={alertname="ES/Opensearch CPUUtilization", domain_name="test-store-staging-new", ruleId="1", ruleSource="<https://monitoring.test.com/alerts/edit?ruleId=1>", severity="warning"})
What this resend delay is?
a

Amol Umbark

01/23/2023, 6:43 AM
it means the alert has already been sent. it waits for sometime (30s i think) before resending it.
v

Vinayak Singh

01/24/2023, 9:59 AM
okay thanks can you also let me know whether we can integrate ops-genie with this signoz? @Amol Umbark @Prashant Shahi
p

Prashant Shahi

01/24/2023, 10:11 AM
@Vinayak Singh we do not have direct integration with it yet. You will have to use custom webhook to achieve it.
v

Vinayak Singh

01/24/2023, 10:12 AM
can you share few links?
p

Prashant Shahi

01/24/2023, 10:12 AM
It looks like something wrong with the exporter. Something related to the syntax. Sometime its working sometime not
If it works sometime, it means the endpoint is correct. Not sure about issue with Otel exporter. @Srikanth Chekuri might be able to help with it.
v

Vinayak Singh

01/24/2023, 10:14 AM
okay thanks
s

Srikanth Chekuri

01/24/2023, 10:22 AM
It looks like something wrong with the exporter. Something related to the syntax. Sometime its working sometime not
Not clear to me, what syntax are you talking about here. What is working sometimes and not working?
v

Vinayak Singh

01/24/2023, 10:25 AM
this is what im using in the prometheus clodwatch exporter
Copy code
config: |-
  # This is the default configuration for prometheus-cloudwatch-exporter
  region: us-east-1
  period_seconds: 240
  metrics:
  - aws_namespace: AWS/ES
    aws_metric_name: OpenSearchRequests
    aws_dimensions: [ClientId, DomainName]
    aws_statistics: [Sum]

  - aws_namespace: AWS/ES
    aws_metric_name: DeletedDocuments
    aws_dimensions: [ClientId, DomainName]
    aws_statistics: [Sum]

  - aws_namespace: AWS/ES
    aws_metric_name: FreeStorageSpace
    aws_dimensions: [ClientId, DomainName]
    aws_statistics: [Average]

  - aws_namespace: AWS/ES
    aws_metric_name: ClusterUsedSpace
    aws_dimensions: [ClientId, DomainName]
    aws_statistics: [Average]

  - aws_namespace: AWS/ES
    aws_metric_name: ElasticsearchRequests 
    aws_dimensions: [ClientId, DomainName]
    aws_statistics: [Sum]

  - aws_namespace: AWS/ES
    aws_metric_name: CPUUtilization
    aws_dimensions: [ClientId, DomainName]
    aws_statistics: [Average]

  - aws_namespace: AWS/ES
    aws_metric_name: SearchableDocuments
    aws_dimensions: [ClientId, DomainName]
aws_statistics: [Average]
But when im tryiing to add more metrices like RDS its not working. And also is there any way where i can flush the old data from the clickhouse DB
s

Srikanth Chekuri

01/24/2023, 10:28 AM
I am not very familiar with the config of the cloudwatch exporter. You can set TTL in the settings of SigNoz application or you can manually truncate once if you want to start fresh.
25 Views
Powered by