Hi All, I'm using the docker setup for a self-hosted SigNoz and I'm having 2 problems that I could...
r
Hi All, I'm using the docker setup for a self-hosted SigNoz and I'm having 2 problems that I could use some help with! 1. Creating alerts is giving me an error - does this feature work for self-hosted SigNoz? I've setup email via the SMTP credentials but the notification tests are also erroring; has anyone been able to get this setup and working in self hosted? 2. I'm trying to get my SigNoz instance to send itself OTEL data (logs, metrics, exceptions, etc...) and it's not working, is there default OTEL instrumentation setup in SigNoz? Do y'all use SigNoz to self-monitor? Thank you for any help/insights you can provide!
b
Btw, do you see any issues with this? The most my custom metrics are not showing up on my self hosted version. Checked metrics, they are not shown. They might appear time to time when I redeploy my service. The same metrics and dashboards are working pretty well with Signoz Cloud though.
@Hien Le what is the best way of debugging where the data is getting dropped?
Some data points: • I use Auto instrumentation. • All my traces appear on both (Signoz Cloud and Self hosted) servers • Signoz Cloud: All my Metrics appear on Cloud • Self Hosted: Metrics appear for a second during deployment and no more data • I have literally the same Dashboards on Signoz Cloud and Self Hosted one
Another interesting observation: All the autoinstrumented metrics is getting ingested. And I see the data
h
Whenever I'm testing processor changes, I update the OTel config on all my collectors to show every event:
Copy code
exporters:
  debug:
    verbosity: normal # basic: aggregate count, normal: one line per telemetry record, detailed: peek at attributes

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [filter/drop_http_duration, attributes/upsert, resource/upsert, batch]
      exporters: [debug]
That's all I can think of.
Suggest also posting the code that's emitting your custom metric. That'd help the Signoz engineers (not me) see if you're using an incorrect type, name, or temporality.