We are doing the same thing. How we set it up:
• SigNoz cluster with an ingest DNS routing traffic to the signoz collector
◦ This takes OTEL data and transforms it into SigNoz ClickHouse to all your data is saves and can be plotted on things like dashboards, alerted on, etc.
• Our other clusters "ship" OTEL data to this collector. We just set the host to that ingress DNS from above. This is done in 3 different flavors:
◦ k8s-signoz chart installed collector to ship k8s metrics for the cluster (node counts, pod states, pod logs, etc.)
◦ annotated pods to "inject" a sidecar that ships APM data
◦ OTEL SDK baked into our apps that ship APM data as well as custom metrics