Hi, We're trying this stack to have logs, traces, metrics in one ui. Currently we use kube prometheus stack, I see the defaults are good, shipping metrics and logs is very easy from otel but we're missing some metrics of Prometheus service monitors and pod monitors. Is there any way to push service monitors to signoz? Or is there any alternative which signoz provides?

but we’re missing some metrics of Prometheus service monitors and pod monitors

These are the k8s objects watched by (Prometheus) operator. You could use the OpenTelemetry operator + Prometheus receiver with

kubernetes_sd_configs

.

Hey thanks for the information, I configured the above using target allocator and open telemetry operator. Though I see the metrics are being scrapped but I cannot see the metrics inside clickhouse db signoz.metrics.v2 table Is there some sort of format matching required?

No, there is no such format requirement. How did you confirm they are being scraped but not appearing in DB?

I figured it out

We don’t convert the promql to ClickHouse SQL. SigNoz works as a remote read system.

It was a config issue, the otl exporter should be from the open telemetry operator to the signoz otel collector, I was trying to make it work from the otelDeploy

You could use either jaeger or istio otel extension.

I configured the otel extension but cannot see any columns in clickhouse db I'm using your signoz custom collector, hitting the otlp endpoint from istio mesh config.

Do you see any data in Services/Traces tab of the SigNoz UI?

but cannot see any columns in clickhouse db

Where are you searching for this info? i.e. table and DB.

No I don't see any data in signoz ui as well

Can you share the logs of the Collector?

I tried to enable logging exporter but I think the collector implementation of signoz collector doesn't support that

I tried to enable logging exporter but I think the collector implementation of signoz collector doesn’t support that

We support the logging exporter.

Oh okay ill enable and check.

Does istiod show the successful message?

I'm not seeing any logs from traces after enabling log exporter We don't see any errors on istiod side too, could be like the istio is not sending traces to otel? Figuring out issue is with collector or the istio itself.

This is high likely an issue with the istio not able to send the traces to the collector since you don’t see any traces from the log exporter.

Thanks alot, we solved the issue, the sampling rate was low on istio and the collector port for otlp should be named as grpc-otlp, only then it works. I want to ask a generic question, what is the difference between the open telemetry contrib collector and the signoz collector? Is everything which is support by the contrib collector is support by the signoz one?

Mostly yes, except we remove some vendor speicific components and components which we believe make the collector bloated. In addition to that, there is some SigNoz specific work we do, such as our own processor components to keep backward compatibility and exporters of our own with ever evolving changes. Overall The SigNoz distro makes it easier for us to build on top of the opentelemetry.