https://signoz.io logo
#support
Title
# support
z

Zakaria Jawwad

04/04/2023, 6:13 PM
Hi, We're trying this stack to have logs, traces, metrics in one ui. Currently we use kube prometheus stack, I see the defaults are good, shipping metrics and logs is very easy from otel but we're missing some metrics of Prometheus service monitors and pod monitors. Is there any way to push service monitors to signoz? Or is there any alternative which signoz provides?
s

Srikanth Chekuri

04/04/2023, 7:23 PM
but we’re missing some metrics of Prometheus service monitors and pod monitors
These are the k8s objects watched by (Prometheus) operator. You could use the OpenTelemetry operator + Prometheus receiver with
kubernetes_sd_configs
.
z

Zakaria Jawwad

04/05/2023, 9:10 AM
Hey thanks for the information, I configured the above using target allocator and open telemetry operator. Though I see the metrics are being scrapped but I cannot see the metrics inside clickhouse db signoz.metrics.v2 table Is there some sort of format matching required?
s

Srikanth Chekuri

04/06/2023, 7:41 AM
No, there is no such format requirement. How did you confirm they are being scraped but not appearing in DB?
z

Zakaria Jawwad

04/06/2023, 7:42 AM
I figured it out
Do you guys have any logs or mechanism where we can see what clickhouse query was generated by the promql we specified in signoz ui?
s

Srikanth Chekuri

04/06/2023, 7:44 AM
We don’t convert the promql to ClickHouse SQL. SigNoz works as a remote read system.
I figured it out
What was the issue?
z

Zakaria Jawwad

04/06/2023, 7:46 AM
It was a config issue, the otl exporter should be from the open telemetry operator to the signoz otel collector, I was trying to make it work from the otelDeploy
@Srikanth Chekuri do you know if we can send istio traces to clickhouse and explore the traces using signoz? What approach should we take like use zipkin or Jaeger or send directly using istio otel extension?
s

Srikanth Chekuri

04/10/2023, 7:36 AM
You could use either jaeger or istio otel extension.
z

Zakaria Jawwad

04/10/2023, 9:54 AM
I configured the otel extension but cannot see any columns in clickhouse db I'm using your signoz custom collector, hitting the otlp endpoint from istio mesh config.
s

Srikanth Chekuri

04/10/2023, 10:04 AM
Do you see any data in Services/Traces tab of the SigNoz UI?
but cannot see any columns in clickhouse db
Where are you searching for this info? i.e. table and DB.
z

Zakaria Jawwad

04/10/2023, 10:04 AM
No I don't see any data in signoz ui as well
signoz_traces database, I see no rows are being added but from istiod I see it's pushing data to otel
s

Srikanth Chekuri

04/10/2023, 10:05 AM
Can you share the logs of the Collector?
z

Zakaria Jawwad

04/10/2023, 10:05 AM
I tried to enable logging exporter but I think the collector implementation of signoz collector doesn't support that
I am directly hitting the signoz collector on 4317, I'm not using the otel contrib.
s

Srikanth Chekuri

04/10/2023, 10:06 AM
I tried to enable logging exporter but I think the collector implementation of signoz collector doesn’t support that
We support the logging exporter.
z

Zakaria Jawwad

04/10/2023, 10:07 AM
Oh okay ill enable and check.
s

Srikanth Chekuri

04/10/2023, 10:07 AM
Does istiod show the successful message?
z

Zakaria Jawwad

04/10/2023, 11:10 AM
I'm not seeing any logs from traces after enabling log exporter We don't see any errors on istiod side too, could be like the istio is not sending traces to otel? Figuring out issue is with collector or the istio itself.
s

Srikanth Chekuri

04/10/2023, 11:17 AM
This is high likely an issue with the istio not able to send the traces to the collector since you don’t see any traces from the log exporter.
z

Zakaria Jawwad

04/11/2023, 7:12 AM
Thanks alot, we solved the issue, the sampling rate was low on istio and the collector port for otlp should be named as grpc-otlp, only then it works. I want to ask a generic question, what is the difference between the open telemetry contrib collector and the signoz collector? Is everything which is support by the contrib collector is support by the signoz one?
s

Srikanth Chekuri

04/11/2023, 7:38 AM
Mostly yes, except we remove some vendor speicific components and components which we believe make the collector bloated. In addition to that, there is some SigNoz specific work we do, such as our own processor components to keep backward compatibility and exporters of our own with ever evolving changes. Overall The SigNoz distro makes it easier for us to build on top of the opentelemetry.
3 Views