Hi all, I'm new to SigNoz software, I just installed an instance of Docker Standalone on a VM on Debian 11 (dest). The purpose is to collect data from another Debian 10 VM (source) on the same subnet. On the source VM I've installed otelcol-contrib_0.66.0_linux_amd64.deb package, then latest docker/standalone/config.yaml where the otlp endpoint has been set to the dest VM. The setup seems working correctly, if I run on source the command:

telemetrygen traces --traces 1 --otlp-endpoint localhost:4317 --otlp-insecure

then I can see on the Traces section the telemetrygen Service data (lets-go and okey-dokey). So I've generated HostMetrics dashboards for the source VM and imported the resulting json to the Dashboards section on the dest VM. The HostMetrics Dashboard - <source-VM-Name> appeared, but all the panels are empty. I've also tried troubleshooting using:

troubleshoot checkEndpoint --endpoint=dest-IP:4317

and the answer was:

INFO    workspace/main.go:28    STARTING!
INFO    checkEndpoint/checkEndpoint.go:41       checking reachability of SigNoz endpoint
INFO    workspace/main.go:46    Successfully sent sample data to signoz ...

There is no firewall between the two VM and tcpdump show traffic packets flowing from source to dest and back. Where am I wrong? Any help is very appreciated, tks.

can you share screenshot of the empty dashboard? Can you open a widget and try plotting any of the metrics below? https://signoz.io/docs/tutorial/opentelemetry-binary-usage-in-virtual-machine/#list-of-metrics

Hi Prashant, thank you for your help. Here is the screenshot of empty dashboard: And the screenshot of Traces section. I also attached the log of otel-collector on source VM; it show a "connection refused" that seems replied by application, because there is no firewall between VMs and if from source I try to telnet the destination I obtain the following:

Trying 213.152.204.19...
Connected to <http://signoz.xtserver.it|signoz.xtserver.it>.
Escape character is '^]'.
@PuTTY^]
telnet>

Thanks again for your support. Riccardo

@Riccardo Leonardi which version of signoz are you using? also, did you select hostname from top selection bar?

Signoz v0.19.0

From logs, it seems like SigNoz-Otel-Collector is either not accessible or not healthy.

I already have attached the log from otel collector on source VM

The one you shared would be logs from otel-collector binary.

yes

But the signoz cluster contains of two collector components: signoz-otel-collector and signoz-otel-collector-metrics.

could you please tell me where to find those logs?

how did you install signoz? where is it deployed?

I've installed it as docker standalone, following tutorial on signoz.io

Logs of signoz otel-collector:

docker logs clickhouse-setup_otel-collector_1

clickhouse-setup_otel-collector_1.log

2023-06-06T125226.041Z error clickhousetracesexporter/writer.go:416 Could not append span to batch:

"name": "TestingSpan", "startTimeUnixNano": 11651379494838206464, "serviceName": "<nil-service-name>",

...

error": "clickhouse: dateTime overflow. timestamp must be between 1925-01-01 000000 and 2283-11-11 000000"}

I see dummy trace data with

2339

year as start time. Can you make sure you have correct time set on the instance or the environment? cc @Ankit Nayan @Srikanth Chekuri

the date should be correct:

root@signoz:~# date
Wed 07 Jun 2023 10:35:53 AM CEST

@Srikanth Chekuri could you please look into this?

yes, I could, but I have to tell you that I tried with 0.78.0 binary and it was not working (a lot of errors on startup) So I had to fall back to the version named in the tutorial, because I'm not expert in signoz

You can find all the release assets here: https://github.com/open-telemetry/opentelemetry-collector-releases/releases/tag/v0.76.1

I have installed 0.76.1 binary, but now the binary log show the following:

Jun 07 12:12:00 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: 2023-06-07T12:12:00.553+0200        info        prometheusreceiver@v0.76.3/metrics_receiver.go:289        Starting scrape manager        {"kind"
Jun 07 12:12:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: 2023-06-07T12:12:31.080+0200        error        scraperhelper/scrapercontroller.go:212        Error scraping metrics        {"kind": "receiver"
Jun 07 12:12:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport|go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport>
Jun 07 12:12:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]:         <http://go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:212|go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:212>
Jun 07 12:12:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1|go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1>
Jun 07 12:12:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]:         <http://go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:191|go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:191>
Jun 07 12:13:00 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: 2023-06-07T12:13:00.863+0200        error        scraperhelper/scrapercontroller.go:212        Error scraping metrics        {"kind": "receiver"
Jun 07 12:13:00 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport|go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport>
Jun 07 12:13:00 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]:         <http://go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:212|go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:212>
Jun 07 12:13:00 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1|go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1>
Jun 07 12:13:00 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]:         <http://go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:191|go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:191>
Jun 07 12:13:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: 2023-06-07T12:13:31.035+0200        error        scraperhelper/scrapercontroller.go:212        Error scraping metrics        {"kind": "receiver"
Jun 07 12:13:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport|go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport>
Jun 07 12:13:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]:         <http://go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:212|go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:212>
Jun 07 12:13:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1|go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1>
Jun 07 12:13:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]:         <http://go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:191|go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:191>
Jun 07 12:14:01 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: 2023-06-07T12:14:01.048+0200        error        scraperhelper/scrapercontroller.go:212        Error scraping metrics        {"kind": "receiver"
Jun 07 12:14:01 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport|go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport>
Jun 07 12:14:01 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]:         <http://go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:212|go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:212>
Jun 07 12:14:01 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1|go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1>
Jun 07 12:14:01 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]:         <http://go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:191|go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:191>

What’s the SDK you are using?

Currently no SDK, just otel-collector binary installed on a Debian 10 Next step will be to evaluate wordpress (php) instrumenting

Can you share what is the main problem here?

The main problem: the dashboard is empty:

Share the signoz version and dashboard json you are using.

signoz v0.19.0