Riccardo Leonardi
06/06/2023, 1:01 PMtelemetrygen traces --traces 1 --otlp-endpoint localhost:4317 --otlp-insecure
then I can see on the Traces section the telemetrygen Service data (lets-go and okey-dokey).
So I've generated HostMetrics dashboards for the source VM and imported the resulting json to the Dashboards section on the dest VM.
The HostMetrics Dashboard - <source-VM-Name> appeared, but all the panels are empty.
I've also tried troubleshooting using:
troubleshoot checkEndpoint --endpoint=dest-IP:4317
and the answer was:
INFO workspace/main.go:28 STARTING!
INFO checkEndpoint/checkEndpoint.go:41 checking reachability of SigNoz endpoint
INFO workspace/main.go:46 Successfully sent sample data to signoz ...
There is no firewall between the two VM and tcpdump show traffic packets flowing from source to dest and back.
Where am I wrong?
Any help is very appreciated, tks.Prashant Shahi
06/07/2023, 4:55 AMRiccardo Leonardi
06/07/2023, 7:59 AMTrying 213.152.204.19...
Connected to <http://signoz.xtserver.it|signoz.xtserver.it>.
Escape character is '^]'.
@PuTTY^]
telnet>
Thanks again for your support.
RiccardoPrashant Shahi
06/07/2023, 8:12 AMRiccardo Leonardi
06/07/2023, 8:15 AMPrashant Shahi
06/07/2023, 8:15 AMRiccardo Leonardi
06/07/2023, 8:16 AMPrashant Shahi
06/07/2023, 8:17 AMRiccardo Leonardi
06/07/2023, 8:17 AMPrashant Shahi
06/07/2023, 8:17 AMRiccardo Leonardi
06/07/2023, 8:19 AMPrashant Shahi
06/07/2023, 8:20 AMRiccardo Leonardi
06/07/2023, 8:21 AMdocker ps
all containers seem to be working:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
095aa079cd0c signoz/frontend:0.19.0 "nginx -g 'daemon of…" 23 hours ago Up 21 hours 80/tcp, 0.0.0.0:3301->3301/tcp, :::3301->3301/tcp frontend
e0d39167b1b5 signoz/alertmanager:0.23.1 "/bin/alertmanager -…" 23 hours ago Up 21 hours 9093/tcp clickhouse-setup_alertmanager_1
f07b23de2200 signoz/signoz-otel-collector:0.76.1 "/signoz-collector -…" 23 hours ago Up 21 hours 0.0.0.0:4317-4318->4317-4318/tcp, :::4317-4318->4317-4318/tcp clickhouse-setup_otel-collector_1
e254a8502327 signoz/signoz-otel-collector:0.76.1 "/signoz-collector -…" 23 hours ago Up 21 hours 4317-4318/tcp clickhouse-setup_otel-collector-metrics_1
80ed2daaa665 signoz/query-service:0.19.0 "./query-service -co…" 23 hours ago Up 21 hours (healthy) 8080/tcp query-service
9ac491ef7bee clickhouse/clickhouse-server:22.8.8-alpine "/entrypoint.sh" 23 hours ago Up 21 hours (healthy) 0.0.0.0:8123->8123/tcp, :::8123->8123/tcp, 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp, 0.0.0.0:9181->9181/tcp, :::9181->9181/tcp, 9009/tcp clickhouse
ac8aed417d9f bitnami/zookeeper:3.7.1 "/opt/bitnami/script…" 23 hours ago Up 21 hours 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 0.0.0.0:2888->2888/tcp, :::2888->2888/tcp, 0.0.0.0:3888->3888/tcp, :::3888->3888/tcp, 8080/tcp zookeeper-1
79a1737c0e82 jaegertracing/example-hotrod:1.30 "/go/bin/hotrod-linu…" 23 hours ago Up 21 hours 8080-8083/tcp hotrod
07ab59b1e368 grubykarol/locust:1.2.3-python3.9-alpine3.12 "/docker-entrypoint.…" 23 hours ago Up 21 hours 5557-5558/tcp, 8089/tcp
Prashant Shahi
06/07/2023, 8:25 AMdocker logs clickhouse-setup_otel-collector_1
docker logs --tail 200 clickhouse
Riccardo Leonardi
06/07/2023, 8:30 AMPrashant Shahi
06/07/2023, 8:35 AM2023-06-06T125226.041Z error clickhousetracesexporter/writer.go:416 Could not append span to batch:
"name": "TestingSpan", "startTimeUnixNano": 11651379494838206464, "serviceName": "<nil-service-name>",
...
error": "clickhouse: dateTime overflow. timestamp must be between 1925-01-01 000000 and 2283-11-11 000000"}I see dummy trace data with
2339
year as start time.
Can you make sure you have correct time set on the instance or the environment?
cc @Ankit Nayan @Srikanth ChekuriRiccardo Leonardi
06/07/2023, 8:36 AMroot@signoz:~# date
Wed 07 Jun 2023 10:35:53 AM CEST
Prashant Shahi
06/07/2023, 8:37 AM0.66.0
to 0.76.x
?0.76.1
to be exactRiccardo Leonardi
06/07/2023, 8:40 AMPrashant Shahi
06/07/2023, 10:05 AMRiccardo Leonardi
06/07/2023, 10:16 AMJun 07 12:12:00 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: 2023-06-07T12:12:00.553+0200 info prometheusreceiver@v0.76.3/metrics_receiver.go:289 Starting scrape manager {"kind"
Jun 07 12:12:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: 2023-06-07T12:12:31.080+0200 error scraperhelper/scrapercontroller.go:212 Error scraping metrics {"kind": "receiver"
Jun 07 12:12:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport|go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport>
Jun 07 12:12:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:212|go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:212>
Jun 07 12:12:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1|go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1>
Jun 07 12:12:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:191|go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:191>
Jun 07 12:13:00 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: 2023-06-07T12:13:00.863+0200 error scraperhelper/scrapercontroller.go:212 Error scraping metrics {"kind": "receiver"
Jun 07 12:13:00 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport|go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport>
Jun 07 12:13:00 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:212|go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:212>
Jun 07 12:13:00 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1|go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1>
Jun 07 12:13:00 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:191|go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:191>
Jun 07 12:13:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: 2023-06-07T12:13:31.035+0200 error scraperhelper/scrapercontroller.go:212 Error scraping metrics {"kind": "receiver"
Jun 07 12:13:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport|go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport>
Jun 07 12:13:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:212|go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:212>
Jun 07 12:13:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1|go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1>
Jun 07 12:13:31 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:191|go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:191>
Jun 07 12:14:01 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: 2023-06-07T12:14:01.048+0200 error scraperhelper/scrapercontroller.go:212 Error scraping metrics {"kind": "receiver"
Jun 07 12:14:01 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport|go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport>
Jun 07 12:14:01 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:212|go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:212>
Jun 07 12:14:01 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1|go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1>
Jun 07 12:14:01 <http://isp1.yndetech.com|isp1.yndetech.com> otelcol-contrib[17452]: <http://go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:191|go.opentelemetry.io/collector/receiver@v0.76.1/scraperhelper/scrapercontroller.go:191>
Srikanth Chekuri
06/07/2023, 3:54 PMRiccardo Leonardi
06/07/2023, 4:13 PMSrikanth Chekuri
06/08/2023, 1:43 AMRiccardo Leonardi
06/08/2023, 7:09 AMhttps://files.slack.com/files-pri/T01HWUTP0LT-F05B56Z1Y22/image.png▾
Srikanth Chekuri
06/08/2023, 8:12 AMRiccardo Leonardi
06/08/2023, 9:05 AM