This message was deleted.

Slackbot

12/19/2022, 9:24 AM

This message was deleted.

Prashant Shahi

12/19/2022, 10:33 AM

Hi @Quyet Nguyen 👋 cluster value should be set to

cluster

itself as those are hardcoded in migration files of OtelCollector. Also, makes sure the clickhouse endpoint and user credentials are accurate and have access to create DBs.

Quyet Nguyen

12/19/2022, 11:52 AM

Hi, changing the cluster to

cluster

helps me able to install chart 0.5.1. Still unable to install 0.6.0 as

otel-collector

and

otel-collector-metric

0.66 backoff with seg fault:

Copy code

2022-12-19T11:52:09.999Z info service/telemetry.go:111 Setting up own telemetry...
2022-12-19T11:52:09.999Z info service/telemetry.go:141 Serving Prometheus metrics {"address": "0.0.0.0:8888", "level": "Basic"}
2022-12-19T11:52:09.999Z info components/components.go:30 Stability level of component is undefined {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "stability": "Undefined"}
time="2022-12-19T11:52:09Z" level=info msg="Executing:\nCREATE DATABASE IF NOT EXISTS signoz_metrics ON CLUSTER cluster\n" component=clickhouse
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x3864f79]

goroutine 1 [running]:
<http://github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.NewPrwExporter(0xc0017d8300|github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.NewPrwExporter(0xc0017d8300>, {{0xc0004198f0, {0x529fd20, 0xc00108a480}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, {0x4931ce2, ...}, ...}})
 /src/exporter/clickhousemetricsexporter/exporter.go:90 +0x2f9
<http://github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.createMetricsExporter(|github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.createMetricsExporter(>{0x52d8d28, 0xc00007e018}, {{0xc0004198f0, {0x529fd20, 0xc00108a480}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}}, ...)
 /src/exporter/clickhousemetricsexporter/factory.go:50 +0x9f
<http://go.opentelemetry.io/collector/component.CreateMetricsExporterFunc.CreateMetricsExporter(...)|go.opentelemetry.io/collector/component.CreateMetricsExporterFunc.CreateMetricsExporter(...)>
 /go/pkg/mod/go.opentelemetry.io/collector/component@v0.66.0/exporter.go:149
<http://go.opentelemetry.io/collector/service/internal/pipelines.createExporter({0x52d8d28|go.opentelemetry.io/collector/service/internal/pipelines.createExporter({0x52d8d28>?, 0xc00007e018?}, {{0xc0004198f0, {0x529fd20, 0xc00108a480}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}}, ...)
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:374 +0x269
<http://go.opentelemetry.io/collector/service/internal/pipelines.buildExporter({0x52d8d28|go.opentelemetry.io/collector/service/internal/pipelines.buildExporter({0x52d8d28>, 0xc00007e018}, {0xc000418e70, {0x529fd20, 0xc00108a480}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}, ...)
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:360 +0x2ec
<http://go.opentelemetry.io/collector/service/internal/pipelines.Build({0x52d8d28|go.opentelemetry.io/collector/service/internal/pipelines.Build({0x52d8d28>, 0xc00007e018}, {{0xc000418e70, {0x529fd20, 0xc00108a480}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}, ...})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:243 +0x1a85
<http://go.opentelemetry.io/collector/service.(*service).initExtensionsAndPipeline(0xc00108a400|go.opentelemetry.io/collector/service.(*service).initExtensionsAndPipeline(0xc00108a400>, 0xc001a9f540)
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/service.go:162 +0x2bf
<http://go.opentelemetry.io/collector/service.newService(0xc0014cf540)|go.opentelemetry.io/collector/service.newService(0xc0014cf540)>
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/service.go:75 +0x525
<http://go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents|go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents>(0xc001a9fa88, {0x52d8d28, 0xc00007e018})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/collector.go:149 +0x1db
<http://go.opentelemetry.io/collector/service.(*Collector).Run(0xc001a9fa88|go.opentelemetry.io/collector/service.(*Collector).Run(0xc001a9fa88>, {0x52d8d28, 0xc00007e018})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/collector.go:190 +0x46
<http://go.opentelemetry.io/collector/service.NewCommand.func1(0xc000816c00|go.opentelemetry.io/collector/service.NewCommand.func1(0xc000816c00>, {0x48fa9ce?, 0x1?, 0x1?})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/command.go:53 +0x479
<http://github.com/spf13/cobra.(*Command).execute(0xc000816c00|github.com/spf13/cobra.(*Command).execute(0xc000816c00>, {0xc00006e050, 0x1, 0x1})
 /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916 +0x862
<http://github.com/spf13/cobra.(*Command).ExecuteC(0xc000816c00)|github.com/spf13/cobra.(*Command).ExecuteC(0xc000816c00)>
 /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bc
<http://github.com/spf13/cobra.(*Command).Execute(...)|github.com/spf13/cobra.(*Command).Execute(...)>
 /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968
main.runInteractive({{0xc0012cfe90, 0xc0012db050, 0xc0012da2d0, 0xc0012cfad0}, {{0x4934385, 0x15}, {0x4931ce2, 0x15}, {0x48f6f03, 0x6}}, ...})
 /src/cmd/signozcollector/main.go:36 +0x5e
main.run(...)
 /src/cmd/signozcollector/main_others.go:8
main.main()
 /src/cmd/signozcollector/main.go:29 +0x1d8

Prashant Shahi

12/20/2022, 5:49 AM

@Quyet Nguyen This is happening because otel-collectors were unable to resolve the passed ClickHouse service name triggering the error. You could either scale the otel-collectors down to 0 and back to 1. Or, you could try removing the pods, which will be re-created automatically. cc @Ankit Nayan @Srikanth Chekuri

Quyet Nguyen

12/20/2022, 9:25 AM

The value file is unchanged between 0.5.1 (which run fine) and 0.6.0 (segfault), the cluster and service is on the same namespace as signoz. Is there anything change that I must be aware of ?

Prashant Shahi

12/20/2022, 9:28 AM

SigNoz OtelCollector version was bumped up from 0.63 to 0.66, but it is a non-breaking change.

Prashant Shahi

12/20/2022, 9:29 AM

can you share output of the following?

Copy code

kubectl get pods,svc -n platform

Quyet Nguyen

12/20/2022, 9:31 AM

sure, here it is (note that it was on 0.5.1 and is running )

Copy code

NAME                                                    READY   STATUS    RESTARTS   AGE
pod/chi-signoz-cluster-0-0-0                            1/1     Running   0          21h
pod/signoz-query-service-0                              1/1     Running   0          21h
pod/signoz-frontend-85c76b49d7-6bpcn                    1/1     Running   0          21h
pod/signoz-alertmanager-0                               1/1     Running   0          21h
pod/signoz-k8s-infra-otel-agent-2nt48                   1/1     Running   0          21h
pod/signoz-otel-collector-65cc5cdddc-dw6fj              1/1     Running   0          21h
pod/signoz-otel-collector-metrics-d6b6bc8fb-552n6       1/1     Running   0          21h
pod/signoz-k8s-infra-otel-agent-vwftv                   1/1     Running   0          21h
pod/signoz-k8s-infra-otel-agent-6mhvd                   1/1     Running   0          21h
pod/signoz-k8s-infra-otel-deployment-76487d54fb-v6bvn   1/1     Running   0          21h

NAME                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                          AGE
service/clickhouse-signoz                  NodePort    10.152.183.30    <none>        8123:32123/TCP,9000:32200/TCP                    21h
service/chi-signoz-cluster-0-0             ClusterIP   None             <none>        8123/TCP,9000/TCP,9009/TCP                       21h
service/signoz-alertmanager-headless       ClusterIP   None             <none>        9093/TCP                                         21h
service/signoz-query-service               ClusterIP   10.152.183.80    <none>        8080/TCP,8085/TCP                                21h
service/signoz-k8s-infra-otel-agent        ClusterIP   10.152.183.148   <none>        13133/TCP,8888/TCP,4317/TCP,4318/TCP             21h
service/signoz-otel-collector-metrics      ClusterIP   10.152.183.74    <none>        13133/TCP                                        21h
service/signoz-alertmanager                ClusterIP   10.152.183.20    <none>        9093/TCP                                         21h
service/signoz-k8s-infra-otel-deployment   ClusterIP   10.152.183.52    <none>        13133/TCP,8888/TCP,4317/TCP,4318/TCP             21h
service/signoz-otel-collector              ClusterIP   10.152.183.158   <none>        14250/TCP,14268/TCP,8888/TCP,4317/TCP,4318/TCP   21h
service/signoz-frontend                    ClusterIP   10.152.183.213   <none>        3301/TCP                                         21h

Prashant Shahi

12/20/2022, 4:02 PM

This one looks good. Except

clickhouse-signoz

running as NodePort, which by default should be

ClusterIP

, but not an issue.

Prashant Shahi

12/20/2022, 4:02 PM

Could you share the the issue you are talking about in chart signoz 0.6.0?

Prashant Shahi

12/20/2022, 5:02 PM

override-values.yaml

would help

Prashant Shahi

12/20/2022, 5:04 PM

Could you share the the issue you are talking about in chart signoz 0.6.0?

Here, I meant the pod status and list of services after upgrade.

Quyet Nguyen

12/21/2022, 7:30 AM

Hi, I can consistently reproduce the error. Below is logs and k8s resources. I think it can relate to the zookeeper since the cluster is single node and have no zoo installed.

Copy code

helm install -n platform -f bare-metal/signoz/helm-values.yaml signoz6 signoz/signoz

Helm value file:

Copy code

clickhouse:
  enabled: false

externalClickhouse:
  host: clickhouse-signoz
  cluster: cluster
  user: admin
  password: password
  existingSecret: clickhouse-credentials
  existingSecretPasswordKey: admin-password
  database: signoz_metrics_060
  traceDatabase: signoz_traces_060

otel-collector

Copy code

2022-12-21T07:24:00.732Z info service/telemetry.go:111 Setting up own telemetry...
2022-12-21T07:24:00.732Z info service/telemetry.go:141 Serving Prometheus metrics {"address": "0.0.0.0:8888", "level": "Basic"}
2022-12-21T07:24:00.732Z info components/components.go:30 Stability level of component is undefined {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "stability": "Undefined"}
time="2022-12-21T07:24:00Z" level=info msg="Executing:\nCREATE DATABASE IF NOT EXISTS signoz_metrics_060 ON CLUSTER cluster\n" component=clickhouse
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x3864f79]

goroutine 1 [running]:
<http://github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.NewPrwExporter(0xc001574600|github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.NewPrwExporter(0xc001574600>, {{0xc000431650, {0x529fd20, 0xc000142980}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, {0x4931ce2, ...}, ...}})
 /src/exporter/clickhousemetricsexporter/exporter.go:90 +0x2f9
<http://github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.createMetricsExporter(|github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.createMetricsExporter(>{0x52d8d28, 0xc000084018}, {{0xc000431650, {0x529fd20, 0xc000142980}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}}, ...)
 /src/exporter/clickhousemetricsexporter/factory.go:50 +0x9f
<http://go.opentelemetry.io/collector/component.CreateMetricsExporterFunc.CreateMetricsExporter(...)|go.opentelemetry.io/collector/component.CreateMetricsExporterFunc.CreateMetricsExporter(...)>
 /go/pkg/mod/go.opentelemetry.io/collector/component@v0.66.0/exporter.go:149
<http://go.opentelemetry.io/collector/service/internal/pipelines.createExporter({0x52d8d28|go.opentelemetry.io/collector/service/internal/pipelines.createExporter({0x52d8d28>?, 0xc000084018?}, {{0xc000431650, {0x529fd20, 0xc000142980}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}}, ...)
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:374 +0x269
<http://go.opentelemetry.io/collector/service/internal/pipelines.buildExporter({0x52d8d28|go.opentelemetry.io/collector/service/internal/pipelines.buildExporter({0x52d8d28>, 0xc000084018}, {0xc000430850, {0x529fd20, 0xc000142980}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}, ...)
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:360 +0x2ec
<http://go.opentelemetry.io/collector/service/internal/pipelines.Build({0x52d8d28|go.opentelemetry.io/collector/service/internal/pipelines.Build({0x52d8d28>, 0xc000084018}, {{0xc000430850, {0x529fd20, 0xc000142980}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}, ...})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:243 +0x1a85
<http://go.opentelemetry.io/collector/service.(*service).initExtensionsAndPipeline(0xc000142800|go.opentelemetry.io/collector/service.(*service).initExtensionsAndPipeline(0xc000142800>, 0xc0017cf540)
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/service.go:162 +0x2bf
<http://go.opentelemetry.io/collector/service.newService(0xc001247540)|go.opentelemetry.io/collector/service.newService(0xc001247540)>
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/service.go:75 +0x525
<http://go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents|go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents>(0xc0017cfa88, {0x52d8d28, 0xc000084018})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/collector.go:149 +0x1db
<http://go.opentelemetry.io/collector/service.(*Collector).Run(0xc0017cfa88|go.opentelemetry.io/collector/service.(*Collector).Run(0xc0017cfa88>, {0x52d8d28, 0xc000084018})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/collector.go:190 +0x46
<http://go.opentelemetry.io/collector/service.NewCommand.func1(0xc000374600|go.opentelemetry.io/collector/service.NewCommand.func1(0xc000374600>, {0x48fa9ce?, 0x1?, 0x1?})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/command.go:53 +0x479
<http://github.com/spf13/cobra.(*Command).execute(0xc000374600|github.com/spf13/cobra.(*Command).execute(0xc000374600>, {0xc00006e050, 0x1, 0x1})
 /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916 +0x862
<http://github.com/spf13/cobra.(*Command).ExecuteC(0xc000374600)|github.com/spf13/cobra.(*Command).ExecuteC(0xc000374600)>
 /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bc
<http://github.com/spf13/cobra.(*Command).Execute(...)|github.com/spf13/cobra.(*Command).Execute(...)>
 /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968
main.runInteractive({{0xc000f0cd50, 0xc000f0def0, 0xc000f0d170, 0xc000f0c990}, {{0x4934385, 0x15}, {0x4931ce2, 0x15}, {0x48f6f03, 0x6}}, ...})
 /src/cmd/signozcollector/main.go:36 +0x5e
main.run(...)
 /src/cmd/signozcollector/main_others.go:8
main.main()
 /src/cmd/signozcollector/main.go:29 +0x1d8

otel-collector-metric

Copy code

2022-12-21T07:24:02.515Z info service/telemetry.go:111 Setting up own telemetry...
2022-12-21T07:24:02.515Z info service/telemetry.go:141 Serving Prometheus metrics {"address": "0.0.0.0:8888", "level": "Basic"}
2022-12-21T07:24:02.515Z info components/components.go:30 Stability level of component is undefined {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "stability": "Undefined"}
time="2022-12-21T07:24:02Z" level=info msg="Executing:\nCREATE DATABASE IF NOT EXISTS signoz_metrics_060 ON CLUSTER cluster\n" component=clickhouse
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x3864f79]

goroutine 1 [running]:
<http://github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.NewPrwExporter(0xc0014a6300|github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.NewPrwExporter(0xc0014a6300>, {{0xc000219110, {0x529fd20, 0xc0003ede80}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, {0x4931ce2, ...}, ...}})
 /src/exporter/clickhousemetricsexporter/exporter.go:90 +0x2f9
<http://github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.createMetricsExporter(|github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.createMetricsExporter(>{0x52d8d28, 0xc000084018}, {{0xc000219110, {0x529fd20, 0xc0003ede80}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}}, ...)
 /src/exporter/clickhousemetricsexporter/factory.go:50 +0x9f
<http://go.opentelemetry.io/collector/component.CreateMetricsExporterFunc.CreateMetricsExporter(...)|go.opentelemetry.io/collector/component.CreateMetricsExporterFunc.CreateMetricsExporter(...)>
 /go/pkg/mod/go.opentelemetry.io/collector/component@v0.66.0/exporter.go:149
<http://go.opentelemetry.io/collector/service/internal/pipelines.createExporter({0x52d8d28|go.opentelemetry.io/collector/service/internal/pipelines.createExporter({0x52d8d28>?, 0xc000084018?}, {{0xc000219110, {0x529fd20, 0xc0003ede80}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}}, ...)
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:374 +0x269
<http://go.opentelemetry.io/collector/service/internal/pipelines.buildExporter({0x52d8d28|go.opentelemetry.io/collector/service/internal/pipelines.buildExporter({0x52d8d28>, 0xc000084018}, {0xc000218460, {0x529fd20, 0xc0003ede80}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}, ...)
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:360 +0x2ec
<http://go.opentelemetry.io/collector/service/internal/pipelines.Build({0x52d8d28|go.opentelemetry.io/collector/service/internal/pipelines.Build({0x52d8d28>, 0xc000084018}, {{0xc000218460, {0x529fd20, 0xc0003ede80}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}, ...})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:243 +0x1a85
<http://go.opentelemetry.io/collector/service.(*service).initExtensionsAndPipeline(0xc0003ede00|go.opentelemetry.io/collector/service.(*service).initExtensionsAndPipeline(0xc0003ede00>, 0xc0019a7540)
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/service.go:162 +0x2bf
<http://go.opentelemetry.io/collector/service.newService(0xc0011c7540)|go.opentelemetry.io/collector/service.newService(0xc0011c7540)>
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/service.go:75 +0x525
<http://go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents|go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents>(0xc0019a7a88, {0x52d8d28, 0xc000084018})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/collector.go:149 +0x1db
<http://go.opentelemetry.io/collector/service.(*Collector).Run(0xc0019a7a88|go.opentelemetry.io/collector/service.(*Collector).Run(0xc0019a7a88>, {0x52d8d28, 0xc000084018})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/collector.go:190 +0x46
<http://go.opentelemetry.io/collector/service.NewCommand.func1(0xc000148600|go.opentelemetry.io/collector/service.NewCommand.func1(0xc000148600>, {0x48fa9ce?, 0x1?, 0x1?})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/command.go:53 +0x479
<http://github.com/spf13/cobra.(*Command).execute(0xc000148600|github.com/spf13/cobra.(*Command).execute(0xc000148600>, {0xc00006e050, 0x1, 0x1})
 /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916 +0x862
<http://github.com/spf13/cobra.(*Command).ExecuteC(0xc000148600)|github.com/spf13/cobra.(*Command).ExecuteC(0xc000148600)>
 /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bc
<http://github.com/spf13/cobra.(*Command).Execute(...)|github.com/spf13/cobra.(*Command).Execute(...)>
 /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968
main.runInteractive({{0xc0005fcd80, 0xc0005fdf20, 0xc0005fd1a0, 0xc0005fc9c0}, {{0x4934385, 0x15}, {0x4931ce2, 0x15}, {0x48f6f03, 0x6}}, ...})
 /src/cmd/signozcollector/main.go:36 +0x5e
main.run(...)
 /src/cmd/signozcollector/main_others.go:8
main.main()
 /src/cmd/signozcollector/main.go:29 +0x1d8

This is the Clickhouse cluster:

Copy code

apiVersion: "<http://clickhouse.altinity.com/v1|clickhouse.altinity.com/v1>"
kind: "ClickHouseInstallation"
metadata:
  name: "signoz"
  namespace: "platform"
spec:
  defaults:
    templates:
      podTemplate: clickhouse-per-node
      serviceTemplate: svc-node-port
  configuration:
    clusters:
      - name: cluster
        layout:
          shardsCount: 1
          replicasCount: 1
    users:
      default/access_management: 1
      default/networks/ip: "10.0.0.0/8"
      default/k8s_secret_password: platform/clickhouse-credentials/admin-password
      admin/k8s_secret_password: platform/clickhouse-credentials/admin-password
      admin/profile: default
      admin/quota: default
      admin/networks/ip: "0.0.0.0/0"
      readonly/k8s_secret_password: platform/clickhouse-credentials/readonly-password
      readonly/profile: readonly
      readonly/quota: default
      readonly/networks/ip: "0.0.0.0/0"
    profiles:
      readonly/readonly: 1
  templates:
    serviceTemplates:
      - name: svc-node-port
        generateName: clickhouse-{chi}
        spec:
          ports:
            - name: http
              port: 8123
              nodePort: 32123
            - name: tcp
              port: 9000
              nodePort: 32200
          type: NodePort

    podTemplates:
      - name: clickhouse-per-node
        spec:
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: "<http://clickhouse.altinity.com/app|clickhouse.altinity.com/app>"
                        operator: In
                        values:
                          - "chop"
                  topologyKey: "<http://kubernetes.io/hostname|kubernetes.io/hostname>"
          volumes:
            - name: local-path
              hostPath:
                path: /data/signoz
                type: DirectoryOrCreate

Prashant Shahi

12/21/2022, 12:51 PM

The following values should not be altered:

Copy code

database: signoz_metrics
  traceDatabase: signoz_traces

Prashant Shahi

12/21/2022, 12:52 PM

we have the database names hardcoded in migration files, due to the limitation of go-migrate.

Quyet Nguyen

12/22/2022, 2:54 AM

I changed that because we have running cluster using signoz_metrics and signoz_traces on 0.5.1.

Quyet Nguyen

12/22/2022, 2:54 AM

Can we upgrading to 0.6.0 keeping the current data ?

Prashant Shahi

12/22/2022, 4:16 PM

I changed that because we have running cluster using signoz_metrics and signoz_traces on 0.5.1.

@Quyet Nguyen That won't be possible. You would have to either use the SigNoz clickhouse or use other external clickhouse.

Prashant Shahi

12/22/2022, 4:17 PM

Can we upgrading to 0.6.0 keeping the current data ?

yes, you can.

Prashant Shahi

12/22/2022, 4:17 PM

Provided that you do not alter cluster name

cluster

, or the database names.

Quyet Nguyen

12/23/2022, 3:15 AM

Upgrade to 0.6.0 with our current helm value cause otel and otel-metrics backoff on segfault error The install works normally when I upgrade it back to 0.5.1 again. Helm value file:

Copy code

clickhouse:
  enabled: false

externalClickhouse:
  host: clickhouse-signoz
  cluster: cluster
  user: admin
  password: password
  existingSecret: clickhouse-credentials
  existingSecretPasswordKey: admin-password

clickhouse installation resource:

Copy code

apiVersion: "<http://clickhouse.altinity.com/v1|clickhouse.altinity.com/v1>"
kind: "ClickHouseInstallation"
metadata:
  name: "signoz"
  namespace: "platform"
spec:
  defaults:
    templates:
      podTemplate: clickhouse-per-node
      serviceTemplate: svc-node-port
  configuration:
    clusters:
      - name: cluster
        layout:
          shardsCount: 1
          replicasCount: 1
    users:
      default/access_management: 1
      default/networks/ip: "10.0.0.0/8"
      default/k8s_secret_password: platform/clickhouse-credentials/admin-password
      admin/k8s_secret_password: platform/clickhouse-credentials/admin-password
      admin/profile: default
      admin/quota: default
      admin/networks/ip: "0.0.0.0/0"
      readonly/k8s_secret_password: platform/clickhouse-credentials/readonly-password
      readonly/profile: readonly
      readonly/quota: default
      readonly/networks/ip: "0.0.0.0/0"
    profiles:
      readonly/readonly: 1
  templates:
    serviceTemplates:
      - name: svc-node-port
        generateName: clickhouse-{chi}
        spec:
          ports:
            - name: http
              port: 8123
              nodePort: 32123
            - name: tcp
              port: 9000
              nodePort: 32200
          type: NodePort

    podTemplates:
      - name: clickhouse-per-node
        spec:
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: "<http://clickhouse.altinity.com/app|clickhouse.altinity.com/app>"
                        operator: In
                        values:
                          - "chop"
                  topologyKey: "<http://kubernetes.io/hostname|kubernetes.io/hostname>"
          volumes:
            - name: local-path
              hostPath:
                path: /data/signoz
                type: DirectoryOrCreate

Otel logs:

Copy code

2022-12-23T03:13:48.759Z info service/telemetry.go:111 Setting up own telemetry...
2022-12-23T03:13:48.759Z info service/telemetry.go:141 Serving Prometheus metrics {"address": "0.0.0.0:8888", "level": "Basic"}
2022-12-23T03:13:48.759Z info components/components.go:30 Stability level of component is undefined {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "stability": "Undefined"}
time="2022-12-23T03:13:48Z" level=info msg="Executing:\nCREATE DATABASE IF NOT EXISTS signoz_metrics ON CLUSTER cluster\n" component=clickhouse
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x3864f79]

goroutine 1 [running]:
<http://github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.NewPrwExporter(0xc00158cf00|github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.NewPrwExporter(0xc00158cf00>, {{0xc0001b13b0, {0x529fd20, 0xc000c9e000}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, {0x4931ce2, ...}, ...}})
 /src/exporter/clickhousemetricsexporter/exporter.go:90 +0x2f9
<http://github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.createMetricsExporter(|github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.createMetricsExporter(>{0x52d8d28, 0xc00007e018}, {{0xc0001b13b0, {0x529fd20, 0xc000c9e000}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}}, ...)
 /src/exporter/clickhousemetricsexporter/factory.go:50 +0x9f
<http://go.opentelemetry.io/collector/component.CreateMetricsExporterFunc.CreateMetricsExporter(...)|go.opentelemetry.io/collector/component.CreateMetricsExporterFunc.CreateMetricsExporter(...)>
 /go/pkg/mod/go.opentelemetry.io/collector/component@v0.66.0/exporter.go:149
<http://go.opentelemetry.io/collector/service/internal/pipelines.createExporter({0x52d8d28|go.opentelemetry.io/collector/service/internal/pipelines.createExporter({0x52d8d28>?, 0xc00007e018?}, {{0xc0001b13b0, {0x529fd20, 0xc000c9e000}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}}, ...)
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:374 +0x269
<http://go.opentelemetry.io/collector/service/internal/pipelines.buildExporter({0x52d8d28|go.opentelemetry.io/collector/service/internal/pipelines.buildExporter({0x52d8d28>, 0xc00007e018}, {0xc0001b0930, {0x529fd20, 0xc000c9e000}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}, ...)
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:360 +0x2ec
<http://go.opentelemetry.io/collector/service/internal/pipelines.Build({0x52d8d28|go.opentelemetry.io/collector/service/internal/pipelines.Build({0x52d8d28>, 0xc00007e018}, {{0xc0001b0930, {0x529fd20, 0xc000c9e000}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}, ...})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:243 +0x1a85
<http://go.opentelemetry.io/collector/service.(*service).initExtensionsAndPipeline(0xc0003b1e80|go.opentelemetry.io/collector/service.(*service).initExtensionsAndPipeline(0xc0003b1e80>, 0xc00182f540)
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/service.go:162 +0x2bf
<http://go.opentelemetry.io/collector/service.newService(0xc00147b540)|go.opentelemetry.io/collector/service.newService(0xc00147b540)>
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/service.go:75 +0x525
<http://go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents|go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents>(0xc00182fa88, {0x52d8d28, 0xc00007e018})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/collector.go:149 +0x1db
<http://go.opentelemetry.io/collector/service.(*Collector).Run(0xc00182fa88|go.opentelemetry.io/collector/service.(*Collector).Run(0xc00182fa88>, {0x52d8d28, 0xc00007e018})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/collector.go:190 +0x46
<http://go.opentelemetry.io/collector/service.NewCommand.func1(0xc0001b7200|go.opentelemetry.io/collector/service.NewCommand.func1(0xc0001b7200>, {0x48fa9ce?, 0x1?, 0x1?})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/command.go:53 +0x479
<http://github.com/spf13/cobra.(*Command).execute(0xc0001b7200|github.com/spf13/cobra.(*Command).execute(0xc0001b7200>, {0xc00006e050, 0x1, 0x1})
 /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916 +0x862
<http://github.com/spf13/cobra.(*Command).ExecuteC(0xc0001b7200)|github.com/spf13/cobra.(*Command).ExecuteC(0xc0001b7200)>
 /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bc
<http://github.com/spf13/cobra.(*Command).Execute(...)|github.com/spf13/cobra.(*Command).Execute(...)>
 /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968
main.runInteractive({{0xc000da0060, 0xc000da1200, 0xc000da0480, 0xc000cc9c80}, {{0x4934385, 0x15}, {0x4931ce2, 0x15}, {0x48f6f03, 0x6}}, ...})
 /src/cmd/signozcollector/main.go:36 +0x5e
main.run(...)
 /src/cmd/signozcollector/main_others.go:8
main.main()
 /src/cmd/signozcollector/main.go:29 +0x1d8

Otel metric logs:

Copy code

2022-12-23T03:14:28.486Z info service/telemetry.go:111 Setting up own telemetry...
2022-12-23T03:14:28.486Z info service/telemetry.go:141 Serving Prometheus metrics {"address": "0.0.0.0:8888", "level": "Basic"}
2022-12-23T03:14:28.486Z info components/components.go:30 Stability level of component is undefined {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "stability": "Undefined"}
time="2022-12-23T03:14:28Z" level=info msg="Executing:\nCREATE DATABASE IF NOT EXISTS signoz_metrics ON CLUSTER cluster\n" component=clickhouse
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x3864f79]

goroutine 1 [running]:
<http://github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.NewPrwExporter(0xc00158e300|github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.NewPrwExporter(0xc00158e300>, {{0xc000421030, {0x529fd20, 0xc0001ac280}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, {0x4931ce2, ...}, ...}})
 /src/exporter/clickhousemetricsexporter/exporter.go:90 +0x2f9
<http://github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.createMetricsExporter(|github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.createMetricsExporter(>{0x52d8d28, 0xc00007e018}, {{0xc000421030, {0x529fd20, 0xc0001ac280}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}}, ...)
 /src/exporter/clickhousemetricsexporter/factory.go:50 +0x9f
<http://go.opentelemetry.io/collector/component.CreateMetricsExporterFunc.CreateMetricsExporter(...)|go.opentelemetry.io/collector/component.CreateMetricsExporterFunc.CreateMetricsExporter(...)>
 /go/pkg/mod/go.opentelemetry.io/collector/component@v0.66.0/exporter.go:149
<http://go.opentelemetry.io/collector/service/internal/pipelines.createExporter({0x52d8d28|go.opentelemetry.io/collector/service/internal/pipelines.createExporter({0x52d8d28>?, 0xc00007e018?}, {{0xc000421030, {0x529fd20, 0xc0001ac280}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}}, ...)
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:374 +0x269
<http://go.opentelemetry.io/collector/service/internal/pipelines.buildExporter({0x52d8d28|go.opentelemetry.io/collector/service/internal/pipelines.buildExporter({0x52d8d28>, 0xc00007e018}, {0xc000420380, {0x529fd20, 0xc0001ac280}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}, ...)
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:360 +0x2ec
<http://go.opentelemetry.io/collector/service/internal/pipelines.Build({0x52d8d28|go.opentelemetry.io/collector/service/internal/pipelines.Build({0x52d8d28>, 0xc00007e018}, {{0xc000420380, {0x529fd20, 0xc0001ac280}, {0x52a7c60, 0x7d3d1a0}, 0x0}, {{0x4934385, 0x15}, ...}, ...})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:243 +0x1a85
<http://go.opentelemetry.io/collector/service.(*service).initExtensionsAndPipeline(0xc0001ac200|go.opentelemetry.io/collector/service.(*service).initExtensionsAndPipeline(0xc0001ac200>, 0xc001ba9540)
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/service.go:162 +0x2bf
<http://go.opentelemetry.io/collector/service.newService(0xc001383540)|go.opentelemetry.io/collector/service.newService(0xc001383540)>
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/service.go:75 +0x525
<http://go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents|go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents>(0xc001ba9a88, {0x52d8d28, 0xc00007e018})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/collector.go:149 +0x1db
<http://go.opentelemetry.io/collector/service.(*Collector).Run(0xc001ba9a88|go.opentelemetry.io/collector/service.(*Collector).Run(0xc001ba9a88>, {0x52d8d28, 0xc00007e018})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/collector.go:190 +0x46
<http://go.opentelemetry.io/collector/service.NewCommand.func1(0xc0003aa600|go.opentelemetry.io/collector/service.NewCommand.func1(0xc0003aa600>, {0x48fa9ce?, 0x1?, 0x1?})
 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/command.go:53 +0x479
<http://github.com/spf13/cobra.(*Command).execute(0xc0003aa600|github.com/spf13/cobra.(*Command).execute(0xc0003aa600>, {0xc00006e050, 0x1, 0x1})
 /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916 +0x862
<http://github.com/spf13/cobra.(*Command).ExecuteC(0xc0003aa600)|github.com/spf13/cobra.(*Command).ExecuteC(0xc0003aa600)>
 /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bc
<http://github.com/spf13/cobra.(*Command).Execute(...)|github.com/spf13/cobra.(*Command).Execute(...)>
 /go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968
main.runInteractive({{0xc001512000, 0xc0015131a0, 0xc001512420, 0xc001509c20}, {{0x4934385, 0x15}, {0x4931ce2, 0x15}, {0x48f6f03, 0x6}}, ...})
 /src/cmd/signozcollector/main.go:36 +0x5e
main.run(...)
 /src/cmd/signozcollector/main_others.go:8
main.main()
 /src/cmd/signozcollector/main.go:29 +0x1d8

Prashant Shahi

12/27/2022, 5:51 PM

@Quyet Nguyen Perhaps we can get on call to get this resolved.

Quyet Nguyen

12/29/2022, 4:45 AM

I'd glad to. Can you pick a time slot

Prashant Shahi

12/29/2022, 8:32 AM

@que

Prashant Shahi

12/29/2022, 8:32 AM

@Quyet Nguyen can you share you email over DM?

Prashant Shahi

12/30/2022, 11:03 AM

https://signoz-community.slack.com/archives/C01HWQ1R0BC/p1671607824010739?thread_ts=1671441854.466139&cid=C01HWQ1R0BC

Prashant Shahi

12/30/2022, 11:13 AM

https://github.com/Altinity/clickhouse-operator/blob/master/docs/chi-examples/99-clickhouseinstallation-max.yaml#L482

Quyet Nguyen

12/30/2022, 11:19 AM

2022-12-30T111731.034Z info service/telemetry.go:111 Setting up own telemetry... Dec 30, 2022

Prashant Shahi

12/30/2022, 11:22 AM

Copy code

kubectl -n platform run test-app --image=djbingham/curl \
  --restart='OnFailure' -i --tty --rm --command -- curl \           
  <http://clickhouse-svc:8123>

Quyet Nguyen

12/30/2022, 11:28 AM

2022-12-30T112738.460Z info service/telemetry.go:111 Setting up own telemetry... 2022-12-30T112738.460Z info service/telemetry.go:141 Serving Prometheus metrics {"address": "0.0.0.0:8888", "level": "Basic"} 2022-12-30T112738.460Z info components/components.go:30 Stability level of component is undefined {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "stability": "Undefined"} Error: cannot build pipelines: failed to create "clickhousetraces" exporter, in pipeline "traces": error connecting to primary db: code: 139, message: There is no Zookeeper configuration in server config 2022/12/30 112738 application run finished with error: cannot build pipelines: failed to create "clickhousetraces" exporter, in pipeline "traces": error connecting to primary db: code: 139, message: There is no Zookeeper configuration in server config

Prashant Shahi

12/30/2022, 11:30 AM

https://github.com/bitnami/charts/tree/main/bitnami/zookeeper/#tldr

269 Views

Open in Slack

Previous Next

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.