https://signoz.io logo
Title
a

Arvin Corpuz

02/10/2023, 9:15 AM
hello. I want to export logs of our k8s clusters to signoz, so I followed this article - https://signoz.io/blog/kubernetes-monitoring/ and it works. Upon reading the documentation, i saw this - https://signoz.io/docs/tutorial/kubernetes-infra-metrics/ and it installs k8s-infra helm chart. executed it with the command
helm --namespace signoz-infra-metrics install signoz-infra-metrics signoz/k8s-infra  \
  --set otelCollectorEndpoint=<secure grpc endpoint> --set otelInsecure=false
then the k8s-infra-otel-deployment pod is giving this error
2023-02-10T09:04:25.988Z	info	extensions/extensions.go:48	Extension started.	{"kind": "extension", "name": "health_check"}
2023-02-10T09:04:25.988Z	info	pipelines/pipelines.go:74	Starting exporters...
2023-02-10T09:04:25.988Z	info	pipelines/pipelines.go:78	Exporter is starting...	{"kind": "exporter", "data_type": "metrics", "name": "otlp"}
2023-02-10T09:04:25.988Z	info	service/service.go:115	Starting shutdown...
2023-02-10T09:04:25.988Z	info	healthcheck/handler.go:129	Health Check state change	{"kind": "extension", "name": "health_check", "status": "unavailable"}
2023-02-10T09:04:25.988Z	info	pipelines/pipelines.go:118	Stopping receivers...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x3e0d3b9]

goroutine 1 [running]:
<http://github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver.(*kubernetesReceiver).Shutdown(0xc00158b358|github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver.(*kubernetesReceiver).Shutdown(0xc00158b358>?, {0x0?, 0x0?})
	<http://github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver@v0.66.0/receiver.go:101|github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver@v0.66.0/receiver.go:101> +0x19
<http://go.opentelemetry.io/collector/service/internal/pipelines.(*Pipelines).ShutdownAll(0xc000670cd0|go.opentelemetry.io/collector/service/internal/pipelines.(*Pipelines).ShutdownAll(0xc000670cd0>, {0x75de9c0, 0xc0000be028})
	<http://go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:121|go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:121> +0x499
<http://go.opentelemetry.io/collector/service.(*service).Shutdown(0xc000cb6000|go.opentelemetry.io/collector/service.(*service).Shutdown(0xc000cb6000>, {0x75de9c0, 0xc0000be028})
	<http://go.opentelemetry.io/collector@v0.66.0/service/service.go:121|go.opentelemetry.io/collector@v0.66.0/service/service.go:121> +0xd4
<http://go.opentelemetry.io/collector/service.(*Collector).shutdownServiceAndTelemetry(0xc00158ba88|go.opentelemetry.io/collector/service.(*Collector).shutdownServiceAndTelemetry(0xc00158ba88>, {0x75de9c0?, 0xc0000be028?})
	<http://go.opentelemetry.io/collector@v0.66.0/service/collector.go:264|go.opentelemetry.io/collector@v0.66.0/service/collector.go:264> +0x36
<http://go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents(0xc00158ba88|go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents(0xc00158ba88>, {0x75de9c0, 0xc0000be028})
	<http://go.opentelemetry.io/collector@v0.66.0/service/collector.go:166|go.opentelemetry.io/collector@v0.66.0/service/collector.go:166> +0x286
<http://go.opentelemetry.io/collector/service.(*Collector).Run(0xc00158ba88|go.opentelemetry.io/collector/service.(*Collector).Run(0xc00158ba88>, {0x75de9c0, 0xc0000be028})
	<http://go.opentelemetry.io/collector@v0.66.0/service/collector.go:190|go.opentelemetry.io/collector@v0.66.0/service/collector.go:190> +0x46
<http://go.opentelemetry.io/collector/service.NewCommand.func1(0xc0003f4c00|go.opentelemetry.io/collector/service.NewCommand.func1(0xc0003f4c00>, {0x69107b9?, 0x1?, 0x1?})
	<http://go.opentelemetry.io/collector@v0.66.0/service/command.go:53|go.opentelemetry.io/collector@v0.66.0/service/command.go:53> +0x479
<http://github.com/spf13/cobra.(*Command).execute(0xc0003f4c00|github.com/spf13/cobra.(*Command).execute(0xc0003f4c00>, {0xc0000b4030, 0x1, 0x1})
	<http://github.com/spf13/cobra@v1.6.1/command.go:916|github.com/spf13/cobra@v1.6.1/command.go:916> +0x862
<http://github.com/spf13/cobra.(*Command).ExecuteC(0xc0003f4c00)|github.com/spf13/cobra.(*Command).ExecuteC(0xc0003f4c00)>
	<http://github.com/spf13/cobra@v1.6.1/command.go:1044|github.com/spf13/cobra@v1.6.1/command.go:1044> +0x3bd
<http://github.com/spf13/cobra.(*Command).Execute(...)|github.com/spf13/cobra.(*Command).Execute(...)>
	<http://github.com/spf13/cobra@v1.6.1/command.go:968|github.com/spf13/cobra@v1.6.1/command.go:968>
main.runInteractive({{0xc0004e9f80, 0xc0004fd170, 0xc0004fc3c0, 0xc0004e9c20}, {{0x69388ae, 0xf}, {0x69b56ff, 0x1f}, {0x690a299, 0x6}}, ...})
	<http://github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:32|github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:32> +0x5d
main.run(...)
	<http://github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:11|github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:11>
main.main()
	<http://github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:25|github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:25> +0x1d8
anyone bumped into this?
n

nitya-signoz

02/10/2023, 9:20 AM
@Prashant Shahi any idea on this?
s

Srikanth Chekuri

02/10/2023, 11:13 AM
Is this reproducible? I would like to know waht conditions create the nil cancel function https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/release/v0.66.x/receiver/k8sclusterreceiver/receiver.go#L101
p

Prashant Shahi

02/10/2023, 8:19 PM
@Arvin Corpuz can you verify that the
<secure grpc endpoint>
in the
helm install
command is set correctly?
s

Srikanth Chekuri

02/11/2023, 1:51 AM
How does that relate to this nil pointer crash?
a

Arvin Corpuz

02/13/2023, 3:20 AM
yup, this is set correctly with <ip>:443 @Prashant Shahi
p

Prashant Shahi

02/13/2023, 5:38 AM
@Arvin Corpuz may I ask how you have set up
<ip>:443
for SigNoz OtelCollector? Is it configured to support gRPC?
a

Arvin Corpuz

02/13/2023, 6:54 AM
yes, it is using nginx ingress. it is working with this https://signoz.io/blog/kubernetes-monitoring/ (not using helm chart)