https://signoz.io logo
Join Slack
Powered by
Hi there, first of all thanks for Signoz, it seems...
# support
b
Hi there, first of all thanks for Signoz, it seems to be a quite nice product! I'm currently trying it on one of my kube but I stumble upon something weird. I'm following the quick start guide. I got the signoz up and running but now I'm trying to deploy k8s-infra helm in the same NS. Following the doc, I set the value as follow, but I see no error but no metrics either in signoz UI. Am I missing something obvious ? I see logs and otel metric pipeline looks good to me. (signoz-otel-collector exists and has a endpoint) [Signoz 0.75.0 / k8s-infra helm 0.12.1]
Copy code
global:
  cloud: others
  clusterName: test-k01
  deploymentEnvironment: prod
otelCollectorEndpoint: signoz-otel-collector:4317
otelInsecure: true
presets:
  otlpExporter:
    enabled: true
  loggingExporter:
    enabled: false
n
i think the otelCollectorEndpoint is wrong here i use something like this http://signoz-otel-collector.observability.svc.cluster.local:4317
where signoz-otel-collector is the deploymentName observability is the namespace and the svc.cluster.local is the kubernets pattern
b
yep
I'll try that then
svc.cluster.local should work...
thing is I see logs
If endpoint would be wrong, I shouldn't have log at all, no ? Anyway, I tried with FQDN url, without luck
n
the otel collector is running ? can you see the logs ?
b
yes
n
my values.yaml have only this
Copy code
presets:
  logsCollection:
    enabled: false

otelCollectorEndpoint: <http://signoz-otel-collector.observability.svc.cluster.local:4317>
and works well
b
image.png
It makes me think collector is working for logs at least
you have some metrics in infra motinoring / Kube ?
which verison oh k8s-infra helm are you running ?
tried your light config no luck 😕
n
Copy code
helm show chart signoz/signoz                                                                                                                                                                                      
apiVersion: v2
appVersion: 0.75.0
dependencies:
- condition: clickhouse.enabled
  name: clickhouse
  repository: <https://charts.signoz.io>
  version: 24.1.15
- condition: signoz-otel-gateway.enabled
  name: signoz-otel-gateway
  repository: <https://charts.signoz.io>
  version: 0.0.1
description: SigNoz Observability Platform Helm Chart
home: <https://signoz.io/>
icon: <https://signoz.io/img/SigNozLogo-orange.svg>
keywords:
- SigNoz
- opentelemetry
- apm
- monitoring
- distributed tracing
- distributed logging
maintainers:
- email: hello@signoz.io
  name: SigNoz
  url: <https://signoz.io>
- email: prashant@signoz.io
  name: prashant-shahi
  url: <https://prashantshahi.dev>
name: signoz
sources:
- <https://github.com/signoz/charts>
- <https://github.com/signoz/signoz>
- <https://github.com/signoz/alertmanager>
- <https://github.com/signoz/signoz-otel-collector>
type: application
version: 0.73.0
Copy code
appVersion: 0.109.0
description: Helm chart for collecting metrics and logs in K8s
home: <https://signoz.io>
icon: <https://signoz.io/img/SigNozLogo-orange.svg>
keywords:
- SigNoz
- OpenTelemetry
- apm
- monitoring
- logs
maintainers:
- email: <mailto:hello@signoz.io|hello@signoz.io>
  name: SigNoz
  url: <https://signoz.io>
- email: <mailto:prashant@signoz.io|prashant@signoz.io>
  name: prashant-shahi
  url: <https://prashantshahi.dev>
name: k8s-infra
sources:
- <https://github.com/signoz/charts>
- <https://github.com/open-telemetry/opentelemetry-collector-contrib>
type: application
version: 0.12.1
i am confused about signoz version if it is 0.73 or 0.75
b
Helm chart version 0.73 signoz 0.75
Same here
Same version 0.12.1 for k8s-infra :/
Do you have something specific on signoz deploy ?
I have simply declared my fontend ingress in signoz deploy
Maybe I have missed something in the "base" signoz deploy.
n
that is my values.yaml to signoz
Copy code
global:
  storageClass: gp3
  cloud: aws

clickhouse:
  installCustomStorageClass: true

frontend:
  enabled: true
  replicaCount: 1
  ingress:
    enabled: true
    className: nginx
    annotations:
      <http://nginx.ingress.kubernetes.io/rewrite-target|nginx.ingress.kubernetes.io/rewrite-target>: /
    hosts:
      - host: <http://signoz.eks.hml.io|signoz.eks.hml.io>
        paths:
          - path: /
            pathType: Prefix
            port: 3301

otelCollector:
  enabled: true
  mode: deployment

  replicaCount: 2

  resources:
    requests:
      memory: "512Mi"
      cpu: "250m"
    limits:
      memory: "1Gi"
      cpu: "500m"

  ingress:
    enabled: true
    className: nginx
    annotations:
      <http://nginx.ingress.kubernetes.io/rewrite-target|nginx.ingress.kubernetes.io/rewrite-target>: /
    hosts:
      - host: <http://otel-grpc.eks.hml.io|otel-grpc.eks.hml.io>
        paths:
          - path: /
            pathType: Prefix
            port: 4317
      - host: <http://otel.eks.hml.io|otel.eks.hml.io>
        paths:
          - path: /
            pathType: Prefix
            port: 4318
b
I'll try with some of your additions
Nope... ;/
Will continue digging ;)
n
🙏
b
several attempts later, no clue ...
n
this is very strange, i had used this deploy configs in a develop and production cluster and it works fine
b
I noticed you setup on aws cloud type, i'm on prem... Might be related
n
hmmm maybe
b
I got it
(hi)
It was a clickhouse issue due to SIMDJSON and the virtualisation mechanism used
I tried to disable use of SIMDJSON without luck but I was able to change cpu type exposed by the hypervisor
n
nice
b
might deserve something somewhere ? not sure if it's a usual issue
and it pops with "recent" CH
n
Hey @babs Thanks for reporting the issue, just wanted to know if you were able to resolve the issue?
b
I did
By changing the CPU exposed by the hypervisor
n
and you're running signoz on prem?
b
Yep
On kube running on proxmox servers
n
okay cool, do you think it's something we should have in our docs ?
b
It might bit not specifically to change the cpu type as it's not doable everywhere but it might be interesting to explain how to inject disable smidjson in the embedded clickhouse
I wasn't able to "try" this or at least my attempts were not successful at this, but I'm no CH specialist https://github.com/ClickHouse/ClickHouse/issues/60661
I tried some options to inject allow_simdjson to 0 into the CH config
18 Views
Open in Slack
PreviousNext
Powered by