https://signoz.io logo
Join Slack
Powered by
Also I cannot find out the reason why I have a mem...
# support
g
Also I cannot find out the reason why I have a memory issue with the newer version of signoz.
Copy code
{
  "kind": "exporter",
  "data_type": "metrics",
  "name": "clickhousemetricswrite",
  "error": "code: 241, message: (total) memory limit exceeded: would use 14.90 GiB (attempt to allocate chunk of 4.39 MiB bytes), current RSS: 13.52 MiB, maximum: 13.82 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker",
  "interval": "23.836708773s"
}
I don't find anything problematic/relevant in the DB
v
g
Hi replace clickhousemetricswrite by 
signozclickhousemetrics
?
v
Yes, which version are you on?
It's not a simple replacement, you will need to perform some steps
g
Of signoz ?
0.90.1
v
g
I have no prod data yet so I can erase everything and start again. In this case do I have just to switch exporter and adapt dashboard accordingly to the new naming convention ?
v
Yes that's pretty much it. There's a migration job which will convert the alerts and dashboards containing metrics to the dot format.
g
Ok I will read that. It's in the link you put ?
v
Yes
g
Your tutorial focus on K8s install... I'm purely Linux and cannot use docker and co
Do I have to do some migration magic with the db tables ?
Also @Vibhu Pandey do you have some doc about how to configure this exporter ? I have errors about invalide entrypoint.
ok endpoint became dsn
and no tls: insecure: true available
I still have
Copy code
{"kind": "exporter", "data_type": "metrics", "name": "signozclickhousemetrics", "error": "code: 241, message: (total) memory limit exceeded: would use 15.01 GiB (attempt to allocate chunk of 4.18 MiB byte
s), current RSS: 7.18 MiB, maximum: 13.82 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker", "interval": "21.560104219s"}
{"kind": "exporter", "data_type": "metrics", "name": "signozclickhousemetrics", "error": "code: 241, message: (total) memory limit exceeded: would use 15.01 GiB (attempt to allocate chunk of 4.18 MiB byte
s), current RSS: 7.18 MiB, maximum: 13.82 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker", "interval": "21.560104219s"}
No idea @Vibhu Pandey Because I'm stuck
Ok I have restart the DB and it's seems to have no more issue. But I have no metrics in the db. Do I have to migrate some tables ? and how to ?
m
When upgrading SigNoz to latest stable, my metrics are no longer in dots with
Copy code
signoz:
  additionalEnvs:
    DOT_METRICS_ENABLED: "true"
any idea @Vibhu Pandey? its blocking our upgrade to latest
v
did you switch to the new exporter?
So 2 broad steps: 1. The environment variable (set to true) 2. Enable the new exporter. Apart from that, we have migration scripts to convert your existing 
alerts
and 
dashboards
to the new dot metrics.
This is the 10,000ft overview.
m
Yes, everything is already migrated to dot notation. Upgrading to latest just swaps the metrics back to underscores
v
An you have the env variable set as well?
Which chart version? the latest?
m
Latest chart version indeed. Right now we run 0.87.1, bumping to 0.88.1 breaks it
v
I'm pretty sure something is up with our charts here. Do you see the env variable being created in the signoz statefulset?
Copy code
kubectl get sts/signoz -o yaml
m
well im not going to upgrade again since all our alerts/dashboards will break then
👍 1
v
@Nagesh Bansal something to check here.
n
@Matti, in release 0.88.1, DOT_METRICS_ENABLED is set to true by default. Let me check why you might be facing this issue
m
I will be available in 2 hours
👍 1
30 Views
Open in Slack
PreviousNext
Powered by