Hi all, we have a self hosted Signoz (v0.53.0) col...
# support
p
Hi all, we have a self hosted Signoz (v0.53.0) collecting logs in our workspace. At a certain point of time, most of the dashboards suddenly stopped working. There was no change in the infrastructure on the server as well as the exporters' sides. Can you provide some ideas on how can this be debugged? Attaching the screenshots for reference. Sometimes I also see the following error on one of the queries in the network tab.
Copy code
Query:
{"query":"SELECT JSONExtractString(labels, 'k8s_namespace_name') AS k8s_namespace_name\nFROM signoz_metrics.distributed_time_series_v4_1day\nWHERE metric_name = 'k8s_pod_cpu_time' and JSONExtractString(labels, 'k8s_cluster_name') = {{.k8s_cluster_name}} AND JSONExtractString(labels, 'k8s_node_name') IN {{.k8s_node_name}}\nGROUP BY k8s_namespace_name","variables":{"k8s_cluster_name":"default","k8s_node_name":[],"k8s_namespace_name":"default","start_timestamp_ms":1750965089000,"end_timestamp_ms":1750966889000,"start_timestamp_nano":1750965089000000000,"end_timestamp_nano":1750966889000000000,"start_timestamp":1750965089,"end_timestamp":1750966889}}

Response:
{
    "status": "error",
    "errorType": "bad_data",
    "error": "code: 62, message: Syntax error: failed at position 288 ('BY') (line 4, col 7): BY k8s_namespace_name. Expected one of: token, Dot, OR, AND, IS NOT DISTINCT FROM, IS NULL, IS NOT NULL, BETWEEN, NOT BETWEEN, LIKE, ILIKE, NOT LIKE, NOT ILIKE, REGEXP, IN, NOT IN, GLOBAL IN, GLOBAL NOT IN, MOD, DIV, alias, AS, GROUP BY, WITH, HAVING, WINDOW, ORDER BY, LIMIT, OFFSET, FETCH, SETTINGS, UNION, EXCEPT, INTERSECT, INTO OUTFILE, FORMAT, end of query"
}
Any pointers appreciated! Thanks! CC: @Shaoxian Xu