https://signoz.io
Join Slack
Just Looking from some support with a local signoz allocation. When trying to perform a query I alw...
# support
j
Just Looking from some support with a local signoz allocation. When trying to perform a query I always get the following error: SELECT DISTINCT JSONExtractString(labels, 'host_name') FROM signoz_metrics.time_series_v2 Received exception from server (version 24.1.2): Code: 173. DB:Exception Received from 127.0.0.1:9000. DB:Exception Couldn't allocate 333 bytes when parsing JSON: while executing 'FUNCTION JSONExtractString(labels :: 0, 'host_name' :: 1) -> JSONExtractString(labels, 'host_name') String : 2'. (CANNOT_ALLOCATE_MEMORY) Any idea how to solve it? Thanks in advance
s
Use this query instead
Copy code
SELECT JSONExtractString(labels, 'host_name') as host_name
FROM signoz_metrics.distributed_time_series_v4_1day
WHERE metric_name = 'system_load_average_1m' AND JSONExtractString(labels, 'deployment_environment') = {{.deployment_environment}} AND unix_milli >= toUnixTimestamp(now() - INTERVAL 1 DAY) * 1000
GROUP BY host_name;
j
Hello. Thanks for your answer. Unfortunately, the issue continues as follows: When running: ''' SELECT * FROM signoz_metrics.time_series_v4_1day WHERE metric_name = 'system_cpu_load_average_1m' ''' ┌─env─────┬─temporality─┬─metric_name────────────────┬─description─────────────────────┬─unit─┬─type──┬─is_monotonic─┬──────────fingerprint─┬────unix_milli─┬─labels────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ default │ Unspecified │ system_cpu_load_average_1m │ Average CPU Load over 1 minute. │ 1 │ Gauge │ false │ 1006716870333484151 │ 1714694400000 │ {"__name__":"system_cpu_load_average_1m","__temporality__":"Unspecified","host_name":"signoz-host","os_type":"linux"} │ │ default │ Unspecified │ system_cpu_load_average_1m │ Average CPU Load over 1 minute. │ 1 │ Gauge │ false │ 9087904535405703063 │ 1714694400000 │ {"__name__":"system_cpu_load_average_1m","__temporality__":"Unspecified"} │ │ default │ Unspecified │ system_cpu_load_average_1m │ Average CPU Load over 1 minute. │ 1 │ Gauge │ false │ 12871807438238536041 │ 1714694400000 │ {"__name__":"system_cpu_load_average_1m","__temporality__":"Unspecified","host_name":"emulate-153","os_type":"linux"} │ └─────────┴─────────────┴────────────────────────────┴─────────────────────────────────┴──────┴───────┴──────────────┴──────────────────────┴───────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ Only 3 results are obtained, what makes me think that the signoz collector is not collecting the data properly docker ps returns the following: ------------------------------------------------------- CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES bb180e54d813 signoz/frontend:0.43.0 "nginx -g 'daemon of…" 6 minutes ago Up 5 minutes 80/tcp, 0.0.0.0:3301->3301/tcp, :::3301->3301/tcp signoz-frontend 9f4744e6bfc3 gliderlabs/logspout:v3.2.14 "/bin/logspout syslo…" 6 minutes ago Up 5 minutes 80/tcp signoz-logspout 4d9eac6b4292 signoz/alertmanager:0.23.5 "/bin/alertmanager -…" 6 minutes ago Up 5 minutes 9093/tcp signoz-alertmanager 4a58b2c7d884 signoz/signoz-otel-collector:0.88.20 "/signoz-collector -…" 6 minutes ago Up 5 minutes 0.0.0.0:4317-4318->4317-4318/tcp, :::4317-4318->4317-4318/tcp, 0.0.0.0:8888-8889->8888-8889/tcp, :::8888-8889->8888-8889/tcp signoz-otel-collector 1e3a01afaddb signoz/query-service:0.43.0 "./query-service -co…" 6 minutes ago Up 5 minutes (healthy) 8080/tcp signoz-query-service bce93c5d7fae clickhouse/clickhouse-server:24.1.2-alpine "/entrypoint.sh" 6 minutes ago Up 6 minutes (healthy) 0.0.0.0:8123->8123/tcp, :::8123->8123/tcp, 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp, 0.0.0.0:9181->9181/tcp, :::9181->9181/tcp, 9009/tcp signoz-clickhouse 92178664ee59 bitnami/zookeeper:3.7.1 "/opt/bitnami/script…" 6 minutes ago Up 6 minutes 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 0.0.0.0:2888->2888/tcp, :::2888->2888/tcp, 0.0.0.0:3888->3888/tcp, :::3888->3888/tcp, 8080/tcp signoz-zookeeper-1 ------------------------------------------------------- Otel/config.yaml looks like this: hostmetrics: collection_interval: 10s scrapers: cpu:{} In the docker compose file, I have removed the # from the ports as well to check - "8888:8888" # OtelCollector internal metrics - "8889:8889" # signoz spanmetrics exposed by the agent
s
What do you mean "not collecting properly"
j
I cannot see any new metrics comming to the clickhouse DB. After the Signoz collector is running for several hours, still only have the same 3 rows for system_cpu_load_average_1m, which should not happen
Hello everyone, I'm trying to run the hostmetrics dashboard (following the instructions in signoz.io/docs>> Infrastructure Monitoring>>Hostmetrics Dashboard , but everything is blank. At the beginning the first error was when performing the
Copy code
SELECT DISTINCT JSONExtractString(labels, 'host_name')
FROM signoz_metrics.time_series_v2
in the variable "host_name". when replicating the query used in the json from the clickhouse client I got the following:
Copy code
Received exception from server (version 24.1.2):
Code: 173. DB::Exception: Received from 127.0.0.1:9000. DB::Exception: Couldn't allocate 333 bytes when parsing JSON: while executing 'FUNCTION JSONExtractString(labels :: 0, 'host_name' :: 1) -> JSONExtractString(labels, 'host_name') String : 2'. (CANNOT_ALLOCATE_MEMORY)
Now, I have changed it to
Copy code
SELECT DISTINCT(simpleJSONExtractRaw(labels, 'host_name'))
FROM signoz_metrics.time_series_v2.
I can finally see the name of my host (UEcollector) in the list on hostmetrics dashboard, but when selecting it, the dashboard is empty. When I see the information of a panel that for example internally runs the query
Copy code
sum by(host_name) (system_memory_usage{host_name="{{.hostname}}",state="used"}) / sum by(host_name) (system_memory_usage{host_name="{{.hostname}}"})
, it shows the error:
Copy code
API responded with 400 - error in prom queries status: error, errors: {"A":"1:52: parse error: unexpected identifier \"UEcollector\" in label matching, expected \",\" or \"}\""}
When running the query
Copy code
SELECT *
FROM signoz_metrics.time_series_v2
WHERE metric_name = 'system_memory_usage'
directly in clickhouse DB, it only returns 6 rows in the form:
Copy code
│ system_memory_usage │ 17253326157141099054 │ 1715081635962 │ {"__name__":"system_memory_usage","__temporality__":"Cumulative","host_name":"UEcollector","os_type":"linux","state":"used"}                   │ Cumulative  │ Bytes of memory in use. │ By   │ Sum  │ false        │
Also I cannot see any value for the metric in the label section. Could anyone provide some support? Thanks in advance
s
Copy code
SELECT DISTINCT(simpleJSONExtractRaw(labels, 'host_name'))
FROM signoz_metrics.time_series_v2.
This is the reason for failure. Please use the original query and allocate some more memory to avoid.
Copy code
Received exception from server (version 24.1.2):
Code: 173. DB::Exception: Received from 127.0.0.1:9000. DB::Exception: Couldn't allocate 333 bytes when parsing JSON: while executing 'FUNCTION JSONExtractString(labels :: 0, 'host_name' :: 1) -> JSONExtractString(labels, 'host_name') String : 2'. (CANNOT_ALLOCATE_MEMORY)
j
how can I allocate more memory to the clickhouse container or what do you mean? Also, when querying directly the database, as mentioned, cannot really see new incoming rows about hostmetrics. If I'm monitoring each 30s, I assume a new query should appear in the DB after this time. or maybe I'm not understanding this properly. Thanks
s
how can I allocate more memory to the clickhouse container or what do you mean?
The error from ClickHouse says there is not enough memory. How much memory did you provision? Use the 
simpleJSONExtractRaw
is the source of the issue.
Also, when querying directly the database, as mentioned, cannot really see new incoming rows about hostmetrics. If I'm monitoring each 30s, I assume a new query should appear in the DB after this time. or maybe I'm not understanding this properly.
There might some errors in the collector logs or clickhouse. Please find and address them
j
I have assigned 4GB to the clickhouse container so I dont think that is the problem Here is the dockercompose:
Copy code
services:
  clickhouse:
    <<: *clickhouse-defaults
    container_name: signoz-clickhouse
    hostname: clickhouse
    ports:
      - "9000:9000"
      - "8123:8123"
      - "9181:9181"
    volumes:
      - ./clickhouse-config.xml:/etc/clickhouse-server/config.xml
      - ./clickhouse-users.xml:/etc/clickhouse-server/users.xml
      - ./custom-function.xml:/etc/clickhouse-server/custom-function.xml
      - ./clickhouse-cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
      # - ./clickhouse-storage.xml:/etc/clickhouse-server/config.d/storage.xml
      - ./data/clickhouse/:/var/lib/clickhouse/
      - ./user_scripts:/var/lib/clickhouse/user_scripts/
    mem_limit: 4g  # Set memory limit to 4 GB
Also, could you please provide a detailed solution on how to access and see the errors in the collector logs or clickhouse? Finally, according to my understanding, the query 
SELECT DISTINCT(simpleJSONExtractRaw(labels, 'host_name'))
FROM signoz_metrics.time_series_v2.
is only used to get the names of the host in the hostmetrics dashbord. Then if from one of the panels from the SigNoz Hostmetrics dashboard (lets use memory usage as an example), in the promQL Query, I manually replace the 
{{.hostname}}
variable by the name of my host (like below), I should get the data
Copy code
# in a query such as:

sum by(host_name) (system_memory_usage{host_name="{{.hostname}}",state="used"}) / sum by(host_name) (system_memory_usage{host_name="{{.hostname}}"})

# i replace the variable with my host name, such as 
sum by(host_name) (system_memory_usage{host_name="UEcollector",state="used"}) / sum by(host_name) (system_memory_usage{host_name="UEcollector"})
However, I don´t get any data. This leads me to think there is a different error rather with collecting the data or how I try to access to it
112 Views
Open in Slack
PreviousNext
Powered by

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.

Powered by