https://signoz.io logo
Title
m

Muhammed Hussein Karimi

02/28/2023, 9:17 AM
Hi, After adding json parser to my logs in
k8s-infra
log counter in
logs
section works but there is no data in events section, also fields are loaded correctly
a

Ankit Nayan

02/28/2023, 9:25 AM
cc: @nitya-signoz
n

nitya-signoz

02/28/2023, 9:39 AM
Can you share the configuration? Also if your fields are loaded but not visible in the UI there might be issues with timestamp. Can you run the commands and share output
kubectl exec -n platform -it chi-my-release-clickhouse-cluster-0-0-0 -- sh

clickhouse client

use signoz_logs;

select timestamp,observed_timestamp from distributed_logs order by timestamp desc limit 10;
m

Muhammed Hussein Karimi

02/28/2023, 9:41 AM
┌───────────timestamp─┬──observed_timestamp─┐
│ 1677577033101391005 │ 1677577033114197754 │
│ 1677577033087210293 │ 1677577033114149167 │
│ 1677577033081940586 │ 1677577033113942963 │
│ 1677577033079232086 │ 1677577033114117714 │
│ 1677577033074985781 │ 1677577033114088974 │
│ 1677577033071399878 │ 1677577033113746096 │
│ 1677577033068867909 │ 1677577033114495821 │
│ 1677577033058288277 │ 1677577033114040042 │
│ 1677577033054256687 │ 1677577033114446222 │
│ 1677577033036741379 │ 1677577033114394174 │
└─────────────────────┴─────────────────────┘
image.png
here you go
n

nitya-signoz

02/28/2023, 9:43 AM
interesting, can you share the API response of
/logs
endpoint from the network tab?
Along with the request params ^
m

Muhammed Hussein Karimi

02/28/2023, 9:49 AM
There is no request with
/logs
only
aggregate
n

nitya-signoz

02/28/2023, 9:50 AM
What version of SigNoz are you using?
@Palash Gupta can there be a case where /logs is not triggered on the UI ?
m

Muhammed Hussein Karimi

02/28/2023, 9:51 AM
v0.16.0
Oh found some after a page refresh
{
  "status": "error",
  "data": "Failed to fetch logs from the DB",
  "errorType": "internal",
  "error": "code: 241, message: Memory limit (total) exceeded: would use 3.62 GiB (attempt to allocate chunk of 5064994 bytes), maximum: 3.60 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker.: (while reading column attributes_string_value): While executing MergeTreeReverse"
}
here is the clickhouse resources
resources:
        requests:
          cpu: 100m
          memory: 200Mi
        limits:
          cpu: 2000m
          memory: 4Gi
I should increase resources? or there are other settings?
n

nitya-signoz

02/28/2023, 9:56 AM
Yeah please increase the resources and try it out first. There are issues about this on clickhouse https://github.com/ClickHouse/ClickHouse/issues/17631 . But you can start by doubling up the resources and trying it out.
p

Palash Gupta

02/28/2023, 10:34 AM
can there be a case where /logs is not triggered on the UI ?
I am not aware of such case... @Muhammed Hussein Karimi can you please share the loom video for the same
m

Muhammed Hussein Karimi

02/28/2023, 10:45 AM
Fixed by increasing the memory limits
p

Palash Gupta

02/28/2023, 10:48 AM
oh great ... @nitya-signoz can we update same in the docs