This message was deleted.

Are you sure there are cert-manager logs in the selected time range?

Yes 100% certain, I extended the range to 1 week. I used cert-manager as an example, but filtering by k8s_namespace_name simply is not working for any namespace at all. As an example using a specific log entry with

id '2OZrlPONFe1F2RDNZoX2meXseev'

that has the following fields:

k8s_namespace_name ('cert-manager')

k8s_node_name ('00000a')

k8s_pod_name ('cert-manager-9997bf6c9-5t98x')

Filter

k8s_namespace_name ('cert-manager')

does NOT work. Filter

id IN ('2OZrlPONFe1F2RDNZoX2meXseev')

works! Filter

id IN ('2OZrlPONFe1F2RDNZoX2meXseev') AND k8s_namespace_name IN ('cert-manager')

does NOT work. Filter

k8s_node_name ('00000a')

works! Filter

k8s_node_name ('00000a') AND k8s_namespace_name IN ('cert-manager')

does NOT work. Filter

k8s_pod_name ('cert-manager-9997bf6c9-5t98x')

works! Filter

k8s_pod_name ('cert-manager-9997bf6c9-5t98x')  AND k8s_namespace_name IN ('cert-manager')

does NOT work. @Srikanth Chekuri Thanks for the response.

@Nityananda Gohain would be the best person to answer this.

Hi @Nityananda Gohain In the UI if I attempt to filter logs by

k8s_namespace_name IN ('lecreuset')

it results in No logs lines found I see the following in the query service logs:

2023-04-24T19:48:02.058Z     [35mDEBUG [0m  clickhouseReader/reader.go:3612 SELECT toInt64(toUnixTimestamp(toStartOfInterval(toDateTime(timestamp/1000000000), INTERVAL 1 minute))*1000000000) as ts_start_interval, toFloat64(count()) as value FROM signoz_logs.distributed_logs WHERE (timestamp >= '1682362080929000000' AND timestamp <= '1682365680929000000' ) AND ( k8s_namespace_name IN ('lecreuset')  )  GROUP BY ts_start_interval ORDER BY ts_start_interval
2023-04-24T19:48:02.060Z     [34mINFO [0m   app/server.go:277   /api/v1/logs/fields timeTaken: 18.868263ms
2023-04-24T19:48:02.060Z     [35mDEBUG [0m  clickhouseReader/reader.go:3468 SELECT timestamp, id, trace_id, span_id, trace_flags, severity_text, severity_number, body,CAST((attributes_string_key, attributes_string_value), 'Map(String, String)') as  attributes_string,CAST((attributes_int64_key, attributes_int64_value), 'Map(String, Int64)') as  attributes_int64,CAST((attributes_float64_key, attributes_float64_value), 'Map(String, Float64)') as  attributes_float64,CAST((resources_string_key, resources_string_value), 'Map(String, String)') as resources_string  from signoz_logs.distributed_logs where ( timestamp >= '1682362080929000000' and timestamp <= '1682365680929000000' ) and ( k8s_namespace_name IN ('lecreuset') )  order by timestamp desc limit 50
2023-04-24T19:48:02.072Z     [34mINFO [0m   app/server.go:277   /api/v1/logs/aggregate  timeTaken: 31.461604ms
2023-04-24T19:48:02.076Z     [34mINFO [0m   app/server.go:277   /api/v1/logs    timeTaken: 35.186776ms
2023-04-24T19:48:09.901Z     [34mINFO [0m   app/server.go:277   /api/v1/version timeTaken: 19.7µs
2023-04-24T19:48:09.901Z     [34mINFO [0m   app/server.go:277   /api/v1/version timeTaken: 47.901µs

If I query clickhouse directly using `k8s_container_name IN ('bizApp')`it works and notice that the log entry returned contains 'k8s_namespace_name':'lecreuset'

SELECT
    timestamp,
    id,
    trace_id,
    span_id,
    trace_flags,
    severity_text,
    severity_number,
    body,
    CAST((attributes_string_key, attributes_string_value), 'Map(String, String)') AS attributes_string,
    CAST((attributes_int64_key, attributes_int64_value), 'Map(String, Int64)') AS attributes_int64,
    CAST((attributes_float64_key, attributes_float64_value), 'Map(String, Float64)') AS attributes_float64,
    CAST((resources_string_key, resources_string_value), 'Map(String, String)') AS resources_string
FROM signoz_logs.distributed_logs
WHERE ((timestamp >= '1682363725243000000') AND (timestamp <= '1682367325243000000')) AND (k8s_container_name IN ('bizApp'))
ORDER BY timestamp DESC
LIMIT 1

Query id: 0a4b97c0-78a0-46aa-991b-b6f703fc2cb5

┌───────────timestamp─┬─id──────────────────────────┬─trace_id─┬─span_id─┬─trace_flags─┬─severity_text─┬─severity_number─┬─body───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─attributes_string──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─attributes_int64─┬─attributes_float64─┬─resources_string───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 1682367322861417200 │ 2OZrlPONFe1F2RDNZoX2meeVMcU │          │         │           0 │               │               0 │ <Source>EtwEvent</Source><Time>2023-04-24T20:15:19.000Z</Time><Provider idGuid="{E13C0D23-CCBC-4E12-931B-D9CC2EEE27E4}"/><DecodingSource>DecodingSourceXMLFile</DecodingSource><Execution ProcessID="21848" ThreadID="15420" /><Level>None</Level><Keyword>0x40000000</Keyword><EventID Qualifiers="82">82</EventID><EventData><ClrInstanceID>44</ClrInstanceID><Reserved1>0</Reserved1><Reserved2>0</Reserved2><FrameCount>41</FrameCount><Stack>0x7FFE7978127D</Stack><Stack>0x7FFE797842D8</Stack></EventData> │ {'time':'2023-04-24T20:15:22.8614172Z','logtag':'F','log_file_path':'\\var\\log\\pods\\lecreuset_bizApp-6b59cbf5f7-qzsjf_b8866b01-eb0b-400a-af43-27b132db6d45\\bizApp\\0.log','log_iostream':'stdout','env':'prod','region':'east-us'} │ {}               │ {}                 │ {'k8s_namespace_name':'lecreuset','k8s_pod_name':'bizApp-6b59cbf5f7-qzsjf','k8s_container_restart_count':'0','k8s_pod_uid':'69563f93-a351-4a75-ad18-9994c5e652c5','k8s_container_name':'bizApp','host_name':'agentpool-000000','signoz_component':'otel-agent','k8s_cluster_name':'','k8s_pod_ip':'10.20.0.73','os_type':'windows','k8s_node_name':'agentpool-000000','k8s_pod_start_time':'2023-04-21 01:12:19 +0000 GMT'} │
└─────────────────────┴─────────────────────────────┴──────────┴─────────┴─────────────┴───────────────┴─────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────┴────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.035 sec. Processed 8.01 thousand rows, 9.90 MB (227.97 thousand rows/s., 281.89 MB/s.)

If I use the same query, but simply replace (k8s_container_name IN ('bizApp') with (k8s_namespace_name IN ('lecreuset') it does not work:

SELECT
    timestamp,
    id,
    trace_id,
    span_id,
    trace_flags,
    severity_text,
    severity_number,
    body,
    CAST((attributes_string_key, attributes_string_value), 'Map(String, String)') AS attributes_string,
    CAST((attributes_int64_key, attributes_int64_value), 'Map(String, Int64)') AS attributes_int64,
    CAST((attributes_float64_key, attributes_float64_value), 'Map(String, Float64)') AS attributes_float64,
    CAST((resources_string_key, resources_string_value), 'Map(String, String)') AS resources_string
FROM signoz_logs.distributed_logs
WHERE ((timestamp >= '1682363725243000000') AND (timestamp <= '1682367325243000000')) AND (k8s_namespace_name IN ('lecreuset'))
ORDER BY timestamp DESC
LIMIT 1

Query id: f9b2e396-c25c-43fa-9463-faeb855eb002

Ok.

0 rows in set. Elapsed: 0.018 sec.

@Srikanth Chekuri @Nityananda Gohain Should the query actually be

WHERE resources_string['k8s_namespace_name'] IN ('lecreuset')

instead of k8s_namespace_name IN ('lecreuset') ? I would really appreciate some support on this topic as we scope our customer resources by namespace and therefore filtering logs by namespace is essential for monitoring and troubleshooting deployments.