https://signoz.io logo
Join Slack
Powered by
Hey, any idea why some filters don't seem to work?...
# support
p
Hey, any idea why some filters don't seem to work? For instance if I go to logs, click on a given log, open the attributes, click on something like k8s_pod_name, it will filter by that pod name. but nothing shows up...
s
That shouldn’t be the case. Can you share some screenshot/small video where selecting the pod name result in no data?
p
so like
but clicking k8s_node_name or k8s_namespace_name works just fine
like k8s_pod_name isn't indexed or something...
oh and I upgraded a few times via the helm chart.
Copy code
SELECT k8s_container_name FROM logs WHERE k8s_container_name != '' LIMIT 10;

SELECT k8s_container_name
FROM logs
WHERE k8s_container_name != ''
LIMIT 10

Query id: 16ad9816-9627-4161-9da9-69ace8e23398

Ok.

0 rows in set. Elapsed: 0.462 sec. Processed 171.96 million rows, 1.55 GB (372.60 million rows/s., 3.35 GB/s.)
thats probably the cause...
Copy code
│ CREATE TABLE signoz_logs.logs
(
    `timestamp` UInt64 CODEC(DoubleDelta, LZ4),
    `observed_timestamp` UInt64 CODEC(DoubleDelta, LZ4),
    `id` String CODEC(ZSTD(1)),
    `trace_id` String CODEC(ZSTD(1)),
    `span_id` String CODEC(ZSTD(1)),
    `trace_flags` UInt32,
    `severity_text` LowCardinality(String) CODEC(ZSTD(1)),
    `severity_number` UInt8,
    `body` String CODEC(ZSTD(2)),
    `resources_string_key` Array(String) CODEC(ZSTD(1)),
    `resources_string_value` Array(String) CODEC(ZSTD(1)),
    `attributes_string_key` Array(String) CODEC(ZSTD(1)),
    `attributes_string_value` Array(String) CODEC(ZSTD(1)),
    `attributes_int64_key` Array(String) CODEC(ZSTD(1)),
    `attributes_int64_value` Array(Int64) CODEC(ZSTD(1)),
    `attributes_float64_key` Array(String) CODEC(ZSTD(1)),
    `attributes_float64_value` Array(Float64) CODEC(ZSTD(1)),
    `k8s_pod_name` String MATERIALIZED attributes_string_value[indexOf(attributes_string_key, 'k8s_pod_name')] CODEC(LZ4),
    `k8s_container_name` String MATERIALIZED attributes_string_value[indexOf(attributes_string_key, 'k8s_container_name')] CODEC(LZ4),
    `service_name` String MATERIALIZED resources_string_value[indexOf(resources_string_key, 'service_name')] CODEC(LZ4),
    `event_domain` String MATERIALIZED attributes_string_value[indexOf(attributes_string_key, 'event_domain')] CODEC(LZ4),
    `event_name` String MATERIALIZED attributes_string_value[indexOf(attributes_string_key, 'event_name')] CODEC(LZ4),
    `k8s_namespace_name` String MATERIALIZED attributes_string_value[indexOf(attributes_string_key, 'k8s_namespace_name')] CODEC(LZ4),
    `k8s_cluster_name` String MATERIALIZED resources_string_value[indexOf(resources_string_key, 'k8s_cluster_name')] CODEC(LZ4),
    `k8s_node_name` String MATERIALIZED resources_string_value[indexOf(resources_string_key, 'k8s_node_name')] CODEC(LZ4),
    `os_type` String MATERIALIZED resources_string_value[indexOf(resources_string_key, 'os_type')] CODEC(LZ4),
    `k8s_deployment_name` String MATERIALIZED resources_string_value[indexOf(resources_string_key, 'k8s_deployment_name')] CODEC(LZ4),
    `enduser_id` String MATERIALIZED attributes_string_value[indexOf(attributes_string_key, 'enduser_id')] CODEC(LZ4),
    INDEX body_idx body TYPE tokenbf_v1(10240, 3, 0) GRANULARITY 4,
    INDEX id_minmax id TYPE minmax GRANULARITY 1,
    INDEX k8s_container_name_idx k8s_container_name TYPE bloom_filter(0.01) GRANULARITY 64,
    INDEX service_name_idx service_name TYPE bloom_filter(0.01) GRANULARITY 64,
    INDEX event_name_idx event_name TYPE bloom_filter(0.01) GRANULARITY 64,
    INDEX event_domain_idx event_domain TYPE bloom_filter(0.01) GRANULARITY 64,
    INDEX trace_id_idx trace_id TYPE bloom_filter(0.01) GRANULARITY 64,
    INDEX k8s_pod_name_idx k8s_pod_name TYPE bloom_filter(0.01) GRANULARITY 64,
    INDEX k8s_deployment_name_idx k8s_deployment_name TYPE bloom_filter(0.01) GRANULARITY 64,
    INDEX enduser_id_idx enduser_id TYPE bloom_filter(0.01) GRANULARITY 64
)
ENGINE = MergeTree
PARTITION BY toDate(timestamp / 1000000000)
ORDER BY (timestamp, id)
TTL toDateTime(timestamp / 1000000000) + toIntervalSecond(1209600)
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1 │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.009 sec.
looks like pod name and container name are stored as 
resources_string.k8s_pod_name
in signoz, but the index has it as 
attributes_string.k8s_pod_name
s
All the 
k8s_*
should be part of resource attributes only. Looks like there is some issue. Did you make any changes to charts and otel collector config?
p
nope...
my helm values:
Copy code
clickhouse:
  layout:
    shardsCount: 3
  resources:
    limits:
      cpu: 4
      memory: 6Gi
k8s-infra:
  otelDeployment:
    resources:
      limits:
        cpu: 1
        memory: 2Gi
  presets:
    hostMetrics:
      enabled: true
    kubeletMetrics:
      enabled: true
    logsCollection:
      blacklist:
        namespaces:
          - kube-system
          - openshift*
          - grafana-stack
          - platform
          - istio-system
otelCollector:
  replicaCount: 3
  resources:
    limits:
      cpu: 4
      memory: 8Gi
otel-agent-config.yaml
otel-agent-config.yaml
s
Entirely not clear what’s happening. Will have to debug deeper. I think there was some similar issue let me check.
p
ok thanks!
oh and while we are at it, when I select a field in logs it gets added twice
image.png
don't know if thats related
and I can't seem to add "body" to the table view
n
Hi Patrick, this was an issue which went out in one of our releases, you can fix it by following this guide https://signoz.io/docs/userguide/logs_troubleshooting/#k8s-attribute-filtering-issue-in-logs
2 Views
Open in Slack
PreviousNext
Powered by