Join Slack

This message was deleted.

# support

Slackbot

12/07/2023, 5:15 PM

This message was deleted.

Nicolas Rakover

12/07/2023, 8:17 PM

cc @Samuel Bacaner

👀 1

Nicolas Rakover

12/07/2023, 10:08 PM

In an effort to confirm my hypothesis, I logged in to the ClickHouse instance and inspected the

signoz_logs.logs_attribute_keys

table:

Copy code

SELECT *
FROM signoz_logs.logs_attribute_keys

Copy code

┌─name──────────┬─datatype─┐
│ log_file_path │ String   │
│ log_iostream  │ String   │
│ logtag        │ String   │
│ time          │ String   │
└───────────────┴──────────┘
┌─name──────────┬─datatype─┐
│ log_file_path │ String   │
│ log_iostream  │ String   │
│ logtag        │ String   │
│ time          │ String   │
└───────────────┴──────────┘
┌─name──────────────────┬─datatype─┐
│ attributes            │ String   │
│ body                  │ String   │
│ chunker_config        │ String   │
│ data                  │ String   │
│ distance_metric       │ String   │
│ dropped_attributes    │ Float64  │
│ embedding_digest      │ String   │
│ embedding_size        │ Float64  │
│ exception_message     │ String   │
│ exception_stacktrace  │ String   │
│ exception_type        │ String   │
│ func                  │ String   │
│ log_file_path         │ String   │
│ log_iostream          │ String   │
│ logtag                │ String   │
│ max_batch_size        │ Float64  │
│ max_input_char_length │ Float64  │
│ object                │ String   │
│ otelServiceName       │ String   │
│ otelSpanID            │ String   │
│ otelTraceID           │ String   │
│ otelTraceSampled      │ Bool     │
│ resource              │ String   │
│ session_id            │ String   │
│ severity_number       │ String   │
│ severity_text         │ String   │
│ solver_service_log    │ Bool     │
│ span_id               │ String   │
│ time                  │ String   │
│ timestamp             │ String   │
│ trace_flags           │ Float64  │
│ trace_id              │ String   │
│ unixtime              │ Float64  │
└───────────────────────┴──────────┘

41 rows in set. Elapsed: 0.007 sec.

Nicolas Rakover

12/07/2023, 10:09 PM

If I'm interpreting this correctly, there is in fact a

body

attribute. I'm not sure how the query service around that deals with the ambiguity between

body

and

attributes.body

, but the symptoms I'm seeing suggest this is a likely culprit

Nicolas Rakover

12/07/2023, 10:10 PM

Would it be safe to surgically delete these rows in the

signoz_logs.logs_attribute_keys

table to restore service?

Nicolas Rakover

12/07/2023, 11:31 PM

Hi @nitya-signoz the Logs Explorer is completely unusable for us right now -- any guidance would be appreciated

nitya-signoz

12/08/2023, 4:00 AM

When you open the explorer page, can you share the response of query range api from the network tab ?

Nicolas Rakover

12/08/2023, 4:51 AM

This is the response from the

api/v3/autocomplete/attribute_keys

route:

Copy code

{
  "status": "success",
  "data": {
    "attributeKeys": [
      {
        "key": "k8s_cluster_name",
        "dataType": "string",
        "type": "resource",
        "isColumn": true,
        "isJSON": false
      },
      {
        "key": "k8s_container_name",
        "dataType": "string",
        "type": "resource",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "k8s_container_restart_count",
        "dataType": "string",
        "type": "resource",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "k8s_deployment_name",
        "dataType": "string",
        "type": "resource",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "k8s_namespace_name",
        "dataType": "string",
        "type": "resource",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "k8s_node_name",
        "dataType": "string",
        "type": "resource",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "k8s_pod_name",
        "dataType": "string",
        "type": "resource",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "k8s_pod_start_time",
        "dataType": "string",
        "type": "resource",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "k8s_pod_uid",
        "dataType": "string",
        "type": "resource",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "log_file_path",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "log_iostream",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "logtag",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "time",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "attributes",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "body",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "chunker_config",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "data",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "distance_metric",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "dropped_attributes",
        "dataType": "float64",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "embedding_digest",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "embedding_size",
        "dataType": "float64",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "exception_message",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "exception_stacktrace",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "exception_type",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "k8s_pod_ip",
        "dataType": "string",
        "type": "resource",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "max_batch_size",
        "dataType": "float64",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "max_input_char_length",
        "dataType": "float64",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "object",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "otelServiceName",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "otelSpanID",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "otelTraceID",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "otelTraceSampled",
        "dataType": "bool",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "resource",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "service_name",
        "dataType": "string",
        "type": "resource",
        "isColumn": true,
        "isJSON": false
      },
      {
        "key": "service_sha",
        "dataType": "string",
        "type": "resource",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "service_version",
        "dataType": "string",
        "type": "resource",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "severity_number",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "severity_text",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "span_id",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "telemetry_auto_version",
        "dataType": "string",
        "type": "resource",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "telemetry_sdk_language",
        "dataType": "string",
        "type": "resource",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "telemetry_sdk_name",
        "dataType": "string",
        "type": "resource",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "telemetry_sdk_version",
        "dataType": "string",
        "type": "resource",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "timestamp",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "trace_flags",
        "dataType": "float64",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "trace_id",
        "dataType": "string",
        "type": "tag",
        "isColumn": false,
        "isJSON": false
      },
      {
        "key": "span_id",
        "dataType": "string",
        "type": "",
        "isColumn": true,
        "isJSON": false
      },
      {
        "key": "trace_flags",
        "dataType": "int64",
        "type": "",
        "isColumn": true,
        "isJSON": false
      },
      {
        "key": "severity_text",
        "dataType": "string",
        "type": "",
        "isColumn": true,
        "isJSON": false
      },
      {
        "key": "severity_number",
        "dataType": "int64",
        "type": "",
        "isColumn": true,
        "isJSON": false
      },
      {
        "key": "body",
        "dataType": "string",
        "type": "",
        "isColumn": true,
        "isJSON": false
      },
      {
        "key": "trace_id",
        "dataType": "string",
        "type": "",
        "isColumn": true,
        "isJSON": false
      }
    ]
  }
}

Nicolas Rakover

12/08/2023, 4:51 AM

You can see two occurrences of

body

, one with no type (I assume the top-level/native key) and another as a

tag

nitya-signoz

12/08/2023, 4:52 AM

Try refreshing the page, i need the response of query range api

Nicolas Rakover

12/08/2023, 4:53 AM

got it, here it is

Copy code

{
  "status": "success",
  "data": {
    "resultType": "",
    "result": [
      {
        "queryName": "A",
        "series": null,
        "list": null
      }
    ]
  }
}

nitya-signoz

12/08/2023, 5:04 AM

do you still have the pipeline?

nitya-signoz

12/08/2023, 5:07 AM

you can disable the pipeline and truncate the

logs_attribute_keys

table by running the following on your clickhouse cli

Copy code

truncate table logs_attribute_keys on cluster cluster

Nicolas Rakover

12/08/2023, 5:50 AM

Thanks @nitya-signoz. I no longer have the pipeline, and I've truncated that table. However, filtering on

body

is still completely nonfunctional

nitya-signoz

12/08/2023, 7:40 AM

when you click on run then are there any errors ? check the network tab as well.

Nicolas Rakover

12/08/2023, 4:46 PM

I see no errors when I click

Run Query

, neither on the console nor the network tab. This is what I get back from `query_range`:

Copy code

{
  "status": "success",
  "data": {
    "resultType": "",
    "result": [
      {
        "queryName": "A",
        "series": null,
        "list": null
      }
    ]
  }
}

Nicolas Rakover

12/08/2023, 4:50 PM

I've truncated the

signoz_logs.logs_attribute_keys

table, which now contains the following rows:

Copy code

SELECT *
FROM signoz_logs.logs_attribute_keys

Query id: d821aeea-cd57-4595-83ae-9f67b4053eea

┌─name──────────┬─datatype─┐
│ log_file_path │ String   │
│ log_iostream  │ String   │
│ logtag        │ String   │
│ time          │ String   │
└───────────────┴──────────┘
┌─name──────────┬─datatype─┐
│ log_file_path │ String   │
│ log_iostream  │ String   │
│ logtag        │ String   │
│ time          │ String   │
└───────────────┴──────────┘
┌─name──────────┬─datatype─┐
│ log_file_path │ String   │
│ log_iostream  │ String   │
│ logtag        │ String   │
│ time          │ String   │
└───────────────┴──────────┘
┌─name─────────────────┬─datatype─┐
│ exception_stacktrace │ String   │
│ exception_type       │ String   │
│ log_file_path        │ String   │
│ log_iostream         │ String   │
│ logtag               │ String   │
│ otelServiceName      │ String   │
│ otelSpanID           │ String   │
│ otelTraceID          │ String   │
│ otelTraceSampled     │ Bool     │
│ time                 │ String   │
└──────────────────────┴──────────┘
┌─name──────────┬─datatype─┐
│ log_file_path │ String   │
│ log_iostream  │ String   │
│ logtag        │ String   │
│ time          │ String   │
└───────────────┴──────────┘

26 rows in set. Elapsed: 0.003 sec.

Nicolas Rakover

12/08/2023, 4:51 PM

This looks good to me, I don't see all the attributes that had been denormalized, such as

body

Nicolas Rakover

12/08/2023, 4:52 PM

However, the

attributes_keys

route in my networking tab is still returning a long list of keys including those that should no longer be present:

Nicolas Rakover

12/08/2023, 5:08 PM

As you can see in the request payload for the call to

query_range

Copy code

{
  "start": 1702051527000,
  "end": 1702055127000,
  "step": 60,
  "variables": {},
  "compositeQuery": {
    "queryType": "builder",
    "panelType": "list",
    "builderQueries": {
      "A": {
        "dataSource": "logs",
        "queryName": "A",
        "aggregateOperator": "noop",
        "aggregateAttribute": {
          "id": "------false",
          "dataType": "",
          "key": "",
          "isColumn": false,
          "type": "",
          "isJSON": false
        },
        "filters": {
          "items": [
            {
              "id": "4e73571b",
              "key": {
                "key": "body",
                "dataType": "string",
                "type": "tag",
                "isColumn": false,
                "isJSON": false,
                "id": "body--string--tag--false"
              },
              "op": "contains",
              "value": "127"
            }
          ],
          "op": "AND"
        },
        "expression": "A",
        "disabled": false,
        "having": [],
        "stepInterval": 60,
        "limit": null,
        "orderBy": [
          {
            "columnName": "timestamp",
            "order": "desc"
          }
        ],
        "groupBy": [],
        "legend": "",
        "reduceTo": "sum",
        "offset": 0,
        "pageSize": 100
      }
    }
  }
}

the filter is being defined on the

body

key of type

tag

, which is wrong (and as far as Clickhouse claims no longer exists)

Nicolas Rakover

12/08/2023, 5:09 PM

There must be some other source of truth being used to determine the valid attribute keys, and this is falling out of sync with the Clickhouse

signoz_logs.logs_attribute_keys

table

nitya-signoz

12/09/2023, 4:53 AM

can you truncate

tag_attributes

table as well

Nicolas Rakover

12/11/2023, 5:36 PM

Yup, ended up truncating

tag_attributes

and that did the trick 🙌

9 Views

Open in Slack

Previous Next

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.