https://signoz.io logo
Join Slack
Powered by
Hi, I want to add labels to logs in signoz, i have...
# support
t
Hi, I want to add labels to logs in signoz, i have JSON body logs and i want to catch key from this JSON to be label like container_name and cluster_name, i'm using kubernetes signoz
n
t
i'm using this file https://github.com/SigNoz/charts/blob/main/charts/k8s-infra/values.yaml to ship the logs to signoz I have key under body json called
Copy code
tenantId
how i can use this one as label Can you help me on that ?
n
t
I will modify the default one, but what's the change should be in this case
Copy code
logstransform/internal:
        operators:
        - if: '"trace_id" in attributes or "span_id" in attributes'
          output: remove_trace_id
          span_id:
            parse_from: attributes.span_id
          trace_id:
            parse_from: attributes.trace_id
          type: trace_parser
        - if: '"traceId" in attributes or "spanId" in attributes'
          output: remove_traceId
          span_id:
            parse_from: attributes.spanId
          trace_id:
            parse_from: attributes.traceId
          type: trace_parser
        - field: attributes.traceId
          id: remove_traceId
          if: '"traceId" in attributes'
          output: remove_spanId
          type: remove
        - field: attributes.spanId
          id: remove_spanId
          if: '"spanId" in attributes'
          type: remove
        - field: attributes.trace_id
          id: remove_trace_id
          if: '"trace_id" in attributes'
          output: remove_span_id
          type: remove
        - field: attributes.span_id
          id: remove_span_id
          if: '"span_id" in attributes'
          type: remove
the content of logstransform/internal
n
It’s better to create a new one for parsing your logs. If you need help in that you will have to share your logs you see in UI.
t
this is the log json
i want to make tenantId as label
what i should do in this case
@nitya-signoz
n
Copy code
otelCollector:
  config:
    processors:
      logstransform/parse_tid:
        operators:
          - default: noop
            id: router_signoz
            type: router
            routes:
              - expr: 'body matches ".*\\\"tenantId\\\":\\\"(?P<tenantId>\\w+)\\\""'
                output: extract_tenantId
          - id: extract_tenantId
            type: regex_parser
            regex: '.*\"tenantId\":\"(?P<tenantId>\w+)\"'
            parse_from: body
            output: noop
          - id: noop
            type: noop
    service:
      pipelines:
        logs:
          receivers: [otlp]
          processors: [logstransform/parse_tid, logstransform/internal, batch]
          exporters: [clickhouselogsexporter]
t
testing it, thank you
i tested it but it's still not appeared in the new logs
even this is not appeared in INTERESTING FIELDS
i'm sorry i found something, it's body inside body
how i can do it in this case
the key tenant id is
body > body > tenant id
s
can you share a complete json log?
t
one moment
Copy code
{
  "timestamp": 1692187765868840000,
  "id": "2U48JQCA8Ob7k3tYQZZouyDNA6A",
  "trace_id": "",
  "span_id": "",
  "trace_flags": 0,
  "severity_text": "",
  "severity_number": 0,
  "body": "{\"name\":\"xxx\",\"hostname\":\"xxx-xxx-644d656d99-c24x5\",\"pid\":17,\"component\":\"nodeWorkerSQS\",\"level\":\"INFO\",\"messageType\":\"integration_process\",\"body\":{\"type\":\"integration_process\",\"integrationType\":\"twitter\",\"tenantId\":\"xxxx\",\"onboarding\":false,\"integrationId\":\"xxxx\",\"retryStreams\":[{\"id\":\"xxxxx\",\"retryCount\":1,\"stream\":{\"value\":\"mentions\",\"metadata\":{\"page\":\"\"}}}],\"remainingStreams\":[{\"value\":\"followers\",\"metadata\":{\"page\":\"\"}}]},\"msg\":\"Sending xxx sqs message!\",\"time\":\"2023-08-16T12:09:25.868Z\",\"v\":0}",
  "resources_string": {
    "k8s_cluster_name": "xxxx",
    "k8s_container_name": "xxx",
    "k8s_container_restart_count": "0",
    "k8s_deployment_name": "xxx-xxx",
    "k8s_namespace_name": "default",
    "k8s_node_name": "xxxxx",
    "k8s_pod_name": "xxxx",
    "k8s_pod_start_time": "2023-08-14 10:08:54 +0000 UTC",
    "k8s_pod_uid": "b1e86f6d-42b6-46f9-903c-d5269a6b44a9"
  },
  "attributes_string": {
    "log_file_path": "/var/log/pods/default_xxx-xxx-644d656d99-c24x5_b1e86f6d-42b6-46f9-903c-d5269a6b44a9/xxx/0.log",
    "log_iostream": "stdout",
    "logtag": "F",
    "time": "2023-08-16T12:09:25.868839814Z"
  },
  "attributes_int": {},
  "attributes_float": {}
}
this is the complete log f
s
Okay, if I look at body, it is
Copy code
{
  "name": "xxx",
  "hostname": "xxx-xxx-644d656d99-c24x5",
  "pid": 17,
  "component": "nodeWorkerSQS",
  "level": "INFO",
  "messageType": "integration_process",
  "body": {
    "type": "integration_process",
    "integrationType": "twitter",
    "tenantId": "xxxx",
    "onboarding": false,
    "integrationId": "xxxx",
    "retryStreams": [
      {
        "id": "xxxxx",
        "retryCount": 1,
        "stream": {
          "value": "mentions",
          "metadata": {
            "page": ""
          }
        }
      }
    ],
    "remainingStreams": [
      {
        "value": "followers",
        "metadata": {
          "page": ""
        }
      }
    ]
  },
  "msg": "Sending xxx sqs message!",
  "time": "2023-08-16T12:09:25.868Z",
  "v": 0
}
t
Yes
s
t
@nitya-signoz can you edit the example to shared above
s
Copy code
otelCollector:
  config:
    processors:
      logstransform/parse_tid:
        operators:
          - type: json_parser
            id: parser-docker
            output: extract_metadata_from_filepath
            timestamp:
              parse_from: body.time
              layout: '%%Y-%%m-%%dT%%H:%%M:%%S.%%LZ'
    service:
      pipelines:
        logs:
          receivers: [otlp]
          processors: [logstransform/parse_tid, logstransform/internal, batch]
          exporters: [clickhouselogsexporter]
With this, you should see be able to see an attribute 
body.body.tenantId
with value as your tanantId
if you want to rename the attribute, you can use
Copy code
otelCollector:
  config:
    processors:
      logstransform/parse_tid:
        operators:
          - type: json_parser
            id: parser-docker
            output: extract_metadata_from_filepath
            timestamp:
              parse_from: body.time
              layout: '%%Y-%%m-%%dT%%H:%%M:%%S.%%LZ'
          - type: move
            from: body.body.tenantId
            to: attributes["k8s.tenantId"]
    service:
      pipelines:
        logs:
          receivers: [otlp]
          processors: [logstransform/parse_tid, logstransform/internal, batch]
          exporters: [clickhouselogsexporter]
t
Copy code
2023/08/16 12:37:40 application run finished with error: failed to build pipelines: failed to create "logstransform/parse_tid" processor, in pipeline "logs": operator 'extract_metadata_from_filepath' does not exist: {"operator_id":"parser-docker"}
n
Copy code
otelCollector:
  config:
    processors:
      logstransform/parse_tid:
        operators:
          - default: noop
            id: router_signoz
            routes:
              - expr: 'body matches "^{.*}$"'
                output: parse_json
            type: router
          # json parser
          - id: parse_json
            type: json_parser
            parse_from: body
            parse_to: attributes.temp
            output: move_tenantId
          - type: move
            id: move_tenantId
            from: attributes.temp.body.tenantId
            to: attributes.tenantId
            if: '"body" in attributes.temp and "tenantId" in attributes.temp.body'
            output: remove_temp
          - type: remove
            id: remove_temp
            field: attributes.temp
            output: noop
          - id: noop
            type: noop
    service:
      pipelines:
        logs:
          receivers: [otlp]
          processors: [logstransform/parse_tid, logstransform/internal, batch]
          exporters: [clickhouselogsexporter]
t
working like charm, thank you, i have another question what if i want to add another field, should i add new processor or can one processor do both
n
Add one more operator like this in the above config.
Copy code
- type: move
            id: move_tenantId
            from: attributes.temp.body.tenantId
            to: attributes.tenantId
            if: '"body" in attributes.temp and "tenantId" in attributes.temp.body'
            output: remove_temp
24 Views
Open in Slack
PreviousNext
Powered by