Syed Muhammad Hassan
04/24/2023, 4:24 PM2023-04-23T12:09:52.345193051Z stdout F 10.110.245.37 - - [23/Apr/2023:12:09:52 +0000] "GET / HTTP/1.1" 200 615 "-" "Wget" "-"
Below is my otel-agent Configmap
receivers:
filelogs:
include: [/var/log/pods/*_php-nginx*_*/*/*.log]
start_at: beginning
operators:
- type: regex_parser
regex: '^(?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z)\s+(?P<stream>stdout)\s+(?P<severity>[A-Z])\s+(?P<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s+-\s+-\s+\[(?P<datetime>\d{2}/[A-Za-z]{3}/\d{4}:\d{2}:\d{2}:\d{2}\s+\+\d{4})\]\s+"(?P<method>[A-Z]+)\s+(?P<path>[^ ]+)\s+(?P<protocol>HTTP\/\d\.\d)"\s+(?P<status>\d+)\s+(?P<size>\d+)\s+"(?P<referer>[^"]+)"\s+"(?P<user_agent>[^"]+)"\s+"(?P<extra>[^"]+)"$'
output: extract_metadata_filepath
timestamp:
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
parse_from: attributes.time
- type: regex_parser
id: extract_metadata_filepath
regex: '(?P<host>[^ ]+)'
- from: attributes.host
to: resource["my.hosts"]
type: move
Ankit Nayan
nitya-signoz
04/24/2023, 6:13 PMSyed Muhammad Hassan
04/24/2023, 6:33 PMnitya-signoz
04/25/2023, 4:09 AMSyed Muhammad Hassan
04/25/2023, 5:25 AM2023-04-25T05:24:06.222112278Z stdout F 10.110.178.97 - - [25/Apr/2023:05:24:06 +0000] "GET / HTTP/1.1" 200 615 "-" "Wget" "-"
2023-04-25T05:24:07.226824031Z stdout F 10.110.178.97 - - [25/Apr/2023:05:24:07 +0000] "GET / HTTP/1.1" 200 615 "-" "Wget" "-"
2023-04-25T05:24:08.231429132Z stdout F 10.110.178.97 - - [25/Apr/2023:05:24:08 +0000] "GET / HTTP/1.1" 200 615 "-" "Wget" "-"
2023-04-25T05:24:09.236194447Z stdout F 10.110.178.97 - - [25/Apr/2023:05:24:09 +0000] "GET / HTTP/1.1" 200 615 "-" "Wget" "-"
2023-04-25T05:24:10.241420179Z stdout F 10.110.178.97 - - [25/Apr/2023:05:24:10 +0000] "GET / HTTP/1.1" 200 615 "-" "Wget" "-"
2023-04-25T05:24:11.246190982Z stdout F 10.110.178.97 - - [25/Apr/2023:05:24:11 +0000] "GET / HTTP/1.1" 200 615 "-" "Wget" "-"
2023-04-25T05:24:12.250794012Z stdout F 10.110.178.97 - - [25/Apr/2023:05:24:12 +0000] "GET / HTTP/1.1" 200 615 "-" "Wget" "-"
2023-04-25T05:24:13.2557379Z stdout F 10.110.178.97 - - [25/Apr/2023:05:24:13 +0000] "GET / HTTP/1.1" 200 615 "-" "Wget" "-"
2023-04-25T05:24:14.260639057Z stdout F 10.110.178.97 - - [25/Apr/2023:05:24:14 +0000] "GET / HTTP/1.1" 200 615 "-" "Wget" "-"
2023-04-25T05:24:15.265771256Z stdout F 10.110.178.97 - - [25/Apr/2023:05:24:15 +0000] "GET / HTTP/1.1" 200 615 "-" "Wget" "-"
2023-04-25T05:24:16.270501506Z stdout F 10.110.178.97 - - [25/Apr/2023:05:24:16 +0000] "GET / HTTP/1.1" 200 615 "-" "Wget" "-"
"details": {"parse_from": "attributes.times"}}, "action": "send", "entry": {"observed_timestamp":"2023-04-25T05:27:53.480511285Z","timestamp":"0001-01-01T00:00:00Z","body":"2023-04-25T05:27:53.373058091Z stdout F 10.110.178.97 - - [25/Apr/2023:05:27:53 +0000] \"GET / HTTP/1.1\" 200 615 \"-\" \"Wget\" \"-\"","attributes":{"host":"10.110.178.97","log.file.name":"0.log","logtag":"F","stream":"stdout","time":"2023-04-25T05:27:53.373058091Z"},"severity":0,"scope_name":""}}
But how will these attributes be captured in signoz frontend?nitya-signoz
04/25/2023, 7:08 AMtimestamp
parser helps to replace that with the correct timestamp the log was generated.
So you are saying that it is working once you changed/removed the timestamp parser ?Syed Muhammad Hassan
04/25/2023, 7:11 AMnitya-signoz
04/25/2023, 7:12 AMSyed Muhammad Hassan
04/25/2023, 7:14 AMnitya-signoz
04/25/2023, 7:17 AMSyed Muhammad Hassan
04/25/2023, 7:17 AMnitya-signoz
04/25/2023, 8:37 AMoperators:
- type: regex_parser
regex: '^(?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z)\s+(?P<stream>stdout)\s+(?P<severity>[A-Z])\s+(?P<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s+-\s+-\s+\[(?P<datetime>\d{2}/[A-Za-z]{3}/\d{4}:\d{2}:\d{2}:\d{2}\s+\+\d{4})\]\s+"(?P<method>[A-Z]+)\s+(?P<path>[^ ]+)\s+(?P<protocol>HTTP\/\d\.\d)"\s+(?P<status>\d+)\s+(?P<size>\d+)\s+"(?P<referer>[^"]+)"\s+"(?P<user_agent>[^"]+)"\s+"(?P<extra>[^"]+)"$'
output: extract_metadata_filepath
timestamp:
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
parse_from: attributes.timestamp
- type: regex_parser
id: extract_metadata_filepath
regex: '(?P<host>[^ ]+)'
- from: attributes.host
to: resource["my.hosts"]
type: move
Can you try this out, else we can schedule a call.Syed Muhammad Hassan
04/25/2023, 9:08 AMnitya-signoz
04/25/2023, 9:24 AM(?P<host>[^ ]+)
is not correct. Eg:- if the log line is
2023-04-25T05:24:06.222112278Z stdout F 10.110.178.97 - - [25/Apr/2023:05:24:06 +0000] "GET / HTTP/1.1" 200 615 "-" "Wget" "-"
and you want my_host as 10.110.178.97
?
For other fields that are extracted, on the left-hand side there are selected and interesting fields. If you convert interesting to selected fields it will appear in the main page. Else when you click on the expand button in the beginning of the log line you will be able to see the parsed values.operators:
- type: regex_parser
regex: '^(?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+Z)\s+(?P<stream>stdout)\s+(?P<severity>[A-Z])\s+(?P<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s+-\s+-\s+\[(?P<datetime>\d{2}/[A-Za-z]{3}/\d{4}:\d{2}:\d{2}:\d{2}\s+\+\d{4})\]\s+"(?P<method>[A-Z]+)\s+(?P<path>[^ ]+)\s+(?P<protocol>HTTP\/\d\.\d)"\s+(?P<status>\d+)\s+(?P<size>\d+)\s+"(?P<referer>[^"]+)"\s+"(?P<user_agent>[^"]+)"\s+"(?P<extra>[^"]+)"$'
output: move_host
timestamp:
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
parse_from: attributes.timestamp
- from: attributes.ip
id: move_host
to: resource["my.hosts"]
type: move
if you can spare sometime for a short call?Sure once we are clear on the above comments we can get on a call if required.
Syed Muhammad Hassan
04/25/2023, 9:29 AMnitya-signoz
04/25/2023, 9:30 AM- type: regex_parser
id: extract_metadata_filepath
regex: '(?P<host>[^ ]+)'
what are you trying to extract is what I was trying to ask?Syed Muhammad Hassan
04/25/2023, 9:30 AMnitya-signoz
04/25/2023, 9:30 AMprocessors:
logstransform/parse_log:
operators:
- default: noop
id: router_signoz
routes:
- expr: 'body matches ".*user-id=.*trace-id=.*span-id.*line"'
output: parse_regex
type: router
- id: parse_regex
type: regex_parser
parse_from: body
regex: '.*INFO[ ]+(?P<filename>\S+)[ ]+user-id= (?P<user_id>\S+)[ ]+trace-id=.*span-id.*line:(?P<line>\d+)[ ]'
parse_to: attributes
- id: noop
type: noop
Syed Muhammad Hassan
04/26/2023, 8:10 PM---
apiVersion: v1
kind: ConfigMap
metadata:
name: k8s-infra-otel-agent
namespace: platform
labels:
<http://app.kubernetes.io/component|app.kubernetes.io/component>: otel-agent
data:
otel-agent-config.yaml: |-
exporters:
otlp:
endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT}
headers:
signoz-access-token: Bearer ${SIGNOZ_API_KEY}
tls:
insecure: ${OTEL_EXPORTER_OTLP_INSECURE}
insecure_skip_verify: ${OTEL_EXPORTER_OTLP_INSECURE_SKIP_VERIFY}
extensions:
health_check:
endpoint: 0.0.0.0:13133
pprof:
endpoint: localhost:1777
zpages:
endpoint: localhost:55679
processors:
attributes:
actions:
- key: host
from_attribute: host
action: insert
# actions:
# - key: new_hosts
# pattern: '(?P<new_hosts>[^ ]+)'
# action: extract
batch:
send_batch_size: 10000
timeout: 200ms
k8sattributes:
extract:
metadata:
- k8s.namespace.name
- k8s.pod.name
- k8s.pod.uid
- k8s.pod.start_time
- k8s.deployment.name
- k8s.node.name
filter:
node_from_env_var: K8S_NODE_NAME
passthrough: false
pod_association:
- sources:
- from: resource_attribute
name: k8s.pod.ip
- sources:
- from: resource_attribute
name: k8s.pod.uid
- sources:
- from: connection
resourcedetection:
detectors:
- env
- system
override: true
system:
hostname_sources:
- dns
- os
timeout: 2s
receivers:
filelog:
include: [/var/log/pods/*_php-apache*_*/*/*.log]
exclude: [/var/log/pods/platform_*/*/*.log]
start_at: end
include_file_name: false
include_file_path: true
operators:
- type: regex_parser
#regex: '^(?P<time>[^ ]+) (?P<stream>stdout)\s+(?P<severity>[A-Z])\s+(?P<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s+-\s+-\s+\[(?P<datetime>\d{2}/[A-Za-z]{3}/\d{4}:\d{2}:\d{2}:\d{2}\s+\+\d{4})\]\s+"(?P<method>[A-Z]+)\s+(?P<path>[^ ]+)\s+(?P<protocol>HTTP\/\d\.\d)"\s+(?P<status>\d+)\s+(?P<size>\d+)\s+"(?P<referer>[^"]+)"\s+"(?P<user_agent>[^"]+)"\s+"(?P<extra>[^"]+)"$'
regex: '^(?P<time>[^ ]+) (?P<stream>stdout)\s+(?P<severity>[A-Z])\s+(?P<client_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s+-\s+-\s+\[(?P<datetime>\d{2}/[A-Za-z]{3}/\d{4}:\d{2}:\d{2}:\d{2}\s+\+\d{4})\]\s+"(?P<method>[A-Z]+)\s+(?P<path>[^ ]+)\s+(?P<protocol>HTTP\/\d\.\d)"\s+(?P<status>\d+)\s+(?P<size>\d+)\s+"(?P<referer>[^"]+)"\s+"(?P<user_agent>[^"]+)"'
output: extract_metadata_from_filepath_2 #move_host
timestamp:
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
parse_from: attributes.time
- from: attributes.client_ip
id: move_host
to: resource["my.hosts"]
type: move
- id: extract_metadata_from_filepath_2
parse_from: attributes["log.file.path"]
regex: ^.*\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]+)\/(?P<container_name>[^\._]+)\/(?P<restart_count>\d+)\.log$
type: regex_parser
- from: attributes.container_name
to: resource["k8s.container.name"]
type: move
- from: attributes.namespace
to: resource["k8s.namespace.name"]
type: move
- from: attributes.pod_name
to: resource["k8s.pod.name"]
type: move
- from: attributes.restart_count
to: resource["k8s.container.restart_count"]
type: move
- from: attributes.uid
to: resource["k8s.pod.uid"]
type: move
filelog/k8s:
exclude:
- /var/log/pods/kube-system_*/*/*.log
- /var/log/pods/platform_*/*/*.log
- /var/log/pods/kubecost_*/*/*.log
- /var/log/pods/*_hotrod*_*/*/*.log
- /var/log/pods/*_locust*_*/*/*.log
- /var/log/pods/*_nginx-test*_*/*/*.log
- /var/log/pods/*_php-apache*_*/*/*.log
include:
- /var/log/pods/*/*/*.log
include_file_name: false
include_file_path: true
operators:
- id: get-format
routes:
- expr: body matches "^\\{"
output: parser-docker
- expr: body matches "^[^ Z]+ "
output: parser-crio
- expr: body matches "^[^ Z]+Z"
output: parser-containerd
type: router
- id: parser-crio
output: extract_metadata_from_filepath
regex: ^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
timestamp:
layout: "2006-01-02T15:04:05.000000000-07:00"
layout_type: gotime
parse_from: attributes.time
type: regex_parser
- id: parser-containerd
output: extract_metadata_from_filepath
regex: ^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
timestamp:
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
parse_from: attributes.time
type: regex_parser
- id: parser-docker
output: extract_metadata_from_filepath
timestamp:
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
parse_from: attributes.time
type: json_parser
- id: extract_metadata_from_filepath
parse_from: attributes["log.file.path"]
regex: ^.*\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]+)\/(?P<container_name>[^\._]+)\/(?P<restart_count>\d+)\.log$
type: regex_parser
- from: attributes.stream
to: attributes["log.iostream"]
type: move
- from: attributes.container_name
to: resource["k8s.container.name"]
type: move
- from: attributes.namespace
to: resource["k8s.namespace.name"]
type: move
- from: attributes.pod_name
to: resource["k8s.pod.name"]
type: move
- from: attributes.restart_count
to: resource["k8s.container.restart_count"]
type: move
- from: attributes.uid
to: resource["k8s.pod.uid"]
type: move
- from: attributes.log
to: body
type: move
start_at: beginning
hostmetrics:
collection_interval: 30s
scrapers:
cpu: {}
disk: {}
filesystem: {}
load: {}
memory: {}
network: {}
kubeletstats:
auth_type: serviceAccount
collection_interval: 30s
endpoint: ${K8S_NODE_NAME}:10250
extra_metadata_labels:
- container.id
- k8s.volume.type
insecure_skip_verify: true
metric_groups:
- container
- pod
- node
- volume
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
max_recv_msg_size_mib: 4
http:
endpoint: 0.0.0.0:4318
service:
extensions:
- health_check
- zpages
pipelines:
logs:
exporters:
- otlp
processors:
- resourcedetection
- k8sattributes
- batch
receivers:
- otlp
- filelog
- filelog/k8s
metrics:
exporters:
- otlp
processors:
- resourcedetection
- k8sattributes
- batch
receivers:
- otlp
metrics/generic:
exporters:
- otlp
processors:
- resourcedetection
- k8sattributes
- batch
receivers:
- hostmetrics
- kubeletstats
traces:
exporters:
- otlp
processors:
- resourcedetection
- k8sattributes
- batch
receivers:
- otlp
telemetry:
metrics:
address: 0.0.0.0:8888