Jason Loong
10/22/2022, 3:01 AM{"level":"info","message":"PassportModule dependencies initialized","service":"logger-service","timestamp":"2022-10-21T10:51:31.050Z"}
Following this doc: https://signoz.io/docs/userguide/collect_logs_from_file/
Docker run command:
docker run -d --name signoz-otel-collector --user root -v $(pwd)/backend.log:/tmp/backend.log:ro -v $(pwd)/signoz-winston-collector.yaml:/etc/otel/config.yaml signoz/signoz-otel-collector:0.55.3
signoz-winston-collector.yaml
receivers:
filelog:
include: [/tmp/backend.log]
start_at: beginning
operators:
- type: json_parser
id: nest-winston-json
timestamp:
parse_from: attributes.timestamp
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
- type: move
from: attributes.message
to: body
- type: remove
field: attributes.timestamp
processors:
batch:
send_batch_size: 10 #low num for testing
send_batch_max_size: 10 #low num for testing
timeout: 10s
exporters:
otlp/log:
endpoint: <http://192.168.200.20:4317>
tls:
insecure: true
service:
telemetry:
logs:
level: 'debug'
pipelines:
logs:
receivers: [filelog]
processors: [batch]
exporters: [otlp/log]
Docker collector container logs
2022-10-22T02:31:52.529Z info service/telemetry.go:103 Setting up own telemetry...
2022-10-22T02:31:52.530Z info service/telemetry.go:138 Serving Prometheus metrics {"address": ":8888", "level": "basic"}
2022-10-22T02:31:52.530Z debug pipelines/pipelines.go:343 Stability level {"kind": "exporter", "data_type": "logs", "name": "otlp/log", "stability": "beta"}
2022-10-22T02:31:52.530Z debug pipelines/pipelines.go:343 Stability level {"kind": "processor", "name": "batch", "pipeline": "logs", "stability": "stable"}
2022-10-22T02:31:52.530Z debug pipelines/pipelines.go:345 Stability level of component undefined {"kind": "receiver", "name": "filelog", "pipeline": "logs", "stability": "undefined"}
2022-10-22T02:31:52.532Z info extensions/extensions.go:42 Starting extensions...
2022-10-22T02:31:52.532Z info pipelines/pipelines.go:74 Starting exporters...
2022-10-22T02:31:52.532Z info pipelines/pipelines.go:78 Exporter is starting... {"kind": "exporter", "data_type": "logs", "name": "otlp/log"}
2022-10-22T02:31:52.532Z info zapgrpc/zapgrpc.go:174 [core] [Channel #1] Channel created {"grpc_log": true}
2022-10-22T02:31:52.532Z info zapgrpc/zapgrpc.go:174 [core] [Channel #1] original dial target is: "192.168.200.20:4317" {"grpc_log": true}
2022-10-22T02:31:52.532Z info zapgrpc/zapgrpc.go:174 [core] [Channel #1] dial target "192.168.200.20:4317" parse failed: parse "192.168.200.20:4317": first path segment in URL cannot contain colon {"grpc_log": true}
2022-10-22T02:31:52.532Z info zapgrpc/zapgrpc.go:174 [core] [Channel #1] fallback to scheme "passthrough" {"grpc_log": true}
2022-10-22T02:31:52.532Z info zapgrpc/zapgrpc.go:174 [core] [Channel #1] parsed dial target is: {Scheme:passthrough Authority: Endpoint:192.168.200.20:4317 URL:{Scheme:passthrough Opaque: User: Host: Path:/192.168.200.20:4317 RawPath: ForceQuery:false RawQuery: Fragment: RawFragment:}} {"grpc_log": true}
2022-10-22T02:31:52.532Z info zapgrpc/zapgrpc.go:174 [core] [Channel #1] Channel authority set to "192.168.200.20:4317" {"grpc_log": true}
2022-10-22T02:31:52.532Z info zapgrpc/zapgrpc.go:174 [core] [Channel #1] Resolver state updated: {
"Addresses": [
{
"Addr": "192.168.200.20:4317",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}
],
"ServiceConfig": null,
"Attributes": null
} (resolver returned new addresses) {"grpc_log": true}
2022-10-22T02:31:52.533Z info zapgrpc/zapgrpc.go:174 [core] [Channel #1] Channel switches to new LB policy "pick_first" {"grpc_log": true}
2022-10-22T02:31:52.533Z info zapgrpc/zapgrpc.go:174 [core] [Channel #1 SubChannel #2] Subchannel created {"grpc_log": true}
2022-10-22T02:31:52.533Z info zapgrpc/zapgrpc.go:174 [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING{"grpc_log": true}
2022-10-22T02:31:52.533Z info zapgrpc/zapgrpc.go:174 [core] [Channel #1 SubChannel #2] Subchannel picks a new address "192.168.200.20:4317" to connect {"grpc_log": true}
2022-10-22T02:31:52.533Z info pipelines/pipelines.go:82 Exporter started. {"kind": "exporter", "data_type": "logs", "name": "otlp/log"}
2022-10-22T02:31:52.533Z info pipelines/pipelines.go:86 Starting processors...
2022-10-22T02:31:52.533Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "batch", "pipeline": "logs"}
2022-10-22T02:31:52.534Z info zapgrpc/zapgrpc.go:174 [core] pickfirstBalancer: UpdateSubConnState: 0xc0008d10d0, {CONNECTING <nil>}{"grpc_log": true}
2022-10-22T02:31:52.534Z info zapgrpc/zapgrpc.go:174 [core] [Channel #1] Channel Connectivity change to CONNECTING {"grpc_log": true}
2022-10-22T02:31:52.534Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "batch", "pipeline": "logs"}
2022-10-22T02:31:52.534Z info pipelines/pipelines.go:98 Starting receivers...
2022-10-22T02:31:52.534Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-10-22T02:31:52.534Z info adapter/receiver.go:54 Starting stanza receiver {"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-10-22T02:31:52.534Z debug pipeline/directed.go:70 Starting operator {"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-10-22T02:31:52.534Z debug pipeline/directed.go:74 Started operator {"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-10-22T02:31:52.534Z debug pipeline/directed.go:70 Starting operator {"kind": "receiver", "name": "filelog", "pipeline": "logs", "operator_id": "remove", "operator_type": "remove"}
2022-10-22T02:31:52.534Z debug pipeline/directed.go:74 Started operator {"kind": "receiver", "name": "filelog", "pipeline": "logs", "operator_id": "remove", "operator_type": "remove"}
2022-10-22T02:31:52.534Z debug adapter/converter.go:143 Starting log converter {"kind": "receiver", "name": "filelog", "pipeline": "logs", "worker_count": 1}
2022-10-22T02:31:52.534Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "filelog", "pipeline": "logs"}
2022-10-22T02:31:52.535Z info service/collector.go:215 Starting signoz-otel-collector... {"Version": "latest", "NumCPU": 6}
2022-10-22T02:31:52.535Z info service/collector.go:128 Everything is ready. Begin running and processing data.
2022-10-22T02:31:52.738Z info fileconsumer/file.go:178 Started watching file {"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "path": "/tmp/backend.log"}
2022-10-22T02:31:52.744Z info zapgrpc/zapgrpc.go:174 [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to READY {"grpc_log": true}
2022-10-22T02:31:52.744Z info zapgrpc/zapgrpc.go:174 [core] pickfirstBalancer: UpdateSubConnState: 0xc0008d10d0, {READY <nil>} {"grpc_log": true}
2022-10-22T02:31:52.744Z info zapgrpc/zapgrpc.go:174 [core] [Channel #1] Channel Connectivity change to READY {"grpc_log": true}
Tried the follow troubleshoot:
docker run -it --rm signoz/troubleshoot checkEndpoint --endpoint=192.168.200.20:4317
2022-10-22T01:40:57.763Z INFO troubleshoot/main.go:28 STARTING!
2022-10-22T01:40:57.764Z INFO checkEndpoint/checkEndpoint.go:41 checking reachability of SigNoz endpoint
2022-10-22T01:40:57.776Z INFO troubleshoot/main.go:46 Successfully sent sample data to signoz ...
Pranay
10/22/2022, 4:09 AMJason Loong
10/23/2022, 2:11 AMoperators:
- type: json_parser
parse_to: body
- type: time_parser
parse_from: body.timestamp
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
# - type: json_parser
# timestamp:
# parse_from: attributes.timestamp
# layout: '%Y-%m-%dT%H:%M:%S.%LZ'
# - type: move
# from: attributes.message
# to: body
- type: remove
field: body.timestamp
Pranay
10/23/2022, 2:29 AMNityananda Gohain
10/23/2022, 9:36 AM