https://signoz.io logo
#support
Title
# support
s

SAMEEL .N

10/21/2022, 3:20 AM
How can I send nginx access and error logs to Signoz directly?
p

Pranay

10/21/2022, 3:48 AM
@nitya-signoz any insights on this?
n

nitya-signoz

10/21/2022, 4:16 AM
Hi @SAMEEL .N can you please share how you are running nginx. I.e directly on a machine /docker/kubernetes ?
s

SAMEEL .N

10/21/2022, 4:17 AM
directly on machine and logs are at /var/log/nginx/
n

nitya-signoz

10/21/2022, 4:19 AM
You can follow this guide https://signoz.io/docs/userguide/collect_logs_from_file/ . While it’s an example for python application, the process will be same for collecting nginx logs.
s

SAMEEL .N

10/21/2022, 4:20 AM
Thanks, will give a try , will Signoz give percentile graps [p95,p99] from nginx logs ?
n

nitya-signoz

10/21/2022, 4:24 AM
So you want to get the p95 and other latencies from your nginx logs itself right ?
s

SAMEEL .N

10/21/2022, 4:53 AM
yes
Will it be possible?
a

Ankit Nayan

10/21/2022, 7:34 AM
@SAMEEL .N should be possible using clickhouse query in dashboards..let us try that and come back
@SAMEEL .N can you share screenshot of you
interesting fields
and
selected fields
s

SAMEEL .N

10/21/2022, 7:45 AM
I was looking for percentile graph for each request(I have around 20 commonly used APIs) corresponding request time
a

Ankit Nayan

10/21/2022, 7:50 AM
that does not need to generate metrics from logs. APM metrics are already stored. You can use
Query Builder
in the dashboard section
Copy code
histogram_quantile(0.9, sum(rate(signoz_latency_bucket{service_name="frontend", operation="HTTP GET /dispatch"}[5m])) by (le))
0.9
=> p90
service_name
=> apply your service name
operation
=> event/operation/api name
you can customise the query Eg, remove service_name, add multiple operations or plot multiple such single queries for each endpoint
there is an option to add query as shown in the image
s

SAMEEL .N

10/24/2022, 6:04 AM
Thanks
May I know why I am getting below error
Copy code
/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.55.0/fileconsumer/file.go:113
2022-10-24T15:42:37.005Z	error	fileconsumer/file.go:197	Failed creating fingerprint	{"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "error": "reading fingerprint bytes: read /var/log/nginx/access.log: is a directory"}
I am trying get logs to Signoz using otel-collector-config.yaml My otel-collector-config.yaml file is
Copy code
receivers:
  filelog:
    include: [ /var/log/nginx/access.log ]
    start_at: beginning
    operators:
      - type: json_parser
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%d,%H:%M:%S %z'
      - type: move
        from: attributes.message
        to: body
      - type: remove
        field: attributes.time
processors:
  batch:
    send_batch_size: 10000
    send_batch_max_size: 153600
    timeout: 10s
exporters:
  otlp/log:
    endpoint: <http://XXXXXXX:4317>
    tls:
      insecure: true
service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [batch]
      exporters: [ otlp/log ]
@Ankit Nayan
a

Ankit Nayan

10/25/2022, 4:19 AM
@SAMEEL .N the error is displayed in logged output
"error": "reading fingerprint bytes: read /var/log/nginx/access.log: is a directory"
cc: @nitya-signoz
s

SAMEEL .N

10/25/2022, 4:43 AM
/var/log/nginx/access.log
ia not a directory , Its nginx log file @Ankit Nayan
@Ankit Nayan :the collecter give error while reading nginx logs do we need it in ay specific format?
Copy code
Failed to process entry	{"kind": "receiver", "name": "filelog", "pipeline": "logs", "operator_id": "json_parser", "operator_type": "json_parser", "error": "ReadMapCB: expect { or n, but found [, error found in #1 byte of ...|[24/Oct/202|..., bigger context ...|[24/Oct/2022:06:46:47 +0000] <http://XXX.XXX.XXX.XXX|XXX.XXX.XXX.XXX> - - 20|...", "action": "send", "entry": {"observed_timestamp":"2022-10-25T07:57:33.427082373Z","timestamp":"0001-01-01T00:00:00Z","body":"[24/Oct/2022:06:46:47 +0000] 162.158.162.219 - - 200 0.085 60 MISS [YYY] \"WWW.WWW.WWW.WWW\" \"GET /context1/context2/context3/context4/context5 HTTP/2.0\"","attributes":{"log.file.name":"access.log"},"severity":0,"scope_name":""}}
n

nitya-signoz

10/25/2022, 8:07 AM
@SAMEEL .N you are using json_parser, but the input to the JSON parser doesn’t seem to be in JSON format. You can either use regex parser in that case or you can output your logs in json format.
31 Views