How can I send nginx access and error logs to Signoz directl SigNoz Community #support

Join Slack

How can I send nginx access and error logs to Sign...

# support

SAMEEL .N

10/21/2022, 3:20 AM

How can I send nginx access and error logs to Signoz directly?

Pranay

10/21/2022, 3:48 AM

@nitya-signoz any insights on this?

nitya-signoz

10/21/2022, 4:16 AM

Hi @SAMEEL .N can you please share how you are running nginx. I.e directly on a machine /docker/kubernetes ?

SAMEEL .N

10/21/2022, 4:17 AM

directly on machine and logs are at /var/log/nginx/

nitya-signoz

10/21/2022, 4:19 AM

You can follow this guide https://signoz.io/docs/userguide/collect_logs_from_file/ . While it’s an example for python application, the process will be same for collecting nginx logs.

SAMEEL .N

10/21/2022, 4:20 AM

Thanks, will give a try , will Signoz give percentile graps [p95,p99] from nginx logs ?

nitya-signoz

10/21/2022, 4:24 AM

So you want to get the p95 and other latencies from your nginx logs itself right ?

SAMEEL .N

10/21/2022, 4:53 AM

yes

SAMEEL .N

10/21/2022, 6:34 AM

Will it be possible?

Ankit Nayan

10/21/2022, 7:34 AM

@SAMEEL .N should be possible using clickhouse query in dashboards..let us try that and come back

Ankit Nayan

10/21/2022, 7:36 AM

@SAMEEL .N can you share screenshot of you

interesting fields

and

selected fields

SAMEEL .N

10/21/2022, 7:45 AM

I was looking for percentile graph for each request(I have around 20 commonly used APIs) corresponding request time

Ankit Nayan

10/21/2022, 7:50 AM

that does not need to generate metrics from logs. APM metrics are already stored. You can use

Query Builder

in the dashboard section

Ankit Nayan

10/21/2022, 7:52 AM

Copy code

histogram_quantile(0.9, sum(rate(signoz_latency_bucket{service_name="frontend", operation="HTTP GET /dispatch"}[5m])) by (le))

Ankit Nayan

10/21/2022, 7:53 AM

0.9

=> p90

service_name

=> apply your service name

operation

=> event/operation/api name

Ankit Nayan

10/21/2022, 7:54 AM

you can customise the query Eg, remove service_name, add multiple operations or plot multiple such single queries for each endpoint

Ankit Nayan

10/21/2022, 7:56 AM

there is an option to add query as shown in the image

SAMEEL .N

10/24/2022, 6:04 AM

Thanks

SAMEEL .N

10/24/2022, 3:44 PM

May I know why I am getting below error

Copy code

/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.55.0/fileconsumer/file.go:113
2022-10-24T15:42:37.005Z	error	fileconsumer/file.go:197	Failed creating fingerprint	{"kind": "receiver", "name": "filelog", "pipeline": "logs", "component": "fileconsumer", "error": "reading fingerprint bytes: read /var/log/nginx/access.log: is a directory"}

I am trying get logs to Signoz using otel-collector-config.yaml My otel-collector-config.yaml file is

Copy code

receivers:
  filelog:
    include: [ /var/log/nginx/access.log ]
    start_at: beginning
    operators:
      - type: json_parser
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%d,%H:%M:%S %z'
      - type: move
        from: attributes.message
        to: body
      - type: remove
        field: attributes.time
processors:
  batch:
    send_batch_size: 10000
    send_batch_max_size: 153600
    timeout: 10s
exporters:
  otlp/log:
    endpoint: <http://XXXXXXX:4317>
    tls:
      insecure: true
service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [batch]
      exporters: [ otlp/log ]

SAMEEL .N

10/24/2022, 6:23 PM

@Ankit Nayan

Ankit Nayan

10/25/2022, 4:19 AM

@SAMEEL .N the error is displayed in logged output

"error": "reading fingerprint bytes: read /var/log/nginx/access.log: is a directory"

cc: @nitya-signoz

SAMEEL .N

10/25/2022, 4:43 AM

/var/log/nginx/access.log

ia not a directory , Its nginx log file @Ankit Nayan

SAMEEL .N

10/25/2022, 8:05 AM

@Ankit Nayan :the collecter give error while reading nginx logs do we need it in ay specific format?

Copy code

Failed to process entry	{"kind": "receiver", "name": "filelog", "pipeline": "logs", "operator_id": "json_parser", "operator_type": "json_parser", "error": "ReadMapCB: expect { or n, but found [, error found in #1 byte of ...|[24/Oct/202|..., bigger context ...|[24/Oct/2022:06:46:47 +0000] <http://XXX.XXX.XXX.XXX|XXX.XXX.XXX.XXX> - - 20|...", "action": "send", "entry": {"observed_timestamp":"2022-10-25T07:57:33.427082373Z","timestamp":"0001-01-01T00:00:00Z","body":"[24/Oct/2022:06:46:47 +0000] 162.158.162.219 - - 200 0.085 60 MISS [YYY] \"WWW.WWW.WWW.WWW\" \"GET /context1/context2/context3/context4/context5 HTTP/2.0\"","attributes":{"log.file.name":"access.log"},"severity":0,"scope_name":""}}

nitya-signoz

10/25/2022, 8:07 AM

@SAMEEL .N you are using json_parser, but the input to the JSON parser doesn’t seem to be in JSON format. You can either use regex parser in that case or you can output your logs in json format.

93 Views

Open in Slack

Previous Next