https://signoz.io logo
Join Slack
Powered by
This message was deleted.
# support
s
This message was deleted.
s
signoz-otel-collector config:
Copy code
exporters:
  clickhouselogsexporter:
    dsn: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_elapsed_time: 300s
      max_interval: 30s
    sending_queue:
      queue_size: 100
    timeout: 10s
  clickhousemetricswrite:
    endpoint: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}
    resource_to_telemetry_conversion:
      enabled: true
  clickhousetraces:
    datasource: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_TRACE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}
  prometheus:
    endpoint: 0.0.0.0:8889
extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  pprof:
    endpoint: localhost:1777
  zpages:
    endpoint: localhost:55679
processors:
  batch:
    send_batch_size: 1000
    timeout: 10s
  resourcedetection:
    detectors:
    - env
    - system
    timeout: 2s
  signozspanmetrics/prometheus:
    dimensions:
    - default: default
      name: service.namespace
    - default: default
      name: deployment.environment
    dimensions_cache_size: 10000
    latency_histogram_buckets:
    - 100us
    - 1ms
    - 2ms
    - 6ms
    - 10ms
    - 50ms
    - 100ms
    - 250ms
    - 500ms
    - 1000ms
    - 1400ms
    - 2000ms
    - 5s
    - 10s
    - 20s
    - 40s
    - 60s
    metrics_exporter: prometheus
receivers:
  fluentforward:
    endpoint: 0.0.0.0:24224
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu: {}
      disk: {}
      filesystem: {}
      load: {}
      memory: {}
      network: {}
  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
      thrift_http:
        endpoint: 0.0.0.0:14268
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  otlp/spanmetrics:
    protocols:
      grpc:
        endpoint: localhost:12345
  prometheus:
    config:
      global:
        scrape_interval: 30s
      scrape_configs:
      - job_name: otel-collector
        static_configs:
        - targets:
          - ${MY_POD_IP}:8888
service:
  extensions:
  - health_check
  - zpages
  pipelines:
    logs:
      exporters:
      - clickhouselogsexporter
      processors:
      - batch
      receivers:
      - otlp
      - fluentforward
    metrics:
      exporters:
      - clickhousemetricswrite
      processors:
      - batch
      receivers:
      - otlp
    metrics/generic:
      exporters:
      - clickhousemetricswrite
      processors:
      - resourcedetection
      - batch
      receivers:
      - hostmetrics
      - prometheus
    metrics/spanmetrics:
      exporters:
      - prometheus
      receivers:
      - otlp/spanmetrics
    traces:
      exporters:
      - clickhousetraces
      processors:
      - signozspanmetrics/prometheus
      - batch
      receivers:
      - jaeger
      - otlp
  telemetry:
    metrics:
      address: 0.0.0.0:8888
fluent-bit config:
Copy code
#[INPUT]
#    name cpu
#    tag  cpu.local
#
#    # Read interval (sec) Default: 1
#    interval_sec 1

[INPUT]
    Name        tail
    Path        <path-to-log-file>/file.log

[OUTPUT]
    Name          forward
    Match         *
    Host          signozoteldns
    Port          24224
    Shared_Key    secret
    Self_Hostname HOSTNAME

#[OUTPUT]
#    Name                 opentelemetry
#    Match                *
#    Host                 signozoteldns
#    Port                 4318
#    Metrics_uri          /v1/metrics
#    Logs_uri             /v1/logs
#    Log_response_payload True
#    add_label            app fluent-bit
#    add_label            color blue
Sample Log:
Copy code
{
  "event": "127.0.0.1 - - [03/Nov/2022:07:54:53 +0000] \"POST /v3/get_animation_presets_unauthenticated HTTP/1.0\" 200 458704 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36\"",
  "level": "info",
  "timestamp": "2022-11-03T07:54:53.133360Z",
  "pid": 3156,
  "thread": "ThreadPoolExecutor-1_3",
  "logger": "gunicorn.glogging",
  "file": "/home/runner/work/backend/backend/.venv/lib/python3.8/site-packages/gunicorn/glogging.py",
  "lineno": 344,
  "func": "access",
  "dd.trace_id": 0,
  "dd.span_id": 0,
  "dd.env": "stage",
  "dd.service": "servicename",
  "dd.version": ""
}
By using otlp Endpoint, we are getting following:
No logs are being received with fluent forwarder
p
cc @nitya-signoz
n
Hey Shivvinay, did you expose the port in otel collector for receiving logs. We missed this in our docs, I have updated it here https://github.com/SigNoz/signoz.io/pull/600/files . If you have already done the above can you check if you can see any error logs in fluentBit ?
For the second question I assume that you are sending logs from the other otel collector to signoz otel collector using otlp. But can you help us with how you are receiving the logs in the other otel collector which is forwarding logs? Since there are various operators and processors to parse your logs https://signoz.io/docs/userguide/logs/#operators-for-parsing-and-manipulating-logs . We can recommend on based on how you are receiving it.
s
In both cases we are sending via fluentbit, but i guess the first approach of sending from fluentbit to fluentforwarder is the correct one.
✅ 1
for the first scenario ports are open, did that in values.yaml as our setup is in k8s
n
No this is for the receiver that is running inside the otel-collector. But otel-collector will also have to expose the port so that the receiver can receive it. @Prashant Shahi can you help with where it needs to be configured.
✅ 2
s
Sorry shared a wrong snip, had tried this at 
.Values.otelCollector.ports
to expose the ports:
Copy code
ports:
    fluentforward:
      # -- Whether to enable service port for OTLP gRPC
      enabled: true
      # -- Container port for OTLP gRPC
      containerPort: 24224
      # -- Service port for OTLP gRPC
      servicePort: 24224
      # -- Node port for OTLP gRPC
      nodePort: ""
      # -- Protocol to use for OTLP gRPC
      protocol: TCP
p
@Shivvinay Kanswal The looks good, and if 
fluentforward
is included in 
.Values.otelCollector.config.receivers
In the shared image, it looks like you added it to 
otelAgent
instead.

https://signoz-community.slack.com/files/U048R5RLN6A/F049ABNSBNX/image.png▾

s
Had added for both Agent as well as collector:
This is the config from the container:
Copy code
❯ kubectl exec signoz-otel-collector-6459d64598-668wn -- cat conf/otel-collector-config.yaml
Defaulted container "signoz-otel-collector" out of: signoz-otel-collector, signoz-otel-collector-init (init)
exporters:
  clickhouselogsexporter:
    dsn: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_elapsed_time: 300s
      max_interval: 30s
    sending_queue:
      queue_size: 100
    timeout: 10s
  clickhousemetricswrite:
    endpoint: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}
    resource_to_telemetry_conversion:
      enabled: true
  clickhousetraces:
    datasource: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_TRACE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}
  prometheus:
    endpoint: 0.0.0.0:8889
extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  pprof:
    endpoint: localhost:1777
  zpages:
    endpoint: localhost:55679
processors:
  batch:
    send_batch_size: 1000
    timeout: 10s
  resourcedetection:
    detectors:
    - env
    - system
    timeout: 2s
  signozspanmetrics/prometheus:
    dimensions:
    - default: default
      name: service.namespace
    - default: default
      name: deployment.environment
    dimensions_cache_size: 10000
    latency_histogram_buckets:
    - 100us
    - 1ms
    - 2ms
    - 6ms
    - 10ms
    - 50ms
    - 100ms
    - 250ms
    - 500ms
    - 1000ms
    - 1400ms
    - 2000ms
    - 5s
    - 10s
    - 20s
    - 40s
    - 60s
    metrics_exporter: prometheus
receivers:
  fluentforward:
    endpoint: 0.0.0.0:24224
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu: {}
      disk: {}
      filesystem: {}
      load: {}
      memory: {}
      network: {}
  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
      thrift_http:
        endpoint: 0.0.0.0:14268
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  otlp/spanmetrics:
    protocols:
      grpc:
        endpoint: localhost:12345
  prometheus:
    config:
      global:
        scrape_interval: 30s
      scrape_configs:
      - job_name: otel-collector
        static_configs:
        - targets:
          - ${MY_POD_IP}:8888
service:
  extensions:
  - health_check
  - zpages
  pipelines:
    logs:
      exporters:
      - clickhouselogsexporter
      processors:
      - batch
      receivers:
      - otlp
      - fluentforward
    metrics:
      exporters:
      - clickhousemetricswrite
      processors:
      - batch
      receivers:
      - otlp
    metrics/generic:
      exporters:
      - clickhousemetricswrite
      processors:
      - resourcedetection
      - batch
      receivers:
      - hostmetrics
      - prometheus
    metrics/spanmetrics:
      exporters:
      - prometheus
      receivers:
      - otlp/spanmetrics
    traces:
      exporters:
      - clickhousetraces
      processors:
      - signozspanmetrics/prometheus
      - batch
      receivers:
      - jaeger
      - otlp
  telemetry:
    metrics:
      address: 0.0.0.0:8888
n
Interesting, I will have to try this out, but till then can you check if there are any error logs on the fluentbit side ?
s
nope fluentbit logs are clean
b
@nitya-signoz / @Prashant Shahi which is the recommended agent to send all the three things i.e traces, metrics & logs ? from external nodes
p
@Bunty Manchundiya external nodes as in independent VMs or K8s nodes?
s
vms
p
In case of VM, you can use OtelCollector binary
You can redirect any OTLP data: traces, metrics, logs.
For capturing logs generated in the VM itself, please refer to the logs configuration documentation
h
hey @Prashant Shahi we are using otel-col binary as an agent, however hostmetrics reciever on the otel-agent is not scraping anything
p
@hashfyre can you please share which otel-col binary version you are using? Also, share machine type and OS.
h
ubuntu 20.04
I'll check the version tomorrow
p
Okay. Also share the node type or arch?
a
@Shivvinay Kanswal Is the logs ingestion still not working?
s
We were able to push logs and host metrics to signoz using opentelemetry agent. Now we are trying to fine grain the inputs as per our requirement.
Will post on the channel in case we observe any challanges
n
Cool.
45 Views
Open in Slack
PreviousNext
Powered by