Hello! I would be glad if someone could assist me. We have installed SigNoz Helm chart (0.12.1) and also k8s-infra chart as well. K8s-infra chart is with default values (Oracle OKE 1.23.4). Problem is that kubernetes pod logs are not visible in SigNoz. In otel agent logs I can see that it detects the file to watch in node:

2023-03-20T11:57:39.619Z    info    fileconsumer/file.go:171    Started watching file   {"kind": "receiver", "name": "filelog/k8s", "pipeline": "logs", "component": "fileconsumer", "path": "/var/log/pods/*********/configuration/0.log"}

I have verified that file contains all the logs for the container. There are no errors visible in agent logs, signoz-otel-collector logs or clickhouse logs, but the logs are not visible in SigNoz.

Team, I am trying to get Signoz standalone running using S3 bucket. Do you have detailed steps on how to do this? 1. I have edit the clickhouse-storage.xml and changed the endpoint, access_key_id, and secret_access_key. 2. I have edit the clickhouse-config.xml and change the database to s3 <!-- Default database. --> <default_database>s3</default_database> 3. Stopped and restart the containers but it’s not writing to s3. I know I am missing something.

Hi Team, I am facing an issue while running the sample Golang app following this document https://signoz.io/blog/distributed-tracing-golang/. While running serve -l 5000 frontend I get the following error : serve -l 5000 frontend file😕//usr/local/lib/node_modules/serve/build/main.js:169 const ipAddress = request.socket.remoteAddress?.replace("::ffff:", "") ?? "unknown"; ^ SyntaxError: Unexpected token '.' at Loader.moduleStrategy (internal/modules/esm/translators.js:133:18) at async link (internal/modules/esm/module_job.js:42:21) Can you please help ? Thanks

hi team, I need help on the following issue : serve -l 5000 frontend file😕//usr/local/lib/node_modules/serve/build/main.js:169 const ipAddress = request.socket.remoteAddress?.replace("::ffff:", "") ?? "unknown"; ^ SyntaxError: Unexpected token '.' at Loader.moduleStrategy (internal/modules/esm/translators.js:133:18) at async link (internal/modules/esm/module_job.js:42:21)

Thanks @Prashant Shahi...i updated Node.js version and the issue got resolved....Its not an issue in Signoz

Following this tutorial https://signoz.io/blog/distributed-tracing-golang/ i started all three microservices ...however in the signoz dashboard services are not appearing...I am running signoz as a docker container

On the “Services” page, how is the “Operations per Second” calculated? I see a number here which does not match what I see in the Rate chart when I click through to a specific service.

Hi All, We are working on the parsing of logs and we are facing multiple issues. Below is the one issue that we are facing when we are trying to parse the access logs. Sample log line

"10.21.18.240 - - [22/Mar/2023:08:19:04 +0000] \"GET /status HTTP/1.1\" 200 1249 \"-\" \"kube-probe/1.22+\"\n"

Below is the configuration of custom processor.

logstransform/keepattrs:
        operators:
          - type: remove
            id: remove-1
            field: attributes
      logstransform/parselog:
        operators:
          - type: router
            id: get-access
            routes:
              - output: parser-access
                expr: 'body matches "(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}"'
          - type: grok_parser
            id: parser-access
            parse_from: body
            pattern: '%{IP:ip}%{SPACE}-%{SPACE}-%{SPACE}\[%{HTTPDATE:timestamp}\]%{SPACE}\\"%{WORD:httpmethod}%{SPACE}%{DATA:request}\"%{SPACE}%{NUMBER:status}'
            parse_to: attributes

Grok debugger output. I am getting below error.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x24701e1]

goroutine 1 [running]:
<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/adapter.(*receiver).Shutdown(0xc00107ccf0|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/adapter.(*receiver).Shutdown(0xc00107ccf0>, {0x52eba28, 0xc000076028})
	/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.66.0/adapter/receiver.go:148 +0x81
<http://go.opentelemetry.io/collector/service/internal/pipelines.(*Pipelines).ShutdownAll(0xc000f781e0|go.opentelemetry.io/collector/service/internal/pipelines.(*Pipelines).ShutdownAll(0xc000f781e0>, {0x52eba28, 0xc000076028})
	/go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/internal/pipelines/pipelines.go:121 +0x499
<http://go.opentelemetry.io/collector/service.(*service).Shutdown(0xc000f5a800|go.opentelemetry.io/collector/service.(*service).Shutdown(0xc000f5a800>, {0x52eba28, 0xc000076028})
	/go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/service.go:121 +0xd4
<http://go.opentelemetry.io/collector/service.(*Collector).shutdownServiceAndTelemetry(0xc001717a88|go.opentelemetry.io/collector/service.(*Collector).shutdownServiceAndTelemetry(0xc001717a88>, {0x52eba28?, 0xc000076028?})
	/go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/collector.go:264 +0x36
<http://go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents(0xc001717a88|go.opentelemetry.io/collector/service.(*Collector).setupConfigurationComponents(0xc001717a88>, {0x52eba28, 0xc000076028})
	/go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/collector.go:166 +0x27d
<http://go.opentelemetry.io/collector/service.(*Collector).Run(0xc001717a88|go.opentelemetry.io/collector/service.(*Collector).Run(0xc001717a88>, {0x52eba28, 0xc000076028})
	/go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/collector.go:190 +0x46
<http://go.opentelemetry.io/collector/service.NewCommand.func1(0xc0005cac00|go.opentelemetry.io/collector/service.NewCommand.func1(0xc0005cac00>, {0x490a0cd?, 0x1?, 0x1?})
	/go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/service/command.go:53 +0x479
<http://github.com/spf13/cobra.(*Command).execute(0xc0005cac00|github.com/spf13/cobra.(*Command).execute(0xc0005cac00>, {0xc00006e070, 0x1, 0x1})
	/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:916 +0x862
<http://github.com/spf13/cobra.(*Command).ExecuteC(0xc0005cac00)|github.com/spf13/cobra.(*Command).ExecuteC(0xc0005cac00)>
	/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044 +0x3bc
<http://github.com/spf13/cobra.(*Command).Execute(...)|github.com/spf13/cobra.(*Command).Execute(...)>
	/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968
main.runInteractive({{0xc0006bdb90, 0xc00082aea0, 0xc00082a120, 0xc0006bd7d0}, {{0x4943e48, 0x15}, {0x49417a5, 0x15}, {0x49065d8, 0x6}}, ...})
	/src/cmd/signozcollector/main.go:37 +0x5e
main.run(...)
	/src/cmd/signozcollector/main_others.go:8
main.main()
	/src/cmd/signozcollector/main.go:30 +0x1d8

Hello all, We have a requirement to monitor the thread dump. Anyone with experience on how to do this with SigNoz? @Prashant Shahi, @nitya-signoz, @Ankit Nayan

I hope you are doing well. I have one requirement to monitor my react application and for that I am following for blog (https://signoz.io/blog/opentelemetry-react/) and video (). I have done the same thing for my react application and not able to get this application on signoz dashboard. otel-collector-config.yaml file

receivers:
  filelog/dockercontainers:
    include: [  "/var/lib/docker/containers/*/*.log" ]
    start_at: end
    include_file_path: true
    include_file_name: false
    operators:
    - type: json_parser
      id: parser-docker
      output: extract_metadata_from_filepath
      timestamp:
        parse_from: attributes.time
        layout: '%Y-%m-%dT%H:%M:%S.%LZ'
    - type: regex_parser
      id: extract_metadata_from_filepath
      regex: '^.*containers/(?P<container_id>[^_]+)/.*log$'
      parse_from: attributes["log.file.path"]
      output: parse_body
    - type: move
      id: parse_body
      from: attributes.log
      to: body
      output: time
    - type: remove
      id: time
      field: attributes.time
  opencensus:
    endpoint: 0.0.0.0:55678
  otlp/spanmetrics:
    protocols:
      grpc:
        endpoint: localhost:12345
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        cors:
          allowed_origins:
            - <http://65.0.130.56:3000>
  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
      thrift_http:
        endpoint: 0.0.0.0:14268
      # thrift_compact:
      #   endpoint: 0.0.0.0:6831
      # thrift_binary:
      #   endpoint: 0.0.0.0:6832

Please help me figure out the error

@Prashant Shahi @Ankit Nayan If we start passing TraceID in our logs. Will signoz/otelcollector automatically parse it?

Hi

While connecting to grpc using nginx we are getting the below error. Please have a check. [error] 30#30: *1 upstream rejected request with error 1 while reading response header from upstream, client: , server: , request: "GET /favicon.ico HTTP/1.1", u

Hi, I have an app that's pushing metrics in prometheus format on

localhost:3334/metrics

. I do not want to run a prometheus instance to consume these. What configuration should I add to signoz to consume the metrics from my app's endpoint? Also I do not see a "Metrics" menu item in the Signoz dashboard. Does this need some sort of config to enable?

Hi

issue in accessing grpc port via azure application gateway if signoz deployed in azure.

Hi

@Prashant Shahi @nitya-signoz Is there any relationship between kubernets container metrics and logging? As I stopped pushing logs few metrics of kubernets container like k8s_container_cpu_limit also stopped?

I'm a bit confused about the logs view. When I choose "Last 1 hour", "Last 5 minutes" in the logs view, what does that actually give me? On my installation, choosing either one gives me 3 day old logs. What does it actually filter on?

It's not even consistent. One moment, I get a bunch of 3 day old logs. Then I click refresh, with the SAME settings, and I get no logs at all!

I'm trying to move logs to s3 that are older than 1 hour, I have set the retention policy as such. I've configured clickhouse to have s3 storage, and clickhouse is not showing any errors. However, my logs are not appearing in s3. Where should I look for logs to begin debugging this?

Hi, When I try to create a dashboard and add a time-series panel, i choose clickhouse query option and when i write a query i have to hard code the time range. Tried to use $timeFilter like we do in grafana but i guess it will not work. Don't we have support to provide time interval in query and it will change as user changes the interval from the top-right?

Hi, Can anyone tell what is the reason of getting missing spans? I don't find any detail in the link too https://signoz.io/docs/userguide/traces/#missing-spans

Hi, can anyone help here? I am trying to do manual instrumenting as I find auto-instrument pumps in too much data. I am using Python, and I have created a HTTP OTLPSpanExporter, with the endpoint pointing to the clickhouse url, port=4318. But it always return an error "Failed to export batch code: 404". Does anyone know how to fix this?

hii, Can anyone guide me how to edit this clickhouse-server/config.xml file. to configure the max_suspicious_broken_parts

2023.03.27 09:41:32.756403 [ 7 ] {} <Information> Application: Shutting down storages.
2023.03.27 09:41:32.756425 [ 7 ] {} <Information> Context: Shutdown disk default
2023.03.27 09:41:32.756432 [ 7 ] {} <Information> Context: Shutdown disk s3
2023.03.27 09:41:32.756448 [ 7 ] {} <Information> DiskObjectStorage(DiskS3): Shutting down disk s3
2023.03.27 09:41:32.756455 [ 7 ] {} <Information> DiskObjectStorage(DiskS3): Disk s3 shut down
2023.03.27 09:41:33.197310 [ 76 ] {} <Information> system.metric_log (ee60f679-08f7-43a6-b64a-b1c5b6033ba0): Delaying inserting block by 131.8256738556407 ms. because there are 255 parts
2023.03.27 09:41:33.816440 [ 7 ] {} <Error> Application: DB::Exception: Suspiciously many (20 parts, 259.02 KiB in total) broken parts to remove while maximum allowed broken parts count is 10. You can change the maximum value with merge tree setting 'max_suspicious_broken_parts' in <merge_tree> configuration section or in table settings in .sql file (don't forget to return setting back to default value): Cannot attach table `signoz_logs`.`logs` from metadata file /var/lib/clickhouse/store/5f9/5f9d716d-9913-47d3-90b0-c95f1c62bfa5/logs.sql from query ATTACH TABLE signoz_logs.logs UUID 'dbc363d8-5e0b-40c5-9df5-9e6260cd082f' (`timestamp` UInt64 CODEC(DoubleDelta, LZ4), `observed_timestamp` UInt64 CODEC(DoubleDelta, LZ4), `id` String CODEC(ZSTD(1)), `trace_id` String CODEC(ZSTD(1)), `span_id` String CODEC(ZSTD(1)), `trace_flags` UInt32, `severity_text` LowCardinality(String) CODEC(ZSTD(1)), `severity_number` UInt8, `body` String CODEC(ZSTD(2)), `resources_string_key` Array(String) CODEC(ZSTD(1)), `resources_string_value` Array(String) CODEC(ZSTD(1)), `attributes_string_key` Array(String) CODEC(ZSTD(1)), `attributes_string_value` Array(String) CODEC(ZSTD(1)), `attributes_int64_key` Array(String) CODEC(ZSTD(1)), `attributes_int64_value` Array(Int64) CODEC(ZSTD(1)), `attributes_float64_key` Array(String) CODEC(ZSTD(1)), `attributes_float64_value` Array(Float64) CODEC(ZSTD(1)), `k8s_cluster_name` String MATERIALIZED resources_string_value[indexOf(resources_string_key, 'k8s_cluster_name')] CODEC(LZ4), INDEX body_idx body TYPE tokenbf_v1(10240, 3, 0) GRANULARITY 4, INDEX id_minmax id TYPE minmax GRANULARITY 1, INDEX k8s_cluster_name_idx k8s_cluster_name TYPE bloom_filter(0.01) GRANULARITY 64) ENGINE = MergeTree PARTITION BY toDate(timestamp / 1000000000) ORDER BY (timestamp, id) TTL toDateTime(timestamp / 1000000000) + toIntervalSecond(345600), toDateTime(timestamp / 1000000000) + toIntervalSecond(172800) TO VOLUME 's3' SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1, storage_policy = 'tiered'
2023.03.27 09:41:33.816490 [ 7 ] {} <Information> Application: shutting down
2023.03.27 09:41:33.816548 [ 8 ] {} <Information> BaseDaemon: Stop SignalListener thread
2023.03.27 09:41:33.843565 [ 1 ] {} <Information> Application: Child process exited normally with code 70.

hey SigNoz! We're using signoz in k8s and find that after a few hours we stop collecting logs. Restarting the otel-collector allows us to start collecting logs again. Any idea where to look for what's causing it to crash?

Hi everyone, signoz-otel-collector exporters are failing with Requested cluster 'cluster' not found" component=clickhouse 2023-03-27T11:52:23.613Z info service/service.go:106 Everything is ready. Begin running and processing data. <snip> time="2023-03-27T11:52:30Z" level=error msg="code: 170, message: Requested cluster 'cluster' not found" component=clickhouse 2023-03-27T11:52:31.621Z info exporterhelper/queued_retry.go:426 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "logs", "name": "clickhouselogsexporter", "error": "PrepareBatch:code: 170, message: Requested cluster 'cluster' not found", "interval": "4.0135666s"} <snip> 2023-03-27T11:52:53.793Z info exporterhelper/queued_retry.go:426 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 170, message: Requested cluster 'cluster' not found", "interval": "140.552788ms"} <snip> 2023-03-27T18:20:57.623Z error exporterhelper/queued_retry.go:175 Exporting failed. No more retries left. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "max elapsed time expired code: 170, message: Requested cluster 'cluster' not found", "dropped_items": 1184} go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).onTemporaryFailure /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/exporter/exporterhelper/queued_retry.go:175 go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/exporter/exporterhelper/queued_retry.go:410 go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/exporter/exporterhelper/metrics.go:135 go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/exporter/exporterhelper/queued_retry.go:205 go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1 /go/pkg/mod/go.opentelemetry.io/collector@v0.66.0/exporter/exporterhelper/internal/bounded_memory_queue.go:61 2023-03-27T18:20:58.626Z info exporterhelper/queued_retry.go:426 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 170, message: Requested cluster 'cluster' not found", "interval": "216.619898ms"} time="2023-03-27T18:21:00Z" level=error msg="code: 170, message: Requested cluster 'cluster' not found" component=clickhouse Recently re-installed using helm, and also added a second node but removed it after running into listener conflicts, possibly when not on the same node.

Hi Team/@Srikanth Chekuri, I see we have only 2 panels available (time-series and value) in dashboard. Do we expect to come more like, histogram, graphs, guage etc.? Also, there is an option to import json from grafana. If my existing grafana dashboard is having bar charts and other panel types, will they appear or just skipped by Signoz?

HI Team, I am facing an issue with traces on Signoz, in the traces for this service I can see it logged only 3 calls. but looking at the cloudwatch I can check we have 6 calls, as attached in the screenshot.