signoz-community #support

@Srikanth Chekuri: Is there a way I can see the data which is received to SigNoz not on the UI but at the backend? I just want to see how to data format looks like. I have configured my Telegraf such that it spit out metrics in open telemetry format to otel collector.

Henrik

10/27/2022, 12:26 PM

It appears that SigNoz automatically reads and imports a lof of stuff from /var/log, which includes the output of all containers on the node. It's a fine feature, but given the amount of noise from many of those, I would like to turn that auto-import off or at least control which files it imports. Is there any way to do this?

Abhinav Ramana

10/27/2022, 6:33 PM

@Prashant Shahi We have asyncio loop stuff for graphdb interactions in our main backend, which is why we were using

Copy code

CMD ["uvicorn", "--workers", "2", "--host", "0.0.0.0", "--port", "8000", "--ws", "websockets", "--loop", "asyncio", "--ws-max-size", "100000", "--ws-ping-interval", "20.00", "--ws-ping-timeout", "20.00", "wombo.fastapi:app"]

However, Uvicorn can't be used because https://github.com/open-telemetry/opentelemetry-python-contrib/issues/385#issuecomment-1199088668, so as per your suggestion we migrated to

Copy code

CMD opentelemetry-instrument --traces_exporter otlp_proto_grpc gunicorn wombo.fastapi:app --workers 2 --worker-class  uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000

But this forced us to disable

Copy code

nest_asyncio.apply()

Which inturn breaks a lot of our operations. Do you have any suggestions? I tried into seeing if i could use other worker classes like

gevent

eventlet

for gunicorn https://docs.gunicorn.org/en/latest/design.html#async-workers However they are giving monkey_patch errors which I can't initialize before

Abhinav Ramana

10/27/2022, 6:38 PM

@Prashant Shahi I am unable to see logs , though I can see services and traces successfully

Abhinav Ramana

10/27/2022, 6:49 PM

For celery workers I am getting

Copy code

[2022-10-27 18:47:48,590: ERROR/MainProcess] Callback failed for instrument system.swap.utilization.
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/opentelemetry/sdk/metrics/_internal/instrument.py", line 132, in callback
    for api_measurement in callback(callback_options):
  File "/opt/conda/lib/python3.8/site-packages/opentelemetry/instrumentation/system_metrics/__init__.py", line 423, in _get_system_swap_utilization
    getattr(system_swap, metric) / system_swap.total,
ZeroDivisionError: division by zero

probably every time its sending metric

Abhinav Ramana

10/28/2022, 12:23 AM

I tried to send metrics as per https://open-telemetry.github.io/opentelemetry-python/getting-started.html in celery workers

Copy code

from opentelemetry import metrics

meter = metrics.get_meter("worker_vqgan")

requests_counter = meter.create_counter(
    name="requests",
    description="number of requests",
    unit="1"
)

But i dont see anything in signoz as per the screenshot Also there were no errors

Copy code

[2022-10-28 00:16:21,847: INFO/MainProcess] Connected to <sqs://AKIAWGXQXQ6WCOB7PP5J>:**@sqs.us-east-1.amazonaws.com/426780362668/dev-wombo-paint-standard

in the logs

Henrik

10/28/2022, 7:52 AM

I have a few UI-related questions 1) Is it possible in the Traces view, to only show the "top level" traces, and not the sub-spans by default? For example for a GET HTTP request, it generates perhaps 20 spans on the traces list, but I would like to only see one line for that request in the overview list 2) When viewing Trace Details, the tags seems to be randomly ordered. If I have 5 subspans (SQL calls) in a row, and click them all one by one, the tags on the right seem to jump up and down for all 5, making it hard to compare them. Is there anything do do about that? Perhaps a feature request otherwise? 🙂

Quy Bui

10/28/2022, 8:26 AM

Hi team, does SigNoz support Sampling?

Copy code

// sampling definition
Sampling is frequently used to reduce the number of traces that are collected and stored in the backend. This is often desirable because it is easy to produce more data than can be efficiently stored and queried. Sampling allows us to store only a subset of the total traces produced.

Wejdan

10/30/2022, 9:39 AM

i tried to connect the logs in kubernetes but i faced this error no files match the configured include patterns {"kind": "receiver", "name": "filelog/k8s", "pipeline": "logs", "component": "fileconsumer", "include": ["/var/log/pods/*/*/*.log"], "exclude": []} could anyone help me?

Henrik

10/31/2022, 11:00 AM

I am getting errors in the otel-collector, when I enable a specific service to send to it using OTLP. My three other similar services work fine, but once I enable this one, I am getting these errors from the pod, and nothing gets stored. 2022-10-31T105610.924Z error clickhousetracesexporter/writer.go:101 Could not write a batch of spans {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "error": "clickhouse [Encode]: mismatched len of columns"} github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter /src/exporter/clickhousetracesexporter/writer.go:101 Any idea how I can hunt down the cause for this? There seem to be no further info logged from the pod than this

Alexandre Campelo

10/31/2022, 4:59 PM

I just installed with the docker in local machine, I removed the HOTRD and now I'm trying to create the admin account. Using the Web interface leads to an error "Invite Token required"

Shivvinay Kanswal

11/01/2022, 11:39 AM

Hi everyone, can I disable the default authentication on signoz in favour of an idp proxy such as pomerium.

Alex Popa

11/01/2022, 12:05 PM

Hi everyone, I was doing a test install on gcloud k8s , and afterwards i tried upgrading helm installation setup using the command exposed here: https://signoz.io/docs/instrumentation/troubleshoot-instrumentation/ for k8s where application is in a different cluster; problem is with steps from docs, the setup doesn't work

Paulo Henrique de Morais Santiago

11/01/2022, 1:09 PM

Hello Everyone, i`m trying to setting up signoz with node app express that have a redis. Using this package. https://www.npmjs.com/package/@opentelemetry/instrumentation-redis-4 But dont are working. We have a doc how to configure? The official doc have the package we need to use but not how configure. https://signoz.io/docs/instrumentation/express/#redis-instrumentation After the correct configuration the page database calls (Database Calls RPS, Database Calls Avg Duration) will work?

Marius Prodana

11/01/2022, 2:49 PM

Hi, do you have any idea how I could break the service map to be based on the environment or something else?..to have more trees based on parameter

Allan Li

11/01/2022, 5:20 PM

Hi, I'm building a timeseries panel in the signoz dashboard, how do I access the time range macro? Thanks

George

11/01/2022, 6:50 PM

Hello I am trying to integrate signoz with Magento 2 any help is appreciated

George

11/01/2022, 7:54 PM

any idea why i am getting this error

George

11/01/2022, 7:54 PM

Copy code

Fatal error: Uncaught ArgumentCountError: Too few arguments to function OpenTelemetry\SDK\Common\Attribute\Attributes::__construct(), 1 passed in /var/www/html/pub/index.php on line 71 and exactly 2 expected in /var/www/html/vendor/open-telemetry/opentelemetry/src/SDK/Common/Attribute/Attributes.php:19 Stack trace: #0 /var/www/html/pub/index.php(71): OpenTelemetry\SDK\Common\Attribute\Attributes->__construct(Array) #1 {main} thrown in /var/www/html/vendor/open-telemetry/opentelemetry/src/SDK/Common/Attribute/Attributes.php on line 19

George

11/01/2022, 7:55 PM

my loop is below

George

11/01/2022, 7:55 PM

Copy code

for ($i = 0; $i < 3; $i++) {
    // start a span, register some events
    $span = $tracer->spanBuilder('loop-' . $i)->startSpan();

    $span->setAttribute('remote_ip', '1.2.3.4')
        ->setAttribute('country', 'USA');

    $span->addEvent('found_login' . $i, new Attributes([
        'id' => $i,
        'username' => 'otuser' . $i,
    ]));
    $span->addEvent('generated_session', new Attributes([
        'id' => md5((string) microtime(true)),
    ]));

    $span->end();
}

C H

11/02/2022, 8:11 AM

Hello 🙂 I'm looking for monitoring on Signoz, There is a health check api endpoint for signoz ?

Prashantkumar Suthar

11/02/2022, 9:33 AM

Hi do signoz community version have a support to configure application healthcheck endpoint if it fails to reach it will raise slack notification. Please let me know if this is possible.

Ashna

11/02/2022, 11:13 AM

Ashna

11/02/2022, 11:14 AM

2022/11/02 102124 collector server run finished with error: failed to get config: cannot unmarshal the configuration: unknown exporters type "clickhouselogsexporter" for "clickhouselogsexporter" (valid values: [awsemf clickhouse honeycomb newrelic opencensus prometheusremotewrite skywalking sumologic alibabacloud_logservice file splunk_hec awsprometheusremotewrite awsxray f5cloud jaeger logging prometheus signalfx awskinesis elasticsearch googlecloud kafka logzio sapm zipkin carbon humio loki stackdriver tencentcloud_logservice azuremonitor clickhousemetricswrite datadog dynatrace elastic jaeger_thrift loadbalancing otlp otlphttp parquet influxdb sentry tanzuobservability])

Ashna

11/02/2022, 11:14 AM

I am getting this error while trying to upgrade the version

Shivvinay Kanswal

11/03/2022, 7:49 AM

Hi, Im trying to export logs using fluentbit to signoz. The logs are being sent from ec2 Instances and fluentbit tail operator is being used as input. Signoz is installed in eks cluster. Tried sending it to 2 of the endpoints fluentforwarder and otlp. • While sending to fluentforwarder, no logs are being received at signoz. • While sending to otlp, logs are received in a single field

body

Logs are in JSON format, wanted the logs to be parsed by keys in json as fields so that we get to filter with them. Currently struggling with that. Could someone guide me here on what i might be missing. Sharing all the configurations on this thread.

Shivvinay Kanswal

11/03/2022, 12:30 PM

Hi Team, clickhouse database initialisation is failing on signoz clickhouse statefulset. signoz collector pods are getting error that database signoz_metrics doesn't exist. Could you guide us on how clickouse database is being created by the clickhouse chart, on the chart config we see signoz_metrics database getting used but dont see it getting created anywhere Is the database creation being handled internally?

Alexei Zenin

11/03/2022, 4:18 PM

Just to flag anyone upgrading from 0.11.0 -> 0.11.2 make sure to add the logs configuration (ie https://github.com/SigNoz/signoz/blob/07dcdb51f7e98cb853f32e065d59082bb6bfb20b/deploy/docker/clickhouse-setup/otel-collector-config.yaml#L148) for your signoz-otel-collector otherwise the logs tables won’t exist and the Settings page on the UI won’t load. I believe i didn’t add it for 0.11.0 either but UI still worked for Settings in that page, maybe i just never clicked there