signoz-community #support

Hi Team, I'm having issue with sending logs from logstash to signoz. I followed this guide: https://signoz.io/docs/userguide/logstash_to_signoz/ but in signoz dashboard there's still no logs at all.

Bhavesh Patel

10/10/2022, 6:49 AM

Hi Team, I have tried to configure Signoz Alert Channel as google chat spaces webhook with below parameter. But, it has giving below error. Can you help me on the same? Channel Name: Test Channel Type: Webhook Webhook URL: https://chat.googleapis.com/v1/spaces/{SPACE_ID}/messages?key={Authkey}&token={Authtoken} Error: Failed to send a test message to this channel, please confirm that the parameters are set correctly CC: @Prakshal Shah

Bhavesh Patel

10/10/2022, 11:59 AM

Have you got any chance to see this issue?

Vishal Goel

10/10/2022, 12:16 PM

I am not getting data of apm in signoz, getting this error. [ERROR] [init.py:156- export()] opentelemetry.exporter.otlp.proto.http.trace_exporter: Failed to export batch code: 405, reason: <html> i open 4317 and 4318 port but still not allowed. can someone help into this

Vishal Goel

10/11/2022, 5:28 AM

how i can send data in APM if my python application is using python version below 3.7

Vishal Goel

10/11/2022, 7:50 AM

my user is not working with https but its working on http. when i created user, i am using http but later on i redirected it to https but its only working with http

Kasim Ali

10/11/2022, 8:10 AM

How to get logs in signoz using fastapi python?

nitya-signoz

10/11/2022, 10:42 AM

These are your docker container logs which are automatically collected, you can disable by following this guide https://signoz.io/docs/userguide/collect_docker_logs/#disable-automatic-container-log-collection

Allan Li

10/11/2022, 5:01 PM

Hello! I just wanted to ask a questions regarding instrumentation for Apollo GraphQL. How would we get the .execute traces to display the SQL query string or any specific information regarding the GraphQL query that was made? Thanks ahead of time!

Vishal Goel

10/12/2022, 6:51 AM

Doest it support java version 7?

Kasim Ali

10/12/2022, 9:51 AM

I am using manual and auto instrumentation for my fastapi python application, so when i have to instrumented and write code for app (1) before creating client / before creating gateway (2) after creating gateway

Devops Netapp

10/13/2022, 2:26 PM

Hi, After running helm install on AKS I get the following services:

NAME                                      PORT(S)
chi-my-release-clickhouse-cluster-0-0     8123/TCP,9000/TCP,9009/TCP
my-release-clickhouse                     8123/TCP,9000/TCP
my-release-clickhouse-operator-metrics    8888/TCP
my-release-k8s-infra-otel-agent           13133/TCP,8888/TCP,4317/TCP,4318/TCP
my-release-k8s-infra-otel-deployment      13133/TCP,8888/TCP,4317/TCP,4318/TCP
my-release-signoz-alertmanager            9093/TCP
my-release-signoz-alertmanager-headless   9093/TCP
my-release-signoz-frontend                3301/TCP
my-release-signoz-otel-collector          14250/TCP,14268/TCP,8888/TCP,4317/TCP,4318/TCP
my-release-signoz-otel-collector-metrics  13133/TCP
my-release-signoz-query-service           8080/TCP,8085/TCP
my-release-zookeeper                      2181/TCP,2888/TCP,3888/TCP
my-release-zookeeper-headless             2181/TCP,2888/TCP,3888/TCP

Is this the correct configuration for sending the traces:

otelcolConfig: {
        url: '<http://my-release-signoz-otel-collector.platform.svc.cluster.local:4318/v1/traces>', // URL of opentelemetry collector
      },

When I enable traces to console I’m able to see the output but when configuring it to send to my-release-signoz-otel-collector service on port 4318 I get no results.

Allan Li

10/13/2022, 3:26 PM

Hello, is there a way to visualize and query traces by ranking them based on their GraphQL name?

Mark Raynsford

10/14/2022, 6:15 PM

hello! i wiped my installation from last week, cloned commit

5eed384ffe05fd734bbbd8ae80b921106c505149

(the current

main

) and ran the usual

docker-compose -f docker/clickhouse-setup/docker-compose.yaml up -d

command to set up a fresh signoz. for whatever reason, i'm now unable to get into the UI (there are no existing accounts, i'm trying to create a new account):

signoz.mkv

Mark Raynsford

10/14/2022, 6:15 PM

don't worry about the silly password and fake email address, this is just for local testing on my dev machine. these steps worked fine a couple of weeks ago...

Alexei Zenin

10/14/2022, 8:25 PM

Just wanted to flag that if you misconfigure the URL for clickhouse in the collector you get this error:

panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x343dd12] goroutine 87 [running]: <http://github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.(*PrwExporter).export.func1()|github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.(*PrwExporter).export.func1()> /src/exporter/clickhousemetricsexporter/exporter.go:280 +0xf2 created by <http://github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.(*PrwExporter).export|github.com/SigNoz/signoz-otel-collector/exporter/clickhousemetricsexporter.(*PrwExporter).export> /src/exporter/clickhousemetricsexporter/exporter.go:276 +0x256

Upesh Jindal

10/15/2022, 3:04 AM

Hi, Is the following a known issue: "opentelemetry.sdk.metrics._internal.instrument", "funcName": "callback", "lineno": 139, "message": "Callback failed for instrument system.swap.utilization.", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.10/site-packages/opentelemetry/sdk/metrics/_internal/instrument.py\", line 132, in callback\n for api_measurement in callback(callback_options):\n File \"/usr/local/lib/python3.10/site-packages/opentelemetry/instrumentation/system_metrics/__init__.py\", line 423, in _get_system_swap_utilization\n getattr(system_swap, metric) / system_swap.total,\nZeroDivisionError: division by zero

Ashish

10/17/2022, 11:37 AM

Hi <!channel> I am facing issue in configuring my DB monitoring, can anyone help me regarding this issue ?

Ashish

10/17/2022, 12:52 PM

Hi <!channel> Does anyone know how to check if my mongodb exporter is able to communicate with signoz

Akhilesh Bussa

10/17/2022, 1:05 PM

Hi Team, I have setup signoz in AWS using docker. I have configured the OTel in my django application, although I can see OTel logs, I cant see the same on signoz dashboard. I have also tried troubleshooting endpoint and it was successful, but there are no traces/data on the dashboard (Screenshots for reference) What am I missing here?

Akhilesh Bussa

10/17/2022, 4:20 PM

Hi team, I was trying to setup with Django + uWSGI with this command and the postfork decorator as mentioned in the example provided:

OTEL_RESOURCE_ATTRIBUTES=service.name=hancock_staging OTEL_EXPORTER_OTLP_ENDPOINT="<http://MySignozIP:4318>"  opentelemetry-instrument --traces_exporter otlp_proto_http /home/hancock/env/bin/uwsgi --ini /home/hancock/config/api_uwsgi.ini

But I get this error:

File "/home/hancock/env/lib64/python3.7/site-packages/uwsgidecorators.py", line 10, in <module>
    import uwsgi
ModuleNotFoundError: No module named 'uwsgi'

Can someone help me?

Bhavesh Patel

10/18/2022, 3:56 AM

Hi Team, I want to scale the Signoz infrastructure for handle the heavy load. Can you help us with the same?

Bhavesh Patel

10/18/2022, 6:14 AM

Also, which database we are using for storing data in signoz?

Kasim Ali

10/18/2022, 8:17 AM

currently i am trying to integrate signoz with my django app it can work properly but i have one issu with that i can get traces but receive traces are independently not with child and parent graph.

Daniel Nordberg

10/18/2022, 12:09 PM

Am following the tutorial to get Django instrumentation running, I installed dependencies, ran

opentelemetry-bootstrap --action=install

, then started the server like so

OTEL_RESOURCE_ATTRIBUTES=service.name=django_app OTEL_EXPORTER_OTLP_ENDPOINT="<http://localhost:4318>"  opentelemetry-instrument --traces_exporter otlp_proto_http python manage.py runserver --noreload

, however I can't reach http://localhost:4318

Daniel Nordberg

10/18/2022, 12:10 PM

On another note, do I have to run the docker stack on the same network

Łukasz Herman

10/18/2022, 12:16 PM

Hi, can you help me with configuring fluentD to send correct data? Currently I'm working on two cases: • IIS Logs • Log4Net application logs With IIS I have almost everything as I would like to. The only problem I see that I would like to store some fields as numeric/int and IP address. Currently all is stored as a string. With Log4Net I need to build grok parsing and I'm lost with it... In Log4Net we are using configuration: %d [%t][%P{vRequestId}]*[%P{vSqlServer}]*[%P{vUserName}][%P{vInstance}] %-5p %c - %m%n vRootUrl:%P{vRootUrl} %n vRequestUrl:%P{vRequestUrl} %n Body:%P{vLogExtension}%n And I have no idea how to build grok filter for it.

Title

Łukasz Herman

10/18/2022, 12:16 PM

Ankit Nayan

10/18/2022, 1:07 PM

@nitya-signoz have you worked with grok? Also, can we do the parsing at otel-collector?

nitya-signoz

10/18/2022, 3:55 PM

Have a little idea but would like to understand the problem better. @Łukasz Herman can you please create a GitHub issue with the configuration that you are currently using for both problems, and also share some example logs. While we work towards a solution it will help the community as well.

Łukasz Herman

10/20/2022, 2:14 PM

Where should we parse the logs? Currently I understand that fluentD can use Grok to parse log and slice it into different fields. But on the SigNoz side all of them are strings. Is there any other way to parse logs (on SigNoz side)?

@nitya-signoz @Ankit Nayan Where should I parse logs? In general it will be more suitable for me to do all parsing in one central location like SigNoz. But how? And when somehow I will parse the logs, how to change datatypes (to int, IP, decimal, etc.)?

nitya-signoz

10/25/2022, 8:47 AM

Yes, Ideally all the parsing should be done in the otel-collector-config.yaml file. For parsing the logs there are different operators available https://signoz.io/docs/userguide/logs/#operators-for-parsing-and-manipulating-logs

If you can share some example log lines, we can help you with how to parse it.

Łukasz Herman

10/25/2022, 9:05 AM

Lets focus on log like below: { "timestamp": 1666659631502000000, "id": "2FlLy8mdGBzFof0oKEZy4x4jtMg", "trace_id": "", "span_id": "", "trace_flags": 0, "severity_text": "", "severity_number": 0, "body": ""[216][638022636288199459][SQL123\\SQL2019B][System][saas] INFO Core.WebTasks.Esb.WebTaskJobProcessor - Zaczynam WebTask:AfterUpgrade (saas)\r\n vRootUrl:https://fake.org/ \r\n vRequestUrl:https://fake.org/? \r\n Body:(null)"", "resources_string": {}, "attributes_string": { "app": "app1", "environment": "UAT", "fluent_tag": "log4net", "hostname": "AP20", "logfile": "info.log", "logfilepath": "D:/Data/info.log", "module": "app1" }, "attributes_int": {}, "attributes_float": {} } I would like to parse 'body' and extract some fields like thread in "[", "]" (216), and so on.

nitya-signoz

10/25/2022, 9:16 AM

This is something that you have taken from the signoz UI. Can you share a raw log line from the file/source where you are getting the data, as some operators are already applied here.

Łukasz Herman

10/25/2022, 9:22 AM

I'm grabbing file logs using FluentD and sending to fluentforward receiver. As I would like to parse them on SigNoz side, it doesn't matter what are source logs.

nitya-signoz

10/25/2022, 9:25 AM

Yeah if you are using fluentD then source logs format doesn’t matter, but the output format of fluentD matters. you can use https://docs.fluentd.org/output/stdout to check the output that is going to signoz.

Łukasz Herman

10/25/2022, 10:07 AM

On Windows FluentD is working as a service. I'm not able to get standard output. I can save it to file, but it will the same as saving to fluentforward, isn't it?

nitya-signoz

10/25/2022, 11:41 AM

Yeah it will give me us an idea about whats happening with the logs in fluentD before it’s going to a dowstream.

Łukasz Herman

10/26/2022, 7:34 AM

I was not able to send log to file (it is failing and I'm not able to figure out why) but it's logging in FluentD log showing what data it has. "2022-10-26T09:22:54+02:00\tlog4net\t{\"message\":\"[46] DEBUG - Init WindsorHttpModule\",\"logfilepath\":\"D:/Data/App1/Komunikacja/info.txt\",\"hostname\":\"AP20\",\"app\":\"App1\",\"module\":\"Komunikacja\",\"logfile\":\"info.txt\",\"environment\":\"UAT\"}\r\n"

It's more readable when I grab what is in FluentD buffer. For log4Net: 2022-10-26T09:09:39+02:00 log4net {"message":"[52] DEBUG - Dispose RequestTimerModule","logfilepath":"D:/Data/App1/Administracja/info.txt","hostname":"AP20","app":"App1","module":"Administracja","logfile":"info.txt","environment":"UAT"} For IIS: 2022-10-26T09:37:44+02:00 iis.arr {"message":"W3SVC1 ARR2 10.50.60.122 POST /App1Wcf/App1WcfService.svc X-ARR-CACHE-HIT=0&SERVER-ROUTED=AP23C&X-ARR-LOG-ID=01dcb691-a6c9-4c6d-a30f-9fa6ee484b0b&SERVER-STATUS=200 443 - 10.50.60.163 HTTP/1.1 - - App1.uat 200 0 0 3581 796 11 10.50.60.163:50137","s-sitename":"W3SVC1","s-computername":"ARR2","s-ip":"10.50.60.122","cs-method":"POST","cs-uri-stem":"/App1Wcf/App1WcfService.svc","cs-uri-query":"X-ARR-CACHE-HIT=0&SERVER-ROUTED=AP23C&X-ARR-LOG-ID=01dcb691-a6c9-4c6d-a30f-9fa6ee484b0b&SERVER-STATUS=200","s-port":"443","cs-username":null,"c-ip":"10.50.60.163","cs-version":"HTTP/1.1","cs(User-Agent)":null,"cs(Referer)":null,"cs-host":"App1.uat","sc-status":"200","sc-substatus":"0","sc-win32-status":"0","sc-bytes":"3581","cs-bytes":"796","time-taken":"11","OriginalIP":"10.50.60.163:50137","app":"arr","type":"iis","environment":"UAT"}

nitya-signoz

10/26/2022, 8:45 AM

Got it, let me get back to you on the parsing steps for parsing the above logs.

@Łukasz Herman so from the data that you have shared,

message

is the actual log that is being scraped and you are parsing that in fluentD to extract different values such as

sc-bytes

sc-status

etc. Now since these values are parsed as a string in fluentD itself, it is stored as string in SigNoz. When you are parsing the log in fluentD you can specify the types parameter https://docs.fluentd.org/configuration/parse-section#types-parameter to parse to a specific type.

Łukasz Herman

10/27/2022, 1:01 PM

Ok, thanks. What for log4net example, when I would like to parse it outside fluentD? It's more suitable for me to parse it at destination.

nitya-signoz

10/27/2022, 1:12 PM

The current version of signoz otel collector doesn’t support type casting. Once this issue is fixed https://github.com/SigNoz/signoz-otel-collector/issues/19#issuecomment-1293276797 , you will be able to typecast string values to int.

Łukasz Herman

10/27/2022, 1:25 PM

Because of new version of Fluent Bit (2.0 looks like solving most demanding tasks to have one tool to grab all logs and metrics) I'm wondering if it will best to put few Fluent Bit servers in beetween source and destination for parsing and filtering. Then the whole stack will be http based as I can send all data in Otel format. (I like HTTP more than TCP because I can send all through our load balancers and have everything in control). Then the pipeline will be Source servers with FluentBit (many servers in many datacenters) -> (Maybe) Central local FluentBit per Environment/Datacenter -> Load balancer (entrance to main datacenter where all data will be stored) -> FluenBit farm (scailing servers easily, performing all parsing and basic filtering) -> Load balancer -> SigNoz Otel Collector I need to check If Otel input and output in FluentBit are working as I think and I they can grab data I would like to have. What do you think about that?

nitya-signoz

10/27/2022, 2:38 PM

Keeping the parsing stuff separate from ingestion machines will be definitely a better idea. Though you can replace the intermediate ones with otel-collectors instead of fluentbit if you are parsing the data at the source fluentBit. Since at the end when fluentD/fluentBit sends data to signoz you might have to transform a few things and doing it before it reaches the final signoz-otel-collector is a better idea.

View count: 1