I can see all services and pods of sample application are up & running fine

Hello team, when inviting new users to Signoz (open source), the emails are not being sent. Where can i define the SMTP server to send emails?

Hi Team, how I remove default Services of SigNoz in UI SigNoz ? or anyway maybe delete default Services

frontend
customer
mysql
driver
route
redis

Kind of dumb question, but how do you install agents on machines that you want to monitor, and send data to Signoz installation? Or do I just need to install prometheus agent?

To begin with, all I want is to see System default metrics in Signoz UI, that should be coming from n number of systems (each having it's agent).

Hey folks, We currently have apps running on Kubernetes and scattered observability stack with Metrics on prometheus, Logs on Loki and new Relic. Would like to try out Signoz based observability stack and would love to get some pointers on what might be the best way to migrate to newer platform. General idea is ability to duplicate the data initially across old and new stacks to ensure we are capturing everything properly and ability to cut over to new platform as needed service by service. Thanks in advance.

Hi there, can someone clarify/explain on the managed service pricing you guys have?

I want to sync my cloudwatch logs to Signoz. Is there a way to do it?

Hey folks, Weirdly for my standalone docker deployment, i am seeing errors in my application logs during trace export in production recently. This happens occasionally. Tried running the application locally by setting the

OTEL_EXPORTER_OTLP_ENDPOINT

to the prod signoz deployments and got some error. Below is my django logs

Traceback (most recent call last):
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/sdk/_logs/export/__init__.py", line 259, in _export_batch
    self._exporter.export(self._log_records[:idx])  # type: ignore
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/exporter/otlp/proto/http/_log_exporter/__init__.py", line 142, in export
    resp = self._export(serialized_data)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/exporter/otlp/proto/http/_log_exporter/__init__.py", line 113, in _export
    return <http://self._session.post|self._session.post>(
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/requests/sessions.py", line 578, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 128, in instrumented_request
    return _instrumented_requests_call(
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 245, in _instrumented_requests_call
    raise exception.with_traceback(exception.__traceback__)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 209, in _instrumented_requests_call
    result = call_wrapped()  # *** PROCEED
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 126, in call_wrapped
    return wrapped_request(self, method, url, *args, **kwargs)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 148, in instrumented_send
    return _instrumented_requests_call(
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 159, in _instrumented_requests_call
    return call_wrapped()
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 146, in call_wrapped
    return wrapped_send(self, request, **kwargs)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

And i cant find any way of checking what exactly is happening. I am assuming the otel-collector is somehow not able to ingest the spans in clickhouse but i couldn’t find any way of viewing debug logs for the collector to understand the root cause. Is this probably a scale issue? I checked the troubleshooting guide and running the trouble shooting command gave following error.

> sudo docker run -it --rm signoz/troubleshoot checkEndpoint --endpoint=172.17.0.1:4318
2023-06-13T11:36:21.026Z	INFO	troubleshoot/main.go:28	STARTING!
2023-06-13T11:36:21.026Z	INFO	checkEndpoint/checkEndpoint.go:41	checking reachability of SigNoz endpoint
Error: not able to send data to SigNoz endpoint ...
rpc error: code = Unavailable desc = connection closed before server preface received
Usage:
  signoz checkEndpoint [flags]

Examples:
checkEndpoint -e localhost:4317

Flags:
  -e, --endpoint string   URL to SigNoz with port
  -h, --help              help for checkEndpoint

Been stuck on this and google searches aren’t returning anything helpful.

👋 Hi everyone!

I have recently started using signoz but i don't known how to connect php application with signoz

Does Signoz have a demo project that I can play with to evaluate the system like Uptrace has for example: https://app.uptrace.dev/play

Hey all! I'm Joan, the co-founder and CTO at crowd.dev. We will be implementing Signoz as a self-hosted solution for our monitoring. We want to use it for both logs and monitoring. We use Kubernetes, and we already have structured logs going to Logz, so we would need to: • Add Signoz to our Kubernetes cluster • Redirect the logs to Signoz • Add full monitoring to Signoz We were thinking of having this done by a freelancer, and I thought it would be great if anyone here was up for it.

Hi guys, I just started using signoz. Was wondering how I can remove signoz own logs. I've managed to remove the sample app metrics and traces by following this guide - https://signoz.io/docs/operate/docker-standalone/#remove-the-sample-application-from-signoz-dashboardbut I see the signoz logs are still coming however. How can I remove? My logs interface is quite noisy as a result of the signoz own logs and I'd like to remove so I'm left with only logs for the application being monitored. See screenshot of the signoz logs. Also below. Would appreciate if someone can guide me. Thanks in advance 🙏

2023-06-17T104805+03:00 | 102.220.12.50 - - [17/Jun/20230748:05 +0000] "GET /api/v1/logs/tail?timestampStart=1686986894073000000 HTTP/1.1" 200 73664 "http://signoz.domain.xyz:3301/logs?q=" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-"

2023-06-17T104805+03:00 | 2023-06-17T074805.163Z INFO app/server.go:280 /api/v1/logs/tail timeTaken: 4m50.969067478s

2023-06-17T104805+03:00 | 2023-06-17T074805.163Z DEBUG app/http_handler.go:2261 done!

2023-06-17T104805+03:00 | 2023-06-17T074805.161Z DEBUG clickhouseReader/reader.go:3541 closing go routine : 172.18.0.10:59372

2023-06-17T104804+03:00 | 2023.06.17 074804.850539 [ 47 ] {2107a9a8-84e4-4791-b150-561fbbba4b5d} <Information> executeQuery: Read 798 rows, 6.23 KiB in 0.002790747 sec., 285944 rows/sec., 2.18 MiB/sec.

2023-06-17T104804+03:00 | 102.220.12.50 - - [17/Jun/20230748:04 +0000] "GET /logs?q= HTTP/1.1" 200 791 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36" "-"

2023-06-17T104804+03:00 | 2023-06-17T074804.407Z DEBUG clickhouseReader/reader.go:3553 SELECT timestamp, id, trace_id, span_id, trace_flags, severity_text, severity_number, body,CAST((attributes_string_key, attributes_string_value), 'Map(String, String)') as attributes_string,CAST((attributes_int64_key, attributes_int64_value), 'Map(String, Int64)') as attributes_int64,CAST((attributes_float64_key, attributes_float64_value), 'Map(String, Float64)') as attributes_float64,CAST((resources_string_key, resources_string_value), 'Map(String, String)') as resources_string from signoz_logs.distributed_logs where timestamp >='1686988064851955929' and id > '2RIKkvmFr30zGjfEHHa8mJWn4HL' order by timestamp desc, id desc limit 100

the helm chart's part for alertmanager has the below commented content is this for creating slack alert channel? If yes where can i get the configuration similar to this for Pagerduty?

## Using the config, alertmanager.yml file is created.
  ## We no longer need the config file as query services
  ## delivers the required config.
  # config:
  # global:
  #   resolve_timeout: 1m
  #   slack_api_url: '<https://hooks.slack.com/services/xxx>'

  # templates:
  #   - '/etc/alertmanager/*.tmpl'

  # receivers:
  # - name: 'slack-notifications'
  #   slack_configs:
  #   - channel: '#alerts'
  #     send_resolved: true
  #     icon_url: <https://avatars3.githubusercontent.com/u/3380462>
  #     title: '{{ template "slack.title" . }}'
  #     text: '{{ template "slack.text" . }}'

  # route:
  #   receiver: 'slack-notifications'

<!channel> 👋 Time for OpenTelemetry end-user discussion group APAC session of this month! The end-user group discussion is a place where you can discuss challenges related to your OTel implementations and learn from other OpenTelemetry users in a vendor-neutral space. Here are the details: Date & Time: Wednesday, June 21, 2023 at 11:00 AM to12:00 PM IST Link to register: https://lu.ma/1w129wgu Looking forward to discuss all things OpenTelemetry. 🙌

Hi people! I'm new to Signoz, trying to deploy Signoz on-premise air-gapped Openshift 4.10. being having some problems, do we have any option to change it to get it from another URL on my network?

+ wget -O /tmp/histogramQuantile <https://github.com/SigNoz/signoz/raw/develop/deploy/docker/clickhouse-setup/user_scripts/histogramQuantile>
wget: bad address '<http://github.com|github.com>'

Dear All, We have installed Signoz in EKS, and when trying to delete the namespace platform- it is in a terminating state- when further verified, seeing the below message , NamespaceFinalizersRemaining True Tue, 09 May 2023 064259 +0000 SomeFinalizersRemain Some content in the namespace has finalizers remaining: finalizer.clickhouseinstallation.altinity.com in 1 resource instances Can some one help how to remove clickhouse finalizers ? Thanks in advance.!!

A Pic of our meeting about SigNoz yesterday on 🇧🇷 #devcommunity #opensource My thanks.

Hi all! I'm Nočnica, the SigNoz team's new developer relations lady. I'm super excited to get connected with this community, having worked with Observability and OpenTelemetry for a number of years. Just before I started at SigNoz, I was lucky enough to get to speak at Open Source Summit North America on OpenTelemetry: I'd like to make a small request of anyone reading this: t*ell me what you want to see next.* New tutorials, guides, events? What's the one thing you wish SigNoz was doing? DM me or reply here!

Hello @Nočnica Mellifera and SigNoz team, is there a way that SigNoz can load (pull) logs from another data source apart from ClickHouse? are there plans to do anything like that? Thanks in advance.

This is an interesting question! I’m curious how you’re collecting and exporting logs now. It seems like it should be possible to connect logging to the OpenTelemetry Logginng, e.g. with the Filelogs receiver https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver If I’m off base here please let me know!

Well, it was not about collecting logs, it was more about searching logs from different destinations…. as SigNoz UI is currently designed to query logs that are stored in ClickHouse… My question is, what if logs are stored in other destinations, eg: File system, or ObjectStore (In Azure of GCP). Is there an adaptor layer that can be re-purposed? @Nočnica Mellifera @Pranay ? Thanks again.

A bit new to signoz, unable to get answers to some questions i have over the internet. Please help if possible I want to set up alerts on general exceptions received by Signoz. Currently this is the existing alert description:

This alert is fired when the error count (current value: {{$value}}) crosses the threshold ({{$threshold}})

I want to make the description more descriptive with information on what the error is. I would like to know what all possible variables i can access in the description with ${{}}. Where can i find the list of variables?

anyone here attending Monitorama - do say hi! to @Nočnica Mellifera - https://twitter.com/Serverless_Mom/status/1673384291856977920?s=20

This is great advice for anyone trying to view low-cardinality data

Hi Guys, I'm trying to redact certain sensitive information bits that are in the log files before exporting using the Open Telemetry collector. Problem - I can't seem to figure out how to redact the value "XnHyH4QALxXnQwmvw7XtB2brs63K4pby" I have tried to use both the hash attribute processor and redaction processor without success. See my yaml config below.

processors:
  attributes:
    actions:
      - key: body
        pattern: 'XnHyH4QALxXnQwmvw7XtB2brs63K4pby'
        action: hash

  redaction:
    blocked_values:
      - "XnHyH4QALxXnQwmvw7XtB2brs63K4pby"

See log below - I've applied json formatting to the log so i can apply json_parser on the collector config - that part is working.

{
  "body": "Request full data: {\n    \"username\": \"user\",\n    \"key\": \"XnHyH4QALxXnQwmvw7XtB2brs63K4pby\",\n    \"operator\": \"123\",\n    \"no\": \"123\",\n    \"units\": 20\n}",
  "attributes": {
    "host_and_client": {},
    "channel": "production",
    "context": {},
    "datetime": "2023-06-28T12:51:56.962453+03:00",
    "extra": {},
    "level": 200,
    "level_name": "INFO"
  }
}

Is there anything I'm missing? How can I achieve the redaction? Thanks a lot in advance🙏

Hi Team, can we export otel collector metrics/trace to prometheus server?

I am trying to get logs published to slack from Signoz. The alerts are almost what looking for; just that they are more like a trigger while i try to get the actual log message to slack (with a certain filters applied ofcourse) instead of the a alaert that trh trigger is activated and deactivated. Is that possible with signoz?

Hi, I have just started exploring SigNoz, need one help. How do I collect logs from std out and send to SigNoz for spring boot application?