Hey all I am looking at diving into our company using Signoz

Question

Hey all I am looking at diving into our company using Signoz I want to have a POC where we could have one Signoz instance that can recieve data from multiple data sources from different clusters However I want to ensure that the transfer of this data is secured ideally by some sort of API key I tried reading the docs but I didn t see anything Does anyone have any experience with this

Valentin Baert · Answer

There are some documentations over here <https github com open telemetry opentelemetry collector tree main config configauth> However I didn t managed to make it work on my side If you find a way tell us stuck out tongue

Blake Romano · Answer

This looks like exactly what I was looking for thank you

Prashant Shahi · Answer

< Blake Romano> < Valentin Baert> There is a detailed guide here that should be useful

Valentin Baert · Answer

Yes I saw that but in my case I m using the otel operator for kubernetes and with the sidecar deployment mode I don t know how to configure tls certificates

Prashant Shahi · Answer

You could add additional host that you will be using to communicate between OtelCollectors in CSR JSON and generate the certificates locally

Prashant Shahi · Answer

Followed by `create secret tls` with certificate and key You can refer here <https aws otel github io docs getting started operator install the opentelemetry operator to your eks cluster>

Blake Romano · Answer

I super appreciate the tips < Prashant Shahi> this is def something I will need if I want to open the collector on the public internet to allow for cross cluster vpc communication

Prashant Shahi · Answer

yeah that definitely makes sense when you have cross cluster vpc communication We will be thoroughly be testing and creating tutorials on the same in some time

Prashant Shahi · Answer

Meanwhile if you need a simpler solution you could use BasicAuth as we have tested it as well Config would look something like this ``` otlp protocols http auth authenticator basicauth cors allowed origins <http localhost> extensions health check zpages basicauth htpasswd inline | $ BASIC AUTH USERNAME $ BASIC AUTH PASSWORD service extensions health check zpages basicauth ``` while you would have to include those environment variables for OTel collector ``` environment BASIC AUTH USERNAME=signoz BASIC AUTH PASSWORD=password```

Blake Romano · Answer

and this authentication would work for all the Open Telemetry recievers like Prometheus etc

Prashant Shahi · Answer

This would work for OTLP So I reckon you would need to use another OtelCollector agent in between

Prashant Shahi · Answer

metrics traces in cluster A gt Otel in cluster A across cluster vpc gt Otel in cluster B signoz otel

Blake Romano · Answer

Ahh okay I am still making trying to wrap my head around the Open Telemetry concepts with Signoz but that makes sense As long as I know it s possible with some basic ideas as to where I need to bring it that is super helpful

Blake Romano · Answer

One other quick question < Prashant Shahi> what is the difference between Signoz Alertmanager and Prometheus Alert Manager We already have Prom Alert Manager setup Could we configure that with Signoz to integrate the two Reading the signoz alertmanager readme it seems to be copied from Prom Alert Manager

Prashant Shahi · Answer

< Blake Romano> yes Signoz Alertmanager is a fork used for supporting alerts in SigNoz any particular reason that you would want to integrate them I am not sure how that would work though < Amol Umbark> would be the best person to answer that

Blake Romano · Answer

Well we already have a lot of setup with the Prometheus Rules CRD and also we would like to be able to programatically create alerts for services using source code

Ankit Nayan · Answer

< Blake Romano> the signoz s alertmanger just creates a few apis to add channels dynamically from the SigNoz s UI Also alertmanager initialization is not dependent on config yaml anymore But we have not tested signoz with prometheus alertmanager and thus can break

Ankit Nayan · Answer

also can you please explain this in somewhat more detail gt we would like to be able to programatically create alerts for services using source code Are you talking about applications in python nodejs go creating alerts Or just maintaining a list of yamls containing alerts and rules

Blake Romano · Answer

Be able to maintain a list of yamls that have the rules and alerts more specifically ^

Blake Romano · Answer

is that doable

Ankit Nayan · Answer

It won t be doable right now but I think it can be built We will be releasing a better Ui for writing queries for alerts and dashboard with autosugestions on metric names and labels tags So our data format for rules and alerts is going to change but we can build a feature where we initialise the rules from yamls

Ankit Nayan · Answer

Can you raise a feature request for this at out github Would be useful to track it publically

Blake Romano · Answer

yup can do I think for our use case using a CRD and deploying it that way would be helpful

Ankit Nayan · Answer

I would suggest give signoz a deeper try and if this the only thing restricting you to move to prod We shall pick it up in our sprints

Blake Romano · Answer

Blake Romano · Answer

I added it I am hoping to have a POC in about a month which is when this feature would be more crucial in our steps towards transition

Ankit Nayan · Answer

Cool +1

Blake Romano · Answer

< Prashant Shahi> is there a way to have an ingress for the Otel collector I d like to open up the authenticated port for the Signoz Otel Collector on an ingress Is this something I could PR the helm chart

Prashant Shahi · Answer

we haven t added ingress yet for otel collector and otel collector metrics It could be great if you could raise a PR with those

Blake Romano · Answer

I can probably do that I have a PR out on helm chart now regarding a bug if you don t mind checking it out slightly smiling face

Prashant Shahi · Answer

Thanks for the PR I have reviewed and left comment