https://signoz.io logo
Join Slack
Powered by
This message was deleted.
# support
s
This message was deleted.
šŸ”„ 2
šŸ‘ 1
a
@User that's awesome news šŸŽ‰
b
Thanks
Actually guys, I already followed that tutorial some days back. I understand how to use open telemetry with tomcat. My issue is to get the pods in kubernetes to run their app with open telemetry. I suppose that there are multiple ways to do it, but I would like to use the most simple and straight forward way. The variables in the setenv.sh file can be set in the docker file I suppose, and then it's just a matter of getting the application to launch and see opentelemetry.jar file and use it.
p
I think @User should have some insights here.
b
What I was wondering is, in the signoz dashboard on the metrics page, I see that if we use the agent in k8s, we will see the pod name in the dashboard, is there a way to get the agent to send more identifying information back to signoz, like say the namespace and vertical? And I was wrong, we're using springboot to deploy a java app and not tomcat.
p
i think there should be @User While exporter you are using for sending k8s metrics?
b
Hi @User We have 6 verticals, and 5 of those verticals have a prod and a preprod. One of them has a dev, qa, stage, preprod, and prod. All of our apps are repeated in many different vertical / environment combinations, or if you will, name spaces. So in the metrics tab of the signoz dashboard, we will be looking at many instances of the same service name without knowing which vertical or environment the service is running in. It is such a beautiful app, and my team loves it. But if we want to use it, we will have to have a way to logically segment the app instances by name space. When you first land on the home page of the signoz server, you see "applications" in the "metrics" tab. Is there a way to make the first default view, "name space", and then if we click on that name space, we will see the list of different apps in that namespace? And my other question is, are those service names "front end", "customer", "driver", and "route", are they sample applications running in containers that are there to provide a view of the features that the signoz application provides when you first launch the docker-compose.yml file? Or are they metrics of the signoz server itself? It they're just sample applications, I suppose the way to make them go away from the dashboard would be to kill their respective containers. Any help along those lines would be appreciated.
p
Hey @User the "frontend", "customer" etc. apps are sample apps. To remove them https://signoz.io/docs/operate/docker-standalone/#remove-the-sample-application
b
To help explain the above questions, consider the the following test with a sample docker file:
Copy code
docker run -it -p 8080:8080 -e OTEL_METRICS_EXPORTER=none -e OTEL_EXPORTER_OTLP_ENDPOINT="<http://10.1.130.93:4317>" -e OTEL_RESOURCE_ATTRIBUTES=service.name=cfpb-service,service.namespace=docmagic-dev-torrance -e POD_NAMESPACE=docmagic-dev-torrance cfpb-brian
I could run this again and again and pass a different namespace to open telemetry, but signoz doesn't seem to pick up the difference. As a test, I ran this again on ports 8081:8080 and gave it a different namespace, "`docmagic-qa-torrance` " and signoz didn't show the instance in the metrics dashboard.
Thanks for the sample app removal link @User
p
To your first question, I understand the usecase now. Thanks for explaining in detailšŸ™‚ We are building a way to filter services based on different resource attributes. In fact we are building it this sprint... as this was asked by few other users also. Should ship in 2-3 weeks. Should I add you in the list of beta users who can give initial feedback as we build this feature?
b
Please, I really like your app, and you guys are very forthcoming and accessible. I want to make this work for us.
Oh and btw, the instructions for removing the sample apps didn't work, I ran docker-compose down from the clickhouse dir, I removed the two sections from the docker-compose.yaml file, and finally I ran the install.sh file and went to the dashboard, and there were the sample apps again.
I'm trying this now to see if it gets rid of the sample apps:
Copy code
docker kill $(docker ps -q)
docker rm $(docker ps -a -q)
docker rmi $(docker images -q)
/signoz/deploy/install.sh
A good feature for your app would be to clean up any remaining traces of itself when you shut it down.
Well shoot, even that didn't work.
You can see that these images still get installed by install.sh:
Copy code
Untagged: yandex/clickhouse-server:21.12.3.32
Untagged: yandex/clickhouse-server@sha256:7d58af905fef789288892f1eac0d570ac23ef4c9b326baf684f202ac2894af09
Deleted: sha256:d846490c0466cd43c55d0034dc3e100e95f0e619f5eb2866912e9ce7f2fcc550
Deleted: sha256:ab805998626154211b92e7d53299d57283d7630831caeffdb04728b5209bbdde
Deleted: sha256:d765eeef11ac5d0613dcf1dea8c5def8c3b944202c2675a4d379f3c471d7d9d1
Deleted: sha256:2c80bedcf23ba5f8535cbf301d3e0b702ae6f3ddf479fbd3108cc561849245be
Deleted: sha256:44feb4760a5bce46e020c04aadc35e45fb535e39a997a79927e609a9b32fc629
Deleted: sha256:94317d09d33d38f04bfb96580f45f65615abaae1e5cd7586ec4c071335ca741f
Deleted: sha256:f328700af5a8f2b38476e9ef9cf2eef601a121b6c45e6b3452333bc9d53ace8e
Deleted: sha256:0d9e96dc57e22b4e986cd33ea0c4be572bb8403342246e8727483db5155f3040
Deleted: sha256:9f54eef412758095c8079ac465d494a2872e02e90bf1fb5f12a1641c0d1bb78b
During the install.sh setup it shows:
Copy code
šŸŸ” Starting the SigNoz containers. It may take a few minutes ...

[+] Running 6/6
 ā æ Container clickhouse-setup-clickhouse-1              Started                                                                                                                                                                                            1.2s
 ā æ Container query-service                              Created                                                                                                                                                                                            0.1s
 ā æ Container clickhouse-setup-otel-collector-1          Created                                                                                                                                                                                            0.1s
 ā æ Container clickhouse-setup-otel-collector-metrics-1  Created                                                                                                                                                                                            0.1s
 ā æ Container frontend                                   Created                                                                                                                                                                                            0.1s
 ā æ Container clickhouse-setup-alertmanager-1            Created
I ended up having to delete my local copy of signoz and then re-cloning it. I then went into the clickhouse folder and deleted hotrod from the yaml file and then ran install.sh. Then all the sample apps were gone.
a
@User the steps mentioned by @User should have worked. The apps don't just disappear straight away because in the top right we have a time period of 
30 minutes
as default and hotrod would have some data in the last 30 mins and hence it was displayed. Choosing a lesser period would not have shown hotrod apps. Anyways, you got it fixed šŸ™‚
b
Hi @User, oh I see, that makes sense. Do you have any estimate for how long it will take to finish the namespace sprint? Within our namespaces, our apps are unique to the user, but outside of a namespace, the apps become redundant. Example, there's only one cfpb-service in uwm-prod-torrance, but if it gets grouped together with other namespaces, then you would see who knows how many instances of the same app. If the execution times for an app started to drift, we wouldn't know where to go to troubleshoot it.
Thanks for your help.
I have another thought about signoz. Since our apps run in pods, and there are multiple pods per container, kubernetes distinguishes the pods with names like cfpb-service-jahkdka, does signoz also make a distinction between pods? Or does it only show the app's overall execution time? That's perfectly okay if that's the case, this feature would probably propel the app into the stratosphere, and you'd have to start charging for it.
šŸ‘ 1
I have a question that I forgot to ask you @User @User, we noticed that the error rate stays at 0.00%, do you know what would trigger an error, and how would the error rate go up?
I couldn't find a settings adjustment for the browser page refresh.
@User @User I have an idea guys, in this command here:
Copy code
docker run -it -p 8080:8080 -e OTEL_METRICS_EXPORTER=none -e OTEL_EXPORTER_OTLP_ENDPOINT="<http://10.1.130.93:4317>" -e OTEL_RESOURCE_ATTRIBUTES=service.name=cfpb-service,service.namespace=docmagic-dev-torrance -e POD_NAMESPACE=docmagic-dev-torrance cfpb-brian
We passed in a kubernetes variable:
Copy code
containers:
        - env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
and what your documentation says is a namespace variable:
Copy code
OTEL_RESOURCE_ATTRIBUTES=service.name=cfpb-service,service.namespace=docmagic-dev-torrance
after the comma. Based on one of these "tags" if you will, you could allow the user to customize their perspective, show the user a perspective of all apps within each namespace (<vertical>-<environment>-<site>, essentially the same thing as a context in Kubernetes parlance). The user should be able to see an aggregate view of their entire operationā€™s response times, each namespaceā€™s response times, a particular app cutting across namespaces response times, or any grouping of applications they desire. They should be able to see into the JVM, the heap, see the database calls, the external calls, and they should be able to correlate issues with other external factors.
p
You mean this for the metrics page right? So the services shown should be for a particular filter of k8s attributes e.g services which are on particular names space and a specific env etc. We are currently thinking of filters like this - you can do tag value filtering of this
b
Perfect, that's exactly what I meant. Any idea how soon this feature will be ready?
p
2-3 weeks
b
Awesome, I really like the simplicity and usability of your UI, your documentation is very polished, and the most amazing thing is, I can actually talk with the team members. Here's a window into why we need the namespace segmentation so badly:
Copy code
6 prod environments with 74 applications and services
-----------------------------------------------------
444

6 preprod environments with 74 applications and services
--------------------------------------------------------
444

1 dev environment with 74 applications and services
---------------------------------------------------
74

1 qa environment with 74 applications and services
--------------------------------------------------
74

1 stage environment with 74 applications and services
-----------------------------------------------------
74
-----------------------------------------------------
TOTAL: 1,110 (This includes k8s cron jobs, applications, and services.)
Nagios says our total number of services is 8,994.
p
Thanks @User Understand the need now. Will keep you posted. Just checking, this doesn't need to be tied to k8s namespace right? You primarily want a way to seggregate applications based in environments - correct?
b
Yes, that's true. As I mentioned earlier, we need to segregate by <vertical>-<environment>-<site>, i.e.., docmagic-prod-torrance, eps-prod-austin, etc... We could pass those values into the containers as they spawn in k8s, they don't have to be extracted from k8s itself. I had a couple really important questions for you, as I'm comparing your application to others in a report for my boss. Is there a way to change the refresh time for the dashboard in settings? Ideally, I'd like to be able to set it at any interval I want, 5 seconds, 2 seconds, 1 second, etc.... Is it possible to alert to anything but Slack as of yet? It would be nice if we could use email, pager duty, and I know this is a big ask, but Nagios too. I'm sure we could write a Python script to look for instances of errors, so long as we know how to dig into the logs or java code somehow.
One more question, if you do allow us to segregate our view of various groupings of apps and services, can you give an aggregate data point of the overall processing time for that group? i.e..., I could see from 6:00 AM PDT to 10:00 AM PDT, there 23,272 transactions with an average response time of 3.7 seconds for wf-preprod-torrance. That would be a case where we could look up a time range, but in our normal everyday view in the UI, it would be nice to see the aggregate transaction time for each grouping of apps.
I think the best way to do the application segregation would be to have a group membership design, so that each application, can be a member of several different groups. But when we want to bring up all assets for docmagic-prod-torrance, we can either click on a tab, or perhaps choose a site from a drop down, an environment from a drop down, and a vertical from a drop down. In other words, we build the groups and add the members, and then they appear as drop downs and or tabs in the UI. They don't necessarily need to be site, env, and vertical, they can be any logically grouping of applications that anybody chooses to make.
We make the groups like ā€œSiteā€, ā€œVerticalā€, ā€œEnvironmentā€, and then we start adding applications to them. They would then appear as drop downs in the UI. So if you wanted to see docmagic-prod-torrance applications, you choose those groups from the drop down lists, and then there should be a create button to allow you to set that combination of groups as a tab.
@User how's it going? I have wasted the last two days trying to get skywalking to work in our environment, and I want to let you know that I really appreciate your easy to understand, straight forward instructions about how to deploy signoz. I was wondering, have you guys made any progress on the new environment segmentation of apps?
p
hey @User, we are working on it. Though as I had mentioned earlier - it will take a couple of weeks to be shipped. You can add filters for the attributes you want to filter on
b
Yes, tags work perfect, like the usual POD_NAMESPACE in the deployment.yaml file of the application. We use overlays here with service.yaml files and kustomization.yaml files, and that's actually how the pod knows where it's running, i.e...,
Hi @Pranay I have spent nearly a week unsuccessfully trying to satisfy our business requirements with Skywalking, because they have a requirement that we must declare an agent service name from our entry point script, or we have to export variables inside our Dockerfile. The only solution that works for us so far is Signoz, because we can launch it by passing arguments from the command line to the container. Having that said, has your team made any progress with tagging? If you have, I can abandon Skywalking and go back to Signoz. Thanks.
p
Hey @Brian Bills We have made progress, but we want to do it right so that it enables generic use cases - we are enabling attributes from https://github.com/open-telemetry/opentelemetry-specification/tree/main/specification/resource/semantic_conventions#resource-semantic-conventions . Let me know if you want to get on a zoom call tomorrow or day after morning PT - where we can share what we are building, and get any feedback also. Though would like to understand what does your timeline look like? Do you have to make any internal presentation etc or any hard deadline. I think this would still need a couple of weeks to ship as we are a small team and working on few other features which users have asked like auth & login
b
Hi @Pranay that would be great if we could do a Zoom call. I can do anytime after 11:30 AM PDT if that works for you.
p
ok - please DM me your email ID. will send an invite for Thursday
b
Hi @Pranay I accepted the invite, but now it's not in my calendar.
Hi Pranay, I got your invite already and accepted it, plus I called into the meeting right now and the host hadn't started it.
p
For the benefit of others who might check this thread later, this feature is shipped now in this release - https://github.com/SigNoz/signoz/releases/tag/v0.8.0 Do check it out and let us know of any feedback
šŸ‘ 1
30 Views
Open in Slack
PreviousNext
Powered by