https://signoz.io logo
Join Slack
Powered by
This message was deleted.
# support
s
This message was deleted.
a
@User SigNoz can be set up as a centralized apm which can receive data from different clusters but it does not have multi-tenancy now. But can easily be achieved with the current architecture. Feel free to raise a github issue for this.
The current K8s deployment was tested to be able to receive 5K events/s with 16GB memory and 2 CPU unit.
The min setup needs around 5GB memory in K8s
The architecture is highly scalable and should be able to handle huge load on horizontal scaling
h
What is minimum setup in kubernetes ? Do we need to make changes in helm chart for that /
a
S3 needs to be configured as cold storage
h
Thank you Ankit... Let me try this
I have done setup in kubernetes with helm chart
I am trying with my application will try this cold storage setup as well
a
👍
h
In my Django pod I have set Environment variable - name: OTEL_EXPORTER_OTLP_ENDPOINT value: http://otel-collector.platform.svc.cluster.local:4317
application is accessible but traces not visible
same image was working in docker setup
a
How does your run command look like? Do you use 
runserver
?
h
CMD [ "opentelemetry-instrument", "gunicorn", "-c", "conf/gunicorn_conf.py", "hemantproject.wsgi:application" ]
from Dockerfile
- env: - name: OTEL_RESOURCE_ATTRIBUTES value: service.name=pythonApp - name: SERVICE_NAME value: pythonApp - name: OTEL_EXPORTER_OTLP_ENDPOINT value: http://otel-collector.platform.svc.cluster.local:4317
and above env variables in k8s deployment file
a
ok...did you use 
post_fork
?
h
yes
def post_fork(server, worker): server.log.info("Worker spawned (pid: %s)", worker.pid) server.log.info("OTEL_EXPORTER_OTLP_ENDPOINT: %s", str(os.environ['OTEL_EXPORTER_OTLP_ENDPOINT'])) resource = Resource.create(attributes={"service.name": str(os.environ['SERVICE_NAME'])}) trace.set_tracer_provider(TracerProvider(resource=resource)) # This uses insecure connection for the purpose of example. Please see the # OTLP Exporter documentation for other options. span_processor = BatchSpanProcessor( OTLPSpanExporter(endpoint=str(os.environ['OTEL_EXPORTER_OTLP_ENDPOINT']), insecure=True) ) trace.get_tracer_provider().add_span_processor(span_processor)
a
Is the application showing any relevant logs?
h
like this in Gunicorn conf py file
[2021-08-31 082848 +0000] [1] [INFO] Starting gunicorn 20.1.0 [2021-08-31 082848 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1) [2021-08-31 082848 +0000] [1] [INFO] Using worker: sync [2021-08-31 082848 +0000] [11] [INFO] Booting worker with pid: 11 [2021-08-31 082848 +0000] [11] [INFO] Worker spawned (pid: 11) [2021-08-31 082848 +0000] [11] [INFO] OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector.platform.svc.cluster.local:4317 Overriding of current TracerProvider is not allowed [2021-08-31 082848 +0000] [19] [INFO] Booting worker with pid: 19 [2021-08-31 082848 +0000] [19] [INFO] Worker spawned (pid: 19) [2021-08-31 082848 +0000] [19] [INFO] OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector.platform.svc.cluster.local:4317 Overriding of current TracerProvider is not allowed 172.17.0.1 - - [31/Aug/20210830:28 +0000] "GET / HTTP/1.1" 404 179 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36" 0.081231 <19> 0 81231 172.17.0.1 - - [31/Aug/20210830:28 +0000] "GET /favicon.ico HTTP/1.1" 404 179 "http://3.6.205.77:31418/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36" 0.069556 <11> 0 69556
these are logs... by any means can we see otel collector logs
a
yes..you need to run otel-collector with flag 
--log-level DEBUG
as mentioned in Link . You can also also see metrics emitted from otel-collector
I think you will need to add
Copy code
- "--log-level=DEBUG"
after this line and apply otel-collector yamls again
otel-collector also exports metrics at 8888 where you can see how many spans were received and exported
h
a
h
yes... with this I could see metrics exposed
a
what's the number for received spans?
h
Copy code
# HELP otelcol_processor_spans_received The number of spans received.
# TYPE otelcol_processor_spans_received counter
otelcol_processor_spans_received{processor="",service="",service_instance_id="af39ad64-d440-40e7-8054-4e2c75d8ada6"} 5
a
you are using below line in your 
post_fork
.
Copy code
resource = Resource.create(attributes={"service.name": str(os.environ['SERVICE_NAME'])})
So, you need to set service name using 
SERVICE_NAME
env variable
try passing this env variable from deployment.yaml
Also look for 
otelcol_receiver_received_spans
in otel-collector metrics
h
yes SERVICE_NAME is already defined in yaml... there is no otelcol_receiver_received_spans matrics
Copy code
otelcol_receiver_accepted_spans{receiver="otlp",service_instance_id="af39ad64-d440-40e7-8054-4e2c75d8ada6",transport="grpc"} 6
otelcol_receiver_refused_spans{receiver="otlp",service_instance_id="af39ad64-d440-40e7-8054-4e2c75d8ada6",transport="grpc"} 0
above only 2 otelcol matrics
a
it seems to be receiving spans
Copy code
otelcol_processor_spans_received{processor="",service="",service_instance_id="af39ad64-d440-40e7-8054-4e2c75d8ada6"}
service
seems to be empty and hence curious
what is the API response you get from 
<http://localhost:3000/api/v1/services/list>
?
use relevant host instead of 
localhost
h
Thank you Ankit... It is working now. I have created new image of application and passed ENV from deployment yaml
🎉 1
a
Hi @User, how is it going? Would be great to have some feedback on your usage.
h
Hi Ankit.... I have done initial setup and implemented auto instrumentation with open-telemetry in my django gunicorn application. Only issue is I am not able to see DB queries to postgres which is running as separate pod. Do we need to do any changes in postgres pod as well ?
a
I will have to check..Do you use django ORM for the db queries to postgres?
h
we are using psycopg to connect to db
a
which engine do you use in 
DATABASES
settings?
Copy code
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'dbname',
        'USER': 'postgres',
        'PASSWORD': 'postgres',
        'HOST': '',
        'PORT': '',
    }
}
OR
Copy code
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': 'db_name',                      
        'USER': 'db_user',
        'PASSWORD': 'db_user_password',
        'HOST': '',
        'PORT': 'db_port_number',
    }
}
18 Views
Open in Slack
PreviousNext
Powered by