<@U02SS3ZAMKQ> I am unable to see logs , though I ...
# support
a
@Prashant Shahi I am unable to see logs , though I can see services and traces successfully
The logs are empty
p
Logs of the containers in K8s cluster where signoz is installed is captured by
k8s-infra
chart.
If I remember correctly, you had it disabled as SigNoz is installed in dedicated K8s cluster and only getting signoz container logs there would not make sense.
Also, any instrumentation does not capture logs, but only traces and in case of few SDKs specific metrics.
@nitya-signoz would it possible to send logs via instrumentation from applications running in external K8s cluster or VMs? 🤔
n
Not all languages support OTEL logs SDK, right now python and Java are supported. If they are using a different language they will have to deploy the collector on the application instance.
s
Logging is not enable by default for some reason (more context https://github.com/open-telemetry/opentelemetry-python/issues/2594). You can enable it by setting
OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
but make sure you understand the potential issue with it from the linked issue.
a
I have done
Copy code
logging.basicConfig(level=<http://logging.INFO|logging.INFO>, force=True)  # Should happen before any logging

app = FastAPI(**opts)
app.add_middleware(LoggingMiddleware)
Copy code
class LoggingMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next):
        # Record the actual processing time for the request
        start_time = time.time()
        try:
            response = await call_next(request)
            response_status_code = response.status_code
            stats_json = get_stats_json(start_time, request, response_status_code)
        except ClientDisconnect:
            # We don't want to handle this and logs get cluttered
            stats_json = get_stats_json(start_time, request, status.HTTP_418_IM_A_TEAPOT)
            stats_json["error_reason"] = "ClientDisconnect"
            <http://logger.info|logger.info>(stats_json)
            response = JSONResponse(stats_json, status.HTTP_418_IM_A_TEAPOT)
        except Exception as ex:
            # Since this was unexpected exception
            stats_json = get_stats_json(start_time, request, status.HTTP_500_INTERNAL_SERVER_ERROR)
            json_response_body = {"detail": repr(ex)}
            response = JSONResponse(json_response_body, status.HTTP_500_INTERNAL_SERVER_ERROR)
            logger.error(json_response_body, exc_info=True)
        if request.url.path != "/":
            <http://logger.info|logger.info>(stats_json)
        return response
And udpated the environment variable as you said but I still don't see the logs beta getting populated
s
Remove this and let me know if works
Copy code
logging.basicConfig(level=<http://logging.INFO|logging.INFO>, force=True)
a
ok
It works but doesn't show on my local console anymore 😄
but yeah good enough
s
When you
force=True
it removes handler what’s automatically configure by
opentelemetry-instrument
. And any calls after that with
logging.basicConfig
will not be affective. That’s problem we are talking about here https://github.com/open-telemetry/opentelemetry-python/issues/2594#issuecomment-1099620061.
a
Okay it works on my docker signoz but it doesn't send logs on my jubernetes signoz
@Srikanth Chekuri can you help me getting it on the kubernetes aws one? I m not sure why its not coming there.. It comes on docker local one though
If it helps IDK why it says traces retention setting is failed
s
@Abhinav Ramana are you using
OTLPHandler + OTLPLogExporter
or something else?
a
not sure, but nothing special we use I believe
it also randomly stopped sending metricson Saturday, it was successfully working on Monday
can we sync ?
s
Sure