Hi. I have been trying to instrument a python app ...
# support
s
Hi. I have been trying to instrument a python app to ship logs to SigNoz by following this docs page. I had to make a custom processor for structlog since OpenTelemetry only provided a handler for the standard
logging
library. I think the logs are being sent correctly - there are no errors in the output and my custom fields show up under "INTERESTING FIELDS" in the Signoz logging dashboard. But the actual logs don't show up. How do I go about debugging this? Additional info: • Docker swarm deployment but on single node • gRPC exporter • SigNoz v0.14.0
s
What is your structlog handler code? I would check the timestamp conversion to begin with.
p
check this as well if you have not already -https://signoz.io/docs/userguide/logs_fields/#creating-log-fields
s
Here's what I'm using for timestamp:
Copy code
timestamp = int(
            datetime.fromisoformat(event_dict["timestamp"][:-1]).timestamp() * 1e9
        )
It's a bit hacky because structlog appends a
Z
to timestamps. Here's an example value:
1674806603174459904
The code needs some cleanup but here's the custom processor I'm using:
Copy code
from copy import deepcopy
from datetime import datetime

import structlog
from opentelemetry._logs import get_logger_provider, set_logger_provider, std_to_otel
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
from opentelemetry.sdk._logs import LoggerProvider, LogRecord
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor

# from opentelemetry.sdk._logs.severity import SeverityNumber, std_to_otlp
from opentelemetry.sdk.resources import Resource
from opentelemetry.trace import format_span_id, format_trace_id, get_current_span


class OpenTelemetryExporter:
    """A handler class which writes logging records, in OTLP format, to
    a network destination or file. Supports signals from the `logging` module.
    <https://docs.python.org/3/library/logging.html>
    """

    def __init__(
        self,
        service_name: str,
        server_hostname: str,
    ) -> None:
        logger_provider = LoggerProvider(
            resource=Resource.create(
                {
                    "service.name": service_name,
                    "service.instance.id": server_hostname,
                }
            ),
        )
        # set_logger_provider(logger_provider)

        exporter = OTLPLogExporter(endpoint="localhost:4317", insecure=True)
        logger_provider.add_log_record_processor(
            BatchLogRecordProcessor(exporter, max_export_batch_size=1)
        )

        self._logger_provider = logger_provider
        self._logger = logger_provider.get_logger(__name__)

    def _translate(self, event_dict: structlog.typing.EventDict) -> LogRecord:
        timestamp = int(
            datetime.fromisoformat(event_dict["timestamp"][:-1]).timestamp() * 1e9
        )
        span_context = get_current_span().get_span_context()
        # attributes = self._get_attributes(record)
        severity_number = std_to_otel(
            structlog.stdlib._NAME_TO_LEVEL[event_dict["level"]]
        )
        print(timestamp, event_dict)
        return LogRecord(
            timestamp=timestamp,
            trace_id=span_context.trace_id,
            span_id=span_context.span_id,
            trace_flags=span_context.trace_flags,
            severity_text=event_dict["level"],
            severity_number=severity_number,
            body=event_dict["event"],
            resource=self._logger.resource,
            attributes=event_dict,
        )

    def __call__(
        self,
        logger: structlog.typing.WrappedLogger,
        name: str,
        event_dict: structlog.typing.EventDict,
    ):
        """
        Emit a record.

        The record is translated to OTLP format, and then sent across the pipeline.
        """
        self._logger.emit(self._translate(deepcopy(event_dict)))

        return event_dict
I think I found the issue
It was an issue in timestamp like you said. The conversion from str to datetime was assuming the local timezone instead of UTC. Once I fixed that, the logs are showing up correctly. Thank you for your help!
s
Please help us by contributing structlog handler here https://github.com/open-telemetry/opentelemetry-python/issues/2993.
s
I came across the issue too. Will definitely contribute once I clean it up.
There is another issue, it seems to be because of
structlog.processors.TimeStamper
. The timestamp it adds is not a valid ISO timestamp, so Signoz shows
Invalid Date
even though it definitely knows the timestamp (because the time filter -
Last 1 hour
, etc. works)
s
That’s probably a bug; please create an issue with where example where it breaks.
s
319 Views