hey folks i am contemplating how the `timestamp` might have

Question

hey folks i am contemplating how the `timestamp` might have been corrupted somehow sweat smile ```Row 5 timestamp 16616360830611120685 observed timestamp 1669976312096499717 id 2IJmVr3fofdss3H3b6bL3YvjBKe trace id 234a5bcd543ef3fa53ce929d0e0e4736 span id b4b37b64bb3f6141 trace flags 0 severity text severity number 0 body log level info ts 1912 01 01T11 12 36 901569069+1100 msg hello error eeeeerrrrr trace id 234a5bcd543ef3fa53ce929d0e0e4736 span id b4b37b64bb3f6141 r n stream stdout time 2022 12 02T10 18 31 908222809Z resources string key resources string value attributes string key log file path timestamp attributes string value var log pods default echoer 9eb10131 9899 4112 9867 b1d6e0701d3a echoer 0 log 1912 01 01T11 12 36 901569069+1100 attributes int64 key attributes int64 value attributes float64 key attributes float64 value ``` repro ```kubectl run namespace default it rm restart=Never image=alpine echoer echo level info ts 1912 01 01T11 12 36 901569069+1100 msg hello error eeeeerrrrr trace id 234a5bcd543ef3fa53ce929d0e0e4736 span id b4b37b64bb3f6141 ``` cfg on log collector ``` receivers filelog containers exclude var log pods otelcol opentelemetry collector log include var log pods log include file name false include file path true operators id mslog regex ts P lt timestamp gt d 4 d 2 d 2 T d 2 d 2 d 2 d 1 20 +| d 2 d 2 trace id P lt trace id gt S span id P lt span id gt S timestamp layout Y m dT H M S f z parse from attributes timestamp trace span id parse from attributes span id trace id parse from attributes trace id type regex parser field attributes trace id type remove field attributes span id type remove``` also how to get `timestamp` on the frontend to show `1912 01 01T11 12 36 901569069+1100` pray

Ankit Nayan · Answer

cc < nitya signoz>

nitya-signoz · Answer

Any reason you wanted to parse the year 1912 Right now the timestamp we have set is Uint instead of int <https github com SigNoz signoz otel collector blob 921457ce79766889fe472d0fbf35a77def2390a3 exporter clickhouselogsexporter migrations 000001 init db up sql L3|https github com SigNoz signoz otel collector blob 921457ce79766889fe472d0fbf35a xporter clickhouselogsexporter migrations 000001 init db up sql> Because of this I guess this issue is occurring

Ishwor Gurung · Answer

thanks < nitya signoz> < Ankit Nayan> i guess this means i can t predate UNIX epoch i am not sure it ll be problematic thank you for pointing it out

Ishwor Gurung · Answer

gt Any reason you wanted to parse the year 1912 just testing

Ishwor Gurung · Answer

< nitya signoz> if i have current scenario where log items are missing in clickhouse what would be a good place to start debugging i see that it has left the log collector s cluster but checking `signoz logs logs` i do not see the log entry on log source side ```otc opentelemetry collector agent 9swbc opentelemetry collector 2022 12 02T14 08 44 292Z info fileconsumer file go 159 Started watching file kind receiver name filelog microservice+json pipeline logs component fileconsumer path var log pods default echoer 44add812 1732 4167 8b6f 269945e129a2 echoer 0 log otc opentelemetry collector agent 9swbc opentelemetry collector 2022 12 02T14 08 44 407Z info LogsExporter kind exporter data type logs name logging logs 1 otc opentelemetry collector agent 9swbc opentelemetry collector 2022 12 02T14 08 44 407Z info ResourceLog 0 otc opentelemetry collector agent 9swbc opentelemetry collector Resource SchemaURL otc opentelemetry collector agent 9swbc opentelemetry collector ScopeLogs 0 otc opentelemetry collector agent 9swbc opentelemetry collector ScopeLogs SchemaURL otc opentelemetry collector agent 9swbc opentelemetry collector InstrumentationScope otc opentelemetry collector agent 9swbc opentelemetry collector LogRecord 0 otc opentelemetry collector agent 9swbc opentelemetry collector ObservedTimestamp 2022 12 02 14 08 44 292300155 +0000 UTC otc opentelemetry collector agent 9swbc opentelemetry collector Timestamp 2004 01 01 00 12 36 901569069 +0000 UTC otc opentelemetry collector agent 9swbc opentelemetry collector SeverityText otc opentelemetry collector agent 9swbc opentelemetry collector SeverityNumber Unspecified 0 otc opentelemetry collector agent 9swbc opentelemetry collector Body Str log level info ts 2004 01 01T11 12 36 901569069+1100 msg hello error eeeeerrrrr trace id 234a5bcd543ef3fa53ce929d0e0e4736 span id b4b37b64bb3f6141 r n stream stdout time 2022 12 02T14 08 44 07928492Z otc opentelemetry collector agent 9swbc opentelemetry collector Attributes otc opentelemetry collector agent 9swbc opentelemetry collector gt log file path Str var log pods default echoer 44add812 1732 4167 8b6f 269945e129a2 echoer 0 log otc opentelemetry collector agent 9swbc opentelemetry collector gt timestamp Str 2004 01 01T11 12 36 901569069+1100 otc opentelemetry collector agent 9swbc opentelemetry collector Trace ID 234a5bcd543ef3fa53ce929d0e0e4736 otc opentelemetry collector agent 9swbc opentelemetry collector Span ID b4b37b64bb3f6141 otc opentelemetry collector agent 9swbc opentelemetry collector Flags 0 otc opentelemetry collector agent 9swbc opentelemetry collector kind exporter data type logs name logging otc opentelemetry collector agent sbj9b opentelemetry collector 2022 12 02T14 08 44 449Z info LogsExporter kind exporter data type logs name logging logs 1 ``` on clickhouse there are no new records ```select body from signoz logs logs where log file path like echoer and trace id = limit 10 G 7 rows in set Elapsed 0 043 sec Processed 2 98 million rows 30 60 MB 69 01 million rows s 708 10 MB s ``` any idea

Ishwor Gurung · Answer

gt i see that it has left the log collector s cluster or at least I think it does signify the debug log that it has left the source collector s cluster sweat smile

Ishwor Gurung · Answer

If I replay the log source restart collector it has start beginning set to true it does reach clickhouse

nitya-signoz · Answer

Interestingly if anything fails to export then ideally it will be logged in the otel collector errors Can t think of a case right now but it will be helpful if you can share the configs will get a better idea Or if it s for a specific kind of log then we can try to reproduce with only that kind of log

Ishwor Gurung · Answer

sure

Ishwor Gurung · Answer

straight from the configmap ```apiVersion v1 data relay | exporters logging loglevel debug otlp signoz endpoint 10 248 19 100 4317 tls insecure true extensions health check memory ballast size in percentage 40 processors batch k8sattributes extract metadata k8s namespace name k8s deployment name k8s statefulset name k8s daemonset name k8s cronjob name k8s job name passthrough false pod association sources from resource attribute name k8s pod ip sources from resource attribute name k8s pod uid sources from connection memory limiter check interval 5s limit mib 700 receivers filelog kubernetes exclude var log pods otelcol opentelemetry collector log var log pods default log include var log pods log include file name false include file path true operators id get format routes expr body matches ^ output parser docker expr body matches ^ ^ Z + output parser crio expr body matches ^ ^ Z +Z output parser containerd type router id parser crio output extract metadata from filepath regex ^ P lt time gt ^ Z + P lt stream gt stdout|stderr P lt logtag gt ^ P lt log gt $ timestamp layout 2006 01 02T15 04 05 000000000 07 00 layout type gotime parse from attributes time type regex parser id parser containerd output extract metadata from filepath regex ^ P lt time gt ^ ^Z +Z P lt stream gt stdout|stderr P lt logtag gt ^ P lt log gt $ timestamp layout Y m dT H M S LZ parse from attributes time type regex parser id parser docker output extract metadata from filepath timestamp layout Y m dT H M S LZ parse from attributes time type json parser id extract metadata from filepath parse from attributes log file path regex ^ P lt namespace gt ^ + P lt pod name gt ^ + P lt uid gt a f0 9 + P lt container name gt ^ + P lt restart count gt d+ log$ type regex parser from attributes stream to attributes log iostream type move from attributes container name to resource k8s container name type move from attributes namespace to resource k8s namespace name type move from attributes pod name to resource k8s pod name type move from attributes restart count to resource k8s container restart count type move from attributes uid to resource k8s pod uid type move from attributes log to body type move start at beginning filelog microservice+json exclude var log pods otelcol opentelemetry collector log include var log pods default log include file name false include file path true operators id parser microservice on error send regex ts P lt timestamp gt d 4 d 2 d 2 T d 2 d 2 d 2 d 1 20 +| d 2 d 2 trace id P lt trace id gt S span id P lt span id gt S timestamp layout Y m dT H M S f z parse from attributes timestamp trace span id parse from attributes span id trace id parse from attributes trace id type regex parser field attributes trace id type remove field attributes span id type remove start at beginning jaeger protocols grpc endpoint 0 0 0 0 14250 thrift compact endpoint 0 0 0 0 6831 thrift http endpoint 0 0 0 0 14268 otlp protocols grpc endpoint 0 0 0 0 4317 http endpoint 0 0 0 0 4318 prometheus config scrape configs job name opentelemetry collector scrape interval 10s static configs targets $ MY POD IP 8888 zipkin endpoint 0 0 0 0 9411 service extensions health check pipelines logs exporters logging otlp signoz processors k8sattributes memory limiter batch receivers filelog microservice+json filelog kubernetes metrics exporters otlp signoz processors k8sattributes memory limiter batch receivers otlp prometheus traces exporters logging processors k8sattributes memory limiter batch receivers otlp jaeger zipkin telemetry metrics address 0 0 0 0 8888 kind ConfigMap```

Ishwor Gurung · Answer

gt Interestingly if anything fails to export then ideally it will be logged in the otel collector errors yap have not seen any errors in the otel log collectors

Ishwor Gurung · Answer

i can try sniffing on the hops to `10 248 19 100 4317` but i thought there might be a meta table in clickhouse to check if records ever made it is there a compaction going on btw

Ishwor Gurung · Answer

if there is even such thing in clickhouse

Ishwor Gurung · Answer

even if there is i wouldn t know clickhouse logs show it is chugging along rather well sweat smile

Ishwor Gurung · Answer

gt Or if it s for a specific kind of log then we can try to reproduce with only that kind of log here it is ```kubectl run namespace us it rm restart=Never image=alpine echoer echo level info ts 2022 09 01T11 12 36 901569069+1100 msg hello error eeeeerrrrr trace id 234a5bcd543ef3fa53ce929d0e0e4736 span id b4b37b64bb3f6141 ```

nitya-signoz · Answer

The config files seem to be all good And also here all your logs have the same timestamp as seen on the regex which won t cause any issues Also how did you verify that some logs are missing You can also use a log generator that we have for testing purposes we have used it here

nitya-signoz · Answer

Have tried to replicate this locally but seems all the logs are getting ingested for me Tried using a script which generates logs with a counter to check ordering as well as missing values but all seems to be ingested correctly ```package main import fmt os time log <http github com sirupsen logrus|github com sirupsen logrus> func main customFormatter = amp log JSONFormatter customFormatter TimestampFormat = Mon 02 Jan 2006 15 04 05 MST f err = os OpenFile testlogrus log os O APPEND|os O CREATE|os O RDWR 0666 if err = nil fmt Printf error opening file v err defer f Close log SetOutput f log SetFormatter customFormatter for time Sleep 10 time Millisecond log WithFields log Fields status hello name xyz Info This is a log message + fmt Sprint i ```