i have been trying to instrument a node js application using

Question

i have been trying to instrument a node js application using the guide in the docs i have a problem that once in a while impposible to work the trace exporter is crashing with the following error ```2022 09 23 06 44 57 http panic serving 172 21 0 1 60810 runtime error index out of range 18 with length 18 goroutine 608759 running net http conn serve func1 0x40002406e0 usr local go src net http server go 1802 +0xe4 panic 0x3b35880 0x400475f818 usr local go src runtime panic go 1052 +0x2b4 0x0 0x0 0x0 go pkg mod go opentelemetry io otel sdk v1 7 0 trace span go 398 +0x83c panic 0x3b35880 0x400475f818 usr local go src runtime panic go 1038 +0x224 src processor signozspanmetricsprocessor processor go 644 0x40000188c0 0x4005bc7590 0xf 0x40036c05b0 0x4005293320 src processor signozspanmetricsprocessor processor go 607 +0x6ec 0x4005293320 0x4005bc7590 0xf src processor signozspanmetricsprocessor processor go 502 +0x9c 0x4005863d88 src processor signozspanmetricsprocessor processor go 491 +0xdc 0x47e9bb8 0x4003f4c2d0 0x4005863d88 src processor signozspanmetricsprocessor processor go 279 +0x30 0x47e9bb8 0x400092b050 0x4005863d88 go pkg mod go opentelemetry io collector v0 55 0 receiver otlpreceiver internal trace otlp go 60 +0xcc 0x4005293260 0x4005d89900 0x40001976f8 0x484e850 0x6e41ce0 go pkg mod go opentelemetry io collector v0 55 0 receiver otlpreceiver otlphttp go 47 +0xd0 0x4005293260 0x4005d89900 go pkg mod go opentelemetry io collector v0 55 0 receiver otlpreceiver otlp go 189 +0x118 net http HandlerFunc ServeHTTP 0x40007640e0 0x47bb730 0x4005293260 0x4005d89900 usr local go src net http server go 2047 +0x40 net http ServeMux ServeHTTP 0x4000758180 0x47bb730 0x4005293260 0x4005d89900 usr local go src net http server go 2425 +0x18c 0x47bb730 0x4005293260 0x4005d89900 go pkg mod go opentelemetry io collector v0 55 0 config confighttp compression go 160 +0x174 net http HandlerFunc ServeHTTP 0x4000d5a140 0x47bb730 0x4005293260 0x4005d89900 usr local go src net http server go 2047 +0x40 0x47b9d20 0x4005bd35e0 0x4005d89400 go pkg mod go opentelemetry io contrib instrumentation net http otelhttp v0 32 0 handler go 191 +0xd0c 0x47b9d20 0x4005bd35e0 0x4004c9cc00 go pkg mod go opentelemetry io collector v0 55 0 config confighttp clientinfohandler go 39 +0x26c net http serverHandler ServeHTTP 0x40002fb500 0x47b9d20 0x4005bd35e0 0x4004c9cc00 usr local go src net http server go 2879 +0x45c net http conn serve 0x40002406e0 0x47e9bb8 0x4000747140 usr local go src net http server go 1930 +0xb54 created by net http Server Serve usr local go src net http server go 3034 +0x4ac```

Itai Gendler · Answer

how can i overcome this is there a different way i can export the metrics so overcome this

Itai Gendler · Answer

< Ankit Nayan>

Srikanth Chekuri · Answer

Can you share some sample traces which are causing this This is very unlikely unless the span timestamps are so distant the difference overflows the range

Itai Gendler · Answer

what exactly do you want me to send and how to get the data

Itai Gendler · Answer

i can send the output from the console from debug mode

Srikanth Chekuri · Answer

Does it happen frequently gt i can send the output from the console from debug mode Yes use the console exporter and share the sample traces output that trigger this error

Itai Gendler · Answer

it happens very frequently

Itai Gendler · Answer

i was just able to reproduce it

Itai Gendler · Answer

i see first this output `Inconsistent start and end time startTime gt endTime 1663917517 902433872 1663917517 901867218 `

Itai Gendler · Answer

and then entire log

Itai Gendler · Answer

```Inconsistent start and end time startTime gt endTime 1663917517 902433872 1663917517 901867218 HTTP 69748 outgoing message end HTTP 69748 SERVER socketOnParserExecute 834 items to be sent Span attributes http route express name query express type middleware links events status code 0 endTime 1663917517 901403189 ended true duration 0 63420 clock AnchoredClock monotonicClock Performance EventTarget epochMillis 1663917517901 performanceMillis 74538 869541049 name middleware query spanContext traceId 427ae950a2d2a411759a652a2dce77f5 spanId 28bb4ae1410dd688 traceFlags 1 traceState undefined parentSpanId 024647d0e876b128 kind 0 startTime 1663917517 901339769 resource Resource attributes Object instrumentationLibrary name opentelemetry instrumentation express version 0 31 1 schemaUrl undefined spanLimits attributeValueLengthLimit Infinity attributeCountLimit 128 linkCountLimit 128 eventCountLimit 128 spanProcessor MultiSpanProcessor spanProcessors Array attributeValueLengthLimit Infinity Span attributes http route express name expressInit express type middleware links events status code 0 endTime 1663917517 901846647 ended true duration 0 235080 clock AnchoredClock monotonicClock Performance EventTarget epochMillis 1663917517901 performanceMillis 74538 869541049 name middleware expressInit spanContext traceId 427ae950a2d2a411759a652a2dce77f5 spanId 8e8227ce9c135ca7 traceFlags 1 traceState undefined parentSpanId 024647d0e876b128 kind 0 startTime 1663917517 901611567 resource Resource attributes Object instrumentationLibrary name opentelemetry instrumentation express version 0 31 1 schemaUrl undefined spanLimits attributeValueLengthLimit Infinity attributeCountLimit 128 linkCountLimit 128 eventCountLimit 128 spanProcessor MultiSpanProcessor spanProcessors Array attributeValueLengthLimit Infinity Span attributes http route express name lt anonymous gt express type middleware links events status code 0 endTime 1663917517 902332067 ended true duration 0 353574 clock AnchoredClock monotonicClock Performance EventTarget epochMillis 1663917517901 performanceMillis 74538 869541049 name middleware lt anonymous gt spanContext traceId 427ae950a2d2a411759a652a2dce77f5 spanId b2f25d407ff996a2 traceFlags 1 traceState undefined parentSpanId 024647d0e876b128 kind 0 startTime 1663917517 901978493 resource Resource attributes Object instrumentationLibrary name opentelemetry instrumentation express version 0 31 1 schemaUrl undefined spanLimits attributeValueLengthLimit Infinity attributeCountLimit 128 linkCountLimit 128 eventCountLimit 128 spanProcessor MultiSpanProcessor spanProcessors Array attributeValueLengthLimit Infinity Span attributes http route express name express type request handler links events status code 0 endTime 1663917517 901867218 ended true duration 1 999433346 clock AnchoredClock monotonicClock Performance EventTarget epochMillis 1663917517901 performanceMillis 74538 869541049 name request handler spanContext traceId 427ae950a2d2a411759a652a2dce77f5 spanId af5eaec43dcd81da traceFlags 1 traceState undefined parentSpanId 024647d0e876b128 kind 0 startTime 1663917517 902433872 resource Resource attributes Object instrumentationLibrary name opentelemetry instrumentation express version 0 31 1 schemaUrl undefined spanLimits attributeValueLengthLimit Infinity attributeCountLimit 128 linkCountLimit 128 eventCountLimit 128 spanProcessor MultiSpanProcessor spanProcessors Array attributeValueLengthLimit Infinity Span attributes http url <http localhost 8080 > http host localhost 8080 net host name localhost http method GET http target http user agent Mozilla 5 0 Macintosh Intel Mac OS X 10 15 7 AppleWebKit 537 36 KHTML like Gecko Chrome 105 0 0 0 Safari 537 36 http flavor 1 1 net transport ip tcp attribute1 value1 net host ip 1 net host port 8080 net peer ip 1 net peer port 50573 http status code 304 http status text NOT MODIFIED http route links events Object Object status code 0 endTime 1663917517 904494047 ended true duration 0 3440380 clock AnchoredClock monotonicClock Performance EventTarget epochMillis 1663917517901 performanceMillis 74538 869541049 name GET spanContext traceId 427ae950a2d2a411759a652a2dce77f5 spanId 024647d0e876b128 traceFlags 1 traceState undefined parentSpanId undefined kind 1 startTime 1663917517 901053667 resource Resource attributes Object instrumentationLibrary name opentelemetry instrumentation http version 0 32 0 schemaUrl undefined spanLimits attributeValueLengthLimit Infinity attributeCountLimit 128 linkCountLimit 128 eventCountLimit 128 spanProcessor MultiSpanProcessor spanProcessors Array attributeValueLengthLimit Infinity Instrumentation suppressed returning Noop Span HTTP 69748 have free socket opentelemetry instrumentation http http instrumentation outgoingRequest opentelemetry instrumentation http http ClientRequest return request HTTP 69748 write ret = true HTTP 69748 outgoing message end HTTP 69748 server socket close opentelemetry instrumentation http outgoingRequest on request error Error socket hang up at connResetException node internal errors 693 14 at Socket socketOnEnd node http client 486 23 at Socket emit node events 539 35 at endReadableNT node internal streams readable 1344 12 at process processTicksAndRejections node internal process task queues 82 21 code ECONNRESET stack Error socket hang up n at connResetException node internal errors 693 14 n at Socket socketOnEnd node http client 486 23 n at Socket emit node events 539 35 n at endReadableNT node internal streams readable 1344 12 n at process processTicksAndRejections node internal process task queues 82 21 message socket hang up code ECONNRESET name Error HTTP 69748 CLIENT socket onClose HTTP 69748 removeSocket localhost 4318 writable false HTTP 69748 HTTP socket close opentelemetry instrumentation```

Itai Gendler · Answer

this includes both dialog log level and node js http debug output

Srikanth Chekuri · Answer

That confirms it I have seen this bug earlier with node js SDK

Srikanth Chekuri · Answer

Which version of node are you using

Itai Gendler · Answer

in this case 18 3 0

Itai Gendler · Answer

or you are talkinb about the sdk version not the node js version

Itai Gendler · Answer

i m using the latest ` opentelemetry sdk node` which is `0 33 0`

Srikanth Chekuri · Answer

If you see the output there are negative durations in it This clock dirft was a known issue and should have been fixed in `0 33 0`

Itai Gendler · Answer

can you maybe point me to the github issue related to that pr so i ll post it there

Srikanth Chekuri · Answer

Srikanth Chekuri · Answer

but I think we will handle this gracefully on our collector side as well Is this a blocker for you

Itai Gendler · Answer

preety much

Itai Gendler · Answer

unless there is a different way to pass the traces to signoz

Itai Gendler · Answer

maybe through the collector pattern if you can point me

Srikanth Chekuri · Answer

No it s data problem that should be handled We will do a patch release with the fix Should be available in an hour or two

Itai Gendler · Answer

ty but isn t it possible to use the collector pattern with signoz instead of the built in collector what s the difference

Srikanth Chekuri · Answer

Not sure I understood what you mean by use the collector pattern with signoz instead of the built in collector but we maintain a separate distribution with some customisation to work well with our backend and for generating the APM metrics generated using traces which are very helpful to give the high level service health

Itai Gendler · Answer

ok

Srikanth Chekuri · Answer

< Itai Gendler> can you update the image from `signoz signoz otel collector 0 55 1` to `signoz signoz otel collector 0 55 2` in your deployment and try again

Itai Gendler · Answer

will try soon