https://signoz.io logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
i
how can i overcome this? is there a different way i can export the metrics so overcome this?
@Ankit Nayan
s
Can you share some sample traces which are causing this? This is very unlikely unless the span timestamps are so distant the difference overflows the range.
i
what exactly do you want me to send and how to get the data?
i can send the output from the console from debug mode
s
Does it happen frequently?
i can send the output from the console from debug mode
Yes, use the console exporter and share the sample traces output that trigger this error.
i
it happens very frequently
i was just able to reproduce it
i see first this output 
Inconsistent start and end time, startTime > endTime [ 1663917517, 902433872 ] [ 1663917517, 901867218 ]
and then entire log
Copy code
Inconsistent start and end time, startTime > endTime [ 1663917517, 902433872 ] [ 1663917517, 901867218 ]
HTTP 69748: outgoing message end.
HTTP 69748: SERVER socketOnParserExecute 834
items to be sent [
  Span {
    attributes: {
      'http.route': '/',
      'express.name': 'query',
      'express.type': 'middleware'
    },
    links: [],
    events: [],
    status: { code: 0 },
    endTime: [ 1663917517, 901403189 ],
    _ended: true,
    _duration: [ 0, 63420 ],
    _clock: AnchoredClock {
      _monotonicClock: [Performance [EventTarget]],
      _epochMillis: 1663917517901,
      _performanceMillis: 74538.869541049
    },
    name: 'middleware - query',
    _spanContext: {
      traceId: '427ae950a2d2a411759a652a2dce77f5',
      spanId: '28bb4ae1410dd688',
      traceFlags: 1,
      traceState: undefined
    },
    parentSpanId: '024647d0e876b128',
    kind: 0,
    startTime: [ 1663917517, 901339769 ],
    resource: Resource { attributes: [Object] },
    instrumentationLibrary: {
      name: '@opentelemetry/instrumentation-express',
      version: '0.31.1',
      schemaUrl: undefined
    },
    _spanLimits: {
      attributeValueLengthLimit: Infinity,
      attributeCountLimit: 128,
      linkCountLimit: 128,
      eventCountLimit: 128
    },
    _spanProcessor: MultiSpanProcessor { _spanProcessors: [Array] },
    _attributeValueLengthLimit: Infinity
  },
  Span {
    attributes: {
      'http.route': '/',
      'express.name': 'expressInit',
      'express.type': 'middleware'
    },
    links: [],
    events: [],
    status: { code: 0 },
    endTime: [ 1663917517, 901846647 ],
    _ended: true,
    _duration: [ 0, 235080 ],
    _clock: AnchoredClock {
      _monotonicClock: [Performance [EventTarget]],
      _epochMillis: 1663917517901,
      _performanceMillis: 74538.869541049
    },
    name: 'middleware - expressInit',
    _spanContext: {
      traceId: '427ae950a2d2a411759a652a2dce77f5',
      spanId: '8e8227ce9c135ca7',
      traceFlags: 1,
      traceState: undefined
    },
    parentSpanId: '024647d0e876b128',
    kind: 0,
    startTime: [ 1663917517, 901611567 ],
    resource: Resource { attributes: [Object] },
    instrumentationLibrary: {
      name: '@opentelemetry/instrumentation-express',
      version: '0.31.1',
      schemaUrl: undefined
    },
    _spanLimits: {
      attributeValueLengthLimit: Infinity,
      attributeCountLimit: 128,
      linkCountLimit: 128,
      eventCountLimit: 128
    },
    _spanProcessor: MultiSpanProcessor { _spanProcessors: [Array] },
    _attributeValueLengthLimit: Infinity
  },
  Span {
    attributes: {
      'http.route': '/',
      'express.name': '<anonymous>',
      'express.type': 'middleware'
    },
    links: [],
    events: [],
    status: { code: 0 },
    endTime: [ 1663917517, 902332067 ],
    _ended: true,
    _duration: [ 0, 353574 ],
    _clock: AnchoredClock {
      _monotonicClock: [Performance [EventTarget]],
      _epochMillis: 1663917517901,
      _performanceMillis: 74538.869541049
    },
    name: 'middleware - <anonymous>',
    _spanContext: {
      traceId: '427ae950a2d2a411759a652a2dce77f5',
      spanId: 'b2f25d407ff996a2',
      traceFlags: 1,
      traceState: undefined
    },
    parentSpanId: '024647d0e876b128',
    kind: 0,
    startTime: [ 1663917517, 901978493 ],
    resource: Resource { attributes: [Object] },
    instrumentationLibrary: {
      name: '@opentelemetry/instrumentation-express',
      version: '0.31.1',
      schemaUrl: undefined
    },
    _spanLimits: {
      attributeValueLengthLimit: Infinity,
      attributeCountLimit: 128,
      linkCountLimit: 128,
      eventCountLimit: 128
    },
    _spanProcessor: MultiSpanProcessor { _spanProcessors: [Array] },
    _attributeValueLengthLimit: Infinity
  },
  Span {
    attributes: {
      'http.route': '/',
      'express.name': '/',
      'express.type': 'request_handler'
    },
    links: [],
    events: [],
    status: { code: 0 },
    endTime: [ 1663917517, 901867218 ],
    _ended: true,
    _duration: [ -1, 999433346 ],
    _clock: AnchoredClock {
      _monotonicClock: [Performance [EventTarget]],
      _epochMillis: 1663917517901,
      _performanceMillis: 74538.869541049
    },
    name: 'request handler - /',
    _spanContext: {
      traceId: '427ae950a2d2a411759a652a2dce77f5',
      spanId: 'af5eaec43dcd81da',
      traceFlags: 1,
      traceState: undefined
    },
    parentSpanId: '024647d0e876b128',
    kind: 0,
    startTime: [ 1663917517, 902433872 ],
    resource: Resource { attributes: [Object] },
    instrumentationLibrary: {
      name: '@opentelemetry/instrumentation-express',
      version: '0.31.1',
      schemaUrl: undefined
    },
    _spanLimits: {
      attributeValueLengthLimit: Infinity,
      attributeCountLimit: 128,
      linkCountLimit: 128,
      eventCountLimit: 128
    },
    _spanProcessor: MultiSpanProcessor { _spanProcessors: [Array] },
    _attributeValueLengthLimit: Infinity
  },
  Span {
    attributes: {
      'http.url': '<http://localhost:8080/>',
      'http.host': 'localhost:8080',
      'net.host.name': 'localhost',
      'http.method': 'GET',
      'http.target': '/',
      'http.user_agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36',
      'http.flavor': '1.1',
      'net.transport': 'ip_tcp',
      attribute1: 'value1',
      'net.host.ip': '::1',
      'net.host.port': 8080,
      'net.peer.ip': '::1',
      'net.peer.port': 50573,
      'http.status_code': 304,
      'http.status_text': 'NOT MODIFIED',
      'http.route': ''
    },
    links: [],
    events: [ [Object], [Object] ],
    status: { code: 0 },
    endTime: [ 1663917517, 904494047 ],
    _ended: true,
    _duration: [ 0, 3440380 ],
    _clock: AnchoredClock {
      _monotonicClock: [Performance [EventTarget]],
      _epochMillis: 1663917517901,
      _performanceMillis: 74538.869541049
    },
    name: 'GET /',
    _spanContext: {
      traceId: '427ae950a2d2a411759a652a2dce77f5',
      spanId: '024647d0e876b128',
      traceFlags: 1,
      traceState: undefined
    },
    parentSpanId: undefined,
    kind: 1,
    startTime: [ 1663917517, 901053667 ],
    resource: Resource { attributes: [Object] },
    instrumentationLibrary: {
      name: '@opentelemetry/instrumentation-http',
      version: '0.32.0',
      schemaUrl: undefined
    },
    _spanLimits: {
      attributeValueLengthLimit: Infinity,
      attributeCountLimit: 128,
      linkCountLimit: 128,
      eventCountLimit: 128
    },
    _spanProcessor: MultiSpanProcessor { _spanProcessors: [Array] },
    _attributeValueLengthLimit: Infinity
  }
]
Instrumentation suppressed, returning Noop Span
HTTP 69748: have free socket
@opentelemetry/instrumentation-http http instrumentation outgoingRequest
@opentelemetry/instrumentation-http http.ClientRequest return request
HTTP 69748: write ret = true
HTTP 69748: outgoing message end.
HTTP 69748: server socket close
@opentelemetry/instrumentation-http outgoingRequest on request error() Error: socket hang up
    at connResetException (node:internal/errors:693:14)
    at Socket.socketOnEnd (node:_http_client:486:23)
    at Socket.emit (node:events:539:35)
    at endReadableNT (node:internal/streams/readable:1344:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  code: 'ECONNRESET'
}
{"stack":"Error: socket hang up\n    at connResetException (node:internal/errors:693:14)\n    at Socket.socketOnEnd (node:_http_client:486:23)\n    at Socket.emit (node:events:539:35)\n    at endReadableNT (node:internal/streams/readable:1344:12)\n    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)","message":"socket hang up","code":"ECONNRESET","name":"Error"}
HTTP 69748: CLIENT socket onClose
HTTP 69748: removeSocket localhost:4318: writable: false
HTTP 69748: HTTP socket close
@opentelemetry/instrumentation
this includes both dialog log level and node.js http debug output
s
That confirms it. I have seen this bug earlier with node.js SDK.
Which version of node are you using?
i
in this case 18.3.0
or you are talkinb about the sdk version not the node.js version
i’m using the latest 
@opentelemetry/sdk-node
which is 
0.33.0
s
If you see the output there are negative durations in it. This clock dirft was a known issue and should have been fixed in 
0.33.0
.
i
can you maybe point me to the github issue related to that pr? so i’ll post it there?
s
but I think we will handle this gracefully on our collector side as well. Is this a blocker for you?
i
preety much…
unless there is a different way to pass the traces to signoz
maybe through the collector pattern? if you can point me
s
No, it’s data problem that should be handled. We will do a patch release with the fix. Should be available in an hour or two.
i
ty! but isn’t it possible to use the collector pattern with signoz instead of the built in collector? what’s the difference?
s
Not sure I understood what you mean by use the collector pattern with signoz instead of the built in collector but we maintain a separate distribution with some customisation to work well with our backend and for generating the APM metrics generated using traces which are very helpful to give the high level service health.
i
ok
s
@Itai Gendler can you update the image from 
signoz/signoz-otel-collector:0.55.1
to 
signoz/signoz-otel-collector:0.55.2
in your deployment and try again?
i
will try soon
5 Views
Open in Slack
PreviousNext
Powered by