hi what is the sampling_percentage by default in signoz ?
u
hi what is the sampling_percentage by default in signoz ?
@nitya-signoz can you help here, i am only seeing about 10% of the traces in signoz than my earlier setup on opentelemetry
n
SigNoz doesn’t perform any kind of sampling by default.
u
i can see the diff from the logs , i am hitting 100 requests from a script, in my logs collecteed by promtail then sent to loki reciever i am getting 100 requests from there, but in signoz only 46 of them show up in the http api dashboard or the time series graph or even the total count is incorrect, does the collector not queue them before exporting for cases where a lot of request come in ??
traces are getting dropped.
n
Can yo try using the logging exporter https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/loggingexporter/README.md in your collector config to see if all the data is being received ?
u
hi, have enabled the logging exporter , i started another 200 requests in one second, and i can see in my logs that i have 200 requests being made. in traces, i am getting 34 from the logging exporter logs also i can see only 34 of them. still dropping a lot of traces, my signoz is setup on a vm and application is on k8s.
s
Your application is dropping the data not signoz .
u
but i have given more than sufficient resources to my pods, its a node application and its currenlty having 2gb mem request configured, i have monitored the otel collectered and cant see any dropped spans, how to debug this ?
s
What language and sdk are you using?
u
Nodejs & opentelemetry/sdk-node, this is my tracing.js
Copy code
'use strict';
Object.defineProperty(exports, "__esModule", { value: true });
const process = require('process');
const opentelemetry = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { CompositePropagator } = require('@opentelemetry/core');
const { B3Propagator, B3InjectEncoding } = require('@opentelemetry/propagator-b3');
const api = require('@opentelemetry/api');
const exporterOptions = {
    url: '<endpoint>'
};
const traceExporter = new OTLPTraceExporter(exporterOptions);
const sdk = new opentelemetry.NodeSDK({
    traceExporter,
    instrumentations: [getNodeAutoInstrumentations()],
    resource: new Resource({
        [SemanticResourceAttributes.SERVICE_NAME]: 'kyc-app'
    }),
});
// initialize the SDK and register with the OpenTelemetry API
// this enables the API to record telemetry
sdk.start();
// <http://logger.info|logger.info>("======Started Tracing======")
api.propagation.setGlobalPropagator(new CompositePropagator({
    propagators: [new B3Propagator(), new B3Propagator({ injectEncoding: B3InjectEncoding.MULTI_HEADER })]
}));
// gracefully shut down the SDK on process exit
process.on('SIGTERM', () => {
    sdk.shutdown()
        .then(() => console.log('Tracing terminated'))
        .catch((error) => {
        if (error instanceof Error) {
            console.log('Error terminating tracing', error.message);
        }
        else {
            console.log('Error terminating tracing', error);
        }
    })
        .finally(() => process.exit(0));
});
exports.default = sdk;
//# sourceMappingURL=tracing.js.map
u
Copy code
No modules instrumentation has been defined, nothing will be patched
Copy code
Instrumentation suppressed, returning Noop Span
@opentelemetry/instrumentation-http http instrumentation outgoingRequest
@opentelemetry/instrumentation-http http.ClientRequest return request
@opentelemetry/instrumentation-http outgoingRequest on response()
@opentelemetry/instrumentation-http outgoingRequest on end()
statusCode: 200 {"partialSuccess":{}}
@opentelemetry/instrumentation-http outgoingRequest on request close()
getting this from debug logs