Hello everyone 👋 Trying out SigNoz cloud, so I've signed up for trial and added the agents/collectors to our EKS development cluster. Logs aggregation and host/node metrics work out of the box, really great 💯 I have also tried to add a few custom metrics in a sample python app. Basically, counters that are incremented by one. I can see them in the dashboard, however the behaviour is a bit strange or I misunderstand something. For example, one of the counters has been incremented X times. If I select "Last one hour" it shows 23. If I choose "Last 6 hours", it shows 19. "Last 1 day" is 17 and "Last 3 days" is 12. That is the opposite of what I would expect 🤷 Number should go up with longer time range.

Were there any counter resets?

Not that I am aware of (when/how is counter reset?). I just followed the tutorial in the docs for the python metrics, sample snippet from the repl:

$ poetry run python
Python 3.9.5 (default, Mar 20 2023, 11:38:27)
[Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from opentelemetry.metrics import set_meter_provider, get_meter_provider
>>> from opentelemetry.sdk.metrics import MeterProvider
>>> from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
>>> from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
>>> reader = PeriodicExportingMetricReader(OTLPMetricExporter())
>>> set_meter_provider(MeterProvider(metric_readers=[reader]))
>>> from opentelemetry import metrics
>>> meter = metrics.get_meter("local.console")
>>> counter = meter.create_counter(name="test_tasks_completed", unit="1", description="The number of tasks completed")
>>> counter.add(1)
... (a number of counter.add() calls)

It is reset if a process restarts, use delta temporality

Thank you for explanation. How can I use delta temporality? In the dashboard panel I have two options in the metrics dropdown:

Increase

or

Rate

.

Please set the env

OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta

Thanks, I will try it out 🙂

@Jan Gnezda I'm trying to replicate the code snippet above. I'm adding metrics to a reference implementation and can't seem to get them to land in my local instance of SigNoz. Traces and logs were there. Metrics appear to be going to the bit bucket. Are you able to share a bit more of your code, setup, etc?

@Doug Ramirez There really isn't anything more to it than the snippet above. Have you set the env variables as described in the docs? I did the following to test the metrics:

OTEL_RESOURCE_ATTRIBUTES=service.name=test_console_local
OTEL_EXPORTER_OTLP_ENDPOINT="<https://ingest.eu.signoz.cloud:443>"
OTEL_EXPORTER_OTLP_HEADERS="signoz-ingestion-key=my_key"
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta

Replace

my_key

with a valid ingestion key.

Thanks, @Jan Gnezda. I've been doing this with a local instance. I'll try again with the hosted version and 🤞