This message was deleted.

This message was deleted.

This message was deleted.

Hi SigNoz community! I am running on a k3s Kubernetes cluster, with SigNoz deployed via the SigNoz Helm chart. I can get logs flowing in OK with the

filelog

reciever. However I see no metrics coming in. I am pretty sure that I have configured the

hostmetrics

reciever fine as a Daemonset. Any insights on what I missed would be greatly appreciated. Details in this thread.

How can SNMP collection be implemented with SigNoz? Do we need to use a collector other than OpenTelemetry to achieve this? Could an expert provide some guidance? Thank you.

This message was deleted.

Hello I keep seeing these error logs in the

signoz-otel-collector

pod

024-03-20T08:22:57.138Z	error	clickhousetracesexporter/writer.go:421	Could not write a batch of spans to index table: 	{"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "error": "clickhouse [Append]:  clickhouse: expected 37 arguments, got 36"}
<http://github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans|github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans>
	/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:421
<http://github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData|github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData>
	/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:424
<http://go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export|go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export>
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.88.0/exporterhelper/traces.go:60
<http://go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send|go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send>
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.88.0/exporterhelper/timeout_sender.go:41
<http://go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send|go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send>
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.88.0/exporterhelper/retry_sender.go:138
<http://go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send|go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send>
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.88.0/exporterhelper/traces.go:177
<http://go.opentelemetry.io/collector/exporter/exporterhelper.(*queueSender).start.func1|go.opentelemetry.io/collector/exporter/exporterhelper.(*queueSender).start.func1>
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.88.0/exporterhelper/queue_sender.go:126
<http://go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).Start.func1|go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).Start.func1>
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.88.0/exporterhelper/internal/bounded_memory_queue.go:52
2024-03-20T08:22:57.138Z	info	exporterhelper/retry_sender.go:177	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "error": "clickhouse [Append]:  clickhouse: expected 37 arguments, got 36", "interval": "8.596695142s"

What could cause this issue? Checked the clickhouse container utilisation, it's normal

After upgrading to the last version, i noted a massive number of the same error being throw in the logs from the clickhouse container:

signoz-clickhouse      | 2024.04.30 15:33:58.073145 [ 623 ] {} <Error> MergeTreeBackgroundExecutor: Exception while executing background task {1d6b00a2-62a1-426a-a406-f72d7dd97975::20240430_1_122_8}: Code: 117. DB::Exception: Invalid flag for negative store: (while reading from part /var/lib/clickhouse/store/1d6/1d6b00a2-62a1-426a-a406-f72d7dd97975/20240430_1_36_7/ in table signoz_metrics.exp_hist (1d6b00a2-62a1-426a-a406-f72d7dd97975) located on disk default of type local, from mark 0 with max_rows_to_read = 8192): While executing MergeTreeSequentialSource. (INCORRECT_DATA), Stack trace (when copying this message, always include the lines below):
signoz-clickhouse      | 
signoz-clickhouse      | 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in /usr/bin/clickhouse
signoz-clickhouse      | 1. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000721a243 in /usr/bin/clickhouse
signoz-clickhouse      | 2. DB::DDSketchDenseLogarithmic::deserialize(DB::ReadBuffer&) @ 0x000000000d5260e0 in /usr/bin/clickhouse
signoz-clickhouse      | 3. DB::SerializationAggregateFunction::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const @ 0x00000000108b58e9 in /usr/bin/clickhouse
signoz-clickhouse      | 4. DB::ISerialization::deserializeBinaryBulkWithMultipleStreams(COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, DB::ISerialization::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::unordered_map<String, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::hash<String>, std::equal_to<String>, std::allocator<std::pair<String const, COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>*) const @ 0x00000000108b01d9 in /usr/bin/clickhouse
signoz-clickhouse      | 5. DB::MergeTreeReaderCompact::readRows(unsigned long, unsigned long, bool, unsigned long, std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>&) @ 0x000000001250a6b9 in /usr/bin/clickhouse
signoz-clickhouse      | 6. DB::MergeTreeSequentialSource::generate() @ 0x000000001251ca4d in /usr/bin/clickhouse
signoz-clickhouse      | 7. DB::ISource::tryGenerate() @ 0x000000001297acf5 in /usr/bin/clickhouse
signoz-clickhouse      | 8. DB::ISource::work() @ 0x000000001297a743 in /usr/bin/clickhouse
signoz-clickhouse      | 9. DB::ExecutionThreadContext::executeTask() @ 0x000000001299371a in /usr/bin/clickhouse
signoz-clickhouse      | 10. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001298a170 in /usr/bin/clickhouse
signoz-clickhouse      | 11. DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x0000000012989928 in /usr/bin/clickhouse
signoz-clickhouse      | 12. DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x0000000012998017 in /usr/bin/clickhouse
signoz-clickhouse      | 13. DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x00000000129981d3 in /usr/bin/clickhouse
signoz-clickhouse      | 14. DB::MergeTask::ExecuteAndFinalizeHorizontalPart::executeImpl() @ 0x000000001233b6f2 in /usr/bin/clickhouse
signoz-clickhouse      | 15. DB::MergeTask::ExecuteAndFinalizeHorizontalPart::execute() @ 0x000000001233b64b in /usr/bin/clickhouse
signoz-clickhouse      | 16. DB::MergeTask::execute() @ 0x0000000012340d99 in /usr/bin/clickhouse
signoz-clickhouse      | 17. DB::MergePlainMergeTreeTask::executeStep() @ 0x0000000012723517 in /usr/bin/clickhouse
signoz-clickhouse      | 18. DB::MergeTreeBackgroundExecutor<DB::DynamicRuntimeQueue>::threadFunction() @ 0x00000000123532c4 in /usr/bin/clickhouse
signoz-clickhouse      | 19. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000c8eb0c1 in /usr/bin/clickhouse
signoz-clickhouse      | 20. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c8ee8fa in /usr/bin/clickhouse
signoz-clickhouse      | 21. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000c8ed6fe in /usr/bin/clickhouse
signoz-clickhouse      | 22. ? @ 0x00007f8adb8b3609
signoz-clickhouse      | 23. ? @ 0x00007f8adb7d8353
signoz-clickhouse      |  (version 24.1.2.5 (official build))

Hello, I have added a custom resource attribute to my traces via the sidecar collector:

resource/k8s:
        attributes:
          - key: test.foo
            value: bar
            action: insert
          - key: k8s.cluster.name
            from_attribute: k8s-cluster
            action: insert
          - key: k8s.namespace.name
            from_attribute: k8s-namespace
            action: insert

I can use that resource attribute to filter in the Traces area of Signoz, but the Services area will not show it as a filter option - only the signoz collector id shows up as filterable. Any ideas on why it wouldn't be allowing the Service filter on another but the collector id even thought the data is there and filterable in the traces area?

This message was deleted.