I have tried Open Search, but what a mess that is

Hello, I'm trying to lower the log output from ClickHouse, through the Helm chart you provide. I tried to use the help in the

values.yaml

to do so, but it seems not to be working. Here my `values.yaml`: (I know I duplicated

default/log_queries=0

but it's on purpose to try to make it work).

clickhouse:
      defaultProfiles:
        default/allow_experimental_window_functions: "1"
        default/allow_nondeterministic_mutations: "1"
        default/log_queries: "0"
        logger:
          level: warning
      enable: true
      profiles:
        default/log_queries: "0"
        logger:
          level: warning
##this block doesn't work, clickhouse get error about unknown settings:
##      logger/level: "warning"
##end of test
      resources:
        limits:
          cpu: 2000m
          memory: 4Gi
        requests:
          cpu: 50m
          memory: 4Gi

But then, the pod gets deployed, and still shows information logs such as:

clickhouse 2023.03.06 16:32:04.571158 [ 234 ] {4483843e-d883-4d01-af4d-2872f07b4545} <Information> executeQuery: Read 170 rows, 674.05 KiB in 0.015092185 sec., 11264 rows/sec., 43.62 MiB/sec.
clickhouse 2023.03.06 16:32:04.598816 [ 234 ] {bd514de5-4a53-4cbe-a959-d8ccb2a02ddf} <Information> executeQuery: Read 1190 rows, 2.22 MiB in 0.027129067 sec., 43864 rows/sec., 81.98 MiB/sec.
clickhouse 2023.03.06 16:32:04.774338 [ 235 ] {7dfafbe7-0766-49c4-a97f-cc583c6a5e54} <Information> executeQuery: Read 1169664 rows, 8.92 MiB in 0.15812709 sec., 7396986 rows/sec., 56.43 MiB/sec.
clickhouse 2023.03.06 16:32:05.360248 [ 224 ] {7556b233-89b2-4a20-8b11-b0c82da94f3d} <Information> executeQuery: Read 1169664 rows, 8.92 MiB in 0.129886739 sec., 9005261 rows/sec., 68.70 MiB/sec.
clickhouse 2023.03.06 16:32:05.456336 [ 218 ] {776364b3-20cb-41f4-8736-acc3b165b0b3} <Information> executeQuery: Read 452 rows, 11.19 KiB in 0.002209947 sec., 204529 rows/sec., 4.95 MiB/sec.
clickhouse 2023.03.06 16:32:05.485650 [ 228 ] {ddbec803-a941-4f7b-9a19-0bac720bfb82} <Information> executeQuery: Read 3618 rows, 88.59 KiB in 0.006745317 sec., 536372 rows/sec., 12.83 MiB/sec.
clickhouse 2023.03.06 16:32:05.564685 [ 240 ] {8bf6ccc3-6663-41f3-a444-54803c07f2d7} <Information> executeQuery: Read 3617 rows, 88.59 KiB in 0.01204406 sec., 300314 rows/sec., 7.18 MiB/sec.

and on the pod, I can check some config files:

bash-5.1$ cat etc/clickhouse-server/config.d/01-clickhouse-02-logger.xml 
<yandex>
    <logger>
        <!-- Possible levels: <https://github.com/pocoproject/poco/blob/develop/Foundation/include/Poco/Logger.h#L105> -->
        <level>information</level>
        <log>/var/log/clickhouse-server/clickhouse-server.log</log>
        <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
        <size>1000M</size>
        <count>10</count>
        <!-- Default behavior is autodetection (log to console if not daemon mode and is tty) -->
        <console>true</console>
    </logger>
</yandex>

Could you tell me what I'm missing here please? Thanks a lot

Hi Team, I'm having a particular issue with a micro-service (springboot) that was instrumented (automatic instrumentation) and some metrics are not being seen. If I check /actuator/prometheus I see:

http_server_requests_seconds_count{exception="None",method="PUT",outcome="SUCCESS",status="200",uri="/api/v1/payment-orders/{id}/cancel",} 12.0
http_server_requests_seconds_sum{exception="None",method="PUT",outcome="SUCCESS",status="200",uri="/api/v1/payment-orders/{id}/cancel",} 0.919633034

But I I try to run PromQL

sum(rate(http_server_requests_seconds_count{uri=~"{{.uri}}"}[1m]))

I don't get any data. I've instrumented other services the same way and I'm getting that metric in particular. Also I'd like to mention that I'm getting traces from that service

Greetings Everyone, Kindly I want to ask few questions: 1- Is there a way to create/register the first admin user of signoz (first time login from UI) using signoz helm chart ? 2- Is there a way to import signoz dashboards from helm chart. 3- Is there a way to modify the retention period for metrics, logs and traces from helm chart ? We are trying to automate these things in order to deploy on multiple clusters. Thanks in advance 🙏

Can SigNoZ monitor big data processing components such as Yarn, HDFS, Hive, Spark and so on?

What is the most used storage for clickhouse ? Do we use gp3 or efs or fsx for lustre ? Also what is recommanded CPU/Memory for clickhouse ?

<!channel> 👋 Time for OpenTelemetry end-user discussion group APAC session of this month! The end-user group discussion is a place where you can discuss challenges related to your OTel implementations and learn from other OpenTelemetry users in a vendor-neutral space. Here are the details: Date & Time: Wednesday, March 15, 2023 at 11:00 AM to12:00 PM IST, Link to register: https://lu.ma/1w129wgu Looking forward to discuss all things OpenTelemetry. 🙌 You can also check out notes from our last end-user group discussion here: https://opentelemetry.io/blog/2023/end-user-discussions-02/

Hello. I am new be here and I am really interested on everything related with Go.

I wonder if I can cooperate with this excellent team members.

Thanks.

Hi, I am a AIOPS engineer, I am using signoz as the data source for my algorithms. How can I access the data in signoz by python scripts?

<!here> those using SigNoz in production, can you please comment with your experience in a reddit thread going on? I think some folks need feedback. Also, feel free to share both friction points and useful things about SigNoz. We shall take the feedback seriously 🙂 https://www.reddit.com/r/devops/comments/11n272q/who_uses_signoz_in_production/

Hello, I have a question about the

query-service

, I noticed in the logs many queries going to https://api.segment.io and https://app.posthog.com what is it? And how can I turn it off (the gateway show too many errors and trigger some alert on our monitoring stack).

signoz-query-service posthog 2023/03/13 13:52:22 ERROR: sending request - Post "<https://app.posthog.com/batch/>": dial tcp 3.91.127.143:443: connect: connection refused
signoz-query-service segment 2023/03/13 13:52:22 ERROR: sending request - Post "<https://api.segment.io/v1/batch>": dial tcp 44.228.49.115:443: connect: connection refused

Hi Team, Can I use Signoz for Data pipeline observability use case to ingest logs from Apache Airflow, Spark and then correlate logs to find root cause of pipeline failure?

How do I invite a team mate in free version. do I need to specify a mail server, port somewhere in configmap? do not see any setup option in UI.thanks!

Hi, do you know how to get the value of "Service -> Overview Metrics -> Latency, Rate(ops/s), Error Percentage" with SQL search in ClickHouse?

Does Signoz support collecting k8s kubelet logs, or kubelet traces from v1.25 onwards?

Greetings, Does signos support exporting the logs/metrics/traces to a file ?

Hello greetings 🙂

@Ankit Nayan https://signoz-community.slack.com/archives/C01HWUTP4HH/p1679037162491089?thread_ts=1678458641.025019&amp;cid=C01HWUTP4HH

This is a great product! Thank you :)

#signoz #support #installation I'm doing a setup in the AKS of signoz i've used below helm charts helm repo add signoz https://charts.signoz.io changed the namespace to signoz and used our own standalone clickhouse cluster setup along with zookeeper Post installation i'm obeserving below issues in OTEL_COLLECTOR Error is below PS D:\Signoz> kubectl logs po/signoz-otel-collector-65fbf66f4f-zdntv -n signoz Defaulted container "signoz-otel-collector" out of: signoz-otel-collector, signoz-otel-collector-init (init) 2023-03-17T10:58:29.571Z info service/telemetry.go:111 Setting up own telemetry... 2023-03-17T10:58:29.572Z info service/telemetry.go:141 Serving Prometheus metrics {"address": "0.0.0.0:8888", "level": "Basic"} 2023-03-17T10:58:29.572Z info components/components.go:30 Stability level of component is undefined {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "stability": "Undefined"} time="2023-03-17T10:58:29Z" level=info msg="Executing:\nCREATE DATABASE IF NOT EXISTS signoz_metrics ON CLUSTER cluster\n" component=clickhouse time="2023-03-17T10:58:29Z" level=info msg="Executing:\nCREATE TABLE IF NOT EXISTS signoz_metrics.samples_v2 ON CLUSTER cluster (\n\t\t\tmetric_name LowCardinality(String),\n\t\t\tfingerprint UInt64 Codec(DoubleDelta, LZ4),\n\t\t\ttimestamp_ms Int64 Codec(DoubleDelta, LZ4),\n\t\t\tvalue Float64 Codec(Gorilla, LZ4)\n\t\t)\n\t\tENGINE = MergeTree\n\t\t\tPARTITION BY toDate(timestamp_ms / 1000)\n\t\t\tORDER BY (metric_name, fingerprint, timestamp_ms)\n\t\t\tTTL toDateTime(timestamp_ms/1000) + INTERVAL 2592000 SECOND DELETE;\n" component=clickhouse time="2023-03-17T10:58:29Z" level=info msg="Executing:\nCREATE TABLE IF NOT EXISTS signoz_metrics.distributed_samples_v2 ON CLUSTER cluster AS signoz_metrics.samples_v2 ENGINE = Distributed(\"cluster\", \"signoz_metrics\", samples_v2, cityHash64(metric_name, fingerprint));\n" component=clickhouse time="2023-03-17T10:58:30Z" level=info msg="Executing:\nALTER TABLE signoz_metrics.samples_v2 ON CLUSTER cluster MODIFY SETTING ttl_only_drop_parts = 1;\n" component=clickhouse time="2023-03-17T10:58:30Z" level=info msg="Executing:\nSET allow_experimental_object_type = 1\n" component=clickhouse time="2023-03-17T10:58:30Z" level=info msg="Executing:\nCREATE TABLE IF NOT EXISTS signoz_metrics.time_series_v2 ON CLUSTER cluster(\n\t\t\tmetric_name LowCardinality(String),\n\t\t\tfingerprint UInt64 Codec(DoubleDelta, LZ4),\n\t\t\ttimestamp_ms Int64 Codec(DoubleDelta, LZ4),\n\t\t\tlabels String Codec(ZSTD(5))\n\t\t)\n\t\tENGINE = ReplacingMergeTree\n\t\t\tPARTITION BY toDate(timestamp_ms / 1000)\n\t\t\tORDER BY (metric_name, fingerprint)\n\t\t\tTTL toDateTime(timestamp_ms/1000) + INTERVAL 2592000 SECOND DELETE;\n" component=clickhouse time="2023-03-17T10:58:30Z" level=info msg="Executing:\nCREATE TABLE IF NOT EXISTS signoz_metrics.distributed_time_series_v2 ON CLUSTER cluster AS signoz_metrics.time_series_v2 ENGINE = Distributed(\"cluster\", signoz_metrics, time_series_v2, cityHash64(metric_name, fingerprint));\n" component=clickhouse time="2023-03-17T10:58:30Z" level=info msg="Executing:\nALTER TABLE signoz_metrics.time_series_v2 ON CLUSTER cluster DROP COLUMN IF EXISTS labels_object\n" component=clickhouse time="2023-03-17T10:58:30Z" level=info msg="Executing:\nALTER TABLE signoz_metrics.distributed_time_series_v2 ON CLUSTER cluster DROP COLUMN IF EXISTS labels_object\n" component=clickhouse time="2023-03-17T10:58:30Z" level=info msg="Executing:\nALTER TABLE signoz_metrics.time_series_v2 ON CLUSTER cluster MODIFY SETTING ttl_only_drop_parts = 1;\n" component=clickhouse 2023-03-17T10:58:31.476Z info kube/client.go:101 k8s filtering {"kind": "processor", "name": "k8sattributes", "pipeline": "metrics", "labelSelector": "", "fieldSelector": "spec.nodeName=aks-nodepool1-18518278-vmss00000f"} 2023-03-17T10:58:31.477Z info components/components.go:30 Stability level of component is undefined {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "stability": "Undefined"} 2023-03-17T10:58:31.676Z info clickhousetracesexporter/clickhouse_factory.go:146 Patching views {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces"} 2023-03-17T10:58:32.786Z info clickhousetracesexporter/clickhouse_factory.go:116 Running migrations from path: {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "test": "/migrations"} 2023-03-17T10:58:41.160Z info clickhousetracesexporter/clickhouse_factory.go:128 Clickhouse Migrate finished {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces"} Error: cannot build pipelines: failed to create "clickhousetraces" exporter, in pipeline "traces": code: 62, message: Syntax error: failed at position 2290 (')') (line 43, col 3): ) ENGINE MergeTree() PARTITION BY toDate(timestamp) ORDER BY (durationNano, timestamp) TTL toDateTime(timestamp) + INTERVAL 604800 SECOND DELETE SETTING. Expected one of: table property (column, index, constraint) declaration, INDEX, CONSTRAINT, PROJECTION, PRIMARY KEY, column declaration, identifier 2023/03/17 10:58:41 application run finished with error: cannot build pipelines: failed to create "clickhousetraces" exporter, in pipeline "traces": code: 62, message: Syntax error: failed at position 2290 (')') (line 43, col 3): ) ENGINE MergeTree() PARTITION BY toDate(timestamp) ORDER BY (durationNano, timestamp) TTL toDateTime(timestamp) + INTERVAL 604800 SECOND DELETE SETTING. Expected one of: table property (column, index, constraint) declaration, INDEX, CONSTRAINT, PROJECTION, PRIMARY KEY, column declaration, identifier need help to understand this at the earliest please

#2466 Signoz setup in k8s otel collectors crashing Issue created by Vaibhavi-Tanla I'm doing a setup in the AKS of signoz i've used below helm charts helm repo add signoz https://charts.signoz.io changed the namespace to signoz and used our own standalone clickhouse cluster setup along with zookeeper Post installation i'm obeserving below issues in OTEL_COLLECTOR Error is below PS D:\Signoz> kubectl logs po/signoz-otel-collector-65fbf66f4f-zdntv -n signoz Defaulted container "signoz-otel-collector" out of: signoz-otel-collector, signoz-otel-collector-init (init) 2023-03-17T10:58:29.571Z info service/telemetry.go:111 Setting up own telemetry... 2023-03-17T10:58:29.572Z info service/telemetry.go:141 Serving Prometheus metrics {"address": "0.0.0.0:8888", "level": "Basic"} 2023-03-17T10:58:29.572Z info components/components.go:30 Stability level of component is undefined {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "stability": "Undefined"} time="2023-03-17T10:58:29Z" level=info msg="Executing:\nCREATE DATABASE IF NOT EXISTS signoz_metrics ON CLUSTER cluster\n" component=clickhouse time="2023-03-17T10:58:29Z" level=info msg="Executing:\nCREATE TABLE IF NOT EXISTS signoz_metrics.samples_v2 ON CLUSTER cluster (\n\t\t\tmetric_name LowCardinality(String),\n\t\t\tfingerprint UInt64 Codec(DoubleDelta, LZ4),\n\t\t\ttimestamp_ms Int64 Codec(DoubleDelta, LZ4),\n\t\t\tvalue Float64 Codec(Gorilla, LZ4)\n\t\t)\n\t\tENGINE = MergeTree\n\t\t\tPARTITION BY toDate(timestamp_ms / 1000)\n\t\t\tORDER BY (metric_name, fingerprint, timestamp_ms)\n\t\t\tTTL toDateTime(timestamp_ms/1000) + INTERVAL 2592000 SECOND DELETE;\n" component=clickhouse time="2023-03-17T10:58:29Z" level=info msg="Executing:\nCREATE TABLE IF NOT EXISTS signoz_metrics.distributed_samples_v2 ON CLUSTER cluster AS signoz_metrics.samples_v2 ENGINE = Distributed("cluster", "signoz_metrics", samples_v2, cityHash64(metric_name, fingerprint));\n" component=clickhouse time="2023-03-17T10:58:30Z" level=info msg="Executing:\nALTER TABLE signoz_metrics.samples_v2 ON CLUSTER cluster MODIFY SETTING ttl_only_drop_parts = 1;\n" component=clickhouse time="2023-03-17T10:58:30Z" level=info msg="Executing:\nSET allow_experimental_object_type = 1\n" component=clickhouse time="2023-03-17T10:58:30Z" level=info msg="Executing:\nCREATE TABLE IF NOT EXISTS signoz_metrics.time_series_v2 ON CLUSTER cluster(\n\t\t\tmetric_name LowCardinality(String),\n\t\t\tfingerprint UInt64 Codec(DoubleDelta, LZ4),\n\t\t\ttimestamp_ms Int64 Codec(DoubleDelta, LZ4),\n\t\t\tlabels String Codec(ZSTD(5))\n\t\t)\n\t\tENGINE = ReplacingMergeTree\n\t\t\tPARTITION BY toDate(timestamp_ms / 1000)\n\t\t\tORDER BY (metric_name, fingerprint)\n\t\t\tTTL toDateTime(timestamp_ms/1000) + INTERVAL 2592000 SECOND DELETE;\n" component=clickhouse time="2023-03-17T10:58:30Z" level=info msg="Executing:\nCREATE TABLE IF NOT EXISTS signoz_metrics.distributed_time_series_v2 ON CLUSTER cluster AS signoz_metrics.time_series_v2 ENGINE = Distributed("cluster", signoz_metrics, time_series_v2, cityHash64(metric_name, fingerprint));\n" component=clickhouse time="2023-03-17T10:58:30Z" level=info msg="Executing:\nALTER TABLE signoz_metrics.time_series_v2 ON CLUSTER cluster DROP COLUMN IF EXISTS labels_object\n" component=clickhouse time="2023-03-17T10:58:30Z" level=info msg="Executing:\nALTER TABLE signoz_metrics.distributed_time_series_v2 ON CLUSTER cluster DROP COLUMN IF EXISTS labels_object\n" component=clickhouse time="2023-03-17T10:58:30Z" level=info msg="Executing:\nALTER TABLE signoz_metrics.time_series_v2 ON CLUSTER cluster MODIFY SETTING ttl_only_drop_parts = 1;\n" component=clickhouse 2023-03-17T10:58:31.476Z info kube/client.go:101 k8s filtering {"kind": "processor", "name": "k8sattributes", "pipeline": "metrics", "labelSelector": "", "fieldSelector": "spec.nodeName=aks-nodepool1-18518278-vmss00000f"} 2023-03-17T10:58:31.477Z info components/components.go:30 Stability level of component is undefined {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "stability": "Undefined"} 2023-03-17T10:58:31.676Z info clickhousetracesexporter/clickhouse_factory.go:146 Patching views {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces"} 2023-03-17T10:58:32.786Z info clickhousetracesexporter/clickhouse_factory.go:116 Running migrations from path: {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "test": "/migrations"} 2023-03-17T10:58:41.160Z info clickhousetracesexporter/clickhouse_factory.go:128 Clickhouse Migrate finished {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces"} Error: cannot build pipelines: failed to create "clickhousetraces" exporter, in pipeline "traces": code: 62, message: Syntax error: failed at position 2290 (')') (line 43, col 3): ) ENGINE MergeTree() PARTITION BY toDate(timestamp) ORDER BY (durationNano, timestamp) TTL toDateTime(timestamp) + INTERVAL 604800 SECOND DELETE SETTING. Expected one of: table property (column, index, constraint) declaration, INDEX, CONSTRAINT, PROJECTION, PRIMARY KEY, column declaration, identifier 2023/03/17 10:58:41 application run finished with error: cannot build pipelines: failed to create "clickhousetraces" exporter, in pipeline "traces": code: 62, message: Syntax error: failed at position 2290 (')') (line 43, col 3): ) ENGINE MergeTree() PARTITION BY toDate(timestamp) ORDER BY (durationNano, timestamp) TTL toDateTime(timestamp) + INTERVAL 604800 SECOND DELETE SETTING. Expected one of: table property (column, index, constraint) declaration, INDEX, CONSTRAINT, PROJECTION, PRIMARY KEY, column declaration, identifier need help to understand this at the earliest please I'm doing a setup in the AKS of signoz i've used below helm charts helm repo add signoz https://charts.signoz.io changed the namespace to signoz and used our own standalone clickhouse cluster setup along with zookeeper Post installation i'm obeserving below issues in OTEL_COLLECTOR Error is below PS D:\Signoz> kubectl logs po/signoz-otel-collector-65fbf66f4f-zdntv -n signoz Defaulted container "signoz-otel-collector" out of: signoz-otel-collector, signoz-otel-collector-init (init) 2023-03-17T10:58:29.571Z info service/telemetry.go:111 Setting up own telemetry... 2023-03-17T10:58:29.572Z info service/telemetry.go:141 Serving Prometheus metrics {"address": "0.0.0.0:8888", "level": "Basic"} 2023-03-17T10:58:29.572Z info components/components.go:30 Stability level of component is undefined {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "stability": "Undefined"} time="2023-03-17T10:58:29Z" level=info msg="Executing:\nCREATE DATABASE IF NOT EXISTS signoz_metrics ON CLUSTER cluster\n" component=clickhouse time="2023-03-17T10:58:29Z" level=info msg="Executing:\nCREATE TABLE IF NOT EXISTS signoz_metrics.samples_v2 ON CLUSTER cluster (\n\t\t\tmetric_name LowCardinality(String),\n\t\t\tfingerprint UInt64 Codec(DoubleDelta, LZ4),\n\t\t\ttimestamp_ms Int64 Codec(DoubleDelta, LZ4),\n\t\t\tvalue Float64 Codec(Gorilla, LZ4)\n\t\t)\n\t\tENGINE = MergeTree\n\t\t\tPARTITION BY toDate(timestamp_ms / 1000)\n\t\t\tORDER BY (metric_name, fingerprint, timestamp_ms)\n\t\t\tTTL toDateTime(timestamp_ms/1000) + INTERVAL 2592000 SECOND DELETE;\n" component=clickhouse time="2023-03-17T10:58:29Z" level=info msg="Executing:\nCREATE TABLE IF NOT EXISTS signoz_metrics.distributed_samples_v2 ON CLUSTER cluster AS signoz_metrics.samples_v2 ENGINE = Distributed("cluster", "signoz_metrics", samples_v2, cityHash64(metric_name, fingerprint));\n" component=clickhouse time="2023-03-17T10:58:30Z" level=info msg="Executing:\nALTER TABLE signoz_metrics.samples_v2 ON CLUSTER cluster MODIFY SETTING ttl_only_drop_parts = 1;\n" component=clickhouse time="2023-03-17T10:58:30Z" level=info msg="Executing:\nSET allow_experimental_object_type = 1\n" component=clickhouse time="2023-03-17T10:58:30Z" l… SigNoz/signoz

Hi, we are considering using SigNoz as a tool to help us monitor logs for PCI DSS compliance. Does anyone know of a set of pre-configured Queries that will generate alerts related to PCI DSS compliance? Or has anyone in the community used SigNoz in a similar manner?

That was quite a thing to make Metrics with the JS examples working ... but I got it.

OpentelemetryJs has changed - I make a new PR for the SigNoz examples and add some words of wisdom to it.

Is there a way in the logger to put custom fields to the detail view? I am using NodeJS/winson (just for fun).

This is giving me nuts again 😞 I don't know what I am doing wrong... Logs - in general work. But where is this one specific log?

Greetings, Kindly I need help in getting all logs and traces for specific trace_id Thanks in advance.