Siddhant Sahay
08/05/2022, 11:49 AMPatrick Sauerborn
08/06/2022, 7:07 PMDavid Castillo
08/08/2022, 8:27 AMAnil Kumar Bandrapalli
08/08/2022, 9:24 AMAlexei Zenin
08/08/2022, 6:02 PMTable signoz_traces.top_level_operations doesn't exist
Did I miss updating the collector, didn’t see anything mentioned in the release notes, wish the query service would be the component that handles schema creation instead of the collectors. Seems like the minor version bump might have had a breaking change if not aligned with the right collector?William Wang
08/10/2022, 3:13 AMUpesh Jindal
08/10/2022, 5:22 AMUpesh Jindal
08/10/2022, 5:23 AMUpesh Jindal
08/10/2022, 5:23 AMUpesh Jindal
08/10/2022, 6:02 AMopentelemetry-bootstrap --action=install
🤦♂️Upesh Jindal
08/10/2022, 6:24 AMSrikanth Chekuri
08/10/2022, 6:25 AMUpesh Jindal
08/10/2022, 8:51 AMTraceback (most recent call last):
File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/opentelemetry/sdk/metrics/_internal/instrument.py", line 132, in callback
for api_measurement in callback(callback_options):
File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/opentelemetry/instrumentation/system_metrics/__init__.py", line 563, in _get_system_network_connections
for net_connection in psutil.net_connections():
File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/psutil/__init__.py", line 2158, in net_connections
return _psplatform.net_connections(kind)
File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/psutil/_psosx.py", line 248, in net_connections
cons = Process(pid).connections(kind)
File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/psutil/_psosx.py", line 350, in wrapper
raise AccessDenied(self.pid, self._name)
psutil.AccessDenied: (pid=27843)
Callback failed for instrument system.network.connections.
Traceback (most recent call last):
File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/psutil/_psosx.py", line 343, in wrapper
return fun(self, *args, **kwargs)
File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/psutil/_psosx.py", line 500, in connections
rawlist = cext.proc_connections(self.pid, families, types)
PermissionError: [Errno 1] Operation not permitted (originated from proc_pidinfo(PROC_PIDLISTFDS) 1/2)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/opentelemetry/sdk/metrics/_internal/instrument.py", line 132, in callback
for api_measurement in callback(callback_options):
File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/opentelemetry/instrumentation/system_metrics/__init__.py", line 563, in _get_system_network_connections
for net_connection in psutil.net_connections():
File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/psutil/__init__.py", line 2158, in net_connections
return _psplatform.net_connections(kind)
File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/psutil/_psosx.py", line 248, in net_connections
cons = Process(pid).connections(kind)
File "/Users/upeshjindal/Virtual/healthpe.api.backend.py/lib/python3.10/site-packages/psutil/_psosx.py", line 350, in wrapper
raise AccessDenied(self.pid, self._name)
psutil.AccessDenied: (pid=27843)
Upesh Jindal
08/10/2022, 8:51 AMRaghu
08/11/2022, 1:04 AMRaghu
08/11/2022, 1:05 AMUpesh Jindal
08/11/2022, 3:39 AMUpesh Jindal
08/11/2022, 5:14 AMrakesh shah
08/11/2022, 6:40 AMrakesh shah
08/11/2022, 6:42 AMUpesh Jindal
08/11/2022, 9:58 AMAnant Vashishtha
08/11/2022, 11:55 AMChris Ahern
08/11/2022, 5:59 PMkubectl -n platform get pods
NAME READY STATUS RESTARTS AGE
my-release-zookeeper-0 1/1 Running 0 77d
chi-signoz-cluster-0-0-0 1/1 Running 0 77d
my-release-signoz-alertmanager-0 1/1 Running 0 77d
my-release-signoz-frontend-6b7dbccbc7-fgbnv 1/1 Running 0 77d
my-release-signoz-otel-collector-metrics-68bcfd5556-7tjks 1/1 Running 0 77d
my-release-signoz-otel-collector-66c8c7dc9d-xqxbd 1/1 Running 0 77d
clickhouse-operator-7f7f84b899-dcgs4 2/2 Running 0 21h
my-release-signoz-query-service-0 0/1 CrashLoopBackOff 201 18h
In the description of the my-release-signoz-query-service-0 pod, it shows:
Containers:
signoz:
Container ID: <containerd://aebb11f5525602fd8205e1aa49414724c6afc7c5b1784993bfe58d3e5a7e931>5
Image: <http://docker.io/signoz/query-service:0.8.0|docker.io/signoz/query-service:0.8.0>
Image ID: <http://docker.io/signoz/query-service@sha256:2febce16a8b8feb6bf96439eccccba4e6ab8e7ef401de96ea38fd1d42c3d9353|docker.io/signoz/query-service@sha256:2febce16a8b8feb6bf96439eccccba4e6ab8e7ef401de96ea38fd1d42c3d9353>
Port: 8080/TCP
Host Port: 0/TCP
Args:
-config=/root/config/prometheus.yml
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Thu, 11 Aug 2022 10:54:03 -0700
Finished: Thu, 11 Aug 2022 10:54:42 -0700
Ready: False
Restart Count: 201
Limits:
cpu: 750m
memory: 1000Mi
Requests:
cpu: 200m
memory: 300Mi
I would increase the memory limit but don't see a deployment for the my-release-signoz-query-service-0 pod:
kubectl -n platform get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
my-release-signoz-frontend 1/1 1 1 77d
my-release-signoz-otel-collector-metrics 1/1 1 1 77d
my-release-signoz-otel-collector 1/1 1 1 77d
clickhouse-operator 1/1 1 1 77d
How do I fix this?Aditya Tripathi
08/12/2022, 8:08 AMcredentials: grpc.credentials.createInsecure(),
^
TypeError: Cannot read properties of undefined (reading 'credentials')
at Object.<anonymous> (/home/xxx/src/tracer.ts:17:21)
at Module._compile (node:internal/modules/cjs/loader:1103:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
at Module.load (node:internal/modules/cjs/loader:981:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Module.require (node:internal/modules/cjs/loader:1005:19)
at require (node:internal/modules/cjs/helpers:102:18)
at Object.<anonymous> (/home/xxx/src/main.ts:13:1)
at Module._compile (node:internal/modules/cjs/loader:1103:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
Pranay
Krishna Chaitanya V
08/16/2022, 2:46 PMAlexei Zenin
08/17/2022, 3:47 PMreceivers:
otlp/spanmetrics:
protocols:
grpc:
endpoint: localhost:12345
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
processors:
batch:
send_batch_size: 10000
send_batch_max_size: 11000
timeout: 10s
signozspanmetrics/prometheus:
metrics_exporter: prometheus
latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 50ms, 100ms, 250ms, 500ms, 1000ms, 1400ms, 2000ms, 5s, 10s, 20s, 40s, 60s ]
dimensions_cache_size: 10000
dimensions:
- name: service.namespace
default: default
- name: deployment.environment
default: default
exporters:
clickhousetraces:
datasource: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=signoz_traces&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PWD}
clickhousemetricswrite:
endpoint: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=signoz_metrics&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PWD}
resource_to_telemetry_conversion:
enabled: true
prometheus:
endpoint: 0.0.0.0:8889
extensions:
health_check:
endpoint: 0.0.0.0:13133
service:
telemetry:
metrics:
address: 0.0.0.0:8888
extensions: [health_check]
pipelines:
traces:
receivers: [otlp]
processors: [signozspanmetrics/prometheus, batch]
exporters: [clickhousetraces]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [clickhousemetricswrite]
metrics/spanmetrics:
receivers: [otlp/spanmetrics]
exporters: [prometheus]
Note: I am not sending directly to this collector but via an agent collector on the instance which forwards to this one via OTLPMukesh Chaudhary
08/19/2022, 5:34 AMsomething went wrong
.
The log of otel is below:
2022-08-19T04:48:22.427Z error clickhousetracesexporter/writer.go:101 Could not write a batch of spans {"kind": "exporter", "name": "clickhousetraces", "error": "code: 81, message: Database signoz_traces doesn't exist"}
<http://github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter|github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter>
/src/exporter/clickhousetracesexporter/writer.go:101
2022-08-19T04:48:32.432Z error clickhousetracesexporter/writer.go:101 Could not write a batch of spans {"kind": "exporter", "name": "clickhousetraces", "error": "code: 81, message: Database signoz_traces doesn't exist"}
<http://github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter|github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter>
/src/exporter/clickhousetracesexporter/writer.go:101
2022-08-19T04:48:42.430Z error clickhousetracesexporter/writer.go:101 Could not write a batch of spans {"kind": "exporter", "name": "clickhousetraces", "error": "code: 81, message: Database signoz_traces doesn't exist"}
<http://github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter|github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter>
/src/exporter/clickhousetracesexporter/writer.go:101
2022-08-19T04:48:52.444Z error clickhousetracesexporter/writer.go:101 Could not write a batch of spans {"kind": "exporter", "name": "clickhousetraces", "error": "code: 81, message: Database signoz_traces doesn't exist"}
<http://github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter|github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter>
/src/exporter/clickhousetracesexporter/writer.go:101
2022-08-19T04:49:02.443Z error clickhousetracesexporter/writer.go:101 Could not write a batch of spans {"kind": "exporter", "name": "clickhousetraces", "error": "code: 81, message: Database signoz_traces doesn't exist"}
<http://github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter|github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter>
/src/exporter/clickhousetracesexporter/writer.go:101
2022-08-19T04:49:12.437Z error clickhousetracesexporter/writer.go:101 Could not write a batch of spans {"kind": "exporter", "name": "clickhousetraces", "error": "code: 81, message: Database signoz_traces doesn't exist"}
<http://github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter|github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter>
/src/exporter/clickhousetracesexporter/writer.go:101
2022-08-19T04:49:22.438Z error clickhousetracesexporter/writer.go:101 Could not write a batch of spans {"kind": "exporter", "name": "clickhousetraces", "error": "code: 81, message: Database signoz_traces doesn't exist"}
<http://github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter|github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter>
/src/exporter/clickhousetracesexporter/writer.go:101
Logs
Also in UI there is no metrics section.Upesh Jindal
08/21/2022, 5:33 AMBlake Romano
08/23/2022, 9:36 PM