Hi Team, our Signoz server went down today morning...
# support
d
Hi Team, our Signoz server went down today morning, I tried restarting with the
.install.sh
script, but getting error on restarting the services
s
What do ClickHouse logs show?
d
Copy code
(version 22.8.8.3 (official build))
2023.04.10 03:13:59.964880 [ 1 ] {} <Information> Application: Shutting down storages.
2023.04.10 03:13:59.964905 [ 1 ] {} <Information> Context: Shutdown disk default
2023.04.10 03:14:01.365702 [ 1 ] {} <Error> Application: DB::Exception: Suspiciously many (11 parts, 0.00 B in total) broken parts to remove while maximum allowed broken parts count is 10. You can change the maximum value with merge tree setting 'max_suspicious_broken_parts' in <merge_tree> configuration section or in table settings in .sql file (don't forget to return setting back to default value): Cannot attach table `signoz_traces`.`dependency_graph_minutes` from metadata file /var/lib/clickhouse/store/c1a/c1a64bd6-789f-4539-8542-15aa1eb29d06/dependency_graph_minutes.sql from query ATTACH TABLE signoz_traces.dependency_graph_minutes UUID 'cdbf4368-4dfd-41b2-a552-b02aa85903f5' (`src` LowCardinality(String) CODEC(ZSTD(1)), `dest` LowCardinality(String) CODEC(ZSTD(1)), `duration_quantiles_state` AggregateFunction(quantiles(0.5, 0.75, 0.9, 0.95, 0.99), Float64) CODEC(Default), `error_count` SimpleAggregateFunction(sum, UInt64) CODEC(T64, ZSTD(1)), `total_count` SimpleAggregateFunction(sum, UInt64) CODEC(T64, ZSTD(1)), `timestamp` DateTime CODEC(DoubleDelta, LZ4)) ENGINE = AggregatingMergeTree PARTITION BY toDate(timestamp) ORDER BY (timestamp, src, dest) TTL toDateTime(timestamp) + toIntervalSecond(604800) SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1
2023.04.10 03:14:01.365776 [ 1 ] {} <Information> Application: shutting down
2023.04.10 03:14:01.365878 [ 47 ] {} <Information> BaseDaemon: Stop SignalListener thread
@Srikanth Chekuri I could find this as last logs
s
Yes, force restore and restart the container
d
Copy code
/var/lib/clickhouse/flags/force_restore_data
where would be this file in our case ?
I dont see
clickhouse
in lib
Copy code
/home/centos/signoz/deploy/docker/clickhouse-setup/data/clickhouse/flags
?
s
Yes, this is folder mounted in docker container