Hi. I'm facing readonly table issues. I restarted clickhouse pods many times but not worked. What should I do then? The manaual said " Whether the replica is in read-only mode. This mode is turned on if the config does not have sections with ClickHouse Keeper, if an unknown error occurred when reinitializing sessions in ClickHouse Keeper, and during session reinitialization in ClickHouse Keeper." There's something to do with zookeeper?

"select database, table, is_leader, is_readonly, parts_to_check, zookeeper_name from system.replicas where is_readonly=1"
metrics time_series_v2  0       1       0       default
metrics time_series_v3  0       1       0       default
prod_logs   logs_resource_keys      0       1       0       default
prod_logs   tag_attributes  0       1       0       default

Yes, please share logs of ClickHouse. It only goes into readonly mode when zk is not available or something wrong with the running zk

Thanks. Main reason was lack of resources of k8s nodes. The readonly tables unlocked after making 'force_restore_data' file and restart. >> $ touch /var/lib/clickhouse/flags/force_restore_data ^^ belows are part of the logs.

2024.10.01 08:48:20.954141 [ 870 ] {88273e6e-fe7f-4dd7-aef3-4946d0bbb1b0} <Error> executeQuery: Code: 252. DB::Exception: Too many parts (3002 with average size of 1.93 KiB) in table 'signoz_logs.tag_attributes (f009532e-9025-41dc-8092-c9b477353a11)'. Merges are processing significantly slower than inserts. (TOO_MANY_PARTS) (version 24.1.2.5 (official build)) (from [::ffff:10.10.222.138]:48390) (in query: INSERT INTO signoz_logs.tag_attributes (timestamp, tagKey, tagType, tagDataType, stringTagValue, int64TagValue, float64TagValue) VALUES), Stack trace (when copying this message, always include the lines below):
2024.10.01 08:48:20.954289 [ 870 ] {88273e6e-fe7f-4dd7-aef3-4946d0bbb1b0} <Error> TCPHandler: Code: 252. DB::Exception: Too many parts (3002 with average size of 1.93 KiB) in table 'signoz_logs.tag_attributes (f009532e-9025-41dc-8092-c9b477353a11)'. Merges are processing significantly slower than inserts. (TOO_MANY_PARTS), Stack trace (when copying this message, always include the lines below):
2024.10.01 08:48:22.372947 [ 690 ] {079405c4-89bb-4575-a58f-0ad2a808064d} <Trace> signoz_logs.logs_resource_keys (c9fe09a7-f012-4966-b5b2-b07e9727ec5b) (Replicated OutputStream): ZooKeeperRetriesControl: commitPart: setUserError: error=242 message=Table is in readonly mode: replica_path=/clickhouse/tables/c9fe09a7-f012-4966-b5b2-b07e9727ec5b/4/replicas/chi-signoz-clickhouse-cluster-4-0
2024.10.01 08:48:23.138744 [ 827 ] {7f0b43ff-f6f1-4bf8-ad45-ee99012afd4f} <Trace> signoz_logs.logs_resource_keys (c9fe09a7-f012-4966-b5b2-b07e9727ec5b) (Replicated OutputStream): ZooKeeperRetriesControl: commitPart: setUserError: error=242 message=Table is in readonly mode: replica_path=/clickhouse/tables/c9fe09a7-f012-4966-b5b2-b07e9727ec5b/4/replicas/chi-signoz-clickhouse-cluster-4-0
2024.10.01 08:48:23.940329 [ 875 ] {79779f16-5488-4d1f-b8a1-25b73dd51adc} <Error> executeQuery: Code: 252. DB::Exception: Too many parts (3002 with average size of 1.93 KiB) in table 'signoz_logs.tag_attributes (f009532e-9025-41dc-8092-c9b477353a11)'. Merges are processing significantly slower than inserts. (TOO_MANY_PARTS) (version 24.1.2.5 (official build)) (from [::ffff:10.10.220.147]:34690) (in query: INSERT INTO signoz_logs.tag_attributes (timestamp, tagKey, tagType, tagDataType, stringTagValue, int64TagValue, float64TagValue) VALUES), Stack trace (when copying this message, always include the lines below):
2024.10.01 08:48:23.940488 [ 875 ] {79779f16-5488-4d1f-b8a1-25b73dd51adc} <Error> TCPHandler: Code: 252. DB::Exception: Too many parts (3002 with average size of 1.93 KiB) in table 'signoz_logs.tag_attributes (f009532e-9025-41dc-8092-c9b477353a11)'. Merges are processing significantly slower than inserts. (TOO_MANY_PARTS), Stack trace (when copying this message, always include the lines below):
2024.10.01 08:48:24.076759 [ 794 ] {97c98293-71ae-4a21-b9b9-5f237de8e46b} <Trace> signoz_logs.logs_resource_keys (c9fe09a7-f012-4966-b5b2-b07e9727ec5b) (Replicated OutputStream): ZooKeeperRetriesControl: commitPart: setUserError: error=242 message=Table is in readonly mode: replica_path=/clickhouse/tables/c9fe09a7-f012-4966-b5b2-b07e9727ec5b/4/replicas/chi-signoz-clickhouse-cluster-4-0
2024.10.01 08:48:24.832409 [ 807 ] {ac0aa67d-a87a-4e19-adf7-bf0c045de18f} <Trace> signoz_logs.logs_resource_keys (c9fe09a7-f012-4966-b5b2-b07e9727ec5b) (Replicated OutputStream): ZooKeeperRetriesControl: commitPart: setUserError: error=242 message=Table is in readonly mode: replica_path=/clickhouse/tables/c9fe09a7-f012-4966-b5b2-b07e9727ec5b/4/replicas/chi-signoz-clickhouse-cluster-4-0
2024.10.01 08:48:27.180602 [ 829 ] {51e66819-8a13-4422-b045-8a9664d4adfb} <Trace> signoz_logs.logs_resource_keys (c9fe09a7-f012-4966-b5b2-b07e9727ec5b) (Replicated OutputStream): ZooKeeperRetriesControl: commitPart: setUserError: error=242 message=Table is in readonly mode: replica_path=/clickhouse/tables/c9fe09a7-f012-4966-b5b2-b07e9727ec5b/4/replicas/chi-signoz-clickhouse-cluster-4-0
2024.10.01 08:48:27.268369 [ 813 ] {e2c214be-41c0-4431-919c-196cb955295c} <Trace> signoz_logs.logs_resource_keys (c9fe09a7-f012-4966-b5b2-b07e9727ec5b) (Replicated OutputStream): ZooKeeperRetriesControl: commitPart: setUserError: error=242 message=Table is in readonly mode: replica_path=/clickhouse/tables/c9fe09a7-f012-4966-b5b2-b07e9727ec5b/4/replicas/chi-signoz-clickhouse-cluster-4-0