Hi, we are self-hosting SigNoz with Clickhouse replica set to 2 to be fault tolerance. But we met issue when searching logs we are randomly losing half of the logs which seems like 2 Clickhouse replicas are not sync data between each other. I checked the logs table definition and seems not using the right Replicated engine but just MergeTree. Can someone suggest what we should do for it?
SHOW CREATE TABLE logs
Query id: 0affbaf0-0688-4d18-b7cf-e08c9052139c
ββstatementβββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β CREATE TABLE signoz_logs.logs
(
`timestamp` UInt64 CODEC(DoubleDelta, LZ4),
`observed_timestamp` UInt64 CODEC(DoubleDelta, LZ4),
`id` String CODEC(ZSTD(1)),
`trace_id` String CODEC(ZSTD(1)),
`span_id` String CODEC(ZSTD(1)),
`trace_flags` UInt32,
`severity_text` LowCardinality(String) CODEC(ZSTD(1)),
`severity_number` UInt8,
`body` String CODEC(ZSTD(2)),
`resources_string_key` Array(String) CODEC(ZSTD(1)),
`resources_string_value` Array(String) CODEC(ZSTD(1)),
`attributes_string_key` Array(String) CODEC(ZSTD(1)),
`attributes_string_value` Array(String) CODEC(ZSTD(1)),
`attributes_int64_key` Array(String) CODEC(ZSTD(1)),
`attributes_int64_value` Array(Int64) CODEC(ZSTD(1)),
`attributes_float64_key` Array(String) CODEC(ZSTD(1)),
`attributes_float64_value` Array(Float64) CODEC(ZSTD(1)),
`attributes_bool_key` Array(String) CODEC(ZSTD(1)),
`attributes_bool_value` Array(Bool) CODEC(ZSTD(1)),
INDEX body_idx body TYPE tokenbf_v1(10240, 3, 0) GRANULARITY 4,
INDEX id_minmax id TYPE minmax GRANULARITY 1,
INDEX severity_number_idx severity_number TYPE set(25) GRANULARITY 4,
INDEX severity_text_idx severity_text TYPE set(25) GRANULARITY 4,
INDEX trace_flags_idx trace_flags TYPE bloom_filter GRANULARITY 4
)
ENGINE = MergeTree
PARTITION BY toDate(timestamp / 1000000000)
ORDER BY (timestamp, id)
TTL toDateTime(timestamp / 1000000000) + toIntervalSecond(432000)
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1 β