Hello, We have a use case where we are reading a file with traces in JSON format, after which we send it to the Signoz Otel collector. What we are noticing is that the data which is ingested into Clickhouse is not being deduped. There are duplicates shown in the UI but another bug is that once I click on a specific trace the correct value is being shown. We are using

file_storage

for the

otel-collector-1

so that it keeps a track of the file offset, along with that we are also using the same

file_storage

for

sending_queue

when it is exported . But that doesn't seem to be working, is there something we can do at signoz otel-collector to dedup when data is ingested into Clickhouse?