https://signoz.io logo
Join Slack
Powered by
I have just updated my retention period on my clic...
# support
m
I have just updated my retention period on my clickhouse data but data is not clearing. Is this triggered on a daily cronjob or something? How can I trigger the actual removal? (my PVs are running out of storage)
s
Updated TTL applies only for the ingested after the update is made.
m
Do you have some cleanup script somewhere?
I updated TTL from 15 to 10 since our cluster is going full
Copy code
pvc-b507399a-0f69-463c-96ed-b7937e2f5ac1  data-volumeclaim-template-chi-signoz-clickhouse-cluster-2-0-0                                           infra      aks-default-16275264-vmss00000f  chi-signoz-clickhouse-cluster-2-0-0                 data-volumeclaim-template                            589Gi  540Gi   48Gi       91.70  879055  38442545  2.24   
 pvc-0b410112-7ff9-4f90-892a-807bda3a4ed1  data-volumeclaim-template-chi-signoz-clickhouse-cluster-1-0-0                                           infra      aks-default-16275264-vmss00000k  chi-signoz-clickhouse-cluster-1-0-0                 data-volumeclaim-template                            589Gi  526Gi   62Gi       89.38  860516  38461084  2.19   
 pvc-ac1fbfae-f8e7-4bc2-81a7-10e993e2eef7  data-volumeclaim-template-chi-signoz-clickhouse-cluster-0-0-0                                           infra      aks-default-16275264-vmss00000j  chi-signoz-clickhouse-cluster-0-0-0                 data-volumeclaim-template                            589Gi  551Gi   38Gi       93.50  880991  38440609  2.24
+-40GB left
s
Can you exec into them and share the output of
Copy code
SELECT
    database,
    table,
    formatReadableSize(sum(data_compressed_bytes) AS size) AS compressed,
    formatReadableSize(sum(data_uncompressed_bytes) AS usize) AS uncompressed,
    round(usize / size, 2) AS compr_rate,
    sum(rows) AS rows,
    count() AS part_count
FROM system.parts
WHERE (active = 1) AND (database LIKE '%') AND (table LIKE '%')
GROUP BY
    database,
    table
ORDER BY size DESC;
m
Copy code
┌─database─────────┬─table───────────────────────┬─compressed─┬─uncompressed─┬─compr_rate─┬────────rows─┬─part_count─┐
│ signoz_logs      │ logs_v2                     │ 259.44 GiB │ 2.66 TiB     │      10.52 │  1449884785 │        171 │
│ signoz_metrics   │ samples_v4                  │ 71.74 GiB  │ 1.36 TiB     │      19.35 │ 53268158976 │        238 │
│ signoz_traces    │ signoz_index_v3             │ 53.13 GiB  │ 604.67 GiB   │      11.38 │   553459000 │        142 │
│ signoz_metrics   │ samples_v4_agg_5m           │ 33.77 GiB  │ 367.89 GiB   │       10.9 │  6584686353 │        206 │
│ signoz_metrics   │ time_series_v4              │ 22.86 GiB  │ 826.97 GiB   │      36.18 │   972656951 │        186 │
│ signoz_metrics   │ time_series_v4_6hrs         │ 13.29 GiB  │ 369.09 GiB   │      27.77 │   535670092 │        173 │
│ signoz_metrics   │ samples_v4_agg_30m          │ 11.50 GiB  │ 84.02 GiB    │        7.3 │  1503593624 │        189 │
│ signoz_metrics   │ time_series_v4_1day         │ 11.07 GiB  │ 295.28 GiB   │      26.68 │   464399671 │        160 │
│ signoz_metrics   │ time_series_v4_1week        │ 7.55 GiB   │ 217.21 GiB   │      28.76 │   424821560 │         35 │
│ system           │ zookeeper_log               │ 7.48 GiB   │ 58.62 GiB    │       7.84 │   197521387 │          7 │
│ signoz_traces    │ trace_summary               │ 7.11 GiB   │ 13.65 GiB    │       1.92 │   257061077 │        106 │
│ system           │ query_views_log             │ 5.95 GiB   │ 74.03 GiB    │      12.44 │    60709168 │         95 │
│ system           │ asynchronous_metric_log     │ 1.42 GiB   │ 17.13 GiB    │      12.08 │  1081735857 │         96 │
│ system           │ metric_log                  │ 1.10 GiB   │ 3.08 GiB     │        2.8 │     2565533 │         79 │
│ system           │ query_log                   │ 1.07 GiB   │ 5.50 GiB     │       5.15 │     7249249 │         11 │
│ signoz_traces    │ dependency_graph_minutes_v2 │ 802.75 MiB │ 1.33 GiB     │        1.7 │      151698 │         71 │
│ system           │ trace_log                   │ 649.16 MiB │ 7.25 GiB     │      11.44 │    24359142 │          6 │
│ system           │ part_log                    │ 558.24 MiB │ 3.87 GiB     │       7.09 │     7484388 │         13 │
│ signoz_logs      │ tag_attributes_v2           │ 320.83 MiB │ 2.73 GiB     │       8.71 │    40217643 │         44 │
│ signoz_metrics   │ metadata                    │ 166.79 MiB │ 1.08 GiB     │       6.66 │    11305479 │         81 │
│ signoz_logs      │ logs_v2_resource            │ 19.30 MiB  │ 450.04 MiB   │      23.32 │      762550 │          8 │
│ signoz_traces    │ tag_attributes_v2           │ 2.61 MiB   │ 31.34 MiB    │      12.01 │      674502 │         23 │
│ system           │ session_log                 │ 2.26 MiB   │ 11.72 MiB    │       5.19 │       78548 │         44 │
│ signoz_traces    │ signoz_error_index_v2       │ 344.96 KiB │ 10.24 MiB    │       30.4 │        2032 │         24 │
│ signoz_analytics │ rule_state_history_v0       │ 168.45 KiB │ 2.15 MiB     │      13.08 │        4572 │         47 │
│ signoz_traces    │ traces_v3_resource          │ 75.64 KiB  │ 20.57 MiB    │     278.41 │       20998 │          1 │
│ signoz_traces    │ top_level_operations        │ 33.86 KiB  │ 155.95 KiB   │       4.61 │        3638 │          7 │
│ signoz_logs      │ usage                       │ 11.18 KiB  │ 16.39 KiB    │       1.47 │          78 │          3 │
│ signoz_metrics   │ usage                       │ 9.37 KiB   │ 13.86 KiB    │       1.48 │          66 │          2 │
│ signoz_traces    │ usage                       │ 9.05 KiB   │ 13.68 KiB    │       1.51 │          66 │          2 │
│ signoz_traces    │ span_attributes_keys        │ 1.07 KiB   │ 1.99 KiB     │       1.85 │          99 │          1 │
│ signoz_logs      │ schema_migrations_v2        │ 512.00 B   │ 358.00 B     │        0.7 │          10 │          2 │
│ signoz_metrics   │ schema_migrations_v2        │ 474.00 B   │ 570.00 B     │        1.2 │          17 │          1 │
│ signoz_traces    │ schema_migrations_v2        │ 442.00 B   │ 504.00 B     │       1.14 │          15 │          1 │
│ signoz_analytics │ schema_migrations_v2        │ 350.00 B   │ 94.00 B      │       0.27 │           2 │          2 │
│ signoz_metadata  │ schema_migrations_v2        │ 350.00 B   │ 94.00 B      │       0.27 │           2 │          2 │
└──────────────────┴─────────────────────────────┴────────────┴──────────────┴────────────┴─────────────┴────────────┘
s
I want to see what is taking up the space so we can act accordingly
Which tables did you try update the retention for? As a next we can try dropping the entire partitions for the old data
m
I updated it through the general settings
metrics/traces/logs
just need some space until the old collector is decommissioned in your next release
s
To get the partitions
Copy code
SELECT DISTINCT partition
FROM system.parts
WHERE (database = 'signoz_logs') AND (table = 'logs_v2') AND active
ORDER BY partition ASC
To drop some partition
Copy code
ALTER TABLE signoz_logs.logs_v2 DROP PARTITION '...';
m
do I need to do this on all clickhouse servers?
or will this be synced if I remove on 1 node
s
Add 
ON CLUSTER cluster
for the alter table and it should trigger on all shards
m
Copy code
ALTER TABLE signoz_logs.logs_v2 DROP PARTITION '2025-06-23' ON CLUSTER cluster;
s
No before DROP
Copy code
ALTER TABLE signoz_logs.logs_v2 ON CLUSTER cluster DROP PARTITION '...';
m
thanks Srikanth
this should give us some room
Copy code
┌─database─────────┬─table───────────────────────┬─compressed─┬─uncompressed─┬─compr_rate─┬────────rows─┬─part_count─┐
│ signoz_logs      │ logs_v2                     │ 176.03 GiB │ 1.79 TiB     │      10.39 │   954662193 │        132 │
│ signoz_metrics   │ samples_v4                  │ 71.76 GiB  │ 1.36 TiB     │      19.35 │ 53275283764 │        238 │
│ signoz_traces    │ signoz_index_v3             │ 53.14 GiB  │ 604.77 GiB   │      11.38 │   553554713 │        140 │
│ signoz_metrics   │ samples_v4_agg_5m           │ 33.78 GiB  │ 367.94 GiB   │      10.89 │  6585558494 │        210 │
│ signoz_metrics   │ time_series_v4              │ 22.87 GiB  │ 827.23 GiB   │      36.17 │   972895990 │        188 │
│ signoz_metrics   │ time_series_v4_6hrs         │ 13.30 GiB  │ 369.35 GiB   │      27.76 │   535909127 │        170 │
│ signoz_metrics   │ samples_v4_agg_30m          │ 11.50 GiB  │ 84.01 GiB    │        7.3 │  1503580539 │        182 │
│ signoz_metrics   │ time_series_v4_1day         │ 11.08 GiB  │ 295.54 GiB   │      26.67 │   464640664 │        160 │
│ signoz_metrics   │ time_series_v4_1week        │ 7.57 GiB   │ 217.47 GiB   │      28.74 │   425060200 │         35 │
│ system           │ zookeeper_log               │ 7.55 GiB   │ 59.15 GiB    │       7.84 │   199241226 │         10 │
│ signoz_traces    │ trace_summary               │ 7.12 GiB   │ 13.65 GiB    │       1.92 │   257118380 │        106 │
│ system           │ query_views_log             │ 5.95 GiB   │ 74.05 GiB    │      12.44 │    60727962 │         94 │
│ system           │ asynchronous_metric_log     │ 1.42 GiB   │ 17.14 GiB    │      12.08 │  1081944619 │         96 │
│ system           │ metric_log                  │ 1.10 GiB   │ 3.08 GiB     │        2.8 │     2566023 │         78 │
│ system           │ query_log                   │ 1.07 GiB   │ 5.54 GiB     │       5.15 │     7302406 │         10 │
│ signoz_traces    │ dependency_graph_minutes_v2 │ 803.05 MiB │ 1.34 GiB     │        1.7 │      151750 │         67 │
│ system           │ trace_log                   │ 654.66 MiB │ 7.31 GiB     │      11.44 │    24564507 │         10 │
│ system           │ part_log                    │ 562.99 MiB │ 3.90 GiB     │        7.1 │     7546524 │         12 │
│ signoz_logs      │ tag_attributes_v2           │ 320.86 MiB │ 2.73 GiB     │       8.71 │    40219605 │         46 │
│ signoz_metrics   │ metadata                    │ 166.23 MiB │ 1.08 GiB     │       6.64 │    11231576 │         81 │
│ signoz_logs      │ logs_v2_resource            │ 19.32 MiB  │ 450.29 MiB   │      23.31 │      762961 │          6 │
│ signoz_traces    │ tag_attributes_v2           │ 2.61 MiB   │ 31.35 MiB    │         12 │      674651 │         25 │
│ system           │ session_log                 │ 2.26 MiB   │ 11.73 MiB    │       5.19 │       78561 │         44 │
│ signoz_traces    │ signoz_error_index_v2       │ 344.96 KiB │ 10.24 MiB    │       30.4 │        2032 │         24 │
│ signoz_analytics │ rule_state_history_v0       │ 168.45 KiB │ 2.15 MiB     │      13.08 │        4572 │         47 │
│ signoz_traces    │ traces_v3_resource          │ 75.64 KiB  │ 20.57 MiB    │     278.41 │       20998 │          1 │
│ signoz_traces    │ top_level_operations        │ 32.94 KiB  │ 154.98 KiB   │        4.7 │        3617 │          4 │
│ signoz_logs      │ usage                       │ 11.18 KiB  │ 16.39 KiB    │       1.47 │          78 │          3 │
│ signoz_metrics   │ usage                       │ 9.37 KiB   │ 13.86 KiB    │       1.48 │          66 │          2 │
│ signoz_traces    │ usage                       │ 9.05 KiB   │ 13.68 KiB    │       1.51 │          66 │          2 │
│ signoz_traces    │ span_attributes_keys        │ 1.07 KiB   │ 1.99 KiB     │       1.85 │          99 │          1 │
│ signoz_logs      │ schema_migrations_v2        │ 512.00 B   │ 358.00 B     │        0.7 │          10 │          2 │
│ signoz_metrics   │ schema_migrations_v2        │ 474.00 B   │ 570.00 B     │        1.2 │          17 │          1 │
│ signoz_traces    │ schema_migrations_v2        │ 442.00 B   │ 504.00 B     │       1.14 │          15 │          1 │
│ signoz_analytics │ schema_migrations_v2        │ 350.00 B   │ 94.00 B      │       0.27 │           2 │          2 │
│ signoz_metadata  │ schema_migrations_v2        │ 350.00 B   │ 94.00 B      │       0.27 │           2 │          2 │
└──────────────────┴─────────────────────────────┴────────────┴──────────────┴────────────┴─────────────┴────────────┘
It seems like the retentions still dont apply properly 🤔
I have logs from back to 2nd of July and metrics for over a month while all my retentions are set to 10 days
image.png,image.png
s
When did you change the retention? The change only applies to new data after the change. The data ingested prior to retention will continue to have old retention setting.
m
a while ago
is there a command to drop all metrics/logs/traces older than 10 days?
s
m
and for metrics/traces, where are those stored primarily?
s
samples_v4
, 
signoz_index_v3
❤️ 1
2 Views
Open in Slack
PreviousNext
Powered by