https://signoz.io logo
Join Slack
Powered by
We started using Signoz in production yesterday We...
# support
s
We started using Signoz in production yesterday We have data coming from 4 clusters to one signoz It consumed 20GB in 10 hours How to know which logs or metrics of which cluster and pod consumed how much space
Logs of signoz clickhouse itself is consuming the most space than all our production data Can we get some help here
Getting this error in signoz clickhouse pod continuously
signiz_clickhouse.log
s
Did you have materialised columns for 
k8s_namespace_name
?
Can you share the output of this query?
Copy code
SELECT
    database,
    table,
    formatReadableSize(sum(data_compressed_bytes) AS size) AS compressed,
    formatReadableSize(sum(data_uncompressed_bytes) AS usize) AS uncompressed,
    round(usize / size, 2) AS compr_rate,
    sum(rows) AS rows,
    count() AS part_count
FROM system.parts
WHERE (active = 1) AND (database LIKE '%') AND (table LIKE '%')
GROUP BY
    database,
    table
ORDER BY size DESC;
a
Did you have materialised columns for 
k8s_namespace_name
?
this could be coming from logs when you convert an interesting field to selected field cc @nitya-signoz
n
There might some issue with clickhouse, can you run this command 
OPTIMIZE TABLE signoz_logs.logs final
s
@Srikanth Chekuri
@nitya-signoz run the command
Also column "attributes_string_value" is consuming 5 to 6 times more data than "body"
how to reduce attributes and resources data consumption they are consuming more data than actual logs(body)
n
The size of attributes depends on the attributes that you have extracted from your logs, you will have to check your logs configuration about what attributes are you extracting from logs.
s
we have deployed default otel collector no modifications
k8sinfra chart is deployed as is
n
Okay, so you will have to check what attributes you don’t want by default and then add remove operators in your config file to remove them from getting inserted. https://signoz.io/docs/userguide/collect_kubernetes_pod_logs/#filterexclude-logs You can also set retention settings for your logs here https://signoz.io/docs/userguide/retention-period/
s
ok thanks
Anyway to mitigate signoz-clickhouse pod consuming most data
n
You can use retention setting to set the time for which data is stored on disk, this will reduce disk usage.
s
@nitya-signoz we have not set any retention settings but yestarday's logs are gone and all users are also removed
n
Metrics and traces? are they present?
s
all are gone
n
It means the pv’s are wiped. @Prashant Shahi can you help here.
s
In that case, Wait first i will check with my devops team internally first, if they have done any changes or reinstallation
@nitya-signoz can you help me with excluding "log_file_path" in "attributes_string" ""attributes_string": { "log_file_path": "/var/log/pods/..."
Copy code
k8s-infra:
presets:
  logCollection:
    # whether to enable log collection
    enabled: true
    blacklist:
      # whether to enable blacklisting
      enabled: true
      additionalExclude: ["log_file_path"]
Is this enough
n
For that you will have to convert the value of this https://github.com/SigNoz/charts/blob/dc36a2e33c4887db2f59ddb2748a4b67927a28f9/charts/k8s-infra/values.yaml#L77 to false. And comment out https://github.com/SigNoz/charts/blob/dc36a2e33c4887db2f59ddb2748a4b67927a28f9/charts/k8s-infra/values.yaml#L125 to line no 147. Also replace 
extract_metadata_from_filepath
which will finally become like this
Copy code
presets:
  loggingExporter:
    enabled: false
  otlpExporter:
    enabled: true
  logsCollection:
    enabled: true
    include:
      - /var/log/pods/*/*/*.log
    startAt: beginning
    includeFilePath: false
    includeFileName: false
    blacklist:
      enabled: true
      signozLogs: true
      namespaces:
        - kube-system
      pods:
        - hotrod
        - locust
      containers: []
      additionalExclude: []
    operators:
      # Find out which format is used by kubernetes
      - type: router
        id: get-format
        routes:
          - output: parser-docker
            expr: 'body matches "^\\{"'
          - output: parser-crio
            expr: 'body matches "^[^ Z]+ "'
          - output: parser-containerd
            expr: 'body matches "^[^ Z]+Z"'
      # Parse CRI-O format
      - type: regex_parser
        id: parser-crio
        regex: '^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$'
        output: move-body
        timestamp:
          parse_from: attributes.time
          layout_type: gotime
          layout: '2006-01-02T15:04:05.000000000-07:00'
      # Parse CRI-Containerd format
      - type: regex_parser
        id: parser-containerd
        regex: '^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$'
        output: move-body
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
      # Parse Docker format
      - type: json_parser
        id: parser-docker
        output: move-body
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
      # Clean up log body
      - type: move
        id: move-body
        from: attributes.log
        to: body
s
thanks
p
It means the pv’s are wiped. @Prashant Shahi can you help here.
@Shravan Kgl can you share what happened here to cause that? Did anyone from team triggered upgrade command or removed PVs manually?
s
123 Views
Open in Slack
PreviousNext
Powered by