https://signoz.io
Join Slack
hello everyone, i receive log entries via fluent bit for cpu, mem, and other application logs. i'm h...
# support
n
hello everyone, i receive log entries via fluent bit for cpu, mem, and other application logs. i'm having a hard time to understand how to visualize and aggregate these. for example, i have log entries like this:
Copy code
{
  "body": "{\"cpu0.p_cpu\":0.5,\"cpu0.p_system\":0,\"cpu0.p_user\":0.5,\"cpu1.p_cpu\":0.5,\"cpu1.p_system\":0.1,\"cpu1.p_user\":0.4,\"cpu_p\":0.55,\"environment\":\"production\",\"hostname\":\"cf-production-app-02\",\"system_p\":0.05,\"user_p\":0.5}",
  "id": "2nsj5xS7GmYAZ0LzWrZ582QWqbQ",
  "timestamp": "2024-10-24T11:14:57.298656942Z",
  "attributes": {},
  "resources": {},
  "severity_text": "",
  "severity_number": 0,
  "span_id": "",
  "trace_flags": 0,
  "trace_id": ""
}
how can i, say, create a simple "value" panel that shows me the average for 
system_p
over the selected time period? the problem is that i cannot use 
Avg
in the query builder with a sub-field of body it seems. i cannot set 
Avg
- 
body.system_p
. i get a 500. only thing i have managed is displaying the amount of log entries, but nothing about the content of an entry. i'm using self-hosted signoz and ingest logs via the otel-collector on port 4318. i'm very thankful for a hint! 🙂
n
hi, please use pipelines to parse your log and convert to attributes https://signoz.io/docs/logs-pipelines/introduction/
then you can do aggregations on those attributes.
there is a guide for json as well https://signoz.io/docs/logs-pipelines/guides/json/
n
hey nitya, thanks for your reply - will try this out, thx!
well, those pipelines seem a bit flaky. i have a very simple pipeline that matches 
severity_number = 0
. i add an "Add" processor that adds the field 
attributes.foo
with value 
foo
. but it doesn't show in the previewed processings. i must be doing something essential wrong...
also there seems to be a bug: when i use this as a filter: 
body.environment = production
i get all the preview results. but when i save this and try to add a json processor (just like the docs suggest), i get this error a lot:
Copy code
{
  "level": "error",
  "ts": 1730115251.3416162,
  "caller": "router/transformer.go:59",
  "msg": "Failed to process entry",
  "kind": "processor",
  "name": "logstransform/pipeline_test",
  "pipeline": "logs",
  "operator_id": "router_signoz",
  "operator_type": "router",
  "error": "running if expr: interface conversion: interface {} is map[string]interface {}, not string (1:21)\n | body != nil && body matches \"^\\\\s*{.*}\\\\s*$\"\n | ....................^",
  "stacktrace": "<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/router.(*Transformer).Process|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/router.(*Transformer).Process>\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.111.0/operator/transformer/router/transformer.go:59\ngithub.com/open-telemetry/opentelemetry-collector-contrib/processor/logstransformprocessor.(*logsTransformProcessor).converterLoop\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/processor/logstransformprocessor@v0.111.0/processor.go:213"
}
n
@Raj Kamal please have a look at this ^
n
i think what i want to achieve isn't that hard, is it? i want these fields to become top-level attributes. isn't that what the JSON parser processor in the pipelines is for? do you have any idea why it's not working?
some more context:
appreciate the help! ❤️
n
I think it’s something to do with the the fluent receiver, it sends the body as an object and not as a string. Can you try moving one of them directly without using a json processor. i.e update the json processor to copy processor.
Copy code
From body["error_message"]
To attributes["error_message"]
n
doesn't seem to resolve it, see the screenshots -
the processed preview results still don't show any change
when i click on the first preview result, this is still what is shown:
what is this 
attributes_bool
key by the way? what does this tell me?
n
Which version of signoz are you using ?
n
the very latest, i have just run 
git pull
before starting the containers again.
Copy code
commit 952ab580234f9c8093d116f745603ca511d4c6a1 (HEAD -> develop, origin/develop, origin/HEAD)
Author: Srikanth Chekuri <srikanth.chekuri92@gmail.com>
Date:   Mon Oct 28 14:06:43 2024 +0530

    chore: bump signoz-otel-collector version (#6290)
this is how i have started it: https://signoz.io/docs/install/docker/#install-signoz-using-docker-compose i have just used the 
docker-compose-minimal.yaml
so i don't start the example apps.
i'll try again from scratch with the latest stable release 0.56.0 ...
n
It shouldn’t be a installation problem, I checking it out if I can help you more with the pipeline else Raj will help you out
Can you try this and check if something in preview is changing?
Copy code
from: body
            to: attributes.tmp
n
yes, give me a second
i get a 500 error when i try to preview. the otel-collector container shows this error:
Copy code
=> /tmp/signoz-otel-collector.err <==
{"level":"info","timestamp":"2024-10-28T12:02:48.278Z","caller":"signozcol/collector.go:121","msg":"Collector service is running"}
{"level":"error","timestamp":"2024-10-28T12:02:48.278Z","caller":"opamp/server_client.go:216","msg":"failed to apply config","component":"opamp-server-client","error":"failed to reload config: /var/tmp/collector-config.yaml: collector failed to restart: failed to build pipelines: failed to create \"logstransform/pipeline_test\" processor, in pipeline \"logs\": failed to set outputs on route: operator e42d25f2-fdf3-4b96-9666-7bb3d10788e1 does not exist: {\"operator_id\":\"router_signoz\"}","stacktrace":"github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:216\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onMessageFuncHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:199\ngithub.com/open-telemetry/opamp-go/client/types.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/types/callbacks.go:162\ngithub.com/open-telemetry/opamp-go/client/internal.(*receivedProcessor).ProcessReceivedMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/receivedprocessor.go:131\ngithub.com/open-telemetry/opamp-go/client/internal.(*wsReceiver).ReceiverLoop\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/wsreceiver.go:57\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runOneCycle\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:243\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runUntilStopped\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:265\ngithub.com/open-telemetry/opamp-go/client/internal.(*ClientCommon).StartConnectAndRun.func1\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/clientcommon.go:197"}
n
try disabling the pipeline and applying it ?
n
error remains. i guess the config is messed up somewhere, because there is no 
piepline_test
anymore. i have removed it and renamed the processors.
i will recreate the setup cleanly.
will get back to you then
r
Hi Nico 🙂 You can't use a severity based filters in pipelines in the version you are using This will be fixed in the next open source release
Regarding the latest 500 error you have pasted. That error tends to happen if the first operator in the pipeline is disabled - the fix for that is also already in develop branch and will be released soon - but in the meantime, if your first pipeline operator is disabled, try deleting it and trying again
n
hey raj, thanks for reaching out. what would be the appropriate way then here? apparently i cannot use 
body.environment EXISTS
either, or something like that. i ingest my logs through opentelemetry, and all information i can influence is the content of the 
body
. => what can i do to run a pipeline over all logs that are coming in?
r
for now, you should be able to use 
body != nil
as a filter - that will match all logs except logs with body containing a string with value "nil"
there is an open issue for addressing this, we should be taking that up soon
n
alright, cool, thx! with this, the suggestion of @nitya-signoz worked:
now let's see if the json parser processor works...
n
Instead of a JSON parser try moving from 
attributes.tmp.<key>
directly and check as well?
r
The json processor error you mentioned should be happening because the body in logs being received is not a string but an object SigNoz converts bodies to strings before storing them in the DB. Since the sample for pipeline previews is fetched from the DB (with bodies already converted to string), the preview doesn't run into that error
so try using the body as a map/object without parsing it with a JSON processor
n
do you mean like this?
r
yes. did that work?
The preview will not work, you will have to see what happens to actual logs after saving the pipeline
n
yes, that works. however, isn't that rather cumbersome? i thought that's exactly what the JSON processor is there for?
and also - a side question - how can i make use of this filter here directly?
r
Ya this is also being tracked as an issue This one is not so simple because there is no simple way to get historical actual raw/unprocessed logs for pipeline preview samples - since the DB only stores processed logs
n
it's there in the actual logs:
n
That environment is mapped to 
resources["deployment.enviroment"]
if you move your environment here it will work.
n
cool, will try this out. is there a doc resource somewhere that shows these mappings?
couldn't find one so far
n
Don’t think we have one, these are based on otel standards, but thanks for pointing it out. Will create one
n
alright. maybe that answer is also obvious for people more familiar with otel. i haven't used it before.
n
Makes sense, we definitely need a doc
n
okay, i'll see how far i can get from here. thank you so much both for your help!
That environment is mapped to 
resources["deployment.enviroment"]
if you move your environment here it will work.
this doesn't work BTW. the pipeline won't get deployed, it will show Deployment Failed in 
Change History
. will provide a screenshot in a sec
r
try using 
resource["deployment.enviroment"]
n
my bad can you change it to resource instead of resources?
r
"resource" not "resources"
n
okay, that leaves me with this: i still cannot search for these in the log explorer.
r
You will have to use the 
[]
. 
resource["deployment.enviroment"]
n
it looks different when i use the notation you have suggested:
however, it's still not searchable
ah, because typo... 1sec
now it works. perfect 🙂
thank you, guys! much appreciated. ❤️
the one for hostname, is it 
resource["deployment.hostname"]
?
ah, got it. i guess this would be the docs for this: https://opentelemetry.io/docs/specs/semconv/resource/
ok, but neither 
resource["deployment.hostname"]
nor 
resource["deployment.host.name"]
or 
resource["deployment.host"]
seem to work for this field. what's the expected name, @Raj Kamal? 😄
ah, it's without deployment i guess. sorry for spamming you guys. i guess that's what you get when you start answering support requests on slack. 😛
(still no luck with the Hostname field...)
r
@nitya-signoz should be able to help you with this one
no worries about the messages. Thanks for putting up with the imperfections and incoveniences
n
it’s 
resource["hostname"]
n
mhh :(
another question - is there a way to export alert- and pipeline configurations as JSON or something similar to what you can do with dashboards? i did a whole lot of clicking around today and i'd like to keep the config somewhere as code
n
Yeah, this mapping is incorrect, someone from the FE team will fix this in the next release. cc @Vikrant Gupta For now, you can map it to attributes, but I would suggest keeping it in resources as that’s the correct place.
n
hi guys - i wonder if there' a way to rotate the logs somehow? because the clickhouse DB grows rapidly and i haven't found anything about this so far.
what's operation best practices for signoz self-hosted is what i'm essentially asking i guess.
n
You can check out this guide for retention period https://signoz.io/docs/userguide/retention-period/
39 Views
Open in Slack
PreviousNext
Powered by

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.

Powered by