question on multiline parsing (from here:https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver#example---multiline-logs-parsing). or maybe a best practice question, overall. i'm assuming we can't configure multiple config blocks of identical type. so if we're configuring multiple multiline definitions, do we just make a list? basically, we're going to be ingesting different log types via

filelog/app

and will need to define multiple multilines. also, can you mix filelog ingestion of single and multiline logs? 1. to define multiple multiline configurations, do we just create a list like

[ regex1, regex2, ...]

2. can you ingest single and multiline logs easily by defining just the multiline config, and anything else is considered single line?

@nitya-signoz Do you have any suggestions here/

To keep the configurations simple it will be a better idea to create multiple receivers. eg:- •

filelog/app1

for logs of type x •

filelog/app2

for logs of type y this will help you keep your configurations easier to manage. regarding this https://signoz-community.slack.com/archives/C01HWQ1R0BC/p1709828210950359?thread_ts=1709754971.177789&cid=C01HWQ1R0BC the config looks correct, can you share me a part of the file, I will try the config locally.

@nitya-signoz - here you go:

14:31:54 Trace started for  on  12/7/2023
14:31:54 Unable to establish a connection to the database. It may be down. 

Connection string=Provider=OraOLEDB.Oracle;Data Source=db01;USER ID=dbuser;Password=dbpass;

Error=ORA-12560: TNS:protocol adapter error

Stack trace=   at System.Data.OleDb.OleDbConnectionInternal..ctor(OleDbConnectionString constr, OleDbConnection connection)
   at System.Data.OleDb.OleDbConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningObject)
   at System.Data.ProviderBase.DbConnectionFactory.CreateNonPooledConnection(DbConnection owningConnection, DbConnectionPoolGroup poolGroup, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionInternal.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
   at System.Data.OleDb.OleDbConnection.Open()
   at V02Net.CSIBasicWeb.getSessMgrConn() in C:\M5\M5-AppCode\App_Code\CSIBasicWeb.cs:line 1856
14:31:54 Trace ended

i'd also have questions about configuring multiple multiline formats. maybe i'm not reading the documentation very well - but it looks like we can configure separate log formats for ingestion w/ separate blocks of

filelog/XYZ

? is there an order of precedence? for example, if i wanted to ingest as multiline based on the 24hr timestamp, but also, ingest multiline as starting with

14:31:54 Trace started

and ending with

14:31:54 Trace ended

?

I tried processing the log file https://signoz-community.slack.com/archives/C01HWQ1R0BC/p1710349681753369?thread_ts=1709754971.177789&cid=C01HWQ1R0BC , against the multiline config that you shared. It working as expected. 1st - 143154 Trace started for on 12/7/2023 2nd - 143154 Unable to establish … 3rd - 143154 Trace ended From the second question that you mentioned do you mean the above three logs should be a single log line ? Or have I got it wrong.

yeah - i had to remove all the old data and rebuild the signoz install from scratch in order to get the config change for that

filelog

multiline block to properly process or display previously ingested logs. there may be a better way to apply an updated config to existing, ingested logs but i couldn't find one. so that's working now. as far as having multiple configs for multiple apps, what's best practice for managing that as you onboard new apps, new logs, etc? just update the

otel-collector-config.yaml

and rerun docker compose? final question, is there documentation out there outlining how you can configure multiple apps using filelog? the documentation i read just made it seem like you have a single block of

filelog/app

- but now it seems that you can do multiple blocks of

filelog/app1

, `filelog/app2`` ...

filelog/appX

for receivers, yes you need to update

otel-collector-config.yaml

and restart the otel-collector. While some parts of the config(pipelines) are updated directly from the UI but manually updating the file needs a restarts. I got your issue with the documentation, it’s mentioned in a small portion in opentelemetry docs

<https://opentelemetry.io/docs/collector/configuration/>

Note that receivers, processors, exporters and pipelines are defined through component identifiers following the type[/name] format, for example otlp or otlp/2. You can define components of a given type more than once as long as the identifiers are unique

@Chitransh Gupta we should add the above to our documentation

thank you. the tool, so far, seems pretty great, but yeah there do seem to be some gaps in documentation. help here in slack has been good, but would be nice to have a full config doc w/ real world examples including configuration for multiple sources of the same type of log/metric/etc. question re signoz cloud - are configuration options found in

otel-collector-config.yaml

and the like performed thru a web console instead?

Even for signoz cloud, you will need to have one collector running in your machine, and that you have to configure it the same way you did for self hosted i.e using otel-collector-config.yaml