question on multiline parsing (from here:<https://...
# support
b
question on multiline parsing (from here:https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver#example---multiline-logs-parsing). or maybe a best practice question, overall. i'm assuming we can't configure multiple config blocks of identical type. so if we're configuring multiple multiline definitions, do we just make a list? basically, we're going to be ingesting different log types via
filelog/app
and will need to define multiple multilines. also, can you mix filelog ingestion of single and multiline logs? 1. to define multiple multiline configurations, do we just create a list like
[ regex1, regex2, ...]
2. can you ingest single and multiline logs easily by defining just the multiline config, and anything else is considered single line?
well here's what i have for the filelog
receivers
config:
Copy code
filelog/app:
    include: ["/cloudadmins/logs/20240304_process_monitor.log","/cloudadmins/logs/**/*.txt"]
    start_at: beginning
    multiline:
      line_start_pattern: '^([01][01]?[0-9]|2[0-3]):[0-5][0-9]:[0-5][0-9]'
but when it comes to logs such as the following, it's not capturing the multiline output.
Copy code
16:40:32 scriptName=/M5WEB/REPORTS/WORKREQUESTS/WOWORKREQPARM.ASPX ip=10.208.61.9 ==>saveSession failure<== Caught critical error==>Thread was being aborted.<== trace=   at System.Data.Common.UnsafeNativeMethods.ICommandText.Execute(IntPtr pUnkOuter, Guid& riid, tagDBPARAMS pDBParams, IntPtr& pcRowsAffected, Object& ppRowset)
   at System.Data.OleDb.OleDbCommand.ExecuteCommandTextForMultpleResults(tagDBPARAMS dbParams, Object& executeResult)
   at System.Data.OleDb.OleDbCommand.ExecuteCommandText(Object& executeResult)
   at System.Data.OleDb.OleDbCommand.ExecuteReaderInternal(CommandBehavior behavior, String method)
   at System.Data.OleDb.OleDbCommand.ExecuteNonQuery()
   at V02Net.CSISession.saveSessionState(String delWindow) in C:\M5\M5-AppCode\App_Code\Session\CSISession.cs:line 551
anyone able to identify something wrong with the filelog config or my regex for multiline?
it's definitely valid regex for the logs in question
p
@nitya-signoz Do you have any suggestions here/
n
To keep the configurations simple it will be a better idea to create multiple receivers. eg:- •
filelog/app1
for logs of type x •
filelog/app2
for logs of type y this will help you keep your configurations easier to manage. regarding this https://signoz-community.slack.com/archives/C01HWQ1R0BC/p1709828210950359?thread_ts=1709754971.177789&amp;cid=C01HWQ1R0BC the config looks correct, can you share me a part of the file, I will try the config locally.
b
@nitya-signoz - here you go:
Copy code
14:31:54 Trace started for  on  12/7/2023
14:31:54 Unable to establish a connection to the database. It may be down. 

Connection string=Provider=OraOLEDB.Oracle;Data Source=db01;USER ID=dbuser;Password=dbpass;

Error=ORA-12560: TNS:protocol adapter error

Stack trace=   at System.Data.OleDb.OleDbConnectionInternal..ctor(OleDbConnectionString constr, OleDbConnection connection)
   at System.Data.OleDb.OleDbConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningObject)
   at System.Data.ProviderBase.DbConnectionFactory.CreateNonPooledConnection(DbConnection owningConnection, DbConnectionPoolGroup poolGroup, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionInternal.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)
   at System.Data.OleDb.OleDbConnection.Open()
   at V02Net.CSIBasicWeb.getSessMgrConn() in C:\M5\M5-AppCode\App_Code\CSIBasicWeb.cs:line 1856
14:31:54 Trace ended
i'd also have questions about configuring multiple multiline formats. maybe i'm not reading the documentation very well - but it looks like we can configure separate log formats for ingestion w/ separate blocks of
filelog/XYZ
? is there an order of precedence? for example, if i wanted to ingest as multiline based on the 24hr timestamp, but also, ingest multiline as starting with
14:31:54 Trace started
and ending with
14:31:54 Trace ended
?
n
I tried processing the log file https://signoz-community.slack.com/archives/C01HWQ1R0BC/p1710349681753369?thread_ts=1709754971.177789&cid=C01HWQ1R0BC , against the multiline config that you shared. It working as expected. 1st - 143154 Trace started for on 12/7/2023 2nd - 143154 Unable to establish … 3rd - 143154 Trace ended From the second question that you mentioned do you mean the above three logs should be a single log line ? Or have I got it wrong.
b
yeah - i had to remove all the old data and rebuild the signoz install from scratch in order to get the config change for that
filelog
multiline block to properly process or display previously ingested logs. there may be a better way to apply an updated config to existing, ingested logs but i couldn't find one. so that's working now. as far as having multiple configs for multiple apps, what's best practice for managing that as you onboard new apps, new logs, etc? just update the
otel-collector-config.yaml
and rerun docker compose? final question, is there documentation out there outlining how you can configure multiple apps using filelog? the documentation i read just made it seem like you have a single block of
filelog/app
- but now it seems that you can do multiple blocks of
filelog/app1
, `filelog/app2`` ...
filelog/appX
basically, our rough idea is to test signoz as a POC with an app, then maybe add some more, then maybe try the cloud offering 30 day trial once we have established some processes and guidelines, gotten experience managing, etc
n
for receivers, yes you need to update
otel-collector-config.yaml
and restart the otel-collector. While some parts of the config(pipelines) are updated directly from the UI but manually updating the file needs a restarts. I got your issue with the documentation, it’s mentioned in a small portion in opentelemetry docs
<https://opentelemetry.io/docs/collector/configuration/>
Copy code
Note that receivers, processors, exporters and pipelines are defined through component identifiers following the type[/name] format, for example otlp or otlp/2. You can define components of a given type more than once as long as the identifiers are unique
@Chitransh Gupta we should add the above to our documentation
b
thank you. the tool, so far, seems pretty great, but yeah there do seem to be some gaps in documentation. help here in slack has been good, but would be nice to have a full config doc w/ real world examples including configuration for multiple sources of the same type of log/metric/etc. question re signoz cloud - are configuration options found in
otel-collector-config.yaml
and the like performed thru a web console instead?
n
Even for signoz cloud, you will need to have one collector running in your machine, and that you have to configure it the same way you did for self hosted i.e using otel-collector-config.yaml