can we create custom clickhouse tables to store ap...
# supports
Shivam Soni
08/21/2023, 2:09 PMcan we create custom clickhouse tables to store application logs?
v
Vishal Sharma
08/22/2023, 1:57 AMSorry I didn’t understand why you want to create custom Clickhouse tables for logs? What’s your usecase?
cc: @nitya-signoz
s
Shivam Soni
08/22/2023, 7:07 AMBecause we are using our own format for logs and don’t need extra things that signoz save
Shivam Soni
08/22/2023, 7:07 AMI think we can save space if we use our own format for logs instead of generic one provided by signoz
v
Vishal Sharma
08/22/2023, 7:09 AMYou will need current schema to support queries.
You go with space optimized schema then the performance will not be as good.
Vishal Sharma
08/22/2023, 7:10 AMYou can move data to S3 to save costs.
s
Shivam Soni
08/22/2023, 7:11 AMIf ,I create custom schema to store logs .Then ,We need to do required changes in query layers also??
v
Vishal Sharma
08/22/2023, 7:11 AMYes and at otel collector too
n
nitya-signoz
08/22/2023, 7:11 AMFew questions:-
• What is the different in space occupied that you are seeing between your and signoz schema
• How do you plan to ingested data in custom schema.
• Our query layer and frontend is written for custom schema, do you not intend to use signoz ui in that case ?
s
Shivam Soni
08/22/2023, 7:14 AM Copy code
{
timestamp : '',
requestId : '',
functionName : "",
thisnameSpace : '',
message : '',
} this are the only fields we want to store in clickhouse customLogs tableShivam Soni
08/22/2023, 7:15 AMWe are using pino logger in our application and transport to signoz otel collector .
Shivam Soni
08/22/2023, 7:16 AMAnd need to transport these logs to clickhouse customLogs table
Shivam Soni
08/22/2023, 7:17 AMand then we are able to filter our logs on requestId,functionname, and partinal string serach on message
Shivam Soni
08/22/2023, 7:18 AMJust want to know how much efforts it will take to setup this configguration through signoz ??
n
nitya-signoz
08/22/2023, 7:19 AMI don’t think space wise there will be much difference.
s
Shivam Soni
08/22/2023, 7:19 AMAnd changes that we need to do on signoz code base??
n
nitya-signoz
08/22/2023, 7:19 AMit will be mostly entire codebase rewrite.
s
Shivam Soni
08/22/2023, 7:21 AMokay
2 Views