This message was deleted.

I understand that SigNoz provides a method to forward CloudWatch logs, but I'm interested in bypassing CloudWatch completely. https://signoz.io/docs/userguide/send-cloudwatch-logs-to-signoz/ The reason for this is that, even after rationalizing the CloudWatch logs, they still incur significant costs in a production environment, somewhere between $10,000 to $15,000 USD.

@Chitransh Gupta @Priyansh if you have some idea here

sure @S Sandhu, it is possible to bypass cloudwatch completely. Here’s a workflow you can try. Have some data in your S3 bucket → make a lambda function in AWS → add trigger to the lamba function (your S3 bucket, set trigger as PUT/All object creation) → add required policy so you can run the lambda function → Load the objects from S3 → perform log parsing and convert them to JSON dictionary → send them to SigNoz otel-collector endpoint. This is a simple overview of how you can do it. We have in-draft full fledged blog on how to send any logs from S3 to signoz, I hope it will do live this week. @Ashu Just make sure the logs are either automatically being collected in S3, or you can upload it there and then make the transfer to signoz. Here are some links for you to. configure your services so you can send logs automatically to S3. • VPC Flow Logs to S3- https://docs.aws.amazon.com/vpc/latest/tgw/flow-logs-s3.html • ELB Logs Collection to S3 - https://docs.aws.amazon.com/elasticloadbalancing/latest/application/enable-access-logging.html#enable-access-logs • Lamba logs -> S3 https://aws.amazon.com/blogs/compute/using-aws-lambda-extensions-to-send-logs-to-custom-destinations/ • S3. logs -> S3 - https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-server-access-logging.html

Thanks for your help @Priyansh @Ashu that looks promising, and it's something I would explore. However there are two minor issues with this approach: The S3 logs exporter appears to not have been updated in a while and is not officially supported: https://github.com/aws-samples/aws-lambda-extensions/tree/main/s3-logs-extension-demo-zip-archive, apart from cost of running lambdas to parse the logs. Additionally, there is one problem with this approach: it would not capture metrics and traces. I was wondering if there is a native exporter for SigNoz that will export these similar to the one for Datadog: https://aws-otel.github.io/docs/partners/datadog and one for newrelic

# New Relic OTLP Collector configuration

exporters:

otlp:

endpoint: <http://otlp.nr-data.net:4317|otlp.nr-data.net:4317>

headers:

api-key: <YOUR_NEW_RELIC_LICENSE_KEY>

Anything in pipeline?

Additionally, there is one problem with this approach: it would not capture metrics and traces

yes this approach is only for logs

Thanks for the suggestions! @Priyansh Gives me something to work with Here are two approaches that I will try: 1) Parse and forward S3 logs to signoz as suggested by you. 2) Attaching the OpenTelemetry layer to AWS Lambda functions and configuring the OTLP HTTP protocol to send logs, metrics, and traces to the signoz endpoint: https://signoz-service-host.com:8082 Theoretically it seems possible. I was hoping I could some clarity on whether the second approach is viable and worth trying. But thanks for your help, will report my findings.

@S Sandhu refer to this rough draft, I have already achieved the poc to send logs via lambda - https://zriyansh.notion.site/zriyansh/Send-any-logs-from-S3-bucket-to-SigNoz-via-HTTP-06e3f71dc4ab465e8b2c65607e9dda77 it has code as well for you to try it out.