https://signoz.io logo
#general
Title
# general
s

S Sandhu

01/28/2024, 12:31 AM
I recently came across Siznoz, and it looks really exciting! It combines a few of my favorite tools and languages: ClickHouse, Golang, and React! I have a newbie question: Is it possible to configure AWS Lambda functions to send logs directly to Signoz OTLP collector using the OpenTelemetry Protocol, bypassing CloudWatch Logs? Most of my Lambda functions are written in Python and Java, and it seems that AWS Lambda has support for OTLP. https://aws-otel.github.io/docs/getting-started/lambda https://aws-otel.github.io/docs/components/otlp-exporter
I understand that SigNoz provides a method to forward CloudWatch logs, but I'm interested in bypassing CloudWatch completely. https://signoz.io/docs/userguide/send-cloudwatch-logs-to-signoz/ The reason for this is that, even after rationalizing the CloudWatch logs, they still incur significant costs in a production environment, somewhere between $10,000 to $15,000 USD.
a

Ashu

01/28/2024, 5:23 AM
@Chitransh Gupta @Priyansh if you have some idea here
p

Priyansh

01/28/2024, 3:53 PM
sure @S Sandhu, it is possible to bypass cloudwatch completely. Here’s a workflow you can try. Have some data in your S3 bucket → make a lambda function in AWS → add trigger to the lamba function (your S3 bucket, set trigger as PUT/All object creation) → add required policy so you can run the lambda function → Load the objects from S3 → perform log parsing and convert them to JSON dictionary → send them to SigNoz otel-collector endpoint. This is a simple overview of how you can do it. We have in-draft full fledged blog on how to send any logs from S3 to signoz, I hope it will do live this week. @Ashu Just make sure the logs are either automatically being collected in S3, or you can upload it there and then make the transfer to signoz. Here are some links for you to. configure your services so you can send logs automatically to S3. • VPC Flow Logs to S3- https://docs.aws.amazon.com/vpc/latest/tgw/flow-logs-s3.html • ELB Logs Collection to S3 - https://docs.aws.amazon.com/elasticloadbalancing/latest/application/enable-access-logging.html#enable-access-logs • Lamba logs -> S3 https://aws.amazon.com/blogs/compute/using-aws-lambda-extensions-to-send-logs-to-custom-destinations/ • S3. logs -> S3 - https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-server-access-logging.html
s

S Sandhu

01/29/2024, 6:04 AM
Thanks for your help @Priyansh @Ashu that looks promising, and it's something I would explore. However there are two minor issues with this approach: The S3 logs exporter appears to not have been updated in a while and is not officially supported: https://github.com/aws-samples/aws-lambda-extensions/tree/main/s3-logs-extension-demo-zip-archive, apart from cost of running lambdas to parse the logs. Additionally, there is one problem with this approach: it would not capture metrics and traces. I was wondering if there is a native exporter for SigNoz that will export these similar to the one for Datadog: https://aws-otel.github.io/docs/partners/datadog and one for newrelic
# New Relic OTLP Collector configuration
exporters:
otlp:
endpoint: <http://otlp.nr-data.net:4317|otlp.nr-data.net:4317>
headers:
api-key: <YOUR_NEW_RELIC_LICENSE_KEY>
Anything in pipeline?
I am wondering if this would work? OTEL provides http exporter, configure it to send logs, metrics to signoz service? https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter#otlphttp-exporter
Copy code
exporters:
  otlphttp:
    endpoint: https://<signoz-service-host>.com:8082
https://signoz.io/docs/userguide/send-logs-http/#construct-the-curl-request-1
p

Priyansh

01/29/2024, 7:51 AM
Additionally, there is one problem with this approach: it would not capture metrics and traces
yes this approach is only for logs
OTEL provides http exporter, configure it to send logs, metrics to signoz service?
yes, we would be using this only
The S3 logs exporter appears to not have been updated in a while…
well that is something not in anyone’s control, signoz needs data ingested, the source does not matter
I was wondering if there is a native exporter for SigNoz that will export these similar to the one for Datadog:
this is not present right now imo
s

S Sandhu

01/29/2024, 8:54 AM
Thanks for the suggestions! @Priyansh Gives me something to work with Here are two approaches that I will try: 1) Parse and forward S3 logs to signoz as suggested by you. 2) Attaching the OpenTelemetry layer to AWS Lambda functions and configuring the OTLP HTTP protocol to send logs, metrics, and traces to the signoz endpoint: https://signoz-service-host.com:8082 Theoretically it seems possible. I was hoping I could some clarity on whether the second approach is viable and worth trying. But thanks for your help, will report my findings.
p

Priyansh

01/29/2024, 9:07 AM
@S Sandhu refer to this rough draft, I have already achieved the poc to send logs via lambda - https://zriyansh.notion.site/zriyansh/Send-any-logs-from-S3-bucket-to-SigNoz-via-HTTP-06e3f71dc4ab465e8b2c65607e9dda77 it has code as well for you to try it out.