This message was deleted.

Does it produce any telemetry data?

@Srikanth Chekuri I would be interested this question too, spark jobs can produce prometheus metrics https://spark.apache.org/docs/latest/monitoring.html Spark jobs can produce metrics by turning this feature on.

spark.ui.prometheus.enabled=true

In my case the spark jobs run on EKS cluster. Additionally, fluent bit is used as logs processor and forwarder at cluster level.

If they produce prometheus metrics, you can configure the receiver to collect metrics https://signoz.io/docs/userguide/send-metrics/#enable-a-prometheus-receiver

thanks @Srikanth Chekuri can we send the logs from spark pods to signoz? My setup at the moment uses fluent bit.

See https://docs.fluentbit.io/manual/pipeline/outputs/opentelemetry