https://signoz.io logo
Join Slack
Powered by
Hello, I'm new to signoz and opentelemetry I deplo...
# support
v
Hello, I'm new to signoz and opentelemetry I deployed signoz on kubernetes using the helm chart And I have a Java Spring application running with the open telemetry java agent So far so good the java app is able to send the traces over grpc, it works well. Then I wanted to setup oauth authentication between the agent and otel collector. But the documentation at https://opentelemetry.io/docs/collector/configuration/#authentication is very succint and I was not able to reproduce the setup. Can you help me figure this out ?
Currently this is my signoz collector config. I have taken the original config from the helm chart and added otlp/auth receiver, extension and pipeline as explained in the documentation :
Copy code
receivers:
  otlp/spanmetrics:
    protocols:
      grpc:
        endpoint: localhost:12345
  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
      thrift_http:
        endpoint: 0.0.0.0:14268
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu: {}
      load: {}
      memory: {}
      disk: {}
      filesystem: {}
      network: {}
  otlp/auth:
    protocols:
      http:
        endpoint: 0.0.0.0:4317
        auth:
          authenticator: oidc
processors:
  batch:
    send_batch_size: 1000
    timeout: 10s
  signozspanmetrics/prometheus:
    metrics_exporter: prometheus
    latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 50ms, 100ms, 250ms, 500ms, 1000ms, 1400ms, 2000ms, 5s, 10s, 20s, 40s, 60s]
    dimensions_cache_size: 10000
    dimensions:
      - name: service.namespace
        default: default
      - name: deployment.environment
        default: default
  # memory_limiter:
  #   # 80% of maximum memory up to 2G
  #   limit_mib: 1500
  #   # 25% of limit up to 2G
  #   spike_limit_mib: 512
  #   check_interval: 5s
  #
  #   # 50% of the maximum memory
  #   limit_percentage: 50
  #   # 20% of max memory usage spike expected
  #   spike_limit_percentage: 20
  # queued_retry:
  #   num_workers: 4
  #   queue_size: 100
  #   retry_on_failure: true
extensions:
  health_check: {}
  zpages: {}
  oidc:
    issuer_url: <https://auth.review.mydomain.com/>
    audience: <https://api.mydomain.com/>
exporters:
  clickhouse:
    datasource: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_TRACE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}
  clickhousemetricswrite:
    endpoint: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}
    resource_to_telemetry_conversion:
      enabled: true
  prometheus:
    endpoint: "0.0.0.0:8889"
service:
  extensions: [health_check, zpages, oidc]
  pipelines:
    traces:
      receivers: [otlp/auth, jaeger]
      processors: [signozspanmetrics/prometheus, batch]
      exporters: [clickhouse]
    metrics:
      receivers: [otlp/auth, hostmetrics]
      processors: [batch]
      exporters: [clickhousemetricswrite]
    metrics/spanmetrics:
      receivers: [otlp/spanmetrics]
      exporters: [prometheus]
And this is my java agent config :
Copy code
extensions:
  oauth2client:
    client_id: ${MYDOMAIN_OTEL_CLIENT_ID}
    client_secret: ${MYDOMAIN_OTEL_CLIENT_SECRET}
    token_url: ${MYDOMAIN_OTEL_TOKEN_URL}

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317

exporters:
  otlp/auth:
    endpoint: <http://signoz-otel-collector.apm.svc.cluster.local:4317>
    auth:
      authenticator: oauth2client

service:
  extensions:
    - oauth2client
  pipelines:
    traces:
      receivers:
        - otlp
      processors: []
      exporters:
        - otlp/auth
The signoz collector loads properly, however the java agent somehow tries to connect over localhost:4317 instead of http://signoz-otel-collector.apm.svc.cluster.local:4317 It repeatedly logs the following error : 
[OkHttp <http://localhost:4317/...>] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export spans. The request could not be executed. Full error message: Failed to connect to localhost/0:0:0:0:0:0:0:1:4317
Do you have any idea why ? Could you point me towards what I am doing wrong ?
p
cc @Srikanth Chekuri
v
As far as I understand the issue, it seems that the agent completely ignores what I put inside the configuration file I start the java application with the environnement variable : OTEL_JAVAAGENT_CONFIGURATION_FILE that points to the agent configuration file But no matter what I put in that file, the agent happily accept it without any warning or error. I'm completely lost here
s
Copy code
otlp/auth:
    protocols:
      http:
        endpoint: 0.0.0.0:4317
        auth:
          authenticator: oidc
@Valentin Baert Why are using the 4317 (which is for gRPC) in http endpoint? When you say agent are you referring to java agent? I couldn't really follow. What is working and what is not working?
v
Hello @Srikanth Chekuri thanks for taking the time to help me So about the http endpoint on the collector side, this was a mistake, I have now replaced it with grpc in the collector config. Yes I am referring to the java agent. I have a java application instrumented with the the jar from https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v1.14.0/opentelemetry-javaagent.jar and I was trying to follow the example given at https://opentelemetry.io/docs/collector/configuration/#authentication where it says "On the agent side, this is an example that makes the OTLP exporter obtain OIDC tokens, adding them to every RPC made to a remote collector:"
So I start my java application using :
Copy code
export OTEL_METRICS_EXPORTER=none
export OTEL_RESOURCE_ATTRIBUTES='service.name=myapp'
export OTEL_JAVAAGENT_CONFIGURATION_FILE='/path/to/opentelemetry.yaml'

java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
                    -Dotel.metrics.exporter=none \
                    -jar \
                    myapp.jar
and the following opentelemetry.yaml for the java agent (as explained in the doc at https://opentelemetry.io/docs/collector/configuration/#authentication under the sentence "On the agent side, this is an example (...)" ) :
Copy code
extensions:
  oauth2client:
    client_id: xxxxx
    client_secret: xxxxxx
    token_url: <https://xxxxxxx/oauth/token>

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317

exporters:
  otlp/auth:
    endpoint: remotecollectorxxxxx:4317
    auth:
      authenticator: oauth2client

service:
  extensions:
    - oauth2client
  pipelines:
    traces:
      receivers:
        - otlp
      processors: []
      exporters:
        - otlp/auth
But the java agent seems to completely ignore the config file and keep caling only localhost:4317
s
I am not sure what's going on. Are you passing the collector config yaml for java agent? They are totally unrelated.
v
I'm not sure I understand then, why does the documentation say "On the agent side, (...)" then it shows the yaml config ? Isn't it the config I can use for the java agent ? How do I configure the Java agent to export to the collector using oauth then ? Like I said I'm new to tracing and opentelemetry and I am really confused about all of this. The standard setup without authentication works well but I need authentication for a proper production setup.
s
Ok, The usage of word agent is loose and used in different context. The agent can also be used to address the collector running along side you application
v
hmm ok, so if I want authentication I need to run an additional collector that plays the role of a proxy between the java app agent and the actual collector (on signoz) ? That would explain why nothing works ^^ I was expecting the java agent could be configured with an authenticated oauth exporter directly.
This is a bit weird to me because in that case the data flow would be split in half, the first part between the java agent and the "proxy" collector would still be unauthenticated, then the second part between the "proxy" collector and the actual collector on signoz would be authenticated. Which means "anyone" could bypass authentication by just talking to the proxy to get free authentication injected by the proxy
s
I will have to go through again what you are doing to get a sense of what is happening.
v
ok thank you @Srikanth Chekuri I think I have understood what I need to do now. Your message https://signoz-community.slack.com/archives/C01HWQ1R0BC/p1654869052135829?thread_ts=1654849579.838479&amp;cid=C01HWQ1R0BC made it a lot more clear for me. I had indeed understood that the "agent" the documentation was talking about was referring to the java agent. But now I understand there are two agents : 1. the otel agent which is just a otel collector configured to relay its data to another otel collector on signoz server. 2. the java agent which only have the ability to send data to an unauthenticated otel collector.
So since I'm deploying my applications under kubernetes I think the best solution for me would be to deploy the otel agent in the same pod as my application as a sidecar container
Currently looking at https://github.com/open-telemetry/opentelemetry-operator on how to set it up to automatically inject that agent with proper oauth authentication
12 Views
Open in Slack
PreviousNext
Powered by