https://signoz.io logo
Join the conversationJoin Slack
Channels
contributing
contributing-frontend
general
github-activities
incidents
k8s_operator_helm
reactjs-config
signoz-alert-testing
support
testsupport
watercooler
webhook-dev
write-for-signoz
Powered by Linen
support
  • b

    Bast

    05/25/2022, 10:06 AM
    Hey, I' ve got a JS error on trace detail.. On a similar trace there is arround 2204 Span. Maybe the trace is broken ?
    p
    • 2
    • 4
  • e

    EzzEddin Abdullah

    05/25/2022, 12:14 PM
    Hello guys, I'm just getting started with SigNoz using Python. After spinning up SigNoz docker image through the
    ./install
    command, I see that the dashboard is already up and running on
    <http://localhost:3301/application>
    . After configuring the environment variables with this command:
    OTEL_RESOURCE_ATTRIBUTES=service.name=Flask_App OTEL_METRICS_EXPORTER=none OTEL_EXPORTER_OTLP_ENDPOINT="<http://localhost:4317>" opentelemetry-instrument python3 app.py
    , I don't see I don't see Flask_App is shown on the dashboard. Any idea what I'm doing wrong here?
    s
    • 2
    • 25
  • d

    Darshan Mistry

    05/25/2022, 12:42 PM
    When i have check the logs of signoz/otelcontribcol container. I am getting
    2022-05-25T07:32:08.158Z        warn    zapgrpc/zapgrpc.go:191  [core] grpc: Server.Serve failed to create ServerTransport:  connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"POST /v1/traces HTTP/1.1\""  {"grpc_log": true}
    2022-05-25T07:33:03.158Z        warn    zapgrpc/zapgrpc.go:191  [core] grpc: Server.Serve failed to create ServerTransport:  connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"POST /v1/traces HTTP/1.1\""  {"grpc_log": true}
    2022-05-25T08:49:58.216Z        warn    zapgrpc/zapgrpc.go:191  [transport] transport: http2Server.HandleStreams failed to read frame: read tcp 172.19.0.5:4317->52.20.38.216:38694: read: connection reset by peer {"grpc_log": true}
    2022-05-25T12:39:21.929Z        warn    zapgrpc/zapgrpc.go:191  [core] grpc: Server.Serve failed to create ServerTransport:  connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"GET / HTTP/1.1\\r\\nRange: b\""      {"grpc_log": true}
    s
    • 2
    • 18
  • d

    Darshan Mistry

    05/25/2022, 12:42 PM
    Please any one help here?
  • c

    Chris Ahern

    05/25/2022, 4:30 PM
    I'm having a problem installing on kubernetes. I followed the instructions on https://signoz.io/docs/install/kubernetes/others/
    helm repo list
    NAME URL signoz https://charts.signoz.io
    kubectl create ns platform
    namespace/platform created
    helm --namespace platform install my-release signoz/signoz
    Error: parse error at (signoz/charts/clickhouse/charts/zookeeper/charts/common/templates/_secrets.tpl:84): function "lookup" not defined Anyone else get this?
    ✅ 1
    s
    p
    p
    • 4
    • 5
  • a

    Alexei Zenin

    05/31/2022, 7:47 PM
    anyone else getting this while accessing the signoz website?
    Internal Server Error - Request ID: 01G4DTGZDHQKC3PYDSJSP1R8KP
    p
    • 2
    • 2
  • c

    Chris Ahern

    06/01/2022, 12:53 AM
    I am unable to capture any traces. The UI never shows any activity. I have signoz installed with kubernetes (by helm chart). I've also setup an ingress (with tls cert) to direct to ports 3301 for the UI and 4317 for the collector. The UI works fine opening on https. I also have test opentelemetry clients for .NET Core and python. My problem is that I so far can not see any traces in the UI. As a test, if I wget the collector URL (https://sigznos.api.myserver.local), I can see an error in the collector log, so can confirm the message is getting to the backend (I obviously don't expect it to generate a span). Both client apps are instrumented and I can see the output in the console. For the .NET Core app, the only exporter when I see any activity in the log is ZipkinExporter. Including/excluding this line makes no difference: AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true); Here's my .NET code: static void Main(string[] args) { //AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true); using var tracer2Provider = Sdk.CreateTracerProviderBuilder() .SetResourceBuilder(ResourceBuilder.CreateDefault()) .AddSource("MyTest.Console") .AddAspNetCoreInstrumentation() .SetResourceBuilder(ResourceBuilder.CreateDefault() .AddTelemetrySdk()) .AddZipkinExporter(otlpOptions => { otlpOptions.Endpoint = new Uri("https://signoz.api.myserver.local"); otlpOptions.ExportProcessorType = ExportProcessorType.Simple; }) .AddConsoleExporter() .Build(); using (Activity activity = source.StartActivity("RunProgram")) { activity?.AddTag("CMA-app-Name", "MyTest.App.Name"); var program = new Program(); program.SigNozTest(); // Makes an http call to an API } Here's the output on the console: Activity.TraceId: 14d3c904ec9ee6583014177882f9f590 Activity.SpanId: 9603d4cf46d8c592 Activity.TraceFlags: Recorded Activity.ActivitySourceName: MyTest.Console Activity.DisplayName: RunProgram Activity.Kind: Internal Activity.StartTime: 2022-06-01T00:40:43.5888893Z Activity.Duration: 00:00:01.4237944 Activity.Tags: CMA-app-Name: MyTest.App.Name Resource associated with Activity: telemetry.sdk.name: opentelemetry telemetry.sdk.language: dotnet telemetry.sdk.version: 1.3.0.498 service.name: MYTEST application: MYTEST The .NET app (code and trace above) generates this error in the collector log: 2022-05-31T23:03:51.352Z warn zapgrpc/zapgrpc.go:191 [core] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"POST / HTTP/1.1\\r\\nHost: s\"" {"grpc_log": true} The python client app shows no sign of connecting to the backend. It's the sample flask app from https://signoz.io/blog/opentelemetry-flask/ with the following env vars set: OTEL_EXPORTER_OTLP_ENDPOINT="https://signoz.api.myserver.local" OTEL_EXPORTER_OTLP_INSECURE=true OTEL_RESOURCE_ATTRIBUTES=service.name=my-flask-app I've also tried setting OTEL_EXPORTER_OTLP_CERTIFICATE to the value of the cert. The message I get on the flask console is: Transient error StatusCode.UNAVAILABLE encountered while exporting span batch, retrying in 2s. What am I missing?
    👆 1
    👀 1
    s
    t
    • 3
    • 6
  • m

    Med Ghaith Sellami

    06/01/2022, 4:37 PM
    Hi guys, am facing some issues here: • Alerts stopped working after the latest update v0.8.0 • Move data to S3, for both metrics and traces, is not working (when you define a period, it's takes so long loading, and then when you refresh the page, nothing saved) • gRPC status and method in the trace table are still not displayed correctly. even though it's mentioned as fixed in the latest release v0.8.0 Am I doing it wrong? 🤔
    p
    a
    v
    • 4
    • 9
  • a

    Anthony Hathaway

    06/01/2022, 5:35 PM
    Hello everyone, I am running into a little bit of trouble with this on PHP. I have followed the example provided here https://signoz.io/docs/instrumentation/php/#instrumenting-a-sample-php-application-with-opentelemetry, but getting some PHP errors with it.
    use GuzzleHttp\Client;
    use GuzzleHttp\Psr7\HttpFactory;
    use OpenTelemetry\Contrib\OtlpHttp\Exporter as OTLPExporter;
    use OpenTelemetry\SDK\Common\Attribute\Attributes;
    use OpenTelemetry\SDK\Trace\SpanProcessor\SimpleSpanProcessor;
    use OpenTelemetry\SDK\Trace\TracerProvider;
    
    ...
    
    $exporter = new OTLPExporter(
       new Client(),
       new HttpFactory(),
       new HttpFactory()
    );
    
    $tracerProvider =  new TracerProvider(
       new SimpleSpanProcessor(
          $exporter
       )
    );
    Error: Trying to access array offset on value of type bool in /vendor/open-telemetry/opentelemetry/src/SDK/Resource/Detectors/Process.php:31 Seems to be thrown when creating a new TracerProvider. None of my data is getting sent to Signoz, though I've run the troubleshooting steps of printing out the telemetry data and also running the test to see if my docker container can hit signoz, both worked as expected.
    p
    p
    • 3
    • 11
  • k

    Kenichi Nakamura

    06/01/2022, 8:24 PM
    is there an example expression to use with alerts that counts exceptions?
    p
    s
    • 3
    • 4
  • r

    Rishabh Tripathi

    06/02/2022, 8:07 AM
    Untitled.txt
    s
    • 2
    • 1
  • r

    Rishabh Tripathi

    06/02/2022, 8:07 AM
    hi I tried to do Java OpenTelemetry Instrumentation over k8 pod to get metrics but it's giving above logs
  • m

    Med Ghaith Sellami

    06/02/2022, 12:48 PM
    Hey guys, am facing a hard time defining the clubhouse pvc resizable. (using latest helm chart) config:
    clickhouse:
      cloud: aws
      installCustomStorageClass: true
      storageClass: gp2-resizable
    the
    gp2-resize
    storageClass is created but not assigned correctly to the
    data-volumeclaim-template-chi-signoz-cluster-0-0-0
    pvc. any idea?
    p
    p
    • 3
    • 2
  • v

    Valentin Baert

    06/02/2022, 2:18 PM
    Hello, I'm new to signoz and I am currently testing it to collect traces from java backend using the opentelemetry java agent. As a proof of concept it works well. However I couldn't find anything explaining how the receiver authentication work. Is it just unsecure and open or am I missing something ? Currently it seems that anything without authentication can send trace data to signoz receiver. I tried to inspire from the example documented in this section but the agent seem to completely ignore the config https://opentelemetry.io/docs/collector/configuration/#authentication Is there a complete step by step guide on how to properly configure authentication betwen the java agent and signoz ?
    s
    p
    • 3
    • 4
  • s

    Satyasanjibani Rautaray

    06/02/2022, 4:36 PM
    Hi, I have deployed SigNoz as a part of my docker env and want to monitor a kubernetes cluster and application over it is there any way i can do that?
  • s

    Satyasanjibani Rautaray

    06/02/2022, 4:36 PM
    and documentation will help…
    ✅ 1
    p
    • 2
    • 9
  • s

    Satyasanjibani Rautaray

    06/02/2022, 4:38 PM
    sorry for the dumb question i am trying SigNoz for the first time…
    p
    • 2
    • 1
  • r

    Rahul Tiwari

    06/03/2022, 9:27 AM
    Hello Team, i was looking to deploy mongodb-exporter as k8s pod and svc i got the link but it has installation with docker-compose file (https://signoz.io/blog/opentelemetry-mongodb/) can anyone help me to deploy mongodb exporter in kubernetes types.
    p
    • 2
    • 2
  • r

    Ritek Saxena

    06/04/2022, 5:03 AM
    Hi Team, I have a doubt regarding the service name I was trying to monitor a node application with opentelemetry and signoz and it was doing fine. The only problem was it's not getting the service name resource attribute and the service is being shown as "unknown_service:node" in the dashboard. I have tried to provide the service name using semantic attributes in opentelemetry node sdk. Thanks for you time.
    p
    • 2
    • 8
  • t

    Tushar Kanti Nath

    06/05/2022, 3:40 AM
    Hi Team, Could any one can look into these issue ? https://github.com/SigNoz/sample-ASPNETCore-app/blob/main/Startup.cs Signoz is not working with docker for .NET 6 projects. Current below oltp nuget packages for .net <PackageReference Include="OpenTelemetry" Version="1.3.0" /> <PackageReference Include="OpenTelemetry.Contrib.Instrumentation.MySqlData" Version="1.0.0-beta1" /> <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.3.0" /> <PackageReference Include="OpenTelemetry.Exporter.Console" Version="1.3.0" /> <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.0.0-rc9.4" /> <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9.4" /> <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc9.4" /> <PackageReference Include="OpenTelemetry.Instrumentation.SqlClient" Version="1.0.0-rc9.4" /> <PackageReference Include="MongoDB.Driver.Core.Extensions.OpenTelemetry" Version="1.0.0" /> <PackageReference Include="MongoDB.Driver.Core.Extensions.DiagnosticSources" Version="1.1.0" /> My .net 6 startup.cs file code using System; using System.Collections.Generic; using System.IO; using System.Reflection; using Microsoft.AspNetCore.Builder; using Microsoft.AspNetCore.Hosting; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Options; using Microsoft.OpenApi.Models; using OpenTelemetry; using OpenTelemetry.Exporter; using OpenTelemetry.Instrumentation.AspNetCore; using OpenTelemetry.Metrics; using OpenTelemetry.Resources; using OpenTelemetry.Trace; namespace Examples.AspNetCore { public class Startup { private MeterProvider meterProvider; public Startup(IConfiguration configuration) { this.Configuration = configuration; } public IConfiguration Configuration { get; } public void ConfigureServices(IServiceCollection services) { services.AddControllers(); services.AddSwaggerGen(c => { c.SwaggerDoc("v1", new OpenApiInfo { Title = "My API", Version = "v1" }); var xmlFile = $"{Assembly.GetExecutingAssembly().GetName().Name}.xml"; var xmlPath = Path.Combine(AppContext.BaseDirectory, xmlFile); if (File.Exists(xmlPath)) { c.IncludeXmlComments(xmlPath); } }); var attributesDict = new Dictionary<string, object> { { "host.name", Environment.MachineName } }; var resourceBuilder = ResourceBuilder .CreateDefault() .AddService(this.Configuration.GetValue<string>("Otlp:ServiceName")) .AddEnvironmentVariableDetector() .AddAttributes(attributesDict) ; // Configure the OpenTelemetry SDK for tracing services.AddOpenTelemetryTracing(tracerProviderBuilder => { // Step 1. Declare the resource to be used by this tracer provider. tracerProviderBuilder .SetResourceBuilder(resourceBuilder); // Step 2. Configure the SDK to listen to the following auto-instrumentation tracerProviderBuilder .AddAspNetCoreInstrumentation(options => { options.RecordException = true; }) .AddHttpClientInstrumentation() .AddSqlClientInstrumentation(options => { options.SetDbStatementForStoredProcedure = true; options.SetDbStatementForText = true; options.RecordException = true; }) .AddMongoDBInstrumentation() //.AddRedisInstrumentation() ; tracerProviderBuilder //.AddConsoleExporter() .AddOtlpExporter(options => { options.Endpoint = new Uri(this.Configuration.GetValue<string>("Otlp:Endpoint")); //options.Protocol = OtlpExportProtocol.HttpProtobuf; }); }); services.AddOpenTelemetryMetrics(meterProviderBuilder => { // Step 1. Declare the resource to be used by this meter provider. meterProviderBuilder .SetResourceBuilder(resourceBuilder); // Step 2. Configure the SDK to listen to the following auto-instrumentation meterProviderBuilder .AddAspNetCoreInstrumentation() .AddHttpClientInstrumentation(); meterProviderBuilder //.AddConsoleExporter() .AddOtlpExporter(options => { options.Endpoint = new Uri(this.Configuration.GetValue<string>("Otlp:Endpoint")); //options.Protocol = OtlpExportProtocol.HttpProtobuf; }); }); } public void Configure(IApplicationBuilder app, IWebHostEnvironment env) { if (env.IsDevelopment()) { app.UseDeveloperExceptionPage(); } // app.UseHttpsRedirection(); app.UseSwagger(); app.UseSwaggerUI(c => { c.SwaggerEndpoint("/swagger/v1/swagger.json", "My API V1"); }); app.UseRouting(); app.UseEndpoints(endpoints => { endpoints.MapControllers(); }); } } }
    c
    p
    • 3
    • 6
  • t

    Tushar Kanti Nath

    06/05/2022, 3:43 AM
    Docker file #See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging. FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS base WORKDIR /app EXPOSE 80 EXPOSE 443 FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build WORKDIR /src COPY ["Examples.AspNetCore.csproj", "."] RUN dotnet restore "./Examples.AspNetCore.csproj" COPY . . WORKDIR "/src/." RUN dotnet build "Examples.AspNetCore.csproj" -c Release -o /app/build FROM build AS publish RUN dotnet publish "Examples.AspNetCore.csproj" -c Release -o /app/publish FROM base AS final WORKDIR /app COPY --from=publish /app/publish . ENTRYPOINT ["dotnet", "Examples.AspNetCore.dll"]
    c
    • 2
    • 2
  • r

    Rishabh Tripathi

    06/06/2022, 9:18 AM
    getting the following error
    ✅ 1
    p
    n
    p
    • 4
    • 18
  • b

    Benedikt

    06/07/2022, 11:47 AM
    Hey SigNoz Team, I've a question about spans an correlation-ids. In our infrastructure a middleware generates a correlation-id with the first request. This correlation-id is the same for all further internal reuests (http and message-broker/rabbitmq). So if an error occured, we can track it down to that correlation id. Is it possible to aggregate a span in a way, that it contains everything that belongs to one correlation-id? E.g. we make a request from /auth/register to /internal/auth/register-company and the second endpoints throws an error, that the span contains the entire call-trace?
    ✅ 1
    p
    s
    a
    • 4
    • 35
  • b

    Benedikt

    06/07/2022, 11:48 AM
    Thank you very much 🙂
  • m

    manohar mirle

    06/07/2022, 2:54 PM
    Hi, When the OTEL collector is stopped due to some reason, the APIs which are emitting the traces to the collector start throwing the connection error. This error will not disappear even after restarting the collector. This requires the APIs to be restarted. This becomes a limitation. What is the solution for this ?
    p
    p
    s
    • 4
    • 12
  • b

    Benedikt

    06/07/2022, 7:45 PM
    Hey SigNoz team! I'm trying to visualize exceptions/errors in SigNoz logged using ASP.NET Core 6 ILogger interface. E.g.:
    logger.Error("Some error message");
    I've added OTLP logging to the service:
    services.AddLogging(logging =>
                {
                    logging.ClearProviders();
    
                    logging.AddConsole();
    
                    logging.AddOpenTelemetry(options =>
                     {
                         options.SetResourceBuilder(resourceBuilder);
    
                         options.IncludeScopes = true;
                         options.ParseStateValues = true;
                         options.IncludeFormattedMessage = true;
    
                         if (UseConsoleExporter(monitoringSettings.LoggingExporter))
                             options.AddConsoleExporter();
    
                         if (UseOtlpExporter(monitoringSettings.LoggingExporter))
                         {
                             options.AddOtlpExporter(otlpOptions =>
                             {
                                 otlpOptions.Endpoint = new Uri(monitoringSettings.OtlpEndpoint);
                             });
                         }
                     });
                });
    But the exception page is always empty (see the screenshot). Do I need to add something to the otel-collector? I could not find and "extension" or additional "service. Traces and Metrics were send successfully.
    s
    a
    • 3
    • 2
  • v

    Valentin Baert

    06/08/2022, 9:01 AM
    Hello, I want to deploy a signoz cluster using the helm chart however the documentation is a bit inconsistent and I'm not sure what's true. In the documentation at https://signoz.io/docs/install/kubernetes/gcp/ it says to override the following values to set the storage class
    clickhouse:
      cloud: gcp
      installCustomStorageClass: true
      persistence:
        storageClass: gce-resizable
    But the documentation then links to the github (https://github.com/SigNoz/charts/tree/main/charts/signoz#configuration) but these values are not documented. There are some of them that are similar but on a different component
    alertmanager.persistence.enabled	Enable volume persistence	true
    alertmanager.persistence.storageClass	Set storage class for persistent volume	nil
    alertmanager.persistence.accessModes	Set access mode for persistent volume	[ReadWriteOnce]
    alertmanager.persistence.size	Set storage size	100Mi
    ✅ 1
    p
    p
    • 3
    • 22
  • a

    anand h

    06/09/2022, 9:19 AM
    Hello, My Application is not showing up in "Metrics" tab in frontend. I am new to Signoz I have locally setup my application and signoz. I can see my traces in the "traces" tab but metrics are not showing up. My application is a tomcat webapp. My env variables are
    export CATALINA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 -javaagent:/Users/anand/libs/opentelemetry-javaagent.jar"
    export OTEL_RESOURCE_ATTRIBUTES=service.name=MyApp
    export OTEL_METRICS_EXPORTER=otlp
    p
    s
    a
    • 4
    • 13
  • m

    manohar mirle

    06/09/2022, 1:44 PM
    Hello, How to download the source code & docker image of the Signoz collector to the local system ?
  • v

    Valentin Baert

    06/10/2022, 8:26 AM
    Hello, I'm new to signoz and opentelemetry I deployed signoz on kubernetes using the helm chart And I have a Java Spring application running with the open telemetry java agent So far so good the java app is able to send the traces over grpc, it works well. Then I wanted to setup oauth authentication between the agent and otel collector. But the documentation at https://opentelemetry.io/docs/collector/configuration/#authentication is very succint and I was not able to reproduce the setup. Can you help me figure this out ?
    p
    s
    • 3
    • 20
Powered by Linen
Title
v

Valentin Baert

06/10/2022, 8:26 AM
Hello, I'm new to signoz and opentelemetry I deployed signoz on kubernetes using the helm chart And I have a Java Spring application running with the open telemetry java agent So far so good the java app is able to send the traces over grpc, it works well. Then I wanted to setup oauth authentication between the agent and otel collector. But the documentation at https://opentelemetry.io/docs/collector/configuration/#authentication is very succint and I was not able to reproduce the setup. Can you help me figure this out ?
Currently this is my signoz collector config. I have taken the original config from the helm chart and added otlp/auth receiver, extension and pipeline as explained in the documentation :
receivers:
  otlp/spanmetrics:
    protocols:
      grpc:
        endpoint: localhost:12345
  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
      thrift_http:
        endpoint: 0.0.0.0:14268
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu: {}
      load: {}
      memory: {}
      disk: {}
      filesystem: {}
      network: {}
  otlp/auth:
    protocols:
      http:
        endpoint: 0.0.0.0:4317
        auth:
          authenticator: oidc
processors:
  batch:
    send_batch_size: 1000
    timeout: 10s
  signozspanmetrics/prometheus:
    metrics_exporter: prometheus
    latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 50ms, 100ms, 250ms, 500ms, 1000ms, 1400ms, 2000ms, 5s, 10s, 20s, 40s, 60s]
    dimensions_cache_size: 10000
    dimensions:
      - name: service.namespace
        default: default
      - name: deployment.environment
        default: default
  # memory_limiter:
  #   # 80% of maximum memory up to 2G
  #   limit_mib: 1500
  #   # 25% of limit up to 2G
  #   spike_limit_mib: 512
  #   check_interval: 5s
  #
  #   # 50% of the maximum memory
  #   limit_percentage: 50
  #   # 20% of max memory usage spike expected
  #   spike_limit_percentage: 20
  # queued_retry:
  #   num_workers: 4
  #   queue_size: 100
  #   retry_on_failure: true
extensions:
  health_check: {}
  zpages: {}
  oidc:
    issuer_url: <https://auth.review.mydomain.com/>
    audience: <https://api.mydomain.com/>
exporters:
  clickhouse:
    datasource: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_TRACE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}
  clickhousemetricswrite:
    endpoint: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}
    resource_to_telemetry_conversion:
      enabled: true
  prometheus:
    endpoint: "0.0.0.0:8889"
service:
  extensions: [health_check, zpages, oidc]
  pipelines:
    traces:
      receivers: [otlp/auth, jaeger]
      processors: [signozspanmetrics/prometheus, batch]
      exporters: [clickhouse]
    metrics:
      receivers: [otlp/auth, hostmetrics]
      processors: [batch]
      exporters: [clickhousemetricswrite]
    metrics/spanmetrics:
      receivers: [otlp/spanmetrics]
      exporters: [prometheus]
And this is my java agent config :
extensions:
  oauth2client:
    client_id: ${MYDOMAIN_OTEL_CLIENT_ID}
    client_secret: ${MYDOMAIN_OTEL_CLIENT_SECRET}
    token_url: ${MYDOMAIN_OTEL_TOKEN_URL}

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317

exporters:
  otlp/auth:
    endpoint: <http://signoz-otel-collector.apm.svc.cluster.local:4317>
    auth:
      authenticator: oauth2client

service:
  extensions:
    - oauth2client
  pipelines:
    traces:
      receivers:
        - otlp
      processors: []
      exporters:
        - otlp/auth
The signoz collector loads properly, however the java agent somehow tries to connect over localhost:4317 instead of http://signoz-otel-collector.apm.svc.cluster.local:4317 It repeatedly logs the following error :
[OkHttp <http://localhost:4317/...>] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export spans. The request could not be executed. Full error message: Failed to connect to localhost/0:0:0:0:0:0:0:1:4317
Do you have any idea why ? Could you point me towards what I am doing wrong ?
p

Prashant Shahi

06/10/2022, 8:33 AM
cc @Srikanth Chekuri
v

Valentin Baert

06/10/2022, 10:04 AM
As far as I understand the issue, it seems that the agent completely ignores what I put inside the configuration file I start the java application with the environnement variable : OTEL_JAVAAGENT_CONFIGURATION_FILE that points to the agent configuration file But no matter what I put in that file, the agent happily accept it without any warning or error. I'm completely lost here
s

Srikanth Chekuri

06/10/2022, 12:25 PM
otlp/auth:
    protocols:
      http:
        endpoint: 0.0.0.0:4317
        auth:
          authenticator: oidc
@Valentin Baert Why are using the 4317 (which is for gRPC) in http endpoint? When you say agent are you referring to java agent? I couldn't really follow. What is working and what is not working?
v

Valentin Baert

06/10/2022, 1:29 PM
Hello @Srikanth Chekuri thanks for taking the time to help me So about the http endpoint on the collector side, this was a mistake, I have now replaced it with grpc in the collector config. Yes I am referring to the java agent. I have a java application instrumented with the the jar from https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v1.14.0/opentelemetry-javaagent.jar and I was trying to follow the example given at https://opentelemetry.io/docs/collector/configuration/#authentication where it says "On the agent side, this is an example that makes the OTLP exporter obtain OIDC tokens, adding them to every RPC made to a remote collector:"
So I start my java application using :
export OTEL_METRICS_EXPORTER=none
export OTEL_RESOURCE_ATTRIBUTES='service.name=myapp'
export OTEL_JAVAAGENT_CONFIGURATION_FILE='/path/to/opentelemetry.yaml'

java \
  -javaagent:/path/to/opentelemetry-javaagent.jar \
                    -Dotel.metrics.exporter=none \
                    -jar \
                    myapp.jar
and the following opentelemetry.yaml for the java agent (as explained in the doc at https://opentelemetry.io/docs/collector/configuration/#authentication under the sentence "On the agent side, this is an example (...)" ) :
extensions:
  oauth2client:
    client_id: xxxxx
    client_secret: xxxxxx
    token_url: <https://xxxxxxx/oauth/token>

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317

exporters:
  otlp/auth:
    endpoint: remotecollectorxxxxx:4317
    auth:
      authenticator: oauth2client

service:
  extensions:
    - oauth2client
  pipelines:
    traces:
      receivers:
        - otlp
      processors: []
      exporters:
        - otlp/auth
But the java agent seems to completely ignore the config file and keep caling only localhost:4317
s

Srikanth Chekuri

06/10/2022, 1:46 PM
I am not sure what's going on. Are you passing the collector config yaml for java agent? They are totally unrelated.
v

Valentin Baert

06/10/2022, 1:47 PM
I'm not sure I understand then, why does the documentation say "On the agent side, (...)" then it shows the yaml config ? Isn't it the config I can use for the java agent ? How do I configure the Java agent to export to the collector using oauth then ? Like I said I'm new to tracing and opentelemetry and I am really confused about all of this. The standard setup without authentication works well but I need authentication for a proper production setup.
s

Srikanth Chekuri

06/10/2022, 1:50 PM
Ok, The usage of word agent is loose and used in different context. The agent can also be used to address the collector running along side you application
v

Valentin Baert

06/10/2022, 1:52 PM
hmm ok, so if I want authentication I need to run an additional collector that plays the role of a proxy between the java app agent and the actual collector (on signoz) ? That would explain why nothing works ^^ I was expecting the java agent could be configured with an authenticated oauth exporter directly.
This is a bit weird to me because in that case the data flow would be split in half, the first part between the java agent and the "proxy" collector would still be unauthenticated, then the second part between the "proxy" collector and the actual collector on signoz would be authenticated. Which means "anyone" could bypass authentication by just talking to the proxy to get free authentication injected by the proxy
s

Srikanth Chekuri

06/10/2022, 1:58 PM
I will have to go through again what you are doing to get a sense of what is happening.
v

Valentin Baert

06/10/2022, 2:13 PM
ok thank you @Srikanth Chekuri I think I have understood what I need to do now. Your message https://signoz-community.slack.com/archives/C01HWQ1R0BC/p1654869052135829?thread_ts=1654849579.838479&amp;cid=C01HWQ1R0BC made it a lot more clear for me. I had indeed understood that the "agent" the documentation was talking about was referring to the java agent. But now I understand there are two agents : 1. the otel agent which is just a otel collector configured to relay its data to another otel collector on signoz server. 2. the java agent which only have the ability to send data to an unauthenticated otel collector.
So since I'm deploying my applications under kubernetes I think the best solution for me would be to deploy the otel agent in the same pod as my application as a sidecar container
Currently looking at https://github.com/open-telemetry/opentelemetry-operator on how to set it up to automatically inject that agent with proper oauth authentication
View count: 9