Blake Romano
06/30/2022, 5:15 PM2022-06-30T17:14:03.114Z info service/collector.go:124 Everything is ready. Begin running and processing data.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1063552]
goroutine 211 [running]:
<http://github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousemetricsexporter.(*PrwExporter).export.func1()|github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousemetricsexporter.(*PrwExporter).export.func1()>
/src/exporter/clickhousemetricsexporter/exporter.go:279 +0xf2
created by <http://github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousemetricsexporter.(*PrwExporter).export|github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousemetricsexporter.(*PrwExporter).export>
/src/exporter/clickhousemetricsexporter/exporter.go:275 +0x256
Blake Romano
07/01/2022, 4:07 PMUsman Ali
07/04/2022, 2:21 PMteja
07/05/2022, 6:14 AMteja
07/05/2022, 6:14 AMteja
07/05/2022, 6:14 AMunable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
Brian Bills
07/05/2022, 9:34 PMdocker run -it -p 8080:8080 -e SW_AGENT_NAME=docmagic-dev-torrance::bluelagoon -e POD_NAMESPACE=docmagic-dev-torrance cfpb-brian bash
docker run -it -p 8081:8080 -e SW_AGENT_NAME=docmagic-preprod-torrance::jetblue -e POD_NAMESPACE=docmagic-preprod-torrance cfpb-brian bash
docker run -it -p 8082:8080 -e SW_AGENT_NAME=docmagic-qa-torrance::chinaair -e POD_NAMESPACE=docmagic-qa-torrance cfpb-brian bash
docker run -it -p 8083:8080 -e SW_AGENT_NAME=docmagic-stage-torrance::evaair -e POD_NAMESPACE=docmagic-stage-torrance cfpb-brian bash
docker run -it -p 8084:8080 -e SW_AGENT_NAME=docmagic-dev-torrance::manadarinair -e POD_NAMESPACE=docmagic-dev-torrance cfpb-brian bash
docker run -it -p 8085:8080 -e SW_AGENT_NAME=docmagic-stage-torrance::starlux -e POD_NAMESPACE=docmagic-stage-torrance cfpb-brian bash
docker run -it -p 8086:8080 -e SW_AGENT_NAME=docmagic-qa-torrance::tigerair -e POD_NAMESPACE=docmagic-qa-torrance cfpb-brian bash
docker run -it -p 8087:8080 -e SW_AGENT_NAME=docmagic-preprod-torrance::delta -e POD_NAMESPACE=docmagic-preprod-torrance cfpb-brian bash
docker run -it -p 8088:8080 -e SW_AGENT_NAME=boa-preprod-torrance::blue22 -e POD_NAMESPACE=boa-preprod-torrance cfpb-brian bash
Dockerfile:
FROM <http://harbor.docmagic.com/library/dm-openjdk:8-jdk-slim-buster-012122164131|harbor.docmagic.com/library/dm-openjdk:8-jdk-slim-buster-012122164131>
ADD agent /app/agent
COPY cfpb-service.war /app
COPY entrypoint.sh /usr/local/bin/
ENTRYPOINT ["entrypoint.sh", "-j", "/app/cfpb-service.war"]
entrypoint.sh:
#!/bin/bash
while getopts ":j:f:h" opt; do
case ${opt} in
j)
JAR=$OPTARG
;;
f)
JVM_EXTRA_FLAGS=$OPTARG
;;
h)
echo "USAGE: entrypoint.sh -j app.jar"
echo " -j: /path/to/jar"
echo " -f: "JVM_EXTRA_FLAGS" (optional)"
exit 0
;;
\?)
echo "Invalid option: $OPTARG" 1>&2
exit 1
;;
:)
echo "Invalid option: $OPTARG requires an argument" 1>&2
exit 1
;;
esac
done
shift $((OPTIND -1))
if [ -f "$JAR" ]; then
JAVA_BIN=$(/usr/bin/which java)
umask 002
exec $JAVA_BIN \
-Dfile.encoding=ISO-8859-1 \
-Dserver.port=8080 \
-Djava.security.egd=file:/dev/./urandom \
-XX:+UnlockExperimentalVMOptions \
-XX:+UseContainerSupport \
-Dcom.sun.management.jmxremote \
-Djava.rmi.server.hostname=127.0.0.1 \
-Dcom.sun.management.jmxremote.port=1083 \
-Dcom.sun.management.jmxremote.rmi.port=1083 \
-Dcom.sun.management.jmxremote.local.only=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=true \
-Dcom.sun.management.jmxremote.access.file=$JAVA_HOME/conf/jmx.access \
-Dcom.sun.management.jmxremote.password.file=$JAVA_HOME/conf/jmx.password \
$JVM_EXTRA_FLAGS \
-javaagent:/app/opentelemetry-javaagent.jar \
-jar \
$JAR
else
echo "File $JAR does not exist"
exit 1
fi
Brian Bills
07/05/2022, 9:49 PM[otel.javaagent 2022-07-05 14:48:40:134 -0700] [OkHttp <http://localhost:4317/...>] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export spans. The request could not be executed. Full error message: Failed to connect to localhost/127.0.0.1:4317
Brian Bills
07/05/2022, 10:23 PMdocker run -it -p 8080:8080 -e OTEL_EXPORTER_OTLP_ENDPOINT="<http://10.1.130.93:4317>" -e OTEL_RESOURCE_ATTRIBUTES=service.name=bluelagoon -e POD_NAMESPACE=docmagic-dev-torrance cfpb-brian bash
Leon Jones
07/06/2022, 12:07 AMOTEL_RESOURCE_ATTRIBUTES: 'service:name=a_service'
Edson F Cunha
07/06/2022, 8:26 PMVikash Kashyap
07/07/2022, 4:38 AMShreyas Mishra
07/07/2022, 11:30 AMSandeep Sarpe
07/08/2022, 11:51 AMSandeep Sarpe
07/09/2022, 6:09 AMShiwam Jaiswal
07/09/2022, 10:26 AMalertmanager:
image: signoz/alertmanager:0.23.0-0.1
volumes:
- ./data/alertmanager:/data
expose:
- "9093"
ports:
- "9093:9093"
# depends_on:
# query-service:
# condition: service_healthy
restart: on-failure
command:
- --queryService.url=172.17.0.1:8085
- --storage.path=/data
I have mapped my host port to dockers port both are same i.e. 9093. So my local alert manager will send alerts to this port which will be received by the application running on port 9093 in docker. Here I am trying to create a new alert from the frontend and then I am trying to ingest it into the DB, before it goes to the DB the following function gets triggered.
apiError := r.alertManager.AddRoute(receiver)
Here is AddRoute method's defn.
func (m *manager) AddRoute(receiver *Receiver) *model.ApiError {
receiverString, _ := json.Marshal(receiver)
amURL := prepareAmChannelApiURL()
response, err := <http://http.Post|http.Post>(amURL, contentType, bytes.NewBuffer(receiverString))
if err != nil {
zap.S().Errorf(fmt.Sprintf("Error in getting response of API call to alertmanager(POST %s)\n", amURL), err)
return &model.ApiError{Typ: model.ErrorInternal, Err: err}
}
if response.StatusCode > 299 {
err := fmt.Errorf(fmt.Sprintf("Error in getting 2xx response in API call to alertmanager(POST %s)\n", amURL), response.Status)
zap.S().Error(err)
return &model.ApiError{Typ: model.ErrorInternal, Err: err}
}
return nil
}
The error occurs at line
response, err := <http://http.Post|http.Post>(amURL, contentType, bytes.NewBuffer(receiverString))
the amURL is as follows:
<http://localhost:9093/api/v1/routes>
I am getting the error as connection refused:
accessJwt eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6InNoaXdhbUBzaWdub3ouaW8iLCJleHAiOjE2NTczNTg3NTMsImdpZCI6ImMyMjk5NGM1LTU2ZWItNGRkYy04NmM0LWU4MTBjY2JjYmY0NSIsImlkIjoiZTZlZmZhMmUtZTc3NC00MDI5LWIyNjQtMDBlMTRlNTE4ZGE3In0.sZi3OYomyl28bb78hpWI0RdIESleXSst1CanBhb_4TA
user &{{e6effa2e-e774-4029-b264-00e14e518da7 <mailto:shiwam@signoz.io|shiwam@signoz.io> 0 c22994c5-56eb-4ddc-86c4-e810ccbcbf45} }
2022-07-09T14:26:01.365+0530 ERROR alertManager/manager.go:63 Error in getting response of API call to alertmanager(POST <http://localhost:9093/api/v1/routes>)
%!(EXTRA *url.Error=Post "<http://localhost:9093/api/v1/routes>": dial tcp 127.0.0.1:9093: connect: connection refused)
<http://go.signoz.io/query-service/integrations/alertManager.(*manager).AddRoute|go.signoz.io/query-service/integrations/alertManager.(*manager).AddRoute>
/home/ella/dev/signoz/pkg/query-service/integrations/alertManage
Also there is this error that I see during the startup:
s=2022-07-09T08:55:07.844081945Z caller=log.go:168 component=notifier level=error alertmanager=<http://127.0.0.1:9093/api/v1/alerts> count=0 msg="Error sending alert" err="Post \"<http://127.0.0.1:9093/api/v1/alerts>\": dial tcp 127.0.0.1:9093: connect: connection refused"
2022-07-09T14:25:53.204+0530 DEBUG auth/auth.go:294 Login method called for user:
Setting the following env variable didnt help as well:
ALERTMANAGER_API_PREFIX=http://localhost:9093/api/Shiwam Jaiswal
07/10/2022, 7:27 PMRishabh Tripathi
07/11/2022, 5:09 AMYuriy
07/11/2022, 11:16 AMphp 2-send-trace-to-collector.php
(from https://github.com/SigNoz/sample-php-app), but I not visible data in panel (http://localhost:3301/application).Blake Romano
07/11/2022, 1:12 PMBlake Romano
07/11/2022, 1:21 PMblackmoja
07/12/2022, 7:31 AMTamir Shkolnik
07/12/2022, 12:09 PMShashank Gupta
07/12/2022, 5:48 PMShashank Gupta
07/13/2022, 5:51 AMAlexei Zenin
07/13/2022, 5:46 PMQuyet Nguyen Duc
07/14/2022, 5:27 AMQuyet Nguyen Duc
07/14/2022, 8:42 AMAnil Kumar Bandrapalli
07/14/2022, 10:38 AMAnil Kumar Bandrapalli
07/14/2022, 10:40 AMAnil Kumar Bandrapalli
07/14/2022, 10:40 AMAnkit Nayan
07/14/2022, 11:16 AM2m
or 5m
histogram_quantile(0.99, sum(rate(signoz_latency_bucket{service_name="workflow-service", operation="/api/task/complete"}[5m])) by (le)) > 100
Anil Kumar Bandrapalli
07/14/2022, 11:22 AMAnkit Nayan
07/14/2022, 1:15 PMAnil Kumar Bandrapalli
07/14/2022, 1:15 PMAnkit Nayan
07/14/2022, 1:22 PMv0.9.2
and follow migration docs to do thatPriyansh
07/14/2022, 2:32 PMRahul Tiwari
07/15/2022, 5:59 AM-- -host=my-release-clickhouse -port=9000 -userName=admin -password=27ff0399-0d3a-4bd8-919d-17c2181e6fb9kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead. If you don't see a command prompt, try pressing enter. Writing samples to DB 2022/07/15 05:44:09 Error while writing samples to DB code: 60, message: Table signoz_metrics.samples_v2 doesn't exist Session ended, resume using 'kubectl attach signoz-migrate-clickhouse-56767c457-sqpl2 -c signoz-migrate-clickhouse -i -t' command when the pod is running [ec2-user@ip-10-0-4-191 ~]$ [ec2-user@ip-10-0-4-191 ~]$ kubectl logs signoz-migrate-clickhouse-56767c457-kfmdj -n platform -f my-release-clickhouse 9000 admin 27ff0399-0d3a-4bd8-919d-17c2181e6fb9 signoz_metrics Total Rows: 63262424 There are total 63262424 samples rows, starting migration... Total Rows: 2555 There are total 2555 time series rows, starting migration... Writing samples to DB 2022/07/15 05:58:08 Error while writing samples to DB code: 60, message: Table signoz_metrics.samples_v2 doesn't exist [ec2-user@ip-10-0-4-191 ~]$
Anil Kumar Bandrapalli
07/15/2022, 10:43 AMAnkit Nayan
07/15/2022, 11:44 AMAmol Umbark
07/15/2022, 11:44 AMAnil Kumar Bandrapalli
07/15/2022, 11:48 AMAmol Umbark
07/15/2022, 11:52 AMAnil Kumar Bandrapalli
07/15/2022, 12:27 PMAnkit Nayan
07/15/2022, 12:56 PMhistogram_quantile(0.99, sum(rate(signoz_latency_bucket{service_name="workflow-service", operation="/api/task/complete"}[1m])) by (le))
Anil Kumar Bandrapalli
07/15/2022, 1:00 PMAnkit Nayan
07/15/2022, 1:01 PM[2m]
Anil Kumar Bandrapalli
07/15/2022, 1:06 PMAnkit Nayan
07/15/2022, 1:09 PMAnil Kumar Bandrapalli
07/15/2022, 1:10 PMAnkit Nayan
07/15/2022, 1:11 PMexec -it
into your clickhouse container and connect to db by running clickhouse client
inside the containerAnil Kumar Bandrapalli
07/15/2022, 1:12 PMAnkit Nayan
07/15/2022, 1:13 PMuse signoz_metrics;
and
select * from time_series_v2 where metric_name='signoz_latency_bucket';
and try to search for rows which has workflow-service
and /api/task/complete
2m
time range..your alert won't workAnil Kumar Bandrapalli
07/15/2022, 1:40 PMAmol Umbark
07/15/2022, 1:42 PMclickhouse client --host localhost --port 9000
Ankit Nayan
07/15/2022, 2:14 PMPrashant Shahi
07/15/2022, 3:16 PMkubectl -n platform exec -i --tty pod/chi-signoz-cluster-0-0-0 -- bash
Followed by:
clickhouse-client
Anil Kumar Bandrapalli
07/15/2022, 4:09 PMAnkit Nayan
07/15/2022, 4:32 PMAnil Kumar Bandrapalli
07/15/2022, 4:38 PMAnkit Nayan
07/15/2022, 4:51 PMAnil Kumar Bandrapalli
07/15/2022, 5:08 PMAnkit Nayan
07/16/2022, 10:56 AMRahul Tiwari
07/18/2022, 6:10 AMPrashant Shahi
07/18/2022, 6:27 AMVishal Sharma
07/18/2022, 6:33 AMRahul Tiwari
07/18/2022, 6:53 AMVishal Sharma
07/18/2022, 6:54 AMRahul Tiwari
07/18/2022, 6:57 AMVishal Sharma
07/18/2022, 6:58 AMkubectl -n platform delete pod signoz-migrate
Rahul Tiwari
07/18/2022, 7:01 AMAnil Kumar Bandrapalli
07/18/2022, 1:08 PMAmol Umbark
07/18/2022, 1:11 PMAnil Kumar Bandrapalli
07/18/2022, 1:13 PMAmol Umbark
07/18/2022, 1:14 PMAnil Kumar Bandrapalli
07/18/2022, 1:14 PMAmol Umbark
07/18/2022, 1:15 PMAnil Kumar Bandrapalli
07/18/2022, 1:15 PMAmol Umbark
07/18/2022, 1:17 PMAnil Kumar Bandrapalli
07/18/2022, 1:23 PMAmol Umbark
07/18/2022, 1:25 PMAnil Kumar Bandrapalli
07/18/2022, 1:27 PMAmol Umbark
07/18/2022, 1:27 PMAnil Kumar Bandrapalli
07/18/2022, 1:27 PMAmol Umbark
07/18/2022, 1:28 PMAnil Kumar Bandrapalli
07/18/2022, 1:32 PMAmol Umbark
07/18/2022, 1:37 PMAnil Kumar Bandrapalli
07/18/2022, 1:37 PMAnkit Nayan
07/18/2022, 3:54 PMhistogram_quantile(0.99, sum(rate(signoz_latency_bucket{service_name="workflow-service", operation="POST /api/task/complete"}[1m])) by (le))
try changing the [1m]
to [5m]
. Does the chart plot now?Anil Kumar Bandrapalli
07/18/2022, 5:48 PMAnkit Nayan
07/18/2022, 6:14 PMAnil Kumar Bandrapalli
07/18/2022, 6:16 PMAnkit Nayan
07/18/2022, 6:19 PMAnil Kumar Bandrapalli
07/18/2022, 6:20 PMAnkit Nayan
07/25/2022, 9:20 AMSrikanth Chekuri
07/25/2022, 10:16 AM