Elias
04/17/2023, 9:37 AMSrikanth Chekuri
04/17/2023, 9:49 AMElias
04/17/2023, 9:51 AMPrashant Shahi
04/17/2023, 10:37 AMhelm upgrade
?
override-values.yaml
alertmanager:
persistence:
size: 1Gi
helm upgrade
, remove the alertmanager statefulset and retry the command.Elias
04/17/2023, 12:23 PMhelm upgrade signoz-monitoring signoz/signoz -f override-values.yaml --namespace=platform
AlterManager is still pending since 8min. Same message in the pod logs. Pod description isn't showing an error:
Name: signoz-monitoring-alertmanager-0
Namespace: platform
Priority: 0
Service Account: signoz-monitoring-alertmanager
Node: <none>
Labels: <http://app.kubernetes.io/component=alertmanager|app.kubernetes.io/component=alertmanager>
<http://app.kubernetes.io/instance=signoz-monitoring|app.kubernetes.io/instance=signoz-monitoring>
<http://app.kubernetes.io/name=signoz|app.kubernetes.io/name=signoz>
controller-revision-hash=signoz-monitoring-alertmanager-6d448ccf6d
<http://statefulset.kubernetes.io/pod-name=signoz-monitoring-alertmanager-0|statefulset.kubernetes.io/pod-name=signoz-monitoring-alertmanager-0>
Annotations: checksum/config: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/signoz-monitoring-alertmanager
Init Containers:
signoz-monitoring-alertmanager-init:
Image: <http://docker.io/busybox:1.35|docker.io/busybox:1.35>
Port: <none>
Host Port: <none>
Command:
sh
-c
until wget --spider -q signoz-monitoring-query-service:8080/api/v1/version; do echo -e "waiting for query-service"; sleep 5; done; echo -e "query-service ready, starting alertmanager now";
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xfvkb (ro)
Containers:
signoz-monitoring-alertmanager:
Image: <http://docker.io/signoz/alertmanager:0.23.0-0.2|docker.io/signoz/alertmanager:0.23.0-0.2>
Port: 9093/TCP
Host Port: 0/TCP
Args:
--storage.path=/alertmanager
--queryService.url=<http://signoz-monitoring-query-service:8085>
Requests:
cpu: 100m
memory: 100Mi
Liveness: http-get http://:http/ delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:http/ delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_IP: (v1:status.podIP)
Mounts:
/alertmanager from storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xfvkb (ro)
Volumes:
storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: storage-signoz-monitoring-alertmanager-0
ReadOnly: false
kube-api-access-xfvkb:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
<http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events: <none>
C:\Users\elias\signoz>kubectl describe pvc --namespace=platform storage-signoz-monitoring-alertmanager-0
Name: storage-signoz-monitoring-alertmanager-0
Namespace: platform
StorageClass: ionos-enterprise-hdd
Status: Pending
Volume:
Labels: <http://app.kubernetes.io/component=alertmanager|app.kubernetes.io/component=alertmanager>
<http://app.kubernetes.io/instance=signoz-monitoring|app.kubernetes.io/instance=signoz-monitoring>
<http://app.kubernetes.io/name=signoz|app.kubernetes.io/name=signoz>
Annotations: <http://volume.beta.kubernetes.io/storage-provisioner|volume.beta.kubernetes.io/storage-provisioner>: <http://cloud.ionos.com|cloud.ionos.com>
<http://volume.kubernetes.io/selected-node|volume.kubernetes.io/selected-node>: prod-performance-o3niqbe464
<http://volume.kubernetes.io/storage-provisioner|volume.kubernetes.io/storage-provisioner>: <http://cloud.ionos.com|cloud.ionos.com>
Finalizers: [<http://kubernetes.io/pvc-protection|kubernetes.io/pvc-protection>]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: signoz-monitoring-alertmanager-0
Events:
Type Reason Age From
Message
---- ------ ---- ----
-------
Normal Provisioning 7m52s (x63 over 3h33m) cloud.ionos.com_csi-ionoscloud-547ff5c6cf-xf55x_ec448156-856a-40a1-b1d4-0d307d8bc24b External provisioner is provisioning volume for claim "platform/storage-signoz-monitoring-alertmanager-0"
Normal ExternalProvisioning 3m16s (x842 over 3h33m) persistentvolume-controller
waiting for a volume to be created, either by external provisioner "<http://cloud.ionos.com|cloud.ionos.com>" or manually created by system administrator
C:\Users\elias\signoz>kubectl get pvc --namespace=platform
NAME STATUS VOLUME
CAPACITY ACCESS MODES STORAGECLASS AGE
data-signoz-monitoring-zookeeper-0 Bound pvc-b675e319-b64e-4039-bea8-748869aed061 8Gi RWO ionos-enterprise-hdd 3h34m
data-volumeclaim-template-chi-signoz-monitoring-clickhouse-cluster-0-0-0 Bound pvc-4bf90e0e-2ff5-4dd4-888c-473d87822e0f 20Gi RWO ionos-enterprise-hdd 3h34m
signoz-db-signoz-monitoring-query-service-0 Bound pvc-ca1049e9-de4d-4dbd-a9a3-677f4338e02d 1Gi RWO ionos-enterprise-hdd 3h34m
storage-signoz-monitoring-alertmanager-0 Pending
ionos-enterprise-hdd 3h34m
Prashant Shahi
04/17/2023, 1:07 PM