Yesterday the Signoz VM CPU just spiked to 100% on...
# supporto
Ola Ahlman
09/16/2024, 7:01 AMYesterday the Signoz VM CPU just spiked to 100% on all 4 cpu cores given to it. Unable to access it through ssh or Proxmox GUI terminal. Memory and disk activity stable the whole time.
A soft reboot from Proxmox didn't take so I had to forcefully shut VM off and on again. Even after reboot CPU keept running at full load on all cores for some time, after it settled down. Maybe 5-10 min after reboot.
Not sure how to debug it to find the cause. Any suggestion welcome. Has been noticed/reported before?
Ola Ahlman
09/16/2024, 8:53 AMSeems it was left in a broken state after reboot. Couldn't login any more. Now attempting a re-install
Ola Ahlman
09/16/2024, 8:53 AMshould signoz/migrate keep restarting over and over?
Ola Ahlman
09/16/2024, 8:55 AMimage.png
Ola Ahlman
09/16/2024, 8:56 AMAfter re-install I can at least access web gui again.
Ola Ahlman
09/16/2024, 9:04 AMUpdated to 0.54, same state of containers after with migrate restarting and the rest being up.
Ola Ahlman
09/16/2024, 9:11 AMMight have been my fault for migrate container. I think I noticed a while ago that some containers didn't start automatically. So I ran:
Copy code
docker update --restart unless-stopped $(docker ps --all -q)s
Srikanth Chekuri
09/16/2024, 9:45 AMShare the error logs of the migrator
o
Ola Ahlman
09/16/2024, 8:48 PMLike I mentioned, reason for it restarting was most likely on my end due to me changing restart mode to unless-stopped for all containers. A re-install most likely didn't change this as created date say 7 months ago for this contianer.
I'm more concerned about why it went to 100 % cpu and stayed there. Though I guess I wont be able to figure that out now. If it happens again I will let you know. Can disregard this one for now.
22 Views