Please visit our donations page for more details.
Yesterday evening around 22:00
everything suddenly went down. After a quick inspection of VMware running on
MrWhite we saw something horrific.
Lost connectivity to SSD storage device
Say what now?
The SSD in question is the disk where MrWhite runs off (which include all our web services, databases, etc.)
The machine fully lost internal connectivity to its SSD storage disk. After a few minutes of research and pondering on what to do we decided there was no other option than rebooting the entire machine. This was another scary exercise because VMware itself also has to load from the SSD drive in question....
Luckily, against all expectations, the machine rebooted correctly AND saw its SSD connected again!
As we no longer really trust the disk we decided to make a few full backups of the hosts using that disk (hence last night’s downtime) just in case it's going to fail us completely.
For now everything seems to be back up and running stable again. If you find everything is down again then you now know what is most likely the cause.