UK Hosting Provider Accidentally Deletes Customer Sites
Uk based hosting service provider has accidentally deleted all its customers websites and data from its server. The issue was arised on April-16, the customers who was maintaining the server mistakenly deleted all server data from its server and cause all VPS websites hosted on it were offline.
123-reg is a UK based hosting and domain provider, which accidentally deleted all its VPS customers data from its server and posted the status as follows.
Created: 16 April 2016, 09:43
Last Updated: 18 April 2016, 15:42
Some customers on our VPS platform may currently experience connectivity issues. Our engineers are aware of this issue and are currently in the process of resolving this.
We apologise for any inconvenience this may cause.
We will provide a further update as soon as this has been resolved.
Sys Admins are still investigating and we have all hands on deck to get this fixed.
This morning, our teams were made aware of issues affecting the performance of our VPS product. Our teams are continuing to investigate which has led to some customers experiencing denigrated service levels.
Our tech teams are working hard to rectify for all customers and we thank you for your patience.
123-reg would like to apologise to customers affected and will continue to provide further updates.
We are still working on finding appropriate solutions for the affected VPS customers. We have allocated extra resources to expedite this. Thank you so much for the patience.
Our teams are continuing to work to restore affected VPS services back to normal throughout the evening and night, using both internal and external experts.
If you are currently offline and would like to restore from your own backup to save time we can set you up a new VPS image. Please let our support teams know.
An email will be sent shortly to each customer effected with more information on each case. We will continue to use the status page for main updates and emails to individual customers.
Customers that have a local backup of their VPS are advised to rebuild their servers following the instructions in this support article: https://www.123-reg.co.uk/support/answers/How-to-rebuild-your-123-reg-VPS-5130/
A more comprehensive update in regards to the undergoing work on bringing the servers back online will shortly follow.
We are currently working on restoring your VPS packages using data recovery tools; in other words, we are attempting to recover your packages bit by bit, while checking data integrity at the same time.
The process is quite time consuming and the undergoing work is being done in parallel on multiple servers, in several locations.
Due to the nature of this process, we are unable to prioritise the recovery of specific hosts, but we are focused on restoring bulks of data at the moment. Progress is visible, and we have already managed restoring a certain number of hosts. However, we are unable to estimate when will the recovery be finished on all of the affected packages.
We can assure you that our teams are doing everything possible in order to get back on track.
Please do note that response times on our communication channels might be a bit higher than usual, at the moment.
We would like to thank you for your patience on the matter.
As the data recovery work continues for those remaining VPS we ask that you contact our service teams through the email address : firstname.lastname@example.org
The team reviewing these emails are able to see the status of the work so far and update you where possible.
Please note that we are actively updating all outstanding customers on a case by case basis as we get information from the recovery team. This update will be via email. We would ask that you wait for the email update. If you have questions about the recovery or need other information please do email the team.
123-reg customer, software company INNmaster, contacted The Register directing our attention to its post on the topic, claiming a rogue script had deleted customer sites.
“At 7am on Saturday morning someone at our hosting provider ran a script that had a catastrophic error in it. The result was that the script deleted the servers and websites of all their customers!”
INNmaster’s post said that company has implemented a disaster recovery plan, running up a backup server in a different location, but expresses sympathy for anybody without a proper backup.
The Register has contacted a support e-mail address at 123-reg to ask whether INNmaster’s account of events is accurate.
However, more serious is the apparent silence from 123-reg. The company’s Twitter account has stuck to variations on “we’re working on the problem” since the outage began, with the last Tweet at the time of writing promising an e-mail to affected customers.