A startup with $25 million in funding is in crisis mode because an employee deleted the wrong files
GitLab
Basically, GitLab provides a virtual workspace for programmers to work on their code together, merging individual projects into a cohesive whole. It's a fast-growing alternative to the leading $2 billion GitHub, the high-profile Silicon Valley startup.
And as of Wednesday morning, GitLab was only just starting to come back online. But even worse than the embarrassment of such major downtime, the company now has to warn a handful of its customers that some of their data might be gone forever.
A bad day
The bad day started on Tuesday evening, when a GitLab system administrator tried to fix a slowdown on the site by clearing out the backup database and restarting the copying process. Unfortunately, the admin accidentally typed the command to delete the primary database instead, according to a blog entry.
And by the time he noticed and scrambled to stop the deletion "of around 300 GB only about 4.5 GB is left," the blog explains. Oops. The site had to be taken down for emergency maintenance while they figured out what to do, keeping users apprised via its blog, Twitter, and a Google Doc that the GitLab team kept updated as new developments arose.
Making matters worse, they couldn't just restore: "Out of 5 backup/replication techniques deployed none are working reliably or set up in the first place" the blog said. "We ended up restoring a 6 hours old backup." Which means that any data created in that six-hour window may be lost forever, Anglade says.
Bad news, good news
While in the process of restoring that older version of the database, the site went completely down for at least six hours, Anglade says. Worse, intermittent failures while they got the service back online took another several hours, with everything only starting to get back to normal on Wednesday morning.
The good news, says Anglade, is that the database that was affected didn't actually contain anyone's code, just stuff like comments and bug reports. Furthermore, Anglade says that the many customers who installed GitLab's software on their own servers weren't affected, since that doesn't connect up to GitLab.com. And most paying customers weren't affected at all, the company said, which minimize the financial impact.
GitLab
The outage is bad, as is the looming possibility that some of that data might be gone, Anglade acknowledges, but nobody is going to have to start rewriting their software from scratch, and only around 1% of GitLab's users will see any lasting effects from this incident.
As for the systems administrator who made the mistake, Anglade is hesitant to place blame, since it was really the whole team's fault that none of their other backup systems were working. "It's fair to say it's more than one employee making a mistake," he says.
We accidentally deleted production data and might have to restore from backup. Google Doc with live notes https://t.co/EVRbHzYlk8
- GitLab.com Status (@gitlabstatus) February 1, 2017
- Colon cancer rates are rising in young people. If you have two symptoms you should get a colonoscopy, a GI oncologist says.
- I spent $2,000 for 7 nights in a 179-square-foot room on one of the world's largest cruise ships. Take a look inside my cabin.
- An Ambani disruption in OTT: At just ₹1 per day, you can now enjoy ad-free content on JioCinema
- In second consecutive week of decline, forex kitty drops $2.28 bn to $640.33 bn
- SBI Life Q4 profit rises 4% to ₹811 crore
- IMD predicts severe heatwave conditions over East, South Peninsular India for next five days
- COVID lockdown-related school disruptions will continue to worsen students’ exam results into the 2030s: study
- India legend Yuvraj Singh named ICC Men's T20 World Cup 2024 ambassador
- JNK India IPO allotment date
- JioCinema New Plans
- Realme Narzo 70 Launched
- Apple Let Loose event
- Elon Musk Apology
- RIL cash flows
- Charlie Munger
- Feedbank IPO allotment
- Tata IPO allotment
- Most generous retirement plans
- Broadcom lays off
- Cibil Score vs Cibil Report
- Birla and Bajaj in top Richest
- Nestle Sept 2023 report
- India Equity Market