Yesterday evening, I was testing an upcoming new feature on my development machine. This feature required a configuration update. When making that change, I swapped in the wrong credentials.
I used the production database credentials instead of the staging database credentials I wanted to test against. Because of how the tests work, the database was cleared of all data.
I deleted all the data in Signature.io’s production database. Yes, all of it.
We believed the data loss was temporary so we got to work trying to restore the data. We host our database with Heroku Postgres so we contacted them immediately for help.
Because it was the evening, the response did not come till hours later.
In the meantime, we were starting to re-create some of the data from a combination of logs and a very very old backup.
When we finally heard back from Heroku this morning, it was that they could not do anything.
Signature.io was running on the $9/month starter plan - part of the Starter tier. It turns out they don’t have extensive recovery capabilities on the starter tier and could not recover our data.
We even had Heroku pgbackups enabled. We were mistakenly under the impression that it created daily backups for all paid plans. It turns out it only create daily backups for $50/month plans and up.
So while we have restored some data, the majority is lost, and likely you will find yourself having to signup again and swap out your API tokens.
Obviously, this should have never happened. It should be very difficult to cause data loss like this and very easy to recover from it.
Our plan moving forward:
- Upgrade our Heroku Postgres database to the $50/month crane plan - part of the Production tier. This is already done.
- Confirm that daily pgbackups are functioning.
- Move our testing to a continuous integration server. This will make it very difficult to change a config setting to a production value.
We’re EXTREMELY sorry for this.
Please email email@example.com if you need to discuss the outage further.