author Posted by Tunnelr, 2017-03-06

From approximately 12:00AM PST 03.06.2017 until 3:00PM PST 03.06.2017, Tunnelr experienced a fairly large scale outage.


All of our nodes at one of our larger providers were shutdown due to a number of open abuse notices for a particular exit node. This effectively took down *all* of our core infrastructure - website, mail, DNS, database servers, workers and several exit nodes.

While all services and nodes have been brought back up and *should* be operational, it is possible that there may be lingering issues as a result of the mass-shutdown.

--------------

We receive a lot of abuse notices - it is not unexpected, it is also not unexpected to have a node get shutdown from time to time by service providers while they wait for an abuse complaint to be resolved. What is unexpected though is to have all unrelated nodes get shutdown as a result of abuse notices for other nodes.

While we make sure that we have web, dns and database redundancy, our core infrastructure is all hosted with one provider. Given this most recent incident, we will have to readdress this.

As a direct result of this outage we will be doing the following:

- talking with our primary provider on what can be done to ensure this does not happen again
- distributing our infrastructure across a number of different cloud providers
- automating additional abuse notice handling for certain providers
- separating our core-infra account from our exit node account (which we believe is the primary cause for the mass-shutdown)

Please accept our sincere apologies for the issues. We'll make sure that this sort of an issue does not occur again.