Under Investigation 07/08/2025 - Network Interruptions NL

KH-DanielP said:
Tell them they are welcome to rage at me. But they need to bring the coffee, and we'll have a nice long chat. It's 3am here and this outage sadly started right as I was about to fall asleep, but who needs sleep I normally wake up in a little over an hour anyway to start my day.
Click to expand...
Back in 2017, I had to move all my clients from one VPS to another, cause of a catastrophic incident on a city where the datacenter was. I've spent 72 hours of my life, sleepless to move all clients from my HDD backup that I had made some days ago, to another web host. I wasn't paid for this, some clients didn't even notice.
Since then, I have set up a WHM backup remotely to my Google Drive account and I can get them out within a day if that happens again.
We all have bad days, we can be prepared to bypass this kind of mess, if we schedule a step further than today's problems.

Daniel, you're probably one of the few CEOs that stay on the boat during a bad day for your clients, I feel you because I'm like that, most of us here right now are like that. Cheers and if you ever come to Crete, Greece, we'll have a beer :)
 
I got a question regarding the possible loss of data: server contents will be back with full integrity, ok, but what about incoming email messages during the downtime? Will they be lost? Senders will be receiving email bounces from our hosted email accounts until the mail servers are back online?
 
Quick update, a plan has been formulated and sent to Equinix, their basically reliant on their remote hands to rectify the situation.

In theory it's only 5-10 minutes of work (swap the power feeds between the bad/good PDU's) but simple things are never quite simple with remote hands as the request was put in to them about 20~ mins ago according to PF.

So it's progress just frustratingly slow progress.
 
Touda said:
I got a question regarding the possible loss of data: server contents will be back with full integrity, ok, but what about incoming email messages during the downtime? Will they be lost? Senders will be receiving email bounces from our hosted email accounts until the mail servers are back online?
Click to expand...
That's going to depend on the sending mail server and how it's configured. Some are configured to bounce and hard-fail if it can't connect, others have a 24-48 hour timeout where they continue to try and re-send the message.

If it fails to send either way then the sender would get the bounce notice.
 
Customer care isn’t just about solving problems — it’s about showing up when it matters the most. And Daniel is doing just that. He is not only leading the company with integrity, but also enabling so many others to build their livelihoods on the foundation of his hard work.

That said, today’s outage has been a serious jolt. Most clients don’t understand the technical complexities behind such issues — nor do they want to. For them, it’s about results. They assume that what they’re paying guarantees them 100% uptime, and when that expectation breaks, they don’t care about explanations — just that their business is affected.

What makes it worse is that a few providers in Europe are offering services at throwaway prices, which only adds more pressure and unrealistic expectations. If an incident like this repeats, many won’t wait — they’ll simply leave. Not because of dissatisfaction with support, but because in their world, service equals availability — nothing more, nothing less.
 
KH-DanielP said:
Quick update, a plan has been formulated and sent to Equinix, their basically reliant on their remote hands to rectify the situation.

In theory it's only 5-10 minutes of work (swap the power feeds between the bad/good PDU's) but simple things are never quite simple with remote hands as the request was put in to them about 20~ mins ago according to PF.

So it's progress just frustratingly slow progress.
Click to expand...
I think you should consider another provider for future
 
KH-DanielP said:
That's going to depend on the sending mail server and how it's configured. Some are configured to bounce and hard-fail if it can't connect, others have a 24-48 hour timeout where they continue to try and re-send the message.

If it fails to send either way then the sender would get the bounce notice.
Click to expand...
OK, that was what I guessed. Thanks for the quick answer and for keeping us informed :)
 
rajivkrchugh said:
Customer care isn’t just about solving problems — it’s about showing up when it matters the most. And Daniel is doing just that. He is not only leading the company with integrity, but also enabling so many others to build their livelihoods on the foundation of his hard work.

That said, today’s outage has been a serious jolt. Most clients don’t understand the technical complexities behind such issues — nor do they want to. For them, it’s about results. They assume that what they’re paying guarantees them 100% uptime, and when that expectation breaks, they don’t care about explanations — just that their business is affected.

What makes it worse is that a few providers in Europe are offering services at throwaway prices, which only adds more pressure and unrealistic expectations. If an incident like this repeats, many won’t wait — they’ll simply leave. Not because of dissatisfaction with support, but because in their world, service equals availability — nothing more, nothing less.
Click to expand...

Data-centers are not infallible, and for me this is a good lesson to diversify my auxiliary backup solutions in the future.
 
KH-DanielP said:
That's already in process to ensure not only redundant paths within a provider (which turn out to not be redundant...) but also fully diverse providers.

We've already done this in WA/GA, and NL was already on our list to complete this process as well.
Click to expand...
When will this be completed. I am very much worried about if this happens again anytime soon.
 
Thanks for all the updates Dan, could I recommend you edit the first post on this thread to save us going through 5 pages?
 
rajivkrchugh said:
Customer care isn’t just about solving problems — it’s about showing up when it matters the most. And Daniel is doing just that. He is not only leading the company with integrity, but also enabling so many others to build their livelihoods on the foundation of his hard work.

That said, today’s outage has been a serious jolt. Most clients don’t understand the technical complexities behind such issues — nor do they want to. For them, it’s about results. They assume that what they’re paying guarantees them 100% uptime, and when that expectation breaks, they don’t care about explanations — just that their business is affected.

What makes it worse is that a few providers in Europe are offering services at throwaway prices, which only adds more pressure and unrealistic expectations. If an incident like this repeats, many won’t wait — they’ll simply leave. Not because of dissatisfaction with support, but because in their world, service equals availability — nothing more, nothing less.
Click to expand...

Very true. One of our customer just called me and asked me change the name servers of their domain. They have moved to a different provider. They restored their website over there from their local backup.
 
I moved my data center from WA to NL, but that was my mistake as I wasn’t aware of the infrastructure limitations in the NL data center.
 
You must log in or register to reply here.
Top