Applied DDos Outage

[Jim Jensen]

Interesting to see that Applied suffered a DDoS outage this week. The email doesn't indicate which products were affected. Here's one of the times that I'm glad we're still on LAN Tam instead TOL or EOL. I know it's rare, but general internet outage has always been a concern too for that. Fortunately that is also rare now and we have a cell-based backup to our fiber connect to reduce those odds. Glad to see that, unlike Rackspace email, the outage only lasted about 45 minutes.

[Mark]

Where did the DDoS information come from?  Support told me that it was a DNS server issue.  Was DNS DDoS'd?  Was it a cloudflare thing?

[Jim Jensen]

Where did the DDoS information come from?  Support told me that it was a DNS server issue.  Was DNS DDoS'd?  Was it a cloudflare thing?

email from Applied:
Dear valued customer,

We hope you are having a good day.

On Tuesday, July 18, we experienced an external attack on our networks that temporarily suspended access to your management system from 2:30 p.m. - 3:15 p.m. CT. The DDoS (distributed denial-of-service attack) hit our third-party security provider, Cloudflare, which intentionally interrupted access to supported systems because of an exponential surge of traffic to the security site. This event did not result in any access to our systems or your data – it was strictly designed to limit access to systems to create business disruption.

Our monitoring technology identified the disruption early, and we quickly engaged Cloudflare and reconfigured networks to minimize the duration of the interruption. We are awaiting additional root cause analysis from Cloudflare to ensure that we take the necessary steps to protect against future similar disruptions.

Applied has invested millions in security infrastructure, both third-party and our own internal security applications, to ensure that your business and its data can be secure and protected. Know that this investment kept your business and data safe during this disruption, and we will continue to invest in our infrastructure to ensure the safety and continuity of your business.
We appreciate the opportunity to be your Indispensable Partner in the growth and success of your business. Please reach out to Applied Support with any questions.

Best,
Applied Support

Stay Up to Date with SMS Text Notifications
Applied also strives to proactively notify customers via SMS text and Applied Community alerts, and we hope you found those alerts to be informative yesterday. If you aren't already signed up, you can text JOIN to APPSYS (277-797) to subscribe to SMS text alerts for product updates and Cloud disruptions like the one this week. You can also sign up by accessing your profile on the Applied Community, entering a mobile number, your country, and selecting the SMS Opted In checkbox. You can review this article for more information on Applied Systems Service Alerts.

[Mark]

ahh, ok.  I didn't get that email yet.  Thanks!

[Ric]

We were down for 45 minutes but the service was flaky the rest of the day.

Things were fine the next day.

[Charlie Charbonneau]

I received the email as well.  We were affected from about 2:45 until 5 p.m.  It wasn't a complete no access to Epic deal.  Some areas didn't work such as printing and accessing attachments.  They came back online around 4ish but were sluggish for a bit.

We were hesitant to move away from LAN as well for the same reasons however when we had the ice storm and were out of the office for 4 days, and no Internet access for 6 days at the office, we needed a better backup working plan.   So far in the 2-3 years we've been on Epic downtime has been minimal (not really much different than LAN downtimes) and well worth the change.  Features and portability have allowed us to grow and expand our operations in ways we hadn't been able to with LAN.

[Billy Welsh]

There are much more knowledgeable nerds in here than me, but from my perspective it seems like this worked out reasonably well and as designed.  And they are using it to tweak their setup for better performance in future attacks.

No downtime is ideal, but that is not realistic in any environment.  Stuff happens, and the more complex the tech gets the more unpredictable the problems get.  If we had been subjected to a DDoS attack when we were on TAM LAN but heavily dependent on local internet service for so much else, I can promise we would have been effectively down for much longer than 45 minutes.

[Tom Fisher]

Very few saas clouds have the infrastructure to deal with a major ddos attack like this.  that's why services like cloudflare exist and they're pretty friggen awesome.  +1 for AS dealing with this in probably the best possible manner.

[Jim Jensen]

If we're out of power, we're down, of course (except for generator possibilities), but we now have cell-service backup for internet if our fiber optic service goes out. It's an extra $20 or so on our AT&T internet bill. Supposed to auto switch, just like a standby generator. That at least helps with the rare interent outage. Of course, that assume cell service is good and still up. We lost power at home after a storm, so internet/wifi was out too, of course. Power outage was big enough that the closest AT&T tower was out too and we had 1 bar of signal. Talk about old-school! No lights, no phones, no internet. We weren't sure what to do! Couldn't even hardly get text messages to go. Apparently that AT&T tower doesn't have a standby generator like the one behind my office does (Verizon). They used to keep a large portable (towable) generator here, but they installed a permanent one a few years ago.