We Were Outgunned: Cybercrime Files

Wade Anderson | Bay & Bay Transportation

It was my fourth day of employment as CIO for a 77-year-old family owned transportation company; an industry very new to me. Our modest, but healthy, mid-sized trucking and logistics company Bay & Bay Transportation was hacked in a most heinous way.

This is our story.

After spending 25 years in industries like manufacturing, insurance and high tech, I was excited to join the transportation industry. The transport of goods is fundamental to the economy in the United States and globally, and I truly believe the future for our industry is very bright. One thing I didn’t anticipate was that investments in technology, like insurance, are often filtered through a must-have prioritization model.

Bay & Bay had some great systems and processes in place. However, attention to and investment in cybersecurity was lagging. On Thursday, July 12, 2018, an unplanned investment in cybersecurity was required to recover critical systems. That investment would later yield big fruit, but not immediately. At 1:54 p.m., a nasty ransomware called “I Apologize” quickly infected nearly all IT systems within the company computers, causing a material loss of operations and productivity.

A variant of the notorious SamSam ransomware, this cyberfraud malware has been wreaking havoc for about three years infecting more than 200 companies and causing more than $30 million in ransoms and damage, according to the FBI. SamSam ransomware encrypts files essential for systems to operate. The encrypted files are accompanied by readable files asking for ransom payment in the form of bitcoin cryptocurrency. Most of our approximately 80 servers were crippled in an instant by this software, forcing us to make some tough decisions.

We immediately set up an internal command center, enlisting help from managed-service providers and security experts. We stripped security rights from most user accounts, made some firewall changes, and attempted to recover all systems by restoring files from backups. We started by restoring our email and trucking systems, since our logistics systems are cloud-based and therefore not impacted. Email was up quickly due to how it had been backed up. Our trucking systems were another story.

Once the bitcoin payment was made and all the security keys were transferred to us via our cyber security partner, it took the extended IT team about 36 hours to bring 95 percent of all systems and data online.

The TruckMate restore process started running at 9:40 p.m. and ran for about five hours, and then IT HAPPENED! An active hacker was still in our network and was able to stop the restore process and delete backup points from our systems. The technician overseeing the restore process noticed the hacker, tried to message them with no success, and ended up shutting everything down. This was the right approach. He contacted me, and we moved to Recovery Plan B.

By 6 a.m. Friday morning, with just three hours of sleep for the team members, we reassembled. After several more conversations with more than a dozen security experts I’ve worked with over the past several decades, I engaged our CEO and CFO and I recommended we go down a new and necessary path – that of paying the ransom. A hacker had successfully thwarted our efforts to recover on our own. We were outgunned, and they had us. Our CEO agreed. After attempting to procure bitcoin on our own, we enlisted help from insurance, legal and security companies. Working closely with an amazing cybersecurity company named Kivu throughout the day, we were able to oust the hackers, contact the cybercriminals, obtain proof they could decrypt our files, and ultimately paid the ransom in bitcoin.

In the full cyber forensic analysis performed in the following days, we were able to learn and confirm a few important things:

1. No data was ever transferred outside of the Bay & Bay network. This was very important and great news. No employee, customer, or carrier data were compromised.
2. There were two groups of cyber criminals, and they tag-teamed the infiltration, installation, and execution of the attack a full 12 hours prior to the shutdown.
3. They exploited a common open firewall port used for technical support (RDP – Remote Desktop Protocol). This was not an employee clicking on a phishing email link, as some had suspected.
4. The other tools (Mimikatz, PsExec) used in the attack were free and easy to get.

Once the bitcoin payment was made and all the security keys were transferred to us via our cybersecurity partner, it took the extended IT team about 36 hours to bring 95 percent of all systems and data online. The only data not recovered was several hours of email and two very old servers, which were easily replaced.

In the following weeks, Bay & Bay made the much-needed investment into perimeter fortification, partner alignment, and strengthening of security policies and are in the process of end-user education and testing.

Our starting recommendations for helping improve your company’s security posture are:

1. Fortify the perimeter. This includes firewall rules (e.g. close open RDP ports), access controls (e.g. eliminate inbound non-U.S. traffic) and use the principle of least privilege (PoLP) when granting access to any network or system resource.
2. Develop and strengthen security policies and your security culture. Any company’s weakest link is its employees – not its firewalls.  All employees share a most important responsibility to protect the sacred data of its employees, customers, and carriers. Institute a cybersecurity policy. Train it, test it, live it.
3. Continuously test and improve. Most companies are doing phishing tests now, which is great. Social engineering and internal/external penetration scans should also be standard practice.

Bay & Bay is a much stronger company now, for our customers, carriers, and employees. Though we had to react to an unplanned and unfortunate situation, we are in a better position than we have ever been. With the changes our company has made and with continuous improvement, we do believe we are greatly reducing our exposure to costly, future interruptions such as these.

Update

Finally, after spending many hours with my new friends at the FBI, I am happy to share that the two cybercriminals committing this fraud have been identified and indicted. Two Iranian citizens have been charged, but because they are suspected to be in Iran, there is no easy way to extradite them at this time. After three years of digital destruction, this one ransomware is now out of commission. Sadly, it is just the tip of the iceberg. As threats like SamSam continue to evolve, diligence with cybersecurity is more important than ever for companies in our great transportation industry and beyond.

Wade Anderson is Chief Information Officer for Bay & Bay Transportation in Eagan, MN. He can be reached at [email protected]ay.com