BronyTales Under Fire - Chapter 2 - A Wild DDoS Appears!
We resume our tale from before. April 12th, 2020. I've just discovered that the source of the server issues we've been having over the past two days were being caused by an unknown attacker with no declared motive.
Chapter 2: A Wild DDoS Appears! / The Hunt
I am now in full sleuth mode. I am become, Derp, P.I. - sysadmin with a target. A folder is created to log everything surrounding this attack. I contact the datacenter for assistance with fixing the mitigation schema to better handle the attacks. During this time I also open up my email to check for notifications and find more. I find spam. Spam that is not spam. Automated notifications regarding DDoS detection and mitigation that an overzealous spam filter ever so helpfully removed from my inbox.
Things are now moving. I have a goal. I work to start documenting everything regarding the attacks. Times, patterns, methods, volume. The first series of attacks look to be fake UDP packets with a large random-length payload. I received information that SquareHorse was also attacked with a DDoS attack during the same time as we did on the 10th, as well as a basic activity dump. After processing it out, the dump has the same “fingerprint” as the attack we received.
A little background on how TCP and UDP work for the laypony: All IP packets have a sender and destination address. TCP is a stateful connection type, in that the order and integrity of the data sent and received is very important. There are multiple steps to any TCP packet transmission that involve a back-and-forth exchange between the sending and destination network interfaces to verify that data is in order and intact.. UDP doesn’t do any of this. UPD doesn’t care if you receive the packets in order, or even at all. This is handy for time-sensitive data where it’s not essential to have 100% of the packets, such as a video stream or a voice call. It also means that if you can sneak specially crafted packets past your edge router, you can spoof the return address to look like it came from anywhere you want and send bogus traffic somewhere without leaving an obvious trail.
How a DDoS (Distributed Denial of Service) attack typically works is by saturating the connection link to the target or flooding the service that handles requests with so much bogus traffic such that legitimate packets being sent to the server don’t have a spot to queue and get dropped before they can get sent to the server or times out in transit. As this is very much illegal to do to a server that you do not own or have authorization to do so, most criminals who do this will use a random set of IP addresses - but it’s usually a specific list being used.
The packets I’m finding at this point are this second type: traffic that’s being sent without regard for a legitimate return address, and always “from” the same list of IP addresses. Two of the addresses being spoofed are in fact the legitimate IP addresses of public DNS servers. Now, I can in theory block this list of addresses at the firewall, but all this does is protect the server itself from trying to respond to all of these packets to tell a nonexistent server that it’s connecting to the wrong port. It doesn’t change the fact that the router or the router behind that router is still going to try to send me that traffic. People trying to connect to the server would still have difficulty doing so. Besides, if I blocked all of those IPs, I could potentially block legitimate people and services from being able to access the server.
At this point after obtaining the attack fingerprint and a few other details, I file out an FBI report with all of the information I have. DDoS attacks against a service that you do not have authorization to test in such a manner is a Federal crime under the United States Computer Fraud and Abuse Act (18.47, §1030), and I fully intend to use the full force of law that I am able to keep this server safe. Even after I have that initial case filed, I continue to keep looking for more evidence as the case is processed. As long as this attacker is a threat to the server, I am vigilant to watch and document. Every piece of unique information can potentially be used to identify the attacker, and I am not going to sit idle.
The saga continues.
April 12th. 5:50 PM. The attacker hits me at home. I call Comcast, but cannot get past the automated system due to the COVID-19 working restrictions. Apparently an Internet company can’t be bothered to get their customer support to work remotely. Someone in the Discord notices right away and suggests that maybe the attacker has my home address. I have to fend off some very panicked people that yes, I am still ok. I do contact some people I know in the local police as a precaution, though.
April 13th. 2:50 PM. While browsing tech news at work, I find a new article labeled “Cloudflare for SSH, RDP and Minecraft”. Cloudflare has decided to provide a trimmed-down version of Spectrum services for non-enterprise customers - specifically for Minecraft servers. I purchase the service soon after I clock out from work and start transferring traffic over.
April 14th. 6:35 PM. The attacker hits me at home again. I call Comcast, and manage to get to a text-messaging support system with an overseas support center. Great. Ultimately, I cannot convince the poor tech-illiterate soul on the other end that my modem was in fact not responsible for the internet drops I was getting, and he hangs up on me when he can’t schedule a technician to visit my house. At this point I’m quite frustrated and decide to use the time I’ve been kicked offline to create a homemade ethernet tap to watch the network traffic from my modem to my router.
Over the course of the month, I collect many gigabytes of TCP dumps and logs, including two more attacks on my home internet. The attacks continue daily until the 20th, after which they become a little more spread out. My sleuthing does turn up quite a bit more than I ever expected to find, though...
April 17th. 11:27 PM. Bedtime? What bedtime? Sleep is not for the sysadmin with a perpetrator on the loose! While examining this latest attack, I notice some new things in the dumps that I had not been looking for. Up until this point I’d just assumed that all of the attacks were solely with the intent to disrupt service. Someone is trying to pretend that they’re reconnecting to an SSH session. I also notice that I had forgotten to re-enable the auto-firewall that I’d disabled while testing and turn that back on. Examining the source, it looks to be a DigitalOcean server that someone is using to try to hack into the server. I fill out an abuse report and send that off to the datacenter to let them know of illegal activity originating on their servers.
April 26th. 4:21 PM. I am getting a little frustrated with the datacenter’s mitigation system and the fact that no changes have been made in two weeks despite all of the information I’ve provided them. The attacks have been slowing down, but they still kick people off of the server when they happen. I send a somewhat sour note on the ticket asking why nothing has been done. The firewall is updated less than twelve hours later.
April 27th. 9:23 PM. It is now seventeen days since the first attack. I’ve just learned three days ago that a company based out of Connecticut intends to use one of my games commercially for profit without my permission and without providing any payment for it. At this point I decide to officially apply for copyright, even though I’ve had the game online for some months now. That’s a story for another day. While I’m going through documents related to this other project, I notice something odd going on in the Minecraft server console. A player who had been banned for griefing and raiding a few days prior was trying to rejoin the server. Over and over and over. 20 times in 20 minutes.
Now in general, players trying to rejoin during a tempban is not uncommon. The ban-kick message tells you how long you have until the ban expires. But trying more than twice in a row was very peculiar for anyone. Your one-month ban is not going to suddenly expire in thirty seconds. Even more peculiar when you start trying to join on the exact second that I get an email notification that a DDoS attack just started. Even stranger still, this player had an alt account on the server. At the time, I had been allowing it to stay unbanned to give them a chance to come clean, even though they had adamantly denied having one. Why would they be trying to connect on a banned main account when they had an alt? Before this, I had not even considered that the attacker could have been someone trying to play on the server when the attacks were going on. This seemed a little too coincidental, though. This needed more investigation.
April 28th. 7:52 AM. Bright-eyed, bushy-tailed, the thrill of the hunt, plus a bit of trepidation at what I could find, I created a new file in the attack evidence folder. A spreadsheet. I started by plotting out every attack notification I’ve received to the hour that I received it in. That done, I then started plotting out every time this player had joined, tried to join, or pinged the server MOTD, dating back to the first of April. They had joined before this at some point, but had never walked past the entrance. What I found was frankly incredible. More on that in a bit.
5:09 PM. Only minutes after clocking out at work and the attacker is at it again. I keep an eye on it, but it doesn’t seem to be affecting any services. I also work on finding more data to plot on my spreadsheet, which is proving to be more and more irrefutable.
9:41 PM. I am still seeing malicious traffic coming from that DigitalOcean server, so I update the ticket asking for an update. The server is taken offline five hours later. I also find malicious traffic from another server on their network and report this one as well. I don’t get an update on that ticket, but I also don’t see any more traffic from this server after the takedown.
April 29th. 4:58 PM. This is the last DDoS we got hit with. The system changes are holding admirably, and absolutely no one has even noticed any disruptions since the changes were made. The attacker tried switching up attack vectors to try to impact other services, but ultimately was unable and eventually gave up.
So, who was that masked man?
Chapter 2: A Wild DDoS Appears! / The Hunt
I am now in full sleuth mode. I am become, Derp, P.I. - sysadmin with a target. A folder is created to log everything surrounding this attack. I contact the datacenter for assistance with fixing the mitigation schema to better handle the attacks. During this time I also open up my email to check for notifications and find more. I find spam. Spam that is not spam. Automated notifications regarding DDoS detection and mitigation that an overzealous spam filter ever so helpfully removed from my inbox.
Things are now moving. I have a goal. I work to start documenting everything regarding the attacks. Times, patterns, methods, volume. The first series of attacks look to be fake UDP packets with a large random-length payload. I received information that SquareHorse was also attacked with a DDoS attack during the same time as we did on the 10th, as well as a basic activity dump. After processing it out, the dump has the same “fingerprint” as the attack we received.
A little background on how TCP and UDP work for the laypony: All IP packets have a sender and destination address. TCP is a stateful connection type, in that the order and integrity of the data sent and received is very important. There are multiple steps to any TCP packet transmission that involve a back-and-forth exchange between the sending and destination network interfaces to verify that data is in order and intact.. UDP doesn’t do any of this. UPD doesn’t care if you receive the packets in order, or even at all. This is handy for time-sensitive data where it’s not essential to have 100% of the packets, such as a video stream or a voice call. It also means that if you can sneak specially crafted packets past your edge router, you can spoof the return address to look like it came from anywhere you want and send bogus traffic somewhere without leaving an obvious trail.
How a DDoS (Distributed Denial of Service) attack typically works is by saturating the connection link to the target or flooding the service that handles requests with so much bogus traffic such that legitimate packets being sent to the server don’t have a spot to queue and get dropped before they can get sent to the server or times out in transit. As this is very much illegal to do to a server that you do not own or have authorization to do so, most criminals who do this will use a random set of IP addresses - but it’s usually a specific list being used.
The packets I’m finding at this point are this second type: traffic that’s being sent without regard for a legitimate return address, and always “from” the same list of IP addresses. Two of the addresses being spoofed are in fact the legitimate IP addresses of public DNS servers. Now, I can in theory block this list of addresses at the firewall, but all this does is protect the server itself from trying to respond to all of these packets to tell a nonexistent server that it’s connecting to the wrong port. It doesn’t change the fact that the router or the router behind that router is still going to try to send me that traffic. People trying to connect to the server would still have difficulty doing so. Besides, if I blocked all of those IPs, I could potentially block legitimate people and services from being able to access the server.
At this point after obtaining the attack fingerprint and a few other details, I file out an FBI report with all of the information I have. DDoS attacks against a service that you do not have authorization to test in such a manner is a Federal crime under the United States Computer Fraud and Abuse Act (18.47, §1030), and I fully intend to use the full force of law that I am able to keep this server safe. Even after I have that initial case filed, I continue to keep looking for more evidence as the case is processed. As long as this attacker is a threat to the server, I am vigilant to watch and document. Every piece of unique information can potentially be used to identify the attacker, and I am not going to sit idle.
The saga continues.
April 12th. 5:50 PM. The attacker hits me at home. I call Comcast, but cannot get past the automated system due to the COVID-19 working restrictions. Apparently an Internet company can’t be bothered to get their customer support to work remotely. Someone in the Discord notices right away and suggests that maybe the attacker has my home address. I have to fend off some very panicked people that yes, I am still ok. I do contact some people I know in the local police as a precaution, though.
April 13th. 2:50 PM. While browsing tech news at work, I find a new article labeled “Cloudflare for SSH, RDP and Minecraft”. Cloudflare has decided to provide a trimmed-down version of Spectrum services for non-enterprise customers - specifically for Minecraft servers. I purchase the service soon after I clock out from work and start transferring traffic over.
April 14th. 6:35 PM. The attacker hits me at home again. I call Comcast, and manage to get to a text-messaging support system with an overseas support center. Great. Ultimately, I cannot convince the poor tech-illiterate soul on the other end that my modem was in fact not responsible for the internet drops I was getting, and he hangs up on me when he can’t schedule a technician to visit my house. At this point I’m quite frustrated and decide to use the time I’ve been kicked offline to create a homemade ethernet tap to watch the network traffic from my modem to my router.
Over the course of the month, I collect many gigabytes of TCP dumps and logs, including two more attacks on my home internet. The attacks continue daily until the 20th, after which they become a little more spread out. My sleuthing does turn up quite a bit more than I ever expected to find, though...
April 17th. 11:27 PM. Bedtime? What bedtime? Sleep is not for the sysadmin with a perpetrator on the loose! While examining this latest attack, I notice some new things in the dumps that I had not been looking for. Up until this point I’d just assumed that all of the attacks were solely with the intent to disrupt service. Someone is trying to pretend that they’re reconnecting to an SSH session. I also notice that I had forgotten to re-enable the auto-firewall that I’d disabled while testing and turn that back on. Examining the source, it looks to be a DigitalOcean server that someone is using to try to hack into the server. I fill out an abuse report and send that off to the datacenter to let them know of illegal activity originating on their servers.
April 26th. 4:21 PM. I am getting a little frustrated with the datacenter’s mitigation system and the fact that no changes have been made in two weeks despite all of the information I’ve provided them. The attacks have been slowing down, but they still kick people off of the server when they happen. I send a somewhat sour note on the ticket asking why nothing has been done. The firewall is updated less than twelve hours later.
April 27th. 9:23 PM. It is now seventeen days since the first attack. I’ve just learned three days ago that a company based out of Connecticut intends to use one of my games commercially for profit without my permission and without providing any payment for it. At this point I decide to officially apply for copyright, even though I’ve had the game online for some months now. That’s a story for another day. While I’m going through documents related to this other project, I notice something odd going on in the Minecraft server console. A player who had been banned for griefing and raiding a few days prior was trying to rejoin the server. Over and over and over. 20 times in 20 minutes.
Now in general, players trying to rejoin during a tempban is not uncommon. The ban-kick message tells you how long you have until the ban expires. But trying more than twice in a row was very peculiar for anyone. Your one-month ban is not going to suddenly expire in thirty seconds. Even more peculiar when you start trying to join on the exact second that I get an email notification that a DDoS attack just started. Even stranger still, this player had an alt account on the server. At the time, I had been allowing it to stay unbanned to give them a chance to come clean, even though they had adamantly denied having one. Why would they be trying to connect on a banned main account when they had an alt? Before this, I had not even considered that the attacker could have been someone trying to play on the server when the attacks were going on. This seemed a little too coincidental, though. This needed more investigation.
April 28th. 7:52 AM. Bright-eyed, bushy-tailed, the thrill of the hunt, plus a bit of trepidation at what I could find, I created a new file in the attack evidence folder. A spreadsheet. I started by plotting out every attack notification I’ve received to the hour that I received it in. That done, I then started plotting out every time this player had joined, tried to join, or pinged the server MOTD, dating back to the first of April. They had joined before this at some point, but had never walked past the entrance. What I found was frankly incredible. More on that in a bit.
5:09 PM. Only minutes after clocking out at work and the attacker is at it again. I keep an eye on it, but it doesn’t seem to be affecting any services. I also work on finding more data to plot on my spreadsheet, which is proving to be more and more irrefutable.
9:41 PM. I am still seeing malicious traffic coming from that DigitalOcean server, so I update the ticket asking for an update. The server is taken offline five hours later. I also find malicious traffic from another server on their network and report this one as well. I don’t get an update on that ticket, but I also don’t see any more traffic from this server after the takedown.
April 29th. 4:58 PM. This is the last DDoS we got hit with. The system changes are holding admirably, and absolutely no one has even noticed any disruptions since the changes were made. The attacker tried switching up attack vectors to try to impact other services, but ultimately was unable and eventually gave up.
So, who was that masked man?