BronyTales Under Fire - Chapter 3 - A Technical Problem
We resume our tale from before. It's now April 28th, 2020. Exactly two weeks to the day the attacks started, I discover a wild array of connections to a previously unconsidered individual. Background information and DDoS postmortem to wrap up our miniature saga.
Chapter 3: A Technical Problem
Here’s the part where about a dozen people are going to say “I told you so!” - while the details may seem obvious in hindsight, we cannot take actions without definitive evidence. Moving on:
In my personal office, I have many screens. Regularly four, although I have a dock for two more and another laptop I keep to my right for occasional status information for extracurricular hobbies. On one of these regularly displayed screens is dedicated to housing a terminal screen that shows real-time information about the Minecraft server that BronyTales runs on. When that screen starts flashing red, there's a fire to put out. On this fateful evening as I'm doing otherwise menial tasks, I notice some alerts popping up on this screen at the same time a DDoS alert pops up on my phone. The Minecraft server terminal doesn't have anything to display DDoS, so the juxtaposition of the two is curious, especially given the recent events surrounding the source of the particular alerts I'm seeing.
Let's go back a little and fill you in on the backstory.
TechnicalRarity first joined the server in January of 2020. He complained about needing to disconnect from a VPN before joining, but nothing otherwise of note regarding when he joined. Well, aside from the little detail that he never made an attempt to continue down the spawn area to get to the main server once he had access to the chat. They came back every once in a while and would hang out in chat for a while before disconnecting.
April 10, 2020, about 2:00 AM their local time (3:12 AM my time, for those keeping track), they joined the server again. Instead of hanging out in chat, they connected and disconnected over and over. It’s not the first time they’ve done that, though, and it’s not unusual activity on its own. However, it happened to also be right at the start of a DDoS attack. Five minutes after it started. He complained in chat that he was unable to connect, and continued to disconnect and reconnect a number of times until their connection on the server timed out. They suggested that the network issues must be a DDoS attack, and for the first five minutes of being online didn’t move an inch. Not even to look around them. They left after chatting a bit about twenty minutes after the DDoS protection cleared. In fact, they talked more on the server in that hour than they had in the past four months combined. The attack ended about 3AM, right after they left.
April 10, 2020, 5:15 PM. A new DDoS attack starts. 5:18 PM, Technical joins the server and almost right away starts complaining about the super high latency. Remember, this is also about the time I start to do some network maintenance on the server, which is why I don’t notice right away that the issues aren’t connected to what I’m doing. After a while of troubleshooting the issues, I jump into a voice chat room, mostly to vent a bit of the issues I’m seeing “rubber ducky” style. See, when working out a tough logic issue that you’re stumped on, simply explaining what the issue is to someone else (or a personified object such as a rubber duck) can help you discover a solution. That, and being in a call with people in the server could also help me explain in real-time what was going on and getting that information shared with the community without needing to stop what I was doing to hang out in the text channels.
As may be recalled from our previous escapade into this time frame, you may recall I didn’t mention a certain individual who wanted to talk about DDoS and system security and how they had DDoSed their school in the past. I’m pretty sure you could gander a guess as to who this was now, especially in the context of this tale. They also showed a great interest in knowing what datacenter BronyTales was hosted out of, and when I don’t tell him he tells me it’s OVH (which is mostly true), and then goes on a tirade about how awful the tech support is. Having experience working in a datacenter, I was very skeptical of his opinions, especially given the language he was using. Skilled techs are more likely to pay attention if you treat them well. Even more so if you can competently speak their lingo.
Well, another thing they mentioned also caught my attention to the point it distracted me from what I was working on to listen in on. I don’t recall the entirety of the conversation being distracted and sleep deprived as I was, but they showed a great deal of interest in a SWAT call at a hotel. They shared images, which I immediately recognized as being from the bomb threat at HarmonyCon 2019. I was doubly suspicious when they kept talking about how they’ve been “swatted” before, but never directly mentioned that the images were from the convention or named the convention. A convention I learned later that he was in attendance at the time.
(The first image is from after the event, but that specific door I’ve never seen and can’t find, so I suspect it’s one they took. The second one was posted on a 4chan thread detailing what happened)
Warning sign number 2, the day after the incident, they posted a blurb about “last night” in general chat:
Previously they were talking about just been woken up by a text alert, so it’s fairly safe to assume that the “it was fun last night” had nothing to do with the creative map chaos that ensued afterwards. I’m not sure how hanging out in a voice call while complaining about how hard it is to join the server could be fun, but it could, so idk. But to follow that statement immediately afterwards with “It’s super easy to hit a mc server” ?? And to mention the hosting company by name, which I had not discussed at all in the call, too.
Enter more drama, stage left
April 13th, 2AM. Or Late-Late April 12th, for you night owls. During another mysterious DDoS, Technical convinces another BT player in voice chat to join him on another server, and proceeds to grief a random house with a lavacast. Following this incident, and especially since they had used the BT server to organize the attack, Technical was issued a 3-month server ban.
Things that were considered in the ban were:
- Helped organize and participate in the raid and grief of a fellow pony server.
- Continually spread fear about the attacks in server chat, despite requests to stop.
- Kept trying to blame other random people/things, generally inciting panic.
- Generally: Inciting drama.
Before and after this raid, another account had logged into BT for about 10 seconds. Which is more interesting when each time this new account connected, Technical had disconnected less than one minute earlier.
Two days after promising to not use an alt, a new account joins, but doesn’t link their account in the discord. A curious one, seeing as they seem to conveniently already have been doing work to develop a DDoS protection system..
On the 16th, my home internet got hit with an attack. Nothing too notable, aside from frustrations with the ineptitude of Comcast’s level 1 technical support. Well, nothing technically notable, although a certain someone on the server somehow seemed to know that I was getting DDoSed at that moment and also knew how I was being targeted.
This “not an alt” somehow also knew that I almost never go offline.
They made a very not-subtle hint that they believed they had the ability to call a SWAT against my home address, and I spent the next hour after trying to assure everyone that messaged me in a panic asking if I was alright that yes, I was alright and no-one had tried to kill me in the past hour.
And they had a home network exactly like how Technical said theirs was set up. Even named their Xbox “Rarity.” And their talking points drifted routinely to the same ones that Technical used a lot, such as how much they like to cuddle their plushies and how much they hated quarantine.
After taking this use of an alt for ban evasion, plus the other unnecessary drama, I formally issued a ban for his Discord account. He had left the Discord server after getting the ban for the raid, so I had to ban the account by Discord ID. I did not take action against the alt, though, as I wanted to give him a chance to come clean on his own. Were he to have told the truth, I would have been more lenient towards his active sentence. Everyone makes mistakes. It’s what you do when you think no-one can see what you’re doing that defines your character.
Technical told me a day after joining with this alt that they thought their ban was unwarranted, at which point I hinted that I knew that they had an alt and that he would not be doing himself any favors by evading the ban.
He didn’t seem to get the hint, though. A day later, this alt account used a WDL mod to download portions of our map. Not cool.
I wrote parts of that mod, for context. Fyi, downloading a public server’s map without their permission - especially for builds that you do not own the rights to - is very not cool. Theft is bad, okay? He bragged about stealing the map the next day to another player on the server:
I had a few other people bring the fact that this GlimmerShimmers / terabyte4000 character was an alt account of Technicals, too. Most pointed to the behavior similarities, but some did some sleuthing to dig up more:
Another interesting piece of information: April 18th, while doing more work scouring for evidence I noticed that I had neglected to re-enable a hacking defense script I keep on my public-facing servers. Shortly after re-enabling this script, something interesting popped up in chat..
(filtered chat to only show the alt)
It would seem that the connection that this alt was using suddenly got blocked and they had to switch connections. Weird, huh?
By the 20th, the rest of the staff team had figured out that this account was an alt of Technical’s. He wasn’t exactly being subtle about it. Timely confronted them about it, even directly naming the alt account, which Technical vehemently denied. I did have a talk with Timely a bit after this, though (sorry, Timely, need to bring this up) - see, the burden of proof should always lie with the prosecution, not the defense. You can’t prove that you have never said something or been somewhere, etc - you can only prove what you have said or where you’ve been.
Note here that he knows that this "GlimmerShimmers" is a Discord account.
We are a Minecraft server.
This alt joined after he left the server, so how would he know that this is a Discord user?
The plot thickens
Technical messaged me directly again, asking if he could appeal the permban. Still wanting to give him a chance to admit to the alt, I tell him to “include as much information” as he could to clear up any misunderstandings. I also try to make it clear that he will only have one shot, so he had better make it good. I may be a softie sometimes, but I do not appreciate being taken advantage of. Speaking of which, he messaged me back a few days later asking for help setting up his server to be like BronyTales.
Note also here he mentions a "/spell" command - his main account has never used this command.
This message was sent to me the day after terabyte4000 discovered it.
This last line he gives to defend himself struck me particularly odd. Firstly, appeal to authority is automatically an invalid argument. Holding a position of authority does not make your decisions or behavior automatically correct, even if they are related to your position. He doubles down on it here, though, stating that being a 3-day unpaid volunteer at a pony convention automatically means that you are a professional in everything you do.
I have been to many conventions. I have been to many pony conventions. I have staffed many pony conventions. I have hung out at staff dead-dog parties at conventions that I did not staff. I have many personal friends who are career pony convention staff. I know very well that people who volunteer at conventions are not “professional” or even any more trustworthy than any of the con-goers themselves. The fact that he decided to appeal to that particular authority, especially as a defense, struck me very interested. What was he trying to hide behind? I started to reach out to a few contacts, to see if anyone knew anything about him. More on that later.
We didn’t get hit by any DDoS attacks during peak hours for about a week, though. Ever since the day after Technical got banned, everything had been going very smoothly. Maybe a tad coincidental, but I was just happy that the network wasn’t being a problem anymore. At the time I just chalked it up to the DigitalOcean server that I reported.
Enter more drama, stage right
April 25, 2020. PonyFest 2.0. We had a party room planned out in the BronyTales server and started streaming in one of the video game rooms, much like we had at the first PonyFest. Well, we tried, anyway. A conveniently-timed DDoS attack kinda left us dawdling around awkwardly and I jumped off to do more information gathering to trace the source. Oh, and more foreshadowing for you, Technical started trying to join BronyTales less than seven minutes after the attack started, and tried to rejoin thirteen times in the span of ten minutes.
About an hour after the attacks started and about a half hour since last trying to join BronyTales even though he was still banned after trying so many times in a row, Technical starts sending private messages in Discord to all of the BronyTales build team that were online at the time, asking them if they wanted to help him build a pony server.
Even sending a friend request to those who he didn’t share a server with:
We report the fact that he’s trying to use the PonyFest Discord to send unsolicited advertisements to the PonyFest moderation team, and they serve him a warning, telling him to knock it off. Most Discord servers have rules against this, and PonyFest is no exception. Besides, it’s just rude to randomly message staff of a server - especially one you’ve been banned from - just to ask them to join your server instead. We found out later that he’d also messaged a few of the really active players to try to get them to build his server while BronyTales was having troubles.
At this point, I’d had enough. He had his chance, and he spent it.
But wait - there’s more!
He tried again two days later with Lego, our new developer. Granted, Lego also was working with the SEAPony con Minecraft server and the Ponyville Plaza server, so that on its own wasn’t of note. However, after Technical inevitably asks Lego to help with his server as well, Lego declines and brings up the alt on BronyTales as part of the reason. Technical again denies that this alt could possibly be his. He also knows that we’ve banned the user, although he doesn’t admit he knows who it was. His reason for it not being an alt is because “RarityIsAwesome is [the] only account I paid for” and also claims that if the IP that the account connected from isn’t from New Mexico then it can’t be his.
If you’ve been paying attention this whole time, you might recall a little tidbit I mentioned very early on. See, Technical doesn’t trust that websites he goes to or servers he plays on won’t try to attack him. From information I was able to find from his past, he has good reason to. He has a habit of making himself a target. He has posted his personal IP to 4chan at least twice. Surprise, he got DDoS’d both times. More on that later. After all of this excitement, he tries to be a little more careful with how he browses the internet. Not in the way he conducts himself, mind you; he just uses VPN services to hide his IP.
So when he declares the ultimatum of “If it’s not from New Mexico it’s not me,” knowing all that you have learned up to this point, it’s probably pretty safe to assume that he never connects when he doesn’t have a VPN proxy hooked up. On BronyTales, I use a VPN detection service to deter most VPN services to prevent troll account spamming. He knows this - he’s complained about it before. A clever techie with time and money can bypass these sometimes.
That said, you think it’s worth taking him up at his offer? I thought it was.
I’ve censored the IPs, for obvious reasons. On the left are all of the IPs that Technical’s main account, RarityIsAwesome / 706bf6cf-3579-4762-852b-30178c297576 have used to connect from which passed the VPN check. On the right are all of the IPs that this suspect alt account, terabyte4000 / 97452c18-8b7a-4b8f-b886-d44d01a7f9aa have used to connect with. The coloration comes from Notepad++ Compare. Notice the two lines that are not green or red? Those are lines that match exactly. Not only did they use the same IP once, but they did twice! Sometimes even on the same day, minutes apart!
When you add the IPs that they’ve tried to connect to but didn’t pass the VPN check, it gets even more interesting - both accounts tend to try to ping from a VPN. They even use the exact same servers. Crazy, huh?
Also, remember how the terabyte account was asking if I had ever released my home IP? Well, it also appears as though they remembered what it was, despite joining the Discord after everything was back on the main server. Both RarityIsAwesome and terabyte4000 have attempted to join my testing server multiple times, long after I took it down. And they both would try to connect from the same VPN server.
Shortly after this discussion - later on that same day, in fact - I notice him trying to join the server over and over again, at the exact moment another DDoS attack hits. Remember how I said that I found the results to be incredible?
This is a scatter plot centered around the time that I receive a notification that a DDoS attack has been detected. The points are mappings of the time difference between when the attack starts and the first time that Technical starts joining the server. I started the chart by simply counting how many times each hour he tried to join. Excepting two moments when the alt joined for a few seconds on the 24th, never in 16 activity periods over 18 days did he ever join during a time that the server was not under attack. Driven by curiosity, I narrowed the window even further to try and pinpoint the time that the attack starts and when he starts joining. See those eleven dots on the right, centered almost perfectly around zero?
Once is a coincidence. Twice is curious. Three times is a pattern. Eleven? That’s intent.
But Data!, you might say, What if he just joins at the same time every day? What if the attacks are timed and he just happens to be trying to join when the real attacker’s schedule kicks off an automated attack?
Nope. The times are all over the place. Even the gaps line up. He was banned from the server (and this was before the visitor rank was self-assignable), so there was no way for him to know an attack was happening so soon after it started without being involved. In fact, the last five dots on that scatter plot are after DDoS mitigations had been improved to the point that no-one on the server even noticed a performance drop.
You can have a look at my spreadsheet for yourselves if you would like:
With this in mind, I start reviewing my server logs again. This time, I’m looking for something very specific: I want to see every time any of his IP addresses, including his VPNs, have ever tried to connect to the server.
So not only had he been using an alt to evade his ban, not only had he used that alt to download portions of the map, not only had he tried to lure staff to help build his server, but he had also been trying to hack into the server itself. That first screenshot is his main home IP while I didn’t have the denyhosts service running (forgot to re-enable it while doing firewall maintenance). That IP tried to connect to the management SSH service 61,685 times over four days. The other two are VPNs that he had used to connect to the server with. While using a VPN to hack isn’t instantly attributable to only Technical, aside from maybe the hack was run during a time that the server was under an attack, what is noteworthy is that the last one on the list isn’t a public VPN. It’s a private server, hosted out of the same datacenter that BronyTales is hosted out of. (one that he has root access to, as confirmed by private key authentication coupled with the fact his Minecraft server is hosted there)
So that’s the end of it, right? Caught him red handed, mystery solved, everyone goes home happy?
If only things were that simple.
On May 10th, Technical joined the Ponyville Plaza Minecraft server. The Plaza is a Discord community that started a Minecraft server earlier that year and that I’ve shared some of our privately developed plugins in return for a mutual advertising agreement. They did not have the same security considerations built into their server as I had on BronyTales, so Technical was able to get a full plugin list, as well as some information about the plugins. Using this, he then sends me an email from Spigot to my developer account asking if he could pay for a copy of the plugin.
I don’t make secret that my jascotty2 Minecraft account is linked to my DataByte Discord account, and a little over an hour later he figures out that it’s me that he messaged and sends me a private message on Discord.
My turn to play.
Using the information I’d dug up on him earlier from personal contacts plus a somewhat lucky treasure trove in a rar archive, I compose him a note. To his personal, unlisted email. Using his first name. Remember how he said he works at conventions? Remember how I said I started asking around? Turns out a personal friend of mine who I play games with and joins occasionally when I host cookouts happens to be his direct supervisor at the one convention that Technical hasn’t been barred from staffing at.
What I got in response was the most bizarre email I have ever received.
Somewhere in there are four periods. “I” is capitalized exactly five times, among the 97 times it’s used. He defends that he is a “real” brony as though I had questioned that. He mentions he sometimes hangs out with the “wrong crowds” and has gotten in trouble for it a few times. He expresses that he’s envious of the community that BronyTales has and that he wants something like that. What’s particularly interesting, though, is he didn’t reply to my email - he opened a new one to type whatever was on his mind, but never once refutes or confirms anything that I mention in the original email. He tries to appeal to emotion several times, and while I do sympathise with his predicament, I also know that he has had this exact same conversation with other people, three years prior.
I mull over this for a few days, not wanting to react rashly. It certainly sounds like an attempt at an apology, but focuses on justifying his view rather than actually apologizing for anything. After some consideration, I ask for more information.
He immediately defends that he has never tried to hack into my console, then says he wanted the stuff because he was jealous of what we had and wanted it for himself. He claims that he would never DDoS anyone even though he knows how to, and that he launches attacks against his own equipment to test it (which is perfectly legal if he also owned the line or had authorization to do so at a shared host).
I press for more details, and he admits to the “lesser” crime of ban evasion and downloading the map, and promises that he deleted the map that he downloaded.
Thing is, for his implication made that he would never DDoS anyone, I have found some evidence from his past that he has done just that. Multiple times.
Also, remember how I said I had been going down my list of contacts in the convention world? I found that not only was he known within upper circles, but also that he has been blacklisted from staffing at Harmony, Everfree, Trotcon, PonyFest, and fully banned from attending at Babs. He may have been blacklisted from staffing BronyCon, too, but that also could have been from staffing in a specific department. I never bothered trying to clarify that one. He was also suspected to have been involved in the Whinny City 2017 bomb threat, although nothing concrete had been pinned on him. In the chat log archive that I had gained access to, I found he had in fact bragged about it to some people..
However, as far as I was able to find, some of those claims he made were outright fabrications. There was no overarching investigation into him, there was no plane ticket, there was no interview with a detective, there was no stipulation to allow him to staff again if he surrendered his phone. In fact, at the time of the messages, all upper con leadership were focused on PCF, not WCPC.
I did find some more information, but due to the sensitive nature of some of them (sexual themes, private conversations, law enforcement topics) and off topic (like general harassment, trying to frame BT on his server, and creepy trolling), I think it would be prudent to leave that much unsaid. This writeup on the attack on BronyTales was exhaustive enough. All I’ll add is that my heart goes out to a certain individual I saw on their Minecraft server. I honestly hope that Technical has learned to be a good friend if they are still hanging out with him after some of the logs I’ve seen.
So, why might he be trying to connect during a DDoS attack?
My current theory around this revolves around how some Minecraft servers are set up in relation to ban tracking. See, in order to synchronize a ban database between servers on a network, you need a database server. On many small Minecraft servers it’s too expensive to run it yourself, so people use an external server - like a cheap web server - to handle the database. Even some larger servers do this, as it makes displaying the current bans on the website easy to automatically update. If you flood a server’s network so much data that it starts dropping packets to keep up, it’s possible that a database request for a ban lookup might time out. Rather than fail-secure, game servers tend to instead fail-functional, and will allow you to join. BronyTales does not use a remote server for database synchronization, though, so any attempt to flood the server to attempt a bypass hack like that will fail.
So why would he want to play on a server while he’s attacking it?
Probably for the same reason arsonists tend to hang around to watch the building they light on fire burn down. The same reason why it’s fun to play Among Us as the imposter. There’s a real sense of accomplishment to watch the fallout from your machinations take form, especially in real time with people around you to watch their reactions. Every time someone says “wow the lag,” he gets to see that as “they noticed!” and “They don’t suspect a thing! I’m so devious!” - every complaint, every rogue idea is catharsis and reward for their actions. Going through the logs, you would also notice that they are the single most active source of ideas for who the attacker could be, even though there is no consensus that there is even an attacker. They’re trying to drive the conversation to get more of that serotonin high. (This incessant fear mongering was a major portion of extending the initial ban term, too)
Why would he attack the server in the first place?
As a disclaimer, this is pure speculation, but founded around available evidence. He states several times that he wants to be a good figure in the fandom. He is always the first to mention he notices the server is under a DDoS attack. He mentions several times that he has been working on developing a DDoS scrubbing service to mitigate server attacks.
Scenario: there’s a brand new Minecraft server that just popped up out of nowhere. The owner isn’t known for running Minecraft servers. They are hosting the server at a datacenter you know has a security flaw. You know you can do better than they can. They probably won’t want to let someone new have access to the server, though. After all, one of the easiest ways to get banned from some servers is to ask for op after joining. It’ll look too suspicious if you start a conversation with “so, I noticed you have an unlocked back door - want me to fix it for you?” (I had a similar issue at SHG when I offered to give them a fixed build of LWC when they were having issues). So you need to show them why security is important. You start small, when the server isn’t busy, test to make sure your attack actually works. Then you kick it off later at a really busy time. Right when everyone is panicking, you swoop in and offer to help. After trying and failing to fix it themselves, they’ll be sure to let you in. Make a few changes, stop the attacks, you’ll be the hero who saved the server! You’ll surely get a ranking position in the community after that!
The old “start a fire in the building so I can show everyone how useful having sprinklers can be” trick.
What makes this theory more compelling is that he’s done it twice.
SEAPony con. May 25th, 2020. They had a modded Minecraft set up for hosting a convention virtually, which had just completed the day before. They were discussing keeping the server online for a while longer so more people could get to see the work that the builders had put into the world. Mysteriously, they get hit with a DDoS attack. It doesn’t last for long. Lo and behold, Technical starts talking with some staff, and offers to generously host their server for them for free, on his server with DDoS protection. He even convinces them that he works in infosec professionally.
I get in touch with their leadership after hearing about Technical trying to obtain their server files and offer to share everything I knew about his behavior so they can be best informed if they still wanted to take him up on his offer. Once they hear about the DDoS, they mention that they also had been hit by an attack - less than 20 hours before Technical contacted them. Mercifully, it was after the con had officially ended. After some discussion among their leadership, it was decided to not move services over to his servers.
So what happens now?
Hopefully, nothing. I have not pursued any legal action at this point, and sincerely hope that it won’t need to come to that. There are so many more constructive uses of my time and his time than to go at each other, especially since after confronting Technical about the attacks and the other things I caught him doing, BronyTales has not experienced any more DDoS attacks. Technical has kept to his word and maintained his distance, although he still denies having been behind the attacks. BronyTales did recover, although it took longer than he may ever know to undo the damage he caused. My sleep cycle recovered after a few weeks, although I will never get back the time spent going over this nonsense.
For those who decided to scroll to the bottom instead of reading all the way through the twenty-some pages of text that constitutes this minor epic, I’ll summarize for your benefit. In short, TechnicalRarity has a history of causing trouble while seeking attention, and some of that trouble goes from a minor nuisance and/or general harassment all the way up to being a potential danger to people he wants the attention of - up to and including physical danger, in some contexts.