A day in the life of a server admin
Hey y’all! Another late-night blog with little more than an idea and a deadline: How about a blurb about what goes into managing a server like BronyTales? More after the break!
Most weekends and most nights, I tend to spend more time working on server-related tasks than playing games, personal tasks like cleaning out my fridge, or visiting with family. Making sure that the community and the server is looked after and running smooth consumes a majority of my free time. I might be a little obsessed. Seriously, I love you guys. Ya’ll are crazy and drive me up the wall sometimes, but I wouldn’t have it any other way. ❤️
A few weeks ago, I spent my entire weekend doing just doing “paperwork” and thanks to a checklist with crossed-out items I can give you a rough idea what “paperwork” can mean when a Minecraft server is involved. Even fun and games have a good deal of paperwork involved, and a continually evolving game is a never ending list of chores more often than not.
We’ll start on Thursday night. January 14th, 2021. It’s a little after getting off from work. It's cold outside. Food in hand, planning to do. We’re planning a large party for Friday, and there are things to be done. I won’t be able to join until later because of work, but I can help make sure everything is ready. We’re planning on doing a scavenger hunt with a new race at the end, and there are a few final checks to make sure it will work when needed. I hang out in voice while going over configuration checks, and eventually get caught up in a game of Clue in the creative map! I didn’t win that one.
Afterwards, in a burst of midnight-fueled inspiration, I figure, “hey, why just release Griffon? Timely has a thing for an Umbrum.. Why not release that at the same time?”
With a great deal of help from Penny, Abella (Silverstream), and Moonstory (Shining Armor), we create a hidden secret area where people can obtain Umbrum for themselves! If they can brave the mobs and beautifully edgy architecture, that is. And, just because our most dedicated and charismatic Unicorn/Umbrum player is currently online, whisper a little rumor that there is a secret out that they may find interesting.. Jump out to help a player with a chest that needs to be unlocked, jump back, and Sombra’s already found it, heh.
I finally get to bed around 2:20 AM. I need to wake up at 6:30 AM. Work in the morning is going to be fun.
I survive Friday mostly, and manage to join in the festivities just in time to get in on two rounds of spleef. Also, over 50 players online at the same time! That’s amazing! Shortly after griffon is officially available, the server starts to have a panic attack. We are able to isolate why later, but at the time we were pretty focused on trying to make sure the server stayed online. The server didn’t quite crash, but caused a large number of players to disconnect at its worst point. After we were back up, a game of chess with Trixie! I managed to win that one. And then banning a new player for asking for child pornography. Yes, that happened. That’s not going to fly here. Nope. Don’t even joke about it. Their friend managed to autoban themselves with the language filter right after. That taken care of, then hosting some Jackbox games in general voice chat! During this, I notice there’s an error popping up in the console related to the treasure hunt - somehow a null was getting into the rewards. Update the code to check for that, push out to the server, all good to go! Then more Jackbox games until 1 AM.
I stayed up far too late for my normal bedtime hanging out on BT’s voice chat and playing games and consequently slept in a little later than I normally get up. Well, aside from a brief un-nap because a certain puppy will not be denied her 6am potty break. Some days she tells time better than I do - she regularly wakes me up a few minutes before my morning alarm goes off. 6:30, wake up to dog nose. Walk dog to back door. Hide behind door. Outside is cold. Hide in warm bed until dog returns. Close door. Return to land of long naps.
Having a heating blanket is amazing.
Sometime around 9, finally surrender the comfort of my comfy respite, make some toast, and start tackling my todo list.
Oh no. The server crashed. That’s bad. It has been down since 5. This is bad. Thankfully, it’s a known glitch, easy to fix. Still pretty annoying.
Welp, I’m up now. First fire put out, what else is there? Turns out quite a bit. Some stats and one config file got corrupted. Some people didn’t get the boots that were supposed to be available for Griffon for the scavenger hunt, and a number of people no longer had the Griffon race. That’s also very not good. Like, really not good. Those two things aren’t supposed to be related. Long list of things to do ahead of me, plus the fact that this is the first time this sort of glitch has happened, I just do some manual updates for players and try to keep things generally running. That missing config value explains why the error from the day before was happening, though.
First on my list of things to check up with was why one of the plugins was occasionally throwing errors in the server console. The night before in voice chat, someone had mentioned trying to get a villager to switch jobs and none of the trades with enchanting books had any enchantments on them. At the same time they were trying to get the trades to work, the errors were popping up in the console. This had been an error I’d noticed before, but had not been able to trace down what was causing it. Now I had a lead.
As this particular plugin is opensource, I was able to dig through the code to isolate where it was happening - turns out they were trying to change the runtime code of the Minecraft server to allow modifying a protected variable. However, the particular method that they were using to accomplish this had been deprecated for a long time and had been removed in the version of Java that the server is using. After verifying that I could reliably reproduce the error, I modified that section to not call the missing function, tested again, and found that the code worked without it just fine. Tested with the older Java, and that worked fine, too. Pushed out a version of the code without those deprecated functions to a fork, created a pull request to merge my changes with their project, uploaded my new build to BronyTales, all good! Next!
An issue I’d never had to really address, but was encountering a lot while working on race resets, was that I’d forgotten to include a feedback message for the manual pony level upgrade admin command. You could run the command, but it wouldn’t let you know what happened until they were a max level. That’s an easy enough fix, though. Add in the message, build the new version, push to the server. Done!
Spent some time coding an easy method for me to upload changelogs to the website. This part’s not particularly interesting, but I did manage to simplify it enough that not only is the main packend page easier for me to work with now, but it also automagically plugs in the formatting I wanted! Fun! These pages are now live at Rules and Policy Changes and Service Changes
Three hours later, noticing that Daring is online and since I’m working on tracking down bugs, there’s another one that I’d heard about and wanted to verify. I’d heard that moving stuff into chests was appreciably causing server lag, and I wanted to verify and possibly track down what was causing it. With some help from Daring Shepard and Kanade Tachi (Celestia), we manage to isolate the issue to specifically clicking on inventory slots, and the source of the excessive lag originated in the kits plugin in use for the fight club! Pulling apart the program, the reason becomes immediately clear, and it’s not an easy fix. It’s not using a very efficient method of checking for admin gui access, and it’s causing a lot of unnecessary processing every time anyone clicks into any inventory interface slot. I remove the plugin for now, and close off the fight club entrance until I can properly sit down and wrestle the plugin into working order. We have a bigger problem to fix. Yes, bigger than extreme lag.
Shortly after finding this last issue, we start having an unusual symptom start showing up everywhere on the server where some players were mysteriously unable to connect to the server. While looking into it, it suddenly happens to my account! I’m unable to join for more than a second - it loads in, I can see stuff around me, but it immediately closes the connection without any reason given. After messing with profiles and locations, I check the proxy server and notice that it’s complaining about being unable to allocate more memory. Well, that’s an easy enough fix: reboot! While doing the reboot, also lined up an update for the bedrock translator. One player got banned for something odd that happened during that glitch, so unbanned that player.
Next up on the list of issues, I’d noticed that spectators were still generating heart particles in the world when hugged. Not normally a big issue, but could be somewhat distracting in the fight club arena. A quick change to the code to check for gamemode before selecting which players to send the particles to and we’re in business! Next!
Next issue is.. Drama. Great. A very lively conversation in the general chat room spawned from an @-everyone announcing that the memes channel was being temporarily restricted so that it could be purged of inappropriate content that a user had been posting. Generally neutral chaos until two individuals turned to pointed insults in lieu of constructive conversation. Please be kind to other people, people. We’re all people at the end of the day. There’s no need to be tearing each other down.
I work more on the website and on some administrative stuff relating to the January 24th rules update, and later on end up talking about client side mods until 2AM. Staying up far too late seems to be a recurring theme..
Falling further into the realm of sleep depravity, I sleep in a tad later than the previous day, and it’s nearly noon when I finally shuffle into my office to discover the server has gone down. Again. Same issue as yesterday. Bleh. First thing on my todo after fixing the server down issue isn’t actually on my todo, just adding autocomplete to a staff utility. Autocomplete is so nice to have.
Another on my list of quick fixes for improving quality of life: When I originally created the inventory rollback for pony temples, I had intended it to be a “just-in-case” sort of failsafe: it shouldn’t ever fail, but just in case it does, it should have an easy way to restore the lost inventory. Nothing is quite as frustrating as a glitch causing you to lose a special item. It’s a good thing I did add that in, too, as we’ve had a few really weird and impossible to reproduce glitches do just that. Unfortunately, the format the rollback command used was not meant to be user-friendly. A few tweaks to formatting and some raw json magic, and now I have a somewhat easier to read display! Got that pushed out to the server without much hassle. Next!
That done, and several more race fixes later, thinking back to the server crash.. There has to be a better way of making sure the server doesn’t go down for long than just checking the server console in the morning. Kinda wishing someone would have called me or something… wait. I just created a texting notification service at work a while back, where it’ll send a text alert to my work phone if any servers go down. What if I did that for BT? Cue hacker montage. Yep!
While developing and testing this new monitoring service, I notice a very questionable Minecraft username join the server and complete the trials very quickly. A cursory check for their Minecraft skin, and it’s a cartoon character in a soiled diaper. Yeah, no. That’s not going to fly here, thank you very much. Got on the server to tell them to change their skin, and they left to do just that. They never rejoined, oddly enough. One did change their skin, but the other didn’t. :shrug:
Jumped on the build server to work a little on a future project until 10 PM. A little late, but not impossible for getting up for work.
On Monday, coincidentally right about when I was done with work for the day, my phone went off with a server monitoring alert - the server was down again! During operating hours! Logging into an admin terminal and poking around, I find what may have been the source of all of the previous crashes: a glitch in another service on the machine was filling up log files with literal gigabytes of spam, which would get rolled over and the space freed up on its own periodically. Deleted the junk files, fixed the error causing it, spent four hours running a delete operation to free up database storage, and added additional checks to my monitoring service to send me an alert if disk space, RAM, or CPU becomes restrictive in the future. This should provide enough warning to give me time to clean up issues before they affect the server too much. Thankfully, the server hasn’t gone down since, and no more data glitches! Well, aside from a nickname issue, but I’m still working on hunting down what’s causing that.
And now it’s 3 AM. See what I mean by a recurring theme?
Night all, seeya next blog!
Image
Most weekends and most nights, I tend to spend more time working on server-related tasks than playing games, personal tasks like cleaning out my fridge, or visiting with family. Making sure that the community and the server is looked after and running smooth consumes a majority of my free time. I might be a little obsessed. Seriously, I love you guys. Ya’ll are crazy and drive me up the wall sometimes, but I wouldn’t have it any other way. ❤️
A few weeks ago, I spent my entire weekend doing just doing “paperwork” and thanks to a checklist with crossed-out items I can give you a rough idea what “paperwork” can mean when a Minecraft server is involved. Even fun and games have a good deal of paperwork involved, and a continually evolving game is a never ending list of chores more often than not.
We’ll start on Thursday night. January 14th, 2021. It’s a little after getting off from work. It's cold outside. Food in hand, planning to do. We’re planning a large party for Friday, and there are things to be done. I won’t be able to join until later because of work, but I can help make sure everything is ready. We’re planning on doing a scavenger hunt with a new race at the end, and there are a few final checks to make sure it will work when needed. I hang out in voice while going over configuration checks, and eventually get caught up in a game of Clue in the creative map! I didn’t win that one.
Afterwards, in a burst of midnight-fueled inspiration, I figure, “hey, why just release Griffon? Timely has a thing for an Umbrum.. Why not release that at the same time?”
With a great deal of help from Penny, Abella (Silverstream), and Moonstory (Shining Armor), we create a hidden secret area where people can obtain Umbrum for themselves! If they can brave the mobs and beautifully edgy architecture, that is. And, just because our most dedicated and charismatic Unicorn/Umbrum player is currently online, whisper a little rumor that there is a secret out that they may find interesting.. Jump out to help a player with a chest that needs to be unlocked, jump back, and Sombra’s already found it, heh.
I finally get to bed around 2:20 AM. I need to wake up at 6:30 AM. Work in the morning is going to be fun.
I survive Friday mostly, and manage to join in the festivities just in time to get in on two rounds of spleef. Also, over 50 players online at the same time! That’s amazing! Shortly after griffon is officially available, the server starts to have a panic attack. We are able to isolate why later, but at the time we were pretty focused on trying to make sure the server stayed online. The server didn’t quite crash, but caused a large number of players to disconnect at its worst point. After we were back up, a game of chess with Trixie! I managed to win that one. And then banning a new player for asking for child pornography. Yes, that happened. That’s not going to fly here. Nope. Don’t even joke about it. Their friend managed to autoban themselves with the language filter right after. That taken care of, then hosting some Jackbox games in general voice chat! During this, I notice there’s an error popping up in the console related to the treasure hunt - somehow a null was getting into the rewards. Update the code to check for that, push out to the server, all good to go! Then more Jackbox games until 1 AM.
I stayed up far too late for my normal bedtime hanging out on BT’s voice chat and playing games and consequently slept in a little later than I normally get up. Well, aside from a brief un-nap because a certain puppy will not be denied her 6am potty break. Some days she tells time better than I do - she regularly wakes me up a few minutes before my morning alarm goes off. 6:30, wake up to dog nose. Walk dog to back door. Hide behind door. Outside is cold. Hide in warm bed until dog returns. Close door. Return to land of long naps.
Having a heating blanket is amazing.
Sometime around 9, finally surrender the comfort of my comfy respite, make some toast, and start tackling my todo list.
Oh no. The server crashed. That’s bad. It has been down since 5. This is bad. Thankfully, it’s a known glitch, easy to fix. Still pretty annoying.
Welp, I’m up now. First fire put out, what else is there? Turns out quite a bit. Some stats and one config file got corrupted. Some people didn’t get the boots that were supposed to be available for Griffon for the scavenger hunt, and a number of people no longer had the Griffon race. That’s also very not good. Like, really not good. Those two things aren’t supposed to be related. Long list of things to do ahead of me, plus the fact that this is the first time this sort of glitch has happened, I just do some manual updates for players and try to keep things generally running. That missing config value explains why the error from the day before was happening, though.
First on my list of things to check up with was why one of the plugins was occasionally throwing errors in the server console. The night before in voice chat, someone had mentioned trying to get a villager to switch jobs and none of the trades with enchanting books had any enchantments on them. At the same time they were trying to get the trades to work, the errors were popping up in the console. This had been an error I’d noticed before, but had not been able to trace down what was causing it. Now I had a lead.
As this particular plugin is opensource, I was able to dig through the code to isolate where it was happening - turns out they were trying to change the runtime code of the Minecraft server to allow modifying a protected variable. However, the particular method that they were using to accomplish this had been deprecated for a long time and had been removed in the version of Java that the server is using. After verifying that I could reliably reproduce the error, I modified that section to not call the missing function, tested again, and found that the code worked without it just fine. Tested with the older Java, and that worked fine, too. Pushed out a version of the code without those deprecated functions to a fork, created a pull request to merge my changes with their project, uploaded my new build to BronyTales, all good! Next!
An issue I’d never had to really address, but was encountering a lot while working on race resets, was that I’d forgotten to include a feedback message for the manual pony level upgrade admin command. You could run the command, but it wouldn’t let you know what happened until they were a max level. That’s an easy enough fix, though. Add in the message, build the new version, push to the server. Done!
Spent some time coding an easy method for me to upload changelogs to the website. This part’s not particularly interesting, but I did manage to simplify it enough that not only is the main packend page easier for me to work with now, but it also automagically plugs in the formatting I wanted! Fun! These pages are now live at Rules and Policy Changes and Service Changes
Three hours later, noticing that Daring is online and since I’m working on tracking down bugs, there’s another one that I’d heard about and wanted to verify. I’d heard that moving stuff into chests was appreciably causing server lag, and I wanted to verify and possibly track down what was causing it. With some help from Daring Shepard and Kanade Tachi (Celestia), we manage to isolate the issue to specifically clicking on inventory slots, and the source of the excessive lag originated in the kits plugin in use for the fight club! Pulling apart the program, the reason becomes immediately clear, and it’s not an easy fix. It’s not using a very efficient method of checking for admin gui access, and it’s causing a lot of unnecessary processing every time anyone clicks into any inventory interface slot. I remove the plugin for now, and close off the fight club entrance until I can properly sit down and wrestle the plugin into working order. We have a bigger problem to fix. Yes, bigger than extreme lag.
Shortly after finding this last issue, we start having an unusual symptom start showing up everywhere on the server where some players were mysteriously unable to connect to the server. While looking into it, it suddenly happens to my account! I’m unable to join for more than a second - it loads in, I can see stuff around me, but it immediately closes the connection without any reason given. After messing with profiles and locations, I check the proxy server and notice that it’s complaining about being unable to allocate more memory. Well, that’s an easy enough fix: reboot! While doing the reboot, also lined up an update for the bedrock translator. One player got banned for something odd that happened during that glitch, so unbanned that player.
Next up on the list of issues, I’d noticed that spectators were still generating heart particles in the world when hugged. Not normally a big issue, but could be somewhat distracting in the fight club arena. A quick change to the code to check for gamemode before selecting which players to send the particles to and we’re in business! Next!
Next issue is.. Drama. Great. A very lively conversation in the general chat room spawned from an @-everyone announcing that the memes channel was being temporarily restricted so that it could be purged of inappropriate content that a user had been posting. Generally neutral chaos until two individuals turned to pointed insults in lieu of constructive conversation. Please be kind to other people, people. We’re all people at the end of the day. There’s no need to be tearing each other down.
I work more on the website and on some administrative stuff relating to the January 24th rules update, and later on end up talking about client side mods until 2AM. Staying up far too late seems to be a recurring theme..
Falling further into the realm of sleep depravity, I sleep in a tad later than the previous day, and it’s nearly noon when I finally shuffle into my office to discover the server has gone down. Again. Same issue as yesterday. Bleh. First thing on my todo after fixing the server down issue isn’t actually on my todo, just adding autocomplete to a staff utility. Autocomplete is so nice to have.
Another on my list of quick fixes for improving quality of life: When I originally created the inventory rollback for pony temples, I had intended it to be a “just-in-case” sort of failsafe: it shouldn’t ever fail, but just in case it does, it should have an easy way to restore the lost inventory. Nothing is quite as frustrating as a glitch causing you to lose a special item. It’s a good thing I did add that in, too, as we’ve had a few really weird and impossible to reproduce glitches do just that. Unfortunately, the format the rollback command used was not meant to be user-friendly. A few tweaks to formatting and some raw json magic, and now I have a somewhat easier to read display! Got that pushed out to the server without much hassle. Next!
That done, and several more race fixes later, thinking back to the server crash.. There has to be a better way of making sure the server doesn’t go down for long than just checking the server console in the morning. Kinda wishing someone would have called me or something… wait. I just created a texting notification service at work a while back, where it’ll send a text alert to my work phone if any servers go down. What if I did that for BT? Cue hacker montage. Yep!
While developing and testing this new monitoring service, I notice a very questionable Minecraft username join the server and complete the trials very quickly. A cursory check for their Minecraft skin, and it’s a cartoon character in a soiled diaper. Yeah, no. That’s not going to fly here, thank you very much. Got on the server to tell them to change their skin, and they left to do just that. They never rejoined, oddly enough. One did change their skin, but the other didn’t. :shrug:
Jumped on the build server to work a little on a future project until 10 PM. A little late, but not impossible for getting up for work.
On Monday, coincidentally right about when I was done with work for the day, my phone went off with a server monitoring alert - the server was down again! During operating hours! Logging into an admin terminal and poking around, I find what may have been the source of all of the previous crashes: a glitch in another service on the machine was filling up log files with literal gigabytes of spam, which would get rolled over and the space freed up on its own periodically. Deleted the junk files, fixed the error causing it, spent four hours running a delete operation to free up database storage, and added additional checks to my monitoring service to send me an alert if disk space, RAM, or CPU becomes restrictive in the future. This should provide enough warning to give me time to clean up issues before they affect the server too much. Thankfully, the server hasn’t gone down since, and no more data glitches! Well, aside from a nickname issue, but I’m still working on hunting down what’s causing that.
And now it’s 3 AM. See what I mean by a recurring theme?
Night all, seeya next blog!
Image