Summer server shuffling - Scintilla's Server Shuffle

During the summer break the Scintilla Operator Team executed one of its biggest operations in years, which involved upgrading Scintilla’s two most important servers. Johan Verzijden, Chairman of the SOT, will tell you all about it and what let up to it.

To properly tell this story we’ll have to go back to last April. We were doing some long overdue software updates when we discovered a severely broken disk in Benedictus, Scintilla’s backup server. Because the disk was mirrored (an exact copy of the data was on a second disk) we hadn’t lost data, but we had to replace the disks as soon as possible. We had planned to simply replace the disks and be done with it. But unfortunately, it wasn’t that easy. After replacing one of the disks, Benedictus did not boot anymore.

It took us a whole afternoon to discover that the power supply was faulty and that it had fried the disk and the motherboard. We were able to get the server back online again quite quickly, using a thin client motherboard and a power supply fellow SOT member Sijmen had brought from home, but we concluded that a permanent solution had to be found.

During the next SOT meeting, it became clear that it would be best to replace Benedictus completely (except the disks of course, as these had just been replaced). But as previously mentioned, Benedictus is ‘only’ our backup server and therefore doesn’t need to run on brand-new hardware. That is why we decided to instead give our main server, called Alexia, an upgrade and use its old hardware for Benedictus. This made the whole operation a bit more complex, but it also meant that we could upgrade the whole server infrastructure of Scintilla just by buying one new server.

As you can imagine, such an operation would take a big part of Scintilla’s digital systems offline. That is why we decided on doing it during the summer break. That way, we still had time to organise it and it wouldn’t bother that many people. Based on previous maintenance projects, we estimated needing one day for each server, but decided on reserving a whole week to be on the safe side.

Now that we had decided on the what and the when, the next hurdle to be taken was the cost. The components we wanted to buy were not exactly cheap and we hadn’t expected such an expense when the budget of Scintilla was set for 2023, so there was no money set aside yet. That is why I wrote a letter to the General Meeting to ask for a budget of €3,000, which you can find here if you want to read it. Luckily the GM approved our request. In the meanwhile we had chosen a supplier and received a suitable quote from them, so we placed the order there and continued with the rest of the preparations.

Unless you are a member of SOT, you’re probably asking yourself by now “Wait, I didn’t know Scintilla had that many servers, are there more?”. Well, good that you’re asking! Let me explain. Scintilla owns four servers:

1. Alexia. It provides the main storage, on which all files and mail of both committees and users are stored. Next to that, it is also host to a lot of services, like Linscin, the mail server and Multivers¹. Don’t confuse it with the other Alexia by the way, that is a project of Inter-Actief (and some other associations) and has nothing to do with this server.
2. Benedictus. We of course have backups of all important data and those are all stored on Benedictus. It was originally named after pope Benedictus, but because he isn’t alive anymore, we found this an appropriate moment to rename the server to Franciscus, the current pope. Throughout this article the old name will be used though to prevent confusion.
3. Britt. If you are a frequent visitor of the Scintilla Room you probably have heard this name, as this server is responsible for the music in the Scintilla Room. But it does a lot more: the Balcony TV and SK TV are connected to it, it provides WiFi for ASM-ELSA (the fridge) and (sometimes) even sends you an SMS!
4. Utelscin. This server was only bought and installed last year, but its name dates back more than 20 years! Its only job right now is hosting the Scintilla Minecraft server, but we plan to use it for more in the near future. These four servers are placed in a couple of different locations. Alexia and Utelscin live together in the Seinhuis, a server building of the UT. As it is bad practice to put your backups physically next to the real data, Benedictus has its place in the other server building, Teehuis. Both are on campus, so you can see them on the campus map (numbers 7 and 14). And you probably have seen Britt already without knowing it, because it is located in the Board Room on one of the cabinets.

Now that you know a bit more about Scintilla’s servers and their purpose, let’s go back to Scintilla’s Server Shuffle. Due to our own agendas, we started on Tuesday. We unpacked the new components, which had already been assembled by the supplier, and made some space in the Board Room to work. After that it was time to shut down both Alexia and Benedictus and retrieve them from the Seinhuis and Teehuis respectively. We don’t have access to these buildings ourselves, so I made an appointment with the people of the UT NOC (Network Operation Center) beforehand so we could get to our servers. With both servers and the new chassis in the Board Room, we started by transferring all the disks of Alexia from the old chassis to the brand new one.

After a little reconfiguration, Alexia happily booted with its new motherboard and case. Now we could start to move the disks of Benedictus to the old chassis of Alexia, so we could get rid of the old chassis of Benedictus. This transfer took a little bit more time, as the hard disks were screwed down in the old case (see the picture at the start of the article) and they had to be screwed into separate trays instead so they would fit in the new case.

After that little bit of Mechanical Engineering, we encountered a problem more suited for Electrical Engineers: we weren’t able to get a stable signal from the VGA port for the display. Luckily for us, this motherboard had two other ways to communicate with it: a good old serial port and IPMI². These still worked, so using these we renamed Benedictus to Franciscus and did the rest of the reconfiguration to get it working in its new case. The only thing left to do now was bringing the servers back to their rooms in the Seinhuis and Teehuis. By Wednesday evening both servers were operational again and Scintilla’s Server Shuffle was done.

After all those preparations, I was really pleased to see that we were able to execute the operation within the two days we had estimated at the start, and that we didn’t encounter any major problems. I would like to thank my fellow SOT members Willem Mulder, Stan Groen and Sijmen Schoon for their help in bringing this project to a good end.

Do you want to know more about the Scintilla Operator Team, what we do or do you want to join? Don’t hesitate to send us an e-mail at sot@scintilla.utwente.nl or come by at one of our weekly tinkering evenings in the Scintilla Room!