Multi-Gaming Community

Working our way through the bugs...

Building towards a stable hosting platform. It's been quite a long process to get the new machine up and running as its suppose to. It took us loads of troubleshooting, hours of brain breaking mysteries , kilo’s of weed and litters of red bull to locate and fix the issue. In the end it turned out to be faulty BIOS memory settings and a faulty BIOS causing the L2-cache write from the memory to the CPU to perform so bad that it caused the whole system to screw up. Luckily we finally could locate the issue so we can go on with the rest of the work, which mostly got put on hold because of all working hours going into the new system. The new system now runs the servers nicely on the expected CPU load and with a rock solid 990 FPS. Now we can put all of our focus on the next issue which needs solving. The freezing issue. The issue wasn’t always around. For about 1 ½ year we were able to run everything smoothly. At one point, which no one can remember exactly when or how it started, these freeze’s started to show up. It’s incredible hard to locate the source of these freeze’s, as they occur absolute randomly without – from what it looks like – any pattern what so ever. To locate this issue we need to perform a lot of testing. Now I am a network engineer myself, so I can cover that part. But who will cover it at OS level? And who at gameserver level? Sure I know enough from Linux and srcds to troubleshoot and run it properly, but when it comes down to real testing, much help is required. We think the issue we experience is network related, as when we play a game on MrBlonde, we also notice hiccups in our voice communication trough mumble, located at MrWhite, at the same time as we experience the freeze’s. 2 Machines, 2 different applications, looks like network. So we did a lot of testing already, but all we can see is *some* packetloss, lower then 1% of the total traffic. In networking country, packetloss below 1% is accepted, as normally you wouldn’t notice a packet dropping left and right. But in our case, this are bursts of packetloss, rather then a packet every now and then. So we don’t go above the 1%, but still experience it as a highly annoying issue. The trick here is to provide proof to Leaseweb that our problems are caused by their equipment. What we need to do is somehow monitor the traffic, capture the packetloss bursts, and match this against the ingame freeze’s. Here is were you’ll see the power of the techteam combined. Ditch and Quitschbo are working together on writing custom scripts to perform networking tests on OS level, Demm is writing a Sourcemod script so we can easily store the times that freeze’s occur. All this data will get combined and analyzed by me, so I can make a solid case towards Leaseweb so they’ll solve the issue. As this is our highest priority work at the moment, I expect that we’ll be able to get rid of these freeze’s this month once and for all. Besides troubleshooting, we’ve also made more plans for the future. How our environment should look like, and how it should perform. Currently we have the following things planned on short term. We’ll be ordering 3 new harddisks for MrWhite and 1 for MrBlonde. As we had a major crash a month ago or so, we need to build in more stability and uptime insurance in our main server. We’re going to try to accomplish this by installing a RAID0+1 config in the server. This will cause the server to have quicker access to files on the harddisks, as well as backup functionality. With this config, if a harddisk fails, the system will be able to rebuild the raid simply by removing the broken disk, and insert a new one. No re-installation etc required. The 4th harddisk will get located in MrBlonde for daily external backups of our database and most important files. More work in terms of security will be done as well. We’ll be ordering and implementing a new to get Cisco ASA. This is a highly advanced hardware firewall which will be responsible of securing all of our servers and loads of important communication channels. As little extra’s, we’ll also be setting up a special, easy to use, and highly effective way to beat the German law restrictions on games, so also our German friends will be able to enjoy the game full of blood and ripped off body parts like we do! The future is looking good! - Saint K.

Comments

The professor says

"Good news everyone!" :D

Ha, ha! Exactly. Nice

Ha, ha! Exactly. Nice technical report, Saint.

:D

Way to go, I hope these freezes will be a thing of the past soon.. them coming up at wars is just baaaaaad~ :(

Crikey - you guys are

Crikey - you guys are spending ages on this. Big Thank You