Its really hard to blame our host for the lag, this is the same server that allowed us to run 88 players on 1.8 with little to no lag (until our map got huge). We have the best single threaded performance that money can buy, a i7 4790k @ 4.7 ghz, 32gb 1866mhz ram, 250gb 850 evo SSD, 1gbps network. Other servers running this exact machine but with spigot have proven to run 600+ players on the same server (no bungee/proxy). Apples to Apples, same machine, the only thing different is the server jar file, even if there is something we are doing wrong, a difference of 550 players is huge. Vanilla is just not cut out to run this many players.
The reason 1.9 is "100 times worse" and the reason it could lag more with half as many players is because of this issue: https://bugs.mojang.com/browse/MC-98822 and this one https://bugs.mojang.com/browse/MC-94438
Essentially, entities do not cause lag in a linear fashion, they are exponential.
Let's play with some numbers...
His results:
512 : 82 FPS ; Very few lag.
1024 : 2 FPS ; Unplayable, I can't even see the top of the debug pie.
2048 or higher : Minecraft crash, so I can't even take a screenshot.
let's say everyone was able to keep to a 100 entity limit cap and we had 50 players on, that's 5000 entities, over double what it took to make his minecraft crash, it's a miracle that we are even able to run with more then 5 players.
You say that other servers have a huge number of players yet don't have this much lag.
#1. As far as I know, we have the most players out of any Vanilla server, so unless you are talking about none spigot servers, this is a completely unfair comparison.
#2. Most vanilla servers have had to drop their render distance to deal with the lag, some even going as as low as they can go (3 chunk render distance). If you want horse blinders then 3 chunk render distance is for you.
#3. I hear non-stop complaints about lag from other vanilla servers all the time, 1.9 has really made a mess.
Ok Crafty, now that I know we are fucked, what are you going to do about it?
Hopefully mojang will be releasing 1.9.1 soon that will come with a patch for the entity issue, they already have some pre-releases, but they haven't fixed it yet.
To fix the network issues we are moving data centers and at the same time we will be upgrading to a 6700k also at 4.7ghz, hopefully this will give us about 3% more power, and with the new way I hope to setup the server maybe even more.
Until this happens, all we can do is watch our entities, all it takes is one inconsiderate player to let their mob grinder get away from them and the server is brought to its knees.
Other solutions are decreasing our render distance, but i doubt that players would want this as we are already close to the minimum possible value.
Money wise, I did go all out on this new server, I made sure it had extra SSD and HD storage so that we don't have to waste performance writing backups to a different, slower hard drive. Knowing we had the money and donors to do this is what allows me to not compromise performance for money.
- Crafty