Its really hard to blame our host for the lag, this is the same server that allowed us to run 88 players on 1.8 with little to no lag (until our map got huge). We have the best single threaded performance that money can buy, a i7 4790k @ 4.7 ghz, 32gb 1866mhz ram, 250gb 850 evo SSD, 1gbps network. Other servers running this exact machine but with spigot have proven to run 600+ players on the same server (no bungee/proxy). Apples to Apples, same machine, the only thing different is the server jar file, even if there is something we are doing wrong, a difference of 550 players is huge. Vanilla is just not cut out to run this many players.

The reason 1.9 is "100 times worse" and the reason it could lag more with half as many players is because of this issue: https://bugs.mojang.com/browse/MC-98822 and this one https://bugs.mojang.com/browse/MC-94438

Essentially, entities do not cause lag in a linear fashion, they are exponential.
Let's play with some numbers...

His results:
512 : 82 FPS ; Very few lag.
1024 : 2 FPS ; Unplayable, I can't even see the top of the debug pie.
2048 or higher : Minecraft crash, so I can't even take a screenshot.

let's say everyone was able to keep to a 100 entity limit cap and we had 50 players on, that's 5000 entities, over double what it took to make his minecraft crash, it's a miracle that we are even able to run with more then 5 players.

You say that other servers have a huge number of players yet don't have this much lag.

#1. As far as I know, we have the most players out of any Vanilla server, so unless you are talking about none spigot servers, this is a completely unfair comparison.

#2. Most vanilla servers have had to drop their render distance to deal with the lag, some even going as as low as they can go (3 chunk render distance). If you want horse blinders then 3 chunk render distance is for you.

#3. I hear non-stop complaints about lag from other vanilla servers all the time, 1.9 has really made a mess.

Ok Crafty, now that I know we are fucked, what are you going to do about it?

Hopefully mojang will be releasing 1.9.1 soon that will come with a patch for the entity issue, they already have some pre-releases, but they haven't fixed it yet.

To fix the network issues we are moving data centers and at the same time we will be upgrading to a 6700k also at 4.7ghz, hopefully this will give us about 3% more power, and with the new way I hope to setup the server maybe even more.

Until this happens, all we can do is watch our entities, all it takes is one inconsiderate player to let their mob grinder get away from them and the server is brought to its knees.

Other solutions are decreasing our render distance, but i doubt that players would want this as we are already close to the minimum possible value.

Money wise, I did go all out on this new server, I made sure it had extra SSD and HD storage so that we don't have to waste performance writing backups to a different, slower hard drive. Knowing we had the money and donors to do this is what allows me to not compromise performance for money.

- Crafty

@Shapedsundew9 If we want Mojang to spend time on this we need to put it in thier face. It is all about visibility. Who ever is managing this is probably only looking at the top 5, maybe 10, issues each week.b

Yes, we need votes on this, a long time ago I created a hack (sign hack) and kept it as a private bug report. Said bug report was not even looked at for about 2 months, by this point I had already been able to "fix" it on my end by making it impossible for players to op themselfs.

About 3 weeks later, wurst came out with the hack and spigot, bukkit, vanilla, all servers were affected, spigot was patched very quickly but it took almost another month for vanilla to even touch the issue. Servers were getting destroyed left right and center.

I think what is more funny is the features that come to vanilla are almost always a shitty version of a plugin that had been made like a year before. Pistons, Hoppers (buildcraft/tekkit pipes), end island, the list goes on, the funny thing is that spigot 1.9 released the same week as vanilla 1.9, removing any early access benefit to using vanilla. I think it's safe to say that vanilla is just getting royally f**ked.

That's the thing, mojang doesn't make money off servers, especially vanilla servers. They sell the features and a shitty client, then the community cleans up their shit and makes a server that can handle more then 3 players as well as mods and youtube videos.