Lets be honest, the chances of Mojang fixing the lag problem is like finding a skeleton with full diamond armor at night, and I've only ever seen one in my years of playing Minecraft. So with that said, this conversation will continue to come up unless the whole community and Crafty decide to change the server. And even then, someone will find a new problem to talk about. Yes this lag is so bad that many active and great players are leaving and considering not coming back, not I of course. This is a really nice community and I would hate to see everyone leave because of something that the server, nor the owner is not responsible for. There really is no solid answer for us to do besides play the lag infected game, or leave....

@BlueAce56 Lets be honest, the chances of Mojang fixing the lag problem is like finding a skeleton with full diamond armor at night, and I've only ever seen one in my years of playing Minecraft. So with that said, this conversation will continue to come up unless the whole community and Crafty decide to change the server. And even then, someone will find a new problem to talk about. Yes this lag is so bad that many active and great players are leaving and considering not coming back, not I of course. This is a really nice community and I would hate to see everyone leave because of something that the server, nor the owner is not responsible for. There really is no solid answer for us to do besides play the lag infected game, or leave....

It’s apparently been fixed
https://bugs.mojang.com/plugins/servlet/mobile#issue/MC-151772

While I feel that sticking to our vanilla routes is important, I know that this server is more than just a server or block game. In its more simplest form, it is yes, but this server provides a getaway for people. It is a way to distract from real life for a couple of hours and have some fun you might otherwise not. It has given me people I'd call my closest friends. I've even met a couple of people in person that I've played with for years on here. When I first joined the server I was in a dark place with my depression and family issues. When I had free time I could come on for a little bit and play with others. I didn't feel as alone and isolated from the world. I had people to spend time with to make the bad times go just a little bit faster and be a little more enjoyable. I think that our community is so strong and if we wish to stick to the principal of being vanilla the server will die out and we will lose all the creative, loving and caring people that make our server special and unique. As much as I would hate losing our vanilla status, I would rather lose that than each and every one of you lovely people that kept me going in those dark times and keep me company to this day :)

I hear you did something overnight Crafty.

Oh my god the improvement at the moment... solid 20 ticks per second for the first time in months... running so well. NICE WORK. Here's hoping it holds up as the "bad hours" hit later on.

Edit - Like, I can't even express how good it is. Mobs are everywhere in caves and it's actually crazy fun. So much fighting, everything moving fast. Seriously how did this happen! So good :D

Unfortunately during the "bad hours" things did indeed turn to real mush like any other night. The hours I mention in this post all refer to Canada (Ottawa) time.

It seems from roughly 2pm to 8pm things get insanely bad; that's what I've noticed for several nights now anyway. Tonight was perhaps maybe possibly a bit better (?), but it was still riding on the edge of unplayable from my perspective. Things were down to 7 TPS.

I'd be hesitant to just write it off as player count, because a) the difference is drastic -- going from 6 to 12 players can't solely be the cause of such a big change, and b) tonight it has clearly performed better with 16 players at around 9pm than it performed with 11 players at around 6pm.

Once it started improving again (after around 8pm), it was definitely starting to act more like the magical improvement we saw earlier today. It wasn't nearly as good yet, but it did appear to be recovering in a way that didn't happen in previous days, made most evident by the occasional return to 20 TPS. I don't think the server has really hit 20 TPS (or even came close) in the last month until your recent magical fix.

Anyway, just thoughts and observations. I guess you guys already have logs and stats that can tell you as much without my fuzzy descriptions. So a summary:
Good hours = mind-blowing good. Drop any thought of spigot like a brick if CM can hold onto that performance.
Bad hours = still super bad and really frustrating. Personal suspicion about how the performance drop seems to sync up with (real world) time of day more than player count.

And all that said, playing today on Crafty while it was good was some of the funnest Minecrafting I've had. It was sooo good; especially with raids, and caves just packed with never-ending mobs.

@Jkeller4000 Adding on to what Crafty said, Mojang has historically neglected the benefits of focusing on performance and (ideally, but not likely) shifting over to a multi threaded approach to the game logic. There's no way you can tweak the variables of the vanilla server to get the max performance possible without radically changing the server jar's programming (with something like nSpigot), unless you want to severely diminish the server experience.

We live in an era where we have 8 or even 16 cores in consumer grade CPUs, and for every vanilla and normal Spigot server, every single core in their CPU is twiddling their thumbs around except for one, which is busy handling chunk loading, entity processing, command blocks, chunk and world serialization, and literally every other aspect about game logic aside from chat (???). This is just old lazy code carried over from the Notch days, and Mojang, despite developing one of the biggest games of all time and having the resources to do so, still hasn't done jack to better the performance of their game.

The only practical solutions here are to wait for Mojang to get their shit together and fix the many issues they've introduced with 1.14, or replace the server software with something that actually focuses on high performance like Spigot.