A few issues with this thought.
If it kicks a player that has over 500 entities and that player was alone at the time then the chunks will simply unload. Items do not despawn in unloaded chunks, so the entities would still be there when the player was allowed back online.
Then the player would get kicked again because the entities are still over 500. It would be an endless loop.
Also, how would they ever get picked up? If any player got kicked once 500 entities were reached anyone that went near the entities would be kicked and not able to clean up the mess.
Just the counting of entities is laggy, we have systems in place for keeping track of entities. If there are too many of one type of entity it randomly kills some of them until it's under a set amount. If there are way too many entities in the world it will pause the spawning of entities. We do quite a bit to limit entities as well as staff checking players. I have learned that there's a balance between automation with functions and lag. Too much checking/anti-cheat code causes lag, about 15-20% of the lag comes just from anti dupe, anti lava, hacking detection, and other functions. The reason I have implemented so much in the panel is to remove as much as I can from the slow Minecraft functions.