Maybe it's because of the gobies? They warned us along time ago something like this may happen. Hopefully those panfish just moved out because a big Muskie was lurking around.
Gobies eat the eggs of other fish and hover around nests and dart in and out eating the eggs. Larger bass can protect the nest and probably eat the gobies that get close. Perch and sunfish don't have that luxury. That is why invasive species are such a problem.