The wisps only spawn when someone is on the island and it takes 2-3 minutes to get a decent spawn. Just begin farming the pooh.

When you leave the island (and nobody else stays) they despawn very fast.