A few fallen leaves on soil are normal and can even protect the surface a bit. But if you let a thick layer build up, it can block air flow and trap too much moisture at the top. That creates a friendly environment for mould, fungus and pests.
Sweeping or gently removing extra fallen leaves keeps the soil surface breathable. Water can soak in evenly, and you can easily see if the soil looks too dry or too wet. It also discourages insects that like dark, damp hiding spots.
Plus, your pots simply look neater and more cared-for. A quick tidy-up every few days is usually enough.