Looks like a standard metal mesh reusable filter! The mesh is probably stainless steel, perhaps with a gold-tinted coating, sandwiched between a plastic frame. They're alternatives to paper filters for filter/drip coffee. I find the outcome to be a little gritty/grainy; more like French press than a regular paper filter.
In general, if your filter is still performing well, I think you're probably fine with a shake-out, rinse, or occasional scrub. However, any residue might impart off-tastes in your next brew (hopefully what's left is just coffee remnants!). The leftover coffee stuck to your filter (that you didn't wash off) will oxidise, taste stale, etc. (See also this question about coffee going stale -- one answer says it only takes minutes! And this question about what's happening with stale coffee.)
Wet coffee grounds will mould if allowed to sit too long as you report. So if you leave coffee in it, very wet, for a long time, you might be inviting flora and fauna that you don't want. You reported this in your question, but I'm surprised if it would happen very quickly. Clearly, when you have mould growing, you'll want to give it a good scrub. If you store your filter (grounds knocked out, even if not washed) in such a way that the filter can dry out more quickly (e.g., hanging, separate from the coffee maker, propped up to permit air flow, etc.) it seems that mould would have a harder time to grow.
As for getting clogged and occasionally needing a really thorough cleaning... I use a similar device for brewing tea, and it gets clogged sometimes. In fact, I wrote this question/answer on Cooking.SE recently about how it had become clogged after a decade of use; see that for more on how to get such a metal mesh really clean.
A probably-obvious alternative is to use a paper filter. You could toss the whole thing into your compost bin (filter and spent grounds and all!) and everything on the coffee maker will dry out more quickly.