I got a tea infuser with a very fine mesh and put my grounds in that. It holds fewer grounds than I used with the "raw" plunge, but it's much more pleasant to use: I just pluck the floating infuser out of the pitcher and empty the grounds from it. I may get a second filter, or a larger one, but for now I'm content making coffee more often because it's less of a chore each time.
There's no visible particulate in the resulting drink, though if I run the brew through a filter (which is a heckuva lot faster, as there's no clogging of grounds at all!) it comes out with a small amount of lint-like residue. And the last cup in the pitcher is much stronger than it used to be, but I'm not exactly complaining about that.
You could also go for an intermediate approach, using a tea strainer before you run the brew through a regular filter: that'll get the big stuff out fast so your filtering doesn't slow down. (Also, put a teaspoon in the funnel under the filter, so the grounds don't stick to the sides of the funnel through the paper. It gives the liquid more room to flow.)