Because it is quite hard to take the basket out of the portafilter, I tend to leave it there. But this means I can't clean the space between them. Is that OK?
The space between them is very hot to be cleaned right after preparing the espresso, actually. It is better to wait for a while and then remove the portafilter to clean if you are not planning to prepare a second one right after. See image below from a manual.
The space in between can be cleaned after given number of use. E.g. after every 50 use.
Normally, I tend to use decalcifying agents after per 100-150 use. Then, I also remove the parts and clean thoroughly. Otherwise, I just wipe the space in between with a wet cloth.
I guess the answer is it all depends on your taste.
The espresso purest would tell you that there are coffee oils that are left on all surfaces of the brew path after the water flows through the grounds and these oils quickly become rancid. I'm not really sure how much oil is left on the portafilter after a shot is prepared, but needless to say, this oil will be present, and in order for it to not affect the flavor of any subsequent shots, needs to be cleaned.
If you cannot detect a flavor difference in the second shot and like the way it tastes, don't worry about it and enjoy your doppio. Otherwise do some experimenting to really see if cleaning it gives you a better result that you desire.