Yes, it makes a difference to the taste. As for what to pick - what you like.
My favorite is "Vermont Coffee" - ie, use maple syrup (or sugar, but syrup is more cost-effective, at least near a production area where the cost is not much shipping.)
Friends in the UK introduced me to Demerara, which is a not-quite fully refined sugar with a delicate carmel taste. Some of the "raw sugar" products sold in the USA approximate this, and some better than others.
At times I use USA brown sugar, which is basically white sugar sprayed with molasses, and not nearly as delicately flavored as a result.
Plain old white sugar is what it is, and if that's what you want, that's fine too.
I certainly have no difficulty distinguishing all of these from each other, in coffee - your taste buds/nose may vary.
Grain size affects only speed and ease of dissolving. Particularly in summer I tend to keep a bottle of 2 sugar:1 water simple syrup in the fridge for easily mixed, pre-dissolved sugar (not just for coffee.)