As has been mentioned, the species of coffee makes a difference. There are something like a hundred different species, with Robusta and Arabica being the most common commercially grown by far. Liberica is another species that is grown in small quantities and has a very distinctive taste.
Within a single species such as Arabica, there can be a huge number of varietals. Some are naturally occuring (such as Bourbon and Typica). Maragogype ("elephant bean") is also a naturally occuring mutation from Brazil. Others have been bred by research institutes. For example SL-28 and SL-34 are common Kenyan varietals.
Another famous varietal is the "Gesha" (sometimes romantically but incorrectly called "Geisha"). This came to light when an estate in Pananma "La Esmarelda" noted something interesting when they were cupping their coffee. By separating their harvest, they finally identified a small section which was contributing to the overall cup. This coffee went on to dominate the Cup of Excellence competition and set a world record price at the time. It's since been planted on other farms which brings up the next point.....
As has been mentioned, things like altitude, soil and micro-climate affect the taste. So the Gesha (or any other) varietal from one farm won't taste exactly the same as Gesha from another farm.
The way of preparing the green beans - washed, dry-processed, or a number of other methods, all have their effect on the final flavour. So even the identical cultivar harvested from the same farm but prepared differently will produce different flavours. (The difference between dry and wet-processed is very signficant.) The actual care taken during preparation is also hugely important - I suspect that if one is starting out with a decent high grown bean, this is what really separates speciality coffee from the mass produced stuff. It takes a lot more care to prep the coffee carefully.
And I haven't even mentioned roasting and brewing....
I think this is what makes coffee so interesting - there is so much variety to try and enjoy.