We raised sheep for 30 years. The difference is this: people who know how to cook lamb and those who don't.
Beyond that, and most of it has been mentioned on here: breed (be it a meat breed or wool breed), what it's fed, how it's processed all play into it. But if you don't know how to cook it, you won't like it.
On the mutton vs. lamb thing, most U.S. folks were raised to not like lamb because that's what was passed down from generations...particularly after WWII. Soldiers were fed mutton, not lamb. We had several lamb roasts where we cooked a whole lamb. We had lots of people who would swear they hated lamb, but damn did they eat a lot of it.
And if you see lamb labeled as lamb here in the U.S., despite what somebody in this thread tried to falsely claim, it's from a lamb that is less than 12 months in age. If both cannon joints break on the front legs of a carcass, it's called a lamb. If not, it's called either yearling mutton or mutton. Once a lamb reaches a year in age, the joints in the cannon joints harden and they don't break.
http://www.ams.usda.gov/AMSv1.0/getfile?dDocName=STELDEV3060365
New Zealand and Australia use the presence of incisor teeth to determine if it's mutton or lamb.
http://www.interest.co.nz/lambgrade.pdf
The page on wikipedia is a perfect example of why you don't believe wikipedia unless you confirm it. It says lamb in the U.S. is determined by the fact it's meat from an ovine animal. Obviously written by a Ozzie or Kiwi who is skairt of that good U.S. lamb from English breed animals...which is probably why so many people like the lamb they've eaten in the UK. It's probably from Suffolk, Hampshire, Shropshire breeds, or maybe the only white face breed worth having, a Dorset.