Why Do Britons and Americans Spell Words Differently?
The first question is why are British and American spellings different for certain words?
Noah Webster
The first answer is to blame Noah Webster, of Webster's Dictionary fame. He believed it was important for America, a new and revolutionary nation, to assert its cultural independence from Britain through language. He wrote the first American spelling, grammar, and reading schoolbooks and the first American dictionary. He was also an ardent advocate of spelling reform, and thought words should be spelled more like they sound.
Many years before he published his well-known American Dictionary of the English Language, he published a much smaller, more radical dictionary he called a Compendious Dictionary that included spellings such as w-i-m-m-e-n for "women" and t-u-n-g for "tongue." That dictionary was skewered and he dialed it back a lot on the spelling reform for his final masterpiece. Yet still, Noah Webster, his affection for spelling reform, and the success of his final dictionary in 1828, are the reasons Americans spell words such as "favor" without a "u" (1), "theater" with an "-er" instead of an "-re" at the end, spell "sulfur" with an "f" and not a "ph" in the middle, and why "aluminium" became "aluminum (2)."
A Separated Population
There are some word differences that we can't lay at Webster's feet. For example, "while" and "whilst" mean the same thing, but as far as I can tell, nobody really knows why "whilst" survived in Britain but not America. According to World Wide Words (3), "whilst" is considered to be more formal than "while," even in Britain. So if I had to guess, I'd say "whilst" probably fell out of favor in America because we are a less formal nation, and geographic separation of the two populations also let the language change differently in the two countries, but really, I'm just making things up at this point. If anyone has a better answer, please post it in the comments.
Why Do Britons and Americans Use Single and Double Quotation Marks Differently?
On to a difference where I at least have a hint of an answer!
In America we use double quotation marks to enclose a quotation, and single quotation marks if we need to enclose another quotation inside the first quotation. In British English, it's the opposite. Single quotation marks are used for everyday purposes such as enclosing a stand-alone quotation (4, 5).
In 1908, an influential British style guide called The King's English, states that "The prevailing [method] is to use double marks for most purposes, and single ones for quotations within quotations." So to spell it out for you, the author, Fowler, was saying that at the time the British did it the same way we do it now in America. But Fowler went on to advocate for single quotations marks, saying it is more logical to use them for regular quotations, and to reserve double quotation marks for quotations within quotations (6). He didn't explain why he thought it was more logical; he just said it was. Given that the British method now follows Fowler's stated preference, I presume that Fowler is the reason the British now use single quotation marks where Americans primarily use double quotation marks--that he was influential enough to make that change happen. But that one little paragraph from The King's English is the only proof I have, so if you know of some other reason Britons made the change, please leave a comment below.
Typesetters Quotations Versus Logical Quotations
There's another difference in how Americans and Britons treat quotation marks. In the U.S. we put periods and commas inside quotation marks, and in Britain they put periods and commas outside quotation marks. My admittedly U.S.-centric memory trick is to remember "Inside the U.S., inside quotation marks. Outside the U.S., outside quotation marks."
The reason for this difference begins with the introduction of movable type. Before typesetting, nobody paid too much attention to where they put periods and commas relative to quotation marks, but periods and commas became a problem with the advent of typesetting because they were so tiny. Printers found that the periods and commas were more stable when they were placed inside closing quotation marks, so that's the way they started doing it (7, 8).
Again, our British friend Fowler seems to have made the difference in his book The King's English. (9) Typesetting technology had advanced to the point where it wasn't necessary to shield periods and commas anymore, and he argued for what he considered a more logical system of letting the context of the sentence determine where the period and comma should go. The British seem to have taken his suggestion to heart and Americans seem to have ignored it.
Because of these origins, it is sometimes said the British use logical quotations and Americans use typesetters quotations.
Pronunciation Differences
Finally, you may be wondering why there are pronunciation differences between British and American speakers of English (not to mention Canadians, Australians, and others). The general idea is that regional and national pride and changing ideas about what sounded like "proper" speech, at least to some degree played a role in changing the British sounding speech of the American colonists to what we hear today in America. It's far too complex to cover here, so I'll refer you to a PBS show called "Do You Speak American?" which talks about regional dialects too (10).
Summary
So, in summary, American English is different from British English because of the revolutionary leanings of a dictionary writer (Noah Webster), typesetting conventions, geographical separation, and the opinion of one influential style guide author (H.W. Fowler).