Curling Quotes in HTML, SGML, and XML

Summary

If you’re creating HTML, SGML, and XML directly, perhaps using a text editor or writing a program, the safest approach is to use “decimal numeric character references” for curling single and double quote characters (these marks are called “smart quotes,” “curly quotes,” “curled quotes,” “curling quotes,” or “curved quotes”). In other words, for left and right double quotation marks, use “ and ” - and for left and right single quotation marks (and apostrophes), use ‘ and ’ - and you’ll be glad you did. This approach complies with all international standards, and works essentially everywhere.

Here’s a table showing what I mean.

To show	In HTML, SGML, or XML use	Displays on your system as
Left Double Quotation Mark	“	“
Right Double Quotation Mark	”	”
Left Single Quotation Mark	‘	‘
Right Single Quotation Mark (including English possessives and contractions)	’	’

By doing this, your text will look good on a very wide variety of browsers and viewers, and you can easily cut-and-paste portions of data between HTML, SGML, and XML documents (letting you dynamically query and create new material from existing material, without having to deal with the complexities of translating between character sets).

If you don't want to do this directly, use tools that will do it for you. If you're using simple ASCII text files, SmartyPants can do this for you.

The best alternative is using UTF-8. UTF-8 is fantastic, but other charsets are still in use and can cause problems.

Rationale

There are many advantages to this particular recommendation. These are the official, standard, vendor-neutral encodings for these characters according to both Unicode and ISO-10646, so you don’t need to worry about them not working in the future. They also work across XML, HTML, and SGML, simplifying data extraction - alternatives such as named character entity references do not easily work across XML and HTML (in particular). Systems which can display curling quotes (with the current fonts) will do so, and practically without exception will gracefully go back to neutral (vertical) characters if they can’t - even if they’re a somewhat old browser. I’ve tested this approach on several versions of Internet Explorer, Netscape (the old 4.5 and 6.X), Mozilla (0.9.9 and 1.0), and lynx (a text browser), on a variety of systems (Windows, Linux, Sun Solaris). The one minor problem is that on some older X windows systems with old fonts, the left single quotation mark may get mapped to a character that is an angled character for the right single quotation mark - but it doesn’t look bad, the alternatives look far worse everywhere else, and this solution is “future-proof”.

Do not use the various alternatives:

• Don’t use HTML’s character entity references assigned for this purpose: “, ”, ‘, and ’. Character entity references won’t work in SGML or XML in general, because they aren’t a predefined entity in SGML or XML (see the XML specification version 1.0 on predefined entitities for more information). They are predefined in modern HTML implementation, and you could define them in both SGML and XML, but this makes it harder to use data fragments - if you take parts of the material, the definitions probably won’t come along. Are you sure your information will never be used again? Indeed, one of the main points of XML is that you can manipulate the resulting data, and using these conveniences interferes with that process. Another problem is that they are not supported by older browsers (such as Netscape 4.5) and tools, and remember, it takes some users a long time to upgrade. Some older text browsers don’t support them - and text browsers are important for accessibility, because they’re the basis of most readers for the blind. It’s also easy to make mistakes with character entity references - earlier versions of this document incorrectly used “lsquot” instead of “lsquo” (note the excess letter t). If you’re sure you’ll never use the text in SGML or XML, you could consider using these symbols in a few years as browsers retire, but it’s not worth it. You’re probably much better off following the recommendation above; it will be easier to combine your data with other data (e.g., to create dynamic results) by following this recommendation.
• Don’t use HTML hexadecimal numeric character references, such as “, ”, ‘, and ’. Hexadecimal numeric character references are nice because the official documents that define the character standards also use hexadecimal. However, support for hexadecimal is a recent feature with inconsistent support: older browsers (like Netscape 4.5) don’t support it, and many other SGML and XML processors don’t support it. Indeed, SGML doesn’t include this ability at all. Since they're rarely used (compared to the decimal versions), there is also a higher risk of hitting a bug with them.
• Don’t embed the UTF-8 (or UTF-16) characters directly into the text and depend on setting the UTF-8 charset for now. For example, for right single quotation mark (including English possessives and contractions), use ’ and do do not use the UTF-8 encoding 11100010 10000000 10011001 (binary) aka e2 80 99 (hexadecimal) aka 342 200 231 (octal) aka 226 128 153 (decimal). UTF-8 is fantastic, but it not yet universal, and the problem is that other charsets are still in use. The UTF-8 encoding won’t work on some text browsers (e.g., lynx, and thus many blind readers that depend on text rendering). It’s possible in XML and HTML to specify that characters should be interpreted according to a particular character set (charset), but requiring a particular charset has many drawbacks. Setting the charset to utf-8 does work in many places, but only if you explicitly set the charset; failing to set the charset will cause this to fail on many systems. Fundamentally, this makes combining your material with other sources harder, because they’re likely to use other charsets. For example, Microsoft’s non-standard character sets (discussed next) interfere with it, so using the UTF-8 encoding can cause trouble when trying to combine with data from some Microsoft and MacOS tools in some circumstances. In the longer term, hopefuly everyone will switch to UTF-8 and UTF-16, and then this would a reasonable alternative. For now, don’t do it.
• Don’t use Microsoft’s non-standard character sets (such as positions 145 through 148 for the windows-1252 charset). Not only are they nonstandard, they conflict with the standards - the specification for HTML clearly forbids the use of character positions 128 through 159, because Unicode and ISO 10646 reserve positions 128 to 159 as control characters. Windows’ non-standard characters are not supported on all browsers and viewers. In fact, using them can cause trouble even if everyone only uses Microsoft’s products. Using them at all (even in browsers that support them) requires that you set the “charset” to a non-standard value (typically windows-1252) at the beginning of your document. If you set the charset to windows-1252, it works fairly widely, but doing this makes it more difficult to combine the resulting text with other material (because there’s no guarantee that your other material uses this charset - there are many other charsets in use, even if you only use Microsoft products). Thus, just as with the embedded utf-8 solution, this approach makes it harder to combine data with other sources - if the other source doesn’t use the same charset (very likely!), it will stop working. Many browsers do try to get these to work as long as this particular charset is set, since many Microsoft products generate them, but it’s touch-and-go... even with Microsoft’s own products. Oddly enough, specifying these Microsoft code positions using numeric identifiers often works on existing systems, but doing that is a complete botch to the standards (they’re officially forbidden), so it’s quite unsafe to depend on this. (the W3C specifically notes this.) Because there are multiple Microsoft character sets, doing this sort of thing causes complications in a Microsoft-only world - you can easily have two files, both using different Microsoft charsets, that can’t be easily combined. Larisa Thomason’s article in NetMechanic noted the same thing - that even in a Microsoft-only world these characters caused problems. And besides all that, Microsoft has already stated that it’s moving towards ISO 10646/Unicode, and it has already modified much of their software to do this, so even if you only use Microsoft products this isn’t where Microsoft itself is headed. Microsoft calls them “smart quotes” and people often complain about smart quotes - the problem is not that Microsoft automatically curls quotation marks by default (that’s okay), but that Microsoft uses these non-standard characters that interfere with standards to do so. So, this approach is proprietary, it requires you to set a nonstandard charset (which makes doing this reliably harder and interferes with merging text), and Microsoft has already declared that it’s moving toward ISO 10646 instead, so don’t do this.
• In fact, avoid the need for charsets if you are transmitting English documents. There are other charsets, such as Apple’s many charsets, but they cause no end of problems. Sticking with a simple ASCII-character-only file, and using decimal numeric character references for all non-ASCII characters, may take a little more space but it completely eliminates a vast number of problems. Getting a system to correctly determine the character set while it’s already trying to read the file is problematic, and there are a vast number of configuraton settings that can go wrong - as well as client systems that may not handle things correctly. UTF-8 at least means the same thing everywhere, so you have some hope if you really want to use a non-ASCII character set and use UTF-8.  But since not all documents use UTF-8, and other charsets (like the Windows charsets) interfere with them, sticking with the approach recommended here has the best chance of success in HTML, SGML, and XML.

Now, why is this a problem? Normal English uses matched pairs of curled single quotation marks and double quotation marks to indicate quotation. Unfortunately, the original designers of the ASCII character set didn’t define a standard method for identifying properly curved quotation marks, so computers have had problems with properly exchanging quotation marks ever since.

Other Sources of Information

Markus Kuhn’s “ASCII and Unicode Quotation Marks” describes the general problem well. He summarizes this way:

Please do not use the ASCII grave accent (0x60) as a left quotation mark together with the ASCII apostrophe (0x27) as the corresponding right quotation mark. Your text will otherwise appear rather strange with most modern fonts (e.g., on Windows and Mac systems). Only old X Window System fonts and some old video terminals show ASCII 0x60/0x27 as left and right quotation marks, while most modern systems follow the ISO and Unicode standards instead. If you can use only ASCII’s typewriter characters, then use the apostrophe character (0x27) as both the left and right quotation mark. If you can use Unicode characters, nice directional quotation marks are available in the form of characters U+2018 and U+2019.

Unfortunately, Kuhn doesn’t describe how to specifically deal with the problem in HTML, XML, and SGML, which is why I wrote this page.

If you’re curious, here are the text pages I used to examine the issue on a wide variety of machines:

• Test page, UTF-8 charset
• Test page, default charset
• Test page, Windows-1252 (proprietary and obsolete) charset

Note that this approach means that if you're trying to generate simple ASCII text from HTML, SGML, or XML, you will need to translate curved quotes into straight quotes. But this is true in general - if you start with a richer character set (such as HTML, SGML, or XML when using numeric character references) and have to move to a poorer character set, you should expect that some characters will need to be translated. There are many other characters you have to handle anyway, so this is a step you would would have to do anyway.

After I wrote this page, I found that others have come to the same conclusion (for the same reasons). For example, Peter K. Sheerin’s The Trouble with EM ‘n EN proposes the same solution, for many of the same reasons (although he doesn’t note the issues with SGML and XML, which I think are important too). He also discusses proper use of the em dash (—), which is used to indicate a sudden break in thought, the en dash, (–), which is used to indicate a range or connection between things, and the single prime (′), which is used to represent feet or minutes. Again, the solution is to use decimal numeric character references.

Note that the W3C recommends only using such escapes as an exception. They suggest using a Unicode-based encoding (UTF-8, UTF-16, or UTF-32), and for XML using UTF-8 or UTF-16. The problem is that we're still in a transition period where tools don't all handle them so well, and the recommendation made here will ALWAYS work (now and in the future).

If you want detailed specifications on some of this, here are a few pointers: here is the Microsoft Windows Codepage 1252 (Windows Latin 1), as well as the Microsoft Windows Codepage 1253 (there are many more). A summary of the PalmOS code page is available. Possibly more importantly, here are some mapping documents that show how to convert from some of these character encodings into Unicode/ISO 10646: Microsoft Windows 1252 to Unicode, MacOS Roman to Unicode, and here are the set of mappings from various encodings into Unicode/ISO 10646.

Tools

I make a open source software / free software (OSS/FS) tool, quoter, available which uses heuristics to try to fix quotation marks in HTML, XML, or SGML (it’s smart enough to leave quotes alone when used in tags). It’s free, so feel free to use it (it requires a Unix-like system or Cygwin on Windows).

The demoroniser program fixes many incompatible Microsoft punctuation marks so that they comply with standards, but unfortunately, the last version I’ve seen (published January 1998) only converts the Microsoft quotation marks into the straight ASCII quotation marks instead of implmenting the approach described here. The demoroniser results at least look better and are more interoperable than doing nothing, but they aren’t as good as the approach recommended here. This perhaps makes sense; in January 1998, there were still some old tools that did not handle quotation marks well, but at this time that is unnecessary. My quoter tool does a better job of translating quotation marks; you can use demoroniser after using quoter if you’d like to fix other characters.

Composer, the HTML editor in Mozilla and Netscape 6 (and later), will normally correctly edit files that include curled quotes defined this way. In other words, if the file has them, and you edit the file, they’ll be fine. However, if you set the Content-type value in the HTML file, be sure to use a setting such ascii or iso-8859-1. Here’s an example of the HTML codes you should set, if you choose to set the Content-type (often a good idea):

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">

Plucker, as of version 1.2, handles these quotes correctly.

MacOS X’s Cocoa supports curling quotes, both entering and displaying them, using the standard Unicode character values advocated here. However, users may not remember how to enter the curly quotes. Andrew C. Stone shows how to automatically add curly quotes to Cocoa’s Text system.

If you have existing text in one character set, particularly a non-standard one like Windows’, you can use one of many tools to convert it to something else. Unix-like systems such as GNU/Linux usually have iconv, which will let you convert between the character sets to a single uniform character set (iconv comes with the GNU C library). Changing everything using iconv to something standard (like utf-8), and then running a simple program to change all non-ASCII characters into decimal numeric character references, would be a very good way to turn random text in various character sets into a single, uniform result.

Feel free to see my home page or my paper Why OSS/FS? Look at the Numbers!.