Letters with Accents

Emma's farm shop now sells jalapeños and pumpkin purée, and words like these need extra care. Here's why.

A computer stores a text as a row of bytes, small numbers from 0 to 255. The texts in your programs, like most texts in the world, use the UTF-8 encoding: every letter of the English alphabet, every digit, space and punctuation mark takes one byte, but a letter with an accent, such as é or ñ, takes two (and some characters, such as emoji, take four). Lua's string functions don't know about letters at all: they count and cut bytes. In a text without accents, one byte is one character, so you don't notice. With accents, you do, in four places.

1. Length. # and len count bytes. To count characters, use utf8.len from the library utf8, which Lua has for exactly this:

local word = "purée"
print(#word, word:len(), utf8.len(word))

This prints:

6	6	5

2. Capital and small letters. upper and lower change only the letters from a to z; letters with accents stay as they are:

print(("purée"):upper())
print(("CRÈME"):lower())

This prints:

PURéE
crÈme

When you need such a word in capital letters, write it that way in the code: "PURÉE".

3. Cutting. The positions in sub are bytes too. In "purée", the é takes the positions 4 and 5, so the first four letters are sub(1, 5), and sub(1, 4) cuts the é in half:

local word = "purée"
print(word:sub(1, 5))
print(word:sub(1, 4))
print(word:sub(1, 4) == "puré")

This prints:

puré
pur�
false

Half a letter isn't a character at all, so when it's printed, you see � in its place. The same goes for reverse, which swaps the two bytes of every letter with an accent: ("purée"):reverse() gives e��rup. Cutting is safe at positions that find gave you, because they're bytes too, and with negative positions when the end of the text has no accents.

4. Widths. %-10s fills a text up to 10 bytes, so a word gets one space too few for every letter with an accent:

print(("%-10s|"):format("carrots"))
print(("%-10s|"):format("purée"))

This prints:

carrots   |
purée    |

The fix is to add the spaces yourself: as many as the width minus the number of characters. A small function does it:

local function pad(text, width)
    return text .. (" "):rep(width - utf8.len(text))
end

print(pad("carrots", 10) .. "|")
print(pad("purée", 10) .. "|")

This prints:

carrots   |
purée     |

When a text is longer than the width, the count is negative, and rep gives an empty text, so nothing is cut off, just as with %-10s.

The patterns in the next exercise have the same problem with accents; you'll see it there. The rule for all of it: printing, joining and searching texts with accents is always fine. When you count, cut, align or change letters, think of the bytes.