Bytes, hexadecimal and text

The letter A is stored as 65, or 01000001. In UTF-8 a Korean syllable such as U+D55C takes three bytes.

An everyday puzzle first. (This is a made-up example.) One student types the word Korean in a chat, six letters. A friend types the same word in Korean script, which takes three syllables. Which message is bigger? Probably not the one you would guess. Saved as UTF-8, the six English letters are 6 bytes. The three Korean syllables are 9 bytes, three each. RFC 3629, the standard that defines UTF-8, uses exactly this word as one of its examples and spells out its bytes: ED 95 9C EA B5 AD EC 96 B4. Two things explain the gap. You need a short way to write bytes, and you need the agreement that turns letters into numbers.

Two hex digits for every byte

Those pairs of letters and digits are hexadecimal, or hex. Long rows of 0s and 1s are hard to read, so programmers use this shorthand. Hex has sixteen digits: 0 to 9, then A to F for ten to fifteen. One hex digit covers exactly four bits, so a byte is always two hex digits. Split 01000001 into 0100 and 0001. Those are 4 and 1, so the byte is 0x41, where 0x only says "this is hex". In decimal it is 4 × 16 + 1 = 65.

You have probably met hex without noticing. Colours on web pages are often written as #RRGGBB, one byte each for red, green and blue light (MDN). #FF0000 is full red: 255 in the first byte, 0 in the other two. #FFFFFF is white. The largest byte, 11111111, is 0xFF, and 0xFF = 15 × 16 + 15 = 255.

A is 65 because everyone agreed it would be

A computer has no idea what a letter is. It stores a number, and everyone agrees in advance which number stands for which letter. The agreement used on early networks was ASCII, and RFC 20 set it out for the ARPANET in 1969. ASCII uses 7 bits, so it has 128 codes. Capital A is 65, B is 66, and so on up to Z at 90. Small a is 97. That is exactly 32 more than A, a difference of a single bit. Digits are codes too. The character 0 is 48, so the character "7" is stored as 55, not as the number 7.

Why a Korean syllable needs three bytes

128 codes are plenty for English and nowhere near enough for Korean, Arabic, Chinese or emoji. Unicode gives every character in the world's writing systems its own number, called a code point, up to U+10FFFF. UTF-8 (RFC 3629) is the most common way to turn those numbers into bytes, using one to four bytes per character. The 128 ASCII characters keep their old one-byte codes, so plain English text is identical in ASCII and UTF-8. The RFC's table puts every code point from U+0800 to U+FFFF in the three-byte row, and the Korean syllables sit in that range. An accented é takes two bytes, a Korean syllable such as U+D55C takes three, and an emoji such as 😀 takes four. The first byte even announces the length. ED is 11101101, and its three leading 1s say that this character is three bytes long.

Garbled text is the right bytes read with the wrong agreement

A plain text file is these bytes in a row and nothing else. The word Hello saved as UTF-8 is five bytes: 48 65 6C 6C 6F in hex, or 72, 101, 108, 108, 111 in decimal. The two l's are the same byte twice. Now open that file, or one in Korean, with the wrong agreement, say an old Korean or Japanese encoding instead of UTF-8. The program reads the very same bytes as different characters. Nothing in the file is broken, and opening it again as UTF-8 brings the text back.

Where the missing 34 GB went

Because 2 to the power 10 is 1024, close to 1000, people long used "kilobyte" for both. In December 1998 the IEC approved separate names, and NIST lists them: 1 kB is 1000 bytes but 1 KiB, a kibibyte, is 1024 bytes; 1 MB is 1,000,000 bytes but 1 MiB is 1,048,576 bytes. Drive makers count in powers of ten. A drive sold as 500 GB holds 500,000,000,000 bytes. An operating system that counts in powers of two divides by 1,073,741,824 and shows about 465.66 GiB. The roughly 34 that seem to be gone were never there to lose. The two numbers simply use different units. The gap grows with the unit: a KiB is 2.4% bigger than a kB, a GiB about 7.4% bigger than a GB.