What is Unicode? Defin­i­tion and ex­plan­a­tion

Unicode is an in­ter­na­tion­al standard for encoding, dis­play­ing, and pro­cessing text char­ac­ters from nearly all the world’s writing systems. Each character is assigned a unique code point, which can be stored in various character encodings like UTF-8 or UTF-16. This allows Unicode to provide con­sist­ent rep­res­ent­a­tion and pro­cessing of texts across different platforms and languages.

What is Unicode?

Unicode stands for ‘Universal Character Encoding’ and is a global standard for rep­res­ent­ing text char­ac­ters in binary form. It enables con­sist­ent storage, exchange, and pro­cessing of text across different digital systems and platforms.

Unicode is in­nov­at­ive in that it is not tied to the formats and encodings of a single alphabet of a par­tic­u­lar human language. Rather, Unicode was created with the aim of serving as a unified standard for rep­res­ent­ing all writing systems and char­ac­ters developed by humans.

Since the release of Unicode 1.0 at the end of 1991, the standard has fulfilled its purpose. Unicode is in­tern­ally used by browsers and operating systems as a unified format. With the release of version 16.0 by the Unicode Con­sor­ti­um in 2024, the Unicode Standard now en­com­passes a rep­er­toire of 154,998 char­ac­ters. The character set covered by the Unicode Standard is com­pletely identical to the ‘Universal Coded Character Set’ (UCS), which is in­ter­na­tion­ally stand­ard­ised as ISO/IEC 10646.

Technical basis for character encoding

First, it’s important to un­der­stand that all in­form­a­tion present in a digital system consists of endless chains of zeros and ones on a deeper level. This is also referred to as ‘binary rep­res­ent­a­tion’. The binary code is somewhat like an alphabet in itself. However, in binary code, there are only two ‘letters’: zeros and ones. Each position within a sequence of zeros and ones is called a ‘bit’.

The basic trick of digital in­form­a­tion tech­no­logy is to represent char­ac­ters from different alphabets as sequences of zeros and ones. This allows for encoding numbers and letters, as well as any other dis­tin­guish­able states. Usually, these are called ‘symbols’. The longer the sequence of zeros and ones for rep­res­ent­ing a single symbol, the more symbols can be depicted. With each added bit, the number of possible symbols doubles.

A concrete example: Imagine we have binary ‘words’ that are two bits long. This would allow us to encode four numbers:

If we add another bit to the beginning of the sequence, the number of possible bit-words doubles. These consist of the already known bit sequences, each preceded by a zero or one. Thus, we can encode eight numbers:

For sim­pli­city, we’ve shown the encoding of numbers as an example here. However, the same principle applies to digital systems for encoding letters or any other char­ac­ters and states. Here is a highly sim­pli­fied example of binary encoding of letters:

The graphic rep­res­ent­a­tion of a character is called a glyph. Depending on the font used, there are different glyphs for the same character, and even within a single font, there can be multiple vari­ations for a glyph. Think, for instance, of different weights, ligatures, italics, etc. Here is an expanded rep­res­ent­a­tion that includes the mapping from the character to the glyph:

Ter­min­o­logy of character encoding

Digital character encoding involves a range of specific terms and concepts. In everyday usage, some of these may be used in­ter­change­ably, but in technical contexts — es­pe­cially when working with Unicode — it’s important to dis­tin­guish them clearly. Below are key terms along with their defin­i­tions:

Overview of common character encodings

Before the advent of Unicode, there was a wide variety of specific encodings. The norm was to use a distinct encoding for each language or language family. This often led to display errors and data in­con­sist­en­cies. To counter this, character encodings were fre­quently modelled as backward-com­pat­ible supersets of an existing standard. The modern Unicode standard builds on the earlier ISO Latin-1 encoding, which in turn is based on the ASCII character code.

Structure of the Unicode Standard

The Unicode Standard defines char­ac­ters and cor­res­pond­ing code points for letters, syl­lab­ar­ies, ideograms, punc­tu­ation marks, special char­ac­ters, and numerals. It supports the Latin, Greek, Cyrillic, Arabic, Hebrew, and Thai alphabets. Ad­di­tion­ally, it includes Japanese (Katakana, Hiragana), Chinese, and Korean scripts (Hangul). There are also math­em­at­ic­al, com­mer­cial, and technical special char­ac­ters, as well as his­tor­ic­al control char­ac­ters for teletypes.

The char­ac­ters are compiled in a series of character tables. We provide an overview of the most common character tables here.

Writing systems of the Unicode Standard

Symbols and punc­tu­ation of the Unicode Standard

What is Unicode used for?

The Unicode Standard primarily serves as a universal found­a­tion for pro­cessing, storing, and ex­chan­ging text in any language. Most modern software com­pon­ents, such as libraries, protocols, databases, etc., that operate on text are based on Unicode. We il­lus­trate the range of possible uses with the following examples.

Operating systems

Unicode is the internal standard for text rep­res­ent­a­tion in most modern operating systems. Some operating systems, like Apple’s macOS, allow the use of Unicode char­ac­ters in file names.

Websites

The Unicode variant UTF-8 has become the standard for encoding HTML documents. As early as 2016, more than 80 percent of the world’s most visited websites used UTF-8 for storing and dis­play­ing their HTML documents. The Punycode standard has become es­tab­lished for using non-ASCII letters in domain names.

Pro­gram­ming languages

Many modern pro­gram­ming languages use Unicode as the basis for text pro­cessing. A recent de­vel­op­ment is the ability to use Unicode char­ac­ters for naming variables and functions. This is possible in ECMAScript/JavaS­cript, as il­lus­trated in the following code:

let ︎👍 = true;
let 👎 = false;
if (bool_var === ︎👎) {
 // …
}

Databases

The popular and widely used database MySQL supports the complete Unicode character set with the character encoding ‘utf8mb4’. In contrast, using the ‘utf8’ encoding results in the loss of char­ac­ters whose code points encompass more than three bytes.

Fonts

Fonts contain the glyphs used for the graphic rep­res­ent­a­tion of text. Due to the large number of char­ac­ters included in the Unicode Standard, there is no font that contains all char­ac­ters. Even the subset of the Basic Mul­ti­lin­gual Plane is covered com­pletely by only a few fonts. Here are a few examples:

How is Unicode used?

In many cases, users employ Unicode without ever being aware of it. Digital text is presented in most documents and ap­plic­a­tions as Unicode and can be freely copied, pasted, and edited by users. Sometimes, the end user may need to insert a specific Unicode character into text. There are various methods for doing this, which we will present below.

Special software keyboards

The use of special software keyboards is probably the most common method to insert Unicode char­ac­ters into text. Ubi­quit­ous on mobile devices, software keyboards allow for switching between languages and their re­spect­ive alphabets. The key layout changes, with all char­ac­ters ori­gin­at­ing from the Unicode rep­er­toire. These char­ac­ters can be mixed and combined freely in texts.

A good example of this is emojis: Emojis are regular Unicode char­ac­ters like letters, numbers, and special symbols. As with digital char­ac­ters, the rep­res­ent­a­tion of emojis is in­de­pend­ent of their internal modelling. Each operating system displays the same emoji slightly dif­fer­ently.

The useful software keyboards are not only found on mobile devices. They’re also available on desktops. They can be easily accessed in Windows, macOS, and many Linux dis­tri­bu­tions, dis­play­ing a different set of char­ac­ters depending on the selected language. Since the number of keys is limited, not all Unicode char­ac­ters are shown. Instead, there’s a language-specific selection of the most commonly used char­ac­ters.

Unicode character tables

Besides software keyboards, Unicode character tables are probably the most useful way to access Unicode char­ac­ters. Remember, a character set (‘Coded character set’) is the col­lec­tion of all char­ac­ters along with their cor­res­pond­ing unique code points. Such a structure lends itself to a table format, and indeed the Unicode Standard includes exactly such tables called Unicode Code Charts. From these tables, users can copy specific char­ac­ters to use elsewhere. Al­tern­at­ively, end users can read the cor­res­pond­ing code point, for example, to use it as a numeric character reference—more on this in the next section.

Many desktop operating systems also include a Unicode character table. This provides an overview of all available Unicode char­ac­ters along with their code point, de­scrip­tion, and glyph. A character can be inserted or copied with a click. A character table can also be created with just a few lines of code. Later in this article, we’ll show an example using the Python pro­gram­ming language.

Numeric character reference

The core of the Unicode Standard is the mapping of char­ac­ters to code points. Knowing a character’s code point allows it to be used to embed the cor­res­pond­ing character in various contexts. On Windows, entering Unicode symbols is done using the standard hardware keyboard with a special key com­bin­a­tion. Note that the code point number typically needs to be entered in hexa­decim­al format.

Pro­gram­mers most often need numeric character ref­er­ences. The hexa­decim­al rep­res­ent­a­tion of code points allows for the mapping of a Unicode character into char­ac­ters of the ASCII character set. We demon­strate this approach in HTML; fun­da­ment­ally, it works the same in Python, C++, etc.

The general scheme for embedding a character using a numeric reference includes the reference itself, as well as an opening and closing term: In HTML documents, the numeric reference starts with &#x and ends with ;. In between, without any spaces, the two- to four-digit hexa­decim­al code point is entered, resulting in the pattern &#xNNNN;.

To insert the copyright symbol ‘©’ into an HTML document by example, we proceed with the following scheme:

1. Search for the character in a Unicode table.

2. Read the code point as­so­ci­ated with the character. In our example, the code point is listed as ‘U+00A9’, which is the hexa­decim­al rep­res­ent­a­tion.

3. Compose the character reference and enter it into HTML source code or a Markdown document. In our case, we input ©; this renders the character ‘©’.

A less common approach allows for the use of code points in decimal rather than hexa­decim­al rep­res­ent­a­tion. In this case, the numeric reference begins with &# (without the ‘x’) and ends as usual with ;. In between, the code point is written in decimal form. In our example, the numeric reference © results in the copyright symbol.

Named character entities

Since the notation of Unicode char­ac­ters as numeric ref­er­ences is not intuitive for humans, there is another method: named character entities. These are defined for commonly used char­ac­ters and assign a short, memorable name to the character. A named character entity starts with the ampersand & and ends with a semicolon ;. The defined name is placed in between without spaces. To insert the copyright symbol ‘©’ in HTML, simply write ©.

Pro­gram­ming languages

Most pro­gram­ming languages include basic functions to convert char­ac­ters and code points. The cor­res­pond­ing functions are often called ord(character) and chr(code point). The following applies:

chr(ord(character)) == character

Note that it is always possible to determine the code point cor­res­pond­ing to a character. Con­versely, the as­sign­ment only works for numbers that are actually defined as code points of the character code. We demon­strate the basic scheme here with a short Python example:

# Determine the decimal code point of a character
ord('A') # `65`
# Determine the hexadecimal code point of a character
hex(ord('A')) # `0x41`
# Determine the character corresponding to a code point
chr(65) # `'A'`
chr(0x41) # `'A'`
chr(0x110001) # Error, because code point > `0x110000`

With the help of these functions, it’s easy to create a character table for code points of the Unicode character set. For this, you iterate the code points and output the cor­res­pond­ing char­ac­ters. With Python, this can be done in just a few lines of code:

# Start `range` at `32` to avoid control characters being printed
# Print ASCII character set
for code_point in range(32, 128):
	print(code_point, hex(code_point), chr(code_point))
# Print ISO Latin-1
for code_point in range(32, 256):
	print(code_point, hex(code_point), chr(code_point))

Program Library ICU

The In­ter­na­tion­al Com­pon­ents for Unicode (ICU) are con­sol­id­ated in a program library provided by the Unicode Con­sor­ti­um. The library is released under an open-source license and can be used on many operating systems. The software serves the purpose of pro­gram­mat­ic In­ter­na­tion­al­isa­tion (often ab­bre­vi­ated as ‘i18n’). Its ap­plic­a­tions include:

• Pro­cessing of Unicode texts
• Support for regular ex­pres­sions in Unicode
• Parsing and format­ting of calendar dates, times, numbers, cur­ren­cies, and messages

The ICU library is available in two versions:

• ‘icu4c’ is written in C/C++ and provides an API for these languages.
• ‘icu4j’ is written in Java and provides an API for this language.

The use of the com­pon­ents provides con­sist­ent results re­gard­less of the un­der­ly­ing platform.

Charset meta tag in the head of HTML documents

Most HTML documents today use the UTF-8 character encoding. To ensure that visitors see the document without erroneous char­ac­ters, a ‘Charset’ meta tag should be placed in the head of the HTML document. This instructs the browser to interpret the retrieved document as UTF-8 and is il­lus­trated below:

<head>
<meta charset="utf-8">
<!-- additional head elements -->
</head>

Instagram fonts

The popular social network Instagram does not allow text format­ting for biography in­form­a­tion, posts, or stories. This limits users’ creative options. However, clever de­velopers have found a work­around: Instagram uses Unicode, making it possible to compose text that appears formatted using special char­ac­ters. This often involves char­ac­ters that resemble Latin letters. The easiest way to create such text is with an Insta Fonts Generator. Ad­di­tion­ally, using Instagram fonts also works in other social networks.