Natural language processing (NLP) is a subfield of artificial intelligence that allows computers to understand, process, and manipulate human language.
NLP has its origin ins the 1950s. Two important figures in the development of computational NLP.
Do you think LLMs are able to pass the Turing Test?
The NLP research of the 1950’s focused on rules-based systems. Linguists would craft large sets of rules to capture the grammatical syntax and vocabulary of specific languages.
Though important at the time, these systems were quite limited. They weren’t able to capture the nuances and many exceptions in languages. They were poor at capturing slang.
The creation and maintenance of such rule based systems for every language was not scalable.
These systems focused predominantly on syntax and vocabulary. They were not able to capture the deeper meaning and context of the texts they were analyzing.
See for example WordNet, (Wikpedia:WordNet).
Due to the limitations or rule-based systems coupled with a steady increase in computational power and large collections of data, this period of time saw the advent of machine learning algorithms applied to language processing.
In contrast to rule-based systems, these statistical models learned patterns from data. This allowed them to better handle the variations and complexities of natural language. IBM research developed a set of important machine translation models, called the IBM alignment models.
The concepts of Recurrent Neural Networks were introduced in 1986 in the paper, “Learning Representations by Back-Propagating Errors,” by (Rumelhart, Hinton, and Williams 1986).
In addition, the use of n-gram models became more formalized and widely adopted. An n-gram language model predicts the probability of a word based on the previous n-1 words in a sequence, making it a fundamental tool in natural language processing for tasks like text prediction and speech recognition.
The 2000s ushered in the era of deep learning. This is when we saw the application of neural networks to NLP.
Here are some of the notable developments.
For the rest of this lecture we will cover:
Machine learning models for NLP are not able to process text in the form of characters and strings. Characters and strings must be converted to numbers in order to train our language models.
There are a number of ways to do this. These include
However, prior to creating a numerical representation of text, we need to tokenize the text.
Tokenization is the process of splitting raw text into smaller pieces, called (drum-roll please), tokens. Tokens can be individual characters, words, or sentences.
Examples of character and word tokenization are shown for the following raw text
Show me the money
Character tokenization:
['S', 'h', 'o', 'w', 'm', 'e', 't', 'h', 'e', 'm', 'o', 'n', 'e', 'y'].
Word tokenization:
['Show', 'me', 'the', 'money']
This code block demonstrates both of these tokenization techniques.
# Character and word tokenization
sentence = "Show me the money"
word_tokens = sentence.split()
print(word_tokens)
character_tokens = [char for char in sentence if char != ' ']
print(character_tokens)['Show', 'me', 'the', 'money']
['S', 'h', 'o', 'w', 'm', 'e', 't', 'h', 'e', 'm', 'o', 'n', 'e', 'y']There are advantages and disadvantages to different tokenization methods. We showed two very simple strategies.
However, there are other strategies, such as subword and sentence tokenization, see for example Byte-Pair Encoding, and SentencePiece.
With tokenization, our goal is to not lose meaning with the tokens. With character based tokenization, especially for English (non-character based languages) we certainly lose meaning.
Here is a demo of how to tokenize using the transformers package from Huggingface.
from transformers import AutoTokenizer, logging
logging.set_verbosity_warning()
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
tokens = tokenizer.tokenize(sentence)
print(tokens)
# Try a more advanced sentence
sentence2 = "Let's try to see if we can get this transformer to tokenize."
tokens2 = tokenizer.tokenize(sentence2)
print(tokens2)['Show', 'me', 'the', 'money']
['Let', "'", 's', 'try', 'to', 'see', 'if', 'we', 'can', 'get', 'this', 'transform', '##er', 'to', 'token', '##ize', '.']Associated to each token is a unique token ID. The total number of unique tokens that a model can recognize and process is the vocabulary size. The vocabulary is the collection of all the unique tokens.
The tokens (and token ids) alone hold no (semantic) information. What is needed is a numerical representation that encodes this information.
There are different ways to achieve this. One encoding technique that we already considered is one-hot encodings. Another more powerful encoding method, is the creation of word embeddings.
We have previously considered the following sparse representations of textual data.
Example
Given the words cat, dog, and emu here are sample one-hot encodings
\[ \begin{align*} \text{cat} &= [1, 0, 0]^{T}, \\ \text{dog} &= [0, 1, 0]^{T}, \\ \text{emu} &= [0, 0, 1]^{T}. \\ \end{align*} \]
Example
Suppose we have the following sentences
| Sentence | the | cat | sat | on | mat | dog | log | emu |
|---|---|---|---|---|---|---|---|---|
| “The cat sat on the mat.” | 2 | 1 | 1 | 1 | 1 | 0 | 0 | 0 |
| “The dog sat on the log.” | 2 | 0 | 1 | 1 | 0 | 1 | 1 | 0 |
| “The emu sat on the mat.” | 2 | 0 | 1 | 1 | 1 | 0 | 0 | 1 |
Example
The TF-IDF representations corresponding to the previous sentences.
| cat | dog | log | mat | on | emu | sat | the | |
|---|---|---|---|---|---|---|---|---|
| Sentence 1 | 0.4698 | 0.0000 | 0.0000 | 0.4698 | 0.3546 | 0.0000 | 0.3546 | 0.7093 |
| Sentence 2 | 0.0000 | 0.4698 | 0.4698 | 0.0000 | 0.3546 | 0.0000 | 0.3546 | 0.7093 |
| Sentence 3 | 0.0000 | 0.0000 | 0.0000 | 0.4698 | 0.3546 | 0.4698 | 0.3546 | 0.7093 |
Word embeddings represent words as dense vectors in high-dimensional spaces.
The individual values of the vector may be difficult to interpret, but the overall pattern is that words with similar meanings are close to each other, in the sense that their vectors have small angles with each other.
The similarity of two word embeddings is the cosine of the angle between the two vectors. Recall that for two vectors \(v_1, v_2\in\mathbb{R}^{n}\), the formula for the cosine of the angle between them is
\[ \cos{(\theta)} = \frac{v_1 \cdot v_2}{\Vert v_1 \Vert_2 \Vert v_2 \Vert_2}. \]
Word embeddings can be static or contextual. A static embedding is when each word has a single embedding, e.g., Word2Vec. A contextual embedding (used by more complex language model embedding algorithms) allows the embedding for a word to change depending on its context in a sentence.
A language model is a statistical tool that predicts the probability of a sequence of words. It helps in understanding and generating human language by learning patterns and structures from large text corpora.
We previously covered RNNs and LSTMs.
We’ll discuss N-grams briefly followed by a deep dive on Transformers.
Transformers are a deep learning model for processing sequential (text) data Vaswani et al. (2017).
To introduce the Transformer architecture we will consider machine translation. This is an application of sequence-to-sequence modeling.
The encoder and decoder work together to transform input sequences into meaningful output sequences.
The input to the encoder are word embeddings.
To establish the order of the words, we use positional encodings.
The authors of Vaswani et al. (2017) proposed the following function for positional encodings \[ \mathbf{t}_{j}^{(i)} = \begin{cases} \sin{\left(\frac{i}{10000^{2j/n}}\right)} & \text{for~$j$~even}, \\ \cos{\left(\frac{i}{10000^{2j/n}}\right)} & \text{for~$j$~odd}. \\ \end{cases} \]
Here is sample code to generate an illustration of the positional encodings for embedding dimension \(n=64\) and \(m=10\) tokens.
import numpy as np
import matplotlib.pyplot as plt
def get_angles(pos, i, d_model):
angle_rates = 1 / np.power(10000, (2 * (i//2)) / np.float32(d_model))
return pos * angle_rates
def positional_encoding(position, d_model):
angle_rads = get_angles(np.arange(position)[:, np.newaxis],
np.arange(d_model)[np.newaxis, :],
d_model)
# apply sin to even indices in the array; 2i
angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2])
# apply cos to odd indices in the array; 2i+1
angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2])
pos_encoding = angle_rads[np.newaxis, ...]
return pos_encoding
tokens = 10
dimensions = 64
pos_encoding = positional_encoding(tokens, dimensions)
plt.figure(figsize=(7, 5))
plt.pcolormesh(pos_encoding[0], cmap='viridis')
plt.xlabel('Embedding Dimensions', fontsize=16)
plt.xlim((0, dimensions))
plt.ylim((tokens, 0))
plt.ylabel('Token Position', fontsize=16)
plt.colorbar()
plt.show()
The positionally encoded vectors are transposed and stacked to form an input matrix. This input matrix is fed into the bottom of the encoder block.
The number of rows \(m\) is the number of words in the sentence an the number of columns \(n\) is the embedding dimension. \[ X = \begin{bmatrix} & & {\mathbf{x}^{(1)}}^{T} & & \\ & & {\mathbf{x}^{(2)}}^{T} & &\\ & & \vdots & & \\ & & {\mathbf{x}^{(m)}}^{T} & & \\ \end{bmatrix} \in\mathbb{R}^{m\times n}. \]
Attention in language models is a mechanism that allows the model to focus on relevant parts of the input sequence by assigning different weights to different words, enabling it to capture long-range dependencies and context more effectively.
Attention is needed in order to understand sentences such as,
The elephant didn't cross the river because it was tired.Attention allows a language model to understand correctly that it refers to the elephant and not the river.
The building blocks of the attention mechanism are query \(\mathbf{q}\), key \(\mathbf{k}\), and value \(\mathbf{v}\) vectors.
The query and key vectors \(\mathbf{k}, \mathbf{q}\in\mathbb{R}^{d}\) and the value vector \(\mathbf{v}\in\mathbb{R}^{n}\).
At a high level
Each \(\mathbf{q}^{(i)}\), \(\mathbf{k}^{(i)}\), and \(\mathbf{v}^{(i)}\) is associated with \(\mathbf{x}^{(i)}\). To compute the query, key, and value vectors, compute \[ \begin{align*} \mathbf{q}^{(i)} &= {\mathbf{x}^{(i)}}^{T}W^{Q}, \\ \mathbf{k}^{(i)} &= {\mathbf{x}^{(i)}}^{T}W^{K}, \\ \mathbf{v}^{(i)} &= {\mathbf{x}^{(i)}}^{T}W^{V}, \\ \end{align*} \]
where \(W^{Q}, W^{K}\in\mathbb{R}^{n\times d}\) and \(W^{V}\in\mathbb{R}^{n\times n}\) are trainable matrices of weights.
This operation can be vectorized to compute \[ \begin{align*} Q &= XW^{Q}, \\ K &= XW^{K}, \\ V &= XW^{V}. \\ \end{align*} \]
Given a query, key, and value vector, we compute attention in the following sequence of operations
Let \(\mathbf{s}\in\mathbb{R}^{n}\), recall \(softmax(\mathbf{s})_i = \frac{e^{s_i}}{\sum e^{s_{j}}}\).
Attention can be easily vectorized. The procedure is then
Recall the original sentence to be translated, “The elephant did not cross the river because it was tired.”
A hypothetical visualization would demonstrate the links of the original sentence to the word it.
The darker the color, the more this word attends to the word it.
Proposed in Ba, Kiros, and Hinton (2016).
The architecture of a decoder block (in an encoder-decoder transformer) is nearly identical to than of an encoder block.
The decoder blocks consists of
The major differences in the decoder
The final output of the decoder blocks is a linear layer followed by a softmax layer.
The linear layer outputs a vector that is fed into a softmax layer.
The softmax layer outputs the probability of a potential word being generated.
The word with the maximum probability is chosen and output by the model.
Encoder-Decoder – used in sequence-to-sequence tasks, where one text string is converted to another (e.g., machine translation)
Encoder – transforms text embeddings into representations that support variety of tasks (e.g., sentiment analysis, classification) Model Example: BERT
Decoder – predicts the next token to continue the input text (e.g., ChatGPT, AI assistants) Model Example: GPT4, GPT4
Proposed in Devlin et al. (2019).
Proposed in Brown et al. (2020).
In this course we introduced you to the Tools for Data Science.
We covered:
Please be sure to fill out the course evaluations. They can be found at the following links:
