Modern AI systems like GPT, Claude, Gemini, and Llama are built on three foundational concepts: tokens, embeddings, and transformers. These concepts work together to convert human language into mathematical representations that neural networks can process efficiently.
Why LLMs Cannot Read Raw Text
Computers do not understand words directly. Before a Large Language Model can process language, text must first be converted into numerical representations. This conversion pipeline starts with tokenization, continues through embeddings, and is processed through transformer neural networks.
Tokens: Splitting Language into Pieces
A token is the smallest unit an LLM processes. Tokens are not always full words. Depending on the tokenizer, they may represent syllables, punctuation, spaces, or partial words. For example, the word 'unbelievable' may split into multiple smaller tokens.
from transformers import AutoTokenizer
model_name = 'gpt2'
tokenizer = AutoTokenizer.from_pretrained(model_name)
text = 'Transformers changed AI forever.'
# Split text into tokens
tokens = tokenizer.tokenize(text)
# Convert tokens into numerical IDs
token_ids = tokenizer.encode(text)
print(tokens)
print(token_ids)Different AI models use different tokenizers. OpenAI models, Llama models, and Mistral models may split identical text differently. This directly affects pricing, latency, and context window usage because most AI APIs charge per token.
Context Windows & Token Limits
Every LLM has a context window, which defines how many tokens it can process simultaneously. Context includes system prompts, user messages, uploaded documents, tool outputs, and generated responses.
If the total token count exceeds the model limit, older information gets truncated or requests fail entirely. This is why production AI systems aggressively optimize prompts and chunk large documents before retrieval.
article = 'Very long document text...'
# Example token estimation
estimated_tokens = len(tokenizer.encode(article))
print(f'Total Tokens: {estimated_tokens}')Embeddings: Converting Meaning into Vectors
After tokenization, language is transformed into embeddings. Embeddings are high-dimensional numerical vectors that capture semantic meaning. Instead of storing only words, embeddings store relationships between concepts mathematically.
For example, embeddings allow models to understand that 'doctor' and 'physician' are semantically similar even though they are different words. This semantic understanding powers search systems, recommendations, clustering, and Retrieval-Augmented Generation (RAG).
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
sentences = [
'Artificial Intelligence is growing rapidly',
'Machine learning is transforming industries'
]
embeddings = model.encode(sentences)
print(embeddings.shape)Vector Similarity Search
Embeddings are useful because similar concepts generate vectors that are mathematically close together. Vector databases use similarity algorithms like cosine similarity to retrieve the most relevant information based on meaning rather than exact keywords.
This is the foundation of semantic search systems. Instead of searching for exact phrases, AI systems retrieve information based on conceptual similarity.
from sklearn.metrics.pairwise import cosine_similarity
similarity = cosine_similarity(
[embeddings[0]],
[embeddings[1]]
)
print(similarity)Transformers: The Brain of Modern LLMs
Transformers are deep learning architectures designed specifically for sequence understanding. Introduced in the famous paper 'Attention Is All You Need', transformers replaced older recurrent neural networks (RNNs) because they process text more efficiently and scale far better.
The key innovation of transformers is self-attention. Instead of processing words one-by-one sequentially, transformers examine relationships between all tokens simultaneously.
Self-Attention Mechanism
Self-attention allows the model to decide which previous words matter most when predicting the next token. This enables transformers to capture long-range dependencies and contextual meaning.
Consider the sentence: 'The trophy did not fit in the suitcase because it was too large.' Humans understand that 'it' refers to the trophy. Self-attention helps the model learn these relationships mathematically.
sentence = 'The trophy did not fit in the suitcase because it was too large.'
attention_map = {
'it': ['trophy', 'large']
}
print(attention_map)Positional Encoding
Transformers process tokens in parallel, which creates a challenge: the model must still understand word order. Positional encoding solves this by injecting positional information into token embeddings.
Without positional encoding, the sentences 'Dog bites man' and 'Man bites dog' would appear identical mathematically.
Training Transformers
During training, transformers repeatedly predict missing or future tokens across billions of text samples. The neural network adjusts its internal weights using backpropagation and gradient descent to minimize prediction errors.
Over time, the model learns grammar, reasoning patterns, programming syntax, world knowledge, and even subtle language structures entirely from statistical relationships.
input_text = 'AI is transforming'
possible_next_tokens = {
'industries': 0.74,
'everything': 0.18,
'quickly': 0.08
}
predicted = max(possible_next_tokens, key=possible_next_tokens.get)
print(predicted)Inference: Generating Responses
When users interact with ChatGPT or similar systems, the model performs inference. It generates one token at a time, continuously predicting the next most probable token based on the entire conversation context.
This iterative generation process creates surprisingly coherent responses despite the model fundamentally operating as a probability engine.
response = client.chat.completions.create(
model='gpt-4o-mini',
messages=[
{
'role': 'user',
'content': 'Explain transformers simply'
}
],
temperature=0.5
)
print(response.choices[0].message.content)Why This Matters for AI Engineers
Understanding tokens, embeddings, and transformers is essential for building production-grade AI systems. These concepts directly impact prompt engineering, RAG pipelines, latency optimization, token costs, chunking strategies, hallucination prevention, and model selection.
AI engineers who deeply understand these foundations can design systems that are faster, cheaper, more accurate, and significantly more reliable in real-world applications.