Embedding
Definition
A learned dense vector representation that maps discrete data like words, tokens, or items into continuous numerical space, capturing semantic relationships and similarities.
Embeddings convert high-dimensional sparse data (like a vocabulary of 50,000 words) into dense low-dimensional vectors (typically 256-4096 dimensions) where similar items are close together. Word embeddings like Word2Vec and GloVe showed that vector arithmetic can capture analogies (e.g., "king" - "man" + "woman" ≈ "queen"). Modern transformer models learn contextual embeddings where the same word gets different vectors depending on context. Embeddings are essential for search (finding semantically similar documents), recommendation systems, and as the input layer for most neural networks. Vector databases like Pinecone and Weaviate specialize in storing and searching embeddings at scale, powering RAG systems and semantic search.
Related Terms
Transformer
A neural network architecture introduced in 2017 that uses self-attention mechanisms to process sequ...
Tokenizer
A component that splits text into smaller units called tokens (words, subwords, or characters) that ...
RAG (Retrieval-Augmented Generation)
A technique that enhances LLM responses by first retrieving relevant documents from an external know...