Encoder-Decoder
Definition
A neural network architecture with two components: an encoder that compresses input data into a compact representation, and a decoder that generates output from that representation.
The encoder-decoder architecture is fundamental to sequence-to-sequence tasks where the input and output have different lengths or formats. The encoder processes the input and produces a fixed or variable-length representation capturing its meaning. The decoder then generates the output step by step, conditioned on this representation. The original transformer architecture uses both an encoder and decoder. BERT uses only the encoder (good for understanding tasks), while GPT uses only the decoder (good for generation). T5 and BART use the full encoder-decoder architecture. Beyond language, encoder-decoder structures appear in image segmentation (U-Net), speech recognition, and machine translation systems.
Related Terms
Transformer
A neural network architecture introduced in 2017 that uses self-attention mechanisms to process sequ...
Attention Mechanism
A technique that allows neural networks to focus on the most relevant parts of the input when produc...
Embedding
A learned dense vector representation that maps discrete data like words, tokens, or items into cont...