The Linear Representation Hypothesis and the Geometry of LLMs

Published by

yjchoe.github.io

on •1 min read

The Linear Representation Hypothesis and the Geometry of LLMs

Informally, the “linear representation hypothesis” is the idea that high-level concepts are represented linearly as directions in some representation space. In this paper, we address two closely related questions: What does “linear representation” actually mean?

Linear Representations Representation Geometry Softmax Geometry Causal Inner Product Large Language Models Gemma Llama

Learn more