IV.10 / Transformers and attention
Calculate a tiny attention mixture.
Use two value vectors, (1,0) and (0,1), with attention weights 0.8 and 0.2. Their weighted output is (0.8,0.2). Change the weights while keeping the values fixed, then change the values while keeping the weights fixed. This separates what is selected from what is carried by the selected representations.
