J'Log

Pre-Norm vs. Post-Norm: The Identity Path That Stabilizes LLMs

Sun, 16 Aug 2026 00:00:00 +0000

The central idea: Pre-Norm matters because every Transformer block retains a residual route that normalization does not transform. That direct identity path lets gradients cross many layers. It mainly improves the optimization stability of deep models; it does not imply that Pre-Norm always has greater final expressive power.

The one-line difference

Let $F_l$ denote either attention or the feed-forward network, and let $N$ denote LayerNorm or RMSNorm. The two arrangements differ by one line:

Pre-Norm

\[x_{l+1}=x_l+F_l(N(x_l))\]

Post-Norm

\[x_{l+1}=N(x_l+F_l(x_l))\]

In Pre-Norm, only the branch computation sees normalized input. The residual stream $x_l$ reaches the next layer unchanged. In Post-Norm, the residual is still present, but the sum must pass through normalization before becoming $x_{l+1}$.

This diagram is the argument in its most useful form: Pre-Norm has an unmodified blue highway through depth. Post-Norm has a residual connection, but no unmodified identity highway.

Why the identity path changes backpropagation

Differentiate one block. Pre-Norm gives

\[\frac{\partial x_{l+1}}{\partial x_l} =I+J_{F_l}J_N.\]

The $I$ term is explicit. When the learned branch is small, as it often is near initialization, the block Jacobian is close to identity:

\[\frac{\partial x_{l+1}}{\partial x_l}\approx I.\]

A gradient can therefore travel through dozens of blocks without depending entirely on the attention, MLP, or normalization Jacobians.

Post-Norm instead gives

\[\frac{\partial x_{l+1}}{\partial x_l} =J_N(I+J_{F_l}).\]

Every backward route includes $J_N$. Across a deep stack, the gradient repeatedly encounters those normalization Jacobians. Their products can rescale and rotate it, and LayerNorm projects out particular shift and scale directions. The residual addition has not disappeared; what disappeared is the untransformed identity route.

That distinction is more precise than saying “Post-Norm cuts the residual connection.” It does not. It makes normalization unavoidable on the path between successive residual states.

What Xiong et al. actually established

Xiong et al. (2020), On Layer Normalization in the Transformer Architecture, analyzed the models at initialization using mean-field arguments. Their key observation was a layerwise imbalance:

In Post-LN, parameter gradients near the output are large at initialization. Applying the full target learning rate immediately can make the first updates unstable.
In Pre-LN, gradient magnitudes are better behaved across depth, so their translation and BERT experiments could train without learning-rate warmup.

This explains why learning-rate warmup is especially important for classical Post-Norm Transformers: warmup makes the early updates small while the optimizer and network leave the fragile initialization regime.

Two qualifications matter:

This is an analysis of initialization under simplifying assumptions, not a theorem that every Post-Norm run diverges.
Modern Pre-Norm LLM training still commonly uses warmup. Warmup also helps adaptive-optimizer state, large batches, mixed precision, and other parts of the training system.

The defensible claim is therefore “Pre-Norm is less sensitive and easier to optimize deeply,” not “Pre-Norm never needs warmup.”

A runnable stability stress test

The code accompanying this article trains the same small 12-layer causal Transformer three ways:

Pre-Norm at a constant learning rate of $2\times10^{-3}$;
Post-Norm at the same constant learning rate;
Post-Norm with a linear warmup lasting two thirds of the run (120 steps in the default 180-step experiment) to the same target rate.

Everything else—data, initialization seed, optimizer, depth, width, and number of updates—is held fixed. The task is deliberately simple: given an arithmetic token sequence, predict the next token. There is no gradient clipping.

On my CPU run, the results were:

Configuration	Final loss	Maximum global gradient norm
Pre-Norm, no warmup	0.00165	1.004
Post-Norm, no warmup	4.21205	6.960
Post-Norm, 120-step warmup	0.01738	2.759

The result is intentionally not “Post-Norm is incapable of learning.” Warmup rescues it. The result is that Pre-Norm tolerates the aggressive target learning rate immediately, while this particular Post-Norm run does not.

This is a pedagogical stress test, not a reproduction of Xiong et al.: the dataset is synthetic, the model is tiny, and a different seed or hyperparameter can move the stability boundary. The useful exercise is to sweep --layers and the learning rate and observe how much tuning each topology needs.

Run it with JAX on CPU or GPU

The complete script lives beside this article:

From the repository root:

# CPU, including Apple Silicon. The full run takes about 26 seconds here.
python -m pip install --upgrade jax flax optax matplotlib pillow
python _posts/pre_norm_vs_post_norm_demo.py --device cpu

# NVIDIA GPU on Linux with JAX's CUDA 13 wheel.
python -m pip install --upgrade "jax[cuda13]" flax optax matplotlib pillow
python _posts/pre_norm_vs_post_norm_demo.py --device gpu

# Prefer a JAX GPU backend when present, otherwise use CPU.
python _posts/pre_norm_vs_post_norm_demo.py --device auto

The script writes both img/pre-vs-post-norm-stability.webp and the underlying CSV. It is JAX end to end: the model is written with Flax, gradients come from jax.value_and_grad, updates come from Optax, and the training step is compiled with jax.jit. On macOS, the supported default is Apple Silicon CPU; the JAX project does not currently ship an official macOS GPU backend. The architectural switch itself is only this:

if self.pre_norm:
    x = x + attention(attention_norm(x))
    return x + mlp(mlp_norm(x))

x = attention_norm(x + attention(x))
return mlp_norm(x + mlp(x))

The compiled update is likewise ordinary JAX:

loss, gradients = jax.value_and_grad(loss_fn)(params)
updates, optimizer_state = optimizer.update(gradients, optimizer_state, params)
params = optax.apply_updates(params, updates)

The full default run is already small enough for CPU use. For a faster smoke test, reduce the work:

python _posts/pre_norm_vs_post_norm_demo.py \
  --device cpu --layers 6 --steps 30

That shorter setting checks that the program and selected backend work; it is not guaranteed to reproduce the full stability separation.

Why Pre-Norm is not simply “better”

Pre-Norm buys optimization stability by leaving the residual stream itself unconstrained inside the stack. Expanding the recurrence shows the trade-off:

\[x_L=x_0+\sum_{l=0}^{L-1}F_l(N(x_l)).\]

Every layer writes directly into the same accumulating stream. As $\lVert x_l\rVert$ grows, a new update can become a smaller fraction of the existing state. Later blocks can approach identity transformations and contribute less than their depth suggests. Later work describes related symptoms as representation collapse, residual-stream growth, massive activations, or the “curse of depth.” The exact growth law depends on initialization, training, correlations, and residual scaling; it is not universally exponential.

Post-Norm has the opposite appeal: it re-normalizes the state after every residual addition, tightly controlling forward scale. Its cost is making normalization unavoidable in backpropagation. That is why several newer architectures try to keep the best property of each design rather than treating either as universally optimal:

Design	What it tries to preserve
Pre-Norm	Clean identity path and robust optimization
Post-Norm	Controlled block-output scale
DeepNorm / residual scaling	Post-Norm quality with bounded updates
NormFormer	More balanced layerwise gradients in Pre-Norm
Peri-LN / sandwich norms	Identity residual path plus bounded sublayer output
Final norm	A stable scale before the language-model head

The final norm has a different job

A typical modern Pre-Norm LLM also applies a final normalization:

\[z=N_{\mathrm{final}}(x_L),\qquad \mathrm{logits}=W_{\mathrm{vocab}}z.\]

This does not undo the identity-path benefit. The block-level residual stream remains clean throughout depth; normalization is applied once before the output head.

Ignoring epsilon and affine parameters, LayerNorm and RMSNorm are approximately zero-homogeneous:

\[N(c x)=N(x),\qquad c>0.\]

The final norm therefore acts as a scale anchor: simply inflating $\lVert x_L\rVert$ cannot inflate the logits. This is separate from the role of the internal Pre-Norms:

Internal Pre-Norm: stabilize each branch input and preserve the identity gradient path.
Final norm: stabilize the scale presented to the language-model head.

The design rule behind the literature

The Pre-Norm versus Post-Norm literature can be condensed into two simultaneous requirements:

Keep a clean residual identity path so gradients can cross depth.
Control the size of updates written into that path so the residual stream does not overwhelm later layers.

Plain Pre-Norm solves the first requirement extremely well, which is why it became the default for deep LLM optimization. It only partially solves the second. DeepNorm, NormFormer, Peri-LN, residual scaling, and related designs are attempts to add scale control without losing the identity highway.

Takeaway

Pre-Norm is important because it changes the optimization geometry of a deep Transformer. In

\[x_{l+1}=x_l+F_l(N(x_l)),\]

the derivative always contains an identity term. That gives the gradient a direct route across depth and makes large models less sensitive to their earliest updates.

The benefit is primarily trainability, not a guarantee of superior final representation. Post-Norm can work—and may sometimes produce stronger representations—when warmup, initialization, or residual scaling controls its optimization. The modern goal is not merely “put normalization first,” but:

Preserve the identity path, then control what each learned branch writes into it.

References

核心观点： Pre-Norm 之所以重要，是因为每个 Transformer block 都保留了一条归一化不会变换的 residual 通路。这条直接的 identity 路径让梯度能穿越许多层。它主要改善深度模型的优化稳定性；这并不意味着 Pre-Norm 总有更强的最终表达能力。

一行之差

令 $F_l$ 表示 attention 或 feed-forward network，令 $N$ 表示 LayerNorm 或 RMSNorm。两种安排只差一行：

Pre-Norm

\[x_{l+1}=x_l+F_l(N(x_l))\]

Post-Norm

\[x_{l+1}=N(x_l+F_l(x_l))\]

在 Pre-Norm 里，只有分支计算看到归一化后的输入。residual stream $x_l$ 原封不动到达下一层。在 Post-Norm 里，residual 仍然在，但求和必须先经过归一化才能成为 $x_{l+1}$。

这张图是论证最有用的形式：Pre-Norm 有一条贯穿深度、未经修改的蓝色高速路。Post-Norm 有 residual 连接，但没有未经修改的 identity 高速路。

为什么 identity 路径会改变反向传播

对一个 block 求导。Pre-Norm 给出

\[\frac{\partial x_{l+1}}{\partial x_l} =I+J_{F_l}J_N.\]

$I$ 项是显式的。当学到的分支较小时——初始化附近常常如此——block Jacobian 接近 identity：

\[\frac{\partial x_{l+1}}{\partial x_l}\approx I.\]

因此梯度可以穿越数十个 block，而不必完全依赖 attention、MLP 或归一化的 Jacobian。

Post-Norm 则给出

\[\frac{\partial x_{l+1}}{\partial x_l} =J_N(I+J_{F_l}).\]

每条反向路径都包含 $J_N$。在深层堆叠里，梯度反复撞上这些归一化 Jacobian。它们的乘积会对其重新缩放和旋转，LayerNorm 还会投影掉特定的平移和缩放方向。residual 加法并没有消失；消失的是未经变换的 identity 通路。

这个区分比说“Post-Norm 切断了 residual 连接”更精确。它没有切断。它只是让归一化在相继 residual 状态之间的路径上变得不可避免。

Xiong et al. 实际建立了什么

Xiong et al. (2020), On Layer Normalization in the Transformer Architecture 用平均场论证分析了初始化时的模型。他们的关键观察是逐层不平衡：

在 Post-LN 中，靠近输出的参数梯度在初始化时很大。立刻用满目标学习率，最初几次更新会不稳定。
在 Pre-LN 中，梯度幅度在深度上更规矩，所以他们的翻译和 BERT 实验可以在没有 learning-rate warmup 的情况下训练。

这解释了为什么 learning-rate warmup 对经典 Post-Norm Transformer 特别重要：warmup 让早期更新保持较小，同时 optimizer 和网络离开脆弱的初始化区间。

有两点限定很重要：

这是在简化假设下对初始化的分析，不是每条 Post-Norm 运行都会发散的定理。
现代 Pre-Norm LLM 训练仍然普遍使用 warmup。Warmup 也有助于 adaptive-optimizer 状态、大 batch、mixed precision 以及训练系统的其他部分。

因此站得住脚的说法是 “Pre-Norm 更不敏感、更容易深度优化”，而不是“Pre-Norm 从不需要 warmup”。

一个可运行的稳定性压力测试

本文配套代码用三种方式训练同一个小型 12 层 causal Transformer：

Pre-Norm，恒定学习率 $2\times10^{-3}$；
Post-Norm，相同的恒定学习率；
Post-Norm，线性 warmup 持续整个 run 的三分之二（默认 180-step 实验中为 120 steps），到达相同的目标学习率。

其他一切——数据、初始化 seed、optimizer、深度、宽度和更新次数——都固定。任务故意很简单：给定一个算术 token 序列，预测下一个 token。没有 gradient clipping。

在我的 CPU 运行中，结果是：

配置	最终 loss	最大全局 gradient norm
Pre-Norm，无 warmup	0.00165	1.004
Post-Norm，无 warmup	4.21205	6.960
Post-Norm，120-step warmup	0.01738	2.759

结果故意不是“Post-Norm 学不会”。Warmup 救了它。结果是 Pre-Norm 能立刻容忍激进的目标学习率，而这次特定的 Post-Norm 运行不能。

这是教学用的压力测试，不是对 Xiong et al. 的复现：数据集是合成的，模型很小，换一个 seed 或超参数就能移动稳定性边界。有用的练习是扫描 --layers 和学习率，观察每种拓扑需要多少调参。

用 JAX 在 CPU 或 GPU 上运行

完整脚本就在本文旁边：

从仓库根目录：

# CPU, including Apple Silicon. The full run takes about 26 seconds here.
python -m pip install --upgrade jax flax optax matplotlib pillow
python _posts/pre_norm_vs_post_norm_demo.py --device cpu

# NVIDIA GPU on Linux with JAX's CUDA 13 wheel.
python -m pip install --upgrade "jax[cuda13]" flax optax matplotlib pillow
python _posts/pre_norm_vs_post_norm_demo.py --device gpu

# Prefer a JAX GPU backend when present, otherwise use CPU.
python _posts/pre_norm_vs_post_norm_demo.py --device auto

脚本会写出 img/pre-vs-post-norm-stability.webp 和底层 CSV。它是端到端的 JAX：模型用 Flax 写，梯度来自 jax.value_and_grad，更新来自 Optax，训练 step 用 jax.jit 编译。在 macOS 上，支持的默认是 Apple Silicon CPU；JAX 项目目前没有发布官方的 macOS GPU backend。架构切换本身只有这些：

if self.pre_norm:
    x = x + attention(attention_norm(x))
    return x + mlp(mlp_norm(x))

x = attention_norm(x + attention(x))
return mlp_norm(x + mlp(x))

编译后的更新同样是普通的 JAX：

loss, gradients = jax.value_and_grad(loss_fn)(params)
updates, optimizer_state = optimizer.update(gradients, optimizer_state, params)
params = optax.apply_updates(params, updates)

完整默认运行已经小到可以在 CPU 上用。若要更快的冒烟测试，减少工作量：

python _posts/pre_norm_vs_post_norm_demo.py \
  --device cpu --layers 6 --steps 30

这个更短的设置用来检查程序和所选 backend 能工作；不保证能复现完整的稳定性分离。

为什么 Pre-Norm 并不只是“更好”

Pre-Norm 通过让 residual stream 本身在堆叠内部不受约束来换取优化稳定性。展开递推式可以看出权衡：

\[x_L=x_0+\sum_{l=0}^{L-1}F_l(N(x_l)).\]

每一层都直接写入同一条累积的 stream。随着 $\lVert x_l\rVert$ 增大，一次新更新可能只占现有状态的更小比例。后面的 block 可能接近 identity 变换，贡献比其深度所暗示的更少。后续工作把相关症状描述为 representation collapse、residual-stream 增长、massive activations，或“深度诅咒”。精确的增长规律取决于初始化、训练、相关性和 residual scaling；它并非普遍指数增长。

Post-Norm 有相反的吸引力：它在每次 residual 加法之后重新归一化状态，严格控制前向尺度。代价是让归一化在反向传播中不可避免。这就是为什么若干更新的架构试图保留每种设计的最佳性质，而不是把任何一种当作普遍最优：

设计	试图保留什么
Pre-Norm	干净的 identity 路径和稳健的优化
Post-Norm	受控的 block 输出尺度
DeepNorm / residual scaling	带有有界更新的 Post-Norm 质量
NormFormer	Pre-Norm 中更平衡的逐层梯度
Peri-LN / sandwich norms	Identity residual 路径加上有界的 sublayer 输出
Final norm	language-model head 之前的稳定尺度

Final norm 有不同的职责

典型的现代 Pre-Norm LLM 还会施加一次最终归一化：

\[z=N_{\mathrm{final}}(x_L),\qquad \mathrm{logits}=W_{\mathrm{vocab}}z.\]

这并不会抵消 identity 路径的好处。block 级 residual stream 在整个深度上保持干净；归一化只在 output head 之前施加一次。

忽略 epsilon 和仿射参数，LayerNorm 和 RMSNorm 近似为零次齐次：

\[N(c x)=N(x),\qquad c>0.\]

因此 final norm 充当 尺度锚：单纯放大 $\lVert x_L\rVert$ 无法放大 logits。这与内部 Pre-Norm 的作用是分开的：

内部 Pre-Norm： 稳定每个分支输入并保留 identity 梯度路径。
Final norm： 稳定呈现给 language-model head 的尺度。

文献背后的设计规则

Pre-Norm 与 Post-Norm 的文献可以浓缩为两个同时要满足的要求：

保持干净的 residual identity 路径，让梯度能穿越深度。
控制写入该路径的更新大小，让 residual stream 不会压倒后面的层。

朴素 Pre-Norm 把第一个要求解决得非常好，这就是它成为深度 LLM 优化默认选择的原因。它只部分解决了第二个。DeepNorm、NormFormer、Peri-LN、residual scaling 以及相关设计，都是在不丢失 identity 高速路的前提下加入尺度控制的尝试。

要点

Pre-Norm 之所以重要，是因为它改变了深度 Transformer 的优化几何。 在

\[x_{l+1}=x_l+F_l(N(x_l)),\]

导数始终包含一个 identity 项。这给梯度一条穿越深度的直接通路，并让大模型对最早的更新不那么敏感。

好处主要是 可训练性，而不是最终表示更优的保证。当 warmup、初始化或 residual scaling 控制了其优化时，Post-Norm 可以工作——有时还可能产生更强的表示。现代目标不仅仅是“把归一化放在前面”，而是：

保留 identity 路径，然后控制每个学到的分支写入其中的内容。

参考文献

Kimi K3: An In-Depth Look at KDA

Sun, 09 Aug 2026 00:00:00 +0000

Kimi K3 (arXiv:2607.24653) is a 2.8T-parameter MoE with 104B active parameters, 93 layers, and a 1M-token context. Two architectural bets carry the report:

Kimi Delta Attention (KDA) — how information moves along the sequence
Attention Residuals (AttnRes) — how information moves along depth

Part 1 — Kimi Delta Attention

What KDA is replacing

Softmax attention keeps every token it has ever seen: generating token $t$ compares the query against $t$ cached keys, so per-token cost and KV cache both grow with the context. At 1M tokens that cache is the deployment bill.

Linear attention takes the other side of the trade. Drop the softmax and the whole history collapses into one fixed-size matrix $\mathbf{S} \in \mathbb{R}^{d_k \times d_v}$ carried from token to token:

\[\mathbf{S}_t = \mathbf{S}_{t-1} + k_t v_t^\top, \qquad o_t = \mathbf{S}_t^\top q_t\]

Constant memory, constant work per token, unbounded context. The catch is that the state never gets bigger, so everything in KDA is about what to write into it and what to erase.

The state is an associative memory

Read the update again as a memory. Writing $kv^\top$ stores the pair $(k, v)$; reading is a dot product against every key at once:

\[\mathbf{S}^\top q = \sum_i (k_i^\top q)\, v_i\]

If the keys are orthonormal, querying $k_1$ returns exactly $v_1$ — a $d_k \times d_v$ matrix holds $d_k$ clean slots. Reality is messier in two ways.

Keys collide. With $k_3$ at 45° to $k_1$, storing $(k_1, v_1)$ then $(k_3, v_2)$ and reading at $k_1$ gives $v_1 + 0.707\,v_2$. Crosstalk.

Rewrites pile up. Store x = 1, later x = 3; plain accumulation hands back both:

write (k1,v1), (k2,v2), then (k1,v1')     read at k1
  plain accumulation   ->  v1 + v1'      (stale value still there)
  delta rule, β = 1    ->  v1'           (old value erased)

A fixed-size memory that only ever adds becomes an increasingly blurry average of everything. Two fixes exist, and KDA uses both.

Fix 1 — forgetting, one rate per channel

Multiply the state down before each write:

\[\mathbf{S}_t = \operatorname{Diag}(\alpha_t)\,\mathbf{S}_{t-1} + k_tv_t^\top\]

Mamba-2 and Gated DeltaNet use a scalar $\alpha_t$: the whole memory fades at one rate. KDA makes $\alpha_t \in (0,1)^{d_k}$ channel-wise — one retention factor per key dimension, chosen per token. Row $j$ of the state fades at its own speed, so the same layer can hold a fast, local channel next to one that keeps information for thousands of tokens:

per-step log-decay $g$	retention $\alpha = e^{g}$	half-life
$-0.0001$	0.99990	6931 tokens
$-0.1$	0.90484	7 tokens

This is also where K3 gets its sense of position. Decay makes “20 tokens ago” measurably weaker than “2 tokens ago”, so the model needs no positional encoding at all (NoPE) — and a model with no RoPE frequencies has nothing to retune when the context window grows from 8K to 1M.

Fix 2 — the delta rule, or writing by correction

Instead of asking what to add, ask what the memory currently gets wrong. The memory should satisfy $\mathbf{S}^\top k_t = v_t$, so define a loss and take one gradient step:

\[\mathcal{L}(\mathbf{S}) = \tfrac{1}{2}\lVert \mathbf{S}^\top k_t - v_t \rVert^2, \quad \nabla_{\mathbf{S}}\mathcal{L} = k_t(\mathbf{S}^\top k_t - v_t)^\top\] \[\mathbf{S}_t = \mathbf{S}_{t-1} - \beta_t \nabla_{\mathbf{S}}\mathcal{L} = \left(\mathbf{I} - \beta_t k_tk_t^\top\right)\mathbf{S}_{t-1} + \beta_t k_tv_t^\top\]

That is the delta rule, and a linear-attention layer is doing online gradient descent on its own memory at inference time, with $\beta_t$ as the learning rate.

The geometry is the part worth keeping. Because KDA L2-normalises its keys, $\lVert k_t\rVert = 1$ and $\mathbf{I} - \beta_tk_tk_t^\top$ is a scaling along $k_t$ and the identity everywhere else. It erases only what was stored at this key and leaves the rest of the memory untouched:

$\beta_t = 1$ — full overwrite: whatever was at $k_t$ is gone, $v_t$ takes its place
$\beta_t = 0$ — no write at all
in between — a partial correction toward $v_t$ (with $\beta = 0.5$ the numbers above give $0.25\,v_1 + 0.5\,v_1'$)

KDA = both, and here is the whole layer

\[\mathbf{S}_t=\left(\mathbf{I}-\beta_t k_t k_t^{\top}\right) \operatorname{Diag}(\alpha_t) \mathbf{S}_{t-1}+\beta_t k_t v_t^{\top}, \qquad \tilde{o}_t=\mathbf{S}_t^{\top} q_t\]

Decay first, then the correction. Everything on the right is produced from the token itself:

term	what it is	how it is produced
$q_t,k_t$	probe / address	ShortConv → Swish → L2Norm
$v_t$	payload	ShortConv → Swish
$\beta_t \in (0,1)$	write strength	$\operatorname{Sigmoid}(\mathbf{W}_\beta x_t)$
$\alpha_t \in (0,1)^{d_k}$	per-channel retention	low-rank logits + per-head bias, then Eq. below

The ShortConv gives each token a short local window before it becomes a key or value; the L2Norm is what makes the erase step a clean projection rather than an arbitrary rescaling.

The update is easiest to implement in its equivalent decay → predict → correct → read form:

decayed_state = row_scale(alpha_t, state)
prediction = transpose_matrix_vector(decayed_state, key_t)
error = vector_sub(value_t, prediction)
state = matrix_add(decayed_state, matrix_scale(beta_t, outer(key_t, error)))
output = transpose_matrix_vector(state, query_t)

This version exposes the idea more clearly than constructing $\mathbf I-\beta kk^\top$: the layer forgets rows of its state, asks what value the decayed state predicts for the current key, and writes only the residual error.

Run the complete tutorial on a CPU

The complete dependency-free Python tutorial uses only the standard library. It is not Kimi K3 and it is not a fast production kernel; it is a small numerical laboratory for the equations in this article. From the repository root:

python3 examples/kda_cpu_tutorial.py

It runs five experiments:

writes an association, repeats it, and shows that the delta rule does not double-count it;
overwrites the same key with a new value and demonstrates the effect of $\beta$;
applies different retention factors to different state rows;
verifies the UT chunk form against token-by-token KDA;
verifies associative segment composition for KDA Context Parallelism and compares the old and new decay maps.

I ran it on CPU. The two ways of evaluating a chunk agreed to floating-point precision:

Maximum output difference:      8.882e-16
Maximum final-state difference: 6.661e-16
Maximum segment difference:     4.441e-16
All numerical equivalence checks passed.

I also ran 100 random recurrence-versus-chunk tests and 100 random segment-composition tests; all passed. These checks validate the educational implementation’s algebra, not the speed or low-precision numerical behavior of FlashKDA’s CUDA kernel.

Big-O: what becomes linear, and what does not

Let $T$ be sequence length, $H$ the number of heads, and $d_k,d_v$ the key and value dimensions per head. Ignoring the Q/K/V projections shared by both designs, causal softmax attention and recurrent KDA scale as follows:

mechanism	sequence work	decode cache	next token
softmax	$O(HT^2d_k)$	$O(HT(d_k+d_v))$	$O(HT(d_k+d_v))$
KDA	$O(HTd_kd_v)$	$O(Hd_kd_v)$	$O(Hd_kd_v)$

When $H,d_k,d_v$ are fixed architectural constants, KDA is $O(T)$ over a sequence and $O(1)$ in persistent cache with respect to context length. “Constant” does not mean free: every new token still reads and updates an entire $d_k\times d_v$ state per head.

Kimi K3 uses $H=96$ and $d_k=d_v=128$, so one KDA layer carries

\[96\times128\times128=1{,}572{,}864\]

state elements per sequence: about 3 MiB in BF16. Across 69 KDA layers that is about 207 MiB before tensor-parallel sharding, convolution histories, allocators, and saved prefix checkpoints. Its size does not grow when the context goes from 1K to 1M tokens.

The production chunk kernel has the per-head attention cost reported in Kimi Linear:

\[6Td_h^2+3TCd_h+TC^2,\]

where $C$ is chunk size and $d_h=d_k=d_v$. This remains $O(T)$ for fixed $C$ and $d_h$. The tutorial’s explicit Python loops are deliberately readable rather than optimized; timing them would measure Python overhead, not FlashKDA.

The whole Kimi K3 model is also not constant-cache. Its 24 MLA layers still keep sequence-growing latent KV caches and perform global attention. The 69 KDA layers remove that growth from roughly three quarters of the attention stack.

Why use KDA?

KDA is useful when generation is long enough that repeatedly reading a growing KV cache dominates cost:

Fixed recurrent memory. Most layers replace per-token KV entries with one state matrix.
Corrective writes. The delta rule overwrites an association instead of accumulating stale and duplicate values.
Different memory timescales. Channel-wise $\alpha$ lets one head maintain fast-forgetting and slow-retaining features simultaneously.
Implicit order and recency. The ordered, data-dependent transitions change when token order changes, so the model can use NoPE in its global layers.
Exact GPU-friendly reformulation. Training uses chunkwise matrix multiplication without changing the mathematical recurrence.

The price is finite capacity. A fixed state can blur unrelated facts or lose exact token details, which is why Kimi K3 retains periodic global MLA rather than using KDA everywhere.

Chunkwise parallel form, step by step

The recurrent loop is ideal for decoding one token at a time, but poor for training: token $r$ needs the state produced by token $r-1$, so a literal implementation launches one small matrix update after another. Chunkwise KDA does not remove the recurrence. It packages most of the work inside a fixed-size chunk into dense matrix multiplications, while passing one state sequentially between chunks.

1. Lay out one chunk

For a chunk of $C$ tokens, stack rows as

\[\mathbf Q,\mathbf K\in\mathbb R^{C\times d_k},\qquad \mathbf V,\mathbf O\in\mathbb R^{C\times d_v},\qquad \mathbf S_{\mathrm{in}}\in\mathbb R^{d_k\times d_v}.\]

For positions $1\le i\le j\le C$, define the channel-wise cumulative retention

\[\gamma^{i\to j}=\prod_{r=i}^{j}\alpha^r, \qquad \gamma^j=\gamma^{1\to j}.\]

Every product is elementwise over the $d_k$ key channels. Let $\mathbf\Gamma\in\mathbb R^{C\times d_k}$ stack $\gamma^1,\ldots,\gamma^C$ row by row.

2. See what the recurrence is hiding

Write one token transition as

\[\mathbf T_r=(\mathbf I-\beta_r k_rk_r^\top)\operatorname{Diag}(\alpha_r), \qquad \mathbf B_r=\beta_rk_rv_r^\top.\]

After the first $r$ tokens of the chunk,

\[\mathbf S^r= \underbrace{\mathbf T_r\mathbf T_{r-1}\cdots\mathbf T_1}_{\mathbf P^r} \mathbf S_{\mathrm{in}} + \sum_{i=1}^{r} \mathbf T_r\cdots\mathbf T_{i+1}\mathbf B_i.\]

The newest transition is on the left. This equation explains the difficulty: every write is transformed by all later decays and delta erasures. Computing those products separately for every output would repeat the same work.

3. Turn relative decay into query/key scaling

Consider a query at row $i$ reading a key written at row $j\le i$. The retention between them is

\[\frac{\gamma^i}{\gamma^j} = \prod_{r=j+1}^{i}\alpha^r.\]

Therefore its decayed similarity can be written as

\[q_i^\top\operatorname{Diag}\!\left(\frac{\gamma^i}{\gamma^j}\right)k_j = (\gamma^i\odot q_i)^\top(k_j/\gamma^j).\]

All such scores appear in one matrix multiplication:

\[\mathbf A= \operatorname{Tril}\!\left[ (\mathbf Q\odot\mathbf\Gamma) (\mathbf K/\mathbf\Gamma)^\top \right] \in\mathbb R^{C\times C}.\]

Tril removes future positions but keeps the diagonal. For $i=j$, the ratio is one: an output reads $\mathbf S_i$ after its own write, so token $i$ must be allowed to read its own pseudo-value.

4. Fold the delta-rule dependencies with the UT transform

Decay scaling alone is not enough, because a delta write stores the residual after accounting for previous writes. Define

\[\mathbf L= \operatorname{StrictTril}\!\left[ \operatorname{Diag}(\boldsymbol\beta) (\mathbf\Gamma\odot\mathbf K) (\mathbf K/\mathbf\Gamma)^\top \right] \in\mathbb R^{C\times C}.\]

The strict lower triangle contains how each earlier key changes the prediction seen by a later key. The UT transform solves that causal system:

\[\mathbf M=(\mathbf I+\mathbf L)^{-1}\operatorname{Diag}(\boldsymbol\beta),\] \[\mathbf W=\mathbf M(\mathbf\Gamma\odot\mathbf K) \in\mathbb R^{C\times d_k}, \qquad \mathbf U=\mathbf M\mathbf V \in\mathbb R^{C\times d_v}.\]

Because $\mathbf I+\mathbf L$ is lower triangular with ones on its diagonal, this is a forward substitution rather than a general matrix inverse. The compact pseudo-values are

\[\widetilde{\mathbf V} = \mathbf U-\mathbf W\mathbf S_{\mathrm{in}} \in\mathbb R^{C\times d_v}.\]

The two terms have direct meanings: $\mathbf U$ contains mutually corrected current-chunk values, while $\mathbf W\mathbf S_{\mathrm{in}}$ subtracts what the incoming state already predicts. Thus each row of $\widetilde{\mathbf V}$ is the effective residual that token contributes after all earlier in-chunk corrections.

5. Produce every output in the chunk

Once $\widetilde{\mathbf V}$ is known,

\[\boxed{ \mathbf O= \underbrace{(\mathbf\Gamma\odot\mathbf Q)\mathbf S_{\mathrm{in}}}_{\text{memory from earlier chunks}} + \underbrace{\mathbf A\widetilde{\mathbf V}}_{\text{writes in this chunk}} }\]

has shape $C\times d_v$. The first term decays and queries the state entering the chunk. The second is a causal weighted sum of corrected writes from the current chunk. This is Kimi K3 Eq. 4.

The state handed to the next chunk is computed from the same pseudo-values:

\[\boxed{ \mathbf S_{\mathrm{out}} = \operatorname{Diag}(\gamma^C)\mathbf S_{\mathrm{in}} + (\mathbf\Delta\odot\mathbf K)^\top\widetilde{\mathbf V} }\]

where row $i$ of $\mathbf\Delta$ is the retention after that write,

\[\Delta_i=\prod_{r=i+1}^{C}\alpha^r,\]

with an empty product of one for the last token. The incoming state experiences all $C$ decays; a write at position $i$ experiences only later decays.

6. What is actually parallel?

The chunks remain recurrent: $\mathbf S_{\mathrm{out}}$ from chunk $t$ is $\mathbf S_{\mathrm{in}}$ for chunk $t+1$. Inside a chunk, the expensive query-key, output, and state-update work is expressed as matrix multiplication. A causal triangular solve remains in the UT transform, so “parallel within a chunk” means the bulk of the arithmetic is tiled dense work, not that every operation is independent.

The CPU tutorial’s official_naive_chunk_kda implements this sequence explicitly: cumulative log-decay, lower-triangular UT solve, $\mathbf W/\mathbf U$, pseudo-values, causal scores, outputs, and final state. Against token-by-token recurrence it produced maximum output and state differences of $8.882\times10^{-16}$ and $6.661\times10^{-16}$. The chunkwise algorithm is an exact algebraic rewrite in real arithmetic; different accumulation order and BF16 storage explain small production-kernel differences.

What K3 changed relative to Kimi Linear

KDA comes from Kimi Linear (arXiv:2510.26692). K3 makes two edits, both small on paper and both about hardware.

1. Lower-bounded decay. Look again at $\mathbf{K}/\mathbf{\Gamma}$: a reciprocal of a product of numbers below 1. Kimi Linear’s decay came from a negative softplus, $g = -e^{A}\operatorname{Softplus}(z) \in (-\infty, 0)$, so that reciprocal can explode and overflow. K3 bounds it with a scaled sigmoid:

\[g_t = g_{\min}\operatorname{Sigmoid}(e^{A_h}z_t) \in (g_{\min}, 0), \qquad g_{\min} = -5\]

Now every step retains at least $e^{-5} \approx 0.0067$, cumulative log-decay over a 16-token tile stays in $(-80, 0)$, and the rescaling factor is at most $e^{80} \approx 5.5\times10^{34}$ — comfortably inside BF16’s $3.4\times10^{38}$. The payoff is concrete: Kimi Linear had to compute the diagonal tiles with explicit position-pair arithmetic, the main intra-chunk bottleneck. With a bounded range every tile, diagonal included, becomes a dense tensor-core matmul. A numerical-stability bound bought a kernel rewrite.

2. Full-rank output gate. The low-rank gate becomes an input-dependent full-rank projection, so each token can decide channel by channel how much of the recurrent read to let through:

\[y_t=\mathbf{W}_o\left[\operatorname{Sigmoid}(\mathbf{W}_g x_t) \odot \operatorname{RMSNorm}(\tilde{o}_t)\right]\]

KDA does not work alone: 3:1 with Gated MLA

A fixed state is lossy by construction, so K3 interleaves exact attention. Each block is 3 KDA layers + 1 Gated MLA layer, and one extra MLA closes the backbone:

$23 \times (3\,\text{KDA} + 1\,\text{MLA}) + 1\,\text{MLA} = 93$ layers, i.e. 69 KDA and 24 MLA.

Only those 24 layers keep a cache that grows with the sequence — against 61 full-attention layers in K2. The MLA layers are NoPE too, and carry the same full-rank output gate.

The division of labour is clean. The 69 KDA layers give recency-weighted, position-aware mixing out of a fixed $d_k \times d_v$ state at constant cost per token; the 24 MLA layers keep a latent cache that grows with $T$ and buy back what a finite state cannot hold — exact access to any earlier token.

At the Kimi Linear scale the hybrid cut KV-cache usage by up to 75% and reached up to 6× decoding throughput at 1M context, while beating full MLA on quality under a matched recipe.

The state is small — but it is serial

Most of K3’s KDA engineering follows from one sentence: the state is cheap to move and impossible to skip ahead in.

FlashKDA — a CUTLASS chunkwise kernel that overlaps intra-chunk math with cross-chunk state propagation, so the SMs are not idle during the serial hand-off. It also serves prefill, as a backend of flash-linear-attention.
KDA Context Parallelism (KCP) — the interesting one. Vanilla linear attention is a plain sum, so every rank can start from $\mathbf{S}=\mathbf{0}$ and the results add up. KDA cannot: in $\mathbf{S}_t = \mathbf{M}_t\mathbf{S}_{t-1} + \beta_tk_tv_t^\top$ with $\mathbf{M}_t = (\mathbf{I}-\beta_tk_tk_t^\top)\operatorname{Diag}(\alpha_t)$, the incoming state is transformed, not just added to. So each rank computes two local quantities — its segment’s cumulative transition $\mathbf{M}$, and the state its own tokens generate from zero — and one all-gather plus a prefix scan composes them exactly. Messages stay fixed-size at any context length, which is what makes 1M-token training affordable.
Prefix caching — KDA state checkpoints land at 512-token boundaries in the same paged pool as the MLA KV cache; a prefix is reusable only if both restore at the same boundary.
Speculative decoding — the state updates in place, so a rejected draft cannot be rolled back. K3 caches the drafts’ much smaller projected inputs and replays the accepted prefix on-chip.

What to remember about KDA

A fixed-size matrix used as an associative memory, read with $\mathbf{S}^\top q$.
Channel-wise decay — every key channel picks its own forgetting rate, which also encodes position (hence NoPE, hence painless 1M extension).
The delta rule — write by erasing what was stored at this key first; it is one step of online gradient descent, and $\beta_t$ is the learning rate.
Chunkwise form — the exact same recurrence expressed as matmuls; K3’s bounded decay pushes the last stubborn tile onto tensor cores.
Hybrid by design — 3:1 with Gated MLA, because a finite state should not be asked to do exact recall.

Part 2 — Attention Residuals

KDA fixes the sequence axis. AttnRes (arXiv:2603.15031) runs the same argument down the depth axis.

The problem

A PreNorm residual looks innocent:

\[h_l = h_{l-1} + f_l(h_{l-1}) = h_0 + \sum_{i=1}^{l} f_i(h_{i-1})\]

Every earlier layer is added with weight 1. Depth is an RNN: all history is crushed into one vector. Hidden-state magnitude grows with depth, so each new layer is a smaller and smaller fraction of the stream — PreNorm dilution. Early information cannot be fetched back on demand.

Sequence modeling had the same bottleneck, and softmax attention replaced the RNN. AttnRes does that for depth: a standard residual is depth-wise linear attention; AttnRes is depth-wise softmax attention.

Three residuals

Standard. Each layer sees only $h_{l-1}$, with fixed mixing weights and one hidden state travelling between layers — $N = 1$ below.

Full AttnRes. Every layer output becomes a key/value, and each layer picks among them:

\[h_l = \sum_{i=0}^{l-1} \alpha_{i \to l}\, v_i, \qquad \alpha_{i \to l} = \mathrm{softmax}_i\big(w_l^\top \mathrm{RMSNorm}(k_i)\big)\]

$w_l \in \mathbb{R}^d$: one learned pseudo-query per layer, decoupled from that layer’s forward pass, so the mix is content-dependent
RMSNorm on the keys stops large-magnitude layers from dominating; queries start at zero, so training begins as a uniform average and does not spike
Compute $O(L^2 d)$, store $O(Ld)$. The real cost is that pipeline parallelism must ship every layer output across stages

Block AttnRes (what ships). Split $L$ layers into $N$ blocks: inside a block an ordinary residual, collapsed to one block vector; across blocks Full AttnRes over the $N$ summaries plus the embedding, with the unfinished block exposing a partial sum. Traffic drops from $O(Ld)$ to $O(Nd)$. $N = L$ is Full, $N = 1$ is Standard, and empirically $N \approx 8$ recovers most of Full. Kimi Linear 48B used 6 layers per block → 9 blocks + embedding = 10 depth sources, for < 4% training overhead and < 2% decode latency.

Experiments

Scaling law. Five sizes, each with Baseline / Block ($N=8$) / Full, all under the baseline’s hyperparameters — a deliberately conservative test. AttnRes is lower loss along the whole compute curve. Largest size: Baseline 1.719, Block 1.693, Full 1.692, and at 5.6 PFLOP/s-days the baseline needs about 25% more compute to match Block. Full is the ceiling; Block is “almost the same, and you can actually train it.”

48B / 3B active, 1.4T tokens. Validation loss is lower throughout and the gap widens during decay. Baseline output magnitude grows monotonically with depth, while Block resets at block boundaries; gradients even out too, since the softmax makes depth sources compete instead of dumping everything on the earliest layers.

	Baseline	AttnRes
MMLU	73.5	74.6
GPQA-Diamond	36.9	44.4 (+7.5)
Math	53.5	57.1 (+3.6)
HumanEval	59.1	62.2 (+3.1)
C-Eval	79.6	82.5

Knowledge moves a little. Multi-step reasoning and code move a lot — consistent with later layers being able to pull earlier representations on demand.

16-layer ablation (loss, lower is better): Baseline PreNorm 1.766, DenseFormer 1.767, mHC 1.747, Full AttnRes 1.737, input-independent mixing 1.749, sigmoid instead of softmax 1.741, no RMSNorm 1.743, Block $S=4$ 1.746. Fixed mixing is clearly worse than learned softmax — content-dependent depth selection is doing real work.

Takeaway

K3 grew depth (K2 had 61 layers; K3 has 93). A unit-weight residual starts to look like an RNN on that axis, exactly as a growing KV cache is the wrong answer on the sequence axis. So both axes get the same treatment: selective, data-dependent retrieval instead of uniform accumulation — KDA across tokens, AttnRes across layers. Together with Stable LatentMoE on the width axis, that is where the reported ~2.5× scaling-efficiency gain over K2 comes from.

Papers: Kimi K3 · Kimi Linear · Attention Residuals. Code: flash-linear-attention.

Kimi K3（arXiv:2607.24653）是 2.8T 参数的 MoE，激活 104B，93 层，1M 上下文。报告最核心的两个架构选择：

Kimi Delta Attention（KDA） —— 信息沿序列方向怎么流
Attention Residuals（AttnRes） —— 信息沿深度方向怎么流

第一部分 —— Kimi Delta Attention

KDA 要替掉什么

Softmax attention 把见过的每个 token 都留着：生成第 $t$ 个 token 要和 $t$ 个缓存 key 比较，单 token 开销和 KV cache 都随上下文增长。到 1M 上下文，这块 cache 就是部署成本本身。

线性注意力走另一条路：去掉 softmax，整段历史被压进一个固定大小的矩阵 $\mathbf{S} \in \mathbb{R}^{d_k \times d_v}$，在 token 之间传递：

\[\mathbf{S}_t = \mathbf{S}_{t-1} + k_t v_t^\top, \qquad o_t = \mathbf{S}_t^\top q_t\]

常数内存、常数单步计算、上下文无上限。代价是状态永远不会变大，所以 KDA 的全部问题就是：往里写什么、擦掉什么。

状态就是一块联想记忆

把更新式当成记忆来读。写入 $kv^\top$ 就是存下 $(k, v)$；读取是一次对所有 key 的点积：

\[\mathbf{S}^\top q = \sum_i (k_i^\top q)\, v_i\]

key 正交时，用 $k_1$ 查询正好取回 $v_1$ —— 一个 $d_k \times d_v$ 的矩阵相当于 $d_k$ 个干净的槽位。现实有两处不干净。

key 会撞。 设 $k_3$ 与 $k_1$ 夹角 45°，先存 $(k_1, v_1)$ 再存 $(k_3, v_2)$，用 $k_1$ 读回来是 $v_1 + 0.707\,v_2$，串味了。

改写会堆积。 先写 x = 1，后写 x = 3，纯累加会把两个都还给你：

依次写 (k1,v1), (k2,v2), (k1,v1')      用 k1 读
  纯累加        ->  v1 + v1'          （旧值还在）
  delta rule, β = 1 ->  v1'           （旧值被擦掉）

只加不减的定容记忆，最后就是一份越来越糊的平均。解法有两个，KDA 两个都用。

解法一 —— 遗忘，而且每个通道一个速率

每次写入前先把状态乘小：

\[\mathbf{S}_t = \operatorname{Diag}(\alpha_t)\,\mathbf{S}_{t-1} + k_tv_t^\top\]

Mamba-2 和 Gated DeltaNet 用标量 $\alpha_t$：整块记忆按同一速率衰减。KDA 把它做成 channel-wise 的 $\alpha_t \in (0,1)^{d_k}$ —— 每个 key 维度一个保留系数，逐 token 决定。状态的第 $j$ 行按自己的速度衰减，于是同一层里可以既有快通道，也有能记住几千 token 的慢通道：

单步 log-decay $g$	保留率 $\alpha = e^{g}$	半衰期
$-0.0001$	0.99990	6931 tokens
$-0.1$	0.90484	7 tokens

K3 的位置信息也从这里来。衰减让「20 个 token 之前」明显弱于「2 个 token 之前」，于是模型 完全不需要位置编码（NoPE）—— 没有 RoPE 频率，上下文从 8K 拉到 1M 时也就没有东西要重调。

解法二 —— delta rule：按误差改写

不要问该加什么，要问这块记忆现在错在哪。记忆应该满足 $\mathbf{S}^\top k_t = v_t$，那就定义损失、走一步梯度：

这就是 delta rule。换句话说，线性注意力层在推理时对自己的记忆做在线梯度下降，$\beta_t$ 就是学习率。

几何意义更值得记住。KDA 对 key 做了 L2 归一化，$\lVert k_t\rVert = 1$，所以 $\mathbf{I} - \beta_tk_tk_t^\top$ 只沿 $k_t$ 方向缩放，其余方向是恒等。它 只擦掉这个 key 上存过的东西，不动记忆里的其他内容：

$\beta_t = 1$ —— 完全覆盖：$k_t$ 上原来的值消失，换成 $v_t$
$\beta_t = 0$ —— 不写
中间值 —— 朝 $v_t$ 部分修正（上面的例子取 $\beta = 0.5$ 得到 $0.25\,v_1 + 0.5\,v_1'$）

KDA = 两个都要，整层长这样

\[\mathbf{S}_t=\left(\mathbf{I}-\beta_t k_t k_t^{\top}\right) \operatorname{Diag}(\alpha_t) \mathbf{S}_{t-1}+\beta_t k_t v_t^{\top}, \qquad \tilde{o}_t=\mathbf{S}_t^{\top} q_t\]

先衰减，再修正。右边所有量都由当前 token 算出来：

项	是什么	怎么来的
$q_t,k_t$	探针 / 地址	ShortConv → Swish → L2Norm
$v_t$	内容	ShortConv → Swish
$\beta_t \in (0,1)$	写入强度	$\operatorname{Sigmoid}(\mathbf{W}_\beta x_t)$
$\alpha_t \in (0,1)^{d_k}$	每通道保留率	低秩 logits + 每头 bias，再走下面的映射

ShortConv 让每个 token 在变成 key / value 之前先看一个小局部窗口；L2Norm 则保证擦除那一步是干净的投影，而不是任意缩放。

把更新式写成等价的 衰减 → 预测 → 修正 → 读取 最容易看懂：

decayed_state = row_scale(alpha_t, state)
prediction = transpose_matrix_vector(decayed_state, key_t)
error = vector_sub(value_t, prediction)
state = matrix_add(decayed_state, matrix_scale(beta_t, outer(key_t, error)))
output = transpose_matrix_vector(state, query_t)

这种写法比显式构造 $\mathbf I-\beta kk^\top$ 更直观：先按行遗忘，再看衰减后的状态对当前 key 预测出什么，只把预测误差写回去。

在 CPU 上运行完整教程

完整的零依赖 Python 教程只用标准库。它不是 Kimi K3，也不是高性能 kernel，而是本文公式的一个小型数值实验室。在仓库根目录运行：

python3 examples/kda_cpu_tutorial.py

脚本会做五组实验：

写入同一个关联两次，验证 delta rule 不会重复累加；
用新值覆盖同一个 key，并展示 $\beta$ 的作用；
给状态的不同行施加不同保留率；
把 UT chunk 形式和逐 token KDA 对齐；
验证 KDA Context Parallelism 的分段结合律，并比较新旧衰减映射。

我在 CPU 上实际运行过。chunk 的两种算法只差浮点舍入误差：

Maximum output difference:      8.882e-16
Maximum final-state difference: 6.661e-16
Maximum segment difference:     4.441e-16
All numerical equivalence checks passed.

我还跑了 100 组随机的递推-vs-chunk 测试，以及 100 组随机的分段组合测试，全部通过。它们验证的是教学实现的代数，不代表 FlashKDA CUDA kernel 的速度或低精度数值行为。

Big-O：哪部分变成线性，哪部分没有

记 $T$ 为序列长度，$H$ 为 head 数，$d_k,d_v$ 为单 head 的 key/value 维度。忽略两种方案共有的 Q/K/V 投影，因果 softmax attention 和递推 KDA 的规模如下：

机制	序列计算	decode cache	下一 token
softmax	$O(HT^2d_k)$	$O(HT(d_k+d_v))$	$O(HT(d_k+d_v))$
KDA	$O(HTd_kd_v)$	$O(Hd_kd_v)$	$O(Hd_kd_v)$

当 $H,d_k,d_v$ 是固定的架构常数时，KDA 对序列长度是 $O(T)$，持久化 cache 对上下文长度是 $O(1)$。“常数”不等于免费：每生成一个 token，仍要读写每个 head 的整个 $d_k\times d_v$ 状态。

Kimi K3 取 $H=96$、$d_k=d_v=128$，所以每个 KDA 层、每条序列的状态有

\[96\times128\times128=1{,}572{,}864\]

个元素，BF16 下约 3 MiB。69 个 KDA 层合计约 207 MiB，还没算张量并行切分方式、卷积历史、内存分配和 prefix checkpoint。上下文从 1K 拉到 1M，这块状态本身不会继续长。

Kimi Linear 给出的生产 chunk kernel 单 head attention 计算量是

\[6Td_h^2+3TCd_h+TC^2,\]

其中 $C$ 是 chunk size，$d_h=d_k=d_v$。固定 $C$ 和 $d_h$ 后仍是 $O(T)$。教程里的显式 Python 循环是为了可读性，不是为了速度；给它计时，测到的主要是 Python 开销，不是 FlashKDA。

整个 Kimi K3 也不是常数 cache：24 个 MLA 层仍然保存随序列增长的 latent KV cache，并执行全局注意力。69 个 KDA 层只是把大约四分之三 attention stack 的增长拿掉。

为什么要用 KDA？

当生成足够长、反复读取不断增大的 KV cache 成为主要成本时，KDA 很有价值：

固定大小的递推记忆。 大部分层用一个状态矩阵取代逐 token KV。
按误差修正。 delta rule 覆盖旧关联，而不是把重复值和过期值一直累加。
多种记忆时标。 channel-wise $\alpha$ 让同一 head 同时拥有快忘和慢记的特征。
隐式顺序与近因性。 有序、随数据变化的转移在 token 顺序改变时也会改变，所以全局层可以使用 NoPE。
精确而适合 GPU 的改写。 训练时用 chunkwise 矩阵乘，但数学递推没有改变。

代价是容量有限。固定状态会把无关事实混在一起，也可能丢失精确 token 细节，所以 Kimi K3 没有把所有层都换成 KDA，而是保留周期性的全局 MLA。

Chunkwise parallel form：一步一步推

逐 token 递推很适合 decoding，但不适合训练：第 $r$ 个 token 必须等第 $r-1$ 个 token 产出状态，直接实现就是连续发射很多小矩阵更新。Chunkwise KDA 没有消灭递推；它把一个固定大小 chunk 里的大部分工作整理成稠密矩阵乘，只在 chunk 之间顺序传递一个状态。

1. 先把一个 chunk 排成矩阵

一个 chunk 有 $C$ 个 token，按行堆叠：

\[\mathbf Q,\mathbf K\in\mathbb R^{C\times d_k},\qquad \mathbf V,\mathbf O\in\mathbb R^{C\times d_v},\qquad \mathbf S_{\mathrm{in}}\in\mathbb R^{d_k\times d_v}.\]

对位置 $1\le i\le j\le C$，定义逐通道累积保留率

\[\gamma^{i\to j}=\prod_{r=i}^{j}\alpha^r, \qquad \gamma^j=\gamma^{1\to j}.\]

这些乘法都在 $d_k$ 个 key 通道上逐元素进行。令 $\mathbf\Gamma\in\mathbb R^{C\times d_k}$ 逐行堆叠 $\gamma^1,\ldots,\gamma^C$。

2. 看清递推里藏着什么

把单 token 转移写成

\[\mathbf T_r=(\mathbf I-\beta_r k_rk_r^\top)\operatorname{Diag}(\alpha_r), \qquad \mathbf B_r=\beta_rk_rv_r^\top.\]

走完 chunk 的前 $r$ 个 token 后，

\[\mathbf S^r= \underbrace{\mathbf T_r\mathbf T_{r-1}\cdots\mathbf T_1}_{\mathbf P^r} \mathbf S_{\mathrm{in}} + \sum_{i=1}^{r} \mathbf T_r\cdots\mathbf T_{i+1}\mathbf B_i.\]

最新的转移在最左边。困难也写在式子里：每次写入都会被后续所有衰减和 delta 擦除再次变换。如果给每个输出单独算这些乘积，会反复做大量相同工作。

3. 把相对衰减变成 query/key 缩放

位置 $i$ 的 query 读取位置 $j\le i$ 写下的 key，两者之间的保留率是

\[\frac{\gamma^i}{\gamma^j} = \prod_{r=j+1}^{i}\alpha^r.\]

所以带衰减的相似度可以写成

\[q_i^\top\operatorname{Diag}\!\left(\frac{\gamma^i}{\gamma^j}\right)k_j = (\gamma^i\odot q_i)^\top(k_j/\gamma^j).\]

所有位置对的分数一次矩阵乘就能得到：

\[\mathbf A= \operatorname{Tril}\!\left[ (\mathbf Q\odot\mathbf\Gamma) (\mathbf K/\mathbf\Gamma)^\top \right] \in\mathbb R^{C\times C}.\]

Tril 去掉未来位置，但 保留对角线。当 $i=j$ 时比例是 1；输出读取的是当前 token 已经写入后的 $\mathbf S_i$，所以 token $i$ 必须能读自己的伪 value。

4. 用 UT transform 折叠 delta-rule 依赖

只有衰减缩放还不够，因为 delta 写入的是扣除先前预测后的残差。定义

\[\mathbf L= \operatorname{StrictTril}\!\left[ \operatorname{Diag}(\boldsymbol\beta) (\mathbf\Gamma\odot\mathbf K) (\mathbf K/\mathbf\Gamma)^\top \right] \in\mathbb R^{C\times C}.\]

严格下三角部分描述「更早的 key 如何改变后续 key 看到的预测」。UT transform 解这个因果系统：

因为 $\mathbf I+\mathbf L$ 是对角线为 1 的下三角矩阵，这一步实际用前向代入，不是通用矩阵求逆。紧凑的 伪 value 是

\[\widetilde{\mathbf V} = \mathbf U-\mathbf W\mathbf S_{\mathrm{in}} \in\mathbb R^{C\times d_v}.\]

两个项都有直接含义：$\mathbf U$ 包含 chunk 内彼此修正后的当前 value，$\mathbf W\mathbf S_{\mathrm{in}}$ 则扣掉入口状态已经能够预测的部分。于是 $\widetilde{\mathbf V}$ 的每一行，都是算入所有更早 chunk 内修正后，这个 token 真正贡献的残差。

5. 一次算出 chunk 里的所有输出

得到 $\widetilde{\mathbf V}$ 后，

\[\boxed{ \mathbf O= \underbrace{(\mathbf\Gamma\odot\mathbf Q)\mathbf S_{\mathrm{in}}}_{\text{来自更早 chunk 的记忆}} + \underbrace{\mathbf A\widetilde{\mathbf V}}_{\text{当前 chunk 的写入}} }\]

形状是 $C\times d_v$。第一项对 chunk 入口状态做衰减后查询；第二项是当前 chunk 中修正后写入的因果加权和。这就是 Kimi K3 的 Eq. 4。

传给下一个 chunk 的状态也从同一组伪 value 得到：

\[\boxed{ \mathbf S_{\mathrm{out}} = \operatorname{Diag}(\gamma^C)\mathbf S_{\mathrm{in}} + (\mathbf\Delta\odot\mathbf K)^\top\widetilde{\mathbf V} }\]

其中 $\mathbf\Delta$ 的第 $i$ 行是该次写入之后经历的保留率，

\[\Delta_i=\prod_{r=i+1}^{C}\alpha^r,\]

最后一个 token 后面没有衰减，空乘积按 1 处理。入口状态经历全部 $C$ 次衰减；位置 $i$ 的写入只经历它后面的衰减。

6. 到底哪里并行？

chunk 之间仍然递推：第 $t$ 个 chunk 的 $\mathbf S_{\mathrm{out}}$ 就是第 $t+1$ 个 chunk 的 $\mathbf S_{\mathrm{in}}$。chunk 内部则把昂贵的 query-key、输出和状态更新整理成矩阵乘。UT transform 里仍有一次因果三角求解，所以「chunk 内并行」的准确含义是 绝大部分算术变成可分块的稠密计算，而不是每一步都完全独立。

CPU 教程里的 official_naive_chunk_kda 显式走完这条链：累积 log-decay、下三角 UT 求解、$\mathbf W/\mathbf U$、伪 value、因果分数、输出和最终状态。它和逐 token 递推相比，最大输出误差与状态误差分别是 $8.882\times10^{-16}$ 和 $6.661\times10^{-16}$。因此 chunkwise 在实数算术下是精确代数改写；生产 kernel 的微小差异来自累加顺序和 BF16 状态存储。

相对 Kimi Linear，K3 改了什么

KDA 来自 Kimi Linear（arXiv:2510.26692）。K3 改了两处，纸面上都很小，但都是冲着硬件去的。

1. 有下界的衰减。 再看 $\mathbf{K}/\mathbf{\Gamma}$：一串小于 1 的数连乘之后取倒数。Kimi Linear 用负 softplus，$g = -e^{A}\operatorname{Softplus}(z) \in (-\infty, 0)$，倒数可以炸掉溢出。K3 改成带缩放的 sigmoid：

\[g_t = g_{\min}\operatorname{Sigmoid}(e^{A_h}z_t) \in (g_{\min}, 0), \qquad g_{\min} = -5\]

于是每步至少保留 $e^{-5} \approx 0.0067$，16 token 的小 tile 上累积 log-decay 落在 $(-80, 0)$，缩放因子最大 $e^{80} \approx 5.5\times10^{34}$，稳稳在 BF16 的 $3.4\times10^{38}$ 之内。收益很实在：Kimi Linear 的对角 tile 必须走显式的 position-pair 计算，这正是 chunk 内的主要瓶颈；范围有界之后，包括对角在内的每个 tile 都变成稠密的 Tensor Core 矩阵乘。一个数值稳定性的下界，换来一次 kernel 重写。

2. 全秩输出门。 低秩门换成随输入变化的全秩投影，每个 token 可以逐通道决定放多少递推读出的内容出去：

\[y_t=\mathbf{W}_o\left[\operatorname{Sigmoid}(\mathbf{W}_g x_t) \odot \operatorname{RMSNorm}(\tilde{o}_t)\right]\]

KDA 不单独工作：和 Gated MLA 3:1 混合

定容状态天生有损，所以 K3 周期性插入精确注意力。每个 block 是 3 层 KDA + 1 层 Gated MLA，backbone 末尾再补一层 MLA：

$23 \times (3\,\text{KDA} + 1\,\text{MLA}) + 1\,\text{MLA} = 93$ 层，也就是 69 层 KDA + 24 层 MLA。

只有这 24 层带着随序列增长的 cache —— 对比 K2 的 61 层全注意力。MLA 层同样是 NoPE，并带上同样的全秩输出门。

分工很清楚：69 层 KDA 用固定的 $d_k \times d_v$ 状态提供偏近期、带位置感的混合，单 token 开销恒定；24 层 MLA 留一份随 $T$ 增长的 latent cache，买回有限状态装不下的东西 —— 对任意早先 token 的精确访问。

在 Kimi Linear 的规模上，这种混合把 KV cache 用量最多降了 75%，1M 上下文下 decoding 吞吐最高 6 倍，同时在相同配方下质量还优于全 MLA。

状态很小 —— 但它是串行的

K3 在 KDA 上的工程，基本都能由一句话推出来：状态搬运便宜，但没法跳着算。

FlashKDA —— 基于 CUTLASS 的 chunkwise kernel，把 chunk 内计算和跨 chunk 的状态传递重叠起来，串行交接时 SM 不空转；它同时服务 prefill，是 flash-linear-attention 的一个后端。
KDA Context Parallelism（KCP） —— 最有意思的一块。普通线性注意力是纯加法，各 rank 从 $\mathbf{S}=\mathbf{0}$ 出发再求和就行。KDA 不行：$\mathbf{S}_t = \mathbf{M}_t\mathbf{S}_{t-1} + \beta_tk_tv_t^\top$，其中 $\mathbf{M}_t = (\mathbf{I}-\beta_tk_tk_t^\top)\operatorname{Diag}(\alpha_t)$，入口状态是被变换，不只是被加。于是每个 rank 本地算两样东西 —— 本段的累积转移 $\mathbf{M}$，以及本地 token 从零生成的状态 —— 一次 all-gather 加一次前缀扫描就能精确拼起来。通信量与上下文长度无关，这才让 1M token 的训练付得起。
Prefix cache —— KDA 状态 checkpoint 按 512 token 边界写入，和 MLA KV cache 共用同一套分页池；两边都能在同一边界恢复，这段前缀才可复用。
投机解码 —— 状态原地更新，草稿被拒绝没法回滚。K3 改成缓存草稿那份小得多的投影输入，在片上重放已接受的前缀。

KDA 记住这五点

一个定容矩阵当联想记忆用，读取就是 $\mathbf{S}^\top q$。
channel-wise 衰减 —— 每个 key 通道自己挑遗忘速率，顺带编码位置（所以 NoPE，所以 1M 扩展不用改任何位置编码）。
delta rule —— 写之前先擦掉这个 key 上的旧值；本质是一步在线梯度下降，$\beta_t$ 就是学习率。
chunkwise 形式 —— 同一个递推的精确矩阵乘写法；K3 的有界衰减把最后一块难啃的对角 tile 也推上了 Tensor Core。
混合是设计的一部分 —— 和 Gated MLA 按 3:1 交错，因为不该指望有限状态做精确召回。

第二部分 —— Attention Residuals

KDA 解决序列这条轴，AttnRes（arXiv:2603.15031）把同一套论证搬到深度这条轴。

问题

PreNorm residual 看起来无害：

\[h_l = h_{l-1} + f_l(h_{l-1}) = h_0 + \sum_{i=1}^{l} f_i(h_{i-1})\]

前面每一层都以 权重 1 加进去。深度方向像 RNN：历史被压成一个向量。hidden 幅度随深度涨，新层占的份额越来越小 —— 这就是 PreNorm dilution。早期信息没法按内容再取回来。

序列建模当年也是这个瓶颈，softmax attention 换掉了 RNN。AttnRes 对深度做同一件事：标准 residual 是深度方向的线性注意力，AttnRes 是深度方向的 softmax 注意力。

三种 residual

Standard。 每层只看见 $h_{l-1}$，混合权重固定，层间只传一个 hidden —— 也就是下面的 $N = 1$。

Full AttnRes。 每一层的 output 都变成 key/value，每层自己去挑：

\[h_l = \sum_{i=0}^{l-1} \alpha_{i \to l}\, v_i, \qquad \alpha_{i \to l} = \mathrm{softmax}_i\big(w_l^\top \mathrm{RMSNorm}(k_i)\big)\]

$w_l \in \mathbb{R}^d$：每层一个 学出来的 pseudo-query，和本层 forward 解耦，所以混合随内容变
key 上的 RMSNorm 防止幅度大的层独占权重；query 零初始化，开训接近均匀平均，不会一上来就炸
计算 $O(L^2 d)$，存储 $O(Ld)$。真正的痛点是 pipeline 并行下每层 output 都要跨 stage 传

Block AttnRes（真正上生产的那个）。把 $L$ 层切成 $N$ 个 block：block 内还是普通 residual，压成一个 block 向量；block 间只对 $N$ 个摘要 + embedding 做 Full AttnRes，当前没走完的 block 再多给一个 partial sum。通信从 $O(Ld)$ 降到 $O(Nd)$。$N = L$ 是 Full，$N = 1$ 是 Standard，经验上 $N \approx 8$ 就能拿回 Full 的大部分收益。Kimi Linear 48B 每 block 6 层 → 9 个 block + embedding = 10 个 depth source，训练开销 < 4%，decode 延迟 < 2%。

实验

Scaling law。 五个规模，每档 Baseline / Block（$N=8$）/ Full，超参一律按 baseline 选 —— 对 AttnRes 更苛刻。AttnRes 整条 compute 曲线 loss 都更低。最大档：Baseline 1.719，Block 1.693，Full 1.692；在 5.6 PFLOP/s-days 上，baseline 大约要多 25% 算力 才追平 Block。Full 是上限，Block 是「几乎一样准，而且真能训」。

48B / 3B 激活，1.4T token。 val loss 全程更低，decay 阶段差距拉大。baseline 的 output magnitude 随深度单调涨，Block 在 block 边界被「重置」；梯度也更均匀，softmax 让各 depth source 相互竞争，而不是把大梯度全砸在最浅几层。

	Baseline	AttnRes
MMLU	73.5	74.6
GPQA-Diamond	36.9	44.4（+7.5）
Math	53.5	57.1（+3.6）
HumanEval	59.1	62.2（+3.1）
C-Eval	79.6	82.5

知识类小涨，多步推理和代码涨最多 —— 和「后面的层能按需把前面的表示捞回来」一致。

16 层消融（loss，越小越好）：Baseline PreNorm 1.766，DenseFormer 1.767，mHC 1.747，Full AttnRes 1.737，固定混合 1.749，用 sigmoid 代替 softmax 1.741，去掉 RMSNorm 1.743，Block $S=4$ 1.746。固定混合明显差于学出来的 softmax —— 内容相关的深度选择是真的在干活。

小结

K3 把深度加上去了（K2 是 61 层，K3 是 93 层）。权重全是 1 的 residual 在这条轴上开始像 RNN，就像不断增长的 KV cache 在序列这条轴上是错的答案。于是两条轴用同一套办法：用有选择、随内容变的检索，取代均匀累加 —— 序列上是 KDA，深度上是 AttnRes。再加上宽度方向的 Stable LatentMoE，这就是报告里相对 K2 约 2.5 倍 scaling 效率提升的来源。

论文：Kimi K3 · Kimi Linear · Attention Residuals。代码：flash-linear-attention。

WIP-Rec System

Sat, 14 Oct 2023 00:00:00 +0000

Guild to Build a Great Rec System

I have worked at TikTok for two year and half in video recommendation system. TikTok has one of the best recommendation system in the world. I have learned a lot from my colleagues and I want to share my knowledge with you. I want to break down this series in 11 posts:

Major Component in Rec System
Modeling Target
Training Data
Loss Optimization
Feature Engineering
Hyperparameter Tuning
Model Structure
Reinforcement Learning in Rec System
Federated Learning in Rec System
AutoML and Application in Rec System
Other

Today we are going to discuss our first topic: Major Component in Rec System.

Major Component in Rec System

In today’s industrial rec system, there are four major components: Recall, Rough Ranking, Fine Ranking, and Mix-Ranker. This design are used by most of the companies in recommendation and ranking system. Of course, some adjustment are made to fit the specific business needs. For example, TikTok user publish millions of videos and our FYP has to select 8 videos out of the pool in real time, thus computation cost and latency is a major concern. However, for some retail E-commerce such as DoorDash or Samsclub.com, the number of viable products are much smaller thus, thus the candidate pool is much smaller, so a rough sort might not be necessary. In this case, a recall + Fine will do the trick just fine.

In this post, I will give introduction to each component, and discuss some design concerns.

Recall

The goal for recall is to select a small subset of candidates from a large pool. This is usually achieved through Multi-Way Recall. Multi-Way Recall builds multiple recall models or rules to select candidates. Some commonly used model targets or rules are:

Recall recent videos
Recall popular videos
Recall Regional Content
Embedding Based Recall
- I2I: Item 2 Item
- U2I: User 2 Item
Approximate Nearest Neighbor Recall

Reference

https://zhuanlan.zhihu.com/p/388603950

Stanford - CS224U, Natural Language Understanding

Mon, 27 Jul 2020 00:00:00 +0000

CS224U: Natural Language Understanding is the first course I took in the Stanford AI Certification Program. This blog is a short summary of the material and what I took away from it.

The instructors for recent terms have been Prof. Christopher Potts and Bill MacCartney, and this year was no exception. Both are major figures in the field; Apple’s Siri is one of the products Bill led. Because of COVID-19, the course went fully online for remote students and Stanford on-campus students alike, taught over Zoom. Combined enrollment was around 120+. I teamed with a Stanford undergrad on the homework. The final project was me, that undergrad, and a CS PhD at Northwestern. The final project became optional because of the pandemic.

Course website

2019 course videos

Lecture 1, Introduction and course overview

Lecture 1 slides

NLP VS. NLU

In the first lecture the instructors gave a short intro and then contrasted NLP (Natural Language Processing) with NLU (Natural Language Understanding). My take is that NLU is a sub-area of NLP: NLU focuses on machine understanding of language, NLP on processing it. The course covers Grounded Language Understanding, a classic NLU problem. They used the Pragmatic Color Describer to introduce Grounding. When you describe a color in language, the words you pick depend on the surrounding context. Enter our orange cats:

When describing the two orange cats we can distinguish them as the lighter orange cat vs the darker one. If the comparison cat is not orange, we do not need to specify the shade; we just say orange cat. For example:

When we use contrastive language we assume the listener shares our understanding of color shades. That shared understanding is Grounding. More on the Pragmatic Color Describer later.

A brief history of NLU

The instructors gave a short history of NLU

1966: Eliza
1988: Latent Semantic Analysis
January 2011: IBM Watson beats Jeopardy! Champions
October 2011: Apple Siri launches in beta
April 2014: Microsoft Cortana demoed
May 2016: Google Assistant

CS224u topics

They then listed the NLU topics for the term: eight topics, two lectures each. Plenty of material.

Vector-space models
Sentiment analysis
Relation extraction
Natural Language Inference
Grounding
Contextual word representations
Adversarial testing
Methods and metrics

The professor then sketched progress in each area and recommended SUPERINTELLIGENCE - NICK BOSTROM. I found it worthwhile: it is about the future of AI and the social issues that might follow. Little technical content, simple language. A decent English-practice read if you have time.

Finally they pointed to a few starter tutorials: setting up the course virtualenv, intro to PyTorch and NumPy, etc. Strongly recommend following the setup tutorial and cloning the course GitHub; there is a lot of useful material.

CS224U virtual environment setup tutorial

Lecture 2, Distributed word representations

This lecture covers the most basic but still central idea in NLP: word vectors. Prof. Potts started from co-occurrence matrices and went through GloVe, Word2Vec, and related methods.

Vector representations are a core idea in deep learning; the running joke is “everything can be an embedding.” In retail, every product and every store can be a high-dimensional vector. You can study product relationships the same way. Alibaba, Walmart, and Amazon have all done substantial work here. A Product2Vec paper if you want to dig in ( Studying Product Competition Using Representation Learning).

Word vector representations are now the default effective method in the field, but they rest on a deep linguistic idea. It goes back to the question: what defines the meaning of a word? The answer here is that the other words that appear with it in a sentence define its meaning.

You shall know a word by the company it keeps. - Firth (1957)

“distributional statements can cover all of the material of a language without requiring support from other types of information. - Firth (1957)

The simplest co-occurrence matrix captures meaning by counting how often words appear together.

How to build a co-occurrence matrix

Building a co-occurrence matrix has two main parameters.

Window Size: the range around a center word in which co-occurrence is counted
Scaling: used to adjust the co-occurrence values

Example from a line of Shakespeare: “For thy sweet love remembered such wealth brings That then I scorn to change my state with kings.”

Choice of these two parameters follows this logic

Larger window size and smoother scaling capture more semantic information
Smaller window size and heavier scaling capture more syntactic information

Compute the co-occurrence matrix over a huge collection of sentences and documents; the counts of how often each word appears with every other word give the simplest word vectors. Each row in the figure below can be treated as a word embedding. Raw counts are not enough; next we look at statistical methods that improve vector quality. Window-based co-occurrence has an obvious flaw: words like the, and, or, is co-occur with almost everything but carry little meaning. Later we cover reweighting methods such as PMI and PPMI that mitigate this.

How to compare two vectors

Once words are vectors we need systematic ways to compare them. The course covers:

Euclidean distance
Euclidean distance with L-2
Cosine distance
Matching-based methods
KL divergence

Euclidean distance

Distance between n-dimensional vectors u, v:

\[{euclidean(u, v)=\sqrt{\sum_{i=1}^{n}{|u_i - v_i|^2}}}\]

Euclidean distance is the straight-line distance between two points. If two vectors point the same way but have different magnitudes, the Euclidean distance is still large. For word vectors direction usually matters more than magnitude; vectors that point the same way tend to have similar meaning, so Euclidean distance is not a good comparison method.

Euclidean distance with L-2 norm

If we L2-normalize the vectors, every vector has magnitude 1. Euclidean distance then ignores magnitude and only reflects direction.

Cosine distance

Cosine distance measures only the angle between vectors; magnitude is irrelevant. It is one of the most common ways to compare word vectors in NLP.

\[{cosine(u, v)=1 - {\frac{\sum_{i=1}^{n}{u_i \times v_i}}{||u||_2 \times ||v||_2}}}\]

One thing worth noting: ranking by Euclidean distance after L2 normalization is the same as ranking by cosine distance. The numeric values differ, but the order when comparing multiple vectors is identical. In word-vector applications they can be treated as equivalent.

Matching-based methods

Matching-based methods are not used much in NLU. The course only mentioned them briefly; I just list the formulas here.

KL divergence

KL divergence is a way to compare two probability distributions. For a deeper dive see KL divergence.

\[D(p || q) = \sum_{i=1}^{n} {p_ilog(\frac{p_i}{q_i})}\]

In NLU, to compare two word vectors we first turn each vector into a probability distribution that sums to 1, then apply KL divergence. The lecture gave this example. A, B, and C have already been turned into distributions that sum to 1. The method only works if the vectors can be interpreted as distributions; negative values make it inapplicable.

Normalizing the co-occurrence matrix

We have covered how to build a co-occurrence matrix and how it yields word vectors. Frequency-based vectors have a serious problem: raw frequency is not very meaningful for language understanding. A simple example: 妖魔鬼怪 vs 魑魅魍魉. The two idioms mean essentially the same thing, but 妖魔鬼怪 appears far more often in student essays because 魑魅魍魉 is hard to write and you lose points if you get it wrong. Frequency is therefore a poor signal; we can reduce its effect with normalization. L2 normalization and converting the vector to a probability distribution are two options. Next is a common method: Observed/Expected.

Observed/Expected is

How should we interpret the expected count? It is the expected frequency if the row word and column word were independent. In the figure, keep appears 60 times and tabs 21 times. If they were independent, the probability they co-occur would be P(keep) * P(tabs) = 60/101 * 21/101. Observed/Expected compares the actual count with that independence assumption.

From Observed/Expected we also get Pointwise Mutual Information (PMI) and Positive PMI (PPMI). PMI is log(Observed/Expected); PPMI sets every negative PMI (from the log) to 0. Note that log(0) is undefined in math, but in this setting we take log(0)=0.

TF-IDF

TF: Term frequency: P(word | document) IDF: Inverse document frequency (IDF)

Detailed explanation of IDF

The professor also mentioned t-test weighting, pairwise distance matrices, etc., but they are not used much in this course so I will not expand on them.

Dimensionality reduction

Co-occurrence matrices are huge; a word representation can easily have thousands or tens of thousands of dimensions. Using them raw does not extract meaning well, so we need dimensionality reduction. The course covers four families of methods:

Latent Semantic Analysis
- Singular value decomposition (SVD)
- Principal Components Analysis (PCA)
- Non-negative Matrix Factorization (NMF)
- Probabilistic LSA (PLSA; Hofmann 1999)
- Latent Dirichlet Allocation (LDA; Blei et al. 2003)
- t-SNE (van der Maaten and Hinton 2008)
Autoencoders
GloVe
Word2Vec

The methods under LSA are less common in NLP today, and SVD and PCA are basic dimensionality-reduction techniques, so I will not go into them. We focus on Autoencoder, GloVe, and Word2Vec.

Autoencoder

An autoencoder is a deep neural architecture used for dimensionality reduction. The lecture showed a simple autoencoder:

The main difference from a usual network is that a standard net needs an input X and a target Y. Training an autoencoder only needs X, because the input and the target are both X. The objective is to make the output X_hat equal the input X. The code below is the autoencoder in the figure; input_dim_ and output_dim_ are equal (the dimension of X).

import torch
import torch.nn as nn
def define_graph(self):
    return nn.Sequential(
        nn.Linear(self.input_dim_, self.hidden_dim),
        nn.Tanh(),
        nn.Linear(self.hidden_dim, self.output_dim_))

GloVe

GloVe is a landmark result in NLP; almost every NLP course covers it. Here is a short look at how it works. The GloVe objective is to learn word vectors whose dot products are proportional to their co-occurrence probabilities.

Before GloVe there were two main families of word-vector methods: 1. Matrix Factorization Methods based on the co-occurrence matrix (everything we discussed above). 2. Shallow window-based methods that use a local context window. That family includes the skip-gram model and the continuous bag-of-words model. Both families have clear weaknesses. In the first, high-frequency words such as the, is, and do not represent meaning well. The second family only sees a local window, so it cannot fully exploit a large corpus.

CS224U: Natural Language Understanding 是我在Stanford AI Certification Program 里的第一节课。这篇Blog对知识点和学习心得做一个简单的总结。

最近几学期这门课的教授一直是Prof. Christopher Potts和Bill MacCartney，今年也不例外。两位都是真*巨佬，苹果的Siri就是Bill领导的产品之一。这学期因为COVID-19疫情的原因，无论是我们Online的学生还是Stanford的在校学生，课程都变成Online了，通过Zoom授课。这学期Online＋在校学生大概有120+人。我这学期作业是跟一个Stanford本科的白人小哥一组，final project 是我跟白人小哥哥加一个在 Northwestern 的CS PhD。然而个final project因为疫情也变成optional的了。

课程官网

2019年课程视频

Lecture 1, Introduction and course overview

Lecture 1 slides

NLP VS. NLU

第一节课，老师简单做了一些自我介绍，然后介绍了一个NLP(Natural Language Processing) 和NLU(Natural Language Understanding)的不同。我的理解是NLU是NLP下的一个小分支， NLU强调的机器对语言的理解，而NLP则强调对语言的处理。门课会涉及到Grounded Language Understanding，是一个典型的NLU问题。这门课用 Pragmatic Color Describer 来简单介绍了Grounding的概念。当我用语言描述一个颜色的时候，我们所用的描述性词汇取决于周围的环境。有请我们的橘猫登场:

我们描述两只橘猫的时候，就可以用浅一点的那个只橘猫VS深色的橘猫来作区分。然而当对比的猫不是橘色的话，我们就需要描述橘色的深浅，只需要说橘猫即可。比如这样:

当我们使用对比描述语言的时候，我们假设说话对象对颜色深浅有跟我们一样的理解。这个理解就是Grounding。有关Pragmatic Color Describer的具体内容后面会详细讲。

A brief history of NLU

这部分老师简单介绍了一下NLU的发展历史

1966: Eliza
1988: Latent Semantic Analysis
2011年1月: IBM Watson beats Jeopardy! Champions
2011年10月: Apple Siri launches in beta
2014年4月: Microsoft Cortana demoed
2016年5月: Google Assistant

CS224u topics

随后介绍了一下这学期会cover的NLU topics，八个topic，每个topic两个lecture。内容还是相当丰富的。

Vector-space models
Sentiment analysis
Relation extraction
Natural Language Inference
Grounding
Contextual word representations
Adversarial testing
Methods and metrics

随后教授介绍了各个方向的发展，并且介绍了一个本书SUPERINTELLIGENCE - NICK BOSTROM, 我读着觉得不错，主要讲的就是对AI的未来发展，和可能带来的相应的社会问题。没什么技术内容，用词简单，有时间的朋友可以读读，当做练习英文也不错。

最后教授介绍了一下几个入门教程，包括设置课程的虚拟环境，PyTorch和Numpy的入门，等。强烈建议跟着教程设置一下虚拟环境，并克隆这门课的Github，有很多有用的东西。

CS224U虚拟环境设置教程

Lecture 2, Distributed word representations

这节课讲的就是NLP里面最基础，但相当重要的Word Vector。 Prof. Potts从最开始的Co-occurrence矩阵，讲到到GloVe和Word2Vec等方法。

Vector representation的思想，是深度学习里的一个核心思想，我们就经常开玩笑说，“万物皆可Embedding”。比如在零售业，每一个商品，每一个门店，都可以用一个高维向量来表示。同理我们也可以通过这种向量来研究商品之间的关系。中国零售大佬阿里巴巴，和美国零售Walmart，Amazon都在这方面有深入的研究，这里附上一篇有关Product2Vec的论文，有兴趣的同学可以读一下( Studying Product Competition Using Representation Learning)。

现在Word vector representation已经是学术界默认的有效方法，然而其背后有着对语言学深刻的理解。这要追溯到这个语言学问题：什么定义了一个词的意思？ 在这里的理解是，与这个词一同出现在一个句子的其他词定义了这个词的意义。

You shall know a word by the company it keeps. - Firth (1957)

“distributional statements can cover all of the material of a language without requiring support from other types of information. - Firth (1957)

最简单的Co-occurrence矩阵通过计算词汇同时出现的次数来反应词义信息。

如何构建Co-occurrence矩阵

Co-occurrence矩阵的构建有两个主要参数。

Window Size: 以一个单词为中心，前后考虑co-occurrence的范围
Scaling: 用于调整co-occurrence的数值

这里用莎士比亚的一句诗来举例: “For thy sweet love remembered such wealth brings That then I scorn to change my state with kings.”

这两个参数的选取，主要遵循以下逻辑

更大的window size和更平滑的scaling能获取更多的semantic information(语义信息)
更小的window size和更多的scaling能获取更多的syntactic information(句子结构信息)

用海量的句子，文章来计算co-occurrence矩阵，每个词与其他词一起出现的次数就能构成最简单的word vector。下图的每一行都可以看做一个word embedding。当然简单的计数并不能达到我们想要的效果，接下来会讲如何通过统计方法提高word vector的质量。然而基于窗口构建的co-occurrence有一个明显的缺点，像the, and, or, is等词汇跟其他词co-occurrence的频率非常高，但是包含词义的信息却非常少。后面会讲到一些reweighting的方法，比如PMI，PPMI，来改善这个情况。

如何比较两个向量

单词变成向量之后，我们需要系统的比较两个向量的方法。这门课介绍了以下几个方法：

Euclidean distance
Euclidean distance with L-2
Cosine distance
Matching-based methods
KL divergence

Euclidean distance(欧几里得距离)

n维向量u, v之间的距离:

\[{euclidean(u, v)=\sqrt{\sum_{i=1}^{n}{|u_i - v_i|^2}}}\]

Euclidean distance描述的是两点之间的直线距离。如果两个向量方向一致，大小不同的话，欧式距离也会非常大。通常word vector的方向比大小更重要，方向一致的word vector通常有着相近的词意，所以Euclidean distance不是比较word vector的合适方法。

Euclidean distance with L-2 norm

如果我们对向量进行归一化，那么向量大小都会变为1，这是用在用Euclidean distance就能排除向量大小的影响，从而只考虑向量方向的影响。

Cosine distance

Cosine distance只测量向量角度，向量大小完全无关。Cosine distance是NLP界最常用的word vector比较方法之一。

\[{cosine(u, v)=1 - {\frac{\sum_{i=1}^{n}{u_i \times v_i}}{||u||_2 \times ||v||_2}}}\]

这里有点值得注意的是, euclidean distance + L-2 Normalization的排序是等同于cosine distance的。这两种方法的数值不完全相同，但是用来比较多个向量距离的时候排序是一样的。在word vector应用领域，可以认为是等价的。

Matching-based methods

Matching based在NLU领域用的并不多，这门课只是简单提了一下，我在这就简单列一下公式。

KL divergence

KL divergence本身是一种比较两个概率分部的方法。想具体了解KL divergence的同学可以看这篇KL divergence。

\[D(p || q) = \sum_{i=1}^{n} {p_ilog(\frac{p_i}{q_i})}\]

在NLU的运用中，如果我们要比较两个word vector我们首先要吧word vector变成一个和为1的概率分布，然后用KL divergence来比较距离。课上给出了这么一个例子。A, B, C都是变成和为1的概率分布了。使用KL的前提是，向量能被理解为概率分布，如果向量里面有负值，那么这个方法就不适用。

标准化Co-occurrence矩阵

我们已经简单讲了如何构建co-occurrence矩阵，并且通过co-occurrence矩阵引申出word vector的概念。然而靠词出现的频率产生的word vector有一个很严重的问题，就是词出现频率在语言理解里的意义并不大。举一个很简单的例子，妖魔鬼怪VS魑魅魍魉。这两个成语的意思基本一样，但是大家写作文的时候可能妖魔鬼怪出现的频率会高很多，因为魑魅魍魉实在太难写了，写错还要扣分不是。所以频率并不是一个非常好的方法，我们可以通过标准化的方法来减弱频率的影响。L-2 normalization和把向量转化为概率的方法。下面会讲一个常用的方法Observe/Expected。

Observed/Expected公式如下

这里该怎么理解这这个expected的计算呢？expected可以理解为在行和列的词都是独立事件的时候，这个词出现频率的期望值。在这张图给出的例子中，keep这个词出现了60次，tabs出现了21次。如果这两个词出现是独立事件的话，那么keep, tabs一起出现的概率应该是P(keep) * P(tabs) = 60/101 * 21/101。Observed/Expected是在比较实际情况和假设为独立事件的时候的不同。

在Observed/Expected的基础上，我们还可以得到Pointwise Mutual Information(PMI)和Positive PMI(PPMI)。PMI是log(Observed/Expected)，而PPMI则是把所有PMI中因为log产生的负数改为0。值得注意的是数学上log(0)是没有定义的，但是在这个应用场景下log(0)=0。

TF-IDF

TF: Term frequency: P(word | document) IDF: Inverse document frequency (IDF)

IDF详细解释

教授还提到了t-test weighting，pairwise distance matrices等，但这些在这门课中用到的并不多，就不在这展开了。

Dimensionality reduction

Co-occurrence矩阵非常大，一个word representation可能会高达上千，上万维度，如果直接用的话并不能很好的提取词义信息，所以这里要用到降维的方法。这门课介绍了四种方法:

Latent Semantic Analysis
- Singular value decomposition (SVD)
- Principal Components Analysis (PCA)
- Non-negative Matrix Factorization (NMF)
- Probabilistic LSA (PLSA; Hofmann 1999)
- Latent Dirichlet Allocation (LDA; Blei et al. 2003)
- t-SNE (van der Maaten and Hinton 2008)
Autoencoders
GloVe
Word2Vec

这里LSA下面的方法在如今的NLP领域已经不那么常用了，而且SVD和PCA是比较基础的降维方法，这里就不展开讲。我们主要看看Autoencoder, GloVe, 和Word2Vec。

Autoencoder

Autoencoder是一类用来降维结构的深度神经网络结构。老师在lecture中给出了一个简单的Autoencoder模型的结构

Autoencoder结构与其他神经网络不一样的地方主要是，训练寻常的神经网络，我们需要一个输入X，和输出Y。但是训练Autoencoder，我们只需要输入X即可，因为这里的输入和输出都是X。优化的目的是让输出层的X_hat等于输入层的X。以下就是图中描述的Autoencoder，这里的input_dim_和output_dim_是相等的都是X向量的维度。

import torch
import torch.nn as nn
def define_graph(self):
    return nn.Sequential(
        nn.Linear(self.input_dim_, self.hidden_dim),
        nn.Tanh(),
        nn.Linear(self.hidden_dim, self.output_dim_))

GloVe

GloVe是NLP界一个里程碑级别的成就，基本上所有NLP相关的课程都会讲到这个，我们就来简单看看GloVe就是怎么实现的。GloVe的优化目标是学习一组word vector使其点乘和他们co-occurrence的概率成正比。

在GloVe之前，生成word vector两大类方法是: 1. 基于co-occurrence矩阵的Matrix Factorization Methods，咱之前讨论的都是这一类方法。2: 基于local context window的Shallow Window-Based Methods。这类方法包括skip-gram model和continuous bag-of-words model。然而这两类方法都有明显的缺陷。第一类中有些高频的词汇，例如the, is, and等并不能很好的代表词意。第二类方法是我是基于local window，所以无法完全利用庞大的语料数据。

Stanford - CS224n, NLP with Deep learning

Wed, 03 Apr 2019 00:00:00 +0000

Lecture 1, Introduction and Word Vectors

Course Website

Lecture Video

Lecture slids

What do we hope to learn?

RNN
Attention
PyTorch
Word meaning, dependency parsing, machine translation, question answering

Traditional NLP use one-hot encoding for words. The problem is that the one-hot matrix can be very big. There is no natural notion of similarity of one-hot vectors because they are orthogonal.

We can use a word similarity tabl, yet the table will be very large (500,000 words X 500,000 words)!

Denotational Semantics VS. Distributional Semantics

Denotational Semantics: signifier (symbol) ⟺ signified (idea or thing)
Distributional Semantics: A word’s meaning is given by the words that frequently appear close-by

Word2vec

Idea:

We have a large corpus of text
Every word in a fixed vocabulary is represented by a vector
Go through each position t in the text, which has a center word c and context (“outside”) words o
Use the similarity of the word vectors for c and o to calculate the probability of o given c (or vice versa)
Keep adjusting the word vectors to maximize this probability

Example windows and process for computing ${P(w_{t+j} | w_t)}$

We want to maximize the overall log likelihood of getting a correct prediction of context words within a window of fixed size m given center word $w_j$. The objective function is:

\[Likelihood = L(\theta)=\prod_{t=1}^{T} \prod_{-m \leq j \leq m \atop j \neq 0} P\left(w_{t+j} | w_{t} ; \theta\right)\]

Then the objective function is the average negative log likelihood.

\[{ J(\theta)=-\frac{1}{T} \log L(\theta)=-\frac{1}{T} \sum_{t=1}^{T} \sum_{-m \leq j \leq m \atop j \neq 0} \log P\left(w_{t+j} | w_{t} ; \theta\right) }\]

Question: How to calculate ${P(w_{t+j} | w_t)}$ ? Answer: Use two vectors per word $w$

$v_w$ when w is center word
$u_w$ when w is a context word

Then for a center word c and a context word o:

\[P(o | c)=\frac{\exp \left(u_{o}^{T} v_{c}\right)}{\sum_{w \in V} \exp \left(u_{w}^{T} v_{c}\right)}\]

HW1

Got problem using virtual environment. The nltk can not find reuters. Deactivate virtual environment solved this. Link to HW1

Lecture 2, Word Vectors and Word Senses

idf-weighting

Lecture 5, Dependency Parsing

After two lectures of mathematical background in deep learning, we can finally started to learn some NLP stuff.

1. Two views of linguistic structure:

Phrase structure: organizes words into nested constituents.

Can represent the grammar with CFG rules

Constituency = phrase structure grammar = context-free grammars (CFGs)

NP: noun phrase

PP: preposition

Det: Determinant

Some of the grammar rules
- NP -> Det N (the cat, a dog)
- NP -> Det (Adj) N
- NP -> Det(Adj) N PP
- PP -> PP NP

Dependency structure

Dependency structure shows which words depend on (modify or are arguments of ) which other words.

Why do we need sentence structure? We, human, need a certain sentence structure in order to convey complex meaning.

Example of prepositional phrase attachment ambiguity

This is an example of why NLP is hard. We human are communicate through languages, but language alone will not do it. We talk to people with the assumption that the other person have similar knowledge as ourself. This is “Common Sense”.

Here is an example of ambiguous sentences: San Jose cops kill man with knife

There are two meaning for this one sentences. First, cops stabbed and killed a man using knife. Second, cops killed a man who is holding a knife. According to our common sense, we tend to believe that the second is true. The meaning of a sentence is depended on the dependency structure of words. This is also why human languages are very different from programming languages.

Catalan numbers: ${ C_n = (2n)! / [(n+1)!n!]}$
An exponentially growing series, which arises in many tree-like contexts. CS228: Prob. Graphic Model.

Coordination Scope Ambiguity

Shuttle veteran and longtime NASA executive Fred Gregory appointed to board.

Either Fred Gregory is a shuttle veteran and a longtime NASA executive, or two people where one is shuttle veteran and the other is NASA executive.

Adjectival Modifier Ambiguity

This is a graduate level class, so prof. actually showed this example in his lecture!!!

Student get first hand job experience

Verb Phrase (VP) attachment ambiguity

The rise of annotated data, Universal Dependencies Treebands

Build a tree band seems a lot slower and less useful than building a grammar. But treebank have following advantages:

Reusability of the labor
- Many parsers, part-of-speech taggers, etc. can be build on it
- Valuable resource for linguistics
Broad coverage, not just a few intuitions
Frequencies and distributional information
A way to evaluate systems

Dependency can cross over: I will give a talk tomorrow on bootstrapping

on bootstrapping cross over talk.

Lecture 7, Vanishing Gradients, Fancy RNNs

Topics: Vanishing (exploding) gradient, LSTM, GRU, bidirectional, Multi-layers

Lecture 1, 引言与词向量

课程网站

讲座视频

讲座幻灯片

我们希望学到什么？

RNN
Attention
PyTorch
词义、依存句法分析、机器翻译、问答

传统 NLP 使用 one-hot 编码表示词。问题是 one-hot 矩阵会非常大。 one-hot 向量之间没有自然的相似性概念，因为它们是正交的。

我们可以用词相似度表，但这个表会非常大（500,000 词 X 500,000 词）！

指称语义 VS. 分布语义

指称语义：能指（符号）⟺ 所指（概念或事物）
分布语义：一个词的含义由其经常出现在附近的词给出

Word2vec

思路：

我们有一个大型文本语料库
固定词表中的每个词用一个向量表示
遍历文本中的每个位置 t，该位置有中心词 c 和上下文（“外部”）词 o
用 c 和 o 的 词向量相似度 来计算给定 c 时 o 的概率（或反过来）
不断调整词向量 以最大化这个概率

计算 ${P(w_{t+j} | w_t)}$ 的窗口示例与过程

我们希望最大化在给定中心词 $w_j$ 时，固定窗口大小 m 内正确预测上下文词的总体对数似然。目标函数 是：

\[Likelihood = L(\theta)=\prod_{t=1}^{T} \prod_{-m \leq j \leq m \atop j \neq 0} P\left(w_{t+j} | w_{t} ; \theta\right)\]

然后 目标函数 是平均负对数似然。

\[{ J(\theta)=-\frac{1}{T} \log L(\theta)=-\frac{1}{T} \sum_{t=1}^{T} \sum_{-m \leq j \leq m \atop j \neq 0} \log P\left(w_{t+j} | w_{t} ; \theta\right) }\]

问题：如何计算 ${P(w_{t+j} | w_t)}$ ？答案：每个词 $w$ 使用两个向量

$v_w$ 当 w 是中心词时
$u_w$ 当 w 是上下文词时

然后对于中心词 c 和上下文词 o：

\[P(o | c)=\frac{\exp \left(u_{o}^{T} v_{c}\right)}{\sum_{w \in V} \exp \left(u_{w}^{T} v_{c}\right)}\]

HW1

使用虚拟环境时遇到问题。nltk 找不到 reuters。停用虚拟环境后解决了。 HW1 链接

Lecture 2, 词向量与词义

idf-weighting

Lecture 5, 依存句法分析

经过两讲深度学习的数学背景后，我们终于可以开始学一些 NLP 内容了。

1. 语言结构的两种观点：

短语结构：将词组织成嵌套的成分。

可以用 CFG 规则表示语法

Constituency = 短语结构语法 = 上下文无关文法 (CFGs)

NP: 名词短语

PP: 介词

Det: 限定词

部分语法规则
- NP -> Det N (the cat, a dog)
- NP -> Det (Adj) N
- NP -> Det(Adj) N PP
- PP -> PP NP

依存结构

依存结构显示哪些词依赖于（修饰或作为论元）哪些其他词。

为什么需要句子结构？我们人类需要一定的句子结构来传达复杂含义。

介词短语附着歧义的例子

这是 NLP 困难的一个例子。我们人类通过语言交流，但仅靠语言不够。我们与人交谈时假设对方拥有与我们相似的知识。这就是“常识”。

下面是一个歧义句子的例子： San Jose cops kill man with knife

这句话有两种含义。第一，警察用刀刺死了一个人。第二，警察杀了一个拿着刀的人。根据我们的常识，我们倾向于相信第二种是真的。句子的含义取决于词的依存结构。这也是人类语言与编程语言非常不同的原因。

Catalan 数： ${ C_n = (2n)! / [(n+1)!n!]}$
一个指数增长的序列，出现在许多树状上下文中。CS228: Prob. Graphic Model.

并列范围歧义

Shuttle veteran and longtime NASA executive Fred Gregory appointed to board.

要么 Fred Gregory 既是航天飞机老兵又是长期 NASA 高管，要么是两个人，一个是航天飞机老兵，另一个是 NASA 高管。

形容词修饰歧义

这是研究生课程，所以教授在讲座中真的展示了这个例子！！！

Student get first hand job experience

动词短语 (VP) 附着歧义

标注数据的兴起，Universal Dependencies 树库

构建树库看起来比构建语法慢得多且用处更小。但树库有以下优点：

劳动的可重用性
- 可以在其上构建许多解析器、词性标注器等
- 对语言学来说是宝贵资源
覆盖面广，不仅仅是一些直觉
频率和分布信息
一种评估系统的方法

依存可以交叉： I will give a talk tomorrow on bootstrapping

on bootstrapping 跨越了 talk。

Lecture 7, 梯度消失，花式 RNN

主题：梯度消失（爆炸）、LSTM、GRU、双向、多层

Python for Absolute Newbies

Wed, 03 Apr 2019 00:00:00 +0000

这篇文章会从安装 Python3.7 开始讲起。接下来的主题会渐渐集中讲一些 Data Science 的一些应用和建模方法。使用 macOS Mojave.

Zen of Python

Python was designed as a successor to the ABC language by Guido van Rossum. Python was initially developed as a “hobby” project that would keep him occuiped during the Christmas break! I mean who would spend their Christmas writing an interpreter!

Eventually, Python become one of the top used programming language today. Python can be used for all kinds of projects from small personal project to large application, and from web development to scientific computation. Here are some famous projects used Python, according to this article

YouTube
Google search engine also used Python for its mainframe
Instagram
Reddit

Let’s start this tutorial with the Zen of Python:

Type this into the Mac terminal.

python
>>> import this

Install Python3.7

Let’s see how to install Python3.6. Mac come with default python2.7 installed. We want to use Python3.7 here.

I found the easiest way here is to install through Anaconda Distribution (Click this link to download directly!)

You can also install through python.org.

Please also install VS Code come with Anaconda. I will talk about Visual Studio Code later. If you installed through python.org, please also install Visual Studio Code separately.

Once you have Python3.7 installed, you should be able to see this in terminal

~ » python3.7
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

Congrats! You know have the latest version of Python ready!

Virtual Environment

Do use a virtual environment, PLEASE!!! Virtual environment allow you to separate your developing environment without interfering with the system.

I want you to have the best programming habit from the beginning. I learned this the hard way.

Lets try to setup virtual environment using virtualenv.

First go to home directory and create a directory Environment to store all your virtual environments

~ » cd ~
~ » mkdir Environment
~ » cd Environment

Then install virtualenv and create a new virtual environment named it python_tutorial

~ » pip install virtualenv
~ » virtualenv ~/Environment/python_tutorial

Activate python_tutorial. Now type which python3. It will return the path to the virtual environment.

~ » source ~/Environment/python_tutorial/bin/activate
~ » which python3
/Users/zhejianpeng/Environment/python_tutorial/bin/python3

Deactivate python_tutorial environment

~ » deactivate

Now every time before you start coding, remember to activate the virtual environment. It’s recommended to create a short alias of the activation command and add it to ~/.bash_profile. You don’t have to type the long version.

~ » code ~/.bash_profile
# User Define Alias, add following line to your bash_profile
alias tutorial='source ~/Environment/python_tutorial/bin/activate'

Here you need to install code command. It’s very easy. Open VS code you installed previously, and type shift + command + p. Then type

Now you have the basic environment setup!!! Let’s start coding

Hello World

这篇文章会从安装 Python3.7 开始讲起。接下来的主题会渐渐集中讲一些 Data Science 的一些应用和建模方法。使用 macOS Mojave.

Python 之禅

Python 被设计为 ABC 语言的继任者，由 Guido van Rossum 开发。 Python 最初是作为一个“业余”项目开发的，好让他在圣诞假期有事可做！谁会在圣诞节写一个解释器啊！

最终，Python 成为当今使用最广泛的编程语言之一。Python 可用于各种项目，从小个人项目到大型应用，从 Web 开发到科学计算。根据这篇文章，以下是一些使用 Python 的著名项目：

YouTube
Google 搜索引擎的主机也使用了 Python
Instagram
Reddit

我们从 Python 之禅开始这个教程：

在 Mac 终端中输入以下内容。

python
>>> import this

安装 Python3.7

看看如何安装 Python3.6。Mac 默认安装了 python2.7。我们这里要用 Python3.7。

我发现最简单的方法是通过 Anaconda Distribution 安装（点击此链接直接下载！）

也可以通过 python.org 安装。

请同时安装 Anaconda 自带的 VS Code。我稍后会讲 Visual Studio Code。如果通过 python.org 安装，请单独安装 Visual Studio Code。

安装好 Python3.7 后，终端里应该能看到这个

~ » python3.7
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

恭喜！你现在已经准备好最新版本的 Python 了！

虚拟环境

一定要用虚拟环境，拜托！！！虚拟环境可以隔离你的开发环境，不会干扰系统。

我希望你从一开始就养成最好的编程习惯。这是我吃过亏才学到的。

我们用 virtualenv 来设置虚拟环境。

首先进入主目录，创建一个 Environment 目录来存放所有虚拟环境

~ » cd ~
~ » mkdir Environment
~ » cd Environment

然后安装 virtualenv 并创建一个名为 python_tutorial 的新虚拟环境

~ » pip install virtualenv
~ » virtualenv ~/Environment/python_tutorial

激活 python_tutorial。现在输入 which python3。它会返回虚拟环境的路径。

~ » source ~/Environment/python_tutorial/bin/activate
~ » which python3
/Users/zhejianpeng/Environment/python_tutorial/bin/python3

停用 python_tutorial 环境

~ » deactivate

现在每次开始编码前，记得激活虚拟环境。建议为激活命令创建一个短别名，并添加到 ~/.bash_profile。这样就不用输入长命令了。

~ » code ~/.bash_profile
# User Define Alias, add following line to your bash_profile
alias tutorial='source ~/Environment/python_tutorial/bin/activate'

这里需要安装 code 命令。很简单。打开之前安装的 VS Code，输入 shift + command + p。然后输入

现在基本环境已经设置好了！！！开始编码吧

term	what it is	how it is produced
\(q_t,k_t\)	probe / address	ShortConv → Swish → L2Norm
\(v_t\)	payload	ShortConv → Swish
\(\beta_t \in (0,1)\)	write strength	\(\operatorname{Sigmoid}(\mathbf{W}_\beta x_t)\)
\(\alpha_t \in (0,1)^{d_k}\)	per-channel retention	low-rank logits + per-head bias, then Eq. below

mechanism	sequence work	decode cache	next token
softmax	\(O(HT^2d_k)\)	\(O(HT(d_k+d_v))\)	\(O(HT(d_k+d_v))\)
KDA	\(O(HTd_kd_v)\)	\(O(Hd_kd_v)\)	\(O(Hd_kd_v)\)

J'Log

Pre-Norm vs. Post-Norm: The Identity Path That Stabilizes LLMs

The one-line difference

Why the identity path changes backpropagation

What Xiong et al. actually established

A runnable stability stress test

Run it with JAX on CPU or GPU

Why Pre-Norm is not simply “better”

The final norm has a different job

The design rule behind the literature

Takeaway

References

一行之差

为什么 identity 路径会改变反向传播

Xiong et al. 实际建立了什么

一个可运行的稳定性压力测试

用 JAX 在 CPU 或 GPU 上运行

为什么 Pre-Norm 并不只是“更好”

Final norm 有不同的职责

文献背后的设计规则

要点

参考文献

Kimi K3: An In-Depth Look at KDA

Part 1 — Kimi Delta Attention

What KDA is replacing

The state is an associative memory

Fix 1 — forgetting, one rate per channel

Fix 2 — the delta rule, or writing by correction

KDA = both, and here is the whole layer

Run the complete tutorial on a CPU

Big-O: what becomes linear, and what does not

Why use KDA?

Chunkwise parallel form, step by step

1. Lay out one chunk

2. See what the recurrence is hiding

3. Turn relative decay into query/key scaling

4. Fold the delta-rule dependencies with the UT transform

5. Produce every output in the chunk

6. What is actually parallel?

What K3 changed relative to Kimi Linear

KDA does not work alone: 3:1 with Gated MLA

The state is small — but it is serial

What to remember about KDA

Part 2 — Attention Residuals

The problem

Three residuals

Experiments

Takeaway

第一部分 —— Kimi Delta Attention

KDA 要替掉什么

状态就是一块联想记忆

解法一 —— 遗忘，而且每个通道一个速率

解法二 —— delta rule：按误差改写

KDA = 两个都要，整层长这样

在 CPU 上运行完整教程

Big-O：哪部分变成线性，哪部分没有

为什么要用 KDA？

Chunkwise parallel form：一步一步推

1. 先把一个 chunk 排成矩阵

2. 看清递推里藏着什么

3. 把相对衰减变成 query/key 缩放

4. 用 UT transform 折叠 delta-rule 依赖

5. 一次算出 chunk 里的所有输出

6. 到底哪里并行？

相对 Kimi Linear，K3 改了什么

KDA 不单独工作：和 Gated MLA 3:1 混合

状态很小 —— 但它是串行的

KDA 记住这五点

第二部分 —— Attention Residuals

问题

三种 residual

实验

小结

WIP-Rec System

Guild to Build a Great Rec System

Major Component in Rec System

Recall

Reference

构建优秀推荐系统指南

推荐系统的主要组件

标准化Co-occurrence矩阵