Kimi KDA Attention LLM

Kimi K3: An In-Depth Look at KDA

Kimi Delta Attention across the sequence, Attention Residuals across depth

· Zhejian Peng · 25 min read

Kimi K3 (arXiv:2607.24653) is a 2.8T-parameter MoE with 104B active parameters, 93 layers, and a 1M-token context. Two architectural bets carry the report:

  • Kimi Delta Attention (KDA) — how information moves along the sequence
  • Attention Residuals (AttnRes) — how information moves along depth

Part 1 — Kimi Delta Attention

What KDA is replacing

Softmax attention keeps every token it has ever seen: generating token \(t\) compares the query against \(t\) cached keys, so per-token cost and KV cache both grow with the context. At 1M tokens that cache is the deployment bill.

Linear attention takes the other side of the trade. Drop the softmax and the whole history collapses into one fixed-size matrix \(\mathbf{S} \in \mathbb{R}^{d_k \times d_v}\) carried from token to token:

\[\mathbf{S}_t = \mathbf{S}_{t-1} + k_t v_t^\top, \qquad o_t = \mathbf{S}_t^\top q_t\]

Constant memory, constant work per token, unbounded context. The catch is that the state never gets bigger, so everything in KDA is about what to write into it and what to erase.

The state is an associative memory

Read the update again as a memory. Writing \(kv^\top\) stores the pair \((k, v)\); reading is a dot product against every key at once:

\[\mathbf{S}^\top q = \sum_i (k_i^\top q)\, v_i\]

If the keys are orthonormal, querying \(k_1\) returns exactly \(v_1\) — a \(d_k \times d_v\) matrix holds \(d_k\) clean slots. Reality is messier in two ways.

Keys collide. With \(k_3\) at 45° to \(k_1\), storing \((k_1, v_1)\) then \((k_3, v_2)\) and reading at \(k_1\) gives \(v_1 + 0.707\,v_2\). Crosstalk.

Rewrites pile up. Store x = 1, later x = 3; plain accumulation hands back both:

write (k1,v1), (k2,v2), then (k1,v1')     read at k1
  plain accumulation   ->  v1 + v1'      (stale value still there)
  delta rule, β = 1    ->  v1'           (old value erased)

A fixed-size memory that only ever adds becomes an increasingly blurry average of everything. Two fixes exist, and KDA uses both.

Fix 1 — forgetting, one rate per channel

Multiply the state down before each write:

\[\mathbf{S}_t = \operatorname{Diag}(\alpha_t)\,\mathbf{S}_{t-1} + k_tv_t^\top\]

Mamba-2 and Gated DeltaNet use a scalar \(\alpha_t\): the whole memory fades at one rate. KDA makes \(\alpha_t \in (0,1)^{d_k}\) channel-wise — one retention factor per key dimension, chosen per token. Row \(j\) of the state fades at its own speed, so the same layer can hold a fast, local channel next to one that keeps information for thousands of tokens:

per-step log-decay \(g\) retention \(\alpha = e^{g}\) half-life
\(-0.0001\) 0.99990 6931 tokens
\(-0.1\) 0.90484 7 tokens

This is also where K3 gets its sense of position. Decay makes “20 tokens ago” measurably weaker than “2 tokens ago”, so the model needs no positional encoding at all (NoPE) — and a model with no RoPE frequencies has nothing to retune when the context window grows from 8K to 1M.

Fix 2 — the delta rule, or writing by correction

Instead of asking what to add, ask what the memory currently gets wrong. The memory should satisfy \(\mathbf{S}^\top k_t = v_t\), so define a loss and take one gradient step:

\[\mathcal{L}(\mathbf{S}) = \tfrac{1}{2}\lVert \mathbf{S}^\top k_t - v_t \rVert^2, \quad \nabla_{\mathbf{S}}\mathcal{L} = k_t(\mathbf{S}^\top k_t - v_t)^\top\] \[\mathbf{S}_t = \mathbf{S}_{t-1} - \beta_t \nabla_{\mathbf{S}}\mathcal{L} = \left(\mathbf{I} - \beta_t k_tk_t^\top\right)\mathbf{S}_{t-1} + \beta_t k_tv_t^\top\]

That is the delta rule, and a linear-attention layer is doing online gradient descent on its own memory at inference time, with \(\beta_t\) as the learning rate.

The geometry is the part worth keeping. Because KDA L2-normalises its keys, \(\lVert k_t\rVert = 1\) and \(\mathbf{I} - \beta_tk_tk_t^\top\) is a scaling along \(k_t\) and the identity everywhere else. It erases only what was stored at this key and leaves the rest of the memory untouched:

  • \(\beta_t = 1\) — full overwrite: whatever was at \(k_t\) is gone, \(v_t\) takes its place
  • \(\beta_t = 0\) — no write at all
  • in between — a partial correction toward \(v_t\) (with \(\beta = 0.5\) the numbers above give \(0.25\,v_1 + 0.5\,v_1'\))

KDA = both, and here is the whole layer

\[\mathbf{S}_t=\left(\mathbf{I}-\beta_t k_t k_t^{\top}\right) \operatorname{Diag}(\alpha_t) \mathbf{S}_{t-1}+\beta_t k_t v_t^{\top}, \qquad \tilde{o}_t=\mathbf{S}_t^{\top} q_t\]

Decay first, then the correction. Everything on the right is produced from the token itself:

term what it is how it is produced
\(q_t,k_t\) probe / address ShortConv → Swish → L2Norm
\(v_t\) payload ShortConv → Swish
\(\beta_t \in (0,1)\) write strength \(\operatorname{Sigmoid}(\mathbf{W}_\beta x_t)\)
\(\alpha_t \in (0,1)^{d_k}\) per-channel retention low-rank logits + per-head bias, then Eq. below

The ShortConv gives each token a short local window before it becomes a key or value; the L2Norm is what makes the erase step a clean projection rather than an arbitrary rescaling.

The update is easiest to implement in its equivalent decay → predict → correct → read form:

decayed_state = row_scale(alpha_t, state)
prediction = transpose_matrix_vector(decayed_state, key_t)
error = vector_sub(value_t, prediction)
state = matrix_add(decayed_state, matrix_scale(beta_t, outer(key_t, error)))
output = transpose_matrix_vector(state, query_t)

This version exposes the idea more clearly than constructing \(\mathbf I-\beta kk^\top\): the layer forgets rows of its state, asks what value the decayed state predicts for the current key, and writes only the residual error.

Run the complete tutorial on a CPU

The complete dependency-free Python tutorial uses only the standard library. It is not Kimi K3 and it is not a fast production kernel; it is a small numerical laboratory for the equations in this article. From the repository root:

python3 examples/kda_cpu_tutorial.py

It runs five experiments:

  1. writes an association, repeats it, and shows that the delta rule does not double-count it;
  2. overwrites the same key with a new value and demonstrates the effect of \(\beta\);
  3. applies different retention factors to different state rows;
  4. verifies the UT chunk form against token-by-token KDA;
  5. verifies associative segment composition for KDA Context Parallelism and compares the old and new decay maps.

I ran it on CPU. The two ways of evaluating a chunk agreed to floating-point precision:

Maximum output difference:      8.882e-16
Maximum final-state difference: 6.661e-16
Maximum segment difference:     4.441e-16
All numerical equivalence checks passed.

I also ran 100 random recurrence-versus-chunk tests and 100 random segment-composition tests; all passed. These checks validate the educational implementation’s algebra, not the speed or low-precision numerical behavior of FlashKDA’s CUDA kernel.

Big-O: what becomes linear, and what does not

Let \(T\) be sequence length, \(H\) the number of heads, and \(d_k,d_v\) the key and value dimensions per head. Ignoring the Q/K/V projections shared by both designs, causal softmax attention and recurrent KDA scale as follows:

mechanism sequence work decode cache next token
softmax \(O(HT^2d_k)\) \(O(HT(d_k+d_v))\) \(O(HT(d_k+d_v))\)
KDA \(O(HTd_kd_v)\) \(O(Hd_kd_v)\) \(O(Hd_kd_v)\)

When \(H,d_k,d_v\) are fixed architectural constants, KDA is \(O(T)\) over a sequence and \(O(1)\) in persistent cache with respect to context length. “Constant” does not mean free: every new token still reads and updates an entire \(d_k\times d_v\) state per head.

Kimi K3 uses \(H=96\) and \(d_k=d_v=128\), so one KDA layer carries

\[96\times128\times128=1{,}572{,}864\]

state elements per sequence: about 3 MiB in BF16. Across 69 KDA layers that is about 207 MiB before tensor-parallel sharding, convolution histories, allocators, and saved prefix checkpoints. Its size does not grow when the context goes from 1K to 1M tokens.

The production chunk kernel has the per-head attention cost reported in Kimi Linear:

\[6Td_h^2+3TCd_h+TC^2,\]

where \(C\) is chunk size and \(d_h=d_k=d_v\). This remains \(O(T)\) for fixed \(C\) and \(d_h\). The tutorial’s explicit Python loops are deliberately readable rather than optimized; timing them would measure Python overhead, not FlashKDA.

The whole Kimi K3 model is also not constant-cache. Its 24 MLA layers still keep sequence-growing latent KV caches and perform global attention. The 69 KDA layers remove that growth from roughly three quarters of the attention stack.

Why use KDA?

KDA is useful when generation is long enough that repeatedly reading a growing KV cache dominates cost:

  • Fixed recurrent memory. Most layers replace per-token KV entries with one state matrix.
  • Corrective writes. The delta rule overwrites an association instead of accumulating stale and duplicate values.
  • Different memory timescales. Channel-wise \(\alpha\) lets one head maintain fast-forgetting and slow-retaining features simultaneously.
  • Implicit order and recency. The ordered, data-dependent transitions change when token order changes, so the model can use NoPE in its global layers.
  • Exact GPU-friendly reformulation. Training uses chunkwise matrix multiplication without changing the mathematical recurrence.

The price is finite capacity. A fixed state can blur unrelated facts or lose exact token details, which is why Kimi K3 retains periodic global MLA rather than using KDA everywhere.

Chunkwise parallel form, step by step

The recurrent loop is ideal for decoding one token at a time, but poor for training: token \(r\) needs the state produced by token \(r-1\), so a literal implementation launches one small matrix update after another. Chunkwise KDA does not remove the recurrence. It packages most of the work inside a fixed-size chunk into dense matrix multiplications, while passing one state sequentially between chunks.

1. Lay out one chunk

For a chunk of \(C\) tokens, stack rows as

\[\mathbf Q,\mathbf K\in\mathbb R^{C\times d_k},\qquad \mathbf V,\mathbf O\in\mathbb R^{C\times d_v},\qquad \mathbf S_{\mathrm{in}}\in\mathbb R^{d_k\times d_v}.\]

For positions \(1\le i\le j\le C\), define the channel-wise cumulative retention

\[\gamma^{i\to j}=\prod_{r=i}^{j}\alpha^r, \qquad \gamma^j=\gamma^{1\to j}.\]

Every product is elementwise over the \(d_k\) key channels. Let \(\mathbf\Gamma\in\mathbb R^{C\times d_k}\) stack \(\gamma^1,\ldots,\gamma^C\) row by row.

2. See what the recurrence is hiding

Write one token transition as

\[\mathbf T_r=(\mathbf I-\beta_r k_rk_r^\top)\operatorname{Diag}(\alpha_r), \qquad \mathbf B_r=\beta_rk_rv_r^\top.\]

After the first \(r\) tokens of the chunk,

\[\mathbf S^r= \underbrace{\mathbf T_r\mathbf T_{r-1}\cdots\mathbf T_1}_{\mathbf P^r} \mathbf S_{\mathrm{in}} + \sum_{i=1}^{r} \mathbf T_r\cdots\mathbf T_{i+1}\mathbf B_i.\]

The newest transition is on the left. This equation explains the difficulty: every write is transformed by all later decays and delta erasures. Computing those products separately for every output would repeat the same work.

3. Turn relative decay into query/key scaling

Consider a query at row \(i\) reading a key written at row \(j\le i\). The retention between them is

\[\frac{\gamma^i}{\gamma^j} = \prod_{r=j+1}^{i}\alpha^r.\]

Therefore its decayed similarity can be written as

\[q_i^\top\operatorname{Diag}\!\left(\frac{\gamma^i}{\gamma^j}\right)k_j = (\gamma^i\odot q_i)^\top(k_j/\gamma^j).\]

All such scores appear in one matrix multiplication:

\[\mathbf A= \operatorname{Tril}\!\left[ (\mathbf Q\odot\mathbf\Gamma) (\mathbf K/\mathbf\Gamma)^\top \right] \in\mathbb R^{C\times C}.\]

Tril removes future positions but keeps the diagonal. For \(i=j\), the ratio is one: an output reads \(\mathbf S_i\) after its own write, so token \(i\) must be allowed to read its own pseudo-value.

4. Fold the delta-rule dependencies with the UT transform

Decay scaling alone is not enough, because a delta write stores the residual after accounting for previous writes. Define

\[\mathbf L= \operatorname{StrictTril}\!\left[ \operatorname{Diag}(\boldsymbol\beta) (\mathbf\Gamma\odot\mathbf K) (\mathbf K/\mathbf\Gamma)^\top \right] \in\mathbb R^{C\times C}.\]

The strict lower triangle contains how each earlier key changes the prediction seen by a later key. The UT transform solves that causal system:

\[\mathbf M=(\mathbf I+\mathbf L)^{-1}\operatorname{Diag}(\boldsymbol\beta),\] \[\mathbf W=\mathbf M(\mathbf\Gamma\odot\mathbf K) \in\mathbb R^{C\times d_k}, \qquad \mathbf U=\mathbf M\mathbf V \in\mathbb R^{C\times d_v}.\]

Because \(\mathbf I+\mathbf L\) is lower triangular with ones on its diagonal, this is a forward substitution rather than a general matrix inverse. The compact pseudo-values are

\[\widetilde{\mathbf V} = \mathbf U-\mathbf W\mathbf S_{\mathrm{in}} \in\mathbb R^{C\times d_v}.\]

The two terms have direct meanings: \(\mathbf U\) contains mutually corrected current-chunk values, while \(\mathbf W\mathbf S_{\mathrm{in}}\) subtracts what the incoming state already predicts. Thus each row of \(\widetilde{\mathbf V}\) is the effective residual that token contributes after all earlier in-chunk corrections.

5. Produce every output in the chunk

Once \(\widetilde{\mathbf V}\) is known,

\[\boxed{ \mathbf O= \underbrace{(\mathbf\Gamma\odot\mathbf Q)\mathbf S_{\mathrm{in}}}_{\text{memory from earlier chunks}} + \underbrace{\mathbf A\widetilde{\mathbf V}}_{\text{writes in this chunk}} }\]

has shape \(C\times d_v\). The first term decays and queries the state entering the chunk. The second is a causal weighted sum of corrected writes from the current chunk. This is Kimi K3 Eq. 4.

The state handed to the next chunk is computed from the same pseudo-values:

\[\boxed{ \mathbf S_{\mathrm{out}} = \operatorname{Diag}(\gamma^C)\mathbf S_{\mathrm{in}} + (\mathbf\Delta\odot\mathbf K)^\top\widetilde{\mathbf V} }\]

where row \(i\) of \(\mathbf\Delta\) is the retention after that write,

\[\Delta_i=\prod_{r=i+1}^{C}\alpha^r,\]

with an empty product of one for the last token. The incoming state experiences all \(C\) decays; a write at position \(i\) experiences only later decays.

6. What is actually parallel?

The chunks remain recurrent: \(\mathbf S_{\mathrm{out}}\) from chunk \(t\) is \(\mathbf S_{\mathrm{in}}\) for chunk \(t+1\). Inside a chunk, the expensive query-key, output, and state-update work is expressed as matrix multiplication. A causal triangular solve remains in the UT transform, so “parallel within a chunk” means the bulk of the arithmetic is tiled dense work, not that every operation is independent.

The CPU tutorial’s official_naive_chunk_kda implements this sequence explicitly: cumulative log-decay, lower-triangular UT solve, \(\mathbf W/\mathbf U\), pseudo-values, causal scores, outputs, and final state. Against token-by-token recurrence it produced maximum output and state differences of \(8.882\times10^{-16}\) and \(6.661\times10^{-16}\). The chunkwise algorithm is an exact algebraic rewrite in real arithmetic; different accumulation order and BF16 storage explain small production-kernel differences.

What K3 changed relative to Kimi Linear

KDA comes from Kimi Linear (arXiv:2510.26692). K3 makes two edits, both small on paper and both about hardware.

1. Lower-bounded decay. Look again at \(\mathbf{K}/\mathbf{\Gamma}\): a reciprocal of a product of numbers below 1. Kimi Linear’s decay came from a negative softplus, \(g = -e^{A}\operatorname{Softplus}(z) \in (-\infty, 0)\), so that reciprocal can explode and overflow. K3 bounds it with a scaled sigmoid:

\[g_t = g_{\min}\operatorname{Sigmoid}(e^{A_h}z_t) \in (g_{\min}, 0), \qquad g_{\min} = -5\]

Now every step retains at least \(e^{-5} \approx 0.0067\), cumulative log-decay over a 16-token tile stays in \((-80, 0)\), and the rescaling factor is at most \(e^{80} \approx 5.5\times10^{34}\) — comfortably inside BF16’s \(3.4\times10^{38}\). The payoff is concrete: Kimi Linear had to compute the diagonal tiles with explicit position-pair arithmetic, the main intra-chunk bottleneck. With a bounded range every tile, diagonal included, becomes a dense tensor-core matmul. A numerical-stability bound bought a kernel rewrite.

2. Full-rank output gate. The low-rank gate becomes an input-dependent full-rank projection, so each token can decide channel by channel how much of the recurrent read to let through:

\[y_t=\mathbf{W}_o\left[\operatorname{Sigmoid}(\mathbf{W}_g x_t) \odot \operatorname{RMSNorm}(\tilde{o}_t)\right]\]

KDA does not work alone: 3:1 with Gated MLA

A fixed state is lossy by construction, so K3 interleaves exact attention. Each block is 3 KDA layers + 1 Gated MLA layer, and one extra MLA closes the backbone:

\(23 \times (3\,\text{KDA} + 1\,\text{MLA}) + 1\,\text{MLA} = 93\) layers, i.e. 69 KDA and 24 MLA.

Only those 24 layers keep a cache that grows with the sequence — against 61 full-attention layers in K2. The MLA layers are NoPE too, and carry the same full-rank output gate.

The division of labour is clean. The 69 KDA layers give recency-weighted, position-aware mixing out of a fixed \(d_k \times d_v\) state at constant cost per token; the 24 MLA layers keep a latent cache that grows with \(T\) and buy back what a finite state cannot hold — exact access to any earlier token.

At the Kimi Linear scale the hybrid cut KV-cache usage by up to 75% and reached up to 6× decoding throughput at 1M context, while beating full MLA on quality under a matched recipe.

The state is small — but it is serial

Most of K3’s KDA engineering follows from one sentence: the state is cheap to move and impossible to skip ahead in.

  • FlashKDA — a CUTLASS chunkwise kernel that overlaps intra-chunk math with cross-chunk state propagation, so the SMs are not idle during the serial hand-off. It also serves prefill, as a backend of flash-linear-attention.
  • KDA Context Parallelism (KCP) — the interesting one. Vanilla linear attention is a plain sum, so every rank can start from \(\mathbf{S}=\mathbf{0}\) and the results add up. KDA cannot: in \(\mathbf{S}_t = \mathbf{M}_t\mathbf{S}_{t-1} + \beta_tk_tv_t^\top\) with \(\mathbf{M}_t = (\mathbf{I}-\beta_tk_tk_t^\top)\operatorname{Diag}(\alpha_t)\), the incoming state is transformed, not just added to. So each rank computes two local quantities — its segment’s cumulative transition \(\mathbf{M}\), and the state its own tokens generate from zero — and one all-gather plus a prefix scan composes them exactly. Messages stay fixed-size at any context length, which is what makes 1M-token training affordable.
  • Prefix caching — KDA state checkpoints land at 512-token boundaries in the same paged pool as the MLA KV cache; a prefix is reusable only if both restore at the same boundary.
  • Speculative decoding — the state updates in place, so a rejected draft cannot be rolled back. K3 caches the drafts’ much smaller projected inputs and replays the accepted prefix on-chip.

What to remember about KDA

  1. A fixed-size matrix used as an associative memory, read with \(\mathbf{S}^\top q\).
  2. Channel-wise decay — every key channel picks its own forgetting rate, which also encodes position (hence NoPE, hence painless 1M extension).
  3. The delta rule — write by erasing what was stored at this key first; it is one step of online gradient descent, and \(\beta_t\) is the learning rate.
  4. Chunkwise form — the exact same recurrence expressed as matmuls; K3’s bounded decay pushes the last stubborn tile onto tensor cores.
  5. Hybrid by design — 3:1 with Gated MLA, because a finite state should not be asked to do exact recall.

Part 2 — Attention Residuals

KDA fixes the sequence axis. AttnRes (arXiv:2603.15031) runs the same argument down the depth axis.

The problem

A PreNorm residual looks innocent:

\[h_l = h_{l-1} + f_l(h_{l-1}) = h_0 + \sum_{i=1}^{l} f_i(h_{i-1})\]

Every earlier layer is added with weight 1. Depth is an RNN: all history is crushed into one vector. Hidden-state magnitude grows with depth, so each new layer is a smaller and smaller fraction of the stream — PreNorm dilution. Early information cannot be fetched back on demand.

Sequence modeling had the same bottleneck, and softmax attention replaced the RNN. AttnRes does that for depth: a standard residual is depth-wise linear attention; AttnRes is depth-wise softmax attention.

Three residuals

Standard. Each layer sees only \(h_{l-1}\), with fixed mixing weights and one hidden state travelling between layers — \(N = 1\) below.

Full AttnRes. Every layer output becomes a key/value, and each layer picks among them:

\[h_l = \sum_{i=0}^{l-1} \alpha_{i \to l}\, v_i, \qquad \alpha_{i \to l} = \mathrm{softmax}_i\big(w_l^\top \mathrm{RMSNorm}(k_i)\big)\]
  • \(w_l \in \mathbb{R}^d\): one learned pseudo-query per layer, decoupled from that layer’s forward pass, so the mix is content-dependent
  • RMSNorm on the keys stops large-magnitude layers from dominating; queries start at zero, so training begins as a uniform average and does not spike
  • Compute \(O(L^2 d)\), store \(O(Ld)\). The real cost is that pipeline parallelism must ship every layer output across stages

Block AttnRes (what ships). Split \(L\) layers into \(N\) blocks: inside a block an ordinary residual, collapsed to one block vector; across blocks Full AttnRes over the \(N\) summaries plus the embedding, with the unfinished block exposing a partial sum. Traffic drops from \(O(Ld)\) to \(O(Nd)\). \(N = L\) is Full, \(N = 1\) is Standard, and empirically \(N \approx 8\) recovers most of Full. Kimi Linear 48B used 6 layers per block → 9 blocks + embedding = 10 depth sources, for < 4% training overhead and < 2% decode latency.

Experiments

Scaling law. Five sizes, each with Baseline / Block (\(N=8\)) / Full, all under the baseline’s hyperparameters — a deliberately conservative test. AttnRes is lower loss along the whole compute curve. Largest size: Baseline 1.719, Block 1.693, Full 1.692, and at 5.6 PFLOP/s-days the baseline needs about 25% more compute to match Block. Full is the ceiling; Block is “almost the same, and you can actually train it.”

48B / 3B active, 1.4T tokens. Validation loss is lower throughout and the gap widens during decay. Baseline output magnitude grows monotonically with depth, while Block resets at block boundaries; gradients even out too, since the softmax makes depth sources compete instead of dumping everything on the earliest layers.

  Baseline AttnRes
MMLU 73.5 74.6
GPQA-Diamond 36.9 44.4 (+7.5)
Math 53.5 57.1 (+3.6)
HumanEval 59.1 62.2 (+3.1)
C-Eval 79.6 82.5

Knowledge moves a little. Multi-step reasoning and code move a lot — consistent with later layers being able to pull earlier representations on demand.

16-layer ablation (loss, lower is better): Baseline PreNorm 1.766, DenseFormer 1.767, mHC 1.747, Full AttnRes 1.737, input-independent mixing 1.749, sigmoid instead of softmax 1.741, no RMSNorm 1.743, Block \(S=4\) 1.746. Fixed mixing is clearly worse than learned softmax — content-dependent depth selection is doing real work.

Takeaway

K3 grew depth (K2 had 61 layers; K3 has 93). A unit-weight residual starts to look like an RNN on that axis, exactly as a growing KV cache is the wrong answer on the sequence axis. So both axes get the same treatment: selective, data-dependent retrieval instead of uniform accumulation — KDA across tokens, AttnRes across layers. Together with Stable LatentMoE on the width axis, that is where the reported ~2.5× scaling-efficiency gain over K2 comes from.

Papers: Kimi K3 · Kimi Linear · Attention Residuals. Code: flash-linear-attention.

Comments

Loading…
Jazzik