这系列方法的本质目的是什么？到底把持续学习的困境转换成了什么问题

💡 真正的遗忘在于分类器，对齐分类器?（SLCA）

持续学习（Continual Learning）的老问题是 catastrophic forgetting——学了新的、忘了旧的。LoRA 这类 PEFT 方法给了一条新路：与其动整个预训练权重，不如只在一个低秩子空间里做更新，把「学习新任务」变成「给老参数开一条不互相打架的小支路」。于是问题从「怎么不忘记」被重新表述成「怎么让新旧任务的更新落在互不干扰的子空间里」。下面 5 篇工作，分别从干扰消除、近正交参数空间、层级梯度相似树、子空间正则、解析路由五个角度回答这同一个问题——也是「把 LoRA 玩出花」的五种玩法。

这 5 篇可以归到两条主线：子空间隔离派（InfLoRA / BiLoRA / CLoRA / AnySSR，都在想办法让更新落在互不干扰的子空间）和 结构共享派（TreeLoRA，反过来利用任务相似性共享 LoRA）。它们共同的核心假设是「遗忘来自参数更新的相互干扰」——但如文末会追问的，真正的遗忘到底在权重里、还是在分类器里，这本身就是个开放问题。

《InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning》（InfLoRA-CVPR2024）

MotivationAlthough existing continual learning methods based on PEFT have demonstrated superior performance compared to those not based on PEFT, most of them do not consider how to eliminate the interference of the new task on the old tasks, which inhibits the model from making a good trade-off between stability and plasticity.

InsightProposition 1 has demonstrated that using our InfLoRA to train the model is equivalent to directly finetuning the pre-trained weight W within the subspace span{bt1, …, btr}. Therefore, before learning the t-th task, we can design matrix Bt such that learning the t-th task in the subspace span{bt1, …, btr} will not interfere with the performance of the model on the old tasks.（没有找到明显的阐述insight的语句，感觉更多的像是发现sequential SVD可以迁移过来）

Method

来一个新任务 → 设计专属子空间 BtB_tBt → 开一条小支路只训练 AtA_tAt。用 local CE 学完后 → 用 DualGPM 保存梯度信息 → 把支路合并回主网络。最后冻结这个支路。结果：不破坏旧任务、参数量保持稳定、能继续学新任务

Mathmatica通过链式法则推导出冻结B与否不受影响

飞书内嵌多维表格：点击查看原表格 (tbloP2TMLVEazrOO)

Limitation对旧任务梯度的近似是否足够准确（这部分都是DualGPM完成）预计算B矩阵的开销；DualGPM 需要为模型的每一层存储一组正交基，这个存储和更新的开销会随着任务数量的增加而增长。学习空间不断缩小是否会抑制“可塑性”？Nt∩Mt⊥ 会随着旧任务的增多而不断被压缩，因为 Mt⊥ 的维度在持续减小 ()。论文提到可以通过调整超参数 `ε_th` 来减缓这个过程尽管可以减缓，但这是否仍然意味着在学习了大量任务后，模型将几乎没有空间来学习一个与过去任务截然不同的全新任务？这可能会在长期学习中导致模型的“可塑性”逐渐丧失。

Supplement

这里实际上可能是次优的，部分参数共用直觉上讲是有利于存储通用知识的：“During this process, the interference of the new task on the old tasks exists due to the shared parameters between new and old tasks, which means fine-tuning a pre-trained model on a new task may interfere with the model’s performance on the old tasks.”

Cite

@article{Liang2024InfLoRA,

《BiLoRA: Almost-orthogonal Parameter Spaces for Continual Learning》（BiLoRA-CVPR2025）MotivationA key challenge is preventing interference between tasks which degrades performance when learning new tasks over previously learned tasks.However, existing PEFT-based continual learning methods such as InfLoRA face fundamental limitations, i.e., they rely on complex optimization procedures to learn orthogonal task-specific spaces which is increasingly difficult as tasks accumulate.InsightOur key insight is that by expanding the parameter space quadratically through two fixed bases, we can achieve “almost orthogonal” task subspaces probabilistically, eliminating the need for explicit interference elimination procedures.

Method

bilinear扩展参数分配空间 DFT实现任务分离

Mathmatica先通过bilinear实现二维扩展，然后证明二维扩展更优，最后证明dft策略的最优；核心在于推导出任务干扰的概率

飞书内嵌多维表格：点击查看原表格 (tblWhikLgHKjTXfE)

Limitation所有频率分量都同等重要吗？该方法通过在 `d²` 维的频率空间中为每个任务随机稀疏地分配 `k` 个分量来实现“几乎正交”。这隐含了一个假设：任何 `k` 个频率分量的组合对于学习一个新任务都是足够且有效的。质疑点：在信号处理中，低频分量通常代表全局、平滑的结构，而高频分量代表细节和噪声。随机分配是否可能导致某些任务分到“无效”或“次优”的频率组合（例如，一个需要识别全局轮廓的任务，却分到了大量高频分量）？这是否会以一种新的形式重新引入它试图解决的“次优分配”问题？Parameter space orthogonal貌似并不能说明functional independent。所以加起来后的表现是不可预测的。一个比较简单的例子是task1的数据同时也能激活task2的weight/我觉得是保证不了，我只能保证学到的东西不同task之间正交。我保证不了input完全不包含其他task的信息。毕竟你改变不了x只能改变w让他有某些性质使得wx有些更好的性质

Supplement

暂无

Cite

@article{Zhu2025BiLoRA,

《TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree》（TreeLoRA-ICML2025）

MotivationAlthough there exist some continual learning methods developed for LPMs , they often do not focus on the efficiency in CL, typically exhibiting computational complexity that scales linearly with the number of tasks. Consequently, this raises a critical new challenge: How to develop an efficient continual learning approach, particularly fitted for large pre-trained models?

InsightSome tasks exhibit similar characteristics and can be grouped together, allowing for the use of task-shared knowledge to improve performance and accelerate adaptation to new tasks.Conversely, tasks that differ significantly or even conflict with others are handled separately to alleviate catastrophic forgetting.

Method

TreeLoRA structure begins with a root node and incrementally builds a hierarchical tree structure as new tasks arrive. For each incoming task, it utilizes a bandit algorithm to identify the most suitable branch that shares similar gradient directions. Sparse gradient updates are then performed to efficiently adapt the model parameters while retaining previously acquired knowledge. At the end of each task stream, the key parameters of that task are recorded, and the TreeLoRA structure is updated accordingly.

Mathmatica后悔界（Theorem 1）

在假设 1 成立的前提下，TreeLoRA 的累计后悔满足： $\text{Reg}(T) \leq \dots$

TreeLoRA 按梯度相似性建树
    ↓
相似任务被分到同一分支 → 树具有“平滑性”（Assumption 1）
    ↓
Bandit 算法在平滑树上搜索时，即使没选最优，也不会太差
    ↓
因此，累计“后悔”很小，且对任务数的依赖从 √N 降到 log N
    ↓
→ 证明了 TreeLoRA 的搜索策略是高效且可扩展的

树结构 = 区域划分，平滑性 = 同区咖啡口味相近，后悔小 = 你不会喝太多难喝的咖啡

飞书内嵌多维表格：点击查看原表格 (tblJ5Q6C1pmbUzqA)

LimitationTreeLoRA的树是增量构建的，这意味着树的最终形态与任务的出现顺序密切相关。一个“糟糕”的任务顺序（例如，先来一系列高度相似的任务，再来一系列完全不相关的任务）可能会导致一个次优的、不平衡的树结构。这是否会永久性地影响后续所有任务的学习效率和效果？TreeLoRA的核心是“寻找并利用相似性”。如果模型遇到一系列彼此之间毫无关联的任务（例如，代码生成、财务报表分析、古诗词创作），TreeLoRA试图寻找相似性的机制本身是否会成为一种负担，甚至导致“负迁移”（Negative Transfer）现象，即相似的结构反而干扰了学习？

Supplement

深浅层特征的说法：“Consider a deep neural network as the learning model, which can be divided into multiple layers of parameters and connected by activation functions. Shallow layers (closer to the input) tend to capture low-level task-shared common patterns, while deeper layers (closer to the output) capture high-level task-specific semantics.”

Cite

@article{Qian2025TreeLoRA,

《Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models》（CLoRA-ACL2025）MotivationHowever, LoRA imposes no restrictions on parameter updates beyond the low-rank constraint, and matrix perturbation theory suggests that even low-rank updates can significantly influence matrix properties.Therefore, LoRA would benefit from more constraints for mitigating catastrophic forgetting. However, more constraints would reduce model capacity for updating, which influences the effectiveness of training. For instance, adding L2 regularization significantly restricts the norm of the updating matrix. Consequently, effective management of the capacityforgetting balancing has become a major concern.InsightOur key insight is that by expanding the parameter space quadratically through two fixed bases, we can achieve “almost orthogonal” task subspaces probabilistically, eliminating the need for explicit interference elimination procedures.

Method

LoRA + 正交正则化

Mathmatica零空间和行空间的提出->增大零空间->正则化

飞书内嵌多维表格：点击查看原表格 (tblkkTPH7bSEfRMW)

Limitation随机正则化矩阵 `P` 的有效性？CLORA 的直觉是，通过将 `P` 所定义的子空间强制变为“更新豁免区”（即 `ΔW` 的零空间的一部分），可以减少对模型输出 `y` 的改变。质疑点：如果这个随机选择的“豁免区”与模型已有知识（例如，通用语言能力、推理能力）所依赖的关键输入子空间毫无关联，那么保护这个随机区域又有什么意义呢？CLORA 强制将 `P` 定义的子空间变为“更新豁免区”。质疑点：如果学习新任务所必需的关键特征恰好就落在这个随机选择的“豁免区”内，那么 CLORA 的约束反而会严重阻碍模型学习新知识，极大地损害其“可塑性”。这种设计似乎存在“误伤”核心学习能力的风险。

Supplement

注意投稿的track

Cite

@inproceedings{Lu2024ControlledLA,

Analytic Subspace Routing: How Recursive Least Squares Works in Continual Learning of Large Language Model（AnySSR-ICCV2025）MotivationExisting techniques either leverage previous data to replay, leading to extra computational costs, or utilize a single parameter-efficient module to learn the downstream task, constraining new knowledge absorption with interference among different tasks.Insight实际就还是解析解

Method

LoRA + 正交正则化

懒得评了，还是那套公式，拿着锤子找钉子，通讯用类似的套路发了很多文章

串起来看：把 LoRA 玩出花之后，还剩什么

把 5 篇放在一起，能看到一条清晰的演进：InfLoRA 用 DualGPM 显式估计「对旧任务无干扰的子空间」，靠的是算；BiLoRA 觉得显式消除太贵，改成用 bilinear+DFT 在概率上达到「几乎正交」，靠的是设计；CLoRA 则更轻，直接用正交子空间正则把一块区域设为「更新豁免区」；AnySSR 把这套正交子空间思路推到解析解（递归最小二乘），干脆不迭代；TreeLoRA 走了另一条路——不追求隔离，反而利用任务间的梯度相似性建树共享。它们都把「持续学习的困境」转换成了同一个更具体的问题：怎么让新旧任务的参数更新落在互不干扰的子空间里。这是「把 LoRA 玩出花」的主旋律。

但有几个反问值得留在最后：

所以「如何把 LoRA 玩出花」——5 篇给出了从显式消除、概率正交、层级共享、正则豁免到解析路由的完整谱系，花是真玩出来了。但花下面那根根——「遗忘到底从何而来、子空间隔离是不是治标」——仍是开放问题。下一阶段，与其继续雕花，不如回头验证这个共同的根假设。