这系列方法的本质目的是什么？到底把持续学习的困境转换成了什么问题？
真正的遗忘在于分类器，对齐分类器？（SLCA）
InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning(CVPR 2024)
Motivation
- Although existing continual learning methods based on PEFT have demonstrated superior performance compared to those not based on PEFT, most of them do not consider how to eliminate the interference of the new task on the old tasks, which inhibits the model from making a good trade-off between stability and plasticity.
Insight
- Proposition 1 has demonstrated that using InfLoRA to train the model is equivalent to directly finetuning the pre-trained weight $W$ within the subspace $\operatorname{span}\lbrace b_{t1}, \ldots, b_{tr} \rbrace$. Therefore, before learning the $t$-th task, we can design matrix $B_t$ such that learning the $t$-th task in the subspace $\operatorname{span}\lbrace b_{t1}, \ldots, b_{tr} \rbrace$ will not interfere with the performance of the model on the old tasks.
没有找到明显阐述 insight 的语句，感觉更多像是发现 sequential SVD 可以迁移过来。
Method
来一个新任务，设计专属子空间 $B_t$，开一条小支路，只训练 $A_t$。
用 local CE 学完后，用 DualGPM 保存梯度信息，再把支路合并回主网络。
最后冻结这个支路。
结果：不破坏旧任务，参数量保持稳定，并且能继续学新任务。
Mathematical Notes
通过链式法则推导出冻结 $B$ 与否不受影响
| 维度 | 书评 |
|---|---|
| writing | 写作很流利，证明简洁易懂 |
| idea | motivation make sense |
| experiment | 消融设计合理，扩展实验丰富 |
Limitation
- 对旧任务梯度的近似是否足够准确？这部分主要由 DualGPM 完成。
- 预计算 $B$ 矩阵有额外开销。DualGPM 需要为模型的每一层存储一组正交基，存储和更新开销会随任务数量增加。
- 学习空间不断缩小是否会抑制可塑性？$N_t \cap M_t^\perp$ 会随着旧任务增多而不断被压缩，因为 $M_t^\perp$ 的维度在持续减小。论文提到可以通过调整超参数 $\epsilon_{\mathrm{th}}$ 来减缓这个过程，但这是否仍然意味着在学习大量任务后，模型几乎没有空间学习一个与过去任务截然不同的新任务？这可能会导致长期学习中的可塑性逐渐丧失。
Supplement
这里实际上可能是次优的。部分参数共用直觉上讲有利于存储通用知识：
During this process, the interference of the new task on the old tasks exists due to the shared parameters between new and old tasks, which means fine-tuning a pre-trained model on a new task may interfere with the model’s performance on the old tasks.
Citation
@article{Liang2024InfLoRA,
title={InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning},
author={Yan-Shuo Liang and Wu-Jun Li},
journal={2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2024},
pages={23638-23647},
url={https://api.semanticscholar.org/CorpusID:268819832}
}
BiLoRA: Almost-Orthogonal Parameter Spaces for Continual Learning(CVPR 2025)
Motivation
A key challenge is preventing interference between tasks, which degrades performance when learning new tasks over previously learned tasks. However, existing PEFT-based continual learning methods such as InfLoRA face fundamental limitations: they rely on complex optimization procedures to learn orthogonal task-specific spaces, which becomes increasingly difficult as tasks accumulate.
Insight
The key insight is that by expanding the parameter space quadratically through two fixed bases, BiLoRA can achieve almost orthogonal task subspaces probabilistically, eliminating the need for explicit interference elimination procedures.
Method
通过 bilinear 方式扩展参数分配空间。
用 DFT 实现任务分离。
Mathematical Notes
先通过 bilinear 实现二维扩展，然后证明二维扩展更优，最后证明 DFT 策略的最优。核心在于推导任务干扰的概率。
| 维度 | 书评 |
|---|---|
| writing | 图画得很简明；对于可能存在的极端情况，即高维随机变量碰撞，也有补丁 |
| idea | insight 很符合直觉；DFT 恰到好处 |
| experiment | 理论足够完善，所以除了常规拼点实验外，实验基本都在佐证 theorem |
Limitation
- 所有频率分量都同等重要吗？该方法通过在 $d^2$ 维频率空间中为每个任务随机稀疏分配 $k$ 个分量来实现几乎正交。这隐含了一个假设：任意 $k$ 个频率分量的组合对于学习新任务都是足够且有效的。质疑点：在信号处理中，低频分量通常代表全局、平滑结构，高频分量代表细节和噪声。随机分配是否可能导致某些任务分到无效或次优的频率组合？例如，一个需要识别全局轮廓的任务，却分到了大量高频分量。这是否会以一种新形式重新引入它试图解决的次优分配问题？
- Parameter space orthogonal 貌似并不能说明 functional independence。所以加起来后的表现是不可预测的。一个简单例子是 task 1 的数据同时也能激活 task 2 的 weight。我觉得这里保证不了 input 完全不包含其他 task 的信息，毕竟只能改变 $w$，改变不了 $x$。
Citation
@article{Zhu2025BiLoRA,
title={BiLoRA: Almost-orthogonal Parameter Spaces for Continual Learning},
author={Hao Zhu and Yifei Zhang and Junhao Dong and Piotr Koniusz},
journal={2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2025},
pages={25613-25622},
url={https://api.semanticscholar.org/CorpusID:280134150}
}
TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree(ICML 2025)
Motivation
- Although there exist some continual learning methods developed for LPMs, they often do not focus on efficiency in CL, typically exhibiting computational complexity that scales linearly with the number of tasks. Consequently, this raises a critical new challenge: how to develop an efficient continual learning approach, particularly fitted for large pre-trained models?
Insight
- Some tasks exhibit similar characteristics and can be grouped together, allowing the use of task-shared knowledge to improve performance and accelerate adaptation to new tasks. Conversely, tasks that differ significantly or conflict with others are handled separately to alleviate catastrophic forgetting.
Method
TreeLoRA begins with a root node and incrementally builds a hierarchical tree structure as new tasks arrive. For each incoming task, it uses a bandit algorithm to identify the most suitable branch that shares similar gradient directions. Sparse gradient updates are then performed to adapt the model efficiently while retaining previously acquired knowledge. At the end of each task stream, key parameters of that task are recorded, and the TreeLoRA structure is updated accordingly.
Mathematical Notes
后悔界（Theorem 1）
在假设 1 成立的前提下，TreeLoRA 的累计后悔满足：
\[\operatorname{Reg}(T) \leq O\left( \sqrt{T |J_\eta| \log \frac{N T}{|J_\eta|}} + \frac{\delta^c}{\eta^{2+c}} \log \frac{N T}{\eta^2} \right)\]TreeLoRA 按梯度相似性建树
↓
相似任务被分到同一分支 → 树具有“平滑性”（Assumption 1）
↓
Bandit 算法在平滑树上搜索时，即使没选最优，也不会太差
↓
因此，累计“后悔”很小，且对任务数的依赖从 √N 降到 log N
↓
→ 证明 TreeLoRA 的搜索策略是高效且可扩展的
树结构 = 区域划分；平滑性 = 同区咖啡口味相近；后悔小 = 你不会喝太多难喝的咖啡。
| 维度 | 书评 |
|---|---|
| writing | 数学证明和文章的 motivation 以及 challenge 是对应的 |
| idea | 比较常规，更多是过去已有想法的适配 |
| experiment | 针对 limitation 有设计，实验看起来很丰富 |
Limitation
- TreeLoRA 的树是增量构建的，这意味着树的最终形态与任务出现顺序密切相关。一个糟糕的任务顺序，例如先来一系列高度相似任务，再来一系列完全不相关任务，可能会导致次优且不平衡的树结构。这是否会永久性地影响后续任务的学习效率和效果？
- TreeLoRA 的核心是寻找并利用相似性。如果模型遇到一系列彼此毫无关联的任务，例如代码生成、财务报表分析、古诗词创作，TreeLoRA 试图寻找相似性的机制本身是否会成为负担，甚至导致负迁移？
Supplement
深浅层特征的说法：
Consider a deep neural network as the learning model, which can be divided into multiple layers of parameters and connected by activation functions. Shallow layers (closer to the input) tend to capture low-level task-shared common patterns, while deeper layers (closer to the output) capture high-level task-specific semantics.
Citation
@article{Qian2025TreeLoRA,
title={TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree},
author={Yuyang Qian and Yuan-Ze Xu and Zhenyu Zhang and Peng Zhao and Zhi-Hua Zhou},
journal={ArXiv},
year={2025},
volume={abs/2506.10355},
url={https://api.semanticscholar.org/CorpusID:279318978}
}
CLoRA: Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models(ACL 2025)
Motivation
LoRA imposes no restrictions on parameter updates beyond the low-rank constraint, and matrix perturbation theory suggests that even low-rank updates can significantly influence matrix properties. Therefore, LoRA would benefit from more constraints for mitigating catastrophic forgetting. However, more constraints would reduce model capacity for updating, which influences training effectiveness. For instance, adding $L_2$ regularization significantly restricts the norm of the update matrix. Consequently, effective management of the capacity-forgetting balance becomes a major concern.
Insight
通过正交子空间正则化约束 LoRA 更新，使更新尽可能避开某些被保护的子空间，从而缓解 continued training 中的 catastrophic forgetting。
Method
LoRA + orthogonal subspace regularization.
Mathematical Notes
零空间和行空间的提出，增大零空间，再进行正则化。
| 维度 | 书评 |
|---|---|
| writing | 文章质量较差，有很多明显 typos |
| idea | 理论思路很简明 |
| experiment | 实验做得很多，理想当时还不怎么缺卡 |
Limitation
- 随机正则化矩阵 $P$ 的有效性？CLoRA 的直觉是，通过将 $P$ 所定义的子空间强制变为更新豁免区，即 $\Delta W$ 的零空间的一部分，可以减少对模型输出 $y$ 的改变。质疑点：如果这个随机选择的豁免区与模型已有知识所依赖的关键输入子空间毫无关联，那么保护这个随机区域有什么意义？
- CLoRA 强制将 $P$ 定义的子空间变为更新豁免区。质疑点：如果学习新任务所必需的关键特征恰好落在这个随机选择的豁免区内，那么 CLoRA 的约束反而会严重阻碍模型学习新知识，损害可塑性。
Supplement
注意投稿的 track。
Citation
@inproceedings{Lu2024ControlledLA,
title={Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models},
author={Yuheng Lu and Bingshuo Qian and Caixia Yuan and Huixing Jiang and Xiaojie Wang},
booktitle={Annual Meeting of the Association for Computational Linguistics},
year={2024},
url={https://api.semanticscholar.org/CorpusID:273507397}
}
Analytic Subspace Routing: How Recursive Least Squares Works in Continual Learning of Large Language Models(ICCV 2025)
Motivation
Existing techniques either leverage previous data to replay, leading to extra computational costs, or utilize a single parameter-efficient module to learn the downstream task, constraining new knowledge absorption with interference among different tasks.
Insight
实际还是解析解。
Method
LoRA + orthogonal subspace regularization.
懒得评了，还是那套公式，拿着锤子找钉子。