Hi, I have a small question about MEND edit procedure in test time:
In your paper's Algorithm 2, line 7 (updating the weight matrix of layer $l$), you wrote $\tilde{W}l \gets W_l-\tilde{\delta}{l+1} \tilde{u}_l ^T$, and there is no learning rate in this equation. However, in mend.py, line 227,
updates = {n: lr * g for lr, (n, g) in zip(self.edit_lrs, mean_grads.items())},
there actually is learning rate when updating, right?
Thank you!
Hi, I have a small question about MEND edit procedure in test time:$l$ ), you wrote $\tilde{W}l \gets W_l-\tilde{\delta}{l+1} \tilde{u}_l ^T$, and there is no learning rate in this equation. However, in
In your paper's Algorithm 2, line 7 (updating the weight matrix of layer
mend.py, line 227,updates = {n: lr * g for lr, (n, g) in zip(self.edit_lrs, mean_grads.items())},there actually is learning rate when updating, right?
Thank you!