Tensor
Python
└─ torch._C.TensorBase
└─ C++ at::Tensor ← 两个指针
├─ → TensorImpl ← 数值区:data_ptr(数据) / sizes(形状) / strides(步长) / device(设备)
└─ → AutogradMeta ← 梯度区
AutogradMeta中
| 字段 | Python 层 | 装什么 |
|---|---|---|
grad_ | x.grad | backward 算完的梯度值。平时是空的,backward 才填 |
grad_fn_ | x.grad_fn | 指向反向节点的指针(伴随模型)。None = 叶子 |
grad_accumulator_ | 不可见 | 叶子的梯度累加器——AccumulateGrad 节点 |
tensor本质是一个多维数组,shape方法可以查询每一个轴(dim)的长度。dim的角标以0开始,-1结束。
tensor运算时会进行广播,原则是从右向左对齐,长度为1或缺失的维度会被复制。
x * y # 广播数乘
x @ y # 矩阵乘法创建Tensor
有多种方法可以创建张量
torch.tensor([1,2,3]) #(3,) 从 Python 数据建(会拷贝)
torch.zeros(2, 3) #(2,3) 全 0
torch.ones(2, 3) #(2,3) 全 1
torch.full((2,3), 7) #(2,3) 全 7(size 要传元组)
torch.arange(0, 10, 2) #(5,) 左闭右开 [0,10),步长 2
torch.linspace(0, 1, 5) #(5,) 闭区间 [0,1] 均匀取 5 个点
torch.eye(3) #(3,3) 单位矩阵
torch.randn(2, 3) #(2,3) 标准正态 N(0,1)
torch.rand(2, 3) #(2,3) [0,1) 均匀分布
torch.empty(2, 3) #(2,3) 不初始化,垃圾值
torch.zeros_like(x) #同x形状/dtype 跟x完全一样(ones_like/full_like/randn_like 同理)
x.clone() #同x深拷贝(改副本不动原 tensor)创建张量时常用参数有
| 参数 | 作用 |
|---|---|
dtype | 数据类型:torch.float32(默认)/ torch.int64 / torch.bool / torch.float16 |
device | 放哪:'cpu' 或 'cuda' |
requires_grad | True 时记录梯度(训练用) |
layout | 内存布局,默认 torch.strided;稀疏用 torch.sparse_coo |
pin_memory | True 时用锁页内存,CPU→GPU 拷贝更快 |
形状操作
tensor有如下形状操作:
x.view(2, 6) #(3,4) → (2,6) 共享底层内存(视图),元素总数必须一致;要求内存连续
x.reshape(2, 6) #(3,4) → (2,6) 同 view 但自动处理不连续,更安全
x.permute(1, 0) #(3,4) → (4,3) 按给定顺序重排所有轴(不拷贝数据)
x.transpose(0, 1) #(3,4) → (4,3) 只交换两个轴
x.t() / x.T #2D 转置 快捷方式
x.mT #任意维,交换最后两轴 批量矩阵转置
x.squeeze(0) #(1,3,4) → (3,4) 去掉指定轴(不指定就去掉所有长度为 1 的轴)
x.unsqueeze(1) #(3,4) → (3,1,4) 在 dim=1 插入长度 1 的新轴
x[:, None] #(5,) → (5,1) 你已会,就是 unsqueeze(1) 的下标写法
x.expand(2, 3, 4) #(1,3,4) → (2,3,4) 广播式扩展,不拷贝;原轴必须是 1;-1 表示保持原长
x.repeat(2, 1, 1) #(1,3,4) → (2,3,4) 真拷贝复制
x.flatten() #(2,3,4) → (24,) 全拉平
x.flatten(1) #(2,3,4) → (2,12) 从 dim=1 开始拉平permute()和transpose()都会使得内存不连续,因此需要.contiguous()才能.view()
数学运算
tenser的数学运算如下所示:
| 类别 | 写法 |
|---|---|
| 四则 | + - * / **(幂)//(整除)%(取余) |
| 函数版 | torch.add sub mul div pow sqrt rsqrt exp log log2 log10 abs neg |
| 三角 | sin cos tan arctan … |
| 取整 | floor(向下)ceil(向上)round(四舍五入)trunc(截断)sign(符号) |
| 限幅 | torch.clamp(x, min=-1, max=1) —— 低于 min 拉回 min,高于 max 拉回 max |
| 比较 | == != > < >= <= —— 逐元素,返回 bool tensor |
| 逻辑 | torch.logical_and logical_or logical_not(或 & | ~,注意括号) |
# 有些含有参数
torch.add(x, y, alpha=2) # x + 2*y
torch.div(x, y, rounding_mode='floor') # 整数除法,'trunc' 截断矩阵运算
x = torch.randn(3, 4) # (3,4)
y = torch.randn(4, 5) # (4,5)
z = x @ y # (3,5)
q = torch.randn(2, 3, 4) # 批量:前面维度广播,最后两维做矩阵乘
k = torch.randn(2, 4, 5)
q @ k # (2,3,5)| 函数 | 规则 |
|---|---|
x @ y / torch.matmul(x, y) | 通用:1D/2D/批量都行 |
torch.mm(a, b) | 只收 2D,报错更早 |
torch.bmm(a, b) | 只收 3D 批量,batch 维必须一致(不广播) |
torch.einsum('bij,bjk->bik', q, k) | 爱因斯坦求和:字符串里写清每个轴,-> 前是输入轴、后是输出轴。批量矩阵乘、点积、转置、trace 全都能写 |
einsum 就是最自由的矩阵语法
torch.einsum('i,i->', a, b) # 点积 → 标量
torch.einsum('ij->ji', a) # 转置
torch.einsum('bik,bjk->bij', q, k) # 批量"qk^T"注意力打分归约运算
| 函数 | 作用 |
|---|---|
x.sum() / x.mean() / x.prod() | 求和 / 平均 / 连乘(全归约 → 标量) |
x.sum(dim=1) | 沿 dim=1 归约,该轴消失 |
x.max() / x.min() | 全局最大 / 最小(标量) |
x.argmax(dim=1) | 沿 dim=1 最大值的索引 |
x.std() / x.var() | 标准差 / 方差 |
x.any() / x.all() | bool 归约:有没有真 / 是不是全真 |
x.count_nonzero() | 非零元素个数 |
x = torch.randn(3, 4) # (3,4)
x.sum(dim=1) # (3,) ← 轴 1 被消掉
x.sum(dim=1, keepdim=True) # (3,1) ← 保留一个空壳轴,方便广播
x.sum() # 标量Tensor Code
01_ ReLU
Implement the ReLU (Rectified Linear Unit) activation function from scratch.
ReLU(x) = max(0,x)
Signature
def relu(x: torch.Tensor) -> torch.Tensor:
...Rules
- Do NOT use
torch.relu,F.relu,torch.clamp, or any built-in activation - Must support autograd (gradients should flow back)
Example
Input: tensor([-2., -1., 0., 1., 2.])
Output: tensor([ 0., 0., 0., 1., 2.])
💡 Hint for Implement ReLU:
ReLU(x) = max(0, x). Think about element-wise comparison with zero.import torch
# ✏️ YOUR IMPLEMENTATION HERE
def relu(x: torch.Tensor) -> torch.Tensor:
# return torch.maximum(x, torch.zeros_like(x))
# return x.clamp(min = 0)
return x * (x > 0).float()
| 写法 | x=0 处梯度 |
|---|---|
torch.maximum(x, zeros_like(x)) | 0.5(相等处均分) |
x.clamp(min=0) | 1.0(含等号) |
x * (x > 0).float() | 0.0(严格大于,0 不入选) |
x * (x >= 0).float() | 1.0(含等号) |
02_Softmax
Implement the Softmax function from scratch.
softmax(x_i)=\frac{e^{x_i}}{\sum_j e^{x_j}}Signature
def my_softmax(x: torch.Tensor, dim: int = -1) -> torch.Tensor:
...Rules
- Do NOT use
torch.softmax,F.softmax, ortorch.nn.Softmax - Must be numerically stable (hint: subtract
maxbeforeexp)
Example
Input: tensor([1., 2., 3.])
Output: tensor([0.0900, 0.2447, 0.6652]) # sums to 1.0
💡 Hint for Implement Softmax:
softmax(x)_i = exp(x_i) / sum(exp(x_j)). Subtract max(x) first for numerical stability.import torch
def my_softmax(x: torch.Tensor, dim: int = -1) -> torch.Tensor:
# pass # Replace this
x = x - x.max(dim=dim, keepdim=True).values
# └─ .values:max(dim=...) 返回 (最大值, 下标) 两个东西,取第一个
# └─ keepdim=True:保住维度形状,减法才能广播
e = torch.exp(x)
# ② 归一化:沿 dim 求和,同样 keepdim
return e / e.sum(dim=dim, keepdim=True)减去最大值不影响softmax最终的值,但可以维持自然指数的数值稳定性
dim参数限定操作轴,如果不指定keepdim=True则该维度消失
03_Linear
Implement a fully-connected linear layer: y = xW^T + b
Signature
class SimpleLinear:
def __init__(self, in_features: int, out_features: int): ...
def forward(self, x: torch.Tensor) -> torch.Tensor: ...Requirements
self.weight: shape(out_features, in_features), init withrandn * (1/√in_features)self.bias: shape(out_features,), init as zeros- Both must have
requires_grad=True forward(x)computesx @ W^T + b- Do NOT use
torch.nn.Linear
import torch
import math
# ✏️ YOUR IMPLEMENTATION HERE
class SimpleLinear:
def __init__(self, in_features: int, out_features: int):
# pass # Initialize weight and bias
self.weight = torch.zeros(out_features, in_features, requires_grad = True)
self.bias = torch.zeros(out_features, requires_grad = True)
def forward(self, x: torch.Tensor) -> torch.Tensor:
# pass # Compute y = x @ W^T + b
return x @ self.weight.transpose(-2,-1) + self.bias
transpose(a,b)交换ab维转置,bias被广播
之前仅对传入张量直接做运算,对于需要计算梯度的参数需要手动加requires_grad = True
04_Layernorm
Implement Layer Normalization from scratch.
LayerNorm(x) = \gamma \cdot \frac{x - \mu}{\sqrt{\sigma^2+\epsilon}}+\betawhere and are computed over the last dimension.
Signature
def my_layer_norm(
x: torch.Tensor, # input
gamma: torch.Tensor, # scale (same size as last dim)
beta: torch.Tensor, # shift (same size as last dim)
eps: float = 1e-5
) -> torch.Tensor:
...
Rules
Must support autograd
Do NOT use F.layer_norm or torch.nn.LayerNorm
Normalize over the last dimension only
import torch
# ✏️ YOUR IMPLEMENTATION HERE
def my_layer_norm(x, gamma, beta, eps=1e-5):
mean = x.mean(dim=-1, keepdim=True) # 每行自己的均值
var = x.var(dim=-1, keepdim=True, unbiased=False) # 每行自己的总体方差
return (x - mean) / (var + eps).sqrt() * gamma + beta 由于计算方差时,我们掌握全部数据,因此是无偏估计,需要参数unbiased=False
在运算后进行线性变换是为了恢复数值的表征能力,避免因归一化造成表达坍缩
05_Attention
Implement the core attention mechanism used in Transformers.
Attention(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}})VSignature
def scaled_dot_product_attention(
Q: torch.Tensor, # (batch, seq_q, d_k)
K: torch.Tensor, # (batch, seq_k, d_k)
V: torch.Tensor, # (batch, seq_k, d_v)
) -> torch.Tensor: # (batch, seq_q, d_v)
...
Rules
- Do NOT use
F.scaled_dot_product_attention - You may use
torch.softmaxandtorch.bmm - Must support autograd
- Must handle cross-attention (seq_q ≠ seq_k)
import torch
import math
# ✏️ YOUR IMPLEMENTATION HERE
def scaled_dot_product_attention(Q, K, V):
_, _, D = Q.shape
score = Q @ K.transpose(-1,-2) #(batch, seq_q, seq_k)
score = score / math.sqrt(D)
return torch.softmax(score, dim = -1) @ V06_Multihead_attention
Implement Multi-Head Attention from scratch — the core building block of the Transformer.
MultiHead(QKV)=Concat(head_1,....,head_n)W^o \\ head_i = Attention(QW_i^Q,KW_i^K,VW_i^V)
Signature
class MultiHeadAttention:
def __init__(self, d_model: int, num_heads: int): ...
def forward(self, Q, K, V) -> torch.Tensor: ...
Requirements
- Use
nn.Linear(d_model, d_model)forself.W_q,self.W_k,self.W_v,self.W_o d_k = d_model // num_headsper headforward(Q, K, V): Q is(B, seq_q, d_model), K/V are(B, seq_k, d_model)- Must support cross-attention (
seq_q != seq_k) - Do NOT use
torch.nn.MultiheadAttention - You may use
torch.softmaxandtorch.matmul
Steps
- Project:
q = self.W_q(Q),k = self.W_k(K),v = self.W_v(V) - Reshape to
(B, num_heads, seq, d_k) - Scaled dot-product attention per head
- Concat heads →
(B, seq_q, d_model) - Output projection:
self.W_o(concat)
import torch
import torch.nn as nn
import math
# ✏️ YOUR IMPLEMENTATION HERE
class MultiHeadAttention:
def __init__(self, d_model: int, num_heads: int):
# Initialize W_q, W_k, W_v, W_o
self.d_model = d_model
self.num_heads = num_heads
self.d_k = d_model // num_heads
self.W_q = nn.Linear(d_model, d_model)
self.W_k = nn.Linear(d_model, d_model)
self.W_v = nn.Linear(d_model, d_model)
self.W_o = nn.Linear(d_model, d_model)
def forward(self, Q, K, V):
# Implement multi-head attention
B, s_q, d = Q.shape
_, s_k, _ = K.shape
q = self.W_q(Q) #(B, seq_q, d_model)
k = self.W_k(K)
v = self.W_v(V)
q_split = q.view(B, s_q, self.num_heads, self.d_k).transpose(1,2) #(B, num_heads, seq_q, d_k)
k_split = k.view(B, s_k, self.num_heads, self.d_k).transpose(1,2)
v_split = v.view(B, s_k, self.num_heads, self.d_k).transpose(1,2)
attn = torch.softmax(q_split @ k_split.transpose(-1,-2) / math.sqrt(self.d_k),dim = -1) @ v_split
combined = attn.transpose(1,2).contiguous().view(B, s_q ,d)
return self.W_o(combined)transpose()不保证内存连续,而view()要求内存连续,因此需要先用contiguous()处理
07_Batchnorm
Implement Batch Normalization with both training and inference behavior.
In training mode, use batch statistics and update running estimates:
BN(x)=\gamma \cdot \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} + \betawhere and are the mean and variance computed across the batch (dim=0).
In inference mode, use the provided running mean/var instead of current batch stats.
Signature
def my_batch_norm(
x: torch.Tensor,
gamma: torch.Tensor,
beta: torch.Tensor,
running_mean: torch.Tensor,
running_var: torch.Tensor,
eps: float = 1e-5,
momentum: float = 0.1,
training: bool = True,
) -> torch.Tensor:
# x: (N, D) — normalize each feature across all samples in the batch
# running_mean, running_var: updated in-place during training; used as-is during inference
Rules
- Do NOT use
F.batch_norm,nn.BatchNorm1d, etc. - Compute batch mean and variance over
dim=0withunbiased=False - Update running stats like PyTorch:
running = (1 - momentum) * running + momentum * batch_stat - Use
running_mean/running_varfor inference whentraining=False - Must support autograd w.r.t.
x,gamma,beta(running statistics should be treated as buffers, not parameters requiring gradients)
Batchnorm和Layernorm的区别是前者在样本内归一化而后者在整个批次的数据中做归一化,换句话说是对特征取了均值,Batch Norm主要用在CNN等结构上。
对于序列信息,LayerNorm按行来归一化保证了每个样本有归一化的信号强度,样本间的差异被抹平,保留了样本内部的特征相对大小。保留语义信息。
对于并行信息,BatchNorm按列来归一化保证了每个特征有归一化的信号强度,特征间的差异被抹平,保留了特征内部的各个样本的差异。保留样本间特征的相对差异。
对于transformer这样的任务中不能使用BatchNorm,因为batchmate的信息会串通到序列中导致语义不可复现,而且Batch的平均句子没有任何意义。同时序列的长度也不一致,padding后的全零项没有意义。在CNN这样的任务中LN的表现不如BN,因为特征得到的归一化特征更不容易损失信息。
为什么 CNN 里能忍,Transformer 里不能
| CNN(图像) | Transformer(文本) | |
|---|---|---|
| 每通道统计量的样本数 | N×H×W ≈ 32×224×224 ≈ 160 万个值 | 每个位置只有 N = 32 个值(batch=32) |
| 测量信噪比 | 极高——统计量几乎不抖 | 极低——校准本身是噪声源 |
| 批内样本关系 | 同一平稳过程的i.i.d(identically distributed)样本,”典型图像”的通道统计是稳定、有意义的参考 | 语义独立的句子,token 级激活分布由内容决定,批内平均没有意义 |
| 被平均的东西 | 像素——可交换(exchangeable),一个像素和另一个像素同类,都是光强样本 | token——有语义身份和位置信息 |
| 任务容忍度 | 分类 loss 逐样本独立,耦合只是噪声(甚至当正则用) | 生成时输出必须是本序列的纯函数 |
需要使用runing_mean和runing_var变量的原因是推理的时候不能用批次统计,假如batch=1,则归一化后参数归零。推理时需要使用训练期间累积的长期校准值。
import torch
# ✏️ YOUR IMPLEMENTATION HERE
def my_batch_norm(
x,
gamma,
beta,
running_mean,
running_var,
eps=1e-5,
momentum=0.1,
training=True,
):
if training:
mean = x.mean(dim=0)
var = x.var(dim=0, unbiased=False)
running_mean.mul_(1 - momentum).add_(momentum * mean.detach())
running_var.mul_(1 - momentum).add_(momentum * var.detach())
else:
mean, var = running_mean, running_var
return gamma * (x - mean) / torch.sqrt(var + eps) + beta.detach方法切断了mean和var的梯度,避免mean和var在反向传播中被更改。
08_RMSNorm
Implement Root Mean Square Layer Normalization — the normalization used in LLaMA, Gemma, etc.
RMSNorm(x) = \frac{x}{RMS(x)}\cdot w, RMS(x)=\sqrt{\frac{1}{d}\sum{x_i^2}+\epsilon}Signature
def rms_norm(x: torch.Tensor, weight: torch.Tensor, eps: float = 1e-6) -> torch.Tensor:
# Normalize over the last dimension. No mean subtraction (unlike LayerNorm).
Rules
- Do NOT use any built-in norm layers
- Normalize over
dim=-1 - Must support autograd
import torch
# ✏️ YOUR IMPLEMENTATION HERE
def rms_norm(x, weight, eps=1e-6):
rms = torch.sqrt(x.pow(2).mean(dim=-1, keepdim=True) + eps)
return x / rms * weight 这里要注意
9_Causal Self-Attention
Implement causal (masked) self-attention — the attention used in GPT-style decoders.
Same as softmax attention, but each position can only attend to itself and earlier positions (no peeking at future tokens).
text{scores}_{ij} = \begin{cases} \frac{Q_i \cdot K_j}{\sqrt{d_k}} & \text{if } j \le i \\ -\infty & \text{if } j > i \end{cases}Signature
def causal_attention(Q, K, V):
# Q, K, V: (batch, seq, d) → output: (batch, seq, d_v)
Rules
- Do NOT use
F.scaled_dot_product_attention - Position can only attend to positions
- You may use
torch.softmax,torch.bmm,torch.triu
import torch
import math
# ✏️ YOUR IMPLEMENTATION HERE
def causal_attention(Q, K, V):
b, s, d = Q.shape
atten = Q @ K.transpose(-1, -2) / math.sqrt(d)
mask = torch.triu(torch.ones_like(atten ,dtype = torch.bool), diagonal = 1)
atten = atten.masked_fill(mask, float('-inf'))
return torch.softmax(atten, dim = -1) @ V10_Grouped Query Attention (GQA)
Implement Grouped Query Attention — used in LLaMA 2, Mistral, etc. to reduce KV cache size.
Like MHA, but with fewer KV heads than Q heads. Each group of Q heads shares the same K/V head.
Signature
class GroupQueryAttention:
def __init__(self, d_model: int, num_heads: int, num_kv_heads: int): ...
def forward(self, x) -> torch.Tensor: # self-attention
Requirements
self.W_q:nn.Linear(d_model, d_model)— full Q projectionself.W_k:nn.Linear(d_model, num_kv_heads * d_k)— reduced K projectionself.W_v:nn.Linear(d_model, num_kv_heads * d_k)— reduced V projectionself.W_o:nn.Linear(d_model, d_model)— output projectiond_k = d_model // num_heads- Expand KV heads with
repeat_interleaveto match Q heads - When
num_kv_heads == num_heads, should behave like standard MHA
import torch
import torch.nn as nn
import math
# ✏️ YOUR IMPLEMENTATION HERE
class GroupQueryAttention:
def __init__(self, d_model, num_heads, num_kv_heads):
self.d_model = d_model
self.num_kv_heads = num_kv_heads
self.num_heads = num_heads
self.g = num_heads // num_kv_heads
self.d_k = self.d_model // self.num_heads
self.W_q = nn.Linear(d_model, d_model)
self.W_k = nn.Linear(d_model, num_kv_heads * self.d_k)
self.W_v = nn.Linear(d_model, num_kv_heads * self.d_k)
self.W_o = nn.Linear(d_model, d_model)
def forward(self, x): #(b, s, d)
b, s, _ = x.shape
q = self.W_q(x) #(b, s, d)
k = self.W_k(x).view(b, s, self.num_kv_heads , self.d_k).repeat_interleave(self.g, dim = 2).transpose(1,2) #(b, n, s, d_k)
v = self.W_v(x).view(b, s, self.num_kv_heads , self.d_k).repeat_interleave(self.g, dim = 2).transpose(1,2) #(b, n, s, d_k)
q_head = q.view(b, s, self.num_heads, self.d_k).transpose(1,2)#(b, n, s, d_k)
score = q_head @ k.transpose(-1,-2) #(b, n, s, s)
atten = (torch.softmax(score, dim = -1) @ v).transpose(1,2) #(b, s, n, d_k)
return atten.contiguous().view(b,s,self.d_model)
关键语法.repeat_interleave(self.g, dim = 2)