Tensor

Python 
   └─ torch._C.TensorBase
        └─ C++ at::Tensor ← 两个指针
             ├─ → TensorImpl   ← 数值区:data_ptr(数据) / sizes(形状) / strides(步长) / device(设备)
             └─ → AutogradMeta ← 梯度区

AutogradMeta中

字段Python 层装什么
grad_x.gradbackward 算完的梯度值。平时是空的,backward 才填
grad_fn_x.grad_fn指向反向节点的指针(伴随模型)。None = 叶子
grad_accumulator_不可见叶子的梯度累加器——AccumulateGrad 节点

tensor本质是一个多维数组,shape方法可以查询每一个轴(dim)的长度。dim的角标以0开始,-1结束。

tensor运算时会进行广播,原则是从右向左对齐,长度为1或缺失的维度会被复制。

x * y # 广播数乘
x @ y # 矩阵乘法

创建Tensor

有多种方法可以创建张量

torch.tensor([1,2,3])	    #(3,)     从 Python 数据建(会拷贝)
torch.zeros(2, 3)	        #(2,3)    全 0
torch.ones(2, 3)	        #(2,3)    全 1
torch.full((2,3), 7)	    #(2,3)    全 7(size 要传元组)
torch.arange(0, 10, 2)	  #(5,)     左闭右开 [0,10),步长 2
torch.linspace(0, 1, 5)	  #(5,)     闭区间 [0,1] 均匀取 5 个点
torch.eye(3)	            #(3,3)    单位矩阵
torch.randn(2, 3)	        #(2,3)    标准正态 N(0,1)
torch.rand(2, 3)	        #(2,3)    [0,1) 均匀分布
torch.empty(2, 3)	        #(2,3)    不初始化,垃圾值
torch.zeros_like(x)	      #同x形状/dtype 跟x完全一样(ones_like/full_like/randn_like 同理)
x.clone()	                #同x深拷贝(改副本不动原 tensor)

创建张量时常用参数有

参数作用
dtype数据类型:torch.float32(默认)/ torch.int64 / torch.bool / torch.float16
device放哪:'cpu' 或 'cuda'
requires_gradTrue 时记录梯度(训练用)
layout内存布局,默认 torch.strided;稀疏用 torch.sparse_coo
pin_memoryTrue 时用锁页内存,CPU→GPU 拷贝更快

形状操作

tensor有如下形状操作:

x.view(2, 6)	      #(3,4) → (2,6)	    共享底层内存(视图),元素总数必须一致;要求内存连续
x.reshape(2, 6)	    #(3,4) → (2,6)	    同 view 但自动处理不连续,更安全
x.permute(1, 0)	    #(3,4) → (4,3)	    按给定顺序重排所有轴(不拷贝数据)
x.transpose(0, 1)	  #(3,4) → (4,3)	    只交换两个轴
x.t() / x.T	        #2D 转置	            快捷方式
x.mT	              #任意维,交换最后两轴  批量矩阵转置
x.squeeze(0)	      #(1,3,4) → (3,4)	  去掉指定轴(不指定就去掉所有长度为 1 的轴)
x.unsqueeze(1)	    #(3,4) → (3,1,4)	  在 dim=1 插入长度 1 的新轴
x[:, None]	        #(5,) → (5,1)       你已会,就是 unsqueeze(1) 的下标写法
x.expand(2, 3, 4)	  #(1,3,4) → (2,3,4)	广播式扩展,不拷贝;原轴必须是 1;-1 表示保持原长
x.repeat(2, 1, 1)	  #(1,3,4) → (2,3,4)	真拷贝复制
x.flatten()	        #(2,3,4) → (24,)	  全拉平
x.flatten(1)	      #(2,3,4) → (2,12)	  从 dim=1 开始拉平

permute()和transpose()都会使得内存不连续,因此需要.contiguous()才能.view()

数学运算

tenser的数学运算如下所示:

类别写法
四则+ - * / **(幂)//(整除)%(取余)
函数版torch.add sub mul div pow sqrt rsqrt exp log log2 log10 abs neg
三角sin cos tan arctan …
取整floor(向下)ceil(向上)round(四舍五入)trunc(截断)sign(符号)
限幅torch.clamp(x, min=-1, max=1) —— 低于 min 拉回 min,高于 max 拉回 max
比较== != > < >= <= —— 逐元素,返回 bool tensor
逻辑torch.logical_and logical_or logical_not(或 & | ~,注意括号)
# 有些含有参数
torch.add(x, y, alpha=2)          # x + 2*y
torch.div(x, y, rounding_mode='floor')  # 整数除法,'trunc' 截断

矩阵运算

x = torch.randn(3, 4)      # (3,4)
y = torch.randn(4, 5)      # (4,5)
z = x @ y                  # (3,5)

q = torch.randn(2, 3, 4)   # 批量:前面维度广播,最后两维做矩阵乘
k = torch.randn(2, 4, 5)
q @ k                      # (2,3,5)
函数规则
x @ y / torch.matmul(x, y)通用:1D/2D/批量都行
torch.mm(a, b)只收 2D,报错更早
torch.bmm(a, b)只收 3D 批量,batch 维必须一致(不广播)
torch.einsum('bij,bjk->bik', q, k)爱因斯坦求和:字符串里写清每个轴,-> 前是输入轴、后是输出轴。批量矩阵乘、点积、转置、trace 全都能写

einsum 就是最自由的矩阵语法

torch.einsum('i,i->', a, b)        # 点积 → 标量
torch.einsum('ij->ji', a)          # 转置
torch.einsum('bik,bjk->bij', q, k) # 批量"qk^T"注意力打分

归约运算

函数作用
x.sum() / x.mean() / x.prod()求和 / 平均 / 连乘(全归约 → 标量)
x.sum(dim=1)沿 dim=1 归约,该轴消失
x.max() / x.min()全局最大 / 最小(标量)
x.argmax(dim=1)沿 dim=1 最大值的索引
x.std() / x.var()标准差 / 方差
x.any() / x.all()bool 归约:有没有真 / 是不是全真
x.count_nonzero()非零元素个数
x = torch.randn(3, 4)              # (3,4)
x.sum(dim=1)                       # (3,)      ← 轴 1 被消掉
x.sum(dim=1, keepdim=True)         # (3,1)     ← 保留一个空壳轴,方便广播
x.sum()                            # 标量

Tensor Code

01_ ReLU

Implement the ReLU (Rectified Linear Unit) activation function from scratch.

ReLU(x) = max(0,x)

Signature

def relu(x: torch.Tensor) -> torch.Tensor:
    ...

Rules

  • Do NOT use torch.relu, F.relu, torch.clamp, or any built-in activation
  • Must support autograd (gradients should flow back)

Example

Input:  tensor([-2., -1., 0., 1., 2.])
Output: tensor([ 0.,  0., 0., 1., 2.])

💡 Hint for Implement ReLU:
   ReLU(x) = max(0, x). Think about element-wise comparison with zero.
import torch
# ✏️ YOUR IMPLEMENTATION HERE

def relu(x: torch.Tensor) -> torch.Tensor:
    # return torch.maximum(x, torch.zeros_like(x))
    # return x.clamp(min = 0)
    return x * (x > 0).float()
写法x=0 处梯度
torch.maximum(x, zeros_like(x))0.5(相等处均分)
x.clamp(min=0)1.0(含等号)
x * (x > 0).float()0.0(严格大于,0 不入选)
x * (x >= 0).float()1.0(含等号)

02_Softmax

Implement the Softmax function from scratch.

softmax(x_i)=\frac{e^{x_i}}{\sum_j e^{x_j}}

Signature

def my_softmax(x: torch.Tensor, dim: int = -1) -> torch.Tensor:
    ...

Rules

  • Do NOT use torch.softmax, F.softmax, or torch.nn.Softmax
  • Must be numerically stable (hint: subtract max before exp)

Example

Input:  tensor([1., 2., 3.])
Output: tensor([0.0900, 0.2447, 0.6652])  # sums to 1.0
💡 Hint for Implement Softmax:
   softmax(x)_i = exp(x_i) / sum(exp(x_j)). Subtract max(x) first for numerical stability.
import torch
def my_softmax(x: torch.Tensor, dim: int = -1) -> torch.Tensor:
    # pass  # Replace this
    x = x - x.max(dim=dim, keepdim=True).values
    #    └─ .values:max(dim=...) 返回 (最大值, 下标) 两个东西,取第一个
    #    └─ keepdim=True:保住维度形状,减法才能广播
    e = torch.exp(x)
    # ② 归一化:沿 dim 求和,同样 keepdim
    return e / e.sum(dim=dim, keepdim=True)

减去最大值不影响softmax最终的值,但可以维持自然指数的数值稳定性

dim参数限定操作轴,如果不指定keepdim=True则该维度消失

03_Linear

Implement a fully-connected linear layer: y = xW^T + b

Signature

class SimpleLinear:
    def __init__(self, in_features: int, out_features: int): ...
    def forward(self, x: torch.Tensor) -> torch.Tensor: ...

Requirements

  • self.weight: shape (out_features, in_features), init with randn * (1/√in_features)
  • self.bias: shape (out_features,), init as zeros
  • Both must have requires_grad=True
  • forward(x) computes x @ W^T + b
  • Do NOT use torch.nn.Linear
import torch
import math
# ✏️ YOUR IMPLEMENTATION HERE

class SimpleLinear:
    def __init__(self, in_features: int, out_features: int):
        # pass  # Initialize weight and bias
        self.weight = torch.zeros(out_features, in_features, requires_grad = True)
        self.bias = torch.zeros(out_features, requires_grad = True)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # pass  # Compute y = x @ W^T + b
        return x @ self.weight.transpose(-2,-1) + self.bias
        

transpose(a,b)交换ab维转置,bias被广播

之前仅对传入张量直接做运算,对于需要计算梯度的参数需要手动加requires_grad = True

04_Layernorm

Implement Layer Normalization from scratch.

LayerNorm(x) = \gamma \cdot \frac{x - \mu}{\sqrt{\sigma^2+\epsilon}}+\beta

where μ\mu and σ2\sigma^2 are computed over the last dimension.

Signature

def my_layer_norm(
    x: torch.Tensor,      # input
    gamma: torch.Tensor,   # scale (same size as last dim)
    beta: torch.Tensor,    # shift (same size as last dim)
    eps: float = 1e-5
) -> torch.Tensor:
    ...

Rules

Must support autograd

Do NOT use F.layer_norm or torch.nn.LayerNorm

Normalize over the last dimension only

import torch
# ✏️ YOUR IMPLEMENTATION HERE

def my_layer_norm(x, gamma, beta, eps=1e-5):
    mean = x.mean(dim=-1, keepdim=True)                    # 每行自己的均值
    var  = x.var(dim=-1, keepdim=True, unbiased=False)     # 每行自己的总体方差
    return (x - mean) / (var + eps).sqrt() * gamma + beta  

由于计算方差时,我们掌握全部数据,因此是无偏估计,需要参数unbiased=False

在运算后进行线性变换是为了恢复数值的表征能力,避免因归一化造成表达坍缩

05_Attention

Implement the core attention mechanism used in Transformers.

Attention(Q,K,V) =  softmax(\frac{QK^T}{\sqrt{d_k}})V

Signature

def scaled_dot_product_attention(
    Q: torch.Tensor,  # (batch, seq_q, d_k)
    K: torch.Tensor,  # (batch, seq_k, d_k)
    V: torch.Tensor,  # (batch, seq_k, d_v)
) -> torch.Tensor:   # (batch, seq_q, d_v)
    ...

Rules

  • Do NOT use F.scaled_dot_product_attention
  • You may use torch.softmax and torch.bmm
  • Must support autograd
  • Must handle cross-attention (seq_q ≠ seq_k)
import torch
import math
# ✏️ YOUR IMPLEMENTATION HERE

def scaled_dot_product_attention(Q, K, V):
    _, _, D = Q.shape
    score = Q @ K.transpose(-1,-2) #(batch, seq_q, seq_k)
    score = score / math.sqrt(D)
    return torch.softmax(score, dim = -1) @ V

06_Multihead_attention

Implement Multi-Head Attention from scratch — the core building block of the Transformer.

MultiHead(QKV)=Concat(head_1,....,head_n)W^o \\ head_i = Attention(QW_i^Q,KW_i^K,VW_i^V)

Signature

class MultiHeadAttention:
    def __init__(self, d_model: int, num_heads: int): ...
    def forward(self, Q, K, V) -> torch.Tensor: ...

Requirements

  • Use nn.Linear(d_model, d_model) for self.W_q, self.W_k, self.W_v, self.W_o
  • d_k = d_model // num_heads per head
  • forward(Q, K, V): Q is (B, seq_q, d_model), K/V are (B, seq_k, d_model)
  • Must support cross-attention (seq_q != seq_k)
  • Do NOT use torch.nn.MultiheadAttention
  • You may use torch.softmax and torch.matmul

Steps

  1. Project: q = self.W_q(Q), k = self.W_k(K), v = self.W_v(V)
  2. Reshape to (B, num_heads, seq, d_k)
  3. Scaled dot-product attention per head
  4. Concat heads → (B, seq_q, d_model)
  5. Output projection: self.W_o(concat)
import torch
import torch.nn as nn
import math

# ✏️ YOUR IMPLEMENTATION HERE

class MultiHeadAttention:
    def __init__(self, d_model: int, num_heads: int):
        # Initialize W_q, W_k, W_v, W_o
        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads
        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)

    def forward(self, Q, K, V):
        # Implement multi-head attention
        B, s_q, d = Q.shape
        _, s_k, _ = K.shape
        
        q = self.W_q(Q) #(B, seq_q, d_model)
        k = self.W_k(K)
        v = self.W_v(V)

        q_split = q.view(B, s_q, self.num_heads, self.d_k).transpose(1,2) #(B, num_heads, seq_q, d_k)
        k_split = k.view(B, s_k, self.num_heads, self.d_k).transpose(1,2)
        v_split = v.view(B, s_k, self.num_heads, self.d_k).transpose(1,2)

        attn = torch.softmax(q_split @ k_split.transpose(-1,-2) / math.sqrt(self.d_k),dim = -1) @ v_split
        combined = attn.transpose(1,2).contiguous().view(B, s_q ,d)
        return self.W_o(combined)

transpose()不保证内存连续,而view()要求内存连续,因此需要先用contiguous()处理

07_Batchnorm

Implement Batch Normalization with both training and inference behavior.

In training mode, use batch statistics and update running estimates:

BN(x)=\gamma \cdot \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} + \beta

where μB\mu_B and σB2\sigma_B^2 are the mean and variance computed across the batch (dim=0).

In inference mode, use the provided running mean/var instead of current batch stats.

Signature

def my_batch_norm(
    x: torch.Tensor,
    gamma: torch.Tensor,
    beta: torch.Tensor,
    running_mean: torch.Tensor,
    running_var: torch.Tensor,
    eps: float = 1e-5,
    momentum: float = 0.1,
    training: bool = True,
) -> torch.Tensor:
    # x: (N, D) — normalize each feature across all samples in the batch
    # running_mean, running_var: updated in-place during training; used as-is during inference

Rules

  • Do NOT use F.batch_norm, nn.BatchNorm1d, etc.
  • Compute batch mean and variance over dim=0 with unbiased=False
  • Update running stats like PyTorch: running = (1 - momentum) * running + momentum * batch_stat
  • Use running_mean / running_var for inference when training=False
  • Must support autograd w.r.t. x, gamma, beta (running statistics should be treated as buffers, not parameters requiring gradients)

Batchnorm和Layernorm的区别是前者在样本内归一化而后者在整个批次的数据中做归一化,换句话说是对特征取了均值,Batch Norm主要用在CNN等结构上。

对于序列信息,LayerNorm按行来归一化保证了每个样本有归一化的信号强度,样本间的差异被抹平,保留了样本内部的特征相对大小。保留语义信息。

对于并行信息,BatchNorm按列来归一化保证了每个特征有归一化的信号强度,特征间的差异被抹平,保留了特征内部的各个样本的差异。保留样本间特征的相对差异。

对于transformer这样的任务中不能使用BatchNorm,因为batchmate的信息会串通到序列中导致语义不可复现,而且Batch的平均句子没有任何意义。同时序列的长度也不一致,padding后的全零项没有意义。在CNN这样的任务中LN的表现不如BN,因为特征得到的归一化特征更不容易损失信息。

为什么 CNN 里能忍,Transformer 里不能

CNN(图像)Transformer(文本)
每通道统计量的样本数N×H×W ≈ 32×224×224 ≈ 160 万个值每个位置只有 N = 32 个值(batch=32)
测量信噪比极高——统计量几乎不抖极低——校准本身是噪声源
批内样本关系同一平稳过程的i.i.d(identically distributed)样本,”典型图像”的通道统计是稳定、有意义的参考语义独立的句子,token 级激活分布由内容决定,批内平均没有意义
被平均的东西像素——可交换(exchangeable),一个像素和另一个像素同类,都是光强样本token——有语义身份和位置信息
任务容忍度分类 loss 逐样本独立,耦合只是噪声(甚至当正则用)生成时输出必须是本序列的纯函数

需要使用runing_mean和runing_var变量的原因是推理的时候不能用批次统计,假如batch=1,则归一化后参数归零。推理时需要使用训练期间累积的长期校准值。

import torch
# ✏️ YOUR IMPLEMENTATION HERE

def my_batch_norm(
    x,
    gamma,
    beta,
    running_mean,
    running_var,
    eps=1e-5,
    momentum=0.1,
    training=True,
):
    if training:
        mean = x.mean(dim=0)
        var = x.var(dim=0, unbiased=False)
        running_mean.mul_(1 - momentum).add_(momentum * mean.detach())
        running_var.mul_(1 - momentum).add_(momentum * var.detach())
    else:
        mean, var = running_mean, running_var
    return gamma * (x - mean) / torch.sqrt(var + eps) + beta

.detach方法切断了mean和var的梯度,避免mean和var在反向传播中被更改。

08_RMSNorm

Implement Root Mean Square Layer Normalization — the normalization used in LLaMA, Gemma, etc.

RMSNorm(x) = \frac{x}{RMS(x)}\cdot w, RMS(x)=\sqrt{\frac{1}{d}\sum{x_i^2}+\epsilon}

Signature

def rms_norm(x: torch.Tensor, weight: torch.Tensor, eps: float = 1e-6) -> torch.Tensor:
    # Normalize over the last dimension. No mean subtraction (unlike LayerNorm).

Rules

  • Do NOT use any built-in norm layers
  • Normalize over dim=-1
  • Must support autograd
import torch
# ✏️ YOUR IMPLEMENTATION HERE

def rms_norm(x, weight, eps=1e-6):
    rms = torch.sqrt(x.pow(2).mean(dim=-1, keepdim=True) + eps)
    return x / rms * weight 

这里要注意rms2=var+mean2rms^2 = var + mean^2

9_Causal Self-Attention

Implement causal (masked) self-attention — the attention used in GPT-style decoders.

Same as softmax attention, but each position can only attend to itself and earlier positions (no peeking at future tokens).

text{scores}_{ij} = \begin{cases} \frac{Q_i \cdot K_j}{\sqrt{d_k}} & \text{if } j \le i \\ -\infty & \text{if } j > i \end{cases}

Signature

def causal_attention(Q, K, V):
    # Q, K, V: (batch, seq, d) → output: (batch, seq, d_v)

Rules

  • Do NOT use F.scaled_dot_product_attention
  • Position  can only attend to positions 
  • You may use torch.softmax, torch.bmm, torch.triu
import torch
import math

# ✏️ YOUR IMPLEMENTATION HERE

def causal_attention(Q, K, V):
    b, s, d = Q.shape
    atten = Q @ K.transpose(-1, -2) / math.sqrt(d)
    mask = torch.triu(torch.ones_like(atten ,dtype = torch.bool), diagonal = 1)
    atten = atten.masked_fill(mask, float('-inf'))
    return torch.softmax(atten, dim = -1) @ V

10_Grouped Query Attention (GQA)

Implement Grouped Query Attention — used in LLaMA 2, Mistral, etc. to reduce KV cache size.

Like MHA, but with fewer KV heads than Q heads. Each group of Q heads shares the same K/V head.

Signature

class GroupQueryAttention:
    def __init__(self, d_model: int, num_heads: int, num_kv_heads: int): ...
    def forward(self, x) -> torch.Tensor:  # self-attention

Requirements

  • self.W_q: nn.Linear(d_model, d_model) — full Q projection
  • self.W_k: nn.Linear(d_model, num_kv_heads * d_k) — reduced K projection
  • self.W_v: nn.Linear(d_model, num_kv_heads * d_k) — reduced V projection
  • self.W_o: nn.Linear(d_model, d_model) — output projection
  • d_k = d_model // num_heads
  • Expand KV heads with repeat_interleave to match Q heads
  • When num_kv_heads == num_heads, should behave like standard MHA
import torch
import torch.nn as nn
import math
# ✏️ YOUR IMPLEMENTATION HERE

class GroupQueryAttention:
    def __init__(self, d_model, num_heads, num_kv_heads):
        self.d_model = d_model
        self.num_kv_heads = num_kv_heads
        self.num_heads = num_heads
        self.g = num_heads // num_kv_heads
        self.d_k = self.d_model // self.num_heads
        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, num_kv_heads * self.d_k)
        self.W_v = nn.Linear(d_model, num_kv_heads * self.d_k)
        self.W_o = nn.Linear(d_model, d_model)

    def forward(self, x): #(b, s, d)
        b, s, _ = x.shape
        q = self.W_q(x) #(b, s, d)
        k = self.W_k(x).view(b, s, self.num_kv_heads , self.d_k).repeat_interleave(self.g, dim = 2).transpose(1,2) #(b, n, s, d_k)
        v = self.W_v(x).view(b, s, self.num_kv_heads , self.d_k).repeat_interleave(self.g, dim = 2).transpose(1,2) #(b, n, s, d_k)
        q_head = q.view(b, s, self.num_heads, self.d_k).transpose(1,2)#(b, n, s, d_k)
        score = q_head @ k.transpose(-1,-2) #(b, n, s, s)
        atten = (torch.softmax(score, dim = -1) @ v).transpose(1,2) #(b, s, n, d_k)
        return atten.contiguous().view(b,s,self.d_model)
        
        

关键语法.repeat_interleave(self.g, dim = 2)

Related Posts