Linear Algebra

Vector: ordered list of numbers (a point in n-dimensional space)

  v = [v1, v2, ..., vn]    (row notation)
  v = (3, 4)               (2D vector)
  v = (1, 0, -2)           (3D vector)

Addition:       [a, b] + [c, d] = [a+c, b+d]
Scalar mult:    k * [a, b] = [ka, kb]
Magnitude:      |v| = sqrt(v1^2 + v2^2 + ... + vn^2)
Unit vector:    v̂ = v / |v|     (magnitude 1, same direction)

u · v = u1*v1 + u2*v2 + ... + un*vn    (scalar result)

Properties:
  u · v = |u| * |v| * cos(θ)    where θ = angle between u and v
  u · v = 0  iff  u ⊥ v          (perpendicular/orthogonal)
  u · u = |u|^2                  (square of magnitude)

Applications:
  Angle between vectors:  cos(θ) = (u · v) / (|u| * |v|)
  Projection of u onto v: proj_v(u) = (u · v / v · v) * v
  Similarity:             cosine similarity in ML/search

u × v = (u2*v3 - u3*v2, u3*v1 - u1*v3, u1*v2 - u2*v1)

Properties:
  |u × v| = |u| * |v| * sin(θ)    (magnitude = area of parallelogram)
  u × v is perpendicular to both u and v
  u × v = -(v × u)                (anti-commutative)
  u × u = 0

Matrix: rectangular array of numbers (m rows × n columns)

  A = | a  b |     2×2 matrix
      | c  d |

  A = | 1  2  3 |  2×3 matrix
      | 4  5  6 |

Addition:      Add element-wise (same dimensions required)
Scalar mult:   Multiply each element by scalar
Transpose:     A^T swaps rows and columns: (A^T)_{ij} = A_{ji}

Matrix multiplication (A is m×n, B is n×p → result is m×p):
  (AB)_{ij} = sum_{k=1}^{n} A_{ik} * B_{kj}

  | a  b |   | e  f |   | ae+bg  af+bh |
  | c  d | × | g  h | = | ce+dg  cf+dh |

  NOT commutative: AB != BA in general
  Associative:     (AB)C = A(BC)
  Distributive:    A(B+C) = AB + AC

Identity matrix I:
  | 1  0 |        AI = IA = A
  | 0  1 |

Inverse A^(-1) (only square matrices):
  A * A^(-1) = A^(-1) * A = I

  For 2×2:
  | a  b |^(-1) = (1/det) * |  d  -b |
  | c  d |                   | -c   a |

  where det = ad - bc
  Inverse exists iff det != 0

det | a  b | = ad - bc
    | c  d |

Geometric meaning: signed area of parallelogram formed by column vectors
If det = 0: matrix is singular (columns are linearly dependent)

det | a  b  c |
    | d  e  f | = a(ei - fh) - b(di - fg) + c(dh - eg)
    | g  h  i |

Expand along first row: alternating signs, multiply by 2×2 minors

det(AB) = det(A) * det(B)
det(A^T) = det(A)
det(A^(-1)) = 1 / det(A)
det(kA) = k^n * det(A)    (n×n matrix)
Swapping two rows negates the determinant
A row of zeros → det = 0
Two identical rows → det = 0

System:         2x + 3y = 8
                4x + y  = 6

Matrix form:    Ax = b

  | 2  3 | | x |   | 8 |
  | 4  1 | | y | = | 6 |

Solution: x = A^(-1)b   (if A is invertible)

Augmented matrix: [A | b]

  | 2  3 | 8 |     Row operations:
  | 4  1 | 6 |     R2 = R2 - 2*R1

  | 2  3 |  8 |
  | 0 -5 | -10|    R2 / -5

  | 2  3 | 8 |
  | 0  1 | 2 |     y = 2, back-substitute: 2x + 6 = 8, x = 1

Reduced row echelon form (RREF):
  | 1  0 | 1 |     Read solution directly: x = 1, y = 2
  | 0  1 | 2 |

Unique solution:     det(A) != 0, rank = n
No solution:         Inconsistent (parallel planes)
Infinite solutions:  rank < n (free variables)

Av = λv

Eigenvector v:  a non-zero vector whose direction is unchanged by A
Eigenvalue λ:   the scalar factor by which v is scaled

To find:
  det(A - λI) = 0    (characteristic equation)
  Solve for λ, then solve (A - λI)v = 0 for each λ

Example:
  A = | 2  1 |
      | 1  2 |

  det(A - λI) = (2-λ)^2 - 1 = λ^2 - 4λ + 3 = (λ-3)(λ-1) = 0
  λ1 = 3, λ2 = 1

  For λ1 = 3: (A - 3I)v = 0 → v1 = (1, 1)
  For λ2 = 1: (A - I)v = 0  → v2 = (1, -1)

PageRank:        dominant eigenvector of web graph
PCA:             eigenvectors of covariance matrix → principal components
Stability:       system stable if all |λ| < 1
Diagonalization: A = PDP^(-1) where D = diag(eigenvalues)
  Powers:        A^n = PD^nP^(-1)  (fast computation)
  Matrix exp:    e^A = Pe^DP^(-1)

Vector space: a set with addition and scalar multiplication satisfying axioms

Subspace:     subset that is itself a vector space
  Must contain zero vector
  Closed under addition and scalar multiplication

Span:         set of all linear combinations of given vectors
  span{v1, v2} = {a*v1 + b*v2 : a, b in R}

Linear independence:
  Vectors are independent if none is a linear combination of others
  {v1, ..., vn} independent iff c1*v1 + ... + cn*vn = 0 implies all ci = 0

Basis:        linearly independent set that spans the space
  Standard basis for R^3: {(1,0,0), (0,1,0), (0,0,1)}

Dimension:    number of vectors in a basis
  R^n has dimension n

Rank of matrix A:
  = number of linearly independent rows
  = number of linearly independent columns
  = number of pivot positions in RREF
  = dimension of column space

Rank-nullity theorem:
  rank(A) + nullity(A) = n    (number of columns)
  nullity = dimension of null space (solutions to Ax = 0)