Table of Content
Chapter 1 Mathematical Background
1.1 Infinite series
1.1.1 Geometric series
1.1.2 Binomial Series
1.2 Approximation
1.2.1 Taylor approximation
1.2.2 Exponential series
1.2.3 Logarithmic approximation
1.3 Integration
1.3.1 Odd and even functions
1.3.2 Fundamental theorem of calculus
1.4 Linear Algebra
1.4.1 Why do we need linear algebra in data science?
1.4.2 Everything you need to know about linear algebra
1.4.3 Inner products and norms
1.4.4 Matrix calculus
1.5 Basic Combinatorics
1.5.1 Birthday paradox
1.5.2 Permutation
1.5.3 Combination
Chapter 2 Probability
2.1 Set Theory
2.1.1 Why study set theory?
2.1.2 Basic concepts of a set
2.1.3 Subsets
2.1.4 Empty set and universal set
2.1.5 Union
2.1.6 Intersection
2.1.7 Complement and difference
2.1.8 Disjoint and partition
2.1.9 Set operations
2.1.10 Closing Remark
2.2 Probability Space
2.2.1 Sample space
2.2.2 Event space
2.2.3 Probability law
2.2.4 Measure zero sets
2.2.5 Summary of the probability space
2.3 Axioms of Probability
2.3.1 Why these three probability axioms?
2.3.2 Axioms through the lens of measure
2.3.3 Corollaries derived from axioms
2.4 Conditional Probability
2.4.1 Definition of conditional probability
2.4.2 Independence
2.4.3 Bayes’ theorem and law of total probability
2.4.4 Prisoner's dilemma
Chapter 3 Discrete Random Variables
3.1 Random Variables
3.1.1 A motivating example
3.1.2 Definition of a random variable
3.1.3 Probability measure on random variables
3.2 Probability Mass Function
3.2.1 Definition
3.2.2 PMF and probability measure
3.2.3 Normalization Property
3.2.4 PMF vs Histogram
3.2.5 Estimating histograms from real data
3.3 Cumulative Distribution Function
3.3.1 Definition
3.3.2 Properties of CDF
3.3.3 Converting between PMF and CDF
3.4 Expectation
3.4.1 Definition
3.4.2 Existence of Expectation
3.4.3 Properties of Expectation
3.4.4 Moments and Variance
3.5 Common Discrete Random Variables
3.5.1 Bernoulli Random Variable
3.5.2 Binomial random variable
3.5.3 Geometric random variable
3.5.4 Poisson random variable
Chapter 4 Continuous Random Variables
4.1 Probability Density Function
4.1.1 Some intuition about probability density functions
4.1.2 More in-depth discussion about PDFs
4.1.3 Connecting with PMF
4.2 Expectation, Moment, and Variance
4.2.1 Definition and properties
4.2.2 Existence of Expectation
4.2.3 Moment and Variance
4.3 Cumulative Distribution Function
4.3.1 CDF for continuous random variables
4.3.2 Properties of CDF
4.3.3 Retrieving PDF from CDF
4.3.4 CDF: Unifying discrete and continuous random variables
4.4 Median, Mode, and Mean
4.4.1 Median
4.4.2 Mode
4.4.3 Mean
4.5 Uniform and Exponential Random Variables
4.5.1 Uniform Random Variable
4.5.2 Exponential Random Variable
4.5.3 Origin of exponential random variable
4.5.4 Applications of exponential random variables
4.6 Gaussian Random Variables
4.6.1 Definition of a Gaussian random variable
4.6.2 Standard Gaussian
4.6.3 Skewness and Kurtosis
4.6.4 Origin of Gaussian random variables
4.7 Functions of Random Variables
4.7.1 General principle
4.7.2 Worked examples
4.8 Generating Random Numbers
4.8.1 Principle
4.8.2 Examples
Chapter 5 Joint Distributions
5.1 Joint PMF and Joint PDF
5.1.1 Probability measure in 2D
5.1.2 Discrete random variables
5.1.3 Continuous random variables
5.1.4 Normalization
5.1.5 Marginal PMF and marginal PDF
5.1.6 Independent random variables
5.1.7 Joint CDF
5.2 Joint Expectation
5.2.1 Definition and interpretation
5.2.2 Covariance and correlation coeffcient
5.2.3 Independence and correlation
5.2.4 Computing correlation from data
5.3 Conditional PMF and PDF
5.3.1 Conditional PMF
5.3.2 Conditional PDF
5.4 Conditional Expectation
5.4.1 Definition
5.4.2 Law of total expectation
5.5 Sum of Two Random Variables
5.5.1 Intuition through convolution
5.5.2 Main result
5.5.3 Sum of common distributions
5.6 Random Vector and Covariance Matrices
5.6.1 PDF of random vectors
5.6.2 Expectation of random vectors
5.6.3 Covariance matrix
5.6.4 Multi-dimensional Gaussian
5.7 Transformaiton of Multi-dimensional Gaussian
5.7.1 Linear transformation of mean and covariance
5.7.2 Eigenvalues and eigenvectors
5.7.3 Covariance matrices are always positive semi-definite
5.7.4 Gaussian whitening
5.8 Principal Component Analysis
5.8.1 The main idea: Eigen-decomposition
5.8.2 The Eigenface problem
5.8.3 What cannot be analyzed by PCA?
Chapter 6 Sample Statistics
6.1 Moment Generating and Characteristic Functions
6.1.1 Moment Generating Function
6.1.2 Sum of independent variables via MGF
6.1.3 Characteristic Functions
6.2 Probability Inequalities
6.2.1 Union bound
6.2.2 Cauchy-Schwarz's inequality
6.2.3 Jensen's inequality
6.2.4 Markov's inequality
6.2.5 Chebyshev's inequality
6.2.6 Chernoff's bound
6.2.7 Comparing Chernoff and Chebyshev
6.2.8 Hoeffding's inequality
6.3 Law of Large Numbers
6.3.1 Sample average
6.3.2 Weak law of large numbers (WLLN)
6.3.3 Convergence in probability
6.3.4 Can we prove WLLN using Chernoff's bound?
6.3.5 Does weak of large numbers always hold?
6.3.6 Strong law of large numbers
6.3.7 Almost sure convergence
6.3.8 Proof of strong law of large numbers
6.4 Central Limit Theorem
6.4.1 Convergence in distribution
6.4.2 Central Limit Theorem
6.4.3 Examples
6.4.4 Limitation of the Central Limit Theorem
Chapter 7 Regression
7.1 Principles of Regression
7.1.1 Intuition: how to fit a straight line?
7.1.2 Solving the linear regression problem
7.1.3 Extension: Beyond a straight line
7.1.4 Over-determined and under-determined systems
7.1.5 Robust linear regression
7.2 Over-ftting
7.2.1 Overview of overfitting
7.2.2 Analysis of the linear case
7.2.3 Interpreting the linear analysis results
7.3 Bias and variance trade off
7.3.1 Decomposing the testing error
7.3.2 Analysis of the bias
7.3.3 Variance
7.3.4 Bias and variance on the learning curve
7.4 Regularization
7.4.1 Ridge regularization
7.4.2 LASSO regularization
Chapter 8 Estimation
8.1 Maximum-Likelihood Estimation
8.1.1 Likelihood function
8.1.2 Maximum-likelihood estimate
8.1.3 Application 1: Social network analysis
8.1.4 Application 2: Reconstructing images
8.1.5 More examples on ML estimation
8.1.6 Regression vs ML estimation
8.2 Properties of ML Estimates
8.2.1 Estimators
8.2.2 Unbiased estimators
8.2.3 Consistent estimators
8.2.4 Invariance principle
8.3 Maximum-A-Posteriori Estimation
8.3.1 The trio of likelihood, prior, and posterior
8.3.2 Understanding the priors
8.3.3 MAP formulation and solution
8.3.4 Analyzing the MAP solution
8.3.5 Analysis of the posterior distribution
8.3.6 Conjugate Prior
8.3.7 Linking MAP with regression
8.4 Mean-Square Error Estimation
8.4.1 Positioning the mean square error estimation
8.4.2 Mean square error
8.4.3 MMSE solution = conditional expectation
8.4.4 MMSE estimator for multi-dimensional Gaussian
8.4.5 Linking MMSE and neural networks
Chapter 9 Confidence and Hypothesis
9.1 Confidence Interval
9.1.1 The randomness of an estimator
9.1.2 Understanding confidence intervals
9.1.3 Constructing a confidence interval
9.1.4 Properties about the confidence interval
9.1.5 Student's
-distribution9.1.6 Comparing Student's
-distribution and Gaussian
9.2 Bootstrap
9.2.1 A brute force approach
9.2.2 Bootstrap
9.3 Hypothesis Testing
9.3.1 What is a hypothesis?
9.3.2 Critical-value test
9.3.3
-value test9.3.4
-test and -test
9.4 Neyman-Pearson Test
9.4.1 Null and alternative distributions
9.4.2 Type 1 and type 2 error
9.4.3 Neyman-Pearson decision
9.5 ROC and Precision-Recall Curve
9.5.1 Receiver Operating Characteristic (ROC)
9.5.2 Comparing ROC curves
9.5.3 ROC curve in practice
9.5.4 Precision-Recall (PR) curve
Chapter 10 Random Processes
10.1 Basic Concepts
10.1.1 Everything you need to know about a random process
10.1.2 Statistical and temporal perspectives
10.2 Mean and Correlation Functions
10.2.1 Mean function
10.2.2 Autocorrelation function
10.2.3 Independent Processes
10.3 Wide Sense Stationary Processes
10.3.1 Definition of a WSS process
10.3.2 Properties of
10.3.3 Physical Interpretation of
10.4 Power Spectral Density
10.4.1 Basic concepts
10.4.2 Origin of the power spectral density
10.5 WSS Process through LTI Systems
10.5.1 Review of a linear time-invariant (LTI) system
10.5.2 Mean and autocorrelation through LTI Systems
10.5.3 Power spectral density through LTI systems
10.5.4 Cross-correlation through LTI Systems
10.6 Optimal Linear Filter
10.6.1 Discrete-time random processes
10.6.2 Problem formulation
10.6.3 Yule-Walker equation
10.6.4 Linear prediction
10.6.5 Wiener Filter