📊 Data Science Week 1 of 14 MSc · S1 ⏱ ~50 min

Week 1: Bayesian Methods, MCMC & Regularization Theory

Deep dive into Bayesian inference, Markov Chain Monte Carlo, regularization theory, and advanced model selection techniques for research-level ML.

University of America

DS501 — Lecture 1 · MSc S1

🎬 CC Licensed Lecture

0:00 / —:—— 📺 MIT OpenCourseWare (CC BY-NC-SA)

🎯 Learning Objectives

Derive the Bayesian posterior for common conjugate models
Implement Markov Chain Monte Carlo (MCMC) from scratch
Prove the Lasso and Ridge regularization properties
Apply cross-validation theory for model selection

Topics Covered This Lecture

Bayesian Inference & Conjugate Priors

MCMC: Metropolis-Hastings & Gibbs Sampling

Regularization Theory: Lasso & Ridge

Information Criteria: AIC, BIC, WAIC

📖 Lecture Overview

This first lecture establishes the foundational framework for Advanced Statistical Learning. By the end of this session, you will have the conceptual grounding and practical starting point needed for the rest of the course.

        Why this matters
        Deep dive into Bayesian inference, Markov Chain Monte Carlo, regularization theory, and advanced model selection techniques for research-level ML. This lecture sets up everything that follows — make sure you understand the core concepts before proceeding to Week 2.
      

Key Concepts

The lecture introduces the four main pillars of this course: Bayesian Inference & Conjugate Priors, MCMC: Metropolis-Hastings & Gibbs Sampling, Regularization Theory: Lasso & Ridge, Information Criteria: AIC, BIC, WAIC. Each will be explored in depth over the 14-week curriculum, with hands-on projects reinforcing theory at every stage.

# Quick Start: verify your environment is ready for DS501
import sys
print(f"Python {sys.version}")

# Check key libraries are installed
try:
    import numpy, pandas, matplotlib
    print("✅ Core libraries ready")
except ImportError as e:
    print(f"❌ Missing: {e} — run: pip install numpy pandas matplotlib")

This Week's Focus

Focus on mastering: Bayesian Inference & Conjugate Priors and MCMC: Metropolis-Hastings & Gibbs Sampling. These are the prerequisites for everything in Week 2. The concepts build on each other — do not skip the practice exercises.

📋 Project 1 of 3 50% of Final Grade

DS501 Project 1: Bayesian Regression with MCMC

Implement Bayesian linear regression with MCMC sampling using PyMC or Stan. Compare posterior predictive distributions against frequentist estimates on a regression dataset.

MCMC sampler implementation (from scratch or PyMC)
Convergence diagnostics (R-hat, effective sample size)
Posterior predictive checks and calibration plots
Comparison with MLE and LASSO estimates

50%

3 Projects

20%

Midterm Exam

30%

Final Exam

📝 Sample Exam Questions

These represent the style and difficulty of questions you'll see on the midterm and final. Start thinking about them now.

Conceptual Short Answer

Derive the posterior distribution for a Gaussian likelihood with a Gaussian prior on the mean.

Analysis Short Answer

What is the Metropolis-Hastings algorithm? Prove that it converges to the target distribution.

Applied Code / Proof

When does Lasso produce exact zeros in coefficients? Prove using the KKT conditions.