{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Bayesian Inference in Machine Learning: Part 1\n", "\n", "Rafiq Islam \n", "2024-07-28" ], "id": "e1aa2543-d26a-48df-86b8-b3283c5068a4" }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/html" }, "source": [ "" ], "id": "a1ff1693-28a7-4804-a705-24eb5ba5962a" }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction" ], "id": "6802fbdc-9330-45fe-85d1-c770e5b6b694" }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/html" }, "source": [ "

" ], "id": "04f4c3f6-4f35-4d44-b4b5-b61ee96a6206" }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "Bayesian inference is a powerful statistical method that applies the\n", "principles of Bayes’s theorem to update the probability of a hypothesis\n", "as more evidence or information becomes available. It is widely used in\n", "various fields including machine learning, to make predictions and\n", "decisions under uncertainty." ], "id": "ab069c71-8a50-4f27-83b3-d341eaf41091" }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/html" }, "source": [ "

" ], "id": "8a2adb48-c8b8-4b54-901e-7b2bd1a39527" }, { "cell_type": "markdown", "metadata": {}, "source": [ "Bayes’s theorem is a fundamental result in probability theory that\n", "relates the conditional and marginal probabilities of random events.\n", "Mathematically, \n", "$$\\mathbb{P}(A|B)=\\frac{\\mathbb{P}(B|A)\\mathbb{P}(A)}{\\mathbb{P}(B)}\\hspace{4mm} \\implies \\mathbb{P}(A|B) \\propto \\mathbb{P}(B|A)\\mathbb{P}(A)$$\n", "\n", "where, $A$ and $B$ are events and $\\mathbb{P}(B)\\ne 0$.\n", "\n", "- $\\mathbb{P}(A|B)$ is a conditional probability which states the\n", " probability of occuring the event $A$ when the event $B$ is given or\n", " true. The other name of this quantity is called\n", " *posterior probability* of $A$ given\n", " the event $B$ or simply, *posterior\n", " distribution.* \n", "- $\\mathbb{P}(B|A)$ is a conditional probability which states the\n", " probability of occuring the event $B$ when the event $A$ is given or\n", " true. In other terms, $\\mathbb{P}(B|A)$ is the likelihood: the\n", " probability of evidence $B$ given that $A$ is true. \n", "- $\\mathbb{P}(A)$ or $\\mathbb{P}(B)$ are the probabilities of occuring\n", " $A$ and $B$ respectively, without any dependence on each other.\n", " $\\mathbb{P}(A)$ is called the *prior* probability or prior\n", " distribution and $\\mathbb{P}(B)$ is called the marginal likelihood\n", " or marginal probabilities.\n", "\n", "**Example 1** \n", "Consider a medical example where we want to diagnose a disease based on\n", "a test result. Let:\n", "\n", "- $D$ be the event that a patient has the disease. \n", "- $T$ be the event that the test result is positive.\n", "\n", "We are interested in finding the probability that a patient has the\n", "disease given a positive test result, $\\mathbb{P}(D|T)$.\n", "\n", "Given:\n", "\n", "- $\\mathbb{P}(T|D) = 0.99$ (the probability of a positive test result\n", " given the patient has the disease). \n", "- $\\mathbb{P}(D) = 0.01$ (the prior probability of the disease). \n", "- $\\mathbb{P}(T|D') = 0.05$ (the probability of a positive test result\n", " given the patient does not have the disease).\n", "\n", "First, we need to calculate the marginal likelihood $P(T)$:\n", "\n", "Now, we can apply Bayes’s theorem: \n", "\n", "So, the probability that the patient has the disease given a positive\n", "test result is approximately $16.67\\%$.\n", "\n", "**Example 2**" ], "id": "41d25e03-cf44-40f6-bd51-1f97454ae351" }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/html" }, "source": [ "

" ], "id": "d8ee8920-93e2-4b5c-a2e7-474f85c0727e" }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "Assume that you are in a restuarant and you ordered a plate of 3\n", "pancakes. The chef made three pancakes with one in perfect condition,\n", "that is not burnt in any side, one with one side burnt, and the last one\n", "burnt in both sides. The waiter wanted to stack the pancakes so that the\n", "burnt side does not show up when served. However, the chef recommended\n", "not to hide the burnt side and asked her to stack the pancakes randomly.\n", "What is the likelyhood that the fully burnt pancake will be on the top?\n", "

To solve this problem, we can use Bayesian approach. We denote\n", "the event $X$ as the pancake without any burnt, $Y$ with one side burnt,\n", "and $Z$ both side burnt. Then we have the following conditional\n", "probabilities" ], "id": "e3915402-6a19-4503-96f7-522386872fc3" }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/html" }, "source": [ "

" ], "id": "119b51bb-7bb1-4288-9576-2584d7ee45f4" }, { "cell_type": "markdown", "metadata": {}, "source": [ "The probability of picking a pancake irrespective of their burnt\n", "condition is $\\frac{1}{3}$. So, \n", " \n", "The marginal probability of having burnt side in the top position \n", "\n", "Now, we can only have a burnt side on top if either $Z$ is placed in the\n", "top or the burnt side of $Y$ is placed in the top.\n", "\n", "So the probability of having the fully burnt pancake on the top is\n", "$\\frac{2}{3}$.\n", "\n", "## Why Bayesian Inference in Machine Learning?\n", "\n", "Bayesian inference plays a crucial role in machine learning,\n", "particularly in areas involving uncertainty and probabilistic reasoning.\n", "It allows us to incorporate prior knowledge and update beliefs based on\n", "new data, which is especially useful in the following applications:\n", "\n", "### Bayesian Networks\n", "\n", "Bayesian networks are graphical models that represent the probabilistic\n", "relationships among a set of variables. Each node in the network\n", "represents a random variable, and the edges represent conditional\n", "dependencies. Bayesian networks are used for various tasks such as\n", "classification, prediction, and anomaly detection.\n", "\n", "### Bayesian Regression\n", "\n", "Bayesian regression extends linear regression by incorporating prior\n", "distributions on the model parameters. This approach provides a\n", "probabilistic framework for regression analysis, allowing for\n", "uncertainty in the parameter estimates. The posterior distribution of\n", "the parameters is computed using Bayes’s theorem, and predictions are\n", "made by averaging over this distribution.\n", "\n", "### Sampling Methods\n", "\n", "In Bayesian inference, exact computation of the posterior distribution\n", "is often intractable. Therefore, sampling methods such as Markov Chain\n", "Monte Carlo (MCMC) and Variational Inference are used to approximate the\n", "posterior distribution. These methods generate samples from the\n", "posterior distribution, allowing us to estimate various statistical\n", "properties and make inferences.\n", "\n", "**Markov Chain Monte Carlo (MCMC)**\n", "\n", "MCMC methods generate a sequence of samples from the posterior\n", "distribution by constructing a Markov chain that has the desired\n", "distribution as its equilibrium distribution. Common MCMC algorithms\n", "include the Underdamped and Overdamped Langevin dynamics,\n", "Metropolis-Hastings algorithm and the Gibbs sampler.\n", "\n", "**Example: Metropolis-Hastings Algorithm**\n", "\n", "Consider a posterior distribution $P(\\theta|D)$ where $\\theta$\n", "represents the model parameters and $D$ represents the data. The\n", "Metropolis-Hastings algorithm proceeds as follows:\n", "\n", "1. Initialize the parameters $\\theta_0$.\n", "2. For $t = 1$ to $T$:\n", " - Propose a new state $\\theta'$ from a proposal distribution\n", " $Q(\\theta'|\\theta_t)$.\n", " - Compute the acceptance ratio\n", " $\\alpha = \\frac{P(\\theta'|D) \\cdot Q(\\theta_t|\\theta')}{P(\\theta_t|D) \\cdot Q(\\theta'|\\theta_t)}$.\n", " - Accept the new state with probability $\\min(1, \\alpha)$. If\n", " accepted, set $\\theta_{t+1} = \\theta'$; otherwise, set\n", " $\\theta_{t+1} = \\theta_t$.\n", "\n", "The samples $\\theta_1, \\theta_2, \\ldots, \\theta_T$ form a Markov chain\n", "whose stationary distribution is the posterior distribution\n", "$P(\\theta|D)$.\n", "\n", "### Bayesian Inference in Neural Networks\n", "\n", "Bayesian methods are also applied to neural networks, resulting in\n", "Bayesian Neural Networks (BNNs). BNNs incorporate uncertainty in the\n", "network weights by placing a prior distribution over them and using\n", "Bayes’s theorem to update this distribution based on the observed data.\n", "This allows BNNs to provide not only point estimates but also\n", "uncertainty estimates for their predictions.\n", "\n", "In the next parts, we will talk about different applications of the\n", "Bayesian inferences, specifically, sampling problem using Langevin\n", "dynamics.\n", "\n", "### Reference\n", "\n", "- [Pancake problems on\n", " mathstackexchance](https://math.stackexchange.com/q/660788)\n", "\n", "**Share on**\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "" ], "id": "fd786605-6462-4ede-96bd-8ac211ab6056" }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/html" }, "source": [ "" ], "id": "45026d44-adcf-4686-a3e1-f057a2ea860a" }, { "cell_type": "markdown", "metadata": {}, "source": [], "id": "e2ed2aa3-2fce-4645-a4be-a3507faad414" }, { "cell_type": "raw", "metadata": { "raw_mimetype": "text/html" }, "source": [ "" ], "id": "05227c0e-3c42-416b-a9d2-dfcc268bba25" }, { "cell_type": "markdown", "metadata": {}, "source": [ "**You may also like**" ], "id": "ed4c4cc9-4067-4b9d-beed-b7fc49208237" } ], "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" } } }