lecture for today

lectures/lecture-12/1 Probability as logic.ipynb

@@ -0,0 +1,199 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Events happen or do not happen. Things exist or do not. Probability is how we think about these."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Thie notebook is based on the original [here](https://gist.github.com/willkurt/6c209c14bf777470543393302d1b523c) (linked from [this excellent page](https://www.countbayesie.com/blog/2019/12/1/probability-and-statistics-in-90-minutes)).\n",
+    "\n",
+    "See [a related book by the author at Uni Freiburg](https://katalog.ub.uni-freiburg.de/opac/RDSIndex/Search?lookfor=Bayesian%20Statistics%20the%20Fun%20Way&source=homepage)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Probability as logic using Python's data model\n",
+    "\n",
+    "In this short notebook we'll show how probability really is just an extension of boolean logic. We'll make a class `P`, that uses Python's data model methods to implement a working example of probability as logic using `-`,`&` and `|`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class P:\n",
+    "    \"\"\"\n",
+    "    Example of Probability as logic using Python's data model\n",
+    "    In this simple example these probabilites are assumed to \n",
+    "    be conditionally independent.\n",
+    "    \"\"\"\n",
+    "    def __init__(self,prob):\n",
+    "        assert prob >= 0, \"probabilities can't be negative!\" \n",
+    "        assert prob <= 1, \"probabilities can't be greater than 1!\"\n",
+    "        self.prob = prob\n",
+    "        \n",
+    "    def __repr__(self):\n",
+    "        return \"P({})\".format(self.prob)\n",
+    "\n",
+    "    def __neg__(self):\n",
+    "        return P(1-self.prob)\n",
+    "    \n",
+    "    def __and__(self,P2):\n",
+    "        return P(self.prob * P2.prob)\n",
+    "    \n",
+    "    def __or__(self,P2):\n",
+    "        return P(self.prob + P2.prob - (self & P2).prob)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "P(0.5)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "P(0.5) & P(0.5)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "True & True"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "P(0.5) | P(0.5)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can then use this to work out probabilities of various events happening using python!\n",
+    "\n",
+    "Suppose, for example, you know that there is a 0.3 probability of rain tomorrow and you'll get rained on if you forget your umbrella or your umbrella is broken. Then let's say you forget your umbrella with a probability 0.1 and you think your umbrella might be the broken, we'll give that a probability of 0.7. \n",
+    "\n",
+    "Now let's use logic to answer: What's the probability you will *not* get wet?\n",
+    "\n",
+    "Let's start with our facts:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "rain = P(0.3)\n",
+    "forget = P(0.1)\n",
+    "broken = P(0.7)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The probability of being `wet` is:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "wet = rain & (forget | broken)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "wet"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "and logically the probability of being dry is:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "-wet"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "With probability as logic, no matter how complicated our reasoning, we can now trivially code up a solution in Python!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Conclusions\n",
+    "\n",
+    "* Probabilities are intuitive\n",
+    "* Do not let mathematical notation of probabilities scare you\n",
+    "* Working mathematically and computationally with probabilities can be very useful\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

lectures/lecture-12/2 Introduction to distributions and random variables.ipynb

@@ -0,0 +1,248 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Learning goals\n",
+    "\n",
+    "Understand the following:\n",
+    "\n",
+    "- Probability distribution\n",
+    "- Probability density function (PDF)\n",
+    "- Gaussian distribution\n",
+    "- Histogram\n",
+    "- Mean\n",
+    "- Standard deviation\n",
+    "- Variance\n",
+    "- Mode\n",
+    "\n",
+    "We will discuss these as we work through this example"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "import scipy.stats as stats"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "sample_size = 30\n",
+    "\n",
+    "true_mean = 10.0\n",
+    "true_variance = 4.0\n",
+    "true_std = np.sqrt(true_variance)\n",
+    "\n",
+    "rng = np.random.RandomState(123)\n",
+    "fish_sample = rng.normal(loc=true_mean, scale=true_std, size=sample_size)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "fish_sample"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "x=plt.hist(fish_sample)\n",
+    "plt.xlabel(\"fish length (cm)\")\n",
+    "plt.ylabel(\"counts\");"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "x"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## What is the mean of the sample of fish lengths?"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "sample_mean = np.mean(fish_sample)\n",
+    "print(sample_mean)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## What is the [mode](https://en.wikipedia.org/wiki/Mode_(statistics)) of the sample of fish lengths? (\"[modus](https://de.wikipedia.org/wiki/Modus_(Statistik))\" in German.)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## What is the sample variance of fish lengths?\n",
+    "\n",
+    "We put the result in the variable `length_variance`. We use the `ddof=1` keyword argument to `np.var()` because we are estimating variance from a sample here."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "sample_variance = np.var(fish_sample, ddof=1)\n",
+    "print(sample_variance)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Probability Density Functions (PDFs)\n",
+    "\n",
+    "- for working with continuous variables (vs. probability mass functions for discrete variables)\n",
+    "- here, the area under the curve gives the probability (in contrast to probability mass functions where we have probabilities for every single value)\n",
+    "- the area under the whole curve is 1. In other words, some outcome must happen.\n",
+    "\n",
+    "## Normal Distribution (Gaussian Distribution)\n",
+    "\n",
+    "- unimodal and symmetric\n",
+    "- many algorithms in machine learning & statistics have normality assumptions\n",
+    "- two parameters: mean (center of the peak) and standard deviation (spread); $N(\\mu, \\sigma)$\n",
+    "- we can estimate parameters $\\mu$ and $\\sigma$ by the sample mean ($\\bar{x})$ and sample variance ($s^2$)\n",
+    "- univariate Normal distribution:\n",
+    "\n",
+    "$$N(\\mu, \\sigma) = f(x \\mid \\mu, \\sigma^2) = \\frac{1}{\\sqrt{2\\pi\\sigma^2}} \\; \\exp\\bigg(-\\frac{(x-\\mu)^2}{2\\sigma^2}\\bigg)$$\n",
+    "\n",
+    "- standard normal distribution with zero mean and unit variance, $N(0, 1)$:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def univariate_gaussian_pdf(x, mean, variance):\n",
+    "    return (1. / np.sqrt(2*np.pi*variance) * \n",
+    "            np.exp(- ((x - mean)**2 / (2.*variance))))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "mean = 0\n",
+    "stddev = 1\n",
+    "x = np.arange(-5, 5, 0.01)\n",
+    "y = univariate_gaussian_pdf(x, mean, stddev**2)\n",
+    "plt.plot(x, y)\n",
+    "plt.xlabel('x')\n",
+    "plt.ylabel('Probability Density Function (PDF)')\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "mean = 10\n",
+    "stddev = 2.0\n",
+    "x = np.arange(5, 15, 0.01)\n",
+    "y = univariate_gaussian_pdf(x, mean, stddev**2)\n",
+    "plt.plot(x, y)\n",
+    "plt.xlabel('fish length (cm)')\n",
+    "plt.ylabel('Probability Density Function (PDF)')\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "plt.hist(fish_sample, density=True, label='samples')\n",
+    "\n",
+    "x = np.linspace(5.0, 15.0, 100)\n",
+    "y = univariate_gaussian_pdf(x, true_mean, true_variance)\n",
+    "plt.plot(x, y, lw=5, label='theoretical (mean=%.1f, variance=%.1f)'%(true_mean,true_variance))\n",
+    "\n",
+    "sample_mean = np.mean(fish_sample)\n",
+    "sample_variance = np.var(fish_sample, ddof=1) # for a sample variance, always use ddof=1. Here is one explanation why https://mortada.net/computing-sample-variance-why-divide-by-n-1.html\n",
+    "y2 = univariate_gaussian_pdf(x, sample_mean, sample_variance)\n",
+    "plt.plot(x, y2, label='empirical (mean=%.1f, variance=%.1f)'%(sample_mean,sample_variance))\n",
+    "\n",
+    "plt.xlabel(\"length (cm)\")\n",
+    "plt.ylabel(\"probability\");\n",
+    "plt.legend();\n",
+    "# plt.savefig('fish-gaussian.png');"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Discussion points\n",
+    "\n",
+    "Here are a few questions you should be able to answer:\n",
+    "\n",
+    " - what is plotted on the x axis?\n",
+    " - what is plotted on the y axis?\n",
+    " - what is the area under the curve from -∞ to ∞?\n",
+    " - according to our Guassian distribution, what is the most likely fish length?\n",
+    " - according to our histogram, what is the most likely fish length?"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

lectures/lecture-12/3 Bayes' theorem.ipynb

@@ -0,0 +1,69 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Bayes' theorem\n",
+    "\n",
+    "\n",
+    "$  \\newcommand{\\thetavec}{\\boldsymbol{\\theta}} \\newcommand{\\pr}{\\textrm{p}}$\n",
+    "\n",
+    "Bayes' theorem with $\\thetavec$ the vector of parameters we seek.\n",
+    "\n",
+    "$$\n",
+    "  \\overbrace{\\pr(\\thetavec \\mid \\textrm{data})}^{\\textrm{posterior}} =\n",
+    "  \\frac{\\color{red}{\\overbrace{\\pr(\\textrm{data} \\mid \\thetavec)}^{\\textrm{likelihood}}} \\times\n",
+    "   \\color{blue}{\\overbrace{\\pr(\\thetavec)}^{\\textrm{prior}}}}\n",
+    "   {\\color{darkgreen}{\\underbrace{\\pr(\\textrm{data})}_{\\textrm{evidence}}}}\n",
+    "$$\n",
+    "\n",
+    "If we view the prior as the initial information we have about $\\thetavec$, summarized as a probability density function, then Bayes' theorem tells us how to <em>update</em> that information after observing some data: this is the posterior pdf.\n",
+    "\n",
+    "\n",
+    "We will go through [this post, \"Bayes' Theorem with Lego\"](https://www.countbayesie.com/blog/2015/2/18/bayes-theorem-with-lego).\n",
+    "\n",
+    "## Discussion: the Bayesian Brain hypothesis.\n",
+    "\n",
+    "Here is an [interesting article](https://towardsdatascience.com/the-bayesian-brain-hypothesis-35b98847d331), which might serve as a starting point, about this idea.\n",
+    "\n",
+    "## Further material\n",
+    "\n",
+    "* [Bayesian Statistics the Fun Way](https://katalog.ub.uni-freiburg.de/opac/RDSIndex/Search?lookfor=Bayesian%20Statistics%20the%20Fun%20Way%20&source=homepage)\n",
+    "\n",
+    "* [Doing Bayesian Data Analysis : A Tutorial with R, JAGS, and Stan\n",
+    "John Kruschke](https://katalog.ub.uni-freiburg.de/opac/RDSIndex/Search?lookfor=Bayesian+Data+analysis+Kruschke&type=AllFields&limit=10&sort=py+desc)\n",
+    "\n",
+    "* https://www.3blue1brown.com/, especially https://www.youtube.com/watch?v=HZGCoVF3YvM\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}