{ "cells": [ { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "b82b3443534132b3771ea1bd5dce7292", "grade": false, "grade_id": "cell-1d6811de1e292425", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "In this notebook, we explore how to detect a moving object on a stationary background." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "c182d2be009c8c695eef8625a316ad12", "grade": false, "grade_id": "cell-c101bf39d596ef9a", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from scipy import ndimage\n", "\n", "# If import imageio fails, you can install it like this in Jupyter:\n", "# !pip install 'imageio[ffmpeg]'\n", "# or like this at the command line:\n", "# python -m pip install 'imageio[ffmpeg]'\n", "import imageio" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "e2c79f491e8c8c3c610c17a2c388b4ac", "grade": false, "grade_id": "cell-7161f550b1a10a31", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# Here we define a helper function which we call below to make sure\n", "# an image is an \"unsigned 8 bit integer\". This way, we know they\n", "# take only a single byte per pixel and have a value from 0..255.\n", "\n", "def ensure_dtype_uint8(arr):\n", " return arr.astype(np.uint8)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "ff7991ce2a103a43b9fcbdd9c12f49d4", "grade": false, "grade_id": "cell-108186f7f14d851d", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Here we load the first frame of a movie file saved in the Straw Lab of a fly walking." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "10aa39ce56f805abd276ebf6158c31cf", "grade": false, "grade_id": "cell-721f647abba6218f", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "fname = \"data/short-movie20170810_182130.mp4\"\n", "reader = imageio.get_reader(fname)\n", "for frame in reader:\n", " frame0 = frame[:,:,1] # take only green channel\n", " break" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "2190e9954dbf33018d3260e5707d16ef", "grade": false, "grade_id": "cell-6589753ba9570c6b", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Now, we plot the first frame (the first image) in the video. We do this once in grayscale and once in false color." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "2d78181caa0fb6a2c3400add13dfdad9", "grade": false, "grade_id": "cell-f8d6f9c412400ac4", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "plt.imshow(frame0, cmap='gray')\n", "plt.colorbar();" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "4c4dc1664274ccb8975819355360a08b", "grade": false, "grade_id": "cell-79ff5ac403dbec51", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "plt.imshow(frame0, cmap='jet')\n", "plt.colorbar();" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "c136511dff9e32106ddf90e1e3b30a3d", "grade": false, "grade_id": "cell-b4d51617498cdf00", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Note how the false color image lets you see more easily approximately what pixel intensity values are present." ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "16d3565781d1ef6c69094952f33fd0f1", "grade": false, "grade_id": "cell-bb6b6c1075212bfa", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Now, list make a histogram of the pixel intensity values." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "9973e9aa97f86e66e016d6c94a3e4270", "grade": false, "grade_id": "cell-1f03feb12c388aa0", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "plt.hist(frame0.flat, bins=30);" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "3f08ed66615d8dd3ab93e8976c026f5b", "grade": false, "grade_id": "cell-b8ef9c796dcc7da1", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Now we will load all the frames into a single large data structure from disk into memory." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "af923e6f1df358188b7bf9d73afc8bef", "grade": false, "grade_id": "cell-d8d4b833e722c306", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "# get all frames into big 3D array\n", "(height,width) = frame0.shape\n", "all_frames = []\n", "reader.set_image_index(0) # return to start of file\n", "for frame in reader:\n", " all_frames.append( frame[:,:,1] )\n", "all_frames = np.array(all_frames)\n", "print(all_frames.shape)\n", "n_frames = len(all_frames)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "02d2ad93fdb67de427d32f3fc0c7e833", "grade": false, "grade_id": "cell-3ef9393de49f1ec7", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Now the entire movie is stored as a large 3D array of size: number of frames x width x height." ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "0b950953dde6ce892c13c4217c9ad3a4", "grade": false, "grade_id": "cell-c2722d719e790f09", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Now, we want to find a fly in the image. The fly is moving, but the background is stationary, so we should be able to determine a single \"background image\". We can try to find the background image in at least two ways: with a \"mean image\" (also \"average image\"), which should average over the fly's position. And with a \"median image\" which should completely eliminate the fly from the backround image if the fly was absent for most of the time from each pixel." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "de347ea8c88b9edf97f3b28e0f1ebf5a", "grade": false, "grade_id": "cell-2f4ea267c6c2ccd9", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "mean_frame = np.mean(all_frames, axis=0)\n", "median_frame = np.median(all_frames, axis=0)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "1b60dd865c78115948b238ed3c5a44f1", "grade": false, "grade_id": "cell-e2365cb18d5b20d1", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "plt.imshow(mean_frame, cmap='jet')\n", "plt.colorbar();" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "e1117202150673346ec746b98ce66971", "grade": false, "grade_id": "cell-58ec1fa04a85c5db", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "plt.imshow(median_frame, cmap='jet')\n", "plt.colorbar();" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "41b6632af41f7da6e03705041ef382d1", "grade": false, "grade_id": "cell-5faa20bd7e208895", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Now that we have the \"background\", we can find the difference from background, which should emphasize anything that is moving." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "074b256543714ac267d6a652a9c816bf", "grade": false, "grade_id": "cell-074247293d712cac", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "frame0_absdiff = abs(frame0 - median_frame)\n", "plt.imshow(frame0_absdiff, cmap='jet')\n", "plt.colorbar();" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "33636375e38961ac449739f12d0c677b", "grade": false, "grade_id": "cell-798add9cb796580b", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Now lets make a histogram of the `frame0_absdiff` image." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "b743c16e5470a4e0ecba1b1830b7b644", "grade": false, "grade_id": "cell-c71b6c10e0ee3018", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "plt.hist(frame0_absdiff.flat, bins=30);" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "c470d43abe6189c7992104c8517126cb", "grade": false, "grade_id": "cell-2e5d36068fff4d1d", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "We want to find a threshold for this \"absolute difference image\" which will separate the fly from the background. From frame to frame the background changes very little - only noise in the light levels, camera sensor and caused by lossy compression to a movie format cause changes. But when the fly moves over the background, there is a large change in luminance values. This happens only in very few pixels. So few that they do not show on the histogram. However, matplotlib will automatically scale the axes, so we know there must be few pixels with values at 150 or above." ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "b0af9598ca270ca26efc5400cf2e19b5", "grade": false, "grade_id": "cell-d05a54e4c069f9aa", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "## Questions Part A\n", "\n", "Enter a threshold below and describe why you chose this value. Put your answer in the variable `threshold`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "nbgrader": { "cell_type": "code", "checksum": "71fcb58983ffdeb9114e35ad94690907", "grade": false, "grade_id": "cell-1ef3868b1b1b045c", "locked": false, "schema_version": 3, "solution": true } }, "outputs": [], "source": [ "# YOUR CODE HERE\n", "raise NotImplementedError()" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "nbgrader": { "cell_type": "markdown", "checksum": "77322ebaaa5f2b4b29f0e3d0488add63", "grade": true, "grade_id": "cell-c848bbfece87e521", "locked": false, "points": 1, "schema_version": 3, "solution": true } }, "source": [ "YOUR ANSWER HERE" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "fdcf39f570e2d445f8fddba75e926a82", "grade": true, "grade_id": "cell-edadac5626867908", "locked": true, "points": 1, "schema_version": 3, "solution": false } }, "outputs": [], "source": [ "# If this runs without error, it means the answer in your previous cell was a number.\n", "assert(threshold+0==threshold)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "f2f208aff8f5e0442143c952eb9bbcb0", "grade": false, "grade_id": "cell-f6fc50107a0d129a", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Now we want to \"binarize\" or \"threshold\" the image so that it is all zeros and ones (or true and false). This will let use use analyses like connected components labeling later." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "07af0cac3c4f77c9758189d926502bac", "grade": false, "grade_id": "cell-fcef9de824e56bd5", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "binarized = ensure_dtype_uint8(frame0_absdiff>threshold)\n", "plt.imshow(binarized);\n", "plt.colorbar();" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "ff1db2b54f0da4954fa6a7d47ee0a5ad", "grade": false, "grade_id": "cell-0338555f51c06105", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Now we will use **connected components labeling** via the `ndimage.label()` function.\n", "\n", "We will do more with **connected components labeling** later. For now, you can [read about connected components labeling](https://en.wikipedia.org/wiki/Connected-component_labeling).\n", "\n", "We will use these labels to perform object detection. We will not bother to detect the fly on every single frame. Only on the frames where we have exactly one label do we say that we detected the fly. We skip the other frames, ignoring them." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "c33765aa2c06d7fa5451d15226f9e5c0", "grade": false, "grade_id": "cell-9dae893dac3022a4", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "labels, num_labels = ndimage.label(binarized)\n", "plt.imshow(labels, cmap='jet')\n", "plt.colorbar();" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "a62a1a753e77ef467e8ba67f399843eb", "grade": false, "grade_id": "cell-bf4c1b87ad01a8bb", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "## Questions Part B\n", "\n", "Run the object detection algorithm for each frame and plot the result. Here, we are going to make lists `frames`, `xs`, `ys` to save the results of our object detection step.\n", "\n", "Try re-running the above cells (from \"Questions Part A\") to the cell below with varying values of the `threshold` value until you get a good tracking result. You should ideally have zero warnings printed, but one or two are OK. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "code", "checksum": "bd98d21e0aa72c9a4ea72292a21e56a8", "grade": false, "grade_id": "cell-cb7ec6b50e3d024a", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "outputs": [], "source": [ "frames = []\n", "xs = []\n", "ys = []\n", "for i in range(n_frames):\n", " frame = all_frames[i,:,:]\n", " frame_absdiff = abs(frame - median_frame)\n", " binarized = ensure_dtype_uint8(frame_absdiff>threshold)\n", " labels, num_labels = ndimage.label(binarized)\n", " if num_labels!=1:\n", " print('WARNING: num_labels %d on frame %d, skipping' % (num_labels,i))\n", " continue\n", " y,x = np.mean(np.nonzero(labels==1),axis=1)\n", " frames.append(i)\n", " xs.append(x)\n", " ys.append(y)" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "c55362805bd45921198f09157c1d21f2", "grade": false, "grade_id": "cell-c1e4e9b97c4899cb", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Make with an image of the first frame (saved in the variable `frame0` above) in the background and then the fly trajectory overlaid on top of the image.\n", "\n", "You can do this by first calling `plt.imshow()` and then `plt.plot()`. Your result should look like this:\n", "\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "nbgrader": { "cell_type": "code", "checksum": "d156a2cb11d8b53eca4ce0c2c6d98c5a", "grade": true, "grade_id": "cell-7250d526d95bf828", "locked": false, "points": 1, "schema_version": 3, "solution": true } }, "outputs": [], "source": [ "# YOUR CODE HERE\n", "raise NotImplementedError()" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "editable": false, "nbgrader": { "cell_type": "markdown", "checksum": "e7bff9942e7efca03e28e36dafa297a3", "grade": false, "grade_id": "cell-06fde843bcd5b77e", "locked": true, "schema_version": 3, "solution": false, "task": false } }, "source": [ "Make a plot of the timeseries of the x and y values.\n", "\n", "We want a plot that looks like the following. You need to fix the following code to do this:\n", "\n", "```\n", "fig, axes = plt.subplots(nrows=2, ncols=1, sharex=True)\n", "axes[0].plot(time_values,x_values,'x')\n", "axes[0].set_ylabel('x (px)')\n", "axes[1].plot(time_values,y_values,'x')\n", "axes[1].set_ylabel('y (px)')\n", "axes[1].set_xlabel('time (frames)')\n", "```\n", "\n", "As a side note, the `sharedx` links the two panels together so they have the same X axis. When using matplotlib interactively (highly recommended!!!), panning and zooming in one panel also pans and zooms the other panel. This is incredibly useful.\n", "\n", "" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "deletable": false, "nbgrader": { "cell_type": "code", "checksum": "a1268da25c1f3b257c2e16acfde6b9d4", "grade": true, "grade_id": "cell-8680c2c7c9c5cb22", "locked": false, "points": 1, "schema_version": 3, "solution": true, "task": false } }, "outputs": [], "source": [ "# YOUR CODE HERE\n", "raise NotImplementedError()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Questions Part C\n", "\n", "Describe: How did we get the background image? How did we compute a binary image? What did we do with the binary image?" ] }, { "cell_type": "markdown", "metadata": { "deletable": false, "nbgrader": { "cell_type": "markdown", "checksum": "8f8394bf41b5ede928f69dff168e85f4", "grade": true, "grade_id": "cell-0a341a483941e5cd", "locked": false, "points": 2, "schema_version": 3, "solution": true } }, "source": [ "YOUR ANSWER HERE" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.10" } }, "nbformat": 4, "nbformat_minor": 4 }