pm21-dragon/exercises/release/exercise-14/Deep Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fac96be8-dc47-4cac-9591-1d4dddd45580",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "56281969fd314d8abd60bb730bc6733c",
     "grade": false,
     "grade_id": "cell-0ec3e572e00b624c",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# Deep Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c33c9c87-b7a4-4f26-9865-34d06d9f3067",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "1a85ab040d5086ce0062ee76e1160425",
     "grade": false,
     "grade_id": "cell-1be808b348d19ef2",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "## YOLO (version 8) video detection\n",
    "\n",
    "We will use YOLO (You Only Look Once), a very popular *object detector* trained using deep learning. Read this [good article about what is YOLOv8](https://blog.roboflow.com/whats-new-in-yolov8/).\n",
    "\n",
    "As our first step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90a07dd6-6ab8-4014-a29a-82358b1176c9",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e19967d111f53da29bb3e1c9d523874a",
     "grade": false,
     "grade_id": "cell-411e1ce5e3e3171d",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "!pip install ultralytics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c95ea0b3-75fc-4612-80cb-113ba7bba00f",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "d0d6bd700b391a4bb2bfde9dbc668c53",
     "grade": false,
     "grade_id": "cell-bec97cac599022a5",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "from ultralytics import YOLO\n",
    "from ultralytics.utils.plotting import Annotator, colors\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dae71ee3-ae2a-444a-80e9-e344edfd9bea",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "d9750a219862774bb13834baaa125ce4",
     "grade": false,
     "grade_id": "cell-6bf6aaac22f685db",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "In this exercise, we use a pre-trained model. Ultralytics has trained this model on the [COCO 2017 dataset by Microsoft](https://cocodataset.org). They have trained 80 classes of objects, from `apple` to `wine glass` including `person`, `bicycle`, and `car`. They make several versions of the object detection model available (and they have other models for tasks beyond object detection). We will use the smallest (and fastest) of these, the \"nano\" version, `yolov8n`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cbfaf12-f167-46e4-bd3f-3024542929cd",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b20451e51a59ca2637f81e32ebd387b5",
     "grade": false,
     "grade_id": "cell-c78347e3794f28a3",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "model = YOLO(\"yolov8n.pt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb7956c2-c064-40dd-aafb-442685ee3cbf",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "f81e3dd44ac6d1483c8d3ef5076a5155",
     "grade": false,
     "grade_id": "cell-95c4d55bfbcc11d4",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "## Q1\n",
    "\n",
    "Upload a video which you would like to run the YOLO object detector on. If you want to use a video of ours, you can try this video: https://strawlab-cdn.com/assets/samples-for-object-detection/BiologieI-bike.mp4 . Upload the video to the folder where this notebook lives by dragging and dropping it into the file browser in JupyterLab.\n",
    "\n",
    "Put the name of your video in the variable `movie_filename`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "698ab95e-58c0-49d0-bcb4-0ffed2a4d877",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e1d2a866b43bbe45344f69c635afcb37",
     "grade": false,
     "grade_id": "cell-f4654a9b8a23da31",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cfbdadd-cde2-4db0-bb5d-69a21b64d0c9",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "75e0fe2c0779b190f089891e63fe4530",
     "grade": true,
     "grade_id": "cell-ef12e91efc25735d",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "assert os.path.exists(movie_filename), 'No file with the name in `movie_filename`'\n",
    "assert os.path.getsize(movie_filename) >= 10, 'Not video file: no more than 10 bytes'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de630ada-5901-4142-a917-13320e79dd82",
   "metadata": {},
   "source": [
    "Now we run YOLO and create an output video where each object detected by YOLO is drawn with a bounding box."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ac2b138-6a81-4a6a-8e70-85982d0215c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "output_movie_filename = os.path.splitext(movie_filename)[0] + \"-output.mp4\"\n",
    "\n",
    "print(f'\"{movie_filename}\" -> \"{output_movie_filename}\"')\n",
    "\n",
    "cap = cv2.VideoCapture(movie_filename)\n",
    "w, h, fps = (int(cap.get(x)) for x in (cv2.CAP_PROP_FRAME_WIDTH, cv2.CAP_PROP_FRAME_HEIGHT, cv2.CAP_PROP_FPS))\n",
    "\n",
    "out = cv2.VideoWriter(output_movie_filename, cv2.VideoWriter_fourcc(*'mp4v'), fps, (w, h))\n",
    "\n",
    "for _ in range(100):\n",
    "    ret, im0 = cap.read()\n",
    "    if not ret:\n",
    "        print(\"Video frame is empty or video processing has been successfully completed.\")\n",
    "        break\n",
    "\n",
    "    results = model.predict(im0, verbose=False)\n",
    "    annotator = Annotator(im0, line_width=2)\n",
    "\n",
    "    for box in results[0].boxes:\n",
    "        cls = int(box.cls)\n",
    "        annotator.box_label(box.xyxy[0], model.names[cls], color=colors(cls))\n",
    "    out.write(im0)\n",
    "\n",
    "out.release()\n",
    "cap.release()\n",
    "print(\"done\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c56534d-b32a-4c9f-82de-341e13c77261",
   "metadata": {},
   "source": [
    "In practice, one may improve performance of a pre-trained model by *fine-tuning* for a particular task by using labelled data from your task. To limit the computational resources (and time) required, we did not perform any training in this exercise. Even the modest amount of tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc0d505e-df76-4cea-97b5-2b521b5a6f16",
   "metadata": {},
   "source": [
    "## Q2\n",
    "\n",
    "Now describe some aspects of the object detection that are functioned but also describe some aspects which did not work well. (Two or three sentences.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b796689b-edc8-43de-b7e7-15202706e455",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "747796a397c411b96072c68d84e6ddf6",
     "grade": true,
     "grade_id": "cell-0ee9df636020a250",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a4336ab-8261-42a6-8fd3-7c51a6348f89",
   "metadata": {},
   "source": [
    "## LLMs\n",
    "\n",
    "For the next questions, sign up on ILIAS for \"AI Chat\" as shown in the movie during lecture. Answer the questions to enable \"Pilotierung: AI Chat für das Studium\".\n",
    "\n",
    "Now, go to \"AI Chat (Modell: llama3.3:70b, Betrieb: RZ)\" as shown in the movie during lecture."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d74188e-482d-4486-a415-e48617bc3dc3",
   "metadata": {},
   "source": [
    "## Q3\n",
    "\n",
    "With the AI Chat, ask the model to explain something to you which you have learned about in one of your biology courses. Copy and paste your prompt and the answer in the next cell:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b93d2c66-e0da-43ec-9764-7701d75ee657",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "25edd3bbcd007a31d1f0b9fd7942da17",
     "grade": true,
     "grade_id": "cell-63f10d82a4d5ebd9",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "960b6e5a-47ca-408d-ad2f-37845ad808bb",
   "metadata": {},
   "source": [
    "## Q4\n",
    "\n",
    "Now, comment on the the quality of the response from the AI Chat: (two or three sentences)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1b9ea17-47f2-461a-99d5-f9f025de6380",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "ace265dc87dc26191ee49a7b9fcad653",
     "grade": true,
     "grade_id": "cell-9dcc354a187c8738",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}