{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Network and web technologies\n", "\n", "(I.e. how can we interact with services on other computers from Python?)\n", "\n", "[Teletype machines](https://en.wikipedia.org/wiki/Teleprinter) \"were adapted to provide a user interface to early mainframe computers and minicomputers, sending typed data to the computer and printing the response.\"\n", "\n", "![WACsOperateTeletype.jpg](WACsOperateTeletype.jpg) WWII era teletype machines. \n", "Source: https://commons.wikimedia.org/wiki/File:WACsOperateTeletype.jpg" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Let's start with an idealized \"serial port\"\n", "\n", "![serial-port.png](serial-port.png)\n", "\n", "Data is carried as a *serial* stream of bits (1s and 0s, high and low voltage).\n", "\n", "We can think about teletype machines, USB, ethernet cables, WiFi/WLAN, and so on:\n", "\n", "(From https://www.electroschematics.com/wp-content/uploads/2010/01/usb-wiring-connection.jpg?fit=1024%2C768)\n", "![usb-wiring-connection.jpg](usb-wiring-connection.jpg)\n", "\n", "![receiver-wiring-to-FC.png](receiver-wiring-to-FC.png) Source: https://www.expresslrs.org/quick-start/receivers/wiring-up/\n", "\n", "(Ethernet cables get more sophisticated. https://serverfault.com/questions/449416/why-do-ethernet-cables-have-8-wires . This is a theme that repeats itself in computer networking.)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "But this is not a \"network\", only two computers\n", "\n", "## From serial to network\n", "\n", "We can build a robust network composed of many individual serial links by defining a common language -- namely packets with framing information - such as a destination address - and a payload. The individual nodes in the network use the framing information to forward the packet or to consume it.\n", "\n", "This standardization has allowed hardware and software producers to interoperate by having common interfaces. Here is an ethernet frame:\n", "\n", "![Ethernet_Type_II_Frame_format.svg.png](Ethernet_Type_II_Frame_format.svg.png) Source: https://commons.wikimedia.org/wiki/File:Ethernet_Type_II_Frame_format.svg\n", "\n", "USB also uses the idea of frames to establish a standard for a robust single link \"network\". Here is a USB frame:\n", "\n", "![640px-USB_signal_example.svg.png](640px-USB_signal_example.svg.png) \"Data packets would have address field and payload between the packet ID and the end of packet.\" Source: https://commons.wikimedia.org/wiki/File:USB_signal_example.svg\n", "\n", "![640px-Speedport_W_921V,_Deutsche_Telekom-7294.jpg](640px-Speedport_W_921V,_Deutsche_Telekom-7294.jpg)\n", "Source: https://de.m.wikipedia.org/wiki/Datei:Speedport_W_921V,_Deutsche_Telekom-7294.jpg\n", "\n", "![640px-Speedport_W_921V,_Deutsche_Telekom-7296.jpg](640px-Speedport_W_921V,_Deutsche_Telekom-7296.jpg)\n", "Source: https://commons.wikimedia.org/wiki/File:Speedport_W_921V,_Deutsche_Telekom-7296.jpg\n", "\n", "\n", "![Wireless-router-network-diagram.png](Wireless-router-network-diagram.png) Source: https://www.conceptdraw.com/How-To-Guide/what-is-a-wireless-network" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# DHCP - Dynamic Host Configuration Protocol\n", "\n", "DHCP is the protocol used when your computer first joins a network and asks for an IP address.\n", "\n", "![DHCP-protocol.png](DHCP-protocol.png) Source: https://www.boardinfinity.com/blog/dynamic-host-configuration-protocol-dhcp/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Layers of networking\n", "\n", "![UDP_encapsulation.png](UDP_encapsulation.png)\n", "Source: https://en.wikipedia.org/wiki/File:UDP_encapsulation.png\n", "\n", "From http://www.cellbiol.com/bioinformatics_web_development/chapter-1-internet-networks-and-tcp-ip/the-tcpip-family-of-internet-protocols/\n", "\n", "![tpc-ip-and-osi-model-cellbiol.com_.png](tpc-ip-and-osi-model-cellbiol.com_.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "These notes give an overview of the most important of these levels with the idea of understanding what your computer is doing when you use the web browser or how a program you write would interact automatically with, for example, a remote server to perform a BLAST query." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## TCP and UDP\n", "\n", "From https://microchipdeveloper.com/tcpip:tcp-vs-udp\n", "![tcp-vs-udp.jpg](tcp-vs-udp.jpg)\n", "\n", "In addition to source and destination address, TCP and UDP frames have \"port\" and thus address has a 2^16 (65536) TCP ports and 2^16 UDP ports." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "UDP joke https://micromismanagement.com/cartoons/udp/\n", "\n", "![udp-joke.png](udp-joke.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Socket\n", "\n", "From https://www.codeproject.com/Articles/1264257/Socket-Programming-in-Cplusplus-using-boost-asio-T\n", "\n", "![socket.png](socket.png)\n", "\n", "From http://ithare.com/network-programming-socket-peculiarities-threads-and-testing/\n", "\n", "![socket-cartoon.png](socket-cartoon.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# DNS and IP addresses\n", "\n", "From https://blog.octo.com/en/comic-introduction-to-networks-ip-addresses/\n", "\n", "![dns-ip.png](dns-ip.png)\n", "\n", "From https://foxutech.com/what-is-dns-and-how-it-works/\n", "\n", "![dns.webp](dns.webp)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "DNS https://howdns.works/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "https://ruslanspivak.com/lsbaws-part1/\n", "\n", "https://jvns.ca/blog/2019/09/06/how-to-put-an-html-page-on-the-internet/\n" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Image source https://twitter.com/b0rk/status/1161997141876903936 (linked from [Julia Evans' blog](https://jvns.ca/blog/2019/09/12/new-zine-on-http/))\n", "\n", "![urls.jpg](urls.jpg)\n", "\n", "# -----\n", "\n", "Image source https://ruslanspivak.com/lsbaws-part1/\n", "\n", "![web-server.png](web-server.png)\n", "\n", "# -----\n", "\n", "Image source https://twitter.com/b0rk/status/1155318552129396736\n", "\n", "![http-basics.jpg](http-basics.jpg)\n", "\n", "# -----\n", "\n", "Image source https://twitter.com/b0rk/status/1145362860136177664\n", "\n", "![anatomy-of-http-request.jpg](anatomy-of-http-request.jpg)\n", "\n", "\n", "# -----\n", "\n", "Image source https://twitter.com/b0rk/status/1161679906415218690\n", "\n", "![http-request-methods.jpg](http-request-methods.jpg)\n", "\n", "# -----\n", "\n", "Image source https://twitter.com/b0rk/status/1145896193077256197\n", "\n", "![http-response-methods.jpg](http-response-methods.jpg)\n", "\n", "# -----\n", "\n", "Image source https://twitter.com/b0rk/status/1146054159214567424\n", "\n", "![post-requests.jpg](post-requests.jpg)\n", "\n", "# -----\n", "\n", "Image source https://twitter.com/b0rk/status/1159839824594915335\n", "\n", "![http-exercises.jpg](http-exercises.jpg)\n", "\n", "# -----\n", "\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", " \n", " \n", " Error 404 (Not Found)!!1\n", " \n", " \n", "

404. That’s an error.\n", "

The requested URL /bananas was not found on this server. That’s all we know.\n" ] } ], "source": [ "!curl http://google.com/bananas" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's practice with the [Star Wars API](https://swapi.py4e.com/).\n", "\n", "```curl https://swapi.py4e.com/api/people/1/```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We will use the [requests](https://pypi.org/project/requests/) library." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "import requests" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'{\"name\":\"Darth Vader\",\"height\":\"202\",\"mass\":\"136\",\"hair_color\":\"none\",\"skin_color\":\"white\",\"eye_color\":\"yellow\",\"birth_year\":\"41.9BBY\",\"gender\":\"male\",\"homeworld\":\"https://swapi.py4e.com/api/planets/1/\",\"films\":[\"https://swapi.py4e.com/api/films/1/\",\"https://swapi.py4e.com/api/films/2/\",\"https://swapi.py4e.com/api/films/3/\",\"https://swapi.py4e.com/api/films/6/\"],\"species\":[\"https://swapi.py4e.com/api/species/1/\"],\"vehicles\":[],\"starships\":[\"https://swapi.py4e.com/api/starships/13/\"],\"created\":\"2014-12-10T15:18:20.704000Z\",\"edited\":\"2014-12-20T21:17:50.313000Z\",\"url\":\"https://swapi.py4e.com/api/people/4/\"}'" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "response = requests.get('https://swapi.py4e.com/api/people/4/')\n", "response.text" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "requests.models.Response" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "type(response)" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "str" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "type(response.text)" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'Date': 'Fri, 10 Jan 2025 09:07:09 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Vary': 'Accept-Encoding, Accept, Cookie', 'X-Frame-Options': 'SAMEORIGIN', 'ETag': 'W/\"84f396e29689886d73163ffeea41d4b8\"', 'Allow': 'GET, HEAD, OPTIONS', 'X-Clacks-Overhead': 'GNU Terry Pratchett', 'Content-Encoding': 'gzip', 'Age': '10', 'Cache-Control': 'max-age=14400', 'cf-cache-status': 'HIT', 'Report-To': '{\"endpoints\":[{\"url\":\"https:\\\\/\\\\/a.nel.cloudflare.com\\\\/report\\\\/v4?s=k1tyqow%2BTHfXBggdLlKOGfY3YUvPQIZw2P1EowugfuLk%2F2nzXWw0X%2FjfUeLAhQcRikRxCoEH%2FDEcASi998U%2FcN70vLqQIRPjvoxABihBwDr9ApD2b%2BpBaqDMJ2ujpKeewA%3D%3D\"}],\"group\":\"cf-nel\",\"max_age\":604800}', 'NEL': '{\"success_fraction\":0,\"report_to\":\"cf-nel\",\"max_age\":604800}', 'Server': 'cloudflare', 'CF-RAY': '8ffb83dfbba02c0a-STR', 'alt-svc': 'h3=\":443\"; ma=86400', 'server-timing': 'cfL4;desc=\"?proto=TCP&rtt=4615&min_rtt=4583&rtt_var=1314&sent=4&recv=7&lost=0&retrans=0&sent_bytes=2830&recv_bytes=781&delivery_rate=630387&cwnd=251&unsent_bytes=0&cid=59f4e5c9b008e54f&ts=22&x=0\"'}" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "response.headers" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'name': 'Darth Vader',\n", " 'height': '202',\n", " 'mass': '136',\n", " 'hair_color': 'none',\n", " 'skin_color': 'white',\n", " 'eye_color': 'yellow',\n", " 'birth_year': '41.9BBY',\n", " 'gender': 'male',\n", " 'homeworld': 'https://swapi.py4e.com/api/planets/1/',\n", " 'films': ['https://swapi.py4e.com/api/films/1/',\n", " 'https://swapi.py4e.com/api/films/2/',\n", " 'https://swapi.py4e.com/api/films/3/',\n", " 'https://swapi.py4e.com/api/films/6/'],\n", " 'species': ['https://swapi.py4e.com/api/species/1/'],\n", " 'vehicles': [],\n", " 'starships': ['https://swapi.py4e.com/api/starships/13/'],\n", " 'created': '2014-12-10T15:18:20.704000Z',\n", " 'edited': '2014-12-20T21:17:50.313000Z',\n", " 'url': 'https://swapi.py4e.com/api/people/4/'}" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x = response.json()\n", "x" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/plain": [ "'Darth Vader'" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x = response.json()\n", "print(type(x))\n", "x['name']" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'C-3PO'" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "requests.get('https://swapi.py4e.com/api/people/2/').json()['name']" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Serving HTTP on port 4448 ...\n", "waiting for connection\n", "Got connection from ('127.0.0.1', 61404)\n", "GET / HTTP/1.1\n", "\n", "waiting for connection\n" ] }, { "ename": "KeyboardInterrupt", "evalue": "", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)", "Cell \u001b[0;32mIn[14], line 13\u001b[0m\n\u001b[1;32m 11\u001b[0m \u001b[38;5;28;01mwhile\u001b[39;00m \u001b[38;5;28;01mTrue\u001b[39;00m:\n\u001b[1;32m 12\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mwaiting for connection\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m---> 13\u001b[0m client_connection, client_address \u001b[38;5;241m=\u001b[39m \u001b[43mlisten_socket\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43maccept\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 14\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mGot connection from \u001b[39m\u001b[38;5;132;01m{}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;241m.\u001b[39mformat(client_address))\n\u001b[1;32m 15\u001b[0m request_data \u001b[38;5;241m=\u001b[39m client_connection\u001b[38;5;241m.\u001b[39mrecv(\u001b[38;5;241m1024\u001b[39m)\n", "File \u001b[0;32m~/anaconda3/envs/wm01-dragon/lib/python3.11/socket.py:294\u001b[0m, in \u001b[0;36msocket.accept\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 287\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21maccept\u001b[39m(\u001b[38;5;28mself\u001b[39m):\n\u001b[1;32m 288\u001b[0m \u001b[38;5;250m \u001b[39m\u001b[38;5;124;03m\"\"\"accept() -> (socket object, address info)\u001b[39;00m\n\u001b[1;32m 289\u001b[0m \n\u001b[1;32m 290\u001b[0m \u001b[38;5;124;03m Wait for an incoming connection. Return a new socket\u001b[39;00m\n\u001b[1;32m 291\u001b[0m \u001b[38;5;124;03m representing the connection, and the address of the client.\u001b[39;00m\n\u001b[1;32m 292\u001b[0m \u001b[38;5;124;03m For IP sockets, the address info is a pair (hostaddr, port).\u001b[39;00m\n\u001b[1;32m 293\u001b[0m \u001b[38;5;124;03m \"\"\"\u001b[39;00m\n\u001b[0;32m--> 294\u001b[0m fd, addr \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_accept\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 295\u001b[0m sock \u001b[38;5;241m=\u001b[39m socket(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mfamily, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtype, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mproto, fileno\u001b[38;5;241m=\u001b[39mfd)\n\u001b[1;32m 296\u001b[0m \u001b[38;5;66;03m# Issue #7995: if no default timeout is set and the listening\u001b[39;00m\n\u001b[1;32m 297\u001b[0m \u001b[38;5;66;03m# socket had a (non-zero) timeout, force the new socket in blocking\u001b[39;00m\n\u001b[1;32m 298\u001b[0m \u001b[38;5;66;03m# mode to override platform-specific socket flags inheritance.\u001b[39;00m\n", "\u001b[0;31mKeyboardInterrupt\u001b[0m: " ] } ], "source": [ "# webserver1.py from https://ruslanspivak.com/lsbaws-part1/\n", "\n", "import socket\n", "HOST, PORT = '0.0.0.0', 4448\n", "\n", "listen_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n", "listen_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n", "listen_socket.bind((HOST, PORT))\n", "listen_socket.listen(1)\n", "print(f'Serving HTTP on port {PORT} ...')\n", "while True:\n", " print(\"waiting for connection\")\n", " client_connection, client_address = listen_socket.accept()\n", " print(\"Got connection from {}\".format(client_address))\n", " request_data = client_connection.recv(1024)\n", " print(request_data.decode('utf-8'))\n", "\n", " http_response = b\"\"\"\\\n", "HTTP/1.1 200 OK\n", "\n", "Hello, World! Guten tag.\n", "\"\"\"\n", " client_connection.sendall(http_response)\n", " client_connection.close()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With telnet:\n", "\n", "```\n", "telnet 127.0.0.1 4448\n", "GET / HTTP/1.1\n", "Host: blahblah.com\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# HTML, CSS, and Javascript and WASM - browser languages\n", "\n", "https://developer.mozilla.org/en-US/docs/Web\n", "\n", "E.g. https://codepen.io/picks/pens/\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# HTTP APIs\n", "\n", "Computer to computer over the internet\n", "\n", "Image from https://atesterthing.wordpress.com/2019/03/16/api-and-web-service-definition-difference-and-example/\n", "\n", "![web-api.webp](web-api.webp)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Serialization \n", "\n", "From https://medium.com/@dilankam/java-serialization-4ff2d5cf5fa8\n", "\n", "![serde.png](serde.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# JSON\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", "\n" ] }, { "data": { "text/plain": [ "{'key1': [1.2, {'32': 64, 'c': \"sdfg'dfsg\"}, 3, 4]}" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import json\n", "\n", "data1 = {'key1':[1.2,{32: 64,\"c\":\"sdfg'dfsg\"},3,4]}\n", "\n", "print(type(data1))\n", "buf = json.dumps(data1)\n", "print(type(buf))\n", "# buf\n", "\n", "data2 = json.loads(buf)\n", "print(type(data2))\n", "data2" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'{\"key1\": [1.2, {\"32\": 64, \"c\": \"sdfg\\'dfsg\"}, 3, 4]}'" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "buf" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# SSH\n", "\n", "Remember SSH?\n", "\n", "Image from https://www.hostinger.com/tutorials/ssh-tutorial-how-does-ssh-work\n", "\n", "![ssh.jpg](ssh.jpg)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.7" } }, "nbformat": 4, "nbformat_minor": 4 }