straw/pm21-dragon
1
2
Fork 23
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity
pm21-dragon/lectures/lecture-05/1 - Command line, files, and processes.ipynb
Andrew Straw 9f59285c76 for today
2024-11-15 10:15:56 +01:00

21 KiB
Raw Blame History

None <html> <head> </head>

What is the unix command line (also known as shell)?

Why would I use the unix command line?

  • faster when you know what you are doing (slower when you are new)
  • full capabilities
  • automation
  • remote access (especially SSH)
  • running self-hosted cloud services
  • your router, NAS drive, and other devices probably run it

Matrix-SSH.png

How can I get access to a unix command line?

On your existing computer

  • On Windows, install WSL2 to run linux. Also native Windows command prompt very similar conceptually to unix command line, but most individual commands are slightly different.

  • On Mac or Linux: native terminal window

On a new computer

  • Puchase a Raspberry Pi computer. I recommend something like the Raspberry Pi Zero 2 W full starter kit with HDMI adapter, SD card, case, USB adapter, and power supply. This should cost about 45 EUR. This particular seller is listed as an example that I know from personal experience is OK.
  • Inexpensive normal PC or laptop. Also: Linux runs great on old hardware.

Remotely

Concepts

  • SSH
  • Shell prompt
  • Running programs from the command line
  • Reading command line arguments in Python
  • Exit codes
  • Navigating in the command line
  • Environment variables, especially PATH

SSH

Image from https://www.hostinger.com/tutorials/ssh-tutorial-how-does-ssh-work

ssh.jpg

Also great with tmux to preserve your session if you logout or if your network connection drops.

Login to the course server using SSH

Step 1 - ensure you are in the Uni Freiburg network

If you are not connected via a Uni Freiburg computer (including EduRoam), you must setup a VPN ("virtual private network"). See this page for details about how to install the relevant software and use the university VPN.

Step 2 - start a terminal program

  • On Windows, go to the start menu and type "Cmd" and then <Enter> to start the "Command Prompt" program. (Alternative: the oldie-but-goodie PuTTY program.)
  • On macOS, go to "Applications -> Utilities -> Terminal" (German: "Programme -> Dienstprogramme -> Terminal")
  • On linux, start the "Terminal" program.

Step 3 - your username and passoword

Your username is your the lowercase version of your lastname (e.g. Cem Özdemir would have a username of oezdemir). Your password is your matriculation number followed by "x" (e.g. 12345678x).

Step 4 - login with SSH

The following steps will only work from the Uni Freiburg network. If you are remote, you can use the VPN solution provided by the university.

Type this command in your terminal program:

ssh <username>@python-course.strawlab.org

First login with SSH

For your first login, the terminal should print something like:

The authenticity of host 'python-course.strawlab.org (10.4.66.100)' can't be established.
ED25519 key fingerprint is SHA256:MWMheGkCSvu8drY2gjS/9SQ7ZBBTRh+bQlj0SNblzyo.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])?

Type yes to continue.

Then you will see something like:

Warning: Permanently added 'python-course.strawlab.org' (ED25519) to the list of known hosts.

Entering password

After your first login, the steps in the section above will not happen as the SSH server will be "known" to your own system.

<username>@python-course.strawlab.org's password:

Here you type your password. You will not be able to see the characters as you type them.

Upon success, the following should be printed:

Linux dragon 6.1.0-27-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.115-1 (2024-11-01) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
straw@dragon:~$

The final line (<username>@dragon:~$) is called the command prompt and is where you type your commands.

When you press the enter (or return) key, the command will be executed by the remote computer. Your commands must be given in the "shell" language. Although there are some slight differences between different shell languages, we are going to stick with a common core of unix shell in this class.

Editing files from the command line

Demo of the use of the nano program as a plain text editor.

Note that the mouse usually does not work in the terminal!

Viewing files from the command line

Use of the cat program (which stands for concatenate).

Also the less program.

Demo: create and run Python from command line

Let's make my_program1.py which will print "Hello world".

Using our anaconda environment

The python we used above is not Python managed by Anaconda but rather the built-in python to the server's operating system. To use all of our libraries, we need to load our Anaconda environment. From the command line, run this:

eval "$(/anaconda3/bin/conda shell.bash hook)"

Upon success, your prompt should change from

straw@dragon:~$

to

(base) straw@dragon:~$

Notice the extra (base) at the beginning. This shows that you are in the "base" anaconda environment.

We can confirm this by typing which python, which should return /anaconda3/bin/python.

Finally, make sure our simple program still runs with this python. Type python my_program1.py:

(base) straw@dragon:~$ python my_program1.py
Hello world
(base) straw@dragon@strawlab-course-server:~$

Saving a plot of data

Let's now make a program to do some plotting and data analysis. Let's first make a program which simply plots some fake data and saves it to a .png file.

Create a new text file called plot1.py and set it to this content:

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

df = pd.DataFrame({'sepal length (cm)': [5.1, 4.9, 4.7, 4.6, 5.0, 5.4, 4.6, 5.0, 4.4, 4.9, 5.4, 4.8, 4.8, 4.3, 5.8, 5.7, 5.4, 5.1, 5.7, 5.1, 5.4, 5.1, 4.6, 5.1, 4.8, 5.0, 5.0, 5.2, 5.2, 4.7, 4.8, 5.4, 5.2, 5.5, 4.9, 5.0, 5.5, 4.9, 4.4, 5.1, 5.0, 4.5, 4.4, 5.0, 5.1, 4.8, 5.1, 4.6, 5.3, 5.0, 7.0, 6.4, 6.9, 5.5, 6.5, 5.7, 6.3, 4.9, 6.6, 5.2, 5.0, 5.9, 6.0, 6.1, 5.6, 6.7, 5.6, 5.8, 6.2, 5.6, 5.9, 6.1, 6.3, 6.1, 6.4, 6.6, 6.8, 6.7, 6.0, 5.7, 5.5, 5.5, 5.8, 6.0, 5.4, 6.0, 6.7, 6.3, 5.6, 5.5, 5.5, 6.1, 5.8, 5.0, 5.6, 5.7, 5.7, 6.2, 5.1, 5.7, 6.3, 5.8, 7.1, 6.3, 6.5, 7.6, 4.9, 7.3, 6.7, 7.2, 6.5, 6.4, 6.8, 5.7, 5.8, 6.4, 6.5, 7.7, 7.7, 6.0, 6.9, 5.6, 7.7, 6.3, 6.7, 7.2, 6.2, 6.1, 6.4, 7.2, 7.4, 7.9, 6.4, 6.3, 6.1, 7.7, 6.3, 6.4, 6.0, 6.9, 6.7, 6.9, 5.8, 6.8, 6.7, 6.7, 6.3, 6.5, 6.2, 5.9], 'sepal width (cm)': [3.5, 3.0, 3.2, 3.1, 3.6, 3.9, 3.4, 3.4, 2.9, 3.1, 3.7, 3.4, 3.0, 3.0, 4.0, 4.4, 3.9, 3.5, 3.8, 3.8, 3.4, 3.7, 3.6, 3.3, 3.4, 3.0, 3.4, 3.5, 3.4, 3.2, 3.1, 3.4, 4.1, 4.2, 3.1, 3.2, 3.5, 3.6, 3.0, 3.4, 3.5, 2.3, 3.2, 3.5, 3.8, 3.0, 3.8, 3.2, 3.7, 3.3, 3.2, 3.2, 3.1, 2.3, 2.8, 2.8, 3.3, 2.4, 2.9, 2.7, 2.0, 3.0, 2.2, 2.9, 2.9, 3.1, 3.0, 2.7, 2.2, 2.5, 3.2, 2.8, 2.5, 2.8, 2.9, 3.0, 2.8, 3.0, 2.9, 2.6, 2.4, 2.4, 2.7, 2.7, 3.0, 3.4, 3.1, 2.3, 3.0, 2.5, 2.6, 3.0, 2.6, 2.3, 2.7, 3.0, 2.9, 2.9, 2.5, 2.8, 3.3, 2.7, 3.0, 2.9, 3.0, 3.0, 2.5, 2.9, 2.5, 3.6, 3.2, 2.7, 3.0, 2.5, 2.8, 3.2, 3.0, 3.8, 2.6, 2.2, 3.2, 2.8, 2.8, 2.7, 3.3, 3.2, 2.8, 3.0, 2.8, 3.0, 2.8, 3.8, 2.8, 2.8, 2.6, 3.0, 3.4, 3.1, 3.0, 3.1, 3.1, 3.1, 2.7, 3.2, 3.3, 3.0, 2.5, 3.0, 3.4, 3.0], 'petal length (cm)': [1.4, 1.4, 1.3, 1.5, 1.4, 1.7, 1.4, 1.5, 1.4, 1.5, 1.5, 1.6, 1.4, 1.1, 1.2, 1.5, 1.3, 1.4, 1.7, 1.5, 1.7, 1.5, 1.0, 1.7, 1.9, 1.6, 1.6, 1.5, 1.4, 1.6, 1.6, 1.5, 1.5, 1.4, 1.5, 1.2, 1.3, 1.4, 1.3, 1.5, 1.3, 1.3, 1.3, 1.6, 1.9, 1.4, 1.6, 1.4, 1.5, 1.4, 4.7, 4.5, 4.9, 4.0, 4.6, 4.5, 4.7, 3.3, 4.6, 3.9, 3.5, 4.2, 4.0, 4.7, 3.6, 4.4, 4.5, 4.1, 4.5, 3.9, 4.8, 4.0, 4.9, 4.7, 4.3, 4.4, 4.8, 5.0, 4.5, 3.5, 3.8, 3.7, 3.9, 5.1, 4.5, 4.5, 4.7, 4.4, 4.1, 4.0, 4.4, 4.6, 4.0, 3.3, 4.2, 4.2, 4.2, 4.3, 3.0, 4.1, 6.0, 5.1, 5.9, 5.6, 5.8, 6.6, 4.5, 6.3, 5.8, 6.1, 5.1, 5.3, 5.5, 5.0, 5.1, 5.3, 5.5, 6.7, 6.9, 5.0, 5.7, 4.9, 6.7, 4.9, 5.7, 6.0, 4.8, 4.9, 5.6, 5.8, 6.1, 6.4, 5.6, 5.1, 5.6, 6.1, 5.6, 5.5, 4.8, 5.4, 5.6, 5.1, 5.1, 5.9, 5.7, 5.2, 5.0, 5.2, 5.4, 5.1], 'petal width (cm)': [0.2, 0.2, 0.2, 0.2, 0.2, 0.4, 0.3, 0.2, 0.2, 0.1, 0.2, 0.2, 0.1, 0.1, 0.2, 0.4, 0.4, 0.3, 0.3, 0.3, 0.2, 0.4, 0.2, 0.5, 0.2, 0.2, 0.4, 0.2, 0.2, 0.2, 0.2, 0.4, 0.1, 0.2, 0.2, 0.2, 0.2, 0.1, 0.2, 0.2, 0.3, 0.3, 0.2, 0.6, 0.4, 0.3, 0.2, 0.2, 0.2, 0.2, 1.4, 1.5, 1.5, 1.3, 1.5, 1.3, 1.6, 1.0, 1.3, 1.4, 1.0, 1.5, 1.0, 1.4, 1.3, 1.4, 1.5, 1.0, 1.5, 1.1, 1.8, 1.3, 1.5, 1.2, 1.3, 1.4, 1.4, 1.7, 1.5, 1.0, 1.1, 1.0, 1.2, 1.6, 1.5, 1.6, 1.5, 1.3, 1.3, 1.3, 1.2, 1.4, 1.2, 1.0, 1.3, 1.2, 1.3, 1.3, 1.1, 1.3, 2.5, 1.9, 2.1, 1.8, 2.2, 2.1, 1.7, 1.8, 1.8, 2.5, 2.0, 1.9, 2.1, 2.0, 2.4, 2.3, 1.8, 2.2, 2.3, 1.5, 2.3, 2.0, 2.0, 1.8, 2.1, 1.8, 1.8, 1.8, 2.1, 1.6, 1.9, 2.0, 2.2, 1.5, 1.4, 2.3, 2.4, 1.8, 1.8, 2.1, 2.4, 2.3, 1.9, 2.3, 2.5, 2.3, 1.9, 2.0, 2.3, 1.8], 'species': ['setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica']})

sns.stripplot(x="species", y="sepal width (cm)", data=df)
plt.savefig('plot1.png')

If we use the Jupyter Notebook, we can open our newly saved file and verify that it looks like this:

plot1.png

Command line arguments and exit codes

Running a program from the command line has many conceptual similarities to calling a function within Python. There are arguments passed to the program, the program returns a final result, and there are "side effects". Typically the programs run more for the "side effects" in this view. The final result of a program is merely a single integer that is 0 in the case of no error, and any non-zero number in case of error. This final result, an integer, is called the "exit code" or the "error code".

On the command line, arguments are given with a space. In fact, when we call python with our script, we are actually running the program python with our program as the first command line argment, e.g. my_program1.py.:

python my_program1.py

We can call it with additional arguments like this:

python my_program1.py arg1 arg2

If we run our program above, we see it still runs as before (printing Hello world) but at least there is no error. In a moment we will see how to read these arguments from Python. For now, let's check the exit code. We do this with the command:

echo $?

What this does is call the echo program with a special variable named ?. At the command line, the $ means to get the variable, and ? means the last exit code. So you must be careful to run this command immediately after a command where you want to know the exit code. So, this command is again using a command line argument ($?) to the program echo. It should print 0 in case of no error.

Now let's use these command line arguments from our python program. For this we import sys and use sys.argv which is a list of all command line arguments. If we create a program my_program2.py to have the following contents:

import sys
print(sys.argv)

Run it like so:

(pm21-tiger) <username>@strawlab-course-server:~$ python my_program2.py
['my_program2.py']
(pm21-tiger) <username>@strawlab-course-server:~$

We can also run it with arguments:

(pm21-tiger) astraw@strawlab-course-server:~$ python my_program2.py arg1 arg2
['my_program2.py', 'arg1', 'arg2']
(pm21-tiger) astraw@strawlab-course-server:~$

Note that all arguments are strings.

It is common to pass filenames as command line arguments. The python program will receive filenames (like all command line arguments) as strings and your program will have to interpret them as filenames.

(Power user: argparse library for parsing command line arguments.)

topic: "current working directory"

Every program has a "current working directory".

To view it in Python:

import os

print(os.getcwd())

Navigation at the command line

(Mouse typically does not work.)

Up arrow, down arrow, <tab>.

(Power user: <Ctrl-r> search)

pwd Present working directory

ls list files

ls -a list all files, including hidden files

ls -l list "long"

ls -al list all files "long"

cd change directory (to home dir)

cd somedir change directory to somedir

(Power user: cd -)

cat filename will print the contents of the file filename to the screen

Paths (also known as filesystem locations)

Directories are separated with /. Paths can be "absolute" and start with / or relative, in which case they may start with ./.

  • . is the current directory
  • .. is the parent directory

Environment variables at the command line

Environment variables are set by the operating system in the command line environment and are used to pass information to programs without explicitly using arguments. They are thus like variables in python in the global frame which are accessed from within a function.

We already saw $?.

$HOME

$PATH To start a program, this is searched unless the program name has a path (either absolute: /some/path/program-name or relative: ./program-name)

Note: anaconda works by setting the $PATH and related $PYTHONPATH.

Useful unix programs

ssh

cat view a file ("concatenate" it to the command line)

less view a file, page by page

nano (or, for a power user: Visual Studio Code with Remote SSH extension)

Programs we won't use, but are very useful:

rg ripgrep - quickly search in recursively descended file tree

curl download from URLs

scp secure copy

tmux terminal multiplexer

Reading and plotting data from the command line

Copy the iris.csv file

cp /data/iris.csv .

Now create plot2.py to have the following contents:

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import sys

fname = sys.argv[1]
print(f'reading {fname}')

df = pd.read_csv(fname)
sns.stripplot(x="species", y="sepal_width", data=df)
plt.savefig('plot2.png')

Now we can run this with: python plot2.py iris.csv

Further resources

</html>