23 KiB
What is the unix command line (also known as shell)?¶
Why would I use the unix command line?¶
- faster when you know what you are doing (slower when you are new)
- full capabilities
- automation
- remote access (especially SSH)
- running self-hosted cloud services
- your router, NAS drive, and other devices probably run it
How can I get access to a unix command line?¶
On your existing computer¶
On Windows, install WSL2 to run linux. Also native Windows command prompt very similar conceptually to unix command line, but most individual commands are slightly different.
On Mac or Linux: native terminal window
On a new computer¶
- Puchase a Raspberry Pi computer. I recommend something like the Raspberry Pi Zero 2 W full starter kit with HDMI adapter, SD card, case, USB adapter, and power supply. This should cost about 45 EUR. This particular seller is listed as an example that I know from personal experience is OK.
- Inexpensive normal PC or laptop. Also: Linux runs great on old hardware.
Remotely¶
- Accounts on university computers (Rechenzentum, the course server we will use for the exercise, ...).
- Online "cloud" providers: https://www.hetzner.com/cloud, https://www.linode.com/pricing/#compute-shared, https://www.digitalocean.com/products/droplets and many more. Typically starting around 5 EUR per month with hourly billing.
Concepts¶
- SSH
- Shell prompt
- Running programs from the command line
- Reading command line arguments in Python
- Exit codes
- Navigating in the command line
- Environment variables, especially PATH
SSH¶
Image from https://www.hostinger.com/tutorials/ssh-tutorial-how-does-ssh-work
Also great with tmux to preserve your session if you logout or if your network connection drops.
Login to the course server using SSH¶
Step 1 - ensure you are in the Uni Freiburg network¶
If you are not connected via a Uni Freiburg computer (including EduRoam), you must setup a VPN ("virtual private network"). See this page for details about how to install the relevant software and use the university VPN.
Step 2 - start a terminal program¶
- On Windows, go to the start menu and type "Cmd" and then
<Enter>to start the "Command Prompt" program. (Alternative: the oldie-but-goodie PuTTY program.) - On macOS, go to "Applications -> Utilities -> Terminal" (German: "Programme -> Dienstprogramme -> Terminal")
- On linux, start the "Terminal" program.
Step 3 - your username and password¶
Your username is your the lowercase version of your lastname (e.g. Cem Özdemir would have a username of oezdemir). Your password is your matriculation number followed by "x" (e.g. 12345678x).
Step 4 - login with SSH¶
The following steps will only work from the Uni Freiburg network. If you are remote, you can use the VPN solution provided by the university.
Type this command in your terminal program:
ssh <username>@python-course.strawlab.org
First login with SSH¶
For your first login, the terminal should print something like:
The authenticity of host 'python-course.strawlab.org (10.4.66.100)' can't be established.
ED25519 key fingerprint is SHA256:MWMheGkCSvu8drY2gjS/9SQ7ZBBTRh+bQlj0SNblzyo.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])?Type yes to continue.
Then you will see something like:
Warning: Permanently added 'python-course.strawlab.org' (ED25519) to the list of known hosts.Entering password¶
After your first login, the steps in the section above will not happen as the SSH server will be "known" to your own system.
<username>@python-course.strawlab.org's password:Here you type your password. You will not be able to see the characters as you type them.
Upon success, the following should be printed:
Linux dragon 6.1.0-27-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.115-1 (2024-11-01) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
straw@dragon:~$The final line (<username>@dragon:~$) is called the command prompt and is where you type your commands.
When you press the enter (or return) key, the command will be executed by the remote computer. Your commands must be given in the "shell" language. Although there are some slight differences between different shell languages, we are going to stick with a common core of unix shell in this class.
Editing files from the command line¶
Demo of the use of the nano program as a plain text editor.
Note that the mouse usually does not work in the terminal!
Viewing files from the command line¶
Use of the cat program (which stands for concatenate).
Also the less program.
Demo: create and run Python from command line¶
Let's make my_program1.py which will print "Hello world".
Using our anaconda environment¶
The python we used above is not Python managed by Anaconda but rather the built-in python to the server's operating system. To use all of our libraries, we need to load our Anaconda environment. From the command line, run this:
eval "$(/anaconda3/bin/conda shell.bash hook)"Upon success, your prompt should change from
straw@dragon:~$
to
(base) straw@dragon:~$
Notice the extra (base) at the beginning. This shows that you are in the "base" anaconda environment.
We can confirm this by typing which python, which should return /anaconda3/bin/python.
Finally, make sure our simple program still runs with this python. Type python my_program1.py:
(base) straw@dragon:~$ python my_program1.py
Hello world
(base) straw@dragon@strawlab-course-server:~$Saving a plot of data¶
Let's now make a program to do some plotting and data analysis. Let's first make a program which simply plots some fake data and saves it to a .png file.
Create a new text file called plot1.py and set it to this content:
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
df = pd.DataFrame({'sepal length (cm)': [5.1, 4.9, 4.7, 4.6, 5.0, 5.4, 4.6, 5.0, 4.4, 4.9, 5.4, 4.8, 4.8, 4.3, 5.8, 5.7, 5.4, 5.1, 5.7, 5.1, 5.4, 5.1, 4.6, 5.1, 4.8, 5.0, 5.0, 5.2, 5.2, 4.7, 4.8, 5.4, 5.2, 5.5, 4.9, 5.0, 5.5, 4.9, 4.4, 5.1, 5.0, 4.5, 4.4, 5.0, 5.1, 4.8, 5.1, 4.6, 5.3, 5.0, 7.0, 6.4, 6.9, 5.5, 6.5, 5.7, 6.3, 4.9, 6.6, 5.2, 5.0, 5.9, 6.0, 6.1, 5.6, 6.7, 5.6, 5.8, 6.2, 5.6, 5.9, 6.1, 6.3, 6.1, 6.4, 6.6, 6.8, 6.7, 6.0, 5.7, 5.5, 5.5, 5.8, 6.0, 5.4, 6.0, 6.7, 6.3, 5.6, 5.5, 5.5, 6.1, 5.8, 5.0, 5.6, 5.7, 5.7, 6.2, 5.1, 5.7, 6.3, 5.8, 7.1, 6.3, 6.5, 7.6, 4.9, 7.3, 6.7, 7.2, 6.5, 6.4, 6.8, 5.7, 5.8, 6.4, 6.5, 7.7, 7.7, 6.0, 6.9, 5.6, 7.7, 6.3, 6.7, 7.2, 6.2, 6.1, 6.4, 7.2, 7.4, 7.9, 6.4, 6.3, 6.1, 7.7, 6.3, 6.4, 6.0, 6.9, 6.7, 6.9, 5.8, 6.8, 6.7, 6.7, 6.3, 6.5, 6.2, 5.9], 'sepal width (cm)': [3.5, 3.0, 3.2, 3.1, 3.6, 3.9, 3.4, 3.4, 2.9, 3.1, 3.7, 3.4, 3.0, 3.0, 4.0, 4.4, 3.9, 3.5, 3.8, 3.8, 3.4, 3.7, 3.6, 3.3, 3.4, 3.0, 3.4, 3.5, 3.4, 3.2, 3.1, 3.4, 4.1, 4.2, 3.1, 3.2, 3.5, 3.6, 3.0, 3.4, 3.5, 2.3, 3.2, 3.5, 3.8, 3.0, 3.8, 3.2, 3.7, 3.3, 3.2, 3.2, 3.1, 2.3, 2.8, 2.8, 3.3, 2.4, 2.9, 2.7, 2.0, 3.0, 2.2, 2.9, 2.9, 3.1, 3.0, 2.7, 2.2, 2.5, 3.2, 2.8, 2.5, 2.8, 2.9, 3.0, 2.8, 3.0, 2.9, 2.6, 2.4, 2.4, 2.7, 2.7, 3.0, 3.4, 3.1, 2.3, 3.0, 2.5, 2.6, 3.0, 2.6, 2.3, 2.7, 3.0, 2.9, 2.9, 2.5, 2.8, 3.3, 2.7, 3.0, 2.9, 3.0, 3.0, 2.5, 2.9, 2.5, 3.6, 3.2, 2.7, 3.0, 2.5, 2.8, 3.2, 3.0, 3.8, 2.6, 2.2, 3.2, 2.8, 2.8, 2.7, 3.3, 3.2, 2.8, 3.0, 2.8, 3.0, 2.8, 3.8, 2.8, 2.8, 2.6, 3.0, 3.4, 3.1, 3.0, 3.1, 3.1, 3.1, 2.7, 3.2, 3.3, 3.0, 2.5, 3.0, 3.4, 3.0], 'petal length (cm)': [1.4, 1.4, 1.3, 1.5, 1.4, 1.7, 1.4, 1.5, 1.4, 1.5, 1.5, 1.6, 1.4, 1.1, 1.2, 1.5, 1.3, 1.4, 1.7, 1.5, 1.7, 1.5, 1.0, 1.7, 1.9, 1.6, 1.6, 1.5, 1.4, 1.6, 1.6, 1.5, 1.5, 1.4, 1.5, 1.2, 1.3, 1.4, 1.3, 1.5, 1.3, 1.3, 1.3, 1.6, 1.9, 1.4, 1.6, 1.4, 1.5, 1.4, 4.7, 4.5, 4.9, 4.0, 4.6, 4.5, 4.7, 3.3, 4.6, 3.9, 3.5, 4.2, 4.0, 4.7, 3.6, 4.4, 4.5, 4.1, 4.5, 3.9, 4.8, 4.0, 4.9, 4.7, 4.3, 4.4, 4.8, 5.0, 4.5, 3.5, 3.8, 3.7, 3.9, 5.1, 4.5, 4.5, 4.7, 4.4, 4.1, 4.0, 4.4, 4.6, 4.0, 3.3, 4.2, 4.2, 4.2, 4.3, 3.0, 4.1, 6.0, 5.1, 5.9, 5.6, 5.8, 6.6, 4.5, 6.3, 5.8, 6.1, 5.1, 5.3, 5.5, 5.0, 5.1, 5.3, 5.5, 6.7, 6.9, 5.0, 5.7, 4.9, 6.7, 4.9, 5.7, 6.0, 4.8, 4.9, 5.6, 5.8, 6.1, 6.4, 5.6, 5.1, 5.6, 6.1, 5.6, 5.5, 4.8, 5.4, 5.6, 5.1, 5.1, 5.9, 5.7, 5.2, 5.0, 5.2, 5.4, 5.1], 'petal width (cm)': [0.2, 0.2, 0.2, 0.2, 0.2, 0.4, 0.3, 0.2, 0.2, 0.1, 0.2, 0.2, 0.1, 0.1, 0.2, 0.4, 0.4, 0.3, 0.3, 0.3, 0.2, 0.4, 0.2, 0.5, 0.2, 0.2, 0.4, 0.2, 0.2, 0.2, 0.2, 0.4, 0.1, 0.2, 0.2, 0.2, 0.2, 0.1, 0.2, 0.2, 0.3, 0.3, 0.2, 0.6, 0.4, 0.3, 0.2, 0.2, 0.2, 0.2, 1.4, 1.5, 1.5, 1.3, 1.5, 1.3, 1.6, 1.0, 1.3, 1.4, 1.0, 1.5, 1.0, 1.4, 1.3, 1.4, 1.5, 1.0, 1.5, 1.1, 1.8, 1.3, 1.5, 1.2, 1.3, 1.4, 1.4, 1.7, 1.5, 1.0, 1.1, 1.0, 1.2, 1.6, 1.5, 1.6, 1.5, 1.3, 1.3, 1.3, 1.2, 1.4, 1.2, 1.0, 1.3, 1.2, 1.3, 1.3, 1.1, 1.3, 2.5, 1.9, 2.1, 1.8, 2.2, 2.1, 1.7, 1.8, 1.8, 2.5, 2.0, 1.9, 2.1, 2.0, 2.4, 2.3, 1.8, 2.2, 2.3, 1.5, 2.3, 2.0, 2.0, 1.8, 2.1, 1.8, 1.8, 1.8, 2.1, 1.6, 1.9, 2.0, 2.2, 1.5, 1.4, 2.3, 2.4, 1.8, 1.8, 2.1, 2.4, 2.3, 1.9, 2.3, 2.5, 2.3, 1.9, 2.0, 2.3, 1.8], 'species': ['setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'setosa', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'versicolor', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica', 'virginica']})
sns.stripplot(x="species", y="sepal width (cm)", data=df)
plt.savefig('plot1.png')
If we use the Jupyter Notebook, we can open our newly saved file and verify that it looks like this:
Command line arguments and exit codes¶
Running a program from the command line has many conceptual similarities to calling a function within Python. There are arguments passed to the program, the program returns a final result, and there are "side effects". Typically the programs run more for the "side effects" in this view. The final result of a program is merely a single integer that is 0 in the case of no error, and any non-zero number in case of error. This final result, an integer, is called the "exit code" or the "error code".
On the command line, arguments are given with a space. In fact, when we call python with our script, we are actually running the program python with our program as the first command line argment, e.g. my_program1.py.:
python my_program1.py
We can call it with additional arguments like this:
python my_program1.py arg1 arg2
If we run our program above, we see it still runs as before (printing Hello world) but at least there is no error. In a moment we will see how to read these arguments from Python. For now, let's check the exit code. We do this with the command:
echo $?
What this does is call the echo program with a special variable named ?. At the command line, the $ means to get the variable, and ? means the last exit code. So you must be careful to run this command immediately after a command where you want to know the exit code. So, this command is again using a command line argument ($?) to the program echo. It should print 0 in case of no error.
Now let's use these command line arguments from our python program. For this we import sys and use sys.argv which is a list of all command line arguments. If we create a program my_program2.py to have the following contents:
import sys
print(sys.argv)
Run it like so:
(pm21-tiger) <username>@strawlab-course-server:~$ python my_program2.py
['my_program2.py']
(pm21-tiger) <username>@strawlab-course-server:~$We can also run it with arguments:
(pm21-tiger) astraw@strawlab-course-server:~$ python my_program2.py arg1 arg2
['my_program2.py', 'arg1', 'arg2']
(pm21-tiger) astraw@strawlab-course-server:~$Note that all arguments are strings.
It is common to pass filenames as command line arguments. The python program will receive filenames (like all command line arguments) as strings and your program will have to interpret them as filenames.
(Power user: argparse library for parsing command line arguments.)
topic: "current working directory"¶
Every program has a "current working directory".
To view it in Python:
import os
print(os.getcwd())
Navigation at the command line¶
(Mouse typically does not work.)
Up arrow, down arrow, <tab>.
(Power user: <Ctrl-r> search)
pwd Present working directory
ls list files
ls -a list all files, including hidden files
ls -l list "long"
ls -al list all files "long"
cd change directory (to home dir)
cd somedir change directory to somedir
(Power user: cd -)
cat filename will print the contents of the file filename to the screen
Paths (also known as filesystem locations)¶
Directories are separated with /. Paths can be "absolute" and start with / or relative, in which case they may start with ./.
.is the current directory..is the parent directory
Environment variables at the command line¶
Environment variables are set by the operating system in the command line environment and are used to pass information to programs without explicitly using arguments. They are thus like variables in python in the global frame which are accessed from within a function.
We already saw $?.
$HOME
$PATH To start a program, this is searched unless the program name has a path (either absolute: /some/path/program-name or relative: ./program-name)
Note: anaconda works by setting the $PATH and related $PYTHONPATH.
Useful unix programs¶
ssh
cat view a file ("concatenate" it to the command line)
less view a file, page by page
nano (or, for a power user: Visual Studio Code with Remote SSH extension)
Programs we won't use, but are very useful:
rg ripgrep - quickly search in recursively descended file tree
curl download from URLs
scp secure copy
tmux terminal multiplexer
Markdown (.md) files¶
Markdown files are another file type (like Python .py files) which can be edited as "plain text". In this case, it is used for writing documentation with things like bold words and links.
It is very common to have a README.md file in directories with important information about its contents.
Reading and plotting data from the command line¶
Copy the iris.csv file
cp /data/iris.csv .
Now create plot2.py to have the following contents:
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import sys
fname = sys.argv[1]
print(f'reading {fname}')
df = pd.read_csv(fname)
sns.stripplot(x="species", y="sepal_width", data=df)
plt.savefig('plot2.png')
Now we can run this with: python plot2.py iris.csv
Getting files from remote computer with scp¶
The command scp is a sister to ssh and stands for "secure copy" and lets you download a file from a remote computer. The syntax is:
scp username:computer.example.com:filename.png .
So, for the file generated above, I can download it from my laptop with:
scp straw@python-course.strawlab.org:plot2.png .
On Mac, I can open a file from the command line with open plot2.png.