Note

Notebook name: 01a__Intro-Jupyter-Python.ipynb (download .ipynb)
Alternative view with nbviewer - sometimes the formatting below can be messed up as it is processed by nbsphinx

Introduction to Jupyter & Python

Authors: Ashley Smith

Abstract: A brief introduction to using Jupyter and Python

Preface

This resource is intended to be a whirlwind tour of using Python within Jupyter notebooks. It covers the key knowledge that is expected within the rest of the notebooks in this repository - consider it as an ordered collection of pointers for further study to “level up” your skills.

Jupyter basics

A Jupyter notebook is a file (extension .ipynb) which can contain both (live) code and its outputs, as well as explanatory text. Content is organised into cells, where each cell may be either Code or *Markdown* (or others). A notebook can be rendered non-interactively in a web browser, converted to other formats (HTML, PDF …), or can be run interactively through a range of software. The standard software is JupyterLab which can be interacted with in a web browser, while the server backend (where the code itself runs) can be run either locally on your own machine, or on a remote machine as is the case with the *Virtual Research Environment (VRE)*. A notebook must be connected to a kernel which provides the software executable used to run code within the notebook (see the top right “Python 3” in JupyterLab). When accessing the VRE, your Python code and data associated with a notebook are stored and running on a remote machine - you only view the inputs and outputs from the code such that data transfer over the internet to your web browser is minimal. The main advantage is that there is zero setup required - all you need is a web browser.

When running interactively, notebook cells can be added, deleted, edited, and executed using buttons in JupyterLab and with keyboard shortcuts. Double click on this text to see the raw Markdown - re-render it by running the cell: use the “play” button at the top or use Ctrl-Enter (run) or Shift-Enter (run and advance to the next cell).

A notebook switches between two modes: command and edit. Enter edit mode by double clicking inside a cell, or by pressing Enter. Enter command mode by pressing Esc and then the notebook can be navigated with the arrow keys.

The following cell is set to “Code”. Run the cell - you should see that the output of the last line of the cell is displayed as the output of the cell. This is a convenient way to check the state of variables rather than having to use print() as below. To suppress this behaviour, end the line with;

Cells can be run in any order and the memory is persistent - see the incrementing counter [1], [2] on the left - so be careful when you run (or re-run) cells out of order!

[1]:
a = 1
b = 2
a + b
[1]:
3
[2]:
print(a, ",", b)
a, b
1 , 2
[2]:
(1, 2)

Markdown cells

These cells provide a way to document and describe your code. These can include rich text, equations, embedded images and video, and HTML. Consult a reference (or see Help / Markdown Reference in JupyterLab) for more details, but some things to get you started (again, double click this cell to see the markdown):

Use # Title, ## Subtitle ... to create headings to structure documents.

Use *..*, **..**, for italics and bold.

Use - ... at the beginning of successive lines to make an unordered list:

  • list item 1
  • list item 2

Use 1. ... to make an ordered list:

  1. list item 1
  2. list item 2

Use $...$ to insert mathematics using Latex to create them inline like \(\frac{dy}{dx}=\sin{\theta} + 5k\), or $$...$$ to create them in a centred equation style like

\[\frac{dy}{dx}=\sin{\theta} + 5k\]

Use --- to insert a horizontal line to break up sections


Use `...` to insert raw text like code: print(), or,

```python
# insert python code here
```

to render Python code with syntax highlighting like:

for i in range(5):
    print(i)

Jupyter tricks and shortcuts

As well as keyboard shortcuts to help interact with notebooks, there are many extra tricks that you will learn over time. Some of these are:

use ! to execute shell commands:

[3]:
!pwd      # print working directory
/home/jovyan/Swarm_notebooks
[4]:
!ls -l    # list files
total 9332
-rw-r--r-- 1 jovyan users  102804 Mar 12 14:33 01a__Intro-Jupyter-Python.ipynb
-rw-r--r-- 1 jovyan users  446851 Mar 12 14:33 01b1_Pandas-and-Plots.ipynb
-rw-r--r-- 1 jovyan users  540431 Mar 12 14:31 02a__Intro-Swarm-viresclient.ipynb
-rw-r--r-- 1 jovyan users   37787 Mar 12 14:33 02b__viresclient-Available-Data.ipynb
-rw-r--r-- 1 jovyan users  178725 Feb 28 16:53 02c__viresclient-API.ipynb
-rw-r--r-- 1 jovyan users   66643 Feb 28 17:06 02d__viresclient-Large-Data.ipynb
-rw-r--r-- 1 jovyan users   34638 Feb 28 16:45 02z1__Template-Basic.ipynb
-rw-r--r-- 1 jovyan users  395663 Mar 12 14:34 03a1_Demo-MAGx_LR_1B.ipynb
-rw-r--r-- 1 jovyan users  125653 Mar 12 14:34 03a2_Demo-MAGx_HR_1B.ipynb
-rw-r--r-- 1 jovyan users  164559 Mar 12 14:34 03b__Demo-EFIx_LP_1B.ipynb
-rw-r--r-- 1 jovyan users  594831 Mar 12 14:34 03c__Demo-IPDxIRR_2F.ipynb
-rw-r--r-- 1 jovyan users   83900 Mar 12 14:33 03d__Demo-TECxTMS_2F.ipynb
-rw-r--r-- 1 jovyan users  383922 Mar 12 14:34 03e1_Demo-FACxTMS_2F.ipynb
-rw-r--r-- 1 jovyan users 2305556 Feb 28 17:04 03e2_Demo-FAC_TMS_2F.ipynb
-rw-r--r-- 1 jovyan users  111731 Feb 28 17:02 03f__Demo-EEFxTMS_2F.ipynb
-rw-r--r-- 1 jovyan users   29026 Feb 28 17:03 03g__Demo-IBIxTMS_2F.ipynb
-rw-r--r-- 1 jovyan users  870588 Mar 12 14:31 04a1_Geomag-Models-VirES.ipynb
-rw-r--r-- 1 jovyan users 1194103 Feb 28 17:05 04b1_Geomag-Models-eoxmagmod.ipynb
-rw-r--r-- 1 jovyan users  744539 Feb 17 22:32 05a1_Polar-Region-Plots.ipynb
-rw-r--r-- 1 jovyan users  657666 Mar 12 14:33 LCS-1.shc
-rw-r--r-- 1 jovyan users    1067 Feb 16 13:38 LICENSE
-rw-r--r-- 1 jovyan users    3863 Feb 28 16:30 notebooks.json
-rw-r--r-- 1 jovyan users    1986 Mar 12 14:36 README.md
-rw-r--r-- 1 jovyan users  428072 Mar 12 14:31 test_file.cdf

use % to access `*IPython magics* <https://ipython.org/ipython-doc/3/interactive/magics.html>`__:

e.g. %time or %timeit to test execution time

[5]:
%timeit [x**2 for x in range(100)]
22 µs ± 375 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

use ? to get help:

[6]:
?print()
Object `print()` not found.

Also try using Tab to auto-complete and Shift-Tab to see quick help on the object you are accessing.

Python basics

Values are assigned to variables with =

[7]:
x = 5
y = 2

Operators perform operations between two given variables - the outcomes are obvious for arithmetic situations such as in this table, but behaviour of operators can be defined for other object types (as with lists and strings as below).

. Addition Subtraction Multiplication Division Exponent Modulo . Equal Not equal Less than and more…
Example: 5 + 2 5 - 2 5 * 2 5 / 2 5 ** 2 5 % 2 . 5 == 2 5 != 2 5 < 2
Output: 7 3 10 2.5 25 1 . False True False

We can use conditional operators in an “if statement” to control program flow, as below. Indentation / whitespace at the beginning of the line is used to define scope in the code, where other languages might use brackets or an endif. You should be able to find the bug in this code for some values of x and y, and fix it by adding additional elif cases before the final else

[8]:
x, y = 5, 2
if x > 2:
    print(x, ">", y)
else:
    print(x, "<", y)
5 > 2

Python data types

You should be familiar with some of the core Python data types: integer, floating point number, string, list, tuple, dictionary:

[9]:
a = 1
b = 1.0
c = "one"
d = [a, b, c]
e = (a, b, c)
f = {"a": a, "b": b, "c": c, "d": d, "e": e}
a, b, c, d, e, f
[9]:
(1,
 1.0,
 'one',
 [1, 1.0, 'one'],
 (1, 1.0, 'one'),
 {'a': 1, 'b': 1.0, 'c': 'one', 'd': [1, 1.0, 'one'], 'e': (1, 1.0, 'one')})

Use type() to see what the type of an object is:

[10]:
type(a), type(b), type(c), type(d), type(e), type(f)
[10]:
(int, float, str, list, tuple, dict)

Jupyter provides tools to help coding. Create a new code cell here, type f. then press Tab - you will see a list completion options. Extract the keys from the dictionary with f.keys(). Here we have used . to access one of the methods the object we assigned to f. The parentheses are necessary as this is a function call - arguments could be passed here.

[11]:
f.keys
[11]:
<function dict.keys>
[12]:
f.keys()
[12]:
dict_keys(['a', 'b', 'c', 'd', 'e'])

Let’s use ? to get help on the dictionary we created. This can be used to get a quick look into the documentation for any object.

[13]:
f?

This shows us that we can create such a dictionary using the dict() function - before we used the shortcut syntax with curly brackets, {}. Let’s create a new dictionary using dict() instead. Click inside the parentheses in dict(..) and press Shift-Tab to access the quick help. Follow the example text near the end of the quick help to create a dictionary.

[14]:
my_dict = dict()
my_dict
[14]:
{}

You should be able to extract values from the dictionary using the key names. Remember that dictionaries consist of (key, value) pairs. Use my_dict["key_name"] to extract a value, or if you want to supply a default value when they key hasn’t been found:

[15]:
my_dict.get("key_name", "empty!")
[15]:
'empty!'

Python objects

Everything in Python is an object. An object is a particular instance of a class, and has a set of associated behaviours that the class provides. These behaviours come in the form of properties (attributes that the object carries around, and involve no real computation time to access), and methods (functions that act on the object). These are accessed as object.property and object.method().

To demonstrate, here we create a complex number, then separately access the real and imaginary components of it which are stored as object properties. Finally we use .conjugate() to evaluate its complex conjugate - this returns a new complex number object that can be manipulated in the same way.

[16]:
z = 1 + 2j
print(type(z))
print(z)
print(z.real)
print(z.imag)
print(z.conjugate())
<class 'complex'>
(1+2j)
1.0
2.0
(1-2j)

This is the basis of object-oriented programming which is a popular programming paradigm that Python software takes advantage of.

Manipulating lists and strings

Lists and strings are closely related objects - a string can be thought of as a list of characters. Python comes with an in-built toolbox to help work with these objects (*string methods* and *list methods*). Here we use the string methods split() and join() to split a phrase up then rejoin the words differently. Afterwards we show that the replace() method already provides this functionality, replacing instances of a chosen character(s) (in this case, space) with another.

[17]:
s = "some words"
l = s.split()
"; ".join(l)
[17]:
'some; words'
[18]:
s.replace(" ", "; ")
[18]:
'some; words'

Let’s look at some list manipulation now using our new list, l. We can use append() to append a new item to the end (this is done in-place, updating the state of l, in contrast to the string methods above which return new strings/lists), then sort the resulting three words alphabetically.

[19]:
l
[19]:
['some', 'words']
[20]:
l.append("more")
l
[20]:
['some', 'words', 'more']
[21]:
l.sort()
l
[21]:
['more', 'some', 'words']

We can access items from a list using [] and the index number within the list, starting from zero. Colon, a:b, can be used to perform list slicing to return all values in the list from index a up to, but not including, b.

[22]:
l[0]
[22]:
'more'
[23]:
l[0:2]
[23]:
['more', 'some']

In addition to list methods, which are particular to list objects and so are accessed as list.method(), there are other more fundamental operations that are provided as built-in functions that you could try to apply to any object so are accessed as function(list). Below we evaluate the length of the list, and of the string which is the first item of the list.

[24]:
len(l), len(l[0])
[24]:
(3, 4)

Another way to add items to a list is to concatenate two lists together, which could be done using list1.extend(list2) as opposed to append() above (which just adds one item to a list). Let’s achieve the same thing by using the simpler + operator which for lists, simply joins them together:

[25]:
[10, 11, 12] + [5]
[25]:
[10, 11, 12, 5]

But what if we wanted to add the number to each item in the list (i.e. we are trying to do some arithmetic)? In this case we would have to loop through each item in the list and add 5 to each one.

Loops

For-loops are constructed with for a in b:, followed by an indented block as with if-statements. a is a variable which will change on each iteration of the loop, and b is an iterable - in a simple case like below this can just be a list, something containing items that can be iterated through. The length of b therefore determines the number of iterations the loop will make. We could create an index counter to iterate through and access items in a list (more normally, this would be achieved with for i in range(3)...) …

[26]:
my_list = [10, 11, 12]
for i in [0, 1, 2]:
    print(my_list[i])
10
11
12

but it is more Pythonic to iterate directly through the list items themselves:

[27]:
my_list = [10, 11, 12]
for i in my_list:
    print(i)
10
11
12

To create a new list with each item incremented by 5, we first create an empty list, then append the new number on each iteration of the loop:

[28]:
my_list = [10, 11, 12]
my_list2 = []
for i in my_list:
    my_list2.append(i + 5)
my_list2
[28]:
[15, 16, 17]

Alternatively, this can be done in-place on the original list if we iterate through the index:

[29]:
my_list = [10, 11, 12]
for i in range(len(my_list)):
    my_list[i] = my_list[i] + 5
my_list
[29]:
[15, 16, 17]

To be properly Pythonic, we should use a list comprehension which can implement looping behaviour while constructing a list directly:

[30]:
my_list = [10, 11, 12]
[i+5 for i in my_list]
[30]:
[15, 16, 17]

Functions

Functions can be defined using def function_name(arguments): ... followed by an indented block. Usually something should be returned by the function by putting return ... on the last line.

[31]:
def add_5(input_list):
    output_list = [i+5 for i in input_list]
    return output_list

add_5([10, 11, 12])
[31]:
[15, 16, 17]

Arguments (args) for the function can be provided separated by commas, followed by optional keyword arguments (kwargs) like x=<default value>. We should also document the function with a docstring which goes inside triple-quotes """ at the top of the function, and comments with #. Adding a docstring here makes the code self-documenting - we can now access this through the notebook help, which is revealed by Shift-Tab when typing print_things(...)

[32]:
def print_things(a, b, c=None):
    """Print 'a; b; c' or 'a; b'"""
    # A shortcut to do if-else logic
    output = [a, b, c] if c else [a, b]
    # Cast each of a, b, c to strings then join them
    print("; ".join([str(i) for i in output]))

print_things(1, 2, 3)
1; 2; 3

Extending functionality by importing from other packages

Obviously the above is an overly complex way to just add two sets of numbers together. This is because lists are not the appropriate object for this, and instead an array is neccessary. Array functionality can be imported from the package, **numpy** (numerical Python). Array objects behave like vectors and matrices in mathematics, and can be higher dimensional - hence the ndarray name, n-dimensional array.

[33]:
import numpy
[34]:
a = numpy.array([10, 11, 12])
print(type(a))
a
<class 'numpy.ndarray'>
[34]:
array([10, 11, 12])
[35]:
a + 5
[35]:
array([15, 16, 17])

It is customary to shorten the name used to refer to the imported package - this is done with import ... as ... - here we import numpy and rename it as np so that we can refer to anything within numpy as np.[...]. Another option is to just import only the part we want with from ... import ...

[36]:
import numpy as np
from numpy import array

l1 = np.array([1, 2, 3])
l2 = array([1, 2, 3])

np.array_equal(l1, l2)
[36]:
True

Arrays come with new properties and methods for mathematical operations, as well as many functions under the numpy namespace (what we can access with np.[...]). More advanced tools can be found in **scipy** (e.g. from scipy import fft) or elsewhere.

The standard tool for making figures is **matplotlib**. Numpy and matplotlib should be quite familiar to those coming from Matlab - you just need to get used to working from the numpy and matplotlib namespaces, and to start counting from zero instead of one!

The “pyplot” module of matplotlib can be used in a similar way to Matlab, with repeated calls to plt.[...] updating the state of the figure being created:

[37]:
import matplotlib.pyplot as plt
[38]:
x = np.arange(-2*np.pi, 2*np.pi, 0.1)
y = np.sin(x)
plt.plot(x, y)
plt.xlabel("x")
plt.ylabel("y")
plt.title("y = sin(x)");
../_images/Swarm_notebooks_01a__Intro-Jupyter-Python_70_0.png

However, for full control of figures, it is more flexible to use the object oriented approach as below. For more, refer to the matplotlib tutorials

[39]:
# Create a "figure" object that contains everything
fig = plt.figure()
# Add an "axes" object as part of the figure
ax = fig.add_subplot(111)
# NB: Instead of the above, it is more convenient to use:
#       fig, ax = plt.subplots(nrows=1, ncols=1)
# Add things to the axes
ax.plot(x, y)
ax.set_xlabel("x")
ax.set_ylabel("y")
ax.set_title("y = sin(x)");
../_images/Swarm_notebooks_01a__Intro-Jupyter-Python_72_0.png

The demonstrations above show that there are quite a few lines of code necessary to do the typical data analysis flow of loading/generating data, performing some computation, and plotting the results, in addition to having to pass around variables (like x and y above). This code may very often be quite similar so you might imagine standardising this process. It is for this reason that **pandas** has become popular, which provides the DataFrame object (inspired by the R language). This enables us to create a single tabular object to pass around (called df below) that contains our data (in this case numpy arrays), that has built-in plotting methods to rapidly inspect the data (creating matplotlib figures). Pandas accepts a wide range of input formats to load data, and contains computational tools that are particularly useful for time series, as well as connecting to **Dask** for doing larger computations (larger than memory, multi-process, distributed etc.).

[40]:
import pandas as pd
[41]:
df = pd.DataFrame({"x": np.arange(-2*np.pi, 2*np.pi, 0.1),
                   "y": np.sin(np.arange(-2*np.pi, 2*np.pi, 0.1))})
df.head()
[41]:
x y
0 -6.283185 2.449294e-16
1 -6.183185 9.983342e-02
2 -6.083185 1.986693e-01
3 -5.983185 2.955202e-01
4 -5.883185 3.894183e-01
[42]:
df.plot(x="x", y="y");
../_images/Swarm_notebooks_01a__Intro-Jupyter-Python_76_0.png

Moving beyond pandas, we come to **xarray**, which extends the pandas concepts to n-dimensional data.