Note

Notebook name: 01b1_Pandas-and-Plots.ipynb (download .ipynb)
Alternative view with nbviewer - sometimes the formatting below can be messed up as it is processed by nbsphinx

Intro to Pandas and Plotting

Authors: Ashley Smith

Abstract: This is a short tutorial to motivate usage of Pandas and demonstrates some basic functionality for working with time series. Previous knowledge of Numpy and Matplotlib is assumed. See also: https://pandas.pydata.org/docs/getting_started/10min.html We show various plots, including basic time series statistics and error intervals.

[1]:
%load_ext watermark
%watermark -i -v -p pandas,matplotlib
2020-03-12T14:37:09+00:00

CPython 3.7.6
IPython 7.11.1

pandas 0.25.3
matplotlib 3.1.2
[2]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

Organising data

Suppose we run an experiment with two variables, x and y. We could store these data as two numpy arrays assigned to variables x and y:

[3]:
# Generate a sinusoidal signal with random noise
x = np.linspace(-2*np.pi, 2*np.pi, 744)
y = np.sin(x) + 0.5*np.random.rand(len(x))
print(x[:3], "...")
print(y[:3], "...")
[-6.28318531 -6.26627229 -6.24935928] ...
[0.18381547 0.3904922  0.05996459] ...

To manipulate our data (e.g plotting), we must pass both of these variables around

[4]:
def plot_data_1(x, y):
    fig, ax = plt.subplots()
    ax.plot(x, y)
    ax.set_xlabel("x")
    ax.set_ylabel("y")
    return fig, ax

plot_data_1(x,y);
../_images/Swarm_notebooks_01b1_Pandas-and-Plots_6_0.png

This is manageable as there are only two variables, but if we were to add more variables to our collection (e.g. other concurrent measurements), or if we wanted to store extra information about each variable, this would rapidly become annoying to handle.

To organise the variables better, we could assign them to a dictionary, d. Now we only have to pass one object to our plotting function. We also use the key names in the dictionary to describe our data (in this case, just x and y).

[5]:
d = {"x": x, "y": y}
print("x:", d["x"][:3], "...")
print("y:", d["y"][:3], "...")
x: [-6.28318531 -6.26627229 -6.24935928] ...
y: [0.18381547 0.3904922  0.05996459] ...
[6]:
def plot_data_2(d, xvar="x", yvar="y"):
    fig, ax = plt.subplots()
    ax.plot(d[xvar], d[yvar])
    ax.set_xlabel(xvar)
    ax.set_ylabel(yvar)
    return fig, ax

plot_data_2(d);
../_images/Swarm_notebooks_01b1_Pandas-and-Plots_9_0.png

We could go on like this, adding new measurements to this dictionary, and defining different datasets (including their various measurements) as different dictionaries. But there is a better way.

Working with dataframes

The pandas.DataFrame organises tabular data and provides convenient tools for computation and visualisation. Dataframes act much like a spreadsheet (or a SQL database) and are inspired partly by the R programming language. They consist of columns (here, we named them x and y), and rows. Each column should contain the same number of elements, and each row refers to some related measurements. Dataframes also have an index which identifies the rows - in our case they have just been labelled as integers that match the indexes in the original input arrays.

Pandas comes with many I/O tools to load dataframes. We can also create one from a dictionary:

[7]:
df = pd.DataFrame.from_dict(d)
df
[7]:
x y
0 -6.283185 0.183815
1 -6.266272 0.390492
2 -6.249359 0.059965
3 -6.232446 0.082151
4 -6.215533 0.368317
... ... ...
739 6.215533 0.369342
740 6.232446 0.411543
741 6.249359 0.081776
742 6.266272 0.008885
743 6.283185 0.029586

744 rows × 2 columns

There is an underlying Numpy array that can be accessed through the .values property:

[8]:
df.values[:5, :]
[8]:
array([[-6.28318531,  0.18381547],
       [-6.26627229,  0.3904922 ],
       [-6.24935928,  0.05996459],
       [-6.23244626,  0.08215086],
       [-6.21553324,  0.36831692]])

We can still extract the separate arrays (x and y) similar to interacting with a dictionary:

[9]:
type(df["x"].values)
[9]:
numpy.ndarray

We can append new data to the dataframe just like appending a variable to a dictionary. The new data should be the same length as the dataframe, but if a constant is supplied then that is used for every row:

[10]:
df["y_error"] = 0.3

Some of the real advantages of using Pandas come when we employ a more useful index. If our measurements are taken at different times, we can set the index as a time-aware object. This uses the Pandas DatetimeIndex which is related to the datetime standard library

[11]:
# Generate some sample times at hourly intervals over a month
df["time"] = pd.date_range("2020-01-01", "2020-02-01", periods=745, closed="left")
df = df.set_index("time")
df
[11]:
x y y_error
time
2020-01-01 00:00:00 -6.283185 0.183815 0.3
2020-01-01 01:00:00 -6.266272 0.390492 0.3
2020-01-01 02:00:00 -6.249359 0.059965 0.3
2020-01-01 03:00:00 -6.232446 0.082151 0.3
2020-01-01 04:00:00 -6.215533 0.368317 0.3
... ... ... ...
2020-01-31 19:00:00 6.215533 0.369342 0.3
2020-01-31 20:00:00 6.232446 0.411543 0.3
2020-01-31 21:00:00 6.249359 0.081776 0.3
2020-01-31 22:00:00 6.266272 0.008885 0.3
2020-01-31 23:00:00 6.283185 0.029586 0.3

744 rows × 3 columns

[12]:
type(df.index)
[12]:
pandas.core.indexes.datetimes.DatetimeIndex

It is reasonable to use dataframes to contain your data, use it for easily reading and writing to files, and to apply Numpy-based transformations and other computation, together with plotting routines using Matplotlib. However there are also many ways to use Pandas for manipulating data which are not covered here.

Below we show a few basic ways to plot time series.

Plotting with dataframes

We can use the .plot() method to access the Pandas plotting API which itself creates Matplotlib objects. This mechanism is rather complex but enables many convenient shortcuts to creating complex figures.

[13]:
df.plot()
[13]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fdfffc97b90>
../_images/Swarm_notebooks_01b1_Pandas-and-Plots_23_1.png

It takes some time to get familiar with this API but after that it becomes very useful for rapid feedback and iteration while playing with data, particularly in combination with Jupyter notebooks.

[14]:
df.plot(y="y")
[14]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fdffeb705d0>
../_images/Swarm_notebooks_01b1_Pandas-and-Plots_25_1.png

Let’s try some things to better visualise this time series.

We can use the resampling system to change the data from hourly samples to daily samples based on the mean of the measurements taken each day. We can directly feed the derived dataframe into a plotting command.

[15]:
df.resample("1d")
[15]:
<pandas.core.resample.DatetimeIndexResampler object at 0x7fdffeb70910>
[16]:
df.resample("1d").mean().plot(y="y")
[16]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fdfffcb4410>
../_images/Swarm_notebooks_01b1_Pandas-and-Plots_28_1.png

A related method is rolling calculations:

[17]:
df.rolling(24).mean().plot(y="y")
[17]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fdffe9d9050>
../_images/Swarm_notebooks_01b1_Pandas-and-Plots_30_1.png

For more, see e.g. https://ourcodingclub.github.io/2019/01/07/pandas-time-series.html

Rather than just creating the plot straight away as above, we can instead instantiate a Matplotlib axes, and then direct Pandas to plot onto it. This enables some more flexible configuration, like plotting aspects from two dataframes onto one axes:

[18]:
fig, ax = plt.subplots()
df.plot(y="y", ax=ax)
df.rolling(24, center=True).mean().plot(y="y", ax=ax, label="y-smoothed")
[18]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fdffe8fbc10>
../_images/Swarm_notebooks_01b1_Pandas-and-Plots_32_1.png

As an example of the more configurable plotting options through .plot(), let’s show the error bars on measurements. To make them visible, we first subselect down to only every 24th measurement using .iloc.

[19]:
df.iloc[::24].plot(x="x", y="y", kind="scatter", yerr="y_error")
[19]:
<matplotlib.axes._subplots.AxesSubplot at 0x7fdffe7e3b90>
../_images/Swarm_notebooks_01b1_Pandas-and-Plots_34_1.png

Beware! If we had resampled the dataframe like in the previous steps, the resampling logic would also have applied to the y_error column. You would instead need to supply the correct error propagation mechanism yourself.

Note that we had to resort to plotting against the “x” data instead of the index (time) because of a limitation in what the “scatter” option can do. Plotting against the index is not supported - this could be worked around by creating an extra column to use: df["time"] = df.index. Another option is to create the plot ourselves using the Matplotlib API:

[20]:
_df = df.iloc[::24]
fig, ax = plt.subplots(figsize=(12, 3))
ax.errorbar(_df.index, _df["y"], yerr=_df["y_error"], marker="o")
ax.set_ylabel("y")
ax.set_xlabel("time");
../_images/Swarm_notebooks_01b1_Pandas-and-Plots_36_0.png

Above, we subsampled the data in order to create a clear visualisation with vertical error bars. To show the error intervals on the original data it is better to shade the area with fill_between:

[21]:
fig, ax = plt.subplots(figsize=(12, 3))
x = df.index
y = df["y"]
y1 = y - df["y_error"]
y2 = y + df["y_error"]
ax.plot(x, y)
ax.fill_between(x, y1, y2, color="grey")
ax.set_ylabel("y")
ax.set_xlabel("time");
../_images/Swarm_notebooks_01b1_Pandas-and-Plots_38_0.png

Rather than using the provided error values, we might choose to plot the spread in the data itself. In the example below we create a new df_daily dataframe that contains the daily means and standard deviations of measurements. We use these to plot the mean and spread in the measurements.

[22]:
def plot_daily_means(df, ax):
    df_daily = (
        df.resample("1d").mean()
        .drop(columns=["x", "y_error"])
        .rename(columns={"y": "y_mean"}))
    df_daily["y_std"] = df["y"].resample("1d").std()

    ax.plot(df_daily.index, df_daily["y_mean"])
    ax.fill_between(
        df_daily.index,
        df_daily["y_mean"] - df_daily["y_std"],
        df_daily["y_mean"] + df_daily["y_std"],
        color="lightgrey")
    ax.set_ylabel("y");

fig, ax = plt.subplots(figsize=(12, 3))
plot_daily_means(df, ax)
../_images/Swarm_notebooks_01b1_Pandas-and-Plots_40_0.png

When we create a figure like this, it is useful to define the plotting routine as a function that applies to a Matplotlib Axes object. This way we can control the figure setup (geometry, other plots etc.) separately from the detailed plotting commands. The figure can be manipulated more cleanly in this way. Other subplots can be easily combined into one figure, or other configurations applied: for example, we might later add on grid lines without modifying the original plotting code:

[23]:
ax.grid(True)
fig
[23]:
../_images/Swarm_notebooks_01b1_Pandas-and-Plots_42_0.png

To xarray

There are some limitations with pandas.DataFrame that make it not so suitable for the physical sciences. **Xarray** fills some of these gaps and is mostly compatible with Pandas, providing a similar API. We can transform a pandas.DataFrame into a xarray.Dataset with .to_xarray():

[24]:
ds = df.to_xarray()
ds
[24]:
<xarray.Dataset>
Dimensions:  (time: 744)
Coordinates:
  * time     (time) datetime64[ns] 2020-01-01 ... 2020-01-31T23:00:00
Data variables:
    x        (time) float64 -6.283 -6.266 -6.249 -6.232 ... 6.249 6.266 6.283
    y        (time) float64 0.1838 0.3905 0.05996 ... 0.08178 0.008885 0.02959
    y_error  (time) float64 0.3 0.3 0.3 0.3 0.3 0.3 ... 0.3 0.3 0.3 0.3 0.3 0.3

Similar quick plotting can be peformed, but the mechanism is different due to the greater complexity of the data structure.

[25]:
ds["y"].plot.line()
[25]:
[<matplotlib.lines.Line2D at 0x7fdffc0ee310>]
../_images/Swarm_notebooks_01b1_Pandas-and-Plots_46_1.png

The primary advantage of xarray is that it extends Pandas-like functionality to n-dimensional data. In Pandas, each column is limited to contain a 1-dimensional array (though this can be worked around by using a MultiIndex). In xarray, each “data variable” (itself an xarray.DataArray) can hold an n-dimensional array, with each dimension carrying a dimension name. To provide label-based access, dimensions can have associated coordinates. In our example, the data variables (x, y, y_error) have the time dimension which has datetime-based coordinates.

We might add more complex data, v, which has a spatial component as well. We need to provide dimension names in order to do this (see also: xarray.Dataset.assign)

[26]:
v = np.random.rand(len(ds["time"]), 3)
ds["v"] = (("time", "space"), v)
ds
[26]:
<xarray.Dataset>
Dimensions:  (space: 3, time: 744)
Coordinates:
  * time     (time) datetime64[ns] 2020-01-01 ... 2020-01-31T23:00:00
Dimensions without coordinates: space
Data variables:
    x        (time) float64 -6.283 -6.266 -6.249 -6.232 ... 6.249 6.266 6.283
    y        (time) float64 0.1838 0.3905 0.05996 ... 0.08178 0.008885 0.02959
    y_error  (time) float64 0.3 0.3 0.3 0.3 0.3 0.3 ... 0.3 0.3 0.3 0.3 0.3 0.3
    v        (time, space) float64 0.6593 0.698 0.3692 ... 0.7362 0.8055 0.2324

Another advantage of xarray is support for metadata. For example, we can add units and a description by changing the .attrs (attributes) property of the DataArray:

[27]:
ds["v"].attrs = {"units": "m/s", "description": "A velocity vector"}

Plotting commands can automatically handle the multi-dimensional aspect, as well as adding the provided units to the axis labels.

[28]:
ds["v"].plot.line(x="time");
../_images/Swarm_notebooks_01b1_Pandas-and-Plots_52_0.png

To-do: tutorial on indexing and other aspects - see http://xarray.pydata.org/en/stable/indexing.html