Introduction to Swarm access through VirES¶
Authors: Ashley Smith
Abstract: VirES is a server/client architecture to help access time series of Swarm data and models. Access is enabled through a token generated on the website, and a Python package, viresclient, which provides the connection with the Python ecosystem (e.g. xarray).
# %load_ext watermark # %watermark -i -v -p viresclient,pandas,xarray,matplotlib
VirES (Virtual environments for Earth Scientists) is a platform for data & model access, analysis, and visualisation for ESA’s magnetic mission, Swarm
This tutorial introduces the Python interface to VirES,
viresclient. We demonstrate usage of the primary Swarm magnetic dataset (
SW_OPER_MAGA_LR_1B) and geomagnetic field models produced as part of the Swarm mission. Some knowledge of
matplotlib is assumed.
For more information see: - https://vires.services/ (Web interface) - https://viresclient.readthedocs.io (Python interface) - https://swarm-vre.readthedocs.io/ (Virtual Research Environment / JupyterLab) - https://earth.esa.int/web/guest/missions/esa-eo-missions/swarm/mission-overview
To access VirES through Python code, you first need to generate an access token from your VirES account:
- Create a user account at https://vires.services/ if you haven’t already done so
- Login there and go to settings (top right: the button with your user name)
- Select “Manage access tokens” and follow the instructions to create a token (see first part of video above)
- Copy the token into the box which will appear when you run the code below
- The default has now been configured and you won’t need to provide the token again.
This can also be done via the command line. For more info, see https://viresclient.readthedocs.io/en/latest/installation.html#first-usage-configuration
Enter your access token¶
Run this code and you will be prompted to enter the access token. The token and site url are stored in a configuration file at
~/.viresclient.ini. You may generate and set a new token, or revoke old tokens, at any point.
NB: You will need to repeat this procedure if you are running viresclient from different machines / user spaces.
from viresclient import set_token set_token("https://vires.services/ows", set_default=True) # (user is now prompted to enter the token)
Generate a token at https://vires.services/accounts/tokens/
Enter token: ································
Token saved for https://vires.services/ows
1. Fetching some data¶
SwarmRequest object which provides the VirES interface, and
datetime which gives convenient time objects which can be used by
from viresclient import SwarmRequest import datetime as dt
The following code will fetch one day (i.e. around 15 orbits) of the scalar (
F) measurements from Swarm Alpha.
Several keyword arguments can be provided to
.set_products() to specify the type of data you want. The
measurements available depend on the collection chosen in
.set_collection(). The same set of
auxiliaries are available for all collections - here we also choose to fetch the
MLT - magnetic local time.
sampling_step="PT10S" downsamples the data to 10 seconds, from the
MAGx_LR default of 1 second. If no
sampling_step is provided, the data will be accessed in its original form (i.e. here, 1-second sampling). These strings to choose the
sampling_step should be provided as ISO 8601 durations (e.g.
"PT1M" for 1-minute sampling).
.get_between() together provide the time window you want to fetch data for - executing this line causes the request to be processed on the server and the data returned to you. NB: this returns data up to but not including
end_time. Alternatively we can provide the start and end times as ISO_8601 strings.
# Set up connection with server request = SwarmRequest() # Set collection to use request.set_collection("SW_OPER_MAGA_LR_1B") # Set mix of products to fetch: # measurements (variables from the given collection) # models (magnetic model predictions at spacecraft sampling points) # auxiliaries (variables available with any collection) # Also set additional configuration such as: # sampling_step request.set_products( measurements=["F"], sampling_step="PT10S", auxiliaries=["MLT"] ) # Fetch data from a given time interval data = request.get_between( start_time=dt.datetime(2016,1,1), end_time=dt.datetime(2016,1,2) )
[1/1] Processing: 100%|██████████| [ Elapsed: 00:01, Remaining: 00:00 ] Downloading: 100%|██████████| [ Elapsed: 00:00, Remaining: 00:00 ] (0.428MB)
The data is now contained within the object which we called
<viresclient._data_handling.ReturnedData at 0x7fd88588e6d8>
The data is temporarily stored on disk and not yet loaded into memory - the
ReturnedData object is actually a wrapper around a temporary CDF file which could be written to disk directly:
Data written to test_file.cdf
… but it is possible to directly transfer it to a
df = data.as_dataframe() print(type(df)) df.head()
… or a
ds = data.as_xarray() print(type(ds)) ds
<xarray.Dataset> Dimensions: (Timestamp: 8640) Coordinates: * Timestamp (Timestamp) datetime64[ns] 2016-01-01 ... 2016-01-01T23:59:50 Data variables: Spacecraft (Timestamp) object 'A' 'A' 'A' 'A' 'A' ... 'A' 'A' 'A' 'A' 'A' Latitude (Timestamp) float64 -72.5 -73.13 -73.76 ... 29.82 30.46 31.1 MLT (Timestamp) float64 1.733 1.507 1.263 ... 17.16 17.16 17.16 Longitude (Timestamp) float64 92.79 93.09 93.41 ... -95.37 -95.37 -95.37 F (Timestamp) float64 4.694e+04 4.691e+04 ... 3.83e+04 3.861e+04 Radius (Timestamp) float64 6.834e+06 6.834e+06 ... 6.823e+06 6.823e+06 Attributes: Sources: ['SW_OPER_MAGA_LR_1B_20160101T000000_20160101T235959_050... MagneticModels:  RangeFilters: 
Try plotting some things to visualise the data. The following shows the variation in field strength measured by the satellite as it passes between high and low latitudes, varying from one orbit to the next as it samples a different longitude.
df.plot(y="F") df.plot(y="F", x="Latitude") df.plot(y="Latitude", x="Longitude") df.plot(y="Latitude", x="Longitude", c="F", kind="scatter");
2. Fetching model evaluations at the same time¶
Various (spherical harmonic) models of the geomagnetic field are produced as Swarm mission products and these are available through VirES. They are evaluated on demand at the same points and times as the data sample points. Here we ask for the
MCO_SHA_2D model, a dedicated core field model produced from Swarm data. By supplying
residuals=True we will get the data-model residuals, named in the dataframe as
request = SwarmRequest() request.set_collection("SW_OPER_MAGA_LR_1B") request.set_products( measurements=["F"], models=["MCO_SHA_2D"], residuals=True, sampling_step="PT10S" ) data = request.get_between( start_time=dt.datetime(2016,1,1), end_time=dt.datetime(2016,1,2) ) df = data.as_dataframe() df.head()
[1/1] Processing: 100%|██████████| [ Elapsed: 00:01, Remaining: 00:00 ] Downloading: 100%|██████████| [ Elapsed: 00:00, Remaining: 00:00 ] (0.359MB)
The core field has been removed from the data so the amplitudes are much smaller. Can you interpret the new signals in terms of external fields, i.e. from the ionosphere and magnetosphere?
df.plot(y="F_res_MCO_SHA_2D") df.plot(y="F_res_MCO_SHA_2D", x="Latitude") df.plot(y="Latitude", x="Longitude", c="F_res_MCO_SHA_2D", kind="scatter");
3. More complex model handling¶
We can also remove a magnetospheric field model at the same time, by specifying a new model (which we call
MCO_MMA here, but can be named whatever you like) which is the sum of core and magnetospheric models.
See smithara/viresclient_examples/model_details.ipynb for more examples of this - it is also possible to specify the spherical harmonic degrees (min/max) to use, and to provide your own
The remaining signal is now primarily due to the ionosphere.
Note that here I am instead using the CI (comprehensive inversion) core and magnetosphere models (
request = SwarmRequest() request.set_collection("SW_OPER_MAGA_LR_1B") request.set_products( measurements=["F"], models=["MCO_MMA = 'MCO_SHA_2C' + 'MMA_SHA_2C-Primary' + 'MMA_SHA_2C-Secondary'"], residuals=True, sampling_step="PT10S", auxiliaries=["MLT"] ) data = request.get_between( start_time=dt.datetime(2016,1,1), end_time=dt.datetime(2016,1,2) ) df = data.as_dataframe() df.head()
[1/1] Processing: 100%|██████████| [ Elapsed: 00:03, Remaining: 00:00 ] Downloading: 100%|██████████| [ Elapsed: 00:01, Remaining: 00:00 ] (0.429MB)
df.plot(y="F_res_MCO_MMA") df.plot(y="F_res_MCO_MMA", x="Latitude") df.plot(y="Latitude", x="Longitude", c="F_res_MCO_MMA", kind="scatter");