{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# onc library tutorial\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The _onc_ library is developed based on the Python _requests_ library, which is a popular library for making HTTP requests. In fact, you can use _requests_ directly to interact with the Oceans 3.0 API. But there are cases when you will find the _onc_ library useful (helper methods including boolean parse, one-click data product download, and ability to automatically download all pages). For more information you can check the [Oceans 3.0 API guide](https://oceannetworkscanada.github.io/Oceans3.0-API/API_Guide.html).\n",
"\n",
"This tutorial will demonstrate both approaches.\n",
"\n",
":::{Tip}\n",
"This is a Jupyter notebook. You can download the file [here](https://github.com/OceanNetworksCanada/api-python-client/blob/main/doc/source/Tutorial/onc_Library_Tutorial.ipynb).\n",
":::\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# Install some libraries\n",
"\n",
"# 1. onc: this is an onc library tutorial, right?\n",
"# 2. request: an alternative (vanilla) way to make HTTP requests to Oceans 3.0 API.\n",
"# 3. pandas: because it's useful and fun!\n",
"\n",
"import sys\n",
"\n",
"!{sys.executable} -m pip install --upgrade requests pandas onc"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To begin using the library, you need to create an instance from the `ONC` class, which is a wrapper class for Oceans 3.0 API requests.\n",
"All the library's functionality is provided as methods of this class.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# Setting up onc\n",
"from onc import ONC\n",
"\n",
"onc = ONC(\"43edcb6d-a3b0-44f0-9a57-ecac79ab3cc3\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Besides the token parameter, there are several other parameters you can modify to change the behavior of the `ONC` class. For example, you can use\n",
"\n",
"```python\n",
"onc = ONC(\"43edcb6d-a3b0-44f0-9a57-ecac79ab3cc3\", showInfo=True, outPath=\"YOUR_DIRECTORY\")\n",
"```\n",
"\n",
"to indicate that you want to see verbose messages after running each method, and you want the download directory to be \"./YOUR_DIRECTORY\" instead of the default \"./output\" when calling methods that involve downloading files like `orderDataProduct` and `downloadArchivefile`.\n",
"\n",
"For more information, check the API reference of the [ONC](https://oceannetworkscanada.github.io/api-python-client/autoapi/onc/index.html#onc.ONC) class.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Setting up requests\n",
"import requests\n",
"\n",
"token = onc.token"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Setting up pandas\n",
"import pandas as pd\n",
"\n",
"# For not overflowing the max-width of sphinx-rtd-theme. Can be safely ignored.\n",
"pd.set_option(\"display.max_colwidth\", 30)\n",
"pd.set_option(\"display.max_columns\", 5)\n",
"pd.set_option(\"display.max_rows\", 5)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## 1. Searching with discovery services\n",
"\n",
"To download data from Ocean Networks Canada's database (Oceans 3.0) , you need to specify the type of data you are interested in and where in particular (i.e. location, from specific instrument (device)) it originates from.\n",
"\n",
"In the Oceans 3.0 API, there are a unique codes that identify every location, device, property, data product type, etc. You include these codes in a group of filters to retrieve the information you're interested in.\n",
"\n",
"The Oceans 3.0 API **Discovery services** allow you to explore the hierarchy of ONC's database to obtain the codes required for your filters to obtain the information/data you are interested in (they work like a \"search\" function).\n",
"\n",
"The example below uses the _getLocations_ method, which is querying the locations database tables that include _\"Bullseye\"_ in their name (i.e. _\"Clayoquot Slope Bullseye Vent\"_).\n",
"It returns a list with all locations that match the search filters provided.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using ONC library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 1. Define your filter parameter\n",
"params = {\n",
" \"locationName\": \"Bullseye\",\n",
"}\n",
"\n",
"# 2. Call methods in the onc library\n",
"onc.getLocations(params)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using requests library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# 1. Define your filter parameter\n",
"params_requests = {\n",
" \"locationName\": \"Bullseye\",\n",
" \"token\": token,\n",
"}\n",
"\n",
"# 2. Define your base url for this query\n",
"url = \"http://data.oceannetworks.ca/api/locations\"\n",
"\n",
"# 3. Run your request\n",
"r = requests.get(url, params=params_requests)\n",
"\n",
"# 4. Parse the json file\n",
"r.json()"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Each entry of this list contains more meta data information for that location, e.g. the locationName, the geographical coordinates and depth, a description field and the URL for **Oceans 3.0 Data Search tool**.\n",
"The parameter **locationCode** contains the string \"NC89\", which is needed for the next steps.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### What device categories are available here at the location NC89?\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using ONC library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 1. Define your filter parameter\n",
"params = {\n",
" \"locationCode\": \"NC89\",\n",
"}\n",
"\n",
"# 2. Call methods in the onc library\n",
"result = onc.getDeviceCategories(params)\n",
"\n",
"# 3. Read it into a DataFrame\n",
"pd.DataFrame(result)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using requests library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 1. Define your filter parameter\n",
"params_requests = {\n",
" \"locationCode\": \"NC89\",\n",
" \"token\": token,\n",
"}\n",
"\n",
"# 2. Define your base url for this query\n",
"url = \"http://data.oceannetworks.ca/api/deviceCategories\"\n",
"\n",
"# 3. Run your request\n",
"r = requests.get(url, params=params_requests)\n",
"\n",
"# 4. Read it into a DataFrame\n",
"pd.DataFrame(r.json())"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### What properties are available for the device category CTD at this location NC89?\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using ONC library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 1. Define your filter parameter\n",
"params = {\n",
" \"locationCode\": \"NC89\",\n",
" \"deviceCategoryCode\": \"CTD\",\n",
"}\n",
"\n",
"# 2. Call methods in the onc library\n",
"r = onc.getProperties(params)\n",
"\n",
"# 3. Read it into a DataFrame\n",
"pd.DataFrame(r)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using requests library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 1. Define your filter parameter\n",
"params_requests = {\n",
" \"locationCode\": \"NC89\",\n",
" \"deviceCategoryCode\": \"CTD\",\n",
" \"token\": token,\n",
"}\n",
"\n",
"# 2. Define your base url for this query\n",
"url = \"http://data.oceannetworks.ca/api/properties\"\n",
"\n",
"# 3. Run your request\n",
"r = requests.get(url, params=params_requests)\n",
"\n",
"# 4. Read it into a DataFrame\n",
"pd.DataFrame(r.json())"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### What data product types are available for the device category CTD at this location NC89?\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using ONC library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 1. Define your filter parameter\n",
"params = {\n",
" \"locationCode\": \"NC89\",\n",
" \"deviceCategoryCode\": \"CTD\",\n",
"}\n",
"\n",
"# 2. Call methods in the onc library\n",
"r = onc.getDataProducts(params)\n",
"\n",
"# 3. Read it into a DataFrame\n",
"pd.DataFrame(r)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using requests library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 1. Define your filter parameter\n",
"params_requests = {\n",
" \"locationCode\": \"NC89\",\n",
" \"deviceCategoryCode\": \"CTD\",\n",
" \"token\": token,\n",
"}\n",
"\n",
"# 2. Define your base url for this query\n",
"url = \"http://data.oceannetworks.ca/api/dataProducts\"\n",
"\n",
"# 3. Run your request\n",
"r = requests.get(url, params=params_requests)\n",
"\n",
"# 4. Read it into a DataFrame\n",
"pd.DataFrame(r.json())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Downloading data\n",
"\n",
"Once you determine the exact filters that identify the information you are interested in, there are different methods available to download data.\n",
"\n",
"1. Near real-time scalar data sensor readings for a given timeframe\n",
"2. Near real-time raw data for a given timeframe\n",
"3. Download archived files containing raw data or processed data\n",
"4. Download data products that are also available via Oceans 3.0 Data Search Tool\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.1 Near real-time scalar data download\n",
"\n",
"In this example we want to download 10 seconds of **Pressure** sensor data from a **CTD** at location \"Bullseye\" (locationCode: **NC89**)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using ONC library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 1. Define your filter parameter\n",
"params = {\n",
" \"locationCode\": \"NC89\",\n",
" \"deviceCategoryCode\": \"CTD\",\n",
" \"dateFrom\": \"2017-01-20T00:00:00.000Z\",\n",
" \"propertyCode\": \"pressure\",\n",
" \"dateTo\": \"2017-01-20T00:00:10.000Z\",\n",
"}\n",
"\n",
"# 2. Call methods in the onc library\n",
"r = onc.getScalardata(params) # getScalardataByLocation(params) also works\n",
"\n",
"# 3. Return the dictionary keys (fields) of the query to get a sense what is contained in your returned message\n",
"print(r.keys())\n",
"\n",
"# 4. Read the data from parameter field \"sensorData\" it into a DataFrame - this is the data from your requested \"Pressure\" sensor\n",
"pressure = pd.DataFrame(r[\"sensorData\"][0][\"data\"])\n",
"pressure"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using requests library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 1. Define your filter parameter to obtain scalar data for 10 seconds\n",
"params_requests = {\n",
" \"locationCode\": \"NC89\",\n",
" \"deviceCategoryCode\": \"CTD\",\n",
" \"dateFrom\": \"2017-01-20T00:00:00.000Z\",\n",
" \"propertyCode\": \"pressure\",\n",
" \"dateTo\": \"2017-01-20T00:00:10.000Z\",\n",
" \"token\": token,\n",
"}\n",
"\n",
"# 2. Define your base url for this query\n",
"url = \"https://data.oceannetworks.ca/api/scalardata/location\"\n",
"\n",
"# 3. Run your request\n",
"r = requests.get(url, params_requests)\n",
"\n",
"# 4. Return the dictionary keys (fields) of the query to get a sense what is contained in your returned message. Note this is a JSON object for this method.\n",
"print(r.json().keys())\n",
"\n",
"# 5. Read the data from parameter field \"sensorData\" it into a DataFrame - this is the data from your requested \"Pressure\" sensor\n",
"pressure = pd.DataFrame(r.json()[\"sensorData\"][0][\"data\"])\n",
"pressure"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.2 Near real-time raw data readings\n",
"\n",
"In this example we want to download 10 seconds of raw data from a **CTD** at location \"Bullseye\" (locationCode: **NC89**)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using ONC library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 1. Define your filter parameter\n",
"params = {\n",
" \"locationCode\": \"NC89\",\n",
" \"deviceCategoryCode\": \"CTD\",\n",
" \"dateFrom\": \"2017-01-20T00:00:00.000Z\",\n",
" \"dateTo\": \"2017-01-20T00:00:10.000Z\",\n",
"}\n",
"\n",
"# 2. Call methods in the onc library\n",
"r = onc.getRawdata(params) # getRawdataByLocation(params) also works\n",
"\n",
"# 3. Return the dictionary keys (fields) of the query to get a sense what is contained in your returned message\n",
"print(r.keys())\n",
"\n",
"# 4. Read the content of parameter field data it into a DataFrame. The column \"readings\" contains the raw data and column \"times\" the Oceans 3.0 timestamps associated with each data reading.\n",
"pd.DataFrame(r[\"data\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using requests library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 1. Define your filter parameter\n",
"params_requests = {\n",
" \"locationCode\": \"NC89\",\n",
" \"deviceCategoryCode\": \"CTD\",\n",
" \"dateFrom\": \"2017-01-20T00:00:00.000Z\",\n",
" \"dateTo\": \"2017-01-20T00:00:10.000Z\",\n",
" \"token\": token,\n",
"}\n",
"\n",
"# 2. Define your base url for this query\n",
"url = \"http://data.oceannetworks.ca/api/rawdata/location\"\n",
"\n",
"# 3. Run your request\n",
"r = requests.get(url, params_requests)\n",
"\n",
"# 4. Read it into a DataFrame\n",
"pd.DataFrame(r.json()[\"data\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 2.2.1. Downloading more data\n",
"\n",
":::{Admonition} Pagination of response due to too many data rows\n",
":class: note\n",
"\n",
"If the row of the data is above 100,000, not all the data will be returned. The rest of the data can be queried based on the _next_ key in the response.\n",
"\n",
"1. If you use **onc**.\n",
"\n",
"`getRawdata` supports a boolean `allPages` parameter. When set to `True`, it will try to retrieve all the pages.\n",
"\n",
"2. If you use **requests**.\n",
"\n",
"You have to manually query the next pages until the `next` key in the response json is `None`, and concatenate all the data together.\n",
":::\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using ONC library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 1. Define your filter parameter with a longer date range (1 day of data)\n",
"params_longer_range = {\n",
" \"locationCode\": \"NC89\",\n",
" \"deviceCategoryCode\": \"CTD\",\n",
" \"dateFrom\": \"2017-01-20T00:00:00.000Z\",\n",
" \"dateTo\": \"2017-01-21T00:00:00.000Z\",\n",
"}\n",
"\n",
"# 2. Call methods in the onc library\n",
"r = onc.getRawdata(\n",
" params_longer_range, allPages=True\n",
") # getRawdataByLocation(params_longer_range, allPages=True) also works\n",
"\n",
"# 3. Read it into a DataFrame\n",
"pd.DataFrame(r[\"data\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using requests library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 1. Define your filter parameter with a longer date range\n",
"params_requests_longer_range = {\n",
" \"locationCode\": \"NC89\",\n",
" \"deviceCategoryCode\": \"CTD\",\n",
" \"dateFrom\": \"2017-01-20T00:00:00.000Z\",\n",
" \"dateTo\": \"2017-01-21T00:00:00.000Z\",\n",
" \"token\": token,\n",
"}\n",
"\n",
"# 2. Define your base url for this query\n",
"url = \"http://data.oceannetworks.ca/api/rawdata/location\"\n",
"\n",
"# 3. Run your request (the url is still the same)\n",
"r = requests.get(url, params_requests_longer_range)\n",
"\n",
"# 4. Read it into a DataFrame\n",
"pd.DataFrame(r.json()[\"data\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Now check the parameter field \"next\"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"r.json()[\"next\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### Update the **dateFrom** parameter to get the next page\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"params_requests_longer_range[\"dateFrom\"] = r.json()[\"next\"][\"parameters\"][\"dateFrom\"]\n",
"r = requests.get(url, params_requests_longer_range)\n",
"pd.DataFrame(r.json()[\"data\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Now check the parameter field \"next\"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(r.json()[\"next\"])"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### 2.3. Downloading archived files\n",
"\n",
"A faster way to download data products and processed data files that are available in Oceans 3.0 (if it suits your needs) is to leverage how ONC scripts auto-generate and archive data products of different types at set time intervals. You can directly download these data product files from our files archive, as long as you know their unique filename.\n",
"\n",
"In the following example, we get the list of archived files available for a camera (deviceCategoryCode: **VIDEOCAM**) at Ridley Island (locationCode:**RISS**) for 5-minute timerange.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using ONC library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 1. Define your filter parameter\n",
"params = {\n",
" \"locationCode\": \"RISS\",\n",
" \"deviceCategoryCode\": \"VIDEOCAM\",\n",
" \"dateFrom\": \"2016-12-01T00:00:00.000Z\",\n",
" \"dateTo\": \"2016-12-01T00:05:00.000Z\",\n",
"}\n",
"\n",
"# 2. Call methods in the onc library\n",
"r = onc.getArchivefile(params) # getArchivefileByLocation(params) also works\n",
"\n",
"# 3. This is the list of archived files:\n",
"r[\"files\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using requests library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# 1. Define your filter parameter\n",
"params_requests = {\n",
" \"locationCode\": \"RISS\",\n",
" \"deviceCategoryCode\": \"VIDEOCAM\",\n",
" \"dateFrom\": \"2016-12-01T00:00:00.000Z\",\n",
" \"dateTo\": \"2016-12-01T00:05:00.000Z\",\n",
" \"token\": token,\n",
"}\n",
"\n",
"# 2. Define your base url for this query\n",
"url_location = \"https://data.oceannetworks.ca/api/archivefile/location\"\n",
"\n",
"# 3. Run your request\n",
"r = requests.get(url_location, params_requests)\n",
"\n",
"# 4. This is the list of archived files:\n",
"r.json()[\"files\"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Once we have the file names, you can use the method `downloadArchivefile()` to download individual files:\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using ONC library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 1. Call methods in the onc library with the filename. The file is downloaded in the output folder by default,\n",
"# which can be customized in the outPath parameter of the ONC class.\n",
"onc.downloadArchivefile(\n",
" \"AXISQ6044PTZACCC8E334C53_20161201T000001.000Z.jpg\", overwrite=True\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using requests library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"# 1. Define your filter parameter with the filename\n",
"params = {\n",
" \"filename\": \"AXISQ6044PTZACCC8E334C53_20161201T000001.000Z.jpg\",\n",
" \"token\": token,\n",
"}\n",
"\n",
"# 2. Define your base url for this query\n",
"url_location = \"https://data.oceannetworks.ca/api/archivefile/download\"\n",
"\n",
"# 3. Run your request (the url is still the same)\n",
"r = requests.get(url_location, params)\n",
"\n",
"# 4. Save the file\n",
"# with open(params[\"filename\"], \"wb\") as f:\n",
"# f.write(r.content)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### 2.4 Downloading data products\n",
"\n",
"Other than using Oceans 3.0 [Data Search](https://data.oceannetworks.ca/DataSearch), we can request the ONC server to generate a **data product**. This is done through the data product delivery services methods.\n",
"\n",
":::{Hint}\n",
"This service should **ONLY** be used when the requested files are not already provided using the **ArchiveFiles** services (see **2.3** above). The data product delivery services will re-generate files using ONC's web machines and this process can often take very long time to generate these results. If you request data files for very long-time ranges and large file sizes, ONCs system will sometimes slow down and stall and requires some manual actions.\n",
"\n",
"We therefore encourage you to check other services before requesting data through this service. If you are unsure what to use feel free to contact u.\n",
":::\n",
"\n",
"**This process will require three steps before you will be able to see the downloaded data product on your computer:**\n",
"\n",
"1. **Request** the data.\n",
"2. **Run** the Request.\n",
"3. **Download** the data.\n",
"\n",
"The following example downloads two **PNG** files with plots for 30 minutes of data from a CTD (find them in the **\"output\"** folder beside this jupyter notebook). The filter includes codes for **location**, **deviceCategory**, and **dataProduct**, as well as the file **extension** and a time interval. They also include a couple of filters to configure this specific data product type (starting with the **\"dpo\\_\"** prefix) which can be obtained from the [Data Product Options documentation](https://wiki.oceannetworks.ca/display/O2A/Available+Data+Products). You can download more than 120 different types of data products including audio & video.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using ONC library\n",
"\n",
"The helper method `orderDataProduct` in onc library contains all three steps (methods) in one call. So this is the preferred library to use over the requests library.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 1. Define your filter parameter\n",
"params = {\n",
" \"locationCode\": \"NC89\",\n",
" \"deviceCategoryCode\": \"CTD\",\n",
" \"dataProductCode\": \"TSSP\",\n",
" \"extension\": \"png\",\n",
" \"dateFrom\": \"2017-01-19T00:00:00.000Z\",\n",
" \"dateTo\": \"2017-01-19T00:30:00.000Z\",\n",
" \"dpo_qualityControl\": \"1\",\n",
" \"dpo_resample\": \"none\",\n",
"}\n",
"\n",
"# 2. Call methods in the onc library\n",
"onc.orderDataProduct(params)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using requests library\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# 1. Request the data\n",
"\n",
"# Define your base url for this query\n",
"url_request = \"https://data.oceannetworks.ca/api/dataProductDelivery/request\"\n",
"\n",
"# Define your filter parameter\n",
"params_requests = {\n",
" \"locationCode\": \"NC89\",\n",
" \"deviceCategoryCode\": \"CTD\",\n",
" \"dataProductCode\": \"TSSP\",\n",
" \"extension\": \"png\",\n",
" \"dateFrom\": \"2017-01-19T00:00:00.000Z\",\n",
" \"dateTo\": \"2017-01-19T00:30:00.000Z\",\n",
" \"dpo_qualityControl\": \"1\",\n",
" \"dpo_resample\": \"none\",\n",
" \"token\": token,\n",
"}\n",
"\n",
"request = requests.get(url_request, params=params_requests)\n",
"request.json()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"##### requests continued #####\n",
"\n",
"# 2. Run the request\n",
"\n",
"# Note: you have to execute this cell multiple times until the return shows the \"status\": \"complete\"\n",
"# Note: Depending on your request, you can have more than one file ('fileCount').\n",
"# You will need to individually download these files by using the index parameter.\n",
"\n",
"url_run = \"https://data.oceannetworks.ca/api/dataProductDelivery/run\"\n",
"\n",
"requestID = request.json()[\"dpRequestId\"]\n",
"\n",
"params_requests = {\n",
" \"dpRequestId\": requestID,\n",
" \"token\": token,\n",
"}\n",
"\n",
"r = requests.get(url_run, params_requests)\n",
"r.json()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"##### requests continued #####\n",
"\n",
"# Find the RunID for the next step\n",
"RunId = r.json()[0][\"dpRunId\"]\n",
"RunId"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"##### requests continued #####\n",
"\n",
"# 3. Download the data\n",
"\n",
"url_download = \"https://data.oceannetworks.ca/api/dataProductDelivery/download\"\n",
"\n",
"params_requests = {\n",
" \"dpRunId\": RunId,\n",
" \"token\": token,\n",
" \"index\": \"1\",\n",
"}\n",
"\n",
"r = requests.get(url_download, params_requests)\n",
"r # Rerun this cell until the response code is 200.\n",
"\n",
"# r.headers[\"Content-Disposition\"] has the format \"attachement; filename=XXX.png\"\n",
"# with open(r.headers[\"Content-Disposition\"][22:], 'wb') as f:\n",
"# f.write(r.content)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
":::{admonition} Another option to get the data\n",
":class: tip\n",
"\n",
"Obtain your downloads from your user FTP directory (More -> User Directory) in Oceans 3.0.\n",
"Navigate to the folder that contains the runId: You will see the files in this folder.\n",
"\n",
"\n",
"\n",
":::\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.6"
}
},
"nbformat": 4,
"nbformat_minor": 2
}