{ "cells": [ { "cell_type": "markdown", "id": "16473b52-64fd-425b-bab4-356708192bab", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "# Faulty Zeroes Filter\n", "## QC protocol for Private Weather Stations" ] }, { "cell_type": "markdown", "id": "957fd8e7-df3f-47d8-b57c-e60e26513338", "metadata": {}, "source": [ "This notebook presents how to use the 'Faulty Zeroes filter' in the Python package `pypwsqc`, a quality assurance protocol developed for automated private weather stations (PWS). \n", "\n", "The package is based on the original R code available at https://github.com/LottedeVos/PWSQC/.\n", "\n", "[Publication: de Vos, L. W., Leijnse, H., Overeem, A., & Uijlenhoet, R. (2019). Quality control for crowdsourced personal weather stations to enable operational rainfall monitoring. Geophysical Research Letters, 46(15), 8820-8829](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2019GL083731)\n", "\n", "`pypwsqc` depends on the `poligrain`, `xarray`, `pandas` and `numpy` packages. Make sure to install and import the required packages first." ] }, { "cell_type": "code", "execution_count": 1, "id": "78857b63-6c25-4391-be95-119a6e906aeb", "metadata": {}, "outputs": [], "source": [ "import poligrain as plg\n", "import xarray as xr\n", "\n", "import pypwsqc" ] }, { "cell_type": "markdown", "id": "d852ea47-40ab-4956-ac7c-f9aaa1dee996", "metadata": {}, "source": [ "## Download example data\n", "\n", "In this example, we use an open PWS dataset from Amsterdam, called the \"AMS PWS\" dataset. By running the cell below, an example NetCDF-file will be downloaded to your current repository (if your machine is connected to the internet)." ] }, { "cell_type": "code", "execution_count": 2, "id": "25b78fd5-c92b-4854-9a56-cefe8450f734", "metadata": {}, "outputs": [], "source": [ "!curl -OL https://github.com/OpenSenseAction/OS_data_format_conventions/raw/main/notebooks/data/OpenSense_PWS_example_format_data.nc" ] }, { "cell_type": "markdown", "id": "e420966c-eba1-4a40-aa4b-e1f10e7bbe26", "metadata": {}, "source": [ "## Data preparations" ] }, { "cell_type": "markdown", "id": "fa7460be-a65e-4549-831d-f11fa418a21c", "metadata": {}, "source": [ "This package handles rainfall data as `xarray` Datasets. The data set must have `time` and `id` dimensions, `latitude` and `longitude` as coordinates, and `rainfall` as data variable.\n", "\n", "An example of how to convert .csv data to a `xarray` dataset is found [here](https://github.com/OpenSenseAction/OS_data_format_conventions/blob/main/notebooks/PWS_example_dataset.ipynb).\n", "\n", "We now load the data set under the name `ds_pws`." ] }, { "cell_type": "code", "execution_count": 2, "id": "9a8f4054-4282-42a0-bfff-c12a55241672", "metadata": {}, "outputs": [], "source": [ "ds_pws = xr.open_dataset(\"OpenSense_PWS_example_format_data.nc\")" ] }, { "cell_type": "markdown", "id": "092bfba0-ecee-46cd-84ee-bea92a08ec8f", "metadata": {}, "source": [ "We then slice the time series to 1 month of data to shorten the computation period of the example." ] }, { "cell_type": "code", "execution_count": 3, "id": "b3851df5-c745-4ff6-85eb-b6648b2b4b52", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
<xarray.Dataset> Size: 9MB\n",
       "Dimensions:    (time: 8640, id: 134)\n",
       "Coordinates:\n",
       "  * time       (time) datetime64[ns] 69kB 2017-07-01 ... 2017-07-30T23:55:00\n",
       "  * id         (id) <U6 3kB 'ams1' 'ams2' 'ams3' ... 'ams132' 'ams133' 'ams134'\n",
       "    elevation  (id) <U3 2kB ...\n",
       "    latitude   (id) float64 1kB ...\n",
       "    longitude  (id) float64 1kB ...\n",
       "Data variables:\n",
       "    rainfall   (id, time) float64 9MB ...\n",
       "Attributes:\n",
       "    title:                 PWS data from Amsterdam\n",
       "    file author:           Maximilian Graf\n",
       "    institution:           Wageningen University and Research, Department of ...\n",
       "    date:                  2022-10-18 10:32:00\n",
       "    source:                Netamo PWS\n",
       "    history:               Data derived and reformated from the originally pu...\n",
       "    naming convention:     OpenSense-0.1\n",
       "    license restrictions:  CC-BY 4.0 https://creativecommons.org/licenses/by/...\n",
       "    reference:             https://doi.org/10.1029/2019GL083731\n",
       "    comment:
" ], "text/plain": [ " Size: 9MB\n", "Dimensions: (time: 8640, id: 134)\n", "Coordinates:\n", " * time (time) datetime64[ns] 69kB 2017-07-01 ... 2017-07-30T23:55:00\n", " * id (id)