Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Topics - Pankaj Dey

Pages: 1 [2] 3 4 ... 29
Physically based distributed hydrologic models require geospatial and time-series data that take considerable time and effort in processing them into model inputs. Tools that automate and speed up input processing facilitate the application of these models. In this study, we developed a set of web-based data services called HydroDS to provide hydrologic data processing ‘software as a service.’ HydroDS provides functions for processing watershed, terrain, canopy, climate, and soil data. The services are accessed through a Python client library that facilitates developing simple but effective data processing workflows with Python. Evaluations of HydroDS by setting up the Utah Energy Balance and TOPNET models for multiple headwater watersheds in the Colorado River basin show that HydroDS reduces the input preparation time compared to manual processing. It also removes the requirements for software installation and maintenance by the user, and the Python workflows enhance reproducibility of hydrologic data processing and tracking of provenance.


•Web-based data services developed for preparation of input data to selected distributed hydrologic models.
•Services are accessed through a Python client library and facilitate development of simple and effective Python workflow scripts.
•Services reduce time for hydrologic model input preparation, enhance reproducibility of hydrologic data processing, and enable tracking of data provenance.



Freely available and reliable meteorological datasets are highly demanded in many scientific and business applications. However, the structure of publicly available databases is often difficult to follow, especially for users who only deal with this kind of dataset on occasion. The “climate” R package aims to fill this gap with an easy-to-use interface for downloading global meteorological data in a fast and consistent way. The package provides access to different sources of in-situ meteorological data, including the Ogimet website, atmospheric vertical sounding gathered at the University of Wyoming’s webpage, and hydrological and meteorological measurements collected by the Institute of Meteorology and Water Management—National Research Institute (i.e., Polish Met Office). This article also provides a quick overview of the key functionalities available within the climate R package, and gives examples of an efficient and tidy workflow of meteorological data within the R based environment. The automation procedures included in the packages allow one to download data in a user-defined time resolution (from hourly to annual), for a user-defined time span, and for a specified group of stations or countries. The package also contains metadata, including a list of available stations, their geospatial information, and measurement descriptions with their units. Finally, the obtained datasets can be processed in R or exported to external tools (e.g., spreadsheets or GIS software).

The PraCTES workshop is a series of demos and hands-on tutorials for practical computing by earth scientists and for earth scientists. The goal of the workshop is to introduce practical computational tools and concepts so that earth scientists can spend more doing science and less time debugging data analysis code, processing large data sets, deciphering model source code, and other frustrating and time-consuming tasks of modern earth science research. We aim to make the workshop accessible and useful to scientists with all levels of programming proficiency and will cover topics ranging from introductory concepts in programming to state-of-the-art software tools for wrangling big data on the Cloud.

Each two-hour session will be highly interactive: instructors will swap between presenting background information on topics (e.g. What is an Earth System Model? How does github work behind the scenes?), demonstrating computing concepts with live demos and leading hands-on code tutorials and exercises, which attendees can follow along with in real-time on their personal laptops. Attendees should feel free to opt-in and opt-out at whichever point in the curriculum they feel is appropriate. Whenever possible, our demos and hands-on tutorials will be agnostic of programming language and earth science subfield, recognizing the many ways in which people engage with computation in earth science. To get the most out of the workshop, bring a laptop and follow along!

Runoff prediction in ungauged catchments is a significant hydrological challenge. The common approach is to calibrate hydrological models against streamflow data from gauged catchments, and then regionalise or transfer parameter values from the gauged calibration to predict runoff in the ungauged catchments. This paper explores the potential for using parameter values from hydrological models calibrated solely against readily available remotely sensed ET (RS‐ET) data to estimate runoff time series. The advantage of this approach is that it does not require observed streamflow data for model calibration and is therefore particularly useful for runoff prediction in poorly gauged or ungauged regions. The modelling experiments are carried out using data from 222 catchments across Australia. The results from the RS‐ET runoff‐free calibration are encouraging, particularly in simulating monthly runoff and mean annual runoff in the wetter catchments. However, results for daily runoff and in the drier regions are relatively poor, and further developments are needed to realise the benefit of direct model calibration against remotely sensed data to predict runoff in ungauged catchments.

Changes in the Earth's climate have been increasingly observed. Assessing the likelihood that each of these changes has been caused by human influence is important for decision making on mitigation and adaptation policy. Because of their large societal and economic impacts, extreme events have garnered much media attention—have they become more frequent and more intense, and if so, why? To answer such questions, extreme event attribution (EEA) tries to estimate extreme event likelihoods under different scenarios. Over the past decade, statistical methods and experimental designs based on numerical models have been developed, tested, and applied. In this article, we review the basic probability schemes, inference techniques, and statistical hypotheses used in EEA. To implement EEA analysis, the climate community relies on the use of large ensembles of climate model runs. We discuss, from a statistical perspective, how extreme value theory could help to deal with the different modeling uncertainties. In terms of interpretation, we stress that causal counterfactual theory offers an elegant framework that clarifies the design of event attributions. Finally, we pinpoint some remaining statistical challenges, including the choice of the appropriate spatio-temporal scales to enhance attribution power, the modeling of concomitant extreme events in a multivariate context, and the coupling of multi-ensemble and observational uncertainties.

Green fractional vegetation cover  is an important phenotypic factor in the fields of agriculture, forestry, and ecology. Spatially explicit monitoring of  via relative vegetation abundance (RA) algorithms, especially those based on scaled maximum/minimum vegetation index (VI) values, has been widely investigated in remote sensing research. Although many studies have explored the effectiveness of RA algorithms over the past 30 years, a literature review summarizing the corresponding theoretical background, issues, current state-of-the-art techniques, challenges, and prospects has not yet been published. The overall objective of the present study was to accomplish a comprehensive and systematic review of RA algorithms considering these factors based on the scientific papers published from January 1990 to November 2019. This review revealed that the key issues related to RA algorithms is the determination of the appropriate normalized difference vegetation index (NDVI) values of the full vegetation cover and bare soil (denoted hereafter by NDVI and NDVIs, respectively). The existing methods used to correct for these issues were investigated, and their advantages and disadvantages are discussed in depth. In literature trends, we found that the number of reported studies in which RA algorithms were used has increased consistently over time, and that most authors tend to utilize the linear NDVI model, rather than other models in the RA algorithm family. We also found that RA algorithms have been utilized to analyze the images with spatial resolutions ranging from the sub-meter to kilometer, most commonly, using images of 30-m spatial resolution. Finally, current challenges and forward-looking insights in remote estimation of  using RA algorithms are discussed to guide future research and directions.

In this paper we develop a conceptual and observational case in which soil water patterns in temperate regions of Australia switch between two preferred states. The wet state is dominated by lateral water movement through both surface and subsurface paths, with catchment terrain leading to organization of wet areas along drainage lines. We denote this as nonlocal control. The dry state is dominated by vertical fluxes, with soil properties and only local terrain (areas of high convergence) influencing spatial patterns. We denote this as local control. The switch is described in terms of the dominance of lateral over vertical water fluxes and vice versa. When evapotranspiration exceeds rainfall, the soil dries to the point where hydraulic conductivity is low and any rainfall that occurs essentially wets up the soil uniformly and is evapotranspired before any significant lateral redistribution takes place. As evapotranspiration decreases and/or rainfall increases, areas of high local convergence become wet, and runoff that is generated moves downslope, rapidly wetting up the drainage lines. In the wet to dry transitional period a rapid increase in potential evapotranspiration (and possibly a decrease in rainfall) causes drying of the soil and “shutting down” of lateral flow. Vertical fluxes dominate and the “dry” pattern is established. Three data sets from two catchments are presented to support the notion of preferred states in soil moisture, and the results of a modeling exercise on catchments from a range of climatic conditions illustrate that the conclusions from the field studies may apply to other areas. The implications for hydrological modeling are discussed in relation to methods for establishing antecedent moisture conditions for event models, for distribution models, and for spatially distributing bulk estimates of catchment soil moisture using indices.

Large-scale distributed watershed models are data-intensive, and preparing them consumes most of the research resources. We prepared high-resolution global databases of soil, landuse, actual evapotranspiration (AET), and historical and future weather databases that could serve as standard inputs in Soil and Water Assessment Tool (SWAT) models. The data include two global soil maps and their associated databases calculated with a large number of pedotransfer functions, two landuse maps and their correspondence with SWAT’s database, historical and future daily temperature and precipitation data from five IPCC models with four scenarios; and finally, global monthly AET data. Weather data are 0.5° global grids text-formatted for direct use in SWAT models. The AET data is formatted for use in SWAT-CUP (SWAT Calibration Uncertainty Procedures) for calibration of SWAT models. The use of these global databases for SWAT models can speed up the model building by 75–80% and are extremely valuable in areas with limited or no physical data. Furthermore, they can facilitate the comparison of model results in different parts of the world.

"Soil moisture at the headwater scale exhibits a huge spatial variability and single or even distributed TDR measurements yield non-representative data" (Zehe et al., 2010, p. 874 l. 11–14). This suggests that even a huge amount of observation points would not be able to capture soil moisture variability. Here we ask whether spatial variability is the dead-end to spatially distributed point sampling – or whether point networks yield representative data on dynamic changes nevertheless?

We present a measure to capture the spatial dissimilarity, or dispersion, and its change over time. Statistical dispersion among observation points is related to their distance to describe spatial patterns. We analyzed the temporal evolution and emergence of these patterns and use Mean shift clustering algorithm to identify and analyze clusters. We found that soil moisture observations from the Colpach catchment in Luxembourg to cluster in two fundamentally different states. On the one hand, we found rainfall-driven data clusters, usually characterized by strong relationships between dispersion and distance. Their spatial extent roughly matches the hillslope scale. On the other hand, we found clusters covering the vegetation period. In drying and then dry soil conditions there is no particular spatial dependence in soil moisture patterns, but the values are highly similar beyond hillslope scale.

By combining uncertainty propagation with information theory, we were able to calculate the information content of spatial similarity with respect to measurement uncertainty (when are patterns different outside of uncertainty margins?). We were able to prove that the spatial information contained in soil moisture observations is highly redundant and can be compressed to only a fragment of the original data volume without significant information loss.

Our most interesting finding is that even a few soil moisture time series bear a considerable amount of information about dynamic changes of soil moisture. We argue that distributed soil moisture sampling reflects an organized catchment state, where soil moisture variability is not random. Thus, only a small amount of observation points is necessary to capture soil moisture dynamics.

Satpy is a python library for reading, manipulating, and writing data from remote-sensing earth-observing meteorological satellite instruments. Satpy provides users with readers that convert geophysical parameters from various file formats to the common Xarray DataArray and Dataset classes for easier interoperability with other scientific python libraries. Satpy also provides interfaces for creating RGB (Red/Green/Blue) images and other composite types by combining data from multiple instrument bands or products. Various atmospheric corrections and visual enhancements are provided for improving the usefulness and quality of output images. Output data can be written to multiple output file formats such as PNG, GeoTIFF, and CF standard NetCDF files. Satpy also allows users to resample data to geographic projected grids (areas). Satpy is maintained by the open source Pytroll group.
The Satpy library acts as a high-level abstraction layer on top of other libraries maintained by the Pytroll group including:
  Go to the Satpy project page for source code and downloads.
Satpy is designed to be easily extendable to support any meteorological satellite by the creation of plugins (readers, compositors, writers, etc). The table at the bottom of this page shows the input formats supported by the base Satpy installation.

The hydrogen and oxygen stable isotope ratios of water have been used to identify sources, transport pathways, and phase-change processes within the water cycle, supporting hydrologic, forensic, ecologic, and hydroclimatic investigations. Here, we introduce a unique, open-access, global database of stable water isotope ratios (?18O, ?17O, and ?2H) from various water types. This database facilitates data preservation, supports standardized metadata collection, and decreases the time investment for meta-analytic research and reference dataset discovery. As of July 2019, the database includes 231?586 samples from 52?210 sites, associated with 218 projects, spanning 1949 through 2019. Key information stored includes the hydrogen and oxygen isotope ratios, water type, collection date and time, site location, and project information. To promote rapid data discovery and collaboration, the database exposes metadata such as data owner contact information of embargoed data, but only permits downloads of public data. The database is supported by two companion apps, one for processing and upload of analytical data from laboratories and the other an iOS application that supports the digital collection of sample metadata.

It has been demonstrated that the application of time-varying hydrological model parameters based on dynamic catchment behaviour significantly improves the accuracy and robustness of conventional models. However, the fundamental problems for calibrating dynamic parameters still need to be addressed. In this study, five calibration schemes for dynamic parameters in hydrological models were designed to investigate the underlying causes of poor model performance. The five schemes were assessed with respect to the model performance in different flow phases, the transferability of the dynamic parameters to different time periods, the state variables and fluxes time series, and the response of the dynamic parameter set to the dynamic catchment characteristics. Furthermore, the potential reasons for the poor response of the dynamic parameter set to the catchment dynamics were investigated. The results showed that the underlying causes of poor model performance included time-invariant parameters, compensation among parameters, high dimensionality, and abrupt shifts in the parameters. The recommended calibration scheme exhibited good performance and overcame these problems by characterizing the dynamic behaviour of the catchments. The main reason for the poor response of the dynamic parameter set to the catchment dynamics may be the poor convergence performance of the parameters. In addition, the assessment results of the state variables and fluxes, and the convergence performance of the parameters provided robust indications of the dominant response modes of the hydrological models in different sub-periods or catchments with distinguishing catchment characteristics.


Hydrological sciences / How to make advances in hydrological modelling
« on: October 26, 2019, 10:17:27 PM »
After some background about what I have learned from a career in hydrological modelling, I present some opinions about how we might make progress in improving hydrological models in future including how to decide whether a model is fi t for purpose; how to improve process representations in hydrological models; and how to take advantage of Models of Everywhere. Underlying all those issues, however, is the fundamental problem of improving the hydrological data available for both forcing and evaluating hydrological models. It would be a major advance if the hydrological community could come together to prioritise and commission the new observational methods that are required to make real progress.

Hydrological sciences / Are changes in small and big floods different?
« on: October 18, 2019, 06:04:50 PM »
2. To estimate regional flood trends, we use a non-stationary regional flood frequency approach consisting of a regional Gumbel distribution, whose median and growth factor can vary in time with different strengths for different catchment sizes. A Bayesian Monte Carlo Markov Chain (MCMC) approach is used for parameter estimation. We quantify regional trends (and the related sample uncertainties), for floods of selected return periods and for selected catchment areas, across Europe and for three regions where coherent flood trends have been identified in previous studies. Results show that, in the Atlantic region, the trends in flood magnitude are generally positive. In small catchments (up to 100?km2), the 100-year flood increases more than the median flood, while the opposite is observed in medium and large catchments, where even some negative trends appear, especially over the southern part of the Atlantic region. In the Mediterranean region flood trends are generally negative. The 100-year flood decreases less than the median flood and, in the small catchments, the median flood decreases less compared to the large catchments. Over Eastern Europe the regional trends are negative and do not depend on the return period, but catchment area plays a substantial role: the larger the catchment, the more negative the trend. The process causalities on the effects of return period and catchment area on the flood trends are discussed.

Pages: 1 [2] 3 4 ... 29