Author Topic: Quantile Mapping for bias correction  (Read 6329 times)

Naveen Reddy

  • Team river
  • ***
  • Karma: +-1/-0
  • be the change that you want to see in the world
    • View Profile
  • Institute : National Institute of Technology Warangal
  • Programming language : C,Matlab,R
Quantile Mapping for bias correction
« on: June 22, 2016, 10:33:55 AM »
Dear members,

I am doing quantile mapping of climate projections into future using historical time data in 1970 to 2004 and future time slices 2010 to 2039, 2039 to 2070 and 2070 to 2099.

As the sample sizes should be equal, how should i proceed for 2039 to 2070 where there is a mismatch between months as well as an extra day because of leap year?

Can I ignore the data of that day and proceed further?

Another question is how to check if the bias correction has improved my result or not as the historical data is available from 1969 to 2004, which is not equal to 60 years?

with best wishes,
naveen
V Naveen Reddy Pothula

Sat Kumar Tomer

  • Administrator
  • Team sea
  • ***
  • Karma: +116/-0
    • View Profile
    • ambhas
  • Institute : IISc
  • Programming language : R, Python and Matlab
Re: Quantile Mapping for bias correction
« Reply #1 on: June 22, 2016, 11:23:59 AM »
Quote
I am doing quantile mapping of climate projections into future using historical time data in 1970 to 2004 and future time slices 2010 to 2039, 2039 to 2070 and 2070 to 2099.

As the sample sizes should be equal, how should i proceed for 2039 to 2070 where there is a mismatch between months as well as an extra day because of leap year?

If I understand your question correctly, there is no need of same sample size. Only for the historical period, data should be available form both the sources. If you are taking 1970 to 2004 as the historical period, you should have data from both the sources for this period. If not, you need to adjust your historical period to where both the data are available.

Quote
Another question is how to check if the bias correction has improved my result or not as the historical data is available from 1969 to 2004, which is not equal to 60 years?
Why 60 years? Any specific reason?

Naveen Reddy

  • Team river
  • ***
  • Karma: +-1/-0
  • be the change that you want to see in the world
    • View Profile
  • Institute : National Institute of Technology Warangal
  • Programming language : C,Matlab,R
Re: Quantile Mapping for bias correction
« Reply #2 on: June 22, 2016, 03:29:14 PM »
Dear Sat kumar Sir,

I have framed my question  in different ways.

When we are doing bias correction of future time slice, by quantile mapping,we assume that the future CDF  of GCM data is same as the  historical GCM CDF and then do the matching of it with CDF of observations to do correction.

But my question is if the Future GCM sample period is different from historical sample period.will this not have a bearing on bias correction as the sample size will alter the parameters of the CDF. In other ways can we do bias correction of GCM data for the entire period from 2010 to 2099 in a single stretch if we have historical data both for observations and GCM data for a period of 1970 to 1999 or we have to do it seperately for three slices 2010 to 2039, 2040 to 2070 and 2070 to 2099. In other ways my question is

1.If we have a SVM model fitted for regression with daily data for climate change impact on a variable from 1951 to 1980.The model parameters are dependent on sample size of predictors.Can the model be used to predict climate change for the entire stretch at a single go 2010 to 2099 ? or be used for predictions for three different slices of equal lengths as that of historical period?

2.If we calibrate a hydrologic model SWAT for 10 years  and validate it for 3 years.Should it be used for prediction for period 2010 to 2099 at a go .

In short I am not asking about the Stationarity assumption of climate change .I am asking for the practicle effect of sample size on predictions.Is there any thing like the period of prediction is less than the calibration period as in the case of independent validation for example: 70 % period for calibration-30 % period for validation.

Kindly excuse me, if my question is stupid.

with best wishes,
naveen
V Naveen Reddy Pothula

Sat Kumar Tomer

  • Administrator
  • Team sea
  • ***
  • Karma: +116/-0
    • View Profile
    • ambhas
  • Institute : IISc
  • Programming language : R, Python and Matlab
Re: Quantile Mapping for bias correction
« Reply #3 on: June 23, 2016, 10:30:06 AM »
Dear Naveen,

Quote
When we are doing bias correction of future time slice, by quantile mapping,we assume that the future CDF  of GCM data is same as the  historical GCM CDF and then do the matching of it with CDF of observations to do correction.

But my question is if the Future GCM sample period is different from historical sample period.will this not have a bearing on bias correction as the sample size will alter the parameters of the CDF. In other ways can we do bias correction of GCM data for the entire period from 2010 to 2099 in a single stretch if we have historical data both for observations and GCM data for a period of 1970 to 1999 or we have to do it seperately for three slices 2010 to 2039, 2040 to 2070 and 2070 to 2099.
Either way you will get the same results.


Quote
In other ways my question is

1.If we have a SVM model fitted for regression with daily data for climate change impact on a variable from 1951 to 1980.The model parameters are dependent on sample size of predictors.Can the model be used to predict climate change for the entire stretch at a single go 2010 to 2099 ? or be used for predictions for three different slices of equal lengths as that of historical period?
Yes. You can do it in a single go.

Quote
2.If we calibrate a hydrologic model SWAT for 10 years  and validate it for 3 years.Should it be used for prediction for period 2010 to 2099 at a go .
You can do it. However, if you can increase your calibration/validation period, it will be good. Ten years is too small period to check for trends in data. You might be knowing that there are some frequencies in rainfall of around 7 years also. If you can increase your calibration/validation period upto 30 years it will be better.

Quote
In short I am not asking about the Stationarity assumption of climate change .I am asking for the practicle effect of sample size on predictions.Is there any thing like the period of prediction is less than the calibration period as in the case of independent validation for example: 70 % period for calibration-30 % period for validation.
Usually assumption of stationarity is made in this sort of analysis. To relax this assumption you will have to find out how is CDF (mean, variance, skewness  and other moments) changing with time. To analyze this, you may require a longer time series. I have not done this sort of analysis. Let's see if someone working in this area comments on this.

Quote
Kindly excuse me, if my question is stupid.
No questions are stupid, if asked with clarity.

guillaume

  • Team tributary
  • *
  • Karma: +0/-0
    • View Profile
  • Institute : ESCER Center
  • Programming language : Matlab, R, Fortran, Pyton, IDL
Re: Quantile Mapping for bias correction
« Reply #4 on: January 26, 2017, 12:22:26 AM »
Dear Sat Kumar Tomer,

I read many discussions in the forum and I saw that you work with Python and you developped a script to make a bias correction using quantile mapping (i work with matlab and started to deal with python last week).

Here's my problem:
I have to make a bias correction on a RCM dataset using quantile mapping. I made a matlab script to do it but i'm wondering if with python could be faster? I'm using an ensemble of 60 simulations on 10km resolution over north america at daily scale...
I attached my code if it could help any Matlab users...It deals with Netcdf files.

Many thanks in advance.

Sat Kumar Tomer

  • Administrator
  • Team sea
  • ***
  • Karma: +116/-0
    • View Profile
    • ambhas
  • Institute : IISc
  • Programming language : R, Python and Matlab
Re: Quantile Mapping for bias correction
« Reply #5 on: January 26, 2017, 05:21:40 PM »
I am not sure if Python is much faster than the MATLAB.
Have you tried bias_correction function provided in the ambhas Python library (https://pypi.python.org/pypi/ambhas/0.4.0)?

guillaume

  • Team tributary
  • *
  • Karma: +0/-0
    • View Profile
  • Institute : ESCER Center
  • Programming language : Matlab, R, Fortran, Pyton, IDL
Re: Quantile Mapping for bias correction
« Reply #6 on: January 31, 2017, 01:15:35 AM »
Thanks for your message.
I asked my computer scientist to install your python library: "ambhas".

I have now the  __version__ = '0.1.0'.

But when I call the bias_correction function as: 
 from ambhas.errlib import bias_correction

I got this error message:
  File "<stdin>", line 1, in <module>
ImportError: cannot import name bias_correction

Do I have to download something else ? or is my ambhas version correct ?

Many thanks

Guillaume Dueymes

subash

  • Team river
  • ***
  • Karma: +16/-0
    • View Profile
  • Programming language : Python, R and Matlab
Re: Quantile Mapping for bias correction
« Reply #7 on: January 31, 2017, 06:52:31 PM »
@ Guillaume Dueymes. The bias_correction function is in stats file.

You should import the bias correction method as below:

Code: [Select]
from ambhas.stats import bias_correction

Subash
« Last Edit: February 01, 2017, 10:38:05 AM by subash »

guillaume

  • Team tributary
  • *
  • Karma: +0/-0
    • View Profile
  • Institute : ESCER Center
  • Programming language : Matlab, R, Fortran, Pyton, IDL
Re: Quantile Mapping for bias correction
« Reply #8 on: February 01, 2017, 08:56:32 PM »
Ho, thanks !
I'm new in python's world.


NARESHAADHI

  • Team tributary
  • *
  • Karma: +0/-0
    • View Profile
  • Institute : University College of Engineering(A), Osmania University
  • Programming language : MATLAB, Python
Re: Quantile Mapping for bias correction
« Reply #9 on: January 23, 2020, 09:30:56 AM »
Hi every one,

Can anyone share the python script of quantile
mapping for bias correction?
Aadhi Naresh
Ph.d Scholar
Dept. of Civil Engineering
Osmania University, Hyderabad

Sat Kumar Tomer

  • Administrator
  • Team sea
  • ***
  • Karma: +116/-0
    • View Profile
    • ambhas
  • Institute : IISc
  • Programming language : R, Python and Matlab
Re: Quantile Mapping for bias correction
« Reply #10 on: January 23, 2020, 02:06:07 PM »
You can try ambhas python library.

NARESHAADHI

  • Team tributary
  • *
  • Karma: +0/-0
    • View Profile
  • Institute : University College of Engineering(A), Osmania University
  • Programming language : MATLAB, Python
Re: Quantile Mapping for bias correction
« Reply #11 on: January 25, 2020, 09:45:26 AM »
Dear Sat kumar sir,

Thanks for the reply. Can we apply this quantile mapping for bias correction.
Aadhi Naresh
Ph.d Scholar
Dept. of Civil Engineering
Osmania University, Hyderabad

Sat Kumar Tomer

  • Administrator
  • Team sea
  • ***
  • Karma: +116/-0
    • View Profile
    • ambhas
  • Institute : IISc
  • Programming language : R, Python and Matlab
Re: Quantile Mapping for bias correction
« Reply #12 on: January 25, 2020, 11:32:47 AM »
Yes, there is a bias correction function in the library.

NARESHAADHI

  • Team tributary
  • *
  • Karma: +0/-0
    • View Profile
  • Institute : University College of Engineering(A), Osmania University
  • Programming language : MATLAB, Python
Re: Quantile Mapping for bias correction
« Reply #13 on: January 29, 2020, 08:18:05 AM »
Ok sir

Thank you
With regards
Aadhi Naresh
Aadhi Naresh
Ph.d Scholar
Dept. of Civil Engineering
Osmania University, Hyderabad