Indian Forum for Water Adroit

Writing two dimensional netCDF file in python-Reg

Diwan

  • ***
  • Thanked: 28 times
  • +12/-0
  • Be a learner or Teacher or helping to learn
    • View Profile
  • Institute : National Institute of Technology Karnataka, Surathkal
  • Programming language : Python, R, Matlab
Writing two dimensional netCDF file in python-Reg
« on: February 08, 2018, 12:45:42 AM »
Dear all

I tried to write a two dimensional nc file in python. I could able to generate the file but that spatial extent and resolution is not in proper. so it is not showing properly in ArcGIS.

Here is the code
Code: [Select]
from netCDF4 import *
from numpy import *
from openpyxl import *
import time

#Load the data sheet
wb = load_workbook('D:\\R_Workspace\\UB_Try.xlsx')
ws = wb['UB_details']

#To get the prec variable
ppt = []
for i in range(2,1696):
   ppt.append(ws.cell(row=i,column=3).value)

ppt_arr = asarray(ppt)

#Writing nc file
nc_file = Dataset('D:\\R_Workspace\\Test.nc','w',format='NETCDF4')

nc_file.description = 'Example dataset'
nc_file.history = 'Created on ' +time.ctime(time.time())

#Defining dimensions
lon = nc_file.createDimension('lon',57)
lat = nc_file.createDimension('lat',52)

#Creating variables
latitude = nc_file.createVariable('Latitude',float32,('lat',))
latitude.units = 'Degree_North'
longitude = nc_file.createVariable('Longitude',float32,('lon',))
longitude.units = 'Degree_East'

prec = nc_file.createVariable('prec',float32,('lon','lat'),fill_value = -9999.0)
prec.units = 'mm'

#Writing data to variables
lats = arange(16.875,19.425,0.05)
lat_reverse = lats[::-1]
lons = arange(73.325,76.175,0.05)
latitude[:] = lat_reverse
longitude[:] = lons
temp = zeros(52 * 57, dtype=float32)
temp[:ppt_arr.size] = ppt_arr
prec[:] = temp
nc_file.close() 

created nc file is attached
any suggestions pls.. Thank you..:)
« Last Edit: February 08, 2018, 09:25:05 AM by Diwan »
Research Scholar
Dept. of Applied Mechanics & Hydraulics
National Institute of Technology Karnataka
Surathkal Mangalore - 575025
 

subash

  • *****
  • Thanked: 32 times
  • +16/-0
    • View Profile
  • Programming language : Python, R and Matlab
Re: Writing two dimensional netCDF file in python-Reg
« Reply #1 on: February 08, 2018, 05:30:35 PM »
Hi Diwan

I think the problem is with the variableName used. The variables associated with the dimensions should have the same name as the dimensions. Change these lines

Code: [Select]
latitude = nc_file.createVariable('Latitude',float32,('lat',))
longitude = nc_file.createVariable('Longitude',float32,('lon',))
 


Modified :

Code: [Select]
#Creating variables
latitude = nc_file.createVariable('lat',float32,('lat',))
longitude = nc_file.createVariable('lon',float32,('lon',))
« Last Edit: February 08, 2018, 05:33:22 PM by subash »
 
The following users thanked this post: Diwan

Diwan

  • ***
  • Thanked: 28 times
  • +12/-0
  • Be a learner or Teacher or helping to learn
    • View Profile
  • Institute : National Institute of Technology Karnataka, Surathkal
  • Programming language : Python, R, Matlab
Re: Writing two dimensional netCDF file in python-Reg
« Reply #2 on: February 08, 2018, 09:41:44 PM »
Hi Subash

Thank you for spending your time.

It is also not working. variable name is not a problem. I am missing proper syntax to write a nc file it seems.

Even I tried for sub-setting and rewrite a existing nc file. But facing the same issue

You may have a look for this

Code: [Select]
from netCDF4 import *
from numpy import *
import time

#Read and subset
nc_file = Dataset('D:\\R_Workspace\\19971207.nc','r')

latbound = [16,20]
longbound = [73,77]

lats = nc_file.variables['Latitude'][:]
lons = nc_file.variables['Longitude'][:]

lat_lb = argmin(abs(lats-latbound[0]))
lat_ub = argmin(abs(lats-latbound[1]))

lon_lb = argmin(abs(lons-longbound[0]))
lon_ub = argmin(abs(lons-longbound[1]))

sm_sub = nc_file.variables['soil_moisture_x'][lon_lb:lon_ub,lat_ub:lat_lb]
lat_sub = nc_file.variables['Latitude'][lat_ub:lat_lb]
lon_sub = nc_file.variables['Longitude'][lon_lb:lon_ub]
nc_file.close()

#write to new file
my_file = Dataset('D:\\R_Workspace\\subset\\19971207.nc','w', format = 'NETCDF4')

my_file.discription = 'Test file'
my_file.history = 'Created on: ' +time.ctime(time.time())

#Dimensions
ldim = lat_lb-lat_ub # getting the no of element
lndim = lon_ub-lon_lb

lon = my_file.createDimension('lon',lndim)
lat = my_file.createDimension('lat',ldim)

#Variables
latitude = my_file.createVariable('Latitude',float32,('lat',))
latitude.units = 'Degree_north'
longitude = my_file.createVariable('Longitude',float32,('lon',))
longitude.units = 'Degree_east'

sm = my_file.createVariable('SM',float32,('lon','lat'),fill_value = -9999.0)
sm.units = 'percent'

#Load values to the variables
latitude[:] = lat_sub
longitude[:] = lon_sub
sm[:] = sm_sub
my_file.close()

something is wrong but I am not getting it. any suggestion pls.

Thank you

Research Scholar
Dept. of Applied Mechanics & Hydraulics
National Institute of Technology Karnataka
Surathkal Mangalore - 575025
 

subash

  • *****
  • Thanked: 32 times
  • +16/-0
    • View Profile
  • Programming language : Python, R and Matlab
Re: Writing two dimensional netCDF file in python-Reg
« Reply #3 on: February 08, 2018, 10:26:13 PM »
Hi Diwan,

I am dumping the gdalinfo to see what is wrong with the test.nc file u shared.


Code: [Select]
Warning 1: No UNIDATA NC_GLOBAL:Conventions attribute
Driver: netCDF/Network Common Data Format
Files: /home/subash/Downloads/test.nc
Size is 52, 57
Coordinate System is `'
Metadata:
  NC_GLOBAL#description=Example dataset
  NC_GLOBAL#history=Created on Thu Feb 08 09:22:33 2018
  prec#units=mm
  prec#_FillValue=-9999
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,   57.0)
Upper Right (   52.0,    0.0)
Lower Right (   52.0,   57.0)
Center      (   26.0,   28.5)
Band 1 Block=52x1 Type=Float32, ColorInterp=Undefined
  NoData Value=-9999
  Unit Type: mm
  Metadata:
    NETCDF_VARNAME=prec
    units=mm
    _FillValue=-9999

We can see from the dump that the file is missing the lat-long coordinates information.

So, to understand what could be likely error i created a similar  test case as below:

Code: [Select]
nc_file = Dataset('Test1.nc','w',format='NETCDF4')

nc_file.description = 'Example dataset'
nc_file.history = 'Created on ' +time.ctime(time.time())

#Defining dimensions
lon = nc_file.createDimension('lon',57)
lat = nc_file.createDimension('lat',52)

#Creating variables
latitude = nc_file.createVariable('lat','f4',('lat',))
latitude.units = 'Degree_North'
longitude = nc_file.createVariable('lon','f4',('lon',))
longitude.units = 'Degree_East'

prec = nc_file.createVariable('prec','f4',('lon','lat'),fill_value = -9999.0)
prec.units = 'mm'

[code]#Writing data to variables
lats = np.arange(16.875,19.425,0.05)
lat_reverse = lats[::-1]
lons = np.arange(73.325,76.175,0.05)
latitude[:] = lat_reverse
longitude[:] = lons

prec[:] = np.arange(0,prec.size,1)
nc_file.close()

U can see the gdal header info for the file Tes1.nc


Code: [Select]
abc@Lenovo-G50-80:~$ gdalinfo Test1.nc
Warning 1: No UNIDATA NC_GLOBAL:Conventions attribute
Driver: netCDF/Network Common Data Format
Files: /home/subash/Downloads/Test1.nc
Size is 52, 57
Coordinate System is `'
Origin = (19.449999229580747,76.150000027247842)
Pixel Size = (-0.049999985040403,-0.050000054495675)
Metadata:
  lat#units=Degree_North
  lon#units=Degree_East
  NC_GLOBAL#description=Example dataset
  NC_GLOBAL#history=Created on Thu Feb  8 22:21:15 2018
  prec#units=mm
  prec#_FillValue=-9999
Corner Coordinates:
Upper Left  (  19.4499992,  76.1500000)
Lower Left  (  19.4499992,  73.2999969)
Upper Right (  16.8500000,  76.1500000)
Lower Right (  16.8500000,  73.2999969)
Center      (  18.1499996,  74.7249985)
Band 1 Block=52x1 Type=Float32, ColorInterp=Undefined
  NoData Value=-9999
  Unit Type: mm
  Metadata:
    NETCDF_VARNAME=prec
    units=mm
    _FillValue=-9999

I had run the case on Ubuntu 14.04 LTS and  nc.__version__ =  '1.2.4'
« Last Edit: February 08, 2018, 10:44:15 PM by subash »
 

Diwan

  • ***
  • Thanked: 28 times
  • +12/-0
  • Be a learner or Teacher or helping to learn
    • View Profile
  • Institute : National Institute of Technology Karnataka, Surathkal
  • Programming language : Python, R, Matlab
Re: Writing two dimensional netCDF file in python-Reg
« Reply #4 on: February 09, 2018, 12:08:45 PM »
Thank you so much Subash for spending your valuable time..:)

subset is working now... some problem in c library for netcdf in my windows. it is working fine in ubuntu. you are right problem with the variable name also.. I should give same name as dimension.

regarding my earlier case (writing the data from excel sheet), I need to write the values for corresponding grids. I have 1694 grids in 57*52 domain with 0.05 degree resolution.

So I did like this

Code: [Select]
wb = load_workbook('D:\\R_Workspace\\UB_Try.xlsx')
ws = wb['UB_details']

#To get the prec variable
ppt = []
for i in range(2,1696):
   ppt.append(ws.cell(row=i,column=3).value)

ppt_arr = asarray(ppt) # the size is 1694
.
.
.
.
# to load the data

temp = zeros(52 * 57, dtype=float32) # Temporary array with dimension size
temp[:ppt_arr.size] = ppt_arr  #put the corresponding values (1694 size) to temp and remaining write as no data
prec[:] = temp

Is this the way? any thing wrong in that?  If I print the variable (prec) it is showing like one dimensional array not in 2D
Code: [Select]
[25.838  9.957 42.521 ...  0.     0.     0.   ]

but your test case the variable looks correct as two dimensional array. we can see
Code: [Select]
[[0.000e+00 1.000e+00 2.000e+00 ... 4.900e+01 5.000e+01 5.100e+01]
 [5.200e+01 5.300e+01 5.400e+01 ... 1.010e+02 1.020e+02 1.030e+02]
 [1.040e+02 1.050e+02 1.060e+02 ... 1.530e+02 1.540e+02 1.550e+02]
 ...
 [2.808e+03 2.809e+03 2.810e+03 ... 2.857e+03 2.858e+03 2.859e+03]
 [2.860e+03 2.861e+03 2.862e+03 ... 2.909e+03 2.910e+03 2.911e+03]
 [2.912e+03 2.913e+03 2.914e+03 ... 2.961e+03 2.962e+03 2.963e+03]]

how do I write that in case I am writing from some files?

Thank you
Research Scholar
Dept. of Applied Mechanics & Hydraulics
National Institute of Technology Karnataka
Surathkal Mangalore - 575025
 

subash

  • *****
  • Thanked: 32 times
  • +16/-0
    • View Profile
  • Programming language : Python, R and Matlab
Re: Writing two dimensional netCDF file in python-Reg
« Reply #5 on: February 09, 2018, 04:38:50 PM »
Hi Diwan,

No problem.

I am not sure what is the problem with data turning into 1D. Try to feed the data as 2D when ur writing to a .nc file. After u set check precp[:],if the data turns out to be 2-D we are good, otherwise need to check what is turning 2D to 1D.


HTH