In one of my previous blogs, I converted ERA 5 datasets downloaded in NetCDF format to csv in python.
To be honest with you all, I was not much satisfied by the code as it was taking a long time to perform the conversion, even for small data. So, I was constantly in a search for a quicker and efficient way to quickly perform the conversion even for bulk datasets.
Finally, by using R, we can now successfully, very efficiently convert netCDF data to data frames.
Before I begin, I would like to thank Prof. Patrick J. Bartlein, Dept. Geography, Univ. Oregon for his helpful links and prompt response to any query and a big thanks to my dearest friend and senior Alaa Mahwish (NASA) for being very generous and motivating, and always available to discuss and troubleshoot the bugs.
Let's start :
So in the present analysis, we will be using ALBEDO data (Forecast albedo) sourced from ERA5.
Let's take data for a small domain for our learning purpose
Albedo file we will be using in this tutorial is available in the link below :
First, let's have a quick look at this data in panoply :
So we have 'fal' (Forecast albedo); it has three dimensions and their shapes (time(72), latitude (5), longitude (5 ).
Let's quickly look at the data array in panoply itself. For this tutorial purpose, I have taken small data into consideration
So longitude range is 83.5 - 84.5 E and latitude range is 28.0-29.0N
TIme coordinate has data from 5th May - 8th May at certain hours of the day at hourly resolution.
So above are the data at different (lat/lon) at 2016-05-05(00:00) UTC, this data focuses on the Pokhara region of Nepal.
Now we want to generate our data frame such that we have: date, the latitude, the longitude, and corresponding values of albedo. We will write our script such that Our final data should somewhere look like this:
So let's code and convert this netCDF to .csv or .txt
####netcdf to csv#######
###Countless attempts :( :(####
###**Testscript
##dedicated to my dearest friend and mentor Alaa Mhawish :D#####
###Clear and clean workspace###
rm(list=ls(all=TRUE))
#####Requires netcdf libarary #######
library(ncdf4) #####Load netcdf libarary #######
######Set working directory###########
setwd("D:/flight_data/ERA5/")
ncfname<- "albedo.nc" ######load data file#######
#####load nc file####
ncin<- nc_open(ncfname)
#####get longitude from file#####
lon <- ncvar_get(ncin, "longitude")
nlon<- dim(lon) ##check length or dimension###
head(lon) ##print first few columns##
#####get latitude from file#####
lat<- ncvar_get(ncin, "latitude")
nlat<- dim(lat)
head(lat)
######Get time or dates########
tt<- ncvar_get(ncin, "time")
units<- ncatt_get(ncin, "time", "units")
ntt<-dim(tt)
head(tt)
print(c(nlon, nlat, ntt))
dname1 <- "fal"
####Pull the albedo data####
fal.array1 <- ncvar_get(ncin, dname1)
###Get other information for the data###
dlname1<- ncatt_get(ncin, dname1, "long_name")
dunits1<- ncatt_get(ncin, dname1, "units")
fillvalue<- ncatt_get(ncin, dname1, "_FillValue")
###Check dimension of albedo data####
dim(fal.array1)
##(ncdf4) automatically converts Fillvalues to NA######
nc_close(ncin) ###Addition by Alaa###(Saves your RAM removes netCdf from memory)
######Convert albedo into vector form#######
fal.vec.long1<- as.vector(fal.array1)
####Check length of vector
length(fal.vec.long1)
####Create a matrix with longitude*latitude*time#######
tlonlat1<- as.matrix(expand.grid(lon, lat, tt))
#####bind albedo vector with matrix (lat/lon/time) by row######
fal_final <- data.frame(cbind(tlonlat1, fal.vec.long1))
#####Var3 here is the date column, since the data format is 'hours since 1900-01-01 00:00:00, we convert to local time ########
fal_final$Var3<- as.POSIXct(fal_final$Var3*3600,origin='1900-01-01 00:00')
####Rename headers of the column######
names(fal_final)<- c( "longitude", "latitude","date", "albedo")
#####Convert fal_final into data.frame format#####
df<- as.data.frame(fal_final)
###save data frame to txt######
write.table(df, "albedo.txt", sep = " ", row.names = FALSE)
#####save data frame to csv ######
write.csv(df, "albedo.csv, header = TRUE, row.names = FALSE)
The link to the R script is available in the following link :
So, likewise, we can convert any NetCDF files from ERA or any other source very quickly in R using the ncdf4 library.
Hope this blog was useful
Happy coding !!!!!
TAKE CARE AND STAY SAFE.
Comments