Digital Geography

25. April 2016

From Cloud to GIS: Getting Weather Data.

Introduction

In this post we are going to look at how to get weather forecast data and display it as a layer in a desktop geographical information system (GIS). The tools that you need are:

  • A desktop GIS. I am going to use QGIS (free software) but I have done this with ESRI ArcMap as well
  • A spreadsheet application. I will be using MS Excel but I have also tried this with Softmaker Office and I am sure LibreOffice (free software) will work or indeed any other spreadsheet application
  • The Natural Earth Quick Start data set. This is a collection of public domain map data that describe the planet, (http://www.naturalearthdata.com/downloads/).   You only need this if you don’t already have some data describing coastlines, countries etc.

We are going to use weather forecast data from the Global Forecast System (GFS) of the USA’s National Weather Service (NWS).

GFS is a global numerical weather prediction system that is run 4 times a day on a powerful super computer.  The forecast data is released under an OpenData licence and is accessible directly at the following address

http://nomads.ncep.noaa.gov/

However if you navigate to the above link you will be confronted with data files in some difficult to use scientific formats such as GRIB (Gridded Binary). Fortunately the GFS forecast is also available on an ERDDAP server. ERDDAP is Cloud based data broker designed to act as the intermediary between developers/users and scientific data formats (particularly gridded data). Please see my previous post for an introduction to ERDDAP.

Getting the Data

Navigate one of the following ERDDAP servers:

On the landing page enter ‘gfs’ into the search box’.

This will return two data sets, select the GFS one by clicking on the  ‘Graph’, this will bring you  to the data query form (see figure) which we can use to get data and to construct a URL query string that can be reused at later date.

The query form for the GFS data, parameters to change are highlighted with green.

On the query (graph) form web page make following changes:

  • Graph type to Vector
  • Vector X to ugrd10m
  • Vector Y to vgrd10m
  • Click on the map to get the data for area of the Earth that you are interested in (try the North Atlantic for this example).
  • Optional: change the ‘Stop’ time. This is time of forecast and it is default at last time of the forecast, click on the ’-’ to move the time backwards
  • File type to CSV (comma separated values file)

Once you have made the necessary changes click on the ‘Download Data or Image’ button to get your data. Copy the data from where it downloaded to a working directory. Below is a link to the query  I used when I was writing this post.

https://bluehub.jrc.ec.europa.eu/erddap/griddap/NCEP_Global_Best.htmlTable?ugrd10m[(2016-04-25T18:00:00Z)][(34.0):(70.5)][(323.0):(359.5)],vgrd10m[(2016-04-25T18:00:00Z)][(34.0):(70.5)][(323.0):(359.5)]&.draw=vectors&.vars=longitude|latitude|ugrd10m|vgrd10m&.color=0x000000

Processing the Data

The next stage is to convert the data into a format that we can use in the GIS. Firstly we need to do the following:

  • Import the CSV into your spreadsheet of choice. The delimiter is Comma
  • Delete the second row of data
  • Delete the time column, we won’t be using it
  • Save the file in a native format of the spreadsheet (csv is text format)

You should now have the following four columns of data:

  • latitude: This data is fine and won’t need processing
  • longitude: This data needs to be converted to a format usable by the GIS
  • ugrd10m & vgrd10m: These two columns represent the U and V component of the wind vector and we need to use these to calculate the wind speed and wind direction

Conversion of the longitude parameter

GIS programs all want data with longitude values within -180 to 180, but many data sets have longitude values within 0 to 360. So if the data set already has longitude -180 to 180 values you do not need to do this conversion. However in this example we do need to convert.

Longitude in the GFS model output starts at the Greenwich meridian 0 degrees and moves east at regular increments up to 359.5 degrees. This representation needs to be converted to more usual:

  • -180 to 0 to 180

Add the following equation to an empty column, in the first cell enter New_Long

= IF(MOD(B2,360) < 180, MOD(B2,360),MOD(B2-180,360)-180)

Where B2 is the original longitude column

Now calculate the new longitude by dragging the equation cell down the spread sheet to last row of longitude data.  A quick way of doing this is to select the cell with the equation, then Shift + Left Click the last ell in the column that you want to populate to and finally Ctrl D (this is the process works in Excel).

Calculating the Wind Direction

To calculate the wind direction we need to apply the arctangent (atan2) with the arguments U & V.

While that atan2 function is available in most spreadsheets and programming languages it is not ubiquitous

The format of atan2 in spreadsheets is different from that used in programming languages and some GIS application in that it has the parameters reversed.

  • Atan2 (vgrd10m, ugrd10m)

So to calculate wind direction enter the following equation into an empty cell of row 2:

=Atan2 (D2, C2)

where D2 is column vgrd10m and C2 is ugrd10m
Populate the column by dragging the equation down to the last row with data. Enter the name Wind_Dir into the first cell of the column.

Calculating the Wind Speed

The steps for calculating wind speed should be a familiar process now.  This time enter the following equation into an empty cell of row 2:

=SQRT(C2 * C2 + D2 * D2)

where C2 is ugrd10m and D2 is vgrd10m
Populate the column down to the last data row and put the name Wind_Speed into the first cell of the column.

Collating the Data

Open a new spreadsheet in you worksheet and copy the following columns to the spreadsheet:

  • latitude
  • longitude, use the new longitude data, make sure to paste just the values instead of the equation and values
  • Wind_Dir, make sure to paste just the values instead of the equation and values
  • Wind_Speed, make sure to paste just the values instead of the equation and values

Give the spreadsheet a name and save it as CSV file.

Importing the Data into a GIS

Finally we come to section where we import the wind data in a GIS and display it as a layer. In many ways this step is the simplest since all the difficult data wrangling has already been done.

To start open a new blank project in QGIS (or ArcMap, the process is very similar) and click on the ‘Add Delimited Text Layer’ button which is usually found on the left hand side of the screen. This will trigger ‘Create a Layer from a Delimited Text File’ form.  Click on the ‘Browse’ button and navigate to, and select, the CSV file that contains the processed data. Check everything on the ‘Create a Layer…’ form making sure that the CSV option is selected and that the x field is longitude and the y field is latitude. Click on the OK button select ‘WGS 84’ from the coordinate reference systems of the world.

Displaying the Data

You should now see the data displayed as points on the GIS layer.  On the layers panel right click on the layer and select the properties option. Do the following steps:

  • Change the ‘Single Symbol’ option to ‘Graduated’
  • In the Column option choose Wind_Speed (or whatever you named you wind speed column)
  • Click on the Symbol and choose the simple marker of and arrow pointing up. Now change the symbol layer type to ‘Font marker’ scroll down to the bottom of the font grid and select the arrow pointing upwards.
  • Now select a field for the rotation of the arrow, this is to the right hand side of the rotation text box, the field you need to activate should be Wind_Dir, unless you have used a different name.
  • Click ‘OK’
  • Back at the properties section choose:
    • A colour ramp
    • Equal Interval as the mode in the Classes section
    • Click on the ‘Classify’ button
    • Click on the ‘OK’ button

When you are returned to the GIS the layer should be populated by arrows which are pointing in the direction of wind flow, the arrows will also be coloured based on wind speed.

Adding Some Geographical Context

To put the wind data into geographical context we can now add a layer of global coastline data. This is contained in the Natural Earth data folder that you download earlier. Click on the ‘Add Vector Layer’ button in the left hand side of QGIS and navigate to the Natural Earth Data folder and select the file ‘ne_10m_coastline.shp’.

The final result

Conclusion and Next Steps

A small bit of effort will greatly improve the presentation of the wind data. In the figure below I added some data layers from the Natural Earth data resource and tweaked the colours for wind speed.

A small bit of effort to improve presentation

The work flow involved in adding other GFS parameters is very similar to what I have shown here, so why not try adding more forecast parameters? The available parameters are here:

Another option is to add data from other models, once again so long as you are using ERRDAP the work flow is very similar to what I have explained here. As a suggestion you can try the following:

So long as you pay attention to dates and projections you too can be using GIS to make ‘mashups with heterogeneous data’! Enjoy!

  • Paolo Cavallini

    Thanks. Why not consolidating all this into a QGIS plugin?

    • That is a great idea, I will see what I can do, but don’t hold your breath I am pretty busy 🙂

    • Conor Delaney

      That is a great idea, I will see what I can do.

      • Paolo Cavallini

        Glad you like it. If you need help, just let me know.

  • Lutra

    In the section for downloading it as CSV, you can choose NetCDF format. Crayfish plugin for QGIS will allow you to display and animate the format without any conversion:

    https://github.com/lutraconsulting/qgis-crayfish-plugin
    http://www.lutraconsulting.co.uk/products/crayfish/

    • Conor Delaney

      Very nice. I thought there must be a suitable plugin somewhere. I hope that you are finding the ERDDAP technology useful?

      • Lutra

        We haven’t used it yet but good to know the service.

  • MH Ali

    Thank u
    how can we relate different parameters(wind
    pressure,humidity,presipitation,rain falling) with temperature in GIS
    (Arcmap). I mean i want to find the trend of change in temperature of a region
    w.r.t. above given parameters.

  • Felipe

    In the next International QGIS Conference, that will take place in Girona (Spain), my collegue Mauricia will explain a new plug in that will be realesed soon, the title of the talk is “THREDDS Explorer plug-in, bringing meteoceanographic operational data into QGIS” you can hace a look to the abstract here.. http://www.sigte.udg.edu/jornadassiglibre/international-qgis-user-and-developer-conference/conferencia-qgis/

  • Ilya Khamushkin

    That’s a pretty detailed and low-level tutorial! Love it!

    Excuse me for bringing my own stuff inside this topic 🙂

    I’m eager to learn about struggles that data engineers and scientists are having with data processing and generally data access.

    I’m working at PlanetOS, where we created a simple point based (probe) API to access similar climate/weather datasets (we have GFS, NWW3, HYCOM and bunch of others).

    If anybody is up to share some feedback, please, let me know.

    You can learn more about available datasets at http://data.planetos.com

    Links to API documentation, examples (Python, JS) are shared in this blog post

    https://planetos.com/blog/six-new-datasets-developer-tools-and-our-own-slack-community/

    I will look for feedback here, but we also have a chat tool on our website, where you can provide feedback or ask questions.

    I hope Digital Geography Team doesn’t mind such comment, if it’s against the rules, I will delete my comment, just let me know 🙂

    • Conor Delaney

      Hi Ilya, thanks for youy comment. I have only been able to have brief look at the PlanetOS api. I was looking at GFS (NCEP) and the first thing that struck me was how do I get two parameters at the same time? I was trying to recreate the workflow I described above but using PlanetOS instead of ERDDAP. I am sure it can be done, I just haven’t had enough free time to look at it.

  • Hewerton da silva

    Dear Conor Delaney,

    I would like to congratulate you for the explanation. I’m finding it difficult to generate the wind field direction in Qgis, I can not identify whether the error is in the formula. I am a doctoral student at the Federal University of Pernambuco in Brazil and my research is to identify natural seeps of oil in the Plateau of Pernambuco.

    I thank you.

    • Conor Delaney

      Hi Hewerton, I only just saw your question. Are are you still having the problem?

      • Hewerton da silva

        Hi Conor Delaney,

        I’m not able to rotate the vectors to give the wind direction, as tenten several times and it still fails. I’ve redid the calculation several times and the results are the same and also I do not know if I’m erring on Qgis, no longer know what to do. I send you herewith a part of my spreadsheet.

        I plan to use this data for my doctoral research.

        Thank you for helping me.

        • Conor Delaney

          I haven’t have a chance to look at this. Can you send me the are of interest you are looking at so that I can see if I can reproduce the problem.

        • Conor Delaney

          Hi Hewerton, sorry for the delay. I think I know what the problem is. First of all I haven’t been able to work with GFS data as the servers in Hawaii seem to be slow today. So I worked with RTOFS data instead (it is hosted on a different server) and I noticed that QGIS will not allow you to set a wind direction field if #Value! appears anywhere in the column, I did a CTRL-H on the spreadsheet and replaced them all with 0. Once I did this I could set the wind_dir (and wind_speed) fields in the layer. Looking at the snippet of data you have above, you should consider reducing the precision by rounding the data to 2 or less decimal points. This is easy to do in QGIS by applying a function to the column, however it may take a while. You might not want to do this for modelling, but for displaying data it will greatly help. Let me know if you work things out?

  • Javier

    Hi Conor,

    Congrats for this wonderful post! I’m a PhD student from Madrid, Spain, and I’m interested in fungal spores dispersion through wind, animals, etc. I was using the information of your post some weeks ago to perform an R script to automatize all the tasks that you describe here, but today, suddenly, the gfs wind database has disappeared! It is only the sea wave data… do you have some information? Thanks in advance anyway! And congrats again or this great job!
    Cheers
    Javi

    • Javi Fernández-López

      The server is up again, thanks! 🙂
      Cheers,
      Javi

      • Conor Delaney

        Sorry in the delay in getting back to you. With ERDDAP when a remote data server goes down it is automatically removed from the catalog. ERDDAP keeps contacting the remote server until it is ready again and then adds the data back into the catalog.

        You might find this data useful as well : http://coastwatch.pfeg.noaa.gov/erddap/griddap/esrlNcepRe.graph

        It is the global reanalysis weather forecast data back to 1948.

        Cheers

        Conor

  • Javi Fernández-López

    Dear Community! I was following this great post and I was developing several functions in R language to automatize this tasks. I put all of them in an R package called rWind, hosted in CRAN repository. Here you have a brief tutorial to use it!

    http://allthiswasfield.blogspot.com.es/2016/12/rwind-r-package-released.html

    Cheers, Javi

    • Conor Delaney

      Fantastic work Javi, I just tweeted the link

      • Javi Fernández-López

        Thanks for the feedback Conor! And thanks for the global reanalysis weather forecast data, I’m thinking to change my R functions to obtain the data directly from this database… are there many differences between them, apart dates?

        I’m interested also in global sea currents… do you know any general database from ERDDAP to get this data? I’m working to implement also some functions to compute least cost paths having into account this kind of data to perform anisotropic analysis, I’ll add to my package as soon as possible… I’ll inform you!

        • Conor Delaney

          Hi, the GFS data in ERDDAP is only updated daily, so if you go straight to the source (http://nomads.ncep.noaa.gov) You will get data at higher resolutions. Any some stage this data will be on ERDDAP. As to sea currents, the two modelling sources that are available to you are the Real Time Ocean Forecast System (RTOFS) from NOAA, which is on ERDDAP or the EU equivalent which can be found here http://marine.copernicus.eu/services-portfolio/access-to-products/.

          Looking forward to seeing your new functions. By the way if you start using RTOFS please be aware that the the geographic grid in the data is different from the Polar Circles onwards