Downloading and formatting historical wind generation and load data

This process refers to the Midwest ISO, but I’m sure other ISOs have a similar process. Thanks goes to Jason Watters at MISO Client Relations for pointing me to the data.

  1. Visit MISO “Library” page.
  2. Click “Market Reports
  3. Choose “Summary,” from the Report Type field.
  4. Select “Include Archived Files.”
  5. The Report Name “Daily Regional Forecast and Actual Load” gives you access to the load data.
  6. The Report Name “wind” gives you access to the wind generation data.

The wind data should be easy enough to download and collate by hand and then import into matlab or R or whatever.

Regarding the load data, it’s kind of a pain to download the ~1000 (and growing) files by hand, so I highly recommend the DownloadThemAll! firefox extension for this. Moreover, the format of this data is somewhat clumsy to work with: separate excel files for each day (containing hourly predicted and actual load information for the previous day and only predicted load information for the day itself). I used the following steps to get all the hourly data together, in a similar format as the wind generation data:

  1. Make sure you have OpenOffice installed.
  2. Download PyODConverter, and read the instructions on the website.
  3. Write a script to run the convert command on all files. Just to peeve my unix friends, I did this as a windows batch script: loopFiles.bat 🙂
  4. Write a script to concatenate the files and remove the unwanted information. In this case, I used Cygwin and piped together a couple of bash commands to do the job in a snap:
    head -q -n 31 *.xls.csv | grep ",,," -v | grep "Market Day" -v > winddata.csv
  5. Depending on your needs, you may want to combine the date and hour fields into a single timestamp. This can be done a number of ways, but it’s easy enough to use Excel or OpenOffice to do the formatting for you. Remember, the numerical representation of dates is in days.

And there you have it! Years of hourly wind generation and grid load data at your fingertips.

8 thoughts on “Downloading and formatting historical wind generation and load data

  1. Very cool, have you created any interesting graphs with the data? I had been looking at this too – I had been downloading daily csvs, I didn’t realize they had archives. 🙂

  2. Sure have! Though, most of them use this data as input to some demand shaping models I’m developing. I’m preparing a paper for publication, so I’m holding off on posting the results.

    But, judging from your blog (interesting to read, BTW!), you’d probably rather have the data and make some graphs yourself. 😉
    MISO Wind Data: http://cl.ly/1O0w2l0H2G0K0s2v171d
    MISO Load Data: http://cl.ly/1q02422w3X1r2J38331s

    Particularly interesting is an autocorrelation of the load data. I knew there would be daily, weekly, and yearly patterns, but it’s pretty neat to see it for yourself. There’s also a 180 day pattern that I can’t quite figure out. Maybe it’s the school year?

  3. FWIW, if you search for “Historical Regional Forecast ” you’ll get yearly csv files, easier to deal with than the 1000 or so you got 😉

    Thanks for the links to your csv files, I had wind done but hadn’t grabbed load yet (though it’ll be easier with the tips above)

  4. Well, I already have my EE degree, so I am naturally better at some of these things 😉 Sounds like you are working on interesting stuff, good luck!

  5. Er, last note – the historical load stuff is in .xls, and in a different format, more difficult to work with. Argh. I’ll just pull down your csv data, thanks again!

  6. Thanks for the information
    I was looking to find the Wind Power actual + Wind Power Forecast, but I found the actual only , do you know if they have the forecast as well. Also, Do they have an hourly data for a year for the hourly spot market price ?
    I appreciate your help
    Thanks,

Leave a Reply

Your email address will not be published. Required fields are marked *