A Script for Pulling Oil and Gas Prices

This small price fetching script comes from a much larger program in Faro’s library which allows a company to get a total view of a basin’s metrics in an incredibly short time frame.  The program will:

1. Clean all the production, geologic, and sundry data that it is fed.
2. Create other metrics that can be calculated from government / state data, the FracFocus database, and other sources.
3. Normalize and high-point align production data for all wells.
4. Use our proprietary curve fitting algorithm to automate fitting all production streams for wells over 12 months old.
5. Utilizing the metrics from those forecasts, along with the general well data, use machine learning algorithms to properly classify the younger vintage wells and predict type curve areas for new locations.
6. Use best estimate capital costs and lease operating expenses to forecast all well economics – and if the data is available, perform look backs.
7. Generate geologic, production, EUR, and IRR maps (in both Python and other mapping software).

About the Program and Price Sites

Before you use this script, you will have to install a webdriver for the web browser you prefer (if you don’t already have the driver).  In this case, we use Firefox.  Chrome is great, but when it comes to looking at what is going on in the background of a website we prefer Firefox’s tools.  So, to get GeckoDriver for Firefox, go to this page.

Typically, when you are looking at a site from which to pull data, you can get an idea of the table format by just right clicking anywhere on the page and choosing to view the page source.  For example, if you go to the EIA page for Cushing, OK WTI daily historic prices and view the page and its source, you will see:

Getting the Data

In the case of the CME site, when you try to view the source, no tables or tags will pop up, but there is an easy work around to get to the originating data url.  Go to the CME site in your Firefox browser and in the upper right hand corner there should be a drop down menu.  From there, go to Web Developer and select Network.  There are some options at the top of that window – “All  HTML  CSS  JS  XHR…”.  Pick XHR and reload the page.  You will see one code in particular that keeps popping up with a long, changing number after the refresh. That is going to be the site with the actual prices. Right click it, open up the link, and select “Raw Data” at the top of that page.  That is all of the data which you will parse out using the descriptors in front of the values you want.

Jupyter Notebook Script

We have posted the full code for pulling both the CME prices and monthly historic prices from the EIA website on our github repository HERE.  We only show oil in this example, but it is just a matter of changing the urls and switching your variable names to reflect gas.

Additions You May Want to Make

One issue with the data you get is that there could be a break in the months between historic and futures prices. The EIA is slow to update history, and futures prices move quick. We account for this in our main program with some other pulls, but you can easily add two months with your estimated/known prices to the bottom of the resulting dataframe and re-sort it if you decide to put them together.

Interested in Learning Python?

If you are new to python or are interested in getting started on it, one of the more popular distributions is Anaconda.  It is free and, in our opinion, more friendly than other python distributions when it comes to set up, managing packages, and building environments.  You can download it here and will be able to set up your own Jupyter Notebook IDE.