Downloading stock market quotes from Yahoo! finance

From WikiJava

The author suggests:

buy this book

Yahoo! finance is in my opinion the best free resource over the internet to read financial news and data. One of the features I love the most is the ichart service which allows you to simply download daily stock market data from any stock and index as a CSV (comma separated value) just using the HTTP Protocol.

In this article I'm going to explain you a simple java class that you can use for downloading any quote you need using Yahoo's ichart service.

The best of all? It's for free.

the ichart service

The ichart service is designed to be very simple to use, you can test it by just hitting with your browser on:

http://ichart.finance.yahoo.com/table.csv?s=YHOO&a=06&b=9&c=1996&d=06&e=20&f=2010&g=d

this will make you download a comma separated value (CSV) file, containing Yahoo's stock market values since 6th September 1996 until 20th, June 2010.

The content of the file looks like this:

Date,Open,High,Low,Close,Volume,Adj Close
2010-06-18,15.66,15.67,15.47,15.54,12639600,15.54
2010-06-17,15.72,15.72,15.44,15.60,10680400,15.60
2010-06-16,15.58,15.65,15.34,15.49,15920300,15.49
2010-06-15,15.29,15.69,15.23,15.65,13888300,15.65
2010-06-14,15.46,15.49,15.15,15.17,12493100,15.17
...
1996-07-12,17.00,18.00,17.00,17.50,1696000,0.73
1996-07-11,16.00,17.25,15.50,17.25,3510400,0.72
1996-07-10,18.75,18.75,16.00,16.37,5899200,0.68
1996-07-09,19.50,20.00,18.50,18.50,2112000,0.77

The URI above is very simple, I'm going to split and detail it:

http://ichart.finance.yahoo.com/: this is the domain name for this service, you don't want to change this part.
table.csv: this is the name of the service on Yahoo's servers, you are not going to change this either.
?s=YHOO: the first parameter (actually the ordering of the parameters shouldn't matter) s is the symbol of the stock quote you want to download. YHOO is Yahoo, MSFT is Microsoft, and so on.
&a=06: a, b and c parameters are represent the starting date you want to download, a is the month.
&b=9: b is the day
&c=1996: c is the year
&d=06: e, e and f parameters are represent the last date you want to download, a is the month.
&e=20: e is the day
&f=2010: f is the year
&g=d: g specifies the interval data: d for daily, w for weekly, m for montly

You can just tweak the values in the parameters above to obtain the stock data you want.

In the next paragraph I'm going to explain how to automate this in a little Java library.

Java can do it

Of course it can, and it's actually pretty simple. As I don't want to bother about http details and about csv parsing details I've used in this Library two open source libraries:

The httpClient (version 3.1), from the apache commons
the Java csv (version 2.0), from this sourceforge project

Using these two libraries the only tasks my little library is left to do are:

Generate the URL: which is basically a String
Execute the call: don't worry the nitty gritty is done by the Http Client library
Parse the CSV data obtained: the hard part is done by the Java CSV library

I'll give you the details in the next sections.

The string builder

As I mentioned above, the URL is just a string, so we can write the buildUri method so that it takes as parameters all the data it requires, and returns a String properly formed.

So I write the following:

	private String buildURI(Product product, TradingDay start, TradingDay end) {
		StringBuilder uri = new StringBuilder();
		uri.append("http://ichart.finance.yahoo.com/table.csv");
		uri.append("?s=").append(product.getSymbol());
		uri.append("&a=").append(start.getMonth());
		uri.append("&b=").append(start.getDay());
		uri.append("&c=").append(start.getYear());
		uri.append("&d=").append(end.getMonth());
		uri.append("&e=").append(end.getDay());
		uri.append("&f=").append(end.getYear());
		uri.append("&g=d");
 
		return uri.toString();
	}

Where Product and TradingDay are simple java beans, representing respectively the financial product I want to obtain data for, and the calendar dates I want to use. The StringBuilder is used to speed up the generation of the String.

Nothing much more to comment here, as this method is pretty simple.

Execute the call

Executing the call is just as simple as generating the URI, using the Http Client Library, all you got to do is to:

instantiate a HttpClient Object to do the call
instantiate a HttpMethod Object, containing the details about what you want the HttpClient to do (i.e. getting the URI you built in the previous section)
do the call and check the response code
convert the body of the Http response to a String

The following method doCall does just this. Here's the code:

	private String doCall(String uri) throws DownloaderException {
		HttpClient httpClient = new HttpClient();
		HttpMethod getMethod = new GetMethod(uri);
 
		try {
			int response = httpClient.executeMethod(getMethod);
 
			if (response != 200) {
				throw new DownloaderException("HTTP problem, httpcode: "
						+ response);
			}
 
			InputStream stream = getMethod.getResponseBodyAsStream();
			String responseText = responseToString(stream);
			return responseText;
 
		} catch (HttpException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
 
		return null;
	}

It takes a String containing the URI as parameter, instances the two objects,HttpClient, and HttpMethod, (note that the URI is passed as parameter to the constructor of this last one).

Then it calls:

int response = httpClient.executeMethod(getMethod);

That executes the call and returns the http status code.

The body of the http response, contains our CSV and can be retrieved as an InputStream with the following statement:

InputStream stream = getMethod.getResponseBodyAsStream();

from the HttpMethod object.

Finally it calls the method responseToString to convert the response to a string, for easier handling. Follows the code of the responseToString method, which just converts the InputStream to a String:

	private String responseToString(InputStream stream) throws IOException {
		BufferedInputStream bi = new BufferedInputStream(stream);
 
		StringBuilder sb = new StringBuilder();
 
		byte[] buffer = new byte[1024];
		int bytesRead = 0;
		while ((bytesRead = bi.read(buffer)) != -1) {
			sb.append(new String(buffer, 0, bytesRead));
		}
		return sb.toString();
	}

This completes the doCall Method which is responsible for doing the Http call, and returning the String containing the unparsed Data.

There are some Exceptions to be caught, but for the purpoose of this tutorial we'll just print them and forget about handling them properly.

Parse the data obtained

All that is left to do is to parse the String received from Yahoo as CSV, and insert the parsed data into java objects, easier to handle.

We are going to write the extractValues method for this, it will take as parameters the Product and the String containing the body of the Http response. The method is very simple, although quite long, as it's repeating the same operations on all the six fields of the CSV data file.

The code is as follows:

	private Product extractValues(Product product, String responseBody)
			throws IOException {
 
		// intantiate the original data ranges, and add them to the product
		OriginalDataRange openDR = new OriginalDataRange(product,
				DataRangeType.Open);
		OriginalDataRange closeDR = new OriginalDataRange(product,
				DataRangeType.Close);
		OriginalDataRange maxDR = new OriginalDataRange(product,
				DataRangeType.Max);
		OriginalDataRange minDR = new OriginalDataRange(product,
				DataRangeType.Min);
		OriginalDataRange adjDR = new OriginalDataRange(product,
				DataRangeType.Adjusted);
		OriginalDataRange volumeDR = new OriginalDataRange(product,
				DataRangeType.Volume);
 
		// add the ranges to the product
		product.addRange(openDR);
		product.addRange(closeDR);
		product.addRange(maxDR);
		product.addRange(minDR);
		product.addRange(adjDR);
		product.addRange(volumeDR);
 
		CsvReader csvReader = new CsvReader(new StringReader(responseBody));
		csvReader.readHeaders();
 
		// populating the lists
		while (csvReader.readRecord()) {
			// Date,Open,High,Low,Close,Volume,Adj Close
			// extract & parse data
			String dateStr = csvReader.get(0);
			String openStr = csvReader.get(1);
			String highStr = csvReader.get(2);
			String lowStr = csvReader.get(3);
			String closeStr = csvReader.get(4);
			String volumeStr = csvReader.get(5);
			String adjStr = csvReader.get(6);
 
			// convert the strings to the appropriate classes
			String[] splitted = dateStr.split("-");
			int year = Integer.parseInt(splitted[0]);
			int month = Integer.parseInt(splitted[1]) - 1;
			int day = Integer.parseInt(splitted[2]);
			TradingDay date = TradingDay.getSingleton(new GregorianCalendar(
					year, month, day));
			Double open = Double.parseDouble(openStr);
			Double close = Double.parseDouble(closeStr);
			Double high = Double.parseDouble(highStr);
			Double low = Double.parseDouble(lowStr);
			Double volume = Double.parseDouble(volumeStr);
			Double adj = Double.parseDouble(adjStr);
 
			// Create objects
			StandardQuote openQuote = new StandardQuote(product, date,
					new StandardValue(open));
			StandardQuote closeQuote = new StandardQuote(product, date,
					new StandardValue(close));
			StandardQuote maxQuote = new StandardQuote(product, date,
					new StandardValue(high));
			StandardQuote minQuote = new StandardQuote(product, date,
					new StandardValue(low));
			StandardQuote volumeQuote = new StandardQuote(product, date,
					new StandardValue(volume));
			StandardQuote adjQuote = new StandardQuote(product, date,
					new StandardValue(adj));
 
			// add today rates to the lists.
			openDR.add(openQuote);
			closeDR.add(closeQuote);
			maxDR.add(maxQuote);
			minDR.add(minQuote);
			volumeDR.add(volumeQuote);
			adjDR.add(adjQuote);
		}
 
		return product;
	}

There are a few new classes mentioned here, it's nothing rocket science here, we are just populating these classes to form a structure of objects that represents our domain.

A StandardQuote object contains the value of one specific list of values in a specific moment in time. OriginalDataRange is a class containing a List of StandardQuote objects. We are basically converting the data in the CSV (which contains all the values grouped per each specific day) to a sort of dual space representation (where the values are grouped by what they represent). More details on these classes in A filter system, for retrieving data from a List.

Here it will be sufficient to say that Product contains a collection of OriginalDataRange, that can be added via the Product.add(DataRange) method, and that OriginalDataRange contains a Collection of StandardQuote that can be added via the OriginalDataRange.add(StandardQuote) method.

Putting it all together

All the methods we wrote so far in the article are completely disjointed each other, what we need now is something to armonize it all, to make it usable.

What we need to write now is a controller for the operations of the library, so that we can just use it by calling a simple method of the class that will hide all the complexities explained above. Such method will basically keep the interface clean and easy to understand.

We are going to write the only public method of our YahooDownloader Library. This is not yet the orchestrator, but it's a useful indipendence tool that makes the code clearer. Here's its code:

	public Product getQuotationRange(Product product, TradingDaysRangeA range)
			throws DownloaderException {
 
		Product result = doYahooDownload(product, range.getStart(), range
				.getEnd());
		return result;
	}

This method serves just as a level of decoupling between the real call and the caller, it helps also maintaining the names of the methods clear. It's not strictly required to the economy of the library, but it's sometimes useful having such methods, for example if I'll want to add logging to this library, this will be certainly a method I'll want to have some logging for.

There's one more Class introduced here, the TradingDaysRangeA which basically contains two TradingDay objects and represents a range of dates.

The real method doing the orchestration is doYahooDownload. Which follows:

private Product doYahooDownload(Product product, TradingDay start,
			TradingDay end) throws DownloaderException {
		assert start != null;
		assert end != null;
		assert !start.isAfter(end);
		assert product != null;
		assert product.getSymbol() != null;
		assert !product.getSymbol().isEmpty();
 
		// creating the request URI
		String uri = buildURI(product, start, end);
 
		System.out.println("calling :" + uri);
 
		// doing the call
		String responseBody = doCall(uri);
		// System.out.println(responseBody);
 
		// parse results
		try {
			product = extractValues(product, responseBody);
		} catch (IOException e) {
			throw new DownloaderException(e);
		}
 
		return product;
	}

The assert statements at the beginning of the method serve to validate the input data, in this way it will be easier to troubleshoot errors with the parameters. Note that it is generally a bad idea to use assert in user inserted input. Normally we should always prefer other ways, such conditional validation methods (for example "if" statements) and exception throwing. This validation part should be modified if we want to use this library in a production environment.

The other parts of this method are quite immediate, and there's not much to say about them.

A side note that by the name of the methods the software using the library calls a method with a very generic name, it may not know where the data comes from, but right after I'm in the library, the doYahooDownload method is called, and will appear in any stacktrace, helping out in the if I'll have to debug any exception or problem with the library.

Using the Library

Having the class ready, I can now use it very simply, for example in the following spike program:

/**
 * This file is part of the StockFriend Project and it's contents are strictly confidential. 
 * The project and this file are owned by Giulio Giraldi, if you have received this file 
 * without explicit consent from the owner please delete it and contact immediately 
 * Giulio Giraldi (dongiulio@gmail.com).
 */
package org.wikijava.stockfriend.backend.controller;
 
import org.wikijava.stockfriend.backend.data.YahooDownloader;
import org.wikijava.stockfriend.model.DownloaderException;
import org.wikijava.stockfriend.model.calendar.TradingDay;
import org.wikijava.stockfriend.model.calendar.TradingDaysRange;
import org.wikijava.stockfriend.model.product.BaseProduct;
import org.wikijava.stockfriend.model.quotations.filters.DefaultFilters;
 
/**
 * <Replace this with a short description of the class.>
 * 
 * @author Giulio Giraldi
 */
public class DownloadDataAndRetrieveOpens {
 
	/**
	 * <Replace this with one clearly defined responsibility this method does.>
	 * 
	 * @param args
	 */
	public static void main(String[] args) {
		YahooDownloader yahooDownloader = new YahooDownloader();
 
		BaseProduct product = new BaseProduct("YHOO", "Yahoo! inc.");
		TradingDay start = TradingDay.getSingleton(-300);
		TradingDay end = TradingDay.getSingleton();
 
		TradingDaysRange range = new TradingDaysRange();
		range.setStart(start);
		range.setEnd(end);
 
		try {
			yahooDownloader.getQuotationRange(product, range);
		} catch (DownloaderException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
 
		System.out.println(product.getRanges(DefaultFilters.closes));
 
	}
}

Note that the last command requires you to have followed, and implement the related article A filter system, for retrieving data from a List. Which I recommend to read anyway.

The complete code

package org.wikijava.stockfriend.backend.data;
 
import com.csvreader.CsvReader;
 
import java.io.BufferedInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.StringReader;
import java.util.GregorianCalendar;
 
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.HttpMethod;
import org.apache.commons.httpclient.methods.GetMethod;
import org.wikijava.stockfriend.model.DownloaderException;
import org.wikijava.stockfriend.model.calendar.TradingDay;
import org.wikijava.stockfriend.model.calendar.TradingDaysRangeA;
import org.wikijava.stockfriend.model.product.Product;
import org.wikijava.stockfriend.model.quotations.DataRangeType;
import org.wikijava.stockfriend.model.quotations.OriginalDataRange;
import org.wikijava.stockfriend.model.quotations.StandardQuote;
import org.wikijava.stockfriend.model.quotations.StandardValue;
 
/**
 * 
 * @implements DayDownloaderI
 * 
 * @author Giulio
 */
public class YahooDownloader implements DayDownloaderI {
 
	/**
	 * Creates a new instance of LoadEndOfDay
	 * 
	 * @throws DownloaderException
	 */
	private Product doYahooDownload(Product product, TradingDay start,
			TradingDay end) throws DownloaderException {
		assert start != null;
		assert end != null;
		assert !start.isAfter(end);
		assert product != null;
		assert product.getSymbol() != null;
		assert !product.getSymbol().isEmpty();
 
		// creating the request URI
		String uri = buildURI(product, start, end);
 
		System.out.println("calling :" + uri);
 
		// doing the call
		String responseBody = doCall(uri);
		// System.out.println(responseBody);
 
		// parse results
		try {
			product = extractValues(product, responseBody);
		} catch (IOException e) {
			throw new DownloaderException(e);
		}
 
		return product;
	}
 
	private Product extractValues(Product product, String responseBody)
			throws IOException {
 
		// intantiate the original data ranges, and add them to the product
		OriginalDataRange openDR = new OriginalDataRange(product,
				DataRangeType.Open);
		OriginalDataRange closeDR = new OriginalDataRange(product,
				DataRangeType.Close);
		OriginalDataRange maxDR = new OriginalDataRange(product,
				DataRangeType.Max);
		OriginalDataRange minDR = new OriginalDataRange(product,
				DataRangeType.Min);
		OriginalDataRange adjDR = new OriginalDataRange(product,
				DataRangeType.Adjusted);
		OriginalDataRange volumeDR = new OriginalDataRange(product,
				DataRangeType.Volume);
 
		// add the ranges to the product
		product.addRange(openDR);
		product.addRange(closeDR);
		product.addRange(maxDR);
		product.addRange(minDR);
		product.addRange(adjDR);
		product.addRange(volumeDR);
 
		CsvReader csvReader = new CsvReader(new StringReader(responseBody));
		csvReader.readHeaders();
 
		// populating the lists
		while (csvReader.readRecord()) {
			// Date,Open,High,Low,Close,Volume,Adj Close
			// extract & parse data
			String dateStr = csvReader.get(0);
			String openStr = csvReader.get(1);
			String highStr = csvReader.get(2);
			String lowStr = csvReader.get(3);
			String closeStr = csvReader.get(4);
			String volumeStr = csvReader.get(5);
			String adjStr = csvReader.get(6);
 
			// convert the strings to the appropriate classes
			String[] splitted = dateStr.split("-");
			int year = Integer.parseInt(splitted[0]);
			int month = Integer.parseInt(splitted[1]) - 1;
			int day = Integer.parseInt(splitted[2]);
			TradingDay date = TradingDay.getSingleton(new GregorianCalendar(
					year, month, day));
			Double open = Double.parseDouble(openStr);
			Double close = Double.parseDouble(closeStr);
			Double high = Double.parseDouble(highStr);
			Double low = Double.parseDouble(lowStr);
			Double volume = Double.parseDouble(volumeStr);
			Double adj = Double.parseDouble(adjStr);
 
			// Create objects
			StandardQuote openQuote = new StandardQuote(product, date,
					new StandardValue(open));
			StandardQuote closeQuote = new StandardQuote(product, date,
					new StandardValue(close));
			StandardQuote maxQuote = new StandardQuote(product, date,
					new StandardValue(high));
			StandardQuote minQuote = new StandardQuote(product, date,
					new StandardValue(low));
			StandardQuote volumeQuote = new StandardQuote(product, date,
					new StandardValue(volume));
			StandardQuote adjQuote = new StandardQuote(product, date,
					new StandardValue(adj));
 
			// add today rates to the lists.
			openDR.add(openQuote);
			closeDR.add(closeQuote);
			maxDR.add(maxQuote);
			minDR.add(minQuote);
			volumeDR.add(volumeQuote);
			adjDR.add(adjQuote);
		}
 
		return product;
	}
	private String doCall(String uri) throws DownloaderException {
		HttpClient httpClient = new HttpClient();
		HttpMethod getMethod = new GetMethod(uri);
 
		try {
			int response = httpClient.executeMethod(getMethod);
 
			if (response != 200) {
				throw new DownloaderException("HTTP problem, httpcode: "
						+ response);
			}
 
			InputStream stream = getMethod.getResponseBodyAsStream();
			String responseText = responseToString(stream);
 
			return responseText;
		} catch (HttpException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
 
		return null;
	}
 
	private String responseToString(InputStream stream) throws IOException {
		BufferedInputStream bi = new BufferedInputStream(stream);
 
		StringBuilder sb = new StringBuilder();
 
		byte[] buffer = new byte[1024];
		int bytesRead = 0;
		while ((bytesRead = bi.read(buffer)) != -1) {
			sb.append(new String(buffer, 0, bytesRead));
		}
		return sb.toString();
	}
 
	private String buildURI(Product product, TradingDay start, TradingDay end) {
		StringBuilder uri = new StringBuilder();
		uri.append("http://ichart.finance.yahoo.com/table.csv");
		uri.append("?s=").append(product.getSymbol());
		uri.append("&a=").append(start.getMonth());
		uri.append("&b=").append(start.getDay());
		uri.append("&c=").append(start.getYear());
		uri.append("&d=").append(end.getMonth());
		uri.append("&e=").append(end.getDay());
		uri.append("&f=").append(end.getYear());
		uri.append("&g=d");
		uri.append("&ignore=.csv");
 
		return uri.toString();
	}
 
	@Override
	public Product getQuotationRange(Product product, TradingDaysRangeA range)
			throws DownloaderException {
 
		Product result = doYahooDownload(product, range.getStart(), range
				.getEnd());
		return result;
	}
}

Comments from the users

To be notified via mail on the updates of this discussion you can login and click on watch at the top of the page

Comments on wikijava are disabled now, cause excessive spam.

Navigation

Portals

Contribute

Highlights