Ruby: Stock Project – Source Data
Now that we have installed ruby, postgres and have tested the connection between ruby and postgres, we need to find some data.
Google search for - stock list file
The search request brings back several hits. This link looked promising, plus free code. CodeProject: AJAX Stock Symbol Drop-down List. Free source code ...
The code is Java, but maybe if I need some ideas, I'll go back and skim over it.
Near the bottom of the page is a list of sites that the author used to get the free stock symbol files. The site I’m going to use is the NASDQ site. NASDQ is being very helpful they have provided their securities, plus AMEX and the NYSE. We won't have to go anyplace else to get our data.
The nasdq link for the data is http://www.nasdaq.com//asp/symbols.asp?exchange=Q&start=0. When looking at the other links the only thing that changes is exchange=? where the ? mark is exchange identifier. Downloading the file shows that it is a CSV file, and that it also contains quite a bit of information.
The data for these files was last updated on the 31st of December 2007. It's recent enough, especially in this slow market, Hopefully this file is updated about once a quarter, otherwise we may have to look for other sources, but for now this will work.
Now that we know what our initial data source data looks like we can now start to design the ruby scripts and the database table layouts.
For starters we are going to break this project into three areas.
1. Database
2. Ruby scripts (back end)
3. Rails / Other tools user front end
Retrieving your source data, and any other kind of updates, like stock prices and other financial data will be back end processes.
So lets get started on getting our data.
As noted earlier the url for retrieving the files changes very little, only the parameter of the exchange changes. Instead of writing 3 separate steps we can write 1 step that accepts parameters. I'm going to start out by creating a back end process file that will contain processes that are repeated continuously.
save this as "back_end_functions.rb"
----------- Start File -------------------
require 'net/http'
def get_source_stock(url,path_param,file_name)
Net::HTTP.start(url) { |http|
resp = http.get(path_param)
open(file_name, "wb") { |file|
file.write(resp.body)
}
}
end
-------------- End File ------------------------
This is the function that will retrieve the actual CSV files from the NASDQ web site and save them where you would like them saved based on the file name passed in the parameters.
With the function created we can now create our retrieval script. Your path will most likely need to changed or created. I like to keep data and scripts in different locations. This reduces the chance that you accidentally delete your script while trying to delete data files.
-------------- Stat File ------------------------
require "back_end_functions" # contains of fuctions
nasdq_url = "www.nasdaq.com" # defines the url to retrieve files from
url_path = "//asp/symbols.asp?exchange=Q&start=0" #path and parameters to requesst file
file_name = "nasdq.csv" #Name we want to assign the file
file_path = "/data/ruby/stock/data/" #path that the file will be written to
file = file_path+file_name #combined value of file and path to pass to function
p file
get_source_stock(nasdq_url,url_path,file)
# data to get AMEX file exchange = 1
url_path = "//asp/symbols.asp?exchange=1&start=0"
file_name = "amex.csv"
file = file_path+file_name
p file
get_source_stock(nasdq_url,url_path,file)
#Change data to get NYSE file exchange = N
url_path = "//asp/symbols.asp?exchange=N&start=0"
file_name = "nyse.csv"
file = file_path+file_name
p file
get_source_stock(nasdq_url,url_path,file)
-------------- End File ---------------------------
Execute the script above and you should end up with 3 csv files in your data directory. From what I can tell everything has to be done from the same drive. I know you can change drives, just haven't tried that. No reason to change so far in this project.
Next step will be to look at all of the data files and start looking at how to design the database
Labels: Ruby, Stock