Jr. Parrot on Command Line

Stock Valuations in R: Quantmod -package [part 1]

2012-02-25T07:10:00.001-08:00

We earlier covered how to get information in Python here but it is a waste of time. There is a wrapper in R that fetches data from different services, it is a bit limited for example in the amount of queries you can do but anyway a good tool when other tools stop working. We encourage our sailors to deepen their knowledge with R -style questions here and take this as a challenge. Technical knowledge is nothing without the social aspect. Hope to make your speculation a bit less painful, have fun!

Code

library(quantmod)
getSymbols("VT")
plot(VT)

EU crisis, R and Reinhart-Rogoff -rule

2012-01-21T07:07:00.000-08:00

The ambiguous Reinhart-Rogoff rule is that the gvt gross ratio over 90% will slow down economy otherwise not a big issue in a way such as [1]. It is very hard to create here some sort of causality so always good to verify things on your own! Have fuN!

[1] http://www.economist.com/blogs/freeexchange/2010/07/debt

# Description: vizualise the gvt gross debt in EU
# License: GPL 3 or later
# Author: MathHarbour.com

png('./euGroshDebt07022012.png')
par(mfrow=c(1,2), bg='white')

system('wget -O tmp.gz http://epp.eurostat.ec.\
europa.eu/NavTree_prod/everybody/\
BulkDownloadListing?file=data/tsieb090.tsv.gz')
tab <- read.table(gzfile('tmp.gz'), header=TRUE)
tab<-tab[c(41,44,48,49,50,66,70),] # DE,SE,FR,FI,UK,EU27
colors<-rainbow(length(tab[,1]))

m <- t(tab[,-1])   #only years and vals
mode(m) <- "numeric"
matplot(m, type="l", 
 pch = 1:nrow(m), 
 xaxt="n", col=colors, 
 ylab="% Gvt gross debt in EU")
axis(1, at=1:nrow(m), labels=rownames(m))

plot(1,1, axes=F, xlab='', ylab='')

legend('center', legend=tab[,1], col=colors, pch="*", lty=1:7)
title('Data source: Eurostat.')

Work with Large Spreadsheets in R

2012-01-14T07:07:00.000-08:00

It is essential that you can fast move between your editor and R. Here is how I do it, it is important to do it this way (not closing the R) because you won't lose the cache with precious vars etc. When you combine this tip with the earlier tip about Vim here, you get an excellent powerhouse!

> source('test.r')
> system('vim test.r')
> readLines('myDataXYZ')  # vectorize, capital!
> edit(someVar)           # very useful command!

Financial Time-serie Modelling in R [part 0]

2012-01-07T07:07:00.000-08:00

It is a rarity that a financial powerhouses provide clean data [1]. It is unfortunate, leaving room for speculation. The code below may look trivial but it is not hard from it to do more advanced things with small changes such as ARIMA -forecasting -- of course in R. Ironically, John Bogle once explained something that he likes fairer share for small guys. Luckily, there are certain firms (not Vanguard) offering historical valuations quite painlessly so you can have fun with this code. The below code requires that you have CSV -files to read. Have FuN!

Code

# Author: MathHarbour.com
# License: GPL 3 or later
#
# Description: R in financial modelling
#  

path<-'./Trial/*.csv'

colors<-rainbow(length(Sys.glob(path)))
names(colors)<-Sys.glob(path)

b<-lapply(Sys.glob(path), function(file) {
    e <- read.table(file)
    data.frame(as.Date(e$V1, "%d.%m.%Y"), e$V2)
})
names(b)<-Sys.glob(path)

png('./Public/logVals06012012.png')

vals<-do.call(rbind, b)[[2]]
vals<-log(vals)

plot( b[[1]][[1]], log(b[[1]][[2]]), 
 type='l', ylim=range(vals),
 col=colors[[Sys.glob(path)[1]]],
 xlab="Time",
 ylab="Log Valuation[ln(EUR)]") #init

Map(function(c, dat) 
 lines( dat[[1]], log(dat[[2]]), col=c ), 
 colors[-1], b[-1])

title("Logarithmic valuations")
#legend('top',
 #do.call(rbind, 
 #as.list(Sys.glob("./Trial/*.csv"))),
 #col=colors, lty=1:7, pch="*", cex=0.8)

dev.off()

References
1. http://www.bogleheads.org/forum/viewtopic.php?p=1011852

VT World Fund Calendar Heatmap with R

2012-01-01T19:07:00.000-08:00

Install the necessary packages as specified by the code, eg. install.packages('chron') in R, and then just:

$ wget https://raw.github.com/tavisrudd/\
r_users_group_1/master/calendarHeat.R
$ sed -i 's@if(FALSE@if(TRUE@g' calendarHeat.R
$ sed -i "s@MSFT@VT@g" calendarHeat.R
$ Rscript calendarHeat.R

...don't ask where I need this kind of toys, perhaps to visualize hotel bookings like here? What about researching the so-called January -effect on your own, more here. How is plot different with VIOV or VIOO?

P.s. ...do not believe that you could exploit the abnormality anymore, use this toy for your plays instead.

Monte Carlo, Python and Efficient Frontier -analysis

2011-12-07T07:07:00.000-08:00

We used Monte Carlo simulation in latter post, here [1], to visualize risk-return -profiles with efficient frontiers.Although MC is very imprecise, the result of MC converging very slowly \frac{1}{\sqrt(N)}, it is used because of its convenience. Shortly, the MC is strikingly different to other methods, such as Trapetz and Simpson, where lower dimensional polynomials are used to calculate the area while MC uses statistical method with random numbers.

Matlab

density = 100000
x = rand(density);
y = rand(density);
i = (x^2 .+ y^2)^0.5 < 1;
Area3 = (sum(i) / density)*4;

Python

import numpy as np

density=int(1e5)
x = np.random.random(density)
y = np.random.random(density)
i = (x**2.0+y**2.0)**0.5 < 1
Area3 = sum(i)/float(density)*4.0

And as a comparison a method that uses polynomials to approximate the area, trapetz rule below.

Matlab

x=0:0.001:pi
plot(x,sin(x))
Area=trapz(x,sin(x))

Python

import numpy as np
x=np.linspace(0,np.pi,99)
Area=np.trapz(x, np.sin(x))

[1] http://finance.mathharbour.com/2011/09/crash-analysis-idiosyncratic-risk-and.html

Scatter Plot, Python and Decision Making

2011-11-07T19:07:00.000-08:00

Investing is the most lucrative when you think as a business. Well, let's consider what it actually infers. Here some payouts with different strategies outlined for a commodity business. There is a fascinating connection between risk-management of a portfolio and corporate investing, more on page 213 [2]. There is always a cost for capital (p.211 [2]) so you should always allocate it as efficiently as possible, sometimes the cost is as a form of economic cost so harder to realize. Indexers are not meeting their benchmarks (p.207 [2]), a bit like private investors' under-performance. Good time to go back to basics.

The model is similar to the traditional efficient frontiers. Now the risk is not to do losses but create less profit than with other strategies. If you think it in terms of economic profit, they are intrinsically the same, now just with the difference to some case such as the worst or the average. I used VaR here as a risk measure, the VaR is the profit under which you get with the probability $p$. I think it is actually quite good indicator, it is often usual to make sure your strategies cover the fixed costs (so you do not run of out the business if a cycle lasts longer than expected).

The red points are first-order stochastic dominant. Interesting thing is that second-order stochastic dominance points are not the same as the first order. Now the pareto-optimal points are the same as the second order dominant points (15,12,8), try it with different cones -- but they are not always so! It is actually quite amazing.

P.s. if you want to dig more into this topic, please, review the convex analysis material covered by Standord University [1]. They are excellent, they formulate the whole concept of pareto optimality to new level, yes it scales to larger surfaces as well!

Data

$ cat Input_1
1  0 0
2  0.3 46 
3  0.73 67.45 
4  0.73 113.45
...

The Code

import numpy as np
import matplotlib.pyplot as plt

colors = {
    "great": "red",
    "poor": "green"
    }

class Point(object):
    "class for point objects"
    def __init__(self, id, x, y):
        self.id = id
        self.x = x
        self.y = y
        self.c = colors["poor"]

def read_files():
    "reads the files and return a list of points"
    the_points = []
    for line in open("Input_1"):
        try:
            L = line.strip().split()
            if len(L) != 3:
                continue
            id, x, y = int(L[0]), float(L[1]), float(L[2])
            point = Point(id, x, y)
            the_points.append(point)
        except:
            pass
    return the_points

the_points = read_files()


for point in the_points:

 #special poinst with different marker
 if (point.id == 8 or point.id == 12 or point.id == 14 or point.id == 15):
  plt.scatter(point.x,point.y, marker="^", c=colors['great'])
 else:
  plt.scatter(point.x,point.y, marker="o", c=point.c)

        plt.annotate( point.id, xy=(point.x,point.y))


plt.axis([0,1, 0, 300])
plt.show()

References
[1] http://www.stanford.edu/~boyd/cvxbook/bv_cvxslides.pdf
[2] Triumph of the Optimists: 101 Years of Global Investment Returns

PE10 and Python to Avoid Euphoria

2011-09-26T09:28:00.000-07:00

I want to continue discussions and help other people to investigate this issue, more here [2] and now categorized under Skeptic -index [1]. You can find the code below!

References

Boglehead -skeptic index here
investigations about PE10 here and here

# Description: plots historical PE10 graphs to avoid euphoria
# License: OpenBSD license
# Author: 10801


import urllib
import xlrd
from pylab import *

plot([1,2,3])
# Uncomment the below for the first time
# be kind and do not populate the Rob's server
# 
# url = "http://www.econ.yale.edu/~shiller/data/ie_data.xls"
# fXls = 'ie_data.xls'
# urllib.urlretrieve(url, fXls)

wb = xlrd.open_workbook(fXls)
sh = wb.sheet_by_name(u'Data')

x=sh.col_values(0)
y=sh.col_values(10)

subplot(211)
plot(x[128:-1], y[128:-1])
title('Graphs are based on R. Shiller data \n\n Historical PE10 versus Time')


#2nd plot 
p=sh.col_values(7)[8:-10]
d=sh.col_values(8)[8:-10]
e=sh.col_values(9)[8:-10]

y = [(ee+dd)/pp for ee,dd,pp in zip(e,d,p)]
x=sh.col_values(10)[8:-10]

subplot(212)
plot(x[128:-1], y[128:-1])
title('Real Return versus PE10')


show()

Vanguard Historical Fund Data With Python

2011-07-29T04:58:00.000-07:00

Vanguard does not offer data over the whole history of their funds, only annual data here and look how other companies provide the data here (much better!). Related discussion in Bogleheads.org here where people are unable to analyze the products due to the lack of data. I have contacted Vanguard about the issue but they provided no excuse. It is very short-sighted to let private investors not to know where they are investing in! I hope they will provide better service so private investors can concentrate on investing rather than awkward parsing.

If the parsing feels annoying, send Vanguard feedback here. Please, inform them with something like that you want historical data (not just annual data). The data is important to provide easy historical analysis about their funds. Ask for quality -- data over the whole history and in clean text -files (not in the obscure HTML -form currently) -- like their competitors such as iShares, Seligson and many other companies do.

Happy sailing on frontiers!

import urllib
import re

# Description: retrieves the annual data about a Vanguard fund
# so that you can more easily see historical data.
# Very bad parsing because Vanguard does not provide the 
# historical data over the whole history.
#
# Tested with Windows XP, Python 2.x and Urllib.
#
# License: OpenBSD License
# Author: 10801
#
# It saves the data to HTML -files where you still need to get it.
# TODO:
# - get the data from files to CSV -file
# - let user to control the years
# - fix the year -ending overgoing



startYear = "1993"
WorkingUrl = "https://personal.vanguard.com/us/funds/tools/pricehistorysearch?radio=1&results=get&FundType=VanguardFunds&FundIntExt=INT&FundId=0085&Sc=0&fundName=0085&fundValue=0085&radiobutton2=1&beginDate=07%2F28%2F2010&endDate=07%2F28%2F2011&year=#res"

test = WorkingUrl.split("&")
years= [x+int(startYear) for x in range(19)]
end = "&end=#res"

print "Data about Vanguard VTSMX over the years coming..." 
print years


print "URls are "

for year in years:
   res2 = re.sub("[0-9]{4}", str(year+1), test[-2])
   res = re.sub("[0-9]{4}", str(year), test[-3]) 
 
   myUrl =  "&".join(test[0:-3]) + "&" + res + "&" + res2 + end
   nameFile = str(str(year)+ "_" + str(year+1) + "_" + "VTSMX" + ".html")
 
   print ("File name is %s." % nameFile)
   urllib.urlretrieve(myUrl, nameFile)

P.s. you can manually use third-party services such as Google Finance to retrieve the historical data to files over the whole history but it is far from perfect.

AWK to find Benjamin Graham Valuation Metrics

2011-03-17T01:28:00.000-07:00

Benjamin Graham is famous for starting to quantify investment decisions. One of his metric is to find stocks with the lowest PE and PB multiplication -- so-called 22 rule. And, kosh, if you want to sort out the awful valuations like today with iShares S&P Small Cap 600 (DE), CS ETF (IE) on MSCI USA Small Cap, Vanguard Small Cap Growth ETF and Vanguard Mid-Cap Growth ETF (all above 30 with PE*PB) then use the command to find the popular 22 rule!

% awk -F"\t" '{if ($5*$6 < 22) print $5*$6 "\t", $1}' file | sort -n
12.155   CS ETF (IE) on MSCI Japan Small Cap 
13.3947  Dimensional Emerging Mkts Val B EUR
14.586   CS ETF (IE) on MSCI EMU Small Cap
14.949   iShares EURO STOXX Small (DE)
15.2724  Vanguard FTSE All-Wld ex-US SmCp Idx ETF
18.4146  iShares STOXX Europe Small 200 (DE)
19.206   CS ETF (IE) on MSCI UK Small Cap
20.559   Seligson Suomi indeksirahasto A

Edit Large Spreadsheets in VIM

2011-03-09T20:00:00.000-08:00

Large files returned nice segmentation errs with Excel, fed up to report bubble-wrap problems so turned to old good VIM. No more stupid time lag and wasted working hours. Some useful configurations to have your own "Spreadsheet program":

:set ts=20
:set list (:set nolist to off)
:set nowrap (:set wrap to off)

Stock Dump with Yahoo API

2010-03-24T04:24:00.000-07:00

Efficient.

Related. yahoo.py

>>> include yahoo
>>>print yahoo.get_all('BRCM')
{'stock_exchange': '"NasdaqNM"', 'market_cap': '16.948B', '200day_moving_avg': '30.1363', '52_week_high': '34.30', 'price_earnings_growth_ratio': '0.92', 'price_sales_ratio': '3.92', 'price': '34.20', 'earnings_per_share': '0.127', '50day_moving_avg': '31.2944', 'avg_daily_volume': '8453730', 'volume': '0', '52_week_low': '19.11', 'short_ratio': '1.00', 'price_earnings_ratio': '269.29', 'dividend_yield': '0.23', 'dividend_per_share': '0.08', 'price_book_ratio': '4.36', 'ebitda': '94.2M', 'change': '0.00', 'book_value': '7.853'}

Get Stock Prices

2010-03-20T14:48:00.000-07:00

Excellent, we may now begin our business. Abbut!

$ curl -s 'http://download.finance.yahoo.com/d/quotes.csv?s=goog&f=l1'

Bargain-hunting with Crontab

2010-03-20T12:20:00.000-07:00

I wish to engage your services.

$ crontab -l
   @daily source findBargains.sh

$ cat findBargains.sh
   export Url="www.JuniorParrotBargaining.com"
   mkdir -p ~/logs/Bargain-`date '+%F'` && \\
   cd ~/logs/Bargain-`date '+%F'` && wget --mirror -l 1 $Url