From the site:

So, how realistic is the Social Security Trustees Middle Projection assumption on what the US GDP will be 27 years from now?

There is a (0.911121914 - 0.48167789) / 0.138352754 = 3.10397886261 standard deviation chance of it-or a chance of 0.000954684857590260, (about a tenth of a percent,) which is about one chance in a thousand, that the US GDP, 27 years from now, will be as bad, or worse, than their assumption.

A very conservative assumption, indeed.

Once our data feed series and we have something up and working again, one of the first things I will be talking about is the Brownian motion discussion which is all over the place on John's site.

Travis is a financial planner and in that little blurb he raises some excellent points - as he does in many sections of his site. He references many well known books and investing gurus.

It is a good little read to occasionally refer back to. Were I to jump up and down and wave my arms around more, trying to appear more confident and telling you how these formulas are a sure thing - maybe I'd have more readers.
But I'm instead going to keep going through the discussion of the analysis and then formal testing so that we can look and see what works and what doesn't.

I will warn you though that for the most part, you are going to see that technical indicators don't really work. There are some that do, but probably not in the way that you are thinking. We can get to this and more once the free data feed series is resolved and we can then write code to analyze the data that we have and test which technical indicators, if any, actually work.
There are many people who will jump up and down and wave their arms and say that if you are saying indicator XYZ doesn't work, then you aren't using it correctly. They very well may be right, but be careful those people aren't overconfident in nothing more than apophenia.

For those of you that are aware of RSS feeds, you may have subscribed to our site and been enjoying the full text feed that we offer. For those of you that haven't - now is your chance.

One of the upsides of an RSS feed is that you can provide a "light" path for your users who want to get your posts, but don't necessarily want to visit the site everyday.

One of the downsides is that the person/people running the site offering said feed doesn't get a good idea of how many people are reading the feed if they aren't visiting the site (as you may or may not have noticed, we have a few counter/tracker things in place to see what sort of traffic we get).

Well, thanks to the very cool Feedburner service, we can now track the RSS readership. What this means for you is really nothing - you don't have to worry about changing your links or RSS feed - I handled that on the server side with a mod_rewrite reference which points all RSS feeds towards Feedburner. It is also smart enough to format all RSS and Atom versions appropriately for your browser, and it even fancies it up a bit so that if you view the XML in a browser, it should be human readable.
I have also updated the ping service for Stensland Systems updates to notify Feedburner when there are new updates, so it should always have the proper and updated feed.

All in all, a great service.

If Feedburner stops behaving properly, I can simply remove the feed redirects and instantly you (assuming you are pointing to our local feeds), will go back to getting the feed we publish locally.

Also, as you can see at the top of this post, and also in the sidebar, there is a counter for the feed. As more people use it and refresh their feeds, then we can get a more accurate assessment of how many people are hitting the feed. (still not entirely clear how often that updates)

The nerd in me rejoices.

(I had taken it down some time back, but I think it is a great feature for any site - so please feel free to use it)

Two things to note from doing that:

1) When I pasted them into a text document, I am on a Mac, so I just double-clicked the old tickers file and pasted in the new data over that. When I double-clicked, it defaulted to TextEdit. When I saved that out, it must have worked some Mac "magic" on it, and it put a Mac newline character at the end ("^M"). When I ran the script, it saw that as one very long line with an illegal character and it deleted it all. I checked via the command line this was the case and then in order to fix this I pasted the data back in via BBEdit, which is much better about acting properly and not forcing Mac things around (there are many ways around this, that was the way I chose to go).

2) I ran the script again after that and we can now note that here are the before and after figures:
Before: 11780
After: 11000

So now we have an even 11K stock tickers that we are going to look at and try to get data for, and update their data at the end of the day (late at night actually).

So the next part coming in the series is going to be how to automate it all in a way that we can start getting data in, but we won't hammer the Yahoo servers. Readjusting our storage requirements to somewhere between 600 and 700MB.
In order to automate, we can essentially use the scripts which we already have, we just have to put in full paths so that when they are run by a cronjob, it will know where to look. Some might argue that had we put in the full paths from the start, we wouldn't need to go back and change this. That is completely and totally correct - but I know from my own experience on my server that this way is easier with this setup in terms of explaining it to someone else.

So stay tuned and hopefully learn something along the way.

I thought you'd be interested in a bit I found out. Using "exchange=O", returns the OTCBB stocks (which I'm interested in personally). However, there aren't verbose descriptions in the CSV file returned, as there are with the major exchanges. In other words, there are only two columns in the returned file - the company name and the ticker symbol.

That of course is referring to that link in this post. So in addition to NASDAQ (Q), NYSE (N), and AMEX (A), you can now get those OTC stocks as well with "O" (that is an "oh" and not a zero).

Thanks Dan for sending that in.

#getDailyCloseData.pl
use strict;
use LWP::Simple;
use Finance::Quote;

my $ticker = '';
my $file = '';
my $data = '';
my $count = 0;
#get all tickers in the stocks directory
my @allTickers = ;
my @updateThese = ();
my @possibleIssues = ();
my @args = ();
my @dataArr = ();
my $newRow = '';
my $test = '';
#note that we will want to pull off the "stocks" part 
#when we use that in the lookup part

sub convertDate($){
    #takes in one date form (m/d/yyyy)
    #spits out the Yahoo form (d-M-yy)
    my $inDate = shift;
    my @tmpArr = split('/',$inDate);#m,d,yyyy
    my $day = int($tmpArr[1]);
    my $month = int($tmpArr[0]) - 1;
    my $year = int($tmpArr[2]);
    my @monthStrings = qw(Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec);
    my $outDate = $day . '-' . $monthStrings[$month] . '-' . substr($year,2);
    
    return $outDate;
}

#now loop over those
for $file (@allTickers){
    #check to see if the file has more than 10 lines of data in it
    $ticker = substr($file,7);
    $count = `wc -l < $file`;
    die "wc failed: $?" if $?;
    chop($count);
    $count = int($count);
    if($count < 10){
        #not enough, try to get full history
        #we could call out to our external "getsingleHistory.pl" instead
        $data = get('http://itable.finance.yahoo.com/table.txt?s=' . $ticker . '&a=1&b=1&c=1998&ignore=.txt');
        #write it out to disk (clobber)
        open(OUTFILE, '+>' , 'stocks/' . $ticker) or die "Could not write out data file: $!\n";
        print OUTFILE $data;
        close(OUTFILE) or die "Could not close data file: $!\n";
        #it should get the newest data in there
        #print "More data for $ticker \n";#uncomment for debugging
    }
    else{
        #put it on the update list
        push(@updateThese, $ticker);
        #print "Update $ticker \n";#uncomment for debugging
    }
}

#now do the updating
#pass them into Finance::Quote
my $q = Finance::Quote->new;
$q->require_labels(qw/date open high low close volume/);
my %quotes  = $q->fetch("usa",@updateThese);

for $ticker (@updateThese){
    #check to see if there is data in there
    #if not, put it in the Issues array
    #otherwise, update the data
    $test = $quotes{$ticker,"date"};
    if(length($test) > 2){
        #2 seems odd, but will get around whitespace issues
        #in here we are good
        #open this ticker file, just reading in
        open(TICKER,"stocks/$ticker") or die "Bad things on open: $!\n";
        @dataArr = ();#clear it out from last time
        while(){
            chomp;#kill the newlines
            push(@dataArr, $_);
        }
        close(TICKER) or die "Bad things on close: $!\n";
        
        #create the new row
        #date,open,high,low,close,volume (ignore adj close)
        $newRow = '';#clear it out from last time in the loop
        $newRow .= convertDate($quotes{$ticker,"date"});#append on
        $newRow .= ',' . $quotes{$ticker,"open"};
        $newRow .= ',' . $quotes{$ticker,"high"};
        $newRow .= ',' . $quotes{$ticker,"low"};
        $newRow .= ',' . $quotes{$ticker,"close"};
        $newRow .= ',' . $quotes{$ticker,"volume"};
        
        splice(@dataArr,1,0,$newRow);#put it in the first position after the descriptors
        
        #now write it back out to the file, clobber it
        open(TICKER,'+>',"stocks/$ticker") or die "Could not open: $!\n";
        for(@dataArr){
            print TICKER "$_\n";
        }
        close(TICKER) or die "Could not close: $!\n";
    }
    else{
        #in here we are bad
        push(@possibleIssues,$ticker);
    }
}

#dump the possible issues to the screen
#if we are running it manually, then we will see them
#if we are running via cron, they will get emailed to us
for(@possibleIssues){
    print"$_\n";
}
#and done

1. We are totaly ignoring the "Adj(usted) Close" in the Yahoo data for now - we can get into that more later.
2. It would probably be good to add in code that would figure out what the date would be if it had current data and then if that date is already in the first line, don't bother trying to add more.
3. Another option is that we could manually hit Yahoo's site and pull down ticker data, then parse it ourselves, and then put that into files. This is still easier. I did it the other way another time and this is easier. Trust me.
4. Of course you could do all of this far more easily if you just bought a proper data feed, and I inevitably get 10 emails a week telling me that sort of thing. It would also have better/cleaner data which you could be sure to trust (there are many that don't trust public feeds and trusting your data is a big deal in the end with your analysis). But this particular post isn't about setting up that sort of data feed - instead it is just how to setup a free data feed.
5. Also note that yes we could put this into a SQL database instead of flat files. But for our purposes, that may just be overkill. There are several arguments I can think of as to why a SQL database would make several stages of the code easier, but it would also involve writing code to interface to the database. I may eventually write a post which discusses how to do exactly that, but for now, this particular one does it with flat files (well, CSV files of a sort I guess).
6. Technically if you wanted to be impolite to Yahoo, you could cheat a bit and write a script that iterates over the stock list and dumps them all out to files named after the stock into our stocks dir. You can put whatever you want in there, as long as it is less than 10 lines (a single space would be just fine). Then run this script and it will go through and get all of the updates for you as part of the process. It would be slow and it would be getting a lot of data, but it would be a way to force your "database" to fill.
7. This code is something I have written before, lost, and now am imporving on from what I have in my head from last time. I work on it in spurts and am not trying to make it perfect. I am trying to make it work. Over time we can update/tweak it so that it is much better and easier to read, but for now, these are really just rough examples. So while you are free to email me and tell me about some coding style, I am likely to ignore it until I spend more time later to do anything cleaned up (whcih may be never on here).

• Account for errors. Sometimes when the data is pulled from the server, it will be old data. Technically included in here we also need to account for American holidays, during which the exchange will be closed (we will treat "close early" holidays as a normal day, for good or bad). Or if data isn't updating on a stock, that is probably a sign that it has delisted. We will want to alert ourselves to that so that we can then lookup those tickers and we can see what they changed their tickers to or if they delisted entirely.
• Web access. Currently our scripts are designed from a command line interface perspective. This is the interface of the gods, the interface through which we live the true experience. But perhaps we want to add pretty colors and pictures of puppies - and why the hell not? Or perhaps we want to share our results with friends? Then onward and upward - with a web interface.
• Automate them all. Set it up so that they are run via a cronjob - this will require being far more careful about the paths.
• Are there indexes in there? If not, we will want to look to see how Yahoo thinks of indexes and then add those into our data feed - with historical data and updates. There is certainly analysis code which will be glad to have some indexes to look at (if nothing more than for comparisons).
• It would be nice to get this into a MySQL database and then use that to pull data out of and see if that is faster/easier - that way it would answer all of the questions I get from people wanting to know if the text file method is worse/better than a SQL database method.
• The site currently doesn't appear to be wide enough to easily display the code without causing some issues. So the site could use a redesign so that it can be wider, and it would also be good to offer up the scripts in a single zipfile so that they can be downloaded and edited easily.

So I am going to have to change it to this now:
"http://itable.finance.yahoo.com/table.txt?s=TICKER&a=1&b=1&c=1998&ignore=.txt" - it currently appears to be letting me use that.

I will go and add notes to the other posts now so that future searches will see it.

Apophenia is the experience of seeing patterns or connections in random or meaningless data. The term was coined in 1958 by Klaus Conrad, who defined it as the "unmotivated seeing of connections" accompanied by a "specific experience of an abnormal meaningfulness".

Conrad originally described this phenomenon in relation to the distortion of reality present in psychosis, but it has become more widely used to describe this tendency in healthy individuals without necessarily implying the presence of neurological or mental illness.

In statistics, apophenia would be classed as a Type I error. Apophenia is often used as an explanation of paranormal and religious claims. It has been suggested that apophenia is a link between psychosis and creativity.

#getSingleHistory.pl
use strict;
use LWP::Simple;#might eventually need a UserAgent if Yahoo starts requiring it 

my $stockDir = 'stocks/';#the directory we are writing the stocks into
my $ticker = shift(@ARGV);#incoming from the command line.
my $data = '';

#now we will grab the data
print "$ticker\n";#comment out this line if you don't want to see the progress
$data = get('http://itable.finance.yahoo.com/table.csv?s=' . $ticker);
#write it out to disk (clobber)
open(OUTFILE, '+>' , $stockDir . $ticker) or die "Could not write out data file: $!\n";
print OUTFILE $data;
close(OUTFILE) or die "Could not close data file: $!\n";

#and done

#cleanup.pl
use strict;

my $count = 0;
my $file = '';
my @questionableFiles = ();

#get all of the stock filenames
my @filenames = ;
#iterate over them
for $file (@filenames){
    #get their linecount
    $count = `wc -l < $file`;
    die "wc failed: $?" if $?;
    chop($count);
    $count = int($count);
    if($count < 10){
        #if in here, we should look into it
        #get all text after "stocks/"
        $file = substr($file,7);       
        #put it into an array
        push(@questionableFiles, $file);
    }
}

#now dump that out to a file
open(OUTPUT,'+>', 'questionableFiles.txt') or die "Bad things man: $!\n";
for(@questionableFiles){
    print OUTPUT "$_\n";
}
close(OUTPUT) or die "Couldn't close the file: $!\n";

#and done

1. Load that file so that we have each ticker in a spot in an array. Then iterate over that array and get all of its historical data. For each ticker, take its data and save it into a file named with the ticker symbol.
2. We will then want code that will get the end of day data for each ticker and update the data files, and we will want to automate this to happen late at night.

#getAllHistoricData.pl
use strict;
use LWP::Simple;#might eventually need a 
                        #UserAgent if Yahoo starts requiring it 

my $runCount = 100;#this is how many tickers are going to try to get data for
#we want to keep that fairly low since it is rude to hit Yahoo overly hard

my $stockDir = 'stocks/';#the directory we are writing the stocks into
my @allTickers = ();
my $ticker = '';
my $data = '';

#open the tickers file
open(TICKERS, 'needData.txt') or die "Could not open needData.txt: $!\n";
while(){
    chomp;
    push(@allTickers, $_);
}
close(TICKERS) or die "Could not close needData.txt: $! \n";

#check that the number of tickers in the allTickers array is 
#greater than or equal to runCount
#if not, then make runCount equal to the length of allTickers
if(scalar(@allTickers) < $runCount){
    $runCount = scalar(@allTickers);
}

#now we will grab the data
while($runCount > 0){
    $ticker = pop(@allTickers);
    print "$ticker\n";#comment out this line if 
                              #you don't want to see the progress
    $data = get('http://itable.finance.yahoo.com/table.txt?s=' . $ticker . '&a=1&b=1&c=1998&ignore=.txt');
    #write it out to disk (clobber)
    open(OUTFILE, '+>' , $stockDir . $ticker) or die "Could not write out data file: $!\n";
    print OUTFILE $data;
    close(OUTFILE) or die "Could not close data file: $!\n";
    --$runCount;
}

#now dump out the tickers that we have left into the needData.txt file
#clobber - meaning '+>' 
#which will overwrite the existing data in that file with this
open(TICKERS,'+>','needData.txt') or die "Could not open needData.txt: $!\n";
for(@allTickers){
    print TICKERS "$_\n";
}
close(TICKERS) or die "Could not close needData.txt: $!\n";

#and done

1. Get all of the ticker symbols.
2. Put those tickers into a file (in alphabetical order) so that we can later grab this file and use it to know which tickers to look up for data.
3. Load that file so that we have each ticker in a spot in an array. Then iterate over that array and get all of its historical data. For each ticker, take its data and save it into a file named with the ticker symbol.
4. We will then want code that will get the end of day data for each ticker and update the data files, and we will want to automate this to happen late at night.

#stripTickers.pl
use strict;

my @tickersIn = ();
my @tickersOut = ();
my $count = 0;

#get all of the tickers into an array
open(TICKERS, 'tickers.txt') or die "Could not open tickers.txt: $!\n";
while(){
    chomp;
    push @tickersIn, $_;
    $count++;
}
close(TICKERS) or die "Could not close tickers.txt: $!\n";

print "Before: $count\n";
$count = 0;

#now iterate over those and if the ticker doesn't contain our bad chars, 
#then we keep it
for my $ticker (@tickersIn){
    #if it doesn't have the chars, then we add it to the keepers
    #check for "/"
    if(!($ticker =~ /\//) && !($ticker =~ /\^/)){
        push @tickersOut, $ticker;
        $count++;
    }
}

print "After: $count\n";

#rename the old tickers.txt file to tickers_old.txt
rename 'tickers.txt', 'tickers_old.txt';

#dump back out (clobber)
open(TICKERS,'+>','tickers.txt') or die "Could not open tickers.txt: $!\n";
for(@tickersOut){
    print TICKERS "$_\n";
}
close(TICKERS) or die "Could not close tickers.txt: $!\n";

#and done

First off, I have to admit that one bleary eyed morning I stupidly typed in the wrong command on my server and wiped out my stock database which I use for analysis.
As a result, I will need to rebuild that from scratch and setup the updating process. This actually works out well because I had been writing up something along those lines on here anyway. Since I will be going over the process from scratch on my end and it might change slightly from what I had before - I am just going to start explaining it again on here - from scratch.

Also, this is running an older version of MovableType, so I will be upgrading to the newest version at some point "soon" (feasibly even tonight). This shouldn't impact you as the end user in anyway - but if you see anything odd, it might be that I am upgrading the site.

That's it! More soon...

Do note that Mandelbrot has a book coming out soon, and this is taken from the book, so it is highly feasible that this is essentially a way for him to garner attention towards said book.

Since each financial institution feels, and perhaps rightfully so I should add, that their own techniques are unique and what makes them money - they aren't likely to want to join into a global effort to resolve something that wouldn't give them any sort of edge over anyone else.