Code:

data _null_;
  y = today();
  x = intnx('month',y,-1);
  call symput('prev_month',strip(put(x,monname9.)));
  call symput('today',put(y,yymmddn8.));
run;

data _null_;
 set sashelp.vmacro;
 where value > "" and scope = "GLOBAL"
       and substr(name, 1, 3) not in ('SQL','SYS');
 put "Macro Variable: " name  "resolves to: " value;
run;

Output into the Log File:

1    data _null_;
2      y = today();
3      x = intnx('month',y,-1);
4      call symput('prev_month',strip(put(x,monname9.)));
5      call symput('today',put(y,yymmddn8.));
6    run;

NOTE: DATA statement used (Total process time):
      real time           0.02 seconds
      cpu time            0.00 seconds

7   data _null_;
8    set sashelp.vmacro;
9    where value > "" and scope = "GLOBAL"
10          and substr(name, 1, 3) not in ('SQL','SYS');
11   put "Macro Variable: " name  "resolves to: " value;
12  run;

Macro Variable: PREV_MONTH resolves to: March
Macro Variable: TODAY resolves to: 20120403
NOTE: There were 2 observations read from the data set SASHELP.VMACRO.
      WHERE (value>' ') and (scope='GLOBAL') and
      SUBSTR(name, 1, 3) not in ('SQL', 'SYS');
NOTE: DATA statement used (Total process time):
      real time           0.04 seconds
      cpu time            0.01 seconds

The where clause of the second data step can be edited to include the SQL and SYSTEM macro variables if need be and / or the scope can be change to print just the local macro variables with a macro (if you put the code inside of the macro).

This whole thing was spurred by my dabbling in d3.  It’s pretty much the most awesome and only visualization library you’ll need.   I’ve been slowly adding links to all of the example graphics I’ve put together to the d3 Examples Page.

There are three out there right now that’ll let you see what I’ve done so far…

• Last.fm_bar – Bar chart of a Last.fm user’s top 50 artists.  Calls the Last.fm api and parses the returned json.  Will update as the user scrobbles more tracks to Last.fm. Feel FREE to make fun of my music choices in the comments….
• Rolling 7 day Training Load - Area chart of my cumulative 7 day training load for miles run.    Much more useful at analyzing the total stress and cycles that weekly or monthly miles.
• Running Calendar - Calendar view of the daily miles run.  Similar to the R running calendar.   Adapted from the d3 example which was inspired by Rick Wicklin and Robert Allison’s winning poster (pdf).

There are a few data sets that I really want to work with but I need to do some serious data cleaning in SAS first.  Writing an json exporter from SAS is next up on the list when I get  a down moment.

The following  python script scrapes the data off of Daytum and outputs a .csv file.

# Program: scrapper.py
# Author : Andy McNeice
#
# Purpose: Scrapes my daytum running log and creates a csv file of the
# entries.
import re, urllib2, math, csv
from BeautifulSoup import BeautifulSoup
from datetime import datetime

# Daytum Url
log = urllib2.urlopen('http://daytum.com/amcneice/items/1183813')

# csv output
csv = csv.writer(file(r'data.csv','wb'))

# Constants
parse = [] # holder for Beautiful Soup finds
data = [] # final array

# Use BeautifulSoup to scrap to the Webpage
html = log.read()
soup = BeautifulSoup(html)

# Iterate through the soup to find the text matching the regular expressions
for text in soup.findAll(text=True):
  if re.search(r"\d{1,}\.\d{1,}", text):
    if not (re.search('Total',text) or re.search('DOC',text)):
      parse.append(text.strip())
  elif re.search(r"\d{2}\/\d{2}\/\d{4}", text):
      temp = re.compile(',').split(text)
      parse.append(datetime.strptime(temp[0].strip(),"%m/%d/%Y").strftime("%Y-%m-%d"))
      parse.append(temp[1].strip())

# Manipulate parsed data from long form into multidimensional array
for i in range(0, len(parse)/3):
 data.append([])
 for j in xrange(3):
   data[i].append(parse[i*3+j])

# Sort by day
data.sort(key=lambda x: x[1])

# csv headers for label data
data.insert(0, [])
data[0].append("Miles")
data[0].append("Time")
data[0].append("Time of Day")

# Write out csv to use data elsewhere
csv.writerows(data)

The Beautiful Soup code is a bit of a mess and will only work for quantitative data at the moment.  It was easier for me to use regular expressions to pluck the important info out of the soup that try to parse based on the underlying html tags.

Right now the script is kicked off by a cron job each hour to capture any updates.  Eventually I’ll have a pretty awesome dashboard (using d3) to monitor each of my training cycles.

I’m not a python guru, so if you’ve got suggestions on how to clean up the code,  I’m all ears.

Full code and output .csv available on github.

Unleash the following while inside the log’s directory while on any *nix based OS.

egrep '(ERROR:|WARNING)'

I save a lot of time scanning the 10,000+ line log files some of the automated jobs create.

%macro daysUntil(event, date);
  data _null_;
    diff = datdif(today(), input("&date.", anydtdte10.), 'act/act');
    call symput('days_until', strip(diff));
  run;

  %put There are &days_until days until &event;
%mend;

Macro call is straight forward:

%daysUntil(Christmas, 12/25/2011);
%daysUntil(Columbia_Marathon, 03/10/2012);

I could have used various intnx() commands to account for the differences in months, days and years.  There’s some good stuff out there on how to pull that off if it’s what you fancy.

The datdif() documentation is worth looking at.

As always, full source file is available here on bitbucket.

G. pointed out in the previous comments, I could have just used

options(gsubfn.engine = "R")

instead of going through the trouble of installing the tcltk binaries.

If you’ve got a happy distribution of R that comes with the working binaries, you can run sqldf and not have to worry about anything. Otherwise update the gsubfn and use the options setting.

When in doubt, check and actually READ (whoops..) the FAQ: http://code.google.com/p/sqldf/

I took a quick break from packing to condense the data to a csv and write up a very rough R script. Note: sqldf rocks but installing tcltk (if you have too) can be a bit of a pain. Look here for help with tcltk.

library(ggplot2)
library(sqldf)

salaries <- read.csv("2011Salary.csv", header=T, sep=",")

result <- sqldf('select
                  a.Conference,
                  sum(a.SchoolPay) / b.spc as avg_pay
                from
                  salaries  as a
                join
                  (select Conference, count(*) as spc
                  from salaries
                  where SchoolPay > 0
                  group by Conference) as b
                on
                  a.Conference = b.Conference
                group by
                  a.Conference')

chart <- qplot(result$Conference, result$avg_pay,
                geom="bar",
                stat="identity",
                fill = I("grey50"),
                main = 'Average Coaches Salary by Conference',
                xlab = 'Conference',
                ylab = 'Average Pay')

chart + opts(axis.text.x=theme_text(angle=-45))

Outputs the following

Most surprising result? PAC-12 coaches average ~ $400,000 less than the Big East.

Full code is available on bitbucket.

Edited per G.'s suggestion: sqldf rocks, tcltk can be tricky.

Generated by the following code:

#Sample Code based on example program at:
source(file = "calendarHeat.R")

run<- read.csv("log.csv", header = TRUE, sep=",")
sum(run$Distance)

date <- c()

for (i in 1: dim(run)[1]){

    if(run$DistanceUnit[i]== 'Kilometer'){
      miles <- c(miles,run$Distance[i] * 0.62)
    }
    else{
      miles <- c(miles,run$Distance[i])
    }
}
sum(miles)

calendarHeat(run$Date, miles, varname="Andy's Running Mileage")

You can see the definite shift in the length of my daily runs over the past 12 months.  Race times have dropped a chunk over that same period.

The heat map still needs some cleaning up,  using 26.2 as the max intensity is preventing the differences in my shorter runs from coming through.  Capping it at max of 15 (shortest of my ‘long’ runs) is probably a good idea to further show the shift in my daily mileage.

One of these days I need to transfer the 4,534.62 miles I have logged into notebooks to the computer.   Buying a Garmin Forerunner and having the data automatically updated to Runningahead.com  made it super easy to get the dataset.

Full code and source files are available on bitbucket:  RunningHeatMap

NOTE: There were 0 observations read from the data set WORK.TEST.
      WHERE 0 /* an obviously FALSE WHERE clause */ ;
NOTE: The data set WORK.TEST has 0 observations and 15 variables.
NOTE: DATA statement used (Total process time):
      real time           0.07 seconds
      cpu time            0.00 seconds

There’s a software developer with a sense of humor somewhere in Cary, NC.

For the example, I’m using the yr1001 data set from sashelp to avoid sending proc ttest a variable with qualitative data. Be forewarned, there are 505 variables within the data set, you may want to choose a set with fewer variables if you have performance concerns.

Variable names come from proc contents. If you want to iterate over each data set in the library instead, swap our sashelp.yr1000 with somelib._all_. Proc contents spits out a lot of information about each variable, but for this example, we only need the name.

*--- Get all the variable names from the data set ---;
proc contents data = sashelp.yr1001
		     out = names (keep = name)
		     noprint;
run;

Select each variable name into a unique macro variable. SAS is smart enough to only keep the variables it actually uses in memory. Count is the stop variable to keep the loops from going out of bounds.

*--- Put count and distinct names into macro variables ---;
proc sql;
 select
 	distinct name,
	count(distinct name)
 into
 	:name1 - :name9999,
	:count
 from
	names
 where
 	substr(name, 1, 1) = 'S';
quit;

Stamp the log with each variable. Useful for checking to make sure the code worked the way it should.

*--- List all of the variables in the log ---;
%macro loop();
 %do i = 1 %to &count.;
   %put &&name&i.;
 %end;
%mend;
%loop();

Perform some procedure over each variables. Be careful of variable types and syntax for any procedure run.

*--- Instead of listing to the log, perform some other proc ---;
%macro loop2();
 %do i = 1 %to &count.;
  proc ttest data = sashelp.yr1001;
   var &&name&i.;
  run;
 %end;
%mend;
%loop2();

That should get you started on splitting apart data sets and libraries using proc contents.  I’ve found the ability to send the proc contents results to a file the most important feature of proc contents by far.

Full program is available for download from bitbucket: varNamesToMacros.sas