Stats and things

Access Your Google Location History in R

2016-04-25T14:31:00.000-05:00

For those of us with Google location history turned on on their android phones, the Google maps page at maps.google.com/locationhistory containing our location history is quite interesting. Interesting enough that I wanted to try to pull it into R for further analysis. You can use Google Takeout to download your entire location history, or just about any data associated with your Google account. I was hoping to come across something a little lighter.

So, first off, location history data is not available from any of Google's API services. That leaves us with few options other than to simulate a browser session in R using rvest to get the data that way. There's a Stackoverflow post about what structuring a URL to get the desired KML data that I used as my guide.

Ever since creating my first R package with the help of Hilary Parker's blog post on creating R packages, I've created a bunch of them. I mostly keep them to myself, but even just for personal use, a structured R package makes things easier for me to document my own code and reuse it. Any how, I created a small package to download Google location history called GoogleLocationHistory.

I'll give an example of how it works via a real life example. I flew out to California in 2014 to go to the UseR conference at UCLA. In a bid to visit all MLB ballparks, I flew out to San Diego first, to go to a Padres game. Then I drove up the coast to a rental condo to go to the conference. I made it to an Angels and Dodgers game on this trip too. All three teams were home during the conference! The problem was that I didn't have the electronic toll thingie in my car and by the time I got back to my room to pay the toll online, I had no idea what toll roads I had taken. This info is required to pay the tolls online. So, I fired up Google location history and was able to figure it out that way. Creepy, but at least it served a use for me. Let's look at the data from that day…

Note that the “login” function assumes Google's two factor authentication is turned on. This is the only thing I can test given that it's turned on for my account. The “login” function will ask for the six digit code that is texted to you by Google when you attempt to login.

library("devtools")  # this is needed to install my package from Github
install_github("corynissen/GoogleLocationHistory")
library("GoogleLocationHistory")

sess <- login(username="corynissen@gmail.com", password="mypassword")
df <- location_history(session=sess, date="2014-06-29")

Here's a look at what the data looks like. The “when” and “coord” are data returned from Google. The library attempts to parse this information and creates the “lon”, “lat”, and “time” columns.

tail(df)

##                              when                     coord       lon
## 372 2014-06-29T20:52:33.211-07:00 -118.4767475 34.0597392 0 -118.4767
## 373 2014-06-29T20:53:19.280-07:00 -118.4752259 34.0587244 0 -118.4752
## 374 2014-06-29T20:54:04.294-07:00 -118.4721347 34.0600617 0 -118.4721
## 375 2014-06-29T20:55:41.607-07:00 -118.4711439 34.0619441 0 -118.4711
## 376 2014-06-29T20:56:54.865-07:00  -118.471275 34.0635299 0 -118.4713
## 377 2014-06-29T20:56:54.865-07:00    -118.471017 34.06378 0 -118.4710
##          lat                time
## 372 34.05974 2014-06-29 20:52:33
## 373 34.05872 2014-06-29 20:53:19
## 374 34.06006 2014-06-29 20:54:04
## 375 34.06194 2014-06-29 20:55:41
## 376 34.06353 2014-06-29 20:56:54
## 377 34.06378 2014-06-29 20:56:54

Maybe it would be cool to map the data…

library("ggmap")
maptile <- get_map(location=c(lon=((max(df$lon) + min(df$lon)) / 2),
                            lat=((max(df$lat) + min(df$lat)) / 2)), zoom=4)
mymap <- ggmap(maptile) + geom_path(data=df, aes(x=lon, y=lat), size=1.1) +
  theme_void()
mymap

I created this to check out my own data, but if it can be useful for anybody else, that's great too. Let me know if you are able to do anything cool with it.

My Vehicle History 1996 - 2015

2016-01-20T12:06:00.002-06:00

Here's a post under the "and things" category in the Stats and things blog... it's my vehicle history. Note, only the Lexus picture is my actual vehicle. The others are the closest thing I could find in a quick google search.

1985 Toyota Cressida - Bought from a mechanic friend of my uncle. It was his Mom's car, she blew the engine. He fixed it up and sold it to me for $2,800ish. Good car until air conditioning, transmission problems and rusty suspension components made it too expensive to fix.

1986 Mercury Grand Marquis - Bought this from family. It was my great Grandpa's car from Florida. It was beige. It was old, but I got it with 78,000 miles. I put a subwoofer in the trunk. Somebody must have heard it, because they broke into it and stole everything. It was a crushing blow during my summer home from college. It ran great for me, but I wanted something a little more modern.

1994 Ford Explorer - My Mom's old car. She sold it to me for $5,000. I got a loan for most of that. It was awesome and got me through grad school. When the air conditioning went out and the forecast was predicting weather in the 90's, I traded it in for the Civic.

2003 Honda Civic ex - My first new vehicle. It was a black EX with a moon roof and 5-speed. I put 117,000 miles on it and sold it to a guy to give to his son for his 16th birthday. I had to have a strut replaced and the head gasket, timing belt, water pump. Very reliable vehicle.

2005 Nissan Xterra - We bought this new for my wife. It was the "off road" package in yellow. The off road package had a locking rear diff, but Erica wanted that package for the moulded stripe on the doors. Another very reliable vehicle. I think all I did on this one was a u-joint. We put 155,000 miles on it and sold it to a guy from down south. He flew up here to drive it back home.

1998 Chevy 2500 6.5L - I bought this when I had my Civic. I had a house and it was tough to get house stuff in the Civic. It was the 6.5L diesel engine with an extended cab and 8 foot bed. It was long and slow, but it got 18mpg. Things started to rust on this one. First, a rear brake line broke in the cold. Then the brake booster started to leak. Then the fuel tank rusted through and leaked diesel in my driveway. I sold it to a young kid with all of those things fixed at around 190,000 miles. Still ran great.

2004 Volkswagen Touareg - Got this to replace the Xterra for Erica. It was $15,500 with just 33,000 miles on it. That's because the first year Touaregs were riddled with problems. But it had all the options, V8, air ride, moon roof... it was awesome. I bought a warranty and had to use it when the vehicle wouldn't shut off even after we took the key out. A new ECU fixed that, but that was the only problem we had with it. Sold it at 80,000 miles to a guy from down south when the warranty ran out and the timing belt was due to be changed.

2010 Toyota Tundra - My current truck. I wanted the 5.7L engine, 4x4, and double cab. Ended up getting that in a base model work truck. Vinyl seats, rubber floor, and column shifter. Perfection. Bought it for $22,000 with 32,000 miles from a dentist in Michigan. He was coincidentally buying a motor home in Illinois, so I met him and bought it from him with cash at the motor home shop. It's had a new front differential and a whole new bed replaced under warranty. Still going at 88,000 miles

2011 Volkswagen Jetta Sportwagen TDI - I flew to Washington DC to buy this one with 6,000 miles on it for Erica to replace the Touareg. It gets 40mpg with the TDI engine. Unfortunately, the EPA has found out that VW cheated on the emissions tests with this engine and it's value has tanked. We are waiting to see what they do with the recall / buy back. It's at 111,000 miles now with only a couple minor warranty repairs.

1992 Lexus SC300 - Bought this from my uncle in pristine condition. It's a factory 5-speed with around 125,000 miles. I am saving money to add a turbo to it and bump up the horsepower from a stock 225hp to somewhere in the 400hp range.

Creating Styled Google Maps in ggmap

2015-05-27T16:01:00.000-05:00

In R, my go to package for creating maps is ggmap. Using this, one can create maps using a variety of services, including Google maps. I just recently discovered the ability to create a styled version of a Google map. Let's go through this process to create a black and white map of Chicago, with the parks shown in red.

First, go to the Styled Maps Wizard and type in “Chicago IL”.

You are shown the stock Google map of Chicago, with some sliders to change various attributes of the map.

Let's take the “saturation” slider down to -100 and bump up the “gamma” slider to .5.

Then, we can work on making the parks red. First, click the “Add” button on the right side.

Next, on the left side at the top, choose “Point of Interest”, then “Park”.

Then, click the “Color” box and type in “#ff0000” into the box to specify red for the parks You should see the parks highlighted in red now.

The last thing we need is to grab the JSON for these styles. On the right side, at the bottom of the “Map Style” box, click “Show JSON”.

With this JSON, we can create the map in ggmaps.

library("RJSONIO")
library("ggmap")
library("magrittr")
style <- '[
  {
    "stylers": [
      { "saturation": -100 },
      { "gamma": 0.5 }
    ]
  },{
    "featureType": "poi.park",
    "stylers": [
      { "color": "#ff0000" }
    ]
  }
]'
style_list <- fromJSON(style, asText=TRUE)

create_style_string<- function(style_list){
  style_string <- ""
  for(i in 1:length(style_list)){
    if("featureType" %in% names(style_list[[i]])){
      style_string <- paste0(style_string, "feature:", 
                             style_list[[i]]$featureType, "|")      
    }
    elements <- style_list[[i]]$stylers
    a <- lapply(elements, function(x)paste0(names(x), ":", x)) %>%
           unlist() %>%
           paste0(collapse="|")
    style_string <- paste0(style_string, a)
    if(i < length(style_list)){
      style_string <- paste0(style_string, "&style=")       
    }
  }  
  # google wants 0xff0000 not #ff0000
  style_string <- gsub("#", "0x", style_string)
  return(style_string)
}

style_string <- create_style_string(style_list)
mymap <- ggmap(get_googlemap("chicago", size=c(800,800), style=style_string), extent="device")
ggsave(filename="pics/mymap.png", width=8, height=8)

I wrote the helper function “create_style_string” to take the style parameters given from Google as JSON, and create a string to use in the API request. Put that style_string in the “style” parameter in get_googlemap() and you get your map back.

The possibilities are endless to create maps with colored placenames, water, places of interest, etc. Very cool stuff.

fitbitScraper 0.1.2 Now on CRAN

2015-03-26T14:23:00.000-05:00

I've updated the fitbitScraper to 0.1.2 with the help of a friend that gave me access to his fitbit account so I could work out the heart rate API endpoints (thanks Shane!). This is a package that enables you to read daily or intraday data from Fitbit into R for graphing and analysis.

Here's some examples:

I'll download all my steps for last Saturday, March 21, 2015, then make a graph similar to how it appears on the fitbit dashboard. As you can see, I spent some time playing basketball in the morning and pretty much took it easy the rest of the day. Note that the function name has changed to get_intraday_data() from get_15_min_data() in version 0.1.1.

# install.packages("fitbitScraper")  
library("fitbitScraper")
# just reading from file to hide pw and to make .Rmd document to work...
creds <- readLines("pw.txt")
cookie <- login(email=creds[1], password=creds[2]) 
df <- get_intraday_data(cookie, what="steps", date="2015-03-21")  
library("ggplot2")  
ggplot(df) + geom_bar(aes(x=time, y=steps, fill=steps), stat="identity") + 
             xlab("") +ylab("steps") + 
             theme(axis.ticks.x=element_blank(), 
                   panel.grid.major.x = element_blank(), 
                   panel.grid.minor.x = element_blank(), 
                   panel.grid.minor.y = element_blank(), 
                   panel.background=element_blank(), 
                   panel.grid.major.y=element_line(colour="gray", size=.1), 
                   legend.position="none")

Now, let's take a look at the heart rate data. This is Shane's data… I have no idea what he was doing on this day. Note that whereas intraday data for steps is given in 15 minute intervals, fitbit provides the heart rate data in 5 minute intervals.

# just reading from file to hide pw and to make .Rmd document to work...
creds <- readLines("pw2.txt")
cookie <- login(email=creds[1], password=creds[2]) 
df <- get_intraday_data(cookie, what="heart-rate", date="2015-03-20")
library("ggplot2")  
ggplot(df) + geom_bar(aes(x=time, y=`heart-rate`, fill=`heart-rate`,
                          colour=`heart-rate`), stat="identity") + 
             xlab("") +ylab("heart-rate") + 
             theme(axis.ticks.x=element_blank(), 
                   panel.grid.major.x = element_blank(), 
                   panel.grid.minor.x = element_blank(), 
                   panel.grid.minor.y = element_blank(), 
                   panel.background=element_blank(), 
                   panel.grid.major.y=element_line(colour="gray", size=.1), 
                   legend.position="none")

And finally, the daily heart rate data comes with three data points for each day. Zone1, zone2, and zone3 presumably represent the time in a given heart rate zone. I suspect they correspond to “fat burn zone”, “cardio zone”, and “peak zone” respectively. More information on fitbit heart rate zones is available here.

# just reading from file to hide pw and to make .Rmd document to work...
creds <- readLines("pw2.txt")
cookie <- login(email=creds[1], password=creds[2]) 
df <- get_daily_data(cookie, what="getTimeInHeartRateZonesPerDay", 
                     start_date="2015-02-17", end_date="2015-02-23")
library("reshape")
df <- melt(df, id="time", variable_name="zone_num")
library("ggplot2")  
ggplot(df) + geom_bar(aes(x=time, y=value, fill=zone_num), 
                      position=position_dodge(), stat="identity") + 
             xlab("Date") +ylab("Minutes in Zone") + 
             theme(axis.ticks.x=element_blank(), 
                   panel.grid.major.x = element_blank(), 
                   panel.grid.minor.x = element_blank(), 
                   panel.grid.minor.y = element_blank(), 
                   panel.background=element_blank(), 
                   panel.grid.major.y=element_line(colour="gray", size=.1))

I'd love to work out the details for the GPS in the fitbit Surge. Unfortunately, I don't have one. If you are interested in working with me on integrating that data into the package, hit me up… corynissen@gmail.com

R Package to Download Fitbit Data

2015-01-22T20:52:00.000-06:00

Edit 4/8/2015: Updated to use newer functions…
Edit 3/26/2015: fitbitScraper 0.1.2 is now on CRAN with new heart rate data capabilities.
Edit 2/2/2015: fitbitScraper 0.1.1 is now on CRAN

Fitbit is a device that tracks your daily activity. It's basically a pedometer, but it does a little more. It has an altimeter, so it can count flight of stairs climbed. It can detect your sleeping activity and give you a read on how often you are tossing and turning. Any how, I received a Fitbit One for Christmas and have been unexpectedly into this thing. I absolutely did not expect to be motivated by monitoring my fitness activity. It's been fun.

The problem is that Fitbit doesn't allow you to export your data under their “free” data tracking tier. It costs $50 a month to upgrade to “premium” to allow for data export. It's my data, I should be able to get it. So, I made an R package to do just that.

fitbitScraper is my new R package, hosted on Github for now, to download Fitbit data. It uses the internal API that fitbit uses to populate your dashboard. Here, they give data as granular as every 15 minutes. Also, there's daily aggregate data available. Basically, the package logs into fibit.com using your email / password, and is returned a cookie. Using that cookie, the internal API is accessible. For the time being, I can only get that cookie if you use email / password to login. Facebook and Google login are available, but I haven't put any effort into figuring out how to get a cookie if one of those methods are used to login. One could copy and paste their cookie information into R and use this package that way…

Here's a quick example of downloading and graphing step data for a given day. On this particular day, I played basketball from about 6am until 9am.

install.packages("fitbitScraper")  
library("fitbitScraper")
# just reading from file to hide pw and to make .Rmd document to work...
mypassword <- readLines("pw.txt")
cookie <- login(email="corynissen@gmail.com", password=mypassword) 
df <- get_intraday_data(cookie, what="steps", date="2015-01-10")  
library("ggplot2")  
ggplot(df) + geom_bar(aes(x=time, y=steps, fill=steps), stat="identity") + 
             xlab("") +ylab("steps") + 
             theme(axis.ticks.x=element_blank(), 
                   panel.grid.major.x = element_blank(), 
                   panel.grid.minor.x = element_blank(), 
                   panel.grid.minor.y = element_blank(), 
                   panel.background=element_blank(), 
                   panel.grid.major.y=element_line(colour="gray", size=.1), 
                   legend.position="none")

Here's an example of the daily data.

df <- get_daily_data(cookie, what="steps", start_date="2015-01-13", end_date="2015-01-20")  
library("ggplot2")  
ggplot(df) + geom_bar(aes(x=time, y=steps), stat="identity") + 
             xlab("") +ylab("steps") + 
             theme(axis.ticks.x=element_blank(), 
                   panel.grid.major.x = element_blank(), 
                   panel.grid.minor.x = element_blank(), 
                   panel.grid.minor.y = element_blank(), 
                   panel.background=element_blank(), 
                   panel.grid.major.y=element_line(colour="gray", size=.1), 
                   legend.position="none")

I have to give a should out to this guy for inspiration. I don't know perl, so I couldn't really use anything here, but it gave me confidence that I could figure it out.

I hope this is of some use to some of you R hackers out there…

Using rvest to Scrape an HTML Table

2015-01-08T09:36:00.000-06:00

I recently had the need to scrape a table from wikipedia. Normally, I'd probably cut and paste it into a spreadsheet, but I figured I'd give Hadley's rvest package a go.

The first thing I needed to do was browse to the desired page and locate the table. In this case, it's a table of US state populations from wikipedia. Rvest needs to know what table I want, so (using the Chrome web browser), I right clicked and chose “inspect element”. This splits the page horizonally. As you hover over page elements in the html on the bottom, sections of the web page are highlighted on the top.

Hovering over the blue highlighted line will cause the table on top to be colored blue. This is the element we want. I clicked on this line, and choose “copy XPath”, then we can move to R.

First step is to install rvest from CRAN.

install.packages("rvest")

Then we it's pretty simple to pull the table into a dataframe. Paste that XPath into the appropriate spot below.

library("rvest")
url <- "http://en.wikipedia.org/wiki/List_of_U.S._states_and_territories_by_population"
population <- url %>%
  html() %>%
  html_nodes(xpath='//*[@id="mw-content-text"]/table[1]') %>%
  html_table()
population <- population[[1]]

head(population)

##   Rank in\nthe Fifty\nStates,\n2014
## 1                           !000001
## 2                           !000002
## 3                           !000003
## 4                           !000004
## 5                           !000005
## 6                           !000006
##   Rank in all\nstates\n& terri-\ntories,\n2010 State or territory
## 1                                      !000001         California
## 2                                      !000002              Texas
## 3                                      !000004            Florida
## 4                                      !000003           New York
## 5                                      !000005           Illinois
## 6                                      !000006       Pennsylvania
##   Population estimate for\nJuly 1, 2014 Census population,\nApril 1, 2010
## 1                            38,802,500                        37,253,956
## 2                            26,956,958                        25,145,561
## 3                            19,893,297                        18,801,310
## 4                            19,746,227                        19,378,102
## 5                            12,880,580                        12,830,632
## 6                            12,787,209                        12,702,379
##   Census population,\nApril 1, 2000 Seats inU.S. House,\n2013–2023
## 1                        33,871,648                        !000053
## 2                        20,851,820                        !000036
## 3                        15,982,378                        !000027
## 4                        18,976,457                        !000027
## 5                        12,419,293                        !000018
## 6                        12,281,054                        !000018
##   Presi-\ndential\nElectors\n2012–\n2020
## 1                                !000055
## 2                                !000038
## 3                                !000029
## 4                                !000029
## 5                                !000020
## 6                                !000020
##   2014 Estimated pop.\nper\nHouse seat
## 1                              732,123
## 2                              748,804
## 3                              736,789
## 4                              731,342
## 5                              715,588
## 6                              710,401
##   2010 Census pop.\nper\nHouse\nseat[4] 2000 Census pop.\nper\nHouse\nseat
## 1                               702,905                            639,088
## 2                               698,487                            651,619
## 3                               696,345                            639,295
## 4                               717,707                            654,361
## 5                               712,813                            653,647
## 6                               705,688                            646,371
##   Percent\nof total\nU.S. pop.,\n2014[5]
## 1                                 12.17%
## 2                                  8.45%
## 3                                  6.24%
## 4                                  6.19%
## 5                                  4.04%
## 6                                  4.01%

There's some work to be done on column names, but this is a pretty pain free way to scrape a table. As usual, a big shout out to Hadley Wickham for making this so easy for us.

geocodeHERE 0.1 is on CRAN

2014-11-06T14:07:00.000-06:00

In my previous blog post, I detailed how I created my first R package called geocodeHERE. This package is a convenient wrapper for Nokia's HERE geocoding API. The cool thing about this API is that it allows for bulk geocoding. So, instead of doing n API calls to geocode n addresses, you can do it with just a couple API calls. Also, you can run 10,000 API calls per day vs. Google's 2,500.

Any how, I went through the process to submit the package to CRAN and it was accepted. It took me 3 attempts, but I did it and it should be replicated across the mirrors by now. You can check it out on CRAN here.

Here's how it works, starting with downloading and installing the package (you'll need the httr package installed)…

install.packages("geocodeHERE", repo="http://cran.rstudio.com",
                 dependencies=TRUE)

## Installing package into ‘/home/c/R/x86_64-pc-linux-gnu-library/3.1’
## (as ‘lib’ is unspecified)
## trying URL 'http://cran.rstudio.com/src/contrib/geocodeHERE_0.1.tar.gz'
## ...
## * DONE (geocodeHERE)

Now, we can try to run some simple queries. Given an address or place name, it will return the latitude, longitude, or NA if it can't find anything

library(geocodeHERE)  
geocodeHERE_simple("wrigley field chicago IL")

## $Latitude
## [1] 41.95
## 
## $Longitude
## [1] -87.65

geocodeHERE_simple("navy pier chicago IL")

## $Latitude
## [1] 41.94
## 
## $Longitude
## [1] -87.7

geocodeHERE_simple("the bean chicago IL")

## [1] NA

That's kinda boring though. There exists functions in other packages to do this same thing, namely ggmap::geocode(). What's interesting about Nokia HERE's API is that you can do gecoding of many addresses in bulk.

data(chicago_landmarks)
addresses <- chicago_landmarks[,"Address"]
# tack on "chicago IL" to the end of these addresses
addresses <- paste(addresses, "chicago IL")
# make a dataframe with an id column so you can match the lat, lngs back
addresses_df <- data.frame(id=1:length(addresses), addresses=addresses)
str(addresses_df)

## 'data.frame':    375 obs. of  2 variables:
##  $ id       : int  1 2 3 4 5 6 7 8 9 10 ...
##  $ addresses: Factor w/ 374 levels "1001 W. Belmont Avenue chicago IL",..: 127 149 160 161 345 290 312 206 318 203 ...

address_str <- df_to_string(addresses_df)
request_id <- geocodeHERE_batch_upload(address_string = address_str,
                                       email_address = "test@test.com")
# wait about 15 seconds... status should go from "running" to "completed"
Sys.sleep(15)
geocodeHERE_batch_status(request_id)

## [1] "completed"

# download the data
geocode_data <- geocodeHERE_batch_get_data(request_id)
# match it back to your original addresses dataframe
addresses_df <- merge(addresses_df, geocode_data, by.x="id", by.y="recId", all.x=T)
str(addresses_df)

## 'data.frame':    375 obs. of  16 variables:
##  $ id              : int  1 2 3 4 5 6 7 8 9 10 ...
##  $ addresses       : Factor w/ 374 levels "1001 W. Belmont Avenue chicago IL",..: 127 149 160 161 345 290 312 206 318 203 ...
##  $ SeqNumber       : int  1 1 1 1 1 1 1 1 1 1 ...
##  $ seqLength       : int  1 1 1 1 1 1 1 1 1 1 ...
##  $ displayLatitude : num  41.9 41.9 41.9 41.9 41.8 ...
##  $ displayLongitude: num  -87.6 -87.6 -87.6 -87.6 -87.6 ...
##  $ houseNumber     : int  300 333 35 3600 NA 6901 860 4605 9326 4550 ...
##  $ street          : chr  "W Adams St" "N Michigan Ave" "E Wacker Dr" "N Halsted St" ...
##  $ district        : chr  "Loop" "Loop" "Loop" "Lake View" ...
##  $ city            : chr  "Chicago" "Chicago" "Chicago" "Chicago" ...
##  $ postalCode      : int  60606 60601 60601 60613 60637 60649 60611 60640 60643 60640 ...
##  $ county          : chr  "Cook" "Cook" "Cook" "Cook" ...
##  $ state           : chr  "IL" "IL" "IL" "IL" ...
##  $ country         : chr  "USA" "USA" "USA" "USA" ...
##  $ matchLevel      : chr  "houseNumber" "houseNumber" "houseNumber" "houseNumber" ...
##  $ relevance       : num  1 1 1 1 0.63 1 0.99 1 1 1 ...

All of the previous commands were done under the DEMO license keys. If you are going to use this a lot, it's probably best to get your own API keys. You can do that here.

If you have any questions, hit me up on twitter or shoot me an email at corynissen_AT_gmail.com

Making an R Package to use the HERE geocode API

2014-10-23T09:29:00.000-05:00

HERE is a product by Nokia, formerly called Nokia maps and before that, Ovi maps. It's the result of the acquisition of NAVTEQ in 2007 combined with Plazes and Metacarta, among others. It has a geocoding API, mapping tiles, routing services, and other things. I'm focused on the geocoding service. Under the “Base” license, you can run 10,000 geocoding requests per day. According to wikipedia, that is the most among the free geocoding services. On top of that, HERE does bulk encoding where you submit a file of things to geocode and it returns a link to download a file of results. This sure beats making thousands of requests one at a time.

I figured coding up a quick function to use this service would be the perfect reason to build my first R package. So, after getting my HERE API keys, I fired up devtools and got started with the R package.

First, I had to install the latest version of devtools and roxygen2

install.packages("devtools", repos="http://cran.rstudio.com/")
install.packages("roxygen2", repos="http://cran.rstudio.com/")
library(devtools)
library(roxygen2)

Next, I chose a directory, and ran the create() function from devtools. This creates a package skeleton that contains the basic structure needed for an R package.

devtools::create("geocodeHERE")

Several files and directories are created after running create(). I moved over to the 'R' directory and created a file called “geocodeHERE_simple.R”. I put my function to use the HERE geocoding API in there. This function was designed to be minimalist and similar to the ggmap geocode() function.

#' Attempt to geocode a string
#'
#' Enter a string and have latitude and longitude returned using the HERE API
#' @param search A string to search
#' @param App_id App_id to use the production HERE API. Get one here... http://developer.here.com/get-started. If left blank, will default to demo key with an unknown usage limit.
#' @param App_code App_code to use the production HERE API. Get one here... http://developer.here.com/get-started. If left blank, will default to demo key with an unknown usage limit.
#' @keywords geocode
#' @export
#' @examples
#' \dontrun{
#' geocodeHERE_simple("chicago")
#' geocodeHERE_simple("wrigley field chicago IL")
#' geocodeHERE_simple("233 S Wacker Dr, Chicago, IL 60606")
#' }
#' geocodeHERE_simple
geocodeHERE_simple <- function(search, App_id="", App_code=""){
  if(!is.character(search)){stop("'search' must be a character string")}
  if(!is.character(App_id)){stop("'App_id' must be a character string")}
  if(!is.character(App_code)){stop("'App_code' must be a character string")}

  if(App_id=="" & App_code==""){
    App_id <- "DemoAppId01082013GAL"
    App_code <- "AJKnXv84fjrb0KIHawS0Tg"
    base_url <- "http://geocoder.cit.api.here.com/6.2/geocode."
  }else{
    base_url <- "http://geocoder.api.here.com/6.2/geocode."
  }

  search <- RCurl::curlEscape(search)

  final_url <- paste0(base_url, format, "?app_id=", ids$App_id, "&app_code=",
                      ids$App_code, "&searchtext=", search)

  response <- RCurl::getURL(final_url)
  response_parsed <- RJSONIO::fromJSON(response)
  if(length(response_parsed$Response$View) > 0){
    ret <- response_parsed$Response$View[[1]]$Result[[1]]$Location$DisplayPosition
  }else{
    ret <- NA
  }
  return(ret)
}

Note the text on the top of the code. That is added in order to automatically create help documentation for the function.

Now, if you look closely, I am using two other packages in that function… RCurl and RJSONIO. Also, notice that I'm not making any library() calls. This must be done in the DESCRIPTION file that is automatically generated by create(). In addition to calling out what packages to import, you input the package name, author, description, etc.

Package: geocodeHERE
Title: Wrapper for the HERE geocoding API
Version: 0.1
Authors@R: "Cory Nissen <corynissen@gmail.com> [aut, cre]"
Description: Wrapper for the HERE geocoding API
Depends:
    R (>= 3.1.1)
License: MIT
LazyData: true
Imports:
    RJSONIO,
    RCurl

Then, I set my working directory to the package root and ran the document() function from devtools that automatically creates package documentation.

setwd("geocodeHERE")
document()

From this point, you can upload your package to github and use install_github(), or use the install() function to install your package locally.

setwd("..")
install()

As far as the HERE geocoding API goes, I find it pretty decent at doing “fuzzy matching”. That is, given a place name instead of an address, it does a good job of correctly returning the coordinates. The downside is that you must register for an API key, where Google does not require registration to use it's geocoding service. But, you only get 2500 requests per day via Google, HERE offers 10,000 per day.

Any how, a big shout out to Hilary Parker for her blog post on creating an R package using devtools, Hadley Wickham for the devtools package (among others), and RStudio for the fantastic and free (and open source) IDE for R.

The geocodeHERE package is available on github. I'll be adding bulk geocoding functionality as time permits. You can install the package with the following code:

devtools::install_github("corynissen/geocodeHERE")
library(geocodeHERE)
geocodeHERE_simple("wrigley field chicago il")

Visualizing Bus Stops with rCharts

2014-06-11T10:28:00.000-05:00

I wanted to create a quick visualization of Bloomington IL bus stops. This data is in pdf file format spread across multiple files. The first step, before any mapping can occur, is downloading those files, parsing them to get the bus stop locations and times.

First, I need to get a list of all of the files. This was a little complicated by the fact that the URL for the buses didn't play nice with some of the usual html R tools (RCurl). Alas, the httr package was the solution. First get the html dump, then look for a table with the id of fsvItemsTable and make your way down the tree to get the hrefs for all of the files. I imagine there's a way to avoid the grep at the end of this snippet, but it works, so I stopped...
EDIT: The url to get the list of pdf files appears to show different content to different users. I've uploaded all of the pdf files to the git repo, so if you wish to execute the code and this snippet doesn't work, you can skip it and move to the next part...

Next, use this list of links to download the files. Again, the normal way of doing things, download.file(), failed, but downloader::download() did work.

Now that we have a directory filled with pdf files, what do we do with it? Well, there's a function called readPDF() in the tm package that can be used to read the data in a pdf file. And using code ripped straight from stack overflow, it was pretty easy to get the data.

This leaves you with a single string for each row of data in the pdf table. A little grepping will separate the data in to separate columns in a data frame. See the full code linked at the bottom of the page for these details.

Now we must geocode the bus stop locations so we can plot them on a map. For this, the ggmap package has a simple function called geocode().

At the end of all of this, we finally have a data set to map. Here's what it looks like...

...well, not exactly. To use the toGeoJSON() function in the rCharts package, the df must be transformed into a list. Also, I add in a color for each route so we can tell them apart on the map, and format the text for the tooltip for each point.

Again, in keeping with using other people's code, I reused some code that Ramnath Vaidyanathan had done for the foodborne chicago map a while back to create a leaflet map of the bus stops. He is the author of the rCharts package, is super helpful via twitter and github with random issues, and is doing a tutorial at useR_2014 in LA. I can't wait to meet him... The last part of this code snippet creates a github gist out of it. I had some trouble using it on my network, so I just used the .save() method to create an html file and copy-pasted it as a gist.

And here's the result. There's still some work to be done on the geocoding end of things. As you can see if you click on a dot on the map, the location doesn't always line up with where the map tooltip says it should be.

All of the code can be found on github.

Finding Named Entities using R

2014-02-11T08:35:00.002-06:00

Occasionally, I'll need to pick out names (first name, last name) from text. These days, the text I'm working with is usually tweets. Any how, I didn't see any solution out there (that worked for me) when I developed this, so hopefully it can be a starting point for somebody else with similar needs...

First, I start out with a list of names from the census bureau. I downloaded male first names, female first names, and last names and same them as variables in R. I do take out some of the names as "exceptions" that screw up my process here. Names like "In", "An", "Chi", "So", and so on.

Then, I split my target text up into bigrams, that is, adjacent pairs of words in the original text...

This returns every pair of words in the tweet. From here, I can look through each of these bigrams for names. To make the search for names a little easier, I throw out any bigrams that don't have capital letters for the first and last names.

Now that I have a list of bigrams in which both words start with capital letters, I can compare the words to the name list to see if they are names. I start with the last name. If the second word in the bigram doesn't appear in the last name list, we can stop... there's no need to check the first name. If the second word is a last name, then we check the first word against the first names list. If both of these check out, we have ourselves a name. Here's the code for that...

The full code for this can be found here... https://github.com/corynissen/cook-county-tweet-dashboard/blob/master/cctweets/findNames.R

I have tried the openNLP package for this and couldn't get it to work reliably and quickly, so I made my own. If you have any suggestions on how to do this better, let me know!

Follow me on twitter... https://twitter.com/corynissen

Graphing my MPG vs Temperature

2013-12-09T11:34:00.000-06:00

Updated: 7/9/2014

I use Fuelly to track the fuel mileage in my Toyota Tundra pickup truck. They have a mobile app, so while I'm waiting for the truck to fill up, I log the odometer, and the number of gallons that I put in and then also the price per gallon of fuel and an estimate of my city driving percentage.

Any how, I've noticed a substantial drop in fuel economy when the weather turns cold. So, remembering that I've decided to record the temperature at O'hare every day, I figured I'd try to take a look at the data. Now, I live 40 miles or so north of O'hare, so it's not perfect, but it was a good estimate.

To merge the daily temperature data with the sporadic fillup data, I chose to take the average high temperature for the days between the fillup[i] and fillup[i-1] to use as temperature[i]. I hope that makes sense. Basically, I take the average temperature over the tank of gas.

Let's see what a simple graph of average temperature by MPG looks like…

ggplot(fuel, aes(x=avg.temp, y=mpg)) + geom_point() + 
  geom_smooth(method = lm, se=FALSE) + xlab("AVG Temp") + 
  ylab("MPG") + theme_bw()

There's a fairly obvious pattern here. As temperature goes up, my MPG goes up, just as I suspected was happening.

Now, I know this next graph goes against normal data visualization guidelines, but I just think it makes the relationship easier to understand for a lay person. Here, I've plotted both temperate and MPG on the same graph, with MPG in black with it's axis on the left and temperature in red with it's axis on the right.

grid.newpage()

# two plots
p1 <- ggplot(fuel, aes(x=date, y=mpg)) + geom_line() + 
  theme_bw() + xlab("Date") + ylab("MPG (Black)")
p2 <- ggplot(fuel, aes(x=date, y=avg.temp)) + 
  xlab("Date") + ylab("AVG Temp (Red)") +
  geom_line(colour="red") + theme_bw() + 
  theme(axis.title.y=element_text(angle = -90)) %+replace% 
  theme(panel.background = element_rect(fill = NA))

# extract gtable
g1 <- ggplot_gtable(ggplot_build(p1))
g2 <- ggplot_gtable(ggplot_build(p2))

# overlap the panel of 2nd plot on that of 1st plot
pp <- c(subset(g1$layout, name == "panel", se = t:r))
g <- gtable_add_grob(g1, 
                     g2$grobs[[which(g2$layout$name == "panel")]], 
                     pp$t, pp$l, pp$b, pp$l)

# axis tweaks
alab <- g2$grobs[[which(g2$layout$name=="ylab")]]
ia <- which(g2$layout$name == "axis-l")
ga <- g2$grobs[[ia]]
ax <- ga$children[[2]]
ax$widths <- rev(ax$widths)
ax$grobs <- rev(ax$grobs)
ax$grobs[[1]]$x <- ax$grobs[[1]]$x - unit(1, "npc") + 
  unit(0.15, "cm")
g <- gtable_add_cols(g, g2$widths[g2$layout[ia, ]$l], 
                     length(g$widths) - 1 )
g <- gtable_add_cols(g, g2$widths[g2$layout[ia, ]$l], 
                     length(g$widths) - 1 )
g <- gtable_add_grob(g, ax, pp$t, length(g$widths) - 2, pp$b)
g <- gtable_add_grob(g, alab, pp$t, length(g$widths) - 1, pp$b)

grid.draw(g)

Credit to this guy, for his dual axis ggplot2 code.

R code available on github here. Follow me on twitter… https://twitter.com/corynissen

Using R to Analyze Yahoo Fantasy Football Data

2013-12-06T10:00:00.001-06:00

Updated: 9/16/2014

I recently created a gist that demonstrated how to authenticate with the Yahoo API, using the httr package. In this post, I will expand on this a little to downloading personal Yahoo fantasy football data and creating a graph showing my regular season results.

The first step, obviously, is to authenticate with the Yahoo API and get a signature to attach to all of the requests. This entails registering an application with Yahoo and getting API keys. This page has information about the auth process. I have my credentials saved to a text file that I read in, but you can use environment variables too, whatever you are comfortable with.

library(httr)
library(XML)
library(RJSONIO)
library(ggplot2)

# saved my yahoo keys to a file, now, read them in...
creds <- read.table("~/cn/personal/keys/yahoo.txt", stringsAsFactors=F)
consumer.key <- creds[1,1]
consumer.secret <- creds[2,1]
oauth_endpoints("yahoo")
myapp <- oauth_app("yahoo", key = consumer.key, secret = consumer.secret)
token <- oauth1.0_token(oauth_endpoints("yahoo"), myapp)

Next, you have to discover your game key, league key, and team key. You will need your league id that is in the URL when you visit your normal Yahoo fantasy football page. My league id should not work for you, as mine is a private league.

# need to get game id for my league...
ff.url <- "http://fantasysports.yahooapis.com/fantasy/v2/game/nfl?format=json"
game.key.json <- GET(ff.url, config(token = token))
game.key.list <- fromJSON(as.character(game.key.json), asText=T)
game.key <- game.key.list$fantasy_content$game[[1]]["game_key"]

# my personal leagueid, you will have to use your own, mine is private
league.id <- "262101"
league.key <- paste0(game.key, ".l.", league.id)
league.url <- "http://fantasysports.yahooapis.com/fantasy/v2/league/"

Next, I had to figure out my team id. This was trial and error for me. I'm not sure if there's a better way. Using the matchups endpoint, I can get the data for all of my games for the regular season (2 games as of today). The data is available as XML or JSON. I choose JSON with the “?format=json” at the end of the URL. Then, using the RJSONIO package, I parsed the data to create two vectors, my score for every week, and my opponent's score for every week. The JSON is nested deeply, so this is kinda ugly…

my.team.id <- "4"
my.team.key <- paste0(league.key, ".t.", my.team.id)
team.url <- "http://fantasysports.yahooapis.com/fantasy/v2/team/"
# lots of endpoints to play with, more here... 
# http://developer.yahoo.com/fantasysports/guide/
my.team.stats.json <- GET(paste0(team.url, my.team.key, "/stats?format=json"), 
                          config(token = token))
my.team.standings.json <- GET(paste0(team.url, my.team.key, 
  "/standings?format=json"), config(token = token))
my.team.matchups.json <- GET(paste0(team.url, my.team.key, 
  "/matchups?format=json"), config(token = token))
my.team.matchups.list <- fromJSON(as.character(my.team.matchups.json), asText=T)

# number of games played
game.num <- 2

# get the opponent scores for my matchups for the entire season
tmp <- my.team.matchups.list$fantasy_content["team"][[1]][[2]]$matchups
opp.score <- tmp$'0'$matchup$`0`$teams$`1`$team[[2]]$team_points["total"]
opp.score <- c(opp.score, sapply(as.character(1:(game.num-1)),   
  function(x)tmp[x][[x]]$matchup$`0`$teams$`1`$team[[2]]$team_points$total))
my.score <- tmp$'0'$matchup$`0`$teams$`0`$team[[2]]$team_points["total"]
my.score <- c(my.score, sapply(as.character(1:(game.num-1)),   
  function(x)tmp[x][[x]]$matchup$`0`$teams$`0`$team[[2]]$team_points$total))

Almost there… now we just have to make a dataframe in the format that ggplot2 likes to have…

my.df <- data.frame(cbind(game=rep(1:length(my.score), 2), 
  team=c(rep("me", length(my.score)), rep("them", length(my.score))),
  score=as.numeric(c(my.score, opp.score))))
my.df$game <- factor(my.df$game, levels=1:game.num)
my.df$score <- as.numeric(as.character(my.df$score))

Then we can graph it using ggplot2. Here's the code and output…

p1 <- ggplot(my.df, aes(x=game, y=score, color=team, group=team)) + 
  geom_point() + geom_line() + scale_y_continuous()
ggsave("FF_regular_season.jpg")

There's a bunch more that can be done with this API, the point of this post was to show how to get through the auth stuff and show a simple demonstration of the API.

Full code can be found on github

Follow me on twitter for more R tidbits like this… https://twitter.com/corynissen

Exploring Chicago's Food Inspections

2013-12-02T19:24:00.001-06:00

In my last post, I made a plot of the Foodborne Chicago submissions. There was a notable pattern, specifically, that most of the reported restaurants were in the loop or north of the loop, with very few on the south side of the city. This may be because there are fewer restaurants on the south side. To explore that hypothesis, I have obtained a list of all of the retail food establishments in Chicago along with their risk level. The risk level is based on the type of food served and the preparation involved, with 1 being the highest risk and 3 being the lowest risk.

The file I received, Retail Food 1006 Licenses 2013.xlsx, contains the address for the facility, but not the latitude and longitude. I will need latitude and longitude for mapping purposes, so I needed to find a way to geocode them. Google offers a free service, but limits the number of requests to 2500 per day. The data set has 15176 facilities, so that would take a week to do. I decided to download the food inspection data from the data portal. This has the latitude and longitude that I need, along with the restaurant license number. I also have license number in the data set from above. Now clearly, not all of the restaurants listed in the first data set will have had an inspection, but many will. I used the Google API to geocode the remaining restaurants. With this process, the geocoding went from a week to two days.

The last step with this file was to take the newly generated latitudes and longitudes get the ward number for each restaurant. To do this, I used an API provided by the Chicago Works For You project.

Before I get too far, let me list the assumptions that I've made up to this point.

The license number appearing in Retail Food 1006 Licenses 2013.xlsx corresponds to the same restaurant and license number in the food inspections data set on the data portal.
The latitude and longitude that I receive from the Google API and the latitude and longitude from the city of Chicago's internal data are both interchangeable. That is, either source is valid for finding the location of a given restaurant.
The restaurants that cannot be geocoded using either method or have a ward of zero after using the CWFY API are randomly distributed among all restaurants, and leaving them out of the analysis introduces no bias and is generally not harmful or wrong to do.

In summary:

Get Retail Food 1006 Licenses 2013.xlsx file from Chicago Department of Public Health
Match file with food inspections data to get some latitude longitude pairs
Use Google API to get remaining latitude longitude pairs
Use CWFY API to get ward number from latitude longitude pairs
End with geocoded_risk_data_w_wards.csv with some intermediate files created along the way

Now that the data is ready, let's take a look at it. First, I want to plot the number of restaurants per ward on a map. Here's the result...

It looks like there's definitely a concentration of restaurants around the loop. Ward 42 in particular has nearly 1600 restaurants in it, where the average across all wards is about 300. This makes the difference between the other wards a little difficult to see, since the color scale is dominated by ward 42's extreme number of restaurants. To get around this, let's ignore ward 42 and redraw the map...

Now it's more clear that there's an abundance of restaurants in the wards around the loop, specifically to the north and west.

Another way to look at this would be to run a 2d density on the latitude longitude pairs. Here's that graph...

The preceding graphs were restaurant counts across all risk levels. Let's split out the risk levels and see if there's a pattern. Specifically, let's look at the proportion of a given risk level to all of the restaurants for each ward. For example. If a ward has 24 High risk restaurants out of 154 total, let's look at that proportion... 15.6%.

Now we are seeing something. The west and south side have a much lower proportion of high risk restaurants compared to the loop and north side. Let's see about the medium risk level...

And low risk level...

Mapping Foodborne Chicago Reports

2013-11-25T10:13:00.002-06:00

I help out with a project called Foodborne Chicago, where we search twitter for people in Chicago mentioning "food poisoning" and @reply to them with a link where they can file a report with the city of Chicago's Department of Public Health. The application has been live since March 2013. Since then, we have passed on about 125 reports to the city. For the majority of these reports, an inspection work order is generated and an inspection occurs within about a week of the incident submission.

So... I decided to take a look and see where these submissions were by plotting them on a map. To start, I had a file from our app with three columns, id which is a unique identifier of the submission, restaurant.address which is the address for the submission, and created.at which represents the date that the complaint was submitted. The first step to plotting these submissions on a map was to get the latitude and longitude for the restaurant addresses. I used the ggmap package for this. Originally, I had a custom function that used the free Google geocoding API and parsed the results, then I found the ggmap package and changed over. The results are the same, but it makes the code cleaner and easier to maintain.

Then, I Since I've used the OpenStreetMap package in R in another project, I started with that to make a map. The only gotcha here was that the OpenStreetMap package needs the points in mercator, so there's a conversion step.

The ggmap package makes doing this sorta thing, in conjunction with ggplot2, very simple. Here's the same map as above, using ggmap and Google maps.

Now, since I'm from the area and am not even sure where Chicago's borders are on that map, I downloaded the shapefiles for Chicago from the city of Chicago and added that as an overlay on the map. All I did here was take the last plot (p2) and add the geom_polygon with a low alpha to the map.

This gives a pretty good representation of where our Foodborne Chicago submissions are coming from. Nearly all of them are on the northern half of the city, with a few from beyond the city limits altogether. It's pretty remarkable how easy the ggmap and ggplot2 folks have made creating maps like this. Thanks guys...

The whole project, including making an animated gif and doing some cloudmade maps is on github here... https://github.com/corynissen/foodborne_chicago_report_map

Check out Foodborne Chicago on twitter... https://twitter.com/foodbornechi

And myself on twitter... https://twitter.com/corynissen

Google Prediction API update

2013-09-10T09:38:00.000-05:00

A quick update on the last blog post...

Mr Hadley Wickham has made the authentication for this much easier...

Then you can use the curl script like in my last blog post.

Google Prediction API example

2013-06-13T11:05:00.000-05:00

Google's prediction API offers a blackbox way of doing some prediction. They had advertised an R package, but it doesn't seem to work with the new version of the prediction API or their OAuth2 authentication mechanism. So, in an effort to check out the prediction API, I tried to go through the authentication process myself. I got it to work, but couldn't get it ported to RCurl successfully, so if you have any suggestions, I'd love to hear them. Since I was unable to get RCurl working, I am using a system() call with 'curl' that is installed on my machine.

The first step is to fill out your API keys and all of that sort of stuff. Note that this must be done in strict accordance with Google's rules. The scope must match what you've entered on the google auth page, same with the redirect_uri.

Then, use those things to submit a request for an authorization code. You will use this code later to request an authorization token. browseURL() opens a page in your browser window. Log into google (if you aren't already) and look at the resulting URL, it contains your code. Example response would be like this... https://oauth2-login-demo.appspot.com/code?state=/profile&code=4/P7q7W91a-oMsCeLvIaQm6bTrgtp7 You want everything after '&code='

Okay, if successful, you now have an authorization code. Cut and paste that from your browser into the your R code and continue on with a authorization token request.

Now, if you are successful, 'token_code' contains what you need to make a request to the google prediction API. Here's how you try it out on a demo language guessing model.

You should end up with a response like this

All of the code together... https://gist.github.com/corynissen/5774784

Using R to communicate via a socket connection

2013-05-28T09:39:00.000-05:00

Occasionally, the need arises to communicate with R via another process. There are packages available to facilitate this communication, but for simple problems, a socket connection may be the answer. Nearly all software languages have a socket communication package, so this is very simple to implement.

First, start off with your R "server". In this case, I'm accepting text as input and upper casing it as a trivial example.

Then, use the client script to connect to the server and upper case some text. Here's an example in R...

Here's an example of the client in python. This will enable communication and data transfer between python and R.

To pull it all together, on the same machine, start up two command prompts. In the first one, start the R server via "Rscript server.R". Then in the other window, run the client script via "python client.py" or "Rscript client.R". In the client window, you should see a prompt to enter some text. Do so, and be amazed as you see your text returned from R in upper case.

Obviously, this can be reversed, with python or some other process being used as a "server" and R as the "client".

All of the code is included in a github repo

Shout out to my colleague JoeO for an early incarnation and an early use case for this process.

Follow me on twitter

3D Model Using the rgl Package

2013-04-25T14:40:00.000-05:00

A while back, I had a unique opportunity to work with some very talented interns for the summer while I was working at Allstate insurance. Without giving away details of the project that Allstate may prefer to keep secret, let's just say that we had access to a laser range finder, and were testing it on a cardboard cutout. The laser range finder would give angle and distance in a fan shape directly in front of it. It's used in robotics to detect obstacles and whatnot. We mounted the scanner on a platform and moved it vertically. That way, we could take a fan shape of measurements incrementally moving up and down giving a 3D view of the target.

The data file was great, but we needed a way to visually inspect the data to check that the trig the interns had to do to get x,y,z data from the angle, angle, distance raw data was correct. So, I fired up R, found the rgl package, and got to work.

There's really not much to the code, first I read in the data. They gave me a 2D matrix of z values. I created some dummy x and y data, fitted it to some colors, smoothed out some errant laser readings and ran the viewer...

It opens a viewer that you can zoom, tilt, and pan.

There are options to adjust the artificial lighting, the colors, and a bunch of other things, but as you can see, it came out fairly well. This was a 3d cardboard "car" taped to a wall. You can even see evidence of the tape coming out of the roof and windshield.

The code and data are on github here... https://github.com/corynissen/3D_model_demo

follow me on twitter... https://twitter.com/corynissen

ggplot dodged vs faceted bar chart

2013-04-17T12:10:00.000-05:00

Updated: 7/9/2014

I've been bowling once per year at a charity event for the last few years and have kept track of the outcomes to share my group. I used ggplot2 to create a bar chart for the scores. Below are two graphs, one is dodged, the other is faceted. There's no right way here, but you can see the code and decide for yourself which is more aesthetically pleasing.

First is the dodged one…

library(ggplot2)
df <- read.csv("bowling.csv", strip.white=T)
a <- 1:200

ggplot(df, aes(x=factor(name), y=score)) +
  geom_bar(position="dodge", stat="identity",aes(fill=factor(game))) +
  coord_cartesian(ylim=c(50, 150)) + scale_y_continuous(breaks=a[a%%10==0]) +
  xlab("") + ylab("Score") + scale_fill_discrete(name="Game") +
  theme(axis.text.x = element_text(size=14))

And next is the faceted one…

ggplot(df, aes(x=factor(game), y=score)) +
  geom_bar(stat="identity",aes(fill=factor(game))) + facet_grid(.~name) +
  coord_cartesian(ylim=c(50, 150)) + scale_y_continuous(breaks=a[a%%10==0]) +
  scale_fill_discrete(name="Game") + xlab("") + ylab("Score")

The color is redundant in the faceted one, since there are labels for the game number. I kept the color in to make comparison with the dodged one easier.

Here's the code to produce these two graphs. The full code with the data to produce these plots and also the blog post using Rmarkdown is available in my github repo.

@corynissen on twitter

Unshorten URLs in R

2013-04-15T11:44:00.000-05:00

Well, of course, this tip comes out one week after I needed it. The author uses the RCurl package to request the header of the shortened URL and then parse the "location" parameter on the return. This sort of operation tends to be needed frequently, especially when using data from twitter. Twitter now shortens even already shortened links using their own t.co service. Every link in every tweet now has to be put through a process like this to resolve it to the full url.

My solution is very similar to the RLangTip, but instead of using RCurl, I am using a system call to "curl", and repeatedly requesting the header for each url returned until no location attribute is found... and that's the final url. It's a little ugly, and I'm sure it can be sped up, and improved upon, but it works well enough...

Find me on twitter... https://twitter.com/corynissen

Use foursquare to locate a twitter user using R

2013-04-08T13:56:00.001-05:00

I've been doing some work with Twitter data. In much of this work, my life would be so much easier if we could geographically locate the origin of the tweets. There are some ways to do this using the twitter APIs. For example, if a user has geo-location turned on, you can get the precise lat-lng info for a specific tweet. Also, each user has the option to set a location in their profile. Using this free-form info, you can get an idea of where the user is located, but this is not relevant to each specific tweet, it's a user attribute. So, a Chicago native traveling and tweeting in Florida will be a problem. Using this information, you can get some information for some people. But it's not complete... so I thought about trying some other ways of location a twitter user.

My first crack at this was to see if a given twitter user is a foursquare user. Foursquare users use the service to check in to places for points or other purposes. Any how, using the foursquare API, you can retrieve a lat-lng pair for a given checkin. My idea was to look at a user's tweet history to see if there are any fourquare links. Take these foursquare links and use the foursquare API to get lat-lng pairs. Then, cluster these points and choose the mid point of the largest cluster, that is, the cluster with the most points, as the guess for the twitter users location.

First, we start off using the twitteR package to download n of a users most recent tweets...

Then, I extract the links and resolve them using a link expander service. Once that is done, I can take the foursquare links, and bounce them off of the foursquare API to get a lat-lng pair. I made a function to do this. Note that you will need a foursquare API key saved in a file as noted in the function.

Now, you have a list of lat-lng pairs from foursquare. I use the Mclust to find the largest cluster. Then, using the points in that largest cluster, take the average lat and average lng to be the center of the cluster. That is my guess for where my twitter user of interest lives.

Next, I create an openstreets map to display this. The red dots represent the various lat-lng pairs from foursquare, and the blue dot is the cluster center.

The code can be found on github.