cl-twitter

Using mongodb as a twitter persistence store

noreply@blogger.com (mohegskunkworks) — Thu, 05 May 2011 23:12:00 +0000

MongoDB is a nosql datastore developed by 10gen in New York City. MongoDB stores collections of key-value pairs called documents. The values associated with a key can even be other documents. A group of documents form a collection and collections make up a database.

Installing mongoDB is easy. There are downloads available for linux, mac osx and windows. Nevertheless, if you can't find a pre-built executable, building one is pretty straightforward too.

Because there are drivers for a wide variety of programming languages, data stored in a mongoDB database can be accessed and analyzed by non lisp tools like the natural language toolkit (nltk) or lingpipe.

MongoDB has many features like powerful querying facilities , map-reduce processing and data distribution through sharding. Consult the mongoDB website or books like MongoDB: The definitive guide , MongoDB in action or The definitive guide to MongoDB.

The elephant store discussed in the previous post does not have a lot of these facilities and is probably the right choice if you just want to store data for analysis using lisp based tools.

You start by loading the :cl-twitter package into you image. Next enable the in-memory cache with the (use-cache) command. The driver specification consists of the host, port and database you'd like to connect to. Here's an example session:

CL-USER> (use-cache)
STYLE-WARNING:
   redefining REGISTER-TWITTER-OBJECT (#
                                       #) in DEFMETHOD
#
CL-USER> (use-db :twitter-mongodb-driver :spec '(host localhost port 27017 db twitter))
;[...]
#
CL-USER> (db-status)
mongodb driver loaded
last error : [{n : 0} {connectionId : 4} {err : NIL} {ok : 1.0d0} ]
last error : [{err : NIL} {n : 0} {nPrev : -1} {ok : 1.0d0} ]
(type-of CL-MONGO:MONGO) [name : TWITTER ]
{[id : 6A90ACAF-D370-4C7F-9A0E-DEE88B6184EF] [port : 27017] [host : localhost] [db : twitter]}

               trend-list :   3.0d0
                    trend :   18.0d0
                    tweet :   51953.0d0
             twitter-user :   50208.0d0
NIL
CL-USER>

In the session above the specification list will be saved as the default specification. You can use the :tag keyword to save a non-default specification. The next time you load the driver you only need to supply the driver keyword (or the tag name if it's not the default).

Each tweet entity is converted to a document which is essentially a hash map. Each document has a unique id. This id is stored with a magic key name. If this key is not present the server will assign a unique key. The mongoDB driver sets the unique id before sending the document to the mongoDB server. This id is either the id returned by the twitter api or is generated from attribute values of the tweet entity.
Each tweet entity has its own separate collection in the database. When an entity is embedded in an other entity (like twitter-user in tweet), the embedded entity will be stored in it's own collection.

You can access the data in MongoDB using the cl-mongo lisp package. As mentioned there are drivers in a variety of languages available.

The internals of the driver are very similar to those of the elephant driver discussed earlier.

That particular particular post also shows how to use the other interface methods work.

Using elephant as a twitter peristence store

noreply@blogger.com (mohegskunkworks) — Tue, 19 Apr 2011 22:54:00 +0000

In my previous post I discussed the persistence interface. This post will focus on the elephant driver. To quote from the introduction of the manual: 'Elephant is a persistent object protocol and database for common lisp'.
In order to use the elephant driver, you'd need to have an elephant data store set up. The first step would be to install the elephant package and it's dependencies. I've had the best results using v 1.0 in the darcs repository :

darcs get http://www.common-lisp.net/project/elephant/darcs/elephant-1.0

The back end I'm the most familiar with is Berkeley db (BDB) . To use it you need to build version 4.5 of BDB for your platform. Next select a configuration in my-config.sexp, as detailed in the elephant manual. Comment out (or remove) all other configurations from the file. Next try to open a data store. BDB stores data on the file system so you'd need to specify a path to the location on your file system where you'd like to store your data :

(use-package :elephant)

(open-store '(:BDB {path to db files}))

If this is successful, load the cl-twit-repl package and the elephant db driver :

CL-USER> (use-package :cl-twit-repl)
T
CL-USER> (use-cache)
STYLE-WARNING:
redefining REGISTER-TWITTER-OBJECT (#
#) in DEFMETHOD
#
CL-USER> (use-db :twitter-elephant-driver :spec '(:BDB "/Users/fons/Data/bdb/twitter/"))

; loading system definition from /Users/fons/.sbcl/systems/elephant.asd
; into #
; registering # as ELEPHANT
; loading system definition from /Users/fons/.sbcl/systems/uffi.asd into
; #
; registering # as UFFI
Attempting to load libmemutil.dylib...
[......]
STYLE-WARNING: Implicitly creating new generic function CHECKPOINT.
STYLE-WARNING: Implicitly creating new generic function BUILD-BTREE-INDEX.
STYLE-WARNING: Implicitly creating new generic function POPULATE.
#
CL-USER>
The next step is to call (public-timeline) and (trends) from the repl. This should cause the first batch of tweets and users the be saved in the data store. You can check to see if this happened by calling (db-status). You should see something similar to this :

CL-USER> (db-status)

store : BDB-STORE-CONTROLLER....

walking over packages in the root
root               TREND :   #
root          TREND-LIST :   #
root               TWEET :   #
root        TWITTER-USER :   #
             TWITTER-USER :   100
                    TWEET :   100
               TREND-LIST :   1
                    TREND :   10
NIL

Obviously the results shown represent more than a single call.
Notice that you need to issue a (use-cache) before loading the driver with (use-db ..). As mentioned in the previous post (use-cache) sets up the methods to save the twitter entities in the cache and these methods call the generic db-store-object. Each driver 's specification can be tagged to distinguish between different data store locations, with the default tag being :default. The next time around you can just call (use-db :twitter-elephant-driver).
Here's the implementation of (use-db..)

(defun use-db (driver-package &key (spec nil specp) (tag :default))
(when driver-package
    (require driver-package)
    (use-package driver-package)
    (when specp (write-driver-spec (list :TAG tag :DRIVER driver-package :SPEC spec)))
    (let ((spec (gethash (cons tag driver-package) (read-driver-spec))))
      (when spec (db-initialize (cadr (member :SPEC spec)))))))

The driver-package is an asdf module which specifies the location of the driver source files which implement the persistence interface mentioned in the previous post.
The implementation of db-initialize looks like this:

(defmethod db-initialize (&rest args)
(open-twitter-db (car args))
(create-caches store-twitter-object))

In the case of this driver it assumes that the argument list consists of a specification list. The specification list should contain all the data necessary to initialize the elephant data store. This specification list is passed on to elephant's open-store method. In the next step of the initialization overrides the default implementation of the db-store-object method which is called each time an item is added to the in-memory cache. Not all types of twitter entities are stored. The twitter entities stored in the cache (and hence the data store) are read from the cache.spec file at initialization.
(create-caches ..) is a macro which takes as it's argument an other macro. It reads the entity keywords from the cache.spec file and passes this on the macro passed in as an argument. The assumption is that this macro will in fact create the appropriate db-store-object method override for the entity in question :

(defmacro create-caches (defmethod-maker-macro)
(let ((lst (list 'progn )))
    (dolist (spec (read-cache-spec))
      (push `(,defmethod-maker-macro ,spec) lst))
    `,(nreverse lst)))

The db-store-object interface method is defined as part of the store-twitter-object macro which passed into create-caches

(defmacro store-twitter-object (type)
`(progn
    (defmethod db-store-object ((collection ,type) (lisprec cons))
      (with-twitter-db ()
    (let ((twitter-obj-db (twitter-object-db (type-of collection)))
          (db-key        (unique-id collection)))
    (unless (get-value db-key twitter-obj-db)
        (setf (get-value db-key twitter-obj-db) lisprec)))))))

Note, that the object stored in the elephant data store is the S-expression representing the response of the twitter api call.
Three more methods round out the driver interface : db-map-reduce, db-status and db-shutdown.

The db-map-reduce methods are defined as follows :

(defmethod db-map-reduce ( (collection symbol) (map-fn function) (reduce-fn function) &key (from nil) (to nil) (from-end nil) (initial-value nil))
(let ((lst (map-btree map-fn (twitter-object-db collection) :collect t :from-end from-end :start from :end to)))
    (reduce reduce-fn lst :initial-value initial-value :from-end from-end)))

(defmethod db-map-reduce ( (collection symbol) (map-fn function) (reduce-fn (eql nil)) &key (from nil) (to nil) (from-end nil))
(let ((lst (map-btree map-fn (twitter-object-db collection) :collect t :from-end from-end :start from :end to)))
    lst))

collection is the name of a twitter entity, like tweet or twitter-user. The names of the entities can be found in the cache.spec. map-fn is a function with two arguments : The first argument is the key; the next argument is the value. map-btree is a function provided by the elephant api and yts keys can be passed in to db-map-reduce as well. The result of map-btree is stored in a lst which is either returned or is passed into reduce. reduce-fn is a single-valued function working on the individual elements of this list. Should you not provide a reduce function the second specialization is used.

Here's an example of it's use:

(db-map-reduce 'tweet (lambda (x y) (format t "~A~A~%" x y)) nil)
[....]
                (PROFILE-USE-BACKGROUND-IMAGE . T) (ID . 180148716)
                   (IS-TRANSLATOR) (UTC-OFFSET . -21600)
                   (PROFILE-BACKGROUND-COLOR . ffffff))
                  (ID . 50724431609344000) (IN-REPLY-TO-SCREEN-NAME)
                  (IN-REPLY-TO-USER-ID))
50724431915520000((IN-REPLY-TO-STATUS-ID-STR)
                  (TEXT . am just hapi.it's a great day) (CONTRIBUTORS)
                  (RETWEETED) (IN-REPLY-TO-USER-ID-STR)
                  (RETWEET-COUNT . 0) (ID-STR . 50724431915520000)
                  (SOURCE
                   . Mobile Web)
                  (GEO) (TRUNCATED)
                  (CREATED-AT . Thu Mar 24 01:03:41 +0000 2011) (PLACE)
                  (IN-REPLY-TO-STATUS-ID) (FAVORITED) (COORDINATES)
                  (USER (FOLLOWING)
                   (PROFILE-BACKGROUND-IMAGE-URL
                    . http://a3.twimg.com/a/1300727311/images/themes/theme1/bg.png)
                   (FAVOURITES-COUNT . 1) (DESCRIPTION)
                   (SHOW-ALL-INLINE-MEDIA) (LISTED-COUNT . 0)
                   (CONTRIBUTORS-ENABLED) (GEO-ENABLED) (TIME-ZONE)
                   (PROFILE-TEXT-COLOR . 333333) (URL)
                   (PROFILE-IMAGE-URL
                    . http://a3.twimg.com/sticky/default_profile_images/default_profile_2_normal.png)
                   (PROFILE-SIDEBAR-FILL-COLOR . DDEEF6)
                   (ID-STR . 270838024) (PROFILE-BACKGROUND-TILE)
                   (LOCATION) (SCREEN-NAME . darelistic)
                   (STATUSES-COUNT . 2) (FOLLOWERS-COUNT . 0)
                   (IS-TRANSLATOR) (LANG . en) (VERIFIED)
                   (CREATED-AT . Wed Mar 23 10:28:44 +0000 2011)
                   (PROFILE-LINK-COLOR . 0084B4) (NOTIFICATIONS)
                   (PROFILE-SIDEBAR-BORDER-COLOR . C0DEED) (PROTECTED)
                   (NAME . Damilare olaniyan) (FOLLOW-REQUEST-SENT)
                   (PROFILE-USE-BACKGROUND-IMAGE . T) (ID . 270838024)
                   (UTC-OFFSET) (FRIENDS-COUNT . 1)
                   (PROFILE-BACKGROUND-COLOR . C0DEED))
                  (ID . 50724431915520000) (IN-REPLY-TO-SCREEN-NAME)
                  (IN-REPLY-TO-USER-ID))
50724433815552000((IN-REPLY-TO-STATUS-ID-STR)
                  (TEXT
                   . Novo EP do Nirvana dia 16 de Abril http://bit.ly/hGmV1P)
                  (CONTRIBUTORS) (RETWEETED) (IN-REPLY-TO-USER-ID-STR)
                  (RETWEET-COUNT . 0) (ID-STR . 50724433815552000)
                  (SOURCE . web) (GEO) (TRUNCATED)
                  (CREATED-AT . Thu Mar 24 01:03:42 +0000 2011) (PLACE)
                  (IN-REPLY-TO-STATUS-ID) (FAVORITED) (COORDINATES)
                  (USER (STATUSES-COUNT . 1662) (FOLLOWING)
                   (PROFILE-BACKGROUND-IMAGE-URL
                    . http://a3.twimg.com/profile_background_images/204786069/jeff_buckley_1280v2.jpg)
                   (FAVOURITES-COUNT . 0) (FRIENDS-COUNT . 238)
                   (DESCRIPTION
                    . Rock & Entretenimento: Bandas, Bandas de Metal, Bandas Novas, Matérias de Rock, Resenhas / Animes, Cinema, Séries, HQs. http://alienacaorock.blogspot.com/)
                   (TIME-ZONE . Brasilia) (PROFILE-TEXT-COLOR . 999999)
                   (URL . http://www.alienacaorock.blogspot.com/)
                   (PROFILE-IMAGE-URL
                    . http://a0.twimg.com/profile_images/1175841120/jeff-buckley-hallelujah_normal.jpg)
                   (LISTED-COUNT . 23) (CONTRIBUTORS-ENABLED)
                   (PROFILE-SIDEBAR-FILL-COLOR . 000000)
                   (ID-STR . 118215328) (PROFILE-BACKGROUND-TILE . T)
                   (LOCATION . Curitiba ) (SCREEN-NAME . AlienacaoRock)
                   (FOLLOWERS-COUNT . 295) (LANG . en) (VERIFIED)
                   (CREATED-AT . Sat Feb 27 23:42:51 +0000 2010)
                   (PROFILE-LINK-COLOR . 850404) (IS-TRANSLATOR)
                   (NOTIFICATIONS) (PROFILE-SIDEBAR-BORDER-COLOR . 000000)
                   (PROTECTED) (SHOW-ALL-INLINE-MEDIA) (GEO-ENABLED . T)
                   (NAME . Alienação Rock)
                   (PROFILE-USE-BACKGROUND-IMAGE . T) (ID . 118215328)
                   (FOLLOW-REQUEST-SENT) (UTC-OFFSET . -10800)
                   (PROFILE-BACKGROUND-COLOR . ffffff))
                  (ID . 50724433815552000) (IN-REPLY-TO-SCREEN-NAME)
                  (IN-REPLY-TO-USER-ID))
50724435476480000((IN-REPLY-TO-STATUS-ID-STR)
                  (TEXT
                   . RT @Ugglytruth: La mujer necesita un hombre que no le de vergüenza en mostrarle lo que siente así sea en publico o delante sus amigos.)
                  (CONTRIBUTORS) (RETWEETED) (IN-REPLY-TO-USER-ID-STR)
                  (RETWEET-COUNT . 0) (ID-STR . 50724435476480000)
                  (SOURCE
                   . ÜberSocial)
                  (GEO) (TRUNCATED)
[....]
As mentioned earlier, the json response string is converted to an alist and the alist is stored in the elephant data store. The elephant db supports object persistence using it's persistent-metaclass metaclass. Reading objects with this meta class back would require access to their class definitions in the twitter api. One the design goals here is make the retrieval of the data from the stro not depend on the twitter api. Reading and processing alists is standard lisp practice and requires no dependency on the twitter api internals.

db-status shows the number of items in each collection and its use was demonstrated earlier on in the post.

db-shutdown provides a way to shutdown the data store by calling the elephant api's close-store on the data store.

Saving twitter data with cl-twitter's built-in persistence.

noreply@blogger.com (mohegskunkworks) — Tue, 05 Apr 2011 22:56:00 +0000

The previous post discussed cl-twitter's built-in in-memory cache which maybe all you need. In addition to the in-memory cache cl-twitter's database api provides a minimal persistence interface to a data store. The api is defined as follows :

(defgeneric db-initialize (&rest args))
(defgeneric db-shutdown (&rest args))
(defgeneric db-store-object (collection lisprec))
(defgeneric db-map-reduce (collection map-fn reduce-fn &key))
(defgeneric db-status (&rest args))

Currently there are two drivers which implement this protocol. One for the elephant object store which is build on top of several data stores. The other driver is for the non-sql database mongodb. elephant is designed to work well with lisp objects. mongodb is a document bases non-sql database which has drivers for many different programming languages.

Drivers are loaded using the use-db method. db-initialize is called when the driver package is loaded. db-store-object is called anytime an object is stored on the cache. As mentioned in the previous post the cache needs to be enabled in order to use the persistence api. Persistence is transparent to the user because db-store-object is called right after the object is stored in the cache. db-status returns information about the persistence store, like connection information or the number of objects currently stored. db-map-reduce can be used to perform operations on the data in particular collection. db-shutdown provides a clean way to shutdown the data store. The persistence interface only supports inserts into the data store. I assume that most of the data management and analysis will be done outside of the cl-twitter package, perhaps even in a programming environment different from lisp.

As I mentioned drivers are loaded by calling use-db with the driver package name, a connection specification and an identifying tag, viz. : (defun use-db (driver-package &key (spec nil specp) (tag :default))

driver-package is a symbol identifying the driver package. The spec is an S-expression contains the login to and location of the server. Server specifications are stored in the driver.spec file located in the access sub-directory of the twitter package.The tag is used to label the specification in the driver.spec file. The tag name is used later on as an alias to the specification expression. The initial specification is saved and tagged as the default one, and subsequent invocations only require the driver name.

When you load the elephant driver with a berkeley db back end for the first time you'd specify the path to the berkeley db files:
(use-db :twitter-elephant-driver :spec '(:BDB ))
Similarly for the mongo db driver :
(use-db :twitter-mongodb-driver :spec '(:host "localhost" :port 27017 :db "twitter") )
The specification is stored with the default tag and subsequent invocations can just use :
(use-db :twitter-elephant-driver).

After the data store's driver has been loaded twitter objects are stored in the store. (db-status) can be called to see data store connection information and statistics, viz :

CL-USER> (db-status)
mongodb driver loaded
last error : [{n : 0} {connectionId : 7} {err : NIL} {ok : 1.0d0} ]
last error : [{err : NIL} {n : 0} {nPrev : -1} {ok : 1.0d0} ]
(type-of CL-MONGO:MONGO) [name : TWITTER ]
{[id : CC067A61-F777-4635-862B-B734923399D1] [port : 27017] [host : localhost] [db : twitter]}

trend-list : 3.0d0
trend : 18.0d0
tweet : 51953.0d0
twitter-user : 50208.0d0
NIL
CL-USER>

The use of db-map-reduce is similar to the cache-map-reduce function discussed in earlier posts.

Cache your Tweets : How cl-twitter implements the cache.

noreply@blogger.com (mohegskunkworks) — Tue, 29 Mar 2011 10:15:00 +0000

Sometimes you just want to save things. And sometimes the things you want to save are tweets or other assorted twitter entities. You want to do this because you just received your copy of Mining the Social Web. Or you want to analyze tweets using lingpipe or the Natural Language Toolkit.

This post provides a high level overview of the cl-twitter's memory cache. The next few posts
will go over the db interface.

The basic idea is that each twitter entity has a unique id and is saved in a collection.
For some entities, like the tweet or user entity, the twitter api provides a unique id. For other entities cl-twitter will need to provide it's own unique id. For example, the result returned by the social graph api call is just a list of user id's and a cursor id and the results of the trends api call also lack a unique id. In cases like these, to meaningfully distinguish between different results you need to enrich the return with (some of) the arguments of the api call as well as a unique id.

You start using the cache by simply calling (use-cache). This enables the cache and is the first step in enabling persistence overall. Starting the cache causes a set of implementations of register-twitter-object to be generated for a select set of types. These types are read from the cache.spec file located in the db sub-directory of the cl-twitter package.
(defgeneric register-twitter-object (ref lisprec)) is called when the json returned by twitter is turned into an lisp object. The ref argument is in fact that object and the lisprec is an S-expression representing the json. Objects are cached because this method is overridden for the object type in question. register-twitter-object enables the introduction of side effects when the results returned by twitter are processed. It's return value is ignored. The default implementation of register-twitter-object is to do nothing.

The contents of cache.spec can be manipulated using write-cache-spec and inspected using read-cache-spec. The cache spec currently has the following types:

TWITTER-USER
TREND-LIST
SEARCH-REF
TWEET
TREND
SOCIAL-GRAPH-CURSOR-ID

register-twitter-object stores the twitter object in a hash table. The key of the hash table is determined by calling unique-id on the object. unique-id is a generic function whose default implementation returns the unique id of the twitter object. As mentioned before, in some case twitter does not assign a unique id, and unique-id needs to be specialized appropriately.
After storing the object in memory register-twitter-object calls db-store-object which is part of the generic persistence interface.

The (caches) call shows the current object count of the cache and cache-map-reduce can be used to access the cache. It's defined as follows :
(defgeneric cache-map-reduce (collection map-fn reduce-fn &key))
The collection is the type of twitter object you want to apply this to.The map function and reduce function are both optional.
When neither map-fn nor reduce-fn are specified, cache-map-reduce returns a list of all the objects in the collection. When a map function is specified, it will apply the map function to each object in the collection before adding it to the list. If an additional reduce function is supplied, the list will passed on to lisp's reduce function and same optional keywords are also available.

The macro map-reduce-cache makes things a little easier. It's defined as :
(defmacro map-reduce-cache (collection &key (map nil) (reduce nil) )
`(progn
     (if ,map
   (cache-map-reduce (quote ,collection) (lambda (k v) (declare (ignore k)) (funcall ,map v)) ,reduce :collect nil)
   (cache-map-reduce (quote ,collection) nil ,reduce :collect t))))

Here's an example :

CL-USER> (caches)
((TREND-LIST . 2) (TREND . 10) (TWEET . 80) (TWITTER-USER . 80))
CL-USER>

CL-USER> (show (map-reduce-cache trend))
Law & Order http://search.twitter.com/search?q=Law+%26+Order
-----------------------------
Cibele Dorsa http://search.twitter.com/search?q=Cibele+Dorsa
-----------------------------
Umständen http://search.twitter.com/search?q=Umst%C3%A4nden
-----------------------------
Escócia http://search.twitter.com/search?q=Esc%C3%B3cia
-----------------------------
Léo Batista http://search.twitter.com/search?q=L%C3%A9o+Batista
------------------------------
Hocus Pocus http://search.twitter.com/search?q=Hocus+Pocus
-------------------------------
#iseewhyyoumad http://search.twitter.com/search?q=%23iseewhyyoumad
-------------------------------
❒Single ❒Taken http://search.twitter.com/search?q=%E2%9D%92Single+%E2%9D%92Taken
--------------------------------
#wealldoit http://search.twitter.com/search?q=%23wealldoit
-------------------------------
#100factsaboutme http://search.twitter.com/search?q=%23100factsaboutme
------------------------------

aliasing commands in the cl-twitter repl mode

noreply@blogger.com (mohegskunkworks) — Sat, 19 Feb 2011 14:33:00 +0000

As I wrote in an earlier post the cl-twitter repl client should have the ability to create simple commands from the api calls.

Aliasing ranges from a shorter name for a command , to a command plus a set of options (curried api call), to a combination of commands (a lambda). These aliases need to persist between sessions.

The cl-twit-repl package in cl-twitter provides two macros unsurprisingly called alias and unalias to handle aliasing.

alias has three modes : called without arguments it"ll show a list of all the currently installed aliases:



CL-USER> (alias)

MOH                =>   (GET-AUTHENTICATED-USER "mohegskunkworks")

PT                 =>   #:PUBLIC-TIMELINE

RL                 =>   #:RATE-LIMIT-STATUS

WHOAMI             =>   (LAMBDA NIL (WHEN *TWITTER-USER* (#:VERIFY-CREDENTIALS)))

H                  =>   (#:COLLECT-HOME-TIMELINE)

FT                 =>   FRIENDS-TIMELINE

SFOLU              =>   (LAMBDA (SN N) (MAPCAR (FUNCTION #:SHOW-USER-BY-ID) (SUBSEQ (#:COLLECT-FOLLOWER-IDS SN) N (+ N 10))))

@                  =>   @MENTION

UL                 =>   USER-TIMELINE

@?                 =>   MENTIONS

DROP-LAST          =>   (LAMBDA NIL (DELETE-TWEET (CAR (USER-TIMELINE))))

TR                 =>   TRENDS

S                  =>   DO-SEARCH

STARTUP            =>   MOH

HOME               =>   COLLECT-HOME-TIMELINE

TY                 =>   (LAMBDA (X Y) (+ X Y))

PERSIST-ON         =>   (LAMBDA NIL (PROGN (USE-CACHE) (USE-DB :TWITTER-BDB-DRIVER)))

Called with the name of an alias it shows its definition :



CL-USER> (alias ul)

USER-TIMELINE

CL-USER>

An alias is defined by providing its name as the first argument and its definition as the second argument.
The first way to define an alias is by linking it to an existing api call. For example, (public-timeline) returns the current public time line. To alias it to the much terser (pt) you'd do something like this :



CL-USER> (alias pt public-timeline)

NIL

CL-USER> (alias pt)

PUBLIC-TIMELINE

CL-USER> (pt)

knears                                                                                                                  Thu Feb 17 11:13:07 +0000 2011

今日久しぶりに音声チャットをしたよね( 　ﾟ,_ゝﾟ)楽しいね



ANAYASMEL                                                                                                               Thu Feb 17 11:13:07 +0000 2011

Bueno dias tengan un super dia.....



erikafromhawaii                                                                                                         Thu Feb 17 11:13:07 +0000 2011

"I'm no detective-- OH! Wait a minute, where am I? I'M A DETECTIVE!" Beautiful, stunning, flawless, etc. Scott Caan, I adore you.

[...]

The second way to define an alias is to curry an api call. For example (dosearch search-term :max-pages pages) will search twitter for the search term you provide and will return the number of pages of 100 tweets each specified with the max-pages keyword). Here's how you create an alias for the word lisp. You can specify the number of pages when you actually call the alias :



CL-USER> (alias lisp (do-search "lisp")) 

NIL

CL-USER> (lisp :max-pages 1)

From: salsipuedes69                                                                                                                               To: 

http://twitpic.com/40vd0h - Cupid's bow seemed a little skewed. (aramisdeceives.com) Pray for the kindness of lisp. Socks

Fri, 18 Feb 2011 00:48:40 +0000



From: miyabipopo                                                                                                                                  To: 

ごめんよｗｗｗバスの運転手さんｗｗｗ気味の声よりもLISPの声のほうがいいんだもんｗｗｗ

Fri, 18 Feb 2011 00:46:35 +0000



From: t_fridge                                                                                                                                    To: 

RT @peccul: "Emacs Lisp Snippets - http://snipplr.com/all/language/emacs-lisp "

Fri, 18 Feb 2011 00:45:07 +0000



From: PMLEGEND                                                                                                                                    To: 

RT @LexiLondonBaby: Has serenity got some kind of lisp ? #muzikradio

Fri, 18 Feb 2011 00:42:19 +0000

----------------------------------------------------------

Lastly you have the ability to specify a lambda. Here's an example:



CL-USER> (alias ps (lambda (x y) (* x y)))

NIL

CL-USER> (ps 2 3)

6

CL-USER>

When an alias is defined it's added to an internal hash table and stored in a file which can be found in the alias sub directory of cl-twitter's repository.

Calling unalias is removes an an alias from the hash table, but not from the file.
To remove a single alias you'd call unalias with the alias name as the argument :
For example, this is how you remove the pt alias defined previously:



CL-USER> (unalias pt)

PT

CL-USER> (alias pt)

NIL

CL-USER> (pt)

; 

; caught STYLE-WARNING:

;   undefined function: PT

; 

; compilation unit finished

;   Undefined function:

;     PT

;   caught 1 STYLE-WARNING condition

; Evaluation aborted.

CL-USER>

Calling unalias without arguments removes all aliases.



CL-USER> (unalias)

#

CL-USER> (alias)

NIL

CL-USER>

As mentioned, the alias file is not updated and that provides some protection against an inadvertent (unalias).

The final sync of the has table with the alias file happens when the (done-twittering) is issued. This command closes the cl-twit-repl session and restores the default printers.
On the other hand, calling (cl-twit-repl) restores the alias table :





CL-USER> (cl-twit-repl)

mohegskunkworks              timezone : Eastern Time (US & Canada)   url     : http://www.mohegan-skunkworks.com                           created   : Sat Feb 06 00:16:20 +0000 2010

name : "fons haffmans"       statuses : 66                           friends : 98                                      followers : 18      following : NIL



--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

CL-USER> (alias)

[...]

PT                 =>   PUBLIC-TIMELINE

RL                 =>   #:RATE-LIMIT-STATUS

WHOAMI             =>   (LAMBDA NIL (WHEN *TWITTER-USER* (#:VERIFY-CREDENTIALS)))

H                  =>   (#:COLLECT-HOME-TIMELINE)

FT                 =>   FRIENDS-TIMELINE

SFOLU              =>   (LAMBDA (SN N) (MAPCAR (FUNCTION #:SHOW-USER-BY-ID) (SUBSEQ (#:COLLECT-FOLLOWER-IDS SN) N (+ N 10))))

@                  =>   @MENTION

UL                 =>   USER-TIMELINE

@?                 =>   MENTIONS

DROP-LAST          =>   (LAMBDA NIL (DELETE-TWEET (CAR (USER-TIMELINE))))

TR                 =>   TRENDS

S                  =>   DO-SEARCH

STARTUP            =>   MOH

HOME               =>   COLLECT-HOME-TIMELINE

TY                 =>   (LAMBDA (X Y) (+ X Y))

[...]

CL-USER>

Authorizing cl-twitter as a twitter client

noreply@blogger.com (mohegskunkworks) — Sun, 09 Jan 2011 23:40:00 +0000

Initially logging into your twitter account was fairly straight forward. You fired up your favorite client and supplied a user id and password and you were all set. That used the be known in twitterland as basic authorization.

Halfway last year twitter switched to using open authorization exclusively. A user under OAuth has the ability to share private resources with a client without handing over the it's login credentials to the client. So instead of providing a client application with your twitter access credentials so it could log into twitter on your behalf (and perhaps other unbeknown to you try those same credentials to access other resources or your private twitter information) you essentially log into twitter on the clients behest. twitter responds by giving the client a set of tokens which can be used to enable subsequent logins.
You now also have the ability to deny a client access to your account as well.
In addition, for this to work, the client application needs to be registered with twitter.

cl-twitter always had support for OAuth, but it was not set up to be used as a valid client application from the repl. That has now changed.Below are the steps you need to take when you use cl-twit-repl for the first time.

First you need to load the cl-twit-repl package. Then, do a (use-package :cl-twit-repl) followed by (cl-twit-repl). You"ll see a url printed by the repl and you" ll be challenged to provide an id (see the screen shot below).

Copy the url from the repl and past it into a web browser :

Twitter will ask you to allow cl-twit-repl access to your twitter account. After you press the allow button twitter responds with a pin number.

You copy that pin number and paste it back into the repl, where it says to enter the pin.

You should now see your user credentials appear on the repl.

Your access credentials are stored in a file called access.ht in the access subdirectory of the cl-twitter repository. .gitignore is set up to exclude access credentials from being tracked. I realize that this is very limiting assumption at the moment and I may provide an override hook in a later release.
If you want to customize this, the function to look for is access-file :

(defun access-file ()
(default-access-path "access/" *access-file*))

You exit twitter mode by calling (done-twittering).

Once access credentials can be located, subsequent calls of (cl-twit-repl) will not require this OAuth handshake with twitter. In addition, you won't be automatically logged in either. You would need to issue a (get-authenticated-user ) call.

If get-authenticated-user is unable to locate credentials for the login id you supply, it will take you through the whole handshake process. This allows you to manage multiple user accounts, although you can be credentialed for only one account at a time. get-authenticated-user can be called from the repl without being in "twitter mode" so you can use the api without being in "twitter mode".

The cl-twitter repl's pretty-printer

noreply@blogger.com (mohegskunkworks) — Thu, 06 Jan 2011 00:13:00 +0000

In my previous post I listed three minimum requirements for a useful twitter repl mode.

Sending tweets is all well and good but the cl-twitter repl has to provide a meaningful print representation of the data returned by the other twitter api calls.

Each json structure returned by twitter has it's own type in cl-twitter and I"ll call their instances 'twitter objects'. The default printed representation of those objects is pretty basic. For example, under the normal repl the tweets returned by (public-timeline) just show some basic information :

(public-timeline)
(#< tweet id:22800422360064000="" omiibby=""> 
 #< tweet fz_r="" id:22800417641472000="">
 #< tweet id:22800415959552000="" nattunkana="">
 #< tweet fcominhavidals="" id:22800412922880000="">
[....])

cl-twitter's show is a generic function which can be used to get a more detailed print out :



(defgeneric show (obj &optional stream ))

The results printed by show applied to the list returned by public-timeline is human readable:



(show *)

OmiiBby                                                                                                                 Wed Jan 05 23:43:38 +0000 2011

- #maarwaarom szeg je gewoon niet dat je gay bent ?



fz_r                                                                                                                    Wed Jan 05 23:43:37 +0000 2011

ベッドにつまずきながらかけこんだ。



nattunkana                                                                                                              Wed Jan 05 23:43:37 +0000 2011

何気ない言動でも人を幸せにできます＞ネットの書き込み、注意してる？～何気ない言動でも人を傷つけます！～



FCOminhavidaLS                                                                                                          Wed Jan 05 23:43:36 +0000 2011

eu tenho 3 caderno do luan ,2 boné e 4 CD e 5 DVDS ! tudo original ganhei pela central de fãs *-* acho que vou fazer promoooooooo =X

show's default specialization prints the default representation of non-twitter objects:



(defmethod show ( (obj t) &optional (s *standard-output*))

          (format s "~&~1t~S~%" obj))

On the other hand each twitter type has its own specialization. Here is how a tweet is displayed :



(defmethod show ((tweet tweet) &optional (s *standard-output*))

   (format s "~&~150<~a~; ~a~>"

(twitter-user-screen-name (tweet-user tweet))

                                      (tweet-created-at tweet))

    (format s "~&~A" (tweet-text tweet))

    (format s "~&~A" *seperator*))

The user information contained in the tweet is only printed selectively.
Containers of twitter objects are handled by mapping show over their content :



(defmethod show ((obj cons) &optional (s *standard-output*))

    (length (mapcar (lambda (el) (show el s)) obj)))



(defmethod show ((obj hash-table) &optional (s *standard-output*))

    (maphash (lambda (k v) (declare (ignore k)) (show v s)) obj))

The special cl-twitter repl mode overrides the default pretty printer with the show command for cl-twitter entities.

When the cl-twitter repl mode is invoked, the common lisp function copy-pprint-dispatch is called to store save current dispatch table by binding it to a dynamic variable. This is done using (pprint-dispatch 'cons) with the default pretty printer for conses as well. We need to do this because show 's specialization for conses would be invoked for all of them and not just the ones containing cl-twitter objects.

The alternative pretty printer pp-show is installed into the dispatch table using set-pprint-dispatch by mapping it over a list of twitter types :

 (defun install-new-dispatchers ()

     (mapcar (lambda (type) (set-pprint-dispatch type #'pp-show)) *twitter-types*))

pp-show is designed to route cons and hash table containers to the right pretty printer for their contents:



(defun pp-show (stream o)

    (if (or (twitterp o) (and (consp o) (twitterp (car o))) 

                                                       (and (hash-table-p o) (twitterp (car-ht-value o))))

      (show o (or *cl-twit-repl-stream* stream))

            (funcall *saved-pprint-dispatch-cons* stream o)))

If the object passed into pp-show is in the list of twitter objects it's routed to show. If the object is a container (either a cons or a hash table) one element is extracted and examined (car-ht-value s function which returns one value from a hash table). If that element is a twitter object, the object is similarly passed to show.

Containers returned by twitter contain only twitter objects. However, it 's possible for a user to create a cons of mixed objects. If the first element is not a twitter object the regular pretty printer will be used.

*cl-twit-repl-stream* is set to standard out and twitter objects are will show up differently on the repl from other objects.

The dispatch table is restored when the twitter mode of the repl is turned off.
So it turns out that it's pretty easy to customize the pretty printer and to make the repl a bit more usable for twittering.

twittering from the repl : Motivation and Requirements

noreply@blogger.com (mohegskunkworks) — Tue, 04 Jan 2011 03:12:00 +0000

A simple repl based client is part of the cl-twitter project. At first I thought of this more as a gag than as something potentially useful. Why would anyone use something other than one of the many twittter gui's ? Those gui's certainly look a lot more polished than a slime/swank repl.

But what if you wanted to filter out all the links in your time line ? Or wanted to see who your 20567th follower is ? Sure, you can search google for twitter managment tools that will give you that capability. But having the flexibility to script your own functions would be one potential advantage a repl based twitter client would give you.

I decided that the cl-twitter repl needed to meet at least these three requirements to be useful:

A useful representation of twitter's various classes and structures.
Automatic authentication when used for the first time.
The ability to create simple commands.

Let's look at each requirement in detail:

The default pretty print for twitter objects is pretty basic : It's a string naming the object, plus one or two fields. For example, search results are stored in a hash table and that's all that's printed at the repl :



CL-USER> (do-search "hello" :max-pages 1)

< hash-table 100="" :count="" :test="" equal="" {10039eef21}="" >

CL-USER;

It's obviously a lot more useful to see expanded results printed to stdout. cl-twitter's show command is set up to accomplish this.Calling show on the output of do-search :



(show (do-search "hello"))

results in output like this :



[....]

From: OhMyBiebsss                                                                                                                       To: CorbinBleu

@corbinbleu it would be legit if you tweeted me a simple HELLO EUNICE. (:

Mon, 03 Jan 2011 23:54:09 +0000



From: maupadilla89                                                                                                                                To: 

Feeling like: Hello, I love you Won't you tell me your name? Hello, I love you Let me jump in your game... Hello, Hello, Hello,I want you

Mon, 03 Jan 2011 23:54:09 +0000



From: OmgashBieber                                                                                                                                To: 

RT @Mrs_LautnerMars: @OmgashBieber Hello Yasmin. ♥

Mon, 03 Jan 2011 23:54:08 +0000



From: CastielTheOnly                                                                                                                To: GritsandCoffee

@GritsandCoffee Hello.

Mon, 03 Jan 2011 23:54:08 +0000



From: _KONKRETE_ROSE                                                                                                                              To: 

MOTHERRRR!!! MJ voice! Tell her I said HELLO! ^__^ @OrangeBone_ About to dye my mom's hair I wish she would keep it gray. Gray is beautiful.

Mon, 03 Jan 2011 23:54:07 +0000



From: Beauty_Bahja                                                                                                                                To: 

Hello everybody Missed Me ? Lol

Mon, 03 Jan 2011 23:54:07 +0000

The first requirement requires replacing the default pretty printer for the objects defined in cl-twitter with the show command.

The second requirement aims to streamline the authentication process for first time users. That way you get the initial OUATH handshake with twitter out of the way. There is by the way no need to login for most read operations, like searches. You just have a much lower rate limit.

The third requirement covers the creation of (simple) functions without necessarily creating an additional lisp source file. I basically envisaged three use cases :

aliasing : Rather than type out (public-timeline) every time you my want to alias it so that you can use (pt) instead.
currying : Suppose you want to search for "#lisp" on a regular basis, you should be able to do something like :



(alias s (do-search "#lisp"))

Subsequent invocations would still respect the :max-pages key word of do-search viz.:



(s :max-pages 1)

(s :max-pages 5)

lambda : You should have the ability to create you own functions by combining cl-twitter api calls with regular lisp functions. For example, suppose you wanted to write a function to get more details on a particular user's followers. collect-follower-ids returns a user's followers as a list of the twitter user ids. You can then map show-user-by-id over (part of the) list to get each user's details. Each call of show-user-by-id goes against the rate limit so you don't want to just blindly apply it. A reasonable implementation would be :



(alias lof (lambda (screen-name count) 

               (mapcar #'show-user-by-id 

                   (subseq (collect-follower-ids screen-name)

                              count (+ count 10))

That way (lof "twitterapi" 100) would return 10 user credentials, starting at position 100 in the list returned by (collect-follower-ids "twitterapi").

Clearly, the following three declarations should be equivalent :



(alias pt public-timeline)

(alias pt (public-timeline))

(alias pt (lambda () (public-timeline)))

Initially I was quite intrigued by land of lisp's approach of constructing you're own repl. But it turned out that such a radical approach was not necessary. In fact, it was fairly easy to tweak the existing repl to meet all three requirements, as I"ll detail in the next few blog posts.

creating list time lines with cl-twitter.

noreply@blogger.com (mohegskunkworks) — Sun, 26 Dec 2010 21:54:00 +0000

A list on twitter is an alternative way to keep track of status updates (or tweets) from a group of people without necessarily befriending or following them. A twitter user has one of three relationships to a list: he's either the list owner, a member of the list or a subscriber to the list.

If you look at twitter as an information gathering device, lists that are somehow linked to you are a reasonable way to discover information that maybe relevant to you. This is pretty obvious when you're the creator or subscriber of a list. Equally, when someone has added you as a member that's an indication they think you and that group of users have something in common.

This post will show how you can use the twitter api (through cl-twitter ) to get status updates (or tweets) from the lists a user has created, the lists he's a member off and the lists he has subscribed to.

Twitter provides a simple call to see the tweets of the people you follow like the home timeline or the friends timeline. The equivalent list statuses api is not so straightforward to use, as you need to provide not only the list owner but also the list id. We need to combine this with the user list api call to get all the (public) lists this user has created. Similarly the membership and subscriptions api calls provide the lists the user is a member of or is subscribed to.

The list statuses api breaks the results into pages and the page option can be used to create a read cursor over the result set. This is twitter's usual way to handle (potentially) large results set of tweets. cl-twitter's has a macro that encapsulates this paging mechanism. There's an additional macro that will handle results sets consisting of tweets.
Here are the functions used to retrieve the time lines :



(defun collect-user-list-statuses (list-owner list-id &key (max -1)

                                                            (skip 0) 

                                                  (container (make-hash-table  :test 'equal

                                            :size 100)))

     (collect-tweets (:max max :skip skip :container container) 

                               statuses-user-lists        :list-owner list-owner

:id list-id

                                        :page 1))



(defun user-list-timeline (list-owner &key (max-per-list 1) )

    (let ((user-list-ids (mapcar #'list-type-id 

                (cursor-user-lists-lists (get-user-lists :list-owner list-owner))))

               (container (make-hash-table  :test 'equal :size 100)))

                (dolist (list-id user-list-ids)

                   (setf container

(collect-user-list-statuses list-owner

                                           list-id 

                                           :max max-per-list

                                           :container container)))

     container))



(defun member-list-timeline (user &key (max-per-list 1) )

    (labels ((user-list-data (ul)

         (list (twitter-user-screen-name (list-type-user ul)) (list-type-slug ul)))) 

       (let ((ulst (mapcar #'user-list-data 

                             (cursor-user-lists-lists (memberships-user-lists :screen-name user))))

         (container (make-hash-table  :test 'equal :size 100)))

                     (dolist (item ulst)

            (setf container

(collect-user-list-statuses (car item)

                                          (cadr item) 

                                          :max max-per-list

                                          :container container)))

      container)))



(defun subscriber-list-timeline (user &key (max-per-list 1) )

    (labels ((user-list-data (ul)

          (list (twitter-user-screen-name (list-type-user ul)) (list-type-slug ul)))) 

       (let ((ulst (mapcar #'user-list-data 

                       (cursor-user-lists-lists (subscriptions-user-lists :screen-name user))))

         (container (make-hash-table  :test 'equal :size 100)))

              (dolist (item ulst)

           (setf container

(collect-user-list-statuses (car item)

                                          (cadr item) 

                                          :max max-per-list

                                          :container container)))

         container)))

collect-user-list-statutes uses the collect-tweets macro to retrieve tweets by paging through the result set. The api does not provide a good stopping criterium. collect-tweets stops collecting when the size of the hash tables doesn't change anymore, or when the number of pages specified by the :max keyword parameter is reached.

user-list-timeline, member-list-timeline and subscriber-list-timeline all have a similar structure: After collecting the lists the user either owns, is a member off or subscribes to, it collects all the tweets of the users in those lists in a hash table. In all three cases I retrieve only the first page. Here's a quick demo to illustrate this :

I 'm subscribed to two lists, both of which are set up by other twitter users :



(show (collect-user-list-subscriptions "mohegskunkworks"))



semantic-web                      @mark_l_watson/semantic-web                            

id : 956750 owner : mark_l_watson members : 6 mode :public 



functional                        @crtvzen/functional                                    clojure/scala and related

id : 6028588 owner : crtvzen members : 37 mode :public

To get the status updates use :



(show (subscriber-list-timeline "mohegskunkworks"))

kendall                                                                                                                 Sun Dec 26 20:34:43 +0000 2010

Generally, ferreting out an assumption of a universal value isn't a very potent gotcha in my view.

kendall                                                                                                                Sun Dec 26 20:05:42 +0000 2010

@kendall Btw, not at all clear to me 'science' can even be coherently described w/out reference to many values, most of all truth seeking.

kendall                                                                                                                 Sun Dec 26 19:33:54 +0000 2010

My rabbit loin and liver cigars with chard and fennel turned out very well. http://yfrog.com/h6sufrj

[...]

debasishg                                                                                                               Sun Dec 26 10:36:40 +0000 2010

I like Erlang syntax .. is that weird ? It so closely models the semantics of the language.



jorgeortiz85                                                                                                            Sun Dec 26 06:19:47 +0000 2010

Just got out of TRON 3D!!! And I have to be at the airport in 5 hours... No sweat. (@ Cinépolis Valle Oriente) http://4sq.com/eedJmQ

This returns forty status updates at the moment, which I've abbreviated above to just a few.
The argument to the three list timelines above can be any twitter user. In order to scan through the lists user fogus is a member of you'd do :



(member-list-timeline "fogus")

I've not used the show function to print out the results set of the hash table returned by member-list-timeline.



(user-list-timeline "fogus")

As you can see this user has set up a list, which generates at least one page of tweets.

Exploring Mid Town Manhattan with Twitter's geo search api

noreply@blogger.com (mohegskunkworks) — Fri, 24 Dec 2010 17:02:00 +0000

What if you were new in town, pulled out your mobile device and immediately saw all the restaurants a mile or so around you ? What if you could also see which of them friends in your network had visited ? Or better yet, the tweets the local food critic sent out ?

Twitter extended it's api to search for tweets about places which would be one step in that direction. In addition, twitter 's geo resources api allows you to look up location information.
goe search allows you to search for locations given a longitude and latitude.

In cl-twiter you can execute the api call like this :

(defun geo-search (lat long &rest args &key (query nil) (granularity nil) (max-results 20) (ip nil) (accuracy nil) (contained-within) (attribute->street_address) )

(declare (ignore query granularity max-results ip accuracy contained-within attribute->street_address))

(apply 'twitter-op :geo/search :lat lat :long long args ))

The granularity can be "poi" (point-of-interest), "neighboorhood", "admin", "city" or "country", where "neighborhood" is the default.
To look for "points of interest" around Times Square (Latitude = 40.7590, Longitude = -73.9845 according to this satellite signals website ) you'd do something like this:

(show (geo-search 40.7590 -73.9845 ))

 The output looks like this :

 neighborhood  Theater District - Times Square, New York                        NIL                                     United States

neighborhood  Midtown Center, New York                                         NIL                                     United States

neighborhood  Clinton, New York                                                NIL                                     United States

city          Manhattan, NY                                                    NIL                                     United States

city          New York, NY                                                     NIL                                     United States

By adding the :granularity "poi" option, we get all the points of interest at this location :

TWIT> (show (geo-search 40.7590 -73.9845 :granularity "poi" :max-results 100)) 

poi           Sbarro, New York                                                 701 7th Ave, Frnt 1                     United States

poi           Pig'n'  Whistle, New York                                        165 W 47th St # 2                       United States

poi           New York Guest, New York                                         1560 Broadway                           United States

poi           McDonald's, New York                                             1560 Broadway, Frnt 2                   United States

[.....]

poi           Palace Theatre, New York                                         1564 Broadway                           United States

poi           Adam's Family Merchandise, New York                              205 W 46th St                           United States

poi           Charlotte Wilcox & Co, New York                                  1560 Broadway                           United States

poi           Times Square Gift Shop, New York                                 1560 Broadway                           United States

poi           Actors Equity, New York                                          1560 Broadway                           United States

neighborhood  Theater District - Times Square, New York                        NIL                                     United States

neighborhood  Midtown Center, New York                                         NIL                                     United States

neighborhood  Clinton, New York                                                NIL                                     United States

city          Manhattan, NY                                                    NIL                                     United States

city          New York, NY                                                     NIL                                     United States

Moving slightly south we get a different set of points :



(show (geo-search 40.7580 -73.9845 :granularity "poi" :max-results 100))

poi           Le Marais, New York                                              150 W 46th St                           United States

poi           Shubert Archive, New York                                        149 W 45th St                           United States

poi           Lyceum Theatre, New York                                         149 W 45th St                           United States

poi           O'Lunney's, New York                                             145 W 45th St                           United States

poi           Nails Town, New York                                             157 W 46th St                           United States

poi           Drummers World, New York                                         151 W 46th St # 3                       United States

poi           Havana Central Times Square, New York                            151 W 46th St                           United States

[...]

poi           Margon Restaurant, New York                                      136 W 46th St # 1                       United States

neighborhood  Theater District - Times Square, New York                        NIL                                     United States

neighborhood  Midtown Center, New York                                         NIL                                     United States

neighborhood  Midtown South Central, New York                                  NIL                                     United States

city          Manhattan, NY                                                    NIL                                     United States

city          New York, NY                                                     NIL                                     United States

To be honest, the point of the geo api is somewhat lost on me. Judging from the results the goal of the api seems to be to provide a geographic anchor to the status updates. Since the inputs are longitude and latitude, the api feels like it's designed to work with a device, where this type of data is fairly easily available.

The api supports query strings, but in my admittedly limited testing I've not been able to get data returned using just a search query.

The api would be more useful in my mind if it returned a all the places by category (like restaurants or museum) within a reasonable distance. Those results could than be cross referenced with status updates from users in the same area or the businesses themselves. That way you could connect with someone at the location and get some idea if it's worth going there.

cl-twitter's api is feature complete

noreply@blogger.com (mohegskunkworks) — Mon, 13 Dec 2010 01:07:00 +0000

When I started this project I created a spreadsheet showing the state of each rest api call. The initial version of cl-twitter covered 53 out of a total of 96 rest calls of specified by the twitter api.
At this stage cl-twitter covers the twitter api completely.

What that means is that all the rest api calls can be made through cl-twitter, and the reply is converted to a data type. The data types cover a large part of the twitter entities. What's missing is support for tweet entities and for some of the geo data (notably yahoo's where on earth identifiers (woeid)) .

There is benefit to adding entities, and it should be straightforward. Adding WOEID's on the other hand is not high on the priority list.

The next stage is to build out the twitter repl application, which will be based on the api calls. In addition, I"ll also take a look at the persistence interface.

Searching twitter with cl-twitter

noreply@blogger.com (mohegskunkworks) — Tue, 07 Dec 2010 10:45:00 +0000

Twitter's search api provides the interface to twitter's huge collection of tweets. For anyone studying the social phenomena that is twitter, having access to these is invaluable.

The search api is somewhat distinct from the other rest api calls. To underscore this point, there is an outstanding bug where the client id's returned by the search api don't match up with those returned by the rest of the api. In addition, because the search api has its own rate limiting, calling rate-limit-exceeded won't give you the remaining number of calls.

The search api returns at most 1500 results, in chunks of at most 100. The chunk (or page) size is set by the rpp (results per page) option. Subsequent pages are accessed by the page page parameter. This feature of the search api is similar to the cursor functionality of the friends and followers api calls.

The basic api call is search-twitter :

(defun search-twitter (query &rest args 
                                  &key (callback nil) 
                                       (lang nil) 
                                       (locale nil) 
                                       (rpp nil) 
                                       (page nil)
                                       (since-id nil) 
                                       (until nil) 
                                       (geocode nil) 
                                       (show-user nil) 
                                       (result-type nil) )
  
     (declare (ignore callback lang locale 
                     rpp page since-id until 
                     geocode show-user 
                     result-type ))
      (apply 'twitter-op :search :q query 
              (rem-nil-keywords args '(:callback 
                                       :geocode 
                                       :lang 
                                       :until))))

The query is the main argument. The keywords :rpp and :page control the paging as mentioned above.
Paging is encapsulated by the with-paging macro. The main search api call do-search is set up to page over the largest possible result set :

(defun do-search (query &key (max-pages 15) )  
   (let ((ht (make-hash-table  :test 'equal :size 1500)))    
           (labels ((collect-it (slst)        
                    (dolist (item slst)   
                        (setf (gethash (search-ref-id item) ht) item))))      
    (with-paging (:collector #'collect-it :max-pages max-pages :test #'rate-limit-exceeded ) 
                         (search-twitter query))     
   ht)))

As you can see the result are returned in a hash map keyed on the (unique) search id. The value of the has table is a search reference :

(define-element search-ref ( (metadata search-ref-metadata))
  "An individual search reference"
  (id "Integer ID of message" nil)
  (text "The text of the tweet" nil)
  (to-user "Screen name" nil)
  (to-user-id "ID of receiver" nil)
  (from-user "Screen name" nil)
  (from-user-id "ID of sender" nil)
  (iso-language-code "Two character language code for content" nil)
  (source "Source of the result" nil)
  (profile-image-url "The profile image of the sender")
  (created-at "The date of message creation" nil)
  (metadata nil "")
  (geo "FIXME unparsed"))

You can use the print-search-ref function to pretty-print the output.

A typical session would look like this :

TWIT> (setf *R* (do-search "Ponies"))
#
TWIT> (maphash (lambda (k v) (print-search-ref v)) *R*)
[...]
[Mon, 06 Dec 2010 07:54:53 +0000] TheMightyAlexx_:    I love ponies and sunshine and *falls* privates!
[Mon, 06 Dec 2010 07:52:51 +0000] HabboGuy:           @Dq231 I LIKE PONIES AND SUNINE AND DOOSH SNOOKIE
[Mon, 06 Dec 2010 07:51:41 +0000] stationbred:        That fancy Spanish horse and his grassyarse. Why can't he just say "thanks" like the rest of us common or garden ponies...
[Mon, 06 Dec 2010 07:37:24 +0000] DSchiller:          Off to see the ponies..in the rain! Looove
[Mon, 06 Dec 2010 07:29:21 +0000] BarackOBoogie:      "@Meezy_TaughtYou: @snazmindanielle Did you just call us ponies?? YOU WERE A DOLPHIN!" (Remember That Noise!)
[Mon, 06 Dec 2010 07:28:44 +0000] Meezy_TaughtYou:    @snazmindanielle Did you just call us ponies?? YOU WERE A DOLPHIN!
[...]

Querying the social graph with cl-twitter

noreply@blogger.com (mohegskunkworks) — Tue, 30 Nov 2010 02:40:00 +0000

Among other things the twitter api provides the ability to obtain the user ids of a friends and followers of a twitter user. You can use a cursor when there are more than 5000 friends or followers. In fact, if you don't use a cursor and the result set exceeds some predefined limit the call will return an error code.

The cl-twitter api of the basic calls is fairly simple. The ids of the users friends are returned by :

(defun friend-ids (screen-name &key (cursor -1))
  (apply 'twitter-op :friends/ids :screen-name screen-name 
                     :cursor cursor nil ))

(defun follower-ids (screen-name &key (cursor -1))
  (apply 'twitter-op :followers/ids :screen-name screen-name 
                     :cursor cursor nil ))

The real fun begins when the list of followers or friends is large enough to require cursors. Cursor mode is invoked by setting the cursor to -1. As mentioned before the use of a cursor is optional, but will result in an error should the size of the results returned exceed a predefined limit. There is in other words, no reason not to issue each call with the cursor set to -1, the initial cursor value. If there are multiple result pages the response will include the next cursor value to use on the subsequent call. The friend-ids and follower-ids functions shown above can be used to issue subsequent calls by setting :cursor to the next cursor value. However, it's easier to let the with-cursor macro handle all of those details.

Here's an example of how with-cursor is used :

(defun collect-follower-ids (screen-name &key (max -1) (skip 0))
  (let ((lst))
    (labels ((collect-it (l)
        (setf lst (nconc lst l))))
      (with-cursor (:skip skip :max max :extractor #'cursor-id-ids 
                    :controller #'cursor-id-next-cursor 
                    :collector #'collect-it :test #'rate-limit-exceeded ) (follower-ids screen-name)))
    lst))


(defun collect-friend-ids (screen-name &key (max -1) (skip 0))
  (let ((lst))
    (labels ((collect-it (l)
        (setf lst (nconc lst l))))
      (with-cursor (:skip skip :max max :extractor #'cursor-id-ids 
                    :controller #'cursor-id-next-cursor 
                    :collector #'collect-it :test #'rate-limit-exceeded ) (friend-ids screen-name)))
    lst))

with-cursor manages the retrieval of successive calls to get the full social graph. It allows the caller to specify a collector as well as a test. Typically the collector will be a closure used to collect the result set into an appropriate data structure. In this case it's simply a list.
The test function is used to give the caller the ability to control the loop in which the data retrieval is done. rate-limit-exceeded returns true if twitter's rate limits have been reached.
Here's a typical session. Notice the decrease in available queries.

CL-USER> (rate-limit-status)
#
CL-USER> (length (collect-followers "sarahpalinusa"))
318181
CL-USER> (rate-limit-status)
#
CL-USER>

a note on api design

noreply@blogger.com (mohegskunkworks) — Sun, 21 Nov 2010 20:50:00 +0000

An api to a web service api like cl-twitter has two purposes. The first one is to translate the web service api into native language primitives, constructs and conventions. This is the api as 'driver' to the web service. At this level the api simply exposes the twitter api to the developer using language constructs he's familiar with, and he should be able to build applications using the api.

For cl-twitter this means things as simple as dashes instead of underscores in names, to using conses as lists, to using macros to define a dsl to specify return types, like the define-element macro. Required fields are function arguments and optional fields can be passed in using keywords.

There is also a second level api. Here, driver level api calls are combined to form building blocks by combining driver calls into logical units. Perhaps they form a simple sample application.
cl-twitter will have some of this as well, but the focus is on providing a complete driver level api first.
In the case of cl-twitter the example application allows twitter to be accessed from the repl.

At this stage there's also some functionality in cl-twitter which allows tweets to be stored in a database using the elephant package. This is a separate package. I haven't looked at this yet. Functionality like that is certainly useful, but would not want to tie it to a specific package.
Here I think the api should be simple enough so that multiple db packages can be used through package specific driver interfaces.

Extending the api

noreply@blogger.com (mohegskunkworks) — Tue, 16 Nov 2010 03:15:00 +0000

I compared the current twitter api specification and cl-twitter's. Obviously things have changed a bit since cl-twitter was first conceived.
In cl-twitter api calls are defined using a macro called "define-command". For example, the following specifies the public-timeline call:

(define-command statuses/public-timeline (:get (:tweet))
(twitter-app-uri "statuses/public_timeline.json")
"Returns 20 most recent statuses from non-protected users"

  :trim_user "When set to either true, t or 1, each tweet returned in a timeline will include a user object including only the status authors numerical ID. Omit this parameter to receive the complete user object."

:include_entities "When set to either true, t or 1, each tweet will include a node called entities. ")

The :get keyword means that it's a get request. The (:tweet) specification defines the return type as a list of tweets. Tweets are a class defined in cl-twitter.
I decided to keep naming conventions in the command definitions as close to the ones in the twitter api specification as possible. That way it's easy to determine which ones have been defined and which ones haven't.
As it stands now, the api commands defined by cl-twitter fall into three categories:

Fully functional where the return type is a well defined (list of) structure.
Functional, but where the return type is an association list.
Not implemented

An example of the first case are the time-line resources.
An example of the second case are the users/suggestions. Currently cl-twitter doesn't have structure or class encapsulating a 'twitter suggestion'. An other example are the saved searches resources or the geo resources.
Some calls fall into both classes, depending on whether you want to use the cursor facility. Cursors are currently not supported very well.
The third case covers the streamed tweets resources a.k.a. the streaming api's as well as the the three list management resources.
You can see a summary of the current capability here. Out of the 96 calls, 53 are already fully functional, and 19 are functional, but have an association list as a return type. Once that's fixed, cl-twitter would cover 75 % of the twitter api.

cl-twitter blog

noreply@blogger.com (mohegskunkworks) — Sun, 14 Nov 2010 17:01:00 +0000

The purpose of this blog is to let you know what's going on with cl-twitter.
The original cl-twitter code can be found here, and can be installed using
"darcs get http://www.common-lisp.net/project/cl-twitter/darcs/cl-twitter".

Unfortunately, the code found at that location doesn't compile.
In addition, twitter discontinued it's simple authorization scheme. All applications are expected to follow twitter's oath protocol.

My goal is to fix the code where necessary and extend it to cover the full twitter api.