I’m not going to do a dissertation on the nuances of using cURL. Instead, I’d like to discuss how to process data after you’ve gotten the HTML string from a page using the curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); setting. The general process using cURL to get the data from a password protected site goes something like this:

1. Make sure your “cookie jar” is set up and working so that your session can be saved while you’re scraping other pages after login.
2. Get the HTML for the login page.
1. Get the form post location.
2. Get any special tokens or hidden variables.
3. Post to the login script with your authorized username and password.
4. Scrape any pages you need to get and process them.

Getting auth tokens and stuff

Sometimes login pages make it difficult to figure out what to post because of “authentication tokens” or similar variables that change every time a page is hit and then have to be posted back along with the username and password. This is most common among .NET applications since it’s built right in to the .NET form creation classes. The process of finding these variables is made a lot easier if you use phpQuery. It’s a project that attempts to replicate what jQuery does but with PHP. You pass phpQuery a string with all the HTML content in it and then you can perform selectors and traverse the DOM just like with jQuery. What you do with the HTML after that is up to you.

In the above example, I already had the HTML string of the login page and simply used phpQuery to get all the inputs on the page with a very jQuery-like selector syntax and then looped through the results to get all my input fields for the form so I could use them in my next call to post the data back to the login script.

Getting poorly formatted data and stuff

I consider “well formatted” data to be in formats like JSON, XML, CSV, etc. that are specifically meant for data transfer. But what heppens when you need to scrap data from HTML tables or DIVs?

phpQuery to the rescue! You can do the same thing as above but parse the newly acquired HTML string of the data you’re looking for. Here’s an example of looking at a two column table of data to get all sorts of neat stuff and format it into an array. Then I check for an element outside of the table loop and set another variable.

Using the data and stuff

After you’re done with that, you have all this useful data in an array and you can basically do whatever you want to it. Since this example I created originated from code used in a real life client project (with variable names changed to protect the innocent), I went on to take that array and save it out to a JSON file that I could easily read in and do what I wanted to with the data.

You may need to go through some tests to get the right kind of cleaning or data sanitization for your values, but phpQuery makes it really easy to scrape this kind of data if you’re used to jQuery’s selectors and traversing. Instead of scraping the site each time to get the data, I like saving out the html string to a file and then playing with selectors and traversing to get the data I want form a local file so I don’t put a lot of strain on the server.

And don’t forget stealth…

If you’re not sure how friendly the server (or serveradmin) is to you doing this, make sure you set the cURL user agent for each request. I like  pretending I’m GoogleBot, but any common user agent string would suffice so that the server doesn’t explicitly know you’re PHP trying to log in.

By default, cURL spits out a user agent like this (changes based on operating system and version of cURL):

curl/7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5

If a serveradmin sees that in their logs, they might freak out… so  I like these user agent strings for sneaky data pulls:

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.77 Safari/535.7

Audience participation

How do you use cURL and phpQuery? Any useful scripts or resources to point people to other than their respective documentation sites?

Tell us a little about what Skookum Digital Works does and how it was founded?
Skookum Digital Works is a technical partner for anyone with a startup dream. We mainly work with entrepreneurs—building out their products or customizing software for businesses. 

My co-founder and I, Bryan Delaney, founded Skookum in 2005. We were roommates at UNC Charlotte, both graduated with computer science degrees, and both went to work for the Department of Defense for a few years before deciding we had a better mousetrap. 

Bryan and I are Charlotte natives, and we’re happy to have located our office Uptown. 

Do you specialize in any particular types of apps or client base?
I don’t think its crass of me to say we like to work with funded startups. Our typical client is the non-technical entrepreneur; someone with business skills and ideas but with no programming background. We allow a non-technical entrepreneur to get started on their digital product without having to find a technical co-founder or trying to hire engineers they are not qualified to vet. 

Our clients have often heard the word “No” elsewhere. We have expertise in the mobile web, complicated software integrations, and real-time web collaboration. 

What are some examples of apps Skookum has created?
A publishing company wanted a marketing tool to promote their books. SDW gave them a digital revenue stream.

Some D.C. folks knew independent voters were eager to take collective action. SDW built them a data mining and people matching system.

A group of investors-and-avid-golfers hated the 100+ scoring apps already available. We made players’ phones talk to each other. (!)

A neighborhood of New York businesses disliked Groupon keeping their margins. We created a localized model they collectively controlled. 

What do you think will be the next innovation in smartphone applications?
Mobile apps are going to be easier to download and live outside the walls of the Apple iTunes store and the Android Marketplace. Companies can now place their applications on iPhones and iPads without Apple taking 30% of the cut. 

Mobile applications are also slowing making their way into retail. Most stores know they can use smartphones and tablet devices to enhance their store experience, but the smaller chains (and certainly the local guys) are waiting to see what the big guys do before making the investment. 

Charlotte isn’t really seen as a tech town. Will that ever change?
It’s fundamentally a marketing problem and one that we and 70+ other Charlotte tech leaders have addressed face to face with Mayor Foxx. Aside from numerous local startups and technical partners like ourselves, all of the banks are essentially technology companies. 

For just one example, If you took all the programmers out of Bank of America, they would comprise the tenth largest tech company in the world. BoA has technology needs that make engineers at IBM cry. 

We like to think we’re doing our part recruiting talent to the area and flying the flag in front of national entrepreneurs. We have clients all over the place happy to come see us and come visit Charlotte. 

Charlotte is already a tech hub, but the city definitely needs to get better about spreading that message. 

weinre. Web Inspector Remote. This is what happens when developers make and market a product.

weinre.

Weinre is a tool for remotely debugging your web pages. Let’s say you’re building a responsive web site or a mobile web app. iOS’s Debug Console certainly can’t be the most helpful option for peeking behind the curtain. This is where weinre comes in.

Recently being brought into the Phonegap fold (and therefore Apache and the Apache Incubator), weinre is undergoing active development. You can learn about weinre at
phonegap.github.com/weinre/Home.html or callback.github.com/callback-weiner (callback is the Apache Incubator github account).

Watch a demo
on Youtube

Here, you’ll learn that if you’re on OSX there is an app to get you up and running fast. You’ll learn that it’s a javascript-only implementation of the Webkit Inspector. You’ll learn that there are three parts (each with the word “Debug” in it’s name). But what you won’t learn is how to open up your localhost development server to outside devices. For that you came here.

To begin using weinre to remotely debug your projects, simply follow this six step program.

1. Download the mac package from https://github.com/callback/callback-weinre/archives/master
2. Create the directory and file

   ~/.weinre/server.properties

   with the following content:

   boundHost:  -all-

3. Grab this bookmarklet Weinrize it!
4. Launch weinre
5. Run local server and weinre on the same machine while opening the website on your remote device.
6. Bask in the glory of Webkit Inspector and iOS Development.

Disclaimer: Opening up your computer with all boundHosts has security implications.

Happy debugging.

A little back story:

I wrote a pretty neat WordPress plugin and had it hosted on GitHub. Basically, it takes an iTunes App Store application ID and fetches all the data about it and displays it on a WordPress site (while caching it for you in the WordPress database).

So I wanted to release it to the world using the WordPress plugin directory… the go-to location for finding plugins. If it didn’t live there, it basically had zero chance of getting any usage whatsoever. Nobody goes searching GitHub for WordPress plugins.

There was one problem…

WordPress uses SVN as the required source code management tool for publishing plugins. You clone check out the repository, make your changes, commit them and then tag the new version and the WordPress plugin directory detects the change and within a few minutes, your updated plugin is released to the world and every site that is using your plugin gets a little message at the top of the admin bar asking them to update their version of the plugin.

Except I haven’t used SVN in about 4 years. Basically the whole world has moved to GIT. There’s been a exodus from SVN, CVS, Perforce, Mercurial, etc. because of GIT’s ease of use and distributed model.

GIT to the rescue

Recent versions of GIT have really focused on integrating SVN as much as possible. The git-svn command is powerful enough to clone an entire SVN repository and allow you to work on it like it’s a GIT repository, then commit your history back to the SVN server, and the SVN server is none the wiser.

So I knew there had to be an easy and/or automated way to publish and then submit subsequent updates to my application.

tl;dr bro

I know, I know…

Which brings me to my point:

With a little Google-fu, I found Brent Shepherd’s blog post about his Automated WordPress Plugin Deployment script. Basically, it’s a bash script that you download and put into every plugin repos and change a few variables. When you’re don’e working on some features, just run ./deploy.sh and it automatically, checks out the SVN plugin, updates it with all your changes, making sure to ignore your GitHub “README.md” and the deploy script itself. Then it tags the new version and commits everything back to the WordPress server. It even aborts if it detects your readme.txt file stable version number doesn’t match your plugin’s declared version.

Thanks, Brent, for the fantastic deploy script. It worked flawlessly on the first try and my new plugin has been published.

Octocat to the rescue.

Having your app register a URL scheme on a device means that you can open an installed application on a user’s iPhone or iPad in ONE CLICK from their email (or the web, or another application). There are all sorts of apps already using this functionality.

Use Case

We just used this for a project where the client would send confirmation emails to the user after signup. The email would contain an link to a URL. The URL opens in mobile Safari, then launches the app and seamlessly logs the user into the app.

That’s one use case; there’s probably others (let us know yours). Another one could be a URL embeded in text—say someone is reading on their iPad and the content could prompt them to hop over to the app.

Try It Out

If a user has your app installed on their device, all you have to do is send them to my-great-app://whatever/ and Safari will resign to the background and your app will open. Why don’t you try it? I know you’ve got the Facebook app installed on your iPhone. Just clear your address bar and type (or click the link): fb://profile

Go ahead, I’ll wait.

It opened the Facebook app and took you to your profile page (or gave you a big fat cryptic error message if you don’t have the Facebook app installed). Facebook allows a LOT of options that you can pass to the application to take you into various parts of the app.

Process

So enough about the Facebook app. What does a developer do if they want to pass a bunch of data to the app—without actually defining what needs to be passed using a bunch of if/else or switch statements?

It’s fairly trivial to add url scheme to your application’s Info.plist:

1. Register the url scheme in your Info.plist. Lets say: my-great-app://
2. In the application delegate, implement this method and return YES:

   application:openURL:sourceApplication:annotation:

3. That’s it!

I wrote a method you can add to your application that allows the passing of data through the url with key/pair values like so:

my-great-app://user_id/27/token/really9385long5892data/email/user@yourservice.com/

When the user clicks a link like that, or you auto-redirect them from your site to that URL, you can parse the URL into an NSDictionary with this code. Here’s the method and an example implementation along with some pretty badass comments so you know what the heck is going on:

Let me know in the comments if you find this useful at all or if you make changes to it.

Never is developmental speed more apparent than when I think back to my pre-SDW sales life.

I ran a software company.

Same industry.
Same clients.
Same challenges.
And two completely different approaches to solving those same business needs.

Perhaps my impressionable years of growing up in a Parochial school operated by nuns armed with wooden rulers lead to habits that played a role in my partiality for an inflexible, doctrinaire approach to application development.

I thought our 350+ page specifications documents were awesome. I thought our inflexible system produced predictable results. I thought our lumbering process equated thoroughness.

That all changed when I fell in love with Skookum. SDW was the company I always envisioned building.

SDW was nimble.
SDW was able to change course and turn on a dime.
SDW was fast, unfussy, AND thorough.

Importantly, SDW also had a full toolbox. At my .NET shop, we had one hammer, so every problem looked like a nail.

Skookum has its own creative department. Instead of waiting for innovation to come from Microsoft, SDW makes their own. Gone are the square buttons and pull downs of the .NET environment (and expensive licensees), and instead, I can now give clients, chic, state-of-the art solutions emphasizing intuitive design and natural functionality.

I admit it, I’ve become an Agile convert and enjoy nothing more than winning over clients both former and new. I like showing how they too can be better, faster, and more nimble. Ahead of schedule, and under budget. No fuss, just shipped code.

I used to think complex web-based application development meant one thing—mass.

Now I know I was wrong. Whoever said “Speed Kills…” has obviously never worked with Skookum Digital Works before.

Most MVC Frameworks today come with some form of sanitization or input filtering class built in. The problem with many of these libraries is that they fail to clean some of the more creative attack vectors. To combat this, some people use drop in libraries like OWASP AntiSamy or HTML Purifier to ensure their data is getting scrubbed clean. HTMLPurifier, for instance, uses a Smoke Test via the ha.ckers.org XSS attack list (the de-facto standard for finding attack vectors to test, might I add) to ensure they’re cleaning anything and everything.

So What’s The Problem?

Cleaning a string is slow, to say the least. A slow load time is the bane of your site’s existence. Customers and visitors hate waiting. The tools mentioned above, as well as any equivalent unmentioned tools, accomplish the task via brute force stripping of characters and numerous regular expressions. Here are a few examples of some common XSS filters out there:

• Drupal: filter_xss
• Kohana 3: Security::xss
• node-validator: xss.js
• Rails 3: SanitizeHelper

To optimize these libraries, we need to step back a minute and think about what it is we’re trying to accomplish. In this case, how can we go about skipping all of the string manipulations and pattern matching with no loss in security?

So What’s The Solution?

There is no catch-all solution, especially if you’re trying to sanitize HTML coming in from a WYSIWYG editor. There is, however, a very worth-while tweak you can use to make your site’s sanitization much, much faster under most circumstances. The simple solution is to start your sanitization with a conditional statement which checks to see if the string is already clean of tom-foolery, and return it if so. Grunt work avoided. The reason such a simple trick works is because 99.9% of user input is going to be alpha-numeric with a few sprinkles of punctuation. Don’t quote me on that.

The Wrap-Up

It’s debatable as to what additional characters you can add to this without overly complicating the solution and potentially introducing security holes. I’ll leave it up to you as the reader to suggest alternative whitelist regexes or punch holes in the solution (like hey guy, how do you know if the character set is in utf-8?).

Back in the spring, we had the fabulous Aaron Draplin re-work our logo which is just ripe for interpretation. We’ve heard it called everything from the “mother ship” to the “alien egg” to (my favorite) the “space volcano.” (Also, I’m the one who says “space volcano.”)

Recently, our SDW design whizzes have been leading ad-hoc Photoshop classes for the guys who do nothing but stare at the Terminal all day. One of those forays quickly resulted in a round-robin PSD free-for-all with the results included below.

I know the one, Mr. Draplin would be horrified to see his meticulously crafted logo butchered in all manners presented; I hope to do a full writeup on the identity process at some point to show off some of the other great things we’ve yet unleashed.

Skookum Digital Works Logo Photoshop Roundup (Gizmodo Style).

Honorable Mentions

From the Land of Obvious…nice try, Jason (probably the first time he’s used PSD, so good effort there).

Body Art from Hunter and James, respectively.

The I-see-what-you-did-there from Dustan.

An homage to the Carolinas, also from Jason (learning quickly).

If you pay your bills to SDW, you pay them to Don. And Don likes golf.

Banksy from Rich.

Winners

5. Tron from Jim.

4. “Starbury” from Chris (this got some laughs).

3. Mischief. Mayhem. Node. From Rich again.

2. Skookums Helped Build the Pyramids, from Jim again (two top fives, nice).

1. 2001, from your’s truly.

Rigged

This was my blog post; you didn’t thing I was going to lose, right? (Also, I’m the only cinefile here, so I accept this first place award in protest. Y’all need to get right with Kubrick.)

Would you like to play?

Yeah, wow.

It’s a little surprising this revelation has not received further attention from [allow me, cough] progressive web and mobile application developers given the basis of their existing skill-set can soon be applied to the desktop development. It could be, perhaps, that a lot of forward-leaning digital manufacturers don’t USE Microsoft products and therefore are ambivalent to advancements in its production environment. Though for the business-minded application development shop, it’s hard to overstate this new opportunity.

And for lifelong .NET, VisualStudio, and Silverlight software developers who long ago exclusively hitched their wagon to whatever-came-out-of-Redmond, their reaction to news relating to the downgrade of their respective skill-sets falls somewhere between the bothersome and ulcer-inducing. Understandably.

Since Microsoft Vice President Julie Larson-Green first spoke of this transition, the company has mostly been mum on the issue. Their silence has largely not been comforting to their installed developer base. Not all the Windows 8 details have been worked out, and since their marketing and product development teams are still honing their message prior to an expected Windows 8 Spring 2012 release date, Microsoft may continue to be working on features (i.e. some throwback SDK tooling) that will calm the fears of their existing devs.

For Microsoft, switching over to technologies aimed at “invigorating their Windows-developer ecosystem” is an extremely intelligent business move. The sheer number of worldwide HTML5 and JavaScript professionals vastly outnumbers Microsoft “Sofities” (apparently, what Microsoft Windows devs call themselves, who knew?).

It’s rumored all the “cool” Windows 8 UI will be powered by this new-to-them dev stack—one of the main features being visual scalability of the apps to work across Windows 8 powered touchscreen tablets. (In other words, responsive Microsoft software.) Old-school Windows applications will still run and will still be produced using hammers and chisels², but only in that weird-OS 9-to-OS X-classic-environment way. That’s a working theory, anyway.

At Skookum Digital Works, we’ve now got a little skin in the Microsoft game. We’ve already been using HTML5 options to power cross platform mobile (iOS, Android, Blackberry, tablet, etc.) experiences that also look great on a larger browsers from desktops or laptops. In addition, we’re experts at bridging the browser-software-to-desktop-software chasm; to now think we’re now in the business of all web, all mobile, AND all (new) Windows platform apps is quite exciting.

Though anyone’s who’s spent hours or days rewriting IE rules can say…how exactly Microsoft decides to adopt HTML5 and JavaScript will be in the details.

¹ hyberbole
² there too

…but until then, it’s our job to go out there and whack them around.

Two makes it a ritual

Though Skookum Digital Works was founded in 2005, the (official) first annual golf trip didn’t happen until 2010. Funny enough, it came at my expense.

See, just a little more than a year ago, I was a client. I was the digital guy at an agency in New England, and the crew at SDW had built out a bunch of executions for me over a couple years.

And I totally screwed them.

The details won’t be interested to anyone outside of advertising, but on a Friday during a known, several-month timeline, I called in a nuke. I begged to have three of SDW’s finest crank over the weekend. They made it so I could make my client happy—I sent them a fat check.

Though we had a great relationship, Founder & VP Bryan needed some convincing.“Dude, If I do this for you, I’m actually going to shut down for a day next week to apologize to my team for ruining their weekend. Like, I’m literally going to take part of your rush money and bring everyone golfing.”

Over the phone, I laughed because I knew he was serious. Also, I was fine with this arrangement.

Second Time Around

SDW makes non-office company time a priority, so we plan group outings on the reg. (*See also: paintball.)

However, sometimes getting away as a group is very hard. We are teaming with startup businesses and custom software development, so our shop is slammed. Alas, a few us had to stay behind and tend projects at headquarters (better luck next time with the short straws Corey, Hunter, Chris S., Jon, Kenny, Rebecca).

And as you can see from the video, the 2nd Annual Skookum Digital Works Golf Outing was long on smiles and short on technique. We hope to share even more of these fun moments as our studio continues to grow. (And grow, and grow, and G R O W.)

Join the team, will ‘ya? Besides, how many other places can you play golf a week before Thanksgiving? (Charlotte FTW.)