I’ve been using jQuery a lot recently and it occurred to me that I could use jQuery’s selectors to target the information that I’m interested in a web page, and then using Ajax POST it to my own script that would be ready waiting to then do something useful with the data, e.g. validate it and save it in a database.  For educational purposes I was keen to keep this completely client-side if possible (except for a script to receive the information).  See later on for a server-side solution.

The situation I was up against was a page that had a heap of data in a table (about 90 items), but the table was interspersed with random images to split it up and make it more pleasing to the eye.  Fortunately for me, all of the data that I wanted was neatly wrapped in <div class=”information”></div> tags.  Selecting these div tags with jQuery is really easy by using $(’div.information’).

My first problem was that in order to use jQuery, the web page you’re looking at has to be using it.  Fortunately there’s a quick bookmarklet called jQuerify that allows you to load jQuery onto any web page.  Once you’ve got that then you can write further bookmarklets of your own to do stuff.

So, my evil evil plan was to combine a jQuery selector, jQuery’s each() construct, and jQuery’s ajax support to post the content of each div to a “scraper” script, like so:

$('div.information').each(function(){
  $.post('http://localhost/scraper.php',{
    data: this.innerHTML
  });
});

I loaded my source page, clicked the jQuerify bookmarklet and then pasted the code above into the Firebug console (what, oh you’ll need that…) and it was flawless … except that the browser security model stepped in and prevented the ajax call because the XHTTPRequest object is not allowed to post information from one domain to another.  I was stuck - I googled around for a while looking for workarounds, and investigated the use of JSONP but the transport method seemed more weighted at retrieving information rather than posting it.

So, I was stuck with a simple question: “How can I get information from one site to another by using the browser?” - the simplest answer to this question is of course to have a form on the source website, that when submitted posts to the target.  Thanks to the power of JavaScript, modifying the DOM of a loaded web page is a doddle.  Therefore it should be simple to create a form on the page after it has loaded (client side, remember), create and populate some form fields with data and then submit the form to my scraper script.

Suddenly my intentions had outgrown a bookmarklet, but I would still need one for jQuerify and one for my “Scraper Utils”.  My new bookmarket simply asked jQuery to load a local JavaScript file in exactly the same was that jQuery was loaded in the first place:

javascript:$.getScript('http://localhost/scraper.js');

Now I had the freedom of writing chunk loads of stuff in my local scraper.js file.

Scraper = {};
Scraper.createForm = function()
{
  var form = document.createElement('form');
  form.setAttribute('method', 'POST');
  form.setAttribute('action', 'http://localhost/scraper.php');
  document.getElementsByTagName('body')[0].appendChild(form);
  return form;
}
 
Scraper.createSubmitButton = function()
{
  var button = document.createElement('input');
  button.setAttribute('type', 'submit');
  return button;
}
 
Scraper.createFormField = function(name)
{
  var field = document.createElement('textarea');
  field.setAttribute('name', name);
  field.setAttribute('rows', 10);
  field.setAttribute('cols', 50);
  return field;
}		
 
var ScraperForm = Scraper.createForm();
$('div.information').each(function(){
  var field = ScraperForm.appendChild(Scraper.createFormField('data[]'));
  field.value = this.innerHTML;
});
// Create a field that we can post with:
ScraperForm.appendChild(Scraper.createSubmitButton());

You can see here that I’ve set up a few functions, createForm(), createFormField(), createSubmitButton() and then at the bottom I wrap them all together with the $(’div.information’).each(…) construct.  The end result of this is that when I click my bookmarklet that includes the scraper.js script, a form is created at the bottom of the page and a textarea for each div.information is created that holds the innerHTML from that div.

Then, by clicking the Submit button, the browser posts all of that information across to http://localhost/scraper.php where I then collect the information from $_POST['data'] and poke it into a database.

It’s pretty rough and ready but could easily be extended to do other things like allow you to specify the selector and target URL for the post when you click the Scraper bookmarket.

Server Side Solution

On my travels I also came across the “PHP Simple HTML DOM Parser” which claims a similar ability like so:

// Create DOM from URL or file
$html = file_get_html('http://www.google.com/');
 
// Find all images
foreach($html->find('img') as $element)
       echo $element->src . '<br/>';
 
// Find all links
foreach($html->find('a') as $element)
       echo $element->href . '<br/>';

You can get a hold of this from Sourceforge at the PHP Simple HTML DOM Parser website.

Anyway, at work we use the Lightning calendar plugin for Thunderbird, with the Google Calendar provider in order to collaborate on a calendar.  For the most part this works well as when not in the office you can fall back to Google Calendar.

I went down the path of installing Lightning into Thunderbird (download the XPI, browse to it etc…) but after the installation Lightning seemed broken.  The UI was mostly there but it looked buggy and nothing worked.  After hunting around for a reason, I came across this thread that suggested that the problem was that the libstdc++5 package had to be installed.

I was skeptical, but after reading half a dozen “me too” posts where the problem had been fixed I got stuck in.

1. Uninstall the Lightning plugin from Thunderbird
2. Open a terminal, and run this command: sudo aptitude install libstdc++5
3. Reinstall Lightning from the XPI you downloaded

Then things came to life nicely.  I was disappointed that the state of Lightning without libstdc++5 appeared to be a buggy application rather than a specific error.

Considering my broadband/phone bill currently sits at about $120 per month this will be a nice saving, plus an excuse to play with some technology :)  I’ll be using Snap for the naked DSL, only because Orcon (my current provider) haven’t unbundled in Dunedin yet and it looks like it’ll be quite a while.

If you don’t mind a bit of downtime on your phone line or broadband service, the steps are:

1. Port your phone number to 2talk.
2. Get naked DSL provisioned on your home line.
3. Configure your system so that your phone rings at home by connecting to 2talk’s servers

You’ll need some hardware, and some charges (termination charges with your current broadband provider for example) may apply.

What is fail2ban?

It’s a python script (that runs as a daemon) which monitors log files in your /var/log file.  It monitors them for specific entries, for example “Failed password”, and then updates iptables rules to deny network access for the offending IP for a configured amount of time.

A good example of this is that if you try to ssh into my system three times unsuccessfully, you won’t be able to try again for 10 minutes.  This is sufficient to make automated brute force attacks useless.

Do you need it?

If you have a public-facing server with the ability to log into it (including web applications even) then you need this.  If you’re curious to see if you’ve been targeted for attacks, try running these commands as root on your server:

cat /var/log/auth.log | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort | uniq -c
zcat /var/log/auth.log* | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort | uniq -c

The first command examines your current auth.log file and the second examines your historical auth.log.[0-9] files. In my recent history (prior to configuring fail2ban properly) I had over 6,000 failed SSH login attempts on a single day just after Christmas.

What next?
The steps are:

bob@server:~$ sudo apt-get install iptables fail2ban
bob@server:~$ sudo /etc/init.d/fail2ban start

Now, you can check to see if it’s working by “pinging” the service:

bob@server:~$ sudo fail2ban-client ping
Server replied: pong

And you can get information on what’s currently been banned by examining the ssh “jail” - the jail is term used to describe the configuration and current black list for access from remote hosts:

bob@server:~$ sudo fail2ban-client status
Status
|- Number of jail:	1
`- Jail list:		ssh
bob@server:~$ sudo fail2ban-client status ssh
Status for the jail: ssh
|- filter
|  |- File list:	/var/log/auth.log
|  |- Currently failed:	0
|  `- Total failed:	52
`- action
   |- Currently banned:	0
   |  `- IP list:
   `- Total banned:	7

To test everything is working, simply try to log into your system incorrectly three times.  When you’ve done this and you look at the results of “fail2ban-client status ssh” you will see your remote IP in the list.  To unblock your IP, simply restart the fail2ban daemon (i.e. sudo /etc/init.d/fail2ban restart)

These pages were very useful when reading about fail2ban:

• The Art of Web ~ System: fail2ban and iptables
• The manual for fail2ban 0.8 (very readable)

#1: A Stein-smashingly good time!

#2: I have days like these …

#3: Just too true

#4: Fandom

#5: White-belly-making-cereal

Rt. Hon Steven Joyce
Minister for Communications and Information Technology
Care of Distribution Services
Parliament Buildings
Wellington 6160

Dear Steven,

I am writing to you in urgency regarding the proposed Section 92 of the Copyright Amendment Act that assumes guilt upon accusation, forcing the termination of internet connections and websites without evidence, a fair trial, or punishment for false acusations of copyright infringement.

I work for a web development company and cannot let this “roll over” into law without voicing my protest. To know that any one of our clients, or indeed ourselves could be falsely accused without proof required and have our internet access terminated would have dire ramifications as our livelihoods and futures rely on providing these services.

I urge you to carefully consider the ramifications of the proposed Section 92 and speak out on behalf of myself and others who have contacted you regarding this.

I am available for further comment if required.

Yours Sincerely

Bob Brown
Web Developer
(my address)
Dunedin

I’ve also just added the introduction to classes tutorial so go and check it out.

I started digging and after ruling out intentional upload/download from inside my network (using wireshark on my LAN segment) I then started looking at the wireless side of things. I have a server on my LAN that is connected via wireless, and this is also a public facing server with sshd running on it.

I had a quick page through the /var/log/auth.log file and to my surprise I found repeated attempts to log in with various usernames (491 different ones so far) from various locations. These were coming in at the rate of one every four seconds. I can’t see how this would account for 15GB (or 1.5GB for that matter) but when I called Orcon they said they saw a definite increase in traffic over the last two days, which also corresponds to the first entry in the auth.log file.

Here’s what the log entries look like:
Sep 27 13:29:59 pandora sshd[17366]: Failed password for invalid user oracle from 60.190.133.228 port 45662 ssh2
Sep 27 13:30:03 pandora sshd[17368]: Failed password for invalid user michael from 60.190.133.228 port 45857 ssh2
Sep 27 13:30:07 pandora sshd[17374]: Failed password for invalid user ftp from 60.190.133.228 port 46079 ssh2
Sep 27 13:30:12 pandora sshd[17376]: Failed password for invalid user test from 60.190.133.228 port 46301 ssh2
Sep 27 13:30:15 pandora sshd[17379]: Failed password for invalid user webmaster from 60.190.133.228 port 46553 ssh2
and so on …

The requests have come from a range of IP addresses in China, Europe, Canada and Bangladesh. It’s likely to be a distributed and targeted attack.

Interesting. I did install fail2ban some time ago which I was told was supposed to prevent this kind of thing but it was an install-and-forget excercise. I’ll have to research it a bit to find out how it actually works.

Secondly, an explanation of what it is I’m doing here. A while back I saw some screencasts re Inkscape at screencasters.heathenx.org. I was impressed - it was a great way to teach people how to use Inkscape.

I had a play around and managed to get to the point where I can now do the following:

• Record my screen, or any particular window
• Use my Logitech USB headset to record the audio
• Play the resulting movie back, sound through speakers or headset

For some reason I seem to be having a ridiculously hard time determining where the audio output goes for my Linux apps. In Windows, when I plug my headset in, it becomes the default audio device for all applications. Currently running applications still use their normal sound output, until restarted. If I dig deep enough I might even be able to find settings in each program that allows me to specify which sound device (headset or sound card) should get the output.

In Linux (Ubuntu 8.04, Gnome to be precise) I discovered under System > Preferences > Sound an application that allows me to adjust the sound preferences. In the dropdown list for each playback type I have seven possible playback device options (Auto, USB Audio, Intel ICH5, Intel ICH5 - IEC958, ALSA, OSS & PulseAudio). It seems that if I set them all to USB Audio then I get lucky and most things play through the headphones. Some things, especially the Adobe Flash Player plugin insists on playing through my desktop speakers regardless. I saw someone complaining that this was Adobe’s fault, not Linux but I can’t comment. In order to avoid all this confusion I found out how to specify the sound device that you want to use for each operation. This is pretty much the crux of this post.

Anyway, here’s what I did …

Recorded My Desktop

There’s a package in the Ubuntu universe repository called recordmydesktop which as the name suggests can be used to record your desktop, with sound. This is great for making recordings of how to do things to share with others. I installed the recordmydesktop package, along with the GTK front end to save me having to look up the commands all the time..

bob@sparkie:~$ sudo apt-get install recordmydesktop gtk-recordmydesktop

Once installed I then hit Alt-F2 (to bring up the Run Application dialog) and typed in gtk-recordmydesktop.

It seemed that by default it was trying to record sound via the mic input on my sound card. Fat chance it would have had because I don’t have a mic in my soundcard, instead I have a USB headset.

After Googling around I discovered that if I click the Advanced button in gtk-recordmydesktop I could then enter the sound device to use to capture audio. I used the device name “plughw:Headset,0,0″ and was able to record sound via my USB headset.

The key to this was finding out what to type in for the device name. I discovered the aplay and arecord utilities that allow you to play and record sound. Each utility has a -l switch which lists the available devices. Here’s the output of my arecord -l:

bob@sparkie:~$ arecord -l
**** List of CAPTURE Hardware Devices ****
card 0: ICH5 [Intel ICH5], device 0: Intel ICH [Intel ICH5]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 0: ICH5 [Intel ICH5], device 1: Intel ICH - MIC ADC [Intel ICH5 - MIC ADC]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 0: ICH5 [Intel ICH5], device 2: Intel ICH - MIC2 ADC [Intel ICH5 - MIC2 ADC]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 0: ICH5 [Intel ICH5], device 3: Intel ICH - ADC2 [Intel ICH5 - ADC2]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: Headset [Logitech USB Headset], device 0: USB Audio [USB Audio]
Subdevices: 1/1
Subdevice #0: subdevice #0

You can see beside each “card” there is a device name, either ICH5 (my sound card) or Headset. Each of those cards has a number of subdevices - in the case of the sound card you can see there are four possible subdevices I could have chosen (numbered 0 to 3) and for the headset it’s a free ride because there’s only one to choose from (as I would have expected).

So, now that plughw:Headset,0,0 should make a bit more sense. With that all set up, I hit record, wiggled my mouse around, spoke some gibberish and created a .ogg file of my session.

Sound Recording and Playback

While I was debugging my sound devices I fiddled around a bit with arecord and aplay. Using the command arecord -Dplughw:Headset,0,0 -f S16_LE -c2 -r44100 > myfile.wav I created a 44kHz wav file using the headset mic as my recording device.

When I played it back with aplay myfile.wav, it played through my desktop speakers. To play it through my headset I just specified the device name as I did with the arecord command, thus aplay -Dplughw:Headset,0 myfile.wav

If you want to use mplayer to play something back using a specific device for audio output, this command will do the trick for you: mplayer -ao alsa:device=hw=1.0 ep069.flv. Device 1.0 is card #1, device 0 - and that’s the headset.

If you need to adjust the volume, I found you can use the alsamixer. To control each card you can specify which one with the -c switch. In my case card 0 is the default and internal sound card, and card 1 is the USB Headset: alsamixer -c 1

That’s all for now!