Random thoughts about nothing (usually) so special

Procedure and ruby script to remove duplicated tracks from iPod in Last.fm client before scrobbling

2009-07-07T00:40:00.002-03:00

Since I bought an iPod Touch, it became my main Last.fm scrobbler device. However, I am facing a recurrent problem, every track that I listen in my iPod is scrobbled between 2 and 20 times with the same timestamp! The picture bellow shows the result:

I am using the Last.fm official client with the iTunes plug-in. I have searched a solution in several sites (e.g, here), lots of people are facing the same problem, but I did not find any solution yet.

To make the problem even worse, lately the remove button in the Last.fm tracks page don’t remove a single occurrence of a track anymore, instead it removes all occurrences in a given timestamp. So, once a track is scrobbled multiple times, I can not remove the duplicates anymore, because once I remove one, it removes all the entries.

So, while I don’t find a good solution, I created a manual procedure that I am using to solve the problem:

Turn off the Internet connection;
Ask the last.fm client to scrobble the tracks;
Since it cannot scrobble the tracks because the lack of connection, the tracks (with the multiple occurrences) are kept in the local history (C:\users\windows_user\Local Settings\Application Data\Last.fm\Client\lastfm_user_submissions.xml);
Close the Last.fm client;
Run the ruby script bellow that I have written that excludes the duplicates tracks from the local history xml file;

ruby scrobbler_fix.rb "C:\users\windows_user\Local Settings\Application Data\Last.fm\Client\lastfm_user_submissions.xml"

Connect to the internet;
Open the last.fm client and scrobble the history;
Check the tracks page to see if everything was ok.

As usual, I kept the source code in my utility scripts repository at GitHub, but the code is listed bellow as well. I hope I find a better solution soon, because this problem even with this procedure sucks!

# Script to remove duplicate items from last.fm submission xml file
# author: Douglas Fernando da Silva 

require "rexml/document"
include REXML

history = nil
File.open(ARGV[0]) do |file|
 history = Document.new(file)
 tracks_to_remove = []

 last_timestamp = nil
 history.elements.each("submissions/item") do |element|
   ts = element.elements["timestamp"].text
   if ts == last_timestamp
     puts element.elements["track"].text
     tracks_to_remove.push(element);
   else
     last_timestamp = ts
   end
 end

 for i in 0..tracks_to_remove.length-1 do
   element_to_rm = tracks_to_remove[i]
   element = history.elements["submissions/item"].parent.delete(element_to_rm)
 end

end

formatter = REXML::Formatters::Default.new
File.open(ARGV[0], "w") do |result|
 formatter.write(history, result)
end

Douglas

Ruby script for setting lyrics in iTunes Library tracks

2009-06-22T02:47:00.001-03:00

One of the cool features of the iPod Touch and iPhone is that is possible to see the lyrics of a track while it’s playing. The picture bellow shows that.

But the problem is that you have to insert the lyrics manually using the iTunes, for each track you have to use the “Get Info” option and then set the track, just like the pictures bellow.

That’s a problem, because if you have hundreds of tracks, it will take lots of clicks and time. In order to solve that problem, I created a Ruby script that tries to set lyrics for all the tracks in my iTunes Library.

The script interacts with the iTunes COM components to get all the tracks in my Library, after that, it uses the LyricWiki web service to get the lyrics for each track and set to it.

The script works only on Windows and it’s available in my github repository:

git://github.com/dougfernando/itunes-helper.git

http://github.com/dougfernando/itunes-helper

Now, every time I get a new album, all I have to do is to run the script again, like the following images :

The script code is also bellow:

require 'win32ole'
require 'soap/wsdlDriver' 
require 'iconv'

class ITunesHelper
  def initialize(lyrics_service)
    @itunes = WIN32OLE.new('iTunes.Application')
    @tracks = @itunes.LibraryPlaylist.Tracks
    @lyrics_service = lyrics_service
  end

  def show_all_tracks
    @tracks.each do |track|
      print track.Name() + "\n"
    end
  end

  def set_all_lyrics
    puts "Number of tracks to be analyzed: #{@tracks.count}"
    index = 1
    @tracks.each do |track|
      if self.is_lyrics_empty(track.lyrics)
        lyrics = @lyrics_service.get_lyrics(track.Artist(), track.Name())
        
        if self.is_lyrics_empty(lyrics)
          self.print_track(index, @tracks.count, track, "lyrics WAS NOT found")
        else
          track.lyrics = @lyrics_service.get_lyrics(track.Artist(), track.Name())
          self.print_track(index, @tracks.count, track, "lyrics WAS found and set")
        end
      else
        self.print_track(index, @tracks.count, track, "already had lyrics")
      end
      index = index + 1
    end
  end

  def is_lyrics_empty(lyrics)
    lyrics.empty? or lyrics == "Not found"
  end

  def print_track(index, total, track, result) 
    puts "Track #{index}/#{total}: #{track.Artist()} - #{track.Name()}: #{result}"
  end
end

class LyricsServiceProxy
  def initialize
    @driver = SOAP::WSDLDriverFactory.new("http://lyricwiki.org/server.php?wsdl").create_rpc_driver
    puts "Lyrics web service initialized"
  end

  def get_lyrics(artist, track_name)
    Iconv.iconv("LATIN1", "UTF-8", @driver.getSong(artist, track_name).lyrics).to_s
  end
end

itunesH = ITunesHelper.new(LyricsServiceProxy.new)
itunesH.set_all_lyrics

Douglas

Powershell script for converting Word documents to PDF format

2009-06-12T16:25:00.005-03:00

In my job, we only send documents to clients in pdf formats. Sometimes we have to send several documents at once and it takes a good time to open each one and save it as a pdf, it’s not a very pleasant task.

In order to minimize this problem, I’ve created a simple powershell script that opens a Word document and saves it as pdf without any user interaction. With such script, I can convert a lot of files at once, just like this:

ls . *.doc* –Recurse | %{ ~\scripts\doc2ps1.ps1 $_.fullname }

With this command, I list recursively each doc/docx file in a directory and save each one as PDF.

By default, Office 2007 apps are not able to save files as PDF, so you have to install the add-in from Microsoft first to use the script. The add-in is available here.

Bellow is the script, it’s also available in my PowerShell scripts repository at GitHub (http://github.com/dougfernando/utility-scripts/tree/master) as doc2pdf.ps1.

param (
 [string]$source = $(Throw "You have to specify a source path."))

$extensionSize = 3
if ($source.EndsWith("docx")) {
$extensionSize = 4
}

$destiny = $source.Substring(0, $source.Length - $extensionSize) + "pdf"
$saveaspath = [ref] $destiny
$formatPDF = [ref] 17

$word = new-object -ComObject "word.application"
$doc = $word.documents.open($source)
$doc.SaveAs($saveaspath, $formatPDF)
$doc.Close()

echo "Converted file: $source"

ps winword | kill

An important note, the script closes every Word instance in the end, so save any work previous using it.

ASPY Player Beta 4 released - S60 Player with Last.fm scrobbling built-in. Open signed online now possible!

2008-04-09T02:33:00.001-03:00

I've just released the beta 4 version of the ASPY Player. Currently I have only few hours (~3h) a week to work on it, so that's the reason for few enhancements in this release, however I think some of them are very important.

The changes in this release are:

Possible to sign the sis file through open signed online;
Blinking "now playing" screen bug fixed;
Better handled id3v2 only files (id3v2 not supported yet);
Track number info being used in the app;
Better tracks ordering handling;
Better handling of non-US ASCII characters in directories names;
Application code design improved;

As usual, all kinds of feedbacks are welcome. They have been essential to improve the player quality, thank you guys!

ASPY Player Beta 3 released - S60 Player with Last.fm scrobbling built-in

2008-03-21T21:08:00.000-03:00

After coming back to work from vacations, it has been hard to find time to work on the player. Currently I'm leading 5 projects in my work! However I got some time (most of it while traveling) and released the beta 3 of my player. Bellow is a list of the major changes:

Non-US ASCII bug fixed, now files path and id3 content can have é, è, ã, etc..
Now it's possible to view the current music history, ie, tracks not sent to last.fm yet
After selecting an artist, now it's possible to select all tracks of that artist as before or drill-down by artist album, just like the native player
AudioScrobbler protocol hard error handled correctly
Application icon added
Correct version number set (major/minor)
Now it's possible to play tracks in random mode
Several bugs fixed

As usual, the source are here and here there's a step-by-step tutorial on how to run it without signing using the python shell (thanks Brixton).

The python snake image I found using Google Images so I don't know who created it. Any information about it, please, tell me so that I can ask the creator the usage authorization.

I also have to find a more beautiful image for the "Now playing screen", preferentially a dark one to help with the energy consumption. I have also to make the now playing screen stop blinking too. I'll try to stabilize the current feature set (fix bugs and refine the design) then start adding more features.

Ahh, BTW, because of my efforts to build this player, the guys from Build Last.fm gave me 12 months of subscription!! Great incentive! Thank you guys again!

I'd like to thank the guys who are using the player and sending me feedbacks.

Douglas

Some new pics from this version:

Second release of my Last.fm S60 Client - aspyplayer

2008-03-01T22:50:00.004-03:00

Today I released the second beta version of my last.fm client for S60 devices (although only tested in my N95-1). I rewrote the whole UI, now it's a little less ugly and it kinda looks like the native player. Now it's also possible to play/stop/forward/back/volume up/volume down using the keyboard in the "Now playing" screen like the native player.

The major bug in the moment is that it cannot handle well files with path or title that contain special characters, such as 'é', 'è', 'ç', 'ã'. I'm Brazilian, so it's a problem for me because in portuguese, words with that kind of characters are so common. However, for me it's strange how the pyS60 handles unicode, I didn't get it yet. It pissed me off and I gave up for the moment, so the bug is there. I have to correct some spelling errors too and now, I don't know why, the sis only works on C:

For those who can't sign the sis files and want to try the player, here can be found a good tutorial explaining how to sign sis files. Besides that way, it's possible to run the application using the sources. To do so, get the aspylayer.py and now_playing_bg.jpg files from the repository and copy them to c:\python or e:\python. After that, run it using the Python Shell like in the image bellow.

Thank you guys for the feedbacks (I'm waiting for more :)). I think soon a more stable/usable version will be released...

Some screeenshots for this release:

My Last.fm S60 Client - aspyplayer

2008-02-27T17:53:00.004-03:00

In the last few days I've been working on a S60 music player with Last.fm scrobbling capabilities because most of time I listen to musics in my N95. I'd the idea to build this application months ago, but only now, in my last week of vacation I had time and today I released the first beta version (ugly UI and barely working) at: http://code.google.com/p/aspyplayer/

Some screenshots for this first beta release are available at: http://code.google.com/p/aspyplayer/wiki/screenshots

I'm writing the application in python (pyS60), which for me, currently is the best development platform for Symbian devices. However, it has a lot of limitations compared to C++ on S60 devices.

The application architecture kinda follows DDD, with a basic MVP for the presentation layer. The last time I did some development in python was about 5 years ago, so the code is not that good yet. The sources can be seen at: http://code.google.com/p/aspyplayer/source/browse. Code reviews are welcome.

Now I'll work on the UI, which currently is very ugly and I have to refactor some code to decouple the file system libraries from the model putting it in the infrastructure layer. Besides that, I have to create more unit tests.

Any kind of feedback is welcome.

BTW, yesterday I found an amazing last.fm client for iPhone/iPod Touch, it's called MobileScrobbler. It's too bad that the S60 interfaces sucks when compared to iPhone interface which is fantastic.

My current "must read" technical readings

2008-02-20T21:44:00.004-03:00

Although I read more than one hundred of blogs (~4500 posts in the last 30 days) using the kickass Google Reader and be a member of several mailing lists, I don't read everything with the same priority. I read ASAP only few blogs/lists that I consider important/cool, others sometimes I read months after I noticed them in my google reader or gmail.

The following items are my current "must read" technical readings:

Top 10 Feeds (no specific order):

Ayende @ Rahien => http://ayende.com/Blog/Default.aspx
Did it with .NET => http://diditwith.net/
Martin Fowler's Bliki => http://martinfowler.com/bliki/
Jeremy D. Miller => http://codebetter.com/blogs/jeremy.miller/default.aspx
Planet ThoughtWorks => http://blogs.thoughtworks.com/
Coding Horror => http://www.codinghorror.com/blog/
Object Mentor => http://blog.objectmentor.com/
InfoQ => http://www.infoq.com/
{Codesqueeze} => http://www.codesqueeze.com/
LeadingAnswers => http://leadinganswers.typepad.com/leading_answers/

Brazilian bloggers

Fragmental => http://blog.fragmental.com.br/
Daniel Sato => http://www.dtsato.com/blog/default/
Guilherme Chapiewski => http://gc.blog.br/
A Maldita Comédia => http://malditacomedia.blogspot.com/

Mailing lists

Castle Project Users => http://groups.google.com/group/castle-project-users
Alt.Net => http://tech.groups.yahoo.com/group/altdotnet/
Domain Driven Design => http://tech.groups.yahoo.com/group/domaindrivendesign/

BTW, the address of my google reader shared items is: http://www.google.com/reader/shared/11574825750058829373.

In paper I usually read non-technical (computer) stuff, currently I'm reading the applied psychology books from "The Personal MBA", the current one it's my third and till now all were great readings.

Douglas

I'm in love with my Nokia N95

2007-10-12T19:46:00.000-03:00

I fought with myself for weeks, but about 2 weeks ago I could not resist anymore and finally bought a Nokia N95. I have been using it since then and what I can tell is that the device is amazing!

For those who don’t now the device yet, it’s a GSM smartphone with a 5MP camera, Wi-Fi and embedded GPS running on S60v3 (Symbian OS). A great generic review is available in the link bellow:

Nokia N95 review: Nokia’s crown jewel
http://www.gsmarena.com/nokia_n95-review-141.php

My digital camera is broken; I needed an mp3 player because I have been spending about 10h a week in a bus because of my current project and I like to listen to music while traveling, so I decided to buy this phone, because it can be used for these functions pretty well.

Currently I’ve been running some very cool softwares on it. For example:

Putty for ssh;
Python for some cool scripts (graphics, text editor, etc.) that I found;
Quake 1;
Google Maps integrated with the GPS, which, btw, absolutely rocks;
DivX Player to watch TV series episodes while traveling;
Fring to make Skype calls over Wi-Fi, which it’s also very very cool;
Gmail client for reading e-mails which is very close to the standard gmail, but gmail mobile is ok too;
Nokia podcasting software for listening to podcasts;
Nokia mp3 player is very nice too; most of my favorite albums are in the phone already, etc.

And now I can take nice pictures (5MP) whenever I want!

Using it for web browsing over Wi-Fi is very cool also. Currently more than half of my feeds I read on it using Mobile Google Reader, which is also very cool. The standard browser of N95 is very good in general; it uses the same engine of Safari and it’s a real browser (not a “wap” browser)! I can access most of my “standard” sites using it and everything works pretty well. I have installed Opera mini, but I think it’s too heavy. So, for reading and listening to music I've been using my N95 more than my laptop.

There are some things that I want to do in the future with it, for example, install a ruby interpreter, install a software for use it as a webcam that I cannot remember the name, buy more memory; install a software to download maps of Sorocaba and Volta Redonda to use with GPS without having to use any Internet connection (it already has a pretty good map for São Paulo), download more games like Quake, write some apps in python for it, send photos to flickr directly using it, etc.

I’m also impressed about how easy is to write graphical applications with python for S60. I’m thinking about writing some apps in python to explore the GPS and also, if possible, a Nokia mp3 player integration with last.fm site.

In my opinion, for advanced users, there’s no comparison with iPhone, which is VERY cool because of its UI, but it’s also much closed for custom software. I could not do most of the things that I’m doing with my N95 if I had an iPhone!

Of course, it has some problems, for example, the battery sucks, the camera flash sucks, internal RAM is not enough for running more than 3 applications in parallel, it’s very expensive (~2,3K reais in Brazil), it’s kind of big, etc. But, for me, up to now, it rocks!

Some images of my N95 running:

Some of my applications

A skype call

Route: Parents' home in Sorocaba -> my apartment in São Paulo

Fibonnaci in python

The mp3 player

Some of my photos

Using google reader

Tools that I Love

2007-06-25T00:30:00.000-03:00

Common Tools

Launchy - I rarely use the "Start Menu" with it. It's the closest Windows tool to Mac Quicksilver. I used to use SlickRun, which is also a great tool, but I moved to Launchy because of the better user experience;
Notepad2 - The best "simple" text editor around, it looks like the traditional Notepad, but much more powerful;
Process Explorer - Much better than the traditional Windows Task Manager. Using it you can know what is really going on in your machine.
Unlocker - Do you want to delete some file and Windows does not allow you because some other process is using it? Unlocker solves this kind of problems perfectly;
TrueCrypt - Strong cryptography made easy, specially for usb sticks;
Paint.NET - Similar to Notepad2; simple as the traditional Paint, however much much more powerful;
Foxit - Is it so hard to read a pdf file? I can't even remember the Acrobat Reader nowadays with Foxit;
Free Download Manager - Kick ass download manager without ads and stuff like that, although the name is not the most "creative" I ever seen;
Firefox - I cannot figure out why people still use Internet Explorer. Firefox is great, but its add-ons are even better: del.icio.us, greasemonkey, flashgot, firebug, fasterfox, etc. are fantastic...

.NET Developer Tools (I haven't developed in Java for the last 2 years, so my Java tools will be in another post)

Resharper - It's impossible to use Visual Studio without Resharper if you ever seen Java IDEs like IntelliJ and Eclipse. It makes you at least 25% more productive. I rarely use mouse in VS because of it...
Reflector - Visual Studio Class Browser is a shame when compared with Reflector. Reflector is much more user friendly, more powerful, great disassembler and have nice plug-ins;
Subversion - Simply the best version control system around if your team is not 200+, TortoiseSVN is a great client as well;
Cruise Control .NET - I cannot imagine myself working in a team without a CI server anymore and Cruise Control .NET does a fantastic job. However, it could get a better default report page, like the new Cruise Control Enterprise. CCTray is great as well;
Test Driven .NET - The best way to launch your unit tests inside Visual Studio. I love the Resharper, but its unit testing tools are not so clean/light/not intrusive as Test Driven .NET. Run Last Test feature is fantastic;
NCover - Very simple but efficient way to check your tests coverage;
FxCop - Great inspection tool, Microsoft could create more tools like that, however it sucks for creating custom inspection rules because of its API;
ActiveWriter - Do you want to create a full domain model compatible with Active Record and/or NHibernate? Using ActiveWriter is the simplest way to implement it;

Resharper Jedi

2007-05-23T00:25:00.000-03:00

Reading the blogs from some guys that went to DevTeach, I've learned a new term that emerged (I think) from there, "Resharper Jedi".

Oren (Ayende) seems to be the official Resharper Jedi, elected by the guys during DevTeach. In his Hibernating Rhinos series you can see how fast he uses a lot of features from Resharper.

In my teams I try to convince every developer to learn at least the most important Resharper shortcuts, it really really improves the programming speedy. I personally rarely use a mouse inside visual studio due Resharper features shortcuts.

I cannot imagine myself using Visual Studio without Resharper anymore; it seems only a text editor with an integrated compiler/debugger. However, I'm happy with VS + Resharper, but I still would like a "JetBrains IDE for .NET", kinda "IntelliN".

Talking about Resharper, I just found a new blog from a JetBrains' developer about Resharper. It looks like a nice blog:
http://resharper.blogspot.com/

Fast system <> real time system

2007-05-13T22:29:00.000-03:00

Why a lot of people think that systems that have fast response are "real time" systems? I often hear things like "It's a real time system, the values are constantly updated in real time.", "It's a real time system, it responds immediately after it receive a request". Real time is about compromise to give a response in a window of time, I mean, the system has to produce a response in x seconds, no matter what, it has to produce it in x seconds, point. A formal definition from http://www.dedicated-systems.com/:

A Real-Time System responds in a (timely) predictable way to unpredictable external stimuli arrivals. In short, a Real-Time System has to fulfill under extreme load conditions:

Timeliness: meet deadlines, it is required that the application has to finish certain tasks within the time boundaries it has to respect;
Simultaneity or simultaneous processing: more than one event may happen simultaneously, all deadlines should be met.;
Predictability: the real-time system has to react to all possible events in a predictable way;
Dependability or trustworthiness: it is necessary that the real-time system environment can rely on it.

Common examples of real time system are: automobile airbag systems, temperature control system of a nuclear power plants, etc.

In one of my current projects we have to control PLCs for pallets transportation in a warehouse, we have to reply the PLCs quick, otherwise the efficiency of the warehouse goes down, but it's far from being a real time system, even so I bet someone (colleagues, partners, client, providers, etc.) will call it a real time in some moment in the future.

The majority of systems people call real time are .NET or Java (not JSR-001/JSR-282) systems. However, the garbage collector in .NET and Java is not deterministic, so you cannot be sure about when some things will happen. For this reason, you cannot build real time systems in .NET or Java, at least not a “hard” real time system.

With JSR-001/JSR-282, Java will be able to build hard real time system; however, I've never saw anything similar for .NET, unfortunately.

My default .NET data access strategy - NHibernate based approach

2007-05-09T00:42:00.000-03:00

In my new project I and my team are being able to use a set of tools that I have been looking at for some time for data access in .NET. The approach is based on well know open source ORM framework: NHibernate, however, we are using other fantastic tools above it.

Instead of creating NHibernate mapping files, we are using Castle Active Record (AR) attributes to decorate our domain model classes. Analyzing the attributes AR can generate the mapping files dynamically, so we don’t have to deal with a lot of xmls files. We have something like this.

But we also don’t create the domain model directly, we use the Active Writer, which is a VS add-in that works like VS Class Designer, but it generates classes decorated with the AR attributes. So, it’s very fast to create a full domain model mapped using the AR attributes.

However, we don’t use the ActiveRecordBase class as base class for our model in order to avoid getting the persistence methods in our domain model. We prefer to use repositories (DDD flavor).

For the repositories we use a slightly modified version of Rhino Commons ARRepository (generic specialization). The implementation uses the ActiveRecordMediator which already have most of basic persistence operations, such as Save, Delete, FindAll, FindOne, etc.

For custom queries, we try to avoid using the NHibernate query API directly; instead we use the Ayende’s NHibernate Query Generator tool. It allows us to have strong typed queries that feels like SQL Queries, much better/natural in my opinion.

So, we have something like this in the EquipmentRepository:

Equipment FindDougEquip()
{
return FindOne(Where.Equipment.Name == “DouglasEquip” && Where.Equipment.Id > 0);
}

Well, I think this approach is very productive; we have a full persistable domain model and basic repository classes, besides the strong typed queries in few minutes. Really really fantastic open source tools.

First post - Introducing myself

2007-05-07T00:24:00.000-03:00

Well, this is my first post so I think it would be a good idea to introduce myself to anyone who could find this blog or even read it and don't know me yet.

As you can find somewhere in this page, my name is Douglas and I'm from São Paulo, Brazil. I'm an Electrical/Computer Engineer graduated at Poli-USP. I work as a Team Leader at Chemtech, an industrial services company from Siemens group. I usually work at São Paulo office, mainly in software development projects for a range of industries such as metals and mining, logistics, water and electric power, etc. Chemtech is a really nice place to work, I'll post about it in the future.

I'm a huge fan of modern/agile software development practices like refactoring, continuous integration, TDD, DDD and techniques like IoC/DI, ORM, AOP, etc.. Most of my projects uses .NET platform (my favorite), but I already did some huge projects in Java EE world, which I like as well.

I'm always looking for great tools, whenever possible, open source one. In .NET world I (and my teams) use a lot of them, such as Castle Project stack (Windsor, Active Record, Active Writer, Dynamic Proxy), NHibernate, Ayende tools (Rhino Mocks, NQG, NQA, Commons), Cruise Control .NET, NAnt, NUnit, etc.. While JetBrains does not create a kick ass .NET IDE, I use Visual Studio with its fantastic Resharper add-in. In Java world I use the similar ones.

I'm also very interested in project management, I like to compare and merge/use whenever possible agile practices such as Scrum and XP with PMI PMBoK guidance in my projects.

Well, so it's the end of my first post. In the future I'll post more about my life in software development industry and other random stuff.

More about me in Linkedin:

http://www.linkedin.com/in/dougfernando