deixto.com/blog

Linked Data & DEiXTo

noreply@blogger.com (kntonas) — Sun, 19 Feb 2012 06:08:00 +0000

As explained in a previous post, DEiXTo can scrape the content of digital libraries, archives and multimedia collections lacking an API and enable their metadata transformation (through post-processing and custom Perl code) to Dublin Core and subsequently in OAI-PMH or another suitable form, e.g. Europeana Semantic Elements (ESE).
Meanwhile, the Web has become a dynamic collaboration platform that allows everyone to meet, read and more importantly write. Thus, it steadily approaches the vision of Tim Berners-Lee (the inventor of the World Wide Web): the Linked Data Web, a place where related data are linked and information is represented in a more structured and easily machine-processable way.

Linked Data refers to a set of best practices for publishing and connecting structured data on the Web. Its key technologies are URIs (a generic method to identify resources on the Internet), the Hypertext Transfer Protocol (HTTP) and RDF (a data model and a general method for conceptual description of things in the real world). It is an exciting topic of interest and it's expected to make great progress in the next few years. A video that does a nice job of explaining what Linked Open Data is all about can be found here: http://vimeo.com/36752317

Over the last decade, the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) has become the de facto standard for metadata exchange in digital libraries and it's playing an increasingly important role. However, it has two major drawbacks: it does not make its resources accessible via dereferencable URIs and it provides only restricted means of selective access to metadata. Therefore, there is a strong need for efficient tools that would allow metadata repositories to expose their content according to the Linked Data guidelines. This would make digitized items and media objects accessible via HTTP URIs and query able via the SPARQL protocol.
Dr Haslhofer has performed significant research and work towards this direction. He has developed (among others) the OAI2LOD Server based on the D2R Server implementation and wrote the ESE2EDM converter, a collection of ruby scripts that can convert given XML-based ESE source files into the RDF-based Europeana Data Model (EDM). These remarkable tools could turn out very useful for making large volumes of information Linked-Data ready, with all the advantages this brings.
Linked Open Data can change the computer world as we know it. So, there is a lot of potential in combining DEiXTo with Linked Data technologies. Their blend could eventually produce an innovative and useful outcome. Many already believe that Linked Data is the next big thing. Time will tell. Meanwhile, DEiXTo could definitely help you generate structured data in a variety of formats from unstructured HTML pages, either your ultimate goal is Linked Data or not.

DEiXTo components clarified

noreply@blogger.com (kntonas) — Sat, 11 Feb 2012 07:42:00 +0000

From the emails and feedback received, it seems that many people get a bit confused about the utility and functionality of the DEiXTo GUI tool compared to the Perl command line executor (CLE). DEiXToBot is even more confusing for quite a few users. So, let's clarify things.

The GUI tool is freeware (available at no cost but without any source code, at least yet) and it allows you to visually build and execute extraction rules for web pages of interest with point and click convenience. It offers you an embedded web browser and a friendly graphical interface so that you can highlight an element/ record instance as the mouse moves over it. The GUI tool is a Windows-only application that harnesses Internet Explorer's HTML parser and render engine. It is worth noting that it can support simple cooperative extraction scenarios as well as periodic, scheduled execution through batch files and the Windows Task Scheduler. Perhaps its main drawback is that it can execute just one pattern on a page although for several cases (maybe for the majority) one and only extraction rule is enough to get the job done.

On the other hand, the command line executor, or CLE for short, is implemented in Perl and it is freely distributed under the GNU General Public License v3, thus its source code is included. Its purpose is to execute wrapper project files (.wpf) that have previously been created with the GUI tool. It runs on a DOS prompt window or on a Linux/ Mac terminal. Besides the code though, we have built two standalone executables so that you can run CLE either on a Windows or a GNU/Linux machine without having Perl or any prerequisite modules installed. CLE is faster, offers more output formats and has some additional features such as an efficient post-processing mechanism and database support. However, it shares the same shortcoming as the GUI tool: it supports just one pattern on a page. Finally, it relies on DEiXToBot, a "homemade" package that facilitates the execution of GUI DEiXTo generated wrappers.

DEiXToBot is the third and probably the most powerful and well-crafted software component of the DEiXTo scraping suite and it is available under the GPL v3 license. It is a Perl module based on WWW::Mechanize::Sleepy, a handy web browser Perl object, and several other CPAN modules. It allows extensive customization and tailor-made solutions since it facilitates the combination of multiple extraction rules/ patterns as well as the post-processing of their results through custom code. Therefore, it can deal with complex cases and cover more advanced web scraping needs. But it requires programming skills in order to use it.

The bottom line is that DEiXToBot is the essence of our long experience. The GUI tool might be more suitable for most every-day users (due to its visual convenience) but when things get difficult or the situation requires a more advanced solution (e.g. scheduled or on-demand execution and coordination of multiple wrappers on a GNU/Linux server), a customized DEiXToBot-based script is your choice. You can use the GUI tool first to create the necessary patterns and then deploy a Perl script that uses them to extract structured data from the pages of the target website. So, if you are familiar with Perl, you should not find it very hard to write your first deixto-based spider/ crawler!

Federated searching & dbWiz

noreply@blogger.com (kntonas) — Sat, 28 Jan 2012 00:03:00 +0000

Nowadays, most university and college students, professors as well as researchers are increasingly seeking information and finding answers on the open Web. Google has become the dominant search tool for almost everyone. Its popularity is enormous, no need to wonder or analyze why. It's just great and it returns "good enough" results really fast.

However, libraries, in their effort to win some patrons back, have tried to offer a decent searching alternative by developing a new model: federated search engines. Federated searching (also known as metasearch or cross searching) allows users to search simultaneously multiple web resources and subscription-based bibliographic databases from a single interface. To achieve that, parallel processes are executed in real time and retrieve results from each separate source. Τhen, the results returned get grouped together and presented to the user in a unified way.

The mechanisms used for pulling the data from the target sources are broadly two: either through an Application Programming Interface (API) or via scraping the native web interface/ site of each database. The first method is undoubtedly better but very often a search API is not available. In such cases, web robots (or agents) come into play and capture information of interest, typically by simulating a human browsing through the target webpages.
Especially in the academia, there are numerous online bibliographic databases. Some of them offer Z39.50 or API access. However, a large number still does not provide protocol-based search functionality. Thus, scraping techniques should be deployed for those (unless the vendor disallows bots).

When starting my programming adventure with Perl back in 2006, in the context of my former full-time job at the Library of University of Macedonia (Thessaloniki, Greece), I had the chance (and luck) to run across dbWiz, a remarkable open source, federated search tool developed by the Simon Fraser University (SFU) Library in Canada. I was fascinated with Perl as well as dbWiz's internal design and implementation. So, this is how I met and fell in love with Perl.

dbWiz offered a friendly and usable admin interface that allowed you to create search categories and select from a global list of resources which databases would be active and searchable. If you had to add a new resource though, you would have to write your own plugin (Perl knowledge and programming skills were required). Some of the dbWiz search plugins were based upon Z39.50 whereas others (the majority) relied on regular expressions and WWW::Mechanize (a handy web browser Perl object).
The federated search engine developed while working at the University of Macedonia (2006-2008) was named "Pantou" and became a valuable everyday tool for students and professors of the University. The results of this work were presented at the 16th Panhellenic Academic Libraries Conference (Piraeus, 1-3 October 2007). Unfortunately, its maintenance stopped at the end of 2010 due to the economic crisis and severe cuts in funding. Consequently, a few months later some of its plugins started falling apart.

Generally, delving into dbWiz taught me a lot of lessons such as web development, Perl programming and GNU/Linux administration. I loved it! Meanwhile, in my effort to improve the relatively hard and tedious procedure of creating new dbWiz plugins, I put into practice an early version of GUI DEiXTo (which was my MSc thesis being fulfilled in the same period at the Aristotle University of Thessaloniki). The result was a new Perl module that allowed the execution of W3C DOM-based, XML patterns (built with the GUI DEiXTo) inside dbWiz and eliminated, at least to a large extent, the need for heavy use of regular expressions. That module, which was the first predecessor of today's DEiXToBot package, got included in the official dbWiz distribution after contacting the dbWiz development team in 2007. Unfortunately, SFU Library ended the support and development of dbWiz in 2010.

Looking back, I can now say with quite a bit of certainty, that DEiXTo (more than ever before) can power federated search tools and help them extend their reach to previously inaccessible resources. As far as the search engines war is concerned, Google seems to triumph but nobody can say for sure what is going to happen in the next few years to come. Time will tell..

Open Archives & Digital Libraries

noreply@blogger.com (kntonas) — Thu, 19 Jan 2012 22:02:00 +0000

The Open Archives Initiative (OAI) develops and promotes interoperability standards that aim to facilitate the efficient dissemination of content. OAI has its roots in the open access and institutional repository movements and its cornerstone is the Protocol for Metadata Harvesting (OAI-PMH) which allows data providers/ repositories to expose their content in a structured format. A client then can make OAI-PMH service requests to harvest that metadata through HTTP.

openarchives.gr is a great federated search engine harvesting 57 Greek digital libraries and institutional repositories (as of January 2012). It currently provides access to almost half a million(!) documents (mainly undergraduate theses and Master/ PhD dissertations) and its index gets updated on a daily basis. It began its operation back in 2006 after being designed and implemented by Vangelis Banos but since May 2011 it is being hosted, managed and co-developed by the National Documentation Centre (EKT). What makes this amazing searching tool even more remarkable is the fact that it is entirely built on open source/ free software.

A tricky point that needs some clarification is that when a user searches openarchives.gr, the search is not submitted in real time to the target sources. Instead, it is performed locally on the openarchives.gr server where full copies of the repositories/ libraries are stored (and updated at regular time intervals).
The majority of the sources searched by openarchives.gr are OAI-PMH compliant repositories (such as DSpace or EPrints). Therefore, their data are periodically retrieved via their OAI-PMH endpoint. However, it is worth mentioning that non OAI-PMH digital libraries have also been included in its database. This was made possible through scraping their websites with DEiXTo and transforming their metadata into Dublin Core. So, more than 16.000 records from 6 significant online digital libraries (such as the Lyceum Club of Greek Women and the Music Library of Greece “Lilian Voudouri”) were inserted in openarchives.gr with the use of DEiXTo wrappers and custom Perl code.

Finally, it is known that digital collections have flourished over the last few years and enjoy growing popularity. However, most of them do NOT provide their contents in OAI-PMH or another appropriate metadata format. Actually, many of them (especially legacy systems) do NOT even offer an API or an SRW/U interface. Consequently, we believe that there is much room for DEiXTo to help cultural and educational organizations (e.g., museums, archives, libraries and multimedia collections) to export, present and distribute their digitized items and rich content to the outside world, in an efficient and structured way, through scraping and repurposing their data.

Netnography & Scraping

noreply@blogger.com (kntonas) — Tue, 17 Jan 2012 13:05:00 +0000

Netnography or digital ethnography, is (or should be) the correct translation of ethnographic methods to online environments such as bulletin boards and social sites. It is more or less doing the same that ethnographers do in actual places like squares, pubs, clubs, etc: observe what people say and do, and try to participate as much as possible in order to better understand what's involved in action and discourses. Using ethnography may answer a lot of what, when, who and how questions defining several everyday problems. However, netnography differs in many ways compared to ethnography; especially in the fashion it is conducted.

Forums, Wikis as well as the blogosphere are good online equivalents of public squares and pubs. There are not physical identities, but online ones; there are not faces, but avatars; there is no gender, age or any reliable info about physical identities, but there are voices discussing and arguing about common topics of interests.

The more popular a forum is, the more difficult it gets to follow it nethnographically. A nethnographer has to use a Computer Assisted Qualitative Data Analysis (CAQDA) tool (such as RDQA) on certain parts of the texts collected during his research. In a forum use case, these texts would be posts and threads. If the researcher has to browse the forum and manually copy and paste its content, a huge amount of effort would be required. However, this obstacle could be surpassed through scraping the forum with a web data extraction tool such as DEiXTo.

A scraped forum is a jewel: perfectly ordered textual data corresponding to each thread, ready for further analysis. So, this is where DEiXTo comes into play and may boost the research process significantly. To our knowledge, Dr Juan Luis Chulilla Cano, CEO of Online and Offline Ltd., has been successfully utilizing scraping techniques so as to capture the threads of popular Spanish forums (and their metadata) and transform them into a structured format, suitable for post-processing. Typically, such sites have a common presentation style for their threads and offer rich metadata. Thus, they are potential goldmines upon which various methodologies can be tested and applied so as to discover knowledge and trends and draw useful conclusions.

Finally, netnography and anthropology seem to be gaining momentum over the last few years. They are really interesting as well as challenging fields and scraping could evolve to an important ally. It is worth mentioning that quite a few IT vendors and firms employ ethnographers for R&D and testing of new products. Therefore, there is a lot of potential in using computer aided techniques in the context of netnography. So, if you are coming from social sciences and creating wrappers/ extraction rules is not your second nature, why don't you drop us an email? Perhaps we could help you gather quite a few tons of usable data with DEiXTo! Unless terms of use or copyright restrictions forbid it..

Geo-location data, Yahoo! PlaceFinder & Google Maps API

noreply@blogger.com (kntonas) — Thu, 12 Jan 2012 22:03:00 +0000

Location-aware applications have known huge success over the last few years and geographic data have been used extensively in a wide variety of ways. Meanwhile, there are numerous places of interest out there, such as shopping malls, airports, restaurants, museums, transit stations and for most of them their addresses are publicly available on the Web. Therefore, you could use DEiXTo (or a web data extraction tool of your choice) in order to scrape the desired location information for any points of interest and then postprocess it so as to produce geographic data for further use.

Yahoo! PlaceFinder is a great web service that supports world-wide geocoding of street addresses and place names. It allows developers to convert addresses and places into geographic coordinates (and vice versa). Thus, you can send an HTTP request with a street address to it and get the latitude and longitude back! It's amazing how well it works. Of course, the more complete and detailed the address, the more precise the coordinates returned.

In the context of this post, we thought it would be nice, mostly for demonstration purposes, to build a map of Thessaloniki museums using the Google Maps API and geo-location data generated with Yahoo! PlaceFinder. The source of data for our demo was Odysseus, the WWW server of the Hellenic Ministry of Culture that provides a full list of Greek museums, monuments and archaeological sites.

So, we searched for museums located in the city of Thessaloniki (the second-largest city in Greece and the capital of the region of Central Macedonia) and extracted through DEiXTo the street addresses of the ten results returned. At the picture below you can see a sample screenshot from the "INFORMATION" section of the Folk Art and Ethnological Museum of Macedonia and Thrace Odysseus detailed webpage (from which the address of this specific museum was scraped):

After capturing the name and location of each museum and exporting them to a simple tab delimited text file, we wrote a Perl script harnessing the Geo::Coder::PlaceFinder CPAN module in order to automatically find their geo-location coordinates and create an XML output file containing all the necessary information (through XML::Writer). Part of this XML document is displayed right below:

After having all the metadata we needed in this XML file, we utilized the Google Maps JavaScript API v3 and created a map (centered on Thessaloniki) displaying all city museums! To accomplish that goal, we followed the helpful guidelines given in this very informative post about Google Maps markers and wrote a short script that parsed the XML contents (via XML::LibXML) and produced a web page with the desired Google Map object embedded (including markers for each museum). Finally, the end result was pretty satisfying (after some extra manual effort to be absolutely honest):

This is kind of cool, isn't it? Of course, the same procedure could be applied in a larger scale (e.g. for creating a map of Greece with ALL museums or/and monuments available) or expanded to other points of interest (whatever you can imagine, from schools and educational institutions to cinemas, supermarkets, shops or bank ATMs). In conclusion, we think that the combination of DEiXTo with other powerful tools and technologies can sometimes yield an innovative and hopefully useful outcome. Since you have the raw web data at your disposal (captured with DEiXTo), your imagination (and perhaps copyright restrictions) is the only limit!

DEiXTo powers Michelin Maps and Guides!

noreply@blogger.com (kntonas) — Wed, 04 Jan 2012 07:57:00 +0000

One of the biggest success stories of DEiXTo is that it was used a few months ago by the Maps and Guides UK division of Michelin in order to build a France gazetteer web application. If you are going on holiday to France, probably you will need hotel and restaurant guides, maps, atlases and tourist guides relevant to where you are staying or the places you will visit. So, the free online Michelin database can help you find out which ones are for you.

The contribution of DEiXTo in the context of the implementation of this useful service was that it scraped from Wikipedia geo-location data as well as other metadata fields for 36.000+ French communes. In France the smallest administrative region is the commune and Wikipedia happened to have all of this relevant information freely available!

The starting target page contained a list of 95 (or so) departments, each of which containing a large number of communes. Thus, every department detailed page would in turn list all its communes and their corresponding hyperlinks/ URLs. A sample department page looks like this. And last, at a level below, we have the actual pages of interest with all the details needed about each commune. You can see a sample commune Wikipedia page by clicking here and a screenshot from it at the picture below. Meanwhile, this "scenario" also serves as a good example of collaborative wrappers where the output of a wrapper (a txt file with URLs) gets passed as input to a second one.

It should be noted though that there were slight variations in the layout and structure of the target pages. However, the algorithm DEiXTo uses is quite efficient and robust and usually can deal with such cases. To be more specific, the scraper that was deployed, extracted from each commune page the following metadata: region, department, arrondissement, canton and importantly the latitude and longitude.

The precision and recall that DEiXTo achieved with these commune pages was amazing (very close to 100%) and as a result the database was finally enriched with the large volumes of information captured. We are really happy that Michelin was able to successfully utilize DEiXTo and create a free and useful online service. So, if you plan a trip to France, you know where to find an informative online map/ guide! :)

Cooperating DEiXTo agents

noreply@blogger.com (kntonas) — Mon, 02 Jan 2012 12:15:00 +0000

Basically there are two major, broad categories of cooperating DEiXTo wrappers. In the first one, the wrappers are executed and applied on the same, single page so as to capture bits of interest that are scattered all over this particular target page. On the other hand, the second category comprises cases where the output of a wrapper serves as input for a second one. For the latter, typically the output of the first wrapper is a txt file containing the target URLs leading to pages with detailed information.

The first category is not supported directly by the GUI tool. However, DEiXToBot (a Mechanize agent object capable of executing extraction rules previously built with the GUI tool) allows the combination of multiple extraction rules/ patterns on the same page and their results through Perl code. So, if you have come across a complex, data-rich page and you are fluent with Perl and DEiXToBot's interface, you can build the necessary tree patterns separately with the GUI tool and then write a highly efficient set of cooperating Perl robots aiming at capturing all the desired data. It is not easy though since it requires programming skills and custom code.

As far as the second type of collaboration is concerned, we have stumbled upon numerous cases where a first wrapper collects the detailed target URLs from listing pages and passes them to a second wrapper which in turn takes over and gathers all data of interest from the pages containing the full text/ description. A typical case would be a blog or a news site or an e-shop, where a first agent could scrape the URLs of the detailed pages and a second one would visit each one of them extracting every single piece of desired information. If you wonder how you can set a DEiXTo wrapper to visit multiple target pages, this can be done either through a text file containing their addresses or via a list. Both ways can be specified in the Project Info tab of the DEiXTo GUI tool.

Moreover, for the first wrapper which is intended to scrape the URLs, you only have to create a pattern that locates the links towards the detailed pages. Usually this is easy and straightforward. You should just point at a representative link, use it as a record instance and set the A rule node as "checked" (right click on the A node and select "Match and Extract Content"). The resulting pattern will be something like this:

Then, via executing the rule you can extract the "href" attribute (essentially the URI) of each matching link and export the results to a txt file, say target_urls.txt, which subsequently will be fed to the next wrapper. Please note that if you provide just the A rule node as a pattern, you will capture ALL the hyperlinks found on the page but we guess you don't want that (we want only those leading to the detailed pages).

In conclusion, DEiXTo can power schemes of cooperative robots and achieve very high precision. Especially for more advanced cases, synergies of multiple wrappers are always needed. Their coordination though usually needs some careful thought and effort. Should you have any questions, please do not hesitate to contact us!

Robots.txt & access restrictions

noreply@blogger.com (kntonas) — Tue, 27 Dec 2011 22:33:00 +0000

A really serious matter that often many people ignore (deliberately or not) is the access and copyright restrictions that several website owners/ administrators impose. A lot of websites want robots entirely out. The method they use to keep cooperating web robots out of certain site content is a robots.txt file that resides in their root directory and functions as a request that visiting bots ignore specified files or directories.

For example, the next 2 lines indicate that robots should not visit any pages on the site:
User-agent: *
Disallow: /
However, a large number of scraping agents violate the restrictions set by the vendors and content providers. This is a very important issue and it raises significant legal concerns. Undoubtedly, there has been an ongoing raging war between bots and websites with strict terms of use. The latter deploy various technical measures to stop robots (an excellent white paper about detecting and blocking site scraping attacks is here) and sometimes even take legal action and resort to courts. There have been many cases over the last years with contradictory decisions. So, the whole issue is quite unclear. You can read more about it in the relevant section of the "Web scraping" Wikipedia article. Both sides have their arguments, so it's not at all an easy verdict.

The DEiXTo command line executor by default respects the robots.txt file of potential target websites (through the use of the WWW::RobotRules Perl module). Nevertheless, you can override this configuration (at your own risk!) by setting the -nice parameter to 0. It is strongly recommended though that you comply with webmasters' requests and keep out of pages that have access restrictions.

Generally speaking, data copyright is a HUGE issue, especially in today's Web 2.0 era, and has sparked endless discussions and spawned numerous articles, opinions, licenses, disputes and legitimacy issues.

By the way, it is worth mentioning that currently there is a strong movement in favor of openness in data, standards and software. And according to many, openness fosters innovation and promotes transparency and collaboration.

Finally, we would like to suggest to everyone using web data extraction tools to comply with the terms of use that the websites set and think twice before deploying a scraper, especially if the data is going to be used for commercial purposes. A good practice is to contact the webmaster and ask for permission accessing and using their content. Quite a few times the website might be interested in such a cooperation mostly for marketing and advertising reasons. So, as soon as you get a "green light", start building your scraper with DEiXTo and we are here to help you!

Downloading images with DEiXTo and wget

noreply@blogger.com (kntonas) — Sun, 25 Dec 2011 09:34:00 +0000

Many people often download pictures and photos from various websites of interest. Sometimes though, the number of images that someone wants to download from certain pages is large. So large that doing it manually is almost prohibitive. Therefore, an automation tool is often needed to save users time and repetitive effort. Of course, towards this goal, DEiXTo can help.

Let's suppose that you want to get all images from a specific web page (with respect to terms of use). You can easily build a simple extraction rule by pointing at an image, using it as a record instance and setting the IMG rule node as "checked" (right click on the IMG node and select "Match and Extract Content"). The resulting pattern will be like this:

Then, via executing the rule you can extract the "src" attribute (essentially the URI) of each image found on the page and export the results to a txt file, let's say image_urls.txt. And last, you can use GNU Wget (a great free command line tool) in order to retrieve the files. You can download a Windows (win32) version of wget here. For example, on Windows you can then just open a DOS command prompt window, change the current working directory to the folder wget is stored (via the 'cd' command) and enter:
wget.exe -i image_urls.txt
where image_urls.txt is the file containing the URIs of images. And voilà! The wget utility will download all the images of the target page for you!
What about getting images from multiple pages? You will have to explicitly provide the target URLs either through an input txt file or via a list. Both ways can be specified in the Project Info tab of the DEiXTo GUI tool.

Thus, if you have the target URLs at hand or you can extract them with another wrapper (generating a txt file), then you can just pass them as input to the new image wrapper and the latter will do the laborious work for you.

In case all the above are a bit unclear, we have built a sample wrapper project file (imdb_starwars.wpf) that downloads all Star Wars (1977) thumbnail photos from the corresponding imdb page. Please note that we set the agent to follow the Next page link so as to gather all thumbnails since they are scattered across multiple pages. However, if you would like to get the large size photos you will have to add another scraping layer for extracting the links of the pages containing the full size pictures.

Anyway, in order to run the sample wrapper for the thumbnails, you should open the wpf (through the Open button in the Project Info tab) and then press the "Go!" button. Or alternatively you can use the command line executor instead on a DOS prompt:
deixto_executor.exe imdb_starwars.wpf
Finally, you will have to pass the image_urls.txt output file to wget in order to download all thumbnails and get the job done! May the Force be with you! :)

Can DEiXTo power mobile apps? Yes, it can!

noreply@blogger.com (kntonas) — Thu, 22 Dec 2011 19:11:00 +0000

Web content scraped with DEiXTo can be presented in a wide variety of formats. However, the most common choice is probably XML since it facilitates heavy post-processing and further transformations so as to make the data suit your needs. A potentially interesting scenario would be to output bits of interest from a target website into an XML file and then transform it to HTML through XSLT (Extensible Stylesheet Language Transformations). This could be very practical and useful for creating in real time a customized, "shortened" version of a target web page specifically for mobile devices (e.g. Android and iOS devices).
As you all know smartphones and tablets over the last few years have changed the computer world. So, we thought it would be challenging and hopefully useful to build a web service capable of repurposing specified pages on the fly (through the use of a DEiXTo-based agent), keeping only the important/ interesting stuff and returning it in a mobile-compatible fashion, suitable to fit small screens by harnessing XML, XSLT and CSS. We did not fully implement the service but we got a simple prototype ready to try our idea. And the results were quite encouraging!

For the needs of our demo we used Tech Box, a popular technology news blog covering a plethora of interesting and fun topics around the IT industry. Tech Box currently has a remarkable iPhone app available allowing the user to browse through the latest news as well as save and share favorite stories. The tech.box mobile app, built by NSLoom (an innovative start-up creating seamless iOS applications), is really awesome. However, there is always room for improvement and often web scraping has something new to offer and extend current functionality.
In the context of the demo, we supposed that we wanted to search the Tech Box site with "iPhone" as a keyword. So, we built a quick (and dirty) test scraper able to extract all the records found on a result page and generate an XML document with the data captured. With the use of an elegant XSLT and a CSS we achieved a nice, usable and easy to navigate structure, suitable for a smartphone screen (illustrated in the picture above). You can see live how the output XML file (containing 10 sample headlines) looks like on an online iPhone simulator at the following address:

http://deixto.com/mobile/techbox.html

The concept of the proposed web service is the following: suppose that you are an app developer or a website owner/ administrator and that you need to display content inside your app or the mobile version of your site, either from a website of your control (meaning that you have access to its backend) or from another, "external" site (with respect to copyright and access restrictions). Often, though, it's not easy to retrieve data from the target website or simply you don't know how to do it.. Therefore, a service that could listen to requests for certain pages/ URIs and return their important data in a suitable form could potentially be very useful. For example, ideally an HTTP request like http://deixto.com/webservice.pl?uri="http://example.com/.." would result in a good-looking XML chunk (formatted with XSLT and CSS) containing the data scraped from the original page (specified with the uri parameter).
Finally, we would like to bring forward the fact that DEiXToBot contains best of breed Perl technology and allows extensive customization. Thus, it facilitates tailor-made solutions so as to make the data captured fully fit your project's needs. And towards this direction, deploying XSLT and XML-related technologies in general can really boost the utility and value of scraping and DEiXTo in particular!

APIs vs Scraping - Cl@rity & Yperdiavgeia

noreply@blogger.com (kntonas) — Sat, 17 Dec 2011 09:20:00 +0000

Typically there are two main mechanisms to search and retrieve data from a website: either through an Application Programming Interface commonly known as an API (if available) or via screen scraping. The first one is better, faster and more reliable. However, there is not always a search API available or perhaps even if there exists one, it may not fully cover your needs. In such cases, web robots, also called agents, are usually used in order to simulate a person searching the target website/ online database through a web browser and capture bits of interest by utilizing scraping techniques.

An API that has attracted some attention over the last few months in Greece is the Opendata API that the "Cl@rity" program ("Διαύγεια" in Greek) is offering. Since the 1st of October 2010, all Greek Ministries are obliged to upload their decisions and expenditure on the Internet, through the Cl@rity program. Cl@rity is one of the major transparency initiatives of the Ministry of Interior, Decentralization and e-Government. Each document uploaded is digitally signed and given a transaction unique number automatically by the system.

The Opendata API offers a variety of search parameters such as organization, type, tag (subject), ada (the unique number assigned), signer and date. However, there are still a lot of parameters and functionality missing such as full text search as well as searching by certain criteria like beneficiary's name, VAT registration number (ΑΦΜ in Greek), document title and other metadata fields.

A remarkable alternative for searching effectively through the documents of the Greek public organizations is yperdiavgeia.gr ("ΥπερΔιαύγεια" in Greek), a web-based platform built by the expert in digital libraries and institutional repositories Vangelis Banos. Yperdiavgeia is a mirror site of Cl@rity that gets updated on a daily basis and it provides a powerful and robust OpenSearch API which is far more usable and easy to harness. Its great advantage is that it facilitates full text searching. Currently, it lacks some parameters support but it seems that they are going to be added soon since it is under active development.

Even though both APIs mentioned above are really remarkable (especially for communicating and exchanging data with third party programs) there is still some room for utilizing scraping techniques and coming up with some "magic". In a previous post we had described in detail an application we developed mostly for downloading a user-specified number of the latest PDF documents of a specific organization uploaded to Cl@rity. We believe that this little utility we created (offering both a GUI as well as a command line version) can be quite useful for many people working in the public sector and potentially save a lot of time and effort. For further information about it, please check out this post (although it is written in Greek).

So, in this short post we just wanted to point out that there are quite a lot of great APIs out there, provided mostly by large organizations (e.g., firms, governments, cultural institutions and digital libraries - collections) as well as the major players of the IT industry such as Google, Amazon, etc, offering amazing features and functionality. Nevertheless, scraping the native web interface of a target site can still be useful and sometimes come up with a solution that overpasses difficulties or/and inefficiencies of APIs and yield an innovative outcome. Moreover, there are numerous websites that do not offer an API, thus a scraper could perhaps be deployed in case data searching, gathering or exporting would be needed. Therefore, the "battle" between APIs and scraping still rages.. and we are eager to see how things will evolve. Truth be told, we love them both!

DSpace & Institutional Repositories

noreply@blogger.com (kntonas) — Tue, 13 Dec 2011 21:23:00 +0000

Institutional Repositories (IRs) have emerged over the last few years and became very popular in the academic library world. The system that has dominated the "market" is DSpace, an exciting, functionality-rich, open source software package that is installed at over 1.100 institutions around the globe. It offers an OAI-PMH web service for harvesting the metadata of the repository and getting its entire contents in Dublin Core format. However, OAI-PMH does not provide advanced search by certain criteria such as title, author, supervisor, etc. Even the REST API which is under construction does not facilitate searching with these metadata fields, at least to the best of our knowledge. Moreover, the default DSpace OpenSearch support still seems incomplete and a bit buggy. Therefore, a potential solution for searching in real time a DSpace repository could be submitting a query and scraping the results returned through its native web interface. This could probably be useful for building a federated search engine or perhaps for creating a mobile app (currently there is no mobile version for DSpace).

Having in mind the lack of an "advanced", multiple-criteria enabled, search API / mechanism by DSpace, we thought it would be interesting and perhaps useful to write a test scraper that could submit queries to a DSpace repository and fetch the search results through its website. So, we built a simple, DOM-based, extraction rule (wrapper) with the DEiXTo GUI tool and then wrote a short DEiXToBot-based script that submits a sample query to Psepheda (the IR of University of Macedonia) and scrapes the results returned. The following picture illustrates the 10 first results for a sample query by title.

To get a better idea of how a Perl, DEiXToBot-based script works, below you can find the code that scrapes the 10 first items containing "programming" in title. The pattern used captures five metadata fields: detailed URL, title, date, authors and supervisor, and prints them on the screen. Of course this script can be easily extended to submit user specified queries as well as navigate through all the result pages by following the Next page link ("επόμενη" is the inner text of this link in Greek).

use DEiXToBot;
use Encode;
my $agent = DEiXToBot->new();
$agent->get('http://dspace.lib.uom.gr/simple-search?query=((title:programming))');

$agent->load_pattern('dspace_pattern.xml');
$agent->ignore_tags( [ 'em' ] );

$agent->build_dom();

$agent->extract_content();

for my $record (@{$agent->records}) {

print encode_utf8(join("\n",@{$record})),"\n\n";

}

DEiXToBot is written is Perl, thus it is portable and can run on multiple operating systems provided you have all the prerequisite Perl modules installed. You can download the lines of code given above along with the necessary pattern by clicking here. This short script serves as a good, simple example for utilizing the power and flexibility of DEiXToBot (a Mechanize agent object, essentially a browser emulator, which is able to execute patterns/ extraction rules previously built with the GUI tool).

Generally, IRs have huge potential and in the next few years they are expected to play an increasingly important role in storing and preserving digital content, academic or not. By the way, a great federated search engine harvesting numerous Greek digital libraries and institutional repositories is openarchives.gr which is mostly based upon OAI-PMH. It harnesses innovative technologies and has grown a lot since 2006 when it was first launched.

Last but not least, DEiXTo was used quite long ago by "Pantou", the federated search engine of the University of Macedonia, in order to scrape (in real time) multiple online resources simultaneously via their web interface/ site. It is worth noting that a predecessor of the current DEiXToBot module, back in 2007, was included in the official dbWiz distribution, a remarkable, open source, federated search software package upon which pantou was built.

myVisitPlanner & DEiXTo

noreply@blogger.com (kntonas) — Mon, 28 Nov 2011 15:50:00 +0000

We are very happy to announce that DEiXTo is going to power myVisitPlanner, an exciting project funded by the Greek Ministry of Education and Lifelong Learning, under the national action "COOPERATION 2011". myVisitPlanner is coordinated by the University of Macedonia and it is aiming at creating a personalized system for cultural itineraries planning. The consortium participants include Athena - Research and Innovation Center, GNOMON Informatics SA, the Ethnological Museum of Thrace and the West Macedonia Development Company (ANKO) SA.

DEiXTo-based wrappers are going to be deployed in order to automatically retrieve regional cultural itineraries and points of interest from various, heterogenous target websites. For a pre-defined set of webpages, DEiXTo scrapers will run periodically and monitor them for new content and coming cultural events. However, besides the traditional DOM-based tree patterns, new “smart” and innovative techniques, such as text mining and NLP algorithms, will be used so as to also contend with sites of unknown structure and layout.

myVisitPlanner officially started a few months ago and its duration is 36 months. We are really glad that we are participating in this challenging and exciting project and we hope that DEiXTo will help myVisitPlanner towards implementing its ambitious goals.

TEL-MAP & DEiXTo

noreply@blogger.com (kntonas) — Tue, 08 Nov 2011 20:36:00 +0000

DEiXTo was recently used successfully in the context of the TEL-MAP FP7 project in order to scrape the metadata of several European Technology Enhanced Learning (TEL) projects from the CORDIS website as well as from the European Commission website.

TEL-MAP is a Coordination and Support Action funded by the European Commission under the Technology-Enhanced Learning programme. It is coordinated by the Brunel University of London and it focuses on exploratory / roadmapping activities for fundamentally new forms of learning to support the adoption of those new forms, via awareness building and knowledge management on the results of EU RTD projects in TEL and socio-economic evaluations in education.

Currently, the TEL-MAP team is building a new portal called "Learning Frontiers". This portal aspires to become a widely recognized, single-point-of-access source of information for all European TEL. Among others, it offers a Projects space that contains a lot of detailed information about numerous TEL EU-funded projects and their participants (essentially the data scraped from the target web pages). It is really worth noting that through using the geo-location info of each participant / organization, Learning Frontiers provides an interactive map of TEL in Europe!

We are really happy that we have helped the Learning Frontiers portal a little towards the implementation of its ambitious vision to increase EU-wide and global dissemination, adoption and impact of EU TEL. We wish them best success!

DEiXTo & Athos Memory

noreply@blogger.com (kntonas) — Mon, 31 Oct 2011 18:45:00 +0000

In the context of our collaboration with the awarded Veria Central Public Library, a really remarkable Greek, online digital collection, Athos Memory, has been scraped through DEiXTo in order to be added to the European Library. Athos Memory has been a giant effort of the monastic community of the Holy Mountain to preserve and disseminate the unique religious tradition of the Eastern Orthodox Church on this peninsula of Chalcidice. Numerous people have worked tirelessly for years to make this endeavour possible. We would really like to congratulate and thank them for their great efforts and for providing open access to this magnificent collection.

The metadata of 27.223 photographs, documents and digitalized manuscripts from Athos, the Sacred Mountain of Christianity, have been transformed into Europeana Semantic Elements (ESE) format so that they could then be inserted into the Hellenic Aggregator's database.

To give you a better idea of the transformation process, check out the picture below. It's a screenshot of a typical item of Athos Memory archives.

Now, this record, after extracting and repurposing its metadata based on Dublin Core, gets the following form, suitable for exporting it:

Finally, it should be noted that this was the fourth digital library that was included in the Europeana with the help of DEiXTo. And we are eager to add more online resources and help Europeana enrich further its huge cultural and scientific collection!

DEiXTo & Veria Central Public Library

noreply@blogger.com (kntonas) — Sat, 08 Oct 2011 06:38:00 +0000

One of DEiXTo's most important success stories is our collaboration with Veria Central Public Library in the context of the EuropeanaLocal project. Veria Central Public Library is a really remarkable library that embraces technology and constitutes a successful model for libraries in Greece and around the world. That's why it received a 1$ Million international award from Bill & Melinda Gates Foundation in 2010.

DEiXTo powers the Hellenic Aggregator for Europeana, created by Veria Central Public Library. DEiXToBot based Perl scripts have enabled the metadata extraction of the Music Library of Greece “Lilian Voudouri”, the Greek Educational TV and Corgialenios Digital Library in a format suitable for further processing. Once extracted, their rich content was repurposed through customized Perl code and transformed into Europeana Semantic Elements (ESE) format so that it could then be inserted into the aggregator's database.

This is the reason why DEiXTo was cited at the Symposium "Europeana in Greece" that took place on 19 October 2010 in Athens, Greece, as well as at the 19th Hellenic Academic Libraries Conference (3-5 November 2010, Athens).

Hopefully, more digital libraries/archives will use DEiXTo in the next few months in order to be able to export their metadata to the great Europeana collection (more than 15 million items from 1.500 institutions!). And we are more than glad to help Europeana enrich its content even more!

DEiXToBot & Lack of JavaScript Support

noreply@blogger.com (kntonas) — Sat, 17 Sep 2011 14:23:00 +0000

Perhaps the major drawback of DEiXToBot (the Perl browser emulator object capable of executing GUI DEiXTo generated patterns) is the lack of JavaScript support, which derives from the fact that WWW::Mechanize doesn't operate on JavaScript. In many cases though, a solution is possible by figuring out what the JavaScript code is doing and simulating it via Perl programming. But in certain, more difficult cases that depend heavily on Javascript, this is very hard, if not impossible, because essentially you cannot reach the actual html source code of interest.

However, a workaround that sometimes works is to download the target pages of interest locally (after executing their Javascript segments of code) and then pass them to DEiXToBot for offline scraping.

Two remarkable tools for getting complex Javascript-enabled pages for this purpose are:

- Selenium, an amazing web browser automation tool and

- spynner, a powerful web browsing module with Ajax support for Python

Please note that these two great tools also work fine on GNU/Linux, which is really important, especially for server use and scheduled, periodic execution of wrappers.

So, once a target page is stored locally to your disk, the DEiXToBot agent can easily get the page through the file:// scheme and extract bits of interest in the usual manner.

Μεταφόρτωση PDF αρχείων από τον Δικτυακό τόπο της Διαύγειας

noreply@blogger.com (kntonas) — Sun, 26 Jun 2011 14:02:00 +0000

Εδώ και σχεδόν 9 μήνες, στο πλαίσιο της λειτουργίας του προγράμματος «Διαύγεια», όλα τα κυβερνητικά όργανα, οι φορείς του στενού και ευρύτερου δημόσιου τομέα και οι Ανεξάρτητες αρχές υποχρεούνται πλέον να αναρτούν το σύνολο των αποφάσεων και των δαπανών τους στο Διαδίκτυο και συγκεκριμένα στο δικτυακό τόπο της Διαύγειας (http://et.diavgeia.gov.gr).

Σε καθημερινή βάση λοιπόν, πολλοί εργαζόμενοι στο Δημόσιο έχουν επιφορτιστεί με την ευθύνη να ανεβάζουν στη Διαύγεια μεγάλο όγκο από εντάλματα πληρωμής και αποφάσεις σε pdf μορφή. Και μάλιστα συνήθως μετά την ανάρτησή τους και αφού έχουν πάρει ΑΔΑ (Αριθμός Διαδικτυακής Ανάρτησης), ο εκάστοτε αρμόδιος υπάλληλος πρέπει να τα "κατεβάσει" χειροκίνητα κάνοντας απανωτά κλικ στους αντίστοιχους συνδέσμους "Λήψη Αρχείου" και να τα τυπώσει, κατά βάση για γραφειοκρατικούς λόγους. Η διαδικασία αυτή ωστόσο είναι χρονοβόρα, ιδιαίτερα όταν το πλήθος των αρχείων είναι μεγάλο. Τυγχάνει να το γνωρίζω αυτό από πρώτο χέρι όντας εργαζόμενος στο Διεθνές Πανεπιστήμιο Ελλάδος.

Στην εικόνα που ακολουθεί φαίνονται μερικές τυπικές εγγραφές στο site της Διαύγειας για κάποιο φορέα. Βλέπετε δεξιά τους υπερσυνδέσμους (links) για μεταφόρτωση των ενταλμάτων πληρωμής.

Σκεφτήκαμε λοιπόν να φτιάξουμε μία μικρή εφαρμογή η οποία θα εντοπίζει τις διευθύνσεις (URLs) των pdf αρχείων πάνω σε μία σελίδα αποτελεσμάτων της Διαύγειας, π.χ. των Χ τελευταίων που ανέβασε κάποιος την προηγούμενη μέρα, και στη συνέχεια θα τα κατεβάζει μαζικά. Ένα τέτοιο πρόγραμμα πιθανότατα θα μπορούσε να βοηθήσει αρκετούς δημόσιους υπαλλήλους να εξοικονομήσουν κόπο και χρόνο.

Η εφαρμογή είναι διαθέσιμη τόσο σε εκδοχή για χρήση σε γραμμή εντολών (σε Windows και Linux) όσο και για χρήση μέσω γραφικής διεπαφής (GUI) σε περιβάλλον Windows (για το δεύτερο, δείτε στο τέλος του post).

Σε γραμμή εντολών των Windows (DOS prompt) τρέξτε το diavgeia-downloader.exe (ή το diavgeia-downloader-linux αντίστοιχα σε ένα terminal εάν έχετε Linux) περνώντας ως παράμετρο τη διεύθυνση/URL στόχο και βάζοντας στην παράμετρο limit της διεύθυνσης της σελίδας τον επιθυμητό αριθμό pdf αρχείων. Για παράδειγμα, έστω η σελίδα:

http://et.diavgeia.gov.gr/f/pamak/find/unit:4652/from:0/limit:50

που περιέχει τις τελευταίες 50 δαπάνες-αποφάσεις του ΕΛΚΕ του Πανεπιστημίου Μακεδονίας. Για λήψη αυτών των αρχείων τοπικά (στο φάκελο στον οποίο βρίσκεται και το εκτελέσιμο) αρκεί να δoθεί η ακόλουθη εντολή:

diavgeia-downloader.exe -url http://et.diavgeia.gov.gr/f/pamak/find/unit:4652/from:0/limit:50

Η εφαρμογή υποστηρίζει δύο επιπλέον προαιρετικές παραμέτρους: [-dir folder] [-sleep N]

όπου folder το όνομα του φακέλου στον οποίο θα αποθηκευθούν τα αρχεία και
N o αριθμός των δευτερολέπτων των χρονικών παύσεων μεταξύ των εντολών μεταφόρτωσης ώστε να μην επιβαρύνεται ιδιαίτερα ο server της Διαύγειας.

Μπορείτε να κατεβάσετε τόσο τον πηγαίο κώδικα (σε Perl) όσο και τα εκτελέσιμα αρχεία (για Windows & Linux αντίστοιχα)! Η άδεια χρήσης του προγράμματος είναι η GNU General Public License version 3. Για περιβάλλον Windows μάλιστα υπάρχει και η γραφική διεπαφή (GUI) που κάνει προφανή όλα τα, μάλλον, πολύπλοκα για πολλούς παραπάνω. Πιο κάτω βλέπετε ένα screenshot από το GUI εργαλείο. Η χρήση του είναι πολύ απλή, αρκεί ο χρήστης να δώσει με copy paste τη διεύθυνση της επιθυμητής σελίδας από το site της Διαύγειας και να πατήσει Go!

Ελπίζουμε η εφαρμογή αυτή να φανεί χρήσιμη. Σχόλια και προτάσεις ευπρόσδεκτα!

deixto.com/blog

noreply@blogger.com (kntonas) — Tue, 21 Jun 2011 21:54:00 +0000

Are you looking for a web content extraction tool to scrape data from websites of interest? DEiXTo can probably help you!

DEiXTo is an ongoing effort that started back in 2007. It is freely available to download and to our knowledge it is being used by many users as well as some organisations and companies all over the world. Indicatively, DEiXTo's site received more than 5.700 visits from 103 different countries during the last 12 months.

Today, we launch the blog of DEiXTo aiming a) to keep you apprised you about the wealth of applications and the utility of this exciting tool and b) to bring forward interesting topics around web scraping.

We really hope that DEiXTo can be useful for you and this blog helps you towards this direction.