Web Spider Research Blogs

David & Miranda

2006-04-26T13:17:00.000-07:00

The paper so far includes all of the goals and purpose of the project. We also include a summary of the process used to complete different projects. Next week we'll edit the whole thing so its presentable to the CREU.

As far as the programming goes, we added a button to the site that fetches all of the new MD5 signatures. This makes the site much more efficient by allowing one to go to the page without being required to wait for every single page in the database to be examined. We did this mostly because we need to change our program to work with the Webguide database instead of our small test database. Having the page take a signature of every site in the official database would take a very long time.

Miranda & David

2006-04-19T13:01:00.000-07:00

Today we finally got our Approve buttons to work! We've added in a new test page on the local server on order to easily play around with this (adding changes). One simply presses 'approve' in order to input the new md5 signature into the database.

Other than our direct site, we have begun working on our final report for this project. Right now we have an outline of all the work we've done during the year--it has turned out to be a lot more than we remembered.

Miranda & David

2006-04-12T13:06:00.000-07:00

These past two weeks, we'd been having a lot of issues getting our buttons to somehow call a function that will properly correspond to the row we desire. We've switched between normal buttons, to a button template, to a button column within our DataGrid and still had issues. We finally got a related example to work on our machines, so next week we'll be working on adjusting this to our needs.

Miranda & David

2006-03-29T13:17:00.000-08:00

This week we managed to squeeze "approve" buttons into our table (now a datagrid instead of generic table). The code behind the datagrid buttons do not execute any real functions yet but that should be easy to add next week.

We originally ordered our urls using a "link ID" as an index. We now use the actual url instead of the link ID with an understanding that no two sites can possibly have the same url.

Our goal for next week is to get the approve button working so that a person can visit the website given with a different signature and "Approve" that change by inputting the new signature in the table. From then on the site will be checking the database for MD5 comparisons with the newly approved page instead of the old page.

Of course we still need INSERT() and DELETE().

Miranda & David

2006-03-15T13:01:00.000-08:00

Today we finally finally got an MD5 signature to be recorded for websites listed in our database. New columns include spaces for MD5 signatures and boolean values indicating changes from previous signatures. The Object Oriented aspects of ASP.NET have made it easier for us to organize the disparate functions necessary to get new signatures and connect to the database.

Spring Break next week.

In two weeks time we plan to import the functions from our old ASP page to the new .NET page (insert/delete entries). Additionally we are hoping to allow the user to approve the page before updating an entry's signature.

Miranda & David

2006-03-08T12:57:00.000-08:00

Our goals for the day were to get the website to fetch HTML from a separate site and MD5() the text. We did get MD5() to finally work with ASP.NET however getting the HTML was a lot harder. We spent the bulk of our time trying to figure out how to use the internet transfer control to get HTML from a given URL. We were confused as to how one would declare the variable used in many examples called "inet" that would store and retrieve site information. Most of the online examples do not show which namespaces were included in order to showcase their abilities.

We have sent Dr. Couch an email and plan on grilling him about this issue tomorrow. The examples he found for us last week do not work with ASP.NET

Miranda & David

2006-03-01T12:59:00.000-08:00

Today we were able to get our connection to the server database back up using ASP.NET

This was a lot harder than we thought it would be. It seemed as if there are many different ways to connect to a database and only when we copied an example line by line from our book did it work.

Next week we hope to get HTML and use MD5.

love,

M&D

Miranda & David

2006-02-23T11:43:00.000-08:00

Today we realized we were writing all of our code in ASP and not ASP.NET. We found this out by trying to run various snippets of code from example sites. In an attempt to get MD5() up and running we were unable to get the "System" namespace to work correctly.

We have subsequently printed out our previous database manipulation code and next meeting we plan to translate this from VBScript to Visual Basic.

Tufts runs on a Monday Schedule this week so we will not be able to meet with Dr. Couch until next week. This gives us more time to catch up with the new language requirements.

Miranda & David

2006-02-15T12:51:00.000-08:00

Today we added the option of adding new data entries to the database. A person can type in new data values at the top of the page and the database will log and refresh the interface to show the new entries added. This proved difficult due to still learning VBScript and finding an index into our database id list which did not exist already. We had to rewrite a simple search 3 times to fix logical errors.

Tomorrow we meet with Dr. Couch to discuss ways of going forward. We have a good idea of what we want to do next. Upon hitting a "refresh" button every entry's web address in the database should be checked for broken links and changes from previous refreshes.

Return 0;

Miranda & David

2006-02-13T12:54:00.000-08:00

Today we worked on an interface for the database in which we can delete entries. This is the first step to being able to fully manipulate the cfw database. Next time we plan on adding insertion and updates to the database that show up on the interface. Right now we are using a temporary database so that we do not mess with our copy of the real cfw system.

Once our schedule permits we will implement an md5 change detection system which will be reported on the interface.

Miranda & David

2006-01-25T12:31:00.000-08:00

We met for the first time after the break. Our web page is now listed as a link on the CREU website which at the very least ranks us higher on google.

We turned in our midway reports and received positive responses from the program directors. David tried to set up the CFW webguide database system on his computer but for some reason SQL would not install correctly.

Today, we created a simple database and tried to pull values off the server as a test. We keep running into a problem where the server tells us we are using the file even though we closed all programs that edit the file. We set up a new weekly meeting schedule with Dr. Couch and we look forward to meeting with him next week to strategize .

We updated the website.

David & Miranda

2005-12-13T15:17:00.000-08:00

Recently we have been trying to get our ASP code to work with our server and have started work on the interface for the content management system.

Our ASP code has had a hard time viewing and manipulating the fake database we have constructed. Dr. Couch is still working on getting the webguide database into a usable format.

We expect to write our end of the semester report soon. It will include many of the problems we have encountered thus far.

Miranda & David

2005-11-06T20:58:00.000-08:00

Hello, world.

We finally were able to connect to the server but we found that our usernames do not have administrative acccess. Therefore we are unable to test ASP code and continue our research as planned. Even though we did spend much time on trying to find a way to test ASP code on our own computer and other websites we had access to, we were able to learn a few more concepts about ASP but only in an abstract sense.

Dr. Couch has been very busy lately and it is understandable that we are unable to make much progress when updates on our server access come weeks at a time. We will continue to try to learn ASP strictly from the books we've been given as well as online tutorials including the Comp20 website. Learning PERL and ASP while trying to build a content management system from scratch is an arduous task. However we are confident that we will be able to make a lot more progress once we have a server up and running. We should have an interesting report ready by the end of the semester.

David Dennis

2005-10-22T12:23:00.000-07:00

On Oct 20th we met again with Dr. Couch. We spent a little time before the meeting reading a bit from each book and informing each other about some interesting aspects.

Dr. Couch set up a server in his office that we could access from within the building using a remote desktop connection. We now have log-in names so we can start to use the database on there as a practice for our ASP programs. Dr. Couch emphasized that it would behoove the both of us to learn ASP as soon as possible.

The Offline Checker can be done in PERL so its now up to Miranda or I if we wish to start working on that in ASP or PERL. This is the easiest program of the three so we might as well work on this first.

David Dennis

2005-10-22T10:53:00.000-07:00

On Oct. 13th Miranda and I met with Dr. Couch to get some new guidance as well as set up a regular meeting schedule. We agreed that at 4:30 PM every Thursday we would meet in Dr. Couch's office. Dr. Couch literally sketched out for us the complete design of the system (images upcoming) and gave us two books. Miranda took the one on content mamnagement while I took the ASP tutorial book. The sketch showed us that the design for managing a website could be illustrated as a state machine where the document is either being submitted for listing, accepted, rejected, reviewed again, or non-existent (its master link leads nowhere).

Several Questions were also outlined:

1. What do we store about each page?
- How much?
- How long?
2. What constitutes a substantative change (When do we flag the reviewer?) ?
- Avoid false positives
- Make small changes non-problematic
3. How do we best present information to a user?
- A cached copy?
- Only changes highlighted?

Miranda and I now know the system contains only 3 programs:

1. Content Management "core"
- Present sites to database for re-review
2. Offline checker
- Given a list of links, the checker will log for the database what links
are traversable and what links do not go anywhere.
3. Display program
- List for the website all the links that are approved.

David Dennis

2005-10-09T17:30:00.000-07:00

After working hard on this preparation project, Miranda and I felt that it was time to go directly after our project goals. On 10/6 we quit trying to copy the html of the webguide after realizing the complexity of the project was starting to outweigh the benefit of the exercise.

Our next project shall be to create a web interface for our project where a list of sites can be added and their md5 signatures monitored every night. From this model we can start to see how to handle the specific change cases outlined in our proposal.

We will start our interface in PERL. We have plans to change to ASP.NET next semester.

David & Miranda

2005-10-02T13:04:00.000-07:00

Today we worked three hours. At the beginning we tried to eliminate repeated links on the first page being examined by the website copying program.

We succeeded at making the program list all important first-depth links. We continued by opening a mother folder for the entire website in our directory called "guide" and then made the program recurse through all links and create folders where folders exist in the url extension and create files where files exist in the extenstion. We did not yet handle first level files or implied files (such as those urls only listing folders at the end and relying on automatically finding index.html), but this should be easy.

We have not yet put the HTML content into their respective files. Dr. Couch has been very busy and has not gotten back to us regarding the database access we requested.

Miranda's serverspace quota has been increased to 500MB and David's has been increased to 200MB. This is in preparation for all of the application code needed to run the webguide website programs.

We are getting a better feel for PERL with every meeting.

Our next meeting should consist of finishing the copying program by starting link recursion and incorporating the MD5() algorithm to look for changes.

David & Miranda

2005-09-17T14:41:00.000-07:00

Today we looked at spider.pl and tried to adapt the program to only retrieve links with a starting url that we define. We had a little success but need to look into how to avoid duplicate urls being listed. We also need to make this program recursive. In order to fully "steal" the CFW website we need to be able to open new directories and copy new files into them. We started studying this by learning how to open and close files and directories in PERL.

Dr. Couch still had not gotten back to us regarding the CFW database access we requested. We are still learning ASP.

David Dennis

2005-09-11T20:03:00.000-07:00

Miranda and I met at 4pm on Thursday the 8th to really see where each of us were in terms of programming in PERL. We acknowledged that we still had not touched the assignment so that was a good place to start our meeting. We managed to make a PERL program that was able to go the Webguide website and use the MD5 algorithm to generate a 128-bit number based on the HTML content. This was very impressive to both us seeing as we thought our PERL skills were still extremely basic. After this hard work we met with Dr. Couch in his office to discuss where to go from here. Dr. Couch praised our progress and decided that now was the time to get all the Webguide website's content copied to our accounts. We concluded that we would need this website and all of its programming in order to understand how the entire system works currently, and how we can take it to the next level. Dr. Couch informed us that now that we had conquered PERL (somewhat) we needed to start learning how to use ASP and its abilities to query databases. Miranda has had experience with databases as she has taken the CS@Tufts databases course. I have has no experience save for hearing "MySQL" every once in a while. The Comp20 website does have a lot of documentation on these subjects so I see no reason why I should not be able to learn all that there is to know about table generation and ASP soon. Dr. Couch is currently contacting those in charge of the CS databases and trying to obtain a user account for Miranda and I. Until he contacts us and informs us that we have both an account and a copied version of the Webguide, Miranda and I will try to learn all that there is to know about playing with this system.

I am very excited about our progress. A sample of the program's output so far:

sunfire13{ddenni01}56: perl my_program
Received the website's content
Digest is "0e755daf497d6bfdd649d36d7855cdb9"

The output states that a response was heard from the Website's server and that it's unique 128-bit number (Shown in Hexidecimal) is as seen.

Miranda Steed

2005-08-26T11:56:00.000-07:00

So I flew into Boston on a red-eye Tuesday night. I read DryDock on the plane and now have a bit more of an understanding of how this project will work, though I still think I'm lacking comprehension in a few areas.

Turns out my house doesn't have internet yet, so I spent this afternoon in Halligan (the computer science and electrical engineering building) and spent some time going through the advanced Perl lectures on the Comp 20 website. I didn't get to a lot of it, but I printed it all out, so that way I can do some learning in my internet-less home, without trekking the 15 minutes to Halligan.

David Dennis

2005-08-09T23:25:00.000-07:00

I read DryDock tonight. I can see why Dr. Couch wanted us to read this. It's a good example of a program like ours designed to deal with a lot of documents. And while the DryDock program was designed to let managers approve documents before being pushed out to the world, our program will have to approve documents every day to amke sure they can stay inside the webguide.

I messed with some PERL today. I thought about converting our research page to a cgi that would manually grab our posts from blogger instead of going through feedburner. That way we could easily customize text styles to make this look a little nicer.

Miranda Steed

2005-08-08T15:09:00.000-07:00

Finished going through all of Prof. Couch's basic Perl lectures from the Comp20 course website. I'm hoping to finish all the advanced Perl lectures before I head back to Massachusetts on the 23rd. Maybe even get to some of the readings/research Prof. Couch recommended.

My life at the moment, however, is mostly occupied by studying for the GRE general test, which I'm taking on the 22nd. I set off this afternoon for Spider-related activities, but I think I'll have to start studying some new vocabulary words for the rest of the evening. I have so much work still to go!

When I get back to campus, I'll be rehearsing with the Jackson Jills most of the time for our Freshmen Orientation shows, but will be spending most of my free time working on Spider-related things. Luckily, without homework/classes yet and the GRE general test over (though I still have the CS subject test in November), I won't have much else to do during that time other than eat and sleep.

David Dennis

2005-08-07T23:05:00.000-07:00

We started reading Drydock today. The document is a whole research article about a program called Drydock that was created to force system administrators to review all inside documents before they are displayed to the world. Our project is similar but different in that we are designing a program that reviews all outside documents before they are displayed on the webguide website.

I made this really nice site art today. I proboably should have been working on the program instead. I have one more week of work before I can really devote myself to getting started.

I gave Dr. Couch and Miranda the username and password needed to start posting on the site.

I'm trying to make the feed simply say "Web Spider Research Blogs" instead of "David's".

More to come.

David Dennis

2005-08-06T17:17:00.000-07:00

I took a look at DryDock today and found out that only the abstract is available online without a membership to the USENIX site.

I uploaded the new website and created an interface where our research updates can occur simply by posting a new blog on blogger.com

I downloaded spider.pl but didn't really look at it.

I'm not sure I totally understand how the site feed works or if I'm even doing it right but it works now so I'll just leave it at that.

I found out frontpage is 1000 times better to use when compared to Dreamweaver.

I hope to format the text soon to make it a little more beautified.