A Fuzzzy blog

Experiences with Fuzzzy social bookmarking and ontology creation

2011-09-20T07:40:00.000-07:00

Fuzzzy.com is a social bookmarking tool with a twist. It lets users add and share bookmarks but it also has a folktology (mix between folksonomy and ontology) where users can create tags that have meaningful relations between them. Some of the things you can do on fuzzzy.com is to create tags, create tags relations between tags. Vote on relations between tags and between bookmarks and tags. With this voting feature you over time evolve more relevant tag-bookmark and tag-tag relations.

In this blog post I'll share some of my experiences with folktologies. Unfortunately I don't have an updated scientific study for the fuzzzy.com's folktology so you'll have to do with my personal observations. To learn more about fuzzzy.com and the folktology read my paper Metadata Creation in Socio-semantic Tagging Systems: Towards Holistic Knowledge Creation and Interchange.

Folksonomies, as we have come to know them on many social media sites, let users create any tag. This tag becomes available to everyone else. An Ontology on the other hand is a set of concepts within a domain, and the relationships between those concepts. Ontologies are the backbone of the Semantic Web so Fuzzzy was a project to see if the imprecise folksonomies could be replaced with more semantic folktologies. Some of the problems with folksonomies are synonyms, ambiguous tags, overly personalised and inexact, homonyms, plural and singular forms, conjugated words and compound words etc [1]. While folksonomies work just fine for a blog, it does not scale as well if semantics is important. As the folksonomy grows it grows into a large flat list of tags where a large percentage of tags just don't make any sense. To fix this problem they usually just show the tags that are used most frequently.

How the folktology of fuzzzy.com was used
A folktology seemed to be a smart use of Topic Maps. The folktology was seen as a way to make tagging more semantic but here's what happened:

Few users created relations between tags.
Few users assigned subject identifiers (needed to tell tags apart or if they represented the same thing)
Few users voted on relations between tags.
Few users voted on relations between tags and bookmarks.

Users did however use tags as if they were folksonomies. Users created both tags and bookmarks and added tags to bookmarks. Tags did have better quality [3] as users could edit tags and make them more consistent. Tags where more intuitive and made more sense when used on bookmarks. Common folksonomy problems such as synonyms, ambiguity, overly personalised, homonyms, plural and singular form tags etc were to a large degree weeded out.

I can only speculate on why the new way of using tags (folktology) was not adopted. Here's my personal hunches. I don't think there was a single reason but a mix of these reasons:

People did already have a fixed view on how tagging should work and did not care to learn about the other tag features.
People are in a hurry when tagging. They bookmark because thy don't have time to read the webpage. So its very important to be able to add bookmarks quickly and to assign tags just as quick.
Bookmarking is personal even though you use a social service. Every user has his own reasons for saving those particular bookmarks so he has no incentives to create an pretty shared ontology or tag network.
The rewards for creating tag to tag relations are perceived as low. Users don't see a direct value. With a built out folktology or semantic network, users can have more relevant bookmarks suggested to them. This is obviously a nice to have feature but it does not justify the work that goes into managing the folktology.

These interpretations are based on typical use of the system. Analyzing this further we begin to see contours of some general underlying issues.

Problems with a folktology
The social and semantic bookmarking service fuzzzy.com has a shared semantic tag set. This introduces several issues in comparison to plain and simple folksonomies. I summarize these issues as context and authority.

Who gets to update the individual tags? A tag is shared and reused by all. If a person decides to create a tag, say “vintage furniture” and another person want to use the same tag but would rather have a different description for the tag, or maybe he thinks the tag should have a different name all together. (Language and vocabularies naturally evolve over time). Then what? Who gets to decide? The meaning of a tag all depends on the context of the person. His situation and background will decide what is meaningful to him. No two persons has the exact same context.
For tags to be semantically interoperable it needs to have an addressable identifier (a PSI in Topic Maps jargon) and a particular semiotic meaning attached to the tag. An authority or other entity must make sure the Id and the meaning of the tags are fairly stable. In an open web environment this is not trivial.
One persons definition of a term might be slightly different from the other person because they have different backgrounds. It would not be right to force them to use a definition that does not reflect how they see the world.
What if a tag name is to be changed because language is changing. If it is changed other users might not find what they are looking for. In this case one might present old tag names marked as deprecated etc but it still enforces a new world view onto users. In some cases (maybe on a corporate Intranet) this might be a good thing.
Who will garden and have the last say? What if a user want to delete a semantic tag he has created but this tag have been adopted and used by many other users?

Comparing folksonomies and folktologies are not straight forward. In general we can say that a folksonomy provide more serendipity and discovery and the folktology provide more semantics that can be used to provide more relevant or contextual aware content.

Conclusion
An ontology claims to be a generic, commonly agreed upon speci fication of a conceptualization of a domain [2]. This definition is not compatible with open online social web 2.0 environments where people do not have a shared understanding. This and above mentioned issues suggest that a folktology such as on fuzzzy.com is best suited for personal use or for small coherent teams that have large amount of bookmarks.

To fix these folktology issues a solution could be to have personal topic maps that overlap with other users topics maps. For this to work with some degree of semanticity you could have subject identifiers for each node and/or deduce node similarity from looking at nodes close by.

References

A Hebbian adaptive semantic triplestore

2011-09-04T22:06:00.000-07:00

While reading the book; Making Things Work: Solving Complex Problems in a Complex World by Yaneer Bar-Yam and his chapter on networks and collective memory I got the idea to mix Hebbian theory with RDF. I have also played with similar thoughts before with Topic Maps technology in my research paper Quality, Relevance and Importance in Information Retrieval with Fuzzy Semantic Networks. This time my thoughts were more on adaptive knowledge.

So here's my idea for a Hebbian triplestore
Each triple in the triplestore database has an array of say up to 10 rows. Each array row has a datetime value. When a SPARQL query is processed the current date is added to the array as a new row. The array acts a FIFO queue so the first date is removed if the array is full. Whenever more triples are added to the database it checks if the database is full. If it is full it will delete triples that are least used. Whenever the database is not in use it will perform routine checks to find out what triples are least used. The database could then be regularly fed with new triples and it would over time automatically adapt to the domain where it is being used (queried). So the Hebbian adaptive semantic triplestore is a knowledge store that evolves and becomes more relevant in the environment where it is being used.

Another interesting feature of this triplestore is that it would know what triples are most used. Based on correlation of dates in arrays for triples it can suggest relevant or extended SPARQL queries.

Sparql.us online RDF SPARQL query builder

2011-06-18T03:10:00.000-07:00

Just made a simple site for testing SPARQL queries.
Using this site you don't have to install triple stores, RDF engines or other Semantic Web stack stuff before testing your SPARQL queries.

Check it out at http://sparql.us

The tool has restrictions on RDF graph size. It's only meant for learning SPARQL and testing/debugging SPARQL queries on the fly.

Koios, the free open and collaborative problem solving platform

2011-06-04T02:29:00.000-07:00

This year I have been prototyping a solution for solving complex social challenges online.

Many people don't get what it is I am developing. Most people who do get the idea think I'm mad to even try to do it. Some also think I am naive and wasting my time chasing a fairytale unicorn.
In this blog post I share my vision and some of my thoughts about the system and why the project is worth doing.

I have had the idea for this system for many years. I love watching documentaries. Documentaries often present social problems. Take for example social issues such as the obesity epidemic, ethics and abortion, poverty, racism, human trafficking, corruption, bullying. The list goes on and on and there are small issues on a personal level and there are the really big issues like climate change.
I have always liked to develop new innovative web solutions. Developing this kind of problem solving service seemed like the ultimate challenge. But, I always knew that it would be too big of a challenge to do on my spare time alone.

After searching for months on the web for similar projects, I could not find any sites that had tried to do this. There are, of course, thousands of sites that target various social problems. Some communities also have a set of web2.0 tools like a forum and a blog or wiki. On these sites you get to discuss the problem but there are no built in support for solving the problems in a systematic and analytic way. There is however some portals for social issues. www.worldchanging.com is a good example of this. Sites like this are a step closer but they mostly let people gather around the problem and share ideas. Again they are not really solving the problem just creating awareness and helping out with social networking, donations etc.

Last year, to my surprise, I got funding by the Norwegian government for doing this project so I set out to design the system. I had already spent a few years doing research on my spare time so I had a good starting point.

After a year of iterative design, prototyping, testing, reviewing and further research I finally have a mock-up that can be viewed online. This pre alpha demo version is put online so I can more easily show it to other remote people and get feedback. It also helps other interested parties to more easily find it so to get even more feedback.

Now to the interesting part. Why on earth do I think it can work and what makes it new?
When wikipedia started out not many people would envision that it would become the de facto reference source website and that it would replace traditional reference books.

My vision is to take the process of social problem solving to the masses. Anyone who wants to help solve a social issue should be able to just go to the website and be able to use proper tools and get help from others to solve the problem.

We recognize you don't really solve complex social issues. You just make the situation better or worse. By making variables/indicators go below an acceptable agreed upon level we can say that a problem is solved.

To achieve this vision I want to provide a clear process to follow so that users can follow an intuitive workflow with predefined steps. Users are not bound to follow the workflow step by step but users should go through all the steps to make sure all sides of the problem have been considered. It is up to the site to scaffold/guide the user through the process and help the user to describe models with feedback loops, do scenario planning, do stakeholder analysis and all the other activities that are required to solve a complex problem. In this way we can really “solve” problems on a mass scale.

There is also a competition/game aspect. This is important to attract users that are not domain experts, analysts or already highly motivated to solve a problem. Everyone gets points and badges as they contribute.
What makes it revolutionary is that it is intended for mass scale collaboration. I want to support expert problem solvers, analysts, domain experts and researchers but most importantly support thousands of common people collaborating in connected problem spaces. In this environment you have a few experts doing the supervision and coordination and hundreds of users carrying out tasks like finding facts, testing hypotheses. This is crowd sourcing and collective intelligence taken to the limits. The system is unique in that it lets people from all over the world collaborate intelligently guided by a sound analytic process. This is truly global Collective Intelligence.

Today many communities use a set of typical web2.0 tools to solve problems. Because there are no single portals or platforms that can support the whole range of features required to solve problems they often use a mix of tools. Here is a list of limitations of using a mix of tools like email, skype, forums, blogs, wikis, QnA sites, Google docs, or more customized platforms like Ning, Drupal, SharePoint or project management like tools such as Basecamp etc:

Data/conversation gets fragmented in different places and is difficult to find later on.
Important insights/data gets lost in offline conversations.
Different tools/portals are de-motivating for new potential contributors as they are presented with all kinds of designs/layouts/user interfaces in different locations. This mess is simply unattractive for new people.
There are no fixed slots to put certain data. Users are not sure how to organize insights, hypotheses, facts etc. using SharePoint or other tools there are no predefined process or structure so this have to be decided by each group.
Transparency is difficult when you don’t know where things are and the different systems have different user accounts.
There are no built in support for verifiability, confidence and reviewing. All this has to be done manually somehow.
There are no way to store all data in a structured way that makes it possible to holistically connect problem spaces to synthesize new knowledge based on the underlying data shared across all problem spaces.
No uniform way to share findings using open data formats.
The underlying data cannot be made accessible as web services/API’s and queried by remote systems to allow for Research2.0/science grid/Open linked data.
Limitations on the ability to find similar problems/solutions.
Limitations on the ability to find potential collaborators.
Does not make it easier for people who would like to collaborate on several problems with different groups.

So to summarize; I believe only experts or highly motivated coherent groups can use traditional tools effectively. Traditional tools also have big limitations when it comes to potential future benefits.

I hope I have been able to show the potential gains of such a system and that it is both important and very relevant to try to develop. Even if the project fails it will bring lots of new knowledge to the fields of web science, usability, information visualization, sensemaking, computer-supported collaborative argumentation, virtual learning communities, knowledge management and other fields related to web and system development.

You are welcome to try out Koios at http://koios.org

Do not hesitate to send me unfiltered feedback. Anything from conceptual, design, to feature requests are highly appreciated.
The current koios version (Alpha preview as of June 2011) is not a working tool but should give a good indication of what I am trying to develop. The site is continuously updated.

A few words to help Google out. This website is about Collaborative Web Centric problem solving, Soft Systems Modelling, crowd Sourcing, collective Intelligence, Information infrastructure and complex societal issues resolution, solving wicked problems, tackling social challenges, changing the world, making the world a better place.

Checklist before launching a website

2011-05-25T11:52:00.000-07:00

If you are developing a small website for a hobby project or similar you might not have test scripts or a test plan. Here's a simple check list you can use for testing your site.

Design / Sign off

All relevant persons/stakeholders have been given the chance to comment on the design.
Relevant persons understand the concept/point of the site.
Someone other than you can use the site with ease.
Design has been tested on old projector with poor contrast. (There are tools to do this also)

Features

All pages have valid title and meta tags.
Other typical SEO guidelines.
404 page.
Internal Server Error page.
Content can be printed? (Print style sheet).
Meta tags.
Fav icon.
Added analytics. Website visitor tracking statistics. E.g. Google analytics.
Site map.
Form validation.
Share buttons: Twitter, facebook, LinkedIn etc.
Contact form works.

Performance

Yslow.
Stess test.
Pages checked for page size. Page footprint of more than 1Mb is bad.
Caching evaluate. (Page output caching, data caching etc, web server caching, E-tags etc)
Log been checked for errors.
Home page downloads within 10 seconds or less.

Stability / accessibility

Tested in all relevant browsers such as IE, Chrome, Firefox, Safari, Opera on different OS and with different versions?
Site tested without JS enabled in the browser.
Tested with various screen resolutions both small and large.
Site tested with W3C CSS validator.
Site tested with different screen resolutions and on iPad.
Site tested on mobile.
Site tested using accessibility toolbar etc.
Site tested with full JS error notification turned on.
Do you know your monthly bandwidth limit? Make sure your site resources are not to big and visited by to many people.
All assemblies checked for memory leaks.

Security

Backup.
Robots file.
System been evaluated against security threats.
OWASP guidelines evaluated.
Need to set up e-mail or SMS alerts? (There are monitoring services you can buy)

Content

Content placed consistently.
Tense/Style of writing consistent.
Dead links checker.
No empty pages.
No Lorem Ipsum pages.
Pages spellchecked.
Content formatting consistent.
Alt text on images.
You have contact info, privacy, feedback, terms and copyright info.
Site tested with realistic content.
Site checked for missing resources (404 on images , js, etc.).

Content publishing related editor support

Nested lists.
Links.
Images in text.
Other text formatting.

Why do software development projects for large customers take unproportionally longer time than for smaller customers

2011-03-09T20:50:00.000-08:00

Occasionally, I meet other developers who tell me that they can't understand how we could have spent the amount of money and time that we did on a project. Often they will even say things like; "My small company could have developed this in half the time". So, why is it that solutions developed for large organizations takes so much more time to develop than a similar type of solution for a small company?

Here's what I think; Time and resources are NOT a linear function of the number of features to be developed. And the reason for this? In one word I like to blame it on complexity.

Lets take a look at the definition of complexity: In most dictionaries you'll find that complexity tends to be used to characterize something being intricate and compounded. Ok, so to have an intricate and compounded project you have to have many things together. A complex project therefore must consist of many things. This seems logical. Large companies have more stuff, right? Stuff? yes more people, more projects, more sub systems, more interfacing systems, more history, more data, more legacy code, more policies, more legal requirements, more constraints, more processes, more methodologies, more processes, more guidelines, more internal and external stakeholders and so on. All these things together form a complex project. Changes to one part of the project will often have consequences for other parts. In the following sections I'll try to delve into what complexity consist of, bare with me:

The uncertainty factor

Projects for large customers have more stuff. No one can have a full overview of all the stuff and all the details of all the stuff. Naturally this leads to situations where things are just unknown or uncertain. When the number of things we have to deal with gets over a certain size it gets hard for us manage them. Often more people are brought in to deal with the mess. Also a risk management [1] consultant might be hired to tackle the uncertainty by calculating risks. Computers are great at managing large amount of data. The human brain, on the other hand, tends to overlook, forget, make faulty assumptions, bias etc. This naturally leads to errors and wasted time. To fix the uncertainty factor we will usually try to seek information to make sense of the mess. This search for just-in-time knowledge will be overlapping. Several people will try to dig for the same information over time and the same information is prepared over and over for various purposes with slight adjustments and the information prepared are often not found again for others to reuse. And when the information is found, the reader might even misinterpret the information or apply it to a different situation where the knowledge don't apply. These are typical problems that the field of Knowledge Management has tried to solve for a couple of decades now.

Uncertainty also causes problems later in the project. Humans are very bad a comprehending complex systems so they are bound to under estimate. This under estimation leads to schedules being broken later in the process which again leads to less time to do the job right. So you end up with less quality which again later can cause big problems and lots of wasted time.

The dependency factor

As we have seen complexity is being intricate and compounded. This implies there are relations between the stuff. Stuff are dependent on other stuff. An example of a not so complex project is a one man programming shop doing a small project for a small customer. He would have relatively few relations to other people and between the things he is working on.

We can use graph theory and network complexity as examples in this regard. When the number of nodes increases, the number of possible connections increases at a higher rate. This means that the people involved will spend more and more time on communication and dealing with more and more stuff.

According to triangular polygonal numbers; if you have 3 nodes the maximum number of relations between these nodes is 3. If you have 4 nodes the number of relations can be 6. For 8 nodes there are 28 connections and so on.

In many cases there is not a direct relation and communication needs to take several node hops which obviously take more time. This also increases the possibility of communication error so the project takes even longer time.

Are there really more stuff to deals with on projects for large customers?

Below is a listing of what I have experienced during my work as a freelancer, and as a consultant for everything from small companies to large multinational corporations.

Processes

Large companies usually have higher demands on quality assurance.
Large customers are more likely to have a number of parallel projects where internal resources have to be coordinated between projects. Often you will have to wait for some other project.
Large companies often have rigid processes with decision gates and formal project planning steps that just takes time. Decisions need to be approved on many levels.
You are probably more people and because many are dependent on you, you can not just sit wherever you like and do the project. Less work situation flexibility usually results in less effective work as you have to spend more time on travel instead.
In large organizations there's usually more beurocracy, which means more waiting if you need to order something, if you need access rights etc etc. You might have internal rules preventing you from fixing your computer when it breaks etc.
There are usually more meetings in large organizations as there are more people involved that wants to have their say. A two hour meeting with 10 persons equals 20 hours.
You are likely to have more guidelines, compliance or regulations on your process so you need to spend more time reporting and writing documentation.
You often can't choose the tools, methods, frameworks you like. You have to learn and reuse what is already in use. This makes your process less lean.
Working with large customers you are more likely to also work on several projects for the same customer. When working on several projects at once, a lot of time will be wasted because of all the chaos that comes with doing parallel work.

People

For large customers there's a bigger chance that there are specialists or experts to deal with certain parts of the project. When these persons quit or leave for another customer, project or even new job it takes time to replace them.
There are probably more people involved and there might be some misunderstandings when something is communicated verbally and new people who come on the project don't get the same piece of information.
For large customers there is a bigger chance that some of the new people on the project is or will get less motivated and slows down progress or create tension in the group. The new guys have less knowledge about the project and often feel they have less to contribute. Often the developers who have been on the team from the start will get an "I'll just fix it my self" mentality where he or she just solves the problem him self because he knows that it will take considerably longer time for the new person who don't know the system just as much. This again leads to the situation where the new person feels left out or less worth.
For large companies there are usually more stakeholders and end users involved. The chance is bigger that relevant stakeholder and/or users do not get to give feedback or input to the development process. This often cause problems later and leads to longer development time and possibly even conflicts.
As a side effect of the project taking more time, you probably need more developers to deliver the project on time. Everyone can't be super programmers. The project often has to "pick from the bottom of the barrel" and so there will be novice developers that are less effective.
More people also mean there's greater chance that someone will become a problem. This can be because they don't fit into the group, because they simply didn't want to be on the project, because they had other expectations to what they were going to work on etc.
More people needed often results in more staffing problems. Some people might feel that they are just thrown on the project.

Politics

For larger customers there might be collaboration problems or unhealthy competition between departments, suppliers, contractors, partners etc.
The larger the organization, the more layers of management. The more layers of management, the harder it is for bad news to get to the top. No one wants to be the one to communicate the bad news.
When there's many departments involved you will often get into a situation where you have to find out how to please everyone. You don't want to have opposition to you project.
You might even meet managers that have their own personal agenda, and you will have to deal with what comes along with that.
It's more likely that you are required by IT politics to use large frameworks or platforms that make the development more difficult than it has to.
Large organizations usually have more requirements on branding.

Technical

For larger customers there's usually more infrastructure, more servers etc. This means more time goes into moving files here and there for testing etc.
The build and test environments are usually larger because of higher degrees of QA and Application Lifecycle Management requirements. Managing access to these environments takes more time.
There are usually more technologies involved and you can't know them all so you have to spend your work day google how to do this and that.
You are more likely to integrate your solution with some legacy system or third party product. This requires the team to learn new API's.
There's probably more systems and infrastructure involved in your work process so when one sub system is down the entire team might be impeded.
With more server infrastructure comes more security management.

Communication

For large customers there are usually more people involved so there's more waiting on getting in touch with other people. Maybe you need a confirmation on some issue so you wait for others to get around or wait for them to pick up the phone. Of course you don’t sit waiting doing nothing until you get the answer but this takes your concentration away and it makes your day more chaotic as you have a bunch of thing to do and follow up on.
There's going to be more coordination needed. Coordination between core team developers, other developers, between developers and designers, testers, managers, product owners etc.
When there are more people, you are more often distracted. Maybe someone needs you to do something. This again takes your attention away and once attention is lost you go for coffee, start chatting or read the online news paper etc.
You probably have to work with external agencies or contractors. Either because they want to have the best people or because they are not able to find the required skills within either your or their own organization. So they outsource parts of the project. Then, you are left having to communicate with the other remote contractor. Teams not co-located have more communication and trust problems [2] and communication is of course much slower.

Code

Large organizations usually have more history and therefore more legacy code. Writing code is very fast when there is little code. As the code base gets bigger things slow down. Rewriting and refactoring takes more time. More time is needed to architect and engineer the code.
It's more likely that code is shared between projects. When requirements change it is usually more difficult to make changes if the code is reused on several projects. If these projects are not co-located it can become a nightmare to coordinate changes on shared code.
Because the organization has more history it will probably have more code. As the code base gets larger, compilation takes longer time.
More code makes it more difficult for developers to get an overview, so new features or sub systems might not get implemented in the best way. Often this code will need an overhaul later when it has escalated into some kind of a problem.
When the code base is large and there are many developers there is a tendency for developers to be afraid to change the code of other developers. Especially if there are no unit tests. Developers fear it might have consequences they don't know of or they are afraid other developers don't agree on the code change etc. They might even feel a pressure to just get the job done so they don't do any refactoring and just adds more code making it less readable.
When there are more developers involved not everyone will have the same feeling of responsibility. Some developers will avoid doing to big changes to the code because they don't want to be the one messing up the system if something doesn't work as planned.
As the system gets larger and time goes by there's more places and chances to put in even more code. You get the requirement creep [3], where stakeholders push for more features.
Large organizations usually have more users so more work need to be done at scaling the system either through code or by adding more server infrastructure.
Because there are more users you have to develop fall back solutions and cater for more type of users.
With more users you need to do more stress testing and performance tuning.

Time leads to more time

As the project drags out there will be more opportunities to make changes. Suddenly someone decides that the system should be based on a completely different platform because of some top management strategic decision.
If the project goes on for a long time some of the project participants will probably get tired or bored. This naturally slows down the pace.

Conclusion
Projects for large customers take more time simply because the project will be more complex.

References
[1] Managing Project Uncertainty, David Cleden.
[2] Geographically-Colocated Subgroups in Globally Dispersed Teams: A Test of the Faultline Hypothesis.
[3] The Pragmatic Programmer: From Journeyman to Master.

Why not use Flash/Silverlight on public websites

2011-02-28T21:37:00.000-08:00

This blog post is about why not to use Flash/Silverlight on your website. You can find hundreds of people who have already written on this topic but I still meet people who think it is a good idea. There are a few cases where it makes sense to use these technologies but in most cases you, your customer(s) and your end users are better off, not using Flash or Silverlight.

Why not use Flash/Silverlight

A page with Flash/Silverlight will load slower. Especially if the page also has a lot of javascript on it.
Inconsistencies with web standards. A flash movie gives a different feeling than ordinary web content:
- Print does not work the same.
- Copying content might not work.
- Navigation, including back button don't work as you are used to.
- Enlarging text doesn't always work the same.
- Input fields, buttons etc look different and can confuse end users.
- Bookmarking or linking to your site is more difficult since you're probably not able to link to the exact section within the Flash/Silverlight movie.
- Searching text on the page (ctrl+F) doesn't work.
- Less usable for the disabled.
Internationalization and localization is more complicated.
SEO is more difficult and in most cases will be less good than a plain HTML solution.
It is more difficult to maintain the code as the resulting .swf/.xap file is closed and require special skills other than regular web development skills.
Flash design is usually hard coded. You can not easily reapply a new style with such as with CSS. Silverlight makes this easier with XAML but XAML again requires knowledge and special tools.
It is usually more difficult to update Flash/Silverlight content. There's usually no wysiwyg support.
Q.A. gets more difficult (automated testing and code review is more difficult because it requires other skills and tools).
Analytics using tools such as Google Analytics is more difficult and usually less precise.
In most cases you don't have control over if the end user has Flash/Silverlight installed. Even if the website is an Intranet and everyone is supposed to have the same setup there's usually someone with a non-standard client that will complain.
Flash will not run at all platforms. (e.g. Flash on iOS)
Flash/Silverlight is more difficult when it comes to progressive rendering. Normal web pages load and render parts of the page before the entire page with all the page resources are sent over the wire. With Flash/SilverLight you might need to create pre-loaders or put extra work into the implementation to make it load smoothly.
You might have to create fall back solutions for those who have not installed the Flash/Silverlight plugin.
Flash/Silverlight usually requires more bandwidth.
Some developers and designers will get tempted to put fancy features or overly fancy designs into the Flash/Silverlight and they often do. These features can often become a source of irritation as they waste the time of the user or make the user think. Often they will create fancy animations on their top notch Mac computers and forget to test it on slower computers. Heavy use of animations requires more CPU usage which again is less environmentally friendly.
As a developer you can get problems with stacking order if you have layers that will popup over the Flash movie. The Flash will always position it self on top, regardless of z-index. might work. Often you don't get into this problem until long after the Flash is developed and you need to update the Flash to fix the problem.

Now that the web is getting more ubiquitous with mobiles and more and more client types appear (like tablet computers) it gets more important to adhere to standards and reuse best practices.

References:

Jakob Nielsen's Alertbox, October 29, 2000: Flash: 99% bad

Green living for busy computer geeks

2010-12-08T07:20:00.000-08:00

Check out my list of tips for green living especially suited for those of you who spend most of your time in front of a computer. The list will not require you to make big changes in your life or invest in new expensive stuff.

Use both sides of paper when taking notes or when printing from your computer. Most printers have a double sided print option.
Buy clothes made from natural fabrics like cotton, wool or silk.
Turn down the brightness of your monitor.
Don't buy high tech fancy clothes made for sub freezing temperatures unless you are going to the North Pole. These clothes require both non eco materials and none eco production methods.
Read your PDF-document on the computer, tablet/iPad instead of printing all 200 pages.
Have a personal mug at work. Don't use dispensable paper or plastic cups.
Don't throw a way clothes and other stuff. Give them away.
When buying stuff, try to choose the one that come in the least amount of plastic packaging.
Switch your computer all of when you go away for days.
Line dry your clothes rather than using an drying machine.
Unplug the cell phone battery charger when you're not charging.
Turn of the lights in the rooms you are not in.
Put the computer in standby when you leave.
If you use a screen saver use a blank all black background.
Stop buying stuff you don't really need. To help the economy buy services rather than physical products.
Don't leave the tap water running while brushing your teeth.
Before you decide to buy a new computer make sure you have tuned and cleaned up your computer making it as fast as it can be.
Skip rinsing your plates before putting them in the dishwasher.
Turn off all the light when you leave the apartment or your work.
You don't need to switch to a new cell phone every 3 months?
Have power saving enabled on your computer.
Lower your winter heating and put on some more clothes instead.
Get more energy efficient light bulbs.
Use less water when taking a shower.
Walk or take the bicycle rather than the car if you can.
Buy used, second hand stuff.
Bulk purchases.
Travel less. Use the telephone or online meeting and conference services.
Reduce junk postal mail.
Buy locally grown food that has not been shipped from the other side of the world.
Don't throw away old cell phones and other electronics in the trash can.
Share stuff instead of buying the same stuff as you friends or neighbour.
Buy fewer products for Christmas and birthdays; instead buy services/adventures/experiences.
cook food with a lid on the casserole.
Switch from black on white text to white on black text on your monitor if you can change the desktop, your application colours. This can save up to 25% electricity usage.
Put on some more clothes at home instead of heating the apartment.
Choose online services that have green data centres.
For your next computer, consider a laptop instead of desktop. A laptop uses much less electricity.
Buy recycled paper for anything from toilet paper to printer paper.
Do you really need a big house and a big car?
Write notes on your phone, tablet etc. we're consuming billions of disposable pens each year.
Don't pre-heat the oven. Turn the oven on after you put in the dish.
Before buying a new computer, consider upgrading the hardware.
Make less noise. With less noise others around you can think more clearly. More clear thinking means more effective work. More effective works means less energy needed to do the same work.
Download e-books rather than ordering books from Amazon that has to be shipped.
Make better plans before going somewhere so there is less detours.
Use matches instead of lighters.
Have fewer coins. Having a lot of coins prevents them from being circulated and this result in more coins having to be minted. Producing coins takes a lot of energy.
Buy less bottle water/soda, or at least buy a local brand.

Benefits of using OAuth as your login provider

2010-09-28T20:45:00.000-07:00

OAuth has become a popular mechanism to facilitate sharing of data between applications. Another way to take advantage of OAuth is to use it as your login provider. In this post I'll discuss the pros and cons of using it as the login provider for your web application.

To use OAuth for login all you have to do is to use some OAuth code library and create a consumer application that will connect to an existing OAuth provider such as twitter, linkedIn, Facebook etc. Store the OAuth token in your database, use cookies to save the user login state in the browser and most of the work is done.

The alternative to using oAuth as the login provider can be to implement OpenId or some proprietary login mechanism. Here are some advantages and disadvantages compared to develop or buy a component for login and profile management.

From the end user's point of view:

Advantages:

Don't have to create another profile on the net.
Fewer passwords to remember.
Do not have to submit a password to your application if he / she does not completely trust us.
User can prevent access to the application from the OAuth Provider.
Allows for exciting extra functionality and synergies when taking advantage of the social graph and other data and features made available by the OAuth provider. This may provide much added value.

Disadvantages:

User can not tailor the profile for your application (would require additional development).
Can be a bit confusing for the user having to create an account with OAuth providers if he / she does not have an account there already.

From the organization's point of view:

Advantages:

Save time on developing authentication, display of user profile and social interaction as friends lists, status updates, profile, photos, etc.
Do not need support for password renewal, forgotten password, authentication of users, and support to let users remove themselves from the service, etc.
Lower risk and fewer bugs in connection to authentication when using a ready-made proven API.
Low-risk for ID theft, etc. The service already has good support to prevent this. Authentication takes place at provider, the OAuth tokens is encrypted and not in our application.
If the OAuth standard is extended with support for info cards or other functionality in the future, it would be supported in your application automatically.
Easier to manage / maintain / configure than forexample extranet login models with mixed authentication (form based saml etc)
Less data to store on your servers.

Disadvantages:

No connections to existing user accounts e.g. in Active Directory if needed.
If the remote service is down you have no login (unless the users authentication cookie is valid).
Requires some logic if you will allow the user to log in with multiple OAuth providers.
Maybe you (or your company or brand) do not want to associate your self with certain OAuth providers.
Maybe the end-user do not want to share his OAuth provider data (LinkedIn / twitter / facebook / Yammer, etc) with your application.

More information

Technical description of OAuth

http://oauth.net/
http://dev.twitter.com/pages/auth

Sample application that shows how login with OAuth in practice

http://www.twitarcade.com/

SharePoint or Asp.Net MVC for building a new and simple web site

2010-08-22T14:12:00.000-07:00

I was recently on a project where we were faced with the choice of what platform to base a scoped down prototype web site on. Based on the team competencies and the fact that we had already developed several SharePoint sites for the customer we were basically down to two choices, SharePoint or developing an Asp.Net MVC site from scratch. I consider Asp.Net MVC superior to Asp.Net so a regular Asp.Net WebForms project was out of the question. SharePoint is mainly a collaboration solution best suited for Intranets and portals. Even so, many companies use the platform for their company website because of the growing amount of SharePoint developers and other benefits of the platform such as extensibility.

The purpose of this blog post is to share knowledge for others who might be faced with the same choice of using SharePoint or Asp.Net MVC for a very small web site with few requirements.

About the team
We were 4 developers, a scrum master, a part time interaction designer and a part time graphical designer on the team. None of the team members were SharePoint or Asp.Net MVC specialists. None had any experience worth mentioning on SharePoint 2010. The average experience in years with programming of the team members were less than 10 years. Most had worked with SharePoint for some years but none of the team members had in-depth expert knowledge on all sides of SharePoint development.

A short note about the scope of the project
Time span of the project was about 2 months including planning, design, development and testing. Without going to much into details, here are the features of the site:

Integration of some external social media widgets.
Very basic publishing of simple content.
Branding and fancy visual design.
Some commenting, mailing list sign up and feedback features.

The site was a pilot project so we did not have strict requirements on quality assurance etc.

Platform choice
The decision was to use SharePoint as most of the developers were more familiar with SharePoint over Asp.Net MVC. The decision was inline with the company IT-strategy and most of the developers were keen on trying out the new SharePoint 2010.

Experiences
At the time of the project it just happened that I was starting a new hobby project using Asp.Net MVC on my spare time so I had the opportunity to compare SharePoint and Asp.Net MVC development side by side. The two projects were similar in many ways. In the comparison below I only discuss issues that are relevant to both platforms. Many of these problems have also been encountered in previous projects so they might be said to be typical problems you might expect to run into.

Problems we encountered while starting a new SharePoint 2010 project

We had to switch to a new development environment and install Windows 7 since SP2010 requires Windows 7. Installing Windows 7, setting up SharePoint and configuring all your nice to have software typically takes two days.
We later got licensing problems for the OS and Visual Studio. This is not a big problem but all of these small non-code related things combined make a lot of noise.
SP Feature mess: We had trouble setting up page layouts. It kept getting ghosted automatically. We found out that some meta tags attributes on the page layout caused it to automatically getting ghosted during deployment. This resulted in several wasted dev hours.
SharePoint has a lot of built in JS such as the init.js file. One of the social media widgets we wanted to use would not run along side the SharePoint JS on IE7. Debugging minified widget code and thousands of SharePoint lines was not anything we wanted to do so we had to write some custom code. Again several hours wasted on debugging and custom development.
SharePoint 2010 still renders bad html and lots of other page elements, js and other markup that is not needed for a Internet facing site. The site even tried loading Silverlight when used as an Internet facing site. Chrome crashed because of the Silverlight plugin. This was fixed by adding a SPSecurityTrimmedControl around the ribbonrow.
We added several web parts to the web part gallery and we encountered problems that made the add-webpart ribbon unusable since the add button got hidden beneath the ribbon bottom edge on some browsers.
We encountered problems with the NoteField. We wanted more than 255 characters for the text. When deciding to use the html rich text field instead we had to change the content type and this is not a walk in the park if you have already created some test pages you would like to keep.
Each time a new content type field is added there's often problems for the other developers. The other developers need to upgrade their solution.
VS caching bug. While doing deploy from SharePoint within VS we sometimes got this error: deploying Activating feature 'OurFeatureName' ...
Error occurred in deployment step 'Activate Features': The field with Id {e3575ee4-1b6e-43fc-b243-77f727f33e16} defined in feature {a36d44c1-b523-4235-950a-59ed763d2588} was found in the current site collection or in a subsite. This error had to do with some state being cached by Visual Studio 2010. In this case we had to restart VS.
SharePoint includes a bunch of js and css files. To optimize page rendering on the client we cleaned up and customized the minimal.master masterpage using editmode controls etc.
We created XSL for the Content by Query (CBQ) web parts and had to figure out why the CBQ didn't list our pages. Debugging this was time consuming.
We also had problems with web parts not being deployed because of inconsistencies between .dwp and .webpart files.
Another problem related to web parts not being deployed had to do with mismatch in the SP2010 .spdef files that might be caused by code merging problems.
A typical error many do on SharePoint. XSL files packaged incorrectly and end user in anonymous mode don't have access to the xsl.-files and the page breaks.
Then we had problem with the server. The system was a small pilot project so the server we got was an old outdated server. SharePoint 2010 simply did not run on the old server without extensive reconfiguration.
We copied a SharePoint feature from another project. We forgot to change a GUID and things just did not work. Debugging, again not trivial.
SharePoint 2010 does not support IE6 so we had to add a custom warning message for IE6 users. The lack of IE6-support has in retrospective been a problem because it also resulted in down prioritizing IE6 for the front end anonymous view of the site and we have got several complaints from end users who did not understand why the site looked bad on their browser. Also showing of the site is also problematic because the company still used IE6 as the default browser so this lead to embarrassment as we didn't always have control of the installed browser at the location where the web site was presented.
The page layout or list we tried to add at some point simply would not be added to SharePoint. No easy way to figure out what was wrong with the xml. The schema.xml for a list is not particularly pretty and inviting to work with. There are of course tools to help you with this but the work all becomes very complex and awkward unless this is the only thing you do for a living.
Then, one of us had problems updating the web.config. Updates did not affect the site on the developer’s machine. After an hour wasted work we found out that SharePoint 2010 use a different web.config file. The new web.config file is under the layouts folder in the 14 hive.
Next problem; SharePoint redirected mobile devices to a mobile site by default so we had to customize this also.
Then we got problems on the Publishing Images doc lib. Some unexpected error occurred when navigating to it. Still don't remember what was done to fix it. Simply too many problems and solutions to keep track of.
We had a lot of static html code that were inserted in content editor web parts. Part of this static html was some javascripts used for client side paging and other GUI effects. In SharePoint this had the affect that the JavaScript altered the content of the web part while the user was editing the web part. When saving the web part the user would be saving the jQuery altered version of the html. To fix this we had to link to a separate file instead and update this one rather than editing the web part content itself. With Asp.Net MVC this would be a simple text field with html. The wysiwyg of SharePoint just made things more time consuming here.
As with other typical SharePoint projects we had to set up content types, set default page layouts for a page library, what page layouts are available for a site etc. All these things take time to configure in the site definition. This is typically tedious and boring work.

Problems I encountered when starting a new Asp.Net MVC project

First problem: You code everything from scratch this can be a frightening enough it self. Using some sample code as a starting point might help a lot.
You get 'Object reference not set to an instance of an object' errors and you can't figure out what is wrong. Asp.Net MVC uses a lot of convention over configuration so there's a lot happening under the hood. Debugging this might be difficult because there a lot of abstractions and inner workings that you might not want to dive into.
More convention over configuration problems. If you miss a convention or have a convention implemented wrong you will often not get a compile time error message and it’s difficult to locate the error.
Figuring out how the routing works on XP with IIS5.1 is difficult. Not very intuitive if you are still on XP.

Conclusion
Asp.Net MVC is simply a web application framework. SharePoint is a software platform for collaboration and web publishing and has a rich set of capabilities out of the box. So basically what you are doing in SharePoint development is to customize the existing features, tweaking features, overriding stuff, extending features or even stripping them way. On Asp.Net MVC, on the other hand, you develop the site basically from scratch. For most developers it will be more interesting and exciting to develop something from scratch that they have full control over, not having to abide by the complexities of SharePoint and having to sit and configure XML files all day long.

If you know that you are going to use some built in feature of SharePoint you should consider SharePoint, but if you are not going to use any of the strong selling points of SharePoint such as document management, workflow, compliance, publishing, document search etc you should definitely build it your self.

Benefits of Asp.Net MVC over SharePoint for small and simple web sites

Asp.Net MVC uses convention over configuration. In most cases this makes development very fast as the conventions reduces the work to be done as well as providing a model (MVC) for architecting and for how to arrange code in a consistent and clean manner that further reduce complexity and speeds up development. Faster development means cheaper development and enables early testing which is important for a prototype.
You feel more productive as everything you do has a purpose and is directly affecting the solution. Every line of code has direct impact on the actual feature you are working on. SharePoint xml configuration is often used to configure some feature that you don't even need in the solution but you have to configure it anyway just because it's there and available be default to users. A lot of tasks on SharePoint simply seem like a waste of time since you are not working on the required features but instead on removing the features that are not required.
Most developers find it more fun to write code than writing XML. When writing code you get to be creative and show how good or smart you are. With XML configurations you have to stick to the schema, no room for creativity.
By using Asp.Net MVC we would have Azure as an alternative or backup hosting model.
SharePoint development requires a lot of competency. To customize a SharePoint site you need to know web development as well as XSL, the SharePoint object model, feature stapling, site definitions, WSP packaging, SharePoint security model and a bunch of other things. Since very few developers are experts in all of these areas it takes a lot of coordination and teamwork. More dependencies means more development time.
A part from the social aspects and skills required there are the complexities of the technical solution it self. In general, SharePoint introduces a lot of complexity that makes development less fast. Lots of different type of code in different places, lots of classes, lots of different configurations on different levels, various tools such as STSADM, lots of scripting commands to remember, many folder locations. Lots of concepts and features. If you want to look into the database to try figure out what SharePoint is doing you'll also meet a wall of complexity there also. More complexity equals developer headache.

Benefits of SharePoint over Asp.Net MVC for small and simple web sites

SharePoint has a lot of built in features, some of these might be exactly what you need. For example, Web parts that can be moved between web part zones was one such feature we made use of.
As with all IT-projects you must take into consideration future requirements. SharePoint's comprehensive platform might be what you need for your site in the future when your site grows larger and features such as versioning, document management etc may be required.

The big take-away from this blog post

Complexity = more development time.

Requirements reminder

2010-06-23T02:07:00.001-07:00

In smaller agile teams, short projects often start without much consideration to requirements other than the functional requirements at hand or some design sketches. This is a dangerous situation but also a typical case when there are no solution architects involved. Once the team have chosen a technical path it might be very expensive to redo the system. The longer the system has been used by end users the harder it is to replace it.

In this blog post I list a bunch of possible requirements, of different type, just to remind myself not to forget to check if there are requirements related to these topics when working on smaller projects.

Accessibility, WAI
Accuracy
Adaptability
Audit and control
Availability (Service level agreement, downtime)
Backup
Branding
Browser support, clients
Capacity
Certification
Compatibility
Compliance
Completeness
Configuration management
Configure-ability (can it be reconfigured)
Correctness (no bugs)
Cultural and political
Currency (data is up to date)
Data structures
Dependency on other parties
Dependability
Deployment (deployment procedures, life cycle turn around and velocity)
Design (and layout)
Disaster recovery
Documentation
Editorial, publishing
Efficiency (resource consumption IT, energy, human, performance vs effort)
Emotional factors (fun, absorbing)
Environmental tolerance
Escrow
Error tolerance (wrong user input)
Ethical considerations
Extensibility (adding features, upgrades)
Failure management
Failure tolerance (defect in system execution)
Install-ability (easy to install?)
Internationalization (e.g., different countries, languages)
Interoperability (properly interfaced with and working together with something else)
Language (style of writing, target group)
Learnability (easy to learn)
Legal (licensing issues, patents-infringement-avoidability)
Logging
Logistics
Maintainability
Memorability (easy to remember how to do thing in the system)
Mobility (Can it be used on the move)
Modifiability
Network topology
Open source
Operational , manual procedures
Operability (possible to perform tasks in accordance with operations manual?)
Packaging
Performance / response time (performance engineering)
Personalization (personal user experience)
Performance (Jitter, Response time, Latency)
Personell (and resources)
Platform compatibility
Portability
Precision of data
Price
Privacy
Provisions
Protocols
Quality (faults delivered or fixed)
Reuse-ability
Recovery (e.g. mean time to recovery - MTTR)
Reliability (e.g. mean time between failures - MTBF)
Reporting
Resilience
Resource constraints (processor speed, memory, disk space, network bandwidth)
Response time
Robustness
Reliability (operates without failure under specified use)
Robustness (also functions under abnormal conditions)
Safety (preventing / dealing with accidental harm)
Scalability (horizontal, vertical)
Schedule-ability
Scope
Security (tolerance against malicious harm)
Source code
Stability
Subset-ability (easy to make different functionality subsets)
Support (end user support)
Survivability (essential services continue in spite of accidental or malicious harm)
Style
Templates
Testability
Throughput
Traceability
Training
Transportability (possible to move system)
Unambiguity
Usability (ease of use)
Usefulness (useful to use)
User experience (system perceived as fun, satisfying, enjoyable, entertaining, helpful, motivating, aesthetically pleasing, emotionally fulfilling, rewarding and supportive of creativity etc)
Verifiability
Variability (degree to which different variants exist)
Withdraw-ability (can a specific version be withdrawn and replaced with a previous, version)
Workflow
Workload (low perceived workload by user)

References

http://en.wikipedia.org/wiki/Non-functional_requirement

http://www.idi.ntnu.no/grupper/su/fordypningsprosjekt-2005/eide-fordyp05.pdf

Scrum taken to the extreme, doing design of features in the same sprint as they are to be developed

2010-02-06T23:54:00.000-08:00

On one of the projects I participated on last year I had the opportunity to work on a large project (more than 15 participants, several locations and several interfacing systems). It was decided to do design of products/features in the same sprint/iteration as it was to be developed. This is not the ideal situation as several others have experienced before me [1],[2],[3],[4],[5] and as described by best practices [6],[7],[8],[9],[10].

The background for this decision was that the clients company profile rebranding project was not finished so the designers did not have what was needed to do the complete design job beforehand. Therefore the design team could not really start their job before the developers. Because of this it was decided to have sort of sub iterations within each sprint. For each sprint a set of products were planned and then designers started designing products as on an assembly line. As each product was designed it was delivered to the developers at once to be implemented.

About the project and team composition

Both developers and designers were quite flexible and competent. The development team had some experience with design and the design team had technical knowledge and could even do frontend programming. The project aim was to develop a new Intranet on a new platform. Re-branding was a key goal. It must be mentioned that the developers outnumbered the designers considerably. There was a fixed deadline and the total project length was about 8 months.

This is what happened

The project started out in good spirit. Everyone was confident and happy but after a while things started to go less smooth. As with all projects with a certain level of complexity there is not just one single thing slowing down production. I'll try to concentrate on the issue of in-sprint design and development (design and development of a product in one sprint).

Planning

Since design was not complete at the start of each sprint, developers could not estimate the job since they did not know what exactly they were to develop. This had the affect that the project lead did not have the slightest idea how much would be completed after each sprint.

Doing design in iteration i+0 (the same sprint as it is to be developed) also made the progress flow extremely unpredictable. The time required for design is often hard to estimate and designers often meet unexpected problems in their design as we developers do in implementation. This means without any lag between design and development the risk of encountering blocking impediments are doubled.

Specification and Handover

As designers were to handover several smaller designs it was easy for them to to make readjustments to the design specs, this naturally resulted in frustration on the developers who had already started developing the design.

Because there were no big formal handover and the designers were already overworked it was easy for designers to get in to the trap of handing over items that did not follow the required documentation guidelines. Items got sent by e-mail and people forgot to upload it on the correct place and adding the correct metadata etc. This again led to difficulties in tracking changes and finding the specs.

Since the designers worked more or less in parallel with development there was always some spec lacking. Developers constantly needed to ask the designers for specs and after some time all the designer was doing was feeding developers with design. This situation where the designer was pressured also had the effect that designers did not always have time to do good overall design making the entire design harmonize. Another side affect was that designers had less time to prepare and get a strong acceptance on the next design. On a few occasions design did not go through a quality assurance loop with the customer resulting in bad features that had to be redesigned later.

As each user story was a minor handover stakeholders could not really be involved in the acceptance of the design as it would then slow down the project considerable if several parties were to be included for each of these small sub deliveries. Instead the designers controlled the process and sought out who to contact to get the necessary feedback. This had the effect that some persons did not get the chance to see the design and give feedback before it was delivered to the developers. In some cases this proved disastrous as some important stakeholders had important information that did reached neither the design nor the development team. Some participants felt overlooked as they did not get the chance to participate in the design work or not even being able to comment on the design proposals.

Communication and noise

Often specs was not very detailed and in many cases completely absent. Less documentation is part of the agile manifesto but since we had developers at several locations this became a problem. Less documentation means more face to face communication and communication over messenger, phone, e-mail but in large projects participants tend to have a lot of meetings and thus not being available for communication about specs. Developers often found themselves sitting and waiting for people to come back from meetings when having questions about some design that may have had an incomplete spec. In other cases they just started on some other product which in turn led to a chaotic work environment when the developer ended up with working on 4 products at the same time. Not being able to complete a product adds noise in the project as the project needs to bring this product up in each scrum meeting and you have to tell once again why the product is not completed. Several products also seemed half finished since there were no complete specs and relevant persons were not available at the same time to clear up design issues.

The incomplete or lacking specs had the effect that everyone on the project needed to communicate a lot more. Time was wasted as messages had to be repeated.

As much of the work surrounding specs or the specs themselves was replaced by face to face communication people outside the core group had to ask for information, this resulted in multiple and overlapping communications.

The IA/GD/UX team will always have some questions for the developers but since both design and development is done continuously and in parallel there will be a continuous flow of inquiries for the developers. This is not optimal for a typical developer. Developers will always be prompt and in a mode to delve in to planning, problem solving and answering questions in the planning phase of the sprint but as they get down into the nitty gritty programming many developers will have their head wrapped around difficult problems and will not spend much time on starting to analyse some small design issue put forth by a designer. Many programmers are also competitive and non-programming related distractions in the middle of the day are not always what the developer is most keen on. This has the effect that some design problems are not well thought through.

Experienced developers want to know what features should be developed next so they can chunk together related work including fixing bugs on related stuff. But as they now live in this day by day design hand over they did not know what features would come the next day and ended up scratching their head trying to figure out this himself or trying to find this out from the IA or designer. The designer works on other things so there is noise. Not being able to chunk related work led to less efficiency.

Workflow, predictability and work conditions
In the early sprints developers tended to have little work in the start of the sprints as they were waiting for design. In the end of the sprint developers were often unmotivated from taking on new design since it was something completely new that they might not have seen before. Completely new design are more intimidating than a design that has been slowly setting in the developers head during the sprint and where the developer has gotten the time to figure out how to best solve the task.
As designers were fully entangled in pushing design they had little time to look over the end result and developers had to go ask designers to test if the implementation of the design was ok.
Developers did not know when they would get a design and what it would look like. This led to unpredictability and unsystematic work making the flow less streamlined and more ad-hoc.
It all became very unstructured. Some people, often the creative ones thrive in chaotic environments but for many developers often consisting of systematic engineers it became very messy a nightmare.
For new developers, especially the juniors, coming on to the project late it must have been a nightmare not having the knowledge that had been communicated only by mouth and feeling that they add even more noise in the project. For some of the seniors on the project it did result in a lot of extra guidance and support on these new participants. Incomplete specs were even more problematic for new project participants because they did not have the big picture. So again more noise.

Technical forward thinking, architecting and code quality

Since we did not know what design would come later we could not estimate the time, therefore the tech lead did not know if we had time available to do refactoring and other things that would benefit the project in the long run. All he could do was either concentrate on getting things up and running or taking the chance on doing refactoring and risking not being able to complete later features.

In general there was a lot of unsystematic work and unpredictability making it harder for the tech lead to plan in advance and see where to do refactoring etc. Unit testing was out as everything was changing day by day. Setting up unit tests takes time and is a waste of time if the requirements change and the code is likely to be totally rewritten the next week.

As specs seemed incomplete or being suddenly changed developers often became demotivated knowing that what they developed would probably have to rewritten. This resulted in lower productivity, poor code quality and features that were only partially completed.

Since designers were overworked they where not always available to do these kind of in front of the screen 2 minute designs that often are needed for some small thing that was overlooked in the design phase. In this case the developer just design the thing him self.

Testing

Some developers felt stressed out as some design was delivered just few days before the end of the sprint. Most developers want to deliver as much as possible so this results in a situation where the developer wants to produce new features but gets overworked and forgets the importance of testing or preparing for testing of what has already been implemented.

Because of poor specs, test scripts were poor and this resulted in test results that were not realistic and testers did not know what had been completely developed. Some products were set as ready for test before they should have.

Because specs was hard to find specs were in some cases even skipped. Some features were even developed without a developer looking at the specification. In some cases features were developed without a spec as there where no spec available. In many cases this was not a problem as the feature worked out quite well. The designer could then go over afterwards and suggest improvements.

Because design was immediately delivered to the developers when ready it did not get to mature and therefore some design was a bit premature and ended up as features that eventually had to be replaced. If the designers had a week or two to digest and circulate the design it is more likely that the design would be better and that it would lead to a better solution.

Coupling design specs to prototypes to issue/task tracking and to test scripts became difficult. For project participants not sitting next to the designer it was hard to know what design was finished.

Incomplete specs led to vague definitions of the done criteria and test scripts were hard to write.

Summary of problems with doing design in the same sprint as it is to be developed

Difficulties planning and estimating work
Increased risk
Poor documentation and traceablity and findability of documents
Often changes in specifications
Design not always harmonizing
Redesign and reimplementation since first version is not good enough
Relevant persons are not included in the design phase and feel overlooked
Noise, duplicate messages and repeating things
More waiting for clarifications
More difficult for new and remote participants
Distractions
Unpredictability and abrupt work flow
Design and development cycle partly breaks down
Unit testing more difficult and poor code quality
Lower productivity
Many half finished features
Difficulties in writing tests and organizing tests
Poor tests and poor test results
Chaos and poor work environment

Discussion

Again, it is difficult to exactly know what factors led to the problems listed above. There were several other decisions that are likely to have contributed to this situation. Having analysed the situation and thinking back on other projects I do believe the list of problems are representative to the issue of doing design in the same iteration as it is to be developed.

Being a developer on this project I am certainly not without some bias. So I'm inclined to say something positive about the process. But hey, as I'm sitting here writing this blog post I can't really come up with any benefits of setting up the process this way. Doing design in iteration i+0 might work perfectly well for small co-located teams but for a project with scaling factors [11] in a large corporation having strict requirements on compliance, traceability etc with a large distributed team this just don't work very well.

What can you learn from this?

Do not do design in the same sprint as it is to be developed. The design team should always be at least one sprint ahead of the developers.

The optimal solution would be for the designers to have a complete initial design ready before the bulk of developers start development. During this design phase selected developers, stakeholders and customer representatives etc help form the initial design. During development designers are tightly integrated into the group but are mainly working on evolving the design based on user feedback and on unexpected technical limitations etc. They would also focus on specifying details or things that have been overlooked.

How to avoid this situation

Plan and make sure managers, project lead, steering committee, product owner, decisions makers or other relevant persons are aware that designers should start ahead of developers.

What to do if you end up in this bad situation

Have the developers implementing core features without any UI design while the designers are working on the initial design. Get the solution up and running. After all this is what Agile is all about; making sure you have something working and then iterating to improve it.
For projects with scaling factors [11], formal design specifications are more important than on smaller co-located projects. A person can be assigned as responsible for documentation making sure specs are in order. If designers don't have time to do deliver good specifications then someone else should be in charge of writing up and formatting the specs.
Make sure the project is staffed up with plenty of designers. If there are 7 developers you might do with 5 designers such as an Information architect, an user experience designer a creative director a web interaction designer and a graphical designer.
Using teams of designers and developers that have previously worked together on projects with the same technologies. This will minimize the friction between designers and developers and reduce uncertainties.
Designer(s) must be tightly integrated and co-located with the project.

References

[1] CHI 2009 ~ Case Studies ~ Experience with Software & System Development and Evaluation

[2] Bringing User Centered Design to the Agile Environment

[3] UX Matters - Clash of the Titans: Agile and UCD

[4] IXDA - Discussion Archives - Interaction Design in an Agile Environment

[5] IXDA - Discussion Archives - UCD and Agile Programming - models of involvement

[6] Prior to Sprint Zero: A Note on Jakob Nielsen’s “Agile User Experience Projects”

[7] Agile + UX: (un)Synchronizing UX and development

[8] IBM Design - An Agile approach to User Experience and Design

[9] MSDN - Magazine. Usability in Practice, Agile Ux Development

[10] InfoQ - Agile User Interface Development

[11] IBM - Agility@Scale: Strategies for Scaling Agile Software Development

Link bombing spam ring discovered

2009-10-24T05:31:00.000-07:00

For not to many weeks ago I wrote about how spammers are killing me and my project fuzzzy.com.

After posting the blog post I deliberately shut down the site to see what happened, hoping the spammers would leave for good. After two weeks down time I put the site online again but without the SMTP server connection so that new users would not get the e-mail confirmation during the signing up process.

Guess what. Suspicious users still registered with the same rate even if they could not log in. This continued for 3 weeks. After three weeks, I turned the SMTP server back on and in an hour or two the spam started trickling in again.

So here's the things I have tried and did not work:

E-mail confirmation.
Captcha.
Wrote rigorously that the site is for web enthusiasts only.
Wrote that spam would be deleted without notice.
Added a human only answerable question to the sign up form. ('Are you human?')
Change the URL's of the pages most used by spammers to post spam.

Another thing I tried was to add a question where the answer is commonly known to the target group. The question I added was; 'Who invented the web?'. This actually had some effect. Spam went down by about 50% after adding this question to the sign up page. It seems that most spammers don't know the answer to this question and moves on to other sites to do their spamming.

One thing I have learned is to be very careful not to deploy pages that let users enter html so users can create url's in free text. Once this gets out amongst the spam ring you will get a hard time fighting them off even if you remove the ability to add hyperlinks.

Looking at the IP addresses of about 100 the spammers these are the typical ISP's:

Mango Teleservices, Bangladesh
Philippine Long Distance Telephone, Manila, Philippines
Digitel Mobile Philippines Inc., Philippines
National Internet Backbone, India
FibreNet Communications Ltd.Dhaka Bangladesh
Smart Broadband Incorporated, Sorsogon Philippines
TATA Communications formerly VSNL is Leading ISP, Ahmadabad, India
Smart Broadband Incorporated, Quezon City, Philippines
Bharti Broadband, Delhi, India
VietNam Post and Telecom Corporation, Vinh, Vietnam
Telefonica del Peru, Peru
Grameenphone is the largest telecommunication Orga, Dhaka, Bangladesh
NIB (National Internet Backbone), Sivakasi, India
FASTER CZ spol. s r.o., Brno, Czech Republic
Makedonski Telekom, Skopje, Macedonia
SC AVA TELECOM INTERNATIONAL SRL, Bucharest, Romania
Vietel Corporation, Hue, Vietnam
Telekom Malaysia Berhad, Kuala Lumpur, Malaysia
PTCL Triple Play Project, Islamabad, Pakistan
RELIANCE COMMUNICATIONS, Madras, India
SIA Lattelekom, Priekule, Latvia
Sify Limited, Calcutta, India
SATNET, Quito, Ecuador
SC AVA TELECOM INTERNATIONAL SRL, Bucharest, Romania

The list shows that most of the spammers come from poor countries or countries with high unemployment rates.

Looking further at the activities carried out and the spam they add my hunch is that there is a link bombing spam ring. Since most automated robots don't get past captcha's and other blockers, organized spam cartels will outsource spamming to poor people in developing countries.

One might think that these spammers are the scum of the earth.. wait scum of the web. But if we look at things from a higher perspective we will probably find that the digital-divide, the socio-technical and global networked economics of the world and the immature stage in the evolution of the web is what really has caused this cancerous spam situation.

So fighting the spammers is like slapping around poor thieves caught in their act.

Bashing up the thief will only make him sink deeper into the black hole his already in. Getting rid of one spammer only leaves room for another spammer. Instead we should focus on prevention and helping people out of their miserable situation. How do we do that? Fair trade is a good solution. Another good solution is to work on innovative R&D projects that will evolve the web.

Getting the right job

2009-09-29T21:28:00.000-07:00

This post is about how to find the job you will still like two years from now.

I have a friend that was recently sacked. The guy is sales person and in his new job he didn't live up to the expectations of the top management. Some of the reason was of course the recession but I believe much of it was a direct result of just choosing the best paid job without analyzing and weighting all the other things that matter.

Most people have a very simple approach to deciding the new job. If(newSalary > (oldSalary*1.3) {StartNewJob();} or something like that. Others just go by gut instinct.

I myself have for the two last occasions where I got a new job done a thorough analysis before starting a new job where I know I’m likely to be for the next years. Just starting somewhere because it sounded cool or the salary was good is not very smart when taking into consideration all the noise and problems that can come from choosing the wrong job.

So here's a To-Do list based on what I did before choosing my current employer Bouvet.

1) Write a list of all the things you look for in an employer.

2) Prioritize what things are the most important for you. Remember to take into consideration both short term and long term goals.

3) Investigate, find and learn about possible employers.

4) Try to get job offers from companies that seem to fit your list.

5) Create a matrix and give scores to each company based on information from job interviews, information you get from others and what you can find on the web.

	Weight	Company A	Company B
Knowledgeable co-workers
Salary, bonus
High spirit, positive environment, happy staff
Cool technology
Mixed workforce with diverse skills
Travel distance from home
Paid overtime
Possibility for travel
Stress
Cantina with good food
Not to much travel
Cantina with good food
Nice and interesting co-workers
Stable company economy
Competent co-workers
Long term career potentials
Short term ladder advancements possibilities
Managers are good role models
Ethical customer portfolio
Interesting projects and customers
Green environment
Nice premises/facilities
Other social benefits
Workout/health club
Possibilities to go on conferences and seminars
Possibilities to take expensive courses
Somewhere to relax in the office building
Not to much overtime
Flexibility (Don't have to be at work from 9 to 5)
Company style. Professional/informal

Fill in the table, replace the items with things that makes sense to you. Give points for each company and multiply with the weight column. At the bottom of the spreadsheet you'll see what company to choose.

And of course if you do not get any offers or know you are not going to get any offers. Go make yourself more valuable by learning new stuff, getting more relevant experience or getting a degree.

Spammers are killing me

2009-09-20T14:09:00.000-07:00

I have over the last years developed an experimental socio-semantic bookmarking service as a part hobby and part academic research project. The site can be said to be under early Beta testing. I have a lot of plans for the site but as it is being developed on my spare time the progress is not as fast as I could have hoped for. Also, as I write the project is kind of on the shelf while I explore another exiting hobby project in the area of social location based services.

Here come the spammers

In the middle of July 2009 I suddenly saw a rise in the number of new members signing up on my bookmarking service fuzzzy.com. Surely it's some robots spamming fuzzzy I thought. But after looking into the actions performed I soon figured out the spam where human generated. Based on the type of actions and data/metadata entered I could tell these where not generated solely by automated agents. It very much looked like coordinated spamming from a spam ring. There were no pattern in the IP addresses used. Captchas and human-readable-only questions on the sign up page did not stop them. Looking in the log-files at the seconds between actions, links added in the form of bookmarks and free html text links where added and modified in a typical human workflow. Some spammers also added tags and comments.

Why are they doing this

Obviously Google pagerank is the root of all evil link bombing spam. Often called link bombing, google bombing, spamdexing, referer spam, spammers add links to sites in order to promote a site and make it rank higher on Google and other search engines using page rank and similar ranking algorithms.

Spammers killing me slowly

For the last weeks I have got about 30 spam links every day and the process of removing the spam is killing me. Instead of using my scarce time on development and learning new stuff I am tied down for 10 minutes each day just verifying links and deleting spam. 10 minutes a day is not that much but its the feeling of fighting against a mob of EVIL EVIL EVIL real world spammers that really makes me just feel sad and frustrated. On just about every page on fuzzzy there is now text saying the site is a community site for people interested in web science and web development. Still, people keep bombing the site with spam links.

So what should I do

As the site is still in sort of early closed beta I don't have a bunch of users that can report, moderate and delete spam.

The few options I see are:

Close the site until I decide to focus 100% on the site and a real community is built around the site.
Keep deleting spam every day.
Develop a spam blacklisting service my self.
Report the spam to some third party black list.
Develop functionality that favours user with high reputation. Links posted by new users are just not shown until the link or the user is voted up or something like that.

If you have any ideas for how to fight the spam please let me know.

The last option or similar approaches seems to be the way to go but it does seem futile to fight the spam mob. If I can free my site of the spammers they will only move on as parasites to new victims. This only shows how primitive the current state of the web really is.

Disadvantages of using frames

2009-09-09T23:12:00.000-07:00

I often encounter developers who think it is a good idea to use frames or iFrames to implement web applications. The most often used argument is that with frames you don't have to update the entire window page. This blog post is an attempt to set the record straight and show why it is not a good idea to use frames unless you have to because of legacy reasons etc.

Benefits of using frames

Navigation can be fixed and always visible without scrolling.
Logo always visible which can be used to strengthen branding.
Less payload with page requests as only sections of the windows is refreshed.
You are able to put anything in the frames making the application seem more integrated or as one without having to open new windows or send the user to new sites.
If you mess up the markup of your page it might not destroy the entire layout of the site.

Most of the above can be solved without frames using different techniques although some of them would require more advanced technologies such as Ajax or DHTML.

Drawbacks of using frames

Facilitating search engines is more difficult and the indexing results by search engines are likely to be less good.
Linking to your site might gets tricky. If you are using frames you might need to set up some javascripts to recreate the frame structure.
The end user can not use the browser bookmarking or favourite functionality as normal.
Many people do not know how to copy the actual page address of a page within a frame setup. The end user can not copy the address of the current page from the browser URL address field since it probably just displays the root page url.
If you do not have full control over who creates pages for your frames you can not be sure that branding and style guidelines are followed.
Normal navigation for end users with back and forward buttons does not work as expected. (The browser address field is not updated)
Hyperlinks on the page inside the frames might intentionally or unintentionally open other external content in your frames, degrading user experience and branding.
Printing the pages gets more difficult.
Printed pages might loose it's contextual setup such as the breadcrumb trail and it will be difficult for the user to get an overview and find out where the page was printed from.
Frames might not be supported very well by many types of mobile or other type of compact clients.
The use of frames increase the chance of multiple and horizontal scrollbars when users do not maximize the browser window or when she has low screen resolution.
Frames are less usable for handicap users (see WAI/WCAG).
Page rendering is slower since firing up frames and doing multiple page requests usually takes more time.
The page rendering might seem unnatural as things are not rendered in a linear fashion.
You need to make sure hyperlinks all point to the correct sub frames which require additional testing.
Frames impose restrictions on the graphical design and layout.
The refresh-button seldom functions as expected.
You can get into copyright problems or confusion about the origin of content when intentionally or unitentionally presenting external content.
Many web application frameworks or content publishing systems are not designed for frames and can thus be an obstacle.
Presenting footer information gets more difficult to present in a consistent matter when using frames (not iFrames).
The user might experience the site as less trustworthy as the current page URL seems to be cloaked.
Use of frames is out of date and the site can be perceived as not being modern.
The use of frames requires more http requests which, to some degree, outweigh the benefits of less byte size payload.
The end user does not get a clear and consistent confirmation on when a new page is loaded.
Frames mess up the page metaphor and will make the site more confusing for novice end users.
If your frames are using session state you might encounter timeout or login forms in the wrong frames which cause a lot of confusion.
As a developer you might experience problems when the front end code base gets large and cross frames code and other javascripts start to make site browsing seem slow or awkward.
When using frames it is harder to debug front end issues.

Conclusion

Avoid frames whenever you can. If used, you should definetly know what you're doing and frames should only be used for application type of sites that do not need the typical document/page metaphor.

Dynamic resizing of iFrame, pros and cons

2009-07-13T05:11:00.000-07:00

If you are a web developer you have probably encountered iFrame resizing requirements.
Because of legacy code, your portal framework or some other reason you have to use an iFrame and because of the fancy layout from the designer you are required to introduce an iFrame that fills some section of the page that adjusts itself depending on the content that goes into the iFrame, and of course you do not want extra scroll bars appearing on this iFrame and you do not want a vertical scrollbars except from on the outer right side window edge when the page is long.

So how do you go about solving this problem? There is tons of sample code that shows what you can do but you will find few articles summing up pros and cons of alternative solutions.

Basically there are two solutions. I will explain them shortly and present a list of pros and cons. Examples of the two alternatives, I am sure you can find your self. jQuery can be used for both solutions.

Alternative 1) Resize iFrame from within inner frame
Description: With javascript on both the outer page (the one hosting the iFrame) and inner page, iFrame is resized after the inner page has checked its own height and sends this value to its parent. In order to allow cross frame scripting you may need to set the document.domain property on both pages.

Pros:

We get a scrollbar on the outer most right side (browser window) when the page is long. Same as on all other type of long pages.
Script only needed on long pages.

Cons:

Might require changes to browser security settings or modification of trusted sites list.
When the inner iFrame page has dynamic content (uses ajax/dhtml) that expands the initial page, multiple scrollbars will appear unless you attach your javascript to these dhtml functions.
Requires that you have control over pages that are to be hosted within the iFrame. javascript must be included on each page (or a shared include/template file).
Requires setting document.domain property using javascript which might cause future unexpected problems because of dependencies?.
Both hosted page and parent page need to have the same domain names (not the same subdomain) using the same protocol (http/https).

Alternative 2) Resize iFrame to fit page from outside
Description: Using javascript, iFrame is automatically resized to fit outer window on window load and resize event.

Pros:

Approach is guarantied to work (if JS is enabled). This means less risk and less chance of bugs.
Less complex solution and therefore more likely to work over time as you or your IT department do browser upgrades or make changes to browser settings.
Can be used to show external resources not under your control.

Cons:

The scrollbar will not span the entire height of the window. Only from the position of the top navigation menu or so and down to the window bottom.
While resizing the window there will be some flickering and scrollbars might seem sluggish.
iFrame must span entire width of window, this might put some restrictions on layout design.
Script to resize iFrame always have to be triggered on page load (and resize) and might contribute to slower rendering on very slow machines.

Using Twitter for your Intranet

2009-03-31T22:44:00.000-07:00

When developing Intranets these days, Twitter often comes up as a possible service to integrate. This blog post takes a look at Twitter for enterprise Intranet solutions.

There are of course other micro blogging/messaging services that may be more geared towards the Enterprise (such as Yammer) but Twitter obviously has a great advantage in the large user mass.

Benefits of Twitter in enterprise Intranets:

Enables spontaneous knowledge sharing.
It supports status updates by letting people easily twit what they are doing at the moment.
It strengthens your social network.
It can be used for solving problems faster by letting you broadcast questions freely.
It enables new exciting possibilities for collaboration and communication.
It enables user centered subscription and full control of what information to receive.
The distance or divide between managers and employees is less when managers start twittering.
The barrier to communication is lower. You can still twit if you are in a hurry.
You can receive breaking news faster.
It can be used to profile the company as futuristic and innovative by using new media.

Drawbacks of using Twitter in enterprise Intranets:

It can become a waste of time. (like Facebook often is)
Often interruptive. (Especially if you use a twitter client that shows popup messages)
It can be seen as just another tech hype. (It will be replaced with something else in a year or two. Employees will have yet another service to sign up on.)
Because of the format and the low barrier to publish information it can be more easily used to spread gossip and rumors.
It can contribute to make a skewed perspective of some issue or subject because twitter is all bottom-up in contrast to editorial quality assured top-down information that takes longer time to prepare.
Higher chance that employees will leak confidential or sensitive information.
We don't know how Twitter will develop it's service and how we must evolve our systems to support and maximize the benefits of the service.
The long term consequences of use in the enterprise is unknown
Integration of service will cost money, depending on how you integrate it.

Sprint retrospective helper

2009-01-12T05:04:00.000-08:00

If you're a developer doing SCRUM having regular sprint retrospective meetings you have probably experienced the situation where you are supposed to write down what went well and what could have been done better. You try to come up with some neat suggestions but you feel that a bunch of topics have been forgotten.

To make it easier to remember stuff for the retrospective you can always start jotting them down when you have them fresh in memory so you are sure to have them on the retrospective.

The list below was made to help remember stuff for the retrospective. It's a sample list or example of keywords to jog your memory and help you brain storm. This list might also be used for small informal project evaluations etc.

Process
standup, daily, meetings, deployment, out of control, reproducing, testing, planning, demos, keeping track, improving, design, results are not lost, handling issues, stabilization, complex, analysis, integration, velocity, timing, flow, pace, think, sprint planning, charts, start and stop, costs, tendencies, peer review, progress, feedback, goals, sticky notes, whiteboard, stepping on toes, race, hand over, significant events, timeline, consensus, management, senior, anchoring.

Policies
source control, decisions, documentation, who decides, hand over, scope and deadlines, quality, staffing, QA, code review, getting credit, mission, vision, risks, bug tracking, issues, automation, analysis, refactoring, activities, accountability, user stories, phases, patterns, architecture, holism, holidays, culture, flow, dummy data, data creation, silo, not used, bounce, ripple effects, peer review, boundaries, proof of concept, skills, metrics, completing stuff before moving on, training, knowledge management, help desk, manual steps, documentation, prototype, risk analysis, cost analysis, contract.

Work environment
collaboration, work hours, new members, number of members, organization, customer, development tools, scrum tool, communication tools, roles, stakeholders, customer involvement, engagement, uncertainties, environments, optimizations, tasks, boring, exciting, change, skills, food, chaos, familiarity, fun, stress, computers, software, 3rd party, external , actors, energy, air, sit/stand, down time, remote/local, crash, branch, control, differences, conformance, competition, constructive, freedom, job security, fatigue, hopelessness, knowledge, oversight, feelings, blaim.

Communications
project participants, dependence, overview, noise, out of sync, expectations, errors, bug, reports, backlogs, requirements specifications, availability, members, stakeholders, members are up to date, lessons learned, loops, acceptance, rejection, hours, timeboxing, impediments, design, styling, wireframes, business goals, unity, talking to seniors, backlog, details, understanding, agenda, conclusions, vague, abstract, appreciations, secrets, honest, creativity, confront, good enough, impact, consequences, questions, ask, feasibility, summary.

Hopefully I'll not get burned out

2008-09-17T12:31:00.000-07:00

I recently read a blog post by a highly regarded co-worker of mine at Bouvet. I found his blog post, which was about getting burned out, to be both helpful and very important as more and more people are getting burned out nowadays. I started to write a type of get well comment on his post and found that my response was getting too long so I decided to write my own blog post on the subject of getting burned out.

Occasionally I’m asked how I live the double life I do with full time work, doing research and development on the spare time, instructing martial arts and having a girlfriend. My typical answer is: Prioritizing, lifestyle and planning. But this way of life is not without a cost. My work and computer related interests typically have me sitting in front of the computer 10 hours a day, 6 days a week and it has been like this for some years now. This is of course not what some million years of evolution had in mind for me. I have for some years been wondering if I am on the verge of getting burned out myself after symptoms like dizziness, neck pain, headaches and general fatigue.

After reading about Lars Marius' experience with getting burned out I am surely going to be very careful to follow up on my precautionary measures. As I understand, the burned out illness or Chronic Fatigue Syndrome (CFS) can be characterized as a total breakdown in the immune system. To strengthen the immune system and to avoid CFS in the first place, a combination of diet, nutrition, exercise, and stress reduction is said to help [1] [2], [3]. The fact that CFS is poorly understood in western medicine indicates that there might be many factors together that are causing the illness. I personally believe I would have crashed totally long ago if I had skipped some of the points below. The list below seems to be the right combination for me:

Stress management
Healthy food
Breaks during the day
Less alcohol
Workout
Less coffee and other stimulants
Drink lots of water
Stretching exercises
Relaxation exercises
Good sleep
Right posture and sight
Having some fun

For those who fear to be in the danger zone. Here is a more in-depth explanation of what I do to stay on top. Please note that I’m not a doctor and I am not saying that this is how you should live you life. So here goes.

I believe stress is the most important factor. Read a stress management book. Take control of your life, make plans. Think before you act. Make sure you avoid worries. A relevant example these days: Don’t take up loan you can’t pay back if the interest rates go up. Hakuna matata, yes you can look that up on wikipedia. Don't worry too much. Think positive. Avoid conflicts. If they happen then meet them head on with honesty, self respect and courage. Avoid negative people. Healthy food to me means a lot of fruits and vegetables. Herbs, fibers, vitamins, Omega-3, antioxidants and cutting down on what’s generally unhealthy food. Take a food intolerance and allergy test. You don't need to drink 4 cups of coffee a day. Drink some herbal tea instead. If you are tired at work then you should see to it that you get your sleep. Coffee and other stimulants will remove symptoms like tiredness. Listen to your body. If your body say it needs sleep make sure it gets enough over the week. Drink lots of water. Make sure to have a bottle of water within arms distance all day. Breaks during the day are important. Do some outrageous movements to stretch those muscles you will not use during the day. Working with computers more than 7 hours a day will make you weak from the lack of movement. Adding some work related stress will make you stiff in addition to weak. Work out, preferably with lots of versatile movements. Cardiovascular aerobic training is better than anaerobic which generally makes you more compact and stiff. Workout or at least take a walk. It will strengthen immune system and increase the resistance to the effects of stressors. Find time to relax, find harmony. Learn to meditate or do breathing exercises. Find that place where you can escape weekly to enjoy the surroundings. Pamper yourself. Make sure you get sun. Get glasses made for computer work if you don’t have perfect sight. Learn good touch typing skills and make sure you sit in an upright ergonomic position [4]. The combination of physical aerobic exercise, healthy food, lots of water will help the body to flush out the toxins in the body. Research suggest that toxins are the cause of CFS [3][5][6][7][8] I also believe that moral is a key issue. Being a good person helps you to feel good about yourself which will reduce stress and you will become more at ease. You do not have to be religious but try finding that something that can make you have a stable and clear vision in life. Another important factor: Don’t be a slave. Live free and not a slave to what others thinks of you. If you feel that you should have the latest and most expensive cell phone when you don’t really need it, then you are a slave to capitalism. A decent lifestyle is important. And last but not least, don't forget to have fun. If you are having a job where you just go to work because you feel a responsibility then you should get a job you like. (I know that this is not an option for many)

[1] C-Helath- Chronic Fatigue Syndrome
[2] Cleveland clinic - Diet, Exercise, Stress and the Immune System
[3] PHOENIX RISING - A Guide To Chronic Fatigue Syndrome (ME/CFS)
[4] pcworld.com - Essential Ergonomics
[5] healingessences.com - Toxicity
[6] Holistic Health Topics - Chronic Fatigue Syndrome from a Holistic Constitutional Perspective
[7] Campaign for a National ME / CFS clinic - A neurotoxin called Ciguatera discovered in Chronic Fatigue Syndrome
[8] Myalgic Encephalomyelitis/Chronic Fatigue Syndrome Forum

Checklist for deciding implementing Topic Maps

2008-01-15T12:02:00.000-08:00

When should I go for Topic Maps? Why should I use Topic Maps and where do we use use Topic Maps? As a system developer or architect you may have asked these questions. The use of Topic Maps does undoubtedly increase complexity of an IT development project but at the same time does provide other very valuable benefits in comparison to plain relational databases.

This checklist can be used as a tool to decide if to use Topic Maps technology. Note that this only discuss using the Topic Maps paradigm, it does not discuss to what extent is should be used or on what parts of the system etc.

[ ] We need a flexible data model (3)
Do you need to model and represent knowledge that is evolving (evolving ontology), where concepts, their metatypes and relations change over time? Do you have information consisting of diverse concepts and need to make distinctions between their type, their relations and the roles the associations play among each other and you are not sure if this conceptualization will change in the future?

[ ] We need a high degree of (semantic) interoperability (2)
With Topic Maps PSI’s you get a strong and accurate identification of subjects between loosely coupled systems. Do you need to represent knowledge such as “Oslo is in Norway, Oslo is a City and Norway is a country”? With Topic Maps this can be expressed in interoperable formats that make sense to both humans and computers. Topic Maps enable import and export of both complete knowledge structures and of fragments only.

[ ] We need to merge semantic information or arbitrary knowledge from diverse sources (2)
Merging is a built in feature of the Topic Maps standard. Queries on the data will work after a merge.

[ ] We are developing a knowledge centric application (1)
Are you going to integrate or build a Knowledge Management system where knowledge generation, codification, and transfer is essential? Is there a need for a paradigm that can collate everything known about any given set of subjects? Do you need a Ontology (a conceptualization of a domain)? Do you have users that will want to interact with the system to create new knowledge? When using Topic Maps people often see new potential relations and thus increasing knowledge leading to new insight, innovation etc.

[ ] We need a technology to access information using different criteria or navigation paths (1)
Do you need a technology that is tailored for displaying multiple navigational aids or multiple views/paths in a networked information structure? Built into the standard is the concept of scopes that let the modeler contextualize information. With Topic Maps you get two-way or navigation. If you need to provide faceted navigation or search where users navigate by related concepts, then Topic Maps may fit your needs because of its support for bidirectional associations and scopes.

[ ] We need a strong separation between metadata and resources (1)
Do you have persons that work with metadata and others that add or edit resources? For instance your organization might have a vocabulary that is maintained by a set of dedicated specialists and you might have authors that create content which is categorized with the taxonomy. Is there a need to manage some form of complex networked information structure or other information structure such as a controlled vocabulary, concept map or taxonomy?

[ ] We have a strategic goal to be among the first to adopt new technologies (1)
Topic Maps is similar or complementary to the Semantic Web initiative, which are seen by many as the precursor for Web3.0 and eventually the raise of Artificial Intelligence and collective wisdom for the web. For many organizations it is a marketing strategy to identify itself with such technologies.

Conclusion
Add the points for each of the sections you answered yes to. The points are indicated from 1 to 3 at the right-hand side of each section. If you got a sum of 3 or more points you might have a good reason for embarking on a Topic Map development project.

Topic Maps pros and cons in comparison to Relational databases

Pros
Provides a semantic interoperable knowledge structure for use by both humans and machines. It will give you tremendous power in modelling a domain. Topic Maps have built in constructs to make changes to a model less painstaking after the system have been deployed and is in production. Last but not least, Topic Maps is an enabling technology that opens up new possibilities for use of the embedded knowledge within the Topic Map.

Cons
Topic Maps projects require more development time. It often leads to highly increased complexity, which again results in systems that are harder to maintain and require special competence. It also requires large amount of cognitive efforts under specification, design and development. Topic Maps also increase the risk in the project due to higher complexity and higher demands on competence. It will often require higher demands on computer hardware because of more complex database processing.

Last updated 5. April 2008.

Global tagging with Fuzzzy

2007-10-05T12:42:00.000-07:00

Fuzzzy 2.0 was launched October 1.st. Among the many new features is global tagging support.

Global tagging enables everyone around the world to participate on building tags that are connected. Instead of adding dumb text categories to your Flickr photos, del.icio.us bookmarks or your weblog posts, you can now use a shared pool of tags. By moving away from simple folksonomy tags that are only valid within the single site you can move on to semantic tags that bring with them connectivity and meaning. This increases knowledge sharing by having a shared vocabulary. It also increases findability, learning and consistency.

Try it out for your self:
A. Distributed tagging
If you open the tag "Topic Maps" http://www.fuzzzy.com/tag/?id=1607
you can click the icon that looks like three red connected nodes. Then some dummy related tags from the tag network will be loaded. At this moment there are just two tag servers.

B. Tag server functionality.
On this dummy blog http://om.niscio.us/cowboyroy/ you can tag the blog post with tags from the Fuzzzy tag server.

How was it made possible?
The simple version for those of you who don’t use most of your spare time reading geeky stuff: Global tagging is made possible by the organic tag-set of Fuzzzy which is built upon the Topic Map ISO standard and an underlying infrastructure with web services. Members of the online community can create tags and relations between them. Users can also vote, comment and in a democratic way participate to build a shared tag vocabulary.

Now the geek version: By using Topic Maps for semantic interoperability and the built in identification model with Public Subject Identifiers (PSI’s) tags becomes universally identifiable. The globally connected tags become a shared vocabulary for both people and computers. By using web services with simplified XTM fragments any peer tagging site can request from all other peers related tags for tags that are found throughout the network. If in two different domains the same subject is mentioned then these domains are be connected when the subject has a published identifier. When a user makes a request for related tags at the global level a request goes out to all peers. The list of remote sites are replicated and stored on each peer. A webservice call such as this one http://www.fuzzzy.com/ws/GetTagByPSI/?psi=http://www.fuzzzy.com/tag/3053 is issued and all related tags are returned as XTM. The associated tags at the remote site can then be viewed or imported.

Why was fuzzzy developed

2007-05-24T02:07:00.000-07:00

Fuzzzy was part of Roy Lachica's masters' thesis. The purpose of the project was to:

Test a new type of metadata categorization, namely the "folktology" (folk + ontology) a semantic tagging system which resembles folksonomies but are built on Topic Maps.
To investigate polyscopic, democratic, collaborative and holistic aspects of knowledge creation.
For personal benefits. To learn Topic Maps and to become more nerd.