Kirsanov.net

The way AI has subtly changed our tools

2024-07-22T19:48:41+00:00

Yesterday I was doing a thing that seemed to be nearly impossible a couple of years ago. I translated video from one language to another, so that it would be “professionally narrated” in voice and language that I prefer. I did it using the latest update of SoundWorks, my tool of trade for sound and multimedia. But the point is – I counted, how many AI tools are used in process, behind the scenes. And it turned out – quite a few.

SoundWorks takes your original video and produces new one, with either added voice, or replaced one. A must have thing for training videos. Apart from obvious FFMPEG stuff, such as extracting and attaching audio track, it transcribes existing video (1), rewrites text, if needed (2), creates new voiceover (3), and if needed – translates it to another language. And that’s 4 different AI engines, where in fact there are more, but we choose any four. For example – we can translate with DeepL, or do it with Azure. We can rewrite using ChatGPT, or do it with Claude.

Even without AGI (Artificial General Intelligence, basically true intellect), we are expanding our toolsets without noticing. It’s one of those changes you don’t notice, which slowly changes life until it hardly resembles the one we had 10 years ago.

Since I find this feature of translating and automatically voicing videos and audios to be extremely useful, here is the video that is showing how it works. SoundWorks is free, by the way.

Free DALL-E from Microsoft, with a catch

2024-03-14T12:51:41+00:00

As part of Office 365, Microsoft released Designer – basically, a service to generate images using DALL-E.

https://designer.microsoft.com/image-creator

The catch is that while OpenAI doesn’t claim any rights for your images, Microsoft states, that:

“By using the Service, posting, uploading, inputting, providing or submitting content you are granting Microsoft, its affiliated companies and necessary sublicensees permission to use your content in connection with the operation of their Internet businesses (including, without limitation, all Microsoft Services), including, without limitation, the license rights to: copy, distribute, transmit, publicly display, publicly perform, reproduce, edit, translate and reformat your content; and the right to sublicense such rights to any supplier of the Services.

No compensation will be paid with respect to the use of your content, as provided herein. Microsoft is under no obligation to post or use any content you may provide, and Microsoft may remove any content at any time in its sole discretion.”

Microsoft wouldn’t be Microsoft, if it wouldn’t claim rights for what you already paid.

Another word about Rabbit R1

2024-03-07T14:07:00+00:00

It’s only weeks until a revolutionary gadget to be released, so for the sake of history, let’s join the pieces of information we have regarding this prototype device. I’m saying prototype, because it’s the first of a kind, and even if it will fail, the idea behind it is what makes it a piper of new industrial revolution.

Disclaimer: I’ll quote Rabbit team below, so when I say “we” it means “they”. All of this information is first-hand and is not my opinion. It’s not necessarily true, but what the Rabbit declares as truth. It’s here to compare with reality later, when first reviews will be published.

The most commonly used example is DoorDash, U.S. food delivery service. That’s because Rabbit representatives are quite hungry until their devices become popular, take over the market and allow them to dictate the rules.

The Rabbit device, priced at only two hundred dollars, is a surprisingly affordable piece of modern technology. What's even more surprising is that there are no subscription fees attached to it. Rabbit has confirmed that they have no plans to introduce any subscription fees, although this could potentially change in the future. The reason for this affordability is the efficiency of the large action model and the Rabbit OS, which are less expensive to run than you might think. Once you purchase the device, it's yours to use without any additional charges.

There seems to be some confusion about the difference between a large language model and a large action model. Many people assume that it's just a marketing term or a large language model with a few extra features. However, this is not the case. A large language model, based on the Transformers architecture, is excellent at predicting the next token in a series of tokens, or the next word in a sentence. But this wouldn't have worked well with the Rabbit application.

What Rabbit does is convert your natural language into the action you want to accomplish, and then performs that action on a hidden operating system. Imagine using your phone to order food from a restaurant via DoorDash. Rabbit does all of that for you, figuring out how to navigate through the interface. It doesn't connect to an API, which makes it more efficient and less costly to run, hence the lack of a subscription fee.

Jesse Liu, in a podcast, explained the difference between a large language model and a large action model. He clarified that Rabbit is not creating a large language model, which is based on Transformer and requires a significant amount of GPU on the cloud to train and get things right. Instead, Rabbit works with the best language models and small language models, and potentially open-source language models in the future. They continuously monitor the performance of these models and can switch between them as needed. This is how Rabbit OS works.

Rabbit focuses on the large action model, which is neurosymbolic. While the Transformer-based language models were designed to understand language better, they perform poorly at completing tasks. Rabbit doesn't work with APIs because not everyone will provide an API, and even if they do, they often don't replicate the full feature of the application. Instead, Rabbit aims to create a universal solution, an AI that can handle any application, whether it's an Android app, iOS app, or Windows app. I think it’s important to note, that “Windows” in this case is the same as Linux was for early .NET. In 2001, Microsoft did think that .NET will be cross-platform, but didn’t do a thing about it. Same is here, Rabbit isn’t “Windows-first”, but of course the LAM will be compatible with Windows. Same as LLM is not just Python and NVidia. Did you know that it’s not?

To achieve this, Rabbit started working with data labelling companies about two and a half years ago. They assembled test groups to interact with different software like Uber and Spotify, and recorded these interactions. These recordings were then fed into the large action model, which reads the clips frame by frame.

Many people have questioned why Rabbit couldn't have just been an app on a phone instead of a standalone device. The answer is that Rabbit is trying to break the mold and show us the future of computing, which they didn't believe could be achieved with just an app. When you use your phone, you navigate through apps and find what you need to do using your finger. But with Rabbit, you speak in natural language and an AI agent performs the actions on your behalf. This wouldn't have been possible if Rabbit was just an app.

As for the concern of having to carry around two devices, it's worth noting that many people already carry around multiple devices.
In today's world, we are surrounded by various devices - phones, laptops, earbuds, and now AR and VR devices. As an early adopter, I find it exciting to carry around another device, especially one that is sleek, small, and lightweight. This device, which I believe represents the future of computing, is something I don't mind carrying around.

Many people wonder if Siri and Google Assistant will eventually incorporate the functionalities of this new device. While it's likely they will add some of these features, they've had over 15 years to do so and haven't quite hit the mark. Things are bound to change rapidly in the future, and I'm excited to see some competition in the market.

Apple and Google are trying to avoid the pitfalls of the innovator's dilemma. They have existing products with established frameworks and ways of thinking, making it difficult to break out of that mindset. Even if Siri and Google Assistant evolve, they will still have to work within the constructs of what they've previously built.

Rabbit, on the other hand, wanted to innovate and do something completely new. Jesse Liu, the creator of Rabbit, explains why he decided to build his own hardware device instead of just creating an app. He believes that the Rabbit is faster and more intuitive than other devices. For instance, when he needs to define a word or look up some information, he finds it quicker to use the Rabbit than to open a new tab and start typing on his phone or laptop.

The Rabbit also has a music feature, which Liu describes as a second iteration of the classic iPod with even simpler controls. He uses the Rabbit for music and searches most frequently, with searches making up about 70% of his usage.

Privacy and security are also major concerns for many users. Rabbit takes these concerns seriously and has a dedicated page on their website about privacy. Like, every other website and company. The device has a physical way to block its camera and does not have an always-listening mode. It only hears you when you hold down the physical push-to-talk button. The Rabbit OS also features a gyroscope-enabled privacy mode where all modes of input and output are disabled when the device is facing down.

As for the information you share with the Rabbit, the company ensures that it stays with the device. They will not use personally identifiable information for any purpose other than serving you. You can choose to have Rabbit OS remember or forget things through their Rabbit Hole web portal. They are committed to never selling or sharing your data with any third party without your explicit permission. This commitment to privacy is crucial in today's digital age.
Rabbit OS is a secure and innovative system that allows you to interact with your own Large Action Model (LAM) in a dedicated and isolated environment on your cloud. When you use Rabbit OS, you are assigned this environment, which is solely for you and your agent.

When you need tasks performed, Rabbit will use your own accounts, which you have “securely granted them control” over through the Rabbit Hole web portal. This could be for services like DoorDash, Spotify, or any other service. You securely log into the web portal using SSL, providing authentication permission just like you would with any other third party.

We prioritize your security and privacy. We do not store your passwords for these devices. During the execution of any tasks, especially those involving sensitive actions such as payments, Rabbits will ask for your permission and clarification. This ensures that you won't accidentally order $500 worth of Snickers bars from DoorDash.

Rabbits are a safe and respectful representation of a legitimate user. We do not create fake accounts, spam accounts, or engage in any abnormal access patterns or traffic to the target app. The AI agents we use look and behave like humans. We do not attempt to reverse engineer any service or aim to replace existing API equivalents. We do not solve captures and we do not seek to significantly alter the user base of the apps.

The agent performing an action on your behalf should be no different from you performing it yourself. The only difference is that it saves you a ton of time by doing it for you.

Rabbit OS is a startup, constantly evolving and improving. We strive for transparency and innovation.

Your passwords are not stored on the device. All security measures are done securely in the cloud. The hardware device simply acts as an interaction method between you and the agent.

Jesse Liu, a representative of Rabbit OS, explains how authentication works. When you tell Rabbit to order sushi for a family of five, for example, it knows how to use the Uber Eats or DoorDash app. This request is then sent to your service somewhere in the cloud. You have already authenticated your Uber Eats and DoorDash account in a web interface.

The device doesn't have any software pre-baked in or anything pre-installed. It's just an AI. You choose what kind of service you want to enable and how advanced you want this device to be.

We have a web portal that helps with all the authentication settings and feature management. You go to the website and choose whatever services you want to unlock. You then click the connect with Spotify button, for example. It will redirect you to Spotify login, and we don't save your credential.

We have a superhost computer that interacts virtually with a DoorDash app or website when you talk to your Rabbit R1 about ordering a hamburger from DoorDash. We then re-render Rabbit-themed UI on your device to get your result.

Conclusion and bets:

They are using hosts, and your device is not independent. That means – subscription is imminent. I’d estimate it to be the same as Amazon Prime or Disney+.
Big sharks will either buy them, or make their own “revolutionary devices”, depending on success of the Rabbit OS.
Rabbit is bound to become next Evil Corp in case of success.

AI job replacement: Another one bites the dust

2024-02-07T21:08:02+00:00

Despite the frequent reassurances that artificial intelligence (AI) will not render our jobs redundant, there is a caveat that we must acknowledge. While AI is not going to disrupt every professional sphere, it is certainly on track to replace roles that were initially established as mundane, non-creative tasks, often performed merely to maintain status quo.

Let's explore an example of such a role , the weather announcer. Regardless of whether the task entails live TV broadcasts or crafting narratives for online platforms, it essentially remains a monotonous duty. Enter a revolutionary website that has successfully replaced the human factor in weather forecasting, and quite impressively so.

This pioneering platform presents daily and weekly weather forecasts through a variety of mediums, including animated videos, audio podcasts, and elaborative texts. What's incredible about this service is not just its comprehensive automation, but the value it adds by going beyond mere data presentation. Rather than merely listing out weather elements like temperature, humidity, and so on, it performs a complete analysis, turning the raw data into a practical, user-friendly dialogue about what tomorrow will look like, hour by hour, and what it will actually mean for you.

At this stage, the service caters to a select number of cities (such as Dublin, Riga or Moscow), positioning itself as a trail-blazing technological experiment that combines various AI applications to produce one innovative solution. One of the key features of this service is its multilingual capacity, designed to accommodate cities with diverse linguistic landscapes. The weather analysis generated by the platform is then translated, vocalised, and spun into engaging videos that are uploaded onto locally popular hosting platforms such as YouTube and Rutube for Russia - even Telegram and their own website.

This glimpse into the transformative power of AI provides a vision of what our not-so-distant future may look like. It offers a promise of practical assistance in our daily lives, in ways we may not have imagined possible. This is the AI revolution - not a takeover, but a redefinition of how we perceive and interact with the world around us.

Chasing the Artificial Intelligence Rainbow

2024-02-05T02:58:58+00:00

A recent trend that sticks out like a sore thumb in the buzzing world of tech is the splashy, high-profile marketing of 'AI' laced devices. A galaxy of household name companies (pun intended), who ironically have made minimal contributions to real AI development, are selling nothing more than illusions wrapped in layers of hyped advertising and fad appeal.

Samsung Galaxy S24 case

Let's take the Samsung Galaxy S24 for instance. The successor to the S23, Samsung heralded it as its "first AI phone". Just to be clear, as its name suggests, the Galaxy S24 Ultra, and its siblings ain't exactly lemons. They're backed by Samsung, after all. Yet, a closer look reveals trickery and shortcuts they took to roll out their 'AI' capabilities.

Well, long story short: the 'AI' built into the S24 is about as useful as a battery-less flashlight. And yes, there are few free features, like the language translator during calls but rest assured, their generosity has an expiration date much like the contents of your fridge.

These “AI functions” support features you won’t need and don’t support what you might want. Hence this can be said about all AI functions: they are “there”, but you won’t use them because they won’t fit into your scenario or “aren’t there yet”. Aren’t, and won’t be, no matter how strong your faith is.

AI Translation; A Bridge Too Far?

The in-call translation feature is a neat idea, no denying that. Yet, before you start planning your next business trip abroad, take note; the supported languages are a bit… well, let's just say unless you're an Ivy league linguistics graduate or have a sudden urge to start conversations in Korean or Vietnamese the feature is practically moot. The translation feature does also support French and German but the results are on the unimpressive side of the line, somewhere between toddler babblings and Google Translate's beta version. Besides, Germans perfectly speak English. French might like it because they don’t like English, and Spaniards – well, perhaps they might love it.

The Battle of the Clients

It's becoming increasingly clear that our smartphones are transitioning from hardware-heavy tech pieces to simple hardware vessels crammed with complex software - a striking move towards the concept of thin clients. The upcoming 'Rabbit R1', is a classic example of this thin client trend in the works, just as the Galaxy S24 Ultra is for a thick client (devices that have the hardware brawn to run operations offline). Interestingly though, they both share a common pitfall: they're nothing more than glossy promissory notes. I'd dare to say, the S24's check has already bounced.

What about Rabbit R1?

And speaking of Rabbit, it's flexing its muscles on the scene with an ambitiously priced AI device pegged at a moderate $200 without any subscription fees. The offer itself should raise more eyebrows than the last season finale of 'Westworld'. Also, and quite worryingly so, the lack of transparency regarding other hidden costs reminiscent of a velociraptor hiding in the bushes, is a major red flag. While Rabbit's exact intentions remain undetermined, their seemingly innocent flirtations with Matrix-themed marketing lingo ('Rabbit hole', 'white rabbit') may provide a clue to their true intent.

Where do we stand?

Is the maturity check for the fabled AI devices clearing yet? The answer, thus far, is a resounding 'No'. Even Samsung's best attempts at AI integration, to be frankly honest, reek of empty promises dressed up in shiny PR. It looks like the tech industry's journey towards Authentic AI nirvana is set to trudge along two contrasting paths - the thick clients led by tech behemoths like Samsung (who, by the looks of it, are bound for a tumble), and the thin clients, which are more likely to depend upon private data centres with, hopefully, a decentralized setup enabling users to own their backend machines or choose their processing centres. For us, the eager spectators of this AI-centric bout, this competitive heat is nothing but good news. Bring on the popcorn folks, we're in for a show!

Design Language Is Not Design, by Bill Buxton, 2012

2023-12-12T01:25:21+00:00

A speech by Bill Buxton in Dublin in 2012, which is worth reading even today, as most ideas about design are rather timeless, and so are some books. Enjoy!

It's been a while since I last visited Dublin or Ireland, with my last visit dating back to 1982. I remember climbing Carentoon Hill during that trip. Now, let's dive into the topic at hand.

In a recent survey, we asked how many of you are designers and how many are developers. But what about the rest of you? Are there any architects, students, CEOs, or management executives among you? How many of you are involved in the business or finance side of things? Ideally, we should have an equal representation of people from design, technology, and business disciplines. These are the three pillars necessary for any business enterprise in today's world.

Now, preparing for this talk was a bit challenging, not because I had nothing to say, but because I was trying to figure out what would be worth saying. I first used a computer in 1971 to compose music for a film soundtrack. That computer had a mouse, real-time sound synthesis, computer graphics, two-handed input, music notation, and a piano keyboard. It was easy to use, and I've spent the rest of my life trying to recreate that simplicity.

My first published work on pen-based input was in 1978, and we were working on multi-touch technology by 1984. This long history in the field has made me realize the importance of getting things right. We are on the cusp of a major change, with a new generation of technologies about to launch. The game is changing from simply making things work to creating smooth, elegant experiences. The challenge now is not just about making things, but deciding what to make and how to make it.

People often say that being at the beginning of a new technology or trend is advantageous because there's a lot of low-hanging fruit. But being first isn't what matters. What matters is having the commitment to see your ideas through, even if it takes 20 years.

Now, let's talk about the concept of beautiful apps. What does it mean for an app to be beautiful? We often say that beauty is in the eye of the beholder, but this phrase is biased towards visual aesthetics. As a musician, I find this offensive because beauty isn't just about what we see. The language we use shapes how we think and approach things, including making apps and starting businesses. So, let's broaden our understanding of beauty and apply it to our work.
In the realm of interaction design, the best work often goes unnoticed. This is because when a design is executed brilliantly, it becomes so transparent that it's almost invisible. I like to refer to this as "out-of-your-face" design, as opposed to interface design. It's elegant, it's seamless, and it's just there. People are delighted by what they're doing and they might even say, "Wow, that felt great." However, your work may not get featured in a design magazine because it doesn't photograph well. Design, after all, is often about looking cool in stylized photographs.

In the traditional design world, the focus is often on the visual aspect. But the true beauty of a design lies in the experience it provides. If you want to create beautiful apps, you need to consider not just the visual aspect, but also the feel, the smell, the hearing, the kinetics, the emotions, the culture, the mind, and all other factors that contribute to the user experience. Before we even begin to discuss beauty, we need to understand its dimensions. What are the aspects that everyone else is forgetting while focusing on creating something that looks gorgeous?

The essence of a design lies in its experience. For instance, if you give a flutist two flutes that look identical, one being a student flute and the other a beautiful instrument, you could tell which one is the better flute just by watching the person hold it. The same goes for a violin. You don't have to play a note. Just pick it up, feel the weight, feel the action, and you'll see immediately whether the person caresses it or merely tolerates it in their hands. Does the instrument show contempt for the hours spent learning to master it, or does it show respect and attempt to make every hour pay back with a factor of two instead of one?

In essence, beauty lies in the experience of the beholder. The experience is the composite takeaway from all the factors mentioned earlier. The challenge then is how to incorporate this understanding into our design process.

This brings us to the topic of design versus design language. There's often confusion between these two terms, not just in the Windows environment but in other environments as well. Words are important. If we want to change the culture of thought about experience design within our organizations, we need to start by developing a common vocabulary. Cultivating a particular kind of culture within an organization can be achieved through vocabulary and language.

Let's consider English. English is a language, but was it designed? No, it evolved. Now, let's consider C-sharp, a programming language. Was it designed? Yes, it was. It evolved from C, but it was an explicit reformulation. The lesson here is that languages can either evolve or be designed. Both are acceptable. The notion of a language being designed is something we understand. It doesn't have to be that way, but it can be.
In the realm of user experience, we often encounter various interaction languages. Some of these languages are meticulously designed from scratch, while others are assembled from existing elements, evolving over time. They are codified, or standardized, in a way that may seem less designed, but they are defined by this codification. This is the language we use, and this is how we use it.

Consider this: just because a statement is grammatically correct doesn't necessarily mean it's true. A statement can be well-formed, yet still be false, culturally inappropriate, or even a lie. The untruth of a statement can have consequences that extend beyond its immediate semantic context. This principle applies not only to everyday language, but also to design languages.

Take, for example, the sentence, "Yates was an Irish poet who won the Nobel Prize in 1923." This is a true statement, expressed in proper English. However, despite its truth and grammatical correctness, it is not poetry. It is merely a utilitarian statement of fact. This is similar to a lot of interaction design, where designs are often created within the vocabulary of a system.

Now, consider this excerpt from Yeats' "The Magi": "Now as at all times I can see in the mind's eye, in their stiff, painted clothes, the pale unsatisfied ones appear and disappear in the blue depth of the sky..." This is also English, and it is well-formed. But it goes beyond a mere factual sentence. It is poetry, filled with implicit truths that require deep analysis to fully understand.

Understanding the difference between beautifully articulated, factually useful information and poetry is crucial. If we don't understand this difference, how can we aspire to elevate our language beyond its utilitarian function? You can't create a beautiful app unless you understand these differences and know how to apply them.

The lesson here is that while one can write poetry in English, not all English is poetry. Furthermore, not all English is true. This is a key point to remember.

Consider another example from Yeats, this time in Irish. Even though it's not strictly English, it can still be considered great English language poetry. This shows that you don't have to be confined by the strict rules of a language. You can augment the language, inventing new words that fit the context and enhance the point you're trying to make. For instance, I often use the word "skeptimus," a term I invented to describe someone who is half optimist, half skeptic. Using such a word, even though it's not officially recognized, can make your point more memorable because it deviates from the usual language.
In any language, including interaction language, it's important not to take the guidelines too seriously. There's always room for flexibility and innovation. For instance, consider the concept of "trustification," a term I coined while working in telepresence and video conferencing.

Despite the initial skepticism towards video conferencing, with many claiming it didn't improve productivity, I argued that its value lay not in the bandwidth of communication or productivity, but in the ability to read body language and determine trust. It's about understanding the emotional state of the person on the other end of the line. If you fail to recognize this, you'll end up measuring the wrong things.

This brings us to the evolution of language. Just as people working in C started to build functions and realized they should be fundamental primitives in the language, we can also introduce new vocabulary and evolve the language. This is how progress happens. The words you invent can become part of the language almost instantly.

However, there's a caveat. When you introduce new words, initially people might not understand what you're talking about, which can be disruptive. But with the right context and body language, you can assert the meaning. The key is to ensure the meaning is consistent with the conversation at hand.

This brings me to an article I read recently, which criticized the use of skeuomorphism in design - the practice of making e-books look like paper, for example. The article argued that this was a misuse of new technologies. However, I believe this argument is flawed.

Yes, there's a risk of bad design if you take things too literally. You could end up designing an automobile that drives like a carriage. But it's important to differentiate between skill and what that skill is applied to. If you simply emulate what you were doing before with new technology, you risk falling into the trap that the article warns against.

However, the article fails to recognize that acquiring a new skill is extremely expensive. There's something called the power law of practice, which states that skill acquisition is a power function, not a linear one. To become highly skilled at something is really, really expensive. Therefore, it's not always wrong to use new technologies to utilize existing skills.
In our profession, we must be cautious not to design interfaces that require fundamentally new skills when existing skills could suffice. This is because we're imposing a cost of use on the system, which takes time and effort away from the primary purpose of using the system.

The design we create should aim to accelerate the process by which novices can perform like experts. This doesn't mean making you a better architect, for example, but rather making the interface more user-friendly.

The difference between emulation and innovation in this context is how literal the transfer of skills is. Skilled transfer, or taking an existing skill and applying it in a new context, is much cheaper because the only thing you have to learn is that the skill applies in this new context. This reduces the cost and accelerates the acquisition of expertise on the operational aspects of using your system.

If there's a one-to-one correspondence, the same skill for the same task, then we're in the realm of skeuomorphism. But if there is a gap, the wider the gap between the skill and where it's being applied, the more innovative it is.

Consider this example: everyone has a strong response compatibility between pressure and speed. If you've driven a car, you know that the harder you push, the faster you go. We've been able to put force-sensitive resistors under the left mouse button for pennies since 1982. Yet, we still can't point at the scroll arrow and push, and the harder we push, the faster it scrolls. This is absurd.

The reason this hasn't happened isn't because it's hard, but because we haven't realized that this was useful or that it was possible. This isn't skeuomorphism because we've never done that before. However, it builds upon a well-established existing skill and applies it in a new context, which could be considered an example of innovation.

The key is to always look for ways to save users from having to learn something new. There are times when it's worthwhile developing a totally new skill, but it's also important to leverage existing skills where possible.

The critical thing is to avoid getting sucked into debates and instead focus on understanding what's actually going on. It's a design problem that requires understanding the merits of both sides and figuring out on what it depends.

When we start looking at the interaction language of a system, we need to understand what makes it different. In music, there's a term for this: iron. All of you play an instrument, right? That's the essence of the language.

By the way, I'm a Doyle. My grandmother's a Doyle, so I have some legitimacy here. Passport. That's right.
When we examine the design of live tiles, we can see that they are more than just graphically altered icons. They may appear as simple squares on a grid, but there's more to them than meets the eye. The concept of live tiles introduces a new context to the use of icons, a concept we've been familiar with since the days of Dreyfus and his book on symbols.

However, the use of graphics in live tiles can sometimes be overemphasized. While graphics are important, they can also lead to confusion. For instance, when live tiles are filled with photographs, it can be difficult to quickly understand what each tile represents. The shape of the tile doesn't provide much information, and the graphic design of the tile needs to be distinct enough to stand out from the others.

Consider two tiles filled with photographs. One represents stocks, the other a newsfeed. Without any context, it's hard to tell them apart. The stock tile uses news images to reflect the state of the stock market, but without this knowledge, the images could be confusing.

There are tiles that use traditional iconography, reminiscent of Windows 3.0 or the Uri Macintosh. Some tiles are sparse, not fully filled, creating a pattern that catches the eye. These tiles are interesting and iconic, demonstrating the skill of the graphic artist while still maintaining the live tile concept.

The design constraints we work with are not dictated by the design language of Microsoft or Apple, but by the limitations and potential of the users' motor, sensory, and cognitive skills. The design process starts with the design language, but it doesn't end there. The final product needs to make sense to the user.

It's important to differentiate between a design and a design language. Having a consistent design language across products doesn't necessarily mean the design is good. Just as using correct English doesn't guarantee the truth or quality of a statement, conforming to a design language doesn't guarantee a well-designed product.

My composition teacher in music school taught me that you need to learn the rules before you can break them. This applies to design as well. Most rule-breaking is done unintentionally, leading to unpredictable and uncontrollable outcomes. Understanding the rules gives you options and allows you to make informed design choices.

So, how do we know if a design language is well-defined enough to allow for good design? I would argue that a design language should never be fully defined, as this would limit creativity and make the design process overly complicated. When the Macintosh first came out, its interaction design language was largely informed by what was going on at PARC and with the Parc Star. This allowed for a degree of freedom and creativity in the design process.
The original Macintosh team didn't copy verbatim or use universal operators, but there were many similarities with previous systems. However, the user interface guidelines book wasn't published until after the Mac was released. The team did two significant things.

Firstly, they built a toolkit, known as Mac App, and other development tools, to exploit the path of least resistance. This means that they designed their tools, libraries, and documentation to make it easier for developers to follow a preferred path. The idea was to make it easier to do things the right way, rather than the wrong way, and thus guide developers down a preferred path.

Secondly, they developed applications like Mac App, Mac Paint, and Mac Write. These applications were somewhat useless at the beginning, but they served as templates for developers.

I applied this approach when I developed a product called the Portfolio Wallet. This product was designed for automotive studios and had to be a walk-up and use interface. We built a behavioural prototype that behaved with the flow and action we wanted, even though the code behind it was unusable.

We also gave a specification in human terms. The specification was that any employee of General Motors, from the president to the janitor, had to be able to learn 80% of the functionality in 3-5 minutes with a 90% retention a week later. If this wasn't achieved, it was considered a stop-ship bug, even if every line of code was correct and every function was implemented.

This approach changed behaviors. It wasn't about ticking off a list of features. Developers are ambitious and creative, and if you reward them for adding features, they will add more features that you never asked for. They will design these features themselves and they will appear in the code because they have the power to do it.

Developers are very good at developing features, but providing access to those features through the interface is the challenge. If the person developing the access doesn't fully understand the features, they can't develop the access to the features.

In my experience, if you study computer science, you want to assert your expertise and creativity by how elegantly you can program. The designer should work closely with the developer, like a structural engineer and an architect, to create the best product.

When you set the specification as a challenge, engineers want to beat the spec. In the case of the Portfolio Wallet, the engineers did everything I asked for and more. They proved their skills by recognizing relationships, eliminating complexity, and beating the spec.

The language you use, the examples you provide, and what you consider a spec are all crucial. This approach may not work for something on the scale of Windows, but it worked for Sketchbook Pro, a product developed by my team.
On July 1st, we embarked on a mission to create a product for the launch of the tablet PC. We had a tight deadline, aiming to ship by November 7th of the same year. This meant we had to go from having no team to shipping a product in just over four months. By October 1st, we had the product ready, thanks to a focused approach and the use of prototypes. This process was unlike the one used for products like Office or Windows, but it was similar to the process used for apps on systems like the tablet PC.

One of the key strategies we used to speed up the process was to draw from our past experiences. We didn't start from scratch. We realized that we could learn a lot from the use of live tiles in film. The first use of live tiles was in the 1966 film Grand Prix, where the screen was split into different tiles. This technique was later used in Expo 67, the Montreal World's Fair, where it was used to tell stories and convey messages effectively.

In our profession, stepping out of our comfort zone and exploring other areas can be both entertaining and enlightening. For instance, watching films that use live tiles can be a great way to learn and get up to speed. If I were developing apps, I would probably organize film screenings on Friday afternoons or encourage everyone to watch as many films as possible.

One of the films I would recommend is the first Thomas Crown Affair, directed by Norman Juisson. Juisson was inspired by the films he saw at Expo 67 and used the same technique of splitting the screen into different tiles. This was done using optical printing, long before digital technology was available.

Looking at these live tiles in the context of visual language, I realize that there's nothing new under the sun. Everything is just a variation of something older. By going back to these older techniques and learning from them, we can broaden our experience and bring more to the table.

This is just one example. There are many other aspects, like touch and gesture, that we can learn from. The internet is full of examples if we know where to look. For me, eBay is a valuable development tool. I have a large collection of gadgets that I bought for a small price. Each one is a working prototype that I can pass around for people to experience and learn from.

In conclusion, there's a lot we can do if we're willing to learn from the past and adapt to the present. Just like in music, where you have to change the arrangement of a piece to suit different instruments, we have to adapt our techniques to suit different platforms. The key is to understand the nuances of each platform and to pay attention to the details.
The stylus presents a significant opportunity in the realm of technology. While I am a proponent of touch and multi-touch technology, having published the first paper on multi-touch in 1985, I am aware that it is not the solution for every situation. Each technology has its strengths and weaknesses, and the idea that one interaction technique can cater to all needs is simply not true.

Consider the Windows 8 style, formerly known as Metro, and its application on mobile phones. If you were to use the touch and tiles interface while driving, it would be a dangerous distraction. That's why we developed Sync, a system that changes the interaction language to speech activation when you're driving.

Imagine being on a call, receiving a text that is read out to you by your car, and then responding to it, all without touching your phone. When you park and leave your car, the call is seamlessly transferred to your handset, and the interaction changes back to touch and tiles. This seamless transition, despite a 90% change in the hardware supporting the call, allows for a consistent conversation.

The key is not consistency in user interface, but rather in the conversation. The interaction style may need to change completely and seamlessly depending on the situation, especially when using a slate that isn't anchored by a large processor and plugged into a wall.

Now, let's talk about the stylus. I want to show you an example of a product that was released around the same time as Windows 3.0. This product, Wang Freestyle, allowed you to annotate any document with your pen, record your voice, and synchronize your voice with what you're pointing at and marking. You could then send this as an email.

Imagine being able to review a business plan, expense report, novel, or artwork, and send your comments as if you were looking over the recipient's shoulder, even if they're across the continent. This capability is still not available today.

For business purposes, the ability to annotate existing documents, images, videos, spreadsheets, novels, artwork, and more, and then send it, could be incredibly valuable. This would require minimal bandwidth, as you're only sending a still image, mouse data over time, and speech.

Imagine being able to review and mark up reports from the comfort of your couch, just like you would with paper, instead of having to print everything out. The tools for this have been available since 1990, yet we still can't do it, despite having about 100,000 times the computing power today.

This is an example of how we can learn from the past. Many technologies failed not because they were bad ideas, but because the timing wasn't right. I believe that a product like Wang Freestyle could be successful today.
Imagine this: you're on a ski trip and you've just come down a thrilling slope. You take out your Windows slate and snap a picture of the hill. You mark up the photo, indicating where you started, the path you took, where you tumbled, and where your friend Buxton nearly went over a cliff. You then narrate your adventure, pointing out the marked spots on the photo, and send it off as a rich image via email.

Now, developers, wouldn't it be great to have a tool that could do this? And wouldn't you use such a tool if it were available? This is just one of the many ideas I've come up with.

But let's talk about patents. They last for 20 years. So, if a patent was filed in 1990, it would have expired in 2010. Yet, no one has capitalized on this expired patent. I don't mean to trivialize the issue of patents. I've worked as an expert witness on a case and have seen how complex patent law can be. But as a developer, you shouldn't be worrying about that. Your focus should be on getting your ideas out there.

It's all about cost-benefit analysis. If your idea is valuable enough, it's worth the legal costs. If it's marginal, then it's not worth the effort. But you need to delve deep enough to determine the value of your idea. That's where sketching techniques come in handy. They allow you to explore different possibilities without spending too much.

Remember, it takes 20 years for an idea to go from inception to a billion-dollar industry. Most of that time, the idea is under the radar. Any idea that will become a billion-dollar industry in the next 10 years is already 10 years old.

Now, let's talk about a concept I've worked on extensively: the radial menu, also known as the marking menu. It's a pop-up menu where you make a choice based on the direction you go, not the distance. No matter how many times you've used a file menu, you'll never be able to differentiate between 'save' and 'save as' or 'print' without looking. But with a radial menu, you can easily select options in the same way you can point to the eight points of the compass without looking.

This type of interaction is especially useful in touch and pen-based devices, which are becoming more prevalent. It's a simple pop-up menu, but the selection is made by direction, making it more intuitive and user-friendly.
Every time you make a selection, you're training your motor memory to learn the gesture. The key is to delay the pop-up menu. If the menu pops up immediately, people will read it and it will slow them down. However, if there's a slight delay, you have to push and hold, and then the menu comes up for you to make your selection.

Once you've learned the gestures, you can simply perform the stroke and it becomes a gestural interface. This accelerates the path for a novice to behave like an expert, much like training wheels. The menu is familiar, but it's tailored to make the selection based on your biological, neurological, and cognitive wiring, unlike any other linear menu.

This allows for a seamless, smooth transition to start using gestures, depending on what you frequently use. For example, when you stroke to the left and the right to switch forward and backward on images, you're using this kind of menu. But because you've memorized it due to good stimulus response compatibility, the menu never even appears.

This is a radial menu, where you've mapped west to go backwards and east to go forwards in the slides. It can also be hierarchical. For instance, if you're choosing groceries, you go up to groceries, then when you're getting groceries, you select meat, bread, staples, miscellaneous, or junk. If you know where the fruit and vegetables are, you can simply do the stroke and the menu never appears.

This can be done with your finger, a stylus, or a mouse, and it works on every platform. It's a speed improvement of almost ten times compared to pop-up menus.

For example, in Windows 8, if you want to type an "O", you can simply tap it. But if you want to do accents, this is a radial menu. If you go up to the north, you get an accenting, to the left, you get a circumflex, and to the left down, you get a tilde.

Most people don't know that this even exists, but it does. It's right there on the keyboard. Even if it didn't exist, you could invent it. You can augment the language and make it a larger part of the design.

There's also a hot virtual keyboard that uses the same thing. If you set it up right, if you do an upward stroke, you get the uppercase, so you don't have to use the shift key. If you stroke to the left anywhere, you get backspace. If you stroke to the right, you get space. If you stroke down to the left, you get return. This can save you time and improve your text entry speed by about 15%.

These are just some examples of how you can incorporate these features into your designs. It's all about paying attention to the details. For instance, in a sketchbook, you can draw with pretty nice ink. You can come to your palette on the Lagoon, and the menu just comes like that. So red's to the right, black's up. You can easily switch between black and red without having to look.
In this lecture, we will discuss the functionality of a pen-centric application that has been designed to work seamlessly with touch devices like the iPhone and iPod Touch. The application's ability to respond to pen pressure and control makes it unique and different from other touch-based applications.

One of the key features of this application is its ability to mimic the physical properties of paint and paper. For instance, when you apply paint to the virtual paper, you can see the texture of the paper. You can also mix different colors of paint, and the application will blend them as if they were real paint. This is achieved by using a blending brush tool, which doesn't leave any paint but allows you to smear the existing paint on the canvas.

The application also allows you to choose the materials of the paint and pastels. However, the level of control you have over these features depends on whether you're using your finger or a stylus. While the application works with your finger, it's hard to control the details. But with a stylus, you can control the pressure and create variations in lines.

It's important to note that capacitive pens, which are often used on phones and capacitive screens, are not styluses. They are simply proxies for your finger, and the application doesn't recognize them as a stylus. This means you can't assign a different function to them.

The application also allows you to use two hands to scale the canvas. If you bring your pen down, you can ink, and if you bring your finger down, it lays down paint. However, if the application can distinguish between a finger and a pen, you could assign different functions to each. For instance, you could use a single finger for the smear command and the paintbrush to lay down paint.

This application is a great example of how pen and touch can work together. It provides a wonderful example of graceful degradation, where you can drop tools to the side but still maintain functionality. It's not about choosing between pen or touch, but about how they can work together to create a seamless user experience.

In conclusion, the beauty of this application lies in its ability to provide a multi-sensory experience. It doesn't rely solely on haptics for feedback but creatively uses non-speech audio to give users a sense of tactile feel. It also allows for the subtle use of pen pressure and sets reasonable defaults when you're just using your finger.

The application also has great speech recognition, allowing you to change tools and colours without any interface on the screen. This leaves the entire canvas available for working. The key takeaway here is not to eliminate any sensory modalities of the human experience, but to consider them all and then explicitly eliminate those that are not necessary.
Consider the value of having everything available when it might be beneficial. You don't need to include all features in the first release. Instead, plan for two or three releases ahead, giving yourself a path to evolve and incrementally improve. This approach allows you to prioritize, as you need to generate revenue to fund research and development for the next generation.

Don't limit your thinking to just one product ahead. Build in the capacity to incorporate future developments. If you're thinking long-term, consider how the market has evolved. For instance, the iPhone was released in 2007, and handheld devices have since transformed our business. Now, we're on the verge of slates coming into their own. In five years, we can expect even smaller devices to become prevalent.

I've been working with Jeff Hahn from Perceptive Pixel on wall-mounted displays and how these devices interact. If you want your business to last, consider the ecosystem in which your products will exist.

I brought the 2002 version of Alias Sketchbook Pro to demonstrate that a product created by a team of five people in five months is still relevant today. Your products should be designed to remain relevant in the ecosystem you anticipate will exist in five to ten years. Plan for success and longevity, not obsolescence. Build in the capacity to evolve and bring your customers along with you.

Understanding the fine details, such as the difference between a pen or even what the difference is amongst pens, is crucial. When I buy a slate, I consider whether it has the N-Trig digitizer and stylus or the Wacom stylus. They're different, with subtle differences, much like buying a Yamaha saxophone. The artistry lies in exploiting these properties.

In projects, we use personas, but it's important to remember that you can't rely solely on user groups or focus groups for design. Steve Jobs once told me that he doesn't talk to users, he talks to markets. There's a larger audience there.

You need to understand both markets and individuals. It would be unwise to generalize based on analysing a single individual or a small group. Personas have some value, but it's also important to consider the context in which a product is used.

One of my favourite design books is Henry Dreyfuss's "Designing for People." He was the first person to write about personas, long before Alan Cooper. I highly recommend getting the first edition of this book for its beautiful typography and paper.

In addition to personas, consider the concept of "placona," a term I coined. It refers to the notion of situated computation, where the design is influenced by the social, physical, and cultural context in which it is used. This affects the design constraints. So, when planning a product, consider not only the persona but also the placona.
The crux of my argument is that the significant shift in our world is not due to technology. Yes, technology has become smaller, faster, cheaper, and more abundant, but it's essentially the same as it was in the 80s. The real change lies in the human, cultural, and social aspects of our lives. The key questions are: who is doing what, where, when, why, for how much, and with whom?

This perspective shifts the focus from a technocentric approach to a more human-centric one. It's all about transitions and changes. During any transaction, the people involved change, the physical location changes, the technologies change, and the ecosystem changes. We need to adapt to these changes.

I am delighted to have had the opportunity to share these thoughts with you. I look forward to returning and continuing this discussion. For those interested in delving deeper into these topics, my website is a treasure trove of information. I also have a YouTube channel, WAS Buxton, where you can find a plethora of videos, including the one I showed today. Feel free to explore these resources and reach out to me if you need further assistance. Thank you for your time and attention.

Functional Testing: The Unsung Hero of the Testing Spectrum?

2023-12-11T04:00:00+00:00

Functional Testing is like Eagon Spengler, the brains behind the Ghostbusters team, often overshadowed by more flamboyant characters but indispensable for their success. That accomplished yet humble individual belies the very essence of functional tests. Just like him, functional testing plays a crucial yet understated role in the software development process.

In the world of software testing, we are so often enamoured by the technological characters: UI testing with its flashy complexities, API testing with its mysteries of wire protocols and data payload formats. Yet, the quietly diligent and most dependable hero - functional testing - is often left in the shadows.

The Gentle Giant in the Testing World

What converse do API and UI testing share with functional testing? Why it's none other than their intrinsic focus on technology and implementation details. Functional tests, however, march to the beat of a different drum. They elegantly waltz in the realm of domain logic, taking scant note of these implementation trivialities.

The primary intent here, is the application's business functionality. In fact, this type of testing seems so easy-going, that often, the tools and terminologies utilized in conducting functional tests are easily comprehensible even to customers and domain experts.

Who Authors Functional Tests?

Greater surprises await! Functional tests, in certain cases, may be scribed by domain experts! Could a greater twist of irony exist in the software testing world? And quite unlike their showy kin - the unit tests, which are notch above in their coded form - functional tests have the liberty of non-programming forms!

No, I'm Not Rebranding!

Now, before you stir from your seats, behold, dear readers - I am not preaching a new mantra. Whilst it's true that functional testing is also known by the sobriquets of ‘acceptance tests,’ ‘customer tests,’ or ‘executable requirements,’ I find the term ‘Functional Tests’ is a better moniker. For, it is - plain, simple, and tells it how it is. Functional Tests focus solely on domain functionality. Need I add more?

Often, one must tread cautiously while referring to price specs of a domain, lest one falls into the yawning pit of vocabulary tangles.

Your Trusty Functional Testing Tools

I like singeing my marshmallows till they are golden, not burnt to unsightly cinders. Thus, to better equip you on the smoky battlegrounds of testing, I gift you your trusty swords: Cucumber and Fitness.

Equipped with these panaceas, you can now traverse the rugged terrains of an application’s functionality sans any programmatic writing. While Cucumber comes with the Gherkin syntax, a popular name in the BDD (Behaviour-Driven Development) school of thought, Fitness offers a more tabular expression to test requirements.

Interestingly, both Cucumber and Fitness are, but, two of many. Seek, and thou shall find many more friendly tools in this cadre. All have one shared vision - to ease the customer’s interaction with, and creation of, functional tests.

The Caveats

However, beware, there are limitations to even the best weapons of trade. Just as the One Ring forged in the fires of Mount Doom promised supreme power, it also held within it a deadly will of its own.

Functional Testing is no exception. This approach to testing may seem the best course given its strengths and popularity, and yet, adoption and buy-in can often seem monumental. Wooing customers and domain experts and getting their commitment becomes an inevitable part of the process!

A Tough Climb, but Absolutely Worthwhile

Maintaining an extra layer of code to wire the expression language of these tests to the code under test may seem hard, but let's be real. Functional tests offer no more than what unit tests would. It’s true; it’s an additional technology to comprehend, a learning curve to be surmounted. However, I can guarantee that there are few other ways to ensure your complex and vital business logic remains steadfast and functional!

So, the climb up the Functional Testing peak may seem steep, but let me tell you dear readers – the sublime vista from the peak is unmatchable. It's an unparalleled sense of accomplishment once you traverse this rocky trail. And at the end of it all, you’ll have bulletproof, effective, and reliable software ready for the market.

Demystifying API Testing

2023-12-06T04:00:00+00:00

In the realm of software systems, the concept of application programming interfaces (APIs) is commonplace. Often, you'll find services or components in these systems that are deployed with over-the-wire APIs. But, what do these APIs entail? Are they exclusive to backend services instructed by various user interfaces via some remote access protocol such as HTTP? Not quite. The term "API" may typically be used in this context, but it isn't necessarily limited to it. In this exploration of API testing, we are defining an API as a remote backend service that is an integral segment of a software system.

So, what exactly is API Testing?

API Testing, in a nutshell, involves examining the test surface of an API, which is usually a deployed service layer interface. This testing schema has evolved over the years and it predominantly involves transmitting HTTP requests to a deployed service and subsequently parsing the HTTP response received. The essence of this process is to ascertain and verify that the payload returned aligns with your expectations.

Conventionally, such APIs will have real user interfaces that call upon them. However, in API testing, this layer is bypassed in the quest for more granular testing. But is this approach what service tests, as depicted in the middle part of the test pyramid, represent? Let's clear that up.

API Testing and the Test Pyramid

According to Mike Cohn, the creator of the test pyramid model, service tests pertain to logical business services. His view of service tests does not outwardly clarify that they are deployed services tested over a remote API. In other words, service tests bring to view a broader breakdown of code that provides a full business function to the application, instead of restricting to just single class or method testing as in unit tests.

However, terminology confusion arises in the tech industry when certain words carry multiple meanings based on context. 'Service,' in relation to software development, is one such word, and unfortunately, the middle layer of the test pyramid succumbs to this semantic trap.

Automated Tests with API Testing Tools

The industry is inundated with several tools that have been designed to write and execute automated tests against a deployed services API. An example is Postman, a tool that houses saved HTTP GET and POST operations and allows tests of varied kinds to be written on the responses of these operations. This helps to confirm either the HTTP status code in the response or perhaps, the response body.

Consider an HTTP GET call to a prescription application - you can order a retrieval of prescribable medications from the API and get the results immediately. If you desire, you can even design a test to scrutinize and verify the expected values in that response.

Most of these API tools require subscription for sustained company usage. Alternatively, unit tests using a network or wire library can be written to test an API in place of a dedicated tool like Postman. However, it's key to note that the choice of tool depends entirely on the tester's preference and comfort.

The Purpose of API Testing

The specific objectives that necessitate API testing are often for regression and smoke purposes - a term unseen till now but will be discussed soon. Essentially, automated tests against an API are predominantly configured not to aid in the construction of new builds but to detect issues in prior work. This form of testing, like a deployed API, has an inherent test surface targeted.

On the technology front, API testing is a highly technical type of testing that doesn't generally require dedicated coding sessions, but demands familiarity with protocols used for sending requests and receiving responses from the API. Currently, REST over HTTP with JSON data as the payload stands as the popular choice. Tools like Postman are widely-accepted in this space, but a multitude of other options are equally available.

The Pros and Cons of API Testing

API testing offers optimum leverage when the product in question is a service. In this scenario, there's no UI that the team builds or controls - their task is solely to create one or more services that other teams or products then utilize. Such a team will most certainly incorporate automated API testing.

Furthermore, if the application utilizes a multi-service or micro-service architecture, API testing provides a substantial benefit. Another instance where API testing triumphs is when the service layer comes with its inherent complexity, complete with logic built into it as a part of the configuration rather than the deployed code. An example is an API gateway, commonly used in modern times as a front-end for services with complex features that require extensive testing - a task aptly handled by API test automation tools.

On the flipside, API tests can be slower compared to unit tests due to the myriad components being tested "over the wire" in a live service environment. As such, API testing might not be the best choice for testing the domain logic implemented in the code behind the API. In these cases, unit testing or functional testing can be a superior option.

The varied facets of API testing – including technical, testing tools, methodology, and pros and cons - show that there are numerous factors to consider when selecting the ideal approach for your system. This reinforces the importance of thoroughly understanding your system architecture and capabilities to ensure optimal testing practices.

Converging Test Automation and UIs

2023-12-05T04:00:00+00:00

Software testing has undergone a dramatic evolution over the years. It has metamorphosed from mundane, manual tasks to a truly dynamic and automated process. Specifically noteworthy is the potent concept of automated testing via the user interface(UI) realized through the advent of profoundly adept modern technologies.

In contrast to traditional unit tests that primarily perforate lines of code and evaluate their intended functionality, automated tests foster interaction and inspection of the application in its full glory. They robustly scrutinize the complete assembled system via its UI. Consider the tests as an intrepid globetrotter, navigating through the labyrinth of web applications, smart devices, virtually or physically, Android or Apple. The main idea is to simulate real user interactions and accurately replicate user experience.

Running Automated Tests: A Game Changer

I personally categorize these automated tests as "through-the-UI tests." Their enactment necessitates the entire application to be deployed, distinguishing them from unit tests that target only the UI code. Widely acknowledged industry jargons like end-to-end tests often sum up their essence adequately, albeit not in all circumstances. We must keep in mind that some applications exclusively function as a service and technically lack a UI, hence an end-to-end test may not typically encompass these.

Automated Testing Case Study: An In-depth Analysis

For example, we'll use a web application underpinned by a Java backend simulating the process of prescribing medications. This application features a specific rule limiting the prescription duration of certain classes of medication such as opioids and benzodiazepines to no more than thirty days. Any attempt beyond the predefined interval triggers a warning message, effectively blocking the prescription process.

A running instance of this application is subjected to an automated test of this business rule via a URL. The test utilizes driver libraries, provided by Selenium, for automated browser interaction, with JUnit facilitating the test execution. Contrary to what one might infer, these tests are not unit tests, but web driver executions, literally interacting and interpreting the UI components like a real user would.

The Challenges and Evolution of Testing

How does automated UI testing fit into the proverbial "test pyramid" that we often refer to? You may wonder: "Should the majority of tests focus on the user interface?" The answer may surprise you.

Firstly, the landscape has changed monumentally since the inception of the test pyramid concept. The proliferation of devices and diverse screen sizes amplified the test space exponentially. The reality of today’s applications being accessible from desktops, tablets, and smartphones is a daunting challenge by itself. This Herculean task is further amplified when applications spawn multiple native front-ends for various smart devices. Enter modern cloud vendors. They provide a gamut of services, including virtual and physical devices and parallelism, to cater to our diverse testing needs.

Real-World Impact and Value of UI Testing

Automated UI tests offer some undeniable advantages. Yes, they might be slower than conventional tests, but they comprehensively evaluate large parts of the system under test. For an application supporting multiple device types, they are indispensable. This approach may seem overused and arguably not the perfect fit in all scenarios, such as when focusing on complex back-end business logic. But, the use cases where it delivers invaluable insights are substantial.

Seamlessly blending technology and strategy, automated testing through the UI is a compelling paradigm shift. Just remember: deliberate, purpose-driven testing always prevails. From unravelling nuanced insights about your application to enhancing the user experience, automated UI testing etches a place for itself in the pantheon of software testing approaches.

The Role of Unit Testing in Modern Software Development

2023-11-29T04:00:00+00:00

As we glide forward on the cutting edge of the technological era, software development practices mature and evolve. Among the myriad of practices that mark the life-cycle of software development, unit testing forms a cornerstone. This indispensable process, intertwined with the act of writing code itself, has been fundamental in shaping this industry.

Reflecting on the incipient stages of my career, at the dawn of the digital age, I recall a time when the concept of unit testing was nascent, untapped, and devoid of the supportive tools or libraries that we now take for granted. The process, though cogent in theory, was far from being systematized. Diligent developers would carefully write small code snippets, run them to make sure they functioned as desired, and repeat this process incrementally. It was a time-consuming and monotonous task, demanding exactness, but it was critical for assuring the essential functionality of the code.

The Arrival of Unit Testing Libraries

Fast forward to the digital landscape of today, and the vision is altogether different and far more dynamic. Today, we have access to a plethora of mature, feature-packed unit testing libraries. These tools empower developers to swiftly build test harnesses and progressively test their code with a simple push of a button. The crux of modern unit testing is the ability to test the smallest changes to code in a matter of milliseconds. This instantaneous feedback mechanism is an effective sentinel, keeping a constant check by validating established expectations or detecting new issues.

The evolution of unit testing tools has been nothing short of revolutionary—an epoch-making occurrence, if you will, that has indelibly transformed the face of the software development industry. It would be no exaggeration to state that the advent of these tools signals one of the most significant advancements in tech history. In fact, I'd go as far as to say that I'd rather regress to the traditional method of manually encoding instructions into a computer—through the arduous task of punching holes in cards—than forfeit the luxury and simplicity offered by unit testing libraries.

Not Just for Developers

It could be easy to fall into the trap of viewing unit testing as an exclusive realm of developers. However, this practice extends beyond the confines of the developer's world and into the realm of professional testers. It's worth noting that any endeavour towards testing, unit testing included, feeds into the broader mission of refining the product quality. Thus, the effectiveness of a team's unit testing strategy is an area focused upon by the entire team, including testers.

Adept testers, with a keen understanding of which parts of the system are adequately covered by unit tests and which parts aren't, can identify and prioritize areas where additional testing activities may be needed. Unit tests can also serve the dual purpose of documentation and offer valuable insights into how a system is implemented—a resource that a tester can leverage. Additionally, unit test libraries are often utilized for writing and running various types of test automation.

Demystifying Unit Testing with an Illustration

To shed more light on the topic, let's explore a unit test for a portion of a medication prescribing application. This application, written in Java, also uses Java for its unit testing—this use of the same language for coding and testing is a common practice.

Each method within this unit test is a self-contained test. They all collectively aim to validate that a code segment, called the duration parser, accurately converts a string describing a certain number of days into an equivalent integer.

If I were to make any changes to the method in question, I could easily rerun these test cases to ascertain if anything has been disrupted. If so, I would receive instant feedback and could either revert the change or attempt to remedy it.

Now, say I required a new feature. I could create a new test case that encapsulates and tests the feature before it's been implemented. This practice, dubbed "test-driven development", entails cycling between writing tests and writing code, with the tests providing guidance at each step. This approach serves as a roadmap for developers throughout an entire feature.

Unit tests can sometimes be quite granular, which can present a challenge when trying to discern the business functionality being tested. However, our example does give clear hints about the business functionality—the code we're testing is adeptly converting a text string portion into a prescription duration represented in integer days.

Unit Tests in Ideal World

I would say, that unit tests serve as a form of living documentation for the code they test. They are an essential cog in the software development wheel, contributing to the overall quality of the product. They offer valuable insights for both developers and testers, showing us what individual codes are designed to do and highlighting how well they're performing.

Unit tests are designed to check a specific scope, ranging from a minuscule code segment to a whole subsystem. However, "true unit testing", the ideal form that we strive to achieve, typically zeroes in on a relatively small snippet of code. There are times when a unit test that works a larger section of an application may be labelled a ‘component test’.

I’d like to emphasize an important point - unit test tools can serve a much broader spectrum than just unit testing. Tools providing unit testing capabilities could be utilized to run an end-to-end test interacting with the user interface of a deployed system. Despite being outside the strict definition of a unit test, the unit test library can be of immense help in writing and executing these tests.

Do You Actually Need Unit Tests?

Well, my friend, that is a big question, deserving the most common answer in software development world – “it depends!”. Which also means – not always. Unit tests are not an axiom, and you should apply common sense and avoid implementing some paradigm for the sake of “doing it by the book”.

You may not be fond of “test-driven development” approach (which, let’s be honest, doesn’t work in most places), and argue that there is no point in having unit tests in functions which are tightly integrated with external components – that’s what we have integration tests for. But there is one thing, in context of modern CI/CD, which make unit tests valuable – tests, that involve compatibility with 3rd party libraries.

For example – let’s imagine, that you have a JSON serializing library, and some functions depend on it. What would happen, if that library would stop working for you? Your unit test of the function that uses that library would fail, that’s what.

However, unit tests are executed during the CI phase, not CD. Meaning – when your project is built, not when it’s deployed, and often we’d like to test, if our application works, after it’s deployed in test environment. Said library could be missing, or be incompatible with some system settings, and we want to be able to detect such failure before our testers start wasting their time.

And that’s where integration test project comes into play, and we often may find, that functions in unit tests would be duplicated in integration test. And that’s where “depends” comes from – you certainly need either unit or integration test, but the question is – whether you need both.

My suggestion is – if your budget allows, have both. Even if unit test will cover just 1% of your code, it may save you time by failing earlier than that code is deployed to test environment. However, in many cases you can resort to integration tests with higher degree of efficiency and satisfaction.

A Developer's Best Friend

Unit tests are primarily designed to assist in the process of development. Their significance is heightened during regression testing, where they help to pinpoint bugs that may have surreptitiously crept into pre-existing features. However, this is more the exception than the rule. The reality is that a unit test failure usually implies the developer is working on that piece of the puzzle and expects it to fail—it isn't often that a unit test selflessly reveals an unsuspected regression bug. Therefore, the scope and depth of unit tests typically mean they are an underling in the bigger scheme of regression testing.

The primary interface for true unit tests is the system's code. They uniquely interact with the system code sans the need for it to be deployed. The beauty of true unit tests lies in their cohabitation with the actual code in the system's source code repository. The language of the unit test library typically aligns with the code being targeted for testing—a Java application would see Java unit tests, a C# application would see C# unit tests, and so on.

Unit testing is synonymous with custom-developed systems; it forms an integral part of the ecosystem. While it isn't necessary to create a unit test for each line of source code, any bespoke system should have comprehensive coverage. The crucial segments of the system—those high-impact areas—are the best suited for intensive unit testing.

The reverse side of this medal is that writing unit tests consumes time, which consumes project budget, which could be used for developing actual features. However, they simplify the coding process significantly, when the stars align. That is – when the function is large and complex, and / or when developer is slow at that particular task. Maintaining unit tests does come with a price tag attached. Whenever significant changes are made to the production code of a system, such as redesigns or major movements, the unit tests must be updated to fit the new context.

As we progress, we'll be delving into a particular kind of test that, although typically written using a unit test library, fundamentally differs from a unit test: user interface or end-to-end tests. So stay tuned for the next chapter of this unfolding saga of software testing!