This is not the first time I’ve made this resolution and it often tails off into adhoc-ery all too quickly but I’m hoping that being intentional about why I am doing this will help it stick more.

To that end there are a few things that I’m changing in how I do my 1:1s this year, including:

1. Diarising time each week on Monday to prepare for my 1:1s. Here I need to admit that usually I’ve been collecting topics on post it notes throughout the week and then winging it through the 1:1 meeting, basically triaging on the fly the most important topics that I or my team member are bringing in that day. This means that we tend to deal with urgent things but not always important things.
2. Creating a template for each of us to complete in preparation for the meeting, and to help ensure that we regularly touch on important topics that might otherwise be overlooked by urgent things. More on this below.
3. Taking proper notes and actions in the meeting, and making sure the actions get acted on. Basic stuff really, the former wasn’t really happening and the latter could be a little hit and miss. Using the template as a way to build accountability for the actions.

With that in mind, here is my current iteration of the template I’m planning to use. I’ve used it only twice so far and already iterated it a little (hence the v2 in the heading) and I expect I will continue to iterate it more and more over time. So far its been pretty positively received, but it is still far, far from perfect. Feedback welcome.

The idea is that each week in preparation for the 1:1 my direct reports, who are mostly Research Managers, fill out this template and take the time to reflect on each of these items. I also have a section I need to fill in. We both contribute topics in addition to the standard items.

Template for my 1:1 Meetings

Date of the 1:1 Meeting

(My actual template is in Confluence and look lots better as it has lots of colourful emojis all over it. Annoyingly WordPress won’t play nicely for me with emojis here so I’ve prioritised making this accessible over giving a screenshot of the pretty one instead)

During the meeting I will then capture a tonne of notes as the meeting progresses (my background as a qual researcher has prepared me well for this!). These notes are shared on a Confluence page so that both of us can add more, annotate etc.

As soon as the 1:1 meeting is completed, I update the template with next weeks items, and the agreed actions captured.

I am a little bit concerned that the extensive ‘structured’ section might not leave enough time to focus on the more ‘urgent’ topics – especially given these currently tend to take up the entire time allowed right now. But, that is also somewhat deliberate, so perhaps that’s not a bad thing.

The 4Ls – an alternative approach I might use on a monthly basis.

When I was talking about this approach with my colleague Dom Price, he shared another 1:1 format that he likes to use. I’m not sure it would work quite so well for me on a weekly basis but I might experiment with using this every 4-6 weeks as I think it does provide a really different perspective on how people are doing in their work and how you might be able to help them be more successful.

Dom’s approach asks people to reflect on these four categories:

• Loved – what I loved doing this month
• Longed for – what I longed to be doing but was unable to find the time/etc
• Loathed – what I really did not enjoy doing this month
• Learned – what I learned this month

The idea is to try to support the person you are managing to increase the loved and learned, enable the longed for and remove the loathed.

I’m sure there are many more great frameworks from people who have given this a good deal more thought than I have. I’d love to hear what’s worked well for you and what you’d recommend.

Over the last seven months I have been working from home, remotely from my team. It has just been in the past couple of weeks that I’ve been able to come up with a way of describing a particular kind of lack that I’ve been feeling.

There are many things we lack (and gain) in working remotely, but this is one I’ve not considered before, and I don’t hear other people talking about it either.

I call it ambient reassurance. (Almost certainly the organisational psychologists have another term for it but I can’t find it!)

Ambient reassurance is the experience of small, unplanned moments of interaction with colleagues that provide reassurance that you’re on the right track. They provide encouragement and they help us to maintain self belief in those moments where we are liable to lapse into unproductive self doubt or imposter syndrome.

In hindsight I realise, these moments flowed naturally in an office environment.

Sometimes we seek them out in an ad hoc way – a conversation in the hallway about the thing you’re working on right now, a request for someone to quickly look at something and give a tiny bit of feedback, a tiny moan about something you’re struggling with.

Sometimes they are completely unintended – someone looks over your shoulder at something you’re working on, or gives you a few encouraging words as you enter or leave a tough meeting, or just happens to comment positively on something they saw you do recently.

It is possible for these to happen when we are all remote, but it takes more effort and intentionality. As a result, I think we experience much less of this ambient reassurance when we work remotely.

Concerned about disrupting people’s flow with messaging, we’re much less likely to send that tiny message of encouragement or positivity. Without the visibility of whether people are in focus mode or have a moment of availability for a small interaction, we keep things to ourselves. We only reach out and demand someones attention if it feels sufficiently important or well thought out.

So many of our interactions now are textual. More visible, audit-able, traceable. Interactions that make us think twice. Far from a reassuring smile across the room or a secret thumbs up from the audience.

Getting the balance right is hard. Protecting our colleagues work time flexibility and their focus time helps deliver some of the advantages of working remotely.

And yet, in the absence of these tiny, human interactions, we’re more dependent on our own, individual self assurance. I never realised, until COVID and this long stretch of remote work, how dependent my self assurance was on ambient reassurance from others. In its absence, the natural peaks and troughs we experience – from confidence in our abilities to despair that we will never be good enough – feel more frequent and more extreme.

So knowing this, I experiment. With reaching out and sharing more than I might otherwise. Both about what I’m working on and how I feel about it, and also, with micro reassurance for others. I worry about the extra load I might be placing on others. And I ponder how our tools might take this new (to me) need into account as well.

Meanwhile, we muddle through. And I wonder if you’re experiencing this absence of ambient reassurance as well?

So, here’s a little reassurance for you right now – whatever you’re working on right now – you’re almost certainly doing better than you think you are. Don’t be so hard on yourself. Don’t be afraid to reach out for some feedback or just plain reassurance. Keep going! and stay safe.

If you found this interesting, you might also be interested in some research my team at Atlassian recently shared on what makes a difference to how people are experiencing remote work during the pandemic. Read more here.

I never really loved mathematics. I am much more of a big picture person than a tiny detail person. But I usually did ok in maths tests because you got marks not just for the answer but for showing all the thinking you did to get there. I may not always get the answer right, usually as a result of a simple mistake along the way, but you can see how I am thinking and understand where to intervene to correct. We both learn.

I apply the same approach to research practice, especially when working with teams who may not have a particularly strong understanding of how and why we do things as we do. An open research practice has multiple benefits including:

• learning about how and why you think about and make decisions and actions at each stage
• understanding what tradeoffs are being made and the impact this has (there are always tradeoffs)
• understanding how we move from data to insight and deeply understanding and trusting what we have learned.

Openness in research requires the willingness to adapt, to not always be right and perfect, to go slower than you want to (and often far slower than people expect) and as a result requires decent amount of bravery.

There are three crucial stages for openness in research practice:

• study design
• fieldwork
• analysis and synthesis

Openness in study design

Being open in the study design phase means bringing your team into the process about considering who we want to talk to (and, as a result who we choose not to include), and what we want to talk with the about. In particular, the consideration of what kinds of differences matter in our audience base is an important one to have and one that thinking about research recruitment can help to facilitate.

Openness in fieldwork

Being open during fieldwork refers to a researcher’s willingness to have team mates observe the research as it happens. There are many different ways that you can enable this, and different levels of interactivity that your team might have with the participant during the study. Being comfortable with having your team observe as you conduct research can be really challenging for researchers at first. Once this becomes standard practice though, it quickly becomes an essential part of our practice and help us to demonstrate the differences between the research questions we want to answer and the questions we need to ask participants in order to answer those questions.

Many researchers are concerned that participants will observe one session and run off to change the entire product based on a single data point – although this is a commonly voiced concern it is usually easily managed by clearly setting expectations that everyone on the team is required to observe at least two sessions before being allowed to participate in the analysis process (from which the findings emerge). We often use UIE’s Exposure Hours requirement for at least 2hrs of observation every 6 weeks as a metric that helps encourage team mates to experience more than a single session in any one research study.

Openness in analysis and field work

While giving team mates the opportunity to observe their customers and users first hand has obvious benefits, allowing them to participate in the analysis process is arguably even more important. This is where we truly pull back the curtain and show the hardest work of research which is making sense of all the stories we have heard and things we have observed.

Robust analysis and synthesis is probably one of the most overlooked aspects of the research process – all too often we see examples of people observing a number of sessions, taking a few bullet point notes et voila – the findings immediately emerge.

If only it were really that simple. Analysis and synthesis is hard, time consuming work when done properly. Doing it properly is essential if you want to do the work required to rid yourselves of as many of those annoying cognitive biases as possible – in particular the confirmation bias and recency effect.

Allowing and encouraging team mates to participate in research analysis gives them an opportunity to get much closer to more of the data, but it also helps them to understand the way that we process that data in order to make sense of it and draw conclusions. It allows them to challenge the ways we are forming narratives about what we believe that data means and demonstrates the traceability of those claims back to the original source data.

This blog post and video describe how I’ve done collaborative analysis successfully with teams.

Open research is challenging but worthwhile

It is beyond dispute that working in an open way – research as a team sport – is slower and more painful for researchers than putting our heads down and getting through the work alone. It is, on the surface, less efficient and more annoying. Nonetheless, if you want to grow understanding and respect for the research craft in your organisation, it is very much worth the overhead to take the time and effort to open up your processes to your team and invite them in to participate actively. Overtime, the increased Research IQ in your team will pay dividends and your ability to be impactful more efficiently will increase.

It is important to remember that the most important thing is not the time to deliver the report, but the impact our research has on our teams ability to make good decisions for our customers and our users. Let’s make sure we’re optimising for efficiency to  the right outcome.

Here are five common dysfunctions that we are contending with.

1. Teams are incentivised to move quickly and ship, care less about reliable and valid research
2. Researching within our silos leads to false positives
3. Research as a weapon (validate or die)
4. Quantitative fallacies
5. Stunted capability

In this post, we’re looking at what happens when the research practice in an organisation fails to mature.

A great first step

Testing one user is 100 percent better than testing none – Steve Krug, Don’t Make Me Think

Many organisations get started doing research with customers and users off the back of encouragement from people like Steve Krug and his classic book ‘Don’t Make Me Think’. In this and other books Steve makes simple usability testing accessible and achievable to almost anyone.

Steve and others like him are evangelists reaching out to those companies who are afraid to engage with their customers to understand opportunities for them to improve. This is important work. Their message is usually that talking to customers is not hard or scary, and that we’ll be better for doing a bit of it, even not perfectly, than not doing it at all.

The first step can be scary

And they are right. Having anyone in the company talking to just one user (and hopefully some more) is a fabulous first step. But it is intended to be just that – a first step. An encouragement to realise the benefits of involving people outside our offices in the process of designing and developing products and services. And help to overcome the fear of engaging with customers and users and an opportunity to experience how beneficial this can be.

For those of us who work with research participants on a regular basis, it may be hard to recall exactly how terrifying those first few research sessions felt. Even trained and experienced researchers continue to experience some background fear (or exhilaration?)  of all the things that could go wrong in the research study – and there are plenty!

The thing about first steps, though, is that they are usually intended to be followed by second steps. Once we break through the fear (or in some cases, just lack of awareness), the idea is that we continue to increase the maturity of our practice.

And this is where many organisations seem to hit a roadblock. More and more people in the organisation might be out and eagerly involving customers in the process of shaping their products, but they often don’t invest in either improving their own skills in research or investing in hiring people who have training and experience doing research. 

Talking to users is not research

One important realisation we need to have on the path to maturity is recognising that ‘talking to customers’ is actually not the same thing as doing research. Talking to customers or watching customers use our products and services has many benefits – in particular it can increase our empathy for our customers and users, it can help expose us to scenarios of use that are dramatically different to our own and what we would expect, and it can provide clues as to where the biggest problems may like. All of these are good outcomes.

If we want to use research as evidence for decision making – either for product strategy or design decisions – then we need to be able to do more to ensure that the insights we are gleaning are sufficiently reliable and valid.

Research doesn’t need to be ‘perfect’, just valid and reliable.

 ‘I don’t need the research to be perfect, I just need enough to help me make a decision’.

Often this is said in response to the suggestion that the research we should be doing will take longer or be more difficult and expensive than our speaker would like. In this situation, there is often a pre-existing ‘hunch’ and they are looking to users for validation. Or perhaps they are stuck between two options and seeks a tie breaker.

Any specialist researcher has almost certainly had their recommended approach discredited as ‘too academic’, and sometimes it is true. Sometimes the research methodology is overdone for the question the business is seeking to answer. But what often follows is a bit of a race to the bottom where considered sample design and appropriate methodology are quickly discarded in favour of whatever is fastest and easiest.

Without the right experience and training, all too often interviewers ‘cut to the chase’ so we get more or less directly to the topic at hand. Somewhere in the world right now a product manager under pressure to make a decision is asking questions like these in a customer interview:

‘here’s what we’re thinking of making, what do you think about it?’

or, perhaps worse…

‘if we made this, would you pay for it’

It can be easy and tempting – so much faster and often quantitative – to mistake the research question for the interview question.

Even with training, it seems that the urge to be able to say that 10 out of 12 people said they would pay for it is almost irresistible. ‘Beating around the bush’ to get the question answered seems like a waste of everyone’s time, in this time where the bias to action and desire to ship at velocity is most valued. 

(It shouldn’t really be a surprise that lack of research capability maturity exposes us to the previous four dysfunctions).

Matching methodology to risk

Whilst we should have plenty of sympathy for this desire for lightweight research and simplicity, it is important to ensure that the methods employed are matched to the risk involved in the decision, rather than the most compressed timeframe.

As our organisations grow, the decisions we take using evidence from our customers can become more and more substantial – the gains of getting it right are greater and the risks of getting it wrong get uglier.

In the same way, our research maturity needs to continue to grow so that we can continue to match the size of the risk of getting it wrong.

This is not to say that mature organisations only ever do serious, time consuming research. Rather, that we invest where the risk is highest.

Investment might look like hiring trained researchers who can design and recruit the right sample and conduct the research in a way that reduces bias. Or investment might look like iterative research with an every increasing number of increasingly diverse participants, sprint after sprint – allowing the team to continue to learn, This can work beautifully when the team is able to be responsive to that learning over time.

Investing too much

Conversely, there are situations where the investment in research is far too high for the decision being made. This often happens where the organisations design process has broken down, or where designers have entirely lost confidence in being able to make relatively conventional design decisions. In these situation we design complex studies to ‘validate’ one micro design treatment over another. In this case,  the mismatch in risk to research investment can result in large quantities of what I would consider to be wasteful and often unreliable research. 

Beware Dunning Kruger

User Research is particularly susceptible to Dunning Kruger syndrome, wherein a relatively small amount of knowledge can result in an excess of confidence. Many people claim a ‘background in research’ when they could mean they watched someone else do a bunch of usability studies in their last job, or they did a research based degree at university.

Many designers and product managers are entirely happy with the outcomes they get from research and how it enables their practice – and often loudly object to the suggestion that anyone could get a better result from the research than they do.

Yet, at the same time, the harsh reality is that the work that is done is often resulting in misleading outcomes that can put their product and their organisation at risk. 

It also undermines the reputation of research in the organisation when people claim when a ‘researched’ product goes into the world and doesn’t succeed as expected. ‘We did research before and it didn’t work’.

In the same way that often both design and product management capabilities require an engineering led organisation to move through the stages from unconscious incompetence through to conscious competence ,  the very same is true for the research capability.

Achieving research maturity

And so, at the end of our five dysfunctions, what can be done to help provoke an organisation to not only involve users in the process of creating products and services but to start and continue to grow their ability to do so by revealing the important insights that are both reliable and valid?

Here are some things that have worked for me.

Perhaps through improving business fluency. By talking less about empathy and more about the risk to the business of getting it wrong. Talking less about customer obsession and more about the reliability and validity of the different types of evidence we can use to make decisions. And by running an open research practice – getting out of the black box, removing any mystery about our work, showing our workings and involving others in the process.

Make use of existing momentum – bringing new shape and substance to whatever your organisation is using to bring its attention to its customers – whether its an NPS survey, a customer convention, a feedback form, or a guerilla research practice – start by shaping the existing connections into something more insightful, more reliable and valid.

Be brave, but be patient and we’ll get there.

Here are five common dysfunctions that we are contending with.

1. Teams are incentivised to move quickly and ship, care less about reliable and valid research
2. Researching within our silos leads to false positives
3. Research as a weapon (validate or die)
4. Quantitative fallacies
5. Stunted capability

In this post, we’re looking at what happens when research is ‘weaponised’ in teams.

Dysfunction #4 –  Quantitative fallacies

I often say that when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science, whatever the matter may be. – Lord Kelvin

I fear many people feel like Lord Kelvin.

There seems to be an intuition that knowledge unable to be expressed numerically is less satisfactory than knowledge that fits into a graph. In order for an assertion to be worth considering as serious, it must have a number associated with it. Everything else is anecdote.

Perhaps we have inherited this from finance. Finance are, after all, the masters of presenting future fictions with bold numbers and graphs. Finance whose authority appears to be rarely challenged.

Organisations love quantitative research because it is fast and feels definitive.

Smash out a survey, launch an experiment, categorise customer feedback by keyword, look at the product analytics. Somehow, numbers just feel more reliable. More trustworthy.

The McNamara fallacy (also known as quantitative fallacy), named for Robert McNamara, the US Scretary of Defense from 1961 to 1968, involves making a decision based solely on quantitative observations (or metrics) and ignoring all others. The reason given is often that these other observations cannot be proven.

–  Daniel Yankelovich “Corporate Priorities: A continuing study of the new demands on business.” (1972)

In my experience, presenting a number boldly is much less likely to be challenged than any assertion backed up by more qualitative evidence. Yet surprisingly few people seem to be inclined (or able) to ensure that the work done to establish that number has any rigour.

Take surveys. How many organisations the the time to do cognitive interviewing to ensure that the data collected in the survey is valid and reliable? Very few. Most don’t know it is even something you should do, and the others don’t want to spend the time.

Do we just have blind faith that our survey respondents will make sense of the questions the same way as us? Or do we actually not really care so much about the validity? We just want an answer. A definitive sounding answer. Some data to show that we are evidenced based.

How many teams when A/B testing their two versions of the design using unmoderated research watch the videos to make sure that people did really complete the task in a way that could be considered an adequate user experience? To check that the people who undertook the research have any resemblance to who they said they were in the screener? To ensure that they things they say and the scores they give make sense when compared to the experience they actually had?

All sounds a bit time consuming doesn’t it, when all you really want is data to tell you what to do. To take the decision out of your hands.

We’ve managed to convince ourselves with a large enough volume of respondents, these problems go away. But the fact is, these numbers can easily be completely misleading. People don’t understand the survey question and answer anyway. To get the incentive. To find out what other questions you’re asking, because some of us are completists.

Recently my team recently did some survey testing – we were testing a feature prioritisation survey (not my favourite). We observed people who told us they didn’t understand what a features as described in the survey. Regardless, it sounded cool and they then went on to prioritise it highly against other features in the survey regardless.

How often does this happen? No one knows.

The first step is to measure whatever can be easily measured. This is OK as far as it goes.

The second step is to disregard that which can’t be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading.

The third step is to presume that what can’t be measured easily really isn’t important. This is blindness. The fourth step is to say that what can’t be easily measured really doesn’t exist.

This is suicide.

–  Daniel Yankelovich “Corporate Priorities: A continuing study of the new demands on business.” (1972)

There are multiple, related quantitative fallacies.

Some like McNamara and Lord Kelvin, believe that quantitative data is superior. But others are more complex – they trust that the trade off for speed and convenience does not have a dangerous impact to validity and reliability. Other fallacies result from absence of experience and ability in defending qualitative data and critiquing quantitative methods.

The fastest and most ‘definitive’ sounding methodologies (and the tools that enable them) have never been more popular. While it is encouraging that more and more people are keen to take a more human centred approach to product design,  experienced researchers need to intervene to make sure that these methods are being used, and critiqued, appropriately.

We need to ensure that our organisations don’t over index to the rapid, quantitative methods because they play well with senior leadership. And when we do use these methods , we need to ensure that we maintain a high enough quality standard that we can genuinely stand behind the numbers and believe they have some reliability and validity.

You can read about the next dysfunction here.

Here are five common dysfunctions that we are contending with.

1. Teams are incentivised to move quickly and ship, care less about reliable and valid research
2. Researching within our silos leads to false positives
3. Research as a weapon (validate or die)
4. Quantitative fallacies
5. Stunted capability

In this post, we’re looking at what happens when research is ‘weaponised’ in teams.

Dysfunction #3 – Research as a weapon (validate or die)

Over reliance on research, without care to the quality level of the research, can also be a symptom of another problem in our organisations – lack of trust between disciplines in a cross functional team.

In particular the relationship between design and product management can have a substantial impact on the way that research is used in product teams. If the relationship is strong, aligned and productive research is often used to support real learning in team. But where the relationship is less healthy, it is not uncommon to see research emerge as a form of weaponry. 

Winning wars with research

How does research become weaponry? When it is being used primarily for the purpose of winning the argument in the team.

Using research as evidence for decision making is good practice, but as we have observed in earlier dysfunctions, the framing of the research is crucial to ensuring that the evidence is reliable and valid. Research that is being done to ‘prove’ or ‘validate’ can often have the same risk of false positives that comes from the silo dysfunction.

This is because the research will often be too tightly focussed on the solution in question and there is little or no interest from the team around the broader context. This lack of realistic context can result in teams believing that solutions are more successful than they will ultimately turn out to be in the realistic context of use.

Data as a crutch for design communications

Another reason to see research being used as weaponry is to compensate for a lack of confidence or ability in discussing the design decisions that have been made. Jen Vandagriff, who I’m very fortunate to work with at Atlassian, refers to this as having a ‘Leaky Design Gut’.

Here we see research ‘data’ being used instead of (not as well as) the designer being able to explain why they have made the design decisions they have made. Much as I love research, it is foolish to believe that every design decision needs to be evidenced with primary research conducted specifically for this purpose. Much is already known about design decisions that can enhance or detract from the usability of a system, for example. 

In a team where the designer is able to articulate the rationale and objectives for their design decisions, and there is trust and respect amongst team members, the need to ‘test and prove’ every decision is reduced.

Validation can stunt learning

Feeling the need to ‘prove’ every design decision quickly leads to a  validation mindset – thinking, ‘I must demonstrate that what I am proposing is the right thing, the best thing. I must win arguments in my team with ‘data”. .

Before going straight to research as validation’, it is worth considering whether supporting designers to grow on their ability to be more deliberate in how they make and communicate their design decisions could be a more efficient way to resolve this challenge.

Sometimes it is entirely the right thing to run research to help understand whether a proposed approach is successful or not. The challenge is to ensure that we avoid our other dysfunctions as we do this research. And to make sure that this doesn’t become the primary role of research in the team – to validate and settle arguments. Rather, it should be part of a ‘balanced diet’ of research in the team.

If we focus entirely on validation and ‘proof’, we risk moving away from a learning, discovery mindset. We prefer the leanest and apparently definitive practices. A/B testing prototypes and the creation of scorecards are common outputs that result from this mindset. We’re incentivised to ignore any flaws in the validity of the method if we’re able to generate data that proves our point. 

Alignment over evidence

Often this behaviour comes from a good place. A place where teams are frustrated with constant wheel spinning based on everyone having an opinion. Where the team is trying to move away from opinion based decision making, where either the loudest voice always wins or the team feels frustrated by their inability to make decisions to move forward. Using research as a method to address these frustration does make sense and should be encouraged.

Validation research can provide short term results to help move teams forward, but it can reinforce a combative relationship between designers and product managers. Often this relationship comes from a lack of alignment around the real problems that the team are setting out to solve. Investing more in more ‘discovery’ research, done collaboratively, as a ‘team sport’ can be incredibly powerful in helping create a shared purpose across the team that can help promote a more constructive and supporting teamwork environment.

Support from an experienced researcher with sufficient seniority can help the team avoid the common pitfalls of seeking the fastest and most definitive ‘result’, but to achieve a shared understanding of both the problem and the preferred solution. Here the practice of research, done collaborative as a team, can help not only to inform the situation to achieve more confident decision making, but also to heal some tensions in the team, by bringing the team together around a shared purpose – solving real problems for their customers or users.

You can read about the fourth dysfunction here.

Here are five common dysfunctions that we are contending with.

1. Teams are incentivised to move quickly and ship, care less about reliable and valid research
2. Researching within our silos leads to false positives
3. Research as a weapon (validate or die)
4. Quantitative fallacies
5. Stunted capability

In this post, we’re looking at the impact of our organisation structure on research outcomes.

Dysfunction #2 – Researching within our silos leads to false positives

The larger the organisation, the more fragmentation and dependencies you tend to get across teams. Teams are organised by product or platform, and then often by the feature set they work on. Occasionally teams are organised by a user type, and very rarely you find some arranged by user journey.

Even in this complex ecosystem of teams where dependencies are rife, the desire for autonomy in teams remains. Between teams, we tend to seek to avoid reliance other teams where possible. We don’t want our own team velocity or ability to ship to be decreased by anyone else. In this environment, collaboration between teams tough. It can be hard to coordinate, there’s no incentive to take this time and trouble. And this leads to greater focus, which, in theory is great, except….

Beware the Query Effect

When it comes to research, we know how critical getting the right research question is. Getting the ‘framing’ of the research right is crucial because, as the Query Effect tells us (and as we know from our own personal experience) you can ask people any question you like and you’ll very likely get data in return.

Whenever you do ask users for their opinions, watch out for the query effect:

People can make up an opinion about anything, and they’ll do so if asked. You can thus get users to comment at great length about something that doesn’t matter, and which they wouldn’t have given a second thought to if left to their own devices. – Jakob Nielsen

By focussing our research around the specific thing our team is responsible for, we increase our vulnerability to the query effect.  That little feature is everything to our product team and we want to understand everything our users might think or feel about it, but are we perhaps less inclined to question our team’s own existence in our research?

Researchers are encouraged to keep the focus tight, to not concern themselves with questions or context that the team cannot control or influence.

I like to use this visual illustration of what that is problematic. Take a quick look at the image below. What strange sea creature do we have here do you think? Looks quite scary, right?

Oh but wait, when you pull back just a little more you realise the story is completely different, and all we have here is a little duck, off for a swim, nothing to worry us at all.

How often is our research so tightly framed on the feature our team is interested in that we make this mistake?

We think something is important when in actually, in proper context of the real user need, it is not so important at all? Or conversely, we focus so tightly on something we think is important when what our users care about is just out of frame. Just outside the questions we are asking, that they are so busy now, helpfully answering. Even though it is not the important thing.

I fear this is one of the most common dysfunctions that we see in product teams doing research in the absence of people who are sufficiently experienced and with seniority and confidence to encourage teams to reshape their thinking.

What is the risk?

Research that is focussed too tightly on a product or a feature increases the risk of a false positive result. A false positive is a research result which wrongly indicates that a particular condition or attribute is present.

False positives are problematic for at least two reasons. Firstly they can lead teams to believe that there is a greater success rate or demand for the product or feature they are researching than is actually the case when experienced in a more realistic context. And secondly, they can lead to a lack of trust in research – teams are frustrated because they have done all this research and it didn’t help them to succeed. This is not a good outcome for anyone.

The role of the trained and experienced researcher is to not only have expertise in methodology but also to help guide teams to set focus at the right level, to avoid misleading ourselves with data. To ensure we not only gather data, but we are confident we are gathering data on the things that really matter. Even if that requires us to do research on things our team doesn’t own and cannot fix or to collaborate with others in our organisation. In many cases, the additional scope and effort can be essential to achieving a valid outcome from research that teams can trust to use to move forward.

You can read about the third dysfunction here.

So we move on to the second generation question. How do we scale this user centred behaviour?

Depending on where in the world you are – and your access to resources – your answer is usually to hire more researchers and/or to have other people in the team (often designers and product managers) to do the research. This is often known as ‘democratising research’.

Almost certainly this is the time that an organisation starts looking to hire designers and product managers with a ‘background in research’ and to establish some research training programs, interview and report templates and common ways of working.

This all sounds eminently sensible, but there are some fairly structural issues in how we work that can undermine our best intentions. At best, it can render our research wasteful and inefficient, and at worst it can introduce significant risks in the decision making that our teams make.

Each of these are systemic issues and anyone doing research is likely to be impacted when working as part of a cross functional product team.

So, let’s assume that people doing research have had adequate training on basic generative and evaluative research methods – here are five common dysfunctions that we will need to contend with.

1. Teams are incentivised to move quickly and ship, care less about reliable and valid research
2. Researching within our silos leads to false positives
3. Research as a weapon (validate or die)
4. Quantitative fallacies
5. Stunted capability

Here we will start with the first, which is one that many will find familiar.

Dysfunction #1.
Teams are incentivised to move quickly and ship, care less about reliable and valid research

The most popular research tools are not the ones that promise the most reliable or valid research outcomes, but those that promise the fastest turnaround. One well known solution promises:

Make high-confidence decisions based on real customer insight, without delaying the project. You don’t have to be a trained researcher, and there’s no need to watch hours of video.

It sounds so appealing and it is a promise that a lot of teams want to buy. Speed to ship or velocity is often a key performance indicator for teams. It’s not a coincidence that people usually start with ‘build’ and rushing to MVP when talking about the ‘learn, build, measure’ cycle.

Recruitment trades offs made for speed

The challenge is that doing research at pace requires us to trade off characteristics  are important to the reliability and validity of research.

One of the most time consuming aspects of research is to recruit participants who represent the different attributes that are important for understanding user needs the product seeks to meet. The validity of the research is constrained by the quality of the participant recruitment.

What do we mean by validity? In the simplest terms, it is the measure of how well our research understands what we intend for it to understand.

Most of the speedy research methods – whether that’s guerrilla research at the coffee shop or using an online tool – tend to compromise on participant recruitment. Either you just take whoever you can get from the coffee shop that morning, or you recruit from a panel of participants online and trust that they are who they say they are and that they won’t just tell you nice things so you don’t give them a low star rating and they get to keep this income source.

There are many kinds of shortcuts to be taken around recruiting – diversity of participants, ‘realistic-ness’ of participants or number of participants being a few. Expect to see some or all of these short cuts in operation in product teams where speed to ship is the primary goal.

Being fast and scrappy can be a great way to do some research work, but in many teams the only kind of research they are doing is whatever is fastest. This is like eating McDonalds for every meal because you’re optimising for speed… and we all know how that works out.

Teams are trading off research validity for speed every day. Everyone in the organisation understands the value of getting something shipped, and this is often measured and rewarded. Not so many people understand risks associated with making speed related trade offs in research.

What is the risk?

Misleading insights from the research work can send a team in the wrong direction. That can direct a team to spend time creating and shipping work that does not improve their users experience or meet their users needs. That does not increase the desirability or necessity of their product, and thereby negatively impacts their productivity and the profitability of their organisation.

Does this mean that speed to ship is bad? Should all research be of an ‘academic standard’?

No.

Testing to identify some of the larger usability issues can often be done with small participant numbers and less care to find ‘realistic’ respondents. But if the work that results from your research findings is going to take more than one person more than a week to implement, it might be worth increasing the robustness of your research methodology to increase confidence that this effort is well spent.

People doing research need to be clear with their teams about the level of confidence they have in the research findings (it is fine for some research to result in hunches rather than certainty as long as it is clearly communicated). And teams should plan to ensure they are using a healthy diet of  both fast and more robust research approaches.

Organisations need to ensure they have someone sufficiently senior asking questions (and understanding how to critique the answers)  about not just the existence of data from user research but also looking under the hood to evaluate the trade offs being made, and as a result the level of confidence and trust we should place in the insights and claims made.

You can read about the second dysfunction here.

I also showed a slide that was a photo of a team, gathered in a meeting around a wall covered with index cards. No one in the meeting had their laptops out.

This is what people seemed to want to talk about over coffee later in the day. Could it be true that I, spokesperson (one of many) for Jira, would possibly want to see stuff on the wall?  Surely it should all just be in Jira right?

People told me they messaged photos of the slide back to their teams to show them – ‘look, the Atlassian person says it is ok to put things on the walls!’

omg yes. I love walls with post its and index cards stuck on it and sketches on whiteboards. I like walls for planning, for thinking, for communicating and for analysing. And then you capture it all in a tool, like Jira.

This is why.

Walls make it easier to iterate

Digital things look ‘finished’ too soon. when something is a work in progress on a wall, it looks unfinished, so you keep working on it. moving things around, reshaping things, connecting things, erasing things, and making them again. Walls make it easier to iterate. Iteration, in my opinion, is massively correlated with quality.

This is why walls are good for sketching out design ideas and processes. This is why they are amazing for research analysis (don’t care what anyone says, post it notes are still the best tool for research analysis for exactly this reason – no one ever does three (or more) rounds of synthesis using a digital tool).

Walls make it easier to collaborate (in a single location)

There is something about a group of people standing in front of a wall full of sketches, or index cards or post it notes. Its a different kind of collaboration than you get around a table, or in a digital tool. You’re usually standing up, so you’re paying attention, you’re focussed. People physically pick up the card that they are talking about and something about that seems to pull focus even more. Doing a stand up at a physical wall and moving the cards across to done has always felt a lot like the physical act of crossing something off the to do list – so much more satisfying than updating a status on an issue. The messiness of a room full of post it notes when you’re doing analysis almost compels you to finish making that sweep through the data… finding the best place, for now, for every sticky note of data. There is something about the physicality and the embodiment of the work that I have always felt binds teams together more, drives us to do better and more complete thinking about the work we’re doing. There is no science to this just many years of experience. Walls just work better for me, when I’m lucky enough to work in the same location as my team. Walls do suck at remote and distributed teams.

Walls make it easier to communicate

Sometimes the walls are not for you but for other people. Sometimes walls are to send a message. They can say – ‘look how many things people want us to do, this is insane and someone needs to prioritise this’, they can say ‘look how much we’ve done this sprint, yay!’ or ‘look how much we have left to do, uh oh!’, they can say ‘these things are really important to our team, this is what we believe it’, or they can say ‘here is what we’re working on at the moment’.

I’ve been in, and observed many teams who use walls to communicate either the most important messages to the team in a kind of omnipresent way – this is what we believe it, or this is what we are focussing on right now, or these are our values, or here is our goal.

Or sometimes they are designed to communicate to bosses and stakeholders – those walls might say ‘we’d get the things we’ve promised done if you didn’t always sneak in all this unplanned work’ (I’ve seen a few of those).

Some people I’ve know have had jobs that include keeping the digital tool up to date with the wall. Or the other way around. Its not inefficiency. The wall is doing different things for the team.

And that’s another great thing about walls, it doesn’t need to be a zero sum game.

If you’re using Jira, using a wall makes perfect sense to me. I don’t know why you’d do without.

related reading: Alan Cooper – Know whiteboards, know design

So what should we do with this research?

This is a question that researchers often hear at the end of a playback session. Especially one where we’re sharing findings or insights and not detailed recommendations of what to do next.

Most of the time there are two questions that teams should ask themselves:

1. Which of these problems/opportunities do we care about now? If you were going to prioritise, which are the most pressing? Which might contribute most to the team meeting goals?
2. What do we think we could do that might make things better for our users? What different things could we do that might address this opportunity?

A good researcher can help a team understand what opportunities are available to pursue. They will help you to see a problem in a different way – to frame the problem from the users point of view.

But you shouldn’t expect the researcher to come back and ‘tell you what to do’.

From insights to actions

Getting to actions from insights is a team sport that requires a range of inputs. The researchers role is to make the ‘user’ input as rich and insightful as possible. They should then to work with the team explore and evaluate the possibilities that emerge.

What makes an insight actionable?

To make a research insight actionable it must answer two key questions:

• what is happening?
• why is it happening?

Research that is not actionable answers only the first of these questions. If we don’t know why something is happening, we are not well placed to contemplate what action we should take.

The better the ‘why’ explanation, the better equipped a team will be to come up with clear and confident actions in response.

Research alone won’t tell you what to do

Sometimes when people say they want the research to be actionable, what they really mean is that they want the research to tell them what to do. They want research to answer a third question:

• what should we do?

Sometimes the right course of action is 100% obvious, but often that is not the case. It would be a foolish or naive researcher who thinks they have the full set of knowledge required to provide this answer.

User research is just one of the pieces of information that product managers or designers need to decide what they should do.

Lenses for decision making

To make a good decision about what to do next, the team really needs to look through at least four lenses:

• what is the user perspective?
• how does this align to our product strategy?
• what are the technical (feasibility) issues?
• what are the financial/business implications? (cost / revenue)

Or, to use a more familiar framework, is the solution desireable, feasible and viable.

Most of the time, user researchers aren’t in possession of this full set of information. They will likely have strong and informed views. But don’t be disappointed if they can’t point you straight to the perfect solution.

Designers and product managers are usually much more expert in coming up with and evaluating solutions.

Designers are trained to take a problem and think about how you might be able to take many different approaches to solving it. Teams should use the designer to make sure they’re generating and evaluating lots of possible solutions.

Product Managers tend to be the experts in balancing all the different needs and helping the team to choose the best of the solutions on offer.

Researchers can help represent the end user perspective through out this process. They can play a role in helping design a way of evaluating proposed solutions from the users point of view.

Other team members are also vital in this process.

Engineers and technical representatives giving the feasibility perspective (and quite often some pretty amazing possible solutions that the designers might have missed).

Analysts and data scientists providing a different useful data sets to contribute to evaluating solutions. Sometimes a colleague from legal, or marketing, or other parts of the organisation can be very useful in this process too.

Getting from insights to actions is a team sport

Its the responsibility of the researcher to make sure that the insights they bring to the team are useful. They need to explain the why and not just the what. But moving from insights to actions is a team sport and needs all the players to participate.