Findability Day 2013 is just a few weeks away and the agenda is now finalized. We will have great keynote speakers and cases for inspiration and of course the approximately 200 attendees will create a valuable networking opportunity.

The event takes place in Stockholm on May 30^th and as organizers of the event we are proud to present the following list of speakers and topics:

Martin White – The future of search

Daniel Bergqvist, Google – The Star Trek Computer

Bjørn Olstad, Microsoft – Unveil the hidden values in your organization

Ravi Mynampaty, Harvard Business School – Developing a Search & Findability Practice for the Enterprise 

Kristian Norling, Findwise – The 2013 Findability Survey

Sebastian Forseland, Husqvarna – Master Data management + Enterprise Search = User Satisfaction

Christian Finstad, Meltwater – Big data for online insight

Martin Öhléen, SKF – Search as a driver in Mobile applications

Jonas Berg, Svensk Byggtjänst – The next generation business search engine

Johan Johansson, Municipality of Norrköping – Governance and the role of search in user satisfaction

Niclas Lillman, Scania – Search as a service

DJ Skillman, Senior Director Technical Services, Splunk – Big data

Check out the agenda here for more details and for registration. There are just a few seats left so make sure to register today!

It promises to be a great event and a day full of inspiration, knowledge sharing and networking opportunities to help develop your business, personal skills and professional network.

Hope to see you there!

To achieve a high NPS, Net Promoter Score, the customer experience (cx) is crucial and a critical factor behind a positive customer experience is the ease of doing business. For companies who interact with their customers through the web (which ought to be almost every company these days) this of course implies a need to have good Findability and search on the website in order for visitors to be able to find what they are looking for without effort.

The concept of NPS was created by Fred Reichheld and his colleagues of Bain and Co who had an increasing recognition that measuring customer satisfaction on its own wasn’t enough to make conclusions of customer loyalty. After some research together with Satmetrix they came up with a single question that they deemed to be the only relevant one for predicting business success “How likely are you to recommend company X to a friend or colleague.” Depending upon the answer to that single question, using a scale of 0 to 10, the respondent would be considered one of the following:

The Net Promoter Score model

The idea is that Promoters—the loyal, enthusiastic customers who love doing business with you—are worth far more to your company than passive customers or detractors. To obtain the actual NPS score the percentage of Detractors is deducted from the percentage of Promoters.

How the customer experience drives NPS

Several studies indicate four main drivers behind NPS:

• Brand relationship
• Experience of / satisfaction with product offerings (features; relevance; pricing)
• Ease of doing business (simplicity; efficiency; reliability)
• Touch point experience (the degree of warmth and understanding conveyed by front-line employees)

According to ‘voice of the customer’ research conducted by British customer experience consultancy Cape Consulting the ease of doing business and the touch point experience accounts for 60 % of the Net Promoter Score, with some variations between different industry sectors. Both factors are directly correlated to how easy it is for customers to find what they are looking for on the web and how easily front-line employees can find the right information to help and guide the customer.

Successful companies devote much attention to user experience on their website but when trying to figure out how most visitors will behave website owners tend to overlook the search function. Hence visitors who are unfamiliar with the design struggle to find the product or information they are looking for causing unnecessary frustration and quite possibly the customer/potential customer runs out of patience with the company.

Ideally, Findability on a company website or ecommerce site is a state where desired content is displayed immediately without any effort at all. Product recommendations based on the behavior of previous visitors is an example but it has limitations and requires a large set of data to be accurate. When a visitor has a very specific query, a long tail search, the accuracy becomes even more important because there will be no such thing as a close enough answer. Imagine a visitor to a logistics company website looking for information about delivery times from one city to another, an ecommerce site where the visitor has found the right product but wants to know the company’s return policy before making a purchase or a visitor to a hospital’s website looking for contact details to a specific department. Examples like these are situations where there is only one correct answer and failure to deliver that answer in a simple and reliable manner will negatively impact the customer experience and probably create a frustrated visitor who might leave the site and look at the competition instead.

Investing in search have positive impacts on NPS and the bottom line 

Google has taught people how to search and what to expect from a search function. Step one is to create a user friendly search function on your website but then you must actively maintain the master data, business rules, relevance models and the zero-results hits to make sure the customer experience is aligned. Also, take a look at the keywords and phrases your visitors use when searching. This is useful business intelligence about your customers and it can also indicate what type of information you should highlight on your website. Achieving good Findability on your website requires more than just the right technology and modern website design. It is an ongoing process that successfully managed can have a huge impact on the customer experience and your NPS which means your investment in search will generate positive results on your bottom line.

More posts on this topic will follow.

/Olof Belfrage

One of the major trends at the 35th annual European Conference on Information Retrieval was event related data. The conference took place between the 24th and 27th of March this year in a snowy Moscow, Russia. It attracted around 300 participants from all over the globe, 3 of them findwizards. While ECIR 2013 provided talks on a large variety of topics from across the field, event related data was definitely a buzz word.

The keynote speaker opening the second day of conference was Rutgers University assistant professor and Mahaya inc. CTO Mor Naaman. In his talk, Mr Naaman let the following image explain why Mahaya inc. are in business.

The past two papal elections.

The image above clearly shows that the way people act at events has changed considerably in the past few years, nowadays everyone is a reporter and their stories can be found on social media. Using platforms such as Twitter, Facebook and YouTube as data sources Naaman’s company creates products which not only extracts, but also synchronizes event coverage. One interesting feature in their latest product is the synchronization of video clips, making it possible for a user to easily switch view when watching video footage of for example a concert.  An arguably even stronger feature of this use of social media is the fact that news and event footage can reach the world even if no press is present at the scene. Slides from this inspiring talk can be found here.

Another presentation the same day displayed promising results in the task of automatic event detection. Using machine learning algorithms a team of researchers from Hanover, Germany have designed a system for detecting and summarizing entity related events from Wikipedia edit history data. Basically the idea is that when a Wikipedia article is edited by a large amount of users in a short period of time that can mark an important event considering the subject of the article. More information about this research can be found here.

The last day of the conference opened with a presentation from Jimmy Lin of Twitter. His talk centered on the importance of fast real-time indexing in social media platform architecture. One of the strengths of Twitter is presenting the users with information about events as they happen. As an example of this he used the event of an earthquake hitting eastern USA in 2011. Tweets from locations closer to the epicenter of the earthquake reached Twitter users in New York City before the actual quake did. I have to admit “Twitter, faster than earthquakes” is a pretty good slogan.

So whether it’s using social media data to let people (re)visit events, automatic event detection in open source dictionaries, making sure your indexing is fast enough to let your users cover events as they happen or something else, event based data seems to be one of the driving forces in the field of IR at the moment.

Big Data is also a Big Challenge for a number of companies that would like to be ahead of the competition. I think Findwise can help a lot with both technical expertise in text analytics and search technology but also with how to put Big Data to use in a business.

During the last days of February I had the pleasure to attend IDG Big Data conference in Warsaw, Poland. It brought plenty of people from both vendors and industry that shared interesting insights on the topic. In general, big vendors that try to be associated with Big Data dominated the conference. IBM, SAS, SAP, Teradata has provided massive marketing information on software products and capabilities around Big Data. Interestingly every single presentation had its own definition on what Big Data is. This is probably caused by the fact that everybody tries to find the best definitions for fitting own products into it.

From my perspective it was very nice to hear that everyone agrees text analytics and search components are of big importance in any Big Data solution. In multiple applications analysis (both predictive and deductive) and for mass social media one must use advanced linguistic techniques for retrieving and structuring the data streams. This sounded especially strong in IBM and SAS presentations.

A couple of companies revealed what they have already achieved in so called Big Data. Orange and T-Mobile presented their approach of extending traditional business intelligence to harness Big Data. They want to go beyond standard data collected in transaction databases and open up for all the information they have from calls (picked and non-answered), SMS, data transmission logs, etc. Telecom companies consider this kind of information to be a good source for data about their clients.

But the most interesting sessions were held by companies that openly shared their experience about evolution of their Big Data solutions based mainly on open source software. In this way Adam Kawa from Spotify showed how they based their platform on Hadoop cluster starting from a single server to a few hundreds nowadays. To me that seems like a good way to grow and adapt easily to changing business needs and altering external conditions.

Nasza Klasa – a Polish Facebook competitor had a very good presentation on several dimensions connected to challenges in Big Data solutions that might be used for summarisation of this post:

1. Lack of legal regulations – Currently there are no clear regulations on how the data might be used and how to make money out of it. It is especially important for social portals where all our personal information might be used for different kinds of analysis and sold in aggregated or non-aggregated form. But the laws might be changed soon, thus changing the business too.
2. Big Data is a bit like research – it is hard to predict return on investment on Big Data as it is a novelty but also a very powerful tool. For many who are looking into this the challenge is internal, to convince executives to invest in something that is still rather vague.
3. Lack of data scientists – even if there are tools for operating on Big Data, there is a huge lack of skilled people – Big Data operators. These are not IT people nor developers but rather open-minded people with a good mathematical background able to understand and find patterns in a constantly growing stream of various structured and unstructured information.

As I stated at the beginning of this post, Big Data is also a Big Challenge for a number of companies that would like to be ahead of the competition. I truly believe we at Findwise can help a lot within this area, we have both the technical expertise and experience on how to put Big Data to use in a business.

Don’t miss the opportunity to visit the biggest search event in Northern Europe focusing entirely on how to find and display corporate information.

Last year we took the first steps towards creating a new industry event for everyone interested in search and findability. This year we are taking it to the next level!

The agenda is work in progress but we can promise a day full of inspiration, knowledge sharing and networking opportunities to help develop your business, personal skills and professional network.

The event takes place in Stockholm on May 30th. For more details check out the event here.

Hope to see you there!

We’ve already written about the new concept called Graph Search, which is being popularized by Facebook. Wouldn’t it be cool if we applied this to the enterprise as well, as I wrote in an earlier blog post on Enterprise Graph Search? That’s what Australian startup company Lumanetix thinks, when they created the SPAR-K graph search engine for the enterprise.

As seen in the screenshots of the product, the product do queries against relational databases with linked data objects such as Movies linked to People in Casts, or Managers of Departments in an organization. One difference to Facebook graph search is the more Google-like query syntax which is keyword-based where Facebook uses natural language processing to describe specific queries.

It’s exciting to see that the market is picking up speed with new innovations in the enterprise search field, as Lumanetix SPAR-K is an example of.

/Christian Ubbesen

At Findwise we are experts in helping organizations setting up systems to find their corporate information and presenting it in every way imaginable. But we are not only good at finding the right information, we must also be good at finding the right people to come work for us.

The people working here are highly skilled consultants within different areas such as business consulting, information management, text analytics, user experience, system design etc. They all have two things in common; they were handpicked to work here because of their unique expertise and passion for search technology and they could all easily have chosen to work someplace else.

Our way of finding these people is based on the notion that talent attracts talent. That means in order to find new talented people we must make the ones who are already working here thrive and come to work each day filled with joy and anticipation. That creates the ripple effect we need to compete for talent with the giants of the IT industry.

How do we accomplish that? Well, talented people must be respected as equals and be given the freedom to create and innovate. You don’t hire a talent to tell him or her exactly how to do what they are talented at. That would be like hiring Michael Jackson and then telling him how to write a hit song. We want our talented people to feel encouraged to act independently and bravely, that is how their talents best are put to use for Findwise and our clients.

Within the corporate world today these are still surprisingly uncommon ideas and two of the major daily newspapers in Sweden have both written about our approach to management, an article in SvD last fall and one recently in Dagens Nyheter.

To summarize the news, Findwise approach to management is to continue to uphold an open, trusting environment with a flat corporate structure, in which flexible working hours, freedom and own responsibility are principal. »Use your own judgment« is the golden rule which encourages a fighting spirit and the desire to develop new ideas.

We gladly walk this talk. And it has paid off so far. We are now employing more than 90 people, have managed more than 700 client projects and have enjoyed steady and profitable growth since the start in 2005. We are well on our way towards becoming a world-leading enterprise in our field, and it is all thanks to the talented people who work here.

/Olof Belfrage

Tomorrow morning I leave Gothenburg to attend the PROMISE Technology Transfer day @ CeBIT 2013 in Hanover, Germany.

The event is a workshop introducing its participants to methodologies for the systematic evaluation and monitoring of search engines, and for discussing future trends and requirements for the next generation of information access systems. In other words, it is right up our alley at Findwise.

As Director of Research at Findwise I will speak about Search as a Service. If you are at the event or just nearby I would be happy to meet up and have a chat.  I will be around from Tuesday March 5 until Thursday March 7. Feel free to email me, henrik.strindberg@findwise.com or give me a call at +46709443905.

Hope to see you there!

Leaving both the SharePoint Conference in Las Vegas and the recent European SharePoint Conference in Copenhagen behind, Findwise continues sharing impressions about the new search in SharePoint 2013! We have previously given an overview of what is new in search in SharePoint 2013 and discussed Microsoft’s focus areas for the release. In this post, we focus more on the ranking of the search results using the query rules.

Understanding user intent in search is one of the key developments in the new release. The screenshots below, showing out-of-the-box functionality on some sample content, exemplify how the search engine adapts to the user query. Keywords such as ‘deck’, ‘expert’, or ‘video’ can express the user’s needs and expectations for different search results and information types, and what the search engine does in this case is promoting those results that have a higher probability to be relevant to the user’s search.

Source: Microsoft

The adaptability of the search results can seem remarkable, as we see in these examples, aiming to provide more relevant search results through a better understanding of the user intent. Actually, this is powered by a new feature in SharePoint 2013 called query rules. Even more interesting maybe is that you can define your own custom query rules matching your specific needs without writing any code!

The simplest query rule would be to promote a specific result for a given search query. For example, you can promote a product’s instruction manual when the users search for that product name. Previously, in SharePoint 2010, you were able to define such promoted results (or “best bets”) using the Search Keywords. The query rules in SharePoint 2013 extend this functionality, providing an easy way to create powerful search experiences that adapt to user intent and business needs.

When defining a query rule, there are two main things to consider: conditions and corresponding actions. The conditions specify when the rule will be applied and the actions specify what to do when the rule is matched. There are six different condition types and three action types that can be defined.

For example, a query condition can be that a query keyword matches a specified phrase or a term from a dictionary (such as ‘picture’, ‘download’ or a product name from the term store), or that the query is more popular for a certain result type (such as images when for example searching for ‘cameras’), or that it matches a given regular expression (useful for matching phone numbers for example). The correlated actions can consist of promoting individual results on top of the ranked search results (promoting for example the image library), promoting a group of search results (such as image results, or search results federated from a web search engine), or changing the ranking of the search results by modifying the query (by changing the sorting of results or filtering on a content type). Another thing to consider is where you define the rule. Query rules can be created at Search Service Application, Site Collection, or Site level. The rules are inherited by default but you can remove, add, configure and change the order of query rules at each level. Fortunately, it also allows you to test a query and see which rules will fire.

There is one more thing though that you need to take into account: some features of query rules are limited in some of the licensing plans. Some plans only allow you to add the promoted results, and the more advanced actions on query rules are disabled. Check TechNet for guidelines on managing query rules and a list of features available across different licensing plans.

With the query rules, you have the freedom and power to change the search experience and adapt it to your needs. Defining the right keywords to be matched on the user queries and mapping the conditions with the relevant actions is easy but the process must undoubtedly be well managed. The management of the query rules should definitely be part of your SharePoint 2013 search governance strategy.

Let’s have a chat about how you can create great search experiences that match your specific users and business needs!

Each year CIO Sweden conducts a trend survey among Sweden’s CIOs. They also host an annual event where they discuss the results and the CIOs from some of Sweden’s largest companies talk about their vision. On February 6 I attended this year’s CIO Trends event at Münchenbryggeriet in Stockholm.

The main conclusion from this year’s survey is that compared to last year not that many things have changed. However, one interesting change this year was that last year’s most important trend, cloud and cloud solutions, had been kicked down by Mobility. Mobility as in easiness to move around not only in the office but also in large scales around the world. Information should always be on your fingertip no matter the device or connection. The Cloud is still a hot topic and focus on that is still high among companies. I guess henceforth we will see more of a combination of the two where you use cloud to create more mobility.

Fun fact of the day from the CIOs of Sweden: The most common CIO in Sweden is Male (84%), around 45-49 years old (33%) and don’t like shopping (2%).

//Ludvig Aldrin still Sweden’s youngest CIO (CIO’s under 30,  1%)

The 4th Swedish Language Technology Conference (SLTC) was held in Lund on 24-26 October 2012.
It is a biennial event organized by prominent research centres in Sweden.
The conference is, therefore, an excellent venue to exchange ideas with Swedish researchers in the field of Natural Language Processing (NLP), as well as present own research and be updated of the state-of-the-art in most of the areas of Text Analytics (TA).

This year Findwise participated in two tracks – in a workshop and in the main conference.
As the area of Search Analytics (SA) is very important to us, we decided to be proactive and sent an application to organize a workshop on the topic of “Exploratory Query Log Analysis” in connection with the main conference. The application was granted and the workshop was very successful. It gathered researchers who work in the area of SA from very different perspective – from utilizing deep Machine Learning to discover users’ intent,  to looking at query logs as a totally new genre. I will do a follow-up on that in another post. All the contributions to the workshop will also be uploaded on our research page.

As for the main conference, we had two papers accepted for presentation. The first one dealt with the topic of document summarization – both single and multidocument summarization
(http://www.slideshare.net/findwise/extractive-document-summarization-an-unsupervised-approach).
The second paper was about detecting Named Enities in Swedish
(http://www.slideshare.net/findwise/identification-of-entities-in-swedish).

These two papers presented de facto state-of-the-art results for Swedish both when it comes to document summarization and Named Entity Recognition (NER). As for the former task, there is neither a standard corpus for evaluation of summarization systems, nor many previous results and just few other systems which made it unfeasible to compare our own system with. Thus, we have contributed two things to the research in document summarization – a Swedish corpus based on featured Wikipedia articles to be used for evaluation and a system based on unsupervised Machine Learning, which by relying on domain boosting achieves state-of-the-art results for English and Swedish. Our system can be further improved by relying on our enhanced NER and Coreference resolution modules.

As for the NER paper, our Entity recognition system for Swedish achieves 74.0% F-score, which is 4% higher than another study presented simultaneously at SLTC (http://www.ling.su.se/english/nlp/tools/stagger). Both systems were evaluated on the same corpus, which is considered a de facto standard for evaluation of different NLP resources for Swedish. The unlabelled score (i.e. no fine-grained division of classes but just entity vs non-entity) of our system achieved 91.3% F-score (93.1% Precision and 89.6% Recall). When identifying people, the Findwise NER system achieves 78.1% Precision and 90.5% Recall (83.9% F-score).

So, what did we take home from the conference? We were really happy to see that the tools we develop for our customers are not something mediocre but rather something that is of very high quality and is the state-of-the-art in Swedish NLP. We actively share our results and our corpora for research perposes. Findwise showed keen interest in cooperating with other researchers in developing better tools and systems in the area of NLP and Text Analytics. And this I think is a huge bonus to all our current and prospective customers – we actively follow the current trends in the research community and cooperate with researchers, and our products do incorporate the latest findings in the field, which make us leverage both high quality and cutting-edge technology.

As we continuously improve our products, we have also released a Polish NER and some work has been initiated on Danish and Norwegian ones. More NLP components will be soon available for demo and testing on our research page.

Facebook will soon launch their new Graph Search to the general public, and it has received a lot of interest lately.

With graph search, the users will be able to query the social graph that millions of people have constructed over the years when friending each other and putting in more and more personal information about themselves and their friends in the vast Facebook database. It will be possible to query for friends of friends who have similar interests as you, and invite them to a party, or to query for companies where people with similar beliefs as you work, and so on and so forth. The information that is already available, will all the sudden become much more accessible through the power of graph search.

How can we bring this to an enterprise search environment? Well, there are lots of graphs in the enterprise as well to query, both social and other types. For example, how about being able to query for people that have been members of a project in the last three years that involved putting a new product successfully to the market. This would be an interesting list of people to know about, if you’re a marketing director that want to assemble a team in the company, to create a new product and make sure it succeeds in the market.

If we dissect graph search, we will find three important concepts:

1. The information we want to query against don’t only need to be indexed into one central search engine, but also the relations and attributes of all information objects need to be normalized to create the relational graph and have standard attributes to query against. We could use the Open Graph Protocol as the foundation.
2. We need a parser that take human language and converts it to a formal query language that a search engine understands. We might want to query in different human languages as well.
3. The presentation of results should be adapted to the kind of information sought for. In Facebook’s example, if you query for people you will get a list of people with their pictures and some relevant personal information in the result list, and if you query for pictures you will get a collage of pictures (similar to the Google image search).

So the recipe to success is to give the information management part of the project a big focus, making sure to create a unified information model of the content to be indexed. Then create a query parser for natural language based on actual user behavior, and the same user studies would also give us information on how to visualize the different result set types.

I believe we will see more of these kind of solutions in the coming years in the enterprise search market, and look forward exploring the possibilities together with our clients.

At the end of November 2012 top predictive analytics experts, practitioners, authors and business thought leaders met in London at Predictive Analytics World conference. Cameral nature of the conference combined with great variety of experiences brought by over 60 attendees and speakers made a unique opportunity to dive into the topic from Findwise perspective.

Dive into Big Data

In the Opening Keynote, presented by Program Chairman PhD Geert Verstraeten, we could hear about ways to increase the impact of Predictive Analytics. Unsurprisingly a lot of fuzz is about embracing Big Data.  As analysts have more and more data to process, their need for new tools is obvious. But business will cherish Big Data platforms only if it sees value behind it. Thus in my opinion before everything else that has impact on successful Big Data Analytics we should consider improving business-oriented communication. Even the most valuable data has no value if you can’t convince decision makers that it’s worth digging it.

But beeing able to clearly present benefits is not everything. Analysts must strive to create specific indicators and variables that are empirically measurable. Choose the right battles. As Gregory Piatetsky (data mining and predictive analytics expert) said: more data beats better algorithms, but better questions beat more data.

Finally, aim for impact. If you have a call center and want to persuade customers not to resign from your services, then it’s not wise just to call everyone. But it might also not be wise to call everyone you predict to have high risk of leaving. Even if as a result you loose less clients, there might be a large group of customers that will leave only because of the call. Such customers may also be predicted. And as you split high risk of leaving clients into “persuadable” ones and “touchy” ones, you are able to fully leverage your analytics potencial.

Find it exciting

Greatest thing about Predictive Analytics World 2012 was how diverse the presentations were. Many successful business cases from a large variety of domains and a lot of inspiring speeches makes it hard not to get at least a bit excited about Predictive Analytics.

From banking and financial scenarios, through sport training and performance prediction in rugby team (if you like at least one of: baseball, Predictive Analytics or Brad Pitt, I recommend you watch Moneyball movie). Not to mention Case Study about reducing youth unemployment in England. But there are two particular presentations I would like to say a word about.

First of them was a Case Study on Predicting Investor Behavior in First Social Media Sentiment-Based Hedge Fund presented by Alexander Farfuła – Chief Data Scientist at MarketPsy Capital LLC. I find it very interesting because it shows how powerful Big Data can be. By using massive amount of social media data (e.g. Twitter), they managed to predict a lot of global market behavior in certain industries. That is the essence of Big Data – harness large amount of small information chunks that are useless alone, to get useful Big Picture.

Second one was presented by Martine George – Head of Marketing Analytics & Research at BNP Paribas Fortis in Belgium. She had a really great presentation about developing and growing teams of predictive analysts. As the topic is brisk at Findwise and probably in every company interested in analytics and Big Data, I was pleased to learn so much and talk about it later on in person.

Big (Data) Picture

Day after the conference John Elder from Elder Research led an excellent workshop. What was really nice is that we’ve concentrated on the concepts not the equations. It was like a semester in one day – a big picture that can be digested into technical knowledge over time. But most valuable general conclusion was twofold:

• Leverage – an incremental improvement will matter! When your turnover can be counted in millions of dollars even half percent of saving mean large additional revenue.
• Low hanging fruit – there is lot to gain what nobody else has tried yet. That includes reaching for new kinds of data (text data, social media data) and daring to make use of it in a new, cool way with tools that weren’t there couple of years ago.

Plateau of Productivity

As a conclusion, I would say that Predictive Analytics has become a mature, one of the most useful disciplines on the market. As in the famous Gartner Hype, Predictive Analytics reached has reached the Plateau of Productivity. Though often ungrateful, requiring lots of resources, money and time, it can offer your company a successful future.

Over 10,000 attendees from 85 countries, more than 200 sponsors and exhibitors, and over 250 sessions. Besides these impressive numbers, the 2012 SharePoint conference in Las Vegas has also marked the launch of the new version of SharePoint. Findwise was there to learn and is now sharing with you the news about enterprise search in SharePoint 2013.

In the keynote presentation on the first day of the conference, Jared Spataro (Senior Director, SharePoint Product Management at Microsoft) mentions the three big bets made for the SharePoint 2013 product: CLOUD, MOBILE, and SOCIAL. This post tries to provide a brief overview of what these three buzzwords mean for the enterprise search solution in SharePoint 2013. Before reading this, also check out our previous post about search in SharePoint 2013 to get a taste of what’s new in search.

Search in the cloud

While you have probably heard the saying that “the cloud has altered the economics of computing” (Jared Spataro), you might be wondering how to get there. How to go from where you are now to the so-called cloud. The answer for search is that SharePoint 2013 provides a hybrid approach that helps out in this transition. Hybrid search promises to be the bridge between on-premises and the cloud.

The search results from the cloud and those from on-premise can be shown on the same page with the use of the “result blocks”. The result block, new to SharePoint 2013, is a block of results that are individually ranked and are grouped according to a “query rule”. In short, a query rule defines a condition and an action to be fired when the condition is met. With the use of the result blocks, you can display the search results for content coming from the cloud when searching from an on-premises site and the other way around (depending whether you want the search to be one-way or bidirectional), and you can also conditionally enable these result blocks depending on the query (for example, queries matching specific words or regular expressions).

Screenshot from the post Hybrid search of the Microsoft SharePoint Team Blog showing how results from the cloud are integrated in the search results page when the user searches from an on-premises SharePoint 2013 site.

Before making the decision to move to the cloud, it is wise to check the current features availability for both online and on-premise solutions on TechNet.

Mobile devices

With SharePoint 2013, Microsoft has added native mobile apps for Windows, Windows Phone, iPhone, and iPad, and support across different mobile devices (TechNet), which provides access to information and people wherever the users are searching from.

Also important to mention when talking about mobile, is that the improved REST API widens the extensibility options and allows easy development of custom user experiences across different platforms and devices. The search REST API provides access to the keyword query language parameters, and combining this with a bit of JavaScript and HTML allows developers to quickly start building Apps with custom search experiences and making all information available across devices.

Social search

In the same keynote, Jared Spataro said that Microsoft has “integrated social very deeply into the product, creating new experiences that are really designed to help people collaborate more easily and help companies become more agile.” This was also conveyed by the presence of the two founders of the enterprise social network Yammer in the keynote presentation. The new social features integration means that the information about people following content, people following other people, tags, mentions, posts, discussions, are not only searchable but can be used in improving the relevance of the search results and improving the user experience overall. Also, many of the social features are driven by search, such as the recommendations for people or documents to follow.

Whether you are trying to find an answer to a problem to which the solution has already been posted by somebody else, or whether you are trying to find a person with the right expertise through the people search, SharePoint 2013 provides a more robust and richer social search experience than its previous versions. And the possibilities to extend the out-of-the-box capabilities must be very attractive to businesses that are for example looking to combine the social interactivity inside SharePoint with people data stored in other sources (CRM solutions, file shares, time tracking applications, etc).

Stay tuned!

It was indeed an awesome conference, well organized, but most of the times it was hard to decide which presentation to choose from the many good sessions running at the same time. Luckily (or wisely), we had more than one Findwizard on location!

This post is part of our series of reports from the SharePoint 2012 Conference. Keep an eye on the Findability blog for part two of our report from the biggest SharePoint conference of 2012!

Today many IT departments are investing in mobile technology to make their internal enterprise content accessible in employees mobile phones and other mobile devices. We all want to be able to work without being at the office, and without having to run around with the job laptop. Imagine being at a business lunch and you want to pull up some presentation you have on the company intranet, why not just use the mobile phone?

In some organizations this is possible, and in some it still isnt’t. And in most organizations you don’t have access to all the documents and content available internally in document management systems, file shares and databases. And even if you did have access to the content in your mobile phone, you wouldn’t want to start browsing for it because it’s just too cumbersome to find it.

Here’s an idea for you: why not utilize the enterprise search platform to make the content both accessible, findable and readable?

First step is to make the content accessible. Since all content is already being indexed by the search engine, it’s already in one central place, at least in text representation. If you have a solution in place for having mobile phones access the company intranet, it should be fairly simple to open up for mobile devices to access the enterprise search web interface as well, with security credentials still in place.

Secondly the content need to be findable, and what better way to find information on a mobile phone is there than to search for it? With mobile search user interface patterns this will be much more efficient than traditional browsing for information.

And third, when you have found your document, you can use search engine features such as fingernail previews, automatic summarization and HTML conversion to make it easily readable on the mobile device.

Check out my presentation on SlideShare on accessing content with mobile search as well.

If you already have an enterprise search platform in place, why not start researching how to utilize it to make your enterprise content accessible on your mobile phone?

And if you don’t have an enterprise search platform in place, I suppose you now have yet another reason to add to your business case for investing in one.

Google released Google Search Appliance, GSA 7.0, in early October. Magnus Ebbesson and I joined the Google hosted pre sales conference in Zürich where we had some of the new functionality presented and what the future will bring to the platform. Google is really putting an effort into their platform, and it gets stronger for each release. Personally I tend to like hardware and security updates the most but I have to say that some of the new features are impressive and have great potential. I have had the opportunity to try them out for a while now.

In late November we held a breakfast seminar at the office in Gothenburg where we talked about GSA in general with a focus on GSA 7.0 and the new features. My impression is that the translate functionality is very attractive for larger enterprises, while the previews brings a big wow-factor in general. The possibility of configuring ACLs for several domains is great too, many larger enterprises tend to have several domains. The entity extraction is of course interesting and can be very useful; a processing framework would enhance this even further however.

It is also nice to see that Google is improving the hardware. The robustness is a really strong argument for selecting GSA.

It’s impressive to see how many languages the GSA can handle and how quickly it performs the translation. The user will be required to handle basic knowledge of the foreign language since the query is not translated. However it is reasonably common to have a corporate language witch most of the employees handle.

The preview functionality is a very welcome feature. The fact that it can highlight pages within a document is really nice. I have played around to use it through our Jellyfish API with some extent of success. Below are two examples of usage with the preview functionality.

A few thoughts

At the conference we attended in Zürich, Google mentioned what they are aiming to improve the built in template in the GSA. The standard template is nice, and makes setting up a decent graphical interface possible for almost no cost.

My experience is however that companies want to do the frontend integrated with their own systems. Also, we tend to use search for more purposes than the standard usage. Search driven intranets, where you build intranet sites based on search results, is an example where the search is used in a different manner.

A concept that we have introduced at Findwise is search as a service. It means that the search engine is a stand-alone product that has APIs that makes it easy to send data to it and extract data from it. We have created our own APIs around the GSA to make this possible. An easy way to extract data based on filtering of data is essential.

What I would like to see in the GSA is easier integration with performing search, such as a rest or soap service for easy integration of creating search clients. This would make it easier to integrate functionality, such as security, externally. Basically you tell the client who the current user is and then the client handles the rest. It would also increase maintainability in the sense of new and changing functionality does not require a new implementation for how to parse the xml response.

I would also like to see a bigger focus of documentation of how to use functionality, previews and translation, externally.

Final words

My feeling is that the GSA is getting stronger and I like the new features in GSA 7.0. Google have succeeded to announce that they are continuously aiming to improve their product and I am looking forward for future releases. I hope the GSA will take a step closer to the search as a service concept and the addition of a processing framework would enhance it even further. The future will tell.

Vårdaktörsportalen (VAP) is a portal for health care providers made by Västra Götalandsregionen (VGR). This portal makes information from a number of reliable and authorised sources findable and accessible for the people who need it in their daily work. This stretches from doctors and nurses to medical secretaries primarily located in the region of Västra Götaland, Sweden. The site and most of its features and information is also accesible (in Swedish) to anyone through http://vap.vgregion.se. In November 2012 the first version of this site went live and Findwise had a big role in the creation of this search centric site. The main source of information for VAP is the regional guidelines found in a document repository within VGR but some other external sources are also included. These include trustworthy authorities like

• Socialstyrelsen
• Läkemedelsverket
• SBU
• TLV
• 1177 - the public health care information site for citizens

This search solution is built around the open source search engine Apache Solr and our common tools for processing and indexing. For this site we have also implemented a rather unique metadata enhancement service that automatically extracts keywords from the document to index and attaches it as metadata. The keyword extraction is based on information from the medical term database SweMeSH. More information (in Swedish) can be found on Google Code. We also include synonyms to keywords to increase recall, making it easier to find documents regardless of what synonym used.

The metadata enhancement service was included because the quality of metadata on the external sites was not very good. VGR will work with the above mentioned authorities to try and make them understand that they would benefit if they improved their data. The source 1177 stands out with very good meta data and overall good quality texts.

We have conducted a user study to see how this first version is able to satisfy user demands. The result of that study shows that some local sources are missing, but a general positive feedback on the idea and the graphical design was collected.

Findwise is looking forward to continue working on VAP with VGR in the future to make it an even better tool.

Related links:

• http://vap.vgregion.se
• http://webbfunktion.com/grafisk-form-pa-vardaktorsportalen/
• http://vardaktorsportalen.se/

Enterprise search in practice presentation has two main focuses. First, to present some interesting and sometimes rather contradicting findings from the Enterprise Search and Findability survey 2012. Second, to introduce an holistic approach to implementing search technology involving five different aspects that are all important to succeed and to reach findability rather than just the ability to search.

Presented at Gilbane Conference 2012 in Boston USA on the 28th of November by Mattias Ellison.

Best Practices for Enterprise Search, from a practitioner perspective. The content of this presentation is more focused on actionable tasks and processes than search technology. The best practices are based on data from the Enterprise Search and Findability Survey, previous research, empirical evidence and of course the accumulated collective experience by Findwise consultants.

Your comments on this “Best Practices for Enterprise Search” presentation are very much appreciated!

Presented at Intranett 2012 in Oslo, Norway, 22 November 2012 by Kristian Norling.

This was presented 8 November at J. Boye 2012 Conference in Aarhus, Denmark, by Kristian Norling.

Presentation Summary

There is a lot of talk about social, big data, cloud, digital workplace and semantic web. But what about search, is there anything interesting happening within enterprise search and findability? Or is enterprise search dead?

In the spring of 2012,  we conducted a global survey on Enterprise Search and Findability. The resulting report based on the answers from survey tells us what the leading practitioners are doing and gives guidance for what you can do to make your organisation’s enterprise search and findability better in 2013.

This presentation will give you a sneak peak into the near future and trends of enterprise search, based on data form the survey and what the leaders that are satisfied with their search solutions do.

Topics on Enterprise Search

•  Help me! Content overload!
• The importance of context
• Digging for gold with search analytics
• What has trust to do with enterprise search?
• Social search? Are you serious?
• Oh, and that mobile thing

This tutorial was done on the 6th of November at J. Boye 2012 conference in Aarhus Denmark. Tutorial was done by Kristian Norling.

Findability and Your Content

As the amount of content continues to increase, new approaches are required to provide good user experiences. Findability has been introduced as a new term among content strategists and information architects and is most easily explained as:

“A state where all information is findable and an approach to reaching that state.”

Search technology is readily used to make information findable, but as many have realized technology alone is unfortunately not enough. To achieve findability additional activities across several important dimensions such as business, user, information and organisation are needed.

Search engine optimisation is one aspect of findability and many of the principles from SEO works in a intranet or website search context. This is sometimes called Enterprise Search Engine Optimisation (ESEO). Getting findability to work well for your website or intranet is a difficult task, that needs continuos work. It requires stamina, persistence, endurance, patience and of course time and money (resources).

Tutorial Topics

In this tutorial you will take a deep dive into the many aspects of findability, with some good practices on how to improve findability:

• Enterprise Search Engines vs Web Search
• Governance
• Organisation
• User involvement
• Optimise content for findability
• Metadata
• Search Analytics

Brief Outline

We will start some very brief theory and then use real examples and also talk about what organisations that are most satisfied with their findability do.

Experience level

Participants should have some intranet/website experience. A basic understanding of HTML, with some previous work with content management will make your tutorial experience even better. A bonus if you have done some Search Engine Optimisation (SEO) for public websites.

Approaches for Identifying Information Access Needs and to Build a Business Case for Enterprise Search and Findability

We have defined a number of alternative approaches to identify the need and value of search-driven findability to support an organisation or a specific process. In other words, different methods to build a business case for enterprise search in a specific organization or process.

Task oriented

Analysing information access needs in relation to specific work task within a business process (by utilizing e.g. the method developed by Byström/Strindberg or the Customer Carewords method).

Process oriented

Mapping the process flow of sequential and dependent (value-adding) activities and the related information access needs, Analysing the dependencies/accessibility of information systems in the different activities (e.g. by using some kind of Business Process Modeling, like the Astrakan-method).

Decision oriented

Identifying and analysing the decision points and the related information access needs within a process.

Risk oriented

Analysing situations within a process or for decision points where the right information was not available. Or even worse if there only was old and unvalid information available? What would have been the outcome of the situation if the desired/needed information had been available? How can we avoid for this scenario to be repeated? Inspired by Lynda Moulton at LWM Technology Services and Martin White of IntranetFocus.

Effect oriented

Determine the desired effects from search-driven findability and define measuring point to follow up the effects over time. Includes also identification of the related target groups/personas and their information access needs to be fulfilled for the effects to be reached (based on the InUse method and previous work at Ericsson (Case Study) and Forsmark (Case Study). An enhanced variant of this method is currently being developed in a project at Chalmers.

Our ambition is to use these methods to help organisations identify information access needs and findability barriers and to help motivate search investments. The analysis could for example be performed by our Findability Business Consultants as part of an in-depth findability review focusing on either an existing application or a specific business process.

Presented by Paula Petcu and Ludvig Aldrin at Microsoft Campus Days, #cddk12, 31 October 2012, in Copenhagen Denmark.

Learn how easy it is to build powerful search experiences using SharePoint 2013.
The presentation will showcase the Search in SharePoint 2013 and provide a technical and functional walkthrough of what is new.  The presentation will take you through the out-of-box search experience, and you will get tips and tricks on how to extend the search platform to create a great custom experience for your users. Also discussed is the new search architecture and how search plays a central role in the new SharePoint 2013.

The presentation is divided into three parts. The first part will include an overview of search and will walk you through the out-of-the-box search experience, showcasing the new or improved functionalities and discussing how this affects the search experience. This part is all about finding what the users are looking for and getting answers to their questions. The new product revolves around the user more than ever, and you will be able to see this in the new search experience.

Then information about about the new search architecture, and this will make the transition to the second part of the presentation, which is all about extending. And a bit about executing queries under the new architecture and more specifically on how to extend the way they are executed.

Prior to SharePoint 2013, the only way to inspect and manipulate managed property values for items before being added to the search index was by extending the item processing pipeline in FAST Search for SharePoint. Clients using SharePoint search were out of luck as the functionality was not available to them. Now, MS has introduced three new items for content processing and enrichment: parsers, custom entity extractors, and web service callouts. These new features will be featured and one of the demoed.

But what happens next to the search engine? The third part of the presentation will be about the governance of your search solution. More specifically, it will focus on search analytics.

There has been a lot of buzz about the upcoming release of Microsoft’s SharePoint 2013, how about the search in SharePoint 2013? The SharePoint Server 2013 Preview has been available for download since July this year, and a few days ago the new SharePoint has reached Release to Manufacturing (RTM) with general availability expected for the first quarter of 2013.

If you currently have an implementation of SharePoint in your company, you are probably wondering what the new SharePoint can add to your business. Microsoft’s catchphrase for the new SharePoint is that “SharePoint 2013 is the new way to work together”. If you look at it from a tech perspective, amongst other features, SharePoint 2013 introduces a cloud app model and marketplace, a redesign of the user experience, an expansion of collaboration tools with social features (such as microblogging and activity feeds), and enhanced search functionality. There are also some features that have been deprecated or removed in the new product, and you can check these on TechNet.

Let’s skip now to the new search experience provided out-of-the-box by SharePoint 2013. The new product revolves around the user more than ever, and that can be seen in search as well. Here are just a few of the new or improved functionalities. A hover panel to the right of a search result allows users to quickly inspect content. For example, it allows users to preview a document and take actions based on document type. Users can find and navigate to past search results from the query suggestions box, and previously clicked results are promoted in the results ranking. The refiners panel now reflects more accurately the entities in your content (deep refiners) and visual refiners are available out-of-the-box. Social recommendations are powered by users’ search patterns, and video and audio have been introduced as new content types. Some of the developers reading this post will also be happy to hear that SharePoint 2013 natively supports PDF files, meaning that you are not required anymore to install a third-party iFilter to be able to index PDF files!

Search Overview in SharePoint 2013, Source: Microsoft SharePoint Team Blog

While the out-of-the-box SharePoint 2013 search experience sounds exciting, you may also be wondering how much customization and extensibility opportunities you have. You can of course search content outside SharePoint and several connectors that allow you to get content from repositories such as file shares, the web, Documentum, Lotus Notes and public Exchange folders are included. Without any code, you can use the query rules to combine user searches with business rules. Also, you can associate result types with custom templates to enrich the user experience. Developers can now extend content processing and enrichment, which previously could have only be achieved using FAST Search for SharePoint. More than that, organizations have the ability to extend the search experience through a RESTful API.

This post does not cover all the functionalities and if you would like to read more about what changes the new SharePoint release brings, you can start by checking the TechNet material and following the SharePoint Team Blog and the Findwise Findability Blog, and then get in touch with us if you are considering implementing SharePoint 2013 in your organization or company.

Findwise will attend the SharePoint Conference 2012 in Las Vegas USA between 12-15 November and this will be a great opportunity to learn more about the upcoming SharePoint. We will report from the conference from a findability and enterprise search perspective. Findwise has years of experience in working with FAST ESP and SharePoint, and is looking forward to discussing how SharePoint 2013 can help you in your future enterprise search implementation.

Every second, more and more information is created and stored in various applications. corporate websites, intranets, SharePoint sites, document management systems, social platforms and many more – inside the firewall the growth of information is similar to that of the internet. However, even though major players on the web have shown that navigation can’t compete with search, the Enterprise Search and Findability Report shows that most organisations have only a small or even a non-existing budget for search.

Web Search and Enterprise Search

Web search engines like Google has made search look easy. For enterprise search, some vendors give promises of a magic box. Buy a search engine, plug it in and wait for the magic to happen! Imagine the disappointment when both search results and performance are poor and users can’t find what they are looking for…

When you start planning your enterprise search project you soon realize the complexity and challenge – how do you meet the expectations created by Google?

The Presentation

This presentation was originally presented at the joint NSW KM Forum and IIM September event in Sydney, Australia by Mattias Brunnert. It contains topics as:

• Why search is important and how to measure success
• Why Enterprise Search and Information Management should be friends
• How to kick off your search program

Understanding what your customer thinks about your company, your products and your service can be done in many different ways. Today companies regularly analyze sales statistics, customer surveys and conduct market analysis. But to get the whole picture of the voice of customer, we need to consider the information that is not captured in a structured way in databases or questionnaires.

I attended the Text Analytics Summit earlier this year in London and was introduced to several real-life implementations of how text analytics tools and techniques are used to analyze text in different ways. There were applications for text analytics within pharmaceutical industry, defense and intelligence as well as other industries, but most common at the conference were the case studies within customer analytics.

For a few years now, the social media space has boomed as platforms of all kinds of human interaction and communication, and analyzing this unstructured information found on Twitter and Facebook can give corporations deeper insight into how their customers experience their products and services. But there’s also plenty of text-based information within an organization, that holds valuable insights about their customers, for instance notes being taken in customer service centers, as well as emails sent from customers. By combining both social media information with the internally available information, a company can get a more detailed understanding of their customers.

In its most basic form, the text analytics tools can analyze how different products are perceived in different customer groups. With sentiment analysis a marketing or product development department can understand if the products are retrieved in a positive, negative or just neutral manner. But the analysis could also be combined with other data, such as marketing campaign data, where traditional structured analysis would be combined with the textual analysis.

At the text analytics conference, several exciting solutions where presented, for example an European telecom company that used voice of customer analysis to listen in on the customer ‘buzz’ about their broadband internet services, and would get early warnings when customers where annoyed with the performance of the service, before customers started phoning the customer service. This analysis had become a part of the Quality of Service work at the company.

With the emergence of social media, and where more and more communication is done digitally, the tools and techniques for text analytics has improved and we now start to see very real business cases outside the universities. This is very promising for the adaptation of text analytics within the commercial industries.

“The Why and How of Findability” presented by Kristian Norling at the ScanJour Kundeseminar in Copenhagen, 6 September 2012. We can make information findable with good metadata. The metadata makes it possible to create browsable, structured and highly findable information. We can make findability (and enterprise search) better by looking at findability in five different dimensions.

Five dimensions of Findability

1. BUSINESS - Build solutions to support your business processes and goals

2. INFORMATION - Prepare information to make it findable

3. USERS - Build usable solutions based on user needs

4. ORGANISATION - Govern and improve your solution over time

5. SEARCH TECHNOLOGY - Build solutions based on state-of-the-art search technology

We have moved our Atom/RSS feed for this blog to http://feeds.findwise.com/findwise/findabilityblog

Please update your feeds.

4HHG2HPKVG23

Yesterday we (Kristian Hjelseth and Kristian Norling) participated in a great World Cafe event arranged by Steria in Norway. We did a Pecha Kucha inspired presentation (scroll down to the bottom of this blog post for the presentation) to introduce the subject of Enterprise Search and Findability and how to work more efficiently with the help of enterprise search. Afterwards there was a set of three round-table workshop with practitioners, where search related issues were discussed. We found the discussions very interesting, so we thought we should share some of the topics with a broader audience.

The attendees had answered a survey before coming to the World Cafe. In which 83,3% stated that finding the right information was critical for their business goals. But only 20,3% were satisfied with their current search solution, because 75% said it was hard or very hard to find the right information. More stats from a global survey on enterprise search that asked the same questions.

Unified Search

To have all the information that you would like to find in the same search was deemed very important for findability by the participants. The experience of search is that the users don’t know what to search for, but to make it even worse, they do not know where to look for the information! This is also confirmed by the Enterprise Search and Findability Survey that was done earlier this year. The report is available for download.

Trust

Google web search always comes up as an example of what “just works”. And it does work because they found a clever algorithm, PageRank, that basically measures the trustworthiness of information. Since PageRank is heavily dependent on inbound links this way of measuring trust is probably not going to work on an intranet where cross-referencing is not as common based on our experience. Most of the time it is not even possible to link stuff on the intranet, since the information is not accessible through http. Read more about it in this great in-depth article series on the difference between web search and enterprise search by Mark Bennet.

So how can we make search inside the firewall as good as web search? I think by connecting the information to the author. Trust builds between people based on their views of others. Simply put, someone has the authority over her peers either through rank (=organisation chart) or through trust. The trustworthiness can be based on the persons ability to connect to other people (we all probably know someone who knows “everyone”) or we trust someone based on the persons knowledge. More reading on the importance of trust in organisations. How to do this in practice? Some ideas in this post by BIll Ives. Also a good read: “How social is Enterprise Search?” by Jed Cawthorne. And finally another good post to read.

Metadata

By adding relevant metadata to information, we can make it more findable. There was discussions on the importance of strict and controlled metadata and how to handle user tagging. For an idea on how to think about metadata, read a blog post on how VGR used metadata by Kristian Norling.

Search Analytics

Before you start to do any major work with your current enterprise search solution, look at the search log files and analyze the data. You might be surprised in what you find. Search analytics is great if you want insight into what the user expects to find when they search. Watch this video for an introduction to Search Analytics in Practice.

Other subjects

• Access control and transparency
• Who owns search?
• Who owns the information?
• Personalization of search results

Understanding and utilizing the context of both people and topic (subject) is the future of enterprise search and findability. As we have seen the last few years, the amount of information that is created within organisations and elsewhere is growing exponentially. This makes it harder, day-by-day, to find the information that is relevant at any given moment. By organizing information based on topic, by using text analytics, better metadata, adding user tagging, sentiment analysis etc. it is possible to make findability better. A few examples are mentioned in this blog post series on information flow from 2010. The whole point of findability boils down to improving the information flow and access at any given time. Example on Information Flow from the Intranet of Region Västra Götaland.

In order to make sense of any arbitrary information we as humans usually need the help of someone familiar with the topic to help us makes sense of it and understand it. By both addressing the challenge of finding people with the right knowledge and finding the right information, we can contextually make the information more relevant and easier to find.

For example by doing search analytics and looking at usage patterns in general or by looking at how people with the same usage (search) patterns are going about finding information, we can give better suggestions. Also, recommendations of information produced or liked by people who are like you have a better chance of being relevant to you. By using Social Network Analysis, we should be able to find patterns in what information is in demand and how the informations flows. The analysis can of course also be used to find the supernodes, meaning the people through which information and connections flow. For example, email is a under-utilized source of information flow, knowledge, context and social network analysis.

On the 28th of August, at the World Café in Oslo, Kristian Norling will talk about findability and collaboration, with a focus on people and topic centric solutions. Examples from Region Västra Götaland and other projects will be made.