Information retreival: Ask someone you trust

Exploring why some users can be reluctant to use search engines, and how to make information retrieval a more 'human' process.

By Jonathan Trim.

I will let you into a secret… The secret of the very best user interface for search applications, which has been around since before search engines even existed: ask somebody else to do the search for you. In particular, ask somebody you trust and who has knowledge of the subject matter you are looking for.

It is a disappointing fact that no matter how much time is spent specifying, designing and tuning a search application, there is always going to be a (significant) group of users who will not use it and choose to let someone else do the search for them, especially where a firm’s culture is one of ‘support’ rather than ‘self-service’.

Why is this and how can we make search more attractive to use? In order to understand, we must first look at what discourages the use of search engines; how people use them; and how this differs from asking your friendly information professional.

What puts users off search?
1. They don’t know where to start
Different entry points for different types of information – for example, multiple search engines for different types of information raise questions such as ‘Where do I start’, or ‘I don’t know which repository the information is in’.

Also, users may not know what search terms to start with: ‘I don’t know enough background information about what I’m searching for’ or ‘I don’t know. what I don’t know’.

2. They don’t get good results
Search can often pinpoint irrelevant or incomplete information – we cannot afford for our lawyers to miss anything important, but if we give them too much they will give up searching. This is a classic ‘recall versus precision’ issue.

3. They find it too awkward to use
This can be due to a complex user interface or difficult query language. Also, the more sophisticated search engines become, the more difficult it can be to understand why certain results are returned. From stop-words to Bayesian or Probabilistic Latent Semantic Analysis, it can be disconcerting when search engines return results which contain very few or even none of the query terms, but are still reported to be 80 per cent relevant.

4. They find it too slow
No matter how good the results are, only very patient users will use a slow search engine and these are in short supply.

5. It interrupts the flow of their day to day work
They need to stop what they are working on and open another application. Overall, users are required to adapt their behaviour to the way the search engine works – and why would they do this when it’s far easier to ask someone else?

Understanding how users use search
When we actually do get users to work with search engines how can we expect them to behave?

Using very few search terms – looking at our own usage statistics, 60 per cent of users use two or less query terms, while 80 per cent use three or fewer. The reasons for this may include not knowing exactly what to search for, or keeping the query broader to ensure no results are missed (a longer query may result in very few or no results). The outcome can be a results set so large and broadly that they do not help;
Using search terms only vaguely related to what they are looking for – users are not necessarily subject-matter experts and they may not know the best search terms to use. They may only have a vague recollection or snippet of information about the content they need, hence it may not be apparent what search terms to use. The outcome could be irrelevant content being returned and this is compounded by the tendency to use few terms;
Only browsing the top results – a Cornell university study[1] showed that users rarely open results beyond the top two. Certainly, if the results do not appear on the first page, then they are unlikely to get viewed;
Revising query rather than scrolling through results – the study also found that users are more likely to change the query or try again, than scroll or page through the results;
Relying on titles and summaries to select results – it was also observed that when users review the results, they spend 30 per cent of the time reading the title and 43 per cent reading the summary or snippet. This obviously demonstrates the importance of having a good title for the content, but also of showing a useful summary;
Giving up – after a number of attempts, if they have not found what they are looking for, they will stop searching. If this happens a number of times, the overall confidence in the search solution fails and they may choose not to use it again;
‘Ego’ searching – looking for content which they have authored themselves. While not something likely to be performed regularly, if a user cannot find everything they have written, then they are likely to feel the search engine is not working properly, which may contribute to them not using it.

We have also found that browsing for content is at least as popular as searching, if not more so. Maybe this is because users know where to look for content; maybe they have no faith in the search; or maybe browsing enables them to start with a broad subject and steadily refine this to the information that they are looking for.

Why is the ‘human-human’ interface so good?
At the simplest level, putting a human being in front of a software application can remove any complexity of the user interface. In addition to this, they can:

Perform rudimental validating of queries, such as spell checking;
Take the activity of searching offline and only return to the users when results are found – in the meantime users can get on with their work;
After retrieving and reviewing results, they can suggest ways to narrow the search (for example, by industry) and then re-run the search on the user’s behalf.

However, this also introduces a communication overhead to the process, so there must be more benefit than this alone. In fact, the real benefit comes with getting an information professional to perform the search on behalf of users, so they are adding more value than just providing a set of hands to operate the computer. In addition, they can:

Apply knowledge of the business, its users, the type of work, and content to clarify and improve search queries;
Ask relevant questions to users to help refine queries;
Provide knowledge of where to look for the best, and different types,
of content;
Directly suggest particularly useful content to users – for example, best practice guides;
Pre-review results to ensure quality and relevance;
Provide access to a network of other people to contact in order to get the best information;
Learn from the interaction and use this experience in the future.

Figure one shows how this might work in practice and what value an information officer can bring. Such individuals can address most of the reasons why users are put off searching and compensate for behaviour they may exhibit if performing the search themselves.

By recommending that users talk to experts in the areas of interest, they are adding value that a simple search engine would never do and providing access to tacit knowledge outside of any repository.

Of course, for information officers this is their job and so they can bring the expertise and time required to provide this service. However, there is also still the overhead of communication and the issue of the time taken for this transaction, especially where the information officer needs to perform a search offline, before returning to the user to ask further questions or to finally deliver the results. There is also the question of the cost of providing this human interface.

How can we make search ‘more human’?
The advantages of the human interface can be summarised into two main areas:

1. The dialogue, which enables the user to start with a vague idea of what they are looking for and then refine this to a more specific set of criteria;
2. The subject matter expertise, which means that this dialogue is targeted at the needs of the user and their area of business.

The objective is therefore to try to simulate the dialogue, while making the search experience as accessible and personalised as possible to the user.

A single point of access is crucial. By providing a single interface for searching all information, including all internal content and external content providers, the problem of users not knowing where to start can be removed.

In addition, by collating information about the user and applying this to the search, the initial scope of a search can then be narrowed down and personalised to them. This will provide a significantly smaller and more targeted set of content. For example:

Retrieving information about their practice area and current jurisdiction from the HR system;
Enabling users to create personal preferences, which indicate their common areas of interest;
Using their past behaviour to suggest content which could be of interest;
By tailoring the content based on the current matter or transaction they are working on.

Since the user may initially choose search terms that are incorrect, too broad or even too narrow, dialogue needs to be simulated so that all three of these factors are addressed as part of the interaction. An example of how this could be achieved is:

1. User submits query with spelling mistake;
2. Search engine suggests correction – this provides a validation of the initial query. The important thing here is that the spell check needs to use the content set itself, rather than a dictionary. In this way, it can automatically suggest spelling corrections to names and highly technical terms, which would otherwise need to be added manually to the dictionary;
3. Search engine also suggests synonyms and related terms – this provides a mechanism to expand the query, in case the user has been too specific, or may have used a niche variation of a term. Synonyms are probably best handled manually to make sure they are truly synonymous, but related terms could be a mixture of manually associated terms and those automatically derived by the search engine;Search engine returns ‘best bets’ at the top of the results – these are cherry-picked content manually configured to be returned when specific queries are entered. For example, return a best-practice guide for confidentiality when a user searches for ‘NDA’;
5. Results are returned from a number of repositories, but the user can choose which to show – either in a single list or grouped by type of content. This enables the user to see a broad selection of content. They can then quickly narrow to a single repository and then return back out to see the full set;
6. Search engine suggests categories, which are relevant to the current results – this enables the user to filter by category, progressively narrowing down the number of results. For example, they could narrow the results to a single jurisdiction, client or author. This method can reduce an initially huge set of results down to single figures very quickly. This is also known as faceted searching;
7. Results include summaries relevant to query with query terms highlighted – this is fairly common practice, but nonetheless very important. If the user cannot quickly determine what a result is about, or why it has been returned, then they are unlikely to view it. An extension to this is to provide a ‘quickview’, which shows a version of the whole document with the query terms highlighted throughout;
8. Learning from past users – by tracking popular content (most viewed or commented on), you can boost results to place emphasis on those found useful by other users;
9. Type ahead – another useful method is to suggest query terms as the user is typing, narrowing the list down the more they type. This effectively satisfies the spell check on the fly and suggests complete queries that users have searched for in the past.

Connecting people
In addition to returning documents, allow users to find people who have relevant expertise. Based on a combination of explicit expertise (written bios and user profiles) and implicit expertise (the documents written, hours billed to particular transaction types, seniority in firm, and so on), the search engine will return a ranked list of people who can potentially help with the user’s query;

Embedding search into day to day working
Rather than expecting users to leave what they are working on to access a search interface, make it available from whatever application they most commonly use. The two most common applications used are e-mail and word processing. By providing better access to search from within these, it makes it more accessible to users. This could take the simple form of a search toolbar, which ideally returns results to a ‘docked pane’, such as Microsoft’s Research Pane; ranging to automatically suggesting content based on the text of the document or e-mail.

Other methods of seamlessly embedding search are to automatically highlight key concepts in the document – for example, companies, legislation, citations – and enable users to click on these to run searches.

Through these approaches, the end result is that the user has effectively created an advanced search query, without needing to go to the ‘advanced search’ screen.

A number of these features are well demonstrated by internet search engines, such as ASK.COM, which uses type-ahead, spell check and faceting search, as well as suggesting terms to expand the query.

Last word: Ask something you can trust
The objective is not only to make a search engine as easy to use as asking someone else, but also to make it faster.

None of this is any use if users do not trust the results of the search engine and this can only be achieved by them using it, hence a catch-22 situation. This stresses the importance of involving key business users through the design and testing process, so that the solution delivers on its promise and to encourage them to act as champions promoting it throughout the firm. A word of warning though: make sure these users are not predominantly information professionals or the emphasis will be on advanced search features and the human-human search interface will live on. ?

Jonathan Trim is knowledge systems programme manager at Clifford Chance. He can be contacted at jonathan.trim@cliffordchance.com.

References

[1] Cornell University – Granka, L., Joachims, T., and Gay, G. (2004). Eye-Tracking Analysis of User Behaviour in WWW Search. In Proceedings of the 28th Annual ACM Conference on Research and Development in Information and Retrieval. (SIGIR ‘04). Sheffield, UK.

Note: To receive a PDF version of Figure one referred to in this masterclass please e-mail the editor at kclifton@ark-group.com.