Category : General

Measuring Quality of Search in your own Application

Think about the data-driven products or applications that are in your portfolio. How many of them support some sort of search? Some might allow a search of structured DB data, such as for a given consumer, business, or product, while others may allow searching of unstructured data, such as articles, documents, or other artifacts.

Most applications at least keep a count of search ‘hits’ vs. ‘no-hits’, simply counting the number of times a search is satisfied or not satisfied. We think there is so much more to learn about the quality of search in addition to counting hits.


  1. If the search application presents a ranked list of search results and allows the user to select the correct entry, how often does the user pick the first (highest-ranked) entry? How often do they pick the last (lowest-ranked) entry, or one in between?  This key metric can be clearly presented in a histogram. The sample analysis below of 50 search results shows that the user selected nothing (the 0 column) 8 times, selected the first result in the list (1 column) 4 times, selected the second result 7 times, and so on.
    So, out of 50 searches, the user selected one of the top 3 results only 16 times (4 + 7 + 5), and failed to select any result 8 times.

  2. If the results are paginated, how often does the user go past the first page? How much past? If your users are scrolling past the first page over 20% of the time and then picking a result that is a good indicator that search options may need to be expanded to allow greater selectivity.

  3. If a user selects one of the presented search results, how often is it the “right one” leading to the desired user action, whether it’s ordering a credit report, printing a document, or some other activity?

  4. How often does the user select nothing (the no-hit condition), refine their search and try again? If possible, it’s very useful to track how the user is refining the search (the “search session”).  For example, if searching for a given business using a business name, city name, and state, did they refine their search to use a ZIP code instead of city and state?  Did they add a telephone number to the search? Did they strip out a noise word? If those refinements of the search query allowed the user to find what she was looking for that is valuable input to your search algorithm or indexing strategy.
    It’s not always easy to determine which search is a refinement of a previous search. This is usually a trial and error effort using various text clustering algorithms.

The metrics above give a good perspective of not only how often your user finds what they are looking for, but also how much effort they had to put into it.   Both are important to know, especially if search is a component of revenue-generating activity in your application.

Designing a screen or supporting a story?

Where would you start if you’re given the task of User Experience (UX) design for a new product?

Often, since we imagine we intuitively know how users will use the product, we jump to screen design. But unless it’s the simplest one-dimensional application, screen-centered design may not result in an efficient, pleasing, user experience. After all, while your users might look at the application one screen at a time, they are likely performing an activity that spans screens, or even spans an extended period of time.

Story-centric UX Design

In working through the UX of a non-trivial application we have found that following these 6 steps is a great roadmap:

    Story-centric Design Flow

  1. First, write down the 3 most important user stories critical for the success of your product or application. Those stories, of course, depend on the product. For example, on an automobile manufacturer’s site, those stories might be 1) Configuring a vehicle, 2) Comparing vehicles, and 3) Searching inventory for availability.

  2. Capture the high-level steps in each of those stories.  Breaking down the stories into smaller steps will be a great help in figuring out how screens should be organized. Be sure to capture timeframes in each of those steps as needed. For example, a step in the inventory search story may be an email exchange with a salesperson that could span days.

  3. List the main entities involved in the stories, and their major properties, including those critical to the stories in question. For example, continuing our above example, a vehicle has a body style, engine size, color, etc.We generally find that steps 2 & 3 are iterative.  You may cycle through them a couple of times.  As you can probably see, these initial steps borrow directly from the Object-Oriented Analysis methodology (after all, good ideas never die, right?). The output is all text, with flow diagrams as needed. All of these artifacts are reviewed with the application business owners.

  4. Using the output from steps 1-3, you can start storyboarding with simple pencil and paper low-fidelity screen wireframes. Now is the time to pay attention to the major content on each screen, screen transitions, major client-side interactions not resulting in a screen transition Storyboard(AJAX), and screen aspect ratio and size. Make sure you review the resulting storyboards with the subject matter experts. Tape them to the wall, arrange them on a conference table, or take over your local Starbucks. It’s sanity check time. If you are designing a responsive application (UI varies depending on screen size), you may want to wireframe the major screen size options as well.   Note that it’s wise to storyboard major features that may not be slated for the initial release of the product, as very often the stability of the UX is important through successive product iterations.

  5. After review of the low-fidelity wireframes, more detailed mock-ups are created, filling in specific fields, buttons, and establishing the general visual style of the application. This step is also dependent on any additional business analysis needed to identify additional relevant stories or user interactions.

  6. Depending on budget, user availability, target audience, and other factors, you may choose to build a simple click-able prototype using your favorite design tool.  We have found that Powerpoint or KeyNote offer quick and powerful capabilities, but have also used more specialized tools.  At any rate, a review of the mockups or prototype by both subject matter experts as well as day-to-day users of the application is really important at this point.   This is the time to kick the tires and make changes.

    If the application is a product then customer feedback is highly encouraged here. Very often user research uncovers issues or concerns the in-house business team may not be aware of. At the very least your key customers will feel valued and engaged.