Understanding Search Relevance

Search Relevance and User Satisfaction

If your users rely on search to find what they need on your site, delivering a great search experience is crucial to user satisfaction. A great search experience has the following traits:

1. Delivers relevant results

2. Easy to use

3. Fast

A user may tolerate a non-intuitive search interface or slow results but if they cannot find what they want they will quickly turn into an unhappy customer and start looking for alternatives. But what does it mean for results to be relevant? In order to truly understand and improve search relevance, it is necessary to first define it in an objective way. In this article we are going to:

1. Define search relevance in a way it can be measured

2. Understand how relevance is modeled in Elasticsearch

3. Discuss the common challenges when attempting to improve search relevance

What is Relevance?

According to information retrieval theory, relevance is driven by 2 metrics:

• Recall: The percentage of relevant documents returned over total relevant documents

• Precision: The percentage of documents in the results that are relevant

Recall and Precision Example Consider the following product documents and their product names:

1. Hiking boots (Black)

2. Black leather boots

3. Black boots with laces

4. Black boot cleaner

5. Black boots

A search query for "black boots" may return:

1. Black boots with laces

2. Black boot cleaner

3. Black boots

In this case, the recall is 60% since only 3 out 5 relevant documents were returned. Similarly, the precision is 66% because "black boot cleaner" may be considered irrelevant if the user was searching for boots and not boot cleaners.

Relevance vs Similarity

The formal definition of relevance still does not answer what it means for a document to be relevant to a user's query. It also does not address how 2 different documents can both be considered relevant even though one of them is more relevant than the other. Going back to our "black boots" example, the product with the name "black boots" can be considered more relevant than the product with name "black boots with leather". Elasticsearch uses the concept of similarity to measure how relevant the contents of each field are to the provided query. By default, search results are ranked by this similarity score so that the most relevant results are shown first.

Relevance Challenges

Relation Between Recall and Precision

There is a hidden relation between recall and precision where attempts to improve one negatively affects the other. Let's examine that relation with a few practical examples where we attempt to improve recall and observe how it affects precision.

Letter Casing

If letter casing is strict, the query "apple" does not return Apple products. Ignoring letter casing will improve recall and return Apple products but will also end up returning products like apple cider and apple sauce thus decreasing precision.

Spelling Mistakes and Typos

If searching does not tolerate minor spelling mistakes and typos, queries may not return all intended results therefore reducing recall. At the same time, searching for "black boots" when meant to search for "black boats" will reduce precision by returning irrelevant results.

Missing Context

Similarity alone is not always sufficient to return relevant results. The context of what the user is actually looking for is not always known. This is especially the case when you have single search box that is used to search across multiple fields. Consider a search for "black dress" that returns products in the following order:

1. Black dress shoes

2. Back dress with white belt

In terms of text similarity, the order makes sense. However, relevance changes based on context:

• The user is searching for shoes and not dresses

• The user has never bought shoes on this site before but has bought dresses in the past

Next Steps

Now that we have defined relevance in a measurable way and understand how it is modeled in Elasticsearch, it possible to come up with a strategy for improving relevance. The next article presents such strategy while also addressing the inverse relation between improving recall and precision. Did you like this article? Subscribe to our blog by adding your email address to the form below. You can also email me at andreas@inventaconsulting.net or schedule a call to find out how Inventa Consulting can help you with Elasticsearch.