Pull to refresh

Open source tool for intent-based search quality validation

Reading time 2 min
Views 806
IT systems testing *Development for e-commerce *Search engine optimization *
No matter how cool your search solution is, without a reliable way of testing it you will not be able to do any improvement without breaking something valuable.

Even a tiny increase in overall search result quality might have a noticeable effect on conversion rate. Naturally, you can use exit-rate to judge if your search needs improvements and do required adjustments, but will you be able to make sure those changes are safe? What if improved relevancy for a specific category breaks search for several other categories?


Moreover, high exit-rate means no matter what you are going to do, the problem has taken place, so dozens of customers already didn’t manage to find what they were up to. Basically, the only way to proceed is to have a reliable search quality validation approach.

Intent-based validation

Clear intent makes search result validation way more simpler and straightforward. There are several validation strategies possible. The actual strategy implementation is out of scope of this project.


Search Quality API defines the following set of «verdicts» for a product being returned: relevant, near, misplaced, irrelevant, undefined:
  • Relevant The product ranking is completely relevant.
  • Near The product ranking is not a perfect match, but it is clearly reasonable for it to be ranked highly.
  • Misplaced It is obvious why the search engine returned the result, but it clearly shouldn't be ranked highly.
  • Irrelevant The result has no apparent relationship to the customer’s search.
  • Undefined For some reason the result is undefined and can not be validated.

Intent-based approach is pretty intuitive and straightforward. So, if you are searching for «fiskars splitting axe» you do not expect to see any other brands, any other tools etc. Obviously, chopping axe is definitely not an option, as well as any other type on an axe.

Searching visitors usually have more specific shopping intents, comparing to visitors who are simply browsing, so it is crucial to make sure search always works as expected.


  • Extendable test set
  • Automated and reliable search quality verification
  • Various validation/verification strategies
  • Detailed reporting with root cause analysis
  • Historical trends


High-level statistics is available in a way it can help to quickly understand the actual situation and overall trend.

Passing tests

Passing test still provides all the details regarding the actual result set and the products. So, if any deep inspection needed it can be done.

Failing test

In case test fails deep investigation is a must, so the solution provides all the required details, including the list of the product returned for the query. Useful marks highlights non relevant products, simplifying the analysis.


  1. Intent-based Search Quality
  2. Search Quality API
Total votes 7: ↑6 and ↓1 +5
Comments 0
Comments Leave a comment