Open source tool for intent-based search quality validation

    No matter how cool your search solution is, without a reliable way of testing it you will not be able to do any improvement without breaking something valuable.

    Even a tiny increase in overall search result quality might have a noticeable effect on conversion rate. Naturally, you can use exit-rate to judge if your search needs improvements and do required adjustments, but will you be able to make sure those changes are safe? What if improved relevancy for a specific category breaks search for several other categories?

    image

    Moreover, high exit-rate means no matter what you are going to do, the problem has taken place, so dozens of customers already didn’t manage to find what they were up to. Basically, the only way to proceed is to have a reliable search quality validation approach.

    Intent-based validation


    Clear intent makes search result validation way more simpler and straightforward. There are several validation strategies possible. The actual strategy implementation is out of scope of this project.

    image

    Search Quality API defines the following set of «verdicts» for a product being returned: relevant, near, misplaced, irrelevant, undefined:
    • Relevant The product ranking is completely relevant.
    • Near The product ranking is not a perfect match, but it is clearly reasonable for it to be ranked highly.
    • Misplaced It is obvious why the search engine returned the result, but it clearly shouldn't be ranked highly.
    • Irrelevant The result has no apparent relationship to the customer’s search.
    • Undefined For some reason the result is undefined and can not be validated.

    Intent-based approach is pretty intuitive and straightforward. So, if you are searching for «fiskars splitting axe» you do not expect to see any other brands, any other tools etc. Obviously, chopping axe is definitely not an option, as well as any other type on an axe.

    Searching visitors usually have more specific shopping intents, comparing to visitors who are simply browsing, so it is crucial to make sure search always works as expected.

    Features


    • Extendable test set
    • Automated and reliable search quality verification
    • Various validation/verification strategies
    • Detailed reporting with root cause analysis
    • Historical trends


    Trends


    High-level statistics is available in a way it can help to quickly understand the actual situation and overall trend.
    image

    Passing tests


    Passing test still provides all the details regarding the actual result set and the products. So, if any deep inspection needed it can be done.
    image

    Failing test


    In case test fails deep investigation is a must, so the solution provides all the required details, including the list of the product returned for the query. Useful marks highlights non relevant products, simplifying the analysis.
    image

    Links


    1. Intent-based Search Quality
    2. Search Quality API
    Share post
    AdBlock has stolen the banner, but banners are not teeth — they will be back

    More
    Ads

    Comments 0

    Only users with full accounts can post comments. Log in, please.