AI Re-Ranking For Semantic Search
Your users can find the information data they need at the right moment. Learn how AI reranking can help your users find the correct information today.
Search is not just about finding keywords that match – semantic Search is even more critical.
Semantic Search is about finding the correct information at the right moment for the searcher.
This goes beyond simply finding the right keywords or concepts and speculates how searchers will interact with the results.
Reranking artificial intelligence (AI) will use information about people who search to tailor search results to them.
This could be done at a group level and change results based on popularity, seasonality, or trends.
You can also do it individually, altering results according to the searcher’s wishes.
Although it is challenging to implement AI reranking in search engines, it can tremendously impact conversions and user satisfaction.
Re-Ranking with Artificial Intelligence
AI-driven search engine reranking can increase search results regardless of the engine’s underlying ranking algorithm.
This is because search results that are relevant and useful go beyond textual relevance. They also include business metrics such as raw popularity.
Good results consider other signals and do this on a per-query basis.
Let’s examine the popularity business metric to see why this is so important.
Although it’s a significant ranking signal, it can be elusive for specific queries. A search for a “red dress” may bring up two options: “Backless dress with red accents” or “Summer dress in bright red.”
Backless dresses are more popular than overall dresses and products.
However, in this particular case, it is not what customers want.
They are looking for a red dress with no red accents.
Isn’t that a signal for the search engine to rank the summer dress higher
Search Analytics
The above example illustrates that reranking is about understanding what searchers are doing.
Clicks and conversions are the most common events you should track.
These are generally the only necessary events and must come from a search.
This example highlights an additional essential consideration: events should be linked to specific queries.
This allows the search engine to learn from user interactions and the interplay of different result sets. This makes the “red dress” summer dress appear higher in search results.
A product may be less popular than its neighbor for different queries.
You’ll need to weigh your events differently when weighing them.
Clicking on a result indicates interest while clicking on it (or any other conversion metric) signifies that you are committed.
This should be reflected in the ranking.
It doesn’t have to be complicated.
It is possible to say that conversions are worth twice as many clicks.
It is essential to test the proper ratio in your Search.
It is also possible to discount events based on the search engine result ranking when it was viewed.
We know that the position of a result can influence its clickthrough rate (CTR).
You may find that the top results are more firmly rooted because they have more interactions. This keeps them higher in search engines.
Seasonality and Freshness
This self-reinforcing loop can be broken by discounting events according to the time since the event.
This is because every event in the past has a decreasing impact on reranking. This is until it ceases to have any effect.
You might, for example, divide each event’s impact by two daily for 30 days. After 30 days, you can stop ranking the event.
The nice thing about using freshness in the algorithm for reranking is that it introduces seasonality to the results.
You will no longer recommend viral videos many years ago but are boring today. Instead, you will recommend videos to “learn how to swim” in the summer and “learn to ski” in the winter.
YouTube is designed to provide freshness and seasonality in its algorithm.
Using Signals To Re-rank
Once you have the signals and how they decay over time, you can use them in the search results.
We often associate artificial intelligence with something complex and unfathomable.
AI can be as simple as taking time-series data and making decisions, as we are doing here.
It is simple to take a set of results and rerank them according to a score.
This number of results is usually small (10-20). Next, score them.
As we have discussed, the score could be added up to the number of conversions multiplied by two and the number of clicks.
The addition of a decay function increases complexity. However, the same principle applies to discounting based upon result position.
This reranking system has one drawback: you can only rerank a limited number of results.
Learn To Rank
A result that isn’t popular but not high in the rankings won’t be noticed.
This system requires that you have events on the records and queries you wish to rerank.
It will not work for new product launches, user-generated material (UGC), and often in and out of the search index.
These issues can be addressed by learning to rank (LTR).
LTR works similarly to the reranking discussed above. It is based on the belief that records searchers who interact with them are more valuable than those who don’t.
When a query is tied to the previous reranking technique, it boosts or burys results.
LTR, however, is more flexible. LTR works by using popular results to boost or bury results.
LTR uses machine learning to identify similar queries (e.g., “video games” or “gaming console”)
It can then rank results for less-popular queries based upon interactions with the more popular ones.
LTR does not just generalize on queries. It also generalizes to records.
LTR models can learn the most popular results, such as the Nintendo Switch game “Legend of Zelda: Breath of the Wild.”
It can then connect to similar results (“Legend of Zelda Skyward Sword”) to boost them.
LTR is a powerful tool that provides more excellent query and record coverage than reranking.
In other words, it generalizes better.
LTR is more complicated and requires more in-house machine learning expertise (ML).
It is also more difficult to understand why specific results are located in particular places.
The first type of reranking allows you to compare the conversions and clicks for each record over time.
LTR is an ML model that creates connections that may not always be obvious.
Are “Breath Of The Wild” & “Sonic Colors” similar?
Personalization
Re-ranking is effective across all search engines, but personalization sounds more personal.
Personalization is about personalizing results that are already relevant.
Although there are many opinions on how web search engines like Google use personalized results for their users, personalization often affects the results from on-site search engines.
It’s a great way to increase search interaction and convert from Search.
Search Analytics
Personalization, just like reranking, depends on how users interact with search results.
You can track clicks and converts to get a better idea of what results from the user is looking for.
There is a significant difference between personalization and reranking on this front. You might need to adjust how you apply personalization depending on what Search you are doing.
If you are a grocery seller, it is a good idea to suggest products purchased in the past.
If your website sells books, you will not recommend books that customers have already purchased. You may even want those books to be moved down in search results.
However, it is also true that personalization should not be pushed so hard that users see only what they have interacted with previously.
Both discovery and finding are possible with Search. If they return to the search bar, they should be open to finding something else.
Do not rank results solely based on personalization. Use other ranking signals.
Personalization, just like reranking, can also benefit from event decay.
A search that reduces the impact of older events will better reflect a user’s current tastes.
It can be viewed as your seasonality.
Personalization Across Users
Personalization that we have seen so far relies on individual interactions. However, you can combine it with other search results.
This approach significantly impacts situations where the user has never interacted with items in the search results.
You can’t boost/bury search results based on past interactions because the user hasn’t interacted with them.
You can instead look at similar users and personalize based on what they have done.
Let’s say, for example, that you have a user unfamiliar with your style but have bought many handbags.
You can search for others with similar tastes and who have interacted with dresses.
Intuitively, other customers who love the same handbags as us should also like the same dresses.
Personalization and Re-Ranking For Discovery
Personalization and reranking can have a significant impact on Search. These same tools can be used for discovery.
Your home page and your category pages should be viewed as search results.
You can use search tools to get the same results as you do for searching.
A home page, for example, can be compared to a search page that doesn’t have a query. A category landing page looks much like a search engine page but with a filter.
These pages can become less static if you add personalization or reranking. They will show users what they want to see and can push more popular items up for customers.
Don’t worry about personalization or reranking. They can be combined with editorial decisions on these pages and inside searches.
This can be done by fixing desired results in specific places and then reranking around them.
Personalization and reranking have been proven effective in enhancing search results by combining user interaction with relevant signals.
The interactions allow you to let your users influence the outcome.
Little by little, these interactions reveal to the search engine which items should rank higher.
Searchers ultimately benefit from a better search experience. You also benefit from more clicks, conversions, and other benefits.