Problem

Trendwatching is a powerful tool for driving strategic innovations. It helps to discover new teсhnologies, business models and products that may be used for idea generation and technology transfer. It is a powerful tool for product managers, business stream managers, top managers and "strategists" and is mostly used on a regular basis.

Most trendwatchers use meta analysis of ready-made reviews and reports. However, an objective and undistorted approach means that you also have to scan the emerging "signals": news, startups, venture deals, academic articles and patents day by day. In this case you have to analyze the huge amount of text data...Big Data. Indeed 50 million startups are created globally each year, 4 mln per month. Each year, over 2 million new research articles are published in more than 30,000 peer-reviewed journals across all fields of study, 167,000 per month. How to succeed in trendwatching with limited resources and tight deadlines?

Solution

Natural Language Processing (NLP) and AI are definitely what we need to cope with Big Text Data in time. Entity Recognition and Text summarization: these are the two tasks that require some analysis. Let's start with the first one. Let's imagine that we want to put perspective technologies that appeared over the past year in a certain business or technology sector on our "TechRadar". Let's focus on "AI" for example.

Pipeline: Tech Trends in AI

Download startup descriptions (2023) using AI tags from Crunchbase and Pitchbook
Add general up-to-date information from the startup main page. We used a parsing script (Python) to get data and automatically translate it into English. This data was merged with data from Crunchbase and Pitchbook. It took 2 minutes.
Apply an AI model in Python script to extract technologies (Entity Recognition) mentioned in the descriptions. We used the ensemble of three AI models: roberta-base-squad2 (extractive model), GPT 3.5 Turbo (generative model) and microsoft/phi-2 (generative model). You may look at some technical details of implementation in the Colab Notebook here. Model Recall is over 92% in the AI dataset for technology extraction tasks. You will get the list of candidates, that is to be cleaned by experts to remove false positives (in some cases AI models confuse technology with products or generate "hallucinations"). You also have to merge synonyms into one technological entity. It takes around 1-2 hours in total.
Prioritize technologies by the number of startups that mention this technology in the description (A) or by total investments (B). 1 minute with script and Pitchbook/Crunchbase API.
Prepare Top10 list of technological trends. Analyze the drivers, constraints and the prospects of each trend using an expert assessment. 1 day for analysis of reports and reviews through Perplexity.
Apply generative model, GenAI (for example GPT Turbo) to extract basic applications and Use Cases for each technology. Collection of startup descriptions is used as the input. 5 minutes.

AI - Expert Collaboration in Trendwatching Pipeline

The total time for analysis is around 1-2 days.

The resulting list of Top 10 technologies looks like this: Generative AI, Computer Vision, Predictive Analytics, Data Insights and BI, AI Tools, AI in Cybersecurity, Semantic Search, AI in Blockchain, Transparent and Safe AI, AI in Edge Computing.

Let's look at the example of Top 5, most popular Use Cases for Transparent and Safe AI: diagnosis of diseases and treatment recommendations (1), creditworthiness analysis (2), transactions monitoring (3), investment decisions (4), defense and security (5). Not bad for the MVP.

What is the value of this methodology? Clear logic, use of primary data, high objectivity сonsequently. You can use it on a regular basis (per quarter, per month), use academic papers and patents for monitoring emerging trends as well.

Pipeline: Business Trends

The current approach has one drawback: we miss business models and products that don't mention technology. Actually sometimes innovation lies within the business model - Uber, BNPL players and so on. So, we need to change something in our model to move towards a more flexible approach. This time we look for trends in Q1 2024, FinTech industry:

Get startup descriptions (2024, Q1) using FinTech tags from Crunchbase and Pitchbook
Add general up-to-date information from the startup main page. We used parsing script (Python) to get data and automatically translate it into English. This data was merged with data from Crunchbase and Pitchbook.
Let's ask GPT Turbo API "What are the most popular diversified trends without company titles (in the format 'Technology or Product - Value for the user') mentioned here. Give a list of 10 items". Collection of 121 startups splitted in chunks is the input. Context provided: "You are a FinTech expert". GPT Turbo parameters: temperature = 0, top_p = 0.9, presence_penalty = 2.

The resulting list looks like this:

AI & Cybersecurity - Protecting against generative AI-enabled fraud
Advanced AI for Currency Trading - Accessible and effective cryptocurrency trading
Crypto Payments Platforms - Facilitating stablecoin and blockchain-based payments
B2B Fintech for Self-Clearing Firms - Helping firms receive account transfers with AI
DeFi Protocol for Leveraged Derivatives Trading without Liquidation
Enabling smart receipts in mobile banking apps
AI-Optimized Asset Maintenance Management
Cryptocurrency Analytics Platform - Tracking, analyzing, and creating cryptocurrency indexes
Decentralized Finance (DeFi) Investment Management
Blockchain Community Marketplace - buying and selling through blockchain technology

We definitely need to add some Use Case examples to our trends. We ask model through one more iteration: "Give title and information (maximum 2 sentences) about one company using 'Advanced AI for Currency Trading'". Here is the model reply:

"NVolume is a FinTech company that develops advanced AI/ML models for autonomous digital currency trading. Their user-friendly platform allows users to navigate the dynamic cryptocurrency market, offering options to make trading more accessible and effective".

Now we can check the company's presence in the collection of startup descriptions (csv file), validate its connection with trend and add the website link - NVolume. Iterate over all trends and complete the report. Colab Notebook here.

Conclusion

We showed some examples of AI applications for trendwatching: with use of GenAI and extractive models. Results correlate with our analysis of open reports and reviews of AI Tech Trends in 2023 and Q1 2024 FinTech sector. However the considered models have to pass further validation of the expert community. I will appreciate any discussion and feedback. You may contact me by telegram: @Sergey_Kamenshchikov. Thanks!

Sergey Kamenshchikov, PhD

How to speed up Trendwatching with AI