In our last blog post, we introduced your new AI Country Risk Assistant and explained how it can help you quickly and easily interrogate our quantitative risk scores. Now, we’ve given the Assistant a powerful new feature: the ability to answer questions about the contents of thousands of publicly available, long-form reports from over 50 reputable sources. 

Your new party trick    

Country risk research typically begins with a need for specific information, which might be prompted by a third-party request or the requirements of a longer research report. In turn, this will trigger a task or question, which might be as simple as “What are the underlying risks to Algeria’s external position?” or “What information is available about beneficial ownership registration?”; or more advanced, like “Compare the climate change vulnerabilities associated with Indonesia and Thailand”. 

Let’s look at a relatively straightforward example. Suppose you want to answer a research question about Kazakhstan’s climate-related risks. In the past, you’d probably search the web for reports that are likely to contain the information you need, such as the IMF’s Article IV report on Kazakhstan. Then, you’d download the PDFs, search them for relevant keywords, and write a summary of what you find. 

Now, let’s see how your improved AI Country Risk Assistant can take much of the legwork out of answering such a research question.

1. Pick your reports

You’re probably familiar with CountryRisk.io’s Reports page, where you can search and filter all of the reports available on the platform by country, topic, source, or date. Now, many of these reports feature a new icon indicating that the AI Country Risk Assistant can answer questions about their contents. To give you as much control as possible, you choose which reports the AI Country Risk Assistant is allowed to consider before asking your question. 

For simplicity, we selected only the Article IV Kazakhstan report for this example:

2. Ask a question

Once you’ve selected the report(s) you want the AI Country Risk Assistant to look at, you can ask your question. In this case, we want to know what the Kazakhstan report has to say about climate change risks. So, we might ask:

3. Get an answer

Then, just hit Enter to get an answer based on the contents of the Article IV report: 

4. Check the answer and get the source

To verify the answer or find its source, click the Source button beneath the question field. This will reveal the original passages upon which the Assistant based its answer, along with links to their source documents: 

5. Ask follow-up questions

You can also see everything you’ve already asked about the topic at hand. So, if you didn’t get everything you need from the first answer, you can continue the conversation by asking follow-up questions:

6. Export your answers

Once you have what you need, you can copy any and all of the answers before pasting them into your report, presentation, or email.

We’ve yet to resolve the tradeoff between the amount of knowledge our custom LLM is capable of drawing upon and the inclusion of a full conversation history as part of that knowledge base. Until we can find a way to give the AI Country Risk Assistant a better conversational memory, then, it will answer each question in isolation.

Even so, we think you’ll agree this new feature is already powerful enough make answering many research questions significantly more efficient, so you can spend more of your time on valuable, creative work.

How we did it

So, how does this feature actually work? The illustration below shows the workflow.

As you might have guessed, the most important component of the workflow is the knowledge base. Indeed, you might already have some sort of knowledge base on a shared drive consisting of all the reports you thought might be useful for future research. The problem is that shared drives tend to quickly turn into disorganised PDF graveyards that don’t give up their secrets easily. After all, maintaining a well-structured knowledge base is hard work. 

But it’s also something at which we at CountryRisk.io are well practiced. We already maintain a custom knowledge base consisting of an augmented version of our risk scores, which is what enables the LLM underlying the AI Country Risk Assistant to provide consistently high-quality answers to questions about those scores. Now, the Assistant can tap into a second custom knowledge base consisting of around 200,000 long form reports from more than 50 reputable sources we find valuable for country risk analysis. These include, among many, IMF Article IV country reports and FATF country assessments. All of these reports are technically available to anyone for free; but in practice, locating and acquiring them from their original sources can involve a lot of legwork.

The Assistant interrogates this custom report database using Retrieval-Augmented Generation (RAG), which combines the strengths of extractive (i.e. return specific text from the custom knowledge base) and generative (i.e. generate a custom answer based on available information via the LLM) question answering models. This means we don’t need to worry about keeping the reports organised—we can simply dump them into the knowledge base so you can find and choose the best content for your question via the search tool.

That said, the knowledge base is “custom” in that we must first do some data pre-processing on each report. As we’ve seen, LLMs have limited contextual memories, and some reports run to over 200 pages. Moreover, the more contextual information the LLM has to review, the greater the performance, financial, and environmental costs associated with each query. Until these technical issues are resolved, then, we must make each report as easy for the LLM to digest as possible.

As with any data analytics project, this process is complex enough to fill a blog post of its own. In short, though, we begin by converting the PDF to plain text and using sentence cleaning to strip out any non-content elements, like tables of contents, bibliographies, and footnotes. Then, we split the resulting ream of unstructured content into topic-specific chunks that the LLM can digest. This process can be fuzzy: although we aim for chunks of 10 sentences, we might need to experiment with different chunk lengths or using semantic breaks for splitting wherever a report devotes more or fewer than 10 sentences to a particular topic. 

Once we’ve entered the “chunkified” report into the custom knowledge base, the LLM needs to be able to determine which chunks are relevant to the question by comparing their semantic similarities. To summarise another involved process, we use word embedding—the representation of words as vectors, in which semantically similar words are closer in the vector space than semantically dissimilar words—generated via OpenAI’s text-embedding-ada-002 model, which we then store alongside the corresponding chunks in a vector database.

So, when you ask a question, the AI Country Risk Assistant converts it into a vector and retrieves the top n chunks with the highest semantic similarity until it reaches a length of 1,000 tokens across all chunks. The Assistant then hands the question and retrieved chunks to the LLM for a completion request, along with two instructions: first, the LLM can base its response only on information contained in the retrieved chunks; and second, that it should respond as if it were an analyst tasked with assessing country risks.

Turbo-charge your research

The AI Country Risk Assistant is a powerful tool that can help you answer specific questions from colleagues, write your own reports, or conduct a sovereign rating analysis more quickly and easily, especially when it comes to assessing qualitative risks like effectiveness of fiscal rule or strength of banking sector regulation. 

If you’re an enterprise client, you can even add proprietary or in-house reports to the existing repository to create your own bespoke knowledge base, so you and your colleagues can use the Assistant to get instant answers about any topic that’s salient for your organisation—which could come in handy at your next credit risk committee meeting. 

Let us know what you think

If you’d like a one-on-one demo of how you can use your own reports to create a custom knowledge base or you have ideas for improvements, you can send us an email at [email protected].