Testing out PriBot, the new AI privacy policy analysis chat bot
Today we took the new PriBot privacy policy analyzer and chat bot tool out for a test run. The PriBot chat bot uses AI to analyze privacy policies — many of which are verbally and legally complex — and returns bite-size nuggets of information to those interested. The overall impact of using the chat tool in our test runs is that we could find answers to key questions very quickly, without digging through entire policies. In the test run for this blog post, we used the WPF privacy policy to show you how the PriBot chat bot works.
The bot’s query box has helpful hints on what questions you may want to ask; they appear in a drop-down menu. And if the website’s privacy policy you are interested in isn’t already analyzed in the bot, you can give the URL to the bot and it will conduct an on-the-spot analysis.
PriBot’s drop down menu
After we ran the bot through a number of privacy policies, from highly complex to fairly straightforward, we found that this tool tends to respond in straightforward ways if an answer to a question is clear. For example, if a policy states its data retention policy is to retain data for two years, the bot will pull that information for you and show it to you. In our tests, the bot returned the paragraph where the relevant information is discussed in the policy. When PriBot looked at WPF’s policy, it was 95% confident it had found where data retention is discussed in our policy.
PriBot’s contextual response to a question about data retention, with a high confidence level of 95%.
When there is nuance involved, the bot will give a lower score for certainty, and it selects the most relevant paragraphs from the privacy policy and shows them to you as possibilities. The confidence score of the answer is the key to knowing how much you can rely on the bot’s answer. The more nuance, the more contextual paragraphs the bot will display as possibilities. We asked the bot to tell us about WPF’s data collection, below.
Contextual response to question about data collection; the AI wasn’t so sure about this, note the lower confidence score of 58%.
The best use of PriBot chat bot is not for a deep analysis of a privacy policy. But it is a potentially useful tool to quickly parse dense privacy policies, or to quickly compare a number of policies based on a limited set of issues, like data retention and data security. The PriBot chat tool has its limitations, so any analysis should be backed up by a careful reading of the policy in full. More nuanced and complex policies will require a deeper reading.
A key for using the PriBot chat bot correctly is to heed the confidence percentages. In the version we tested, the confidence score was in clearly displayed in the lower right hand corner of the results boxes.
The researchers who created the PriBot chat bot also created an associated tool, Polisis, which is an AI-powered privacy policy analyzer that creates an overall summary of the privacy policy. In our test runs, we didn’t find this summary tool as clear or as helpful as the PriBot chat tool.
Take your own test run here, and let us know what you think of the PriBot chat bot.