Vero AI’s VIOLET Impact Model Used to Gauge Responsible AI |

Vero AI unveiled its first-ever “Generating Responsibility: Assessing AI Using AI” report, offering a detailed evaluation featuring measurable ratings of ten leading generative AI models. The aim is to assist businesses in gauging how these tools adhere to responsible AI standards, as outlined by Vero AI’s VIOLET Impact Model™, which was developed by industrial-organizational psychologists and seasoned AI experts.

“As generative AI continues to rapidly evolve, organizations are increasingly challenged to grasp its benefits and potential risks,” said Eric Sydell, PhD., CEO and co-founder. “Although there have been some attempts to quantify and assess components of popular generative AI models for fairness and compliance, the criteria in these studies have been too narrow in scope to provide valuable recommendations. To fully harness AI in a responsible manner, especially with the emergence of new AI regulations, a broad approach accompanied by a scientific method of measuring AI systems at scale is needed.”

Using its AI-powered analytical engine Iris™, combined with human experts, Vero AI evaluated publicly available documentation of some of the more popular LLMs and generative models, including Google’s Gemini, Open AI’s GPT-4, Meta’s LLAMA2, and more. Iris allows for automatic processing of vast amounts of unstructured information. The models were then assigned scores based on key components of the VIOLET Impact Model, including Visibility, Integrity, Optimization, Legislative Preparedness, Effectiveness, and Transparency. Vero AI’s VIOLET Impact Model is a holistic, human-centered framework of elements and methodologies that provide a comprehensive and objective view of the impact of algorithms and advanced AI architectures.

The generative AI models analyzed showed varying strengths and weaknesses according to the criteria evaluated

The average effectiveness score was 81%.
The lowest average score was on optimization (at 69%) while visibility (76%) and transparency (77%) were less than 10 points higher. These results underscore the importance of vendors giving equal weight to all components of an algorithm when designing and building their models, and continuing to monitor them to make sure they are meeting responsible AI standards.

Generative AI models are aiming for a responsible approach to AI, but the task at hand is large

Most generative AI models have posted responses to calls from the White House to manage the risks posed by AI, on their websites. Additionally, many have clear feedback channels for users to reach out with model experience feedback, questions, or privacy and data related concerns.
The majority of generative AI vendors could benefit, however, from increased efforts related to transparency about their model algorithms, training data sources, and data quality, as well as documentation about how they ensure fairness and prevent biased outputs.
Although individual scores ranged from as low as 56% in certain categories to a high of 86%, some strengths stood out for each of the evaluated models. For example:
- Google’s Gemini, Meta’s LLAMA2, Inflection’s INFLECTION2, Big Science’s BLOOM all scored high for accountability
- OpenAI’s GPT-4, Cohere’s COMMAND and Amazon’s TITAN TEXT, AI21Labs’ JURASSIC 2 have made noticeable efforts in risk management

There is a clear path forward to achieving responsible AI, prioritizing evaluation and transparency

There are many AI frameworks across the globe, even the top generative AI models did not score perfectly on the VIOLET Impact Model and demonstrated room for growth. Responsible AI results in the equitable and beneficial use and downstream effects of AI for all of humanity. As companies contemplate integrating AI into their operations, Vero AI makes the following recommendations:

Have your model independently evaluated for effectiveness and make these results clearly and easily accessible to end users.
Provide clear information pertaining to human annotation rules practiced in the development of the system and information outlining the scale of human annotation.
Be transparent regarding data sources – what methods were used to ensure data quality? How were humans involved?

Derived from a global approach to AI ethics and regulation, incorporating best practice frameworks and legislation from across a variety of countries and cultures along with scientific practices, VIOLET ensures that both business effectiveness and human interests are served.

There is a full list of scores for each of the 10 generative AI models in the report. Vero AI is also offering a full version of its comprehensive report, including more details on its methodology and measurement scale for full price. Both versions of the Vero AI’s VIOLET Impact Model report can be found at the website here,

Leadership in the Age of AI Report Released by Kearney and Egon Zehnder