HonestyMeter - A Free Open Source Framework for Evaluating the Objectivity and Credibility of Media Content

"By embracing HonestyMeter, you can join the vanguard of a movement that champions media objectivity and transparency. The more people who adopt this tool, the more we can create a well-informed society where the truth prevails over bias and misinformation" Read the full article in MTS

Introduction:

In today's world, media plays a crucial role in shaping public opinion and influencing decision-making processes. However, the proliferation of fake news and misinformation has significantly blurred the lines between fact and fiction, and between objective and biased reporting. This challenge is intensified by the use of sophisticated, yet subtle, manipulative techniques. Techniques such as sensationalism, framing, and selective reporting, among multiple others, are commonly employed by media outlets. These tactics can lead audiences to form distorted perceptions of reality, hindering their ability to make well-informed decisions.

To address this issue, we have developed the HonestyMeter framework – a free, AI-powered tool designed to assess the objectivity and bias of media content. Utilizing neural networks and advanced language models, HonestyMeter meticulously analyzes various media elements to identify potential manipulative tactics. It generates a comprehensive objectivity report, which includes an objectivity score, a list of detected manipulations, and recommendations for mitigating bias within the text. This tool empowers authors to craft more objective content and enables the audience to make better-informed decisions.

Process:

The HonestyMeter framework uses a multi-step process to evaluate the objectivity and bias of media content:

Input: The user provides a link to the media content, which can include text, images, audio, or video.
Analysis: The framework uses large language models to analyze the media content and identify any manipulative techniques that may be present. The analysis includes evaluating the tone, sentiment, and language used in the content.
Scoring: Based on the analysis, the framework provides an overall objectivity score for the media content on a scale of 0-100. Additionally, the framework scores the objectivity level for each side represented in the content.
Reporting: The framework generates a report summarizing the analysis, scores, and feedback provided for the media content.
Feedback: The framework provides feedback to the user on the manipulative techniques identified and the areas of the content that may be biased or lacking in objectivity and suggests possible improvements.
Improvement: The user can take the feedback provided by the framework and use it to improve the objectivity of the content.

Example HonestyMeter Report Screenshot:

(GPT-4 Generated Article Explores Imaginary Debates Between Fictional Candidates in a Hypothetical Country)

Challenges and Solutions

Currently, the HonestyMeter version is just an experimental demo. The reports are not as deep, accurate, or consistent as we would like them to be, for several reasons:

The GPT-3.5-Turbo model is used in production to reduce costs, since the tool is free and self funded.
The LLM struggles with long texts, often becoming "confused."
LLMs are not sufficiently effective in handling complex, multi-step tasks, especially with lengthy texts.

To enhance the results, we are undertaking the following steps, which are part of ongoing research and have not yet been implemented in production.

We conducted in-depth research on manipulation techniques, gaining a comprehensive understanding of manipulation categories. We revised the list of manipulations and created a more concise and well-structured list that covers all manipulation groups without redundancy or omissions.

We are moving away from our initial approach of relying on the LLM's "magic" with broad instructions, which was intended to prove the concept and showcase its potential. We are now working on providing the LLM with thorough, step-by-step instructions for detecting each manipulation technique.

We broke down the evaluation process into the smallest possible tasks. We are currently testing distinct services for each micro-step, which involve chains of prompts, autonomous agents and individual models that are specifically trained and fine-tuned for certain tasks. This approach is anticipated to not only improve output consistency but also reduce the system's dependence on any single model and simplify the process of replacing existing models with better or more cost-effective open-source alternatives when necessary.

We are experimenting with the most advanced Large Language Models (LLMs) and closely monitoring their exponential progress. By incorporating upcoming, newly released advanced models into our workflow, we expect to achieve significant enhancements in each component of our system, thereby leading to an overall elevation in performance.

Accuracy, consistency, and deterministic outputs.

It's important to note that as long as the system correctly identifies the broad objective or biased nature of an article in most cases, it can provide statistically valuable insights into bias levels, even if report accuracy and consistency are not perfect. This can be achieved by analyzing large volumes of content and calculating average scores from multiple iterations over the same articles.

For instance, by analyzing multiple articles from several sources and repeating the analysis of each article multiple times, we can identify which sources are more or less biased relative to each other. This approach, even with relatively low analysis accuracy, can yield statistically high confidence in the results.

Consequently, the effectiveness of the system isn't a binary choice between perfect operation and complete failure. It involves attaining a minimum required level of performance, followed by gradual improvements towards maximum effectiveness.

We are optimistic that through ongoing research and development, the efficiency and capabilities of HonestyMeter will steadily improve, gradually approaching its maximum potential. Even now, in its experimental demo phase, HonestyMeter frequently provides insights that are difficult for humans to discern.

Future Plans:

In our ideal future vision, we aspire to create a comprehensive media objectivity analysis tool that supports images, video and audio content analysis, evaluating combinations of text and images in articles, voice tonality in audio and video content, background images and video footage, as well as body language and facial expressions in video content. This represents the challenging goal of creating a process that considers all possible modalities and analyzes how they are integrated with each other in any piece of content, be it an article, book, podcast, or video. Further, we consider to connect to fact-checking sources and compare multiple sources to ensure a robust and reliable analysis.

Special thanks to:

1littlecoder, Yohei Nakajima, Matt Wolfe for the great inspiring content that made us fall in love with AI-powered apps. It was this inspiration that led us to create HonestyMeter, and we're grateful for their contribution!

Important Considerations When Using the HonestyMeter Framework:

When using the tool for the first time, you may be shocked by the high levels of subjectivity even in the content of the most well-known and authoritative mass media sources. It is essential to acknowledge that no one can be entirely objective, and some degree of bias is inevitable. Furthermore, a low objectivity score does not necessarily indicate malicious intent on the part of the mass media or journalists. Many instances of biased content are created unknowingly, with the best of intentions.

Our goal is not to blame anyone, but to provide a valuable tool for content creators and consumers alike that can help improve objectivity in media content. By using the HonestyMeter framework thoughtfully and with an understanding of its limitations, we can take a step towards creating a more reliable and trustworthy source of information for all.

Conclusion:

The HonestyMeter framework has the potential to be a game-changer in addressing media bias and misinformation. It's widespread adoption could increase transparency and objectivity in mass media, by helping journalists and content creators to produce more objective content, empowering users to make informed decisions with ease and becoming an essential tool for anyone seeking truthful and unbiased information.

Join Us in Shaping the Future of Media Truth

Up to this day, HonestyMeter has been fully self-funded. We invest our own time and money in research, development, and maintenance. Though we are fully capable of progressing independently, we are open to the possibility of partnering with those who resonate with our vision and can offer a substantial contribution, whether it be financial, promotional, or in terms of expertise.

If you share our vision of truthful media and are interested in making a contribution that has the potential for major advancement, please feel free to reach out to us at info@honestymeter.com.

Together, we can let the truth triumph.

Honest Disclosure:

This text was evaluated by the HonestyMeter and found to be highly biased towards promoting mass media transparency and the use of the HonestyMeter. 😊