NOVI SAD 10th International Conference THE BRIDGES OF MEDIA EDUCATION 2018

Fighting fake news and hoaxes in the age of convergence: evaluation of fact-checking and debunking organizations’ performance

Bissera Zankova, Media 21 Foundation,Bulgaria



The successful fight against fake news and hoaxes requires not only a careful study of the nature and impact of these phenomena but identifying the possible paths and instruments for combating them as well. Against the background of the measures taken on a European level the article will discuss the activities of the fact-checking and debunking organizations with a particular focus on the evaluation of their performance. 


Keywords: fake news, information disorder, fact-checking and debunking organizations’s performance.


  1. Introduction: the modern term of terms

Journalists, politicians and scientists have agreed with the real news that fake news is the Word of the Year for 2017. Defined as “false, often sensational, information disseminated under the guise of news reporting”, “fake news” will now be added to the next print edition of Collins Dictionary (The Independent 2017). 

False news has changed our understanding of the information society we are immersed in and more particularly of the role of the media as opinion makers. Therefore it is not about a popular term only but about an issue people have to deal seriously with. Media messages which carry misleading information usually accompany stories and events that impact considerably our lives (Rannard 2017). Moreover the trust of society in the new media is severely undermined. Not only social media can be blamed for spreading false stories. Traditional media relying on cosy relationships with political parties or grabbing uncritically news from the networks are also agents (wittingly or unwittingly) of their dissemination. Though all media pursue a public function and are accountable for their presentation public service media by their mission are obliged to provide diverse, accurate and unbiased information. This media can be “a bulwark against fake news”.(Morgan 2018).

Fake news and hoaxes require complex responses but the measures should not imperil free Internet and free communication. Vertesi (2016:7 – 8) argues “a novel set of challenges evolves as our community addresses the production of knowledge in the contemporary context and these challenges require us to revisit, interrogate, and expand our theoretical, methodological, and practical toolkits.” The solutions policy makers could come up with are not easy and must rely on the people’s will to vigorously react against lies in the media. To do so users have to be digitally empowered and the process should be ongoing. 

The European Commission (EC) and the social networks companies themselves have made the first steps against fake news and hoaxes online. Facebook (FB) is trying users to potential misinformation by displaying fact-checked articles next to disputed stories and Twitter has expanded its rules to cover what is classed as hateful or harmful behaviour on the platform. Google has invested 300 mln dollars to elevate quality journalism and has started pursuing projects on media literacy. 

However, the attempts made by private entities do not always unconditionally coincide with the public interest. That is why the efforts to include new stakeholders in the fight against disinformation represent a positive move towards broadening the ground of this struggle. Now fact-checking and debunking organizations comprise a promising party in the battle with fake news and hoaxes. Before concentrating on their performance it is important to elucidate fake news as a phenomenon in the modern world in order to react adequately to its consequences. The next section will discuss how deviations from normal communication in the form of false and misleading information through the media result in a wider phenomenon – information disorder.

  1. The broader perspective: information disorder

In the digital world we come across a variety of communications problems each of different scope and impact. False content and hoaxes are not isolated messages and generate wide and diverse repercussions – social, political, economic, digital. Bounegru et al. (2017: 6) argue that “the significance of fake news cannot be fully understood apart from its circulation online.” Therefore in order to identify the possible measures against fake news we have to take into account their reach and all the negative implications in the media environment. The Digital, Culture, Media and Sport Committee (DCMSC) in Britain investigating disinformation as a result of the Cambridge Analytica data scandal has produced a report which confirms the frailty of the term ‘fake news’ referring to a variety of practices. According to Des Freedman (2018) fake news stretches”from falsehoods deliberately concocted to undermine elections and referenda through to perspectives that are simply seen as unwelcome and controversial the government should, from now on, refer to misinformation instead”(Des Freedman, 2018). In the same vein Wardle & Derakhshan (2017) call these negative phenomena “information pollution” or “information disorder” and suggest scientists and policy makers to comprehensively examine them and the related challenges. They introduce a more elaborate conceptual framework for examining information disorder, distinguishing between three different types: mis-, dis- and mal-information, each of them characterized by a certain degree of falseness and dangerousness. Mis-information is the case when false information is shared, but no harm is meant, dis-information is when false information is knowingly shared to cause harm and mal-information is when genuine information is shared to cause harm, often by moving information designed to stay private into the  public sphere.Titled a “multidisciplinary framework for research and policy-making” the report goes beyond simple checking or blatant censorship and is aiming to lay down a new methodology (Wardle & Derakhshan 2017).

Such approach proves fruitful, firstly, because it goes beyond the incidental fake news distribution and draws public attention to its broader effects. Secondly, it allows other information failures, sometimes interrelated, to be explored. Thirdly, it points to the necessity that a range of measures at different levels have to be taken to combat negative effects. Fourthly, it serves as a signal to stakeholders that all of them have to participate if they wish the fight against the different forms of information disorder to bring tangible results. In the next section I am going to explore the measures against fake news and information disorder taken at a European level.

  1. European efforts against information disorder

Disinformation can have immediate perilous consequences and can make the public policy implementation much harder. The interplay of rights is essential with regard to the formulated measures – on the one hand, freedom of speech and pluralism should be respected and on the other, limitations on the dissemination of certain types of content which are false and misleading should be imposed.

Realizing this complexity the European Union (EU) undertook a political intervention through the European External Action Service East Stratcom Task Force that ran the ‘EU vs Disinformation’ campaign which had identified and debunked over 3.500 disinformation cases between September 2015 and November 2017. Despite these concrete outcomes, Nyhan and Reifle (2015) argue that misinformation (even being the mildest form of information disorder – B.Z.) may have lasting effects after it is discredited. Therefore, only debunking is not sufficient but it must be complemented by an alternative explanation. 

This reasoning and the quick spread of fabricated stories across Europe led to the establishment of a dedicated expert group in 2018 with the task to formulate solid policy proposals. The high-level group of experts (“the HLEG”) had to discard simplistic solutions and to avoid any form of censorship either public or private. Based on the independent HLEG report (European Commission 2018b) as well as wider consultations, the EC defines disinformation as verifiably false or misleading information that is created, presented and disseminated for economic gain or to intentionally deceive the public, and may cause public harm” (European Commission Press release 2018).

The aim of the HLEG’s recommendations was to ensure the effectiveness of the longer-term responses and their continuous evaluation. Fake news and disinformation were considered a “moving target” that required constant checking whether measures taken were really efficient from a human rights perspective. The central goal of the report was to reinforce the transparency of the combating activities in all its dimensions. Transparency was considered essential with respect to maintaining the sustainability of the media ecosystem as well as to strengthening the position of the newly emerging group of fact checkers. 

The involvement of the fact checking and debunking organizations in the battle against disinformation merits a particular focus because in practice it represents the recognition of a new stakeholders’ group among others. The HLEG set a high-profile task on it suggesting to use its capacity to increase the transparency and efficiency of the overall fact-checking practices.The idea is based on the multi-stakeholders’ approach premised on cross-sector and cross-border cooperation.Having in mind the role they are assigned to play the main issue here is about the efficiency of the organizations themselves which may contribute to the improvement of the whole process of combating fake news and hoaxes or downgrade it. The next section will focus on the performance of the fact checking and debunking organizations and the need for its regular assessment.

  1. The performance of the fact-checking and debunking organizations 

The role of platforms, news media and factchecking organisations in the fight against disinformation is crucial as they stay most closely to the production and distribution of messages. The European Commission, together with the Member States,”should support cooperation between media organisations, platforms, academic researchers, fact-and source checkers, advertising industry and civil society organisations to ensure the necessary level of public scrutiny and balance in the definition of transparency standards.”(European Commission 2018b: 22) These conclusions are very general and do not provide what kinds of transparency standards specifically are needed. It is also striking that the performance of the fact checkers in the EU is still relatively fragmented and respectively the knowledge about it. That is why more collaborative work should be done either at the media level or with other stakeholders within the Member States and across Europe. Such joint activities could reassure society at large that fact-checkers continuously improve their working methods. Adjusting to the changes in the environment they should advance their journalistic approaches, too. The higher quality of journalistic work can stem from the cooperation with the public service media and the adoption of their standards, for instance. These relationships are not easy because as Graves & Cherubini (2016) conclude :”Whatever their organisational form, research practices, and funding model, all fact-checking outlets still rely in a large part on existing news media to publicise their work. All have a digital presence across their own website and various social media channels but also work in a sometimes uneasy relation with the existing media that they aim to influence or provide an alternative to.”(Graves & Cherubini 2016: 30). 

The EC (European Commission Press release 2018) formulates two proposals with respect to fact checkers related to their organization and methods: 

  • To establish an independent European network of fact-checkers with common working methods to exchange best practices, and work to achieve the broadest possible coverage of factual corrections across the EU and
  • To create a secure European online platform on disinformation to support the network of fact-checkers and relevant academic researchers with cross-border data collection and analysis, as well as access to the EU-wide data.

However, the accomplishment of these objectives depends on the evidence that these organizations have the potential to fulfill them. The efficiency of their activities merits equal attention as their transparency and openness of action.

Difficulties to trace the quality of the fact-checking and debunking organizations performance may stem from the lack of a common definition as scientists generally apply a variety of methods to describe them. A recent study by Bae Brandtzaeg & Føolstad (2017), for instance, divides fact-checking services into three general categories based on their areas of concern: 1) political and public statements in general; 2) online rumors and hoaxes and 3) specific topics, controversies. particular conflicts or narrowly scoped issues and events (Bae Brandtzaeg & Føolstad 2017). Graves & Cherubini speak of two models of fact checking organizations – the journalistic news room model and the NGO mode l(Graves & Cherubini 2016: 8 – 10). None of the authors cited, however, have delved into their internal structure and management of resources.

The International Fact-Checking Network (IFCN) launched in September 2015 to support fact-checking initiatives by promoting best practices and exchanges among organizations adopted a Code of Principles in 2016 representing a range of professional standards similar to the media codes. However, neither the network, nor the organizations themselves have passed clear and publicly announced criteria for the assessment of their performance so far. 

Obviously developing a workable methodology for a performance analysis of fact-checking and debunking organizations is an important goal having in mind the higher expectations for the results accomplished by these organizations. In a recent study Pavleska, Školkay, Zankova, Ribeiro & Bechmann (2018) deal in depth with these problems in their manuscript “Performance analysis of fact-checking organizations and initiatives in Europe: a critical overview of online platforms fighting fake news”. The authors use the Wardle and Derakhshan’s conceptual scheme and extract the basis of their methodology from the existing frameworks and reports on the performance issues of two sectors: governmental and non-governmental. Such approach allows for complementing the efforts in both sectors and to count on their interaction.  The study of the real performance of fact checking and debunking organizations is premised on an initial survey designed and carried out through a set of questions based on the identified indicators. Through it 50 European organizations located in 27 countries were approached.

The purpose here is to not repeat the analysis and arguments in the quoted article but building on the idea of performance evaluation to move towards a discussion of the necessary measures that can make fact checking and debunking organizations really effective and efficient. The formulation and deployment of an array of performance indicators is the first step to achieving this goal. In addition on the basis of the data collected in the survey and recent developments interesting insights concerning the problems of fact checking and debunking organizations can be summarized. In the next section the problems of the fact-checking and debunking organizations will be examined.

  1. The fact checking and debunking organizations’ problems

The biggest reported challenge for these organizations relates to the insufficient stakeholders’ awareness of the issues concerning information disorder and the lack of adequate resources for their counteracting. Fact checkers should be encouraged to publicise widely information about their methods and the outcomes accomplished. Factually articles concerning their work are increasing particularly on the Poynter’s International Fact-Checking Network site where materials presenting local themes can also be found. However, specific information about the management and staff as well as about financing of projects and campaigns are missing. Graves and Cherubini (2016: 28 – 29) underline that the “budgets for these efforts span a wide range, but remain quite low for the typical outlet.” In practice there are two main sources of funding known – by the media industry and through donations – each carrying risks for the independence of these entities. Greater transparency of financial resources would contribute to their better using and controlling, to finding more donors and new channels of funding which eventually may enhance performance and ultimately strengthen the independence of these organizations.

Another problem is the lack of clarity in the objectives which concern the internal divisions, procedures, staff management and operation processes as required for non-governmental organizations notwithstanding the fact they can operate as journalistic organizations or as typical NGOs.(Governance and internal control) Reforms that can be implemented to discharge properly their broader functions demand considerable improvement of the organizational culture and internal practices about which there is not sufficient information (at least at the moment). Such re-organizational steps can help fact-checking organizations connect to new audiences.As reported, a large number of readers and viewers have not been reached – not only partisan voters who are skeptical about media, but non-voters who are not engaging fully in civic life. (Greenblatt 2017)

The majority of fact checking and debunking organizations are ‘specialized’ in one type of content only. Specific visual content (photos, YouTube), although known to have far greater impact on the proliferation of fake news and hoaxes than text, is checked to a lesser extent. However, as Greenblatt (2017) states fact-checking organizations are prone to finding new formats, including better use of graphics and visual aids to make their information more accessible. Optimistically this can be considered a good move towards the control of a new content and its better presentation but has to be traced and explored further.

With respect to the working methods of the fact checking and debunking organizations most of them strive to embrace innovative methods, but there is still a significant number that have not yet considered this option due possibly to financial reasons or lack of skills. The IT companies can help them providing advice and technical support and social media giants can also consider the opportunity to investing in such empowering projects. However, the new methods may generate new challenges. As reported at the Summit of the fact checking organizations organized by the American Press Institute in 2017 it became clear that fact-checks had among the highest engagement of any of the news organization’s offerings on Snapchat. This non-traditional tool was used as a way of presenting information where readers and viewers live, not waiting for them to navigate their way through a website.However, fact-checks that are presented primarily on visual platforms like Snapchat may be limited to topics that are already fairly well-known and thus don’t require much in the way of explanation and an in-depth discussion about claims, counterclaims and sources. GIFs, Instagram Stories and Twitter Moments are other ways of presenting information through new channels. Digital-only fact checks on platforms such as Facebook have become more common. Other efforts are already underway, such as “Share the Facts” app for Alexa, Amazon’s home assistant device, that can answer questions about some claims; and ClaimBuster, a tool developed at the University of Texas at Arlington, designed to quickly call out falsehoods on Twitter. There should be better understanding of these apps and opportunities among the audience which is also an element of the transparency policy of the fact checking organizations. 

Generally the transparency of the majority of fact checkers (in terms of methodology, funding and operation) in Europe remains blurred. This fact demands more intense civil society and public involvement.In the same vein are the recommendations of the HLEG and the EC – “transparency is a key element in the response to digital disinformation”. (European Commission 2018b: 22). Fact checkers should do their best to accomplish a high quality job and gain the trust of the public through better communication policies. The latter should make clear why certain claims are subject to checks, why some sources of information, such as statistics from government agencies, are seen more reliable than others, they should provide sufficient proof  that a variety of fact-checkers agree on the substance behind a controversial topic and that a publicly available database of verified facts is created. In the digital age trust manifested through various forms of communication and interaction becomes a central category. As Pavleska and Jerman-Blazic argue current systems do not account for important factors when calculating trust and system design should be changed as to allow for improving both the user-experience and the system-performance.Taking this into account fact checking organizations can be shaped organizationally and functionally in a manner that supports trustful behaviour. This also may lead to limiting the three types of information disorder: disinformation, misinformation and malinformation.(Pavleska & Jerman-Blazic 2013)

These inferences can serve as a ground for more diverse and nuanced policy initiatives aiming at promoting better performance and encouraging a high quality contribution on the part of the fact-checking and debunking organizations to the fight against information disorder. The general conclusion is that notwithstanding shortcomings and deficits their activities comprise grass root initiatives that should be given a chance to develop.

  1. The way forward: performance improvement and effective cooperation policies

Fact checking and debunking organizations in order for to be effective and efficient should improve their journalistic and organizational performance including through the changes in their management and culture. Clear-cut criteria for the evaluation of their performance should be designed in this respect. The article referred to in this publication is only the beginning of the process which we believe will accelerate (Pavleska et al. 2018). The academic community can engage more thoroughly with the exploration of the issues regarding the analysis, formulation and testing of a set of adequate performance indicators. Solutions may combine both technical and human characteristics. The big problem that appears, however, is who will monitor and evaluate the performance of these organizations, summarize and publish the results or “who will check the fact-checkers”.

Fact checking activities should not remain confined to debunking and correction of disinformation only. They are of importance to the implementation of other projects in the digital environment related to better digital literacy, ethical civic and professional journalism. Therefore it is necessary evaluation to follow also the diversified inputs of fact checking organizations.

Realistically speaking fact-checkers could not solve the problems with fake news and hoaxes alone and their collaboration with other stakeholders is mostly needed. More allies should support and promote the work of the fact checking and debunking organizations. The media is not a unified group and despite social media public media can be considered another relevant partner to fact checkers. However, politicization or the poor financial condition of public service media can complicate these relationships which at the moment remain underdeveloped. There is no interaction with the media regulators and the media accountability bodies, too.

The adoption of new communication and regulatory policies and a consistent digital information literacy strategy becomes more and more urgent in Europe. These long awaited documents could also benefit from the experiences of the fact checking and debunking organizations. 


Bae Brandtzaeg, P., & Følstad, A. (2017) Trust and Distrust in Online Fact-Checking Services. Communications of the ACM, 60 (9), 65-71.

Bounegru, L., Gray, J., Venturini, T.,Mauri, M. (eds.) (2017) A Field Guide to Fake News: A Collection of Recipes for Those Who Love to Cook with Digital Methods. Public Data Lab, Research Report.

Des Freedman. (2018) “‘Fake news’ and Facebook: symptoms not causes of democratic decline”, Inforrms Blog. Retrieved 1 December 2018, URL:

European Commission. (2018a) Final results of the Eurobarometer on fake news and online disinformation. Retrieved 1 December 2018, URL:

European Commission. (2018b) A multi-dimensional approach to disinformation.Report of the independent High Level Group on fake news and online disinformation. Directorate-General for Communication Networks, Content and Technology. Retrieved 1 December 2018, URL:

European Commission Press release. (2018) Tackling online disinformation: Commission proposes an EU-wide Code of Practice. Retrieved 1 December 2018,URL:

Governance and Internal Control in Non-governmental Organisations. Retrieved 1 December 2018,URL: 

Greenblatt, A. (2017) How fact-checkers will respond to new challenges: Some solutions. Retrieved 1 December 2018,  URL:

Graves, L., & Cherubini, F. (2016) The Rise of Fact-checking Sites in Europe. Reuters Institute for the Study of Journalism.

Morgan, T. (2018) Public service TV can be a 'bulwark' against fake news, Lord Puttnam says in Goldsmiths Press book. Retrieved 1 December 2018,  URL:  

Nyhan, B., &  Reifler, J. (2015) Displacing Misinformation about Events: An Experimental Test of Causal Corrections. Journal of experimental political science 2 (1), 81 – 93. 

Pavleska,T., & Jerman - Blazic, B.(2013) A holistic approach for designing human-centric trust systems. Systemic practice and action research, vol. 26, no. 5, 417- 450. 

Pavleska, T., Skolkay, A., Zankova, B., Ribeiro, N., & Bechmann, A. (2018) Performance analysis of fact-checking organizations and initiatives in Europe: a critical overview of online platforms fighting fake news (manuscript)

Rannard, R. (2017) How fake news plagued 2017, Retrieved 1 December 2018,  URL:

The Independent. (2017) Fake news' named Collins Dictionary's official Word of the Year for 2017. Retrieved 1 December 2018, URL:  

The International Fact-Checking Network's code of principles. Retrieved 1 December 2018,  URL:

Vertesi, J. (2016) Seizing the Digital. Engaging Science, Technology, and Society 2, 180 -192,  

Wardle, C., & Derakhshan, H. (2017) Information Disorder. Toward an interdisciplinary framework for research and policymaking. Strasbourg: Council of Europe.

The information is prepared by the team of the COMPACT project.

COMPACT is a Coordination and Support Action funded European Commission under framework Horizon 2020.

The objective of the COMPACT project is to increase awareness (including scientific, political, cultural, legal, economic and technical areas) of the latest technological discoveries among key stakeholders in the context of social media and convergence. The project will offer analyses and road maps of related initiatives. In addition, extensive research on policies and regulatory frameworks in media and content will be developed.

Let's talk

If you want to get a free consultation without any obligations, fill in the form below and we'll get in touch with you.