Fighting fake news and hoaxes in the age of convergence: performance evaluation and policies against information disorder

Bissera Zankova

Media 21 Foundation, Bulgaria

Abstract

With the proliferation of the new information and communications technologies, the quality of the information environment is a question that becomes of paramount importance for every person and society. The veracity of the disseminated content is a big challenge nowadays, both for those that disseminate it, as well as for those that consume it. As the spreading of fake news is on the rise, false stories and perverted information on the Internet have become a powerful factor for disrupting the information order and turning it into information disorder. However, the successful counteracting of the negative consequences of this phenomenon requires not only a careful study of its nature and impact but the possible paths and instruments for fighting it. The article will discuss the policy measures taken on an European scale and the activities of the fact-checking organizations and their efficiency in particular. 

Keywords: fake news, information disorder, factchecking organizations, performance evaluation, policies against fake news

1. Introduction – the modern term of terms

Journalists, politicians and scientists have agreed with the real news that fake news is the Word of the Year for 2017. Defined as “false, often sensational, information disseminated under the guise of news reporting”, “fake news” will now be added to the next print edition of Collins Dictionary. 

It will not be an exaggeration to state that false news has changed our understanding of the information society we are immersed in and more particularly of the role of the media as opinion makers. Therefore it is not about a popular term only but about an issue people have to deal seriously with because of its strong impact on their life perceptions. The Reuters Institute Digital News Report (Nielsen 2017:10) stresses that only a quarter (24%) of the respondents think “social media do a good job in separating facts from fiction, compared to 40% for the news media.” This conclusion points to the fact that “users feel the combination of a lack of rules and viral algorithms are encouraging low quality and allowing fake news to spread quickly.” (Ibid) However, not only social media can be blamed for spreading ungrounded stories. Traditional media relying on cosy relationships with political parties or grabbing uncritically news from the networks are also agents (wittingly or unwittingly) of fake news dissemination. Though all media pursue a public function and are accountable for their presentation the greatest responsibility to inform and educate an enlightened public falls on public service media which by their mission are obliged to provide diverse, accurate and unbiased information. Public service media can be “a bulwark against fake new” is the position of Lord Puttnam shared lately in a new book (Morgan 2018).

Tackling these problems must be premised on the people’s will to vigorously react against lies in the media. To do so they have to be digitally empowered and possess the necessary skills to identify false content and expose it. Such process in the digital environment should be ongoing. As Vertesi (2016:7 – 8) argues “a novel set of challenges evolves as our community addresses the production of knowledge in the contemporary context and these challenges require us to revisit, interrogate, and expand our theoretical, methodological, and practical toolkits. Digital systems considered symmetrically with other practices and tools provide renewed opportunities for theory-building and analysis, if we pay attention to how they intersect our sites of interest.”

The BBC journalist Georgina Rannard (2017) provides examples of media messages which carry false and misleading information usually accompanying stories and events that impact considerably our lives. “While 2018 will probably see more fake news circulating around the big stories, there are efforts to try and limit its impact” she optimistically anticipates. A general comment about this is that fake news requires complex responses but the measures against it should not imperil free Internet and free communication. The first steps have already been made by social networks companies  – Facebook is trying users to potential misinformation by displaying fact-checked articles next to disputed stories and Twitter has expanded its rules as to what is classed as hateful or harmful behaviour on the platform. Google has invested 300 mln dollars to elevate quality journalism and has boosted its projects on media literacy. However, the attempts of these private entities do not always coincide unconditionally with the public interest. The European Commission has also formulated proposals aiming at a common European approach against disinformation.It is important to know, however, who the enemy is in order for to formulate adequate measures against it. The next section will discuss how fake news and other deviations from normal communication caused by the spread of ill content result in a wider phenomenon – information disorder.

2. The broader perspective – information disorder

Nowadays fake news represents not merely harmless rumours, fency stories, or gossips. In the digital world we can witness a variety of information and communication deviations each of different scope and impact. Fake news and hoaxes are not isolated messages on the web. They generate wide and diverse repercussions – social, political, economic, digital.  Bounegru et al. (2017:6) striving to study the phenomenon in its entirety argues that “fake news may be considered not just in terms of the form or content of the message, but also in terms of the mediating infrastructures, platforms as and participatory cultures which facilitate its circulation. In this sense, the significance of fake news cannot be fully understood apart from its circulation online.” Therefore in order to identify the possible measures against fake news we have to take into account their reach and all the modifications caused in the media environment. The Digital, Culture, Media and Sport (DCMS) Committee in Britain investigating disinformation and fake news following the Cambridge Analytica data scandal came up with a report which acknowledges the frailty of the term ‘fake news’ which refers to hugely different practices. According to Des Freedman (2018) starting from the premise that fake news stretches ”from falsehoods deliberately concocted to undermine elections and referenda through to perspectives that are simply seen as unwelcome and controversial” and supports the conclusion that “government should, from now on, refer to misinformation instead”. Wardle and Derakhshan (2017) call all the negative consequences “information pollution” or “information disorder” and suggest scientists and policy makers to comprehensively examine these phenomena and the related challenges. Claiming that information pollution is a complex notion they introduce a new more elaborate conceptual framework for examining information disorder, distinguishing between its three different types: mis-, dis- and mal-information. Each of them is characterized by a certain degree of harm and falseness:

▪Mis-information is when false information is shared, but no harm is meant. 

▪Dis-information is when false information is knowingly shared to cause harm.

▪Mal-information is when genuine information is shared to cause harm, often by moving information designed to stay private into the  public sphere.

It is not by chance that Wardle and Derakhshan describe the framework for studying and counteracting information disorder as a “multidisciplinary framework for research and policy-making” (Ibid.) which goes beyond simple checking or blatant censorship. 

This approach proves fruitful because it escapes from the incidental character of fake news distribution and draws attention to its broader effects. Second to this, it allows other failures in information order to be explored, sometimes interrelated to each other. Third, devising a complex construct points to the necessity that a range of measures at different levels have to be taken on board to combat the risks to the normal communications processes. Fourth, it serves as a signal to stakeholders that all of them have to participate in the fight against the deviations of the information order – everyone within its scope and at various levels – if they wish this fight to bring tangible results. In the next section the measures against fake news and information disorder taken at a European level will be explored.

3. European efforts against information disorder

Information disorder is an all embracing term that can relate to any type of harmful content. Disinformation can have immediate perilous results and what proves really alarming is that it can make the public policy implementation much more difficult. Here also the interplay of rights is essential – on the one hand, freedom of speech and pluralism should be respected and on the other, limitations with regard to disseminating disinformation should be imposed. However, such approach is salient as people have to have a chance to make informed choices and take grounded decisions.

Recognizing this, a political intervention was undertaken by the European Union (EU) in this regard: the European External Action Service East Stratcom Task Force that ran the ‘EU vs Disinformation’ campaign with partners have identified and debunked over 3.500 disinformation cases between September 2015 and November 2017. Despite these concrete outcomes, Nyhan and Reifle (2015) argue that misinformation (even being the mildest form of information disorder – B.Z.) can prove hard to correct and may have lasting effects even after it is discredited. Therefore, only debunking it is not sufficient – it must be supplanted by an alternative causal explanation. 

These arguments and the still quick spread of fabricated and misleading information across Europe have been considered an important and urgent task to be tackled. A step forward was to set up a dedicated expert group which would come up with solid policy proposals. In January 2018, the European Commission established a high-level group of experts (“the HLEG”) to advise on policy initiatives to combat fake news and disinformation spread online. The main deliverable of the HLEG was a report designed to review best practices in the light of the fundamental principles of communication and freedom of expression and to formulate suitable responses premised on them. The HLEG had to advise the Commission against simplistic solutions and to avoid clearly any form of censorship either public or private. Based on the independent HLEG report published in March 2018 (HLEG Report 2018) as well as wider consultations carried out over the past six months, the Commission defines disinformation as “verifiably false or misleading information that is created, presented and disseminated for economic gain or to intentionally deceive the public, and may cause public harm”.

Being a multistakeholders’ entity the HLEG pursued a multidimensional approach rooted in human rights and the values of the European societies. Though not discussing in detail the various forms of information disorder, the HLEG’s recommendations aimed to elaborate longer-term responses to increase societal resilience to disinformation in particular. It also decided to produce a framework for ensuring the effectiveness of these responses and its continuous evaluation as well as the development of new evidence-based responses instead of providing short-term solutions to the most pressing problems. Fake news and disinformation is considered a “moving target” that requires constant checking whether measures taken are really efficient.

The main principle applied by the group was to reinforce the transparency of the combating activities in all its dimensions especially of the ecosystem and of the newly emerging group of fact checkers. Another one was the diversity and sustainability of the European news system. The empowerment of users served as a more personalized navigation to correct disinformation. A new type of literacy which should be developed by the European countries is the media information literacy. The increased support for quality journalism is underlined to be vital in the fight against false information. Research and development had been raised as an important pan-European goal. Among these policy measures the preparation of a multistakeholders’ code of practice regarding platforms and advertisers is stressed to be the first starting point to encouraging self-regulation and high professional standards. The idea to create a multistakeholders’ coalition against fake news could lend an organizational support to the code. 

Among the set of policy proposals the inclusion of fact checking and debunking organizations in the battle against disinformation merits a particular focus because in practice it represents an involvement of a new stakeholders’ group in the fight against information disorder. The HLEG suggested the capacity of fact checking and debunking organizations’ actions to be used to increase the transparency and efficiency of fact-checking practices.The idea is based on the multi-stakeholders’ approach that recognises the need for cross-sector and cross-border cooperation as a precondition for comprehensive tackling of disinformation.The next section will focus on the performance of the fact checking and debunking organizations and the need to evaluate it.

4. The fact-checking and debunking organizations and their performance

The role of platforms, news media and factchecking organisations in the fight against disinformation is crucial as they stay most closely to the production and distribution of false messages. The activities of these organizations for combating information disorder are not new. They have applied artificial intelligence and machine learning “to tackle specific facets of the disinformation phenomenon and filtering systems enabling the exposure of fact-checked information”. (HLEG Report: 16). As reported “trust measures have been developed as an outcome of the partnership between online platforms, publishers and independent fact-checkers.”(Ibid.) This conclusion is very general, however, and does not analyze the established relationships in depth. What is striking is that the performance of fact checkers and of fact-checking activities in general in the EU are still relatively fragmented and respectively the knowledge about it as well. That is why more collaborative work is necessary to be done either at the media level by fact-checkers, verification organizations, and professional newsrooms or at other levels with other stakeholders within the EU Member States and across the EU. A cooperation with the media subjects could build on the existing partnerships with certain online platforms in a few EU countries and be extended at the EU level, while developing innovative approaches and transparent exchanges of information. The HLEG does not provide examples of good practices in this regard and this proposal sounds somehow vague. The goal of such cooperation is to make sure that fact-checkers continuously improve the transparency of their working methods. It should be added that according to the changes in the environment they should advance their journalistic approaches, too. Graves and Cherubini (2016) express the opinion that “fact-checking outlets all share the laudable goal of promoting truth in public discourse. But political fact-checking always attracts controversy. Even simple factual questions can leave surprising room for disagreement, and fact-checkers often come under attack from critics who disagree with their verdicts.” Therefore the conclusions they come to should be not only transparent but well-grounded, as well. With respect to the improvement of the quality of their journalistic work cooperation with public service media and adoption of their standards can be very helpful.

All these briefly formulated recommendations to fact checkers are quite pertinent as they should not turn into gate keepers or censors pursuing petty interests in the current situation but follow the highest professional rules and provide high quality work examples. 

The EC (Press release, 2018) formulates two proposals with respect to fact checkers related to their organization and methods: 

• To establish an independent European network of fact-checkers: this will establish common working methods, exchange best practices, and work to achieve the broadest possible coverage of factual corrections across the EU; they will be selected from the EU members of the International Fact Checking Network which follows a strict International Fact Checking NetworkCode of Principles and
• To create a secure European online platform on disinformation to support the network of fact-checkers and relevant academic researchers with cross-border data collection and analysis, as well as access to EU-wide data.

However, to support the accomplishment of these objectives data concerning the organizations’ management and operation proving that they have the potential to fulfill the significant tasks set on them is lacking. If they are recognized to be one of the chief fighters against fake news the efficiency of their activities merits particular focus. With respect to the public their transparency and openness of action is crucial.

There is no single definition of fact-checking organizations and scientists apply a variety of methods to determine and describe these organizations. A recent study by Bae Brandtzaeg and Føolstad (2017), for instance, divides fact-checking services into three general categories based on their areas of concern: 1) political and public statements in general; 2) online rumors and hoaxes and 3) specific topics, controversies. particular conflicts or narrowly scoped issues and events. Graves and Cherubini (2015: 8 – 10) speak of two models of fact checking organizations the journalistic news room model and the NGO model. None of the authors cited, however, have delved into their internal structure and management of resources.

Fact checkers around the globe have also set up an entire professional network. The International Fact-Checking Network (IFCN) launched in September 2015 to support fact-checking initiatives by promoting best practices and exchanges among organizations in this field is a unit of the Poynter Institute dedicated to bringing together fact-checkers worldwide. The association also adopted a Code of principles in 2016. The International Fact-Checking Day on April 2 is promoted by the IFCN in partnership with fact-checking organizations around the world. The day advances the idea of fact checking as a common undertaking and a basis for successful action – in order for an accurate information ecosystem to operate it requires everyone to do their part. 

The principles in the Code represent a bunch of professional standards such as nonpartisanship and fairness, transparency of sources, transparency of methodology and open and honest corrections. These comprise the values on which the fact-checking activities are premised and they are similar to journalistic and media codes of work. However, these organizations have not adopted clear criteria for the assessment of their performance which are also publicly known. In this respect how human and machine actions complement each other and what is the overall effectiveness can be a special focus of research. As Rubin and Conroy (2012) and Wineburg et al. (2016) claim, one serious drawback of the fact-checking and debunking activities is related to the fact that human observers perform poorly in the detection of fake news, and machines even slightly outperform humans on certain tasks. The observation raises a series of essential issues: how effective are these organizations, how efficient are the tools aimed to combat disinformation,  how human effort and machine operation relate to each other, which factors affect their performance and eventually how to evaluate the performance.

In a recent study Pavleska, Školkay, Zankova, Ribeiro and Bechmann (2018) deal in depth with these problems in their manuscript “Performance analysis of fact-checking organizations and initiatives in Europe: a critical overview of online platforms fighting fake news” which attempts at distilling performance indicators rooted in the interaction of the non-governmental and governmental sectors. The toolbox scholars suggest comprises internal and external coordination, tracking impact, tracking progress, clarity of objectives, transparency, self-assessment, alignment with the regulatory policies and incentives’ policy.“ The analysis of all of the identified indicators are “implicitly or explicitly embedded into the analysis of the effectiveness and efficiency indicators whose maximization is the most desirable performance aspect”, the authors claim.

In addition to defining performance indicators for the evaluation of fact checking organizations, an initial survey was also designed and carried out containing a basic set of questions addressing the identified indicators. This survey was piloted in several organizations of this type. Based on an extended search, 50 European organizations located in 27 countries were approached. On the basis of the data collected the scholars come up with a range of interesting insights concerning the problems faced by fact checking and debunking organizations and the possible policy solutions for fostering their activities. In the next section the difficulties encountered by fact-checking and debunking organizations will be summarized and discussed.

5. The fact checking and debunking organizations’ problems

The problems faced by fact-checking and debunking organizations require both internal and external steps towards their proper settling. The biggest reported challenge for these organizations relate to the insufficient stakeholders’ awareness of the issues concerning information disorder and the lack of adequate resources for their counteracting. Fact checkers should be encouraged to publicise widely information about their activities, methods and outcomes accomplished. Factually articles concerning their work are increasing particularly on Poynter’s International Fact-Checking Network site where materials presenting local themes can also be found. However, specific information about the management and staff are missing including data about financing of projects and campaigns. Graves and Cherubini (2016: 28 – 29) underline that the “budgets for these efforts span a wide range, but remain quite low for the typical outlet.” In practice there are two main sources of funding – by the media industry and through donations – each of which carry its risks for the independence of these entities. Greater transparency would contribute to better using and controlling of resources and to finding more donors which eventually will enhance the fact checking organizations’ performance.

Although there is a clear general goal set for all of the fact checking and debunking organizations, there is a lack of clarity on the part of the organizations in the sub-goals and objectives that concern the internal bodies, procedures, staff management and operation processes as required for non-governmental organizations notwithstanding the fact they can operate as journalistic organizations or as typical NGOs.(Governance and internal control) The reforms that shall be implemented with a view of strengthening these entities demand considerable improvement of the organizational culture and internal practices pursued by them about which there is no transparency (at least at the moment). Such re-organizational steps can help also fact-checking organizations connect to new audiences and thinking about what kind of audience is open to the concept of fact-checking and how best to reach them.As reported, a large number of readers and viewers isn’t being reached – not only partisan voters who are skeptical about media, but non-voters who are not engaging fully in civic life. (Greenblatt 2017)

The majority of fact checking and debunking organizations are ‘specialized’ in one type of content only. Specific visual content (photos, YouTube), although known to have far greater impact in the proliferation of fake news and hoaxes than text, is checked to a lesser extent. However, as Greenblatt (2017) states fact-checking organizations are prone to finding new formats, including better use of graphics and visual aids to make their information more accessible; considering shorter writing; making sources more visible; and taking advantage of all social media platforms. This piece of information can be considered as an optimistic move made by fact checkers towards the control of a new content and its better presentation.

The number of debunked news/hoaxes (last three-months average) varies highly across countries and is very much context dependent – influenced by both general political situation and ad hoc events such as election campaigns which means that there is uneven working pace of these organizations which obviously speeds up under certain conditions. Such observations point to the necessity of securing good governance and internal coordination as basic conditions for effective performance as well as of the availability of appropriate guarantees for preserving their independence and non-politicization and serving only the objective of truth in times of political conflicts. There also appears a significant overlapping in tackling information disorder, when there are countries with two or even more separate fact checking and debunking organizations and such situation poses the problem of fixing the scope of every one of these organizations and achieving good coordination among them.

With respect to the working methods of the fact checking and debunking organizations the results show that the extent to which automated and semi-automated software are employed in these projects is very low, although there are already complete end-to-end computational fact-checking solutions available. Most of the organizations do pay attention to embracing new tools and adopting of innovative methods, but there is still a significant number of them that have not yet considered this option due possibly to financial reasons. IT companies can help in this regard providing advice and technical support to these organizations and social media giants can also consider the opportunity to investing in such projects. However, the new methods may generate new challenges.

Still most of the fact checkers do not track, monitor and evaluate any impact (human, political and social), which may jeopardize in practice their long-term results and generally the far reaching effect envisaged. One can come across on little or no considerations at all with regard to the sustainability plans of these organizations which can put at stake their activities in the future.

The transparency of the majority of fact checkers (in terms of methodology, funding and operation) in Europe remains blurred. This fact raises the issue of assuring more intense civil society and public involvement.In the same vein are the recommendations of the HLEG and the European Commission – “transparency is a key element in the response to digital disinformation. It is a cross-cutting issue that concerns the whole digital media value chain and aims at making news production and distribution less opaque with a view to supporting users’ ability to better discern between journalistic quality content and various kinds of disinformation”. (Report: 22). Efforts should be made on the part of fact checkers to do a high quality job and gain the trust of the audience through better communication such as share with their audience information about why they’re checking claims, explain why certain sources of information, such as statistics from government agencies, are seen as more reliable than others, bring diverse organizations together to check important claims, providing assurance that a variety of fact-checkers agree on the substance behind a controversial topic, create a database of verified facts available for the public to share and analyze, etc. In the digital age trust manifesting all dimensions becomes a central category. As Pavleska and Jerman-Blazic argue current systems do not account for important factors when calculating trust and system design should be changed for improving the user-experience and system-performance.If taking into account these recommendations fact checking organizations can be shaped in a way that supports and incentivize trustful behaviour. This implies limiting all three types of information disorder: disinformation, misinformation and malinformation.(Pavleska, and Jerman-Blazic 2013)

These inferences can serve as a common ground for policy initiatives aiming at entrenching the fact-checking and debunking organizations’ position and encouraging a better contribution to the fight against information disorder. The general conclusion however, is that notwithstanding shortcomings in their work the activities of fact checking and debunking organizations are grass root initiatives that should be supported and expanded.

6. The way forward: improvement of performance and effective cooperation policies

Fact checking and debunking organizations in order for to be effective should improve their journalistic and organizational performance including through the changes in their internal management and culture. Their intentions to modernize their methodology, structure and manner of work should be supported by other stakeholders. Clear-cut criteria for evaluation should be designed in this respect. Solutions may combine both technical and human characteristics.

On a professional scale fact checking and debunking organizations should broaden their methodological means for approaching fact-checking issues.This means relying on a variety of experts from different fields, and being open for employing novel approaches including computational semantic analysis. (Pavleska, et al.) The latter even appears to be urgent, considering the limited, imperfect, slow and costly human-based approaches to fact-checking and debunking, and, moreover, the emergence of artificial intelligence techniques for creation and distribution fake news and hoaxes (so called digital factories). The need for greater speed in bringing fact-checks closer in time to false claims is crucial. More than this fact-checkers should find ways to incorporate fact-checks into people’s regular news consumption behavior.

In this respect, close cooperation with IT companies and the business sector at large proves crucial. The academic community in particular can engage with the exploration of the performance assessment based on additional set of criteria/indicators, such as economic indicators and by and large of the structure and functions of fact-checking and debunking organizations with the purpose of improving their performance. 

Fact-checking and debunking activities could not solve the problems with fake news and hoaxes alone. In this respect their collaboration with other stakeholders proves crucial. There is a need for a fundamental change in the communication and the regulatory policies and practices. The problem of pursuing new digital information literacy becomes more and more urgent. The elaboration and the effective implementation of these policies is an obligation of the public authorities in Europe which could also take into account the experiences of the fact checking and debunking organizations. 

The efforts of a stronger and independent civil society should underpin the process of awareness-raising among stakeholders and the public at large in order to give more credibility to the fact checking work and bring to the fore the issues related to tackling information disorder. At the same time, the civil society debate over the activities of stakeholders and the  fact checkers in particular is an essential element of the overall scrutiny in a democratic society for greater openness and transparency of all organizations and institutions. Allies to fact checking and debunking organizations are mostly needed to support and promote their activities. A very promising alignment can be established with the media on a national, European and global level. National organizations can broaden their partnerships with local media, offering up information that perhaps could be presented more prominently to capture local audience. In this partnership not everything is smooth. Graves and Cherubini (2016) draw attention to the fact “in media environments dominated by partisan outlets, fact-checkers worried that their reputation would be stained by journalists who distort their work or cite it selectively.” Having in mind this admonition it is necessary journalists to be aware of the important function pursued by fact checking organizations and relations of mutual trust and assistance to be established. However, the media is not a unified group of subjects and despite social media public media can be considered another relevant partner to fact checkers. However, politicization or the poor financial condition of public service media can complicate these relationships which at the moment at least remain underdeveloped.

Another possible cooperation which is expected to expand in the future is the one between the media accountability bodies and the network of fact checking and debunking organizations for the exchange of good practices for fostering professional and responsible journalism.

This policy paper is a side-result of the activities taken within the COMPACT Project. The COMPACT Project represents a Cooperation and Support Action under the EC H2020 umbrella, which deals with broader themes of social media and convergence.

Sources

Bae Brandtzaeg, P., and Følstad, A. (2017) Trust and Distrust in Online Fact-Checking Services.In: Communications of the ACM 60 (9): 65-71.
Bounegru, L., Gray, J., Venturini, T.,Mauri, M. (eds.) (2017) A Field Guide to Fake News: A Collection of Recipes for Those Who Love to Cook with Digital Methods. Public Data Lab, Research Report.
Dean, J. (2018) Google’s pledges $300m to support quality journalism, 21 March 2018. Available at https://www.thetimes.co.uk/article/google-s-pledges-300m-to-support-quality-journalism-pkqgjdmf0 (Accessed 10 August 2018).
Des Freedman. (2018) ‘Fake news’ and Facebook: symptoms not causes of democratic decline, 5 August 2018, Inforrms Blog. Available at https://inforrm.org/2018/08/05/fake-news-and-facebook-symptoms-not-causes-of-democratic-decline-des-freedman/(Accessed 10 August 2018).
European Commission. (2018a) Final results of the Eurobarometer on fake news and online disinformation, 12 March 2018. Available at https://ec.europa.eu/digital-single-market/en/news/final-results-eurobarometer-fake-news-and-online-disinformation (Accessed 10 August 2018) .

European Commission. (2018b) A multi-dimensional approach to disinformation.Report of the independentHigh level Group on fake news and online disinformation. Directorate-General for Communication Networks, Content and Technology.

European Commission Press release. (2018) Tackling online disinformation: Commission proposes an EU-wide Code of Practice, 26 April 2018.Available at http://europa.eu/rapid/press-release_IP-18-3370_en.htm (Accessed 10 August 2018).
European Union Code of practice on disinformation. (2018) Available at http://ec.europa.eu/information_society/newsroom/image/document/2018-29/msf_on_disinformation_17_07_2018_-_proofread_99F78DB7-9133-1655-990805803CDCCB67_53545.pdf (Accessed 10 August 2018).
EU v. Disinfo, EU vs Disinformation campaign. Available at https://euvsdisinfo.eu/about/ (Accessed 10 August 2018).
Governance and Internal Control in Non-governmental Organisations. Available at http://cpas.icac.hk/UPloadImages/InfoFile/cate_43/2016/208ec0bd-878e-4dc0-b755-c4a230b39bdd.pdf (Accessed 10 August 2018).
Greenblatt, A. (2017) How fact-checkers will respond to new challenges: Some solutions, 4 April 2017. Available at https://www.americanpressinstitute.org/publications/reports/white-papers/fact-checkers-solutions/ (Accessed 10 August 2018).
Graves, L., and Cherubini, F. (2016) The Rise of Fact-checking Sites in Europe. Reuters Institute for the Study of Journalism.
Independent. (2017) Fake news' named Collins Dictionary's official Word of the Year for 2017, 2 November 2017. Available at https://www.independent.co.uk/news/uk/home-news/fake-news-word-of-the-year-2017-collins-dictionary-donald-trump-kellyanne-conway-antifa-corbynmania-a8032751.html (Accessed 10 August 2018).
Morgan, T. (2018) Public service TV can be a 'bulwark' against fake news, Lord Puttnam says in Goldsmiths Press book, 13 April 2018. Available at https://www.gold.ac.uk/news/fotv/ (Accessed 10 August 2018).
Nielsen, R. Kleis. (2017) Digital News Report 2017. Oxford: Reuters Institute for the Study of Journalism.
Nyhan, B.,  and Reifler, J. (2015) Displacing Misinformation about Events: An Experimental Test of Causal Corrections.In: Journal of experimental political science 2 (1): 81 – 93. 
Pavleska,T., and Jerman - Blazic, B.(2013) A holistic approach for designing human-centric trust systems.In: Systemic practice and action research, ISSN 1573-9295, 2013, vol. 26, no. 5 : 417- 450, doi: 10.1007/s11213-012-9259-3.
Pavleska, T., Skolkay, A., Zankova, B., Ribeiro, N., and Bechmann, A. (2018) Performance analysis of fact-checking organizations and initiatives in Europe: a critical overview of online platforms fighting fake news (manuscript)
Rannard, R. (2017) How fake news plagued 2017, BBC News, 31 December 2017. Available at https://www.bbc.com/news/world-42487425 (Accessed 10 August 2018).
Rubin, V.L., and Conroy, N. (2012) Discerning truth from deception: Human judgments & automation efforts.In: First Monday 17:  3 – 5.
The International Fact-Checking Network's code of principles. Available at https://ifcncodeofprinciples.poynter.org/(Accessed 10 August 2018).
Vertesi, J. (2016) Seizing the Digital. Engaging Science, Technology, and Society 2: 180 -192,  DOI:10.17351/ests2016.75. 
Wang, S. (2018) Google announces a $300M ‘Google News Initiative’ (though this isn’t about giving out grants directly to newsrooms, like it does in Europe), 20 March 2018. Available at http://www.niemanlab.org/2018/03/google-announces-a-300m-google-news-initiative-though-this-isnt-about-giving-out-grants-directly-to-newsrooms-like-it-does-in-europe/(Accessed 10 August 2018).
Wardle, C. , and Derakhshan, H. (2017) Information Disorder. Toward an interdisciplinary framework for research and policymaking. Strasbourg: Council of Europe.
Wineburg, S., McGrew,S., Breakstone,J., and Ortega, T. (2016) Evaluating Information: The Cornerstone of Civic Online Reasoning. Stanford Digital Repository.