Today Civil Society Organizations (CSOs) from around the world are using IT tools and public open data as essential ingredients in their watchdog activities. Often CSO’s are using the data already available online, while in other cases they put tremendous efforts in acquiring the public data that is not available. This influences their ability to spark civic engagement and inform citizens about the work of state institutions.
This article intends to help CSOs in their efforts to design and develop open data projects*. It highlights the most common Do’s and Don'ts and can serve as a roadmap and a source of inspiration for civil society organizations that are just entering the field of technology.
It explores learnings from developing open data projects in several countries from Central, Eastern, and South-Eastern Europe and provides closer look into their rationale, links with target audiences, and the main stages of planning and development.
Inputs of six civil society organizations ePanstwo Foundation, Poland; Civil Kapocs, Hungary; K-Monitor, Hungary; Fair-play alliance, Slovakia; Transparency International, Lithuania; IRSI, Russia; were gathered through a survey questionnaire designed to reflect their experience in the processes of developing an open data project.
The central question that we have tried to answer is:
How to develop and launch an open data project that fits the needs of the target audience?
*In this article we define an open data project as a project initiated and developed by a civil society organization, aimed to inform citizens on certain policy issue through gathering public data, opening the data, and making it available for further use, distribution or analysis.
Running an open data project in most of the cases is not just about opening the data. It is about revealing information that otherwise would remain untold or impartial. In this chapter, we will explore how project ideas are born and what are the reasons behind their existence?
According to our survey, lack of governmental transparency is one of the key reasons why many CSOs develop open data projects. In many cases, their effort is a response to the institutional failure to properly inform people on the issues of public interest.
By gathering and presenting the data on decision-making processes or fiscal transparency to the public, CSOs, along with the media, are aiming to reveal cases of wrongdoing and power misuse, and hold the governments accountable.
With this in mind, Transparency International Lithuania has developed their open data platform Stirna.info, aimed to bring greater transparency over the media ownership in the country.
“TI Lithuania had a purpose to bring more transparency into the media landscape and give better opportunities to analyze media content by creating a user-friendly platform which shows the connections among media, business and politicians in a simple manner”, said Rugile Trumpyte from TI Lithuania.
They intended to use Ministry of Culture datasets but from the start hit the wall. There was no information available in open data formats. It required a lot of manual work to gather and systematize it to make Stirna.info a workable solution.
In another similar case, the organization Fair Play Alliance from Slovakia developed a project called Datanest as a response to the lack of structured and searchable information on the distribution of public procurements.
“This platform serves as a database for tracking public money which helps to control the flow of public sources in Slovakia. It maps the handling of money in public procurements, sponsorship of political parties, subsidies or tax breaks and more”, said Pavol Lacko from Fair Play Alliance, Slovakia.
However, at the early stage of the development it became clear that the data they need is not publicly available at all. Thus, Fair Play Alliance had no other choice than to gather the data themselves by sending requests for information to the public institutions, and later to transfer it in digital formats to “make it open”.
According to Lacko, even if information is not available publicly, it still can be requested from state institutions and brought to the people in a coherent way. However, his work would have been more efficient if the public data was already opened by its main holder - the state institutions.
Another platform of Fair Play Alliance Znasichdani.sk is created to trace the public spendings by synthesizing data from Public Procurement Bulletin and Slovak Business Registry and providing access to the public to the companies involved in public procurements and individuals somehow involved or connected to those companies in a same platform. However, citizens or watchdogs and journalists don’t always have the time and energy to look into the details of such a specific topics.
This requires a middleman with the capacity to perform research and present it’s highlights in an easy-digestible way. Sandor Lederer who leads the Hungarian public procurements monitoring platform Red Flags designed it specifically to make it clear and user-friendly for journalists and watchdogs, and support their investigations by bringing the most troubling trends to the surface.
At the same time, a lot of open data projects are developed in a hope that making the information searchable, digitally standardized, and accessible will influence the work of public institutions that will use CSOs solutions as a model for good practices.
This technocratic positivism is challenged every day by the harsh reality in which public institutions operate in the region: they are often outdated and unmotivated to innovate.
“They [public institutions] also need clear indicators showing the effectiveness of their work. Most of them still have lack of skills on organizing the process of open data publication”, says Viacheslav Romanov, director of Analytics at Russian Infometer project center.
Therefore, it’s instrumental for the success of any open-data project to bring on board all the stakeholders, including those from public sector. We need to take into account the knowledge gap that often lies between the developers who have a technical solution and authorities who have the data to make this solution work.
One of the most important aspects that was pointed out by almost all respondents was the importance of developing a projects relevant to the needs of the target audiences. Advocating public interest is central to CSO’s work and should be fundamental for any open data project.
There is no magic recipe or a single solution that will bring relevance to your project, but there are few tips shared by our interviewees that can help to bring your audience closer and through this to amplify the impact of your initiative.
Good targeting of your project should lead to a stronger engagement of the audience and, as a consequence, stronger impact.
Mailing lists and phone calls is a cheap and straightforward way to establish the initial contact.
“We used mailing lists and telephone calls to develop first contact with the governmental bodies, then we interacted via our web-platform. The best way to make them active and motivated is to show you can ease their duties and also make them look better in the [global transparency] ratings”, said Viacheslav Romanov, director of Analytics at Infometer project center.
Interviewing is a great way to test the direction of your work, make improvements and keep your audience updated and engaged.
For example, while working on redflags.eu, K-Monitor reached out to future users (journalists, public procurement experts, practitioners) and conducted interviews with them at several phases of the project. Having all stakeholders on board while working on the project is also essential for finding out-of-the-box solutions and holding reality checks.
In the same way surveys and questionnaires can provide feedback from the audience that can help to improve project’s functionality and user experience.
“We repeatedly asked our target audiences for feedback on how they use or don’t use Datanest. How user friendly is the design, what to they miss in it? We tried to implement their suggestions or wishes if possible. Especially in later phases of development we were looking for their expertise and experience in using Datanest”, said Pavol Lacko from Slovakian Fair-play Alliance.
Lastly, face to face meetings and focus groups can provide a lot of information and generate discussions that will help you to go beyond your initial agenda. It is hard to substitute a live discussion, but good video conference tools potentially can help you to achieve similar results with fewer investments.
According to the results of our survey most of open data projects, while having different objectives, are still following similar conceptual path towards success.
Knowing your audience and its needs is only one essential element. Others include thorough planning, testing, and maintenance.
So what are the most important stages which shouldn’t be skipped in the process of your open data project development?
To understand how to move forward you’ll need to perform an overall evaluation when the project will be in a full swing. You should be able to detect positive and negative trends and use them for future improvements or new projects.
While each step mentioned above is important and requires equal attention and commitment, things can go wrong almost at any point of development. Specifically sensitive are stages of obtaining the data, choosing the technology solution, and testing.
Very often you’ll have to combine various approaches to obtain the data as others will fail. For example, in the case of Datanest.sk, one type of information was gathered automatically by using ETL tools (data capture software for real-time data integration). This way the team was able to extract data from state online publications. However, other types of data was not available online and had to be requested from the state institutions. They forwarded it, but not in a machine-readable format. So the team had to take additional effort to manual transcribe state owned data before they could add it to the existing database, analyse or interpret.
There will always be a clash between the budgetary constraints and wishful thinking for your open data project. However, there are many existing open source solutions that can be applied without (or with minimal) financial implications. You just need to find the one that will be compatible with your idea.
“Ideally it's better to reuse technologies. Practically it depends on many factors such as the quality of already existing code, support from the team that created previous solution, and how well it is suited to our needs” said Krzysztof Madejski, from the Polish ePanstwo Foundation.
Finally, piloting can become a lifesaver for your project since making any changes after the official launch is usually much more complicated and costly than at earlier stages.
“Often huge and complicated solutions are being designed at the beginning and later on they are implemented one to one without sufficient testing which would reveal many “teething problems” of the project” - said Palo Lacko Pavlo from Fair-play Alliance.
In the final part of this article we will touch upon some of the external factors that may have an influence on your open data project.
We already mentioned that availability of open data in a machine-readable formats is a major challenge for civic activists in the region. Very often your initiative might be jeopardized by the hostile political climate, apathy, and sometimes resistance (or in some cases even competition) of public institutions when it comes to cooperation. This creates additional costs and the need for more human, time, and financial resources.
According to our respondents, another major factor influencing open data projects is related to the short-term grant-based model of financing CSOs employ. It often leads to problems with further sustainability and maintenance of the project.
Good news is that sustainability and maintenance challenges can be addressed through early and detailed planning. Financial and human resources are crucial and failing to secure them in a long-term perspective puts all your efforts in danger. Although, some donors are changing their project-based mindset and provide long-term support of open data initiatives, finding sustainable financial support remains to be a major struggle for CSO’s if they are not part of a wider program.
“The budget of the project allowed us to build only a part of indicators we planned. Since data publication continuously changes, the program has to be adapted to these changes”, said Sandor Lederer from K-Monitor.
Many CSO’s are trying to solve this issue by combining different grants. However, finding matching funds is challenging. At the same time, obsession with constant “innovation” and “success culture” makes it much easier to get support for a new project than for the maintenance or improvement of an existing one.
Some CSO’s are exploring alternative ways to achieve financial stability, such as social entrepreneurship or crowdfunding, yet many still lack expertise and experience to embrace them successfully.
Nevertheless, early financial planning for implementation, improvement, and maintenance of your open data project is crucial for finding applicable solutions.
It’s important to realise that if you are aiming to achieve lasting social or behavioral change, launching the website will be just the beginning of your work - not the end of it.
As Pavlo Lacko puts it: “it is crucial to concentrate not only on the IT development of the data project but also on further education and training of your potential audience, so it would know how to use it and why. Opening up data is only the beginning. Reaching out to the public must be the next step to give the open data solution real impact and social sense”.
Riste Zmejkoski is former Program Manager at the Balkan Investigative Reporting Network, responsible for developing several open data platforms and digital tools in Macedonia, including internationally recognized and awarded database "Skopje 2014 Uncovered" and the "Foreign Investments Uncovered" database, shortlisted for the global Data Journalism Award in the Small newsrooms category for 2017.