Daniel Antal | Reprex

Learn R with Reprex

Fri, 07 Oct 2022 12:35:00 +0200

Big Data Creates Inequalities

Only the largest corporations, best-endowed universities, and rich governments can afford data collection and processing capacities that are large enough to harness the advantages of AI.

Fullscreen: F

Next: ️> or Space | Previous :️<
Start: Home | Finish: End
Overview: Esc| Speaker notes: S
Zoom: Alt + Click 🖱️

Big data that works for all

No matter how big is the problem or how small is your team, `Reprex` fill your reports, dashboards, newsletters, books with data and its visualization.
Learn R with us: you can reduce the inequalities by joining the open source movement, learning to run open source software, ask for help, improve the tutorials, the documentation, and eventually learn to make the computer work for you.
Contributor Covenant: Participating in open source is often a highly collaborative experience. We’re encouraged to create in public view, and we’re incentivized to welcome contributions of all kinds from people around the world. This makes the practice of open source as much social as it is technical.

Get Inspired

Find more interesting and better data: you don’t have to be a data scientist or write code to contribute to our projects.
Data feminism: Catherine D’Ignazio and Lauren Klein present a new way of thinking about data science and data ethics—one that is informed by intersectional feminist thought. Highly inspirational, free, open-source book.
RLadies is a world-wide organization to promote gender diversity in the R community.

Contributor Covenant

We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.

Run code from tutorials

retroharmonize.dataobservatory.eu
🖱 Get started
[🖱️ Articles](https://retroharmonize.dataobservatory.eu/articles/index.htm

Find help, ask for help: reprex

Documentation for better tutorials

Debugging and testing code

Contribute to documentation

R is a functional language

R is both a statistical environment and a programming language
R, the open source and further developed version of the S language, is mainly functional
If you did a task at least twice, the 3rd time you better write a function script to keep doing it forever.
Most of your effort will be to find a well-written function for your work
If you cannot find a function, you will modify somebody else’s function, or eventually write your own

R + YAML + markdown = web ready

Learn YAML in Y minutes: tell the computer what you want to do with a document
R Markdown basics: it is just a plain markdown that allows you to insert little R program ‘chunks’.
Awesome markdown editors and pre-writers: find a convenient tool
Google Docs to markdown: practice by translating your Google Docs text to markdown. It is very easy.

Package and release: a team effort

Our open source development projects

🔢 dataset: Synchronize datasets with global knowledge hubs #️⃣ statcodelists: Make your data codes understood globally ♻️ iotables: Create economic or environmental impact assessments in any EU country 🌍 regions: Create from raw survey data more granular statistics in any EU country ✅ retroharmonize: Harmonize questions banks, recycle answers from past surveys ⏭️ all in on one page

Create with us

Questions?

Email | Keybase

LinkedIn: Daniel Antal - Reprex | Home

Learn R with Reprex

Fri, 07 Oct 2022 12:35:00 +0200

Big Data Creates Inequalities

Only the largest corporations, best-endowed universities, and rich governments can afford data collection and processing capacities that are large enough to harness the advantages of AI.

Fullscreen: F

Next: ️> or Space | Previous :️<
Start: Home | Finish: End
Overview: Esc| Speaker notes: S
Zoom: Alt + Click 🖱️

Big data that works for all

No matter how big is the problem or how small is your team, `Reprex` fill your reports, dashboards, newsletters, books with data and its visualization.
Learn R with us: you can reduce the inequalities by joining the open source movement, learning to run open source software, ask for help, improve the tutorials, the documentation, and eventually learn to make the computer work for you.
Contributor Covenant: Participating in open source is often a highly collaborative experience. We’re encouraged to create in public view, and we’re incentivized to welcome contributions of all kinds from people around the world. This makes the practice of open source as much social as it is technical.

Data Feminism

Get Inspired

Find more interesting and better data: you don’t have to be a data scientist or write code to contribute to our projects.
Data feminism: Catherine D’Ignazio and Lauren Klein present a new way of thinking about data science and data ethics—one that is informed by intersectional feminist thought. Highly inspirational, free, open-source book.
RLadies is a world-wide organization to promote gender diversity in the R community.

Contributor Covenant

We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.

Run code from tutorials

retroharmonize.dataobservatory.eu
🖱 Get started
🖱️ Articles

Find help, ask for help: reprex

Documentation for better tutorials

Debugging and testing code

Contribute to documentation

R is a functional language

R is both a statistical environment and a programming language
R, the open source and further developed version of the S language, is mainly functional
If you did a task at least twice, the 3rd time you better write a function script to keep doing it forever.
Most of your effort will be to find a well-written function for your work
If you cannot find a function, you will modify somebody else’s function, or eventually write your own

R + YAML + markdown = web ready

Learn YAML in Y minutes: tell the computer what you want to do with a document
R Markdown basics: it is just a plain markdown that allows you to insert little R program ‘chunks’.
Awesome markdown editors and pre-writers: find a convenient tool
Google Docs to markdown: practice by translating your Google Docs text to markdown. It is very easy.

Package and release: a team effort

Our open source development projects

Create with us

Questions?

Email | Keybase

LinkedIn: Daniel Antal - Reprex | Home

Surveyharmonization

Sun, 25 Sep 2022 12:00:00 +0000

Survey Harmonization Workflow

Controls

Next: Right Arrow or Space
Previous: Left Arrow
Start: Home | Finish: End
Overview: Esc | Speaker notes: S | Fullscreen: F

Principles

Generic concept of surveying, i.e. examining and record the area and features of (an area of land) to construct a map, plan, or description.
Structured data collection of the missing information, harmonization of knowledge.
Reproducibility and not automation. On a small scale, anything can be done with Ctrl C + Ctrl V. But it should be recorded, documented for future Ctrl C + Ctrl V.

Timeline

Concepts

We harmonize knowledge concepts. Because knowledge concepts are very abstract, the harmonization of concepts requires an iteration of desired output and questionnaire or form items, and it will be carried on throughout the project. The harmonization of concepts will allow us to link our survey data to pre-existing survey data, financial information, or any other source of information.

Data Model

Data modeling enables us to place the information we gain from existing sources, for example, by recycling pre-existing questionnaire items and answers to a knowledge graph together with our data. A knowledge graph is a more flexible, future-proof, generalized database that connects pre-existing information with new information.

Question Bank

The questionnaire harmonization includes the harmonization of the question or entry form label (In the past 12 months, how many times have you been to a concert) and the response scale (1, 2, Do not remember, Decline to say). The harmonization must be made with other knowledge concepts (i.e. the concept of the concert) and survey questionnaires or annual report information fields.

Translations

We must be able to work with translators and standardized translated labels. We must have the question bank ready by the end of October. Ensure that we do not use URIs but IRIs for identifying questionnaire items. Labesl are translated or localized. A generic ‘French’ label is often unsuitable for French speakers in Belgium.

Fieldwork

We must carry out fieldwork, i.e. surveying music-related problems. We will conduct the fieldwork with a cheap online tool (LimeSurvey or SurveyMonkey). The fieldwork will likely remain fully online or may contain a small, hybrid online interview element. The integration of fieldwork implementation is the least important task for us. Use whatever is convenient.

Code and Save

We must record the information into coded datasets that are saved into files. The success of the output harmonization will depend on the use of harmonized coding (we will use, whenever possible, SDMX code definitions, such as ‘F" = ‘female’) and the use of machine-readable, open, portable file formats. Potential users are small entities, and we will avoid the use of databases and favor the use of knowledge graphs instead.

Output harmonization

We will harmonize the data, which means that we will join the coded answers considering the question labels, the value labels, and various forms of missing information across all languages (i.e., English or German versions of a question and answer options.)

Presentation

We will report the harmonized information using graphic visualizations, tables placed into presentation slides, books or web pages. We should have the templates based on test data ready in January.

Questions?

LinkedIn: Daniel Antal - Digital Music Observatory

Reprex Open Collaboration NLAIC 2022

Wed, 21 Sep 2022 18:00:00 +0200

Reprex is looking for new collaborations within the NLAIC

Questions?

Email | Keybase

LinkedIn: Daniel Antal - Reprex

Reprex

Wed, 21 Sep 2022 16:00:00 +0200

Big Data Creates Inequalities

Only the largest corporations, best-endowed universities, and rich governments can afford data collection and processing capacities that are large enough to harness the advantages of AI.

Fullscreen: F

Next: ️> or Space | Previous :️<
Start: Home | Finish: End
Overview: Esc| Speaker notes: S
Zoom: Alt + Click 🖱️

Big data that works for all

Reprex: No matter how big is the problem or how small is your team, we fill your reports, dashboards, newsletters, books with data and its visualization.

Connected financial and sustainability reporting based on open data

Eviota: We map your material impacts in your value chain and connect it with environmental or social data that is re-used from the public sector.

Data problems: Reprex

Most SMEs, and civil society organizations do not have a data scientist/engineer in their team, maybe not even an IT person or a HR professional to make such a hire.
When these organizations must solve novel problems, like connecting their financial accounts with environmental and social impact data or connecting to automated transaction systems (like in music), they need novel solutions that do not require managing a database within their organization.

Data problems: Eviota

Most SMEs, and civil society organizations do not have a data scientist/engineer in their team, maybe not even an IT person or a HR professional to make such a hire.
To access green bank loans, insurance products, subsidies, or investments, or to keep track of their sustainability goals in line with the Paris Accord or gender equality plan, organizations must connect their accounting system to external environmental data. We connect their accounts with impact estimates from reliable scientific sources.

Data problems (examples)


	The cost of questionnaire-based market research (survey) is increasing exponentially and offers mediocre results without an enormous question bank and harmonization with other surveys.(See 🖱 blogpost)
	Manual data acquisition is an error-prone and boring task for humans that requires many working hours (often not credited in consultancies, law firms, or research institutes.)
	Wrangling spreadsheet tables or word processor documents by people without data knowledge is the 🖱 data Sisyphus.

Our solution: Reprex

We create data ecosystems with the modernization of the EU/OECD/UN-endorsed 'data observatory' concept. Our data observatory 3.0 uses the knowledge graphs of the web of data.
We acquire and process data on a scale in our data observatories. We acquire and process data on a scale in our data observatories. Our approach significantly reduces the cost of data acquisition and opens invisible, reliable governmental and scientific data sources. We are currently building five observatories, and one of them is already mature enough to be considered for official EU recognition (serving the music industry).
We provide applications, for example, our Eviota application, which connects financial accounts with environmental and social data, and crates reliable indicators and benchmarks for the requirements of the sustainable finance package.

Our solution: Eviota (Non-Financial)

We create data ecosystems with the modernization of the EU/OECD/UN-endorsed 'data observatory' concept. Our Green Deal Data Observatory uses the knowledge graphs of the web of data and gives access to reliable, often unseen, hard-to-access ESG data sources.
We acquire and process data on a scale in our data observatories. Our approach significantly reduces the cost of data acquisition and opens invisible, reliable governmental and scientific data sources.
We provide applications, for example, our Eviota application, which connects financial accounts with environmental and social data, and crates reliable indicators and benchmarks for the requirements of the sustainable finance package. Unlike our competitors, we can serve SMEs, too, at a competitive cost.

Our solution: Eviota (For Banks)

We create data ecosystems with the modernization of the EU/OECD/UN-endorsed 'data observatory' concept. Our Green Deal Data Observatory uses the knowledge graphs of the web of data and gives access to reliable, often unseen, hard-to-access ESG data sources.
We acquire and process data on a scale in our data observatories. Our approach significantly reduces the cost of data acquisition and opens invisible, reliable governmental and scientific data sources.
We provide applications, for example, our Eviota application, which connects financial accounts with environmental and social data, and crates reliable indicators and benchmarks for the requirements of the sustainable finance package. We are validating our product in the regulatory sandbox of a central bank to show that we provide a cost-effective solution to many regulatory problems opened by the new [sustainable finance package of the EU](https://finance.ec.europa.eu/publications/sustainable-finance-package_en).

UNIIQ

Objectives, including product roadmap (technology/time/money)
Schematic overview of developments since inception

Market

Competition

Know-how and integration of open source components

Reprex has a special know-how to map and connect private datasets managing the boundaries of organizations that often have conflicting interests. Our know-how was developed over 10 years, and the data of about 60, often conflicting music industry actors in 12 countries.
Our team has many years of experience with working public sector information reuse, or 'open data', and have built reliable open source software to process legally open, not readily downloadable, and very valuable information that is not available for market vendors.
We use RDF (linked open data) and other technologies to link scattered small data to big data; we use our own R libraries to test and process various data into reliable statistical data or indicators.
Based on our unique data access and software we are developing the Eviota App to connect financial accounts and environmental, social and governance data.

Team [rewrite with GDO]


	The two co-founders, 🖱 Daniel Antal, CFA and 🖱 Andrés García Molina, PhD, and the core team manage the ecosystems’ development, develop knowledge management, and direct the software development. 🖱 Team on full screen
	Each observatory has a broader team of users, data and knowledge curators, and developers. The most developed 🖱️ Digital Music Observatory has 16 institutional users and a team of about 20 music and data professionals. The newer observatories have a smaller, initial service development and data curatorial team.

Timeline

Inception: Yes!Delft AI+Blockchain Product Market Fit Validation with the Digital Music Observatory
New observatory development started with computational antitrust, ESG reporting, and
Several, peer reviewed software releases
DMO has more than 20 curators, 3 million euro budget for 3 years, increasing user base.
Eviota and the Green Deal

We are part of rOpenGov and have access to very special knowledge working with national accounts data and ESG data used by governments to keep track with the Paris Accord. We can access data cheaper, faster, better than our competitors.
We have a know-how to manage conflicts of interest and very complex data use rights.

Data observatories 3.0

Reprex is offering shared data ecosystems. Our observatories are great solutions for organizations without a data specialization:


🌳 Organizations that cannot afford to build a large enough data team to sustain consistent, extensive data collection and processing (many large institutions and companies)
🪴 Who cannot hire even a single data engineer or a data scientist (medium-sized companies, NGOs)
🌱 Who do not even have a permanent IT function (about 2 million European small enterprises and civil society organizations)


	The European Union, the World Bank, OECD, and UN have facilitated the creation of more than 80 so-called ‘data observatories’ to help companies, researchers, NGOs, and governments systematically collect data and knowledge.
	We are currently building one prototype for the European Music Observatory financed by the European Union and music industry players (cc 3-4 million euros.) We would like to take over existing or start new observatories in 2 years at least 5)
	Our observatories are competitive, because they use high-quality open source scientific software; they exploit the new Data Governance Act and Open Data Directive, deploy web 3.0 data synchronization, and offer great value-added research products.

Platform products	Value added data applications
The European Union, the World Bank, OECD, and UN have facilitated the creation of more than 80 so-called ‘data observatories’ to help companies, researchers, NGOs, and governments systematically collect data and knowledge.	The different observatories offer different types of knowledge products, such as statistical yearbooks, various apps, and database access.
Most of them use web 1.0 technologies, inefficient knowledge accumulation. Already 20 of them have been discontinued.	We are developing software solutions that exploit our platforms: we harmonize surveys, statistical data, automate research reporting, elements of market monitoring or ESG reporting.
We are currently building one prototype for the European Music Observatory financed by the European Union and music industry players (cc 3-4 million euros.) We would like to take over existing or start new observatories in 2 years at least 5)	Each observatory gives us intimidate customer access to 3-4 large universities, 1-2 large consultancies, and various specialist institutions.

Marketing strategy


	Buma/Stemra like copyright management agencies, music export offices, festivals and venues, University of Amsterdam, Sant’Anna, Economic University of Bratislava, ministries of culture, grant agencies.
	University of Amsterdam, Europeana, Sant’Anna, Hungarian Film Fund
	Connected financial and sustainability reporting: bank consultancies, big four audit companies, large environmental NGOs.
	Antitrust agencies, law firms, economics consultancies working with mergers and other competition related issues.

Target market size


The observatory platforms usually have a build-up cost of about 3-5 million euros and an annual running costs of 0.1-3 million euros.	We hope to gain at least 10% global market share on the observatory platform management market to pay our basic data science team and R&D.
Our existing observatories give us access to the market and public surveying markets (cc € 30-40 bn in the developed nations), particularly to its software component (€ 10 billion euros).	retroharmonize integrates pre-existing questionnaire-based surveys and new surveys. We see interest from the biggest global players.
Our existing observatories gave us access to environmental impact assessment and currently we build an ESG reporting tool with a central bank, a value bank, and a big four company.	Connected ESG reporting has a €4 bn market in the EU alone, and our Eviota product is very competitive. Due to regulatory pressure, we can harvest a decent share if we are able to attract venture capital.

The observatory platforms usually have a build-up cost of about 3-5 million euros and an annual running costs of 0.1-3 million euros.

We hope to gain at least 10% global market share on the observatory platform management market to pay our basic data science team and R&D.

Our existing observatories give us access to the market and public surveying markets (cc € 30-40 bn in the developed nations), particularly to its software component (€ 10 billion euros).

retroharmonize integrates pre-existing questionnaire-based surveys and new surveys. We see interest from the biggest global players.

Our existing observatories gave us access to environmental impact assessment and currently we build an ESG reporting tool with a central bank, a value bank, and a big four company.

Connected ESG reporting has a €4 bn market in the EU alone, and our Eviota product is very competitive. Due to regulatory pressure, we can harvest a decent share if we are able to attract venture capital.

Team


	The two co-founders, 🖱 Daniel Antal, CFA and 🖱 Andrés García Molina, PhD, and the core team manage the ecosystems’ development, develop knowledge management, and direct the software development. 🖱 Team on full screen
	Each observatory has a broader team of users, data and knowledge curators, and developers. The most developed 🖱️ Digital Music Observatory has 16 institutional users and a team of about 20 music and data professionals. The newer observatories have a smaller, initial service development and data curatorial team.

Traction


💻 Our free scientific software products have a steadily growing user base (several thousand users globally.)
📈 We are able to convert this to paying research automation services at a higher growth rate.
🚀 We won four competitive tenders this year, but we feel that the slow tendering/acquisition/cash cycle is hampering our growth, we see far more opportunities that we can serve. Therefore we are looking for investors.

Funding


We have a good track record in EU tenders, but we would like to build up this reputation in the Netherlands, too, mainly for new platforms.
We help our non-profit users, such as cultural heritage organizations, music export offices, collective rights management agencies to get funding to use our platforms and services
Our for profit-users need a more polished, user-friendlier front-end. Some are interested in joint ventures (like exploiting our survey capabilities). Venture capital would be preferred, as demand outstrips growth.

Do it Smarter

They usually do not exchange standard data with statistical agencies, they are not synchronized on knowledge graphs of the Europeana or national libraries, and their research output is usually not to be found on open science repositories.
The Hague is the winner of the World Smart City Award 2021, and we would like to attract the planned European Music Observatory and other, EU/UNESCO recognized institutions into the town building on the innovations of Reprex and the ecosystem of the Hague.

Questions?

Email | Keybase

LinkedIn: Daniel Antal - Reprex

Home - One Pager ImpactCity Startup Support XL

Reprex

Wed, 21 Sep 2022 16:00:00 +0200

Big Data Creates Inequalities

Only the largest corporations, best-endowed universities, and rich governments can afford data collection and processing capacities that are large enough to harness the advantages of AI.

Fullscreen: F

Next: ️> or Space | Previous :️<
Start: Home | Finish: End
Overview: Esc| Speaker notes: S
Zoom: Alt + Click 🖱️

Data problems


	The cost of questionnaire-based market research (survey) is increasing exponentially and offers mediocre results without an enormous question bank and harmonization with other surveys.(See 🖱 blogpost)
	Manual data acquisition is an error-prone and boring task for humans that requires many working hours (often not credited in consultancies, law firms, or research institutes.)
	Wrangling spreadsheet tables or word processor documents by people without data knowledge is the 🖱 data Sisyphus.

Data observatories 3.0

Reprex is offering shared data ecosystems. Our observatories are great solutions for organizations without a data specialization:


🌳 Organizations that cannot afford to build a large enough data team to sustain consistent, extensive data collection and processing (many large institutions and companies)
🪴 Who cannot hire even a single data engineer or a data scientist (medium-sized companies, NGOs)
🌱 Who do not even have a permanent IT function (about 2 million European small enterprises and civil society organizations)


	The European Union, the World Bank, OECD, and UN have facilitated the creation of more than 80 so-called ‘data observatories’ to help companies, researchers, NGOs, and governments systematically collect data and knowledge.
	We are currently building one prototype for the European Music Observatory financed by the European Union and music industry players (cc 3-4 million euros.) We would like to take over existing or start new observatories in 2 years at least 5)
	Our observatories are competitive, because they use high-quality open source scientific software; they exploit the new Data Governance Act and Open Data Directive, deploy web 3.0 data synchronization, and offer great value-added research products.

Platform products	Value added data applications
The European Union, the World Bank, OECD, and UN have facilitated the creation of more than 80 so-called ‘data observatories’ to help companies, researchers, NGOs, and governments systematically collect data and knowledge.	The different observatories offer different types of knowledge products, such as statistical yearbooks, various apps, and database access.
Most of them use web 1.0 technologies, inefficient knowledge accumulation. Already 20 of them have been discontinued.	We are developing software solutions that exploit our platforms: we harmonize surveys, statistical data, automate research reporting, elements of market monitoring or ESG reporting.
We are currently building one prototype for the European Music Observatory financed by the European Union and music industry players (cc 3-4 million euros.) We would like to take over existing or start new observatories in 2 years at least 5)	Each observatory gives us intimidate customer access to 3-4 large universities, 1-2 large consultancies, and various specialist institutions.

Marketing strategy


	Buma/Stemra like copyright management agencies, music export offices, festivals and venues, University of Amsterdam, Sant’Anna, Economic University of Bratislava, ministries of culture, grant agencies.
	University of Amsterdam, Europeana, Sant’Anna, Hungarian Film Fund
	Connected financial and sustainability reporting: bank consultancies, big four audit companies, large environmental NGOs.
	Antitrust agencies, law firms, economics consultancies working with mergers and other competition related issues.

Target market size


The observatory platforms usually have a build-up cost of about 3-5 million euros and an annual running costs of 0.1-3 million euros.	We hope to gain at least 10% global market share on the observatory platform management market to pay our basic data science team and R&D.
Our existing observatories give us access to the market and public surveying markets (cc € 30-40 bn in the developed nations), particularly to its software component (€ 10 billion euros).	retroharmonize integrates pre-existing questionnaire-based surveys and new surveys. We see interest from the biggest global players.
Our existing observatories gave us access to environmental impact assessment and currently we build an ESG reporting tool with a central bank, a value bank, and a big four company.	Connected ESG reporting has a €4 bn market in the EU alone, and our Eviota product is very competitive. Due to regulatory pressure, we can harvest a decent share if we are able to attract venture capital.

The observatory platforms usually have a build-up cost of about 3-5 million euros and an annual running costs of 0.1-3 million euros.

We hope to gain at least 10% global market share on the observatory platform management market to pay our basic data science team and R&D.

Our existing observatories give us access to the market and public surveying markets (cc € 30-40 bn in the developed nations), particularly to its software component (€ 10 billion euros).

retroharmonize integrates pre-existing questionnaire-based surveys and new surveys. We see interest from the biggest global players.

Our existing observatories gave us access to environmental impact assessment and currently we build an ESG reporting tool with a central bank, a value bank, and a big four company.

Team


	The two co-founders, 🖱 Daniel Antal, CFA and 🖱 Andrés García Molina, PhD, and the core team manage the ecosystems’ development, develop knowledge management, and direct the software development. 🖱 Team on full screen
	Each observatory has a broader team of users, data and knowledge curators, and developers. The most developed 🖱️ Digital Music Observatory has 16 institutional users and a team of about 20 music and data professionals. The newer observatories have a smaller, initial service development and data curatorial team.

Traction


💻 Our free scientific software products have a steadily growing user base (several thousand users globally.)
📈 We are able to convert this to paying research automation services at a higher growth rate.
🚀 We won four competitive tenders this year, but we feel that the slow tendering/acquisition/cash cycle is hampering our growth, we see far more opportunities that we can serve. Therefore we are looking for investors.

Funding


We have a good track record in EU tenders, but we would like to build up this reputation in the Netherlands, too, mainly for new platforms.
We help our non-profit users, such as cultural heritage organizations, music export offices, collective rights management agencies to get funding to use our platforms and services
Our for profit-users need a more polished, user-friendlier front-end. Some are interested in joint ventures (like exploiting our survey capabilities). Venture capital would be preferred, as demand outstrips growth.

Do it Smarter

They usually do not exchange standard data with statistical agencies, they are not synchronized on knowledge graphs of the Europeana or national libraries, and their research output is usually not to be found on open science repositories.
The Hague is the winner of the World Smart City Award 2021, and we would like to attract the planned European Music Observatory and other, EU/UNESCO recognized institutions into the town building on the innovations of Reprex and the ecosystem of the Hague.

Questions?

Email | Keybase

LinkedIn: Daniel Antal - Reprex

Home - One Pager ImpactCity Startup Support XL

Reprex Nominated for The Hague Innovators Award

Tue, 13 Sep 2022 08:12:00 +0200

Reprex is a finalist for The Hague Innovators Award 2022, and the prize of the audience, in the startup category with our respectable competitors, Sibö, WECO, STHRIVE and ECOBLOQ.

The transition towards a sustainable and inclusive economy depends on collaboration. That is why we are bringing together startups, scale-ups, investors, policymakers, and other impact makers from around the world in The Hague for the 7th edition of ImpactFest.

With the The Hague Innovators Challenge, the municipality of The Hague challenges startups, scale-ups, and students to present their innovative ideas for global issues, as described in the UN Sustainable Development Goals (SDGs).

Hague Innovators Award 2022

Wed, 17 Aug 2022 12:00:00 +0000

Data Observatory 3.0

Controls

Next: Right Arrow or Space
Previous: Left Arrow
Start: Home
Finish: End
Overview: Esc
Speaker notes: S
Fullscreen: F
Zoom: Alt + Click
PDF Export: E

Shared evidence ecosystems: data observatories

Organizations that cannot afford to build a large enough data team to sustain consistent, extensive data collection and processing (many large institutions and companies)
Who cannot hire even a single data engineer or a data scientist
Who do not even have a permanent IT function (about 2 million European small enterprises and civil organizations)

What are data observatories?

There are more than 60 functional, and about 20 already discontinued data observatories, i.e. long-term, usually triangular (business, academic, policy) data collection institutions recognized by the EU, OECD or UNESCO, including the European Observatory on Infringements of Intellectual Property Rights of the EU or the European Audiovisual Observatory of the Council of Europe.

Do it Smarter

They usually do not exchange standard data with statistical agencies, they are not synchronized on knowledge graphs of the Europeana or national libraries, and their research output is usually not to be found on open science repositories.
The Hague is the winner of the World Smart City Award 2021, and we would like to attract the planned European Music Observatory and other, EU/UNESCO recognized institutions into the town building on the innovations of Reprex and the ecosystem of the Hague.

Strategic objectives

Develop our data observatories as Open Scholarly Infrastructure
Place our Digital Music Observtory, Cultural and Creative Data Observatory, and Green Deal Data Observaotry on knowledge graphs of Europeana, Wikidata, and other open knowledge sytems
Harmonize research artefacts with open repositories such as Zenodo and Figshare.
Achieve EU/UNESCO/OECD recognition for our self-governing, triangular, science-policy-busines triangular data ecosystems as data observatories

Digital Music Observatory

Listen Local in Horizon Europe OpenMuse WP Diversity, Creative Europe MusicAIRE: connected and curated data on 10,000s of music works
Our aim is to describe the entire, currently legally available music repertoire of Slovakia and Lithuania at first, and a large part of Ukraine.
Connected with name authorities, web services.

Possible Collaboration

Connect national collective management organization, national library, and various services (Spotify, YouTube) to make the national repertoire more visible
Create use statistics for cultural diversity policies and monitoring local content regulations
Provide best practice example and open source tools for replication

Creative and Cultural Sectors Industries Data Observatory

Possible Collaboration

The CCSI Data Observatory already has some data assets on Zenodo, and we can upgrade its API (both as Rest API with datacube and with a simple RDF serialization)
Create use statistics for cultural heritage objects and other cultural heritage policy data
Revisit some modest deliverables of RECREO and seek new funding.

Green Deal Data Observatory

Possible Collaboration

The Green Deal Data Observatory is currently developed to provide free or very accessible environmental, social and governance reporting tools to the cultural sector.
It could also be used to provide ecological context to cultural heritage objects (CHO) for greater awareness.

Technical Features

Reprex | Documentation

FAIR

FAIR metadata: Dublin Core & DataCite referential metadata
Integration to FigShare and Zenodo for automated releases and publications

Web 3.0

supported with optional, open source APIs to retrieve the data
supported with RDF serialization

Dissemination Support

support automated publishing and releasing of data, visualizations, newsletters, and long-form documentation in auto-refreshing websites, blogposts, or articles, or even books.
develop an ecosystem of open source software that helps the professional collection, processing, documentation of data conforming the Data Governance Act, and supporting data sharing and data altruism.

Research Automation

Research automation

support research automation

Questions?

LinkedIn: Daniel Antal - Digital Music Observatory

Slides

Wed, 17 Aug 2022 12:00:00 +0000

Big Data Creates Inequalities

Only the largest corporations, best-endowed universities, and rich governments can afford data collection and processing capacities that are large enough to harness the advantages of AI.

Fullscreen: F

Next: > or Space | Previous: <
Start: Home | Finish: End
Overview: Esc| Speaker notes: S
Zoom: Alt + Click

Data problems


	The cost of questionnaire-based market research (survey) is increasing exponentially and offers mediocre results without an enormous question bank and harmonization with other surveys.(See 🖱 blogpost)
	Manual data acquisition is an error-prone and boring task for humans that requires many working hours (often not credited in consultancies, law firms, or research institutes.)
	Wrangling spreadsheet tables or word processor documents by people without data knowledge is the 🖱 data Sisyphus.

Data observatories 3.0

Reprex is offering shared data ecosystems. Our observatories are great solutions for organizations without a data specialization:


🌳 Organizations that cannot afford to build a large enough data team to sustain consistent, extensive data collection and processing (many large institutions and companies)
🪴 Who cannot hire even a single data engineer or a data scientist (medium-sized companies, NGOs)
🌱 Who do not even have a permanent IT function (about 2 million European small enterprises and civil society organizations)


	The European Union, the World Bank, OECD, and UN have facilitated the creation of more than 80 so-called ‘data observatories’ to help companies, researchers, NGOs, and governments systematically collect data and knowledge.
	We are currently building one prototype for the European Music Observatory financed by the European Union and music industry players (cc 3-4 million euros.) We would like to take over existing or start new observatories in 2 years at least 5)
	Our observatories are competitive, because they use high-quality open source scientific software; they exploit the new Data Governance Act and Open Data Directive, deploy web 3.0 data synchronization, and offer great value-added research products.

Platform products	Value added data applications
The European Union, the World Bank, OECD, and UN have facilitated the creation of more than 80 so-called ‘data observatories’ to help companies, researchers, NGOs, and governments systematically collect data and knowledge.	The different observatories offer different types of knowledge products, such as statistical yearbooks, various apps, and database access.
Most of them use web 1.0 technologies, inefficient knowledge accumulation. Already 20 of them have been discontinued.	We are developing software solutions that exploit our platforms: we harmonize surveys, statistical data, automate research reporting, elements of market monitoring or ESG reporting.
We are currently building one prototype for the European Music Observatory financed by the European Union and music industry players (cc 3-4 million euros.) We would like to take over existing or start new observatories in 2 years at least 5)	Each observatory gives us intimidate customer access to 3-4 large universities, 1-2 large consultancies, and various specialist institutions.

Marketing strategy


	Buma/Stemra like copyright management agencies, music export offices, festivals and venues, University of Amsterdam, Sant’Anna, Economic University of Bratislava, ministries of culture, grant agencies.
	University of Amsterdam, Europeana, Sant’Anna, Hungarian Film Fund
	Connected financial and sustainability reporting: bank consultancies, big four audit companies, large environmental NGOs.
	Antitrust agencies, law firms, economics consultancies working with mergers and other competition related issues.

Target market size


The observatory platforms usually have a build-up cost of about 3-5 million euros and an annual running costs of 0.1-3 million euros.	Some of our basic products are included in the platform service.
Our existing observatories give us access to the market and public surveying markets (cc 30-40 billion euros in the developed nations), particularly to its software component (about 10 billion euros).
retroharmonize integrates pre-existing questionnaire-based surveys and new surveys. It is aimed at large, international survey companies (Kantar, Gfk) and large international survey programs (Eurostat, GESIS).
regions improves the granularity of existing market research with ‘small area statistics’.

Our existing observatories gave us access to environmental impact assessment and currently we build an ESG reporting tool with a central bank, a value bank, and a big four company.
We hope to gain at least 10% global market share on the observatory platform management market to pay our basic data science team and R&D.

Team


	The two co-founders, 🖱 Daniel Antal, CFA and 🖱 Andrés García Molina, PhD, and the core team manage the ecosystems’ development, develop knowledge management, and direct the software development. 🖱 Team on full screen
	Each observatory has a broader team of users, data and knowledge curators, and developers. The most developed 🖱️ Digital Music Observatory has 16 institutional users and a team of about 20 music and data professionals. The newer observatories have a smaller, initial service development and data curatorial team.

Traction

We have a good, organic growth, but we mainly work with very large private and public entities with complicated, lengthy and not startup-friendly procurement with a cash-conversion cycle of up

Funding

We have a good track record in EU tenders, but we would like to build up this reputation in the Netherlands, too, mainly for new platforms.
We help our non-profit users, such as cultural heritage organizations, music export offices, collective rights management agencies to get funding to use our platforms and services. We have a track record in the EU, Slovakia, Lithuania, but not in the Netherlands.
Our for profit-users need a more polished, user-friendlier front-end. Some are interested in joint ventures (like exploiting our survey capabilities). Venture capital would be preferred, as demand outstrips growth.

«««< HEAD

Do it Smarter

They usually do not exchange standard data with statistical agencies, they are not synchronized on knowledge graphs of the Europeana or national libraries, and their research output is usually not to be found on open science repositories.
The Hague is the winner of the World Smart City Award 2021, and we would like to attract the planned European Music Observatory and other, EU/UNESCO recognized institutions into the town building on the innovations of Reprex and the ecosystem of the Hague.

Questions?

Email | Keybase

LinkedIn: Daniel Antal - Reprex

Developing a software-as-service solution for micro-, and small enterprises

Thu, 09 Jun 2022 09:40:00 +0100

The music sector must increase its environmental and social (ESG) sustainability management to meet the challenges of the climate emergency and to make the music sector a fairer, more just workplace for womxn and artists coming from minorities, small countries. The European Union will make target setting and audited reporting mandatory in environmental and social sustainability for large companies. The application of these new accounting, reporting and disclosure rules are optional for the music sector where almost all entities are micro-, or small enterprises and civil society organizations.

Even if music organizations are not pushed by regulators to adopt these new standards, it is in their best interest to take the initiative on the principle of subsidiarty, and develop tools that can be applied as an extension to their simplified financial and tax reporting. Music organizations and businesses that can prove that they are making progress in reducing their carbon footprint, making their water use more sustainable, and they provide equal opportunities for womxn, they will be eligible for new, green bank and insurance products (which are particularly important in live music) and can attract new sponsors and donors.

Compliance with these new rules is very costly, because tools are being developed for stock-exchange listed big companies and financial institutions. The Commission’s impact assessment (SWD/2021/150 final) estimates the cost of compliance with the Corporate Social Responsibility Directive exceeding 4 bn euros for the European companies or around 10,000 euros per company. Reprex, working together with large accounting, audit and value-based banking partners, scientific, research and industry partners in the Digital Music Observatory open knowledge collaboration, hopes to bring down this cost below 500 euros, which will immediately pay off when a music organization receives green money.

We are working on a simple interface that can connect the accounting system of micro and small enterprises with new methodologies, starting with greenhouse gas reporting with Reprex’s open source EEIO application iotables. We will keep many aspects of our software and data solution open, so that later methodological innovations and scientific achievements can be easily incorporated into the system. Reprex’s minimum viable product will be created in four iteration rounds in Malta, Czechia, Bulgaria and Belgium. However, our testing is open for any amount of donations to any music entities in the European Union who can provide input data in English or Dutch, or be able to pay for their translation and localization costs.

Link: Final List of Awareded Projects by MusicAIRE

Trustworthy Autonomous Recommender Systems on Music Streaming Platforms

Mon, 28 Feb 2022 19:00:00 +0100

Currently almost 60% of the global recording industry sales are made via streaming platforms. Given the enormity of choice on these platforms, and that music listening is a low-key, routine consumption choice, consumers are more and more relying on the recommendations of autonomous recommendation systems. Streaming platforms are two-sided markets, where recommendations are deployed to enhance the user experience on the consumer side, but they also decide the fate of the investments that composers, lyricists, producers, and performers made into the music. We are going to contribute to a research on how such systems may lead to potentially tilted competition field between the content providers, and more specifically, between major labels and independents.

Reprex maintains the Digital Music Observatory and the Listen Local system for granular microdata about music use in small territories (i.e., on small country or sub-national level.) We will provide data/expertise in music streaming and recommendation systems and links to many relevant stakeholders with our considerable experience running experiments on music platforms.

A research team of the University of East Anglia (UEA) the University of Liverpool (UoL), The University of London (City), and King’s College (KCL), supported by the Competition Market Authority of the United Kingdom and Reprex won a prestigious research grant to understand how recommender systems on music streaming platforms can employ trustworthy AI.

The researchers will explore the relationship between the autonomous recommendation systems and entry barriers via simulation. Working closely with Reprex, they will simulate sets of users, and iteratively generate recommendation lists, which the simulated users will react to by deciding how long to engage for and which recommendations to listen to. Through their engagement their user profiles will be updated based on what they listen to which will feed into future recommendations.

See our Feasibility Study for Listen Local.

The empirical experiments of the project want to explore how autonomous recommendation systems are driving consumer choice in a real-life setting, and to establish causality between the recommendation systems and the barrier to entry. As part of the second work package, the researchers will conduct randomised trials by inviting participants to stream music through our own user interface. Reprex has extensive experience conducting similar experiments in the music domain (for various online, field experiments, and high-quality surveys.)

Link: Eight new TAS research projects announced

Reproducible Economic Impact Assessment

Sun, 19 Dec 2021 13:00:00 +0100

Get started with iotables.

We made an important, peer-reviewed release of iotables in the last week as a preparation to increase the functionality of our open-source software. The official release of the iotables R package currently works with economic impact assessments, and can evaluate the likely employment, tax, wage, or gross value added direct, indirect and multiplied impacts of various policy changes in about 30 countries.

Originally the package was developed to calculate the economic impact of the Hungarian film tax shelter and the impact of the music sector on the Slovak economy. (See Slovak Music Industry Report).

The new CRAN release improved the documentation of the function and removed most outdated dependencies. The new, development version (which did not go through peer-review yet) is adding new functionality for environmental impact analysis with the following pollutants: Carbon dioxide without emissions from biomass (CO2), Carbon dioxide from biomass (Biomass CO2), Nitroux oxide (N2O), Methane (CH4), Perfluorocarbons (PFCs), Hydrofluorocarbons (HFCs), Sulphur hexafluoride (SF6) including nitrogen trifluoride (NF3), Nitrogen oxides (NOx), Non-methane volatile organic compounds, (NMVOC), Carbon monoxide (CO), Particulate matter < 10μm (PM10), Particulate matter < 2,5μm (PM2,5), Sulphur dioxide (SO2), Ammonia (NH3) and their combinations (see Reference Metadata in Euro SDMX Metadata Structure (ESMS)).

Our aim is to develop new sustainable finance applications, and understand the sustainability impacts of bank’s lending activities and insurer’s underwriting activities on climate change mitigation and adoption, biodiversity, preservation of water reservers, preventing pollution, and promoting the circular economy.

EU Taxonomy on Sustainable Activities

The European Commission created an created an EU Taxonomy Compass, which provides a visual representation of the contents of the EU Taxonomy, starting with the Delegated Act on the climate objectives, as adopted on 4 June 2021. Whilst you can download the EU Taxonomy in xlsx or json format, they are not tidy datasets, and they are not particularly well-suited for calculations, filtering, or inclusion in applications.

Reprex created a tidy version of the EU Taxonomy for developing better sustainability indicators into the Green Deal Data Observatory.

Open Data

EU Taxonomy on Sustainable Activities (Tidy) download.

Using our iotables is not for the faint heart. It is a scientific software, and it requires a good command of national accounts, input-output economics and sustainability to work with. Our Green Deal Data Observaotry is designed to be an API of scientific software, and produce clean, ready to use data for researchers, policy-makers and business planners who do not have the skills to work with scientific software. We are planning to release well-designed datasets that go through dozens of checks to make sure they have the best data quality.

Do you want to develop input-output models for any European country to measure the direct and indirect green house gas impacts of policy actions? Do you need well-formatted data on interindustry linkages or other relevant topics for sustainable economy or susitainable finance research? Get in touch with us – we are happy to help and test our new software tool with data you need, and create high-quality, open datasets that are ready to use.

Jumping Ahead With the Digital Music Observatory

Thu, 02 Dec 2021 13:00:00 +0100

Our Digital Music Observatory project spent a year in the JUMP Music Market Accelerator’s program. Over the course of 9 months, co-founder Daniel Antal could meet many stakeholders from almost all European countries, meet other new music technology startups and projects, and got mentoring and other professional help to further develop the project.

The Digital Music Observatory is one of the several initiatives to fill the data gaps of the fragmented European music ecosystems. While most of Europe’s music is available and promoted on data-heavy, AI-driven autonomous platforms like TikTok, Spotify, YouTube, Deezer, music labels, publishers, national export offices are lacking the necessary data solutions to remain competitive.

Daniel is pitching for partnership with Music Tech Europe on Linechech and finding a music city that wants to be the seat of the future European Music Observatory. Photo: Wen Liu.

One of the recurring themes of 2021 was the notion that the music streaming economy is broken. Several JUMP fellows are working on various projects that aim to fix this, and our Digital Music Observatory has both the data and track record to provide evidence and test ideas about possible solutions – change in pricing, better targeting in export and domestic markets, and checking for algorithmic biases. See what we have done in the field this year in the UK IPO-initiated Music Creators’ Earning project; understanding algorithmic recommendation problems with the support of the Slovak Arts Council, and making recommendations about better music metadata and copyright regulation with our research consortium.

The other very interesting theme of the year was the emergence of new, immersive music tech companies. We hope that our Digital Music Observatory can grow into a hub for their data needs, too. How is the world of 2.7 billion gamer and music lovers is forming a new market for Ristband? We would also like to curate data about the healing effects of sound, and work in the future with immersive, functional music providers like Flower of Sound who place music and sound design into a less stressful, more healthy acoustic environment.

We were often criticized for placing too little emphasis on data visualization. Our next priority is to provide clear, beautiful infographs and charts to all of our datasets.

There were many professionals who helped us in the JUMP program. We are particularly thankful for Alessanra di Caro (partnership building), Elodie Crouzet (program coordination), Steve Farris (mentoring), Veronique Friedrich (team building), Thierry Giesler (improving our pitch) and Anna Zò (Music Tech Europe).

Are you a data user? Give us some feedback! Shall we do some further automatic data enhancements with our datasets? Document with different metadata? Link more information for business, policy, or academic use? Please ive us any feedback!

How We Add Value to Public Data With Better Curation And Documentation?

Mon, 08 Nov 2021 09:00:00 +0000

In this example, we show a simple indicator: the Turnover in Radio Broadcasting Enterprises in many European countries. This is an important demand driver in the Music economy pillar of our Digital Music Observatory, and important indicator in our more general Cultural & Creative Sectors and Industries Observatory. Of course, if you work with competition policy or antitrust, than any industry may be interesting to you–but not all of them are well-serverd with data.

This dataset comes from a public datasource, the data warehouse of the European statistical agency, Eurostat. Yet it is not trivial to use: unless you are familiar with national accounts, you will not find this dataset on the Eurostat website.

The data can be retrieved from the Annual detailed enterprise statistics for services NACE Rev.2 H-N and S95 Eurostat folder.

Our version of this statistical indicator is documented following the FAIR principles: our data assets are findable, accessible, interoperable, and reusable. While the Eurostat data warehouse partly fulfills these important data quality expectations, we can improve them significantly. And we can also improve the dataset, too, as we will show in the next blogpost.

Table of Contents

Findable Data

Our data observatories add value by curating the data–we bring this indicator to light with a more descriptive name, and we place it in a domain-specific context with our Digital Music Observatory and Cultural & Creative Sectors and Industries Observatory and a policy-specific context with our Competition Data Observatory and Green Deal Data Observatory. While many people may need this dataset in the creative sectors, or among cultural policy designers, most of them have no training in working with national accounts, which imply decyphering national account data codes in records that measure economic activity at a national level. Our curated data observatories bring together many available data around important domains. Our Digital Music Observatory, for example, aims to form an ecosystem of music data users and producers.

We added descriptive metadata that help you find our data and match it with other relevant data sources.

We added descriptive metadata that help you find our data and match it with other relevant data sources. For example, we add keywords and standardized metadata identifiers from the Library of Congress Linked Data Services, probably the world’s largest standardized knowledge library description. This ensures that you can find relevant data around the same key term ("Radio broadcasting") in addition to our turnover data. This allows connecting our dataset unambiguously with other information sources that use the same concept, but may be listed under different keywords, such as Radio–Broadcasting, or Radio industry and trade, or maybe Hörfunkveranstalter in German, or Emitiranje radijskog programa in Croatian or Actividades de radiodifusão in Portugese.

Accessible Data

Our data is accessible in two forms: in csv tabular format (which can be read with Excel, OpenOffice, Numbers, SPSS and many similar spreadsheet or statistical applications) and in JSON for automated importing into your databases. We can also provide our users with SQLite databases, which are fully functional, single user relational databases.

Tidy datasets are easy to manipulate, model and visualize, and have a specific structure: each variable is a column, each observation is a row, and each type of observational unit is a table. This makes the data easier to clean, and far more easier to use in a much wider range of applications than the original data we used. In theory, this is a simple objective, yet we find that even governmental statistical agencies–and even scientific publications–often publish untidy data. This poses a significant problem that implies productivity loses: tidying data will require long hours of investment, and if a reproducible workflow is not used, data integrity can also be compromised: chances are that the process of tidying will overwrite, delete, or omit a data or a label.

Tidy datasets are easy to manipulate, model and visualize, and have a specific structure: each variable is a column, each observation is a row, and each type of observational unit is a table.

While the original data source, the Eurostat data warehouse is accessible, too, we added value with bringing the data into a tidy format. Tidy data can immediately be imported into a statistical application like SPSS or STATA, or into your own database. It is immediately available for plotting in Excel, OpenOffice or Numbers.

Interoperability

Our data can be easily imported with, or joined with data from other internal or external sources.

All our indicators come with standardized descriptive metadata, and statistical (processing) metadata. See our API

All our indicators come with standardized descriptive metadata, following two important standards, the Dublin Core and DataCite–implementing not only the mandatory, but the recommended descriptions, too. This will make it far easier to connect the data with other data sources, e.g. turnover with the number of radio broadcasting enterprises or radio stations within specific territories.

Our passion for documentation standards and best practices goes much further: our data uses Statistical Data and Metadata eXchange standardized codebooks, unit descriptions and other statistical and administrative metadata.

Reuse

All our datasets come with standardized information about reusabililty. We add citation, attribution data, and licensing terms. Most of our datasets can be used without commercial restriction after acknowledging the source, but we sometimes work with less permissible data licenses.

In the case presented here, we added further value to encourage re-use. In addition to tidying, we significantly increased the usability of public data by handling missing cases. This is the subject of our next blogpost.

Are you a data user? How could we serve you better?

Shall we do some further automatic data enhancements with our datasets? Document with different metadata? Link more information for business, policy, or academic use? Please get in touch with us!

How We Add Value to Public Data With Imputation and Forecasting

Mon, 08 Nov 2021 10:00:00 +0100

Public data sources are often plagued by missng values. Naively you may think that you can ignore them, but think twice: in most cases, missing data in a table is not missing information, but rather malformatted information. This approach of ignoring or dropping missing values will not be feasible or robust when you want to make a beautiful visualization, or use data in a business forecasting model, a machine learning (AI) applicaton, or a more complex scientific model. All of the above require complete datasets, and naively discarding missing data points amounts to an excessive waste of information. In this example we are continuing the example a not-so-easy to find public dataset.

In the previous blogpost we explained how we added value by documenting data following the FAIR principle and with the professional curatorial work of placing the data in context, and linking it to other information sources, such as other datasets, books, and publications, regardless of their natural language (i.e., whether these sources are described in English, German, Portugese or Croatian). Photo: Jack Sloop.

Completing missing datapoints requires statistical production information (why might the data be missing?) and data science knowhow (how to impute the missing value.) If you do not have a good statistician or data scientist in your team, you will need high-quality, complete datasets. This is what our automated data observatories provide.

Table of Contents

Why is data missing?

International organizations offer many statistical products, but usually they are on an ‘as-is’ basis. For example, Eurostat is the world’s premiere statistical agency, but it has no right to overrule whatever data the member states of the European Union, and some other cooperating European countries give to them. And they cannot force these countries to hand over data if they fail to do so. As a result, there will be many data points that are missing, and often data points that have wrong (obsolete) descriptions or geographical dimensions. We will show the geographical aspect of the problem in a separate blogpost; for now, we only focus on missing data.

Some countries have only recently started providing data to the Eurostat umbrella organization, and it is likely that you will find few datapoints for North Macedonia or Bosnia-Herzegovina. Other countries provide data with some delay, and the last one or two years are missing. And there are gaps in some countries’ data, too.

See the authoritative copy of the dataset.

This is a headache if you want to use the data in some machine learning application or in a multiple or panel regression model. You can, of course, discard countries or years where you do not have full data coverage, but this approach usually wastes too much information–if you work with 12 years, and only one data point is available, you would be discarding an entire country’s 11-years’ worth of data. Another option is to estimate the values, or otherwise impute the missing data, when this is possible with reasonable precision. This is where things get tricky, and you will likely need a statistician or a data scientist onboard.

What can we improve?

Consider that the data is only missing from one year for a particular country, 2015. The naive solution would be to omit 2015 or the country at hand from the dataset. This is pretty destructive, because we know a lot about the radio market turnover in this country and in this year! But leaving 2015 blank will not look good on a chart, and will make your machine learning application or your regression model stop.

A statistician or a radio market expert will tell you that you know more-or-less the missing information: the total turnover was certainly not zero in that year. With some statistical or radio domain-specific knowledge you will use the 2014, or 2016 value, or a combination of the two and keep the country and year in the dataset.

Our improved dataset added backcasted (using the best time series model fitting the country’s actually present data), forecasted (again, using the best time series model), and approximated data (using linear approximation.) In a few cases, we add the last or next known value. To give a few quantiative indicators about our work:

Increased number of observations: 65%
Reduced missing values: -48.1%
Increased non-missing subset for regression or AI: +66.67%

If your organization is working with panel (longitudional multiple) regressions or various machine learning applications, then your team knows that not havint the +66.67% gain would be a deal-breaker in the choice of models and punctuality of estimates or KPIs or other quantiative products. And that they would spent about 90% of their data resources on achieving this +66.67% gain in usability.

If you happen to work in an NGO, a business unit or a research institute that does not employ data scientists, then it is likely that you can never achieve this improvement, and you have to give up on a number of quantitative tools or visualizations. If you have a data scientist onboard, that professional can use our work as a starting point.

Can you trust our data?

We believe that you can trust our data better than the original public source. We use statistical expertise to find out why data may be missing. Often, it is present in a wrong location (for example, the name of a region changed.)

If you are reluctant to use estimates, think about discarding known actual data from your forecast or visualization, because one data point is missing. How do you provide more accurate information? By hiding known actual data, because one point is missing, or by using all known data and an estimate?

Our codebooks and our API uses the Statistical Data and Metadata eXchange documentation standards to clearly indicate which data is observed, which is missing, which is estimated, and of course, also how it is estimated. This example highlights another important aspect of data trustworthiness. If you have a better idea, you can replace them with a better estimate.

Our indicators come with standardized codebooks that do not only contain the descriptive metadata, but administrative metadata about the history of the indicator values. You will find very important information about the statistical method we used the fill in the data gaps, and even link the reliable, the peer-reviewed scientific, statistical software that made the calculations. For data scientists, we record the plenty of information about the computing environment, too-–this can come handy if your estimates need external authentication, or you suspect a bug.

Avoid the data Sisyphus

If you work in an academic institution, in an NGO or a consultancy, you can never be sure who downloaded the Annual detailed enterprise statistics for services (NACE Rev. 2 H-N and S95) Eurostat folder from Eurostat. Did they modify the dataset? Did they already make corrections with the missing data? What method did they use? To prevent many potential problems, you will likely download it again, and again, and again…

See our The Data Sisyphus blogpost.

We have a better solution. You can always rely on our API to import directly the latest, best data, but if you want to be sure, you can use our regular backups on Zenodo. Zenodo is an open science repository managed by CERN and supported by the European Union. On Zenodo, you can find an authoritative copy of our indicator (and its previous versions) with a digital object identifier, in this case, 10.5281/zenodo.5652118. These datasets will be preserved for decades, and nobody can manipulate them. You cannot accidentally overwrite them, and we have no backdoor access to modify them.

Get the data

How can we do better?

Are you a data user?

Shall we do some further automatic data enhancements with our datasets? Document with different metadata? Link more information for business, policy, or academic use? Please get in touch with us!

Reprex Joins RECREO Research Consortium To Develop Innovation Indicators

Sat, 06 Nov 2021 16:00:00 +0000

The Scuola Superiore di Studi Universitari e di Perfezionamento Sant’Anna and Università degli Studi di Trento (Italy); University of Glasgow (United Kingdom); Universiteit van Amsterdam and Stichting Europeana from the Netherlands; the National University of Ireland Maynooth (Ireland); Tartu Ulikool (Estonia); Szegedi Tudományegyetem (Hungary); Fundacion Santa Maria La Real del Patrimonio Historico from Spain; the Katholieke Universiteit Leuven, (Belgium); Culture Action Europe AISBL and IDEA Strategische Economische Consulting (Belgium) and Reprex created theREshaping CCSI REsearch: Open data, policy analysis and methods for evidence-based decision-making consortium consortium, which will mainly develop new policy evidence in the field of innovation and inclusiveness for the creative and cultural sectors, industries. The Consortium applies for a Horizon Europe grant with the HORIZON-CL2-2021-HERITAGE-01-03 Cultural and creative industries as a driver of innovation and competitiveness call of the European Commission.

Policymakers face challenges when trying to implement a strict evidence-based approach to decision-making in the field of cultural and creative sectors and industries (CCSI). This is mostly due to four phenomena:

Evidence dissonances in mapping, measuring and analysis of key indicators, which lead to improper generalizations and gaps in decisionmakers’ knowledge and stakeholders’ awareness
Fragmentation of hubs of production and concentration of platforms, which create statistical biases and have features that hardly fit with traditional impact assessment methods;
Datafication, which is revolutionizing CCSI but remains difficult to investigate, thus broadening knowledge gaps; and
Stakeholders’ fragmentation and conflicting interests, which hinders their engagement, awareness-raising and uptake of policy inputs.With its cross-disciplinary consortium of academics, practitioners and a strong network of stakeholders, engaged via participatory research strategies, RECREO will help policymakers and stakeholders tackling such challenges, by generating new knowledge and methods to fill in knowledge and awareness gaps. RECREO will achieve this goal through four actions.

First, it will generate a wide array of horizontal and sector-specific datasets, made openly accessible via the CCSI Data Observatory and the Evidence Synthesis Platform. Second, it will offer an unprecedented EU and comparative mapping and impact assessment of key regulatory and policy measures relevant for CCSI, made available on the Law and Policy Observatory. Third, it will develop innovative methods to measure and assess CCSI innovation, competitiveness and spill-over effects, emphasizing inclusiveness, diversity and sustainability. Last, it will offer policy recommendations and best practices aimed at supporting the sustainable growth and competitiveness of culturally diverse CCSI, and their cross-fertilization with cultural heritage promotion and preservation.

Reprex on MaMA

Fri, 15 Oct 2021 19:00:00 +0000

Reprex’s co-founder, the main developer of the Digital Music Observatory, Daniel Antal and Digital Music Observatory curator, Marie Zhorová participated in the MaMA Festival & Convention in Paris on 13-15 October within the JUMP Music Market Accelerator Program Program. We introduced our Digital Music Observatory to national music organizations and encouraged them to try out a cooperation with us. (See Use Cases below)

Our main aim was to find new users to our Digital Music Observatory, and to find partners for a future Horizon Europe R&D project to develop the scientific pillars of the Observatory in a manner that meets practical industry needs and the feature requirements laid out in hte Feasiblity Study for a Euroepan Music Observatory.

Our concept was introduced in Le Trianon to a wider audience during the JUMP Music Market Accelerator Pitch Session and in one-to-one meetings to representatives of French national organizations. We have also started to investigate the possibility to cooperate with two startups to bring our data services closer to artists, labels, and publishers.

Use Cases

Fair Streaming

Daniel introduced our work made for the UK IPO’s Music Creators’ Earnings in the Digital Era Project about the justified and not-justified differences among music rightsholders earnings and the diminishing market value of streams. We believe that our UK approach is a particularly interesting addition to join with the distribution analysis performed by the Centre Nationale de la Musique and Deloitte in France.

Fair Value

Daniel introduced to collective management professioanls our innovative approach for private copying valuation, royalty price setting, estimating the values of value transfer to media platforms, and other topics of interests for collective management and rights management organizations. Our approach has a proven track record to increase revenues for creators.

Open Music Observatory

We introduced our approach to building the European Music Observatory in a decentralized way, relying not only on the resources of Creative Europe but also on Open Science, Horizon Europe, bringing the music industry, music research in universities and cultural policy under one open collaboration. Because France is building its own music observatory of a kind, the decentralized approach could particularly benefit French stakeholders.

Listen Local

Why Data Observatory?

Our use cases highlight the value of having a wide range of data available for the industry players, researchers and policy-makers. In the era of big data, and when open data is becoming legally more and more available, it is important to have one place with a single data collection method. Copernicus built a permanent observatory for the ongoing observation of celestial bodies. We built an automated data observatory to permanently collect data about music.

CCSI Data Observatory

Wed, 06 Oct 2021 16:00:00 +0000

The creative and cultural sectors and industries are mainly made of networks of freelancers and microenterprises, with very few medium-sized companies. Their economic performance, problems, and innovation capacities hidden. Our open collaboration to create this data observatory is committed to change this. Relying on modern data science, the re-use of open governmental data, open science data, and novel harmonized data collection we aim to fill in the gaps left in the official statistics of the European Union.

We believe that introducing Open Policy Analysis standards with open data, open-source software and research automation can help better understanding how creative people and their enterprises and institutions add value to the European economy, how they create jobs, innovate, and increase the well-being of a diverse European society. Our collaboration is open for individuals, citizens scientists.

The new observatory can be reached on ccsi.dataobservatory.eu and will be institutionally hosted by IViR, the Institute for Information Law of the University of Amsterdam, where Reprex’s co-founder, Daniel Antal will coordinate the development of this new, open scientific tool. Reprex will continue to develop the working model of the data observatory and continue to build open source software tools within the rOpenGov community and the R-Universe initative of ROpenSci.

The Scuola Superiore di Studi Universitari e di Perfezionamento Sant’Anna and Università degli Studi di Trento (Italy); University of Glasgow (United Kingdom); Universiteit van Amsterdam and Stichting Europeana from the Netherlands; the National University of Ireland Maynooth (Ireland); Tartu Ulikool (Estonia); Szegedi Tudományegyetem (Hungary); Fundacion Santa Maria La Real del Patrimonio Historico from Spain; the Katholieke Universiteit Leuven, (Belgium); Culture Action Europe AISBL and IDEA Strategische Economische Consulting (Belgium) and Reprex created the the RECREO consortium, which will mainly develop new policy evidence in the field of innovation and inclusiveness for the creative and cultural sectors, industries. The Consortium applies for a Horizon Europe grant with the HORIZON-CL2-2021-HERITAGE-01-03 Cultural and creative industries as a driver of innovation and competitiveness call of the European Commission.

Research & Analysis: Music Creators’ Earnings in the Digital Era

Thu, 23 Sep 2021 08:00:00 +0000

Reprex with its Digital Music Observatory team was commissioned to prepare an analysis on the justified and not justified differences in music creators’ earnings. We have posted our most important findings in an earlier blogpost (Music Creators’ Earnings in the Streaming Era. United Kingdom Research Cooperation With the Digital Music Observatory.

The UK Intellectual Property Office has published the entire report on the music creators’ earnings, and we have made our detailed analysis available in a side-publication. Reprex also signed an agreement with the researchers of the Music Creators’ Earnings project to deposit all data published in the report in the Digital Music Observatory, and to promote the building of the observatory further.

The research questions asked in this report are related to the Music Creator Earnings’ Project (MCE), exploring issues concerning equitable remuneration and earnings distributions. We were tasked with providing a longitudinal analysis of earnings development and relating our findings to equitable remuneration. The starting point of our work was centred around a very broadly defined problem: how much money music creators (rightsholders) earn from streaming, how these earnings are distributed, and how the earnings and their distribution have developed during the last decade.

The highly globalized music industry generates two important international reports, as well as several national reports, but these are not suitable for the analysis of the typical or average rightsholder, nor for small labels and publishers who do not represent a large and internationally diversified portfolio of music works or recordings. Copyright and neighboring right revenues are collected in national jurisdictions. Because British artists are almost never constrained by their use of language, and the UK Music Industry is highly competitive in the global music markets, even relatively less known rightsholders earn revenues from dozens of national markets. The lack of market information on music sales volumes, prices for each jurisdiction, and the unaccounted for national, domestic, and foreign revenues makes the analysis of the rightholder’s earnings, or the economics of a certain distribution channel like music streaming or media platforms, impossible.

The Effect of International Diversification on Revenues - a combination of international price differences and exchange rate fluctuations.

While total earnings are reported by international and national organizations, they hide five important economic variables: changes in sales volumes, changes in prices, market share on various national jurisdictions (which have their own volume and price movements), the exchange rates applied, and the share of the repertoire exploited. Even worse, the global music industry has no comprehensive database of rightsholders, music works, and recordings – this is the data gap that we would like fill with the Digital Music Observatory.

Our report highlights some important lessons. First, we show that in the era of global music sales platforms it is impossible to understand the economics of music streaming without international data harmonization and advanced surveying and sampling. Paradoxically, without careful adjustments for accruals, market shares in jurisdictions, and disaggregation of price and volume changes, the British industry cannot analyze its own economics because of its high level of integration to the global music economy. Furthermore, the replacement of former public performances, mechanical licensing, and private copying remunerations (which has been available for British rightsholders in their European markets for decades) with less valuable streaming licenses has left many rightsholders poorer. Making adjustments on the distribution system without modifying the definition of equitable remuneration rights or the pro-rata distribution scheme of streaming platforms opens up many conflicts while solving not enough fundamental problems. Therefore, we suggest participation in international data harmonization and policy coordination to help regain the historical value of music.

Context

The idea of our Digital Music Observatory was brought to the UK policy debate on music streaming by the Written evidence submitted by The state51 Music Group to the Economics of music streaming review of the UK Parliaments’ DCMS Committee¹.

The music industry requires a permanent market monitoring facility to win fights in competition tribunals, because it is increasingly disputing revenues with the world’s biggest data owners. This was precisely the role of the former CEEMID² program, which was initiated by a group of collective management societies. Starting with three relatively data-poor countries, where data pooling allowed rightsholders to increase revenues, the CEEMID data collection program was extended in 2019 to 12 countries.The final regional report, after the release of the detailed Hungarian, Slovak and Croatian reports of CEEMID was sponsored by Consolidated Independent (of the state51 music group.)

CEEMID was eventually to formed into the Demo Music Observatory in 2020³, following the planned structure of the European Music Observatory, and validated in the world’s 2nd ranked university-backed incubator, the Yes!Delft AI+Blockchain Validation Lab. In 2021, under the final name Digital Music Observatory, it became open for any rightsholder or stakeholder organization or music research institute, and it is being launched with the help of the JUMP European Music Market Accelerator Programme which is co-funded by the Creative Europe Programme of the European Union.

In December 2020, we started investigating how the music observatory concept could be introduced in the UK, and how our data and analytical skills could be used in the Music Creators’ Earnings in the Streaming Era (in short: MCE) project, which is taking place paralell to the heated political debates around the DCMS inquiry. After the state51 music group gave permission for the UK Intellectual Property Office to reuse the data that was originally published as the experimental CEEMID-CI Streaming Volume and Revenue Indexes, we came to a cooperation agreement between the MCE Project and the Digital Music Observatory. We provided a detailed historical analysis and computer simulation for the MCE Project, and we will host all the data of the Music Creators’ Earnings Report in our observatory, hopefully no later than early July 2021.

The Digital Music Observatory contributes to the Music Creators’ Earnings in the Streaming Era project with understanding the level of justified and unjustified differences in rightsholder earnings, and putting them into a broader music economy context.

We started our cooperation with the two principal investigators of the project, Prof David Hesmondhalgh and Dr Hyojugn Sun back in April and will start releasing the findings and the data in July 2021.

Join us

Do you need high-quality data for your music business or institution? Are you a music researcher? Join our open collaboration Digital Music Observatory team as a data curator, developer or business developer.

Footnote References

state51 Music Group. 2020. “Written Evidence Submitted by The state51 Music Group. Economics of Music Streaming Review. Response to Call for Evidence.” UK Parliament website. https://committees.parliament.uk/writtenevidence/15422/html/. ↩︎
Artisjus, HDS, SOZA, and Candole Partners. 2014. “Measuring and Reporting Regional Economic Value Added, National Income and Employment by the Music Industry in a Creative Industries Perspective. Memorandum of Understanding to Create a Regional Music Database to Support Professional National Reporting, Economic Valuation and a Regional Music Study.” ↩︎
Antal, Daniel. 2021. “Launching Our Demo Music Observatory.” Data & Lyrics. Reprex. https://dataandlyrics.com/post/2020-09-15-music-observatory-launch/. ↩︎

The Data Sisyphus

Thu, 08 Jul 2021 09:00:00 +0000

Sisyphus was punished by being forced to roll an immense boulder up a hill only for it to roll down every time it neared the top, repeating this action for eternity. This is the price that project managers and analysts pay for the inadequate documentation of their data assets.

When was a file downloaded from the internet? What happened with it sense? Are their updates? Did the bibliographical reference was made for quotations? Missing values imputed? Currency translated? Who knows about it – who created a dataset, who contributed to it? Which is an intermediate format of a spreadsheet file, and which is the final, checked, approved by a senior manager?

Big data creates inequality and injustice. On aspect of this inequality is the cost of data processing and documentation – a greatly underestimated, and usually not reported cost item. In small organizations, where there are no separate data science and data engineering roles, data is usually supposed to be processed and documented by (junior) analysts or researchers. This a very important source of the gap between Big Tech and them: the data usually ends up very expensive, ill-formatted, not readable by computers that use machine learning and AI. Usually the documentation steps are completely omitted.

“Data is potential information, analogous to potential energy: work is required to release it.” – Jeffrey Pomerantz

Metadata, which is information about the history of the data, and information how it can be technically and legally reused, has a hidden cost. Cheap or low-quality external data comes with poor or no metadata, and small organizations lack the resources to add high-quality metadata to their datasets. However, this only perpetuates the problem.

The hidden cost item behind the unbillable hours

As we have shown with our research partners, such metadata problems are not unique to data analysis. Independent artists and small labels are suffering on music or book sales platforms, because their copyrighted content is not well documented. If you automatically document tens of thousands of songs or datasets, the documentation cost is very small per item. If you, do it manually, the cost may be higher than the expected revenue from the song, or the total cost of the dataset itself. (See our research consortiums’ preprint paper: Ensuring the Visibility and Accessibility of European Creative Content on the World Market: The Need for Copyright Data Improvement in the Light of New Technologies)

In the short run, small consultancies, NGOs, or as a matter of fact, musicians, seem to logically give up on high-quality documentation and logging. In the long run, this has two devastating consequences: computers, such as machine learning algorithms cannot read their documents, data, songs. And as memory fades, the ill-documented resources need to be re-created, re-checked, reformatted. Often, they are even hard to find on your internal server or laptop archive.

Metadata is a hidden destroyer of the competitiveness of corporate or academic research, or independent content management. It never quoted on external data vendor invoices, it is not planned as a cost item, because metadata, the description of a dataset, a document, a presentation, or song, is meaningless without the resource that it describes. You never buy metadata. But if your dataset comes without proper metadata documentation, you are bound, like Sisyphus, to search for it, to re-arrange it, to check its currency units, its digits, its formatting. Data analysts are reported to spend about 80% of their working hours on data processing and not data analysis – partly, because data processing is a very laborious task that can be done by computers at a scale far cheaper, and partly because they do not know if the person who sat before them at the same desk has already performed these tasks, or if the person responsible for quality control checked for errors.

Uncut diamonds need to be cut, polished, and you have to make sure that they come from a legal source. Data is similar: it needs to be tidied up, checked and documented before use. Photo: Dave Fischer.

Undocumented data is hardly informative – it may be a page in a book, a file in an obsolete file format on a governmental server, an Excel sheet that you do not remember to have checked for updates. Most data are useless, because we do not know how it can inform us, or we do not know if we can trust it. The processing can be a daunting task, not to mention the most boring and often neglected documentation duties after the dataset is final and pronounced error-free by the person in charge of quality control.

Our observatory automatically processes and documents the data

The good news about documentation and data validation costs is that they can be shared. If many users need GDP/capita data from all over the world in euros, then it is enough if only one entity, a data observatory, collects all GDP and population data expresed in dollars, korunas, and euros, and makes sure that the latest data is correctly translated to euros, and then correctly divided by the latest population figures. These task are error-prone,and should not be repeaeted by every data journalist, NGO employee, PhD student or junior analyst. This is one of the services of our data observatory.

The tidy data format means that the data has a uniform and clear data structure and semantics, therefore it can be automatically validated for many common errors and can be automatically documented by either our software or any other professional data science application. It is not as strict as the schema for a relational database, but it is strict enough to make, among other things, importing into a database easy.
The descriptive metadata contains information on how to find the data, access the data, join it with other data (interoperability) and use it, and reuse it, even years from now. Among others, it contains file format information and intellectual property rights information.
The processing metadata makes the data usable in strictly regulated professional environments, such as in public administration, law firms, investment consultancies, or in scientific research. We give you the entire processing history of the data, which makes peer-review or external audit much easier and cheaper.
The authoritative copy is held at an independent repository, it has a globally unique identifier that protects you from accidental data loss, mixing up with unfinished an untested version.

Cutting the dataset to a format with clear semantics and documenting it with the FAIR metadata concep exponentially increases the value of data. It can be publisehd or sold at a premium. Photo: Andere Andre.

While humans are much better at analysing the information and human agency is required for trustworthy AI, computers are much better at processing and documenting data. We apply to important concepts to our data service: we always process the data to the tidy format, we create an authoritative copy, and we always automatically add descriptive and processing metadata.

The value of metadata

Metadata is often more valuable and more costly to make than the data itself, yet it remains an elusive concept for senior or financial management. Metadata is information about how to correctly use the data and has no value without the data itself. Data acquisition, such as buying from a data vendor, or paying an opinion polling company, or external data consultants appears among the material costs, but metadata is never sold alone, and you do not see its cost.

In most cases, the reason why there is no gold rush for open data is that fact that while the EU member states release billions of euros’ worth data for free, or at very low cost, annually, it comes without proper metadata.

Data-as-Service

Reusable, legal, easy-to-import, interoperable, always fresh data in tidy formats with a modern API. Photo: Edgar Soto.

If the data source is cheap or has a low quality, you do not even get it. If you do not have it, it will show up as a human resource cost in research (when your analysist or junior researcher are spending countless hours to find out the missing metadata information on the correct use of the data) or in sales costs (when you try to reuse a research, consulting or legal product and you have comb through your archive and retest elements again and again.)

The data, together with the descriptive and administrative metadata, and links to the use license and the authoritative copy can be found in our API. Try it out!

Including Indicators from Arab Barometer in Our Observatory

Mon, 28 Jun 2021 09:00:00 +0000

A new version of the retroharmonize R package – which is working with retrospective, ex post harmonization of survey data – was released yesterday after peer-review on CRAN. It allows us to compare opinion polling data from the Arab Barometer with the Eurobarometer and Afrorbarometer. This is the first version that is released in the rOpenGov community, a community of R package developers on open government data analytics and related topics.

Surveys are the most important data sources in social and economic statistics – they ask people about their lives, their attitudes and self-reported actions, or record data from companies and NGOs. Survey harmonization makes survey data comparable across time and countries. It is very important, because often we do not know without comparison if an indicator value is low or high. If 40% of the people think that climate change is a very serious problem, it does not really tell us much without knowing what percentage of the people answered this question similarly a year ago, or in other parts of the world.

With the help of Ahmed Shabani and Yousef Ibrahim, we created a third case study after the Eurobarometer, and Afrobarometer, about working with the Arab Barometer harmonized survey data files.

Ex ante survey harmonization means that researchers design questionnaires that are asking the same questions with the same survey methodology in repeated, distinct times (waves), or across different countries with carefully harmonized question translations. Ex post harmonizations means that the resulting data has the same variable names, same variable coding, and can be joined into a tidy data frame for joint statistical analysis. While seemingly a simple task, it involves plenty of metadata adjustments, because established survey programs like Eurobarometer, Afrobarometer or Arab Barometer have several decades of history, and several decades of coding practices and file formatting legacy.

Variable harmonization means that if the same question is called in one microdata source Q108 and the other eval-parl-elections then we make sure that they get a harmonize and machine readable name without spaces and special characters.
Variable label harmonization means that the same questionnaire items get the same numeric coding and same categorical labels.
Missing case harmonization means that various forms of missingness are treated the same way.

For the evaluation of the economic situation dataset, get the country averages and aggregates from Zenodo, and the plot in jpg or png from figshare.

In our new Arab Barometer case study, the evaulation of parliamentary elections has the following labels. We code them consistently 1: free_and_fair, 2: some_minor_problems, 3: some_major_problems and 4: not_free.

“0. missing”	“1. they were completely free and fair”
“2. they were free and fair, with some minor problems”	“3. they were free and fair, with some major problems”
“4. they were not free and fair”	“8. i don’t know”
“9. declined to answer”	“Missing”
“They were completely free and fair”	“They were free and fair, with some minor breaches”
“They were free and fair, with some major breaches”	“They were not free and fair”
“Don’t know”	“Refuse”
“Completely free and fair”	“Free and fair, but with minor problems”
“Free and fair, with major problems”	“Not free or fair”
“Don’t know (Do not read)”	“Decline to answer (Do not read)”

Of course, this harmonization is essential to get clean results like this:

For evaluation or reuse of parliamentary elections dataset get the replication data and the code from the Zenodo open repository.

In our case study, we had three forms of missingness: the respondent did not know the answer, the respondent did not want to answer, and at last, in some cases the respondent was not asked, because the country held no parliamentary elections. While in numerical processing, all these answers must be left out from calculating averages, for example, in a more detailed, categorical analysis they represent very different cases. A high level of refusal to answer may be an indicator of surpressing democratic opinion forming in itself.

Survey harmonization with many countries entails tens of thousands of small data management task, which, unless automatically documented, logged, and created with a reproducible code, is a helplessly error-prone process. We believe that our open-source software will bring many new statistical information to the light, which, while legally open, was never processed due to the large investment needed.

We also started building experimental APIs data is running retroharmonize regularly. We will place cultural access and participation data in the Digital Music Observatory, climate awareness, policy support and self-reported mitigation strategies into the Green Deal Data Observatory, and economy and well-being data into our Economy Data Observatory.

Further plans

Retrospective survey harmonization is a far more complex task than this blogpost suggest. Retrospective survey harmonization is a far more complex task than this blogpost suggest, because established survey programs have gathered decades of legacy data in legacy coding schemes and legacy file formats. Putting the data right, and especially putting the invaluable descriptive and administrative (processing) metadata right is a huge undertaking. We are releasing example codes, datasets and charts for researchers to comapre our harmonized results with theirs, and improve our software. We are releasing example codes, datasets and charts for researchers to comapre our harmonized results with theirs, and improve our software.

Use our software

The retroharmonize R package can be freely used, modified and distributed under the GPL-3 license. For the main developer and contributors, see the package homepage. If you use it for your work, please kindly cite it as:

Daniel Antal (2021). retroharmonize: Ex Post Survey Data Harmonization. R package version 0.1.17. https://doi.org/10.5281/zenodo.5034752

Download the BibLaTeX entry.

Tutorial to work with the Arab Barometer survey data

Daniel Antal, & Ahmed Shaibani. (2021, June 26). Case Study: Working With Arab Barometer Surveys for the retroharmonize R package (Version 0.1.6). Zenodo. https://doi.org/10.5281/zenodo.5034759

For the replication data to report potential issues and improvement suggestions with the code:

Daniel Antal, & Ahmed Shaibani. (2021). Replication Data for the retroharmonize R Package Case Study: Working With Arab Barometer Surveys (Version 0.1.6) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.5034741

Experimental API

We are also experimenting with the automated placement of authoritative and citeable figures and datasets in open repositories. For the climate awareness dataset get the country averages and aggregates from Zenodo, and the plot in jpg or png from figshare. Our plan is to release open data in a modern API with rich descriptive metadata meeting the Dublin Core and DataCite standards, and further administrative metadata for correct coding, joining and further manipulating or data, or for easy import into your database.

Join our open source effort

Want to help us improve our open data service? Include Lationbarómetro and the Caucasus Barometer in our offering? Join the rOpenGov community of R package developers, an our open collaboration to create the automated data observatories. We are not only looking for developers, but data curators and service design associates, too.

Open Data - The New Gold Without the Rush

Fri, 18 Jun 2021 17:00:00 +0000

If open data is the new gold, why even those who release fail to reuse it? We created an open collaboration of data curators and open-source developers to dig into novel open data sources and/or increase the usability of existing ones. We transform reproducible research software into research- as-service.

Every year, the EU announces that billions and billions of data are now “open” again, but this is not gold. At least not in the form of nicely minted gold coins, but in gold dust and nuggets found in the muddy banks of chilly rivers. There is no rush for it, because panning out its value requires a lot of hours of hard work. Our goal is to automate this work to make open data usable at scale, even in trustworthy AI solutions.

There is no rush for it, because panning out its value requires a lot of hours of hard work. Our goal is to automate this work to make open data usable at scale, even in trustworthy AI solutions.

Most open data is not public, it is not downloadable from the Internet – in the EU parlance, “open” only means a legal entitlement to get access to it. And even in the rare cases when data is open and public, often it is mired by data quality issues. We are working on the prototypes of a data-as-service and research-as-service built with open-source statistical software that taps into various and often neglected open data sources.

We are in the prototype phase in June and our intentions are to have a well-functioning service by the time of the conference, because we are working only with open-source software elements; our technological readiness level is already very high. The novelty of our process is that we are trying to further develop and integrate a few open-source technology items into technologically and financially sustainable data-as-service and even research-as-service solutions.

Our review of about 80 EU, UN and OECD data observatories reveals that most of them do not use these organizations’s open data - instead they use various, and often not well processed proprietary sources.

We are taking a new and modern approach to the data observatory concept, and modernizing it with the application of 21st century data and metadata standards, the new results of reproducible research and data science. Various UN and OECD bodies, and particularly the European Union support or maintain more than 60 data observatories, or permanent data collection and dissemination points, but even these do not use these organizations and their members open data. We are building open-source data observatories, which run open-source statistical software that automatically processes and documents reusable public sector data (from public transport, meteorology, tax offices, taxpayer funded satellite systems, etc.) and reusable scientific data (from EU taxpayer funded research) into new, high quality statistical indicators.

We are taking a new and modern approach to the ‘data observatory’ concept, and modernizing it with the application of 21st century data and metadata standards, the new results of reproducible research and data science

We are building various open-source data collection tools in R and Python to bring up data from big data APIs and legally open, but not public, and not well served data sources. For example, we are working on capturing representative data from the Spotify API or creating harmonized datasets from the Eurobarometer and Afrobarometer survey programs.
Open data is usually not public; whatever is legally accessible is usually not ready to use for commercial or scientific purposes. In Europe, almost all taxpayer funded data is legally open for reuse, but it is usually stored in heterogeneous formats, processed into an original government or scientific need, and with various and low documentation standards. Our expert data curators are looking for new data sources that should be (re-) processed and re-documented to be usable for a wider community. We would like to introduce our service flow, which touches upon many important aspects of data scientist, data engineer and data curatorial work.
We believe that even such generally trusted data sources as Eurostat often need to be reprocessed, because various legal and political constraints do not allow the common European statistical services to provide optimal quality data – for example, on the regional and city levels.
With rOpenGov and other partners, we are creating open-source statistical software in R to re-process these heterogenous and low-quality data into tidy statistical indicators to automatically validate and document it.
We are carefully documenting and releasing administrative, processing, and descriptive metadata, following international metadata standards, to make our data easy to find and easy to use for data analysts.
We are automatically creating depositions and authoritative copies marked with an individual digital object identifier (DOI) to maintain data integrity.
We are building simple databases and supporting APIs that release the data without restrictions, in a tidy format that is easy to join with other data, or easy to join into databases, together with standardized metadata.
We maintain observatory websites (see: Digital Music Observatory, Green Deal Data Observatory, Economy Data Observatory) where not only the data is available, but we provide tutorials and use cases to make it easier to use them. Our mission is to show a modern, 21st century reimagination of the data observatory concept developed and supported by the UN, EU and OECD, and we want to show that modern reproducible research and open data could make the existing 60 data observatories and the planned new ones grow faster into data ecosystems.

We are working around the open collaboration concept, which is well-known in open source software development and reproducible science, but we try to make this agile project management methodology more inclusive, and include data curators, and various institutional partners into this approach. Based around our early-stage startup, Reprex, and the open-source developer community rOpenGov, we are working together with other developers, data scientists, and domain specific data experts in climate change and mitigation, antitrust and innovation policies, and various aspects of the music and film industry.

Our open collaboration is truly open: new data curators,developers and service designers, even volunteers and citizen scientists are welcome to join.

Our open collaboration is truly open: new data curators, data scientists and data engineers are welcome to join. We develop open-source software in an agile way, so you can join in with an intermediate programming skill to build unit tests or add new functionality, and if you are a beginner, you can start with documentation and testing our tutorials. For business, policy, and scientific data analysts, we provide unexploited, exciting new datasets. Advanced developers can join our development team: the statistical data creation is mainly made in the R language, and the service infrastructure in Python and Go components.

There are Numerous Advantages of Switching from a National Level of the Analysis to a Sub National Level

Wed, 16 Jun 2021 12:00:00 +0000

The new version of our rOpenGov R package regions was released today on CRAN. This package is one of the engines of our experimental open data-as-service Green Deal Data Observatory , Economy Data Observatory , Digital Music Observatory prototypes, which aim to place open data packages into open-source applications.

In international comparison the use of nationally aggregated indicators often have many disadvantages: they inhibit very different levels of homogeneity, and data is often very limited in number of observations for a cross-sectional analysis. When comparing European countries, a few missing cases can limit the cross-section of countries to around 20 cases which disallows the use of many analytical methods. Working with sub-national statistics has many advantages: the similarity of the aggregation level and high number of observations can allow more precise control of model parameters and errors, and the number of observations grows from 20 to 200-300.

The change from national to sub-national level comes with a huge data processing price: internal administrative boundaries, their names, codes codes change very frequently.

Yet the change from national to sub-national level comes with a huge data processing price. While national boundaries are relatively stable, with only a handful of changes in each recent decade. The change of national boundaries requires a more-or-less global consensus. But states are free to change their internal administrative boundaries, and they do it with large frequency. This means that the names, identification codes and boundary definitions of sub-national regions change very frequently. Joining data from different sources and different years can be very difficult.

Our regions R package helps the data processing, validation and imputation of sub-national, regional datasets and their coding.

There are numerous advantages of switching from a national level of the analysis to a sub-national level comes with a huge price in data processing, validation and imputation, and the regions package aims to help this process.

You can review the problem, and the code that created the two map comparisons, in the Maping Regional Data, Maping Metadata Problems vignette article of the package. A more detailed problem description can be found in Working With Regional, Sub-National Statistical Products.

This package is an offspring of the eurostat package on rOpenGov. It started as a tool to validate and re-code regional Eurostat statistics, but it aims to be a general solution for all sub-national statistics. It will be developed parallel with other rOpenGov packages.

Get the Package

You can install the development version from GitHub with:

devtools::install_github("rOpenGov/regions")

or the released version from CRAN:

install.packages("regions")

You can review the complete package documentation on regions.dataobservaotry.eu. If you find any problems with the code, please raise an issue on Github. Pull requests are welcome if you agree with the Contributor Code of Conduct

If you use regions in your work, please cite the package as: Daniel Antal, Kasia Kulma, Istvan Zsoldos, & Leo Lahti. (2021, June 16). regions (Version 0.1.7). CRAN. http://doi.org/10.5281/zenodo.4965909

Join us

Join our open collaboration Economy Data Observatory team as a data curator, developer or business developer. More interested in environmental impact analysis? Try our Green Deal Data Observatory team! Or your interest lies more in data governance, trustworthy AI and other digital market problems? Check out our Digital Music Observatory team!

Open Data is Like Gold in the Mud Below the Chilly Waves of Mountain Rivers

Thu, 10 Jun 2021 07:00:00 +0000

Open data is like gold in the mud below the chilly waves of mountain rivers. Panning it out requires a lot of patience, or a good machine.

As the founder of the automated data observatories that are part of Reprex’s core activities, what type of data do you usually use in your day-to-day work?

The automated data observatories are results of syndicated research, data pooling, and other creative solutions to the problem of missing or hard-to-find data. The music industry is a very fragmented industry, where market research budgets and data are scattered in tens of thousands of small organizations in Europe. Working for the music and film industry as a data analyst and economist was always a pain because most of the efforts went into trying to find any data that can be analyzed. I spent most of the last 7-8 years trying to find any sort of information—from satellites to government archives—that could be formed into actionable data. I see three big sources of information: textual,numeric, and continuous recordings for on-site, offsite, and satellite sensors. I am much better with numbers than with natural language processing, and I am improving with sensory sources. But technically, I can mint any systematic information—the text of an old book, a satellite image, or an opinion poll—into datasets.

For you, what would be the ultimate dataset, or datasets that you would like to see in the Economy Data Observatory?

I am a data scientist now, but I used to be a regulatory economist, and I have worked a lot with competition policy and monopoly regulation issues. Our observatories can automatically monitor market and environmental processes, which would allow us to get into computational antitrust. Peter Ormosi, our competition curator, is particularly interested in killer acquisitions: approved mergers of big companies that end up piling up patents that are not used. I am more interested in describing systematically which markets are getting more concentrated and more competitive, in real time. Does data concentration coincide with market concentration?

To bring an example from the realm of our Digital Music Observatory, which was a prototype to this one, I have been working for some time on creating streaming volume and price indexes, like the Dow Jones Industrial Average or the various bond market indexes, that talk more about price, demand, and potential revenue in music streaming markets all over the world. We did a first take on this in the Central European Music Industry Report and recently we iterated on the model for the UK Intellectual Property Office and the UK Music Creators’ Earnings project. We want to take this further to create a pan-Europe streaming market index, and we will be probably the first to actually be able to report on music market concentrations, and in fact, more or less in a real-time mode.

We would like to further developer our 20-country streaming indexes into a global music market index.

Is there a number or piece of information that recently surprised you? If so, what was it?

There were a few numbers that surprised me, and some of them were brought up by our observatory teams. Karel is talking about the fact that not all green energy is green at all: many hydropower stations contribute to the greenhouse effect and not reduce it. Annette brought up the growing interest in the Dalmatian breed after the Disney 101 Dalmatians movies, and it reminded me of the astonishing growth in interest for chess sets, chess tutorials, and platform subscriptions after the success of Netflix’s The Queen’s Gambit.

The Queen’s Gambit’ Chess Boom Moves Online By Rachael Dottle on bloomberg.com

Annette is talking about the importance of cultural influencers, and on that theme, what could be more exciting that Netflix’s biggest success so far is not a detective series or a soap opera but a coming-of-age story of a female chess prodigy. Intelligence is sexy, and we are in the intelligence business.

But to tell a more serious and more sobering number, I recently read with surprise that there are more people smoking cigarettes on Earth in 2021 than in 1990. Population growth in developing countries replaced the shrinking number of developed country smokers. While I live in Europe, where smoking is strongly declining, it reminds me that Europe’s population is a small part of the world. We cannot take for granted that our home-grown experiences about the world are globally valid.

Do you have a good example of really good, or really bad use of data?

FiveThirtyEight.com had a wonderful podcast series, produced by Jody Avirgan, called What’s the Point. It is exactly about good and bad uses of data, and each episode is super interesting. Maybe the most memorable is Why the Bronx Really Burned. New York City tried to measure fire response times, identify redundancies in service, and close or re-allocate fire stations accordingly. What resulted, though, was a perfect storm of bad data: The methodology was flawed, the analysis was rife with biases, and the results were interpreted in a way that stacked the deck against poorer neighborhoods. It is similar to many stories told in a very compelling argument by Catherine D’Ignazio and Lauren F. Klein in their much celebrated book, Data Feminism. Usually, the bad use of data starts with a bad data collection practice. Data analysts in corporations, NGOs, public policy organizations and even in science usually analyze the data that is available.

You can find these examples, together with many more that our contributors recommend, in the motivating examples of Create New Datasets and the Remain Critical parts of our onboarding material. We hope that more and more professionals and citizen scientist will help us to create high-quality and open data.

The real power lies in designing a data collection program. A consistent data collection program usually requires an investment that only powerful organizations, such as government agencies, very large corporations, or the richest universities can afford. You cannot really analyze the data that is not collected and recorded; and usually what is not recorded is more interesting than what is. Our observatories want to democratize the data collection process and make it more available, more shared with research automation and pooling.

You cannot really analyze the data that is not collected and recorded; and usually what is not recorded is more interesting than what is. Our observatories want to democratize the data collection process and make it more available, more shared with research automation and pooling.

From your perspective, what do you see being the greatest problem with open data in 2021?

I have been involved with open data policies since 2004. The problem has not changed much: more and more data are available from governmental and scientific sources, but in a form that makes them useless. Data without clear description and clear processing information is useless for analytical purposes: it cannot be integrated with other data, and it cannot be trusted and verified. If researchers or government entities that fall under the Open Data Directive release data for reuse in a way that does not have descriptive or processing metadata, it is almost as if they did not release anything. You need this additional information to make valid analyses of the data, and to reverse-engineer them may cost more than to recollect the data in a properly documented process. Our developers, particularly Leo and Pyry are talking eloquently about why you have to be careful even with governmental statistical products, and constantly be on the watch out for data quality.

Our API is not only publishing descriptive and processing metadata alongside with our data, but we also make all critical elements of our processing code available for peer-review on rOpenGov

What do you think the Economy Data Observatory, and our other automated observatories do, to make open data more credible in the European economic policy community and be accepted as verified information?

Most of our work is in research automation, and a very large part of our efforts are aiming to reverse engineer missing descriptive and processing metadata. In a way, I like to compare ourselves to the working method of the open-source intelligence platform Bellingcat. They were able to use publicly available, scattered information from satellites and social media to identify each member of the Russian military company that illegally entered the territory of Ukraine and shot down the Malaysian Airways MH17 with 297, mainly Dutch, civilians on board.

How we create value for research-oriented consultancies, public policy institutes, university research teams, journalists or NGOs.

We do not do such investigations but work very similarly to them in how we are filtering through many data sources and attempting to verify them when their descriptions and processing history is unknown. In the last years, we were able to estore the metadata of many European and African open data surveys, economic impact, and environmental impact data, or many other open data that was lying around for many years without users.

Open data is like gold in the mud below the chilly waves of mountain rivers. Panning it out requires a lot of patience, or a good machine. I think we will come to as surprising and strong findings as Bellingcat, but we are not focusing on individual events and stories, but on social and environmental processes and changes.

Join us

Where Are People More Likely To Treat Climate Change as the Most Serious Global Problem?

Sat, 06 Mar 2021 00:00:00 +0000

library(regions)
library(lubridate)
library(dplyr)
if ( dir.exists('data-raw') ) {
data_raw_dir <- "data-raw"
} else {
data_raw_dir <- file.path("..", "..", "data-raw")
}

The first results of our longitudinal table were difficult to map, because the surveys used an obsolete regional coding. We will adjust the wrong coding, when possible, and join the data with the European Environment Agency’s (EEA) Air Quality e-Reporting (AQ e-Reporting) data on environmental pollution. We recoded the annual level for every available reporting stations [not shown here] and all values are in μg/m3. The period under observation is 2014-2016. Data file: https://www.eea.europa.eu/data-and-maps/data/aqereporting-8 (European Environment Agency 2021).

Recoding the Regions

Recoding means that the boundaries are unchanged, but the country changed the names and codes of regions because there were other boundary changes which did not affect our observation unit. We explain the problem and the solution in greater detail in our tutorial that aggregates the data on regional levels.

panel <- readRDS((file.path(data_raw_dir, "climate-panel.rds")))
climate_data_geocode <- panel %>%
mutate ( year: lubridate::year(date_of_interview)) %>%
recode_nuts()

Let’s join the air pollution data and join it by corrected geocodes:

load(file.path("data", "air_pollutants.rda")) ## good practice to use system-independent file.path
climate_awareness_air <- climate_data_geocode %>%
rename ( region_nuts_codes : .data$code_2016) %>%
left_join ( air_pollutants, by: "region_nuts_codes" ) %>%
select ( -all_of(c("w1", "wex", "date_of_interview",
"typology", "typology_change", "geo", "region"))) %>%
mutate (
# remove special labels and create NA_numeric_
age_education: retroharmonize::as_numeric(age_education)) %>%
mutate_if ( is.character, as.factor) %>%
mutate (
# we only have responses from 4 years, and this should be treated as a categorical variable
year: as.factor(year)
) %>%
filter ( complete.cases(.) )

The climate_awareness_air data frame contains the answers of 75086 individual respondents. 17.07% thought that climate change was the most serious world problem and 33.6% mentioned climate change as one of the three most important global problems.

summary ( climate_awareness_air )
## rowid serious_world_problems_first
## ZA5877_v2-0-0_1 : 1 Min. :0.0000
## ZA5877_v2-0-0_10 : 1 1st Qu.:0.0000
## ZA5877_v2-0-0_100 : 1 Median :0.0000
## ZA5877_v2-0-0_1000 : 1 Mean :0.1707
## ZA5877_v2-0-0_10000: 1 3rd Qu.:0.0000
## ZA5877_v2-0-0_10001: 1 Max. :1.0000
## (Other) :75080
## serious_world_problems_climate_change isocntry
## Min. :0.000 BE : 3028
## 1st Qu.:0.000 CZ : 3023
## Median :0.000 NL : 3019
## Mean :0.336 SK : 3000
## 3rd Qu.:1.000 SE : 2980
## Max. :1.000 DE-W : 2978
## (Other):57058
## marital_status age_education
## (Re-)Married: without children :13242 18 :15485
## (Re-)Married: children this marriage :12696 19 : 7728
## Single: without children : 7650 16 : 5840
## (Re-)Married: w children of this marriage: 6520 still studying: 5098
## (Re-)Married: living without children : 6225 17 : 5092
## Single: living without children : 4102 15 : 4528
## (Other) :24651 (Other) :31315
## age_exact occupation_of_respondent
## Min. :15.0 Retired, unable to work :22911
## 1st Qu.:36.0 Skilled manual worker : 6774
## Median :51.0 Employed position, at desk : 6716
## Mean :50.1 Employed position, service job: 5624
## 3rd Qu.:65.0 Middle management, etc. : 5252
## Max. :99.0 Student : 5098
## (Other) :22711
## occupation_of_respondent_recoded
## Employed (10-18 in d15a) :32763
## Not working (1-4 in d15a) :37125
## Self-employed (5-9 in d15a): 5198
##
##
##
##
## respondent_occupation_scale_c_14
## Retired (4 in d15a) :22911
## Manual workers (15 to 18 in d15a) :15269
## Other white collars (13 or 14 in d15a): 9203
## Managers (10 to 12 in d15a) : 8291
## Self-employed (5 to 9 in d15a) : 5198
## Students (2 in d15a) : 5098
## (Other) : 9116
## type_of_community is_student no_education
## DK : 34 Min. :0.0000 Min. :0.000000
## Large town :20939 1st Qu.:0.0000 1st Qu.:0.000000
## Rural area or village :24686 Median :0.0000 Median :0.000000
## Small or middle sized town: 9850 Mean :0.0679 Mean :0.008151
## Small/middle town :19577 3rd Qu.:0.0000 3rd Qu.:0.000000
## Max. :1.0000 Max. :1.000000
##
## education year region_nuts_codes country_code
## Min. :14.00 2013:25103 LU : 1432 DE : 4531
## 1st Qu.:17.00 2015: 0 MT : 1398 GB : 3538
## Median :18.00 2017:25053 CY : 1192 BE : 3028
## Mean :19.61 2019:24930 SK02 : 1053 CZ : 3023
## 3rd Qu.:22.00 EL30 : 974 NL : 3019
## Max. :30.00 EE : 973 SK : 3000
## (Other):68064 (Other):54947
## pm2_5 pm10 o3 BaP
## Min. : 2.109 Min. : 5.883 Min. : 66.37 Min. :0.0102
## 1st Qu.: 9.374 1st Qu.: 28.326 1st Qu.: 90.89 1st Qu.:0.1779
## Median :11.866 Median : 33.673 Median :102.81 Median :0.4105
## Mean :12.954 Mean : 38.637 Mean :101.49 Mean :0.8759
## 3rd Qu.:15.890 3rd Qu.: 49.488 3rd Qu.:110.73 3rd Qu.:1.0692
## Max. :41.293 Max. :123.239 Max. :141.04 Max. :7.8050
##
## so2 ap_pc1 ap_pc2 ap_pc3
## Min. : 0.0000 Min. :-4.6669 Min. :-2.21851 Min. :-2.1007
## 1st Qu.: 0.0000 1st Qu.:-0.4624 1st Qu.:-0.49130 1st Qu.:-0.5695
## Median : 0.0000 Median : 0.4263 Median : 0.02902 Median :-0.1113
## Mean : 0.1032 Mean : 0.1031 Mean : 0.04166 Mean :-0.1746
## 3rd Qu.: 0.0000 3rd Qu.: 0.9748 3rd Qu.: 0.57416 3rd Qu.: 0.3309
## Max. :42.5325 Max. : 2.0344 Max. : 3.25841 Max. : 4.1615
##
## ap_pc4 ap_pc5
## Min. :-1.7387 Min. :-2.75079
## 1st Qu.:-0.1669 1st Qu.:-0.18748
## Median : 0.0371 Median : 0.01811
## Mean : 0.1154 Mean : 0.06797
## 3rd Qu.: 0.3050 3rd Qu.: 0.34937
## Max. : 3.2476 Max. : 1.42816
##

Let’s see a simple CART tree! We remove the regional codes, because there are very serious differences among regional climate awareness. These differences, together with education level, and the year we are talking about, are the most important predictors of thinking about climate change as the most important global problem in Europe.

# Classification Tree with rpart
library(rpart)
# grow tree
fit <- rpart(as.factor(serious_world_problems_first) ~ .,
method="class", data=climate_awareness_air %>%
select ( - all_of(c("rowid", "region_nuts_codes"))),
control: rpart.control(cp: 0.005))
printcp(fit) # display the results
##
## Classification tree:
## rpart(formula: as.factor(serious_world_problems_first) ~ .,
## data: climate_awareness_air %>% select(-all_of(c("rowid",
## "region_nuts_codes"))), method: "class", control: rpart.control(cp: 0.005))
##
## Variables actually used in tree construction:
## [1] age_education isocntry
## [3] serious_world_problems_climate_change year
##
## Root node error: 12817/75086: 0.1707
##
## n= 75086
##
## CP nsplit rel error xerror xstd
## 1 0.0240566 0 1.00000 1.00000 0.0080438
## 2 0.0082703 3 0.92783 0.92783 0.0078055
## 3 0.0050000 5 0.91129 0.91425 0.0077588
plotcp(fit) # visualize cross-validation results

summary(fit) # detailed summary of splits
## Call:
## rpart(formula: as.factor(serious_world_problems_first) ~ .,
## data: climate_awareness_air %>% select(-all_of(c("rowid",
## "region_nuts_codes"))), method: "class", control: rpart.control(cp: 0.005))
## n= 75086
##
## CP nsplit rel error xerror xstd
## 1 0.024056592 0 1.0000000 1.0000000 0.008043837
## 2 0.008270266 3 0.9278302 0.9278302 0.007805478
## 3 0.005000000 5 0.9112897 0.9142545 0.007758824
##
## Variable importance
## serious_world_problems_climate_change isocntry
## 31 26
## country_code BaP
## 20 8
## pm2_5 ap_pc1
## 4 3
## age_education pm10
## 2 2
## education ap_pc2
## 2 1
## year
## 1
##
## Node number 1: 75086 observations, complexity param=0.02405659
## predicted class=0 expected loss=0.1706976 P(node): 1
## class counts: 62269 12817
## probabilities: 0.829 0.171
## left son=2 (25229 obs) right son=3 (49857 obs)
## Primary splits:
## serious_world_problems_climate_change < 0.5 to the right, improve=2214.2040, (0 missing)
## isocntry splits as RRLLLRRRLLRLRLLLLLLLLLLRRLLLRLL, improve= 728.0160, (0 missing)
## country_code splits as RRLLLRRLLRLLLLLLLLLLRRLLLRLL, improve= 673.3656, (0 missing)
## BaP < 0.4300347 to the right, improve= 310.6229, (0 missing)
## pm2_5 < 13.38264 to the right, improve= 296.4013, (0 missing)
## Surrogate splits:
## age_education splits as ----RRRRRR-RRRRRRRRRR-RRRRRRRRRR-RRRRRRRRRR-RRRRRRRRRR-RRRRRL-RRR-RRRRRRRRR--RRRLLR--R-R, agree=0.664, adj=0, (0 split)
## pm10 < 7.491315 to the left, agree=0.664, adj=0, (0 split)
##
## Node number 2: 25229 observations
## predicted class=0 expected loss=0 P(node): 0.3360014
## class counts: 25229 0
## probabilities: 1.000 0.000
##
## Node number 3: 49857 observations, complexity param=0.02405659
## predicted class=0 expected loss=0.2570752 P(node): 0.6639986
## class counts: 37040 12817
## probabilities: 0.743 0.257
## left son=6 (34631 obs) right son=7 (15226 obs)
## Primary splits:
## isocntry splits as RRLLLRRRLLRLRLLLLLLLLLLRRLLLRLL, improve=1454.9460, (0 missing)
## country_code splits as RRLLLRRLLRLLLLLLLLLLRRLLLRLL, improve=1359.7210, (0 missing)
## BaP < 0.4300347 to the right, improve= 629.8844, (0 missing)
## pm2_5 < 13.38264 to the right, improve= 555.7484, (0 missing)
## ap_pc1 < -0.005459537 to the left, improve= 533.3579, (0 missing)
## Surrogate splits:
## country_code splits as RRLLLRRLLRLLLLLLLLLLRRLLLRLL, agree=0.987, adj=0.957, (0 split)
## BaP < 0.1749425 to the right, agree=0.775, adj=0.264, (0 split)
## pm2_5 < 5.206993 to the right, agree=0.737, adj=0.140, (0 split)
## ap_pc1 < 1.405527 to the left, agree=0.733, adj=0.126, (0 split)
## pm10 < 25.31211 to the right, agree=0.718, adj=0.076, (0 split)
##
## Node number 6: 34631 observations
## predicted class=0 expected loss=0.1769802 P(node): 0.4612178
## class counts: 28502 6129
## probabilities: 0.823 0.177
##
## Node number 7: 15226 observations, complexity param=0.02405659
## predicted class=0 expected loss=0.4392487 P(node): 0.2027808
## class counts: 8538 6688
## probabilities: 0.561 0.439
## left son=14 (11607 obs) right son=15 (3619 obs)
## Primary splits:
## isocntry splits as LL---LLR--L-L----------LL---R--, improve=337.5462, (0 missing)
## country_code splits as LL---LR--L-L--------LL---R--, improve=337.5462, (0 missing)
## age_education splits as ----LLLLLL-LLLRRRRRRR-RRRRRRRRRL-RRRRRRLLRR-RRRRLLRLRL-RRLRRR-RRR-LLLLRRR-----LR-----L-R, improve=294.0807, (0 missing)
## education < 22.5 to the left, improve=262.3747, (0 missing)
## BaP < 0.053328 to the right, improve=232.7043, (0 missing)
## Surrogate splits:
## BaP < 0.053328 to the right, agree=0.878, adj=0.485, (0 split)
## pm2_5 < 4.810361 to the right, agree=0.827, adj=0.271, (0 split)
## ap_pc2 < 0.8746175 to the left, agree=0.792, adj=0.124, (0 split)
## so2 < 0.3302972 to the left, agree=0.781, adj=0.078, (0 split)
## age_education splits as ----LLLLLL-LLLLLLLRLR-LRRLRRRRRR-RRRRLLLLLR-LRLRLLRRLL-LLRLLR-LLR-RRLLLLL-----RR-----R-L, agree=0.779, adj=0.071, (0 split)
##
## Node number 14: 11607 observations, complexity param=0.008270266
## predicted class=0 expected loss=0.3804601 P(node): 0.1545827
## class counts: 7191 4416
## probabilities: 0.620 0.380
## left son=28 (7462 obs) right son=29 (4145 obs)
## Primary splits:
## age_education splits as ----LLLLLL-LRRRRRRRRR-RRLRRLRRLL-RRRRLRLLRR-RLRLLLRLRL-RR-RR--RRL-L-LLRRR------------L-R, improve=123.71070, (0 missing)
## year splits as R-LR, improve=107.79460, (0 missing)
## education < 20.5 to the left, improve= 90.28724, (0 missing)
## occupation_of_respondent splits as LRRLRRRRRLRLLLRLLL, improve= 84.62865, (0 missing)
## respondent_occupation_scale_c_14 splits as LRLLLRRL, improve= 68.88653, (0 missing)
## Surrogate splits:
## education < 20.5 to the left, agree=0.950, adj=0.861, (0 split)
## occupation_of_respondent splits as LLLLRLLRRLRLLLRLLL, agree=0.738, adj=0.267, (0 split)
## respondent_occupation_scale_c_14 splits as LRLLLLRL, agree=0.733, adj=0.251, (0 split)
## is_student < 0.5 to the left, agree=0.709, adj=0.186, (0 split)
## age_exact < 23.5 to the right, agree=0.676, adj=0.094, (0 split)
##
## Node number 15: 3619 observations
## predicted class=1 expected loss=0.3722023 P(node): 0.04819807
## class counts: 1347 2272
## probabilities: 0.372 0.628
##
## Node number 28: 7462 observations
## predicted class=0 expected loss=0.326052 P(node): 0.09937938
## class counts: 5029 2433
## probabilities: 0.674 0.326
##
## Node number 29: 4145 observations, complexity param=0.008270266
## predicted class=0 expected loss=0.4784077 P(node): 0.05520337
## class counts: 2162 1983
## probabilities: 0.522 0.478
## left son=58 (2573 obs) right son=59 (1572 obs)
## Primary splits:
## year splits as L-LR, improve=40.13885, (0 missing)
## occupation_of_respondent splits as LRLLRRRRRLRLLLRLLL, improve=18.33254, (0 missing)
## marital_status splits as LRRRLRRRLRRLRLLRRRRRRLRLRLLRR, improve=17.86888, (0 missing)
## type_of_community splits as LRLRL, improve=17.55254, (0 missing)
## age_education splits as ------------LLRRRRRRR-RR-RL-RR---LRRR-R--LR-R-R---R-R--RR-RR--RR------RRR--------------R, improve=14.66121, (0 missing)
## Surrogate splits:
## type_of_community splits as LLLRL, agree=0.777, adj=0.412, (0 split)
## marital_status splits as RRLLLLLRLLLLLLLRRRLLLLLLRLRLL, agree=0.680, adj=0.155, (0 split)
## isocntry splits as LL---LL---L-R----------LL------, agree=0.669, adj=0.127, (0 split)
## country_code splits as LL---L---L-R--------LL------, agree=0.669, adj=0.127, (0 split)
## o3 < 83.06345 to the right, agree=0.650, adj=0.076, (0 split)
##
## Node number 58: 2573 observations
## predicted class=0 expected loss=0.4240187 P(node): 0.03426737
## class counts: 1482 1091
## probabilities: 0.576 0.424
##
## Node number 59: 1572 observations
## predicted class=1 expected loss=0.43257 P(node): 0.02093599
## class counts: 680 892
## probabilities: 0.433 0.567
# plot tree
plot(fit, uniform=TRUE,
main="Classification Tree: Climate Change Is The Most Serious Threat")
text(fit, use.n=TRUE, all=TRUE, cex=.8)
## Warning in labels.rpart(x, minlength: minlength): more than 52 levels in a
## predicting factor, truncated for printout

saveRDS ( climate_awareness_air , file.path(tempdir(), "climate_panel_recoded.rds"), version: 2)
# not evaluated
saveRDS( climate_awareness_air, file: file.path("data-raw", "climate-panel_recoded.rds"))

What is Retrospective Survey Harmonization?

Thu, 04 Mar 2021 00:00:00 +0000

Reproducible ex post harmonization of survey microdata

Retrospective survey harmonization allows the comparison of opinion poll data conducted in different countries or time. In this example we are working with data from surveys that were ex ante harmonized to a certain degree – in our tutorials we are choosing questions that were asked in the same way in many natural languages. For example, you can compare what percentage of the European people in various countries, provinces and regions thought climate change was a serious world problem back in 2013, 2015, 2017 and 2019.

We developed the retroharmonize R package to help this process. We have tested the package with about 80 Eurobarometer, 5 Afrobarometer survey files extensively, and a bit with Arabbarometer files. This allows the comparison of various survey answers in about 70 countries. This policy-oriented survey programs were designed to be harmonized to a certain degree, but their ex post harmonization is still necessary, challenging and errorprone. Retrospective harmonization includes harmonization of the different coding used for questions and answer options, post-stratification weights, and using different file formats.

Eurobarometer, Afrobaromer, Arab Barometer and Latinobarómetro make survey files that are harmonized across countries available for research with various terms. Our retroharmonize is not affiliated with them, and to run our examples, you must visit their websites, carefully read their terms, agree to them, and download their data yourself. What we add as a value is that we help to connect their files across time (from different years) or across these programs.

The survey programs mentioned above publish their data in the proprietary SPSS format. This file format can be imported and translated to R objects with the haven package; however, we needed to re-design haven’s labelled_spss class to maintain far more metadata, which, in turn, a modification of the labelled class. The haven package was designed and tested with data stored in individual SPSS files.

The author of labelled, Joseph Larmarange describes two main approaches to work with labelled data, such as SPSS’s method to store categorical data in the Introduction to labelled.

Two main approaches of labelled data conversion.

Our approach is a further extension of Approach B. Survey harmonization in our case always means the joining data from several SPSS files, which requires a consistent coding among several data sources. This means that data cleaning and recoding must take place before conversion to factors, character or numeric vectors. This is particularly important with factor data (and their simple character conversions) and numeric data that occasionally contains labels, for example, to describe the reason why certain data is missing. Our tutorial vignette labelled_spss_survey gives you more information about this.

In the next series of tutorials, we will deal with an array of problems. These are not for the faint heart – you need to have a solid intermediate level of R to follow.

Tidy, joined survey data

The original files identifiers may not be unique, we have to create new, truly unique identifiers. Weighting may not be straightforward.
Neither the number of observations or the number of variables (which represents the survey questions and their translation to coded data) is the same. Certain data may be only present in one survey and not the other. This means that you will likely to run loops on lists and not data.frames, but eventually you must carefully join them.

Class conversion

Similar questions may be imported from a non-native R format, in our case, from an SPSS files, in an inconsistent manner. SPSS’s variable formats cannot be translated unambiguously to R classes. retroharmonize introduced a new S3 class system that handles this problem, but eventually you will have to choose if you want to see a numeric or character coding of each categorical variable.
The harmonized surveys, with harmonized variable names and harmonized value labels, must be brought to consistent R representations (most statistical functions will only work on numeric, factor or character data) and carefully joined into a single data table for analysis.

Harmonization of variables and variable labels

Same variables may come with dissimilar variable names and variable labels. It may be a challenge to match age with age. We need to harmonize the names of variables.
The harmonized variables may have different labeling. One may call refused answers as declined and the other refusal. On a simple choice, climate change may be ‘Climate change’ or Problem: Climate change. Binary choices may have survey-specific coding conventions. Value labels must be harmonized. There are good tools to do this in a single file - but we have to work with several of them.

Missing value harmonization

There are likely to be various types of missing values. Working with missing values is probably where most human judgment is needed. Why are some answers missing: was the question not asked in some questionnaires? Is there a coding error? Did the respondent refuse the question, or sad that she did not have an answer? retroharmonize has a special labeled vector type that retains this information from the raw data, if it is present, but you must make the judgment yourself – in R, eventually you will either create a missing category, or use NA_character_ or NA_real_.

That’s a lot to put on your plate.

It is unlikely that you will be able to work with completely unfamiliar survey programs if you do not have a strong intermediate level of R. Our package comes with tutorials for Eurobarometer, Afrobarometer and our development version already covers Arab Barometer, highlighting some peculiar issues with these survey programs, that we hope to give a head start for less experienced R users.

Eurobarometer Surveys Used In Our Project

Wed, 03 Mar 2021 00:00:00 +0000

In our tutorial series, we are going to harmonize the following questionnaire items from five Eurobarometer harmonized survey files. The Eurobarometer survey files are harmonized across countries, but they are only partially harmonized in time.

All data must be downloaded from the GESIS Data Archive in Cologne. We are not affiliated with GESIS and you must read and accept their terms to use the data.

Eurobarometer 80.2 (2013)

GESIS Data Archive, Cologne. ZA5877 Data file Version 2.0.0, https://doi.org/10.4232/1.12792

Data file: ZA6595 data file (European Commission 2017).
Questionnaire: Eurobarometer 83.4 Basic Bilingual Questionnaire
Citation: ZA6595 Bibtex

QA1a Which of the following do you consider to be the single most serious problem facing the world as a whole? (single choice)

QA1b Which others do you consider to be serious problems? (multiple choice)

QA2 And how serious a problem do you think climate change is at this moment? Please use a scale from 1 to 10, with '1' meaning it is "not at all a serious problem (scale 1-10)

QA4 To what extent do you agree or disagree with each of the following statements? - Fighting climate change and using energy more efficiently can boost the economy and jobs in the EU (agreement-disagreement 4-scale)

QA4 To what extent do you agree or disagree with each of the following statements? - Reducing fossil fuel imports from outside the EU could benefit the EU economically (agreement-disagreement 4-scale)

QA5 Have you personally taken any action to fight climate change over the past six months? (binary)

Eurobarometer 83.4 (2015)

European Commission, Brussels; Directorate General Communication COMM.A.1 ´Strategy, Corporate Communication Actions and Eurobarometer´GESIS Data Archive, Cologne. ZA6595 Data file Version 3.0.0, https://doi.org/10.4232/1.13146

Data file: ZA6595 data file (European Commission 2018).
Questionnaire: Eurobarometer 83.4 Basic Bilingual Questionnaire
Citation: ZA6595 Bibtex

Eurobarometer 87.1 (2017)

European Commission, Brussels; Directorate General Communication, COMM.A.1 ‘Strategic Communication’; European Parliament, Directorate-General for Communication, Public Opinion Monitoring Unit GESIS Data Archive, Cologne. ZA6861 Data file Version 1.2.0, https://doi.org/10.4232/1.12922

Data file: ZA6861 data file.
Questionnaire: Eurobarometer 90.2 Basic Bilingual Questionnaire
Citation: ZA6861 Bibtex

QC1a Which of the following do you consider to be the single most serious problem facing the world as a whole? (single choice)

QC1b Which others do you consider to be serious problems? (multiple choice)

QC2 And how serious a problem do you think climate change is at this moment? Please use a scale from 1 to 10, with '1' meaning it is "not at all a serious problem (scale 1-10)

Qc4 To what extent do you agree or disagree with each of the following statements? - Fighting climate change and using energy more efficiently can boost the economy and jobs in the EU (agreement-disagreement 4-scale)

Qc4 To what extent do you agree or disagree with each of the following statements? - Promoting EU expertise in new clean technologies to countries outside the EU can benefit the EU economically (agreement-disagreement 4-scale)

Qc4 To what extent do you agree or disagree with each of the following statements? - Reducing fossil fuel imports from outside the EU can benefit the EU economically (agreement-disagreement 4-scale)

Qc4 To what extent do you agree or disagree with each of the following statements? - Reducing fossil fuel imports from outside the EU can increase the security of EU energy supplies (agreement-disagreement 4-scale)

Qc4 To what extent do you agree or disagree with each of the following statements? - More public financial support should be given to the transition to clean energies even if it means subsidies to fossil fuels should be reduced. (agreement-disagreement 4-scale)

Qc5 Have you personally taken any action to fight climate change over the past six months? (binary)

Eurobarometer 90.2 (2018)

European Commission, Brussels; Directorate General Communication, COMM.A.3 ‘Media Monitoring and Eurobarometer’ GESIS Data Archive, Cologne. ZA7488 Data file Version 1.0.0, https://doi.org/10.4232/1.13289

Data file: ZA7488 data file (European Commission 2019a)
Questionnaire: Eurobarometer 90.2 Basic Bilingual Questionnaire
Citation: ZA7488 Bibtex

QB5 To what extent do you agree or disagree with each of the following statements? - Fighting climate change and using energy more efficiently can boost the economy and jobs in the EU (agreement-disagreement 4-scale)

QB5 To what extent do you agree or disagree with each of the following statements? - Promoting EU expertise in new clean technologies to countries outside the EU can benefit the EU economically (agreement-disagreement 4-scale)

QB5 To what extent do you agree or disagree with each of the following statements? - Reducing fossil fuel imports from outside the EU can benefit the EU economically (agreement-disagreement 4-scale)

QB5 To what extent do you agree or disagree with each of the following statements? - Reducing fossil fuel imports from outside the EU can increase the security of EU energy supplies (agreement-disagreement 4-scale)

QB5 To what extent do you agree or disagree with each of the following statements? - More public financial support should be given to the transition to clean energies even if it means subsidies to fossil fuels should be reduced. (agreement-disagreement 4-scale)

Eurobarometer 91.3 (2019)

European Commission, Brussels; Directorate General Communication, COMM.A.3 ‘Media Monitoring and Eurobarometer’ GESIS Data Archive, Cologne. ZA7572 Data file Version 1.0.0, https://doi.org/10.4232/1.13372

Data file: ZA7572 data file (European Commission 2019b).
Questionnaire: Eurobarometer 91.3 Basic Bilingual Questionnaire
Citation: ZA7572 Bibtex

QB4 To what extent do you agree or disagree with each of the following statements? - Taking action on climate change will lead to innovation that will make EU companies more competitive (N) (agreement-disagreement 4-scale)

QB4 To what extent do you agree or disagree with each of the following statements? - Promoting EU expertise in new clean technologies to countries outside the EU can benefit the EU economically (agreement-disagreement 4-scale)

QB4 To what extent do you agree or disagree with each of the following statements? - Reducing fossil fuel imports from outside the EU can benefit the EU economically (agreement-disagreement 4-scale)

QB4 To what extent do you agree or disagree with each of the following statements? - Adapting to the adverse impacts of climate change can have positive outcomes for citizens in the EU (agreement-disagreement 4-scale)

QB5 Have you personally taken any action to fight climate change over the past six months? (binary)

References

European Commission, Brussels. 2017. “Eurobarometer 80.2 (2013).” GESIS Data Archive, Cologne. ZA5877 Data file Version 2.0.0, https://doi.org/10.4232/1.12792. https://doi.org/10.4232/1.12792.

———. 2018. “Eurobarometer 83.4 (2015).” GESIS Data Archive, Cologne. ZA6595 Data file Version 3.0.0, https://doi.org/10.4232/1.13146. https://doi.org/10.4232/1.13146.

———. 2019a. “Eurobarometer 90.2 (2018).” GESIS Data Archive, Cologne. ZA7488 Data file Version 1.0.0, https://doi.org/10.4232/1.13289. https://doi.org/10.4232/1.13289.

———. 2019b. “Eurobarometer 91.3 (2019).” GESIS Data Archive, Cologne. ZA7572 Data file Version 1.0.0, https://doi.org/10.4232/1.13372. https://doi.org/10.4232/1.13372.

Daniel Antal | Reprex

Learn R with Reprex

Big Data Creates Inequalities

Slide navigation

Big data that works for all

Get Inspired

Contributor Covenant

Run code from tutorials

Find help, ask for help: reprex

Documentation for better tutorials

Debugging and testing code

Contribute to documentation

R is a functional language

R + YAML + markdown = web ready

Package and release: a team effort

Our open source development projects

Create with us

Questions?

Learn R with Reprex

Big Data Creates Inequalities

Slide navigation

Big data that works for all

Data Feminism

Get Inspired

Contributor Covenant

Run code from tutorials

Find help, ask for help: reprex

Documentation for better tutorials

Debugging and testing code

Contribute to documentation

R is a functional language

R + YAML + markdown = web ready

Package and release: a team effort

Our open source development projects

Create with us

Questions?

Surveyharmonization

Survey Harmonization Workflow

Controls

Principles

Timeline

Concepts

Data Model

Question Bank

Translations

Fieldwork

Code and Save

Output harmonization

Presentation

Questions?

Reprex Open Collaboration NLAIC 2022

Reprex is looking for new collaborations within the NLAIC

Questions?

Reprex

Big Data Creates Inequalities

Slide navigation

Big data that works for all

Connected financial and sustainability reporting based on open data

Data problems: Reprex

Data problems: Eviota

Data problems (examples)

Our solution: Reprex

Our solution: Eviota (Non-Financial)

Our solution: Eviota (For Banks)

UNIIQ

Market

Competition

Know-how and integration of open source components

Team [rewrite with GDO]

Timeline

Data observatories 3.0

Marketing strategy

Target market size

Team

Traction

Funding

Do it Smarter

Questions?

Reprex

Big Data Creates Inequalities