Social media algorithms are shrouded in secrecy. We’re trying to change that

Over the past 20 years, social media has transformed how we communicate, share information and form social connections. A federal parliamentary committee is currently trying to come to grips with these changes, and work out what to do about them.

The social media platforms where we spend so much time are powered by algorithms that exercise significant control over what content each user sees. But researchers know little specific detail about how they work, and how users experience them.

This is because social media companies closely guard information about their algorithms and operations. However, in recent weeks my colleagues and I announced a new national infrastructure project to help us find out what they are up to.

Our project, the Australian Internet Observatory, will investigate how social media users interact and the content on their feeds. But the federal government can also help by forcing tech companies to let some light in to the closed black boxes that power their business.

Resistance to data access

To understand the impact of social media, we need to first understand its inner workings. This requires observing the content shared by users and the algorithms that control what content is visible and recommended.

We must also observe how users interact with these platforms in an everyday setting.

This is important because social media is personal and increasingly ephemeral. Content differs for every user and quickly disappears from feeds.

This makes it challenging to draw general conclusions about the experiences of users and the broader impact of social media on society.

But the companies behind social media platforms refuse to let the public peer under the hood. They often cite privacy concerns and competitive interests as reasons for limiting data access.

These concerns are possibly valid. But they are often cynically deployed. And they should not preclude the possibility of more transparent and ethical research data access.

As a result, my colleagues and I have had to be inventive to gain insights into the inner workings of social media. We use methods such as scraping public data, platform audits and other forensic methods.

However, these methods are often limited and fraught with legal risk.

The Australian Internet Observatory

In the absence of direct platform data access, we are also using other methods, such as data donation, to understand how social media platforms operate.

Data donation enables people to voluntarily share specific parts of their social media experience for independent study conducted under strict ethical guidelines. This provides invaluable insights while respecting user privacy and autonomy.

Two data donation projects have already improved our understanding of internet search and targeted advertising in Australia.

Over the next four years we will rapidly expand the scope of data donation through the new Australian Internet Observatory. This research infrastructure will collect and analyse the data of users of social media platforms such as Facebook, TikTok and YouTube.

This will shed new light not just on on how people interact on social media platforms but also on what content they see and how it is distributed. This enhanced visibility will improve our knowledge of the algorithms that power social media platforms – and their impact on society.

For example, since its launch in 2021, the Australian Ad Observatory has amassed nearly 800,000 Facebook ad donations from over 2,100 ordinary Australians.

This significant corpus of Facebook advertising data has allowed us to uncover illegal gambling advertising and track the prevalence of scam ads. We have also used this evidence to inform inquiries into unhealthy food advertising and “green washing”.

More than just being able to uncover what forms of advertising are prevalent and to whom they are targeted, this work has also helped us uncover details about the algorithmic targeting process itself.

The Australian Internet Observatory aims to further deepen our understanding of this and similar processes across many more platforms. We will soon be inviting members of the public to donate data from their social media platforms to help us achieve this.

Legislating data access in Australia

The Australian government has attempted to regulate various aspects of the internet and social media.

The Online Safety Act and recently proposed legislation targeting misinformation and disinformation illustrate the government’s concern over the influence of digital platforms.

However, these regulatory efforts have been flawed. Crucially, they are often proceeding without a comprehensive understanding of the actual activities and interactions taking place online.

Without this knowledge, regulations risk being either too broad, impacting legitimate expression and access, or too narrow, failing to address the root causes of online harms.

To strengthen the efforts of researchers to understand the impact social media platforms are having on society, it’s essential the Australian government follow the lead of the European Union by passing legislation which compels social media platforms to provide access to crucial data.

This would allow increased platform accountability. It would also empower researchers to conduct vital, independent, public-interest research with the transparency and support necessary to safeguard our digital future.