Predicting the Next Big Hit - Big Data and the Music Industry

The problem in the music industry

Not long ago, there was a strong belief that the internet was killing the music industry. For years now, the industry has failed to keep up with the rapid pace of technological advancement. This clearly meant that consumers turned to illegal downloading for the sake of convenience as well as pricing.

Current solutions that the music industry came up with, such as Apple’s iTunes, enjoyed some success. But they were always hampered by device compatibility and digital rights management issues. 

Then came streaming. Streaming finally offered a solution with the potential to overcome the chief draw of illegal downloading –which was that the videos could be consumed for free - as well as convenience, since services offered by YouTube were free and tunes could be streamed on any device.

And music and technology became allies, yet again.

When Big Data came along

Big Data and Analytics played a major role in this modern-day romance.

From recommendation engines to choosing the perfect individual playlist and IoT-enabled pop concerts, data is redefining the dynamics of the music industry and the relationship between music and its listeners, in more creative ways than ever.

A decade ago, the music industry barely had any understanding of their audience, who was buying their LPs, CDs, or cassettes. With downloading services taking over, these companies found ways to track the listening habits of users and making recommendations, the same way that Amazon does for books.

However, with the latest streaming model, the floodgates stand wide open. Companies have access to detailed information such as when, how, where, and who is listening to what.

The aim of the industry, now, is to use these customer behaviour insights together with knowledge of the music itself, which is made possible only with Big Data. The raw music that is produced is essentially like unstructured data. In the digital era, this raw music can be easily digitized and analyzed.

The music industry predicting the future

The music industry lays great emphasis on predicting the future. This happens at all levels of granularity – from deciding what the individual user of a streaming service wants next on their playlist, to discovering the next Gangnam Style. It has recently been shown that Big Data has the ability to do just that.

Researchers at the University of Antwerp were able to create an algorithm to predict the position that the dance records would chart at in the Billboard Dance Singles chart. This was fairly accurate.

Of late, the Internet of Things has also found its footing in pop music. This year, attendees at the Taylor Swift concert were given LED bracelets controlled with RFID technology that changes colour and pulse in tune with the music.

With a large chunk of the music industry’s revenue coming from live music performances, we can expect increasingly creative ways of creating new experiences for live audiences. 

Pandora Media

Since 1999, the Musical Genome Project, developed by Pandora Media, has been using the process of structuring music data with the help of manual classification as well as automated algorithms. There are up to 450 data points that are collected with every song in the database, which currently stands at around 30 million.

These parameters include the instruments that are in use, the gender of the vocalist, the style of the back-up vocals, and the tempo of the rhythm. Each of these tracks is studied by specially trained musicians, similar to the way that Netflix employs people to watch and classify their content.

Due to this structuring of unstructured data that comes from raw music, the tracks can be compared to each other and judgements can be made, algorithmically, about what the user would like to listen to next.

Spotify

Spotify, a commercial music streaming service, was launched in 2008. It currently has over 24 million registered active users, of which 6 million are paying users. With over 20 million songs online, 20,000 new ones are added to its database every day.

Users all over the world have created over 1 billion playlists and over $500 million has been paid out to right holders since the launch of this database. This makes it abundantly clear that without Big Data, Spotify would cease to exist.

Spotify positions itself as a data driven company, which means that data is made use of in all the aspects of the organization. And there are numbers to prove this –

1. Users at Spotify create 600 GB of data per day and 150 GB of data per day via different services.
2. 4TB of data are generated daily in Hadoop – a 700 node cluster that runs over 2,000 jobs per day.
3. Spread across 4 data centres across the world, the company has 28 PB of storage.

The company has also developed an open source workflow manager called Luigi. Luigi is a python framework for data definition and execution. The manager is also used to crunch tons of data. Most of this data is, however, user centric. For instance, the billions of log messages which allow the database to provide music recommendations or select the next song on the radio.

This data is also used in decision making and providing forecasting information and business analytics.
Spotify also uses this data in various other forms. For example, in 2013, the database used streaming data to predict the winner of the Grammys. They made this possible by breaking down the user’s listening habit by taking into account the song and the album that was being streamed to determine the popularity of the music. By the end of this experiment, 4 out of 6 of their predictions turned out to be right.

Spotify would not have turned out the way it did without Big Data.

With a growing presence in many countries and a rapidly growing listener base, there will be a lot more data created in the coming years.

With more data comes better predictions, better recommendations, and more users, which results in better payouts to those with the rights. 

Preparing for a career in Data Science? Take this test to know where you stand!

Big Data has become such an essential part of our lives that all our technology will cease to exist without it. Name any industry – fashion, music, technology, food, marketing, business- and we find that Big Data has made its mark there.

The demand for skilled Big Data Engineers has never been higher. If you are a part of this enticing industry, or wish to become a part of it, up your game with a certification. Take up a course in Big Data.

Big Data Hadoop Training Course

What are you waiting for?

Get out there and get certified today!