Radio: Preparing WAV files for cloud-based radio

Chris Mendez -

The team at USC Radio Labs has been working to solve a pretty exciting problem: converting 60k+ audio files from WAV format to something more internet-ready. For context, in the 1990s, many radio stations started moving towards automating their music programming. The benefits were clear: no skips, no repeated songs, no "dead air," and for music directors, a way to plan and strategize their music curation.

If you remember, the 1990s was the decade of Microsoft Windows. Everyone was making Windows applications, and therefore, most of the high-end broadcast software was built on Windows using wav as their default file format.

For the most part, wav format is fantastic. It's like the perfect wrapper for storing music regardless of its encoding. For example, if you've ever worked with a zip file, wav is very similar to that. You can do much more with wav than just package a song.

The challenge? wav isn't built for internet distribution. They are big files to send over the web. A 3-minute song can average 40Mb. The second problem is that WAV isn't designed to store metadata information. So there's no way no add song title, composer, genre, year, and all of those other goodies –unless you start getting unnecessarily creative [1].

Going back to my team, as we begin exploring the idea of building a "cloud-based radio station," we thought it would be essential to learn how to convert our existing catalog into a new system that better prepares us for the future.

The goal was to convert wav to another uncompressed, lossless format that is still high-quality yet capable of handling ID3 data and open source.

The rest of this article will explain:

  • How we converted wav file to flac
  • How we automated the process of finding metadata using our source material.
  • How we injected that metadata into flac.
  • How we created Mp4 renditions.

Hello FLAC

FLAC is a technology that offers most of our key benefits, including:

  • Lossless audio compression.
  • Slightly smaller file sizes.
  • ID3 metadata is a first-class citizen.
  • License-free
  • Mac and PC friendly
  • Supported by a robust open-source community
  • An excellent suite of command-line tools

Step 1 - Creating a Process

From WAV to FLAC, a love story

One of my first tasks is figuring out converting data from WAV to FLAC. Thankfully, there are many options available, but I chose a tool that would allow me to create a scriptable, automation-friendly process.

As a starting point, I will use ffmpeg to convert my WAV file to FLAC format.

The only problem is that beets don't do well with WAV files. Therefore, our first step is to use ffmpeg to convert a WAV file to FLAC. Unfortunately, there is very little WAV support on beets, so we'll need to run this command on its own.

Once you have a FLAC file, you can run some fingerprinting.

Since I'm on a Mac, I installed the ffmpeg command-line through the Homebrew package manager to help with the conversion.

You must first have Rake and Homebrew installed to use the script below.

Create a file titled Rakefile and paste this code.

namespace :audio do
  desc %Q{ Install ffmpeg encoder. }
  task :install do
    # https://gist.github.com/predakanga/2376835
    sh %{ brew install ffmpeg boost taglib }    
  end
  
  desc %Q{ Encode WAV to FLAC }
  task :convert, [:input, :output] do |task, args|
    input = args.input
    output = args.output
    sh %{ ffmpeg -i #{input} -c:a flac #{output} }
  end
end

Download and install ffmpeg.

rake ffmpeg:install

Convert a WAV file to FLAC.

rake ffmpeg:convert[/path/to/file.wav,/path/to/file.flac]

Here are a few other examples showing you how to convert files to different sample rates

Step 2 - Injecting ID3 data

Now that my WAV files are in FLAC format, I want to figure out a way to find the metadata and inject it into the file.

Considering that I have 60k files, the only plausible way to automate this is to use the fingerprinting technology you see on products like Shazam.

Ideally, I also want to scour the Internet somehow and find the album art for each song. Here's how we can do both using beets.

Getting Started

So here's the deal, Music Fingerprinting is a fantastic technology with incredible benefits, but it requires a little bit of configuration.

Since I work on radio, it makes sense for me to learn the intricacies of creating a Shazam-like tool, but the barrier might be a little too high for most people. Therefore, I've created a script that automatically installs the necessary software for you.

You can choose to install the necessary libraries yourself by visiting the docs

Add these new namespaces to your Rakefile.

namespace :fingerprint do  
  desc %Q{ ›› Start Fingerprinting }
  task :install => [:dependencies] do
    puts "Great! We're ready to get started."
  end
  
  desc %Q{ ›› Install Fingerprinting Dependencies. }
  task :dependencies do
    # Install Chromaprint's tool fpcalc
    Rake::Task["fpcalc:install"].invoke("~/Desktop/fpcalc.tar.gz")
    # Install flac library
    sh %{ brew install flac }
    # Install pyacoustic so that we can interact with fpcalc from pythong
    sh %{ pip install pyacoustid }
    # Install beets
    sh %{ pip install beets[chroma] }
  end
end 

namespace :fpcalc do  
  desc %Q{ ›› Download fpcalc. }
  task :download do |task, args|
    source  = "https://github.com/acoustid/chromaprint/releases/download/v1.4.2/chromaprint-fpcalc-1.4.2-macos-x86_64.tar.gz"
    dest    = args.dest || "chromaprint.tar.gz"
    # Remove any previous downloads
    sh %{ rm -f #{dest} }
    # Download it as a new filename
    sh %{ curl -L #{source} -o #{dest} }
  end

  desc %Q{ ›› Install fpcalc. rake fpcalc:install[~/Desktop/tmp] }
  task :install, [:dest] => [:download] do |task, args|
    dest = args.dest || "chromaprint.tar.gz"
    # Unpack store the executable in /usr/local/bin
    # -x --extract = extract files from an archive
    # -v, --verbose = verbosely list files processed
    # -z, --gzip = gzipped files eg. for tar.gz packages
    # -f, --file ARCHIVE = use archive file or device ARCHIVE
    # -C, change directory
    sh %{ tar -xzvf #{dest} -C #{Dir.home}/Desktop --strip-components=1 }
    # Move fpcalc to /usr/loca/bin
    sh %{ mv fpcalc /usr/local/bin/fpcalc }
    # Double check your work.
    sh %{ which fpcalc }
    # Remove the desktop download
    sh %{ rm -f #{dest} }
  end
end

Now all you have to do is open up Terminal and run this command.

The command will install flac, pyacoustid, and fpcalc.

rake fingerprint:install
  • fpcalc is an open-source fingerprinting technology is available within a command-line package called fpcalc. If you're on a mac, download fpcalc library and copy the executable file to your /usr/local/bin.
  • We need pyacoustid package to interact with the Chromaprint library from Python.
  • beets is a command-line tool for Python that can automatically catalog your collection, transcode audio files, check for duplicates and help you fingerprint songs through chroma print. We are also including chroma plugin.

Configuring Beets

Now that we've installed all the necessary packages, we need to configure the beets application.

You will first need to configure beets so that you can install plugins.

Create a new config.yaml.

beet config -e

Here is an example of a config file. The file is located in ~/.config/beets/config.yaml.

# Copy cleaned-up music into that empty folder using beets’
directory: ~/Desktop/my_beets_app/my_music_folder
# Location of Beets music database
library:   ~/Desktop/musiclib.blb

import:
  # If no data is available, move on
  quiet_fallback: skip
  # Do not copy songs to another directory
  copy: no
  # Inject ID3 data into the song if found.
  write: yes

# Use the Chroma plugin
plugins: chroma

chroma:
  # When you type beets import, auto fingerprinting will begin.
  auto: yes

# You can do multiple processes at once. This is somewhat CPU intensive.
threaded: yes
# Show the command line
verbose: yes

Download a WAV file

For this example, let's use a Creative Commons WAV file from Nine Inch Nails titled "999,999".

Music Fingerprinting - Finding a Song's info based on its "sound."

Now that our development environment is ready. Let's try and fingerprint a WAV file that contains no metadata.

Acoustic fingerprinting is a technique for identifying songs from the way they –sound– instead of from their existing metadata.

Step - Import Music

Singletrack

Import a single track and include a log file.

beet import -A -C -s -l my_log_file.txt  my_music_folder/album/02.flac
  • -A do not auto-tag anything; import the files.
  • -C do not copy files to a new directory.
  • -s is the singleton method we should use when we're trying to access a single file.

Source

Entire Albums

Import a single track and include a log file.

beet import -A -C -l my_log_file.txt  my_music_folder/album/02.flac

Step - Beet Queries

Beet Queries

List out the music you've imported.

beet list

Resources

  1. Resource Interchange File Format (RIFF) ↩︎