IP-Audio Networks in the Real World: MPR and Univision
by Clark Novak
Cleveland, Ohio, USA
Most broadcasters agree that the audio systems of the future will be IP-based. However, many do not know that systems using Internet Protocol ad-dressing with Switched Ethernet to transport audio are already widely deployed in broadcasting facilities. In fact, IP-Audio has been solving real-world problems for several years.
This paper will explore the challenges faced by broadcasters, and show, using case studies, how some prominent broadcasters have used IP-Audio to meet these challenges. Topics covered will in-clude cost benefits, ease of configuration, on-and off-site remote administration, satellite content distribution, automated event handling, reduced operator learning curve and training requirements and accessibility to non-IT engineering staff.
As with analog audio installations, Livewire set-ups range from the very simple to complex facility-wide installations with hundreds of ports. This section is aimed primarily at those who will be building large systems.
"At my first radio job in 1981, carts were king. Everything played from carts... commercials, jingles, Legal ID's, bits, the EBS test, even the music — hundreds and hundreds of red Fidelipac cartridges loaded into a wall of IGM Instacarts just outside the studio. The King is Dead, Long Live the King"
Introduced at the 1959 NAB Convention, Collins/ATC's P-150 cart machines quickly obsoleted the 16-inch vinyl transcriptions stations had formerly used to store audio productions. By the time I entered the business, carts had been the industry standard for more than 20 years.
Fast-forward to 1990. Carts were still king. But we read about some guy named Freedman in New Mexico who'd used a couple of IBM PCs loaded with those new digital audio cards to automate a nighttime airshift. I remember our GM turning red in the face from laughter at this news (he'd just finished a mammoth 4-hour download session to get the latest Tapscan data, at 300 bps). PCs required constant attention, and those early 8-bit mono cards delivered audio that was only passable at best. "It'll never work," he laughed. "They'll have to have someone standing by all night just to re-boot the computers every 10 minutes!"
One year later, a PC software application sim-ply called "Cart Machine" racked up over 2,000 Compuserve downloads just 90 days after its re-lease. That same year our Production Director be-gan fooling with an editing program called SAW – "Software Audio Workshop".
By 1996, cart machines – the rulers of the studio for over 30 years – were dead as the dodo, killed by the disruptive technology of the personal computer.
Today, another type of computer technology is poised to make traditional audio distribution infra-structure as obsolete as the cart machine: IP-Audio networking. These networks employ the same IP (Internet Protocol) addressing technology that powers business networking and the Internet to send sound and data to its intended destinations, eliminating the discrete-wired model we've used since the beginning of radio.
Everyone is talking about this new technology, and just about everyone agrees that IP-Audio Networks are the future — but they also have questions about how IP Audio can be used today, in the real world.
The Future is Now
Why are IP-Audio networks considered to be the future of the broadcast plant? Consider this thought from Cisco Systems Vice President Peter Alexander on the benefits of IP-based systems:
"With an IP infrastructure, all data, voice, and video applications can be integrated onto a single, secure network. New applications work reliably with existing applications because they are all based on the same protocol. Features that were un-available or too expensive with traditional systems...can be deployed relatively easily.
"Plus, with just one network to manage and maintain, a company is better able to scale its network to meet changing business needs and user requirements. Furthermore, having an IP network enables a small or medium-sized business to use the same IP applications as a large enterprise, which improves its competitive advantage."
If you translate Alexander's comments from the business sales environment to that of broadcast, his points remain valid. IP-Audio networks can allow broadcasters to cut costs by utilizing a common transport mechanism for audio, messaging and other forms of data traffic. They can give broadcasters the flexibility to easily grow and change that traditional nailed-up systems lack. And because IP-Audio networks are standards-based, utilizing common components already proven in the multi-billion dollar computer networking arena, even smaller stations can afford to deploy them, imparting operational advantages previously available to only the wealthiest operators.
The delivery of real-time IP-Audio in the IT world is already commonplace, as seen in the accelerating migration from traditional telephone services to Voice-over-IP (VoIP) technology.
Microsoft Chairman Bill Gates has thrown his considerable weight behind converged IP-based networks:
"And this idea that we're going to unify on the network voice, video and data...that's some of the work we're doing, and some of the phone companies are now moving toward that unified network. That's another one of those Holy Grails...But that's really coming to the fore."
According to data from Infonetics Research, sales of VoIP-based PBX systems outstripped traditional TDM systems for the first time during 2005, with revenues growing at a steady 11% per quarter. And the growth curve is increasing: Infonetics estimates that by 2008, VoIP systems will account for over 90% of PBX sales (a whopping $8.3 billion in revenue), while traditional phone systems' share will decline to just 8%.
Accordingly, the Telecommunications Industry Association projects that the number of VoIP access lines in the US will quadruple, increasing from 6.5 million in 2004 to 26 million by 2008. And in September of 2005, Cisco, reporting the sale of its 6-millionth IP phone, said that VoIP is displacing up to 8,000 traditional circuit-based telephones every business day.
Other interesting delivery mechanisms are adding to the attractiveness of IP-Audio networks. Take, for instance, NPR's forthcoming Content-Depot® program distribution system, which will employ IP-over-satellite technology already proven in the video world. ContentDepot's IP-Audio delivery system will allow stations to automate content-on-demand by integrating with their digital delivery systems.
Instead of being slaves to real-time program feeds, NPR affiliates will be able to browse a list of programs, arrange feeds when they're most convenient, and download metadata including promos, audio samples and rights information. A station will be able to download an hour-long show in just five minutes if it needs it immediately. No more missed program feeds, mad scrambles to obtain a dub or manually cutting up blocks of promo feeds; subscribers will have total control thanks to IP-Audio distribution.
All of this leads to the inescapable conclusion that the broadcasting industry is on the verge of an IP-fueled revolution in distribution and infrastructure design.
How IP-Audio Works
The particular mechanics of how IP-Audio works have been discussed at length elsewhere, so we won't dwell on details here.
In a nutshell, the process is as follows:
Individual audio sources are connected to "audio nodes" located in local studios and/or central server areas. These nodes convert incoming analog or AES/EBU signals to uncompressed 48 kHz, 24-bit digital audio, which is further converted to packet data suitable for Ethernet transmission. Each audio node input and output is assigned a unique IP address for identification and routing purposes.
Logic ports on each audio device (which provide on/off/start pulses, tally lamp closures, etc.) are connected to GPIO nodes which convert these commands to packet data also.
Each node, connected to a local QoS-compliant Ethernet switch, makes its audio and control data available to the network for subscription. Gigabit Ethernet or fiber-optic links between switches provide extremely high system capacity (many thousands of stereo signals per system).
Each studio's local Ethernet switch is connected to the other rooms via core switches or daisy-chain style. In this manner, control of all audio devices and their output streams are made available for on-demand use anywhere within the network. Since these audio streams are now in routable data form, automated software control of routing paths and switching configurations is possible. Entire routing networks can be remapped with a single mouse click.
To simplify I/O even further, IP-Audio inter-faces can be built-in to broadcast equipment, al-lowing devices to send and receive audio channels to and from the network by simply connecting them to a local switch with an Ethernet cable.
Think of how this changes the installation of, say, a multi-line broadcast phone system. Traditional connection methods require dedicated wiring for two audio inputs, two audio outputs, pro-gram-on-hold input, logic connections to the con-sole modules, and another control circuit for PC screening software — that's more than half-a dozen separate cables.
The same phone system equipped with an IP-Audio interface requires only one Ethernet connection; multiple I/O and logic connections all travel on the same wire, making very short work of installation.
Now consider computer-based delivery systems. Traditionally, playout computers are loaded with audio cards, with multiple stereo connections to a router or other distribution system, and multiple start/stop/record logic controls as well.
Using IP-Audio, an integrated audio/control driver can be installed on the playout PC, allowing digital audio to travel directly from the computer to the network using the computer's NIC — no sound cards or D/A/D conversions. And of course, logic for each audio channel travels with it. Again, a single Ethernet cable takes the place of a handful of discrete pairs.
At the time of this writing, several major delivery system providers have announced their systems' compatibility with IP-Audio, including ENCO, Prophet, Scott Studios, BSI, Pristine and iMediaTouch, with more announcements imminent. IP-Audio-aware versions of broadcast phone systems, ISDN codecs, audio processors and satellite receivers are now in development and should debut shortly.
A not-insignificant side benefit of IP-Audio technology is lower cost. Both short- and long-term savings are realized in several different areas, including materials (cabling and mainframe router/switcher gear), installation (reduced labor costs) and maintenance (simplified infrastructure). Users of IP-Audio networks typically report an installed cost 20% to 35% less than that of traditional hardwired studios.
"All of this sounds terrific for five or 10 years in the future," you say. But what's shocking is that IP-Audio is transforming the broadcast environment now. Let's take a look at how.
For years, broadcasters have built "mirror" studios; clusters of on-air and production rooms with identical layouts that can be used interchangeably. Once air talent or producers are trained in a single room, they are trained in all rooms – confusion and multiple learning curves are eliminated. Standardized design makes equipment and parts identical and exchangeable between rooms, and the entire complex benefits from more flexibility.
Unfortunately, the major objective – the ability to take any room to air as needed – is a logistical stumbling block with hardwired systems.
While the program output of a given room can be switched to feed a single airchain easily enough using a program select switcher, what if you want to make each room's outputs available to, say, three broadcast airchains and two network feeds? Suddenly, the switching mechanism becomes much more complex.
Taking a source to air in Studio "A" that originates from a delivery device in Studio "C" presents another set of concerns. Where will shared inputs appear on each studio console? Will machine control logic be available for shared sources? If so, what is the mechanism? And how much more will it cost?
With a TDM routing system, shared audio sources must be sent from studios to a central routing frame via dedicated pairs of wires (usually within a multi-pair cable bundle). Once there, the router receives commands from controllers mounted within the studio consoles (which require their own circuits to communicate with the central frame). Distribution circuits from the routing frame send requested audio back to the individual studios.
Until recently, most TDM routing systems were not able to route machine logic associated with audio feeds. Recent offerings allow logic commands to be multiplexed into the system, but this requires additional hardware not generally provided as standard equipment.
By contrast, due to its decentralized, "shared data" approach to audio and logic routing, IP-Audio networks simplify construction and use of identical studios. Audio sources local to each studio (and their associated machine logic) are linked to each studio's Ethernet switch; these "edge" switches are then connected using Gigabit Ethernet links.
Since Gigabit Ethernet has the capacity to carry several hundred simultaneous channels of stereo audio per link, the many pairs of home-and-back audio and control cabling required by traditional systems are eliminated, along with attendant labor and material costs. Audio, logic and program-associated data all travel the same CAT.6 cable.
An example where IP-Audio has delivered such simplification and cost savings can be found at WOR's new origination studio complex in Manhattan. In addition to each day's local programming, WOR generates unique program feeds des-tined for Internet streams, several different satellite networks, and the occasional television program.
These differing concurrent uses dictated a uniform design for WOR's studios, so that talent could use any available studio to generate programming. Four pairs of identical talk studios and control rooms were planned, along with six news booths, a Master Control, production room and technical center, all of which could require access to any audio in the building at any given time.
When estimates for such a facility from suppliers of TDM routing equipment were found to be hundreds of thousands of dollars higher than the allotted equipment budget, WOR investigated IP-Audio and found that it was able to accommodate all of their technical needs, including the ability to immediately access any source in any location, and to automate the switching of specified feeds to pre-determined destinations. Failsafe operation was achieved with the use of redundant core switches.
Tom Ray, Corporate Director of Engineering for Buckley Broadcasting/WOR Radio, says:
"Studio switching was greatly simplified, as the audio simply 'happens' in the studio, and it shows up exactly where we tell it to go. The router portion, which is actually integrated with the entire system, has become extremely flexible thanks to software control of Ethernet data, and can be changed at a moment's notice without giving it much thought.
"Sending local commercial cues to our radio networks has been vastly simplified, as the closures go into the system and get routed through the switcher along with the audio. No need to run separate control cabling from Master Control to each studio; we just use a GPIO node to route these commands over the network."
WOR found that building with IP-Audio not only satisfied their complex operational needs, but saved them roughly 25% of the cost of the same studios built with traditional means.
Hardwired facilities, by their very nature, are not amenable to growth. Multi-pair cables are easily outgrown when need outstrips system capacity; the only solution is to purchase and install additional cable. Often, cable trays and conduits must be added as well, which often means breaking into walls and ceilings.
TDM routers are easier to expand, but face a similar dilemma: the central frames that house their input cards can quickly reach their limit when new studios are added to the facility. Adding more inputs requires purchase of additional frames and cards and of course, the attendant wiring infrastructure to support the expansion. Since these frames are custom-built with proprietary technology, the expense can be considerable.
IP-Audio networks are not subject to these types of capacity limits or scalability roadblocks. Since the underlying transport structure is standard Switched Ethernet, expanding an IP-Audio network can be accomplished as easily as expanding a business computing network. All that's required to add a new studio to the net is to connect its audio nodes to a local Ethernet switch; that switch links to the core switch using a run of CAT.6 cable. The engineer can then quickly assign IP addresses to the new inputs using a standard Web browser.
This ability to scale at will with minimal effort gives IP-Audio networks a significant advantage over other tech. And while IP-Audio networks cannot scale upward infinitely, their ability to carry tens of thousands of stereo channels per system is enough to satisfy the needs of most facilities.
An example of IP-Audio's scalability can be found at the headquarters of Minnesota Public Radio in Saint Paul.
Faced with a growing amount of daily content that feeds two statewide networks, a nationwide satellite classical music service and many long form productions, MPR decided to greatly expand its facilities, which already consisted of eight control rooms, five on-air and production studios, two full recording studios and several small editing rooms.
MPR's planned expansion called for doubling the size of their facilities, adding another eight control rooms and studios, a news/announce booth and 10 edit/production rooms, as well as an auditorium space. After the new studios were built, existing rooms would be converted and brought on to the net as well, so the ability to handle large amounts of audio and expand easily were high priorities.
While MPR had used a TDM routing system for many years, they determined that IP-Audio's easy scalability, along with its enormous system capacity, was a better fit for their future.
MPR Chief of R&D Ethan Torrey:
"Late in 2003, we began planning our new technical infrastructure with a thorough examination of the distributed rout-ing/control surface model. Our goal was to determine if it would give us operational advantages. The answer to our re-search was a resounding yes...
"The lower cost of entry was a factor in our decision but not the driving force; we believe the combination of Ethernet and IP functionality is [IP-Audio's] biggest advantage. An IP-based infrastructure is scalable, and economic forces beyond the broadcast industry will continue to add capacity and functionality to that structure."
The nature of modern Ethernet also enabled MPR to make their audio network fully redundant and self-healing, an expensive proposition (when possible) with hardwired routing systems.
Studios are connected in pairs to edge switches, themselves connected via twin high-capacity fiber links to a Cisco Catalyst 6500-series core switch. The core contains two bladeservers, each backing up the other. With this configuration, if any portion of the system's core should experience service interruption, the other portion instantly and automatically takes over. Even in the unlikely event of a catastrophic core switch failure, the individual studios and their edge switches would remain operational and on-air.
As MPR engineers look to the future, they anticipate expanding their network even further to accommodate possible HD Radio™ multicast con-tent generation. As in WOR's case, building studios with IP-Audio cut their costs by roughly 33% compared to other studio build methods.
As anyone who's ever designed a studio knows, change happens throughout the process — sometimes even after the equipment has been specified, ordered and delivered.
With router/switchers, making quick changes or additions to designs can prove difficult. A router designed to handle a certain number of signals may reach a "plateau" in terms of capacity; only a few more inputs may be needed, but those few could require purchase of an entire additional routing frame, adding considerably to the project's cost.
IP-Audio networks solve this problem because they are both scalable and truly modular, enabling changes and additions to be made without punitive expense.
An illustration of this flexibility is found at Canadian Satellite Radio (also known as XM Canada). This project comprised not one, but two studio complexes in Montreal and Toronto for origination of XM's Canadian content channels. In the Toronto facility, studios are spread across two floors and consist of three production rooms, a control room and talk studio on one floor, and a talk studio, control room, production room and recording studio complex at street level. Programming generated in Canada is then fed back to XM headquarters in Washington, DC via broadband OC-3 connection.
Pippin Technical of Saskatoon was hired to de-sign and install both sets of studios. According to Tyler Everitt, Pippin's Sales Manager:
"One of the best things about IP-Audio is its ease of change. Because of the scope of the [XM] projects, a certain amount of changes to the initial plans were inevitable. But because we were building an IP-Audio plant, these changes were easily accommodated.
"Ethernet has a scalability and flexibility that other systems don't, so building with it lets you take a much more a la carte approach. If a studio in progress takes a hard right turn, you don't have to worry about maxing out the router, or how many inputs you have left.
"And down the road, additions are easy to accommodate. Need three more audio nodes? All you have to do is plug 'em in, whereas with other tech you reach a capacity plateau that requires more router cages and input cards."
IP-Audio networks' ability not only to scale but to co-exist with other systems makes it easy for broadcasters to begin migrating to the new technology without being forced to make wholesale changes to existing studios.
Why would this matter? Let's imagine that a radio station, having outgrown its existing facilities, plans on moving to a larger space in the future but needs another studio now.
Instead of upgrading the entire existing facility to an audio network, IP-Audio components are deployed within the new studio only. Program outputs, remote inputs and other audio feeds enter and exit the room as analog audio, and are translated to and from the networked format courtesy of an audio node.
This method can also be used to "stage" studio remodeling, updating the facility and retiring old gear on a studio-by-studio basis until the entire remodel is complete. This has the added benefit of allowing upgrade costs to be spread over time as well.
This is the method employed at two Univision clusters in Texas. In McAllen, Chief Engineer Jorge Garza has expanded and rebuilt his facilities, one room at a time, using IP-Audio components:
"We have three stations in McAllen... We had decided to upgrade our studio complex, starting with KBTQ. Univision has put switching/routing systems in several stations, so we started with that. And the air studio was in need of some freshening.
"We learned that [the] Ethernet backbone scales, like a computer network. All we'd have to do to grow is connect more nodes and surfaces, maybe add another Ethernet switch. We didn't have to commit to buying equipment for all of our studios at once."
"The station had already budgeted for two studios' worth of [routing and surfaces]. Then we learned about all the additional capabilities that an IP-Audio net-work could give us. The IP-Audio system actually cost a little more than the other one, but the additional flexibility couldn't be ignored. We probably made up for the extra dollars in installation time saved.
"An especially compelling factor was that we didn't have to blow our entire budget at once; that we could spread out the cost of remodeling several studios by re-working them one-at-a-time. We'll be replacing our old digital consoles with IP-Audio surfaces and routing as time and money permit."
Painless Configuration and Documentation
Documentation of installed systems is one of the most tedious and unwelcome tasks associated with building studios, especially those with complex audio switching systems.
Of course, any studio system should be thoroughly documented, but those that employ multi-pair cable demand it. This is needed to ensure that empty pairs can be readily accessed, and also in order to make sense of the massive amounts of wiring present when troubleshooting or system diagnoses are called for. It's safe to say that the job of assigning numbers and affixing labels to hundreds of individual pairs of wire and entering row upon row of them into a spreadsheet is disliked only a little less than that of changing the light bulbs in the GM's office.
There is also the joy of breaking out wire pairs from the cable bundle, connecting them to punch blocks, and soldering dozens (maybe hundreds!) of XLR connectors.
IP-Audio networks nearly eliminate this mind-numbing work. Let's trace the path of one stereo channel from a codec in the Tech Center to the control surface in Studio A:
The outputs of the satellite receiver are connected to an analog or AES/EBU audio node using premade XLR cables and a pre-made XLR-to-RJ45 dongle.
The audio node (with 8 stereo inputs and 8 stereo outputs) is connected to a 100Mbps port on a local Ethernet switch by one CAT.5e cable.
The local switches' Gigabit Ethernet port is connected to the system's core switch with one CAT.6 cable.
The core switch sends the audio channel to Studio A's edge switch over a second run of CAT.6.
The audio engine in Studio A presents the satellite audio to the control surface for mixing. The control surface and all other local nodes are attached to the local switch with CAT.5e cable.
Sound like a lot of Category cable? Not really, when you consider that each bidirectional Gigabit link can transport up to 250 stereo audio channels at one time. Multi-pair is eliminated, as are home-and-back cable runs, punch blocks and soldering. So is most infrastructure troubleshooting, as the plant's complexity is dramatically reduced.
But what about the audio channels themselves? Surely they must have unique identifiers — won't those need to be documented?
Of course. But the network's use of standard IP addressing makes short work of even that.
In an IP-Audio network, as in a standard Ethernet computer network, each node is assigned a Unicast IP address. The nodes, containing built-in webservers, can then be configured using a computer equipped with a Web browser.
During configuration, each node's inputs (and outputs) are given a channel number and descriptive text, e.g. "CD 1" or "Zephyr A". Behind the scenes, the node's software assigns each input and output a unique Multicast IP address.
These names and channel numbers follow the input's audio throughout the network, and are dis-played whenever a user browses or "takes" available feeds.
How does this simplify documentation? Terrence Dupuis, Chief Engineer at the University of Missouri's KWMU-FM in Saint Louis:
"I've always hated system documentation. I'd be hard pressed to name anyone who doesn't. It's massively time-consuming and boring, but of course it's absolutely necessary.
"An unexpected and very welcome benefit of our IP-Audio network was that it completely eliminated having to do this work by hand.
"Since input and output channel names and numbers are stored in the audio nodes, and the nodes display this in-formation on their internal web pages, all I had to do was hit 'print' on my browser when I finished configuring a node. I printed all of this data directly to PDFs stored on a network drive, and then I printed out a hard copy for backup. Instant system documentation! That's nice."
Remote Administration and Control Finally, since all the parts of an IP-Audio network are have assigned IP addresses, the ability to remotely administer the system is built in. All that's required is a network gateway to provide secure access to the world, and engineers are free to configure and troubleshoot their audio infrastructure off-site in the same manner as any net-work administrator.
Bonus: since studio consoles in the IP-Audio environment are just human interface devices controlling digital mixing engines, software applications can even enable talent to board-op themselves from remote locations.
But Is It Ready For Prime-Time?
Every so often, while I'm discussing the concept and mechanics of IP-Audio, someone will stop me and say "I've heard audio on the Internet. It stops and starts and just sounds awful. I can't put that stuff in my radio station!"
Let me make this clear: IP-Audio is not Internet audio!
The Internet is a tremendous open system, and as such it constantly falls prey to traffic congestion, bandwidth hogs, node failures and more. And that is why audio streamed on the Net so often sounds like a robot with a case of severe hiccups.
But IP-Audio networks are not Internet based — rather, they are carefully controlled environments where traffic overloads are not allowed to exist.
Ethernet networking, routing and switching systems have come a long way since their infancy. The computer industry has relentlessly improved Ethernet's capabilities, so that digital media signals can be reliably transported over controlled Ethernet audio networks with guaranteed quality of service (QoS).
Indeed, giant cable providers and entertainment companies such as Time-Warner, Cox and Disney have spent amazing sums to ensure that their IP-based TV-on-demand systems will work as advertised. Cisco, HP, IBM and Dell have likewise invested millions to perfect and incorporate this technology in their mission-critical Ethernet switching equipment.
In the same manner, IP-Audio networks employ switches with guaranteed QoS, along with careful system design and specialized transport protocols to deliver real-time, no-loss, synchronized Ethernet audio. WOR, the world's first large-scale deployment of IP-Audio networking, is proof of this; at the time of this writing, their net-work has been continuously delivering multiple program channels around the clock for more than a year.
The numerous operational benefits of IP-Audio networking have been and are being continuously proven by professional broadcasters around the world each and every day. Just as computers have revolutionized our daily lives since the first appearance of the IBM PC, data transport technology from the computer world will work a sea-change in conventional broadcast facilities, and sooner rather than later.
Those looking forward with this technology have the opportunity today to catch the express elevator to the top... those who miss it will find themselves painfully climbing the stairs instead.
Church, Steve. 2004. "Ethernet for Studio-Audio Systems." Cleveland: Telos Systems.
Dosch, Michael. 2004. "Axia – A Network-Enabled Radio Console Architecture." Cleveland: Axia Audio.
ContentDepot is a registered trademark of National Pubic Radio. HD Radio is a trademark of iBiquity Digital Corp. Zephyr is a trademark of TLS Corp.