WHEAT:NEWS RADIO
JANUARY 2017
Vol 8, No.1
Got feedback or questions? Click my name below to send us an e-mail. You can also use the links at the top or bottom of the page to follow us on popular social networking sites and the tabs will take you to our most often visited pages.
Hacking IT
Let’s go back to the beginning of 2016, when there was no Zika, no President Trump, and Apple phones still came with headphone jacks.
We knew then that hackers were out there, but it happened to other industries, other people. Then, it happened to broadcasting’s own. First, stations in Colorado and Texas were hijacked by a cast of X-rated characters in animal costumes. Then, a group of Midwestern stations was knocked off the air by malware and held for ransom. By the time a hacker broke into a station’s EAS system during an emergency test in October and changed an EAS code to a line straight out of the Dr. Seuss book, Green Eggs and Ham, we all knew.
We no longer lived in the Age of Innocence. “Saga had been hit a couple of times, and although we have gone to great lengths to protect ourselves, it still keeps me up at night,” says Mark Spalding, the CE of seven Saga Communications stations in Illinois. The Illinois cluster hasn’t been hit but Saga stations elsewhere have, including an entire cluster that had been knocked off the air after malware had wormed its way into the automation system, corrupting all the stations’ music and data files.
The broadcast industry has been put on notice. We are all at risk and we all play a role in the solution, both manufacturer and broadcaster. At Wheatstone, we’re tightening security on our products and systems, from manufacturing practices to the very code that goes into the WheatNet-IP audio network. At broadcast facilities everywhere, we’re tightening security. The good news is that we can prevent malware and hackers from taking over if we work together.
Separate from the public internet. “WheatNet-IP and probably any other studio network should be kept either on a separate VPN or a physically separate network, not exposed to either in-house enterprise networks or the internet. Anything Internet-facing is vulnerable,” says Wheatstone Systems Engineer Scott Johnson.
We can’t stress this point enough. Read on.
Use proxies to isolate access. Ransomware made it past Saga’s antivirus software from a station PC to the automation computer, apparently masquerading as an email from a record label and corrupting the entire music library and associated files affecting all stations in the multi-station cluster. The group now isolates the automation and the studio network from the enterprise network through proxy computers. The automation computers used to have direct access to the internet, but no more. The PD, traffic and production computers all access the automation system through proxy computers. No exceptions.
Secure the WAN. Encrypted VPNs are the standard go-to option for securing networks. Connecting employee home PCs or studio proxy computers to a VPN creates a secure tunnel on an inter/intranet that handles traffic as if it were on a private circuit. VPN tunneling options vary from the more traditional Layer 2 options to more recent broadband VPN options that work at the Layer 3 level and use infrastructure (virtually or physically) separate from the public internet. For strictly security purposes, it’s hard to beat a dedicated separate leased line, especially if it’s fiber optic. These are generally more expensive than VPN options. But if you’re sending live-action feeds from a sports stadium to the home studio a state away for live mixing and final production, you might consider leasing “dark” fiber from a provider, as these links are virtually impenetrable to hackers. Whatever your link, whether it’s an existing ATM or newer MPLS, you’ll need to use VPN tunneling and encryption as protection against hackers.
Another option that is growing increasingly popular for short-distance WANs or STL hops (generally less than 20 miles, line of sight) are licensed-frequency IP radios. These microwave links are fairly secure from hackers, and they’re relatively affordable.
Saga is using a combination of encrypted VPNs and IP radios as its primary defense against hackers. “To keep us off the internet, we have encrypted VPN tunnels from one place to another and we’re using IP radios to transmit that data or portion of the network to the transmitter site or building or other location,” explains Spalding.
Back up all music and relevant files. “Can you believe they wanted us to give them $5,000 to unlock the files? That wasn’t going to happen,” says Spalding, referring to a ransomware attack at one Saga cluster. The local engineer was able to bring the stations back on the air by downloading Saga’s backup of the music library and data base, stored on drives and kept off site. The idea is to hope for the best, but prepare for the worst. Whether you back up to a drive at a sister station or use offsite storage in a cloud isn’t so important as actually sticking to a backup schedule. Always keep a running backup of your critical files and music, and make sure to use a secure link for the transfer. Better yet, advises Johnson, keep a mirrored image of your entire system on a drive located offsite, which will make it that much easier to get back on the air should the unthinkable happen.
How often should you back up files? Saga backs up music in 30 day intervals and commercials in daily intervals. The intervals are important. Less than 30 days, and they take the risk that they might back up malware along with the music library without knowing it. More than 30 days, and critical music files will not be included. Backup routines will vary station to station.
Segment your network. Relying strictly on the network’s DMZ to protect your station is like putting all your money in a room, and having one security guard protecting it. It’s far better to spread out your wealth – that is, group studios into separate segments each with edge switches, and then connect those segments to one central, redundant switch bank so that if one part of the system gets infected, the rest is isolated.
It’s worth noting that if you are connecting studios with our WheatNet-IP audio network, a lot of your wealth is already spread out in separate I/O access units we call BLADEs. Each BLADE has built in audio tools such as mixing for summing and mixing audio at any point in the network, plus will reconfigure itself in an emergency – and, in fact, can recover settings for your entire network! (BLADE-3s can also be ordered with audio playback option, which can automatically be triggered for playback by silence detection within the BLADE or triggered directly from your console.)
Restrict access. A few broadcast stations were hacked simply because they didn’t change equipment password from the default “password.” That happened in 2015. In 2017, we’ll need to get much, much better at restricting access. Most broadcast equipment manufacturers offer at least basic authentication protection, so use it. You should establish employee levels of access based on job function, from the presets on your Wheatstone console on out to the padlock on the door to the transmitter building and all major points of access in between. A word about passwords: complex passwords that include symbols as well as upper and lower case letters are good. But longer passwords are better because it’s harder to crack a 25-letter password than a 10-letter one using “brute force,” the common practice of using automation and a dictionary to hack into a network (see comic). What about complex, long passwords? According to Johnson, the problem with long and symbol-filled passwords is that they’re hard to remember and the risk goes up that someone will write them down somewhere. “Human factors override technology. It happened at Three Mile Island and it can happen at a radio station,” he says. For this reason, Johnson suggests using long passwords with simple, easy-to-remember words.
`
(note: thanks to http://xkcd.com/936/ for use of the comic for this article).
Sound Off
Someone once said that life would be so much easier if we only had the source code. We’ve been thinking about those words as we start a new year at Wheatstone, our 42nd, actually. This is the time of year when we tend to get caught up in new ideas, new products, and new ways of doing broadcasting. But we haven’t forgotten what’s important. The closest we’ve come to the source code at Wheatstone are the words of our customers. Here’s some of that wisdom, straight from the source.
On virtualization: There’s a lot to be said for being able to have a single studio, yet have the ability to take mics and other gear from one venue to another and control them from a distance. – Mickey Morgan, audio engineer, Eastern New Mexico University
On adding cameras in studios: It was discussed here, but that would require our morning show to wear pants! – Mark Spalding, CE, Saga Communications, Illinois.
On the latest network frontier: We're automating a lot more downloading of spots and programs and automatically ingesting them into our automation system. It’s all just networked in, and it’s no small deal for the cluster because there are 400 spots a week that have to be changed out here. It’s the better part of a fulltime job that someone doesn’t have to do. – Mike Maciejewski, Market Engineering Manager Townsquare Media, Michigan.
On contingency planning: Simple plan. We take the output of one of our BLADEs, and we map that to copper so we can still get through to our console and out to air if a core switch goes down. – Joe Schloss, Market Engineer, Saga Communications, Iowa.
On networking in general: Everything is networked today. In fact, I just networked in the audio processing this morning. – Mark Spalding, CE, Saga Communications, Illinois.
On connectivity: I think you’ll see a lot more people going fiber as the interfaces come down in price. – Mickey Morgan, audio engineer, Eastern New Mexico University
On mixing a remote feed from the studio: We’re doing it. The Free Beer & Hot Wings show does a lot of out-of-market remotes for promotions. We mix the show from here at the studio because we ran into so much trouble on the local side finding people to do it. And it’s easier to mix it here than having to mix in a roomful of loud people. – Mike Maciejewski, Market Engineering Manager Townsquare Media, Michigan.
On the reality of virtualization: I can send the morning crew out on a remote with Glass E on a laptop, and as long as we can jump to the console (in the studio) through a gateway VPN, operators can roll audio for the weather bed and do everything they need to do from the (studio) LX-24 (console). And, here’s the thing: I don’t have to bring in another operator to run the board for a three-hour show at a remote site. – Joe Schloss, Market Engineer, Saga Communications, Iowa.
On the expanding role of engineers: We’re pulling in more and more from the network. Automating spot downloads is just one example, and it’s great … until it doesn’t work, and then I have to deal with it. – Mike Maciejewski, Market Engineering Manager Townsquare Media, Michigan.
7-Year Old Edits Call with VoxPro!
Your IP Question Answered
Q: I’m looking into IP audio networking for mixing live remotes in our master studio, or what everyone calls “at-home live production.” Are there other applications for your WheatNet-IP audio system that might be useful?
A: Yes. Quite a few, actually. For example, the same I/O access unit (we call them BLADEs) that you normally use for at-home production can be used when covering live speeches down at the city hall. The news conference becomes another point on the network, so you can turn on the reporter’s mic and all the relevant processing settings – and even bring in another reporter located elsewhere in the network, plus set up mix-minuses for IFBs from one central location. You can place BLADEs in the van, on a stage or throughout the ballfield, and then connect them together over fiber or CAT6 via a network switch and transport audio between all those locations. The possibilities are endless, because each BLADE has routing, processing, mixing and logic controls for mics and other devices in one rack unit – sort of like a “studio in a box” that essentially extends your studio WheatNet-IP audio network, and all the routing and control that goes with it, to anywhere you need it. By the way, we also have a specialty network unit if you want to take advantage of 5.8 GHz unlicensed wireless IP radios as a line-of-sight link to the truck or studio. Our EDGE unit connects directly into the IP wireless radio through RJ-45 connectors to carry audio between locations. We can’t wait to hear from you about all the different ways you’re using the system!
Wheatstone
-
Zee Network (Dubai, UAE) purchased two LX-24 control surfaces with WheatNet-IP audio network and AirAura X1 audio processor through Horizon Avionics & Electronics.
-
WCCB-TV (Charlotte, NC) purchased an I/O BLADE and EDGE network unit for remote audio I/O to an existing Dimension Three TV audio console.
-
CBS (Philadelphia, PA) purchased a WheatNet-IP audio network system.
-
KTRS-AM (St. Louis, MO) purchased an LX-24 control surface, two L-8 control surfaces, a TS-22 talent station, five TS-4 talent stations and I/O BLADEs through BSW.
-
U.S. General Accountability Office (GAO) (Washington, DC) purchased a Series Four TV audio console, five L-12 control surfaces and a WheatNet-IP audio network through CEI.
-
Entercom (Gainesville, FL) purchased two LX-24 control surfaces, an IP-12 and IP-16 control surface and WheatNet-IP audio network.
-
C-Span (Washington, DC) purchased a TS-4 talent station, IP-MTR64 “wall of meters” GUI, and three I/O BLADEs for streaming audio from Capitol Hill.
-
WBMA-TV (Birmingham, AL) purchased an E-6 control surface and WheatNet-IP audio network.
-
WSB-AM (Atlanta, GA) purchased a SideBoard surface and two I/O BLADEs for an existing WheatNet-IP audio network.
-
CBC Radio (Toronto, ON) purchased a MADI BLADE for an existing WheatNet-IP audio network through Marketing Marc Vallee.
-
Leighton Broadcasting (Winona, MN) purchased an M4IP-USB four channel mic processor and I/O BLADE for an existing WheatNet-IP audio network.
-
Grove Broadcasting (Kingston, Jamaica) purchased an LX-24 control surface through Recording Media & Equipment.
-
Townsquare Media (Lafayette, LA) purchased additional I/O BLADEs for an STL.
-
KKDA-FM (Dallas, TX) purchased three LXE and two L-12 control surfaces and WheatNet-IP audio networking with ScreenBuilder app.
-
iHeartMedia (Atlanta, GA) purchased additional I/O BLADEs for an existing WheatNet-IP audio network.
-
Humber College (Toronto, ON) purchased an LXE control surface, TS-4 talent station and WheatNet-IP audio network I/O BLADEs through Ron Paley Broadcast.
-
Rogers Broadcasting (Toronto, ON) purchased two split-surface LXE control surfaces through Ron Paley Broadcast.
-
iHeartMedia (Charlotte, NC) purchased an LX-24 control surface and I/O BLADEs for an existing WheatNet-IP audio network.
Audioarts Engineering
-
Broadcast and Studio Company (Bangkok, Thailand) purchased an R-55e console.
-
Offerdahl Broadcast Service (Fosston, MN) purchased an Air-4 console.
-
WDBN-FM (Wrightsville, GA) purchased an Air-4 console.
-
Dixie State University (St. George, UT) purchased an Air-5 console.
-
Woof Boom Radio (Muncie, IN) purchased an Air-1 console.
Wheatstone Audio Processing
-
Pacific Media Group (Kahului, HI) purchased four FM-55 audio processors.
-
iHeartMedia (Charlotte, NC) purchased an M4IP-USB four channel mic processor.
-
Noalmark Broadcasting Corp. (El Dorado, AR) purchased an FM-55 audio processor.
-
Energy FM (Kingston, Jamaica) purchased an AirAura X3 audio processor.
-
Calhoun County (Blountstown, FL) purchased an M2 dual channel mic processor and WDM driver.
-
WKAK-FM (Albany, GA) purchased an FM-55 audio processor.
-
Cumulus Media (Wichita Falls, KS) purchased four FM-55 audio processors and three M2 dual channel mic processors.
-
Cumulus’ WYNN-FM (Florence, SC) purchased an FM-55 audio processor.
-
Cumulus’ KRBE-FM (Houston, TX) purchased two M2 dual channel mic processors.
-
Cumulus (Macon, GA) purchased an FM-55 audio processor.
-
Cumulus (Nashville, TN) purchased 18 M2 dual channel mic processors.
-
Cumulus’ WGLF-FM/WBZE-FM (Tallahassee, FL) purchased 10 M2 dual channel mic processors.
-
Cumulus (Harrisburg, PA) purchased 8 M2 dual channel mic processors.
-
iHeartMedia (Tucson, AZ) purchased an M1 mic processor.
VoxPro
-
KRLD-FM (Dallas, TX) purchased a VoxPro6 digital audio recorder/editor.
-
WBT-AM (Charlotte, NC) purchased a VoxPro6 digital audio recorder/editor.
-
WPLJ-FM (New York, NY) purchased a VoxPro6 digital audio recorder/editor.
-
Local Media San Diego (California) purchased a VoxPro6 digital audio recorder/editor.
-
KJWL-FM (Fresno, CA) purchased a VoxPro6 digital audio recorder/editor.
-
Entercom (New Orleans, LA) purchased a VoxPro6 digital audio recorder/editor.
-
Oakwood Broadcast (Mississauga, ON) purchased a VoxPro6 digital audio recorder/editor.
-
iHeartMedia (San Antonio, TX) upgraded to a VoxPro6 digital audio recorder/editor.
-
Humber College (Toronto, ON) purchased a VoxPro6 digital audio recorder/editor through Ron Paley Broadcast.