London, UK, 27 August 2020: Mo-Sys, world leader in precision camera tracking solutions for virtual studios and augmented reality, provided key technology to EVOKE Studios, which allowed it to develop exciting extended reality environments for the performances at this year’s AIM Awards, from the Association for Independent Music, on the 12th August 2020. Mo-Sys’ StarTracker provided the precision camera tracking data, enabling EVOKE to seamlessly blend (real-time compositing) live action with photo-realistic virtual elements.
“The challenges with extended reality lie in the smoothness of tracked content, frame delays, and having a close to faultless set extension,” said Vincent Steenhoek, founder of EVOKE Studios. “Our experience with StarTracker is that it gives us ultra-reliable, highly accurate positional data and ultra low latency. Building on technologies like StarTracker enables awards shows like the AIM Awards to be presented in broadcast quality virtual environments.”
Critical for virtual studio and augmented reality production is to track the position of each camera in three-dimensional space and with all 6 degrees of movement (pan, tilt, roll, x, y, z) plus lens focal length and focus. StarTracker from Mo-Sys is proven as the most precise and reliable camera tracking package, using dots on the studio ceiling (“stars”) which are placed at random and tracked to plot camera positions with extraordinary accuracy.
EVOKE and its creative partners shot a number of guest performances for the awards show, recording as live with no post production. The shoot took place at the new Deep Space studios at Creative Technology in Crawley, a studio which is already set up for StarTracker, including a camera jib for free movement.
Performances captured in extended reality included AJ Tracey and MoStack surrounded by larger-than-life burgers and fries, and Pioneer Award winner Little Simz who was made to appear underwater.
“This is a great example of what StarTracker delivers,” said Michael Geissler, CEO of Mo-Sys. “It is designed for live work, providing completely reliable positioning data into the graphics engines. It allowed the EVOKE team to build really complex extended reality and virtual environments in combination with LED walls and floor, then let the performers go with the music confident that they would capture all the action flawlessly.”
For more information on StarTracker, please visit the Mo-Sys website: www.mo-sys.com
CJP Broadcast Press Release: Ross-On-Wye, UK, 20 August 2020:
CJP Broadcast announces the successful completion of a video production and live streaming project for three of the highest profile games in the European sports calendar. A complete system centred on a CJP Live Sports Production System captured content to supplement terrestrial and satellite coverage of the events. CJP staff active at the matches included Managing Director Chris Phillips supervising technical setup, James Ruddock operating as Technical Manager, Kieron Sharpe and Kieran Phillips providing AV rigging support, Rob Dyton as Production Director and Chris Hollier as Remote Camera Operator.
“Covid-19 restrictions meant the venue was unable to host the capacity crowds normally present at semi-finals and finals,” Chris Phillips comments. “We were asked to augment the traditional broadcast coverage with behind-the-scenes content. This included manager reactions during the match, player interactions in the tunnel area and a focus on key international players. The resultant video would then be made available online for easy access by supporters during and after each match. Our role was to provide a complete production system plus an experienced installation and operations crew.”
“A key part of the challenge was capturing content from ‘red zone’ areas such as the substitutes’ bench and the technical control area. We provided four JVC cameras with motorised pan/tilt/zoom which were operated from our control base on the gantry in front of the press area. Two of the cameras were positioned in the tunnel. The other two were focused on the managers’ and subs’ benches. In addition to the robotic cameras, we had feeds from two Sony FS7 cameras provided by the host and operated by their own experienced freelancers, plus a JVC GY-HM660RE live streaming camcorder.”
“The Streamstar iPX allowed us to record ISO-style feeds from the six cameras and make these accessible in 15-minute segments to remotely located video editors so they could start producing final edited content while the game was still in progress. We expanded the 1 terabyte of onboard video storage in the iPX with a 16 terabyte GB Labs F-8 Studio recorder using a 512 gigabyte Nitro SSD Layer for ingest while editing. Content for editing was accessed from the F-8 by four edit suites. The system also generated an H.264 RTMP live stream for practically instant publication on social media. An HD/SDI clean programme feed was also provided from our system to the host broadcaster for use as an optional contribution feed within the terrestrial and satellite transmission.”
Available in several versions supporting up to eight camera inputs, the CJP Live Sports Production System provides a wide range production and streaming capabilities in an easily transportable unit. Its facilities include ISO recording, four-layer graphics, transitions, real-time replay, slow-motion replay, on-the-fly highlights creation, advert insertion, clip insertion and audio mixing. Operation is via a touchscreen and keyboard with the option of an external joystick for pan/tilt/zoom camera control. Full multiscreen monitoring facilities are included with the option of a second screen for commentator positions. An H.264 live stream can be fed directly to a TV station or third-party OB control suite via a 10 megabits per second link or via 4G mobile, with the ability to simulcast to multiple platforms and in-stadium screens. Up to 96 terabytes of RAID5 storage can also be attached.
About CJP Broadcast
CJP Broadcast Service Solutions Limited (www.cjp-bss.co.uk)) was established in 2011 to provide broadcast manufacturers and engineering companies with professional ITIL based service desk solutions. In 2016 the company expanded its portfolio to include digitisation of broadcast tape and film media to provide restoration of historical media archives into modern file-based formats. In 2018 CJP expanded its operation further, providing live production solutions, professional broadcast TV studio system integration and technical support services.
By Mike O'Connell | Pixel Power | Published 21st July 2020
Having lived with COVID-19 for a few months now, what have we in the broadcast industry learned? I can offer two take-home lessons.
First: we have to be a whole lot more agile than we ever thought. As broadcast engineers we have lived with the idea that a project takes a couple of years, from deciding what we want to do, through meeting vendors, maybe a proof of concept or two, then on to a final spec, installation, testing, training and rehearsals.
In 2020 we have had to roll out emergency solutions to keep our stations on the air in hours, not months or years.
Second, we have to have a much more flexible and perhaps bolder attitude to finance. An important part of that two year project timescale would have included an outline budget for first stage approval, then refining the project costs through to final contract.
But if we need a solution up and running this afternoon, we need to be able to pay for it by tomorrow – or maybe even this morning. The death of the capital budget is another transformative change that COVID-19 has brought to our lives. Opex is another of those buzzwords that has been around the industry for a while, without ever gaining too much traction. Inertia ruled and since broadcasters have always worked on capex it was felt no reason to change. Well now there is a reason.
First and foremost, Opex provides a tight alignment to output. Whether it is 3D graphics or a virtualized software only playout channel, an enlightened vendor and pay-as-you-play cloud hosting, you should pay for only the functions you need, when you need them. Pixel Power has allowed you to buy licenses outright, or by time, or by output for several years – a pure Opex model right here and right now.
Capex encourages you to have equipment sitting in the rack in case you need the facilities. While sitting in the rack that equipment is taking up space, power and air-conditioning even if it is not giving you an output and earning you money. If you only use 3D graphics for Saturday sport, why pay for the machine to be running the other six and a half days when it’s not working?
That principle applies on a bigger scale. We have a presidential election this coming November (although thanks to the pandemic, it is going to be a strange one). Do we really want to spend a lot of capital on facilities that, after November 3rd, we are going to have to find work for?
The other advantage of Opex is that the money can be found from multiple sources. If you need specific functionality – like complex graphics for an election – you can fund the licenses, for precisely when you need them, through the production budget.
The consensus of opinion during this pandemic seems to be that we may run out of content in August. To fill the gap, we are going to have to do things differently because we simply can’t have people working physically closely together any more.
If we need new equipment and techniques for a short time (we all sincerely hope that the pandemic will be over soon) then it makes no sense to spend capital on temporary equipment and facilities. And if it makes no sense now, is it not time to make Opex the primary way forward?
By Mike O'Connell | Pixel Power | Published 8th July 2020
As an industry we have talked about disaster recovery for a while. We know about business continuity, about how our audiences – not to mention our revenues – will disappear so fast if we drop off the air.
The problem is, most of us have been talking about the wrong sort of disaster. Received wisdom has been that, if our building catches fire or there is a gas leak up the street, we need to be able to relocate (our staff) to somewhere a distance away, where we have a duplicate set of equipment to keep us on air.
Well, now the disaster has come. And it is nothing like the disaster we prepared for.
Our buildings and our technical installations are all there in the different location. But, thanks to COVID-19, we can’t get our people there! As those who rely on traditional broadcast hardware technology for playout have found, it really cannot operate hands-off. It relies upon people.
At Pixel Power we have been talking about and deploying the solution to this for several years. It is virtualizable, software defined playout. You build exactly the production or playout automation workflows you need, from elemental, modular components. These software modules sit on standard IT hardware (Commercial and Off The Shelf, COTS), and are slickly managed to deliver precisely to your requirements.
Why is this so important? Because it leads to near-perfect protection from the unexpected, like COVID-19 – when you need to flip your operators to work remotely, from home.
A virtualized architecture can run on dedicated hardware in your broadcast machine room. Or it can take the processor and storage requirements it requires while residing in your corporate data centre. Or it can live entirely in the public cloud. Or it can exist across any combination of these scenarios.
Playout in the cloud means you never have to attend to the hardware. You can control every aspect of your output – even live interventions on premium channels – from a laptop, wherever you have an internet connection.
A lot of people are now talking about virtualized playout from the public cloud. Indeed, it’s great to welcome some of our industry rivals to the club. At NAB 2016 Pixel Power demonstrated premium channel playout, including live interventions, with sophisticated graphics & branding, hosted entirely in the AWS cloud. We did it in 2017, 2018 and 2019, too, and would have made it five in a row if NAB2020 had happened. But regardless of big tradeshows happening, the beauty is that we can demonstrate this to you right here, right now – because it’s ready in the AWS cloud.
Other vendors still view this as science fair stuff. We have this nailed. It is proven and it is delivered.
And it gives you flexibility. For those broadcasters who want to retain the traditional infrastructures, your primary playout can be in your building, with disaster recovery in the cloud. Or you can make the jump and put everything in the cloud, relying on AWS or whichever vendor you choose to sort out the geo-diverse protections for non-stop service.
No-one predicted the extraordinary impact COVID-19 has made on all of our lives. Who knows what’s next. It is your responsibility to put the best possible business continuity plans in place. We can help you make that happen.
By James Gilbert | Pixel Power | Published 5th June 2020
It is almost compulsory to start any blog at the moment by saying these are difficult times. What I mean is that we are having to review so many things to work with isolation and social distancing, while ensuring a minimal impact on what we have always been proud to call broadcast quality and production values. But ‘luck’ is when preparation meets opportunity and in the last few years we have been in the fortunate position to help our customers prepare well.
We have had to quickly find ways to work remotely, whether that is in production or in delivery. Remote control of playout automation is an obvious requirement, and that has focused a very harsh light on legacy systems, many of which simply cannot be managed at a distance. Virtualised, software-centric technology is capable of control from anywhere, by its very nature, from a remote facility or a home office.
Broadcasters need new interstitials, promoting new programming created in lockdown, and offering public service information. With on-screen talent working from less than ideal environments at home, good graphics are more important than ever. An ideal system would see an editor working from home create a template which an automated production platform could then populate into all the different versions required.
Obviously I am talking about the technology for which Pixel Power is famous. But the same ideas apply across the board. Broadcasters have to find new ways of working. To succeed, that depends upon two linked factors.
First and most obvious, they need this technology now. We don’t have the luxury of six months research and extensive meetings at NAB or IBC (we didn’t have NAB and no longer have IBC!). We want something that works now – today – that may not work the way we used to, but which will deliver the goods.
That in turn leads to the second point, which is that you have to be able to rely on the vendors supplying this kit. If you are not going to spend months in discussions and developing proof of concept solutions, you have to have 100% faith that what is proposed will do what you want.
You need a vendor that understands your business. We started Pixel Power in 1987 to develop broadcast graphics systems then moving on to master control, automation and playout. 33 years later the company still has a rock solid dedication to solely broadcast solutions. To ensure we really understand what broadcasters and content companies need, we recruit key staff from them: people like Toria Farrell, a former transmission controller and Malorie Delaporte, a former head of systems engineering.
For more than 30 years Pixel Power was a thriving independent business that in 2018 became part of Rohde & Schwarz – another privately owned, engineering led organisation. We share the same values of always working to understand our customers’ businesses and delivering the best engineered solutions whether software or hardware. The combined result of engineering stability allows us to serve our customers with the right technology that helps them move forward: creatively, technically and commercially.
It is too early today to say what the new normal will be. But it is clear that, for broadcasting and media streaming to recover, it will need to be more agile and able to respond rapidly to change. It will rely much more heavily on automation, remote operation in the first instance and later virtualization. Preparing for that new world of opportunity will demand software defined solutions and ever stronger bonds of trust between vendor and customer, to implement effective solutions quickly and accurately.
We understand that, and we are here. Whether you need a quick fix or a long-term strategy, talk to Pixel Power.
By Nikhil Pereira | Digital Studio ME | Published 27th February 2020
What are your plans for CABSAT 2020?
The great news is that both Pixel Power and Rohde & Schwarz are together on the same booth at CABSAT. We will be there to talk to customers about all our solutions in studio production, post production, delivery and distribution.
Give us more insight into the products and services you will be exhibiting at the show?
We are showing the following areas: Create and StreamMaster Produce graphics creation tools — with a 30 year heritage in graphics creation Pixel Power can show you how your brand can come to life, how you can automate your branding workflows and how to create a stunning new look and feel for both premium channels and across all your channel portfolio. StreamMaster Deliver (channel-in-a-box) and Gallium Playout (complete workflow automation for playout) will be one of the prominent demos on the booth. Full broadcast chain monitoring solution and multiviewing software using the Rohde & Schwarz Prismon solution set. Satellite amplifiers will be the star of the show with the new PKU-100 satellite uplink amplifier.
Who is your target audience and what should they look out for in your offerings?
We are looking for customers interested in innovative solutions where they want to save money or make money in their broadcast network. Customers who are interested in workflow solutions, future proof software defined tools that can scale and change with their business.
What is the USP of your products?
There are many unique selling points of our products and services but one of the most important things to note about Pixel Power and Rohde & Schwarz is that we focus on partnering with our customers throughout the whole journey from first discussion about a new project right through to the ongoing creative, operational and technical support over many years. We are an engineering led business and driven by the desire to deliver great solutions for great broadcasters for great content.
What challenges is the market facing in the market sector your firm operates in?
“Buzz” and the need to be realistic. There is a lot of talk about new standards, new resolutions, new consumer features. But the reality is that most broadcasters don’t simply want to hear about the dream, they want to see the reality of how they can create a steady pathway to that dream over a period of time. In the past decades we have had the ‘time in motion’ studies, the ‘just in time’ movement, the ‘time to market’ period. In the 2020s I believe it’s the ‘time to trust’ decade.
What are your plans for 2020?
We will keep developing and delivering innovative solutions for our broadcaster and telecommunication customers who want to create and deliver exciting television programmes and movies — then help them deliver that to you and me on our screens at home or on the move, in the cinema or the stadium, to enjoy with family and friends.
By James Gilbert | KitPlus Magazine | Published January 2020 Edition
The first days of 2020 seem like a good time to reflect on what will happen to our industry in the coming months and years.
There are some things about which we can do nothing. The global political situation seems to be in turmoil. Populists seem to stop at nothing, and are certainly not hampered by the need to be truthful; the Middle East is in turmoil (growing rapidly worse as this article takes shape); and of course we still have no idea what the impact will be of Brexit, both for British companies and those based in the EU. All of these and more will be affecting the decisions of vendor companies every day in 2020 and beyond.
But what of our industry? The first point to make is that we are now in the end stage of the transition from broadcasting to content delivery, from linear television to content everywhere. The new model demands much more of delivery systems, yet technical budgets are not growing in step. Indeed, they are shrinking, so content companies are going to be looking for ever-more clever solutions.
Thanks to the growing power of COTS hardware and well-thought through software-defined architectures, both the technology and the solutions are available. The challenge relates to ‘people’.
First, the users of the technology need to buy into the new workflows and operational practices. Simply like-for-like replacement of a playout chain with a software-defined equivalent is not enough: it is definitely not a clever solution. We need to work together to create smarter ways to work, that take advantage of the new opportunities.
Secondly, we are not a young industry. It is something we regularly say, but we need to offer encouragement for the best people to join us. That has to start with STEM initiatives at school.
For all the talk of software-centric solutions, we will still need “broadcast engineers” in the future. It is about more than simply writing code. You may be able to add functionality to social media applications, or secure banking log-ons, but you still need a special set of skills and understanding to engineer a live sports broadcast. 2020 sees the Olympics and Paralympics: there will be very many engineers, with very many years of experience to their names, heading for Tokyo to deliver the quality and intensity of coverage that audiences demand.
Japanese broadcaster NHK will be claiming the headlines for its 8k coverage. But it is also important to remember that in large parts of the world SD is still the norm, and very few ordinary people will watch in anything above HD.
Just as the move from SDI to IP has proved slower than people predicted, this underlines a simple truism we should all remember: if you don’t need to be cutting edge, don’t make life difficult. If your SDI installation works, do not feel the need to throw it out any time soon, to replace it with an infrastructure which does not perform the way you expect it, and which needs additional skills which are still in very short supply.
That also applies to other new technologies for which wild claims have been made but have failed to deliver. There is a growing realisation that the idea of using the public cloud for large swathes of broadcast applications is a real case of the emperor’s new clothes.
That is not what the cloud is for; not what it is good at. By all means, set up disaster recovery in the cloud, to sit there costing very little unless the dreaded day should arrive. But for many customers 24/7 playout in the cloud really does not stand up to sensible financial and commercial analysis.
Having said all that, I am very optimistic for the future. We see good reasons for excitement in 2020, not least a tangible sales pipeline and a successful continuing recruitment programme. We are investing in a stronger team to support the delivery of better solutions.
I know we have been talking about solutions for many years, but it is truer today than ever that media companies want to buy working systems, not point products. They simply do not have the staff to decide what they need to do and evaluate the marketplace.
But systems integrators have to change, too. They are no longer stack and rack merchants, judged on the neatness of their wiring looms above all else. Today’s integrators have to manage the orchestration of the technology into the workflows that are going to deliver commercial and operational benefits.
Broadcasters and content deliverers still want to feel that they are buying the best of breed, and certainly do not want to get locked in to single-vendor solutions. So the need is for smart, capable integrators who can develop this orchestration, creating original software as well as configuring products from multiple vendors.
For companies like Pixel Power, that means we have to have strong resources to support integrators, helping them get the best out of our technology to achieve the desired goals of the end customer.
That, in turn, means that successful businesses will need a certain scale. Pixel Power became part of the Rohde & Schwarz group over a year ago now, and continues to retain its own identity. That seems to me an ideal solution: we have big company resources while retaining specialist skills.
2020 is definitely going to be an interesting year. I hope you are looking forward to it as much as I am.
By Adrian Pennington | IBC 365 | Published 5th March 2020
As more media organisations enforce home working, could the spread of the coronavirus hasten the transition to remote production?
While the coronavirus continues to wreak havoc and hysteria across the globe, the epidemic has accelerated the deployment of remote-working software to such an extent that many businesses may never look back.
“What has changed in the last few weeks is that working remotely is no longer a work-life balance argument, or a nice-to-have, it is now a question of business continuity,” says Daniella Weigner, owner, Cinegy. “Crisis is forcing change right now. This is a catalyst. It is also a major opportunity to get change done.”
By Adrian Pennington | The Broadcast Bridge | Published 15th May 2020
Playout automation has been enabling fewer people to control more channels for decades but we’re not quite at the point where human interaction can be eliminated altogether. Since most linear broadcasters will either move to a software-based deployment for their channels themselves or give them to a service provider that carries out that transformation for them. The first of a two part article assesses the layout and establishes the benefits of software playout and MCR operations.
For more static or thematic channels which are exclusively scheduled by traffic there is less need for an operator to actively ‘run’ things, but operators will still need experienced staff around to react if there is a problem, or to manage breaking news which would interrupt the schedule.
In the playout environment of managed services provider Red Bee Media, automation is used to assist in the execution of repetitive operational processes, so that human interaction is focused on exceptions to the norm or on complex tasks that humans are simply better at carrying out.
“There are some historically manual operational tasks that now happen entirely automatically and there are some channels that, providing certain rules are followed, can be run with a very light touch,” explains Red Bee Media's Richard Cranefield, Head of Product for Playout Services. “But as many of the channels we deliver inherently need to accommodate multiple live events and late changing commercial content, we need to retain an ability to deal with unpredictability and react quickly to complex and altering scenarios. For that reason, we retain our highly experienced playout staff but make sure they have the support of automated software tools in order to raise the reliability of our output.
Within operations, automation is less about removing people from the equation and more about supporting them in delivering a quality output. It is at the beginning of a new channel’s life, in its deployment, that automation has had most impact on the workforce.
In Red Bee Media’s MCR, automation is used primarily for control and monitoring of acquisition, routing, processing, and distribution whereas it still uses ‘hands and eyes’ for broadcast support, project support, 24/7 service desk and remote assistance with events, studios and news teams.
“Our MCR (and most of the broadcasting industry) is now in a hybrid state, supporting both traditional SDI and IP routing under software-defined control and monitoring technology platforms,” says Kristian Langbridge, Head of Distribution Services, Red Bee Media. “To handle both, you need edge devices capable of converting between the two formats. We won’t be able to fully convert to a software-defined set-up until SDI becomes a legacy format.”
Leaving that aside for one moment, the move to software defined playout and master control is already underway. James Gilbert who co-founded Pixel Power, thinks the lockdown will focus people’s attention on the timing of investments.
“The crisis will adjust the list of requirements for operators shopping for a system. The ability to operate remotely has not been high up the list of considerations until now.”
In the current situation we find ourselves in, a software only approach makes it easier to maintain playout, even if staff need to vacate the building in an emergency.
Evertz reports a number of its customers performing playout from home, which with a hardware-based solution, is just not possible.
All playout systems vendors claim to be software defined and have been tracking this way in their tech development for five or six years. The benefits of a software only approach are just too strong to ignore.
“Upgrades can be effected more quickly and systems are more scalable, enabling broadcasters and media companies to be more reactive to business changes. Integration with third party systems is usually more effective,” says Daniel Robinson, Head of R&D, Pebble Beach. “For solutions which are deployed on virtual machines, the host hardware can be shared with other applications and reused for different applications if, for example, a virtualised channel is decommissioned at the end of a season or event.”
Monitoring by exception can be hugely helpful in enabling efficient operations across multiple channels, reducing headcount whilst not removing the opportunity for human intervention altogether. Having a consolidated view across all channels of any upcoming errors or missing media means that a single operator is presented only with the information that requires attention, enabling them to take remedial action as soon as an issue arises.
Time to market is a key factor. For operators like Red Bee, the benefits of automation and software deployments are felt more at the beginning of a channel’s life. This is not an inconsequential impact. Historically the launch costs of starting a channel, simply in manpower, could be half of the effort dedicated to a basic channel over a five-year period.
When channel infrastructure is built out of bespoke appliances they needed to be unboxed, racked, wired, configured, and tested. At a rough estimate Red Bee reckons an investment of 400 days of effort to get a channel from idea to on-air. In contrast, a refined software deployment can cover the same ground in less than a day.
“Scripts can now launch replicas of a known good channel configuration, for example,” says Cranefield. “Software emulations of appliances from multiple vendors are automatically connected across the existing network and end-to-end testing is also partly automated. We don’t wait for boxes to arrive, no wiremen go into the racks room, there’s no SDI router to expand or appliances to soak test.”
Moving to software only affords a greater range of service flexibility but operational spaces are obviously still physical. In many cases the aim is to emulate the appliance-based way of working in a software world without impacting the operational user experience.
In the MCR, software provides the ability to scale up quickly across private and public cloud infrastructure, add new services and new features without having to refresh expensive hardware.
“It will inevitably reduce the monolithic software stack of many of today’s broadcast solutions into micro services, licensed on a pay-as-you-go basis,” says Langbridge. “This will drive increased competition and ultimately reduce costs for the broadcasters.”
What automation and software deployments have enabled Red Bee Media to do is get greater efficiency and quality out of these operational spaces and the teams that operate within them. Cranefield says that these spaces now have a far less machinery deployed within them, making the rooms cheaper to cool, less noisy and therefore also more pleasant places to work in.
The quality of production output can also be expected to increase as a result of using software only tools. Evertz says its software-defined systems permit customers to increase from one operator per channel, up to as high as one operator per 50 channels, with the human operator providing a safety net to ensure programming is running. That cost saving can be returned back in to the content production, presentation or acquisition.
“It’s worth noting, that the greatest return on investment is when our Mediator-X solution is used to deal with the entire content factory,” says Martin Whittaker, Technical Product Director, Evertz. “That process starts right from the beginning, whether its ingesting or managing production edits or scripts, integrating with Avid and Premiere Pro, processing content through third party automated checks, utilising AI and ML where possible, even producing, managing and delivering packages for VOD and Direct to Consumer platforms.”
By Adrian Pennington | The Broadcast Bridge | Published 22nd May 2020
Playout automation has been enabling fewer people to control more channels for decades but we’re not quite at the point where human interaction can be eliminated altogether. Since most linear broadcasters will either move to a software-based deployment for their channels themselves or give them to a service provider that carries out that transformation for them, The Broadcast Bridge assesses the benefits and the challenges in so doing. Part II examines the crucial role of IP and the workflows and skillsets needed to operate such infrastructure.
The major premise of software-defined operations is to consign proprietary and hard to interoperate equipment and siloed workflows to history.
IP, and especially uncompressed IP, is the stepping stone. That said, the vast majority of playout infrastructures are still SDI, and a baseband solution is inherently incapable of being software only.
“Key to the transition will be the widespread adoption of open standards which enable interoperability between different vendors’ solutions in the IP environment,” says Daniel Robinson, Head of R&D, Pebble Beach.
Initiatives from AMWA, the Advanced Media Workflow Association who are developing NMOS (Networked Media Open Specification) are helping to drive this forward.
James Gilbert, Pixel Power, agrees, “We need standards and ST2110 only touches the surface of what is really needed to have software-defined best of breed systems in which you can connect video between different vendor’s products. What is not standardised is the control layer and that needs a big amount of work although AMWA and NMOS are working in the right direction.
“Inevitably there will be certain pieces of the puzzle which will stay with the vendor since it will be impractical and inefficient to open it up completely.”
There are many benefits to be gained from deploying a channel in the cloud. For service providers, MCOs, sports broadcasters, and corporates, virtualised playout can deliver an affordable option to deploy or contract IP-based channels instantly without the burden of racks of complicated hardware, and weeks or months of setup and provisioning.
But questions remain about the economic, logistical and technical benefits to the end user, and judging by the high volume of on-premises playout solutions that Pebble Beach install and commission – whether IP or baseband - it’s clear that this path is not one that every broadcaster or media company is ready to follow.
“Adding a virtualised infrastructure adds an extra layer of complexity and specific new requirements into the mix,” explains Robinson. “Don’t underestimate the level of in-house expertise you will need access to in order to implement a full-scale virtualised platform. Make no mistake, you will need to understand every nut and bolt of your virtual environment. In the more traditional set-up you will own the playout device and the vendor will take full responsibility for how that device performs, what benchmarks it complies with etc.
“However, with a virtualised solution, the vendor is simply the software provider, meaning that you, or your nominated representative, have responsibility for the overall performance of the virtualised platform and networks.”
Robinson adds that buying a bare metal box, a certain amount of RAM and a number of CPU cores will give you a reasonably predictable performance under given circumstances, but when you put your application on to a hypervisor, you are adding a whole new layer of software between you and the hardware which has a potentially huge number of ‘tweakables’.
“Don’t forget to check that your chosen hypervisor supports the disk drives and storage you want to use with your COTS hardware. If you need to change your hypervisor will your hardware be supported?”
Failure scenarios and failover contingencies needs to be considered. Who or what will be switching IP streams? If your VM fails, you may lose the transport stream altogether. Can your downstream distribution deal with no stream at all? Where are your IP streams going? Can you test them?
Shift in skillsets and workflows
Moving to any new workflow, whether it is an all IP or an all software architecture, does require new training and/or skill sets. Gilbert finds that operators are usually on board with any change in working practice.
“We don’t encounter resistance to that and part of that secret is involving them in procurement process so they can look at alternate solutions and feedback.
That’s not to say that the lines between traditional broadcast and IT aren’t blurring. In future there will be no distinction. In the past you needed technicians who understood how to hook up video signal and monitor Tektronix scopes. Now its Dev Ops and agile scrum developments to orchestrate solutions from different manufacturers. They need to understand the language of rest APIs as well as how to operate a scope.”
In the MCR, broadcast engineering competence will shift towards Python and C suite, high bit rate media transport technologies SMPTE 2110 / 2022, TCP/IP, and containerisation.
Such skills may simply be part and parcel of the incoming workforce. “Software defined technology will help attract young generations into the industry,” Gilbert says.
Aside from client training, Evertz have also tried to eliminate the learning curve, by creating UIs and feature sets to provide something that feels the same as operators are used to.
“For example, when we come to routing in a MCR environment, Evertz MAGNUM and VUE provides the same source, destination take familiarity, even if in the background its routing feeds up and down from the public cloud,” says Martin Whittaker, Technical Product Director, Evertz.
It is hard to be specific about what workflows might emerge but we can say that workflows are no longer set in stone and they evolve constantly and iteratively. We are now in a world where we are continuously tweaking and refining our operational practice and customer experience.
“Leveraging faster release cycles from the vendors we partner with as well as carrying out a significant amount of development in-house allows us to solve small issues with quick feature releases or small applications,” says Richard Cranefield, Head of Product for Playout Services, Red Bee Media.
It’s important to note that you won’t just need to measure the behaviour of the playout software application; you also need to monitor the behaviour of the entire infrastructure. Simply verifying that video and audio are playing does not give you the full picture. The range of available monitoring options in an SDI environment usually far exceeds those available in the IP domain. Diagnostics can be harder for IP too, so you’ll need to investigate what tools are at your disposal, as well as staff who are able to interpret the results.
“Operational monitoring is also critically important, especially in public cloud scenarios,” Robinson alerts. “As well as monitoring latencies and considering how and where your operators will monitor the playout, you need to consider any control latencies that will need to be added. Playout automation may need to send out control commands taking into account the monitoring latency for the user.”
The transport streams a playout infrastructure generates will go through the enterprise network switches and so can overload the network bandwidth potentially impacting on-air performance. That’s despite the fact that the playout software application may be running on a completely separate network.
Robinson’s advice is to check that the playout software vendor will give you access to the raw data that shows how the application is really performing on the virtualised platform. Among the multitude of parameters that can be measured, the sleep/wake time of processors of certain hypervisors may not be good enough for real time playout, he says. Latencies and behaviour will vary depending on the hypervisor you test.
The principal barriers to achieving a transition to truly virtualised playout and MCR operation are time and expertise, according to Red Bee Media which has gone through the process. Its platform is entirely software-based, including multi-viewers, and alarms and monitoring.
“We believe that you have to emulate all of your legacy appliances in software and not just some of them,” reports Cranefield. “To do so has meant integrating emerging technologies from multiple vendors and taking on responsibility for the performance of the hardware that those technologies now run on. In taking software from broadcast vendors and then running on our own cloud we’ve taken on responsibility for the fabric that everything runs on. The cloud infrastructure is now the part of the system that would have been the PCB in an old-world appliance. Who is accountable for a ‘device’ not working is now much more blurred, so we have had to upskill our engineering and network teams to deal with a greater level of responsibility in keeping the platform working, or understanding why parts of it are not.”
The R&D project that got Red Bee to this position lasted two years and was deliberately not pegged to any customer projects until they’d nailed it.
“Many broadcasters who want to undertake software transformation for themselves will still have to do this work, but they may struggle to dedicate the time and cost of developing the intellectual property required to get it right and can only leverage that investment over their own channels,” he says.