Senior Production And Tech Support Engineer

Summary

Join NBCUniversal's Production Software Engineering (PSE) team as a Senior Production & Technology Support Engineer. You will provide crucial 24/7 support for virtualized production infrastructure and applications across various NBCU networks. Responsibilities include troubleshooting, resolving user issues, collaborating with cross-functional teams, and developing SOPs. This role demands extensive experience in supporting large-scale applications, proficiency in agile methodologies and ITSM tools, and a deep understanding of broadcast technologies. The position requires weekend and holiday work and offers a competitive salary and comprehensive benefits package. The ideal candidate is highly responsive, organized, and possesses strong problem-solving and communication skills.

Requirements

• 5+ years of experience supporting large-scale, mission-critical applications
• Proficient in agile, Github, and ITSM tools like Auto Now Service Now
• Experience troubleshooting a broad range of applications, desktop & mobile operating systems, browsers, etc
• Must have experience working directly with software development teams in a Continuous Delivery and SDLC environment
• Knowledge of a broad range of enterprise technologies
• Must have ability to review and interpret application, infrastructure, and service logs
• Must have basic knowledge of AWS services (IAM, EC2, RDS, S3, etc.)
• Fluent in Microsoft Office applications, including Outlook, Teams, Word, Excel, etc
• Must be willing to work evenings, overnights, weekends, holidays with little or/no notice in support of 24x7x365 critical system availability
• Must be willing to travel on an as-needed basis
• Experience within a News or Television Production Environment / Media & Entertainment a must
• Knowledge of cloud providers (AWS, Azure, GCP) a plus
• Fully Remote: This position has been designated as fully remote, meaning that the position is expected to contribute from a non-NBCUniversal worksite, most commonly an employee’s residence

Responsibilities

• Provide Support to Virtualized Productions and Production Software Applications Support live programming for NBCU Local, MSNBC, CNBC, TODAY, Nightly News, Dateline, Telemundo, and other NBCU shows and networks
• Responsible for the support of live production operations within the facility for on-air production
• Respond to and resolve reported issues from users and L1 using best practices to ensure production reliability
• Communicate and escalate issues to L3 support & engineering managers as needed
• Passionate about the production process, excited by new technologies and opportunities and ready to help develop and improve the existing process and infrastructure
• Self-directed ownership of tasks and assignments
• Able to provide adequate status updates to senior and executive leadership communicating both problems and solutions to technical and non-technical personnel alike to help develop new processes, technologies, and techniques, which will help to improve the funning of live on air shows and productions
• Validation of Software Deployments: Assist and collaborate with the development team in validating new platform software releases
• Identify bugs and workflow gaps through operational testing
• Documentation and Workflow Development: Identify and resolve gaps in documentation
• Create Standard Operating Procedure (SOP) documents for efficient issue resolution for use by technical operations and other support staff to resolve issues efficiently. Regulates, documents and implements Standard Operating Procedures (SOPs)
• Create new SOPs and operational guidelines and train others in these SOPs, issue resolution, and recovery
• Researching and documenting production issues or potential on-air discrepancies- providing detailed write-ups of occurrences resulting in potential outages or degraded performances
• Application Support: Monitor incoming user issues (tickets and/or emails) and provide hands-on application support and troubleshooting for 24x7x365 virtualized productions and production software applications, escalating to the appropriate group when required staying in alignment with set SLAs
• Collaborate with cross-functional teams to define support SOPs and create /define playbook documentation. Including 3rd party vendors
• Performs routine system updates, security patching, collaboration with various tech teams regarding planned maintenance, and provides on call 24x7x365 on-call coverage
• 24x7x365 availability for production outages, emergencies, deployments, and escalations
• Collaboration and Coordination: Work closely with leaders to coordinate maintenance and build activities around production schedules
• Collaborate with various teams, including Scrum Leads, Software Engineers, QA Engineers, Product Manager, Project Managers, Producers, Operational Partners (Contractors and/or Staff) and third-party vendors
• Works with product /business engagement teams to learn business objectives, development teams to plan support needs, QA to understand test strategy, and SRE on environments, deployments, and regular maintenance
• Participates in Scrums, demos, and other Agile ceremonies and ensures accurate and timely status updates to the team
• Hands-on support and documentation for all on-prem & cloud (AWS) environments (Servers, Storage, Databases, Networking, Security, etc.)
• Observes and responds to all application, infrastructure, and services monitoring alerts and dashboards
• Recognizes and identifies potential operational impacts of application or enterprise change controls which could affect our applications and customers
• Helps improve performance, scalability, and reliability
• Performs routine system updates, security patching, collaboration with various tech teams regarding planned maintenance, and provides on call 24x7x365 on-call coverage
• 24x7x365 availability for production outages, emergencies, deployments, and escalations
• Outage Response and Root Cause Analysis: Own outage response, management, resolution, and root cause analysis. Incident, Service, Knowledge & Change Management
• Establish subject matter expertise for all applications and workflows
• 24x7x365 availability for production outages, emergencies, deployments, and escalations
• Metrics and Continuous Improvement: Implements gathering and utilization of metrics to measure accomplishments and monitor progress, ensuring SLAs are met and customers are satisfied with their support experience to measure accomplishments and monitor progress
• Contribute to process improvement ideas and support innovation

Preferred Qualifications

• Knowledge of NoSQL data stores (MarkLogic, MongoDB, Cassandra, DynamoDB, Couchbase, PostgreSQL, etc.) a plus
• Knowledge on all broadcasting equipment and included but not limited to Production Control Switchers, Broadcast Routers, CG, DV, Audio and Video Mixers, RTS, Riddle, Clearcom, EVS, Icon Stations, Ross XPression, Tessera / Graphite / Carbonite, Dreamcathcer, Media Asset Management (NAM, DAM), Encoders/ Decoders, Rasterizers, Evertz AIR Cleaners, Motion, FTP client and Omneon Spectrum, Versio, Vue Routers, Nexio, Utah routers, LiveU, TVU, Evertz Magnum UI, EXQ and EXE routers
• Strong Knowledge of automation and IT systems. (Automations: ITX, Harris, Morpheus and Mediator X)
• Well-versed on Broadcast operations and procedural requirements to confirm accuracy and on-air readiness of scheduled material by checking the IRD, Traffic log and automation playlists. Creation of VOD for streaming platforms
• Troubleshooting Audio, Video and other technical issues that may arise in the uplink/downlink signal path and be able to recover quickly and effectively
• Optimizing workflows related to all broadcast systems including linear and IP fabric for video infrastructures, virtualized Windows and Linux environments, Encode / Decode platforms (software / hardware), newsgathering tools, Non-Linear Editing, Distribution/ Transmission systems
• Familiarity with AWS services and cloud providers is a plus
• Responsive and organized, with a high tolerance for stressful situations
• Inclusive and collaborative approach with strong interpersonal and communication skills
• Detail-oriented and resourceful with a drive to learn
• Strong problem-solving and written communication skills
• Flexibility to work according to production software engineering schedules, including evenings, overnights, weekends, and holidays
• Passion for monitoring, metrics, and continuous improvement
• Extremely responsive and organized, letting nothing fall through the cracks
• Solves problems quickly and automates processes for the future
• Knows when to place a call to a user rather than continuing a long email thread or chat
• Ability to take responsibility, jump in and get things done, and be self-directed with limited supervision
• Ability to recognize priorities and pivot activities accordingly. Ability to focus on deadlines and deliverables. Ability to work and participate in a team with an open and collaborative style of communication
• Demonstrated self-composure, high tolerance for stressful situations, and unquestionable integrity under pressure
• Strong interpersonal and communication skills with ability to effectively speak with people at all technical levels, from CTOs and CIOs to users and business stakeholders
• Highly customer focused and very responsive. Listens, understands and acts
• An approach that is inclusive of other viewpoints and perspectives; confidence to make tough decisions about difficult issues with limited data
• Big Picture thinker
• Demonstrated ability to work independently as well as part of a cross-functional team
• Takes accountability for action items and requests even those requiring actions outside of their direct control, following up and leaving nothing to chance

Benefits

• Medical, dental and vision insurance
• 401(k)
• Paid leave
• Tuition reimbursement
• A variety of other discounts and perks