Augmented Reality Is on the Verge of Reality in the Electric Power Industry
It wasn’t just Arnold Schwarzenegger’s classic one-liners (“I’ll be back” and “Hasta la vista, baby”) that made 1984’s Terminator such a huge blockbuster. Also driving its quick ascension into the cultural zeitgeist was the futuristic technology built into Schwarzenegger’s cyborg character.
In one memorable scene, the audience experiences Schwarzenegger’s point of view as he approaches and enters a biker bar and looks at motorcycles, people, and other objects. A red-tinted screen is filled with real-time information describing the objects he encounters—including their height and weight—along with quantitative assessments that, presumably, only a cyborg could understand.
Three decades later, augmented reality is moving steadily from science fiction to becoming a tool that utilities can use to work more efficiently, cost-effectively, and safely, and EPRI is leading a multifaceted research effort to advance the technology.
“When Schwarzenegger walks into that biker bar and information is overlaid on what he sees, it’s spooky how close that is to what we envision right now in the electric power industry,” said Matt Buck, an EPRI senior technical leader who is developing digital and augmented reality tools to bolster power plant operations and maintenance. “Here’s what we’re imagining for plant technicians: They view equipment through augmented reality–enabled smart glasses, which automatically overlay information about the components and provide various indicators of condition and performance.”
From Fast Food Joints to Utilities
EPRI’s augmented reality research is investigating and assessing diverse potential applications in the electric power industry—from power plant operations and maintenance to transmission and distribution line work. EPRI’s cross-sector research encompasses laboratory work, field tests with utilities, and technology assessments and development, with potential application in EPRI’s nuclear, generation, power delivery and utilization, and energy and environment sectors.
EPRI Technical Executive John Simmins spearheaded EPRI’s augmented reality efforts in 2011, shortly after EPRI switched from blackberries to iPhones. “Like any nerdy guy, I downloaded apps right and left, and one was an augmented reality app to locate fast food restaurants,” he said. “You looked at your phone, and it would tell you there was a McDonald’s 2.4 miles away. I was talking with a colleague and I asked, ‘Why couldn’t we overlay utility data on reality?'”
Simmins’ first project collaborated with four utilities in successfully demonstrating that utility workers could view grid components through augmented reality glasses or helmets along with information about the equipment’s specifications and maintenance history. Now, Simmins is collaborating with 10 utilities to identify applications such as worker safety and storm response. EPRI is assessing the necessary capital investments, quantifying the potential benefits for utility operations, and analyzing cost effectiveness.
“We help utilities identify and document their applications and then develop tests to measure efficiency gains and reductions in human errors,” said Simmins. “For example, if a worker saves 15% of his time completing a task with augmented reality, what does that mean in dollars and cents?”
Con Edison Pursues Safety and Efficiency
New York’s Con Edison is working with EPRI to test various uses for augmented reality and sees potential for bolstering worker safety and grid operations and maintenance. One application under investigation is substation switching. When done incorrectly, this task can put workers at risk, disrupt electricity delivery, and result in large maintenance expenses.
Augmented reality equips workers with a second set of eyes to prevent errors and point them to the right piece of equipment. “You can look at the switches and see that the one that should be operated is lit up,” said Simmins. “If you’re not in the right substation—which does happen—you don’t see any switches light up.”
Another promising application is substation inspections and maintenance. “When you make a round of inspections at a substation, it would be good to have the relevant maintenance history and equipment specifications when you look at a transformer, circuit breaker, disconnect switch, or other piece of equipment,” said Sergo Sagareli, a senior engineer at Con Edison.
Do No Harm
While augmented reality shows promise for reducing human errors and increasing productivity, it is essential to demonstrate that workers can safely wear the technology’s glasses and helmets.
Marquette University Mechanical Engineering Professor Richard Marklin, who has long collaborated with EPRI on ergonomics research, is working with EPRI Senior Technical Leader Eric Bauman to understand more precisely the impacts on workers of using augmented reality equipment.
“Is it safe to use this technology? Does it affect workers’ health and safety, and if so, how?” said Bauman.
Research focuses on muscle and skeletal strain as a result of extended use of glasses and helmets, eyestrain, and situational awareness.
“When you look at an augmented reality display for a long time, there is the potential for dry eyes, eye strain, and muscle strain in the neck,” said Bauman. “With situational awareness, the concern is that you become so focused on the display or alerts that you could forget that you’re at elevation in a power plant or standing next to energized equipment.”
At a test site, Bauman and Marklin plan to monitor 10 electric utility workers equipped with augmented reality helmets and glasses as they complete three typical utility tasks. In the glasses and helmets, a tiny camera aimed at the workers’ pupils will detect blinks and other indicators of eye strain. Future tests may involve gauging balance problems after extended use of augmented reality equipment.
In other research, EPRI is developing and testing wearable technologies to enhance worker safety. These include sensors to monitor workers’ heart rate and temperature and alert them to danger of a heat stroke. Other sensors can alert workers to nearby hazards such as construction zones, energized equipment, and moving vehicles. EPRI is assessing what is important to monitor and how to interpret the information. While the research is still early, Bauman believes such wearable technologies offer the potential to improve safety of augmented reality use.
From Pencil and Paper to Augmented Reality
The current approach to equipment inspections and repairs in power plants is low-tech. “Many coal and natural gas plants still use paper and pencil,” said Matt Buck. “When workers inspect equipment, they often make a written note if something needs work and wait until they get back to a PC before they enter a work request.”
Recognizing that the younger generation of workers is more digitally inclined, utilities are eager to upgrade this approach with augmented reality and other digital technologies. “If you put a young guy in a power plant and hand him a clipboard and tell him to write stuff down, he’ll look at you like you’re crazy,” said Buck.
In 2015 and 2016, Buck and his EPRI colleagues published guides to encourage greater use of mobile devices in power plant operations and maintenance. In 2017, he organized the first “digital worker” interest group—a forum for utilities to discuss current efforts to implement mobile technology for field workers, share experiences and best practices, and assist EPRI in determining research priorities.
These efforts pointed to various ways to manage work more accurately and efficiently. For instance, if a worker determines that a piece of equipment is malfunctioning, instead of taking notes he can scan a tag to initiate a work request. The worker might photograph the component, helping those who will complete the repair to bring the right tools. “You can begin and end the work electronically,” said Buck.
Buck views such digital work management as an essential precursor to implementing augmented reality in power plants. EPRI is examining potential uses of the Microsoft HoloLens*, a headset that superimposes augmented reality images in a person’s field of vision. Researchers constructed mockups of pipes, valves, and other power plant equipment and used the HoloLens to overlay images with information to identify components, menu options for various tasks, and detailed directions on how to accomplish those tasks. A worker wearing the HoloLens headset sees the equipment along with this information and may make selections based on her job responsibilities.
“If you click on a component, a drop-down menu will give you the choice to enter a work request, see assigned work, perform an inspection, and do a lock out/tag out on the equipment,” said Buck. “The vision is a much more human version of Schwarzenegger’s cyborg, walking through a power plant and using the information to diagnose and fix problems.”
Buck recognizes that it’s one thing to make such a tool work in a controlled environment with equipment mockups, but that it’s quite another to implement it in dynamic, diverse field situations.
“In a power plant, there is a lot of rotating equipment, piping, valves, and other diverse components,” said Buck. “How do you incorporate the necessary information so that the device will recognize all this equipment? That is a big jump. You need a lot of computing power that can sit on your nose or head and is lightweight enough that it remains comfortable even with extended use.”
It will take time to develop headsets into a field-ready tool. In the meantime, Buck and his colleagues are working on an augmented reality application that enables workers to hold an iPad in front of a component and view all the same information. They expect it to be ready for field testing in 2018.
“We have to walk before we can run,” said Buck. “But in the not-too-distant future, augmented reality could become a standard tool for utility workers in the field.”
*Microsoft HoloLens is a registered trademark of Microsoft Corporation. EPRI Journal is an independent publication and is neither affiliated with nor authorized, sponsored, or approved by Microsoft Corporation.
Key EPRI Technical Experts:
Matt Buck, John Simmins, Eric Bauman
For more information, contact techexpert@eprijournal.com.
Further Resources:
Artwork by Kirk Anderson
