September/October 2018

The PE Exam: Behind the Curtain

BY EVA KAPLAN-LEISERSON

IN THE EARLY 2000s, THE PE EXAMS MOVED TO A FULLY MULTIPLE-CHOICE FORMAT, WHICH ENABLED MACHINE GRADING AND QUICKER RESULTS. CREDIT: NCEES

The three components of professional engineering licensure: education, examination, and experience. While the order of the last two can vary according to state licensing board requirements, exams remain a critical part of ensuring the competency of PEs sworn to protect the public.

The two licensing exams—the Fundamentals of Engineering and the Principles and Practice of Engineering—haven’t stayed static, however. Their administration and format have morphed over the years, along with their content. The most recent changes are taking place with the PE exams. Here we take a look behind the curtain at past and current transitions.

A Brief History

The National Council of Examiners for Engineering and Surveying—the organization responsible for developing, administering, and scoring the licensing exams—conducts a Professional Activities and Knowledge Study on each of the 25 PE exams every six to eight years. The PAKS helps the council adjust content and keep the tests up-to-date on the profession.

However, it’s not just the content of the exams that has changed over the years, but also the format. Even on their surface, the PE exams of today would look quite unfamiliar to professional engineers of the past.

Prior to 1965, each state licensing board was responsible for conducting its own exams, leading to varied formats. NCEES’s director of examination services, Tim Miller, P.E., notes that when his father took his PE exam in Iowa in the early 1950s, it included an oral interview.

But having individual state exams impeded mobility, as PEs who wanted to get licensed in multiple states would have to take multiple exams. To standardize the exams and facilitate licensure in multiple jurisdictions, the then National Council of State Boards of Engineering Examiners took them over.

The EIT exam (now known as the FE exam) was first offered nationally in 1965, and the PE exams began transitioning in 1966. By 1984, all states had switched over to the new national system.

“I’ve got the best job in the world,” says Miller, of working with the volunteers. “When I walked out of my exam, I was like, ‘I never want to do that again!’ We’ve got people who want to come in all the time and they want to help take the exam, help give feedback.”

Toward Today

Even as the exams become standardized, tests of the past remained quite different than current ones. The first national PE exams were “show your work,” essay-type exams, Miller explains. They had to be hand-graded, with candidates’ papers shipped out to volunteers who scored them using a rubric. “Even though you had one rubric for a particular problem,” says Miller, “trying to get 10 people to grade the same way was difficult at best.”

Exams might also vary in difficulty level, he says. “If you took the exam in April and I took one in October, if my exam was more difficult than yours, good for you and tough for me.” Candidates had to get a certain number of points to pass, regardless of difficulty.

And the process took a long time. Miller took the exam in the mid-1980s, on October 29. He didn’t receive results until February 6, 100 days later.

After a format made up of part constructed responses and part multiple choice answers, the PE exams moved to a 100% multiple choice format in the early 2000s. Machine grading enabled objective scoring and improved validity and reliability, Miller says.

In their current format, exams’ passing scores can be adjusted so everyone is measured against the same standard. A less difficult exam requires a higher passing score, and a more difficult one requires a lower one. “You’re not rewarded if your exam is less difficult,” Miller says, “and I’m not penalized if mine’s more difficult.” In addition, candidates receive results more quickly—on average, within 39 days.

CBT Transition

The PE exams are now transitioning from pencil and paper to computer-based testing, following the similar change in the FE exam in 2014. The exams should all be moved to CBT by 2024.

One major advantage will be the standardization of testing locations. Currently, examinees take the exams in such venues as church basements, fire stations, fairgrounds, hotel ballrooms, civic centers, and university classrooms—“wherever we can get a big group of people together,” Miller says.

NCEES aims to ensure a more uniform test experience, with Pearson VUE again selected as its testing vendor. The company, which provides testing services for a wide range of academic, government, and professional programs, has centers around the country with quiet, private carrels.

Security is tight. Palm vein scanning, pictures, and digital signatures confirm examinees’ identities. They’re asked to turn out pockets, and tattoos are even checked for added scribbles. In addition, videos cameras above each desk enable surveillance.

Test-takers are monitored by trained and certified Pearson employees instead of part-time test proctors who may interpret processes and rules differently. “We’re trying to make sure [test takers] in Maine and New Mexico are being treated the same,” says Miller.

EXAMINEES TAKE THE PAPER-BASED PE EXAM IN LARGE GROUPS, WITH SUITCASES FULL OF REFERENCE BOOKS. CREDIT: NCEES

Range of Benefits

A major advantage of CBT is the increased availability for many of the exams. Those with large examinee volumes—such as the exams in civil, mechanical, and electrical engineering—can be offered year-round instead of just twice a year. Each examinee for those large-volume exams receives a unique but equivalent test compiled from a bank of questions (called “items”).

NCEES staff and volunteers work continually on hundreds of test items to fill the bank—versus the previous linear process of developing a single 80-question exam.

Nicole Jenkins, P.E., has volunteered with NCEES since 2005 and currently serves on three different committees. (“NCEES is my passion!” she says.) As the incoming chair of the architectural engineering exam committee explains, year-round testing enables candidates to take the exams as soon as they feel professionally ready. In addition, exams can be easily rescheduled if something comes up.

For exams with fewer candidates, such as those in petroleum and fire protection engineering, computer-based testing will replicate the once-annual format, with a single-day, standard test for everyone. Less than 10% of candidates take these exams, according to NCEES, with each having typically no more than a couple hundred examinees per year.

Another advantage to computer-based testing: alternative item types. Subject matter experts developing items will have new ways to test knowledge, such as point-and-click to identify an area on a drawing, drag-and-drop for matching, and fill-in-the-blank.

In addition, hard copies of exams won’t have to be shipped around the country, or answer sheets returned. That will not only improve security but further shrink waiting times. Candidates should get results within 7–10 days.

CBT will also enable the council to collect data that is not currently available, such as how long examinees spend answering questions. If they’re spending too long on a question, it can be adjusted or removed.

Point of Reference

In the current pencil-and-paper format, the PE exams are open book. Candidates can bring in as many references as they want—and they do. “You’ll see rolling suitcases, hand trucks with multiple milk crates, banker boxes,” says Miller. “I’ve even been at a site where someone came in with a rolling bookcase.”

However, the better prepared test takers are, the fewer references they actually need, says Tommy Caldwell, P.E., a longtime NCEES volunteer and past chair of the PE chemical exam committee. “People who are test ready, they tend to walk in with a small armload, not the wheelbarrow.”

The large number of references brought in has been a security concern for NCEES—it’s hard to control what’s coming in and, even more importantly, what’s going out (such as exam items). And as Ashley Cheney, the council’s manager of exam publications, notes, the testing centers are a lot smaller than exhibit halls and won’t accommodate wagonloads of books.

Thus, each PE exam taker will be supplied with a single, searchable electronic reference manual, provided on a split screen with the exam. The development of these manuals—hundreds of pages compiled for each exam over a couple of years—has been a significant undertaking, requiring NCEES to ramp up staffing and outsource some production.

“It has certainly been a challenge,” says Cheney. But she praises the volunteers working on the manuals, saying she can’t stress enough how impressed she’s been with them. “We’re really asking them to do quite a bit of work in such a short amount of time.”

She also emphasizes their willingness to jump into something completely new. The volunteers, largely drawn from industry, had little previous experience with book authoring.

Resistance, Then Dedication

It wasn’t just the exams that needed to convert. Both Caldwell and Jenkins describe a reluctance within their committees to move towards CBT and a single reference manual.

When the chemical exam committee first learned of the changes, Caldwell says, members “fought tooth and nail.” And Jenkins says the reaction of the mechanical HVAC and refrigeration exam committee was, “No way—we’re not giving up all our books!”

Engineers hate change, Caldwell says. But now he acknowledges that the transition “is actually really good.” For instance, he says CBT allows the committee to more easily examine which items perform best at sorting the qualified from the not qualified. “Over the years the items are just going to get better and better,” he says. “Even though it’s fantastic right now, I see the potential of continuous improvement.”

And Jenkins points out that other professions have been using CBT for years. “This makes us current…. It’s an exciting time to be involved with the exam.”

Hundreds of Hours

The chemical exam was the first PE exam to convert to CBT, this past January. To produce the reference manual, teams of six to 10 volunteers worked with two editors, a desktop publisher, and an illustrator to produce the 597-page reference manual in 18 months—a feat that Cheney calls “phenomenal.”

According to Caldwell, each primary chapter author typically spent several hundred hours on it. That included researching their topic from all commonly used texts and handbooks. For his chapter, Caldwell ended up with a collection of several dozen textbooks and hundreds of online references.

And, he explains, the work came with additional challenges. For example, committee members found errors and inconsistencies in previous references. That required them to check their work against items that were already in the exam bank.

“Every equation, even the most basic and universally accepted, was questioned and multiply reviewed,” Caldwell says. “We didn’t [take] anything for granted.”

Some of the smaller volume exams, and their reference manuals, are developed by technical societies working in cooperation with NCEES. The PE nuclear exam will be the first of the lower volume exams to convert to computer-based testing, with the inaugural CBT exam slated to be administered in October 2018. The American Nuclear Society (ANS) Professional Engineering Examination Committee has spent the last four years preparing for the conversion in conjunction with NCEES, explains its chair, John Bennion, P.E.

The work included substantially increasing the number of questions in the exam bank to aid the CBT process and creating the 643-page reference handbook. Six volunteers assembled the handbook, and 21 past and present members of the PEEC reviewed the draft, Bennion says.

Nate Carstens, P.E., a nuclear engineer at the Pacific Northwest National Laboratory, served as the primary editor. He explains that the manual includes materials from public domain sources, ANS textbooks, and other external sources. In a few cases, the committee even needed to create content.

Ongoing Work

Throughout the development process, NCEES emphasizes that the handbooks are not meant to be general references but instead to simply provide material for the tests that examinees aren’t expected to memorize. The manual is not a general book of knowledge or a teaching tool, says Miller. “We give them the equation, but we don’t need to teach them how to use it.”

The manuals also include supplemental references, such as codes and standards. Working with publishers to get permissions and electronic versions for these is another huge challenge, according to Miller.

After each reference manual is completed, it’s pretested with the exam. While six or seven people may work on the reference, another 10 or so might be involved with this process.

Every handbook will not only be included with the exam at the test center but also made available online prior to the CBT launch, to allow examinees to familiarize themselves with it.

But once each committee completes its reference manual, the work isn’t done. Items are added frequently and retired when they become outdated, explains Cheney. The references will be updated annually, including consideration of examinee feedback.

Advancing the Profession

More than 800 volunteers work on the NCEES exam committees. Like Cheney, Miller emphasizes that he “can’t say enough for them,” as well as the in-house staff. “We can’t do what we do without them.”

As a reference manual editor, Carstens says he put in a lot of night and weekend work. Why? “I’m a big believer in the PE exam,” he says, and “my skillset and background lent themselves to helping.”

Caldwell says he wanted to help grow the profession. “Plus, it was fun. Challenging your head.”

Jenkins also points to the chance to use her skills and brain in a way that’s different than her day-to-day work as a project engineer. And she wants to give back to the field she loves. “Of anything I’ve ever done, [volunteering with NCEES] has been the most rewarding for me,” she says. “I come back from meetings with a great sense of satisfaction and accomplishment.”

“I’ve got the best job in the world,” says Miller, of working with the volunteers. “When I walked out of my exam, I was like, ‘I never want to do that again!’ We’ve got people who want to come in all the time and they want to help take the exam, help give feedback.” Volunteers meet two or three times a year and get to know each other, he explains.

They are multigenerational, ranging from new professionals in their 20s to experienced engineers in their 70s, Caldwell notes. “We like the camaraderie,” he says. In addition, “because of the friendships developed while working on this committee, I have trustworthy and lifelong contacts from essentially every aspect of industry. Not many engineers can make that claim.”