PapaJohns.com Usability Test
Summary
Overview
Papa Johns is falling behind the competition in online pizza orders. In an effort to increase our competitiveness in this space the user experience team conducted research on papajohns.com.
To identify the pain points of the interface we first needed to identify the expectations of the website. Individuals primarily come to papajohns.com to:
View menu and order pizza
Find promotions and deals
Find the nearest Papa John’s location
Usability Testing and Recruitment
Participants were recruited via a phone screener that specifically targeted advanced users. All participants in this study are considered advanced users who have ordered pizza online before.
We asked the participants to agree to our NDA and recorded them performing key tasks on the website. We studied the recordings and came up with key findings and recommendations.
Key Findings
The pizza divider toggles are not immediately apparent to most users.
Text and email offers were overlooked by most participants.
If a customer had a bad experience with Papa Johns they would rather talk to someone over the phone rather than fill out a form.
Participants expected a summary of their customized pizza toppings.
Key Recommendations
Make the pizza divider more prominent on the order builder screen. Consider adding labels with the pizza icons.
Add Text & Email Offers to the Papa Rewards page.
Add the corporate phone number to the Feedback Form page.
Show a list of pizza toppings on the Order Builder page and on the Your Cart page at checkout.
Website Being Assessed
In this study we assessed https://www.papajohns.com/. The websites primary function is to drive sales of pizzas through online orders. The primary audience will likely come to the website to view the menu, view the latest specials, order pizza, and find the nearest Papa John’s location. Auxiliary audience members may include job seekers and current or potential franchise owners.
Expectations of Ordering Online
Over the years customers have begun to expect more and more out of their online food ordering experiences. Some of these expectations are:
Ease of correcting mistakes
Robust, but easy to learn ordering process
Confidence that their order was submitted successfully
Accurate delivery and/or pickup time estimates
Customers expect the online ordering process to have all the same offerings and clarity as ordering over the phone. Confidence in the online ordering process is key.
Methodology
According to usability.gov, “User research focuses on understanding user behaviors, needs, and motivations through observation techniques, task analysis, and other feedback methodologies.” For our study we conducted a usability test that incorporated task analysis through direct observation of the user.
Before we could conduct the usability test we first developed a screener that would specifically target advanced users. We then scheduled each participant for an hour-long session and offered them a $100 visa gift card as compensation. We purposefully crafted the tasks and moderated the sessions in such a way that would not bias the results or lead the participant down any particular path.
Participants in the study were recruited with the following screener.
Participant Information
Testing took place between June 11th and June 17th 2018. Out of the four participants tested, two were male, and two were female. Three of the participants were Caucasian one was of unknown ethnicity. All participants were in their late twenties to early thirties and use the internet at least three hours per day.
List of Tasks
The tasks performed during this usability test are as follows:
Task 1
Imagine that the people you are with are fans of Papa John’s and that is where you will be ordering the pizza. There are bunch of people at this party and you need to order 3 pizzas. They are:
Mushroom and pepperoni.
Half onion and half sausage with light sauce.
A specialty pizza, because you want some variety.
Task 2
You want to sign up for deals and coupons, but you don’t want to register. You just want to give them your email. Can you do this and what do you think you will receive by email?
Task 3
Your pizza arrives and it is terrible! Your driver was rude and you are really upset about what just happened. You call the local store but get nowhere. You need to contact the corporate office. How would you do that?
Participant Interviews
Synopsis of Findings
A clear pattern emerged in each of the testing session. Most of the participants were able to complete task 1 but had difficulty finding the split toppings feature. On task 2, most of the participants were unable to find the Text & Email Offers link. They would most often search the Papa Rewards page before abandoning the task. On task 3 we had mixed results with two out of the four participants finding the corporate number and half only finding the feedback form before abandoning the task.
Task Completion Rate
Task completion rate measures the percentage of participants that were able to successfully complete each task. This table summarizes the results for this test.
Findings and Recommendations
This table summarizes the key findings of the usability testing sessions and categorizes them with a high medium, and low rating based on their impact on the user experience. Recommendations on how to fix these usability problems are shown in the right column.
Analysis of Tasks
Task 1 Findings
Even though 100% of the participants were eventually able to complete this task successfully, it wasn’t without some friction. The summary of the findings are listed below.
Split pizza toppings toggle was not immediately apparent
Specialty pizzas were not fully customizable
Some participants were confused by the description modal on the menu page (see figure 1).
Was the interface successful in helping people complete the task?
The interface was successful at helping the participants complete the task. A participant remarked “I like how it also has the picture so that I can actually see where I’m putting the toppings” (Participant 2 7:45). Although almost all did not see the divider toggle right away. One participant thought that the pizza image was interactive and tried to click on it.
Did individuals complete them in a similar or different way?
Most participants did complete this task in a similar way. Although two of the participants did not like the “pop out” modal on the menu page and would have rather the “Order Now” button take you directly to the pizza builder page.
Quotes from Participants
Participant 1
“Where’s the divider? I know that there’s a divider.” (10:28)
“I can click these stupid buttons I think” (figure 2). (10:34)
Participant 2
“I like how it also has the picture so that I can actually see where I’m putting the toppings.” (7:45)
“Oh, so it doesn’t let me delete all of the specialty items?” (9:25)
Participant 1
“I can click these stupid buttons I think”
Participant 3
“I’m wondering if there should be a finish button” (10:39)
“I wish that the specialty Pizza wasn’t locked in its customization because I clearly couldn’t modify certain things” (15:50)
Participant 4
“I caught that, for a second I was wondering where that would be… I order Pizza a lot but I feel like for some people it would be a little confusing” (11:52)
“I think the location is okay, I just feel like it should say something like half and half whole” (12:46)
Task 1 Conclusion
Summary of findings for Task 1.
Finding
Split pizza toppings toggle not immediately apparent
Source of Error
Participants struggled to find the split toppings toggle button.
Recommendation
Make the split toppings toggle more apparent on the page and possibly add labels to the icon.
Finding
Specialty pizzas were not fully customizable
Source of Error
System would not allow The participants to fully customize a Specialty Pizza
Recommendation
Add the applied toppings at the top of the page when the user first clicks customize. Allow the user to fully customize the pizza.
Finding
Participants expected a confirmation of their pizza toppings.
Source of Error
No easy way to confirm pizza toppings once the pizza was added to cart.
Recommendation
Add list of customized options on the checkout screen.
Task 2 Findings
Only one out of the four participants successfully completed this task. The summary of the findings are listed below.
Could not find Text & Email Offers
Expected Text & Email Offers to be on the Papa Rewards page
Phone number input field was three text input fields instead of one
Was the interface successful in helping people complete the task?
The interface did not help participants complete the task. Users spent time searching for the text and email offers.
Did individuals complete them in a similar or different way?
The one that completed the task did eventually find the text and email offers in the footer after searching on the Papa Rewards page.
Quotes from Participants
Participant 1
“Lets see if Papa Johns Rewards will do that” (14:40)
“If I go into join I think I’m going to have to put all my information
in again. Because I don’t want to create an account.” (15:12)
“Yeah, I wouldn’t even know where to go.”(15:13)
Participant 2
“Let’s see what sign up does, so that basically gets me to the same window.” (11:39)
“I don’t see a clear way to do that.” (12:08)
Participant 3
“Its weird that its three separate categories for my phone number and not just one… also why are you validating my phone number?” (see Figure 2) (18:00)
Participant 4
“I just feel like umm, I don’t know, I don’t feel like I can JUST give them my email.” (21:29)
Task 2 Conclusion
Summary of findings for Task 2.
Finding
Could not find Text & Email Offers
Source of Error
Text & Email Offers only listed in footer in body copy font size and only featured on certain pages in the footer.
Recommendation
Make Text & Email Offers more prominent.
Finding
Expected Text & Email Offers to be on the Papa Rewards page.
Source of Error
Content was not where participants thought it would be.
Recommendation
Add Text & Email Offers to the Papa Rewards page.
Finding
Phone number input field was three text input fields instead of one
Source of Error
Unknown programming error.
Recommendation
Make the phone number one input field.
Task 3 Findings
Task 3 performed the worst out of the three with a 0% completion rate. This can be explained by the fact that the corporate phone number is not listed on the website. The summary of the findings are listed below.
Participants could not find the corporate phone number
The feedback form was the only way that participants felt like they could contact the corporate office.
Participants want to talk to someone over the phone if they have a bad experience.
Was the interface successful in helping people complete the task?
None of the participants were able to successfully complete this task. A participant remarked “If it was a horrible experience I would make sure I would get someone on the phone” (15:52). This quote captures the overall sentiment of the participants during this task.
Did individuals complete them in a similar or different way?
All of the participants abandoned this task on the feedback form.
Quotes from Participants
Participant 1
“I would go to customer service, that is most likely what I would do” (16:25)
“I would probably do the feedback form, but I would still probably call someone.” (16:34)
Participant 2
“I don’t see a more corporate number, I just see local numbers”(See Figure 3) (13:37)
“If it was a horrible experience I would make sure I would get someone on the phone” (15:52)
Participant 3
“I would probably go to the bottom of the page and look for contact.. which I don’t see or customer service.” (19:10)
Participant 4
“I’m more of a caller if I’m really angry about something.” (23:00)
“It doesn’t look like there’s a corporate number, unless I fill out that form.” (25:01)
Task 3 Conclusion
Summary of findings for Task 3.
Finding
Participants could not find the corporate phone number
Source of Error
Corporate number was not listed on the Contact Us page
Recommendation
Add a corporate office number to the Contact Us page
Finding
The feedback form was the only way that participants felt like they could contact the corporate office.
Source of Error
This is not an error, the feedback form is the only way to contact corporate.
Recommendation
Add the corporate number to the feedback form as an alternative method of contact.
Finding
Participants want to talk to someone over the phone if they have a bad experience.
Source of Error
Not listing the corporate phone number on the website.
Recommendation
Add the corporate number to the website.
Miscellaneous Feedback
Participants want to see a list of their customized pizza toppings on the Your Cart view. After completing 3 orders some of them wanted to confirm what toppings they had added to each of their pizzas (see figure 6.) The website only displays “Large Original Crust Pizza” with no other descriptors.
Follow Up Research Questions
Because of the usability issues identified, I am highly recommending doing iterative usability studies so that we can test the impact of our new designs. For example, informal formative testing allows us gain insights early in the design process which allows us to pivot design directions quickly without wasting development time. For our next testing session some follow up questions that need to be considered are:
After the redesign, what elements of the site improved and did the design meet the needs of the users?
Will we see a decline of Papa Reward enrollments if we make the Email & Text Offers more prominent on the site?
If we add a corporate phone number to the website, will the call center become congested?