I received a survey request last week, and I feel like it’s the perfect occasion to do a small case study.
I lived in Japan between 2015 and 2017 but didn't learn Japanese at the time. The thing is, I hate textbooks! It’s just not the way I learn things. I tried other ways, but nothing stuck until I discovered 2 apps: WaniKani and Bunpro.
They are both using a spaced repetition system. You learn new items, and you review them in spaced repetition.
Today, I want to talk about Bunpro. I received an email a few days ago asking me for feedback. And I was happy to give it to them!
So here’s my review of the experience beginning by the email.
First, no personalization is better than bad personalization. They don't store the first name or last name in the application. So instead, they use my username.
The first question is a familiar one:
How would you feel if you could no longer use Bunpro? [Don't worry, we aren't going anywhere 😅]
The product/market fit question! I like how they added the note to reassure their customers. I also find the way the question is formulated to be a bit worrying but also effective. This note is a great addition!
On the other hand, the question had only 3 choices (very disappointed, somewhat disappointed, and not disappointed). But there might be people receiving the email that are no longer using the application yet still have interesting feedback for the other questions.
It skews the results without any benefits. I would still have "Not applicable" as a 4th choice.
Please help us understand why you selected this answer
I found this question too vague. I would go with conditional logic, this way I could personalize the question depending on the answer of the previous one.
If customers would be "Very disappointed", I would like to know why they are satisfied with the application. I would use a question like "Why does Bunpro work best for you?".
In the other cases, I would try to understand where they do have problems. "What parts of Bunpro aren’t as good as you’d like them to be?" is the question I'd use.
How often do you use Bunpro? (once a day, once a week, once a month, other)
Don’t ask questions you can answer with analytics.
In this case, I would have tried to understand why some customers don't use the application daily. That would be more interesting.
What would you use if Bunpro were no longer available?
What is the main benefit you receive from Bunpro?
What type of people do you think would most benefit from Bunpro?
These three questions are also taken from the product/market fit survey, and I don't have anything much to say about them.
Why are you learning Japanese? (work (teaching related), work (business related), academic study, hobby, other)
That's probably the most interesting question of the survey, and sadly it's last. It's an important question because it can be used to segment your customers and learn about their specific needs.
Right now, Bunpro is 3$/m. They need a lot of customers to be profitable. But if they have enough customers using the application for work-related reasons, maybe they can have a premium offer for them.
Small details can have tremendous effects on surveys, and it's always important to keep in mind the best practices.
I recommend you read the following article: 28 Tips for Creating Great Qualitative Surveys. It includes lots of great actionable advice for the next time you want to run a survey.
Finally, follow-up emails are a great tactic to get even more value from surveys. Take a look at the following article by Moritz Dausinger, founder of Refiner: 17 Goal-Oriented NPS Follow-Up Emails You Can Use Right Now (Ready for Copy & Paste).