No one who has existing customers wants to launch a new version of their product without a trial. People who pay or subscribe to your business expect it to continue working and for your company to honor the commitments.
TalkTalk had a trial team as many traditional businesses do, separate from the Product Owner, designer and engineering team that depend on the trial results in order to iron out technical and product issues.
Roles and responsibilities in the trial
- As a designer, you need to be as close as to the customer feedback as possible in order to adapt the design.
- As a product owner you need to be as close to the feedback as possible in order to prioritize issues.
- As a developer you need to know the exact issue, how to replicate it and how important it is.
- As a trial lead you need to be able to encourage feedback quickly, get participants to respond and to answer the questions from the participants.
- As a participant you want to be engaged, report issues easily and know that your feedback matters and issues are being solved.
TalkTalk used to run trials using Survey Monkey software. Survey Monkey is good for one off surveys but fail when you need daily feedback, reaching out to the participants quickly and triaging the issues.
In December 2016, we were about to launch a new TV interface. We wanted to get it to the market faster than our competitors and with least issues. I identified the problem that we do not have a system that served us well. Specifically, we failed to engage the customer, respond in time and fix issues. I looked into better ways of running the trial.
After testing a number of options, I singled out Web based trial/feedback software called Dscout. What Dscout provides is a mobile app that participants can install on their phones and provide feedback based on the questions from the trial team. Since we are testing a TV interface, it allows participants to use their phone while they are in front of the TV, record videos or take photos of the issues they encounter. For the team behind the trial, it allows to:
- Send mobile notifications to all or single user with new questions
- Send direct mobile messages to the single or group of users to encourage feedback
- Allow access to the Product Owner, Designer and Developers to the user feedback directly
- Create macros for triaging and tracking issues
- Create smart questionnaires with “or/if” statements
Learnings
You do not need to know how the soup tastes by eating the entire pot. You mix it up and taste a spoonful.
- People are less interested in a reward than in knowledge they brought value. Reward participants when you can but most importantly tell them how they helped.
- Set clear tasks for testing. Set clear way of reporting.
- Screen your participants for those who want to provide in depth feedback, do not test with thousands, test with a couple of hundred. We tested with 1200 people, daily feedback was overwhelming, about 200 entries each day – only 10 of them were not repeating issues. This proved difficult to manage and respond to.
- Respond to every question within reasonable time or they will leave the tial.
- Be a person, not a machine, use your own language, apologize and encourage when needed.
What is the appropriate reward for participants?
It is a common misconception that reward should be monetary. There is a great book by Dan Ariely “Predictably Irrational”, and something I learnt from it that giving someone a $5 for feedback results in people being unhappy with their reward versus their time than those who get a lovely box of chocolate that costs $1. Most importantly people value the importance of their contribution.
Personally responsible
- We launched the software 4 months ahead of competition with a major uptake on connected users
- We were nominated as the best TV provider by Wich.com
- We changed the way we run trials at TalkTalk
- I trained the product and trials team in using the software, communication with participants and creation of surveys
- I negotiated a xx% discount with Dscout in return for our feedback on the software
- I had learnt a heck of a lot about running the trials
- We had the most successful and issue free launch