We share our thoughts on how to use online qualitative approaches – and when and where they are most effective.
Why would you do financial qualitative research online?
As in other market sectors, qualitative research in financial services has traditionally relied on personal interviewing – face to face in the case of group discussions and depth interviews, or on the phone as an alternative way of conducting depth interviews.
But sometimes it is just not practical or desirable to adopt such a personal approach, yet qualitative insight is still what is required.
Strictly Financial’s approach
In our experience, an online approach offers benefits in easing sample pressures, and works well in gaining responses to questions about what people think, as well as for picking ‘winners’ from a range of stimulus such as positionings or advertising executions.
Adopting an online approach could make the difference between gaining useful insights at a reasonable cost, and the research being impossible to put together or so expensive and time-consuming as to be impractical.
We recommend using a bulletin board approach as it has significant advantages over online groups/ chat rooms. Using online bulletin boards has enabled us to conduct qualitative research ‘groups’ with people too far away from each other to be able to take part together in the same physical location, and with relatively small samples (e.g. customer lists or difficult to find samples, advisers, SMEs, C-suite etc): the key recruitment criterion becomes willingness to take part, rather than willingness and availability on a given day.
The flexibility to fit the research into the respondents’ day/ week enables them to participate in a project which might otherwise be impossible for them to fit into their schedule. And because they are able to pick their own time to take part during their working day, they are focused on the research when they participate in it, rather than trying to fit it in among other – and from their point of view more important – distractions.
In terms of content, we feel online qualitative research is most effective in exploring thoughts, opinions and behaviours. It lends itself naturally to exploration of rational preferences and decision making. In the specific area of communications research an online approach can clearly reveal preferred approaches and executions.
Clearly, as with any methodology there are weaknesses as well as strengths: online approaches lack the immediacy of face to face, and in our view lend themselves less well to the exploration of deeper feelings and emotions.
However, there are often cases and circumstances where the pros clearly outweigh the cons, and where online is the obvious solution. So how do we go about designing and running a qualitative research project online?
Chat room or bulletin board?
There are a number of different ways in which qualitative research can be conducted online. The two best known are probably the live chat room and bulletin board approaches.
We prefer to use a bulletin board approach, where both questions and answers are spread out over time.
In our experience, the bulletin board approach is generally to be preferred. Respondents can to choose what time of day to take part, which helps recruitment by incorporating – and signalling – flexibility in the design as well as overcoming some of the sampling difficulties (which remain in the chat room scenario).
Different respondents can answer the same questions at different times of day to suit their own preferences. The staggered approach also gets over issues such as the speed of the internet connection (which could reduce their ability to respond to complex visual stimulus they need to react to) or the respondent’s typing skills.
All respondents answer all questions – so unlike in a group discussion or chat room, where people can simply ‘agree’ with someone else’s point of view, a bulletin board ‘forces’ everyone to present their own perspective. Participants can be allowed to see the responses of their peers, but when they do so is controlled by the moderator. In this way interaction can be encouraged, but also managed.
The other benefit of staggering is that respondents are able to think about their answers – both at the time of initial exposure to the question and subsequently – they are able to return to previous comments and amend/ expand on them as further issues occur to them. This means we receive fuller and more considered responses overall.
All of these benefits are also available to client observers – they can log on and review progress at any stage (and at their own convenience), can ask the moderators to follow up comments with more probes and gain early feedback on key issues.
We have also conducted live chat rooms, but while they address geographic issues with sample, in our experience this type of online environment exacerbates the challenges of running a traditional group: it becomes harder for the moderator to manage the varying quality, quantity and timing of input from individual respondents, and the logic and flow of the discussion can be lost as responses arrive at different times due to differences in respondent internet connections and typing ability.
Online questioning technique
The crucial difference between an online and face to face approach is the flexibility to ask questions and follow-up probes in a conversational setting. Online questions need to be fewer in number and more focused than the equivalent questions in a group topic guide.
In our bulletin board approach, we favour a relatively short ‘main’ question, which is supplemented with a more explanatory question indicating the type of information that we are seeking. In practice what would be follow-up probes in a face to face topic guide can be incorporated into the ‘main’ question in an online approach.
This approach clearly signals to respondents what we expect from them, keeps them focused and on track.
Just as in a face to face group, where unexpected answers can prompt new questions which are not on the topic guide the moderator is able to write new ones in response to the answers that are coming back. In this respect the online approach is similar to traditional groups, where individual participants can be asked to expand on/ explain an answer they have given.
It is also possible to seek feedback from the respondents by inviting their comments and suggestions for improvement in the questions being put to them. This can be very helpful in a larger and longer duration project, where their suggestions can be incorporated into changes, e.g. to the wording of the questions being posed, as the ‘fieldwork’ progresses.
Recruitment
From the recruitment point of view, there can be real benefits to taking an online approach, and recruitment can sometimes be more straightforward than it is for groups:
- It is an easier recruitment task than for a group discussion – it feels less ‘intense’ and less daunting for participants
- We are able to recruit from anywhere (rather than being limited to a few miles radius around a single point)
- We recruit a greater range of people from different areas, some of whom do not normally get asked to take part in groups. This makes the sample more inclusive with fewer unintentional biases present
- People are often very amenable to taking part, and happier to commit to a daily 15 minutes for a week than giving up an evening to attend a two hour group
We still need, as always, to over-recruit. We typically recruit 12 respondents for an online group in the hope that 9-10 will take an active part in the research, and will do so until the end of the process.
The incentive is paid on completion of the week’s research. We find that participants are happy with this and accept it as part of the deal.
Managing a bulletin board
When it comes to managing the project, bulletin boards present advantages over traditional qualitative groups or chat rooms:
Moderation & stimulus
The moderator puts questions to the group as a whole, obtains responses, and follows these up with supplementary questions to the group and/ or probes to individual respondents (e.g. based on what they said in reply to the initial question).
Good moderation is good moderation – and we apply the same basic tenets to a bulletin board as we would in traditional qualitative research.
Participation in the group is asynchronous (people see questions and give answers at different times over a period of days rather than all taking part during the same two hours), so there is both a requirement and an opportunity to use different moderation techniques.
- Questions can be set up to ensure that each respondent gives their own views before seeing what other people have said. This means that we obtain ‘clean’ responses from all participants before they build on/ respond to other people’s comments
- Ad-hoc follow-up questions can be put either to the whole group or to individuals, although this should be done sparingly, so as not to ask more of the participants than they thought they were signing up to when they were recruited
- We are able to ‘spread’ follow up questions over time and among different respondents. This is partly to spread the load of extra questions among the respondents, but also to indicate to everyone that we are actively ‘listening’ to what they are saying, and to motivate them to continue
Stimulus is easy to manage online, as the moderator has complete control over who sees what, and when. This also means that when participants comment about different pieces of stimulus, it is always clear which they are referring to (which is not always the case in a recording of a face to face discussion).
Online also lends itself to the collection of other input from the participants: for example, diaries, photos and video clips including video diaries or ‘vox pop’ pieces to camera.
Respondents’ answers
Just as the dynamic of an online group is different from its face to face equivalent, there are significant differences in the type of answer you get back as well.
In running online business groups in financial services, we have found that one key difference is that participants’ answers and comments online are more considered than in a face to face group:
- They are almost always on topic (no irrelevant answers), and often more clearly expressed than in a face to face discussion
- They are often longer than in-group responses, paragraphs rather than sentences
- At the same time, they are also often more concise, with no wandering around the subject in the answer, and the very nature of the method automatically provides verbatims
- Participants often provide specific illustrations of, for example, perceived provider strengths and weaknesses, resources etc.
- Because online responses tend to be more considered, they are also more complete than in face to face conversation.
As with face to face groups, there is still the opportunity for respondents to talk to each other directly as part of the discussion – not all the dialogue has to go through the moderator, though of course he/ she can also see and get involved in any such discussion.
Output and Analysis
We get immediate and complete transcripts from the online approach, and the answers are organised by the question structure (all comments about a particular subject area are in the same place). It is always possible to see who said what, and thus to link comments with individuals and assess how widely these sentiments are shared.
The structure inherent in a bulletin board avoids any possible stimulus confusion – it is clear what is being referred to. This makes it considerably easier to manage the information, and as a result the analysis process with online groups is considerably faster.
The same applies to any photographic or video footage gathered as part of the research (scrapbooks, video diaries, vox pops, etc.). This too arrives in a more structured and ‘already compiled’ form, making it easier to organise and work with.
However, there are also some weaknesses to counter-balance the strengths of an online approach when it comes to analysis:
- Some depth and granularity are lost compared to face to face methods. Without the tone of voice, hesitations, face and body language that you get face to face, some of the nuance and subtleties are lost.
- While the design of an online question enables us to build in follow-up probes, these are often contained in the original question (rather than always being posed as a separate follow-up), and they are sometimes ignored
In weighing up these weaknesses, it is worth bearing in mind that just how important depth and granularity actually are will depend on the context: what is being researched, and among whom. Equally, not everyone answers every question in a face to face group either, so this limitation is not unique to an online approach.
Case Study
To see a case study of recent work we have conducted amongst advisers using the bulletin board approach click here.