What is a concierge experiment / test?

COVID killed my testing budget to $0.

Due to COVID, any experiment that involves an invoice like online adds are a no go.

One idea I had for a ‘free’ experiment was to ask the frontline call centre staff to use content from a concept I’m working on when they are making a sale and let us know the reactions of the customer.

Is that a concierge experiment? Because it involves a person…

What google had to say about a Concierge test

• Interview based
• In the customer’s native environment
• Ask if they would be interested in the solution
• It’s bad when when we have a clear idea of the solution, good when you’re not sure what the solution is

So asked some people who had experience with a concierge.

Person 1 thoughts

“Here’s what I think of as a concierge experiment.

It usually involves taking a small group of customers, and co-creating solutions based on their specific circumstances.

It can be a really effective way to get rapid iterations of a solution done, because you work with the same group of people, and they provide ongoing feedback during solution design.

The risk with this, is that it’s possible to cater too much to their needs, and it might not represent the needs of others.

So, in your case, it depends on who you’re designing for. If you’re designing a solution to help frontline staff, and you’re able to work with a group of them in an ongoing way, trying things, taking their feedback and iterating, then absolutely, that is a concierge experiment.

However, if you’re ultimately designing a solution for an end customer (e.g. the frontline consultant’s customer) then I’m not sure I’d call it a concierge. Because, while the frontline consultant is certainly a subject matter expert, they’re not actually the customer.”

Person 2 thoughts

“You pretend in their environment and it’s fake and you intervene.

We validated a new digital experience in a bank branch. We created a sign in the branch if you want to check your balance. A line forms and your team says:

“Hi there, my name is blabla, can you fill in this tablet so we can process for you.” Give them the tablet and observe them with your prototype.

“Thanks it will be just one minute,” and you go confirm with a branch teller, and you come back here.

I wouldn’t intervene, just observe, and afterwards ask some qual question.

It’s more a validation of the solution if you’re half confident — before touching a line of code, trial this in the wild.

Get into their environment.”

Person 3

Customer Interviews vs Concierge

“So both of these activities involve real customers, but the strength of evidence they product changes. As a rule of thumb, the closer the evidence is to behaviour, the more stronger the evidence is.

So this means that talking to people < people doing things, in terms of evidence. Usually, when we think about customer interviews, there is a spectrum from not knowing anything and learning (i.e. customer discovery) to knowing something and getting input on the thing (e.g. user testing a prototype). Customer interviews mainly live in the world of “opinions” — you show something to the customer, and then you get an opinion.

A concierge is when you are actually solving the problem for the customer, and you are solving this in a way which is manually assisted.

The key here is that the customer engages with your thing and actually gets some value from the interaction.”

Wizard of OZ with frontline staff

In the end I concluded what I’m doing is not a concierge — but more a Wizard of OZ in a different way — using the call centre staff instead of the website, and the ‘prototype’ is the content.

Next thing I want to unpack is the difference between a concierge and a Wizard of OZ in another article…