#362 – Bridging the Gap: AI’s Impact on Market Research with Justin Chen
Prepare to unlock the secrets of split testing in e-commerce and discover how it can revolutionize your business in our riveting conversation with Justin Chen, co-founder of PickFu. We’re promising you deep insights into the world of split testing, where you’ll learn to leverage this powerful tool to refine and optimize your product line. We’re not just talking about images here, we’re taking you into the realm of videos and even audio comparisons, to give you a comprehensive understanding of product testing.
Our exploration doesn’t stop there. With Justin, we delve deeper into how PickFu harnesses AI-generated images and videos to offer you actionable insights on product promotion. Ever wondered about optimizing Amazon product images? We’ve got you covered. We guide you through the process, emphasizing cost-effectiveness and the value of human feedback. Along our journey, we also touch upon early product validation using hypothetical tests and potential barriers one might face in this space.
But wait, there’s more! Our conversation expands to encompass the growth journey of PickFu, and how it has evolved into a leader in consumer research and testing. We discuss their foray into international markets and their mission to democratize consumer research. We wrap up our enlightening discussion with a look into the future of split testing and the challenges it might pose. So tune in, as you’re about to gain the tools and knowledge needed to transform your business.
In episode 362 of the AM/PM Podcast, Kevin and Justin discuss:
- 01:30 – Kevin Helped Putting PickFu In The Map?
- 03:33 – How PickFu Got Started
- 06:35 – Where Do They Get Their Market Research Audiences?
- 08:27 – Can They Validate Shopping History?
- 09:14 – How Long Does The Responses Take
- 10:12 – What Type Of A/B Tests Can You Run?
- 12:32 – Other Creative Way To Run Tests
- 13:04 – Integrating AI Capability Inside PickFu
- 15:30 – PickFu Over Amazon’s Manage Your Experiments? Or Both?
- 18:46 – Get Validation Much Earlier Before Investing Time And Money
- 21:04 – Using AI To Create, Test, & Improve Main Images Before Sourcing
- 23:29 – Market Research Test For Videos
- 26:00 – Tips On How To Get Solid Data From Polls
- 29:53 – Demographics Data For These Polls
- 31:20 – Interesting Case Studies From PickFu’s Clients
- 33:29 – Serving Clients From Fortune 500 Companies
- 35:43 – Does PickFu Work Outside The US?
- 37:04 – How Kevin King Did Market Research Before PickFu
- 38:56 – Kevin’s Idea For An AI Feature For Polls
- 40:17 – NDAs and Other Security Measures
- 43:14 – Why Should You Be Testing?
- 45:10 – How To Get In Touch With Justin Chen And PickFu
- 45:24 – This Week’s Words Of Wisdom From Kevin
Transcript
0:00:00 – Kevin King:
Welcome to episode 362 of the AM/PM Podcast. This week, my guest is Justin Chen, one of the co-founders of PickFu. We’re going to be talking about the importance of split testing your images, your videos, everything about your product, and a whole lot more in this episode. It’s going to be a lot of good information. Hope you enjoy it. Welcome to the AM/PM Podcast. Welcome to the AM/PM Podcast. Welcome to the AM/PM Podcast, Mr PickFu himself. Justin, how are you doing, man?
Justin:
Good Finally on the podcast, super happy to be here.
Kevin King:
Yeah, this is awesome. For those of you that don’t know who PickFu is, PickFu is one of the actually I think, the original split testing service in this space. There’s a few people that have kind of followed in your footsteps since then, but you’re the OG, the guys who basically started it all right, that’s right.
Justin:
I think you’re the one who put us on the map for e-commerce Me. What did I do? You mentioned us. What was it? Was it Global Sources back in 2018?
Kevin King:
Yeah, I might have that’s when you started, right.
Justin:
Well, so we started way before that, but we weren’t really on the map for Amazon in e-commerce until I think you mentioned it, and then I think Manny mentioned it on this podcast, on the AM/PM Podcast, and then we definitely saw a lot more Amazon sellers coming in, but before that we weren’t really targeted towards Amazon or e-commerce.
Kevin King:
Oh really. So when did, was it called PickFu then as well?
Justin:
Yeah, it was still called PickFu. So my partner, john, and I we built it oh my God, I mean probably over a dozen years ago as a side project and we were building a completely different business like not, you know, it was a website. It was something we were doing before this and we were working on a redesign for that website actually, and so we were trying to figure out which way to go, and we couldn’t decide between the two of us. We wanted to get actual consumer feedback. So, being engineers, we built this as a side project, kind of threw it up on the internet as things happen, put a paywall on it, you know PayPal button back in the day, and, yeah, it kind of lived there, took on the life of its own for a while and, you know, meandered through a few different industries.
Kevin King:
So you guys were doing all the coding. Were you just the two of you, or did you have a team?
Justin:
Yeah, so back then it was just the two of us and we built the initial version of it. We had a lot of. We thought entrepreneurs would be our target use case, being entrepreneurs ourselves, but we found a lot of authors using it for book titles and book covers. You know Tim Ferriss for our work week. He was kind of famously talking about using Google Ads to test his book titles and so people were trying to get more data driven about testing book titles and so we had a lot of self publishing authors and you know using to test that kind of stuff. So that was. That was really cool. We have a lot of bestselling authors that use a pick for that kind of stuff still today. In fact, James Clear was an early user of pick for as well.
Kevin King:
Oh, really Adtomic habits yeah.
Justin:
Yeah, that’s right. Yeah yeah, so really cool to see that kind of content coming through and then to kind of meander through. You know, mobile gaming is an interesting space for us. A lot of just software startups using it. Duck Go, if you know. The privacy search engine, yeah yeah, Gabriel their founder was he was one of our biggest supporters in the very beginning Loves using it, and so they continue to use it on kind of almost every creative decision in their in their company.
Kevin King:
So whether it’s ads or UX or anything like that, and when it first started you were kind of just jobbing out the back side because you got an army of people We’ll talk about that here in a minute that actually participate and help do this. But in the beginning wasn’t like kind of like almost like a jobbing out to like a mechanical Turks or something. And then you grew big enough to where you could actually start cultivating your own audience.
Justin:
Yeah, so we tried to stitch together. Yeah, we did start with that, and now we’re stitching together the different panels and trying to bring kind of the best, best quality to our consumers without having to know, like, where that quality comes from.
Kevin King:
So for those of you that don’t understand what, if you’ve never heard of PickFu PickFu is basically a split testing service. It started out pretty basic, where you could compare, you know, two images. You could say if, like what he said with authors, you could say which of these book covers should I go with? Which one looks more appealing? Or you can ask a question, whatever question you want. But you’re trying to isolate rather than just asking your friends and family, who are close to you and might not give you their honest opinion, or that they’re a little bit more jaded, or this total strangers that don’t know you from Adam, don’t know anything about what you do, don’t know your personality, don’t know anything and get their opinion and it’s not just vote for which one you like is best, and maybe that’s how it started. But you guys now have where they actually have to say why and we’ll talk about some of that in a second. There’s different variations of that. But they don’t just say I like A or B best. They say I like A best because and they write a short little couple sentences, or a paragraph in some cases, of why. And it’s really great because then you get to get an insight into not only which one they chose, but why they chose it. And sometimes they’ll say stuff that you didn’t think of. You’re like, oh, I don’t, I didn’t realize. Everybody thought yellow was a mix mix that you want to throw up or whatever they might say, and you see a pattern of that. You’re like holy cow, and so that’s where it started.So the audience that does this, basically the way it works is you put there’s a number of different tests and we’ll talk about some of those in a minute, but it’s core basic is it’s image A against image B, and you could do, you know more than one image if you can do for images whatever, but it’s image A or image B and at its core. And then there’s a group of people. You select how many people you want to actually give you opinions and the most common number is probably 50. I mean, you can go up higher to 200 or whatever you want, but the most common number that way you get a statistically valid response, because you need 30 or more actually is a technical fee and stat statistics. The critical number. So you do 50 and the people you know within a matter of depends on the time of day you do it. If it’s during the middle of the day on a weekday, usually you have your answers within an hour. You know it might take a little bit longer if it’s, you know, at night or on a weekend or something like that. The audience that gives these feedbacks, where do they actually come from? I mean, how big is that audience and where where those people come from?
Justin:
Yeah, so they’re all people who like to be paid for doing surveys, and so we tap now. Now, we tap into all the same enterprise market research panels that you know, say, procter and Gamble would use when they’re running their longer form market research studies, and so, in the US, we are able to tap into over over 10 million different consumers who are, you know, coming from different avenues, who just want to, you know, get paid or get rewarded for doing something. And so the challenge with all these panels is maintaining a high level of quality and then, like you said, being able to target the audience by different demographic attributes, and so that’s kind of the layer that we’re bringing to. It is being able to target, like by, amazon Prime members who are, like you know, like to drink tea and have cats or something like that, and being able to do that and make sure that the quality of responses, the responses that they give you, explaining why they chose something of high quality, and that they’re genuinely, you know, trying to do a good job.
Kevin King:
So these people, they get paid a nominal amount for every survey. The answer so they’re not. Are they just doing yours or they’re part of some big panel that they’re also doing other companies are sitting at home, you know, for a couple of hours one afternoon. It could be a soccer mom, it could be someone that’s just at their office and bored during the lunch hour and they just like to make a little bit extra money. Just, they just bang through different surveys like they get one from you, they get one from somebody else, and for an hour they make 10, 15, 20 bucks on the side or something. So is that kind of how it works?
Justin:
Yeah, pretty much yeah. So they’re definitely doing other people’s surveys as well. Like you said, they’re typically just bored or they want to do something while they’re watching TV or whatever. It is so like kind of side hustle money.
Kevin King:
And how many? You said it’s 10 million that you have access to.
Justin:
Yeah yeah, obviously, like you know, not everyone’s on at a given time and you know finding certain attributes can be a little difficult. So, while most of our our polls do finish in under an hour, if you were to do a lot of targeting, like for a very niche audience, it could take a little bit longer.
Kevin King:
And what Justin means by that is you can go in there and you can just let it be open so you can say here’s image, here’s that my main product, image number one. Here’s my main product, image number two which one do you like? And at a minimum, you should probably say that you want prime members, but that’s still most of the US. That’s not going to limit you too much, yeah, and then you can go. There’s a whole range of demographic and other filters where you can choose. I only want women over 50, or I only want people who work out three to four times a week, or whatever the whatever it may be, and you can filter that down. So that’s what he’s saying, that that can slow it because they have to fit those criteria, which is better for you in a lot of cases. If that’s your target, because you don’t want some random dude you know that has never worn makeup in his life voting on your makeup box, that’s right. For example, I mean, you rather have someone that’s actually in the market to buy that stuff and you have the ability to actually have people that actually have bought Red, versus just their that’s their interest. Do you have access to like databases, like we know this. This person bought from Mary Kay cosmetics or something, and you can’t say exactly where, but you can. Can you overlay that so we actually know they’re spending money on those things?
Justin:
We’re not able to verify that. It is self-reported so. But we do ask Questions like you know, what are the types of products you’ve bought in the past six to twelve months? And so we do ask like kind of specifically, you know, are you buying things in electronics category, or pets or toys or and things like that? But it’s not. We’re not able to validate it against their actual items on shopping history, anything like that.
Kevin King:
And so most of them you said can you get the answer with? Even if you do like 200, you get the answer within an hour in most cases.
Justin:
Yeah, I think if you do a general response poll, which is basically first come, first serve, it would probably be done in under an hour.
Kevin King:
And now now you’ve expanded it now beyond just single images. You can test titles, you can test bullet points, you can test a number of things. Talk about some of the different Options that are that are in pick food for what we’re most commonly used.
Justin:
Yeah, so, like you said, when we first started, it was for comparing two things a versus B. A lot of times it was an image. It can be texts, it could be video we see animated gifts a lot or even audio, and so Basically any kind of media format. For example, we’ve used it to test voiceover actors for, for videos or theme music, that kind of stuff, and so the head-to-head was the original one. We opened it up for up to eight options. So when you’re, when you’re comparing, let’s say, eight variations of your logo, what actually happens is we asked the respondents to rank every single one, so they’re not just choosing oh, I, like you know, the seventh option out of eight, because if they just chose one, you may not get to any kind of you know statistically interesting answer. You’re just gonna get like 12 and a half percent or something like that. So by asking them to rank every single one, we’re actually able to run an instant runoff, kind of like what they do in a lot of elections now, and you actually take into consideration their second place vote, the third place vote and actually come up with a winner. It’s our more most preferred winner Based off their ranking, and so that’s kind of what we do when it’s more than two options.
Kevin King:
If they read it one, it gets eight points. If they read it to get seven points, and so on down. So you could have something that’s rated number two by a lot of people. That exit ends up actually being the best choice, not necessarily one that got the first, the most first place votes, because there might be a. The number one choice might have 30 people like it, but a lot of people gave it a seven. So and then all those people are a vast majority of them gave brain at number two. That actually means number two is actually Probably statistically better, right.
Justin:
Yeah, yeah. So that’s why we kind of call it the most preferred right, like it’s not horribly offensive and like most most generally liked by the audience, so and it’s worked out pretty well. The other way we actually do it is if, if you want and this takes a little bit longer and does cost a little bit more as we Do a full round Robin tournament, where we’ll actually do a versus be, a versus C, a versus the, a versus e and all the permutations, and Essentially do kind of like a tennis tournament, have them match up against each other and the winner of the tournament is actually the winner, and that that’s a ton of responses.
Kevin King:
And we’ll actually see a lot of our larger enterprise companies, or, you know, gaming companies who who just want all their data and that’s the kind of test that they like to run and you also I mean like for Amazon sellers you can actually, you have an option where you can bring in, like imports, a listing from Amazon and Actually imports are listed as if the listing notes the search results right. So yeah and then you could normalize it. So you could actually, if you got 20 people selling a product your dog bowl you could bring in all 20 dog 19 other dog bowls from Amazon page and then put yours in one that randomly put yours in one of those spots, your image, and Then you can go in, you can normalize the review. So so actually make everybody have four star reviews, so there’s not one, that’s, everybody has the same price and everybody is prime, for example, and then then it’s gonna get some really good insight on where the eye goes and what people are clicking on. Right.
Justin:
Yeah, that’s right. So we kind of call it a mock-up tool generator, and so what happens is you can put in your ace in and maybe, maybe you’re not even selling it. Maybe it’s a category you want to go in, like you were talking about, and you want to see if you’re gonna be able to compete against all the other dog bowls or whatever it is, and so you can put in your product concept, maybe the title you’re thinking about using, maybe it’s the price you’re thinking about going in at and seeing if are you gonna be able to steal some of the clicks against, against the top category leaders, or Maybe you’re already in the category and you just want to test out a new main image and see if that’s gonna affect how people perceive you against your competition. So it’s a really important way to gauge, in a hypothetical safe sandbox, how you’re, you’re, you’re gonna do with, you know, either changes to your listing or before even investing into a new product line.
Kevin King:
That’s cool. And then then you go recently, earlier this year or two you added some AI to it where it’s does like the sentiment right, where it can actually read the reviews and end up with the people right you know they’re voting and actually come up with a general sentiment correct.
Justin:
Yeah, so since every one of our polls includes written feedback, like you were saying, that’s a lot of text to analyze and so if you’re doing 50, 100, 200, even 500 responses, that’s a lot of text for for any person to to read through and kind of pull out all the themes and you know the broader likes and dislikes. And so what we’ve done is we’ve integrated with AI to feed all those results into AI and generate a nice three paragraph executive summary that highlights the pros and cons and you know the net, even possible next steps that you may want to take with your poll, and then we also break out likes and dislikes, but every single option using that AI as well.
Kevin King:
What’s the advantage of using something like pick food to do testing versus just using Amazon? Split testing or you can split test the price and people are voting with their wallets versus their mouse Sure, really in that case. Or Testing images, you know, running at Facebook ads to see which, which one, gets clicked on the most. That’s what a lot of people have done before. They knew much about pick foods run 10 different Facebook ads, see which one gets the most clicks. Why should we do your service versus something like that, or should you do both?
Justin:
Yeah, I mean we definitely recommend doing both Pick food and Amazon manager experiments. I think it’s just different, different parts of the process. So the way we recommend it doing is Using pick food earlier in the process. So, as you’re trying out different creative directions and you know maybe you want to try out layouts, that, or product designs or color variations that you know may never see light of day, obviously that’s what you would use pick food for once. You come down to maybe your top two main images then or you know, you centered it on them what the prices are going to be. Then test it live with manager experiments. The benefit is that pick food is just going to be so much faster. Obviously, manager experiments is going to take a number of weeks to gather that data. Even Facebook ads may take a while, and with both of those approaches you’re not going to get any written explanations. Why so? That’s why it’s not great for experimentation, but maybe just fine-tuning what you already know are. You know good options and you’re just trying to like Eek out the best performance out of it.
Kevin King:
I remember earlier this year my, my trainer was he long, he made a journal and he’s like Kevin, I don’t know anything about Amazon, can you help me out? And so I offered to help him out. You know I’m not good, it’s like I’m not gonna do the work for you, but I’ll help you get going. You know, explain it, sure. And so he went out to on. He’s like well, I need to do. I say you need to create a bunch of images. Give him all the steps they need to do. So he went on a fiverr and found somebody and created a like a main image for him and I said one’s not enough. I mean we need like four or five of them. So he gave he had four or five of them done. Some of us just slight color changes, some of us wording changes or layout changes, the way of showing it. And then we did kind of what you did. We didn’t use the round robin option, we just did one after another, but we just put it a number one against number two and then number three against number four, and then a winner of both of those and then came out with a winner out of the four. And then I told him and you know he was happy. He’s a little bit surprised, you know, reading some of the comments like wow, they didn’t like this and it is a little bit surprised. But then we took it and we I I just Grab I don’t think I use the actual tool where you have the 20 or whatever it is Well, I just grabbed four thumbnails of competitors, the top competitors, the ones that are doing the best. They’re kind of similar to him, and we just mocked up, we put those four against each other and said which of these four Would you like the best? And he came out like I think he got I forget the exact number is like six percent of the vote, which was like it was a shot to the gut for him. He’s like you know, he is proud of this thing and like it’s like, it’s a nice thing. But and I’ve seen it, I’ve held it in my hand but people were just like no, we like these others better. So we read the comments, like you said, and they said we like that we can see this page, or we like we can see this example or this color. So he went back and made a bunch of changes on it and we ran the same thing again, against the exact same competition, and this time I think he went up to forget the exact number 16 or 18, maybe 20 percent somewhere around in that in the ballpark, which, which says that, yeah, he would like to have gotten, you know, 70% of it, but getting that percentage is still means that he can take, you know, enough people, there’s enough, yeah, enough depth in that field, that field where you know, even if he’s in the fourth or fifth best seller, he’s still gonna have some decent sales.
Justin:
Yeah, I think you bring up an important point there that you don’t always have to win the test when you’re testing against a competition. Right, you just you want to make improvements and going from six percent to twenty percent show that you’re probably gonna get some clicks and probably gonna get some of sales, as in. Secondly, you know, an interesting way that a lot of people end up using us is that they they come to test out their main image because they think that’s the issue with their, their listing right, like, oh, I’m not getting the clicks, I need to test on my main image and they’ll start testing their either the main image against their competition or just some variations they came up with and the written feedback Actually comes back as people saying they don’t like the product, they don’t like the design, they don’t like the branding, they don’t like the packaging kind of like what the feedback you were reading about, about the journal and Then they realized, oh man, I should have been getting validation much earlier, much earlier. You know, before I spent, you know, ten thousand dollars on this inventory and having a sit in Amazon just trying to optimize a main image, I should have been testing out my concepts, because you know these hypothetical tests, if you do it early enough, will cost you almost nothing, right like you could have. You could just have your your designer mock something up. You could use 3d renders. The whole point is to do these hypothetical tests before you invest a ton of time or money into a product line that you know maybe is not going to do that well and maybe you still would have done it at 6 or 20 percent taking away from the competition. But maybe there’s like five other products that you know you also had ideas for that could do actually better. Maybe that’s worth investing your time there.
Kevin King:
I know someone using AI to do that. They’re actually coming there analyzing you know products on Amazon. They’ll say, hey, I want to do a new I don’t know a new dog bowl, for example, you know new slow feed dog. Well, and they’ll have. They’ll go into AI and do prompting and actually engineer the image. That’s cool. And then come up with a cool design there, you know, refining a little bit mid-journey or whatever it is, and then they’ll load that up On to to PickFu and actually then compared against the competition and you know one of these, these competition comparisons With a number of competition and see what people say. This is before they’ve ever contacted a manufacturer or anything. See what people. See what people say. And I know one person, I think, told me they have to get at least what was the number 50 or 60 percent and they’ll refine it. If they don’t get that, they’ll read what they say. They’ll go back and have the AI, rerender it and, based on what the people say, and then they’ll come back and they’ll try to get it to where, if they can get 50, 60, 70% of the people to vote for them, they’re like, okay, I think we can launch this product. And then they go and actually do the sourcing and everything, and so there’s that’s a great use of something like this, for what’s it cost to do 50 people like I guess it depends on the poll, but 50 bucks or something like that. Yeah, and so I mean you might spend a few hundred bucks, or maybe, you know, even 500 or a thousand bucks if you’re doing a lot of different testing. But that’s a lot better than spending 10 grand or 20 grand on moles and making a new product and bringing it over just to find out that you can’t compete.
0:22:16 – Justin:
Yeah, I mean, I think the pairing of AI generation with actual human feedback is is the perfect combination now, especially because you know our humans are giving you that written feedback and you’re actually going to be able to have that actionable insight into what to then refeed back into the AI. Okay, like people don’t like this color, you know, over this design, like just tweak it this way, like they said, and it’s just gonna hone in on, on something You’re gonna be so confident about going to market with now you guys just do images, right.
Kevin King:
You, I can’t compare videos, right.
Justin:
You can do videos. Yeah, so you can do a video. Is there a length?
Kevin King:
maximum or anything, or how does that work?
Justin:
You just uploaded directly. I think 30 seconds is included With any you know basic pole and then it will just kind of prorate it if you go over that. So if you know went to two minutes, it would probably cost 4x as much. So, yeah, you could do comparison to video or you could just upload a video by itself and then ask people what do you think about this product concept? Maybe it’s like a 3d render spinning or something like that or or whatever it is, but we can’t compare two videos to each other like two.
Kevin King:
How do I know they actually watch the whole video?
Justin:
That’s a little more difficult. Yeah, so we’re working on doing some tracking around that, but we do make sure that they have enough time to watch. Watch all the video and then you can probably tell by the responses as well. So if you you know, depending on how you phrase your question, you could probably get them to Elaborate on, like what’s some of the specific things that they like or dislike about each video and you also have.
Kevin King:
Speaking of video, I mean, you actually have an option where you can out kind of eavesdrop in on someone’s thought process. So they actually they do like a screen capture video, as they’re Walking through the product and then scrolling up and down and they’re commenting, so you can listen in to what they are saying, right?
Justin:
That’s right, yeah, yeah, so we do have a screen recording capability where you might just want to get people to talk through, either them looking at an image or a listing or whatever it is, because there’s definitely Different thoughts that you you surface when you just get people to freeform, as opposed to just writing their final, you know, a written explanation and so that’s a. That’s a really interesting way to kind of get into people’s minds and then, of course, at the end will continue to do that AI summary to kind of synthesize all those, you know, random thoughts that people had.
Kevin King:
so I don’t have to go listen to a bunch of Grandma’s and people with accents and whatever talking about it. The AI will take transcripts and summarize it for me.
Justin:
Yeah, exactly yeah, so you can just go look at the whole summary and just and gather all their thoughts in one place.
Kevin King:
That’s. That’s cool. So why do you think a lot of people Aren’t using this one? Is it because they just don’t want to spend the money and you’re not expending by? Yeah, okay, doing a few tests is maybe a few hundred bucks by the time you do a series of tests, but I don’t. Yeah, why do you think it is? What’s the barrier to more people not actually using something like this to really hone in On what they’re doing?
Justin:
Yeah, I mean, I think the first thing is I don’t think people realize that they can do something like this. I think even when they hear about it, they might still have hesitation around it being too complex. And you know, we we still hear that even for people who have heard of pig food, they go I don’t want to figure out how to use it and I don’t think they’re giving it a fair shake to just log in and try it, like we try to make it, you know, as, as you know, simple, stupid as possible. But we’ve also heard some hesitation around asking the right question and so you know some people, especially when they’re paying money there, they’re not sure like is this the right question? My phrasing it right? And so what we’ve tried to do recently is is have a lot more pre-built templates and questions so that You’re not having to come up with a phrasing of it. So if you’re trying to test in Amazon main image, for example, we’ve got a preset test. That’s got a pretty good question. Obviously, you can still tweak it if you want to add more context, but you know that that reduces the friction, yeah, and then I think there’s like you said, I think they’re the costs, as any Any entrepreneur, amazon seller, like. They’re watching, like every single dollar that they spend, and I think there’s hesitation in just adding yet Another thing that they might potentially have to pay for. We’ve, we’ve tried to reduce some of that friction as well. We’re lowering the minimum number of responses. I know you mentioned, like you know, 30 and 50 responses is being more significant, but In our minds, something is better than nothing, at least getting some directional feedback so you can go as well as 15 responses. So that might be only $15 to you, and we were experimenting with some $5 combos. I know that you saw when it’s only you know five responses, but, and while you wouldn’t want to make a huge decision based off five or 15 responses, it’s still Interesting to hear five people to give you feedback, right, like if you’re going to go into a coffee shop, you probably would only talk to five to 15 people anyways, and so getting the thoughts from those kinds of strangers is still going to be helpful, right, because there’s still, like surface, some objections or whatever it is, and and maybe enough to give you the confidence to spend 50 to get a little bit more significance.
Kevin King:
And you said that some people are worried about asking the right question. What are some tips you could give to make sure you do ask the right question? What are some phrases that you shouldn’t do or that you should do to really make sure you’re getting solid data?
Justin:
Yeah, definitely tried to be as plain, plainly worded and, I guess, and unbiased as possible. So and Simple. So obviously, don’t ask a ton of questions, ask. Ask one question. One of the most common ones is you know which one would you buy? Obviously, that’s very generic. Or which one would you click on? You could add a little bit of context. So, if you were shopping on Amazon for a Teapot, which one would you buy? And that might give the appropriate level of context without skewing it too much. Well, you don’t want to do is. You don’t want to like tip your hand and let them know which one is yours. You know why wouldn’t you buy the red one or something like that. It’s too leading, right. So so just try to be as plainly worded as possible. And then I guess the other tip would be the targeting is something that that’s great, but it could also be a Downfall for some people. I think some people Target too granularly and they, they have such a narrow view of who their, their End user of the product is, but not necessarily who the buyer is. And so I would keep that in mind. Right, because, for example, men’s products. Maybe you have something that’s strictly for men, but if you look at your brand analytics, like maybe it’s actually women who are buying it, right, and so you need to know who’s actually buying it. Maybe there’s different times a year where people the Men are buying it, other times a year that women are buying it as gifts, and so you know those kinds of things. Or or just you know being too narrow about thinking about who cooks in the kitchen kitchen or something like that, right, like you know, oh, maybe you think that only women cook in the kitchen, but maybe a lot of men or actually buying this product or whatever it is. So you know, I would be careful about being too targeted. Obviously, like you said, if it’s makeup or something like that in its makeup for women, then you can you can be a little bit safer about that, but try to use as much data if you have analytics as possible To you know, verify that the right, the right targeting.
Kevin King:
Is there a way to actually test, to see, like do you have an option to where you can segment the responses by Demographics as well? Right, I think that’s, and some of the PDF reports you can download. It I’ll show you, like, if certain age group liked it more than another age group or certain Gender liked it more than another gender, or something like that, right?
Justin:
Yeah, that’s right Actually. So there’s kind of two aspects to demographics for our polls, so the first one being targeting who actually answers it. So that would be restricting who can actually see it and respond. So if you only want men to answer, that’s the first option. The second aspect of it is we actually ask a bunch of demographic questions to allow you to segment the responses and also get to know your respondents. So those would include, like age and gender, income or any of the other traits that you can target by. But, say, you just want to know about if they use makeup or if they like to cook in the kitchen or if they have pets, and so what will happen is they’ll not only give you the written explanation but they’ll give all that demographic information. When you hover over every response, it’ll tell you the demographic profile of that person. So you’ll know that it was a male who has dogs and loves to drink tea. And then we have demographic charts where you can then filter and segment like oh okay, so all the men like option A, all the women like option B, all the tea drinkers like option C. So that might be an interesting way if you don’t know who your product is going to resonate with. We’ve actually seen some customers run larger polls 200 or 500 responses, general audience just to see like, okay, I’m really not sure who we should target. I just want to see who this is resonating with. Maybe it’s a product concept really early stage and they just want to see who they can start targeting afterwards. Right, so just doing it very broad first.
Kevin King:
What’s one of the weirdest things you’ve seen? Someone test.
Justin:
Weirdest things Something like what the heck is this? We see a lot of. We see a lot of adult things. Oh, yeah, yeah, we see a lot of adult products. We see a lot of steamy romance novels, which are always fun. A lot of you know, salacious covers and things like that. So, yeah, I don’t know if there’s anything super weird. I think there’s a lot of personal use cases that we’ve seen. So testing the kids names, you know like, oh, I’m about to have a boy and kind of like testing different names. So those are always fine. So a lot of interesting personal use is even like LinkedIn or dating profile pictures.
Kevin King:
Yeah, I say dating pictures. Test your Tinder pictures, or something.
Justin:
Which outfit should I wear? You know, we see people doing that kind of stuff. Oh really yeah.
Kevin King:
Yeah, what about? Do you have a case study or a somewhat example of you can give us someone who just radically changed their business after doing a test? They were at X level and then they came in split tests and all of a sudden their sales went up 10,000% or some crazy thing. Is there some example that you can give us around that?
Justin:
Yeah, I mean. So Thrasio, who’s you know, one of the bigger Amazon aggregators. One of their main product that they kind of got their notoriety for was that Angry Orange pet deodorizer the cleaner that they use and so when they first had that product they wanted to do a rebrand of it, and so they actually used Pik Fu for that. They were trying to justify a big repackaging, a big new packaging change, and I think it was going to cost them $50,000. So before they did it they wanted to make sure that it was going to be validated, and so they ran a bunch of designs through.
PikFu eventually came up with their new Orange bottle packaging and they launched with that, immediately saw sales improve after that and from there they continued to expand the land, expand the product line, all using that same packaging, and so that was a tremendous success for them, obviously a much bigger project than a lot of people are working on. But we’ve had other really simple ones. We had a case study with yes Bar and they were just changing their main image and they agreed to isolate everything else. They didn’t change anything else about it, but they ran a $65 Pik Fu test, just testing a few different images, and made the tweak to that winning one and immediately saw their click-through rate increase by 12% and so immediately netted thousands of dollars in new sales the next week, and ROI was super easy for them.
Kevin King:
So it’s not just smaller Amazon type of sellers using this. You said you have a lot of Fortune 500 companies and eight Madison Avenue agencies and stuff using your service, right.
Justin:
Yeah, so we actually have quite a few traditional CBG companies using us now. I think our goal is actually just to get enterprise grade consumer research into the hands of the people doing the work. So that includes individual Amazon sellers that might be mom and pop shops, but it also includes the e-commerce and Amazon managers at larger CBG companies people creating new products there. It’s just a much faster way to gather consumer research than what’s previously been accessible. Before, large companies had to work with their consumer-sized teams. They had to work with a market research consultant, kind of run these really long form studies, and it wasn’t something that you could rapidly iterate on the design or a concept very quickly, and so we’re definitely seeing much more adoption in the enterprise base. We recently got our SOC2 certification, which is a security certification for software that a lot of corporate companies look for as a checkbox to show that we take security very seriously. As a company, we’re trying to build the best tech platform for self-service consumer research out there.
Kevin King:
Is PickFu just for the US market, or do you have international markets as well?
Justin:
Yeah, we recently expanded to international with Australia, UK, Canada and Germany, and we have a couple more countries coming out later this year. And what’s actually cool about the German audience and any other country that we add in the future that’s not English-speaking is we’ll actually automatically translate the question into the native language, so the question will get translated into German. We’ll show you what it is if you want to verify it or you could. If you speak German, you can just write the question in German. The response forms are all in German. We collect the responses in German and translate it back into English, and, of course, we’ll also show you the actual German responses as well so that you can validate it, and so we’re trying to enable this. Anyone to anyone consumer research so you could be a Chinese seller getting feedback from a German audience and you could use our app completely in Chinese. We’ve localized the app into Chinese as well, so you could use our poll builder in Chinese and write your question in Chinese and it’ll get translated into German and everything back. So that’s kind of the division that we’re working with.
Kevin King:
That’s cool. I remember, before I knew about you guys, this must have been like 2015. I was coming out with a line of makeup mirrors for women and I got five or six samples from China, different factories, and I’m a dude. I don’t know what the heck women want in a little pocket makeup mirror. But I saw there’s an opportunity with the keywords and the research on Amazon. So it’s like all right, this is a good extension of a brand I was doing in the beauty space and I didn’t have a pick for you. So I actually walked into a bar and just sat there and any woman that would come into the bar is like do you mind giving me your opinion? I’ll buy you a drink or something? Give me your opinion. Like you said, I got 10 or 15 people maybe over the course of four or five hours and I mean it’s cool that they could actually hold it and touch it and check it versus online. But even the older days, companies have spent thousands of dollars on focus groups and stuff. Bring 20 people into a conference room and just to get feedback that now you can get for 60 bucks or less in an hour or less. It’s crazy where the technology where we’ve come.
Justin:
Yeah, I mean, I think what’s been amazing is just watching this democratization of everything, right, like, as you know, anyone can sell online, anyone can sell on a marketplace, or they could spin up a Shopify site, and so we’re just trying to add that other piece of being able to validate with consumer insights, because now that you can sell online, you’re competing against every other large company out there, right, there’s no reason that any Amazon seller can’t outsell the larger brands, but they are using consumer insights, they’re using their very expensive market research consultants and all that kind of stuff. So you should be validating as well, right, and this is the tool that you can use to do that.
Kevin King:
So can I test? I can test. You said earlier, you can test video. So I’m just brainstorming here. So if I’ve got a spokesperson and I have a male spokesperson, a female spokesperson, a country healer, a proper nerd or whatever, and I create the exact same video with the exact same script, but all I do is just change out the person and this could even be an avatar, using AI or something. I mean that’s going to go, but and I can do a test to see which one resonates best with the audience. I could do something like that as well, right? For example whoever they, whoever they believe is the most believable or something I can do that.
Justin:
Yeah, that’s a great test. Yeah, so, voiceover actors, actual hosts of videos, you know theme music. When we advice people who are testing video, we try to get them to test different components of the video before they compose like that final thing. Then it’s just like a product right. You want to get feedback as you’re developing it. We’ve even seen people test storyboards. Actually, it’ll be like literally hand sketch storyboards of like here’s the shot by shot of what I’m going to be doing. What do you think? It’s really cool to see things at that idea stage, even text ideas. It doesn’t have to be anything that’s actually 3D rendered or actually photoshopped. It could be pretty raw.
Kevin King:
Is there an NDA that the people who are participating, the voters, signs that they can’t Like? If I did something like that, how do I? Some people are going like well, if I show this prototype of what I’m doing, what if they just copy and steal it from me? That’s going to be a worry of some people.
Justin:
Yeah, they agree to a non-disclosure agreement before they are even able to see the test. That’s the best that we can do. Obviously, people can breach it, but we’ve never had that issue in the past and we’ve worked with a lot of pretty sensitive IP. At this point, I know that every entrepreneur has that worry. Obviously, as entrepreneurs we have that worry. But I think after you’ve been in the game long enough, you realize that everyone wants to execute more on their own idea as opposed to going to stealing someone else’s idea in most cases. But I can’t really say it’s never going to happen. But we do take as many precautions as we can.
Kevin King:
Is there something like with AI, where you’ve done thousands of not tens of thousands or hundreds of thousands of tests where you could actually make recommendations yourself? Or if I ran a test and here’s, 50 people gave me their feedback, that’s great, but could you then say okay, Kevin, here’s a dog well, we’ve ran tests for 37 other dog wells in the past and analyze that and combine it with what my people said and then make suggestions based on that, like some sort of AI suggestions or something based on that? Is that something that could be possible?
Justin:
It’s definitely possible. Yeah, I won’t make any promises, but it’s definitely possible. It’s a good idea.
Kevin King:
Yeah, I mean, there’s a lot of cool stuff that you can do, or even where you could jump on a call, like we’re talking about the focus groups of old, where you sit in a conference room or something where I could even just pull. I’m just brainstorming here. But something that I would find cool is like let’s get 10 people on a 15-minute Zoom call and so that I can hold it up and show them three-dimensionally, or I could even send it to them. I can like, hey, you want a free product? Here’s a code or whatever to go get it on Amazon or somewhere. Okay, go over it now, I’ll send it to them. You can pass their address or whatever and then give it some time to be delivered. And then we say on Tuesday at eight o’clock we’re going to meet and I’d be willing to pay them better money for that so they have it in their hand, and it’s a live group thing and I think you could, depending on the product. You might not want to do that for all products, but something like that could be really, really cool too.
Justin:
Interesting? Yeah, for sure. I mean there’s, and I think there might even be ways to kind of approximate that with different AI models and interaction mechanisms. So a lot of interesting opportunities now.
Kevin King:
Yes. So the important thing here is what’s the most important message to get across, whether you use PickFu or something else. Why should you be testing?
Justin:
The thing that I want to emphasize the most is definitely that people should be testing. I think it’s just a way to de-risk your investment. We’ve had so many stories of people avoiding catastrophe by being able to not go into a product line or not overstock a certain color variation. Like that’s a super common one, right? You know, maybe you’re stocking up different colors of umbrellas or whatever it is, or different patterns of a puzzle, and you need to make a decision on how much to buy each, or even if you should diversify so much. And so those are our favorite stories where people say, like, hey, I used PickFu, I chose the colors variations. I did because of the PickFu feedback and didn’t get left holding the bag on a bunch of ugly colors. So it’s such a risk minimizer that it’s kind of a no-brainer. And of course, it’s going to improve your sales, right, like it’s by minimizing risk, it’s going to improve sales both by better product selection and product development, but also through things like main image and A plus content, all that other stuff that you can improve. So, yeah, whether you use PickFu or something else or actually going through the bar or the coffee shop, it doesn’t really matter, you just need to get some kind of feedback from your target consumers Because, as any entrepreneur knows your friends and family they stop giving you useful feedback after a few times of asking them and they’re, honestly, probably not even your target audience. So you do need to get some actual strangers to give you feedback on things.
Kevin King:
Awesome, Justin. This has been great. If people want to reach out or find out more, reach out to you or find out more about PickFu, what’s the best way for them to do that?
Justin:
Yeah, definitely check out the website pickfu.com. We have a very helpful customer success team there as well, so if you need any help, just ask on the live chat. You could also help on a strategy call with them for free, so that’s always a good way to get started. You can reach out to me on LinkedIn Pretty easy to find under Justin Chen and connect with me there Awesome.
Kevin King:
Thanks, Justin, I appreciate it man.
Justin:
Yeah, thanks for having me on, Kevin.
Kevin King:
Great stuff from Justin. The guys over at PickFu are awesome, really good people. It’s a great tool to use. I use it personally in my business all the time for all kinds of testing. So if you haven’t tried out PickFu, go to pickfu.com. It can make a big difference in your business and your conversion rates and how much money you make. Give it a shot and let me know what you think. We’ll be back again next week with another incredible episode here on the AM/PM Podcast. Make sure you hit that Subscribe button, Go back and download the episodes that you’ve missed A lot of really good ones over the past year. And before we leave, I just want to leave you with some words of wisdom. Just remember a confused mind always says no. A confused mind always says no. See you again next week.
Enjoy this episode? Want to be able to ask questions to Kevin King live in a small group with other 7 and 8-figure Amazon sellers? Join the Helium 10 Elite Mastermind and get monthly workshops, training, and networking calls with Kevin at h10.me/elite
Make sure to subscribe to the podcast on iTunes, Spotify, or wherever you listen to our podcast!
Want to absolutely start crushing it on eCommerce and make more money? Follow these steps for helpful resources to get started:
- Get the Ultimate Resource Guide from Kevin King for tools and services that he uses every day to dominate on Amazon!
- New to Selling on Amazon? Freedom Ticket offers the best tips, tricks, and strategies for beginners just starting out! Sign up for Freedom Ticket.
- Trying to Find a New Product? Get the most powerful Amazon product research tool in Black Box, available only at Helium 10! Start researching with Black Box.
- Want to Verify Your Product Idea? Use Xray in our Chrome extension to check how lucrative your next product idea is with over a dozen metrics of data! Download the Helium 10 Chrome Extension.
- The Ultimate Software Tool Suite for Amazon Sellers! Get more Helium 10 tools that can help you optimize your listings and increase sales for a low price! Sign up today!
- Does Amazon Owe YOU Money? Find Out for FREE! If you have been selling for over a year on Amazon, you may be owed money for lost or damaged inventory and not even know it. Get a FREE refund report to see how much you’re owed!
- Check out our other Amazon FBA podcasts including the Serious Sellers Podcast, as well as our Spanish and German versions!
- You can also listen to the AM/PM Podcast on YouTube here!