In a past life I founded/operated a conversion rate optimization company.
One of our all time most successful tests was related to this story and happened almost by accident.
The client had just rolled out new packaging and wanted help validating the change statistically, very similar to this situation.
Their metric was measuring upsells to a premium product after a purchase, and like Candy Japan, found statistically insignificant results.
But the new packaging was beautiful and we wanted to test it on their landing page.
We tried just replacing the old product image on landing page. Meh results, no real bump.
We tried increasing the size of the image + circling with "Improved" (no joke). 10% bump. Interesting.
We hired a professional model to hold the product and smile. 15% bump. !?
We hired a group of professional models to hold the product and smile. 40% bump in sales. !@%K$!?
We re-hired the same group of people to hold product, look SUPER excited (jumping up and down like their team just won the superbowl). 50%+ bump in sales.
No joke.
If Candy Japan is reading and you'd like some help recreating I would not be shocked to see something similarly group oriented, visceral, and beautiful bump yeh.
Interesting story! I've long wondered if the (over)excited voice of many a startup's marketing/onboarding/help copy is justified by customers liking it more than the alternatives. This seems connected.
>10% bump.
>15% bump.
>40% bump in sales.
>50%+ bump in sales.
Was that cumulative (i.e., 1.1 * 1.15 * 1.4 * 1.5 ≈ 2.7 total) or relative to the base figure?
How about a plain box in some relevant color (sakura blossom pink? :) ), hand-stamped with the Candy Japan logo?
In my mind's eye, that'd remain classy enough not to be creepy/cringy/embarrassing (as https://news.ycombinator.com/item?id=11641715 complains), but would also have sort of a personal touch to it.
Having a couple rubber stamps made and stamping each box during packaging wouldn't probably cost an awful lot...
Seems like a good alternative. There has been a lot of valuable feedback in these comments. I hadn't considered how these might be perceived at a workplace, or that people even receive packages at work.
I second having something more classy. Japan is renowned for its package design, so why not include this aspect as part of what's being sold? Maybe something like how Meiji brands their goods. That illustration/design would seem to me to make you less differentiated from your competitors who are tacking on candy to their other "nerd culture" offerings.
Have you considered that the new box is too "generic" or "corporate?" (For lack of a better word)
In other words, the old box, while plain, adds a certain intimacy. It doesn't look like a huge multinational candy company sent it. The new box, though, looks basically like any other piece of Japanese media.
For me, the plain box gives me cheap Chinese vibes. It brings to mind badly damaged cardboard boxes with dubious electronics inside.
The Japanese illustrated box is innocent and kind of cute (certainly not "rapey" as another commenter put it) and in a Western country, stands out from the things around it. Perhaps if you're in Japan, surrounded by Japanese media it might seem generic but I don't think that's a problem elsewhere.
In addition, Japanese products are not unknown for overpackaging. It's probably not surprising that Apple took some cues in their packaging from Japanese product packaging to signal "upscale" meticulousness and attention to the customer.
I can see how that might be the case. Current subscribers might know me personally or through my blog. That would then mean that as I start to reach out to people who don't know me personally, these results could change.
As a current subscriber (who has been receiving the old packaging), the new packaging is off-putting:
* It looks childish. If you are familiar with the Japanese style of advertising etc., probably not an issue, but as a box to be sent anywhere and to be received in a variety of settings (workplaces etc.) it seems like a poor choice.
* Anime may have poor connotations (unfair IMO) as pointed out elsewhere in the thread, which will reflect on the receiver.
* It feels corporate / like advertising. It may seem silly, but I'm more likely to continue subscribing to a small company if I'm on the fence about canceling.
* It feels like a waste of money. The current packaging is utilitarian, which signals that money is being spent only where it needs to be. The new packaging feels like an unnecessary indulgence, which would make me think I'm getting less value out of the subscription.
I often think like this, then I'm reminded how infomercials are so obnoxious and yet they still work. So the conundrum of personal taste and a good gut instinct vs what works with a lot of people.
was the old box that blue plain box? It's not intimate to me. Compared to the new one, it looks kind of homely
Someone who admires Japanese culture will likely not have the same reaction as someone with little exposure to Japanese culture.
The picture is not even on the same planet as 'rapey' Whoever that was either said it to get a rise or they have some distinct issues.
One solution, perhaps, is avail both packaging options to buyers.
The typeface and font, I think, could be improved. It kind of looks like an afterthought. I don't see "branding". It's not Lobster or Comic Sans... but...
So I think the main issue is how the buyer might perceive the packaging. If a coworker has issues... I think the coworker has issues. It's like someone finding Barbie offensive (for other than body image issues). I can't see it. Are these people who whisper and shuffle when they buy condoms or feminine products at the checkout counter? Do they wrap their wine or beer in brown bags? It's hard to imagine this person.
Humans are social animals. When I buy things, I absolutely consider how that purchase will look to other people. it's one of the primary reasons why brands have value.
You might not like that the world works that way, but as a business it's wise to not deny reality.
Right, but this nettlesome coworker you imagine is, I can only imagine, the lone outsider who can possibly find offense in the packaging. I think most would be like, hey, what's that? Oh, candy from _Japan_? Can I try one> Wow, Oishi!
That's the kind of person you do not want to work with. This the conspiracy theorist who misinterprets benign things and paints them in whatever twisted way they want for whatever odd reason they have.
_That_ person is the problem in the office, not the person who bought sweets online and probably shared them with some of the office.
If anything, the packaging helped reveal an oddball at the office.
> I absolutely consider how that purchase will look to other people. it's one of the primary reasons why brands have value.
Wait, what? In fashion, sure, by definition - but when I'm choosing between Persil and Tide, or Coke and Pepsi, or Pizza Hut and Domino's, I'm primarily looking at how good it is at whatever I want it to do. Which one cleans better, which one tastes better?
There's a certain point where many people start buying things more as social signalling than as actual useful things, but I'd disagree that the vast majority of brands could be fashion items in this way. The rest use their brands to evoke trustworthiness - "I know that this will do what I want".
Masking one's preferences and behavior to avoid offense in a public is a defining element of Japanese society and culture. And while everyone there is more inured to these kinds of graphics, the design of packaging for everyday goods often veers towards the minimal/elegant for precisely this reason.
I hadn't intended to imply that the graphic was offensive, nor do I think that most Japanese people would find it so either. I was just pointing out (perhaps imprecisely) that perceived appropriateness in Japan is highly context dependent, with work environments being amongst the most conservative in this regard. The degree to which people seek to avoid offending others (as compared to the US) is quite at odds with a "if someone doesn't like it, it's their problem" mentality.
Trains and parks, by contrast, are generally considered an appropriate context to consume these kinds of materials. That being said, book covers are also far more commonly used for exactly the same reasons.
That has to be a joke. This has no hint of tentacles or sexualization that I see. It's a couple of youngish* people sharing or (long shot, if you take the greedy view) tugging over the box of sweets.
*The age is hard to tell, it's like trying to divine Buggs Bunny's age from his cartoons. But the age is irrelevant as it's two people with a box of sweets.
Maybe if it was teenage girls all the way around. If a 25+ year old male had that laying around, I guarantee you all the women I know would non-judgmentally classify him as 'probably hentai'.
Yeah I think it's really borderline, besides the product is candy - not girls so why make girls the main / standout part of the very publicly visible packaging. It just looks... Dodgy!
Sort of... It follows Japanese advertising characteristics and they are selling Japanese candy. So.... But, yes, I think the people depicted are too prominent and the candy should be more conspicuous. The packaging might benefit by giving the candy more prominence?
Perhaps, but it's not a huge jump from creepy given how hyper sexualised girls are in modern society coupled with perhaps some good intentioned coworker humour but at the end of the day I'm paying for the interesting candy, not for cartoons of young girls and I find that weird regardless.
I don't agree with the connotation, but unfortunately there are negative associations in American culture with older white men who are into anime or other Japanese cultural artifacts.
Sure, there's a certain stigma attached to anyone who enjoys Japanese culture -- older white men included. But even then, one of the words that pop into my mind when thinking about said stigma isn't rape. It's weeaboo, nerd, otaku, geek, parents basement, neckbeard, etc etc etc.... I say that as someone who enjoys japanese culture from time to time myself. Not anime, but j-dramas, movies and their music. Especially their music. It reminds me of how music was here in the 90s (not as manufactured, still a lot of experimenting going on, etc).
Honest question - what do you see as sexualising in that picture? (I'm not going to question the young girl part, but I think that depends on culture / drawing style / reception)
In Japan you have 30 year olds (and older) who dress in that manner[1] and behave in that manner and speak in that manner. It's just different --so viewing it under a western lens probably introduces misinterpretation.
I run a subscription business through mail myself, and I love your posts.
Have you tried to run introduction offers? 50% off for first shipment for example? If I were you I would try to put the countdown for the next shipment on top of your site. Thats a really powerful psychological factor.
Your CTA is also a little unclear, why do I need to first get an email, then a link? The buy process should be straight forward, not through my email. And if you have a really good reason for doing it this way, you should atleast tell me why I need to leave my email.
I really like the cartoon that explains the process. Very good.
Regarding packaging design, we did a huge remake in 2013. We increased our prices with 30%, and redesigned packaging completely. Doubled our LTV, insane gains. Just keep testing, but try to change provider for carton. We pay $0,1 USD for bigger cartons then yours, in one of the most expensive countries in the world. (With plastic window for address field) We chose to put our three USP on the back of the shipment box, to remind your customers every time they receive a shipment why they subscribed in the first place.
I don't really think A/B testing something like this, with this size of a sample, on a subscription based niche-product can give you any real insights.
Why not poll your subscribers or (assuming you can get the costs to work out) have an option for anime art. Personally I wouldn't want the anime style box, something custom and simpler would be fine.. but thats my personal opinion.
I didn't get in on the survey and got the old box in the A/B test, but I would 100% prefer the white one with the illustration for my subscription. I order my candy to the office to share and people usually see the box from Japan and are curious so they ask me about it and I tell them about the candy subscription.
The white box would be better advertising for you in my case :)
I'd suspect much different results if you conducted the same survey but indicated that the white box may result in cost increases to the customer. I suspect they wouldn't care about the box if it meant a higher cost.
I like my subscription. My biggest issue has been shipments that end up in my mailbox melted. It would probably not be possible for new packaging to fix this, but this would be an area of improvement worth higher cost.
I think candyjapan should add a new form of subscription, one-time shipment, and I believe they would see it being ordered by about 40% of their customers. ;)
One consideration is the user experience is not connectable to a purchase.
If I went to a friends house and saw this, I would never remember the name. If you could combine "making the purchaser look cool" with timing-relevancy, I think your packaging improvement would pay dividends.
Put another way, I believe your A/B ratio test is actually X*Y/Z where X/Z and Y/Z are both < 1. Synergy.
Terrible idea, (possibly) useful for analogy: Instead of optimizing the unboxing experience alone, if you also included cards with QR codes, where every QR code immediately triggers a free candy bar (with user data entry on a mobile app). It should also provide social value to the host in realtime; maybe they also receive an immediate notification that their guest scanned their card plus some other bonus I can't think of.
The point is to get closer to the conversion/churn decision, or provide more value to your consumer.
I signed up for CandyJapan several years ago when they were just getting things started. I canceled after probably 3 shipments. Each shipment I was only hoping that the next would contain better products and actually be worth the value in which I was paying. Unfortunately that never happened. While a cool concept and very eye catching, that's all it is. Importing candy is expensive no matter how you go about it. Most people just want a poke to see how it is, and then move on to something else.
Thanks. Yes it costs much more to send candy in individual boxes from here, so if you want more candy per dollar it is definitely better to visit a Japanese market if there is one near you.
I used to receive Candy Japan in the "plain" box to Australia, and the customs sticker declaring the value & contents took up most of the space on the box.
I would suggest a Bayesian decision-theory test of mean profitability: http://gwern.net/Candy%20Japan (Negative, so the decision is to use the old box.)
Those answers are bizarre. Use a t-test or z-test?! That is both less interpretable* and less efficient than doing a test of difference in proportions (based on a chi-squared) or better yet, doing a generalized linear regression with a binomial link which lets you test multiple interventions and include covariates and other things. These are little harder than using a t-test and work much better.
(Of course, even better would be a Bayesian decision analysis which takes into account the full posterior distribution, costs, and benefits, allowing one to decide whether the evidence is enough to justify the more expensive box or whether one should extend the experiment further to collect more data.)
* quick, if conversions are 0.25%, what does a 0.1SD mean difference between arms of an A/B test mean? 'Uh...' Exactly.
To be fair, a couple answers there mentioned the Chi-squared approach, and a z-test could also be appropriate for testing a difference in proportions. I think your Bayesian approach would make the most sense for this kind of stuff, to examine the posterior especially. There's other things that could be done than what you mentioned too to assess a difference. I'm not convinced the parametric assumptions made by the standard tests mentioned (including difference in proportions, whether a z-test or Chi-square test) would hold up well for this kind of data, it would take further examination. I think a z-test or Chi-square test are a decent starting point, and could be fine to use if the assumptions made for those tests are met (to defend some of the Quora answers). The issue is often tests are used without folks either knowing how to check or ignoring the underlying assumptions, but that isn't an issue with the test itself I'd argue.
> I'm not convinced the parametric assumptions made by the standard tests mentioned (including difference in proportions, whether a z-test or Chi-square test) would hold up well for this kind of data, it would take further examination.
Yes, you're right that chi-squared is non-parametric, my mistake. However, even non-parametric tests can have assumptions that should probably be checked into at least for a given data set. It may be robust to the violations the data collected may exhibit, I'm not sure. This article lists the assumptions for chi-squared test:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3900058/
Honestly, I think it would have been really bizarre if something as unrelated to the central product as the design on the box, changed something as fundamental as retention of customers.
That works under the assumption that humans are purely rational beings, which we are not. Though I don't necessarily think the design chosen was a great choice based on other comments here. Maybe something a bit more generic, yet still colourful (like the candies in games like candy crush) would have been a better choice.
Maybe, and if they were selling something for display, like figurines, or even a quasi-collectible I would be right there with you. I know people who like that kind of thing, and they would like a funky box too. This is JUST candy though, it's just people buying something they want to eat. It's a different mindset, and I think, less likely to be engaged by the aesthetics of the packaging (as in the box). They're interested in the packaging of the actual candy, and the candy itself.
One of our all time most successful tests was related to this story and happened almost by accident.
The client had just rolled out new packaging and wanted help validating the change statistically, very similar to this situation.
Their metric was measuring upsells to a premium product after a purchase, and like Candy Japan, found statistically insignificant results.
But the new packaging was beautiful and we wanted to test it on their landing page.
We tried just replacing the old product image on landing page. Meh results, no real bump.
We tried increasing the size of the image + circling with "Improved" (no joke). 10% bump. Interesting.
We hired a professional model to hold the product and smile. 15% bump. !?
We hired a group of professional models to hold the product and smile. 40% bump in sales. !@%K$!?
We re-hired the same group of people to hold product, look SUPER excited (jumping up and down like their team just won the superbowl). 50%+ bump in sales.
No joke.
If Candy Japan is reading and you'd like some help recreating I would not be shocked to see something similarly group oriented, visceral, and beautiful bump yeh.