Can you really trust my reviews?

Can you trust my reviews?
Reviews aren’t as “neutral” as you might think – there are plenty of programs that each ever so slightly incentivize reviewers to see things in a different light. Let’s talk about those!

Let’s talk about reviews. You know, they’ve been an essential part of my channel and of pretty much every other channel in tech, but there seems to be a range of interpretations of how a review should be done, and, if you’re not doing reviews yourself on the regular, you may not be aware of all the different factors that play into the outcome, the bottom line of a review including all of the twisted and intricate details of how reviewers earn an income by doing reviews, like for example, a sponsorship from Thangs.com where I tell you about how they’re a huge model library with over one and a half million high-quality, 3D printable designs, powerful search tools and, newest feature, augmented reality support. And how you should check them out at Thangs.com when you’re looking for free models to print. That was a real sponsor spot, so thanks Thangs for sponsoring this video. And that sentence just now, that was disclosure, Thangs are paying me to promote their service to you. But the thing is, not everyone is that clear about which parts of a video or an article are sponsored and paid for, and which aren’t. And we’ll get to that. But I think we should first look at what actually constitutes a review to me and to the world.

What’s a “review”?

So, there are actually some official definitions of a “review” from bodies like the FTC or the equivalents in other countries. What it comes down to is that it should be a balanced evaluation of a product or a service that mentions both the good and the bad, so what they’re saying is, it shouldn’t just be mentioning just the positives, which would make it an ad, or just the negatives likewise because both of those extremes go against being an objective and informative piece of information. That means that the type of content that already doesn’t “feel” like a review actually has no reason to call itself that, like the “spec reviews” that online stores sometimes do where they just recite the manufacturer’s claims to fame or the thousands of so-called “niche websites” that claim to compare or test products, but then just list specs and send you on your way to buy the product through Amazon affiliate links. Also, I don’t think that videos or articles that, in the case of a 3D printer, just take it out of the box, get two prints out and then go, “wow, I’m so impressed that this thing somewhat works!”, like those I don’t think qualifies as reviews, either, because they don’t actually take the time to properly evaluate a product, it’s just a first look that doesn’t even have the chance to expose some of the details that would make a product useful and practical or annoying in daily use. It’s also why I don’t label unboxing live streams as “reviews”.

But I think equally important to how a “review” is done are the circumstances it’s happening under.

Conditions for reviews

First off, for me, I’m the one choosing which products I cover, so it’s never another company saying “hey, you need to review this right now”. The amount of products I choose not to review is probably tenfold bigger than the number of products that I do review. 

But it’s not just which reviews you do and don’t do, it’s also when you do them. The thing is, a review is most valuable when the product is brand new. It’s generally going to be seen by way more people than if a review comes out a year late when everyone else has already published their hot takes. Now, that is a bit of a hard spot to be in. Because 3D printer companies haven’t quite learned yet that decent reviews take time. Some still think they can shoot out an email going something like “hey, we’re going to have this cool launch in 10 days, can you have a review ready by then?”, and I’m like, well, first of all, I’m already working on my next video, that’s already locked in, but if you’re saying ten days, that means, your printer is first going to take five days to get here, producing a video, including scripting, filming and editing takes at least another five days, when am I actually supposed to use the product? You know, the unique thing about 3D printers is that everything just takes time. Prints take time. Especially when you’re trying to use the full build volume, that’s easily two or three extra days just for a single print. Compare that to reviewing graphics cards, if you have to, you can run a small set of benchmarks in an afternoon and that gives you maybe not a perfect, but a pretty good sense of how it’s going to perform.

Best case, I think the quickest I’ve ever done a proper review is three or four weeks from getting the product to having the review out. So, manufacturers, if you’re listening, two months advance notice, just to get everything lined up, that would be perfect. That’s especially important for video stuff where you can’t just update an article as you collect your data, on YouTube, once something’s uploaded, the entire video is locked in. You can edit out sections, but you can never add new findings.

Responsibility and rules

I’ve also got a general set of rules about which products I’ll review and which ones I won’t touch. First off, stuff that’s unsafe or borderline illegal under consumer protection law, like 5W laser cutters with no enclosure – those are not just crazy dangerous, they are downright illegal to sell or import into the EU, so I personally don’t want to use them, but I also don’t ever want to encourage anyone to buy and use one of those. I’ve also blacklisted companies for repeatedly making 3D printers that are electrically unsafe or fire hazards. Fool me once, shame on me, but fool me twice, and that’s it. So if you don’t want to see reviews of stuff that can laser-beam your eyes out, get subscribed, because I never do those.

There’s also another category of products that I personally try to stay away from, and that’s stuff on Kickstarter or other products that aren’t actually shipping yet. Because in those cases, more often than not, the product you may or may not be getting if you support a project on a crowdfunding platform or pre-order something that’s not out yet (*cough* games) is going to be drastically different from what they would ship out for early reviewers. And that goes both ways: What if I find a flaw – will it be fixed for the final version? It kinda takes the steam out of any criticism when the answer is always “we’ll fix that before it ships” and then who knows if they actually will. But on the other hand, if I end up really liking something, like high-quality components, there’s absolutely no guarantee that those will be used for the mass-produced version as well. Quite often, parts just get replaced with cheaper alternatives that look similar but aren’t quite made to the same quality standards.

And then, of course, there’s always the risk that they’re not going to ship at all because they underestimated how expensive some steps would be and they run out of cash. I mean, it would suck being the one who recommended a printer and then people get scammed out of their money or end up getting something that performs nowhere near what I saw.

So at the end of the day, while it’s easy to make videos with titles like “This is the best $100 printer I’ve ever used” when it’s a Kickstarter printer, that would actually be a pretty deceptive way to spin it.

Morals

While we’re at it, let’s talk about morals. And you might be saying, like how is this a question of morals, you test a product, you report on that, and that’s the end of the story. Except that it’s not. Especially for 3D printers.

With graphics cards, CPUs, and other stuff you can measure performance, and then that’s definitive facts that are pretty hard to twist around. Unless you’re called Userbenchmark. But a 3D printer is a lot more complex. Like, you can measure a couple of things about a 3D printer, noise, print speed, build volume, but even things like print quality are so multifaceted that it’s often impossible to compare two different printers objectively. And then there’s the entire rabbit hole of build quality, usability, software support, and so on, that may or may not be important for a specific machine. In the end, it’s up to the reviewer to come up with a conclusion that reflects how the printer performed for them. And that process is subjective.

And I should point out that there is still a difference between doing a full, biased, sponsored video for a company that showcases a product and a neutral review, but not everyone actually tells you which parts of a video are an ad and which are their own opinion and experiences. I was watching a video about putting a little camera on a bird the other day and I was thinking “well, this is a cool video, but it feels like I’m watching a 17-minute long ad for the little camera”. There are rules and regulations for when and how to disclose which parts of your content is an ad or sponsored or paid for, but let’s be honest, nobody actually sticks to those. YouTube also has a checkbox you’re supposed to tick if there’s a sponsored segment or product placement in the video, and that triggers that little message you see at the start of a video, but not even channels with 13 million subscribers and two ad spots on every video tick that box, so how is that supposed to be of any use to the viewer when it’s not used consistently. The only thing you’re doing is punishing honest YouTubers because that message is going to make viewers suspicious when it’s just on a handful of videos.

Okay, but the water does start to get muddy when sponsorships and reviews start overlapping. Like, how can you justify doing a critical review when just weeks ago, you had a sponsor spot that recommended that same product? The way I handle it is by being very selective about which products I accept for sponsor spots. Like, if it’s not a product I think is worth spending any money on, it’s not going to be in a sponsor spot on my channel, either. That also solves the implications that you’d get the other way around, where, as a reviewer, you’d have to be afraid of losing a sponsor for future videos if you criticized their product in a review. Well, if it’s a product I have to criticize too much, or if the manufacturer tends to throw a fit when I find stuff wrong with their merchandise, I wouldn’t want them in sponsor spots anyway. I don’t know how other channels and sites handle this, but it’s been working out well for me.

I guess I should also note that I never require or even accept payment for reviews. I know it’s a thing because some manufacturers keep asking how much I charge per review, but I think that’s a very slippery slope, because even when you make it clear that payment wouldn’t influence the outcome of the review, still, if you’re not being nice, they are not going to pay you again for the next one.

Instead, I’ve got fantastic Patreons that make up that difference so that I won’t need to get into shady deals. I know that Patreon isn’t an option that everyone can use, so I don’t mind whatever means other channels use to survive as long as the viewer is aware of it and can form their own critical opinion. More on disclosure and all that in a moment.

Free Samples

But first, let’s talk about free merchandise. I thought it was obvious that almost no reviewer actually buys the products they test. I mean, if you had to spend tens of thousands of Euros on phones, laptops, or 3D printers, that would make it pretty hard to review stuff unless you’re rich to start with. So usually manufacturers will send out products to test because for them, the exposure alone is worth the cost, and also, it doesn’t actually cost them the full retail price; depending on how much they’re worth and how much it costs them to get the products back and into a representable shape again, they also often don’t ask for them back. Cameras, for example, are most often loaners. They’re small, light, and easy to ship around and they don’t really degrade during a week of testing. For 3D printers, on the other hand, shipping a typical low-cost printer from and back to China costs way more than what the printer is even worth, and when they do get it back, it basically needs a full overhaul to give a “good as new” experience again. So most manufacturers in 3D printing will send out printers or components and then let reviewers keep them. I know that might be an incentive for people to appease the manufacturer with a good review so that they keep sending more printers. For me, I’ve decided that I’m never going to turn free products into money, I’m never going to sell them. I give them away or disassemble them for parts, and the better ones, I keep and use them for projects on the channel. But I already have perfectly good printers that do everything I need them to do, and even before I started the channel, I had built myself a printer that even today, would still be fine, so, right now, a free machine just takes up extra space unless it’s actually good – and in that case, they get a positive review anyway.

So even more so, I don’t mind if a company asks for their machines back as long as that’s been communicated beforehand so that I can be careful with it and give it back in a decent shape. I’ve once had a company suddenly ask for their printer back three years after the review, and then threaten me when I told them I had already taken it to pieces, like what are you trying to do.

Affiliates

Alright, let’s close this out with affiliate links. I use them, everybody uses them, but when you think about them, they are basically payment and an incentive for writing positive reviews. Just to make it clear how they work, so in the video description or hot-linked in an article or on one of the many dreadful affiliate spamming sites, you’ll see links to the products shown on the site or in the video. Whenever you click that link and then buy anything on the store you’re forwarded to, the person who placed that link gets a commission on that entire sale. You get the same prices whether you use an affiliate link or if you just type out the URL, but when you click an affiliate link, the shop keeps track of who referred you there.

So now the thing is, when you put the effort in to produce a review and then come to the conclusion of “well, this isn’t a product anyone should buy because it’s horrible”, then you’re giving up all the affiliate commissions you’d be getting when instead, you could also kind of leave out a couple of details and just recommended the product anyway.

It creates an incentive to praise every product you review as long as you have an affiliate link for it.

That’s a problem if all you’re doing is reviews, then affiliate sales on the products you “test” are a huge part of your income – and it’s why all those “niche” sites I mentioned before are being so crummy. I solve that predicament by doing not just reviews and diversifying my content and therefore my income. I mean, YouTube is my job after all. But I’m not dependent on the commissions from every single review, and I would highly recommend everyone else getting into YouTube or writing articles or a blog to really look into not making yourself dependent on affiliate sales, either.

Bias

But that’s not to say that there’s not going to be a bias to a review. MKBHD talked about that once, and in the end, everyone has expectations and previous experiences with the brand they’re reviewing or the type of product that it is. For everyone, it’s going to be different what they’re comparing it against and how they’re weighting different aspects, like for example, I don’t care *that* much about how exactly a 3D printer is constructed as long as it’s sensibly built, does the job and is safe, but I find it really important for a manufacturer to provide proper guidance and software for actually using the printer, which is something that’s often glossed over and deferred to being figured out in Facebook groups. So just keep that in mind – everyone’s take will be different, so I always recommend checking out more than one review and paying attention to which aspects different reviews cover or leave out. 

Then there’s also the entire rabbit hole of what happens psychologically to a reviewer between buyer’s remorse, confirmation bias, and the internet preferring positive content in general, but that would be a bit too much to get into now.

Disclosure and what I will do in the future

My takeaway is that you as the viewer or reader should be able to have your grain of salt with any review you watch or read. As in, you should know which circumstances a review was done in. The gold standard for that, of course, is Gerald Undone, he states exactly what relationship he has with the product and company at the start of every product-focused video. Check out, for example, this video from Gerald!

I think I’m going to start doing the exact same thing for my reviews as well, where I disclose on what terms for example a printer was provided and if there are any sponsorships or anything else I’m doing on the video or with the brand in general. Again, this is something I would encourage everyone to do, and it’s important that it’s not just in a camouflaged sidenote, but upfront and in terms that everyone knows exactly what’s going on. I’ve not always been super clear about for example the fact that printers are generally provided free of charge, I thought that was something that was obvious, but it’s not, and I want to do better in the future. 

So that’s my thoughts on doing reviews. If you’re someone who’s doing reviews and are serious about doing them properly, you will have thought about these things already, but, as a viewer and a consumer of reviews, I think you should also know what’s going into producing a proper review and what kind of challenges and conundrums there are to deal with and to navigate around.

If you found this video helpful, as always, leave it a like, share it, if you think your buddies could find it helpful, and if you want to see the reviews I do, get subscribed and support the channel with a YouTube membership or over on Patreon.


💙 Enjoying the videos? Support my work on Patreon!

Product links are affiliate links – I may earn a commission on qualifying purchases (at no extra cost to you)

All my video gear


You can support me without spending a single penny!