April 11, 2022 | by Scott Bahr
Working in research, one of the tenets we live by is to try and ensure our work can withstand scrutiny. And as researchers, we should be able to take the hits that can come once we put our work out there for this scrutiny.
When I worked in an academic research setting, every Friday the department would have someone present the results of their research (or the status of their research) in a roundtable setting. Once the presentation was over, the participants would offer critiques and questions about the research. These could vary from a simple question about instrumentation to a more harsh critique of the analytical approach. On one occasion, a doctor who presented was taking the typical heat from the table and at one point, began to get visibly upset at the critiques. The department chair looked at him and simply said: “son, you should probably reconsider whether you want to be involved in research.”
This may seem harsh, but it illustrates the position we place ourselves in, especially when we put our work out into the public for scrutiny (or even at the private organization level at times). People are going to look at our work and decide what level of credibility to assign the results. Sometimes this criticism can come from relatively uninformed sources – “you can’t base anything from a sample of that size” – to the more informed – “how did you screen your survey participants?” – to the much better informed – “what is your sample source, how was that list compiled, did they opt-in, was it balanced?”
In all cases, there is an understanding that we should at all times be prepared for the questions and questioning, and if it’s a competitor, then we need to be even more prepared to ensure that what we have done is buttoned up.
How we decide to respond
But as the title of this summary suggests, this isn’t about those external criticisms necessarily, but in how we decide to respond to the work of others and also how we can respond to the criticisms of others when the level of professionalism may be – ahem – lacking.
“It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, who comes short again and again, because there is no effort without error and shortcoming; but who does actually strive to do the deeds; who knows great enthusiasms, the great devotions; who spends himself in a worthy cause; who at the best knows, in the end, the triumph of high achievement, and who at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who neither know victory nor defeat.” – Teddy Roosevelt
Point Number One
The quote is a bit over the top in the context of market research, but you get the point. Let us first consider the manner in which we deliver criticism. My first piece of advice: keep it to yourself. If you see a study that may not have adhered to the standards that you feel it should, unless asked to do it, being paid to do it, or if the study itself could do someone harm, it’s best to simply take note and move on. Use the instance as a way to identify what could have been done differently, how they might have approached the project more effectively, and use it as a learning experience. It can also be helpful to you in how you work with your clients by offering you the opportunity to speak with your clients about methodology and approach and why you do things the way you do.
Point Number Two
My second piece of advice is to keep track of what others are doing and how they do it. Many times the methods of gathering information are not shared, thus a knee-jerk reaction to something might be due to a simple omission of approach, or importantly, history. If you know the history of the organization, the work they produce, and their clients, then you will have a better idea of their portfolio and enable you to better understand what it is they are about.
Point Number Three
My third piece of advice is to consider reaching out to those who are publishing the work. What may first appear to be an egregious mistake, may have been more purposeful than you had originally thought. Information that is pushed out publicly is sometimes (always) picked over and only certain parts of the results are being published (and in many instances, by people who don’t necessarily understand the context). Mixed methodologies are popular for some organizations, others might be using a more narrow band of respondents who are part of a unique set, the results might be part of a larger study and only a portion of the results are available. Additionally, as much as we might try to maintain control of how information is framed when put out to the public, those who write the final copy may not always understand the nuance in word choice that we, as researchers, jump all over when used in a way we don’t see as appropriate for the situation. Which is a long way of saying, take the time and reach out to the organization that put the information out and ask some questions. Engage in a dialog before you decide to be a critic (and hopefully avoid the need for an embarrassing mea culpa).
In my opinion
My other piece of advice on offering criticism is to stay away from social media if you’re going to be a critic. Just don’t.
My final thoughts surround being on the receiving end of criticism. And this is the most difficult part. When you receive criticism, the first thing you should consider is how this impacts you and your organization. Was the criticism made in public (see: social media)? Does it impact your relationships with your clients? Who is the source of the criticism? How severe is the threat to the results of your research is this criticism? If the criticism doesn’t impact your business, then maybe the best thing to do is to simply ignore it. This can be difficult, but in most cases, it’s the best course of action. However, it is incumbent upon you to address the criticism and identify whether this criticism is valid, and ultimately impacts your results.
If you discover that it’s a rounding issue, you may or may not need to say something, but typically, that’s not something you need to get into the weeds on. If you have a more severe error (e.g., sampling, inaccurate data, mislabeling), then you may have to address the issue in a more open manner, but this is something you need to think long and hard about in terms of how to address it in the open. If it is truly a result that impacts what you’re trying to say, then you are going to have to be open and honest and hope for the best in terms of others understanding how this can happen. If you get too defensive and don’t have a great way to back it up, you can severely damage your reputation and client relationships. Take your lumps.
Valid Criticism and Professional Decorum
If the criticism is in your mind not a valid criticism, then it is incumbent on you to jealously defend your results. This is your job, your brand, your reputation and if you don’t defend your results, then you are at the whim of others. Once doubt is introduced into your work, you risk losing credibility for all your work and as a researcher, this is the kiss of death.
If you have done everything correctly, your methods are solid, and your conclusions buttoned up, then stand up for yourself. Don’t engage in a back and forth. It is important to forcefully defend your position and establish your standing as an expert.
You know, research. You know your results. Make sure everyone knows this about you.
It’s a great time to be in market research.
Set up a chat with our team and let us help you build your next market strategy using our plethora of knowledge and depth of experience.