10 Questions to Consider Before Conducting Your Employee Engagement Survey

Every day we are bombarded with all kinds of surveys, tests and assessments. But what is the difference between conducting a survey and making a prediction? Can an employee engagement survey create more problems than it solves? What makes a good survey and, more importantly, should we conduct one? Understanding employee satisfaction and employee engagement is crucial for any organization. When employees are engaged, they put their heart, soul, mind and hands into their work. This completely voluntary effort results in innovation, dedication, improved quality, lower employee turnover, improved customer service, and the list goes on.

Employee engagement surveys play an important role in understanding how this level of engagement changes over time. Their main purpose is to provide vital feedback, predict future performance and identify the factors that influence shaping it. Imagine trying to score a goal when you are in a situation where you cannot determine the distance, direction or even whether you have touched the ball or not! Trying to run a business without proper employee feedback would be just as absurd. In this article, I'll discuss a few key issues to consider in employee engagement surveys and how you can tackle them when creating your own survey setup.

While the methods detailed below are generalizable to any type of survey, I will use an employee engagement survey as an example.

Here are 10 questions to think about before your next employee engagement survey:

1. Do I need to conduct an employee survey?

2. What makes a good employee engagement survey?

3. Can a good employee engagement survey still create problems?

4. What counts as an employee survey?

5. How often should we conduct an employee engagement survey?

6. How long should an employee engagement survey be?

7. How should we design a good employee engagement survey?

8. Does it matter how an employee engagement survey is administered?

9. How should I interpret the results of employee engagement surveys?

10. Are we still not finished?

Do I need to administer an employee survey?

Yes! Still skeptical? OK then, let's consider the following "goal achievement model". When I was preparing my work plan at the university, I divided it into three main parts: the goals I wanted to achieve, the activities that would accomplish those goals, and the methods of evaluating whether the activities achieved the goals. If not, I would use the feedback from the evaluation to revisit the activities and sometimes the objectives.

This iterative process applies to any goal in any sector. Moreover, each step is necessary. Skipping the objectives stage leaves us directionless. Skipping the activities stage leaves us adrift. Skipping the evaluation stage leaves us uninformed. Without feedback, we tend to create our own realities. Often these "realities" are not true. Yet organizations that fail to get feedback often make assumptions about what they think is happening within the organization, and the consequences of these assumptions can be costly! There are, of course, some subtleties to the survey. What makes a good survey? Can even a good survey create problems? And what counts as a survey?

What makes a good employee engagement survey?

Generally speaking, there are two basic requirements when talking about surveys: reliability and validity. The first one, reliability, is the consistency of the survey in terms of administration and results. Validity is the ability of a survey to measure what it claims to measure. The vast majority of problems that arise from a survey can be categorized as violations of reliability or validity.

Can a good employee engagement survey still create problems?

Since we do not live and interact in a closed jar, the simple act of conducting an employee survey can positively change your business results. What is known to social scientists as the observer effect is sometimes a good thing. Research shows that even frequent surveys can improve performance outcomes. Other times it is less positive. If I say that the last song you sang out loud in the shower you should now sing out loud in front of your board of directors, maybe that's an apt analogy. Years ago, we had a client who came to us with a problem - they had just completed a self-made questionnaire asking how worried their employees were about being made redundant. They had been very successful in creating an internal revolt, because by asking that question, they had actually caused people to be more worried about possible layoffs.

In the aforementioned case of collecting information about pending layoffs, the very act of conducting the survey changed the outcome of the survey in a positive way; either the employees hid their true feelings and said, "Oops, I better change my answer or I might get kicked out", or they actually changed their behavior and started thinking, "They are asking about layoffs - should I be worried?". And that raised a validity issue with the questionnaire.

This is not an insurmountable problem, but it requires expertise to minimize potential problems. Another problem is dependency. This often arises with the kind of surveys that are always open. If a manager knows that he or she can read the results of the weekly surveys at any time, it is not uncommon for him or her to become dependent on these results instead of building personal relationships with employees. This is not a problem with the survey (i.e. neither reliability nor validity); it is more a problem of human nature. Unfortunately, people tend to conserve their mental resources, just as water seeks the easiest way down the mountain. The solution might be to redefine our intuitive definition of what a survey is.

What counts as an employee survey?

When we think of employee surveys, the first thing we naturally think of is programs like "the kind of paper where you draw circles around the answers to multiple-choice questions with a pencil". But this ignores the observations we make all the time. Observing a colleague's body language as you pass by is a survey. When you ask how his or her day was, that is a survey. When you interpret his or her words, it is a survey. I write a lot of articles about how fickle our intuition is and how inaccurate our gut instincts are (i.e. reliability and validity, respectively). But when I say try to stay away from making decisions on gut instinct, I also mean make use of those instincts on an ongoing basis. Ideally, a manager should test the validity of formally conducted surveys against his or her own feelings. Yet this is easier said than done. People have years of training on how best to survey a community of employees. So let's explore some best practices with their support.

How often should we conduct an employee engagement survey?

I wish I could give an easy answer for this, but it's all about context. An always-on survey allows for immediate feedback at the expense of possible survey fatigue or biased sampling (we only respond when we are really happy or really pissed off). Attempting to survey annually has the benefits of standardization and the survey being seen as more important, but does not yield immediate results. Think of surveying as a spectrum, at one end there is feedback that is created all the time, and at the other end there is feedback that is created once a year. There are pros and cons to each, and the more you move towards either end of the spectrum, the more you need to think about the disadvantages that come with it.

The important thing to remember here is that while it is tempting to create a survey that can be done quickly and easily, your ultimate goal is to create a survey that will provide valid and useful feedback. So, while you can have your employees complete a 100-item survey or a short survey every week, if they get tired or bored doing it, their hasty responses will not be accurate (you can consider it invalid). Another example is that surveys that employees can fill out at their own convenience (or not at all) tend to skew towards what is known as a bimodal distribution, where extreme views are over-represented. This is why when you read restaurant reviews online, they either praise the food or put the chef through hell. You'll hardly see anyone describing their experience as one that completely lived up to expectations.

An ideal evaluation would be a vertical design (several successive questionnaires over a period of time) that respondents would be highly motivated to answer and would not take too much effort to complete. If you are interested in measuring a change in a specific area (e.g. how performance has changed after a salary increase), then a microgenetic design (several questionnaires over a shorter period of time) may be appropriate. The principle in all cases is to capture several different data points for the same question, so that variables such as "I answered at a bad time" or "I had just returned from vacation" are more controllable for you.

Your contentious issue with what I described in the previous paragraph is probably highly motivated employees. No matter what economists tell you, offering rewards will not improve your employee engagement survey. You might think that if you offered employees a cash reward for completing the survey, they would be more motivated. Yes, but only to fill out the survey... What's more, you'll find that they fill it out faster than ever, almost as if they don't even think about it! Give it a try. So how do you make employees want to put effort (and sincerity) into their answers?

Humans are terrible at drudgery. We place a high value on finding meaning, and we crave a sense that what we do has effects beyond the behavior itself. Research shows that it doesn't take much for us to feel this way. If you want your employees to feel that filling out a survey is worth their time and effort, they need to see the results of their work. This doesn't mean buying a foosball table because everyone wants one. It means that you understand their demand and address why it can or cannot be done (money, productivity, etc.) and address the underlying concern (why they think they need a foosball table). When this happens, they will feel that filling out the survey is worth their time and effort.

How long should an employee engagement survey be?

Again, to answer more succinctly but pointing to larger principles, an employee engagement survey should be long enough to get what you want without sacrificing validity. Of course, this is also very closely related to how often you conduct assessments. The more frequent the assessment, the less comprehensive it is recommended. In the case of more frequent assessments, the biggest validity issue is whether employees will complete the questionnaire to the best of their ability. And one of the biggest pragmatic issues connected to that is, how much budget is allocated for how much time away from their work do they need to do this? Remember, I mentioned that making their efforts count is the biggest part of this. The next part is to show that interest by taking as little of their time as possible. Make the survey short and convenient. If you want qualitative responses, make sure it is a clear enough question that they don't need to interpret it. In practice, we recommend around 10 questions for frequent pulse surveys and 35-50 questions for comprehensive annual employee engagement surveys. But be careful with these numbers. There needs to be a survey that works for your organization. There is no one best way that meets the needs of every organization.

How should we design a good employee engagement survey?

My first little suggestion is: don't do it. Find an expert. Not a day goes by that we don't see a survey that was done in-house and has a terrible design. But if for some reason you don't do it, choose experts trained in survey design, I'll give you plan B. I'll try to summarize years of experience at high cost in one sentence. Find a question and try your best to debunk it. How many possible interpretations are there? Are the words constructed at a sixth grade reading level? Is it so long that your mind wanders while reading it? Is it after a specific answer? One of the most common mistakes I see is bi-purpose questions (two questions within one question). For example, "Would he be promoted with a pay raise or with better equipment?" does that mean a raise, equipment, or both? What does better equipment mean? When you say a raise, what kind of amount are you talking about? Perhaps the amount you are thinking of is insulting.

Another important point in survey design is what we call actionability. In other words, can we take action based on the results of the survey? Is the survey designed in such a way that the organization can explore strengths and areas for improvement and create action plans based on the results? If not, you may be in for a very expensive process with little return. There is an art to crafting good evaluation questions, but if you follow the rule of being your own harshest critic, you stand a good chance of creating valid test items.

Does it matter how an employee engagement survey is administered?

No, it doesn't. But at the same time, yes. In a cognitive sense, it's all about perception, and regardless of how something looks, people want to attach meaning to things. However, the nature of data entry can influence other factors such as fatigue. For your typical office worker, it is much more convenient (and therefore more likely to be completed) to fill in an online questionnaire that comes via email than to fill in a paper with bubbles with a pencil and hand it in somewhere. Conversely, it's probably more convenient for your warehouse workers to gather in the break room, fill out a paper survey, throw it in a box and have a coffee while they're on the clock.

There is no perfect way to administer a survey. The good news is that it allows you to use your creative muscles to find what works for your situation and environment. Use email, texts or paper surveys (phone surveys don't work very well). Set up a box or computer corner that your employees can easily access. Conduct paper surveys in a disciplined manner in a break room or in an environment where it is comfortable to focus.

How should I interpret the results of employee engagement surveys?

I once worked with a client who wanted to measure whether managers' competence in a metric could predict the performance of their teams. They looked at the percentage of managers with high-performing teams and concluded that there was a correlation, and they wanted to set up a program to help increase managers' competence in this skill. Now, the first question mark is to decide whether the manager's high score leads to high-performing teams or whether high-performing teams lead to the manager's high score. But putting that aside for now, the big problem was that because the size of the groups were different, using a ratio gave more score weight to a group. In other words, if I have a group of 1,000 people and another group of 100 people, when I convert the results of both groups into percentages, the scores of 100 people are 10 times higher. In fact, the correlation showed that ultimately there was no relationship between manager scores and team performance. They had acted on information that did not exist (or was not correct). Correlations are therefore highly susceptible to misinterpretation.

This was just one (far from comprehensive) example of how simple analyses can be hastily inaccurate and, at worst, completely misleading. The best practice here would be not to skimp on getting an expert trained in identifying assumptions and possible hidden pitfalls inherent in a data set, and let him or her handle the analysis and interpretation. This is basically the same reason why I didn't have my knee surgery done by my neighbor, despite the fact that he was within easy reach and very skilled in his positive approach.

Are we finished yet?

There are three lessons I want you to take away from this. First, evaluations and surveys are critical for improvement. All learning in human history is based on getting feedback and using that feedback to predict future behavior and events. Second, surveying is a skill that can be greatly improved when expertise is brought to bear, making the results more realistic and very useful. While everything is better served when an expert is involved, there are many factors that can improve your employee satisfaction surveys from day one. And thirdly, it is important to think about and consider surveys not in terms of their feasibility, but in terms of their validity. Just because a survey is feasible does not mean it should be done. The market is flooded with companies that produce employee surveys and their useless results. Remember, a survey is of little use if the organization cannot take action on the results. But if you think of "validity" as the pole star and use an effective and actionable survey, you have one of the most powerful tools to affect the change you want in your organization.