Has Technology Changed Your Surveying Practices?

A review of discussions about technology and surveys in CAE program courses has me convinced that a major upheaval is occurring in member information gathering. Why is that? Back in 1997, course discussions indicated that the typical association undertook a major member survey once every five years. This was usually the extent of formal information gathering conducted by associations to help them understand member needs and expectations.

Seven years later it appears that major member surveys are occurring more frequently, often within a two or three year time frame. In addition to major member needs surveys, a lot of special surveys are occurring - these are usually issue based and quite short. More associations are also getting on the phone to reach out to key segments of their membership to probe in-depth issues and explore product and service possibilities. As we all know, many members won't complete a paper or on-line survey – many associations find their members are willing to express their needs and wants on the phone to independent surveyors. Those that use this method tout the quality of the responses they receive.

Why the shift in practice? Technology is making it possible to support "just-in-time" member input and accountability. Associations are reaping the benefits of reduced survey and administration costs directly linked to continuing improvements in technology. In the case of phone research, targeting calls to about five respondents per member segment doesn't break the bank and probing beyond standard questions provides valuable information.

Smaller associations, never able to afford independent or self-administered surveys, are now using low cost online packages that for $500 or less will get them information from members they never had before.

It appears members, through the use of online surveys, now have more say in the direction and priorities of their association. This raises a lot of questions. Which members are having a say? Will increased frequency reduce response rates? Will a select group that always responds move the association in a direction not consistent with the silent majority? What impact is this having on board and staff leadership initiative? Is the association becoming more reactive? Perhaps it's necessary to step back and look at the source, quality, and use of survey information.

Associations now realize they need to reach out beyond the ten percent in their organization that are most likely to participate. Finding out the real needs of the ninety percent which represent other types of member segments will improve decision making and help the association achieve its core purpose. One student noted that information from the responses of the ten percent "most likely to" is just information - input from the other ninety percent will help develop real knowledge to support decision making.

There is no doubt that technology is enabling new approaches to surveys. While lower costs are always welcome, it should not sacrifice quality. Perhaps policy around survey types, timing, intent, etc. is required to ensure a balanced and strategic approach.

I have a chance to review many completed member surveys, and despite the increase in survey activity I have noticed the key question about overall member satisfaction is often missed. This is a key performance measure that needs to be obtained to allow for internal and external benchmarking activity. In addition, some of the other questions being asked could have benefited from greater thought. If you have limited training in this area, seek the advice of a research professional to comment on the quality of the questions you are asking – this could prevent some embarrassing moments and improve the quality of the information you generate.

Is it possible to get a one hundred percent response rate? The answer is yes. One association indicated they achieved 100% rate. The good rate was attributed to member involvement with the strategic plan. Other helpful suggestions include keeping things as short as possible without impacting on quality and making it easy to respond with minimal clicking.

Those who have embraced the online survey are quick to point out the following advantages:

• Lower costs
• Easy to create
• Getting it out is a few clicks away
• The potential for response rates is high
• The crunching is done within minutes
• Distribution of results is easy

On the other hand there are some things to watch out for such as:

• Open responses are limited as some people have concern about security and confidentiality
• It is not always easy to customize a low cost package
• Some survey packages have limits in terms of responses – a large response may incur a surcharge
• A current email address may not be available for all members nor are all members able to access high speed connections that make downloading the survey instrument easy
• The survey is so easy to create that not enough time is spent on question development and the relationship of the questions to the decisions that need to be made

Other notable observations and lessons learned:

• Even where membership is mandatory, members have opinions and want to provide input - boards and staff need to understand the satisfaction level
• Telephone interviews to a variety of member segments by an independent produces high quality information to support future decision making
• Whether your managing online based surveys in-house or outsourcing, it is important to be sure security needs are met
• Being entered into a draw if you complete a survey is not a major incentive
• Get expert advice on question development and keep the number as short as possible without sacrificing the intent and quality of the survey
• Share the analysis with others as you may be too close and miss key trends or issues
• Let members know when it is coming, when it is available, and what the results were

This information suggests we need to spend more time on our strategy and policies around member strategic information gathering.

Cam Davis, CMRP a Past President and Instructor with the Professional Marketing Research Society and Vice President of Pollara says, "the quantum shift to both qualitative and quantitative market research on the Internet has realized incredible time and cost savings and has revolutionized the business".

Davis suggests associations will need to spend more time on strategy and policies around member strategic information gathering. Given the changes occurring, now is a good time to start making improvements in this area.

This column features innovation and practical solutions applied to challenges, trends, issue and opportunities for the association community. Column editor Jim Pealow, MBA, CMA, CAE is a consultant and the Association Management Education Program Lead Instructor/Coach for CSAE. He can be reached at jim@amces.com