Materials hero image

Materials

Question development

The consultation questionnaire will be the primary mechanism for gathering feedback on your proposals and as such it will play an instrumental role in gathering the views of consultees and in delivery of a meaningful outcome from the consultation. Remember that public consultation is meant to be ‘a dynamic process of dialogue with the objective of influencing decisions, policies and programme of action’, and it is as important to understand why people believe (and which people) something to be an issue as whether they agree or disagree with the proposals.  The consultation questionnaire must be presented in such a way that enables such a process of dialogue and explanation.

A good questionnaire should be easy to navigate. Questions should be set out logically, so the respondent can clearly follow the narrative (for example, reflecting the number of proposals being presented). Placing questions in logically organised sections is helpful, together with summaries of the proposals at each stage.

Be clear about what information is being requested, and keep to an appropriate number of questions, so as not to create a barrier to people responding. Although your questionnaire may contain ‘closed’ questions (that require respondents to tick pre-formulated answers), there should always be a degree of openness that allows respondents to express their own views in their own way. An overly detailed and confusing questionnaire can lead to frustration and/or complaints and may cause significant problems for the consultation.

Recognising that consultation questionnaires need to be tailored to the specific circumstances of each change sponsor and the proposals under consideration (as well as operating effectively through the CAA portal), ICCAN is not proposing to offer specific advice on the drafting of individual questions.  We have however created some guiding principles that could help in developing an effective online questionnaire.

Below are the guiding principles for developing questions for an online questionnaire:

The most important principle when constructing a questionnaire is to ensure that it is rooted in the world of the respondent – that it is easy for them to understand, and that it follows the way they think about issues. This goes beyond mere language (and the care taken not to include jargon or technical terms) but extends into the way respondents express themselves over the issues, and how they see things.

Many of the reasons for questionnaires failing to capture data properly is that respondents do not understand the questions in the way that the person writing them does, and this leads, either to respondents giving up on the questionnaire, not responding to questions, or there being a disjuncture between the mindset of the consultor and the consultee in the answers given.

During Stages 1 and 2 through the engagement undertaken with representative stakeholders and before even considering a questionnaire, a consultor should have undertaken some preparatory work to understand the world of the consultees: what their concerns are; how they express them; the areas where different types of respondent have different views (and therefore a more quantitative exploration via questionnaire is required). Pre-consultation engagement work, along with its many other benefits, provides this opportunity, and consultors need to use this period to consider these issues and to feed them into the process of questionnaire development.

This will allow a questionnaire to be more user-friendly, target issues of concern, and help a questionnaire writer provide tick-box options to closed questions that accurately reflect views, as well as having an understanding of the direction and structure of more open-ended answers for subsequent coding and analysis.

As a change sponsor, you need to be clear about what information you need to gather to support your decision-making process. Once this has been clearly defined, you can decide on the type and nature of the questions to be asked. Consider the following when planning the consultation questionnaire:

What are the scope of the proposals, and what is open to influence (i.e. from Consultation Mandate)

What does the change sponsor need to understand to be able to make an informed decision, for example:

  • What aspects of the proposals are supported and why?
  • What aspects of the proposals are not supported and why?
  • How might the positive impacts of the proposals be maximised?
  • How might the negative impacts of the proposals be minimised?
  • Have might equalities impacts be appropriately mitigated?
  • Are there other options or ideas which should be considered?
  • Are there any impacts which have not been adequately covered?

By considering these questions, in conjunction with the information gleaned from earlier engagement, the change sponsor can identify the information that needs to be collected and provide a focus to the questions asked.

Change sponsors should be careful not to limit the scope of questions asked, so that people feel they cannot raise issues which they believe are important to them (see below).

Before designing the questions, it is important to clearly define the scope of information that is required to support the decision-making process – a useful question to ask is ‘what information are we trying to gather, and for what purpose (in terms of informing our decision- making process)’ and from whom?

The proposals should be at a formative stage, so the consultation provides an opportunity to collect further information both directly and indirectly regarding the people who will be affected by the proposals and the likely impacts.

A further useful consideration at this stage is to be clear about what aspects of your proposals are open to influence, and what information is required to help inform and shape your proposals. Public consultation is an opportunity to identify and understand in more detail the drivers for agreement and/or disagreement with the proposals, and to identify whether there are other options or issues to consider which could help shape the proposals going forward.

There are three main types of question and their use depends on the information you are seeking:

Closed questions: these are designed to ensure a respondent must choose from a pre-set series of answers.  This can be useful in terms of analysis and gauging the level of support for a proposal as they insist on a choice being made – although remember, the consultee is being asked for their views, not being forced into a vote.

It is generally considered good practice that close questions provide a further option of ‘don’t know’. Closed questions will often be followed with an open question, asking for the person to explain their reasoning for the decision given, or to expand on/contextualise their answer.

Scale questions: questions involving ratings or scales can be used to provide insight in to the level of support for a proposal. When using ratings / scales, it is often useful to have a neutral option as well as the positive and negative, and again the option for ‘don’t know’. There are no specific requirements in terms of how many rating positions should be provided, however a typical example uses:

  • A very poor option / Strongly disagree
  • A poor option / Disagree
  • Neither a poor or a good option / neither Agree or Disagree
  • A good option / Agree
  • A very good option / Strongly agree
  • Don’t know

Open questions: allow respondents to provide a full, informed response in an open space. Although they can be harder to analyse, they enable people to provide more information and the reasoning for their decision.

However, be careful, of making the question too open, e.g. ‘Please provide your views on the proposal’ – you run the risk that consultees will leave a longer and less focused response which covers many topics, and this can make analysis complex and difficult. Try to provide a focus for each question, for instance: if this proposal will impact you, either positively or negatively, please could you tell us how, and why you think this? It is much better to have several focused open questions (e.g. either following closed questions, or at the end of each section) than to rely on one large open question at the end of the questionnaire.

Using a combination of different types of question, can gather additional information that can help to pinpoint specific issues which need to be considered during the decision-making process.  

It is important that people can express everything they want to about the proposals, so it is good practice to include a final catch-all question that covers this – but see above, do not use this as the only open question.

By including a question such as ‘is there any other feedback you would like to provide?’, or ‘are there any other issues you wish to raise?’ you provide the respondent with an opportunity to leave their comments outside of the boundary of the set questions. This means that the questionnaire gives everyone the opportunity to express their opinions on the issues that are most important to them and ensures that the questionnaire is robustly prepared.

It is key that the language of the questionnaire should be consistent with that of the consultation materials, so try to use clear, straight forward phrases, and avoid using acronyms and technical jargon as far as possible. Be guided by the first principle above and root the language of the questionnaire and concepts used in the world of the consultees. Using clear language will help to avoid confusion and ensure that respondents are able to understand what they are being asked.

The below suggestions could be adopted to help create a more accessible questionnaire:

  • Use neutral language so that the options are all presented in an open and consistent way and avoid leading questions so that people can provide their responses without any accusation of bias.
  • Avoid using prompts within the question as this could potentially prejudice the response. If you want to know what someone thinks about a specific issue, tailor a specific question about it and ensure that the language used is neutral, for example: ‘Is there a particular issue that concerns you about this proposal such as the frequency of flights or respite periods?’ compared with ’Is there any feedback you would like to provide about the proposals for planned respite?’
  • Simple terms and questions make the questions asked much harder to misread. Using phrases that could be open to interpretation will make it more difficult to fully gauge what people’s views of the proposals are when analysing the feedback.

Ensuring that the questionnaire is created using plain English will also mean that those groups identified as part of the seldom heard strategy should find it more accessible and approachable.

Throughout the design process it is important to remember that the respondent should have a clear understanding of why the consultation is taking pace and what is being asked of them, so questions need to be relevant and clearly presented.  

 As the questionnaire will be hosted on the CAA portal, it will sit within the Citizen Space platform, and this will provide the flexibility to create a structured and easy-to-navigate questionnaire, enabling the respondent to understand which sections are most relevant to them, and what data sets are relevant to answer each question in an informed manner.

The structure of the questionnaire should appear logical and flow as much as possible; the information gleaned from earlier engagement exercises will help you to put yourself in the respondent’s shoes and run through the questionnaire from their perspective. 

Some other things to consider:

  • Gathering personal/demographic information about the respondent can help track against your consultation plan, so that you’re able to map against areas of high / low response. Asking for postcodes and organisation details can help identify if there are any geographic areas or groups that will require top-up engagement before the end of the consultation. Such information will also help you to segment the responses, so that you can see, for example, if there are any differences in response by postcode or by age. Collecting personal information must adhere to GDPR rules, and you must be clear about the use to which the information will be put.
  • The questionnaire should be as clear and focused as possible. Avoid the use of ambiguous terms which could be interpreted differently by others. Remember that people will be responding to questions without the opportunity to clarify the scope and purpose of the question.
  • Although you will have presented the information as clearly as possible in the consultation materials, it is considered helpful to provide a clear introduction to each question to summarise the proposals and clearly explain what you are seeking from the respondent.
  • Create sections that reflect the layout of the consultation materials and act as clear signposts for respondents. In this section we are seeking views on flight XX, further information about these proposals can be found XX

It is often useful to test a questionnaire before launch with independent audiences and different user scenarios to ensure that you cover as many perspectives as possible.