A non-binary sex question in Scotland’s 2021 census

I watched with interest the Census (Amendment) (Scotland) Bill’s Stage One debate (28 February 2019) and was pleased to hear many MSPs affirm the importance of the data quality captured by the census. For example, Claire Baker noted that the census ‘should enable a better understanding of the population over the years, as it is collected’. Stuart McMillan based his views on ‘three points: the ease of gathering the data; how the information gathered will be analysed and used; and the consistency of data gathering’. Alexander Stewart stressed that ‘it is vital that we have a rich set of data’ and that ‘it is necessary to have accurate data’.

Joan McAlpine explained that the Culture, Tourism, Europe and External Affairs Committee Committee voted ‘by a majority, that the sex question should remain binary in order to maximise response rates’. The rationale for this decision did not align with research commissioned by National Records of Scotland (NRS), which found that a non-binary sex question returned a slightly higher response rate than a binary sex question.

During the debate, Stuart McMillan noted his surprise to ‘read in the cabinet secretary’s letter that the NRS testing seemed to indicate that a non-binary question would lead to a higher response rate’, and asked for further information from NRS on the result from their testing.

To address these points, this post presents further information on NRS’s testing of a non-binary sex question.

Question testing

NRS published Scotland’s Census 2021 Sex and Gender Identity Topic Report (2018), a comprehensive report on testing conducted by Ipsos Mori and ScotCen Social Research, two of Scotland’s leading research organisations.

Cognitive testing involves in-depth interviews with a small number of respondents to explore the mental processes respondents use when answering questions. This testing helps to identify any problems with question design or difficulties encountered by respondents. Quantitative testing examines, to what extent, the quality of data collected depends on the questions and response options used.

Both cognitive and quantitative testing are extremely important methods that provide valuable information on whether changes to census questions will impact data quality.

This post focuses on results from the quantitative testing of a non-binary sex question in the 2021 census and its potential impact on data quality and census respondents.

Research findings

Between June and August 2017, Ipsos MORI conducted quantitative testing of binary and non-binary sex questions. Using a sample of randomly selected residential addresses drawn from the Scottish Address Register, 15,579 respondents were invited to participate in the testing. One third of the sample were sent a survey that included a binary sex question, one third a non-binary sex question and one third a gender identity question set (which is not discussed in this post). The question wordings are presented below:

Ipsos MORI concluded:

• ‘The inclusion of the non-binary question does not significantly increase the level of item non-response in comparison to the binary sex question’ (p.45).
• ‘Regardless of the version of the sex/gender identity question received by respondents, there was no evidence of invalidating or tampering with these questions in Scotland’ (p.45).

In fact, a slightly larger proportion of respondents responded to the non-binary sex question than the binary sex question (p.36):

Statistical analysis found no significant interactions between the questionnaire version and response rate (p.37).

In Autumn 2017, NRS commissioned ScotCen Social Research to conduct further testing of a non-binary sex question. The question wording is presented below:

3,300 addresses in Scotland were randomly selected and invited to take part in the research. From this total, 1,530 responses were received (645 online responses and 809 paper responses). Further information on ScotCen’s methodology is presented in Scotland’s Census 2021, 2017 Cognitive and Quantitative Testing (2017). Quantitative testing found that:

• When asked to indicate if they found any questions difficult to answer and, if so, provide feedback in an open text box, less than 1% of respondents stated that they had found the non-binary sex question difficult to answer (p.74).

As demonstrated by Ipsos MORI and ScotCen’s quantitative testing, the introduction of a non-binary sex question did not negatively impact the census response level, respondents’ understanding of questions, or invite question invalidation or tampering.

Why is this important?

The Cabinet Secretary for Culture, Tourism and External Affairs, Fiona Hyslop, stated, ‘The census does not lead public opinion; the census has to reflect society as it is just now and ask questions that maximise the response rate so that the data can be used’.

NRS testing found that the inclusion of a non-binary sex question had no impact on the response rate and was understood by respondents. Furthermore, we must remember that non-binary respondents (ie people who do not define as either male or female) are currently unable to accurately respond to the sex question. The inclusion of a non-binary sex question will enable non-binary people in Scotland to register themselves in a way that accurately reflects Scotland’s population in 2021.

In conclusion:

• Ipsos MORI’s testing of 15,579 randomly selected residential addresses in Scotland found that a slightly larger proportion of respondents responded to the non-binary sex question than the binary sex question.
• From the same testing, Ipsos MORI found no evidence of question invalidation or tampering for the non-binary or binary sex questions.
• ScotCen’s testing of 3,300 randomly selected addresses in Scotland found that less than 1% of respondents stated that they had found the non-binary sex question difficult to answer.

Dr Kevin Guyan is an equality and diversity researcher based in Edinburgh. He is writing in a personal capacity.