Let’s look at the evidence – the sex question in Scotland’s 2021 census

The Census (Amendment) (Scotland) Bill recently returned to Holyrood for its Stage 3 debate (12 June 2019). Parliament agreed that the 2021 census should include voluntary questions on sexual orientation and transgender status/history. However, debate around the wording of 2021 questions remains unresolved as Joan McAlpine MSP called for the sex question to be ‘based on biological sex’, a move that would change the meaning of the question asked in 2011 and force people to answer in a way that went against their lived experiences.

I have previously written about how research has found that a non-binary sex question does not pose a risk to data quality in terms of respondents’ understanding of the question, response rate, question invalidation or tampering.

In this post, I want to also rule-out the asking of a two-step question, which asks a binary question on sex followed by a non-binary question on gender (as presented in the image below). I will refer to this as a sex/gender question.

There exists major concerns with a sex/gender question. For example, in-depth interviews conducted as part of National Records of Scotland’s research into the 2021 census sex questions noted a privacy risk, where the combination of an individual’s answers to a binary sex question and non-binary gender question could potentially ‘out’ a respondent.

This post particularly focuses on the results of quantitative testing commissioned by NRS and conducted by research company Ipsos MORI in 2017, presented in the report Scotland’s Census 2021: Sex and Gender Identity Topic Report.

Ipsos MORI used a sample of randomly selected residential addresses drawn from the Scottish Address Register to invite 15,579 people to participate in their research. One third of the sample were sent a survey that included a binary sex question, one third a non-binary sex question and one third a sex/gender question.

Ipsos MORI wished to determine whether the questions asked impacted people’s likelihood to partly or fully complete the census (in other words, answer the questions online or return the paper version by post).

Research found no impact on response rate:

• The binary sex question had a response rate of 35%.
• The non-binary sex question had a response rate of 36%.
• The sex/gender question had a response rate of 35%.

Ipsos MORI conducted further tests and found no statistically significant interactions between questions asked and response rate.

From this we can see that the type of question asked did not impact people’s likelihood to complete the census (if anything, a non-binary sex question had a slightly higher response rate).

Looking at the submitted/returned censuses in more detail, Ipsos MORI then explored whether the type of question asked impacted people’s likelihood to respond to that specific question (in other words, whether people answered that question within the census)?

Results from this testing found that the level of non-responses to the sex/gender question was significantly higher than non-responses to the binary sex question or to the non-binary sex question.

Ipsos MORI concluded that respondents were less likely to fully answer a binary sex question followed by a non-binary gender question, as noted below:

The accuracy of census data is paramount. That is why we must ensure that the addition of new questions, revision of response options or change to question meaning (for example, whether a question asks about ‘biological sex’ or ‘lived sex’) does not negatively impact data quality nor force respondents to register themselves in a way that fails to reflect the reality of who they are.

I am alarmed by the scant references to this major piece of research, undertaken by a national non-ministerial office, on the specific issue of the sex question in the 2021 census. NRS’s 80-page report is freely available online and, thus far, the most rigorous research conducted in Scotland on the framing of the census sex question.

Those with an interest in the 2021 census all wish for it to remain the ‘gold standard’ in terms of data collection. We must therefore look at the findings from established statistical agencies and research organisations, such as NRS and Ipsos MORI, to ensure that any developments do not present risks and are based on rigorous evidence rather than hyperbole.

Dr Kevin Guyan is an equality, diversity and inclusion researcher based in Edinburgh. He is writing in a personal capacity.