Incomplete data leads to incomplete conclusions

At Senate Estimates I asked the Australian Bureau of Statistics about the accuracy of the data they publish.

Many Australians, politicians, government officials and media should be watching the ABS data for signals that there could be a problem with our COVID response. Births and deaths would be the main indicators.

The ABS are slow in producing this data and don’t appear to understand that these datasets should be produced faster than pre COVID times.

In addition, the ABS has been loading incomplete data and not labelling it as such. After this was pointed out to them during our last senate estimates, the dataset referenced was changed to include the label “incomplete”.

How many other datasets are labelled as final when in fact they are incomplete?

The answers showed that the data for Provisional Mortality only includes doctor-certified deaths (which we knew) but that the comparison baseline includes ALL deaths, including coroner-certified deaths (which we didn’t).

This means the ABS has not been comparing apples with apples, and the figure for Provisional Mortality understates actual deaths by 15%.

What this means is that unexplained deaths in Australia is over 30,000 in 2022. Around 10,000 of those are attributed to COVID.

What are the other 20,000 deaths?

Transcript

Senator Roberts: Thank you all for appearing today. My first questions go to accuracy of data. In the last estimates session, we had a conversation around the accuracy of one of your datasets. I want to follow up on that.  The dataset is births by year and month of occurrence by state. It’s available in your Data Explorer. The conversation was around the reduction in births shown towards the end of 2021, and that reduction was quite dramatic. I accept your position that this effect is caused by delays in reporting of birth, and a lot of December’s reports came through in January. Is this correct so far?

Dr Gruen: That is correct. There’s a pattern, which is repeated every year, which is that the first unrevised estimate of births in December is of the order of 6,000 or 7,000, and then, once you have the final numbers, the final numbers are of the order of 22,000 or 23,000. So, there is an enormous revision for precisely the reason you just mentioned—namely, not everyone has recorded the birth of their child. I think they have other things on their mind than making sure that the ABS gets its numbers right.

Senator Roberts: The dataset is titled ‘birth by month of occurrence’, not ‘births by month of reporting’.  2021 data was not available until 19 October 2022. Why was 10 months insufficient time to completely compile the full 2021 calendar year? I note that December is still showing 6,600 births against an expected 20,000 in your Data Explorer, as you’ve just said. Why is this data still incomplete 17 months later—and still wrong?

Dr Gruen: It’s unrevised; I wouldn’t use the word ‘wrong’. The answer is we have a schedule of births which has been the same schedule for an extended period. We haven’t yet got the revised numbers for 2021, but, when we do, we have a pretty good idea of the order of magnitude that they’ll be. This hasn’t changed. We’ve be doing it on this timetable for many years.

Senator Roberts: The database now carries a warning—thank you for this—’incomplete data’. Have you made a note of where else incomplete data is being loaded into your Data Explorer and ensured incomplete data warnings are attached as you load that data?

Dr Gruen: We provide preliminary data for a range of series, and we did more of that during COVID because we thought it was important for people who were making decisions to have the most up-to-date data that they could possibly have. So, we brought forward some releases, understanding that they would not be complete, and we were transparent about that. It is certainly the case that revisions are part of producing statistics, whether it’s births or the national accounts. The national accounts also get revised. It’s a common feature. We do not revise the quarterly CPI because there are legislative indexation arrangements. Again, it’s a longstanding practice that we do not revise the CPI, but, for many other series, revisions are a standard practice.

Senator Roberts: I don’t think anyone would complain, Dr Gruen, about data needing to be revised.  Maybe the speed of it might be something we might inquire about, but what I was getting to was: are there any other datasets on your Data Explorer that need the words ‘incomplete data’ as a warning? Bad decisions are made off bad data, and it becomes misinformation. 

Dr Gruen: I don’t think it’s misinformation. We are as transparent as we can possibly be about the nature of the data. For instance, we put out provisional data for deaths, which we have actually discussed in previous estimates hearings.

Senator Roberts: Yes.

Dr Gruen: That is based on the available information two months after the end of the reference period, and those are also revised subsequently. When we first started producing that data, again, that was during the early phase of COVID. We did it purely on the basis of doctor certified deaths, which is about 80 to 85 per cent of overall deaths. We’ve managed to include some coroner certified deaths in that series, but it’s still incomplete when it’s first published two months after the period. So there are several datasets where we are very clear about the fact that they’re not the final data and that extra data will come in for the period that we’re talking about.

Senator Roberts: I’m advised that the incomplete data warning arrived after our session last time.

Dr Gruen: That is possible.

Senator Roberts: So I’m just wondering if there are any others. The dataset ‘Causes of Death, Australia’ for calendar year 2021 was released in October last year. Can you confirm that 2022 will be released no later than October this year?

Dr Gruen: I’m sure there’ll be someone here who can tell you for sure. Around October is when we publish the annual data for the previous year, but we can take that on notice and give you an answer, for sure.

Senator Roberts: The provisional mortality figure is still showing that deaths are running above the previous known range. Has the ABS received any request from any minister or department—federal or state—for an explanation of where the increase is or what data the ABS has which could cast light on that substantial increase in mortality?

Dr Gruen: We do talk about provisional deaths, and we do talk about what proportion of those are people who died with, or of, COVID and from other causes, so I don’t think there’s a mystery about what is happening.  We get lots of requests for our data, so I can’t answer the question. Since it’s on the website—

Senator Roberts: They wouldn’t need to ask you.

Dr Gruen: That’s right.

Senator Roberts: I was just wondering, in particular, whether Health had asked, but, as you said, they don’t need to. Do you send reports routinely, or do you just publish on the website?

Dr Gruen: We publish, and we answer media inquiries. We have outposted people in many of the departments in Canberra, and we have continuing discussions with them. If a department had a specific request, it would be straightforward for them to ask us.

Senator Roberts: There’s a disparity between datasets that I would like to ask about. Starting with the publication ‘Provisional mortality statistics, Jan 2020-Dec 2021’, which was released on 30 March 2022, the key statistic is that 149,486 doctor certified deaths occurred in 2021. If I then go to your Data Explorer, the figure for ‘Deaths and infant deaths, year and month of occurrence’, shows deaths in 2021 to be 160,891.

Dr Gruen: Is the subsequent number published? The number you first quoted is the number that was available from doctor certified deaths up until the end of March, and then the second number you quoted comes from more recent data. Is that correct?

Senator Roberts: I don’t know when that was published, but it shows deaths in 2021 to be 160,891, which is higher. So, I understand the difference in deaths because some would be autopsy certified and take time to come through; is that correct?

Dr Gruen: Yes, that’s right. As we say when we publish those provisional death numbers, they are provisional. They are the data that we have available on the date at which we finalised the numbers. As I said earlier, doctor certified deaths are something like 80 to 85 per cent of all deaths, so the number goes up when you add the coroner certified deaths.

Senator Roberts: It includes the autopsies. Is the figure on this graph for the baseline average calculated using provisional mortality or using final data from the ‘Causes of Death, Australia’ dataset?

Dr Gruen: We can check, but I’m pretty confident that it’s final.

Senator Roberts: Would that then include autopsy deaths?

Dr Gruen: Yes.

Senator Roberts: Provisional mortality is a widely shared dataset that informs much debate around our COVID response. It’s running well above our historical range. From today’s exchange, we know that the figure for provisional mortality understates actual rates of mortality. Your dataset does make that clear, so this isn’t a criticism.

Dr Gruen: No.

Senator Roberts: What I would like to know is: by how much does provisional mortality understate actual mortality in percentage terms on average? I think you’re saying 85 per cent?

Dr Gruen: I think the number that we get two months after the reference period is about 85 per cent of the final number.

Senator Roberts: I’d like to go briefly to data collection. A constituent of mine in Queensland has contacted me in person during a listening session in Rockhampton just recently. This elderly lady, who is single—widowed—and lives alone had a terrifying interaction with the Australian Bureau of Statistics that raises questions about either the staff training or your understanding of the fair exercise of power. The ABS maintained a dataset called the National Nutrition and Physical Activity Survey, which apparently involves Australians being selected at random to participate. The survey consists of an Australian Bureau of Statistics officer visiting the selected person’s home and taking their height, weight, blood pressure and waist measurement, which is compulsory. Then the citizen has the option of submitting a voluntary blood and urine sample. Is that correct?

Dr Gruen: I think so. I think that is correct.

Senator Roberts: The constituent in this case advised the ABS worker that she lives alone. After receiving a series of letters they thought was a joke, an ABS field worker came by her home in the dark at 6.30 pm, showed her credentials, asked for her by name and advised that the constituent must submit to the government mandated physical. When the constituent declined, she was threatened by your worker with a fine of $220 per day until she submitted to this physical examination by a complete stranger. Is that how the ABS runs its survey?

Dr Gruen: Well, I can’t comment on a specific event. We obviously do our best to treat people in a dignified way. It is true that the surveys that we run are compulsory, but we also allow for the possibility that people who have extenuating circumstances can apply not to be part of the survey, and people do do that on occasions. It is important, in order to be able to collect data that is representative, that we can indeed choose a representative sample, but it is also true that, for people who are in circumstances in which they find it particularly difficult or who are in the circumstances that you described, we are understanding.

Senator Roberts: That goes to my next question. Why can’t you get this information from hospital records for admitted patients with de-identified data? Why pull names out of a hat, knock on their door, call out for them by name and terrorise them into submission? It seems like a massive overreach when there are alternative ways of doing it. Maybe the alternative ways are not entirely random, but they could be made so, couldn’t they?

Dr Gruen: Just to make it clear: our aim is not to terrify people.

Senator Roberts: This lady was terrified.

Dr Gruen: Well, I’m sorry about that. We obviously train our interviewers to be sensitive to people. On the general issue of being able to find alternative ways to get the data, we are very much alive to those possibilities.  What you’re talking about is an example of using big data instead of surveys, and there’s a worldwide move from national statistical offices to do precisely that both because the big datasets that are becoming available—there are increasing numbers of them. For instance, early in COVID we started using single-touch payroll from the tax office to be able to give high-quality, up-to-date information about employment. That’s an example of a big dataset. But it is also true that response rates around the world are falling because people are, for whatever reason, getting less happy to respond to the surveys of the national statistical offices. That’s another push factor to lead us to do precisely what you’re suggesting. Now, we haven’t accessed the particular dataset that you have talked about, but the general proposition that we are moving in the direction of using big data and taking the burden off individuals and businesses is very much a journey that we’re on.

Ms Dickinson: For some of the surveys that we run, there are not alternative sources that we could avail ourselves of, and the survey that you referred to—the nutrition survey—has quite a range of questions that we ask people before we come to the physical measurements. It’s things like diet. We ask people to recall what they have eaten and sometimes do a food diary. That’s the type of thing that we can’t get from big data and in which there’s quite a range of interests from users, including the Department of Health, Treasury and so on.

Senator Roberts: By big data you mean data that can be automatically collected or harvested from existing datasets?

Ms Dickinson: Yes, such from the example that you gave, such as hospital data.

Senator Roberts: Okay. Have you ever fined someone for refusal?

Dr Gruen: Yes. And we fine a small number of people for not filling in the census.

Senator Roberts: Yes.

Dr Gruen: But not a large number. We have 10 million households fill it in and the number of people we fine is very small.

Senator Roberts: Minister, are you happy that this elderly widow was terrified?

Senator Gallagher: I’m sure the ABS and Dr Gruen would be very happy to follow up an individual matter, if you’re able to support your constituent to raise that—if she felt vulnerable over that. I think that resolving these issues is important and there are ways to do that. I’d certainly encourage you to think about how you could facilitate that. I also totally support the need to seek this information, because it helps in so many ways to understand what’s going on. Currently, for example, I’ve been selected for one of the household surveys—I think it’s for nine months. Do you get selected for that—

Ms Connell: Eight.

Senator Gallagher: Eight months—

Chair: You can—

Senator Gallagher: It was made very clear to me when I inquired about having to do it—the compulsory nature of it—and the consequences for not filling things out every month—

Senator Ruston: They didn’t believe you when you said you were too busy, did they?

Senator Gallagher: I had very helpful advice from the ABS when I rang to try to get out of it! I was told, politely, that those were not grounds for getting out of it. But that’s how we get information about what’s happening across the country.

Senator Roberts: Yes.

Senator Gallagher: And I don’t think that anyone who’s sitting here would say that they took any comfort in thinking that an elderly woman felt terrified by it; that’s not the intent, and I’m sure there are ways to work through that.

Senator Roberts: I applaud your comments about the need to use data in government but I don’t see much of it—and I’m not talking about this government on its own, I’m talking about previous governments as well. One of the sad things is that government doesn’t use data when making policy and legislation, in my view.

Senator Gallagher: But it’s not just for government. So many people rely on the ABS datasets for their work.

Senator Roberts: Dr Gruen, you mentioned something that I took to mean people are becoming more reluctant to share data—

Dr Gruen: More reluctant to participate in surveys.

Senator Roberts: Is that due to the pushback because of—well, what is the cause? Is it due, partly or maybe majorly, to the intrusion into people’s lives during COVID?

Dr Gruen: It’s a phenomenon that predates COVID, and it’s global. It happens in all countries. I’m aware that there has been a gradual decline in response rates to surveys. We have higher response rates than most advanced countries for many of our high-profile surveys, like the Labour Force Survey, which I think must be the one the minister is enrolled in.

Senator Gallagher: Mine is the household one.

Dr Gruen: Oh, can I—

Senator Gallagher: They want to know how many people in my house, what we’re doing and how hard we’re working. I’m skewing the statistics!

Dr Gruen: That’s the Labour Force Survey.

Senator Gallagher: Is it?

Dr Gruen: We have the labour force expert behind us.

Senator Gallagher: Okay!

Senator Roberts: In which way are you skewing the statistics?

Senator Gallagher: Because I work so much! I’m off the scale!

Senator Roberts: Oh, off the scale.

Senator Gallagher: And it’s, ‘Why are you working so hard?’ I fill it all out.

Dr Gruen: On the web?

Senator Gallagher: Yes.

Dr Gruen: Good, I like to hear that.

Senator Roberts: Because a pesky senator is asking questions in Senate estimates! Thank you, Chair.

Chair: I’ve got distracted and entirely lost control of the committee!

Senator Roberts: No, you’re still in control.