Note: Where available, the PDF/Word icon below is provided to view the complete and fully formatted document
Economics References Committee
2016 Census

KALISCH, Mr David, Australian Statistician, Australian Bureau of Statistics

PALMER, Mr Jonathan, Deputy Australian Statistician, Australian Bureau of Statistics

SUTTON, Mr Trevor, Deputy Australian Statistician, Statistical Business Transformation Group, Australian Bureau of Statistics

CHAIR: I remind senators that the Senate has resolved that an officer of a department of the Commonwealth or other state shall not be asked to give opinions on matters of policy and should be given reasonable opportunity to refer the questions asked of the officer to superior officers or to a minister. This resolution prohibits only questions asking for opinions on matters of policy and does not preclude questions asking for explanations of policies or factual questions about when and how policies were adopted. Officers of a department are also reminded that any claim that it would be contrary to the public interest to answer a question must be made by the minister and should be accompanied by a statement setting out the basis for the claim.

You have lodged submission No. 38 with the committee. Would you like to make any amendments or additions to the submission?

Mr Kalisch : I would like to make one small correction but if I could make that at the end of my opening statement.

CHAIR: You have foreshadowed you would like to make an opening statement. We will then open up for questions.

Mr Kalisch : The ABS was requested to provide its submission to this inquiry by 21 September, when we were still receiving many census returns. Since then, I am pleased to report that all the available evidence suggests we will have high quality census data from the 2016 collection—a preliminary participation rate of over 96 per cent of households contributing data to the 2016 census, comprising over 4.9 million online forms and over 3.5 million paper forms, and many personal forms in addition to the household ones.

The initial quality checks we have undertaken to date, covering over half of the total census returns, show low levels of item non-response to the known sensitive questions. The much higher level of online response achieved in the 2016 census compared to past years is expected to lead to higher quality census responses overall, drawing on the 2006 and 2011 experience about the higher quality of online responses. The Census Post-Enumeration Survey is still in the field to assess the coverage of the census count and will undertake further intensive processing and quality testing of census data over the coming six months. As an additional quality mechanism, we are establishing an independent panel to provide their own quality assurance of the 2016 census data, and will publish its report in 2017. Census data for 2016 will allow us to produce new population estimates that, in turn, contribute to federal and state funding allocations and are used for setting of electoral boundaries. New information on regional communities and small population groups will properly inform many future policy and planning decisions.

As I mentioned at our Senate estimates hearing last week, I would like to thank the Australian population for their forbearance and diligence in completing the 2016 census. The ABS has tested the patience and commitment of many households, especially through the difficulties accessing the call centre and the unavailability of the census online form for nearly two days. We made a difficult decision to take the system offline on 9 August to ensure the security of census data, but we should not have got to that point and the IBM system should have been robust to DDoS events. The ABS made a number of poor judgments in our preparation for census 2016 that led to the poor service experienced by many households. I apologise to the community on behalf of the ABS. We managed other aspects of the census very well. Many other households did have a good census experience in 2016 through access to a quick and easy online form or use of paper forms. This is what we should achieve across the entire population.

In the lead up to the 2016 census and right through the census process, we continued to track very high levels of community support for the census. Australians clearly understand the value of the census. The available evidence is that Australians completed the census to the usual extent despite our system difficulties for a few days and the commentary about privacy.

Our submission highlights the long history and importance of collecting name and address in Australian censuses. Over the last century, the ABS has required people to provide names and addresses to produce essential national statistics, and only for statistical purposes. This is international practice. Canada's experiment with an opt-in census in 2011 had less than a 70 per cent response rate and serious data quality issues. Canada's latest 2016 process reverted to a mandatory response to the long form census. Following an Australian tradition to debate census and privacy that goes back to the 1970s, I have been struck by some of the misunderstandings that were promulgated around this issue, including by Bill McLennan who was Australian Statistician more than 16 years' ago.

Our submission provides the facts of how the ABS safely and securely uses census information to produce statistics that are of national importance and has done so for decades. The ABS regularly receives sensitive information from households, businesses and government that enables us to produce the wide array of essential statistics required by the nation. The ABS has a longstanding compact with the community to keep your private information safe and secure. The ABS has extensive protections safeguarding the security of our data holdings, statistical production and dissemination of statistics. We are also constantly vigilant to identify new risks and upgrade our protections.

The overall mission of the ABS is to produce statistics to inform decision making. The functions of the ABS established in 1975—to collect, compile, analyse and disseminate statistics and maximum possible utilisation for statistical purposes of information available to official bodies—are still pertinent to the ABS in the 21st century. The secrecy and disclosure requirements in the Census and Statistics Act 1905 are properly very restrictive for a national statistical agency.

The ABS's statistical capabilities continue to develop, as they have over our 110-year history. The cost of managing big data has declined substantially over recent years. We are making increasing use of the information already collected by governments and the commercial sector. We are looking to reduce the collection burden we directly place on households and businesses. The ABS of today is more complex and advanced from the ABS of just 15 to 20 years ago, as is the broader information environment in which we operate.

The ABS remains one of Australia's iconic organisations, providing trusted statistics on our economy, our society and our environment. Our data integration capabilities developed and used over the past decade have provided new statistics on Indigenous life expectancy, outcomes for migrants, transitions for auto workers since 2006, post-school outcomes for students undertaking school based apprenticeships, and comprehensive statistics for the 2014 review of mental health programs. These are all important public policy issues where the ABS has safely produced reliable statistics.

The decision to retain name and address information for longer than the previous 18 months will mean that the ABS can deliver more and better-quality statistics to inform key policy decisions and contribute more statistical information about the effectiveness of public programs. Statistics New Zealand already provides such statistics safely and effectively, as observed recently by the JCPAA.

I still believe a digital-first approach was appropriate for the 2016 census. Given some of the public reaction to the 2016 census experience and the need for the ABS to restore high levels of public trust in the census process, planning for the 2021 census will necessarily adopt a more rigorous approach and will have all the learnings from the new approach adopted in the 2016 census.

As I mentioned, I would like to take this early opportunity to advise the committee of one correction to the ABS submission, to table 2.1 on page 13. Consistent with the advice I provided to the Senate Economics Legislation Committee supplementary estimates 2016-17 hearing on Wednesday, 19 October, the appropriation funding for the 2016 census over the five years from 2012-13 to 2016-17 should read: $43.8 million, $41.1 million, $57.7 million, $111.5 million and $214.5 million, respectively. That is consistent with the evidence I provided last week.

CHAIR: Thank you very much. Mr Kalisch, in the opening remarks from IBM earlier today they talked about negotiations that have taken place between them and I understand principally the secretary of Treasury in relation to cost defraying or cost recovery arising from the census. Is the ABS involved in those negotiations?

Mr Kalisch : I have been having extensive discussions with the Treasury secretary and we have provided information to Treasury to support their discussions and negotiations with IBM, but the negotiations are largely taking place between the Treasury secretary, John Fraser, and the IBM company.

CHAIR: Is the minister involved in this?

Mr Kalisch : I understand the minister is also being kept apprised of developments.

CHAIR: But he is not participating in the negotiations?

Mr Kalisch : No, this is a negotiation that I understand the Treasury secretary is undertaking on the advice and suggestion of the Treasurer.

CHAIR: So when did the Treasurer make that suggestion?

Mr Kalisch : You would have to ask the Treasury secretary or the Treasurer. I am not aware of that. It was some time ago.

CHAIR: Can you clarify for us, starting from your appointment in December 2014, which ministers had responsibility for the census.

Mr Kalisch : Ultimately, I think it is appropriate to say that the ABS has responsibility for the census. The ABS and the Australian Statistician control the ABS, and that is very much in the legislation. So we have responsibility for the census. There are some aspects where government and ministers do play a role, for example, in terms of establishing the topics in the census, proclaiming the date for the census and establishing the regulations. There are other engagements that we have directly with government processes, as I mentioned at estimates last week, around the government communication processes.

CHAIR: Tell us: which were the relevant ministers since your appointment.

Mr Kalisch : Since my appointment, it was Mr Ciobo first of all, then Ms Kelly O'Dwyer, Mr Alex Hawke and, more recently, Mr Michael McCormack.

CHAIR: During the period that Ms O'Dwyer was the responsible minister, how many meetings or briefings did you have with her about the progress of the census?

Mr Kalisch : Following the question that you provided to us at the last estimates, we have undertaken some preliminary investigations. Those investigations are still continuing, so I am not in a position to be able to provide you with exact numbers of meetings. A number of the meetings are still being verified with ministers' offices. In terms of the total meetings—I think you asked since 2014—we have been able to find at least 40 meetings between the Australian Statistician or the acting Australian Statistician and the relevant minister. Many of those meetings would have at some point or another touched on the census process. Others would have been exclusively about the census process if they largely dealt with the communications issues, the proclamation of the date or the census topics issue.

CHAIR: When you have finished your investigation, could you provide us with a breakdown as to how many meetings were with each minister over that period of time?

Mr Kalisch : Yes, Senator. That is what we were intending to do when we took the question on notice last time.

CHAIR: This might be a question for Mr Palmer as well: can you tell us what concerns ABS raised during those meetings about the progress of the census?

Mr Kalisch : I cannot recall too many meetings that I was party to where concerns were raised about the progress of the census. There were a number of meetings, particularly around the quick resolution around the topics in the census, which was undertaken very quickly within government. Certainly, that eased some issues that we had around the timing of decisions and the timing of the census at that stage. Similarly, early resolution of the date of the census and the proclamation of that was again very much to our satisfaction and to the requirements of running an efficient census.

CHAIR: Some of these meetings would have occurred prior to your appointment, Mr Kalisch.

Mr Kalisch : There may well have been some with Minister Ciobo.

CHAIR: Mr Palmer, are you able to tell me about any concerns you would have raised with the minister about the progress of the census?

Mr Palmer : I had some meetings with Minister Ciobo during 2014. I do not recall raising any concerns in relation to the census. There were still a couple of years to go. We certainly did discuss the census.

CHAIR: I understand that the project head, Mr Duncan Young, sent a memo to ABS staff warning that the census status was red, meaning that the program would not be able to deliver on the current scope, timetable and/or budget. Can you tell me when that occurred and which minister was in the chair at the time?

Mr Sutton : I think it was Minister Ciobo at the time.

Senator KETTER: Was Mr Young's memo, or the contents of it, relayed to the responsible minister?

Mr Kalisch : Sorry, Senator, we have just had some clarification on this.

Mr Palmer : It was in February 2015. It would have then been Minister O'Dwyer.

Mr Kalisch : It was certainly around that time that Minister O'Dwyer and I were engaged in a number of conversations about the future of the census. As you would recall, there was some conversation about potentially doing a census every 10 years and the nature of our broader population and social statistics program. So they were the issues that were pertinent at that stage. We were essentially planning to do the census while we were also looking at contingency plans to not do the census.

CHAIR: Which is confusing.

Mr Kalisch : That is the reality of how we do a lot of things in government. You often do not have a clear pathway through. You have a number of choices, a number of options that you are also simultaneously prosecuting.

CHAIR: Correct me if I am wrong, Mr Kalisch, but I do not think you made reference in your submission to this inquiry about that conjecture on whether it was going to be a five-year or a ten-year census.

Mr Kalisch : I cannot recall that we profiled it heavily in our submission.

CHAIR: But you have just identified it as, perhaps, being one of the factors that might have caused some concern.

Mr Kalisch : I know that there were some questions internally about the nature of some of the responses. But, as I mentioned, then once the budget situation was clarified—in terms of the funding for the census was clarified—and also quick decisions were taken on census topics and a census date we then got back on track. So there were some fairly easy wins that government and ourselves took to get the project back on track very quickly.

CHAIR: Just going back to Mr Young's memo, were the contents of that email relayed to Ms O'Dwyer?

Mr Kalisch : I think there certainly was some discussion that I was having with the minister at the time about the nature of the timing and when we needed to have a decision one way or the other about the census. So there was a question that we were keeping the minister apprised of where, beyond a certain point, we were at the point where we needed to progress with the census. There was no second option, in effect. That was consistent, I suppose, with the nature of the processes within government, the negotiations and the discussions that were taking place with parliamentarians across different political parties at the time and, also, the nature of the resolution that came to it.

CHAIR: What was Minister O'Dwyer's response to that memo?

Mr Kalisch : I am not sure that she actually saw the memo. That was an internal one. But certainly she was very careful about understanding the nature of our processes and also making sure that we had sufficient time and resources to undertake the census if that was the decision that was then taken.

CHAIR: Did Minister O'Dwyer subsequently follow up on the concerns that had been raised in the memo?

Mr Kalisch : She certainly responded to the conversations that we had about the nature of the process that was taking place in the discussions with different political parties and she understood that a decision needed to be made at a particular time and she responded to that by having further discussions within the government—with the Treasurer.

CHAIR: To your knowledge, was the status of the census and Mr Young's concerns ever raised with a meeting of cabinet after that time?

Mr Kalisch : It is not within my remit to talk about that, but I am not aware of anything.

CHAIR: So you are not aware of it, or it did not happen?

Mr Kalisch : I am not aware of anything. Nor would it be appropriate for me to be necessarily aware of that.

CHAIR: After being appointed the minister in July 2016, on what date was Minister McCormack first briefed about the progress of preparations for the census?

Mr Kalisch : My understanding is that the first briefing took place with Minister McCormack on 25 July.

CHAIR: What did you tell him at that point?

Mr Kalisch : It was really just about the nature of the census process; it was largely about that. There was also further discussion about small business statistics, from what I recall, given his other portfolio responsibilities were in the small business environment and we were providing him with a sense of some of the statistics that we collected on small business, some of the collection burden that we placed on small business, necessarily, to provide government and others with further advice about the small business sector. But also we provided some sense of the information that we use from the Australian Taxation Office and from other administrative collections to reduce the burden on small business.

If I can just go back to the census dimensions, it was very much about the nature of the census process. I suppose it is important to keep in mind that the census in 2016 was different to past censuses. We were moving to a digital-first approach. Not everyone was aware of this. At that time the broader public advertising had not begun to the broader population, so we were providing the minister with a clear picture of what the different steps and processes within the census were.

CHAIR: So, between 25 July and census night, can you tell us how frequently ABS and the minister communicated about the census?

Mr Kalisch : This is something that we are still following up with his office to verify the number of meetings, but we did have a number of meetings, including on the issue of privacy. I can recall at least one, and there may have been two, media events where the minister was involved also about the census in the lead-up to the census night.

CHAIR: Can you clarify how many meetings or communications—

Mr Kalisch : This is something we are still clarifying with the minister's office at this stage.

Senator HUME: I wanted to ask you about the limited tender for which IBM were successful. The limited tender was recommended by CapDA?

Mr Kalisch : I will ask Mr Sutton, who was oversighting that process at that stage, to respond.

Mr Sutton : Sorry, I did not quite catch the question.

Senator HUME: The option for a limited tender was recommended by CapDA—is that correct?

Mr Sutton : It was, indeed.

Senator HUME: You accepted that as the right option?

Mr Sutton : Yes, we did. Their advice was that the lowest risk option was to go to a limited tender with IBM, and the factors that were considered as part of that risk were clear, I understand, from the CapDA evidence that was given this morning, which was around time frame, the need to develop a new application—anybody coming into it would have had to have done that from scratch—lack of familiarity with the ABS systems and processes, and also a lack of experience in running an online census. So there were a whole range of risk factors that were considered, along with time frame, as part of CapDA's assessment. We accepted that assessment as correct and subsequently went for a direct select tender with IBM.

Senator HUME: Obviously, part of what you mention there is the benefit of incumbency, if you like; they were already familiar with your processes and what had gone on before. Incumbency obviously played a big part in that decision?

Mr Sutton : I think it played a part in the decision.

Senator HUME: Do you think that IBM were aware of your budgetary constraints?

Mr Sutton : In terms of the census there were no issues with budget. We had a satisfactory budget with respect to the census. It is a separate allocation for the census compared with the rest of the organisation.

Senator HUME: Do you think that the proposal that IBM put to you was tailored around the budget, or was it tailored for the best solution? Was the budget the overriding constraint in the solution that was offered? I think that is what I am trying to say.

Mr Sutton : No, I do not believe the budget was a factor in the way that you are describing. We were looking for IBM to meet our particular requirements and to be value for money. They were the criteria which we used in assessing that IBM was the appropriate provider of an online census. We assessed value for money by looking at benchmarking the previous census, where they had provided that online capability. They came in at a lower value after five years. They had to meet larger scope and also larger scale requirements. So it was less expensive than last time, with a larger scale and a larger scope. The scale would be, for example, going from a census where we had a third of the population use the eCensus in 2011 to, obviously, our target of going up to 65 per cent in the 2016 census. In terms of scope, they also had to connect to a whole range of other technical requirements around mobile devices and so forth and so on. So it was a much bigger scale and a much bigger scope, and they came in at a lower price than they had provided in the previous census. So our assessment was that was value for money, and it met all of our requirements.

Senator HUME: Terrific. Without meaning to go over old ground, I would be very interested in your take on census day, or census night—the four DDOS experiences that occurred.

Mr Sutton : I think I might need to pass that on to my colleague Jonathan Palmer to answer those questions.

Senator HUME: I am particularly interested in the interactions between the ABS, IBM and also the Australian Signals Directorate in those four instances.

Mr Palmer : Certainly. The chronology of the night is pretty well covered in our submission, so I will keep it pretty brief. The first attack was at 10.10 am. After that attack, we had a discussion with IBM about the fact that the traffic may have been coming from overseas, so we talked to them about how we might respond to further attacks, particularly if they came from overseas, and we agreed on a protocol for invoking Island Australia, which is the name given to their geoblocking solution.

The second attack was at 11.45 am. The system became unavailable for a couple of minutes, and IBM switched on the Island Australia defence mechanism. At that point, after that attack, ABS reported this to ASD. I listened in on the IBM submission this morning, and perhaps they failed to remember that. From ASD, we sought some advice on appropriate mechanisms and sought advice as to whether there was anything we should do differently, and we shared that with IBM.

The third attack was at 4.52. Island Australia was in place. It had no discernible impact. There was no disruption to service. Then the fourth attack was at 7.28. I might just point out that IBM referred to that as 7.27; in their submission they say 'about' 7.27. These time frames are within minutes rather than second-perfect, so we essentially have the same chronology here. That attack was again inferred as being overseas-based, and that was the attack that caused us to bring the system down and keep it down. And at that point we re-engaged with Australian Signals Directorate. I was particularly keen to enlist their support in verifying that there had been no breach of the security of the data, and also of course to work with them on protections, should we bring the system back up. I think that covers the broad chronology.

Senator HUME: Who was it specifically at IBM, and who was it specifically at ABS, that was dealing with each one of those incidents?

Mr Palmer : The earlier attacks—I am not sure: we had our management team in place in ABS house—Mr Libreri and Mr Young were there—IBM was in-house for the entire day and, as it turned out, well into the next morning as well, including Kerry Purcell, the managing director, who was there for the later part. Permenthri Pillay, who testified this morning, was there throughout. I am not quite sure what time she arrived in the day. So we had teams in place that were actively talking to each other. We had an operations centre where we were monitoring the census as it progressed—looking at the performance, the number of online forms being submitted. The key interlocutors, if you like, would have been Permenthri and her IBM team, and Mr Young and Mr Libreri and myself and Mr Kalisch.

Senator HUME: How long exactly did it take to establish that there was not a data breach in that last incident?

Mr Palmer : I have to say I was confident all along that there was no breach, because of the nature of the security architecture. We had end-to-end encryption; we had had the architecture well reviewed. So the problem was that we could not explain this—as it turned out—false positive alert, and IBM could not explain to us what it was. We felt it was extremely important that we be able to assure the Australian public that our faith in our security was well placed. So it was not until sometime the next morning, I think 2 am or 3 am—I would have to check—

Mr Kalisch : I think was about 3.25. I received an email from our security adviser saying that the meeting had concluded between IBM, ASD and ABS, and that they had categorically concluded that there was no data exfiltration.

Mr Palmer : Senator, I should have added that we also had an IT security adviser in our team who contacted ASD after each of the attack incidents, so we were in fairly continuous contact there.

Senator HUME: Okay, terrific. Regarding the decision not to bring the census back online for some time: you got the advice at two or so in the morning and it did not go back online for 40 hours or so, wasn't it?

Mr Kalisch : Yes. There were three particular aspects that we wanted to be satisfied about. One was about the security of the data which, as we say, was something that we were assured of after three am the next morning. The second aspect was really related to the nature of the router and that there was a working contingency, and that was something that was still the subject of some discussion on the following morning and through the following day. The third aspect was that—certainly against the backdrop of the system being vulnerable to a DDoS event on the night before—I wanted to be as sure as we could be that it would not be vulnerable to a second DDoS event . And so that was where there was further engagement with ASD—and Telstra certainly assisted in that process as well—making sure that there were a number of additional protections put in place. I then received further advice from ASD—I can give you the exact time, Senator. At 1.16 pm on 11 August I received advice. I then briefed the minister on the nature of that ASD advice, and advised him that the online census system would be reactivated . The advice I also had was that it would take about an hour for that system to be reopened . In the end, it was just before 2.30, so it was just under that one hour. We then issued a press release and held a media conference to confirm to the public that the site was back and running.

Senator HUME: Were you surprised by the extent of the disruption caused by the DDoS events considering on a relative basis they did not seem to be particularly large?

Mr Kalisch : We were certainly surprised that the system was vulnerable.

Senator HUME: And you were assured, I assume, beforehand; otherwise, you were assured that it was invulnerable?

Mr Kalisch : We were assured that that system was robust and was ready to go to a range of different attacks and mechanisms, not just DDoS.

Senator HUME: Were you surprised that the problem had been caused by the Nextgen router, that there had been a breach of protocols there that had not been anticipated? What was your response to that piece of news?

Mr Kalisch : We were also receiving advice and were, at that stage, in close contact with the Prime Minister's senior adviser on cybersecurity and learning more about the nature of the dimensions. I think it is clear to say that on the evening of 9 August and even into the morning of 10 August, there were a number of things that were still being discovered so it was not as if there was a picture perfect sense of what had happened at 7:30 on 9 August. Things were still being learnt and some of the advice that we received turned out to be incorrect at a later stage. We were really disappointed about that.

Senator HUME: After the system went back up online, there were a number of DDoS attacks after that, were there not?

Mr J Palmer : I think at Senate estimates we said there was an attack after it came back up. My understanding was there was an attack while it was still down. I am not aware of any attacks after it came up.

Senator HUME: After it came back up, for the days after it was back up and used, there were no DDoS attacks at all?

Mr J Palmer : No, not that I am aware.

Mr Kalisch : Interestingly as you might appreciate, we were still carefully monitoring the system and so there were the operations that Mr Palmer talked about around the time of census day. Prior to census day it was in operation again so we had people watching the screens, watching the flow of responses on a minute-by-minute basis—if I can refer to it that way—and we were providing advice to the minister and the Treasurer about the nature of the system—that it was still operating—and the nature of the flow of responses closely on about a two-to-three hourly basis.

Senator HUME: When the system went down and you tried to reboot it and one of the routers did not work, was that something that took you by surprise?

Mr Kalisch : I suppose it was just really the nature of the three aspects: that we needed to be sure that the system would operate in that it would be able to cope with any DDoS events; that the hardware and the contingency plans around the system were robust; and that the data was secure. So it was really just a part of that triumvirate of aspects that we were wanting to be sure about.

Senator HUME: IBM said something interesting—that if they had their time again, they would have tested that router by turning it off and turning it on again. That sounds like the sort of level of technical competence I have with my computer. Did that sound like a pretty basic oversight to you?

Mr Kalisch : One thing that did surprise us was that there was a sense that the router had failed. They then progressed to get a second router as a 'just in case' to be operating the next day and then the story changed to it being a 'configuration problem'.

Senator HUME: This issue of not having a backup router bewilders me. I would be interested in your take on that. Was that something that you would have assumed existed? Did you ask about it? Did you feel you needed to ask about it or was it just assumed that it would be there?

Mr Kalisch : They had the two ports in, but I will ask Mr Palmer to provide a further explanation.

Mr Palmer : We were actively interested in the architecture for the census. Although IBM was contracted to provide industry-best-practice secure architecture, we were keen to understand how it worked. We were aware of the design which had two ISP links coming in and two routers. As the engineer from IBM explained, that was an N+1 redundancy, which means that the system was designed to have one more router than was technically required to cope with legitimate load to the application. I think the engineer also said it was pretty standard practice to have that level of redundancy. The architecture was, I guess, unsurprising and appropriate. We were just more surprised that when the router 'failed', there was not a clear explanation of what the problem was. It took some time for them to tell us that it was actually a configuration problem rather than some failure of the hardware. At that point we knew we could operate with one router. We knew the system was designed to have sufficient capacity. But, as Mr Kalisch said, it was just another issue on top of a couple of others that we had to work through. I would say it is a bit like functioning on one kidney but you do not really want to when you have two. We wanted to understand what was wrong with it and make sure it was fixed.

Senator HUME: For the remainder of the time that the census was open, you were essentially functioning on one kidney successfully?

Mr Palmer : No. We had both operating quite nicely until that 7.28 attack.

Senator HUME: Yes, but after that?

Mr Palmer : After that, we were not really operating. We had some people who had active forms that were still able to operate, but we were starting to succumb to the load of the denial of service attack.

Senator HUME: Sorry, I meant after the 40 hours when you—

Mr Palmer : I am sorry, Senator. The second router was brought back online and properly configured so, when we came back up, that was all squared away. There was some talk of having a third router, just out of an abundance of caution. There may have been one brought on site, but I would need to check that.

Senator HUME: Thank you.

Senator XENOPHON: Mr Kalisch, you said you were surprised the system was vulnerable to DDoS. What was the source of that surprise? Was that as a result of representations made to you by IBM and anyone else or by anyone else that the system could withstand that level of DDoS?

Mr Kalisch : It was broadly the comments and exhortations that IBM had made to the ABS about the importance of this contract, the importance of this work to them—

Senator XENOPHON: Hold on. Do you usually get a representation or was it an exhortation dressed up as a representation?

Mr Kalisch : It was a meeting that I had with the senior IBM executives before census day. Perhaps 'exhortation' is not quite the right word. They were certainly very clear about the importance of the contract for them, the importance of them delivering a quality census for the ABS and Australia, and they saw and appreciated very much the importance of them delivering properly on their contract. I understand that similar comments had been made, probably over a number of years, to Jonathan Palmer, who was Acting Australian Statistician, and also to Brian Pink, who was previously Australian Statistician and would have been involved in some of the earlier conversations, certainly with IBM. So it was left in no sense unspoken that they saw and appreciated the importance of this contract.

Senator XENOPHON: I did not have an opportunity to hear IBM's evidence today. Did you hear IBM's evidence today?

Mr Kalisch : I think Mr Palmer was able to hear most of it.

Senator XENOPHON: Mr Palmer, is your understanding substantially in accord with the version given by IBM, or does the ABS have points of difference with IBM's position and reasons for what occurred on 9 August?

Mr Palmer : I think we are substantially in agreement, Senator. There are a few minor potential points of difference. I raised one earlier when I said that IBM had said in their submission or in their testimony that they had not talked to ASD until after the fourth attack, or they thought we did not, but in fact we talked to ASD after the second—or the first, I think. So there are a couple of minor differences. Another was that IBM stated that they activated the overload control after the fourth attack. The overload control was the screen that came up and told people that the system was not available for new sessions. That is something we believe was our decision, so if they activated it we were not aware of that.

Senator XENOPHON: Did they activate it or not?

Mr Palmer : I am not sure. I have sought more information from them. I think it may just be that we all agreed at the time to do it, and there is a little bit of confusion as to who actually made the decision. But in our view the decision was clearly ours to make, and we made it. It was a good thing to do, so it is just a bit of a quibble as to who thought of it first.

Mr Kalisch : There is just one other thing I will add: to some extent it is a little bit irrelevant what IBM believes took place. The judgement that we make about what took place on the night and the evidence that we have seen is largely drawing on the conversations that we have since had with Alastair MacGibbon and his process. So we see that as the source of truth.

Senator XENOPHON: In respect of sources of truth, though, do you concede that, if IBM had erroneous beliefs, as you put it, those erroneous beliefs could have extended into perhaps erroneous advice in terms of the way the software or the system was designed prior to census night?

Mr Kalisch : There was certainly some uncertainty and some corrections to the advice that we received from IBM through that first 24 hours after the DDoS event, particularly around the router and some of the other aspects. In terms of the configuration and the architecture, I think Mr Palmer talked about the broad satisfaction that we had with that process.

Mr Palmer : Yes, I would say there was no misunderstanding. There was considerable clarity as to our requirements and their contractual obligations to meet those. We were building on our experience with two prior censuses, so they had an excellent understanding of what we needed. So I do not think there was any lack of understanding leading up to the census. It is more just being clear precisely what happened on the night. There might be a couple of differences of opinion there.

Senator XENOPHON: Sure. Can I just go to issues of legality. There seems to be a strong view that there is no ability to compel a person to supply their name to the census. Mr McLennan and others have expressed that view. On 3 August 2016, Mr Kalisch, you made a claim on the 7.30 program that you believed on the basis of advice from the AGS, the Australian Government Solicitor, that it is lawful to conduct the census in the way you were doing. I am not misrepresenting your position? Is that correct?

Mr Kalisch : That is correct.

Senator XENOPHON: And you consider that advice to be privileged?

Mr Kalisch : That is the advice that the government has provided to the Senate.

Senator XENOPHON: Which I think is the subject of a challenge. But are you aware that you can indirectly waive privilege by disclosing the crux of the advice?

Mr Kalisch : I am not in a position to provide that advice, but I would make one comment. Actually, the main complaint or concern I have with what Mr McLennan says is not in a legal dimension; it is in a statistical one. So the issue is that, I think just as a matter of history, the ABS has collected names and addresses on a compulsory basis since 1911. So the collection of names has been part of, first of all, the Census and Statistics Act, and then from I think about 1986 it was included in the census regulations, when the census regulations were introduced. But the issue is more something that has been covered very carefully and properly in the Australian Law Reform Commission's report entitled Privacy and the census. I am not sure whether you are familiar with it.

Senator XENOPHON: Yes, I have that in front of me.

Mr Kalisch : We did refer to it quite carefully in our submission. If I can take you to page 42 of our submission:

The Commission also examined the suggestion that the Census might be conducted on an anonymous basis. It concluded that "an anonymous Census, like a voluntary one, would result in an unacceptable level of non-response and an equally unacceptable bias in responses. It would inhibit or prevent the conduct of the post-enumeration survey whose function is to assess the accuracy and completeness of census responses."

So the purpose of us collecting name and address is actually for statistical reasons. If I can take you back to Mr McLennan's latest advice, and this was something he mentioned in the evidence he gave this morning. The advice from the Office of Parliamentary Counsel in 1981 noted: 'This will be broad enough to authorise the acquisition of name and address of particular respondents, as long as it is done for statistical purposes.' The issue I have with Mr McLennan is not so much on the legal dimension but the statistical one. I think he has perhaps forgotten or not recalled the nature in which names have been used, particularly to provide better responses to the census, and then the way in which name is used for the post-enumeration survey to measure the extent of the overcount and the undercount. If you want to go to the issue of address, we use address for a number of purposes, such as making sure that we do not count households multiple times and that we have populations and demographics statistics for states and territories and regions. So we can properly then provide information for electoral purposes, for local government uses and also for transport statistics.

Senator XENOPHON: I understand that, and I do not want to dwell on this too much longer, because I do want to ask you some questions on the issue of procurement and the Commonwealth Procurement Rules, which I understand the secretariat may have a copy of and which hopefully someone can give you in a moment. Going to the 'Brazil Direction' of cabinet, going back many years now, which says that agencies should not claim exemption for documents which have no particular sensitivity and that legal professional privilege should be waived unless some real harm would result from the release of documents—that is essentially about FOI, but the grounding principle is relevant—what specific harm would come from releasing the legal advice that you have relied on quite publicly in relation to the legality of names being required on a census form?

Mr Kalisch : I am not a lawyer, so I am not going to—

Senator XENOPHON: You are very fortunate in that regard.

Mr Kalisch : provide a legal response to that, but I am just going on the advice that I have received from AGS and the Attorney-General's Department that I understand has been conveyed to the Senate by Minister Cormann.

Senator XENOPHON: Can I go to the issue of the contract. You have given a time line that shows the ABS started thinking about the census back in June 2011. That is correct, is it not?

Mr Kalisch : I am not sure about the date, but it certainly was around that 2011 period.

Senator XENOPHON: That is fine—thereabouts in 2011. I note that the Capability Driven Acquisition's report found that there is an urgent need to onboard a prime partner and other external resources as quickly as possible. Indeed, the company went on to state:

… the original prime partner procurement plans … have been delayed as a result of a time taken in relation to key decisions relating to the potential use of Blaise and the scope of any prime partner procurement. Interviewees generally recognised that it would have been desirable to have had a prime partner ‘on board’ sooner than is now achievable.

Can you elaborate on what key decisions the ABS made—or, indeed, failed to make? Rather, what key decisions do you concede the ABS failed to make in the necessary time frame? Conversely, in respect of the question I have asked, what decisions were made and in what time frame?

Mr Kalisch : I will just provide a little bit of context and then ask Mr Sutton to provide a little bit more detail for you, Senator. Around that 2011-12 period, the ABS was, I think, it is probably fair to say, struggling with its fragile and ageing statistical infrastructure. So it was thinking about: are there better solutions; are there better ways of improving our broader statistical infrastructure across the organisation as a whole? The census has previously been seen as an opportunity to enhance the statistical infrastructure of the ABS. I think that is probably one of the contexts. I will ask Mr Sutton, who was involved in some of those deliberations at that time, to provide you with further information.

Mr Sutton : Thank you, Mr Kalisch. Senator, we considered quite a number of options in terms of the 2016 census, as you quite rightly point out, probably beginning back in late 2011. If I were to give you a time line on the various options that we considered during that period, I hope that would be helpful in giving you a context.

We initially went to the market in February 2012 with an RFEOI for an enterprise-wide electronic form solution, along with a whole range of other capabilities. What we were looking to do, which Mr Kalisch touched on, was to see whether we could actually procure something that could provide an e-form solution for both households and businesses and, potentially, the census in 2016. We had a whole range of submissions due to that RFEOI. We assessed those in June 2012 and found that none of them offered a better outcome than our existing product, which is referred to as Blaise, which is a product we license from Statistics Netherlands and we use in our household and business surveys.

We then moved on to look at another option, which was to establish whether Blaise was scalable for a census. Unfortunately, our final testing showed in November 2013, I think it was, that it would not be scalable and that we would then need to look at some other options. As Mr Kalisch said, our attempts here were really to come up with something that could actually meet a range of requirements, including those of the census.

We then looked at exploring our existing census e-form application. You might also be aware that we had a successful e-form capability for the 2006 and the 2011 census. We own the IP on that application. It was something that we did develop in partnership with IBM, but we actually had the IP on that application, and we had also successfully tested that application in parallel in our testing for the 2016 census in 2013—and then, again, in fact, in 2014—with 100,000 households. So we actually had an application there, which had been successful in the past, that we had running in parallel with these other processes.

As you are aware, in February 2014 we then went on to commission an independent and well-respected adviser, CapDA, to provide an assessment of the options going forward. CapDA's report, as you are aware, noted that we did not have sufficient in-house capability to run that application ourselves and that we would need to look at procuring an external partner to host that application.

Senator XENOPHON: Did you consider the idea that an Australian company could be the partner and that a local census application could have been built up and possibly exported to other countries with a need for an e-census capability?

Mr Sutton : That was not something that was of a primary focus. Our focus was to ensure that we had an effective e-census for 2016. As you would be aware, our independent adviser, CapDA, then undertook a market scan to see whether there were any companies in the market.

Senator XENOPHON: Sorry; it is a bit hard; I am getting an echo. I am not sure whether the secretariat is able—through you, Chair—to provide the witnesses with a copy of the Commonwealth Procurement Rules.

CHAIR: Yes. Those are available and are being provided to Mr Kalisch at the moment.

Senator XENOPHON: Thank you, Chair. Can I just go to this issue—and I am concerned about time constraints. The Commonwealth Procurement Rules are quite clear on the criteria under which limited tenders can be used. This was a limited tender; is that correct?

Mr Sutton : That is correct.

Senator XENOPHON: Can I take you to 'Method 3—Limited tender' at 9.10 on page 25?

Mr Sutton : Yes. We have it.

Senator XENOPHON: It says there what a limited tender involves. At 9.11 it talks about procurements at or above the relevant procurement threshold, which is clearly the case here. At 10.3 it says:

A relevant entity must only conduct a procurement at or above the relevant procurement threshold through limited tender in the following circumstances …

It then gives 'in response to an approach to market'. There never was an approach to market here, was there?

Mr Sutton : There was an approach to market in terms of a market scan. We asked our independent adviser to approach a range of potential vendors and test their appetite for, if we were to go to an open market, bidding in that process and also to make some assessment as to their appetite, if you like, and also their ability and the risks associated with going down that path.

Senator XENOPHON: I hope you are not seriously suggesting that what you have described would comply with the Commonwealth Procurement Rules under 10.3, subclause a. Is that what you are saying?

Mr Kalisch : It is a bit hard for us to respond immediately, seeing as we have only just received this document.

Senator XENOPHON: But wouldn't you be guided by the Commonwealth Procurement Rules? It is not a secret document. It is a document which, for a major contract for a procurement such as this, ought to be the guidelines the bureau would be required to comply with.

Mr Sutton : We believe that our approach was in fact consistent with the policy and indeed with the guidance material provided by the Department of Finance on their website. We believe that we did satisfy the conditions in terms of going to market for a select tender.

Senator XENOPHON: May I suggest to you that the way the rules operate is that it must actually be done via AusTender, and that was not done in this case. AusTender were not involved in this, were they?

Mr Sutton : In 'Conditions for limited tender', part e, it talks about 'additional deliveries of goods and services by the original supplier or authorised representative that are intended either as replacement parts, extensions, or continuing services for existing equipment, software, services, or installations,' and so on. We believe we quite clearly meet that.

Senator XENOPHON: That is due to an absence of competition for technical reasons. I am suggesting to you that that was never tested because it did not actually go to market in the first place. You may want to take these issues on notice.

Mr Kalisch : I think that might be helpful. We will provide a full response. The other thing that we have just noted—and admittedly we have only just had these papers put in front of us in the last few minutes—is that all of the conditions a, b, c, d, e, f, g and on to i are all 'or' conditions, so it is not as if all of them need to be satisfied; it is actually just one of those.

Senator XENOPHON: You are not suggesting that you come under d.i., where the requirement is for a work of art? You are not suggesting the census form is a work of art?

Mr Kalisch : No. We are suggesting that part e is the particularly relevant one in this case.

Senator XENOPHON: But presumably you would have received advice or you would have considered seriously the issue of Commonwealth procurement rules in the context of this.

Mr Kalisch : Yes, and I think it probably would help if we did take this on notice and provide you with a full response.

Senator XENOPHON: My understanding is that you indicated urgency, because in order to have a limited tender there must be some urgency in the process. You are not suggesting that there was urgency here, are you? This has been going on for quite a number of years, since the previous census.

Mr Sutton : There was an element of urgency and that was certainly reflected in CapDA's assessment. We obviously had time frames. Whilst it might seem like a long time between censuses and it was about two years out from the next census in this particular time frame, time frames are always a serious consideration, particularly given that we needed to test any application ideally in the 2015 year. I have to say that, if IBM had not proven to be value for money or meet our requirements, we would have seriously gone to an open tender at that point, despite the fact that it would have obviously increased some of the risks that we talked about earlier.

Senator XENOPHON: But didn't the urgency here rest with the ABS? The ABS put itself in a position where it needed to make an urgent sole-source decision on the contractor, which many would consider is never good from a value for money perspective. It became the only option because the ABS did not make a decision earlier.

Mr Sutton : No, it was not the only option; it was the least-risk option. As I have explained, if IBM had not proven to be value for money or meet the additional requirements that we needed for the 2016 census, we would have gone to an open tender.

Senator XENOPHON: If you could get back to me in terms of the procurement process. Wasn't the risk though due to the schedule and the schedule is something that could have been determined by the ABS?

Mr Sutton : Time frames are always a constraint. I certainly would not be suggesting that the time frame was not part of the consideration around risk—it certainly was. Having said that, I believe I have provided you with some of the context leading up to that point. If we had tried to find a solution earlier through, as I mentioned, that market test that we did in 2012 and then the subsequent test of our Blaise product to see whether that was scalable. All of those were legitimate, thoughtful options that we had pursued because they in themselves could have led us to a much more cost-effective solution had they worked out. The reality is they did not. If you are asking me whether we should have seriously considered those because they did put some time pressure on us later in the process, I would still say that we should have tested those options.

Senator XENOPHON: I ask you to table in due course your procurement decision paperwork in respect of this—the documents relating to procurement and the basis for the decision—because they go to the sole-sourcing aspect of this contract.

Mr Sutton : Absolutely. We will provide that on notice.

Senator XENOPHON: Thank you. I know royalties is an issue that my colleague Senator Griff may wish to cover, so I might leave that for now.

CHAIR: I would like to follow that line of questioning on the procurement rules. You have given us some figures that suggest that the total budget for the online portion of the census was in the hundreds of millions of dollars. Adding up all the figures you corrected in your submission—

Mr Kalisch : That was for the total cost of the census. Most of the cost of the census is actually in the field staff. I think there was evidence this morning that the contract that we have with IBM to provide the eCensus was less than $10 million. The cost of the 38,000 field staff that we have to do the follow-up activity and the drop-off of forms in some locations and the further visits to houses—that is where the large proportion of the costs come. We have other costs with the processing of the forms and some of the postal costs that we have with Australia Post for the post that comes back, because that is something we pay for, and then the scanning and processing of those paper forms as well as the processing of the e-forms.

CHAIR: In terms of the eCensus aspect of the project, you mentioned that it just comes in under the $10 million—

Mr Kalisch : That was the cost that we paid for IBM. It was $10.664 million, but that included some of the data capture aspects.

CHAIR: And there were other subcontractors as well. So we are looking at a project that is over $10 million.

Mr Kalisch : Out of a total cost of around $470 million.

CHAIR: What is your understanding of the requirements for a two-stage cabinet approval process?

Mr Sutton : The two-pass process that you mention is one that government applies to projects that meet certain thresholds in terms of IT expenditure as well as risk.

CHAIR: Are you saying that this project did not qualify within the rules for a two-pass cabinet process?

Mr Sutton : That is certainly the conclusion that the Department of Finance reached in that process.

Mr Kalisch : I recall there was some discussion of this at the Finance estimates committee last week.

CHAIR: It is my understanding that these rules are invoked if the total project cost is $30 million or more and the ICT component is at least $10 million. Is that your understanding?

Mr Sutton : That is correct. That is my understanding, although there are, depending on the level of risk that is associated with the project, other factors that can come into play in decisions around whether something will be subject to a two-pass process.

CHAIR: But you would agree that this was a high-risk project?

Mr Sutton : Yes. You could not only describe the census as high risk, in a sense. It involves a lot of complexity. One of the things that would have been considered is the fact that, given that it is a high-risk project, given its complexity and the number of moving parts, it is actually something that is repeated. Mostly these processes apply to new policy proposals that are, by their very name, new and have never been repeated. One of the characteristics, as you would be aware, of the census is that we conduct it every five years. It is not new in that sense, so it is something that is known. We have successfully run censuses for a long time.

CHAIR: But each census is a separate project.

Mr Sutton : Clearly there are different aspects to every census.

Mr Kalisch : And some similarities.

Mr Sutton : And a lot of similarities.

Mr Kalisch : I suppose my understanding is that a decision was taken that that approach would not apply here. That was not something that we took independently. My understanding is that that was something that the Department of Finance agreed to. In other projects, other large IT and infrastructure builds, we have the full suite of different processes subject to our dimensions, in terms of the Gateway review process that we are currently subject to with our statistical business transformation project—that about every nine to 12 months we have independent Gateway tests and evaluation of how the project is going. But in this case, in the census, that was not subject to the same rigour.

CHAIR: So this was a Department of Finance decision?

Mr Kalisch : We cannot independently make that decision ourselves.

CHAIR: When was that decision made?

Mr Sutton : I believe it was somewhere around October 2012.

Mr Kalisch : So that would have been around the time that the deliberations were underway with the previous government around the rephasing of the census and rephasing of the census dollars.

CHAIR: Okay. In terms of the rephasing of the census, I am interested in whether or not there was a period of time when the ABS had ceased working on the 2016 census pending a decision as to whether there was going to be a 10-year census or a five-year census.

Mr Kalisch : Certainly while I was Statistician, there was never a time when we stopped working. As I think I mentioned earlier in response to a question from Senator Hume, we were working on running the census while at the same time we were also looking at the contingencies of not running the census.

CHAIR: And, Mr Palmer, during your period?

Mr Palmer : We were actively planning to run the 2016 census at all times, including during my time in 2014. We had quite separate teams. It was quite a small team working on the alternative option.

CHAIR: I have some questions relating to the night of 9 August which goes to the minister's involvement in the events of that night. Your submission suggests that the first time contact was made with the minister's office was about 8.26 pm. Is that correct?

Mr Kalisch : That is correct. He had tried to contact me a minute or two earlier. I then rang him back for the first time that night.

CHAIR: What did you tell him about the nature of the problems that you were having at that time?

Mr Kalisch : I recall we talked about that it was subject to a DDoS event and that there was an issue with some of the monitoring information at that point. I cannot recall whether we talked about the router issue and whether that was a matter. We then had a number of conversations—

CHAIR: Would you have some diary notes about what you discussed?

Mr Kalisch : I do not, because at that point we were around the operational centre and I did not carry pen and paper with me on that night. But I do recall that we had a number of conversations through the night when we were making a decision on whether or not to keep the system operating, and I understand there were some further conversations around communication aspects to the public.

CHAIR: Did the minister direct you to take any particular action at that time during your first conversation?

Mr Kalisch : No, and it would not have been appropriate for him to do so. As I think I mentioned earlier, the ABS and the Statistician implicitly controls the census. The minister was certainly concerned and very interested in what was happening and was wanting to be kept informed on developments, which was what happened through the night.

CHAIR: Did you contact any other government ministers or officers during the course of the night?

Mr Kalisch : No, I did not, but I was aware that the Prime Minister and the Treasurer were aware. They were advised, I think, by some of their staff.

CHAIR: On how many further occasions did you contact Minister McCormack during that night?

Mr Kalisch : I would have to take that on notice and refer to my recollection. But there certainly were a number of phone calls that night.

CHAIR: Could you also take on notice what updates you provided on each occasion and what response you received from the minister on each occasion?

Mr Kalisch : I can provide you with a broad sense. It was really around the nature of the latest advice that we had about how the system was tracking, what remedial actions were underway at that stage, what the best advice was that we had at the time as to when it might be suitable and the next processes. For example, we did note during the night that the Australian Signals Directorate was going to have further conversations with IBM at their Baulkham Hills site in Sydney, and that it was going to take them some time. We certainly provided some advice that that was taking place from 2 am in the morning. There were also communications with the minister and his office around the communication material and where the advices were in terms of the information that we provided through Twitter and through our website.

CHAIR: Your submission is silent as to when the decision was taken to close down the census. Can you tell us the time that that occurred?

Mr Kalisch : I thought we were actually fairly clear about that matter. We have two aspects. If I can just refer you to page 67 of our submission. At 8:09 we talk about requesting that IBM enable the overload control. Then, at 9:15, we decided to keep the online system closed until the ABS was confident that those three aspects had been dealt with.

CHAIR: It was closed before that, though, wasn't it?

Mr Kalisch : We had enabled the overload control. At that early stage, around 8:10, we still had a skerrick of hope that it might be able to get back up and running that night.

CHAIR: At what time did you actually make the decision to close it down?

Mr Kalisch : I think we are saying that at 9:15 we decided to keep it closed until we were confident that those three conditions were met. Then, at around 10:30 to 10:40 we were very clear that it would not be up and running that night.

CHAIR: Did you advise the minister of this decision before or after it was done?

Mr Kalisch : I think we advised the minister just after it was done. As I said, this is something the ABS has control over and takes responsibility for. It was not a ministerial decision.

CHAIR: Did you advise any other government officers about this decision?

Mr Kalisch : I do recall that I did speak to John Fraser later in the evening about those matters as well.

CHAIR: What was Mr Fraser's response?

Mr Kalisch : Again, he was appreciative that he was being kept informed. It was not a decision—or within his purview to control. But it was certainly appropriate for him, as portfolio secretary, to be kept advised.

CHAIR: Accepting that it is your final decision as to whether to close the census, were requests made to you to close it down?

Mr Kalisch : No.

CHAIR: Did the minister's office contact you for updates about the status of the site, after this time?

Mr Kalisch : We certainly provided ongoing advice to the minister's office through the evening and then also I suppose during the following day as the remedial action was underway, including the teleconference that took place very early on 10 August, and then the briefing that was provided to, amongst others, the Prime Minister, the Treasurer and the minister, at 8 am, and then further advice to the minister following that meeting.

CHAIR: I think your submission says—and could you confirm this—that it was at 2:29 pm on 11 August that the site was fully restored?

Mr Kalisch : Yes, that was the advice I provided earlier to the questions today. I received advice from ASD at around 1:16 pm on the 11th that as far as they were concerned there was as much protection in the site as there could be. At 1:35 I briefed the minister to advise him that the online census would be going back online. And also just taking into account that it takes nearly an hour for the system to get up and running from the time you push the switch, it was then up and running again at 2:29 pm on the 11th.

The other aspect that I do not know has come out very much from the other information and evidence is that by the Sunday night, after the system had been up and running since the Thursday afternoon, we were back on track to our forecast on the number of online forms that we were expecting. So it had taken a few days, and Australians had responded in terms of providing millions of responses. So, by the end of August, in total we had received already seven million census responses.

CHAIR: In the days following the census night there seemed to be some confusion as to the reasons why the site was closed down. I am interested in what discussions you had with the minister, or other officers, on the evening of 9 August and the morning of 10 August about the public explanation that would be provided.

Mr Kalisch : There were certainly, I suppose, a number of speculations, and I would have to say that a number of those speculations came from people who did not know what was actually happening. There was certainly a lot of media reporting suggesting that there was too much load on the site and that that was the cause. That was certainly not the case. It was a DDoS event and certainly the discussions that were held with the Prime Minister's cybersecurity adviser on 10 August, and the information that had been provided by IBM to ASD overnight on the very early morning of 10 August, also confirmed that.

CHAIR: My question was in relation to the minister or his office. Did you have discussions with him about what public explanation would be given?

Mr Kalisch : There were certainly some conversations that took place at 5 am, in a teleconference call that I was party to, where there was some discussion about the nature of the early messages back to the public. I did some radio with ABC News Radio, just before 6 am. I then did a further radio spot on early AM. Then we had a full teleconference at just before 10:30 am. It was a little bit challenging after an hour's sleep.

CHAIR: Sure, I can appreciate that. Who suggested providing public commentary that the site had been the subject of a malicious attack?

Mr Kalisch : The one thing that we, collectively—and I think it is probably fair to say that the ABS and ministers and departmental officials were all in accord that it was important for the ABS and for government to be transparent and open an up-front with the Australian community. This is something where wanted to have their ongoing commitment to the census. We needed to be clear with them around what the nature of what we had seen was and what the nature of the problems and issues were and how we are going to solve them. I think it is quite clear that there was not really one person. Certainly, I was keen to make some public comment from the perspective of the ABS. There was also a sense from government ministers and their advisers that that would be useful.

Senator XENOPHON: Do you acknowledge now that that malicious attack line from the ABS was in fact untrue?

Mr Kalisch : No—I am not quite sure what you—

Senator XENOPHON: The people of Australia were given to understand that there was a malicious attack on the ABS website, which caused the breakdown. The words 'malicious attack' were used, as Senator Ketter has accurately stated. Do you now acknowledge that there was in fact no malicious attack?

Mr Kalisch : There was certainly a DDoS attack, or event. I think the words I had used in the news conference we had at about 10:30 in the morning—I cannot quite recall exactly the words I use at 5:50 am or 6:05 am or 6:10 am, but it was very much around the DDoS events as well.

Senator XENOPHON: My understanding is that you used the words 'malicious attack'. If you did that, do you now resile from that point of view—that there was in fact no malicious attack.

Mr Kalisch : It certainly was a malicious event, and it was a DDoS event. It was quite clear that the intention was to frustrate the use of the census. It was not a benign attack, if I could draw that comparison.

CHAIR: So you have no recollection as to who suggested this terminology of 'malicious attack'?

Mr Kalisch : I cannot even recall using it, quite frankly. I can check the transcripts as to what comments I made in those few radio interviews early on the morning of the 10th or at the later press conference that took place at Parliament House.

CHAIR: Did you have advice from the ASD or other intelligence agencies confirming that there had been an attack on the website at this time?

Mr Kalisch : I think we had confirmation at that stage that it certainly was a DDoS event. The Prime Minister's cybersecurity adviser, I think, was also confident at that stage that that was the root cause of the matter.

CHAIR: So the Prime Minister's office was involved. Was the Treasurer's office involved?

Mr Kalisch : Yes—and the minister's office.

CHAIR: Are you saying that you cannot remember using the term 'malicious attack'?

Mr Kalisch : I cannot recall using it. But irrespective of whether or not someone did use it, I think it is probably fair to say that it was malicious.

CHAIR: Would you have used the word 'malicious'?

Mr Kalisch : It is not generally a word I use.

CHAIR: Was there any attempt by Minister McCormack or any other member of government to direct you to provide some other explanation of the events?

Mr Kalisch : No, it was very clear from the conversations that took place from early in the morning through to as long as I can recall that the only approach was to be open and transparent with the community—that we were to provide a full explanation of what happened.

CHAIR: Did they ask you to stop speaking publicly about the matter?

Mr Kalisch : Not that I can recall. I know there was a time when we were trying to manage who was saying what, but I cannot recall that there was an issue with us not saying some things, or being directed not to say some things.

Senator GRIFF: Earlier today IBM confirmed that whilst you paid for the eCensus software they have the rights to it and are able to sell it, perhaps overseas. Is it not a mistake to relinquish royalties for a leading-edge product developed in Australia?

Mr Palmer : I cannot say that I was party to the negotiations there, but I think we were keen to see the system further developed at other people's expense. My understanding is that we included in the contract a provision for IBM to leverage the intellectual property in the hope that the software would be further enhanced through other applications.

Senator GRIFF: So they were able to sell it for a significant amount of money to other countries, for instance?

Mr Palmer : We would derive some benefit from that, because the enhanced product—

Senator GRIFF: But it would not be a financial benefit, so you are just happy for the development benefit of that?

Mr Palmer : Yes.

Mr Kalisch : I think that is quite clear as my understanding of what happens in the international census world. There is a lot of collaboration and discussion between different countries and, because we all do censuses at slightly different times, there is much learning from each other. We learnt a lot from the Canadian experience earlier this year, the New Zealanders will learn a lot from our experience and then the United Kingdom will learn a lot from us and from New Zealand. It is very much a collaborative community.

Senator GRIFF: Do you have difficulty recruiting field staff? Is recruiting field staff an easy exercise?

Mr Kalisch : It can be difficult in some locations. Certainly having the changed census approach for 2016, where we needed to recruit fewer field staff, was seen as one of the benefits of the new approach. In the previous census we recruited 43,000 field staff. Recruiting field staff in some locations can be tricky, particularly in some of the more remote locations—so getting people with the right characteristics, character, drive and interest. From what I understand, some of the urban areas can also be tricky, particularly for some of the apartment blocks that have gone up, for gated communities and for other areas where it can be difficult to engage field staff.

One of the real advantages of the 2016 census was that we had near real-time information on how the census collection process was going. The advantage we had with that process was that as the enumeration process was progressing through the days and the weeks, we knew the areas that had a lower response and the areas where we had a higher response at that particular time. So we could, and we did, move field staff into areas where we were having a lower response, because the importance of the census is not just to have a high overall rate of community participation but also to have that high rate of participation in local areas. One of the key values of the census is to provide local area information, so we wanted to maximise the participation rate in as many smaller areas as we could. We did actually get some very good field staff. If I could give them a plug to say that they have done a fabulous job.

Senator GRIFF: It was reported yesterday that up to 15,000 field staff have not been paid. Also the report stated that some were in dire straits with some field managers lending them money. Is that the case?

Mr Kalisch : Perhaps if I can start off by saying: do not believe everything you read in the newspapers. It was certainly the case where there were some elaborations made and some assumptions made about a number of things.

Senator GRIFF: At this point, how many field staff would be owed money?

Mr Kalisch : Perhaps Mr Libreri can give you the numbers, but it is a much smaller number and it is against the backdrop that most field staff have been paid. The issue that we are really dealing with, and the subject of some discussion, is some of the additional payments that are required by field staff because they have done a little bit of extra work or they have had more mileage than we expected.

Mr Libreri : We have 38,000 staff at the census and they have all been paid. We assess their payments for their work beforehand and give them a series of payments that cover the time worked, their mileage and other incidentals. Then there are further claims they can make for additional work that they claim to have done. We have some 4,700 of those made to us and up to about 3 November we will have paid nearly 3,000 of those. We expect that there is another 1,500 or 1,800 or so that will be done in the next payday. There are about $100 million in payments that we make to agents and about 5 million of that is made in additional payments. It is a very small amount that we are talking about in this context.

Senator GRIFF: So when someone puts in a claim, what time would it normally take for them to be paid?

Mr Libreri : It depends. We do it on a first-in-best-dressed basis unless there is a compassionate reason. We have had a number of those.

Senator GRIFF: So would it be two weeks, three weeks, four weeks? What would be typical?

Mr Libreri : I think it is between four and six weeks in total elapsed time. But as I said, if there are compassionate grounds, we consider those and do them immediately.

Senator XENOPHON: Has the ABS issued infringement notices or notices warning people that they could be subject to prosecution? If so, how many? And is it still the intention of the ABS to seek to prosecute anyone who has completed a census form but who has failed to provide their name on that form?

Mr Libreri : This question was asked at estimates the other night. I provided some data but I will repeat that for the record here. We have some 10,531 field refusals at the moment. We go through a verification process for those refusals. Are there grounds for that to have happened? Are they actually refusals? Are there mitigating circumstances? I noted that in the 2011 census we actually had 13,000 refusals so we have had substantially fewer refusals in 2016. We have sent out some 1,800 refusal letters. That is just a semi-formal letter outlining obligations.

Senator XENOPHON: Is a refusal not completing the census at all or does that include trying to track down people who have completed the census but for their name?

Mr Libreri : It is the former so it is people that have completely refused or who have not filled in the form. They are the numbers that we have got at the moment. Just for the record, we have sent out some 239 NoDs as of last week.

Senator XENOPHON: Sorry?

Mr Libreri : A notice of direction—that is the formal letter as per our act.

Mr Kalisch : And then it is a matter for the DPP whether they then choose to take that any further.

Senator XENOPHON: If somebody has completed their census form, assuming they have given the details accurately, but for their name, what do you plan to do about those people?

Mr Libreri : For item nonresponse including name but that might include other items such as income, occupation—

Senator XENOPHON: I am only asking about the name now.

Mr Libreri : We are not at that stage of processing. We do not know. We have not dealt with any cases as yet. In the broad, our aim is to produce a high-quality census and we would assess individual item nonresponse such as name in that context.

Senator XENOPHON: So you will not be prosecuting people that have not provided their name at this stage or do you intend to do so? You have obviously reached a policy position on this to date, have you not?

Mr Libreri : We will do exactly the same as we have done for every other census.

Senator XENOPHON: Which is?

Mr Libreri : We will go through a process of quality assurance for all item nonresponse and we will consider then our follow-up action in the context of the quality of the results that we get.

Senator XENOPHON: I do not know what you have done in previous censuses. If somebody completed their census form but for their name, in the past have you sought to prosecute those people?

Mr Libreri : To my knowledge, we have not done that.

Mr Kalisch : The other aspect I will add is that certainly we have had aspects of item nonresponse in the census in past years. Mr Libreri referred to aspects such as income, occupation, age, country of birth. Religion is one that is optional but nonetheless there are a number of compulsory parts of the census where we do get a small but modest level of item nonresponse. In most of those cases, I am not aware that we have prosecuted people.

Mr Libreri : I believe that there are some initial numbers on item nonresponse in our submission.

Mr Sutton : It is probably worth noting that preliminary estimates item nonresponse is lower than in 2011. Our initial investigation around name shows also a very low nonresponse to name as far as we can estimate at this stage.

CHAIR: I have one more question based on your opening statement, Mr Kalisch. You did come out and say that the ABS made poor judgements in relation to the census. You might want to keep your answer as brief as possible given the time, but can you tell us what those poor judgements were.

Mr Kalisch : I think overall there was probably a sense that we certainly appreciated the scale and transformation that the census was undertaking in 2016 compared to past censuses, but there were a number of aspects where I think we will have certainly some clear learnings. One is around effectively identifying and mitigating some of the key risks, such as the DDoS events that we have seen. We are also looking particularly with the communication through to the community to improve the nature of that. I think we underestimated how complex a communication task it was to explain to people the nature of what the census is, the importance of the census as well as the change to the census process. There may be some other aspects around the nature of the initial letter that goes to households to make that clearer and easier for people to respond to. So I think there are a number of dimensions around that.

The one other thing that we are going to look at is perhaps also the focus on the census night. Obviously in past censuses people have had some time to complete the census. The census communication campaign this year did have a strong focus on the census night: 'Go online on August 9'. I think there are some aspects where we can give the population and the community a broader sense that they have a bit more time and latitude to complete the census in a timely manner.

CHAIR: But I think it is fair to say that, proper DDoS protections being in place, it should not have been a problem that the Australian population was trying to do the census on one night.

Mr Kalisch : Yes, and that was certainly our sense—that we had the capability and we had the capacity for people to complete the census on the night, and the DDoS event just should not have occurred.

CHAIR: What about the procurement process?

Mr Kalisch : I think the one aspect that has been raised is that, now that the government has provided us with the extra investment of the $257 million for our statistical business transformation, we can progress that; we do not need to worry about how we would fund and provide those sorts of facilities for the ABS more generally. We expect to have five clear years to plan and implement a very successful 2021 census.

CHAIR: With an open tender process, presumably?

Mr Kalisch : We will need to consider whether we will have internal capacity to look at providing the eCensus component, but certainly if we do go to the market we will go to an open tender.

CHAIR: Thank you very much, gentlemen, for appearing before us.