Government, experts repeatedly warned of issue that caused Telstra outage

Speaker 2: Now, South Australian police have also confirmed today that a report suggesting there was a failed call to 000 due to the Telstra outage that ended in a death are incorrect. Speaker 2: On Wednesday night, Liberal Senator Karen Little posted on social media claiming her office had received a report of a death following an apparent failure to connect to Triple Zero. Speaker 2: She stated an elderly South Australian person had died. Speaker 2: But today, South Australian Police Commissioner Grant Stevens says a thorough investigation has been carried out, and it found that calls made to Triple Zero using a Telstra mobile phone for this particular family did connect, and an ambulance was requested. Speaker 1: This is a tragedy where a family has lost a loved one. Speaker 1: And in the ordinary course of events, they would not have had such significant involvement with the South Australia police, and they would not, this particular person's passing would not have been the subject of such significant scrutiny. Speaker 2: You’ve today reported that Telstra was repeatedly warned by government agencies about this sort of outage occurring. Speaker 2: What exactly were they told? Speaker 1: Yeah, that’s right. Speaker 1: So Telstra, in the years and months leading up to this week's outage, was told that the system that suffered the fault this week was vulnerable to a kind of error like this that could cause some massive issues for the telco. Speaker 1: The system at hand, the one that we’re talking about that’s been identified by Telstra, is its timing system and its dependency on GPS satellites. Speaker 1: Most people think of GPS, they think of location, but another really important function that they serve is also telling the time very accurately to computers all around the world that are used in a lot of infrastructure, including telecommunications. Speaker 1: So, for example, telecommunication networks use this very specific timing to be able to coordinate when one cell tower talks and the other one listens, and also as part of their security, because when a cell tower receives a signal and that it believes it is from another cell tower, the way that it figures out if that's a legitimately from a cell tower, not someone else trying to interfere, is it looks at a security certificate, which is essentially a way of knowing that the signal comes from who it says it is. Speaker 1: That depends on also the time as well. Speaker 2: So when you had the error that happened this week, which reportedly convinced Telstra systems that it wasn’t 2026 but actually 2006, that caused this error, it messed up with these settings that caused the problems cascading across the network. Speaker 2: Now, um, this is something that had been brought up by a government agency, the federal government's critical infrastructure security agency and experts directly to Telstra this year. Speaker 2: And Vicky Brady was asked about uh this reporting today. Speaker 2: Well, I think we need to get underneath exactly what was part of that conversation as I understand it right now. Speaker 2: It was a broad range of things um spoken about, and as I Speaker 2: mentioned, these timing synchronization systems are well known to be critical to a mobile network so now Speaker 2: Cam you mention Australia's government run cyber and infrastructure security centre repeatedly raised timekeeping as a major concern for critical infrastructure providers so aside from raising the issue can the government actually compel or force telcos to act on such a concern Speaker 2: agency that put out those awards is not one that has the powers to do anything but Telstra as a company and a telecommunications provider that has obligations under legislation and rules for critical infrastructure in Australia could be asked to do certain kinds of things. Speaker 2: There was a new rule brought in last year that requires these networks to prepare for all kind of potential hazards to the network presumably having some kind of backup for the the timekeeping system that was affected by this error or just generally having a kind of ability to test and have a resilience when there are issues to its timing system altogether the act also gives the minister in the the the federal government's critical infrastructure security agency and experts directly to Telstra this year. Speaker 2: Now, um, this is something that had been brought up by a government agency, the federal government’s critical infrastructure security center repeatedly raised timekeeping as a major concern for critical infrastructure providers so aside from raising the issue can the government actually compel or force telcos to act on such a concern Speaker 1: agency that put out those awards is not one that has the powers to do anything but Telstra as a company and a telecommunications provider that has obligations under legislation and rules for critical infrastructure in Australia could be asked to do certain kinds of things. Speaker 1: There was a new rule brought in last year that requires these networks to prepare for all kind of potential hazards to the network presumably having some kind of backup for the the timekeeping system that was affected by this error or just generally having a kind of ability to test and have a resilience when there are issues to its timing system altogether the act also gives the minister in the the federal government's critical infrastructure security agency and experts directly to Telstra this year. Speaker 2: Cam Wilson thank you for your time. Speaker 1: Thank you.

Well, more now on the alleged warnings that Telstra is supposed to have been given about its system. ABC National AI Technology reporter Cam Wilson has been following that story and is with me in the studio. Cam, you're also at that media conference with Telstra this morning. What did we learn about this supposed error? So we got a bit more information about the error, but what we heard mostly about today was their response to the error. Telstra's CEO and CFO were both out there talking about how they had heard about it very soon after it happened and how their teams had kicked into response very quickly. But despite that, they said this error continued to cascade across the network, ultimately bringing it down. So, you know, this kind of error is a error that they acknowledged was a known risk, or in fact, well known within the industry, is what I think that they said. But we still did not have much information about how it got to this point. And you reported this morning, Cam that Telstra had been warned in the months, even years leading up to the outage about the error. What warnings had Telstra received before this happened? Yeah, so Telstra acknowledged today that they had received these warnings from government agencies and academics. There is a federal government agency that deals with the security of critical infrastructure, which includes things like telecommunications companies. And they had repeatedly publicly warned that there is a potential issue with the timing system as a vulnerability. Now, timing is something that probably, you know, most people haven't thought much about when we think about the satellites that do timing, probably think about them more in relation to the fact that they also do location. These are GPS satellites that tell us where we are, they help us navigate, but they also serve this important function of telling computers very precisely what time it is. That is important for all kinds of infrastructure, including telcos, which need that information so that their cell towers can talk to each other. You know, they need to know like when one cell tower is talking, the other one's listening, and also as part of their security. When a cell tower receives a signal, it has what's known as a security certificate. This is supposed to be the thing that vouches for the fact that what's incoming is from who it says it's from. Those have expiry dates, and unfortunately, we're told this issue affected their conception of what time it was, which meant that you know these security certificates then were no longer working, which caused the issue. So Telstra said we're aware of this issue. Um government agencies had said, you know, you really need to prepare for this, and even academics have been increasingly sounding the alarm about timing systems as a potential point of failure across telcos and other infrastructure. One who spoke to me said that they had personally gone to Telstra this year saying that they believed it was an issue, but obviously, you know, this has still ended up happening. Let's hear from CEO Becky Brady now about the error. This issue of time synchronization in our network, that's when the first issue occurred. That does take time to propagate across a very large network. Our teams, our technical teams, were working on it almost immediately. So they acknowledged this error that they had known about things like this, but it was also very interesting what they were evasive on during the press conference. Um, they wouldn't say specifically that they had a backup system for the timing system. They just said, you know, generically that they had backups and resilience in their system. So it'll be interesting to see if they do give us some more details on that. And the other thing is they wouldn't actually say that in their legally required risk management plan, had they actually identified this type of error as something that could affect them and prepared strategies to be able to respond to that. That's something they have to do. The government actually has that information. I expect that we'll hear more about this because, of course, this critical infrastructure is something that the government has been putting a lot of pressure on. It's put um it's passed new rules recently saying you need to prepare for these kinds of errors and risks and vulnerabilities. This vulnerability has happened. They didn't seem that prepared. So I guess we'll see how what they had said and whether they had lived up to that or they failed to do the things that they promised to do to address this.

[0.08s] Speaker 2: Meanwhile, Telstra is being questioned as to why it wasn’t prepared for the issue blamed for the network outage after the ABC revealed the federal government's critical infrastructure agency had repeatedly warned telco providers of the potential vulnerability. [16.08s] Speaker 2: A national AI reporter Cameron Wilson broke this story, and he joins me now with more. [21.52s] Speaker 2: Hi there, Ken. [22.32s] Speaker 2: So just take us through what we know about this error. [26.08s] Cam Wilson: So we still don’t know everything, but Telstra has been giving us more information in the days since the outage earlier this week. [32.96s] CamWilson: They've pointed to a software error that happened as a result of an update and have linked it to their GPS system that is used as part of the timekeeping across the network. [49.52s] Cam Wilson: But another important but less known function of GPS satellites is also telling very accurately computer systems what time it is. [57.6s] CamWilson: And timing is a very important kind of system that’s needed across a whole array of very important critical infrastructure, including uh telecommunication networks. [69.12s] Cam Wilson: The role that it fills is that things from as simple as cell towers need to talk to each other, they need to know when to talk, when to listen, and also to matters of security and authorization. [81.36s] CamWilson: So, for example, if I'm a cell tower and I get a message from someone but I don’t know who it is, I’m told it’s another cell tower. [88.24s] Cam Wilson: Um, one of the ways I do that is by looking at their security certificate. [92.16s] CamWilson: And those security certificates rely on uh the time they often have a time and date expiration. [99.36s] CamWilson: Uh so what we're told happened and what has also been reported is that all of a sudden Telstra’s GPS timekeeping system believed that it was not 2026 but 2006, and that created errors that cascaded across the network to cause the outage, but of course, then cause these issues across the country. [118.16s] Speaker 2: And so, what were the warnings that Telstra received? [121.88s] CamWilson: So timing is one of those things that has been coming, has been raised repeatedly by agencies who are looking at Australia’s critical infrastructure and also by experts who've been studying it. [134.76s] Cam Wilson: The Australia’s Critical Infrastructure Security Agency had repeatedly warned critical infrastructure providers, especially telecommunication providers publicly over the last few years that timing was something that they needed to make sure that they were not vulnerable to. [149.96s] CamWilson: They identified it as a potential area that needed to have resilience. [154.76s] Cam Wilson: And so broadly they said, you know, they were looking in particular at these satellite based systems, but also, you know, there are other options for timing. [162.92s] CamWilson: You can have backup systems like atomic clocks. [165.56s] Cam Wilson: Telstra did say that it has a backup system, but that was not able to prevent the error. [170.44s] CamWilson: Um the question of the timing system and preparing for an outright failure is something that can be tested for. [187.96s] Cam Wilson: Of course, it is a very important part of these systems. [199.24s] Cam Wilson: The experts I spoke to spoke about having a variety of ways to prepare for errors like this, whether it is having the backups, doing more testing of what happens uh when something goes wrong to your major provider if it is the satellite provider, and also just generally running drills and even potentially in Australia, having something like the UK just launched, a national timing centre that would not only provide a source of timing for Australian companies and infrastructure, but would also kind of be responsible for keeping in touch and coordinating across sectors in you know preparing for these kinds of failures. [214.92s] CamWilson: The federal government late last year did create a new legal obligation for telecommunication providers to have a risk management plan for all kinds of errors, and one of the specific errors that wasn’t included in this was a failure to these satellite systems. [219.24s] Cam Wilson: So we know that they were tasked with doing it. [229.76s] CamWilson: The Department of Home Affairs confirmed to the ABC that they had received that plan from Telstra. [239.24s] CamWilson: We don’t know what's in it. [241.92s] Cam Wilson: Obviously, you don’t want to just go handing out a document with all of your potential errors. [247.88s] Cam Wilson: That seems like something that's probably not good to have out there under the public. [248.96s] CamWilson: Um so I guess that will be one of the areas of inquiry going forward. [250.12s] Cam Wilson: How did they prepare for this if they identified it as a risk? [253.76s] Speaker 2: Okay, National AI and Technology Reporter CameronWilson there with those details. [254.84s] Speaker 2: Thank [255.00s] Cam Wilson: you. Thank you.