CamFeed

Thank you. But dozens of projects already approved are expected to escape those requirements. An ABC analysis has found this next generation of centres will be much bigger and more power-hungry than those already operating. Cam Wilson is the ABC's national AI reporter, and he joins me in the studio now. Hi Day, Cam. So take us through what the government is proposing here. Yeah, so a couple of months ago, before the recent headlines about how AI is going to kill us all, we were getting a much more positive message about artificial intelligence from the Prime Minister when he delivered a landmark speech in which he said that Australia would love to get this investment from foreign AI companies into Australia. That's an important technology. But saying, you know, we want to do this on our own terms, talking about you know the social license that these companies need to have to build these data centres here. So kind of like you know, rolling out a red carpet but having a bouncer there, saying you can come in, but you've got to uh meet these uh standards. Those were specifically about things like power and water use. Um, many people have probably heard that there's been community concern over the environmental impact of these data centers. And so the government said we're going to create these new national standards uh that we will have to work with the states with uh and we'll try and legislate by early 2027. Okay, so since that announcement from the Prime Minister, we've heard a little bit more about its implementation. What more do we know now? Yeah, I mean, one of the big challenges with this area is that there is so much appetite at the moment uh to build out the data centres that are the infrastructure that kind of powers the um generative AI boom, you know, products like Jack GPT and Gemini and those kinds of things. Uh, and so while the government is kind of working on these things, um data center approvals continue to happen, construction is is keeping happening. And what the Prime Minister said shortly after that speech is that uh these rules, which are not still expected to be introduced into law for at least six months from now, they will not be retrospective, already um uh built and obviously in construction uh data centres will not have these rules apply. Analysis that we did looking at uh planning documents and company records shows there's at least 25 significant data center projects that have already been approved and are currently being built that may still yet be years away from actually you know coming to fruition that will then escape these rules and therefore not have to apply, uh not have to comply with minimum power and water standards. And so, what does this exemption to these new rules then mean for the communities around these new projects? Yeah, so I mean the Prime Minister was saying, you know, we need this social license, we need to have these rules so that people feel comfortable having them in their communities. Um, you know, one person I spoke to in the story who is upset about a local data center development said, you know, I personally didn't want it either way, but he was saying, at the very least, if it is going to come here, I'd like to see it improve its environmental impact. And that person told me, and this is you know, um, not just limited to them, that they are disappointed that this these new rules that are supposed to set these minimum standards won't come into effect for quite a while. So there is this kind of mismatch between the speed of development and the speed at which these new rules are coming into effect. And so, as you said earlier, the Prime Minister was looking to sort of roll out the red carpet somewhat to investment in these data centers. But is the uncertainty over these rules now affecting investment in Australia's data centers? Yeah, according to analysts I spoke to, they're saying that if it is dampening it, there's so much demand that it's not really stopped the you know huge appetite to build this out. I mean, like these data centers, um I mean, I think Westpac estimated at bringing in about 150 billion dollars in in investment just into the data centers themselves. Um, so there's this enormous amount of money coming in. Australia has this situation where we are, you know, a country that has uh a good energy grid, skilled workforce for Pi of Five Eyes, a number of things that makes us a very attractive destination for major AI companies to set up here. Uh, and they are clearly very keen to do so. And so, you know, an analyst I spoke to mentioned well, like developers are used to dealing with the uncertainty, you know, regulations are always changing. That being said, you know, um, the head of the Data Centres Australia Group said, you know, we think that we are doing enough to already address some of these social licence issues, but we can't um comply with rules that are aren't in effect or won't even kind of established yet, aren't even specified. So the fact that we're not necessarily meeting these doesn't mean that we won't, but it's hard to say that we are if they don't exist yet. Cam Wilson, thank you very much. Thank you. Well, Queensland is racing for its first credit rating downgrade in more than 17 years.

And together they defined the spirit of a nation. Announcement about how Australia was going to seize the opportunity but deal with the challenges and risks of AI. And it could be best summarized as almost a that we're rolling out the red carpet for AI companies to come here. But we've got a bouncer. We're setting some kind of rules around what we want here and why we will welcome them in. And that was specifically around the idea that you know these companies are going to bring a lot of foreign investment. The technology is likely to be you know crucially important to industry of the future. But you know, the government is very aware of the growing uh community sentiment around um uncomfort about some of the data centers that are popping up to uh power this generative AI boom. And so as a result, you know the Prime Minister said uh to have social license to allow these data centers to be built here, um, we've got to set minimum standards at a national level around things like uh power use, energy use, and even where they're situated. Okay, now since that announcement, Kent, we've learned a bit more about its implementation from the Prime Minister. What did he say? Yeah, so I mean, obviously, these um new rules are when he announced them, he said, you know, we're going to uh have to figure them out, we're gonna have to hash some stuff out with the states as well because um you know many of these issues are not just purely federal but also involve the states as well. Um, and so subsequent to that, he said that these rules will not be retrospective, meaning that they won't apply to data centers that already exist, but also ones that are being built uh right now. Um, and so ABC analysis based on looking at um state planning documents uh and also company announcements shows that there are at least 25 new data center projects that will escape these rules, even though many of them will not be built for potentially years. Now, some of these may not end up being built, and and no doubt there'll be more as well approved before these new rules come into place. But what's important about this is that these new data centers are this kind of new generation that are bigger, consume more resources than the ones before it because this generative AI boom, you know, the things powering chat GPT, the kind of AI answers now that you get when you search something in Google, they require a lot more computing power than traditional uh uh computing other methods. And so as a result, you know, um we have to build up this infrastructure, these products have been hugely successful and very, very uh, you know, widely used, but also it means that we've got to build these data centers out, and we know that um there's this kind of rush at the moment to get uh new approvals through to build them out, meaning that these new rules that the government says is important for communities to feel comfortable about them built being built, won't apply to them. Yeah, so then what does this exemption from the new rules mean for these communities that have these new projects already going ahead? Yeah, so I mean, look, um it should be clear that even though these national rules aren't already in effect, there still are state planning rules, and um, you know, many people might compare this to the US, which has um kind of significantly um more data centers already and kind of more ones being built, and they tend to be bigger. Australia is not quite there yet, although these new ones that are being proposed, have been approved, are currently being built, are significantly bigger. They still are subject to state planning rules, but the federal government is like, you know, these federal rules will kind of set more minimum standards that we think are necessary for for communities to feel good about it. And so I spoke to you know, people who are um have data centers being built near them, and they um, you know, including some people who are opposing them, and they said, Well, look, you know, I'm I'm I actually don't want it built here, but if it is going to be built here, I'd like it to have you know certain um minimum standards around things like water use. I want to see them use less water. And so, you know, they were saying that they wished that these federal government rules were already in place, um, but but obviously they are not yet. Um so is the uncertainty over these rules affecting investment in Australia's data centers? Well, I mean, an analyst I spoke to uh suggests that uh there is plenty of demand. We are seeing this enormous um uh interest and appetite to build data centres in Australia. Um, you know, certainly if it is dampening interest at all, uh it hasn't counteracted the the this great um appetite for them. Um, you know, I think Westpac said there's about 150 billion dollars worth of data centre projects in the pipeline. Um, the analyst I spoke to said that like there's a lot of interest. Australia is a very attractive place for this, and that this, you know, if we build them here, there'll be a lot of benefit for the um Australian economy. Um so we're definitely seeing it happen. I think the just really fascinating thing is that these data centres are being built, it is going ahead at such a pace that is outpacing the federal government's attempt to set rules around them. Cam Wilson, thank you very much. Thank you.

Australians were among hundreds of thousands of passengers who were affected. Harrison AI is an Australian company. It was founded in 2018 by two brothers. And what it does is sell tools that are used to help radiologists interpret scans. They've trained these tools on a number of, we don't know the exact number, but according to government documents, fewer than 30 million patient scans that were obtained from Australia's largest radiology clinic, IMED, without patients' knowledge that was being used to train those. They've since become a very valuable company. They're backed by some of Australia's startup scene's biggest names, and also the Australian government tipped in $32 million at the start of last year in a fundraising round that valued them in the kind of mid hundreds of millions in terms of evaluation. So they are one of the success stories of Australia's startups in the last decade or so. So what do we know about these layoffs now? So the company has laid off some Australian staff. I don't know the exact number. Several of them have posted online, but you know, based on discussions with people inside the company, we know it's you know in the mid-double digits is what's supposed. I've of course tried to speak to the company who hasn't responded to a number of requests of comment, uh, other than picking up the phone and saying that they don't uh answer unsolicited calls. Um but the layoffs were done as part of the company's restructure. They're talking about really doubling down on AI inside the company. Now they sell AI tools, but internally they're talking about going to a kind of model where every employee isn't just you know a worker themselves, but thinks of themselves as a uh a player coach or uh a manager of AI agents to do their work. Um, so they have laid off some staff. Uh, a company executive in an interview uh uh that was posted online, says this move has also led to other staff um leaving as well. And around the same time, the company has also launched an affiliated US company that is going from selling these AI tools to directly hiring doctors to use the school uh the tools in a teleradiology service. Okay, so just take us through some of the questions now that are being raised by this new business model. Yeah, so experts I spoke to raise the idea that combining someone who both sells the tool but also hires doctors to care for patients raises a potential conflict of interest because you know ultimately a doctor is supposed to be fundamentally just care about their patient care. That's you know their number one um guiding principle. But if you are hired by someone who is giving you a certain tool, um, you know, there are questions about, for example, could a doctor choose not to use that tool if they're employed by the company, or even at a more granular level, there's a well-known concept in uh AI ethics and and uh healthcare, particularly, where they talk about automation bias. And that's the idea that if a person is using a tool that helps it make decisions for them, sometimes they might defer to the suggested decisions, you know, the automated decisions, even though their better judgment might suggest otherwise, they might just trust the machine when maybe they shouldn't. And experts said, you know, there is of course like a kind of implication about if the people who hire you make the tool, whether that kind of then places pressure on the people, the doctors using the tool to accept what it says. Now, um, the company again didn't respond directly to me, but has said um, you know, it's this US affiliate is actually an independent company, and that the doctors who it hires are independent and can question that judgment. And I should also say that the company, as well, you know, based on what the um experts who I spoke to said that you know this could be an opportunity for a um for these doctors to use a health uh uh tool that could make them more efficient. So, you know, there are definitely upsides to this as well. And the company says, you know, based on it's got advertising material on its website for the doctors that it's trying to hire, saying that we expect that you'll be 30% more efficient using this tool. Um so there's definitely an upside, but there's questions about how do you use that responsibly and and are those kind of checks and balances to make sure that ultimately you're doing everything you can to ultimately care for the patients. Right, Cam Wilson, thanks very much. Thank you.

thank you. Thank you. Let's turn now to other news and internal New South Wales government emails obtained by the ABC reveal that AI company Anthropic was interested in locating up to five gigawatts of AI training capacity in the state. Now, this number represents more than three times the entire capacity of Australia's data industry. It's really quite remarkable. Let's get more on this with national AI reporter Cam Wilson, who joins me uh in the studio. Cam, you you are all over this story. Talk to us about these emails. What what does it show? Yeah, so New South Wales Parliament released a tranche of emails from within New South Wales government about um interactions that the various parts of the government had been having with AI companies and data center companies. Those are actually kind of different because while many of us would know major tech companies who do AI, whether it's OpenAI, Meta, Google, Anthropic, there's also a number of companies that are Australian-owned that actually build the physical infrastructure in Australia. Um, earlier this year, Anthropic's uh US executive uh was making a visit to Australia where they end up signing a um MOU, a memorandum of understanding with the Australian government saying essentially we would love to, you know, put more of our you know, do more work here, put more of our data centers here within the kind of conditions that you would like. It's a kind of very high-level document saying, you know, we're we're into it, we'll start looking into it. But prior to that, um, emails that I got from uh inside the New South Wales government shows that we got a little bit more detail about exactly what they were interested in asking for. Specifically, this number of five gigawatts of uh AI training computer capacity in New South Wales alone. And that's kind of the first time that I think we've heard a figure like that to do with the state that gives us some insight into exactly what they want to do and put in Australia. So break down that number for us. What why is that such a significant amount? Well, uh, there's numbers out today from Data Centres Australia, which is a lobby group representing data centres, predictably, um, saying that currently Australia has 1.6 gigawatts in Australia in total. This ask from Anthropic, which I should say was a very preliminary discussion. It's been some months since then, but suggested they wanted up to five gigawatts. So, you know, my mass isn't great, but it's more than three times that, potentially in one state alone. Now, of course, that doesn't necessarily mean that they would get that much, and perhaps if they got more in New South Wales, they wouldn't get as much elsewhere. But clearly, we can kind of see from that that this company is having discussions with government suggesting that they are really interested potentially in making serious investments in Australia. That would mean some pretty serious infrastructure build-outs. Um, you know, again, over what time frame? We don't know. These companies, you know, are dealing with the here and now, but they're also probably trying to think, you know, years, even a decade in advance. This stuff building out takes a long time. So, you know, I'm sure they're all trying to figure out how to get their ducks in a row. Um, but still, that that number is an enormous amount that kind of lets us understand that we've been hearing that data center companies, AI companies want to come to Australia, they want to build up this capacity, but the amount that they are potentially asking for is such a large amount. It's not just a big thing for them, it's a big thing for Australians, for politicians, for people who are building these things out to be like, let alone do we want it, but can we even give them that much even if we really want to go for it? Yeah, you know, Cam, there's been a lot of discussions about the um uh data centers and of course the kind of resources they would need, uh, what kind of a strain that would be on Australia's own infrastructure and resources as well, but also they are bringing in some investment as well. And this is a hot topic that's going to be discussed in the National Cabinet today. What are you expecting to come out of that? Yeah, so National Camet today was looking at these um what the Albanese government calls its Australian standards. This came out of a speech that the Prime Minister gave about a month ago where he said that we want to get major AI companies to come to Australia to invest here, to you know, employ people here. Um, but also we realize that we don't want a kind of US-style backlash that you're seeing to the data centers being built across that country. And so, you know, they talk about the idea of social license. How do we, you know, um allow or set the parameters for building out this kind of infrastructure in a way that Australians feel like they're not getting taken advantage of, but they're benefiting from. And that uh what his you know recipe for that included a number of things, but the two things that are coming up at National Cabinet today are about energy and water, because um, you know, those are things that the state and uh federal governments have to come to some agreement on. Um we understand that you know some of the states, Queensland, Northern Territory, are not keen on suggestions that uh that um all data centers would have to come with a commensured amount of renewable energy. So you come on in, you say we want this much power, you've then got to source that much renewable energy yourself. Um we don't know what the outcome of that is, but that is like one of the main kind of planks of this idea that the government says if we do this right, we think that we can have it so that Australians feel comfortable and happy that these companies are coming here and setting up their infrastructure. Very interesting. Uh no doubt the conversation and the controversy will continue. Cam Wilson there, AI reporter. Thank you. Thank you.

Thanks for having me on. Well, AI created music will be excluded from Australia's official music charts from this week. The Australian Recording Industry Association says it'll exclude AI generated music from Friday, unless it's substantially human made, does not raise manipulation concerns, and uses legal AI services. For more, I'm joined now by our national AI reporter, Cameron Wilson in the studio. Hi there, Cameron. So, what's brought on this move exactly? Yeah, going from you know what's powering AI to what's coming out of AI. This is something that has kind of come out of rumblings around the world about how AI generated music is uh becoming a really big presence. Um it's very easy now to produce AI music. Um, global charts, uh including Australia's have had songs that people either suspect or know to have significant AI parts of it in it, um do quite well. And so a few months ago, uh a global record association put out some rules saying that we think that these are the kind of ways to distinguish between what is a song that uses AI to assist, but it's kind of a human-made song, versus like an AI generated song. And they said either um, you know, associations around the world should follow this if they run charts themselves. And so today um ARIA, which represents uh the record labels in Australia and runs the kind of top music charts, says it's changing its rules uh to exclude um most, like either wholly or mostly produced um AI music. And um one of the precipitating factors I think that's kind of really seems to have um contributed to this, perhaps sped it up, was a song that was the most uh listened to song in Australia that I think we have a clip of now. When you call my name, it's like a little red. So, yeah, that was a song produced by an Australian producer. Um, and the ABC did some reporting on it saying that people suspected that it had AI kind of involvement uh in it, and then later on, the producer behind it actually updated um information online saying that the vocals and drums had AI as part of it. As I kind of mentioned before, this was like the most song, most listened to song in Australia in July, did quite well in the ARIA charts. So clearly we've gone from this you know idea where you know maybe AI will be creating things in the future to now we are actually listening to songs that are being seriously made with AI. And so what does this change mean for artists? So I mean what ARIA says that this is doing is kind of setting a line in the sand so people know exactly what is and isn't considered you know AI generated. Because like it's actually quite a like difficult thing to figure out exactly where the line is sometimes. Um a lot of computing that we use today involves AI. Like, you know, for example, whenever you take a photo with your phone, it uses computational photography, which involves AI. Is that AI generated? Well, it depends who you ask. In this um, you know, area ARIA is kind of setting down these guidelines saying uh unless you can prove that the um music was um made substantially by a human, hasn't had chart manipulation behind it, and it involves AI, they're gonna say that it can't be on the ARIA chart. Obviously, there's ramifications to that, whether it's from you know winning an ARIA award, or to like you know the fact that ARIA charts are used as an indication to the industry, to the to the public, you know, to history as well, about what has been a big song. Um and the other thing is that uh ARIA has also come out and say we hope that the uh radio stations pay attention to this as well, because we want to support Australian creativity, when it will support Australian uh human creativity and artistry, and so they're also now saying we are encouraging the radio stations not to play songs that we call, you know, right, mostly AI generated and instead play real artists. Thank Cameron Wilson, thanks very much. you.

Thanks, Barbara. Privacy experts and legal advocates have criticized an AI mass surveillance pilot program being rolled out in Western Australia. Since the trial began in June, cameras linked to facial recognition software have scanned more than 130,000 people in Perth and Fremantle, comparing their faces against a police watch list in real time. National AI reporter Cam Wilson has the story and is with me in the studio to tell us all about it. Hi, Cam. Hi Ross. So what is this program? What's it about? So starting in June, WA Police did the first ever Australian trial by police of a live facial recognition scanning program. People might be familiar with facial recognition stuff we have in our phones. That's kind of a one-to-one scan. So it checks your face against the one that you know you're supposed to have to open it. This is a kind of more complex one, which is known as one-to-many. And so it scans, you know, you can look at a crowd, it can scan all the faces and compare it against a watch list to see if there are any matches. WA police says that they have a watch list of about 4,000 people. That includes people who've been accused of a serious crime, uh, people who are have an outstanding warrant, missing persons, and also people who are believed to be a risk to themselves or others. And they have been trialing using this technology to pick up people in the community. Um so they've kind of taken out a van, they go out, they scan the faces as very clear signage as part of this trial saying we're doing this now. And when their system pings, it thinks that it has found someone who matches the watch list. They have someone like a police officer or officers go out and meet the person, see who it is, and then potentially arrest them if that is what the appropriate thing is to do. In this trial, it's been going on for more than a month. They've only released the first week of data so far. They said that they had 33 flags that led to 18 arrests, but also one false alert that identified someone as someone who would who they were not in this early part of the trial. And this has obviously raised privacy concerns. Yeah, so I mean, facial recognition software is quite well known for having racial biases. That's not something that is, you know, kind of inherent to the technology, but is just based on the fact that most facial recognition software and technology has been trained on a corpus of data that is primarily white middle-aged faces. And so outside of that main demographic, they're known to have more accuracy issues because obviously, you know, you train how it works is that they create a neural network. All you gotta know is like kind of like anyone else learning something. You look at something, and as you get to know it more, you're better at kind of telling small differences apart. But if you don't have many examples of it, it's kind of hard to be able to do as good a job determining between them. We so when WA police launched this trial, they said there's a UK trial of a similar technology a few years ago, and they found there was no statistically significant difference between demographic or ethnicity groups for this exact same technology. So they said, hey, look at this, they've done this trial. We know that it actually can be done in a way that doesn't have these biases. However, experts that we spoke to said that that trial was in very different circumstances, both in kind of ideal conditions, so like straight on, good lighting in worse conditions. They they the trial actually showed that it had worse outcomes. And also just purely based on the fact that Australia's demographics are not the same. In particular, the thing that we've really looked at is First Nations people, because First Nations people are already subject to over policing in Australia. And so combining that with the fact that we have not received any confirmation from either police or the technology provider that they trained specifically on First Nation faces, leads experts to believe that there is a good chance that this technology, as it's being rolled out, would be less accurate on First Nations people and therefore more likely to, you know, potentially identify them as someone they're not and lead to that over policing. And how do police respond to that? So I mean, look, they said that this is just a trial at the moment, and this is kind of very early days. Andy Burns and I are have been looking into this, and we've put it repeatedly to police to say, you know, what have you done to mitigate this risk? And they say, you know, let's bear this trial out and see what happens. Um we we hope to hear some more information from police soon to find out how this has worked, to understand more, even about in terms of the people who've been identified and arrested, how many of them are First Nations people, what actual crimes they were for arrested for, what the warrants for. There's a lot that we don't know about this, and so you know, I don't think there's very few people who say this technology, if works well, if it's able to protect privacy, help policing in a safe way, that that's not something that people would like. It's just, of course, like how it's actually working in practice. Have these known risks been mitigated? Um, WA police have yet to kind of show that they've taken serious steps towards that. Thanks, Ross. Thanks, Cam.

Thank you. Privacy experts and legal advocates have criticized an AI mass surveillance pilot program being rolled out in Western Australia. Since the trial began in June, cameras linked to facial recognition software have scanned more than 130,000 people in Perth and Fremantle, comparing their faces against a police watch list in real time. Tell us about this program. Yeah, so WA Police has been trialing this program since June, and as you mentioned, it is a uh AI facial recognition um uh software that is used to um compare faces that are scanned using normal CCTV cameras against a watch list of around 4,000 people. WA police say that the people on the watch lists include those accused of serious offenses, people with a warrant out against them, and people who are missing people, as well as people who've been identified as a risk to themselves or others. This has been now happening for a couple of weeks. It is a trial and is the first time an Australian police force has used this technology live. And WA police say that it is something that they believe that other state police forces are watching quite closely. Obviously, this has raised privacy concerns. Yes, so facial the facial recognition software and technology is well known for having potential racial biases, and that's because it's essentially this is uh AI technology that's trained on uh thousands, tens of thousands, hundreds of thousands of faces, and by through that process, it learns how to recognize faces compared to other faces. Um typically this data uh is um uh has like overrepresented um white um demographic other faces, uh, and so it it sometimes faces issues of with accuracy when it comes to demographics that aren't um as necessarily well represented in that training data. Now, um a similar uh the same technology trialled in the UK underwent official testing, and um that testing showed that there was no statistical significant uh difference between demographics and gender in ideal conditions. Um, but the issue that is raised is whether this technology is being used in those exact conditions as it is being trialled. For example, the conditions that were tried are, you know, people going through a crowd straight on in perfect sunlight. Obviously, that's not exactly the way the world works, you know, different weather, different angles, and in particular, questions about how it might struggle with First Nations faces. That's because it seems very unlikely that this data has been, that this technology has been trained on a corpus of data that has First Nations faces in it. We asked WA police, we asked the provider, and neither said that they did. And so as a result, there's reason to suspect based on that that it is unlikely that it would do as good a job as recognising those faces. What's the ramification of that? Well, if you're not as good at recognising those faces, you might fail to make matches or you might make false matches. And so in this case, what might end up happening is people would get uh wrongly recognized and accused of being on that watch list. Um, obviously, as we uh well know that um Indigenous Australians face already over policing. So there are real questions about um what WA police has done to prepare for this very well known issue. We of course have put it to them and they have declined to say, but they've just said, you know, this is part of what the trial will be looking at. Okay, Wilson, thank you. Thank you.

Speaker 2: Fears about AI evolving at breakneck pace have intensified amid reports of so-called agents escaping from testing systems into the real world. Speaker 2: Now the ABC can reveal the first known Australian case of an autonomous AI hack, and it's raising fresh questions about who bears responsibility. Speaker 1: Up early, getting Speaker 1: a spot in the popular morning gym classes isn’t easy. Speaker 1: So Andrew asked his assistant to book them for him. Speaker 1: Except his assistant is not a person, it's artificial intelligence. It reads my personal emails and manages my calendar and um uh books me in for restaurants and things. Speaker 1: It found a way to manage the bookings, but it went further. Speaker 1: Hacking into his gym’s website, kicking someone off the wait list, and bumping him up. Speaker 2: That Speaker 2: was definitely not what I had intended for it to do. Speaker 2: Andrew is one of the growing number of Australians using AI agents. Speaker 2: Unlike chatbots, AI agents can take a goal, use tools like internet, email, and even credit cards, and carry out tasks for you, sometimes doing things you might not expect. Speaker 1: Last month, leading AI companies were forced to admit their AI models went rogue. Speaker 1: The company says it lost control of the advanced agent in what it described as an unprecedented security incident. Speaker 2: For years, experts have warned that the more capable and autonomous AI becomes, the greater the risk of harm. Speaker 2: It’s been expected for a long time that this sort of thing would happen, even though it happened now. Speaker 1: Soon. Legal experts say the law is yet to catch up. Speaker 1: Is the deployer liable? Speaker 1: Is some third-party vendor or the underlying model liable in that context? Speaker 2: That’s the unknown um area of liability in Australia that we're facing right now. Speaker 2: The Albanese government is already preparing for a future filled with AI agents. Speaker 2: But with AI becoming more powerful and more accessible, that’s looking increasingly urgent. Speaker 2: And the government is being warned that any delay could leave Australians exposed. Speaker 2: Cam Wilson, ABC News.

Fears about artificial intelligence have intensified in recent weeks amid reports of AI agents autonomously hacking into websites. Well, the ABC can now reveal the first known Australian case of an autonomous AI hack. National AI reporter Cam Wilson joins us now from Sydney. Cam, take us through the details of this case. Hi, good morning. Yeah, look, this was an incident that was a little bit uh lower stakes than some of the other ones that people might have been hearing in the last uh couple of weeks. But I thought it was one that is it feels very uh a little bit closer to home. So a um a Melbourne man uh was using a commercially available piece of software, one that anyone can download that was powered by one of the very well-known um AI um uh models that again anyone can get if they just have a credit card and a couple bucks. And he put it together to act as his uh AI assistant. He gave it access to his email, the internet, uh his calendar. And what he did was he gave it a task. He said, Hey, and he contacted it and speaks to it via WhatsApp and he said, Hi, these morning gym classes, you know, they're hard to get into. Uh, can you book into these classes for me? The AI assistant went away, uh, did the um did uh uh went looked at the website, KPAC and said, not only can I do that, I can do uh things that go far beyond what uh you actually asked me to do, I can book you more than a month in advance, far beyond what anyone else could do. Uh, and also um I can actually kick other people out of the class, and I've gone ahead and done that to get you into a spot. And so uh he immediately heard that this happened. You know, he got this message and said, Whoa, whoa, can you undo that? Can you put the person back in? And the bot went, hmm, I'm sorry, I actually don't have the uh capacity to do that. And so this was an example that was like, again, you know, very low stakes, but was an example of how AI agents, this new form of AI computing that not only you know answers questions but goes away and does tasks for you, are capable of doing things beyond what people might expect. You know, you only asked it to book his class, but uh, you know, AI, because of its uh so much stuff is done online and because of its ability to you know analyze um code and do things that it is very capable of doing, was able to um exploit this vulnerability, technically hacking into the website to get it. Um this is uh uh something that's not quite some of these, like some of these big hacks we've heard in recent weeks, but it just shows how this now capability is not only um you know available to anyone, but is actually out in the real world now, potentially causing harm. Yeah, I mean, in this case, it's it's a funny one, I guess. Um the stakes pretty low. But why should we be concerned about it? So people might have heard the headlines over the last couple of weeks uh of companies like uh OpenAI, Anthropic, Meta, uh now even a Chinese lab called Moonshot, who've all disclosed that they're cutting edge, um, top of the line AI models have actually escaped their systems, gone onto the internet and hacked into other companies. Um, this is as a result of them giving them internal tasks, saying, hey, you know, we're testing out the capabilities, but what they found is that their capabilities went far beyond what they expected and were able to kind of break out of enclosures and potentially, you know, in these cases, actually literally get into other systems and expose them. The reason that this is kind of worrying is because um, you know, while not all of the world is software, a lot of the systems that we depend on, you know, our water systems, our electricity grid, a grid, are powered by computers, by systems that are vulnerable to this kind of um attack. And I think you know, this incredible um build in the capability of this technology, the fact that it now is getting better and better at carrying out these attacks, um, but also in some cases seems to be doing it without being explicitly asked, like whether it's booking a gym class or hacking into another company's servers, you know, that in all these situations, the AI was not explicitly told to go and do this by hacking into it, but went and did it as part of trying to accomplish a task that it was given. And this shows one of the big issues in AI that's been talked about for decades by AI researchers, by philosophers who've said, you know, what happens when we get this technology that's capable of doing things, we give it tasks, but we don't understand it and uh how it might actually go and accomplish them. You know, in these situations, it's going, oh, what I need to do to complete the job that I've been given is to break into these other systems. Now, you know, if I said to you, Gemma, like, hey, can you book me into a gym class? I would probably expect that you wouldn't go and uh, you know, hack the website. That's kind of like you know, the unwritten, I guess, social construct. Um, AI doesn't have this, and so now when we have this technology that is exposed to the internet that can do these things for people, but doesn't necessarily have that kind of human understanding of context, it's now a real problem. How do we make sure these really powerful technologies do what we expect and don't unintentionally cause harm, even when we're not asking them to do so? And what is being done to address these issues? So, I mean, look, the federal government has started to address this. You know, we've had ministers talking about this stuff for the first time ever. Uh, it again, you know, all this sounds like sci-fi, but really it is now a real risk that we all need to think about. Um, and they've started to even uh they've funded CSIRO to do some initial research into well, how do you make sure in the future we have this super intelligent AI? How do you make sure that it actually obeys what humans uh say to do and carries out these tasks and all the you know subtasks that it needs to do in a ways that are consistent with you know what we want them to do, and how do we make sure they actually are doing what we ask them to do? Um, but broader than that, you know, there is this real um fear around the world. We're seeing because of these headlines that have been happening the last few weeks, where it really is here now, you know, you're seeing calls for the for governments, in particular the US government, to step in and say, you know, these top level AI labs which are producing these AI technologies that they say are are so powerful. Well, we can't necessarily just trust them to secure that themselves. Clearly it hasn't gone perfectly so far. We actually need governments to start stepping in. So that's kind of the stage that we're at. We're saying that we're we're the technology has gotten so capable and potentially uh harmful that now we can't trust the companies that are producing it, and now we're saying maybe someone else needs to make sure there is a basic level of security so these things don't go you know worse and actually cause real damage. National AI reporter Ken Wilson, thank you. Thank you.

Thank you for having me. Well, fears about artificial intelligence have intensified in recent weeks. This is, of course, is because of reports of AI agents autonomously hacking into websites. Well, now the ABC can reveal the first known Australian case of an autonomous AI hack after an AI agent hacked into a gym class website. National AI reporter Cam Wilson joins us now from Sydney. Cam, this sounds a bit unusual. What can you tell us? What happened here? Yeah, good morning. Uh, so this is a story about how someone who is just using um what are available to anyone, which is some a basic piece of software and access to a commercially available AI product, put them together and was using it as as their um kind of personal assistant. The term in AI that we're seeing increasingly uses AI agents. This is like different from a chatbot where you might say, here's a question, and it'll give you an answer. Instead, you might say, Here's a question or here's a request, and it uses access to the internet, access to your emails, maybe even your credit cards to go and do things on the internet. Um, and what we're finding is that you know they're able to do a whole lot of things because so much of our life is digital now, and so you can do actually a lot just being an AI agent online. But one of the challenges uh that has been kind of feared for a long time, but is now really kind of kind of coming home to Roos, is this idea that you might ask it to do something, and even when it's doing, you know, when it thinks it's doing exactly what you might want it to do, it might do something unintentionally harmful or not like what you want to do. And so in this case, a uh just a Melbourne man was using this this product that anyone could use. He said, Can you please book me into my um gym class? It's in the morning, it's a real pain to get into the class. And to do so, it went and looked and looked at the website of the gym booking website and said, hmm, well, look, I think you can actually, if I do this, and it actually found a vulnerability in the website, booked him into classes that shouldn't have been able to book him into, and then actually later on even kick someone out of a waiting list for the class because he was saying, you know, hey, I kind of want to get into this class, can you do it? These are all things he shouldn't have been able to do, but it was able to do it because these AI agents are incredibly capable, and it doesn't really know the difference between just looking at a website normally and and uh potentially um you know exporting a vulnerability. And in this case, we know we're seeing this problem in a much bigger way happen with all these AI companies who are currently saying we're seeing some crazy behavior coming out of these new models. Cam, how worried should we be about all this? Because I feel like a lot of us are pretty stressed about it anyway. Yeah, look, I it's gonna, it is, it feels a bit stressful, and I think it is one of those things that is now gone from being one of these science fiction plots of you know, AI taking over the world, hacking into systems, to something that we're hearing about not just every day, but as I reveal in the story, you know, something that anyone could accidentally do. I think we're starting to see um governments, companies take this really seriously, but it is a threat that is worth um some serious attention because these AI that uh models that they are developing have both gotten, you know, much better at um you know carrying out tasks and also much more capable of potentially committing these hacks and and doing things for a long time and ending up doing things that people might not have expected. So I I think it is a challenge that that is a serious one. We are going to hear more about it for the average person at home. Like, so you know, I don't think you need to change anything at the moment. I don't really think there's much you can do. This is one of these tasks that governments companies are really, really need to take seriously, and they're increasingly looking at it as a major risk. Yeah, well, what can they do about it? Are there things that can that can put safeguards in place, or is it a case of well, it might just work around those two? Well, so one of the challenges is that many of these major AI companies do have safeguards around their products, and these recent stories like OpenAI and Anthropic having their um AI models break out of what they thought were kind of safe enclosures and ending up going onto the internet and hacking into other companies. In some cases, the AI um um models didn't even know that they were doing the wrong thing. In fact, you know, one of them looked at uh the model after the fact, kind of went through the logs, which is kind of the records of what they did, and the the AI model thought that it was actually still in a testing environment. It still thought it was given a challenge in this environment, and that it had that that the internet that had it connected to wasn't actually real. And so that kind of shows this challenge when you've got these AI models which are being given tasks but can't really know the real world in the same way, and they might do things that they think, even if they have been trained and told not to do the wrong things, might end up doing harmful things. But what we do know is that um, you know, in being serious about these safeguards, ensuring that they are being tested in safe ways, and I think increasingly we're hearing calls for governments to say that we can't just leave it up to companies who say they're developing these products that they say are very, very um powerful, that they are testing safely. Clearly, that's not something we just leave to companies and take their word for it. It's something that uh countries like the US are looking very seriously at. Okay, Cam Wilson, thanks for your time. Thank you.