James Pearce, TelecomTV (00:09):
Our final session of the morning is going to look at transformation velocity. So I'm going to ask Naresh to go up first.

Naresh Chauhan, GSMA (00:17):
Hi everyone. Can everyone hear me okay? Good. Hi. So I'm Naresh Chauhan. I'm part of the GSMA. I work in the AI technology and strategy team, and AI is across the whole of the GSMA in lots of different ways, everything from regulatory and policy all the way through to frameworks that we have for responsibility, and then how AI is going to be utilised within the networks in both 5G and 6G environments or 6G futures. And then other strands that we're working on around building telco-grade AI, building inclusive AI, as well as looking at monetisation frameworks and agents in action or AI in action as well. So just concentrating on a couple of the things. The first thing is around building telco-grade AI. So for a couple of years or for a few years now, we've been monitoring how AI is handling and coping with telco-specific data.

(01:20):
So why is that important? Well, we've heard over the past couple of days how operators and the community in general is trying to look at network operations and how to optimise those and automate those, building towards fully autonomous networks as well. So as you can see, this is a model from Stanford and their AI index. The past 12 years have seen vast improvements in terms of AI capability, the ability to handle mathematical equations, complex constructs within AI, as well as looking at particular Q&A responses as well.

(02:04):
And so some vast strides have been made, and even in this last year alone, we've seen a huge amount of progress within AI overall. However, they still don't properly speak telco, so they still don't really help operators. And what we have been seeing is operators looking more in depth at how to utilise AI, as we've all heard. But again, we're still seeing a lack of accuracy, and that's one of the things that we want to try and improve. Why is that important? Well, from a recent GSMA Intelligence research report earlier this year, it's not rocket science to say that a huge percentage or a third of all operational costs that network operators are spending has been on network operations, yet their spend in AI doesn't match that. It's only at 16%, whereas they're focusing on things like customer experience and other uses of AI within their environment.

(03:11):
So there's an opportunity there. And when we're talking about an industry that spends billions, that opportunity is actually large monetarily as well.

(03:23):
So what are we doing about this? Well, earlier this year at Mobile World Congress in Barcelona, GSMA launched the OpenTelco AI platform, and this is a community resource. So it's important to state that this is community. It's not just the telco industry. It is academia, it is operators, it is the frontier LLM companies, as well as innovative startups that are focused on different things. And what we're doing is making available large datasets. So you all know the 3GPP specifications, but those have all been contributed to the OpenTelco platform as well as what we call the PRDs. So these are all the documents that go to building the actual standards themselves. So that's a vast amount of data that has actually been encoded and made available open source for people to utilise. So when we see the leaderboards, what we're also doing is benchmarking people's LLMs that they are building.

(04:34):
So everything from the frontier model companies as well as models that are being built by operators. So that is where we really start to see some of the real insights and benefits as to how and why operators are using LLMs.

(04:53):
And if you look at the leaderboard now, which I can show you outside, what you'll see is that AT&T are topping the charts, and they've just recently released the OTEL 2 model, which is now performing at about 90—in certain categories, it's performing at 96% accuracy. So this has helped them to understand how to adopt it and modify and utilise AI within the networks. But predominantly, they're now looking at instant use cases of different areas of the network, everything from configuration through to management, through to root cause analysis, and then onwards onto self-healing when they go towards building their autonomous level of network as well.

(05:43):
So this was actually built using the largest corpus of data that we released just about six weeks ago, which is based on 400 billion parameters, and that is available for all the community to use. So moving on from that, so this is about building telco-grade AI. The other really useful thing, and one of the key things that I think is coming through in this, is that voice is important, and AI and voice is going to play a major part. So last year we launched a pilot project with the African Union concentrating on building languages for Africa. So there are 2,000 dialects in Africa under major groupings of languages such as Swahili, Yoruba, Kikuyu, Xhosa, all of these things. So what we're building is another community around African languages and building that at scale. So ATLAS is African Tongues and Languages at Scale. Umoja is unity in Swahili.

(06:57):
And what this aims to do is to encode all the different languages from an inclusive point of view. So in the last series of revolutions that we've seen on the internet through over mobile, a lot of the African population and many populations around the world have been left out because they don't have access to smartphones or to full services. So what ATLAS Umoja is aiming to do is make those languages available to people via voice or via smartphones or whatever mechanisms possible and bring them into the fold. So the idea is to actually build out these languages over a period of time and involve governments as well as broadcasters, linguists, culturalists, historians to amass all the data in that country and encode that into an LLM and make those LLMs available nationally for others to lay on their services. So what we are now starting to see is some very clever and inclusive services, everything from a farmer in rural Uganda being able to describe a symptom on his farm with his crop and have a response back in his natural language, in his natural dialect as to what to do with that crop as well.

(08:21):
So why am I telling you all this? Well, it's really simple. We need more and more people involved in the community. We need more and more contributions from everyone who's there, and all of these resources are available. We have a lot of interaction mechanisms. So we launched campaigns about challenges where at Shanghai this year, we recently launched an agentic evaluation. So this is something that Philippe and Sergey were talking about as to how to evaluate agents and what guardrails and what controls and what impact they're having on the network directly are. So those challenges are means of engagement, and that's where we engage academia, students and other companies as well. And then ATLAS Umoja, again, the demonstration shown by Sanas earlier on today is a good example of how language is playing a major part of it. To a certain extent, we could almost say that this is signalling the death of the app economy because voice is predominant, everyone knows how to use voice.

(09:25):
So it's an alternative view. But with that, I thank you.

James Pearce, TelecomTV (09:32):
Brilliant, Naresh, thank you so much if you can come and join us. I'm going to ask you some questions before we let Gaurav do his presentation. Obviously you've launched, as you said, around MWC, the OpenTelco AI project. Can you give us a little bit more light as to how many people have been involved in it and to the kind of shape of it and the structure and how things are going in that way?

Naresh Chauhan, GSMA (09:55):
Sure. So we sort of set an objective of engaging 20 operators around the world by the end of this year. What does that mean? So that means actually having their baseline models or variety of models benchmarked against the specific datasets that we'd laid out there. So these datasets were either contributed or synthesised or operator data that had been stripped of any personal information as well. So we now have 18 of those operators already on board engaged and working with us and benchmarking their models. So as I mentioned, AT&T have been very prolific in this in that they have the top performing model against those benchmarks right now, but they actually released 40 models to the open-source community. So all of these models, the open weights for those models, not the actual data, but the open weights for those models are available on Hugging Face as a resource for anybody to use and start to use as a baseline for theirs.

(10:58):
So the aim here is to try and get everyone 80% of the way along, and then we know that the remaining 20% will be tailored specifically to their environment. So we're at 18 out of 20 now. The other metric is what we were seeing say at six months ago was Gemini was performing really well. It was at 75, 80% accuracy.

(11:24):
And so the frontier models were kind of up at the top end of the league table, but now what we're seeing is six months later on, six out of the top 10 models that are performing the best are from operators. So AT&T, LGU+, SKT, China Telecom, they're all—SoftBank as well, are part of that whole initiative. So what we're seeing is real momentum moving towards that accuracy improving, but also we're starting to see a trend where things are—we've started to see real use cases where people are deploying and getting that operational savings as well.

James Pearce, TelecomTV (12:03):
18 out of 20 is very close to your target. So how confident are you that you're going to hit that by the end of the year?

Naresh Chauhan, GSMA (12:08):
We'll achieve that definitely. We'll exceed that by any means, but our focus is now to sort of say, well, great, we've proven a point here. What we want to see is diversification. So I see it probably in two ways. One is specialisation of models and how those can grow. So it may well be configuration, it may well be management, it may well be self-healing. And then what data is required for that as well. The second metric we think is going to be an agentic framework. And one of the key things we're working on right now is an evaluation framework, which I think many of the operator community, or just the general community, will benefit from when it looks at—one of the other presenters this morning said there's going to be compound errors when agents start to deploy, and that may well quickly get to a stage where it's irrecoverable.

(13:09):
So there'll be so much debt that you won't be able to ever get back. So what we're trying to do very quickly is put in place an evaluation framework, again as an open-source tool that anybody in the community can use to sort of say, well, what is the actual impact these agents are having?

James Pearce, TelecomTV (13:27):
One of the reasons you gave behind why you launched this project was that you said that AI doesn't speak telco, which I thought was a really interesting way of phrasing it. Can you maybe build on that a little bit as to what you mean and why that might be the case?

Naresh Chauhan, GSMA (13:45):
Yeah. I mean, AI 101 is a lot of what you feed into it. In terms of the tokenisation process, many of the important characters and coding, things like parentheses or asterisks, these get stripped away. In natural language, that's not a problem, but in coding language, that's a huge issue. So when we get to the stage where we actually want operations engineers to be able to code using AI, that coding needs to be maintained, that encapsulation needs to be maintained to maintain integrity.

James Pearce, TelecomTV (14:28):
I think it was a really cool example of a project, the one in Africa that you mentioned, ATLAS Umoja. You gave some examples of that being used in action. How wide-reaching has it been so far? What's the take-up? What's the reaction to it been like?

Naresh Chauhan, GSMA (14:44):
Yeah, again, that launched on the 23rd of July. We launched at the African Union ITU Plenipotentiary, and we had a minimum requirement of five countries. So it originally started as a pilot project with Nigeria, who with MTN launched what they call the N-Atlas, the Nigerian Atlas. So again, at a national level, a Nigerian model that speaks three different dialects, Hausa, Igbo, Yoruba, available to the populace for them to layer on top their own services. So when it came to launch, we actually set out a minimum requirement of five countries. We actually exceeded that by six, but there were about 12, well, I would say about eight others that were in the pipeline ready to sign an MOU and commit to that, because you have to think about this in the sense that this isn't a glib statement, this is a government committing resources to that and committing money, committing people to doing that.

(15:50):
So we have a target of, right, I think it was at least, I would say 10 at the end of this year, if I remember rightly. We're already at about 13 in terms of commitments. That's out of 54 nations.

James Pearce, TelecomTV (16:05):
That's really impressive. Thank you so much for the presentation. We'll come back to you with some questions from myself and from the audience. I'm now going to hand over to Gaurav to do his presentation. So if you'd like to take the podium, please.

Gaurav Jain, Rakuten Symphony (16:18):
Sure, thank you.

(16:22):
Okay. So hi, I'm Gaurav. I am VP for Data and AI Products in Rakuten Symphony. Today we are talking about a real journey that we have gone through in telecom with our partner or our parent company, which is Rakuten Mobile. So what we have done there is that we tried to make autonomous Level 4 things on our NOC side, but when we went to that level, meaning autonomous Level 4 is something which is technically a journey which we have to follow. We completed it in a way that there is a process, there are softwares, everything going on. But if I get into the details of it, there are two things which are causing a lot of apprehension in ourselves. One is token consumption. So when we are doing this, we saw that we are very much mindful about what we are consuming.

(17:17):
So there is a lot of token that we are consuming. And as Naresh said, the telco models versus the open models that we have, whether it is SaaS models or PaaS models, they're only up to 80%. So we are looking for something—we don't want to reinvent the wheel, we don't want to do something which is out of the box. What we want to do is partner with everybody who is doing this and try to bring some value to our business. So that's what we did in our case. So we created a model zoo kind of thing where we keep all the models which are making sense for us, where to use, what to use. That's the framework that we have written by ourselves. We'll talk about it in a few slides, but what exactly we have done is to minimise our token consumption, plus the sovereignty that we really want, we kept it as one box.

(18:13):
What we have done, we have taken according to the use cases, all that, we trained the models up to a limit where they make sense to us, they can give us what we really want from that model instead of giving us an 80% accurate answer. On the other side of it, which is more realistic, is: are we really solving the problem? So these technologies, AI and everything, are good, but is it making sense for me to make me autonomous Level 4, where the core answer to it lies in autonomous Level 3? Am I really solving my problem, which is I know my RCAs, I know how to solve it and what to solve, and with what accuracies? So this is a big thing, and these two things together are what I need to make my autonomous Level 4. So I'll go to the next slide with this.

(19:09):
So I got stuck with a lot of tooling, a lot of gravity of data, then the sovereignty that I cannot share this outside. So such models are somewhere going out of scope either, or if I am consuming it, I have to anonymise a lot of things. And the final thing is with all this, I am not making an operational closure. Why am I not making an operational closure? Because it's a probabilistic solution in all cases, because I have not trained it. Somebody else trained it, I'm consuming it. And every time I have to doubt myself, is my solution actually going to solve my problem? Whereas up to Level 3 is what is making a remediation problem for me. So Level 4 is fine—meaning if I know that I'm doing right, I just have to go to Level 4 and execute it. Possibly in future, at Level 5, I can give that level up and then things are automatically solving, erupting, closing.

(20:09):
That's the final thing that we are looking for. But even at Level 4, when we are doing self-healing, how confident are we, and how do we make ourselves confident? That is at Level 3. So we have to be so sure about it that at Level 3 we are doing closed-loop automation—whatever remediation we are doing, we should be sure before executing it. So with that, what we have done is we have created one single platform, because when we are doing it with multiple platforms, multiple things coming together, there are latencies involved, and every new latency is making me distant from my real-time problem solution and the real thing that I'm trying to solve. So what we've done: we keep data in one single platform and then do all analytics before doing any AI—the basic analytics, the type of AI-nisation that I need for my own problems to solve.

(21:05):
We have done that. And eventually we created the AI-nisation thing. So that is also in three parts. First is governance should be right, because telcos are something which are very specific with guidelines. So we kept that as a prime thing, and then we did all those agentic workflows, and eventually we maintained it for a long time. And every time when we are running it, we are very much conscious that we are making the right decision. How we do it is the question. So the first thing, the base layer, is sovereignty. As you see, we need an air-gapped solution because we cannot give this data in any shape or form to the internet in any scenario. So we kept an air-gapped platform. The second is data and memory. So every time you solve a problem, you need that memory to be maintained, because a lot of energy, a lot of prospects and aspects should be taken care of.

(22:02):
Even when human-in-the-loop is coming in, then what exactly have they solved, why did they solve it in that way? That learning is actually adding every plus-plus to my knowledge base. So I have to keep that together. The third one is I don't want to rely on one single model every time. There is no silver bullet in the world, at least in the operational world. You have to learn that every time when you're solving it. So I'll talk about this in more detail. So we kept every option open, whether it is a machine learning model, a small language model or a large language model. So that is what we have done in the orchestration layer. We try to create acyclic graphs. We know what our tollgates are, and at every tollgate we ask that question to ourselves, and that's a framework that we have written ground up because that is where we need our confidence.

(22:52):
The last layer is the agent plane, where—every agent is unique. There are multiple different types of agents which are needed at multiple stages of the work. So that agent, because I am sure at my orchestration layer, we just went ahead and executed it. So that's what we have done. Now coming to it: what model, which model, when we have used it. So we try to—it's not a generalised statement, but what we try to do is solve it in a very basic way. All telemetry things somewhere are time series. So we try to make it very clear that we are going to solve it through simple machine learning models, because classical machine learning models can better solve it, as they know the seasonality and there are very few aspects that we have to bring in. So we should not invest—because we are very mindful about our compute layer.

(23:48):
We don't want to exhaust our GPUs, CPUs on something which can be solved very linearly. So we kept it that way. The other part is there are very specific problems which need an SME to solve, and no large language model should be brought into it, because then it advances the complexity which is not needed at all. It's a very specific conceptual job which should be done by some trained agent. So with Hugging Face, every problem has a solution; some model here or there is going to solve it. So what we have done is we took that particular solution. We are not reinventing the wheel either in the case of large models or small models, or even from the perspective of classical ML sometimes. So we took that particular model, distilled it—no previous data knowledge that we need; we need the computational knowledge and the basic ingrained knowledge which the creator of that particular language model has kept it.

(24:46):
So we use that knowledge and we are giving it back to them sometimes. So that's the way we have done it. So very incidental problems, we try to solve through small language models, and all the complex problems, ambiguous things where we are not sure, we try to do that.

(25:04):
That being said, the human brain is still not replaceable. So there are cases, at least in our network, which humans are still solving. And that's a problem for now, because there is legacy involved, there are systems involved, and there are sometimes human aspects involved. So there are still unsolved problems which our humans are still solving, but we are training this back into our model. So that's how our current models are running, and we are trying to achieve what we are achieving. For doing that, the basic thing that we talked about is the orchestration layer. So here the point is we try to do it through DAGs. So we try to create a framework. The framework has been in production for quite a long time now, where at every step and every time...

(26:06):
So in production-class use cases—if I talk about, let's say, a storage system running, there may be the same error coming or multiple errors coming. So every time the problem is unique, but possibly this unique problem happened 100 times previously. But when we do it, there are 99% chances the same problem is erupting. But you should be confident about it; for that, you have to analyse it every time. That does not mean you are going to burn the same energy that you spent the first time. So what we do is try to create tollgates. If my tollgates are getting cleared, I feel that this is the same problem I'm solving. If at any tollgate I feel that this is not solved the way the previous problem was solved, then at that moment I try to initiate—and that's what an acyclic graph is all about. If it is all ticking down, I'm taking it as an automation journey. As soon as at any tollgate I feel, no, this is not solving...

(26:56):
So first I have to go back. In the first session I saw that we were going for the backup path, but in our case we are not going for backup paths because we are in the middle of a journey. We cannot go back to the basics again. So we are in a problem, we have to solve this, and whatever mess we have created ourselves, we have to solve this as well. So now we have two problems to solve: one is what we have done, and what we have to solve. We bring it together, start a journey, and then we are solving it. So the entire remediation plan we are trying to create from that particular path. So it's like automation with every tollgate. At that particular tollgate, we decide, no, this is not now an automation, we have to go back to the basics and try to create an acyclic graph.

(27:40):
So that is all about that. Memory layer: as these systems need a memory layer, so there is a periodicity that we have kept in the memory. Every problem, at least every solution, and all knowledge is part of the knowledge base. Finally, in any scenario, it's an operational problem, so we have to solve it. That's what we have done. Now, how we are solving it is the question. So how we are doing it is you will see multiple error codes coming from multiple parts of the system. As I said, I'm just taking an example that, okay, a storage system is running within hardware, with a network, and if it is a distributed one, like in our case, we have active-active instances where if you're writing at one place, it is simultaneously routing to two different places. And once all three places or at least two have been written, then only we will commit it in a way that yes, we have written it.

(28:34):
So in this particular type of complexity where we have 100% assurance that we are going to write something and it will be there whatever happens... So in a geography like Japan, it is more probabilistic in nature. So that's how we have done it. In this particular scenario, you don't know when you are getting five different error codes, five different systems failing, which one to prioritise, what to solve first, or what is the critical error versus all other four. So there is a way to solve it. How we have done it is through acyclic graphs. So we created a kind of internal graph system that we have created in which we find out, in a prioritised, weight-driven way, which to solve and which is going to solve in which format, and how much we have to parallelise it and what we have to do in a sequential manner. So that graph will get created, then there is a tollgate.

(29:30):
And then according to the tollgate, we try to scope it and then start solving it. While solving it, there is another thing: diagnose something which is not going to change my real world again—meaning I'm not managing a problem, plus I am trying to solve it in a way that I am solving it, plus in parallel I am not damaging my own system. So I have to be mindful enough that while solving one thing, I'm not creating another problem. So we did it together. So in parallel, when we are solving, we are also diagnosing that there is no new problem coming. Eventually we have to execute it in all scenarios. Finally, so in our service model, we solve it. We say that it's solved, but until the time I verify it, I will not close it. So we kept two different things, and until the time it is not closing any new problem, still this is a problem which is in the system.

(30:25):
So my system is always taking that new acyclic graph, still taking the previous problem into consideration. Altogether, it is solved from that perspective. So that's how my entire network is running as of now. We have done it for the entire NOC as of now up to L2 level, and L3 level is almost half done. So that's what we are doing. From the deployment perspective, we are currently running it in Rakuten Mobile. We are trying to bring this to other partners also. So it's not a concept, it is more of a scaling thing, and we have solved it for almost 25 petabytes of data with all the network that we are running. The last thing that we want to talk about is: does it matter? Yes, then there are three points which are very important. First is every time when we go into it, we solve Level 3 instead of Level 4, because if you solve Level 3, Level 4 is a solvable problem.

(31:27):
Second is MTTR—meaning what is the cost of it? Because if you are burning a lot of tokens or you are doing a lot of stuff where every time, possibly, you will get a right answer, but you will get exhausted very quickly. So you have to know where to stop and what to do. The last point is: is my agent not blocking me in any scenario? So I have to be very specific that I'll execute fast and every time I'm making the right decision. So that's all from our side. Anything that you feel, any questions that you have in mind, I'm here. I can answer you. Thank you very much.

James Pearce, TelecomTV (32:08):
Brilliant. Thank you, Gaurav. If you'd like to come back and join us, we're actually going to go straight to the audience as you've kind of led us in nicely. So I believe there's an Alex in the audience still lurking around. If anyone has any questions for either of our speakers about their brilliant presentations, then please raise your hand. We've got one just up here.

Alex Farrant, TelecomTV (32:28):
I have one. I'm on my way.

James Pearce, TelecomTV (32:30):
Thank you.

Baisan Abbas, Ericsson (32:32):
Hi, my name is Baisan. I'm an account manager at Ericsson and I work with AI and cloud in general. And I have a question regarding unique events. So when we're looking at RCAs and automated RCAs and of course remedies for the problems, very often in telco you find that the issue is quite unique and the machine learning hasn't caught up. What do we do here?

Naresh Chauhan, GSMA (33:01):
Good question. Put it to the test. I mean, in the past year we've run lots of root cause analysis challenges out to the community involving people like Google, the ITU, China Telecom and so on. So what we've tried to do is put the problem or series of challenges in front of an outsourced community of people who've come back and analysed it and looked at the root cause. What we found from that was the actual LLMs had to be slightly finely tuned, but they were coming up with solutions to the problems. And so after a bit of tuning, using that wider community, we actually found what the root cause was. So we tend to look at it as an operator community, we tend to look at it very much in our own environment. We're in an echo chamber. So whereas what we've learned is there's a huge amount of talent out in academia, in the industry wider than just the telco industry.

(34:13):
So I'm not saying that's the quick answer, but it's a good way of actually finding out by recruiting people from outside of your environment. Gaurav?

Gaurav Jain, Rakuten Symphony (34:24):
Yeah. So it's absolutely right. My point is that every problem is unique and you have to solve it whether you are an operator or the partner. In this particular scenario, as I previously mentioned, 90% of the time you will get a solution, but 10% is something you have to train by yourself. You have to bring your own brain somewhere. There are tools and technologies available across the world. But yes, that final 10%, the last mile, should be your own brain and your training.

James Pearce, TelecomTV (34:59):
Brilliant. Thank you. Have we got any other questions in the audience? See a hand up. Yes. I'm on my way.

Wadii Bellaa, Vodafone (35:18):
Thank you. Wadii from Vodafone Group. My question relates to the architecture. Basically in slide eight, you presented the architecture; in slide five, you presented the gate after the diagnose phase. My question here: which gate have you implemented? And are these gates specific to use cases or common to all the use cases as a reusable capability?

Gaurav Jain, Rakuten Symphony (35:51):
Yeah. So first thing is, what is the definition of a gate? So a gate is something where you can close at least one thing. So it's not like it's a stage. It's a gate where you will feel that I have at least completed one milestone in my problem. And why we created gates is absolutely for creating frameworks. So the problem, every time, is going to repeat without some shape and change—there will be some level of change, but it's a repeatable problem. Operationally, there are very few problems which exist in a very unique way where you have to solve it in a very different way. But 90% of the time, there is a fixed framework. Only thing is there is a 10 or 20% slight deviation, and you have to tackle it in a different way. And that's the reason in the framework that we have created, we kept a gate.

(36:44):
So if the gate is achieved, which means a milestone is achieved, we are entering into the different phase of it. So that's how we have solved it.

James Pearce, TelecomTV (36:55):
Brilliant. Have we got any other questions in the audience? No, I think everybody wants to go for lunch, but I'm going to keep you just for a little bit longer with a question of my own. One for you, Gaurav, actually: you mentioned right at the start of the presentation about some of the challenges and problems with the current Level 3 autonomous networks and why that might present problems moving to Level 4. And then you obviously went into how your solution deals with that. How wide are these problems? Is it something that you're seeing a lot when you're having discussions with partners? Do a lot of these troubles exist in a lot of telcos?

Gaurav Jain, Rakuten Symphony (37:34):
Yeah, so these problems actually form a matrix. One is a vertical level and another is a horizontal level. Each system itself may have different levels of problems, and all systems together will have problems. So it's a matrix problem; it's not a linear or a horizontal problem. And when it comes to a matrix, there is a space where you have to see that, okay, this problem is occurring multiple times at multiple levels, so you have to find a fine-grained solution for it. And that's the reason we talked about multiple types of ways to solve it.

James Pearce, TelecomTV (38:10):
Naresh, for you: you mentioned benchmarking during your presentation. I just want to build on that because if you saw the panel before, we were talking about measurability and stuff. Specifically we were talking about financial measurements, but I think benchmarking across the board is something the industry does really well in certain areas, but not so well in others. Can you talk us through a little bit more about what you guys are doing there?

Naresh Chauhan, GSMA (38:33):
Yeah, so the benchmarking is really done in aggregate across lots of different indices and different types of datasets that we have, everything from complex Q&A pairs all the way through to mathematical and then on towards reasoning as well. The biggest challenge has been data, and getting the data to which... Well, it's twofold. One is data in that getting access to data is critical and we just need more. Ideally, we want real-world operator data where possible from lots of different vendors so that we can start to grow and dimension and benchmark against specific vendors as well, and then different use cases like whether it's optimisation or configuration or management as well. So that's the challenge that we have: getting that data as well. A lot of operators are beginning to see that yes, there is a volume of data that they can give to us as long as it's shredded of any kind of PII or personal information disclosure and it's old.

(39:48):
It would just live in cold storage forever, but now it can actually be of use to somebody. So getting that contribution is vital. And so it's an iterative process. We've moved a lot in the last six months. I think we can do a lot more in the next six months as well.

James Pearce, TelecomTV (40:04):
I'm going to finish with a quick-fire question for you both that I want you to both answer, on the topic that we were talking about in the presentation around transformation velocity. In terms of operators transforming, whether it be moving to Level 4 as you mentioned, or whether it's embracing AI and taking on some of the projects that you've talked about or hitting better benchmarking, what's the biggest obstacle? If you could name one obstacle, what do you think is the biggest that is slowing down or affecting the velocity of these transformations? Naresh, do you want to go first? Just one thing that you think is the biggest obstacle that we maybe need to tackle.

Naresh Chauhan, GSMA (40:45):
I mean, the sense I'm getting, not just from the previous six months or the last year we've been doing this, but even during this conference and through Guy's excellent index report, is uncertainty. People are unsure as to where their investment should go right now and they're holding back because they're just saying—it's good to see because people are going, "Are we doing the right thing? Are we investing in the right areas? Do we need massive GPU farms when a lot of what we could do could be achieved on small models on CPUs?" So it's good to see that check, because what we don't want is huge investments and then having to claw that back. So that uncertainty for me is kind of the sentiment I'm picking up.

James Pearce, TelecomTV (41:34):
Gaurav, what about you? What one thing would you think is maybe affecting the velocity of change in the industry?

Gaurav Jain, Rakuten Symphony (41:40):
So from my perspective, the operational problem that we see is the versions. So practically, when you have a lot of network elements, then they have different versions, and this is a layered problem—meaning at every level there are versions. So integrity between them—that when what your subject matter expert is saying versus operationally what is deployed at every level, they have to all be in sync while finding the right solution. So sometimes the solution is right, only the stage is wrong. So that level of thing—if we are really doing an autonomous thing, everything should be in sync. So sync and integrity is a question.

James Pearce, TelecomTV (42:25):
Perfect. I'm going to ask you to just stay on stage a second while I share some messages, but unfortunately we're going to have to stop now. You'll be glad to hear that we have a one-hour lunch break. Lunch is being served in the lobby, and please don't forget to visit our exhibitor pods out there—all the guys out there would love to answer any questions that you have, whether it's about the presentations that you've seen here or about any of the other conversations that we've had here. And we will see you back here at 2:00 PM when Guy will take over, but let's get a round of applause for our guests, please.