Guy Daniels, TelecomTV (00:13):
So for our second panel, we are going to be discussing how telcos can overcome these challenges and ensure successful AI scaling. So before we get into our discussion proper, we need to introduce our guests. I'm going to start on my far side by asking David to briefly introduce himself and we'll come down the line.

David Warnock, Blue Planet, a division of Ciena (00:33):
Okay, thank you guys. So David Warnock from Blue Planet, which is the OSS software division of Ciena. My team, I have a team of solutions architects and me and my team, we have a very high level of customer engagement, so scale is at the forefront of that. So I'm very interested to hear what the panel discusses today and also any questions that we get from the audience.

Nastasi Karaiskos, Wind River (00:56):
Thanks David. Nastasi Karaiskos. I am the Global Vice President for the telecoms vertical at Wind River.

Nabil Lahyani, Nokia (01:05):
Thanks very much. Good afternoon. My name is Nabil and I'm the global head of analytics in autonomous network organization in Nokia. My pleasure to be today in the second panel.

Guy Daniels, TelecomTV (01:14):
Yep, thank you. Good to see you again, bill. And

Ahmed Hafez, Deutsche Telekom (01:16):
Hi everyone, it's me again. So from Deutsche Telekom headquarters, I'm SVP technology strategy and also responsible for data and AI networks.

Guy Daniels, TelecomTV (01:25):
Fantastic. Well thanks very much for joining us. Let's get straight in because as we've stated in the title of this session, scaling is an imperative for ultimate success and we've just heard in fact that Techos face huge difficulties in going from pop to scaling out. Obviously they do. So what do they need to do to achieve this and how do they minimize the risks associated with this move towards scaling? David, are you able to start

David Warnock, Blue Planet, a division of Ciena (01:58):
Some of your thoughts? Yes, certainly. Yeah, so our customers are some of the largest telcos across the globe. If you think about fixed line and mobile operators as they have millions of subscribers, right? So very big numbers there already. And also their networks are hugely complex. If you think about, they're very highly virtualized at the moment, especially in things like 5G, which is almost all virtualized now. So they're very, very complex. We have a customer in North America who receives a million alarms every day and 2.5 billion performance metrics every hour. Now how can an operations team deal with that volume? Okay, I'm talking specifically about OSS here. The only way they can deal with it is intelligent automation. You need to get rid of the background noise and the operations team needs to understand what's relevant at the moment and how to act upon that.

(02:50):
And AI plays a key part in that. So that's the problem statement. If I could offer maybe at a very high level three ways to address that. Certainly in the OSS space, the first thing is platform and applications. I think they need to be cloud native or Vive talked earlier about AI native. So they need to be able to scale up and scale out and scale down to meet demand even. The next thing is the integration technology. So we need open standardized interfaces. George talked about that in the TM forum and we are seeing more and more of things like Kafka now as well, which is a great enabler, but also we're now starting to see agent to agent and MCP and that makes sense. So the ability of agents to talk to each other and to get information from external systems I think is important.

(03:43):
And the final thing I think is clean data. So AI is known to hallucinate, it will give an answer no matter what. And so in order to give it the best chance of a good answer, I think we need clean data. So I think things like dynamic inventory and inventory federation's important there. We were saying in Blue Planet you can't AI what you can't see. So it's important to understand what the clean data is and also specifically my background's in assurance. And so my take on that is to do closed loop, you need to understand what the problem is. So accurate root cause is also key there and I'm also interested in the question of culture as well around that. So I'd like to see how the discussion.

Guy Daniels, TelecomTV (04:28):
Yeah, absolutely David. And we're going to touch on a couple of areas that you mentioned there a bit more details as we develop our discussion. But should we just continue down with some initial problem statements and what generally telcos need to do and how they minimize the risk associated?

Nastasi Karaiskos, Wind River (04:45):
So scaling AI is well beyond a technology problem. It's an operational challenge. The telecom operators are managing diverse domains across core and it each have different workflows, each have unique data performance requirements, and then there are legacy siloed systems that are maybe not AI ready, they do not scale. And then there is the cloud native journey that some operators are now on to transform their networks. And we believe that having a cloud native platform is foundational to scaling ai. Thanks Stanzi.

Nabil Lahyani, Nokia (05:34):
85% of PoCs either they fail or they fail scaling up to deployment. 75% of time in AI projects are spent on understanding data. 74% of operators, they claim that they have to use three or more tools in complex project. To answer the question, what is the problem statement? Still these are data that are documented from sources, from analysts, from the search. And this is an eyeopener to understand that maybe the problem is much simpler that we need first to define the problem. We talk about ai, we can talk about hallucination, we can talk about a lot of stuff, even people, the organization. But I think Philippe in the previous session, he mentioned very well, maybe we don't need AI everywhere. Maybe sometimes we need ai, sometimes we need agent ai, but we need to really understand that the data is key in our problem statement. Then we can address it from process from operation perspective. But that understanding from the entire organization, from the strategy perspective, what is the issue and where AI will help us. I think the driver always will be data in our experience what we are talking to customers.

Guy Daniels, TelecomTV (06:52):
Sure, yeah. And those data points you mentioned, they were very revealing. Ahmed, we've spoken all day about this, but ultimately we do need to scale. We have to get up there and it's not easy.

Ahmed Hafez, Deutsche Telekom (07:05):
No indeed it's not easy. And I like the challenges from Philip in the previous slide so we can have a good conversation around it. I don't want to talk about data again, I think everybody's aware of the data problem, but just small comment on the data. Data has many dimensions. It's not only availability of data or quality of data, but also it's the whole governance around data and its ontology and lifecycle because you can have it and then you have problems afterwards. You have problems in the chain afterwards. So let's say we have data foundations to sort out, this includes the data availability, quality and all the facets with it. But there is also AI foundations and I would maybe double down a little bit on the AI foundations element. The problem with the agen ai, I mean it's the value and the problem and we're not new for challenges by the way.

(07:57):
We have all encountered new technologies in telecoms, right? And in the beginning we had a lot of challenges and as we went through, we solved the challenges one by one and we decided in some cases to not do things because the challenge outweighs the value. So this will be the same. It's nothing dramatically different except for the speed. We have not encountered such speed before. So we've always been working on things that we have software update every two years. Now we have something new every two weeks, what is the pace? And the problem is the pace much more than it's the exact problems. We will solve these problems. Most of them will be solved one by one. And if we are cognizant about the problems, then this is 50% of the past to solve. So the AI foundations here means that, okay, how do we handle when we deploy agents, let alone development, how do we operate them?

(08:53):
If I have handful of agents we can manage, there's no big problem because there's no scale. But once we have 100 agents in the network, how would that look like? How do we utilize the MCP service? Do we use one for all tools, 10 for multiple tools or one for every tool? These are design decisions. I'm not claiming that we know that the best answer. We have something that we go through, but I'm not claiming it anything because we have to be humble with ai. Again, this is one problem. The other problem is the culture definitely. I mean you are going to people telling them that you are actually going to automate, but listen, that agenda, AI will do your job. So this is also something that we need to be also understanding and also thoughtful about how do we handle and approach that? Because reality is people work will change, it's not will disappear, it will change.

(09:46):
And then you would have actually to monitor agents, we will not go away. I mean we will need to monitor agents, we will need still to give the intent to the agents. They'll not run completely in autonomy and we will have long path until we get the agents to work precisely and get the intent we want. So the understanding of the journey itself, the understanding of the challenges and taking them on is for me the most important element when we talk about these elements or these things, we have been there and we will continue the journey.

Guy Daniels, TelecomTV (10:16):
You said there's something new every two weeks at the moment it feels that every two days I'm reading about some new development. It's a crazy cadence there we're seeing, but we should pick up on that, on the skills because this is an area we've covered data a little bit already, but David, let me come back to you because you mentioned this is an area that interests you as well.

David Warnock, Blue Planet, a division of Ciena (10:34):
Yeah. And Nabil, you talked about the fact that 85% of PoCs fail. So we were, I'll tell you an anecdote about that. So we were involved in A PoC last year with a European national service provider. They'd had a legacy solution in for 30 years and they wanted an AI powered solution to replace that. Now the person who implemented that 30-year-old system was still there and in fact he was actually assessing our PoC and every turn he compared how our solution compared to the legacy solution. And so he was really blocking the PoC in that respect and in some ways he was going against their own business case. So I think these are the kind of obstacles that we need to overcome. And the way that we found from overcoming them, there are always technology evangelists and AI evangelists within an organization. They're the ones that you need to focus on and even to elevate them within their own organization because they're the ones that will get you through the PoC. So in that case, we were successful and we're actually doing an implementation there now.

Guy Daniels, TelecomTV (11:42):
I mean it's something we've, and thanks for that note there. Every DSP leaders will form event we do. One of the sessions is all about culture, it's about skills. Every cloud native telco event we do, it's about the skills, it's about the culture, it's about the people, people, people, people. We lose track of the people at times. We focus on the technology and the technology is complex. We've just seen the agen AI slide from Philippe and it hugely complex. So any more thoughts about how we continue this never ending problem of attracting the right talent at right time for the right roles?

Nastasi Karaiskos, Wind River (12:18):
Yeah, so we're seeing emerging skill set requirements for both AI and ml but also edge experience across Kubernetes, CI/CD containers. These are the areas that are growth and I think culturally leadership needs to support that transformation and development and also that experimentation within these groups to develop them and encourage them and fear maybe is another one of that cultural change. And we see very quickly, very minor outages in some cases becoming worldwide news. We saw that with AWS less than an hour an outage in DNS earlier this week, but it's made worldwide headlines the week before, a hundred thousand Vodafone subscribers in the UK at home or on the mobile network again becoming World World news. And we all saw last summer the impact that the CrowdStrike outage had. Again, simply software changes not being fully tested and rolled out and the impact of that within 30 days they'd lost 30 billion of their market value and thankfully they've recovered.

(13:27):
But we are in a world where any simple errors can really make huge global headlines and impacts, which nobody wants here at Wind River part of our business and our history. And if anyone's flown here on an Airbus or a Boeing, it's highly likely that that operating system was based on Wind River on our real-time operating system. So we know all about keeping systems working without any failures or outages and we've transferred a lot of that knowledge and experience into our Wind River cloud platform and orchestration and analytics tool that we provide to the service providers.

Guy Daniels, TelecomTV (14:05):
I'm just wondering about the types of skills. Obviously we need a whole range of skills, but Nabil, think about your role. If you were tasked with hiring X number of new engineers, software engineers, are you looking for specific AI skills or is it more important that they can coordinate tools out there and act as this overseer of various tools?

Nabil Lahyani, Nokia (14:28):
I think I will disappoint you if I don't tell you three magic skills. I don't pretend to come up with something that is the reality is telling us that the first basic skill is required is leadership. Actually we don't have a clear, in many cases we see there is not top down clear understanding mandate. This is why we need AI and this is the benefit I will bring to my company. Use cases operational efficiency can be sustainability, whatever. Number one, clear vision mandates top down that people follow the same of that leadership, sorry, I believe it's relevant. Now when we come to AI itself, let's look at other industries, even AI and data. I put them in the same package, banking, gaming consumers. There is a lot of progress and return of investment there and this is why the CSP still saying, hey, I think we discussed that here before about how to generate revenues through ai.

(15:26):
So if we want to bring the value, definitely we need AI data scientists. But to me, based on my humble experience, the key one is that possibility to coordinate different skills. I counted last week I had a workshop more than 20 companies, they claim to do anomaly detection, AI ml, well very good hyperscaler without naming them, they all claim to do also anomaly detection. If I tell you now, I haven't found one single operator with a snap of fingers able to answer the question, how many subscribers I have who are facing issues in buffering in Netflix and where they are located and where the root cause analysis. To answer this question, you need skills of ai, of ML of cloud. But most importantly please let not forget this, the domain knowledge, what does it mean a drop call? What does it mean the congestion? Is it coming from control plane? Is it coming from user plane? That correlation of data. So I really emphasize the domain knowledge combined partnering or bringing acquiring skills on AI ml. That will be the nice cocktail, but it's not like, hey, if I hire the best AI engineer, my business will grow, in my opinion.

Guy Daniels, TelecomTV (16:44):
It's very good. And you're nodding here, your thoughts?

Ahmed Hafez, Deutsche Telekom (16:47):
Absolutely. Actually one of the things that maybe we also discussed it previously is that the domain knowledge in my personal view is more important than the AI knowledge today because you can actually as a company, we can also hire, get people from outside, get companies to help us with AI and data, but no one can help us with the domain knowledge we need in-house domain knowledge. So if I talk about networks, domain knowledge is number one. The second one for me is not the data scientist or the data engineer is an AI engineer, which is a new term that's coming out, which is something else. It's somebody who takes the problem of a domain and try to create a design of a solution. Then the AI scientists will step in and the data engineer will step in to solve the problem. But I need the designer of that.

(17:34):
So this AI engineer type of profile which is now popping up a little bit here and there is becoming more and more demanded. So I need these people with knowledge and usually these people work with the domain knowledge. So either you take the domain knowledge people and evolve them to become the AI engineer without having to know a lot of details behind the data science and the data engineering itself, the AI engineer, they can become an AI engineer. So we classify that you have three classes. Either you have an exploratory user, a power user or a developer. Someone can develop things or we want to transition everybody to be at least a power user. Not stop exploring, but use the AI for something, be able to do it, use it in your work and use it in your network in whatever you do. And then some of them will evolve of course to be developers or can develop things with AI developers.

(18:24):
I don't mean develop and developing code, but even using no code to develop solutions. And then the AI engineer comes in that place where you evolve towards that. So of course every one of us have a lot of training programs and a lot of different workshops. There's no doubt that everybody was does that. But what I believe on top is very important is learning by doing. And the experience shows it's a completely different knowledge gain when you actually put more people into the projects. So we started doing that. We have more people you can really see the ramp up of the knowledge. All the people coming into AI projects knowing nothing about ai, nothing. So almost just JGBT, that's their knowledge level and they come in and a year through they become really deep into ai. But by doing take the courses you want, people forget, but doing things you remember you are in.

Guy Daniels, TelecomTV (19:19):
Yeah, quite right.

David Warnock, Blue Planet, a division of Ciena (19:20):
Just to add to that, I mean I think you said people knowing nothing about ai, but there's also people who know nothing about OSS trying to move into the space. So the likes of the cloud providers, the Googles, the Microsofts, the Amazons of this world OSS is full of standards, full of processes and full of people who operate those processes at the moment and it's not an easy area to just walk into and take over from the cloud provider's perspective.

Guy Daniels, TelecomTV (19:48):
True.

Nabil Lahyani, Nokia (19:49):
If I may add on top of Ahmed and David, I fully agree by the way, I always recently I was discussing about this ramp up and also of capabilities and my conclusion with the team was it may take us as you said, couple of gemba projects to acquire speed up the ramp up on AI ml but how long it'll take you to become core plus RAN subject matter expert

Ahmed Hafez, Deutsche Telekom (20:14):
10 years?

Nabil Lahyani, Nokia (20:17):
But we tend to think no if a hire couple of AI experts then problem solved. Absolutely agree.

Nastasi Karaiskos, Wind River (20:23):
And I think this is a great real life example that amis just given there so many people, we've talked about it earlier today, I'm going to lose my job, I'm going to be role's going to be redundant and taken over by ai. But we are absolutely not seeing that. And you've just proven that example in one of the world's biggest operators and there is a development path and it is exciting for people to go into this. Companies need to, as I said before, leadership needs to champion this development path for people and encourage them to take up. And we're all doing it. We're all doing it at home as well. And we heard earlier about coding and the process and how quickly, for example, one of the partners here can liven up BSS, right? This is a great example.

Guy Daniels, TelecomTV (21:05):
I'm just wondering are some of these new skills we're discussing and new roles you mentioned they're the AI engineer that's a recent development. Are these skills and people and roles, okay, I get it for the PoCs, I get it for the testing, get it for the exploratory work. Are they the same people and roles that will then take this and scale for telcos,

Ahmed Hafez, Deutsche Telekom (21:26):
Right? Maybe I make a comment here. In DT, so I'm telling my teams I don't want anybody to put any objectives as proof of concept anymore. So we are not putting that, that's not an objective. That would be a means towards an objective and objectives to have minimum viable product or a launch of implementation. So PoCs are not an objective anymore. So yes, in my case, in our case, that would be actually yes, implementation. So that would be the same thing. They have to prove things until it gets to at least one of our markets up and running. When this is up and running then we want to just scale to other natcos and so on. Then of course they can step back and other teams can step in. But yeah, implementation.

Guy Daniels, TelecomTV (22:09):
I'm so glad you mentioned that. That's made my day the end of the PoC maybe. Let's hope. Thanks very much. Let's move on because another area I do want to cover on this panel is integration and integration challenges always comes up when we look at new technologies and areas. I'm wondering if you can help me out and where do we see the greatest integration challenges when we start to embed AI in our existing workflows? Has anyone got any thoughts? Want to start us off with on that one? Nastasi if you want think about integration?

Nastasi Karaiskos, Wind River (22:42):
It's a challenge because as I'd mentioned before, a lot of legacy systems OSS/BSS systems that maybe based maybe are not already AI native and not scalable and then there's a skillset gap as well. There are umbrella systems out there that can assist and mesh these together, but the partners that have the expertise at doing this and doing it at speed, we had an example with a partner, a customer of ours, an operator in the us. They were facing some end of contract challenges with their existing cloud vendor and part of the fear for them was how long it was going to take to move from that existing cloud partner. And we used AI and our automation engine to migrate over 24,000 sites after a proof of concept in a period of less than seven weeks. So that really just proved and demonstrated to them that it was possible to scale and automate the migration. And for them in terms of savings per site, they were averaging three servers. They've gone down to one significant savings, not just operationally but also in the energy consumptions.

Guy Daniels, TelecomTV (23:51):
Thanks Nastasi. Nabil.

Nabil Lahyani, Nokia (23:53):
Yeah, I think there is no one single pattern when it comes to the challenges in integration due to the fact of the complexity of the network and the other fact that Ahed mentioned, which is speed. We want to run fast and although we have such complex network in Nokia in autonomous network, we have something called the fabric leveraging the data suite. And that's why we believe having access to the data in two dimensions is very important. Most of the networks are multi-vendor. So when you integrate, let's say I know Nokia very well, I know the interfaces, but half of the network sometimes the entire network is other vendors. So we must have, if we go back to the domain knowledge to understand how quickly we can integrate whatever is a new AI model agent, whatever it is. And when it comes to the domain, it's also very, very important when we talk about, and I think we forget, go back to the basic, what is the business issue we are trying to solve?

(25:03):
When you start asking those questions, it can be sustainability, it can be quality of experience, it can be, okay, what is the experience of the gamers society is changing how operators data is consumed is different. And to answer those questions from the business perspective, you end up in the problem that, oh, I need to understand how to integrate to core, how to integrate to run as I mentioned before. And that is a paradigm shift. It's not about the network anymore. Yes, network is important, it's about the centricity of the subscriber, where is the real experience a user is having and then to integrate to different interfaces and the platforms, it's becoming the real challenge. If I put again on top of it, the speed, I think so multivendor, different platforms and last comment if you allow me, I think if you don't do that with the ecosystem of partners, it's mission impossible. So nobody will claim to say because there are different deployment models. Some they want SaaS, some they want hybrid, they want fully cloud, they want on-prem. So I think these are complex architecture design issues that are making difficult integration of AI.

Guy Daniels, TelecomTV (26:13):
Thanks Nabil. David, from your domain, what do you see as the greatest integration challenges when integrating AI and how is that holding back the scaling challenge?

David Warnock, Blue Planet, a division of Ciena (26:26):
Yeah, I mean I think it comes back to the problem of having clean data and sources from where to acquire that clean data. There's a lot of legacy inventory management systems out there. They're very static. They do discovery maybe once a day. A lot of the data in there is out of date and if you're going to make business decisions based upon that data, then you're off to a bad start. So I think it's important to have dynamic inventory, federated inventory across the different domains and of course you need expertise in each of those domains. If you think about mobile service, it goes across the the core, the transport layer. You need to have good data from all of those different domains in order to do end-to-end. We talked about experience management in order to be able to do end-to-end management of the services across those domains. So I think that's very important.

Guy Daniels, TelecomTV (27:19):
Thanks David. Ahmed, any thoughts and ideas on integration challenges?

Ahmed Hafez, Deutsche Telekom (27:23):
We look into integration in three areas. One area is the inputs to ai. The other area is the output of the AI and the one within AI itself. So if you talk about scaling, the one within AI becomes profound because scaling means that I have multiple agents, they're doing different things and they need to talk to each other. Thankfully there is a to a coming up and A to A also have been developed and will develop continuously until IT reach, let's say the best level that we can use but it's still usable today. This will enable us to make sure that we can scale agents and the agents can talk to each other. So we are not replicating functions, we are consuming functions properly. When it comes to the biggest challenge is I think already well said is data. And because data is not just one integration level that's source, you have to take the data from the sources, you have to put them into ingestion layer, you have to select that from the industrial layer to the data platform, from the data platform upwards to ai. And the AI has a chain, there is a pipeline. So it's a whole set of integration and this is the most complex part of the integration. The integration towards the network for closed loop automation is not the biggest worry. I see. So that's something that we have well-defined interfaces, we can manage that. So we have a lot of experience with that if we do things. But the data challenge and the AI to AI part.

Guy Daniels, TelecomTV (28:45):
Well thanks very much everyone. Let's bring in our audience, shall we because and Chris. A hand goes up straight away in the front row. You don't know what I was going to ask you. I love to ask our audience what challenges they see and the problems that they may have.

Audence Member (Chris Stallmach, Red Hat) (29:00):
Yeah, this is a great collective of experts for this question. We talked a lot about AI today of course, but we haven't really talked a lot about the models themselves. So I would love to get the perspectives from the panel in terms of when it's a good idea to use existing models, when is it a good idea to develop your own models in terms of resources and other things are i'd involved in that. So I'd just love to get a perspective on, because we have operators, network equipment providers, operating system representation to get a perspective on that.

Guy Daniels, TelecomTV (29:42):
Hey, it's perfect opportunity. Good question. Thank you much indeed. Who would like to tackle that one first?

David Warnock, Blue Planet, a division of Ciena (29:48):
I think the existing models are a good starting point. So I come across a lot of organizations that have a whole bunch of workflows written and they're based on scripts and I think it's difficult to just take a big bang approach to that, right? So you've got to acknowledge that the customer has that already. You've got to put in place a system that can leverage those scripts and then over a period of time, I think you need to think how you can move away from 'em more towards like an agenda AI kind of approach I think is the best way of doing that. But I don't see a big bang as being a good fit for most customers.

Ahmed Hafez, Deutsche Telekom (30:24):
David, maybe if I can, I would agree with David and actually if we go back to speed, I mean the speed of developments and the LLMs and the tiny models and the recursive models and it's something that we cannot easily keep the same pace so we should use what's available and then test and then if it's not satisfactory then we go to the next stages. So I would say that this phase where we are learning and also monitoring that we can apply what we have and what we can use from three models, other models, available models and then at the latest stage we can see if there is any need to do those. So I'm not sure who was mentioning, I think Philip was mentioning that I was pleasantly surprised that the frontier models have a lot of knowledge about telecom today. I mean I was talking to my boss today by chance and I was exactly talking about the same thing with him because effectively they have put a lot of information about three GPP and networks and so on in the corpus of these models, you'd be surprised how knowledgeable they are.

(31:23):
So it is not about the fanciness of having my own model and then training my own model, my own data sounds nice, but is it really needed? That's a question that we need to check before we get into this because it can take us into a rabbit hole as well. I mean you train the model that once you train the model there's a new one coming, should you take the new one and train it again or should you use that? So that's why I think I would agree with David that we go with the existing models and only at the latest stage you can think about training models.

Guy Daniels, TelecomTV (31:49):
Great, thanks.

Nabil Lahyani, Nokia (31:51):
Yeah, I agree. What having said and actually it's a very good question. I would pick up what you said we are learning. I think that's very, very key because we start as a baseline with some known models, some existing models, but I would say it depends on each use case. I go backwards because sometimes we pretend to jump immediately to unsupervised model that doesn't require any injection from external knowledge or really to give time to the model to learn. I tested myself some of the correlation of anomalies between core and RAN in different projects. I mean the same customer, different countries and it was disappointing somewhere and we had to tune completely the model. So I think that possibility to learn fast and adjust, it's the right answer of combination. But again, sometimes we think we bring them and that it just as an note was an escalation call and I realized we had more engineers than lines of code for one model because at the end you end up with people just putting a knowledge. So I think it's, I said by the colleagues, it's a combination of both.

Guy Daniels, TelecomTV (33:08):
Thanks Nabil. Nastasi, do you want to add?

Nastasi Karaiskos, Wind River (33:10):
I see sometimes we mentioned earlier why so many the high percentage of PoCs not succeeding and when those models are thought through and the use cases is the success criteria or the exit criteria achievable? And is this one of the reasons why so many are failing? Because are we always thinking about the customer, whether that's an internal customer, another department or the end customer, the consumer? What are these AI models going to bring in the benefits? What's the return of investment going to be? Is it fewer faults, is it faster fault resolution, quality of service, for example, improved customer experience, personalized customer experience? These have to be the goals and the end achievements that we're looking to gain to make these use cases successful.

Guy Daniels, TelecomTV (33:56):
Great, thanks very much everyone for that great question. Thank you very much indeed. We have time for another question. Anybody else want to quiz our audience or does anybody else think that the challenges we've been talking about are too limited, there's more challenges we haven't yet discussed. Maybe there's something we've overlooked that you are facing.

Nabil Lahyani, Nokia (34:15):
Questions are filled of chairs.

Guy Daniels, TelecomTV (34:16):
Yeah, come on. Anything at the back here?

Audience Member (34:27):
Thank you. So there was a lot of topic about domain knowledge, which is an asset we all have in this room sitting here and others may might not have. Could you elaborate a little bit? What is the operator part? What is the vendor part there? I remember ahed one sentence, you spoke in the morning about the vendor providing curated documentation that is machine readable. This is a new impulse I've heard for the first time and I think there's a responsibility on the vendor and on the operator side, how do you see the split?

Guy Daniels, TelecomTV (35:03):
Thanks very much.

Ahmed Hafez, Deutsche Telekom (35:04):
So I can start. So when we talk about domain knowledge, take the RAN. So when we develop the rank guardian for instance, the first thing that we would do is to find out how you want to deal with your problem. So the problem statement, what we defined and then how exactly are we going to deal with it, which KPIs should we put? How do we monitor these KPIs? Do we create the thresholds or boundaries? Do we have a combination of the KPIs to consider? So the logic itself of how you operate the network, how you manage it, how you close loop it, it's all coming from domain experts. Then the AI experts, AI engineers first, which actually take that. So can I put that into solution? Can I solve it better than the human process? Now I understand the intent and the critical angles. Can I bypass part of the process The human does because not always you have to follow the exact same.

(35:56):
When you employ machines and especially agent ai, you may need to break down things into smaller tasks, not like humans. Humans are much more intelligent and then can deal with a myriad of tasks and a large chain of thought. AI and agent AI is not as intelligent as us. So you need to split how do you split the split itself as a design that you can get help of an engineer, an AI engineer, but the domain expert has to explain the problem and how the problem escalate or how the problem is built up. So they have to work together very closely.

Guy Daniels, TelecomTV (36:30):
So can I just add, when you say the question was about machine readable as well, does that mean it's like a JSON structured? Is that what you mean by that?

Ahmed Hafez, Deutsche Telekom (36:38):
For example, JSON is a very good example. So if you define it this way, but I can give you another additional example. The documentation we get for example about lock files, which every one, so we say for example it describes the latency of the flow. This is very short sentence. Even an expert, if you're not domain expert in the, you wouldn't understand it. So it's not even today, not even human friendly. It is an expert friendly, yeah, that's the one. I know these ones. But if you want to explain that to the agent agents, you need to explain the relevance of this field, the importance of this field and if it has a relation with some other parameters. So you need to have much bigger explanation that gives a little bit more context to that attribute, then the attribute becomes of value. Some other attributes say, so this is a redundant attribute used for X because if you go for example data we have at least for every machine you have 10 different types of logs or files that are generated and then every one of them have a hundred attribute. That's a thousand attribute. Are they well-defined? Not they're not well-defined. And I can be challenged here. If any suppliers want to challenge me back, I'm happy to take the challenge, not well-defined. Some of them, very few are well-defined, the rest is not, which makes us a lot of effort. We have projects to actually try to get data explaining this data so that cannot be solved easily by the suppliers.

Nabil Lahyani, Nokia (38:04):
I would like to compliment maybe with another analogy, the reason why we believe having the Lego block from the data perspective, which we call data product on the data mesh architecture. Because when you have to answer certain use cases or business issues, which data you have to send the agent to and for that design phase, as Ahmed said, it's crucial, it's not even important. It's crucial to have the domain knowledge, the architect who going to design the solution. I give you an example we were looking at it's it's not only for Nokia data product to give an answer through LLM about the video quality. It was a sport event. The operator was complaining that they had a lot of calls about people streaming and video. They will not get in the expected quality. And if you send an agent to go to the counters or the KPIs, like Amro saying there are KPIs that you would think, oh they're providing information about blocking or even the counter and video quality.

(39:10):
Now we realize that you have to go to the event session level correlating the different domain estimation before there are KPIs like time to play in video. There are KPIs like negative bit rate. If you are not a domain expert, I can go through the list. I think the list was huge. If you don't have an expert domain experts in, hey look Mr Agent or Mrs Agent, this is what you have to correlate. And by the way a lot of information behind it. If you have such amount of subscribers in an area like I don't know, like Allianz Arena for instance in Munich, in a football event to control the video quality of all those subscribers, you need huge amount of data. Guess what? To get the data from probes, from core, from RAN, you will just pump all the information. If someone from hyperscale will say, yeah, bring all the data please, but from a perspective from TCO perspective, they say it's very expensive. I need the domain expert who will tell me which is the exact information I have to filter out to answer this question of. So this is an example where we face in the reality why this domain knowledge is important.

Guy Daniels, TelecomTV (40:11):
Okay, thanks very much, David.

David Warnock, Blue Planet, a division of Ciena (40:12):
Yeah, I think your question also raises the problem that we have at the moment with silos. So we have the call team, we have the transport team, we have the RAN team, and when something goes wrong they all point the finger at each other. I think that's both a challenge, but it's also an opportunity and I think that agen AI working across all of those domains, eventually once we get there, once we evolve to that is a great opportunity to break down those silos.

Guy Daniels, TelecomTV (40:37):
Yeah, great. Thanks David. Okay, well thank you very much for that. We have a question I think. Oh yeah, we can squeeze in a quick one.

Andrew Collinson, Connective Insight (40:45):
Thank you. Forgive me for asking this question but Dean isn't here and I feel I have to, if I was an investor, a telco investor and you were the board of the telco and you told me that 80% of proof of concepts fail to progress and let's say you are the board of a vendor or the board of a telco as an investor, I would be starting to think why are you spending 80% of this time on this activity? Why is so much effort going into something which is demonstrably not working? Because if you listen to Danielle, when she was talking about it, she said everything's got a target and we're testing it within four weeks. When you listened to George Glass earlier, he was saying Gentech AI is great, and Philippe was saying great and powerful, but it's exposing a lot of danger. Think 20 times before you get into it, why is there so much proof of concept? Why not more proof of value? Why not more integrated clear targets? Why emperor's new clothes question.

Guy Daniels, TelecomTV (41:53):
Let's ask the question.

Andrew Collinson, Connective Insight (41:54):
I don't mean to put you guys on this spot, but I think it's a very important question,

Guy Daniels, TelecomTV (41:57):
Right, we're asking who wants to go first? Well I can help you if you want go on then. Ahmed, you go.

Ahmed Hafez, Deutsche Telekom (42:03):
It wasn't my statement by the way. So it wasn't a big statement, but I agree with it. It happens. It's documented, but no, no, it's document. I'm just joking. I was joking. So that's not attributed to us, but there are reasons. If your target is to have a proof of concept from the beginning, then there is a likelihood that you fail because you haven't really looked into problem statement, the business owners and you haven't thought it through. If I'm successful, what would I do? And usually, and that's why we try to change the mentality there is say no objective is to implement it. The proof of concepts is just your path to prove it before you go implementation like testing, but you do not go. And that's not an objective because if I do 1000 proof of concepts, this is zero value on the business I need to implement.

(42:55):
And we have found out that just changing that perspective flips the coin. So 80% is success, not 80% is a failure because that's not objective. So now the teams has to think, okay, so what is the problem I'm solving? What's the business value? What is that? Can I implement who I'm implement with? Where are the integration points? So they go through these all problems before they start. They don't start and go to the lab and then see maybe good or coincident will happen and by magic we implement. No, they go the other way around. So this is how we try to solve the problem of proof of concepts not seeing the light of the day. But yeah, i'll leave...

Nabil Lahyani, Nokia (43:35):
I really love the question and for being bold, thanks for that. I appreciate it. Let me challenge you something and I love your PoC zero concept. I will adopt it. Okay, for your information, would you buy something from me without PoC?

Ahmed Hafez, Deutsche Telekom (43:47):
Well, no, but actually you have the answer, but I would not even start to discuss with you unless I have a plan to implement it and I stop a lot of suppliers from coming to me telling me proof of concept. If I do not have an assurance that from Nokia what you offer me, okay, so I have a problem here.

Guy Daniels, TelecomTV (44:05):
Yeah, just use that.

Ahmed Hafez, Deutsche Telekom (44:06):
So for example, you come to me, sir, I'm going to sell you an anomaly detection solution and I do not have a plan to implement it. I will not make a proof of concept with you

Nabil Lahyani, Nokia (44:17):
Implemented. You mean commercially the problem.

Ahmed Hafez, Deutsche Telekom (44:19):
Commercially launch it. So basically you're right, but it's on step in the past testing. So I need to make sure that you're giving me something of value. But I tell you, and I'm really honest on that, we stop a lot of proof of concepts and we don't even go through them because we do not see the outlook for implementation. So I save the suppliers time and our time say, guys, I don't see that we are going to do something so better not, and I block a lot of them. I mean there could suppliers here, they can witness that. So yes, we don't.

Guy Daniels, TelecomTV (44:48):
It's great. Sorry.

Nabil Lahyani, Nokia (44:49):
No, I think Andrew, I think he said it sometimes it looks very easy, very nice on the slide where then you want to execute it and he say, okay, give me a PoC halfway. We even don't get the data security, whatever. And then it failed. But it's like a pre requirement. I think if we don't have clear understanding what we are after, sometimes it's like a fashion business. Like this year, last year it was about ai. The year after, it was about gen ai next year, about agent ai. How many of you were involved in rp, robotics, blockchain, self optimizing network. But it's like we are forced to bring AI and implement it. It sound otherwise would be a failing company. But then when we go to the customer, they said, Hey, to get the investment from shareholders, as you mentioned, I need to prove the value.

(45:47):
I need to go back and show them. I run A PoC and I tell you why I've been having this. And finally it's changing in energy. I tell you we have a very mature product, energy efficiency. It's shown the electricity bill is drop medicine. No, you have to prove it in my network. I have to go back and prove it. Then you run many, many PoCs without getting into the commercial. But I would like to emphasize in many cases there is a systemic issue. There is no roadmap like saying we have to agree on success criteria. Many times we start PoCs without even understand what we are looking for. But now if we have the success criteria, hey, if we are successful, you're having this impact in my operations financially, whatever's the KPI SLA, then you have this roadmap. There is a commitment to deploy. I think this is one of the reason why it has been failing as well.

Guy Daniels, TelecomTV (46:34):
Great. Fantastic. This is really good Nastasi.

Nastasi Karaiskos, Wind River (46:37):
I think this is a great question, Andrew. For me, I think one, it shows the desire in telecoms to innovate. So the percentage, whatever it might be, 50, 60, 80%, that's not a problem, right? Because we want to innovate, we want to constantly improve some of the PoCs. There may be one that's failed that had certain, an exit criteria did meet it, but the parallel one did. So we're continually improving. We're continually innovating and improving the success rate of those PoCs. Yes, we need to do more because this is how we're learning, this is how we're innovating.

David Warnock, Blue Planet, a division of Ciena (47:10):
But sometimes I've been involved with quite a few PoCs, you're already 5% figure. Sounds about right to me. You meet all the success criteria, but it still doesn't go anywhere. So it is a great question to raise that. And so what I insist on now when we get involved with the PoC is that there's a path to revenue us as an OSS vendor. And so we don't just need to be confident that we can meet the success criteria. We need to understand that there's a real business problem behind that that we're going to solve. That's at the end of the day, going to either save money for the customer or improve their chances of revenue with it.

Guy Daniels, TelecomTV (47:47):
Thanks very much, David. Yeah, please.

Ahmed Hafez, Deutsche Telekom (47:49):
I want to add quick one. So there are exceptions when it comes to r and d, just to give the complete picture r and d and research for long term. They are allowed to do proof of concept to proof technologies, but these are very limited and very controlled. So it's not in the scope here, but just to make sure that the answer is complete.

Guy Daniels, TelecomTV (48:06):
Good point. Great. Thanks so much for the question. We are pretty much out of time for this panel out of time for day one. We're in overtime, but before we go, because you're here next to me on the stage, we spoke briefly at the beginning of the day and we look back, I got my years slightly wrong, but we look back at the origin of AI native and what we need to do and we've been talking all day about what we need to do to move it towards reality. What's your quick thoughts at the end of a day of presentations? You've been sat here for the day. Are we talking about the right things? Are we moving things forward as fast and as safely as you think we should?

Ahmed Hafez, Deutsche Telekom (48:44):
Yeah. If I look back two years when we started discussions of AI on AI in this forum, I see we have really gone long way through and I also, when I hear the developments and the thoughts and the maturity of thoughts at least, and thinking and directions from all presenters here and from people around. The people I talk to, we are actually going in the right direction. And the good thing is that we are also sharing with each others what we see and what value we see and how can we approach challenges. And it's not a problem, by the way, to have skepticism around ai. This skepticism also helped us to adjust and tune and move around the problems and understand them. So I really see that we're going well. And I think also this forum is also playing a good role there. So thanks.

Guy Daniels, TelecomTV (49:26):
Yeah. Fantastic. Fantastic. That's it for today. A huge round of applause, please for all our panelists. Thank you very much.