Gaurav Jain, Rakuten Symphony (00:09):
Hi, I'm Gaurav. I am VP for Data and AI Products in Rakuten Symphony. Today we are talking about a real journey that we have gone through in telecom with our partner or our parent company, which is Rakuten Mobile. So what we have done there is that we tried to make autonomous Level 4 things on our NOC side. But when we went to that level, meaning autonomous Level 4 is something which is technically a journey which we have to follow, we completed it in a way that there is a process, there is software, everything going on. But if I get into the details of it, there are two things which are causing a lot of apprehension in our own selves. One is token consumption. So when we are doing this, we saw that we are very much mindful about what we are consuming.

(01:03):
So there is a lot of tokens that we are consuming. And as Naresh said, the telecom models versus the open models that we have, whether it is SaaS models or PaaS models, they are only up to 80%. So we are looking for something—we don't want to reinvent the wheel, we don't want to do something which is out of the box. What we want to do is partner with everybody who is doing this and try to bring some value to our business. So that's what we did in our case. So we created a model zoo kind of thing where we keep all the models which are making sense for us, where to use, what to use. That's the framework that we have written by ourselves.

(01:33):
We'll talk about it in a few slides, but what exactly we have done is to minimise our token consumption, plus the sovereignty that we really want, we kept it as one box. What we have done, we have taken according to the use cases, all that, we trained the models up to a limit where they make sense to us, they can give us what we really want from that model instead of giving us an 80% accurate answer. On the other side of it, which is more realistic, is: are we really solving the problem? So these technologies, AI and everything, are good, but is it making sense for me to make me autonomous Level 4, where the core answer to it lies in autonomous Level 3? Am I really solving my problem, which is I know my RCAs, I know how to solve it and what to solve, and with what accuracies? So this is a big thing, and these two things together are what I need to make my autonomous Level 4. So I'll go to the next slide with this.

(02:54):
So I got stuck with a lot of tooling, a lot of gravity of the data, then the sovereignty that I cannot share this outside. So such models are somewhere going out of scope either, or if I am consuming it, I have to anonymise a lot of things. And the final thing is with all this, I am not making an operational closure. Why am I not making an operational closure? Because it's a probabilistic solution in all cases, because I have not trained it. Somebody else trained it, I'm consuming it, and every time I have to doubt myself, is my solution actually going to solve my problem? Whereas up to Level 3 is what is making a remediation problem for me. So Level 4 is fine—meaning if I know that I am doing right, I just have to go to Level 4 and execute it. Possibly in future, at Level 5, I can give that level up and then things are automatically solving, erupting, closing.

(03:59):
That's the final thing that we are looking for. But even at Level 4, when we are doing self-healing, how confident are we, and how do we make ourselves confident? That is at Level 3. So we have to be so sure about it, that at Level 3 we are doing closed-loop automation—whatever remediation we are doing, we should be sure before executing it. So with that, what we have done is we have created one single platform, because when we are doing it with multiple platforms, multiple things coming together, there are latencies involved, and every new latency is making me distant from my real-time problem solution and the real thing that I'm trying to solve.

(04:39):
So what we've done: we keep data in one single platform and then do all analytics before doing any AI—the basic analytics, the type of AI-nisation that I need for my own problems to solve. We have done that, and eventually we created the AI-nisation thing. So that is also in three parts. First is governance should be right, because telcos are something which are very specific with guidelines. So we kept that as a prime thing, and then we did all those agentic workflows, and eventually we maintained it for a long time. And every time when we are running it, we are very much conscious that we are making the right decision. How we do it is the question.

(05:20):
So the first thing, the base layer, is sovereignty. As you see, we need an air-gapped solution because we cannot give this data in any shape or form to the internet in any scenario. So we kept an air-gapped platform. The second is data and memory. So every time you solve a problem, you need that memory to be maintained, because a lot of energy, a lot of prospects and aspects should be taken care of. Even when human-in-the-loop is coming in, then what exactly have they solved, why did they solve it in that way? That learning is actually adding every plus-plus to my knowledge base. So I have to keep that together.

(05:59):
The third one is I don't want to rely on one single model every time. There is no silver bullet in the world, at least in the operational world. You have to learn that every time when you're solving it. So I'll talk about this in more detail. So we kept every option open, whether it is a machine learning model, a small language model or a large language model. So that is what we have done in the orchestration layer. We try to create acyclic graphs. We know what our tollgates are, and at every tollgate we ask that question to ourselves, and that's a framework that we have written ground up because that is where we need our confidence. The last layer is the agent plane, where every agent is unique. There are multiple different types of agents which are needed at multiple stages of the work. So that agent, because I am sure at my orchestration layer, we just went ahead and executed it. So that's what we have done.

(06:57):
Now coming to it: what model, which model, when we have used it. So we try to—it's not a generalised statement, but what we try to do is solve it in a very basic way. All telemetry things somewhere are time series. So we try to make it very clear that we are going to solve it through simple machine learning models, because classical machine learning models can better solve it, as they know the seasonality and there are very few aspects that we have to bring in. So we should not invest—because we are very mindful about our compute layer. We don't want to exhaust our GPUs, CPUs on something which can be solved very linearly. So we kept it that way.

(07:42):
The other part is there are very specific problems which need an SME to solve, and no large language model should be brought into it, because then it advances the complexity which is not needed at all. It's a very specific conceptual job which should be done by some trained agent. So with Hugging Face, every problem has a solution; some model here or there is going to solve it. So what we have done is we took that particular solution. We are not reinventing the wheel either in the case of large models or small models, or even from the perspective of classical ML sometimes. So we took that particular model, distilled it—no previous data knowledge that we need; we need the computational knowledge and the basic ingrained knowledge which the creator of that particular language model has kept it.

(08:32):
So we use that knowledge and we are giving it back to them sometimes. So that's the way we have done it. So very incidental problems, we try to solve through small language models, and all the complex problems, ambiguous things where we are not sure, we try to do that. That being said, the human brain is still not replaceable. So there are cases, at least in our network, which humans are still solving, and that's a problem for now, because there is legacy involved, there are systems involved, and there are sometimes human aspects involved. So there are still unsolved problems which our humans are still solving, but we are training this back into our model. So that's how our current models are running, and we are trying to achieve what we are achieving.

(09:28):
For doing that, the basic thing that we talked about is the orchestration layer. So here the point is we try to do it through DAGs. So we try to create a framework. The framework has been in production for quite a long time now, where at every step and every time... So in production-class use cases—if I talk about, let's say, a storage system running, there may be the same error coming or multiple errors coming. So every time the problem is unique, but possibly this unique problem happened 100 times previously. But when we do it, there are 99% chances the same problem is erupting. But you should be confident about it; for that, you have to analyse it every time.

(10:11):
That does not mean you are going to burn the same energy that you spent the first time. So what we do is try to create tollgates. If my tollgates are getting cleared, I feel that this is the same problem I'm solving. If at any tollgate I feel that this is not solved the way the previous problem was solved, then at that moment I try to initiate—and that's what an acyclic graph is all about. If it is all ticking down, I'm taking it as an automation journey. As soon as at any tollgate I feel, "No, this is not solving..." So first I have to go back. In the first session I saw that we were going for the backup path, but in our case we are not going for backup paths because we are in the middle of a journey. We cannot go back to the basics again. So we are in a problem, we have to solve this, and whatever mess we have created ourselves, we have to solve this as well.

(11:02):
So now we have two problems to solve: one is what we have done, and what we have to solve. We bring it together, start a journey, and then we are solving it. So the entire remediation plan we are trying to create from that particular path. So it's like automation with every tollgate. At that particular tollgate, we decide, "No, this is not now an automation, we have to go back to the basics and try to create an acyclic graph."

(11:25):
So that is all about that. Memory layer: as these systems need a memory layer, so there is a periodicity that we have kept in the memory. Every problem, at least every solution, and all knowledge is part of the knowledge base. Finally, in any scenario, it's an operational problem, so we have to solve it. That's what we have done.

(11:49):
Now, how we are solving it is the question. So how we are doing it is you will see multiple error codes coming from multiple parts of the system. As I said, I'm just taking an example that, okay, a storage system is running within hardware, with a network, and if it is a distributed one, like in our case, we have active-active instances where if you're writing at one place, it is simultaneously routing to two different places. And once all three places or at least two have been written, then only we will commit it in a way that yes, we have written it. So in this particular type of complexity where we have 100% assurance that we are going to write something and it will be there whatever happens... So in a geography like Japan, it is more probabilistic in nature. So that's how we have done it.

(12:33):
In this particular scenario, you don't know when you are getting five different error codes, five different systems failing, which one to prioritise, what to solve first, or what is the critical error versus all other four. So there is a way to solve it. How we have done it is through acyclic graphs. So we created a kind of internal graph system that we have created in which we find out, in a prioritised, weight-driven way, which to solve and which is going to solve in which format, and how much we have to parallelise it and what we have to do in a sequential manner. So that graph will get created, then there is a tollgate, and then according to the tollgate, we try to scope it and then start solving it.

(13:21):
While solving it, there is another thing: diagnose something which is not going to change my real world again—meaning I'm not managing a problem, plus I am trying to solve it in a way that I am solving it, plus in parallel I am not damaging my own system. So I have to be mindful enough that while solving one thing, I'm not creating another problem. So we did it together. So in parallel, when we are solving, we are also diagnosing that there is no new problem coming. Eventually we have to execute it in all scenarios.

(13:53):
Finally, so in our service model, we solve it. We say that it's solved, but until the time I verify it, I will not close it. So we kept two different things, and until the time it is not closing, any new problem is still a problem which is in the system. So my system is always taking that new acyclic graph, still taking the previous problem into consideration. Altogether, it is solved from that perspective. So that's how my entire network is running as of now.

(14:25):
We have done it for the entire NOC as of now up to L2 level, and L3 level is almost half of the things done. So that's what we are doing. From the deployment perspective, we are currently running it in Rakuten Mobile. We are trying to bring this to other partners also. So it's not a concept, it is more of a scaling thing, and we have solved it for almost 25 petabytes of data with all the network that we are running.

(14:53):
The last thing that we want to talk about is: does it matter? Yes, then there are three points which are very important. First is every time when we go into it, we solve Level 3 instead of Level 4, because if you solve Level 3, Level 4 is a solvable problem. Second is MTTR—meaning what is the cost of it? Because if you are burning a lot of tokens or you are doing a lot of stuff where every time, possibly, you will get a right answer, but you will get exhausted very quickly. So you have to know where to stop and what to do. The last point is: is my agentic not blocking me in any scenario? So I have to be very specific that I'll execute fast and every time I'm making the right decision. So that's all from our side. Anything that you feel, any questions that you have in mind, I'm here. I can answer you. Thank you very much.