STFC-RAL-CR03 R61: Reconstruct our data.
STFC-RAL-CR03 R61: And currently, I think most of us are really using machine learning on a daily basis. I'm currently writing a Grant proposal an Erc grant.
STFC-RAL-CR03 R61: And I'm using it every single day to do things like this. You know, I have a big text reference, and the the proposal needs references, and I ask it to give me the proper style reference. So I keep telling my supervisor he's currently director at Eth that
STFC-RAL-CR03 R61: you can use machine learning in your everyday life to be 1,000 times more productive. And I do pay 20 bucks per month to use Tpp for because it increases my productivity 100 fold.
STFC-RAL-CR03 R61: So we're all using machine learning to some extent when we're using our facial recognition on the phones. And the reason I'm bringing up Gbt, here is that whereas these models are great, they're also huge. So we know, is roughly 175 billion parameters, which is point 16, the neuron to have in your brain and the current. Gp, 4, that I'm paying
STFC-RAL-CR03 R61: 20 bucks for a month. It's roughly the same number of neurons as you have in your brain. So these models are getting really, really massive. The reason they're performing so well is because they're absolutely huge. So they can just remember a lot of stuff.
STFC-RAL-CR03 R61: And when you even want to train one of these models, you need massive supercomputers. So openai the company behind Chat, Vt. They have their own super computer. From Microsoft, with 300,000 CPU cores, 10,000 gpus, all connected via 5 400 GB per second links.
STFC-RAL-CR03 R61: And it's trained on 300 billion for so 25, the longer high stack of books training time
STFC-RAL-CR03 R61: at inference time. So when you ask Gp. Something, and it gives you reply. You wait a few seconds and you see what it gives you back. We don't really know where that's running, but if you have a model that has trillions of parameters, it needs to run on very powerful hardware. And the problem these days is that the models are out growing our hardware. So the size of these models grow way faster than the hardware we have to train them on and to actually run inference on.
STFC-RAL-CR03 R61: And so, for instance, for Gpt, it would use roughly, and if you have the full model it would use an order of magnitude more, and memory single. And even if we use the numerical position of all the numbers of the model, you need roughly 11 and vda very expensive, gpus to even fit the model and run it. So every time you ask, give me this reference. It needs 11, Gpus. To give you your answer.
STFC-RAL-CR03 R61: It is currently becoming a problem. So here you can see for a range of types of models. You have these transformers like Gbt in red. So this is okay, and computer vision models that you see here in blue.
STFC-RAL-CR03 R61: These are growing by a massive amount per year. So by a factor of 240 every 2 years, whereas our Gpus are only growing by a factor of 2 each year. So there's 2 different trends. There's 2 different trends happening in machine learning. Now.
STFC-RAL-CR03 R61: one of them is big models that can do crazy things like Dolly and GET. The second thing is what we call efficient machine learning. So making the models faster and smaller, so that your user experience is better every time you use your phone to recognize a song or recognize your face. You can't. Don't wait 10 s. Then you're gonna buy another phone. This is called efficient machine learning.
STFC-RAL-CR03 R61: Right? So what we see with these models is that they get better, the more compute you throw at it when you train them. So the loss that means it goes down. The model is better. As you increase the amount of compute you use to train the model. The performance gets better. If you increase the data set size, the performance gets better. If you increase the number of parameters in your model, the performance gets better.
STFC-RAL-CR03 R61: and we have no idea where this will stop. The idea is you can have a brilliant artificial general intelligence model simply by scaling it up and simply by increasing the data set size. So this is the current trend we're seeing.
STFC-RAL-CR03 R61: So you have a problem. You know, our models just get better with the hardware we have.
STFC-RAL-CR03 R61: and but the hardware is growing with the amount of computing provided for the hardware isn't going fast enough.
STFC-RAL-CR03 R61: So am I saying all this? What is this have to do with the Lmc. Well, we just saw that every time you ask Gpc, something, it's running on €1110,000 Gpus, and you wait a few seconds and give you an answer where I work and what I'm working on. You have to run everything on one tiny little chip on an Fpga, and you have to do it in a billion at the second, really, really fast.
STFC-RAL-CR03 R61: So that means
STFC-RAL-CR03 R61: hmm, large language processing, they need smaller and faster models. And we are currently pioneering the work in designing smaller, faster models are power efficient.
STFC-RAL-CR03 R61: So this is becoming more and more relevant as we have self-driving cars that needs to do inference computer vision inside of the car very, very fast. Your smart watches, and also need to run inference on the device and that Cms we have to do in on the device which happens to be a detector.
STFC-RAL-CR03 R61: So efficient AI is the main topic of my talk today is, how can we do brilliant things very, very efficiently, and within nanoseconds this is relevant for the large Haveron collider, but it's also relevant. For instance, in astro particle physics. So the Ruben observatory. They look at electromagnetic waves from outer space. They have 1 million of a third generating 20 TB of data per night. And I need to process that data fast.
STFC-RAL-CR03 R61: Secondly, you have Ligo gravitational wave experiments. They need to do matching of the weights patterns that they observe with patterns they've already saved, and they want to be able to detect a merger of of stars before it happens. They also need to do machine learning very, very fast.
STFC-RAL-CR03 R61: and then you have shrank of telescope arrays that generates a lot of data, 72 terabits per second of data and need to do real time event filtering.
STFC-RAL-CR03 R61: And then and where I work. So here you can see the amount of data that goes through the system of various companies or experiments.
STFC-RAL-CR03 R61: Okay? And so here you see the amount of data that's going through a given system versus the time you have to process that data. And if you look at the large habit provider. Specifically the hardware event filter system. We have way, less time have to process way, more data than, for instance, or the Google Cloud. So we generate. We have to process an extreme amount of data and a very short amount of time.
STFC-RAL-CR03 R61: And so where's all this coming from? So most of you here in this room probably know the Lc. Very, very well. You can see the ring and 4 different experiments located around the ring, but this is where I work the same experiment.
STFC-RAL-CR03 R61: In here we collide bunch of protons roughly 11,000 times per second, and the Lxc is at any given point during data taking filled with roughly 2,500 bunches of these protons that we bring to collide in the center of the particle detectors like.
STFC-RAL-CR03 R61: So what happens is that every time these bunch of protons collide and they collide, the energy of the incoming protons is converted to new output particles that shower to the detector, and that showering produces a massive amount of data. It's just hits inside the detector. So we have billions of collisions per second, producing petabytes of data every single second. So for comparison, that's an order of magnitude more than what's uploaded to you
STFC-RAL-CR03 R61: 2 in 24 h that we produce every single second, it's a lot of data.
STFC-RAL-CR03 R61: Now, ideally, we would like to read out all of that data and look at the collisions and analyze them. But what does that mean that means that we need chips to acquire that data inside that zoom, we need cables for powering to get the data out.
STFC-RAL-CR03 R61: So if we wanted to get all of that data out of the detector in this in this frequency. So we have portions every 25 s. We would need a lot of that material. We don't have active material as partners sucks. Basically. So you may be trade off. We either use less material, and we have less of the data, or we have.
STFC-RAL-CR03 R61: So for that reason, we actually don't read out all of the data, and luckily we can afford not to read out all of that data. So if you don't look at all of the lines here, this thing that this dash line is the energy we currently have the Lvc, this is the probability of producing anything. Yeah, let's see, the probability of producing something interesting is only one in a billion or something like that. So either, even if we could save all of these collisions.
STFC-RAL-CR03 R61: they wouldn't even be very useful or interesting to us. We can afford to get rid of a lot of them. So that's how our experiments actually work. So in Atlas and Cms, what we do is that we reduce the amount of data both in hardware and software, via 2 steps. So hardware stuff and a software set
STFC-RAL-CR03 R61: in Lxcb, they use a full software data reduction system. And then at least, you can afford to actually read out continuously because we have less data continuously in software.
STFC-RAL-CR03 R61: And since that's what I have on. I'm gonna talk about how these 2 step rate production systems work. So I'm gonna focus on Cns, but it generalizes to what's being done in Atlas.
STFC-RAL-CR03 R61: Okay, so what are we doing? Cms, a collision happens. And we keep all of that data from the collision inside of the detector. So that data never leaves the detector. It's locked inside pipelines inside the detector.
STFC-RAL-CR03 R61: Then we take the small subset of that data that we have decided upon it in advance to say, okay, we can use this amount of data to figure out. And if this collision is interesting enough, that data we send to a radiation shielded tavern located right next to the detector itself, right next to the detector, because we can't afford to transmit data up and out because the latency is so small.
STFC-RAL-CR03 R61: So here we have 1,000 pgas. I'll talk about what they are in a minute. That has the job of deciding which that's to keep and which to reject within a few microseconds. So currently, it's 4 microseconds. So that's very little time. And that time comes from the buffering we have inside the
STFC-RAL-CR03 R61: so the detector can only keep data for 4 microseconds. If you're not done with your decision by then you're gonna lose that data. So you have 4 microseconds to make your decisions. you make a decision, and you have to reject over 98% of the collision events at this stage.
STFC-RAL-CR03 R61: If your decision is 0 event is not interesting. You discard that collision, you'll never see it again. You're not interested in it, and you'll never have access to to that information again.
STFC-RAL-CR03 R61: If it is one. You're interested in looking closer at this collision. Maybe there's a heat something interesting. Then we send a signal back to the detector. And we say, Hey, this is interesting. I wanna look closer at your data.
STFC-RAL-CR03 R61: Only at this point does the data actually leave the detector. So it never left underground. It never left the detector. And only at this one does it actually go to surface.
STFC-RAL-CR03 R61: So here we reduce the amount of data by 99%. And we send the data to the surface. What's called a high level trigger. High level trigger still has too much data. Now, we have less back pressure. So here we can run more complicated algorithms in software and attempt to further reduce the data rate because the data rate is still too high. And we reject a further 99% of that 99 that one
STFC-RAL-CR03 R61: only at an accept from the high level trigger, do we send the data out for offline processing. So currently, that's roughly 0 point 0 0 25% of the collision events remaining. Everything else has been discarded either in the hardware system
STFC-RAL-CR03 R61: or in the software system. So obviously, we have to be sure that when we do discovery, we want to find really rare processes. We have to make sure, we select the right data because it's very little data. So algorithms need to be really fast to get more data through and accurate to select the right data, otherwise potential signs of new visits.
STFC-RAL-CR03 R61: Now the Higgs boson is want things
STFC-RAL-CR03 R61: now. We know that there must be other new interesting physics out there. The standard model is an effective theory. It breaks down at certain energy scale. So there has to be more. And so far we haven't found anything. But what we're doing is looking through analysis.
STFC-RAL-CR03 R61: Now, if we wanna produce new physics, we saw that probability producing a Higgs is one in a billion producing new physics, the probability is one in a trill. So if you wanna measure these really rare processes, we need more data.
STFC-RAL-CR03 R61: and that more data will be provided by what's called the high luminosity. Lhc, so the produce one and a trillion. We need more collisions to see rare processes.
And the way we're gonna do that is by upgrading the other experiments to have a factor of 10 more in data, basically. And the way we're gonna do that
STFC-RAL-CR03 R61: is, by increasing the amount of protons, and each of these bunches is matched together, squeezing them tighter together so that there's higher probability that they overlap so more collisions, more data, a higher probability of finding potential in physics
site.
STFC-RAL-CR03 R61: And the problem is that this comes at a very rare fast.
STFC-RAL-CR03 R61: So at the current. Lhc, we have an average, probably 60 protons for each bunch, and crossing each other by with every single crossing. How do we not see Lhc will have up to 200 of these. So you can see that the event complexity going from here to here
STFC-RAL-CR03 R61: is huge, and we are going to have to analyze that data. And we're going to have to analyze that data in 12 microseconds. So very, very fast
STFC-RAL-CR03 R61: in addition, since we have these more complex data topologies. We're investing in more granular detectors. So Cms, we're buying this very expensive which will take us from 85,000 readout channels today to 6 million readout channels in the future. So we have more collisions, and we have more readout channels. So that means you're gonna need a lot of compute.
STFC-RAL-CR03 R61: We also know that our computing budget is basically flat. We can't buy as many Fpgas as we want. We can't buy as many gpus as we want. So if you look at the amount of computer that I need is sort of CPU as a month of the year for Cms and Atlas.
STFC-RAL-CR03 R61: this blue line, this band, is what we can actually afford. This blue line is that we keep the status quo of all of our algorithms. And these dash lines is only if you do, aggressive aren't need into new algorithms. So it's not a choice to invent better algorithms. We actually have.
STFC-RAL-CR03 R61: So that's why, we're looking at modern machine learning in order to become way faster, become better and to do more with the data that we have. And so we're really looking at more complex architectures and algorithms to deal with the increased data complexity.
STFC-RAL-CR03 R61: One of the things that would change the most is the level one system. These guys right here, which I know a lot of you here in this group work on like phone.
STFC-RAL-CR03 R61: And so we'll be new in the system. So we'll have the inclusion of chart particle tracks
STFC-RAL-CR03 R61: having short particle tracks, particle direction and energy, and it allows us to combine that information with calorimeter information from the new one chamber in something that we call particle flow. We have more brand new visitors. So we're including this, and we're including more granular. What does that mean? So I mean that way, more data is coming in to this Fpga farm.
STFC-RAL-CR03 R61: That is 5 of the total Internet traffic in a cavern, 100 meters underground 1,000 Fpgas. So this
STFC-RAL-CR03 R61: is a very complex and advanced system. The time we have a process, those 5% of the Internet traffic is 12 microseconds. So that's gonna be very, very challenging.
STFC-RAL-CR03 R61: So in order to cope with that, we are looking into doing machine learning on specialized hardware. That's either using machine learning inside the detector on tiny chips to read out basically better data.
STFC-RAL-CR03 R61: It is to improve the reconstruction and selection algorithms on these Fpga in the level one trigger, or it is to improve the reconstruction and selection efficiency on Gp on the surface.
STFC-RAL-CR03 R61: So I want to show you quickly what this level one trigger looks like. We're still here. So here you have the detector. You have these racks for that Pj's monitor on custom boards. And here you can see what it looks like. So it's just racks with the with boards. Basically.
STFC-RAL-CR03 R61: the system is kind of hierarchical. So you have, like 400 refugees doing a calorimetry, 200 refugees just doing tracking 100 for nuance. That information is combined in the one that called particle flow. Before you have this final decision after 12 microseconds.
STFC-RAL-CR03 R61: Now, that doesn't. That means you don't have 12 microseconds for any single algorithm. This guy has to be done after 5 microseconds. That means at most for any single algorithm. You have a few microseconds, usually. So you need extremely fast machine learning algorithm. When I unlock my phone like this. And I think everything is real time.
STFC-RAL-CR03 R61: That's millisecond latency. Maybe so, it's orders of magnitude slower.
STFC-RAL-CR03 R61: So that's why we are working for. And with companies like Siemens, for instance, we're really interested in the technology we are developing for level one trigger.
STFC-RAL-CR03 R61: And as you also work on here, I've been told, tracking is an extremely complex task that needs to be done within microseconds. That's also something we're very interested in using machine learning. For.
STFC-RAL-CR03 R61: So one thing I wanted to mention first, this, because the question that we often get is, why are we using Fpga's another launcher, again, should probably let some answer that. So in other experiments like Lhcb, they've swapped to completely software based way of reducing the rate they use gpus and cpus. Actually, now, they have a full Gpu reconstructions was 300. Gpus. Why aren't we doing that?
STFC-RAL-CR03 R61: Well, the problem is this, pipeline is the buffering inside the detectors. We can't afford to read out all the data, so we have to keep it inside the detector. The more buffering time we give, the more dead material we have inside the detector.
STFC-RAL-CR03 R61: So
STFC-RAL-CR03 R61: the main issue for us is that, Alexev, they've already read off the detector. It's not in the detector and pipelines anymore. So if they have no latency constraints, so our frontline buffers are strictly limited. We can't make them bigger, because then we have crappy detector and also the raw event rate of Cms raw. The event data size factor is 10 larger than Alexev, all of that level. One event. This is roughly the same order of magnitude, so it would fit on it. So it's mainly latency.
STFC-RAL-CR03 R61: So latency, latency latency. You can't do much on the Gpu and 4 microseconds or 12, and so on. Fpga, you can work on different part, different parts of the problem and different data simultaneously. So, for instance, what we're doing is that it's not like. This is one event that is working on just one event. It's one, and then it's another, and it can kind of work on 18 events simultaneously. So you could really pipeline this, we call it to work very and parallel.
STFC-RAL-CR03 R61: So so latency deterministic when my simulation after place and Rob tells me, okay, this is gonna be 3.4 nanoseconds. Then you can be pretty sure that it's not gonna be. You can have software based interrupts. And suddenly it's slow. And suddenly it's fast. And you really can't afford that in this system.
STFC-RAL-CR03 R61: Yeah, then it has a high bandwidth. As I said, we are going to process in a new system, 5% of the total internet track. So
STFC-RAL-CR03 R61: we need a large number.
STFC-RAL-CR03 R61: So we wanna put machine learning on these Fpgas. And we are physicists, we are, we? Many of you are engineers, that's I'm not. And I think, program in Fpga is really really hard. So usually, what you have to do. This is the easy way. Even if you write some C decorated with some constraints from the compiler like. I want this to be very parallel. I don't want this to be very parallel, etc.
STFC-RAL-CR03 R61: Then you pass this code to a compiler which depends on which vendor you're using. We are using. Zoom, you have these compilers here, and then you compile that into a hardware description language, another lower level abstraction before you have a firmware block that you can put on your Fpga. So someone has to do this part.
STFC-RAL-CR03 R61: And here's an example. yes, also in this part, if you're going to do inference of machine learning algorithm and 100 nanoseconds right in this part. That's that needs to be good. That's not gonna be easy. Peasy kind of non optimized. That needs to be highly optimized.
STFC-RAL-CR03 R61: So you can see an example code and a hardware description language, what it looks like if you write it out. So we kind of have this thought that like.
STFC-RAL-CR03 R61: if you are writing a machine learning algorithm, it's maybe a deep neural network with some weight multiplied by the input of the bias term is very, very generic. So you can optimize that. And many people can use it. And that started and our project and making a generic super fast implementation writing that C plus cost code for you to be highly optimized and really fast
STFC-RAL-CR03 R61: and give it to physicists. They can design algorithms for their subsystem.
STFC-RAL-CR03 R61: So that's what we did and design these libraries that can go from any kind of machine learning library type model translated, using our tools and get basically a firmware block that you can then integrate implement into your system as easy as possible. And then way it works is that you start by some model that you have to massage a lot to make it small enough.
STFC-RAL-CR03 R61: Once you have that model, you pass it to us that will take that model and convert it to our own internal representation of that model. Just some representation. You can work more easily with.
STFC-RAL-CR03 R61: And then we write that.
STFC-RAL-CR03 R61: and we can then emulate that code. We can check. How fast is it? How accurate is it exactly as it will be on the hardware in python, and then actually running the synthesis resulting in either co-processing kernel. You wanna use an accelerator to make your inferences faster, or custom designs like we're using in the trigger, or even a 6 tiny chips that you can put inside the detector.
STFC-RAL-CR03 R61: So here, I just wanted to show a really, really quick example. This is all the code you need to take a computer vision model and put it basically onto Fpga.
STFC-RAL-CR03 R61: The reason I put this picture is because
STFC-RAL-CR03 R61: this, all everything that's colored here, that means it's resources that's occupied. This is a tiny tiny model. And as soon as you want to put a model on the chip with no external memory, you're gonna blow up this thing instantly, because there is no space.
STFC-RAL-CR03 R61: So that means that we can't put a transformer Gpt on these chips because their ship is going to be too large.
STFC-RAL-CR03 R61: So ideally. we want this. But in reality we have this
STFC-RAL-CR03 R61: most big companies. They have an external memory where they store some of the model for us that doesn't work, because the time it takes for the compute to talk to the memory is too slow. We actually can't afford to store the model. Externally, everything needs to be on the tiny shift.
STFC-RAL-CR03 R61: And that's why we are able to reach the lowest latencies when we compare to other tools.
STFC-RAL-CR03 R61: So in order to be able to do that. We have been inventing engineering a lot of techniques and not inventing but pioneering methods at least
STFC-RAL-CR03 R61: to make this possible one is quantization so reducing numerical precision of networks. One is pruning, removing weights that don't really matter for the network paralyzation. Everything that can be done in parallel should be done in parallel, and also something called knowledge distillation, which I won't talk about today. But it's technique to train a big model and make small model learn what the big.
I wanna talk about the most powerful one first, which is this thing called
STFC-RAL-CR03 R61: so. This is a deep neural network. It has a stack of X layers, and everything is done in floating point 32 precision. So you can represent 4 billion numbers in a very, very large range. So you have this very, very granular sign, resolution of each weight. And also it can represent a very large dynamic range, so fine, large dynamic range.
STFC-RAL-CR03 R61: The problem is that we already know is that
STFC-RAL-CR03 R61: floating point 32 is very expensive. It takes up a lot of memory, and also, when you compute with these numbers, it takes a lot of time. So if you can even reduce this to floating point 16, and you can basically have the size of the model.
STFC-RAL-CR03 R61: And that's what the one you're using on your computer is already, and Fpga's are really flexible. You can reduce the numerical precision to basically whatever you want on the CPU or Gpu, you need a specific core that can handle that type of
STFC-RAL-CR03 R61: for instance, in 8.
STFC-RAL-CR03 R61: So what we wanna do is rather use something called fixed point decision where you have basically a fixed number digest before after the decimal point. So, for instance, for a bit with 8, you can have it with a port, you can have 2 to the 4 numbers in this range.
STFC-RAL-CR03 R61: So here you basically only have 60 numbers you can represent. And the range is really small, so less granular, smaller range, the range we don't care so much about usually in neural networks. The range is anyways pretty small. You're not gonna have to represent 1,000,385
STFC-RAL-CR03 R61: but the granularity gets worse. So we want to make the weights all the weight, the numbers in little buckets.
STFC-RAL-CR03 R61: If you do this naively. So this is just whatever algorithm. And here you can see the accuracy as you reduce that buckets bucket size. You have fewer buckets, basically, at some point, you might as well just throw a coin to coin. Toss algorithm isn't doing anything else because it has so little granularity.
STFC-RAL-CR03 R61: So what we instead did
STFC-RAL-CR03 R61: in a collaboration together with Google was that we decide to libraries that talk monthly together. One has the job of training this network to know which bucket the weights will go into.
STFC-RAL-CR03 R61: So when it's training, you tell the network, hey.
STFC-RAL-CR03 R61: on the Fpga. That number will actually only be in these buckets, so everything will be rounded into this buckets. So it learns the network perfectly to adapt the weights to the precision you will have on firmware. So it's kind of a code design between software and and firmware. So the network perfectly well knows what the wait number will be, and adapt accordingly for technicality and back propagation, we do everything in in floating point. Precision would be redo that loop.
STFC-RAL-CR03 R61: and most importantly, this should be easy. So everyone wants to design a machine learning algorithm for the level. One trigger or for the tranquil telescope array should be able to do so. So it should be cara slide. Just put a queue in front, and whatever precision you want as easy as possible will develop the tools you'll develop the application.
STFC-RAL-CR03 R61: Now, we also know that it's hard to choose which precision you're going to use. So are you going to use this bit with 4. Are you gonna use 6? Are you gonna use 8? Are you gonna use 42. So a lot of things to choose between. And some layers can really tolerate more quantization than others. Some can be binary, for instance.
STFC-RAL-CR03 R61: and we know that this is the case.
STFC-RAL-CR03 R61: So we want to have the best possible quantization for highest accuracy and the lowest resource consumption consumption. So we've also designed as a way that you can scan over all of the quantization parameters and choose the best possible model.
STFC-RAL-CR03 R61: So here you can see an example of such a scan. Here you have the size of the model itself here, and the accuracy, and in the end we want the model that is the smallest, with the highest accuracy, and you can pick and choose your model, and that is the best, and you put that on your chip.
STFC-RAL-CR03 R61: The effect of that you can see here. So before we saw how the accuracy would drop off at a certain precision in a certain bucket. That's this line. Using these tools we developed together with Google, you can go down to very, very small small buckets and still maintain the same accuracy. And currently, for the level one phase 2, almost all the machine learning algorithms correct me, are using per ml or conifer to develop these algorithms. And this one session where training?
STFC-RAL-CR03 R61: Okay, I need this kind of just for illustration. So if you're working on and on a chip. It is an A and p chick. You can do 200 times more operations. Once you quantize down to you, put numbers in buckets, basically.
STFC-RAL-CR03 R61: And then you can reduce the memory. So you can do more operations. And you can reduce the memory with print significantly.
STFC-RAL-CR03 R61: Okay, a second technique is what's called pruning. So pruning basically means that every time you train a network, a lot of the weights are redundant, you just make some big network. It's great. A lot of the weights aren't really doing much. So pruning just means that you remove some of them during the training, you see, which weights don't really matter so much, and you snip them away to get rid of them.
STFC-RAL-CR03 R61: The way it's done technically is that we just set those way to 0. So this is the way distribution before. And this is the way distribution gets removed. So that means this whole site of wait is just gone from. So you never have to run those operations. You can save a lot in terms of time and resources.
STFC-RAL-CR03 R61: and how that's done is usually iteratively so. We will train the model, we'll see which weights are very small and don't matter much. We click them away, and then we retrain. See which weights are small, click them away gradual way. So you some, then more than one video having removed 75% of the network. And we usually see you can remove up to usually 75% of all of the weights.
STFC-RAL-CR03 R61: The reason you can do that is because of something called the lottery ticket hypothesis. Lottery ticket hypothesis is based on the following, you have a big network that's most times over parameterized. So a lot of weights aren't really carrying any weight.
STFC-RAL-CR03 R61: And if you prune it in a smart way. Remove ways and matches in a smart way. You can get models that are better
STFC-RAL-CR03 R61: than the ones that are not actually so you can really find a lot of new tickets within your network. And that's something, no matter what you're doing. If you're doing machine learning, you should do that because you have faster, better networks.
STFC-RAL-CR03 R61: that are also smaller.
STFC-RAL-CR03 R61: Now, one more thing because everyone is in this type of brack neural networks and transformers and insane models. I want to give a shout out to trees. And so who's the decision? Trees are very powerful. And why are they very, very powerful for us because we have really powerful summary statistics.
STFC-RAL-CR03 R61: You can't make a summary statistic of an image. You have pixels. How are you gonna combine them? We can do invariant masses. You can do just rapidity. We have very good. You learn data. So in our field decision trees are still, I would say, one of the most powerful things you can do. And in the machine learning literature there's also a big new hype about these 3 based models, really outperforming deep learning.
STFC-RAL-CR03 R61: and the neural networks are, in fact, decision. So always start out with a decision tree that can be more robust, more stable, easier to train and great for our type of tabular data. Actually, yesterday, Monday, I was connected to this Iml workshop at Cern, where there was exactly a guy, one of the founders from psychic learn
STFC-RAL-CR03 R61: who said, Hey, decision! Trees are great. And I said, Yes, they're great. So this is really a trend that's changing, and we've done it for a long time at certain. So we should just go back to to decision trees. I'm no longer so excited about neural networks, and we support that. So we also have collaboration with Google, sir, and find an interface. Conifer was old. That's a library that they take exchange models. And
STFC-RAL-CR03 R61: yeah, well, Tmva, and translate them into firmware like per month. Now we also support this tensorflow decision for us, and what we see is that if you have a quantized neural network versus the Vdt. For a lot of tasks you get the same accuracy, but a fraction of the latency and resources so always start out with the Vdt. If you don't have pixelated information, that's always the best.
STFC-RAL-CR03 R61: Alright. Then, finally, and a little bit on applications. So we're back into this system.
STFC-RAL-CR03 R61: 12 microsecond latency processing type of the Internet traffic traffic. What we're seeing so far with the developments ongoing, we just have a machine learning app level one workshop at 30 December. Every single one of these thinkers represents the development of a machine learning algorithm, either deep neural network or the decision tree for any one of these subsystems
STFC-RAL-CR03 R61: if we count. If we count based on how much data is coming into that system, how many inferences will be doing 40 billion inferences per second. And if we include all of the projects, and that is same as all of the inferences that's done internally at Google and phone. So that's a lot mostly coming from the 25 s.
STFC-RAL-CR03 R61: So how are we currently using this? Or do we wanna use this? The Lhc, so one example is inside the detectors themselves. I talk about this calorimeter. It's a a resolution
STFC-RAL-CR03 R61: and even timing. And the idea here is that you can way better identify, and which type of articles generated by the shower as well as the.
STFC-RAL-CR03 R61: So here you can also see what timing is doing. You have no timing how all of these pilots and the things you're actually interested in, and like the photon and eject from this Vvip signal is in this big mess of pile up, whereas when we put the time window, you can really clean up the events
STFC-RAL-CR03 R61: so often great, we're gonna do great physics and highlights that I see. The problem is that we cannot read out all of these channels fast enough for the level once to trigger. So that means we have this great information, but we actually can't take advantage of it, which is a little bit of a shame.
STFC-RAL-CR03 R61: So what we're doing is trying to compress that data on the detector itself. So this is one of the the sensors for that system. You have as 6 mounted on these sensors that sends out the data, 40 megahertz and to the trigger to that one, and it also sends the data out at 750 kilohertz to pull me that right to the surface.
STFC-RAL-CR03 R61: You're in this high radiation environment. And the whole thing is pulled minus 30. So any algorithm that's running in there needs to be very, very low power. So, and you have 1.5 microseconds of latency to take that data and get it out and send it to the level one trigger.
STFC-RAL-CR03 R61: So how can we compress that data? So one of the things we're looking at doing is something called variational auto encoder, which is trained to give us some input, it will compress it to some bottleneck, a lower dimensional embedded space and then learn to decompress it again. So it learns the essential information, higher dimensional data. Basically.
STFC-RAL-CR03 R61: Now, what we wanna do is make an algorithm learn that process. Very, very well.
STFC-RAL-CR03 R61: then, on an Asic inside the detector, we compress the sensor data into this lower dimension embedded space. We send that lower, dimensional data, so fewer bits through the high speed optical links into another one.
STFC-RAL-CR03 R61: and there you can do whatever you want. You can either reconstruct the data again, or we just upgrade directly on that information, so you can really take advantage of the granularity of it.
STFC-RAL-CR03 R61: So this runs at roughly 100 milliwatts. And what's kind of interesting is that sensory inside the detector? You can have? What's it called bit flips from the radiation. So suddenly, the network basically changed. So all of the weights and biases are duplicated in case you have 3 copies of each, just to make sure that nothing can go wrong.
STFC-RAL-CR03 R61: Alright. So what we want do we want to do in the level? One triggers, one of the big challenges will do that. Someone in this room is working for face to tracking
STFC-RAL-CR03 R61: you.
STFC-RAL-CR03 R61: Well, I did so. So one of the things that's been kind of community challenge for us
STFC-RAL-CR03 R61: is to see how we can do charge particle tracking, using machine learning. So you have thousands of hits coming from hundreds of tracks at 40. Megahertz, right? You have 5 s basically to process that.
STFC-RAL-CR03 R61: And so a lot of our. He is going into using these graph neural networks from fast charge article tracks. We have some demonstration that you can reach quite a good accuracy of this from classical methods, and you can also do it relatively fast.
STFC-RAL-CR03 R61: And so here you can see for one graph neural network, either optimized for a high tool meaning a low latency. So it's exactly the same model. You can optimize it to have a low latency. But then you're using the full Fpga.
STFC-RAL-CR03 R61: or you can optimize it to be a little bit slower. So this is a trade-off you always have to do with these, either high throughput or optimize for resources.
STFC-RAL-CR03 R61: We also, since we have the chart particle tracks we'll have particle flow. We will have particles in the level, one trigger for the very first time, and these particles we can cluster into jets. And these jets you can operate on. Someone's also working on checks.
STFC-RAL-CR03 R61: I think maybe not here.
STFC-RAL-CR03 R61: So we wanna try to tag these jets in the level, one trigger. And here we're trying to figure out which algorithms are the best. So either we treat all these particles inside the jets set. So you have. These particles are on order for mutation invariant, or their graph. So they have some connection. And we want to use that information.
STFC-RAL-CR03 R61: So we've currently been studying is which sort of networks are the most powerful to do, jet.
STFC-RAL-CR03 R61: And since we don't have so many particles inside each jet, we wanna, we also know that how many input particles you have depends on the accuracy. So here you can see this is more accurate models. This is a classical p, 4, world deep neural network.
STFC-RAL-CR03 R61: This is a set based algorithms. You treat the particles as a set. And that's this yellow line. And this blue one is a graph based network. So it seems about all of the particles and how they talk to each other. What we see is that if you have very few particles. Project doesn't matter if you use a simple network or if you use one of these more complex ones, once you have a lot of jet constituents, a lot of particles inside the jets.
STFC-RAL-CR03 R61: You need networks that are permutation invariant, for instance, so that you can reorder the articles inside the deaf, and the answer doesn't matter. Then you gain a lot by using one of these more and set-based or graph based one over F. Pr. One over false, positive rate. So it's kind of a tagging. Yeah.
STFC-RAL-CR03 R61: I think that variable.
STFC-RAL-CR03 R61: Okay, one thing we're also doing is doing things completely different. This we're not just trying to improve our current algorithms. We're trying to do new things. So I worked for Vsm searches for 5 years or something didn't start or anything, and we've been excluding a lot of this new physics base all of these new as a search for new physics.
STFC-RAL-CR03 R61: But there is still a lot of space for exploration. So, for instance, region up here, it's very heavy and large mass scales. We can't explore. Essentially, this one is uncovered. But there's also these blobs here, where you have this very low coupling or trigger limited searches, where you just don't have access to them, basically based
STFC-RAL-CR03 R61: due to this level on trigger. And you have signatures that we haven't even thought about. We wanna use machine learning to start probing. At least, he said, here, that are automatically accessible.
STFC-RAL-CR03 R61: We want to format them. Not so Lhc, and on 3 explore the full capabilities of the Lhc. And be a bit more generic. We can start probing all of these at once.
STFC-RAL-CR03 R61: So scenario one is that there is some new physics signature we haven't thought of, and so we don't really trigger on it. We just use lose it already at the trigger level scenario 2 is that even for signatures we're already looking for, some regions are simply out of reach, basically because of the level one trigger. So how can we deal with those?
STFC-RAL-CR03 R61: So the current limitations that we have is that we have to get rid of the rate, and usually we do that by cutting on. For instance, the jet transfers momentum or or making faster scenarios, and say.
STFC-RAL-CR03 R61: everything above this cuff is interesting. Everything you'll always find interesting. But obviously you can imagine that there's something interesting where we are not looking. So is there a smart way to select.
STFC-RAL-CR03 R61: So we wanna do is rather look at the data itself rather than defining the signal hypothesis of priority. Which is what we're doing here, we're saying, save on this year, we're biased.
STFC-RAL-CR03 R61: We wanna look at the data and try to classify the collisions as a normal data or anonymous data. And in the machine learning literature, we know how to do this.
STFC-RAL-CR03 R61: It's using something called again off way coders that we talked about before. So the idea is that you again train an algorithm to compress, then you compress some information.
STFC-RAL-CR03 R61: Now, the algorithm is trained on normal data. So this is often used for financial fraud, detection. It's trained directly on all of the transactions, like normal transactions or or or
STFC-RAL-CR03 R61: fraudulent transactions, but mostly on normal transactions. Now. this reconstruction, if it's normal data that it's seen a lot during training, we'll do this perfectly. And the difference between the output and the input is a good metric of performance.
STFC-RAL-CR03 R61: If you have an outlier in the data, there's not enough information here to memorize everything.
STFC-RAL-CR03 R61: If the data outfire comes through, it won't be able to do this compress decompressing very well. And like I said, this is something you see in financial product section. You're trained on those products as actors as well. But the network don't have the capacity to learn. So the difference between the output and the input is basically a degree of abnormality. If this is really off with respect what came in. You have something anonymous. So this is your degree of anomalies.
STFC-RAL-CR03 R61: So we're currently working on now for Cms and run 3 is an event selection algorithm that keeps everything that is deemed abnormal by this algorithm and reject everything that is normal. And that's so within 50 nanoseconds. And so we're hoping to deploy this for data. Taking this year.
STFC-RAL-CR03 R61: Here you can see an event display of one of these events team anomalous in the Cms experiment. So a lot of a lot of soft, soft jets, a lot of activity.
STFC-RAL-CR03 R61: And here you can see a picture of me in the level one triggers, we have something really cool and Cms.
STFC-RAL-CR03 R61: so we have this final global trigger that makes the final selection right? It gives Sarah and want to get some input. And it says, yes or no.
STFC-RAL-CR03 R61: We have a new copy of that system. So it's receiving exactly the same thing. But they are, but it can trigger Cms. They can say that all the data. It just gets all the information, and you can kind of monitor and what it does. So we've also been integrated in that system for data ticking last year, and to validate the performance and rates, etc. So that's a really cool thing. If you wanna do machine learning algorithms, you can actually type
them in real fence type.
STFC-RAL-CR03 R61: Another thing and that will become more and more relevant is that we have very changing environments in our pictures. So
STFC-RAL-CR03 R61: in the beginning of data, taking, we have transparent sensors. For instance, at the end of data thinking we have opaque sensors. Can our algorithms cope with that? So you see your radiation damage on a silicon?
STFC-RAL-CR03 R61: And there are many reason these things can happen. So how can our algorithms be robust to these things?
STFC-RAL-CR03 R61: So one of the things we're looking at something called continual learning, where you have a foundation model. In this case you have some decision tree that you train and you trust, and it's great, and then you can continually adapt it. So you train it a little bit of new incoming data adaptive, and then you in this case add a second tree on top to adapt to changing and conditions you want. And that's called continual learning as something we're exploring for most of the algorithms
STFC-RAL-CR03 R61: have running online.
STFC-RAL-CR03 R61: Okay, just one brief mention here mainly for don't see. I wanted to quickly mention the high level triggers. So this is also inference on specialized hardware, so new, and Cms. We used to have only cpus and Cms. 30,000 cpus. Now we reduce the amount of cpus, and we've equipped each node with Gpus for password processing.
STFC-RAL-CR03 R61: And so here we have more time, but it's still on the order of self-driving car, so it's still not to be kind of discarded.
STFC-RAL-CR03 R61: So let me relatively fast.
STFC-RAL-CR03 R61: So we have these CPU notes. And before we get data from the level one and the CPU would do basically everything until you send it out.
STFC-RAL-CR03 R61: Now, we instead equipped each of these notes with a Gpu, so we could offload by a factor of almost 2 and reduce the power. So now you
STFC-RAL-CR03 R61: take the data. We popped some of them over here that some of this compute heavy work that's faster and give you. We send it back to other stuff, and then through the decision.
STFC-RAL-CR03 R61: Now, one of the problem is is again, that this data transfer is often the bottleneck in terms of time. So you are maybe faster on anything here, but then sending the data back on the lot of your time. But that's always you always want to do everything on a single device for
STFC-RAL-CR03 R61: and then you have to write the cuda kernel for that. The algorithm we want to run which is not straightforward.
STFC-RAL-CR03 R61: One thing that's cool is that neural networks are live very natively on a Gp. So what we've been looking at is how we can reconstruction fully on a Gpu. So going from this classical way of combining particles, and rather using graph neural networks to do the same. And we see we can get almost better or the same performance as classical particles. So if you can't do better by do it well, that's because of scaling. So here we can see how
STFC-RAL-CR03 R61: and the particle flow algorithm scales in terms of time. It takes you to process each of them versus the number of particles. And you see, this is definitely not linear, whereas these networks scale linearly with the number of images
STFC-RAL-CR03 R61: so and more efficient.
STFC-RAL-CR03 R61: Great
STFC-RAL-CR03 R61: kind of final work work on hardware. And Yvonne just told me on this caliber project that you have here in the Uk. We're also really interested in looking at New AI hardware. All the big companies are bringing out their own AI engines. We're especially interested in this process called Versaill AI.
STFC-RAL-CR03 R61: So these are processes that you can program C plus and have these special AI processes that can talk to the Fpga, and they can move back and forth between that PGA and the AI engines. We're going to looking at that for real time and tracking.
STFC-RAL-CR03 R61: One more thing where real time machine learning and Fpga's is becoming relevant is also for not very latency problems. So
STFC-RAL-CR03 R61: current 2 and Cms, we use 60% of all of our compute on reconstruction basically running reconstruction algorithms. It takes a lot of our budget simulation and reconstruction.
STFC-RAL-CR03 R61: So if we could use and machine learning for this reconstruction, which you see are already doing for particle flow, and we can run them on accelerators that would really take the load off in terms of compute for us.
STFC-RAL-CR03 R61: So one thing we're looking at is to offload some of the inferences, some of the machine learning work that's currently being done at tier 0. Offer them on Fpga co-processors, the Fpga work as an accelerator, and to make you more efficient. And this is something we've been able to do with pretty good success so far.
STFC-RAL-CR03 R61: we're not doing it by default. But it's another cool thing, is Lhcb, so they have this thing of being fully gpu based, but they still have 200 fpgas for vap.
STFC-RAL-CR03 R61: So they have 200 fpgas for Dq. That's running when they're stable beams. Then they're on. They're connected to the computing grid, and everything. When the beam is off is off. They're just standing around. They're not being used to do is do the reconstruction on these fpgas, when there is no beam to really utilize all of the hardware that you have. That obviously means you have to
STFC-RAL-CR03 R61: reprogram those Fpgas between Bill. So it's it's a lot of work. But I think the idea of videos utilizing what you have makes makes a lot of sense
STFC-RAL-CR03 R61: great. So I think that's it. I have one more example. I wanted to show this because I think he's cute, and we also work on triggering and other experiments. You have this very cool neutrino detectors on the seabed in the Mediterranean, where you try to detect high energy neutrinos from outer space.
STFC-RAL-CR03 R61: They apparently have a lot of background. And one of their biggest backgrounds is bioluminescence from fish.
STFC-RAL-CR03 R61: So they're really frustrated because they have this queue in like a box on the sea bed. And they're like a lot of power. So that's not great, as we're also working with them on having an Fpga and running real time computer vision on the sea bed. Send that data to sure.
STFC-RAL-CR03 R61: And we also do a lot of work outside of physics. Self-driving car brain implants quantum control fusion do, and unfortunately, not anymore.
STFC-RAL-CR03 R61: Many cool thanks. So, if you're working on anything else or interesting any of this. Please join us in the past machine learning lab, where nonprofits is soon, hopefully, we're a collaboration of of researchers working towards the same goal. We meet once once a week. New techniques, methods, exchange things. Also, we have people from industry. Thank you very much.