STFC-RAL-CR03 R61: Than they are about intellect.
STFC-RAL-CR03 R61: transport, or eating meat, and other factors that typically used to be very high in conservice. So we want something that is very energy efficient.
STFC-RAL-CR03 R61: And that's also useful for stuff that you might want to put in space, or in medical devices. And we also want fast imprints, again, for help, for stuff on edge, in space, self-driving cars, list as well.
STFC-RAL-CR03 R61: Hello, how I do physics, I'm sure you all know, machine learning has increased quite a lot. I made a little plot of,
STFC-RAL-CR03 R61: I scraped, the PEP Living Review of Machine Learning, which is just a web page that relates all of the machine learning papers that are published in PAP, and I'm able to plot off the number of papers on there year by year.
STFC-RAL-CR03 R61: This part was made at the end of, kind of, autumn 25, so the last bar is kind of not complete yet. But I mean, you can, you can clearly see, unsurprisingly, number of, machine learning papers in the field is growing quite frankly.
STFC-RAL-CR03 R61: And then for triggers, we really care about cost inference. We've all… we already do machine learning on SPGAs, in many of the experiments, but the need for it seems to be increasing.
STFC-RAL-CR03 R61: So I've put a little poll here. You should be able to scan this QR code with your phone.
STFC-RAL-CR03 R61: To answer the poll, you can just, you don't have to sign in, you can just click skip and answer the poll. So the question, and this should obviously work on Zoom as well, the question is how much of the energy consumption and cost do you think that inference to just evaluating the model.
STFC-RAL-CR03 R61: Accounts for in a usual machine learning pipeline.
STFC-RAL-CR03 R61: So, what I'm basing this on is what kind of NVIDIA, Amazon, like, AWS, Google, Facebook, their self-reported figures.
STFC-RAL-CR03 R61: On what they and their users say that it takes up.
STFC-RAL-CR03 R61: Just give it my bill slime.
STFC-RAL-CR03 R61: Let's see what we get.
STFC-RAL-CR03 R61: Okay, good.
STFC-RAL-CR03 R61: Excellent. So it seems that they win ahead, 30-50%.
STFC-RAL-CR03 R61: And the correct answer here, again, by correct, I mean what they sell, they report in their own papers.
STFC-RAL-CR03 R61: Is, 70-90%.
STFC-RAL-CR03 R61: 50-70 is probably, you know, still kind of correct, especially for certain applications. But most of the time, what the figures that they're reporting around papers is 70-90%.
STFC-RAL-CR03 R61: I certainly didn't know that before looking into this, but it kind of goes to show that inference is actually a lot more important than we think it is, especially with LLMs, I think.
STFC-RAL-CR03 R61: Our impression might be that because they're such huge models, it takes so much resource to train them, and it does.
STFC-RAL-CR03 R61: But there's also millions of uses, so the inference matters. And that's not only the case for LLMs, but also for other machine learning applications. We find that inference actually takes up both a lot of the time and the energy at most. So, it's an important topic.
STFC-RAL-CR03 R61: So, how do we do our inference? It depends on the task.
STFC-RAL-CR03 R61: And I just wanted to give a quick overview of some of the devices. So, the one you'll be most familiar with is CPUs.
STFC-RAL-CR03 R61: We actually did… you might have seen my name out in a survey on, sending out a survey on machine learning, you know, how I do physics, and we had a survey recently asking how people train their machine learning and how they use the machine learning app.
STFC-RAL-CR03 R61: And one of the things we found is that about half of people who use, MLN head actually use their own CPUs. They use their own laptops for them, as opposed to, like, a cluster, or a GPU, or a dedicated device, which is quite high, because machine learning is quite taxing on a laptop.
STFC-RAL-CR03 R61: So, it's absolutely fine for something like a small boost position tree, a small model will do fine on a CPU.
STFC-RAL-CR03 R61: But I do think we found that, in fact, we do use maybe CPUs a bit more than we should for machine learning.
STFC-RAL-CR03 R61: The next step up is GPUs.
STFC-RAL-CR03 R61: And it used to be a bit hard to program GPUs. These days, it's very easy. In packages like PyTorch, it's very easy to convert a GPU. I think, based on our survey, one of the issues we do have is that they are sometimes a bit harder to access,
STFC-RAL-CR03 R61: the access to GPUs in the field is not entirely even. You have some universities that have fantastic clusters, and then you have others that have more than two clusters.
STFC-RAL-CR03 R61: But GPUs are probably the most used chip for machine learning.
STFC-RAL-CR03 R61: And then you have the more specialized ones, like tensor processing units, or neural processing units. As far as I know, they're not as much used in HAP. They're a bit more specialized, usually used by Meta, Google,
STFC-RAL-CR03 R61: And then we have FPGAs, which is the device we'll talk a lot about today.
STFC-RAL-CR03 R61: I'm gonna go into a bit more detail, but one of the general ideas is that with FPGAs, you can get really fast inference, but it's a bit tricky because you're much more resource constrained.
STFC-RAL-CR03 R61: So how to get your machine learning onto FPGA is a big non-trip.
STFC-RAL-CR03 R61: And then, if you want to go really extreme, there's also ASICS, application-specific integrated circuits. So these are devices that are bespoke. So you design your circuit, you print your circuit. And we have lots of them directly on the detectors, both for Atlas and CMS.
STFC-RAL-CR03 R61: And the trend that we're finding is that as our data gets bigger, as our computing needs get bigger, we are drifting, kind of, from this end here over to this end here. We're looking at the more specialized devices, how can we make things more efficient, how can we make it faster?
STFC-RAL-CR03 R61: But in general, the tricky thing as well is that, in general, it gets more and more per college program, the further you go down. So that is, certainly a trade-off.
STFC-RAL-CR03 R61: And another thing to keep in mind that we'll come back to, is that no matter which chip we use, no matter how we program it, ultimately it is translated to binary logic.
STFC-RAL-CR03 R61: And that is something that we will use later.
STFC-RAL-CR03 R61: So just to hone in a little bit on FPGAs, this is just kind of a mock-up, of how an FPGA might look.
STFC-RAL-CR03 R61: So an FPGA, for those of you who… I know there's a lot of expertise, here about FPGAs, but for those of you who don't know, an FPGA is essentially a chip you can… where you can reconfigure the physical circuit by programming it.
STFC-RAL-CR03 R61: So, as I mentioned, it has more of a barrier to entry of programming, so typically you can use,
STFC-RAL-CR03 R61: There are languages, like, very logical BHDL. They certainly have a barrier to them. They are a bit more tricky. Another alternative is high-level synthesis, where you write even Python-like code.
STFC-RAL-CR03 R61: Or, but more commonly, something like C++ or C Sharp, something like that, and then it translates it into, into something like PhDL or Verlook.
STFC-RAL-CR03 R61: Famously, there's also, a tool which is called HLS4ML, which is also a tool developed by someone.
STFC-RAL-CR03 R61: And that takes machine learning models, typically written in Python.
STFC-RAL-CR03 R61: and translates it to something that PGS can use.
STFC-RAL-CR03 R61: So that is also a popular package.
STFC-RAL-CR03 R61: And FPGAs, they… you can have very low latency. They're also… one part of the reason is that you can parallelize things very well in them.
STFC-RAL-CR03 R61: And they also don't have any caches, so a CPU or a GPU will have a cache, so it will try to, you know, predict what's going to happen next, and stuff like that.
STFC-RAL-CR03 R61: And there's, like, different levels of memory, so you have to kind of strategize, or maybe it happens in the operating system, how the memory access is done.
STFC-RAL-CR03 R61: Fpgas don't have that, and what that means is that you actually have, deterministic latency.
STFC-RAL-CR03 R61: So, that means that you… if you time something, it's gonna be the same every time. It's not going to vary depending on how things were in your cache, because there is no cache.
STFC-RAL-CR03 R61: And that is a really nice thing. If you need something that can really depend on, for instance, in space, where you need to know that the result is going to come at a certain time, or in medical fields, this can be a very nice thing.
STFC-RAL-CR03 R61: And there's also radiation, tolerant.
STFC-RAL-CR03 R61: parts of this, for instance, like we have in Texas.
STFC-RAL-CR03 R61: Orange does.
STFC-RAL-CR03 R61: So one thing we do need to understand a little bit, to kind of get a foundation for where we're going to go next with this, are some of the fundamental building blocks of an FPGA.
STFC-RAL-CR03 R61: So, usually, with… if you want to do machine learning on an FPGA, you use what I call, DSPs, digital signal processes.
STFC-RAL-CR03 R61: They are kind of general arithmetic, units on the FPGS. So you can do multiplications, additions, and so on.
STFC-RAL-CR03 R61: And usually, when you have a neural net, you have a lot of additions and multiplications, and that'll be put in this, this sort of extraction.
STFC-RAL-CR03 R61: They are quite limited on an FPGA, so it's not uncommon that if you have a machine learning method, you kind of have to make your neural nets a little bit smaller, so that it fits into these computing units on an FPGA.
STFC-RAL-CR03 R61: Another part of a network yet is a lookup table, and that's going to be quite important for some of the matters we're going to talk about.
STFC-RAL-CR03 R61: Output table is exactly how it sounds. It's just a table of inputs and outputs.
STFC-RAL-CR03 R61: So typically, on an FPGA, it would look something like this. You usually have 4 to 6 inputs and an output.
STFC-RAL-CR03 R61: The advantage of these is that you're just looking something up, so there's no computation per se.
STFC-RAL-CR03 R61: And that means that you can get really fast results this way. So it takes just one kind of tick of the cook, computing-wise, to get your answer.
STFC-RAL-CR03 R61: You also typically have a lot more of these than you do of the DSPs.
STFC-RAL-CR03 R61: And that's going to become important when we talk about different types of neural nets you can do with logic nets.
STFC-RAL-CR03 R61: So yeah, here's a more detailed example if you want to see, kind of, how the computation works, but I don't think it's too important, so I'm going to skip through that.
STFC-RAL-CR03 R61: As I mentioned, it is sometimes tricky to try and fit a neural net onto an FPGA. So, there are lots of techniques to try and do that, and these techniques are used not only for that, but also for making our machine learning faster, or more power efficient.
STFC-RAL-CR03 R61: So these are some of the… there's lots of techniques out there, but these are maybe the three main ones for taking a big model and making it smaller, making it faster, making it more efficient.
STFC-RAL-CR03 R61: The first one is pruning. In pruning, essentially, you just iteratively try to remove stuff in your neural nets. You try to remove nodes, you try to remove connections, and you see, you just don't get the same accuracy.
STFC-RAL-CR03 R61: And you can even do this while you're training, if you want to.
STFC-RAL-CR03 R61: And that typically can reduce lots of parameters. Part of that comes from something in machine learning that's called old parameterization. Usually, our models are bigger than we technically would need them to be.
STFC-RAL-CR03 R61: But discovering the smallest architecture possible that has the accuracy we need is hard. That's why usually we make them bigger than we need to, and we're fine with that. But when we're talking about really quick machine learning, we're not fine with that.
STFC-RAL-CR03 R61: So that's one method. Another one, especially relevant for FPGAs, is conjugation. So, typically, in a regular neural net, you have, you know, lots of decimal points, you have high bit precision.
STFC-RAL-CR03 R61: But very often, you can reduce that and still get away with a pretty good accuracy. So it's essentially just rounding the numbers to make a smaller memory. And that is especially relevant for FPGAs, where you have limited resources.
STFC-RAL-CR03 R61: Again, that's something you can do while you're training, so that you can learn the best representation. You can even do it differently in different layers of your neural nets to figure out, which parts do I need to be really accurate, and which ones do not.
STFC-RAL-CR03 R61: And then there's another method called knowledge distillation.
STFC-RAL-CR03 R61: And here, you have the teacher model and a student model.
STFC-RAL-CR03 R61: And the teacher model is usually a very big neural net, and then the student model is usually quite small, and it just tries to learn the behavior of the bigger model.
STFC-RAL-CR03 R61: So it just learns to imitate it, essentially.
STFC-RAL-CR03 R61: This is something that is used, for anomaly detection, currently deployed in the CMS Level 1 trigger, because we… we have a big, kind of CNN model.
STFC-RAL-CR03 R61: So a convolutional model, that learns to, to find anomalies in the data, but it's not small enough to fit on a PDF, so we have a student model that is smaller, that knows how, kind of, learns how to imitate the teacher.
STFC-RAL-CR03 R61: So these are some of the main methods, but they're not really enough.
STFC-RAL-CR03 R61: And… First of all, it's a bit… as you can kind of see here, it's a…
STFC-RAL-CR03 R61: It's a bit… it's a very thing in machine learning. It's a bit wishy-washy. It's kind of, you… you try, you fail, you try again. It's quite iterative, it's not…
STFC-RAL-CR03 R61: It's not very satisfying, mathematically.
STFC-RAL-CR03 R61: And also, there's several theoretical results showing, kind of, the limits of these methods. And one of the reasons for that is that you're kind of starting big and you're trying to reduce it. You're not going for something small and building up.
STFC-RAL-CR03 R61: And there is a fundamental difference in those two strategies.
STFC-RAL-CR03 R61: And another thing is that the hardware here usually is almost secondary. We make a good machine learning model, and then we say, okay, does this fit on our hardware? Instead of saying, this is our hardware, what machine learning should we do?
STFC-RAL-CR03 R61: So, we could, perhaps there is some potential for improvement there.
STFC-RAL-CR03 R61: So there are some paradigm shifts that aim to kind of shift how we think about machine learning in this way. A few of them would be neuromorphic computing.
STFC-RAL-CR03 R61: Which is where we try to kind of have machine learning, or just computing, that mimics how the brain works. So here, you can even go as far as having artificial tissue that tries to replicate computations.
STFC-RAL-CR03 R61: And one of the reasons that is so appealing is that if you look at the power needed for computations in our brain, it is so, so low. Like, you can't beat evolution that way. So that is a very attractive idea, but also similar with quantum computing,
STFC-RAL-CR03 R61: They're quite, not entirely new fields, there has been research for a long time, but we're still quite far away from having scaled machine learning on these things.
STFC-RAL-CR03 R61: So though they're very interesting fields, we all want to learn.
STFC-RAL-CR03 R61: And just to even more convince you that, that scaling
STFC-RAL-CR03 R61: inference is really, really important. Here's a little plot I made, I analyzed, abstracts from NERD and ICML, which are the two biggest machine learning, conferences, and we looked for keywords that mention
STFC-RAL-CR03 R61: Either, like, scaling up, so making a model bigger, or inference,
STFC-RAL-CR03 R61: And you can see that, scaling is becoming… sorry, inference, so fast inference is becoming more important.
STFC-RAL-CR03 R61: But scaling is…
STFC-RAL-CR03 R61: the main topic, and the reason for that is kind of, as you can see, it's the LLMs, it's the large language models. So people are really focusing on how do we make it bigger, how do we make it more accurate, and then we're kind of forgetting, okay, how do we make it more efficient and quick?
STFC-RAL-CR03 R61: Yes.
STFC-RAL-CR03 R61: So, we've seen… Also, for instance, the needs are increasing.
STFC-RAL-CR03 R61: And we aren't quite getting where we need to with the methods we have.
STFC-RAL-CR03 R61: And we need it more than ever.
STFC-RAL-CR03 R61: So, what do we do? Well, one potential solution is logical node, of course.
STFC-RAL-CR03 R61: I have a very intense slide coming up.
STFC-RAL-CR03 R61: So I hope you don't get scared. I just wanted to have this as an overview here, and don't worry if most of this is kind of a lot in one, but there are links to all of these papers if you are interested in learning about this field. So, just wanted to give a bit of a background of how this works in general.
STFC-RAL-CR03 R61: And the fields of, logic… sometimes I say logic gate neuromats, sometimes I say logic neuromats. Sorry about that, it's kind of, a nomenclature that has changed within the field recently. I mean the same thing.
STFC-RAL-CR03 R61: It has had two, kind of, mainstreams.
STFC-RAL-CR03 R61: So one is to learn without voice.
STFC-RAL-CR03 R61: So in a normal neural net, you have neurons, and you have connections. And the thing that you learn is how much each connection matters. So you have a weight that you keep changing as you're learning.
STFC-RAL-CR03 R61: And then some people thought, okay, that's really annoying, maybe we can just learn without the weights.
STFC-RAL-CR03 R61: And one really cool way of doing that is that you can actually just map… map your inputs to your outputs directly.
STFC-RAL-CR03 R61: So, the first way of doing that was, like, back in the 60s. And You still train?
STFC-RAL-CR03 R61: In the sense that you have lots of examples, and you learn, kind of, you learn how to map your inputs to a certain output in your memory. And then when you get a new example that looks a little bit different.
STFC-RAL-CR03 R61: You know already that it's pretty similar to this mapping you've already created, so you can use that for your algorithm.
STFC-RAL-CR03 R61: So, actually, that was some of the first vision, like, real-time vision, image recognition was with this sort of approach.
STFC-RAL-CR03 R61: And that was this paper over there.
STFC-RAL-CR03 R61: was actually a, like, multi-class classification. So, similar to some of you might have heard of the MLS dataset, which is, like, handwritten digits. It's that sort of thing. You take an image and you can classify and say, okay, this is a 1, this is a 2, or something like that. And that was actually something that we had way back then.
STFC-RAL-CR03 R61: It wasn't as good or as accurate as what we have now. The idea is pretty good.
STFC-RAL-CR03 R61: And then, things didn't maybe change here for a long time, and then in 2023,
STFC-RAL-CR03 R61: They came up with several clever tricks, to make this kind of adaptable to machine learning in the modern day.
STFC-RAL-CR03 R61: And then at the same time.
STFC-RAL-CR03 R61: there has been another idea in the field, and that is context learning. So again, going back to the idea of these wicks, they are pretty annoying, they take up a lot of memory, they take up a lot of time to learn.
STFC-RAL-CR03 R61: So, in 2016, there was a very interesting paper on binary neural nets.
STFC-RAL-CR03 R61: And here, you change your weight so that they're just zero points.
STFC-RAL-CR03 R61: So, essentially, what you're learning, it's almost like… connections between moods.
STFC-RAL-CR03 R61: And we might think you're going to lose a lot of information that way if you just put it to zoom.
STFC-RAL-CR03 R61: And you… you do it a little bit. You do have to compensate with other things in your network, like the architecture.
STFC-RAL-CR03 R61: But interestingly, you can still learn that way.
STFC-RAL-CR03 R61: And then there were some other developments, for that, so they recognized that, okay, maybe… so there was, like, a specific gate they used here, won't go into the details of that, but then, yeah, some of the developments were, okay, we can use logic gates as well.
STFC-RAL-CR03 R61: So, instead of just what this stream was doing up here was kind of just mapping
STFC-RAL-CR03 R61: So, this screen up here was kind of just the bad. Yeah.
STFC-RAL-CR03 R61: So their weightless learning was kind of just mapping an input zone output. And then over here, they kind of realized, okay, we can… we can have logic here as well, and then we can have math logic.
STFC-RAL-CR03 R61: So you can still use lookup tables, you can still map and Appleton inputs, but maybe you could even implement logic gates in your lookup tables.
STFC-RAL-CR03 R61: And that brings us to one of the more exciting developments, which was in 2022, which really kick-started a lot of the developments in this field. So as you can see, it's been going on for a few years before, but this was one of the things that really
STFC-RAL-CR03 R61: Made it quite relevant again. So, again, yeah, there's a link here to that paper. So this was presented at Nurex, which is, again, the biggest machine learning conference.
STFC-RAL-CR03 R61: And this is the method that we'll talk through in a bit more detail, of how we can use Logicades for these computers.
STFC-RAL-CR03 R61: There's also some other developments, that won't go into detail, so some new developments, but there's all links for here for that.
STFC-RAL-CR03 R61: Yeah, and again, more details, just so you can see, kind of.
STFC-RAL-CR03 R61: Why there are so many different flavors, and which kind of correction each of them brought, because it is very much a developing field.
STFC-RAL-CR03 R61: So, what I want to talk about is logic gate neural networks, specifically.
STFC-RAL-CR03 R61: I'm… I think it's best to show how these work officially.
STFC-RAL-CR03 R61: So, here's an animation, it's not mine. If you have an image, you can convert it to binary numbers.
STFC-RAL-CR03 R61: And then you send it through a circuit of logic.
STFC-RAL-CR03 R61: At the end, you can accumulate the results in order to get your classification.
STFC-RAL-CR03 R61: Okay? So, at this point, it's kind of similar to a regular romance. How come with that, so you have more duplicates.
STFC-RAL-CR03 R61: And you have to binarize your inputs.
STFC-RAL-CR03 R61: And this is really, really false.
STFC-RAL-CR03 R61: First of all, because you're using binary numbers.
STFC-RAL-CR03 R61: Secondly.
STFC-RAL-CR03 R61: you have logic gigs, and that means that your compiler can actually simplify it. Even if you say, okay, I need, you know, a thousand logic gigs, the compiler might say, okay, I can simplify that logic, and I can make it smaller.
STFC-RAL-CR03 R61: And there's no matrix multiplications.
STFC-RAL-CR03 R61: And that is what usually takes a lot of time.
STFC-RAL-CR03 R61: Matrix multiplications are, are, yeah, they take a lot of time, whereas here, you just kind of pass it through the network.
STFC-RAL-CR03 R61: Now, the challenge is that logic gates aren't differentiated.
STFC-RAL-CR03 R61: because they are binary. And usually in machine learning, we need to be able to differentiate. And the reason for that is that that's how we know, kind of, how to update our model. So in each iteration, when you do machine learning, you kind of try some parameters.
STFC-RAL-CR03 R61: And then you learn, where should I go next? You measure how much was I wrong or right, and then you try to update your parameters better next time.
STFC-RAL-CR03 R61: If you don't have something differentiable, usually you can't really know what parameters should I try next.
STFC-RAL-CR03 R61: So there is a clever little trick in here.
STFC-RAL-CR03 R61: To make binary logic differentiable.
STFC-RAL-CR03 R61: So we actually approximate the logic gates.
STFC-RAL-CR03 R61: With some regular continuous maths that we can differentiate.
STFC-RAL-CR03 R61: And that leads to some weird things. Again, I'm going to show you more visually, so I think that is best.
STFC-RAL-CR03 R61: So, for each node in the neural net.
STFC-RAL-CR03 R61: we actually use continuous numbers while we train. So, key difference here is that training and inference looks very different.
STFC-RAL-CR03 R61: But during training, what we have is that each node contains 16 different logic gates.
STFC-RAL-CR03 R61: And during training, we try to learn which logic gate to use at each node.
STFC-RAL-CR03 R61: So, as we train, you can kind of see we shift, and we see which logic edge should I use here.
STFC-RAL-CR03 R61: And then, at inference, we choose the best one.
STFC-RAL-CR03 R61: So, you have this really weird effect. During training, you have 16 different logic gates.
STFC-RAL-CR03 R61: For each node. During inference, you just select the best one. So your train and your inference looks very different.
STFC-RAL-CR03 R61: But then for inference, you have a somewhat small neural net that just contains logic rates.
STFC-RAL-CR03 R61: So, no matrix multiplications, and that makes it really, really fast.
STFC-RAL-CR03 R61: It has also been extended to convolutional neural nets. I won't go too much into the details there, just because of time, but essentially, you retain the idea of convolution as you do in machine learning, and there's just a couple of tricks, to work with logic. Yeah, but I won't go into too much detail there.
STFC-RAL-CR03 R61: So…
STFC-RAL-CR03 R61: The really cool thing about this is when this first came out, it had really, really good results. So…
STFC-RAL-CR03 R61: on a specific dataset, the MS dataset that I mentioned, handwritten digits, they had…
STFC-RAL-CR03 R61: the quickest inference of anyone. So they really had an inference that was unparalleled with the accuracy that they were serving.
STFC-RAL-CR03 R61: So they show that we're able to do this really, really fast.
STFC-RAL-CR03 R61: And there's, so there's two main papers, and the links are, are in the slides here, so you can have a look if you want to.
STFC-RAL-CR03 R61: And then there's been some development since then. Again, I won't go into too much details. There are some issues. Like, training is pretty slow, because you have to evaluate these 16 different logic at each points.
STFC-RAL-CR03 R61: And some other, some other issues. We've got very similar types of research.
STFC-RAL-CR03 R61: Another thing to note is that there are actually also several startups in this area.
STFC-RAL-CR03 R61: So, this guy who wrote this NERTS paper, it's called Felix Peterson, he has a startup. He's a little… he is… he's a friend of ours, we do talk to him a lot, but, he's a bit secretive about his, his start job, so we're not quite sure, where his researchers are.
STFC-RAL-CR03 R61: But yeah, he has a startup with around, I think, 11 employees, and there's also several other startups in this area.
STFC-RAL-CR03 R61: Because file transfer is something that many, many companies need.
STFC-RAL-CR03 R61: Another really cool thing about using logic is
STFC-RAL-CR03 R61: It's that, potentially, the more verifiable.
STFC-RAL-CR03 R61: And the more interpretable, the more components.
STFC-RAL-CR03 R61: But this is a normal… I should say, normal neural net.
STFC-RAL-CR03 R61: You have continuous inputs, continuous outputs.
STFC-RAL-CR03 R61: Alright, so essentially, you more or less have an influence database.
STFC-RAL-CR03 R61: So if you try to evaluate, you know, how… how well, or how a model will respond to something, that can be really hard.
STFC-RAL-CR03 R61: really, really hard. And either you try to solve it completely, and kind of look at all the possible, things that can happen, but that takes a lot of computer resource, and for the big models, it's also actually not possible with the methods we have.
STFC-RAL-CR03 R61: Or you try to make approximations. Which, you know, it can work, but it could also, especially with LLMs, safety is a big concern.
STFC-RAL-CR03 R61: So that is, that is all.
STFC-RAL-CR03 R61: With elephants!
STFC-RAL-CR03 R61: You have binary numbers, as inputs, and as outputs.
STFC-RAL-CR03 R61: That means, and you also have logic in there, so that means you actually have a final exercise.
STFC-RAL-CR03 R61: And that means you can actually just evaluate, okay, you… what would happen if I passed these values in, what would I get at?
STFC-RAL-CR03 R61: Because you have advanced assessments. And you don't have to even evaluate all the possibilities, there are ways to simplify it. It's not trivial, it's still…
STFC-RAL-CR03 R61: takes a bit of compute to do, and it might not scale up hugely well. There is one paper on this, but it is something that is very attractive, especially for business. There might be some very good potential here.
STFC-RAL-CR03 R61: Just to kind of hand it down on the point of, of how much, we don't understand, and we can't evaluate how the big models work at the moment, I have, two polls. I'm gonna give you this one.
STFC-RAL-CR03 R61: So they found that there's lots of different prompting tricks that improves LLM performance. So if you want, please answer the poll right now. So which of these prompting tricks do you think improves LLM performance?
STFC-RAL-CR03 R61: Funny.
STFC-RAL-CR03 R61: So if you're talking to an LLM, and it doesn't give you the answer it wants.
STFC-RAL-CR03 R61: what can you say to make a deal? Do you think any of these will work?
STFC-RAL-CR03 R61: Okay, good.
STFC-RAL-CR03 R61: Nice.
STFC-RAL-CR03 R61: So you're all kind of right, so the answer is all of the above. All of these things work.
STFC-RAL-CR03 R61: And in a way, it shouldn't, right? And there's also lots of other weird things about this, like, it was other fault, we won't do this, but they've also found that in December, if ChatGPT gives you shorter responses as compared to May.
STFC-RAL-CR03 R61: Right. And actually, at philanthropic, so that people who have the model code.
STFC-RAL-CR03 R61: They actually even have a team that is dedicated to looking at the mood of the models.
STFC-RAL-CR03 R61: It sounds a bit insane,
STFC-RAL-CR03 R61: It sounds a bit insane, and it sounds like we're really anthropomorphizing it, but what happens is drift. So, a model can kind of change behavior over the time as you're talking to it.
STFC-RAL-CR03 R61: And because we can't evaluate the model, we can't just map all the inputs and all the outputs, because that probability space, is almost infinite. Therefore, we have these teams that just sit there and try to talk to it, and try to see what happens. But mathematically, we can't quantify it, we can't predict it that way, that we would like to.
STFC-RAL-CR03 R61: And obviously, that is kind of why some people don't always have machine learning and physics, because that is kind of uncomfortable if we can't understand the behavior.
STFC-RAL-CR03 R61: And logic is, you know, it's, again, it's not trivial, but they do open the door to doing mathematical proofs in the machine learning to show exactly why it has made the decisions it has.
STFC-RAL-CR03 R61: So… We've shown that, okay, logic neural mats, they can do stuff fast.
STFC-RAL-CR03 R61: There might be more explainability in it.
STFC-RAL-CR03 R61: And it's quite appealing in the sense that instead of just learning a surrogate and then trying to map it to hardware, you're starting by thinking of your hardware.
STFC-RAL-CR03 R61: I'm going for math.
STFC-RAL-CR03 R61: Which is one of the reasons you have it so fast.
STFC-RAL-CR03 R61: So, I'll just very quickly go through some of the results we saw when we tried this, for the cicada data. So, in the CMS, we have a calorimeter, and from that, we make, kind of, lots of the energy deposits in the calorimeter, it's quite, quite a small image, which is one of the reasons this works.
STFC-RAL-CR03 R61: And then there's an autoencoder that learns to reconstruct an image.
STFC-RAL-CR03 R61: And then, based on how well it reconstructs the image, we know whether this is something it's used to seeing, or it's something it's not used to seeing. And from that, you can trigger and say, okay, the model has a really high reconstruction loss, it's not used to seeing this, therefore it's anonymous.
STFC-RAL-CR03 R61: So, that is something that is implemented on the trigger, and therefore it's the level 1 trigger, so it has a very short latency.
STFC-RAL-CR03 R61: And we tried to then take this model and implement it. So again, as I mentioned, this is kind of a teacher-student situation, so we took the student model, the small model that is implemented on the trigger, and we tried to use the differentiable logic gate neural method with that.
STFC-RAL-CR03 R61: So I'll just go through the results somewhat quickly.
STFC-RAL-CR03 R61: Essentially, the most important part is probably this one. So, this is our model.
STFC-RAL-CR03 R61: And this, in yellow is what is implemented on the tree.
STFC-RAL-CR03 R61: And we saw that we actually do get a better accuracy, than the baseline model, because the baseline model has a lot of tricks, right? It has the quantization, it has the pruning, those kind of standard methods of making a model smaller.
STFC-RAL-CR03 R61: So we actually saw that with this sort of method, we can achieve a very good accuracy, and maybe more importantly, we can achieve a very good latency.
STFC-RAL-CR03 R61: So there's lots of different flavors of that model that they've tried. The one at the bottom here is our model.
STFC-RAL-CR03 R61: and our latencies in clock cycles, so that's just kind of how, how many clock cycles, it takes. We do see that we actually have, a very, very quick model.
STFC-RAL-CR03 R61: Compared to what, is currently there.
STFC-RAL-CR03 R61: So there are still some issues to be figured out, but there is promising evidence that we can have both higher accuracy and a low latency.
STFC-RAL-CR03 R61: And when we can have lower latency, it opens up the door to have more complex models that can have better physics accuracy. Because very often, we have to kind of do a trade-off of, okay, we need it to be this fast, therefore we have to be okay with losing a bit of accuracy.
STFC-RAL-CR03 R61: And maybe this can open up for having better accuracy.
STFC-RAL-CR03 R61: So our work so far, one of the main things we are doing is
STFC-RAL-CR03 R61: As I mentioned, the guy, the main guy who develops this, he has a startup, so he's a bit secretive. Some of his code is public.
STFC-RAL-CR03 R61: We took that and we continued to develop it with new models and more support.
STFC-RAL-CR03 R61: So that library is open, so if you want to try it, you can click that link, and
STFC-RAL-CR03 R61: And play with it and try it out. We've also, published a couple papers on this, with a few, yeah, it's kind of, maybe not very interesting, it's a bit technical, and we are continuously submitting papers and continuing to develop this.
STFC-RAL-CR03 R61: I think there could be a lot of potential applications for this in high-end deposits. Triggering, obviously, is a very interesting one.
STFC-RAL-CR03 R61: Could it even be something that is relevant for FCC?
STFC-RAL-CR03 R61: With really big data, again, we do need good latency.
STFC-RAL-CR03 R61: Maybe even it could be used for fast inference offline, you do actually have good performance on GPU for this sort of model as well.
STFC-RAL-CR03 R61: So that could be relevant.
STFC-RAL-CR03 R61: And again, really appealing thing is that it had opened up for more verifiability in machine learning.
STFC-RAL-CR03 R61: Now, just to kind of show some of the challenges in the field.
STFC-RAL-CR03 R61: If we compare normal neural nets and logic neural nets.
STFC-RAL-CR03 R61: We are kind of in 2008.
STFC-RAL-CR03 R61: Compared to this. Where we're at with our models, how well they perform, is how well we did with normal neurons in 2008.
STFC-RAL-CR03 R61: And we're finding that scaling these models up is quite tricky. They take a long time to train.
STFC-RAL-CR03 R61: And also, there are some engraving issues and other issues still to outcome. But it is an active field of research, but right now, yeah, it's good to be aware that there is a limit to how big these models so far has gotten and still see success.
STFC-RAL-CR03 R61: So… Just to summarize, yeah, we just discussed this, so… but essentially.
STFC-RAL-CR03 R61: These modes are false, there's no matrix multiplications.
STFC-RAL-CR03 R61: Environment Verifiability.
STFC-RAL-CR03 R61: But they can be a bit harder to train, and the threshold to using them is definitely higher, ignored them by tonic machine learning.
STFC-RAL-CR03 R61: So, to finish up… You would ever consider using these?
STFC-RAL-CR03 R61: Let me switch it onto the talk.
STFC-RAL-CR03 R61: That means the talk has been affecting us.
STFC-RAL-CR03 R61: Okay, thank you. But then also, if you want to get started on this, there is a really good overview paper that talks about all the different flavors of the models that I talked about now.
STFC-RAL-CR03 R61: And also, I know some of this is quite complex, so it can take a bit of time to dig in. I'd recommend that paper and the Kingston papers that are…
STFC-RAL-CR03 R61: Also, as I said, that COVID's open, so hopefully around for COVID, and then, you know, I'll be around for lunch, or book me if you want, and ask questions now, and you can also feel free to email me.
STFC-RAL-CR03 R61: Thank you. Thank you.
STFC-RAL-CR03 R61: Questions from room?
STFC-RAL-CR03 R61: So,
STFC-RAL-CR03 R61: So what you showed was, using, gates with outputs for classification. Can you also do it for, say, regressions? Yes, yeah, exactly. The only difference is how you kind of, sum it at the end. So, usually you kind of just group some neurons together.
STFC-RAL-CR03 R61: And for classification, you can just look at the binary outputs,
STFC-RAL-CR03 R61: And then for regression, you kind of just scale the outputs, to get more aggressive value.
STFC-RAL-CR03 R61: on.
STFC-RAL-CR03 R61: Thanks. One question about the…
STFC-RAL-CR03 R61: it's a logic neural network, or you mentioned this training at the input side. I wonder, where does the training happen? If the training is also happening in the APK, or the training happened in the GPU or somewhere, and then after that, you, you sort of convert into the APGA. Okay, exactly.
STFC-RAL-CR03 R61: This was all completely new to me, and thank you very much for the talk. In my head, there's some internal sort of debate. There actually…
STFC-RAL-CR03 R61: a node or… with 16 logic gates can also just be represented by a neural network, and actually, at some level, they become the same, or if this is… so is this a repackaging of something at the very top level that is the same? Okay, so if I… if I understand your question correctly, so yes, you can take a normal neural net and translate it into logic gates. Is that kind of what you were thinking? So if you infinitely expand it, say, what you've got on the
STFC-RAL-CR03 R61: Yeah. Within each node you've got.
STFC-RAL-CR03 R61: Well, at some point, you showed 16…
STFC-RAL-CR03 R61: logic gates within one node, and thinking, surely you could have a neural network with the right weights that represents that exact same behavior? Yes. Yeah. If that makes any sense. I don't know if there's a mathematical equivalency, but a different representation going on here. Yeah.
STFC-RAL-CR03 R61: So, you absolutely can.
STFC-RAL-CR03 R61: And kind of the trick, because, yeah, it's easy to train the neural net with, like, continuous ways, right?
STFC-RAL-CR03 R61: And the trick is kind of going to the buying numbers. Yeah, sure. You could just translate a normal neural net into logic gates, and that actually was a really active field of research in, like, the 2000s. They found you can't just…
STFC-RAL-CR03 R61: you can't just map it one-to-one into Logicate very easily, and you end up losing a lot of information.
STFC-RAL-CR03 R61: So actually, there is a lot of research, because that was what I thought as well when I first came to the field. And also, that means you wouldn't have to retrain, you could just translate what you already had, because it's dead easy. But research-wise, it looked to be a dead end, which is why training in Scratch seemed to work better, but yeah.
STFC-RAL-CR03 R61: Yeah, if you go back to the slide onto paper… Yeah.
STFC-RAL-CR03 R61: Yeah, so it looks like, yeah, you get… you… I think you showed on the other slide as well that you get better results.
STFC-RAL-CR03 R61: with this LGN. Yeah, and then if you go back to the resource table… Yes.
STFC-RAL-CR03 R61: And it looks like the resource usage is lower.
STFC-RAL-CR03 R61: did you try implementing a larger LGN so that that would then give you even better results? Yeah, not so far. So one of the reasons is that the results here are a normal,
STFC-RAL-CR03 R61: normal neural net, that you see results for here. Our next steps was to synthesize, in this method, and that was a bit more tricky to fit into the chip, so that is something I'm still working on. But in what way was it tricky to fit into the chip? It was…
STFC-RAL-CR03 R61: increased resources compared with the LGO numbers here? Yes, it was the synthesis that was the… that was the issue. I don't… yeah, I actually have not specifically saying that. Okay, yeah, yeah, yeah, because normally, well, for this particular application.
STFC-RAL-CR03 R61: It's basically, you have certain resources in the chip, then you also have to meet timing in the chip, so there are problems with meeting timing the… Exactly, yeah, that could go real, but yeah, I didn't do the synthesis, but yeah, I'm sorry, but yeah, good question. But we are… we are continuing to work on that and trying to, ultimately, yeah, to assist in that.
STFC-RAL-CR03 R61: In my incredibly limited experience, quantization seems to be a huge problem, and I don't know if you have any tricks to deal with it, or if it's more or less just trial and error.
STFC-RAL-CR03 R61: So, the nice thing about this is that a conversation has already happened.
STFC-RAL-CR03 R61: Because you only… when you do your inference, you're only operating binary numbers, like, it translates your input to binary, so it's already completely binary, so you don't have to worry about quantization anymore.
STFC-RAL-CR03 R61: Same with pruning. It's kind of built into the architecture of this. So whereas with normal neural nets, as you say, it's a big problem with quantization and pruning, it's… it's really messy. Here, it comes baked in, from the sound.
STFC-RAL-CR03 R61: Actually, that leads into my question.
STFC-RAL-CR03 R61: Whoopi goes.
STFC-RAL-CR03 R61: looking at with regards to this is, are you limited by the designs of, FPGAs? For example, look at tables that only take…
STFC-RAL-CR03 R61: maybe, 6 or 8 inputs to a local machine. So, for example, in standard machine learning, the way you get around it, you can have a layer of 100 nodes, and an input could go to a…
STFC-RAL-CR03 R61: You know, one input going to the 107 nodes, or vice versa, one input taking
STFC-RAL-CR03 R61: yeah, a thousand inputs, but you can't do that on an PGA because the logic error. Is that a problem you find, or… Yeah, so, absolutely, I think that is… that is a problem. I think the… and it will be more of a problem
STFC-RAL-CR03 R61: In the future. I think right now, the limiting factor themselves is the fact that if you try to scale this up to something big, you lose gradients. So there's, like, gradient issues in the machine learning that people are really trying to solve, but I think that will be
STFC-RAL-CR03 R61: And also, another issue is that usually in machine learning, as you go deep, you sometimes have to add, like, residual layers, and it's not entirely obvious how that would be in logic gates. So it's kind of that that is the issue, but I think you're absolutely right. Once that is solved, then the resource utilization on the FDA will probably be the only factor.
STFC-RAL-CR03 R61: Great.
STFC-RAL-CR03 R61: So…
STFC-RAL-CR03 R61: completely naive with machine learning, so this might be a silly question. But, so when you've got these 16 gates, and you kind of… the example you gave, there was one that kind of, on the waiting, became very obvious, and it was that one that was chosen. I mean, I can imagine that there are cases where
STFC-RAL-CR03 R61: you know, two are very similar weights within those 16. Is that then… is there then an option to iteratively do this, where you then say, okay, that one logic gate, I got to… I can split it, add a third gate to recombine kind of thing? Is there kind of an iterative, you know, kind of build the logic gates, more complexity into it as you go?
STFC-RAL-CR03 R61: That's a really neat idea. As far as I know, no one has tried that in the field yet. I mean, it is a very insightful question, because that is one of the reasons why the gradients disappears, because some of the logic gates almost, like, cancel out, and some of them do have very similar signals.
STFC-RAL-CR03 R61: I think, yeah, as far as I know, no one has tried to kind of, like, figure it out that way by adding logic out. That's a neat idea. Yeah, maybe that'll work.
STFC-RAL-CR03 R61: Just coming to your page where you talked about where this was in the, sort of, timeline, compared to other things, and you… 2008,
STFC-RAL-CR03 R61: What are the… it's another page where you have the, access.
STFC-RAL-CR03 R61: Oh, that one, yeah, yeah.
STFC-RAL-CR03 R61: So what… what sort of criteria are you using in order to say, oh, it's like 2008 in comparison? Just accuracy on standard data sets.
STFC-RAL-CR03 R61: okay, but that's…
STFC-RAL-CR03 R61: So you mean, like, these numerals that… Yeah, so there are many… that dataset, and then there are many other, like here, that are usually used as benchmarks, and have been useful as benchmarks for a long time. And just based on the accuracy it achieves on that, then it is kind of…
STFC-RAL-CR03 R61: Okay, so that… so that's sort of…
STFC-RAL-CR03 R61: The biggest network we can achieve implement at the moment, and that's… that's the…
STFC-RAL-CR03 R61: No.
STFC-RAL-CR03 R61: Thank you for the nice talk. I'm…
STFC-RAL-CR03 R61: So I was a bit curious about, how difficult it is to train like this compared to conventional models, because it does seem that, once you do train it, it has, like, lower number of parameters.
STFC-RAL-CR03 R61: lower latency, higher latitudency to, use MPGAs. But is the training, like, actually, like, harder, or is it actually similar? Yeah.
STFC-RAL-CR03 R61: Very good question, yeah, your intuition there is right, it is,
STFC-RAL-CR03 R61: It is absolutely, harder to train. So, on these kind of typical benchmarks, very often you can train and get really good accuracy in, like, a few minutes. These logic gate neural ads might take an hour.
STFC-RAL-CR03 R61: They are harder to train, both in the code you have to write, in how sensitive they are to the parameters you use, so you might have to iterate more, and also just in pure training time.
STFC-RAL-CR03 R61: So yeah, they are. They are definitely not the…
STFC-RAL-CR03 R61: Yeah, you mentioned that you don't have the problem with quantization here, but then you have, in some sense, a similar one in these cases, like Craig mentioned, where…
STFC-RAL-CR03 R61: when the training is going on, maybe the conclusion is that you have, you know, you want to… when you've got this, combination of 16 logic gates, you have two of them which seem very promising, because that's giving a 30-50 contribution. Yes. How's that… is there something in the training which…
STFC-RAL-CR03 R61: at the moment, at least, kind of, forces the trainee to prefer a single output? Yes.
STFC-RAL-CR03 R61: Yeah, very good question. So, the thing that forces it to prefer something is a soft max function. So, it does kind of push it to do that, but then you're also right, there is, something that we call a discretization gap.
STFC-RAL-CR03 R61: So, you are definitely right, that once you go from that training and having continuous weights to having binary weights, you do lose your bones. So there is definitely something lost there. There is a paper on it on trying to reduce that. One of the things that people have found is that if you inject
STFC-RAL-CR03 R61: actually help with that, and make it, you know, further updates instead of just healing up. So typically, when you train, you see something like this, where there's a gap between your training reference, and yeah, there are a few tricks to mitigate it. But yeah, you're absolutely right.
STFC-RAL-CR03 R61: Let's take a question from Zoom. David, please unmute and go ahead.
David Sankey: Yep, I mean, the other obvious, game in town for look-based immigrants is boosted decision trees.
David Sankey: No, which give you a lot of the, when you have the explainability, you've got the…
David Sankey: No, you can do the differentiability at some level, so, I mean, where you're ending up seems very close to that. Do you care to comment?
David Sankey: how… how you're different from the, sort of, BDT route with, sort of, you know,
David Sankey: Quonifer rather than HLS for ML and that sort of game.
STFC-RAL-CR03 R61: Absolutely. So, there is… you're absolutely right that you could do this with disposition trees, and there is a paper called Tree LUT that is linked here that came out last year, that covers exactly that. They do get very good performance. The way the computations are done
STFC-RAL-CR03 R61: are different. Kind of the structure of having,
STFC-RAL-CR03 R61: having… the way the notes work is a little bit different, and you can read the paper if you're interested, but yeah, they are, the way they're trained, the architectures, do end up being slightly different, but yeah, you're absolutely right. There is definitely similarity there.
STFC-RAL-CR03 R61: Great, thank you. Any other questions?
STFC-RAL-CR03 R61: Any other questions from Zoom?
STFC-RAL-CR03 R61: Looks like not, so let's thank our speaker for the question.
STFC-RAL-CR03 R61: Thank you. I'm sorry, yes.