STFC-RAL-CR03 R61: It's also going to be showing things like the…
STFC-RAL-CR03 R61: Tim is also gonna be showing things later, Chris.
STFC-RAL-CR03 R61: He's actually done this.
STFC-RAL-CR03 R61: And we were trying to build a seminar slot, and Bridge asked whether we would talk about what we're currently doing around large language models. So I figured I'd go provocative and say, please stop calling it AI.
STFC-RAL-CR03 R61: He didn't want to convey that in the semin bike, but that's how I feel about it.
STFC-RAL-CR03 R61: There's an abstract somewhere, there's a large disclaimer that basically says, I know nothing about AI. So I shouldn't be talking, somebody else should be talking. If you've got any interjections or objections, you want to burn the slides and do something new, just do it better than I do, feel free.
STFC-RAL-CR03 R61: I'll talk about why we're here, different things, AI, as far as I want to go into it, and as far as I can go into it, and then the question of what do we do?
STFC-RAL-CR03 R61: How we got here is, in September, we had Andy Chisel from Birmingham here, and he showed us his bubble chamber project, which I've seen about a year ago. I kept thinking we should do something with it.
STFC-RAL-CR03 R61: We had seen in film, and we felt like we could maybe do something more, so there was a question, would we present this in Masterclasses? And we basically used Masterclass students as a sort of crowdsourcing for seeding event displays, and getting tracking done on that.
STFC-RAL-CR03 R61: So I started messing around with GitHub Copilot, which many of you probably know. It's a sort of coding agent support thing in Visual Studio that most of you will find easy to get to.
STFC-RAL-CR03 R61: Use the personal account, free-tier version of that, and very quickly spent my 50 free requests on that.
STFC-RAL-CR03 R61: But I got something running that was written in JavaScript, used SVGs, A little bit graphically slow.
STFC-RAL-CR03 R61: Don't know. SVG of an image format, or better graphics.
STFC-RAL-CR03 R61: as a display, and also for the data input, which I thought was quite intriguing. I had no idea how it worked. I had no idea about JavaScript. I hate Java, to start with. So JavaScript, for me, is an abomination. However, it works.
STFC-RAL-CR03 R61: So, to me, there was a realization that there's magic happening.
STFC-RAL-CR03 R61: And whilst I can read the code that comes out and understand what it does, I don't need to know how to write it to actually get something done.
STFC-RAL-CR03 R61: So that was an epiphany to me.
STFC-RAL-CR03 R61: I then asked around in SCFC, started with the IA Service Desk. Chris kindly responded and said.
STFC-RAL-CR03 R61: Christopher Kelly variety, for those of you who remember.
STFC-RAL-CR03 R61: said, we can pay £315 per person, and then we can get a license, which I assume is per annum, because that's the scale at which these things are charged.
STFC-RAL-CR03 R61: Apparently, some other department have bought hundreds of licenses of this, so we're spending around six figures or so on co-pilot at this point.
STFC-RAL-CR03 R61: So then, A, I wasn't sure we did want to go down that road, I haven't quite looked at it enough, I didn't know there was appetite in departments, or I didn't have any money to spend on it. There we go. I figured, let's try and look at other options. And it turns out that these days.
STFC-RAL-CR03 R61: Large language models, which are sort of…
STFC-RAL-CR03 R61: basic part of running an agentic coding support. They are open source, many of them you can just download and run yourselves. It depends a bit on hardware. I may have used the opportunity to convince my wife to buy a new computer, and I've started messing about with it at home.
STFC-RAL-CR03 R61: I then badjerked Chris Broom for a very, very long time, which felt like a week, maybe two.
STFC-RAL-CR03 R61: And then he let me have one of our decent computers, because we do have quite a bit of hardware, and I… I'm pretty sure it's not overutilized at the moment. When it becomes overutilized, I would happily hand it back, but at the moment, it's taken out of the queue.
STFC-RAL-CR03 R61: And we're running a Docker container on it that runs language models. I've downloaded a few
STFC-RAL-CR03 R61: Stuart, in particular, gave it a very decent try, and asked it questions around late 1989 events in various countries, and figured that some of these models have an intrinsic bias, where you might worry that they've been trained in particular ways.
STFC-RAL-CR03 R61: Some other models do report history accurately, as far as we're aware.
STFC-RAL-CR03 R61: So we're currently running an NVIDIA model, which was published around Christmas last year. It can do two parallel threads on the machine that it's running on. It's got a very large context window. Don't be fooled by that.
STFC-RAL-CR03 R61: I'll talk about context a little bit.
STFC-RAL-CR03 R61: But, that doesn't mean that it's very good in large context settings.
STFC-RAL-CR03 R61: As do many. At home, I'm currently running Wend3 Coder Next, which is an Albava open source model. That one clearly has a political bias in it. However, it is the gold standard under the open source models in terms of coding.
STFC-RAL-CR03 R61: It won't.
STFC-RAL-CR03 R61: Taking one step back, why not AI?
STFC-RAL-CR03 R61: I was gonna say something a bit about machine learning. As far as I'm concerned, machine learning started somewhere with linear regression about 200 years ago.
STFC-RAL-CR03 R61: To me, much of it's still very much that. It's trying to forecast a trajectory from something that you put in.
STFC-RAL-CR03 R61: whether that trajectory is a value, an assessment of quality, an object found in an image, it doesn't really matter. It takes an input, and it mathematically transforms that into an output, number, vector, whatever. So, linear algebra is somewhere in there.
STFC-RAL-CR03 R61: 1950s saw the term machine learning brand it, and a lot of that was to do with how true you are.
STFC-RAL-CR03 R61: turned from just linear regression into something that was trained based on a known answer. So, we started looking at backpropagation as a mechanism of training. We learned a lot more about that later.
STFC-RAL-CR03 R61: Training, or better.
STFC-RAL-CR03 R61: I was gonna briefly say, what does a simple neural network do? Who does not know what a neural network is? Who has never looked at details of neural networks at all?
STFC-RAL-CR03 R61: It's a safe environment here, you can speak up.
STFC-RAL-CR03 R61: First time I looked at neural networks was in 1998, and it was written in a book that could be bought in a normal bookshop, so I'm pretty sure they've been around for a while, people probably heard about them.
STFC-RAL-CR03 R61: Take your input vector, multiply by it as a scalar product by a weight vector.
STFC-RAL-CR03 R61: You get typically one constant input that's not defined as the actual input, which allows you to offset any
STFC-RAL-CR03 R61: any bias that you have in your impairment level that you need to get to, to then apply a threshold to the sum of… to the… to that scalar product, and then you get now a value. That's a single neuron. So to first order, scalar product does some sort of linear action on your values.
STFC-RAL-CR03 R61: And then you apply a threshold, typically a soft threshold or a hard threshold.
STFC-RAL-CR03 R61: If you can accept that, then what does that mean? That thing?
STFC-RAL-CR03 R61: does single… Plain cuts in your input vector.
STFC-RAL-CR03 R61: It can… it can form one plane on whatever dimensional space you have.
STFC-RAL-CR03 R61: And define a cup on that.
STFC-RAL-CR03 R61: What's up?
STFC-RAL-CR03 R61: Do people agree or not agree? Is that an acceptable statement?
STFC-RAL-CR03 R61: trying to make this interactive, so… Yes. Cool. So if you accept that, then if you wanted to define, say, a cut sectional space, I want to define… I have my easy two-dimensional space, and I just want to derive a band that says yes.
STFC-RAL-CR03 R61: In my space, whatever my input vector is, then I need two of these neurons to define the two thresholds, and then I need a second layer that forms a judgment based on the output of the two neurons.
STFC-RAL-CR03 R61: Accepted.
STFC-RAL-CR03 R61: Pro tip. I see Chris on here, so that's always good.
STFC-RAL-CR03 R61: So yes, two nodes in parallel can apply different plane thresholds. A second layer can then apply a cut based on multiple of these thresholds. So, as you're adding layers of neurons, complexity gets added in.
STFC-RAL-CR03 R61: You can imagine that if you do multiple plane cuts, and you try and fold them in all together.
STFC-RAL-CR03 R61: If you're trying to do a square cut.
STFC-RAL-CR03 R61: You can't do that purely based on
STFC-RAL-CR03 R61: No, you could do that purely based on a two-layer network, where the first layer defines, say.
STFC-RAL-CR03 R61: two vertical cut planes, two horizontal cut planes, and the third layer always defines the difference between those two planes. What you'll find is that there is still a problem that the linear sum brings you into corners.
STFC-RAL-CR03 R61: So if you do actually want to cut just on the square, you need a third layer already, just to define that spatial object.
STFC-RAL-CR03 R61: What I'm trying to say is…
STFC-RAL-CR03 R61: A neural network with two hidden layers cannot do image recognition. It cannot tell you whether there's a cat in the image, which is what people tend to say a lot.
STFC-RAL-CR03 R61: hidden layers add complexity, and adding two or three hidden layers makes everything possible. It really doesn't.
STFC-RAL-CR03 R61: You can, of course, define an infinite number of
STFC-RAL-CR03 R61: Neurons in your first hidden layer that allows you to apply an infinite number of thresholds.
STFC-RAL-CR03 R61: And if you did that, and then made a selection in the second layer on your infinite number of thresholds, then you could probably define, say, a per pixel
STFC-RAL-CR03 R61: accepting or not in an image, and it would totally kill the system. You could never train this, but it could then clearly get to the point where it identifies all the possible cat pictures in the world.
STFC-RAL-CR03 R61: It is a lot of effort for not much gain.
STFC-RAL-CR03 R61: So, hidden layers and complexity, you need lots of hidden layers to do something that is really complex. Two, three, four hidden layers don't do very much.
STFC-RAL-CR03 R61: Okay.
STFC-RAL-CR03 R61: Moving on. Deep learning. Deep learning? Yeah, go for it. So you've seen…
STFC-RAL-CR03 R61: just four hidden layers cannot do much, irrespective of the number of nodes within the layer? No, not irrespective of the numbers, but it's very inefficient if you're trying to do something complex. And so there's things like,
STFC-RAL-CR03 R61: My favorite example, neural network clusters was in Atlas Pixels.
STFC-RAL-CR03 R61: Which throws a 7x7 matrix of time over threshold values into the mix, and assumes that it can do pattern recognition. And it really can't.
STFC-RAL-CR03 R61: And this is a two-hidden layer network, so it's called a third output layer.
STFC-RAL-CR03 R61: So there's 3 notes, so we can find squares.
STFC-RAL-CR03 R61: Of almost arbitrary shape.
STFC-RAL-CR03 R61: But it can't do anything more complex than SWAT.
STFC-RAL-CR03 R61: it can't do charge distribution into two, sort of, sigmoid-shaped objects or something. It… it can't. And so, looking at that network, I've always worried that people seem to infer too much from what a few hidden layers can do, and I'm pretty sure four hidden layers can't do much more.
STFC-RAL-CR03 R61: It's when we're talking about neural networks later, you see how many layers there are.
STFC-RAL-CR03 R61: And then it becomes interesting. Okay, I'm just… I'm cautious about what?
STFC-RAL-CR03 R61: Yeah. That includes in the…
STFC-RAL-CR03 R61: I would say those… there's lots of factors, so the number of nodes in the layer, also training and inputs, so 4 on me sounds a lot, right?
STFC-RAL-CR03 R61: 4 is a lot of layers, but it really depends on your geometry of the in layers. So typical neural network designs, either half the number of intimate layers in the first… half the number of intimate nodes into the first layer, or they double it.
STFC-RAL-CR03 R61: The doubling is clearly what I said earlier, if you wanted to find a threshold on a certain
STFC-RAL-CR03 R61: parameter access in your input space, then you need two nodes to define a band within those thresholds. So doubling is a tendency that people do, which, for example.
STFC-RAL-CR03 R61: in the neural network with clusters within Atlas, it's not done.
STFC-RAL-CR03 R61: So number of hidden nodes in the first layer is exactly the same as the number of input notes, and that really doesn't help with any headphones recognition. So it's… it's trying to make people aware of what can each layer do.
STFC-RAL-CR03 R61: That will then later on, help you figure out how many layers do you need to do something complex.
STFC-RAL-CR03 R61: And yes, 4 layers is a lot, but again, 3 layers can find a square.
STFC-RAL-CR03 R61: Or… Yeah,
STFC-RAL-CR03 R61: trapezoid, presumably, or something, but you… you get the gist. Yeah, I guess three layers don't do… so adding one more layer.
STFC-RAL-CR03 R61: Doesn't do much more.
STFC-RAL-CR03 R61: I guess my point is also architecture is a… architecture of those layers is important. Like, for example, you say if you go from input to the first layer, and you double the nodes, then that's how… Yes, yes. Number of nodes will always have an impact, and there's always a question
STFC-RAL-CR03 R61: How many numbers do you feed in, and how much do you go up in the first dimension, and then scale down to whatever your output dimensionality is. Often enough, it's one.
STFC-RAL-CR03 R61: Brilliant.
STFC-RAL-CR03 R61: Good. There's a lot more detail around machine learning, and that's a completely separate topic where Manny will do a seminar one day and teach us all about the mathematics of machine learning.
STFC-RAL-CR03 R61: What about deep learning? People keep talking about deep learning like it's magic.
STFC-RAL-CR03 R61: It is, and it's not. It's lots of hidden layers. One and two hidden layers were a thing around the 2000s, it was something that people could understand. People clearly did more.
STFC-RAL-CR03 R61: But it didn't really work, and there were… there was a phase transition where, at some point, people figured out how to properly train lots and lots of hidden layers. Because as you fold many values into one output value, you clearly need to understand very carefully what your… what your first hidden layer weights are.
STFC-RAL-CR03 R61: To make sure that your output weights are correct.
STFC-RAL-CR03 R61: So there's that vector propagation off. I know the answer, I've got my input vector over how many variables affect the output?
STFC-RAL-CR03 R61: That is the challenge. So when you add and add hidden layers, it becomes more and more challenging to get the input weights well defined.
STFC-RAL-CR03 R61: As in the first weight.
STFC-RAL-CR03 R61: Manny can interject if he thinks that's wrong.
STFC-RAL-CR03 R61: People figured out deep learning at the time, sometime around 2010,
STFC-RAL-CR03 R61: Again, training algorithms have improved massively, and hardware has improved massively, so we can do a lot more. Cool.
STFC-RAL-CR03 R61: Last language models.
STFC-RAL-CR03 R61: I know this is quick, and this wasn't going to be a training course or something, I don't know enough about it. Generative pre-transformers is the term for GPT, and that's why it's called ChatGPT.
STFC-RAL-CR03 R61: What they do is, I don't know, our current model probably has
STFC-RAL-CR03 R61: 512 nodes per hidden layer, maybe 1,000, maybe 2,000.
STFC-RAL-CR03 R61: And they've got up to 40, 50 hidden layers stacked on top of one another. So earlier when we said first node is a threshold, second node is a bend, third node is a…
STFC-RAL-CR03 R61: Spatial object of some description.
STFC-RAL-CR03 R61: You can imagine what lots and lots of layers stacked on top of that will start adding agents.
STFC-RAL-CR03 R61: They are complex piece. What they do is they take a piece of input text.
STFC-RAL-CR03 R61: Texts get split up into tokens. Tokens are typically a syllable in Chinese, for example, they would be a character, typically.
STFC-RAL-CR03 R61: In European languages, they tend to be either one or multiple letters, and it depends very much on what you're reading.
STFC-RAL-CR03 R61: When it's code, it really matters. When it's language in a typical English sentence, then it's a little sort of scale of what the quote should be.
STFC-RAL-CR03 R61: And tokens then define the language content, what is trying to be fed in.
STFC-RAL-CR03 R61: In a first iteration, after these tokens are generated, there will be an attention layer that tries to identify what's the important bit in this. Is the end just waffling, and what is the part that I need to actually process?
STFC-RAL-CR03 R61: That will have a larger impact on how well that thing works or not, whether it figures out the right bits or context in your input.
STFC-RAL-CR03 R61: And it then takes what context you fed in, it processes it into an output token, a single token. So every transformer, what it does is it takes your old context.
STFC-RAL-CR03 R61: however pre-processed, and then process that… processes that into one output token. That one output token then gets added into the context, unless that output token says.
STFC-RAL-CR03 R61: End of context, or something, or end of…
STFC-RAL-CR03 R61: Sweet. So basically, it's self-feeding.
STFC-RAL-CR03 R61: And it tries to complete the conversation.
STFC-RAL-CR03 R61: What these things are trained on.
STFC-RAL-CR03 R61: It's that if I start a sentence like, press through this.
STFC-RAL-CR03 R61: then it'll complete it, an IT specialist in PPE or something, because it's read that somewhere, and that is a reasonable end for the sentence or something.
STFC-RAL-CR03 R61: That's what it learns.
STFC-RAL-CR03 R61: So…
STFC-RAL-CR03 R61: That's where, again, I feel like, is that AI? Not really. It forecasts the end of the sentence based on the start of the sentence.
STFC-RAL-CR03 R61: It can do that over a very large set of tokens, depending on the model, depending on how about the strain.
STFC-RAL-CR03 R61: But all it does is it produces text in response to a given input text.
STFC-RAL-CR03 R61: Okay?
STFC-RAL-CR03 R61: Okay. There is one other difference in that.
STFC-RAL-CR03 R61: Those large language models are not deterministic.
STFC-RAL-CR03 R61: If you run it a step and time, you'll say Chris Brew is a…
STFC-RAL-CR03 R61: Cyclist. Yes, yes. They have temperature, and they put thresholding in various spaces, and they do weird things in terms of ranking potential outputs. So they generate internally
STFC-RAL-CR03 R61: A list of potential output tokens, and they then select the output token based on a random seed and on a temperature.
STFC-RAL-CR03 R61: Yes.
STFC-RAL-CR03 R61: Each model will come with a recommendation on how to run it.
STFC-RAL-CR03 R61: part of that is, yeah, a large part of that is they know how they trained it, they have tested it in a test bench, and they understand how to get the best results out. It also means that you don't always get the same results.
STFC-RAL-CR03 R61: Details.
STFC-RAL-CR03 R61: Prep in, prep out is the one thing. Some models are better than others with dealing with crap, but by and large, if you feed the wrong instructions in, the wrong cell comes out. Do first order.
STFC-RAL-CR03 R61: Number of parameters help, the vocabulary set, so the input vector size for a token.
STFC-RAL-CR03 R61: matters. If you've got a very large input character set, then your model can handle more complex tasks.
STFC-RAL-CR03 R61: Basically.
STFC-RAL-CR03 R61: And then tools make a massive difference, which I think you will hear more about content Basically.
STFC-RAL-CR03 R61: Tools allow the language model, basically, you tell your language model, I need an answer to my question, and by the way, if you think you need more information, then please tell me by saying, I need this tool called. I need to ask Google what the time is in New York, or something, because the language model clearly doesn't know what the time is.
STFC-RAL-CR03 R61: So then it queries that, and then it can reply what the time in New York is.
STFC-RAL-CR03 R61: So by handing it tools and allowing it to call those tools, you're turning a language model in itself, which is just a thing that turns one string into another, into something that can behave as an agent.
STFC-RAL-CR03 R61: It can fulfill an action for you.
STFC-RAL-CR03 R61: question that I have here.
STFC-RAL-CR03 R61: And that is mostly why I'm speaking today, is do we want to do more in that direction?
STFC-RAL-CR03 R61: We can clearly sell post, we've got multiple models, I tried a few, I don't know what Manny's done so far in terms of running it.
STFC-RAL-CR03 R61: Tim's played around with it, others may or may not have tried looking at them.
STFC-RAL-CR03 R61: I used the model, I used it, I didn't train anyone. So, what we haven't done well, lots of details around agentic systems, but Tim will show you in a bit.
STFC-RAL-CR03 R61: What we also haven't done is we haven't looked at training of models.
STFC-RAL-CR03 R61: Large language models, by and large, get trained on a basic
STFC-RAL-CR03 R61: language. They can talk a language so that they can speak, and then they often get fine-tuned in separate layers, so this is a layer separation.
STFC-RAL-CR03 R61: which allows them to understand specific contexts better. So, for example, if you wanted something that is trained on coding, you go for a specific model. If you want something that's trained on poetry.
STFC-RAL-CR03 R61: Whatever.
STFC-RAL-CR03 R61: Then you pick another model rather than calling, because they tend to be better at one thing, not everything.
STFC-RAL-CR03 R61: Tuesday.
STFC-RAL-CR03 R61: Okay, so… Just get a little biased.
STFC-RAL-CR03 R61: My little bit is going to be based on, a one-hour tutorial session which was put on by the Hartree Center a couple of weeks ago now, on basically concepts and practical implementations using agentic AI Tools.
STFC-RAL-CR03 R61: Let me get a heads up who was in that training.
STFC-RAL-CR03 R61: Okay, great.
STFC-RAL-CR03 R61: So, they started showing how you could start, basically, from an empty Python shell.
STFC-RAL-CR03 R61: build up a little routine, but then they quickly moved on to using a pre-existing, developed framework for this to just handle the plumbing easier than to get to the, sort of, the interesting parts. So, this demo then is built using this, this API. It's called Langraph.
STFC-RAL-CR03 R61: And with this, you can build a graph, which then enacts a sequence of actions and links everything together, and you can build an agent from this.
STFC-RAL-CR03 R61: So, this example here is sort of the simplest example of a graph you might want to build. So far, I haven't actually had to go beyond this particular graph structure. So you go into an LL node here, which
STFC-RAL-CR03 R61: Receives a query, and then it's going to produce text as an output.
STFC-RAL-CR03 R61: The output of it goes into this conditional edge, and there's two options from here. Either the conversation or the task of the agent can be finished, and it goes down to the end node, or if it decides it has more work to do, it can go down this action node, and its particular query can be dispatched to a tool.
STFC-RAL-CR03 R61: which lets it then gain additional information. That information feeds back into the LLM.
STFC-RAL-CR03 R61: And this cycle here on the right could continue multiple times until… We're processing fish.
STFC-RAL-CR03 R61: So, to give an example, this is Hartree's example.
STFC-RAL-CR03 R61: They define three tools. There's a Wikipedia search. Search Wikipedia and return the summary, e.g, what is Agentic AI, and you can see this does return wikipedia.summaryquerySentences equals 3, using an existing Wikipedia Python API.
STFC-RAL-CR03 R61: There's a stock price tool, which has a Yahoo Finance API, and just returns, yahoofinance.ticker, and returns the ticker is currently trading at price.
STFC-RAL-CR03 R61: And then finally, a calculator tool, which simply does Python, string, evaluate, expression, to do some maths.
STFC-RAL-CR03 R61: The way that these tools, you see they have a Python annotation, so these are integrated. The large language model knows docstring, so it's for humans to understand the tool, but it's also for the LLM to understand how to use this tool.
STFC-RAL-CR03 R61: And then the final thing which is needed is a prompt. So, you need to give your LLM node its job description, if you will.
STFC-RAL-CR03 R61: So, the one they use here, you're an autonomous AI agent, you may use tools to help answer users' questions. Don't mention the fact we used a tool, etc. If you used it, incorporate results naturally into response, and these are free tools available to you.
STFC-RAL-CR03 R61: So, you can then run this.
STFC-RAL-CR03 R61: I can… I can give some input. Look up a few prominent FTSE stocks, and tell me their current prices. If I do this, then the model starts going away, churning off on its own, so it starts…
STFC-RAL-CR03 R61: asking Wikipedia for C100 constituents. It does it twice, I don't know why. Then there's index. It seems to get itself a bit stuck in the loop, to be honest here. It keeps on going down, asking it various combinations in Wiki until it finally decides, okay, I'm gonna ask for structural for BP.
STFC-RAL-CR03 R61: Do a few more wiki searches, world.shell ticker, shell.
STFC-RAL-CR03 R61: HSBC, first it gets it wrong, it says HSBA, realizes it's making a mistake, it does HSBC, it does AZN, which is AstraZeneca, and then finally, after all of these, after it's done all of this, took about 30 seconds in total, it prints the final result. Here are a few well-known FCSC, 100 constituents and the ground share prices.
STFC-RAL-CR03 R61: Which were… which were correct, so it did manage to correctly go off.
STFC-RAL-CR03 R61: perform the task I gave it, using the resources I gave it access to, and, and come back.
STFC-RAL-CR03 R61: So…
STFC-RAL-CR03 R61: That is all using this basic graph node. As I say, I've not had an opportunity… I've not really had a reason yet to go beyond this, but you could think about… you could have multiple of these LLM nodes. You could give them different prompts, so that they have different behaviors, and then they can… the model can talk to itself with sort of different hats on.
STFC-RAL-CR03 R61: You could have different nodes which go to different back-end large language models, because, of course, some are more specialized nodes should be able
STFC-RAL-CR03 R61: One which is more specialized for coding, looking at coding problems. You can have another one which is maybe better at natural text generation, looking at emails and things like that. And, have one at the top, which is deciding, okay, does this go to the coding box, does this go to the email box, etc.
STFC-RAL-CR03 R61: So, to which agent, which large language model did you use? Was that the Google one they suggested in the… Okay, so I started this using, so this was run on a Google Colab.
STFC-RAL-CR03 R61: In a tutorial by connecting to Gemini. But, within, like, 10 minutes, I ran out of tokens. Gemini gives you 20 tokens, 20 requests per day.
STFC-RAL-CR03 R61: Each of these is a request, so you can see this one command here would instantly have run out of my Gemini. So this wasn't your Gemini, because you'd only run out? No, no, so this is all running backed locally by the LLM
STFC-RAL-CR03 R61: Running on Mercury, which, which Jen's set up.
STFC-RAL-CR03 R61: And yeah, I kind of wanted to continue the exercise, so my first thing was really just get this current exercise running, so I can play with it a bit more, because without paying for Gemini, you very quickly
STFC-RAL-CR03 R61: I can't remember anymore.
STFC-RAL-CR03 R61: So, that then gets the Hartree example up and running, up to the end of the tutorial. So, I've had a little bit of more playing around, sort of saying, okay, this is great, but I don't really need to know the stock price of AstraZeneca right now. How can it help me more, with my day-to-day stuff?
STFC-RAL-CR03 R61: So, I introduced, I gave it a few more tools. I gave it a very simple, I call it the coding standards tool. In Atlas, we have, one page, or one web page.
STFC-RAL-CR03 R61: of Markdown, which has all of the standards by which our C++ and Python code should perform.
STFC-RAL-CR03 R61: So I gave it a very simple way to search for terms and return paragraphs from our coding standards tool. I also gave it access to a couple of Unix command line tools, so I gave it access to Find.
STFC-RAL-CR03 R61: Am I given access to Git?
STFC-RAL-CR03 R61: An autonomics, you're an autonomous AI agent, which is an expert developer of Athena, a large C++ and Python framework used by the Atlas experiments. You may use tools to help answer users' questions.
STFC-RAL-CR03 R61: I also found that it was doing some very wrong things in Git initially, so I, needed to give it a little bit of a hint, but remember that source files are all located within subfolders, packages in Athena. You'll often need to give a full path to a file when using Git and your tools.
STFC-RAL-CR03 R61: And so, with that, I could ask it, please review code within this package, trigger, trigger analysis, trigger naps within AMT. This is happening in one of my packages, so I thought I'd give it that as a test.
STFC-RAL-CR03 R61: It went off and it did Git LS files, it, queried the coding standards tool for the naming, it then did some more Git, show, head.
STFC-RAL-CR03 R61: Show head of the main header and the main source files, and coding standards towards, again, error handling.
STFC-RAL-CR03 R61: Based on that, it then produced a large markdown output, where it did a reasonable, to the extent where it is definitely a useful review of the code. I would have liked it to have looked a bit more at the different areas of the coding standards, but for a first
STFC-RAL-CR03 R61: first try, basically. Already very impressive stuff, and I link a full, full example output of what it took you there.
STFC-RAL-CR03 R61: So, given that very promising startup, I've actually been using it for the last couple of weeks, so I'm currently doing Atlas Code Review shifts. Every afternoon, I sign on and look for merge requests which have been flagged, for, some more review, and I look through them.
STFC-RAL-CR03 R61: But I'm also now running a little bit of Python code, where I put in the merge request number and the number of commits in the merge request.
STFC-RAL-CR03 R61: I then run a Git MR origin checkout, just using, GitLab Python specifics to check out a specific merge request locally.
STFC-RAL-CR03 R61: And then, once it's checked out, I can tell the model, please perform correctly due to the code we checked out Athena branch, the developer's work is going to sketch the last 8 commits on the branch, and it does a very similar thing. It does Git tools, it does…
STFC-RAL-CR03 R61: coding standards, it looks… it's not going to find special geometry to or identify coding standards, so that was a bad request, but it did get more hits for naming conventions.
STFC-RAL-CR03 R61: It then, looks for a cert, and it does some finds. So this is a bit where I'm a little bit conflicted.
STFC-RAL-CR03 R61: Find is a very powerful Unix tool, but it has lots of edge cases. In this case, the LLM is using it very powerfully. It's doing find.name, asterisk.yphon, and then execrep on that for geometry drawing. That's great, that's a really powerful way of doing it, but of course, if you're doing exec, there's more…
STFC-RAL-CR03 R61: Things which could happen in there.
STFC-RAL-CR03 R61: So, yeah, so, excuse me. So, as a complete, you know, ignoramus.
STFC-RAL-CR03 R61: I know all kinds of shifts, but I never heard of a code review shift. What are you, actually? So, yeah. In order to keep the quantity of my codebase reasonably high, when somebody wants to make a change.
STFC-RAL-CR03 R61: They open what we call a merge request, which is where they bundle their changes into a little package. They write a description of what they want to do, and then this gets presented on the webpage. They see the description.
STFC-RAL-CR03 R61: You see the list of all, like, the diff, basically, of all of the code changes, and we can interactively comment
STFC-RAL-CR03 R61: discuss the changes with the developer, iterate over the changes until everybody's happy that they are in a state to be merged. At that point, the release coordinator will press a merge button, which then integrates their changes into the codebase, and then they become part of the next nightly.
STFC-RAL-CR03 R61: So, presumably, it's a good way for people to learn. I mean, these people on the shift are going to be experts, and somebody trying something may be not an expert, and you can really… sounds good. Absolutely, yes.
STFC-RAL-CR03 R61: And this has been really… I class myself as an expert of our code.
STFC-RAL-CR03 R61: But this has been a really helpful thing to… it gives me a second opinion, basically. I can look over it, I then look at what the large language models spat out, from running through its thoughts. I compare them to mine.
STFC-RAL-CR03 R61: See, maybe it picks up something I missed, especially if you're reading through quite fast, it's quite easy to miss the occasional thing here and there. So,
STFC-RAL-CR03 R61: I don't think it's at the stage where it's going to replace me doing code review, at least not yet, but as a second opinion, I'm finding it's saving me a lot of time, and it's letting me go through this faster than it otherwise would.
STFC-RAL-CR03 R61: I have a question on the previous page, very quickly. So, how is the connection between the query in the form of human message, and the calls that are generated without that?
STFC-RAL-CR03 R61: So, shall we look at that later? Yeah, maybe we can… Or maybe… We can come back to… it's Python. Okay.
STFC-RAL-CR03 R61: less than 200 lines, I think. You'll see here at the beginning, it's how it's human message. So, in essence, what you're doing is you're just building up a long conversation history, like a chat log, where every message is flagged, and it's going to be flowing human message, tool message, AI message.
STFC-RAL-CR03 R61: There might be another couple of categories, but those are the main ones, and you just build up a long
STFC-RAL-CR03 R61: Like a tapestry, as it progresses down.
STFC-RAL-CR03 R61: down through all these stages.
STFC-RAL-CR03 R61: Yes, okay, so my thoughts, are that, it really took very little effort to write up a simple agent graph with one node being the large language model node, and a suite of custom tools available to it.
STFC-RAL-CR03 R61: General question, is it better to be routing through local open source large language, probably Copilot?
STFC-RAL-CR03 R61: I would say yes, we have more control over exactly what information gets sent. I would feel if we're putting… starting to put, sort of, more sensitive data in it as well, it's nice to know that everything is local and nothing's having to go off
STFC-RAL-CR03 R61: to the cloud. I would say it's very important to audit the tool requests a large language model is making.
STFC-RAL-CR03 R61: So far, nothing outright malicious or dangerously stupid has been observed.
STFC-RAL-CR03 R61: And the LLM, when you're… when it's executing commands by Python, it's not doing it in a shell, so it can't use pipes, it can't choose redirects, and things like that, so that somewhat limits the sort of danger footprint. But, I mean, as you saw on the previous page, if you give it to find, then it's already…
STFC-RAL-CR03 R61: Got a way to do more expressive
STFC-RAL-CR03 R61: Commands, and there's a trade-off.
STFC-RAL-CR03 R61: you give it more powerful tools, it can do more powerful things, but there's more potential for misuse. Can you audit it before it runs? You can, you can. You can put a human in the loop node, although… Or you can run explicit command checks, but… Yeah.
STFC-RAL-CR03 R61: Yeah, I think it's… I already… I did put in the chat that it cannot…
STFC-RAL-CR03 R61: Include the string ROM, space, RO space.
STFC-RAL-CR03 R61: anywhere in the fine tool. I put that one in explicitly. I also didn't want to have to be sort of sitting here, is this command? Okay, press enter, is this command.
STFC-RAL-CR03 R61: It's developing that level of trust. Yeah, yeah.
STFC-RAL-CR03 R61: Yeah, so I'm already using this to augment my merger review shift work. I'm working on getting controlled access to my emails, it's my next thing. And yeah, please think, what else could it do? And the code I've been using so far, is available on the local AI coding there.
STFC-RAL-CR03 R61: That's it for me.
STFC-RAL-CR03 R61: So I can battery for somewhere, yeah.
STFC-RAL-CR03 R61: Yeah, what do we have? We do have a locally hosted language model. We are looking at
STFC-RAL-CR03 R61: catching Compreh, through… the… through SCD, that would…
STFC-RAL-CR03 R61: be slightly more up-to-date, slightly better in terms of total memory, but otherwise very similar to what we're currently running, and it would likely run similar models. Models keep evolving, and what I would say is the
STFC-RAL-CR03 R61: The open source model site, of things. It's probably…
STFC-RAL-CR03 R61: Maybe half a year or so behind what the commercial models are, in terms of how…
STFC-RAL-CR03 R61: Fast and how well they process things.
STFC-RAL-CR03 R61: However, they are catching up.
STFC-RAL-CR03 R61: It was definitely more than a year behind not too long ago.
STFC-RAL-CR03 R61: And you can see how they're catching up with a lot of speed.
STFC-RAL-CR03 R61: so we're getting a machine ready to run this as an explicit host for it.
STFC-RAL-CR03 R61: Rather than stealing one of the attacks.
STFC-RAL-CR03 R61: It's got web interface if you want to play with it, as in talk to your local
STFC-RAL-CR03 R61: Agent, whatever. Then you could talk to
STFC-RAL-CR03 R61: We haven't done much more. I mean, Tim is actively pursuing this, because there are benefits to it, but a lot of people are also hesitant about it.
STFC-RAL-CR03 R61: But I'd be happy to hear feelings about it. I'm more hesitant to ask co-finder about things, but…
STFC-RAL-CR03 R61: Yep.
STFC-RAL-CR03 R61: Why don't we have more? Tim has spent a full day, setting up, so you saw earlier, to run against the local agent, and to do a bit of apps coding standard, and…
STFC-RAL-CR03 R61: all of that, but it was just one day. However, it's taken the day out of his life, so he needed to catch that back.
STFC-RAL-CR03 R61: I've probably spent 2 days total around the whole hosting setup.
STFC-RAL-CR03 R61: A lot of that was figuring out which model, what parameters, which machine, Docker, finicky bits, whatever.
STFC-RAL-CR03 R61: And… I think bottom line is if we do want more, if we are keen to try more.
STFC-RAL-CR03 R61: then we'll need to put more time into it. When, what, this is basically up to you.
STFC-RAL-CR03 R61: But I figured this would start the discussion.
STFC-RAL-CR03 R61: There are some links?
STFC-RAL-CR03 R61: Hug and Face is the repository for open source large language models, and a lot of things around
STFC-RAL-CR03 R61: what the modern world calls AI, and we should be calling large language models.
STFC-RAL-CR03 R61: Unstop, in particular, is… I keep wanting to say a company, but I'm not quite sure.
STFC-RAL-CR03 R61: But they produce a lot of the open-source quantized models that exist out there, and they tend to apply fixes to the models as delivered by the person originally owning it.
STFC-RAL-CR03 R61: Quite often, there are problems with the communication template, as in how we feed things into the model, and Unsolved is one of those groups that will fix those and make it work.
STFC-RAL-CR03 R61: They have a lot of articles around how to train, how to fine-tune, how to use RAG, whatsoever.
STFC-RAL-CR03 R61: Lots of time being spent reading that, if you want to.
STFC-RAL-CR03 R61: then graph, that is what Tim just showed.
STFC-RAL-CR03 R61: There is… I put a link in there for an example of LaneGrath together with OpenWeb UI. We can look at what the web interface for the LLM looks like, which is Open Web UI.
STFC-RAL-CR03 R61: But in principle, it doesn't need to be just the language model. You can have a whole agent behind your chat interface, and talk to the agent, and give it the resources that it needs to do your work for you.
STFC-RAL-CR03 R61: Open code running in Docker, sorry, that's not linked yet, but that is a link.
STFC-RAL-CR03 R61: I have been using OpenCode a lot at home, But they could work.
STFC-RAL-CR03 R61: And I've started to encapsulate it. This has been on a chat that we have in PVD Teams, the AI coding channel.
STFC-RAL-CR03 R61: the suggestion is that if you wanted to use a coding agent, or if you wanted to use your LLM together with something that is basically a coding agent in a box.
STFC-RAL-CR03 R61: then, as Tim just said, you want to avoid it being able to access all your files.
STFC-RAL-CR03 R61: So the easiest way to do that is run it in a Docker container, only target one specific project folder, whatever it does in there, if it's your fresh git checkout in a new folder, it can recap off on that, you kill the thing.
STFC-RAL-CR03 R61: You delete the folder, you start fresh.
STFC-RAL-CR03 R61: So that would be the worst that happens. And I'll find that the Dopplerized version doesn't run any worse than anything else.
STFC-RAL-CR03 R61: So… That's fun.
STFC-RAL-CR03 R61: Have people seen this?
STFC-RAL-CR03 R61: Who has looked at self-hosted model?
STFC-RAL-CR03 R61: 1, 2, 3, 4, 5, 6.
STFC-RAL-CR03 R61: Okay.
STFC-RAL-CR03 R61: This is where I'm like, huh?
STFC-RAL-CR03 R61: We… can we take a question in the meantime? Yes, Brish, go ahead.
STFC-RAL-CR03 R61: Hello?
Brij Kishor Jashal: Hi. Yes. Okay, I mean, I can wait for the demonstration. Please, finish it, and then I'll come back.
STFC-RAL-CR03 R61: You don't have to focus.
Brij Kishor Jashal: Alright, so while you're preparing, just, I mean,
Brij Kishor Jashal: Thank you, Jens and Tim, for preparing this at such short notice.
Brij Kishor Jashal: So, before I come to my question.
Brij Kishor Jashal: I just wanted to mention that the idea behind this Let's Talk It AI series of seminars is to only share and discuss ideas and ongoing work, even if it isn't fully formed yet. The goal is to encourage honest conversation.
Brij Kishor Jashal: And, help us better understand PPD's engagement and interest in this area.
Brij Kishor Jashal: So, if you have any ideas for the next seminar which you'd like to share with us, then please do let us know, and then we'll book you in. Now, my question is,
Brij Kishor Jashal: from the experience of hosting this local LLM in our local infrastructure.
Brij Kishor Jashal: I just wanted to ask, did you try enabling MCP agents which can fetch?
Brij Kishor Jashal: Information from internet directly?
Brij Kishor Jashal: Or they're… these are for fully enclosed LLM models, which we have hosted.
STFC-RAL-CR03 R61: So the LLM model that is hosted runs without any connection to the outside world. It's purely internal. However, what you've seen, what Tim has shown us, uses other tools to run the queries that the LLM might want to run.
STFC-RAL-CR03 R61: So if you're setting it up in a way that we present it with tools, and these tools can then access the internet.
STFC-RAL-CR03 R61: Then, yes, it could fetch information from the internet.
STFC-RAL-CR03 R61: Yeah, we're being very careful about this. I think we're being quite slow compared to maybe other people out there.
STFC-RAL-CR03 R61: But you want to be mindful of what it might do if you get it.
STFC-RAL-CR03 R61: just an HTTP.
STFC-RAL-CR03 R61: interface.
Brij Kishor Jashal: Yep.
Brij Kishor Jashal: Toop.
Brij Kishor Jashal: Then, a second question I had was, in terms of storage, I mean, with your usage of it, what… I mean, how do you see a single user with…
Brij Kishor Jashal: Bitter.
Brij Kishor Jashal: for example, having… coding something in Athena, scale of software stack.
Brij Kishor Jashal: How much storage, in terms of the context, we will need to have in our infrastructure, and how it is going to grow, just to…
Brij Kishor Jashal: Just to see, I mean, how we… yeah.
STFC-RAL-CR03 R61: I mean, cold storage is one thing, I guess, and Athena is, Jim was saying it earlier, a few gigabytes.
STFC-RAL-CR03 R61: Only feeding little bits at a time. Yep. Context windows. For large language models, the…
STFC-RAL-CR03 R61: Decent coding models have somewhere around 250,000 up to a million tokens. Nemotron is technically not a coding model, the thing that we're currently running, and it's got a million token input window.
STFC-RAL-CR03 R61: I am quite sure that when you feed it more than 30,000 tokens.
STFC-RAL-CR03 R61: You will see the quality of the output degrading.
STFC-RAL-CR03 R61: So there's a… there's an interesting question whether we need to run a different model with smaller context window, but with much better coherence up to the limit of the context window.
STFC-RAL-CR03 R61: Other models are slower, depending on which model we choose.
STFC-RAL-CR03 R61: But we haven't really had that conversation, like, people haven't come up and said, could you do this, or could we host that? So it's worthwhile bringing this out here and saying, if you would fancy trying something, we could probably try something, it's just…
STFC-RAL-CR03 R61: In absence of having any input, I just run whatever I feel like. You can then sell me off, or it won't work.
Brij Kishor Jashal: Thank you.
STFC-RAL-CR03 R61: That here is the chat interface, so if you just go to havex16.p.rl.ac.uk on port 8000, you've got a non-encrypted web interface.
STFC-RAL-CR03 R61: there's bits that I need to fix about all of this, which is why it's not quite public.
STFC-RAL-CR03 R61: But it is the response mechanic that will chat to you.
STFC-RAL-CR03 R61: If you look at the same thing, same web interface.
STFC-RAL-CR03 R61: but on what is called the OpenAI interface.
STFC-RAL-CR03 R61: Did I misspelled?
STFC-RAL-CR03 R61: Oh, okay.
STFC-RAL-CR03 R61: Then, for example, you'd get JSON replies for what is currently being hosted. OpenAI, at some point, went in and defined a standard
STFC-RAL-CR03 R61: weapon interface based on GET and…
STFC-RAL-CR03 R61: host methods to be able to send things to the web server and get the price back.
STFC-RAL-CR03 R61: all of it encoded in JSON. So when you're interacting with a model like Tim does, then what's happening is you're using that web interface to chat to
STFC-RAL-CR03 R61: It runs metrics in the background, which we can now see how much Tim has been using this. Not quite, because it doesn't really keep user sets.
STFC-RAL-CR03 R61: But we've processed… 280,000 prop tokens, which isn't a lot.
STFC-RAL-CR03 R61: Much of that might be nuclear state.
STFC-RAL-CR03 R61: Yep.
STFC-RAL-CR03 R61: Chris.
STFC-RAL-CR03 R61: Hopefully I'm safely far enough away from you to ask this question. Can you just give a brief
STFC-RAL-CR03 R61: summary of SDFC policy on this? No, no. So what I went through when I asked the I about this is that
STFC-RAL-CR03 R61: Chris, who was very nice, said he can forward me to information security, and they can talk to me about posting language models or something, which they really can't they? What they have done is they have vetted Copilot, and in particular, Copilot checked.
STFC-RAL-CR03 R61: Which means we can use Copilot when we have the appropriate licenses to process work-related documents.
STFC-RAL-CR03 R61: And that is guaranteed to keep them safe, as far as physical sex is concerned.
STFC-RAL-CR03 R61: They haven't done much else on the AFib.
STFC-RAL-CR03 R61: There is people in SCD who run a small
STFC-RAL-CR03 R61: So, a small little stamp holder.
STFC-RAL-CR03 R61: Which is a line completion model, so when you're in your editor and you're starting to type a line, and it then suggests, oh, you might want to finish it like this, because I know your variable names, and I know what you're doing, so I'll just suggest it to you, and then you press tap, and it's done.
STFC-RAL-CR03 R61: They run that model over an SCD, they're not running anything bigger. I think you asked them about it, there was zero response, really.
STFC-RAL-CR03 R61: So they haven't come back and said, oh, if you're hosting this, we'd be keen to try. Nothing.
STFC-RAL-CR03 R61: So we've asked around in various departments, not much came back.
STFC-RAL-CR03 R61: They have then… so these guys in SCD have said that they've done a risk assessment for Starbuilder, which is a… an AI something or other risk assessment.
STFC-RAL-CR03 R61: I have started filling one out and sent that to the, primary source of all risk assessments, which is Justine Manino in SCC.
STFC-RAL-CR03 R61: See, she had a look at it, turned it into an Excel spreadsheet, made it really complex using ChatGPT, added lots of solutions using ChatGPT, which I then broke down into what I'm going to do for now, because a lot of these things sound very intelligent, but also make a lot of work.
STFC-RAL-CR03 R61: And don't really have much benefit.
STFC-RAL-CR03 R61: So, there is a risk assessment for this currently running in SCFC in PPE.
STFC-RAL-CR03 R61: as a playground. If we wanted to turn this into more, we might need to look at book success again, but by and large, nobody objected to it.
STFC-RAL-CR03 R61: Yeah.
STFC-RAL-CR03 R61: Well, UKRA has banned DeepSeek.
STFC-RAL-CR03 R61: Yes. So that was the only thing that came back from InfoSec, I think. They said, can't be done. Which is odd, because it can be run locally without sending things to…
STFC-RAL-CR03 R61: Well, and yeah, and any locally run model, as far as information security is concerned, it's none of their business.
STFC-RAL-CR03 R61: If you know that it doesn't send things out because you're hosting it yourself in an engine that you know doesn't send requests to the outside world, then…
STFC-RAL-CR03 R61: Whatever. Your problem.
STFC-RAL-CR03 R61: However, it is found not one, because politically it's challenging, and this is the exact same problem that we have with models like Plan 3 Bogger. If you ask the question that I just typed into Nematron about Germany in 1989, if you change Germany or China, the reply in Nematron is very good, and it summarizes what happened.
STFC-RAL-CR03 R61: If you ask the same thing in 23 Polar, it suggests that you might be violating law or committing a crime by
STFC-RAL-CR03 R61: insinuating something, and you're literally just asking a question, but because you mentioned China in 1989 in one sentence, it will right away stop processing your request and tell you to go
STFC-RAL-CR03 R61: Let me do something.
STFC-RAL-CR03 R61: Stuart has had lots of fun trying to make it speak to him in young big pentometer, and therefore forcing a reply that was actually quite accurate. However, it was so twisted that the model couldn't quite identify what was happening and just deliver the factual reply, rather than
STFC-RAL-CR03 R61: Trying to be stopped from it.
STFC-RAL-CR03 R61: so there are questions like, would we run deep-seq?
STFC-RAL-CR03 R61: Don't There are so many open source models that derive from DeepSeek that you… it's very hard to…
STFC-RAL-CR03 R61: say, you're running a model that has never been touched by anything Chinese, so…
STFC-RAL-CR03 R61: And I don't think we should, because they're really good at trying to push about racism. So there's… but that's politics. I don't know if they want to be too involved.
STFC-RAL-CR03 R61: There's a risk assessment for it, we should say, for that. Go for it.
STFC-RAL-CR03 R61: Totally.
STFC-RAL-CR03 R61: Because, we are running into my free area.
STFC-RAL-CR03 R61: AI models like LLM for project management. So, if we download…
STFC-RAL-CR03 R61: one of the open source, do we need to do some sort of advertisement? For example, we don't follow any internet connection to that AI model that is downloaded into MyQuery.
STFC-RAL-CR03 R61: So, we should have a separate conversation about that. In principle, you need to do a risk assessment for any activity that you do in this organization, because that's what we strive to do.
STFC-RAL-CR03 R61: That's what we're here for. But, yes, it definitely helps to not open ports to the outside world, which I think you will find hard to do within the organization, and if you do open ports on your computer, somebody will find out and eventually tell you off.
STFC-RAL-CR03 R61: Which this, on the…
STFC-RAL-CR03 R61: On HEPAC, we do see scans happening that find reports that are open and stop.
STFC-RAL-CR03 R61: Probably. So there's a question whether that is gonna throw somebody off or not soon enough.
STFC-RAL-CR03 R61: The other question is, what are you running? Why wouldn't you run against this model, as in centrally?
STFC-RAL-CR03 R61: Because this runs hot.
STFC-RAL-CR03 R61: Whatever hardware you have.
STFC-RAL-CR03 R61: I'm pretty sure this runs faster.
STFC-RAL-CR03 R61: And with larger context, larger… larger parameter space, unless I'm missing something that's happening for the project officer.
STFC-RAL-CR03 R61: But we should have a conversation about that.
STFC-RAL-CR03 R61: I heard…
STFC-RAL-CR03 R61: Two questions. One is, like, so when you're loading the Nematron, it's pre-trained, and my point is, like, you're using an infrastructure which is probably not as simple to…
STFC-RAL-CR03 R61: like, I don't know, if I was loading the model onto a GPU, I would have longer… I'd take longer than 2 days to load this model. So, is this something that's already provided?
STFC-RAL-CR03 R61: Just from the run commands on… run, run computation on the GPU. So, computation on GPU. At home, I run these models on a… on two RTX 56 GTIs with 16GB each.
STFC-RAL-CR03 R61: And they easily run Nemotron in a forward organization with high precision input.
STFC-RAL-CR03 R61: Processing, so it's 24 gigabytes or so total size.
STFC-RAL-CR03 R61: And with 130K context, and that runs as fast as here.
STFC-RAL-CR03 R61: Here, we're running on two A100s with 80GB of ATM memory.
STFC-RAL-CR03 R61: So they should be faster, however, their previous generation, so in terms of how they process
STFC-RAL-CR03 R61: one by itself, they're just not quite understood. But how do they use the… like, is there internal settings that you can just specify? Like, I'm giving you this GPU, or you have this many GPU cores, this compute capability, and it automatically adjusts? So, with, normal CPP, which is the go-to solution for lots of people.
STFC-RAL-CR03 R61: If you want to have fine brain control.
STFC-RAL-CR03 R61: You can tell it which layers to offload to which GPU.
STFC-RAL-CR03 R61: You can tell it to keep the cache in system memory or put that on GPU.
STFC-RAL-CR03 R61: you can… yeah, there's all sorts of tweaks you can do. It gets infinitely complex. I haven't done any of that. I've literally just set it, you've got two GPUs.
STFC-RAL-CR03 R61: Fire away.
STFC-RAL-CR03 R61: Here's the problem.
STFC-RAL-CR03 R61: One of the things that I start noticing is because it puts the key value cache on one of the graphics cards. One of the graphics cards has 60GB of VRAM use, and the other one's only got 40GB of VRAM use. So we could, in principle, run more queues. At the moment, we've got two parallel queues for this model.
STFC-RAL-CR03 R61: So two parallel requests can run.
STFC-RAL-CR03 R61: We could, in principle, run more if I started doing things like moving a larger fraction of the layers to the second GPU to keep more KP cache on the first GPU.
STFC-RAL-CR03 R61: The second question that I had was related to the original goal, which is to have an interface running with VS Code, which I tried. Is that already solved?
STFC-RAL-CR03 R61: Many lots, hopefully. In the AI Coding Teams channel, there's shared files, there's a DS Code config, which I think I updated.
STFC-RAL-CR03 R61: Problem was there was a…
STFC-RAL-CR03 R61: the annual version and the JSON version or something, and I don't know whether both of them are updated or not.
STFC-RAL-CR03 R61: Because at some point, we changed the model from 3 totaler to Nematron.
STFC-RAL-CR03 R61: Yeah, it depends.
STFC-RAL-CR03 R61: Oh, I've used it with,
STFC-RAL-CR03 R61: a YAML configuration for VS Code, and it works, and it's nice. Does that do a bit of line completion stuff? It does that. So you have to ask your queries and stuff like that. Line completion are technically… they're slightly different in how they run.
STFC-RAL-CR03 R61: And I think the configuration isn't set up
STFC-RAL-CR03 R61: To auto-complete through this model, because it's a very costly model to run, and line completion you can do with smaller models.
STFC-RAL-CR03 R61: So what we could do is, given that this already risk assessment for it, we could take stop order 2, which is a 2 billion parameter model, and we could load that on the same GPUs.
STFC-RAL-CR03 R61: And just run that as the OpenComplete form. If you run in the same instance, it would serve a different purpose, so therefore, through the OpenAI interface, you ask for a specific task to be done.
STFC-RAL-CR03 R61: And embedding would be one of those, autocomplete would be another.
STFC-RAL-CR03 R61: Yeah, I just want to make a comment, Elaine, to say,
STFC-RAL-CR03 R61: the stuff that you've been doing is really, really nice, and that, even I'm just starting to learn what you can do with these large language models, and I mean, it was Jacob just showed me, I've sat with him, and he showed me how he refractured
STFC-RAL-CR03 R61: you know, built a whole new code network, and they all use large language models, which typically you have to pay for, as we've seen, like, Gemini, they'll entice you in with some tokens, and then they run out core part. It's all very expensive, however, what we have here in the deployment is something that's free, and okay, it's not exactly the same, but
STFC-RAL-CR03 R61: I've not been able to tell the difference at the level that I've been working at, but, these large algorithms will be able to plug into many, many things, which I think we're all going to start to explore and
STFC-RAL-CR03 R61: I think it's very, very valuable. If you want to use things like phone call or timer properly integrated, then you will find that they are better at
STFC-RAL-CR03 R61: Following your instructions and delivering something functional on press present.
STFC-RAL-CR03 R61: Cloth is insanely extensive. There's a huge question around fairness to me.
STFC-RAL-CR03 R61: Some people in the department can afford… I can afford to buy a computer for myself, right? Or I can run it at home.
STFC-RAL-CR03 R61: other people can afford to pay for a globe license. And we shouldn't end up in a situation where somebody in a corridor has the ability to afford a globe code license, and therefore they can do three times as much coding work as other
STFC-RAL-CR03 R61: And then get away without having to lap cleaning, because it's all good calling. We want to make sure that we understand, as a department, how we're engaging this.
STFC-RAL-CR03 R61: And if the decision is to just paint more coal buyer, actually, by me.
STFC-RAL-CR03 R61: It's just a number and spreadsheet. But we need to make the choice, and I think
STFC-RAL-CR03 R61: at least from my end, I would say we have more appetite in understanding the mechanics of it and doing more with it.
STFC-RAL-CR03 R61: Rather than just using what we did.
STFC-RAL-CR03 R61: But, I mean, I would also say something more, I mean, my perspective, it's not just a number in a spreadsheet. I mean, you were saying six… you were saying six figures that… Yes. …for a department? Yes. Can we do better? And I think if we can… So for our department, we're talking 30,000 pounds a year.
STFC-RAL-CR03 R61: Right. A year, and… Which is a budget to refer black equipment… Exactly. So, whatever, so…
STFC-RAL-CR03 R61: There is the other stuff that I've not tried Copilot for, but it does have the potential to
STFC-RAL-CR03 R61: you know, read your email for you, and flag up stuff that should be done, write documents for you, given other stuff, summarize stuff. Does this have the potential to do that? I know you said something about… I have certainly done text into this.
STFC-RAL-CR03 R61: Not saying that was in any way related to raisins or something. I said, my English isn't good enough to get the tone of this ride.
STFC-RAL-CR03 R61: please fix this. And it does, and then when I read it, I realize what it's done, and it does read better. It's just that I couldn't naturally come up with this.
STFC-RAL-CR03 R61: So, yes, it does text processing to a level where it's really being helpful.
STFC-RAL-CR03 R61: You can dump some types of input files into the web interface.
STFC-RAL-CR03 R61: One of the things that I try doing with it is make it my keypad for me.
STFC-RAL-CR03 R61: So I wanted a basic silicone drift detector design.
STFC-RAL-CR03 R61: And rather than starting to write the lines of TCAT myself, I just say, make me a silicon drip detector design, and by the way, here's the Centaurus device.
STFC-RAL-CR03 R61: Manual appendix on the commands.
STFC-RAL-CR03 R61: Use these.
STFC-RAL-CR03 R61: If you don't give it the command manual, it perfectly well hallucinates you a file that has nothing to do with said false device. However, if you do give it the manual, the commands work, and the producer structure isn't perfect, it's not good.
STFC-RAL-CR03 R61: What notes?
STFC-RAL-CR03 R61: But by and large, if you give it within its context, the right input to process your request, it does things. And so, attaching a text file with apps coding standards or something, and then saying, read me this C++ file and tell me whether it complies, will definitely work.
STFC-RAL-CR03 R61: And we have a question on Zoom, because it's waiting for a long time. Yes, go ahead.
Tim Adye: Yes, I, I just…
Tim Adye: I was going to ask about the, use of containers, to make this, rather safer.
Tim Adye: So, how… how have… has this been… have you… have you used this already? Have you tested this? I'm thinking about how you… how you get the commands into the container and then out again, whether you could… the things that Tim was describing were,
Tim Adye: I guess we're not done in the container, because you… you want… you don't want to.
STFC-RAL-CR03 R61: You would pull a standard Ubuntu Python container or something, you would mount a subpoer that has your script in it, you would then tell the container to run the first command, or you would run it interactive.
STFC-RAL-CR03 R61: And then just run that command. The thing that I've done with OpenCode is I've used… there's a repository that dockerizes OpenCode, which, again, is a sort of agentic coding system.
STFC-RAL-CR03 R61: That just needs an LLM in the back of it.
STFC-RAL-CR03 R61: it can make mistakes, it browses through your folder hierarchy and deletes other projects and stuff, and it's definitely all been seen. So, you want to containerize that. People have done that.
STFC-RAL-CR03 R61: You literally just call it in its authorized version on a folder where you want to run it.
STFC-RAL-CR03 R61: And the Docker container will start to mount that folder as the workspace, and you can then run on that workspace.
STFC-RAL-CR03 R61: yeah, I thought, for a lot of the stuff, you can mount all the stuff read-only.
Tim Adye: And only the bit you want it to modify, could… or could be alone.
STFC-RAL-CR03 R61: The agenda is to make it do stuff, so you want to allow it to edit some files.
STFC-RAL-CR03 R61: And this is where you would mount the one folder that you need, that
STFC-RAL-CR03 R61: It's maybe a temporary copy of what you're trying to work on.
STFC-RAL-CR03 R61: And then run And yes, we've got first examples of how to run that, so happy to try.
STFC-RAL-CR03 R61: If you wanted to try.
Tim Adye: Yeah, I was actually, I mean, struck by Tim's example, where actually you don't need any modifications there. It's… it's asking… I mean, you need modifications internally to check out the right,
Tim Adye: The, the right,
Tim Adye: merge request, but after that, it's just a… it's a matter of searching the existing code, so that could all be mounted read-only.
STFC-RAL-CR03 R61: Yeah, that's it, yeah.
Tim Adye: But… and I think probably there's a lot of cases where you want it to suggest things to improve, and so it can do that in its own little sandbox, and then you, actually make the changes.
STFC-RAL-CR03 R61: It can do that in Sandbox, it can even do the git commits for you with appropriate messages, and then file a merge request.
Tim Adye: Yeah, that's where I start to get a little nervous about it.
STFC-RAL-CR03 R61: But then it depends on API tokens, how you allow Git to interact with the repo, with the GitLab repo.
STFC-RAL-CR03 R61: So that you do actually get a virtual request in, rather than a forced push that against the whole reboot.
Tim Adye: Yup.
STFC-RAL-CR03 R61: So I think that there's multiple ways of attacking this so that you're isolated from the world in a way that it can't do harm.
STFC-RAL-CR03 R61: Yep.
STFC-RAL-CR03 R61: You Only Live Once is a phrase being tossed around a lot with large language wrong.
Tim Adye: So I'm just wondering whether there's… whether we can,
Tim Adye: If we're… if we're worried about, kind of, risk assessments and that sort of thing, whether we can.
Tim Adye: make a code of practice, or a kind of, example, so that everybody… so people using this can set it up, and, people, who have thought about the… the dangers can,
Tim Adye: can… can set up, or help set up something like this. I'm wondering if we all… if we all do this with,
Tim Adye: with containers, are we going to all come up with our own solution, as you suggest, but then, perhaps I won't spot the,
Tim Adye: the clever way that the, AI could, could delete all my code, or could, submit a merge request that, that would, cause great embarrassment.
STFC-RAL-CR03 R61: we started the repository group on GitLab last week, when Tim had put to submit into it. Since then, nobody's done any
STFC-RAL-CR03 R61: well, Tim's done more work on his version, but, like…
STFC-RAL-CR03 R61: People haven't started looking at this, so this is the first iteration.
STFC-RAL-CR03 R61: I'd be happy to say we can make a,
STFC-RAL-CR03 R61: a containerizer repo or something, and start talking about, or we can add a doctor file to Git's repo, Tim's repo, and,
STFC-RAL-CR03 R61: start talking about how you would run that using Docker.
STFC-RAL-CR03 R61: it's just, we haven't gotten there yet.
Tim Adye: Yeah, yeah, I was just suggesting, yeah, that would be a nice thing, and I'd be interested to… to help work with that… work on that.
STFC-RAL-CR03 R61: Cool.
STFC-RAL-CR03 R61: This is a question, right, for you and Chris, I mean.
STFC-RAL-CR03 R61: Obviously, it doesn't address everything, but does it make sense to get an OLAMA server running centrally on one of the machines, rather than everyone else having to set up the server, and then people can access publicly accessible
STFC-RAL-CR03 R61: So, at the moment, the link that I showed live earlier, that is accessible from within PBT Network.
STFC-RAL-CR03 R61: people in TV have tried and have failed, so I'm assuming it's only visible within PBD network?
STFC-RAL-CR03 R61: No, right, FCD and DI would be able to get to it, but yeah. We had others. But it's… it's central, Manny uses it, I use it, Tim uses it, we're not running multiple links.
STFC-RAL-CR03 R61: And we could expose it quite easily. That's a specific model, or is it accessing lots of them?
STFC-RAL-CR03 R61: It is… currently a specific model, because I know I've got one.
STFC-RAL-CR03 R61: We have had issues when you load more than one model, so in principle, you can make more than one model available, but then I would need to figure out more of the mechanics of model loading and unloading automatically.
STFC-RAL-CR03 R61: Which I hadn't done yet. I think that's why Olama might be worth looking at, because it does all that, I think.
STFC-RAL-CR03 R61: In some way. Yes.
STFC-RAL-CR03 R61: Ish. Do you do this? Yeah.
STFC-RAL-CR03 R61: Okay.
STFC-RAL-CR03 R61: But then there's this idea of this machine in SCD.
STFC-RAL-CR03 R61: Right.
STFC-RAL-CR03 R61: They would run it on a slightly bigger machine, and not one of our repurposed DPU nets, but…
STFC-RAL-CR03 R61: So, for the others who are interested, the Llama and LM Studio are two things that sit on top of Llama CPP, which is the basic sort of open source repo that runs transformer time.
STFC-RAL-CR03 R61: large language models.
STFC-RAL-CR03 R61: And they… they add features around it. They strip out some of the complexity of configuring it.
STFC-RAL-CR03 R61: yep.
STFC-RAL-CR03 R61: They do usually run on your normal desktop machine. They're slow when you don't have a GPU.
STFC-RAL-CR03 R61: If you've got a decent old-hand driver for your integrated GPU, then you might get a bit more out than if you're just running on CPU.
STFC-RAL-CR03 R61: If you have a Mac.
STFC-RAL-CR03 R61: with…
STFC-RAL-CR03 R61: say, an M4 Max and lots of unified memory, then you can get quite decent rates out of it.
STFC-RAL-CR03 R61: Got it.
STFC-RAL-CR03 R61: Med volt, especially.
STFC-RAL-CR03 R61: Thanks, yeah, that's a nice talk.
STFC-RAL-CR03 R61: A few weeks ago, I went to the States to see what they're doing with agentic AI. Yeah. So Trump has pumped in a few hundred million dollars into the Genesis program at the Navy, and they want to create an AI-enabled workforce.
STFC-RAL-CR03 R61: In the States, the labs are, within the next few months, going to be preparing demonstrations
STFC-RAL-CR03 R61: uses of digantic systems.
STFC-RAL-CR03 R61: I think we, as the facilities here, will also start to use them mainly as co-pilots, and they have their uses right in documentation and things like this.
STFC-RAL-CR03 R61: But is there any… push towards…
STFC-RAL-CR03 R61: understanding the true cost of these systems. Like, we've heard that they're free, but they're not free. Yeah, yeah, there's a lot of questions. So, look, to first order when these things write code for you, when they're not.
STFC-RAL-CR03 R61: When they're not clawed on the latest revision or something, they tend to produce code that is of the quality of a reasonably junior person.
STFC-RAL-CR03 R61: making first steps at writing their own code or something. It feels a bit…
STFC-RAL-CR03 R61: Like, it needs a senior person to throw ice over it. So what it does replace is what I would call a typical PhD student.
STFC-RAL-CR03 R61: And that's where it becomes worrying, because you want to teach PhD students to know how to write code, before we then let them use the details.
STFC-RAL-CR03 R61: And if we…
STFC-RAL-CR03 R61: Look at funding prices that may or may not be impeding on us, and we start asking
STFC-RAL-CR03 R61: Can we afford to pay PhD students, or do we just run our local AI to do our coding for us?
STFC-RAL-CR03 R61: And that question gets answered in the wrong way, then we're killing our own future.
STFC-RAL-CR03 R61: So yes, there are intriguing questions around this. For facilities, I'd say that we want to be able to understand what is the cost of,
STFC-RAL-CR03 R61: running some model versus the cost of employing someone to do the same, right? Yeah.
STFC-RAL-CR03 R61: And then, what is the benefit, and all of this stuff. So, if we're at the early stages now, and you were talking about working with SCD, I would envision some SCD-hosted facilit… some, you know, something for the facilities. Given that we're also, as a country, pumping
STFC-RAL-CR03 R61: a few hundred million or so into AI something or other monsters. Not sure we'll get it, but yeah. There must be resource somewhere to run these things slightly more efficiently than we do, yeah. But I imagine in the future, the question is going to be, what's the, you know, what's the cost of running this thing? You know, how do I… how do I…
STFC-RAL-CR03 R61: run a facility with an agentic system such that I'm actually saving money rather than spending more in order to achieve the same performance. So, if I think about a department like SE, which may or may not be the department that has bought 300 co-pilot licenses.
STFC-RAL-CR03 R61: Then we're talking 100,000 pounds a year.
STFC-RAL-CR03 R61: co-pilot?
STFC-RAL-CR03 R61: It's gonna make a lot of those people who are in large buying software More effective at some things.
STFC-RAL-CR03 R61: But doesn't… what costs the environment as well, right? So, if we're supposed to be measuring our CO2, like, are we… Yes.
STFC-RAL-CR03 R61: My concern is that we see this as a
STFC-RAL-CR03 R61: a helpful tool, which it is, but we overuse it, and then, we could look at token generation costs per token. Well, they're usually measured in megatokens or something.
STFC-RAL-CR03 R61: So you can tell here that we've processed 270,000 input opens so far on this running instance. I don't know whether it was restarted, that's a major downside.
STFC-RAL-CR03 R61: But it's…
STFC-RAL-CR03 R61: typical chat sessions that I would run are 72,000 tokens or so. So, mega tokens is the sort of scale that could be established for energy use or something, and then we could try and evaluate how much is that? How much… how much would I be using in a week if I, every day.
STFC-RAL-CR03 R61: up through a million focus or something.
STFC-RAL-CR03 R61: I'm doubtful that's gonna…
STFC-RAL-CR03 R61: unique outcome. Not for sure, but I think it's… No, as in, we wouldn't like what it says. Not exactly, right? But I think it's something that we have to keep an eye on, right? Maybe not for PPD, for a facility, if we were spending X amount of money, fine, but if it adds to our footprint as well, it's really worth it.
STFC-RAL-CR03 R61: But… Yes. Great question. And maybe it's less… that is the other question, is adding a person.
STFC-RAL-CR03 R61: At the moment, you couldn't met the person, because you can't really hire.
STFC-RAL-CR03 R61: Powerful when it's very business-critical or something.
STFC-RAL-CR03 R61: So, at the moment, a go-to solution for fixing quite a few problems is to turn to AI.
STFC-RAL-CR03 R61: Responded on that to our environmental footprint, and there's other False.
STFC-RAL-CR03 R61: It is something where we can't actually spend on.
STFC-RAL-CR03 R61: Okay, well, the chance.
STFC-RAL-CR03 R61: Person is…
STFC-RAL-CR03 R61: anywhere between 100 and 150,000 a year. There's still short-term thinking, but I'm talking long-term. If we roll out these systems, you know, quite heavily in facilities, which is what they want to do in places like the States, you know, are we going to have to add an extra data center just to keep our facilities running?
STFC-RAL-CR03 R61: So anyway, it's just a comment that maybe we should be aware of these things and think about them from the start. I think not, but we should… we could look at it. We could look at evaluating how much power is actually being consumed by this.
STFC-RAL-CR03 R61: I also… it depends on how high responsibility you want.
STFC-RAL-CR03 R61: Lots of these data center type applications, they're aimed around training these systems, making them better.
STFC-RAL-CR03 R61: And B, about response times. So that's a lot, I don't know. Google has explicit clones of Wikipedia to be faster than querying Wikipedia, which is why.
STFC-RAL-CR03 R61: Name sometimes. Takes seconds to run.
STFC-RAL-CR03 R61: Well, it's largely because Wikipedia query is actually slow, compared to when you've got a whole lot of Wikipedia stored on your favorite cluster, running 24-7, waiting for a request to come in.
STFC-RAL-CR03 R61: So I think a lot of the power footprint that we're talking about when we're talking big AI machines is this sort of super-fast response time for training. So, what I'm saying there on that point is that,
STFC-RAL-CR03 R61: we can… balance this, right? We can define what is a good balance, and then we can define what
STFC-RAL-CR03 R61: what we should be using, as opposed to these mixed systems that are off the shelf. Yeah.
STFC-RAL-CR03 R61: I have asked whether there is a policy in any shape or form. People have said, sure, there's policies, but nothing that can allow me to evaluate whether I want to run something or not.
STFC-RAL-CR03 R61: And so far, the conversation just hasn't progressed too long.
STFC-RAL-CR03 R61: Actually, however.
STFC-RAL-CR03 R61: is running training courses on how to make your own AI agents, so there's a… currently quite a disconnect between us and Artery in terms of what they're doing and what we're doing.
STFC-RAL-CR03 R61: I am finally talking to them on Friday.
STFC-RAL-CR03 R61: An hour.
STFC-RAL-CR03 R61: Where that goes, no idea.
STFC-RAL-CR03 R61: Yeah, it seems that they're looking at this as a way to write customer interaction agents.
STFC-RAL-CR03 R61: Because that's a two-digital application where you replace two people working at a call center with Computer.
STFC-RAL-CR03 R61: subsidies, just so much better.
STFC-RAL-CR03 R61: There's a few people in facilities I wouldn't tell you.
STFC-RAL-CR03 R61: So it'd be… it'd be nice to know if this made the round, so that we can have an actual coherent conversation across SAT, or even their life about it. But at the moment, whenever you prompt somebody about it, they just go, I've done this, fine.
STFC-RAL-CR03 R61: So it's not really a living discussion yet.
STFC-RAL-CR03 R61: Okay, let's thank… I don't see any questions, so let's thank him and Jens again.
STFC-RAL-CR03 R61: He wanted to look at some.
STFC-RAL-CR03 R61: Let us know.
STFC-RAL-CR03 R61: Presumably, you're on the outside as far as people.