STFC-RAL-CR03 R61: But…
STFC-RAL-CR03 R61: Right. So thanks for that introduction, very nice. Indeed, as you've probably heard from that, I focus a lot on anything to do with statistics and physics.
STFC-RAL-CR03 R61: And that is where I get super excited, so when they invited me to come and give a talk here, I thought, what can I talk about? I already mentioned to Tim that I had given a bunch of seminars about, unfolding, but with Tim in the room, I figured you all know everything about that already.
STFC-RAL-CR03 R61: Right? I also, as was mentioned, have been an expert for a whole bunch of years, and with future provider things going on, that's a topic that I talk about a lot, about what we should do for future providers, and what we should think about, and especially focusing usually about
STFC-RAL-CR03 R61: What don't get people need from these new collaborations and new experiments?
STFC-RAL-CR03 R61: But I thought instead I'll try to do a very new topic for me, not as a topic, but at the seminar topic, which I started calling Validation and Evaluation in Physics, meaning how to do statistics behind the geophysics, and how we need to change it from machine learning purposes.
STFC-RAL-CR03 R61: Now, while I was writing this, I decided, actually, I should have called it What We Don't Know and What We Did Wrong, so I'll try to focus a little bit about, in the history, when we did our statistics wrong, why it was wrong, and what we could have done and said, and how we changed our procedures afterwards.
STFC-RAL-CR03 R61: And with that, try to look forward to how we might need to change things for machine learning.
STFC-RAL-CR03 R61: I'm hoping that this can be a little bit more interactive, as in, if there's something you don't understand, please just interrupt right away, don't wait till the end of the jaw, at least take it right away.
STFC-RAL-CR03 R61: And as you might have realized, this is not a topic where the answers are very well known, so I can't really tell you what we have to do for machine learning, how we change the way we do our statistics, but I can point out a few directions that
STFC-RAL-CR03 R61: are currently being investigated, because again, as was mentioned in the introduction, I work a lot with actual professional statisticians, not people like me doing statistics in physics, but people do that for their entire career. And we try to figure that out together.
STFC-RAL-CR03 R61: They generally like working with us, by the way, because we have a lot of data, which means that our
STFC-RAL-CR03 R61: Anything other than statistical uncertainties, so systematic uncertainties that we usually call them, becomes very interesting, for them. So we have a lot of fun use cases for them to give some obscure, in their words, new tests for statistics.
STFC-RAL-CR03 R61: Alright, so…
STFC-RAL-CR03 R61: This explicit I really like. This is basically the basis of what we… how to do statistics and physics. It's… there's a bunch of data points, and usually we try to fix something to it.
STFC-RAL-CR03 R61: So this one kind of says, what does it mean when you do certain things? So, you know, when you do a linear line, and you might… you did say, I did a regression to still use machine learning terminology. You have a bunch of it where you say, okay, I wanted to do something sophisticated, so it looks sophisticated.
STFC-RAL-CR03 R61: I click smooth lines in Excel, and it just skinks every single point. It's probably about too many parameters in your bit there. Or, please don't look outside of the range of my plot, because things get really, really weird, but as long as you just don't look at this part, everything looks fine.
STFC-RAL-CR03 R61: Yeah, so this is sort of… which one do you then choose? Based on the data points that we see here?
STFC-RAL-CR03 R61: I personally think they all look pretty bad, as in, I really don't know which one is the best one, so that's why we need to figure out which one is the one you need to do. All you need to know physics and know why your model should have a certain shape.
STFC-RAL-CR03 R61: All right. So, I'm going to talk about physics analysis in a nutshell.
STFC-RAL-CR03 R61: I know you all know how to do this, but mostly such that when I say something later, you understand the terminology I use.
STFC-RAL-CR03 R61: So… There's one term here, which is simulation-based inference. It's currently very, very popular in machine learning.
STFC-RAL-CR03 R61: What we've been doing classically is basically simulation-based inference.
STFC-RAL-CR03 R61: We just haven't folded that. But the idea is, you have some theory.
STFC-RAL-CR03 R61: You do some simulation of some detector system, whichever detector you work for, and you get ahead, look at some data and compare the two.
STFC-RAL-CR03 R61: But then you do some inference, because you want to say something about the theory, depending on your data. So if this is simulation-based inference.
STFC-RAL-CR03 R61: The machine learning version of that is still called simulation-based inference, or machine learning-based simulation-based inference, and it's the same idea, it's just you've trained on which of these steps.
STFC-RAL-CR03 R61: Oh my god.
STFC-RAL-CR03 R61: So, if we add a little bit of the, of the terms, here we have some branches, usually called View.
STFC-RAL-CR03 R61: I'm going to use me for any parameter of your theory. I don't usually use it as the ratio between the number of events measured divided by the number of expected events, but it could be anything here. It's just your parameter of interest.
STFC-RAL-CR03 R61: Which comes from some model, which we call the standard model, usually.
STFC-RAL-CR03 R61: Then you have a detector, which means there's some function of X, given… which is your data, given these parameters mu and beta. You have some multicola sampling, so you get some
STFC-RAL-CR03 R61: data, this Monte Carlo data X, and then you have some observables. So this could be, for example, your PT
STFC-RAL-CR03 R61: Distribution, or whatever, the mass distribution that you put on the x-axis on the plot.
STFC-RAL-CR03 R61: So then you take the actual measured Y… Sorry, could you distinguish X and Y? Okay, so X is your actual data points. So this is the… for each point of your data, what the four vectors are, or each of each of its values. This is the histogram, or distribution, that you're looking at.
STFC-RAL-CR03 R61: So that's what I call an observable, and the S I call the data.
STFC-RAL-CR03 R61: So usually what we look at is our histograms wide, which are now real… filled with our real data.
STFC-RAL-CR03 R61: We have some likelihood of this distribution, given our parameters, and we try to infer another, this parameter of the model again.
STFC-RAL-CR03 R61: Okay.
STFC-RAL-CR03 R61: So, the question we usually try to answer is, does my hypothesis describe the data?
STFC-RAL-CR03 R61: So the basic question is what… what we want from the answer. Whenever you're trying to set up an analysis, this is the first thing you should do. What am I trying to learn from this, right? So you want to have a clearly understandable number, which is why we like things like p-values. It's one number, and it gives you a lot of all the information you maybe wanted.
STFC-RAL-CR03 R61: Then it's something, and this one is often, again, forgotten, but we would like to match it within point.
STFC-RAL-CR03 R61: If it says that something is very obvious, we would like to be able to see it, and if it says it's not very obvious, we would not like to be able to see it super easily.
STFC-RAL-CR03 R61: Just because it makes it easier for us to understand the number.
STFC-RAL-CR03 R61: And we would want to have a meaningful interpretation in terms of the likelihood.
STFC-RAL-CR03 R61: So…
STFC-RAL-CR03 R61: Basically, we get plots like this. I have a bunch of data, I generated that for this… for this talk, and you have some sort of background distribution, and you want to figure out, is there a signal here?
STFC-RAL-CR03 R61: Who thinks there's a signal here?
STFC-RAL-CR03 R61: Who thinks there's definitely not a signal here.
STFC-RAL-CR03 R61: Okay.
STFC-RAL-CR03 R61: So, it kind of looks like there might be a signal here.
STFC-RAL-CR03 R61: Around, well, whatever this parameter is, 30, maybe?
STFC-RAL-CR03 R61: It really does look like a signal, if I make this plot. It's more than 3 sigma.
STFC-RAL-CR03 R61: If I use this, this test here, likelihood of background, if I divide the background plus zero, what?
STFC-RAL-CR03 R61: So, I can already tell you it's not.
STFC-RAL-CR03 R61: it's… I generated it, it's completely bullshit, but I did very annoyingly draw
STFC-RAL-CR03 R61: through this line in here, and that tricks you. You told us there was a background. I told you that was my guess for the background, but it's… it's just a line for a background, so if you don't do your background right, it'll be a signal. But, yeah, there was nothing… there was no signal or anything.
STFC-RAL-CR03 R61: So…
STFC-RAL-CR03 R61: Then you wonder, do we… like, just looking at it, and then seeing… drawing the line, and then seeing how many thickness there are, it's maybe not a good idea, might be a bit dangerous. So, do we actually use that? Yes?
STFC-RAL-CR03 R61: We do, actually quite a lot. So I did see some… these are some not-too-recent examples from Atlas CMS. Here's one from Atlas, this is the diagon residents, where you have this plot, and you just get good agreement is observed by optical inspection of pools between the data and the background prediction.
STFC-RAL-CR03 R61: The word calls here just literally means the difference between… the part we see here, the difference between the prediction and the data points.
STFC-RAL-CR03 R61: Here's some examples from CMS. This test, optical inspection, shows good agreement.
STFC-RAL-CR03 R61: And good agreement about optical inspection between data and expectations were understandable.
STFC-RAL-CR03 R61: So we actually use this a lot. I very much do not recommend it.
STFC-RAL-CR03 R61: Optical inspection helps you, because you want to know that these numbers that you get out of your model actually seem to be reasonable.
STFC-RAL-CR03 R61: If you get something out that isn't reasonable, reconsider what you're doing!
STFC-RAL-CR03 R61: But, this is happening still quite often. We just say, oh, actually, looks pretty sad and blah blah blah. Main result is today, but…
STFC-RAL-CR03 R61: So here we have a bunch more signals. Again, I just drew some lines and some… some beads, so now you won't be tricked by me. None of these are, at all signals, these are all background fluctuations, basically.
STFC-RAL-CR03 R61: So if you want to know more about that, just look up this term, which is the look elsewhere effect. Basically, that means if you just look at somewhere and try to find a signal, you can usually find a signal.
STFC-RAL-CR03 R61: Just by manipulating your data.
STFC-RAL-CR03 R61: So you don't want to know exactly… unless you know exactly where your peak is going to be,
STFC-RAL-CR03 R61: you can usually find a peak. This is also why, in a lot of papers, we quote things like local p-value versus a global p-value. The local p-value is this, showing the significance of these numbers, the significance of each peak.
STFC-RAL-CR03 R61: Well, the global p-value corrects for the fact that you might just be inventing it, basically.
STFC-RAL-CR03 R61: If you're similar?
STFC-RAL-CR03 R61: They like these local p-values, because they can still say that there might be something there, and we are generally quite conservative in the experiments, so that's why we still called them.
STFC-RAL-CR03 R61: Alright.
STFC-RAL-CR03 R61: So, if we go back to this picture, We have…
STFC-RAL-CR03 R61: Right. I wanted… I've been skipping over this one parameter here, which is the SETA.
STFC-RAL-CR03 R61: Which is basically all systematic.
STFC-RAL-CR03 R61: So, where these uncertainties come from, so some of them obviously come from our theory.
STFC-RAL-CR03 R61: So we have the soft physics process simulation, the physics process simulation, there's the hard scatter, the proton structure functions, but also from the experiment and from Atlas, as you've heard, so I use a lot of Atlas pictures.
STFC-RAL-CR03 R61: So there's the Atlas detective simulation, or the detector reconstruction, which have a lot of uncertainties. I've put down 5 here, as probably you all know, that's usually more like 200.
STFC-RAL-CR03 R61: I went the wrong way again.
STFC-RAL-CR03 R61: So, I wanted to start with a fun discovery. So, this is the top quilt discovery at UA1.
STFC-RAL-CR03 R61: So they had a process, which is W to the top and a bottom, and the top to a B or U.
STFC-RAL-CR03 R61: There's a difference. So I ignore that. So their analysis selection was two objects, a charged electron, and some missing energy.
STFC-RAL-CR03 R61: This is the final plot, where here that you can see these lovely six events, so that's what they found. They found six, found 6 events.
STFC-RAL-CR03 R61: They plot the dopel mass against the BL mu mass, where the new missing energy.
STFC-RAL-CR03 R61: And the WMF was in the right place.
STFC-RAL-CR03 R61: So that gave them confidence. So they managed to put the top mass at 40 GV,
STFC-RAL-CR03 R61: For whoever forgot it from their school days. The correct math is 173 GRE, approximately.
STFC-RAL-CR03 R61: Yeah, so that turned out to be just some background events and some very creative selection events.
STFC-RAL-CR03 R61: They… this was actually before selection cuts, completely felled, and they just took a little band which… where the mass of the lovely would be in the right place, cut out everything else, also on both sides, and found the mass of the top cork.
STFC-RAL-CR03 R61: So, that is not a method that we want to use, and we have learned something from that.
STFC-RAL-CR03 R61: So, then now I wanted to mention this one, which is Wilt's theorem. Wilt's theorem is something we can still use every day in every analysis, but I don't think anybody realises that, so I thought I'd bring it up.
STFC-RAL-CR03 R61: So what is here? The idea is you have an H null, a null hypothesis.
STFC-RAL-CR03 R61: Then it basically says any additional parameters, as by some alternative hypothesis, are not needed. So, for example, you have the standard model without any explosion.
STFC-RAL-CR03 R61: Then, if a theorem is correct, then according to Wilk's theorem.
STFC-RAL-CR03 R61: You have a chi-squared which matches with your likelihood ratio.
STFC-RAL-CR03 R61: So this means that when you draw your light through the gray shadow.
STFC-RAL-CR03 R61: Which we tend to do like that.
STFC-RAL-CR03 R61: Right? These are sort of the parameters… the likelihoods that we're used to.
STFC-RAL-CR03 R61: Then you can draw a straight line across from this, and give your confidence interval at the place where these intersect.
STFC-RAL-CR03 R61: That's basically the idea of Rope Serum.
STFC-RAL-CR03 R61: Imagine the fact that this follows a chi-squared function is the straight line on this part.
STFC-RAL-CR03 R61: So, one of the things that's worked here in Omni applies for nested hypotheses.
STFC-RAL-CR03 R61: Meaning, if H0 is a first-order polynomial, and H1 is a second-order polynomial, it works.
STFC-RAL-CR03 R61: If one is the standard model, the other one is the standard model plus an export, it works.
STFC-RAL-CR03 R61: If… If they are not matching, H0 is the first order, depending on what H1 is an exponential.
STFC-RAL-CR03 R61: It doesn't work. It doesn't work at all.
STFC-RAL-CR03 R61: So your two hypotheses that you're testing
STFC-RAL-CR03 R61: Have to be in the same family.
STFC-RAL-CR03 R61: There's a bunch of others!
STFC-RAL-CR03 R61: Boundary conditions, one of which is that this point here, your minimum of your likelihood, falls inside your space of inside your parameter space that you're scanning.
STFC-RAL-CR03 R61: Meaning not from the edge.
STFC-RAL-CR03 R61: Meaning it can't be zeroed with where a negative number of events are not allowed. You're affecting zero events.
STFC-RAL-CR03 R61: This doesn't actually hold, because zero, your expected number of events, lies not inside the parameter space that you're scanned.
STFC-RAL-CR03 R61: And we generally don't test that, but basically what that means is that this line is not quite straight, so these boundaries that we choose are not by right.
STFC-RAL-CR03 R61: Worth checking. Oh, by the way.
STFC-RAL-CR03 R61: So, let's assume that it does work.
STFC-RAL-CR03 R61: If we're interested in a mu that maximizes those likelihood.
STFC-RAL-CR03 R61: We can use both theorem, and instead consider that the maximum of this likelihood, and then I say minus 2 of the log of the likelihood ratio.
STFC-RAL-CR03 R61: Is the NLL, which we call NLL.
STFC-RAL-CR03 R61: So, in the large N limit, is that a chi-squared distribution with K degrees of freedom, so then this thing, the maximum of the likelihood, is the negative log likelihood is a chi-squared.
STFC-RAL-CR03 R61: Oops, too.
STFC-RAL-CR03 R61: That's basically what I drew on the picture then, but I find it easier in pictures.
STFC-RAL-CR03 R61: Nope.
STFC-RAL-CR03 R61: So here it is, Dan.
STFC-RAL-CR03 R61: So, it makes it quite easy to construct this confidence interval.
STFC-RAL-CR03 R61: So, your confidence interval, which is this distance here, is then defined as this region that is below some value C.
STFC-RAL-CR03 R61: Okay.
STFC-RAL-CR03 R61: The threshold values of C,
STFC-RAL-CR03 R61: for different components, depending on the dimensionality of the two components interval. So, what I said here is that it has the chi-squared of dimension k, so if you are looking at a two-dimensional plot, rather than a one-dimensional plot, it actually changes.
STFC-RAL-CR03 R61: So, for one dimension, these numbers are 1 and 4 or 9, so that the normal
STFC-RAL-CR03 R61: confidence level that we actually quote. But if you're going, for example, to a two-dimensional one, make sure you adjust it to the chi-squared of the two dimension… for two dimensions. So these are actually at 2.3, 6.2, and 11.8.
STFC-RAL-CR03 R61: slightly less… nice, less round numbers. As the Statistics Committee in Atlas, we get this quite a lot, where people say, hey, our confidence intervals don't make intuitive sense, because we looked at the 1D version, and it worked fine, and then we looked at the 2D version, and it…
STFC-RAL-CR03 R61: Somehow, the confidence intervals didn't shrink or grow the way we expected them, because they forgot to do this. They forgot to put their limits in a different test.
STFC-RAL-CR03 R61: So, the other part is setting limits.
STFC-RAL-CR03 R61: So, an upper limit on parameter, in contrast to the screw-2-sided interval, but basically what you do is you take this NLL and you turn it to zero in the other half.
STFC-RAL-CR03 R61: And then the C is a little bit more tricky.
STFC-RAL-CR03 R61: Since this one follows a chi-squared only enhancement distribution.
STFC-RAL-CR03 R61: But you get that. You, for example, for a 95% confidence level, it goes to 2.85. So also then, change it a little bit.
STFC-RAL-CR03 R61: Park.
STFC-RAL-CR03 R61: So what do we do with these news as parameters?
STFC-RAL-CR03 R61: One way to look at it is for, for the new parameters theta, is to look at this post-impact of an individual nuisive parameter on the determination of you.
STFC-RAL-CR03 R61: So this is these annoying plots. I think everybody's seen these kind of plots before.
STFC-RAL-CR03 R61: And I think most people go over them and say, I guess it looks funny.
STFC-RAL-CR03 R61: I don't really know what's in there, and it has the bars that switch colors, and it's probably fine. What is… what happens, actually, in this law, is that every time you take the music's parameter, you fix it to plus 1 and minus 1,
STFC-RAL-CR03 R61: of the one sigma uncertainty on that user's parameter. So every user's parameter is added as a constraint term to the likelihood.
STFC-RAL-CR03 R61: So you set what your one sigma difference is for that user's bound to, you shift it up and down.
STFC-RAL-CR03 R61: You redo the entire regular fit, but with that one parameter fixed at a plus one or minus 1 sigma, and you see what… how the central value of your mu changes. So that's these…
STFC-RAL-CR03 R61: these dots, that's what we call the pull, usually. Now, this is one where I can already tell you it's very frustrating, because we use impact and pull and…
STFC-RAL-CR03 R61: 5 other words for this, and they are also mixed up, and they're not the same between Atlas and CMS.
STFC-RAL-CR03 R61: I'm hoping that we can get this sorted still before the end of the year. There's a document with a proposal that is currently being satisfied within Atlas and CMS.
STFC-RAL-CR03 R61: between logist this week, to make sure that the names stay correct, so look out for that. But that's just these black dots, so you want them to sort of not move too far away from zero, and the uncertainty's not too far away from these lines, which is the one before you move, your leases around to.
STFC-RAL-CR03 R61: So then, in fact, it's basically what happens to that uncertainty itself, and which direction does it pull in, switches your products.
STFC-RAL-CR03 R61: Okay?
STFC-RAL-CR03 R61: So, again, if things sort of… Don't look too crazy.
STFC-RAL-CR03 R61: We consider it a good enough uncertainty. Why do you do this? Why do we look at this?
STFC-RAL-CR03 R61: Well, we look at this because if they get really, really small uncertainties, or it gets pulled very far away, you probably are not including the right tone for that particular uses parameter. The right… not the right… and not including the right constraint.
STFC-RAL-CR03 R61: Either your data is giving you some more information about this solutions parameter than you previously thought.
STFC-RAL-CR03 R61: or you're actually misidentified how this character behaves. One of the ones where we see that quite often is what we call two-pointers planning.
STFC-RAL-CR03 R61: That's things like, we did this analysis with Pythia and with Sherpa, and the two do not actually match.
STFC-RAL-CR03 R61: Deciding what your one sigma away is, is pretty much impossible, because you just have two points in a parameter space, no way to know which one is right, or how they go between the two.
STFC-RAL-CR03 R61: So then sometimes it turns out that your data heavily savors one of those predictions, and it gets fooled quite a lot.
STFC-RAL-CR03 R61: And it's good to know that.
STFC-RAL-CR03 R61: Okay.
STFC-RAL-CR03 R61: Yeah, oh, this is the nomenclature I put in. After working with statisticians, I have to put it. We call a lot of things fitting, because we fit the branches of our model.
STFC-RAL-CR03 R61: The statisticians call this gram transformation, and it would mean something completely different from spitting.
STFC-RAL-CR03 R61: So, in case you want to know more about this, and you want to look up, like, hardcore statistics papers on it, you Google brand resolution, and not just shitting, otherwise you do not get this.
STFC-RAL-CR03 R61: So, and here you see the pre-fit and the post-fit, so you see these different colored bars, one of them is empty, one of them is full. Basically, no constraint means that the post-fit is not smaller than the pre-fit.
STFC-RAL-CR03 R61: And if this stays at 0, plus or minus 1, you have no problem.
STFC-RAL-CR03 R61: So, the other thing we look at are correlation matrices.
STFC-RAL-CR03 R61: They're ugly. You usually need a magnifying glass, especially when you get a lot of news disruptors to be able to see anything in them. What is twice as important? Well, a lot of correlation generally make your fit unstable.
STFC-RAL-CR03 R61: That is not necessarily a problem. It doesn't say that you're gonna find the wrong answer, at least.
STFC-RAL-CR03 R61: But you can do stuff about it. So usually change a basis can help you if you just change to a basis where you take, basically, between two correlated parameters, you say we take a new parameter, which is a fully correlated part, and we take one parameter, which is the fully non-correlated part.
STFC-RAL-CR03 R61: And that can make these sensory explicit.
STFC-RAL-CR03 R61: But consider carefully what you're doing when you're rebranchising, because of course, we're introducing a lot of extra
STFC-RAL-CR03 R61: expert problems. So this is what you see. This example comes from an EFT measurement, from a SNES measurement, and this is where they call it the fitting basis.
STFC-RAL-CR03 R61: So they do switch to the spaces where they say, okay, here we go, actually,
STFC-RAL-CR03 R61: how we can do this uncorrelated version. Basically, we take our eigen directions of the matrix, and we can do the fit there much easier, and then we can turn it back
STFC-RAL-CR03 R61: But when you send it back, you don't always get all the same information.
STFC-RAL-CR03 R61: Okay. Sorry, you suggested we interrupt. Yes. Why are you pointing to constraints on nuisance parameters as a problem, but not correlations between them?
STFC-RAL-CR03 R61: It's the same thing. It is the same thing. So usually the constraints that you see here come from correlations between the parameters. So this is kind of why I put them together, these two slides together, is that when you see that there's a large, large constraint on something, maybe you're missing a large correlation.
STFC-RAL-CR03 R61: And you can look at this and try to figure out what's going on, and maybe you can repramicize it that you're… you can do a fit that doesn't have this problem.
STFC-RAL-CR03 R61: All right. Another example of something where I go wrong.
STFC-RAL-CR03 R61: This is the A2 meson, so it's here.
STFC-RAL-CR03 R61: It's a nice peek?
STFC-RAL-CR03 R61: They thought… oh.
STFC-RAL-CR03 R61: So, there you found, if you look here, you have the single peak version, or you have the two-pick version that you could fit.
STFC-RAL-CR03 R61: So this was first reported in 1967, and it was the sixth stigma Fest, but it was definitely split. There was actually two peaks in there.
STFC-RAL-CR03 R61: It was confirmed in 1969, which you can see here, where you have the original data.
STFC-RAL-CR03 R61: you have a selection that was the first selection that was applied, and a second selection that was applied, and here you can very nicely see those two peaks, and they fitted them, and it was clear, and they published that. In 1971, it went away.
STFC-RAL-CR03 R61: It was also against a creative analysis coach.
STFC-RAL-CR03 R61: But also, it was a thing where they really, really wanted to see it. So the 1969 one, they found it because in 1967 they found it.
STFC-RAL-CR03 R61: They really wanted to measure the same thing.
STFC-RAL-CR03 R61: There were also a bunch of different groups who did not manage to find it, so they didn't publish.
STFC-RAL-CR03 R61: That's unfortunate. As far as we know, it's not slipped.
STFC-RAL-CR03 R61: Alright, so this is a… comes back to this question of how many parameters do I actually need in my fit?
STFC-RAL-CR03 R61: Sorry.
STFC-RAL-CR03 R61: There can be a hypothesis that you have to add more parameters to your background, or you can have a hypothesis that you actually need a signal on it.
STFC-RAL-CR03 R61: Which one do you take? So here, for example, clearly, I just had the wrong background function, rather than that there was a signal, but which one is it? Because you don't really know from just the data.
STFC-RAL-CR03 R61: So, this is, again, all to do with these background estimations.
STFC-RAL-CR03 R61: If you have your background well estimated, this one looks pretty decent, this whole plot will look very decent, but you have to basically know how well, you know, your, backgrounds.
STFC-RAL-CR03 R61: There is some… oh, yeah. There… I have some slides in the backup, about when to stop, but basically, you can do something which is called the goodness of it.
STFC-RAL-CR03 R61: This is a text where you basically check how well your chi-squared changes based on adding more parameters to your background bin, and it gives you an idea on the change in the chi-squared of your stopping criteria.
STFC-RAL-CR03 R61: This, of course, has a difficulty, which is that if there is a signal, you will also add more parameters until you include the signal, which is why you should generally do this binding procedure, and make sure that you do that test on your side and normally.
STFC-RAL-CR03 R61: That does assume, you know, where your signal might be.
STFC-RAL-CR03 R61: That's quite complicated.
STFC-RAL-CR03 R61: Okay, so another fun experiment. This is the, CHI 8.3. This was discovered in 1984 by the Crystal Dow experimenter diesel.
STFC-RAL-CR03 R61: So they measured the energy of the photons, you can see the thing here, you saw a peak.
STFC-RAL-CR03 R61: So the signals were E plus E minus to this psi and the photon. It was a 4.2 sigma effect. So here you see the raw data. Here you see the fifth, with this background plus signal fifth.
STFC-RAL-CR03 R61: And here you see the background subtractift.
STFC-RAL-CR03 R61: It really looks pretty convincing to me, this one.
STFC-RAL-CR03 R61: So, one more data was taken in 1985, the peak went away, and this was that problem. They had just basically decided to… to fix a background plus signal fit a peak, because they could see the peak there, and therefore they found it.
STFC-RAL-CR03 R61: Okay, so when do things go wrong? I like this quote of Michelangelo, when he says, it was easy, I just got a block of marble and chipped away anything that didn't look like
STFC-RAL-CR03 R61: David. So that might be a good way of creating a software. Very bad at doing physics, but that's basically the problem of all the different mistakes that I showed you before.
STFC-RAL-CR03 R61: So to resist temptation to do that, because we're very good at human, we're extremely good at pattern recognition. We like to see these googs. We like to be… we actually, if you ask someone to draw a function through a set of data points, they manage to find the chi-squared minimum for the fit in there quite quickly, just by hand.
STFC-RAL-CR03 R61: I find it incredibly impressive. But it's not really what you want when you're doing sort of scientific research, right?
STFC-RAL-CR03 R61: So for searches, we devise cuts before looking at the data. We use Monte Carlo simulation to not look at the data. We look at data with sidebands, basically blind to the data, and only when the cuts are optimized do we look… do we unblind we look in the box.
STFC-RAL-CR03 R61: So, measurements where we already know something, one of the standard procedures, for example, in a bar or in Edit B is to do a secret offset. So basically, you just hard-code some offset into that parameters data, which is
STFC-RAL-CR03 R61: random, but always the same, such that you can predict it later. You look at all your data, you measure the exact number, and only when you're finished, you can take it away so that you can measure something that is slightly off from where you expected something to be.
STFC-RAL-CR03 R61: So, I saw something going for a partial summary here.
STFC-RAL-CR03 R61: It means this one is definitely what we go for from my third slide. We do a confidence interval, and according to XPSD, it says, listen, science is hard, but I'm a serious person doing my best.
STFC-RAL-CR03 R61: So, that's pretty good, that's this idea, where we set… take the confidence interval between the… between the points, and we look at all these four plots.
STFC-RAL-CR03 R61: Okay.
STFC-RAL-CR03 R61: So, pretty good so far. We've set up our different methodologies to make sure we don't accidentally, or on purpose cheat with our data.
STFC-RAL-CR03 R61: So, why would anything need to change from gene length?
STFC-RAL-CR03 R61: We had a workshop last year here, which is called Stack Meets Now.
STFC-RAL-CR03 R61: I really like this one, someone just heard it. Some statistics are some weird cracks in whatever it is that we're looking at.
STFC-RAL-CR03 R61: We've put a frame around it, we call it machine learning, and then we have also intelligence, which is what we presented also to the IP.
STFC-RAL-CR03 R61: This is how I started, right? With simulation-based inference, it's not new, it's exactly what we've already done, it just has a new frame around it.
STFC-RAL-CR03 R61: And it's kinda true.
STFC-RAL-CR03 R61: But actually, for the way we do our statistics, or the way we set up our securities for making sure we do our statistics right, things do actually change.
STFC-RAL-CR03 R61: And this is what I want to take the last part of the… oh!
STFC-RAL-CR03 R61: So first question I always ask is, why is your machine learning model doing better than what you were doing before?
STFC-RAL-CR03 R61: So, there are a few reasons this could be. One of them is that it's multidimensional.
STFC-RAL-CR03 R61: So, especially when you have correlated parameters of parameters that you're not directly looking at, maybe in your users' parameters, but maybe in some underlying parameters of the model that you don't know, which could be in your parameters of interest as well, you could have some actual information that comes through, which you couldn't see by
STFC-RAL-CR03 R61: The other part is that it's often based on event-by-event, so rather than a histogram where you plump a bunch of events together that, in that particular vertical, you look at, so you're looking at your Y, you can look directly at your X, at your actual date, event-by-event data.
STFC-RAL-CR03 R61: Another thing that people often say is, oh, the machine learning model is so much faster.
STFC-RAL-CR03 R61: One of the ways that it does that is because it can often parallelize things. It can look at different groupings of the data very quickly in parallel, which otherwise you do it by hand, you do sequentially. But very often, the case is that it's just improvised.
STFC-RAL-CR03 R61: He uses a simplified likelihood, or he did not include any systematics, and it goes faster because of that.
STFC-RAL-CR03 R61: And the other thing is wrong assumption. So, sometimes we assume things because it makes our calculations easier, which might not necessarily be true, and it doesn't learn the same.
STFC-RAL-CR03 R61: That's great, right? This is one of the reasons why you should do machine learning. Another reason why I started with this is that last time when I talked about how we should change our statistics for machine learning, I got told that I hate machine learning. I love machine learning. I think it's a great thing. I just do think we have to carefully look at how we do it.
STFC-RAL-CR03 R61: So, what kind of machine learning do they use for physics? I think this is stuff people generally, especially if they're using machine learning, they know. So, one of them is called supervised learning. This is everything where you do classification or regression.
STFC-RAL-CR03 R61: Is it… is my event signal or background? It's the most common way people use machine learning.
STFC-RAL-CR03 R61: That's what we… into machine learning languages, or supervised learning. Then there's unsupervised learning, so what we mean with that is clustering, dimensionality reproduction, or compression, or outlier protection.
STFC-RAL-CR03 R61: So, clustering is generally what we use when we try and do some tracking information with machine learning, reconstructing events from the detector.
STFC-RAL-CR03 R61: Outline detection, that's anomaly detection, is also quite optical. It's basically trying to say, okay, if we just teach it the standard model, can it tell me some of the events that do not fit?
STFC-RAL-CR03 R61: And then there's weak and steady supervised learning. So weak is everything where we just move labels, or corrections, where we re-weight the loss, or multiple instance learning. Basically, we on purpose fudge the data.
STFC-RAL-CR03 R61: To make sure that it doesn't learn things that we might not actually have.
STFC-RAL-CR03 R61: So this is sort of the equivalent of this offset that we… that I talked about before that is used, for example, there's V.
STFC-RAL-CR03 R61: And then the semi-supervised learning is things like pseudo-labeling with confidence thresholds, so basically saying, don't just tell me if it's a signal or a background. If you're not sure, don't give it a label. Don't tell me if it's a signal or a background. Only if you're sure enough, tell me this event is a signal and this event is a background.
STFC-RAL-CR03 R61: Sorry, what could go wrong? My student made this slide. We've had a project which worked on transformers, where we gave in a bunch of information, and every time the student would come back.
STFC-RAL-CR03 R61: and say, oh, I found it, I can perfectly separate the signal from the background, which I couldn't do at all classically. It was used in the PTF-8 jet,
STFC-RAL-CR03 R61: When, actually, we knew that this simulation was only after adopting 7 jets.
STFC-RAL-CR03 R61: After that, we literally had no idea, and it was filling random numbers. So you could very well separate things in Dora Monte Carlo, because it was using very false information.
STFC-RAL-CR03 R61: So you run into some multiple issue. You didn't learn anything about the actual physics.
STFC-RAL-CR03 R61: One of the other problems which we see a lot, and which I really want to warn about, is people say, well, oh, look, I found the answer so fast, I got such a nice precision on it.
STFC-RAL-CR03 R61: And they gave the step size of your minimization process as the uncertainty.
STFC-RAL-CR03 R61: And it actually has no clue how sure they are about something, it just gave some number.
STFC-RAL-CR03 R61: said, okay, I did some classification, it says this is my signal, and therefore, the mass of the W boson is whatever it is. But they didn't include any uncertain training in their analysis in any of that, in any of those steps.
STFC-RAL-CR03 R61: So they get something, they gave the step size of the minimization process of machine learning as their uncertainty, and they got something exceptionally accurate.
STFC-RAL-CR03 R61: without any sentences on it, and, you know, that is not right. And the other thing is data scopes, and I think this is the one that people are the most aware of. I see the picture went off. But basically, it's the same idea as this
STFC-RAL-CR03 R61: Discovery of the top quartet raw mass that we saw before, where it just cuts away a bunch of stuff until everything that's left over looks like signal, because it was asked to find signals.
STFC-RAL-CR03 R61: So it found you a sign.
STFC-RAL-CR03 R61: You might think this hasn't actually happened, but even this year, I… in Atlas.
STFC-RAL-CR03 R61: re-blinded an analysis after they told me that after unblinding, they found an excess of 7 sigma, which I could not see by eye in the place where they saw it.
STFC-RAL-CR03 R61: No matter what cuts you were applying. So I told them that maybe they should do some cross-checks, they were very, very pissed at me. They found a bug in their code, and it went away.
STFC-RAL-CR03 R61: They were quite unhappy with me, I have to admit, but they came to me and asked me, how could it be to the statistics committee, and they said, oh, how could it be that we're actually finding Seven Sigma, but can't see anything?
STFC-RAL-CR03 R61: And the answer is you're doing things wrong. So data… data salting really happens quite often in the human anyway, because you're asking it to find something.
STFC-RAL-CR03 R61: So, there's some special cases. So one of them is physics-informed machine learning, which is writing and gaining some traction. So, the idea here is that you're grounding the data-driven network of internet knowledge.
STFC-RAL-CR03 R61: And the idea is that, is that you get an accurate, robust, and interpretable outcome. What do I mean with this? If you have a transformer, which were some of this big.
STFC-RAL-CR03 R61: a big matrix of information, you basically match a bunch of shit, and you say you don't need to figure out what that is, because we know that the things that are interesting are the four vectors of these particles, or something like that, right? You're giving it specific information, where you say you don't have to learn everything.
STFC-RAL-CR03 R61: This part we already know.
STFC-RAL-CR03 R61: that one of the things, especially here, is this interpretability. Someone told me interpretability is a warm, fuzzy feeling, or more than that. So what is the idea with that? It's that you don't just want it to give you some number at the end, some significance, you want to understand what that means.
STFC-RAL-CR03 R61: You want to understand not just that there is a new particle, but what mass air has, what particles it interacts with, and why you haven't seen it before. If you can't answer those questions, none of us really get happy about it.
STFC-RAL-CR03 R61: Because we want to understand why we found what we found.
STFC-RAL-CR03 R61: The other thing is uncertainty-aware machine learning. So basically, what the idea there is, you try and make it learn the confidence interval.
STFC-RAL-CR03 R61: And not whether it's a signal or a background. You're trying to not find the central value, but the confidence interval.
STFC-RAL-CR03 R61: The disadvantage is, you have to give it all those 200 nuisance parameters to start with, and it has to train on all of them, which means your training all of a sudden gets a lot more larger, and a lot more complicated, and a lot more expensive.
STFC-RAL-CR03 R61: But in the end, you get something that is closer to what we already do. So there's some ideas on how we can improve on that, and this is a fairly new thing that we're trying to work on. And why is this new? Because we think machine learning is so widespread. We're definitely not the only ones doing that in physics.
STFC-RAL-CR03 R61: But if you look at, for example, a marketing company, they want to know whether they should stock
STFC-RAL-CR03 R61: beer and diapers, and they don't really care what their uncertainty interval is, because no matter what, they will do that thing that comes out as the best thing, and try it out for a while. They don't need to know that there's a 20% uncertainty on that.
STFC-RAL-CR03 R61: On whether they should stop the diaper since the beginning.
STFC-RAL-CR03 R61: We care a lot about these intervals, so for us, this makes more sense to look at.
STFC-RAL-CR03 R61: Okay.
STFC-RAL-CR03 R61: So if we go back to this Michelangelo thing about sculpting our data.
STFC-RAL-CR03 R61: This is… finding David out of a block of marble is actually quite often the way our machine learning works. We say finding a signal, and it finds a signal.
STFC-RAL-CR03 R61: And in this case, you just want to make sure you actually found David, not your life, or one of some other weird sculpture that maybe you weren't waiting for.
STFC-RAL-CR03 R61: If we say, okay, to resist temptation, we can do this dip binding and things like that, this… all this stuff is still true, and we still have to do, but it's often harder to do for machine learning, because we do accidentally add the information of the JET to this training data. So we have to think very carefully
STFC-RAL-CR03 R61: If what the information is that we provide, and what it might learn from that, which is hard, because the whole point of asking the machine to learn it rather than us is because we don't know what it might learn from that, and might learn more things than we know.
STFC-RAL-CR03 R61: But this secret offset still works.
STFC-RAL-CR03 R61: We don't do that, though. I haven't seen a single machine learning analysis that actually uses secret offsets.
STFC-RAL-CR03 R61: But we could definitely still do that.
STFC-RAL-CR03 R61: Okay, so what do you make… how do you make sure that you're not finding a David in your, in your block marble? So these are some of the, techniques to prevent it in machine learning. One of them is called regularization.
STFC-RAL-CR03 R61: So this basically would do it as a penalty term to the loss function to discourage large parameter values.
STFC-RAL-CR03 R61: This we're also used to for unfolding.
STFC-RAL-CR03 R61: If I would have talked about that, I would have talked about the regularization one, but basically what it means is you're not allowed to use single outliers with a large weight to decide whether or not something is there or not. We give a penalty for outliers.
STFC-RAL-CR03 R61: The other thing is dropout. This we do a lot. This we actually have started to become common practice. So during training, we randomly pick sets of fractions, that we set to zero in each iteration.
STFC-RAL-CR03 R61: And that makes sure that this… that we don't look at particular clusters of events. So, you should find the same thing.
STFC-RAL-CR03 R61: Even if a few events fall out. If a few events are away, or a section of 20% of our data is not there, it can't really change
STFC-RAL-CR03 R61: What the outcome of machine learning is, because it can never learn from just those few events.
STFC-RAL-CR03 R61: So why not drop some of the data on some of the neurons?
STFC-RAL-CR03 R61: Actually, we do both. So, depending on which analysis, you could do either one. Generally, this has… it's just a little bit safer, but either one would work.
STFC-RAL-CR03 R61: The other one is barely stopping. Basically, if you're trying to minimize something, before you're actually at the minimum, you stop and you say, we're good, we're close enough.
STFC-RAL-CR03 R61: Basically, as soon as your validation loss stops improving, and therefore you reduce the fit of the whole stream.
STFC-RAL-CR03 R61: The other one is data augmentation, so you increase the effective size of the training set, by applying physically valid transformations. So basically, you make toy datasets, a lot of extra toy data sets, to make sure that any kind of statistical fluctuations are not there, and you use those as well.
STFC-RAL-CR03 R61: Each of them has their risks as well.
STFC-RAL-CR03 R61: So, if we go to this difficult part where we say we don't really know how to do these findings, there are a few checks that you can do.
STFC-RAL-CR03 R61: One of them is an injection test.
STFC-RAL-CR03 R61: This one, I think you should always do.
STFC-RAL-CR03 R61: Inject some signal, and try to figure out what you're… what… after you're trained on non-signal data.
STFC-RAL-CR03 R61: Try and find out what your model would say happens to that.
STFC-RAL-CR03 R61: This is also to avoid doing this too much.
STFC-RAL-CR03 R61: Because sometimes, I see people do all of these.
STFC-RAL-CR03 R61: all of the possible checks to make sure that they wouldn't find the signal, and trust me, they do not find the signal, no matter how big the signal is that they inject. If you inject the 7 Sigma signal, I would certainly call my machine learning to find it.
STFC-RAL-CR03 R61: Submit our adjusted input data, we would just take a variation on our input data and see whether it comes out the same.
STFC-RAL-CR03 R61: Testing on a multicolor from a different generator, this seems obvious, we don't often do.
STFC-RAL-CR03 R61: We usually try and take something that is closer to what we trained with, because we think that either one of the generators might be better for our purpose, but taking our validation dataset from a different generator really helps with not using this H-Jet information that we might not have.
STFC-RAL-CR03 R61: Because it might have a different problem in that Montefalo generator, comparative to what you'd use to train on.
STFC-RAL-CR03 R61: And the other thing is stay blind.
STFC-RAL-CR03 R61: So we usually don't… we don't often, use real data to train on.
STFC-RAL-CR03 R61: But when you do, make sure that you don't accidentally learn it. And what do I mean with that? So, for example, someone gave a very nice talk where they could find the math that they did so well.
STFC-RAL-CR03 R61: It was beautiful. But it turns out that even though they didn't have the mass of… the mass distribution as an input, they gave all the four vectors and all the things, which meant that the machine learning very quickly figured out how to calculate the invariant mass of the system from the four vectors.
STFC-RAL-CR03 R61: So I just learned where the Higgs leak was, because the data that we included had the Higgs in them.
STFC-RAL-CR03 R61: Great!
STFC-RAL-CR03 R61: I'm glad it defined that, but it was really not the goal. So, make sure that when you say, I didn't include something, that you also didn't include the information to infer it.
STFC-RAL-CR03 R61: And that is quite hard to do.
STFC-RAL-CR03 R61: This is a topic called fairness in machine learning.
STFC-RAL-CR03 R61: So if you want to learn more about that, that's the way it starts in team learning, you can look it up, there's some fun papers on that, to see how you can make sure that in physics, you're not accidentally giving it all the information anyway, that you are actually unblinded your teeth already.
STFC-RAL-CR03 R61: If you take that example, then it's…
STFC-RAL-CR03 R61: you know, the way that things used to be done, where you would put in, you know, 10 parameters or something, then it was easy to make sure that you didn't give enough information to reconstruct the full vector and get the mass. But now that we tend to throw all of the low-level information at it, so it's a lot harder, so what would your answer be? Yeah, so what we found is that, for example, if you divide out
STFC-RAL-CR03 R61: of the PT of the particle out of each of the parameters, then it couldn't learn the mass distribution for this particular example.
STFC-RAL-CR03 R61: But you have to… basically, you just have to check it for analysis, that you're sure that you're not actually learning that… that you're sure what you're learning. Because indeed, at lower level information, it's harder, because you give it all more information, so it learns more information.
STFC-RAL-CR03 R61: And it's easy to spot if that calculation happens?
STFC-RAL-CR03 R61: Really, really important.
STFC-RAL-CR03 R61: Yeah, what helps is to try and see if it can,
STFC-RAL-CR03 R61: Basically, if it can plot the distribution that you're trying to make sure that it doesn't know.
STFC-RAL-CR03 R61: Because if it can do that, it… Yeah. If that one is flat, then you're fine.
STFC-RAL-CR03 R61: In the case of maths, you can look at the score as a function of maths, right? Yeah, that's the capital value.
STFC-RAL-CR03 R61: So that helps. And then the weird thing is, it needs to be flat, but not…
STFC-RAL-CR03 R61: not completely flat, so it can be linear, it just can't have a peak there. And how to decide when it's a little bit of a bump and when it's flat, a flat enough, that part is still being thought over by statisticians. Like, how do we de-quantify whether something is flat enough when it isn't even flat?
STFC-RAL-CR03 R61: Turns out, not that easy. But, for now, I would say look at the score distribution for that parameter that you're trying not to include, and see if it looks by eye, which I said at the beginning, don't do that. That's enough.
STFC-RAL-CR03 R61: Okay. Sorry, can I ask on the previous point? Yes, go ahead. I could ask, on testing of Carlo using a different generator, that seems to kind of rely on there being
STFC-RAL-CR03 R61: enough generators that you can do this. Is there a way you kind of make a generator do…
STFC-RAL-CR03 R61: something wrong.
STFC-RAL-CR03 R61: In order to, yeah, so one of the things, for example, you could always use a different tune of the same generator. It won't have quite as…
STFC-RAL-CR03 R61: much effect as using an actual different generator, but generally heating on different tubes of the same generator will already give you something, or dropping out information, just saying, specifically.
STFC-RAL-CR03 R61: throwing away…
STFC-RAL-CR03 R61: the high PT information or the gen information are places where you don't expect the thing to learn anything from.
STFC-RAL-CR03 R61: and check whether, indeed, it didn't go back. So it's a little bit of the…
STFC-RAL-CR03 R61: Of the dropout, but not dropping out random bits, but dropping out very specific bits of information.
STFC-RAL-CR03 R61: pick the 8th jet. Dropping the 8th jet helps a lot. India doesn't know how to do it.
STFC-RAL-CR03 R61: All right.
STFC-RAL-CR03 R61: So, that gets me to a summary.
STFC-RAL-CR03 R61: We… this is sort of the rules that we set for results.
STFC-RAL-CR03 R61: We use a high level of significance, let's say this Phi Sigma. We're the only field in the world that does that. Everybody else says 1 sigma is usually enough. Some are vigorous, and they say 3 sigma. We say 5 or 6 sigma.
STFC-RAL-CR03 R61: History tells us that's necessary. One of the other reasons is we have a high output of results. If you look at what comes out of Apploss or CMS, it's basically a paper every day.
STFC-RAL-CR03 R61: If this is 99% confidence level, it's 5 sigma, right? It's 3 Sigma.
STFC-RAL-CR03 R61: Yeah, 3 single. We would have 3 or 4 papers every year that are claiming a discovery that is not there. We don't want to do that! So, to make the public believe in us, we set it a little bit higher.
STFC-RAL-CR03 R61: We use mind analyses, which we have to keep doing, which is slightly more difficult for machine learning. We focus on these intervals rather than the best estimates, because we're serious scientists.
STFC-RAL-CR03 R61: We look at independent validations, such as ATLAS versus CMS, and here I want to put the caveat, rumors are really hard to get rid of. Turns out, they did a social science study at SUD, where they injected a rumor that ATHLAS saw a residence somewhere.
STFC-RAL-CR03 R61: Within 20 minutes, everybody in Atlas knew that everybody in CMS knew. It wasn't real, but it was a very fun… the social science articles about CERN are really funny. They hired a bunch of social scientists to also see at the dynamics between people, that we don't come off very well. But…
STFC-RAL-CR03 R61: These are kind of funny, that the rumors are really, really fast.
STFC-RAL-CR03 R61: They also tracked how they get spread, because everybody immediately might guess that we had marriages between people who come out and seen us. Yeah, they went to culprits. But these are really, really hard to get rid of, and it does happen sometimes where you think, oh, I'm going to…
STFC-RAL-CR03 R61: work on an analysis, because CMS are something.
STFC-RAL-CR03 R61: And that already is a little bit of a bias, right?
STFC-RAL-CR03 R61: Use common sense, and a very healthy dose of skepticism. If someone comes and tells you you have a 7 sigma test somewhere and you don't see it.
STFC-RAL-CR03 R61: First, consider that something is wrong. Don't get too excited just yet.
STFC-RAL-CR03 R61: I put a link here, which is the Saturn's Pride Side event, which was called Statley Savannah, which was here in London last year. There's some amazing talks there on very specific… very specific machine learning models, what the statistics behind them is, and how, for that particular model, you could change it to be more safe.
STFC-RAL-CR03 R61: And I want to mention as well that there's another effort coming out, which is called Verify, which is a 5-stat
STFC-RAL-CR03 R61: Basically, a proceedings, which is based off of the follow-up of this meeting, which is currently being planned, but where we've paired a bunch of very, very clever statisticians with a bunch of very clever physicists to write a series of 10 articles.
STFC-RAL-CR03 R61: About, specifically, statistics for machine learning, what the tests are that we should be doing according to them, which will come out also later this year.
STFC-RAL-CR03 R61: So I can't really tell you what the result of that is, because quite a lot of them don't have the results yet.
STFC-RAL-CR03 R61: So with that, if it looks like that we quite slip it up, it probably is a nook. Thanks!
STFC-RAL-CR03 R61: Yes.
STFC-RAL-CR03 R61: So there were a lot of tricks on how to be cautious to not find access.
STFC-RAL-CR03 R61: And I… like, there was one trick which said, like, okay, you should still have a model which should try to find an excess in case there is an excess. I'm… I'm a bit concerned. I know in machine learning, we're going to the…
STFC-RAL-CR03 R61: place where we might find an excess if it doesn't exist, but if there is an excess, and we don't find it because we're too conservative, it might…
STFC-RAL-CR03 R61: it might prevent us from the next step, and I'm wondering if there is a balance to be achieved, or…
STFC-RAL-CR03 R61: There is, in my opinion. It means you have to have an accurate description of your uncertainties, also on your machine learning model.
STFC-RAL-CR03 R61: If you look… I mentioned the fact that if you would have Three Sigma, we would have a couple of papers every year that claim the three sigma test, which isn't there. We do 5 sigma, it's not an infinite number of Sigmas, we should still have a bunch of Three Sigma assesses every once in a while, that aren't there.
STFC-RAL-CR03 R61: And as you can see, we don't actually do that. If you actually look at the distribution of the significances of each of the excesses, it goes… follows the distribution that you expect for about two, two and a half sigma, then it drops.
STFC-RAL-CR03 R61: Because if it's near 3 sigma, we decide to inflate our uncertainties quite often.
STFC-RAL-CR03 R61: It is not a good way of doing things. Because we are… if you overestimate your uncertainties, you get exactly to this point. You get to the point where you don't see something that you could have actually seen.
STFC-RAL-CR03 R61: We just have to describe it honestly. Say, look, this is what comes out. You should still publish it.
STFC-RAL-CR03 R61: And this is the same problem with machine learning, and this is also why I'm quite excited about this uncertainty-aware machine learning efforts,
STFC-RAL-CR03 R61: physics inspired machine learning, because if you combine those, you get a reduction in the difficulty for the machine learning, and at the same time, you get your right uncertainties. Having the correct uncertainties should mean
STFC-RAL-CR03 R61: that you still get your discovery when it is there. For that, if you want to learn more about it, I would very much recommend to look… to look up the, the statistics then called coverage.
STFC-RAL-CR03 R61: Coverage means you get… if you stay 68% of the time, 68% confidence level, that 68% of the time, you fall within that uncertainty, and the rest of the time you don't.
STFC-RAL-CR03 R61: That's a legal coverage, and that is something we should have. We shouldn't over-cover either.
STFC-RAL-CR03 R61: More questions in the room, yes. Okay. I was wondering if you could say something more about… you talked about measurements and shifting the parameters to, kind of, blinder measurements as such. I mean, I guess, similar to what we've been talking about, there's a balance to this in terms of you're going to train your thing if you shift parameters
STFC-RAL-CR03 R61: too far, you train on something that isn't useful. If you don't shift enough, you're… you might as well be training without the shift. I think you mentioned LHCB use this a lot. I wonder if you could say, how do you find that balance, and what kind of parameters at the moment? Right. Sorry.
STFC-RAL-CR03 R61: First of all, you should only ever use this for a measurement, and you shouldn't use it if you're doing a search. Because if you're doing a search by moving things around, you really do
STFC-RAL-CR03 R61: Bites yourself into finding things or not finding things.
STFC-RAL-CR03 R61: Singapore, the offset should be very specific and the same for everything.
STFC-RAL-CR03 R61: That does mean you have to make sure that you don't accidentally learn what the offset is. I know there was an analysis in Baba where they measured something, the parameter to be 3, where the edge of the range was 1.
STFC-RAL-CR03 R61: So they figured out what the obstacle was.
STFC-RAL-CR03 R61: If you're looking at a ratio, it really… that should be between 0 and 1, it really shouldn't suddenly become 3.
STFC-RAL-CR03 R61: But…
STFC-RAL-CR03 R61: In general, look, taking a small offset of order of cent on a ratio is a good system to go for. That way, you get something of the order of your uncertainties. If you have a one signal uncertainty, throw a random number according to the caution of your uncertainty.
STFC-RAL-CR03 R61: And pick that number of your offset. And then you should be near enough
STFC-RAL-CR03 R61: To your parameter that you aren't biasing yourself?
STFC-RAL-CR03 R61: But at the same time, you don't know whether the mass of the W is exactly what you thought it would be there.
STFC-RAL-CR03 R61: distance.
STFC-RAL-CR03 R61: Do you look at combined performance at all from a statistics point of view? I'm just wondering, as we do more and more machine learning.
STFC-RAL-CR03 R61: Places that are not physics analyses, but influence what comes out.
STFC-RAL-CR03 R61: Annapolis, you mean? Yep.
STFC-RAL-CR03 R61: Not really. We put a bunch of calls out to say, if you're doing things that are not too trivial, please come and tell us, and come and talk to us about it.
STFC-RAL-CR03 R61: We do the same for analyses, we say the same thing. If you're going to do anything new, try and come and talk to us about it. And I always say, please don't do that the day before it has to be approved for modeling.
STFC-RAL-CR03 R61: So, after I've started saying that, people are about 2 days beforehand.
STFC-RAL-CR03 R61: It's an improvement, but really not where I want to be. More often than not, there's some convener, also a CP group that comes to me and says, the car is also really weird.
STFC-RAL-CR03 R61: Or an EV. Most times, EVs, the editorial board panelists, they come and say, this is beyond our debt, it looks weird, can you check?
STFC-RAL-CR03 R61: I would like it if the CP groups started coming as well. The few times that I've actually asked them to, they say, oh, but we talked to the machine learning forum. Machine learning forum is great, but we should really integrate that with these kind of issues.
STFC-RAL-CR03 R61: Yes, please.
STFC-RAL-CR03 R61: So, I wonder what your view is on, analysis preservation in this domain, because it gets harder and harder to
STFC-RAL-CR03 R61: have results which are… or does it? You know, to have results which can then be stored. Let's think of, like, a rivet analysis, or something like that, and how you do that when you're using machine learning, and whether you think that's actually within the scope of a statistical perspective, or it's just a much bigger question.
STFC-RAL-CR03 R61: Okay, that's a good question. In general, I'm very in favor of an outsourced reservation, and it doesn't happen yearly enough, in my opinion.
STFC-RAL-CR03 R61: I think the easiest way to do analysis preservation is to save the full one payload.
STFC-RAL-CR03 R61: You don't have to describe to keep everything.
STFC-RAL-CR03 R61: But keeping the full likelihood means that you can reinterpret it a lot easier than if you would Supply.
STFC-RAL-CR03 R61: Just the plots on the paper.
STFC-RAL-CR03 R61: Keeping a full likelihood when you're especially going to likelihood-free inference, which a lot of machine learning-based models are using, becomes a lot harder.
STFC-RAL-CR03 R61: What you can do is to save a copy of the pre-trained model. So you don't have to keep all of Monte Carlo data in that case, but you can keep a trainer's model. Because if you do that, it basically becomes a single algorithm, it doesn't take up too much space, and you should be able to at least reproduce the results again on new data.
STFC-RAL-CR03 R61: Yeah, it's not something that's meaningful to a theorist. It's not something that's meaningful to a theorist, but the very… the likelihoods are meaningful for a theorist.
STFC-RAL-CR03 R61: The pre-trained model usually is not, but it does at least mean that you… when something else finds something completely different, you can apply the same model and see if it's a problem from the machine learning model, or whether it's a problem of the
STFC-RAL-CR03 R61: Beta.
STFC-RAL-CR03 R61: Okay, thanks.
STFC-RAL-CR03 R61: No questions in the room.
STFC-RAL-CR03 R61: Are there any questions on Zoom? I don't see. Okay, then let's thank the speaker again.
STFC-RAL-CR03 R61: Maybe that's the more statistics question, they can send me. Well, I mean, we're heading for lunch now, I don't want to go, let's say, ask people to join.
STFC-RAL-CR03 R61: Absolutely.