05 - SN Partial Differential Equations and Applications Webinar
4.63K subscribers
SUBSCRIBED
Join George Em Karniadakis of Brown University and MIT as he presents a new approach to develop a data-driven, learning-based framework for predicting outcomes of physical and biological systems, governed by PDEs, and for discovering hidden physics from noisy data. Don’t want to miss another webinar? Register here: https://go.sn.pub/lXgEaxHave an article on the neural networks? Submit Your Article: https://go.sn.pub/GT525VLearn more about publishing with SN Partial Differential Equations and Applications: https://go.sn.pub/UFulhi
00:09
It's really my privilege and my owner as well as
00:13
my pleasure to introduce our speaker today.
00:16
Professor George Canada kiss from Brown University,
00:19
now in a way George doesn't need an introduction.
00:22
He one can safely say that he's one of the
00:25
world's Preeminent Applied Mathematicians and computational scientists with many,
00:31
many pioneering contributions over the years.
00:34
So but very quickly, George received his PhD from MIT
00:37
in 1987.
00:38
He was a faculty member at Princeton University.
00:41
Before he moved to Brown University in 1993,
00:44
where he is shown, Palmer,
00:46
Bairstow, Professor and the Charles Pitts Professor.
00:50
He's also affiliated with MIT,
00:52
where his research scientists and with the Pacific Northwestern National
00:57
Labs where he has a big group and he has
00:59
many different groups.
01:01
As you can see from his slides now here.
01:03
As I said, he has many similar contributions in various
01:07
areas of computational science and discontinuous collection methods.
01:12
Spectral element methods. Generalized polynomial chaos fractional beedis microfluidics,
01:18
nanofluidics and more recently in machine learning and really pioneering
01:23
very important part of machine learning that he's going to
01:27
talk about pins and its applications in scientific computing.
01:32
George has his member of all fellow sorry of all
01:35
professional societies that you can think of.
01:38
Siam is me.
01:41
Many awards and honors. The list is too long,
01:44
but let me mention the Ralph Kliment rise from Siam
01:49
in 2015 among others.
01:50
So George, it's really our our pleasure to host you
01:55
and maybe one just to testify.
02:00
Testifying your impact is the fact that you have action
02:03
index which is more than 100 now,
02:05
which is really remarkable. So your impact applied mathematics is
02:08
massive and we are truly delighted to have you here
02:11
and we're looking forward to your talk.
02:14
The floor is yours.
02:16
Thank you very much. I said that very generous of
02:20
you and can you hear me very well,
02:23
very well that's great. I just invested on this nublu
02:27
technology for for a microphone that's good.
02:30
So I want to talk about.
02:34
Computational solutions of partial differential equations.
02:39
An using neural networks kind of different take on what
02:43
you may have heard so far,
02:45
because I want to talk about not just functional approximation
02:49
but functional approximation and operators approximation and just so that.
02:55
I give you the main theme from the very beginning
02:59
of what you saw in this movie that I play
03:03
before you play it again.
03:05
I can play this anyway.
03:07
What you see here is.
03:11
Some data which some of my friends from Germany La
03:14
Vision gave me and drink coffee every morning and just
03:19
have my coffee.
03:20
I was very curious to see what is the maximum
03:24
velocity over my espresso Cup and can I quantify?
03:28
The pressure field and the velocity field everywhere,
03:32
given that they give me a temperature.
03:35
So that's like something that curious and and have some
03:38
data,
03:39
but obviously it's an ill posed problem.
03:41
So the main theme of my today's talk is can
03:44
we solve problems using piddies which we cannot solve easily
03:49
with conventional methods.
03:50
OK, so that's why I'll get back to the coffee
03:53
later on.
03:54
But let me, by the way of introduction and outline,
03:57
give you some Glossary that I will be using today.
04:01
Pin is a physics informed neural network that about I
04:05
will talk about this in the context of function approximation.
04:09
DF net is functional approximation.
04:12
I will give you some application and how what kind
04:15
of network should approximate the functional?
04:18
Very useful for autonomy. Another big applications important applications then
04:23
my favorite one is Depot net.
04:25
It's paper that will appear in nature machine intelligence soon
04:30
and it has 16 different applications.
04:33
Ann is basically neural network at the level of of
04:36
operator of system identification and nonlinear operator and then how
04:41
you can use if I have time if I don't
04:44
run out of time I will talk about deep Eminem,
04:47
this sweet new framework we introduce how you can use
04:51
Depot Nets as building blocks to solve multiphysics and multiscale
04:56
problems.
04:56
Very very fast, maybe 10 thousand 100,000 times faster than
05:01
the state of the art and that's our current.
05:04
Project with DARPA and we have to produce results every
05:08
week with this project that so I'm very excited about
05:11
it.
05:12
So just to get started.
05:14
Universal function approximation first was developed by your side Pangle
05:19
when he was Professor Assistant professor here at not far
05:23
from where I live in Boston at Tufts University.
05:26
He proposed if the function is continuous function,
05:30
then single hidden layer with a activation function Sigma,
05:34
which is non polynomial, would be a sig model at
05:38
the time.
05:38
What they said. Of course we know now this activation
05:42
function could be some other.
05:44
Nonlinear non polynomial function, but a single layer is enough
05:49
to approximate this at the arbitrary dysfunction at arbitrary accuracy.
05:54
This is sort of the basis of almost everything.
05:57
The subsequent work by massacre.
05:59
One of my collaborators, currently my new project on.
06:05
Newly that we're kicking off tomorrow.
06:07
Actually this this new Murray if you want to watch
06:11
out.
06:11
It's a multi University collaboration between myself Brown at Brown
06:16
and Caltech and Stanford University of Utah on machine learning
06:21
for Piddies,
06:22
which is the topic of my talk today.
06:26
So about discontinuous functions. This theory also covers this continuous
06:31
functions,
06:32
but shallow networks do not work very well,
06:35
but deep networks work very expressive and work really well
06:40
for for discontinuous functions,
06:42
and I want to demonstrate that with something new that
06:47
we have one of my collaborators,
06:49
doctor young, young, John Shin.
06:53
Develop the following. Thierry will not go through the theory
06:57
on to tell you about the sort of a neural
06:59
network.
07:00
If you have functions which are discontinuous or solutions of
07:03
Piddies which are discontinuous like in hyperbolic conservation laws,
07:08
then the weights explode one of the observations you can
07:11
make is that the way to explode.
07:13
So the question is can you can you come up
07:15
with a universal approximation theorem when you have a bound
07:19
on the weights.
07:20
For example, here we have a bound of two we
07:22
don't want.
07:23
Our weights, if you look at the distribution of the
07:26
weights and just for those of you here are the
07:29
weights.
07:29
What we're after is these weights,
07:31
which are basically these lines here and these biases.
07:34
But these weights we don't really want these weights and
07:38
bias to be the big greater than two,
07:40
so we want to contain the growth of the weights
07:43
given some initialization.
07:44
So there's some details here which I will skip,
07:47
but you can see here in this box.
07:49
What I mean is that we can dial in using
07:51
the theory,
07:52
and I want to make the point that the theory
07:54
is very practical tool in this case,
07:57
and I'm I'm after very practical tools so.
07:59
Puritan leaders to an architecture.
08:01
And so I just want to demonstrate it is here.
08:04
Given some inputs for, for example,
08:06
in this theorem, like the input dimension,
08:09
how many discontinuities you have an A controlling the accuracy
08:13
to numbers that control the accuracy that you want,
08:16
and so on. The how you represented some other constants
08:19
you can come up with an architecture.
08:21
Like for example, I want to approximate not just this
08:24
function,
08:25
but actually the advection. So linear advection of these functions
08:29
du DT plus du DX equals 0.
08:30
This is the initial condition and I want to approximate
08:34
this.
08:34
And I will tell you how later,
08:36
but you can see I have an architecture that has
08:39
two inputs and then the first letters,
08:42
20 second layer, 26 euros,
08:43
- Sort of a crazy neuron numbering of neurons per
08:47
layer that this theory gives us.
08:49
But notice that results for long term time integration.
08:53
There's this adaptive way of approximating functions in this.
08:57
In this case, the solution of a linear vector equation,
09:00
which is of course is the deceptively simple equation be
09:04
cause.
09:05
We see no dispersal node node diffusion in this with
09:08
this simple architecture,
09:09
if of course if you dial in any architecture,
09:12
you get Wiggles, you get all the GIF stuff and
09:14
so on.
09:15
But this architecture gives you something good,
09:17
so so there is some value.
09:19
There's a lot of value actually in controlling the way.
09:22
It's not just computing the weights,
09:24
but controlling the weights and connect that to the functions
09:28
that you are targeting.
09:29
As I said here, sort of dial in some of
09:32
the properties that targeting like the number of discontinuity.
09:36
But the main thing I want to talk about today
09:39
is.
09:39
Data how will combined data and PD is in a
09:42
Seamless Way and I work a lot with with the
09:45
Department of Defense and somebody from the Army told me
09:49
gave me this five deal or that the data he
09:52
said their data is not smooth and nice like the
09:56
academic people like their Dinkie which means small,
09:59
dirty, noisy, dynamic streaming and deceptive.
10:02
They said because the Russians everywhere.
10:05
Sorry about that. I didn't mean to offend anybody but
10:10
this is I wanted to quote.
10:12
This I'm not English army person so so if you
10:15
have this data how can you use them in the
10:18
together with PD's?
10:19
Of course there's lots of data simulation methods and so
10:23
on,
10:23
but we know how expensive are current data simulation methods.
10:27
So from the three scenarios where we're used to for
10:31
long time I work on lots of different methods myself
10:34
and I always look at this left scenario where I
10:37
know the physics. In other words,
10:40
I know the PD and I know that.
10:42
Parameters all the parameter Piddies and I love the smoothness
10:45
and then they have small data.
10:47
Presumably this will be my initial and boundary conditions fully
10:51
determined,
10:52
but. More often than not,
10:54
I don't know my initial conditions and boundary conditions.
10:57
I don't know this small parameters in front of the
11:00
important terms.
11:01
Even the Rangers number in Navier Stokes.
11:03
I may not know. I don't there are there many
11:05
properties in multiphysics problems,
11:07
especially constitutive laws, dirty plasmas and so on.
11:10
I partially announced plasmas. There's lots of uncertainty.
11:14
How do you take that into account?
11:16
How you use data to improvise?
11:17
So I'm looking at this day.
11:19
I will be looking at this middle scenario where I
11:22
know some of the physics.
11:23
And I don't know some other part of the physics
11:26
or I may not simply I may not be able
11:28
to resolve it,
11:29
for example in turbulence even in isothermal turbulence,
11:32
I May, I know exactly the equations.
11:34
We know they're valid. However,
11:36
if it's high Reynolds number turbulence,
11:38
there's no way, even with exaflop computers,
11:41
I can resolve your physic,
11:42
your physical turbulence at the scale of 1,000,000 or hundreds
11:46
of millions of in terms of radius number.
11:48
OK, so there's no physics scenario.
11:50
I will skip it today,
11:51
but there's also possible to do something.
11:54
On that, using lots of data and discovering new equations
11:57
or new operators.
11:58
In fact, I'll talk about that later,
11:59
so.
12:00
This brings me to pins physics in form Neural Networks,
12:04
which I already define.
12:07
As any new idea, this idea was resisted too close
12:11
to our two years.
12:13
Actually 2 to get this paper published because nobody wanted
12:18
to use it to publish the paper,
12:20
and now this paper has already.
12:23
I think it has more than $600 or 700 +
12:26
700 citations,
12:27
and it's a very simple concept.
12:30
Let me introduce the concept of a pin.
12:35
They said. This is a neural networks like we said
12:38
before.
12:38
Here we have two layers.
12:40
Sigma is the activation function,
12:41
X is the input. Use the output.
12:43
Of course, if I have lots of data,
12:45
can learn this very, very well.
12:46
I can learn this function no problem.
12:48
Neural networks are extremely well approximating.
12:51
This function, continuous functions and discontinuous functions as I said.
12:55
However, I don't have data in science.
12:57
We never have enough data and Biomedicine would never have.
13:01
Enough data, but we know that you satisfy some conservation
13:05
law here.
13:05
I pretend that conservation laws is parameterized.
13:08
Nonlinear Schrodinger equation could be in the Navier Stokes,
13:12
it could be a multi physics system.
13:14
It could be some equations in some constraints,
13:17
anything one of the difficulties of course is how you
13:20
are you going to go back to standard numerical schemes,
13:23
use finite elements and so on.
13:25
Finite difference spectral elements that I like.
13:29
But if you do that then you will end up
13:31
having a grid.
13:32
A grid is fine if you academic and you do
13:34
homework problems,
13:36
but if you're in an industry.
13:38
Generating management takes six months real match for an airplane
13:43
or or a gearbox or or digital twin of a
13:46
complex system.
13:47
So here instead of using.
13:51
This.
13:53
Conventional methods. Instead, we use automatic differentiation.
13:57
Automatic differentiation is the same method that allows you to
14:00
back propagate the error,
14:02
and in the Standard neural network in terms of law
14:05
in Pytorch.
14:05
In all these frameworks now is standard,
14:08
so we use automatic differentiation.
14:10
Works really well and maybe not so well for very
14:13
high,
14:13
very high order derivatives, but certainly works well we have.
14:17
I'm basically take the derivative of the neural network itself.
14:21
Now I don't have a network here because if I
14:24
take the derivative,
14:25
this becomes a very complicated network.
14:27
So what you see here in the box is actually
14:30
another graph,
14:31
another network which becomes very complicated if you use tensor
14:35
board.
14:35
For example, you can visualize it,
14:37
but it's very complicated. The Nested graph.
14:40
And of course if you take secondary,
14:42
but if every time you take a derivative,
14:44
the depth of this original network fits 10,
14:47
it doubles to 20. If I take a second derivative,
14:50
I will have a really deep network of with 40
14:53
hidden layers.
14:54
So in the process what you have is is 2
14:56
loss functions.
14:57
One will be with a mismatch of the data.
14:59
The data doesn't have to be at.
15:01
The boundary, doesn't have to be boundary conditions.
15:04
Our initial conditions. It could be anything for initial value
15:08
boundary value problem and then you have this extra term
15:12
which will help you.
15:13
It's like supervised learning. That's like the residual that try
15:17
to compute at random points inside the domain.
15:20
I will explain this, but let me just say that.
15:24
Despite the the complex way of describing this.
15:29
Sorry bout.
15:32
This describes despite the complex way of describing.
15:36
It is a very simple concept that very,
15:38
very simple implementation and the industry loves it.
15:42
Becausw because of its simplicity,
15:44
so so companies like NVIDIA,
15:46
arm seasons, Emmons now already started using this.
15:50
In fact, that NVIDIA has already called parallel code that
15:54
they give for free.
15:55
It's pretty good code.
15:58
With almost parallel perfect parallel efficiency on their GPU's and
16:02
so on South,
16:03
so the industry likes it and also the academic people
16:06
like like it now and the reason the academic people
16:09
like it is because they say it's not good enough
16:12
so they want to improve it.
16:14
In fact, recently someone student of a friend of mine
16:17
gave a talk to my group and said that they
16:20
improve it by 11,000 times and I wasn't.
16:22
I wasn't offended. I was very happy to see all
16:26
these people having a job.
16:28
And writing good papers. So anyway,
16:30
let me just explain the concept here using this burger
16:33
equation.
16:34
This various equation with Viscous Burgers Equation.
16:37
So just implement first set up the domain is at
16:40
a time domain and the domain the space domain.
16:42
We never discretize this variables.
16:44
These are continuous variable 60 and we take automatic differentiation.
16:49
Here's the star points at the data points.
16:51
The initial conditions. Here the boundary conditions and you can
16:55
see that on purpose I don't have a lot of.
16:58
Boundary data. I have some boundary data and that will
17:02
be my first term in my loss function.
17:05
Now the second term is the residual F.
17:07
The residual of this of this burgers equation and I
17:11
need to evaluate it at random points inside the domain
17:15
using like let's say a space filling curve or Latin
17:18
hypercube or some important sampling for high dimensional PDS.
17:22
But random points inside your your domain.
17:25
The extra domain. You can evaluate this NF points.
17:29
So in terms of. Implementation and encode it tensorflow.
17:32
You define the Neural Network,
17:34
which I had here on the left.
17:38
One here is just one line to define it,
17:40
and then you need basically to work on the right
17:43
hand side and the right hand side is only three
17:46
4 lines transferred.
17:47
The neural network. Take the derivative first of the neural
17:51
network in spacetime to eyes.
17:53
Someday map put it in the loss function,
17:55
then call your favorite Adam or otherwise minimizer.
17:58
So by doing so that you can see that within
18:01
an hour a graduate student,
18:03
even a professor, I joke these days that even professors
18:07
now can implement their own ideas.
18:09
And you can get solutions.
18:11
Pretty good solutions. Pretty good looking solutions of burgers equation
18:14
in an hour starting from scratch,
18:16
not knowing anything about numerical methods,
18:18
so that's the convenient part I talked about.
18:21
Of course there's a lot of danger with that because
18:23
the solutions the optimizer is will not give you a
18:26
unique solution.
18:26
There's lots of dangers. That's why there's thing there's a
18:30
lot of room for improvement.
18:32
To show you this, I don't know,
18:35
let me ask. My chair seed.
18:37
Can you hear me please?
18:38
Yes I can hear you very well and you have
18:42
not seen this picture before right?
18:44
I don't think so.
18:47
So this is another another extension of pin,
18:50
we call it the generalized domain decomposition.
18:52
It says space time domain decomposition.
18:55
So here we have two neural networks.
18:57
One is the blue, the Blue Sea where the dolphin
19:00
swims and the other one is the dolphin itself.
19:03
It's an arbitrary, it's a gimmick to show you that
19:06
we can.
19:06
We can decompose the domain into subdomains.
19:09
The Blue Domain is 1 neural network.
19:11
We called Neural Network won the dolphin domain is another
19:15
neural network.
19:16
So we can decompose FaceTime.
19:18
We can simultaneously work on both domains,
19:20
so I will speed up the operation by a factor
19:24
of two.
19:24
If I do it in 100 hundred pieces,
19:26
I speed up the operation.
19:28
The operation by almost 100 because I can get almost
19:32
parallel speedup and the reason I get parents without is
19:35
unlike these continuous Gallery methods where I had to compute
19:40
fluxes and continuous galerkin's dealing with stiffness matrices and so
19:45
on.
19:45
Here the extension is very,
19:47
very simple basically. We want to simply make the residual
19:50
of the conservation law continue,
19:52
so that's very easy. So you can have convex non
19:55
convex surfaces and so on.
19:57
And you stitch these solutions together very simply by penalizing
20:01
the jumps of the residual,
20:03
which of course is 0.
20:04
How you have to make it continues in the process
20:07
we created a parallelism of course as I said,
20:10
but also we create great flexibility.
20:12
For example here I can have a different neural network
20:15
on on the Blue Sea and I can have a
20:17
different neural network.
20:19
On their dolphin I can use different activation functions and
20:23
so on and down.
20:23
Here you can see that because I control the number
20:26
of points on the boundary,
20:28
I can have a really seamless integration in space time.
20:31
That's what I wanted to make.
20:33
The point that I have parallelism time,
20:35
which as you can appreciate for initial value problems,
20:39
is not trivial at all.
20:40
You can use Arial, but there are other problems where
20:43
that's not so easy,
20:44
especially hyperbolic problems to do that.
20:48
OK, so that's.
20:50
After one extension I will talk about that later.
20:53
This summarizes generalized domain decomposition.
20:56
Again, if you have an arbitrary domain like this,
20:59
you can have multiple subdomains and then it's sub domain.
21:02
This is especially good for multiphysics,
21:04
multiscale problems, where you have bonded layer.
21:07
You have multi rate for example if you have chemistry.
21:10
Later on I will show you what happens if I
21:12
have chemistry with hypersonic problems.
21:14
Very difficult problems, but this gives you an extra flexibility
21:19
for it.
21:19
Let me show you a couple of applications.
21:22
This is something we published in science last February.
21:26
We call it hidden fluid mechanics because we pretend that
21:30
we have data,
21:31
not only their noisy data,
21:32
but their data on an auxiliary field.
21:35
I don't have a day any data on the velocity
21:38
in the pressure?
21:39
I mean, there's never lost the pressure in fluid mechanics.
21:43
Of course I have Daytona passive scalar,
21:46
for example if I have.
21:47
If you follow clouds, if you have if you have
21:50
smoke.
21:51
I have data on the smoke,
21:53
had data on the dye,
21:54
something that visualize what's going on behind an object.
21:58
Like here I visualize a Patch of my domain of
22:01
interest.
22:02
That is this Patch and I have a video of
22:05
that.
22:05
OK can use this video to extract the velocity and
22:08
pressure there,
22:09
so as long as you have some gradients of the
22:13
die like I have here and some time snapshots,
22:16
then you can have. You can compose a neural network
22:19
which is multi physics problem,
22:21
namely. The passive scalar and the Navier Stokes equations with
22:25
incompressibility constraints very easy to in this framework with LaGrange
22:29
Multipliers too to include any type of constraints.
22:32
But basically using the data,
22:34
the auxiliary data, then you go to primary data.
22:37
That's why we call this hidden fluid mechanics because it's
22:40
sort of an analog.
22:41
It's not exactly, but there's sort of an analog of
22:44
a hidden Markov process.
22:46
You know data is trying to infer the primary state,
22:49
and that's what we do here,
22:50
down here for one snapshot for the velocity and the
22:54
temperature.
22:54
They infer dynamics is within 1 to 2%.
22:57
What you would get if you knew everything,
22:59
and if you were solving the follower problem.
23:02
Of course, here we solve the inverse problem together with
23:06
the with the physics.
23:07
If you have this die going around the object,
23:10
you can see I can even compute the forces on
23:13
the object,
23:14
which is something useful if you do experiments in a
23:17
wind tunnel or a water tunnel and do that now.
23:20
This is another rail post problem.
23:22
At timely we got this from a YouTube video.
23:26
We downloaded that's how we became friends with low vision
23:29
and German company.
23:30
Big company on lasers and visualization.
23:33
So we said OK, can you estimate the can we
23:35
actually infer the velocity field around the mouth and the
23:39
nose of this person?
23:40
This is actually our collaborators Thomas Bergen La Vision and
23:44
when he wears a mask or without the mask and
23:46
here I just show you that we can quantify totally
23:49
in time space the pressure field.
23:51
This is a pressure puff that comes out of his
23:54
mouth.
23:54
We can quantify it. If he wears a mask,
23:57
there's very little pressure field.
23:59
You can see highly reduces a little bit from the
24:02
side because he didn't use his mask very accurately,
24:05
but basically velocity field, which we also computed 10 times
24:09
small,
24:10
smaller with. With the mask,
24:12
this is a another I'm not going to show you
24:15
a lot of applications today.
24:16
I like applications but I just want to make the
24:19
point.
24:20
Working with real data. This is ultrasound testing of materials
24:24
from Wright Patterson Air Force Base.
24:26
They use that technique for the wings of airplanes and
24:29
composites.
24:30
This is an aluminum alloy.
24:32
There is a surface crack you can see that you
24:35
put in ultrasound module here and you send them ultrasound
24:38
to the surface cracking or it will reflect the ultrasound
24:42
and from that you can.
24:43
See where the crack is,
24:45
how you quantify exactly the crack is we use physics
24:48
informed neural network here by having the wave the wave
24:52
equation in this simple problem you can use the elasticity
24:56
full elastic equations. If you are,
24:58
it's an immersive crack, but in this case this was
25:01
a hacker phone by a dozen teams at the last
25:04
year in DARPA there was twelve teams.
25:06
As I said, none of the teams were able to
25:09
do it because they said we didn't have.
25:11
They didn't have enough data.
25:13
We found we had too much data.
25:16
We only use 10% of the data of the ultrasound
25:19
data and we nail this problem because we were able
25:23
to actually quantify the wave speed.
25:25
As you can see up here on the right.
25:28
And if you quantify the wave speed,
25:30
then you can exactly determine where the crack is.
25:33
The size of the crack,
25:35
the orientation and everything. OK,
25:37
now the key to this was of course to include
25:39
physics,
25:40
and by including the physics you don't need so much
25:43
data,
25:43
you also have noise and you regularize the problem instead
25:47
of using the arbitrary weight regularization using neural networks.
25:51
So but another another important development was that we use
25:54
what we call adaptive activation functions,
25:57
and I want to show you what this is.
25:59
We had done a lot of works in the last
26:02
three years on this adaptive activation function.
26:06
So as you know, in neural networks we have one
26:10
nonlinear.
26:11
Activation function website called Sigma before and that is the
26:15
same for all neural networks.
26:17
However, there's no reason for all neurons to fire at
26:21
the same rate.
26:22
There's no reason why all layers fire at the same
26:26
rate,
26:26
so we can make we can make the activation functions
26:30
layer specific,
26:31
or we can make it neuron specific and we can
26:34
introduce a parameter for each one.
26:36
Here A and here I can use a parameterized different
26:40
activation functions to show you.
26:42
That every neuron could have its own mind went to
26:46
fire.
26:46
Now this a becomes also hyperparameters.
26:49
So now my trainable parameters using stochastic gradient descent will
26:53
include awaits the bias and new parameters.
26:56
These are not that many parameters,
26:58
so even if every neuron has its own firing rate,
27:02
it's still it's a small problem compared to the weights.
27:05
For example, where you have all these combinations.
27:10
But but you can do a layer wise,
27:12
so that's actually adds maybe 1020 or 50 parameters at
27:16
most.
27:16
So in fact we found from practice at that layer
27:20
wise activation adaptive activation is even better.
27:24
This factor in here is the factor that the user
27:27
can use to be more aggressive or less aggressive on
27:32
this.
27:32
Just an example here to see how this.
27:35
Let's say we try to learn this function and if
27:38
you use adaptive activation function after a few iterations you
27:42
can nail the function.
27:43
In fact, you can recover it.
27:45
It's fully a spectrum here on the right totally for
27:48
this low high frequency even discontinuity it generates the long
27:52
long tail spectrum.
27:54
You can totally recover it with adaptive activation function with
27:58
a fixed activation function.
28:00
You can see what happens you get stucked into a
28:03
minimum.
28:03
And you're there. But this is a loss,
28:06
but you can see that eventually with adaptive activation function
28:11
you can get the loss tourists more value.
28:14
We have used that for benchmarks like Missy for and
28:18
so on,
28:18
and we beat everyone. So is there a theory behind
28:22
it?
28:22
There's actually theory. One of my postdocs jacked up and
28:26
Maya jacked up,
28:27
and Kenzie Kawaguchi from MIT sale.
28:30
They prove theorem.
28:33
That shows that adaptive activation functions depending on the specific
28:38
case eliminate,
28:39
but minimum, which is big deal.
28:41
Of course, for training and so so so so it's
28:44
very useful,
28:45
especially if you have noise.
28:47
If you have rough landscapes and so on.
28:50
This is very important. There is another part of the
28:53
theorem that don't have it here that shows that effectively
28:57
what you do with your own wise activation function is
29:01
like you have a second order STD.
29:03
Without paying the price for occasion.
29:06
So as you know, people like Jordan and so on
29:08
have done great job in high order methods except nobody
29:12
is using them in practice 'cause they're very expensive if
29:16
added to compute occasions and so on.
29:18
So you can use it.
29:19
That sort of later stages,
29:21
but this one allows you from the very beginning to
29:25
get hired.
29:26
OK, I want to kind of summarize what I said
29:28
here.
29:29
We develop a lot of different methods.
29:32
By far by now, but I have show you anything,
29:35
just standard pins and X pins their pins for high
29:39
speed flows where we can bring conservativity variational pins,
29:43
stochastic pins and blah blah blah blah.
29:46
But here here it says stochastic pins.
29:48
We use guns if you don't know guns.
29:51
Guns is a really great concept.
29:53
Is this adversarial training of a generator that uses arbitrary
29:57
noise as an input?
29:59
It generates fake data that discriminatory,
30:02
which is the target network that you want to.
30:05
Train.
30:07
Both generated data in real data we have shown in
30:11
this paper in Cisc which again take it took three
30:14
years to publish this paper 'cause people don't like to
30:18
see new things we we realize that we can actually
30:22
dial in any stochastic process if you can dial in
30:25
stochastic processes,
30:27
you can definitely do stochastic piddies.
30:29
This is an example of a PD that we solve
30:32
the stochastic PD.
30:34
It's the problem is that app is such that I
30:37
know something about the right hand side which is stochastic
30:41
excitation.
30:42
I know a little bit about the solution,
30:44
some some measurements somewhere, and we know something about the
30:49
hydraulic conductivity we apply this to.
30:52
Real problems like this Hanford side,
30:54
which is the largest nuclear waste site in Hanford close
30:58
to in Seattle,
30:59
northwest of the United States,
31:02
where apinan alias where directing the center.
31:05
We had an application, so this is kind of introducing
31:09
uncertainty in stochastic PD.
31:11
Now we modify the standard gun to have also.
31:14
A generator for all the stochastic feels there right hand
31:17
side and maybe the boundary conditions generator for the conductivity
31:21
and you so then the discriminator.
31:24
See this concatenated vector of realizations from the stochastic fields
31:28
plus some real data.
31:29
This would be real measurements.
31:31
For example if you were looking at polls media,
31:34
there will be like 1000 wells which their data actually
31:38
from this huge nuclear waste site.
31:40
And so on. South. What was the benefit?
31:43
From this? I will not go through the this paper,
31:46
but we could. We were able to go 220 dimensions
31:50
and eventually 10,000 dimensions using a big computer,
31:53
but within quite bitter curse of dimensionality a lot of
31:57
people talk about that,
31:59
but we made the cost almost linear,
32:01
quadratic at most with the number of dimensions.
32:05
Variational pincers, nice mathematical underpinning,
32:08
tweet it say. Let's say we're trying to solve this.
32:13
Passion or passion equation you can integrate by parts once
32:20
or twice and and you can mix spaces.
32:25
One of the interesting things with if you use their
32:28
petrol galerkin approach,
32:29
you can have the neural network being the approximation space
32:33
and there's a good reason for for that I show
32:35
you this adaptive adaptivity built in in that nonlinear space,
32:39
and then you can have the best functions being smooth
32:42
functions.
32:43
There's an elegant way of doing this.
32:45
In fact with simple Ray lewenstein functions you can find
32:48
yes,
32:48
yes, questions that are coming in.
32:50
But my suggestion is that we take all the questions
32:53
at the end,
32:54
is that OK? Yes, yes.
32:55
That 'cause I want to.
32:56
I want to show you the Depot net and I
32:58
cannot afford to just exactly right,
33:00
so let's see.
33:02
This is to show you hear that thank you,
33:04
Steve. This is to show you that actually you can
33:07
write Analytically and do some theory.
33:09
You can actually with a revenue function.
33:12
You can write Analytically also for a couple of more
33:16
layers and cut up you appreciate if actually how this.
33:20
Big spaces how these residuals look and how these are
33:23
different methods and so on.
33:25
So variational pins is interesting.
33:28
Let me briefly talk about the math of pins.
33:30
This is a big frontier.
33:32
This is, I think this audience would be really interested
33:35
in that there is this approximation.
33:37
Especially I will not talk about here,
33:39
but approximation when you have this.
33:41
What I 5D data I said earlier,
33:43
the multiple data, not all data are the same level,
33:46
not. And how you bring that in the nonlinear approximation
33:51
theory?
33:51
But the main issue here is generalization,
33:54
and I want to mention the two papers too.
33:58
Seminole papers by the Chair Submiss Rahu.
34:03
Motivated, he was motivated by some of these applications of
34:07
pins and produced perhaps the best two papers on generalization
34:11
already of pins.
34:12
Generalization error refers to the fact that although you're in
34:16
this neural network,
34:18
this HM space and you can do a good job
34:20
from the approximation in the universal theorems of approximation,
34:24
you can minimize this by making a really big and
34:28
expressive neural network,
34:29
making this box bigger and bigger cloud going closer to
34:34
the.
34:34
So the true solution, then there's always this approximate this
34:38
generalization error becausw you replace your Journal.
34:41
You can never exhaust the training space you can ever.
34:44
You have to use a.
34:45
You go from a integral to Quadratures.
34:48
That's very important error, of course,
34:50
as always, the optimization error.
34:52
The fact that you, which is another beast,
34:54
but you can see now how this analysis of this.
34:59
Approximations for PDS is much more even for function approximation,
35:03
but certainly for pieces is much more complicated.
35:08
The complication is that I'm interested in solving ill posed
35:12
problems so you don't have a standard well posedness you
35:16
have to find conditions or conditional well well posedness and
35:21
that's the paper of of Professor Mr that they exploit
35:25
that type of knowledge.
35:26
Another paper that that some of my collaborators worked on
35:32
is.
35:32
Is based on a different approach to this,
35:35
again tackling the issue of convergence generalization,
35:38
but for linear piddies and elliptic and parabolic's,
35:41
first of all we wanted to show that even without
35:44
the boundary conditions pins converge as you make the space.
35:48
HM, bigger it converge in the L2 space if you
35:51
include also the boundary conditions,
35:53
you can find the H1 converges another paper which.
35:57
We haven't put on the archive yet,
35:59
but it's complete. Just about to submit it.
36:03
Is using equivalence of norms and compactness arguments to derive
36:08
a priority in a posterior estimates in terms of the
36:11
solution or actually in terms of the functional which you
36:14
compute the posteriori and their two formulations,
36:18
that discrete formulation, the continuous formulation and this down?
36:22
Here is a sketch just standard when you have continuous
36:26
form lesson.
36:26
If you have the expressivity of the Neural Network you
36:30
can show convergence generalization.
36:32
We refer to generalization as compatibility compatible networks.
36:37
Or other conditions? I don't have time to go through
36:40
this very technical paper,
36:42
but it will appear in the archive version.
36:46
See I don't know if you can see this movie.
36:49
It's a huge movie so I don't know if you
36:52
can see I can see it perfectly and destroyer real
36:55
destroyer of the US Navy going through North Atlantic is
36:59
supposed to be autonomous. No crew,
37:02
no captain. Can we do it?
37:04
The answer, we cannot do it because if I were
37:07
to use CFD to predict how this the dynamic motion
37:10
of this destroyer takes about one week for one simulation
37:14
for one state, this is state 8 where.
37:17
20 meters waves are in front of you.
37:19
This aerial spectrum of the North Atlantic.
37:22
So standard conventional methods will never meet what you need
37:26
for autonomy.
37:27
For cars or airplanes, and so on,
37:29
or the ship. So we want to use neural networks
37:33
to approximate to as an input will have a stochastic
37:36
input as as the stochastic.
37:38
See elevation as an output.
37:40
I will have the sekeping motion dynamic degrees of freedom,
37:44
so that gives me the opportunity to talk about the
37:48
second theorem.
37:49
That second theorem is by Shannon Shannon.
37:52
It just so happened that Junping Chen,
37:55
who developed this at the same time at your side
37:58
Pango,
37:58
he is now 8687 years old and he sent me
38:02
an email This morning.
38:04
What is going?
38:07
Yes, an an he he proved his theorem.
38:09
I mean is is such and not many people know
38:12
about this theorem.
38:13
He proved this theorem and he published in IEEE Journal
38:17
in 1993.
38:17
So we started. Yes there is a network that can
38:21
approximate that.
38:22
How can you do it?
38:23
It turns out with today's technology we can use analysis
38:27
TM.
38:27
We can use Google's sequence to sequence and so on
38:30
to other stochastic input to different realizations to get like
38:34
3 degrees of freedom.
38:36
And you can see here.
38:37
This is now the first column is for the first
38:41
excitation,
38:41
second column, second excitation. This is all unseen data for
38:45
the train day for the train neural network that we
38:49
can predict the function of.
38:51
It completed whatever you want.
38:53
You can predict the time series here.
38:55
Underneath you can see the CFD in red,
38:58
but clearly we can do a very good job in
39:01
in how fast you can do this.
39:03
You can do this in .01 second on the laptop,
39:06
so now we're meeting real time requirements.
39:09
But this is more interesting and again,
39:11
going back to Chen and Chen at the same time
39:15
a couple of years later they prove something so profound,
39:19
which I these days I spend all my time working
39:22
on this topic,
39:23
and that is.
39:26
If you really want to talk about generalization,
39:28
you have to go to higher levels of abstraction.
39:31
Overparameterized neural networks may do a good job,
39:34
and there's a lot of controversy and lots of interest
39:37
in that,
39:37
but if you go to a higher level of abstraction
39:40
if you learn that operator,
39:42
you know you can do much better.
39:43
You don't want this robot here to go through all
39:46
the calculus that we did,
39:48
because then they have to calculate the way we try
39:50
to calculate using hexa flop computers.
39:53
So this robot will have a huge head like an
39:55
excellent computer.
39:56
With a thermal cooling system which is bigger than my
40:00
house,
40:01
so so that's not the way to go about intelligence
40:04
and Collective Intelligence.
40:06
And so on. Robots, the way to do it is
40:09
by system identification by offline training and so on.
40:13
So this theorem of challenge and basically extends the Cypress
40:18
CYP angle theorem in their own theorem to nonlinear operators.
40:22
So now the input is functions from this space,
40:26
compact Space V. The output are functions and G is
40:29
the mapping of the input of.
40:31
The output is a nonlinear continuous operator,
40:34
so the theorem of Chen and Chen says that any.
40:37
Mapping like that in a non linear operator user input
40:41
can be approximated by this double summation on a neural
40:46
network which is the output here which will call trunk
40:50
and an input which will call the branch so you
40:53
can see the input is here which you sample from
40:57
the space V and then the output is there because
41:00
because that's also the output itself is because you have
41:05
Geo view of Y is also a neural network.
41:09
This is for a signal layer,
41:10
but recently and this is in the paper that is
41:14
coming up in nature machine learning machine intelligence,
41:17
we prove that this could be also extended to an
41:21
arbitrary deep neural network.
41:23
It's not very different from there's some technical arguments,
41:27
the compact space. Of course in application of the compact
41:31
spaces is a big constraint,
41:33
but but as you will see,
41:34
most of my in my examples I take compact space,
41:38
which is a gauss random field which is not non
41:41
compact.
41:42
So it works with non compact,
41:44
but it's impossible to prove it.
41:47
OK, So what does this set up the top is
41:50
here.
41:50
Let's let's look at panel BI have panel be.
41:53
I observe my function UA second function you another 100
41:57
different functions you and then I observe the output at
42:01
a few points.
42:02
Usually observed the input at many more points than the
42:06
Outputs.
42:07
If I go back here to panel a,
42:09
then you give me a new function of from inside
42:12
or outside the space V and I want to give
42:15
you the output in terms of functions.
42:18
OK, now one of the main things when you have
42:21
the approximation theory theorem is as I said,
42:24
Generalization. We talk about Generalization.
42:26
The question is can you?
42:29
Design your networks with small generalization error.
42:34
And today, with there's so many papers out there,
42:37
one can control everything. The space V,
42:40
by the way, is very important as you can appreciate,
42:43
appreciate. So that's also an area of where research can
42:47
go.
42:47
I just want to give you a teaser.
42:49
If we use a Gaussian random field,
42:51
here is a Gaussian, for example with a kernel with
42:55
a correlation length L.
42:56
And if I do a sampling with uniform sampling,
42:59
you can define this Copa here in terms of M,
43:02
the number of points we're sampling in this space V.
43:05
And then you can find that Copa is of the
43:08
order of em square 1 /
43:10
M ^2 number of Points Square is sampling correlation Link
43:14
Square and then you can prove something like that that
43:17
this neural network that approximates this operator G is certainly
43:22
bounded by this error. OK,
43:24
now how materialize this? How do you actually implement this?
43:29
In practice, this is what this is.
43:31
Our contribution that we propose this neural network.
43:34
Let's consider this unstuck case,
43:36
which turns out to be the best in terms of
43:39
generalization error,
43:40
so there's the branch network that implements this big parentheses
43:45
that will relate to the input,
43:47
and then there's a trunk with the output,
43:49
and then here you can see we cross.
43:51
We just do this math here by crossing it on
43:54
P points PPR.
43:55
The points on the output K defines the output.
43:58
M is the number whereas.
44:00
I'm sampling OK so Depot net this deponents based on
44:03
rigorous approximation theory.
44:05
I want to give you a very simple example how
44:08
we actually do this.
44:10
I'll be done in about.
44:12
8 minutes. Is that OK?
44:16
Absolutely, George. Take your time.
44:20
OK, so this is an integral you can see that.
44:24
I used 10,000 functions. I only observe at one point
44:28
and I find that I can learn this integral operator
44:31
and hear the testing error and the training error is
44:34
basically about the same. If you use a another full
44:37
feedforward network or something else the you can see this
44:41
difference is the generalization error so I can learn these
44:45
functions. I can learn exponentially fast.
44:47
This is a pendulum, the input is U,
44:50
the output is 2 functions.
44:51
You can see that I can learn exponentially fast for
44:55
this network.
44:55
But then eventually I saturate,
44:58
was a break and very slow convergence.
45:00
I guess somebody out there maybe in the audience today,
45:04
is smart enough. The smarter than us to design the
45:08
neural network that will maintain this exponential learning rate.
45:12
You can do that for PDS.
45:14
I want to show you that I can do it
45:17
also for differential for fractional differential operators.
45:21
And here I use a puto derivative Caputo Derivative.
45:24
Let me just show you what it is.
45:27
A couple of derivatives. This side is defined up here.
45:30
The singular kernel used in initial value problems for fractional
45:35
piddies.
45:35
So the idea is, is again I use numeric standard
45:39
numerical methods to derive this outputs.
45:41
I use lots of functions as a nipple input and
45:44
then I can.
45:45
It's expensive to evaluate everything in fractional is expensive,
45:49
so can I amortize all that cost?
45:52
Can I learn Depot net that does that for me?
45:54
It turns out yes and this middle plot here is
45:57
to tell me to tell to tell you that the
46:00
mean square error can be very small,
46:02
but it depends so much how you actually.
46:05
Approximate the input space. For example,
46:08
if I use a GRFI get this green line.
46:11
However, if I use Poly Fracton Amiel's,
46:14
which is a spectral methods that I extended to like
46:17
legend polynomials,
46:18
or that they extended to fractional singular value problems.
46:25
Then this eigenfunctions of those,
46:27
if I present my function VI can get.
46:29
I can increase the accuracy tremendously,
46:32
and then I can. I can compute this fractional operators.
46:36
I can infer them very quickly.
46:38
This becomes very important for the Laplacian bounded domains.
46:42
Here, Myspace V is represented by their nikkah polynomials.
46:46
I don't need a lot,
46:47
but you can see again if I use.
46:51
In fact, one can use other Neural Networks CNN.
46:54
This is one of the cases we found in a
46:56
nice domain like this at the CNN.
46:58
I can map this to a nice image Square.
47:01
It could outperform actually deponent sodipo net is very powerful,
47:05
but it's not necessarily the best and I'm sure the
47:08
concept is very strong.
47:09
Other people can find I'll skip the stochastic differential equations
47:14
because it's very similar.
47:15
For example with color noise,
47:17
you can see what we're doing here.
47:19
The input is represented by.
47:23
Michael Cohen live expansion, so the input is bigger and
47:27
the output is of course space is.
47:29
The is becomes bigger I using this truncation I get
47:33
very accurate solutions for samples I want to show you
47:37
this.
47:37
This is deponent now learning Euler equations with non equilibrium
47:42
chemistry.
47:42
OK this is not a trivial even for double you
47:46
know this is a recent paper that my colleague shoe.
47:50
Producing in JCP. Give us the code to generate data.
47:54
Give us the code you can see.
47:56
We have three species who generate this data.
48:00
We only took hundred trajectories for training,
48:03
and we use 50 trajectories for testing.
48:06
Now here's what we did.
48:08
We use the conservation laws we started from initial conditions
48:13
from nitrogen,
48:14
oxygen, and velocities, and so on.
48:16
We can basically predict at anytime and in the next
48:20
XT.
48:20
We can predict all this and we can predict it
48:23
very,
48:24
very quickly what the input is is this parameterized form
48:28
of the jump in the pressure and the smoothness of
48:31
the of the jump.
48:32
Of course, from that you can generate a lot of
48:35
initial conditions,
48:36
so at ETA is very is very continuously between point
48:40
auto and .2 and basically you controlling the gradient of
48:44
this and then jumps in the left and right in
48:47
this Riemann solver for initial condition.
48:50
So that can give you anything.
48:52
But here's an example. After sometime I can predict,
48:55
for example, at Target time this red line starting from
48:58
initial conditions from black you can see for all species
49:02
I can predict again without solving the problem.
49:05
The Depot net can predict this with 10 to the
49:08
minus 5 accuracy.
49:12
We have some theory for it.
49:14
Let me give you a teaser for Conservation laws.
49:17
This is the theorem that shows error estimates on the
49:20
right hand side.
49:21
Here I have the number of points where we sample
49:24
the input.
49:25
Here P is the output,
49:26
1 / P. Then here's the parameters of the branch
49:29
network and these are empty.
49:31
Is the number of the trunk which is the output
49:34
network.
49:34
This looks good, except this middle term.
49:37
Cause here I have square root of the number of
49:40
points and this number of points is the is a
49:42
large number,
49:43
so we're trying to get rid of this.
49:46
Something in log or something like that,
49:48
but to prove convergence. Oh well,
49:50
this is an open problem.
49:52
I invite you to actually do the theory better than
49:55
us.
49:55
We're still working on it ourselves,
49:57
but if you have any ideas this is a good
50:01
place to tackle.
50:03
I. Let's see how much time do I have.
50:06
Let me take 2 minutes to show you this hypersonic
50:09
problem because I want to talk about Depot net dependent.
50:12
M sodipo Nets are building blocks for very complicated problems.
50:16
So this is a current problem of interests around the
50:19
world.
50:20
From the Chinese to Russians and especially the United States
50:24
so.
50:24
I'm looking here at the specific example of five speeches
50:29
now non equilibrium chemistry mark #8 to 10.
50:33
So I use standard solvers to learn Depot Nets for
50:37
all these fields separate fields OK,
50:40
I use 240 trajectories. I use a DNS to produce
50:44
the fields in this Mac #8 to 10.
50:47
You can see I can go from chemistry to velocity
50:51
from and temperature from chemistry.
50:54
Rusty, but these are. These are my datasets in this
50:58
market number I want to show you that it's not
51:02
trivial.
51:02
Look at the velocity, there's a very very steep structure
51:06
both in the species and almost eight orders of magnitude.
51:11
So here we did. Math #8 and 10.
51:13
Then we can predict from temperature,
51:16
velocity and predict all the chemistry in .1 second and
51:20
vice versa.
51:21
But the concept of Eminem is the following is multi
51:25
physics and.
51:26
Basically, in hypersonic problems another electric convection and so on.
51:30
You have some measurements somewhere,
51:32
for example for the slow scales.
51:34
For chemistry you may not have anything except the initial
51:38
conditions.
51:39
But here on the fly literally you may.
51:41
You may have some measurements for velocity and temperature and
51:45
pressure for example.
51:46
So you use this. You have this prefabricated pre train
51:50
networks offline in a split of a second.
51:52
You predict the Newfields. Then you pipe that through this.
51:56
Other Deponents which are pre trained and then with this
52:00
loop is a Standard neural network.
52:02
You now have a loss function which you test your
52:05
operators.
52:06
This is the operator here.
52:08
Basically, why are the fields you trying to predict measures
52:12
the the outputs of the operators and then these are
52:15
the real data and we have shown that we just
52:18
very few data to three data.
52:20
You can really solve multiphysics problems for unseen data and
52:24
this is a paradigm we have not published this yet.
52:27
This is the first time I actually second time I
52:31
showed this show this on Friday.
52:33
But but this shows you the possibilities that,
52:36
so in terms of accuracy,
52:38
the accuracy is pretty good in terms of speed.
52:42
We do 50,000 times faster than DNS.
52:44
In predicting this, let me finish here.
52:47
I want to treat you with my espresso coffee.
52:50
Now you can see the mug has the same thing
52:53
as my background that Sodipo net deponent.
52:56
That's it. That's a pin I want to tell you
52:59
a story about this,
53:01
because La Vision, they give us this data.
53:04
And we predict that the maximum velocity in my Espresso
53:07
Cup is .4 meters per second.
53:09
I said, You know, I'm in engineering.
53:11
I know about these things.
53:13
There's no way that this is .4 meters per second.
53:16
Such a slow flow. And then they said yes,
53:19
they only find .08 meters per second from there.
53:22
From there you know naked,
53:23
naked eye type of measurement.
53:25
Then they set up an experiment with PV particulare modulo
53:29
symmetry,
53:29
and they found that indeed the velocity that the neural
53:33
network predict was point 4.4.
53:35
The .45, so it was a blind comparison we did.
53:37
They were were very gratified to see this,
53:40
and there's a library for those of you who want
53:45
to start.
53:46
Playing with this for any deep XD work for any
53:49
at the level of Education ordered the level of of
53:52
real problems,
53:53
but it's a good way to get started.
53:55
This library I was just told by my collaborator has
53:59
5056 thousand downloads and it's about a year old so
54:02
I guess students are using it a lot.
54:05
I would like to think spring and I like the
54:08
thing seed for inviting me here today and I don't
54:11
know if there's time I can take some questions.
54:14
Yes absolutely. First of all thank you George for this
54:18
superb talk.
54:19
India Tour de force if I may say so with
54:22
lots of different concepts as as we expect from you
54:27
so.
54:27
Delighted that you gave this talk and it has a
54:30
lot of responsible already.
54:32
A lot of questions. Of course we all are busy
54:35
people,
54:36
so I tried to ask you some relevant ones.
54:39
I have the questions in front of me.
54:41
Someone is saying, for instance,
54:44
this one is a relevant question from Nick.
54:47
Nick does the work on STD SDS with Pi guns
54:50
work for Ito Stochastic Piddies as well?
54:53
Ah, really, good question. I was asked this before.
54:58
We don't. Have that yet.
55:01
There is something we published.
55:05
In fact, yesterday every every Saturday at 11 we have
55:09
an ad hoc group working on something like that,
55:12
but we have something that you may be interested in.
55:16
It's called generative. Uh.
55:20
And symbol regression. So we're looking at from unpaired data
55:24
looking stochastic,
55:25
ODS and so on. But we haven't.
55:28
We are considering this, but we don't have anything that
55:32
we can answer this meaningfully for you,
55:35
so, so most of the stochastic stuff we did so
55:38
far is more like color noise with truncated and so
55:42
not not what you have in mind.
55:44
Someone is asking about Caputo fractional derivative,
55:48
but maybe I just built up a little bit on
55:51
this question and ask you yeah,
55:54
this work on fractional. You had this enormous body of
55:58
work and fractional derivatives before and now with this F
56:02
pins.
56:02
So what's the message here?
56:04
Means is that there are certain models,
56:07
refraction, derivatives are good or yeah.
56:13
No, no, you're right, you're right.
56:15
So so so the main idea is to cut the
56:17
cost with fractional pins and that would you know that's
56:21
a good question because I need to clarify.
56:23
There's no automatic integration in any of these frameworks.
56:27
Are automatic differentiation, automatic differentiation?
56:30
Automatic integration is basically what I show you that one
56:34
day we were trying to use Depot net,
56:36
so I was thinking how we can build basically automatic
56:40
integration using this Depot net.
56:42
And of course, if you do that then you can
56:45
deal with integral operators computer.
56:47
Otherwise, in particular, computing the fractional Laplacian on a finite
56:52
domain is really very expensive,
56:55
and I have written papers on H metrics,
56:57
another pay and other people have done that and so
57:01
on.
57:01
But still we want to.
57:03
We want to compute and this comes from a real
57:06
need.
57:06
For example, I was trying to do terrible and Simulation
57:10
Subgrid and I wanted to use.
57:12
Nonlocal models for that, and it was almost impossible,
57:16
although was more accurate. It was very expensive to compute
57:21
the nonlocal subgrid models for closure,
57:24
so so we thought that we need to do something
57:27
like that to compute.
57:31
You know fractional appliance ingredients and so on to compute
57:34
it,
57:34
very precomputed, and then just spit it out very quickly.
57:37
That was sort of the motivation,
57:39
but maybe I I ask you a question,
57:41
just a quick follow up on that because you have
57:43
this very nice machinery of people net which I have
57:46
to understand each time you give me at give a
57:49
talk and I feel that I have to understand this
57:51
more now.
57:52
But so now you had this compactness assumption somewhere in
57:55
the approximation,
57:56
but you have been made available space the input space.
57:59
Here the input space, but we have been using sort
58:02
of Gaussian random fields and fractal normals and so on.
58:05
Bad this assumption. Is feeling right and at the same
58:09
excellent results.
58:10
So yes, so I think you will see also in
58:13
the challenge and if you go through their theorem you
58:17
will see you need that to do that.
58:19
Now if if anybody wants if you are able to
58:22
like with some weighted.
58:26
Space too.
58:29
To remove that and go to Infinity,
58:32
but waited. I think that that's probably just,
58:36
but we haven't been able to do it so,
58:40
but in in your right,
58:42
for example.
58:45
You know you want to do to learn Laplace transform,
58:49
right? Do quickly laplace transform so arbitrary functions and so
58:53
on?
58:53
That that's that's very important,
58:55
and so that's why we're pushing this for the applications.
58:59
But the theory itself doesn't cover us for that Case
59:02
No,
59:02
but of course this is theory is always late to
59:05
the party,
59:06
but another. It's very important.
59:08
I mean, if we can even not removed,
59:10
but like you some waiting,
59:12
waiting spaces too. That I think that will be.
59:15
A huge step forward also and and what did you
59:17
use for the input space?
59:19
For instance, for this compressible euler chemistry problems,
59:22
what was it? Was it a finite dimensional input space
59:25
that yeah,
59:26
finite dimensional space and one of the questions that we
59:29
ask that kind of really interesting question is what happens
59:33
if you extrapolate outside the map #8 to 10?
59:35
Let's say you go to model number 13.
59:37
It turns out actually, you know that's what Darpa's interest,
59:41
of course, and it turns out it works really well,
59:44
for example. In the worst case scenario,
59:46
we get a deponent to give you 20%
59:48
error. OK, without nothing now if you increase the number,
59:53
if you can reach that space V like I use
59:55
only 100 Projectors.
59:57
If I use 300 trajectories the error the extrapolation error.
60:01
That's the interesting thing. The Extrapolation regards down to 10%.
60:05
So you refine more the space.
60:07
The input space. Then you do a better extrapolation which
60:11
makes some sense,
60:12
but it's good to see that maybe depends on this
60:15
on the on the application.
60:17
Also. For example if I go on the other side
60:19
on Mark #7 or 5 or 4.
60:21
Because there's no chemistry, I cannot predict it very accurately,
60:25
of course, so it depends also on how smooth solution
60:28
extends to those.
60:29
So it is really interesting from the point of view
60:32
of the of the application of you,
60:34
as you can imagine absolutely yes.
60:37
So any further questions.
60:40
If not, I think we stop here.
60:42
We all have to go and we thank you.
60:44
Know there is one question.
60:45
Yeah, so it's for next week.
60:47
Yeah, thanks a lot. George again for this very sort
60:50
of as I said it was Real Tour de force
60:53
with so many different components and one really has to
60:56
think a lot in order to just get things absorbed.
60:59
So thank you very much and I think it's highly
61:02
appreciated what you're doing.
61:04
Charm. Thank you. Thank you.
61:06
Thank you very much. Thanks everybody.
61:08
Have a good day.
61:10
Bye bye child thanks adults.
61:11
Thanks for joining everybody.