The following content is provided under a Creative
Commons license.
Your support will help MIT OpenCourseWare
continue to offer high quality educational resources for free.
To make a donation or to view additional materials
from hundreds of MIT courses, visit MIT OpenCourseWare
at ocw.mit.edu.
OLIVIER DE WECK: OK so let's start on the material.
We're following the V-model and the empty boxes
are getting fewer and fewer every week here.
So we're on the right side moving up
toward lifecycle management and operations.
And today's topic is via V&V, verification validation.
And I also want to discuss the FRR
as one of the key milestones, and that's the flight readiness
review.
If you're not building something that flies,
this is your launch to market, right launch to market review.
Are you ready to launch your project or product to market?
The outline is-- we have quite a few things to cover.
So first of all, I want to drill into verification
and validation.
What's the difference?
What is their role?
What's their position in the lifecycle?
Then we'll spend quite a bit of time on the issue of testing.
What kind of testing is done?
Why is it done?
We'll talk about aircraft testing, flight testing,
we'll talk about spacecraft testing, but also some caveats.
Testing is not always the--
it's not free of challenges and difficulties.
Then I want to talk about technical risk management,
which is often covered in classes on project management,
but I think it's essential here as well as a system
engineer to have a good grasp on technical risk management.
So I'll cover the risk matrix, the Iron Triangle,
cost, schedule, scope, and risk, and then
I added a small section on system safety, which is
a very important topic as well.
And we'll finish up with discussing the FRR, the flight
readiness review.
So the readings related to this lecture
are sections 5.3 and 5.4 in the handbook, the System
Engineering Handbook, and then there's
a couple of appendices there, Appendix E And I
that are very, very helpful.
And at least one of these I'll mention in the lecture.
Plus one of the papers, and this is a paper about a decade old--
and I know a lot of work has been done on this since then,
but this is a paper by Professor Leveson.
Nancy Leveson was a colleague of mine
here at MIT who's really an expert in systems safety.
So the system safety model that she's developed
is the subject of that paper.
OK, so let's talk about V&V, verification and validation,
and how they fit together.
You've seen this diagram before, but I just
want to talk through it in some detail again.
So the idea is you start your project,
you're undertaking, in the upper left.
You do your stakeholder analysis.
Who are the stakeholders, the customers, beneficiaries,
the regulators, the suppliers, the partners on the project?
So you really have to do a good job doing your stakeholder
analysis then in order to write your requirements.
And I do have to say I've been very pleased, especially
with your assignment A-2.
You really dug into those 47 requirements for canned set.
You grouped them, you scrubbed them, you did a great job.
And the idea is that for each of these requirements,
you also have target values.
There are certain thresholds or target values
that have to be achieved.
And then you actually do the development.
You do the conceptual design, the detailed design,
and that's written here as functional deployment.
In other words, especially for the functional
and the performance requirements,
how will you actually implement those,
embody those, in technology, hardware, software,
and so forth.
So that's your intended function, your concept,
and then you're implemented design solution.
Now the question is, do you actually
satisfy A, the requirements, and do you
satisfy your stakeholders?
And it is possible that you satisfy the requirements,
but not the stakeholders.
So the way to think about this is
we're going to close the loop.
In fact, we're going to close two loops.
An inner loop-- and I put testing here.
Testing is really one of the ways
to verify whether you meet requirements.
There's other ways too, but testing is often
the most important.
So we close this inner loop.
So what we ask is--
we test our implemented solution,
our implemented design, and ask the question, did we deliver?
Did we actually satisfy the requirements
as they were written?
Do we satisfy the requirements as they were written?
And are these attainable?
Were these requirements attainable?
So this loop here, this inner loop, is the verification loop.
You verify whether you design as implemented satisfies
the requirements as written.
That's what verification is.
And then there's an outer loop where
you take your implemented design solution,
and you essentially bring it all the way back
to the stakeholders.
And usually that also means you're employing it
in a realistic environment, like in the environment
that the stakeholders will actually use the system,
not in a pristine lab environment.
And you have the stakeholders try out your system
and see whether they're satisfied, whether this
meets their original intent.
You remember the CONOPS, concept of operations?
Can you actually do the CONOPS the way
you had envisioned it in a realistic environment?
And that's what we call validation.
And that's the outer loop.
You see the difference?
So a lot of people who don't know system engineering who
have never been exposed to this when they hear verification
and validation, they think it's basically two different words
for the same thing.
It is it is different, it's not the same thing.
And then if you successfully verify and validate,
you end the SE process and you deliver, which is good.
So this is something I pulled out
from the handbook, which is the differences
between verification and validation.
And I'm just summarizing this here.
So one way to ask is, was the end product realized right?
Meaning, did you do the right thing?
Or did you implement it correctly?
So verification is often done during development,
so you verify components, subsystems,
you check if the requirements are met.
Typically, verification is done in a laboratory environment,
or on a test stand, or some environment that
allows you to very carefully control the test conditions.
And verification tends to be component and subsystem
centric.
OK, and then validation is the question
was the right end product realized?
Did you actually build the right thing?
Did you deploy the right solution?
This is often done-- so validation
focuses more on during or after system integration.
It's typically done in a real or simulated mission environment.
You check if your stakeholder intent is met.
And it's often done using the full-up system.
It's difficult to do validation on a subsystem basis alone.
Typically, validation implies you've
got to use the whole system to do it.
Or, you basically use dummy subsystems.
You basically replace the actual subsystems
you're going to have with something
temporary so that you can go back to the stakeholder
and give them as close to the real experience
as they'll have with the actual system.
OK, so that's essentially the distinction here.
So I want to do a quick concept question on this
to see whether this point, this distinction, came across.
So here's a link, SE9VV, these are all caps.
And what I'm listing here is different test activities
or different type of activities.
And I'd like you to check the box here
whether you think this is verification,
whether this is validation, or you're not sure.
All right, testing and handling of a new car in snow conditions
in Alaska.
90% of you said this is validation,
and I would agree with this.
So many car companies, I think all car companies,
once the vehicle has been finished essentially,
the design, it doesn't go to market right away.
There's a very extensive-- usually
it's at least six months of field testing of a new vehicle.
And you go to the desert where it's very sandy and hot,
test your air conditioning systems right at the limit.
And then you go to really cold climates.
In Europe they go up to Sweden and Norway.
And here we tend to go up to Michigan, Minnesota.
And so the idea is that you really
utilize the vehicle in a extreme environment,
but that's realistic of actual operation.
So I agree with this.
Frontal crash test in the lab.
Most of you said it's verification,
and I would agree with.
So those very standardized crash tests
that you've seen in some of the commercials.
The vehicle is prepared, instrumented,
you have the crash test dummies.
And then you have, typically, there's
at least three different kinds of crash tests.
There's frontal, there's side impact--
like the T crash-- and then there's
a rollover tests, that are more and more standardized.
And the test conditions in these are there
they're highly stylized tests.
They're very prescribed exactly the speeds, the angles,
everything is prescribed.
And real accidents, the variability of conditions
is much, much bigger than in these tests.
So I agree because the test conditions are so tightly
defined and constrained, this is verification, not validation.
Testing of a new toy in a kindergarten.
OK, so here you're--
and this is the toy companies, and they essentially--
before, again, they make a big million or billion
dollar decision to mass manufacture toys,
they will actually have kids play with them
in realistic environments.
And so I agree with that this is validation.
Vehicle emission testing.
Obviously, this was the big Volkswagen scandal that we had.
So basically, what this cheating that happened
is essentially software that was embedded in the vehicle, such
that when the vehicle experienced exactly the test
conditions of these drive cycles that are very well known,
very well defined, the vehicle would internally
switch or reconfigure to a verification mode
and really emphasize low emissions
at the expense of fuel economy.
And as soon as the vehicle would detect
that it's in a more general driving conditions,
it would essentially switch that mode off.
So verification, again, is on a dynamo in the lab.
Satellite vibration testing on a shake table.
We'll talk about this.
This is often-- we refer to this as shake and bake
in the spacecraft business.
The spectra, the load spectra, that
are put into these shaking tables
are, again, very stylized and different
for each launch vehicle.
So here we can debate a little bit whether--
is it closer to validation, because the actual test
conditions are so much adapted to each launch vehicle.
But I agree, this is primarily a verification activity.
And then the field testing of the Google glasses.
So you basically produce an initial batch of your product
and then you give it to like lead users.
You have them try it and give you feedback.
This is much closer to validation.
So I think by and large your answers here are very good.
And the real distinguishing factor
is whether this activity happens in a lab,
in a very controlled environment,
under stylized conditions, or whether you're actually
going out in the field, in a realistic mission
environment with real users or real potential users that
are not especially knowledgeable or not specially trained
about the system.
So, very good job.
I think most of you really understand that distinction.
OK.
So let me talk briefly about the--
yes, please, Veronica?
AUDIENCE: Got it today.
OK, are there any has that really
bridged the gap that really could be seen as both.
So I'm thinking about products in particular
that are to be used in a lab setting where you have
a very specific kind of user, where meeting the requirements
is more about how the tool is employed,
and I see the user in that sense is part of the system.
So I'm wondering if in a more clinical sense
there's an action that is both validation and verification.
OLIVIER DE WECK: Yeah, and so you said clinical.
So I think there are situations where the distinction is not
as sharp, not as clear.
You said clinical, so I think in hospital,
you know for medical equipment, like if you
think about surgical equipment and things like this,
where it's very hard to really--
it's very hard to do verification in a stylized way.
The only way to really check it is
to have the equipment embedded and used in a pilot study,
for example, in a hospital.
And in that case, because the human is so involved,
and it's not a general consumer product,
but it's really a tool for specialists,
the only way to really check your requirements
is to actually embed it in a realistic environment
to begin with.
So whenever it's very difficult to design,
very specific, isolated tests where
you can check for each of these requirements one by one,
you almost have to move straight into validation.
And I think in medical equipment that's often the case.
Yeah, go ahead.
AUDIENCE: Would you say that for a spacecraft
really true validation isn't possible?
You have to recreate the conditions in some way,
in some kind of a laboratory.
OLIVIER DE WECK: I think it depends
on the novelty of the spacecraft.
If you're launching something like a standard communications
satellite where you've launched dozens before,
and you know the actual pitfalls and the operating conditions,
you've experienced failure modes in the past
and eradicated most of them, I do
think that you can do a lot in verification.
But then I'll show you one example
of a spacecraft we've actually talked about before,
where there's going to be a lot of residual risk.
And the first time it's deployed,
people are going to sweat, because there's still
a lot of unknowns to be resolved.
OK, so let's look at the product verification process
in particular.
This is from the--
so this is the product verification process
from the NASA System Engineering Handbook.
So what are the inputs?
The end product to be verified.
So you have to have the artifact that you're going to verify.
The specified requirements baseline,
you need this as a reference.
What are you going to check against?
The product verification plan which
is essentially your test plan.
What test cases are you going to run?
How long are you going to run them?
How many repetitions will you do?
And then product verification enabling products,
which would be test software, test equipment,
things that are not part of the product itself,
but our enabling of the verification process.
You then, essentially, go through this process,
and what are the outputs?
The verified end product, the product verification results,
so these would be test protocols,
things like this, product verification report, and then
product verification, any other work products
that come out of it.
So you could have, for example, discrepancy reports.
You failed some tests.
Well, that would be an important output.
And then the question is, is this significant enough
that you have to go redesign or retest?
Or is it a minor issue that you can waive essentially
to move to the next stage?
So let me just give you a quick example
here from my own experience about this verified end
product.
One of the things on the Swiss F18 program that we did
is not just by airplanes and equipment, but also
models of the plane itself.
In particular, finite element models, very detailed
finite element models of the structure.
And these models were very expensive.
Like, some of these models were millions of dollars.
And so I got a phone call, I was a liaison engineer
at the time in St. Louis.
I got a phone call from Switzerland saying,
this is crazy.
How can we be charged millions of dollars
for this particular set of models?
And I said, yeah, that seems pretty expensive, so I'm
going to go negotiate this.
And so I started negotiating, and I
guess either I'm a bad negotiator
or it was really, really clear why these were so expensive.
The reason these models were so expensive
because they were on the right side not on the left side.
So every one of these models that we were purchasing
had been verified using actual physical tests.
So every location was guaranteed under the load conditions
to produce a stress and strain prediction
at that location that was guaranteed to be correct within
plus or minus 5%.
So the model had been very carefully calibrated and tuned
against physical reality as opposed to a finite element
model that's just anybody can make a model
and put some load cases and boundary conditions on.
And you don't know how closely does this mean.
So there's a huge difference in value
between a product, a model that has gone through verification
where at the end of it there's actually a report,
there's a protocol, there are data
that says all these features, all these requirements
that you had against it have actually been checked.
This is a certified product.
And that's the main reason for the price,
because the actual process of verification
is very, very resource intensive.
So even though when you look at it physically, you might say,
I can't tell the difference between pre-verification
and post-verification because physically it's the same.
But in actuality, there's a huge difference
because once it's been verified and certified
against a set of requirements, it's
a much more valuable asset.
Does that make sense?
So keep that in mind when you think about these products
on the right side.
Now what are the types of verification?
So tests we'll talk about, so you're physically testing.
But there's other ways to do it through analysis,
through demonstration, and through inspection.
So analysis essentially means you're doing a calculation
with a mathematical calculation or a simulation that
satisfies you that this requirement is met.
And you're doing this with the input parameters
into the simulation are as accurate as possible
based on the physical reality of the system you have.
But for whatever reason, either because you
don't have the funds for it, or you
can't simulate the operating conditions well enough,
you have to do it through analysis.
Demonstration essentially means you're operating the system,
you're demonstrating the functions that you need,
but you don't necessarily have a lot of instrumentation
on the system.
And you don't certainly do destructive testing.
In other words, a demonstration simply
means you're operating the system as intended
and demonstrating physically that it performs its purpose.
Inspection essentially means you are
physically inspecting the artifact
either visually inspecting--
there's also a lot of techniques called
NDI, nondestructive inspection through with X-rays or eddy
current sensors.
You're checking for the lack of manufacturing flaws
or [INAUDIBLE], whatever it is.
But inspection essentially is you're not
physically operating the system, but you're
inspecting the artifact to make sure that it satisfies
a certain set of requirements.
And then testing typically means that you're putting a stimulus
into the system.
You're operating the system under some test conditions.
You're recording data, which you then
analyze in terms of comparing that to your prediction
or expected behaviors.
So these are analysis, demonstration, inspection,
and tests.
They are all different ways of verification.
Yeah?
AUDIENCE: How do I know when I'm supposed
to use more than one type at the same time?
I mean in and or or.
OLIVIER DE WECK: Yeah, that's a good point.
There's no real general rule of this,
but in general, I would say the more crucial, the more critical
a particular requirement is to the operation of the system,
the more intense the verification will be.
Whether that's just using one of these types,
you know you just run more tests or more different tests,
or you doing a combination of inspection and testing.
There's no there's no general rule in terms of two
out of three, or two out of four,
but the purpose of the V&V plan, the verification and validation
plan is--
and you did a little bit of this in A2.
You did a little bit of thinking into how would we actually
verify this requirement.
The purpose of a V&V plan is to say for each requirement which
of these four methods are we going to use for verification,
and then actually write down each test
that you're going to perform, what kind of equipment
you'll use, what kind of test conditions.
It's a lot of work.
In fact, I think it's fair to say
that the people that do this kind of work, verification
and validation, are typically different people
than the people that do the writing of the requirements
or that do the actual design work.
This is a pretty specialized activity
and the people are a little different.
If you've met people who would do testing or quality
inspection, they're quite different.
It's a different mind set up.
Go ahead.
AUDIENCE: According to what's happening in ESA,
actually this ADIT will be imposed in the specification
prior to your proposal.
And it will give you to a minimum requirements
to just test against.
And they will give you a rough matrix for every requirement
line, whether it's [? ABIT, ?] and then
you have to answer with a validation intense plan,
usually, unless your agency and you are defining
the specification and you have to do it,
and it's mostly based on experience.
And it's some people that have really lots of knowledge that
then make these specifications.
But I think for all of you engineers
here in the next 10 years, you'll
be just hoping that not so many of these ADITS in the specs
you will get, because you will have
to answer as part of the specification actually.
OLIVIER DE WECK: Yeah, and what--
AUDIENCE: I'll just demonstrate it.
OLIVIER DE WECK: Well, and the point you're making,
Voelcker, is that this is a contractual requirement.
This is not optional.
AUDIENCE: [INAUDIBLE]
OLIVIER DE WECK: Yeah.
AUDIENCE: It's not optional and it has to be followed,
the pricing, right in front of [INAUDIBLE]
OLIVIER DE WECK: Yes, very good points.
So the outputs of all of this are discrepancy reports,
if there's any discrepancy reports, waivers,
the verified product itself, and then
the compliance documentation, which
is essentially your test protocols, et cetera,
et cetera.
And as you can imagine now if you had 47 requirements
with can set, and then by the end-- [? Uonna ?],,
what would you say the number of requirements
that we ended up with at the end in A2 were closer to 100,
right?
Most people were around 80, 90.
OK, now imagine-- the good thing is
if you can do tests that actually
check multiple requirements at once, that's a good thing.
If you can do tests that help you verify
multiple requirements through the same tests,
you can save some money.
But the whole testing strategy, the contractual requirements
that Volcker was mentioning, it's a big, big, big deal.
It's a really critical part of system engineering.
OK.
So in terms of the lifecycle phases
where this fits in, most of this activity
happens during phase D.
So you remember this was the NASA lifecycle model,
and so phase D is system assembly, integration, test,
and launch.
So much of the testing that we talked about happens
during phase D. So the system has been fully designed,
it's been assembled, it's been integrated
the way we talked about last week,
and now you're really putting this system through its paces.
And so this phase D is intense, it's expensive.
And if something goes wrong, it sends you back
to the drawing board often.
And you have to do--
you have to figure out whether a failed test, a failed
verification, is a showstopper.
If it is, then you have to redesign the system,
you have to retest.
But if it's a minor thing, then you
might be able to request a waiver
and you can say, OK, we didn't achieve this requirement,
or we failed this test, but we think it's a minor issue.
And instead of holding up the program,
we're going to get a waiver for it, which
means you get an exemption essentially,
and you can move on.
And whether or not a waiver can or cannot be granted is a big
deal and that goes under risk management,
which we'll talk about in a few minutes.
Yes?
AUDIENCE: I had a question on--
so if you're a system integrator,
and you created statements of work for other people
to procure a large optic, or something,
they go and build that [INAUDIBLE] requirements,
and then they do all the verification and testing,
and then they provide all that documentation.
And then like from my experience with anything like procurement
out stuff, that comes back in your in house,
and you go to assemble it, and you essentially
do a lot of that verification and testing
again to double check that supplier.
There is a little bit of a conflict of interest,
obviously, with that supplier doing the work
and also verifying their own work.
Is there any good way to get around that?
So it seems like a very expensive process.
OLIVIER DE WECK: So my experience-- basically,
you're talking about separation of powers.
The good suppliers, they will have internally
separation of powers.
In other words, the people that are
doing the testing and the Q&A, they're
usually people who really enjoy finding mistakes and faults.
And that's why when I'm saying--
I'm trying to be diplomatic when I said it, but people even
in software, people who do software verification,
they love to find bugs.
They love to find problems, because that's their job.
And so the good suppliers, it's in their own self-interest
not to do shortcuts.
Now if you don't trust that, and you
do all your same testing and Q&A again,
that's a duplication of effort.
The way I've seen that done effectively
is that you, as a customer, say you're
going to buy the subsystem or the engine for,
example, from it.
What you do is you send liasion people,
you send representatives, who are
knowledgeable people to the supplier while the testing is
being done.
And so they're present when the testing has been done.
They're very involved with it.
And therefore, you don't have to do it twice.
So there are ways around this.
OK, so what I did here is just search for the word test
and the list of milestones.
And where does it come up?
So the first time it really comes up in a major fashion
is at the CDR. OK, so let me just read this to you.
So the CDR demonstrates the maturity of the design
and is appropriate to support proceeding
with full scale fabrication, assembly, integration,
and tests.
So in other words, even at the CDR
you're blessing the final design,
you should say something at the CDR
about how the testing will be done.
In fact, test planning often is way before the CDR.
But at the CDR, you should you should really
talk about the testing.
Then we have so-called TRR, which
is a test readiness review.
And so for each major test, you would have a separate TRR.
The TRR ensures that the test article, hardware software,
the facilities, the support personnel, the test procedures
are ready for testing data acquisition reduction, meaning
data post-processing and control.
And then at the system acceptance review, the SAR,
that's when you essentially transfer
the ownership of the asset.
And that's at the SAR, at the system acceptance
review, that you're going to review,
not just the product and its documentation,
but all the test data, the analyses
that support verification.
So at CDR, you say this is the testing we'll do.
At the test review itself, you say
everything is ready for the tests to happen,
and then you do them.
And at the system requirements review
you look backwards and you say, what tests actually happen?
What's the documentation?
What were the results?
Are we ready to own the asset now?
Does that does that make sense?
OK, so with that in mind, let's talk about testing itself.
What kind of testing there are, and so testing
is one of the four methods of verification.
And it's the one that we often spend the most money on.
So this is from the handbook, section 5.3.
This is basically an alphabetic list of the testing
that we typically do.
And I'll just highlight a few here,
and then I have a group exercise for you.
So aerodynamic testing, burn-in testing,
which is often done with electronics.
Make sure that you use you burn-in your electronics,
you get them running at the right conditions, drop testing,
pressure testing, pressure limits, thermal testing,
G-loading, human factors testing, thermal testing,
manufacturing random defects--
that's when you do nondestructive inspection--
thermal cycling, vibration testing, and so forth.
So this is 20 or 30 types of testing.
And then within each there's even subtypes,
so there's a lot of different-- and there's
a whole industry actually that is primarily
focused on providing test equipment, sensors,
data logging equipment.
It's a big industry not just in aerospace, but throughout.
OK, so I'd like to do a little turn to your partner exercise.
And the question is I want to ask you what kind of testing
have you been involved in the past.
And if this was like in product design, product development,
that's fine.
If it was for an internship, but even at the University
itself, if you did some experimental work
and experimental testing as part of research, that's fine too.
You can talk about that too.
So what kind of testing have you been involved in the past?
What was the purpose of the testing?
What were the challenges?
What went well?
What were the results?
Maybe if it didn't go well, talk about that too.
All right, good.
So let's see, we're going to go back and forth.
So who wants to start here at MIT?
Who has a good story to tell?
Go ahead.
AUDIENCE: So I worked on the ground station
side of the Lunar Laser Communication Demonstration
program that recently flew.
So I was involved in assembling an integration,
but also doing verification testing in the lab
and at our field site, but then we
did validation when we be moved out to the field
site in New Mexico.
So I was involved in the whole process and it was neat to see.
And we had to go into the clean room
a few times to adjust the optics, because we
saw that there weren't meeting requirements.
OLIVIER DE WECK: The reflector that
was left by the astronauts, are you
using the reflector that was left on the surface?
AUDIENCE: No, this was--
we were using just like [INAUDIBLE]
like optical alignment stuff in the lab.
And then when we were out at the field site,
we were utilizing guidestars to align optics.
OLIVIER DE WECK: So what was--
what went well?
Was there a big difference between indoor and outdoor?
What surprised you in these tests?
AUDIENCE: Yeah, so once you can do
the alignment of the individual telescopes which
were 20 inches in diameter, you can
do that well in the laboratory, but then every time you
assemble and disassemble the system,
you change the alignment of them relative to each other.
So there was a lot of attention paid
to making sure that we could replicate the alignment
to a certain extent.
So that was very difficult in a laboratory setting to get done,
but once we did that, we were able to have
fair confidence that when we were out in the field
that we could match that.
OLIVIER DE WECK: Very good, very good.
What about EPFL?
AUDIENCE: Well, I did an internship
in an aluminum-roll product factory.
So basically, I was doing natural science there.
And there was a whole bunch of tests to do.
And well, all the tests was about heat treatments
and different tempering.
And actually, the alloy that was already produced in the factory
was not the best of what we can have of it.
And it [? applied ?] to change the [INAUDIBLE]
with the heat treatment for a few seconds,
naturally, on the line of production.
Adding this amount of time was totally critical
because it was continuous.
And the rolled aluminum, if it spends
a bit more time in the oven, it would melt.
And that's a bit like for the Swiss plane, actually.
Like I discussed with my boss, saying
that we should maybe change the original power meter.
But at the end, it was really critical to change something
on the line because it could have cost a lot,
like in the modification of the oven or the general machine.
PROFESSOR: So were those tests successful?
Were these heat-treatment changes
eventually implemented on the line?
Or did the tests reveal that it would be too difficult?
AUDIENCE: Unfortunately, I don't know because I
finished my internship before.
PROFESSOR: OK.
Well, you should find out whether it worked out
in the end.
Very good.
Back to MIT, any other examples people want to mention?
Test experiences?
Yes please, go ahead.
AUDIENCE: We bought the CASA-295.
It's a small cargo aircraft.
And we get we got involved in the development
of its simulator.
It was pretty different, the simulator.
Because as the aircraft has no fly-by wiring,
it is pretty light.
So lots of hydraulics to de-motion.
And they brought the flight model from the factory.
And we applied the flight model to the simulator.
But it was not real enough.
So we had to go for flying, like 60 test flying points.
And we have to go back to the simulator
to apply these points to tailor the simulator to meet reality.
PROFESSOR: I see.
So the purpose of this testing-- because the plane
itself had already been certified, it sounds like.
AUDIENCE: Yes.
PROFESSOR: It's Spanish, right?
Spanish airplane?
AUDIENCE: Yes.
PROFESSOR: You tested it specifically
to get flight dynamics and other data to then
tune the simulator to be more reflective of reality.
AUDIENCE: Exactly.
Because the flight model from the factory
was not close to reality at all,
PROFESSOR: Very, very cool, very interesting.
So different purpose, of not testing for certification
of the first airplane, because it had already been certified,
but to get the simulator to be matching more closely.
AUDIENCE: It was development for the simulator.
Because it was sold afterwards as a type delta simulator.
So it was the development of the simulator.
PROFESSOR: OK.
Great, thank you for that example.
This all sounds pretty good.
Anybody involved in test failures?
You know, things that didn't go well?
Yes, [? Narik? ?]
AUDIENCE: Well, it was an interesting experience.
We were designing a wind turbine that we
were 3-D printing in undergrad.
And we had certain requirements on the wind turbine.
And we were supposed to test in the wind tunnel afterwards.
What happened was that the wind turbine matched our performance
prediction fairly closely.
But the generator and the electrical power system
that the test operators consisted of
wasn't designed to handle the current
that we were outputting.
So we caused a small fire.
PROFESSOR: OK.
[LAUGHS]
So this was the test equipment itself?
AUDIENCE: The test equipment.
PROFESSOR: Not the artifact you were testing failed--
AUDIENCE: Yeah.
PROFESSOR: --but the test equipment around it,
because overload.
AUDIENCE: The interesting point was
that we had no control over the test equipment.
It was managed by the university.
So within the requirements that they gave us,
the power output possible didn't match what they had.
PROFESSOR: OK, great.
Great example.
So the test equipment and the test artifact
need to be matched to the test conditions.
Excellent, good.
I do hope that you those of you that have not
had a lot of test experience, that you get to experience it.
It's a lot of work, slow, tedious.
But in many cases, despite modeling and simulation,
there's still a big role to play for actual testing.
OK, so let's talk about aircraft testing.
Typically we distinguish between ground testing
and flight testing.
Weights and balance, I had some experience
with this on the F-18 program.
You think this is the most trivial testing there could
possibly be, you just put an airplane on a scale
and that's it.
Well, it turns out it's actually more involved than you think.
First of all, airplanes are very big.
They're heavy, multi-tons.
And typically it's not just one scale.
You have several scales you put on the landing gear.
So the scales need to be properly calibrated.
If you have differences in calibration
of the different scales, you have an issue.
You need to determine the mass.
Not just the mass, but the CG.
And then the most difficult thing
to experimentally determine, at least in a 1G field,
is the inertia matrix, if you need to experimentally get
the inertia matrix.
Do you remember your Ixx, Ixy, Iyy?
The inertia matrix is tricky because you typically then
have to suspend the airplane.
And just the presence of the cables and the suspension
will pollute the real inertia matrix.
And you have to subtract out the effect of the suspension
system.
So something that seems super trivial, weights and balances--
you just stand on the scale in the morning, there it is--
is actually very tricky.
And there are people, that's all they do.
They do weights and balance testing
for spacecraft, aircraft.
And it's basically a science.
Engine testing, I'll show you some pictures.
This is done in what's called the Hush House.
So Hush House is heavily insulated.
You run an engine through all of its test conditions,
its operational conditions.
And then you integrate it into the airplane
and you run it outdoors.
Fatigue testing, this has been a big issue on the Swiss F-18.
But in general, making sure that the airplane
can satisfy all the static and dynamic structural load
conditions.
Avionics checkout, this is very, very involved.
As we get more and more displays,
mission-control computers, all of the avionic suite
needs to be checked out.
Essentially every function, every button, every menu item
needs to be tested.
And the tricky thing is interactions
among different pieces of avionics.
So you can't just test each box in isolation.
You also have to look at the interactions
of different pieces of avionics, the flight control
software, or the flight software that
is loaded in each of these avionics boxes needs to be
in the right configuration.
It's a very big combinatorial challenge
to do avionics checkout these days.
And then finally, pre-flight testing.
So this is everything you can do on the ground, run the engines,
taxi with the airplanes, basically
turn all the equipment on, turn it off, do the cycling.
You could do a lot of testing before you actually fly.
Flight testing itself falls into different categories.
So flight performance testing, rate of climb,
range, can you meet each point in your prescribed performance
envelope?
Stability and control, this is where test pilots typically
earn their living putting airplanes
into stall conditions, recovering from stalls,
trimming.
Flutter testing is a big deal.
So flutter is a phenomenon whereby at high speeds
you have a coupling between the structural deformations
of the airplane and the actual excitation of, for example,
the wings.
Flutter can be very dangerous.
If you hit a resonance at high speed
you can actually destroy the airplane
because of an instability.
So flutter testing is also very, very interesting
and very tricky.
And then finally, this is primarily
for military airplanes, weapons testing, both guns, missiles,
bombs, live fire testing--
sometimes also using airplanes that are towed--
simulated targets, and then LO stands for a Low Observability.
So this is essentially all the new generation
of military airplanes have measures to reduce their radar
signature, or even make them invisible or quasi-invisible
to radar.
And you know, a lot of this stuff is classified.
But actually checking that an airplane is invisible on radar
or has truly low observability, there's
a lot of testing involved in that.
And that's also quite expensive and very involved.
So let me show you just some pictures
that I've collected over the years.
This is a wind tunnel test model.
This is a model that was developed
as part of the F-18 program.
This is about 1995, vintage.
This model, it's a subsonic wind tunnel model.
And you can see in yellow, you have
all these probes and radomes and things like this.
So it's basically to check whether any modifications you
make to the airplane will affect its performance in airflow.
This is for wind tunnel testing.
This model, by the way, just building this model
is about half a million dollars.
It's very accurate.
It's very precise.
This is a picture--
yes?
Go ahead?
AUDIENCE: Is that model full scale or half scale?
PROFESSOR: No it's, I want to say, like 1/8 scale, something
like this.
Yeah.
OK, here's the Hush House that I was talking about.
This is in St. Louis.
So you can see that the airplane is not painted yet.
And only one engine at a time.
So the engine is being, in this case, with full afterburner,
you can see the airplane itself is secured with these chokes
here.
And there's load cells in these chokes.
So as you fire up the engine, you
can measure the thrust by the load cells that are
attached to these chokes here.
You also see that there's these cables running
in and out of the airplane.
So all the sensors, everything, and the engine
is put through its full different operating profiles.
And a lot of sensor data is recorded
to make sure that the engine responds appropriately,
it has the right thrust for the right throttle setting,
the fuel consumption, all the temperatures in the engine,
that everything is nominal, essentially.
Live Fire Testing, this is a Maverick missile being fired,
an air-to-ground missile.
As you can imagine, there are special ranges and test sites
for doing this kind of work.
So in the US, one of the most well known as China Lake,
out in California.
You have to reserve months and sometimes years ahead.
So if you want to do like a live fire test campaign,
you have to reserve the range at least 18 months
to 2 years ahead of time.
Because a lot of other services, a lot of other programs
are using the same facilities.
In Europe, it's a little harder.
Definitely in Switzerland, because the country
is so small and dense and highly populated,
you can't test live missiles in Switzerland.
You can do guns, air to ground.
But in order to do missile testing, typically
that's done here in the US, or in a more limited fashion,
in Scandinavia, like in Sweden, in northern Sweden,
there are some test ranges up there.
This is the most expensive kind of testing you can do.
So a single test like the one shown here,
a single test like this will probably
cost several million dollars.
Not just the airplane and the weapon itself,
but all the test procedures, the protocols,
airplanes that observe it from all kinds of angles.
It's very, very involved.
And because it's so expensive, you
will typically only do it for something new
that you haven't done before.
Either a new weapon, or a new weapon
integrated on a new platform and so forth.
And obviously, it's very interesting.
But it's very involved.
Yes?
AUDIENCE: Are they just testing accuracy,
Or that the two things work together?
What are they looking for, really?
PROFESSOR: The first thing you look
for is does the weapon fire?
So do you have all the electronics?
All the signals?
The wire bundles?
Did you get it right?
Is there an end-to-end functionality?
That's number one.
Number two, safety.
Does the weapon separate properly from the aircraft?
The worst thing that can happen to you
is if you release the weapon and it collides with the airplane.
And so you can see, the various angles and release conditions
are very tightly prescribed.
And so there's a separation, has to be proper.
And then the third, of course, is accuracy.
So within each of these tests, there
are multiple sub-objectives that you would test for.
But safety always comes first.
OK, any questions?
This was a little bit military-aviation heavy.
If you're testing, whether it's a CASA airplane or a new Airbus
or Boeing commercial airplane, many, many months of testing.
They actually fly the routes.
You'll fly New York to Singapore, to London.
You would actually fly the real routes.
You would record fuel consumption,
a lot of parameters.
Some of this testing is not very exciting.
It's many, many, many, many hours in the air.
But the key is that you have a lot of instruments and sensors
during these tests that you may not have during regular flight
operations, to really make sure there's no surprises.
The airplane flies at least as good as the requirements
that you promised your customers.
And even then, when you think about what
happened to the Dreamliner, the 787
had a lot of battery problems.
Because they used a lot of lithium ion batteries.
There were overheating issues.
Some of these problems didn't show up in testing.
They only showed up in early operations,
once you had a fleet going.
So it's not a guarantee because you're
doing a lot of testing that you're going
to catch all the problems.
But you want to catch as many as you can.
Yes?
AUDIENCE: So my question was about the risk
posture for larger airliners, for Boeing, the Airbuses.
So for military aircraft, there is an escape method
for the pilot.
But for these larger aircraft, how much analysis
do they do before they decide to go ahead and put
a person inside?
They do fly by wire beforehand?
Is that possible. for such large aircraft?
PROFESSOR: So that's where the ground testing,
pre-flight testing becomes very important.
So you basically taxi for many, many hours.
All the flight control surfaces, all the engine,
you have the Hush House testing.
So you essentially try to do as much
as you can on the ground before you do the maiden flight.
All right, let's move to spacecraft.
And it's kind of a similar thing.
You can distinguish the ground testing
versus on-orbit testing.
So the ground testing is really not that different, weights
and balance.
The biggest thing is if your satellite is heavier
than the launch capability of the launcher,
you have a real problem.
So the mass constraint is even tighter in spacecraft.
Then, a lot of testing on antenna and communications.
This is typically done in anechoic chambers
in the near field, and then later in the far field.
Vibration testing, that's the shake part.
Thermal and vacuum-chamber testing, that's the bake part.
And then you also have pre-launch testing,
so off-pad and on-pad.
Off-pad testing is the satellite or the spacecraft
has already been shipped to the launch site,
and it's hooked up.
Like a patient in the hospital, it's
hooked up to a lot of cables and power and cooling and so forth.
And then when it's on the pad, it's pretty limited.
So on the pad means the satellite or the spacecraft
is already integrated into the launch vehicle.
It's on the launchpad.
And then that amount of testing you can do is very limited.
So that's when we say off-pad, on-pad is,
is the spacecraft already been integrated
on the launcher or not?
Once you launch to orbit, you got your eight minutes
of terror.
And hopefully the launch goes well
and the spacecraft is released into its initial target orbit.
And then you do a lot of other tests, like thruster testing.
Can you do station keeping?
Can you turn on and off the thrusters?
You deploy all of your mechanisms, your antennas,
your scientific instruments, and then your communications,
communication and instruments.
And we'll talk more next week, but this typically
is called commissioning.
You're commissioning a spacecraft before you actually
turn it over to the users.
And that commissioning phase could
be anywhere from a few days to several weeks, or even
a couple months.
And again, you don't want to randomly put commands
into your spacecraft.
These test sequences, deployment sequences,
are very, very, very carefully worked out.
Every command, the order in which
you send the commands have been worked out ahead of time.
They've been simulated.
And all you want to see here is confirmation
that the spacecraft behaves as planned.
Some pictures, so this is what typical spacecraft integration
testing looks like.
So this is in a cleanroom environment.
You have people in bunny suits.
And the idea is to not damage the spacecraft
while you're doing the testing.
This is a picture of the Clementine spacecraft.
This is a radio frequency anechoic chamber testing.
So you see these funny cones here.
These are essentially foam cones.
And the idea is to prevent multipath
to prevent echoes in the test chamber,
to test all of the antennas, EMI,
electromagnetic interference and compatibility,
charging and discharging of the spacecraft.
This is one of the failure modes,
is that you have high electrostatic charges that
build up on a spacecraft, create a lot
large voltage potential across the spacecraft.
Some spacecraft have failed because of that.
So all of these things you want to test in a very controlled
environment.
James Webb Space Telescope, I'll send you a link.
I'll send you a link through email.
There's a simulation of this on-orbit deployment.
It truly is amazing.
This spacecraft will be launched in a box, essentially.
And the deployment sequence is very carefully choreographed.
First, typically, you deploy your solar panels
because you need power.
Because you're only running on battery initially.
So if your battery runs out before you've
had a chance to deploy your solar panels
and get fresh power into it, you're in big trouble.
So typically, solar panels first.
Then communications.
And then you start deploying the other subsystems.
So for James Webb, also very tricky,
is this-- this is called the Sunshield.
It's essentially thin layers of insulation.
And the geometry is very important.
The primary mirror, the secondary mirror,
all these things need to be deployed
with very, very high precision.
And it's even to the point where this particular spacecraft is
so lightweight that it cannot support its own weight in a 1G
gravity field.
So there is no way to test, end-to-end,
the full deployment sequence on Earth.
The first time it will happen is in orbit.
Now, they've tested sub sequences or scaled models.
For example, the Sunshield has actually
been deployed at a smaller scale in a 1G field,
but never the full thing.
So this will be kind of scary, after an $8 billion investment.
So let's hope for the best, 2018.
So testing is good, testing, testing, testing.
But testing also has its caveats.
So caveat means limitations, essentially.
So testing is critical, but it's very expensive.
Think about test rigs, test chambers, sensors,
DAQ is Data Acquisition Equipment.
All this stuff is very expensive.
And if you can reuse things between different programs,
that helps.
But still, how much testing should you do of components?
So one of the comments, who mentioned
the vendor, the supplier?
One of you.
You talked about it.
And this is a key question.
Do you trust the parts that come from your vendors?
Or do you retest everything yourself?
Calibration of sensors and equipment,
if you've done some testing and you
forgot to calibrate your displacement sensors,
your thrust sensors, you didn't calibrate
them or they're out of calibration,
that's a big problem.
That's a big problem.
So before you start your tests, make sure
that all your sensors are properly calibrated,
or you can get the wrong conclusions.
This is a mantra that's well-known.
Test as you fly, fly as you test.
Fundamentally, this means that the configuration
of your item--
a spacecraft, aircraft, medical device--
the one that you test should be the same configuration as what
you're actually going to fly.
And it's often failures occur when the test went well,
but then somebody tinkered with it
and modified it before it actually flew.
And that change actually caused a big problem.
So make sure that your test conditions
reflect the actual operations as closely as possible.
Simulated tests, what do we mean by this?
So simulated tests use dummy components.
Maybe your full spacecraft or aircraft isn't ready yet,
you don't have all the pieces.
So you can still start testing, but you
have to replace the missing pieces with dummy components.
At least, they should reflect the right mass distribution.
But maybe you can do more.
Simulated operations, so the 0G versus 1G,
is it representative?
And then, what's often true is that you pass all your tests
and then you still have failures in practice.
And the failures often happen outside of the test scenarios
that you had tested.
So you have to be ready for that.
But try to avoid that.
So here's from Appendix E. This is called a Validation
Requirements Matrix.
Essentially what this is, is an organized way
to organize your V&V activities, in terms
of what's the activity?
What's the objective?
Which facility or lab will you do it in?
What phase?
Who's in charge?
And what are the expected results?
It's pretty straightforward.
It's just a table to organize these activities.
And then appendix I is your more formal V&V plan.
This is a suggested outline for it.
And I'll just say this.
The degree to which you take Verification and Validation
seriously and the resource you make
available for it are critical for success.
So how many dedicated Q&A personnel?
What is the interaction in working with suppliers?
Are you planning ahead for these tests?
How close are you getting to actual end-to-end functional
testing?
Can you piggyback on existing facilities and equipment?
How well do you document all the outcomes
and follow up with discrepancies?
And my last comment here is this work is often not glamorous,
except for some of the very cool flight testing
that I showed you.
Most of this work is really hard work.
It's very detail-oriented.
It's not glamorous.
But it's essential.
If you cut corners, you often pay the price for it.
So any comments or questions?
We'll take a short break, like a five-minute break.
But any questions about testing, verification, validation?
Yes, go ahead.
AUDIENCE: Not really a question, but I
wanted to say that also, flight testing is not so fancy.
I mean, many people think it is.
Well, if they actually fly, maybe it's fun.
But there are not that many.
And you need to prepare them weeks and weeks ahead.
And it's actually very, very boring.
Because you need to make sure that you don't waste time
at all.
Well, at least you can waste a little bit more time
in the lab.
So it's very stressing and not so fun to fly.
PROFESSOR: Do have experience with this?
AUDIENCE: I wasn't flying, because you
need to be certified.
But I was preparing.
And I think it's even worse than testing in the lab.
PROFESSOR: Yeah, yeah.
Which airplane, or which system were you involved with?
AUDIENCE: I was in with the power plane system
for [? Airbus, ?] particularly.
And the aircraft was an A330.
PROFESSOR: A330, OK.
Great.
But I think it's healthy to have this experience.
It really makes you humble.
And you also see, for the things in design, did you design well?
Did you design for testability?
Really, I highly recommend for every one of you
to try to get on some kind of test campaign,
at least once in your career, because it's eye opening.
So thank you for that comment.
Any comments at EPFL?
Any of the students?
[? Voelker? ?] Did you want to add something?
Katya?
Go ahead.
AUDIENCE: Yeah, I guess the comment is,
it seems like sometimes you can meet
all of the requirements in the verification process.
But when you get to the validation part,
for example, maybe the customer has some expectation
that the range for the time of flight
would have been on the maximum edge
and you were on the minimum edge.
But actually, it seems like sometimes you
can meet all the requirements but they're still
not going to be happy.
When do you find that middle ground?
And it's going to be constant, to continue iterating this
over and over again.
Do you try to involve them earlier on, during the testing
process too?
How do you handle that, that difference?
PROFESSOR: Yeah, it's tricky, you know?
So that's when you do need contractual agreements
in place.
You need to have the requirements, baseline.
You need to have the contractual agreements.
And hopefully, any problems that occur
will not lead to some kind of legal dispute.
But sometimes that's unavoidable.
But as a designer, as a manufacturer, unless you
have agreements in place and clear baseline,
how do you decide in the end, is it successful
or is it not successful?
And if there's problems, try to isolate these problems
and say, OK here, by and large, the testing went well.
But we have like, three, four, five issues
that need to be addressed.
And you can tackle these issues one by one.
But if you don't have a contract in place that's
really a good contract, if you don't have a clear requirements
baseline, and then if you don't have
a good relationship with your customer,
you're setting yourself up for big, big problems.
[? Voelker? ?] Go ahead.
AUDIENCE: Yeah.
There is also the one big difference
between commercial operators or commercial customers
that are becoming more and more frequent
compared to the institutional ones.
And often I remember, for some of the [? Global ?] [? Store ?]
[? Iridium ?] series, the customer was only accepting
the hardware six months after they had been commissioned
in orbit.
So there you have a validation that is still
your responsibility in orbit.
You can't go and fix it, but it still has to work.
And he was retaining up to 10% of the full contact value,
even up to the N minus-2-tier-level suppliers,
until he was satisfied it was working in orbit.
So these considerations, it's really the proof of the pudding
when you have to test this up there and can't fix it.
So there's no recall of a satellite
constellation, like, sorry we messed up with the software.
It's not possible there.
PROFESSOR: Yeah.
And of course, those terms and conditions
you've probably negotiated years before.
So you've got to be careful.
That's where risk management, which is actually--
thank you, [? Voelker, ?] for that comment.
Let's take a short break.
And then we'll talk about risk management, which is really
what this ends up being.
So let me talk about risk management.
And this is actually quite prominent in the System
Engineering Handbook.
This is right in the middle here of your System Engineering
Engine, Technical Risk Management, Section 13.
Why is it important?
So first of all, what is risk?
So risk is the probability that a program or project
will experience some undesired effect or event.
And then the consequences or impact or severity
of that undesired event should occur.
And so think of risk as the product of probability times
impact.
And the undesired events could come from a number of things,
technical, programmatic.
So cost overruns, schedules slippage,
safety mishaps, health problems, malicious activities--
cybersecurity is a big thing these days--
environmental impact, failure to achieve
the scientific or technological objectives or success criteria.
And so technical risk management is,
therefore, an organized systematic risk-informed
activity centered around decision making to proactively
identify, analyze, plan, track, control, communicate risks
to increase the likelihood of success of a program.
And so what risk really does is measure the future
uncertainties of achieving your program goals-- technical,
cost, schedule goals--
and think of risks in a holistic way,
all aspects of the technical effort,
technology maturity, supplier capabilities,
performing against plan, and so forth.
And so the idea of risks is that risks have some root cause.
There's something that gives rise to risks.
And then the actual quantification of risks
happens in terms of likelihood and consequences,
which are kept separate, separate dimensions.
So the first thing to think about
is where do risks come from?
Where is the source of risks?
And I want to show you a couple of models
for thinking about this.
The first one is this idea of layers of risk,
that there are layers of risk.
And I want to credit one of my colleagues
here at MIT, Don Lessard from the Sloan School, who really
developed this Layer of Risk Model
and applied it to different industries.
So there's a version of this for the oil and gas industry.
You could make a version for medical, medical technologies.
So this is the version for Mars missions,
so if you're designing a new Mars mission, a new Mars Rover.
So you have, in the bullseye here,
the narrow interpretation is technical or project risk.
So the airbag technology.
If you're using airbags for deployment, will it work?
The rover/motor performance, are you
going to have software bugs?
Those are the risks we typically think about.
And the idea is you have high influence
over these risks as a system engineer, as a project manager.
Then you have a layer around it, which we call
industry or competitive risks.
Will your contractors perform?
Will you have budget stability?
And then there's sort of more country and fiscal risk.
So in the US, we have a budget cycle.
We have four-year administrations.
Will you get your budget?
What is the priorities between human and robotic space
exploration?
And then working with international partners.
And then there's another layer of risk,
which are called market risks.
So if you think in Mars missions, who's your market?
Well, the science community and maybe the public.
So will these missions hold their attention?
Are there new science requirements?
We discovered there's water, probably flowing water on Mars,
maybe with a lot of perchlorates in it.
It's not pristine water.
But that could change the priorities for your mission.
And then finally, the most outer is what we call natural risk.
So this would be things like cosmic radiation,
micrometeorites, uncertainties in the atmospheric density
of Mars as you're doing entry descent and landing.
And you have very low influence.
That doesn't mean you can't protect yourself or take
measures to deal with these risks.
But fundamentally, the occurrence or the probability
is something you can't really do much about.
So that's one way to think about risks.
And the seeds of risks is in these layers.
I know this is very high level, but I find this
to be a pretty useful model.
Yeah, go ahead.
AUDIENCE: So this references the influence
you have, not necessarily the amount that each of these
are a risk to the program?
PROFESSOR: That's correct.
And That will be program specific.
Just the stuff that's in the bullseye here,
you can do a lot about it, and perhaps
both in terms of probability and impact.
And then as you move further out,
there's less and less influence you have as a system engineer,
as a project manager.
Here's another way to organize your thinking around risks.
And this is around the "Iron" Triangle
and project management.
We talk about the "Iron" Triangle
of cost, schedule, and risk.
And we call it "Iron" because the idea
is that if you constrain all three too tightly,
it can be very difficult.
And it's also referred to as the triple constraint
in project management.
So the three dimensions here are technical risks, cost risks,
and schedule risk.
And in the center we have programmatic risk,
which means it's kind of the combination of all three.
And the idea that even if you do a great job on keeping
your budget under control, schedule,
and you're meeting your technical objectives,
you can still fail because the program as a whole
isn't doing the right thing.
Or the market that you had been targeting
is no longer really attractive by the time you launch.
The key idea here is that these risk categories are not
independent of each other.
So let me mention a couple of examples.
So cost risk might limit your funds.
And that could, in itself, induce technical problems which
cause you further cost risk.
So one of the big initiatives at NASA in the '90s
was the faster, better, cheaper program.
We're going to launch more missions, cheaper.
And out of 10 missions, maybe 2 or 3 will fail.
And then seven will succeed.
But we'll get more value out of this as a portfolio.
Unfortunately, it didn't work very well.
Because when the one, two, or three missions
fail out of your portfolio, the media and the public
focuses on the failures rather than the aggregate value
of the whole portfolio.
And eventually, that's probably the main reason
why faster, better, cheaper was abandoned.
So for example, we just talked about testing.
If you have a very limited budget,
what is the first thing people typically cut out?
What's the first thing to go?
Testing.
Testing is very important.
Sam asked me during the break, what's
your typical budget for testing in V&V activities?
And in many programs, it's very substantial,
you know, 40% of the budget, maybe 30%,
40% of the budget easily.
And so you start cutting out tests.
Well, what you do is you introduce technical risk.
And if you have failures because you didn't test,
that could cause you additional rework and more cost.
Similar, schedule slips can induce cost risk.
So as you slow down, you have what's
known as the standing army cost, right?
People are going to charge to your program,
even if it's at a reduced level.
And that will also increase your cost.
So lots of coupling here between risk categories.
This is a very useful Risk Management Framework.
It's essentially a controls framework.
And the idea is you start in the upper right.
You anticipate what can go wrong in your program.
So that's risk identification.
You then analyze these risks, in terms of prioritizing them,
which of these are important?
You plan to take action.
This is often called risk mitigation.
You track these actions.
And then you correct any deviations from your plan
and you communicate throughout, and you cycle through this.
So typically, risk management will
happen on a weekly basis, a monthly basis, at least
quarterly basis for big programs.
Now, how do you actually do this?
First of all, the risk ID and the assessment.
So the risks are typically brainstormed.
So you think about risks.
You have to imagine all the bad stuff that could happen to you
and your program, the probability
that these things will happen, and then
the impact or consequence if they
do happen based on the requirements, the cost,
the schedule, the product and its environment.
And this is where actually having
a mix of younger engineers and more experienced engineers
really comes in handy.
The experienced engineers, they will have
been through several programs.
They will have seen failures in the past.
They will really be able to point to potential risks
that less-experienced people may ignore or just not understand
how important they could be.
So the next step, then, is to aggregate these
into categories, typically not more than 20-or-so categories
or risk items.
Projects often keep so-called risk registers, which is just
a database or a list of risks.
If you have hundreds and hundreds of risks in the risk
register, it's too much.
It's just a long list and really it's
just a check-the-box exercise.
To really take risk management seriously,
you have to focus on few of the risks
that you think are important.
You score them based on a combination
of opinions and data.
And you try to involve all the stakeholders in the risk
management.
And eventually, risks are placed on this matrix
of uncertainty and consequence.
So let me zoom in on the matrix.
There are many, many different versions of the risk matrix.
This is one that NASA typically uses.
And I like this particular version for several reasons
that I'll explain.
But basically, the way it works is
you have the two dimensions, impact and probability.
So probability is how likely is this to occur?
And then impact, if it does occur, what will happen?
What's the consequence of that?
So of the two things that I really like about it,
the first one is that each of these levels,
there's actually some definition behind it.
Not just guessing at the level, but there's some criteria.
So for probability, a level 3 means
it's about equally likely that it will happen and not happen.
So a level 3 means it's about 50/50,
whether this will happen in your program.
And then 4 is very likely.
So maybe that's, I don't know, 75%.
And then near-certainty is like 90% or more.
Improbable is like 10%.
Unlikely is 20% to 30%, something like this.
And then more importantly, the impact so a level 1 impact
is negligible.
It has almost no impact.
A level 2 means your mission performance margins
are reduced on the technical side.
Do you remember margins?
I asked you to assign margins in assignment A2?
So it means you're eating into your reserves.
But you should still be able to meet all your performance.
There should be no visible impact.
Your safety cushion is less.
That's what level 2 means.
Number 3 means your mission is degraded.
So you can still do the mission.
But you're not going to hit all your targets.
4 is you lose the mission, but the asset is still recoverable.
So maybe you could try again in the future.
And then level 5 is a catastrophic failure
that involves a loss of mission and/or loss of crew.
On the cost side, you have some thresholds for cost.
Obviously these numbers have to be adjusted
for different programs.
A $10-million loss is a huge thing in some program
and almost like pocket change in other programs.
And then schedule, so a level 1 milestone would, for example,
be launch.
Good example of this was the Mars Science Laboratory,
their Curiosity mission.
Originally it was supposed to launch in 2009.
They missed that deadline, mainly due
to problems with cryogenic actuators.
They took a lot of the blames, the actuators.
But there were a lot of problems across the board.
They missed that launch window.
And they had to launch in 2011.
So that was considered, from a programmatic standpoint,
a level 5 failure.
Because you missed your main launch window and you
had to wait 26 months for the next one.
So that's good.
Because now when you assign a probability and impact,
you can really look at these criteria.
And it's easier to do that in a repeatable fashion.
The other thing is if you look at the colors on the matrix,
you can see it goes from 1, blue,
which means low risk, to 12, which is the highest risk.
So there's 12 risk levels.
But when you look at the matrix, there's
something peculiar about it.
So look closely at the colors.
And you'll see something special about this matrix.
Anybody notice what I'm talking about?
Let's see at EPFL, do you guys, when you look at those colors,
at the matrix, do you notice something?
Go ahead.
AUDIENCE: It goes from blue to red,
which is a light spectrum, with the blue the lowest radio
waves, waves, and red the highest ones.
PROFESSOR: Right.
AUDIENCE: [INAUDIBLE].
PROFESSOR: Go ahead.
AUDIENCE: It's due to impact is more serious than probability.
PROFESSOR: Right.
So it's asymmetric.
You see that?
It's asymmetric.
So the high-impact, low-probability corner
is weighted more heavily than the low-impact,
high-probability.
And that's intentional because it's been shown in the past
that things that are not likely to happen but if they happen,
they're really bad, in the past, people
have sort of pushed that away and ignored those.
So the purpose of this asymmetry in this matrix
is to elevate the low-probability, high-impact
events to be higher in the risk level
so that people pay more attention to it.
OK?
So most risk matrices don't have that asymmetry in them,
but this one does.
[SNEEZES]
And I like it because of that.
Bless you.
So the question then is, what do you do with this?
The idea is you do your risk management.
You identify your risks.
You place them on this matrix.
And then you track each of these risk items over time.
So here's your 12 risk levels, between 1 and 12.
That's the y-axis.
And then on your x-axis there's Time.
And for each of your risk items, people might disagree.
Some people on your team might say, hey, look,
this is not a big deal.
We've seen this before.
The impact is not big.
We have a quick fix for this.
We know how to deal with this.
And other people disagree and say no, this
is very, very serious.
You have to take it seriously.
So the idea is that for each risk item,
you have this optimistic, expected,
and pessimistic estimate of what is the true level of risk.
That's what these bars are.
And then you track it over time.
And depending on whether you're before PDR or CDR,
you could have very substantial risks.
And that's, I guess, OK still, as long
as you find ways to reduce the level of risk.
For example, by doing extra testing,
by changing your design to put in extra power margins,
bigger solar panels, redundancy, for example.
There's a lot of things you can do
to affect both the probability and the impact, radiation,
extra shielding.
And so the idea is that over time you're
going to reduce these risks gradually below some threshold.
So this red line here is like the acceptable threshold
of risks at that point in the program.
And then as you get closer to launch,
things should be below the threshold.
And if it's below this watch domain,
then you even don't track it.
You don't pay much attention to it.
If it's above this red line, you have a big problem.
You might have to stop the program or do a major redesign,
or repeat a major milestone.
And some programs have been canceled because they just
couldn't get these risks under control.
So the idea is that gradually you transition.
And you do this by actually doing risk mitigation
around that risk management cycle.
Now, the last thing I will say here
is that every mission that is worthwhile doing
is still going to have some residual risk at the end.
The requirement is not that all the risks are
at 0 in the lower-left corner.
You will launch, and you're going to have residual risks.
And you just have to accept those.
But you have to be cognisant of this.
And this is really no different in the automotive industry,
for example.
When you're developing a new car or a medical device
and you're going to launch it to the market,
if your requirement is 0 risk, you will never sell anything.
You will never launch anything.
Because you will always think of something bad
that could happen.
And there will always be people saying it's too risky.
We can't do it.
So knowing how much residual risk
you should be willing to carry is a big part
of being a leader, being a system engineer, really
understanding things.
And in the automotive industry, there
are people whose primary job they have is to do this work.
And they're called Quality Engineers or Warranty
Engineers.
So the Warranty Engineers, their job is twofold.
Before you launch a vehicle to market,
it's actually ranking on this particular vehicle or program,
what are the top 10 things that could cause warranty planes
and problems in the future?
We don't know that they will, but they might.
Right?
And then once a vehicle goes to market
and reports are coming back from users and from the fleet,
actually tracking what these issues are
and then knowing when has it hit a threshold where
you do need to do a recall, you do need to do a retrofit,
this is a big part of it.
And it's really a big deal.
I mean, the amount of money that automotive companies spend on
recalls every year is about the same as what their profits are.
So if you could eliminate recalls and warranty
claims altogether, you'd basically double your profit.
And so depending on what industry
you're in, whether it's automotive, medical,
spacecraft, how much risk and safety is involved,
this is more or less emphasized in the industry.
But it's a big part, I think, of system engineering
job, is to understand this.
This is, again, a flow diagram for how to do this risk
management properly.
You have a Risk Management Plan, your technical risk issues
that are placed on the matrix, any measurements
or data you have.
And then how do you report this?
And then out of it comes a mitigation plan,
a set of actions, technical risk reports, and then
any work product from the technical risk management.
And the idea is that you repeat this process
on a regular basis.
And it's a big part of your milestone reviews as well.
OK, so I'd like to spend a few minutes on system safety.
I am not going to do this justice because there's
a whole class here at MIT on this, taught by Professor Nancy
Leveson.
By the way, who's taken that class?
Or who's been thinking about taking it?
About three, four of you.
So I'm just going to give you a very quick exposure to this.
So this is a book that Professor Leveson wrote several years
ago.
And she basically distinguishes two kinds of failures.
Component failures, which most people think about,
an axle broke or there was a battery caught fire.
And clearly, component failures are real
and they happen, single or multiple component failures.
And usually there's some randomness to them.
So most of the classic accident investigation
techniques and safety techniques focus on component failures.
But there's also component interaction accidents
or failures which are trickier, in a sense,
because you could have a system that
has no single component that's failed.
And yet, you had a system failure.
And so it's the interactions among components.
And this could be related to interactive complexity
in coupling, more and more computers and software,
and then the role of humans and systems.
And this is really what a lot of this is about.
So the traditional safety thinking
is that component failures, you need
to worry about the component failures only.
So here's a classic example of a sequence of events.
This is for a tank failure.
So in this case, we have a tank.
And there's moisture that builds up in the tank.
And then corrosion, essentially, as a result.
The metal gets weakened.
And under the operating pressure of the tank, the tank
itself has been weakened due to corrosion.
The operating pressure causes a tank rupture.
And the tank rupture then causes, essentially,
an explosion.
And fragments or shrapnel from the tank
will be projected and then cause equipment damage
or personnel injury.
And so in this linear chain-of-events model,
the way you think about safety is putting in barriers.
This is often also referred as the Swiss cheese model.
if you take layers of Swiss cheese--
and I guess it has to be Emmentaler, right?
You guys know the Emmentaler at EPFL?
Emmentaler is the one, it's got the big holes.
AUDIENCE: Yes.
PROFESSOR: So you take these slices of cheese.
And if you can look at the cheese,
and there's actually a hole right through,
then the accident can happen.
But if you put another barrier in between,
you can't see through the cheese and the accident is prevented.
That's the classical thinking around system safety
is chain of events.
And then put barriers between these.
And this is, I think, valid for a very particular kind
of accidents, which are these component accidents.
What Professor Leveson says in her STAMP and STPA Framework
is a little different.
This is based on essentially thinking about safety
as a lack of control of a system, lack of control-ability
of a system.
And so if you think about it this way,
I'm showing you here a control loop.
In this case, you have the actual system,
the actual process that you're executing.
The controlled process is here.
You're sensing things about that process, so temperature,
pressure, proper alignment.
And so one problem could be your sensors are inadequate.
You're sensing the wrong information.
And then here's your controller model,
how should you control the system?
So you have the wrong controller or the wrong process model.
And then here's your actuators.
Are you issuing commands at the right time?
Are you issuing the right commands?
Or are you not issuing commands when
you should be to the system?
And that feeds back into the control process itself.
And so process inputs could be wrong or missing.
You could have disturbances into the process that are
unidentified or out of range.
And then eventually, if this process goes unstable,
then you have a failure or an accident.
So it's quite different.
It's essentially thinking of this
as a control problem instead of a chain-of-events problem.
And the argument here is that for safety or failures
that involve a combination of hardware, software and humans,
often this model is able to be more complete,
in terms of identifying hazards and potential mitigation
actions.
So I think we're out of time.
But I want I want to give this to you as a homework
for thinking about this.
This is an accident that happened earlier this year,
I guess, in July.
And this is the Virgin Galactic crash that happened.
Virgin Galactic is one of the space tourism companies.
And during a test flight, the airplane
crashed because the copilot unlocked the brake system.
So it has a kind of feathering mechanism.
And the pilot unlocked it too early
during a high-speed flight phase.
So what I'd like you to do, the link is here.
Just read the story.
There's a more lengthy accident report that's come out.
I'd like you to just read this quickly and then think about
how does this relate to risks?
How does it relate to this particular model
of system safety?
So the System's Theoretic View of Safety
is then that safety is an emergent system property.
Accidents arise from interactions
among system components--
physical, human, social-- constraint violation.
Losses are the result of complex processes,
not simple chain of events.
And that most accidents arise from--
you could have a system that's quite safe when
you start operating it.
But over time it migrates to an unsafe state.
Because sensors fail.
People start bypassing safety procedures.
And gradually, it migrates to high risk.
OK.
So the last thing I want to talk about-- just
for a minute or two-- is the FRR, the Flight Readiness
Review, which is one of the later milestones.
And what happens at the flight FRR?
Essentially, this is your last chance to raise a red flag.
This is the last milestone before launch.
Have all the V&V activities been passed successfully?
Are there any waivers that need to be granted?
What are the residual risks we just talked about?
And then after the FRR has passed,
you actually start the countdown--
T minus X days, Y hours, Z seconds--
to an actual launch, or a product launch, whatever it is.
And then here's from the handbook, the entrance
and success criteria for FRR.
Everything should have been done at this point.
Your design, your integration, your testing,
your operating procedure, your people should be trained.
This is your last chance to raise a red flag.
After the FRR, you're essentially go for launch.
So the stakes are high.
OK, so a quick summary.
Verification and validation are critical.
There's a distinction between the two.
Verification is against the requirements as written.
Validation is you go back to your customer
and you test in a real environment.
Testing, many different kinds of testing.
It's a fundamentally Q&A activity,
and it's really expensive, but it needs to be done right.
Risk management, we have different tools
like the risk matrix, risk identification, mitigation.
And that's really where the rubber meets the road,
in terms of the tension between cost, scope, schedule,
and risk in projects.
System safety, think about not just the chain of events model,
but this control's view as well.
STAMP/STPA is a particular framework for this.
And if you're interested, there's a whole class.
There's a whole set of things you
can learn about just safety.
And then finally, FRR is your last chance
to raise the red flag.
It's sort of the big milestone before you go live
with your system.
OK?
So any last questions or comments?
EPFL?
Yes, please go ahead.
AUDIENCE: So after the FRR you should have
finished your tests, right?
It's the limit?
PROFESSOR: Yes, you should have finished your tests
except for the ones that you're going to do, say, on orbit,
right?
AUDIENCE: Because I was wondering, in the list
there is the go, no-go test.
And I'm wondering if it's not related to the launch,
actually?
PROFESSOR: Yeah, so the actual launch itself, of course,
has the actual launch countdown.
And you can stop the launch.
So this is a review.
The FRR is more like a CDR.
So the countdown hasn't actually started yet.
But if you successfully pass the FRR,
that's when you begin the countdown.
The official countdown starts.
And then you still have a possibility, of course,
of stopping the launch.
But in terms of a formal programmatic review,
this is your last chance.
AUDIENCE: I think the point here is more about the what
is the system boundary?
If you consider the system boundary being
the whole mission, including the launch,
then obviously the system will only
be finished when the mission has finished phase EF.
PROFESSOR: Right.
AUDIENCE: And this FRR, Flight Readiness Review,
is linked to the panel or to the whole satellite,
specifically that you are allowed to go forward and now
start the countdown issues.
And then all the other things.
Like, you'd extend the mission boundaries to the whole space
program or to the whole colonization of Mars,
it will be, obviously, later.