WEBVTT 00:00:00.000 --> 00:00:01.979 align:middle line:90% [SQUEAKING] 00:00:01.979 --> 00:00:03.951 align:middle line:90% [RUSTLING] 00:00:03.951 --> 00:00:06.909 align:middle line:90% [CLICKING] 00:00:06.909 --> 00:00:09.570 align:middle line:90% 00:00:09.570 --> 00:00:11.320 align:middle line:84% HAZHIR RAHMANDAD: So today, we are working 00:00:11.320 --> 00:00:12.980 align:middle line:90% on endogenous diffusion model. 00:00:12.980 --> 00:00:19.940 align:middle line:84% So this model came from basically Jack Homer's work. 00:00:19.940 --> 00:00:23.080 align:middle line:84% And I was interested in the diffusion space 00:00:23.080 --> 00:00:24.580 align:middle line:90% when I was doing my PhD. 00:00:24.580 --> 00:00:28.280 align:middle line:84% And towards the end of it, he shared ideas 00:00:28.280 --> 00:00:32.900 align:middle line:84% for potential thesis areas or research areas. 00:00:32.900 --> 00:00:35.360 align:middle line:84% And then later on, we had a follow-up. 00:00:35.360 --> 00:00:38.440 align:middle line:84% And he shared some very small, nice model 00:00:38.440 --> 00:00:43.960 align:middle line:84% that captured his view of important dynamics. 00:00:43.960 --> 00:00:45.680 align:middle line:84% And I turned that model basically 00:00:45.680 --> 00:00:46.940 align:middle line:90% into a problem definition. 00:00:46.940 --> 00:00:49.760 align:middle line:84% So here, you are seeing a problem definition 00:00:49.760 --> 00:00:55.580 align:middle line:84% that captures what Jack's model had-- a very, very simple model. 00:00:55.580 --> 00:00:59.580 align:middle line:90% So I'll ask him if he's OK. 00:00:59.580 --> 00:01:02.200 align:middle line:84% I'll share that model with you as well. 00:01:02.200 --> 00:01:05.960 align:middle line:84% So basically, you have now worked on it. 00:01:05.960 --> 00:01:07.420 align:middle line:90% So everybody knows the problem. 00:01:07.420 --> 00:01:10.440 align:middle line:84% It's an adoption of a product that itself 00:01:10.440 --> 00:01:14.260 align:middle line:84% is evolving over time with addition of new features, 00:01:14.260 --> 00:01:16.620 align:middle line:84% but also discovery of bugs and problems. 00:01:16.620 --> 00:01:19.480 align:middle line:84% And Jack's thesis was on pacemakers. 00:01:19.480 --> 00:01:22.080 align:middle line:84% It's a life-saving technology for people 00:01:22.080 --> 00:01:23.960 align:middle line:90% who have heart conditions. 00:01:23.960 --> 00:01:26.600 align:middle line:84% And it has evolved a lot from something 00:01:26.600 --> 00:01:29.120 align:middle line:84% that was for a very niche market of people 00:01:29.120 --> 00:01:36.240 align:middle line:84% who have very serious heart problems to now capturing 00:01:36.240 --> 00:01:40.080 align:middle line:84% a much wider range of conditions and providing 00:01:40.080 --> 00:01:44.280 align:middle line:84% incremental benefits for people who otherwise could live pretty 00:01:44.280 --> 00:01:47.960 align:middle line:84% fine without it as well, at least. 00:01:47.960 --> 00:01:51.480 align:middle line:84% So that's the expansion of the scope of use. 00:01:51.480 --> 00:01:56.040 align:middle line:84% And then in his work, there he also 00:01:56.040 --> 00:01:59.720 align:middle line:84% identified impacts of identifying problems 00:01:59.720 --> 00:02:02.420 align:middle line:84% and assessing quality through research. 00:02:02.420 --> 00:02:05.980 align:middle line:84% And so these things come in that are promising. 00:02:05.980 --> 00:02:08.139 align:middle line:90% People may start adopting these. 00:02:08.139 --> 00:02:11.340 align:middle line:84% And this is not-- it's a little bit of different from drugs. 00:02:11.340 --> 00:02:15.800 align:middle line:84% It's not clinical trials that drive ability 00:02:15.800 --> 00:02:17.820 align:middle line:90% to market your product. 00:02:17.820 --> 00:02:21.280 align:middle line:84% But you need to show safety and show effectiveness in the longer 00:02:21.280 --> 00:02:23.022 align:middle line:90% term. 00:02:23.022 --> 00:02:25.480 align:middle line:84% And if there are problems, you need to get them discovered. 00:02:25.480 --> 00:02:29.280 align:middle line:84% Anyway, so research becomes an important source of information 00:02:29.280 --> 00:02:30.800 align:middle line:90% in the pacemaker example. 00:02:30.800 --> 00:02:34.360 align:middle line:84% More broadly, basically, you discover problems. 00:02:34.360 --> 00:02:38.520 align:middle line:84% And those discoveries are typically 00:02:38.520 --> 00:02:40.160 align:middle line:84% done by the customers, many of them. 00:02:40.160 --> 00:02:42.440 align:middle line:84% In many other markets, they just find 00:02:42.440 --> 00:02:44.240 align:middle line:84% the product to be useless or not, 00:02:44.240 --> 00:02:49.120 align:middle line:90% as effective as it was sold for. 00:02:49.120 --> 00:02:51.920 align:middle line:90% It becomes a source of-- 00:02:51.920 --> 00:02:53.520 align:middle line:84% negative word of mouth potentially 00:02:53.520 --> 00:02:57.340 align:middle line:84% may bring down the scope of uses that people would have. 00:02:57.340 --> 00:03:01.440 align:middle line:84% So that tug of war between expanding the scope and use 00:03:01.440 --> 00:03:03.480 align:middle line:84% cases and problems that are discovered 00:03:03.480 --> 00:03:05.520 align:middle line:84% that may need to be fixed, and then 00:03:05.520 --> 00:03:07.980 align:middle line:84% once the problems are fixed, those problems may go away. 00:03:07.980 --> 00:03:09.780 align:middle line:84% That is, the features become more useful. 00:03:09.780 --> 00:03:13.060 align:middle line:90% But some damage is already done. 00:03:13.060 --> 00:03:17.920 align:middle line:90% And this is the big picture. 00:03:17.920 --> 00:03:22.320 align:middle line:84% So let's see what challenges you have faced as you 00:03:22.320 --> 00:03:24.100 align:middle line:90% have worked on this problem. 00:03:24.100 --> 00:03:27.400 align:middle line:84% So what are some of the common challenges 00:03:27.400 --> 00:03:30.760 align:middle line:90% that you have experienced? 00:03:30.760 --> 00:03:33.320 align:middle line:90% Anybody want to share? 00:03:33.320 --> 00:03:35.360 align:middle line:84% AUDIENCE: I think I wasn't sure how 00:03:35.360 --> 00:03:40.620 align:middle line:84% to model the bugs having a problem, 00:03:40.620 --> 00:03:46.440 align:middle line:84% whether it should be a flow of people with buggy pacemakers 00:03:46.440 --> 00:03:52.480 align:middle line:84% or some kind of enduring story from the adopters who did ever 00:03:52.480 --> 00:03:53.720 align:middle line:90% have bugs. 00:03:53.720 --> 00:03:55.060 align:middle line:90% I went with the latter. 00:03:55.060 --> 00:03:57.700 align:middle line:84% But I wasn't very confident in it. 00:03:57.700 --> 00:04:04.440 align:middle line:90% 00:04:04.440 --> 00:04:07.200 align:middle line:84% AUDIENCE: For me, it was a little bit hard 00:04:07.200 --> 00:04:11.560 align:middle line:84% to think about the relation between the scope of use 00:04:11.560 --> 00:04:17.320 align:middle line:84% and the technology advancement, so these two sides. 00:04:17.320 --> 00:04:22.420 align:middle line:84% First of all, [INAUDIBLE] technology advancement 00:04:22.420 --> 00:04:23.860 align:middle line:90% [INAUDIBLE]. 00:04:23.860 --> 00:04:29.093 align:middle line:90% 00:04:29.093 --> 00:04:30.260 align:middle line:90% HAZHIR RAHMANDAD: Very good. 00:04:30.260 --> 00:04:33.680 align:middle line:84% That's also an important question. 00:04:33.680 --> 00:04:35.480 align:middle line:90% Others? 00:04:35.480 --> 00:04:38.920 align:middle line:84% AUDIENCE: I started this one too late, but, on previous ones, 00:04:38.920 --> 00:04:45.400 align:middle line:84% I would formulate each relationship incrementally. 00:04:45.400 --> 00:04:48.440 align:middle line:84% And for this one, that strategy didn't work. 00:04:48.440 --> 00:04:52.360 align:middle line:84% I had to lay out the big picture, 00:04:52.360 --> 00:04:58.720 align:middle line:84% and then go back and do the formulations in detail, 00:04:58.720 --> 00:05:00.460 align:middle line:90% which I didn't complete. 00:05:00.460 --> 00:05:04.840 align:middle line:84% But I think it draws attention to this question 00:05:04.840 --> 00:05:10.480 align:middle line:84% of incremental relationship formulation versus laying out 00:05:10.480 --> 00:05:12.960 align:middle line:84% a picture so that you have an idea of how 00:05:12.960 --> 00:05:15.512 align:middle line:84% the whole thing should work, and then going back and putting 00:05:15.512 --> 00:05:16.720 align:middle line:90% the relationship [INAUDIBLE]. 00:05:16.720 --> 00:05:18.637 align:middle line:84% HAZHIR RAHMANDAD: That's a very good question. 00:05:18.637 --> 00:05:20.080 align:middle line:84% So we should have a chat on that. 00:05:20.080 --> 00:05:25.680 align:middle line:84% Other observations, challenges that you faced with this model? 00:05:25.680 --> 00:05:29.480 align:middle line:84% AUDIENCE: [INAUDIBLE] units were a little bit more complex 00:05:29.480 --> 00:05:34.420 align:middle line:84% than this one, but [INAUDIBLE] didn't work as [INAUDIBLE]. 00:05:34.420 --> 00:05:41.720 align:middle line:90% 00:05:41.720 --> 00:05:44.760 align:middle line:90% HAZHIR RAHMANDAD: Interesting. 00:05:44.760 --> 00:05:45.260 align:middle line:90% Good. 00:05:45.260 --> 00:05:47.780 align:middle line:84% So I think these are good items to look at. 00:05:47.780 --> 00:05:53.320 align:middle line:84% And as you see, this one was even more loosely defined. 00:05:53.320 --> 00:05:54.860 align:middle line:90% And that's by design. 00:05:54.860 --> 00:05:56.800 align:middle line:84% As we move forward, I'm trying to make 00:05:56.800 --> 00:06:00.160 align:middle line:84% the problems a little bit harder by making them less 00:06:00.160 --> 00:06:02.577 align:middle line:84% well-defined, so that you can't just say, 00:06:02.577 --> 00:06:04.660 align:middle line:84% OK, I know this is this chunk, this is this chunk, 00:06:04.660 --> 00:06:05.452 align:middle line:90% this is this chunk. 00:06:05.452 --> 00:06:07.520 align:middle line:84% And the next one would also be somewhat 00:06:07.520 --> 00:06:11.400 align:middle line:84% similar in that you need to decide for yourself 00:06:11.400 --> 00:06:14.900 align:middle line:84% what chunks of formulation are going in. 00:06:14.900 --> 00:06:17.780 align:middle line:84% And that goes also partly to Jason's question. 00:06:17.780 --> 00:06:20.320 align:middle line:84% What's the best approach to modeling, 00:06:20.320 --> 00:06:22.880 align:middle line:84% in terms of having a big picture in place before you 00:06:22.880 --> 00:06:27.760 align:middle line:84% start with chunks or you go more incremental? 00:06:27.760 --> 00:06:34.480 align:middle line:84% And there are two parts in answering that. 00:06:34.480 --> 00:06:39.560 align:middle line:84% One is that the general iterative process for modeling 00:06:39.560 --> 00:06:43.420 align:middle line:90% means you do want to iterate. 00:06:43.420 --> 00:06:45.520 align:middle line:84% And you don't need to think about all the things 00:06:45.520 --> 00:06:50.820 align:middle line:84% that will be, in future, coming into your model's boundary. 00:06:50.820 --> 00:06:53.440 align:middle line:84% You start with what you can manage. 00:06:53.440 --> 00:06:56.640 align:middle line:84% But within any of those iterations-- and usually, 00:06:56.640 --> 00:07:02.000 align:middle line:84% the first iteration is the big part, the bigger challenge. 00:07:02.000 --> 00:07:05.560 align:middle line:84% In that first iteration, I would say you definitely 00:07:05.560 --> 00:07:08.800 align:middle line:84% need to have a big picture view of the model 00:07:08.800 --> 00:07:15.860 align:middle line:84% that, OK, this is the sector and features and scope. 00:07:15.860 --> 00:07:18.620 align:middle line:84% And the way I would think about it is x, y, z. 00:07:18.620 --> 00:07:22.500 align:middle line:84% So I would have features as a stock that is being developed. 00:07:22.500 --> 00:07:26.160 align:middle line:84% And I may have a coflow for bugs or-- 00:07:26.160 --> 00:07:29.040 align:middle line:84% so have it in mind before touching your Vensim 00:07:29.040 --> 00:07:32.600 align:middle line:84% and have that, OK, I think I know how to manage that. 00:07:32.600 --> 00:07:34.800 align:middle line:84% Then there is the question of impact 00:07:34.800 --> 00:07:39.580 align:middle line:84% of features on the scope of use and the number of customers. 00:07:39.580 --> 00:07:42.100 align:middle line:84% Do I want to just have an analytical function there? 00:07:42.100 --> 00:07:43.140 align:middle line:90% Yeah, maybe that's fine. 00:07:43.140 --> 00:07:47.280 align:middle line:84% And then the demand side-- do I use a Bass diffusion model? 00:07:47.280 --> 00:07:51.960 align:middle line:84% Or do I set a feasible scope we're interested in, population, 00:07:51.960 --> 00:07:54.860 align:middle line:84% and just adjust towards that and make the time constant? 00:07:54.860 --> 00:07:57.080 align:middle line:84% So I have those in the back of my mind 00:07:57.080 --> 00:07:59.500 align:middle line:84% when I'm starting with the structure, 00:07:59.500 --> 00:08:04.040 align:middle line:84% I may take quite a bit of time deciding on that. 00:08:04.040 --> 00:08:06.480 align:middle line:84% For the city dynamics model, I remember, 00:08:06.480 --> 00:08:08.940 align:middle line:84% for a few days, I was just walking and thinking, 00:08:08.940 --> 00:08:12.080 align:middle line:84% OK, what are the chunks that would work together and try 00:08:12.080 --> 00:08:18.860 align:middle line:84% to make reasonable decisions on those before I start working? 00:08:18.860 --> 00:08:21.420 align:middle line:84% Because as soon as you start building the model, 00:08:21.420 --> 00:08:24.160 align:middle line:84% it becomes too concrete to deviate. 00:08:24.160 --> 00:08:28.600 align:middle line:84% And therefore, you don't easily change bad decisions 00:08:28.600 --> 00:08:29.900 align:middle line:90% that you have made. 00:08:29.900 --> 00:08:33.100 align:middle line:84% And you start to get attached to bad decisions. 00:08:33.100 --> 00:08:36.860 align:middle line:84% And then they are inconsistent with other parts. 00:08:36.860 --> 00:08:39.000 align:middle line:90% And it just becomes messy. 00:08:39.000 --> 00:08:39.980 align:middle line:90% And you get locked in. 00:08:39.980 --> 00:08:42.600 align:middle line:90% Yeah, so that's the approach. 00:08:42.600 --> 00:08:46.900 align:middle line:84% Now, in the previous assignments, to a large extent, 00:08:46.900 --> 00:08:50.000 align:middle line:84% I had simplified the problem enough that you could read it 00:08:50.000 --> 00:08:55.300 align:middle line:84% and you would have the image of all of those chunks in place. 00:08:55.300 --> 00:08:58.600 align:middle line:84% So you didn't consciously need to go through this process 00:08:58.600 --> 00:09:00.620 align:middle line:90% that you had to do this time. 00:09:00.620 --> 00:09:03.300 align:middle line:84% And you would probably have to do some version of it next time. 00:09:03.300 --> 00:09:06.240 align:middle line:90% So this is exactly the process. 00:09:06.240 --> 00:09:09.940 align:middle line:84% Now, at that stage, you may also say, OK, 00:09:09.940 --> 00:09:13.980 align:middle line:84% I'm going to ignore this dynamic x or dynamic y. 00:09:13.980 --> 00:09:16.380 align:middle line:84% That is, OK, this is my first iteration. 00:09:16.380 --> 00:09:22.440 align:middle line:84% I'm not going to worry about measuring profitability 00:09:22.440 --> 00:09:26.000 align:middle line:84% and how it impacts feature introduction. 00:09:26.000 --> 00:09:27.780 align:middle line:90% This is beyond the scope. 00:09:27.780 --> 00:09:29.360 align:middle line:90% I'm OK with that. 00:09:29.360 --> 00:09:30.620 align:middle line:90% But I'm conscious of it. 00:09:30.620 --> 00:09:32.600 align:middle line:84% It's important so that, if somebody asks, 00:09:32.600 --> 00:09:34.020 align:middle line:90% you say, yes, you're right. 00:09:34.020 --> 00:09:35.760 align:middle line:90% That's an important feature. 00:09:35.760 --> 00:09:38.560 align:middle line:84% That's not part of this version of the model-- next iteration 00:09:38.560 --> 00:09:41.960 align:middle line:90% or five iterations from now. 00:09:41.960 --> 00:09:45.300 align:middle line:84% In a real project, I would have a list of features to be added. 00:09:45.300 --> 00:09:48.380 align:middle line:84% I have one right now for my city dynamics model. 00:09:48.380 --> 00:09:51.040 align:middle line:84% It has-- I don't know-- about 100 rows 00:09:51.040 --> 00:09:52.300 align:middle line:90% with different features. 00:09:52.300 --> 00:09:55.440 align:middle line:84% And about 70 of them are incorporated already. 00:09:55.440 --> 00:09:58.620 align:middle line:84% But there are 30 more to come, even within the current sectors. 00:09:58.620 --> 00:10:00.940 align:middle line:84% And this is beyond adding new sectors. 00:10:00.940 --> 00:10:03.120 align:middle line:84% So it doesn't have the transportation sector 00:10:03.120 --> 00:10:04.100 align:middle line:90% being added and so on. 00:10:04.100 --> 00:10:05.660 align:middle line:90% It still has quite a bit. 00:10:05.660 --> 00:10:08.140 align:middle line:90% So that's the big picture of it. 00:10:08.140 --> 00:10:12.360 align:middle line:84% So let's get started with your models. 00:10:12.360 --> 00:10:17.160 align:middle line:84% So while we are opening those, these 00:10:17.160 --> 00:10:22.260 align:middle line:84% are things that I was thinking about, what I'm looking for. 00:10:22.260 --> 00:10:24.960 align:middle line:84% So for the demand sector, and then the feature development 00:10:24.960 --> 00:10:29.540 align:middle line:84% sector, well, you may take a look at this. 00:10:29.540 --> 00:10:33.500 align:middle line:84% I'll just bring up the models that we have. 00:10:33.500 --> 00:10:51.892 align:middle line:90% 00:10:51.892 --> 00:10:52.400 align:middle line:90% OK. 00:10:52.400 --> 00:10:57.920 align:middle line:84% So let's take a quick look at some of the structures we have. 00:10:57.920 --> 00:10:59.840 align:middle line:90% This is Ross's. 00:10:59.840 --> 00:11:02.640 align:middle line:90% And let me increase the size. 00:11:02.640 --> 00:11:06.320 align:middle line:84% So just quick review of the stock inflow 00:11:06.320 --> 00:11:11.040 align:middle line:84% structures-- so Ross has total population, 00:11:11.040 --> 00:11:13.540 align:middle line:90% potential users, current users. 00:11:13.540 --> 00:11:15.340 align:middle line:90% There is an adoption process. 00:11:15.340 --> 00:11:21.060 align:middle line:90% 00:11:21.060 --> 00:11:21.560 align:middle line:90% Good. 00:11:21.560 --> 00:11:24.320 align:middle line:84% And he's thinking about potential users, those 00:11:24.320 --> 00:11:26.320 align:middle line:90% who may find this relevant. 00:11:26.320 --> 00:11:28.600 align:middle line:90% And that comes from features. 00:11:28.600 --> 00:11:34.440 align:middle line:84% And then those may adopt and become actual users. 00:11:34.440 --> 00:11:36.020 align:middle line:90% So this does make sense. 00:11:36.020 --> 00:11:38.480 align:middle line:84% I like this breaking down as one way 00:11:38.480 --> 00:11:41.240 align:middle line:84% of where the features operate and where 00:11:41.240 --> 00:11:43.840 align:middle line:90% utility might be operating. 00:11:43.840 --> 00:11:49.540 align:middle line:84% There are issues about one-way flow of adopters, users. 00:11:49.540 --> 00:11:52.800 align:middle line:90% But we can talk about those. 00:11:52.800 --> 00:12:00.620 align:middle line:84% Then we have feature development creating side effects. 00:12:00.620 --> 00:12:04.400 align:middle line:84% These side effects are unknown, then become known, 00:12:04.400 --> 00:12:07.800 align:middle line:90% and then they are fixed. 00:12:07.800 --> 00:12:10.420 align:middle line:84% That's also a nice way of doing it. 00:12:10.420 --> 00:12:14.360 align:middle line:90% 00:12:14.360 --> 00:12:16.040 align:middle line:90% Good. 00:12:16.040 --> 00:12:16.760 align:middle line:90% OK. 00:12:16.760 --> 00:12:19.560 align:middle line:90% So let's look at a few more. 00:12:19.560 --> 00:12:24.520 align:middle line:90% So this is Alban's. 00:12:24.520 --> 00:12:29.200 align:middle line:84% We have adopters here coming from basically 00:12:29.200 --> 00:12:32.080 align:middle line:90% some total population of sorts. 00:12:32.080 --> 00:12:36.120 align:middle line:84% And then we have feature expansion 00:12:36.120 --> 00:12:42.720 align:middle line:84% that sets the scope for indicated adopters. 00:12:42.720 --> 00:12:50.560 align:middle line:84% Then we have the bulk generation with feature generation, 00:12:50.560 --> 00:12:56.960 align:middle line:84% undiscovered, and discovered, very similar to Ross's model. 00:12:56.960 --> 00:13:02.080 align:middle line:90% Utility is coming from-- 00:13:02.080 --> 00:13:05.380 align:middle line:84% there is some utility function that is interesting. 00:13:05.380 --> 00:13:10.837 align:middle line:84% It is kind of a coflow of current scope and-- 00:13:10.837 --> 00:13:12.920 align:middle line:84% so it's something I have to look into more details 00:13:12.920 --> 00:13:15.720 align:middle line:84% to understand what exactly is happening. 00:13:15.720 --> 00:13:19.800 align:middle line:84% And then perceived utility informs 00:13:19.800 --> 00:13:23.200 align:middle line:84% what fraction of customers might be interested. 00:13:23.200 --> 00:13:27.680 align:middle line:84% So this is not breaking down the adopters into three sectors, 00:13:27.680 --> 00:13:29.600 align:middle line:90% but into two sectors. 00:13:29.600 --> 00:13:32.900 align:middle line:90% That also works. 00:13:32.900 --> 00:13:33.400 align:middle line:90% OK. 00:13:33.400 --> 00:13:38.120 align:middle line:90% This one is from Sercin. 00:13:38.120 --> 00:13:42.480 align:middle line:84% So we have adopters and their perceived utility, similar idea 00:13:42.480 --> 00:13:47.400 align:middle line:84% as Alban had, though not exactly as a coflow. 00:13:47.400 --> 00:13:53.920 align:middle line:84% And the adopters are coming from total population 00:13:53.920 --> 00:13:58.440 align:middle line:84% and what fraction may find the product to be useful. 00:13:58.440 --> 00:14:02.980 align:middle line:84% And this time to adopt is adjusted based on utility. 00:14:02.980 --> 00:14:05.520 align:middle line:90% 00:14:05.520 --> 00:14:16.160 align:middle line:84% Scope is basically based on investment in R&D. 00:14:16.160 --> 00:14:26.380 align:middle line:84% And usefulness of product is an interesting concept. 00:14:26.380 --> 00:14:28.840 align:middle line:84% So you are trying to capture something 00:14:28.840 --> 00:14:31.380 align:middle line:84% about the marginal utility of the product. 00:14:31.380 --> 00:14:34.700 align:middle line:84% That is, as we increase the scope, 00:14:34.700 --> 00:14:37.060 align:middle line:84% each increment becomes less important. 00:14:37.060 --> 00:14:39.240 align:middle line:84% So that's an important and interesting feature 00:14:39.240 --> 00:14:42.080 align:middle line:84% that we didn't see in other parts. 00:14:42.080 --> 00:14:44.200 align:middle line:84% And then we have unrecognized problems, 00:14:44.200 --> 00:14:47.620 align:middle line:84% recognized problems not fixed, and then those that are solved. 00:14:47.620 --> 00:14:48.360 align:middle line:90% Great. 00:14:48.360 --> 00:14:50.480 align:middle line:84% So I think those are fairly consistent 00:14:50.480 --> 00:14:54.200 align:middle line:90% with the previous ones we saw. 00:14:54.200 --> 00:14:57.040 align:middle line:90% This is Hesam's. 00:14:57.040 --> 00:15:01.440 align:middle line:84% So similar to Ross, we have potential adopters, adopters 00:15:01.440 --> 00:15:07.880 align:middle line:84% in scope, and current adopters, unknown bugs and known bugs, 00:15:07.880 --> 00:15:09.840 align:middle line:90% features. 00:15:09.840 --> 00:15:10.340 align:middle line:90% OK. 00:15:10.340 --> 00:15:16.360 align:middle line:84% So this is also fairly similar to Ross's big picture 00:15:16.360 --> 00:15:18.360 align:middle line:90% of components. 00:15:18.360 --> 00:15:23.920 align:middle line:84% Here, we have Gerardo's model with adopters 00:15:23.920 --> 00:15:25.900 align:middle line:90% and potential adopters. 00:15:25.900 --> 00:15:28.600 align:middle line:84% And this is based on the scope, I guess. 00:15:28.600 --> 00:15:31.880 align:middle line:84% This is a little bit more Bass diffusion-y type. 00:15:31.880 --> 00:15:40.460 align:middle line:84% And then available scope, this is basically the features. 00:15:40.460 --> 00:15:43.360 align:middle line:90% 00:15:43.360 --> 00:15:48.420 align:middle line:84% This is adding to the utility for customers. 00:15:48.420 --> 00:15:52.520 align:middle line:90% 00:15:52.520 --> 00:15:55.360 align:middle line:84% Then we have unknown issues, discovered issues, 00:15:55.360 --> 00:15:59.120 align:middle line:90% and fixed issues. 00:15:59.120 --> 00:16:01.560 align:middle line:84% This is an interesting thing to have a stock here 00:16:01.560 --> 00:16:03.140 align:middle line:90% to calculate some things. 00:16:03.140 --> 00:16:05.680 align:middle line:84% I'm sometimes skeptical when I see this 00:16:05.680 --> 00:16:09.120 align:middle line:84% because this one will be ultimately accumulating 00:16:09.120 --> 00:16:10.280 align:middle line:90% everything. 00:16:10.280 --> 00:16:13.880 align:middle line:84% So whether people care about that in calculating anything 00:16:13.880 --> 00:16:15.060 align:middle line:90% is not clear to me. 00:16:15.060 --> 00:16:17.500 align:middle line:84% But we should look at those equations later. 00:16:17.500 --> 00:16:21.500 align:middle line:84% Is this the one that I was just looking at? 00:16:21.500 --> 00:16:24.000 align:middle line:90% Yes. 00:16:24.000 --> 00:16:26.560 align:middle line:90% This is Jason's. 00:16:26.560 --> 00:16:31.240 align:middle line:84% So we have adopters and non-adopters, 00:16:31.240 --> 00:16:34.440 align:middle line:84% so two underlying stocks there-- discovered 00:16:34.440 --> 00:16:38.920 align:middle line:84% bugs, undiscovered bugs, similar to the previous ones. 00:16:38.920 --> 00:16:45.360 align:middle line:84% This is product capabilities, features that come. 00:16:45.360 --> 00:16:48.900 align:middle line:90% And then we have utility coming. 00:16:48.900 --> 00:16:50.440 align:middle line:84% And this is perception of utility, 00:16:50.440 --> 00:16:54.160 align:middle line:84% the delaying the perception of utility that impacts scope. 00:16:54.160 --> 00:16:55.640 align:middle line:90% OK. 00:16:55.640 --> 00:17:00.880 align:middle line:84% And then is this what we just went through. 00:17:00.880 --> 00:17:05.079 align:middle line:90% So this is Cathy's. 00:17:05.079 --> 00:17:07.520 align:middle line:90% So we have features. 00:17:07.520 --> 00:17:08.780 align:middle line:90% We have bugs. 00:17:08.780 --> 00:17:12.520 align:middle line:90% 00:17:12.520 --> 00:17:16.260 align:middle line:84% There is adopters and addressable market, 00:17:16.260 --> 00:17:20.300 align:middle line:84% which could expand based on population here. 00:17:20.300 --> 00:17:24.400 align:middle line:84% So this is more population dynamics not the scope 00:17:24.400 --> 00:17:26.880 align:middle line:90% impacting that. 00:17:26.880 --> 00:17:34.480 align:middle line:84% Then we have adoption, dynamics impacting what fraction of bugs 00:17:34.480 --> 00:17:38.440 align:middle line:84% are discovered or something about discovery of bugs. 00:17:38.440 --> 00:17:40.920 align:middle line:84% This is actually about customers that 00:17:40.920 --> 00:17:45.800 align:middle line:84% have problems and negative perceptions that come with it. 00:17:45.800 --> 00:17:53.297 align:middle line:90% OK, interesting. 00:17:53.297 --> 00:17:55.380 align:middle line:84% So this is a little bit of a different formulation 00:17:55.380 --> 00:17:58.480 align:middle line:84% we can check out here in Cathy's model. 00:17:58.480 --> 00:18:01.940 align:middle line:90% 00:18:01.940 --> 00:18:03.700 align:middle line:90% Next one, Ali's. 00:18:03.700 --> 00:18:07.560 align:middle line:90% 00:18:07.560 --> 00:18:13.020 align:middle line:84% So we have issues, known and unknown, fixing time, 00:18:13.020 --> 00:18:17.960 align:middle line:84% if the issues that may depend on some additional structure 00:18:17.960 --> 00:18:23.680 align:middle line:84% here for desired time to fix things. 00:18:23.680 --> 00:18:27.340 align:middle line:84% But all of this looks like just a number at the end to me. 00:18:27.340 --> 00:18:31.080 align:middle line:84% So there is no feedback into this structure. 00:18:31.080 --> 00:18:32.042 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:18:32.042 --> 00:18:33.000 align:middle line:90% HAZHIR RAHMANDAD: Yeah. 00:18:33.000 --> 00:18:35.720 align:middle line:90% 00:18:35.720 --> 00:18:45.460 align:middle line:84% Maximum scope of use is coming from current scope of use. 00:18:45.460 --> 00:18:48.360 align:middle line:90% So there is some adjustment. 00:18:48.360 --> 00:18:50.040 align:middle line:90% There is a scope of use. 00:18:50.040 --> 00:18:53.220 align:middle line:84% And this maximum is also exogenous. 00:18:53.220 --> 00:18:54.600 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:18:54.600 --> 00:18:56.910 align:middle line:84% HAZHIR RAHMANDAD: OK, that's fine. 00:18:56.910 --> 00:18:59.160 align:middle line:84% AUDIENCE: [INAUDIBLE] trade-off between the capability 00:18:59.160 --> 00:19:03.685 align:middle line:84% to [INAUDIBLE] scope [INAUDIBLE]. 00:19:03.685 --> 00:19:04.560 align:middle line:90% HAZHIR RAHMANDAD: OK. 00:19:04.560 --> 00:19:06.680 align:middle line:84% Capabilities of timing, what do you mean? 00:19:06.680 --> 00:19:08.640 align:middle line:84% AUDIENCE: The capability of investment 00:19:08.640 --> 00:19:12.408 align:middle line:90% in fixing the bugs [INAUDIBLE]. 00:19:12.408 --> 00:19:13.700 align:middle line:90% HAZHIR RAHMANDAD: I see, I see. 00:19:13.700 --> 00:19:15.672 align:middle line:90% Interesting, OK. 00:19:15.672 --> 00:19:17.480 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:19:17.480 --> 00:19:19.800 align:middle line:84% HAZHIR RAHMANDAD: Yeah, that's fine. 00:19:19.800 --> 00:19:21.540 align:middle line:90% I think this is good. 00:19:21.540 --> 00:19:22.820 align:middle line:90% You had the big picture. 00:19:22.820 --> 00:19:24.780 align:middle line:84% You had these components in your mind. 00:19:24.780 --> 00:19:29.220 align:middle line:84% But then time-wise, you didn't get to fix all the bugs, 00:19:29.220 --> 00:19:31.400 align:middle line:90% so that's OK. 00:19:31.400 --> 00:19:34.220 align:middle line:84% Scope of use impacts the adoption. 00:19:34.220 --> 00:19:37.340 align:middle line:84% And we have here the two-stock version of the adoption. 00:19:37.340 --> 00:19:41.200 align:middle line:90% 00:19:41.200 --> 00:19:47.500 align:middle line:84% And yeah, so this is also fairly similar in the bug fixing. 00:19:47.500 --> 00:19:50.360 align:middle line:90% 00:19:50.360 --> 00:19:51.400 align:middle line:90% Let's see. 00:19:51.400 --> 00:19:54.060 align:middle line:90% This is Jake's. 00:19:54.060 --> 00:19:58.440 align:middle line:90% 00:19:58.440 --> 00:20:06.000 align:middle line:84% So we have three-stock version of the adoption process. 00:20:06.000 --> 00:20:08.940 align:middle line:84% We have the two-stock version of the bug discovery. 00:20:08.940 --> 00:20:18.720 align:middle line:84% We have utility that is coming from capabilities or capability 00:20:18.720 --> 00:20:19.620 align:middle line:90% development rate. 00:20:19.620 --> 00:20:23.160 align:middle line:90% So this might be closer to-- 00:20:23.160 --> 00:20:27.600 align:middle line:84% so this is closer to the coflow idea. 00:20:27.600 --> 00:20:33.320 align:middle line:84% And then perceived utility is impacting something down here 00:20:33.320 --> 00:20:35.240 align:middle line:90% I'm sure. 00:20:35.240 --> 00:20:36.020 align:middle line:90% OK, good. 00:20:36.020 --> 00:20:37.700 align:middle line:84% So I think it's actually interesting. 00:20:37.700 --> 00:20:41.420 align:middle line:84% It's fairly consistent across the models that we have. 00:20:41.420 --> 00:20:45.800 align:middle line:84% So we have two or three stock versions of the adoption chain. 00:20:45.800 --> 00:20:50.200 align:middle line:84% And then we have typically two-stock version 00:20:50.200 --> 00:20:53.420 align:middle line:84% of bugs and bugs that need to be fixed 00:20:53.420 --> 00:20:57.740 align:middle line:84% and one-stock version for features. 00:20:57.740 --> 00:21:00.440 align:middle line:84% And then we have variants for other things. 00:21:00.440 --> 00:21:03.420 align:middle line:84% So some folks have utility explicitly. 00:21:03.420 --> 00:21:10.740 align:middle line:84% Some just capture it within the other pieces of the model. 00:21:10.740 --> 00:21:14.620 align:middle line:90% So let's dive into some of them. 00:21:14.620 --> 00:21:16.320 align:middle line:90% We could start with Jake's here. 00:21:16.320 --> 00:21:25.840 align:middle line:84% So first, on the demand side-- so I 00:21:25.840 --> 00:21:29.200 align:middle line:84% think the more standard formulation is the demand side. 00:21:29.200 --> 00:21:33.080 align:middle line:84% So population out of scope of use, 00:21:33.080 --> 00:21:40.080 align:middle line:84% they become addressable based on total utility. 00:21:40.080 --> 00:21:43.020 align:middle line:84% We should get back to see what this is. 00:21:43.020 --> 00:21:48.280 align:middle line:84% This total utility, how does it turn into the-- 00:21:48.280 --> 00:21:50.425 align:middle line:84% AUDIENCE: Utility, it's going to be like 0 to 1. 00:21:50.425 --> 00:21:51.300 align:middle line:90% HAZHIR RAHMANDAD: OK. 00:21:51.300 --> 00:21:53.980 align:middle line:84% So utility is bounded between 0 to 1. 00:21:53.980 --> 00:21:57.400 align:middle line:84% 1 means everybody who potentially could ever 00:21:57.400 --> 00:22:01.500 align:middle line:84% use pacemaker is now within the scope. 00:22:01.500 --> 00:22:05.400 align:middle line:84% That is, there is a pacemaker that could be useful for them. 00:22:05.400 --> 00:22:11.560 align:middle line:84% And then as you get this ratio closer to-- 00:22:11.560 --> 00:22:17.840 align:middle line:84% and then yeah, you want to bring down this ratio to 1 minus it. 00:22:17.840 --> 00:22:21.360 align:middle line:90% 00:22:21.360 --> 00:22:24.400 align:middle line:90% Yes, that makes sense. 00:22:24.400 --> 00:22:27.680 align:middle line:84% And then this is, over time, these people move 00:22:27.680 --> 00:22:30.680 align:middle line:90% to the right-hand side. 00:22:30.680 --> 00:22:33.900 align:middle line:84% And if we have a utility that goes down-- 00:22:33.900 --> 00:22:36.560 align:middle line:90% 00:22:36.560 --> 00:22:38.920 align:middle line:90% yes, it would get negative. 00:22:38.920 --> 00:22:40.180 align:middle line:90% It should get negative. 00:22:40.180 --> 00:22:42.640 align:middle line:90% Now, this one has a max 0. 00:22:42.640 --> 00:22:45.960 align:middle line:84% So this is an interesting question here. 00:22:45.960 --> 00:22:47.340 align:middle line:90% Should it get negative or not? 00:22:47.340 --> 00:22:50.800 align:middle line:84% So if the utility goes down-- now, maybe in your formulation, 00:22:50.800 --> 00:22:51.740 align:middle line:90% it never goes down. 00:22:51.740 --> 00:22:55.460 align:middle line:84% So you wouldn't have the use case where this is negative. 00:22:55.460 --> 00:22:57.760 align:middle line:84% But conceptually, I could see a situation 00:22:57.760 --> 00:23:01.200 align:middle line:84% where we have expanded the scope because we 00:23:01.200 --> 00:23:02.420 align:middle line:90% thought this is useful. 00:23:02.420 --> 00:23:05.160 align:middle line:84% And then later on, we realize, actually, there 00:23:05.160 --> 00:23:09.040 align:middle line:84% is so much side effects and problems that this 00:23:09.040 --> 00:23:11.000 align:middle line:90% shouldn't be in scope. 00:23:11.000 --> 00:23:15.020 align:middle line:84% And therefore, the addressable population should go down. 00:23:15.020 --> 00:23:19.680 align:middle line:84% So there would perhaps be a negative flow backwards. 00:23:19.680 --> 00:23:22.920 align:middle line:84% And in that case, it's conceivable 00:23:22.920 --> 00:23:29.580 align:middle line:84% that just having this flow not maxed out by 0 be OK. 00:23:29.580 --> 00:23:31.040 align:middle line:90% It may work fine. 00:23:31.040 --> 00:23:33.280 align:middle line:84% There is a little bit of a tricky business 00:23:33.280 --> 00:23:35.760 align:middle line:84% here that usually, when you have side effects, 00:23:35.760 --> 00:23:38.280 align:middle line:84% people decide, oh, this thing is out of scope. 00:23:38.280 --> 00:23:40.920 align:middle line:84% The movement backwards is very fast. 00:23:40.920 --> 00:23:43.780 align:middle line:84% The whole community would know about it. 00:23:43.780 --> 00:23:46.280 align:middle line:84% This is a scandal that has been behind it 00:23:46.280 --> 00:23:48.060 align:middle line:90% that has led to that point. 00:23:48.060 --> 00:23:50.640 align:middle line:84% So the time constant for movement to the right 00:23:50.640 --> 00:23:53.820 align:middle line:84% versus to the left would be potentially asymmetric. 00:23:53.820 --> 00:23:57.900 align:middle line:84% But having that in mind, if you're aware of that, 00:23:57.900 --> 00:23:58.700 align:middle line:90% then it's OK. 00:23:58.700 --> 00:24:00.960 align:middle line:84% I'm OK with this formulation here. 00:24:00.960 --> 00:24:03.240 align:middle line:84% And we have indeed looked into the utility. 00:24:03.240 --> 00:24:06.520 align:middle line:84% Then once you are a potential adopter, 00:24:06.520 --> 00:24:09.960 align:middle line:84% you have the word of mouth formulation here. 00:24:09.960 --> 00:24:12.480 align:middle line:84% And you have a different formulation 00:24:12.480 --> 00:24:17.520 align:middle line:84% based on adoption fraction and adoption gap. 00:24:17.520 --> 00:24:20.920 align:middle line:84% AUDIENCE: So yeah, I mean what you basically just said 00:24:20.920 --> 00:24:23.720 align:middle line:90% is being [INAUDIBLE]. 00:24:23.720 --> 00:24:25.920 align:middle line:84% There is a literal utility, but you're 00:24:25.920 --> 00:24:31.680 align:middle line:84% adding scope that is going to [INAUDIBLE] from population 00:24:31.680 --> 00:24:34.120 align:middle line:84% [INAUDIBLE] use cases to potential adopters. 00:24:34.120 --> 00:24:37.960 align:middle line:84% And then, they consider themselves part of scope 00:24:37.960 --> 00:24:42.880 align:middle line:84% with the perceived utility versus adoptive [INAUDIBLE], 00:24:42.880 --> 00:24:43.990 align:middle line:90% actual [? adopters. ?] 00:24:43.990 --> 00:24:45.740 align:middle line:84% HAZHIR RAHMANDAD: So what I'm saying here, 00:24:45.740 --> 00:24:49.880 align:middle line:84% there is the adoption gap, which is-- 00:24:49.880 --> 00:24:52.800 align:middle line:84% and the units here start to become tricky 00:24:52.800 --> 00:24:55.560 align:middle line:84% because adopters are typically in the units of people 00:24:55.560 --> 00:24:56.800 align:middle line:90% or something. 00:24:56.800 --> 00:24:57.460 align:middle line:90% Let's see. 00:24:57.460 --> 00:25:00.480 align:middle line:90% 00:25:00.480 --> 00:25:08.320 align:middle line:84% So this adoption gap, this is fine. 00:25:08.320 --> 00:25:10.780 align:middle line:84% AUDIENCE: [INAUDIBLE] perceived adopters [INAUDIBLE]. 00:25:10.780 --> 00:25:20.280 align:middle line:90% 00:25:20.280 --> 00:25:22.680 align:middle line:84% HAZHIR RAHMANDAD: So what this formulation does 00:25:22.680 --> 00:25:31.960 align:middle line:84% for you is that it says, what is the gap between fraction that 00:25:31.960 --> 00:25:40.240 align:middle line:84% have adopted, this guy, and fraction that should be here, 00:25:40.240 --> 00:25:43.080 align:middle line:90% but may not still be here. 00:25:43.080 --> 00:25:44.160 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:25:44.160 --> 00:25:46.368 align:middle line:84% HAZHIR RAHMANDAD: Or thinks that they should be here. 00:25:46.368 --> 00:25:51.680 align:middle line:84% And if that number is positive, it allows the flow to continue. 00:25:51.680 --> 00:25:53.580 align:middle line:90% When it goes to 0, it goes to 0. 00:25:53.580 --> 00:25:54.660 align:middle line:90% It shuts off the flow. 00:25:54.660 --> 00:25:55.920 align:middle line:90% That's the idea. 00:25:55.920 --> 00:26:05.440 align:middle line:84% Now, the word of mouth formulation has contact rate. 00:26:05.440 --> 00:26:08.340 align:middle line:84% Word of mouth adoption effectiveness, 00:26:08.340 --> 00:26:11.860 align:middle line:84% potential adopters, adopters total population. 00:26:11.860 --> 00:26:13.460 align:middle line:84% So this is the standard base model. 00:26:13.460 --> 00:26:15.500 align:middle line:84% But you have added the max zero thing. 00:26:15.500 --> 00:26:18.300 align:middle line:84% So where did these max zeros come from? 00:26:18.300 --> 00:26:20.283 align:middle line:84% AUDIENCE: Well, I mean, maybe I was just 00:26:20.283 --> 00:26:22.200 align:middle line:84% sticking too close to the idea of [INAUDIBLE]. 00:26:22.200 --> 00:26:27.070 align:middle line:84% I didn't know if you can remove the pacemaker, but [INAUDIBLE]. 00:26:27.070 --> 00:26:28.320 align:middle line:90% HAZHIR RAHMANDAD: That's fine. 00:26:28.320 --> 00:26:31.600 align:middle line:84% But this formulation should never go negative. 00:26:31.600 --> 00:26:34.520 align:middle line:90% So why do you need a max zero? 00:26:34.520 --> 00:26:37.400 align:middle line:84% AUDIENCE: It's probably at some stage in development 00:26:37.400 --> 00:26:40.425 align:middle line:84% when I was like, why is this graph going negative? 00:26:40.425 --> 00:26:41.300 align:middle line:90% HAZHIR RAHMANDAD: OK. 00:26:41.300 --> 00:26:45.000 align:middle line:84% So this is actually a very good process point. 00:26:45.000 --> 00:26:48.200 align:middle line:84% So if, in your modeling, you see something 00:26:48.200 --> 00:26:50.600 align:middle line:84% going negative that it shouldn't, that 00:26:50.600 --> 00:26:53.720 align:middle line:84% is actually a very good sign to trace and figure out 00:26:53.720 --> 00:26:55.020 align:middle line:90% why it is happening. 00:26:55.020 --> 00:26:59.260 align:middle line:84% Putting a max zero there will basically wipe the problem, 00:26:59.260 --> 00:27:01.620 align:middle line:90% but wouldn't solve the problem. 00:27:01.620 --> 00:27:04.140 align:middle line:84% So the problem is now happening, but you don't see it. 00:27:04.140 --> 00:27:06.015 align:middle line:84% AUDIENCE: But if you run it before and after, 00:27:06.015 --> 00:27:08.360 align:middle line:90% you see what fixed it. 00:27:08.360 --> 00:27:09.960 align:middle line:84% HAZHIR RAHMANDAD: Yes, but it really 00:27:09.960 --> 00:27:12.280 align:middle line:84% hasn't fixed it because you haven't understood what 00:27:12.280 --> 00:27:15.620 align:middle line:90% led to the negative flows here. 00:27:15.620 --> 00:27:17.280 align:middle line:84% So basically, negative flows here 00:27:17.280 --> 00:27:20.100 align:middle line:84% mean your potential adopters have gone negative. 00:27:20.100 --> 00:27:21.200 align:middle line:90% That's my guess. 00:27:21.200 --> 00:27:25.520 align:middle line:84% And my guess for that is this perceived adoption gap 00:27:25.520 --> 00:27:27.452 align:middle line:84% is doing something a little bit problematic. 00:27:27.452 --> 00:27:29.660 align:middle line:84% AUDIENCE: I think I ultimately found it and fixed it. 00:27:29.660 --> 00:27:31.493 align:middle line:84% HAZHIR RAHMANDAD: It might already be fixed. 00:27:31.493 --> 00:27:33.280 align:middle line:84% So that's exactly the point that you 00:27:33.280 --> 00:27:35.700 align:middle line:90% wouldn't know from this signal. 00:27:35.700 --> 00:27:37.660 align:middle line:90% So don't delete the signal. 00:27:37.660 --> 00:27:39.640 align:middle line:84% Signals are good, even if it means 00:27:39.640 --> 00:27:41.800 align:middle line:84% there is a problem with your formulation. 00:27:41.800 --> 00:27:43.840 align:middle line:90% OK. 00:27:43.840 --> 00:27:46.960 align:middle line:84% So the second thing is that you're 00:27:46.960 --> 00:27:50.060 align:middle line:84% kind of double-counting an important concept. 00:27:50.060 --> 00:27:52.880 align:middle line:84% So Bass diffusion model has this limit of, 00:27:52.880 --> 00:27:54.520 align:middle line:84% if I don't have potential adopters, 00:27:54.520 --> 00:27:56.440 align:middle line:90% I wouldn't have any adoption. 00:27:56.440 --> 00:27:57.800 align:middle line:90% So that's already there. 00:27:57.800 --> 00:28:00.520 align:middle line:84% Now, you have added this additional perceived adoption 00:28:00.520 --> 00:28:04.240 align:middle line:84% gap mechanism that says, basically, do I 00:28:04.240 --> 00:28:06.140 align:middle line:90% have enough potential adopters? 00:28:06.140 --> 00:28:09.600 align:middle line:84% If not, I'll shrink adoption rate further. 00:28:09.600 --> 00:28:11.160 align:middle line:84% So this is kind of double-counting 00:28:11.160 --> 00:28:12.680 align:middle line:90% the same mechanism. 00:28:12.680 --> 00:28:15.380 align:middle line:84% And as such, I wouldn't have that. 00:28:15.380 --> 00:28:18.460 align:middle line:84% I would probably go with one of these two pathways. 00:28:18.460 --> 00:28:21.520 align:middle line:84% That is, either you would have a perceived adoption gap-- not 00:28:21.520 --> 00:28:23.420 align:middle line:84% in the dimensionless view of the world, 00:28:23.420 --> 00:28:26.480 align:middle line:84% but in the difference between the current adopters 00:28:26.480 --> 00:28:31.400 align:middle line:84% and what they should be, which is the potential adopters 00:28:31.400 --> 00:28:34.200 align:middle line:90% plus adopters. 00:28:34.200 --> 00:28:37.080 align:middle line:84% So the gap is the potential adopters, basically. 00:28:37.080 --> 00:28:38.640 align:middle line:84% So it would be potential adopters 00:28:38.640 --> 00:28:40.540 align:middle line:90% divided by a time constant. 00:28:40.540 --> 00:28:42.920 align:middle line:84% That time constant says how fast they move. 00:28:42.920 --> 00:28:44.980 align:middle line:84% And that movement, you could say, 00:28:44.980 --> 00:28:46.975 align:middle line:84% depends on how many adopters you have. 00:28:46.975 --> 00:28:49.100 align:middle line:84% If you have a lot adopters, there is word of mouth, 00:28:49.100 --> 00:28:50.020 align:middle line:90% people talk about it. 00:28:50.020 --> 00:28:51.460 align:middle line:90% I can change the time constant. 00:28:51.460 --> 00:28:54.960 align:middle line:84% So that would be a simple alternative formulation here. 00:28:54.960 --> 00:28:55.980 align:middle line:90% OK, great. 00:28:55.980 --> 00:28:58.100 align:middle line:90% So this is the adoption side. 00:28:58.100 --> 00:29:04.640 align:middle line:84% Now, let's go and understand what the supply side of features 00:29:04.640 --> 00:29:05.180 align:middle line:90% come in. 00:29:05.180 --> 00:29:08.000 align:middle line:84% So you have an R&D process that generates 00:29:08.000 --> 00:29:10.000 align:middle line:90% technological capabilities. 00:29:10.000 --> 00:29:13.340 align:middle line:90% This R&D process has utility. 00:29:13.340 --> 00:29:15.140 align:middle line:90% So it is kind of endogenous. 00:29:15.140 --> 00:29:17.240 align:middle line:90% That is, you have a max effort. 00:29:17.240 --> 00:29:21.780 align:middle line:84% And then you have an incentive factor that depends on-- 00:29:21.780 --> 00:29:24.500 align:middle line:90% 00:29:24.500 --> 00:29:27.000 align:middle line:84% AUDIENCE: How much [INAUDIBLE] you're actually going to get. 00:29:27.000 --> 00:29:28.542 align:middle line:84% HAZHIR RAHMANDAD: Yes, so the utility 00:29:28.542 --> 00:29:30.460 align:middle line:84% that you are getting per new development. 00:29:30.460 --> 00:29:34.220 align:middle line:84% So this is an important, interesting mechanism. 00:29:34.220 --> 00:29:36.600 align:middle line:84% That is, it's trying to endogenize 00:29:36.600 --> 00:29:39.540 align:middle line:90% the creation of new features. 00:29:39.540 --> 00:29:41.840 align:middle line:84% Now, the measurement issues, when I see this, 00:29:41.840 --> 00:29:43.940 align:middle line:84% is I don't know how it has been captured. 00:29:43.940 --> 00:29:48.960 align:middle line:84% So I'm interested to see how the utility per new development 00:29:48.960 --> 00:29:50.940 align:middle line:90% feature is incorporated. 00:29:50.940 --> 00:29:52.720 align:middle line:90% That's the core. 00:29:52.720 --> 00:29:57.280 align:middle line:84% Marginal utility of a new feature is key to the decision 00:29:57.280 --> 00:29:59.180 align:middle line:84% to whether to invest into R&D or not. 00:29:59.180 --> 00:30:02.400 align:middle line:84% So as such, conceptually, it is a good concept here. 00:30:02.400 --> 00:30:04.760 align:middle line:84% Now, how it is formulated, we have to see. 00:30:04.760 --> 00:30:07.040 align:middle line:84% But before we go further to see that, 00:30:07.040 --> 00:30:13.400 align:middle line:84% it is R&D incentive multiplied by max R&D effort. 00:30:13.400 --> 00:30:16.880 align:middle line:84% And max R&D effort is the units of capabilities 00:30:16.880 --> 00:30:18.300 align:middle line:90% per unit of time. 00:30:18.300 --> 00:30:20.240 align:middle line:84% And this is, I assume, dimensionless. 00:30:20.240 --> 00:30:22.880 align:middle line:90% 00:30:22.880 --> 00:30:27.040 align:middle line:84% It's the ratio of utility per new development, max utility 00:30:27.040 --> 00:30:30.000 align:middle line:90% gain per development. 00:30:30.000 --> 00:30:30.620 align:middle line:90% That's OK. 00:30:30.620 --> 00:30:34.440 align:middle line:84% I think this is introducing a couple of parameters 00:30:34.440 --> 00:30:37.120 align:middle line:90% that are left out. 00:30:37.120 --> 00:30:41.800 align:middle line:84% And as such, some people may be hesitant to have 00:30:41.800 --> 00:30:43.460 align:middle line:90% as many free parameters. 00:30:43.460 --> 00:30:45.720 align:middle line:84% But I'm OK with it for our purposes 00:30:45.720 --> 00:30:48.940 align:middle line:84% because we are not doing serious cost-benefit analysis. 00:30:48.940 --> 00:30:50.980 align:middle line:84% So it's a good first order approximation. 00:30:50.980 --> 00:30:54.220 align:middle line:84% The question is, is this utility per new development? 00:30:54.220 --> 00:30:55.700 align:middle line:90% You have a question? 00:30:55.700 --> 00:30:58.618 align:middle line:84% AUDIENCE: What if you have [INAUDIBLE]? 00:30:58.618 --> 00:31:00.660 align:middle line:84% HAZHIR RAHMANDAD: For technological capabilities? 00:31:00.660 --> 00:31:02.887 align:middle line:84% If it is not used anywhere, we don't need it. 00:31:02.887 --> 00:31:04.720 align:middle line:84% But it should be used somewhere because this 00:31:04.720 --> 00:31:10.240 align:middle line:84% is what is really determining the current scope of use. 00:31:10.240 --> 00:31:14.800 align:middle line:84% Now, we'll see where it is being used. 00:31:14.800 --> 00:31:15.675 align:middle line:90% It is not being used. 00:31:15.675 --> 00:31:17.175 align:middle line:84% AUDIENCE: Yeah, right now, it's not. 00:31:17.175 --> 00:31:19.040 align:middle line:84% I did try to make it depend on that rather 00:31:19.040 --> 00:31:21.120 align:middle line:84% than the flow [INAUDIBLE], I thought it easier 00:31:21.120 --> 00:31:23.525 align:middle line:90% to make the scope [INAUDIBLE]. 00:31:23.525 --> 00:31:24.400 align:middle line:90% HAZHIR RAHMANDAD: OK. 00:31:24.400 --> 00:31:26.960 align:middle line:90% So this is important, I think. 00:31:26.960 --> 00:31:28.560 align:middle line:84% If you are capturing a stock that 00:31:28.560 --> 00:31:34.680 align:middle line:84% is capturing the key inertial feature here, which is the-- 00:31:34.680 --> 00:31:37.880 align:middle line:84% let's call it features or capabilities-- 00:31:37.880 --> 00:31:39.600 align:middle line:84% then it should be used somewhere. 00:31:39.600 --> 00:31:41.880 align:middle line:84% Now, probably the way you're doing it 00:31:41.880 --> 00:31:45.840 align:middle line:84% is that this guy is going to some other places as well, yes. 00:31:45.840 --> 00:31:48.760 align:middle line:84% So it is not only going to flog, but also it's 00:31:48.760 --> 00:31:50.880 align:middle line:90% going to utility growth rate. 00:31:50.880 --> 00:31:55.800 align:middle line:84% So basically, your utility growth rate 00:31:55.800 --> 00:32:01.060 align:middle line:90% is capturing this inertia. 00:32:01.060 --> 00:32:05.480 align:middle line:84% But you also have this decreasing return built 00:32:05.480 --> 00:32:07.820 align:middle line:90% into your utility growth rate. 00:32:07.820 --> 00:32:10.640 align:middle line:84% So you can think of the total utility 00:32:10.640 --> 00:32:14.800 align:middle line:84% to be a coflow of the technological capabilities. 00:32:14.800 --> 00:32:19.560 align:middle line:84% But that coflow makes the incremental utility 00:32:19.560 --> 00:32:27.440 align:middle line:84% smaller and smaller as you go to higher levels of current utility 00:32:27.440 --> 00:32:28.820 align:middle line:90% or current features. 00:32:28.820 --> 00:32:31.260 align:middle line:84% That's the idea that Jake is capturing. 00:32:31.260 --> 00:32:32.060 align:middle line:90% It's a good idea. 00:32:32.060 --> 00:32:35.360 align:middle line:84% That is, basically, each increment of a technology 00:32:35.360 --> 00:32:40.280 align:middle line:90% is not as valuable. 00:32:40.280 --> 00:32:43.320 align:middle line:90% And then you can calculate-- 00:32:43.320 --> 00:32:43.820 align:middle line:90% OK. 00:32:43.820 --> 00:32:50.120 align:middle line:84% So one part to consider is whether this feedback 00:32:50.120 --> 00:32:52.520 align:middle line:84% on-- so utility of new development 00:32:52.520 --> 00:32:58.440 align:middle line:84% is max utility e to the power of utility drop-off sensitivity 00:32:58.440 --> 00:33:00.180 align:middle line:90% by total utility. 00:33:00.180 --> 00:33:03.320 align:middle line:84% So basically, you have a declining function. 00:33:03.320 --> 00:33:10.540 align:middle line:84% As you build more utility, fewer incremental utilities are added. 00:33:10.540 --> 00:33:13.320 align:middle line:84% So that creates an exponential curve 00:33:13.320 --> 00:33:17.920 align:middle line:84% that converges for utility as you add more features. 00:33:17.920 --> 00:33:21.280 align:middle line:84% That's, I think, the idea that Jake is capturing. 00:33:21.280 --> 00:33:21.900 align:middle line:90% It works. 00:33:21.900 --> 00:33:24.680 align:middle line:84% I think the basic idea is reasonable. 00:33:24.680 --> 00:33:27.360 align:middle line:90% And it should work fine. 00:33:27.360 --> 00:33:35.580 align:middle line:84% Then this utility per development is the same thing. 00:33:35.580 --> 00:33:40.280 align:middle line:90% So this is exactly-- no, sorry. 00:33:40.280 --> 00:33:42.920 align:middle line:84% Utility per-- yes, this is exactly the same thing. 00:33:42.920 --> 00:33:46.000 align:middle line:84% So this is actually the increment of utility 00:33:46.000 --> 00:33:47.540 align:middle line:90% that is added. 00:33:47.540 --> 00:33:49.040 align:middle line:84% AUDIENCE: Jake, you mentioned that-- 00:33:49.040 --> 00:33:50.748 align:middle line:84% I think you mentioned [INAUDIBLE] utility 00:33:50.748 --> 00:33:52.600 align:middle line:90% is found between 0 and 1. 00:33:52.600 --> 00:33:55.880 align:middle line:84% HAZHIR RAHMANDAD: And because max-- 00:33:55.880 --> 00:34:04.280 align:middle line:84% there is-- so for that to be bounded between 0 and 1, 00:34:04.280 --> 00:34:06.280 align:middle line:90% how do you enforce that? 00:34:06.280 --> 00:34:10.679 align:middle line:84% AUDIENCE: [INAUDIBLE] exactly, [INAUDIBLE] utility for 00:34:10.679 --> 00:34:12.838 align:middle line:90% [INAUDIBLE]. 00:34:12.838 --> 00:34:13.880 align:middle line:90% HAZHIR RAHMANDAD: Ah, OK. 00:34:13.880 --> 00:34:20.040 align:middle line:84% If it goes above 1, you kill the inflow. 00:34:20.040 --> 00:34:25.600 align:middle line:84% So this is not a good way to normalize it, to get-- 00:34:25.600 --> 00:34:27.520 align:middle line:84% AUDIENCE: I did it another way at first. 00:34:27.520 --> 00:34:33.219 align:middle line:84% And it never actually converged to 1. 00:34:33.219 --> 00:34:37.029 align:middle line:84% And I was like, well, at some point, it's going to stop. 00:34:37.029 --> 00:34:39.179 align:middle line:84% HAZHIR RAHMANDAD: But this creates a discontinuity. 00:34:39.179 --> 00:34:42.000 align:middle line:84% Basically, you have positive marginal utility at one day. 00:34:42.000 --> 00:34:45.840 align:middle line:84% And the next day, it goes to 0, whereas the actual process 00:34:45.840 --> 00:34:49.840 align:middle line:84% is much more like the underlying slow convergence. 00:34:49.840 --> 00:34:57.840 align:middle line:90% So it's an interesting question. 00:34:57.840 --> 00:35:02.080 align:middle line:84% So you might be able to find an analytical solution for what 00:35:02.080 --> 00:35:05.840 align:middle line:84% would be the maximum-- the total utility at the end of time 00:35:05.840 --> 00:35:09.440 align:middle line:84% using this formulation that you have, 00:35:09.440 --> 00:35:15.480 align:middle line:84% as this is basically a first order system, assuming 00:35:15.480 --> 00:35:19.760 align:middle line:84% that you get max utility-- you get, sorry, 00:35:19.760 --> 00:35:23.440 align:middle line:84% capability development rate to be at its maximum-- that 00:35:23.440 --> 00:35:26.240 align:middle line:84% is, always maximum-- even though there is a feedback here 00:35:26.240 --> 00:35:28.040 align:middle line:90% that slows it down. 00:35:28.040 --> 00:35:30.280 align:middle line:84% Even if you assume it is at its maximum, 00:35:30.280 --> 00:35:32.920 align:middle line:90% what would be the total utility? 00:35:32.920 --> 00:35:37.000 align:middle line:90% What would it converge to? 00:35:37.000 --> 00:35:42.200 align:middle line:84% There should be some analytical solution for that. 00:35:42.200 --> 00:35:44.440 align:middle line:84% And if that is the case, then you 00:35:44.440 --> 00:35:46.700 align:middle line:90% can use that as your maximum. 00:35:46.700 --> 00:35:49.200 align:middle line:84% You divide this stock by that maximum. 00:35:49.200 --> 00:35:54.520 align:middle line:84% So that's one way to normalize it to 1. 00:35:54.520 --> 00:35:57.040 align:middle line:84% AUDIENCE: That's close to [INAUDIBLE]. 00:35:57.040 --> 00:36:02.400 align:middle line:84% Yeah, I just, in my head, was like I wanted it to actually hit 00:36:02.400 --> 00:36:04.480 align:middle line:84% 1, where they have a responsibility 00:36:04.480 --> 00:36:09.050 align:middle line:84% to fix the bugs that are still in there. 00:36:09.050 --> 00:36:10.800 align:middle line:84% HAZHIR RAHMANDAD: And the bugs are not yet 00:36:10.800 --> 00:36:13.180 align:middle line:90% showing up here, yeah. 00:36:13.180 --> 00:36:14.680 align:middle line:84% AUDIENCE: Make it the whole product. 00:36:14.680 --> 00:36:16.080 align:middle line:90% HAZHIR RAHMANDAD: Yeah, OK. 00:36:16.080 --> 00:36:20.760 align:middle line:84% So then you have perception of utility 00:36:20.760 --> 00:36:24.320 align:middle line:84% that starts with the total utility. 00:36:24.320 --> 00:36:28.480 align:middle line:84% And then it looks at bugs being fixed. 00:36:28.480 --> 00:36:33.080 align:middle line:84% So here, we have a little bit of a unit problem 00:36:33.080 --> 00:36:36.760 align:middle line:84% in the way I'm looking at it, based on the fact 00:36:36.760 --> 00:36:40.880 align:middle line:84% that bugs may be corresponding to these units 00:36:40.880 --> 00:36:43.280 align:middle line:90% of technological capabilities. 00:36:43.280 --> 00:36:49.200 align:middle line:84% But they do not directly correspond to units of utility. 00:36:49.200 --> 00:36:52.720 align:middle line:84% And therefore, this comparison here becomes harder. 00:36:52.720 --> 00:36:55.000 align:middle line:84% So you have total utility minus sensitivity 00:36:55.000 --> 00:36:56.440 align:middle line:90% to bugs to bugs being fixed. 00:36:56.440 --> 00:37:00.920 align:middle line:84% So you have this added parameter of sensitivity to bugs 00:37:00.920 --> 00:37:02.960 align:middle line:90% to fix the units. 00:37:02.960 --> 00:37:05.240 align:middle line:84% But then you probably have observed 00:37:05.240 --> 00:37:09.520 align:middle line:84% that total utility could be much smaller than the bugs being 00:37:09.520 --> 00:37:10.360 align:middle line:90% fixed. 00:37:10.360 --> 00:37:15.000 align:middle line:84% And therefore, if you just had total utility minus sensitivity 00:37:15.000 --> 00:37:17.840 align:middle line:84% to bugs multiplied by bugs being fixed, that could go negative. 00:37:17.840 --> 00:37:20.840 align:middle line:84% And then that messes up everything. 00:37:20.840 --> 00:37:23.720 align:middle line:84% So then you have added max zero and the other thing. 00:37:23.720 --> 00:37:26.380 align:middle line:90% And that approach is not right. 00:37:26.380 --> 00:37:29.680 align:middle line:84% That's one of those use cases of max zero 00:37:29.680 --> 00:37:31.400 align:middle line:90% that you don't want to have. 00:37:31.400 --> 00:37:35.440 align:middle line:84% Basically, you should say, OK, what 00:37:35.440 --> 00:37:39.020 align:middle line:84% is the concept that is comparable to bugs? 00:37:39.020 --> 00:37:41.600 align:middle line:84% It is my technological capabilities. 00:37:41.600 --> 00:37:46.480 align:middle line:84% So with respect to the current technological capabilities, 00:37:46.480 --> 00:37:53.000 align:middle line:84% how many bugs do I have that are being fixed that are known? 00:37:53.000 --> 00:37:56.270 align:middle line:90% That becomes a ratio-- 00:37:56.270 --> 00:37:59.710 align:middle line:84% if you keep the units of bugs to be in features-- 00:37:59.710 --> 00:38:03.110 align:middle line:84% so buggy features, think of them-- 00:38:03.110 --> 00:38:05.590 align:middle line:84% so technological capability-- buggy features 00:38:05.590 --> 00:38:08.430 align:middle line:84% divided by technological capabilities 00:38:08.430 --> 00:38:17.110 align:middle line:84% gives you a good rate for what the effective features are. 00:38:17.110 --> 00:38:19.470 align:middle line:84% And those effective features could 00:38:19.470 --> 00:38:23.190 align:middle line:84% be driving what is the current perception of utility 00:38:23.190 --> 00:38:25.590 align:middle line:90% or perceived utility would be. 00:38:25.590 --> 00:38:28.510 align:middle line:84% Now, the problem there is that you are capturing utility 00:38:28.510 --> 00:38:31.970 align:middle line:84% with a stock and, therefore, with an accumulation process. 00:38:31.970 --> 00:38:35.290 align:middle line:84% So now, if I shift the total features, 00:38:35.290 --> 00:38:38.730 align:middle line:84% I can't directly go and see what is the corresponding utility. 00:38:38.730 --> 00:38:40.910 align:middle line:90% And that makes the complication. 00:38:40.910 --> 00:38:46.150 align:middle line:84% So that is one place where I don't necessarily 00:38:46.150 --> 00:38:49.490 align:middle line:84% go with the stock version of total utility. 00:38:49.490 --> 00:38:52.430 align:middle line:84% I would go with a simple function that 00:38:52.430 --> 00:38:55.830 align:middle line:84% says, as you increase technological capabilities 00:38:55.830 --> 00:38:57.790 align:middle line:84% or effective technological capabilities 00:38:57.790 --> 00:39:00.190 align:middle line:84% based on what fractions are bugs, 00:39:00.190 --> 00:39:04.130 align:middle line:84% you are going to have more utility, 00:39:04.130 --> 00:39:05.330 align:middle line:90% but with decreasing returns. 00:39:05.330 --> 00:39:09.750 align:middle line:84% So that would be a simple decreasing return formulation 00:39:09.750 --> 00:39:11.310 align:middle line:90% for-- 00:39:11.310 --> 00:39:13.250 align:middle line:90% and you could use logarithm. 00:39:13.250 --> 00:39:21.030 align:middle line:84% You could use something to the power of a small number smaller 00:39:21.030 --> 00:39:21.610 align:middle line:90% than 1. 00:39:21.610 --> 00:39:23.610 align:middle line:84% Something like that would do the trick. 00:39:23.610 --> 00:39:26.110 align:middle line:84% So that's the trade-off that would-- 00:39:26.110 --> 00:39:30.170 align:middle line:84% but once you have put the stock of total utility, 00:39:30.170 --> 00:39:32.970 align:middle line:84% I see how everything else goes in this direction. 00:39:32.970 --> 00:39:35.950 align:middle line:84% That creates a little bit of a dead end 00:39:35.950 --> 00:39:37.770 align:middle line:90% at this point inside the model. 00:39:37.770 --> 00:39:38.350 align:middle line:90% OK. 00:39:38.350 --> 00:39:42.050 align:middle line:84% I think, conceptually, it's all in the right space. 00:39:42.050 --> 00:39:47.390 align:middle line:84% But this creates some challenges that you have probably faced. 00:39:47.390 --> 00:39:51.470 align:middle line:84% I like the idea of getting some measure of incremental value 00:39:51.470 --> 00:39:54.590 align:middle line:84% of utility for the decision to expand 00:39:54.590 --> 00:39:55.890 align:middle line:90% technological capabilities. 00:39:55.890 --> 00:39:58.630 align:middle line:84% And if you were going with this alternative that I was saying, 00:39:58.630 --> 00:40:00.870 align:middle line:84% then you actually needed to capture 00:40:00.870 --> 00:40:07.510 align:middle line:84% that marginal utility in the curve of technology to utility. 00:40:07.510 --> 00:40:11.670 align:middle line:84% And that marginal utility is-- you probably 00:40:11.670 --> 00:40:15.110 align:middle line:84% needed it either analytically or some simulation-based way 00:40:15.110 --> 00:40:16.345 align:middle line:90% of calculating it. 00:40:16.345 --> 00:40:17.970 align:middle line:84% I would go with an analytical solution. 00:40:17.970 --> 00:40:22.670 align:middle line:84% That is, if I have features to the power of alpha, 00:40:22.670 --> 00:40:25.630 align:middle line:84% then marginal utility is just the derivative of this thing 00:40:25.630 --> 00:40:28.590 align:middle line:90% with respect to features. 00:40:28.590 --> 00:40:31.370 align:middle line:84% So that becomes our marginal utility. 00:40:31.370 --> 00:40:33.150 align:middle line:90% OK. 00:40:33.150 --> 00:40:34.630 align:middle line:90% This is nice. 00:40:34.630 --> 00:40:40.110 align:middle line:90% Let's go to some other models. 00:40:40.110 --> 00:40:41.090 align:middle line:90% Which one is this? 00:40:41.090 --> 00:40:42.470 align:middle line:90% This is Jason's. 00:40:42.470 --> 00:40:46.590 align:middle line:84% AUDIENCE: Just warning-- about 50% of the formulations are not 00:40:46.590 --> 00:40:47.123 align:middle line:90% [INAUDIBLE]. 00:40:47.123 --> 00:40:49.290 align:middle line:84% HAZHIR RAHMANDAD: OK, so there are open things here. 00:40:49.290 --> 00:40:50.650 align:middle line:90% AUDIENCE: A lot. 00:40:50.650 --> 00:40:53.230 align:middle line:90% But the hardest ones. 00:40:53.230 --> 00:40:54.550 align:middle line:90% HAZHIR RAHMANDAD: OK. 00:40:54.550 --> 00:40:56.790 align:middle line:84% Then why don't we focus on things 00:40:56.790 --> 00:40:59.070 align:middle line:84% that you want us to focus on, if there 00:40:59.070 --> 00:41:01.630 align:middle line:90% are places that you think are-- 00:41:01.630 --> 00:41:04.050 align:middle line:90% you were struggling. 00:41:04.050 --> 00:41:05.430 align:middle line:90% Let's look at those. 00:41:05.430 --> 00:41:08.670 align:middle line:84% AUDIENCE: Maybe to continue with the last conversation, I guess, 00:41:08.670 --> 00:41:10.590 align:middle line:90% scope and utility. 00:41:10.590 --> 00:41:11.550 align:middle line:90% HAZHIR RAHMANDAD: Yeah. 00:41:11.550 --> 00:41:16.630 align:middle line:84% AUDIENCE: I like the idea of having scope move 00:41:16.630 --> 00:41:18.150 align:middle line:84% people into the addressable market, 00:41:18.150 --> 00:41:23.790 align:middle line:84% and then utility determining what fraction 00:41:23.790 --> 00:41:25.610 align:middle line:84% of that addressable market is realized. 00:41:25.610 --> 00:41:29.350 align:middle line:84% That was the idea I wanted to capture. 00:41:29.350 --> 00:41:30.670 align:middle line:90% I didn't get there. 00:41:30.670 --> 00:41:33.470 align:middle line:90% HAZHIR RAHMANDAD: OK. 00:41:33.470 --> 00:41:34.610 align:middle line:90% So let me see. 00:41:34.610 --> 00:41:39.550 align:middle line:90% 00:41:39.550 --> 00:41:41.790 align:middle line:84% So there are really two different ideas 00:41:41.790 --> 00:41:44.850 align:middle line:84% that I think are happening in different models. 00:41:44.850 --> 00:41:48.470 align:middle line:84% One is that scope and features determine 00:41:48.470 --> 00:41:50.190 align:middle line:90% the addressable market. 00:41:50.190 --> 00:41:55.230 align:middle line:84% And then bugs determine quality within that space 00:41:55.230 --> 00:41:59.990 align:middle line:84% and the movement of people between the two different-- 00:41:59.990 --> 00:42:03.030 align:middle line:84% I don't think that's what you're doing here 00:42:03.030 --> 00:42:05.070 align:middle line:84% because, if you were doing it that way, then 00:42:05.070 --> 00:42:08.310 align:middle line:84% the non-adopter and adopter, you needed another stock. 00:42:08.310 --> 00:42:11.910 align:middle line:84% You needed the stock of-- so I think what Ross was doing 00:42:11.910 --> 00:42:14.570 align:middle line:90% was closer to that idea. 00:42:14.570 --> 00:42:18.670 align:middle line:84% We should get to other models to see exactly how it is done. 00:42:18.670 --> 00:42:21.670 align:middle line:84% But right now, here, you have indicated fractions 00:42:21.670 --> 00:42:24.270 align:middle line:84% from this idea of perceived utility, which is 00:42:24.270 --> 00:42:27.510 align:middle line:90% coming from indicated utility-- 00:42:27.510 --> 00:42:30.030 align:middle line:84% AUDIENCE: Going from [INAUDIBLE]. 00:42:30.030 --> 00:42:31.090 align:middle line:90% HAZHIR RAHMANDAD: So yes. 00:42:31.090 --> 00:42:33.730 align:middle line:84% And then you have addressable market captured here. 00:42:33.730 --> 00:42:36.250 align:middle line:84% So you're doing two operations at the same time. 00:42:36.250 --> 00:42:36.910 align:middle line:90% This is OK. 00:42:36.910 --> 00:42:39.350 align:middle line:84% This is also similar to what Jake was doing. 00:42:39.350 --> 00:42:40.810 align:middle line:90% I'm not saying that is wrong. 00:42:40.810 --> 00:42:42.570 align:middle line:84% But there are two different ways to do it. 00:42:42.570 --> 00:42:45.630 align:middle line:84% One is exactly this way, that you combine these effects 00:42:45.630 --> 00:42:49.750 align:middle line:84% and just have the indicated adoption from both of them. 00:42:49.750 --> 00:42:51.210 align:middle line:90% Or you separate the two. 00:42:51.210 --> 00:42:54.310 align:middle line:84% One is operating on the first gate of getting people 00:42:54.310 --> 00:42:56.050 align:middle line:90% into the addressable market. 00:42:56.050 --> 00:42:59.710 align:middle line:84% And then the second one works on the gate of moving people 00:42:59.710 --> 00:43:01.710 align:middle line:90% into adopting-- 00:43:01.710 --> 00:43:06.470 align:middle line:84% AUDIENCE: It's clearer to make the second option [INAUDIBLE]. 00:43:06.470 --> 00:43:12.150 align:middle line:84% HAZHIR RAHMANDAD: Yeah, so these two are two different options. 00:43:12.150 --> 00:43:14.730 align:middle line:84% I don't have a strong preference for one of them. 00:43:14.730 --> 00:43:18.350 align:middle line:84% I think the second one is a little bit cleaner, conceptually 00:43:18.350 --> 00:43:19.990 align:middle line:90% easier to follow. 00:43:19.990 --> 00:43:22.190 align:middle line:84% And so maybe I would go with that 00:43:22.190 --> 00:43:23.530 align:middle line:90% if I was building the model. 00:43:23.530 --> 00:43:28.290 align:middle line:84% But I haven't thought through all of the trade-offs here. 00:43:28.290 --> 00:43:30.130 align:middle line:84% So we should see some of those in practice. 00:43:30.130 --> 00:43:31.310 align:middle line:90% Anything else? 00:43:31.310 --> 00:43:34.590 align:middle line:84% AUDIENCE: No, that was the main, I think, conceptual hangup. 00:43:34.590 --> 00:43:40.190 align:middle line:84% It's like, yeah, I'm trying to think. 00:43:40.190 --> 00:43:43.790 align:middle line:84% Oh, for the bug accumulation in discovery, 00:43:43.790 --> 00:43:49.870 align:middle line:84% so I have adoption as a coflow into undiscovered bug 00:43:49.870 --> 00:43:51.310 align:middle line:90% accumulation. 00:43:51.310 --> 00:43:53.153 align:middle line:90% You can see the [INAUDIBLE]. 00:43:53.153 --> 00:43:54.070 align:middle line:90% HAZHIR RAHMANDAD: Yes. 00:43:54.070 --> 00:43:58.950 align:middle line:84% So this is tricky because what you're saying here 00:43:58.950 --> 00:44:02.510 align:middle line:84% is that the units of bugs conceptually 00:44:02.510 --> 00:44:05.750 align:middle line:84% is dependent on how many people are there. 00:44:05.750 --> 00:44:13.370 align:middle line:84% So if you have 100 people, you may discover one bug per month. 00:44:13.370 --> 00:44:17.750 align:middle line:84% But if you have 1,000, you may discover $10 per month. 00:44:17.750 --> 00:44:20.710 align:middle line:84% And conceptually, however, these bugs 00:44:20.710 --> 00:44:24.730 align:middle line:84% are unique bugs, that you want them to be unique bugs. 00:44:24.730 --> 00:44:28.310 align:middle line:84% Whereas, the version that you are highlighting this way, 00:44:28.310 --> 00:44:31.010 align:middle line:84% they are double-counting and triple-counting. 00:44:31.010 --> 00:44:36.150 align:middle line:84% If you and I both have the same bug, both of us 00:44:36.150 --> 00:44:39.630 align:middle line:84% add to this undiscovered bugs stock. 00:44:39.630 --> 00:44:41.445 align:middle line:90% Is that a correct understanding? 00:44:41.445 --> 00:44:43.570 align:middle line:84% AUDIENCE: [INAUDIBLE] it was a correct undertanding 00:44:43.570 --> 00:44:43.790 align:middle line:90% [? because-- ?] 00:44:43.790 --> 00:44:45.332 align:middle line:84% HAZHIR RAHMANDAD: Now, if you do it-- 00:44:45.332 --> 00:44:47.030 align:middle line:84% as soon as you do it that way, things 00:44:47.030 --> 00:44:49.190 align:middle line:84% get messed up because now there is 00:44:49.190 --> 00:44:56.910 align:middle line:84% no comparative stock of bug-free features in the world. 00:44:56.910 --> 00:44:58.250 align:middle line:90% Now, you could create that. 00:44:58.250 --> 00:45:00.670 align:middle line:84% You could basically count the number of features 00:45:00.670 --> 00:45:03.870 align:middle line:84% that you have already sold by number of features 00:45:03.870 --> 00:45:06.030 align:middle line:84% in the product multiplied by the number of people 00:45:06.030 --> 00:45:07.790 align:middle line:90% who are buying the product. 00:45:07.790 --> 00:45:09.610 align:middle line:90% But that's way too convoluted. 00:45:09.610 --> 00:45:11.170 align:middle line:84% And you're not getting much out of it 00:45:11.170 --> 00:45:13.090 align:middle line:84% because then you're dividing it to get-- 00:45:13.090 --> 00:45:15.910 align:middle line:84% or maybe you would need to divide it, at some point, 00:45:15.910 --> 00:45:19.870 align:middle line:84% to get what is the fraction of buggy features. 00:45:19.870 --> 00:45:23.150 align:middle line:84% So just focus on features rather than-- 00:45:23.150 --> 00:45:25.610 align:middle line:90% but why did you have that? 00:45:25.610 --> 00:45:30.510 align:middle line:84% AUDIENCE: Because the thinking was 00:45:30.510 --> 00:45:34.590 align:middle line:84% there's a difference between a fluke, a freak 00:45:34.590 --> 00:45:38.630 align:middle line:90% accident and a legitimate bug. 00:45:38.630 --> 00:45:45.710 align:middle line:84% And you need volume to amount to it being a real bug and not 00:45:45.710 --> 00:45:49.790 align:middle line:90% just a mistake. 00:45:49.790 --> 00:45:51.950 align:middle line:90% HAZHIR RAHMANDAD: OK, that's-- 00:45:51.950 --> 00:45:54.750 align:middle line:84% so to capture that, essentially, what 00:45:54.750 --> 00:45:57.690 align:middle line:84% you have is that, first of all, it is undiscovered bugs. 00:45:57.690 --> 00:46:00.670 align:middle line:84% So if it is undiscovered, it is not yet even known. 00:46:00.670 --> 00:46:03.890 align:middle line:84% So these are bugs that are there in the product, 00:46:03.890 --> 00:46:05.750 align:middle line:90% but we don't know about it. 00:46:05.750 --> 00:46:09.670 align:middle line:84% So that part, as soon as it is developed poorly and released 00:46:09.670 --> 00:46:12.610 align:middle line:84% into the market with those bugs, it goes into this stock. 00:46:12.610 --> 00:46:16.110 align:middle line:84% So this should be a coflow with the feature development. 00:46:16.110 --> 00:46:17.470 align:middle line:90% So that's very clear. 00:46:17.470 --> 00:46:20.190 align:middle line:84% Now, once it is a bug that is undiscovered, 00:46:20.190 --> 00:46:21.770 align:middle line:90% there is a discovery process. 00:46:21.770 --> 00:46:25.510 align:middle line:84% And the time to discover could very much 00:46:25.510 --> 00:46:29.390 align:middle line:84% be dynamically related to your adoption 00:46:29.390 --> 00:46:33.370 align:middle line:84% rate or your recent adopters or something along those lines. 00:46:33.370 --> 00:46:36.190 align:middle line:84% I would probably just anchor it to adoption rate. 00:46:36.190 --> 00:46:39.550 align:middle line:84% But recent adoption rate might be even a better one 00:46:39.550 --> 00:46:42.590 align:middle line:84% because recent adopters are observing, 00:46:42.590 --> 00:46:44.470 align:middle line:90% experiencing those challenges. 00:46:44.470 --> 00:46:47.930 align:middle line:84% That reduces the time to discover bugs. 00:46:47.930 --> 00:46:51.507 align:middle line:84% So that would be the right way of doing it. 00:46:51.507 --> 00:46:52.590 align:middle line:90% AUDIENCE: This is helpful. 00:46:52.590 --> 00:46:53.230 align:middle line:90% HAZHIR RAHMANDAD: Sure. 00:46:53.230 --> 00:46:53.850 align:middle line:90% OK. 00:46:53.850 --> 00:46:56.130 align:middle line:90% So any questions on Jason's? 00:46:56.130 --> 00:46:56.630 align:middle line:90% OK. 00:46:56.630 --> 00:46:59.970 align:middle line:84% Actually, let me just see something else here. 00:46:59.970 --> 00:47:04.750 align:middle line:84% So here, you have product capabilities increasing 00:47:04.750 --> 00:47:06.810 align:middle line:90% based on potential capability. 00:47:06.810 --> 00:47:10.550 align:middle line:84% And potential capability is coming from non-adopters. 00:47:10.550 --> 00:47:15.035 align:middle line:84% So this was just the idea not yet-- 00:47:15.035 --> 00:47:15.910 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:47:15.910 --> 00:47:17.730 align:middle line:84% HAZHIR RAHMANDAD: So I think what Jake had, 00:47:17.730 --> 00:47:21.770 align:middle line:84% which was something about the marginal value of new features, 00:47:21.770 --> 00:47:25.270 align:middle line:84% is a pretty smart way of thinking about these 00:47:25.270 --> 00:47:27.390 align:middle line:90% that economists would like. 00:47:27.390 --> 00:47:29.110 align:middle line:84% Now, whether companies are exactly 00:47:29.110 --> 00:47:31.270 align:middle line:90% tracking marginal utility-- 00:47:31.270 --> 00:47:34.510 align:middle line:84% these days, yes, by and large, the whole product development 00:47:34.510 --> 00:47:36.710 align:middle line:84% process is organized around, OK, let's 00:47:36.710 --> 00:47:39.830 align:middle line:84% assess the value of each new feature for customers 00:47:39.830 --> 00:47:43.610 align:middle line:84% and decide, cost-benefit-wise, does it have the ROI? 00:47:43.610 --> 00:47:46.710 align:middle line:84% So I think it is a decent approximation of what 00:47:46.710 --> 00:47:48.050 align:middle line:90% even happens in the real world. 00:47:48.050 --> 00:47:51.390 align:middle line:84% So we should follow something along those lines 00:47:51.390 --> 00:47:55.870 align:middle line:90% for adding new features. 00:47:55.870 --> 00:47:59.830 align:middle line:84% But it is also tricky because the utility 00:47:59.830 --> 00:48:04.510 align:middle line:84% on the eye of the company is not just this concept utility 00:48:04.510 --> 00:48:06.250 align:middle line:84% that I said, OK, take the features, 00:48:06.250 --> 00:48:09.490 align:middle line:84% raise it to power of some alpha, and that becomes utility. 00:48:09.490 --> 00:48:11.450 align:middle line:84% What do you do with that final utility? 00:48:11.450 --> 00:48:15.030 align:middle line:84% You now take it back into the customer sector. 00:48:15.030 --> 00:48:18.350 align:middle line:84% And you would say, OK, how many customers would I 00:48:18.350 --> 00:48:20.530 align:middle line:90% get with that marginal utility? 00:48:20.530 --> 00:48:24.890 align:middle line:84% You need to calculate that before you can-- 00:48:24.890 --> 00:48:27.150 align:middle line:84% and then what is the profitability I would expect 00:48:27.150 --> 00:48:30.430 align:middle line:84% from getting those additional customers so that you 00:48:30.430 --> 00:48:34.190 align:middle line:84% get to something about the cost benefits? 00:48:34.190 --> 00:48:37.850 align:middle line:84% So there is more that goes into a very serious, 00:48:37.850 --> 00:48:39.210 align:middle line:90% robust formulation there. 00:48:39.210 --> 00:48:44.670 align:middle line:84% But the core is the idea of marginal utility. 00:48:44.670 --> 00:48:45.350 align:middle line:90% OK. 00:48:45.350 --> 00:48:46.670 align:middle line:90% Let's-- 00:48:46.670 --> 00:48:49.110 align:middle line:84% AUDIENCE: [INAUDIBLE] better to include 00:48:49.110 --> 00:48:51.590 align:middle line:84% the decision-making process [INAUDIBLE]. 00:48:51.590 --> 00:48:54.907 align:middle line:90% 00:48:54.907 --> 00:48:56.990 align:middle line:84% HAZHIR RAHMANDAD: So what do you mean by including 00:48:56.990 --> 00:48:58.750 align:middle line:90% in the decision making process? 00:48:58.750 --> 00:49:00.375 align:middle line:90% AUDIENCE: Of the [INAUDIBLE]. 00:49:00.375 --> 00:49:01.750 align:middle line:84% HAZHIR RAHMANDAD: In the company. 00:49:01.750 --> 00:49:04.730 align:middle line:84% Yeah, but how do you want it to be represented? 00:49:04.730 --> 00:49:07.110 align:middle line:84% AUDIENCE: Yeah, like I was thinking [INAUDIBLE], 00:49:07.110 --> 00:49:11.630 align:middle line:84% how should we assess if adding some feature 00:49:11.630 --> 00:49:15.310 align:middle line:84% was basically bringing us customers or profit or whatever. 00:49:15.310 --> 00:49:17.590 align:middle line:90% [INAUDIBLE] 00:49:17.590 --> 00:49:18.250 align:middle line:90% I don't know. 00:49:18.250 --> 00:49:27.550 align:middle line:84% But another thing I was thinking [INAUDIBLE] 00:49:27.550 --> 00:49:30.670 align:middle line:84% having limited resources and having synthetic problems 00:49:30.670 --> 00:49:32.570 align:middle line:90% or developing features. 00:49:32.570 --> 00:49:41.430 align:middle line:84% Then you have to decide [INAUDIBLE] was helping 00:49:41.430 --> 00:49:42.650 align:middle line:90% in that decision-making. 00:49:42.650 --> 00:49:47.250 align:middle line:84% [INAUDIBLE] having this discussion separately 00:49:47.250 --> 00:49:47.750 align:middle line:90% [INAUDIBLE]. 00:49:47.750 --> 00:49:49.583 align:middle line:84% HAZHIR RAHMANDAD: We should talk separately? 00:49:49.583 --> 00:49:52.950 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:49:52.950 --> 00:49:54.267 align:middle line:90% 00:49:54.267 --> 00:49:56.350 align:middle line:84% HAZHIR RAHMANDAD: But you have the discovered bugs 00:49:56.350 --> 00:49:58.270 align:middle line:90% that are accumulated here. 00:49:58.270 --> 00:50:01.230 align:middle line:84% And then once they are fixed, they are in the fixed bugs. 00:50:01.230 --> 00:50:02.810 align:middle line:90% Which one are you saying? 00:50:02.810 --> 00:50:05.850 align:middle line:84% AUDIENCE: I was saying I was going to make that decision. 00:50:05.850 --> 00:50:11.213 align:middle line:84% [INAUDIBLE] this number as well, which is [INAUDIBLE]. 00:50:11.213 --> 00:50:13.630 align:middle line:84% HAZHIR RAHMANDAD: Yeah, so the resource allocation problem 00:50:13.630 --> 00:50:14.290 align:middle line:90% is a real one. 00:50:14.290 --> 00:50:17.390 align:middle line:84% So that is, basically, do I invest in fixing bugs or do I 00:50:17.390 --> 00:50:21.470 align:middle line:84% invest in developing new features/ And that is not always 00:50:21.470 --> 00:50:25.150 align:middle line:84% obvious that I should always spend more on bug fixing 00:50:25.150 --> 00:50:29.390 align:middle line:84% or prioritize bug fixing because there are some bugs that are not 00:50:29.390 --> 00:50:30.250 align:middle line:90% that important. 00:50:30.250 --> 00:50:33.510 align:middle line:90% So in a sense, it is-- 00:50:33.510 --> 00:50:35.650 align:middle line:84% but then bugs that are not so important, 00:50:35.650 --> 00:50:37.890 align:middle line:84% maybe we could ignore them, in a sense. 00:50:37.890 --> 00:50:41.110 align:middle line:84% Defining bugs as things that are high enough a priority that, 00:50:41.110 --> 00:50:44.190 align:middle line:84% once you have discovered them, it is better to fix them first. 00:50:44.190 --> 00:50:50.670 align:middle line:84% That would be my option for a simpler model 00:50:50.670 --> 00:50:53.750 align:middle line:84% without going through too much details there 00:50:53.750 --> 00:50:58.470 align:middle line:84% because the alternative is to think about the marginal utility 00:50:58.470 --> 00:51:02.150 align:middle line:84% of fixing a bug and to capture in that marginal utility 00:51:02.150 --> 00:51:04.770 align:middle line:84% of fixing the bug, or marginal profit from fixing a bug, 00:51:04.770 --> 00:51:10.450 align:middle line:84% you have to trace the impact of that bug on your sales, 00:51:10.450 --> 00:51:13.830 align:middle line:84% on losing customers who are now, because of the bug, disappointed 00:51:13.830 --> 00:51:16.550 align:middle line:84% and are moving into, OK, I'm not going-- 00:51:16.550 --> 00:51:20.050 align:middle line:84% even though I'm within the scope, I'm not going to do this. 00:51:20.050 --> 00:51:21.370 align:middle line:90% And that's too much work. 00:51:21.370 --> 00:51:23.958 align:middle line:90% And it becomes messy, messy. 00:51:23.958 --> 00:51:25.750 align:middle line:84% AUDIENCE: Yeah, I was thinking [INAUDIBLE]. 00:51:25.750 --> 00:51:28.070 align:middle line:84% HAZHIR RAHMANDAD: Yeah, so I think the first order 00:51:28.070 --> 00:51:32.150 align:middle line:84% solution is to actually think of bugs as bad enough 00:51:32.150 --> 00:51:33.650 align:middle line:90% that you should fix them. 00:51:33.650 --> 00:51:37.510 align:middle line:84% It is, cost-benefit-wise, more important 00:51:37.510 --> 00:51:40.390 align:middle line:84% than the new features to be added. 00:51:40.390 --> 00:51:41.830 align:middle line:90% First, fix those bugs. 00:51:41.830 --> 00:51:45.210 align:middle line:84% And once you have defined it with that boundary condition, 00:51:45.210 --> 00:51:48.210 align:middle line:84% then the resource allocation becomes pretty straightforward. 00:51:48.210 --> 00:51:49.490 align:middle line:90% OK, you have a bunch of bugs. 00:51:49.490 --> 00:51:51.630 align:middle line:84% You have a desired bug fixing rate. 00:51:51.630 --> 00:51:54.690 align:middle line:84% That captures a subset of your R&D resources. 00:51:54.690 --> 00:51:59.410 align:middle line:84% The remainder can be allocated to building your new features, 00:51:59.410 --> 00:52:02.610 align:middle line:84% but provided that those new features are worth it. 00:52:02.610 --> 00:52:03.830 align:middle line:90% And that becomes the-- 00:52:03.830 --> 00:52:05.410 align:middle line:90% AUDIENCE: Also, [INAUDIBLE]. 00:52:05.410 --> 00:52:12.942 align:middle line:90% 00:52:12.942 --> 00:52:15.150 align:middle line:84% HAZHIR RAHMANDAD: So adding of a new product feature, 00:52:15.150 --> 00:52:16.210 align:middle line:90% we went through it. 00:52:16.210 --> 00:52:19.850 align:middle line:84% It's more driven by the marginal benefit of the new feature. 00:52:19.850 --> 00:52:21.870 align:middle line:84% I mean, that would be one formulation-- that 00:52:21.870 --> 00:52:25.010 align:middle line:84% is, the marginal benefit of a new feature, what is it. 00:52:25.010 --> 00:52:27.990 align:middle line:84% And calculating that could be also tricky. 00:52:27.990 --> 00:52:33.830 align:middle line:84% We talked about an analytical function from feature 00:52:33.830 --> 00:52:35.790 align:middle line:90% to utility. 00:52:35.790 --> 00:52:39.430 align:middle line:84% Then you have an analytical function from utility 00:52:39.430 --> 00:52:44.150 align:middle line:84% to number of customers who are adopting. 00:52:44.150 --> 00:52:46.870 align:middle line:84% Then these two marginals multiplied by each other 00:52:46.870 --> 00:52:52.430 align:middle line:84% give you the total marginal that you would get. 00:52:52.430 --> 00:52:54.040 align:middle line:90% OK. 00:52:54.040 --> 00:52:56.610 align:middle line:90% So let's look at this one. 00:52:56.610 --> 00:52:59.070 align:middle line:90% It is Hesam's. 00:52:59.070 --> 00:53:04.670 align:middle line:84% So we have the three-piece customer sector 00:53:04.670 --> 00:53:07.470 align:middle line:90% with potential adopters. 00:53:07.470 --> 00:53:09.990 align:middle line:84% And Hesam doesn't have unit errors, 00:53:09.990 --> 00:53:12.950 align:middle line:84% even though he didn't like the units here. 00:53:12.950 --> 00:53:15.990 align:middle line:90% AUDIENCE: [INAUDIBLE]. 00:53:15.990 --> 00:53:18.510 align:middle line:84% I started with the two-stock thing. 00:53:18.510 --> 00:53:24.030 align:middle line:84% Then I moved to three because the expansion of scope 00:53:24.030 --> 00:53:26.390 align:middle line:90% was a goal-seeking behavior. 00:53:26.390 --> 00:53:31.110 align:middle line:84% And then that created a jump at the beginning 00:53:31.110 --> 00:53:35.430 align:middle line:84% in the adoption rate, if we weren't starting at equilibrium. 00:53:35.430 --> 00:53:36.770 align:middle line:90% And then I didn't like that. 00:53:36.770 --> 00:53:37.680 align:middle line:90% So I moved to three. 00:53:37.680 --> 00:53:38.930 align:middle line:90% HAZHIR RAHMANDAD: Sounds good. 00:53:38.930 --> 00:53:41.790 align:middle line:90% So that's one reason to do it. 00:53:41.790 --> 00:53:45.950 align:middle line:84% Now, that one you could manage in other ways. 00:53:45.950 --> 00:53:48.530 align:middle line:84% So I think the better reason-- not the better reason, 00:53:48.530 --> 00:53:51.150 align:middle line:84% but structurally more problematic part, 00:53:51.150 --> 00:53:54.950 align:middle line:84% more challenging part exactly, is related to these marginals. 00:53:54.950 --> 00:54:00.050 align:middle line:84% And if you combine the two effects of bugs and features, 00:54:00.050 --> 00:54:02.490 align:middle line:84% it becomes too complex to trace the marginals. 00:54:02.490 --> 00:54:06.155 align:middle line:84% So separating them might be a little bit easier conceptually. 00:54:06.155 --> 00:54:07.030 align:middle line:90% AUDIENCE: [INAUDIBLE] 00:54:07.030 --> 00:54:10.710 align:middle line:84% HAZHIR RAHMANDAD: Yeah, no, but that's fine. 00:54:10.710 --> 00:54:11.430 align:middle line:90% OK. 00:54:11.430 --> 00:54:15.350 align:middle line:84% So the scope is based on indicated potential adopters. 00:54:15.350 --> 00:54:18.790 align:middle line:84% That comes from total population and the scope of use. 00:54:18.790 --> 00:54:21.590 align:middle line:84% Scope of use comes from features. 00:54:21.590 --> 00:54:24.110 align:middle line:84% AUDIENCE: S-shape of the features [INAUDIBLE]. 00:54:24.110 --> 00:54:26.300 align:middle line:84% So additional features don't get you [INAUDIBLE]. 00:54:26.300 --> 00:54:28.050 align:middle line:84% HAZHIR RAHMANDAD: That's very interesting. 00:54:28.050 --> 00:54:32.790 align:middle line:84% So you're using an S-shaped, which is fine. 00:54:32.790 --> 00:54:36.750 align:middle line:84% In fact, there is an argument for going with an S-shaped 00:54:36.750 --> 00:54:40.350 align:middle line:84% because, at the beginning, if you have no features, 00:54:40.350 --> 00:54:42.210 align:middle line:90% it's a pretty useless thing. 00:54:42.210 --> 00:54:45.030 align:middle line:84% Even the first feature is still not very useful. 00:54:45.030 --> 00:54:47.830 align:middle line:84% So there is a minimum viable feature. 00:54:47.830 --> 00:54:50.510 align:middle line:84% So if you are interested in a startup-- that 00:54:50.510 --> 00:54:53.350 align:middle line:84% is, at the very beginning of the growth of the market 00:54:53.350 --> 00:54:56.670 align:middle line:84% as well-- then having the S-shape makes sense. 00:54:56.670 --> 00:55:01.170 align:middle line:84% Now, for once you have a minimum viable product, 00:55:01.170 --> 00:55:04.310 align:middle line:84% then you really are looking at the decreasing return 00:55:04.310 --> 00:55:06.510 align:middle line:90% part of the scope. 00:55:06.510 --> 00:55:09.550 align:middle line:90% So that part is not as much. 00:55:09.550 --> 00:55:11.330 align:middle line:90% So I'm fine with it. 00:55:11.330 --> 00:55:13.650 align:middle line:84% Now, when I'm thinking about the marginal parts, 00:55:13.650 --> 00:55:15.810 align:middle line:84% as soon as you have an S-shape, I'm saying, 00:55:15.810 --> 00:55:18.630 align:middle line:84% OK, do we have analytical solutions for the S-shape you 00:55:18.630 --> 00:55:19.170 align:middle line:90% have chosen? 00:55:19.170 --> 00:55:19.850 align:middle line:90% Probably, yes. 00:55:19.850 --> 00:55:22.850 align:middle line:84% If this is a logistic curve, we have an analytical solution. 00:55:22.850 --> 00:55:26.733 align:middle line:84% It's easy to use the marginal thinking in that phase as well. 00:55:26.733 --> 00:55:29.150 align:middle line:84% The problem, though, is that the marginal thinking doesn't 00:55:29.150 --> 00:55:32.210 align:middle line:84% work for the introduction of features at the very beginning 00:55:32.210 --> 00:55:34.870 align:middle line:84% because, at the beginning, every startup, 00:55:34.870 --> 00:55:38.252 align:middle line:84% in expectation, is going to be wiped out. 00:55:38.252 --> 00:55:39.710 align:middle line:84% And therefore, nobody would invest. 00:55:39.710 --> 00:55:41.130 align:middle line:90% And therefore, nothing happens. 00:55:41.130 --> 00:55:44.550 align:middle line:84% So that is a little bit additional consideration 00:55:44.550 --> 00:55:46.590 align:middle line:84% in the back of my mind when I look 00:55:46.590 --> 00:55:50.390 align:middle line:90% at this S-shaped alternative. 00:55:50.390 --> 00:55:54.210 align:middle line:84% OK, then I need to define a honeymoon period, 00:55:54.210 --> 00:55:57.690 align:middle line:84% an irrational investment period for the startup phase 00:55:57.690 --> 00:56:00.710 align:middle line:84% before we get into the point that it is marginally 00:56:00.710 --> 00:56:02.050 align:middle line:90% viable to invest anything. 00:56:02.050 --> 00:56:05.230 align:middle line:90% OK, all good. 00:56:05.230 --> 00:56:08.370 align:middle line:84% But I think the structure here looks solid. 00:56:08.370 --> 00:56:13.190 align:middle line:84% You get a scope of use, which is a number between 0 00:56:13.190 --> 00:56:16.910 align:middle line:84% and a max scope, which is what fraction of total population 00:56:16.910 --> 00:56:18.570 align:middle line:90% would ever be interested. 00:56:18.570 --> 00:56:22.190 align:middle line:84% So this is an important concept to keep in mind. 00:56:22.190 --> 00:56:24.810 align:middle line:84% For pacemakers, I don't think ever 00:56:24.810 --> 00:56:26.950 align:middle line:84% we would have 80% of the population 00:56:26.950 --> 00:56:29.795 align:middle line:84% be in scope for having a pacemaker. 00:56:29.795 --> 00:56:30.670 align:middle line:90% I mean, I don't know. 00:56:30.670 --> 00:56:34.010 align:middle line:84% With the current world where it is going, maybe we have this. 00:56:34.010 --> 00:56:36.590 align:middle line:90% 00:56:36.590 --> 00:56:39.490 align:middle line:90% There is a good term for it. 00:56:39.490 --> 00:56:47.070 align:middle line:84% We are biologically enhanced people or something. 00:56:47.070 --> 00:56:50.057 align:middle line:84% Yeah, anyway, but that's the meaning of that parameter. 00:56:50.057 --> 00:56:51.890 align:middle line:84% It's important to keep that meaning in mind. 00:56:51.890 --> 00:56:53.710 align:middle line:84% This is the total fraction of population 00:56:53.710 --> 00:56:56.250 align:middle line:84% that could ever be interested in this product. 00:56:56.250 --> 00:56:58.110 align:middle line:84% Now, if your product is not pacemaker 00:56:58.110 --> 00:57:00.153 align:middle line:84% and it is something else, then it may very well-- 00:57:00.153 --> 00:57:02.070 align:middle line:84% AUDIENCE: I was thinking of potential adopters 00:57:02.070 --> 00:57:05.550 align:middle line:84% being all people who have heart problems, but [INAUDIBLE]. 00:57:05.550 --> 00:57:08.470 align:middle line:84% HAZHIR RAHMANDAD: But I'm highlighting that because then 00:57:08.470 --> 00:57:10.150 align:middle line:84% your definition of total population 00:57:10.150 --> 00:57:13.190 align:middle line:84% is total population with heart problems. 00:57:13.190 --> 00:57:15.750 align:middle line:84% So then you need to make that more clear 00:57:15.750 --> 00:57:18.730 align:middle line:84% when you are talking about this, the main total population. 00:57:18.730 --> 00:57:20.272 align:middle line:84% Anyway, we are all on the same page. 00:57:20.272 --> 00:57:22.230 align:middle line:84% AUDIENCE: I remembered the text of the question 00:57:22.230 --> 00:57:24.730 align:middle line:84% suggested that there would be always a minimum scope, 00:57:24.730 --> 00:57:26.790 align:middle line:84% as in that there are some people that, 00:57:26.790 --> 00:57:31.310 align:middle line:84% no matter how flawed these devices are, have 00:57:31.310 --> 00:57:33.890 align:middle line:90% no other choice but pacemakers. 00:57:33.890 --> 00:57:35.230 align:middle line:90% And that is the main focus. 00:57:35.230 --> 00:57:36.188 align:middle line:90% HAZHIR RAHMANDAD: Yeah. 00:57:36.188 --> 00:57:38.470 align:middle line:84% So in that sense, it is a little bit tricky 00:57:38.470 --> 00:57:40.870 align:middle line:84% because, once you have that minimum scope, then 00:57:40.870 --> 00:57:43.470 align:middle line:90% your definition of feature-- 00:57:43.470 --> 00:57:46.050 align:middle line:84% what is the zero feature, the first feature? 00:57:46.050 --> 00:57:48.630 align:middle line:90% Is it already built? 00:57:48.630 --> 00:57:51.950 align:middle line:84% Zero means there is already a bunch of features there. 00:57:51.950 --> 00:57:54.110 align:middle line:84% And then at that point, do we really 00:57:54.110 --> 00:57:56.170 align:middle line:90% have an S-shape in utility? 00:57:56.170 --> 00:57:59.390 align:middle line:84% So you see, these are kind of small. 00:57:59.390 --> 00:58:01.550 align:middle line:90% But they actually could matter. 00:58:01.550 --> 00:58:07.870 align:middle line:84% So we want to be clear what's our theory of feature to utility 00:58:07.870 --> 00:58:09.990 align:middle line:90% or to potential adopters. 00:58:09.990 --> 00:58:11.910 align:middle line:90% OK, good. 00:58:11.910 --> 00:58:14.710 align:middle line:84% So then you get the potential adopters 00:58:14.710 --> 00:58:17.770 align:middle line:90% that go into in scope adopters. 00:58:17.770 --> 00:58:24.030 align:middle line:84% And then scope adopters turn into adopters 00:58:24.030 --> 00:58:27.110 align:middle line:90% with a time constant. 00:58:27.110 --> 00:58:31.550 align:middle line:84% And that time constant depends on perceived utility 00:58:31.550 --> 00:58:33.532 align:middle line:90% and depends on-- 00:58:33.532 --> 00:58:34.990 align:middle line:84% AUDIENCE: The perceived utility was 00:58:34.990 --> 00:58:38.690 align:middle line:84% coming from a combination of bugs and-- 00:58:38.690 --> 00:58:40.990 align:middle line:84% HAZHIR RAHMANDAD: Yeah, so that formulation 00:58:40.990 --> 00:58:42.650 align:middle line:90% has a little bit of a problem. 00:58:42.650 --> 00:58:45.710 align:middle line:84% So think about the extreme condition 00:58:45.710 --> 00:58:52.090 align:middle line:84% where you had very fast adoption of these in-scope adopters. 00:58:52.090 --> 00:58:55.870 align:middle line:84% Now, pretty much all of them are adopters because this 00:58:55.870 --> 00:58:57.870 align:middle line:90% sounded like a great idea. 00:58:57.870 --> 00:59:00.670 align:middle line:84% And now, a year later, there is research 00:59:00.670 --> 00:59:03.270 align:middle line:84% that shows this thing has lots of bugs. 00:59:03.270 --> 00:59:07.150 align:middle line:84% And now, these people actually want to stop using the product. 00:59:07.150 --> 00:59:09.270 align:middle line:90% They want to move back. 00:59:09.270 --> 00:59:11.570 align:middle line:84% And they don't have any option here. 00:59:11.570 --> 00:59:18.510 align:middle line:84% So that adoption rate here, using just the first order 00:59:18.510 --> 00:59:22.230 align:middle line:84% correction, can create this challenge. 00:59:22.230 --> 00:59:26.670 align:middle line:84% So I would probably go with some version of, OK, this 00:59:26.670 --> 00:59:33.590 align:middle line:84% is based on the current utility, the fraction of these two groups 00:59:33.590 --> 00:59:39.110 align:middle line:84% that needs to be in the adopters box so that the effect of bugs 00:59:39.110 --> 00:59:41.810 align:middle line:84% is much more clearly captured here. 00:59:41.810 --> 00:59:46.270 align:middle line:90% 00:59:46.270 --> 00:59:49.150 align:middle line:84% And then the time to adopt, the way you're formulating it 00:59:49.150 --> 00:59:52.070 align:middle line:90% is a linear-- 00:59:52.070 --> 00:59:54.130 align:middle line:84% AUDIENCE: And perceived utility between 0 and 1. 00:59:54.130 --> 00:59:55.505 align:middle line:84% HAZHIR RAHMANDAD: Yeah, so you're 00:59:55.505 --> 01:00:01.950 align:middle line:84% doing a linear increase from minimum time to adopt to-- 01:00:01.950 --> 01:00:04.950 align:middle line:84% so if the perceived utility goes to 0, 01:00:04.950 --> 01:00:08.225 align:middle line:90% you have minimum time to adopt-- 01:00:08.225 --> 01:00:09.350 align:middle line:90% AUDIENCE: Plus [INAUDIBLE]. 01:00:09.350 --> 01:00:12.870 align:middle line:84% HAZHIR RAHMANDAD: Plus this nominal time to adopt. 01:00:12.870 --> 01:00:16.030 align:middle line:84% So this is a very stylistic thing. 01:00:16.030 --> 01:00:17.830 align:middle line:84% I would probably have minimum time 01:00:17.830 --> 01:00:21.110 align:middle line:84% to adopt multiplied by perceived utility plus 1 minus perceived 01:00:21.110 --> 01:00:23.310 align:middle line:84% utility multiplied by maximum time 01:00:23.310 --> 01:00:25.710 align:middle line:84% to adopt so that the two concepts are 01:00:25.710 --> 01:00:32.310 align:middle line:84% easy, independent, clear, mathematically identical. 01:00:32.310 --> 01:00:39.110 align:middle line:84% And linear, fine, fine, because you have control over those time 01:00:39.110 --> 01:00:41.310 align:middle line:90% constants that you're using. 01:00:41.310 --> 01:00:44.470 align:middle line:84% But in fact, I wouldn't have used perceived utility 01:00:44.470 --> 01:00:46.230 align:middle line:90% in the first place. 01:00:46.230 --> 01:00:49.510 align:middle line:84% So the thing I would have probably used more clearly 01:00:49.510 --> 01:00:52.690 align:middle line:84% is the fraction of adopters or the recent adoption rate 01:00:52.690 --> 01:00:56.970 align:middle line:84% or something that tells if there is more customers, 01:00:56.970 --> 01:00:58.990 align:middle line:84% there is more word of mouth, that increases 01:00:58.990 --> 01:01:00.330 align:middle line:90% the speed of adoption. 01:01:00.330 --> 01:01:02.970 align:middle line:84% So that would be the effect on the time. 01:01:02.970 --> 01:01:06.850 align:middle line:84% And then there is this desired level that you are shifting. 01:01:06.850 --> 01:01:09.970 align:middle line:84% And that desired level is not all of the in-scope adopters, 01:01:09.970 --> 01:01:12.670 align:middle line:84% but may depend on how many people have already 01:01:12.670 --> 01:01:14.610 align:middle line:90% adopted versus potential. 01:01:14.610 --> 01:01:20.350 align:middle line:84% Then you have bug discovery that comes 01:01:20.350 --> 01:01:22.790 align:middle line:90% from fraction of adopters-- 01:01:22.790 --> 01:01:25.630 align:middle line:84% AUDIENCE: I listened to what you were 01:01:25.630 --> 01:01:28.478 align:middle line:84% saying to Jason, which makes a lot more sense than how I-- 01:01:28.478 --> 01:01:30.270 align:middle line:84% HAZHIR RAHMANDAD: Yeah, so this is not-- it 01:01:30.270 --> 01:01:31.970 align:middle line:84% shouldn't come from the total stock. 01:01:31.970 --> 01:01:33.750 align:middle line:84% It should come from the recent adoption 01:01:33.750 --> 01:01:38.430 align:middle line:84% rate, even from adoption rate itself directly 01:01:38.430 --> 01:01:39.962 align:middle line:90% for the bug discovery time. 01:01:39.962 --> 01:01:41.670 align:middle line:84% AUDIENCE: Introducing, right [INAUDIBLE]? 01:01:41.670 --> 01:01:42.628 align:middle line:90% HAZHIR RAHMANDAD: What? 01:01:42.628 --> 01:01:44.692 align:middle line:84% AUDIENCE: You guys are introducing [INAUDIBLE]. 01:01:44.692 --> 01:01:46.150 align:middle line:84% HAZHIR RAHMANDAD: Introducing bugs? 01:01:46.150 --> 01:01:46.650 align:middle line:90% Right. 01:01:46.650 --> 01:01:49.230 align:middle line:84% Yes, that's coming directly from bugs per feature. 01:01:49.230 --> 01:01:53.730 align:middle line:90% Now, yes, this is fine. 01:01:53.730 --> 01:01:57.130 align:middle line:84% I would even think of bugs as buggy features 01:01:57.130 --> 01:01:59.750 align:middle line:84% so that this bugs per feature becomes something 01:01:59.750 --> 01:02:03.630 align:middle line:84% like error rate or error fraction, 01:02:03.630 --> 01:02:05.590 align:middle line:90% so that it is simplified. 01:02:05.590 --> 01:02:09.510 align:middle line:84% Then I can compare bugs and features directly. 01:02:09.510 --> 01:02:15.070 align:middle line:84% That just keeps the formulation a little bit cleaner. 01:02:15.070 --> 01:02:21.310 align:middle line:84% Then you have the debugging R&D budget is going to the-- 01:02:21.310 --> 01:02:23.850 align:middle line:84% AUDIENCE: My budgeting style was the other way around. 01:02:23.850 --> 01:02:26.655 align:middle line:84% What you were talking about makes sense 01:02:26.655 --> 01:02:28.030 align:middle line:84% to go with simple budgeting needs 01:02:28.030 --> 01:02:31.930 align:middle line:90% to go prioritize debugging. 01:02:31.930 --> 01:02:34.433 align:middle line:84% Mine was based on the opportunity left. 01:02:34.433 --> 01:02:35.350 align:middle line:90% HAZHIR RAHMANDAD: Yes. 01:02:35.350 --> 01:02:39.770 align:middle line:84% So basically, you prioritize R&D first and bug fixing second. 01:02:39.770 --> 01:02:40.470 align:middle line:90% OK. 01:02:40.470 --> 01:02:42.850 align:middle line:84% AUDIENCE: And then if the scope gets high enough, 01:02:42.850 --> 01:02:44.210 align:middle line:90% then the opportunity goes down. 01:02:44.210 --> 01:02:45.330 align:middle line:84% And then the majority of budget-- 01:02:45.330 --> 01:02:47.038 align:middle line:84% HAZHIR RAHMANDAD: Yeah, so we haven't yet 01:02:47.038 --> 01:02:50.110 align:middle line:84% looked at that formulation for the budget. 01:02:50.110 --> 01:02:52.805 align:middle line:90% So you have profit margin. 01:02:52.805 --> 01:02:54.430 align:middle line:84% AUDIENCE: And that's a switch because I 01:02:54.430 --> 01:02:56.593 align:middle line:84% wanted to test how it works without this. 01:02:56.593 --> 01:02:57.510 align:middle line:90% HAZHIR RAHMANDAD: Yes. 01:02:57.510 --> 01:03:03.730 align:middle line:84% So you are capturing a total opportunity thingy here, 01:03:03.730 --> 01:03:08.070 align:middle line:84% which is the remainder population. 01:03:08.070 --> 01:03:10.590 align:middle line:84% But the way you are calculating it 01:03:10.590 --> 01:03:14.190 align:middle line:84% is potential adopters plus in-scope adopters divided 01:03:14.190 --> 01:03:17.123 align:middle line:90% by total population. 01:03:17.123 --> 01:03:18.790 align:middle line:84% AUDIENCE: Well, I was thinking that this 01:03:18.790 --> 01:03:20.670 align:middle line:84% is like the firm, the company, is seeing 01:03:20.670 --> 01:03:23.750 align:middle line:84% that this many people who are potentially my customers 01:03:23.750 --> 01:03:26.590 align:middle line:90% haven't bought the product yet. 01:03:26.590 --> 01:03:29.770 align:middle line:84% So I can lure them in with more features. 01:03:29.770 --> 01:03:31.610 align:middle line:84% HAZHIR RAHMANDAD: But in your formulations, 01:03:31.610 --> 01:03:34.710 align:middle line:84% you don't lure them in with more features. 01:03:34.710 --> 01:03:36.117 align:middle line:90% Is that a correct? 01:03:36.117 --> 01:03:38.450 align:middle line:84% AUDIENCE: No, I think the features then feed into scope, 01:03:38.450 --> 01:03:38.650 align:middle line:90% right? 01:03:38.650 --> 01:03:39.270 align:middle line:90% So I want to-- 01:03:39.270 --> 01:03:45.070 align:middle line:84% HAZHIR RAHMANDAD: Yes, you can bring in more in-scope adopters. 01:03:45.070 --> 01:03:50.350 align:middle line:84% But you can't take in-scope adopters 01:03:50.350 --> 01:03:52.890 align:middle line:84% and turn them into adopters with more features. 01:03:52.890 --> 01:03:54.950 align:middle line:84% Whereas, this formulation, it's saying 01:03:54.950 --> 01:03:57.510 align:middle line:84% it is a good idea to invest more if we have 01:03:57.510 --> 01:03:59.510 align:middle line:90% a lot of in-scope adopters. 01:03:59.510 --> 01:04:01.910 align:middle line:90% So it's internally inconsistent. 01:04:01.910 --> 01:04:04.570 align:middle line:90% Is it clear what's happening? 01:04:04.570 --> 01:04:05.190 align:middle line:90% OK. 01:04:05.190 --> 01:04:11.910 align:middle line:84% And then this opportunity thing is not really tied anyway to-- 01:04:11.910 --> 01:04:13.270 align:middle line:90% OK. 01:04:13.270 --> 01:04:17.230 align:middle line:84% So if you have a lot of potential adopters simplified-- 01:04:17.230 --> 01:04:20.290 align:middle line:84% and by the way, also this cannot go above 0.8. 01:04:20.290 --> 01:04:25.270 align:middle line:84% And that 0.8 is a parameter you have made an assumption 01:04:25.270 --> 01:04:26.750 align:middle line:90% about somewhere else. 01:04:26.750 --> 01:04:30.390 align:middle line:84% So it makes this formulation of opportunity 01:04:30.390 --> 01:04:33.510 align:middle line:84% dependent on a parameter that is not part of-- 01:04:33.510 --> 01:04:38.530 align:middle line:84% the scale of this construct is bounded by that other parameter. 01:04:38.530 --> 01:04:41.550 align:middle line:84% So if I change that parameter, the scale of this will change. 01:04:41.550 --> 01:04:43.830 align:middle line:84% Therefore, wherever this is going to go 01:04:43.830 --> 01:04:47.310 align:middle line:84% is going to have a change in scale. 01:04:47.310 --> 01:04:48.330 align:middle line:90% That you don't want. 01:04:48.330 --> 01:04:51.150 align:middle line:84% So this is, again, one of these general rules. 01:04:51.150 --> 01:04:54.630 align:middle line:84% If you have a parameter that is changing 01:04:54.630 --> 01:04:56.910 align:middle line:84% some construct downstream and it's 01:04:56.910 --> 01:05:01.550 align:middle line:84% going to change this construct, then I have to keep in my mind 01:05:01.550 --> 01:05:04.670 align:middle line:84% all the traces where this is going to impact. 01:05:04.670 --> 01:05:07.390 align:middle line:84% Whereas, if I could normalize it right here, 01:05:07.390 --> 01:05:12.450 align:middle line:84% maybe divide it by 0.8, then this is always between 0 and 1. 01:05:12.450 --> 01:05:14.190 align:middle line:84% Then I can use it without worrying 01:05:14.190 --> 01:05:16.270 align:middle line:84% about tracing that impact and making 01:05:16.270 --> 01:05:17.810 align:middle line:90% sure it remains consistent. 01:05:17.810 --> 01:05:19.250 align:middle line:90% And that will always fail. 01:05:19.250 --> 01:05:20.850 align:middle line:90% It's just no doubt about that. 01:05:20.850 --> 01:05:23.350 align:middle line:90% You will forget some pathway. 01:05:23.350 --> 01:05:26.310 align:middle line:84% So as soon as you can, normalize so that you don't 01:05:26.310 --> 01:05:28.190 align:middle line:90% need to keep things in mind. 01:05:28.190 --> 01:05:28.730 align:middle line:90% OK. 01:05:28.730 --> 01:05:31.630 align:middle line:90% 01:05:31.630 --> 01:05:35.370 align:middle line:84% So again, the opportunity idea-- the idea is right. 01:05:35.370 --> 01:05:39.210 align:middle line:84% That is, yes, it becomes smaller as you don't have anybody left. 01:05:39.210 --> 01:05:42.310 align:middle line:84% And that should reduce R&D budget. 01:05:42.310 --> 01:05:45.350 align:middle line:84% But it is not as precise and clean 01:05:45.350 --> 01:05:48.130 align:middle line:84% as a marginal utility type of a calculation. 01:05:48.130 --> 01:05:52.190 align:middle line:84% But it is good to have that idea. 01:05:52.190 --> 01:05:54.550 align:middle line:84% Now, you could use the same idea and say 01:05:54.550 --> 01:05:56.950 align:middle line:90% this is the desired R&D budget. 01:05:56.950 --> 01:06:01.950 align:middle line:84% And then I have the desired bug-fixing budget. 01:06:01.950 --> 01:06:04.270 align:middle line:84% And if you wanted to prioritize that, 01:06:04.270 --> 01:06:07.770 align:middle line:84% get that bug fixing budget it taken care of. 01:06:07.770 --> 01:06:11.470 align:middle line:84% And then whatever remains versus what is the desired 01:06:11.470 --> 01:06:13.210 align:middle line:90% gives you the R&D budget. 01:06:13.210 --> 01:06:16.890 align:middle line:84% If desired, is more than what it remains, you give what remains. 01:06:16.890 --> 01:06:21.590 align:middle line:84% If the desired is less, then you just give what is desired. 01:06:21.590 --> 01:06:22.910 align:middle line:90% Good. 01:06:22.910 --> 01:06:25.930 align:middle line:84% I think we have covered the key formulations here. 01:06:25.930 --> 01:06:29.550 align:middle line:90% So this all makes sense. 01:06:29.550 --> 01:06:30.410 align:middle line:90% Let's see. 01:06:30.410 --> 01:06:32.990 align:middle line:90% 01:06:32.990 --> 01:06:36.770 align:middle line:90% Let's go to Ross's model. 01:06:36.770 --> 01:06:39.670 align:middle line:90% 01:06:39.670 --> 01:06:45.030 align:middle line:90% So we have users. 01:06:45.030 --> 01:06:49.470 align:middle line:84% So here, we have a very similar d fraction of potential users. 01:06:49.470 --> 01:06:52.710 align:middle line:84% This is potential users who are in the scope, so to speak. 01:06:52.710 --> 01:06:54.750 align:middle line:84% This definition of total population 01:06:54.750 --> 01:06:57.010 align:middle line:84% starts with the total people who have a heart disease. 01:06:57.010 --> 01:07:01.470 align:middle line:84% So this is not the total population altogether. 01:07:01.470 --> 01:07:08.510 align:middle line:84% That comes from feature, which has a return on feature thingy. 01:07:08.510 --> 01:07:11.470 align:middle line:90% Let's take a look at-- 01:07:11.470 --> 01:07:13.730 align:middle line:90% so the return on features is-- 01:07:13.730 --> 01:07:17.590 align:middle line:90% 01:07:17.590 --> 01:07:18.550 align:middle line:90% OK. 01:07:18.550 --> 01:07:23.730 align:middle line:84% so this is an interesting formulation. 01:07:23.730 --> 01:07:26.510 align:middle line:84% It's normalized by initial features. 01:07:26.510 --> 01:07:29.290 align:middle line:84% So typically, initial features are smaller than features. 01:07:29.290 --> 01:07:31.650 align:middle line:84% So this number can go easily way above 1. 01:07:31.650 --> 01:07:35.550 align:middle line:84% So we need to keep that in mind, that the scope of use 01:07:35.550 --> 01:07:39.630 align:middle line:90% is now increasing. 01:07:39.630 --> 01:07:41.830 align:middle line:84% And then you have minimum of baseline fraction 01:07:41.830 --> 01:07:46.550 align:middle line:84% of potential users multiplied by scope of use. 01:07:46.550 --> 01:07:49.850 align:middle line:84% And then you have minimum of this and 0, 1. 01:07:49.850 --> 01:07:54.630 align:middle line:84% So Ross, I think I like the first part of your formulation 01:07:54.630 --> 01:07:56.990 align:middle line:84% to think about this scope of use to have decreasing 01:07:56.990 --> 01:07:58.870 align:middle line:90% return with features. 01:07:58.870 --> 01:08:01.770 align:middle line:84% But the way you have formulated it here, 01:08:01.770 --> 01:08:06.110 align:middle line:84% you get a stock in the next step because this scope of use 01:08:06.110 --> 01:08:08.330 align:middle line:90% starts above 1 and goes above 1. 01:08:08.330 --> 01:08:10.730 align:middle line:84% Then you need a parameter that, when multiplied by it, 01:08:10.730 --> 01:08:14.710 align:middle line:84% you want to keep that still below 1. 01:08:14.710 --> 01:08:18.410 align:middle line:84% But then you cannot enforce it because the first part, 01:08:18.410 --> 01:08:22.710 align:middle line:84% which is features to the power of return on features, 01:08:22.710 --> 01:08:23.930 align:middle line:90% is not bounded. 01:08:23.930 --> 01:08:27.430 align:middle line:90% It's not bounded by anything. 01:08:27.430 --> 01:08:29.490 align:middle line:90% And therefore, you get a stock. 01:08:29.490 --> 01:08:33.390 align:middle line:84% You don't know what to do with that. 01:08:33.390 --> 01:08:39.310 align:middle line:84% So what do you do in these situations? 01:08:39.310 --> 01:08:44.590 align:middle line:84% So one way-- there is no easy way because-- 01:08:44.590 --> 01:08:49.450 align:middle line:84% I mean, because the features are growing unlimited, potentially. 01:08:49.450 --> 01:08:51.450 align:middle line:84% And as long as they grow unlimited potentially, 01:08:51.450 --> 01:08:56.990 align:middle line:84% no matter how you normalize it, it can grow above 1. 01:08:56.990 --> 01:09:03.630 align:middle line:84% But I could-- if I was making the feature investment 01:09:03.630 --> 01:09:07.470 align:middle line:84% endogenous to marginal return from features, 01:09:07.470 --> 01:09:10.290 align:middle line:84% then I could, here, rather than initial features, 01:09:10.290 --> 01:09:12.910 align:middle line:84% use the maximum features or something like that 01:09:12.910 --> 01:09:14.790 align:middle line:90% as my normalizing factor. 01:09:14.790 --> 01:09:18.950 align:middle line:84% So I look at the features divided by maximum features 01:09:18.950 --> 01:09:28.390 align:middle line:84% as the factor raised to the power of return on features. 01:09:28.390 --> 01:09:31.310 align:middle line:84% Then as long as I could keep features 01:09:31.310 --> 01:09:34.790 align:middle line:84% under the maximum features, things will be fine. 01:09:34.790 --> 01:09:37.330 align:middle line:84% So that's an easier starting point. 01:09:37.330 --> 01:09:38.870 align:middle line:84% And then I don't also need to worry 01:09:38.870 --> 01:09:42.569 align:middle line:84% about this baseline fraction of potential users. 01:09:42.569 --> 01:09:47.750 align:middle line:84% So I don't introduce additional parameters up there. 01:09:47.750 --> 01:09:52.090 align:middle line:84% Now, keeping the features under the maximum is still tricky. 01:09:52.090 --> 01:09:53.310 align:middle line:90% How do you do that? 01:09:53.310 --> 01:09:59.310 align:middle line:84% If you do the marginal utility calculations, 01:09:59.310 --> 01:10:03.470 align:middle line:90% you probably will get there. 01:10:03.470 --> 01:10:05.610 align:middle line:84% Or there is-- you could get there. 01:10:05.610 --> 01:10:07.870 align:middle line:90% It's not guaranteed either. 01:10:07.870 --> 01:10:10.170 align:middle line:84% You could do something that is more ad hoc, 01:10:10.170 --> 01:10:14.390 align:middle line:84% like putting a bound on feature productivity 01:10:14.390 --> 01:10:19.510 align:middle line:84% that goes to 0 once you get close to the maximum features. 01:10:19.510 --> 01:10:23.710 align:middle line:84% So you could make that, no matter how much you're 01:10:23.710 --> 01:10:27.670 align:middle line:84% investing, if the gap between maximum features is 0, 01:10:27.670 --> 01:10:29.990 align:middle line:84% you basically shut off the productivity of feature 01:10:29.990 --> 01:10:31.110 align:middle line:90% development. 01:10:31.110 --> 01:10:33.890 align:middle line:84% That's an ad hoc solution that I don't like. 01:10:33.890 --> 01:10:35.730 align:middle line:84% But if I don't have any better solutions, 01:10:35.730 --> 01:10:38.210 align:middle line:84% I'm OK to use something like that. 01:10:38.210 --> 01:10:38.710 align:middle line:90% OK. 01:10:38.710 --> 01:10:42.510 align:middle line:84% Now, that creates a fraction that leads 01:10:42.510 --> 01:10:44.330 align:middle line:90% to indicated potential users. 01:10:44.330 --> 01:10:47.750 align:middle line:84% There is a time to adopt or to move. 01:10:47.750 --> 01:10:50.110 align:middle line:90% This all makes sense. 01:10:50.110 --> 01:10:54.470 align:middle line:84% Then you have potential users with a fraction 01:10:54.470 --> 01:10:57.710 align:middle line:84% of these potential users based on perceived utility. 01:10:57.710 --> 01:10:59.870 align:middle line:90% You get the adoption rate. 01:10:59.870 --> 01:11:02.570 align:middle line:84% And let's take a look at what is happening here. 01:11:02.570 --> 01:11:06.570 align:middle line:84% This is indicated users minus users divided by time to adopt. 01:11:06.570 --> 01:11:07.930 align:middle line:90% So this is a two-way flow. 01:11:07.930 --> 01:11:10.590 align:middle line:84% Now, in this case, theoretically, 01:11:10.590 --> 01:11:14.830 align:middle line:84% if the indicated users go down, you could have negative numbers 01:11:14.830 --> 01:11:20.590 align:middle line:84% and, therefore, push people from users go back into potential-- 01:11:20.590 --> 01:11:22.130 align:middle line:90% to indicated users. 01:11:22.130 --> 01:11:23.910 align:middle line:84% That's the right formulation here. 01:11:23.910 --> 01:11:26.590 align:middle line:84% This is actually what you want to have. 01:11:26.590 --> 01:11:30.870 align:middle line:84% That allows reductions in indicated users 01:11:30.870 --> 01:11:33.550 align:middle line:90% if perceived utility goes down. 01:11:33.550 --> 01:11:37.670 align:middle line:84% And perceived utility is coming from known side effects. 01:11:37.670 --> 01:11:43.110 align:middle line:84% Now, the way Ross has formulated known side effects is based 01:11:43.110 --> 01:11:47.930 align:middle line:84% on some sensitivity parameter that you just multiply-- oh, OK, 01:11:47.930 --> 01:11:50.710 align:middle line:84% so this is a more complex formulation-- 01:11:50.710 --> 01:11:53.670 align:middle line:84% 2 divided by 1 plus known side effects 01:11:53.670 --> 01:11:59.350 align:middle line:84% divided by initial side effects to the power of this thingy. 01:11:59.350 --> 01:12:08.350 align:middle line:84% So if known, side effects are the same as initial, 01:12:08.350 --> 01:12:13.150 align:middle line:90% you have perceived utility of 1. 01:12:13.150 --> 01:12:15.710 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:12:15.710 --> 01:12:19.790 align:middle line:90% 01:12:19.790 --> 01:12:20.750 align:middle line:90% HAZHIR RAHMANDAD: Yes. 01:12:20.750 --> 01:12:23.253 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:12:23.253 --> 01:12:24.170 align:middle line:90% HAZHIR RAHMANDAD: Yes. 01:12:24.170 --> 01:12:30.870 align:middle line:84% And if you have a hell of side effects, then it goes to 0. 01:12:30.870 --> 01:12:36.790 align:middle line:84% So this is kind of an S-shaped curve that goes from 0, 01:12:36.790 --> 01:12:45.550 align:middle line:84% is changing slope second derivative at 1, 01:12:45.550 --> 01:12:51.110 align:middle line:84% and then saturates at 2 for the utility. 01:12:51.110 --> 01:12:55.710 align:middle line:84% It's a little bit ad hoc-y for me. 01:12:55.710 --> 01:12:59.630 align:middle line:84% It's not bad because it is capturing 01:12:59.630 --> 01:13:02.890 align:middle line:90% the trend of the utility. 01:13:02.890 --> 01:13:12.250 align:middle line:84% But it leaves too many things on the initial side effects. 01:13:12.250 --> 01:13:17.510 align:middle line:90% 01:13:17.510 --> 01:13:18.530 align:middle line:90% It's an interesting one. 01:13:18.530 --> 01:13:19.370 align:middle line:90% OK, so-- 01:13:19.370 --> 01:13:22.270 align:middle line:90% 01:13:22.270 --> 01:13:24.232 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:13:24.232 --> 01:13:25.690 align:middle line:84% HAZHIR RAHMANDAD: Can you speak up? 01:13:25.690 --> 01:13:26.830 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:13:26.830 --> 01:13:30.190 align:middle line:84% HAZHIR RAHMANDAD: So 2 is basically 01:13:30.190 --> 01:13:32.770 align:middle line:84% what Ross decided to normalize around. 01:13:32.770 --> 01:13:37.230 align:middle line:84% And then when it goes into the fraction of users, 01:13:37.230 --> 01:13:41.110 align:middle line:84% it should not go to 2 because, otherwise, the fraction of users 01:13:41.110 --> 01:13:46.310 align:middle line:90% becomes twice the current. 01:13:46.310 --> 01:13:46.810 align:middle line:90% OK. 01:13:46.810 --> 01:13:51.710 align:middle line:84% So this is actually an actual problem here, Ross. 01:13:51.710 --> 01:13:54.710 align:middle line:90% 01:13:54.710 --> 01:13:56.550 align:middle line:84% First of all, fraction of users ultimately 01:13:56.550 --> 01:13:57.930 align:middle line:90% should be between 0 and 1. 01:13:57.930 --> 01:14:00.150 align:middle line:84% So if you use perceived utility and perceived utility 01:14:00.150 --> 01:14:02.890 align:middle line:84% can go to 2, then fraction of users becomes 2. 01:14:02.890 --> 01:14:06.430 align:middle line:84% Then it doesn't work, if it is fraction. 01:14:06.430 --> 01:14:09.070 align:middle line:84% AUDIENCE: I think I was trying to do the baseline fraction 01:14:09.070 --> 01:14:13.990 align:middle line:84% thing again, but probably [INAUDIBLE]. 01:14:13.990 --> 01:14:17.030 align:middle line:84% HAZHIR RAHMANDAD: You were trying to do what? 01:14:17.030 --> 01:14:20.470 align:middle line:84% AUDIENCE: Like the one that I did in the [INAUDIBLE], 01:14:20.470 --> 01:14:22.692 align:middle line:90% multiplying by the [INAUDIBLE]. 01:14:22.692 --> 01:14:23.650 align:middle line:90% HAZHIR RAHMANDAD: Yeah. 01:14:23.650 --> 01:14:27.670 align:middle line:84% You wanted to multiply it by something to scale it. 01:14:27.670 --> 01:14:31.830 align:middle line:84% You could scale it by that maximum, which, in this case, 01:14:31.830 --> 01:14:33.670 align:middle line:90% was 2 if I-- 01:14:33.670 --> 01:14:39.010 align:middle line:84% yes, maximum is always 2, which comes from known. 01:14:39.010 --> 01:14:42.225 align:middle line:90% 01:14:42.225 --> 01:14:44.050 align:middle line:90% And that could work. 01:14:44.050 --> 01:14:47.590 align:middle line:84% But the more structurally problematic issue 01:14:47.590 --> 01:14:49.350 align:middle line:90% is this indicated users. 01:14:49.350 --> 01:14:52.630 align:middle line:84% It shouldn't be just multiplying fraction of users 01:14:52.630 --> 01:14:54.390 align:middle line:90% by potential users. 01:14:54.390 --> 01:15:02.190 align:middle line:84% It should be multiplying it by potential users plus users. 01:15:02.190 --> 01:15:04.910 align:middle line:90% Am I right? 01:15:04.910 --> 01:15:06.390 align:middle line:90% AUDIENCE: Yeah, yeah. 01:15:06.390 --> 01:15:09.715 align:middle line:90% What changes [INAUDIBLE]? 01:15:09.715 --> 01:15:10.590 align:middle line:90% HAZHIR RAHMANDAD: Hm? 01:15:10.590 --> 01:15:12.048 align:middle line:84% AUDIENCE: What changes [INAUDIBLE]? 01:15:12.048 --> 01:15:15.110 align:middle line:90% 01:15:15.110 --> 01:15:17.470 align:middle line:84% HAZHIR RAHMANDAD: Yeah, that would work. 01:15:17.470 --> 01:15:20.370 align:middle line:84% AUDIENCE: And I am trying to figure out. 01:15:20.370 --> 01:15:23.710 align:middle line:84% So when somebody adopts into a user, 01:15:23.710 --> 01:15:27.310 align:middle line:84% because there's also an indicated potential user, 01:15:27.310 --> 01:15:30.550 align:middle line:84% that adoption would drive someone from total population 01:15:30.550 --> 01:15:33.650 align:middle line:84% to come into the potential users stock, but not through-- 01:15:33.650 --> 01:15:34.858 align:middle line:90% HAZHIR RAHMANDAD: Wait, wait. 01:15:34.858 --> 01:15:36.830 align:middle line:84% From total population, nobody should come here 01:15:36.830 --> 01:15:38.930 align:middle line:90% just because somebody left. 01:15:38.930 --> 01:15:39.530 align:middle line:90% Let me see. 01:15:39.530 --> 01:15:42.570 align:middle line:84% What's the becoming relevant formulation here? 01:15:42.570 --> 01:15:44.390 align:middle line:84% AUDIENCE: I guess indicated potential users 01:15:44.390 --> 01:15:46.270 align:middle line:90% made me think that. 01:15:46.270 --> 01:15:50.070 align:middle line:84% HAZHIR RAHMANDAD: Oh, this is also problematic, yes. 01:15:50.070 --> 01:15:54.790 align:middle line:84% So you don't want to look at the indicated potential users 01:15:54.790 --> 01:15:58.590 align:middle line:84% minus potential users for becoming relevant. 01:15:58.590 --> 01:16:02.950 align:middle line:84% You want to look at indicated potential users minus the sum 01:16:02.950 --> 01:16:06.390 align:middle line:90% of potential users and users. 01:16:06.390 --> 01:16:10.110 align:middle line:84% So that's the total that you're comparing with, 01:16:10.110 --> 01:16:11.490 align:middle line:90% and then closing that gap. 01:16:11.490 --> 01:16:13.270 align:middle line:90% Is that clear, Ross? 01:16:13.270 --> 01:16:14.670 align:middle line:90% AUDIENCE: Yeah, yeah, yeah. 01:16:14.670 --> 01:16:15.545 align:middle line:90% HAZHIR RAHMANDAD: OK. 01:16:15.545 --> 01:16:19.430 align:middle line:84% And similar here, basically here, your indicated users 01:16:19.430 --> 01:16:22.350 align:middle line:84% should be the sum of the potential users and users 01:16:22.350 --> 01:16:24.830 align:middle line:90% multiplied by this fraction. 01:16:24.830 --> 01:16:29.710 align:middle line:84% That becomes the total that you would expect 01:16:29.710 --> 01:16:31.570 align:middle line:90% to be in the users category. 01:16:31.570 --> 01:16:34.990 align:middle line:84% Now, you compare that with the current users. 01:16:34.990 --> 01:16:38.830 align:middle line:90% And you can close that gap. 01:16:38.830 --> 01:16:42.470 align:middle line:84% So it would be indicated users minus users 01:16:42.470 --> 01:16:44.450 align:middle line:90% divided by time to adopt. 01:16:44.450 --> 01:16:47.990 align:middle line:84% But indicated users is the sum of users and potential users 01:16:47.990 --> 01:16:49.910 align:middle line:90% multiplied by this fraction. 01:16:49.910 --> 01:16:54.190 align:middle line:84% And then the time to adopt, you're 01:16:54.190 --> 01:16:56.210 align:middle line:90% looking at endogenizing it. 01:16:56.210 --> 01:16:59.170 align:middle line:84% But you're endogenizing it using the current users. 01:16:59.170 --> 01:17:00.990 align:middle line:90% And that's OK. 01:17:00.990 --> 01:17:03.530 align:middle line:84% It's people who have it have it word of mouth. 01:17:03.530 --> 01:17:06.830 align:middle line:90% So this is the idea. 01:17:06.830 --> 01:17:10.870 align:middle line:84% Now, this formulation here is baseline time 01:17:10.870 --> 01:17:18.050 align:middle line:84% to adopt divided by fraction of users in population plus 1. 01:17:18.050 --> 01:17:21.890 align:middle line:84% So as that fraction goes down, you get to the baseline. 01:17:21.890 --> 01:17:28.290 align:middle line:84% As that fraction goes to 0, you go to baseline. 01:17:28.290 --> 01:17:33.630 align:middle line:84% If that fraction of users in population is 1-- 01:17:33.630 --> 01:17:38.270 align:middle line:84% that's the maximum-- then time to adopt 01:17:38.270 --> 01:17:41.070 align:middle line:90% goes down to half that value. 01:17:41.070 --> 01:17:46.830 align:middle line:84% So this is not a very effective way 01:17:46.830 --> 01:17:52.310 align:middle line:84% of scaling your time constant because, basically, it 01:17:52.310 --> 01:18:01.150 align:middle line:84% limits it between baseline and baseline divided by 2. 01:18:01.150 --> 01:18:03.990 align:middle line:84% And even 2 is very unlikely because this fraction 01:18:03.990 --> 01:18:07.510 align:middle line:84% of users in population is a number that 01:18:07.510 --> 01:18:10.410 align:middle line:90% usually doesn't go that high. 01:18:10.410 --> 01:18:14.830 align:middle line:84% It's bounded by how much potential users can 01:18:14.830 --> 01:18:16.750 align:middle line:90% increase the scope and so on. 01:18:16.750 --> 01:18:20.350 align:middle line:84% So I would go with something more flexible that 01:18:20.350 --> 01:18:25.790 align:middle line:84% has another parameter for maybe some kind of a minimum time 01:18:25.790 --> 01:18:29.150 align:middle line:84% and maximum time, and then scale it using this fraction. 01:18:29.150 --> 01:18:30.710 align:middle line:84% And the fraction could be fraction 01:18:30.710 --> 01:18:32.650 align:middle line:90% of users in the population. 01:18:32.650 --> 01:18:38.030 align:middle line:90% That's fine, but yeah. 01:18:38.030 --> 01:18:42.450 align:middle line:84% So these are the key questions here. 01:18:42.450 --> 01:18:49.810 align:middle line:84% Then feature development here, you're using a fixed rate. 01:18:49.810 --> 01:18:53.430 align:middle line:84% And why do you have this baseline at feature rate divided 01:18:53.430 --> 01:18:55.290 align:middle line:90% by time to add features? 01:18:55.290 --> 01:18:57.910 align:middle line:90% 01:18:57.910 --> 01:19:02.943 align:middle line:84% AUDIENCE: I did not go as far [INAUDIBLE]? 01:19:02.943 --> 01:19:05.610 align:middle line:84% HAZHIR RAHMANDAD: Yeah, I think you can just have one parameter, 01:19:05.610 --> 01:19:06.527 align:middle line:90% if you want to fix it. 01:19:06.527 --> 01:19:09.390 align:middle line:84% But then if you fix it, you will not 01:19:09.390 --> 01:19:14.110 align:middle line:84% be capturing the marginal utility and the decision 01:19:14.110 --> 01:19:18.390 align:middle line:84% to add features endogenous to the return from features and so 01:19:18.390 --> 01:19:19.830 align:middle line:90% on. 01:19:19.830 --> 01:19:27.770 align:middle line:84% And then the bug discovery and fixing right now is a fixed one. 01:19:27.770 --> 01:19:34.870 align:middle line:84% Again, this is skipping R&D structure and formulations 01:19:34.870 --> 01:19:35.370 align:middle line:90% there. 01:19:35.370 --> 01:19:37.310 align:middle line:90% OK, good. 01:19:37.310 --> 01:19:38.990 align:middle line:90% Let's take a break. 01:19:38.990 --> 01:19:41.630 align:middle line:90% I think it's a good time. 01:19:41.630 --> 01:19:44.150 align:middle line:84% But before that, any questions on this last model 01:19:44.150 --> 01:19:46.310 align:middle line:90% that we went through? 01:19:46.310 --> 01:19:48.150 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:19:48.150 --> 01:19:49.790 align:middle line:90% 01:19:49.790 --> 01:19:51.708 align:middle line:84% HAZHIR RAHMANDAD: Yeah, short question. 01:19:51.708 --> 01:19:52.750 align:middle line:90% HAZHIR RAHMANDAD: Mm-hmm. 01:19:52.750 --> 01:19:56.110 align:middle line:84% AUDIENCE: If you want to [INAUDIBLE] stop-- 01:19:56.110 --> 01:20:01.230 align:middle line:84% so for the features we had a maximum feature level 01:20:01.230 --> 01:20:06.070 align:middle line:84% that we preordained and then just close 01:20:06.070 --> 01:20:08.250 align:middle line:90% the gap to that level. 01:20:08.250 --> 01:20:10.990 align:middle line:90% 01:20:10.990 --> 01:20:12.590 align:middle line:90% Is there a problem? 01:20:12.590 --> 01:20:17.350 align:middle line:84% What problems do we have [INAUDIBLE] if the marginal 01:20:17.350 --> 01:20:20.390 align:middle line:90% [INAUDIBLE]? 01:20:20.390 --> 01:20:23.770 align:middle line:84% HAZHIR RAHMANDAD: You don't get anything-- 01:20:23.770 --> 01:20:25.270 align:middle line:90% any big problem. 01:20:25.270 --> 01:20:27.990 align:middle line:84% The only problem we would have is that your marginal 01:20:27.990 --> 01:20:29.790 align:middle line:90% becomes incorrect. 01:20:29.790 --> 01:20:33.950 align:middle line:84% Basically, because you have one pathway that is using marginals 01:20:33.950 --> 01:20:38.550 align:middle line:84% to decide on adding features, and then you 01:20:38.550 --> 01:20:44.710 align:middle line:84% have this other kind of a very strong limiting factor that 01:20:44.710 --> 01:20:46.430 align:middle line:84% ignores the marginals and says, OK, I'm 01:20:46.430 --> 01:20:48.650 align:middle line:84% going to shut off feature development, 01:20:48.650 --> 01:20:51.650 align:middle line:84% even if it is why it is OK marginal-wise. 01:20:51.650 --> 01:20:53.430 align:middle line:84% AUDIENCE: But if you don't build out 01:20:53.430 --> 01:20:55.390 align:middle line:84% the decision making structure, then you 01:20:55.390 --> 01:20:57.910 align:middle line:84% can still capture that marginal effect. 01:20:57.910 --> 01:20:59.570 align:middle line:84% If you don't have the profitability 01:20:59.570 --> 01:21:02.230 align:middle line:84% of the [INAUDIBLE], then you still 01:21:02.230 --> 01:21:05.150 align:middle line:84% get that goodness of marginal return 01:21:05.150 --> 01:21:06.730 align:middle line:90% on investment [INAUDIBLE]. 01:21:06.730 --> 01:21:09.203 align:middle line:90% 01:21:09.203 --> 01:21:10.870 align:middle line:84% HAZHIR RAHMANDAD: I mean, that cap alone 01:21:10.870 --> 01:21:13.630 align:middle line:84% is not a great formula for marginal 01:21:13.630 --> 01:21:16.790 align:middle line:84% because it is disconnected from what 01:21:16.790 --> 01:21:21.030 align:middle line:84% is happening with your fraction of potential users. 01:21:21.030 --> 01:21:23.590 align:middle line:84% This return on feature parameter is the key. 01:21:23.590 --> 01:21:26.750 align:middle line:84% This is really what determines your marginal 01:21:26.750 --> 01:21:31.270 align:middle line:84% or, by the first order determinant of the marginal, 01:21:31.270 --> 01:21:33.590 align:middle line:84% is your return on features parameter. 01:21:33.590 --> 01:21:38.670 align:middle line:84% So the derivative of fraction of potential users with respect 01:21:38.670 --> 01:21:43.130 align:middle line:84% to features is dependent on this parameter. 01:21:43.130 --> 01:21:45.690 align:middle line:84% So in this formulation for add features, 01:21:45.690 --> 01:21:47.390 align:middle line:90% I ignore that parameter. 01:21:47.390 --> 01:21:49.990 align:middle line:84% I am completely ignoring the actual process 01:21:49.990 --> 01:21:51.970 align:middle line:90% by which marginal is generated. 01:21:51.970 --> 01:21:53.230 align:middle line:90% So you don't want that. 01:21:53.230 --> 01:21:56.230 align:middle line:84% And that's why I was a little bit hesitant of adding 01:21:56.230 --> 01:22:00.568 align:middle line:90% that very sharp thingy. 01:22:00.568 --> 01:22:01.610 align:middle line:90% AUDIENCE: That's helpful. 01:22:01.610 --> 01:22:03.670 align:middle line:84% HAZHIR RAHMANDAD: Yeah, OK, cool. 01:22:03.670 --> 01:22:05.210 align:middle line:84% Let's take 10 minutes of a break. 01:22:05.210 --> 01:22:08.130 align:middle line:90% 01:22:08.130 --> 01:22:08.630 align:middle line:90% OK. 01:22:08.630 --> 01:22:12.550 align:middle line:90% We went through Ross's model. 01:22:12.550 --> 01:22:17.410 align:middle line:84% Let's see which models did we miss. 01:22:17.410 --> 01:22:19.250 align:middle line:90% So this is Ali's. 01:22:19.250 --> 01:22:20.210 align:middle line:90% Let's take a look. 01:22:20.210 --> 01:22:21.730 align:middle line:84% Did we go through your model, Ali? 01:22:21.730 --> 01:22:22.438 align:middle line:90% AUDIENCE: Uh, no. 01:22:22.438 --> 01:22:24.950 align:middle line:90% HAZHIR RAHMANDAD: No, OK. 01:22:24.950 --> 01:22:27.010 align:middle line:84% Oh no, we did go through the stocks. 01:22:27.010 --> 01:22:29.010 align:middle line:84% Yeah, but we didn't go through the formulation. 01:22:29.010 --> 01:22:30.990 align:middle line:84% AUDIENCE: So I haven't collected [INAUDIBLE]. 01:22:30.990 --> 01:22:36.378 align:middle line:90% 01:22:36.378 --> 01:22:37.670 align:middle line:90% HAZHIR RAHMANDAD: I see, I see. 01:22:37.670 --> 01:22:39.250 align:middle line:90% AUDIENCE: [INAUDIBLE]. 01:22:39.250 --> 01:22:41.830 align:middle line:90% 01:22:41.830 --> 01:22:44.590 align:middle line:84% The reason that I used this formula [INAUDIBLE] 01:22:44.590 --> 01:22:46.302 align:middle line:90% S-shaped logistic. 01:22:46.302 --> 01:22:49.010 align:middle line:84% HAZHIR RAHMANDAD: So you want to have S-shaped logistic for what? 01:22:49.010 --> 01:22:50.130 align:middle line:90% AUDIENCE: For the scope. 01:22:50.130 --> 01:22:51.005 align:middle line:90% HAZHIR RAHMANDAD: OK. 01:22:51.005 --> 01:22:53.530 align:middle line:84% So the scope right now, you have a maximum scope of use. 01:22:53.530 --> 01:22:56.630 align:middle line:84% And then you are moving towards that maximum scope of use. 01:22:56.630 --> 01:23:00.743 align:middle line:84% And it's basically looking at the gap-- 01:23:00.743 --> 01:23:02.410 align:middle line:84% AUDIENCE: [INAUDIBLE] carrying capacity. 01:23:02.410 --> 01:23:03.368 align:middle line:90% HAZHIR RAHMANDAD: Yeah. 01:23:03.368 --> 01:23:04.930 align:middle line:90% And so you slow down. 01:23:04.930 --> 01:23:08.990 align:middle line:84% So it has a little bit of that reduced investment in the scope 01:23:08.990 --> 01:23:13.190 align:middle line:90% over time built into it. 01:23:13.190 --> 01:23:14.090 align:middle line:90% And then-- 01:23:14.090 --> 01:23:21.390 align:middle line:90% 01:23:21.390 --> 01:23:23.710 align:middle line:90% AUDIENCE: [INAUDIBLE] adopters. 01:23:23.710 --> 01:23:24.670 align:middle line:90% HAZHIR RAHMANDAD: Yeah. 01:23:24.670 --> 01:23:28.430 align:middle line:84% So one thing that I just remembered we haven't discussed 01:23:28.430 --> 01:23:33.910 align:middle line:84% is we talked about this marginal utility or marginal benefit 01:23:33.910 --> 01:23:37.110 align:middle line:90% of adding more features. 01:23:37.110 --> 01:23:40.590 align:middle line:84% So what that really does, in practice, 01:23:40.590 --> 01:23:44.210 align:middle line:84% is, if it is above a threshold-- that is, if it is profitable, 01:23:44.210 --> 01:23:46.370 align:middle line:84% it gives you enough return on investment-- 01:23:46.370 --> 01:23:50.070 align:middle line:84% you want to invest in feature development. 01:23:50.070 --> 01:23:52.550 align:middle line:84% And if it is not, then you don't want 01:23:52.550 --> 01:23:54.550 align:middle line:84% to invest in feature development. 01:23:54.550 --> 01:23:59.310 align:middle line:84% So if you have a maximum capacity for investment, 01:23:59.310 --> 01:24:03.210 align:middle line:84% that maximum capacity-- if you don't make that endogenous, 01:24:03.210 --> 01:24:05.590 align:middle line:84% that maximum capacity is operational, as long 01:24:05.590 --> 01:24:07.510 align:middle line:84% as you are above this threshold, and then it 01:24:07.510 --> 01:24:11.190 align:middle line:84% shuts off or becomes much smaller 01:24:11.190 --> 01:24:16.630 align:middle line:84% when it's not worthwhile building new features. 01:24:16.630 --> 01:24:24.870 align:middle line:84% And that formulation actually doesn't use the marginal utility 01:24:24.870 --> 01:24:29.650 align:middle line:84% to decide on the flow, on the level of product development. 01:24:29.650 --> 01:24:31.130 align:middle line:90% It just is a switch. 01:24:31.130 --> 01:24:32.490 align:middle line:90% It's 0 or 1. 01:24:32.490 --> 01:24:35.250 align:middle line:84% If it is above threshold, you're investing. 01:24:35.250 --> 01:24:38.830 align:middle line:84% If it is below threshold, you are not investing. 01:24:38.830 --> 01:24:42.010 align:middle line:84% And that might be an OK first order approximation. 01:24:42.010 --> 01:24:44.450 align:middle line:84% I'm just highlighting the way I was talking about it 01:24:44.450 --> 01:24:47.870 align:middle line:84% and because I haven't built that part of the model-- 01:24:47.870 --> 01:24:49.690 align:middle line:84% I haven't built this model recently. 01:24:49.690 --> 01:24:51.510 align:middle line:84% I did a version of it many years ago. 01:24:51.510 --> 01:24:54.510 align:middle line:90% 01:24:54.510 --> 01:24:58.170 align:middle line:84% I talk as if it is continuously changing the flow, 01:24:58.170 --> 01:25:01.790 align:middle line:84% but it is not, in that interpretation. 01:25:01.790 --> 01:25:04.830 align:middle line:84% Now, there is a more subtle, complex interpretation 01:25:04.830 --> 01:25:11.350 align:middle line:84% that says the faster you develop features, the more inefficient 01:25:11.350 --> 01:25:13.710 align:middle line:90% is development process. 01:25:13.710 --> 01:25:16.990 align:middle line:84% It's just called time compression. 01:25:16.990 --> 01:25:19.550 align:middle line:90% Diseconomies, this is kind of-- 01:25:19.550 --> 01:25:24.850 align:middle line:84% academics want to generate terms that are fancy-sounding. 01:25:24.850 --> 01:25:27.670 align:middle line:84% So time compression diseconomy is basically telling you 01:25:27.670 --> 01:25:30.550 align:middle line:84% that, if you pack in too many features together 01:25:30.550 --> 01:25:33.150 align:middle line:84% into next product development project, 01:25:33.150 --> 01:25:38.310 align:middle line:84% you have to spend a lot more in getting those features in place. 01:25:38.310 --> 01:25:42.750 align:middle line:84% If you account for that, then this marginal 01:25:42.750 --> 01:25:46.550 align:middle line:84% becomes important because you would pack enough features 01:25:46.550 --> 01:25:49.490 align:middle line:84% into your next development project 01:25:49.490 --> 01:25:53.030 align:middle line:84% so that the time compression diseconomy's cost of developing 01:25:53.030 --> 01:25:56.710 align:middle line:84% those features balances against the benefits that 01:25:56.710 --> 01:25:58.890 align:middle line:90% come from having more features. 01:25:58.890 --> 01:26:02.750 align:middle line:84% And that is coming from your marginal value of new features. 01:26:02.750 --> 01:26:05.270 align:middle line:84% So there is a more complex version 01:26:05.270 --> 01:26:08.870 align:middle line:84% of the world in which the flow, at any point in time, 01:26:08.870 --> 01:26:15.610 align:middle line:84% is dependent on that marginal revenue or profit per feature. 01:26:15.610 --> 01:26:18.170 align:middle line:84% But in that world, to get to that world, 01:26:18.170 --> 01:26:19.290 align:middle line:90% you need another concept. 01:26:19.290 --> 01:26:22.710 align:middle line:84% You need something about these time compression diseconomies. 01:26:22.710 --> 01:26:26.190 align:middle line:84% Absent that, the decision on how many new features I want 01:26:26.190 --> 01:26:30.630 align:middle line:84% to develop-- the decision is infinite until the point-- 01:26:30.630 --> 01:26:35.150 align:middle line:84% infinite flow-- until the point that I get to marginal 01:26:35.150 --> 01:26:37.810 align:middle line:84% goes to negative, below the threshold. 01:26:37.810 --> 01:26:39.070 align:middle line:90% And then I shut off. 01:26:39.070 --> 01:26:42.470 align:middle line:84% That is, go with the maximum features you can pack, 01:26:42.470 --> 01:26:45.670 align:middle line:90% and then go to 0. 01:26:45.670 --> 01:26:48.390 align:middle line:84% That, of course doesn't happen in the real world because 01:26:48.390 --> 01:26:50.910 align:middle line:84% of those time compression diseconomies 01:26:50.910 --> 01:26:52.728 align:middle line:84% and the underlying mechanisms for those. 01:26:52.728 --> 01:26:54.270 align:middle line:84% You who don't know about the customer 01:26:54.270 --> 01:26:55.990 align:middle line:84% taste until you do some other things. 01:26:55.990 --> 01:26:57.870 align:middle line:84% And you don't have the technology 01:26:57.870 --> 01:26:59.730 align:middle line:90% until you do some other things. 01:26:59.730 --> 01:27:07.030 align:middle line:84% So a lot of features are not in the feasible frontier 01:27:07.030 --> 01:27:08.770 align:middle line:90% that is accessible to you. 01:27:08.770 --> 01:27:10.830 align:middle line:84% You have to develop other things before you 01:27:10.830 --> 01:27:14.070 align:middle line:84% get to the technologies that allow you to develop 01:27:14.070 --> 01:27:17.510 align:middle line:90% new features and so on. 01:27:17.510 --> 01:27:23.230 align:middle line:84% Basically, those technology frontiers are typically-- 01:27:23.230 --> 01:27:25.030 align:middle line:84% knowledge and technology frontiers 01:27:25.030 --> 01:27:28.330 align:middle line:84% are what sets the pace of product development. 01:27:28.330 --> 01:27:30.710 align:middle line:84% That is, you're trying to develop 01:27:30.710 --> 01:27:33.550 align:middle line:84% whatever is feasible in that space 01:27:33.550 --> 01:27:35.890 align:middle line:90% that you think makes sense. 01:27:35.890 --> 01:27:39.130 align:middle line:84% And then that increases, that expands the frontier. 01:27:39.130 --> 01:27:40.790 align:middle line:90% Now, new frontiers show up. 01:27:40.790 --> 01:27:42.670 align:middle line:90% And you develop that. 01:27:42.670 --> 01:27:45.310 align:middle line:84% Approximating that with a fixed product development 01:27:45.310 --> 01:27:48.030 align:middle line:84% rate or a max product development rate is OK. 01:27:48.030 --> 01:27:50.670 align:middle line:84% So for this project, that's perfectly fine. 01:27:50.670 --> 01:27:54.870 align:middle line:84% You just need a process to shift towards a point 01:27:54.870 --> 01:27:58.310 align:middle line:84% that, OK, it's not worth it anymore to develop more features 01:27:58.310 --> 01:28:01.350 align:middle line:90% or expand the scope. 01:28:01.350 --> 01:28:06.610 align:middle line:84% So that's just something to keep in mind. 01:28:06.610 --> 01:28:07.110 align:middle line:90% OK. 01:28:07.110 --> 01:28:10.050 align:middle line:84% And this one is a fractional investment. 01:28:10.050 --> 01:28:14.810 align:middle line:84% So it reduces investment as we go forward. 01:28:14.810 --> 01:28:20.190 align:middle line:84% And then you get the maximum adopters from here. 01:28:20.190 --> 01:28:24.030 align:middle line:84% You look at the gap between the current adopters and maximum. 01:28:24.030 --> 01:28:30.270 align:middle line:84% That creates an adoption process with the time to adopt 01:28:30.270 --> 01:28:31.730 align:middle line:90% and some perceived utility. 01:28:31.730 --> 01:28:35.030 align:middle line:84% So how do the three come together? 01:28:35.030 --> 01:28:36.630 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:28:36.630 --> 01:28:42.842 align:middle line:90% 01:28:42.842 --> 01:28:44.550 align:middle line:84% HAZHIR RAHMANDAD: So in this formulation, 01:28:44.550 --> 01:28:47.190 align:middle line:84% you say adopters gap, which is those 01:28:47.190 --> 01:28:53.070 align:middle line:84% who are in a scope minus those who are already adopted, 01:28:53.070 --> 01:28:56.470 align:middle line:84% you multiply that by a perceived utility. 01:28:56.470 --> 01:28:59.990 align:middle line:84% So what fraction of those are interested 01:28:59.990 --> 01:29:01.730 align:middle line:84% is coming from your perceived utility. 01:29:01.730 --> 01:29:05.430 align:middle line:84% So in this formulation, if the perceived utility goes to 0, 01:29:05.430 --> 01:29:07.830 align:middle line:84% you would just shut off the flow. 01:29:07.830 --> 01:29:09.910 align:middle line:84% But you would not have a way for people 01:29:09.910 --> 01:29:14.030 align:middle line:84% to come back because they are seeing a lot of problems 01:29:14.030 --> 01:29:15.270 align:middle line:90% with the product. 01:29:15.270 --> 01:29:18.550 align:middle line:84% And this is also partly the challenge 01:29:18.550 --> 01:29:21.510 align:middle line:84% that we talked about with combining those two 01:29:21.510 --> 01:29:22.730 align:middle line:90% effects together. 01:29:22.730 --> 01:29:30.790 align:middle line:90% 01:29:30.790 --> 01:29:34.830 align:middle line:84% AUDIENCE: It seems fair for a pacemaker [INAUDIBLE]. 01:29:34.830 --> 01:29:37.530 align:middle line:84% HAZHIR RAHMANDAD: So yes, that's right. 01:29:37.530 --> 01:29:42.150 align:middle line:84% So once you have it in this other-- 01:29:42.150 --> 01:29:46.470 align:middle line:84% and there is a question of who is an adopter in this case. 01:29:46.470 --> 01:29:52.030 align:middle line:84% These are people who have a pacemaker or are kind of-- 01:29:52.030 --> 01:29:53.810 align:middle line:84% because pacemaker don't last forever. 01:29:53.810 --> 01:29:59.050 align:middle line:84% So you would go every few years to replace, upgrade. 01:29:59.050 --> 01:30:02.730 align:middle line:84% And that process, we don't model it explicitly. 01:30:02.730 --> 01:30:04.510 align:middle line:84% So if you think about adopters as people 01:30:04.510 --> 01:30:07.870 align:middle line:84% who are sold on the product and either have it right now 01:30:07.870 --> 01:30:10.590 align:middle line:84% or whenever this goes off they will 01:30:10.590 --> 01:30:15.270 align:middle line:84% be willing to replace it, then that replacement rate could 01:30:15.270 --> 01:30:18.233 align:middle line:84% go to 0 and can reduce the number of adopters, 01:30:18.233 --> 01:30:19.650 align:middle line:84% shifting people to the other side. 01:30:19.650 --> 01:30:24.730 align:middle line:84% So it becomes-- there are ways to consider flows back. 01:30:24.730 --> 01:30:27.830 align:middle line:90% 01:30:27.830 --> 01:30:29.630 align:middle line:90% OK. 01:30:29.630 --> 01:30:31.590 align:middle line:90% Then we have perceived utility. 01:30:31.590 --> 01:30:40.110 align:middle line:84% That is coming from the bugs that are known issues. 01:30:40.110 --> 01:30:43.830 align:middle line:84% And by the way, depending on how we are defining issues, 01:30:43.830 --> 01:30:46.550 align:middle line:84% for a pacemaker, I think this is a good formulation, people 01:30:46.550 --> 01:30:49.390 align:middle line:84% who don't know about these problems 01:30:49.390 --> 01:30:51.750 align:middle line:90% until they have adopted it. 01:30:51.750 --> 01:30:56.710 align:middle line:84% And so unknown issues wouldn't count. 01:30:56.710 --> 01:31:00.450 align:middle line:84% But for some other products and services, 01:31:00.450 --> 01:31:03.630 align:middle line:84% there is at the point-- so if you go to a website of a company 01:31:03.630 --> 01:31:06.430 align:middle line:84% and their website is poorly designed, 01:31:06.430 --> 01:31:10.910 align:middle line:84% you may decide not to adopt it at that point in time. 01:31:10.910 --> 01:31:15.310 align:middle line:84% And those bugs are the types of bugs that are-- 01:31:15.310 --> 01:31:17.070 align:middle line:84% even when they are unknown issues-- that 01:31:17.070 --> 01:31:20.390 align:middle line:84% is, the company is not monitoring those yet 01:31:20.390 --> 01:31:24.170 align:middle line:84% or has not identified those yet to start fixing them-- 01:31:24.170 --> 01:31:27.270 align:middle line:84% they could stop customers from adopting. 01:31:27.270 --> 01:31:31.950 align:middle line:84% So for a lot of online businesses, that matters. 01:31:31.950 --> 01:31:36.490 align:middle line:84% You may go to Netflix or Amazon or wherever. 01:31:36.490 --> 01:31:39.110 align:middle line:84% And you may be disappointed with something 01:31:39.110 --> 01:31:42.150 align:middle line:90% and decide not to adopt. 01:31:42.150 --> 01:31:46.350 align:middle line:84% And that may not be in the known issues for the . 01:31:46.350 --> 01:31:48.552 align:middle line:84% Company, they would be investing. 01:31:48.552 --> 01:31:50.510 align:middle line:84% Their product development team would be looking 01:31:50.510 --> 01:31:53.650 align:middle line:90% at data from their online. 01:31:53.650 --> 01:31:56.230 align:middle line:84% So in those cases, you could have a fraction 01:31:56.230 --> 01:31:58.590 align:middle line:84% of these unknown issues also to play 01:31:58.590 --> 01:32:01.570 align:middle line:84% a role in whatever calculation for utility you're doing. 01:32:01.570 --> 01:32:06.270 align:middle line:84% I'm just highlighting that it is plausible to have that idea. 01:32:06.270 --> 01:32:10.916 align:middle line:90% So here, known issues-- 01:32:10.916 --> 01:32:17.990 align:middle line:84% AUDIENCE: [INAUDIBLE] collecting adoption rate 01:32:17.990 --> 01:32:20.360 align:middle line:90% to the [INAUDIBLE]. 01:32:20.360 --> 01:32:23.290 align:middle line:90% 01:32:23.290 --> 01:32:25.210 align:middle line:84% So right now, I'm not doing this, 01:32:25.210 --> 01:32:30.110 align:middle line:84% but instead, I'm commuting the [INAUDIBLE]. 01:32:30.110 --> 01:32:32.990 align:middle line:84% HAZHIR RAHMANDAD: Using the perceived utility to drive that. 01:32:32.990 --> 01:32:38.670 align:middle line:84% AUDIENCE: Yeah, I'm just talking about the bugs [INAUDIBLE] here 01:32:38.670 --> 01:32:41.027 align:middle line:84% for the recovery time or reporting-- 01:32:41.027 --> 01:32:42.610 align:middle line:84% HAZHIR RAHMANDAD: Yes, reporting time. 01:32:42.610 --> 01:32:47.350 align:middle line:84% AUDIENCE: So I'm just using that feedback from the stock 01:32:47.350 --> 01:32:49.033 align:middle line:90% to the time to report. 01:32:49.033 --> 01:32:49.950 align:middle line:90% HAZHIR RAHMANDAD: Yes. 01:32:49.950 --> 01:32:52.190 align:middle line:84% So you're using adopters rather than 01:32:52.190 --> 01:32:55.410 align:middle line:84% adoption rate or recent adoption rate to drive that. 01:32:55.410 --> 01:32:56.310 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:32:56.310 --> 01:32:57.470 align:middle line:84% HAZHIR RAHMANDAD: That's conceptually a little bit 01:32:57.470 --> 01:32:59.630 align:middle line:84% problematic because these adopters, many of them 01:32:59.630 --> 01:33:02.190 align:middle line:84% are very old timers who don't have anything 01:33:02.190 --> 01:33:04.110 align:middle line:90% to say about the current bugs. 01:33:04.110 --> 01:33:06.530 align:middle line:84% And therefore, the unknown issues, 01:33:06.530 --> 01:33:09.032 align:middle line:84% these are people who have very recently adopted. 01:33:09.032 --> 01:33:10.490 align:middle line:84% AUDIENCE: So that's what I thought, 01:33:10.490 --> 01:33:13.390 align:middle line:84% maybe even-- probably even in the case of pacemakers, maybe 01:33:13.390 --> 01:33:15.072 align:middle line:90% it takes like five years or so. 01:33:15.072 --> 01:33:16.030 align:middle line:90% HAZHIR RAHMANDAD: Sure. 01:33:16.030 --> 01:33:18.230 align:middle line:84% AUDIENCE: So that's why I thought maybe even 01:33:18.230 --> 01:33:20.260 align:middle line:90% very old adopters. 01:33:20.260 --> 01:33:21.510 align:middle line:90% HAZHIR RAHMANDAD: That's fair. 01:33:21.510 --> 01:33:25.110 align:middle line:84% And in those cases, I mean, you could break down the adopters 01:33:25.110 --> 01:33:26.830 align:middle line:90% into cohorts. 01:33:26.830 --> 01:33:30.550 align:middle line:84% So this is recent adopters and 2 to 3 years adopters 01:33:30.550 --> 01:33:32.470 align:middle line:90% and 4 to five years, et cetera. 01:33:32.470 --> 01:33:36.870 align:middle line:84% And then you would have different contributions 01:33:36.870 --> 01:33:40.130 align:middle line:84% to bug discovery from these different cohorts. 01:33:40.130 --> 01:33:41.150 align:middle line:90% You could do that. 01:33:41.150 --> 01:33:42.730 align:middle line:84% It's an overkill for this project, 01:33:42.730 --> 01:33:46.130 align:middle line:84% but that would be the way to tackle it. 01:33:46.130 --> 01:33:49.590 align:middle line:90% 01:33:49.590 --> 01:33:54.070 align:middle line:84% So I know you have all these parameters to create 01:33:54.070 --> 01:33:57.590 align:middle line:90% these S-shaped functions. 01:33:57.590 --> 01:34:05.510 align:middle line:84% Again, kind of OK, but perhaps an overkill for some of these. 01:34:05.510 --> 01:34:11.590 align:middle line:84% And also, right now, because of doing the known issues 01:34:11.590 --> 01:34:16.510 align:middle line:84% in its own units-- it is its dimensionless. 01:34:16.510 --> 01:34:18.850 align:middle line:84% I would probably not make it dimensionless. 01:34:18.850 --> 01:34:19.890 align:middle line:90% Bugs have a unit. 01:34:19.890 --> 01:34:24.270 align:middle line:84% It's either a bug or it is in units of tasks that are buggy 01:34:24.270 --> 01:34:27.070 align:middle line:90% or features that are buggy. 01:34:27.070 --> 01:34:29.310 align:middle line:84% But then you have this bug scale thing that 01:34:29.310 --> 01:34:31.330 align:middle line:90% needs to come in and impact. 01:34:31.330 --> 01:34:34.280 align:middle line:90% 01:34:34.280 --> 01:34:37.150 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:34:37.150 --> 01:34:38.790 align:middle line:90% HAZHIR RAHMANDAD: Yeah. 01:34:38.790 --> 01:34:41.830 align:middle line:84% Right now, the unit problems that, I think, Hesam was also 01:34:41.830 --> 01:34:43.850 align:middle line:84% highlighting are showing up here-- 01:34:43.850 --> 01:34:46.970 align:middle line:84% that is, known issues by making it dimensionless. 01:34:46.970 --> 01:34:48.450 align:middle line:84% But this thing can go up and down. 01:34:48.450 --> 01:34:53.630 align:middle line:84% It's determined partly based on the scope. 01:34:53.630 --> 01:34:56.690 align:middle line:84% And the units for scope is also dimensionless. 01:34:56.690 --> 01:34:59.630 align:middle line:84% So this makes it harder for you to really use 01:34:59.630 --> 01:35:03.310 align:middle line:84% the units as a way of setting things right. 01:35:03.310 --> 01:35:08.830 align:middle line:84% So this is another generally good insight. 01:35:08.830 --> 01:35:12.590 align:middle line:84% You can use dimensionless for everything, ignore units. 01:35:12.590 --> 01:35:17.110 align:middle line:84% But you're depriving yourself of a very helpful avenue 01:35:17.110 --> 01:35:19.130 align:middle line:84% for thinking clearly about the problem. 01:35:19.130 --> 01:35:21.070 align:middle line:84% So don't use dimensionless unless something 01:35:21.070 --> 01:35:24.990 align:middle line:84% is really dimensionless so that dimensionless is meaningful. 01:35:24.990 --> 01:35:28.230 align:middle line:84% There are situations that, of the way Vensim 01:35:28.230 --> 01:35:33.390 align:middle line:84% reports unit errors, if you have x to the power of y, 01:35:33.390 --> 01:35:35.770 align:middle line:84% both x and y need to be dimensionless. 01:35:35.770 --> 01:35:37.690 align:middle line:84% Otherwise, Vensim gives you errors. 01:35:37.690 --> 01:35:40.270 align:middle line:84% Then you may need some kind of normalizing 01:35:40.270 --> 01:35:44.310 align:middle line:84% to get things dimensionless for those rare cases 01:35:44.310 --> 01:35:45.570 align:middle line:90% to just deal with Vensim. 01:35:45.570 --> 01:35:49.910 align:middle line:84% But conceptually, always have units that are consistent. 01:35:49.910 --> 01:35:51.950 align:middle line:84% AUDIENCE: It's in scope something. 01:35:51.950 --> 01:35:54.510 align:middle line:84% HAZHIR RAHMANDAD: So this should be, in my view, in features. 01:35:54.510 --> 01:35:56.177 align:middle line:84% Scope should be in the unit of features. 01:35:56.177 --> 01:35:58.270 align:middle line:84% But Ali has put it as dimensionless. 01:35:58.270 --> 01:36:00.890 align:middle line:84% And when you put that as dimensionless, 01:36:00.890 --> 01:36:04.630 align:middle line:84% I can't really track what exactly is measured here, 01:36:04.630 --> 01:36:05.970 align:middle line:90% how it is counted. 01:36:05.970 --> 01:36:09.290 align:middle line:84% So is it this scale of it between 0 and 1? 01:36:09.290 --> 01:36:12.070 align:middle line:90% Or is it between 0 and 100? 01:36:12.070 --> 01:36:16.790 align:middle line:84% And I wouldn't know when you have dimensionless 01:36:16.790 --> 01:36:18.430 align:middle line:90% how to think about that. 01:36:18.430 --> 01:36:24.430 align:middle line:84% But then also, it goes into bugs per change in scope of use. 01:36:24.430 --> 01:36:26.850 align:middle line:84% And it is not really changing scope of use. 01:36:26.850 --> 01:36:33.690 align:middle line:84% It's bugs per scope unit because this thing is feature per month. 01:36:33.690 --> 01:36:37.350 align:middle line:84% And this is also bug per month, if you want to have that. 01:36:37.350 --> 01:36:40.910 align:middle line:84% So you already see the lack of clarity seeps 01:36:40.910 --> 01:36:46.290 align:middle line:84% into the variable name selection as soon as you remove the unit. 01:36:46.290 --> 01:36:49.510 align:middle line:90% So it's pretty helpful to do it. 01:36:49.510 --> 01:36:54.260 align:middle line:84% OK, unknown issues-- anyway, they are in the scale of-- 01:36:54.260 --> 01:36:56.170 align:middle line:90% I assume this is also small. 01:36:56.170 --> 01:36:57.310 align:middle line:90% Yeah, so this is 2. 01:36:57.310 --> 01:37:01.270 align:middle line:84% So these are the scale of what the scope of use is. 01:37:01.270 --> 01:37:06.070 align:middle line:84% So then up here, you come and use this scale thing 01:37:06.070 --> 01:37:11.150 align:middle line:84% to really try to grapple with how many bugs do I have versus 01:37:11.150 --> 01:37:13.950 align:middle line:90% how many adopters do I have. 01:37:13.950 --> 01:37:19.070 align:middle line:84% And so first of all, you shouldn't be comparing bugs 01:37:19.070 --> 01:37:21.350 align:middle line:90% with adopters. 01:37:21.350 --> 01:37:24.230 align:middle line:84% Why are you normalizing against adopters? 01:37:24.230 --> 01:37:26.650 align:middle line:84% It is really bugs compared to features. 01:37:26.650 --> 01:37:31.070 align:middle line:84% So that's the real driver of the normalization, which is missing 01:37:31.070 --> 01:37:34.515 align:middle line:90% from this formulation here. 01:37:34.515 --> 01:37:35.390 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:37:35.390 --> 01:37:36.348 align:middle line:90% HAZHIR RAHMANDAD: Sure. 01:37:36.348 --> 01:37:37.390 align:middle line:90% AUDIENCE: [INAUDIBLE]. 01:37:37.390 --> 01:37:39.710 align:middle line:84% I was thinking that if I put myself 01:37:39.710 --> 01:37:41.813 align:middle line:84% in the shoes of a person who's [INAUDIBLE]. 01:37:41.813 --> 01:37:42.730 align:middle line:90% HAZHIR RAHMANDAD: Yes. 01:37:42.730 --> 01:37:46.070 align:middle line:84% AUDIENCE: And I see how many people have adopted it 01:37:46.070 --> 01:37:50.870 align:middle line:84% and I also see how many bugs that we have so that 01:37:50.870 --> 01:37:52.490 align:middle line:90% is [INAUDIBLE]. 01:37:52.490 --> 01:37:54.690 align:middle line:84% HAZHIR RAHMANDAD: So this is very important. 01:37:54.690 --> 01:37:58.750 align:middle line:84% What is the definition of a bug in your formulation, the way you 01:37:58.750 --> 01:38:00.190 align:middle line:90% are thinking about it? 01:38:00.190 --> 01:38:03.710 align:middle line:84% You kind of switched between two ways of thinking, 01:38:03.710 --> 01:38:06.010 align:middle line:84% maybe even again because of the units. 01:38:06.010 --> 01:38:08.070 align:middle line:84% One is the version that we have seen 01:38:08.070 --> 01:38:10.990 align:middle line:84% in most of the models, which is each bug 01:38:10.990 --> 01:38:15.910 align:middle line:84% corresponds to a feature that is buggy or bugs within a feature. 01:38:15.910 --> 01:38:19.070 align:middle line:84% So no matter you have 100 customers or a million 01:38:19.070 --> 01:38:21.670 align:middle line:84% customers, there is a buggy feature, 01:38:21.670 --> 01:38:23.910 align:middle line:90% it has a negative utility. 01:38:23.910 --> 01:38:25.630 align:middle line:90% Then there was another feature-- 01:38:25.630 --> 01:38:29.830 align:middle line:84% way of defining that we saw in Jason's model, where 01:38:29.830 --> 01:38:34.230 align:middle line:84% he was tracking features multiplied by customers or buggy 01:38:34.230 --> 01:38:35.970 align:middle line:84% features multiplied by customers. 01:38:35.970 --> 01:38:39.950 align:middle line:84% This is how many bugs total are in the population. 01:38:39.950 --> 01:38:43.470 align:middle line:84% That conceptualization then requires 01:38:43.470 --> 01:38:47.610 align:middle line:84% you to normalize also against the total number of customers. 01:38:47.610 --> 01:38:50.550 align:middle line:84% And that's what might be driving your thinking 01:38:50.550 --> 01:38:51.890 align:middle line:90% in the second part. 01:38:51.890 --> 01:38:54.010 align:middle line:84% So in one part, you're using one definition. 01:38:54.010 --> 01:38:56.950 align:middle line:84% In another, you're using another definition. 01:38:56.950 --> 01:39:00.990 align:middle line:84% AUDIENCE: Does it matter for bugs 01:39:00.990 --> 01:39:03.750 align:middle line:84% to be normalized against the number of features 01:39:03.750 --> 01:39:06.490 align:middle line:90% versus just the number of bugs? 01:39:06.490 --> 01:39:07.192 align:middle line:90% Because I mean-- 01:39:07.192 --> 01:39:09.150 align:middle line:84% HAZHIR RAHMANDAD: What do you mean by normalize 01:39:09.150 --> 01:39:11.270 align:middle line:90% against the number of bugs? 01:39:11.270 --> 01:39:14.515 align:middle line:84% AUDIENCE: Not against the number of bugs, but-- 01:39:14.515 --> 01:39:15.890 align:middle line:84% HAZHIR RAHMANDAD: Non-normalized. 01:39:15.890 --> 01:39:18.510 align:middle line:84% AUDIENCE: I [INAUDIBLE] to care that there's 01:39:18.510 --> 01:39:21.470 align:middle line:84% two bugs if it has a million features or two bugs if there's 01:39:21.470 --> 01:39:25.430 align:middle line:84% 10 features [INAUDIBLE] the bugs are a problem. 01:39:25.430 --> 01:39:28.045 align:middle line:90% 01:39:28.045 --> 01:39:29.670 align:middle line:84% HAZHIR RAHMANDAD: It's a good question. 01:39:29.670 --> 01:39:36.270 align:middle line:84% And the way I think about it is how serious those bugs are. 01:39:36.270 --> 01:39:38.710 align:middle line:84% And to get a feel for how serious those bugs are, 01:39:38.710 --> 01:39:41.630 align:middle line:84% you should compare it to the opportunity space 01:39:41.630 --> 01:39:42.530 align:middle line:90% for having bugs. 01:39:42.530 --> 01:39:45.690 align:middle line:84% If you have a million features, you could have a million bugs. 01:39:45.690 --> 01:39:47.870 align:middle line:84% And the fact that you are seeing only two 01:39:47.870 --> 01:39:50.910 align:middle line:84% means this is actually a very good product. 01:39:50.910 --> 01:39:53.870 align:middle line:84% If you had two features and you had two bugs, 01:39:53.870 --> 01:39:56.330 align:middle line:84% that would be a terrible product. 01:39:56.330 --> 01:40:00.630 align:middle line:84% So by not normalizing, we miss the picture on that opportunity 01:40:00.630 --> 01:40:03.550 align:middle line:84% space for having bugs and, therefore, what is considered 01:40:03.550 --> 01:40:05.310 align:middle line:90% a bug, how serious it is. 01:40:05.310 --> 01:40:07.470 align:middle line:90% That's the idea. 01:40:07.470 --> 01:40:10.670 align:middle line:84% Now, when you're doing it for a single product, 01:40:10.670 --> 01:40:14.450 align:middle line:84% that normalization factor may not be as big a deal. 01:40:14.450 --> 01:40:19.070 align:middle line:84% You can deal with it OK by just think of it as a number. 01:40:19.070 --> 01:40:24.050 align:middle line:84% But to make a robust model that is more generalizable-- 01:40:24.050 --> 01:40:26.010 align:middle line:84% but also, in all the formulations, 01:40:26.010 --> 01:40:28.710 align:middle line:84% when you want to have dimensionless effects 01:40:28.710 --> 01:40:31.930 align:middle line:84% of these bugs on utility, on other things, 01:40:31.930 --> 01:40:33.890 align:middle line:90% you do need a normalization. 01:40:33.890 --> 01:40:35.950 align:middle line:84% So I think, for both of those reasons, 01:40:35.950 --> 01:40:39.630 align:middle line:84% it's a good idea to have the normalization. 01:40:39.630 --> 01:40:40.270 align:middle line:90% OK. 01:40:40.270 --> 01:40:42.170 align:middle line:84% I think we covered most of the things here. 01:40:42.170 --> 01:40:44.850 align:middle line:84% Anything else that you want us to look at, Ali? 01:40:44.850 --> 01:40:47.270 align:middle line:84% AUDIENCE: [INAUDIBLE] have a quick question-- a comment 01:40:47.270 --> 01:40:49.650 align:middle line:90% that scope [INAUDIBLE]. 01:40:49.650 --> 01:40:54.610 align:middle line:90% 01:40:54.610 --> 01:40:58.910 align:middle line:84% Is that specific to this model or to Ali's model or just 01:40:58.910 --> 01:41:00.790 align:middle line:84% in general, how you would think about it? 01:41:00.790 --> 01:41:02.665 align:middle line:84% HAZHIR RAHMANDAD: I think, for this exercise, 01:41:02.665 --> 01:41:04.950 align:middle line:84% I would think of scope as something along the lines 01:41:04.950 --> 01:41:07.150 align:middle line:90% of features or-- 01:41:07.150 --> 01:41:09.150 align:middle line:84% AUDIENCE: And then that determines 01:41:09.150 --> 01:41:12.470 align:middle line:90% the maximum population. 01:41:12.470 --> 01:41:14.430 align:middle line:84% HAZHIR RAHMANDAD: Yes, so that transformation 01:41:14.430 --> 01:41:18.650 align:middle line:84% is what we saw in-- we were just looking at Ross's model. 01:41:18.650 --> 01:41:20.990 align:middle line:84% We talked about the possibility of normalizing 01:41:20.990 --> 01:41:24.110 align:middle line:84% that against the maximum viable feature 01:41:24.110 --> 01:41:28.910 align:middle line:84% so that you do have a dimensionless ratio that, then 01:41:28.910 --> 01:41:31.270 align:middle line:84% you can look at it's incremental value 01:41:31.270 --> 01:41:35.950 align:middle line:84% for having more customers, something along those lines. 01:41:35.950 --> 01:41:39.950 align:middle line:90% OK, good. 01:41:39.950 --> 01:41:42.490 align:middle line:84% Let's see what models we haven't looked at. 01:41:42.490 --> 01:41:43.970 align:middle line:90% This one is Kathy's. 01:41:43.970 --> 01:41:47.030 align:middle line:90% So let's look at this now. 01:41:47.030 --> 01:41:48.788 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:41:48.788 --> 01:41:50.330 align:middle line:84% HAZHIR RAHMANDAD: Could you speak up? 01:41:50.330 --> 01:41:52.663 align:middle line:84% AUDIENCE: Yeah, I did discover a unit error at 10:00 AM. 01:41:52.663 --> 01:41:54.110 align:middle line:90% It's in risk perception. 01:41:54.110 --> 01:41:55.390 align:middle line:90% HAZHIR RAHMANDAD: OK. 01:41:55.390 --> 01:41:57.530 align:middle line:84% AUDIENCE: It won't be clean [INAUDIBLE]. 01:41:57.530 --> 01:41:59.910 align:middle line:90% HAZHIR RAHMANDAD: No worries. 01:41:59.910 --> 01:42:02.250 align:middle line:84% It has fewer unit errors than Jason's. 01:42:02.250 --> 01:42:07.650 align:middle line:90% 01:42:07.650 --> 01:42:08.150 align:middle line:90% OK. 01:42:08.150 --> 01:42:18.270 align:middle line:84% So let's take a look at features. 01:42:18.270 --> 01:42:23.910 align:middle line:84% So increasing features is based on a maximum feature idea, 01:42:23.910 --> 01:42:28.230 align:middle line:84% similar to Ali's and remaining features. 01:42:28.230 --> 01:42:30.270 align:middle line:84% You add them with the time constant. 01:42:30.270 --> 01:42:33.550 align:middle line:84% So initially, you add more features. 01:42:33.550 --> 01:42:36.550 align:middle line:84% And over time, you add fewer features. 01:42:36.550 --> 01:42:39.210 align:middle line:84% And this part is pretty much built into it. 01:42:39.210 --> 01:42:43.190 align:middle line:84% That is, the whole function of this part of the model 01:42:43.190 --> 01:42:47.390 align:middle line:84% becomes creating a goal-seeking behavior that 01:42:47.390 --> 01:42:49.810 align:middle line:84% doesn't get any feedback from other parts of the model. 01:42:49.810 --> 01:42:50.490 align:middle line:90% And that's OK. 01:42:50.490 --> 01:42:53.110 align:middle line:84% Again, as we discussed, there are different ways 01:42:53.110 --> 01:42:55.350 align:middle line:90% to formulate it here. 01:42:55.350 --> 01:42:58.810 align:middle line:84% AUDIENCE: It does get feedback from the R&D pressure. 01:42:58.810 --> 01:43:01.190 align:middle line:84% HAZHIR RAHMANDAD: Ah, because the time to add features 01:43:01.190 --> 01:43:03.250 align:middle line:84% is getting feedback from R&D pressure. 01:43:03.250 --> 01:43:06.110 align:middle line:90% OK, good. 01:43:06.110 --> 01:43:11.110 align:middle line:84% And then these features lead to effective features 01:43:11.110 --> 01:43:12.690 align:middle line:90% and marginal utility. 01:43:12.690 --> 01:43:14.370 align:middle line:90% So how is that formulated? 01:43:14.370 --> 01:43:16.630 align:middle line:84% It is features divided by initial 01:43:16.630 --> 01:43:21.310 align:middle line:84% raised to the power of marginal utility reduction curve 01:43:21.310 --> 01:43:22.950 align:middle line:90% strength. 01:43:22.950 --> 01:43:29.230 align:middle line:84% So this is this marginal utility curve strength. 01:43:29.230 --> 01:43:30.290 align:middle line:90% It's a mouthful. 01:43:30.290 --> 01:43:33.590 align:middle line:90% 01:43:33.590 --> 01:43:36.330 align:middle line:84% AUDIENCE: It's from the VCR model formulation. 01:43:36.330 --> 01:43:38.870 align:middle line:84% HAZHIR RAHMANDAD: Yeah, that's OK. 01:43:38.870 --> 01:43:41.550 align:middle line:84% You basically have one parameter here. 01:43:41.550 --> 01:43:45.270 align:middle line:84% Now, you're transforming this parameter-- 01:43:45.270 --> 01:43:48.390 align:middle line:84% calculating this parameter using another parameter. 01:43:48.390 --> 01:43:49.810 align:middle line:90% It's an overkill. 01:43:49.810 --> 01:43:52.590 align:middle line:84% You then you need to explain exactly why. 01:43:52.590 --> 01:43:55.370 align:middle line:90% And I'm OK with it. 01:43:55.370 --> 01:44:00.470 align:middle line:84% But it's not adding much value to have two parameters there. 01:44:00.470 --> 01:44:06.370 align:middle line:84% And then that feature rate leads to marginal feature utility. 01:44:06.370 --> 01:44:12.830 align:middle line:84% This is very similar to what Ross was working on as well. 01:44:12.830 --> 01:44:18.790 align:middle line:84% So this feature utility is now not a number 01:44:18.790 --> 01:44:22.170 align:middle line:90% that is bounded by anything. 01:44:22.170 --> 01:44:23.310 align:middle line:90% It can be above 1. 01:44:23.310 --> 01:44:25.530 align:middle line:90% And it can be below 1. 01:44:25.530 --> 01:44:29.150 align:middle line:90% It goes into the utility. 01:44:29.150 --> 01:44:30.930 align:middle line:90% It's actually interesting. 01:44:30.930 --> 01:44:35.430 align:middle line:84% So right now, the formulation you have here 01:44:35.430 --> 01:44:41.070 align:middle line:84% is a little bit confusing for me because the marginal utility 01:44:41.070 --> 01:44:48.990 align:middle line:84% of features is the derivative of utility 01:44:48.990 --> 01:44:51.790 align:middle line:90% with respect to features. 01:44:51.790 --> 01:44:53.850 align:middle line:84% If you're calling that marginal utility, 01:44:53.850 --> 01:44:55.970 align:middle line:84% that's the definition of marginal utility. 01:44:55.970 --> 01:44:57.590 align:middle line:84% Then this marginal feature utility 01:44:57.590 --> 01:45:00.910 align:middle line:84% would not be going into the utility as an integral-- 01:45:00.910 --> 01:45:05.310 align:middle line:84% to get integrated into the utility. 01:45:05.310 --> 01:45:08.750 align:middle line:90% What you would have is-- 01:45:08.750 --> 01:45:11.390 align:middle line:90% I have this number of features. 01:45:11.390 --> 01:45:14.350 align:middle line:84% That creates a utility for me, which 01:45:14.350 --> 01:45:17.270 align:middle line:84% can come from the effect of features on utility 01:45:17.270 --> 01:45:19.950 align:middle line:90% rather than marginal utility. 01:45:19.950 --> 01:45:22.850 align:middle line:84% So you would have initial utility multiplied by effect 01:45:22.850 --> 01:45:25.110 align:middle line:90% of features on utility. 01:45:25.110 --> 01:45:31.168 align:middle line:84% And that gives you the utility that you want to have. 01:45:31.168 --> 01:45:31.710 align:middle line:90% AUDIENCE: OK. 01:45:31.710 --> 01:45:38.070 align:middle line:84% I might have not really been conceptualizing utility 01:45:38.070 --> 01:45:38.890 align:middle line:90% correctly. 01:45:38.890 --> 01:45:42.710 align:middle line:84% I was thinking of it as like a basic-- 01:45:42.710 --> 01:45:44.910 align:middle line:84% it's meant just to be a number of features 01:45:44.910 --> 01:45:49.470 align:middle line:84% based on that marginal add of each feature. 01:45:49.470 --> 01:45:55.150 align:middle line:84% So for instance, I think most people did utility 01:45:55.150 --> 01:46:01.030 align:middle line:84% as a combination of features and bugs. 01:46:01.030 --> 01:46:02.953 align:middle line:90% But for me, utility is just-- 01:46:02.953 --> 01:46:04.370 align:middle line:84% HAZHIR RAHMANDAD: No, that's fine. 01:46:04.370 --> 01:46:06.525 align:middle line:90% I think yours is not wrong. 01:46:06.525 --> 01:46:07.650 align:middle line:90% I'm not saying it is wrong. 01:46:07.650 --> 01:46:14.750 align:middle line:84% I just see it as a more convoluted way of modeling it. 01:46:14.750 --> 01:46:16.790 align:middle line:90% It is not wrong. 01:46:16.790 --> 01:46:18.670 align:middle line:84% And the way you do it later-- that 01:46:18.670 --> 01:46:23.367 align:middle line:84% is, you then multiplied increase in features by marginal feature 01:46:23.367 --> 01:46:24.950 align:middle line:84% utility-- is correct-- that is, if you 01:46:24.950 --> 01:46:28.230 align:middle line:84% want to calculate total utility using this version. 01:46:28.230 --> 01:46:29.910 align:middle line:84% So what you're doing is that you are 01:46:29.910 --> 01:46:35.610 align:middle line:84% calculating the marginal utility from the current feature level. 01:46:35.610 --> 01:46:37.850 align:middle line:84% And then you accumulate utility that way. 01:46:37.850 --> 01:46:41.410 align:middle line:84% And then this actually becomes the correct marginal utility. 01:46:41.410 --> 01:46:44.170 align:middle line:84% So this is actually mathematically correct. 01:46:44.170 --> 01:46:46.695 align:middle line:84% It's just a little bit more complexity. 01:46:46.695 --> 01:46:48.070 align:middle line:84% Now, sometimes, it is a good idea 01:46:48.070 --> 01:46:50.070 align:middle line:84% to have that complexity because maybe I 01:46:50.070 --> 01:46:53.190 align:middle line:84% know how to calculate the marginal utility 01:46:53.190 --> 01:46:56.430 align:middle line:84% from the current feature level, but I can't calculate 01:46:56.430 --> 01:46:59.190 align:middle line:84% the total utility from current feature level 01:46:59.190 --> 01:47:03.310 align:middle line:84% because that's an integral that I can't take analytically. 01:47:03.310 --> 01:47:07.110 align:middle line:84% Then this formulation allows you to take that integral here 01:47:07.110 --> 01:47:10.750 align:middle line:84% using a different stock and have the utility 01:47:10.750 --> 01:47:13.610 align:middle line:84% and the marginal utility calculated at the same time. 01:47:13.610 --> 01:47:14.750 align:middle line:90% That's great. 01:47:14.750 --> 01:47:17.670 align:middle line:84% So in a way, it is a good trick if you 01:47:17.670 --> 01:47:22.430 align:middle line:84% need a more complex marginal formulation 01:47:22.430 --> 01:47:27.450 align:middle line:84% that you know how it depends on the thing that is driving it. 01:47:27.450 --> 01:47:28.950 align:middle line:90% OK. 01:47:28.950 --> 01:47:34.390 align:middle line:84% Then you have the utility effect on indicated fraction 01:47:34.390 --> 01:47:36.670 align:middle line:90% of adopters. 01:47:36.670 --> 01:47:41.230 align:middle line:84% So you have this ad hoc maximum utility here. 01:47:41.230 --> 01:47:47.990 align:middle line:84% I don't know how you would make sure this is maximum utility. 01:47:47.990 --> 01:47:51.450 align:middle line:84% AUDIENCE: There's a maximum number of features somewhere. 01:47:51.450 --> 01:47:53.030 align:middle line:84% And I just said the maximum number 01:47:53.030 --> 01:47:57.685 align:middle line:84% of features times 1 util per feature is maximum utility. 01:47:57.685 --> 01:47:59.310 align:middle line:84% HAZHIR RAHMANDAD: Yeah, but that is not 01:47:59.310 --> 01:48:03.030 align:middle line:90% your maximum of this integral. 01:48:03.030 --> 01:48:06.030 align:middle line:84% That is, if you increase the features 01:48:06.030 --> 01:48:11.950 align:middle line:84% over time using whatever inflow you want to put into it, 01:48:11.950 --> 01:48:16.070 align:middle line:84% you will get to a utility level that saturates at some point, 01:48:16.070 --> 01:48:16.570 align:middle line:90% perhaps. 01:48:16.570 --> 01:48:19.650 align:middle line:84% I mean, I don't know for sure, given this formulation. 01:48:19.650 --> 01:48:20.810 align:middle line:90% Maybe it never saturates. 01:48:20.810 --> 01:48:22.750 align:middle line:90% Maybe it continues forever. 01:48:22.750 --> 01:48:26.870 align:middle line:84% But once you get to your maximum feature, 01:48:26.870 --> 01:48:28.850 align:middle line:90% it will saturate at some level. 01:48:28.850 --> 01:48:33.050 align:middle line:84% That is the value that should go into the maximum utility. 01:48:33.050 --> 01:48:35.830 align:middle line:84% But you don't know that value beforehand 01:48:35.830 --> 01:48:40.590 align:middle line:84% because it requires solving this integral problem. 01:48:40.590 --> 01:48:43.670 align:middle line:84% Now, you might be able to analytically solve 01:48:43.670 --> 01:48:46.550 align:middle line:90% that integral problem. 01:48:46.550 --> 01:48:51.750 align:middle line:84% So doing that is a little bit of work, back of the envelope 01:48:51.750 --> 01:48:56.710 align:middle line:84% or under some real paper and pen work because it 01:48:56.710 --> 01:48:58.050 align:middle line:90% might be more complex. 01:48:58.050 --> 01:49:00.870 align:middle line:84% But you could potentially do that calculation, 01:49:00.870 --> 01:49:03.910 align:middle line:84% and then use that maximum utility. 01:49:03.910 --> 01:49:05.230 align:middle line:90% That is correct. 01:49:05.230 --> 01:49:07.490 align:middle line:84% The back of envelope one that you have done, 01:49:07.490 --> 01:49:09.950 align:middle line:84% which is features multiplied by 1 and 1 01:49:09.950 --> 01:49:15.790 align:middle line:84% being the maximum marginal utility that you could conceive, 01:49:15.790 --> 01:49:21.710 align:middle line:84% is not quite satisfactory because then it 01:49:21.710 --> 01:49:23.970 align:middle line:90% doesn't scale between 0 and 1. 01:49:23.970 --> 01:49:27.790 align:middle line:84% It scales between 0 and some number. 01:49:27.790 --> 01:49:30.992 align:middle line:84% But that number you don't know what it is. 01:49:30.992 --> 01:49:32.950 align:middle line:84% AUDIENCE: Could you say a little more about how 01:49:32.950 --> 01:49:34.790 align:middle line:90% I would go about doing that? 01:49:34.790 --> 01:49:36.430 align:middle line:90% HAZHIR RAHMANDAD: Sure. 01:49:36.430 --> 01:49:42.210 align:middle line:84% So the solution, whether it is feasible or not, 01:49:42.210 --> 01:49:45.630 align:middle line:84% depends on whether you can solve analytically 01:49:45.630 --> 01:49:49.190 align:middle line:84% the maximum level of utility once you 01:49:49.190 --> 01:49:54.990 align:middle line:84% get the integral of feature from initial feature 01:49:54.990 --> 01:49:57.470 align:middle line:90% to maximum feature. 01:49:57.470 --> 01:50:06.470 align:middle line:84% So it is integral over feature multiplied by marginal utility. 01:50:06.470 --> 01:50:09.630 align:middle line:90% That becomes the total utility-- 01:50:09.630 --> 01:50:14.750 align:middle line:84% the feature going from feature being 01:50:14.750 --> 01:50:21.270 align:middle line:84% equal to minimum or initial until it is maximum. 01:50:21.270 --> 01:50:24.350 align:middle line:84% So it is feature multiplied by effective feature 01:50:24.350 --> 01:50:26.990 align:middle line:84% on marginal utility, which is feature itself 01:50:26.990 --> 01:50:30.630 align:middle line:84% to the power of marginal utility sensitivity. 01:50:30.630 --> 01:50:35.430 align:middle line:84% So it becomes feature f to the power of 1 01:50:35.430 --> 01:50:39.390 align:middle line:84% plus this alpha parameter that you have. 01:50:39.390 --> 01:50:44.350 align:middle line:84% So integral of that from minimum f to maximum f 01:50:44.350 --> 01:50:47.270 align:middle line:84% is analytically calculatable because it is just 01:50:47.270 --> 01:50:48.930 align:middle line:90% a power function. 01:50:48.930 --> 01:50:52.330 align:middle line:84% So you can relatively easily calculate that, 01:50:52.330 --> 01:50:58.270 align:middle line:84% use that calculation to put it into your maximum utility. 01:50:58.270 --> 01:51:00.070 align:middle line:90% Is that clear? 01:51:00.070 --> 01:51:01.870 align:middle line:90% It's a little bit-- 01:51:01.870 --> 01:51:04.170 align:middle line:84% yeah I just hand-wavingly did an integral. 01:51:04.170 --> 01:51:08.830 align:middle line:84% So it may not be obvious what's going in it. 01:51:08.830 --> 01:51:11.330 align:middle line:84% Now, the problem with that is-- it's OK. 01:51:11.330 --> 01:51:14.150 align:middle line:84% But it is a little bit tricky because then the formulation 01:51:14.150 --> 01:51:15.130 align:middle line:90% here is complex. 01:51:15.130 --> 01:51:18.590 align:middle line:84% You have to actually explain quite much with detail. 01:51:18.590 --> 01:51:22.290 align:middle line:84% And it has to come from these parameters. 01:51:22.290 --> 01:51:26.150 align:middle line:84% So the maximum feature, initial feature, 01:51:26.150 --> 01:51:28.070 align:middle line:84% and this marginal utility, all of them 01:51:28.070 --> 01:51:31.950 align:middle line:84% should go into that equation so that it is all internally robust 01:51:31.950 --> 01:51:32.770 align:middle line:90% and consistent. 01:51:32.770 --> 01:51:34.550 align:middle line:90% If you do that, then it is fine. 01:51:34.550 --> 01:51:37.350 align:middle line:84% And then you make this a maximum-- sorry, 01:51:37.350 --> 01:51:39.370 align:middle line:90% initial function. 01:51:39.370 --> 01:51:45.430 align:middle line:84% It is initial value calculated based on those three parameters. 01:51:45.430 --> 01:51:49.230 align:middle line:84% Anyway, once you have done that, it all works. 01:51:49.230 --> 01:51:57.310 align:middle line:84% Then you have a utility effect on fraction 01:51:57.310 --> 01:51:59.370 align:middle line:84% because this number is between 0 and 1. 01:51:59.370 --> 01:52:03.610 align:middle line:84% This indicated fraction will also remain between 0 and 1. 01:52:03.610 --> 01:52:06.550 align:middle line:90% 01:52:06.550 --> 01:52:11.390 align:middle line:84% Now, at this point, I'm inclined not to have 01:52:11.390 --> 01:52:15.150 align:middle line:90% a separate strength of utility. 01:52:15.150 --> 01:52:22.030 align:middle line:84% I'm inclined to make utility act as my fraction of customers 01:52:22.030 --> 01:52:25.670 align:middle line:84% who are interested in this market 01:52:25.670 --> 01:52:29.870 align:middle line:84% because, otherwise, I have this unobservable qualitative 01:52:29.870 --> 01:52:30.770 align:middle line:90% thing in between. 01:52:30.770 --> 01:52:33.470 align:middle line:84% And I have two nonlinear transformations going into it 01:52:33.470 --> 01:52:34.790 align:middle line:90% and coming out of it. 01:52:34.790 --> 01:52:37.050 align:middle line:84% It's very hard to then track what is happening. 01:52:37.050 --> 01:52:38.150 align:middle line:90% And it doesn't add value. 01:52:38.150 --> 01:52:40.030 align:middle line:84% Mathematically, it doesn't add value. 01:52:40.030 --> 01:52:44.070 align:middle line:84% So I would fold this additional parameter 01:52:44.070 --> 01:52:49.430 align:middle line:84% into whatever I think is the effect of this marginal utility 01:52:49.430 --> 01:52:51.010 align:middle line:90% reduction curve strength. 01:52:51.010 --> 01:52:52.813 align:middle line:84% And I'll just take care of it there. 01:52:52.813 --> 01:52:54.730 align:middle line:84% AUDIENCE: And you said something a second ago. 01:52:54.730 --> 01:52:58.950 align:middle line:84% You said the maximum utility would be some initial value. 01:52:58.950 --> 01:53:00.250 align:middle line:90% What do you mean by that? 01:53:00.250 --> 01:53:01.950 align:middle line:84% HAZHIR RAHMANDAD: Maximum utility-- 01:53:01.950 --> 01:53:06.590 align:middle line:84% oh, in Vensim, I would make it an initial equation. 01:53:06.590 --> 01:53:10.350 align:middle line:84% So typically, auxiliary equations or initial equations-- 01:53:10.350 --> 01:53:13.670 align:middle line:84% initial was calculated once and is never updated 01:53:13.670 --> 01:53:14.850 align:middle line:90% throughout the simulation. 01:53:14.850 --> 01:53:17.370 align:middle line:84% So it acts like a parameter for you. 01:53:17.370 --> 01:53:22.230 align:middle line:84% That's why I would think of it that way. 01:53:22.230 --> 01:53:26.530 align:middle line:84% Now, yeah, that's not a big deal. 01:53:26.530 --> 01:53:28.650 align:middle line:84% You could also just have a normal auxiliary. 01:53:28.650 --> 01:53:37.470 align:middle line:84% And then this indicated fraction here is being affected by flaws. 01:53:37.470 --> 01:53:43.190 align:middle line:84% So you have flawed features that go into the features 01:53:43.190 --> 01:53:44.670 align:middle line:90% introduction. 01:53:44.670 --> 01:53:48.310 align:middle line:84% And then the outflow is by fixing 01:53:48.310 --> 01:53:53.290 align:middle line:84% of these through third order delay or something, I assume. 01:53:53.290 --> 01:53:56.670 align:middle line:90% 01:53:56.670 --> 01:54:00.910 align:middle line:84% That's all fine-- again, with the caveat of, OK, then 01:54:00.910 --> 01:54:03.990 align:middle line:84% this is kind of sidestepping the issues related to allocation 01:54:03.990 --> 01:54:07.090 align:middle line:84% of resources between bug-fixing and new features and so on. 01:54:07.090 --> 01:54:14.390 align:middle line:84% But that's OK, as long as you make that choice consciously. 01:54:14.390 --> 01:54:20.070 align:middle line:84% Flaws per feature, it's also well-defined. 01:54:20.070 --> 01:54:27.990 align:middle line:84% But the flaws per feature part is not normalized. 01:54:27.990 --> 01:54:31.910 align:middle line:84% Oh, OK, it is normalized because you 01:54:31.910 --> 01:54:37.950 align:middle line:84% are using the same units for features and flaws. 01:54:37.950 --> 01:54:40.650 align:middle line:90% Good, good. 01:54:40.650 --> 01:54:44.000 align:middle line:84% So that's of keeping it into the same units. 01:54:44.000 --> 01:54:45.440 align:middle line:90% And that makes sense. 01:54:45.440 --> 01:54:51.020 align:middle line:90% 01:54:51.020 --> 01:54:58.288 align:middle line:84% And then you are looking at adopters with flaws. 01:54:58.288 --> 01:54:59.080 align:middle line:90% That's interesting. 01:54:59.080 --> 01:55:07.340 align:middle line:84% So you're saying folks who have problematic items are creating 01:55:07.340 --> 01:55:09.360 align:middle line:90% some kind of a risk perception. 01:55:09.360 --> 01:55:12.900 align:middle line:90% 01:55:12.900 --> 01:55:15.940 align:middle line:84% This has a little bit of a unit issue, I think. 01:55:15.940 --> 01:55:17.520 align:middle line:90% So flaws per feature-- 01:55:17.520 --> 01:55:20.748 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:55:20.748 --> 01:55:22.700 align:middle line:90% 01:55:22.700 --> 01:55:26.340 align:middle line:84% HAZHIR RAHMANDAD: Yeah, it is dimensionless, but it's a scope. 01:55:26.340 --> 01:55:32.460 align:middle line:84% Let's say you have 1,000 features and you have 10 flaws 01:55:32.460 --> 01:55:34.740 align:middle line:90% in those 1,000 features. 01:55:34.740 --> 01:55:37.632 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:55:37.632 --> 01:55:41.980 align:middle line:90% 01:55:41.980 --> 01:55:52.220 align:middle line:84% HAZHIR RAHMANDAD: At that point in time, we are-- so that's 1%. 01:55:52.220 --> 01:55:53.600 align:middle line:90% And we are adopting. 01:55:53.600 --> 01:55:57.500 align:middle line:84% And these adopters have 1% flawed features. 01:55:57.500 --> 01:56:04.300 align:middle line:84% So adopters with flaws are bugginess-- 01:56:04.300 --> 01:56:10.860 align:middle line:84% average bugginess multiplied by how many adopters you have. 01:56:10.860 --> 01:56:18.740 align:middle line:84% You divide that by adopters, you get your average bugginess. 01:56:18.740 --> 01:56:20.540 align:middle line:90% Yeah, that works. 01:56:20.540 --> 01:56:22.105 align:middle line:90% That works, except-- 01:56:22.105 --> 01:56:22.980 align:middle line:90% AUDIENCE: [INAUDIBLE] 01:56:22.980 --> 01:56:25.440 align:middle line:84% HAZHIR RAHMANDAD: Yeah, because it doesn't drain this one. 01:56:25.440 --> 01:56:30.900 align:middle line:84% I mean, death rate-- let's for now put it aside. 01:56:30.900 --> 01:56:34.820 align:middle line:84% Because it doesn't drain, this is very accumulative. 01:56:34.820 --> 01:56:36.920 align:middle line:84% It is not the recent risk perception. 01:56:36.920 --> 01:56:43.460 align:middle line:84% This is the longer, over the whole course of this product, 01:56:43.460 --> 01:56:44.700 align:middle line:90% risk perception. 01:56:44.700 --> 01:56:49.880 align:middle line:84% So I think, conceptually, it looks OK, but with that caveat. 01:56:49.880 --> 01:56:56.820 align:middle line:84% So I would probably have what is the time horizon of performance 01:56:56.820 --> 01:57:02.300 align:middle line:84% that impacts new customers' decision or market reputation. 01:57:02.300 --> 01:57:04.180 align:middle line:84% And that time horizon, after that, 01:57:04.180 --> 01:57:07.740 align:middle line:84% I'll just drain people or these adopters 01:57:07.740 --> 01:57:09.400 align:middle line:90% with flaws from this stock. 01:57:09.400 --> 01:57:11.560 align:middle line:84% And I'll send the adopters into a different stock. 01:57:11.560 --> 01:57:15.540 align:middle line:84% These are people who we are tracking 01:57:15.540 --> 01:57:17.778 align:middle line:84% because their word of mouth matters 01:57:17.778 --> 01:57:19.320 align:middle line:84% because they are the recent adopters. 01:57:19.320 --> 01:57:21.660 align:middle line:84% And then they go into these old adopters 01:57:21.660 --> 01:57:23.800 align:middle line:84% that you don't need to track anymore. 01:57:23.800 --> 01:57:26.660 align:middle line:90% 01:57:26.660 --> 01:57:31.280 align:middle line:84% And then that would create your indicated adopters. 01:57:31.280 --> 01:57:34.340 align:middle line:90% 01:57:34.340 --> 01:57:36.060 align:middle line:84% And indicated adopters, therefore, 01:57:36.060 --> 01:57:41.500 align:middle line:84% is having these two effects together. 01:57:41.500 --> 01:57:47.740 align:middle line:84% And that gap between indicated adopters and total population 01:57:47.740 --> 01:57:50.900 align:middle line:84% drives adoption rate with the time to adopt. 01:57:50.900 --> 01:57:57.680 align:middle line:84% That is affected by fraction of adopters in total market. 01:57:57.680 --> 01:58:00.040 align:middle line:90% So again, this is kind of OK. 01:58:00.040 --> 01:58:03.400 align:middle line:84% But because it is coming from the total adopters, 01:58:03.400 --> 01:58:06.480 align:middle line:84% the old adopters have the same effect as the new adopters. 01:58:06.480 --> 01:58:07.240 align:middle line:90% But that's OK. 01:58:07.240 --> 01:58:11.400 align:middle line:90% It's not a big problem there. 01:58:11.400 --> 01:58:17.020 align:middle line:84% And then you have a couple of parameters going into this. 01:58:17.020 --> 01:58:20.300 align:middle line:90% So this is-- 01:58:20.300 --> 01:58:24.100 align:middle line:84% AUDIENCE: It's just a reverse sigmoid. 01:58:24.100 --> 01:58:28.680 align:middle line:84% And then that max reduction says that we can't take time to 0. 01:58:28.680 --> 01:58:33.080 align:middle line:84% So I think I have it set at the max reduction is 0.75, 01:58:33.080 --> 01:58:37.220 align:middle line:84% so the best we can do is a quarter of the baseline time. 01:58:37.220 --> 01:58:39.120 align:middle line:84% HAZHIR RAHMANDAD: OK, that's all right. 01:58:39.120 --> 01:58:42.640 align:middle line:84% I probably would have-- these nonlinear curves, 01:58:42.640 --> 01:58:44.960 align:middle line:84% then they go into a time constant. 01:58:44.960 --> 01:58:47.080 align:middle line:84% And the time constant is in the denominator. 01:58:47.080 --> 01:58:49.260 align:middle line:90% So it adds another nonlinearity. 01:58:49.260 --> 01:58:51.140 align:middle line:84% The accumulation of these nonlinearities 01:58:51.140 --> 01:58:56.020 align:middle line:84% make it very hard to really mentally track what is happening 01:58:56.020 --> 01:58:59.980 align:middle line:84% and what is the impact of ka or max reduction in time to adapt 01:58:59.980 --> 01:59:02.420 align:middle line:84% or something on the rest of the dynamics. 01:59:02.420 --> 01:59:05.760 align:middle line:84% So my preference is for something simpler, 01:59:05.760 --> 01:59:08.700 align:middle line:84% like a linear function for the time constant 01:59:08.700 --> 01:59:10.980 align:middle line:84% that goes between a minimum and a maximum. 01:59:10.980 --> 01:59:14.300 align:middle line:84% And to get that, you would need a normalized fraction 01:59:14.300 --> 01:59:15.720 align:middle line:90% of adopters in the market. 01:59:15.720 --> 01:59:18.820 align:middle line:84% And it can easily do that for you. 01:59:18.820 --> 01:59:19.640 align:middle line:90% This is not wrong. 01:59:19.640 --> 01:59:23.580 align:middle line:84% It's just a little bit harder to track. 01:59:23.580 --> 01:59:29.700 align:middle line:84% OK, I think-- yeah, pretty much everything 01:59:29.700 --> 01:59:33.380 align:middle line:84% is making sense with a few ideas here and there 01:59:33.380 --> 01:59:36.860 align:middle line:90% that could be improved. 01:59:36.860 --> 01:59:39.200 align:middle line:84% OK, which models haven't we looked at? 01:59:39.200 --> 01:59:42.580 align:middle line:90% So this is Jason's we looked at. 01:59:42.580 --> 01:59:45.040 align:middle line:90% This is Gerardo's. 01:59:45.040 --> 01:59:48.180 align:middle line:90% Let's look at Gerardo's model. 01:59:48.180 --> 01:59:54.540 align:middle line:84% So we have quality issues, unknown quality issues, 01:59:54.540 --> 01:59:56.780 align:middle line:90% unfixed quality issues, fixed. 01:59:56.780 --> 02:00:00.500 align:middle line:90% 02:00:00.500 --> 02:00:03.380 align:middle line:84% Commercially available scope of use, 02:00:03.380 --> 02:00:10.500 align:middle line:84% that is coming from feasible scope of use discovered-- 02:00:10.500 --> 02:00:15.455 align:middle line:84% what is feasible scope of use discovered? 02:00:15.455 --> 02:00:19.340 align:middle line:90% AUDIENCE: [INAUDIBLE] 02:00:19.340 --> 02:00:23.060 align:middle line:84% HAZHIR RAHMANDAD: OK, so that's the maximum. 02:00:23.060 --> 02:00:25.260 align:middle line:84% Then you have available R&D expenditure 02:00:25.260 --> 02:00:29.460 align:middle line:90% for technology development. 02:00:29.460 --> 02:00:33.220 align:middle line:84% And then you have the actual R&D expenditure. 02:00:33.220 --> 02:00:38.380 align:middle line:84% So, Gerardo, could you tell us what 02:00:38.380 --> 02:00:42.940 align:middle line:84% the available R&D expenditure versus R&D 02:00:42.940 --> 02:00:44.880 align:middle line:90% expenditure do for you here? 02:00:44.880 --> 02:00:48.460 align:middle line:90% What is the story? 02:00:48.460 --> 02:00:53.840 align:middle line:84% AUDIENCE: [INAUDIBLE] how much improved [INAUDIBLE]. 02:00:53.840 --> 02:01:01.580 align:middle line:90% 02:01:01.580 --> 02:01:03.560 align:middle line:84% HAZHIR RAHMANDAD: Yeah, I'm just not following 02:01:03.560 --> 02:01:05.540 align:middle line:84% what's the average R&D expenditure 02:01:05.540 --> 02:01:07.240 align:middle line:90% for technological advancement. 02:01:07.240 --> 02:01:10.940 align:middle line:90% What is it really doing? 02:01:10.940 --> 02:01:16.180 align:middle line:84% So it's maximum of 0 and feasible scope of use discovered 02:01:16.180 --> 02:01:22.100 align:middle line:84% minus commercially available scope of use divided-- 02:01:22.100 --> 02:01:25.460 align:middle line:84% so you don't need this max 0 because it should turn off 02:01:25.460 --> 02:01:30.580 align:middle line:84% the scope development anyway-- divided 02:01:30.580 --> 02:01:36.080 align:middle line:84% by average R&D expenditure divided by R&D expenditure. 02:01:36.080 --> 02:01:40.300 align:middle line:90% 02:01:40.300 --> 02:01:48.300 align:middle line:84% AUDIENCE: So the [INAUDIBLE] balance 02:01:48.300 --> 02:01:53.340 align:middle line:84% between the feasible [INAUDIBLE]. 02:01:53.340 --> 02:01:57.236 align:middle line:90% 02:01:57.236 --> 02:02:03.220 align:middle line:84% But the other one would be [INAUDIBLE]. 02:02:03.220 --> 02:02:07.960 align:middle line:90% 02:02:07.960 --> 02:02:19.460 align:middle line:84% And basically, between these two, [INAUDIBLE], 02:02:19.460 --> 02:02:21.092 align:middle line:84% but probably simulation is wrong. 02:02:21.092 --> 02:02:23.300 align:middle line:84% HAZHIR RAHMANDAD: Yeah, so I think the formulation is 02:02:23.300 --> 02:02:25.300 align:middle line:90% at least too complex. 02:02:25.300 --> 02:02:26.720 align:middle line:90% So it's not intuitive. 02:02:26.720 --> 02:02:29.940 align:middle line:84% You have, in the denominator, something again divided 02:02:29.940 --> 02:02:30.720 align:middle line:90% by something. 02:02:30.720 --> 02:02:34.372 align:middle line:90% So this part makes me-- 02:02:34.372 --> 02:02:38.900 align:middle line:84% it just makes it hard for me to track what is happening. 02:02:38.900 --> 02:02:43.580 align:middle line:84% I want just the time constant for these types of adjustments. 02:02:43.580 --> 02:02:52.000 align:middle line:84% Now, these two parameters going there, one of them is changing. 02:02:52.000 --> 02:02:53.760 align:middle line:90% So R&D expenditure is changing. 02:02:53.760 --> 02:02:55.940 align:middle line:90% So I don't even-- 02:02:55.940 --> 02:02:58.540 align:middle line:84% anyway, so this part is a little bit sketchy, 02:02:58.540 --> 02:03:04.080 align:middle line:84% in terms of getting everything easily traceable. 02:03:04.080 --> 02:03:05.600 align:middle line:90% It may work just fine. 02:03:05.600 --> 02:03:09.500 align:middle line:84% I think all that it is doing is creating 02:03:09.500 --> 02:03:12.020 align:middle line:84% a curve for technological advancement rate 02:03:12.020 --> 02:03:14.700 align:middle line:84% that slows down over time and perhaps 02:03:14.700 --> 02:03:17.400 align:middle line:84% maybe a little bit of S-shape, maybe not. 02:03:17.400 --> 02:03:18.920 align:middle line:84% It doesn't have S-shape, I think. 02:03:18.920 --> 02:03:22.420 align:middle line:90% 02:03:22.420 --> 02:03:23.825 align:middle line:90% OK. 02:03:23.825 --> 02:03:25.220 align:middle line:90% AUDIENCE: [INAUDIBLE] 02:03:25.220 --> 02:03:26.613 align:middle line:90% HAZHIR RAHMANDAD: What's that? 02:03:26.613 --> 02:03:28.280 align:middle line:84% AUDIENCE: [INAUDIBLE] shape [INAUDIBLE], 02:03:28.280 --> 02:03:31.262 align:middle line:90% but nothing really [INAUDIBLE]. 02:03:31.262 --> 02:03:32.720 align:middle line:84% HAZHIR RAHMANDAD: So if you run it, 02:03:32.720 --> 02:03:38.620 align:middle line:84% it does create an S-shape or nothing happens. 02:03:38.620 --> 02:03:41.780 align:middle line:84% And so right now, it is just doing a linear thing. 02:03:41.780 --> 02:03:43.620 align:middle line:90% I don't know what is that. 02:03:43.620 --> 02:03:44.620 align:middle line:90% AUDIENCE: [INAUDIBLE] 02:03:44.620 --> 02:03:45.870 align:middle line:90% HAZHIR RAHMANDAD: Oh no, yeah. 02:03:45.870 --> 02:03:48.180 align:middle line:90% 02:03:48.180 --> 02:03:49.740 align:middle line:90% OK. 02:03:49.740 --> 02:03:52.620 align:middle line:84% Anyway, I think this can benefit from something 02:03:52.620 --> 02:03:54.000 align:middle line:90% more standard formulation. 02:03:54.000 --> 02:03:56.380 align:middle line:84% I think the concept of average R&D expenditure 02:03:56.380 --> 02:03:58.500 align:middle line:84% for technological advancement, in the unit 02:03:58.500 --> 02:04:03.660 align:middle line:84% of dollar, that doesn't make much sense because it 02:04:03.660 --> 02:04:05.760 align:middle line:90% doesn't sound like expenditure. 02:04:05.760 --> 02:04:08.500 align:middle line:84% Expenditure is dollars per unit of time. 02:04:08.500 --> 02:04:10.740 align:middle line:90% This looks a pot of money. 02:04:10.740 --> 02:04:16.830 align:middle line:84% But what that pot of money is doing is not clear to me. 02:04:16.830 --> 02:04:22.060 align:middle line:84% AUDIENCE: In my mind, the money [INAUDIBLE]. 02:04:22.060 --> 02:04:25.020 align:middle line:84% HAZHIR RAHMANDAD: OK, this is per unit of technology. 02:04:25.020 --> 02:04:29.980 align:middle line:84% So this would be per task or per unit of scope. 02:04:29.980 --> 02:04:31.820 align:middle line:84% Or this is for total scope, how much 02:04:31.820 --> 02:04:34.860 align:middle line:84% you would be needing to get to the full scope 02:04:34.860 --> 02:04:37.980 align:middle line:90% or something like that. 02:04:37.980 --> 02:04:46.300 align:middle line:84% Then you have the scope that, combined with perceived quality, 02:04:46.300 --> 02:04:53.160 align:middle line:84% impacts scope of use-- qualitatively valid. 02:04:53.160 --> 02:04:58.500 align:middle line:84% So that's similar to what Kathy also had. 02:04:58.500 --> 02:05:04.140 align:middle line:84% With some delay, it is perceived that utility then impacts 02:05:04.140 --> 02:05:08.900 align:middle line:90% your Bass diffusion model. 02:05:08.900 --> 02:05:13.880 align:middle line:84% And it is just a multiplier for that Bass diffusion model. 02:05:13.880 --> 02:05:18.100 align:middle line:84% But this commercially available scope is between 0 and 1. 02:05:18.100 --> 02:05:20.880 align:middle line:84% Perceived quality, I assume, is between 0 and 1. 02:05:20.880 --> 02:05:26.300 align:middle line:84% So in all cases, the perceived utility is above 0. 02:05:26.300 --> 02:05:29.260 align:middle line:84% And if perceived utility is above 0, 02:05:29.260 --> 02:05:31.220 align:middle line:84% Bass diffusion model will ultimately 02:05:31.220 --> 02:05:33.860 align:middle line:90% get everybody to adopt-- 02:05:33.860 --> 02:05:38.340 align:middle line:84% well, assuming you have some initial adopters. 02:05:38.340 --> 02:05:45.080 align:middle line:84% So you wouldn't have a case where any customers return-- 02:05:45.080 --> 02:05:50.500 align:middle line:84% that is, they decide not to adopt it after initial adoption. 02:05:50.500 --> 02:05:53.740 align:middle line:84% Now, again, with the pacemaker example, 02:05:53.740 --> 02:05:59.860 align:middle line:84% it's understandable to not have people go backwards. 02:05:59.860 --> 02:06:01.540 align:middle line:84% But either way, the Bass diffusion 02:06:01.540 --> 02:06:04.000 align:middle line:90% is a very strong exponential. 02:06:04.000 --> 02:06:06.380 align:middle line:84% It has a strong exponential component. 02:06:06.380 --> 02:06:10.780 align:middle line:84% So for this perceived utility not to turn off 02:06:10.780 --> 02:06:15.260 align:middle line:84% the spigot of adoption, it needs to really go to 0. 02:06:15.260 --> 02:06:17.140 align:middle line:84% And for it to really go to 0, you 02:06:17.140 --> 02:06:20.460 align:middle line:84% need the perception of quality to go to 0 because the scope is 02:06:20.460 --> 02:06:23.020 align:middle line:90% always above 0. 02:06:23.020 --> 02:06:29.340 align:middle line:84% So then the quality perception needs to be well-formulated. 02:06:29.340 --> 02:06:32.500 align:middle line:84% Here, the quality perception is coming from-- 02:06:32.500 --> 02:06:34.900 align:middle line:84% well, not only well-formulated, but also be 02:06:34.900 --> 02:06:40.780 align:middle line:84% prone to putting numbers close to 0. 02:06:40.780 --> 02:06:43.540 align:middle line:84% This is, right now, undiscovered quality issues 02:06:43.540 --> 02:06:46.420 align:middle line:84% that are discovered quality issues that are unfixed divided 02:06:46.420 --> 02:06:48.100 align:middle line:90% by fixed quality issues. 02:06:48.100 --> 02:06:49.980 align:middle line:84% This is not a very good formulation here 02:06:49.980 --> 02:06:53.820 align:middle line:84% because, as we discussed, fixed quality issues in the past 02:06:53.820 --> 02:06:56.500 align:middle line:84% are not a good normalizing factor 02:06:56.500 --> 02:06:59.760 align:middle line:84% for the current discovered issues. 02:06:59.760 --> 02:07:03.560 align:middle line:84% So think about a project that is in its first year. 02:07:03.560 --> 02:07:08.060 align:middle line:84% So there is no fixed quality issues in the denominator. 02:07:08.060 --> 02:07:11.500 align:middle line:84% This unfixed ratio becomes a huge number. 02:07:11.500 --> 02:07:15.300 align:middle line:84% Now, if you divide it by discovered plus fixed, 02:07:15.300 --> 02:07:17.900 align:middle line:90% then it is bounded. 02:07:17.900 --> 02:07:23.180 align:middle line:90% So it would go to 1. 02:07:23.180 --> 02:07:26.880 align:middle line:84% But either way, it is not very attractive. 02:07:26.880 --> 02:07:28.500 align:middle line:84% But then if you have a product that 02:07:28.500 --> 02:07:32.100 align:middle line:84% has been around for 40 years, the fixed quality issues 02:07:32.100 --> 02:07:33.060 align:middle line:90% are in hundreds. 02:07:33.060 --> 02:07:35.420 align:middle line:84% And you may have just a couple here. 02:07:35.420 --> 02:07:38.880 align:middle line:84% That ratio doesn't really scale correctly. 02:07:38.880 --> 02:07:43.840 align:middle line:84% So this needs to be normalized against something else. 02:07:43.840 --> 02:07:47.780 align:middle line:84% Now, the something else here is perhaps the scope. 02:07:47.780 --> 02:07:52.500 align:middle line:84% And fractional technological issue 02:07:52.500 --> 02:07:55.860 align:middle line:84% that is being added with each technological advancement rate 02:07:55.860 --> 02:07:57.680 align:middle line:90% could take care of that. 02:07:57.680 --> 02:08:01.020 align:middle line:90% 02:08:01.020 --> 02:08:05.040 align:middle line:84% So the numbers around here are these very small numbers. 02:08:05.040 --> 02:08:11.060 align:middle line:84% So it's 1,000th as the flow variable. 02:08:11.060 --> 02:08:14.900 align:middle line:84% So the perceived quality, after that very early phase, 02:08:14.900 --> 02:08:17.700 align:middle line:90% will become so small-- 02:08:17.700 --> 02:08:23.340 align:middle line:84% so high because the unfixed ratio is close to 0 02:08:23.340 --> 02:08:31.660 align:middle line:84% that, yeah, the effect of quality on all of the dynamics 02:08:31.660 --> 02:08:32.560 align:middle line:90% goes away. 02:08:32.560 --> 02:08:34.480 align:middle line:84% It just wouldn't have much of an effect. 02:08:34.480 --> 02:08:37.160 align:middle line:84% That would be my guess, after the very beginning. 02:08:37.160 --> 02:08:38.960 align:middle line:90% So oops. 02:08:38.960 --> 02:08:42.900 align:middle line:90% 02:08:42.900 --> 02:08:46.280 align:middle line:84% Well, in this case, actually, it does have an effect. 02:08:46.280 --> 02:08:50.820 align:middle line:84% So the sensitivity parameter is strong enough. 02:08:50.820 --> 02:08:51.780 align:middle line:90% AUDIENCE: [INAUDIBLE] 02:08:51.780 --> 02:08:52.940 align:middle line:90% HAZHIR RAHMANDAD: Yeah. 02:08:52.940 --> 02:08:55.820 align:middle line:84% I mean, maybe it is because the unfixed ratio 02:08:55.820 --> 02:08:57.820 align:middle line:90% is changing significantly. 02:08:57.820 --> 02:09:04.820 align:middle line:84% Yes, the unfixed ratio is changing. 02:09:04.820 --> 02:09:07.580 align:middle line:84% Oh, six months of simulation for something 02:09:07.580 --> 02:09:11.820 align:middle line:84% that should go 40 years is also not enough, yeah. 02:09:11.820 --> 02:09:18.580 align:middle line:84% OK, good I think all of these have-- 02:09:18.580 --> 02:09:22.480 align:middle line:84% so we have seen the key issues around it. 02:09:22.480 --> 02:09:23.980 align:middle line:90% Let's see. 02:09:23.980 --> 02:09:26.240 align:middle line:90% Which model did we not discuss? 02:09:26.240 --> 02:09:28.900 align:middle line:90% 02:09:28.900 --> 02:09:30.980 align:middle line:90% Hesam's we discussed. 02:09:30.980 --> 02:09:34.880 align:middle line:84% Sercin's OK, let's take a look at yours. 02:09:34.880 --> 02:09:42.620 align:middle line:84% So scope is changing with R&D investment, effectiveness 02:09:42.620 --> 02:09:44.140 align:middle line:90% of R&D investment. 02:09:44.140 --> 02:09:50.360 align:middle line:84% So this average time to develop new apps is less clear to me 02:09:50.360 --> 02:09:53.460 align:middle line:84% why do we need that because R&D investment should be in dollars 02:09:53.460 --> 02:09:55.220 align:middle line:90% per month or whatever. 02:09:55.220 --> 02:09:58.000 align:middle line:84% And then effectiveness would be features per dollar. 02:09:58.000 --> 02:10:02.700 align:middle line:84% So this dividing by average time to develop 02:10:02.700 --> 02:10:05.260 align:middle line:84% suggests to me that your R&D investment is not 02:10:05.260 --> 02:10:06.320 align:middle line:90% in the right unit. 02:10:06.320 --> 02:10:07.560 align:middle line:90% Yes, it is in dollars. 02:10:07.560 --> 02:10:11.340 align:middle line:84% It should be in dollars per unit of time. 02:10:11.340 --> 02:10:13.460 align:middle line:90% So this is important. 02:10:13.460 --> 02:10:19.460 align:middle line:84% Conceptually, investment is floating. 02:10:19.460 --> 02:10:23.060 align:middle line:84% And then you're getting the investment based on something 02:10:23.060 --> 02:10:26.500 align:middle line:84% that you call research gap, which 02:10:26.500 --> 02:10:31.640 align:middle line:84% come from the technological capabilities of the product. 02:10:31.640 --> 02:10:32.640 align:middle line:90% This is a fixed number. 02:10:32.640 --> 02:10:35.060 align:middle line:90% This is kind of a maximum. 02:10:35.060 --> 02:10:37.300 align:middle line:90% Is it a maximum of sorts? 02:10:37.300 --> 02:10:37.825 align:middle line:90% Yes. 02:10:37.825 --> 02:10:38.700 align:middle line:90% AUDIENCE: [INAUDIBLE] 02:10:38.700 --> 02:10:41.520 align:middle line:84% HAZHIR RAHMANDAD: Yes, maximum number of applications. 02:10:41.520 --> 02:10:46.980 align:middle line:84% So then you are looking at the gap between the maximum 02:10:46.980 --> 02:10:49.260 align:middle line:90% and the current scope. 02:10:49.260 --> 02:10:56.080 align:middle line:84% And then this R&D investment is that gap multiplied by per unit. 02:10:56.080 --> 02:10:59.200 align:middle line:84% So if you have a bigger gap, you're more willing to invest. 02:10:59.200 --> 02:11:01.420 align:middle line:90% That's the idea. 02:11:01.420 --> 02:11:13.220 align:middle line:84% And then the bigger the gap as a fraction of the total-- 02:11:13.220 --> 02:11:14.760 align:middle line:90% so this is a fraction. 02:11:14.760 --> 02:11:17.900 align:middle line:90% 02:11:17.900 --> 02:11:23.220 align:middle line:84% You just multiply to get an effectiveness. 02:11:23.220 --> 02:11:25.180 align:middle line:90% I don't know. 02:11:25.180 --> 02:11:31.780 align:middle line:84% Just multiplying for both of these are a little bit simple. 02:11:31.780 --> 02:11:36.660 align:middle line:90% 02:11:36.660 --> 02:11:42.080 align:middle line:84% If the fraction of applications that are still not realized, 02:11:42.080 --> 02:11:45.700 align:middle line:84% which is this number, is higher, you're 02:11:45.700 --> 02:11:47.260 align:middle line:84% saying effectiveness of investment 02:11:47.260 --> 02:11:50.860 align:middle line:84% is higher and linearly higher with that. 02:11:50.860 --> 02:11:53.640 align:middle line:84% Given that this number is actually between 0 and 1, 02:11:53.640 --> 02:11:56.820 align:middle line:84% I would probably use maybe a power function rather than 02:11:56.820 --> 02:12:01.140 align:middle line:84% just a multiplication of a linear. 02:12:01.140 --> 02:12:01.640 align:middle line:90% OK. 02:12:01.640 --> 02:12:04.380 align:middle line:84% So anyway, all of this, what it does 02:12:04.380 --> 02:12:08.700 align:middle line:84% is that it speeds up initial change in scope 02:12:08.700 --> 02:12:10.380 align:middle line:84% because now you're investing more. 02:12:10.380 --> 02:12:13.080 align:middle line:84% And there is more return per unit of investment. 02:12:13.080 --> 02:12:19.000 align:middle line:84% And as scope grows, that starts to become smaller and smaller. 02:12:19.000 --> 02:12:22.820 align:middle line:84% The extra development gets smaller. 02:12:22.820 --> 02:12:28.340 align:middle line:84% So it just creates a goal-seeking behavior 02:12:28.340 --> 02:12:30.800 align:middle line:90% for the scope, I guess. 02:12:30.800 --> 02:12:41.520 align:middle line:90% 02:12:41.520 --> 02:12:48.140 align:middle line:84% And then scope goes to fraction of population to be served. 02:12:48.140 --> 02:12:51.460 align:middle line:84% So here, you have this sensitivity parameter 02:12:51.460 --> 02:12:54.600 align:middle line:84% for effect of scope on fraction of population served. 02:12:54.600 --> 02:12:57.500 align:middle line:90% 02:12:57.500 --> 02:13:02.260 align:middle line:84% And it is normalized by that technological-- 02:13:02.260 --> 02:13:04.600 align:middle line:90% sorry, technological capability. 02:13:04.600 --> 02:13:05.520 align:middle line:90% So this is good. 02:13:05.520 --> 02:13:08.640 align:middle line:84% This will take care of the normalization. 02:13:08.640 --> 02:13:14.420 align:middle line:84% And it will always be remaining between 0 and 1. 02:13:14.420 --> 02:13:19.960 align:middle line:84% And then you have this decreasing return on the scope 02:13:19.960 --> 02:13:25.420 align:middle line:84% or if you want to have that in the sensitivity parameter. 02:13:25.420 --> 02:13:29.500 align:middle line:84% This gives the fraction of population. 02:13:29.500 --> 02:13:31.780 align:middle line:84% And that fraction is partly dependent 02:13:31.780 --> 02:13:38.340 align:middle line:84% on technological capabilities, so max fraction that can get it. 02:13:38.340 --> 02:13:45.700 align:middle line:84% And then it gives you the max user base. 02:13:45.700 --> 02:13:49.340 align:middle line:84% And you move towards that max user base based on a time 02:13:49.340 --> 02:13:54.980 align:middle line:84% to adopt, which depends on the utility. 02:13:54.980 --> 02:14:01.300 align:middle line:84% So the formulation here is reference utility 02:14:01.300 --> 02:14:04.480 align:middle line:84% divided by perceived utility raised to some power. 02:14:04.480 --> 02:14:09.580 align:middle line:84% So if perceived utility is smaller than reference utility, 02:14:09.580 --> 02:14:12.620 align:middle line:84% then you adopt more slowly because your time 02:14:12.620 --> 02:14:14.820 align:middle line:90% constant increases. 02:14:14.820 --> 02:14:17.535 align:middle line:90% OK. 02:14:17.535 --> 02:14:18.520 align:middle line:90% That makes sense. 02:14:18.520 --> 02:14:26.300 align:middle line:84% And then perceived utility is coming from this indicated value 02:14:26.300 --> 02:14:33.900 align:middle line:84% that itself comes from problems and usefulness. 02:14:33.900 --> 02:14:39.140 align:middle line:84% Problems are recognized and not fixed problems. 02:14:39.140 --> 02:14:42.660 align:middle line:84% It is normalized against some reference number, which perhaps 02:14:42.660 --> 02:14:49.740 align:middle line:84% should be normalized against either total features 02:14:49.740 --> 02:14:55.580 align:middle line:84% or some measure of recent features multiplied 02:14:55.580 --> 02:15:01.900 align:middle line:84% by this number of problems per unit of application. 02:15:01.900 --> 02:15:04.500 align:middle line:90% So right now, this recognized-- 02:15:04.500 --> 02:15:07.740 align:middle line:84% reference number of recognized problems 02:15:07.740 --> 02:15:10.300 align:middle line:84% is just an ad hoc number that you have put. 02:15:10.300 --> 02:15:13.620 align:middle line:84% It is not rooted in any of those other parameters that 02:15:13.620 --> 02:15:15.320 align:middle line:90% determine the scope of it. 02:15:15.320 --> 02:15:18.200 align:middle line:84% And that creates a problem downstream 02:15:18.200 --> 02:15:20.160 align:middle line:84% because then you have to remember, 02:15:20.160 --> 02:15:21.960 align:middle line:84% whenever I change any of these numbers, 02:15:21.960 --> 02:15:23.760 align:middle line:84% come and change this number accordingly. 02:15:23.760 --> 02:15:26.260 align:middle line:90% You don't want that. 02:15:26.260 --> 02:15:30.700 align:middle line:84% But otherwise, the core ideas are good. 02:15:30.700 --> 02:15:34.180 align:middle line:84% Effective usefulness on utility-- 02:15:34.180 --> 02:15:38.280 align:middle line:84% so you already calculated that from the scope. 02:15:38.280 --> 02:15:41.980 align:middle line:90% You got the average usefulness. 02:15:41.980 --> 02:15:44.580 align:middle line:84% Wait, how do you do this total usefulness? 02:15:44.580 --> 02:15:45.080 align:middle line:90% OK. 02:15:45.080 --> 02:15:51.260 align:middle line:90% 02:15:51.260 --> 02:15:54.940 align:middle line:84% So you create a marginal usefulness. 02:15:54.940 --> 02:15:58.200 align:middle line:84% You get to total usefulness of product. 02:15:58.200 --> 02:16:00.820 align:middle line:90% 02:16:00.820 --> 02:16:08.780 align:middle line:84% And then you say per feature what 02:16:08.780 --> 02:16:16.700 align:middle line:84% is unit of usefulness per unit of a scope. 02:16:16.700 --> 02:16:19.740 align:middle line:84% So this marginal calculation is very 02:16:19.740 --> 02:16:22.220 align:middle line:84% similar to Kathy's calculation of marginals 02:16:22.220 --> 02:16:24.060 align:middle line:90% to get to a utility. 02:16:24.060 --> 02:16:26.300 align:middle line:84% This is basically looking at change in scope, 02:16:26.300 --> 02:16:29.160 align:middle line:84% multiplying that by marginal utility. 02:16:29.160 --> 02:16:33.020 align:middle line:84% So that makes sense, in terms of marginals. 02:16:33.020 --> 02:16:35.299 align:middle line:84% But then you get this total usefulness 02:16:35.299 --> 02:16:42.340 align:middle line:84% of product, which is the utility of the scope that we are seeing. 02:16:42.340 --> 02:16:45.660 align:middle line:84% And then you calculate it as average usefulness 02:16:45.660 --> 02:16:49.660 align:middle line:90% per application. 02:16:49.660 --> 02:16:53.780 align:middle line:90% 02:16:53.780 --> 02:16:57.459 align:middle line:84% So as we increase the scope, the average usefulness 02:16:57.459 --> 02:16:59.559 align:middle line:90% per application goes down. 02:16:59.559 --> 02:17:02.379 align:middle line:90% 02:17:02.379 --> 02:17:09.059 align:middle line:84% And then that makes effective usefulness on utility 02:17:09.059 --> 02:17:10.680 align:middle line:90% to go down. 02:17:10.680 --> 02:17:17.860 align:middle line:90% 02:17:17.860 --> 02:17:19.040 align:middle line:90% That's OK. 02:17:19.040 --> 02:17:21.299 align:middle line:84% But again, you have the max usefulness of product 02:17:21.299 --> 02:17:24.059 align:middle line:84% that is hanging in there without having-- 02:17:24.059 --> 02:17:27.500 align:middle line:84% it's a very similar issue as what we saw in Kathy's model, 02:17:27.500 --> 02:17:32.020 align:middle line:84% that you do need some max that is rooted in how 02:17:32.020 --> 02:17:35.780 align:middle line:90% far this usefulness can go up. 02:17:35.780 --> 02:17:38.940 align:middle line:84% But if you do that, then it should work. 02:17:38.940 --> 02:17:41.260 align:middle line:90% Cool? 02:17:41.260 --> 02:17:41.799 align:middle line:90% OK. 02:17:41.799 --> 02:17:44.700 align:middle line:84% I think maybe we have only one more model 02:17:44.700 --> 02:17:47.700 align:middle line:90% that we haven't looked at. 02:17:47.700 --> 02:17:49.280 align:middle line:90% It says Alban's model. 02:17:49.280 --> 02:17:51.540 align:middle line:84% So let's do a quick look at Alban's model, 02:17:51.540 --> 02:17:55.820 align:middle line:84% and then go for the next week's assignment. 02:17:55.820 --> 02:18:00.700 align:middle line:90% So we have scope. 02:18:00.700 --> 02:18:04.940 align:middle line:84% Expansion of scope is coming from R&D investment 02:18:04.940 --> 02:18:06.920 align:middle line:90% and investment productivity. 02:18:06.920 --> 02:18:13.940 align:middle line:84% As you build more scope, your opportunities to build more 02:18:13.940 --> 02:18:16.139 align:middle line:90% goes down, I assume. 02:18:16.139 --> 02:18:23.020 align:middle line:84% So this is balancing the return on investment from the R&D. 02:18:23.020 --> 02:18:25.580 align:middle line:90% That makes sense. 02:18:25.580 --> 02:18:33.139 align:middle line:84% Then this scope is having a coflow with the utility. 02:18:33.139 --> 02:18:36.500 align:middle line:84% And that is affected by this very similar idea 02:18:36.500 --> 02:18:41.420 align:middle line:90% of marginal utility. 02:18:41.420 --> 02:18:48.379 align:middle line:84% But you are getting it from these remaining opportunities 02:18:48.379 --> 02:18:49.719 align:middle line:90% to expand the scope. 02:18:49.719 --> 02:18:53.500 align:middle line:84% So they bring down the average utility 02:18:53.500 --> 02:18:57.160 align:middle line:84% of a new piece of feature or a new-- 02:18:57.160 --> 02:18:59.900 align:middle line:90% 02:18:59.900 --> 02:19:01.719 align:middle line:84% yeah, these are all dimensionless, 02:19:01.719 --> 02:19:04.400 align:middle line:84% which makes it a little bit harder, as we discussed. 02:19:04.400 --> 02:19:07.920 align:middle line:90% But it would be OK, potentially. 02:19:07.920 --> 02:19:19.820 align:middle line:84% So here, this is the decreasing return on utility of new pieces 02:19:19.820 --> 02:19:22.420 align:middle line:90% that go into the total scope. 02:19:22.420 --> 02:19:25.480 align:middle line:84% Then we get the average utility of the product. 02:19:25.480 --> 02:19:30.980 align:middle line:84% This is per feature what is the average utility. 02:19:30.980 --> 02:19:37.733 align:middle line:84% So the indicated utility is actually going down over time? 02:19:37.733 --> 02:19:38.400 align:middle line:90% AUDIENCE: Right. 02:19:38.400 --> 02:19:41.226 align:middle line:90% 02:19:41.226 --> 02:19:43.059 align:middle line:84% HAZHIR RAHMANDAD: And then perceived utility 02:19:43.059 --> 02:19:46.580 align:middle line:84% determines the fraction of population willing to adopt. 02:19:46.580 --> 02:19:51.960 align:middle line:84% So this part is problematic, in the sense that, over time, 02:19:51.960 --> 02:19:54.340 align:middle line:84% your average utility of product should go down 02:19:54.340 --> 02:19:55.280 align:middle line:90% in this formulation. 02:19:55.280 --> 02:19:56.700 align:middle line:90% Is that correct? 02:19:56.700 --> 02:19:59.460 align:middle line:90% Or we could simulate it and see. 02:19:59.460 --> 02:20:04.140 align:middle line:90% Let's not just assume. 02:20:04.140 --> 02:20:08.100 align:middle line:84% So yeah, perceived utility initially goes down 02:20:08.100 --> 02:20:10.720 align:middle line:84% because the average utility of product is going down. 02:20:10.720 --> 02:20:14.620 align:middle line:84% That part is-- I think it's baked into the formulations 02:20:14.620 --> 02:20:16.100 align:middle line:90% you have. 02:20:16.100 --> 02:20:18.340 align:middle line:84% And the way it is baked into it is 02:20:18.340 --> 02:20:23.200 align:middle line:84% because you're dividing the total utility by scope. 02:20:23.200 --> 02:20:27.380 align:middle line:84% And because the marginal utility is going down, 02:20:27.380 --> 02:20:31.460 align:middle line:84% then the average utility will also go down. 02:20:31.460 --> 02:20:35.020 align:middle line:84% And average utility is relevant for some use cases 02:20:35.020 --> 02:20:36.560 align:middle line:90% or some parts of the model. 02:20:36.560 --> 02:20:38.740 align:middle line:84% But it is not the factor that is determining 02:20:38.740 --> 02:20:46.260 align:middle line:84% the total scope of adoption or total fraction of population 02:20:46.260 --> 02:20:47.400 align:middle line:90% willing to adopt. 02:20:47.400 --> 02:20:51.380 align:middle line:90% 02:20:51.380 --> 02:20:55.820 align:middle line:84% So that's important to keep in mind. 02:20:55.820 --> 02:20:59.940 align:middle line:84% So this creates a weird behavior that, initially, the largest 02:20:59.940 --> 02:21:03.000 align:middle line:90% fraction are interested. 02:21:03.000 --> 02:21:06.600 align:middle line:84% And over time, fewer people become interested. 02:21:06.600 --> 02:21:08.900 align:middle line:84% Now, you have this additional feedback 02:21:08.900 --> 02:21:12.140 align:middle line:84% coming from the effect of bugs on utility 02:21:12.140 --> 02:21:16.360 align:middle line:84% that is very high at the beginning. 02:21:16.360 --> 02:21:18.580 align:middle line:84% We should see how that is happening. 02:21:18.580 --> 02:21:19.498 align:middle line:90% AUDIENCE: [INAUDIBLE] 02:21:19.498 --> 02:21:20.540 align:middle line:90% HAZHIR RAHMANDAD: Uh-huh? 02:21:20.540 --> 02:21:23.620 align:middle line:84% AUDIENCE: The scope determined the [INAUDIBLE]. 02:21:23.620 --> 02:21:26.600 align:middle line:90% 02:21:26.600 --> 02:21:28.980 align:middle line:84% HAZHIR RAHMANDAD: Ah, interesting, interesting. 02:21:28.980 --> 02:21:30.120 align:middle line:90% Yes, yes. 02:21:30.120 --> 02:21:34.860 align:middle line:84% So this is not this fraction of potential adopters. 02:21:34.860 --> 02:21:36.780 align:middle line:90% Got it, got it. 02:21:36.780 --> 02:21:38.380 align:middle line:90% Sorry, thank you. 02:21:38.380 --> 02:21:40.620 align:middle line:90% That clarifies. 02:21:40.620 --> 02:21:46.380 align:middle line:84% So max potential adopters is coming from the scope. 02:21:46.380 --> 02:21:49.400 align:middle line:84% So this is actually closer to the formulations 02:21:49.400 --> 02:21:51.720 align:middle line:90% we did three stocks. 02:21:51.720 --> 02:21:54.900 align:middle line:90% But you basically have created-- 02:21:54.900 --> 02:21:59.940 align:middle line:84% you have assumed that first step is very fast, which is good, 02:21:59.940 --> 02:22:01.680 align:middle line:90% which is not a bad assumption. 02:22:01.680 --> 02:22:04.040 align:middle line:84% So you get the max potential adopters from a scope. 02:22:04.040 --> 02:22:06.740 align:middle line:84% And then the perceived utility could go down 02:22:06.740 --> 02:22:14.000 align:middle line:84% because this is, given that scope, how far the utility is. 02:22:14.000 --> 02:22:16.180 align:middle line:90% OK, that's good. 02:22:16.180 --> 02:22:19.900 align:middle line:84% And this gives the indicated adopters. 02:22:19.900 --> 02:22:25.880 align:middle line:84% And then you have adopter adoption versus abandonment. 02:22:25.880 --> 02:22:29.680 align:middle line:84% And you have an asymmetric thing here. 02:22:29.680 --> 02:22:32.600 align:middle line:84% So if the indicated adopters is bigger than the adopters, 02:22:32.600 --> 02:22:35.440 align:middle line:84% then you have time to adopt, otherwise time to abandon. 02:22:35.440 --> 02:22:37.100 align:middle line:84% So that's nice because abandonment 02:22:37.100 --> 02:22:40.540 align:middle line:90% could be faster, perhaps. 02:22:40.540 --> 02:22:44.260 align:middle line:84% And then effect of adopters on adoption time 02:22:44.260 --> 02:22:46.520 align:middle line:90% is what we have discussed. 02:22:46.520 --> 02:22:47.940 align:middle line:90% So that's good. 02:22:47.940 --> 02:22:49.580 align:middle line:90% Great, I think. 02:22:49.580 --> 02:22:52.060 align:middle line:84% And then the bug part with this scope 02:22:52.060 --> 02:22:58.020 align:middle line:84% comes bugs, discovered bugs, some time constant, 02:22:58.020 --> 02:23:00.020 align:middle line:90% create the effect of bugs. 02:23:00.020 --> 02:23:03.860 align:middle line:84% Now, discovered bugs, because you're using dimensionless 02:23:03.860 --> 02:23:09.752 align:middle line:90% for the current scope-- 02:23:09.752 --> 02:23:13.140 align:middle line:90% AUDIENCE: [INAUDIBLE] 02:23:13.140 --> 02:23:16.060 align:middle line:84% HAZHIR RAHMANDAD: Yeah, this needs some normalization 02:23:16.060 --> 02:23:19.260 align:middle line:84% because, right now, you have 200 bugs per unit 02:23:19.260 --> 02:23:21.940 align:middle line:90% of scope going into here. 02:23:21.940 --> 02:23:26.080 align:middle line:84% And then you don't normalize it at all. 02:23:26.080 --> 02:23:28.980 align:middle line:84% So this definitely needs a normalization. 02:23:28.980 --> 02:23:29.500 align:middle line:90% Cool. 02:23:29.500 --> 02:23:31.720 align:middle line:90% Otherwise, it's pretty nice. 02:23:31.720 --> 02:23:32.220 align:middle line:90% OK. 02:23:32.220 --> 02:23:34.040 align:middle line:90% Any questions? 02:23:34.040 --> 02:23:38.860 align:middle line:90% 02:23:38.860 --> 02:23:45.340 align:middle line:84% All right, let's then review our model for next week. 02:23:45.340 --> 02:23:47.880 align:middle line:84% Actually, it's easier to do this way. 02:23:47.880 --> 02:23:55.700 align:middle line:90% 02:23:55.700 --> 02:23:57.020 align:middle line:90% OK. 02:23:57.020 --> 02:24:02.660 align:middle line:84% So next week's assignment is about modeling the growth 02:24:02.660 --> 02:24:05.540 align:middle line:84% of a professional field of practice. 02:24:05.540 --> 02:24:07.243 align:middle line:84% In some ways, it has relationships 02:24:07.243 --> 02:24:08.160 align:middle line:90% with this week's work. 02:24:08.160 --> 02:24:10.780 align:middle line:84% So this week, you were focusing on a product. 02:24:10.780 --> 02:24:13.880 align:middle line:84% Next week, it is a little bit perhaps more complex, 02:24:13.880 --> 02:24:15.100 align:middle line:90% perhaps not. 02:24:15.100 --> 02:24:20.580 align:middle line:84% It is more about a situation where supply is a service. 02:24:20.580 --> 02:24:22.980 align:middle line:90% So think about system dynamics. 02:24:22.980 --> 02:24:26.780 align:middle line:84% Or think about any kind of field within consulting space 02:24:26.780 --> 02:24:30.180 align:middle line:90% or within law. 02:24:30.180 --> 02:24:32.980 align:middle line:84% There are practitioners who do the job. 02:24:32.980 --> 02:24:37.020 align:middle line:84% They are going through a learning process 02:24:37.020 --> 02:24:42.460 align:middle line:84% to become professional, become experts in what they do. 02:24:42.460 --> 02:24:46.540 align:middle line:84% And as they build these skills, they 02:24:46.540 --> 02:24:54.260 align:middle line:84% become better at serving the customers for that space. 02:24:54.260 --> 02:24:57.300 align:middle line:84% They could take on projects perhaps, 02:24:57.300 --> 02:24:58.680 align:middle line:90% even when they are novices. 02:24:58.680 --> 02:25:00.780 align:middle line:84% And they could mess up those projects 02:25:00.780 --> 02:25:04.760 align:middle line:84% and create quality perception issues. 02:25:04.760 --> 02:25:09.360 align:middle line:90% 02:25:09.360 --> 02:25:12.980 align:middle line:84% So there is the quality dynamics and training dynamics. 02:25:12.980 --> 02:25:20.300 align:middle line:84% So rookie/expert type of a structure would be key there. 02:25:20.300 --> 02:25:23.620 align:middle line:84% And then you would have the demand side. 02:25:23.620 --> 02:25:28.100 align:middle line:84% Demand side requires people, outsiders, 02:25:28.100 --> 02:25:29.900 align:middle line:90% to know about this service. 02:25:29.900 --> 02:25:33.100 align:middle line:84% There is a potential maximum market size ultimately. 02:25:33.100 --> 02:25:35.780 align:middle line:84% Not every decision in the world can 02:25:35.780 --> 02:25:41.980 align:middle line:84% benefit from that specific type of consulting, for example. 02:25:41.980 --> 02:25:46.140 align:middle line:84% But from that total market fraction, 02:25:46.140 --> 02:25:50.580 align:middle line:84% people need to know about it and need to have a good sense of, 02:25:50.580 --> 02:25:51.840 align:middle line:90% OK, this is a good thing. 02:25:51.840 --> 02:25:57.460 align:middle line:84% And this is a reputable practice. 02:25:57.460 --> 02:26:00.780 align:middle line:84% And if they know about it and have 02:26:00.780 --> 02:26:08.180 align:middle line:84% a good, positive sense about it, then they bring in work. 02:26:08.180 --> 02:26:13.180 align:middle line:84% And that work motivates more of these professionals 02:26:13.180 --> 02:26:16.380 align:middle line:84% to develop their skills, more of the novices 02:26:16.380 --> 02:26:19.660 align:middle line:84% to spend time and build their skills. 02:26:19.660 --> 02:26:24.620 align:middle line:84% And if there is not much demand, then they wouldn't stick around 02:26:24.620 --> 02:26:26.500 align:middle line:90% and they would leave. 02:26:26.500 --> 02:26:32.180 align:middle line:84% So that becomes the core of the model you will be developing. 02:26:32.180 --> 02:26:35.780 align:middle line:84% Again, a little bit less well-defined. 02:26:35.780 --> 02:26:43.180 align:middle line:84% I've left my purpose to be you decide on what's 02:26:43.180 --> 02:26:46.340 align:middle line:90% the right constructs exactly. 02:26:46.340 --> 02:26:51.760 align:middle line:84% If you found time and you found it easy to add more pieces, 02:26:51.760 --> 02:26:54.560 align:middle line:84% you can think about endogenous learning time. 02:26:54.560 --> 02:27:00.180 align:middle line:84% So basically, if you have a lot of novices, 02:27:00.180 --> 02:27:04.540 align:middle line:84% they can't learn on their own that much. 02:27:04.540 --> 02:27:06.860 align:middle line:84% So working with professionals is where 02:27:06.860 --> 02:27:10.740 align:middle line:84% they learn on the job to do the job better and learn 02:27:10.740 --> 02:27:14.580 align:middle line:90% the tricks of the game. 02:27:14.580 --> 02:27:18.940 align:middle line:84% On the other hand, they also take some time 02:27:18.940 --> 02:27:20.760 align:middle line:90% from the experts for training. 02:27:20.760 --> 02:27:25.200 align:middle line:84% So whether that taking of time from experts is important, 02:27:25.200 --> 02:27:26.900 align:middle line:90% you can think about it. 02:27:26.900 --> 02:27:30.440 align:middle line:84% If you have a job that is well-modularized, 02:27:30.440 --> 02:27:31.640 align:middle line:90% so the novice-- 02:27:31.640 --> 02:27:34.220 align:middle line:90% 02:27:34.220 --> 02:27:40.520 align:middle line:84% so the clerk in the law firm helps with the research, 02:27:40.520 --> 02:27:46.540 align:middle line:84% and then the partner is doing some of the more complex tasks, 02:27:46.540 --> 02:27:47.820 align:middle line:90% then both of them are happy. 02:27:47.820 --> 02:27:52.400 align:middle line:84% And the novice is learning and going up the chain. 02:27:52.400 --> 02:27:56.300 align:middle line:84% On the other hand, if you need a lot of hand-holding, 02:27:56.300 --> 02:28:01.540 align:middle line:84% then that effect might be more challenging 02:28:01.540 --> 02:28:03.260 align:middle line:90% for the professionals. 02:28:03.260 --> 02:28:06.760 align:middle line:84% And if there is not much work to be done, of course, 02:28:06.760 --> 02:28:09.100 align:middle line:84% there is not much opportunity for novices 02:28:09.100 --> 02:28:12.480 align:middle line:84% to learn from this learning on the job side. 02:28:12.480 --> 02:28:15.420 align:middle line:90% So those are extra ideas. 02:28:15.420 --> 02:28:19.600 align:middle line:84% And then if you had time doing some policy analysis, 02:28:19.600 --> 02:28:25.180 align:middle line:84% so thinking about policies like certification-- that is, 02:28:25.180 --> 02:28:27.280 align:middle line:90% who is eligible to do the work? 02:28:27.280 --> 02:28:31.800 align:middle line:84% So in the law, there is a very clear certification process. 02:28:31.800 --> 02:28:34.140 align:middle line:90% In consulting, there is not. 02:28:34.140 --> 02:28:35.940 align:middle line:90% Reputation becomes important. 02:28:35.940 --> 02:28:40.100 align:middle line:84% But certification is not part of it. 02:28:40.100 --> 02:28:45.100 align:middle line:84% Training-- so in some settings, you have professional schools. 02:28:45.100 --> 02:28:47.800 align:middle line:84% We teach system dynamics in the university. 02:28:47.800 --> 02:28:49.500 align:middle line:84% But some types of consulting, there 02:28:49.500 --> 02:28:53.350 align:middle line:84% is not as much professional training. 02:28:53.350 --> 02:28:57.520 align:middle line:84% In law, there is law schools you have to go through and so on. 02:28:57.520 --> 02:29:00.060 align:middle line:90% So how does that play out? 02:29:00.060 --> 02:29:03.820 align:middle line:84% And then matching markets-- how well 02:29:03.820 --> 02:29:11.060 align:middle line:84% you can match customers with expert professionals, 02:29:11.060 --> 02:29:14.140 align:middle line:84% how does that impact the dynamics? 02:29:14.140 --> 02:29:18.380 align:middle line:84% So those are interesting policy levers to think through and try 02:29:18.380 --> 02:29:20.820 align:middle line:90% to incorporate. 02:29:20.820 --> 02:29:22.820 align:middle line:84% Another possibility is thinking about, 02:29:22.820 --> 02:29:27.860 align:middle line:84% if you inject more novices, try to bring people into the field, 02:29:27.860 --> 02:29:31.895 align:middle line:84% advertise and bring lots of folks, how does that play out? 02:29:31.895 --> 02:29:33.020 align:middle line:90% These are actual questions. 02:29:33.020 --> 02:29:34.512 align:middle line:84% A lot of professional societies are 02:29:34.512 --> 02:29:35.720 align:middle line:90% dealing with these questions. 02:29:35.720 --> 02:29:38.080 align:middle line:84% I talked with folks in the system dynamics society. 02:29:38.080 --> 02:29:40.280 align:middle line:84% These are exactly the questions of the day. 02:29:40.280 --> 02:29:45.740 align:middle line:84% So if you've got a chance, try to do some policy analysis 02:29:45.740 --> 02:29:48.020 align:middle line:90% as well. 02:29:48.020 --> 02:29:52.065 align:middle line:90% Questions for-- yeah? 02:29:52.065 --> 02:29:54.580 align:middle line:90% AUDIENCE: [INAUDIBLE] 02:29:54.580 --> 02:30:00.820 align:middle line:90% 02:30:00.820 --> 02:30:04.340 align:middle line:84% HAZHIR RAHMANDAD: So you need some feedback 02:30:04.340 --> 02:30:08.120 align:middle line:84% on the supply demand imbalances on both sides. 02:30:08.120 --> 02:30:10.100 align:middle line:90% So there is a quality feedback. 02:30:10.100 --> 02:30:13.160 align:middle line:84% If you have too much demand and not enough supply, 02:30:13.160 --> 02:30:15.220 align:middle line:84% a lot of the work that gets done gets 02:30:15.220 --> 02:30:17.880 align:middle line:84% done by people who don't know what they are doing. 02:30:17.880 --> 02:30:22.540 align:middle line:84% And that quality reputational impact will be there. 02:30:22.540 --> 02:30:25.140 align:middle line:84% If there's not enough work, novices 02:30:25.140 --> 02:30:27.000 align:middle line:90% will leave professionals. 02:30:27.000 --> 02:30:31.580 align:middle line:84% Experts may stick around, but be less engaged. 02:30:31.580 --> 02:30:34.060 align:middle line:84% They don't get enough incentives. 02:30:34.060 --> 02:30:36.120 align:middle line:90% So those you could capture. 02:30:36.120 --> 02:30:37.980 align:middle line:84% I don't think you need to go all the way 02:30:37.980 --> 02:30:41.580 align:middle line:84% into cost-benefit analysis, tracking 02:30:41.580 --> 02:30:44.200 align:middle line:84% profitability of a typical consultant, 02:30:44.200 --> 02:30:46.760 align:middle line:84% things like that for this purpose. 02:30:46.760 --> 02:30:48.480 align:middle line:84% For this exercise, it's not worth it. 02:30:48.480 --> 02:30:54.820 align:middle line:90% 02:30:54.820 --> 02:30:58.040 align:middle line:84% OK, so that's the plan for next week. 02:30:58.040 --> 02:31:02.620 align:middle line:84% Next week, we will have only the first half focused 02:31:02.620 --> 02:31:06.120 align:middle line:84% on the discussion, and then the second half of this model. 02:31:06.120 --> 02:31:07.700 align:middle line:84% And then the second half, we will 02:31:07.700 --> 02:31:11.300 align:middle line:84% go through your problem definitions and the feedback 02:31:11.300 --> 02:31:14.080 align:middle line:84% from those who have looked at those problem definitions. 02:31:14.080 --> 02:31:18.500 align:middle line:84% So come prepared for those conversations. 02:31:18.500 --> 02:31:20.400 align:middle line:90% Any other questions? 02:31:20.400 --> 02:31:23.720 align:middle line:90% 02:31:23.720 --> 02:31:24.660 align:middle line:90% OK. 02:31:24.660 --> 02:31:26.740 align:middle line:90% So have fun. 02:31:26.740 --> 02:31:28.600 align:middle line:90% And I'll see you next week. 02:31:28.600 --> 02:31:33.140 align:middle line:84% Of course, we have some journal club here right now. 02:31:33.140 --> 02:31:36.810 align:middle line:84% So you can stick around for that as well. 02:31:36.810 --> 02:31:40.000 align:middle line:90%