Type Three Error

Do you Need a Prediction or a Prescription?

noreply@blogger.com (Zohar Strinka) — Wed, 10 Aug 2022 16:16:00 +0000

Machine learning and Predictive Analytics immediately captures the imagination as people try to figure out how to use data to grow their business. “What would we do differently if we knew what was coming?” is a common question in the world of data analytics. However, there are a number of challenges with this problem framing that often reduce the success of data projects. In this article, you will learn about some of those challenges and a better approach to leveraging data.

First, two key definitions:

Predictive Analytics is focused on making a prediction. This is often a prediction of what will happen in the future (a forecast). But it can also be a prediction of an unknown value based on other values (e.g., is this credit card transaction fraud or not?). A lot of “Machine Learning” and “Artificial Intelligence” falls into this category.
Prescriptive Analytics is focused on prescribing an action. Typically, this involves creating a model of your system and the effects possible actions would have. These actions often include preparing for the future (staffing / inventory choices). But they can also reflect a tradeoff of side effects given unknowns (e.g., do we deny or approve this credit card transaction?). The fields of “Optimization” and “Operations Research” tend to study this domain.

In general, there has been a lot of focus on Predictive Analytics. As an example, “Analytics Maturity Frameworks” typically suggest that companies who have already nailed Descriptive Analytics can grow into Predictive Analytics. Then, they should only layer in Prescriptive Analytics once they get good at predictions. However, what you will notice about Predictive Analytics problems is that they only impact the business when they have been translated into an action. A perfect prediction does nothing for your organization until it has actually changed something.

This is not to suggest that predictions are useless. But after reading this article, you will understand the role of prescriptions and how the two can be used together to drive value for your organization.

Going From Data to Action

The key reason Analytics Maturity frameworks place Prescription after Prediction is perfectly valid. If there is a problem with the prediction, there is an opportunity for some process to compensate since a prediction by itself does not change anything. Data generally has many inaccuracies and gaps unless your organization has already spent substantial effort solving those issues. Furthermore, if you are trying to make a decision based on incomplete information, it seems to make sense that the first priority should be to improve that information. Maybe you should start with predicting anything that is unknown yet key to making your decision?

By contrast, a broken prescription seems useless. If we ask our analytics what to do based on inaccurate or unrelated data, what benefit can it provide? Unfortunately, this framing glosses over the fact that predictions guide actions. As mentioned above, a prediction that never contributes to an action cannot improve your business. Fraud detection is called a Predictive Analytics problem, and yet only is useful if you then deny transactions that are predicted to be fraudulent.

You might think the above is fine because these predictive algorithms have been found to be accurate enough to link them to actions. In business when predictions are not accurate enough, they are often presented instead as the input to a human decision-making process. A prediction which is properly trained, tested, and validated will have a quantifiable “accuracy” which engenders trust in the output. It also helps to define progress for future development as data scientists try different features, algorithms, or tuning parameters.

But what if we defined accuracy in terms of the actions, not the predictions? Depending on the link from prediction to action, this translation may be simple or very challenging. In the simple case the benefits of measuring accuracy in terms of action are clear since we will be able to measure in business terms. Dollars of uncaught fraud, number of transactions incorrectly denied. When the link is muddy from prediction to action, it is a lot less clear how to fairly assess the prediction. However, actions often are more forgiving than predictions. Ultimately, the measure of success for an analytics tool should be “how did this improve my business?” Anything less is selling yourself short.

To illustrate this point, consider the problem of predicting how much a house is worth. Getting an accurate prediction is fraught with data challenges including complex markets, changing trends, and text-based features. However, a natural use of a house value prediction is to take an action of making an offer on a house. In the context of that action, a prediction is good if it improves profits. While improved prediction accuracy can help, the context of how it will be used is critical to the value of the effort. Done poorly, prediction accuracy may be phenomenal on houses that are entirely irrelevant to your business and terrible on the ones you care about.

Putting it into Practice

The first question you should ask yourself before starting a data analytics project is “What data do we have that is relevant to our goals?” At this point in the project you must decide what data team to bring together to solve your business problems. In a world where you have an enterprise data warehouse team who can quickly respond to new data needs from the business, it makes sense to focus on building algorithms. By contrast, an organization that has not yet worked out the kinks of defining things like “how many active customers do we have now and at any point in the past?” should ensure that experienced data engineers and architects form the backbone of the team. A helpful heuristic is to think about how you would train a human to do the work you are hoping an algorithm will do for you. If that training process would be unpleasant, you probably have some work to do before you can try to train an algorithm.

Once the correct team is assembled per your analytics maturity, it makes sense to clearly define the “why” of your project. Everyone on the team should be able to articulate what would change for your business if the project were to be successful. This is also your key opportunity to ensure that definition of success includes the action, not just the inputs to the action. A helpful thought experiment to distinguish between prediction and prescription is to imagine you have an Oracle and can make perfect predictions. What actions would you take and how would they be different than in an uncertain world?

With a clear set of goals for the project, comes the fun of the analysis phase. “Proof of Concept” and “Minimum Viable Product” are key concepts that can help your team prioritize competing elements of solving the problem. This is also the phase of the project to test competing ideas for how you might achieve your business objectives. Depending on the reality of your data and how you are trying to use it, this phase can be very quick or much slower. The quicker you can prove that the data you have can solve your business needs, the sooner you can quantify the potential ROI of your project.

Finally, even though this article is focused on the benefits of prescriptive analytics, in reality people are often reluctant to hand control over to an algorithm. Moreover, that reluctance is frequently justified due to gaps in the data or qualitative business rules. Given that awareness, choosing the right way to add your analytics solution to the business process is critical to realizing the return on investment. Non-traditional prescriptive solutions like a “what if” calculator can be extremely effective both for communication with your users and to smoothly integrate into business processes.

Conclusion

Data is often limited in how completely it describes the things you most care about for your business. Predictions help to fill the gap from your data to the things that drive decisions. However, there remains a gap from prediction to action which can be filled in part with Prescriptive Analytics. By framing your business problem all the way from data to action, you can more effectively drive business impact.

"Data Literacy" and Why it Matters

noreply@blogger.com (Zohar Strinka) — Sat, 30 Jul 2022 02:31:00 +0000

Most months I attend the INFORMS Practice Section Happy Hour. Today, the topic of discussion was “Data Literacy,” a term I had not previously paid any attention to. In the course of the discussion and reflecting afterwards, I realized that junior data folks needing to become data literate is a key component of what I see makes it hard to break into the field. It is also a very learnable skill with practice.

The way I had previously described this challenge is that everyone who works with data will eventually learn that dealing with Time zones is hard. A couple years ago I watched this 10 minute video which I *thought* had taught me everything there was to know about the difficulty of dealing with timezones. However, this recent article on tzdb covered even more complexities related to the politics of tracking time zones, and the challenge of who gets to make those decisions.

But even without going that deep, simply knowing what timezone your data is in can be tricky, and is frequently important. My first real data project was trying to develop a trading algorithm. At one point I realized I had accidentally been handicapping my algorithm by several hours because I had not properly handled time zones. That is an example of not having data literacy because it didn’t even occur to me to check until I had lost substantial time trying to figure out why my analysis was not making sense. Compare that to a recent experience when I was working on a database server and was told to filter for an end_date of “12-31-9999 23:59.” I was able to quickly shift to “12-31-9999 17:59” when my first query gave me nothing.

With that new lens, I have a slightly different definition of data literacy than say, Gartner. I would define data literacy as “the ability to know what questions you need to answer about a set of data in order to understand it.” I like this framing because whether you know the domain of a particular data set or not, someone who is data literate can make progress on understanding what the data means. Our discussion spent a lot of time on a debate around the “context” you may need to be data literate. I think it makes sense to have a general concept of “Data Literacy” that is not specific to a domain, i.e., context. A key part of my job is knowing when to guess that something means what I think it does, and when to ask a “Subject Matter Expert (SME)” to walk me through the nuance.

To close, I’ll add some common steps I use or that came up during the happy hour to understand a new-to-you data set:

Look at the column headers for the spreadsheet or read the axis and legend of the graph.
Familiarize yourself with the data itself. Maybe check what the set of values is. Look at the most common values in each column. Make some graphs. Test some guesses about links in the data.
Imagine the process behind the data and see if it explains anything about what you have observed so far. Oftentimes there is a human involved, and that can substantially change the interpretation of what you find. For applications you often can answer a lot of questions just by getting a SME to walk you through how the tool works.
In data science bootcamps you always get taught to look at the first few and the last few rows of a data set. I have found grabbing a random sample (.sample instead of .head or .tail in python) to be extremely helpful for getting an understanding of what you are dealing with. This is particularly useful for understanding which data may be missing a lot of the time.
Consider if anything in the prior steps leads you to believe your understanding of the data might not be quite right.
Think about if the data itself might not be quite right.

After spending some time using these principles, I either have a good understanding of the data, or a good knowledge of what I don’t yet understand. Feel free to add any other tips in the comments!

Why I Call Myself a Data Scientist

noreply@blogger.com (Zohar Strinka) — Tue, 05 Apr 2022 11:29:00 +0000

This week, I am at the INFORMS Business Analytics conference, one of the two conferences I attend regularly as an Operations Research PhD and enthusiast. In fact, INFORMS conferences are the only ones I have attended at all since graduation. In my work, I identify myself as a data scientist. What is interesting about these two facts is that INFORMS is not even on the map when it comes to data science.

What I find most valuable about my background in Operations Research is that by the end of your PhD for sure, and likely after a Masters, you have internalized one key lesson: The problem is always up for discussion. Unfortunately, you don't receive that lesson explicitly. Instead, what you get is a series of courses focused on "reformulating" problems. As an example, you learn that linear problems are easiest mathematically, and so you use your training to rewrite problems as linear subproblems. After spending 2-6 years rewriting problems it becomes crystal clear that the first way you think of writing down a problem is unlikely to be the best.

This mindset puts you in a place to succeed as a data scientist (and also as a consultant). Traditionally, the best data scientists are able to take a business problem and understand how to leverage ML and tools from analytics to solve that problem. Data science training programs focus on teaching you primarily how to use ML algorithms and code. This puts Operations Researchers in an odd position as they have so many of the hard-to-find skills on the business side that make exceptional data scientists, but often are lacking the ML skills that are considered "table stakes" for these roles.

As a result of all this, I call myself a data scientist. It is expedient and people generally hand me the right kinds of problems when I market myself that way. At some point they catch on or I warn them that not all data scientists approach problems the same way I do. Depending on the setting I go so far as to explain Operations Research and why they should consider hiring OR professionals for their data science needs. However, what I really wish is that the mindset of OR became the common framework for all data scientists. By articulating the notion that the problem is always up for discussion, you start to realize how much of your value comes not from just solving the problem you were asked to solve, but from getting to the why and how along the way.

My bathroom non-remodel

noreply@blogger.com (Zohar Strinka) — Sun, 27 Dec 2020 02:15:00 +0000

In graduate school I took a class called “Systems Engineering” somewhat on a whim since it was being taught by the former secretary of the navy and I like systems. In preparing to now teach the same course in the spring quarter for DU, I have been reflecting on the guiding principles of the discipline and how I can convey those to my students.

I feel “mindset” is the most distinctive feature of many of my favorite disciplines (and what makes me a good consultant). What sets Operations Research apart is the perspective that the problem is up for discussion as well, not just the solution. Similarly, lean engineering is a way of understanding and improving production systems. I see systems engineering as also primarily a perspective and set of tools: one focused on how people can design large systems that succeed.

Today’s illustration however is not a large system. Namely, it is a home improvement project I was contemplating but hesitant to move forward with. When we first bought our house 3 years ago, I was suspicious of the shower bench in the master bathroom. It seemed to have serious mold issues, and as a former Michigander I am incredibly suspicious of mold. A month later, I had patched in new tile around the bench and felt confident for the short term but knew within a few years I would want to do the whole shower properly.

After finishing my last project this summer, I have been debating how urgent the master bathroom project is. It seems like the next major project I should tackle, but should I tackle it now? With three kids home and work to do, it seemed like an obvious “no.” But still, the molding grout and caulk shouted for something to be done.

And then I realized I was making a classic mistake I learned about in systems engineering. I considered plenty of “revolutionary” alternatives (should I redo the whole bathroom, or just the shower?), but had forgotten to include an “evolutionary” alternative (leave things mostly the same, but re-caulk). One of the important lessons in Systems Engineering is to contemplate your alternatives carefully. If you are not mindful of the alternatives, you often end up with a sub-optimal solution. Like a large remodel in the middle of a pandemic.

I am happy to report that my shower is once-again mold free. And with just a couple hours invested!

Computers like to cheat

noreply@blogger.com (Zohar Strinka) — Thu, 07 Nov 2019 15:16:00 +0000

One year ago I got to hear Janelle Shane speak about her blog "AI weirdness." Her illustrations helped me start to understand sort of the... logic? of more sophisticated machine learning algorithms like neural nets.

This week, her book on the same topic was published and I got to hear her speak again. First off, I highly recommend both the blog and the book. Her justification for writing "You Look Like a Thing and I Love You" is that while we have many examples in science fiction of super-smart AIs like C3PO and Ultron, we don't actually have examples of AI as we have it today. Machines with brains maybe as powerful as a worm, but who are being trusted to screen applicants and drive cars.

The part that I continue to find most interesting in her presentations is the subject of my blog. She explains that while sometimes you don't have enough data to train an algorithm, much more often the problem is that you asked the computer to solve the wrong problem. You wanted a computer to caption images, but neglected to mention that "I'm not sure" is an acceptable answer. Or you asked a computer to make unbiased hiring decisions, but fed it discriminatory examples. These may sound like isolated examples, but Janelle's presentation helps you to understand that computers are always going to take advantage of the smallest oversights in your problem statement to win at the wrong task.

This intuitive idea that the computer will be trying to "cheat" any way it can is helpful as we navigate hype around AI and when we should actually trust it. Can this image recognition software tell the difference between dogs and wolves? Or has it just learned that wolves are often photographed in snow. Should we trust an algorithm because it has a lot of training data? Or might that data have important holes on issues we care about.

None of this is to say ML and AI don't have a place in the world today. But it does help us as individuals in the modern era understand how our lives may be changing for both good and bad as more decisions are handed over to computers.

I'm also going to throw out her Ted talk from last week, if you're not quite ready to commit to a whole book.

Not that multiverse

noreply@blogger.com (Zohar Strinka) — Fri, 07 Sep 2018 23:09:00 +0000

I recently started reading the book "Fooled by Randomness" by Nassim Taleb. So far it is not a book I would recommend to most people (the person who suggested I read it said he usually recommends people start with his most recent book, Antifragile). The author covers very interesting content, but not in a way that is easy to follow or digest. This is the first of probably (hopefully?) a series of posts trying to translate the subject of Taleb's book to an easier to digest format.

While I lived in Ann Arbor during graduate school there was a turn I had to drive about once a month. The unfortunate thing about this turn was that it was a left turn immediately after taking a left at a light. It was so close that I had to make a decision: either move into the middle lane of the road, which was a left turn lane for traffic coming the opposite direction, or remain in the line of traffic, and wait for any oncoming traffic to be clear.

After a few times of taking the turn, I wondered which of the two not-great options I should choose going forward. It seemed to me that I could either risk a low likelihood of a head-on-collision, or a relatively higher likelihood of being rear-ended in the other lane. I settled on staying in my lane and risking being rear-ended because of how much more destructive head-on collisions are.

A few years after making the decision, I made the left turn, and waited for the oncoming traffic to clear as usual. The person who was driving behind me saw the brake lights and stopped. Unfortunately, the person behind them didn't and bumped the middle car into mine. It was fairly minor damage all around, but it is easy to wonder given what happened if I actually made the right choice.

One of the messages from Taleb is that there is complexity in judging the quality of a decision based on random outcomes. For the person who bought a lottery ticket and won, it was a good decision. However, we should advise each person not to buy lottery tickets because in most versions of the universe, the individual you are talking to does not win.

This notion of "most versions of the universe" is a useful one when talking about randomness since it lets you still give weight to things that didn't happen. And while it can be a good idea to update your estimates of probabilities as you get new information, the fundamentals before an event are the same as they are after. As an example, after being rear-ended I did conclude that maybe I should be a bit more aggressive in taking my turn between oncoming traffic. But the fundamentals of my decision didn't change because of it.

Soft vs. Hard constraints

noreply@blogger.com (Zohar Strinka) — Tue, 17 Apr 2018 11:18:00 +0000

Last week, at a meeting to prepare for an on-site kickoff with a client, I was asked if I had any real-life examples of the "squishy rules" I wanted to discuss with the customer. At first nothing was coming to mind, but my airline helpfully solved that problem for me on my way to the kickoff.

After I had started my first flight, my second flight was cancelled. I found myself in the customer service line behind several other people also trying to figure out how to satisfy their constraints and priorities in the best way possible (scheduled meetings the next day, no private jets, how far were they willing to drive a rental car). What struck me was how much those constraints and priorities varied among the 4 people ahead of me in line. Some people were fine with getting in the next night, others (like me) were willing to give up anything except being on time the next day.

Now, you may have noticed above that I combined constraints and priorities into a single list. When I had booked my flight, I chose to fly to the actual city I was headed to. Once that flight was cancelled, I had a choice to make. What used to be two hard constraints now gave me zero "feasible solutions" -- I could either miss one day of the one-and-a-half day kickoff, or I needed to fly to a different city. Now, some very creative people find themselves in this situation and will fly to some other middle city and then to their destination. But my airline either didn't or couldn't suggest those options, and if you had asked me before the cancellation if I would consider a 3-leg trip, I would have given a flat no. So if I had no possible solutions, what could I do?

Well, this happens a lot. People will often list their preferences as needs until pressed. And as long as there is a feasible solution, it doesn't have to become obvious. One of the people ahead of me in line chose not to give up any of their hard constraints, which meant there were still no options available. It was obvious that something had to give unless the goal had changed -- nevermind, I didn't need to go to that city after all. But knowing which of your rules to turn "squishy" is the key to still achieving your goal.

In my case, I flew to a neighboring city instead. In fact, my boss had flown directly to my alternate city and planned from the start to drive the remaining distance -- he had never made flying to the final city a constraint. As a result of this experience I also finally bought some plane tickets for the summer I had been putting off for weeks. I am now flying to the 2-hour-away airport for less than half the price of the tickets to the actual city.

Have you ever realized you were overconstraining your problem? Which constraints turned out to be a lot squishier than you realized?

You can't inspect in quality

noreply@blogger.com (Zohar Strinka) — Sat, 03 Feb 2018 20:21:00 +0000

This is just a short post on applying industrial engineering principles to daily life.

At some point in my education, someone told me that it is impossible to inspect in quality. At the time it made sense from what I knew about inspections: people are bad at noticing rare events.

Since then I have found a semi common application at home where attempting to inspect in quality is both tempting and a bad idea... Cleaning up broken glass. I have no idea how other people do it, but the system I have found to avoid the unpleasant outcome of stepping on glass is to clean extremely thoroughly twice, and then to conduct my first inspection. If I find any glass, I assume there are several more pieces I missed and do another cleaning pass.

Do you have any tips to speed up this process? Thoughts of other scenarios where it is tempting to try to "inspect in" quality? Leave your thoughts in the comments!

Solving the “real” problem

noreply@blogger.com (Zohar Strinka) — Sat, 14 Oct 2017 18:59:00 +0000

In undergrad when I learned about the field of operations research I assumed people would write down their objective and constraints, get the optimal solution, and then do whatever the model told them to. Eventually I took a class from an adjunct professor my first year of grad school who explained that the hardest part of working in OR was convincing people to implement the output of the model. Basically "decision makers" (aka, people who did not know math) would not believe the output of the model, and so we had to design things so they could follow all the steps in our analysis.

I internalized that people would have reasons not to believe the model, but for a long time continued to believe it was mostly because of mistakes people made. You would build them a beautiful model, and then they would see the solution and realize that they forgot to give you important constraints. Or they would see the result and just determine it too weird and insist on a sub-optimal solution which looked more like what they had been doing. Over time I developed a more complete list of why people would not trust a model, but I still fundamentally thought of the models as right.

At some point though, that changed. I stopped thinking of people as the problem. I started this blog under the premise that not solving the right problem (type 3 error) was avoidable, but took a careful study to get to the problem you should solve. Even now, I have continued to find it challenging to really talk about that mental shift. In fact, this particular blog post has been sitting in purgatory since July while I was figuring out just the right way to convey the distinction.

But yesterday, while reading the HBR article “Are you solving the right problems?” the author described reframing a problem as not simply redefining the “real” problem, but instead recognizing that there is a better problem to solve. Realizing that you could be solving a better problem is not a simple process. It often requires attempting to solve other problems first. Even the notion of a “better” problem is not straightforward. It may have to do with the intractability of your current problem, or the realization that your first solution does not achieve what you thought it would.

If you are reading this and have a problem that could use some reframing, feel free to reach out to me or leave a comment here. Oftentimes, just explaining the situation to someone a bit further from the problem is all it takes to shift your context.

Influencer book review: part 2

noreply@blogger.com (Zohar Strinka) — Thu, 24 Aug 2017 13:23:00 +0000

When I need to explain why I think industrial engineering is different than other engineering disciplines, I often point to the fact that we see the whole world as systems. While all engineers need to model pieces of the world, in my opinion industrial engineers take a wider view largely because they consider people as part of the system. Suddenly the way you approach problems is different because you can't just expect that a person will do what you tell them. You have to accept the reality that people will do what they're going to do, and your job is to design a system where what they decide to do is what you need.

In "Influencer" the authors start by suggesting you identify very explicitly what your goals are in a way that they are actionable. You then identify the moments at which people have a choice to support that goal, or not. Finally, you use "influence" to help them choose to support that goal. The authors then spend the majority of the book on the "6 keys of influence" which are ensuring there are personal, social, and structural: motivation and ability, which are encouraging the person to make the choice which aligns with the goal. The book itself is full of examples of what each of these look like, but you can think of times you were personally unmotivated in doing homework or felt unable to do something you believed you should.

And this brings us back to industrial engineers who see the world as systems. In the first chapters when the authors referred to structural motivation and ability, I had no clue how that was supposed to be different than the personal and social categories. But when I got to the actual chapters, I recognized exactly the mindset industrial engineers use to effect change. We try to design changes to almost be easier than doing things the old way. I remember trying to figure out how on earth you are supposed to have an orderly office supply drawer, and the flash of insight when I was looking at pictures and realized the first step was to have about 1/3 of the items I presently had. The system (a jumble of career fair pens and highlighters and countless other trinkets) made it impossible for me to have an organized drawer. It wasn't that I was personally unable to do this, it was the structure itself that made my goal impossible. This became particularly clear after I fixed the system (removed most of the content of the drawer) and was able to organize the remainder.

In the book the authors do actually make the connection that industrial engineers have been the pioneers of structural ability. However, the goal of the book really is to give the reader a framework to effect change using all 6 approaches at once. The main premise of the book is that most people trying to effect change only use one or two approaches, which is simply not enough. I think the underlying reason it is not enough is that people are different. Some people will happily change their behavior if it will get them a bonus while others will only do it if it would be embarrassing not to. Therefore, by encouraging a certain behavior "on all fronts" persay,

you can hope to actually reach everyone.

Book review: Influencer

noreply@blogger.com (Zohar Strinka) — Wed, 19 Jul 2017 14:35:00 +0000

I have been picking up highly recommended books for years on business-y topics that seemed interesting. However, until a couple months ago when I found a position as a data scientist at Mashey, reading most of them had never topped my to-do list. One of those books was called "Influencer: the new science of leading change."

The book is impressive to me for carefully articulating how little it takes to really create change, but also the complexity behind those small differences. I have noticed that it really does seem to be the small things that determine outcomes, but had not fully articulated what made that set of small things special. In the book the authors describe those things as "crucial behaviors." They give examples of settings where a leader was able to articulate and change one or two problem-specific behaviors which stopped the spread of disease, got inner city kids successfully through college, and many other cases.

While many business books lay out a roadmap to success that may or may not work, this one covers an approach that feels very familiar to me. As an example, I have found that while the everyday interactions matter for my kids, there are pretty infrequent "critical moments" where if I notice and take the opportunity, I can teach them something really important. Further, the whole premise of this blog is that if you can identify the right problem to solve, you will be much more successful in your projects.

I actually am jumping the gun a bit on posting this since I've only made it through part 1 of the book so far, but the framework is genuinely inspiring to me. Hit me up in a month and I can give you the complete run-down.

The research process

noreply@blogger.com (Zohar Strinka) — Thu, 27 Apr 2017 13:44:00 +0000

I naturally am a big picture person who learned during grad school to also have a more detail-oriented mode. I had not gotten to that point in my second year, and my advisor pointed out that I had this habit of working out the bigger picture and then would immediately jump to trying to prove something (though not necessarily the right something). For the next two years I had a post-it note on my computer with "Big -> medium -> details" on my monitor.

Eventually I didn't need the note any more, but the smooth transition between levels of research focus stayed present in my mind as I continued my PhD. The summer before my last year I took some time to work on an independent project. I had my initial ideas of what the big picture was, but discovered in working out the details that there were interesting high-level concepts which I would not have come up with without going through the math. I realized that research is not just a one-shot transition through the levels, but that ideally you may traverse the range of focus levels a number of times to finish a project.

Which brings us to the picture at the top from xkcd. I like the idea of the "research focus knob" because there is no way to get from the big picture to the details without going through the intermediary levels. More than that, I think it makes it clearer that your goal is not to just go in a straight line from big picture to the details, but instead to pick the right level of the research problem at every point in the project.

Thoughts or questions are welcome!

Stochastic Optimization as a mindset

noreply@blogger.com (Zohar Strinka) — Wed, 22 Feb 2017 22:05:00 +0000

Stochastic optimization (finding the best solution when you have randomness) is a tricky topic. As an example, think about picking the fastest route for driving home from work. Depending on the traffic, different paths might be fastest. It might make sense at first to say "I want to pick the path that will be the fastest today." But when that outcome has uncertainty, it is usually impossible to answer until after you actually have driven home.

There are a number of different ways people handle this issue in optimization. For driving home, it probably makes sense to pick the route with the lowest expected transit time. Over the years of driving that path, some days might take longer than you'd like, but in the long run you'll come out ahead. Other times when you have a dinner-appointment, it might make sense to pick the route that has the lowest probability of taking more than 25 minutes so you are not late. If you were managing the electric grid and avoiding power outages, you will account for the uncertainty in a different kind of way.

As you build a model of the world, different objectives will compress or amplify the effects of uncertainty. Frequently when you are driving somewhere, there will be several routes that have basically the same expected transit time. But the likely worst-case (say, average of the worst 5%) of driving times will often be very different across routes. In the first case, the uncertainties are a relatively small issue because they all get averaged away. In the latter case, extreme cases have a much bigger effect, and therefore the uncertainty will too.

Since models are just attempts to represent reality in a useful way, which model to use for a stochastic problem will depend a lot on your best guess of the costs of uncertainty. If you use an expected-value objective but care a lot about the worst-case tails, you are going to have a bad time when you implement your solution. On the other hand, if you optimize for the worst case for a low-stakes situation like how much inventory to order for a promotion, your company probably will not stay in business too long.

While I have been focusing on either worst-case or expected value as an objective, there are countless ways to design your stochastic optimization model. The short version is that in the field, we are typically trying to reduce the random outcome of our decisions to a single number which allows us to pick the "optimal" solution. While there are ways to optimize over "multiple objectives," they still tend to focus on either subjective decision making or weighting the objectives to obtain a single number.

I welcome comments either on the blog or directly to me. I'm getting this topic ready for a short talk, and I think this will be only my second talk on optimization to a room full of not-optimization people.

Rocky Mountain Datacon

noreply@blogger.com (Zohar Strinka) — Sat, 12 Nov 2016 15:59:00 +0000

I spent the previous two days at the first Rocky Mountain Datacon. I haven't yet figured out how to blog during a conference (I have two half-finished posts and a number of ideas), but it was a great experience and I learned a ton. All the talks were filmed, and it was successful enough that the organizers expect to do it again next year.

For the moment, I thought I would share a few thoughts of what I learned at the conference. Feel free to hit me up if you are interested in a discussion about any of them since that will help my eventual posts be more useful and articulate for everyone.

Data has allegories to oil, currency, intellectual property, and inventory.
Data is a tax-free asset (though it does cost money to keep it and use it).
With the current technology and tools, we have distinct classes of big, medium, and small data. Accurately assessing what you have and will have in the future is important for picking the right technology stack.
I picked up a lot of data science 101 including what all the titles should mean, what a technology stack is, how to pronounce the word "munging," what the technology options right now look like, how to "break into the field," and a ton of other things.
And for the OR folks reading, very little of any of this is using optimization yet. Several people threw around "5 years" as the timeframe to get there, so it seems to be a pretty good time for us to join the data science world.

Evaluating health claims

noreply@blogger.com (Zohar Strinka) — Thu, 06 Oct 2016 19:03:00 +0000

There is a large body of evidence (along with personal anecdotes) that getting people to change beliefs is very difficult. Studies have found that providing people with contradictory evidence can make them even more confident in their views. Given these challenges, how should we go about reducing misinformation in the world?

One in-progress study is looking at teaching primary school students in Uganda how to evaluate health claims as well as the evidence they are based on. While I had previously thought that a basic understanding of statistics was our best option, this kind of education is more clearly and directly related to the goal. It will be interesting to see what the results of the study end up being, particularly if we eventually find that learning in one area spills over to increase scientific literacy in general.

Indicators and using noise as signal

noreply@blogger.com (Zohar Strinka) — Sun, 25 Sep 2016 20:26:00 +0000

How should you use an indicator? When you look at the weather report, it seems pretty straightforward. Sunny means you do not need an umbrella. High of 36 means you should wear a coat. But what about when there is a 40% chance of rain? And what if you are trying to figure out if you can go for a hike this weekend?

When talking to my friend Chris Miller recently, he was trying to predict the weather before going on an aggressive hike. He mentioned that the noise in the forecast was part of his signal to decide how seriously he should take the report. If the forecast kept changing in the few days leading up to the proposed hike, that meant there was a decent chance that the weather would be unfavorable the day-of.

This got me thinking about the standard weather forecast as an indicator of the underlying data. I had a friend who was studying to be a meteorologist, and so she would go straight to the NOAA source data to predict the weather. For everyone else the weather report is basically a black box and all we have to work with are the indicators.

And when there are no indicators that answer your question? Or if it is simply impossible to interpret the underlying data? That is when it is time to get creative with the information you do have.

Voting and an intro to some game theory ideas

noreply@blogger.com (Zohar Strinka) — Fri, 02 Sep 2016 14:52:00 +0000

My apartment complex decided to show a movie via projector and sent out a poll with 6 options. We were asked to rank the options and told that the movie that won would be shown at the movie night.

If management only got one response, it would be easy to decide how to vote and which movie won. However, assuming there was more than one response, how should the winner be determined? And given that there will be other voters, what should your vote be? In this game, management has decided on the rules for voting (rank the 6 options) as well as the rule for which movie is selected based on those votes (which they did not tell us). The voters are then left to decide how to vote, given their guess of the rules.

The most likely way to score such a set of votes is that management could give a set number of points for each possible rank-position (i.e., 5 points for being ranked 1st, 4 points for 2nd, etc) and then take the movie with the highest sum. However, there's no reason they couldn't pick the movie with the largest number of times being ranked 1st, and then use the later places as tie breakers. And in general, there can be truly crazy sets of rules. For example, if one movie is far-and-away the favorite in general, we could handicap that movie by saying any vote for it is actually only worth half a vote (this seems ludicrous... but in auction theory counting a "high-valuation" bidder's bid as a fraction of their actual bid is a standard tool to design an "optimal" auction).

Depending on the rules, an individual voter has different incentives. Further, depending on their guess as to how everyone else will vote, they will have additional incentives. An important notion in game theory is a "Nash Equilibrium." A NE is a set of votes for everyone so no individual person will choose to switch their vote. So if you knew that everyone else was following the NE, there is no benefit to you from not following the NE. But there are a lot of assumptions that go into the NE including that it is unique, that there will be no collusion (lets both list our shared second-favorite movie as 1st), and that somehow there being a NE actually leads people to vote accordingly (here is the Wikipedia link on when that will happen).

Given all this complexity, you might wonder how anyone ever decides anything. In a world where so much is uncertain though, a lot of decisions could be the best. If I vote my genuine ranking, I'm at least giving my preferred movie it's best shot to be selected. I could vote strategically and rank my 3rd favorite as top because I think my least favorite movies are the most popular. I could also not vote at all because I think the effort involved in voting is more than the difference between the outcome when I vote or not. Which of these guesses of uncertainty is right is impossible to say until after the votes are in, which are then influenced by what everyone else is guessing to be the case.

Hopefully, this gives a taste of some of the difficulty both in designing the rules of a game, and the subsequent decisions by the voters. Early on in learning about game theory I was told "the devil is in the details," which I have found to be absolutely true. First-past-the-post voting seems sensible, until you realize the incentive issues when there are more than two choices.

Feel free to send me any follow-up game theory questions you have and I will do my best to get them answered!

Classification problems

noreply@blogger.com (Zohar Strinka) — Tue, 23 Aug 2016 22:05:00 +0000

A friend asked me for ideas of a good analogy for classification problems in machine learning. In a classification problem, we have a collection of objects, and we somehow want to separate them into groups. Ideally, when designing this analogy there are a few things we want to convey:

Not all properties of objects are equal when it comes to classification. Some will be highly predictive, while others just help you over-fit your model.
Some properties will be highly correlated, so it can be a waste of energy / data effectiveness to include them all.
Your desired classification informs which properties you should use for your classification.
What do "properties" even look like, and how do they help us get at a classification?

The example I suggested is a person deciding what to eat at a potluck. I usually have two different classification problems to worry about when I'm filling my plate. First off, I want to decide which things I want to eat. In addition, I'm one of these people who will get a main course plate, eat that, and then go get dessert later. So as I survey the food, I have to decide both which things I think will be delicious, and which things I want to get later as dessert.

When trying to figure out what will be delicious, there are a lot of criteria I could use. Since I do not like cucumbers, anything with them is immediately excluded. Other properties besides ingredients could be smell, color, how much of it is available, how much was already eaten, the temperature of the food, anything! Some of these criteria are more helpful than others. I've had a lot of delicious brown things in my life at potlucks. And when I am trying to decide if something is dessert of not, how much people ate is not going to be very informative.

How do you classify food at a potluck?

Organized Brainstorming

noreply@blogger.com (Zohar Strinka) — Thu, 14 Jul 2016 14:31:00 +0000

When you learn about brainstorming, there is often a focus on how "spontaneous" it should be. Don't worry if an idea is good or bad, just add it to the list! During undergrad I took several of Tau Beta Pi's "Engineering Futures" classes. One of the topics was a modified approach to brainstorming.

With the mindset they taught, the quality of the ideas are still unimportant, but you go about producing them in an organized way. Say you are trying to come up with the list of things you need to buy from the store. Instead of simply writing things down, you create categories and then fill in each category. For our shopping example, you might list each room in your house as a category and then think about what things you need for each room. If you are trying to figure out the best way to solve an engineering problem at work you might have categories like "new equipment" and "better software."

What are your thoughts on brainstorming? After a quick read through this link on the topic, it looks like I am describing a more task-oriented version of brainstorming. Which makes sense for an engineering-focused training on the subject.

Using decision trees for sequential decision making

noreply@blogger.com (Zohar Strinka) — Sun, 29 May 2016 14:23:00 +0000

I am a planner by nature. As I get close to leaving my apartment of 6 years to go a third of the way across the country, I've found planning a move to be a complicated project. For most of the past year I have tried to avoid thinking about the move since there was a lot of uncertainty which would be resolved with time. With the move less than a month away there is still a lot of uncertainty, but very little of it will be resolved until weeks, months, or years from now.

Since my research is in stochastic optimization (making decisions under uncertainty), in principle I have a lot of tools at my disposal. But most of the complexity of a move is the sequential decision-making aspect. Initially you have a lot of degrees of freedom. With each decision you eliminate not only the alternatives to that choice, but also the feasibility of any decisions which come later.

In my research I have not come across tools to handle this aspect of empirical decision making. However, with a little thinking outside the box, decision trees can! Decision trees let you visually consider what will happen in a world after a series of decisions have been made and uncertainty resolved. For a situation where the order of decisions matters, you can then draw multiple decision trees for each of those orderings.

I only explicitly drew decision trees for a couple aspects of my move. But for the early decisions, lots of flexibility down the road was an important factor. There were also a couple cases where I did draw the tree so I could articulate the relationship between specific uncertainty and my decisions. In one case I found that the uncertainty I was worried about did not actually change my choice!

In somewhat related news, hopefully I will get back to my once-a-week schedule for posting soon.

Link on stats and journalism

noreply@blogger.com (Zohar Strinka) — Fri, 06 May 2016 12:44:00 +0000

I've linked to his blog before, but this post by Andrew Gelman is a great read. Whether you happen to be a journalist, someone trying to understand how to communicate statistics results to the public, or just want to know what is missing from the standard "science says x cures y!" news stories.

Back to business

noreply@blogger.com (Zohar Strinka) — Thu, 14 Apr 2016 19:31:00 +0000

After defending (and revising) my thesis late last month, I'm back to blogging. This week I was in Orlando for the INFORMS Analytics conference. During the conference they host the Edelman competition with the goal of recognizing a project with huge tangible contributions made possible by OR and analytics.

I made it to three of the six talks, and enjoyed hearing about what people are doing with data.

The NYPD has developed "DAS" which integrates with department phones and provides real-time information about an address and the surrounding area before police even show up at the scene of an event. They also use the data to identify likely locations for crime to target extra enforcement (it seemed somewhat less sinister than minority report).
Until very recently, UPS still had their drivers plan their routes each day. ORION changed that with major cost savings. One of the most interesting parts to me was that before they could roll out this project, they needed to collect much more accurate map data to avoid the problem of "Google maps tells me to drive a mile out of the way to make a u-turn."
360i is a marketing firm which specializes in paid search advertising. Keyword-based advertising at Google is allocated based on the result of an auction. However, there are literally hundreds of thousands of strings that a particular company might want to bid on. 360i developed a set of tools to improve the effectiveness of search advertising by trying to figure out what people intended to search for (their example was you don't want to waste advertising dollars on someone trying to figure out how to sort out relationship problems)

UPS was chosen as the winner this year, and will be joining an impressive list going back to Pillsbury in 1972.

Links for Students

noreply@blogger.com (Zohar Strinka) — Tue, 08 Mar 2016 16:45:00 +0000

To anyone else who will eventually be doing a PhD defense, here are a few useful links I've come across recently.

10 Ways to Successfully Defend Your PhD: This focuses on the presentation itself. There appear to be other useful articles on the website if you want to look around.

Hints from CS at Columbia: This is the link that helped me understand the point of the defense. Specifically, getting everyone on the same page in case some of your committee did not read every section so that if one person asks a specific question, everyone will understand the context when you answer.

Preparation tips for Rochester University: This link talked about the basic logistics. If you are a PhD student and haven't attended a defense yet, make it a priority long before your turn. There are also always snacks.

Prepare your PhD Defense Presentation: This one did a nice job of demonstrating what a research question is.

Good luck!

Edit: Just found a super helpful Defense Outline.

Don't ask "which one is right?"

noreply@blogger.com (Zohar Strinka) — Fri, 19 Feb 2016 13:27:00 +0000

The above quote came from this blog post discussing the PACE trial which was a large-scale randomized controlled study comparing different treatment approaches for Chronic Fatigue Syndrome. I would not necessarily recommend reading the post since it can be hard to follow, and the comments suggest that in fact the study actually only provided evidence that the placebo effect holds...

But separate from all that, people tend to be uncomfortable with uncertainty. The setup of the study was supposedly to demonstrate which of different treatment options were effective. However, different people are likely to have varying responses to the same treatment. For something like CFS, it then makes sense to start your study with a question like "how can we identify the right treatment for each person" rather than "which treatment is right for everyone."

These questions come up a lot in the relatively new field of personalized medicine. While doctors have always used patient-specific information to make healthcare decisions, policy makers typically have not. Sometimes that makes sense: Eating more plants and getting more exercise are good for pretty much everyone. But we should not let our desire to have one right answer get in the way of understanding complex systems.

The Greenfield Bridge in Pittsburgh

noreply@blogger.com (Zohar Strinka) — Sun, 07 Feb 2016 04:47:00 +0000

A speaker in my department reminded us about the bridge-under-the-bridge in Pittsburgh, which involved building a bridge to catch falling debris from traffic-carrying bridge above. The speaker posited that perhaps the resources that went into building the second bridge would have been better spent fixing the broken bridge.

The bridge is finally being replaced (link here). The article actually provides numbers we can use to decide if, in hindsight, the secondary bridge was a good idea.

The bridge is projected to cost 19 million dollars in today's dollars to build, while the secondary bridge cost $625,000 in 2003 dollars. Based on the CPI, $1 in 2003 would be $1.29 today, and therefore the bridge would cost about $800,000 in 2015 dollars. But it turns out that is pretty irrelevant since $800,000 / $19 million = 4.2%. Assuming building new bridges wasn't wildly cheaper back in 2003, this was an obviously good investment since the bridge lasted 12 years, not one.

I will add one caveat: I don't know a lot about how these kinds of capital projects are funded. The replacement bridge project page mentions that 5% of the funding is coming from the city, 15% from the state, and 80% from the federal government. The above analysis in some ways assumes that the breakdown is the same for the secondary bridge. However, it appears likely that the city of Pittsburgh covered the full cost of the under-bridge. That puts their investment in 2003 for the past 12 years of bridge as about the same as for an entirely new bridge which will hopefully last 100 years. From that perspective, the city's best decision is a little less clear.