Apple are evil because they take away control of how we use their devices.

I teased Mike that I would never say “Apple are evil.” On this side of the Atlantic we’d say “Apple is evil.” But in the UK it is accepted usage to say “Apple are evil.”

“Apple” is a collective noun when used to refer to Apple Inc. British English treats collective nouns as plural, but American English treats them as singular. Although the British usage sounds odd to my American ears, it makes sense just as much sense as the American convention. You could argue for plural verbs because corporations are made of individual people, or you could argue for singular verbs because the corporations act as a single entity. See Grammar Girl’s tip on collective nouns for more background.

By the way, I don’t believe Apple is evil. They’re just a company, no more or less virtuous than most other companies.

Apple posts:

I am not an operating system
Inside Steve Job’s brain
Protestant PCs, Catholic Macs

Grammar posts:

Important because it’s unimportant
English grammar
Finding grammatical errors in software

Regular expressions

@RegexTip started over today. It’s a cycle of tips for learning regular expressions. It sticks to the regular expression features common to Python, Perl, C#, and many other programming languages. This account posts Monday through Friday.

Keyboard shortcuts

@SansMouse gives one tip a day on using Windows without a mouse. By practicing one keyboard shortcut a day, you can get into the habit of using your mouse less and your keyboard more. This cycle of tips started over January 29 with the most common and most widely useful shortcuts. I’m also sprinkling in a few extra tips that are less well known. This account also posts Monday through Friday.

Math

I have three mathematical accounts. These post seven days a week.

@AlgebraFact, just started February 2. It will be a mixture of linear algebra, number theory, group theory, etc.

@ProbFact gives one fact per day from probability. Usually these facts are theorems, but sometimes they include a note on history or applications.

@AnalysisFact gives facts from real and complex analysis. The topics range from elementary to advanced.

What if I don’t use Twitter?

You can visit the page for a Twitter account just like any other web page. And every Twitter account has an RSS feed link allowing you to subscribe just as you would subscribe to a blog.

How do you write these?

I write up content for these accounts in bulk. I may sit down on a Saturday and come up with several weeks worth of tips. Then I use HootSuite to schedule the tips weeks in advance. Sometimes I’ll post something spontaneously, such as link to something relevant, but most of the work is done in advance. I use my personal Twitter account for live interaction.

Related links:

Using Windows without a mouse

Regular expressions in

• Python
• PowerShell
• Mathematica
• R
• C++

Chart of probability distribution relationships

Here’s an insightful observation Yegge makes along the way.

And Haskell, OCaml and their ilk … try to force people to model everything. Programmers hate that. These languages will never, ever enjoy any substantial commercial success, for the exact same reason the Semantic Web is a failure. You can’t force people to provide metadata for everything they do. They’ll hate you.

Related post:

Probability of semantic markup being correct

Online diff tool
HTTP flowchart
Astroinformatics
Python propaganda
How to safely store a password
Google Docs dropping IE6 support
Why it’s hard to move Facebook off PHP
Keyboard shortcuts for Windows, Mac, and Linux

Math

Fundamental examples in math
Daily fact from algebra and number theory
62nd Carnival of Mathematics

Miscellaneous

Always wear your seatbelt
Space shuttle repaired with duct tape (lunar rover too)

How to peel a pummelo. YouTube video with nice soundtrack.

I’m taking a turn hosting this month. Tradition dictates that the host begin with some trivia about the number of the post. As this is the 62nd Carnival of Mathematics, here are a few facts about 62.

• 62 is the only number whose cube (238328) consists of 3 digits each occurring 2 times.
• The great rhombicosidodecahedron has 62 sides.
• Louis Pasteur developed the first rabies vaccination at age 62.

And now onto the posts.

Math and science teacher Cory Poole sends in a video that he created along with his partners and students. The video features a 64-foot Sierpenski triangle about of 12,000 tortilla chips. Read more about the story of the video. Also, here are the bloopers from making the video.

St. Swithun’s day is a sort of British analog of America’s Groundhog Day. If it rains on St. Swithun’s day, it is supposed to rain for the next 40 days. Is there some truth to the legend? See Jon McLoone’s article Mathematica Tests the St. Swithun’s Day Proverb posted at Wolfram Blog.

Edmund Harriss from Maxwell’s Demon presents Spirographs and the third dimension. Interesting visually and mathematically.

Rachel Thomas presents Beautiful symmetry provides glimpse into quantum world on News from the world of maths. This article reports on a low-termperature experiment that implies that the exceptional Lie group E8 is at work.

Did you know that sine and cosine are equal for all x? Heather (Xi) submitted a pseudo-proof in A=B implies that 1=1, therefore? by her colleague TwoPi at 360. (If there is ever a 360th Carnival of Mathematics, Heather should host it.)

Update: The 360 blog has agreed to host the 360th Carnival of Mathematics, tentatively scheduled for December 1, 2034. (Mike, I hope it’s OK that I scheduled this date without consulting you. )

Marc West presents The curse of the duck, a post about mathematics and cricket. His blog is Mr Science Show: Where Science Meets Pop Culture.

Dan M presents Appolonian Gaskets and Ford Circles on his blog mathrecreation. This post relates number theory and geometry as Ford circles are related to the Farey sequence.

Rick Regan presents Counting Binary and Hexadecimal Palindromes posted at Exploring Binary. Rick counts base 10 palindromes as a warm-up before diving into new territory with binary and hexidecimal numbers.

Gregory Astley wrote a guest post Maxima Tutorial – plotting direction fields for 1st order ODEs for Mike Croucher’s blog Walking Randomly. (Maxima is part of the SAGE mathematics system. Mike has written several posts about SAGE lately.)

Annarita Ruberto from Matem@ticaMente presents the article “How heavy the fish?” The original post was written in Italian, and here is Google’s translation of the page into English.

Matt McDonnell presents Mathematical Recreations: Tweetable Game Of Life, a guest blog for Loren Shure’s blog Loren on the Art of MATLAB.

For some mental arithmetic shortcuts and an explanation of why they work, see Sol Lederman’s post Trachtenberg speed multiplication: exploring why it works on his blog Wild About Math.

Politics impacts sphere of human activity, including math education. See Jason Dyer’s post Anatomy of a Political Math-Ed Reaction on The Number Warrior.

Peter Rowlett from Travels in a Mathematical World presents Substitution ciphers: Ancient – Renaissance, video included below.

You can keep up with Carnival of Mathematics news on Twitter by following @CarnivalOfMath. You may also be interested in daily math facts on Twitter from @ProbFact (probability), @AnalysisFact (real and complex analysis), and @AlgebraFact (algebra and number theory).

The article is Finding probability distribution parameters from percentiles posted on CodeProject. The article comes with Python source code and some commentary. The article shows how SciPy and the functools module make it possible for the code to be very succinct.

Related posts:

Probability distribution parameters in SciPy
Numerical computing in IronPython with Ironclad
Getting started with SciPy

Python’s SciPy library has a somewhat unusual approach to parameterization with some advantages. SciPy makes every continuous distribution a location-scale family, even those distributions that typically do not have a location or scale parameter. This eliminates, for example, the question of whether an exponential distribution is parameterized by its mean or its rate. There is no mean or rate parameter per se. But there is a scale parameter, which happens to also be the mean.

Some methods on distribution classes have unusual names. For example, the inverse CDF function, often called the quantile function, is ppf for “percentile point function.” The complementary CDF function, or CCDF, is called sf for “survival function.” (Survival function is not an unusual name, though my preference would have been ccdf since that would make the API more symmetric.)

Discrete distributions in SciPy do not have a scale parameter. Also instead of a pdf method the discrete distributions have a pmf method; continuous functions have a probability density function but discrete methods have a probability mass function.

One surprise with SciPy distributions is that the SciPy implementation of the lognormal distribution does not correspond to the definition I’m more familiar with unless the location is 0. In order to be consistent with other continuous distributions, SciPy shifts the PDF argument x whereas I believe it is more common to shift log(x). This isn’t just a difference in parameterization. It actually amounts to different distributions.

For more details, see these notes on distributions in SciPy. See also these notes on distributions in R and in Mathematica for comparison.

Related posts:

Basics of distributions in SciPy
Getting started with SciPy
IEEE floating point arithmetic in Python

Little programs are delightful to write in isolation, but the process of maintaining large-scale software is always miserable. … Technologists wish every program behaved like a brand-new, playful little program, and will use any available psychological strategy to avoid thinking about computers realistically.

Related posts:

Writes large, correct programs
Why there will always be programmers

… every one of the great industrial disasters of the past twenty years — Chernobyl, the Exxon Valdez, Bhopal, Three Mile Island — occurred in the middle of the night. For the most part, those in charge had worked very long hours and built up considerable sleep debt.

So far I’ve found the first episode most interesting. It discusses the “moratorium”, the plan to give Python library authors time catch up with Python 3 before extending the core language further. This sounds like a very smart move.

Related posts:

Good enough for Google and NASA
Plain Python

• How do I configure PowerShell?
• How do I make PowerShell launch faster?
• How do I get documentation?
• Why did PowerShell make some of the design decisions they did?
• Once I’ve written some useful functions and scripts, where do I put them?
• Where can I find more PowerShell resources?

Now I’ve started updating the booklet to reflect changes in PowerShell version 2.0. I haven’t had a lot of experience with version 2.0 and would appreciate your help updating the booklet. I have put a link to an alpha version of the update for version 2.0 on the download page.

Things would be easier if you could ask subject matter experts to express their opinions in statistical terms. You could ask “If you were to represent your belief as a gamma distribution, what would the shape and scale parameters be?” But that’s ridiculous. Even if they understood the question, it’s unlikely they’d give an accurate answer. It’s easier to think in terms of percentiles.

Asking for mean and variance are not much better than asking for shape and scale, especially for a non-symmetric distribution such as a survival curve. Anyone who knows what variance is probably thinks about it in terms of a normal distribution. Asking for mean and variance encourages someone to think about a symmetric distribution.

So once you have specified a couple percentiles, such as the example this post started with, can you find parameters that meet these requirements? If you can’t meet both requirements, how close can you come to satisfying them? Does it depend on how far apart the percentiles are? The answers to these questions depend on the distribution family. Obviously you can’t satisfy two requirements with a one-parameter distribution in general. If you have two requirements and two parameters, at least it’s feasible that both can be satisfied.

If you have a random variable X whose distribution depends on two parameters, when can you find parameter values so that Prob(X ≤ x₁) = p₁ and Prob(X ≤ x₂) = p₂? For starters, if x₁ is less than x₂ then p₁ must be less than p₂. For example, the probability of a variable being less than 5 cannot be bigger than the probability of being less than 6. For some common distributions, the only requirement is this requirement that the x’s and p’s be in a consistent order.

For a location-scale family, such as the normal or Cauchy distributions, you can always find a location and scale parameter to satisfy two percentile conditions. In fact, there’s a simple expression for the parameters. The location parameter is given by

and the scale parameter is given by

where F(x) is the CDF of the distribution representative with location 0 and scale 1.

The shape and scale parameters of a Weibull distribution can also be found in closed form. For a gamma distribution, parameters to satisfy the percentile requirements always exist. The parameters are easy to determine numerically but there is no simple expression for them.

For more details, see Determining distribution parameters from quantiles. See also the ParameterSolver software.

Update: I posted an article on CodeProject with Python code for computing the parameters described here.

Related posts:

Biostatistics software
Diagram of distribution relationships
How to calculate percentiles in memory-bound applications

Related post:

The one thing to remember in economics

Diamond oceans
Plants put the bend in rivers
State of biology data integration

Programming

Developer town (little individual houses as offices)
.NET framework install base
Evolution of a Python programmer
How to recognize a good programmer

Miscellaneous

Crayola history
Visa restriction index
LaTeX search
The perverse economics of college construction
Odds that best potential chess player has never played chess
Paying more for information than for food

Universal time (UTC) is the same as Greenwich Mean Time (GMT), give or take a second. It’s essentially the time in Greenwich, England except it ignores Daylight Savings Time.

The abbreviation UTC is an odd compromise. The French wanted to use the abbreviation TUC (temps universel coordonné) and the English wanted to use CUT (coordinated universal time). The compromise was UTC, which doesn’t actually abbreviate anything.

Sometimes a ‘Z’ is appended to a time to indicate it is expressed in UTC. The NATO phonetic alphabet code for ‘Z’ is ZULU, and so UTC is sometimes called “Zulu Time.”

It’s useful to know how your time zone relates to UTC. (You can look it up here.) For example, I live in the US Central time zone. Central Standard Time (CST) is UTC-6, i.e. we’re 6 hours behind Greenwich. Knowing your time relative to UTC makes international communication easier. It also helps you read computer time stamps since these almost always use UTC.

One of the advantages of UTC is that it avoids Daylight Savings Time. DST is surprisingly complicated when you look at it in detail.  Some countries observe DST and some do not. Countries that do observe DST may begin and end DST on different dates, and those dates can change from year to year. And inside countries that observe DST some regions are exceptions. For example, the United States generally observes DST, but Arizona does not. Actually, it’s even more complicated: The Navajo Nation inside Arizona does observe DST.

Related posts:

Mercator projection
Find distances using longitude and latitude
Visual Source Safe and time zones

The fundamental task of statistics is making decisions in the presence of uncertainty, and that’s hard. You have to make all kinds of simplifying assumptions and arbitrary choices to get anywhere. But after a while you lose sight of these decisions. Or you justify your decisions after the fact, making a virtue out of a necessity. After you’ve worked on a problem long enough, it’s nearly impossible to say “Of course, our whole way of thinking about this might have been wrong from the beginning.”

My concern is not so much “creative” statistics but rather uncreative statistics, rote application of established methods. Statistics is extremely conventional. But a procedure is not objective just because it is conventional.  An arbitrary choice made 80 years ago is still an arbitrary choice.

I’ve taken myself too seriously at times in regard to statistical matters; it’s easy to get caught up in your model. But I’m reminded of a talk I heard one time in which the speaker listed a number of embarrassing things that people used to believe. He was not making a smug comparison of how sophisticated we are now compared to our ignorant ancestors. Instead, his point was that we too may be mistaken. He exhorted everyone to look in a mirror and say “I may be wrong. I may be very, very wrong.”

Related posts:

The IOT test
Bike shed arguments
The data may not contain the answer
Problems versus dilemmas
Approximate problems and approximate solutions

The great liability of the engineer compared to men of other professions is that his works are out in the open where all can see them. His acts, step by step, are in hard substance. He cannot bury his mistakes in the grave like the doctors. He cannot argue them into thin air or blame the judge like the lawyers. He cannot, like the architects, cover his failures with trees and vines. He cannot, like the politicians, screen his sort-comings by blaming his opponents and hope the people will forget. The engineer simply cannot deny he did it. If his works do not work, he is damned.

Related posts:

Architects versus engineers
Catalog engineering and reverse engineering

Here’s a similar problem.  Suppose you gave away an electronic book.  You stated that people are free to distribute it however they want, but that you’d appreciate feedback.  How could you estimate the number of people who have read the book?  If you get email from 100 readers, you know at least 100 people have read it, but maybe 10,000 people have read it and only 1% sent email.  How can you estimate the number of readers and the percentage who send email at the same time?

The key is the variability in the data.  You could never tell by looking at the average alone. If there were an average of 10 burglaries per month, the data would be less variable than if there were an average of 20 burglaries per month with only half reported.  Along the same lines, big variations in the amount of fan mail suggest that there may be many readers but few who send email.

The statistical problem is estimating the parameters (n, p) of a binomial distribution. The expected number of reported events is np where n is the total number of events and p is the probability that an event will be reported. Suppose one town has n₁ actual burglaries with each burglary having a probability p₁ of being reported. The other town has n₂ burglaries with a probability p₂ of being reported. If the expected number of reported burglaries are equal, then n₁p₁ = n₂ p₂ = r. The variance in the burglary reports from the two towns will be r(1 – p₁) and r(1 – p₂). If p₁ is less than p₂ there will be more variance in the data from the first city.

Estimating binomial parameters is complicated and I won’t give the details here.  If n were known, estimating p would be trivial. But when n and p are both unknown, this is a much harder problem. Still, there are ways to approach the problem.

The latest Entrepreneurial Thought Leaders podcast features David Heineimeier Hansson’s talk Unlearn Your MBA which he gave to a room full of MBA students.

The latest Tech Nation podcast from IT Conversations is an interview with Jaron Lanier. Lanier is a virtual reality pioneer and the author of You Are Not A Gadget.

Both Hansson and Lanier have contributed to the “free” culture but both are also critical of it. Hansson says he has benefited greatly from open source software and make his Ruby on Rails framework open source as a way to contribute back to the open source community. But he is also scathingly critical of businesses that think they can make money by giving everything away.

Lanier was an early critic of intellectual property rights but has reversed his original position. He says he’s an empiricist and that data have convinced him he was dead wrong. He now says that the idea of giving away intellectual property as advertising bait is unsustainable and will have dire consequences.

Giving away content to make money indirectly works for some businesses. But it’s alarming that so many people believe that is the only rational or moral way to make money if you create intellectual property. Many people are saying things such as the following.

• Musicians should give away their music and make money off concerts and T-shirts.
• Authors should give away their books and make money on the lecture circuit.
• Programmers should give away their software and make money from consulting.

There’s an anti-intellectual thread running through these arguments. It’s a materialistic way of thinking, valuing only tangible artifacts and not ideas. It’s OK for a potter to sell pots, but a musician should not sell music. It’s OK for teachers to make money by the hour for teaching, but they should not make money from writing books. It’s OK for programmers to sell their time as consultants, and maybe even to sell their time as a programmers, but they should not sell the products of their labor. It’s OK to sell physical objects or to sell time, but not to sell intellectual property.

Related post:

Software profitability in the middle
Emily Dickinson versus Paris Hilton
How to avoid being outsourced or open sourced

The soft maximum of x and y is defined by

g(x, y) = log( exp(x) + exp(y) ).

The most obvious way to turn the definition above into C code would be

double SoftMaximum(double x, double y)
{
    return log( exp(x) + exp(y) );
}

This works for some values of x and y, but fails if x or y is large. For example, if we use this to compute the soft maximum of 1000 and 200, the result is numerical infinity. The value of exp(1000) is too big to represent in a floating point number, so it is computed as infinity. exp(200) is finite, but the sum of an infinity and a finite number is infinity. Then the log function applied to infinity returns infinity.

We have the opposite problem if we try to compute the soft maximum of -1000 and -1200. In this computation exp(-1000) and exp(-1200) both underflow to zero, and the log function returns negative infinity for the logarithm of zero.

Fortunately it’s not hard to fix the function SoftMaximum to avoid overflow and underflow. Look what happens when we shift both arguments by a constant.

log( exp(x – k) + exp(y – k) ) = log( exp(x) + exp(y) ) – k.

This says

log( exp(x) + exp(y) ) = log( exp(x -k) + exp(y-k) ) + k

If we pick k to be the maximum of x and y, then one of the calls to exp has argument 0 (and so it returns 1) and the other has a negative argument. This means the follow code cannot overflow.

double SoftMaximum(double x, double y)
{
	double maximum = max(x, y);
	double minimum = min(x, y);
	return maximum + log( 1.0 + exp(minimum - maximum) );
}

The call to exp(minimum - maximum) could possibly underflow to zero, but in that case the code returns maximum. And in that case the return value is very accurate: if maximum is much larger than minimum, then the soft maximum is essentially equal to maximum.

The equation for the soft maximum implemented above has a few advantages in addition to avoiding overflow. It makes it clear that the soft maximum is always greater than the maximum. Also, it shows that the difference between the hard maximum and the soft maximum is controlled by the spread of the arguments. The soft maximum is nearest the hard maximum when the two arguments are very different and furthest from the hard maximum when the two arguments are equal.

Thanks to Andrew Dalke for suggesting the topic of this post by his comment.

Related links:

Soft maximum
Anatomy of a floating point number
Avoiding overflow, underflow, and loss of precision

1. Numerical linear algebra applies very advanced mathematics to solve problems that can be stated with high school mathematics.
2. Practical applications often require solving enormous systems of equations, millions or even billions of variables.
3. The heart of Google is an enormous linear algebra problem. PageRank is essentially an eigenvalue problem.
4. The efficiency of solving very large systems of equations has benefited at least as much from advances in algorithms as from Moore’s law.
5. Many practical problems — optimization, differential equations, signal processing, etc. — boil down to solving linear systems, even when the original problems are non-linear. Finite element software, for example, spends nearly all its time solving linear equations.
6. A system of a million equations can sometimes be solved on an ordinary PC in under a millisecond, depending on the structure of the equations.
7. Iterative methods, methods that in theory require an infinite number of steps to solve a problem, are often faster and more accurate than direct methods, methods that in theory produce an exact answer in a finite number of steps.
8. There are many theorems bounding the error in solutions produced on real computers. That is, the theorems don’t just bound the error from hypothetical calculations carried out in exact arithmetic but bound the error from arithmetic as carried out in floating point arithmetic on computer hardware.
9. It is hardly ever necessary to compute the inverse of a matrix.
10. There is remarkably mature software for numerical linear algebra. Brilliant people have worked on this software for many years.

Related posts:

Don’t invert that matrix
Searching for John Francis
Applying PageRank to biology
Matrix cookbook
What is the cosine of a matrix?

What do you do if you need to solve Ax = b where A is an n x n matrix? Isn’t the solution A^-1 b? Yes, theoretically. But that doesn’t mean you need to actually find A^-1. Solving the equation Ax = b is faster than finding A^-1. Books might write the problem as x = A^-1 b, but that doesn’t mean they expect you to calculate it that way.

What if you have to solve Ax = b for a lot of different b’s? Surely then it’s worthwhile to find A^-1. No. The first time you solve Ax = b, you factor A and save that factorization. Then when you solve for the next b, the answer comes much faster. (Factorization takes O(n³) operations. But once the matrix is factored, solving Ax = b takes only O(n²) operations. Suppose n = 1,000. This says that once you’ve solved Ax = b for one b, the equation can be solved again for a new b 1,000 times faster than the first one. Buy one get one free.)

What if, against advice, you’ve computed A^-1. Now you might as well use it, right? No, you’re still better off solving Ax = b than multiplying by A^-1, even if the computation of A^-1 came for free. Solving the system is more numerically accurate than the performing the matrix multiplication.

It is common in applications to solve Ax = b even though there’s not enough memory to store A^-1. For example, suppose n = 1,000,000 for the matrix A but A has a special sparse structure — say it’s banded — so that all but a few million entries of A are zero.  Then A can easily be stored in memory and Ax = b can be solved very quickly. But in general A^-1 would be dense. That is, nearly all of the 1,000,000,000,000 entries of the matrix would be non-zero.  Storing A requires megabytes of memory, but storing A^-1 would require terabytes of memory.

Related posts:

Floating point numbers are a leaky abstraction
Ten surprises from numerical linear algebra

No font can display all Unicode characters. According to Wikipedia

… it would be impossible to create such a font in any common font format, as Unicode includes over 100,000 characters, while no widely-used font format supports more than 65,535 glyphs.

However, the biggest problem isn’t the number of characters a font can display. Most Unicode characters are quite rare. About 30,000 characters are enough to display the vast majority of characters in use in all the world’s languages as well as a generous selection of symbols. However Unicode fonts vary greatly in their support even for the more commonly used ranges of characters. See this comparison chart. The only range completely covered by all Unicode fonts in the chart is the 128 characters of Latin Extended-A.

Unifont supports all printable characters in the basic multilingual plane, characters U+0000 through U+FFFF. This includes the 30,000 characters mentioned above plus many more. Unifont isn’t pretty, but it’s complete. As far as I know, it’s the only font that covers the characters below U+FFFF.

Related posts:

Why Unicode is subtle

Entering Unicode characters in Windows, Linux

Beautiful architecture video.  No explicit math, but lots of math behind the scenes.

Some math and some great images:  3-DSpirographs

Advanced math with some pictures: algebraic topology books for download from J. P. May and Allen Hatcher. (Hatcher’s book has more pictures.)

Miscellaneous

There’s really nothing that cannot be innovatived

Failure insurance for students

Genes and patents

Why Donald Knuth does not use email

Match the following technologies with the year of their invention.

First the inventions:

1. The computer mouse
2. Radio frequency identification (RFID)
3. Pull-top cans
4. Bar codes
5. Touch tone phones
6. Cell phones
7. Car airbags
8. Automated teller machines (ATM)
9. Magnetic resonance imaging (MRI)
10. Latex paint

Now the years:

1. 1948
2. 1952
3. 1953
4. 1963
5. 1968
6. 1969
7. 1973
8. 1977

Two of the years are used twice. Quiz answers here.

All examples taken from Big Ideas: 100 Modern Inventions That Have Transformed Our World

The home page of the download site allows you to see all products sorted by date or by name. This page also allows search. A new page lets you see the software organized by tags.

f(x, y) = max(x, y)

can be difficult to work with because it has sharp corners. Sometimes you want an alternative that sands down the sharp edges of the maximum function. One such alternative is the soft maximum. Here are some questions this post will answer.

• What exactly is a soft maximum and why is it called that?
• How is it related to the maximum function?
• In what sense does the maximum have “sharp edges”?
• How does the soft maximum round these edges?
• What are the advantages to the soft maximum?
• Can you control the “softness”?

I’ll call the original maximum the “hard” maximum to make it easier to compare to the soft maximum.

The soft maximum of two variables is the function

g(x, y) = log( exp(x) + exp(y) ).

This can be extended to more than two variables by taking

g(x₁, x₂, …,  x_n) = log( exp(x₁) + exp(x₂) + … + exp(x_n) ).

The soft maximum approximates the hard maximum. Why? If x is a little bigger than y, exp(x) will be a lot bigger than exp(y). That is, exponentiation exaggerates the differences between x and y. If x is significantly bigger than y, exp(x) will be so much bigger than exp(y) that exp(x) + exp(y) will essentially equal exp(x) and the soft maximum will be approximately log( exp(x) ) = x, the hard maximum. Here’s an example. Suppose you have three numbers: -2, 3, 8. Obviously the maximum is 8. The soft maximum is 8.007.

The soft maximum approximates the hard maximum but it also rounds off the corners. Let’s look at some graphs that show what these corners are and how the soft maximum softens them.

Here are 3-D plots of the hard maximum f(x, y) and the soft maximum g(x, y). First the hard maximum:

Now the soft maximum:

Next we look at a particular slice through the graph. Here’s the view as we walk along the line y = 1. First the hard maximum:

And now the soft maximum:

Finally, here are the contour plots. First the hard maximum:

And now the soft maximum:

The soft maximum approximates the hard maximum and is a convex function just like the hard maximum. But the soft maximum is smooth. It has no sudden changes in direction and can be differentiated as many times as you like. These properties make it easy for convex optimization algorithms to work with the soft maximum. In fact, the function may have been invented for optimization; that’s where I first heard of it.

Notice that the accuracy of the soft maximum approximation depends on scale. If you multiply x and y by a large constant, the soft maximum will be closer to the hard maximum. For example, g(1, 2) = 2.31, but g(10, 20) = 20.00004. This suggests you could control the “hardness” of the soft maximum by generalizing the soft maximum to depend on a parameter k.

g(x, y; k) = log( exp(kx) + exp(ky) ) / k

You can make the soft maximum as close to the hard maximum as you like by making k large enough. For every value of k the soft maximum is differentiable, but the size of the derivatives increase as k increases. In the limit the derivative becomes infinite as the soft maximum converges to the hard maximum.

Update: See How to compute the soft maximum.

Related posts:

Convex optimization
Free optimization software from Microsoft

… we have left undone those things which we ought to have done, and we have done those things which we ought not to have done.

The things left undone are called sins of omission; things which ought not to have been done are called sins of commission.

In software testing and debugging, we focus on sins of commission, code that was implemented incorrectly. But according to Robert Glass, the majority of bugs are sins of omission. In Frequently Forgotten Fundamental Facts about Software Engineering Glass says

Roughly 35 percent of software defects emerge from missing logic paths, and another 40 percent are from the execution of a unique combination of logic paths.

If these figures are correct, three out of four software bugs are sins of omission, errors due to things left undone. These are bugs due to contingencies the developers did not think to handle. Three quarters seems like a large proportion, but it is plausible. I know I’ve often written plenty of bugs that amounted to not considering enough possibilities, particularly in graphical user interface software. It’s hard to think of everything a user might do and all the ways a user might arrive at a particular place. (When I first wrote user interface applications, my reaction to a bug report would be “Why would anyone do that?!” If everyone would just use my software the way I do, everything would be OK. )

It matters whether bugs are sins of omission or sins of commission. Different kinds of bugs are caught by different means. Developers have come to appreciate the value of unit testing lately, but unit tests primarily catch sins of commission. If you didn’t think to program something in the first place, you’re not likely to think to write a test for it. Complete test coverage could only find 25% of a projects bugs if you assume 75% of the bugs come from code that no one thought to write.

The best way to spot sins of omission is a fresh pair of eyes. As Glass says

Rigorous reviews are more effective, and more cost effective, than any other error-removal strategy, including testing. But they cannot and should not replace testing.

One way to combine the benefits of unit testing and code reviews would be to have different people write the unit tests and the production code.

Related posts:

The most subtle of the seven deadly sins
Shallow bugs versus reported bugs
Negative space in operating systems

Abstractions are never perfect. Every new layer creates failure points, interoperability hassles, and scalability problems. New tools can hide complexity, but they can’t justify it … The more complex the system, the more difficult it is to fix when something goes wrong.

From the preface to RESTful Web Services.

Related posts:

Obscuring complexity
Baklava code
Leaky abstractions
Less isn’t more; just enough is more