Paul Legato

Dell PowerEdge BIOS Upgrade on Linux

Paul Legato — Sat, 31 Jul 2010 05:13:59 +0000

Dell only provides certain PowerEdge BIOS flash upgrades in the form of Windows-only self-extracting archives (despite the facts that they claim to officially support Linux and that the BIOS flash utility itself is a DOS program.) Ordinary unzip won’t open the self-extracting archive. You need to run it on a Windows system, which I don’t have. Here’s how to flash your PowerEdge BIOS without Windows.

Dell provides the biosdisk utility to make a bootable FreeDOS image with the BIOS upgrade executable on it, but it doesn’t work with the Windows-only PowerEdge 4400 BIOS upgrade, since that won’t run on FreeDOS. They claim that biosdisk isn’t intended for PowerEdge systems, since they all have Linux BIOS flashes available; this does not seem to actually be the case for the 4400.

The Windows self-extractor does run under WINE, but it demands an actual floppy drive device to write to, which my computer doesn’t have. WINE also doesn’t seem to support mounting an image file as a floppy device, for some reason. You’ll need to use something like VMWare or a computer with a physical floppy drive to get this to extract itself.

Once you have a floppy image, here’s how to flash without biosdisk or Windows available:

Use WINE to extract the archive. For maximum fun, the upgrade I have refuses to just extract itself; instead, it requires that it has access to a floppy drive to write to directly. Sigh. Ok. Working on that. Updates to come. winecfg
Install syslinux. (apt-get install syslinux). We are going to use memdisk, a boot loader that creates a RAM drive and then loads an operating system, to load FreeDOS.
Get a FreeDOS boot disk image and add your flashing .exe to it.Mount with something like mount -t vfat /path/to/freedos.img /mnt -o loop, then just cp your flashing .exe to /mnt. (If you got a gzipped image, gunzip it first.
Edit /etc/grub.d/40_custom to add a boot menu entry for memdisk/FreeDOS.
```
menuentry "FreeDOS" {
    set root=(hd0,2) #### <--- Change this to your /boot partition!
    linux16 /memdisk floppy
    initrd16 /freedos.img
}
```
As noted, change the root entry to point to your actual /boot partition, so GRUB can find the images. Also note that the partition normally mounted at /boot will be mounted as the root directory (/) at boot time, so it’s not necessary to prefix /boot before the image file names.

Then, reboot and select FreeDos from the GRUB menu. Run your DOS-based flash program, and all is well.

Print PDF

Swing-Clojure GUI for the Black-Scholes Option Modeler

Paul Legato — Thu, 15 Jul 2010 07:55:56 +0000

Now that we have implemented Black-Scholes in Clojure, let’s make a Swing GUI for it. The Swing GUI will have text boxes for all the necessary inputs, and calculate prices and Greeks when the button is pressed. It’s a simple and straightforward way to get started in Swing GUI programming in Clojure. Here’s what it looks like.

The GUI is written entirely in Clojure with the Swing toolkit. Calculation state is stored in a series of atoms. Watches are used to update the output table automatically when an atom changes, an idea from Kotka. I used the excellent MiG Layout for general layout functionality, and generic Swing widgets (JTextFields and a JTable) for the input and output.

Clojure code

The Black-Scholes GUI is launched with themain function. The Swing GUI is actually created in the initialize-gui function, but since Swing widgets are not threadsafe, we must use the do-swing* helper function from Clojure Contrib to spin off the real GUI creation code into the Swing event thread.

(defn initialize-gui
  []
  (let [frame (JFrame. "Black-Scholes Option Modeler")]

    (doto frame
      (.setDefaultCloseOperation JFrame/DISPOSE_ON_CLOSE)
      (-> .getContentPane
          (.add (miglayout (JPanel.)

                           (input-panel
                            (fn [_]
                              (do-swing
                               (doto frame
                                 (.setVisible false)
                                 (.dispose)))))

                           (result-table)

                           )))

      (.setDefaultCloseOperation JFrame/DISPOSE_ON_CLOSE)
      (.pack)
      (.setVisible true))
     ))

(defn main
  []
  (do-swing* :now initialize-gui))

This is pretty straightforward. We make a (locally scoped) JFrame to be the main container at line 3. Within the context of the let, we add a JPane using MiG Layout to its content area (starting at line 8). The JPanel itself contains two subwidgets, which are those returned from the input-panel and result-table functions.

Note that we are passing an anonymous function to input-panel at line 11. The input panel contains the “Quit” button, and that button needs to know what to do when it’s pressed. We want it to close the GUI; that is, it should destroy the outer JFrame. To do that, it needs a reference to it. However, as a local variable in initialize-gui, the JFrame is not in scope elsewhere; we have to pass it in. The given function closes over the JFrame reference (line 13) and transports it to input-panel, where that function will be called as an event listener when the “Quit” button is pressed.

We could also have simply passed the bare JFrame rather than a function, and moved the frame-closing code into input-panel, but that would imply that the input panel must always be in a JFrame, which will not necessarily the case. Passing in an external anonymous function to be activated by the “Quit” button allows the input panel to be used generically in any sort of container, with the container deciding what to do when “Quit” is pressed.

The Input Panel


(defn- input-panel [quit-action]
    (let [
          ;; Fields
          spot (doto (JFormattedTextField. (NumberFormat/getNumberInstance))
                 (.setColumns 6)
                 (.setValue @*spot*))

          strike (doto (JFormattedTextField. (NumberFormat/getNumberInstance))
                   (.setColumns 6)
                   (.setValue @*strike*))

          days-till-expiry (doto (JFormattedTextField. (NumberFormat/getIntegerInstance))
                             (.setColumns 6)
                             (.setValue @*days-till-expiry*))

          riskfree (doto (JFormattedTextField. (NumberFormat/getNumberInstance))
                     (.setColumns 6)
                     (.setValue @*riskfree*))

          volatility (doto (JFormattedTextField. (NumberFormat/getNumberInstance))
                       (.setColumns 6)
                       (.setValue @*volatility*))

          ;; Buttons
          calculate (JButton. "Calculate")
          quit (JButton. "Quit")
          ]

      (add-action-listener
       calculate
       (fn [_]
         (update-all-calculations
          (Double/parseDouble (.getText spot))
          (/ (Integer/parseInt (.getText days-till-expiry)) *trading-days-per-year*)
          (Double/parseDouble (.getText strike))
          (Double/parseDouble (.getText riskfree))
          (Double/parseDouble (.getText volatility)))))

      (add-action-listener
       quit
       quit-action)

      (miglayout (JPanel.)
                 :layout  [:wrap 2 ]

                 (JLabel. "Spot") [:align "right"]
                 spot

                 (JLabel. "Strike") [:align "right"]
                 strike

                 (JLabel. "Risk-free rate") [:align "right"]
                 riskfree

                 (JLabel. "Volatility") [:align "right"]
                 volatility

                 (JLabel. "Days till expiry") [:align "right"]
                 days-till-expiry

                 calculate
                 quit)))

The input widgets are local variables in the input panel function, along with the two buttons. The widgets are initialized from the state atoms (which are then unused again; see discussion below.) The “Quit” button is given the event callback passed in as an argument (so it can manipulate the enclosing JFrame as per above.) The “Calculate” button is given an event callback that calls the update-all-calculations function with the values in the input widgets.

The lot are then stuffed into a JPanel with a MiG Layout and returned.

The Output Table

(defn- update-cell
  "Updates the given AbstractTableModel cell and fires a change event for that cell."
  [model data row col]
  (.setValueAt model data row col)
  (.fireTableCellUpdated model row col))

(defn- result-table []
  (let [
        column-names ["" "Call" "Put"]
        row-names ["Price" "Delta" "Theta" "Rho" "Gamma" "Vega"]
        table-model (proxy [AbstractTableModel] []
                      (getColumnCount [] (count column-names))
                      (getRowCount [] (count row-names))
                      (isCellEditable [] false)
                      (getColumnName [col] (nth column-names col))
                      (getValueAt [row col]
                                  (condp = col
                                      0 (nth row-names row) ;; Return the row name for column zero
                                      (condp = [row col]
                                          [0 1] @*call-price*
                                          [0 2] @*put-price*

                                          [1 1] @*call-delta*
                                          [1 2] @*put-delta*

                                          [2 1] @*call-theta*
                                          [2 2] @*put-theta*

                                          [3 1] @*call-rho*
                                          [3 2] @*put-rho*

                                          [4 1] @*gamma*
                                          [4 2] "=="

                                          [5 1] @*vega*
                                          [5 2] "=="

                                          nil))))

        table (doto (JTable. table-model)
                (.setGridColor java.awt.Color/DARK_GRAY))
        ]

    (add-watch *call-price* ::update-call-price (fn [_ _ _ newprice] (update-cell table-model newprice 0 1)))
    (add-watch *put-price* ::update-put-price (fn [_ _ _ newprice] (update-cell table-model newprice 0 2)))

    (add-watch *call-delta* ::update-call-delta (fn [_ _ _ newval] (update-cell table-model newval 1 1)))
    (add-watch *put-delta* ::update-put-delta (fn [_ _ _ newval] (update-cell table-model newval 1 2)))

    (add-watch *call-theta* ::update-call-theta (fn [_ _ _ newval] (update-cell table-model newval 2 1)))
    (add-watch *put-theta* ::update-put-theta (fn [_ _ _ newval] (update-cell table-model newval 2 2)))

    (add-watch *call-rho* ::update-call-rho (fn [_ _ _ newval] (update-cell table-model newval 3 1)))
    (add-watch *put-rho* ::update-put-rho (fn [_ _ _ newval] (update-cell table-model newval 3 2)))

    (add-watch *gamma* ::update-gamma (fn [_ _ _ newval] (update-cell table-model newval 4 1)))
    (add-watch *vega* ::update-vega (fn [_ _ _ newval] (update-cell table-model newval 5 1)))

    ;; This shrinks the table's preferred viewport down to its actual size.
    ;; (The default is to make a huge viewport, even though the table is small.)
    (.setPreferredScrollableViewportSize table (.getPreferredSize table))

    (JScrollPane. table)
    ))

The output table itself is a standard JTable. Its model is initialized from the output state atoms. The watches at lines 44-57 are the most interesting part; they allow arbitrary code to be called when the output state atoms change. This allows the Black-Scholes calculation to happen independently of the output widget. When one of those atoms is changed, it self-updates automatically via the watches.

The helper function at lines 1-5 updates the appropriate model cell and fires an event to tell the JTable to repaint it.

The scrollable viewport business at line 61 is to work around a questionable Java design decision where a JTable, by default, tells its container to make its viewport enormous, even though it have very little data. (That’s not an error, that’s subjunctive.)

Design considerations

GUI and business logic decoupling

Above all, the GUI has to be decoupled from the model itself. Internal changes to one should never affect the other, and the two should be independently testable. In such a simple application, this was easy enough. A more complex app that requires several information round trips would be trickier. We now have a good foundation for such an app. No business calculations are performed in the GUI code namespace. The sole point of connection is in the update-all-calculations function:


(defn- update-all-calculations [spot timeleft strike riskfree sigma]
  (swap! *call-price* (fn [_] (bs/call spot timeleft strike riskfree sigma)))
  (swap! *put-price* (fn [_] (bs/put spot timeleft strike riskfree sigma)))

  (swap! *call-delta* (fn [_] (bs/call-delta spot timeleft strike riskfree sigma)))
  (swap! *put-delta* (fn [_] (bs/put-delta spot timeleft strike riskfree sigma)))

  (swap! *call-theta* (fn [_] (bs/call-theta spot timeleft strike riskfree sigma)))
  (swap! *put-theta* (fn [_] (bs/put-theta spot timeleft strike riskfree sigma)))

  (swap! *call-rho* (fn [_] (bs/call-rho spot timeleft strike riskfree sigma)))
  (swap! *put-rho* (fn [_] (bs/put-rho spot timeleft strike riskfree sigma)))

  (swap! *vega* (fn [_] (bs/vega spot timeleft strike riskfree sigma)))
  (swap! *gamma* (fn [_] (bs/gamma spot timeleft strike riskfree sigma))))

Atoms to hold state

As we can see above, set of atoms holds state for the GUI, in keeping with the general Clojure principle of isolating state into transactional memory. (Actually, I cheated a bit. Our full state consists of the input values and the results of our Black-Scholes calculations. Although I created atoms for the input state values as well, I calculate the results directly from the widgets. It is a trivial exercise to alter the “Calculate” button callback to use the atoms. This would be more useful in the case where changes to a widget fire an event that triggers recalculation, which is not done in this demo as per below. I would use the atoms in a real application.)

I/O is inherently stateful, as are GUIs. There was therefore the temptation to use the Swing widget objects themselves as state-holders, and in fact it requires extra code to shuttle data back and forth between the widgets and the atoms. In such a simple app as this, using the GUI widgets as state-holders does not make much practical difference, but in a more complex app, a well designed set of state atoms and watches can automate much of the tedious GUI logic.

Using the widgets directly as state holders implies a tight coupling between the input widgets, the business logic, and the output widgets. Changing any one of them would invariably have required changes to the linking logic.

For example, suppose we were using widgets-as-state, and we changed the “days left till expiry” input to a slider rather than a text box, or suppose we wanted to read it from a file or from an API. We would have to update the code that reads it, calculates the model, and puts the results into the output table. Touching that code implies a non-zero likelihood of breaking it. Why even introduce the potential for breaking something that is unrelated to the task at hand, in this case the business logic and output code?

Again, this seems trivial with such a very small application that is only a few lines long, but the design principle is what’s important. Imagine applying it to a much larger application with hundreds of inputs spread across 4 windows, 3 databases, and 6 web feeds to see why this is a good idea.

Auto-updating results when input changes

I wanted the results to auto-update anytime an input is changed, without resorting to a “Calculate” button. Having the button trigger the calculation and output update allows the state of the input and the output to become decoupled, which can be confusing for the user. Worse, it could be unnoticed by a user who is doing other things at the same time.

Due to typical Java overengineering, there is no straightforward way to simply (add-action-listener) on the JTextField and get notifications when it changes. Instead, each Swing widget implements full-blown model-view separation, and so you have to get the document underlying the JTextField. Yes, the view components of your MVC application themselves contain both models and views. Each and every field contains an entire implementation of the document object and associated cruft, all for itself. If you want to monitor changes in the widget, you need to build out a DocumentListener to watch for such events.

In a real application, I’d bite the bullet and implement it, but it’s not worth the extra time for the purposes of this demo, so we are left with a “Calculate” button and manual updating.

Clojure-Options on Github

The entire package so far is now available on my GitHub page. Patches are welcome. Atomizing the input widgets and adding the appropriate event callbacks and watches to eliminate the “Calculate” button is particularly welcome.

Print PDF

Black-Scholes in Clojure

Paul Legato — Sun, 11 Jul 2010 00:51:27 +0000

The Black-Scholes option pricing model, implemented in Clojure based on the description at Wikipedia and these code samples.

;;
;; Black-Scholes option pricing model
;;
;; Copyright (C) 2010 Paul Legato. All rights reserved.
;; Licensed under the New BSD License. See the README file for details.
;;
;; Disclaimer: This code comes with NO WARRANTY, express or implied.
;; There may be any number of bugs. Use at your own risk.
;;
;; References:
;; - https://secure.wikimedia.org/wikipedia/en/wiki/Black%E2%80%93Scholes
;; - http://www.espenhaug.com/black_scholes.html

(ns clojure-options.black-scholes
  (:require incanter.distributions))

(defn d1
  [spot timeleft strike riskfree sigma]
  (/
   (+ (Math/log (/ spot strike))
      (* (+ riskfree (/ (* sigma sigma) 2))
         timeleft))
   (* sigma (Math/sqrt timeleft))))

(defn d2
  [spot timeleft strike riskfree sigma]
  (- (d1 spot timeleft strike riskfree sigma) (* sigma (Math/sqrt timeleft))))

(defn- n [val] (incanter.distributions/cdf (incanter.distributions/normal-distribution) val))

(defn call
  "Returns the theoretic value of a European call option on the given underlying based on the Black-Scholes model.

 * spot - spot price of the underlying
 * timeleft - time to expiration, in years
 * strike - option strike price
 * riskfree - annual continuous risk-free interest rate
 * sigma - volatility of the underlying

"
  [spot timeleft strike riskfree sigma]
  (- (* spot (n (d1 spot timeleft strike riskfree sigma)))
     (* strike
        (Math/exp (* (- riskfree) timeleft))
        (n (d2 spot timeleft strike riskfree sigma)))))

(defn put
  "Returns the theoretic value of a European put option on the given underlying based on the Black-Scholes model.

 * spot - spot price of the underlying
 * timeleft - time to expiration, in years
 * strike - option strike price
 * riskfree - annual continuous risk-free interest rate
 * sigma - volatility of the underlying

"
  [spot timeleft strike riskfree sigma]
  (- (* strike
        (Math/exp (* (- riskfree) timeleft))
        (n (- (d2 spot timeleft strike riskfree sigma))))
     (* spot (n (- (d1 spot timeleft strike riskfree sigma))))))

(defn call-delta
  [spot timeleft strike riskfree sigma]
  (n (d1 spot timeleft strike riskfree sigma)))

(defn put-delta
  [spot timeleft strike riskfree sigma]
  (- (n (d1 spot timeleft strike riskfree sigma)) 1))

(defn- nprime [val] (incanter.distributions/pdf (incanter.distributions/normal-distribution) val))

(defn gamma
  [spot timeleft strike riskfree sigma]
  (/ (nprime (d1 spot timeleft strike riskfree sigma))
     (* spot sigma (Math/sqrt timeleft))))

(defn vega
  [spot timeleft strike riskfree sigma]
  (* spot (nprime (d1 spot timeleft strike riskfree sigma)) (Math/sqrt timeleft)))

(defn call-theta
  [spot timeleft strike riskfree sigma]
  (-
   (- (/ (* spot (nprime (d1 spot timeleft strike riskfree sigma)) sigma)
         (* 2 (Math/sqrt timeleft))))
   (* riskfree strike (Math/exp (* (- riskfree) timeleft)) (n (d2 spot timeleft strike riskfree sigma)))))

(defn put-theta
  [spot timeleft strike riskfree sigma]
  (-
   (- (/ (* spot (nprime (d1 spot timeleft strike riskfree sigma)) sigma)
         (* 2 (Math/sqrt timeleft))))
   (* riskfree strike (Math/exp (* (- riskfree) timeleft)) (n (- (d2 spot timeleft strike riskfree sigma))))))

(defn call-rho
  [spot timeleft strike riskfree sigma]
  (* strike timeleft (Math/exp (* (- riskfree) timeleft)) (n (d2 spot timeleft strike riskfree sigma))))

(defn put-rho
  [spot timeleft strike riskfree sigma]
  (* (- strike) timeleft (Math/exp (* (- riskfree) timeleft)) (n (- (d2 spot timeleft strike riskfree sigma)))))

This code uses the new Incanter Distributions module, which hasn’t been released yet, so it requires the latest master version from Github.

Most of the values agree with those produced at BloBek’s calculator, with the exception of theta. I imagine this has to do with the units used for the “days left to expiration” field; my calculator uses fractions of a year, while BloBek doesn’t specify whether his days are to be trading days or calendar days. It could also be a bug in one of our implementations.

In Part 2, I’ll build a simple Swing GUI for the calculator.
Update: Part 2, with the Swing GUI, is now available. You can also get the entire project’s source code from GitHub.

Print PDF

A Rebuttal for Kartik Athreya

Paul Legato — Tue, 29 Jun 2010 11:19:48 +0000

Kartik Athreya’s article deriding the unwashed, mouth-breathing public for presumptuously daring to have an opinion about his area of expertise is making the rounds, predictably being slammed by those of us with the audacity to hold an opinion about some subject in which we lack a PhD. I list reasons why professional “expert” economists are useless or even dangerous, and why Athreya’s real problem is American culture.

Economists’ general failure to have made many useful predictions in the past, as a profession. How many trained, qualified PhD economists warned us about the onset of the Great Recession in 2006? Right. About as many as predicted the Great Depression or the Panic of 1873 or the end of the gold standard or the ’70s oil shocks or the ’87 stock market crash or …
I own Business Fluctuations, a 1952 textbook by UC Berkeley economics professor Robert Aaron Gordon, which has a quaint section entitled “IS THE BUSINESS CYCLE OBSOLETE?”

How about bona fide, credentialed economist Irving Fisher‘s famous 1929 pronouncement that “Stock prices have reached what looks like a permanently high plateau”?

It’s not just that one or a few fringe economists occasionally make wacky analyses of the economy that turn out to be totally false; the profession as a whole does it, and has been doing it repeatedly for as long as anyone can remember. These guys were not fringe nutjobs. These views were reasonably mainstream at the time they expressed them.

Theories that do not make falsifiable, testable predictions are pseudoscience. A theory cannot explain every possible outcome, else there is no way to test it and it is not falsifiable.

Embracing their theories as expert advice even after their repeated failure to demonstrate any accurate predictive ability is worse than useless. It’s dangerous. Remember that Alan Greenspan guy? What was his profession, again?
The vast incidence of massively contradictory economic paradigms held by various “experts”. Keynes versus Friedman springs to mind at once, and let’s not even touch Marx or Soros. Efficient market hypothesis, anyone? Ask 10 economists today merely whether and why we will see deflation or inflation in the US, and you will get 10 contradictory, mutually exclusive answers.
To extend Athreya’s cancer analogy, it’s as if one oncologist expert says cancer is caused by genes and requires gene therapy; another says it’s caused by fluoride in the water and needs wholesale tooth extraction; a third says it’s caused by ragweed pollen and just requires an antihistamine; a fourth says the patient’s bodily humours are merely imbalanced, but bloodletting with leeches is a sure cure in his vast and storied professional experience; and yet another says that cancer is caused by Mars having passed too close to Sagittarius on the patient’s birthday, so nothing can be done except perhaps a sacrifice to the gods. And still none of them can predict with any accuracy at all who will get cancer, much less tell you how to cure it with better-than-chance accuracy.

Little respect would naturally develop for such a profession, and members of the public would feel well qualified to add their own opinions and contributions, since they couldn’t possibly do much worse than the self-designated “professionals”. And indeed, that was exactly how medicine worked in premodern times, until doctors began to put forth testable predictions that actually accurately predicted future conditions and developed rigorous, repeatable experiments to test them. Economics has not yet achieved any measure of remotely reliable prediction. Economics is today a protoscience at best.

Athreya himself says, “For my part, seventeen years after my ﬁrst PhD coursework, I still feel ill at ease with my grasp of many issues, and I am fairly conﬁdent that this is not just a question of limited intellect.” (This false humility is ostentatious and self-serving; it was written to say, “hey, this is so hard that even us experts don’t know what’s going on, so you amateurs should just all be quiet.”)

Some amateurs, like Soros, have consistently done far, far better than the pros. The field is simply extremely immature and not capable of providing very good answers to anyone. “Experts” are not special in that regard. They’re barely more aware of what’s going on than anyone else, and often they develop emotional attachment to theories that they have invested professional capital in, resulting in tunnel vision. When they can consistently predict significant economic events before they happen, and consistently do so better than amateurs, then they can begin to merit the respect accorded to oncologists.

As Nicholas Taleb says:

The environment in financial economics is reminiscent of medieval medicine, which refused to incorporate the observations and experiences of the plebeian barbers and surgeons. Medicine used to kill more patients than it saved – just as financial economics endangers the system by creating, not reducing, risk. But how did financial economics take on the appearance of a science? Not by experiments (perhaps the only true scientist who got the prize was Daniel Kahneman, who happens to be a psychologist, not an economist). It did so by drowning us in mathematics with abstract “theorems”. Prof Merton’s book Continuous Time Finance contains 339 mentions of the word “theorem” (or equivalent). An average physics book of the same length has 25 such mentions. Yet while economic models, it has been shown, work hardly better than random guesses or the intuition of cab drivers, physics can predict a wide range of phenomena with a tenth decimal precision.
Economic policy requires concerted action by all, and its results have vast consequences for everyone. A choice of several potential cancer treatments is between the patient and his doctor. If oncologists disagree about which treatment is best, concluding a decision as to which to go with requires convincing at most a small handful of people: the patient and his family and close friends. The consequences of that choice will also rarely radiate much farther than that.
Economic policy, on the other hand, requires the concerted efforts of substantially all of the population, via government agencies and their use (or lack thereof) of tax money (at least in a democracy.) Ordinary people don’t know or care about the abstruse details; they just want to know how it will affect them. Enter politicians and big businesses, who, needless to say, do not always have pure motives. Public opinion can be manipulated easily.

Given that the choice of which policies to implement is a political choice that ultimately rests with the entire population; given that self-interested politicians, investment banks, and other capital institutions are actively seeking to manipulate that opinion for their own benefit; and given the gravity of policy choice ramifications for each and every member of society, Athreya’s position of “just leave it all to us experts and don’t insult us by trying to think about it” begins to seem positively Orwellian in tone.
Perverse incentives reward bland economic thinking by the professionals; the “cover your ass” syndrome.
Working professional economists are not rewarded for making accurate predictions about the future of the economy; they are rewarded for making safe predictions about it. (Safe for them, not for the economy.) You can see this in action every time new economic indicators come out. Bloomberg and the other financial news media run stories with “consensus estimates” which do not usually deviate very much from each other. When there’s a surprise figure, there is rarely a pro willing to go on record and call it in advance.

Why? They have large career incentives to be safely mainstream and risk averse. The reward for correctly predicting an outlier economic data point is relatively small; maybe the financial press will perfunctorily laud you for half a day at most. The penalty for making an on-the-record inaccurate prediction of an outlier event is large in professional terms; other economists will suspect that you’re ignorant of the field’s basics. On the other hand, there is no penalty for making a safe prediction that is in line with the mainstream view. If the mainstream theory is right, all the better; naturally, everyone’s predictions will converge. If the mainstream theory is wrong and an outlier event occurs, no harm done; everyone else was wrong, too. How could anyone have known better?

The same general process plays out in academia. A graduate student will not get a PhD, and a new professor will not get tenure, for making earth-shattering predictions or proposing radical new theories. She gets the PhD and tenure for making very minor improvements to some existing body of theory, fleshing out a few small details here and there, and flattering the preconceptions of a committee who have all invested decades of their lives into some particular paradigm of economics.
Cultural factors related to perceptions of formal credentials. Many cultures in the world adore fancy titles and formal credentials. America is not one of them.
“All men are created equal,” reads our founding document, the Declaration of Independence. It was written by upstart bourgeois traders and lawyers who were, in part, greatly annoyed that their potential for social advancement was limited in British society due to lack of aristocratic birth and an Oxford diploma. They came to America to live somewhere where that did not matter much.

We are hardline egalitarians, proudly derisive of highfalutin institutional authority at every turn. The “experts” are a self-selected bunch who can often contribute something useful. Insofar as they can occasionally do so, great, we’ll keep them around. Just as often, they seek merely to guard their own status as elite members of an exclusive club, to defend their turf from outsiders. Not infrequently, they devolve into groupthink and dogmatism, eventually stifling progress if left unchecked. Mainstream society does not really care, because it pays little to no attention to them — though they are very often totally unaware of this and consider themselves important figures in society.

I think this anti-authoritarian attitude is in no small part exactly what has driven American successes in so many fields of endeavor over the centuries. Sure, there’s nothing wrong with having a PhD here, and you can gain wonderful amounts of knowledge by doing so, which is great. But as far as anyone outside the narrow specialized field is concerned, that doctor’s theories are judged on their own merits, not on reputation or fancy titles. Moreover, we in no way whatsoever feel the need to refrain from commenting on a matter simply because we are not “experts.” (This does not mean that the laymen are necessarily or even often right, of course, which is another matter.) We certainly don’t refrain from criticizing some “experts” just because they are highly placed at a respected institution and bear its seal of approval.

This is very much not the case in, say, Germany or China. Institutions there exist to imbue people with the right to have an opinion, and only after arduous study and prolonged ritualized vetting by the pre-existing high priests of the field. Everyone else simply shuts up and listen to them, because they know that they are not qualified to have an opinion on a complex matter without a PhD in hand.

Immigrants to America, and sometimes their children, naturally carry a cultural legacy from their home country with them. Those who bear fancy degrees and come from countries with high regard for institutionalized authority often have a hard time adapting to the fact that Americans just don’t care about fancy degrees and titles very much. They feel stilted and insulted. Often they don’t even notice the cultural correlation, or if they do they dismissively condemn it as merely an attitude for the ignorant and the stupid, those who are simply too unenlightened to realize the vast power of institutional blessing for one’s opinions.

I think this is the ultimate source of Athreya’s frustration. He’s annoyed that Americans are not exhibiting the proper deference to his institutionally sanctioned writ of authority to hold opinions about economics. His borderline-abusive derision of economics bloggers certainly supports this theory. How dare a “naif” layman hold a view that contradicts his, the great doctor, when he has spent so many years studying, and read so many books! He even goes to the trouble to be affectedly humble, disclaiming having a firm grasp of the subject himself so as to demonstrate just how tough it really is.

I am unaware of his exact cultural provenance, but “Kartik” is the name of a month in Bengali. Indian subcontinent cultures exhibit an extreme case of this appeal-to-authority automatic reverence for institutional knowledge approval. He is trying to impose Bengali cultural norms (or wherever he’s actually from) upon Americans, a curious case of “reverse cultural imperialism”. Perhaps he grew up here, considers himself an American, and doesn’t even realize this cultural dimension himself.

None of this should be construed as a rejection of the formal academic process. Having a PhD is fine, and many PhDs do useful work. There are even many knowledgeable, useful professional PhD-bearing economists! The problem comes when the PhD is not taken as a marker of knowledge but rather as a cudgel with which to bludgeon those lesser beings who somehow have the audacity to fail to obey the expert’s opinion. This becomes a farce when done in a field like economics where there is no certain knowledge at all, a long track record of totally failing to predict major and devastating events clearly declared to be within the field’s purview, and indeed relatively few predictions that can be construed as “scientific” in any sense.

Print PDF

Lazy-spy: Clojure logging/spy for lazy sequences

Paul Legato — Mon, 24 May 2010 02:27:16 +0000

Clojure.contrib’s logging/spy macro is really useful for quick investigation of bugs with unclear stack traces, and to check that intermediate values in a calculation are being returned and composed correctly. Lazy sequences don’t get logged correctly, however:

> (log/spy [1 2 3 4 5])
[1 2 3 4 5] ;;; prints " [1 2 3 4 5] => [1 2 3 4 5]" to the log
> (log/spy (take 3 [1 2 3 4 5]))
(1 2 3) ;;; prints "(take 3 [1 2 3 4 5]) => clojure.lang.LazySeq@8592" to the log

The spy macro logs a string realization of the sequence per se, rather than its constituents. In the case of a concrete, fully realized sequence such as the literal vector in the first example, all values are known in advance, and the vector’s string realization in the log can helpfully be its actual contents. In the case of the lazy sequence returned by take, however, we get a less than helpful class name and some sort of instance identifier code rather than the actual contents of the sequence. A lazy sequence saves resources by merely promising to provide the values when asked. If they’re never asked for, they’re never computed. Since they haven’t been computed, they can’t be printed. A lazy sequence can also potentially be infinitely long, another good reason why they are not realized by default when converting the sequence to a string.

In our case, we don’t care about the resource usage, and we’re sure the sequence is finite. We want to force all values to be computed for manual inspection. (There are several ways of going about this.) The following modification of the spy macro converts the lazy sequence into a regular seq:

(defmacro lazy-spy
  "Evaluates expr and outputs the form and its result to the debug log.
   If the result is a lazy sequence, it is realized concretely in the log.
   Returns the result of expr."
  [expr]
  `(let [a# ~expr] (log/log :debug (str '~expr " => "
                                    (if (= clojure.lang.LazySeq (class a#))
                                      (seq a#)
                                       a#)))
        a#))

This gives us:

> (lazy-spy  (take 3 [1 2 3 4 5]))
(1 2 3) ;;; prints "(take 3 [1 2 3 4 5]) => (1 2 3)" to the log

Print PDF

Log4J and Clojure Configuration

Paul Legato — Sun, 16 May 2010 22:12:48 +0000

Clojure, Log4J, and clojure.contrib.logging can play nicely together. Here’s a reasonable default configuration.

If using Leiningen, add

:dependencies [
                 [log4j "1.2.15" :exclusions [javax.mail/mail
                                              javax.jms/jms
                                              com.sun.jdmk/jmxtools
                                              com.sun.jmx/jmxri]]
]

to your project.clj file and rerun lein deps to auto-install the Log4J JAR.

If not using Leiningen, acquire a Log4J JAR and put it on your classpath.

Create a file called log4j.properties on your classpath and put the following into it:

# Based on the example properties given at http://logging.apache.org/log4j/1.2/manual.html
# Set root logger level to DEBUG and its only appender to A1.
log4j.rootLogger=DEBUG, A1

# A1 is set to be a ConsoleAppender.
log4j.appender.A1=org.apache.log4j.ConsoleAppender

# A1 uses PatternLayout.
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern= %-5p %c - %m%n

That’s the short version. Read on for the gory details.

Logging philosophy

My logging requirements are simple at the moment. I want to write messages to the console, each tagged with a log level (“INFO”, “WARN”, “ERROR”, etc.) I want to be able to set a log level, and only see log messages of that severity or higher. That’s it. I take this to be the canonical base case of logging that covers 80% of needs. All other logging features (e.g. logging to a file, differentiating by class or namespace) are nice, but much less common, and ought to be handled via configurable deviations from this default case.

clojure.contrib.logging does not presently make it easy to do any of this common base case out of the box. The Lispy façade quickly falls apart; you’re dumped into the deep end of the Java overengineering mire without a life vest when you simply want to change the log level temporarily.

What’s wrong with clojure.contrib.logging’s defaults?

clojure.contrib.logging with no special configuration done prefaces each and every logged line on my system with:

[null] May 16, 2010 1:08:48 PM clojure.contrib.logging$eval__2077$impl_write_BANG___2088 invoke

I have no idea why. This makes it very hard to read the log when logging any nontrivial amount of information.

clojure.contrib.logging’s documentation contains a rather cryptic offhanded remark, with no further explanation: Note: your log configuration should display the name that was passed to the logging implementation, and not perform stack-inspection, otherwise you’ll see something like “fn__72$impl_write_BANG__39__auto____81″ in your logs.

My log configuration? Stack inspection? Why do I need to have a log configuration to just dump strings to the console? Why is the default case to print a verbose and useless header for each line? Isn’t this Javaesque mess — obscure 50 line XML configuration files just to get basic logging functionality working — exactly what we’re trying to avoid?

Log4J with `clojure.contrib.logging` configuration

Ah, well, there’s nothing for it, I suppose. Better just get a serious logging library and figure out how to configure it. The docs say that clojure.contrib.logging delegates to the first supported logging implementation that it finds. I am apparently using the builtin JDK logger by default, and Apache Commons’ logger is just another wrapper, so I added Log4J to Leiningen’s project.clj and it installed the jar for me.

Now, I get:

     [null] log4j:WARN No appenders could be found for logger (my-project-name).
     [null] log4j:WARN Please initialize the log4j system properly.

Followed by dead silence.

Hm.

I can only imagine that Log4j was never intended to be used without a custom configuration. Odd. Why people design software that comes without reasonable defaults pre-set to cover the common base case is beyond me. I fully understand that strange setups will require customized configuration, but “dumping strings to the console” does not seem like a bizarre edge case for a logging package. A default case of “send all log messages to the black hole unless configured to do otherwise” makes little sense.

So, how hard can it be to configure Log4J for this base case functionality? The “short introduction to log4j” is 19 pages long. Sigh. (The complete manual is proudly described as “over 200 pages.” I sometimes wonder if they overengineer Java libraries on purpose just to drive book sales.)

I do not want to know what a Nested Diagnostic Context is, or the precise semantic distinction between “Loggers,” “Appenders,” and “Layouts.” I don’t care about “Appender Additivity” and “Named Hierarchies,” and I certainly don’t want to write an XML configuration file. I Just want to log tagged strings to the console, and have them print without a bizarre autogenerated header line. That’s it. I fully appreciate that those things will be useful for some people — maybe even for me, when my project grows to become more complex. I have nothing against their existence; but for the base case, one should not need to know of it.

On page 9 of the “short” manual, after a boatload of irrelevant information, we have an answer: there is a class called BasicConfigurator with a static method that does reasonable initialization for logging to the console. We can just do (org.apache.log4j.BasicConfigurator/configure) and get that.

However, we are still getting log messages like:

     [null] 3702 [8633270@qtp-3149669-2] INFO my.namespace - log message

We have to learn something about Layouts now. The thread names generated by Clojure are apparently not very useful, so we’re going to turn them off. And while interesting when profiling, the number of milliseconds since program start is just going to be visual clutter 98% of the time, so I’m going to turn that off, too, until I actually need it.

The result is the log4j.properties file listed above. Just copy and paste it into your classpath. If you want more or different information in your logs, take a look at the PatternLayout API documentation and alter the ConversionPattern line according to your preferences.

The result:

     [null] INFO  my.namespace - log message

I’m still not sure where the [null] comes from, but overall, it’s very readable.

Print PDF

Why the EU is Screwed

Paul Legato — Tue, 11 May 2010 18:15:48 +0000

Why the EU is screwed, from our friends at Der Spiegel:

Print PDF

Flash BIOS Without a Floppy

Paul Legato — Sun, 09 May 2010 00:09:53 +0000

Flashing the BIOS without a floppy drive can be done with an Ultimate Boot CD (free!) and a USB memory stick. BIOS upgrades usually come in the form of an MS-DOS executable meant to be run from a bootable floppy, but modern computers don’t usually have floppy drives anymore, and who has a copy of DOS lying around anyway?

Download a copy of the free Ultimate Boot CD and burn it to disk. FreeDOS (and a lot of other useful stuff) is preinstalled on the CD. As the name implies, FreeDOS is a free, open source, binary-compatible replacement for MS-DOS.
Download the replacement BIOS from your motherboard’s manufacturer. Unpack it onto a USB flash drive.
Plug your flash drive into the target computer and boot it with your Ultimate Boot CD. Select FreeDOS from the menu.
FreeDOS asks a number of configuration questions while booting. You can usually leave all at their default except for the ASPIUSB driver. ASPIUSB is not enabled by default (at the time of this writing), but is necessary for the use of a USB flash drive under DOS, so enable it when prompted.
If you miss the prompt for the USB driver, don’t worry. Type “menu” at a command prompt and enable USB devices from the popup menu.
Change to your flash drive by typing e.g. R: and execute the BIOS flash utility as per the manufacturer’s instructions. (If you don’t know which drive letter to use, type “menu” and select the option to display active drive letters.)

Print PDF

XML DTD Validation in Clojure: Turning It Off, Parsing Malformed XML

Paul Legato — Thu, 29 Apr 2010 00:29:31 +0000

I wanted to parse some externally-generated and malformed HTML, so naturally I went to the short and sweet clojure.xml/parse function. I got a nasty error:

error: java.io.IOException: Server returned HTTP response code: 503 for URL: http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd

It seems that the W3C blocked access to the DTDs two years ago, but Java still tries to load them by default anyway. The following allows clojure.xml to run without checking external DTDs:

(defn startparse-sax-non-validating [s ch]
  (.. (doto (. javax.xml.parsers.SAXParserFactory (newInstance))
       (.setValidating false)
       (.setFeature "http://apache.org/xml/features/nonvalidating/load-dtd-grammar" false)
       (.setFeature "http://apache.org/xml/features/nonvalidating/load-external-dtd" false)
       (.setFeature "http://xml.org/sax/features/validation" false)
       (.setFeature "http://xml.org/sax/features/external-general-entities" false)
       (.setFeature "http://xml.org/sax/features/external-parameter-entities" false))

       (newSAXParser) (parse s ch))))

Then you can simply (xml/parse "sourcefile.xml" startparse-sax-non-validating). This is not the ideal solution — ideally, we want to use a locally cached DTD — but it works well enough for one-off code. Read on for further information.

W3C’s malformed specification

The W3C webserver was being DDoSed by the massive number of XML parsers trying to download the DTDs constantly, so they just blocked everyone. This is a direct result of the poorly considered technical specification for DTD identification and URIs themselves, then exacerbated by the W3C’s institutional culture and philosophy.

Technical commentary

The root of the problem is, as they put it, that “these [DTD URIs] are not hyperlinks; these URIs are used for identification” (emphasis in original.) This is an abuse of the URL format. If it’s “for identification,” then it doesn’t need to specify a transport protocol! They are relying upon their particular interpretation of the old URI/URL debate, reducing the transport protocol element to a namespace identifier instead, and then assuming everyone will defer to their judgement and wisdom on the matter.

The W3C (and others who share their URL/URI interpretation) decided to conflate the semantic functions of locating and transporting a resource with the very distinct function of identifying a resource. “My name” is not the same thing as an instruction to “get in your car, drive to my street address, pick up someone with my name here, and then drive me back to your house,” even though there is indeed only one person with my name at this address. If you want to identify me, you would not give someone the latter instruction. Most likely, they didn’t think about this at all and just did it. Now everyone suffers for this poor design decision. They are being DDoSed and paying massive amounts of money to keep the servers going, and everyone’s XML parsers are broken because they’re now returing 503s for all the DTD URIs.

The W3C’s proposed solution is that everyone else should a) defer to their views on the URL/URI matter, and 2) rewrite their XML parsers to be massively more complex and build local caching DTD catalogs for every single one-off parse. In practice, most people will just turn DTD validation off, potentially breaking a lot of other things.

Institutional commentary on the W3C itself

The tone of naive incredulity in the W3C’s plea is most amazing of all. They seem genuinely baffled as to why anyone would ever choose to ignore their wisdom on the URL/URI matter, and even more confused as to how anyone could ignore any detail of a baroque and poorly communicated specification like those for XML and DTD cataloging. “We spent a long time writing that, and we have [the self-appointed] institutional authority to write specifications and make determinations like what a URI should be, so surely people would never just ignore us!”

For them, the only conceivable explanation for every detail of the specification not being implemented is sheer ignorance. They even gently chide the readers about their failure to use URIs properly and to build DTD caches into everything, much as a kindergarten teacher would a well-meaning but rather dumb child who continually ignores instructions about how to use his crayons. They think that they only have to patiently remind people enough times that there is indeed a right way to do it, that which is sanctioned by the appropriate institutional authorities (them, of course) — and finally everything will be fine once people get that message.

This is the sort of ingenuous attitude that can only develop in a pure research institute and among people who mostly have very high power distance index scores. The high PDI explains their naive arrogation of global authority and utterly genuine assumption that everyone ought to just do as they say, simply because they’re the (self-)designated authority. The research institute setting means that they’re shut off from the economic real world. Since money is never a big issue, the worship of artificially created fancy titles and institutional authority can become rampant, and it’s perfectly fine to spend extended amounts of time parsing a 2 page XML document “the right way” rather than 15 minutes hacking it and moving on to more pressing problems so your company doesn’t go bankrupt in the interim.

Moreover, they are still intransigently defending the original poor design choice to use URIs with embedded transport protocols as unique identifiers, because they (as an institution) cannot admit the possibility that this was a poor choice. That would create a great deal of cognitive dissonance vis-à-vis their conception of the nature of an institution and its role in dictating what is correct and what is not in the world; their view that institutional authority must be right, because, well, otherwise it wouldn’t be an institution. After all, everyone there does have more fancy titles than most people who are not there.

These attitudes are pervasive throughout most W3C work, and are the main reason that real-world implementations ignore much of their work.

Parsing malformed HTML

So, in any case, I have another problem besides Java trying to load the unavailable DTD. The input HTML is malformed. Not just a little; it’s utterly broken. It’s both externally-generated and unreplaceable, so fixing it at the source is not an option. I’m stuck with it; it’s the only way to get the data I need. (That’s real-world programming as opposed to academia right there.)

Disabling XML validation

Disabling all XML validation is the obvious solution to both problems, but there is no readily apparent way to do that clojure.xml. Calling setValidating(false) on the factory is insufficient. Thanks to Stack Overflow, I found that you must also use
setFeature() with several obscure “SAX2 Standard Feature Flags“, as above.

HTML Tidy to the rescue

This still leaves the problem of the malformed HTML. There does not seem to be any easy way to make Sax ignore errors. ((.setFeature "http://apache.org/xml/features/continue-after-fatal-error" true) has no apparent effect; it still doesn’t continue after a fatal parse error.)

In the end, I just ran the malformed input HTML through HTML Tidy before attempting to parse it. HTML Tidy coerces the malformed input into valid XML, and Sax is finally happy.

(I now get “Exception in thread "Swank REPL Thread" java.lang.RuntimeException: java.lang.IllegalMonitorStateException” from SLIME when I try to actually display the parsed XML in Emacs, but that’s another matter…)

Print PDF

Goldman’s Blankfein before Congress

Paul Legato — Tue, 27 Apr 2010 23:40:57 +0000

And it came to pass, that Goldman’s then-chief overlord was order’d to parade full 7 days and 7 nights through the streets of lower Manhattan, clad in naught save his drawers, bearing tether’d to heavy chains every SEC filing that Goldman hath wrought for 10 years previous; while the peasants of the borough were to gather round to hurl rotting vegetables thence, and cry, “Lo! How the mighty have fallen!”

And only then, once the indignity were suffer’d in full, might he retreat to his penthouse to recover, or to his yacht, or to the coffers of his chalet in southern France, or to his Bahamas estate countinghouse, or to any other sequester’d location whatsoever.

Print PDF

Paul Legato

Dell PowerEdge BIOS Upgrade on Linux

Swing-Clojure GUI for the Black-Scholes Option Modeler

Clojure code

The Input Panel

The Output Table

Design considerations

GUI and business logic decoupling

Atoms to hold state

Auto-updating results when input changes

Clojure-Options on Github

Black-Scholes in Clojure

A Rebuttal for Kartik Athreya

Lazy-spy: Clojure logging/spy for lazy sequences

Log4J and Clojure Configuration

Logging philosophy

What’s wrong with clojure.contrib.logging’s defaults?

Log4J with clojure.contrib.logging configuration

Why the EU is Screwed

Flash BIOS Without a Floppy

XML DTD Validation in Clojure: Turning It Off, Parsing Malformed XML

W3C’s malformed specification

Technical commentary

Institutional commentary on the W3C itself

Parsing malformed HTML

Disabling XML validation

HTML Tidy to the rescue

Goldman’s Blankfein before Congress

Log4J with `clojure.contrib.logging` configuration