Our democracy, which depends on a free and informed electorate, is in danger!

This is not the fault of any political party, and the coming election has nothing to do with it. It has to do with the possible imminent demise of good journalism and the newspapers that support it. And that has to do with both the financial problems of existing newspapers and our apparent unwillingness to pay for something better.

Thomas Jefferson said it eloquently in 1786. “Our liberty depends on the freedom of the press, and that cannot be limited without being lost.” This is also expressed in the clause in the first amendment of our Constitution – “Congress shall make no law … abridging the freedom of speech, or of the press …” And the Universal Declaration of Human Rights, adopted by the United Nations in 1948 makes it even more explicit. “Everyone has the right to freedom of opinion and expression; this right includes freedom to hold opinions without interference, and impart information and ideas through any media regardless of frontiers.”

So – is anyone here opposed to freedom of opinion and expression? Not likely. And we mostly seem to have it. We certainly have ways for LOTS of people to express their opinions – on newspaper websites, and blogs, and Facebook, and Twitter, in self-published books, etc. etc. etc. And this is good. But an informed electorate needs more than JUST lots of opinions, freely expressed and published. We need good journalism that reports the news that’s hard to find, that occurs in places far away from us, that is hidden by special interests (including our government and our politicians). And that reveals opinions that are not always what we already believe and that introduce new vetted facts into our world view.

Traditionally, newspapers have provided most of that kind of journalism. Think about The Washington Post and Watergate, the New York Times and WikiLeaks, The Washington Post again with an expose of the terrible conditions for injured veterans at Walter Reed Medical Center, the mostly-ignored early warnings about the fragility of the big mortgage lenders in The Wall Street Journal and more.

So I feel we have a need for professional journalists. But do we need them to write for print newspapers? Can’t we find out everything we need to know from “the web”? We can see fast breaking news stories without waiting for the next edition of the paper. We can go look whenever (if ever?) we want to know something. Isn’t that enough?

It might be, as long as the websites have a source for their news and commentaries and as long as we remember to go look for more than the latest sports scores. Who does go get those stories, verify the facts and the sources, and write them up? Most of it comes from newspaper companies now. What will happen if the newspaper companies disappear? Can we rely on “crowd-sourcing” and bloggers? Can the typical blogger afford to send reporters around the world to cover world news? Can an individual afford to spend more than a year tracking down the details of the risky activities of Wall Street insiders? Who will keep a watchdog eye on local school boards and town councils? And, most importantly, who will verify that the details are accurate and unbiased? The “free” news on the internet won’t be there if no one pays the reporters to get it

Well, there’s always TV news. National TV news organizations DO currently send their reporters all over the country and the world. But what they typically report on the national evening news is more like entertainment sound bites than investigative journalism. (For a more eloquent description of this trend, visit http://bit.ly/NDyljJ to see the video of Bill Moyers interviewing Marty Kaplan). Local news broadcasts are mostly police There ARE 60 Minutes and Frontline who take up some of the slack. But then, there is also Fox News.

Obtaining and presenting news – especially more than just the surface news – takes money. Where does a news organization get this money?

Whether it’s for TV, radio or print, the money comes from advertisers. Any money that comes from consumers is mostly gravy.

Of course, the advertisers are looking for money from the consumers. They want you to see their ads and buy things – lots of things. They will place their ads, and thus support the publishers of the content around the ads, where they spend the least for the greatest return. That USED to be in the newspaper. Print newspapers used to be king in the classified advertising market, for example. Now there’s Google search, yahoo, Craig’s List, Facebook, Twitter, etc. There are many places to publish ads; this is no longer a captive market for newspapers. The problem is that these other places have no mandate (or even desire) to also give the readers good journalism.

This chart, which you’ve all probably seen, shows the decline in print advertising in US newspapers since 2005.

But newspapers have websites and they’re making this up in revenue from online advertising, right? The Poynter Institute recently published a study that says “Newspapers are losing $7 in print revenue for every $1 in digital gained.” It is probably even worse now since newspaper digital ad growth seems to be slowing down.

The “all digital” movement that’s being preached in newspaper industry circles lately doesn’t seem to be working out too well. Add to that the fact that most newspaper websites are “free”, and this looks like a business model that is failing.

What can newspaper companies do about this? What can we, as citizens in a democracy, do about it?

Too many newspaper websites are just formatted dumps of the content in the print product. It should certainly be possible to present great content on the web and draw the reader in to promote interaction in a way nothing in print can. You can find occasional examples of this – most noticeably in online polls, the opportunity to comment on stories and opinions, and solicitation of reader photos. But news organizations can get MUCH more creative – I’m confident they can.

There’s one kind of quirky example: a website in DC (not a newspaper, unfortunately) that covers only crime stories, but keeps a running update of each one on a dedicated page with links between similar incidents (www.spotcrime.com). Not the most critical commentary perhaps – but a very creative way to use the web format.

Another possible example: Barack Obama has just mandated that government agencies provide data about all of their spending accessible through the web. It’s likely to be just a dump of numbers, hard to organize and compare across agencies. Where could you go to make sense of this information? How about if your local newspaper, a source you know and trust, led you to this data and had software that put it in a format you COULD read?

There’s undoubtedly some exciting interactive experiences coming for us.

But then, when online newspapers start being creative and engaging AND still fulfilling the need for what Steve Buttry of the Journal Register Co. describes as, “commitment to getting the facts right, dedication to seeking and reporting the truth, high ethical standards, holding the powerful — and ourselves — accountable, serving the watchdog role with honor,” it’s up to all of us to be willing to pay for this service. When that website you’ve been jumping to for your news fix or your sports scores puts up a paywall, subscribe and be grateful it’s there. There is no such thing as “free journalism.” Pay for it and use it to become an informed citizen of a free society.

http://linkd.in/svpqAC

We’ve used things we learned from chess programming to create SCS/ClassPag for (newspaper classified pagination).

Chess programming provided a rich environment for computer science research. Debugging chess programs was quite difficult. The complexity of the algorithms meant that hand simulation of the program’s logic was often nearly impossible. Chess programmers quickly realized that writing such programs depended on having the best tool for thinking through the programs’ designs.

McCarthy developed Lisp for his AI projects. It replaced OPS5 when he found he could no longer improve his chess program because he could no longer understand it!

Remember that in the early days of chess programming, the development environment wasn’t interactive. Batch compilations and test runs were the norm. Programmers like Ed Kozdrowicki and Dennis Cooper, authors of the CoKo, wrote in FORTRAN. They invented indenting code to show the nesting levels of loops, etc. CoKo was well over 50,000 lines of code. (A single spaced printout of the code was six inches thick.) Like most chess programs in those days, it was tested using all the computer time the authors could beg, borrow or steal. In CoKo’s case just 12 executions! Imagine doing a live show-and-tell of such minimally tested software at a convention (many of these programs were shown at the Association for Computing Machinery annual meetings) before nearly a thousand of your most respected colleagues!

Ken Thompson developed C and Unix as a platform for his chess programming efforts. He and his chess program/machine, Belle, designed at AT&T’s Bell Laboratories, won the North American Computer Chess Championship in 1978. In 1983, Belle became the first chess program to reach the U.S. master level of playing ability.

Ken basically stole the computer on which he did this skunk works development. When asked what he was up to, he replied that he was developing a text processor that the lab (i.e., Bell Labs) could use in document preparation. When asked for a demo, he had to quickly put together what later was called nroff. To compose type for a phototypesetter he then developed troff.

Thompson and McCarthy were both Turing Award winners, which most regard as the Nobel prize for computer science.

Barbara Huberman Liskov, a graduate student of McCarthy’s, wrote a Lisp based chess playing system for her doctorate. She then developed a new paradigm for programming which she embodied in a programming language of her own design called CLU.  CLU was the first programming language that supported all of the concepts that are fundamental to object oriented programming. For this contribution she received the Turing Award. I suspect the design of CLU was influenced by her chess programming experience.

In a biography of Liskov, John V. Guttag, Head of the MIT Electrical Engineering and Computer Science Departmen,t wrote, “CLU was a truly audacious language. It tastefully combined the then novel ideas of 1) data abstraction, 2) iteration abstraction, 3) exception handling, and 4) polymorphic types. CLU was not a “paper”
programming language. Barbara and her students produced a high quality implementation on a variety of platforms. Along the way, they pioneered a number of implementation techniques, e.g., for raising and catching exceptions, that are still used today. Every important programming language since 1975 has borrowed ideas from CLU. Java, for example, owes its notion of inheritance to Simula and Smalltalk, but almost everything else of interest in it was pioneered in CLU.” [

Here is some correspondence between me and Dr. Liskov:

“Dear Dr. Liskov,

Congratulations on winning the Turing award.

I note that your winning of the Turing award is particularly appropriate since Turing himself and John McCarthy, your adviser, were both students of computer chess.

I remember sitting in the lunch room at the Bank of Delaware in 1972 with my fresh BSCS degree from the U of Delaware reading your PhD dissertation. (Chess programs in LISP! Wow!) You were Barbara Huberman then. I was working in data processing operations because I wasn’t trained to program in 1401 Autocoder or 360 COBOL. There was a recession on (RCA had just let 2,000 data processing staff go) so I was glad for the job.

Anyway the VP of data processing (called the IBM Department) (who didn’t have a degree) picked up what I was reading and shrugged when I tried to explain what it was about. I suspect that he thought my interests were at best unusual. He must have had some inkling that I was special because after that he kept giving me little projects that resulted in challenges to the status quo. I started having fun at the BoD job. To this day I say this was thanks to you.

As you know writing chess programs confers on their authors special insights. Unless you are able to use an algorithmic language that supports clear thinking, your hopes for getting working chess programs are slim.

For our work with newspapers we build numerous domain specific
languages. Not surprisingly much of the software we supply to
newspapers solves their NP-hard problems.

I was thinking of you the other day. While scanning several articles on the web I found myself saying “Oh, this one sounds like Barbara.” I checked the authors and there you were. You have a clarity of thought and expression that makes whatever you write an outstanding pleasure to read.

Once again I am delighted to offer you my personal congratulations on your award.

Thanks,
Richard”

Barbara Liskov wrote on 3/27/09:

“thanks so much for your note. I really appreciate it. It’s great
to learn about your story and I’m glad I was able to help (if in
fact I did)!

Best regards,

Barbara Liskov”

SCS had a contract with American Express to make personal computer
programs for business executives. I think we agreed to write the suite
of programs (called Epics) for $6,000. It included an appointment
scheduler, project planner using the critical path planning method (we
got the algorithm from friends at DuPont which first developed it on a
mainframe), contact manager and several others. The AmEX VP who
contracted with us thought all his executives should have computers of
their own.

The programs were developed in Pascal (UCSD version) with the help of
interns. They worked in an office I outfitted in the lower level of our
home. William started there.

Martha applied to get an AmEX credit card but she was at first refused.
She contacted the VP and said something like, “We are giving you credit
on this project, surely you can return the favor.” SCS got its card
shortly thereafter.

SCS also developed (mostly my code) an information retrieval system
for scientific article citations under contract with Connaught
Laboratories. Their Apple had a 5mb hard disk with a VCR-based backup
tape drive. Complex queries were evaluated in parallel to save disk
accesses because they were very slow. The development machine had 48kb
and three 360kb floppies.

There might not be an SCS today were it not for Steve Jobs and Apple
Computer, the company he co-founded.

Jobs has earned an exalted place in history. He will be missed.

Please enjoy what’s linked to below. There are lessons for SCS today in
what he says.

Steve Jobs: ‘Computer Science Is A Liberal Art’

http://www.npr.org/player/v2/mediaPlayer.html?action=1&t=1&islist=false&id=141118621&m=141128513

Jobs Stanford commencement speech.

Steve Jobs Interview extract.

2. What is perhaps the most valuable thing you learned in college?

“Because I had dropped out and didn’t have to take the normal classes, I
decided to take a calligraphy class. I learned about serif and sans
serif typefaces, about varying the amount of space between different
letter combinations, about what makes great typography great. It was
beautiful, historical, artistically subtle in a way that science can’t
capture, and I found it fascinating.”

3. How was this useful to you?

“Ten years later, when we were designing the first Macintosh computer,
it all came back to me. And we designed it all into the Mac. It was the
first computer with beautiful typography. If I had never dropped in on
that single course in college, the Mac would have never had multiple
typefaces or proportionally spaced fonts.”

4. What lesson did you take from this?

“You can’t connect the dots looking forward. You can only connect them
looking backwards. So you have to trust that the dots will somehow
connect in your future.”

5. What maxim has guided your life?

“You have to trust in something — your gut, destiny, life, karma,
whatever. This approach has never let me down, and it has made all the
difference in my life.”

The politics of allowing advertising departments to control color ad placement is interesting to say the least. We found this out in the late 80’s when we released, sold and installed ColorAdBoss’s predecessor called WebPresser.

WebPresser was unique technology for the newspaper business. It was (and except for ColorAdBoss, still is) the only successfully marketed computer program which actually computes press layouts while the edition is being designed and therefore while more color positions can still be sold.

This sounds like something every advertising department would love to use – and perhaps they would. But it challenges the traditional role of the pressroom foreman and his relationship with the layout staff.

By the end of 1993 we had sold nearly $100K of WebPresser technology to 59 sites, but the importance of WebPresser went far beyond the minimal amount of our revenue that this represented. Its importance was more closely tied to how it distinguished us from competitors who might also market ad dummying software. WebPresser had tactical and strategic importance which helped us protect our market share with Layout-8000. Here is what it looked like in 2002.

Read my memo of March 7, 1989 about the back-story including how we used it tactically to defeat a challenge from Composition Systems Inc. (CSI) that threatened our dominance in the automated dummying business. The memo also contains a good description of the potentially adversarial battle between the advertising department and the pressroom foreman about how much color can be run in the paper and the edge that WebPresser could give the advertising staff.
Here is a summary of some additional details from other folks who were around and involved in the story for those who remember the era.

Facebook

Twitter

Let’s say your marketing department comes up with a product that they claim is “new and brilliant.”  After being impressed, you go to the production and technology departments and say, “Look at this wonderful new product.  Make changes so we can sell and produce them.”  If either of these departments responds with, “I’m not sure this is a good idea,” I wouldn’t blame you for being frustrated and suspicious.  Why can’t you have this wonderful, shiny, new thing to sell?  Why shouldn’t you be innovative and creative?  After all, isn’t that what marketing departments are all about?

Our company, Software Consulting Services, LLC, works with the technology and production departments of newspapers; we also work with the advertising and accounting departments.  Nothing would make us happier than finding new ways to be creative and innovative in the support of increasing ad sales in newspapers.  Clearly, arguing against the idea of creative and innovative ads seems, at least superficially, bullheaded, and yet that’s what I’m going to do here.

Our software, Layout-8000, is a product with a 30-year history of serving newspaper needs.  Layout-8000 helps newspapers design their products.  It computes what are called dummies.  A dummy is a sketch of the design of an edition.  When you design a newspaper edition, you first figure out where the ads are going to go.  This is called display ad dummying.  Page dummies, both graphical and electronic, are produced by Layout-8000, showing where ads are to be placed in editions.   Ad dummies are sent to pagination systems where news is flowed onto pages and display ads are inserted into the locations specified by the dummying package.  Completed pages can then be output.

At many newspapers, desktop publishing tools are used for the news pagination step.  There are interfaces from Layout-8000 to such tools as QuarkXPress® and Adobe’s InDesign®.  (SCS co-invented Quark xTensions®.)  In general, if what you make with QuarkXPress or InDesign can fit on a plate to be installed on a press, then the page image can appear in a newspaper edition.

So what do marketing departments think of when they are talking about creative and innovative ads?  They look at the rectangular boxes so many columns wide and so many inches or centimeters high and say, “We should have ads that aren’t limited to being rectangular boxes.  Maybe they can be “L-shaped” or “H-shaped” or “U-shaped.” Wouldn’t that be more creative?  Why are we limited to the boundaries of columns?  Why shouldn’t we be able to have an ad that is exactly 2.7695 columns wide?  And, this business of having ads that are rectangular is even further limiting.  Why shouldn’t we be able to have an ad that is like a circle or a star?”  Now there’s innovation for you.

There is no limitation to the creativity of marketing departments.   I’m sure you’ve seen ads that are sold to be placed upside down.  Others have tried shadow ads where a black and white page has a spot color shadow of a display ad plopped right on top of copy. I’ve even seen newspapers run ads diagonally across the page or turned ninety degrees to fit vertically into a column.  Believe it or not, last month we had a Brazilian newspaper go live with Layout-8000 and during the first week of working with it, they ran a number of pages in 3-D.   They even gave away the funny looking glasses with each copy.

Suggesting that there might be practical limitations to creativity and innovation puts someone like me in the role of humbug.  Well, I’m willing to be a humbug.

The first thing we need to do is choose a more neutral term for describing these strange artifacts that are blessed with the names “creative and innovative.”  I call them “novelty ads.”  The word novelty is pretty much devoid of either positive or negative connotations.  Using it allows one to more fully appreciate what’s going on.  Two things are at work.  First, advertisers want readers to notice their ads.  Newspapers say, “We will run it far forward, run it on a right-hand side or run it in color.  Take the whole page.  If that’s not enough, let’s be creative and innovative.  What kind of novelty would you like?  Would you like an ad that is mis-shaped, mis-sized or mis-positioned?  We can do that too.”

Here’s the rub.  Let’s say that an advertiser and a salesperson agree on an ad that’s shaped like a “U.”  It goes down the left hand side of the page, across the bottom and up the right.  Surely you can produce a page that has such an ad on it in InDesign.  You might not be able to order it in your advertising order-entry and accounting system, and when it comes to dummying the entire paper, I’ve never met an advertiser willing to have their ad sit in the middle of a “U-shaped” ad placed by another advertiser.  Their ad wouldn’t be noticed at all.

What’s the problem?  You have a “U-shaped” ad and you want it to go on some page.  Somehow you order this ad in the order-entry system so you can price it and bill it.  And, just as equally surely, no other advertiser is going to want to be on that page so you would order the ad as a “single ad” for Layout-8000.  This indicates that the ad should be the only one on the page.  From there, you set up a block that is the size of the page, call it a single ad, place the block on the page and put a page message note, “this has the u-shaped ad.”   Then the paginater can get the u-shaped ad and stick it on the page with the other copy and editorial content that might go there.

Remember that when dealing with such a u-shaped ad, you are not talking about doing classified pages in Layout-8000.  If you wanted to do a u-shaped ad with SCS’s software for classified pagination, you would factor the ad into pieces as if they were folded, long legals and place the pieces on the page and paginate the other classified liners and display ads around it.

To deal with the issues of design and production of novelty ads, experimentation should be encouraged but the ads should be dealt with as the exceptions they are.

There are lessons to be learned from other industries and other new product development situations.

Here’s an example from AutoCad®.  You can use the architectural module of AutoCad to design buildings.  Most buildings, like most ads, come out right side up.  However, there is at least one situation where a building was built to look like it was built upside down.  One of the franchised “Ripley’s Believe It or Not” sites looks like a building that was completely flipped over.  Would you add a button, a configuration, a context for designing upside-down buildings to AutoCad because of this one exception?  I think not.  An exception is an exception and should be dealt with as a one-off situation until it proves that it deserves more than that.

Here’s a second example.  In this example, everything that was a one-off situation became standardized.  This standardization led to a new world order in global commerce.

It used to be that goods were shipped in whatever container they fit in. Packing the ship’s cargo hold amounted to gathering and stacking individual packages.  The process was labor-intensive, slow, costly, subject to error and inviting of pilferage.  Modern cargo ships plow the oceans with containers that are all exactly the same size.  Ships are built to hold thousands of them.  Cranes and trucks are all synchronized to support the easy handling and transport of standardized cargo containers.  The essential standardization of cargo containers would be ill-served by any deviation from specification in the name of creativity or innovation.  Automation through standardization trumps “creative” variability.  It’s true for containers; it’s also true for newspaper ads.  The future for advertising solutions argues for minimizing the presence of novelty ads.

I am a strong proponent of innovation; I’m just an enemy of wasteful novelty.  Innovation is important but there’s a context that must be appreciated.  Newspapers are in decline.  If the newspaper business experiences only a single-digit percentage decline in advertising revenue in a year-to-year quarterly analysis, that’s now regarded as “holding steady” or “good.”

The history of industries in decline is often characterized by a rash of innovation.  This is something that our company tried unsuccessfully for awhile only to later learn that we were typical of those serving declining industries.  Innovation can get in your way.  It can prevent you from doing the right things.

Innovative new products in declining industries cause already stressed sales departments to be confused and even less effective.  It will happen to you, if it hasn’t already.

This business pattern is not unique to newspapers or to vendors serving newspapers.  In the early 1800s, when steam ships were replacing sailing ships, those that built sailing ships innovated at a furious pace.  They worked on making their vessels larger and speedier.  They pointed out that the wind provided free propulsion.  They innovated with things like five-masted schooners.  Nevertheless, steam took over.

There is room for creativity and innovation in newspaper ad sales. Some publishers get it.  Martin Till, publisher of the Express-Times in Easton, PA (serving the counties in and around the Lehigh Valley as well as into New Jersey), has worked on an innovative pricing formula. Instead of having different prices for color versus black and white, the Express-Times has innovated with pricing all ads the same.  One advantage of this is that more color ads are purchased.  Best of all, the more color, the better the paper looks. What is needed, as Mr. Till shows, is innovation in selling, not necessarily innovation in products.  Selling more color ads is better than selling more triangular, circular or star-shaped ads.

There are other areas to innovate in with enhanced automation: publishing more products, paginating and printing the products of others, etc.  Get away from one-off solutions, avoid narrowly focused innovation like dead-ends and blind alleys.

On the positive side, what about selling premium locations?   We have developed and provide a product that manages reservations for premium space.  Two of our metro customers, the San Francisco (CA) Chronicle and the Detroit (MI) News and Free Press, use it as a tool for generating additional revenue.  It can be used in a self-service, online, web-based mode, tracking and allocating premium positions.  It even supports selling by template.  Of course its output can be used by Layout-8000.

Another innovative sales tool from SCS manages standby/remnant advertising.  Here’s a tool for replacing unpaid fillers with paid advertising.  It ties directly into Layout-8000 and manages ads in the context of space available during a dummying session.  You can sell remnant ads like Orbitz sells discounted hotel rooms. It’s another tool for innovative selling.

And we have the first tool to compute press impositions in the context of the ad layout process.  It allows you to run more color more simply.  Let’s say you are running fewer pages.  If you don’t have a collection of tower units, you can still run more color if you link units.  It takes smarts to compute such linking.  Our tool has them.

So here’s my message.  As long as your print product is your primary revenue source, innovate by seeking the highest level of automation.  Perhaps this will mean less novelty in the ads that you are selling, but the savings that you will generate by not dealing with exceptions should more than compensate for any outlier sales you might get from novelty ads.

Robert Gunning helped make the WSJ more readable. Barney Kilgore was the transformative journalist who hired Gunning. This lead to the
Journal becoming one of the most readable newspapers in the country.
Its circulation rapidly grew from 250,000 to over a million as Gunning’s suggestions for clear writing were adopted. (He said you don’t need complex language to explain complex topics. In fact, complex language works against understanding.)

The Audit Bureau of Circulation recently reported that the WSJ was the only newspaper that increased its daily circulation (to over 2 million) between October, 2009 and March, 2010. Is there still a message here about clear writing?

At the American Newspapers Publishers Association Research Institute
(ANPA/RI) in the mid-70s, I was responsible for creating and testing an advanced fog index calculator. The Society of Newspaper Editors
campaigned against this and several other applications we developed,
including one that did relatively intelligent story cutting and fitting. They were opposed to almost any machine analysis of editorial
content. They made quite a stink and the RI backed off.

Later nearly everyone liked my spelling checker. Word processors of the day checked 60 words a minute. My patented device checked 15,000 per second. The software version was pretty fast and allowed the mini-computer based newsroom systems of the day to provide spelling checking. (In our current systems we test words at over 125,000 per second. The tester can work on any set membership task.)

The grant request I read also addressed the idea of measuring quality and/or political bias in metro newspapers. I’m much more curious about local content in local newspapers than in metros. The large national newspapers seem to offer few surprises regarding quality (usually good) and political bias (predictable). I’d like to see some thought put into assessing the quality and depth of local coverage by community newspapers.

There seem to still be plenty of areas where computer science could help newspapers improve. Are they willing to listen?

Sharon: Dad, why would you use a macro instead of a procedure or function?

Dad:  Good question, Sharon.  I have some opinionated experts here to
help answer this important question.  What are your thoughts on the subject?

Sharon: I write a lot of code.   I want it to be reliable, maintainable,
efficient and easily produced.

Pragmatist: Joel (as in “Joel on Software”) says “Don’t repeat yourself.”

Sharon:  Well, if code is to be easily maintainable, it will have to be
readable.   I’ll want terse but easily understood code.

Dad:  Wouldn’t it be nice if programming could be more automated?

Sharon:  You mean like David Parnas wanted to see.  An engineering
discipline of software development, where tested components were
manufactured and then combined into reliable sub-assemblies and from
there into systems?

Dad: Yes, exactly.   Parnas thought doing software needed to be a more
systematic endeavor.

Sharon:  What became of his work?

Dad: The Japanese claimed to have built such a culture and the West
thought it would take over the world.  That was 20 years ago, and it
hasn’t happened yet.

Sharon: Back to the question of macros vs. procedures and functions, please!

Implementer:  My compiler includes a pre-processor that expands macros.
You get inline code on macro invocation.  For procedures and functions
you get reusable code blocks that can be invoked via procedure calls
over and over.  The macros execute faster because they are inline while
they make bigger code images because they are replicated whenever
invoked.   Procedures and functions usually save space but have the
overhead of the procedure or function call and eventual return.

Theorist:  Actually thinking of the implementation doesn’t much help
with understanding the real differences.  You want to write clearer yet
more terse and reliable programs.  This starts with the  notion of type.
Types are the names of sets of values. Variables hold values. The
beauty of types is that with types, programs written in imperative
programming languages offer a way to declare consistency requirements
for operands.  When operands of the same type can be treated the same,
appropriate operators can be defined.  If variables (operands) have
known, that is, static types, then compilers can check for consistent
usage.  This protects the programmer from making many intractable errors
and even allows more optimized code generation.

Declare a type and a set of procedures and functions that work with it
and you have the making of a library of components.  Step one in
engineering software.

Better yet, combine these two ideas and make classes.  You
can make things doubly secure.

Pragmatist:   I’m not so sure.  Making classes helps me encapsulate
abstract types and operators, but how does it work in practice?  Let’s
say I need to process things last in, first out, that is, in stack-like
fashion.   So I know the logic for handling stacks.  Where do I go from
there?   I’d like to have many stacks, each differing by the type of
elements the stack holds and yet having the same operators. It’s
complicated. I’m confused.

Theorist:  No confusion necessary.   You just encapsulate the notion of
stack and apply operators with the same name.  It’s called polymorphism.
It looks the same, but works differently.

Pragmatist:  You mean it looks like a duck, walks like a duck, quacks
like a duck, but isn’t a duck?  What if some stacks are slightly
different from others?

Theorist: No problem.  Just include the parts that are the same and
differentiate that which is different.  It’s called inheritance.

Pragmatist: We have gotten a long way from abbreviating code for clarity
and efficiency.  Is this really going to solve the problem Parnas lays out?

Dad: Hasn’t yet.

Implementer:  You would not believe how complex compilers are getting
these days.

Dad: And then there is the structure clash.  Database management systems
make data access available externally to and independently of
application code.   Encapsulated data certainly cannot be externally
available.  That’s why there are relational to object-oriented mapping
technologies.  It might just be that one notion or another is incompatible.

Sharon:  So what’s wrong with class libraries?

Dad:  Not much.  However the bigger and more versatile they get, the
more complex and less transparent they get. (Even Julian Assange knows
lack of transparency is bad.) If a programmer runs into a situation
where the library routine is almost, but not quite, what he or she
needs, what happens?  If the library is a set of binaries, then a new
module is written.  If there is source available, then the source might
be edited forking the module or increasing its complexity and
potentially introducing instability.

BTW – When Parnas wrote about the software engineering crisis, Fortran
was being promoted as a tool for automated programming.  How times have
changed and how much they have stayed the same.

Sometimes it is better to just make a new language, one that has a
notation that very precisely fits the application domain – a domain
specific language, (DSL).

Sharon:  Where do macros come in?

Dad:  Early on elaborate macro processing was built into assemblers and
text processors (composition systems).

For typography you could name a visual effect and apply it to any type
(i.e., copy) you wished.  Properly done you could extend a macro
processor-based composition language, like RNF, troff or TeX, to many
application areas.  They allowed you to define your own domain specific
languages.

For assemblers, macro facilities made generating appropriate tedious
code sequences easier.  Eventually, macro processors built for
assemblers became so general purpose that the code a programmer wrote
appeared to be written in a high-level structured language.   Macro
processors could even be used to re-target code to different machine
architectures.

Powerful macro processing facilities were built into some compilers.
These allowed programmers to code extensions to the base language that
looked as if they were there from the start.

Theorist:  Yes, but the generality of macro processing must be weighed
against the lack of verification that comes with that power.  You
should have seen the macro processor that was first used to translate
C++ into C.

Pragmatist:  All this theory sounds good, but I still want to make
applications that work and to do so quickly.   I like to use module
builders.   What do you think of them?

Dad: I love module builders.   They are the perfect anti-theoretical
technology.  They work extremely well in practice but offer virtually no
theoretical underpinning.  You couldn’t get a PhD studying them, ever,
but you could build SugarCRM and dozens of other powerful and useful
applications.

If it were not for module builders, the web would be a much less
interesting place.

What we have made and are continuing to develop is technology which
includes generalized module building technology as a component.

Much of this relates back to macros.  Where they started and left off
was as tools for manipulating code.   Code – not objects, abstract data
types, stacks, queues, etc., just code.  Code is what you wanted to make
in the first place, after all.

Only by manipulating code can you deal with non-local issues globally,
as in the case of security technology.  It needs to operate everywhere
but appear nowhere.

Much of what gives theorists heartburn can be dealt with by targeting a
very strongly typed domain specific language.  Its compiler can pick up
type conflicts.

Still, one has to ask about our approach at SCS.  What is it?  Will it work?

Our tools development proceeds both top down and bottom up.   There are
three basic parts – the DSL, called the spice formula language; the Spice
Pattern Language and the Spice Macro Pre-processor.   They function
together to both become and to generate applications.  Bassett frames,
macros, patterns and most other concepts that support automatic program
generation are part of their feature set.   That’s the top down part.

The bottom up part is that as the tools are enhanced they are put to
almost immediate use.  Many features are incorporated based on day-to-day programming experience.   Programmer desire drives feature
development.  The tools are used to build very large mission-critical
applications in which new versions are deployed agilely nearly every
four weeks.  Every iteration includes multiple enhancements (often major
ones) and significant code re-factoring to take advantage of tool
enhancements.

Best of all it works at major companies every day.

Richard J. Cichelli (aka Dad)
President, SCS

Richard is more than an IT expert or a technologist.  What drives him (in addition to his passion about newspapers) is the desire to push the limits of computer science to implement the best algorithms with the best tools he can design. He’s committed to creating software that is fast, efficient, machine and operating system independent and that solves “tough problems.”

That’s what you’ll read about here – the research, the thinking, the design philosophy and the creation of development tools that continues to make this possible.

Martha J. Cichelli