What about shuffling? Using the Fisher-Yates algorithm also known as Knuth algorithm, you can shuffle large files while using almost no memory. But you need random access to your files. Thus it is not applicable to variable-length records. And indeed, the Unix sort command cannot shuffle. (It has a random-sort option, but it is not a shuffle. Meanwhile, the shuf command runs in RAM.)

A solution: Tag each record with a random number. Pick random numbers from a very large set so that the probability that any two lines have the same random number is small. Then use external-memory sorting. You can implement something similar as a single line in Unix.

A better solution? Shuffling is possible in linear time O(n). Sorting is a harder problem (in O(n log n)). Thus, using a sort algorithm for shuffling—as we just did—is inelegant. Can we shuffle in linear time without random access with variable-length records?

Maybe we could try something concrete? Consider this algorithm:

• Read the original file in small blocks. Shuffle each block in RAM. Write them to temporary files. View each shuffled block as a stack of records.
• Select a non-empty block at random. Pick and remove the record on top of the stack. Append it to the result set. Repeat. (The correct probability assignment for each block is the number of records left in the block divided by the total number of records left.)

(As a variation on this algorithm, you can merge the blocks two-by-two.)

Unfortunately, I doubt this algorithm can run in linear time.

Your challenge: Consider variable-length records. Prove or disprove that we can implement an external-memory shuffle in linear time. Alternatively, come up with an algorithm faster than the sorting-based one.

Update: Preston L. Bannister proposed an algorithm which solves the problem to my satisfaction. The same algorithm was described by P. Sanders in Random Permutations on Distributed, External and Hierarchical Memory (Information Processing Letters, 1998).

Reference: This is a follow-up to my blog post External-Memory Shuffles?

Which line runs faster? (The symbol “^” is the XOR.)

for(int k = 0; k < N; ++k) sum+= k;

for(int k = 0; k < N; ++k) sum^= k;

My result: In C++ and Java, both run at the same speed (within 1%).

Disclaimer: I’d be delighted if you could prove me wrong. Please provide Java or C++ source code.

Software programming is a form of communication, usually between human beings and machines. While different in style, programming is a subset of the language of Mathematics. If you dig into the average source code, it is undistinguishable from ramblings, even if you are an expert developer.

Yet, we denigrate programming. Many will even deny that it is a Mathematical language. But Mathematics and Programming are not so different:

Why is programming getting so little respect?

• The intense commercialization of programming has commoditized it. As Paul Graham might say : painters where initially “portrait takers”. It is only when painting lost its commercial function that it became recognized as a noble art. However, just like painters always used their free time to create great art, the best programmers are open sourcing beautiful code all the time.
• The study  of programming itself remains rather informal. You can get degrees in Computer Science, Computing Engineering or Software Engineering, but there is no degree in Programming. Programming is taught in universities, but generally only in the first few courses of a degree. Yet, there are degrees in Communication, Fine Art, Architecture, Music or Dance. While a degree in Computer Science or Software Engineer can make you a better programmer, the fact remains that your professors are not expert practitioners.

How can we fix this? I have this secret dream of setting up the equivalent of “Creative Writing” program, but for programmers. Call it “Creative Programming”. Basically, students would come together to write great code. Yes, such code might be useful commercially, but that would be a secondary consideration. The pursuit of greatness would be the only goal that matters. It would treat programming as a bona fide language. It would attract the best programmers as guest lecturers. Would this ever work out? I do not know.

I am sure that many will point out that my secret dream is impractical. Beauty should not come first : we want cheap, reliable, maintainable code. We also want programmers to be replaceable, inexpensive and practical. However, human beings can both pursue greatness while being practical. Compromise is possible.

Let me conclude by quoting Donald Knuth:

(…) computer programming is an art, because it applies accumulated knowledge to the world, because it requires skill and ingenuity, and especially because it produces objects of beauty. A programmer who subconsciously views himself as an artist will enjoy what he does and will do it better.

Further reading: The best software developers are great at Mathematics? and Is programming “technical”?

• Email forwarding may create infinite email loops. These may disrupt services and require human intervention.
• Invalid or failing remote servers may saturate the local servers as they are unable to forward the emails.
• Professors and management send confidential information by email. Yet, without full control of the email service, the University cannot ensure the needed confidentiality.
• With email forwarding, it may be impossible to ensure and prove that an email was received and read. Thus, homework assignments, administrative inquiries or security advisories may never reach the students, or we may be unable to prove that they reach the students because of email forwarding.
• As a Canadian University, email forwarding puts us at risk that the emails may transit on American servers, where the Canadian law on privacy is not applicable.
• Email forwarding may put students at risk if remote accounts are stolen or lost.

Can you help me debunk or mitigate these arguments? I know that some of these arguments are bogus, but I am looking for solid references. (Not that I expect to change their mind.)

A larger issue: shouldn’t universities stick with research and teaching? I understand that we must have networks, cables, computers, firewalls, but do we need to provide our students with email services?

Update: Turns out that our IT people encourage students who want forwarding to GMail (say) to use the POP3 protocol. It is unclear to me how email forwarding can be a dangerous practice whereas POP3 “forwarding” can be safe.

Maybe I should start my courses with this theorem:

Theorem. If you understand an idea, you can implement it in software.

There is no denying that programming requires a lot of technical knowledge. Most programmers do technical jobs, involving testing, building or refactoring code. But programming is ultimately a communication form. And it is as noble as Mathematics or English. Let us compare:

• Writers are considered sexy and non-technical people. Yet, grammar and spelling are technical. Moreover, most writers earn a living by writing ads for boring products. Some of them make a living with grand novels, but fewer than you think.
• Physicists are great thinkers. Yet, their mathematical derivations are often mind-numbing and technical. Many physicists spend years running extremely technical experiments. And when they don’t, they program extremely complex (and technical) simulations.

For some reason, being a writer is somehow considered more prestigious than being a programmer. If you ask me, Linus Torvalds is every bit as cool J. K. Rowling. And I’d rather have a lunch date with Linus.

1. How much data do I need?

Given your data, you can use cross-validation or A/B testing to measure objectively the effectiveness of a recommender system.

2. We have this system in place, how do we know whether it is sane?

See previous question.

3. My online recommender system is slow!

Laziness is your friend: don’t recompute the recommendations each time you have new data.

4. My customers don’t like the recommendations!

• Keep expectations in check: recommending products is difficult and even human beings have trouble doing it,
• Explain the recommendations: nobody trusts a black box,
• Allow your users to freely explore your data and products in convenient and exciting ways.

5. Which algorithm is best?

You should start with simple algorithms: it worked well enough for Amazon. To do better, a mix of different algorithms is probably best. You can combine them using ensemble methods.

Beyond the fact that software is expressed in Mathematics artefacts (bits, algorithms), are Information Systems fundamentally Mathematical?

For my convenience, I temporarily rephrase the question to something simpler and more concrete:

How are Software Developers limited by their mathematical weaknesses?

I plan several blog posts around this question, but let me start with an example.

A common and powerful language to process XML is XPath. XPath is used within web applications, scripts, databases, and so on. I often ask students the following question about XPath. Are these two expressions equivalent?

$x="some string"

and

not($x!="some string").

(The symbol “!=” means “different from”.)

Invariably, most students conclude that they are equivalent. Wrong!

Let us examine the semantics.

• The expression $x="some string" means that at least one element of $x is equal to "some string".
• The expression $x!="some string" means that some element of $x is different from "some string".
• The negation of $x!="some string" is that all elements of $x are equal to "some string". (Sorry if it sounds confusing.)

Thus, the expression not($x!="some string") is a  more restrictive condition than the expression $x="some string".

Great software developers routinely think through far more complex mathematical problems. Yet, they do not think of them as being Mathematics.

Here is my take on this issue.

1. Sharing can’t hurt the small fish. Almost nobody sets out to beat Daniel Lemire at some conference next year. I have no pursuer. And guess what? You probably don’t. But if you do, you are probably doing quite well already, so stop worrying. Yes, yes, they will give you a grant even if you don’t actively sabotage your competitors. Relax already!
2. Sharing your code makes you more convincing. By making your work easier to reproduce, you are instantly more credible. Trust is important in science. Why would anyone trust that I actually wrote the code and ran the experiments? Because I published my code, that’s why!
3. Source code helps spread your ideas faster. On the long run, you should not care about getting papers accepted at some hot conference. What matters is the impact you have had. Make it easy for me to use your ideas! Help yourself!
4. Sharing raises your profile in industry. Having open source software makes your more attractive to software engineers.
5. You write better software if you share it. While not all code I publish is bug-free, documented or even usable, I care slightly more about my code because I publish it.

Finally, does sharing code works? Do people download and use my software? Here are download statistics for my latest source-code publications:

Related reading: Good prototyping software and The challenge of doing good experimental work by Suresh Venkatasubramanian. And More on algorithms and implementation by Michael Mitzenmacher.

Update: Joachim Wuttke pointed out another potential benefit: your users will debug your code.

And publishing is easy. Write and submit many papers  conforming to the expectations of the editors. Eventually, some of your work will be accepted. And there are thousands of journals, conferences and workshops. Just write a lot.

Yet, don’t publish everything you write—even when what you wrote looks like a research paper. Hold on to it.  Because, publishing everything that looks like a research paper leads to what Feynman famously described as Cargo Cult Science. Indeed, there is a real danger that we become so good at faking science that we are no longer doing science at all! We become dishonest.

In our haste to be published…

• we cut corners in our experiments, when we validate our ideas at all;
• we pretend that our work is applicable in the real world, when it isn’t;
• we don’t take the time to reproduce and reflect on known results;
• we give the positive aspects of our research while omitting to mention the negatives;
• we complexify the issues so that our research looks fancier;
• we get lost in abstract nonsense.

If you want your work to really matter, you should be honest. You should not fool yourself and others. So what do we do? Maybe we should publish carefully. While barely reducing our output rate as academic writers, we can introduce extra steps to keep us more honest. What do we need?

• Diverse point of views: it is easy to fool a small group of like-minded experts, but comparatively more difficult to fool the readers of my blog.
• Time to reflect: if you read what you wrote months ago, and you don’t feel the urgency to communicate it more broadly, maybe it wasn’t all that good to begin with?

The problem is that once a paper is published in a journal or a conference, we tend to move on. Anyhow, we cannot easily revise our published work. Are there other models? Economists regularly publish working papers—commonly known in Computer Science as technical reports. But the difference between computer scientists and economists is that economists revise their working papers. And only when their work has stood the test of time, that is, has been available freely for months or years, do they submit it to conventional peer review.

This year, I will try the following experiment. Both on this blog and on my publication page, I will “publish” working papers and specifically ask readers to be critical of my work. Only after a couple of months have passed (or more) will I submit my work to a journal or conference.

This will introduce some latency in my publication output. Can I trade latency for quality? I plan to report back in a year on this (very public) experiment.

Further reading: Time for computer science to grow up by Lance Fortnow.

Amazon has most of the market for now. Yet, using the kindle store—on the kindle—is painful. Moreover, Amazon ebooks are protected by Digital Right Management (DRM). Amazon sells you crippled ebooks that can stop working if you copy them too often. There are often better alternatives elsewhere.

And, in Canada, there is a two-dollar surcharge for every wireless download using the Kindle. Since most ebooks are 0.5MB or less, the wireless costs 4$ per megabyte! This is insulting! Moreover, if you buy a book by mistake—which is annoying common—Amazon will reimburse the cost of the book itself, but not the fee for the wireless download.

Thankfully, you can grab books compatible with the kindle (in Mobipocket format) elsewhere. Then you can drop the file on the kindle using the USB port.

• You can get nearly 2000 of the great French classic for free on ebookgratuits. This include a large fraction of the work of Honoré de Balzac.
• Project Gutenberg offers 30,000 free e-books in various languages (mostly English).
• WebScription sells DRM-free ebooks in various format. Most books fall into the scifi, young adults and fantasy genres.

I am currently reading You’re Not Fooling Anyone When You Take Your Laptop to a Coffee Shop by Scalzi. I bought it at WebScription for six dollars. It is a compilation of Scalzi’s blog posts on his life as a writer. I am fascinated by how much it ressembles my own life. Well… Except for the fact that I don’t get paid when I publish a paper. Maybe I should put together a compilation of posts about my silly work life. Would anyone buy it for six dollars?

I am also reading Halting State by Stross which I bought on Amazon for ten dollars. I haven’t yet gotten into the mood of the novel.

Further reading:

• According to a Computer Scientist, the iPad could make Computer Science obsolete.
• While I don’t think academic journals will be available on the Kindle any time soon, I think that has mostly to do with how insane academic publishing is.

I have already written much on the topic but let me reiterate my message:

• In this new online world, professors are not content providers. They provide structure and motivation. They are role models. And most importantly, by their reputation, professors can provide certification. If someone gets a reference letter from Michael Mitzenmacher or Dick Lipton, I trust they know something about Computer Science, because I trust Michael Mitzenmacher and Dick Lipton. I suspect it is not easy to get these fellows to write fake reference letters because they have a high degree of independence (job security, good money, and so on) and their greatest asset is their reputation.
• Students are trained to expect classrooms. Many students need structure and constant attention. That is not a good thing! We are effectively training students to be good employees working in large organizations with much structure. Yet, this world made of large and stable organizations has already fallen apart. We urgently need to teach students to learn on their own, using the Web.
• Yes, there will always be campus classes, the same way there will always be physical libraries with actual books, and newspapers printed on paper.

Further reading:

• College students prefer classes with online learning, University of Wisconsin-Madison
• Stanford offers 10 free online Computer Science courses
• From online courses to… automated teaching
• Some myths about online teaching
• Why don’t people use university libraries?

Of course, being rejected does not mean you are original. It could also mean that you are sloppy or uninteresting. However, there may be valid indications of your originality such as:

• You have no competitor. Nobody quite does what you do.
• You cannot be scooped. You read new issues of journals looking for fresh ideas, but without fear that someone made you irrelevant.

As MacLeod put it: Don’t try to stand out from the crowd; avoid crowds altogether.

Further reading: The secret behind radical innovation and A recipe for interesting Computer Science research papers.

Most of my writing time is spent on LaTeX documents. I have tried typical Word processors in the past, but they get in my way. Indeed, by mixing document content and document presentation, Microsoft Word makes it difficult to maintain consistency. Word is meant for short-lived (or throw-away) business documents. It is easy to get started and get 80% of the job done with Word. However, as the document gains complexity, as the number of revisions grow, as the number of collaborators expands, Microsoft Word becomes inadequate.

Oh! I still use OpenOffice or Google Docs to produce quick-and-dirty documents. But for anything that is meant to have lasting value, that is research, I refuse to fall into the Word processor trap. It causes some friction with colleagues, but it is a price I am willing to pay.

I believe every single graduate student should learn to write without a word processor. And serious science students should learn LateX. Even if you do not care for LaTeX, at least explore alternatives to Word such as Scrivener.

In any case, you are unlikely to need more than a text editor to write your prose:  Charles Stross, one of the best scifi writer alive, wrote many of his novels using a primitive text editor (Vim). If you have never written without Microsoft Word, how do you know that Word is not holding you back?

Right now, I write using a regular text editor (Smultron for MacOS) and the TeX Live 2009 distribution. I save all my documents to a subversion tree. Using a version control tool such as Subversion makes collaboration easy, and it allows me to go back in time years ago. It is a good setup.

Programming is also a form of writing. For my experimental work, I program in C++, Java or Python, often using Eclipse. I find it is slightly better for programming than my standard writing setup (using only a text editor). Eclipse has great qualities:

• It stays out of the way. In particular, you can collaborate with people who are not using Eclipse without any problem.  For example, it will happily let you use handcrafted makefiles to compile your C++ programs.
• It offers incremental compilation of Java programs. Basically, it compiles as you type.
• It suggests corrections for many common compilation errors.

Essentially, while Java is still an awful language, Java with Eclipse is almost fun. Eclipse proves that sophisticated software can be helpful to programmers and writers.

Writing is hard and it will always be hard, no matter the tool. But at least, ease your pain!

See also Physical tools to improve research productivity.

Algorithms (and their implementations) also have fundamental properties. Running time and memory usage are the obvious ones. In practice, there is often a trade-off between memory usage and the running time: you can a low memory usage, or a short running time, but not both. Michael Mitzenmacher reminded me this morning of another: correctness. On some difficult problems, you can get a low memory usage and a short running time if you accept an approximate solution.

I believe there are other fundamental properties like latency. Consider problems where the volume of the solution and of the input is large: statistics, image processing, finding some subgraph or sublist, text compression, and so on. In such instances, the solution comes out as a stream. You can measure the delay between the input and the output. For example, a program that compresses text by first scanning the whole text might have high latency, even if the overall running time is not large. Similarly, we can give the illusion that a Web browser is faster by beginning the Web page rendering faster, even if the overall running time of the rendering is the same. As another example, I once wrote a paper on computing the running maximum/minimum of an array where latency was an issue.

It would be interesting to come up with a listing of all the fundamental properties of computing.

The stated goal was to make degrees more accessible. We succeeded.

Yet, we are now facing an intriguing paradox due to this success. Technology, by making access easier than ever to access educational content, is also shaking the very foundation of the University. As an example of this transformation,  Michael Nielsen was pointing out this morning that you can watch 120 hours of lectures on Physics by Lenny Susskind, for free on YouTube. You are in deep trouble if what you are selling in 2009 are mass-produced lectures. The market price just went through the floor.

Lance Fortnow pointed us to a short essay by Martin Rees about technology and universities. Rees’ point is that technology creates a more level playing field as far as location is concerned. A hundred years ago, airplanes made it possible for Indian Mathematicians to travel to Cambridge where they could be taken seriously. In some sense, airplanes made Indian Mathematicians more globally competitive, though only marginally so. The Web—with repositories such as arXiv—pushes this idea further, an order of magnitude further. After all, Gregori Perelman won a million dollar and the equivalent of a Nobel prize by posting a few papers on arXiv.

The revolution is all around us, not just in Science. Recently, an unknown writer, Sam Landstrom, posted his novel MetaGame on the Amazon Kindle. No publisher, no ad campain. Sales rank of his novel? 540. Considering that Amazon reported selling more ebooks than paper books over Christmas, I am sure many authors envy Landstrom success. Yet, Landstrom did not need an office New York City to either write or publish his book. For all I know, he lives in his parents’ basement.

Thankfully, bona fide Universities have some form of monopoly on University degrees. Yet, like Rees, I think that we are coming to the end of the road for mass universities:

Traditional universities will survive insofar as they offer mentoring and personal contact to their students. But it’s less clear that there will be a future for the ‘mass university’ where the students are offered little more than a passive role in lectures (generally of mediocre quality) with minimal feedback.

One thing is clear to me: The value of a lecture in front of 80 students—or the equivalent as a webcasted show—is exactly zero. (From an educational point of view.)

Disclaimer: Yes, I know that I am making ennemies. Feel free to disagree with me.