Programming Languages
In The Hundred-Year Language Paul Graham pretend that the language that we will use in 100 years would be great to have now. I think he is wrong. He probably also think he is wrong because he seems to have stopped working on Arc. In Worse is Better, Richard Gabriel argue that Unix and C came to lead because implementation simplicity is as important as interface simplicity. Every now and them you read a post on Planet Lisp about how we should fix Lisp. Since Paul Graham isn't likely to do it lets look at what it means and how someone could do it
Lisp is more or less a family of programming languages with a few key features though some may argue about the exact minimal set to be called a Lisp: simple sexp syntax, dynamic typing, macros, garbage collection, first class functions, closures and few other things like built-in linked lists and programming in the ast instead of line oriented command lists. There are two members of that family that still have a decent community and who can be used for general programming: Common Lisp and Scheme.
Scheme can be thought the bare minimum you can have to be a Lisp. In fact its probably the bare minimum you can have to be a useful programming language. In Scheme there are no loops, you recurse all the time but the spec makes it mandatory that your implementation will do tail call elimination. Scheme has other power features like continuations. Scheme is really small and that makes it and ideal embeddable scripting language. Sadly the spec don't include a module system and all the implementations now have more or less incompatible ideas of how it should be done. That means that you won't find repositories of libraries for scheme that you can expect to work across many implementations. That very sad point more or less rule out Scheme for general programming and I won't discuss it further.
Common Lisp (CL) is at the other hand of the spectrum. With Common Lisp you have huge language with many features including optional type declarations, selectable levels of compilation (safe, fast, debug, ...), many native types and containers, exceptions (and other "conditions") and an object system so flexible that I would need ten pages just to list its features. There are many cross implementation libraries out there and that makes Common Lisp a great language for general programming. This language is extremely flexible. You typically bent the language into the language you would want to solve a family of problems and use your new language to program into. This is probably new to most programmer but making your own domain specific language can be quite easy and you don't need to mess with a formal grammar if you have macros. The Common Lisp macros let your use Common Lisp to rewrite that code that is already in AST form. You can make code walkers that rewrite definitions when keywords are found, you can add new looping constructs, say to iterate a ternary tree and your new language embeds well into CL and support embedded CL transparently. That being said I often decide to use Python or some other language because of some problems with Common Lisp. I think can we can "fix Lisp", we just need to take the time to diagnose the problems.
Common Lisp was standardized a long time ago. It is impressive that a language could be designed so early and still kick the ass of modern languages feature wise. On the other hand, the designs feels baroque in some ways. A major annoyance is the pathname interface. In modern languages you refer to a file with a string, a path to your file into the directory tree where "/" separate directories. Some languages include a function to merge directories so you don't notice that there are broken systems that don't use "/" for the right thing. Some globing is expected to work almost everywhere like "*.txt". In CL the interface to refer to files is over engineered and included aspects that no modern OS support anymore like versions of a file. In Python you would open(join(base, "foo-"+bar+".txt")), in CL you can be sure that even if raw strings instead of pathnames some implementation will parse your paths and wreck havoc when it find funny chars in your filenames. Yes you can fix that with some packages but since those are not part of the spec, implementations don't include them. So you can't expect that if your mail someone your snippet of code using non-baroque pathnames it will run.
That brings me to my next irritation: shell scripting. Bash is a damn good command interpreter but a really bad scripting language. I mean, foo="blah qwe" should be the same as foo = "blah qwe" don't you think? So I don't script in bash unless what I want to do it really simple. You'd expect that CL with its tons of libraries and specialty looping constructs would be my scripting language but it isn't. The spec is quite strict about the fact that "#" is not a comment char. So no "#!/usr/bin/cl" for you guys. Ok we can work around that, the fix is ugly and still not included with the default install of most implementations but there is more to it. A long time ago, it seems, when the CL spec was drafted, the concept of process wasn't as pervasive as it is today. Common Lisp don't include a way to call another program. Before you over react to that statement, rest assured that all the implementations include a non-standard way to do so. You'll find some really fancy stuff with communication channels much more flexible than pipes but the fact that simple stuff like that isn't portable is argument against using CL as your shell scripting. Needless to say that support to get the command line arguments is strange.
Another problem with is deployment. Most packages these days use ASDF. Unlike Pythons and Perls, there is no support for packaging and deployment of CL modules. ASDF only covers the loading of the module. That makes publishing CL code more painful than it should be so many potential libraries just sit on hard drives and never get released. Here is what Trevor Blackwell has to say about it
It's worth distinguishing a third category of programming environments: one where a solitary hacker uses many open-source modules written by various folks. I consider this a very good environment, unlike pack programming. But it has some requirements in common with it. Various languages succeed or fail dramatically in making this work smoothly. In C/C++, I find I can almost never use other people's stuff conveniently. Most have some annoying requirements for memory management, or use different string types (char *, STL string, GNU String, custom string class), or array/hash types, or streams (FILE *, iostream) or have nasty portability typedefs (int32, Float). CL has fewer such troubles since it has standard memory management & string/hash/array types, but there are often nasty namespace collisions or load order dependencies, especially with macros. Most chunks of CL code I've seen (which are mostly PG's) won't work without special libraries, which have short names and are likely to conflict with other macro libraries or even different versions of the same macro libraries. Perl packages work pretty well, because everyone agrees on how basic types like string, stream, array and hash should work, and the normal use of packages avoids any namespace collisions. I don't think I ever had a open source Perl module break something. I guess Java also prevents conflicts, but you have to give up an awful lot to get it. The huge assortment of open source Perl packages testifies to the ease of writing them. I've taken a few packages that I wrote for my own purpose and found it very easy to make them self-contained and contribute them. Usually, people only write C++ libraries for very large projects, and it requires a different programming style from what you'd use for your own code. Anyway, I suggest that Arc's modularity features should be designed to support use of open-source modules in single-hacker projects, not to support pack programming. This suggests, in addition to what CL already has: - that modularity be a convention (CL, Perl), not something enforced by the compiler (Java). Sadly, I find the CL module system way to cumbersome to actually use. It has to be convenient enough to use in ordinary programming, not a special thing you use when you're writing a module for external use. - there be a sufficient basic library that everyone won't have to write their own basic library. I'm talking about the sort of functions from On Lisp like last1, in, single, append1, conc1, mklist, flatten. If everyone has their own, then you have to package it up with any code you publish (making it awkward to publish) and it'll be hard to read other people's code with different definitions of basic functions.
Now, what do we learn from that? Ok maybe you don't know CL and you think that its a crap language that isn't worth learning. You are probably wrong but the bottom line is that the best language ever designed isn't that useful if it isn't aware of your computing environment. For The Hundred-Year Language, I just don't care how good your language is if it's trapped into autism.
Can we fix CL? I don't know. Should we? No. Paul Graham is right, its time for the next Lisp in the family. But Paul Graham is wrong, its not the time to make another CL, its time to make another C, that is a simple language with a simple implementation that will be useful now, not in 100 years. He is also wrong when he say that his language should not be to script Unix. Scripting Unix still sucks but its still how many great packages are started. What are web apps if not Unix scripting anyway? You look at Perl, Python and Ruby and you know that no matter how bad your language is, as long as it include more Lisp features than the one before it you will have a large hacker base. And the hackers are the ones who make the libraries and thats what makes a language useful for day to day coding. The right language will make simple things real simple and hard stuff doable. CL make it possible to make a symbolic algebra system but way too hard to parse my server logs. Perl got popular when sysadmins decided that bash wasn't good for scripting opted for the next quick hack language.
How do we make this new Lisp? I don't know. Paul Graham make some work and posted a few ideas from other. Dave Moon's comments are interesting. I like the idea of some bits of syntax. Perl has way to much syntax but hash tables are definitely more convenient to use in Python than in CL: hash[key]=foo vs (setf (gethash hash key) foo). We probably can get 80% of non-baroque CL in 20% of the time it takes to make a CL implementation. I think one has to write down a few snippets of code in his new language to see how it looks than draft some spec and to grind down an implementation. One thing that I'm sure it that this implementation should not be done with CL macros. That would make it too tempting to keep plenty of baroque constructs. CL has multi methods but there is something nice with "mono methods": it prevents namespace pollution. When methods are part of the object, methods with the same name can have different signatures. There is a use for both kind of methods, I think don't know how well they could interact. Ok there are many other aspects that need to be looked into and I don't know where to start.
There is an urgent need for a modern Lispish programming language. This quote form Wolfram sums it up What's made it happen for me is Mathematica. I built Mathematica to have a general and very high-level computational environment that would let me do experiments and set up models easily. And it's worked fantastically well. I would never have been able to discover more than a tiny of fraction of what I have without Mathematica.