((( All You Need Is Lisp )))

by Dmytro Pinskyi

Is life too short either for hacking on blubs and for adopting mature lisp ecosystems? Are we strong enough already to lisp out of the blubset?

AYNIL — GitHub repository

«I object to doing things that computers can do».
— Olin Shivers

The following article addresses practical issues related to project AYNIL which is about running lisp over JS. Author relies on three years experience of using Common Lisp for web development.

Keywords: lisp, high-level machine language, information technology, meta-analysis.

«Greenspun's Tenth Rule of Programming: any sufficiently complicated C or Fortran program contains an ad hoc informally-specified slow implementation of ».
— Philip Greenspun

Overview

Lisp (standing for "list processing") is one of the oldest high-level machine languages that are still in use today. Lisp was discovered in October 1958 by John McCarthy, professor at MIT and Stanford, also famous as one of the artificial intelligence discipline founders and the author of the term "AI". He suggested to implement symbolic computations on Fortran, then to include such feature into Algol, but no. Finally he got to the point of creating a new language. Lisp became the very first language with garbage collection (automatic memory allocation) which is now considered a must-have.

Lisp is different than any other language. It's extremely pure and simple syntax is based on the concept of directly representing abstract syntax trees (AST) by the code. This and only this makes possible to process the same code equally as either a piece of data or a program. Lisp users are able to compose programs, which by themselves, out of the previously evaluated data, write another programs and run them just-in-time. Lisp gives users full computation control, so they instantly become language developers.

However, after sixty years of awesomeness, lisp still is definetely not a mainstream information technology. There are various opinions around the internet. Let us surf some top publications and provide needed clarifications about why and how we should probably use lisp today.

"Beating the Averages" by Paul Graham
"Introduction: Why Lisp?" by Peter Seibel
"Why Lisp Failed" by Steven Fuerst

"Beating the Averages"

Origin: Paul Graham — author of "On Lisp", "ANSI Common Lisp", etc.

In this classical text the definition of blub is introduced. I remember, that influenced me very much.

«Programmers get very attached to their favorite languages, and I don't want to hurt anyone's feelings, so to explain this point I'm going to use a hypothetical language called Blub. Blub falls right in the middle of the abstractness continuum. It is not the most powerful language, but it is more powerful than Cobol or machine language.
And in fact, our hypothetical Blub programmer wouldn't use either of them. Of course he wouldn't program in machine language. That's what compilers are for. And as for Cobol, he doesn't know how anyone can get anything done with it. It doesn't even have x (Blub feature of your choice).
As long as our hypothetical Blub programmer is looking down the power continuum, he knows he's looking down. Languages less powerful than Blub are obviously less powerful, because they're missing some feature he's used to. But when our hypothetical Blub programmer looks in the other direction, up the power continuum, he doesn't realize he's looking up. What he sees are merely weird languages. He probably considers them about equivalent in power to Blub, but with all this other hairy stuff thrown in as well. Blub is good enough for him, because he thinks in Blub.
When we switch to the point of view of a programmer using any of the languages higher up the power continuum, however, we find that he in turn looks down upon Blub. How can you get anything done in Blub? It doesn't even have y.
By induction, the only programmers in a position to see all the differences in power between the various languages are those who understand the most powerful one. (This is probably what Eric Raymond meant about Lisp making you a better programmer.) You can't trust the opinions of the others, because of the Blub paradox: they're satisfied with whatever language they happen to use, because it dictates the way they think about programs».

Hereinafter we find evidence that real life needs macros as much as oxygen in the air.

«The source code of the Viaweb editor was probably about 20-25% macros. Macros are harder to write than ordinary Lisp functions, and it's considered to be bad style to use them when they're not necessary. So every macro in that code is there because it has to be. What that means is that at least 20-25% of the code in this program is doing things that you can't easily do in any other language».

Later on is pointed that lexical closures too were introduced by lisp in the early 1970s.

"Introduction: Why Lisp?"

Origin: Peter Seibel — author of the "Practical Common Lisp" book.

«If you think the greatest pleasure in programming comes from getting a lot done with code that simply and clearly expresses your intention, then programming in Common Lisp is likely to be about the most fun you can have with a computer. You'll get more done, faster, using it than you would using pretty much any other language».

I confirm, I really had a lot of pleasure inventing wheels for my mainstream domain. Just sometimes having pleasure all alone is not the main priority and there is a goal to build an application almost totally out of vendor modules glued by a drop of business logic.

As to performance:

The same forces that drove Lisp's feature evolution also pushed the envelope along other dimensions — big AI problems eat up a lot of computing resources however you code them, and if you run Moore's law in reverse for 20 years, you can imagine how scarce computing resources were on circa-80s hardware. The Lisp guys had to find all kinds of ways to squeeze performance out of their implementations. Modern Common Lisp implementations are the heirs to those early efforts and often include quite sophisticated, native machine code-generating compilers. While today, thanks to Moore's law, it's possible to get usable performance from a purely interpreted language, that's no longer an issue for Common Lisp.

The introduction also highlights such Common Lisp features as dynamic typing, condition (signal) system, object system.

"Why Lisp Failed"

Origin: Steven Fuerst — PhD of Computer Science, owner of Lockless Inc.

At first sight the above issue seems pessimistic or provocative. However, not only Neil is mostly right, yet he testifies in favor of the concepts which I am used to follow in designing AYNIL.

«The reason Lisp failed was that it was too successful at what it was designed for. Lisp, alone amongst the early languages was flexible enough that the language itself could be remade into whatever the user required. Programming with the other early languages involved breaking a task into small sub-tasks that could then be implemented. The larger tasks could then be implemented in terms of the smaller ones. Lisp was different, due to its power, a programmer would be able to design a domain-specific language that would perfectly solve the task at hand. Due to the orthogonality of the language, the extensions written would work seamlessly with the core language».

This must be a joke. Programming is always programming (see Turing equivalence), but languages may provide more or less power to the programmer. Should be mentioned that there are packages in Common Lisp which are used similarly (skipping details here) to CommonJS modules, either is an advanced "system definition facility" (module dependency manager).

«So what is the problem with creating domain-specific languages as a problem solving technique? The results are very efficient. However, the process causes Balkanization. It results in many sub-languages all slightly different. This is the true reason why Lisp code is unreadable to others. In most other languages it is relatively simple to work out what a given line of code does. Lisp, with its extreme expressibility, causes problems as a given symbol could be a variable, function or operator, and a large amount of code may need to be read to find out which».

From my point of view, the problem is not actually about the lisp's ability to represent DSLs or whatever. There are two following sides here.

Firstly, I believe too that the language kernel should be as small as possible to cover the host environment functionality. This is the known killing feature of Scheme, meanwhile the Common Lisp standard was consciously bloated so that (theoretically) there should be no need for any vendors, you just go and happily write all the things from scratch. This was good for U.S. industry and army of 80th, but not today.

Secondly, the syntax should also be as simple as possible. On this field Common Lisp is today's leader, it has not only a lot of cosy parentheses, but such a useful feature as separated binding scopes for entities that differently evaluate: variables, functions, macrosymbols, macros, packages, systems... I believe that both of the mentioned two sides of lisp power should be implemented in AYNIL.

«Unfortunately, so much time has passed, and so many Lisp variants have been created, that yet another new language based upon it is probably not the answer. There simply will not be enough users to make a difference. Perhaps the solution is to slowly add Lisp-like features to languages within the ALGOL family. Fortunately, this seems to be what is happening. The newer languages (C#, D, Python etc.) tend to have garbage collectors. They also tend to be even more orthogonal than the older languages. The future may eventually contain a popular language that behaves much like Lisp».

I agree with Steven that crucial is the role of two following successful programming framework's aspects. Firstly, a user-friendly ecosystem: tonns of users, reviewers and contributors. And secondly, a transparent threshold of entry: ability to integrate easily into an existing system so the system lives on. Speaking in general, it is known that evolution always needs a currently working transformation method.