The remaining six Clojure/conj 2011 videos have been released, and all videos are now available on Clojure's blip.tv page.
Thank you once more to all of our speakers and attendees. If we don't see you at Clojure/west or EuroClojure first, we'll see you at the next Conj!
Scheduled Talk Videos
Keynote Rich Hickey
Cascalog: Making Data Processing Fun Again Nathan Marz
Extreme Cleverness: Functional Data Structures in Scala Daniel Spiewak
Performance in the Wild Craig Andera
The Macronomicon Michael Fogus
Programming Music with Overtone Sam Aaron
Enter Rich, with Parentheses
In the beginning, there was a guy with an idea. That guy was Rich Hickey, and his idea was to combine the power of Lisp with the reach of a modern managed runtime. He started with Jfli, embedding a JVM in Lispworks' Common Lisp implementation. When that proved inadequate, he took a two-year sabbatical to write the compiler that would eventually become Clojure: a completely new Lisp for the JVM with language-level concurrency support.
In late 2007, Rich Hickey presented Clojure at a meeting of the New York Lisp users' group, LispNYC. I was there, and I was so excited by what I saw that I wrote one of the first blog articles about Clojure. Three days later, I was asking questions about Java interop on the Clojure mailing list.
Those early days were fun, participating in heady discussions about fundamental language features like nil vs. false and argument order. It felt like the beginning of something genuinely new. The community was tiny, and Rich participated in almost every discussion on the mailing list or IRC.
How times have changed. The Clojure mailing list has over five thousand members, and we just wrapped up the second international Clojure conference with nearly four hundred attendees. Google Groups tells me I've racked up over a thousand posts on the mailing list, which is shocking to me. There are five books and counting about Clojure. People are building businesses and careers on it. Who would have guessed, in 2007, that we would be here in just four years?
Enter Second Stuart
(That was a cheap shot. Hi, Stu! :)
In the Summer of 2008, Stuart Halloway started blogging about Clojure. With his co-founder Justin Gehtland, Stuart H. had already built a business helping big companies navigate from ponderous Java development to more agile practices and more expressive languages like Ruby. Stuart H. decided that Clojure was the next big thing. He wrote the first book about Clojure (soon to get a 2nd edition). When he and Rich met at the 2008 JVM Language Summit, they started a long conversation that would eventually become a partnership.
Clojure Contrib: The Beginning
Around the same mid-2008 time frame, "clojure-contrib" began its life as a Subversion repository where community members could share code. There were twelve committers and no rules, just a bunch of Clojure source files containing code that we found useful. I contributed str-utils, seq-utils, duck-streams, and later test-is.
The growth of contrib eventually led to the need for some kind of library loading scheme more expressive than load-file. I wrote a primitive require function that took a file name argument and loaded it from the classpath. Steve Gilardi modified require to take a namespace symbol instead of a file. I suggested use as the shortcut for the common case of require followed by refer. This all happened fairly quickly, without a lot of consideration or planning, culminating in the ns macro. The peculiarities of the ns macro grew directly out of this work, so you can blame us for that.
Clojure-contrib also prompted a question that every open-source software project must grapple with: how to handle ownership. We'd already gone through two licenses: the Common Public License and its successor, the Eclipse Public License.
Rich proposed a Clojure Contributor Agreement as a means to protect Clojure's future. The motivation for the CA was to make sure Clojure would always be open-source but never trapped by a particular license. The Clojure CA is a covenant between the contributor and Rich Hickey: the contributor assigns joint ownership of his contributions to Rich. In return, Rich promises that Clojure will always be available under an open-source license approved by the FSF or the OSI.
Some open-source projects got stuck with the first license under which contributions were made. Under the CA, if the license ever needs to change again, there would be no obstacles and no need to get permission from every past contributor. Agreements like this have become standard practice for owners of large open-source projects like Eclipse, Apache, and Oracle.
Clojure/core and "New Contrib"
In 2010 I left my cozy academic job and went to work for Relevance, where Stuart Halloway and Rich were discussing a strategic partnership that would eventually become Clojure/core. So what is Clojure/core? It's a business initiative of Relevance (though not an independent business entity) to provide consulting, training, and development-for-hire services around Clojure. Rich Hickey is an advisor to Clojure/core, but not a Relevance employee.
Members of Clojure/core, of which I am one, have made a commitment to spend their 20% time supporting the Clojure ecosystem. Although Rich still personally reviews every patch for the language itself, the job of answering questions and organizing contributions from a 5000-member community is too big for one person, so Rich delegated that responsibility to Clojure/core.
The first big issue Clojure/core had to confront was the distribution of clojure-contrib. With sixty-plus libraries in one binary release, it was already unwieldy. Since clojure-contrib releases were tied to Clojure language releases, which happened infrequently, development had stalled. There was also vast confusion about what, exactly, clojure-contrib was meant to be. Was it an essential component of the language, a nascent standard library, or a load of crap? (I was inclined to the latter view, especially with regard to my own contributions.)
My attempts at modularizing clojure-contrib within a single Git repository failed to improve the situation. Eventually, we settled on the solution of separate Git repositories for each library. This was a huge amount of work: dozens of repositories to create and hundreds of files to move. Many of the contrib libraries were stagnant, their original authors lacking time to continue working on them.
Finally, almost a year later, the situation has stabilized: 28 libraries, each with its own Git repository, test suite, continuous integration, and independent release cycle. The overall code quality is higher and development is moving forward again.
It was a painful transition for everyone, not least for those of us trying to manage it all and bear the brunt of the inevitable carping. On top of everything else, the whole process overlapped with the release of Clojure 1.3, the first release to break backwards-compatibility in noticeable ways (non-dynamic Vars as a default, long/double as default numeric types).
Our technology choices for Clojure and "new contrib" — GitHub, JIRA, Hudson, and Maven — were driven by several concerns:
- to be first-class participants in the Java ecosystem
- to preserve the future-proof licensing structure of the CA
- to give library developers freedom to develop/release on their own schedule
- to ensure changes are made only after a thorough review process
The last point was particularly important for patches to the Clojure language. Clojure is very stable: since its first public release, implementation bugs have been rare and regressions almost nonexistent. Most reported bugs are edge cases in Java interop. But stability has a price: new features come more slowly. The majority of JIRA tickets on Clojure are really feature requests. Rich is extremely conservative about adding features to the language, and he has impressed this view on Clojure/core for the purpose of screening tickets.
To take one prominent example, named arguments were discussed as far back as January 2008. Community members developed the defnk macro to facilitate writing functions with named arguments, and lobbied to add it to Clojure. Finally, in March 2010, Rich made a one-line commit adding support for map destructuring from sequential collections. This gave the benefit of keyword-style parameters everywhere destructuring is supported, including function arguments. By waiting, and thinking, we got something better than defnk. If defnk had been accepted earlier, we might have been stuck with an inferior implementation.
Conversely, the decision to move some libraries into the language, notably my testing library, was probably premature. (Stuart Halloway accepts blame for that one. :) Some of the decisions I made in that library could use revisiting, but now clojure.test is what we're stuck with.
Clojure/dev and the Future
If there was one mistake that I personally made during the 1.3 migration, it was speaking as if Clojure/core owned Clojure and clojure-contrib. We don't: Clojure is owned by Rich Hickey, and clojure-contrib is owned jointly by Rich Hickey and contributors. But we are the appointed stewards (and Stuarts!) of the open-source Clojure ecosystem. In that role, we have to make decisions about what we choose to invest time in supporting. Given limited time, and following Rich's conservative position on new features, that decision is usually "no."
It's a difficult position to be in. Most of Clojure/core's members come from the free-wheeling, fast-paced open-source world of Ruby on Rails. We really don't enjoy saying "no" all the time. But a conservative attitude toward new features is exactly the reason Clojure is so stable. Patches don't get into the language until they have been reviewed by at least three people, one of them Rich Hickey. New libraries don't get added to clojure-contrib without multiple mailing-list discussions. None of the new contrib libraries has reached the 1.0.0 milestone, and probably won't for some time. These hurdles are not arbitrary; they are an attempt to guarantee that new additions to Clojure reflect the same consideration and careful design that Rich invested in the original implementation.
So what is clojure-contrib today? It's a curated set of libraries whose ownership and licensing is governed by the Clojure Contributor Agreement. It could also serve as a proving ground for new features in the language, but this does not imply that every contrib library will eventually make it into the language.
With the expansion of contrib, we've given name to another layer of organization: Clojure/dev. Clojure/dev is the set of all people who have signed the Clojure Contributor Agreement. This entitles them to participate in discussions on the clojure-dev mailing list, submit patches on JIRA, and become committers on contrib libraries. Within Clojure/dev is the smaller set of people who have been tasked with screening Clojure language tickets. Clojure/core overlaps with both groups.
At the tail end of this year's Clojure/conj, Stuart Halloway opened the first face-to-face meeting of Clojure/dev with these words: "This is the Clojure/dev meeting. It's a meeting of volunteers talking about how they're going to spend their free time. The only thing we owe each other is honest communication about when we're planning to do something and when we're not. There is no obligation for anybody in this room to build anything for anybody else."
One consensus that came out of the Clojure/dev meeting was that we need to get better at using our tools, particularly JIRA. We would like to streamline the processes of joining Clojure/dev, screening patches, and creating new contrib libraries. We also need better integration testing between Clojure and applications that use it. Application and library developers can help by running their test suites against pre-release versions of Clojure (alphas, betas, even SNAPSHOTs) and reporting problems early.
But Stu's last point is an important one: no one in the Clojure community owes anybody anything. If you want something, it's not enough to ask for it, you need to be willing to do the work to make it happen. At the same time, don't let a lukewarm response to ideas on the mailing list dissuade you from implementing something you think is valuable. It might just be that no one has time to think about it. Recall keyword arguments: more than two years from inception to completion. We're in this for the long haul. Join us, be patient, and let's see where we can go.
William Byrd is a postdoc at Indiana University researching the applicability of declarative programming techniques to high-performance computing domains. In addition, he is one of the three authors of the fantastic book The Reasoned Schemer that walks the reader through the development of a small but powerful logical programming system in Scheme called miniKanren. Dr. Byrd attended the 2011 Clojure Conj and gave (along with Dr. Friedman) what became one of the best talks of the conference -- during the conference off hours. In this candid installment in the (take) series of micro-interviews Dr. Byrd discusses Clojure, Scheme, Lisp, macros, and miniKanren's evolution.
A trend in the Clojure language and its ecosystem is an increasing emphasis on declarative programming models, including Datalog, logic, and relational programming. Is this progression unique to the Clojure evolution, or is it more emblematic of a more general trend?
Declarative programming has been used in industry for decades: HTML, XSLT, GNU Make, and SQL are all declarative languages. I see a slightly different trend, towards programmers designing their own domain-specific languages, many of which are declarative. Of course Lispers have been doing this for decades--Lisp macros are arguably the most powerful tool for creating DSLs. Therefore Clojure is almost perfectly positioned as a pragmatic language for DSL creation. Almost, but not quite.
Not quite, because Clojure's macro system appears to me to be a kludge, trying to patch up Common Lisp's defmacro without going all the way to real hygienic macros. I don't mean to be overly critical--I'm not happy with the state of Scheme, either, which has not one but two hygienic macro systems, neither of which is ideal.
The more powerful Scheme macro system, syntax-case, gives the programmer the full power of Scheme at macro expansion time, along with the ability to "bend" hygiene when desired. Syntax-case is powerful enough to define Common Lisp's defmacro (and almost certainly powerful enough to define Clojure's defmacro as well). Unfortunately, syntax-case is notoriously complicated. As far as I can tell, mastering syntax-case requires a mental model equivalent in complexity to the macro expander's implementation. As a result, Dan, Oleg, and I refuse to use syntax-case.
The other Scheme macro system, syntax-rules, is strictly less powerful than syntax-case, and can in fact be implemented in syntax-case. However, the mental model required to use syntax-rules is much simpler, as it is essentially a term-rewriting system with hygiene. For vanilla forms such as let, and, and or, syntax-rules works beautifully. For incredibly complicated macros, such as the match pattern matcher used in Kent Dybvig's compiler course at Indiana, the full power of syntax-case is probably required. For almost all other macros, it seems like a system with just a little more power than syntax-rules would be both sufficient and ideal. For example, often a macro writer wishes to concatenate two symbols--currently this requires using syntax-case, even if the rest of the macro is trivial.
If a hygienic macro system that hits the sweet spot between syntax-rules and syntax-case is created, I hope it will be adopted by Schemers and Clojurers alike. A few steps in this direction include the Dylan macro system and Ryan Culpepper's syntax-parse. Also needed is a way to learn how to write non-trivial hygienic macros that doesn't involve earning a PhD in programming languages from Indiana or Northeastern.
Because of the limitations imposed by the JVM, is Clojure unable to address certain classes of problems due to a limitation in its support for CPS?
Clojure is Turing-complete, so in theory the answer to all such questions is 'no'--you can always encode the problem using iteration. In practice, you lose many of the powerful correctness-preserving transformations enabled by CPS (CPS is the gateway drug of program transformations).
Even worse, there is a tremendous hidden cost to the Clojure community. Schemers should be the greatest fans of Clojure; instead, they see the improper handling of tail calls, along with a suspect-looking macro system, and yawn instead of jumping onboard. I think the macro situation can and should be improved; I don't know what to do about tail calls, other than clearly explain the design tradeoffs to Schemers and challenge them to do better.
How do you see the future of Lisp progressing?
Common Lisp has no future--if taking the union of features from multiple Lisp Machine implementations didn't kill it (why have one object system when you can have three?), standardization did. Fortran is still evolving; Common Lisp isn't.
So far the Scheme community has avoided making the same mistakes. Historically, only the intersection of features from the major implementations have made it into the standard. The result is a standard language that is useless for real-world programming (FFI and threads aren't in the current standard, for example) but encourages researchers and implementers to experiment with new features. In fact, Scheme shouldn't be viewed as a complete programming language--rather, Scheme is a dialect of Lisp, and the various implementations (Racket, Chez, Ikarus, etc.) are dialects of Scheme. Scheme will never become a popular language, but popular languages are becoming more like Scheme. For example, Ruby ripped off much of Scheme--alas, Matz decided not to rip off the most important part, hygienic macros, which is why Ruby programmers think the solution to every problem is to hack the interpreter.
One interesting Lisp you didn't mention is Dylan, which was created by Apple in the 90's and then abandoned. Dylan is an object-oriented infix-notation Scheme designed for systems programming, with well-designed libraries, real hygienic macros, and outstanding performance. I'm trying to steal a few cool ideas from Dylan--Clojurers should as well. Dylan is dead, but hopefully its ideas will live on.
Clojure feels like a different beast entirely. Clojure has one canonical implementation, one lead designer, no rigorous standard (this part needs to change!), an enthusiastic and rapidly growing community, and a very practical bent. From that perspective, Clojure sounds much more like Ruby or Python than Lisp. Two major differences I see are the quality of the implementation (Clojure shows Pythoners that pragmatic dynamic languages needn't be slow) and the incorporation of STM, fast immutable data structures, logic programming, and other useful but potentially scary technologies that haven't made their way into the fundamentally conservative world of "scripting" languages (where much of the innovation is in syntactic sugar, since these languages lack hygienic macros).
Will Clojure become as popular as Ruby or Python? Probably not. Will the languages that replace Ruby and Python be influenced by Clojure? Almost certainly. This is one reason it's important for Clojurers to understand the weaknesses of the language, and not just its strengths.
What is it like to work with Dr. Friedman and Oleg Kiselyov?
Dan is a close friend and mentor--working with him is always great fun. We argue constantly over language design--sometimes I'm even right.
Oleg is an emissary from some advanced alien race. Working with him is both exhausting and enlightening.
What are you working on next?
I'm a postdoc working for Andrew Lumsdaine in CREST, the Center for Research in Extreme Scale Technologies at Indiana University. My primary research is on the design, implementation, and application of declarative languages to High Performance Computing. I'm fortunate to be working with Eric Holk, Claire Alvis, Andy Lumsdaine, Arun Chauhan, and several other talented hackers and language designers at Indiana. We're currently working on two languages: Kanor, a message-passing language for programming cluster supercomputers, and Harlan, a declarative language for general purpose GPU programming. Of course both languages are implemented in Scheme. We hope to embed Harlan in the Scheme REPL; if we are successful, perhaps David Nolen can port our implementation to Clojure!
I'm also working with Dan and Claire on improving cKanren, our implementation of constraint logic programming. I can't wait to see what cool tricks David uses in his core.logic version of cKanren.
For fun I run 3D printing and Arduino clubs for undergrads, and am working with Nicole Jacquard, Andy Lumsdaine, and Kylie Peppler to start a Fab Lab at Indiana. Next semester I'll be teaching a course on Arduino, which will rock.
I'm also doing some fun research with Zach Sparks and Claire at Indiana, and Chris Martens at CMU, on using logic programming to write Interactive Fiction.
Finally, I'm working with some Indiana-trained macrologists on a new hygienic macro system for Scheme, and hopefully for Clojure as well. I hope to say much more about that soon.
We're delighted to announce our involvement in EuroClojure 2012, Europe's first Clojure conference, happening in London from May 24th to May 25th.
Clojure/core members Stuart Sierra and Luke VanderHart, authors of Practical Clojure, will be offering our 3-day Clojure training class between May 21st and May 23rd, right before the conference.
Much of the Clojure community is European, and we're excited to see Clojure conferences going global.
EuroClojure brings the number of yearly Clojure-specific conferences to three (the other two being Clojure/west and Clojure/conj, and gives Clojurists world-wide another opportunity to learn and share all things Clojure.
Registration for EuroClojure and training is open. We'll see you there.
We are happy to announce the availability of another six Clojure/conj 2011 videos.
Recordings from both of the 2010 and 2011 events are available, along with other Clojure videos, on Clojure's blip.tv page.
To see talks about Clojure in person, your next opportunity is Clojure/west, the Regular registration of which ends on February 17th!
Scheduled Talk Videos
(Neal's) Master Plan for Clojure Enterprise Mindshare Domination Neal Ford
Predicate Dispatch David Nolen
From Linear to Incremental Christophe Grand
Logs as Data Mark McGranaghan
Modeling the world probabilistically using Bayesian networks in Clojure Chas Emerick
Clojure and Android Daniel Solano Gómez
