CUFP 2009 Abstracts
Keynote: Real world Haskell
Scala at EDF Trading. Implementing a Domain-Specific Language for
Derivative Pricing with Scala
We have been using Scala within a large code base of 300k lines of Java on a on a business-critical, production system, where performance requirements are key. Based on this experience, the second part of the talk will discuss the suitability of Scala for use within this context, comparing the risks of using a niche language against its productivity benefits.
Erlang at hover.in
From our experiences at 'hover.in' I'd like to discuss why we chose using Erlang and got about using it as our bridge across our multi-node cluster. In particular the architectural decisions that went into making our distributed python crawler back-end running off Mnesia with its sharing & fragmentation strategies for tables that span several millions of rows, load-balancing to our 3-node Yaws web servers, tweaks to solve file descriptor & efficiency bottlenecks, experiments in DHT's, our cache worker implementations, our messaging queues, cron's & trade offs in dispatching jobs also throw light on design choices that can fit in distributed and heterogeneous environments. We have also recently built our own in-memory cache workers, persistent stats & logging system, and in the process of now building our own A/B testing framework, that we'd love to talk about.
After an initial quite 1 1/2 years of Erlang in production, we've just launched our developer blog which might give a closer insight of our work:
The Big Board: Teleconferencing over High-res Maps with Haskell
The Big Board aids in filling this technological gap by providing aerial photography, facility for dynamic, geo-referenced content creation, and a mechanism for simulcasting the content to everyone participating in the management of an emergency. The goal is that it should give emergency managers a reliable and flexible way to coordinate on their most natural medium (maps), be as easy to use as pen and paper, quick to train a new user on, and able to run in many environments, some quite technologically out of date.
In The Big Board, users open a map, join a virtual conference room, and then can collaboratively mark up the map with paths, polygons, and placemarks and add content behind these using a wiki that is generated as part of every conference room. As they mark up the map, their annotations are simulcasted to everyone else in the same conference room. Additionally, web-services serving geo-referenced content can be integrated onto the map to overlay things such as real-time vehicle location and sensor data.
This application has been written from the ground up in Haskell, with a PostGIS backend. Haskell was chosen for its laziness, reliability of applications written in it, rapid development, multicore scalability, and native compilation. In this presentation I will describe:
I will also give a demonstration of the application running, showing how it can be used to coordinate a disaster response.
The First Substantial Line of Business Application in F#
Measure: How well are my campaigns working? Are they meeting the goals?
Analyze: Why are the campaigns exceeding goals, or falling short of goals? What sub-segments are worthy of further communications with?
Design: Create communications that are relevant to each individual, driven by the data and insight gained from measurement and analysis.
Execute: Deliver the communications through the most appropriate channels, including email, print and mail, texting and purls.
Why did TFC management choose F#?
F# is a functional language with first class functions and composability, a pattern matching language and strongly typed. At the same time it as a .NET CLR language, fully interoperable with the vast libraries, and fully capable of OOP. So we believed that we would get the best of both worlds — a modern productive language and a full set of libraries to build on. The product is in the market and selling well, so we were rewarded by our decisions. Where was F# applied within the application, and where not?
Most of the business logic was implemented in F#, as was most of the data handling. C# was retained for building the ASP.NET user interface, just because the F# tool support is not there yet.
What were the development benefits?
The application was developed in less time with significantly less code — probably a quarter. The code is more readable and easier to adapt as new market requirements come in.
What were the business benefits?
We were able to get to market much sooner with more feature, and the overall cost of the development was significantly lower.
Conclusion: F# is an excellent language for line of business applications. It somewhat shortens the development cycle and greatly shortens the enhancement cycle.
Functional Programming at Facebook
Erlang has been the right tool for the job. It's well-known that Erlang is a good choice for communications applications and real-time chat in particular, and we'd like to mention that Erlang's strong support for concurrency, distribution, hot code loading, and online debugging have been indispensable, and that it's those same weaknesses that lead us away from C++ when first designing the service. However, we'd like to focus on the factors inside and outside Facebook that enabled us to choose Erlang in the first place. Most importantly, we needed good support for language-independent RPC. Fortunately we rely heavily on Thrift for all our services. Both the channel and Jabber servers need to speak to programs written in PHP and C++, and having infrastructure and engineers comfortable debugging it already in place when we began was invaluable. Our service management tools as well are language-independent; all our services regardless of language are controlled using the same interface. Facebook and the Thrift project are also fortunate to have an active community. The original Thrift binding for Erlang was implemented in-house but substantial portions were reimplemented by outside contributors. As well we've leveraged existing open source products including ejabberd and Bob Ippolito's Mochiweb framework.
The other half of our story consists of the barriers presented and risks taken in choosing Erlang. I mentioned above that much of Facebook's infrastructure is language-agnostic, but our build and deploy systems unfortunately are not. Many of our internal tools weren't written with Erlang or (in particular) hot code reloading in mind, so we needed to write many one-offs to support all the runtime goodness that Erlang affords us. Facebook runs many services, most of which are in C++, Python, or Java, and only two in Erlang, so in some respects we've needed to work harder to integrate. We at Facebook also tend to work in large codebases and share the responsibility of bug fixing. Unfortunately, our Erlang projects live outside of our main repository, and the extra learning necessary to develop or just fix bugs in Erlang tends to keep us isolated from the pool of talent in our department. Unsurprisingly, the job of fixing Chat bugs falls to one of a very small group, whereas bugs in our PHP code have a small army of PHP generalists waiting to squash them. However Erlang has such a steep learning curve only because most undergraduate programs don't stress functional programming, and Facebook in particular doesn't expose most engineers to functional programming, or even information about the strengths and weaknesses of FP versus PHP, C++, and Python. In particular we'd like to touch on the prevalence of object-oriented programming in education and in practice, and the tendency of engineers (even engineers with FP experience) to guide their design by OOP principles even when they don't fit. The recurring theme is "the right tool for the job" — actor-based, functional, imperative, and object-oriented programming are all valuable tools for modeling particular problems, but the most important lessons in our experience are how to lower the barriers to choosing the right tool, and how to arm engineers with the knowledge they need to make an appropriate choice.
FMD - Functional development in Excel
Excel combines two programming models
The functional model allows very rapid development and great transparency, but the limitations of Excel's built-in functions drives developers to use VBA. Soon the spreadsheet dependency graph is lost and the spreadsheet layer is relegated to a GUI on top of tens/hundreds of thousands of lines of VBA. The logic is tightly tied to Excel, and a server-side implementation involves a complete rewrite in another language, which carries both operational risk and developmental cost.
FMD (‘Functional Model Deployment’) prevents these problems by embedding a functional language cleanly into Excel, effectively extending Excel to be a higher order functional environment. Now complex models can be developed without leaving the pure spreadsheet domain:
The business have fully supported this approach, and are enthusiastic about using FMD - as Simon Peyton Jones identified elsewhere, “Excel is the world's most popular functional language”. From their point of view, functional programming in Excel is just an extension of what they've been doing for years!
Building the User Programmable Internet with Erlang
The most popular programming language in the world is a functional one — the humble spreadsheet. The spreadsheet programming paradigm remains tied to the ‘document’ model due to its firm desktop roots.
Hypernumbers are developing a ‘spreadsheet-like’ programming language and platform for the web that will enable non-technical end-users to build dynamic integrated web-applications. The hypernumbers platform is itself implemented in a functional programming language — Erlang.
The platform is currently in private beta testing with selected users and potential customers.
Clear & Simple: Composing a Marketplace
First, easy-to-use often means complex-to-build, but FP's high-level abstractions and referential transparency help us to manage this complexity. Second, as we handle people's money, mistakes could be damaging, but the expressiveness of FP give us confidence in our code, and a lack of side effects facilitates testing. Third, a responsive site is a pleasure to use, so we write parallelizable logic, which FP's avoidance of side effects makes feasible.
This talk will describe Glyde's use of JoCaml and Scala and discuss our experiences with these tools.
Birth of the Industrial Haskell Group
At last year's CUFP, users of several FP languages raised the issue that there was no organisation that they could pay to do the important but boring work of maintaining and improving common infrastructure. Shortly after CUFP, in partnership with major commercial users of Haskell such as Galois and Amgen, we started to set wheels in motion, and in March 2009 we announced the birth of the Industrial Haskell Group (IHG).
The IHG is starting off with a limited set of activities, but already it is having an impact on the state of the Haskell development platform. We expect that as it expands, it will become a significant force driving Haskell forwards.
In this presentation we will talk about the motivation leading to the formation of the IHG, how it has worked thus far and what lessons we can learn that might benefit other FP communities. We will also look at how we can encourage the positive feedback cycle between commercial users and the wider community.
26 June 2009