Clojure as First Language

Tags

, , ,

At the time of writing, I’m organising a ClojureBridge workshop.  ClojureBridge targets underrepresented groups who are either brand new to programming or only have limited experience.  While preparing for the workshop, I struggled to find quality info in the public domain about whether Clojure is a suitable first programming language.  There are lots of introductions to Clojure written from the assumption that the reader/viewer has previous programming experience, but very little substance around choosing Clojure (or any other language for that matter) as your first language.  So this is my attempt to answer the question:

“Is Clojure a good choice as a first programming language?”

Criteria

The get-out-of-jail answer to any question like this is “it depends”, so I’m going to list some key criteria that I think are important for someone choosing their first programming language, even if the significance of some of these criteria might not be immediately apparent.  The criteria I settled on are:

  1. TTHW (Time To Hello World)
  2. Community Support
  3. Funability
  4. Industry relevance

I’ll step through each of these criteria and give a subjective “grade” (i.e., “A” through to “F”)  on how well Clojure meets these criteria, as well has listing other language choices that perform particularly well or poorly on each criteria.

1. TTHW (Time To Hello World)

There is an age-old tradition where programs that show the text “Hello World” on the screen are the first ones you see/construct in any new language.  The essence of “Hello World” programs is that they are simple, quick to write and with very little that can go wrong with them.  “Hello World” programs also provide unambiguous tangible feedback that they are working (or not).  This criteria recognises how important it is for a first language to support novice developers by allowing them to write and run programs as quickly as possible.

With the only prerequisite that you have a computer with a working internet connection and browser, languages that perform well on this criteria allow you to code without the need to install or configure any additional software.  On the other end of the spectrum are languages which require installation and configuration of significant amounts of non-standard software.

Hello World in the Browser

TryClojure, 4Clojure and Repl.it are three online Clojure environments (amongst others) that let you use only your browser to experiment with the Clojure language.  It’s important to note that while these tools are great for the early days of experimenting with a language, you will fairly quickly want/need to write code on your own machine which requires the installation of additional software, and Clojure does have a higher hurdle to clear than some other languages in this respect.

Clojure Grade: A (lots of options, very little friction)

Hello World on your Computer

Clojure runs on top of the Java programming language.  Learning Clojure doesn’t require you to learn Java, but Java must be installed.  Many operating systems (e.g., Windows, OSX on Apple computers) come with a version of Java pre-installed, but in some cases Java wont be installed or be the wrong version for Clojure, so installing Java will be required as well.

Once Java is installed, installing Clojure usually requires that you install Leiningen (another piece of software you use to construct Clojure programs) which will allow you to download and install Clojure easily.  Installing Leiningen requires some familiarity with the command line tools on your Operating System, which can complicate things if you have not used these tools before.

A final step in this process is to install some form of text editor or IDE to write your Clojure programs.  Your computer will come installed with a basic editor already, but it will not be especially useful for Clojure programming in it’s default state so options like Sublime Text, Atom or Cursive are popular choices to quickly provide a beginner developer a decent development environment.

Clojure Grade: C (compared to languages like Ruby and Python which are often pre-installed on most Apple and Linux computers).

2. Support

This criteria could have been renamed “Time to First StackOverflow Answer” as it relates to how much support the language community will/can provide to beginner developers.  This support could come from a variety of channels; official documentation, tool support, online communities like StackOverflow, local user groups, conferences, books, etc.

From a beginner’s perspective, it’s important to understand that languages come with their own unique culture which often flows from their creators and high profile advocates.  My experience been that community support peaks in the early years of a language’s popularity where the “size” of the language is small enough to keep the active members of the community cohesive and passionate about it’s growth.  As languages mature and their reach into the software development world both broadens and slows, the pioneering spirit of the community will most likely dissipate as the need to support programmers of various levels of experience exceeds the ability/desire of the core community.  The upside of this phase of a language’s evolution is that all the beginner problems have been solved many, many times over and Google will answer the vast majority of your questions fairly effectively.

Clojure is a relatively mature language that is attracting the “Early Majority” in the adoption life cycle.  That said, Clojure is a language from the functional programming family of languages and the mathematical and academic background to this group can flow into the online and physical communities (e.g., user groups, conferences) which can portray Clojure as less accessible than necessary.

Clojure Grade: B- (minus marks for some of the pious FP attitude that leaks from the community)

3. Funability

Of all the largely subjective nature of these criteria, this one takes the cake!  How does one measure how much fun it is to program in a language?  But for a beginner programmer, using your first language must generate positive vibes or you’ll quickly lose interest.

Under the premise that a sense of achievement (i.e., accomplishing what I set out to do) will generate feelings of satisfaction, I’m going to use a proxy metric of “how much does the language get in my way of doing what I want” to measure how much fun Clojure is, and I think Clojure has a good story to tell in this sense:

(Note: some of what follows is might not be immediately obvious/understandable to completely novice developers)

  • Dynamic typing – not needing to worry (much) about types increases the speed with which beginner programmers can produce working code.
  • The REPL – having an interactive environment to experiment in not only overcomes the downside of Clojure’s Java dependence but provides instantaneous feedback for a development cohort that have no past experience to give them confidence in their actions.
  • Concise syntax – less syntactic breadth means less mental energy needed when coding.  There’s still the APIs to grok, but almost everything in Clojure code is a function call with a very small set of reserved terms in the language.

And on the negative side…

  • Java/Clojure Stacktraces – it wont be long before every new Clojure developer is presented with their first stacktrace… and even as a seasoned Java developer, I still recoil at the sight of some of these.
  • Prefix notation – this might be a vastly polarising topic, but I suspect a group of people will never get comfortable with prefix notation and the sea of parenthesis that are so commonly associated with Clojure and it’s siblings in the Lisp family.

Clojure Grade: B- (with Ruby as possibly the most “fun” language I’m aware of with all the benefits of dynamic typing and a REPL, but without the pain of reading Clojure-like stacktraces and a more “natural” infix notation).

4. Industry relevance

Of course, many people start coding as part of a career change, so choosing a first language that puts you somewhere along a path to said career might factor very highly in importance.  Breaking into the software development market with your only experience being in “niche isolated language X” is probably not going to stand out on your fledgling CV as “highly popular, widely used language Y”.

It’s hard to deny that the “best” language if you wish to optimise for (broard) industry relevance above everything else is JavaScript.  Ubiquitous on the browser and with considerable popularity on the server via Node.js, Javascript (considerable warts and all) is the ideal choice if you want to bootstrap yourself into a development career as quickly as possible.

Clojure has nowhere near the market presence as Javascript (or Ruby or Python for that matter), but it does have success stories at some fairly high profile brands.  It also has a browser version of the language – ClojureScript – which can give you exposure to front-end web development with little extra investment.

However the biggest advantage Clojure has in industry relevance is the Java connection.  Java is an very mature and widely adopted language that is deeply integrated into the IT systems of tens of thousands of companies.  The JVM (platform on which both Java and Clojure run) will be a permanent part of the global IT landscape for decades to come, and more and more companies with existing experience in the JVM (via Java) are looking to broaden this experience with other JVM languages (like Clojure).

Clojure Grade: B (Javascript is an A+ on this scale.  Clojure looses marks for relative lack of popularity, but gains some back via ClojureScript and the Java connection.)

Conclusion

“Is Clojure a good choice as a first programming language?”

Coming back to the criteria, I judged Clojure as follows:

  • Time To Hello World – A (on browser) and C (on computer)
  • Community Support – B minus
  • Funability – B minus
  • Industry relevance – B

Overall, that’s not a bad report card and while Clojure doesn’t stack up as the obvious choice for a first programming language, it’s not a bad choice by any means.

Individuals may place different weights against these criteria, but if you keep in mind that your first programming language shouldn’t also be your last one (i.e., you can/should learn plenty more over time), there’s very little downside to making Clojure the first one.

Clojure gives you a good first programming experience with little overhead, lots of support avenues and good options to turn your programming hobby into the beginnings of a programming career.

Advertisements

5 things you’ll do as a graduate developer

Background

Universities don’t have the time or scale to give undergraduate software engineering students a complete sense of what it’s like to work in industry, which means there can be some eye opening moments for many graduates once they leave tertiary education and become an IT professional.  I’ve prepared a list below containing some of the big ticket items that might surprise many a graduate upon entering the industry, but will be commonplace to seasoned engineers.

This list is not meant to be definitive or exhaustive, but it does capture key moments that many graduate developers will go through, depending on where they land their first development job.

So here’s the list:

  1. Release code you aren’t proud of
  2. Not coding
  3. Work with someone more experienced than yourself
  4. Work to someone else’s deadline
  5. Work long hours

For each item, I’ve included examples of (likely?) reactions from graduates, trying to cover both ends of the spectrum of reasonableness.  I’ve also included links to seminal text books which provide a lot of extra knowledge around the item in case anyone reads anymore.

1. Release code you aren’t proud of

Firstly, a basic home truth: given enough time, 100% of your code will become something you no longer take pride in 😦 With time comes experience, (hopefully) skill and exposure to alternate ways to write code and all these changes mean you look back at the code you’ve written and see how differently you would write it now.  In extreme cases, you struggle to even associate any of your old code with how you see yourself as a developer now.

But it’s not the above situation that this item is about – it’s when you release code that you know at the time is not your best work, but external factors prevent you from fixing/improving the code.  The typical scenario is project deadlines forcing you to move onto the next piece of work as soon as you have the most basic working version of a solution to your current piece of work.

Good Reaction

Even as a graduate developer, you’ll be starting to build a spidey sense that tells you whether the code you’ve written is ugly-but-perfectly-workable versus ugly-and-potentially-buggy.  Where the code falls on this spectrum should dictate how much energy you use to push back against the forces preventing you from working further on the code.  It also helps to have an estimate of how much time you need to bring the code up to a level you believe is truly production-ready.  Sense checking this feeling with a more experienced developer is critical at this point, lest you are being overly optimistic or pessimistic with your estimate.

The other side of the equation is understanding just what is behind the opposing forces: How tight is the deadline?  Are the consequences of missing the deadline too high?  Is the importance of your code high or low relative to these forces?

Poor Reaction #1

Say nothing, do nothing.  Accept this as a fact of life and move on.

Poor Reaction #2

Use only your own counsel to gauge whether your code is ready for release.  Refuse to budge even an inch until you have achieved code which meets your standards.

Further Reading

Refactoring (Martin Fowler) – For theory and practical advice on how to incrementally improve the quality of the code you’ve written

2. Not Coding

“Not coding” can mean lots of things, but it will most likely be related tasks like testing, writing documentation or working with customer support or operations teams.  You will be often be assigned these activities as part of a larger on-boarding process, and the rationale is usually along the lines of:

  • Increasing your level of knowledge around the business environment in which the code runs
  • Giving you a softer introduction to your role than throwing you in the deep end immediately
  • Buying some time during a critical activity for other people to become free to support you

Good Reaction

If you’re not coding in the early stages of your career, it is key to understand the reasons for this (hopefully they align with the above) and how long this stage will last.  Make the most of the information you collect during this stage and the people you’ll meet.  Use all this context when you finally sit down to solve their problems.

Poor Reaction

“I’m a coder.  I code.  Testers test.  Technical writers write.  And ne’er the twain shall meet.”

Further Reading

The Pragmatic Programmer (Andy Hunt and Dave Thomas) – so many gems of wisdom for programmers of all ages, but includes general career advice as well

3. Work with someone more experienced than yourself

This can come in several forms; some companies have explicit mentor arrangement for junior developers, in other cases, it could be the natural side effect of coming into a team with more experienced developers, or it could be a more disciplined team culture of pair programming that finds you spending a lot of time developing with senior engineers.

Good Reaction

Whether this is a deliberate master/apprentice coupling or just opportunistic, there are so many benefits to working in this scenario.  Senior developers understand the business domain, codebase and technical stack far better than you and will know where the gotchas lie.  They will know the architectural and design patterns intuitively: things that can take a long time to sink in.  They will also (likely) know the key people around the work which will make navigating the informal communication channels far easier.

Having access to a good mentor(s) can have a significant impact on how quickly a graduate developer picks up speed in their career.  Having an open mind and a willingness to listen is vital.  Be careful not to blindly accept everything you hear/see though – working with more experienced people doesn’t mean you turn your brain off and take no responsibility for the work being done.  Good mentors should welcome questions from enthusiastic junior devs.

Poor Reaction

“I’ve got this – hold my beer (after all, I did get great grades in my software subjects at university/college).”

Further Reading

Pair Programming Illuminated (Laurie Williams) – Laurie was one of the first people to do empirical studies on the effectiveness of pair programming.

Image result for pair programming illuminated

4. Work to someone else’s deadline

It’s very, very rare (i.e., unicorn rare) to find a development project with completely elastic deadlines.  Deadlines are derived from pressures; market pressures (e.g., what is the market demanding, what is the competition doing, when is the next big event we’re targeting), budget pressures (e.g., we have to get this done for less than $X), timeline pressures (e.g., end of financial/tax year, in time for the System X upgrade in Q3).

Whatever the pressures are, some poor sucker derives a schedule that will clearly show that the work will get done by the required deadline and then the starter’s gun fires: it’s now down to your team to make the plan a reality (note: massive oversimplification here).  And whatever the pressures are, the presence of a deadline and our general inability to predict what course a development project will follow inevitably means time allocated to work will fall short of the time needed to do the work… and how people react to this reality is often the difference between success and failure, at a individual level, team level and so on up to the overall success (or failure) of the program or even organisation.

Good Reaction

“Is there anything I can see in the work I’m doing that could be done simpler, cheaper, quicker?  Is all the functionality needed for day 1, or can some wait until a later release?  Am I spinning my wheels by struggling to finish my work solo for the benefit of my own knowledge, or am I asking for help regularly, even if it means I don’t understand 100% of what I’ve done?”

Is this deadline affecting everyone involved, or is it really just the development team?

Poor Reaction

“They cannot possibly expect me to finish this work in 2 weeks! We originally estimated 6 weeks and none of the scope has changed.  And what about the 2 month delay to the start of the work?!? Why didn’t that change the deadline?  Even as a graduate, I can see the lunatics are running the asylum!!!”

Further Reading

Software Estimation (Steve McConnell) – sheds some light on the risks around estimating software and will help make you a better estimator as a consequence (p.s., you’ll still be wrong most of the time though :-))

Image result for software estimation

5. Work long hours

Given the likelihood you’ll be working to a tight deadline, a lazy but very common reaction from those ultimately responsible for the delivery of the work (most likely a project or program manager) is to ask the team to work extra hours to meet the deadline.  This request could come in the form of longer days, or weekends, or a block on taking leave, or any combination of these.  For all the enlightened conversations around the IT industry, when push comes to shove, we invariably fall back to cracking the whip to meet a deadline with very little in the way of compensation for our efforts.

But unpleasant though these situations may be, it’s important for graduate developers to experience these environments at least once for there are many lessons to be learnt from these situations.

Good Reaction

Have a good understanding of how much time you’re willing to give for the cause and how critical this time of yours will be to meet the deadline.  There’s no point if you work 80 hours weeks on work that is nowhere near the critical path of a project.

My experience has shown me that short bursts of long hours can help considerably in these situations.  By “short bursts”, I mean effort measured in days and possibly weeks, but definitely not months.

Poor Reaction 1

“My contract says I work 37.5 hours a week.  I’ll see you at 9:00am tomorrow.”  To me, this is a sign that your treating this as a job rather than a career.  Being successful in any career requires you to undertake some personal investment and being completely unwilling to budge is not a path to enlightenment.

Poor Reaction 2

“What day is it?  I’ve been sitting here coding for so long I cannot really see the screen anymore, but I’m sure another 3-4 hours is all I need.   And where’s that pallet of Red Bull I ordered?”

Further Reading

Death March (Ed Yourdon) – sometimes working long hours is taken to the extreme, and these situations have a suitably

The Mythical Man Month (Fred Brooks) – this book isn’t guaranteed to help you fight the battle over long hours, but will equip you to understand how the situation may have arisen.

Mythical man-month (book cover).jpg

 

Study of speaker gender diversity at Australian Tech Conferences

Tags

, , , ,

As a generalist software geek in Australia/NZ, there is not a lot of choice when it comes to conferences. One advantage of the limited menu we have is that it’s relatively easy to track trends of the available conferences over time and one such trend I’m particularly interested in is how many female speakers get an opportunity to present at these conferences.

Below is the information I’ve gathered so far in all it’s empirical glory.

agile-oz-cropped


yow-west-cropped


lambda-jam-cropped


webstock-cropped


last-conf-cropped.png


yow-cropped.png

I’ve focused on conferences where I could get at least three years of data (which is sometimes not as easy as I expected, and I thank conference organisers who archive appropriately) to see how their gender diversity numbers were trending.  Some initial conclusions are:

  1. Webstock in Wellington is a clear winner in terms of this metric.  They were the only conference in this list which had > 50% female speakers at any time (2016 and 2017).  [Personal note: I’ve not attended this conference but colleagues who have speak very highly of it as well]
  2. All conferences included seem to be trending in the right direction apart from YOW West and YOW Lambda Jam, which either have no discernible trend due to small sample size (YOW Lambda Jam) or where diversity is not improving (YOW West).
  3. Based on this data set, the bar is very low in terms of improving.  With the exception of Webstock, diversity ratios of 1:3 or 1:4 is the best I’ve found 😦

Note: Gender presence is only one form of diversity to consider and the approach I’ve used is unscientific and assuming a gender binary world, but I hope you find this information useful even with the shortcomings in my method.  I plan to release this data in a more machine consumable format (e.g., JSON) in the near future.

Full Disclosure: My employer, ThoughtWorks, sponsors both Agile Australia and the YOW conferences and I’ve spoken at several of these events over the years.

Adopting Functional Programming Languages (Part 7)

(Note: This is the seventh and final part of a whitepaper I wrote a couple of years ago which I never had a chance to publish in full.  You’ll find the earlier parts here, herehere, here, here and here.)

Conclusion

Functional languages have been growing for over 50 years, mainly in academia and niche applications. However, recently many have evolved into viable choices for general-purpose software development. At the same time the software industry has increased its need for the benefits these languages provide:

  • The increasing complexity of software has brought focus to the benefits of solutions built around the strengths of functional languages.
  • The need to build robust systems running reliably in concurrent environments has made immutability a de facto requirement of these solutions.
  • Data processing intensive requirements (e.g., analytics) are a natural domain for functional languages with their focus on composability and readiness for parallel execution.
  • The easy integration of Scala, Clojure and F# with existing code running on the JVM and CLR has lowered the cost of adopting these languages in a polyglot environment.
  • The ability of these languages to attract high performing software developers makes their adoption a sound move for organisations looking to increase the productivity of their development teams.
  • Software architecture approaches focusing on service composition provide low risk opportunities to experiment with new languages alongside existing codebases.

Adopting Functional Programming Languages (Part 6)

(Note: This is the sixth part of a whitepaper I wrote a couple of years ago which I never had a chance to publish in full.  You’ll find the earlier parts here, herehere, here and here.)

Adoption

We will discuss language selection approaches in the Language selection section but beyond that decision is the pivotal question of where to first trial a functional language within your organisation.

As a general rule, start small. Starting small helps minimize the risk of the adoption. Starting small usually means working on a small piece of software or a small part of a larger piece of software. For many organisations, this is easier said than done. Although there are prevailing trends towards building large solutions as a collection of smaller independent services, many solutions are still built and deployed from a single monolithic codebase. Typically this codebase is written in a single language and executed by a single homogenous process. In this type of environment, choosing to adopt a new language for a piece of the solution presents these choices:

  • Extract some existing functionality into a separate module and re-implement it in the new language. In this context “separate module” needs to be independent from the main codebase at least in terms of development language. On the JVM, a Clojure or Scala module can be developed independently and then deployed as part of the larger single application if needed. In practice, building this new module as a separately deployable service will provide additional benefits to your architecture.
  • Create a separate module for a new piece of functionality and implement it in the new language. This is applying the above thinking to new functionality rather than existing logic.
  • Look for an area of non-production code where adoption experiments can be carried out in an isolated manner. Developer or tester tooling or in-house monitoring tools are all potential places where new languages can be applied.

Not surprisingly, greenfield development provides the most opportunities to choose new languages for all or part of the solution. In the case of a recent experience we had with a client in the US (http://bit.ly/1fYwbZQ), Clojure was used to build a custom CMS forming part of a larger solution.

Polyglot Programming

Adopting a functional language alongside more traditional enterprise languages is embracing polyglot programming – the selective application of several different programming languages to harness their respective strengths (http://bit.ly/1joh7rS). Even when web applications are developed with a single primary language, you will usually find significant amounts of other languages like SQL and JavaScript. Polyglot programming goes a step beyond that and suggests that you consider options such as a highly productive web framework like Ruby on Rails that dispatches to backend web services written in a combination of Java, Erlang (where the Actor model can assist in building robust distributed subsystems) and Haskell (for some heavy data transformation work) where appropriate.

Irrespective of whether you have a driving need or desire to adopt functional languages within your organisation, making the changes necessary to allow this sort of adoption is beneficial for other reasons, best expressed by this quote: “The recurring theme is ‘the right tool for the job’ … the most important lessons in our experience are how to lower the barriers to choosing the right tool” (http://bit.ly/NsF0iR)

People/recruiting

Most experienced IT people admit (sometimes reluctantly) that choice of technology has a relatively low level of influence on the success or failure of a project or product. These people do generally agree that focusing on softer issues like team capability, culture, customer involvement and effective project management are far more likely to influence the outcome. However, there is a very important connection between programming language choice and these key success factors.

Organisations strive to build teams that harness the latent power of technology to provide innovative solutions to their customers. Building these teams requires people with a strong passion in technology and desire to work in productive environments. Harnessing the surge in interest in functional programming can help position your organisation as a destination employer for these types of developer. This will provide short-term benefits through the application of functional programming as well as longer-term advantages of building highly competent development teams.

Indeed, a growing number of developers are investing more and more of their own time in learning and honing their functional skills. These developers see the benefits in this style of development and appreciate how better tools help them serve their employers.

This is the same type of thinking that led to some developers seeing the benefit of a simple, new C-like language without the overhead of explicit memory management in the mid-late 1990s. This was the first wave of developer to adopt Java.

This is the same type of thinking that led to some developers seeing the benefit dynamic languages brought in terms of expressiveness and development speed over their statically typed cousins and surfed the Ruby/Ruby on Rails wave from the mid 2000s.

This is the same type of thinking that will lead to some developers becoming early adopters of the next waves of software development languages and approaches in the years to come.

These types of developer keep a keen eye on what is evolving on the horizon of their industry. They spend considerable (personal) time researching the topics that interest them. They are likely to be well connected into a network of similarly minded people, either through community user groups or common employers. They often move in groups from destination employer to destination employer as they spot environments that value their presence. They are likely to be energetic contributors to open source software and/or have active GitHub accounts.

In short, these types of developer are highly sought after hires for an employer. It is true that there will be a smaller pool to choose from than for Java or C# developers, but the quality of developer will be much higher.

Language selection

If your development group is keen to trial functional programming, there are two broad approaches they can take in the selection of the language. These choices are similar to popular models for spoken language learning in use the world over.

Approach A: Dipping your toes in

Approach B: Immersion

For an existing enterprise development group, an obvious decision might be to stay aligned with the language platform you are currently using and pick a functional language which fits neatly into that platform (e.g., Scala for a Java shop, F# for a Microsoft shop).

There can be downsides to this approach with Scala as it’s very easy for experienced Java developers to continue writing Java code with the occasional Scala idiom for good measure. The easy blend between core Java syntax and Scala’s myriad extensions can actually increase the difficulty in unlearning many of the lessons of an imperative object-oriented background and fully embrace functional thinking.

In this world, you deliberately choose a language that is syntactically separate from any the current development team has used. The alien syntax is a constant reminder that the mental approach needed for this language is different. Examples of languages that could be used in this case include any of the LISP variants (with Clojure being a good choice for Java environments), Haskell or Erlang.

Each approach has its own merits. Approach A will provide the most immediate benefit in terms of adoption with a longer-term risk that the full power of a functional approach may never be reached because of the ever-present temptation to fall back on old habits.   Approach B will include a steeper initial learning curve but should result in developers that have a clearer and deeper understanding of functional programming.

Evaluation

When adopting a new language, it also makes sense to think about how the adoption should be evaluated.

An excellent presentation discussing the issues surrounding language choice is “Nobody Ever Got Fired for Picking Java” (http://bit.ly/OAkv4O) by Alex Payne and he spends a lot of time discussing the difficulty in choosing objective criteria on which to fairly evaluate a number of potential languages. Alex concludes by recommending using a weighted decision matrix to capture the criteria used for the evaluation.

In 2013, ThoughtWorks used a number of criteria to help with language selection with an Australian-based client in the wealth management industry (see http://bit.ly/1gz2c66 for a presentation on this evaluation). The team spent three weeks doing cross-evaluation of Java, Scala and Clojure, taking regular measurements of the three languages using a combination of objective and subjective criteria. See below for a set of graphs showing one of the objective criteria being measured; the amount of time spent in minutes (y-axis) between production code, test code, tool management and library management per day (x-axis).

fp-whitepaper-img-4

The team also used weightings to balance out seven criteria between short-term (i.e., project) and long-term horizons. These criteria appear on the left hand side of the tables below. On the right hand side are subjective evaluations by the team members on how well they thought Clojure (the selected language) would satisfy each of those criteria. Note that the “Ease of maintenance” scores were based on a “throw it over the wall” support model – the team decided to mitigate this risk by rotating the support developers through the development team on a regular basis.

fp-whitepaper-img-5

This approach is by no means the definitive way to evaluate language selection, but it does highlight some areas you may want to consider. Using weighted criteria during evaluation will also help people involved in the evaluation decide on the critical aspects of the experiment.

Adopting Functional Programming Languages (Part 5)

(Note: This is the fifth part of a whitepaper I wrote a couple of years ago which I never had a chance to publish in full.  You’ll find the earlier parts here, herehere and here.)

Benefits

The main benefits of using functional languages stem from their natural expressiveness and the downstream advantages derived from immutability. Languages that are naturally expressive and favour immutability help fight the accidental complexity that inevitably creeps into large codebases.   These benefits also increase as codebases mature and grow.

Expressivity

Naturally expressive languages allow developers to write less code. Less code is beneficial[1] because:

  • There is less code to understand
  • There is less code to maintain
  • There is less code to test
  • There are fewer places for defects to occur

RedMonk used quantitative analysis of updates to open source code repositories on GitHub to rank the expressiveness of languages (http://bit.ly/1gENQ8y). The theory behind this study is that more expressive languages will result in smaller, more isolated changes for each update, leading to less lines of code being changed (e.g., smaller deltas on average) on each update.

See the charts below for our summary of the Redmonk results comparing the various families of language against the number of times they appear in the most expressive 50% of the results or the least expressive 50%. Screenshot 2016-04-21 19.56.47

Screenshot 2016-04-21 19.56.59

Functional languages rank highly on the highly expressive end of the axis; Clojure (#7), Haskell (#10), Racket (#11), Dylan (#12), Emacs Lisp (#14), R (#17), Scala (#18), OCaml (#20), F# (#21), Erlang (#22) and Common Lisp (#23) consume almost half of the top 23 spots on the graph, whereas Java (#44) and C# (#45) are towards the end of the 52-language list. Whilst the conclusions drawn are subject to lots of uncontrolled variables, the results do correlate with the prevailing impression that functional languages naturally produce more expressive code than their object-oriented equivalents.

Immutability

For many developers, immutability becomes an end unto itself, and the core reasons for the benefit are often forgotten. We mentioned previously that programs with immutable data are inherently easier to reason about because time is no longer a variable that can impact the state of the system. And immutable data structures have another huge advantage that is becoming more and more important with enterprise software:

Immutable objects are naturally thread safe

Immutable objects/data structures provide a consistent view to the outside world at all times after construction. Under this context, multiple threads will see the same version of these objects at all times and avoid classic multi-threaded problems like race conditions and deadlocks that are notoriously difficult to reproduce and debug.

Thread safety is becoming increasingly important as CPUs are built around more, comparatively low powered cores. Software must be capable of running in parallel across multiple cores to take advantage of these architectures.

Counter-intuitively, immutable data structures are usually quite performant, although this was not always the case. When new data structures are created, they share as much structure as they can with the source data structure, rather than automatically taking a separate copy of the source data.

Object creation and garbage collection are not “free” operations so if memory consumption and runtime performance are absolutely critical non-functional requirements, you might want to look at languages that give you more control over these things (e.g., C or C++).   However, the need for this optimisation should be proven rather than anticipated as the benefits of low-level control over memory management are usually outweighed by the ongoing cost of living with code written in such languages. This sentiment is embodied by the well-worn approach of “make it work, make it right and, finally, make it fast”.

Using immutability with OO languages

The value of immutability can also be seen in the amount of effort expended in applying it with languages that don’t typically encourage it. For many years, software engineers have realized the benefits immutable data structures bring in terms of reducing complexity in applications. Therefore, many people have codified practices to design immutable solutions in non-functional languages.

Indeed, the fundamental notion of encapsulation (one of the main goals of good object-oriented design) recognizes the need for a class to provide strict control over who has access to its state and how that state is changed. Although some of the theory of encapsulation is focused on who can read the state of a class, much of the benefit comes from narrowing who can update that state. In immutable systems, good encapsulation is still desirable, but less so as updates to this state are not allowed.

Many developers now favour building immutable classes within their OO language of choice. This approach results in classes that contain only property accessors (i.e., getters) and not mutators (i.e., setters). With this pattern, classes are initialized only during construction and provide no public access to change their state beyond construction. This practice does not prevent internal mutation from occurring, but developers can apply discipline to return new instances of objects when they have been mutated internally. For example, in an Account class, you can write a method to cancel the existing account:

     public void cancel(CancellationReason reason);

or you could write a version which cancels and returns a newly-created instance of Account:

     public Account cancel(CancellationReason reason);

Unfortunately, when the language does not enforce immutability, the discipline falls to the developers to determine when and how the Account has been changed. It would be easy for a developer to ignore the return value and continue to work with the existing Account object. Conversely, in a functional world, developers naturally deal with immutable structures because that approach is inline with the principles of the language: working against these principles generates friction between the language and the developers.

[1] Be careful of fixating on a single metric because of the unwanted behaviours this fixation could produce.

Adopting Functional Programming Languages (Part 4)

(Note: This is the fourth part of a whitepaper I wrote a couple of years ago which I never had a chance to publish in full.  You’ll find the earlier parts here, here and here.)

Resurgence

The renewed interest in functional programming has been driven by three main factors:

  • A number of successful high profile organisations making significant investments in these languages
  • The efforts by language designers to lower the barrier to migration from existing languages and platforms
  • The incorporation of functional aspects into non-functional languages

High profile success stories

Being able to cite other organisations that have trodden a similar path always helps language adoption efforts. And when those other organisations are high profile, technically innovative and speak publicly about their technology choices, they become an even stronger role model for other organisations considering the same adoption.

Scala @ LinkedIn

LinkedIn is one of the high profile companies that have gone public with their implementation of Scala. From usage of Scala at LinkedIn, Norbert (http://linkd.in/1dbZsB8) was produced. Norbert provides a framework for distributing simple client/server architectures over a clustered environment to produce highly scalable and fault tolerant solutions for high load environments.

Seamless Java integration, easy concurrency through the Actor model and support for code reuse are listed as being the main reasons for the adoption (http://bit.ly/1gENnmY).

Scala @ Twitter

Twitter uses Scala for much of its infrastructure services (with Ruby/Ruby on Rails for the front-end) including queuing, the social graph store, people search functionality and streaming API (http://bit.ly/1cXIXrW). Beyond the Java integration and code reuse benefits, Twitter highlighted ready access to the language developers and “fast, fun and good for long-running processes” as major reasons for adopting Scala. Furthermore, Twitter has re-invested in the Scala community by releasing numerous libraries it has created as open source software (https://github.com/twitter).

Erlang @ Facebook

Facebook uses Erlang for its Facebook Chat feature and the nature of these services play to the strengths of Erlang’s strong concurrency model. Needing to support upwards of 800 million messages per day across 7 million chat channels, Erlang has served Facebook well in providing the basis for a scalable reliable set of services, upon which the JavaScript and PHP front-end is based.

Haskell @ Finance

The Haskell language is a popular choice amongst corporate banking and trading houses with its usage within ABN AMRO, Bank of America, Barclays, Credit Suisse, Deutschebank, Standard Chartered Bank, as well as non-financial industry companies like Google and Intel (http://bit.ly/1fGjqOJ). The problem domains common to these industries lend themselves to solutions expressed in terms of mathematical functions, making them ideally suited to functional languages.

Clojure @ Netflix

After over a year of investigation, Netflix engineers started investing in Clojure (https://speakerdeck.com/daveray/clojure-at-netflix) for both internal and production services. Netflix’s choice was driven primarily by the abstractions Clojure provides as well as the interactive development model available to Clojure developers via the REPL (Read Evaluate Print Loop) utility. The Netflix story shows a progressive increase in investment in terms of where they introduced Clojure: as their confidence grew, so too did the importance and size of the code that was written in Clojure.

Easy migration paths

LinkedIn and Twitter both declared that Scala’s tight Java integration is an important reason for its selection. With Scala, developers can easily drop down into the Java language to call upon either core Java libraries or 3rd party code that already provides valuable functionality.   Beyond ease of integration at the language level, running on top of the Java Virtual Machine (JVM) gives Scala and Clojure developers the benefit of the thousands of person years invested in developing and tuning the JVM. The same benefits are available to programmers using F# on the Microsoft Common Language Runtime (CLR).

Another common attribute of the organisations listed above is their ability to adopt these languages selectively in small areas due to the nature of their architecture. By composing applications of many independently buildable and deployable services, these organisations can choose a range of different implementation languages and need standardize only on the communication mechanism between the services (e.g. Apache Thrift for Facebook and Twitter).

Functional aspects in non functional languages

Also lowering the barrier to entry to functional programming is the prominence of functional concepts in traditionally non-functional languages. Being able to program with higher order functions when working within another dominant paradigm can help soften the blow for developers who are worried about leaping completely into a functional language.

The introduction of lambda expressions to Java 8 (http://bit.ly/1kGXMzg) is the most recent sign of functional programming influencing the evolution of imperative/OO languages, and C# developers have had access to lambda expressions and higher order functions since 2007. The lingua franca of the web, JavaScript, has supported higher order functions since its creation. Many dynamic languages like Ruby allow blocks to be passed to, and returned from, other functions – emulating the behavior of higher order functions found in many functional languages. With blocks, standard list comprehensions (functions that take lists as parameters and return variations on those lists) like map and reduce can be easily implemented.

It’s also important to note that these features have been introduced for practical reasons. The appropriate use of these features will provide considerable benefits around readability and maintainability by reducing the amount of boilerplate code that is needed for many algorithms.

Adopting Functional Programming Languages (Part 3)

(Note: This is the third part of a whitepaper I wrote a couple of years ago which I never had a chance to publish in full.  You’ll find the earlier parts here and here.)

History

The dawn of functional languages took place when the first implementation of LISP was released in 1959. Since that time, a steady stream of functional languages has been developed. Like many language families, some progeny live long and prosperous lives while others lack the support and mindshare needed to reach critical mass in the market.   The modern languages which are the focus of this paper are still a long way from reaching their golden jubilee, but they are all built upon foundational languages like LISP and ML (originating in the early 1970s).

fp-languages-timeline

A potted timeline of Functional Programming languages

Adopting Functional Programming Languages (Part 2)

(Note: This is the second part of a whitepaper I wrote a couple of years ago which I never had a chance to publish in full.  You’ll find the first part here.)

Definition

Although the functional programming community struggles to agree on an exact definition of what makes a language inherently functional, an agreed common ground exists:

  • Computation via functional composition
  • Higher-order functions
  • Favouring immutability

Functional languages are also declarative, which means they focus on what operations should be executed, rather than how they should be executed. This distinction immediately puts them at odds with the dominant object-oriented/imperative approach to software development and more inline with other declarative languages like SQL, HTML, CSS and the formulae of Excel.

Functional composition

Not surprisingly, the role of functions takes centre stage in the definition of functional languages. While most languages come with similar ways of creating named blocks of code (e.g., methods, procedures, subroutines, etc.), functional languages define this term very specifically. In this world, a function:

  • Creates no side effects
  • References no global state
  • Is referentially transparent

Here is a simple Java function that violates the strict definition of pure functions by outputting content to the console, which is a form of side effect.

fp-whitepaper-code-1

A Java function with a side-effect.

A side effect is any action that changes the execution environment that is not observable through the value returned by the function.   Other common examples of side effects include updating databases or other data stores within the function. Code producing side effects is much harder to reason about because the scope of change produced by the function is so wide.

Here is another version of the previous function that removes the side effect but accesses global state in the form of a collaborator that provides the current exchange rate.

fp-whitepaper-code-2

A Java function using non-local state.

As with side effects, to reason about how this function works, we also need to understand how the collaborator behaves, increasing the complexity of the problem significantly.

The final version of this function is one that includes no side effects and references no global state.

fp-whitepaper-code-3

A pure Java function.

This function is therefore referentially transparent, which means it will return exactly the same value every single time it is evaluated. Referential transparency offers enormous benefits to both the programmer and compiler in terms of caching, parallelisation and other performance optimisations.

The lack of side effect and global state is what classifies a function as a pure function and languages that only provide pure functions are called pure functional languages (e.g., Haskell). In practice, many functional languages are impure and developers make disciplined use of side effects for things like logging and database access.  Pure functional languages like Haskell use concepts like IO monads (see http://www.haskell.org/tutorial/io.html) to perform these types of operations while maintaining referential transparency and without the downside of side effects.

Higher-order functions

Using functions as the centrepiece of computation is not the only key differentiator for functional languages. Functions are first class in these languages and, as such, can be passed as arguments to other functions, as well as returned as values from functions. The combination of these properties defines higher-order functions.

The judicious use of higher-order functions is what provides much of the expressiveness of functional languages and removes the need for much of the boilerplate code found in non-functional languages. Classic examples of this are performing some arbitrary processing (e.g., sorting, filtering, collecting, etc.) on a list of values. Solutions in imperative languages require explicit looping constructs (e.g., a “for” or “while” loop) and references to the start and end of the list and with the processing logic buried inside of the loop. Conversely, functional languages provide a number of higher-order functions that contain the logic necessary to iterate through the loop. Calling these functions requires an additional predicate function that performs the necessary business logic: all loop iteration initialization and control is abstracted away within the higher-order function.

Below is a trivial example of how Java mimics some of the benefits of higher order functions, by using interfaces (in this case java.util.Comparator) to provide function objects. In this example, the ListSorter parameter to the sortUsingComparator method exists only for its implementation of the compare method.

fp-whitepaper-code-4

Using Java interfaces to mimic a higher order function.

Immutability

Another key difference between functional and imperative/object-oriented languages comes from their respective approaches to state management. Imperative/object-oriented systems are built around the notion of ongoing changes to the state of the software at runtime. Conversely, functional systems emphasize immutability of state and seek to minimize the scope of state that is mutable.

At runtime, a Java system is a network of objects with state, represented by the value of the object’s attributes. Over time, different pieces of code will read and write these attributes, mutating the state of the object and its enclosing system. In order to understand what state a particular object is in, you need to understand what values its attributes contain.

By contrast, functional systems maintain no mutable state. At runtime there is no equivalent of a Java object’s attributes that are changed over time. “Variables” can be initialized with values, but once this initialization is done, the values are persisted unless the variables are re-created. Re-assignment is not permitted so the value of a variable doesn’t vary across its lifetime.

Immutability helps greatly when you need to reason about a piece of code within a system, and one of the earliest times this reasoning needs to occur is when you are writing test cases. If you need to test a Java method that takes two arguments and returns a single value (see below), the number of logical test cases is tied to the set of values the two arguments can contain. If that same method also uses the value of one of its object’s attributes, then all the values of that attribute now need to be considered. And the more this method relies on its state, the harder it is to reason about its behaviour.

fp-whitepaper-code-5Conversely, a function in a language such as Haskell will only act upon its arguments and return a result. Building the test cases for this function is a much smaller, quicker and simpler task.

Immutability is not a property uniquely associated with functional languages. Programmers can use discipline to create immutable data structures in any language (more on this in a later post) and several non-functional languages include immutable data structures by default or via third-party libraries.

Adopting Functional Programming Languages (Part 1)

Tags

, , , , , ,

(Note: This is the first part of a whitepaper I wrote a couple of years ago which I never had a chance to publish in full.  I plan to serialise the major sections into separate blog articles here.)

Introduction

Over recent years, there has been a quiet renaissance in a style of programming once isolated to niche fields of academia and computer science: functional programming.  Functional languages like Scala, Haskell and Clojure are attracting significant attention from developers, and aspects of functional programming are now established in many programming languages/platforms, including the industry heavyweights Java and C#.

But what is the catalyst for this renaissance?  The increasing scale and sophistication required of custom software development has led people to reprioritise the benefits of functional languages (e.g., immutability, expressiveness) as a way of increasing code quality, boosting development productivity and reducing complexity and risk.

Alongside this rising popularity, organisational barriers to entry to functional programming have been lowered by the availability of these languages on existing enterprise development platforms (e.g., Clojure and Scala running on the Java Virtual Machine, and F# on the Microsoft CLR) and in some cases augmenting existing languages, as Scala has done with it’s hybrid functional/object-oriented orientation.

A third factor contributing to this increased interest is the cyclical nature of IT in general, as subsequent generations of practitioners rediscovering old approaches.   Agile software development principles hark back to the days before heavyweight command-and-control process took over the IT landscape. The NoSQL database movement is “new” only in the sense that many people have forgotten that non-relational databases were once a viable choice for many data storage requirements.

See “Resurgence” for more information on the driving forces behind the popularity of functional programming.

Why functional programming should be your next strategic direction

So why should businesses care about this revolution? Isn’t the choice of programming language an arcane detail in a world where most software can be accessed as a commodity service in the cloud? In some cases this is true. Every business has certain capabilities that are essentially the same from one organisation to the next. Examples might be maintaining the general ledger or managing payroll. The software to support these capabilities can be acquired off the shelf and adopted with little or no customisation. The programming language those systems are written should not be a concern to the consumer. On the other hand, every business depends on capabilities or processes that differentiate them from their competitors. Since these processes are unique, there is—by definition—no off-the-shelf software to implement them. In a business landscape that is increasingly technology-dependent, the ability to continually develop and maintain custom software ahead of competitors is a universal requirement. This is where functional programming languages come in. Choosing the right language for the job can make a business more responsive to customers, allow it to adapt to market pressures faster than competitors, and attract a different class of technical professionals.

The benefits of building expertise in functional programming are twofold:

  1. Deliver more robust and scalable software solutions
  2. Position your organisation as an employer of choice in the marketplace for high value developers