More Productive C++ with TDD

The title might read a little like click-bait, and there are certainly some nuances and qualifications here. But, hey! That's what the article is for.

Those that know me know that I have been a practitioner of, and advocate for, TDD in C++ for many years. I originally introduced Catch in an attempt to make that easier. I've given many talks that touch on the subject, as well as giving coaching and consultancy. For the last year I've added to that with a workshop class that I have given at several conferences - in both one-day and two-day forms (at CppCon you can do either!).

But why am I all in on TDD? Is it really that good?

What has TDD ever done for me?

Most of the time, especially for new code (including new parts being added to legacy code) the benefits of using TDD include (but are not limited to):

  1. A decent set of tests with high coverage (100% if you're following a strict approach).
  2. Well factored, "clean code".
  3. (an aspect of 2) code that is easy to change.
  4. A more thoughtful design that is easier to work with.

But attaining these benefits is not automatic. Applying the discipline of TDD steers you towards the "pit of success" - but you still have to jump in! Fully achieving all the benefits relies on a certain amount of experience - with TDD itself - but also with a range of other design principles. TDD is like a spirit guide, nudging you in the right direction. It's down to you to listen to what TDD is telling you and take the right actions.

This is the first hurdle that people trying TDD fall down at. The core steps to TDD are simple and can be taught in about 10-20 minutes. Getting all the benefits from them takes experience. If you start trying to apply TDD to your production code straight away you will almost certainly just see overhead and constraints - with little-to-no benefit!

On your own this may just be a case of doing enough katas and side-projects until you feel confident to slowly start using it in more critical areas. If you're able to take a class (whether mine or someone else's), you can be helped past this initial stage - and also shown the cloud of additional things around TDD that will let you get the best out of it.

Types, Tests and EOP

First of all, I don't consider TDD to be the complete picture. There are some cases where it's just not practical (although these are often not the ones people think - it takes exprience to make that call). Where it is practical (the majority of cases, in my experience) it can form the backbone for your code-design approach - but should be complemented by other techniques.

If you think of these things as a toolbox to select from - then combining the right tools is the only way to be fully effective. Proficiency - if not mastery - with those tools is also necessary,

The tools that I find, again-and-again, work well with TDD are: using the type system to reduce - or eliminate - potential errors, and what I call Expression-Oriented Programming, which is really a distillation of Functional Programming.

These are two big topics in their own right, and I plan to follow up with deeper dives on each. In the meantime you'll get a better idea of my views on FP from my talk, "Functional Programming for Fun & Profit". I've yet to do a talk specifically on my views on how the type system can help, but there are elements in my recent series on Error Handling.

The bottom line, for now, is that the more you can lean on Types and FP, the less there will be left that needs testing.

TDD or not TDD

Alas, poor Yorik...

That's the question.

I've already hinted that it may not always be the best approach (but usually is). But even when it is - it is not sufficient. That is, the tests you are left with through pure TDD are only half the story. They helped you arrive at the design, gave you a users perspective on your code, and the confidence to make changes. But they won't find the edges of your design that you didn't explicitly consider. TDD might help you think of some pathological cases you might have otherwise left out (and leaning on type system may eliminate many more) - but you're going to have to consider other forms of testing to find the things you didn't design for.

Then there's the thorny issue of how to get legacy code under test - or should you even try? There are more tools in the toolbox to help with all of these things. Even though they are not part of TDD itself, to practice TDD well you'll need to be able to be able to weave it into this bigger picture.

Short-cutting the Gordian Knot

So while learning TDD, technically, is easy - mastering it is a far richer proposition. Returning to our opening question - is it worth it? My experience has been a resounding "yes"! All the pieces that come into play (and we've not mentioned them all, here) will give you more tools, greater experience and a better understanding of the bigger picture - even if you choose not to use TDD in a particular case.

But how can we get up to speed? You could do what I did and pick up bits here and there, reconcile it with your own years of experience, observe the results, adjust course and try again - eventually settling on a pattern that works.

Or you could join me on one of my workshops as I try to give you the benefit of that experience in a distilled form. You won't walk out a seasoned expert, but I think I can help you well down the road.

My next outings are:

Or if you'd like me to come into your company to deliver on-site training you can reach me on atdd@philnash.me.

East End Functions

There has been a recent stirring of attention, in the C++ community, for the practice of always placing the const modifier to the right of the thing it modifies. The practice has even been gifted a catchy name: East Const (which, I think, is what has stirred up the interest).

As purely a matter of style it's fascinating that it seems to have split the community so strongly! There are cases for and against, but both sides seem to revolve around the idea of "consistency". For the East Const believers the consistency is in the sense that you can always apply one, simple, rule about what const means and where it goes. For the West Consters the consistency is with the majority of existing code out there - as well as the Core Guidelines recommendation!

Personally I've been an East Const advocate for many years (although not by that name, of course) - and converted the entire Catch codebase over to East Const quite early on.

But there's another style choice that I've not seen discussed quite as much, but has a number of parallels.

As with East vs West Const this is purely a matter of style (it doesn't change what the compiler generates), and one of the arguments in favour is consistency across application (there are some cases where you must do it this way) - but the main argument against is also consistency - with most existing code. Sound familiar? But what is it?

The issue is about where to specify return types on function signatures. For most of C++'s history the only choice has been to write the type before the name of the function (along with any modifiers). But since C++11 we've been able to write the type at the end of the function signature (after a -> - and the function must be prefixed with the keyword auto).

auto someFunc( int i ) -> std::string;
// instead of
std::string someFunc( int i );

So why would you prefer this style? Well, first there's that consistency argument. It's the only way to specify return types for lambdas. You're also required to use trailing return types if the type is a decltype that is dependent on the name of one of the function's arguments. Indeed, that's the motivating case for adding the syntax in the first place. e.g.:

template <typename Lhs, typename Rhs>
auto add( Lhs const& lhs, Rhs const& rhs ) -> decltype( lhs + rhs ) {
    return lhs + rhs;
}

A Foolish Consistency?

Given those cases where it is required, using the same syntax in all other cases would seem to be more consistent.

I'm not sure the consistency argument is as strong here as it is with East Const - there was never much confusion over what the return applied to, after all. But I think it's worth keeping in mind.

The next reason for is consistency with other languages. Many languages, especially functional programming languages, exclusively use the trailing syntax for return types. Quite a few, e.g. Swift, use the same -> syntax.

It's not a strong reason on its own, but combined with the internal consistency argument I think there's something there.

However, for me at least, the most compelling rationale is for readability. Why do I think it's more readable? There are actually two parts to this:

  1. Function declarations tend to line up. Certain qualifiers might spoil this effect, although one approach might be to group similarly qualified functions (e.g. all virtuals) together. This makes glancing through the list of function names much easier.

  2. The name of the function is usually the most important thing when you're browsing the code. If you're more interested in the return type it's usually because you already know which function you're interested in. So making the name the first thing you read (after the auto introducer) seems fitting.

auto doesItBlend() -> bool;
auto whatsYourFavouriteNumber() -> int;
auto add( double a, double b ) -> double;
void setTheControls();

(note that many who prefer this form, including myself, tend to still put void first)

For me the arguments for are compelling. The arguments against really boil down to the same argument against East Const - inconsistency with older code. As Jon Kalb deliberated on in A Foolish Consistency, this sort of thinking can hold us back.

I've been favouring this style for more than a couple of years now. In fact I tracked down a post to the ACCU mailing list (linked here, but I believe you have to be a subscriber to read it) where I talked about it - and made all the same points I'm making here. My opinion since then has not changed much. Other than feeling more confident that it's The Right Thing.

So I think it's time we gave it a catchy name. Unlike East Const it already has a name, "trailing return types". It's not especially galvanising, though. Given the parallels to East vs West Const - and the fact that it, also, relates to the thing in question being placed to the left or the right, I propose East End Functions (vs West End Functions).

What about the redundant auto keyword?

Think of auto, here, as the "function introducer". In other languages it might be spelt fun or func. If it makes you feel better you could always:

#define func auto

... actually don't. The point is, in languages that introduce a function with func, then have a trailing return type, nobody gives it a second thought. auto is the same number of characters as func. It's a shame it's not quite as expressive - but that's the price of legacy. It shouldn't mean we "can't have nice things".

East End Functions

There has been a recent stirring of attention, in the C++ community, for the practice of always placing the const modifier to the right of the thing it modifies. The practice has even been gifted a catchy name: East Const (which, I think, is what has stirred up the interest).

As purely a matter of style it's fascinating that it seems to have split the community so strongly! There are cases for and against, but both sides seem to revolve around the idea of "consistency". For the East Const believers the consistency is in the sense that you can always apply one, simple, rule about what const means and where it goes. For the West Consters the consistency is with the majority of existing code out there - as well as the Core Guidelines recommendation!

Personally I've been an East Const advocate for many years (although not by that name, of course) - and converted the entire Catch codebase over to East Const quite early on.

But there's another style choice that I've not seen discussed quite as much, but has a number of parallels.

As with East vs West Const this is purely a matter of style (it doesn't change what the compiler generates), and one of the arguments in favour is consistency across application (there are some cases where you must do it this way) - but the main argument against is also consistency - with most existing code. Sound familiar? But what is it?

The issue is about where to specify return types on function signatures. For most of C++'s history the only choice has been to write the type before the name of the function (along with any modifiers). But since C++11 we've been able to write the type at the end of the function signature (after a -> - and the function must be prefixed with the keyword auto).

auto someFunc( int i ) -> std::string;
// instead of
std::string someFunc( int i );

So why would you prefer this style? Well, first there's that consistency argument. It's the only way to specify return types for lambdas. You're also required to use trailing return types if the type is a decltype that is dependent on the name of one of the function's arguments. Indeed, that's the motivating case for adding the syntax in the first place. e.g.:

template <typename Lhs, typename Rhs>
auto add( Lhs const& lhs, Rhs const& rhs ) -> decltype( lhs + rhs ) {
    return lhs + rhs;
}

A Foolish Consistency?

Given those cases where it is required, using the same syntax in all other cases would seem to be more consistent.

I'm not sure the consistency argument is as strong here as it is with East Const - there was never much confusion over what the return applied to, after all. But I think it's worth keeping in mind.

The next reason for is consistency with other languages. Many languages, especially functional programming languages, exclusively use the trailing syntax for return types. Quite a few, e.g. Swift, use the same -> syntax.

It's not a strong reason on its own, but combined with the internal consistency argument I think there's something there.

However, for me at least, the most compelling rationale is for readability. Why do I think it's more readable? There are actually two parts to this:

  1. Function declarations tend to line up. Certain qualifiers might spoil this effect, although one approach might be to group similarly qualified functions (e.g. all virtuals) together. This makes glancing through the list of function names much easier.

  2. The name of the function is usually the most important thing when you're browsing the code. If you're more interested in the return type it's usually because you already know which function you're interested in. So making the name the first thing you read (after the auto introducer) seems fitting.

auto doesItBlend() -> bool;
auto whatsYourFavouriteNumber() -> int;
auto add( double a, double b ) -> double;
void setTheControls();

(note that many who prefer this form, including myself, tend to still put void first)

For me the arguments for are compelling. The arguments against really boil down to the same argument against East Const - inconsistency with older code. As Jon Kalb deliberated on in A Foolish Consistency, this sort of thinking can hold us back.

I've been favouring this style for more than a couple of years now. In fact I tracked down a post to the ACCU mailing list (linked here, but I believe you have to be a subscriber to read it) where I talked about it - and made all the same points I'm making here. My opinion since then has not changed much. Other than feeling more confident that it's The Right Thing.

So I think it's time we gave it a catchy name. Unlike East Const it already has a name, "trailing return types". It's not especially galvanising, though. Given the parallels to East vs West Const - and the fact that it, also, relates to the thing in question being placed to the left or the right, I propose East End Functions (vs West End Functions).

What about the redundant auto keyword?

Think of auto, here, as the "function introducer". In other languages it might be spelt fun or func. If it makes you feel better you could always:

#define func auto

... actually don't. The point is, in languages that introduce a function with func, then have a trailing return type, nobody gives it a second thought. auto is the same number of characters as func. It's a shame it's not quite as expressive - but that's the price of legacy. It shouldn't mean we "can't have nice things".

The World’s First Distributed C++ Meet-up (*)

(* Probably)

Last week my London based C++ user group, C++ London, joined forces with SwedenCpp, based in Stockholm, for a distributed event where we shared video streams with each other. The whole thing was hosted by King, who took care of the audio-visual link up, so we just had to organise the speakers.

We did this as a series of lightning talks (5 or 10 minutes each) - 5 in Stockholm and 7 in London.

This was an experiment. The idea was that, if successful, we could do more like this. I know for a fact that other user groups have been waiting to hear how it went as they think about doing something similar too.

So how did it go? Well, you can see for yourself. The video is here (if you view it on YouTube you'll also find a Table Of Contents to jump to individual talks):

Overall I think it went very well. There were certainly some rough edges, and opportunities to do better next time - but the basic concept worked well, I think. Feedback from attendees in both locations also bore that out. Everyone is keen to do this again!

Survey results

Paul Dreik, in Stockholm, sent out a survey to all attendees after the meeting and collected some stats and comments. Regarding the overall event, over 80% said "Great, let's do it again!". Of the rest, all but one said it was at least as good as a normal meet-up. Only one person thought it was a step down (you can't please everyone).

Interestingly, while lightning talks, as a format, was popular (over 50% thought at least most of the time should be spent on them), a sizeable 40% thought more time should be spent on longer talks. So maybe some combination could work well?

Beyond that there were a couple of people who said that it was too long without a break, and some people wanted time for questions as we go (which is hard to make work for lightning talks).

Room for improvement?

So, some of the things that could be improved (and notes for other groups looking to try):

We could have planned more interaction between the locations. We were very much focused on our own schedules and, other than the handover in the middle, there was not much beyond two independent sites that happened to be watching each other. There was an "open questions" section at the end. Jean Guegant, in Sweden, had mentioned that we'd do this in the intro but I'd missed it - and I think others had too, so we weren't really prepared for it. Between that and everyone forgetting any questions they'd had during the talks meant we didn't get many questions. I think we can do better here - perhaps collecting questions during the talks via a web app, or maybe a Twitter hashtag?

Are there other points during the course of the event that we could interact more between the sites? It can be a tough balance, but I think there is scope to experiment here.

And finally, while we're very grateful to King, and their AV staff, for providing, and setting up, the live stream, as well as recording, I think we made too many assumptions about how that was going to work. In particular we were left with an audio recording that was not as good as we had hoped for. In the London audio, especially, the hand mic and ambient mics were mixed together before recording - so we couldn't separate them out - which would have been very useful in the editing.

So. Would we do it again? Yes, definitely! I think this is a great way to expand the reach of our speakers, and give our members more variety of speakers and topics to listen to. In that respect this was a great success, and I'm excited to see what happens next.

The World’s First Distributed C++ Meet-up (*)

(* Probably)

Last week my London based C++ user group, C++ London, joined forces with SwedenCpp, based in Stockholm, for a distributed event where we shared video streams with each other. The whole thing was hosted by King, who took care of the audio-visual link up, so we just had to organise the speakers.

We did this as a series of lightning talks (5 or 10 minutes each) - 5 in Stockholm and 7 in London.

This was an experiment. The idea was that, if successful, we could do more like this. I know for a fact that other user groups have been waiting to hear how it went as they think about doing something similar too.

So how did it go? Well, you can see for yourself. The video is here (if you view it on YouTube you'll also find a Table Of Contents to jump to individual talks):

Overall I think it went very well. There were certainly some rough edges, and opportunities to do better next time - but the basic concept worked well, I think. Feedback from attendees in both locations also bore that out. Everyone is keen to do this again!

Survey results

Paul Dreik, in Stockholm, sent out a survey to all attendees after the meeting and collected some stats and comments. Regarding the overall event, over 80% said "Great, let's do it again!". Of the rest, all but one said it was at least as good as a normal meet-up. Only one person thought it was a step down (you can't please everyone).

Interestingly, while lightning talks, as a format, was popular (over 50% thought at least most of the time should be spent on them), a sizeable 40% thought more time should be spent on longer talks. So maybe some combination could work well?

Beyond that there were a couple of people who said that it was too long without a break, and some people wanted time for questions as we go (which is hard to make work for lightning talks).

Room for improvement?

So, some of the things that could be improved (and notes for other groups looking to try):

We could have planned more interaction between the locations. We were very much focused on our own schedules and, other than the handover in the middle, there was not much beyond two independent sites that happened to be watching each other. There was an "open questions" section at the end. Jean Guegant, in Sweden, had mentioned that we'd do this in the intro but I'd missed it - and I think others had too, so we weren't really prepared for it. Between that and everyone forgetting any questions they'd had during the talks meant we didn't get many questions. I think we can do better here - perhaps collecting questions during the talks via a web app, or maybe a Twitter hashtag?

Are there other points during the course of the event that we could interact more between the sites? It can be a tough balance, but I think there is scope to experiment here.

And finally, while we're very grateful to King, and their AV staff, for providing, and setting up, the live stream, as well as recording, I think we made too many assumptions about how that was going to work. In particular we were left with an audio recording that was not as good as we had hoped for. In the London audio, especially, the hand mic and ambient mics were mixed together before recording - so we couldn't separate them out - which would have been very useful in the editing.

So. Would we do it again? Yes, definitely! I think this is a great way to expand the reach of our speakers, and give our members more variety of speakers and topics to listen to. In that respect this was a great success, and I'm excited to see what happens next.

Catch2 released

Super Catch

I've been talking about Catch2 for a while - but now it's finally here! The big news for Catch2 is that it drops all support for pre-C++11 compilers. Other than meaning that some users will not be supported (you can still use Catch "Classic" (1.x) - which will get some bug fix updates for a while, at least) that's mostly an internal change - however it enables a number of user-facing changes, both now and in the future. Let's take a look at what they are.

New and shiny

New, composable, command line processor

Clara is the command line parser in Catch. In Catch 1.0 I spun it out into its own library (but it's still embedded in the Catch single header). Like Catch 1.0 itself, Clara was constrained to C++98 compatibility. For Catch2 I've rewritten Clara from the ground up, not only to fully embrace C++11, but also to be composable. What that means here is that each individual command line option or argument can be represented using its own, self-contained, parser. A composite parser of all the options is then assembled, or composed, from those smaller parsers.

The main advantage of this approach is that the set of available options is now trivially extendible outside of Catch, so users can easily specify command line options that can tune their test code.

See my lightning talk at CppCon this year for a bit more

Commas in assertions

As Catch assertions are implemented using macros, it was susceptible to the old problem of how macros interpret commas within macro arguments. Commas may occur in contexts that macros don't know about, such as within angle brackets (e.g. for template instantiations) - and so get interpreted as argument separators for the macro itself.

Now that we can rely on C++11, which includes variadic macros, we can make the assertions variadic, and just reassemble all the arguments again inside. That means we can now write code like the following:

  REQUIRE( getPair() == std::pair<true, "banana">() );

Microbenchmarking (experimental)

Catch2 gains initial support for micro-benchmarking. This is where small pieces of code are timed, usually in a loop so they are repeated enough times to be significant compared to the system clock accuracy. Some extra adjustments need to be made to allow for other sources of jitter and slowdown on the host machine - and, even then, multiple samples should be taken so they can be subject to statistical analysis.

There are many shortcomings with micro-benchmarks - not least that the performance of a piece of code in isolation can often be drastically different to how well it performs in conjunction with other code. This is not only due to the way the compiler may inline or otherwise optimise code together, but even on the CPU instructions can be reordered, pipelined or run in parallel - and with cache levels and branch prediction, the relationship between these things becomes hugely unpredictable.

Nonetheless they can still be useful - and it can be convenient to use the same test framework that you use to write functional tests - not least because there is much shared infrastructure.

Catch's benchmarking support is incomplete at time of this writing, lacking the multi-sampling, statistical analysis and richer reporting that fully-fledged frameworks offer. The intention is to grow this, but only if it can be done without any significant impact to non-benchmarking tests. In lieu of full documentation, see the example tests for now.

Performance

Both runtime and compile time performance are becoming increasingly important for Catch, and a lot of work is going into improving both. Runtime performance was a non-goal initially, so there has been plenty of low-hanging fruit. As a result we're already seeing some significant improvements, and there is more to come.

Compile time is harder. This has always been important, but as Catch has grown over the years, it has begun to suffer. Improving it significantly means making some trade-offs. So far some features that drag compile times have been made configurable - e.g. whether breaking into the debugger on a failed assertion happens in the code that caused it (meaning the debugger code gets compiled into every assertion macro) or one level up the stack (so can be "hidden" in a function). Other areas to look at are whether to use (non-standard, potentially brittle) forward declarations of some standard library types. Again, this is an ongoing area of active development - but much is already in Catch2 at launch.

See some of the toggle macros for more details

A new name

Believe it or not "Catch2" is now the name - it's not just a version reference! In fact the current intention is that, even when we move to v3.x it will still be called Catch2. E.g. Catch2 v3.0. Why? Well there have been calls for a name change - for searchability reasons. Catch is obviously a common keyword in C++, and also in unit testing. So getting search terms sufficiently narrow has been tricky. But I didn't want to use an entirely different name (although I did toy with the idea of "Catfish" for a while) - because that would lose too much of the momentum behind Catch overall. A derivative name doesn't fully solve the problem, because people will still refer to it as Catch, casually - but at least it gives a slight advantage. So that's what I've gone with. There are also a few interesting numerological aspects to it. It stops short of being Catch22 - but if you consider the C++11 requirement you could multiply them to get 22. And you can add the digits in C++11 to get 2.

Upcoming features

It's always dangerous to talk about what's planned - and I've fallen into this trap with Catch before. So there are some feature promises that have been outstanding for a long time now. In fact most of those have been deferred to Catch2 for quite some time - either because C++11 has features that make them much easier/ possible to implement (e.g. threading) - or just because they involved a lot of code that gets less noisy in C++11. So we'll talk about those again here.

Threading

This was unfeasible in Catch 1.x due not having C++11 threading primitives, or being able to use external dependencies like Boost for threading. To provide a basic level of support should now be fairly straightforward as a lot of groundwork has been laid (e.g. how singletons are organised).

The idea is that if, within a test case, you use additional threads, you should be able to make assertions from those threads - as long as the test case is still in scope at the time. The aim is for this to be done without locks in the assertions. Running multiple test cases in parallel is not immediately planned (and may be best implemented at the process level, anyway).

Generators/ Property Based Testing

Generators give you what other frameworks might call (Data-)Parameterised Tests - i.e. being able to use the same test code with different inputs. An experimental version of generators was included in Catch from very early on. Other than not being complete and having some limitations it also had a serious issue in that it didn't work at all with Sections! This is because both features relied on the ability to re-enter test cases - but they were independent of each other. I rewrote the test-case tracking code a couple of years ago now to be able to support this properly - and had a proof-of-concept new implementation of Generators working with it - enough to give a demo at a talk I gave. However the implementation was getting noisy with C++98 syntax so I deferred work on it for Catch2. Now that Catch2 is released I'll be looking at this again. Closely related to Generators - in fact it builds on it - is the idea of Property Based Testing. The proof-of-concept I mentioned actually had an initial version of this, too. There's more work involved here to getting it right, but having Generators is a first step.

Breaking Changes

As a major version change we've taken advantage of the permission that Semantic Versioning gives us to introduce a few breaking changes. These should have little, if any, impact on most users - but it's worth checking these before making the move to be sure you're ready.

toString() has been removed

This is probably the biggest change, and the most likely to affect people. For a long time there have been three ways to tell Catch how to convert values into strings for reporting purposes. In order, the pipeline was like this:
  1. toString() overload
  2. StringMaker<> specialisation
  3. ostream& operator << overload
  4. give up and use {?}

If your types already have << overloads for ostream then you're good. If not then, in theory, overloading toString() was the simplest option.

However toString() had a number of limitations - mostly due to the point of template instantiation. Compiler differences with two-phase lookup, and other factors which are implementation defined, mean that toString() overloads were unreliable and caused a lot of confusion - hardly the simplest option after all!

Specialising StringMaker<> is slightly more work, but is more reliable, stable, and flexible. So this is now the recommended way to provide string conversion functionality for your types. In Catch2, toString() has been completely removed!

If you have code that calls toString() there is a new function that plays that role: Catch::detail::stringify(). However, note that (a) this should never be overloaded - it just wraps the call into the pipeline that starts with StringMaker<> and (b) the detail part of the namespace should be a clue that this is really an internal part of Catch and is subject to change.

To specialise StringMaker<> see the documentation.

Other removals and changes

As well as C++98 support and toString(), a number of deprecated features and interfaces have been removed, as well as a few other tweaks and changes that may impact some code-bases. See the "Breaking Changes" section in the release notes for the full list. In fact the release notes in general give a good overview of all the many small changes and improvements that have gone into Catch2 that have not been mentioned here.

A new home

The Catch(2) repository has moved! You may not have noticed as it has been transferred in GitHub, and that means GitHub maintains redirects for all the old links. However they do recommend updating your own urls, in bookmarks, direct download links and, of course, git remotes. We've made this move for two reasons:

1. As Catch has grown it has become more of a community effort. It already has an additional lead maintainer in Martin Hořeňovský, and others may be added. But as the sole owner of my own personal GitHub account there are some things that only I could do (webhooks and other integrations, for example). So as not to be a bottleneck we've created a GitHub "Organization" account, CatchOrg, which allows multiple admin users. That's where we've moved it to.

2. For Catch2 to get any traction as a new name it was important for it to be reflected in the repo name, so we've taken advantage of the move to change the repo name, too. Catch "Classic" (1.x) has also moved here, but is now on a branch. If you cannot move to Catch2 for C++98 compatibility reasons you can stay on Catch Classic on this branch. It will continue to receive critical fixes, at least for now, but is no longer the active development branch. Please try to move to Catch2 as soon as possible.

If you notice anything broken as a result of this move, please let us know so we can fix it.

Thanks!

As always, a huge thanks to all who have supported and contributed to Catch and Catch2 - especially for your patience when I wasn't getting to issues and PRs as quickly as was needed!

An extra thanks to Martin, who has been doing the majority of the work on Catch this year!

Catch2 Released

I've been talking about Catch2 for a while - but now it's finally here

The big news for Catch2 is that it drops all support for pre-C++11 compilers. Other than meaning that some users will not be supported (you can still use Catch "Classic" (1.x) - which will get some bug fix updates for a while, at least) that's mostly an internal change - however it enables a number of user-facing changes, both now and in the future. Let's take a look at what they are.

New and shiny

New, composable, command line processor

Clara is the command line parser in Catch. In Catch 1.0 I spun it out into its own library (but it's still embedded in the Catch single header). Like Catch 1.0 itself, Clara was constrained to C++98 compatibility. For Catch2 I've rewritten Clara from the ground up, not only to fully embrace C++11, but also to be composable. What that means here is that each individual command line option or argument can be represented using its own, self-contained, parser. A composite parser of all the options is then assembled, or composed, from those smaller parsers.

The main advantage of this approach is that the set of available options is now trivially extendible outside of Catch, so users can easily specify command line options that can tune their test code.

See my lightning talk at CppCon this year for a bit more

Commas in assertions

As Catch assertions are implemented using macros, it was susceptible to the old problem of how macros interpret commas within macro arguments. Commas may occur in contexts that macros don't know about, such as within angle brackets (e.g. for template instantiations) - and so get interpreted as argument separators for the macro itself.

Now that we can rely on C++11, which includes variadic macros, we can make the assertions variadic, and just reassemble all the arguments again inside. That means we can now write code like the following:

REQUIRE( getPair() == std::pair<true, "banana">() );

Microbenchmarking (experimental)

Catch2 gains initial support for micro-benchmarking. This is where small pieces of code are timed, usually in a loop so they are repeated enough times to be significant compared to the system clock accuracy. Some extra adjustments need to be made to allow for other sources of jitter and slowdown on the host machine - and, even then, multiple samples should be taken so they can be subject to statistical analysis.

There are many shortcomings with micro-benchmarks - not least that the performance of a piece of code in isolation can often be drastically different to how well it performs in conjunction with other code. This is not only due to the way the compiler may inline or otherwise optimise code together, but even on the CPU instructions can be reordered, pipelined or run in parallel - and with cache levels and branch prediction, the relationship between these things becomes hugely unpredictable.

Nonetheless they can still be useful - and it can be convenient to use the same test framework that you use to write functional tests - not least because there is much shared infrastructure.

Catch's benchmarking support is incomplete at time of this writing, lacking the multi-sampling, statistical analysis and richer reporting that fully-fledged frameworks offer. The intention is to grow this, but only if it can be done without any significant impact to non-benchmarking tests. In lieu of full documentation, see the example tests for now.

Performance

Both runtime and compile time performance are becoming increasingly important for Catch, and a lot of work is going into improving both. Runtime performance was a non-goal initially, so there has been plenty of low-hanging fruit. As a result we're already seeing some significant improvements, and there is more to come.

Compile time is harder. This has always been important, but as Catch has grown over the years, it has begun to suffer. Improving it significantly means making some trade-offs. So far some features that drag compile times have been made configurable - e.g. whether breaking into the debugger on a failed assertion happens in the code that caused it (meaning the debugger code gets compiled into every assertion macro) or one level up the stack (so can be "hidden" in a function). Other areas to look at are whether to use (non-standard, potentially brittle) forward declarations of some standard library types. Again, this is an ongoing area of active development - but much is already in Catch2 at launch.

See some of the toggle macros for more details

A new name

Believe it or not "Catch2" is now the name - it's not just a version reference! In fact the current intention is that, even when we move to v3.x it will still be called Catch2. E.g. Catch2 v3.0.

Why?

Well there have been calls for a name change - for searchability reasons. Catch is obviously a common keyword in C++, and also in unit testing. So getting search terms sufficiently narrow has been tricky. But I didn't want to use an entirely different name (although I did toy with the idea of "Catfish" for a while) - because that would lose too much of the momentum behind Catch overall. A derivative name doesn't fully solve the problem, because people will still refer to it as Catch, casually - but at least it gives a slight advantage. So that's what I've gone with.

There are also a few interesting numerological aspects to it. It stops short of being Catch22 - but if you consider the C++11 requirement you could multiply them to get 22. And you can add the digits in C++11 to get 2.

Upcoming features

It's always dangerous to talk about what's planned - and I've fallen into this trap with Catch before. So there are some feature promises that have been outstanding for a long time now. In fact most of those have been deferred to Catch2 for quite some time - either because C++11 has features that make them much easier/ possible to implement (e.g. threading) - or just because they involved a lot of code that gets less noisy in C++11. So we'll talk about those again here.

Threading

This was unfeasible in Catch 1.x due not having C++11 threading primitives, or being able to use external dependencies like Boost for threading. To provide a basic level of support should now be fairly straightforward as a lot of groundwork has been laid (e.g. how singletons are organised).

The idea is that if, within a test case, you use additional threads, you should be able to make assertions from those threads - as long as the test case is still in scope at the time. The aim is for this to be done without locks in the assertions. Running multiple test cases in parallel is not immediately planned (and may be best implemented at the process level, anyway).

Generators/ Property Based Testing

Generators give you what other frameworks might call (Data-)Parameterised Tests - i.e. being able to use the same test code with different inputs.

An experimental version of generators was included in Catch from very early on. Other than not being complete and having some limitations it also had a serious issue in that it didn't work at all with Sections! This is because both features relied on the ability to re-enter test cases - but they were independent of each other.

I rewrote the test-case tracking code a couple of years ago now to be able to support this properly - and had a proof-of-concept new implementation of Generators working with it - enough to give a demo at a talk I gave. However the implementation was getting noisy with C++98 syntax so I deferred work on it for Catch2. Now that Catch2 is released I'll be looking at this again.

Closely related to Generators - in fact it builds on it - is the idea of [Property Based Testing](http://hypothesis.works/articles/what-is-property-based-testing/]. The proof-of-concept I mentioned actually had an initial version of this, too. There's more work involved here to getting it right, but having Generators is a first step.

Breaking Changes

As a major version change we've taken advantage of the permission that Semantic Versioning gives us to introduce a few breaking changes. These should have little, if any, impact on most users - but it's worth checking these before making the move to be sure you're ready.

toString() has been removed

This is probably the biggest change, and the most likely to affect people. For a long time there have been three ways to tell Catch how to convert values into strings for reporting purposes. In order, the pipeline was like this:

  • toString() overload
  • StringMaker<> specialisation
  • ostream& operator << overload
  • give up and use {?}

If your types already have << overloads for ostream then you're good. If not then, in theory, overloading toString()was the simplest option.

However toString() had a number of limitations - mostly due to the point of template instantiation. Compiler differences with two-phase lookup, and other factors which are implementation defined, mean that toString() overloads were unreliable and caused a lot of confusion - hardly the simplest option after all!

Specialising StringMaker<> is slightly more work, but is more reliable, stable, and flexible. So this is now the recommended way to provide string conversion functionality for your types. In Catch2, toString()has been completely removed!

If you have code that calls toString() there is a new function that plays that role: Catch::detail::stringify(). However, note that (a) this should never be overloaded - it just wraps the call into the pipeline that starts with StringMaker<> and (b) the detail part of the namespace should be a clue that this is really an internal part of Catch and is subject to change.

To specialise StringMaker<> see the documentation.

Other removals and changes

As well as C++98 support and toString(), a number of deprecated features and interfaces have been removed, as well as a few other tweaks and changes that may impact some code-bases. See the "Breaking Changes" section in the release notes for the full list. In fact the release notes in general give a good overview of all the many small changes and improvements that have gone into Catch2 that have not been mentioned here.

A new home

The Catch(2) repository has moved! You may not have noticed as it has been transferred in GitHub, and that means GitHub maintains redirects for all the old links. However they do recommend updating your own urls, in bookmarks, direct download links and, of course, git remotes. We've made this move for two reasons:

  1. As Catch has grown it has become more of a community effort. It already has an additional lead maintainer in Martin Hořeňovský, and others may be added. But as the sole owner of my own personal GitHub account there are some things that only I could do (webhooks and other integrations, for example). So as not to be a bottleneck we've created a GitHub "Organization" account, CatchOrg, which allows multiple admin users. That's where we've moved it to.

  2. For Catch2 to get any traction as a new name it was important for it to be reflected in the repo name, so we've taken advantage of the move to change the repo name, too. Catch "Classic" (1.x) has also moved here, but is now on a branch. If you cannot move to Catch2 for C++98 compatibility reasons you can stay on Catch Classic on this branch. It will continue to receive critical fixes, at least for now, but is no longer the active development branch. Please try to move to Catch2 as soon as possible.

If you notice anything broken as a result of this move, please let us know so we can fix it.

Thanks!

As always, a huge thanks to all who have supported and contributed to Catch and Catch2 - especially for your patience when I wasn't getting to issues and PRs as quickly as was needed!

An extra thanks to Martin, who has been doing the majority of the work on Catch this year!

Catch Up

Trolley

Stock image from Shutterstock

It's been just over six years since I first announced Catch to the world as a brand new C++ test framework!

In that time it has matured to the point that it can take on the heavyweights - while still staying true to its original goals of being lightweight, easy to get started with and low-friction to work with.

In the last couple of years or so it has also increased dramatically in popularity! That sounds like a good thing - and it is - but with that comes a greater diversity of environments and usage, and more people raising issues and submitting pull requests.

Again, it's great to have so much input from the community - especially in the form of pull requests - where other developers have gone to some effort to implement a change, or a fix, and present it back for inclusion in the main project. So it's been heart-breaking for me that, between this increase in volume and finding my meagre free-time stretched even further, so many issues and PRs have been left unacknowledged - many not even seen by me in the first place.

But two things have happened, recently, that completely change this state of affairs. We're moving firmly in the right direction again.

Firstly, as mentioned in On Joining JetBrains, I've recently changed jobs to one that should give me much more time and opportunity to work on Catch - as well as the opportunity to do so in my home office - with stable internet (as opposed to on the train while commuting to and from work). The first few months were a bit of a wash for the reasons discussed in that post, but, as I also suggested there, this year has seen that change and I've been able to put in quite a lot of work on Catch already.

But that's not really enough. There's a huge back-log - and I'm still only doing this part time - and I want to spend time working on Catch2 as well (more on that soon). I don't want to end up back in the situation where everything is backing up and there's no hope of recovery.

I've been hoping to find someone else to be a key maintainer of Catch for a couple of years now. I've not been very active in this search - for all the same reasons - but it's been on my mind.

But, just last month, after I appeared on CppCast talking about JetBrains and Catch, there was a thread on Reddit about it - with many expressing concern over the Catch situation. I brought the subject up on there again and got the attention of one of the commenters.

I didn't know it at the time, but Martin Hořeňovský has been responsible for a good number of those PRs and issues that had been left unaddressed - as well as an active community member in helping address other people's issues. So it's with great pleasure (and relief!) that I can announce that Martin now has full commit rights to Catch on GitHub and has been prolific in working through the currently outstanding tickets.

Martin seems to really "get" Catch, and the design goals around it - so working with him on this the last couple of weeks has been very rewarding. From some queries I just ran on GitHub it looks like 39 issues have been closed and 38 PRs merged or closed in that time! That's compared to 9 new issues and 7 PRs - about half of which were created by Martin and I in the process. And that's not to mention all the labels we've been using to categorise the other tickets - with many marked as "Resolved - pending review" - which usually means we think it's resolved but we're just waiting for feedback (or a chance for more testing).

With 219 open issues and 41 PRs still outstanding, at time of writing, there's a lot more work to do yet - but I hope this reassures you that we're going in the right direction - and fast!

And we're not stopping with Martin. We have at least one other volunteer that I'll be bringing up to speed soon.

Catch2

I've referred to Catch2 a number of times now, and talked a little about what it will be. The biggest reason for making it a major release, according to Semantic Versioning, is that it will drop support for pre-C++11. For that reason Catch Classic (1.x) will continue to receive at least bug fix updates - but no more new features once Catch2 is fully released. A few major features in the pipeline have been explicitly deferred to Catch2: concurrency support and generators/ property-based testing in particular.

Moving to C++11 provides a very large scope for cleaning up the code-base - which has a significant volume of code dedicated to platform-specific workarounds for compiler shortcomings, missing library features such as smart pointers, and boilerplate that will no longer be necessary with things like range-based-for, auto and others. Lambdas will be useful too, but are not quite so important.

Because taking advantage of C++11 has the potential to touch almost every line of code, I'm taking the opportunity to rewrite the core of Catch - primarily the assertion macros and the infrastructure to support that. This is code that is #included in every test file, and expanded (in the case of macros) in every test case or even every assertion. Keeping this code lightweight is essential to avoiding a compile time hit. There's a number of ways this foot-print can be reduced and the rewrite will strive for this as much as possible.

The rest of the code, concerned with maintaining the registry of tests, parsing and interpreting the command line, running tests and reporting results, will be updated more incrementally.

I already have a (not-yet-public) proof-of-concept version of the re-written code. It's not yet complete but, so far, has only one standard library dependency and minimal templates. The compile-time overhead is imperceptible.

In addition to compile-time, runtime performance is also a goal of Catch2. It's not an overriding goal - I won't be obfuscating the code in the name of wringing out the last few milliseconds of performance - but this is a definite change from Catch Classic where runtime performance was a non-goal. This is in recognition of the fact that Catch is used for more than just isolated unit tests - and will also become more important with property based testing.

I don't have a timeline, yet, for when I expect Catch2 to be ready - and in the immediate term getting Catch Classic back under control is the priority. Despite the partial re-write, and the major version increment, I expect tests written against Catch Classic to mostly "just work" with Catch2 - or require very minimal changes in a some rare cases.

You

As already mentioned many developers have also spent time and effort contributing issues, fixes and even feature PRs over the years. So Catch has really been a community project for years now and I'm very grateful for all the help and support. I think Catch has shown that having a low-friction approach to testing C++ code is very important to a lot of people and I'm hoping we'll continue to build on that. Thank you all.

Catch Up

Trolley

It's been just over six years since I first announced Catch to the world as a brand new C++ test framework!

In that time it has matured to the point that it can take on the heavyweights - while still staying true to its original goals of being lightweight, easy to get started with and low-friction to work with.

In the last couple of years or so it has also increased dramatically in popularity! That sounds like a good thing - and it is - but with that comes a greater diversity of environments and usage, and more people raising issues and submitting pull requests.

Again, it's great to have so much input from the community - especially in the form of pull requests - where other developers have gone to some effort to implement a change, or a fix, and present it back for inclusion in the main project. So it's been heart-breaking for me that, between this increase in volume and finding my meagre free-time stretched even further, so many issues and PRs have been left unacknowledged - many not even seen by me in the first place.

But two things have happened, recently, that completely change this state of affairs. We're moving firmly in the right direction again.

Firstly, as mentioned in On Joining JetBrains, I've recently changed jobs to one that should give me much more time and opportunity to work on Catch - as well as the opportunity to do so in my home office - with stable internet (as opposed to on the train while commuting to and from work). The first few months were a bit of a wash for the reasons discussed in that post, but, as I also suggested there, this year has seen that change and I've been able to put in quite a lot of work on Catch already.

But that's not really enough. There's a huge back-log - and I'm still only doing this part time - and I want to spend time working on Catch2 as well (more on that soon). I don't want to end up back in the situation where everything is backing up and there's no hope of recovery.

I've been hoping to find someone else to be a key maintainer of Catch for a couple of years now. I've not been very active in this search - for all the same reasons - but it's been on my mind.

But, just last month, after I appeared on CppCast talking about JetBrains and Catch, there was a thread on Reddit about it - with many expressing concern over the Catch situation. I brought the subject up on there again and got the attention of one of the commenters.

I didn't know it at the time, but Martin Hořeňovský has been responsible for a good number of those PRs and issues that had been left unaddressed - as well as an active community member in helping address other people's issues. So it's with great pleasure (and relief!) that I can announce that Martin now has full commit rights to Catch on GitHub and has been prolific in working through the currently outstanding tickets.

Martin seems to really "get" Catch, and the design goals around it - so working with him on this the last couple of weeks has been very rewarding. From some queries I just ran on GitHub it looks like 39 issues have been closed and 38 PRs merged or closed in that time! That's compared to 9 new issues and 7 PRs - about half of which were created by Martin and I in the process. And that's not to mention all the labels we've been using to categorise the other tickets - with many marked as "Resolved - pending review" - which usually means we think it's resolved but we're just waiting for feedback (or a chance for more testing).

With 219 open issues and 41 PRs still outstanding, at time of writing, there's a lot more work to do yet - but I hope this reassures you that we're going in the right direction - and fast!

And we're not stopping with Martin. We have at least one other volunteer that I'll be bringing up to speed soon.

Catch2

I've referred to Catch2 a number of times now, and talked a little about what it will be. The biggest reason for making it a major release, according to Semantic Versioning, is that it will drop support for pre-C++11. For that reason Catch Classic (1.x) will continue to receive at least bug fix updates - but no more new features once Catch2 is fully released. A few major features in the pipeline have been explicitly deferred to Catch2: concurrency support and generators/ property-based testing in particular.

Moving to C++11 provides a very large scope for cleaning up the code-base - which has a significant volume of code dedicated to platform-specific workarounds for compiler shortcomings, missing library features such as smart pointers, and boilerplate that will no longer be necessary with things like range-based-for, auto and others. Lambdas will be useful too, but are not quite so important.

Because taking advantage of C++11 has the potential to touch almost every line of code, I'm taking the opportunity to rewrite the core of Catch - primarily the assertion macros and the infrastructure to support that. This is code that is #included in every test file, and expanded (in the case of macros) in every test case or even every assertion. Keeping this code lightweight is essential to avoiding a compile time hit. There's a number of ways this foot-print can be reduced and the rewrite will strive for this as much as possible.

The rest of the code, concerned with maintaining the registry of tests, parsing and interpreting the command line, running tests and reporting results, will be updated more incrementally.

I already have a (not-yet-public) proof-of-concept version of the re-written code. It's not yet complete but, so far, has only one standard library dependency and minimal templates. The compile-time overhead is imperceptible.

In addition to compile-time, runtime performance is also a goal of Catch2. It's not an overriding goal - I won't be obfuscating the code in the name of wringing out the last few milliseconds of performance - but this is a definite change from Catch Classic where runtime performance was a non-goal. This is in recognition of the fact that Catch is used for more than just isolated unit tests - and will also become more important with property based testing.

I don't have a timeline, yet, for when I expect Catch2 to be ready - and in the immediate term getting Catch Classic back under control is the priority. Despite the partial re-write, and the major version increment, I expect tests written against Catch Classic to mostly "just work" with Catch2 - or require very minimal changes in a some rare cases.

You

As already mentioned many developers have also spent time and effort contributing issues, fixes and even feature PRs over the years. So Catch has really been a community project for years now and I'm very grateful for all the help and support. I think Catch has shown that having a low-friction approach to testing C++ code is very important to a lot of people and I'm hoping we'll continue to build on that. Thank you all.

C++17 – Why it’s better than you might think

C++20 Horizon

From Mark Isaacson's Meeting C++ talk, "Exploring C++ and beyond"

I was recently interviewed for CppCast and one the news items that came up was a trip report from a recent C++ standards meeting (Issaquah, Nov 2016). This was one of the final meetings before the C++17 standard is wrapped up, so things are looking pretty set at this point. During the discussion I made the point that, despite initially being disappointed that so many headline features were not making it in (Concepts, Modules, Coroutines and Ranges - as well as dot operator and uniform call syntax), I'm actually very happy with how C++17 is shaping up. There are some very nice refinements and features (const expr if is looking quite big on its own) - and including a few surprise ones (structured bindings being the main one for me).

But the part of what I said that surprised even me (because I hadn't really thought of it until a couple of hours before we recorded) was that perhaps it is for the best that we don't get those bigger features just yet! The thinking was that if you take them all together - or even just two or three of them - they have the potential to change the language - and the way we write "modern C++" perhaps even more so than C++11 did - and that's really saying something! Now that's a good thing, in my opinion, but I do wonder if it would be too soon for such large scale changes just yet.

After the 98 standard C++ went into a thirteen year period in the wilderness (there was C++03, which fixed a couple of problems with the 98 standard - but didn't actually add any new features - except value initialisation). As this period coincided with the rise of other mainstream languages - Java and C# in particular - it seemed that C++ was a dying language - destined for a drawn out, Cobolesque, old-age at best.

But C++11 changed all that and injected a vitality and enthusiasm into the community not seen since the late 90s - if ever! Again the timing was a factor - with Moore's Law no longer influencing single-core performance there was a resurgence of interest in low/ zero overhead systems languages - and C++11 was getting modern enough to be palatable again. "There's no such thing as a free lunch" turns out to be true if you wait long enough.

So the seismic changes in C++11 were overdue, welcome and much needed at that time. Since then the standardisation process has moved to the "train model", which has settled on a new standard every three years. Whatever is ready (and fits) makes it in. If it's not baked it's dropped - or is moved into a TS that can be given more real-world testing before being reconsidered. This has allowed momentum to be maintained and reassures us that we won't be stuck without an update to the standard for too long again.

On the other hand many code-bases are still catching up to C++11. There are not many breaking changes - and you can introduce newer features incrementally and to only parts of the code-base - but this can lead to some odd looking code and once you start converting things you tend to want to go all in. Even if that's not true for your own codebase it may be true of libraries and frameworks you depend on! Those features we wanted in C++17 could have a similar - maybe even greater - effect and my feeling is that, while they would certainly be welcomed by many (me included) - there would also be many more that might start to see the churn on the language as a sign of instability. "What? We've only just moved on to C++11 and you want us to adopt these features too?". Sometimes it can be nice to just know where you are with a language - especially after a large set of changes. 2011 might seem like a long time ago but there's a long lag in compiler conformance, then compiler adoption, then understanding and usage of newer features. Those just starting to experiment with C++11 language features are still very common.

I could be wrong about this, but it feels like there's something in it based on my experience. And I think the long gap between C++98 and C++11 is responsible for at least amplifying the effect. People got used to C++ being defined a single way and now we have three standards already in use, with another one almost ready. It's a lot to keep up with - even for those of us that enjoy that sort of thing!

So I'm really looking forward to those bigger features that we'll hopefully get in C++20 (and don't forget you can even use the TS's now if your compiler supports them - and the Ranges library is available on GitHub) - but I'm also looking forward to updating the language with C++17 and the community gaining a little more experience with the new, rapidly evolving, model of C++ before the next big push.

On joining JetBrains

JbOffice

The JetBrains office in St. Petersburg

I recently joined JetBrains as Developer Advocate for their (our) C++ related tools, CLion (a cross-platform C++ IDE), AppCode (for Mac and iOS development - also supporting Swift and Objective-C) and ReSharper C++ (plug-in for Visual Studio).

This was a significant move for me as my previous full-time role had been as a developer at one place for over seven years! In that time I had been able to take time out to do occasional stints of coaching and consulting, as well as travelling to a number of conferences and other events where I always felt like I was more a part of the community. But making that a significant part of my job has been an interesting transition.

JetBrains are a superb software engineering company with a friendly, diverse, set of employees, distributed across a number of offices. I primarily interact with members in St. Petersburg, Russia and Munich, Germany (and have already visited both offices). Personally I work at home - as do the rest of the Developer Advocacy team (their respective homes, that is. There's not quite enough room at my place!)

I'm not going to talk too much more about my role in general, here. I covered it a bit more in an interview published on some of the JetBrains blogs soon after I joined. What I wanted to talk about here is how it affects my other activities - and in particular Catch.

One of the enticing things about taking this role was that I would be expected to continue work on Open Source projects. After all I wouldn't be working on a paid project anymore - but I still need to keep my skills relevant (and not just tuned to small, self-contained, demos). Over the last three months, however, I've found there has been a lot more to get up-to-speed on - as well as opportunity to overbook myself - than I had anticipated. So catching up on Catch has been minimal so far.

But with my first three release-oriented activities under my belt, and a clearer idea of what to expect, I'm looking forward to turning that around early in the new year. I have a lot of issues and PRs to catch up on, and I've started work on Catch2 in parallel, which I'm keen to get to a first release of (I intend to follow-up soon with a post on Catch2). I'm also hoping to find people I can work with (and give commit rights to) to help with all of the above.

Beyond Catch I expect to blog more frequently again (both here and on the JetBrains blogs) and I have some other projects in the pipeline too.

A Game of Tag

One of the tent-pole features of Catch is the ability to write test names as free-form strings. When you run a Catch executable from the command line you can specify a test case by name, to run just that one:

./MyTestExe "a very nice test case"

or you can use wildcards to run a group of test cases (or just one with less typing):

./MyTestExe "*very nice*"

If you want to use wildcards but you're not sure what they'll match you can combine this with the listing option, -l, to see which test cases match the pattern:

./MyTestExe "*very nice*" -l
Matching test cases:
  a very nice test case
  a not very nice test case
2 matching test cases

This is already quite a powerful way to group test cases into ad-hoc "suites". However we don't want to twist our test names into artificial schemes for this purposes (although, early on, that's exactly what I proposed). Instead Catch allows you to add "tags" to test cases.

TEST_CASE( "a very nice test case", "[nice][good]" ) { /* ... */ }
TEST_CASE( "a not very nice test case", "[nice][bad]" ) { /* ... */ }

Now we can run all tests with a certain tag:

./MyTestExe [good]

or combination of tags:

./MyTestExe [nice][good]

also with exclusions:

./MyTestExe [nice]~[bad]

unions are supported with ,:

./MyTestExe [nice],[pleasant]

Very powerful! And this functionality has been around for a while.

More recent, and less well known (mostly because they weren't documented until recently) are a set of "special tags": Instruction Tags, Hiding Tags, Tag Aliases and some automatically generated tags.

Let's see what they're all about.

Instruction Tags

In general all tags that start with a symbol are reserved by Catch (or, put another way, user defined tag names must start with an alpha-numeric character). This allows a nice rich range of namespaces for special tags. Tags that start with the ! character are Instruction tags. They inform Catch something about the test case that they apply to. At time of writing the following are defined:

  • [!hide] This "hides" the test from the default run (i.e. if you run the test executable without specifying any names or tags). This feature was originally introduced with the [hide] tag (note, no: !) - and is still supported, though deprecated. There is also a shortcut form, [.] which we'll revisit in a moment.
  • [!throws] This tells Catch that an exception may be thrown in the course of executing the test - even if it is caught and dealt with. If you've ever tried to track down a rogue exception in your debugger - and so have set the debugger to break on exceptions as they're thrown - you'll know how frustrating all the false positives coming from such tests are! So Catch provides a way to suppress exceptions it is expecting - through the -e or --nothrow options on the command line. This already skips over REQUIRE_THROWS... or CHECK_THROWS... assertions. The [!throws] tag covers you for cases where the exception is caught and handled in the code under test (or your test code).
  • [!shouldfail] This tells Catch that you're expecting this test to fail! Furthermore, if it does fail then it should treat that as a pass!
  • [!mayfail] Rather than explicitly inverting the pass/ fail logic as the previous tag does, this tag just says that the test may fail but that's ok (although it is still reported). It's also ok if it passes.

Hiding Tags

We already looked at [!hide] (and the deprecated [hide]) above, and mentioned that [.] was a shortcut for the same.

It turns that when one of these tags is used it is often combined with another tag that is used when you do want to run the test. The classic example is where you write integration tests in the same executable as unit tests. By default you don't want the integration tests to run as you want the shortest possible path to running just unit tests. So you hide them but also tag them [integration], or something similar (the word "integration" has no significance to Catch). So pairings like, [.][integration] or [.][performance] are frequently found together.

So, as a convenience, Catch now supports . as a tag prefix. The rest of the tag can be completely custom and works exactly like any other normal tag - except that the test is also hidden. Our examples would, thus, be written as [.integration] and [.performance]

One final point to mention about hiding tags is that, due to the way they have evolved through a number of forms (including the severely deprecated "./" name prefix) whichever form is used will not only hide the test, but any of the other forms will match it in a tag pattern. e.g. if you tag a test with [.] you can match it with [!hide].

Tag Aliases

As we saw earlier, tags can be combined in fairly complex ways. While this is powerful and flexible, it can be a bit awkward if you often want to use the same tag expression. Wouldn't it be nice if there was a way of writing the expression once then getting Catch to remember it for you - and associate it with an easier to remember name?

Well there is! You can associate any tag pattern with a name that you can use just like any normal tag - except that it must begin with the @ character.

You create a tag alias, in code, using the CATCH_REGISTER_TAG_ALIAS macro. E.g.

CATCH_REGISTER_TAG_ALIAS( "[@not nice]", "~[nice]~[!hide]" );

This registers a tag alias, [@not nice] which, when expanded will match all tests that are not tagged [nice] but also are not hidden. The second part is important because if you have any hidden tests then they will usually be included any time you use a not expression (~) because the rule is that tests are only hidden if no pattern is specified!

Also did you notice that we had a space in the tag name? Surprised? I never said that tags could not include spaces. Of course they can.

You can register as many aliases as you like and you can put them anywhere you like (as long as catch.hpp is #included). However I recommend keeping them all in your main source file (the one you #define CATCH_CONFIG_MAIN, or equivalent) - simply so you only have to look in one place for them.

Filenames As Tags

The newest special tag form is the result of automatically generating a set of tags. The tags all begin with the # character (I've resisted the urge to call them "hash tags"). The rest of the tag is generated from the name of the source file that the test is implemented in. The full path (as reported by __FILE__) is stripped of its directories and extension - so all tests in /Development/Tests/SquirrelTests.cpp would be tagged, [#SquirrelTests].

At time of writing this feature is only available on the develop branch on GitHub - and must be specifically enabled running with the --filenames-as-tags or -# command line options. It's possible that situation may change by the time it makes it onto master.

The Tag Line

So tags not only provide a rich grouping mechanism in Catch - they also allow you to control some aspects of how Catch runs and treats test cases. Some tags can be generated for you - and some tags can be expanded from simpler forms. We've covered here the complete set of special tags at the time of writing. If you're reading this in the future there may be more - I'll try and be better at keeping the docs up-to-date there. Also any stock price tips you might have from the future would be welcome too.

Naming is hard – or is it?

Following Peter Hilton's excellent ACCU talk, at last week's conference in Bristol, "How to name things - the hardest problem in programming", a few of us were discussing some of the points raised - and some not raised.

He had discussed identifier length without any mention of Uncle Bob's guideline, whereby the length of a variable name should be proportional to it's scope (i.e. large or global scopes need longer, descriptive, names whereas in smaller, local, scopes shorter, more concise - even single letter - names are appropriate). This seemed all the more of an omission given that he later referenced the book, Clean Code.

It wasn't that Peter disgreed with Uncle Bob (who doesn't, half the time?) that surprised me but that he didn't even mention it in passing. I thought it was fairly well known. Actually I double checked and it is not discussed fully in the book, which only says, "The length of a name should correspond to the size of its scope". This is expanded considerably in Clean Coders (video) episode 2. Also, of course, this is not really "Uncle Bob's rule". Kevlin Henney recalls that he first heard of it in the 90s and it may well have been kicking around before that. Bob calls it "The Scope Rule".

Kevlin was one of those discussing this afterwards. After initially toying with The Scope Rule in the 90s he came to consider it not particularly useful. This, too, surprised me as I had found it worked quite well for me. Or so I thought. Further discussion with Kevlin led to the conclusion that I had read more of my own interpretation into The Scope Rule than I had realised! So I started musing over exactly what my interpretation was.

A transparent reference

As it happened a concept key to clarifying matters came from another great talk at the same conference just the day before. Didier Verna's "Referential transparency is overrated". In this talk Didier discussed various ways that useful idioms in Lisp required violating referential transparency. At one point he explained how "hidden" variables may be introduced by one macro that were then used by another. This worked because the thing being referred to was named very generically - so both macros agreed on the name. He drew on the term "Anaphora" from linguistics, which is where one part of an expression - usually a pronoun - stands in for a more specialised part - such as a person's name - introduced earlier in the context. For example, just now I used the word, "he" to refer to Didier Verna. It was clear who I was talking about because his was the most recently specified name in the current context. In fact I used this anaphoric term a couple of times - and many, many, times in this article. If I had had to fully qualify "Didier Verna" every time writing would quickly become very cumbersome. Anaphora is used very frequently in natural language - usually to good effect.

Scope Creep

I believe this is key to understanding why and when shorter identifiers can be used too. When I had been talking with Kevlin it had become apparent that we had different interpretations of the word, "scope". I realised that I had subconsciously expanded the specific technical meaning to include a more general idea of "context" - including the anaphoric context.

To make this clear I might write some (C++) code like this:

	std::string s = getNextString();
	if( !s.empty() )
		std::cout << "received string: " << s << std::endl;

Many corporate, or personal, coding standards would balk at such practice! Single character identifiers? Way to obfuscate the code!

But how has it obfuscated anything? Look at it as an anaphoric entity. In this case the variable name 's' is anaphoric. We know it is "the next string" because we saw it being introduced by the function call, "getNextString()". We then use it twice on the next couple of lines. There are no other strings being introduced in the same vicinity to confuse it with, and the context in which it is used is kept small. There is no ambiguity and the full identity is revealed in the immediate vicinity.

Sustainability

But what if we add more code, or move parts of this elsewhere? Certainly code evolves over time in ways that can make things less clear if we don't change them. That's true regardless. Naming of the entities at play should always form part of your consideration when refactoring or otherwise modifying existing code. Does it make this code less "sustainable" (to reference another property that Kevlin likes to talk about)? I don't think so. In the worst case, if you don't immediately notice that a short name has become unclear because it's usage has drifted out of anaphoric range, you'll notice the next time you look at it and, momentarily, think "why is there an free variable called 's' here? What on earth is that. You'll take a moment to find it's original declaration, work out what it is, then decide to encode that in the name by renaming the variable at that point. Variable renaming is one of the safest and most ubiquituous refactorings around so I have no qualms about deferring such identity expansion to such a time as it is needed.

Why?

But what about the other side of the argument? Is there any advantage to using a short, even single character, variable name in the first place?

This is often cast as a matter of optimising for typing speed - in world where we typically read these names many many times more than we write them.

While introducing, even small, speed bumps to writing code might discourage spending more time than necessary writing code (which in turn may discourage certain refactorings) it's not really about typing performance at all - It's about readability! Consider again the linguistic definition of anaphora: substituting an, unambiguous, subsequent reference to an entity with a shorter form (e.g. a pronoun) that means the same thing. We do this all the time in natural speech and the written word. Why? Because it would sound unnatural and cumbersome to fully qualify every entity we talk about all the time!

The same applies in programming. Where it is perfectly clear from the immediate context what an identifier refers to then using greater verbosity actually increases the cognitive friction! The more unnecessary and redundant noise and ceremony we can strip away from our code the easier it will be to read, in a shorter period of time. That fact that anaphora is so common in natural language should give us a clue as to our ability to code with it's use in a natural and efficient way.

Now I've only mentally organised my thoughts around this as a result of ruminating on those two talks - and some of the offshoot discussions - but I realise this is essentially how I had interpreted The Scope Rule. Now I've worked it through when I go back and compare it with what Mr Martin actually said his version sounds like a poor proxy for the anaphoric interpretation.

So naming - good naming - is still hard. We've only just discussed one narrow aspect here. But perhaps this has made some of it that little bit easier.

Modern C++ Testing

Back in August I took my family to stay for a week at my brother's house in (Old) South Wales. I've not been to Wales for a long time and it was great to be back there - but that's not what this post is about, of course.

The thing about Wales is that it has mountains (and where there are no mountains there are plenty of hills and valleys and cliffs). Mobile cell coverage is non-existent in much of the country - particularly where my brother lives.

So the timing was particularly bad when, just as we were driving along the south cost (somewhere between Cardiff and Swansea, I think), I started getting emails and tweets from people pointing out that Catch was riding high on HackerNews! Someone had recently discovered Catch and was enjoying it enough that they wanted to share it with the community. Which is awesome!

Except that, between lack of mobile reception and spending time with my family, I didn't have opportunity to join the discussion.

When I got back home a week later I read through the comments. One of them stuck out because it called me out on describing Catch as a "Modern C++" framework (the commenter recommended another framework, Bandit, as being "more modern").

When I first released Catch, back in 2010, C++11 was still referred to as C++1x (or even C++0x!) and the final release date was still uncertain. So Catch was written to target C++03. It used a few "modern" idioms of the time - but the modernity was intended more as being a break from the past - where most C++ frameworks were just reimplementations of JUnit in C++. So I think the label was somewhat justified at the time.

Of course since then C++11 has not only been standardised but is fully, or nearly fully, implemented by many leading, mainstream, compilers. I think adoption is still not high enough, at this point, that I'd be willing to drop support for C++03 in Catch (there is even an actively maintained fork for VC6!). But it is enough that the baseline for what constitutes "modern C++" has definitely moved on. And now C++14 is here too - pushing it even further forward.

"Modern" is not what it used to be

What does it mean to be a "Modern C++ Test Framework" these days anyway? Well the most obvious thing for the user is probably the use of lambdas. Along with a few other features, lambdas allow for a lot of what previously required macros to be done in pure C++. I'm usually the first to hold this up as A Good Thing. In a moment I'll get to why I don't think it's necessarily as good a step as you might think.

But before I get to that; one other thing: For me, as a framework author, the biggest difference C++11/14 would make to something like Catch would be in the internals. Large chunks of code could be removed, reduced or at least cleaned up. The "no dependencies" policy means that Catch has complete implementations of things like shared pointers, optional types and function objects - as well as many things that must be done the long way round (such as iterating collections - I long for range for loops - or at least BOOST_FOREACH).

The competition

I've come across three frameworks that I'd say qualify as truly trying to be "modern C++ test frameworks". I'm sure there are others - and I've not really even used these ones extensively - but these are the ones I'll reference in this discussion. The three frameworks are:

  • Lest - by Martin Moene, an active contributor to Catch - and partly based on some Catch ideas - re-imagined for a C++11 world.
  • Bandit - this is the one mentioned in the Hacker News comment I kicked off with
  • Mettle - I saw this in a tweet from @MeetingCpp last week and it's what kicked off the train of thought that led me to this post

The case for test case macros

But why did I say that the use of lambdas is not such a good idea? Actually I didn't quite say that. I think lambdas are a very good idea - and in many ways they would certainly clean up at least the mechanics of defining and registering test cases and sections.

Before lambdas C++ had only one place you could write a block of imperative code: in a function (or method). That means that, in Catch, test cases are really just functions - which must have a function signature - including a name (which we hide - because in Catch the test name is a string). Those functions must be captured somehow. This is done by passing a pointer to the function to the constructor of a small class - who's sole purposes is to forward the function pointer onto a global registry. Later, when the tests are being run, the registry is iterated and the function pointers invoked.

So a test case like this:

    TEST_CASE( "test name", "[tags]" )
    {
        /* ... */
    }

...written out in full (after macro expansion) looks something like:

    static void generatedFunctionName();
    namespace{ ::Catch::AutoReg generatedNameAutoRegistrar
        (   &generatedFunctionName, 
       	    ::Catch::SourceLineInfo( __FILE__, static_cast<std::size_t>( __LINE__ ) ), 
            ::Catch::NameAndDesc( "test name", "[tags]") ); 
    }
    static void generatedFunctionName()
    {
        /* .... */
    }

(generatedFunctionName is generated by yet another macro, which combines root with the current line number. Because the function is declared static the identifier is only visible in the current translation unit (cpp file), so this should be unique enough)

So there's a lot of boilerplate here - you wouldn't want to write this all by hand every time you start a new test case!

With lambdas, though, blocks of code are now first class entities, and you can introduce them anonymously. So you could write them like:

    Catch11::TestCase( "test name", "[tags]", []() 
    {
        /* ... */
    } );

This is clearly far better than the expanded macro. But it's still noisier than the version that uses the macro. Most of the C++11/14 test frameworks I've looked at tend to group tests together at a higher level. The individual tests are more like Catch's sections - but the pattern is still the same - you get noise from the lambda syntax in the form of the []() or [&]() to introduce the lambda and an extra ); at the end.

Is that really worth worrying about?

Personally I find it's enough extra noise that I think I'd prefer to continue to use a macro - even if it used lambdas under the hood. But it's also small enough that I can certainly see the case for going macro free here.

Assert yourself

But that's just test cases (and sections). Assertions have traditionally been written using macros too. In this case the main reasons are twofold:

  1. It allows the expression evaluation to be wrapped in an exception handler.
  2. It allows us the capture the file and line number to report on.

(1) can arguably be handled in whatever is holding the current lambda (e.g. it or describe in Bandit, suite, subsuite or expect in Mettle). If these blocks are small enough we should get sufficient locality of exception handling - but it's not as tight as the per-expression handling with the macro approach.

(2) simply cannot be done without involving the preprocessor in some way (whether it's to pass __FILE__ and __LINE__ manually, or to encapsulate that with a macro). How much does that matter? Again it's a matter of taste but you get several benefits from having that information. Whether you use it to manually locate the failing assertion or if you're running the reporter in an IDE window that automatically allows you to double-click the failure message to take you to the line - it's really useful to be able to go straight to it. Do you want to give that up in order to go macro free? Perhaps. Perhaps not.

Interestingly lest still uses a macro for assertions

Weighing up

So we've seen that a truly modern C++ test framework, using lambdas in particular, can allow you to write tests without the use of macros - but at a cost!

So the other side of the equation must be: what benefit do you get from eschewing the macros?

Personally I've always striven to minimise or eliminate the use of macros in C++. In the early days that was mostly about using const, inline and templates. Now lambdas allow us to address some of the remaining cases and I'm all for that.

But I also tend to associate a much higher "cost" to macro usage when it generates imperative code. This is code that you're likely to find yourself needing to step through in a debugger at runtime - and macros really obfuscate this process. When I use macros it tends to be in declarative code. Code that generates purely declarative statements, or effectively declarative statements (such as the test case function registration code). It tends to always generate the exact same machinery - so should not be sensitive to its inputs in ways that will require debugging.

How do Catch's macros play out in that regard? Well the test case registration macros get a pass. Sections are a grey area - they are on the path of code that needs to be stepped over - and, worse, hide a conditional (a section is really just an if statement on a global variable!). So score a few points down there. Assertions are also very much runtime executable - and are frequently on the debugging path! In fact stepping into expressions being asserted on in Catch tests can be quite a pain as you end up stepping into some of the "hidden" calls before you get to the expression you supplied (in Visual Studio, at least, this can be mitigated by excluding the Catch namespace using the StepOver registry key).

Now, interestingly, the use of macros for the assertions was never really about C++03 vs C++11. It was about capturing extra information (file/ line) and wrapping in a try-catch. So if you're willing to make that trade-off there's no reason you can't have non-macro assertions even in C++03!

Back to the future

One of my longer arcs of development on Catch (that I edge towards on each refactoring) is to decouple the assertion mechanism from the guts of the test runner. You should be able to provide your own assertions that work with Catch. Many other test frameworks work this way and it allows them to be much more flexible. In particular it will allow me to decouple the matcher framework (and maybe allow third-party matchers to work with Catch).

Of course this would also allow macro-less assertions to be used (as it happens the assertions in bandit and mettle are both matcher-like already).

So, while I think Catch is committed to supporting C++03 for some time yet, that doesn't mean there is no scope for modernising it and keeping it relevant. And, modern or not, I still believe it is the simplest C++ test framework to get up and running with, and the least noisy to work with.

The Future Of Computing

The future is already here! - it's just not very evenly distributed.

I have some ideas about what computing will be like in the future but it is composed mostly of pieces we already have - or have the promise of. At the centre of my vision is the evolution of the Post-PC device

What is Post PC anyway?

Many people attribute this term to Steve Jobs, who certainly brought it to the mainstream in 2007, using it to describe iOS devices and how they would come to eclipse "traditional" PCs in sales and use. This is already coming to pass. But it was actually David Clark who coined the phrase, back in 1999. That article is really worth a read. You should go and read it now. Go on. I'll wait. (Actually I'll just carry on writing - but the appearance will be the same).

So while the Jobsian vision (initially, at least) refers to the reset in expectation, interaction and ease of use that iOS devices ushered in, Clarks original words encompass more - including Cloud Services, cashless payment systems, and most interestingly (to me) finer grained distribution of responsibilities.

It's that last one where I think the most opportunities are yet to play out.

For two or three decades we have obsessed over convergence. Traditional PC systems converged to a single device - the laptop. Post-PC devices have taken that to the next level - a single slab, fronted by a piece of glass that is both the display and primary input. These tiny devices also pack in cameras, extra sensors and even fingerprint scanners and replace what used to be dozens of separate devices. But they have also been born into a world where wireless communication technologies are ubiquitous and come in many forms. Many of their capabilities are distributed in "the cloud", or consist of sending things between devices or connecting wirelessly with additional "smart" peripherals such as cameras, fitness trackers, printers and other devices. They are intensely personal yet highly social. Autonomous yet democratised. Functions such as Airplay and its counterparts reinforce the idea that these devices are not isolated computing silos. They are participants in a computing ecosystem that is distributed at many different levels. And all so seamlessly that entire demographics that were previously written off as "computer illiterate" are regularly using these devices. They are barely even considered "computers" anymore. The term has come to be associated with that clunky, finicky, bulky thing you used to struggle to get to do anything you want.

This new generation of devices, finally, "just works".

The NeXT Steps

So where does it go from here? Have we reached the end of the evolution of the personal computing device?

Not by a long shot! We're just getting warmed up!

We have just crossed the threshold from general-purpose computers being primarily for the focused used of businesses and enthusiasts to being something that everyone uses and carries with them everywhere. That in itself has been opening up possibilities that had been hitherto unseen or simply not feasible.

The degree to which these devices and their interconnections have embedded themselves into our lives already is quite breath-taking when you take a step back. While, admittedly, I'm a bit of an early adopter, none of the following is particularly extreme:

On a typical, weekday, morning I am awoken by music served as an alarm from my phone. I get up and go to begin my bathroom routine. Part of that routine involves stepping onto a set of scales that take my weight and fat mass and automatically send the figures, via wi-fi, to a cloud service that is immediately accessible to my phone, collated together with a number of other metrics that are tracked over time.

Once finished and dressed I leave the house and go to my car, which automatically unlocks itself due to the proximity of the key fob in my pocket. I get in and push a button and the car starts. As I start driving the media system in the car has automatically connected, via bluetooth, to my phone, which is also still in my pocket, and continues playing the podcast that I had previously been listening to. I drive to the station and park the car.

As I get out I put my bluetooth headphones on and, at the push of another button, they too have connected to my phone (still in my pocket) and the podcast resumes once again. I get on the train and get my laptop out to do some development work. It connects via a personal wi-fi network to my phone for an internet connection (which, when I pick up LTE, is faster than my home broadband was only a few years ago) - all the time it is still sending audio to my headphones. Later I get off the train and walk to my office. As I walk my steps are being counted by a device on my belt that intermittently sends this information on to my phone via Bluetooth LE, where it is sent to the cloud service that is collating my health related measurements - including heart rate and blood pressure. Along my journey something interesting and unexpected happens. I take out my phone and take a photo, then continue on. As I get near the office a reminder pops up that I had set to go off in that proximity. Eventually I get to my desk where I put my phone in a dock to charge because battery technology is still struggling to keep up with all these demands!

We're only just getting started, so it's not all as seamless as it could be yet, but the story I've just recounted is real and usually all "just works" without a hitch. I think, as time goes on, these sort of experiences will become more reliable and encompass more things.

But that's the present - wasn't I going to be talking about the future? Well I apologise for burying the lede but it's important to remember how much of the future is already here (albeit not evenly distributed). And my vision is really an extension of the things already discussed. That may sound a little uninspiring - but remember that phenomenon of incremental advances suddenly creating whole new opportunities?

Evenly distributed

One of the criticisms often levelled at the current crop of Post-PC devices is that they are great for consumption, but less so for content creation - or "real work". Many contend that you still need a "real" PC for that. I don't think it's quite so black and white - but there do remain many tasks that are cumbersome to undertake with a tablet or smartphone. It won't always be that way, though. Although tablets with keyboards and mice, and hybrid operating systems, exist now - that's not the way of the future.

I believe that in the not too distant future touch-screens, keyboards, and other input devices will all be merely components of a distributed "system" that consists of both cloud services and local sharing of storage and processing. This system will scale seamlessly to the task at hand. Whether you need more computational power, a different input metaphor, or a different way to output you should be able to add what you need without missing a beat. Right now if your needs outgrow a tablet you have to switch to a whole different device (a laptop, say) - which may or may not sync over data you were working on - in this future you would just add the keyboard if you need it (more easily than now), add some extra processing units (you can do this now in certain limited ways), extra storage (again cloud services already play a role here - as does card based storage in some tablets) or even an extra display (technologies like AirPlay are showing the promise of this).

Each of these components would be what we call "smart". That is they are computers in their own right with enough processing power and sensors to be aware of their environment and how they connect and interact. Take a display, for example. The display itself would contain accelerometers and gyroscopes so it is aware of it's orientation in the real world and whether it is being moved - just like your tablet or smartphone does now. It would also know when another display is nearby, and if so how near and in what direction. Of course the display would be a touch-screen. Imagine you have an object on one display. You could start up a new display, place it next to the first one, touch the object and "flick" it over to the second display. All without any need to configure anything.

Now this system, distributed as it is, would need a centralised "brain". It must scale down to a single device that can be used in isolation. It would make sense for this to be what we currently think of as a smartphone. We would need to carry them with us everywhere and use them for communication, so it would be equipped with audio input and output and cameras - just as our current smartphones are. In fact they needn't be much different to the smartphones we have now. They would be more powerful - but needn't be much more powerful as they can scale up the processing power as needed with additional devices and/ or cloud services. And with all data synced to cloud services an alternate device could be picked up and made into your primary hub for the day as necessary.

Everyday revisited

Most of the pieces are already there. There are some challenges - mostly business-oriented rather than technical - but the trend is already in this direction. Yet it all seems very incremental. To see how transformative it would be consider a re-run of my story earlier, reworked to showcase these future technologies (and a few others to spice it up a bit).

It's a typical, weekday, morning. I am awoken by music serving as an alarm on my primary computing device (which will have a really cool name). I get up and go to begin my bathroom routine. Part of that routine involves having various health metrics samples and sent to a cloud service. Another part is that my bathroom mirror presents me with some curated information pertinent to the day ahead - the current weather, traffic conditions and any early appointments I have set. Perhaps also the days news headlines.

Once finished and dressed I leave the house and go to my car, which automatically unlocks itself due to the proximity of the computing device in my pocket. I get in and push a button and the car starts. As I start driving the media system in the car has automatically connected to my computing device, which is still in my pocket, and continues playing the podcast that I had previously been listening to. I drive to the station and park the car. My computing device knows that I have just parked in a car park and automatically communicates with the car park server and pays for my day's stay.

Just before I get out I ask the device to switch it's audio over to the earpieces embedded in my ears and the podcast resumes once again. I get on the train and get my tablet out to do some development work - which is, of course, already online. I might also fish out a keyboard - which automatically connects as it comes into proximity with the tablet. Later I get off the train and walk to my office. As I walk my steps are being counted by the peripheral device on my wrist where it is collated along with my other health measurements and sent to the cloud. Along my journey something interesting and unexpected happens. I bring out my device to take a photo. But I really want a good quality picture, so I quickly fish out a lens with a full size sensor from my bag, which wirelessly connects to my device and instantly beefs up the optics to professional standards. I take a great picture then continue on. As I get near the office a reminder pops up on my wrist that I had set to go off in that vicinity. Eventually I get to my desk where I put my device on the wireless charging pad as it connects to my keyboard and large displays and I continue the work I started on the train.

The task at hand

One consequence of this more distributed way of working is that the single-(main-)tasking metaphor that the iPhone doggedly champions is allowed to survive while still allowing multiple applications to run and be interactive. The metaphor becomes "one app per device". Each device is typically running one interactive application at a time - for some devices it is the same app at any time (a keyboard, for example). For a more general purpose device, such as a tablet, it may run one app, while a different app runs on the "phone" beside it. But the devices can see each other and documents and other data may be shared between them - probably using real-world metaphors like the "flick" mentioned earlier.

Conversely at any one time two or more devices may appear to be running a portion of the same app - but in truth they will be running their own instances - with tight integration between them.

My vision of the future is one of heterogenous, smart devices - some specialised, some generalised - participating in the fabric of a system that surrounds us - and which tends to recede into our surroundings. The seeds are there - and they're growing. I think the next decade is going to be an exciting and transformative time in technology - perhaps even more so than the last!

Postscript...

I had wanted to publish this post by New Year's Eve (2013) but didn't get time to finish up by then. I'm pushing it now, largely un-edited, to try and keep it relatively seasonal (but I may come back and edit more aggressively yet - it's much too rambling for my liking).

As I was finishing I saw blog post by Dave Addey - which he actually posted back in September - covering very similar material. I haven't had a chance to think how to work it in organically to this post (yet) but didn't want to miss the opportunity to link to it - so I'll do that explicitly here. Go read it now. Go on, I'll wait.

Optional streaming

Catch has a number of macros that allow values of arbitrary types to be streamed into an ostringstream. The canonical example is the INFO macro:

INFO( "There were " << bottles.size() << " green bottles, hanging on the wall" );

This macro builds up a string that will be passed to the next assertion to be included as an annotation. Note that, unlike with a naked ostringstream there is no leading <<. This makes it clean and uncluttered when you just want to log a single value (such as a string), for example:

INFO( "Weirdness" );
The obvious way to do this is for the macro to provide the leading << prior to its argument. Conceptually something like this:
#define INFO( log ) { \
	std::ostringstream oss; \
	oss << log; \
	useTheString( oss.str() ); 
}

This all works quite nicely. But there are a few other macros that use this idiom, too: WARN, SUCCEED and FAIL.

The last two are of interest because the logging behaviour is more of a secondary concern. The primary behaviour is to appear like a passing or failing assertion, respectively, without the need to actually assert on anything. SUCCEED can be useful if you otherwise have no assertions in a test and you don't want to see warnings about it. FAIL is useful if the situation that leads to the failure is not captured in an expression for some reason. It can also be useful to force a test to fail, perhaps as a placeholder. These are useful macros to have available, but they are not often needed in practice. So when they are it's nice to be able to annotate their useage inline - hence the streamed argument.

This is all well and good. But I've found there are still enough cases where I don't want to annotate that having to pass an empty string or make something up is a little annoying. I also use a similar idiom in other projects where it would be nice to be able to make the stream completely optional.

This is not as easy as it sounds, though. The first, and most obvious, issue is that this requires support for variadic macros. Catch has made use of variadic macros, where available, for some time now. In theory they are available to any C++11 compiler. In practice most, if not all, compilers that support any reasonable chunk of C++11 support variadic macros - and most supported them as an extension even before that. That's certainly true of Visual C++, GCC and Clang.

The technically more interesting problem, though, is dealing with that initial <<. Remember the first << is being supplied inside the macro. It will still be there even if the caller does not supply an argument to the macro. If we wrote FAIL the same way we presented INFO earlier (but with variadic macros) it might look something like this:

#define FAIL( ... ) { \
	std::ostringstream oss; \
	oss << __VA_ARGS__; \
	notifyFail( oss.str() ); \
}
... which, with no argument provided, would expand to...
{ 
	std::ostringstream oss; 
	oss << ; 
	notifyFail( oss.str() ); 
}

Do you see the problem? With nothing following the << this will not compile.

So do we need a different operator? What properties would we need? It seems we'd need an operator that comes in two forms: a binary operator that allows us to capture an argument, and a unary operator that allows us to omit the argument. Furthermore the binary form must not require its argument to be enclosed in any sort of brackets. Finally it must have higher precedence than << so we can switch over to normal stream insertion at that point.

That's a long list. Does such an operator exist? Fortunately there's not just one but two such operators to choose from! + and -. The only slight hitch is that the unary form is right-to-left associative, whereas the binary form is left-to-right. So how can we work these in?

Let's pick one of the operators. I've gone with +, but I don't think there is any advantage either way. Because unary + is right-to-left associative it needs to prefix something. So we can't use it at the start of our streaming expression. We can, however, use it at the end. Then we'll need an object to apply it to. The object doesn't actually need to do anything else. I've gone with this implementation of StreamEndStop in Catch:

struct StreamEndStop {
    std::string operator+() {
        return std::string();
    }
};
With this definition the expression, +StreamEndStop() now yields an empty string - which is idempotent with a stringstream. Which means we can write:
{
	std::ostringstream oss; 
	oss << +StreamEndStop();
	notifyFail( oss.str() ); 
}
And oss.str() evaluates to an empty string. Perfect. But what about when we do stream something? Well that would expand to:
{
	std::ostringstream oss; 
	oss << something +StreamEndStop();
	notifyFail( oss.str() ); 
}
... where something could be a string or variable or literal of any type. So we need some way for the expression:
something +StreamEndStop()
to yield the value of something. That's where the binary form of operator+ comes in:
template<typename T>
T const& operator + ( T const& value, StreamEndStop& ) {
	return value;
}
Now, whether we supply nothing, a single value or multiple values joined by <<s we'll end up with a stringstream containing what we expect. The relevant bit of code in Catch actually looks like this:
Catch::ExpressionResultBuilder( messageType ) \
	<< __VA_ARGS__ \
	+::Catch::StreamEndStop()
which yields an ExpressionResultBuilder that gets passed on elsewhere. This is all protected by CATCH_CONFIG_VARIADIC_MACROS. Otherwise it falls back to:
Catch::ExpressionResultBuilder( messageType ) << log
So a lot of work to save a few explicit empty strings, but sometimes it's the little things.

The Minimum Viable Wiki

Documentation is usually considered a necessary evil. The truth is probably closer to it being necessary and evil. For most of us it's one of those things we feel we should have more of (or any at all) but never have the time. And how much documentation do we need anyway?

Most project documentation is worse than useless. Why? For the same reason that most comments in code are useless-to-dangerous - only more so. Documentation, when written, is rarely kept up-to-date. It is rarely complete. It was often never "true" to start with.

In short the docs are just not reliable

Oh, there are exceptions, of course. API docs generated from source code, for example. They will be consistent, right? Perhaps. Assuming they are kept maintained within the source. It is still possible for them to lie.

If we are Agile we don't do documentation, do we? After all the manifesto states:

Working software over comprehensive documentation

Of course it doesn't say we don't value documentation - just that we value working software more. And even then it talks about comprehensive documentation.

There is the idea of "Just Enough Documentation". I hear people talk about this and see it referred to, along with some personal interpretation. But I have yet to see a definitive write-up of this idea (please let me know if you know of one). Perhaps this is meant to be deliberately ironic? In any case, my understanding is that this relates to the sort of documents that are often seen as deliverables of a project - for example a requirements doc.

I'm not going to talk so much about those, although the principle I'll describe may often be applied. I want to concentrate on the internal, developer-oriented, documentation that describes things such as how a project is organised, what bits do what, what to expect in certain situations etc. This is often captured in a wiki that is specific to a team.

This sort of documentation is often seen as unnecessary in a Agile team. This is where face-to-face communication between team members is sufficient, isn't it?

That view seems to be borne out on projects that do have a wiki. The wiki is rarely complete or up-to-date. It quickly falls out of use altogether, or is approached with trepidation as it is as likely to mislead as inform. If it is kept up-to-date it is likely due to a Mandate From Above, which leads to more time invested in its maintenance than is worth it.

So is there a way this can be made to work? Can it ever be worth it? Why would we need it at all?

Well that first depends on your project and team. Maybe the conversations alone really are sufficient. My experience is that this reaches a limit pretty quickly. We need something. How many times have you, or someone on your team, wasted a day looking for something or going down a wrong path, only for someone else to then say, "oh, that's over here", or, "that's done differently for x reason". Yes the conversation was had - but time was already wasted

So the problem is knowing when and who to ask - especially for things you would have thought were "obvious"

In terms of the Four stages of competence this is about moving from Unconscious Incompetence (you don't know what you don't know) to Conscious Incompetence (you at least know what things you don't know). This is much more valuable than it sounds at first! If you don't know that adding a setting to a config file dumps the information you need to a log you won't spend six hours writing a whole chunk of ad-hoc logging code to do the same thing, for example. Instead you'll remember, "there was something about being able to enable logging in a config file". Now you know to ask the question and finding the answer should be quick and accurate.

Having a full specification, perhaps along with examples, of every attribute that could go in the config file might sound better. And if that config file was meant for end-user customisation you'd probably need that. But if it's some internal thing then any attempt at such documentation is likely to be incomplete and/ or out of date - if written at all

But a line that says, "logging is configurable", somewhere that people will read leaves out those noisy, unstable, details and simply informs you of what is possible. You now know what you don't know (i.e. how to enable logging), whereas before you didn't even know it was possible. Details can be asked for.

So my recommendation for maintaining team information on a wiki is to stick to the briefest possible comment that moves the reader from Unconscious to Conscious Incompetence. Unless you really need it avoid the level of detail that will leave it unmaintained and incomplete.

Injecting Singletons in Objective-C Unit Tests

I've promised to write this up a few times now. As I've just given another talk that covers it I thought it was time to make good on that promise.

The topic is the use of singletons in UIKit (and AppKit) and how that makes code using them hard to test. These APIs are riddled with singletons and you can't really avoid them. In case you need convincing that singletons are problematic take this contrived function:

NSString* makeWidget() {
    NSString* colour = 
        [[NSUserDefaults standardUserDefaults] stringForKey: @"defaultColour"];
    return [colour stringByAppendingString: @"Widget"];
}

NSUserDefaults is a singleton - the sole instance of which is returned when you call standardUserDefaults.

Monster1

A perturbing problem

Now consider how we might test this code. Obviously in an example this trivial there are various ways we could change the code to make the problem go away. Consider this a scaled down example of a problem that may be deeper in the code - perhaps a legacy code-base (or even some third party library!).

A naive test might set the "defaultColour" key in NSUserDefaults prior to calling makeWidget(). The problem with that is that the environment is left in a changed state after the test. Subsequent tests may now pick up a different value if they use NSUserDefaults. Worse: NSUserDefaults is backed by persistent storage that can potentially leave your whole user account in a changed state!

So, at the very least, we should restore the prior value at the end of the test. This leads to further problems: If the test fails, or an exception is otherwise thrown, the clean-up would not be called. So we'd need to wrap it in a @try-@finally too. Then, can we be sure we know what value to restore it to. It's probably nil - but if it's not the environment is still in a different state. So we should capture the prior value first and hold it in a variable.

Now what if you need to set more than one value. Or you change the keys used. We're starting to do a lot of bookkeeping just to compensate for the fact that a singleton is being used. Not only is it ugly but it's increasingly error prone.

Better if we can avoid this in the first place. If we have the option - prefer to pass dependencies in - rather than have your code reach out to these Dependency Singularities. In our example either pass in the default colour, or failing that, pass in NSUserDefaults.

NSString* makeWidget( NSUserDefaults* defaults ) {
    NSString* colour = [defaults stringForKey: @"defaultColour"];
    return [colour stringByAppendingString: @"Widget"];
}

At first this doesn't seem to buy us much. We still need an instance of NSUserDefaults. Even if we alloc-init it we'll get a copy of the global one. That's better but we'd still be dependent on the environment and have to take steps to compensate. And in other cases we may not even have that option

Monster2

If you can't make it - fake it!

We might not be able to create completely fresh instances of NSUserDefaults - but we can create instances of a stand-in class. Due to Objective-C's dynamic nature we don't even need to subclass - and we only have to implement the methods that are actually called - in this case stringForKey:. We could do that with a Mock Object. Or we can build our own Fake. Let's assume you've written a Fake called FakeUserDefaults, which contains an NSMutableDictionary, a means to populate it (perhaps via an initialiser) and an implementation of stringForKey: that looks the key up in the dictionary. Now we can test like this:

TEST_CASE() {
    id defaults =
        [[FakeUserDefaults alloc] initWithValue: @"Red" 
                                         forKey: @"defaultColour"];
    REQUIRE_THAT( makeWidget( defaults ), StartsWith( @"Red" ) );    
}

Great. That seems to tick all the boxes. We have complete control of the default value and we haven't perturbed our environment. No clean-up is required at the end of the test (not even memory, if we're using ARC)

Assuming you have the freedom to change the code under test, here, of course. If makeWidget() was buried deep in some legacy code, for example, it may not be feasible to make such a change (yet). Even if we can make the change it can be useful to be able to put the test in first to watch your back while you change it. If we need to leave the call to [NSUserDefaults standardUserDefaults] baked into the code under test for whatever reason what else can we do?

Monster3

To catch a singleton we must think like a singleton

What we'd like is that, when standardUserDefaults is called on NSUserDefaults deep in the bowels of the code under test, it returns an instance of our fake class instead - but only while we're testing. Again, due to Objective-C's dynamic nature we can achieve this. But it starts to get messier. It involves gritty low-level functions from objc/runtime.h. Can we package that away somewhere?

Of course we can! Enter TBCSingletonInjector. I've uploaded the code to GitHub, but there's actually not much to it. It exposes one public (class) method:

+(void) injectSingleton: (id) injectedSingleton
              intoClass: (Class) originalClass
            forSelector: (SEL)originalSelector
              withBlock: (void (^)(void) ) code;

The usage is best explained by example:

TEST_CASE() {
    id defaults =
        [[FakeUserDefaults alloc] initWithValue:@"Red" forKey:@"defaultColour"];

    [TBCSingletonInjector injectSingleton: defaults
                                intoClass: [NSUserDefaults class]
                              forSelector: @selector(standardUserDefaults)
                                withBlock: ^ {
            REQUIRE_THAT( makeWidget(), StartsWith( @"Red" ) );
        } ];
}

Magic! How does it work? It uses a technique known as "method swizzling" (Ruby or Pythonists know it as "monkey patching"). In short we replace a singleton accessor method (such as standardUserDefaults) with one we control (actually another, not otherwise exposed, class method of TBCSingletonInjector). More specifically we swap the two implementations. This is so we can swap them back again when we're done. Then we call the code block - all within a @try-@finally - so no matter what happens we always restore everything to its previous state.

What does the method we swap in do? It returns a global variable.

Wait, what? I thought globals and singletons were basically the same thing? Aren't we out of the frying pan into the fire?

In the war against singletons we must fight them with singletons! Well it's not all bad. This global is only in our test code and we have full control over it. It gets set to our "injected" singleton instance (and set back to nil at the end). It's not perfect - we can only use this implementation to handle one singleton at a time. I've not yet needed to handle more than one but I daresay the implementation could be extended to handle it.

Keep it clean

Since we've hand rolled our own fake class here (FakeUserDefaults) we can tidy things up further if we encapsulate the use of the singleton injector within it. Just adding a method like this should do the trick:

-(void) use:(void (^)(void) ) code
{
    [TBCSingletonInjector injectSingleton: self
                                intoClass: [NSUserDefaults class]
                              forSelector: @selector(standardUserDefaults)
                                withBlock: code ];
}

Now the test code becomes:

    FakeUserDefaults* defs = 
        [[FakeUserDefaults alloc] initWithValue: @"Red" 
                                         forKey: @"defaultColour"];
    [defs use:^{
            REQUIRE_THAT( makeWidget(), StartsWith( @"Red" ) );
         }];

Or, if you prefer, even:

    [[[FakeUserDefaults alloc] initWithValue: @"Red" 
                                      forKey: @"defaultColour"]
    	use:^{
            REQUIRE_THAT( makeWidget(), StartsWith( @"Red" ) );
         }];

Not too bad, really. But, still, prefer to avoid the singletons in the first place if you have the option.

Monster4

Mocking a monster

Rather than hand rolling a Fake you might prefer to use a Mock object too. I've found OCMock does the job well enough. I'm sure other mocking frameworks would do so at least as well. I prefer to use mocks when I want to test the behaviour, though. In this context that might equate to testing that some code under test sets a value in a singleton (e.g. sets a key in NSUserDefaults). The Singleton Injector works just as well for that, of course.

So there we have it. When you really have to deal with the beast you now have some tools to do so. If you do it please consider only doing so until you are able to replace the singularity with something better behaved instead.