RSS Feed

Planet Code

May 17, 2012

Paul Grenyer: F# eye for the C# guy

Jon Jagger: the cat in the hat

It's not the event that counts, it's your reaction to the event.

I've got to admit it's getting better.

They're horrible. Who want's some?

This day has been amazing. Thank you.

Two people can't drive at the same time.

I think there's something wrong with your brakes. When was the last time you had them checked?

There's no way to explain this to mom.

Please keep your hands and feet in the KWAN at all times.

Jon Jagger: cyber-dojo Skillsmatter podcast

I had the pleasure of running a CyberDojo at Skillsmatter's 2 day Progressive Java tutorial last week. The session was video'd and is available here.

Jon Jagger: building a rails 3 turnkey image

# cd ~
# apt-get purge ruby-enterprise

# cd ~
# wget http://pyyaml.org/download/libyaml/yaml-0.1.4.tar.gz
# tar xzf yaml-0.1.4.tar.gz
# cd yaml-0.1.4
# ./configure
# make
# make install

# cd ~
# wget http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.3-p125.tar.gz
# tar xzf ruby-1.9.3-p125.tar.gz
# cd ruby-1.9.3-p125
# ./configure --disable-install-doc
# make
# make install

# cd /var/www
# git clone https://JonJagger@github.com/JonJagger/cyberdojo
# chown -R www-data cyberdojo
# chgrp -R www-data cyberdojo
# cd cyberdojo
# gem update --system
# gem update --no-rdoc
# gem install bundle --no-ri --no-rdoc
# bundle install

# cd /etc/apache2/sites-enabled
# sed s/railsapp/cyberdojo/ <railsapp >cyberdojo
# rm railsapp
# cd /etc/apache2/sites-available
# sed s/railsapp/cyberdojo/ <railsapp >cyberdojo
# rm railsapp
# cd /etc/apache2/conf
# sed s/railsapp/cyberdojo/ <railsapp.conf >cyberdojo.conf
# rm railsapp.conf

# cd ~
# apt-get update
# apt-get install libcurl4-openssl-dev
# cd /var/www/cyberdojo
# gem install passenger --no-ri --no-rdoc
# passenger-install-apache2-module

500 ActionView::Template::Error (application.js isn't precompiled)

config.assets.compile = true

ActionView::Template::Error (Could not find a javascript runtime: See https://github.com/sstephenson/execjs for a list...)

gem 'therubyracer'

# cd /var/www/cyberdojo
# gem update --no-rdoc --no-ri
# bundle install
# /etc/init.d/apache2 restart

Jon Jagger: requiem for C head cam

At the ACCU 2012 conference Uncle Bob gave his excellent keynote Requiem for C. Skillsmatter video'd the whole keynote and it will be available soon. As an experiment Uncle Bob gamely agreed to wear a Go Pro 2 head camera whilst giving the keynote! So here, possibly for the first time ever, are a few rough clips of what it's like to give a keynote.

Jon Jagger: Emil's cyber-dojo

I ran a Mastering Agile Practice tutorial with Kevlin Henney at the recent Scandinavian Developer Conference in Sweden. Emil Jönsson attended and liked Cyber-Dojo so much he ran one at his company afterwards. He says:

Just wanted to let you now that the Cyber-Dojo session I organised at work last week went really well. We did the leap year kata in Java and we were eight in total, so we ended up working in four pairs. It was fun to hear all the interesting discussions taking place and it was perfect how we could jump between the different solutions of the pairs afterwards when talking about the code. The feedback after the dojo was all positive and my colleagues liked the Cyber-Dojo way of practicing.

I hope this will become a reoccurring event. Thanks for the excellent work creating the Cyber-Dojo.

Jon Jagger: cyber-dojo in the cloud

Jon Jagger: big cyber-dojo in Beijing

Tim Pizey: Networking, and hence sshd, does not start until after login on Ubuntu

I updated my Ubuntu system and restarted it, as prompted, and left the office.

When I got home I could not ssh into my work machine.

This was because I had not logged into the machine and so networking and hence sshd had not started.

To get networking started before login add the following to /etc/network/interfaces

auto eth0
iface eth0 inet dhcp

May 16, 2012

Paul Grenyer: Jez Higgin's Long Lost Brother?

May 14, 2012

Christof Meerwald: La bohème @ Royal Opera House

Just been to a performance of La bohème at the Royal Opera House.

Allan Kelly: 10 Things to make you Agile adoption successfull

One of the closing slides in my Agile Foundations course includes a quote from Ken Schwaber saying that only 30% of teams who attempt Scrum will be successful. What I find interesting about this quote is that it aligns with many other change management studies. Researchers like Harvard Professor John Kotter regularly say 70% of major change efforts fail.

On his blog Ken Schwaber says he doesn’t remember this and instead suggests only 30% will become “excellent development organizations.”

Either way, the prognosis isn’t optimistic. A few months ago, at the end of the course, someone asked the obvious question, a question so obvious I wonder why nobody has asked it before: “What can we do to ensure that we are in the 30% who make it?”

Given that I had the Managing Director, the Director of Technology and most of the technology team in the room it was an excellent opportunity to set the change agenda. And I fluffed it, despite having written a book on the subject I didn’t have a quick answer to hand. But it set me thinking: “What are the 10 things a team can do to make Agile (any flavour) stick?”

Here then is that list, the team in the room will recognise the first three, it was after that that I had to think.

1) Use a physical board: over the last year I have become convinced that the single biggest difference between teams which successfully adopt Agile working and those who try, fail, or end up stuck is the use of an actual physical board.

I know some teams find this difficult, I know some teams are distributed, I know there is technology out there to do this for you but I stand by my point. If you can make it physical, in a place where many, if not all, can see, then you are more likely to succeed.

2) Start collecting and using statistics and other data: velocity, burn-down, bugs identified, bugs logged, etc. etc. Metrics have a bad name in software development - rightly in most cases. But that only means that have been badly collected, managed and used, it doesn’t mean they aren’t useful. At the very least measure your velocity and create a burn-down chart or cumulative flow diagram of the work to do or arising.

3) Engage a coach/consultant: at the risk of being accused of trying to make work for myself I should say you can adopt Agile all by yourself. You can read the books, you can experiment, you can go on courses. But doing it without help makes the whole process slower and increases the risk that you won’t make it to the 30%.

Personally, I find it difficult to know just how an Agile Coach differs from an Agile Consultant. What ever you call the role you want someone who can:

• Provide advice on which practices and process to adopt, and how to best adopt them
• Offer examples of what they have seen work, and not work, elsewhere, and how other team tackle similar issues
• Observe, examine, query and challenge your thinking on what you are doing
• Challenge your thinking and point out opportunities and idea that you haven’t seen yet

Derek Jones: Undefined behavior is a design decision

Every few years or so some group of people in the C/C++ community start writing about the constructs specified as having undefined behavior in those languages. A topic that always seems to be skipped is why a language committee would choose to specify that the behavior in a particular case was undefined.

A quick refresher for readers on the definition of Undefined behavior, from the C Standard: “behavior, upon use of a nonportable or erroneous program construct or of erroneous data, for which this International Standard imposes no requirements”. The two key features are that the behavior applies when an error has occurred and any behavior whatsoever is permitted after one of these errors occurs. Examples of constructs that have undefined behavior are divide by zero, the result of an arithmetic operation on a signed value not being representable in its type (i.e., overflowing) and indexing an array outside of its defined bounds.

The point to note about all undefined behaviors is that the C/C++ language committee could have chosen to specify the behavior that a conforming implementation is required to support. Some language specifications do attempt to explicitly define the behavior for all constructs, e.g., Java, while other languages have a smaller set of undefined behaviors (e.g., Ada, which uses the term Bounded error instead of undefined behavior; there are 35 of them in Ada 2005). To understand why languages take these different approaches we need to look at the language design aims.

The design aims of C included it being implementable for any processor and for the generated code to be efficient (I’m not sure to what extent these might still be major design aims for C++). Computing systems come in all shapes and sizes, some with hundreds of bytes of memory and others with gigabytes, some raise exceptions when certain operations occur while others set processor status flags and others don’t do anything special.

A willingness to accept whatever behavior happens to occur, in an error situation, is the price that has to be paid for efficient execution on a wide range of disparate processors. The C/C++ designers were willing to pay this price while while the Java designers were not, with the Ada designers willing to tolerate less variability than C/C++.

Undefined behavior need not be nasty behavior, an implementation could chose to generate a helpful message or try to recover from it.

There are C tools and compilers (when certain options are specified) that check, at runtime, for various kinds of undefined behavior. I am in the minority in having Boundschecker installed as my default C compiler (as the name suggests it checks that array and pointer accesses to an object are within the defined bounds); for reasons I don’t understand few C/C++ developers are willing to use tools like this. For production code I use a non-boundschecking compiler; I don’t know whether Ada programs are tested with the mandated bounds checking switched on and then have it switched off for the production version (this is what Pascal developers do, in my experience). Of course Java developers have no choice but to permanently live with checking turned on.

The number of companies that make a living selling runtime checking tools is a small percentage of the number of companies based on selling static analysis tools. There continues to be a steady stream of runtime checking tools appearing and quickly disappearing, but until a developers start being sent to jail for faults in their code I don’t foresee the market growing.

Optimizations to figure out when code need not be generated to perform a bounds check because the access is known to be within bounds is an active research area. These days the performance penalty is not so much executing the checking instructions but the disruption to the instruction pipeline caused by the branches that might be taken (if the bounds check fails).

The cost of all the checking required by Java is that the minimal permitted configuration requires at least 256K of memory (Oracle’s K virtual machine, used by the Java Micro Edition which is intended for embedded systems, also makes floating-point optional and allows implementations some freedom in how some constructs are handled). So the Java motto really out to be “Write once, run anywhere with at least 256K and don’t depend on floating-point”.

I have heard stories of Ada code being liberally scattered with various forms of unchecked (how the developer tells the compiler not to do any runtime checking) but have not seen any empirical analysis (a study of goto usage in Ada did not have any trouble finding plenty of uses to analyze).

Jon Jagger: ACCU conference charity bookstall

A big thank you to the excellent people attending the ACCU 2012 conference who raised £645.77 plus a €10 note plus a very small silver coin of unknown origin! This will be split equally between Paws with a Cause and The Autism Trust.

Claysnow: "Common Objections to TDD" coming to Agile North 2012

If you missed my talk at ACCU 2012, you now have another chance. I'll be presenting it at Agile North (Preston) on June 29th.

There is a chance that it will be recorded, which would be great, because the slide deck doesn't really work on its own.

Permalink | Leave a comment  »

May 13, 2012

Paul Grenyer: Dark Shadows

Paul Grenyer: Avengers Assemble

May 08, 2012

Andy Balaam: Generalising tail call optimised C++

This series: Lightning talk, Explanation, Performance, Generalisation.

In previous posts I discussed the construction of some C++ that does the same job that the tail call optimisation does in some other languages. The example code given showed the case where we know that every function in the recursion will take two long integer parameters, and return a long as well.

In fact, it is perfectly possible to generalise this code to cover more complex cases. Fortunately, the trampoline function doesn’t need to know the arguments taken by the functions being called, only the return value. It looks like this:

template<typename RetT>
const RetT trampoline_templ(
    std::auto_ptr< IAnswer<RetT> > answer )
{
    while( !answer->finished() )
    {
        answer = answer->tail_call()();
    }
    return answer->value();
}

So we can call this with the type of the return value as our template parameter, and supply an object which satisfies the IAnswer<RetT> interface:

template<typename RetT>
class IAnswer
{
public:
    virtual const bool             finished()  const = 0;
    virtual const ICallable<RetT>& tail_call() const = 0;
    virtual const RetT             value()     const = 0;
};

where ICallable looks like this:

template<typename RetT>
class ICallable
{
public:
    typedef std::auto_ptr< IAnswer<RetT> > AnswerPtr;
    virtual AnswerPtr operator()() const = 0;
};

The concrete classes that implement these interfaces need to know the types (and number) of the arguments, but that’s ok because they only get instantiated by code that would otherwise (in the standard, non-tail-call recursion case) call the functions themselves. In the toy case we are using of repeatedly adding up numbers to multiply by two, the outer function looks like this:

const long times_two_tail_call_templ( const long n )
{
    typedef Answer2 AnswerType;

    return trampoline_templ(
        AnswerType::newFn(
            times_two_tail_call_impl, 0, n, 0 )
    );
}

and the inner one looks like this:

std::auto_ptr< IAnswer<long> > times_two_tail_call_impl(
    const long acc, const long i )
{
    typedef Answer2<long, long, long> AnswerType;

    if( i == 0 )
    {
        return return AnswerType::newAns( acc );
    }
    else
    {
        return AnswerType::newFn(
            times_two_tail_call_impl,
            acc + 2, i - 1, 0 );
    }
}

Both of the above use static convenience methods newFn and newAns that I have defined on Answer2 to create smart pointers to newly-allocated Answer2 objects. newAns creates an Answer2 that contains a final answer, and newFn creates an Answer2 specifying another function (with arguments) to call.

Answer2 looks like this:

template<typename RetT, typename Arg1T, typename Arg2T>
class Answer2 : public IAnswer<RetT>
{
private:
    typedef FnPlusArgs2<RetT, Arg1T, Arg2T> FnArgs;
    typedef std::auto_ptr< IAnswer<RetT> > AnswerPtr;

    const bool finished_;
    const FnArgs tail_call_;
    const RetT value_;

private:
    Answer2( const bool finished, const FnArgs tail_call, const RetT value )
    : finished_( finished )
    , tail_call_( tail_call )
    , value_( value )
    {
    }

    static AnswerPtr newPtr(
        const bool finished, const FnArgs tail_call, const RetT value )
    {
        return AnswerPtr( new Answer2<RetT, Arg1T, Arg2T>(
            finished, tail_call, value ) );
    }
public:
    static AnswerPtr newFn(
        const typename FnArgs::fn_type fn,
        const Arg1T arg1,
        const Arg2T arg2,
        const RetT zero_val )
    {
        return newPtr( false, FnArgs( fn, arg1, arg2 ), zero_val );
    }

    static AnswerPtr newAns( const RetT value )
    {
        return newPtr( true, FnArgs::null(), value );
    }

    virtual const bool    finished()  const { return finished_; };
    virtual const FnArgs& tail_call() const { return tail_call_; };
    virtual const RetT    value()     const { return value_; };
};

and uses FnPlusArgs2, which looks like this:

template<typename RetT, typename Arg1T, typename Arg2T>
class FnPlusArgs2 : public ICallable<RetT>
{
private:
    typedef typename ICallable<RetT>::AnswerPtr AnswerPtr;
public:
    typedef AnswerPtr (*fn_type)( const Arg1T, const Arg2T );
private:
    const fn_type fn_;
    const Arg1T arg1_;
    const Arg2T arg2_;

public:
    FnPlusArgs2( const fn_type fn, const Arg1T arg1, const Arg2T arg2 )
    : fn_( fn )
    , arg1_( arg1 )
    , arg2_( arg2 )
    {
    }

    virtual AnswerPtr operator()() const
    {
        return fn_( arg1_, arg2_ );
    }

    static FnPlusArgs2<RetT, Arg1T, Arg2T> null()
    {
        return FnPlusArgs2<RetT, Arg1T, Arg2T>( NULL, 0, 0 );
    }
};

I have continued with the 2 long arguments, and long return value example here, but with the above code it is possible to construct recursive code using more than one function, and the functions can have different numbers of arguments, and different argument types, so long as they all co-operate to produce an answer of the required type. The Source code for this article includes an example, in the file tail_call_templ_2fns.cpp, of two different functions that call each other recursively, and take different arguments, using the trampoline function and interfaces listed above, and Answer3 and FnPlusArgs3 class templates similar to Answer2 and FnPlusArgs2 shown above. Implementing the N-args case using variadic templates (C++11) or template meta-programming is left as an exercise for the reader.

This more realistic case where the number and types of arguments are not known beforehand forces us to use dynamic memory to store our AnswerN objects, and causes more pointer dereferences and virtual function calls, and these do hurt performance. In tests on my machine, this code ran approximately 10 times slower than the version using only stack memory. Perhaps we C++ programmers should comfort ourselves that many languages supporting tail-call optimisation require dynamic memory, virtual functions and pointer indirection to do absolutely everything.

Paul Grenyer: ACCU Conference 2012

This year’s ACCU Conference, in Oxford as usual, was superb. I enjoyed it immensely and I was only there for two sessions, one of which was my own! There was a huge amount of twitter activity which actually made all the sessions very interesting even though I wasn’t there. My wife enjoyed it too, I had huge difficulty getting her out of the bar! I learnt that I featured in Allan Kelly’s presentation minutes after my picture flashed up on one of his slides. Next year the children will be older and I should be able to go to the entire event.  

The Congruent Programmer
Phil Nash

 Phil Nash is simply Phil Nash. I’d be lying if the fact that he was giving the ACCU member keynote didn’t fill (no pun intended) me with envy. Phil was suffering from throat problems, but he’s so softly spoken anyway that you I didn’t really notice. I could hear everything he was saying at every point. In usual Phil Nash style his Mac toys were also giving him trouble. It appeared he was using his iPhone to move his presentation between slides and it was being more than a little temperamental. The keynote was highly amusing, as usual with lots of subtle references. Hopefully this is the first of many keynotes for Phil.  

Walking Skeleton
Paul Grenyer

I hate putting presentations together and I hate practicing them. However when I get to the conference I really enjoy giving them and this one was no different to the last time (over two years ago). I had a great audience which nearly filled the room. The problem with ACCU audiences is that they’re pretty clued up already and I was acutely aware of this all the way through and felt I was only really filling in the gaps. And of course at the end they found all the holes that other audiences had failed to find! Roll on next year.

 

Allan Kelly: Points based contracts? Just Say No.

With the points-mini-series still fresh in the mind now seems a good time to say publicly something which I’ve been saying privately for a long time.

Avoid points based contracts. i.e. don’t outsource work, or undertake work, on the basis of points - be they story points, abstract points, nebulous units of time or any other name you give them.

I have one client at the moment who wants their software supplier to sign a points based contract, I’ve advised against it. Another client is trying to sell points based contracts to their clients, while they are having some success - I think they rushed in before they had enough data to understand the implications.

Why do I say this? Well three reasons

First is Goodhart’s Law: “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” Put it another way: any measurement metric will change behaviour once it is used for control.

In this context it means: points are very good at measuring story size and team velocity, they can be accurate at predicting when a piece of work will be done. But, if you use them for other purposes - like regulating a contract and making payments - they will change their behaviour. They won’t be so useful for predicting end dates, or for controlling contracts for that matter.

This is a problem many readers will be familiar with from traditional time estimation. Estimates are nominally sought to determine how long a piece or work will take, and thus how much it will cost. But they are also used as a means of targeting and for control. They are used as a proxy for commitment and they are gamed (i.e. changed for specific ends) when they don’t give the time/cost numbers desired. (This is something Esther Derby discusses recently in her blog, Estimating is often helpful, estimates are not.)

In fact, time estimates show the same range of problems present in the corporate budgeting processes and which has given rise to the beyond budgeting movement.

One direct result of Goodhart’s Law in this context is...

Inflation, the second reason to avoid points based contracts: points are subjective, they are not grounded in time, complexity, function point analysis, lines of code or any other objective measurement. They are in fact like a fiat currency: they are worth what you can buy with it. If people don’t believe in it, or believe the value will change then it will change. Check out rational expectations theory if you want to understand why.

Overtime points can devalue with the result that point scores increase. Actually, I believe the free floating nature of points is one of the strongest reasons for using them but in terms of signing a contract it makes them useless.

Most teams I see work in low points: 1, 2, maybe a 5, rarely an 8, they score 10, 12 or maybe 20 an iteration. One team I saw worked in tens, and scored hundreds each iteration. It was like one of those old Space Invaders machines were the last digit was a hard coded “0”.

The team’s project manager finished planning meetings with an call for the team to work harder next iteration, to reclaim the lost time. Iteration on iteration velocity increased. Inflation was rampant.

Finally there is practicality. As the recent posts from myself and Vasco Duarte demonstrate points, there is still a lot of debate over points. Personally I have come to the conclusion that exactly how you run iterations and count points makes a big difference. While if you agree with Duarte you might as well dispense with points and sign story based contracts.

Then there is the team: only the team which will do the work can accurately say how many points a piece of work will, or did, take; and then only when they have experience of doing the work. So you shouldn’t sign a points based contract unless you have the team in place and they have done some of the work.

Even a relaxed interpretation of that last point should lead you to conclude you should only sign a points based contract when the team is experienced in using points and you have historical data. If you feel you must sign a points based contract then only do it when you have data.

Still, I’d rather you didn’t do it in the first place.

May 05, 2012

Christof Meerwald: Cologne

I have added some pictures from my stop-over in Cologne to my photo album.

May 04, 2012

Andy Balaam: JavaScript WTFs Videos

I recorded some videos of my JavaScript WTFs presentation. Here is the first one:

You can get the JavaScript WTFs slides.

Update: all six episodes:

• JavaScript WTFs 1 – The Global Object
• JavaScript WTFs 2 – Equality and Inequality
• JavaScript WTFs 3 – Syntax
• JavaScript WTFs 4 – Reserved words and typeof
• JavaScript WTFs 5 – Arithmetic and parseInt
• JavaScript WTFs 6 – Arrays

Chris Oldwood: Beware the Complacency Unit Testing Brings

One phrase that is always sure to raise the ire of any good honest developer when something breaks is:-

“well, it works on my machine”

This simple statement shows a complete disregard for any other sort of testing that you might need to do to ensure that your feature works correctly and is “done, done” not just almost done. But there is a new kid on the block when it comes to showing how little some people understand about software development that I’m beginning to hear with alarming regularity:-

“well, all the unit tests passed”

It seems that modern development practices have unknowingly created the Silver Bullet that Fred Brooks has always told us never existed! Apparently good unit test coverage and automated refactoring tools means that it’s highly unlikely that any bug would only show up during integration and system testing that those ideas are just old fashioned. Or, if not altogether outdated, then reduced to just a footnote in the product’s testing strategy on the basis that there is so much less value in them than unit testing.

Don’t get me wrong I can understand a genuine mistake caused by a seemingly unrelated change - accidents happen and it could be a fault of the design - but changing the configuration file for a service and then not even bothering to see if it starts up is just laziness. Yes, it does take time and effort to do more extensive testing in your sandbox but the feedback loop could still be fast and you won’t annoy your team mates when you cost them a day’s system testing because of a silly mistake.

The rule of thumb about not checking in code until the unit tests pass was designed to make you think about writing fast tests so that the barriers to testing are as low as possible, it was not expected to be used as a justification for short-circuiting the amount of testing you do.

Chris Oldwood: My “Agile SQL Tweet” Slide

Before I put the PowerPoint slides for my ACCU conference talk on Database Development Using TDD up on my web site I agonised over whether to remove the 3rd slide or not. The slide in question just contains the following tweet from Allan Kelly:-

@chrisoldwood is the only person I know with a convincing Agile SQL story -- @allankelly

I felt that viewing the slide in isolation would only amplify the arrogance which could be attached to it and that wouldn’t have a positive effect. But at the same time I felt I should put up what was shown and accept that perhaps it may also raise the same questions that I had that caused me to put it in the first place.

The slide forms part of the prologue and was originally just a picture of myself to act as a backdrop whilst I spent a couple of minutes describing who I was and what I did as I’m not a database developer per-se, I’m mostly a C#/C++ service layer guy who has slipped into the SQL arena like many similar developers. As a consequence I have tried to work with the RDBMS in the same way that I would approach anything else, by using TDD and abstracting the client from the underlying implementation. So, rather than couple the client and database code tightly with client-side SQL[*] I have kept them decoupled to allow the database to develop (and therefore be refactored) independently where possible - I don’t expose public fields in my types[#] so why directly expose tables?

And then this tweet from Allan Kelly appeared. Until that moment I suspected I was the proverbial square peg trying desperately to fit into a round hole, but maybe, just maybe, I might not be barking mad after all. There are many benefits to TDD (and unit testing in particular) over and above proving the functional correctness of a piece of code, but I don’t understand why more people can’t see that.

So the slide should hopefully provoke a reaction that forces you to question what your database development tools and processes are and whether you’re getting the most out of them. If you are then great, it’s working for you, but if you have that uneasy feeling that you’re stymied at every turn because your database can’t evolve quickly enough then perhaps what you’re about to see (and hear, if you attended) will provide food-for-thought about what you could change.

Prior to this the most ‘positive’ reaction I’ve had to what I’ve been suggesting is:-

“I suppose it makes sense, I’ve just never seen it done like that before”

[*] I’m not just talking about explicit embedded SQL here where the SQL code is a string literal (or as a format string) but also where it is implicit as a by-product of a technology such as LINQ-to-SQL. One way or another the schema is likely to be too tightly coupled to the calling code and that means your database design will be harder to change as will remediating performance problems.

[#] Some would argue that a DTO (Data Transfer Object) does exactly that, but a modern RDBMS is so much more than just a data persistence mechanism. Use its power where it makes sense to.

Andy Balaam: Closures in Scheme

In this series on Scheme: Intro, Basics, Closures.

Here’s a presentation I did recently, on Closures in the Scheme programming language. Closures are the way the environment in which a function was created hangs around with it as long as the function itself exists. They allow many flexible programming styles, and in this presentation I demonstrate how a simple object-oriented class structure can be implemented using closures.

Closures in Scheme

May 03, 2012

Pete Barber: More computing humor

A few more computer/programming related jokes.  This time courtesy of the ACCU general mailing list:

• I heard a good concurrency joke recently, about 10,000 threads. It was a bit contentious... (Ewan Milne)

• A programmer walked into a Bar.  I say a Bar, it was actually a Foo. (Frances Buontempo)

• Two programmers are sitting in a bar discussing static and dynamic languages. The next day they get a visit from the RSPCA. When they ask what it's concerning, the RSPCA man says "We got a tip off that you were using ducks for typing". (Anthony Williams)

• An SEO expert walks into a bar, bars, pub, public house, Irish pub, drinks, beer…. (Anders Knatten) 
• Q: WHY DID THE HACKER GO TO THE BEACH? A: EASY ACCESS TO SHELLS (@yeahok)

May 02, 2012

Chris Oldwood: My ACCU Conference Session - Database Development Using TDD

It’s been a busy few weeks as I’ve been preparing to present at the 2012 ACCU Conference in Oxford. This has become somewhat of a pilgrimage and for the second year running I’ve been accepted as one of the speakers; this time to talk about using TDD to develop databases. This was an excuse to piece together some of the ideas that I’ve been blogging about along with the experiences gained with my current client to present one way that can use tests to drive both the development and design of a database.

Session Details

The talk started by walking you through the pre-requisites needed to embrace TDD, such as your own development sandbox and looked at the primary testing mechanism[*], i.e. unit testing SQL. I also spent a fair bit of time covering the notion of a database’s public interface as I personally feel that the modern RDBMS gives you the tools to build an abstraction layer to help decouple the client from the data model to grant freedom of implementation[#].

Naturally I explained the principles of TDD and did a 20 minute live coding session to show how I would use them to develop a reporting style stored procedure. For this I used my own SS-Unit testing framework and SQL Server Express; switching rapidly between test code & production code. With the main body of the talk in the bag I turned to the opportunities that TDD (although really unit testing) opens up such as continuous integration, continuous deployment and schema refactoring.

The PowerPoint slides are available on my web site here.

Abstract

Just for the record this was the abstract that I submitted...

The modern day RDBMS is a complex product that offers so much more than just data persistence. The SQL language, with its vendor specific variants such as T-SQL, provides the ability to develop code in various forms to read, transform & write that data efficiently. This code requires constant testing right from its inception through its various incarnations until it is finally retired.

TDD is a technique that puts writing those tests at the front of the development process, whether that be because you’re writing new code or changing existing code. The knock-on effect of this approach is that your client-based perspective opens your eyes to potential variations in the implementation, and that is where the second ‘D’ in TDD turns from Development into Design. With a solid automated test suite and Continuous Integration under your belt too the world of refactoring opens itself up so that your database design can safely evolve.

This session looks at applying the same principles and disciplines used in other areas of system development to tame the ever increasing complexity that has arisen from the maturity of the RDBMS.

Photo

The above photo of yours truly comes courtesy of Mark Ridgwell. Here are the rest of his photos for Wednesday.

[*] TDD does not prescribe any particular type of testing but I believe that most people associate TDD with unit testing as it probably forms the lions share. I did point out that one of the benefits of a top-design and implementation is that you can start development with a system test and drill down from there.

[#] ORM tools like Entity Framework actually push you in the other direction which may be exactly what you want. But as I’ve said before, Enterprise culture has a habit of routing around nicely designed service layers and attempting to go straight to the heart of the data.

Derek Jones: EU rules that computer languages cannot be copyrighted

The European Court of Justice has published its decision in SAS v WPL; the title of the press release says it all “The functionality of a computer program and the programming language cannot be protected by copyright”. To summarise the background, World Programming Ltd developed a system that was capable of emulating the input/output behavior of programs written in what the SAS Institute Inc were claiming to be their copyrighted scripting language, along with various file formats.

According to the Court of Justice, “the Court holds that neither the functionality of a computer program nor the programming language and the format of data files used in a computer program in order to exploit certain of its functions constitute a form of expression. Accordingly, they do not enjoy copyright protection.”

This EU ruling is not quiet what it seems. The SAS v WPL case is before the High Court in London and the EU Court of Justice has been asked for advice based on European Law. So the UK dispute has not yet been decided, but given that the UK is signed up to adhere by EU laws people who know about the legal stuff seem to think the High Court in London will follow the EU ruling. Assuming this, then…

This ruling is not just bad news for SAS, it is also bad news for their competitors. Competition is likely to lead to better/cheaper products for users of the SAS language, resulting in less incentive for them to move to an alternative (the R language included; incidentally what exactly are The R Foundation for Statistical Computing claiming copyright over in that notice that pops up when R is started?)

The Oracle vs. Google Java API lawsuit involves similar territory. There are plenty of details over at Groklaw and I’m not going to go there.

This ruling makes it much more likely that behave-alike implementations of more ‘corporate languages’ will be created, at least in Europe. Previously the threat of a lawsuit would have been enough to deter most people, irrespective of whether what they wanted to do was legal or not.

What languages might we see implemented any time soon? The one that immediately springs to mind for me is Mathematica, which is the leader in its field and a fork of Maxima that supported the Mathematica language would move it out of the ghetto. Octave and Matlab are already very close, so no change there.

I imagine there are corporate languages scattered over every conceivable application domain. A lot of these domains will be sufficiently specialized that there is a very low probability of somebody creating an open source implementation; if it looks like there is money to be made it has become more likely that an alternative commercial implementation will be created.

It looks like being a compiler writer is back as flavor of the month again

Tim Pizey: Send me the DHCP allocated IP address on boot

Occasionally I reboot, occasionally I get allocated a new IP address.

Create /etc/init.d/sendIpToTimP :

#!/bin/sh                                                                                                                                      
ifconfig eth0 \ 
 |grep "inet addr" \
 |cut -f2 -d":" \
 |cut -f1 -d" " \ 
 | mail timp@paneris.org -s "Work IP address" 

 update-rc.d sendIpToTimP defaults

Tim Pizey: Setup Postfix to use Gmail as a smarthost

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_sasl_tls_security_options = noanonymous
html_directory = /usr/share/doc/postfix/html

 emacs /etc/postfix/sasl_passwd # create password file

smtp.gmail.com GMAIL_USER_NAME@gmail.com:GMAIL_USER_PASSWORD

 postmap hash:/etc/postfix/sasl_passwd
 /etc/init.d/postfix reload
 sudo apt-get install mailutils 

May 01, 2012

Claysnow: "Common objections to TDD" slide deck, now uploaded

Last week I presented my session at ACCU 2012 to a packed audience in an overheated room. Here is the slide deck

Permalink | Leave a comment  »

April 30, 2012

Andy Balaam: Performance of tail call optimised C++

This series: Lightning talk, Explanation, Performance, Generalisation.

After I wrote a version of tail-call optimised code in C++ I became interested in its performance relative to normal recursion. The tail call version can process arbitrarily large input, but how much do you pay for that in terms of performance?

Recap: there are 4 different versions of our function, called times_two. The first, “hardware”, uses the “*” operator to multiply by 2. The second, “loop” uses a for loop to add up lots of 2s until we get the answer. The third, “recursive”, uses a recursive function to add up 2s. The fourth, “tail_call” is a reimplementation of “recursive”, with a manual version of the tail call optimisation – see the original article for details.

Let’s look first at memory usage. Here is the stack memory usage over time as reported by Massif of calling the four functions for a relatively small input value of 100000:

The recursive function uses way more memory than the others (note the logarithmic scale), because it keeps all those stack frames, and the tail_call version takes much longer than the others (possibly because it puts more strain on Massif?), but keeps its memory usage low. Let’s look at how that affects its performance, for different sizes of input:

For these much larger input values, the recursive and tail_call functions take similar amounts of time, until the recursive version starts using all the physical memory on my computer. At this point, its execution times become huge, and erratic, whereas the tail_call function plods on, working fine.

So the overhead of the infrastructure of the tail call doesn’t have much impact on execution time for large input values, but it’s clear from the barely-visible green line at the bottom that using a for-loop with a mutable loop variable instead of function calls is way, way faster, with my compiler, on my computer, in C++. About 18 times faster, in fact.

And, just in case you were wondering: yes those pesky hardware engineers with their new-fangled “*” operator managed to defeat all comers with their unreasonable execution times of 0 seconds every time (to the nearest 10ms). I suppose that shows you something.

Andy Balaam: Tail call optimisation in C++

This series: Lightning talk, Explanation, Performance, Generalisation.

Some programming languages make recursive programming more practical by providing the tail call optimisation. For a tiny talk at the recent ACCU conference I looked at how we might do something similar in C++.

Update: Source code for this article is available.

Multiplying by two

Here’s a toy problem we will use as our example.

Imagine for a second that you want to write a function that multiplies a number by two. OK, we can do that:

long times_two_hardware( long value )
{
    return value * 2;
}

Now imagine that you don’t have the “*” operator. We’re going have to use “+”:

long times_two_loop( long value )
{
    long ret = 0;
    for ( long i = 0; i  value; ++i )
    {
        ret += 2;
    }
    return ret;
}

(Obviously, this is just a silly example designed to be easy to follow.)

Now imagine that you read somewhere that state was bad, and you could always replace a loop with recursion. Then you might get something like this:

long times_two_naive_recursive( long value )
{
    if ( value == 0 )
    {
        return 0;
    }
    else
    {
        return 2 + times_two_naive_recursive( value - 1 );
    }
}

This is fine, but what happens when you run it for a large input?

$ ulimit -S -s 16
$ ./times_two naive_recursive 100000
Segmentation fault

Note that I set my stack size to be very small (16K) to make the point - actually, this will run successfully for very large arguments, but it will eat all your memory and take a long time to finish.

Why does this fail? Because every time you call a function, the state of the current function is saved, and new information is pushed onto the stack about the new function. When you call a function from within a function multiple times, the stack grows and grows, remembering the state all the way down to the place where you started.

So is programming like this useless in practice?

Tail call optimisation

No, because in several programming languages, the compiler or interpreter performs the "tail call optimisation".

When you call a function from within some other code, you normally need the state of the current code to be preserved. There is a special case where you don't need it, though, and this is called a tail call. A tail call is just the situation where you call a function and immediately return its return value as your return value. In this case, we don't need any of the state of the current code any more - we are just about to throw it away and return.

The tail call optimisation throws away this unneeded state before calling the new function, instead of after.

[In practice, in compiled code, this involves popping all the local variables off the stack, pushing the new function parameters on, and jmping to the new function, instead of calling it. This means that when we hit the ret at the end of the new function, we return to the original caller, instead of the location of the tail call.]

Many recursive functions can be re-cast as tail-call versions (sometimes called iterative versions). The one we're looking at is one of those. Here is the tail-call version:

long times_two_recursive_impl( long total, long counter )
{
    if ( counter == 0 )
    {
        return total;
    }
    else
    {
        return times_two_recursive_impl(
            total + 2, counter - 1 );
    }
}

long times_two_recursive( long value )
{
    return times_two_recursive_impl( 0, value );
}

It consists of an outer function times_two_recursive which just hands off control to the inner function times_two_recursive_impl. The inner function uses a counter variable and calls itself recursively, reducing that counter by one each time, until it reaches zero, when it returns the total, which is increased by 2 each time.

The key feature of this implementation is that the recursive function times_two_recursive_impl uses a tail call to do the recursion: the value of calling itself is immediately returned, without reference to anything else in the function, even temporary variables.

So, let's see what happens when we compile and run this:

$ ulimit -S -s 16
$ ./times_two recursive 100000
Segmentation fault

Did I mention that C++ doesn't do the tail call optimisation?*

* Tail call optimisation isn't in the C++ standard. Apparently, some compilers, including MS Visual Studio and GCC, do provide tail call optimisation under certain circumstances (when optimisations are enabled, obviously). It is difficult to implement for all cases, especially in C++ since destruction of objects can cause code to be executed where you might not have expected it, and it doesn't appear to be easy to tell when a compiler will or will not do it without examining the generated assembly language. Languages which have this feature by design, like Scheme (and D?) can do it more predictably.

So how would we write code that is tail call optimised in C++? Possibly of more interest to me personally: if we were generating C++ as the output format for some other language, what code might we generate for tail call optimised functions?

Tail call optimised C++

Let's imagine for a second we have some classes, which I'll define later. FnPlusArgs holds a function pointer, and some arguments to be passed to it. Answer holds on to one of 2 things: either a FnPlusArgs to call later, or an actual answer (return value) for our function.

Now we can write our function like this:

Answer times_two_tail_call_impl( long acc, long i )
{
    if ( i == 0 )
    {
        return Answer( true, null_fn_plus_args, acc );
    }
    else
    {
        return Answer(
            false,
            FnPlusArgs(
                times_two_tail_call_impl,
                acc + 2,
                i - 1
            ),
            0
        );
    }
}

long times_two_tail_call( long n )
{
    return tail_call( Answer(
        false,
        FnPlusArgs( times_two_tail_call_impl, 0, n ),
        0 ) );
}

This has the same structure as times_two_recursive, if a little more verbose. The important point to note, though, is that times_two_tail_call_impl doesn't call itself recursively. Instead, it returns an Answer object, which is a delegate saying that we have more work to do: calling the provided function with the supplied arguments.

The Trampoline

All we need now is some infrastructure to call this function, and deal with its return value, calling functions repeatedly until we have an answer. This function is called a "trampoline", and you can sort of see why:

long trampoline( Answer answer )
{
    while ( !answer.finished_ )
    {
        answer = answer.tail_call_();
    }
    return answer.value_;
}

While the answer we get back tells us we have more work to do, we call functions, and when we're finished we return the answer.

Now all we need to get this working is the definition of Answer and FnPlusArgs:

struct Answer;
typedef Answer (*impl_fn_type)( long, long );

struct FnPlusArgs
{
    impl_fn_type fn_;
    long arg1_;
    long arg2_;

    FnPlusArgs(
        impl_fn_type fn,
        long arg1,
        long arg2
    );

    Answer operator()();
};

impl_fn_type null_fn = NULL;
FnPlusArgs null_fn_plus_args( null_fn, 0, 0 );

struct Answer
{
    bool finished_;
    FnPlusArgs tail_call_;
    long value_;

    Answer( bool finished, FnPlusArgs tail_call, long value );
};

FnPlusArgs::FnPlusArgs(
    impl_fn_type fn,
    long arg1,
    long arg2
)
: fn_( fn )
, arg1_( arg1 )
, arg2_( arg2 )
{
}

Answer FnPlusArgs::operator()()
{
    return fn_( arg1_, arg2_ );
}

Answer::Answer( bool finished, FnPlusArgs tail_call, long value )
: finished_( finished )
, tail_call_( tail_call )
, value_( value )
{
}

The only notable thing about this is that we use operator() on FnPlusArgs to call the function it holds.

Results

Now, when we run this code, we get what we wanted:

$ ulimit -S -s 16
$ ./times_two tail_call 100000
200000

(I.e. it doesn't crash.)

So, it turns out that the tail call optimisation is just a while loop. Sort of.

Derek Jones: Incompetence borne of excessive cleverness

I have just got back from the 24 hour Data Science Global Hackathon; I was an on-site participant at Hub Westminster in London (thanks to Carlos and his team for doing such a great job looking after us all {around 50 turned up from the 100 who registered; the percentage was similar in other cities around the world}). Participants had to be registered by 11:00 UTC, self form into 3-5 person teams ready for the start at 12:00 UTC and finish 24 hours later. The world-wide event had been organized by our London hosts who told us they expected the winning team to come from those in the room; Team Outliers (Wang, Jonny, Kannappan, Bob, Simon, yours truely and Fran for the afternoon) started in an optimistic mood.

At 12:00 an air-quality training dataset + test points was made available and teams given the opportunity to submit eight predictions in each of the two 12 hour time periods. The on-line submissions were evaluated by Kaggle (one of the sponsors, along with EMC) to produce a mean estimated error that was used to rank teams.

The day before the event I had seen a press release saying that the task would involve air-quality and a quick trawl of the Internet threw up just the R package I needed, OpenAir; I also read a couple of Wikipedia articles on air pollution.

Team Outliers individually spent the first hour becoming familiar with the data and then had a get together to discuss ideas. Since I had a marker pen, was sitting next to a white-board and was the only person with some gray hair I attempted to manage the herding of the data science cats and later went on to plot the pollution monitor sites on Google maps as well as producing some visually impressive wind Roses (these did not contribute anything towards producing a better solution but if we had had a client they could have been used to give the impression we were doing something useful).

People had various ideas about the techniques to use for building the best model and how the measurements present in the training set might be used to predict air quality (the training data had names such as target_N_S, where N and S were small integer values denoting the kind of pollution and the site where the measurement was made). The training set included measurements of wind speed/direction data and hours of sunlight, and a couple of members wanted to investigate if these would make good predictors. Team Outliers had people looking at all the fancy stuff you find in textbooks, e.g., ARIMA for time series, svm for machine learning, and I was looking at getting the data into the form needed by OpenAir.

There were all sorts of problems with the data, just like real life, e.g., missing values (lots of them), some kinds of quality (i.e., pollutants) were only measured at one or two sites and fuzzy values such as the ‘most common month’ (what ever that might be). Some people were looking at how best to overcome this data quality problem.

20:30 arrived and some great food was laid out for dinner, no actual predictions yet but they would be arriving real soon now. A couple of hours later my data formatting project crashed and burned (being a 32-bit system R got upset about my request to create a vector needing 5.2 Gig; no chance of using the R-based OpenAir package which needed data in a format different from the one we were given it in). Fifteen minutes to midnight I decided that we either used the eight submissions permitted in the first 12 hours or lost them, and wrote a dozen lines of R that built a linear model using one predictor variable (which I knew from some earlier plots was far from linear, but the coding was trivial and the lm function would take no time to build separate models for each of the 30 odd response variables). I submitted the predictions and we appeared on the score board at number 65 out of 112. Being better than 47 other teams was a bit of a surprise.

Panic over we realised that the 12 hours ended at 12:00 UTC which was 01:00 BST (British Summer time) and we had another hour. Wang made a couple of submissions that improved our score and at around 02:00 I went to grab a few hours sleep.

I was back online at 06:00 to find that team Outliers had slipped to 95th place as the Melbourne and San Francisco teams had improved during their daytime. More good food for breakfast at 08:30.

Jonny drew attention to the fact that the mean absolute error in our team’s current score was almost twice as great as that of the sample solution provided with the data. We had long ago dismissed this solution as being too simplistic (it was effectively a database solution in that it calculated the mean value of the various pollutants in the training set at various chunkID and hour points which were used as keys for ‘looking up’ prediction values required by the test dataset). Maybe team members ought to focus their attention on tweaking this very simple approach rather than our ‘cleverer’ approaches.

I suggested modifying the sample solution to use the median rather than the mean (less susceptible to outliers), this boosted our ranking back into the 60′s. Jonny and Simon tried using a rolling mean, no improvement; Wang tried other variations, no improvement.

Team Outliers finished the Hackathon in equal 61st, along with 22 other teams, out of 114 submissions.

What did we do wrong? Mistakes include:

• trying to do too much for people with our various skill levels in the time available. For instance, I don’t regularly reformat large data tables or try to calculate the error in machine learning models and while I can easily knock out the code I still have to sit down and think about what needs to be written, something that somebody who does this sort of thing regularly would just know how to do; other team members seemed very familiar with the theory but were not used to churning out code quickly.
• spending too much time studying all of the various kinds of measurements available in the training set, many of which were not available in the test dataset. We should have started off ignoring measurements in the training set that were not available in the test set, perhaps looking to using these later if time permitted.

Members of team Outliers enjoyed themselves but were a little crestfallen that our clever stuff was not as good as such a crude, but insightful, approach. Most of us used R, a few made use of awk, Python, spreadsheets and Unix shell.

Our hosts are looking to run more data science hackathons this year, in particular one related to the music industry in a few months time. If you are interested in taking part keep an eye on their website.

Update (later the next day)

At least one team achieved some good results using ARIMA. Fran had started building an ARIMA model, had to leave and nobody else picked it up; I should have been paying more attention to ensure that ideas did not disappear when people left.

April 29, 2012

Christof Meerwald: EPOLLOUT|EPOLLET Behaviour

Continuing my investigation of EPOLLOUT|EPOLLET behaviour I have now looked at different socket types (and updated the test code). This now shows that the behaviour described in the epoll man page actually only applies to TCP sockets and varies quite a bit for other socket types.

For UDP sockets you get an EPOLLOUT event when the send operation completes, whereas with UNIX domain sockets you get an EPOLLOUT event when the data has been read by the peer. And finally, with UNIX pipes you get an event when the send operation completes and another event when the data is read by the peer.

Unfortunately, this doesn't appear to be documented anywhere and you are essentialy left with testing what the behaviour is or trying to read the kernel source code. There was a somewhat related discussion on the Linux netdev mailing list a few years ago regarding the EPOLLOUT behaviour for TCP sockets.

April 27, 2012

Andy Balaam: Lighting talk – Tail call optimisation in C++

This series: Lightning talk, Explanation, Performance, Generalisation.

Here’s the lightning talk I gave at the ACCU 2012 Conference:

Tail Call Optimisation in C++

It’s about how you would generate C++ code that represents a recursive function, without running out of stack space.

Derek Jones: An academic programming language paper about R

The R language has passed another milestone, a paper aimed at the academic programming language community (or at least one section of this community) has been written about it, Evaluating the Design of the R Language by Morandat, Hill, Osvald and Vitek. Hardly earth shattering news, but it may have some impact on how R is viewed by nonusers of the language (the many R users in finance probably don’t care that R seems to have been labeled as the language for doing statistics). The paper is well written and contains some very interesting information as well as a few mistakes, although it will probably read like gobbledygook to anybody not familiar with academic programming language research. What follows has something of the form of an R users guide to reading this paper, plus some commentary.

The paper has roughly three parts, the first gives an overview of R, the second is a formal definition of a subset and the third an initial report of an analysis of R usage. For me and I imagine you dear reader the really interesting stuff is in the third section.

When giving a language overview to people who know other computer languages it makes sense to leverage that knowledge, this is why the discussion has a world view from the perspective of languages rarely associated with R: Scheme, Haskell and CLOS. I found some of the discussion of R constructs to be much more informative and less confusing than that in nearly all R books/tutorials I have read, but then they are written from a detailed operational programming language perspective. One criticism of this overview is that it does not give any hint as to why R has such a large following (saying that users found it more useful than these languages would send the wrong kind of signal .

What is a formal description of a subset of R (i.e., done purely using mathematics) doing in the second part? Well, until recently very little academic software engineering was empirically based and was populated by people I would classify as failed mathematicians without the common sense needed to be engineers. Things are starting to change but research that measures things, particularly people, is still regarded as not being respectable in some quarters. In this case the formal definition is playing the role of a virility symbol showing that the authors are obviously regular guys who happen to be indulging in a bit of empirical research.

A surprising number of papers measuring the usage of real software contain formal definitions of a subset of the language being measured. Subsets are used because handling the complete language is a big project that usually involves one or more people getting a PhD out of the work. The subset chosen have to look plausible to readers who understand the mathematics but not the programming language, broadly handle all the major constructs but not get involved with all the fiddly details that need years of work and many pages to describe.

The third part contains the real research, which is really about one implementation of R and the characteristics of R source in the CRAN and Bioconductor repositories, and contains lots of interesting information. Note: the authors are incorrect to aim nearly all of the criticisms in this subsection at R, these really apply to the current implementation of R and might not apply to a different implementation.

In a previous post I suggested some possibilities for speeding up the execution of R programs that depended on R usage characteristics. The Morandat paper goes a long way towards providing numbers for some of these usage characteristics (e.g., 37% of function parameters are assigned to and 36% of vectors contain a single value).

What do we learn from this first batch of measurements? R users rarely use many of the more complicated features (e.g., object oriented constructs {and this paper has been accepted at the European Conference on Object-Oriented Programming}), a result usually seen for other languages. I was a bit surprised that R programs were only 40% smaller than equivalent C programs. I think part of the reason is that some of the problems used for benchmarking are not the kind that would usually be solved using R and I did not see any ‘typical’ R programs being coded up in C for comparison, another possibility is that the authors were not thinking in R when writing the code.

One big measurement topic the authors missed is comparing their general findings with usage measurements of other languages. I think they will find lots of similar patterns of usage.

The complaint that R has been defined by the successive releases of its only implementation, rather than a written specification, applies to all widely used languages, at least in their early days. Back in the day a major reason for creating language standards for Pascal and then C was so that other implementations could be created; the handful of major languages whose specification was written before the first implementation (e.g., PL/1, Ada) have/are dieing out. Are multiple implementations needed in an Open Source world? The answer seems to be no for Perl and yes for PHP, Ruby etc. The effort needed to create a written specification for the R language might be better invested improving the efficiency of the current implementation so that a better alternative is not needed.

Needless to say the authors suggested committing the fatal programming language research mistake.

The authors have created an interesting set of tools for static and dynamic analysis of R and I look forward to reading more about their findings in future papers.

April 26, 2012

RiverBlade: ACCU Conference 2012

Another great ACCU Conference passes, and with it the usual mix of informative and entertaining sessions, great networking, barhopping, and far too much food and (real!) beer. As ever the schedule was varied, the sessions interesting (for example we attended one on writing Jenkins plug-ins, which is obviously very relevant to Visual Lint Build Server) and entertaining (Tim Lister's bonus lightning talk on "Software projects are hurricanes", or Alan Kelley's passionate rant on software rewrites. It goes without saying that the company was as fun and inspiring as ever. We learnt a bit more about C++ 11 (my copy of the second edition of "The C++ Standard Library is now resting on my desk alongside Anthony Williams' recently arrived "C++ Concurrency in Action" and Allan Kelley's "Business Patterns for Software Developers"...), a lot about other stuff and generally put the world to rights, using Twitter as our medium, and beer as our lubrication. There was even a little free pizza and beer floating around - the latter notably during Olve Maudal's "C++ 11 Pub Quiz" which was (rather appropriately) held in the hotel bar. However, for me the absolute highlight had to be Bob Martin's "A Requiem for C", which was so funny he had us laughing almost continuously all of the way through. The fact that he was wearing a headcam for the whole thing (watching you watching me...") just added to the hilarity. More than one delegate was noted to comment that his nickname should now be "Uncle Borg" rather than "Uncle Bob"... On Wednesday evening I took the plunge and did my first lightning talk, entitled "Don't let the Big Ball of Mud sneak Up on you". Talking to a room of 400 people with a microphone and a 5 minute deadline is an interesting place to be in, but it was great fun and I'd heartily recommend it if you fancy trying your hand at speaking. Skillsmatter filmed some of the sessions, so video should be online soon - as will slides from the presenters themselves. Next up for us should be Agile on the Beach in Falmouth this September.

April 25, 2012

Derek Jones: Using identifier prefixes results in more developer errors

Human speech communication has to be processed in real time using a cpu with a very low clock rate (i.e., the human brain whose neurons fire at rates between 10-100 Hz). Biological evolution has mitigated the clock rate problem by producing a brain with parallel processing capabilities and cultural evolution has chipped in by organizing the information content of languages to take account of the brains strengths and weaknesses. Words provide a good example of the way information content can be structured to be handled by a very slow processor/memory system, e.g., 85% of English words start with a strong syllable (for more details search for initial in this detailed analysis of human word processing).

Given that the start of a word plays an important role as an information retrieval key we would expect the code reading performance of software developers to be affected by whether the identifiers they see all start with the same letter sequence or all started with different letter sequences. For instance, developers would be expected to make fewer errors or work quicker when reading the visually contiguous sequence consoleStr, startStr, memoryStr and lineStr, compared to say strConsole, strStart, strMemory and strLine.

An experiment I ran at the 2011 ACCU conference provided the first empirical evidence of the letter prefix effect that I am aware of. Subjects were asked to remember a list of four assignment statements, each having the form id=constant;, perform an unrelated task for a short period of time and then recall information about the previously seen constants (e.g., their value and which variable they were assigned to).

During recall subjects saw a list of five identifiers and one of the questions asked was which identifier was not in the previously seen list? When the list of identifiers started with different letters (e.g., cat, mat, hat, pat and bat) the error rate was 2.6% and when the identifiers all started with the same letter (e.g., pin, pat, pod, peg, and pen) the error rate was 5.9% (the standard deviation was 4.5% and 6.8% respectively, but ANOVA p-value was 0.038). Having identifiers share the same initial letter appears to double the error rate.

This looks like great news; empirical evidence of software developer behavior following the predictions of a model of human human speech/reading processing. A similar experiment was run in 2006, this asked subjects to remember a list of three assignment statements and they had to select the ‘not seen’ identifier from a list of four possibilities. An analysis of the results did not find any statistically significant difference in performance for the same/different first letter manipulation.

The 2011/2006 experiments throw up lots of questions, including: does the sharing a prefix only make a difference to performance when there are four or more identifiers, how does the error rate change as the number of identifiers increases, how does the error rate change as the number of letters in the identifier change, would the effect be seen for a list of three identifiers if there was a longer period between seeing the information and having to recall it, would the effect be greater if the shared prefix contained more than one letter?

Don’t expect answers to appear quickly. Experimenting using people as subjects is a slow, labour intensive process and software developers don’t always answer the question that you think they are answering. If anybody is interested in replicating the 2011 experiment the tools needed to generate the question sheets are available for download.

For many years I have strongly recommended that developers don’t prefix a set of identifiers sharing some attribute with a common letter sequence (its great to finally have some experimental backup, however small). If it is considered important that an attribute be visible in an identifiers spelling put it at the end of the identifier.

See you all at the ACCU conference tomorrow and don’t forget to bring a pen/pencil. I have only printed 40 experiment booklets, first come first served.

April 24, 2012

Andy Balaam: Scheme: pairs, lists and recursion

In this series on Scheme: Intro, Basics, Closures.

Here’s a presentation I did recently, on the basics of the Scheme programming language.

Scheme: pairs, lists and recursion

April 23, 2012

RiverBlade: ResOrg 2.0.2.17 has been released

The following changes are included in this build:

• Fixed a bug which caused retail builds of ResOrg 2.0.1.16 to erroneously expire 90 days after the build date.
  
• When a ResOrgApp floating toolwindow is shown a check is now performed to ensure that its position is within the visible area of the desktop. If not, the window is moved to the centre of the main window before being shown.

Jon Jagger: The evolution of useful things

Can any single theory explain the shape of a Western saw, which cuts on the push stroke, as readily as an Eastern one, which cuts on the pull?

A French book of advice to students recognised the implicit threat involved in using a weapon at the table, and instructed its readers to place the sharp edge of their knife facing towards themselves… Such actions, coupled with the growing widespread use of forks, gave the table knife its now familiar blunt-tipped blade.

Round chopsticks would tend to twist in the fingers and roll off the table, and so squaring one end eliminated two annoyances in what is certainly a brilliant design.

The stories associated with knives, forks, and spoons also illustrate well how interrelated are technology and culture generally.

Luxury, rather than necessity, is the mother of invention.

The very properties of the material that make it possible to be shaped into a useful object also limit its use.

Engineering is invention institutionalised, and engineers engaged in design are inventors who are daily looking for ways to overcome the limitations of what already works.

It is not the form follows function but, rather, that the form of one thing follows from the failure of another thing to function as we would like.

When sewn into a garment, a piece of thread can be thought of as a continuous and flexible ghost of a needle.

It is 3M's policy (and that of other enlightened companies) to allow its engineers to spend a certain percentage of their work time on projects of their own choosing, a practice known as "bootlegging".

Pete Goodliffe: Installing Realtek USB WiFi drivers on Mac OS Lion

The airport card in my laptop died and I couldn't get it fixed before going away for a week. So I needed a stop-gap solution. There are many USB wifi adaptors on eBay that claim Mac compatibility. Some are super-cute little things that only just stick out of the USB port (how they have enough antenna showing to get a signal is anyone's guess).

They almost all seem to use a Realtek chipset. (You can verify this by going to System Information on your Mac, opening the Hardware>USB page, and selecting the device. The bottom pane shows the device info, included the manufacturer name (look at the Vendor ID line).

If you're lucky, you'll get a driver CD bundled. If you're very lucky it's even got some Mac drivers on it.

Don't even try to use them! They'll be hideously out of date.

And don't go to the Realtek website and download the driver that matches your chip number. no doubt that too will be hideously out of date.

I wasted hours doing that; it reminded me of why the Mac is (usually) a so much better computing environment that Windows and Linux. I wasted hours fiddling with different driver versions, 32-bit and 64-bit kernels, cleaning out old driver installations, Googling forum postings, trying other driver versions, and all sorts of tedious tomfoolery.

The simple solution I found, which works on Lion, with 64 bit kernels and 32 bit kernels, is to install the following Realtek driver:

http://www.realtek.com.tw/downloads/downloadsView.aspx?Langid=1&PNid=21&PFid=48&Level=5&Conn=4&DownTypeID=3&GetDown=false&Downloads=true

Select the RTL8188CUS>Others>Mac OSX 10.7 Install Package (UI ver 1.9.7) version.

Even though the chip number doesn't match my USB key, the software is perfectly compatible with older devices. And it has the added bonus of working.

Remember: the driver doesn't integrate into Apple's airport utility. So you'll have to run a separate client to set up the wireless network. It isn't pretty. But it works.

April 22, 2012

Christof Meerwald: Design and Analysis of Algorithms I

Just completed the final exam for Design and Analysis of Algorithms I (by Tim Roughgarden of Stanford University). I have to admit that I slightly messed up on the final exam and only got 27 out of 30 - but as I got the quizzes and programming assignments during the course right, my total score should still be quite reasonable. Let's wait for the certificate of accomplishment...

April 21, 2012

Christof Meerwald: boost asio thread contention

Looking a bit more closely at what happens behind the scenes for boost asio where you have more worker threads than work to do (i.e. one socketpair with 4 worker threads from my previous boost asio example) we can see some interesting thread contention via strace:

11687 sendmsg(7, {msg_name(0)=NULL, msg_iov(1)=[{"\7\0\0\0", 4}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL <unfinished ...>
11689 <... epoll_wait resumed> {{EPOLLIN, {u32=36459312, u64=36459312}}}, 128, 4294967295) = 1
11687 <... sendmsg resumed> )           = 4
11689 futex(0x7f82a6211dd4, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f82a6211dd0, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1} <unfinished ...>
11687 futex(0x7f82af213dd4, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f82af213dd0, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1} <unfinished ...>
11690 <... futex resumed> )             = 0
11689 <... futex resumed> )             = 1
11690 futex(0x22c5150, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
11689 recvmsg(6,  <unfinished ...>
11690 <... futex resumed> )             = 0
11689 <... recvmsg resumed> {msg_name(0)=NULL, msg_iov(1)=[{"\7\0\0\0", 4}], msg_controllen=0, msg_flags=0}, 0) = 4
11690 epoll_wait(4,  <unfinished ...>
11689 recvmsg(6,  <unfinished ...>
11690 <... epoll_wait resumed> {}, 128, 0) = 0
11689 <... recvmsg resumed> 0x7f82aea12370, 0) = -1 EAGAIN (Resource temporarily unavailable)
11690 epoll_wait(4,  <unfinished ...>
11689 sendmsg(6, {msg_name(0)=NULL, msg_iov(1)=[{"\10\0\0\0", 4}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL <unfinished ...>

What's interesting to see is that there are 3 threads involved with significant locking (via futex) between those threads.

For comparison, the equivalent section from the strace for asyncsrv.cc only shows 2 active threads and no lock contention at all:

11711 sendto(7, "\7\0\0\0", 4, 0, NULL, 0) = 4
11710 <... epoll_wait resumed> {{EPOLLIN, {u32=7674112, u64=7674112}}}, 16, 4294967295) = 1
11711 epoll_wait(3,  <unfinished ...>
11710 recvfrom(6, "\7\0\0\0", 4, 0, NULL, NULL) = 4
11710 recvfrom(6, 0x7518e0, 4, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
11710 sendto(6, "\10\0\0\0", 4, 0, NULL, 0 <unfinished ...>

That's definitely what you would expect.

BTW, I did apply a crude fix for the spurious EPOLLOUT events to boost asio to not get distracted from those.

Christof Meerwald: Spurious EPOLLOUT events

One thing I noticed when looking at epoll scalability was that Linux seems to generate lots of spurious EPOLLOUT events (even when using EPOLLET - edge-triggered). To illustrate the issue, have a look at eptest-out.cc. This clearly shows that Linux generates an EPOLLOUT event for each send syscall even though there is no need as the state hasn't changed.

BTW, the strace output also clearly shows those events:

sendto(4, "\7\0\0\0", 4, 0, NULL, 0)    = 4
epoll_wait(3, {{EPOLLOUT, {u32=4, u64=4}}, {EPOLLIN|EPOLLOUT, {u32=5, u64=5}}}, 16, 4294967295) = 2
recvfrom(5, "\7\0\0\0", 4096, 0, NULL, NULL) = 4
recvfrom(5, 0x7fffa14e1000, 4096, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)
sendto(4, "\10\0\0\0", 4, 0, NULL, 0)   = 4
epoll_wait(3, {{EPOLLOUT, {u32=4, u64=4}}, {EPOLLIN|EPOLLOUT, {u32=5, u64=5}}}, 16, 4294967295) = 2
recvfrom(5, "\10\0\0\0", 4096, 0, NULL, NULL) = 4
recvfrom(5, 0x7fffa14e1000, 4096, 0, 0, 0) = -1 EAGAIN (Resource temporarily unavailable)

Note that in this case the EPOLLOUT event on the sending socket always coincides with an EPOLLIN|EPOLLOUT event on the receiving socket, so you don't get an extra wakeup, but you aren't always that lucky...

Jon Jagger: Adapt - why success always starts with failure

is an excellent book by Tim Harford (isbn 978-0-349-12151-2). As usual I'm going to quote from a few pages

Cross the river by feeling for stones [attributed to Deng Xiaoping]

Accepting trial and error means accepting error.

Darwin, a meticulous observer...

The art of success is to fail productively.

Complexity is a problem only in tightly coupled systems.

Make sure you know when you've failed, or you will never learn.

What Palchinsky realised was that most real-world problems are more complex than we think. They have a human dimension, a local dimension, and are likely to change as circumstances change. His method for dealing with this could be summarised as three 'Palchinsky principles'

• seek out new ideas and try new things
• when trying something new, do it on a scale where failure is survivable
• seek out feedback and learn from your mistakes as you go along

If we are to take the 'variation' part of 'variation and selection' seriously, uniformly high standards are not only impossible but undesirable.

When John Nagl served in Baghdad in 2003, he found that while his young inexperienced soldiers had the authority to kill, he - a major with a doctorate and a decade of experience - didn't have the authority to print his own propaganda pamphlets to counteract the clever PR campaign that the local insurgents were running.

Speciation - the divergence of one species into two separate populations - rarely happens without some form of physical separation.

Tight coupling means the unintended consequences proliferate so quickly that it is impossible to adapt to the failure or to try something different.

The first thing Timpson does when it buys another business is to rip out the electronic point-of-sale machines (there are always EPOS machines) and replace them with old fashioned cash registers. 'EPOS lets people at head office run the business', explains John Timpson. 'I don't want them to run the business.'
... John Timpson describes one instance where he couldn't buy half-price happy hour drinks at a hotel bar, because midway through giving his order, the hour ended and the bar's computerised sales system refused to allow the half-price offer to be applied.
... Timpson's company training manual describes the twenty easiest ways to defraud the company, making it clear that the company understands the risks it is running and trusts its employees anyway - and many people respond to being trusted by becoming more trustworthy.

A central point of the corporation, as a legal structure, is that it is supposed to be a safe space in which to fail. Limited liability companies were developed to encourage people to experiment, to innovate, to adapt - safe in the knowledge that if their venture collapsed, it would merely be the abstract legal entity that was ruined, not them personally.

Fail better. [Samuel Beckett]

Jon Jagger: Curry velocity

Some people's introduction to curry begins with them ordering the hottest dish on the menu. They discover the dish is beyond their curry-capacity and they can't finish it. Amazingly, they often repeat this behaviour. Lot's of curry goes to waste. Dishes are frequently only half-eaten. Curry velocity remains at zero.

Other people's introduction to curry begins with them trying something nearer the cooler end of the Scoville scale. They find the dish is below their curry-capacity. They finish it! No curry is wasted. Their curry velocity is above zero. The next time they may decide to try something a little hotter. But they are in control of the curry, rather than the curry being in control of them.

Jon Jagger: Consilience

The first step to wisdom, as the Chinese say, is getting things by their right names.

The cost of scientific advance is the humbling recognition that reality was not constructed to be easily grasped by the human mind.

Analysis and synthesis, he [Goethe] liked to say, should be alternated as naturally as breathing in and breathing out.

Nothing in science - nothing in life, for that matter - makes sense without theory.

Complexity is what interests scientists in the end, not simplicity.

Consilience among the biological sciences is based on a thorough understanding of scale in time and space.

Complexity theory can be defined as the search for algorithms used in nature that display common features across many levels of organisation.

In a system containing perfect internal order, such as a crystal, there can be no further change.

the brain is a machine assembled not to understand itself, but to survive.

The biologist S. J. Singer has drily expressed the matter thus: I link, therefore I am.

No example of bias-free mental development has yet been discovered.

April 20, 2012

Allan Kelly: Points: conclusions and hypothesis

As anyone with time to spare will know I’ve recently spent a lot of time thinking, and writing about story points. This was in response to Vasco Duarte’s Story Points Considered Harmful blog from a month or two back. For completeness here are the links:

• Vasco Duarte: Story Points Considered Harmful
• Story points considered harmful? 1 of 4 - Journey's start
• Story points 2 of 4 - Duarte's arguments
• Story points 3 of 4 - An example
• Points 4 of 4 - Breakdown

• There is far more work to do on Abstract/Story Points than we, as a community have done to date
• There are many more nuances to the assignment of points, the breakdown of work and the management of the outcomes than I think I previously realised
• I must go and see what Mike Cohn actually says about Story Points before I say any more about his approach
• Stable teams are crucial - but then I’ve been saying this all along

RiverBlade: Introducing VisualLintGui

If you have been following me (@annajayne) on Twitter, you may have noticed me talking about something called "VisualLintGui". This is actually the second of two projects (the first being VisualLintConsole - the command line version of Visual Lint) we got underway after the release of Visual Lint 3.0. Now that VisualLintConsole (the command line version of Visual Lint) is out in the wild, we have turned our attention to VisualLintGui. This is, as the name suggests, a standalone Visual Lint application with a graphical user interface - basically a text editor focused on code analysis:

Derek Jones: Randomizing a list of items using sort() and rand()

I’m busy putting together the experiment I will be running at the ACCU conference next week. If you are attending the conference please reserve your Thursday lunchtime slot for taking part as a subject!

Experiment generation invariably involves randomizing the sequence of items seen by every subject. While few languages support a randomise function many support sorting and random number generation. These two functions can be combined to create a randomize function; simply append a random number to the start of each item, sort it and then strip off the random number. Voilà a randomized list (awk code below).

function rand_items(items)
{
for (v in items)
   {
   items[v]=rand() " " items[v]
   }
asort(items)
for (v in items)
   {
   sp_pos=index(items[v], " ")
   items[v]=substr(items[v], sp_pos+1)
   }
}

This randomization problem is not yet listed on Rosetta code and probably has longer solutions in other languages.

Update (the next day).

The glow I have had for the last 10 years over coming up with a neat solution to a problem has now disappeared. Following the link kindly provided by D. Herring in the comments eventually lead me to the Fisher-Yates shuffle, which has performance (the call to sort is probably . The following shuffles a deck of cards:

for (i = 0; i < 52; i++)
   {
   j = i + (rand() % (52 - i));
   tmp = card[i];
   card[i] = card[j];
   card[j] = tmp;
   }

The proof of the uniform shuffling behavior of Fisher-Yates (also known as Knuth shuffle) is straight forward but not nearly as appealing as using rand and sort.

Phil Nash: Upcoming speaker engagements

I seem to have gotten myself committed to some speaker engagements over the next couple of months:

Next week, on 26th April, I'm giving the thursday keynote at the ACCU 2012 conference in Oxford. My topic there is "The Congruent Programmer" and is about aligning what we do with our motivations and gaining clarity on why we do things.

Then, on 29th June, I'll be over at Mobile East talking about how to TDD your iOS apps. There seems to be growing interest in this area. I've just been engaged with a team at the BBC doing just that. And a few days ago Graham Lee's new book on "Test Driven iOS Development" was released. Graham's book mentions my C++/ Objective-C test framework, CATCH, so it must be good!

Pete Goodliffe: Speaking: Forthcoming talks

• I'm speaking at ACCU 2012 next week. Details of my talks are available here. This year I'm in the "tools" track. Make of that what you will.
• I'm speaking at Mobile East in June, on Advanced iOS Development.

April 19, 2012

Tim Pizey: Configuring MySQL

The defaults that MySQL comes with are probably not what you want.

We got caught by this at work and followed the instructions given in How to calculate a good InnoDB log file size

query_cache_limit = 2M
query_cache_size  = 32M
table_cache       =256

innodb_open_files = 512

innodb_buffer_pool_size = 512M

innodb_additional_mem_pool_size = 512M
innodb_log_file_size = 64M
innodb_log_buffer_size = 8M
innodb_thread_concurrency = 8
innodb_concurrency_tickets = 500
innodb_lock_wait_timeout=200
innodb_autoinc_lock_mode= 2
innodb_commit_concurrency=4
innodb_flush_log_at_trx_commit=2
innodb_support_xa=false
innodb_checksums=0
innodb_doublewrite=0
innodb_max_dirty_pages_pct=15

Discover the location of datadir typically datadir = /var/lib/mysql

service mysql stop
mv /var/lib/mysql/ib_logfile0 /var/lib/mysql/ib_logfile0_old
mv /var/lib/mysql/ib_logfile1 /var/lib/mysql/ib_logfile1_old
service mysql start

Allan Kelly: Points 4 of 4 - Breakdown

This entry directly continues from three earlier ones:

• Story points considered harmful? 1 of 4 - Journey's start
• Story points 2 of 4 - Duarte's arguments
• Story points 3 of 4 - An example

• Whites are smaller pieces of work, they flow through a system more easily. Progress can be seen, tasks tracked, velocity calculated.
• “The Business” aka Product Owner/Manager/BA are not always good at delivering small stories, breaking a blue down gives the developer a chance.
• On some teams the business have been beaten up by development to request really small stories. However these stories lack business value. Because Blues are going to be broken down they can be large enough to have value even if that means that can’t be completed in one iteration/sprint.
• (Yes, you heard that right) Whites are completed during the iteration, when all the Whites, or the essential ones, are completed then the Blue is completed.
• Breaking Blues down to Whites is as much a design exercise as it is an estimation and scheduling one. This allows teams to engage in design and create a shared understanding.
• Breaking Blues down to Whites frequently reveals functionality or assumptions about the Blue requirement which can be removed or postponed.
• Having the Product Manager/Owner/BA in the room during this break down allows for requirements elaboration and knowledge mining.
• Work can be rolled from one iteration to the next. I’m very relaxed about carry over work and I think for a new team its almost unavoidable. However doing it this way allows some points to be counted and illustrates what is happening.

• I have seen the breakdown and estimate approach. As detailed previously in one case it allowed a team to forecast to the day.
• Estimating work allows teams, and individual team members, to raise a warning when work is not understood, defined or involves a lot of risk. For example, a team estimating with planning poker will normally settle on an “average size” of task, e.g. 3 or 5 points. When they suddenly assign 13 or 20 points to a task something is wrong.
• And just in case the warning is ignored the team, the people at the code face, the people doing the work, have a control mechanism. No matter how much the business or a manager bully a team they can still assign a high point score.
• Equally, when differences in estimation appears it is a trigger to discussion, to learning, to understanding. This is desirable.

April 18, 2012

Andy Balaam: What timezone Python thinks I’m in

sudo aptitude install python-dateutil
python -c "import dateutil.tz; import datetime; print datetime.datetime.now( dateutil.tz.tzlocal() ).tzname()"

Allan Kelly: Story points 3 of 4 - An example

This entry continues from two earlier ones:

• Story points considered harmful? 1 of 4 - Journey's start
• Story points 2 of 4 - Duarte's arguments

Allan Kelly: Story points 2 of 4 - Duarte's arguments

This blog entry follow directly from the previous Story Points - Journey’s Start.

Key to Duarte argument, and something I didn’t originally appreciate from his initial Tweets is: he is not saying story points are rubbish, forget about them. What he is saying is: it is simpler and equally accurate to just count the stories as atomic items. This is equivalent to saying “All stories are 1 point” and having done with it.

For a mature team with a good relationship with its stakeholders I could see this working well. However, for less mature teams (who have difficulty agreeing among themselves) or a team with bully-boy stakeholders (or bully boy anyone else for that matter) then I think being able to put a higher point score on the card serves as a useful warning mechanism.

Duarte says in his blog “the best predictor of the future is your past performance!” On this I couldn’t agree more. He then poses three questions - and answers - which I think are worth reviewing.

Q1: Is there sufficient difference between what Story Points and ’number of items’ measure to say that they don’t measure the same thing?

Here he finds there is a close correlation. I’m not surprised here, in fact I would expect this to be the case. Teams are encouraged to write small stories, in fact Scrum almost mandates this because work should be completely done at the end of a sprint. In effect there is an upper bound placed on the size of a story.

Actually, I’m not so keen on this rule. I allow work to be carried from iteration to iteration but I only allow points to be scored when the work is done. Thus I encourage stories to be completed in an iteration but I don’t mandate it. One of the exercises I do with teams on my courses actually sets out to illustrate this point.

At the very least I would expect teams to settle on an “average story size” implicitly. Notice also that the correlation applies whether all stories are of size 1 or of size 2, 3 or any other number. Its a correlation between two series of numbers.

However, given all this Duarte has a point: if your stories are clustered around an average size then you might as well count the stories.

Q2: Which one of the two metrics is more stable? And what does that mean?

Duarte’s analysis says that both stories and story points have similar standard deviation. Thus they are of similar stability. Since these two are closely correlated this isn’t a surprise. In fact, given the correlation, it would be a surprise if one was notably more stable.

Q3: Are both metrics close enough so that measuring one (number of items) is equivalent to measuring the other (Story Points)?

Duarte’s data seems to measure the same thing - again, if they are closely correlated then this is exactly what you would expect. You can write out the equation:
        Story Points ~= (Correlation Co-efficient) x (Number of Stories)

(The ~= is supposed to mean approximately equal.)

With this out of the way Duarte moves on to consider Mike Cohn’s claims for story points.

Claim 1: The use of Story points allows us to change our mind whenever we have new information about a story
Duarte says: Story Points offer no advantage over just simply counting the number of items left to be Done.

I agree here. I’ve long encouraged teams to move away from story pointing work in the distant future. Yes I encourage them to story point some stories in the backlog - say a few months work - and story point tasks - for the next iteration. But for stuff that is “out there” or has just arisen my advice is usually: just assign it your average story point value.

In other words, assume your average story point value is your Correlation Co-efficient. When work gets close then estimate it traditionally, you might find the value changes. When it gets really close break it down into tasks.

Claim 2: The use of Story points works for both epics and smaller stories
Duarte says: there is no significant added information by classifying a story in a 100 SP category

Again I agree. To be honest I’m not a fan of Epics and while some of the teams I work with use them I often encourage teams to dump them. To me an epic is just a collection of stories around a theme.

Actually, what Cohn and Duarte are saying are not at odds here. Cohn doesn’t (seem to) make any additional claims. Its just a scaling question.

Claim 3: The use of Story points doesn’t take a lot of time        
Duarte says: In fact, as anybody that has tried a nontrivial project knows it can take days of work to estimate the initial backlog for a reasonable size project.

Here I have issues with both Cohn and Duarte.
If you are estimating stories then it does take time. Fast as it is, even planning poker takes time. However there is also a lot of design and requirements discussion going on in that activity. Therefore I don’t see this as a problem. In fact I see it as an important learning exercise.

True, on a none trivial project it will take time to estimate a large backlog. But a) that is valuable learning and b) I won’t try. I’d either estimate it in chunks or I’d apply an average estimate to work which wasn’t going to happen anytime soon - see claim #2.

I deliberately delay estimation as long as possible to allow more information to arrive and because work will change. It might be changed out of all recognition or it might go away completely.

Claim 4: The use of Story points provides useful information about our progress and the work remaining
Duarte says: This claim holds true if, and only if you have estimated all of your stories in the Backlog and go through the same process for each new story added to the Backlog.

Again I agree. However I doubt the usefulness of the concept of “work remaining”. Its only work remaining if you think you have a lump of work to do. In my experience work is always negotiable. Its just that people don’t want to negotiate until they accept that they won’t get everything.

One of my clients has gone through the very expensive exercise of estimating all the work they might do. Earlier this year they realised they had 3000 points to do by Christmas. They also realised they had capacity to do less than 1000. This brought home to fact that they couldn’t do everything - something many people on the project had long known or suspected. The company are still working through this issue but at they are having the discussion now, in March and April not September and October.

Claim 5: The use of Story points is tolerant of imprecision in the estimates
Duarte says: there's no data [in Cohn’s book] to justify the belief that Story Points do this better than merely counting the number of Stories Done. In fact, we can argue that counting the number of stories is even more tolerant of imprecisions

Again I agree. But then, if there is a high correlation between story points and stories then this is self-evident. And again, as I said before: we need to work with aggregates and averages.

Claim 6: Story points can be used to plan releases
Duarte says: Fair enough. On the other hand we can use any estimation technique to do this, so how would Story Points [be better than counting the number of stories]

Again, agreement, and with correlation its self evident.

Duarte goes on to give a worked examples in which a project does not achieve the desire velocity and gets cancelled. His story makes no use of story points, simply stories. To be honest I’m missing something here. True the stories in his story have no points, but I don’t see where that makes a difference. What he describes is exactly the way I would play the scenario although I would have story points in the mix.

His conclusion: “Don't estimate the size of a story further than this: when doing Backlog Grooming or Sprint Planning just ask: can this Story be completed in a Sprint by one person? If not, break the story down!”

This is interesting because while Duarte is working at the story level this pretty closely models the way I advise teams to work. I always tell teams:

• “I’d like a story to be small, to fit in one iteration but that isn’t always the way.”
• “In my experience stories need to be broken down, both so they can get done but also so you get flow and as part of a design exercise.”

Andy Balaam: Freeview dvb-t scan gives “tuning failed!!!” after 18th April switchover

Usually when the channel lineup changes, I just have to do this:

$ scan /usr/share/dvb/dvb-t/uk-CrystalPalace > ~/.mplayer/channels.conf

But this time it is failing with:

>>> tune to: 505833000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE
WARNING: >>> tuning failed!!!
>>> tune to: 505833000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_AUTO:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE (tuning failed)
WARNING: >>> tuning failed!!!

(snip)

However, a more low-level scan with w_scan seems to be working today:

$ sudo apt-get install w-scan     # Install w_scan
$ w_scan -X > ~/.mplayer/channels.conf

I still can’t see BBC Four at the moment…

Update
It looks like I have to scan when the channels are actually on air. I did this:

$ cd ~/.mplayer
$ w_scan -X > channels.conf.morning    # I ran this in the morning
$ w_scan -X > channels.conf.evening    # I ran this in the evening
$ cat channels.conf.morning channels.conf.evening | sort | uniq > channels.conf

Now I have both BBC Four and CBeebies, which pretty much covers us!