Pattern: Single CrUD Transaction

Software patterns have their roots in architecture. In 1978, Christopher Alexander published a book called ‘A Pattern Language: Towns, Buildings, Construction‘ (ISBN-13: 978-0195019193) about the patterns he’d discovered designing buildings. A pattern can be thought of as a tried and tested way of doing something which can be applied in different contexts.  Think about how the Observer or Visitor pattern is implemented across languages such as Java, Ruby and JavaScript, where the different language idioms dictate slightly different implementations of the same basic pattern.

Software Patterns became popular with the publishing of the Gang of Four book, “Design patterns: elements of reusable object-oriented software” (ISBN-13: 978-0201633610) in 1994. It contains a number of patterns, most of which every developer should know, even if it’s to know to avoid the likes Singleton. However, these aren’t the only patterns! Indeed, patterns are not created, they are discovered and documented. Whole conferences are dedicated to software patterns (http://www.europlop.net/), where delegates are encouraged to bring their pattern write-ups for appraisal by their peers and the experts.

In 2000 I joined the ACCU, a group for programmers who strive for better software. I was encouraged by another member to write for the group’s magazine, but I didn’t think I’d have anything to contribute that someone better hadn’t already thought of and written about. As I gained experience I found I had quite a lot to write about and to challenge.

In the same way you’d have thought that 23 years after the Gang of Four book most if not all of the software patterns had been discovered and documented. However, it appears not and I was very surprised to find that what I’m calling the “Single CrUD Transaction” pattern, although used by many, doesn’t appear to have been written up anywhere publically. I checked with industry experts and they weren’t aware of it being written-up either.

This is my first software pattern write up and where better to share it for the first time than Norfolk Developers Magazine?

Name

Single CrUD Transaction

Intent

To create, update and delete items in a datastore within a single transaction.

Problem

Sometimes it’s necessary to create, update and delete items in a datastore in a single transaction. Traditional web applications support create, update and delete in separate transactions and require the page to be reloaded between each action.

Modern web applications allow the items of a list to be created, updated and deleted in a browser without any interaction with the server or the underlying datastore. Therefore when the list is sent to the server side it must determine which items are new, which already exist and must be updated and which have been removed from the list and must be deleted.

One simple solution is to delete all of the items from the datastore and simply replace them with the list of line items passed from the browser to the server. There are at least two potential drawbacks with this approach:

  1. If the datastore (such as a relational database) uses unique, numerical ids to identify each item in the list, the size of the ids can become very big, very quickly.
  2. If the datastore (such as a relational database) has other data which references the ids of the items in the list, the items cannot be deleted without breaking the referential integrity.

Solution

The Single CrUD Transaction pattern gets around these drawbacks by performing three operations within a single transaction:

  1. Delete all of the list items from the datastore whose ids are not in the list passed from the browser to the server.
  2. Update each of the items in the datastore whose ids match ids in the list passed from the browser to the server.
  3. Create new items in the datastore for each item in the list passed from the browser to the server which do not yet have ids.
Each action is executed within a single transaction so that if any individual action fails the list is returned to its original state.

Applicability

Use the Single CrUD transaction pattern when:

  • Datastores cannot have new items added, existing items updated and/or items removed in separate transactions.
  • Creating new ids for each item in the list each time the datastore is modified is expensive or cumbersome.
  • Removing all the items of a list from a datastore and recreating the list in the datastore breaks referential integrity.

Advantages and Disadvantages

Advantages


  • Entire update happens within a single transaction.

Disadvantages


  • Three separate calls to the datastore within a single transaction.


On Share And Share Alike – student

When last they met, the Baron challenged Sir R----- to a wager in which, for a price of three coins and fifty cents, he would make a pile of two coins upon the table. Sir R----- was then to cast a four sided die and the Baron would add to that pile coins numbering that upon which it settled. The Baron would then make of it as many piles of equal numbers of no fewer than two coins as he could muster and take back all but one of them for his purse. After doing so some sixteen times, Sir R----- was to have as his prize the remaining pile of coins.

Good Stories Assure the Architecture

One of the problems a team can run into when they adopt a more agile way of working is they struggle to frame their backlog in the terms of user focused stories. This is a problem I’ve written about before in “Turning Technical Tasks Into User Stories” which looked at the problem for smaller units of work. Even if the team can buy into that premise for the more run-of-the-mill features it can still be a struggle to see how that works for the big ticket items like the system’s architecture.

The Awkward Silence

What I’ve experienced is that the team can start to regress when faced with discussions around what kind of architecture to aim for. With a backlog chock full of customer pleasing functionality the architectural conversations might begin to take a bit of a back seat as the focus is on fleshing out the walking skeleton with features. Naturally the nervousness starts to set in as the engineers begin to wonder when the architecture is going to get the attention it rightly deserves. It’s all very well supporting a handful of “friendly” users but what about when you have real customers who’ve entrusted you with their data and they want to make use of it without a moments notice at any hour of the day?

The temptation, which should be resisted, can be to see architectural work as outside the scope of the core backlog – creating a separate backlog for stuff “the business does not understand”. This way can lead to a split in the backlog, and potentially even two separate backlogs – a functional and a non-functional one. This just makes prioritisation impossible. Also burying the work kills transparency, eventually erodes trust, and still doesn’t get you the answers you really need.

Instead, the urge should be to frame the architectural concerns in terms the stakeholder does understand, so that the business can be more informed about their actual benefits. In addition, when “The Architecture” is a journey and not a single destination there is no longer one set of benefits to aim for there are multiple trade-offs as the architecture evolves over time, changing at each step to satisfy the ongoing needs of the customer(s) along the way. There is in essence no “final solution” there is only “what we need for the foreseeable future”.

Tell Me a Story

So, what do I mean by “good stories”? Well, the traditional way this goes is for an analyst to solicit some non-functional requirements for some speculative eventual system behaviour. If we’re really lucky it might end up in the right ballpark at one particular point in the future. What’s missing from this scene is a proper conversation, a proper story – one with a beginning, a middle, and an end – where we are today, the short term and the longer term vision.

But not only do we need to get a feel for their aspirations we also need quantifiable metrics about how the system needs to perform. Vague statements like “fast enough” are just not helpful. A globally accessible system with an anticipated latency in the tens of milliseconds will need to break the law of physics unless we trade-off something else. We also need to know how those exceptional events like Cyber Monday are to be factored into the operation side.

It’s not just about performance either. In many cases end users care that their data is secure, both in-flight (over the network) and at rest, although they likely have no idea what this actually means in practice. Patching servers is a technical task, but the bigger story is about how the team responds to a vulnerability which may make patching irrelevant. Similarly database backups are not the issue it’s about service availability – you cannot be highly available if the loss of an entire data centre potentially means waiting for a database to be restored from scratch elsewhere.

Most of the traditional conversations around non-functional requirements focus entirely on the happy path, for me the conversation doesn’t really get going until you start talking about what needs to happen when the system is down. It’s never a case of “if”, but “when” it fails and therefore mitigating these problems features heavily in our architectural choices. It’s an uncomfortable conversation as we never like discussing failure but that’s what having “grown up” conversations mean.

Incremental Architecture

Although I’ve used the term “story” in this post’s title, many of the issues that need discussing are really in the realm of “epics”. However we shouldn’t get bogged down in the terminology, instead the essence is to remember to focus on the outcome from the user’s perspective. Ask yourselves how fast, how secure, how available, etc. it needs to be now, and how those needs might change in response to the system’s, and the business’s growth.

With a clearer picture of the potential risks and opportunities we are better placed to design and build in small increments such that the architecture can be allowed to emerge at a sustainable rate.

SE-Radio interviews Ron Lichty, must listen for development managers

The Software Engineering Radio podcast has just published an episode with an interview with Ron Lichty. If you’re either thinking about moving from development into management or already have, it’s well worth a listen. Unfortunately there is only so much that can be packed into an hour’s worth of a podcast, but just based on […]

The post SE-Radio interviews Ron Lichty, must listen for development managers appeared first on The Lone C++ Coder's Blog.

Mutable

The mutable keyword seems to be one of the less known corners of C++. Yet it can be very useful, or even unavoidable if you want to write const-correct code […]

The post Mutable appeared first on .

Event: Burkhard Kloss on The Ethics of Software & Panel: Talking to the clouds


Event: Burkhard Kloss on The Ethics of Software & Panel: Talking to the clouds

When: 6 November 2017 @ 6.30pm

Where: Whitespace, 2nd Floor, St James Mill, Whitefriars, Norwich, NR3 1TN

RSVP: https://www.meetup.com/preview/Norfolk-Developers-NorDev/events/239616865

The Ethics of Software - some practical considerations
Burkhard Kloss
@georgebernhard

As Uncle Bob pointed out, software is everywhere, and without software, nothing works.

That gives us great power, and – as we all know – with great power comes great responsibility.

We have to make choices every day that affect others, sometimes in subtle and non-intuitive ways. To mention just a few:

  • What logs should we capture?
  • How does that change if we have to hand them over to the government?
  • Are our hiring practices fair? Are we sure about that?
  • Is there bias in our algorithms that unfairly disadvantages some groups of people?
  • Is the core function of our software ethical? How about if it’s deliberately misused?

I hope to raise a few of these questions, not to provide answers – I don’t have any – but to stimulate debate.

Burhard Kloss

I only came to England to walk the Pennine Way… 25 years later I still haven’t done it. I did, though, get round to starting an AI company (spectacularly unsuccessful), joining another startup long before it was cool, learning C++, and spending a lot of time on trading floors building systems for complex derivatives. Sometimes hands on, sometimes managing people. Somewhere along the way I realised you can do cool stuff quickly in Python, and I’ve never lost my fascination with making machines smarter.


Panel Discussion: Talking to the clouds

Conversational computing, the ability to talk to, an interact with a computer via voice, is becoming more and more prevalent. Most of us now have access to an intelligent assistant like Siri or Alexa, and how we interact with the devices is being defined. But are we going in the right direction. Should we be treating these devices as just "dumb computers", or should we speak to them as we do to other people?

Our panel of experts will discuss this topic with input from the audience as we look at one of the many areas where the question is not "can we?", but "should we?".

Reflections on learning and training (and dyslexia)

Nuremberg_Funnel1910-2017-10-17-16-19.jpg

German readers might recognise the image above as a Nuremberg Funnel. For the rest of you: the Nuremberg funnel represents the idea that a teacher can simply open a students head and pour in learning.

First off, a warning: this blog entry might be more for me than for you, but its my blog!

Still, I expect (hope) some of you might be interested in my thoughts and reflections on how training, specifically “Agile training” sessions. More importantly writing this down, putting it into words, helps me think, structure my thoughts and put things in order.

Shall I begin?

I’m not naive enough to think training is perfect or can solve everything. But I have enough experience to know that it can make a difference, especially when teams do training together and decide to change together.

One of the ways I’ve made my living for the last 10 years is by delivering “Agile training.” I don’t consider myself a “trainer” (although plenty of others do), I consider myself more as a “singer songwriter” – it is my material and I deliver it. I’ve actually considered renaming my “Agile training” as “Agile rehearsal” because thats how I see it. I haven’t because I’d have to explain this to everyone and more importantly while people do google for “Agile training” nobody searches for “Agile rehearsal.”

Recently I’ve been prompted to think again about how people learn, specifically how they learn and adopt the thing we call “Agile”. One unexpected experience and one current challenge have added to this.

A few months ago I got to sit in while someone else delivered Agile training. On the one hand I accepted that this person was also experienced, also had great success with what he did, also claimed his courses were very practical and he wasn’t saying anything I really disagreed with.

On the other hand he reminded me of my younger self. The training felt like the kind of training I gave when I was just starting out. Let me give you two examples.

When I started giving Agile training I felt I had to share as much as possible with the attendees. I was conscious that I only had them for a limited time and I had so much to tell them! I was aiming to give them everything they needed to know. I had to brain dump… Nuremberg funnel.

So I talked a lot, sessions were long and although I asked “Any questions?” I didn’t perhaps give people enough time to ask or for me to answer – ‘cos I had more brain dumping to do!

Slowly I learned that the attendees grew tired too. There was a point where I was talking and they had ceased to learn. I also learned that given a choice (“Would you like me to talk about Foobar for 30 minutes or have you had enough?”) people always asked for more.        

Second, when I started out I used to include the Agile Manifesto and a whistle-stop-tour of Lean. After all, people should know where this came from and they should understand some of the theory, right? But with time I realised that the philosophy of the manifesto takes up space and isn’t really important. Similarly, Lean is very abstract and people have few reference points. To many (usually younger) people who have never lived “the waterfall” it can seem a crazy straw-man anyway.

Over the years I’ve tried to make my introductions to Agile more experiential. That is, I want to get people doing exercises and reflecting on what happened. I tend to believe that people learn best from their own experience so I try to give them “process miniatures” in the classroom and then talk (reflect) on the experience.

These days my standard starter 2-day Agile training course is three quarters exercises and debriefs. My 1-day “Requirements, User Stories and Backlogs” workshop is almost entirely exercise based. I’m conscious that there is still more stuff – and that different people learn in different ways – so I try to supplement these courses with further reading – part of the reason behind printing “Little Book of User Stories” is to supplement “Agile Reader” in this.

I’m also conscious that by allowing people to learn in different mediums, and to flip between them they can learn more and better.

My own thinking got a big boost when I learned about Constructivist learning theory. Perhaps more importantly I’ve also benefited from exploring my own dyslexia. (Reading The Dyslexic Advantage earlier this year was great.)

Why is dyslexia relevant here? Well two reasons…

Firstly, something I was told a long time ago proves itself time and time again: Dyslexics have problems learning the way non-dyslexics do, but the reverse is not true. What helps dyslexics learn helps everyone else learn better too.

Second: dyslexics look at the world differently and we have to construct their own meaning and find our own ways to learn. To me, Agile requires a different view of the world and it requires us to learn to learn. Three years back I even speculated that Agile is Dyslexic, as time goes by I’m more convinced of that argument.

So why am I thinking so much about all this?

Well, I’ve shied away from online training for a few years now – how can I do exercises? how can I seed reflection?

Now I’ve accepted a request to do some online training. I can’t use my existing material, it is too exercise based. I’m having to think long and hard about this.

One thought is to view “online training” as something different to “rehearsal training.” That is, something delivered through the online medium might be more akin to an audio book, it is something that supplements a rehearsal. But that thinking might be self limiting and ignore opportunities offered online.

The other thing is the commercial medium. As my training has got more experiential and better at helping people move from classroom to action it has actually covered less. The aim is to seed change, although the classroom is supplemented the actual material covered in class is less; learn less change more! – Thats a big part of the reason I usually give free consulting days after training.

In a commercial setting where there is a synopsis and a price tag the incentive is to list more and more content. One is fearful of missing something the buyer considers important. One can imagine a synopsis being placed next to a competitor synopsis and the one with the most content for the least price is chosen.

So, watch this space, I will be venturing into online training. To start off with I’m not sure who is going to be learning the most: the attendees or the presenter! (O perhaps I shouldn’t have said that, maybe I’m too honest.)

If you have any experience with training (as a teacher or student), and especially online training, I’d love to hear about them. Please comment below or email me.

Read more? Subscribe to my newsletter – free updates on blog post, insights, events and offers.

The post Reflections on learning and training (and dyslexia) appeared first on Allan Kelly Associates.

Data analysis with a manual mindset

A lot of software engineering data continues to be analysed using techniques designed for manual implementation (i.e., executed without a computer). Yes, these days computers are being used to do the calculation, but they are being used to replicate the manual steps.

Statistical techniques are often available that are more powerful than the ‘manual’ techniques. They were not used during the manual-era because they are too computationally expensive to be done manually, or had not been invented yet; the bootstrap springs to mind.

What is the advantage of these needs-a-computer techniques?

The main advantage is not requiring that the data have a Normal distribution. While data having a Normal, or normal-like, distribution is common is the social sciences (a big consumer of statistical analysis), it is less common in software engineering. Software engineering data is often skewed (at least the data I have analysed) and what appear to be outliers are common.

It seems like every empirical paper I read uses a Mann-Whitney test or Wilcoxon signed-rank test to compare two samples, sometimes preceded by a statement that the data is close to being Normal, more often being silent on this topic, and occasionally putting some effort into showing the data is Normal or removing outliers to bring it closer to being Normally distributed.

Why not use a bootstrap technique and not have to bother about what distribution the data has?

I’m not sure whether the reason is lack of knowledge about the bootstrap or lack of confidence in not following the herd (i.e., what will everybody say if my paper does not use the techniques that everybody else uses?)

If you are living on a desert island and don’t have a computer, then you will want to use the manual techniques. But then you probably won’t be interested in analyzing software engineering data.

I think I’ve worked out why Uni is so stressful.

The study environment generates panic.

I’m back at University after taking a year out to work as a professional software developer.

I got back and instantly found everything so incredibly stressful.
Moreso than I felt before during my earlier years at University and certainly more than I’d felt at all during my work years.
It took me a couple of weeks, but I think I’ve figured out a few reasons why making the transition from professional work, back to being a student is so tough.

Fragmented days

The first, and possibly biggest issue I’m having is the fragmented days.

Most days during the week I have some lectures. Often, it’s two or more different modules on the same day, sometimes in different buildings.

The problem with this is that I spend the whole day wildly context-switching .
Not just switching between discrete tasks, like one might do at work, but entirely different mindsets, technologies, and problems.

I’ll be switching from thinking about my Concurrent and Real Time Systems course, its coursework, and the project I’m making for that. Then break for a couple of hours, during which I might have a meeting or spend time working on plans for the society I lead before switching into the next module’s session (maybe OpenSource Development for example) .

I spend the whole day never spending more than about two hours on a particular module or task before switching to another, never finishing anything before having to move onto another thing.

To exacerbate this is that I might also be moving my physical location as well as my mental one. Having to get up and move across campus for the next session when I haven't reached a good point of closure on the one I’m working on.

This way of working makes me feel like I never manage to finish things. I feel like the end of a particular piece of work is never in sight because I can’t work on it long enough at any one time to feel like I’ve achieved something or gotten closer to completion.

Taking your work home

Because I never manage to get anything substantial done during the working day, I have to bring my work home with me.

Taking one’s work home is a fundamental part of University life, as every student will know.
You are never finished, you’ve always got something you should be doing right now, instead of relaxing or seeing friends.

This results in whenever I’m not working towards my degree, I feel like I should be.
I feel like I’m wasting my time doing whatever else I’m doing because I should be working.

This leads to a lot of stress, because one can never relax properly when they’ve always got work on the mind.

Having to wait for the information

While this might not affect other students who have less prior knowledge and experience on their topic than myself, having to wait to be told things because of the plan the course leader set out is incredibly frustrating.

For the past year, I’ve been used to having the project brief set out and being released to go and produce the project (including, of course, doing all the planning and research needed before implementation).

At University, you know you’re going to have to do something but the lecturers hold off on telling you exactly what you need to be producing.
Instead, they tend to release a new piece of information every week. So no matter how much you already know, or how fast you got the work for this week finished, you’re not going to know what you need to do next until the next scheduled lecture.

This results in me knowing that there's a deadline in the future, but having no idea what I need to be doing to make sure that I manage to hit that deadline with everything finished.

In business, you get told what you’re doing and it’s up to you to plan your time and get done what you need to get done to make sure the project hits the deadline.
Yes, there are hold ups that you can’t always get around. But, because you know the whole scope of the project and what you need to get finished, there is normally something else you can work on so you’re not wasting time.

General stressful environment

A big factor that contributes to my general stress is feeling the stress of everyone around me.

University is so full of people who are in no way relaxed. People who are out of their depths. Lecturers reminding you of the deadlines so you don’t forget.

University just feels buzzing with stress-fuelled energy, and it doesn't help anyone.

Maybe it’s me?

But also, it might just be me.
I’m stressed already and I’m only a couple of weeks into term.
My deadlines feel like they’re on top of me, even though they’re all in November.
My society needs constant attention and planning (with help from a fantastic committee)
I moved back into my parents and had to store most of my stuff.
I need to plan my Final Year Project, somehow (I have no supervisor yet)
I’m speaking at some conferences soon.
I haven’t got any work, and not much money.
Soon I have to start proper work with Coventry Pride again.

So yeah, a lot on my plate and University is not helping much.

Thanks for reading this a-bit-of-a-rant post.

How to read water

is an excellent book by Tristan Gooley (isbn 978-1-473-61522-9). As usual I'm going to quote from a few pages:
One of the universal truths of human observation is that we see more of what we expect to see and less of what we don't expect to see.
Much of my work is not about teaching people to see things that are hard to see, but in showing them how to notice the things that hide in plain sight.
It did not take sailors long to work out that a ship that carries too much may be vulnerable in heavy seas, but sailors were rarely the ones to make the decision about how much cargo a ship could safely carry. The merchants making the profit would have had a different view to the deckhand, especially if the trader never set foot on the vessel. This led to a wrestle between greedy traders and cautious captains that lasted centuries. The first attempts to regulate how much a ship could carry go back over four thousand years to ancient Crete.
Samuel Plimsoll, a nineteenth-century English politician, realized that a low freeboard height can present a problem, but he also appreciated that it becomes the solution if we take a very keen interest in it. In other words, we can tell if there is too much cargo in the boat by looking much more carefully at how high the water rises up the side of the hull. And the easiest way to do this is by drawing a ruler on the side of the ship, calibrated according to an architect's or engineer's understanding of the boat. These lines, which became known as Plimsoll Lines, were such a simple and brilliant success that they became law and proliferated around the world.
From 1833, when the first tide tables were produced by the Admiralty, the emphasis shifted from looking, thinking and understanding, to depending on tables of others' measurements.
There is a stange truth in the profile of beaches: they have evolved in a physical sense to be a near ideal shape to defend themselves against the onslaught of the sea. This means that almost any attempt to engineer a 'solution' to what nature is trying to achieve has as much chance of backfiring as working.
Many sailors use little pieces of fabric, nicknamed 'tell-tails', that are tied to the sails and stays (the wires that give the mast stability), to offer a constant visual reminder of what the wind is doing.
Once the depth of the water is half the wavelength of the waves, it effectively cramps the motion of the waves and it is this that slows them down.
Sailors dislike precision almost as much as they like bureaucracy.
Rivers do not run straight for more than ten times their own width.
There will be an alternating combination of quick water and much slower water and this always happens in a certain way. The quick patches are knowns, perhaps onomatopoeically, as 'riffles' and the slower areas are known as pools. If there is no human tinkering with a river's flow, then there will be a riffle-pool sequence for every stretch of river that is fives times its width.
It is typical for the water at the sides of a river to be travelling at only a quarter of the speed of the water in the centre. The river is being slowed by two things at its sides; when it comes into contact with banks it is slowed by friction and it is also slowed by the shallowing at the sides.
A stream is just a river you can step over.
Swell is the name of the waves that have enough energy to travel beyond the area of wind.


Friday throughts on the Agile Manifesto and Agile outside of software

AgileManifesto-2017-10-13-17-01.jpg

While I agree with the Agile Manifesto I’ve never been a great on for defining “Agile” in terms of it.

As time goes by I find the manifesto increasingly looks like a historic document. It was written in response to the problems in the software industry at the turn of the millennium – problems I recognise because I was there. I worked on the Railtrack privatisation, ISO 9000 certified and so much paper you needed a train to move it. I worked at Reuters as they destroyed their own software capability with a CMM stream roller.

The manifesto is a little like Magna Carta or the US Constitution, you sometimes have to read into it what would fit your circumstances. It was written about software and as we apply agile outside of software you have to think “what would it say?” the same way the US Supreme Court looks at the Constitutions interprets what it would say about the Internet

Perhaps a more interesting question than “What is Agile?” is “Where does Agile apply?” or, even more interesting, “Where does Agile not apply?”

One can argue that since Agile includes a self-adaptation mechanism (inspect and adapt) – or as I have argued, Agile is the embodiment of the Learning Organization – it can apply to anything, anywhere. Similarly it has universal applicability and can fix any flaws it has.

Of cause its rather bombastic to make such an argument and quite possibly anyone making that argument hasn’t thought it through.

So the definition of “Agile” becomes important – and since we don’t have one, and can’t agree on one we’re in a rather tricky position.

Increasingly I see “Agile” (and so some degree Lean too) as a response to new technologies and increasing CPU power. Software people – who had a particular problem themselves – had first access to new technologies (programmable assistants, email, instant messenger, wikis, fast tests and more) and used them to address their own issues.

The problems are important. Although people have been talking about “agile outside of software development” for almost as long as agile software development it has never really taken off in the same way. To my mind thats because most other industries don’t have problems which are pushing them to find a better way.

As technologies advance, and as more and more industries become “Digital” and utilise the same tools software engineers have had for longer then those industries increasingly resembled software development. That means two things: other industries start to encounter the same problems as software development but they also start to apply the same solutions.

Software engineers are the prototype of future knowledge workers.

Which implies, the thing we call Agile is the prototype process for many other industries

“Agile outside of software” becomes a meaningless concept when all industries increasingly resemble software delivery.

Read more? Subscribe to my newsletter – free updates on blog post, insights, events and offers.

The post Friday throughts on the Agile Manifesto and Agile outside of software appeared first on Allan Kelly Associates.

The User-Agent is not Just for Browsers

One of the trickiest problems when you’re building a web service is knowing who your clients are. I don’t mean your customers, that’s a much harder problem, no, I literally mean you don’t know what client software is talking to you.

Although it shouldn’t really matter who your consumers are from a technical perspective, once your service starts to field requests and you’re working out what and how to monitor it, knowing this becomes far more useful.

Proactive Monitoring

For example the last API I worked on we were generating 404’s for a regular stream of requests because the consumer had a bug in their URL formatting and erroneously appended an extra space for one of the segments. We could see this at our end but didn’t know who to tell. We had to spam our “API Consumers” Slack channel in the hope the right person would notice [1].

We also had consumers sending us the wrong kind of authorisation token, which again we could see but didn’t know which team to contact. Although having a Slack channel for the API helped, we found that people only paid attention to it when they noticed a problem. It also appeared, from our end, that devs would prefer to fumble around rather than pair with us on getting their client end working quickly and reliably.

Client Detection

Absent any other information a cloud hosted service pretty much only has the client IP to go on. If you’re behind a load balancer then you’re looking at the X-Forwarded-For header instead which might give you a clue. Of course if many of your consumers are also services running in the cloud or behind the on-premise firewall they all look pretty much the same.

Hence as part of our API documentation we strongly encouraged consumers to supply a User-Agent field with their service name, purpose, and version, e.g. MyMobileApp:Test/1.0.56. This meant that we would now have a better chance of talking to the right people when we spotted them doing something odd.

From a monitoring perspective we can then use the User-Agent in various ways to slice-and-dice our traffic. For example we can now successfully attribute load to various consumers. We can also filter out certain behaviours from triggering alerts when we know, for example, that it’s their contract tests passing bad data on purpose.

By providing us with a version number we can also see when they release a new version and help them ensure they’ve deprecated old versions. Whilst you would expect service owners to know exactly what they’ve got running where, you’d be surprised how many don’t know they have old instances lying around. It also helps identify who the laggards are that are holding up removal of your legacy features.

Causality

A somewhat related idea is the use of “trace” or “correlation” IDs, which is something I’ve covered before in “Causality - A Mechanism for Relating Distributed Diagnostic Contexts”. These are unique IDs for diagnosing problems with requests and it’s useful to include a prefix for the originating system. However that system may not be your actual client if there are various other services between you and them. Hence the causality ID covers the end-to-end where the User-Agent can cover the local client-server hop.

You would think that the benefit of passing it was fairly clear – it allows providers to proactively help consumers fix their problems. And yet like so many non-functional requirements it sits lower down their backlog because it’s only optional [2]. Not only that but by masking themselves it actually hampers delivery of new features because you’re working harder than necessary to keep the existing lights on.

 

[1] Ironically the requests were for some automated tests which they didn’t realise were failing!

[2] We wanted to make the User-Agent header mandatory on all non-production environments [3] to try and convince our consumers of the benefits but it didn’t sit well with the upper echelons.

[3] The idea being that its use in production then becomes automatic but does not exclude easy use of diagnostic tools like CURL for production issues.

Visual Lint 6.0.6.287 has been released

Visual Lint 6.0.6.287 is now available. This is a recommended maintenance update for Visual Lint 6.0, and includes the following changes:
  • Added PC-lint Plus specific compiler indirect files co-rb-vs6.lnt (Microsoft Visual Studio 6.0), co-rb-vs2002.lnt (Microsoft Visual Studio .NET 2002), co-rb-vs2003.lnt (Microsoft Visual Studio .NET 2003) and co-rb-vs2005.lnt (Microsoft Visual Studio 2005) to the installer.
  • Modified the installer to correctly recognise Microsoft Visual Studio 2017 version 15.3 installations.
  • Fixed a bug in the installer which affected the installation of the Atmel Studio plug-in into AVR Studio 5.x.
  • Added PC-lint Plus specific warning 550 suppression directives for the UNREFERENCED_PARAMETER, DBG_UNREFERENCED_PARAMETER and DBG_UNREFERENCED_LOCAL_VARIABLE macros.

    These suppressions are implemented in a new lib-rb-win32.h header file referenced by the existing lib-rb-win32.lnt indirect file supplied within the installer.
Download Visual Lint 6.0.6.287

Don’t Hide the Solution Structure

Whenever you join an existing team and start work on their codebase you need to orientate yourself so that you have a feel for the system’s architecture and design. If you’re lucky there is some documentation, perhaps nice diagrams to give you an overview. Hopefully you also have an extensive suite of tests to tell you how the system behaves.

More than likely there is nothing or very little to go on, and if it’s a truly legacy system any documentation could well be way out of date. At this point you pretty much only have the source code to work from. Whilst this is the source of truth, the amount of code you need to read to become au fait with all the various high-level concepts depends in part on how well it’s laid out.

Static Structure

Irrespective of whether you like to think of your layers in terms of onions or brick walls, all code essentially gets organised on disk and that means the solution structure is hierarchical in nature. In the most popular languages that support namespaces, these are also hierarchical and are commonly laid out on disk to reflect the same hierarchy [1].

Although the compiler is happy to just hoover up source code from the entire solution and largely ignore the relative position of the callers and callees there are useful conventions, which if honoured, allow you to reason and refactor the code more easily due to lower coupling. For example, defining an interface in the same source file as a class that implements it suggests a different inheritance use than when the interface sits externally further up the hierarchy. Also, seeing code higher up the hierarchy referencing types deeper down in an unrelated branch is another smell, of an abstraction potentially depending on an implementation detail.

Navigating the Structure

One of the things I’ve noticed in recent years whilst pairing is that many developers appear to navigate the source code solely through their IDE, and within the IDE by using features like “go to definition (implementation)”. Some very rarely see the solution structure because they hide it to gain more screen real estate for the source file of current interest [2].

Hence the only time the solution structure is visible is when there is a need to add a new source file. My purely anecdotal evidence suggests that this will be added without a great deal of thought as the code can be easy located in future directly by the author through its class name or another reference; they never have to consider where it “logically” resides.

Sprawling Suburbs

The net result is that namespaces and packages suffer from urban sprawl as they slowly accrete more and more code. This newer code adds more dependencies and so the package as a whole acquires an ever increasing number of dependencies. Left unchecked this can lead to horrible cyclic dependencies that are a nightmare to resolve.

I recently had the opportunity to revisit the codebase for a greenfield system I had started a few years before. We initially partitioned the code into a few key assemblies to get ourselves going and so I was somewhat surprised to still see the same assemblies a few years later, albeit massively overgrown with extra responsibilities. As a consequence even their simple home-grown tools had bizarre dependencies dragged in through bloated shared libraries [3].

Take a Stroll

So in future, instead of taking the Underground (subway) through your codebase every day, stop, and take a stroll every now-and-then around the paths. The same rules about cohesion within the methods of a class also apply at the higher levels of design – classes in a namespace, namespaces in an assembly, assemblies in a solution, etc. Then you’ll find that as the system grows it’s easier to refactor at the package level [3].

(For more on this topic see my older post “Who’s Maintaining the 100 Foot View?”.)

 

[1] Annoyingly this is not a common practice in the C++ codebases I’ve worked on.

[2] If I was being flippant I might suggest that if you really need the space the code may be too complicated, as I once did on Twitter here.

[3] I once dragged in a project’s shared library for a few useful extension methods to use in a simple console app and found I had pulled in an IoC container and almost a dozen other NuGet dependencies!

[4] In C# the internal access modifier has zero effect if you stick all your code into one assembly.

Every Commit Needs the Rationale to Support It

Each and every change to a codebase should be performed for a very specific reason – we shouldn’t just change some code because we feel like it. If you follow a checklist (mental or otherwise), such as the one I described in “Commit Checklist”, then each commit should be as cohesive as possible with any unintentional edits reverted to spare our blushes.

However, whilst the code can say what behaviour has changed, we also need to say why it was changed. The old adage “use the source Luke” is great for reminding us that the only source of truth is the code itself, but changes made without any supporting documentation makes software archaeology [1] incredibly difficult in the future.

The Commit Log

Take the following one line change to the JSON serialization settings used when persisting to a database:

DateTimeZoneHandling = DateTimeZoneHandling.Utc;

This single-line edit appeared in a commit all by itself. Now, any change which has the potential to affect the storage or retrieval of the system’s data is something which should not be entered into lightly. Even if the change was done to make what is currently a default setting explicit, this fact still needs to be recorded – the rationale is important.

The first port of call for any documentation around a change is probably the commit message. Given that it lives with the code and is (usually) immutable it stands the best chance of remaining intact over time. In the example above the commit message was simply:

“Bug Fix: added date time zone handling to UTC for database json serialization”

In the same way that poor code comments have a habit of simply stating what the code does, the same malaise can affect commit messages by merely restating what was changed. Our example largely suffers from this, but it also teases us by additionally mentioning that it was done to fix a bug. Suddenly we have so many more unanswered questions about the change.

Code Change Comments

In the dim and distant past it was not unusual to use code comments to annotate changes as well as to describe the behaviour of the code. Before the advent of version control features like “blame” (aka annotate) it was non-trivial to track down the commit where any particular line of code changed. As such it seemed easier to embed the change details in the code itself rather than the VCS tool, especially if the supporting documentation lived in another system; you could just use the Change Request ID as the comment.

As you can imagine this sorta worked okay at first but as the code continued to change and refactoring became more popular these comments became as distracting and pointless as the more traditional kind. It also did nothing to help reduce the overheard of tracking the how-and-why in different places.

Feature Trackers

The situation originally used to be worse than this as new features might be tracked in one place by the business whilst bugs were tracked elsewhere by the development team. This meant that the “why” could be distributed right across time and space without the necessary links to tie them all together.

The desire to track all work in one place in an Enterprise tool like JIRA has at least reduced the number of places you need to look for “the bigger picture”, assuming you use the tool for more than just recording estimates and time spent, but of course there are lightweight alternatives [2]. Hence recording the JIRA number or Trello card number in the commit message is probably the most common approach to linking these two sides of the change.

As an aside, one of the reasons many teams haven’t historically put all their documentation in their source code repo is because it’s often been inaccessible to non-developer colleagues, either due to lack of permissions or technical ability. Fortunately tools like GitHub have started to bridge this divide.

Executable Specifications

One of the oldest problems in software development has been keeping the supporting documentation and code in sync. As features evolve it becomes harder and harder to know what the canonical reason for any change is because the current behaviour may be the sum of all previous related requirements.

An ever-growing technique for combating this has been to express the documentation, i.e. the requirements, in code too, in the form of tests. At a high level these are acceptance tests, with more technical behaviours expressed as unit or integration tests.

This brings me back to my earlier example. It’s incredibly rare that any code change would be committed without some kind of corresponding change to the automated tests. In this instance the bug must have manifested itself in the persistence layer and I’d expect at least one new test to be added (or an existing one fixed) to illustrate what the bug is. Hence the rationale for the change is to fix a bug, and the rationale can largely be described through the use of one or more well written tests rather than in prose.

Exceptions

There are of course no absolutes in life and fixing a spelling mistake should not require pages of notes, although spelling incorrectly on purpose probably does [3].

The point is that there is a balance to be struck if we are to trade-off the short and long term maintenance of the system. It might be tempting to rely on tribal knowledge or the product owner’s notes to avoid thinking about how the rationale is best expressed, but finding a way to encode that information in executable form, such as through tests, provides both the present reviewer and the future software archaeologist with the most usable representation.

 

[1] See my “Software Archaeology” article for more about spelunking a codebase’s history.

[2] I’ve written about the various tools I’ve used in the past in  “Feature Tracking”.

[3] The HTTP “referer” header being a notable exception, See Wikipedia.

Organize for Complexity

is an excellent book by Niels Pflaeging (isbn 978-0-9915376-0-0). As usual I'm going to quote from a few pages:
What Taylor pioneered was the idea of consistently dividing an organization between thinking people (managers) and executing people (workers).
Problem-solving in a life-less system is about instruction.
Problem-solving in a living system is about communication.
Any attempt to motivate can only lead to de-motivation.
Ultimately, organizing for complexity and self-organization are always about empowering teams ... not about empowering individuals.
Actual teams of people working for and with each other.
Nobody is in control. Everybody is in charge.
To be intensively involved in selection [recruiting] is a matter of honor.
A hallmark of great selection is that it is highly time-consuming.
Management is a mindset that will not just go away all by itself.
When employees think for themselves and make entrepreneurial decisions automonomously, you must at all times bear joint reponsibility for those decisions, even if you or other members of the organization might have decided differently.
A "beta" kind of organization produces many such stories: Peculiarities, unusual practices, by which they can be instantly recognized among so many over-managed and under-led organizations.
People do not need to be forced to work. However, this deeply-seated prejudice about people and their relationship to work is what keeps management alive.


Installing specific major Java JDK versions on OS X via Homebrew

In an earlier post, I described how to install the latest version of the Oracle Java JDK using homebrew. What hadn’t been completely obvious to me when I wrote the original blog post is that the ‘java’ cask will install the latest major version of the JDK. As a result, when I upgraded my JDK […]

The post Installing specific major Java JDK versions on OS X via Homebrew appeared first on The Lone C++ Coder's Blog.

Histogram using log scale creates a visual artifact

The following plot appears in the paper Stack Overflow in Github: Any Snippets There?

Histogram of file and function counts: log scale on x-axis

Don’t those twin peaks in the top-left/bottom-right plots reach out and grab your attention? I immediately thought of fitting a mixture of two Poisson distributions; No, No, No, something wrong here. The first question of data analysis is: Do I believe the data?

The possibility of fake data does not get asked until more likely possibilities have been examined first.

The y-axis is a count of things and the x-axis shows the things being counted; source files per project and functions per file, in this case.

All the measurements I know of show a decreasing trend for these things, e.g., lots of projects have a few files and a few projects have lots of files. Twin peaks is very unexpected.

I have serious problems believing this data, because it does not conform to my prior experience. What have the authors done wrong?

My first thought was that a measuring mistake had been made; for some reason values over a certain range were being incorrectly miscounted.

Then I saw the problem. The plot was of a histogram and the x-axis had a logarithmic scale. A logarithmic axis compresses the range in a non-linear fashion, which means that variable width bins have to be used for histograms and the y-axis represents density (not a count).

Taking logs and using the result to plot a histogram usually produces a curve having a distorted shape, not twin peaks. I think the twin peaks occur here because integer data are involved and the bin width just happened to have the ‘right’ value.

Looking at the plot below, the first bin contains values for x=1 (on an un-logged scale), the second bin for x=2, the third bin for x=3, but the fourth bin contains values for x=c(4, 5, 6). The nonlinear logarithmic compression, mapped to integers, means that the contents of three values are added to a single bin, creating a total that is larger than the third bin.

Histogram of 'thing' counts: log scale on x-axis

The R code that generated the above plot:

x=1:1e6
y=trunc(1e6/x^1.5)
log_y=log10(y)
 
hist(log_y, n=40, main="", xlim=c(0, 3))

I tried to mimic the pattern seen in the first histogram by trying various exponents of a power law (a common candidate for this kind of measurement), but couldn’t get anything to work.

Change the bin width can make the second peak disappear, or rather get smeared out. Still a useful pattern to look out for in the future.

Do The Evolution – a.k.

In the last few posts we have taken a look at genetic algorithms, which use simple models of biological evolution to search for global maxima of functions, being those points at which they return their greatest possible values.
These models typically represent the arguments of the function as genes within the binary chromosomes of individuals whose fitnesses are the values of the function for those arguments, exchange genetic information between them with a crossover operator, make small random changes to them with a mutation operator and, most importantly, favour the fitter individuals in the population for reproduction into the next generation with a selection operator.
We used a theoretical analysis of a simple genetic algorithm to suggest improved versions of the crossover operator, as well as proposing more robust schemes for selection and the genetic encoding of the parameters.
In this post we shall use some of them to implement a genetic algorithm for the ak library.

Tax the data

iStock_000008663948XSmall-2017-10-6-13-19.jpg

If data is the new oil then why don’t we tax it?

My data is worth something to Google, and Facebook, and Twitter, and Amazon… and just about every other Internet behemoth. But alone my data is worth a really tiny tiny amount.

I’d like to charge Google and co. for having my data. The amount they currently pay me – free search, free email, cheap telephone… – doesn’t really add up. In fact, what Google pays me doesn’t pay my mortgage but somehow Larry Page and Sergey Brin are very very rich. Even if I did invoice Google, and even if Google paid we are talking pennies, at most.

But Google don’t just have my data, they have yours, your Mums, our friends, neighbours and just about everyone else. Put it all together and it is worth more than the sum of the parts.

Value of my data to Google < 1p
Value of your data to Google < 1p
Value our combined data to Google > 2p

The whole is worth more than the sum of the parts.

At the same time Google – and Facebook, Amazon, Apple, etc. – don’t like paying taxes. They like the things those taxes pay for (educated employees, law and order, transport networks, legal systems – particularly the bit of the legal system that deals with patents and intellectual property) but they don’t want to pay.

And when they do pay they find ways of minimising the payment and moving money around so it gets taxed as little as possible.

So why don’t we tax the data?

Governments, acting on behalf of their citizens should tax companies on the data they harvest from their users.

All those cookies that DoubleClick put on your machine.

All those profile likes that Facebook has.

Sure there is an issue of disentangling what is “my data” from what is “Google’s data” but I’m sure we could come up with a quota system, or Google could be allowed a tax deduction. Or they could simply delete the data when it gets old.

I’d be more than happy if Amazon deleted every piece of data they have about me. Apple seem to make even more money that Google and make me pay. While I might miss G-Drive I’d live (I pay DropBox anyway).

Or maybe we tax data-usage.

Maybe its the data users, the Cambridge Analyticas, of this world who should be paying the data tax. Pay for access, the ultimate firewall.

The tax would be levied for user within a geographic boundary. So these companies couldn’t claim the data was in low tax Ireland because the data generators (you and me) reside within state boundaries. If Facebook wants to have users in England then they would need to pay the British Government’s data-tax. If data that originates with English users is used by a company, no matter where they are, then Facebook needs to give the Government a cut.

This isn’t as radical as it sounds.

Governments have a long history of taxing resources – consider property taxed. Good taxes encourage consumers to limit their consumption – think cigarette taxes – and it may well be a good thing to limit some data usage. Anyway, thats not a hard and fast rule – the Government taxes my income and they don’t want to limit that.

And consider oil, after all, how often are we told that data is the new black gold?
– Countries with oil impose a tax (or charge) on oil companies which extract the oil.

Oil taxes demonstrate another thing about tax: Governments act on behalf of their citizens, like a class-action.

Consider Norway, every citizen of Norway could lay claim to part of the Norwegian oil reserves, they could individually invoice the oil companies for their share. But that wouldn’t work very well, too many people and again, the whole is worth more than the sum of the parts. So the Norwegian Government steps in, taxes the oil and then uses the revenue for the good of the citizens.

In a few places – like Alaska and the Shetlands – do see oil companies distributing money more directly.

After all, Governments could do with a bit more money and if they don’t tax data then the money is simply going to go to Zuckerberg, Page, Bezos and co. They wouldn’t miss a little bit.

And if this brings down other taxes, or helps fund a universal income, then people will have more time to spend online using these companies and buying things through them.

Read more? Subscribe to my newsletter – free updates on blog post, insights, events and offers.

The post Tax the data appeared first on Allan Kelly Associates.

A review: nor(DEV):biz October 2017

The idea “Networking” strikes fear into the heart of many techies, but Norfolk Developers Business or nor(DEV):biz is different. The idea behind the monthly meetings over dinner at the Library Restaurant is to get tech companies in Norwich and Norfolk talking to each other and referring business between themselves and from external parties. It’s not just about tech companies though, we also invite people from academia (City College Norwich was represented tonight and the UEA attended the very first event), those running complementary business (such as accountants, lawyers, recruiters, etc), those looking to engage software companies and even those looking to be employed by them.

"It was relaxed and much like having a good dinner with a selection of your wittiest and most worldly wise friends !"
- Chris Sargisson, CEO Norfolk Chamber

Norwich has networking events coming out of its ears. nor(DEV):biz is different, not just because of the tech focus, but also because of the people who attend. Over the years Norfolk Developers has attracted the biggest personalities in the community (that’s you Dom Davis!) including many senior tech business owners. Yes, everyone has their one minute to speak to the group about who they are, what they do and what they’re looking for, but there’s no bell when your time is up and there’s humour, passion and interaction from the entire group. This isn’t just networking, this is building, bonding and rapport with people you may well work with in the future. It’s more than that, this is fun and raucous and entertaining. It’s a night out with friends rather than a pressure cooker for sales.

“Great event .Who would have thought that by having dinner with a bunch of techies I would learn that tomato ketchup is the best thing for smelly dog issues.. it just shows, never judge a book by its cover.” 
- Chris Marsh, AT&A BUSINESS INSURANCE BROKERS

At each nor(DEV):biz a member has the opportunity, not the obligation, to do a 15 minute spotlight. This is beyond their one minuter and the opportunity to give a more indepth overview of their business or something they are passionate about.

"Great evening arranged by Paul Grenyer and Dom Davis for the Norfolk Developers group. My highlight was Nikki and Tom Bool integrating the basics of Dog Training skills with leading a team in the workplace!"
- Anthony Pryke, Barclays 

For this, the fourth nor(DEV):biz, the spotlight was given by Nikki and Tom Bool. Nikki is a puppy trainer, while Tom runs a language services business, specialising in helping businesses market themselves and grow internationally. Nikki explained how to use positive reinforcement to encourage the right behavior in puppies, with some hilarious anecdotes. Tom went on to describe how similar techniques can be used to help foster the desired behavior in the people you work with. The spotlight fulfilled my favorite criteria by being both informative and entertaining.

"What a smashing group of people and thoroughly enjoyable puppy behaviours reflection on office management. "
 - Mike Peters, Evoke Systems

You know you’re onto a winner when you have to encourage people to leave and the conversation has moved from the table to a huddle by the doorway.  I’m already looking forward to the next nor(DEV):biz in November where we’re hoping to hear from Laura Flood and Anietie Ukpabio of City College, Norwich, about the young people they’re training to be software engineers.

If you’d like to attend nor(DEV):biz, please drop Paul an email on paul@norfolkdevelopers.com.


Expected variability in a program’s SLOC

If 10 people independently implement the same specification in the same language, how much variation will there be in the length of their programs (measured in lines of code)?

The data I have suggests that the standard deviation of program length is one quarter of the mean length, e.g., 10k mean length, 2.5k standard deviation.

The plot below (code+data) shows six points from the samples I have. The point in the bottom left is based on 6,300 C programs from a programming contest question requiring solutions to the 3n+1 problem and one of the points on the right comes from five Pascal compilers for the same processor.

Mean vs standard deviation of sample program SLOC

Multiple implementations of the same specification, in the same language, are very rare. If you know of any, please let me know.

MVP is a marketing exercise not a technology exercise

MVP-2017-10-2-11-03.jpg
… Minimally Viable Product

Possibly the most fashionable and misused term the digital industry right now. The term seems to be used by one-side-or-other to criticise the other.

I recently heard another Agile Coach say: “If you just add a few more features you’ll have an MVP” – I wanted to scream “Wrong, wrong, wrong!” But I bit my tongue (who says I’m can’t do diplomacy?)

MVP often seems to be the modern way of saying “The system must do”, MVP has become the M in Moscow rules.

Part of the problem is that the term means different things to different people. Originally coined to describe an experiment (“what is the smallest thing we could build to learn something about the market?”) it is almost always used to describe a small product that could satisfy the customers needs. But when push comes to shove that small usually isn’t very small. When MVP is used to mean “cut everything to the bone” the person saying it still seems to leave a lot of fat on the bone.

I think non-technical people hear the term MVP and think “A product which doesn’t do all that gold plating software engineering fat that slows the team down.” Such people show how little they actually understand about the digital world.

MVPs should not about technology. An MVP is not about building things.

An MVP is a marketing exercise: can we build something customers want?

Can we build something people will pay money for?

Before you use the language MVP you should assume:

  1. The technology can do it
  2. Your team can build it

The question is: is this thing worth building? – and before we waste money building something nobody will use, let alone pay for, what can we build to make sure we are right?

The next time people start sketching an MVP divide it in 10. Assume the value is 10% of the stated value. Assume you have 10% of the resources and 10% of the time to build it. Now rethink what you are asking for. What can you build with a tenth?

Anyway, the cat is out of the bag, as much as I wish I could erase the abbreviation and name from collective memory I can’t. But maybe I can change the debate by differentiating between several types of MVP, that is, several different ways the term MVP is used:

  • MVP-M: a marketing product, designed to test what customers want, and what they will pay for.
  • MVP-T: a technical product designed to see if something can be build technologically – historically the terms proof-of-concept and prototype have been used here
  • MVP-L: a list of MUST HAVE features that a product MUST HAVE
  • MVP-H: a hippo MVP, a special MVP-L, that is highest paid person’s opinion of the feature list, unfortunately you might find several different people believe they have the right to set the feature list
  • MVP-X: where X is a number (1,2, 3…), this derivative is used by teams who are releasing enhancements to their product and growing it. (In the pre-digital age we called this a version number.) Exactly what is minimal about it I’m not sure but if it helps then why not?

MVP-M is the original meaning while MVP-L and MVP-H are the most common types.

So next time someone says “MVP” just check, what do they mean?

The post MVP is a marketing exercise not a technology exercise appeared first on Allan Kelly Associates.

Experimental method for measuring benefits of identifier naming

I was recently came across a very interesting experiment in Eran Avidan’s Master’s thesis. Regular readers will know of my interest in identifiers; while everybody agrees that identifier names have a significant impact on the effort needed to understand code, reliably measuring this impact has proven to be very difficult.

The experimental method looked like it would have some impact on subject performance, but I was not expecting a huge impact. Avidan’s advisor was Dror Feitelson, who kindly provided the experimental data, answered my questions and provided useful background information (Dror is also very interested in empirical work and provides a pdf of his book+data on workload modeling).

Avidan’s asked subjects to figure out what a particular method did, timing how long it took for them to work this out. In the control condition a subject saw the original method and in the experimental condition the method name was replaced by xxx and local and parameter names were replaced by single letter identifiers. The hypothesis was that subjects would take longer for methods modified to use ‘random’ identifier names.

A wonderfully simple idea that does not involve a lot of experimental overhead and ought to be runnable under a wide variety of conditions, plus the difference in performance is very noticeable.

The think aloud protocol was used, i.e., subjects were asked to speak their thoughts as they processed the code. Having to do this will slow people down, but has the advantage of helping to ensure that a subject really does understand the code. An overall slower response time is not important because we are interested in differences in performance.

Each of the nine subjects sequentially processed six methods, with the methods randomly assigned as controls or experimental treatments (of which there were two, locals first and parameters first).

The procedure, when a subject saw a modified method was as follows: the subject was asked to explain the method’s purpose, once an answer was given either the local or parameter names were revealed and the subject had to again explain the method’s purpose, and when an answer was given the names of both locals and parameters was revealed and a final answer recorded. The time taken for the subject to give a correct answer was recorded.

The summary output of a model fitted using a mixed-effects model is at the end of this post (code+data; original experimental materials). There are only enough measurements to have subject as a random effect on the treatment; no order of presentation data is available to look for learning effects.

Subjects took longer for modified methods. When parameters were revealed first, subjects were 268 seconds slower (on average), and when locals were revealed first 342 seconds slower (the standard deviation of the between subject differences was 187 and 253 seconds, respectively; less than the treatment effect, surprising, perhaps a consequence of information being progressively revealed helping the slower performers).

Why is subject performance less slow when parameter names are revealed first? My thoughts: parameter names (if well-chosen) provide clues about what incoming values represent, useful information for figuring out what a method does. Locals are somewhat self-referential in that they hold local information, often derived from parameters as initial values.

What other factors could impact subject performance?

The number of occurrences of each name in the body of the method provides an opportunity to deduce information; so I think time to figure out what the method does should less when there are many uses of locals/parameters, compared to when there are few.

The ability of subjects to recognize what the code does is also important, i.e., subject code reading experience.

There are lots of interesting possibilities that can be investigated using this low cost technique.

Linear mixed model fit by REML ['lmerMod']
Formula: response ~ func + treatment + (treatment | subject)
   Data: idxx
 
REML criterion at convergence: 537.8
 
Scaled residuals:
     Min       1Q   Median       3Q      Max
-1.34985 -0.56113 -0.05058  0.60747  2.15960
 
Random effects:
 Groups   Name                      Variance Std.Dev. Corr
 subject  (Intercept)               38748    196.8
          treatmentlocals first     64163    253.3    -0.96
          treatmentparameters first 34810    186.6    -1.00  0.95
 Residual                           43187    207.8
Number of obs: 46, groups:  subject, 9
 
Fixed effects:
                          Estimate Std. Error t value
(Intercept)                  799.0      110.2   7.248
funcindexOfAny              -254.9      126.7  -2.011
funcrepeat                  -560.1      135.6  -4.132
funcreplaceChars            -397.6      126.6  -3.140
funcreverse                 -466.7      123.5  -3.779
funcsubstringBetween        -145.8      125.8  -1.159
treatmentlocals first        342.5      124.8   2.745
treatmentparameters first    267.8      106.0   2.525
 
Correlation of Fixed Effects:
            (Intr) fncnOA fncrpt fncrpC fncrvr fncsbB trtmntlf
fncndxOfAny -0.524
funcrepeat  -0.490  0.613
fncrplcChrs -0.526  0.657  0.620
funcreverse -0.510  0.651  0.638  0.656
fncsbstrngB -0.523  0.655  0.607  0.655  0.648
trtmntlclsf -0.505 -0.167 -0.182 -0.160 -0.212 -0.128
trtmntprmtf -0.495 -0.184 -0.162 -0.184 -0.228 -0.213  0.673

Norfolk Developers Magazine: AI

The first issue of the Norfolk Developers magazine (outside  of a conference) is out now and free to download!

This issue focuses on A.I., a topic we thought a good one to kick off with as everyone has an opinion about Artificial Intelligence, it affects our daily lives (see Dom Davis’ column about arguments with Alexa) and it gave us an excuse to use the awesome robot image on the front cover too.

It is because of people like you that we have  such a thriving tech community in Norwich and Norfolk, a community that has turned our Fine City into a Tech City. Without this passionate and dedicated community, there would be no reason for writers to contribute to this magazine, there would be no market for local companies to place adverts for, there would be no events to report from. Mainly, there would be no one to read it so thank you

Utilisation and non-core team members

SplitMan-2017-09-25-10-06.png

“But we have specialists outside the team, we have to beg… borrow… steal people… we can never get them when we want them, we have to wait, we can’t get them enough.”

It doesn’t matter how much I pontificate about dedicated, stable, consistent teams I hear this again and again. Does nobody listen to me? Does nobody read Xanpan or Continuous Digital?

And I wonder: how much management time is spent arguing over who (which individuals) is doing what this week?

Isn’t that kind of piecemeal resourcing micro-management?

Or is it just making work for “managers” to do?

Is there no better use of management time than arguing about who is doing what? How can the individuals concerned “step up to the plate” and take responsibility if they are pulled this way and that? How can they really “buy in” to work when they don’t know what they doing next week?

Still, there is another answer to the problem: “How do you manage staffing when people need to work on multiple work streams at once?”

Usually this is because some individuals have specialist skills or because a team cannot justify having a full time, 100%, dedicated specialist.

Before I give you the answer lets remind ourselves why the traditional solution can make things worse:

  • When a resource (people or anything else) is scarce a queues are used to allocate the scarce resources and queues create delays
  • Queues create delays in getting the work done – and are bad for morale
  • Queues are an alternative cost: in this case the price comes from the cost-of-delay
  • Queues disrupt schedules and predictability
  • In the delay people try to do something useful but the useful thing is lower value, and may cause more work for others, it can even create conflict when the specialist becomes available

The solution, and it is a generic solution that can be applied whenever some scarce resource (people, beds, runways):

Have more of the scarce resource than is necessary.

So that sounds obvious I guess?

What you want is for there be enough of the scarce resource so that the queues do not form. As an opening gambit have 25% resource more than you expect to need, do not plan to use the scarce resource more than 75% of the available time.

Suppose for example you have several teams, each of who needs a UX designer 1-day a week. At first sight one designer could service five teams but if you did that there would still be queues.

Why?

Because of variability.

Some weeks Team-1 would need a day-and-a-half of the designer, sure some week they would need the designer less than a day but any variability would cause a ripple – or “bullwhip effect”. The disruption caused by variation would ripple out over time.

You need to balance several factors here:

  • Utilisation
  • Variability
  • Wait time
  • Predictability

Even if demand and capacity are perfectly matched then variability will increase wait time which will in turn reduce predictability. If variability and utilisation are high then there will be queues and predicability will be low.

  • If you want 100% utilisation then you have to accept queues (delay) and a loss of predicability: ever landed at Heathrow airport? The airport runs at over 96% utilisation, there isn’t capacity to absorb variability so queues form in the air.
  • If you want to minimise wait time with highly variable work you need low utilisation: why do fire departments have all those fire engines and fire fighters sitting around doing nothing most days?

Running a business, especially a service business, at 100% utilisation is crazy – unless your customers are happy with delays and no predicability.

One of the reasons budget airlines always use the same type of plane and one class of seating is to limit variation. Only as they have gained experience have they added extras.

Anyone who tries to run software development at 100% is going to suffer late delivery and will fail to meet end dates.

How do I know this?

Because I know a little about queuing theory. Queuing theory is a theory in the same way that Einstein’s Theory of Relativity is theory, i.e. it is not a theory, its proven. In both cases mathematics. If you want to know more I recommend Factory Physics.

So, what utilisation can you have?

Well, that is a complicated question. There are a number of parameters to consider and the maths is very complicated. So complicated in fact that mathematicians usually turn to simulation. You don’t need to run a simulation because you can observe the real world, you can experiment for yourself. (Kanban methods allow you to see the queues forming.)

That said: 76% (max).

Or rather: in the simplest queuing theory models queues start to form (and predictability suffers) once utilisation is above 76%.

Given that your environment is probably a lot more complicated then the simplest models you probably need to keep utilisation well below 76% if you want short lead times and predictability.

As a very broad rule of thumb: if you are sharing someone don’t commit more then three-quarters of their time.

And given that, a dedicated specialist doesn’t look so expensive after all. Back to where I came in.

<

blockquote>Read more? Join my mailing list – free updates on blog post, insights, events and offers.

The post Utilisation and non-core team members appeared first on Allan Kelly Associates.

An Almanac of the Internet

My search for software engineering data has turned me into a frequent buyer of second-hand computer books, many costing less than the postage of £2.80. When the following suggestion popped up along-side a search, I could not resist; there must be numbers in there!

internet almanac

The concept of an Almanac will probably be a weird idea to readers who grew up with search engines and Wikipedia. But yes, many years ago, people really did make a living by manually collecting information and selling it in printed form.

One advantage of the printed form is that updating it requires a new copy, the old copy lives on unchanged (unlike web pages); the disadvantage is taking up physical space (one day I will probably abandon this book in a British rail coffee shop).

Where did Internet users hang out in 1997?

top websites 97

The history of the Internet, as it appeared in 1997.

internet history viewed from 1997

Of course, a list of web sites is an essential component of an Internet Almanac:

website list from  1997

Visual Lint 6.0.5.285 has been released

Visual Lint 6.0.5.285 is now available. This is a recommended maintenance update for Visual Lint 6.0, and includes the following changes:
  • Added PC-lint Plus specific compiler indirect files co-rb-vs2008.lnt and co-rb-vs2010.lnt.
  • co-rb-vs2013.lnt and co-rb-vs2015.lnt now undefine the preprocessor symbols _CPPRTTI, _CPPUNWIND and _NATIVE_WCHAR_T_DEFINED which are hardcoded in the Gimpel supplied compiler indirect files. These definitions should not be hardcoded as Visual Lint will automatically define these symbols where required by the project configuration.
  • Added additional suppression directives to the PC-lint Plus indirect file rb-win32-pclint10.lnt.
  • Fixed a bug in the expansion of project variables within the "Command Line" page in the Analysis Configuration Dialog.
  • If a category with no issues (e.g. "Internal Errors" in PC-lint Plus) is selected in the Message Lookup Display the "Title" and "Description" fields will now be correctly cleared.
  • When multiple views of a file is opened for editing in VisualLintGui, the contents of the views are now correctly synchronised.
  • VisualLintGui now captions the MDI child window tabs correctly when there are multiple instances of the same file open.
  • Fixed a bug in VisualLintGui which prevented MDI windows from being correctly configured if more than one view of a particular file was open.
  • Added a "Window | Close" command to VisualLintGui. The Ctrl+W accelerator and MDI tab "Close" command are now mapped to this command rather than "File | Close".
  • Fixed a bug in the VisualLintGui "Window | Close All" and "Window | Close All Except" commands which manifested when more than one view of a particular file was open.
  • The VisualLintGui "Window | New" command is now disabled if the active MDI child window is a web browser window.
  • Updated the PC-lint Plus message database to reflect changes in PC-lint Plus RC2.
Download Visual Lint 6.0.5.285

The public release of PC-lint Plus is imminent!

The PC-lint Plus beta test phase is now complete, with three release candidates having been issued since the beginning of September by Gimpel Software (the PC-lint Plus vendor). The following statement has just appeared on the Gimpel website:

PC-lint Plus Release is Imminent

Our testing of PC-lint Plus is nearing completion and we expect a formal release of the product during the 4th quarter of 2017. Unlike PC-lint and FlexeLint, licensing of PC-lint Plus will be based on a Team License. To get a quote, please send an email to sales@gimpel.com with your company information, and a brief description of the Team, including the name of your Team and the number of developers (including consultants) that will be working on the source code that PC-lint Plus will be analyzing.

What is PC-lint Plus?

PC-lint Plus is a rewrite of PC-lint from the ground up. It combines the clang framework with Gimpel Software's 30+ years of static analysis experience to provide a product that supports the latest C and C++ standards and implements the leading edge analysis technology that Gimpel Software is known for. While PC-lint Plus is a new offering from Gimpel Software, it is largely backward compatible with PC-lint /FlexeLint. In particular, most of the same options and messages are supported by PC-lint Plus, indirect files work the same way, and flagship features such as Value Tracking, User-defined Function Semantics, and Strong Types are all available and many features have been significantly enhanced. Read More
While we are waiting for full details we are continuing our testing of the release candidates and (in particular) refining the PC-lint Plus compiler configuration and suppression files installed with Visual Lint. As with PC-lint 9.0, in due course we aim to include a full set of indirect files in the Visual Lint installer - including compiler indirect files for all versions of Visual Studio back to Visual C++ 6.0 - regardless of whether Gimpel provide them (so far PC-lint Plus only includes compiler indirect files for Microsoft Visual Studio 2012, 2013 and 2015). We have already authored compiler indirect files for Visual Studio 2008 and 2010, and others will follow. If you already have a PC-lint Plus RC installation, Visual Lint 6.0 can already analyse projects using both the 32 bit (pclp32.exe) and 64 bit (pclp64.exe) versions of PC-lint Plus and already includes a number of other features (for example multicore per-project analysis) to specifically support PC-lint Plus installations. If you have any specific queries about PC-lint Plus support, just let us know.

Further Still On A Calculus Of Differences – student

For some time now my fellow students and I have been whiling away our spare time considering the similarities of the relationships between sequences and series and those between the derivatives and integrals of functions. Having defined differential and integral operators for a sequence sn with

  Δ sn = sn - sn-1

and
  n
  Δ-1 sn = Σ si
  i = 1
where Σ is the summation sign, we found analogues for the product rule, the quotient rule and the rule of integration by parts, as well as formulae for the derivatives and integrals of monomial sequences, being those whose terms are non-negative integer powers of their indices, and higher order, or repeated, derivatives and integrals in general.

We have since spent some time considering how we might solve equations relating sequences to their derivatives, known as differential equations when involving functions, and it is upon our findings that I shall now report.

Just::Thread Pro v2.5.0 released with coroutines support

I am pleased to announce that Just::Thread Pro v2.5.0 has been released. This adds support for gcc 7, clang 4.0 and clang 5.0, but the big change with this version is the support for coroutines with Microsoft Visual Studio 2017, and clang 5.0 on ubuntu when used with libc++ 5.0.

Just::Thread Pro is our C++ concurrency extensions library which provides an Actor framework for easier concurrency, along with concurrent data structures: a thread-safe queue, and concurrent hash map, and a wrapper for ensuring synchronized access to single objects.

It also includes the new facilities from the Concurrency TS:

Coroutines support is here!

V2.5.0 adds support for coroutines with Microsoft Visual Studio 2017 and clang 5.0. This means that you can now use co_await to wait for a std::experimental::future, and can create coroutines that return a std::experimental::future.

Supported compilers

Just::Thread Pro is now fully supported on the following compiler/OS combinations (32-bit and 64-bit):

  • Microsoft Visual Studio 2015 for Windows
  • Microsoft Visual Studio 2017 for Windows
  • gcc 5 for Ubuntu 14.04 or later
  • gcc 6 for Ubuntu 14.04 or later
  • gcc 7 for Ubuntu 14.04 or later
  • clang 3.8 for Ubuntu 16.04 or later
  • clang 3.9 for Ubuntu 16.04 or later
  • clang 4.0 for Ubuntu 16.04 or later
  • clang 5.0 for Ubuntu 16.04 or later with libc++ or libstdc++
  • gcc 5 for Fedora 22 and 23
  • gcc 6 for Fedora 24 and 25
  • gcc 7 for Fedora 26
  • clang 3.8 for Fedora 24
  • clang 3.9 for Fedora 25
  • clang 4.0 for Fedora 26

Just::Thread Pro v2.2 is also supported with the Just::Thread compatibility library on the following compiler/OS combinations:

  • Microsoft Visual Studio 2005, 2008, 2010, 2012 and 2013 for Windows
  • TDM gcc 4.5.2, 4.6.1 and 4.8.1 for Windows
  • g++ 4.3 or later for Ubuntu 9.04 or later
  • g++ 4.4 or later for Fedora 13 or later
  • g++ 4.4 for Centos 6
  • MacPorts g++ 4.3 to 4.8 on MacOSX Snow Leopard or later

All licences include a free upgrade to point releases, so if you purchase now you'll get a free upgrade to all 2.x releases of Just::Thread Pro. Purchasers of the older Just::Thread library (now called the compatibility library) may upgrade to Just::Thread Pro for a small fee.

Posted by Anthony Williams
[/ news /] permanent link
Tags: , , ,

| Stumble It! stumbleupon logo | Submit to Reddit reddit logo | Submit to DZone dzone logo

Comment on this post

Follow me on Twitter

Emacs 25.3 released

Emacs 25.3 has been released on Monday. Given that it’s a security fix I’m downloading the source as I write this. If you’re using the latest Emacs I’d recommend you update your Emacs. The vulnerability as been around since Emacs 19.29, you probably want to upgrade anyway. Build instructions for Ubuntu and friends are the […]

The post Emacs 25.3 released appeared first on The Lone C++ Coder's Blog.

Detection Idiom – A Stopgap for Concepts

Concepts is a proposed C++ feature which allows succinct, expressive, and powerful constraining of templates. They have been threatening to get in to C++ for a long time now, with the first proposal being rejected for C++11. They were finally merged in to C++20 a few months ago, which means we need to hold on for another few years before they’re in the official standard rather than a Technical Specification. In the mean time, there have been various attempts to implement parts of concepts as a library so that we can have access to some of the power of Concepts without waiting for the new standard. The detection idiom, which is part of the Library Fundamentals 2 Technical Specification, is one such solution. This post will outline the utility of it, and show the techniques which underlie its implementation.


A problem

We are developing a generic library. At some point on our library we need to calculate the foo factor for whatever types the user passes in.

template <class T>
int calculate_foo_factor (const T& t);

Some types will have specialized implementations of this function, but for types which don’t we’ll need some generic calculation.

struct has_special_support {
  int get_foo() const;
};

struct will_need_generic_calculation {
  // no get_foo member function
};

Using concepts we could write calculate_foo_factor like so:

template <class T>
concept bool SupportsFoo = requires (T t) {
  { t.get_foo() } -> int;
};

template <SupportsFoo T>
int calculate_foo_factor (const T& t) {
    return t.get_foo();
}

template <class T>
int calculate_foo_factor (const T& t) {
    // insert generic calculation here
    return 42;
}

This is quite succinct and clear: SupportsFoo is a concept which checks that we can call get_foo on t with no arguments, and that the type of that expression is int. The first calculate_foo_factor will be selected by overload resolution for types which satisfy the SupportsFoo concept, and the second will be chosen for those which don’t.

Unfortunately, our library has to support C++14. We’ll need to try something different. I’ll demonstrate a bunch of possible solutions to this problem in the next section. Some of them may seem complex if you aren’t familiar with the metaprogramming techniques used, but for now, just note the differences in complexity and abstraction between them. The metaprogramming tricks will all be explained in the following section.

Solutions

Here’s a possible solution using expression SFINAE:

namespace detail {
  template <class T>
  auto calculate_foo_factor (const T& t, int)
    -> decltype(std::declval<T>().get_foo()) {
    return t.get_foo();
  }

  template <class T>
  int calculate_foo_factor (const T& t, ...) {
    // insert generic calculation here
    return 42;
  }
}

template <class T>
int calculate_foo_factor (const T& t) {
  return detail::calculate_foo_factor(t, 0);
}

Well, it works, but it’s not exactly clear. What’s the int and ... there for? What’s std::declval? Why do we need an extra overload? The answers are not the important part here; what is important is that unless you’ve got a reasonable amount of metaprogramming experience, it’s unlikely you’ll be able to write this code offhand, or even copy-paste it without error.

The code could be improved by abstracting out the check for the presence of the member function into its own metafunction:

template <class... Ts>
using void_t = void;

template <class T, class=void>
struct supports_foo : std::false_type{};

template <class T>
struct supports_foo<T, void_t<decltype(std::declval<T>().get_foo())>>
: std::true_type{};

Again, some more expert-only template trickery which I’ll explain later. Using this trait, we can use std::enable_if to enable and disable the overloads as required:

template <class T, std::enable_if_t<supports_foo<T>::value>* = nullptr>
auto calculate_foo_factor (const T& t) {
  return t.get_foo();
}

template <class T, std::enable_if_t<!supports_foo<T>::value>* = nullptr>
int calculate_foo_factor (const T& t) {
  // insert generic calculation here
  return 42;
}

This works, but you’d need to understand how to implement supports_foo, and you’d need to repeat all the boilerplate again if you needed to write a supports_bar trait. It would be better if we could completely abstract away the mechanism for detecting the member function and just focus on what we want to detect. This is what the detection idiom provides.

template <class T>
using foo_t = decltype(std::declval<T>().get_foo());

template <class T>
using supports_foo = is_detected<foo_t,T>;

is_detected here is part of the detection idiom. Read it as “is it valid to instantiate foo_t with T?” std::declval<T>() essentially means “pretend that I have a value of type T” (more on this later). This still requires some metaprogramming magic, but it’s a whole lot simpler than the previous examples. With this technique we can also easily check for the validity of arbitrary expressions:

template <class T, class U>
using equality_t = decltype(std::declval<T>() == std::declval<U>());

template <class T, class U>
using supports_equality = is_detected<equality_t, T, U>;

struct foo{};
struct bar{};
struct baz{};

bool operator== (foo, bar);

static_assert( supports_equality<foo,bar>::value, "wat");
static_assert(!supports_equality<foo,baz>::value, "wat");

We can also compose traits using std::conjunction:

template <class T>
using is_regular = std::conjunction<std::is_default_constructible<T>,
                                    std::is_copy_constructible<T>,
                                    supports_equality<T,T>,
                                    supports_inequality<T,T>, //assume impl
                                    supports_less_than<T,T>>; //ditto

If you want to use is_detected today, then you can check if your standard library supports std::experimental::is_detected. If not, you can use the implementation from cppreference or the one which we will go on to write in the next section. If you aren’t interested in how this is written, then turn back, for here be metaprogramming dragons.


Metaprogramming demystified

I’ll now work backwards through the metaprogramming techniques used, leading up to implementing is_detected.

Type traits and _v and _t suffixes

A type trait is some template which can be used to get information about characteristics of a type. For example, you can find out if some type is an arithmetic type using std::is_arithmetic:

template <class T>
void foo(T t) {
     static_assert(std::is_arithmetic<T>::value, "Argument must be of an arithmetic type");
}

Type traits either “return” types with a ::type member alias, or values with a ::value alias. _t and _v suffixes are shorthand for these respectively. So std::is_arithmetic_v<T> is the same as std::is_arithmetic<T>::value, std::add_pointer_t<T> is the same as typename std::add_pointer<T>::type1.

decltype specifiers

decltype gives you access to the type of an entity or expression. For example, with int i;, decltype(i) is int.

std::declval trickery

std::declval is a template function which helps create values inside decltype specifiers. std::declval<foo>() essentially means “pretend I have some value of type foo”. This is needed because the types you want to inspect inside decltype specifiers may not be default-constructible. For example:

struct foo {
    foo() = delete;
    foo(int);
    void do_something();
};

decltype(foo{}.do_something())               // invalid
decltype(std::declval<foo>().do_something()) // fine

SFINAE and std::enable_if

SFINAE stands for Substitution Failure Is Not An Error. Due to this rule, some constructs which are usually hard compiler errors just cause a function template to be ignored by overload resolution. std::enable_if is a way to gain easy access to this. It takes a boolean template argument and either contains or does not contain a ::type member alias depending on the value of that boolean. This allows you to do things like this:

template <class T>
std::enable_if_t<std::is_integral_v<T>> foo(T t);

template <class T>
std::enable_if_t<std::is_floating_point_v<T>> foo(T t);

If T is integral, then the second overload will be SFINAEd out, so the first is the only candidate. If T is floating point, then the reverse is true.

Expression SFINAE

Expression SFINAE is a special form of SFINAE which applies to arbitrary expressions. This is what allowed this example from above to work:

namespace detail {
  template <class T>
  auto calculate_foo_factor (const T& t, int)
    -> decltype(std::declval<T>().to_foo()) {
    return t.get_foo();
  }

  template <class T>
  int calculate_foo_factor (const T& t, ...) {
    // insert generic calculation here
    return 42;
  }
}

int calculate_foo_factor (const T& t) {
  return calculate_foo_factor(t, 0);
}

The first overload will be SFINAEd out if calling to_foo on an instance of T is invalid. The difficulty here is that both overloads will be valid if the too_foo call is valid. For this reason, we add some dummy parameters to the overloads (int and ...) to specify an order for overload resolution to follow2.

void_t magic

void_t is a C++17 feature (although it’s implementable in C++11) which makes writing traits and using expression SFINAE a bit easier. The implementation is deceptively simple3:

template <class... Ts>
using void_t = void;

You can see this used in this example which we used above:

template <class T, class=void>
struct supports_foo : std::false_type{};

template <class T>
struct supports_foo<T, void_t<decltype(std::declval<T>().get_foo())>>
: std::true_type{};

The relevant parts are the class=void and void_t<...>. If the expression inside void_t is invalid, then that specialization of supports_foo will be SFINAEd out, so the primary template will be used. Otherwise, the whole expression will be evaluated to void due to the void_t and this will match the =void default argument to the template. This gives a pretty succinct way to check arbitrary expressions.

That covers all of the ingredients we need to implement is_detected.

The implementation

For sake of simplicity I’ll just implement is_detected rather than all of the entities which the Lib Fundamentals 2 TS provides.

namespace detail {
  template <template <class...> class Trait, class Enabler, class... Args>
  struct is_detected : std::false_type{};

  template <template <class...> class Trait, class... Args>
  struct is_detected<Trait, void_t<Trait<Args...>>, Args...> : std::true_type{};
}

template <template <class...> class Trait, class... Args>
using is_detected = typename detail::is_detected<Trait, void, Args...>::type;

Let’s start with that last alias template. We take a variadic template template parameter (yes really) called Trait and any number of other type template arguments. We forward these to our detail::is_detected helper with an extra void argument which will act as the class=void construct from the previous section on void_t4. We then have a primary template which will “return” false as the result of the trait. The magic then happens in the following partial specialization. Trait<Args...>> is evaluated inside void_t. If instantiating Traits with Args... is invalid, then the partial specialization will be SFINAEd out, and if it’s valid, it’ll evaluate to void due to the void_t. This successfully abstracts away the implementation of the detection and allows us to focus on what we want to detect.


That covers it for the detection idiom. This is a handy utility for clearing up some hairy metaprogramming, and is even more useful since it can be implemented in old versions of the standard. Let me know if there are any other parts of the code you’d like explained down in the comments or on Twitter.


  1. See here for information on why the typename keyword is needed in some places. 

  2. Have a look here for a better way to do this. 

  3. Due to standards defect CWG1558, implementations needed an extra level of abstraction to work on some compilers. 

  4. The reason we can’t use the same trick is that parameter packs must appear at the end of the template parameter list, so we need to put the void in the middle. 

Investing in the gcc C++ front-end

I recently found out that RedHat are investing in improving the C++ front-end of gcc, i.e., management have assigned developers to work in this area. What’s in it for RedHat? I’m told there are large companies (financial institutions feature) who think that using some of the features added to recent C++ standards (these have been appearing on a regular basis) will improve the productivity of their developers. So, RedHat are hoping this work will boost their reputation and increase their sales to these large companies. As an ex-compiler guy (ex- in the sense of being promoted to higher levels that require I don’t do anything useful), I am always in favor or companies paying people to work on compilers; go RedHat.

Is there any evidence that features that have been added to any programming language improved developer productivity? The catch to this question is defining programmer productivity. There have been several studies showing that if productivity is defined as number of assembly language lines written per day, then high level languages are more productive than assembler (the lines of assembler generated by the compiler were counted, which is rather compiler dependent).

Of the 327 commits made this year to the gcc C++ front-end, by 29 different people, 295 were made by one of 17 people employed by RedHat (over half of these commits were made by two people and there is a long tail; nine people each made less than four commits). Measuring productivity by commit counts has plenty of flaws, but has the advantage of being easy to do (thanks Jonathan).

Continuous Digital & Project Myopia

Reprojects-2017-09-11-10-56.png

This seems a little back to the future… those of you who have been following the evolution of Continuous Digital will know the book grew out of the #NoProjects meme and my extended essay.

I think originally the book title was #NoProjects then was Correcting Project Myopia, then perhaps something else and finally settled down to Continuous Digital. The changing title reflected my own thinking, thinking that continued to evolve.

As that thinking has evolved the original #NoProjects material has felt more and more out of place in Continuous Digital. So I’ve split it out – Project Myopia is back as a stand alone eBook and you can buy it today.

More revisions of Continuous Digital will appear as I refactor the book. Once this settles down I’ll edit through Project Myopia. A little material may move between the two books but hopefully not much.

Now the critics of #NoProjects will love this because they complain that #NoProjects tells you what not to do, not what to do. In a way I agree with them but at the same time the first step to solving a problem is often to admit you have a problem. Project Myopia is a discussion of the problem, it is a critique. Continuous Digital is the solution and more than that.

Splitting the book in two actually helps demonstrate my whole thesis.

For a start it is difficult to know when a work in progress, iterating, self-published, early release book is done. My first books – Business Patterns and Changing Software Development – were with a traditional publisher. They were projects with a start and a finish. Continuous Digital isn’t like that, it grows, it evolves. That is possible because Continuous Digital is itself digital, Business Patterns and Changing Software Development had to stop because they were printed.

Second Continuous Digital is already a big book – much bigger than most LeanPub books – and since I advocate “lots of small” over “few big” it makes sense to have two smaller books rather than one large.

Third, and most significantly, this evolution is a perfect example of one of my key arguments: some types of “problem” are only understood in terms of the solution. Defining the solution is necessary to define the problem.

The solution and problem co-evolve.

In the beginning the thesis was very much based around the problems of the project model, and I still believe the project model has serious problems. In describing a solution – Continuous Digital – a different problem became clear: in a digital age businesses need to evolve with the technology.

Projects have end dates, hopefully your business, your job, doesn’t.

If you like you can buy both books together at a reduced price right now.

Read more? Join my mailing list – free updates on blog post, insights, events and offers.

The post Continuous Digital & Project Myopia appeared first on Allan Kelly Associates.

Libc++ v5 PPA for Ubuntu now available

I am please to announce that I have made v5.0 of libc++ available for Ubuntu Linux via a PPA. Initially, the binaries are only available for Ubuntu 16.04 Xenial.

I was frustrated at the lack of an up-to-date build of libc++ for linux, especially with the v5.0 release of clang. Even though the LLVM project provide Ubuntu packages for clang, they don't provide one for libc++, and the version in the Ubuntu repositories is woefully out of date (v3.7). I therefore decided to package it myself, in this PPA.

This is especially good, since in combination with clang v5.0, it provides support for the Coroutines TS!

Enjoy!

Posted by Anthony Williams
[/ news /] permanent link
Tags: , , , ,

| Stumble It! stumbleupon logo | Submit to Reddit reddit logo | Submit to DZone dzone logo

Comment on this post

Follow me on Twitter

The book of five rings

is an excellent book by Miyamoto Musashi, translated by Thomas Cleary (isbn 1-57062-748-7). As usual I'm going to quote from a few pages:
Preface: In common parlance, to do something with a real sword means to do it with utmost earnestness… The Book of Five Rings… explicitly intended to symbolise processes of struggle and mastery in all concerns and walks of life.
The martial way of life practiced by warriors is based on excelling others in anything and everything.
Fourth is the way of the artisan. In terms of the way of the carpenter, this involves skilful construction of all sorts of tools, knowing how to use each tool skilfully, drawing up plans correctly by means of the square and the ruler, making a living by diligent practice of the craft… practice unremittingly.
You should observe reflectively, with overall awareness of the large picture a well as precise attention to small details.
Having attained a principle, one detaches from the principles; thus one has spontaneous independence in the science of martial arts and naturally attains marvels.
As human beings, it is essential for each of us to cultivate and polish our individual path.
Observation and perception are two separate things.
It is essential to be relaxed in body and mind.
If you get to feeling snarled up and are making no progress, you toss your mood away and think in your heart that you are starting everything anew.
In my military science, it is essential that the physical aspect and the mental state both be simple and direct.
Whether in large- or small-scale military science, there is no narrow focus of the vision. As I have already written, by finicky narrowness of focus, you forget about bigger things and get confused, thus letting certain victory escape you.
Things stick in your mind because of being in doubt.
The practice of all the arts is for the purpose of clearing away what is on your mind. In the beginning, you do not know anything, so paradoxically you do not have any questions on your mind. Then, when you get into studies, there is something on your mind and you are obstructed by that. This makes everything difficult to do.


The book of five rings

is an excellent book by Miyamoto Musashi, translated by Thomas Cleary (isbn 1-57062-748-7). As usual I'm going to quote from a few pages:
Preface: In common parlance, to do something with a real sword means to do it with utmost earnestness… The Book of Five Rings… explicitly intended to symbolise processes of struggle and mastery in all concerns and walks of life.
The martial way of life practiced by warriors is based on excelling others in anything and everything.
Fourth is the way of the artisan. In terms of the way of the carpenter, this involves skilful construction of all sorts of tools, knowing how to use each tool skilfully, drawing up plans correctly by means of the square and the ruler, making a living by diligent practice of the craft… practice unremittingly.
You should observe reflectively, with overall awareness of the large picture a well as precise attention to small details.
Having attained a principle, one detaches from the principles; thus one has spontaneous independence in the science of martial arts and naturally attains marvels.
As human beings, it is essential for each of us to cultivate and polish our individual path.
Observation and perception are two separate things.
It is essential to be relaxed in body and mind.
If you get to feeling snarled up and are making no progress, you toss your mood away and think in your heart that you are starting everything anew.
In my military science, it is essential that the physical aspect and the mental state both be simple and direct.
Whether in large- or small-scale military science, there is no narrow focus of the vision. As I have already written, by finicky narrowness of focus, you forget about bigger things and get confused, thus letting certain victory escape you.
Things stick in your mind because of being in doubt.
The practice of all the arts is for the purpose of clearing away what is on your mind. In the beginning, you do not know anything, so paradoxically you do not have any questions on your mind. Then, when you get into studies, there is something on your mind and you are obstructed by that. This makes everything difficult to do.


Definition of Ready considered harmful

Small-iStock-502805668-2017-09-6-12-58.jpg

Earlier this week I was with a team and discussion turned to “the definition of ready.” This little idea has been growing more and more common in the last couple of years and while I like the concept I don’t recommend it. Indeed I think it could well reduce Agility.

To cut to the chase: “Definition of ready” reduces agility because it breaks up process flow, assumes greater role specific responsibilities, introduces more wait states (delay) and potentially undermines business-value based prioritisation.

The original idea builds on “definition of done”. Both definitions are a semi-formal checklists agreed by the team which are applied to pieces of work (stories, tasks, whatever). Before any piece of work is considered “done” it should satisfy the definition of done. So the team member who has done a piece of work should be able to mentally tick each item on the checklist. Typically a definition of done might contain:

 

  • Story implemented
  • Story satisfies acceptance criteria
  • Story has been seen and approved by the product owner
  • Code is passing all unit and acceptance tests

Note I say “mentally” and I call these lists “semi formal” because if you start having a physical checklist for each item, physically ticking the boxes, perhaps actually signing them, and presumably filing the lists or having someone audit them then the process is going to get very expensive quickly.

So far so good? – Now why don’t I like definition of ready?

On the one hand definition of ready is a good idea: before work begins on any story some pre-work has been done on the story to ensure it is “ready for development” – yes, typically this is about getting stories ready for coding. Such a check-list might say:

 

  • Story is written in User Story format with a named role
  • Acceptance criteria have been agreed with product owner
  • Developer, Tester and Product owner have agreed story meaning

Now on the other hand… even doing these means some work has been done. Once upon a time the story was not ready, someone, or some people have worked on the story to make it ready. When did this happen? Getting this story ready has already detracted from doing other work – work which was a higher priority because it was scheduled earlier.

Again, when did this happen?

If the story became “ready” yesterday then no big deal. The chances are that little has changed.

But if it became ready last week are you sure?

And what if it became ready last month? Or six months ago?

The longer it has been ready the greater the chance that something has changed. If we don’t check and re-validate the “ready” state then there is a risk something will have changed and be done wrong. If we do validate then we may well be repeating work which has already been done.

In general, the later the story becomes “ready” the better. Not only does it reduce the chance that something will change between becoming “ready” and work starting but it also minimises the chance that the story won’t be scheduled at all and all the pre-work was wasted.

More problematic still: what happens when the business priority is for a story that is not ready?

Customer: Well Dev team story X is the highest priority for the next sprint
Scrum Master: Sorry customer, Story X does not meet the definition of ready. Please choose another story.
Customer: But all the other stories are worth less than X so I’d really like X done!

The team could continue to refuse X – and sound like an old style trade unionist in the process – or they could accept X , make it ready and do it.

Herein lies my rule of thumb:

 

If a story is prioritised and scheduled for work but is not considered “ready” then the first task is to make it ready.

Indeed this can be generalised:

 

Once a story is prioritised and work starts then whatever needs doing gets done.

This simplifies the work of those making the priority calls. They now just look at the priority (i.e. business value) or work items. They don’t need to consider whether something is ready or not.

It also eliminates the problem of: when.

Teams which practise “definition of ready” usually expect their product owner to make stories ready before the iteration planning meeting, and that creates the problems above. Moving “make ready” inside the iteration, perhaps as a “3 Amigos” sessions after the planning meeting, eliminates this problem.

And before anyone complains saying “How can I estimate something thing that is not prepared?” let me point out you can. You are just estimating something different:

 

  • When you estimate “ready” stories you are estimating the time it takes to move a well formed story from analysis-complete to coding-complete
  • When up estimate an “unready” story you are estimating the time it takes to move a poorly formed story from its current state to coding-complete

I would expect the estimates to be bigger – because there is more work – and I would expect the estimates to be subject to more variability – because the initial state of the story is more variable. But is still quite doable, it is an estimate, not a promise.

I can see why teams adopt definition of ready and I might even recommend it myself but I’d hope it was an temporary measure on the way to something better.

In teams with broken, role based process flows then a definition of done for each stage can make sense. The definition of done at the end of one activity is the definition of ready for the next. For teams adopting Kanban style processes I would recommend this approach as part of process/board set-up. But I also hope that over time the board columns can be collapsed down and definitions dropped.

Read more? Join my mailing list – free updates on blog post, insights, events and offers.

The post Definition of Ready considered harmful appeared first on Allan Kelly Associates.

Learning C++

C++ is one of the most powerful programming languages in the world. However, it can also be one of the most daunting for beginners. There is so much to learn, so many corner cases, so many little languages within the language itself, that mastering it can seem an insurmountable task. Fortunately, it’s not quite as impossible as it looks. This post outlines some resources and advice for learning C++ today. Some of these also will be useful for those who have used C++ in the past and want to get familiar with modern C++, which is like an entirely different language.

If you want to learn C++ from scratch, don’t rely on random online tutorials. Get a good book. One of these. I read the entirety of Accelerated C++ the day before the interview for my current job, and it was enough to get stuck in to real C++ development. It’s a bit outdated now, (i.e. it doesn’t cover C++11/14/17) but is easily the most important book in my career development. The Effective C++ series by Scott Meyers should be mandatory reading for C++ developers, particularly Effective Modern C++, which is the go-to introduction to C++11 and 14. If you prefer online resources, use reputable ones like Kate Gregory’s Pluralsight courses rather than whatever Google throws out. There are a lot of terrible C++ tutorials out there which will only harm you. A good rule of thumb: if you see #include <iostream.h>, run1.

However, no matter how much you read, you need to practice in order to improve. I find it helpful to have a mix of small, self-contained problems to try out new techniques, and larger applications to make sure you can use those ideas in a real-world context. Kind of like unit tests and integration tests for your programming skills; can you use these ideas in isolation, and can you fit them together into something comprehensible and maintainable.

Stack Overflow is a great source of skill unit tests. I learned most of my initial metaprogramming knowledge by answering Stack Overflow questions incorrectly and having people tell me why I was wrong. Unfortunately it can be an unforgiving community, and isn’t friendly to very simple questions. Be sure to read through the rules thoroughly before contributing, and feel free to send me an answer you were going to post to get feedback on it if you’re worried!

For skill integration tests, I’d recommend coming up with an application in a domain you’re interested in and trying to develop it from start to finish. If you haven’t done much of this kind of thing before, it can be a daunting task. The main piece of advice I have on this is to ensure your application has a strict scope so that you always know what you’re aiming towards and when you’re finished. Set a number of checkpoints or milestones so that you have explicit tasks to work on and don’t get lost. Some ideas for fun projects:

  • A compiler for a simple C subset
  • A mini text editor
  • Implement some simplified standard library classes, like std::vector
  • A clone of a simple video game like Snake
  • An ELF file manipulation tool

One of the most valuable knowledge sources is a good C++ community to be part of. Find out if your city or a nearby one has a User Group and go along to meet other developers. They might even have a mentorship program or advice channels for people learning the language. Definitely get on to the C++ Slack Channel and lurk, ask questions, join in discussions. Many of the foremost C++ experts in the world hang out there, and it’s also a very friendly and welcoming environment to those learning. There are even dedicated subchannels for those learning or having problems debugging C++.

Don’t be discouraged when you find things difficult, and, trust me, you will. C++ is hard, and even the experts get things wrong, or can’t even agree on what’s right.

If you do get good, pass on the knowledge. Start a blog, give advice, mentor those who need it, talk at conferences. You do not need to be an expert to do these things, and we need more speakers and writers from a diverse set of experience levels, as well as in background, race, gender, sexuality. The major C++ conferences all have codes of conduct and organisers who care about these issues. Many of them also have student or accessibility tickets available for those who need further assistance to attend. If in doubt, just ask the organisers, or drop me a message and I can get in touch with the relevant people.

Finally, find some good, regular blogs to learn from. Here are some I would recommend:

I hope that this post helps you in your journey to learn C++. It’s a long, but very rewarding path, with a lot of wonderful people to meet, work with, and debate metaprogramming techniques with along the way.


  1. iostream.h rather than just iostream the name for the header before C++ was standardised. People are still writing tutorials which teach using this form and ancient compilers. 

Unit Testing with Mocha, a local instance of dynamoDB & Promises

I'm writing the backend for my current iOS App in Javascript using node.js, AWS Lambda along with DynamoDB.

My AWS Lambda code is mostly AWS Lambda agnostic except for the initial handler methods, this makes them fairly testable outside of AWS. However, they depend on the DynamoDB. It's quite easy to write Unit Tests that run against a live version of DynamoDB but I wanted to run against a local instance, ideally an in-memory instance so that it would be quick (not that running against a real instance is that slow) and so that I could have a clean dB each time.

NOTES

  • As these tests are running against a dB it might be more accurate to call them Integration Tests implemented using a Unit Testing framework but I'll refer to them as Unit Tests (UTs).
  • This is running on MacOS
  • I don't Unit Test the actual AWS Lambda function. Instead I export the underlying functions & objects that the AWS Lambda uses and Unit Test these.
  • I'm a JavaScript, Node, AWS n00b so this if you spot something wrong or that'd bad please comment.
  • I don't like the callback pyramid so I use Promises where I can. I would use async and await but the latest version of Node that AWS Lambda supports doesn't support them :-(

In order to run this locally you'll need:
My goal was to have a clean dB for each individual Unit Test. The simplest way to achieve this I thought was to create an in-memory of dynamoDB and destroy it after each Unit Test.


This uses the child_process npm package to create an instance before each test, store the handle to the process in local-ish variable and following the tasks just kill it. The important points here are that the '-inMemory' option is used meaning that when the dB instance is killed and another re-started everything is effectively wiped without having to do everything.

The problem I had with this approach is that in addition to creating the dB each time I also needed to create a table. Whilst the documentation for local dynamoDB says that one of the differences between the AWS hosted & the local versions is that CreateTable completes immediately it seems that the function does indeed complete immediately the table isn't immediately available. This meant the UT using the table often failed with:

1) awsLambdaToTest The querying an empty db for a user Returns{}:
     ResourceNotFoundException: Cannot do operations on a non-existent table

I'm going to jump ahead and show the completed Unit Test file and explain the various things I had to do in order to get it working. This shows the tests.


before()/after() - creating/destroying dynamoDB


Rather than attempting to create & destroy the dB for each Unit Test I settled with creating it once per  Unit Test file. This is handled in the begin() & after() functions. Here the local instance of dynamoDB is spawned using the child_process package and reference to the process retained. This is then used to kill it afterwards. The important point to note here is the use of the sleep package & function.

I found when I had multiple test files, each with their own begin() & after() functions that did the same as these, even though kill had purported to have killed the processed (I checked the killed flag) it seemed the process hadn't died immediately. This meant that the before() function in the next set of tests would succesfully connect to the dying instance of dynamoDB. Then later when any operation was performed it would just hang until Mocha timed-out the Unit Test/before handler. I tried various ways to detect that the process was really dead but none worked so settled for a sleep.

beforeEach()/afterEach() - creating/destroying the table

Where possible I use Promises. Mocha handles promises quite simply for both hooks (the before*/after* functions) and Unit Tests. The key is to make sure to return the final promise (or pass in the done parameter & call it - though I don't use this mechanism).

Looking at the beforeEach() function createTable() is called which returns a promise (from the AWS.Request type that aws-sdk.DynamoDB.createTable() returns. This promise is then chained too by the synchronous waitFor method. This polls the dB for state of the table. The returned promise will not complete until the table has been created and waitFor has completed.

I am not convinced that waitFor is needed. According to the AWS vs Local DynamoDB guide for local instances tables are created immediately. I added this check as occasionally I was getting resources errors like the one earlier. However, I think the cause for that was because I forgot the return statement before the call to createTable() meaning no Promise was returned to Mocha so it thought the beforeEach() function had completed. I have removed this since in my real Unit Tests and they all seem to work.

Unit Tests

That's really it. The hard part wasn't writing the UTs but getting a local instance of DynamoDB running with the table that the functions to test used in the correct state. Again, due to the functions being tested usually returning promises themselves it is necessary to return Promise. The assertion(s) are made synchronously in a then continuation chained to Promise returned from the function being tested and Promise from the whole chain returned.

If an assertion returns false then even though it's within a continuation Mocha detects this and the test fails. If the function under test throws then Mocha also catches this and the test fails.

And finally

timeout

There's also the this.timeout(5000); at the top of the tests for this file. By default Mocha has 2 second timeout for all tests and hooks. By having the 1 second sleep for starting the dB it's possible for the hook to timeout then causing everything else to fail. The 5 second timeout protects against this.

localhost

When creating the local instance it uses the default port of 8000 and is accessible via the localhost hostname alias. This is used to access the dB when creating the reference to it in before(). A local aws.config is also constructed in each of the functions that access the dB in the actual code under test below.

The actual code being tested


Unit Testing with Mocha, a local instance of dynamoDB & Promises

I'm writing the backend for my current iOS App in Javascript using node.js, AWS Lambda along with DynamoDB.

My AWS Lambda code is mostly AWS Lambda agnostic except for the initial handler methods, this makes them fairly testable outside of AWS. However, they depend on the DynamoDB. It's quite easy to write Unit Tests that run against a live version of DynamoDB but I wanted to run against a local instance, ideally an in-memory instance so that it would be quick (not that running against a real instance is that slow) and so that I could have a clean dB each time.

NOTES

  • As these tests are running against a dB it might be more accurate to call them Integration Tests implemented using a Unit Testing framework but I'll refer to them as Unit Tests (UTs).
  • This is running on MacOS
  • I don't Unit Test the actual AWS Lambda function. Instead I export the underlying functions & objects that the AWS Lambda uses and Unit Test these.
  • I'm a JavaScript, Node, AWS n00b so this if you spot something wrong or that'd bad please comment.
  • I don't like the callback pyramid so I use Promises where I can. I would use async and await but the latest version of Node that AWS Lambda supports doesn't support them :-(

In order to run this locally you'll need:
My goal was to have a clean dB for each individual Unit Test. The simplest way to achieve this I thought was to create an in-memory of dynamoDB and destroy it after each Unit Test.


This uses the child_process npm package to create an instance before each test, store the handle to the process in local-ish variable and following the tasks just kill it. The important points here are that the '-inMemory' option is used meaning that when the dB instance is killed and another re-started everything is effectively wiped without having to do everything.

The problem I had with this approach is that in addition to creating the dB each time I also needed to create a table. Whilst the documentation for local dynamoDB says that one of the differences between the AWS hosted & the local versions is that CreateTable completes immediately it seems that the function does indeed complete immediately the table isn't immediately available. This meant the UT using the table often failed with:

1) awsLambdaToTest The querying an empty db for a user Returns{}:
     ResourceNotFoundException: Cannot do operations on a non-existent table

I'm going to jump ahead and show the completed Unit Test file and explain the various things I had to do in order to get it working. This shows the tests.


before()/after() - creating/destroying dynamoDB


Rather than attempting to create & destroy the dB for each Unit Test I settled with creating it once per  Unit Test file. This is handled in the begin() & after() functions. Here the local instance of dynamoDB is spawned using the child_process package and reference to the process retained. This is then used to kill it afterwards. The important point to note here is the use of the sleep package & function.

I found when I had multiple test files, each with their own begin() & after() functions that did the same as these, even though kill had purported to have killed the processed (I checked the killed flag) it seemed the process hadn't died immediately. This meant that the before() function in the next set of tests would succesfully connect to the dying instance of dynamoDB. Then later when any operation was performed it would just hang until Mocha timed-out the Unit Test/before handler. I tried various ways to detect that the process was really dead but none worked so settled for a sleep.

beforeEach()/afterEach() - creating/destroying the table

Where possible I use Promises. Mocha handles promises quite simply for both hooks (the before*/after* functions) and Unit Tests. The key is to make sure to return the final promise (or pass in the done parameter & call it - though I don't use this mechanism).

Looking at the beforeEach() function createTable() is called which returns a promise (from the AWS.Request type that aws-sdk.DynamoDB.createTable() returns. This promise is then chained too by the synchronous waitFor method. This polls the dB for state of the table. The returned promise will not complete until the table has been created and waitFor has completed.

I am not convinced that waitFor is needed. According to the AWS vs Local DynamoDB guide for local instances tables are created immediately. I added this check as occasionally I was getting resources errors like the one earlier. However, I think the cause for that was because I forgot the return statement before the call to createTable() meaning no Promise was returned to Mocha so it thought the beforeEach() function had completed. I have removed this since in my real Unit Tests and they all seem to work.

Unit Tests

That's really it. The hard part wasn't writing the UTs but getting a local instance of DynamoDB running with the table that the functions to test used in the correct state. Again, due to the functions being tested usually returning promises themselves it is necessary to return Promise. The assertion(s) are made synchronously in a then continuation chained to Promise returned from the function being tested and Promise from the whole chain returned.

If an assertion returns false then even though it's within a continuation Mocha detects this and the test fails. If the function under test throws then Mocha also catches this and the test fails.

And finally

timeout

There's also the this.timeout(5000); at the top of the tests for this file. By default Mocha has 2 second timeout for all tests and hooks. By having the 1 second sleep for starting the dB it's possible for the hook to timeout then causing everything else to fail. The 5 second timeout protects against this.

localhost

When creating the local instance it uses the default port of 8000 and is accessible via the localhost hostname alias. This is used to access the dB when creating the reference to it in before(). A local aws.config is also constructed in each of the functions that access the dB in the actual code under test below.

The actual code being tested


The Best Laid Schemata – a.k.

We have seen how we can exploit a simple model of biological evolution, known as a genetic algorithm, to search for global maxima of functions, being those points at which they return their greatest values.
This model treated the function being optimised as a non-negative measure of the fitness of individuals to survive and reproduce, replacing negative results with zero, and represented their chromosomes with arrays of bits which were mapped onto its arguments by treating subsets of them as integers that were linearly mapped to floating point numbers with given lower and upper bounds. It simulated sexual reproduction by splitting pairs of the chromosomes of randomly chosen individuals at a randomly chosen position and swapping their bits from it to their ends, and mutations by flipping randomly chosen bits from the chromosomes of randomly chosen individuals. Finally, and most crucially, it set the probability that an individual would be copied into the next generation to its fitness as a proportion of the total fitness of the population, ensuring that that total fitness would tend to increase from generation to generation.
I concluded by noting that, whilst the resulting algorithm was reasonably effective, it had some problems that a theoretical analysis would reveal and that is what we shall look into in this post.

The Best Laid Schemata – a.k.

We have seen how we can exploit a simple model of biological evolution, known as a genetic algorithm, to search for global maxima of functions, being those points at which they return their greatest values.
This model treated the function being optimised as a non-negative measure of the fitness of individuals to survive and reproduce, replacing negative results with zero, and represented their chromosomes with arrays of bits which were mapped onto its arguments by treating subsets of them as integers that were linearly mapped to floating point numbers with given lower and upper bounds. It simulated sexual reproduction by splitting pairs of the chromosomes of randomly chosen individuals at a randomly chosen position and swapping their bits from it to their ends, and mutations by flipping randomly chosen bits from the chromosomes of randomly chosen individuals. Finally, and most crucially, it set the probability that an individual would be copied into the next generation to its fitness as a proportion of the total fitness of the population, ensuring that that total fitness would tend to increase from generation to generation.
I concluded by noting that, whilst the resulting algorithm was reasonably effective, it had some problems that a theoretical analysis would reveal and that is what we shall look into in this post.

Documentation is another deliverable and 7 other rules

8295173-2017-08-30-13-43.jpg

“Working software over comprehensive documentation.” The Agile Manifesto

Some have taken that line to mean “no documentation.” I have sympathy for that. Years ago I worked on railway privatisation in the UK. We (Sema Group) were building a system for Railtrack – remember them?

We (the programmers) had documentation coming out our ears. Architecture documents, design documents, user guides, functional specifications, program specifications, and much much more. Most of the documentation was worse than useless because it gave the illusion that everything was recorded and anyone could learn (almost) anything any time.

Some of the documentation was just out of date. The worst was written by architects who lived in a parallel universe to the coders. The system described in the official architecture and design documents bore very little relevance to the actual code but since the (five) architects never looked at the code the (12) programmers could do what they liked.

Documentation wasn’t going to save us.

But, I also know some documentation is not only useful but highly valuable. Sometimes a User Manual can be really useful. Sometimes a developers “Rough Guide” can give important insights.

So how do you tell the difference?

How do you know what to write and what to forego?

Rule #1: Documentation is just another deliverable

Time spent writing a document is time not spent coding, testing, analysing or drinking coffee. There is no documentation fairy and documentation is far from free. It costs to write and costs far more to read (if it is ever read).

Therefore treat documentation like any other deliverable.

If someone wants a document let them request it and prioritise it against all the other possible work to be done. If someone is prepared to displace other work for documentation then it is worth doing.

Rule #2: Documentation should add value

Would you add a feature to a system if it did not increase the value of the system?

The same is true of documentation. If it adds value then there is every reason to do it, if it doesn’t then why bother?

Rule #3: Who will complain if the document is not done?

If in doubt identify a person who will complain if the document is not done. That person places a value on the document. Maybe have them argue for their document over new features.

Rule #4: Don’t write documents for the sake of documents

Don’t write documents just because some process standard somewhere says a document needs to be written. Someone should be able to speak to that standard an explain why it adds more value than doing something else.

Rule #5: Essential documents are part of the work to do

There are some documents that are valuable – someone will complain if they are absent! User Manuals are one reoccurring case. I’ve also worked with teams that need to present documentation as part of a regulatory package to Federal Drug Administration (FDA) and such.

If a piece of work requires the documentation to be updated then updating the documentation is part of the work. For example, suppose you are adding a feature to a system that is subject to FDA regulation. And suppose the FDA regulation requires all features to be listed in a User Manual. In that case, part of the work of adding the feature is to update the user manual. There will be coding, testing and documentation. The work is not done if the User Guide has not been updated.

You don’t need a separate User Story to do the documentation. In such cases documentation may form part of the definition of done.

Rule #6: Do slightly less than you think is needed

If you deliver less than is needed then someone will complain and you will need to rework it. If nobody complains then why are you doing it? And what value is it? – Do less next time!

This is an old XP rule which applies elsewhere too.

Rule #7: Don’t expect anyone to read what you have written

Documentation is expensive to read. Most people who started reading this blog post stopped long ago, you are the exception – thank you!

The same is true for documentation.

The longer a document is the less likely it is to be read.
And the longer a document is the less of it will be remembered.

You and I do not have the writing skills of J.K.Rowling, few of us can hold readers attention that long.

Rule #8: The author learns too

For a lot of documentation – and here I include writing, my own books, patterns, blogs, etc. – the real audience is the author. The process of writing helps the author, it helps us structure our thoughts, it helps us vent our anger and frustration, it forces us to rationalise, it prepares us for an argument, and more.

So often the person who really wants the document, the person who attaches value to it, the person who will complain if it is not done, and the person who will learn most from the document is the author.

Join my mailing list to get free updates on blog post, insights, events and offers.

The post Documentation is another deliverable and 7 other rules appeared first on Allan Kelly Associates.

The fuzzy line between reworking and enhancing

One trick academics use to increase their publication count is to publish very similar papers in different conferences/journals; they essentially plagiarize themselves. This practice is frowned upon, but unless referees spot the ‘duplication’, it is difficult to prevent such plagiarized versions being published. Sometimes the knock-off paper will include additional authors and may not include some of the original authors.

How do people feel about independent authors publishing a paper where all the interesting material was derived from someone else’s paper, i.e., no joint authors? I have just encountered such a case in empirical software engineering.

“Software Cost Estimation: Present and Future” by Siba N. Mohanty from 1981 (cannot find a non-paywall pdf via Google; must exist because I have a copy) has been reworked to create “Cost Estimation: A Survey of Well-known Historic Cost Estimation Techniques” by Syed Ali Abbas and Xiaofeng Liao and Aqeel Ur Rehman and Afshan Azam and M. I. Abdullah (published in 2012; pdf here); they cite Mohanty as the source of their data, some thought has obviously gone into the reworked material and I found it useful and there is a discussion on techniques created since 1981.

What makes the 2012 stand out as interesting is the depth of analysis of the 1970s models and the data, all derived from the 1981 paper. The analysis of later models is not as interesting and doe snot include any data.

The 2012 paper did ring a few alarm bells (which rang a lot more loudly after I read the 1981 paper):

  • Why was such a well researched and interesting paper published in such an obscure (at least to me) journal? I have encountered such cases before and had email conversations with the author(s). The well-known journals have not always been friendly towards empirical research, so an empirical paper appearing in less than a stellar publication is not unusual.

    As regular readers will know I am always on the look-out for software engineering data and am willing to look far and wide. I judge a paper by its content, not the journal it was published in

  • Why, in 2012, were researchers comparing effort estimation models proposed in the 1970s? Well, I am, so why not others? It did seem odd that I could not track down papers on some of the models cited, perhaps the pdfs had disappeared since 2012??? I think I just wanted to believe others were interested in what I was interested in.

What now? Retraction watch offers some advice.

The Journal of Emerging Trends in Computing and Information Sciences has an ethics page, I will email them a link to this post and see what happens (the article in question is listed as their second most cited article last year, with 19 citations).

Microcomputers ‘killed’ Ada

In the mid-70s the US Department of Defense decided it could save lots of money by getting all its contractors to write code in the same programming language. In February 1980 a language was chosen, Ada, but by the end of the decade the DoD had snatched defeat from the jaws of victory; what happened?

I think microcomputers is what happened; these created whole new market ecosystems, some of which were much larger than the ecosystems that mainframes and minicomputers sold into.

These new ecosystems sucked up nearly all the available software developer mind-share; the DoD went from being a major employer of developers to a niche player. Developers did not want a job using Ada because they thought that being type-cast as Ada programmers would overly restricted their future job opportunities; Ada was perceived as a DoD only language (actually there was so little Ada code in the DoD, that only by counting new project starts could it get any serious ranking).

Lots of people were blindsided by the rapid rise (to world domination) of microcomputers. Compilers could profitably sold (in some cases) for tends or hundreds of dollars/pounds because the markets were large enough for this to be economically viable. In the DoD ecosystems compilers had to be sold for thousands or hundreds of thousands of dollars/pounds because the markets were small. Micros were everywhere and being programmed in languages other than Ada; cheap Ada compilers arrived after today’s popular languages had taken off. There is no guarantee that cheap compilers would have made Ada a success, but they would have ensured the language was a serious contender in the popularity stakes.

By the start of the 90s Ada supporters were reduced to funding studies to produce glowing reports of the advantages of Ada compared to C/C++ and how Ada had many more compilers, tools and training than C++. Even the 1991 mandate “… where cost effective, all Department of Defense software shall be written in the programming language Ada, in the absence of special exemption by an official designated by the Secretary of Defense.” failed to have an impact and was withdrawn in 1997.

The Ada mandate was cancelled as the rise of the Internet created even bigger markets, which attracted developer mind-share towards even newer languages, further reducing the comparative size of the Ada niche.

Astute readers will notice that I have not said anything about the technical merits of Ada, compared to other languages. Like all languages, Ada has its fanbois; these are essentially much older versions of the millennial fanbois of the latest web languages (e.g., Go and Rust). There is virtually no experimental evidence that any feature of any language is best/worse than any feature in any other language (a few experiments showing weak support for stronger typing). To its credit the DoD did fund a few studies, but these used small samples (there was not yet enough Ada usage to make larger sample possible) that were suspiciously glowing in their support of Ada.

How much will my software cost?


The question we get asked the second most when speaking to clients and potential clients is “how much will my bespoke software cost to build?” This is extremely difficult to answer without lots of detail and even then the complexities of software development, the complexity of client requirements and clients changing needs over the course of a project make an accurate estimate challenging.

For this reason, most software development companies shy away from including prices on their website. In fact we checked the websites of a number of our competitors and the closest we found was one who offers a range of fee options from fixed price to a daily rate and a couple who ask for your budget when contacting them for more information. As a client, until you get that first email response, phone call or face-to-face meeting you’re no closer to understanding how much your software will cost. Even then it may be some time before you are any the wiser.

We can’t help you understand how much your project will cost until we speak to you. What we can tell you is how much projects have cost our existing clients. We’ve broken the figures down into the types of services we provide, the minimum project cost, the maximum project cost, the average project costs and where in the range most of the projects sit:

* All values are approximate, exclude VAT and are correct as of August 2017

To start investigating how your business problem could be solved with a bespoke application, please contact us for a chat

How much will my software cost?


The question we get asked the second most when speaking to clients and potential clients is “how much will my bespoke software cost to build?” This is extremely difficult to answer without lots of detail and even then the complexities of software development, the complexity of client requirements and clients changing needs over the course of a project make an accurate estimate challenging.

For this reason, most software development companies shy away from including prices on their website. In fact we checked the websites of a number of our competitors and the closest we found was one who offers a range of fee options from fixed price to a daily rate and a couple who ask for your budget when contacting them for more information. As a client, until you get that first email response, phone call or face-to-face meeting you’re no closer to understanding how much your software will cost. Even then it may be some time before you are any the wiser.

We can’t help you understand how much your project will cost until we speak to you. What we can tell you is how much projects have cost our existing clients. We’ve broken the figures down into the types of services we provide, the minimum project cost, the maximum project cost, the average project costs and where in the range most of the projects sit:

* All values are approximate, exclude VAT and are correct as of August 2017

To start investigating how your business problem could be solved with a bespoke application, please contact us for a chat

My Fantasy Gig: Polish Death Metal



It’s no secret that I like death metal. Three of my favorite death metal bands are all from Poland. I’ve been lucky enough to see all of them at least twice individually, but never together. I’ve often wondered why they haven’t all toured together. I’ve never been to Poland either so I’d settle for seeing them all together in their home country.

Decapitated

Opening the show I’d have Decapitated a technical death metal band. Their style, as you would expect, is heavy and progressive. While currently the smaller and less well know of the three bands on this bill, Decapitated are growing in popularity and are poised to step into the shoes of metal titans such as Lamb of God.

After getting into Vader and Behemoth I was really excited to read about another Polish death metal band and I wasn’t disappointed, especially as I also have a soft spot for progressive metal. Often with metal bands who have been around a while, their back catalogue is noisy and unpalatable. Not the case with Decapitated. They’re tight, aggressive and heavy from the first album through to the more recent ones. I’ve seen them play three times now (once even in Norwich) and their live performance demonstrates their skill as musicians.

Behemoth

I’d have Behemoth second on the bill. By far the biggest of the three bands, Behemoth are one of the best metal bands around at the moment. Currently (summer 2017) they are touring the US with Lamb of God and Slayer. I’ve seen them play three times. Once to about 10 people at a club in Bradford and twice to thousands at Bloodstock.

Their singer is often in the press, in Poland and around the world. He famously burned a bible on stage in Poland and was promptly arrested. Later he was diagnosed with and beat cancer.

In the early days Behemoth’s style of ‘blackened death metal’ was heavily influenced by US death metal giants Morbid Angel, but much more palatable. That said they’ve improved on almost every album. Their 2013 album the Satanist is a masterpiece of modern metal. Probably their least heavy album to date, but still crushing.

Vader

Headlining I’d have Vader. I’d describe them as the godfathers of Polish death metal. While not as popular or well selling as Behemoth, they belong at the top. Vader play more traditional death metal, sometimes with trashy tinges. I really struggled to get into their back catalogue. I just wasn’t ready, but every album is superb.

I’ve seen them twice, both times in small clubs. Their sound wasn’t the best, but being a huge fan I put that down to the PA in the clubs. I am sure that atop such a fine bill, they would shine and show what they can really do.

Of course the final encore would comprise of all three bands playing a metal classic together.

My Fantasy Gig: Polish Death Metal



It’s no secret that I like death metal. Three of my favorite death metal bands are all from Poland. I’ve been lucky enough to see all of them at least twice individually, but never together. I’ve often wondered why they haven’t all toured together. I’ve never been to Poland either so I’d settle for seeing them all together in their home country.

Decapitated

Opening the show I’d have Decapitated a technical death metal band. Their style, as you would expect, is heavy and progressive. While currently the smaller and less well know of the three bands on this bill, Decapitated are growing in popularity and are poised to step into the shoes of metal titans such as Lamb of God.

After getting into Vader and Behemoth I was really excited to read about another Polish death metal band and I wasn’t disappointed, especially as I also have a soft spot for progressive metal. Often with metal bands who have been around a while, their back catalogue is noisy and unpalatable. Not the case with Decapitated. They’re tight, aggressive and heavy from the first album through to the more recent ones. I’ve seen them play three times now (once even in Norwich) and their live performance demonstrates their skill as musicians.

Behemoth

I’d have Behemoth second on the bill. By far the biggest of the three bands, Behemoth are one of the best metal bands around at the moment. Currently (summer 2017) they are touring the US with Lamb of God and Slayer. I’ve seen them play three times. Once to about 10 people at a club in Bradford and twice to thousands at Bloodstock.

Their singer is often in the press, in Poland and around the world. He famously burned a bible on stage in Poland and was promptly arrested. Later he was diagnosed with and beat cancer.

In the early days Behemoth’s style of ‘blackened death metal’ was heavily influenced by US death metal giants Morbid Angel, but much more palatable. That said they’ve improved on almost every album. Their 2013 album the Satanist is a masterpiece of modern metal. Probably their least heavy album to date, but still crushing.

Vader

Headlining I’d have Vader. I’d describe them as the godfathers of Polish death metal. While not as popular or well selling as Behemoth, they belong at the top. Vader play more traditional death metal, sometimes with trashy tinges. I really struggled to get into their back catalogue. I just wasn’t ready, but every album is superb.

I’ve seen them twice, both times in small clubs. Their sound wasn’t the best, but being a huge fan I put that down to the PA in the clubs. I am sure that atop such a fine bill, they would shine and show what they can really do.

Of course the final encore would comprise of all three bands playing a metal classic together.

Share And Share Alike – baron m.

Sir R----- my fine fellow! Come join me in quenching this summer eve's thirst with a tankard of cold ale! Might I presume that your thirst for wager is as pressing as that for refreshment?

I am gladdened to hear it Sir! Gladdened to hear it indeed!

This day's sweltering heat has put me in mind of the time that I found myself temporarily misplaced in the great Caloris rainforest on Mercury. I had been escorting the Velikovsky expedition, which had secured the patronage of the Russian Imperial court for its mission to locate the source of the Amazon, and on one particularly close evening our encampment was attacked by a band of Salamanders which, unlike their diminutive Earthly cousins, stood some eight feet tall and wielded vicious looking barbed spears.

Share And Share Alike – baron m.

Sir R----- my fine fellow! Come join me in quenching this summer eve's thirst with a tankard of cold ale! Might I presume that your thirst for wager is as pressing as that for refreshment?

I am gladdened to hear it Sir! Gladdened to hear it indeed!

This day's sweltering heat has put me in mind of the time that I found myself temporarily misplaced in the great Caloris rainforest on Mercury. I had been escorting the Velikovsky expedition, which had secured the patronage of the Russian Imperial court for its mission to locate the source of the Amazon, and on one particularly close evening our encampment was attacked by a band of Salamanders which, unlike their diminutive Earthly cousins, stood some eight feet tall and wielded vicious looking barbed spears.

We hereby retract the content of this paper

Yesterday I came across a paper in software engineering that had been retracted, the first time I had encountered such a paper (I had previously written about how software engineering is great discipline for an academic fraudster).

Having an example of the wording used by the IEEE to describe a retracted paper (i.e., “this paper has been found to be in violation of IEEE’s Publication Principles”), I could search for more. I get 24,400 hits listed when “software” is included in the search, but clicking through the pages there are just 71 actual results.

A search of Retraction Watch using “software engineering” returns nine hits, none of which appear related to a software paper.

I was beginning to think that no software engineering papers had been retracted, now I know of one and if I am really interested the required search terms are now known.

Two approaches to arguing the 1969 IBM antitrust case

My search for software engineering data has made me a regular customer of second-hand book sellers; a recent acquisition is: “Big Blue: IBM’s use and abuse of power” by Richard DeLamarter, which contains lots of interesting sales and configuration data for IBM mainframes from the first half of the 1960s.

DeLamarter’s case, that IBM systematically abused its dominant market position, looked very convincing to me, but I saw references to work by Franklin Fisher (and others) that, it was claimed, contained arguments for IBM’s position. Keen to find more data and hear alternative interpretations of the data, I bought “Folded, Spindled, and Mutilated” by Fisher, McGowan and Greenwood (by far the cheaper of the several books that have written on the subject).

The title of the book, Folded, Spindled, and Mutilated, is an apt description of the arguments contained in the book (which is also almost completely devoid of data). Fisher et al obviously recognized the hopelessness of arguing IBM’s case and spend their time giving general introductions to various antitrust topics, arguing minor points or throwing up various smoke-screens.

An example of the contrasting approaches is calculation of market share. In order to calculate market share, the market has to be defined. DeLamarter uses figures from internal IBM memos (top management were obsessed with maintaining market share) and quote IBM lawyers’ advice to management on phrases to use (e.g., ‘Use the term market leadership, … avoid using phrasing such as “containment of competitive threats” and substitute instead “maintain position of leadership.”‘); Fisher et al arm wave at length and conclude that the appropriate market is the entire US electronic data processing industry (the more inclusive the market used, the lower the overall share that IBM will have; using this definition IBM’s market share drops from 93% in 1952 to 43% in 1972 and there is a full page graph showing this decline), the existence of IBM management memos is not mentioned.

Why do academics risk damaging their reputation by arguing these hopeless cases (I have seen it done in other contexts)? Part of the answer is a fat pay check, but also many academics’ consider consulting for industry akin to supping with the devil (so they get a free pass on any nonsense sprouted when “just doing it for the money”).

Principles of software development revisited

BeachSpainLite-2017-08-15-10-04.jpg

Summer… my traditional time for doing all that stuff that requires a chunk of time… erh… OK, “projects” – only they aren’t well planned and they only resemble projects in the rear-view mirror.

Why now? Why summer? – Because my clients are on holiday too so I’m quiet and not delivering much in the way of (agile) training or consulting. Because my family is away visiting more family. Thus I have a chunk of time.

This year’s projects include some programming – fixing up my own Twitter/LinkedIn/Facebook scheduler “CloudPoster” and some work on my Mimas conference review system in preparation for Agile on the Beach 2018 speaker submissions.

But the big project is a website rebuild.

You may have noticed this blog has moved from Blogger to a new WordPress site, and attentive readers will have noticed that my other sites, allankelly.net and SoftwareStrategy.co.uk have also folded in here. This has involved a lot of content moving, which also means I’ve been seeing articles I’d forgotten about.

In particular there is a group of “The Nature of Agile” articles from 2013 which were once intended to go into a book. Looking at them now I think they still stand, mostly. In particular I’m impressed by my 2013 Principles of Software Development.

I’ll let you can read the whole article but here are the headlines:

Software Development Principles

  1. Software Development exhibits Diseconomies of Scale
  2. Quality is essential – quality makes all things possible
  3. Software Development is not a production line

Agile Software Principles

  1. Highly adaptable over highly adapted
  2. Piecemeal Growth – Start small, get something that works, grow
  3. Need to think
  4. Need to unlearn
  5. Feedback: getting and using
  6. People closest to the work make decisions
  7. Know your schedule, fit work to the schedule not schedule to work
  8. Some Agile practices will fix you, others will help you see and help you fix yourself

The article then goes on to discuss The Iron Triangle and Conway’s Law.

I think that essay might be the first time I wrote about diseconomies of scale. Something else I learned when I moved all my content to this new site is that the Diseconomies of Scale blog entry is by far my most read blog entry ever.

I’m not sure if I’m surprised or shocked that now, four years later, these still look good to me. I really wouldn’t change them much. In fact these ideas are all part of my latest book Continuous Digital.

I’m even starting to wonder if I should role those Nature of Agile essays together in another book – but thats bad of me! Too many books….

Join Allan Kelly’s mailing list to get free updates on blog post, insights and events.

The post Principles of software development revisited appeared first on Allan Kelly Associates.

A Review: Express in Action

Express in Action: Node applications with Express and its companion tools

By Evan Hahn
ISBN: 978-1617292422

This is another excellent JavaScript book from Manning. It contains a great introduction to Express.js and I wish I’d read it sooner as it explains a lot of things about Express.js and how to use it, as well as the tools surrounding it and Node.js, which I had previously worked out for myself. If you’re thinking of writing a web application, especially one in JavaScript, I recommend you read this book first.

The book is far from perfect. It could have been a lot shorter. There is a fair amount of repetition and the chatty style makes it overly verbose and irritating in many places.  The author tries to cover too much and goes beyond Express.js unnecessarily in a few places. However, given that, it’s still not a huge book and quite easy to read.

A Review: Express in Action

Express in Action: Node applications with Express and its companion tools

By Evan Hahn
ISBN: 978-1617292422

This is another excellent JavaScript book from Manning. It contains a great introduction to Express.js and I wish I’d read it sooner as it explains a lot of things about Express.js and how to use it, as well as the tools surrounding it and Node.js, which I had previously worked out for myself. If you’re thinking of writing a web application, especially one in JavaScript, I recommend you read this book first.

The book is far from perfect. It could have been a lot shorter. There is a fair amount of repetition and the chatty style makes it overly verbose and irritating in many places.  The author tries to cover too much and goes beyond Express.js unnecessarily in a few places. However, given that, it’s still not a huge book and quite easy to read.

A review: JavaScript the Good Parts

By Douglas Crockford
ISBN: 978-0596517748

Every JavaScript developer with a pre-existing working knowledge of JavaScript should read this book. JavaScript is a powerful and varied language, but it was developed in a hurry and there’s plenty wrong with it. This book outlines the good bits of the language and highlights the bad bits and the bits you should just avoid. There’s also a fair amount about the author’s JSLint project in the appendices.

This book was written in 2008 and probably needs updating. It’s hard going in places and the diagrams did little to nothing to help my understanding. I’ve come away still wondering about new and constructors, but I know I just need to review them again when I need them and it’ll get clearer.  I’m still not sure which function declaration syntax is best, but I’m not sure it matters too much.


A review: JavaScript the Good Parts

By Douglas Crockford
ISBN: 978-0596517748

Every JavaScript developer with a pre-existing working knowledge of JavaScript should read this book. JavaScript is a powerful and varied language, but it was developed in a hurry and there’s plenty wrong with it. This book outlines the good bits of the language and highlights the bad bits and the bits you should just avoid. There’s also a fair amount about the author’s JSLint project in the appendices.

This book was written in 2008 and probably needs updating. It’s hard going in places and the diagrams did little to nothing to help my understanding. I’ve come away still wondering about new and constructors, but I know I just need to review them again when I need them and it’ll get clearer.  I’m still not sure which function declaration syntax is best, but I’m not sure it matters too much.


Getting to the route of the problem

In 2016, Venkat Subramaniam wrote an incredible book called ‘Test-Driving JavaScript Applications’ which, along with JavaScript tools such as Mocha, Istanbul, Prettier and Eslint, have made me fall in love with JavaScript and Node.js (well for UI development anyway). JavaScript isn’t a proper language, right? For a long time I argued not, because the tools weren’t available to develop software with unit tests, static analysis and code coverage. This has changed and now I’m starting to take JavaScript seriously, even beyond jazzing up a web based UI. I’m almost over the lack of static typing.

I’m currently using Express.js, a web framework for Node.js, a lot and Venkat includes a section on testing Express.js routes in his book. They’re a bit like controllers in the Modal View Controllers pattern:

router.get('/', function(req, res, next) {
task.all(function(err, tasks) {
res.send(tasks);
});
});

Venkat’s example test looks like this:

it('should register uri / for get', function(done) {
    // ...        

    var registeredCallback = router.get.firstCall.args[1];
    registeredCallback(req, res);
});

I’ve left out some mocking and other boilerplate for brevity and so that we can concentrate on the one bit I don’t like. Venkat describes the test in full detail in his book.  Take another look at this line:

    var registeredCallback = router.get.firstCall.args[1];

What it does is get the second argument for the first get route declared with the router. That’s what is returned by firstCall, the first declared route. So if there is more than one get route declared with the router and at some point you change the order in which they are declared or you declare another get route in-between, the test will break. It’s brittle.

In fact it’s worse. To get the second get route you’d use secondCall and so on. So although it’s probably a very large number, there are a finite number of get routes you can get from the router with this method. For me this rang alarm bells.
Google suggested this is the way that everyone is doing it. It appears to be the standard practice. It doesn’t sit at all well with me. I’d much rather be able to look up route in the router by its path. After a while printing all sorts of things to the console to find out the data structures, I was able to develop this:

var rh = {
    findGet: function(router, path) {
        for (var i = 0; i < router.get.args.length; i++)
            if (router.get.args[i][0] === path)
                return router.get.args[i];

        return null;
    },

   // ..
};

module.exports = {
    execGet: function(router, path, req, res) {
        var get = rh.findGet(router, path);
        if (get != null) get[1](req, res);
    },

    // ..
};

The findGet function takes a router and the path to test and returns all of the arguments declared for that route or null if it’s not found.  The execGet function uses those arguments to execute the route, meaning that the test now becomes:

it('should register uri / for get', function(done) {
        // ...

        execGet(router, '/', req, res);
    });

Which is not only far more expressive, but less brittle and less code per test. It means that the declaration order of the routes for the router no longer matters. Of course similar functions can be added to facilitate testing post, put and delete.

I wanted to write this up as I couldn’t find any other solution with Google. Hopefully it will encourage developers to write more tests for Express routes as they become easier and less brittle.


Getting to the route of the problem

In 2016, Venkat Subramaniam wrote an incredible book called ‘Test-Driving JavaScript Applications’ which, along with JavaScript tools such as Mocha, Istanbul, Prettier and Eslint, have made me fall in love with JavaScript and Node.js (well for UI development anyway). JavaScript isn’t a proper language, right? For a long time I argued not, because the tools weren’t available to develop software with unit tests, static analysis and code coverage. This has changed and now I’m starting to take JavaScript seriously, even beyond jazzing up a web based UI. I’m almost over the lack of static typing.

I’m currently using Express.js, a web framework for Node.js, a lot and Venkat includes a section on testing Express.js routes in his book. They’re a bit like controllers in the Modal View Controllers pattern:

router.get('/', function(req, res, next) {
task.all(function(err, tasks) {
res.send(tasks);
});
});

Venkat’s example test looks like this:

it('should register uri / for get', function(done) {
    // ...        

    var registeredCallback = router.get.firstCall.args[1];
    registeredCallback(req, res);
});

I’ve left out some mocking and other boilerplate for brevity and so that we can concentrate on the one bit I don’t like. Venkat describes the test in full detail in his book.  Take another look at this line:

    var registeredCallback = router.get.firstCall.args[1];

What it does is get the second argument for the first get route declared with the router. That’s what is returned by firstCall, the first declared route. So if there is more than one get route declared with the router and at some point you change the order in which they are declared or you declare another get route in-between, the test will break. It’s brittle.

In fact it’s worse. To get the second get route you’d use secondCall and so on. So although it’s probably a very large number, there are a finite number of get routes you can get from the router with this method. For me this rang alarm bells.
Google suggested this is the way that everyone is doing it. It appears to be the standard practice. It doesn’t sit at all well with me. I’d much rather be able to look up route in the router by its path. After a while printing all sorts of things to the console to find out the data structures, I was able to develop this:

var rh = {
    findGet: function(router, path) {
        for (var i = 0; i < router.get.args.length; i++)
            if (router.get.args[i][0] === path)
                return router.get.args[i];

        return null;
    },

   // ..
};

module.exports = {
    execGet: function(router, path, req, res) {
        var get = rh.findGet(router, path);
        if (get != null) get[1](req, res);
    },

    // ..
};

The findGet function takes a router and the path to test and returns all of the arguments declared for that route or null if it’s not found.  The execGet function uses those arguments to execute the route, meaning that the test now becomes:

it('should register uri / for get', function(done) {
        // ...

        execGet(router, '/', req, res);
    });

Which is not only far more expressive, but less brittle and less code per test. It means that the declaration order of the routes for the router no longer matters. Of course similar functions can be added to facilitate testing post, put and delete.

I wanted to write this up as I couldn’t find any other solution with Google. Hopefully it will encourage developers to write more tests for Express routes as they become easier and less brittle.


NorDev: JavaScript Starter Kit – Beginners Full Day Workshop


Date: 9:00 am to 4:45 pm, Thursday 5th October 2017

Location: The King’s Centre, King Street, Norwich, NR1 1PH

Price: £50.00 per person

Level: Beginner

Prerequisites: Laptop with wifi, modern browser, text editor

RSVP: https://www.meetup.com/Norfolk-Developers-NorDev/events/242461849/

JavaScript is amazing.

It is a powerful, simple, infuriating, elegant and sometimes irrational programming language which was born in a hurry and can now do almost anything you can imagine. It can make whizzy websites, speak to databases, and draw maps, it can fly drones, make games, and build apps. You can run it on your watch or on your phone, on any web page or on hundreds of virtual servers.

And if you’re reading this you’re probably contemplating learning it.

This day-long workshop aims to cover enough ground to give you a broad base from which to start your quest. We’ll use plenty of practical exercises to explore the language. We’ll cover some of the tricky parts which often mystify people – especially handling asynchronous code, which is one of the language’s great strengths. We’ll spend most of our time in the browser, but we’ll also play around with node.js, JavaScript’s foremost server-side environment. There’ll be time to survey some of the different tools and frameworks which are popular with JavaScripters at the moment. As well as all this we’ll explore JavaScript’s history, its culture and community, and the factors behind its explosive growth. Perhaps most importantly we’ll introduce a set of resources which’ll help you continue your learning independently.

You’ll need to come equipped with a laptop, and you should have a modern browser installed, along with a text editor you’re comfortable using. You don’t need to have a lot of knowledge or experience to join in, though any familiarity with another programming language will help a lot.

There’s a lot to get into one day, so please bring lunch and Neontribe will be buying the first round in the pub straight after the workshop.

Rupert Redington

Rupert ran away from the theatre to become a web developer at the turn of the century. Since then he’s been making mistakes at Neontribe as fast as he can, learning from a reasonable percentage of them. Recently he’s been using Javascript to help teenagers talk to doctors, Americans to buy airline tickets and everybody to find their way to the loo.

“Rupert did a fine job of making this an entertaining subject and his enthusiasm for js was infectious.” – Matthew Shorten

“Thoroughly enjoyed it! Presenter was excellent. Would be interested in any other JS courses that he runs.” -Stephen Pengilley

“I’d certainly sign up for other courses Rupert hosts in a flash. This was an introduction and as such it was perfectly positioned (in my humble…), but if ever there’s an “intermediate” course which goes into more depth with core principles & real-world use of loops, arrays, functions & objects that would be great.” – Steve Harman

NorDev: JavaScript Starter Kit – Beginners Full Day Workshop


Date: 9:00 am to 4:45 pm, Thursday 5th October 2017

Location: The King’s Centre, King Street, Norwich, NR1 1PH

Price: £50.00 per person

Level: Beginner

Prerequisites: Laptop with wifi, modern browser, text editor

RSVP: https://www.meetup.com/Norfolk-Developers-NorDev/events/242461849/

JavaScript is amazing.

It is a powerful, simple, infuriating, elegant and sometimes irrational programming language which was born in a hurry and can now do almost anything you can imagine. It can make whizzy websites, speak to databases, and draw maps, it can fly drones, make games, and build apps. You can run it on your watch or on your phone, on any web page or on hundreds of virtual servers.

And if you’re reading this you’re probably contemplating learning it.

This day-long workshop aims to cover enough ground to give you a broad base from which to start your quest. We’ll use plenty of practical exercises to explore the language. We’ll cover some of the tricky parts which often mystify people – especially handling asynchronous code, which is one of the language’s great strengths. We’ll spend most of our time in the browser, but we’ll also play around with node.js, JavaScript’s foremost server-side environment. There’ll be time to survey some of the different tools and frameworks which are popular with JavaScripters at the moment. As well as all this we’ll explore JavaScript’s history, its culture and community, and the factors behind its explosive growth. Perhaps most importantly we’ll introduce a set of resources which’ll help you continue your learning independently.

You’ll need to come equipped with a laptop, and you should have a modern browser installed, along with a text editor you’re comfortable using. You don’t need to have a lot of knowledge or experience to join in, though any familiarity with another programming language will help a lot.

There’s a lot to get into one day, so please bring lunch and Neontribe will be buying the first round in the pub straight after the workshop.

Rupert Redington

Rupert ran away from the theatre to become a web developer at the turn of the century. Since then he’s been making mistakes at Neontribe as fast as he can, learning from a reasonable percentage of them. Recently he’s been using Javascript to help teenagers talk to doctors, Americans to buy airline tickets and everybody to find their way to the loo.

“Rupert did a fine job of making this an entertaining subject and his enthusiasm for js was infectious.” – Matthew Shorten

“Thoroughly enjoyed it! Presenter was excellent. Would be interested in any other JS courses that he runs.” -Stephen Pengilley

“I’d certainly sign up for other courses Rupert hosts in a flash. This was an introduction and as such it was perfectly positioned (in my humble…), but if ever there’s an “intermediate” course which goes into more depth with core principles & real-world use of loops, arrays, functions & objects that would be great.” – Steve Harman

What if it is all random?

iStock-512907832Small-2017-08-10-12-37.jpg

What if success in digital business, and in software development, is random? What if one cannot tell in advance what will succeed and what will fail?

My cynical side sometimes thinks everything is random. I don’t want to believe my cynical side but…

All those minimally viable products, some work, some fail.

All those stand-up meetings, do they really make a difference?

All those big requirements documents, just sometimes they work.

How can I even say this? – I’ve written books on how to “do it right.”
I advise companies on how to improve “processes.” I’ve helped individuals do better work.

And just last month I was at a patterns conference trying to spot reoccurring patterns and why they are patterns.

So let me pause my rational side and indulge my cynical side, what if it is all random?

If it is all random what we have to ask is: What would we do in a random world?

Imagine for a moment success is like making a bet at roulette and spinning the wheel.

Surely we would want to both minimise losses (small bets) and maximise wheel spins: try lots, remove the failures quickly and expand the successes (if we can).

I suggested “its all random” to someone the other day and he replied “It is not random, its complex.” And we were into Cynefin before you could say “spin the wheel.”

Dave Snowden’s Cynefin model attempts to help us understand complexity and the complex. Faced with complexity Cynefin says we should probe. That is, try lots of experiments so we can understand, learn from the experiments and adjust.

If the experiment “succeeds” we understand more and can grow that learning. Where the experiment “fails” we have still learned but we will try a different avenue next time.

Look! – it is the same approach, the same result, complexity, Cynefin or just random: try a lot, remove failure and build on success. And hang on, where have I heard that before, … Darwin and evolution; random gene mutations which give benefit get propagated and in time others die out.

It is just possible that Dave is right, Darwin is right and I am right…

Today most of the world’s mobile/cell telephone systems are built on CDMA technology. CDMA is super complex maths but it basically works by encoding a signal (sequence of numbers, your voice digitised) and injecting it into a random number stream (radio spectrum), provided you know the encoding you can retrieve the signal out of the randomness. Quite amazing really.

Further, provided the number sequences are sufficiently different they are in effect random so you can inject more signal into the same space.

That is why we can all use our mobile phones at the same time.

Put it another way: you walk into a party in London, in the room are Poles, Lebanese, Germans, Argentinians and the odd Brit. They are all talking in their own language to fellow speakers. Somehow you can hear your own language and join the right conversation. Everything else is random background noise.

Maybe the same is true in digital business and software development…

Perhaps it is all complex but it is so complex that we will never be able to follow all the cause and effect chains, it is so complex that it looks random. Dave is right with Cynefin but maybe there is so much complexity that we might as well treat it as random and save our time.

Back to CDMA and London parties, faced with apparent randomness there are useful strategies and signals can still be extracted.

Perhaps the way to deal with this complexity is not to try and understand it but to treat it as random. Rather than expend energy and time on a (possibly) impossible task accept it as random and apply appropriate strategies.

After all, if we have learned anything from statistical distributions it is that faced with actual and apparent randomness we can still find patterns, we can still learn and we can still work with, well, randomness.

The post What if it is all random? appeared first on Allan Kelly Associates.

Writing a Good Tech Cover Letter

I review a lot of job applications at Codeplay and most of them make me sad.

For every application I look at, I make a list of plus points and minus points as I read through the candidate’s documents and code. Invariably the first thing I look at is the cover letter, and I would estimate that 90% of them get an instant minus point in my review feedback. As an alternative to ranting about this on the internet, I’ve written some advice on writing a cover letter for a tech job. All advice is from the viewpoint of a small-medium sized tech company and all, well, you know, just like, my opinion, man.

A good job application answers two questions: why the candidate is right for the company and why the company is right for the candidate. The best place to answer these questions is in your cover letter. In fact, with the best cover letters, I even forget to read the CV. If you can’t answer both of these questions on a single side of A4, you probably don’t know the answers.

Step 1 to writing a good cover letter is to actually write a cover letter. If the job advertisement asks for a cover letter, write a cover letter. If it doesn’t ask for a cover letter, write one anyway! Without a concise, targeted description of why you should be considered for a job, your CV needs to work five times as hard for you, as the reviewer needs to try and extract the necessary information from a list of competencies and experience rather than an explicit argument for why the candidate fits the position.

Do not copy-paste a generic cover letter which you send everywhere. Anyone who has reviewed a bunch can spot them instantly, and they make it look like you don’t care about the job enough to actually write something targeted to the company and position.

If the job advert lists criteria for applicants, reference it. Show why you fulfil (or, even better, exceed) the expectations and provide evidence. The best cover letters I’ve seen have gone through all of the required and desired skills and provided short descriptions of experience in the area with links to proof of this experience. And, yes, you can do this on one sheet of paper and still have space for a how-do-you-do.

Don’t spout vague buzzwords. Every time I read about a “team player who is eager to learn” my eyes roll back in my head so hard I lose them. If things like being a team player or eager to learn are in the criteria, fine, but (again) provide evidence.

Let your character and enthusiasm show. These don’t come across in a CV, so show me who you are and why I should want to work with you. Of course, this is much more apparent when it actually comes to interviews, but being enthusiastic about the position and giving some colour to your writing can make the difference between actually getting that interview or not.

If the advert asks for links to your work, provide good ones. Of course, if most of your work is confidential and you don’t work on hobby projects then you may not have any; that’s fine, just justify it. But many applicants link to a Stack Overflow profile with two closed questions on std::vector and send a code sample they wrote ten years ago with using namespace std; in all the headers. Don’t.

Spell-check your letter. English might not be your first language, and whoever is reviewing your application should be sensitive to that, but maeking shoor ur letter doesnt look lyk this can be done by your word processor. This advice should be obvious, but you’d be surprised how many applications I review are riddled with typos. Bad spelling looks lazy. You should put effort into your cover letter, and it should show.

Finally, and most importantly, don’t assume you’re not good enough. An understanding of the gaps in your knowledge, an honest desire to learn the field and evidence that you’re trying goes a really long way.

CppCon 2017 class and presentation on concurrency

I am pleased to announce that I will be running my "Concurrent Thinking in C++" class at CppCon again this year. Here is the course description:

One of the most difficult issues around designing software with multiple threads of execution is synchronizing data.

Whether you use actors, active objects, futures and continuations or mutable shared state, every non-trivial system with multiple threads needs to transfer data between them. This means thinking about which data needs to be processed by which thread, and ensuring that the right data gets to the right threads in the right order. It also means thinking about API design to avoid race conditions.

In this workshop you will encounter a series of scenarios involving multithreaded code, and be guided through identifying the problem areas and the ways of handling them.

You will learn techniques for thinking about the scenarios to ease the analysis, as well as details of the tools we have available in C++ to mitigate the problems. You will also learn how to use the C++ standard library to help enforce the requirements of each scenario in code.

I will also be presenting on "Concurrency, Parallelism and Coroutines":

C++17 is adding parallel overloads of most of the Standard Library algorithms. There is a TS for Concurrency in C++ already published, and a TS for Coroutines in C++ and a second TS for Concurrency in C++ in the works.

What does all this mean for programmers? How are they all related? How do coroutines help with parallelism?

This session will attempt to answer these questions and more. We will look at the implementation of parallel algorithms, and how continuations, coroutines and work-stealing fit together. We will also look at how this meshes with the Grand Unified Executors Proposal, and how you will be able to take advantage of all this as an application developer.

My class is on 17th-18th September, and the main conference is running 19th-23rd, with my presentation on 20th September. If you haven't got your ticket already, head on over to CppCon Registration to get yours now.

Hope to see you there!

Posted by Anthony Williams
[/ news /] permanent link
Tags: , , , ,

| Stumble It! stumbleupon logo | Submit to Reddit reddit logo | Submit to DZone dzone logo

Comment on this post

Follow me on Twitter

void C::C::C::A::A::A::foo() – valid syntax monstrosity

Here’s an odd bit of C++ syntax for you. Say we have the following class hierarchy:

class A {
public:
    virtual void foo() = 0;
};

class B {
public:
    virtual void foo() = 0;
};

class C : public A, public B {
public:
    virtual void foo();
};

The following definitions are all well-formed:

void C::foo(){
  std::cout << "C";
}
void C::A::foo(){
  std::cout << "A";
}
void C::B::foo(){
  std::cout << "B";
}

The first one defines C::foo, the second defines A::foo and the third defines B::foo. This is valid because of an entity known as the injected-type-name:

A class-name is inserted into the scope in which it is declared immediately after the class-name is seen. The class-name is also inserted into the scope of the class itself; this is known as the injected-class-name. For purposes of access checking, the injected-class-name is treated as if it were a public member name. […]

Since A is a base class of C and the name A is injected into the scope of A, A is also visible from C.

The injected-class-name exists to ensure that the class is found during name lookup instead of entities in an enclosing scope. It also makes referring to the class name in a template instantiation easier. But since we’re awful people, we can use this perfectly reasonable feature do horribly perverse things like this:

void C::C::C::A::A::A::foo(){
    std::cout << "A";
}

Yeah, don’t do that.

Books similar to my empirical software engineering book

I am sometimes asked which other books are similar to the Empirical Software Engineering book I am working on.

In spirit, the most similar book is “Software Project Dynamics” by Abdel-Hamid and Madnick, based on Abdel-Hamid’s PhD thesis. The thesis/book sets out to create an integrated model of software development projects, using system dynamics (the model can be ‘run’ to produce outputs from inputs, assuming the necessary software is available).

Building a model of the software development process requires figuring out the behavior of all the important factors and Abdel-Hamid does a thorough job of enumerating the important factors and tracking down the available empirical work (in the 1980s). The system dynamics model, written in Dynamo appears in an appendix (I have not been able to locate any current implementation).

In the 1980s I would have agreed with Abdel-Hamid that it was possible to build a reasonably accurate model of software development projects. Thirty years later, I have tracked down a lot more empirical work and know a more about how software projects work. All this has taught me is that I don’t know enough to be able to build a model of software development projects; but I still think it is possible, one day.

There have been other attempts to build models of major aspects of software development projects (all using system dynamics), including Madachy’s PhD and later book “Software Process Dynamics”, and Buettner’s PhD (no book, yet???).

There are other books that include some combination of the words empirical, software and engineering in their title. On the whole these are collections of edited papers, whose chapters are written by researchers promoting their latest work; there is even one that aims to teach students how to do empirical work.

Dag Sjøberg has done some interesting empirical work and is currently working on an empirical book, this should be worth a look.

“R in Action” by Kabacoff is the closest to the statistical material, but at a more general level. “The R Book” by Crawley is the R book I would recommended, but it is not at all like the material I have written.

Blogger, Blog move & Comments

Frank_Sinatra_Hollywood_star-2017-08-7-15-15.jpg

“And now, the end is near
And so I face the final curtain
My friend, I’ll say it clear
I’ll state my case, of which I’m certain”
My Way, Paul Anka/Frank Sinantra
(public domain picture from WikiCommons)

Don’t panic!

I’m not going away, this blog is not finishing.

But it is changing. And with the change there will be some loss.

Blog entry 1 appeared in April 2005, and after 12 years on Blogger this blog is moving and that move has a downside.

I’ve always been happy on Blogger but in the last few years I’ve been operating three website: the blog on Blogger, allankelly.net – a static site written in RapidWeaver and my commercial site, Software Strategy (www.softwarestrategy.co.uk).

This means my efforts have been split between three sites, and Google has seen me as three sites. Given that my web presence is part of marketing my training and consultancy that might not have been the smartest move.

So last month I decided to merge the three sites and adopt a new corporate identity. Most of the move has been done, Software Strategy is now doing to allankellyassociates.co.uk, the blog is mostly moved – thanks to the great work of the folks at CMS2CMS, highly recommended – and here in lies the problem….

CMS2CMS have done a great job of moving the blog posts, images and comments. But they can’t move all the comments. That is because around October 2014 I switch the comments from Blogger to Google+. The comments on Blogger, before October 2014, can, and have, been moved across, but the comments on Google+ can’t – so all comments left since October 2014 are lost. I regret the decisions to switch to Google+ comments massively.

So many of the comments over the few years (I’m not sure exactly how many!) are about to be lost when the Blogger blog is set to redirect to allankellyassociates.co.uk.

In order that the comments are not lost entirely here are some of my “greatest hits” as PDF files with comments and the view count according to Blogger:

I could go on, I’m sorry I can’t save more comments, it is sad looking now at all the comments, far more than I thought there were. I’m also amazon by how the blog has grown, from early posts with, maybe, a hundred views to a regular audience of thousands and sometimes tens of thousands.

And I completely blame Google and G+ for this, on Blogger Google are quite supportive of blog exports but there is nothing equivalent on Google+.

O, the third site, allankelly.net, this is mostly an archive of my writing down the years and it too will disappear inside www.allankellyassociates.co.uk in the very near future – most of the content is already moved.

The post Blogger, Blog move & Comments appeared first on Allan Kelly Associates.

It’s All In The Genes – a.k.

Last time we took a look at the simulated annealing global minimisation algorithm which searches for points at which functions return their least possible values and which drew its inspiration from the metallurgical process of annealing which minimises the energy state of the crystalline structure of metals by first heating and then slowly cooling them.
Now as it happens, physics isn't the only branch of science from which we can draw inspiration for global optimisation algorithms. For example, in biology we have the process of evolution through which the myriad species of life on Earth have become extraordinarily well adapted to their environments. Put very simply this happens because offspring differ slightly from their parents and differences that reduce the chances that they will survive to have offspring of their own are less likely to be passed down through the generations than those that increase those chances.
Noting that extraordinarily well adapted is more or less synonymous with near maximally adapted, it's not unreasonable to suppose that we might exploit a mathematical model of evolution to search for global maxima of functions, being those points at which they return their greatest possible values.

It’s All In The Genes – a.k.

Last time we took a look at the simulated annealing global minimisation algorithm which searches for points at which functions return their least possible values and which drew its inspiration from the metallurgical process of annealing which minimises the energy state of the crystalline structure of metals by first heating and then slowly cooling them.
Now as it happens, physics isn't the only branch of science from which we can draw inspiration for global optimisation algorithms. For example, in biology we have the process of evolution through which the myriad species of life on Earth have become extraordinarily well adapted to their environments. Put very simply this happens because offspring differ slightly from their parents and differences that reduce the chances that they will survive to have offspring of their own are less likely to be passed down through the generations than those that increase those chances.
Noting that extraordinarily well adapted is more or less synonymous with near maximally adapted, it's not unreasonable to suppose that we might exploit a mathematical model of evolution to search for global maxima of functions, being those points at which they return their greatest possible values.

Thames path improvements – Letter to Oxfordshire County Council

I have sent the following letter to LTS.Team AT oxfordshire.gov.uk in response to https://consultations.oxfordshire.gov.uk/consult.ti/OxfordRiversideRoutes/.

 

Hi,

 

I have submitted the following via the questionnaire:

 

The Oxpens bridge should move upstream to parallel the railway bridge with a path extension besides railway to station. Getting from Thames/railway station to canal towpath and up to Kidlington Airport and Kidlington new housing extension needed. Second bridge, again parallel to rail bridge needed to access Science Park.

 

I have used the Thames towpath for commuting to Osney Mead industrial estate for three months and for work on Cornmarket for three years. I also used the path for three months to commute to Kidlington (Oxford Airport) but it is too arduous and much to my sadness I now cycle on the road, so have trenchant views on the difficulty of cycle commuting in Oxford.

 

I understand that there is a plan to build many new houses in the Kidlington gap, between Summertown and Kidlington.

 

These new commuters would very much appreciate a functioning cycle route into the centre of town.

 

The new Oxford Parkway should be integrated into this cycle route.

 

The problem facing cyclists is not how to get from the towpath to the Centre, this is served by the pipe bridge, the pedestrian crossing down stream and Folley Bridge. What is needed is easy access to the railway station which could be easily achieved by a bridge parallel to the railway bridge and then along railway land to the station itself.

 

Cyclists wishing to get from the Thames to the canal have to continue to Osney, cross the road and then fiddle around the Thames onto the canal path, which is in a very poor state, then up to Kingsbridge and all the way through Kidlington to the Oxford Airport.

 

Finally the Thames path should connect the Science Park, and the planned new housing at Littlemore. This again could be achieved by a cycle bridge parallel to the existing railway bridge down stream from the bypass underpass.

 

I really welcome the proposals but would urge you to consider extending its scope and vision. This could be such a good route and would show that Oxford is a cycling city.

 

best regards Tim Pizey

-- Tim Pizey - http://tim.pizey.uk/

Thames path improvements – Letter to Oxfordshire County Council

I have sent the following letter to LTS.Team AT oxfordshire.gov.uk in response to https://consultations.oxfordshire.gov.uk/consult.ti/OxfordRiversideRoutes/.

 

Hi,

 

I have submitted the following via the questionnaire:

 

The Oxpens bridge should move upstream to parallel the railway bridge with a path extension besides railway to station. Getting from Thames/railway station to canal towpath and up to Kidlington Airport and Kidlington new housing extension needed. Second bridge, again parallel to rail bridge needed to access Science Park.

 

I have used the Thames towpath for commuting to Osney Mead industrial estate for three months and for work on Cornmarket for three years. I also used the path for three months to commute to Kidlington (Oxford Airport) but it is too arduous and much to my sadness I now cycle on the road, so have trenchant views on the difficulty of cycle commuting in Oxford.

 

I understand that there is a plan to build many new houses in the Kidlington gap, between Summertown and Kidlington.

 

These new commuters would very much appreciate a functioning cycle route into the centre of town.

 

The new Oxford Parkway should be integrated into this cycle route.

 

The problem facing cyclists is not how to get from the towpath to the Centre, this is served by the pipe bridge, the pedestrian crossing down stream and Folley Bridge. What is needed is easy access to the railway station which could be easily achieved by a bridge parallel to the railway bridge and then along railway land to the station itself.

 

Cyclists wishing to get from the Thames to the canal have to continue to Osney, cross the road and then fiddle around the Thames onto the canal path, which is in a very poor state, then up to Kingsbridge and all the way through Kidlington to the Oxford Airport.

 

Finally the Thames path should connect the Science Park, and the planned new housing at Littlemore. This again could be achieved by a cycle bridge parallel to the existing railway bridge down stream from the bypass underpass.

 

I really welcome the proposals but would urge you to consider extending its scope and vision. This could be such a good route and would show that Oxford is a cycling city.

 

best regards Tim Pizey

-- Tim Pizey - http://tim.pizey.uk/

Writing a Linux Debugger Part 10: Advanced topics

We’re finally here at the last post of the series! This time I’ll be giving a high-level overview of some more advanced concepts in debugging: remote debugging, shared library support, expression evaluation, and multi-threaded support. These ideas are more complex to implement, so I won’t walk through how to do so in detail, but I’m happy to answer questions about these concepts if you have any.


Series index

  1. Setup
  2. Breakpoints
  3. Registers and memory
  4. Elves and dwarves
  5. Source and signals
  6. Source-level stepping
  7. Source-level breakpoints
  8. Stack unwinding
  9. Handling variables
  10. Advanced topics

Remote debugging

Remote debugging is very useful for embedded systems or debugging the effects of environment differences. It also sets a nice divide between the high-level debugger operations and the interaction with the operating system and hardware. In fact, debuggers like GDB and LLDB operate as remote debuggers even when debugging local programs. The general architecture is this:

debugarch

The debugger is the component which we interact with through the command line. Maybe if you’re using an IDE there’ll be another layer on top which communicates with the debugger through the machine interface. On the target machine (which may be the same as the host) there will be a debug stub, which in theory is a very small wrapper around the OS debug library which carries out all of your low-level debugging tasks like setting breakpoints on addresses. I say “in theory” because stubs are getting larger and larger these days. The LLDB debug stub on my machine is 7.6MB, for example. The debug stub communicates with the debugee process using some OS-specific features (in our case, ptrace), and with the debugger though some remote protocol.

The most common remote protocol for debugging is the GDB remote protocol. This is a text-based packet format for communicating commands and information between the debugger and debug stub. I won’t go into detail about it, but you can read all you could want to know about it here. If you launch LLDB and execute the command log enable gdb-remote packets then you’ll get a trace of all packets sent through the remote protocol. On GDB you can write set remotelogfile <file> to do the same.

As a simple example, here’s the packet to set a breakpoint:

$Z0,400570,1#43

$ marks the start of the packet. Z0 is the command to insert a memory breakpoint. 400570 and 1 are the argumets, where the former is the address to set a breakpoint on and the latter is a target-specific breakpoint kind specifier. Finally, the #43 is a checksum to ensure that there was no data corruption.

The GDB remote protocol is very easy to extend for custom packets, which is very useful for implementing platform- or language-specific functionality.


Shared library and dynamic loading support

The debugger needs to know what shared libraries have been loaded by the debuggee so that it can set breakpoints, get source-level information and symbols, etc. As well as finding libraries which have been dynamically linked against, the debugger must track libraries which are loaded at runtime through dlopen. To facilitate this, the dynamic linker maintains a rendezvous structure. This structure maintains a linked list of shared library descriptors, along with a pointer to a function which is called whenever the linked list is updated. This structure is stored where the .dynamic section of the ELF file is loaded, and is initialized before program execution.

A simple tracing algorithm is this:

  • The tracer looks up the entry point of the program in the ELF header (or it could use the auxillary vector stored in /proc/<pid>/aux)
  • The tracer places a breakpoint on the entry point of the program and begins execution.
  • When the breakpoint is hit, the address of the rendezvous structure is found by looking up the load address of .dynamic in the ELF file.
  • The rendezvous structure is examined to get the list of currently loaded libraries.
  • A breakpoint is set on the linker update function.
  • Whenever the breakpoint is hit, the list is updated.
  • The tracer infinitely loops, continuing the program and waiting for a signal until the tracee signals that it has exited.

I’ve written a small demonstration of these concepts, which you can find here. I can do a more detailed write up of this in the future if anyone is interested.


Expression evaluation

Expression evaluation is a feature which lets users evaluate expressions in the original source language while debugging their application. For example, in LLDB or GDB you could execute print foo() to call the foo function and print the result.

Depending on how complex the expression is, there are a few different ways of evaluating it. If the expression is a simple identifier, then the debugger can look at the debug information, locate the variable and print out the value, just like we did in the last part of this series. If the expression is a bit more complex, then it may be possible to compile the code to an intermediate representation (IR) and interpret that to get the result. For example, for some expressions LLDB will use Clang to compile the expression to LLVM IR and interpret that. If the expression is even more complex, or requires calling some function, then the code might need to be JITted to the target and executed in the address space of the debuggee. This involves calling mmap to allocate some executable memory, then the compiled code is copied to this block and is executed. LLDB does this by using LLVM’s JIT functionality.

If you want to know more about JIT compilation, I’d highly recommend Eli Bendersky’s posts on the subject.


Multi-threaded debugging support

The debugger shown in this series only supports single threaded applications, but to debug most real-world applications, multi-threaded support is highly desirable. The simplest way to support this is to trace thread creation and parse the procfs to get the information you want.

The Linux threading library is called pthreads. When pthread_create is called, the library creates a new thread using the clone syscall, and we can trace this syscall with ptrace (assuming your kernel is older than 2.5.46). To do this, you’ll need to set some ptrace options after attaching to the debuggee:

ptrace(PTRACE_SETOPTIONS, m_pid, nullptr, PTRACE_O_TRACECLONE);

Now when clone is called, the process will be signaled with our old friend SIGTRAP. For the debugger in this series, you can add a case to handle_sigtrap which can handle the creation of the new thread:

case (SIGTRAP | (PTRACE_EVENT_CLONE << 8)):
    //get the new thread ID
    unsigned long event_message = 0;
    ptrace(PTRACE_GETEVENTMSG, pid, nullptr, message);

    //handle creation
    //...

Once you’ve got that, you can look in /proc/<pid>/task/ and read the memory maps and suchlike to get all the information you need.

GDB uses libthread_db, which provides a bunch of helper functions so that you don’t need to do all the parsing and processing yourself. Setting up this library is pretty weird and I won’t show how it works here, but you can go and read this tutorial if you’d like to use it.

The most complex part of multithreaded support is modelling the thread state in the debugger, particularly if you want to support non-stop mode or some kind of heterogeneous debugging where you have more than just a CPU involved in your computation.


The end!

Whew! This series took a long time to write, but I learned a lot in the process and I hope it was helpful. Get in touch on Twitter @TartanLlama or in the comments section if you want to chat about debugging or have any questions about the series. If there are any other debugging topics you’d like to see covered then let me know and I might do a bonus post.

Building Emacs 25.2 on XUbuntu 17.04

I haven’t done much with Ubuntu recently, but had to set up a laptop with XUbuntu 17.04. That came with Emacs 24.5 as the default emacs package, and as skeeto pointed out in the comments, with a separate emacs25 package for Emacs 25.1. I tend to run the latest release Emacs everywhere out of habit, […]

The post Building Emacs 25.2 on XUbuntu 17.04 appeared first on The Lone C++ Coder's Blog.

Signed-magnitude: The integer representation of choice for IoT?

What is the best representation to use for integer values in a binary computer? I’m guessing that most people think two’s complement is the answer, because this is the representation that all the computers they know about use (the Univac 1100/2200 series uses one’s complement; I don’t know of any systems currently in use that make use of signed magnitude, pointers welcome).

The C Standard allows implementations to support two’s complement, one’s complement and signed magnitude (the Univac 1100/2200 series has a C compiler). Is it time for the C Standard to drop support for one’s complement and signed magnitude?.

Why did two’s complement ‘win’ the integer representation battle and what are the chances that hardware vendors are likely to want to use a different representation in the future?

The advantage of two’s complement over the other representations is that the same hardware circuits can be used to perform arithmetic on unsigned and signed integer values. Not a big issue these days, but a major selling point back when chip real-estate was limited.

I can think of one market where signed magnitude is the ‘best representation’, extremely low power devices, such as those that extract power from the radio waves permeating the environment, or from the vibrations people generate as they move around.

Most of the power consumed by digital devices occurs when a bit flips from zero to one, or from one to zero. An application that spends most of its time processing signals that vary around zero (i.e., can have positive and negative values) will experience many bit flips, using a two’s complement representation, when the value changes from positive to negative, or vice-versa, e.g., from 0000000000000001 to 0000000000000000 to 1111111111111111; in signed magnitude a change of sign generates one extra bit-flip, e.g., 0000000000000001 to 0000000000000000 to 1000000000000001.

Simulations show around 30% few transitions for signed magnitude compared with two’s complement, for certain kinds of problems.

Signed magnitude would appear to be the integer representation of choice for some Internet-of-Things solutions.

Writing a Linux Debugger Part 9: Handling variables

Variables are sneaky. At one moment they’ll be happily sitting in registers, but as soon as you turn your head they’re spilled to the stack. Maybe the compiler completely throws them out of the window for the sake of optimization. Regardless of how often variables move around in memory, we need some way to track and manipulate them in our debugger. This post will teach you more about handling variables in your debugger and demonstrate a simple implementation using libelfin.


Series index

  1. Setup
  2. Breakpoints
  3. Registers and memory
  4. Elves and dwarves
  5. Source and signals
  6. Source-level stepping
  7. Source-level breakpoints
  8. Stack unwinding
  9. Handling variables
  10. Advanced topics

Before you get started, make sure that the version of libelfin you are using is the fbreg branch of my fork. This contains some hacks to support getting the base of the current stack frame and evaluating location lists, neither of which are supported by vanilla libelfin. You might need to pass -gdwarf-2 to GCC to get it to generate compatible DWARF information. But before we get into the implementation, I’ll give a more detailed description of how locations are encoded in DWARF 5, which is the most recent specification. If you want more information than what I write here, then you can grab the standard from here.

DWARF locations

The location of a variable in memory at a given moment is encoded in the DWARF information using the DW_AT_location attribute. Location descriptions can be either single location descriptions, composite location descriptions, or location lists.

  • Simple location descriptions describe the location of one contiguous piece (usually all) of an object. A simple location description may describe a location in addressable memory, or in a register, or the lack of a location (with or without a known value).
    • Example:
      • DW_OP_fbreg -32
      • A variable which is entirely stored -32 bytes from the stack frame base
  • Composite location descriptions describe an object in terms of pieces, each of which may be contained in part of a register or stored in a memory location unrelated to other pieces.
    • Example:
      • DW_OP_reg3 DW_OP_piece 4 DW_OP_reg10 DW_OP_piece 2
      • A variable whose first four bytes reside in register 3 and whose next two bytes reside in register 10.
  • Location lists describe objects which have a limited lifetime or change location during their lifetime.
    • Example:
      • <loclist with 3 entries follows>
        • [ 0]<lowpc=0x2e00><highpc=0x2e19>DW_OP_reg0
        • [ 1]<lowpc=0x2e19><highpc=0x2e3f>DW_OP_reg3
        • [ 2]<lowpc=0x2ec4><highpc=0x2ec7>DW_OP_reg2
      • A variable whose location moves between registers depending on the current value of the program counter

The DW_AT_location is encoded in one of three different ways, depending on the kind of location description. exprlocs encode simple and composite location descriptions. They consist of a byte length followed by a DWARF expression or location description. loclists and loclistptrs encode location lists. They give indexes or offsets into the .debug_loclists section, which describes the actual location lists.

DWARF Expressions

The actual location of the variables is computed using DWARF expressions. These consist of a series of operations which operate on a stack of values. There are an impressive number of DWARF operations available, so I won’t explain them all in detail. Instead I’ll give a few examples from each class of expression to give you a taste of what is available. Also, don’t get scared off by these; libelfin will take care off all of this complexity for us.

  • Literal encodings
    • DW_OP_lit0, DW_OP_lit1, …, DW_OP_lit31
      • Push the literal value on to the stack
    • DW_OP_addr <addr>
      • Pushes the address operand on to the stack
    • DW_OP_constu <unsigned>
      • Pushes the unsigned value on to the stack
  • Register values
    • DW_OP_fbreg <offset>
      • Pushes the value found at the base of the stack frame, offset by the given value
    • DW_OP_breg0, DW_OP_breg1, …, DW_OP_breg31 <offset>
      • Pushes the contents of the given register plus the given offset to the stack
  • Stack operations
    • DW_OP_dup
      • Duplicate the value at the top of the stack
    • DW_OP_deref
      • Treats the top of the stack as a memory address, and replaces it with the contents of that address
  • Arithmetic and logical operations
    • DW_OP_and
      • Pops the top two values from the stack and pushes back the logical AND of them
    • DW_OP_plus
      • Same as DW_OP_and, but adds the values
  • Control flow operations
    • DW_OP_le, DW_OP_eq, DW_OP_gt, etc.
      • Pops the top two values, compares them, and pushes 1 if the condition is true and 0 otherwise
    • DW_OP_bra <offset>
      • Conditional branch: if the top of the stack is not 0, skips back or forward in the expression by offset
  • Type conversions
    • DW_OP_convert <DIE offset>
      • Converts value on the top of the stack to a different type, which is described by the DWARF information entry at the given offset
  • Special operations
    • DW_OP_nop
      • Do nothing!

DWARF types

DWARF’s representation of types needs to be strong enough to give debugger users useful variable representations. Users most often want to be able to debug at the level of their application rather than at the level of their machine, and they need a good idea of what their variables are doing to achieve that.

DWARF types are encoded in DIEs along with the majority of the other debug information. They can have attributes to indicate their name, encoding, size, endianness, etc. A myriad of type tags are available to express pointers, arrays, structures, typedefs, anything else you could see in a C or C++ program.

Take this simple structure as an example:

struct test{
    int i;
    float j;
    int k[42];
    test* next;
};

The parent DIE for this struct is this:

< 1><0x0000002a>    DW_TAG_structure_type
                      DW_AT_name                  "test"
                      DW_AT_byte_size             0x000000b8
                      DW_AT_decl_file             0x00000001 test.cpp
                      DW_AT_decl_line             0x00000001

The above says that we have a structure called test of size 0xb8, declared at line 1 of test.cpp. All there are then many children DIEs which describe the members.

< 2><0x00000032>      DW_TAG_member
                        DW_AT_name                  "i"
                        DW_AT_type                  <0x00000063>
                        DW_AT_decl_file             0x00000001 test.cpp
                        DW_AT_decl_line             0x00000002
                        DW_AT_data_member_location  0
< 2><0x0000003e>      DW_TAG_member
                        DW_AT_name                  "j"
                        DW_AT_type                  <0x0000006a>
                        DW_AT_decl_file             0x00000001 test.cpp
                        DW_AT_decl_line             0x00000003
                        DW_AT_data_member_location  4
< 2><0x0000004a>      DW_TAG_member
                        DW_AT_name                  "k"
                        DW_AT_type                  <0x00000071>
                        DW_AT_decl_file             0x00000001 test.cpp
                        DW_AT_decl_line             0x00000004
                        DW_AT_data_member_location  8
< 2><0x00000056>      DW_TAG_member
                        DW_AT_name                  "next"
                        DW_AT_type                  <0x00000084>
                        DW_AT_decl_file             0x00000001 test.cpp
                        DW_AT_decl_line             0x00000005
                        DW_AT_data_member_location  176(as signed = -80)

Each member has a name, a type (which is a DIE offset), a declaration file and line, and a byte offset into the structure where the member is located. The types which are pointed to come next.

< 1><0x00000063>    DW_TAG_base_type
                      DW_AT_name                  "int"
                      DW_AT_encoding              DW_ATE_signed
                      DW_AT_byte_size             0x00000004
< 1><0x0000006a>    DW_TAG_base_type
                      DW_AT_name                  "float"
                      DW_AT_encoding              DW_ATE_float
                      DW_AT_byte_size             0x00000004
< 1><0x00000071>    DW_TAG_array_type
                      DW_AT_type                  <0x00000063>
< 2><0x00000076>      DW_TAG_subrange_type
                        DW_AT_type                  <0x0000007d>
                        DW_AT_count                 0x0000002a
< 1><0x0000007d>    DW_TAG_base_type
                      DW_AT_name                  "sizetype"
                      DW_AT_byte_size             0x00000008
                      DW_AT_encoding              DW_ATE_unsigned
< 1><0x00000084>    DW_TAG_pointer_type
                      DW_AT_type                  <0x0000002a>

As you can see, int on my laptop is a 4-byte signed integer type, and float is a 4-byte float. The integer array type is defined by pointing to the int type as its element type, a sizetype (think size_t) as the index type, with 2a elements. The test* type is a DW_TAG_pointer_type which references the test DIE.


Implementing a simple variable reader

As mentioned, libelfin will deal with most of the complexity for us. However, it doesn’t implement all of the different methods for representing variable locations, and handling a lot of them in our code would get pretty complex. As such, I’ve chosen to only support exprlocs for now. Feel free to add support for more types of expression. If you’re really feeling brave, submit some patches to libelfin to help complete the necessary support!

Handling variables is mostly down to locating the different parts in memory or registers, then reading or writing is the same as you’ve seen before. I’ll only show you how to implement reading for the sake of simplicity.

First we need to tell libelfin how to read registers from our process. We do this by creating a class which inherits from expr_context and uses ptrace to handle everything:

class ptrace_expr_context : public dwarf::expr_context {
public:
    ptrace_expr_context (pid_t pid) : m_pid{pid} {}

    dwarf::taddr reg (unsigned regnum) override {
        return get_register_value_from_dwarf_register(m_pid, regnum);
    }

    dwarf::taddr pc() override {
        struct user_regs_struct regs;
        ptrace(PTRACE_GETREGS, m_pid, nullptr, &regs);
        return regs.rip;
    }

    dwarf::taddr deref_size (dwarf::taddr address, unsigned size) override {
        //TODO take into account size
        return ptrace(PTRACE_PEEKDATA, m_pid, address, nullptr);
    }

private:
    pid_t m_pid;
};

The reading will be handled by a read_variables function in our debugger class:

void debugger::read_variables() {
    using namespace dwarf;

    auto func = get_function_from_pc(get_pc());

    //...
}

The first thing we do above is find the function which we’re currently in. Then we need to loop through the entries in that function, looking for variables:

    for (const auto& die : func) {
        if (die.tag == DW_TAG::variable) {
            //...
        }
    }

We get the location information by looking up the DW_AT_location entry in the DIE:

            auto loc_val = die[DW_AT::location];

Then we ensure that it’s an exprloc and ask libelfin to evaluate the expression for us:

            if (loc_val.get_type() == value::type::exprloc) {
                ptrace_expr_context context {m_pid};
                auto result = loc_val.as_exprloc().evaluate(&context);

Now that we’ve evaluated the expression, we need to read the contents of the variable. It could be in memory or a register, so we’ll handle both cases:

                switch (result.location_type) {
                case expr_result::type::address:
                {
                    auto value = read_memory(result.value);
                    std::cout << at_name(die) << " (0x" << std::hex << result.value << ") = "
                              << value << std::endl;
                    break;
                }

                case expr_result::type::reg:
                {
                    auto value = get_register_value_from_dwarf_register(m_pid, result.value);
                    std::cout << at_name(die) << " (reg " << result.value << ") = "
                              << value << std::endl;
                    break;
                }

                default:
                    throw std::runtime_error{"Unhandled variable location"};
                }

As you can see I’ve simply printed out the value without interpreting it based on the type of the variable. Hopefully from this code you can see how you could support writing variables, or searching for variables with a given name.

Finally we can add this to our command parser:

    else if(is_prefix(command, "variables")) {
        read_variables();
    }

Testing it out

Write a few small functions which have some variables, compile it without optimizations and with debug info, then see if you can read the values of your variables. Try writing to the memory address where a variable is stored and see the behaviour of the program change.


Nine posts down, one to go! Next time I’ll be talking about some more advanced concepts which might interest you. For now you can find the code for this post here

Visual Lint 6.0.4.281 has been released

Visual Lint 6.0.4.281 is now available. This is a recommended maintenance update for Visual Lint 6.0, and includes the following changes:
  • Microsoft Visual C++ Express Edition 2010/2012 system headers can now be used as a source of system headers for PC-lint 8.0/9.0 analysis of VS2013/15/17 projects.
  • Fixed a potential crash in VisualLintGui if a corrupt window layout is restored on startup.
  • Fixed a bug in the parsing of Visual C++ 2010-2017 project (.vcxproj) files in which the configurations are defined in an external property (.props) file located via a user macro.
  • Fixed a bug in the parsing of Green Hills projects which affected whether some subprojects were read correctly.
  • Various changes to the installer:
    1. The installer now prompts for affected applications (Visual Studio, Atmel Studio, AVR Studio, Eclipse, VisualLintConsole and VisualLintGui) to be closed before installation can proceed.
    2. The installer now installs VSIX extensions to Visual Studio 2017 and Atmel [AVR] Studio silently.
    3. Revised the order of registration of the Visual Studio plug-in with each version of Visual Studio so that the newest versions are now registered first.
    4. Uninstallation no longer incorrectly runs "Configuring Visual Studio..." steps if the VS plug-in is not selected for installation.
    5. The "Installing Visual Lint" progress bar is now updated while Visual Studio, Atmel Studio and Eclipse installations are being registered.
    6. Improved the logging of VSIX extension installation/uninstallation.
Download Visual Lint 6.0.4.281

Programmer’s Rorschach test

The picture above, I recently added this picture to Continuous Digital for a discussion of teams. When you look at it what do you see:

An old style structure chart, or an organization chart?

It could be either and anyone who knows of Conway’s Law shouldn’t be surprised.

When I was taught Modula-2 at college these sort of structure charts were considered superior to the older flow charts. This is functional decomposition, take a problem, break it down to smaller parts and implement them.

And that is the same idea behind traditional hierarchical organizational structure. An executive heads a division, he has a number of managers under him who manage work, each one of these manage several people who actually do the work (or perhaps manage more manager who manage the people who do the work!)

Most organizations are still set up this way. It is probably unsurprising that 50 years ago computer programmers copied this model when designing their systems – Conway’s Law, the system is a copy of the organization.

Fast forward to today, we use object oriented languages and design but most of our organizations are still constrained by hierarchical structure, that creates a conflict. The company is structurally decomposed but our code is object oriented.

The result is conflict and in many cases the organization wins – who hasn’t seen an object oriented system that is designed in layers?

While the conflict exists both system and organization under perform because time and energy are spent living the conflict, managing the conflict, overcoming the conflict.

What would the object-oriented company look like?

If we accept that object oriented design and programming are superior to procedural programming (and in general I do although I miss Modula-2) then it becomes necessary to change the organization to match the software design – reverse Conway’s Law or Yawnoc. That means we need teams which look and behave like objects:

  • Teams are highly cohesive (staffed with various skills) and lightly coupled (dependencies are minimised and the team take responsibility)
  • Teams are responsible for a discrete part of the system end-to-end
  • Teams add value in their own right
  • Teams are free to vary organizational implementation behind well defined interface
  • Teams are tested, debugged and maintained: they have been through the storming phase, are now performing and are kept together

There are probably some more attributes I could add here, please make your own suggestions in the comments below.

To regular readers this should all sound familiar, I’ve been exposing these ideas for a while, they draw on software design and Amoeba management, I discuss them at greater length Xanpan, The Xanpan Appendix and now Continuous Digital – actually, Continuous Digital directly updates some chapters from the Appendix.

And like so many programmers have found over the years, classes which are named “Manager” are more than likely poorly named and poorly designed. Manager is a catch all name, the class might well be doing something very useful but it can be named better. If it isn’t doing anything useful, then maybe it should be refactored into something that is. Most likely the ManagerClass is doing a lot of useful stuff but it is far from clear that it all belongs together. (See the management mini-series.)

Sometimes managers or manager classes  make sense, however both deserve closer examination. Are they vestige from the hierarchal world? Do they perform specialist functions which could be packaged and named better? Perhaps they are necessary, perhaps they are necessary for smoothing the conflict between the hierarchal organization and object oriented world.

Transaction costs can explain both managers and manager classes. There are various tasks which require knowledge of other tasks, or access to the same data. It is cheaper, and perhaps simpler, to put these diverse things together rather than pay the cost of spreading access out.

Of course if you accept the symbiosis of organization and code design then one should ask: what should the organization look like when the code is functional? What does the Lisp, Clojure or F# organization look like?

And, for that matter, what does the organization look like when you program in Prolog? What does a declarative organization look like?

Finally, I was about to ask about SQL, what does the relational organization look like, but I think you’ve already guessed the answer to this one: a matrix, probably a dysfunctional matrix.

The post Programmer’s Rorschach test appeared first on Allan Kelly Associates.

On Divisions – student

The Baron's game most recent game consisted of a series of some six wagers upon the toss of an unfair coin that turned up one side nine times out of twenty and the other eleven times out of twenty at a cost of one fifth part of a coin. Sir R----- was to wager three coins from his purse upon the outcome of each toss, freely divided between heads and tails, and was to return to it twice the value he wagered correctly.

Clearly, our first task in reckoning the fairness of this game is to figure Sir R-----'s optimal strategy for placing his coins. To do this we shall need to know his expected winnings in any given round for any given placement of his coins.

On Divisions – student

The Baron's game most recent game consisted of a series of some six wagers upon the toss of an unfair coin that turned up one side nine times out of twenty and the other eleven times out of twenty at a cost of one fifth part of a coin. Sir R----- was to wager three coins from his purse upon the outcome of each toss, freely divided between heads and tails, and was to return to it twice the value he wagered correctly.

Clearly, our first task in reckoning the fairness of this game is to figure Sir R-----'s optimal strategy for placing his coins. To do this we shall need to know his expected winnings in any given round for any given placement of his coins.

Exciting: Something different for those Leading & Managing Agile

“Some readers have found it curious that The Mythical Man Month devotes most of the essays to the managerial aspects of software engineering, rather than the many technical issues. This bias … sprang from [my] conviction that the quality of the people on a project, and their organization and management, are much more important factors in the success than are the tools they use or the technical approaches they take.” Frederick P. Brooks, The Mythical Man Month – Anniversary Edition (1995a, p.276)

This is one of my favourite quotes about software development.

Those of you who followed my management mini-series last year might remember some of my conclusions, specifically:

  • Management is still needed in most software teams
  • Even if you don’t have dedicated managers there is still management work to do; and without dedicated management that management work gets spread around so in a self-organizing team more people need to do management work

I can’t remember if I said it in that series or in another post: the quality of management in software development is often poor.

So now I’m going to do something about it.

Part of the problem is that very very few software engineers ever take time to learn about management. Sure they learn about Java, BDD, AWS, CSS etc. etc. but most people learn “managing” by watching the people who had the job before them. Unfortunately that tends to propagate poor management.

Why don’t people take the time to learn management skills the way they do technical skills?

A few days in the classroom learning about Java is great, why not spend a little time learning about management?

Now most readers probably know that I provide training, usually agile training to teams, but I have long felt that simply offering a two day training course for current and future managers wouldn’t be enough. I’ve long dreamt of offering a course that ran over a series of weeks, that allowed people to mix classroom learning and discussion of topics with hands on practice and, perhaps more importantly, reflection and group discussion.

Now I’m partnering up with Learning Connexions to offer just this course – I’m more excited about this than anything I have been in a while!

It’s going to run this autumn, September to December, half a day every two weeks.

In each session I’ll present some material but more importantly we’ll have discussions between people doing work. We’ll try and apply the material to real work. I expect group discussions will go beyond the stated subjects and allow everyone – me included! – to learn from one another.

Not only is this a new course it is a new way of approaching learning in work. Since it is new the course we are going to keep the price down. We’re running it on Friday afternoons so hopefully it won’t interfere with the usual work week too much.

Here are the hypothesis I’m testing:

  • Software development can be improved if software management can be improved: current software managers, aspiring software managers and software engineers will all do a better job if they take time to pro-actively learn about key issues in the software industry.
  • People will learn more and deeper if it is organised as a little and often.
  • Encouraging self reflection and group discussion about topics and real life problems will enhance learning.

The hypothesis you would be testing by coming along are:

  • I will learn more by discussing key topics with a cross section of my peers
  • An investment of six-half says and a little less than £1000 will pay back in increased effectiveness, productivity, organization, focus….

More information on the workshops on the training pages. (Yes, this is the new website.)

I hope, I very much hope we can make this work…. – please get over to Learning Connexions and book Leading & Managing Agile today. Even if you cannot attend I’d love to know your thoughts on this, please give me feedback – comment here or mail me direct, allan@allankelly.net.

And apologies, I’m going to be talking more about this in the next few weeks as I put material together and get more and more excited!

The post Exciting: Something different for those Leading & Managing Agile appeared first on Allan Kelly Associates.

Vacancy: Executive PA / Office Manager


Naked Element are a software development company based in Norwich looking to recruit a self motivated, outgoing, well organised person looking for variety in a small, yet progressive tech company. There is opportunity for the right person to grow into a more specialised role, based on your strengths, as the company grows.

Salary: £18-20,000 per annum salary (depending on experience)

Hours: 37.5 hours per week

Location: New Patricks Yard, 2 Recorder Road, Norwich, Norfolk, NR1 1NR

Application Deadline: 28th July 2017

Essential skills and qualities:

  • Good client and communication skills
  • Exceptional organisation
  • Self motivated
  • You thrive in a fast-paced office environment
  • Competent user of email systems, document creation and management software packages
  • Ability to prioritise

Desirable skills and qualities:

  • An Interest in Software, Technology, Development, or any wider part of the ICT industry
  • Previous Administration and Office Management experience 
  • A Level 3 qualification or equal in Administration or Business Management

Main Responsibilities:

Office Management

  • Running the office on a day-to-day basis depending on the needs of the business, it’s directors and employees
  • Purchasing stationery and equipment
  • Liaising with suppliers
  • Answering the phone
  • Preparing agendas, documents and contracts

Company Administration

  • Book keeping
  • Managing finances
  • Financial forecasting/producing reports
  • Paying and raising invoices
  • Paying bills
  • General administration
  • Payroll

Social Media & Marketing Assistant

  • Assisting the commercial director in all elements of marketing as required
  • Setting up daily social media
  • Preparing and sending marketing material
  • Attending networking events
  • Exhibiting at events

Sales Assistant

  • Assisting the commercial director in all elements of sales as required
  • Prospecting
  • Warm calling
  • Meeting prospects & clients
  • Sandler training provided

Account & Project Management

  • Day to day running of projects
  • Project reporting
  • Liaising with all stakeholders during projects
  • Regular client reviews & other account management as necessary

 Executive PA

  • Managing diaries for both the Commercial Director & CEO
  • Booking events & meetings
  • Booking travel

Benefits

  • Pension after 3 months
  • 6 Month probationary period
  • A lovely, air-conditioned working environment in the centre of Norwich
  • Flexible working hours

This is the perfect opportunity for those looking for an interesting, never the same each day role to grow their skills. You will benefit from the guidance of staff with over 20 years experience in Business, Finance and Project Management. This is also a chance to gain an in depth insight into the software development industry.

If you would like to apply for this opportunity, please send CV’s to emma@nakedelement.co.uk 

Vacancy: Executive PA / Office Manager


Naked Element are a software development company based in Norwich looking to recruit a self motivated, outgoing, well organised person looking for variety in a small, yet progressive tech company. There is opportunity for the right person to grow into a more specialised role, based on your strengths, as the company grows.

Salary: £18-20,000 per annum salary (depending on experience)

Hours: 37.5 hours per week

Location: New Patricks Yard, 2 Recorder Road, Norwich, Norfolk, NR1 1NR

Application Deadline: 28th July 2017

Essential skills and qualities:

  • Good client and communication skills
  • Exceptional organisation
  • Self motivated
  • You thrive in a fast-paced office environment
  • Competent user of email systems, document creation and management software packages
  • Ability to prioritise

Desirable skills and qualities:

  • An Interest in Software, Technology, Development, or any wider part of the ICT industry
  • Previous Administration and Office Management experience 
  • A Level 3 qualification or equal in Administration or Business Management

Main Responsibilities:

Office Management

  • Running the office on a day-to-day basis depending on the needs of the business, it’s directors and employees
  • Purchasing stationery and equipment
  • Liaising with suppliers
  • Answering the phone
  • Preparing agendas, documents and contracts

Company Administration

  • Book keeping
  • Managing finances
  • Financial forecasting/producing reports
  • Paying and raising invoices
  • Paying bills
  • General administration
  • Payroll

Social Media & Marketing Assistant

  • Assisting the commercial director in all elements of marketing as required
  • Setting up daily social media
  • Preparing and sending marketing material
  • Attending networking events
  • Exhibiting at events

Sales Assistant

  • Assisting the commercial director in all elements of sales as required
  • Prospecting
  • Warm calling
  • Meeting prospects & clients
  • Sandler training provided

Account & Project Management

  • Day to day running of projects
  • Project reporting
  • Liaising with all stakeholders during projects
  • Regular client reviews & other account management as necessary

 Executive PA

  • Managing diaries for both the Commercial Director & CEO
  • Booking events & meetings
  • Booking travel

Benefits

  • Pension after 3 months
  • 6 Month probationary period
  • A lovely, air-conditioned working environment in the centre of Norwich
  • Flexible working hours

This is the perfect opportunity for those looking for an interesting, never the same each day role to grow their skills. You will benefit from the guidance of staff with over 20 years experience in Business, Finance and Project Management. This is also a chance to gain an in depth insight into the software development industry.

If you would like to apply for this opportunity, please send CV’s to emma@nakedelement.co.uk 

Exciting: Something different for those Leading & Managing Agile

HeadacheiStock_000014496990SmallLeft-2017-07-20-10-29.jpg

“Some readers have found it curious that The Mythical Man Month devotes most of the essays to the managerial aspects of software engineering, rather than the many technical issues. This bias … sprang from [my] conviction that the quality of the people on a project, and their organization and management, are much more important factors in the success than are the tools they use or the technical approaches they take.” Frederick P. Brooks, The Mythical Man Month – Anniversary Edition (1995a, p.276)

This is one of my favourite quotes about software development.

Those of you who followed my management mini-series last year might remember some of my conclusions, specifically:

  • Management is still needed in most software teams
  • Even if you don’t have dedicated managers there is still management work to do; and without dedicated management that management work gets spread around so in a self-organizing team more people need to do management work

I can’t remember if I said it in that series or in another post: the quality of management in software development is often poor.

So now I’m going to do something about it.

Part of the problem is that very very few software engineers ever take time to learn about management. Sure they learn about Java, BDD, AWS, CSS etc. etc. but most people learn “managing” by watching the people who had the job before them. Unfortunately that tends to propagate poor management.

Why don’t people take the time to learn management skills the way they do technical skills?

A few days in the classroom learning about Java is great, why not spend a little time learning about management?

Now most readers probably know that I provide training, usually agile training to teams, but I have long felt that simply offering a two day training course for current and future managers wouldn’t be enough. I’ve long dreamt of offering a course that ran over a series of weeks, that allowed people to mix classroom learning and discussion of topics with hands on practice and, perhaps more importantly, reflection and group discussion.

Now I’m partnering up with Learning Connexions to offer just this course – I’m more excited about this than anything I have been in a while!

It’s going to run this autumn, September to December, half a day every two weeks.

In each session I’ll present some material but more importantly we’ll have discussions between people doing work. We’ll try and apply the material to real work. I expect group discussions will go beyond the stated subjects and allow everyone – me included! – to learn from one another.

Not only is this a new course it is a new way of approaching learning in work. Since it is new the course we are going to keep the price down. We’re running it on Friday afternoons so hopefully it won’t interfere with the usual work week too much.

Here are the hypothesis I’m testing:

  • Software development can be improved if software management can be improved: current software managers, aspiring software managers and software engineers will all do a better job if they take time to pro-actively learn about key issues in the software industry.
  • People will learn more and deeper if it is organised as a little and often.
  • Encouraging self reflection and group discussion about topics and real life problems will enhance learning.

The hypothesis you would be testing by coming along are:

  • I will learn more by discussing key topics with a cross section of my peers
  • An investment of six-half says and a little less than £1000 will pay back in increased effectiveness, productivity, organization, focus….

I hope, I very much hope we can make this work…. – please get over to Learning Connexions and book Leading & Managing Agile today. Even if you cannot attend I’d love to know your thoughts on this, please give me feedback – comment here or mail me direct, allan@allankelly.net.

And apologies, I’m going to be talking more about this in the next few weeks as I put material together and get more and more excited!

The post Exciting: Something different for those Leading & Managing Agile appeared first on Allan Kelly Associates.

Exciting: Something different for those Leading & Managing Agile

HeadacheiStock_000014496990SmallLeft-2017-07-20-10-29.jpg

“Some readers have found it curious that The Mythical Man Month devotes most of the essays to the managerial aspects of software engineering, rather than the many technical issues. This bias … sprang from [my] conviction that the quality of the people on a project, and their organization and management, are much more important factors in the success than are the tools they use or the technical approaches they take.” Frederick P. Brooks, The Mythical Man Month – Anniversary Edition (1995a, p.276)

This is one of my favourite quotes about software development.

Those of you who followed my management mini-series last year might remember some of my conclusions, specifically:

  • Management is still needed in most software teams
  • Even if you don’t have dedicated managers there is still management work to do; and without dedicated management that management work gets spread around so in a self-organizing team more people need to do management work

I can’t remember if I said it in that series or in another post: the quality of management in software development is often poor.

So now I’m going to do something about it.

Part of the problem is that very very few software engineers ever take time to learn about management. Sure they learn about Java, BDD, AWS, CSS etc. etc. but most people learn “managing” by watching the people who had the job before them. Unfortunately that tends to propagate poor management.

Why don’t people take the time to learn management skills the way they do technical skills?

A few days in the classroom learning about Java is great, why not spend a little time learning about management?

Now most readers probably know that I provide training, usually agile training to teams, but I have long felt that simply offering a two day training course for current and future managers wouldn’t be enough. I’ve long dreamt of offering a course that ran over a series of weeks, that allowed people to mix classroom learning and discussion of topics with hands on practice and, perhaps more importantly, reflection and group discussion.

Now I’m partnering up with Learning Connexions to offer just this course – I’m more excited about this than anything I have been in a while!

It’s going to run this autumn, September to December, half a day every two weeks.

In each session I’ll present some material but more importantly we’ll have discussions between people doing work. We’ll try and apply the material to real work. I expect group discussions will go beyond the stated subjects and allow everyone – me included! – to learn from one another.

Not only is this a new course it is a new way of approaching learning in work. Since it is new the course we are going to keep the price down. We’re running it on Friday afternoons so hopefully it won’t interfere with the usual work week too much.

Here are the hypothesis I’m testing:

  • Software development can be improved if software management can be improved: current software managers, aspiring software managers and software engineers will all do a better job if they take time to pro-actively learn about key issues in the software industry.
  • People will learn more and deeper if it is organised as a little and often.
  • Encouraging self reflection and group discussion about topics and real life problems will enhance learning.

The hypothesis you would be testing by coming along are:

  • I will learn more by discussing key topics with a cross section of my peers
  • An investment of six-half says and a little less than £1000 will pay back in increased effectiveness, productivity, organization, focus….

I hope, I very much hope we can make this work…. – please get over to Learning Connexions and book Leading & Managing Agile today. Even if you cannot attend I’d love to know your thoughts on this, please give me feedback – comment here or mail me direct, allan@allankelly.net.

And apologies, I’m going to be talking more about this in the next few weeks as I put material together and get more and more excited!

The post Exciting: Something different for those Leading & Managing Agile appeared first on Allan Kelly Associates.

C++17 attributes – maybe_unused, fallthrough and nodiscard

C++17 adds three new attributes for programmers to better express their intent to the compiler and readers of the code: maybe_unused, fallthrough, and nodiscard. This is a quick post to outline what they do and why they are useful.

In case you aren’t familiar with the concept, attributes in C++ allow you mark functions, variables, and other entities with compiler-specific or standard properties. The standard attributes prior to C++17 are noreturn (function doesn’t return), deprecated (entity is going to be removed in a later version) and carries_dependency (used for optimizing atomic operations). You mark an entity with an attribute like this:

[[deprecated]]
void do_thing_the_old_way();

void do_thing_the_new_way();

do_thing_the_old_way(); // Compiler warning: deprecated
do_thing_the_new_way(); // No warning

With that out of the way, on to the new attributes!


maybe_unused

[[maybe_unused]] suppresses compiler warnings about unused entities. Usually unused functions or variables indicates a programmer error – if you never use it, why is it there? – but sometimes they can be intentional, such as variables which are only used in release mode, or functions only called when logging is enabled.

Variables

void emits_warning(bool b) {
     bool result = get_result();
     // Warning emitted in release mode
     assert (b == result);
     //...
}

void warning_suppressed(bool b [[maybe_unused]]) {
     [[maybe_unused]] bool result = get_result();
     // Warning suppressed by [[maybe_unused]]
     assert (b == result);
     //...
}

Functions

static void log_with_warning(){}
[[maybe_unused]] static void log_without_warning() {}

#ifdef LOGGING_ENABLED
log_with_warning();    // Warning emitted if LOGGING_ENABLED is not defined
log_without_warning(); // Warning suppressed by [[maybe_unused]]
#endif

I feel like this attribute is more useful to supress compiler warnings than to document your code for others, but at least we now have a standard way to do so.


fallthrough

[[fallthrough]] indicates that a fallthrough in a switch statement is intentional. Missing a break or return in a switch case is a very common programmer error, so compilers usually warn about it, but sometimes a fallthrough can result in some very terse code.

Say we want to process an alert message. If it’s green, we do nothing; if it’s yellow, we record the alert; if it’s orange we record and trigger the alarm; if it’s red, we record, trigger the alarm and evacuate.

void process_alert (Alert alert) {
     switch (alert) {
     case Alert::Red:
         evacuate();
         // Warning: this statement may fall through

     case Alert::Orange:
         trigger_alarm();
         [[fallthrough]]; //yes, you do need the semicolon
         // Warning suppressed by [[fallthrough]]

     case Alert::Yellow:
         record_alert();
         return;

     case Alert::Green:
         return;
     }
}

The most important function of fallthrough is as documentation for maintainers. The presence of it in the code above shows anyone looking at the code that an orange alert is absolutely supposed to be recorded. Without the fallthrough in the Alert::Red case, it is not obvious whether a red alert is supposed to trigger the alarm and be recorded, or just evacuate everyone.


nodiscard

Functions declared with [[nodiscard]] should not have their return values ignored by the caller. This can be useful if you want to ensure that callers check a return value, or that some scope guard object has a reasonable lifetime. Types can be marked [[nodiscard]] to implicitly mark all functions returning that type as the same.

Functions

[[nodiscard]] error do_something (thing&);
error do_something_else (thing&);

do_something(my_thing); // Warning: ignored return value
do_something_else(my_thing);

Classes

[[nodiscard]]
struct lock_guard;

lock_guard lock (mutex& m);

{
    lock(my_mutex); // Warning: ignored return value
    // critical section
}

Those warnings will help the user notice that do_something_else might be given a bad object, or the critical section won’t be locked.


Compilers have shipped non-standard extensions to express these concepts for years, but it’s great that we now have standard methods to do the same. Leave a comment if you have ideas for other attributes you would like to be added to the language!

For some more examples of using these attributes, see posts by Kenneth Benzie, Bartłomiej Filipek, and Arne Mertz.

A new blog home

After over 10 years blogging on Blogger I’m moving.

Actually, for the last 10 years I’ve been running three websites. The blog, AllanKelly.net and my company, commercial, site, SoftwareStrategy.co.uk. This kind of made sense once upon a time but it has increasingly meant that all three sites were a little neglected.

My plan is to merge all three sites into this one eventually. Hopefully that will also benefit my Google rankings but, it also means I need to keep all those inbound links!

This is going to be a gradual process…

The post A new blog home appeared first on Allan Kelly Associates.

Software systems are the product of cognitive capitalism

Economics obviously has a significant impact on the production of software systems; it is the second chapter of my empirical software engineering book (humans, who are the primary influencers, are the first chapter; technically the Introduction is the first chapter, but you know what I mean).

I have never been happy with the chapter title “Economics”; it does not capture the spirit of what I want to talk about. Yes, a lot of the technical details covered are to be found in economics related books and courses, but how do these technical details fit into a grand scheme?

I was recently reading the slim volume “Dead Man Working” by Cederström and Fleming and the phrase cognitive capitalism jumped out at me; here was a term that fitted the ideas I had been trying to articulate. It took a couple of days before I took the plunge and changed the chapter title. In the current draft pdf little else has changed in the ex-Economics chapter (e.g., a bit of a rewrite of the first few lines), but now there is a coherent concept to mold the material around.

Ecosystems chapter added to “Empirical software engineering using R”

The Ecosystems chapter of my Empirical software engineering book has been added to the draft pdf (download here).

I don’t seem to be able to get away from rewriting everything, despite working on the software engineering material for many years. Fortunately the sparsity of the data keeps me in check, but I keep finding new and interesting data (not a lot, but enough to slow me down).

There is still a lot of work to be done on the ecosystems chapter, not least integrating all the data I have been promised. The basic threads are there, they just need filling out (assuming the promised data sets arrive).

I did not get any time to integrate in the developer and economics data received since those draft chapters were released; there has been some minor reorganization.

As always, if you know of any interesting software engineering data, please tell me.

I’m looking to rerun the workshop on analyzing software engineering data. If anybody has a venue in central London, that holds 30 or so people+projector, and is willing to make it available at no charge for a series of free workshops over several Saturdays, please get in touch.

Projects chapter next.