Further Still On Natural Analogarithms – student

For several months now my fellow students and I have been exploring -space, being the set of infinite dimensional vectors whose elements are the powers of the prime factors of the roots of rational numbers, which we chanced upon whilst attempting to define a rational valued logarithmic function for such numbers.
We have seen how we might define functions of roots of rationals employing the magnitude of their associated -space vectors and that the iterative computation of such functions may yield cyclical sequences, although we conspicuously failed to figure a tidy mathematical rule governing their lengths.
The magnitude is not the only operation of linear algebra that we might bring to bear upon such roots, however, and we have lately busied ourselves investigating another.

A 1948 viewpoint on developer vs. computer time

For a long time now developer time has been a lot more expensive than computer time. The idea that developers should organize what they do, so as to maximize the efficiency of computer time rather than their own time, is considered to be an echo from a bygone age.

Until recently, I thought the transition from this bygone age, when computer time was considered more important than developer time, started in the late 1960s. Don’t ask me why I thought this, put it down to personal bias.

I was recently reading A Survey of Eniac Operations and Problems: 1946-1952, published in 1952, and what did I find:

“Early in 1948, R. F. Clippinger and some of his associates, in the course of coding the solution of …, were forced to adopt a different method of using the Eniac in order to fit their problem on the machine. …. The experience with this method (first discussed in reference 1), led J. von Neumann to suggest the use of a serial code for control of the Eniac. Such a code was devised and employed with the Eniac beginning in March 1948. Operation of the Eniac with this code was several times slower than either the original method of direct programming or the code for parallel operation. However, the resulting simplification of coding techniques and other advantages far outweighed this disadvantage.

In other words, in 1948, the people using one of the few computers in the world, which clocked at 100KHz, considered developer time to be more important than computer time.

How should we organize our teams?

StartingPoint-2018-09-18-12-23.jpg

Q1: How should we organize our teams?
My team is owner of different trading platforms and the core services around it. But we depend heavily on other products (e.g. financial feeds, client identification, services to send orders to stock markets, etc.). And of course each of the team managing these services have other platforms that are their clients.

When Vasco Duarte and I ran the #NoEstimates/#NoProjects workshop (or #NoNoWorkshop as I think of it) in Switzerland last month the attendees asked some good questions. With Project Myopia done and published, and Continuous Digital almost done it seems like a good time to repeat, and elaborate, the answers publicly. This will take a few blog posts to work through.

(I now have several Continuous Digital workshops and briefings available, please let me know what you think. Vasco and I are looking at repeating the workshop in London later this year, please get in touch if you are interested.)

The picture above is the way I see the question, if you have another interpretation, or another scenario please let me know.

The Continuous Digital model is for stable, long standing, autonomous, value seeking teams staffed with all the skills they need. Much of my thinking derives from Amoeba Management. Importantly each team needs to see how it adds value. In this case the business facing teams can see this – they enable the business do make money. But the back office teams find it hard see how they add value.

Now there are several possible answers to this question most of which involve some sort of re-organizations.

Option 1: Share the value

This solution does not involve reorganisation and comes straight from the pages of Amoeba management: allocate some portion of the value earned by the business facing teams to the teams they depend on. For example, the Trading platforms team might generate $10m each year. It could not do this without the services of the other three teams. Therefore some portion of Trading team’s earned value is passed to those teams.

Think about it, Trading Platforms affectively buys the services of three other teams. If those teams did not exist Trading Platforms would need to do that work themselves. Therefore those teams are contributing and deserve some credit.

This requires a serious conversation and probably needs more senior managers to intervene. Indeed, in Amoeba Management, Kazuo Inamori says that such decisions were among the most difficult ones facing Kyocera and often required more senior managers to make the final decision.

Nor is it always clear who buys from who, does a Sales Amoeba earn the value and pass part of it to the manufacturing team who build the product. Or does the Manufacturing Amoeba hire the Sales Amoeba to get their product to customers and therefore book the revenue and pass some to sales?

In the case above one might find it better to consider the value of the whole trading team including both the traders and the programmers who make the platform. Or perhaps the traders rent the platform from the technologists.

According to Inamori Kyocera standing allocations are set between teams. Alternatively one might create an internal market in which teams bought services from others on a piecemeal basis. On the one hand I like that idea model because it would allow for negotiation and trade-offs. On the other hand I imagine it creating a whole new set of bureaucracy, politics and internal sales. On balance, I’d fix the allocations and review periodically.

Option 2: Vertical slice

If you look at the picture above you might replace the word “team” with “library” or “services” and you would have a module dependency chart. Conway’s Law is at work – the organization and system reflect each other. (Although without knowing the history here it is difficult to say whether this was Conway’s Law or Reverse Conway’s Law at work.)

The services can stay as they are but we just disband the back-office teams and pass their responsibilities to the (enlarged) business teams.

Vertical-2018-09-18-12-23.jpg

The three teams will need to co-operate and co-ordinate with each other as they now have shared responsibilities. This itself can be a problem – two developers changing the same code anyone? But the world has moved on. Technology has improved.

In the days of SCCS, Visual Source-Unsafe, manual testing and monthly deployments it was a pain to have two teams working on the same code. But distributed source code control, automated testing and continuous delivery make this option far more viable than it once was.

On the plus side each team can work at their own pace on their own priorities and knowledge is spread around. On the downside teams can still trip up each other, they may duplicate work and specialist knowledge can get lost. (Note I am not saying “nobody has overall design authority” is a downside because while a single Linus can be an advantage it can also be a liability.)

One more problem here: this solution directly breaks Conway’s Law. In theory it could work but quite possibly the homomorphic force behind Conway’s Law might reassert itself. This might create some problems further down the line so needs monitoring.

Option 3: Independence

Taking option 2 to the extreme you might even separate the teams completely. Again there are plus and minuses.

Indepence-2018-09-18-12-23.jpg

On the one hand the teams are completely independent, they can move at their own pace, with their own priorities, value is clearly attributed and there is now resilience in the system and risk is reduced.

However, there is duplication. Not only does this mean more work it means that there may be inconsistencies, a client recognised by Trading might not be recognised by Yet Another.

Both options 2 and 3 demand larger teams and this option might requires more people overall. One can’t be sure because teams might come up with innovative solutions or come up with some new mechanism for sharing.

I’m sure some readers will discount this option very quickly but there are big benefits to complete independence – particularly when teams are separated geographically (e.g. Trading in London, Some Other in Frankfurt and Yet Another in Singapore) or when they are addressing different markets. One of the dangers of shared modules is that they become bloated by generic features nobody really wants but someone has to pay for.

This approach might also be advantageous when the company is in a growth and innovation mode. Let each team grow as fast as they can and innovate. In time a “winner” might emerge or common elements appear naturally.

Another variation on option 3 would be to have one team take the lead. Say Trading, this would be a larger team who developed the share services as part of their business facing work. But they would not “genericise” those services. The other, smaller teams, would do what they needed, when they needed, to service their own value streams.


That is three options. I could come up with some more, none is perfect. The important things are:

  • Create a clear way for teams to see the effects of their work and share in the value.
  • Allow teams autonomy in decision making and reduce dependencies.
  • Keep it simple so everyone can see cause and effect.
  • And of course, keep the teams stable – don’t break them up.

If you have any questions about Continuous Digital and #NoProjects please mail them over and I’ll do my best to answer them in this blog.

Receive these posts by e-mail?

Subscribe to my newsletter & receive a free eBook “Xanpan: Team Centric Agile Software Development”

The post How should we organize our teams? appeared first on Allan Kelly Associates.

Major players in evidence-based software engineering

Who are the major players in evidence-based software engineering?

How might ‘majorness’ of players be calculated? For me, the amount of interesting software engineering data they have made publicly available is the crucial factor. Any data published in a book, paper or report is enough to be considered interesting. How interesting is data published on a web page? This is a tough question, let’s dodge the question to start with, and consider the decades before the start of 2000.

In the academic world performance is based on number of papers published, the impact factor of where they were published and number of citations of those papers. This skews the results in favor of those with lots of students (who tack their advisor’s name on the end of papers published) and those who are good at marketing.

Historians of computing have primarily focused on the evolution of hardware and are slowly moving to discuss software (perhaps because microcomputers have wiped out nearly every hardware vendor). So we will have to wait perhaps a decade or two for tentative/definitive historian answer.

The 1950s

Computers and Automation is a criminally underused resource (a couple of PhDs worth of primary data here). A lot of the data is hardware related, but software gets a lot more than a passing mention.

The US military published lots of hardware data, but software does not get mentioned much.

The 1960s

Computers and Automation are still publishing.

The US military still publishing data; again mostly hardware related.

Datamation, a weekly news magazine, published a lot of substantial material on the software and hardware ecosystems as they evolved.

Kenneth Knight’s analysis of computer performance is an example of the kind of data analysis that many people undertook for hardware, which was rarely done for software.

The 1970s

The US military are still leading the way; we are in the time of Rome. Air Force officers studying for a Master’s degree publish more software engineering data than all academics combined over this and the next two decades.

“Data processing technology and economics” by Montgomery Phister is 720 A4 pages packed with graphs and tables of numbers. Despite citing earlier sources, this has become the primary source for a lot of subsequent researchers; this is understandable in a pre-internet age. Now we have Bitsavers and the Internet Archive, and the cited primary source can be downloaded.

NASA is surprisingly low volume.

The 1980s

Rome falls (i.e., the work gets outsourced to a university) and the false prophets (i.e., academics doing non-evidence based work) multiply and prosper. There are hushed references to trouble makers performing unclean acts experiments in the wilderness.

A few people working in the wilderness, meaning that the quantity of data being produced drops by at least an order of magnitude.

The 1990s

Enough time has passed for people to be able to refer to the wisdom of the ancients.

There are still people in the wilderness howling at the moon, and performing unclean acts experiments.

The 2000s

Repositories of Open source and bug reports grow and prosper. Evidence-based software engineering research starts to become mainstream.

There are now groups of people doing software engineering research.

What about individuals as major players? A vaguely scientific way of rating individual impact, on evidence-based software engineering, is to count the number of papers they have published, that are cited by a book claiming to discuss all the important/interesting publicly available software engineering data (code+data).

The 1,521 papers cited, by such a book, had 3,716 authors, of which 3,095 were different. The authors who appeared most often are listed below (count on the right, and yes, at number 2 is a theoretician; I have cited myself nine times, but two of those are to web sites hosting data).

Magne Jorgensen 17
Anne Chao 11
Dag I. K. Sjoberg 10
Massimiliano Di Penta 10
Ahmed E. Hassan 8
Christian Bird 8
Stanislas Dehaene 8
Giuliano Antoniol 7
Thomas Zimmermann 7
Alexander Serebrenik 6
Dror G. Feitelson 6
Gregorio Robles 6
Krzysztof Czarnecki 6
Lutz Prechelt 6
Victor R. Basili 6

The number of authors/papers follows the usual pattern of many people writing one paper.

Number of evidence-based papers written by an author

Who might I have missed? The business school researchers don’t get a mention because their data is often covered by a confidentiality agreement. The machine learning crowd are just embarrassing.

Suggestions for major players welcome.

Direct access to SonarQube Postgresql Database

I want to change to change the name of a sonarqube project. This cannot be done without performing another analysis. You can just do it in SQL https://stackoverflow.com/questions/30511849/how-to-rename-a-project-in-sonarqube-5-1 but you have to be able to login to the database. Postgresql is very secure. A quick fix is to edit /var/lib//pgsql/pg_hba.conf change local connections from ident to trust:

# TYPE DATABASE USER CIDR-ADDRESS METHOD

# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
# IPv6 local connections:
host all all ::1/128 trust

Now you can edit:

psql -U sonarqube -W sonar
and finally:

UPDATE projects
SET name = 'NEW_PROJECT_NAME',
long_name = 'NEW_PROJECT_NAME'
WHERE kee = 'PROJECT_KEY'

My experience upgrading to Elm 0.19

Elm is unstable, so upgrading to the next version can be painful. Here’s what I needed to do to upgrade from 0.18 to 0.19.

  • Replace elm-package.json and tests/elm-package.json with elm.json – e06f5a1728
  • Switch to the new elm-testb964b7c7a
  • Re-arrange Main, and how we call it from JavaScript – 0c118c49f
  • Stop using eeue56/elm-all-dict (since it’s not ported to 0.19 and porting it looked hard due to a lack of Debug.crash) – fe100f256
  • Replace toString with String.fromX or Debug.toString – 9e78163d0a3
  • Stop “shadowing” names by making new variables with the same name as another in the scope – 9688a621de
  • Adapt to the changed Html.style function – b991ab4f
  • Stop using Debug.crash – f98a70ad1
  • Adapt to the changes in the Regex module – 856762a4
  • Stop using tuples with more than 3 parts – 472c0bb7

The lack of Debug.crash is really, really painful, especially for a library like eeue56/elm-all-dict that has lots of invariants that are hard or impossible to enforce via the type system. On the other hand, if Elm can give a hard guarantee that there will be no runtime errors, this seems pretty cool. The problem is that some code may well have to return the wrong answer silently, instead of crashing, which could be much worse than crashing in some use-cases.

I was annoyed by the lack of more-than-3-part tuples, but even as I did the work to change my code, I saw it get better, so it’s hard to argue with.

The hardest part to work out was how to run the tests. Fortunately the tests themselves needed almost no changes. I just needed to do this:

rm -r tests/elm-stuff
rm tests/elm-package.json
sudo npm install -g elm-test@0.19.0-beta8
elm-test install
elm-test

My next job is to check out the –optimize compiler flag, and the advice on making the code smaller and faster.

Visual Lint 6.5.4.298 has been released

This is a recommended maintenance update for Visual Lint 6.5. The following changes are included:
  • If a "Lint" folder without the hidden attribute exists in a solution/workspace folder Visual Lint will no longer attempt to use it to store analysis results and will create a new ".visuallint" folder instead. This prevents Visual Lint from assuming that a user-created "Lint" folder is one which was created by an earlier (pre-v5.0) version of Visual Lint.
  • Fixed a crash which could occur when files were saved in the Eclipse plug-in. The crash seemed to particularly affect plug-in installations running within Texas Instruments Code Composer Studio and configured for per-project analysis with the "Re-analyse saved files using the preferred method" option set.
  • The project variables $(CEVER), $(ARCHFAM) and $(_ARCHFAM_) are now automatically defined when analysing Visual Studio 2008 projects for the NetDCU9 (ARMV4I) platform.
  • Corrected the "Supported development environments" help topic to reflect the fact that Atmel AVR Studio 5 and Atmel Studio 6.x/7.x are now supported via a dedicated plug-in.
  • Updated the PC-lint Plus message database to reflect changes in PC-lint Plus 1.2. Note that the definitions for Clang errors 5905, 5916 and 5922 have been omitted as the PC-lint Plus -dump_messages directive does not reveal either their titles or descriptions.
Download Visual Lint 6.5.4.298

Business school research in software engineering is some of the best

There is a group of software engineering researchers that don’t feature as often as I would like in my evidence-based software engineering book; academics working in business schools.

Business school academics have written some of the best papers I have read on software engineering; the catch is that the data they use is confidential. For somebody writing a book that only discusses a topic if there is data publicly available, this is a problem.

These business school researchers show that it is possible for academics to obtain ‘interesting’ software engineering data from industry. My experience with talking to researchers in computing departments is that most are too involved in their own algorithmic bubble to want to talk to anybody else.

One big difference between the data analysis papers written by academics in computing departments and business schools, is statistical sophistication. Computing papers are still using stone-age pre-computer age techniques, the business papers use a wide range of sophisticated techniques (sometimes cutting edge).

There is one aspect of software engineering papers written by business school researchers that grates with me, many of the authors obviously don’t understand software engineering from a developer’s perspective; well, obviously, they are business oriented people.

The person who has done the largest amount of interesting software engineering research, whose work I don’t (yet; I will find a way) discuss, is Chris Kemerer; a researcher who has a long list of empirical papers going back to the late 1980s, and rarely gets cited by papers by people in computing departments (I am the only person I know, who limits themself to papers where the data is publicly available).

#NoProjects: Project Myopia is published

ProjectMyopiaNew-2018-09-10-11-17.jpg

Project Myopia – the original case for #NoProjects – has been a long time in the works but it is now done. Published. For sale on Amazon.

Projects fail. Some say 40% of all IT projects fail, some say 70%. And it has been that way for years. Each project fails for its own reasons but they all share one thing in common: the Project Model. Could it be the project model itself which creates failure?

Projects end. Successful software continues. Twenty-first century digital businesses want to continue and grow.

Project Myopia is available to buy on Amazon today – the physical version should joined the eBook in a few days.

Project Myopia gives the case against projects – the hard core #NoProjects arguments. A second book, Continuous Digital will join Project Myopia in a few weeks on Amazon. Right now copyediting isn’t finished on Continuous Digital, plus the physical copy needs to be worked out. In the meantime late drafts of Continuous Digital are available on LeanPub.

The post #NoProjects: Project Myopia is published appeared first on Allan Kelly Associates.

#NoProjects: Project Myopia is published

ProjectMyopiaNew-2018-09-10-11-17-1.jpg

Project Myopia – the original case for #NoProjects – has been a long time in the works but it is now done. Published. For sale on Amazon.

Projects fail. Some say 40% of all IT projects fail, some say 70%. And it has been that way for years. Each project fails for its own reasons but they all share one thing in common: the Project Model. Could it be the project model itself which creates failure?

Projects end. Successful software continues. Twenty-first century digital businesses want to continue and grow.

Project Myopia is available to buy on Amazon today – the physical version should joined the eBook in a few days.

Project Myopia gives the case against projects – the hard core #NoProjects arguments. A second book, Continuous Digital will join Project Myopia in a few weeks on Amazon. Right now copyediting isn’t finished on Continuous Digital, plus the physical copy needs to be worked out. In the meantime late drafts of Continuous Digital are available on LeanPub.

The post #NoProjects: Project Myopia is published appeared first on Allan Kelly Associates.

Reidolized

So Blackie Lawless decided to re-record my favorite album of all time and my initial thoughts are why did he bother. I’ve avoided buying it for months until last Friday when I was trying out some new headphones with the re-recorded guitar solo from The Idol and quite enjoyed the new layers and mix of the new version of the song. Although I’m not a musician, it feels like I know every note and I could really hear the difference. It’s not the original guitarist, and consequently the solo wasn’t the same or as good.


Having now listened to the first disk of the re-record I can only say there is one improvement, the extra songs, at least one of which was taken from WASP’s latest studio album and is magnificent. The removing of the swearing in the songs and intros feels wrong and a cop out and the lead guitar work just isn’t up to Bruce Kulik’s incredible standard.

The second disk is starting now….

Cuboid Space Division – a.k.

Over the last few months we have been taking a look at algorithms for interpolating over a set of points (xi,yi) in order to approximate values of y between the nodes xi. We began with linear interpolation which connects the points with straight lines and is perhaps the simplest interpolation algorithm. Then we moved on to cubic spline interpolation which yields a smooth curve by specifying gradients at the nodes and fitting cubic polynomials between them that match both their values and their gradients. Next we saw how this could result in curves that change from increasing to decreasing, or vice versa, between the nodes and how we could fix this problem by adjusting those gradients.
I concluded by noting that, even with this improvement, the shape of a cubic spline interpolation is governed by choices that are not uniquely determined by the points themselves and that linear interpolation is consequently a more mathematically appropriate scheme, which is why I chose to generalise it to other arithmetic types for y, like complex numbers or matrices, but not to similarly generalise cubic spline interpolation.

The obvious next question is whether or not we can also generalise the nodes to other arithmetic types; in particular to vectors so that we can interpolate between nodes in more than one dimension.

Bulk adding items to Wunderlist using wunderline on Ubuntu MATE

If you use Wunderlist and want to be able to bulk-add tasks from a text file, first install and set up wunderline.

Now, to be able to right-click a text file containing one task per line on Ubuntu MATE, create a file called “wunderlist-bulk-add” in ~/.config/caja/scripts/ and make it executable. Paste the code below into that file.

(Note: it’s possible that this would work in GNOME if you replaced “caja” with “nautius” in that path – let me know in the comments.)

#!/usr/bin/env bash

set -e
set -u
set -o pipefail

function nonblank()
{
    grep -v -e '^[[:space:]]*$' "$@"
}

for F in "$@"; do
{
    COUNT=$(nonblank "$F" | wc -l | awk '{print $1}')
    if [ "$COUNT" = "" ]; then
    {
        zenity --info --no-wrap --text="File $F does not exist"
    }
    else
    {
        set +e
        zenity --question --no-wrap \
            --text="Add $COUNT items from $F to Wunderlist?"
        ANSWER=$?
        set -e

        if [ "$ANSWER" = "0" ]; then
        {
            set +e
            nonblank "$F" | \
                wunderline add --stdin | \
                zenity --progress --text "Adding $COUNT items to Wunderlist"
            SUCCEEDED=$?
            set -e
            if [ "$SUCCEEDED" = "0" ]; then
            {
                zenity --info --no-wrap --text "Added $COUNT items."
            }
            else
            {
                zenity --error --no-wrap \
                    --text "Failed to add some or all of these items."
            }; fi
        }; fi
    }; fi
}; done

Release or be damned

iStock-166161352small-2018-09-6-12-26.jpg

Back when I was still paid to code I had a simple question I posed to troubled development efforts:

“Why can’t we release tomorrow?”

This short simple question turns out to be amazingly powerful. I remember one effort I was involved with in California where a new CEO took over and started cutting jobs. I posed this question to the team and in a week or two we did a “beta release” – we did those sort of things back then. Asking this question was the key that allows us to question everything, to cut the feature list – or rather push work back, it stayed on the to-do list but we didn’t let it stop us from pushing to release.

We rethought what we were trying to achieve: we didn’t need the whole product, we just needed enough of the product to work to show to one specific target customer. Even if they signed there and then we had weeks before they used it in anger. But until we released something, until we had something “done” our team, our product, look like just another “maybe.” We had to draw a line under it so the new CEO wouldn’t draw a line under us.

Saying “only do the essential” is easy and come up again and again, whether it is Minimal Viable Product, Minimal Subset, Must haves in Moscow rules, but it is far easier said than done. One persons “essential” is so often another persons “optional extra.” In this context, when I say “essential” I mean “the parts needed to make the system work end to end” – I’m far closer to the old walking skeleton idea.

I was reminded of this question by a couple of endeavours that came to my attention during the summer. Well, I say came to my attention, I feel a bit responsible. Both endeavours are happening at clients; clients who I had fallen out of touch with. My style of working is to help clients who want help, I don’t like selling myself. These clients didn’t ask for more help so I didn’t jam my foot in the door, in retrospect maybe I should have.

In one case the team were doing very well. They were iterating, they were TDD/BDD’ing, they were demoing, they were working with the client, they were doing everything … except releasing. Then one day the client asked “when will it be done?”

Now think for a moment: What if you could release your product tomorrow?

The thing is, without actual products those around the team look for signs that the team can be trusted, that they team will deliver, that the team are thinking about what is to be done. People ask for proxy-products: plans, schedules, risk-logs, budget forecasts and so on. When stakeholders can’t see progress they look for things to assure them that there is (or will be) progress (soon).

Who needs plans and predictions about the future when the future is here tomorrow?

Actual releases are they key to reaching the new world, they change everything.

So I feel guilty: I should have inflicted myself on these teams, I should have been there again and again bugging them “Go to release”, “Remove that barrier”, “Force it through”.

Being able to ship an update of your product has a transformative effect.

It demonstrates the team have the ability to do the job in hand.
It demonstrates you have quality. It obliterates the need for a test-fix-test-fix aka stabilisation aka hardening phase.
It blows away sunk costs because something has been delivered.
It removes “maybe” and “ready but…”
It is probably the greatest risk mitigation strategy possible.
It creates trust and provides a platform for solid conversations.

Most of all, a released product is a far better statement of progress than any number of plans or forecasts.

This does not mean everything is done. Sure there are things left undone but there will be things left undone when I’m on my deathbed, that is the nature of life. As much as we (especially men) love to collect entire sets there are few prizes in life for completing everything on your bucket list.

Having a released product utterly changes the nature of the conversation. Conversations are no longer full of “ifs” “maybes” “shoulds” “how long will it take?” “what are the quick wins?”. Those questions can go away. In its place you can have serious conversations about prioritisation and “what do you want tomorrow?”

This is all part of the reason I love continuous delivery. Teams can focus on real priorities and stop wasting time on conjecture.

In my book if you don’t have a releasable product at least every two weeks – say every second Thursday – you are not Agile. And if you haven’t released a product to live in the last two weeks you are probably not Agile.

I don’t care how close you get to a releasable product: it isn’t a release if it isn’t released to a live environment – close but no cigar as they say. (OK, I’ll accept the live environment may not be publicly know, or might be called a beta, but it has to be the real thing.)

Nor should you rest on your laurels once you have regular releases (to live) every second week. That is but first base. You have opened the door, now go further. There are at least 13 opportunities to improve.

If you cannot do that now then ask yourself: Why can’t we release tomorrow?

And start working to remove those obstacles:

  • Reduce the number of work items you are aiming to put in the release.
  • Fix show-stopper defects now.
  • Running tests now.
  • Get those people who need to sign-off to sign-off.

Software development has diseconomies of scale: many small is cheaper than few large.

And once you have your release you can turn your attention to making sure these things don’t happen again:

  • Reduce the amount of work you accept into development at one time.
  • Fix every defects as soon as they are found.
  • Automate tests so they can run more often. (Automate anything that moves, and if it doesn’t move, automate it in case.)
  • Find a way to reduce the time it takes to get sign-offs: remove the sign-off, make sure the signer prioritises signing or delegate someone else to sign (or automate the signature.)

If there are essential processes, activities, third-parties (or anything else) that has limited bandwidth which need to be done before release but inject delay then re-orientate your process around that bottleneck. For example, if your code needs to pass a security audit before release (an audit you can’t automate that is) then, downsize all the other activities so that the audit process is 100% utilised. (OK, 100% is wrong, 76% might be better, but thats a long conversation about queuing theory.)

Again and again I seem condemned to learn the lesson: nothing counts but working software which is used.

As for my team, and my job in California, it didn’t save me. I regret not asking the question sooner.

The post Release or be damned appeared first on Allan Kelly Associates.

Writing a new Flarum extension on Ubuntu

In a previous post I described how to install Flarum locally on Ubuntu.

Here is how I set up my development environment on top of that setup so I was able to write a new Flarum extension and test it on my local machine.

Recap: I installed Apache and PHP and enabled mod_rewrite, installed MariaDB and made a database, installed Composer, and used Composer to install Flarum at /var/www/html/flarum.

More info: Flarum Contribution Guide, extension development guide, extension development quick start, workflow discussion.

I decided to call my extension “rabbitescape/flarum-ext-rabbitescape-leveleditor”, so that is the name you will see below.

Here’s what I did:

cd /var/www/html/flarum

Edit /var/www/html/flarum/composer.json (see the workbench explanation for examples of the full file).

At the end of the “require” section I added a line like this:

“rabbitescape/flarum-ext-rabbitescape-leveleditor”: “*@dev”

(Remembering to add a comma at the end of the previous line so it remained valid JSON.)

After the “require” section, I added this:

    "repositories": [
        {
            "type": "path",
            "url": "workbench/*/"
        }
    ],

Next, I made the place where my extension would live:

mkdir workbench
cd workbench
mkdir flarum-ext-rabbitescape-leveleditor
cd flarum-ext-rabbitescape-leveleditor

Now I created a file inside flarum-ext-rabbitescape-leveleditor called composer.json like this:

{
    "name": "rabbitescape/flarum-ext-rabbitescape-leveleditor",
    "description": "Allow viewing and editing Rabbit Escape levels in posts",
    "type": "flarum-extension",
    "keywords": ["rabbitescape"],
    "license": "MIT",
    "authors": [
        {
            "name": "Andy Balaam",
            "email": "andybalaam@artificialworlds.net"
        }
    ],
    "support": {
        "issues": "https://github.com/andybalaam/flarum-ext-rabbitescape-leveleditor/issues",
        "source": "https://github.com/andybalaam/flarum-ext-rabbitescape-leveleditor"
    },
    "require": {
        "flarum/core": "^0.1.0-beta.5"
    },
    "extra": {
        "flarum-extension": {
            "title": "Rabbit Escape Level Editor"
        }
    }
}

In the same directory I created a file bootstrap.php like this:

<?php

return function () {
    echo 'Hello, world!';
};

Then I told Composer to refresh the project like this:

cd ../..   # Back up to the main directory
composer update

Among the output I saw a line like this which convinced me it had worked:

  - Installing rabbitescape/flarum-ext-rabbitescape-leveleditor (dev-master): Symlinking from workbench/flarum-ext-rabbitescape-leveleditor

Now I went to http://localhost/flarum, signed in as admin, adminpassword (as I set up in the previous post), clicked my username in the top right and chose Administration, then clicked Extensions on the left.

I saw my extension in the list, and turned it on by clicking the check box below it. This made the whole web site disappear, and instead just the text “Hello world!” appeared. This showed my extension was being loaded.

Finally, I edited bootstrap.php and commented out the line starting with “echo”. I refreshed the page in my browser and saw the Flarum site reappear.

Now, my extension is installed and ready to be developed! See flarum.org/docs/extend/start for how to get started making it do cool stuff.

Moving to the 12th circle in fault prediction modeling

Most software fault prediction papers are based on a false assumption, i.e., a list of dates when a fault was first experienced, by a program, contains enough information to build a model that has a connection to reality. A count of faults that have been experienced twice is required to fit a basic model that has some mathematical connection to reality.

I had thought that people had moved on from writing papers that fitted yet more complicated equations to one of the regularly used data sets. No, it seems they have just switched to publishing someplace they have not been seen before.

Table 1 lists the every increasing number of circles within circles; the new model is proposed as the 12th refinement (the table is a summary, lots of forks have been proposed over the years). I have this sinking feeling there is another paper in the works, one that ‘benchmarks’ the new equation using a collection of the other regular characters data sets that appear in papers of this kind.

Fitting an equation to data of first experience of a fault is little better than fitting noise.

As Planck famously said, science advances one funeral at a time.

Ubuntu “compose” key for easy unicode character input

I found out on Mastodon recently that the Compose key exists, allowing you to enter special characters using easy-to-remember key sequences (e.g. “<compose>/=” gives you “≠”).

To do this on Ubuntu (and probably many other systems), hold SHIFT, then press AltGr, release them both, and type something like “/=”. You should see the ≠ symbol appear.

In Ubuntu MATE, you can customise what key is the compose key by opening “Keyboard”, clicking “Layouts”, then “Options…” and checking a box under “Position of compose key”. I checked “Right Alt”, meaning my “AltGr” key is now the compose key.

More info: Ubuntu ComposeKey docs.

Want to explore? See the list of the default sequences you can type.

Redirecting all requests to https and www using .htaccess in Apache

I want all requests to artificialworlds.net/rabbit-escape/levels/ to get redirected to use the https protocol, and to include “www.” at the beginning of the URL, and I found lots of Stack Overflow articles, but nothing there worked perfectly for me. Here is how I managed it.

I edited the .htaccess file in the directory where I want this to apply, and added this at the top:

<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteCond %{HTTP_HOST} ^artificialworlds\.net$ [NC]
  RewriteRule ^ https://www.artificialworlds.net%{REQUEST_URI} [R=301,L]

  RewriteCond %{HTTPS} off
  RewriteRule ^ https://www.artificialworlds.net%{REQUEST_URI} [R=301,L]
</IfModule>

Surgical Support Needs Surgical Tools

In the world of IT support there is the universal solution to every problem – turn it off and on again. You would hope that this kind of drastic action is only drawn upon when all other options have been explored or that the problem is known a priori to require such a response and is the least disruptive course of action. Sadly what was often just a joke in the past has become everyday life for many systems as rebooting or restarting is woven into the daily support routine.

In my professional career I have worked on a number of systems where there are daily scheduled jobs to reboot machines or restart services. I’m not talking about the modern, proactive kind like the Chaos Monkey which is used to probe for weaknesses, or when you force cluster failovers to check everything’s healthy; I’m talking about jobs where the restart is required to ensure the correct functioning of the system – disabling them would cripple it.

Sledgehammers

The need for the restart is usually to overcome some kind of corrupt or immutable internal state, or to compensate for a resource leak, such as memory, which degrades the service to an unacceptable level. An example of the former I’ve seen is to flush a poisoned cache or pick up the change in date, while for the latter it might be unclosed database connections or file handles. The notion of “recycling” processes to overcome residual effects has become so prominent that services like IIS provide explicit support for it [1].

Depending on where the service sits in the stack the restart could be somewhat disruptive, if it’s located on the edge, or largely benign if it’s purely internal, say, a background message queue listener. For example I once worked on a compute farm where one of the front-end services was restarted every night and that caused all clients to drop their connection resulting in a support email being sent due to the “unhandled exception”. Needless to say everyone just ignored all the emails as they only added to the background noise making genuine failures harder to spot.

These kind of draconian measures to try and attain some system stability actually make matters worse as the restarts then begin to hide genuine stability issues which eventually start happening during business hours as well and therefore cause grief for customers as unplanned downtime starts occurring. The impetus for one of my first ACCU articles “Utilising More Than 4GB of Memory in a 32-bit Windows Process” came from just such an issue where a service suddenly starting failing with out-of-memory errors even after a restart if the load was awkwardly skewed. It took almost four weeks to diagnose and properly fix the issue during which there were no acceptable workarounds – just constant manual intervention from the support team.

I also lost quite a few hours on the system I mentioned earlier debugging a problem in the caching mechanism which was masked by a restart and only surfaced because the restart failed to occur. No one had remembered about this failure mode because everyone was so used to the restart hiding it. Having additional complexity in the code for a feature that will never be used in practice is essentially waste.

Cracking Nuts

Although it’s not true in all cases (the memory problem just described being a good example) the restart option may be avoidable if the process exposed additional behaviours that allowed for a more surgical approach to recovery to take place. Do you really need to restart the entire process just to flush some internal cache, or maybe just a small part of it? Similarly if you need to bump the business date via an external stimulus can that not be done through a “discoverable” API instead of hidden as part of a service restart [2]?

In some of my previous posts and articles, e.g “From Test Harness To Support Tool”, “Home-Grown Tools” and more recently in “Libraries, Console Apps, and GUIs”, I have described how useful I have found writing simple custom tools to be for development and deployment but also, more importantly, for support. What I think was missing from those posts that I have tried to capture in this one, most notably through its title, is to focus on resolving system problems with the minimal amount of disruption. Assuming that you cannot actually fix the underlying design issue without a serious amount of time and effort can you at least alleviate the problem in the shorter term by adding simple endpoints and tools that can be used to make surgical-like changes inside the critical parts of the system?

For example, imagine that you’re working on a grid computing system where tasks are dished out to different processes and the results are aggregated. Ideally you would assume that failures are going to happen and that some kind of timeout and retry mechanism would be in place so that when a process dies the system recovers automatically [3]. However, if for some reason that automatic mechanism does not exist how are you going to recover? Given that someone (or something) somewhere is waiting for their results how are you going to “unblock the system” and allow them to make progress, without also disturbing all your other users who are unaffected?

You can either try and re-submit the failed task and allow the entire job to complete or kill the entire job and get the client to re-submit its job. As we’ve seen one way to achieve the latter would be to restart parts of the system thereby flushing the job from it. When this is a once in a million event it might make sense [4] but once the failures start racking up throwing away every in-flight request just to fix the odd broken one becomes more and more disruptive. Instead you need a way to identify the failed task (perhaps using the logs) and then instruct the system to recover such as by killing just that job or by asking it to resubmit it to another node.

Hence, ideally you’d just like to hit one admin endpoint and say something like this:

> admin-cli kill-job --server prod --job-id 12345

If that’s not easily achievable and there is distributed state to clear up you might need a few separate little tools instead that can target each part of system, for example:

> admin-cli remove-node –s broker-prod --node NODE99
> admin-cli remove-results -s results-prod --job 12345
> admin-cli remove-tasks –s taskq-prod --job 12345
> admin-cli reset-client –s ui-prod --client CLT42
> admin-cli add-node –s broker-prod --node NODE99

This probably seems like a lot of work to write all these little tools but what I’ve found in practice is that usually most of the tricky logic in the services already exists – you just need to find a way to invoke it externally with the correct arguments.

These days it’s far easier to graft a simple administration endpoint onto an existing service. There are plenty of HTTP libraries available that will allow you to expose a very basic API which you could even poke with CURL. If you’re already using something more meaty like WCF then adding another interface should be trivial too.

Modern systems are becoming more and more decoupled through the use of message queues which also adds a natural extension point as they typically already do all the heavy lifting and you just need to add new message handlers for the extra behaviours. One of the earliest distributed systems I worked on used pub/sub on a system-wide message bus both for functional and administrative use. Instead of using lots of different tools we had a single admin command line tool that the run playbook generally used (even for some classic sysadmin stuff like service restarts) as it made the whole support experience simpler.

Once you have these basic tools it then becomes easy to compose and automate them. For example on a similar system I started by adding a simple support stored procedure to find failed tasks in a batch processing system. This was soon augmented by another one to resubmit a failed task, which was then automated by a simple script. Eventually it got “productionised” and became a formal part of the system providing the “slow retry” path [3] for automatic recovery from less transient outages.

Design for Supportability

One of the design concepts I’ve personally found really helpful is Design for Testability; something which came out of the hardware world and pushes us to design our systems in a way that makes them much easier test. A knock-on effect of this is that you can reduce your end-to-end testing burden and deliver quicker.

A by-product of Design for Testability is that it causes you to design your system in a way that allows internal behaviours to be observed and controlled in isolation. These same behaviours are often the same ones that supporting a system in a more fine-grained manner will also require. Hence by thinking about how you test your system components you are almost certainly thinking about how they would need to be supported too.

Ultimately of course those same thoughts should also be making you think about how the system will likely fail and therefore what needs to be put in place beforehand to help it recover automatically. In the unfortunate event that you can’t recover automatically in the short term you should still have some tools handy that should facilitate a swift and far less disruptive manual recovery.

 

[1] Note that this is different from a process restarting itself because it has detected that it might have become unstable, perhaps through the use of the Circuit Breaker pattern.

[2] Aside from the benefits of comprehension this makes the system far more testable as it means you can control the date and therefore write deterministic tests.

[3] See “When Does a Transient Failure Stop Being Transient” for a tangent about fast and slow retries.

[4] Designing any distributed system that does not tolerate network failures is asking for trouble in my book but the enterprise has a habit of believing the network is “reliable enough”.

Grammar checking in 2018

I am a big fan of using tools to find problems quickly. Polishing the draft material for my evidence-based software engineering book, I have been finding an annoying number of grammatical mistakes :-(.

LanguageTool is what I use to check my grammar; it is the best tool of its kind that I know, and supports lots of different languages.

I also have an awk script that looks for new instances of previous mistakes I have made. It rarely flags anything, I seem to be in a continual state of making new grammatical mistakes.

Stung by a recent series is blatant mistakes, I have been searching for a better tool. What did I find:

So, lots of interesting stuff, but nothing better that is usable.

I keep looking at the interesting things that spaCY can do (if you are looking to integrate language processing in your app, spaCY is currently the best language processing library). Does anybody know of grammar checking work being done using spaCY (LanguageTool is based around a parsing engine that is rather long in the tooth now)?

Anybody interested in organizing a grammar checking tool hack day in London?

Experimental Psychology by Robert S. Woodworth

I have just discovered “Experimental Psychology” by Robert S. Woodworth; first published in 1938, I have a reprinted in Great Britain copy from 1951. The Internet Archive has a copy of the 1954 revised edition; it’s a very useful pdf, but it does not have the atmospheric musty smell of an old book.

The Archives of Psychology was edited by Woodworth and contain reports of what look like ground breaking studies done in the 1930s.

The book is surprisingly modern, in that the topics covered are all of active interest today, in fields related to cognitive psychology. There are lots of experimental results (which always biases me towards really liking a book) and the coverage is extensive.

The history of cognitive psychology, as I understood it until this week, was early researchers asking questions, doing introspection and sometimes running experiments in the late 1800s and early 1900s (e.g., Wundt and Ebbinghaus), behaviorism dominants the field, behaviorism is eviscerated by Chomsky in the 1960s and cognitive psychology as we know it today takes off.

Now I know that lots of interesting and relevant experiments were being done in the 1920s and 1930s.

What is missing from this book? The most obvious omission is equations; lots of data points plotted on graph paper, but no attempt to fit an equation to anything, e.g., an exponential curve to the rate of learning.

A more subtle omission is the world view; digital computers had not been invented yet and Shannon’s information theory was almost 20 years in the future. Researchers tend to be heavily influenced by the tools they use and the zeitgeist. Computers as calculators and information processors could not be used as the basis for models of the human mind; they had not been invented yet.

Example of a systemd service file

Here is an almost-minimal example of a systemd service file, that I use to run the Mastodon bot of my generative art playground Graft.

I made a dedicated user just to run this service, and installed Graft into /home/graft/apps/graft under that username. Now, as root, I edited a file called /etc/systemd/service/graft.service and made it look like this:

[Service]
ExecStart=/home/graft/apps/graft/bot-mastodon
User=graft
Group=graft
WorkingDirectory=/home/graft/apps/graft/
[Install]
WantedBy=multi-user.target

Now I can start the graft service like any other service like this:

sudo systemctl start graft

and find out its status with:

sudo systemctl status graft

If I want it to run on startup I can do:

sudo systemctl enable graft

and it will. Easy!

If I want to look at its output, it’s:

sudo journalctl -u graft

As a reward for reading this far, here’s a little animation you can make with Graft:

Spaceship Operator

You write a class. It has a bunch of member data. At some point, you realise that you need to be able to compare objects of this type. You sigh and resign yourself to writing six operator overloads for every type of comparison you need to make. Afterwards your fingers ache and your previously clean code is lost in a sea of functions which do essentially the same thing. If this sounds familiar, then C++20’s spaceship operator is for you. This post will look at how the spaceship operator allows you to describe the strength of relations, write your own overloads, have them be automatically generated, and how correct, efficient two-way comparisons are automatically rewritten to use them.

Relation strength

The spaceship operator looks like <=> and its official C++ name is the “three-way comparison operator”. It is so-called, because it is used by comparing two objects, then comparing that result to 0, like so:

(a <=> b) < 0  //true if a < b
(a <=> b) > 0  //true if a > b
(a <=> b) == 0 //true if a is equal/equivalent to b

One might think that <=> will simply return -1, 0, or 1, similar to strcmp. This being C++, the reality is a fair bit more complex, but it’s also substantially more powerful.

Not all equality relations were created equal. C++’s spaceship operator doesn’t just let you express orderings and equality between objects, but also the characteristics of those relations. Let’s look at two examples.

Example 1: Ordering rectangles

We have a rectangle class and want to define comparison operators for it based on its size. But what does it mean for one rectangle to be smaller than another? Clearly if one rectangle is 1cm by 1cm, it is smaller than one which is 10cm by 10cm. But what about one rectangle which is 1cm by 5cm and another which is 5cm by 1cm? These are clearly not equal, but they’re also not less than or greater than each other. But speaking practically, having all of <, <=, ==, >, and >= return false for this case is not particularly useful, and it breaks some common assumptions at these operators, such as (a == b || a < b || a > b) == true. Instead, we can say that == in this case models equivalence rather than true equality. This is known as a weak ordering.

Example 2: Ordering squares

Similar to above, we have a square type which we want to define comparison operators for with respect to size. In this case, we don’t have the issue of two objects being equivalent, but not equal: if two squares have the same area, then they are equal. This means that == models equality rather than equivalence. This is known as a strong ordering.

Describing relations

Three-way comparisons allow you express the strength of your relation, and whether it allows just equality or also ordering. This is achieved through the return type of operator<=>. Five types are provided, and stronger relations can implicitly convert to weaker ones, like so:

relation conversion

Strong and weak orderings are as described in the above examples. Strong and weak equality means that only == and != are valid. Partial ordering is weaker than weak_ordering in that it also allows unordered values, like NaN for floats.

These types have a number of uses. Firstly they indicate to users of the types what kind of relation is modelled by the comparisons, so that their behaviour is more clear. Secondly, algorithms could potentially have more efficient specializations for when the orderings are stronger, e.g. if two strongly-ordered objects are equal, calling a function on one of them will give the same result as the other, so it would only need to be carried out once. Finally, they can be used in defining language features, such as class type non-type template parameters, which require the type to have strong equality so that equal values always name the same template instantiation.

Writing your own three-way comparison

You can provide a three-way comparison operator for your type in the usual way, by writing an operator overload:

struct foo {
  int i;

  std::strong_ordering operator<=> (foo const& rhs) {
    return i <=> rhs.i;
  }
};

Note that whereas two-way comparisons should be non-member functions so that implicit conversions are done on both sides of the operator, this is not necessary for operator<=>; we can make it a member and it’ll do the right thing.

Sometimes you may find that you want to do <=> on types which don’t have an operator<=> defined. The compiler won’t try to work out a definition for <=> based on the other operators, but you can use std::compare_3way instead, which will fall back on two-way comparisons if there is no three-way version available. For example, we could write a three-way comparison operator for a pair type like so:

template<class T, class U>
struct pair {
  T t;
  U u;

  auto operator<=> (pair const& rhs) const
    -> std::common_comparison_category_t<
         decltype(std::compare3_way(t, rhs.t)),
         decltype(std::compare3_way(u, rhs.u)> {
    if (auto cmp = std::compare3_way(t, rhs.t); cmp != 0) return cmp;
    return std::compare3_way(u, rhs.u);
  }

std::common_comparison_category_t there determines the weakest relation given a number of them. E.g. std::common_comparison_category_t<std::strong_ordering, std::partial_ordering> is std::partial_ordering.

If you found the previous example a bit too complex, you might be glad to know that C++20 will support automatic generation of comparison operators. All we need to do is =default our operator<=>:

auto operator<=>(x const&) = default;

Simple! This will carry out a lexicographic comparison for each base, then member of the type, in order of declaration.

Automatically-rewritten two-way comparisons

Although <=> is very powerful, most of the time we just want to know if some object is less than another, or equal to another. To facilitate this, two-way comparisons like < can be rewritten by the compiler to use <=> if a better match is not found.

The basic idea is that for some operator @, an expression a @ b can be rewritten as a <=> b @ 0. It can even be rewritten as 0 @ b <=> a if that is a better match, which means we get symmetry for free.

In some cases, this can actually provide a performance benefit. Quite often comparison operators are implemented by writing == and <, then writing the other operators in terms of those rather than duplicating the code. This can lead to situations where we want to check <= and end up doing an expensive comparison twice. This automatic rewriting can avoid that cost, since it will only call the one operator<=> rather than both operator< and operator==.

Conclusion

The spaceship operator is a very welcome addition to C++. It gives us more expressiveness in how we define our relations, lets us write less code to define them (sometimes even just a defaulted declaration), and avoids some of the performance pitfalls of manually implementing some comparison operators in terms of others.

For more details, have a look at the cppreference articles for comparison operators and the compare header, and Herb Sutter’s standards proposal. For a good example on how to write a spaceship operator, see Barry Revzin’s post on implementing it for std::optional. For more information on the mathematics of ordering with a C++ slant, see Jonathan Müller’s blog post series on the mathematics behind comparison.

Installing Flarum on Ubuntu 18.04

I am setting up a forum for sharing levels for my game Rabbit Escape, and I have decided to try and use Flarum, because it looks really usable and responsive, has features we need like liking posts and following authors, and I think it will be reasonably OK to write the custom features we want.

So, I want a dev environment on my local Ubuntu 18.04 machine, and the first step to that is a standard install.

Warning: at the time of writing the Flarum docs say it does not work with PHP 7.2, which is what is included with Ubuntu 18.04, so this may not work. (So far it looks OK for me.)

Here’s how I got it working (as far as the web installer stage, anyway):

sudo apt install \
    apache2 \
    libapache2-mod-php \
    mariadb-server \
    php-mysql \
    php-json \
    php-gd \
    php-tokenizer \
    php-mbstring \
    php-curl

php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"

# Get the neat line from https://getcomposer.org/download/
# Don't copy it exactly!
php -r "if (hash_file('SHA384', 'composer-setup.php') === '544e09ee996cdf60ece3804abc52599c22b1f40f4323403c44d44fdfdd586475ca9813a858088ffbc1f233e9b180f061') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"

mkdir $HOME/bin
php composer-setup.php --install-dir=$HOME/bin/ --filename=composer
rm composer-setup.php

cd /var/www/html
sudo mkdir flarum
sudo chown $(whoami) flarum

# Log out and in again here to get composer to be in your PATH
cd flarum
composer create-project flarum/flarum . --stability=beta

sudo chgrp -R www-data .
sudo chmod -R 775 .

sudo systemctl restart apache2

Go to http://localhost/flarum in your browser, and follow the instructions there to get set up.

If I get further, I will update this post, including on how to set up the MySQL database.

If you want to find and share levels for Rabbit Escape, check up on our progress setting up the forum at https://artificialworlds.net/rabbit-escape/levels.

nor(DEV):mag Youth in Tech out NOW!

Welcome to our Youth in Tech issue – just in time for the new term! It was no coincidence that our August issue is all about younger developers, but we’ve also included articles about how it is important to never stop learning, even when we might think we know everything!

The world of tech is always evolving, sometimes it feels difficult to keep up with new developments and technologies. It’s important to spend some time investing in ourselves and staff to make sure we keep abreast of this ever-changing sector. We hear from Luminous PR and netmatters about the importance of learning in their articles.

Of course it wouldn’t really be a Youth Issue without the view of an actual young developer, and we knew just the chap. Teenager James kindly gave us his opinion on the best bits of being into technology and where he thinks the future of development will be and student developers Emily and David were also kind enough to give us a developers view of their university project ‘Theia’, which aims to use machine learning to help students.

On the lighter side of things developer Adam sent us an amusing code piece (just try not to spit your tea while reading!) and we were also lucky enough to have an insider view from the recent Apple conference in San Jose, California, so if you’ve ever wondered what actually happens inside Apple, see inside!

Download now: https://www.norfolkdevelopers.com/wp-content/uploads/2018/08/norDEVmagazine-201808-e05.pdf

Impact of team size on planning, when sitting around a table

A recent blog post by Allan Kelly caught my attention; on Monday Allan sent me some comments on the draft of my book and I got to ask for a copy of his data (you don’t need your own software engineering data before sending me comments).

During an Agile training course he gives, Allan runs an exercise based on an extended version of the XP game. The basic points are: people form into teams, a task is announced, teams have to estimate how long it will take them to complete the task and then to plan the task implementation. Allan recorded information on team size, time spent estimating and time spent planning (no information on the tasks, which were straightforward, e.g., fold a paper airplane).

In a recent post I gave a brief analysis of team size on productivity. What does this XP game data have to say about the impact of team size on performance?

We don’t have task information, but we do have two timing measurements for each team. With a bit of suck-it-and-see analysis, I found that the following equation explained 50% of the variance (code+data):

Planning=-4+0.8*TimeSize+5*sqrt{Estimate}

There was some flexibility in the numbers, depending on the method used to build the regression model.

The introduction of each new team member incurs a fixed overhead. Given that everybody is sitting together around a table, this is not surprising; or, perhaps the problem was so simply that nobody felt the need to give a personal response to everything said by everybody else; or, perhaps the exercise was run just before lunch and people were hungry.

I am not aware of any connection between time spent estimating and time spent planning, but then I know almost nothing about this kind of XP game exercise. That square-root looks interesting (an exponent of 0.4 or 0.6 was a slightly less good fit). Thoughts and experiences anybody?

The Rich Get Richer – baron m.

Sir R-----! I must say that it is a relief to have the company of a fellow nobleman in these distressing times. That I have had to sell not one, but two of my several hundred antiquities to settle the burden of tax that this oppressive democracy has put upon me, simply to enrich slugabeds I might add, is quite intolerable!

Come, let us drown our sorrows whilst we still have the means to do so and engage in a little sport to raise our spirits.

I have a fancy for a game that I used to play when I was the Russian ambassador to the Rose Tree Valley commune. Founded by the philosopher queen Zway Remington as a haven for downtrodden wealthy industrialists, it was the purest of pure meritocracies; no handouts to the idle labouring classes there!

Migrating videos from YouTube to PeerTube inside a Docker container

I have quite a few videos hosted on YouTube that I would like to upload to my new PeerTube location, but I don’t want to install all the PeerTube dependencies on my machine, so I did it all inside a Docker image.

First I built and started a Docker container:

$ git clone https://github.com/chocobozzz/PeerTube /tmp/peertube
$ cd /tmp/peertube
$ docker build . -f ./support/docker/production/Dockerfile.stretch --tag peertube
$ docker run --tty --interactive peertube bash

Then I ran these commands inside it:

# yarn install --production=false
# node dist/server/tools/import-videos.js -u "https://peertube.mastodon.host" -U "andybalaam" -t "https://www.youtube.com/watch?v=TG0qRDrUPpA"

Of course, it would be better to write this up into its own Dockerfile to make this a one-liner.

References: PeerTube Docker setup, PeerTube video import.

2018 in the programming language standards’ world

I am sitting in the room, at the British Standards Institution, where today’s meeting of IST/5, the committee responsible for programming languages, has just adjourned (it’s close to where I have to be in a few hours).

BSI have downsized us, they no longer provide a committee secretary to take minutes and provide a point of contact. Somebody from a service pool responds (or not) to emails. I did not blink first to our chair’s request for somebody to take the minutes :-)

What interesting things came up?

It transpires that reports of the death of Cobol standards work may be premature. There are a few people working on ‘new’ features, e.g., support for JSON. This work is happening at the ISO level, rather than the national level in the US (where the real work on the Cobol standard used to be done, before being handed on to the ISO). Is this just a couple of people pushing a few pet ideas or will it turn into something more substantial? We will have to wait and see.

The Unicode consortium (a vendor consortium) are continuing to propose new pile of poo emoji and WG20 (an ISO committee) were doing what they can to stay sane.

Work on the Prolog standard, now seems to be concentrated in Austria. Prolog was the language to be associated with, if you were on the 1980s AI bandwagon (and the Japanese were going to take over the world unless we did something about it, e.g., spend money); this time around, it’s machine learning. With one dominant open source implementation and one commercial vendor (cannot think of any others), standards work is a relic of past glories.

In pre-internet times there was an incentive to kill off committees that were past their sell-by date; it cost money to send out mailings and document storage occupied shelf space. In an electronic world there is no incentive to spend time killing off such committees, might as well wait until those involved retire or die.

WG23 (programming language vulnerabilities) reported lots of interest in their work from people involved in the C++ standard, and for some reason the C++ committee people in the room started glancing at me. I was a good boy, and did not mention bored consultants.

It looks like ISO/IEC 23360-1:2006, the ISO version of the Linux Base Standard is going to be updated to reflect LBS 5.0; something that was not certain few years ago.

Visual Lint 6.5.3.296 has been released

This is a recommended maintenance update for Visual Lint 6.5. The following changes are included:
  • Added the ability to override the analysis tool installation folder on a per solution/workspace or project basis [Visual Lint Enterprise and Build Server Editions].
  • The analysis tool executable pathname can now be overridden on a per project basis as well as per solution/workspace [Visual Lint Enterprise and Build Server Editions].
  • Fixed a minor resizing bug in the Analysis Configuration Dialog.
  • Fixed a minor bug in the "Active Analysis Tool" dialog.
Download Visual Lint 6.5.3.296

How to write a programming language articles

Recent Overload journal issues contain my new articles on How to Write a Programming Language.

Part 1: How to Write a Programming Language: Part 1, The Lexer

Part 2: How to Write a Programming Language: Part 2, The Parser

PDF of the latest issue: Overload 146 containing part 2.

This is all creative-commons licensed and developed in public at github.com/andybalaam/articles-how-to-write-a-programming-language

Maximum team size before progress begins to stall

On multi-person projects people have to talk to each other, which reduces the amount of time available for directly working on writing software. How many people can be added to a project before the extra communications overhead is such that the total amount of code, per unit time, produced by the team decreases?

A rarely cited paper by Robert Tausworthe provides a simple, but effective analysis.

Activities are split between communicating and producing code.

If we assume the communications overhead is give by: t_0(S^{alpha}-1), where t_0 is the percentage of one person’s time spent communicating in a two-person team, S the number of developers and alpha a constant greater than zero (I’m using Tausworthe’s notation).

The maximum team size, before adding people reduces total output, is given by: S=({1+t_0}/{(1+alpha)t_0})^{1/{alpha}}.

If alpha=1 (i.e., everybody on the project has the same communications overhead), then S={1+t_0}/{2t_0}, which for small t_0 is approximately S=1/{2t_0}. For example, if everybody on a team spends 10% of their time communicating with every other team member: S={1+0.1}/{2*0.1}approx 5.

In this team of five, 50% of each persons time will be spent communicating.

If alpha=0.8, then we have S=({1+0.1}/{(1+0.8)*0.1})^{1/0.8}approx 10.

What if the percentage of time a person spends communicating with other team members has an exponential distribution? That is, they spend most of their time communicating with a few people and very little with the rest; the (normalised) communications overhead is: 1-e^{-(S-1)t_1}, where t_1 is a constant found by fitting data from the two-person team (before any more people are added to the team).

The maximum team size is now given by: S=1/{t_1}, and if t_1=0.1, then: S=1/{0.1}=10.

In this team of ten, 63% of each persons time will be spent communicating (team size can be bigger, but each member will spend more time communicating compared to the linear overhead case).

Having done this analysis, what is now needed is some data on the distribution of individual communications overhead. Is the distribution linear, square-root, exponential? I am not aware of any such data (there is a chance I have encountered something close and not appreciated its importance).

I have only every worked on relatively small teams, and am inclined towards the distribution of time spent communicating not being constant. Was it exponential or a power-law? I would not like to say.

Could a communications time distribution be reverse engineered from email logs? The cc’ing of people who might have an interest in a topic complicates the data analysis; time spent in meetings are another complication.

Pointers to data most welcome and as is any alternative analysis using data likely to have a higher signal/noise ratio.

Test Language: Behaviours, Not Examples

Naming is hard, as we know from the old adage about the two hardest problems in Computer Science, and naming in tests is no different. I’ve documented my own journey around how I structure tests in two previous posts: “Unit Testing Evolution Part II – Naming Conventions” and “Other Test Naming Conventions”. I’ve also covered some similar ground before quite recently in “Overly Prescriptive Tests” but that was more about the content of the tests themselves, whereas here I’m trying to focus more on the language aspects.

Describing the Example

Something which I’ve observed, both from reviewing Fizz Buzz submissions with tests [1] and from real tests, is that there is often that missing leap from writing a test which describes a single example to generalising the language to describe the effective behaviour [2]. For example, imagine you’re writing a unit test for a calculator, if you literally encode your example as your test name you might write:

[Test]
public void two_plus_two_is_equal_to_four()

Given that you could accidentally implement it with multiplication and still make the test pass you might add another scenario to be sure you don’t fall into that trap:

[Test]
public void three_plus_seven_is_equal_to_ten()

The problem with these test names is that they only tell you about the specific scenario covered by the test, not about the bigger picture. One potential refactoring might be to parameterise the test thereby forcing you to generalise the name:

[TestCase(2, 2, 4)]
[TestCase(3, 7, 10)]
public void adding_two_numbers_together_returns_their_sum(. . .)

One way this often shows up in FizzBuzz tests is with examples for the various rules, e.g.

[Test]
public void three_returns_the_word_fizz()

[Test]
public void five_returns_the_word_buzz()

The rules of a basic calculator are already known to pretty much everyone but here, unless you know the rules of the game Fizz Buzz, you would not be able to derive them from these examples alone and one very important role of tests are to document, nay specify, the behaviour of our code.

Describing the Behaviour

Hence to encode the rules you need to think more generally:

a_number_divisible_by_three_returns_the_word_fizz

There are a couple of issues here, namely that technically any number is divisible by three (just not wholly), and also that it won’t be true once we start bringing in the more advanced rules. It’s not easy trying to be precise and yet also somewhat vague at the same time, but we can try:

a_number_wholly_divisible_by_three_generally_returns_the_word_fizz

Once we bring in the “divisible by three and divisible by five” rule it becomes much harder to be precise in our test names as we’d have to include the overriding rules too which likely makes them harder to read and comprehend:

a_number_wholly_divisible_by_three_but_not_also_wholly_divisible_by_five_returns_the_word_fizz

You might just get away with it this time but its not really scalable and test names, much like code comments, often have a habit of getting out of sync with reality. Even when they break due to new functionality it’s easy to end up fixing the test and forgetting to check whether the “documentation” aspect still reflects the new behaviour.

Hence I personally prefer to use words in test names that suggest “broad strokes” when necessary and guide the reader (top to bottom) from the more general scenarios to the more specific. This, in my mind, is similar to putting the happy path cases before the various error handling ones.

Validating Collections

These examples might be a little too trivial but the impetus for this post came from similar scenarios where the test language talked about the outcome of the example itself rather than the behaviour of the logic in general. The knock-on effect of doing this, apart from making the intent of the example harder to comprehend in the future, was that it also became brittle as the specific scenario outcome was encoded in the test and any change in logic that might be orthogonal to it could break it unnecessarily. (As mentioned earlier, “Overly Prescriptive Tests” looks at brittle tests from a different angle.)

A common place where this shows up is when asserting behaviours around collections. For example imagine writing tests for querying the seats available in a cinema where there are seats in different price bands. When testing the “seat query” method for an exhausted price band you might be inclined to write:

[TestFixture]
public class when_querying_for_seats_and_none_left_in_band
{
  [Test]
  public void then_the_result_is_empty()
  {
    auditorium.Add(“Posh Seats”, new Seats[0]);

    var seats = auditorium.FindAvailableSeats();

    Assert.That(seats, Is.Empty);
  }
}

The example, being minimal in nature, means that technically in this scenario the result will be empty. However that is an artefact of the way the example is expressed and the test has been written. If I were to change the test set-up and add the following line, the test would break:

auditorium.Add(“Cheap Seats”, new Seats[100]);

While the outcome of the example above might be “empty”, that is not the general behaviour of the logic under test and our test language should be changed to describe that:

[Test]
public void then_no_seats_in_that_band_are_returned()

Now we’re not making a statement about what else might or might not be in that result, only what our expectations are for seats in the band in question. Once we have fixed the test language we can address how we validate that in the example. Instead of looking at what is in the collection we should be looking at what isn’t there as the test name tells us to expect that something should be absent, and the assert should reflect that language:

Assert.That(seats.Where(s => s.Band == “Posh Seats”), Is.Empty);

Now I should only be able to break this test by changing the data or logic specific to the example, orthogonal behaviours should not break it by accident. (See “Manual Mutation Testing” for more on how you can test the quality of your tests.)

Invest in Tests

If you’ve ever worked on a codebase with brittle tests you’ll know how frustrating it can be when your feature mushrooms because you broke a bunch of badly written tests. If we’re lucky we see the failed assertion and if it’s not obvious then we can look back at the test name to see if the scenario rings any bells. If we’re unlucky we have to immediately reach for the debugger and likely add “refactor tests” to the yak stack.

If you “pay it forward” by taking the time to write good tests up front you’ll find it easier to sustain delivery in the future.

 

[1] A company I once worked for used Fizz Buzz in their candidate early screening process. Despite being overkill in practice (as was pointed out to candidates) a suite of tests was requested as part of the submission to help get a feel for what style they used. IMHO the tests said much more about candidates than the production code.

[2] Yes, “property based testing” takes this entire concept a step further so that it exercises the behaviour with multiple examples generated differently each time. That’s the destination, this post is about one possible journey.

StatsModels: the first nail in R’s coffin

In 2012, when I decided to write a book on evidence-based software engineering, R was the obvious system to use for data analysis. At the time, lots of new books had “using R” or “with R” added at the end of their titles; I chose “using R”.

When developers tell me they need to do some statistical analysis, and ask whether they should use Python or R, I tell them to use Python if statistics is a small part of the program, otherwise use R.

If I started work on the book today, I would till choose R. If I were starting five-years from now, I could be choosing Python.

To understand why I think Python will eventually take over the niche currently occupied by R, we need to understand the unique selling points of both systems.

R’s strengths are that it supports a way of thinking that is a good fit for doing data analysis and has an extensive collection of packages that simplify the task of applying a wide variety of analysis techniques to data.

Python also has packages supporting the commonly used data analysis techniques. But nearly all the Python packages provide a developer-mentality interface (i.e., they provide an API like any other package), R provides data-analysis-mentality interfaces. R supports a way of thinking that data analysts can identify with.

Python’s strengths, over R, are a much larger base of developers and language support for writing large programs (R is really a scripting language). Yes, Python has a package ecosystem supporting the full spectrum of application domains, this is not relevant for analysing a successful invasion of R’s niche market (but it is relevant for enticing new developers who are still making up their mind).

StatsModels is a Python package based around R’s data-analysis-mentality interface. When I discovered this package a few months ago, I realised the first nail had been hammered into R’s coffin.

Yes, today R has nearly all the best statistical analysis packages and a large chunk of the leading edge stuff. But packages can be reimplemented (C code can be copy-pasted, the R code mapped to Python); there is no magic involved. Leading edge has a short shelf life, and what proves to be useful can be duplicated; the market for leading edge code in a mature market (e.g., data analysis) is tiny.

A bunch of bright young academics looking to make a name for themselves will see the major trees in the R forest have been felled. The trees in the Python data-analysis-mentality forest are still standing; all it takes is a few people wanting to be known as the person who implemented the Python package that everybody uses for XYZ analysis.

A collection of packages supporting the commonly (and eventually not so commonly) used data analysis techniques, with a data-analysis-mentality interface, removes a major selling point for using R. Python is a bigger developer market with support for many other application domains.

The flow of developers starting out with R will slow down, casual R users will have nothing to lose from trying out another language when the right project comes along (another language on the CV looks good and Python is a bigger market). There will be groups where everybody uses R and will continue to use R because that is what everybody else in the group uses. Ten-Twenty years from now R, developers could be working in a ghost town.

We’re Not For Turning – a.k.

We have seen how it is possible to smoothly interpolate between a set of points (xi, yi), with the xi known as nodes and the yi as values, by specifying the gradients gi at the nodes and calculating values between adjacent pairs using the uniquely defined cubic polynomials that match the values and gradients at them.
We have also seen how extrapolating such polynomials beyond the first and last nodes can yield less than satisfactory results, which we fixed by specifying the first and last gradients and then adding new first and last nodes to ensure that the first and last polynomials would represent straight lines.
Now we shall see how cubic spline interpolation can break down rather more dramatically and how we might fix it.

What statistical techniques are useful for software engineering data?

What statistical techniques are of general usefulness for analyzing software engineering data?

The answer depends on the kinds of data likely to be encountered, in software engineering, and the questions likely to be asked.

When I started working on a book, aiming to cover all worthwhile publicly available software engineering data, I was hoping to refer readers to a book (or two) that they ought to read to learn the appropriate techniques. Kabacoff’s “R in Action” comes closest to the book I had in mind as a basic introduction, but there was nothing covering a wider range of topics; so I ended up writing something; I found Crawley’s “The R book”, to be the best book on the subject.

My answer to the kinds of data likely to be available was to work with all the software engineering data I could get obtain (around 600 data sets to date).

What questions should be asked about the data? My selection of questions was driven by whether the data was used in the software engineering half of the book, or the statistical analysis techniques half.

The software engineering material consists of the chapters: Introduction, Human cognitive characteristics, Cognitive capitalism, Ecosystems, Projects, Reliability and Source code. The data appeared in one of these chapters if it could be used to make (what I thought was) a practical point about the topic being discussed.

Data appeared in the statistical analysis techniques chapters, if it could be used to illustrate the technique under discussion.

What happened in practice was the software engineering material was worked on for a year or two, on realizing that bespoke statistical analysis material was needed the existing data was plundered to create the necessary chapters; after this was released, work switched back to the software engineering material (using unplundered and newly acquired data), and of course the earlier chapters plundered data from the yet to be worked on chapters.

This seems to have worked surprisingly well, at least from my perspective of keeping the production line going.

Now most if the data has been analyzed, it’s time to take a global overview and where necessary shuffle things around. I may find that everything is a complete mess; we shall see.

What techniques have I found to be useful?

The number 1, most useful data analysis technique is building a regression model. The one thing I have been consistently able to do, when analyzing other people’s data, is extract more information from it than they did (unless they also built a regression model); at times it has been embarrassing.

At number 2, is bootstrapping. Many widely used techniques only give accurate answers if the data has a normal/gaussian distribution and use of these techniques can involve a lot of arm waving involving claims about the data having a good-enough gaussian-like distribution. This arm waving was necessary before computers became available, because the practical manual techniques required a gaussian distribution. Now we have computers and techniques that don’t require any particular distribution can be used, and which in some cases are more powerful techniques than those designed for manual implementation.

Sitting here, I cannot think of a number 3; there might be one.

What techniques are not generally useful? The various tests containing some combination of the names Wilcoxon, Mann and Whitney are well past their sell-by date. Searching the source of the book I see these names still appear in one or two places; this is a hangover from the early versions from many years ago (when I was following the clueless herd) and will soon be gone.

I thought that extreme value theory might apply to some data, but have only found one data-set to which it might be applied (so not generally useful).

I spent a lot of time watching out for zero-inflated data (data containing more zero values than expected by the common probability distributions). I saw four/five papers containing plots of data that looked zero-inflated and emailed the authors asking for the data (who kindly sent it to me). None of the data turned out to be zero-inflated (I’m not sure what the authors thought about being asked for data that somebody thought was zero-inflated). This does not mean that software engineering data is not zero-inflated, only that it is not common.

My zero-inflated search was motivated by the occasional appearance of zero-truncated data (data with that does not contain zero values). Zero-truncated data occurs when counting starts at one, rather than zero (I have one data-set that is 0/1 truncated; the counting starts at 2).

I was surprised that time-series did not turn out to be widely useful.

Sometimes we are all clueless button pushers, so machine learning gets a few pages. Anybody who knows what they are doing builds regression models.

I will eventually get around to counting how many times each technique is used on the data I have (watch this blog, but don’t hold your breath).

Source code chapter added to “Evidence-based software engineering using R”

The Source Code chapter of my evidence-based software engineering book has been added to the draft pdf (download here).

This chapter has suffered from coming last and there is still lots of work to be done. Almost all the source code related data has been plundered to fill up earlier chapters. Some data did not make the cut-off for release of the draft; a global review will probably result in some data migrating back to this chapter.

When talking to developers about the book I am constantly being asked ‘what is empirical software engineering?’ My explanation uses the phrase ‘evidence-based’, which everybody seems to immediately understand. It is counterproductive having a title that has to be explained, so I have changed the title to “Evidence-based Software Engineering using R”.

What is the purpose of a chapter discussing source code in a book on evidence-based software engineering? Source code is obviously an essential component of the topics discussed in the other chapters, but what is so particular to source code that it could not be said elsewhere? Having spent most of my professional life studying source code, first as a compiler writer and then involved with static analysis, am I just being driven by an attachment to the subject?

My view of source code is very different from most other developers: when developers talk about code, they spend most of the time talking about how they do things, when I talk about code I spend most of the time talking about how other developers do things (I’m a mongrel writer of code). Developers’ blinkered view of code prevents them seeing bigger pictures. I take a Gricean view of code and refrain from using meaningless marketing terms such as maintainability, readability and testability.

I have lots of source code data of interest to compiler writers (who are not the target audience) and I have lots of data related to static analysis (tool developers are not the audience). The target audience is professional software developers and hopefully what has been written is of interest to that readership.

I have been promised all sorts of data. Hopefully some of it will arrive. If somebody tells you they promised to send me data, please encourage them to take some time to sort out the data and send it.

As always, if you know of any interesting software engineering data, please tell me.

Finalizing the statistical analysis material in the second half of the book (released almost two years ago) next.

Perl’s failure to grow and Python takes over

Perl, once the most widely used scripting language, has been in decline for many years; the decline now looks terminal (many decades from now, when its die-hard users have died), what happened?

Python is what happened. Why was this? Did Perl have a major fail, did Python acquire pixie dust that could not be replicated, or something else?

Some commentators point to the failure to produce a timely release of Perl 6; a major reworking of the language announced in 2000 with a stumbling release made available around 2015.

I think the real issue is a failure for Perl to take off outside its core use as a systems language. Perl is famous for its one-liners, but not for writing large programs (yes, it can be done, but would many developers would really want to?); a glance of the categories in its module library shows; those 174,970 modules (at the time of writing) are not widely spread over application domains (i.e., not catering to a wide audience).

Perl 5 was failing to grow outside its base before Perl 6 began its protracted failure to launch.

Language use is a winner take-all game, developers create more packages, support tools, and new users who combine to attract more developers. Continuing support for minority languages comes from die-hard users, existing software that is worth somebody paying to maintain and niche advantages.

These days, language success is founded on the associated package ecosystem (Go and Rust have minuscule package ecosystems, which is why they are living on borrowed time, other languages will eventually take away their sheen of trendiness). Developers use languages to build stuff, the days of writing the code for almost everything are long gone; interesting software is created by taking advantage of packages written by others. Python was in the right place, at the right time to acquire a wide variety of commercial grade packages.

It’s difficult to see Python being displaced as the lingua franca of software development. Its language features are almost irrelevant, its package ecosystem is everything. The winner will eventually take all.

I’m sure the cycle of languages becoming popular for a few years, before disappearing, will continue. There have always been, and will always be, fashionable languages.

TODO or TODO Not – Redux

One of my most “successful” posts was one I wrote way back in 2011 about my dislike for TODO style comments in code – “TODO or TODO Not - There Is No Later”. The premise of that post, which includes a number of examples from real codebases I’ve worked on, is that they are fundamentally pointless because they are almost certainly too low in value to get done. If they were valuable enough they either should be a proper feature on the backlog or be left to be handled as part of a relevant story, e.g. refactoring.

At the time I wrote that post I was convinced about them not living in the codebase (past the feature’s release) but I suggested that any potentially useful ones should be converted into bona fide user stories so they could be formally considered and prioritised along with the other items. After receiving a reply to a tweet that referenced my aging blog post from a company that tries to quantify technical debt by analysing such TODO style comments I felt it must be time for an update. (The TL;DR of this post was my reply to them.)

// Learn to Let Go

One of the hardest lessons I have learned over the intervening 7 years is how to let go of stuff. What caused me to cling onto some of those TODOs that I would run across was a fear of forgetting something important. My daily use of a log book (see “Pen & Paper”) to record notes as I go along exemplifies my apparent need to keep track of my current state as my brain is like the proverbial sieve. In an era where change was much harder because the software development QA process was largely manual this makes sense as it was more popular to batch-up changes. Couple this with a general disregard for anything non-functional, such as an appreciation for refactoring, and it’s all too easy to see why people bury their personal backlog in the code rather than open it up for discussion with non-developers.

There is a familiar ring here and that’s because I’ve walked this path before more recently in 2016’s “Developers Can Be Their Own Worst Enemy”. With a modern development process that puts an emphasis on trust and transparency we can let go of the past and should have more confidence in our peers and the management to see the value in our opinions.

It’s also not acceptable to just leave a cryptic comment in the code about why something should be implemented differently or moved elsewhere – the need to change must be backed up with a reason so that we can understand the value in it, and most TODO comments are throwaway rather than well reasoned arguments.

// Measuring Technical Debt

I’ll be honest, I genuinely cannot see the point of attempting to measure the level of technical debt in a codebase, even if it were possible to do so. For a start you need to decide if you’re taking the original interpretation – a conscious decision to temporarily take a shortcut - or the ever more popular one – crap code. Even if you could do that, what are you ever going to do with the output? I once saw a SonarQube report that said we had £X million of technical debt in a codebase I was working on; how exactly does that inform you?

The reason I find it pointless is because it feels like it’s measuring the wrong thing. Just like the story about the man looking for his keys under the streetlight we apply a tool to measure the quality of the code when what really matters is measuring our ability to deliver. The reason poor quality code affects us is because it makes future change hard and slows us down. Hence if it matters there must be a causal link because if the code is that bad our pace of delivery will slow down and we’ll see that (all other things being equal).

But what is more important in my mind is that even if you did know how much debt you had you would only want to focus on the areas that are subject to change, and you do that already by focusing on the most valuable work in the backlog! And the technique for tackling the debt is refactoring. Hence we already have what we need to address the real problem.

// Trimming the Backlog

Unwinding the stack slightly let me return to the topic of reflecting unfinished work in the product’s backlog.

That sentence should already be making your spider-senses tingle – how can it be unfinished? If we’ve not met the acceptance criteria we’re not done, so carry on. If the acceptance criteria was wrong or incomplete then that’s a discussion to have with the product owner for clarification. Note that “of good quality” is an acceptance criteria inherent in every story we do as it is virtually non-negotiable [1].

Perhaps a better term would be undiscovered work. Woody Zuill has this saying “it’s in the doing of the work that we discover the work that needs doing”. What we often unearth are things which never showed up in our initial analysis and therefore we now have to factor them into our schedule or decide to drop them. When we’re knee deep in the feature it may seem really important to do it now, but given time the need may slowly dissolve, and in the end possibly disappear altogether.

What you might initially put on the backlog (or write as a TODO in the code) could be quite specific, e.g. “Need to handle poisoned messages”. As you continue you might also add other scenarios such as “Should refresh token before it expires”  and “Triage malformed messages separately from those well-formed but still invalid”. These are all valuable behaviours which go towards building a reliable service but the initial focus might not be on reliability, maybe its currently just a tactical fix to enable learning about something else. You don’t want to waste time over-engineering but at the same time you also don’t want to forget what else you have learned doing it.

The problem with this mentality is that the backlog just grows and grows, and we’re back to where we were before with TODOs littered all over the code – it’s just an every growing feature list. As time passes the need to implement some of those features will either happen because they have suddenly become important (it’s no longer tactical) or they will vanish (it remains a mere stop-gap). Leaving the stories on the backlog just makes it harder and harder to prioritise as you keep going over old ground again and again.

Once again we need to let go and rely on the fact that if they are important they will eventually resurface. If you feel really uncomfortable about deleting them entirely then you might consider rolling up related stories back into epic sized ones as part of the backlog grooming. Applying this to our recent example you could easily fold them all into a single “Service Reliability” epic that would be easier to handle. You could turn each card’s title into a checklist item.

// Confidence

That said as I get older I become less tolerant of a never-ending backlog and want to be more aggressive in the backlog grooming. Part of that is down to having more confidence in knowing that issues will be addressed [2] in a more timely fashion and that prioritisation will take into account both the technical and functional features with more or less equal consideration because the team is trusted to do The Right Thing.

Being more experienced there is no doubt more than a grain of truth that my sphere of influence is probably wider but that shouldn’t really matter. Every story must be valuable and ideally stand-up on its own merit, where my experience comes in is probably in being able to express that value more succinctly.

// TODO: Redux

I haven’t seen, heard or read anything in the intervening years that has been able to convince me in the value of using TODO comments in the code as a more successful technique of managing what needs to be done. If anything my appetite for tracking any work outside the next few sprints / weeks has begun to wane simply because it is now so easy to change in direction with only a moment’s notice should the situation deteriorate. This does not mean we should be reckless, far from it, but leaving a TODO in the code as a way of conveying a change in architecture hardly seems optimal either.

As for the notion that a TODO in the code can be equated with an increase in technical debt I don’t buy that either. I would posit it is more likely to indicate a failure within the development process itself as the mismatch between behaviour and acceptance criteria either indicates a bug in the code or a bug in the requirements and neither of these outcomes sounds like something that should be brushed off lightly with a comment in the code.

 

[1] I try not to deal in absolutes as there always seems to be an exception, but either way it must be a conscious decision.

[2] Assuming the organisation is behaving in a moderately agile way and not just paying lip service to a couple of the ceremonies or practices.

Evolutionary pressures on C++, Java and Python

The future evolution of C++, Java and Python is being driven by very different interested parties, and it’s going to be interesting watching events unfold over the next 5-10 years.

I have previously written about how the C++ Standard’s committee is past its sell-by date, has taken off its ball and chain and is now in the hands of bored consultants.

Bjarne Stroustrup was once effectively treated as C++’s Benevolent Dictator For Life (during the production of the first C++ Standard some people were labeled as Bjarne groupees); things have moved on since then, but the ‘old-guard’ are trying to make a comeback. Suggesting that people ought to base their thinking on a book published almost 25-years ago (Stroustrup’s “The Design and Evolution of C++”; a very interesting book that is well worth reading) creates a rather backward looking image. Bored consultants are looking to work on exciting new ideas. The old-guard need to appear modern to attract followers (even if the ideas are old ideas with a fresh coat of paint).

The threat to C++ is from bored consultants, each adding their own pet idea to the language standard; a situation that Stroustrup thinks is starting to happen.

Java, the language, is owned by Oracle, the company (let’s not get too involved in exactly what they own, have copyright on, etc). Oracle are not shy about asking people for licensing fees. Java is now on a 6-month release cycle (at least the Oracle version, there are Open Source implementations) and the free support only applies to the current release; paying a license fee buys support for versions older than 6-months. In the short term, the cheapest solution is for companies to pay for support.

Oracle are always happy to send in the lawyers and if too many customers switch to non-Oracle implementations, I’m sure something can be found to introduce enough uncertainty to discourage work/distribution involving Open Source Java implementations.

Will Java survive Oracle’s licensing? It is not in their interest for Java to die; Oracle will adjust their terms to keep the money flowing in, but over the longer term I think willing Java developers are going to be hard to find.

Guido van Rossum recently removed himself from the post of Python’s Benevolent Dictator For Life. One of the jobs of a benevolent dictator is maintaining some degree of language coherence, which involves preventing people’s pet ideas from being added to the language. Does this mean that Python is slowly going to be become more and more bloated? Perhaps, but I think a more likely problem is a language fork, multiple implementations of slightly different (at first) languages all claiming to be Python.

These days, the strength of Python is its large collection of very useful, commercial grade, packages, and future language details may turn out to be irrelevant. There is a lot to learn from the Python 2/3 transition, but true believers like to think that things will turn out differently for them.

On Blockade – student

Recall that the Baron's game is comprised of taking turns to place dominoes on a six by six grid of squares with each domino covering a pair of squares. At no turn was a player allowed to place a domino such that it created an oddly-numbered region of empty squares and Sir R----- was to be victorious if, at the end of play, the lines running between the ranks and files of the board were each and every one straddled by at least one domino.

Delivery Anti-Pattern: Local Optimisations

The daily stand-up mostly went along as usual. I wasn’t entirely sure why there was a stand-up as it wasn’t so much a team as a bunch of people working on the same codebase but with more-or-less individual goals. Applying the microservices analogy to the team you could say we were a bunch of micro-teams – each person largely acting with autonomy rather than striving to work together to achieve a common goal [1].

Time

But I digress, only slightly. What happened at the stand-up was a brief discussion around the delivery of a feature with the suggestion that it should be hidden behind a feature toggle. The implementer explained that they weren’t going to add a feature toggle because “it was a waste of their time”.

This surprised me. Knowing what I do now about how the team operates it isn’t that surprising but at the time I was used to working in teams where every member works towards common goals. One of those common goals is to try and ensure the delivery of features is a continuous flow and is not disrupted by a bad change which then has to be backed out because rolling back has the potential to create all sorts of disruption, not least the delay of those unaffected changes.

You should note that I’m not disagreeing with their choice of whether or not to use a feature toggle – I did not know anywhere near enough about the change or the system at that time to contribute to that decision. No, what disturbed me was the reason why they chose not to take that approach – that their time is more valuable than that of anyone else in the team, or the business for that matter.

In isolation that paints an unpleasant picture of the individual in question and that simply is not the case. However their choice of words, even if done without real consideration, does appear to reinforce the culture that surrounds them. In essence, with a feeling that the focus is on them and their performance, they are naturally going to behave in a way that favours optimising delivering their own features over that of the team at large.

Quality

Another example of favouring a local optimisation over the longer term goal of sustained delivery occurred when I was assigned my first piece of work. This was not so much a story as a couple of epics funded as an entire project (over 4 months solid work in the end). My instinct, after being shown roughly where in the code I needed to dig, was to ask where the existing tests were so that I knew where to add mine. The tech lead’s immediate response was “you won’t have time to write tests”.

My usual response to this statement is a jovially phrased “how will I know if it works then?” which often has the effect of opening a line of dialogue around the testing strategy and where it’s heading. Unfortunately this time around it only succeeded in the tech lead launching into a diatribe about how important delivery was, how much the business trusted us to deliver on time, blah, blah, blah, in fact almost everything that a good test suite enables!

Of course I still went ahead and implemented the entire project TDD-style and easily delivered it on time because I knew the approach was sound and the investment was more than worth it. The subsequent enhancements and repaying of some technical debt also became trivial at that point and meant that anyone, not just me, could safely and quickly make changes to that area of code. It also showed how easy it was to add new tests to cover changes to the older parts of the component when required later.

In the end over 10% of unit tests of the entire system had been written by me during that project for a codebase of probably over 1/2 million lines of code. I also added a command line test harness and a regression testing “framework” [2] in that time too all in an effort to reduce the amount of hoops you needed to go through to diagnose and safely fix any edge cases that showed up later. None of this was rocket science or in the least bit onerous.

Knowledge

I would consider a lack of supporting documentation one further local optimisation too. When only a select few have the knowledge to help support a system you have to continually rely on their help to nurse it through the bad times. This is especially true when the system has enough quirks that the cost of taking the wrong action is quite high (in terms of additional noise). If you need to remember a complex set of conditions and actions you’re going to get it wrong eventually without some form of checklist to work from. Relying on tribal knowledge is a great form of optimisation until core members of the team leave and you unearth the gaping holes in the team’s knowledge.

Better yet, design away the problems entirely, but that’s a different can of worms…

Project Before Product

I believe this was another example of how “projects” are detrimental to the development of a complex system. With the team funded by various projects and those projects being used as a very clear division on the task board through swim lanes [3] it killed the desire to swarm on anything but a production incident because you felt beholden to your specific stakeholders.

For example there were a number of conversations about fixing issues with the system that were slowing down delivery through unreliability that ended with “but who’s going to pay for that?” Although improvements were made they had to be so small as to not really affect the delivery of the project work. Hence the only real choice was to find easier ways to treat the symptoms rather than cure the disease.

Victims of Circumstance

Whenever I bump into this kind of culture my gut instinct is not to assume they are “incompetent” people, on the contrary, they’re clearly intelligent so I’ll assume they are shaped by their environment. Of course we all have our differences, that’s what makes diversity so useful, but we have to remember to stop once in a while and reflect on what we’re doing and question whether it’s still the right approach to take. What works for building Fizz Buzz does not work for a real-time, distributed calculation engine. And even if that approach did work once upon a time the world keeps moving on and so now we might be able to do even better.

 

[1] Pairing was only something you did when you’d already been stuck for some time, and when the mistake was found you went your separate ways again.

[2] I say “framework” because it was really just leveraging a classic technique: a command line tool reading CSV format data which fired requests into a server, the results of which are then diff’d against a known set of results (Golden Master Testing).

[3] The stand-up was originally run in project order, lead by the PM. Unsurprisingly those not involved in the other projects were rarely engaged in the meeting unless it was their turn to speak.

Agile is the process digital technology needs

1200px-Workers_in_the_fuse_factory_Woolwich_Arsenal_Flickr_4615367952_d40a18ec24_o-2018-07-18-12-19.jpg

In my presentation at Agile on the Beach last week I continued my discussion of Agile and Digital. It is increasingly clear that digital and agile are intrinsically linked. Specifically, business need agile processes to get the most out of digital technology. My “Agile, Digital & the new management paradigms” presentation is online but let me give you the argument here.

There is a long standing model of technology change – so widespread I can’t find the original source – which says change comes in three steps:

  1. First new technology allows the same processes and activities to be done better, faster, cheaper, more efficiently. In this stage new technology is used to do the same things, the processes and practices change little.
  2. Next new technology allows process and practices to be reconsidered and changed to make the most of new technology. Work becomes even better – whether that be faster, cheaper, higher efficiency, superior products, whatever.
  3. Finally new innovations appear because of the technology and new processes. One can see opportunities for new businesses, new business models, the next round of technology innovation and more.

So the whole thing repeats.

Look at the photo above. According to WikiCommons this is a picture of a factory at Woolwich Arsenal sometime in the 1800s. Notice the belts stretching from the ceiling to the workstations. These carried power, or to be more precise motion. Above the workers is the line shaft which turns. The shaft is driven by a central power (motion) source somewhere, probably a water wheel or a steam engine.

This is before electricity. The line shaft and the belts carry the power the factory needs to work. And they break, the longer they are the more prone to breaking they are. Factory design is constrained by the need to have straight lines for the line shaft and short distances between the shaft and the workstation. And factory design dictates layout and processes.

Then came electricity.

Electricity allowed each workstation to have its own motion generator. At first factory owners used electricity to do the same things faster and more reliably. They could dispense with the steam engine and thus the stokers and coal it needed. But at first they didn’t seize all the advantages electricity brought.

It took time to understand how a factory could be laid out more efficiently and how processes could be changed. When they did factories got even more efficient and faster. Some might argue that it took the coming of Lean manufacturing to complete these process changes.

The same story has played out in industry after industry with technology after technology. Think of Word processors: first they helped secretaries do their job faster, then processes changed and everyone wrote themselves, goodbye secretaries. Containerisation in the shipping industry is another. First ships loaded and unloaded faster. Then the shipping companies innovated but more importantly world trade innovated. Some observers claim containerisation was a more significant factor in trade globalisation than free-trade agreements.

Digital technology is like electricity. It changes business, it creates new opportunities for doing things differently. To get the most from digital technology you need new processes. Right now most companies are stuck – even happy – doing things faster. Only when they change processes will they get the full benefits.

Agile processes are that change.

Agile ways of working help companies get more from digital technologies. Without Agile companies using digital technologies are just doing the same old thing faster.

Agile started in software development for two reasons. First software developers had a lot of problems, they had the need to change. Second, programmers had the first access to digital technologies.

Ray Tomlinson, a programmer, was the first person with e-mail. Tim Berners-Lee, a programmer, had the first web-browser. Ward Cunningham, a programmer, had the first Wiki. I could continue.

Software developers created Agile because they needed to and they could.

This is why Agile is taking off in marketing.

Outside of technology itself marketing has probably been more exposed to digital technology than any other part of business. First with digital publishing then with social media. At first digital helped marketing departments do the same work faster. Next it changed what you could do entirely. Marketing is adopting agile because those processes allow marketeers to do a better job when working with new digital technology.

So forget all those arguments about agile being a better way of working (it is but never mind).

Forget all those stories of agile like processes and practices before 1998 (yes they existed but that doesn’t change things).

Forget the debate about waterfall and upfront planning versus agile and just-in-time (that is history).

All you need to know is:

  1. Digital technology is helping you do things faster/better/cheaper.
  2. Agile ways of working allow you to get more from digital tools.
  3. More innovation is coming.

Agile is the process for digital businesses.

Sign-up to receive these posts by e-mail and free eBook of Xanpan

Image of Woolwich Arsenal factory taken from WikiCommons, no known copyright.

The post Agile is the process digital technology needs appeared first on Allan Kelly Associates.

nor(DEV):con 2019 Call For Speakers


nor(DEV):con 2019 Call For Speakers
Thursday 21st to Saturday 23rd February 2019
Kings Centre
63-75 King Street
Norwich
NR1 1PH

nor(DEV):con, the Norfolk Developers Conference is back for 2019 and stronger than ever with new tracks and an updated format. nor(DEV):con is Norfolk and Norwich’s premier and most well attended conference for everyone involved in software development and business.

Call for Speakers

The call for speakers is open from now until Friday 28th of September 2018. To submit a proposal, please send an email to paul@norfolkdevelopers.com with the following:

  • Session Title
  • Session abstract
  • Session topics: Pre-conference workshop, Tech, Process. Workshop, Stephen Fry, Business, Lightning talk
  • Session length: 45 min presentation, 90 minute presentation, 90 minute hands on workshop, Full day hands on workshop, 5 minute Lightning talk
  • Technical level: Beginner, Intermediate, Advanced, Business
  • Speaker(s) Bio
  • Availability for Thursday 21st (full day workshops only), Friday 22nd and Saturday 23rd February.

Further information will be required if you proposal is accepted.

Conference Topics

Pre-Conference Full Day Workshops - We’re looking for three full day pre-conference hands on workshops. These should be on a software development or process theme and we’re looking for one beginner, one intermediate and one advanced workshop. Pre-conference workshops attract between 5 and 30 people.

Tech Track - We’re looking for 45 minute and 90 minute beginners, intermediate and advanced presentations primarily on coding, techniques and libraries. This year we’re particularly keen to include a few Blockchain sessions.

Process Track - Rather than just concentrating on Agile, this year we have a process track for any type of software development process. Yes, you can even propose a session on Waterfall if you’re feeling brave!

Stephen Fry Track - Software Development is more than just code and practices. It’s about people. And people are not logical entities that follow easily understood rules. We’re illogical, irrational, and tend to shy away from difficult subjects like mental illness, inequality, and bias. This track looks at the softer side of Software Development, at the people, how they interact, and how we can work with the whole gamut of humanity, because it’s not just Stephen Fry who has  manic depression.

Workshop Track - We’re looking for 90 minute hands on workshops for up to 20 people. These can be on any of the topics above and delegates bring their own laptops.

Business Track - The business track is the track for everyone! We’re looking for business based sessions with a technical slant.

Lightning Talks - On the Friday evening during the wine reception, which will be in the main hall, we’ll also be opening the floor to lightning talks. A lightning talk is a 5 minute presentation and we encourage new speakers as well as experienced speakers to present.

Speakers Package

  • Free entry to the Friday and Saturday sessions at the conference
  • Free entry to the Friday night speakers dinner
  • Travel and accommodation at the discretion of the conference organise


Filters to help decide who might be a software developer

How do you find people who are likely to be good software developers?

I use the filter approach: start with whoever is available, filter out those who are not likely candidates and go with those that are left (if any).

The first filter is a question: which language do you like to program in?

This question is positive, in that it assumes the other person is a developer; asking for the name of a language makes it a difficult to dodge question for those who don’t know any language. The language itself is irrelevant, apart from as a lead in to further discussion.

Learning to program is easy and a fun thing to do, at least if you are the kind of person likely to become a good developer. Cheap computing hardware has been available since the 1980s, the extra ingredients are a desire to write software and some degree of the necessary skills.

The next filter is a discussion about the largest software system they have written.

The theme of the discussion is how they solved the problems encountered during the implementation. Do the problems sound like something a developer of the person’s experience ought to find a problem? How much perseverance was shown in solving the problems, were they flexible in trying alternatives, what was their approach to problem solving?

Building systems is all about solving problems. People who cannot solve problems will fail, those with problem solving abilities might succeed.

What about paper qualifications?

Demand for developers continues to outstrip supply, creating an opportunity for turkeys to fly.

When getting a university degree was intellectually challenging, it was a sign of cognitive firepower. The stated aim of the UK government is for 50% of 18-year olds to study for a degree, which means that courses requiring high cognitive firepower are dumbed down (otherwise the failure rate goes through the roof and a University’s ranking suffers). If the only option is a turkey shoot, a degree in a subject requiring lots mathematical thinking (e.g., physics, chemistry, some psychology subjects, …) is obviously a much better filter than Medieval French, Modern History, etc.

There are people whose path through life has kept them away from computers when they were younger and university when they were a bit older. Software carpentry seems to be doing good things for such people; I don’t have any direct experience of working with those who have gone that route, and so cannot say anything about it.

Will this filter approach work for you? Well, it depends on the characteristics required of a good developer in your line of work.

Perhaps you need a regular Joe, who does the job, nine-to-five, and sticks to the tried and trusted approached; a solid person who keeps systems reliably maintained and customers happy.

The independent, frontier, mentality that thrives in ‘new’ fields is becoming a less tolerated in software development. The frontier shrinks as more and more software becomes good-enough and those with money to pay for change, spend it on something else.

SYNC THE CITY 2018 – SAVE THE DATE & NEW VENUE

We are really pleased to announce that Sync the City 2018 will be held on November 15-17th at OPEN, Norwich.

This will be our 5th year in which we challenge attendees to Build & Launch a Startup in 54 Hours - for fun or profit.   Last year the event sold out in 9 days, subscribers to this list will be notified of ticket sales 24 hours ahead of public sale (early September).

OPEN Norwich - its BIG, its BEAUTIFUL



Experience the highs, lows, fun, and pressure that make up life at a startup.

Do you have an idea that could be solved using technology? Or perhaps want to know more about how technology startup businesses become the next Facebook?

SyncNorwich's flagship event is Sync the City 2018 is a 54-hour event that brings together entrepreneurs, developers, business managers, marketing gurus, graphic artists, and students to pitch ideas for new startup companies, to form teams around those ideas, and to develop a working prototype, demo & presentation.

In just 54 hours, you will experience the highs, lows, fun, and pressure that make up life at a startup.

CONNECT with people driven to build something new.

DISCOVER where you are on the Entrepreneur's Journey.

LEARN what it really takes to start a company. No book, panel, speaker will teach you what you need to know.

START It’s that simple. Sync the City is designed to get you going, FAST. So grab your ticket syncthecity.com and let's do this.



From Work Experience to Oxford University


"It was an invaluable and pretty unique experience"

Here at Naked Element we’re big into supporting the future of tech, and that means young people. One work experience recruit was Chelsea, an ambitious school student who went on to be accepted at Oxford to study Computer Science. We couldn’t be more proud! Here she tells us a little about her journey and what inspired her to take up tech and join Naked Element for some real world experience.

How did you start your work experience with Naked Element?

One of my teachers at school was friends with Paul Grenyer (MD of Naked Element), and I was looking around for any work experience in the tech/computing/software industry. I had begun thinking about what I wanted to do in the future and had begun thinking of going into a technology based career. I was introduced to Paul and he asked me what I wanted out of the experience and offered me work experience. After completing a week, he offered to let me come and do another week of paid work!

What skills did you bring to Naked Element do you think?

Not sure I brought too many skills but I had some previous programming experience with python and a tiny bit in Java. I’d done my fair share of teamwork and group projects but this was definitely my first experience in a ‘work’ / ‘professional’ environment.

What did your work experience entail?

I was really surprised when I arrived on my first day and was given a real project to work on, and actual code to edit. It was a vastly different experience to other placements I had done and I loved the hands on experience. I got to work on a couple of projects including one that would manage your social media posts. What was really interesting for me was seeing the difference between the theory we learn in class and how it’s actually implemented in real life, such as client and server side processing. When I came in I thought I would be completely out of my depth, but even though I didn’t understand Java to start with everyone at Naked Element was willing to take the time to explain to me how something worked and what it was doing. Even being given a task as simple as going through previous code and fixing mistakes or inefficient parts was a useful experience for me and has helped in me checking to make sure my A Level coursework is as efficient as possible!

Were there any stand out moments during your time with the company?

Definitely the best part about the work experience was the hands on nature of it. I definitely had never had work experience that was so hands on, it would often just be tours of the departments etc. But with Naked Element I actually got to look and work on code for live projects.

What did you learn while at Naked Element?

I learnt a lot on my work experience. I got a serious introduction to Java and experience with JavaScript and CSS as well as what a career in Software Development could entail. As well as the actual tech aspects of the experience. I learnt a lot about the business management side of things and it was interesting going to one of the talks and training sessions and learning how to better advertise the company. It was also brilliant being introduced to other tech companies around Norwich and seeing how they interact.

What made you choose Oxford?

I’ve always been pretty ambitious, and I wanted to apply for Oxford on the chance I got in. The opportunities a degree from Oxford would provide would be almost unmatched and I adore the city. I’d also enjoyed looking round on the open days and the taster lectures I had. I currently hold an offer to study Computer Science at Jesus College, I just need to get the grades now!!

Do you think your time with Naked Element helped with your application?

I definitely think it helped, it was an invaluable and pretty unique experience that helps make you stand out from the thousands of others who apply. I know a lot of people don’t have the opportunity to get work experience.

Do you have any plans for your career future yet?

I’m not sure about a career future. After my work experience I’m definitely looking at software development. I also like the idea of working in cyber security or even AI. I think Computer Science opens up a vast field of job prospects and I haven’t quite got around to choosing one just yet. Luckily I don’t have to.

Do you have any advice for other young people interested in tech?

Firstly go out and get experience. It’s not easy but there’s no harm in asking and you’re not going to find what you enjoy or what you’re good out without trying things out. I don’t think there is any harm in broadening your skills and any experience is good experience.

Allow drag-to-side, but not drag-to-top in Ubuntu MATE (Marco)

I love the “tiling” feature in many window managers including Marco that means I can drag windows to the side of the screen and get them covering one half.

However, I never use the similar feature that allows dragging a window to the top, and it often triggers when I just want to move a window upwards.

Today I discovered that Marco (the non-fancy window manager in Ubuntu MATE and probably other places) does allow me to have one without the other, even though the UI configuration tools don’t expose the option.

Here’s what I did:

gsettings set org.mate.Marco.general allow-top-tiling false

Now my windows can be dragged to the side for half-screen maximisation, but not to the top for full-screen!

Optimistic SQL

One of the benefits of learning other programming languages is the way it teaches you about other paradigms and idioms. This is the premise behind the “Seven Xxx in Seven Weeks” range of books. Although I have the database one on my bookshelf I’ve only ever skimmed it as at the time I bought it I suddenly found myself leaving the world of the classic RDBMS behind and working with other types of DB for real; most notably the document-oriented kind.

Although some of these products like MongoDB and Couchbase have come a long way from their early beginnings as highly available key-value stores they often still lack the full-on transaction support of the old stalwarts like SQL Server and PostgreSQL. Coupled with a high-availability service you have to think differently about how you react to concurrency conflicts as explicit locking is almost certainly never the answer [1].

The impetus for this post was going back into the world of SQL databases and being slightly bemused by a stored procedure that appeared to implement an “upsert” (an UPDATE or INSERT depending on whether the row already exists) as I realised it wasn’t how I’d approach it these days.

The Existing SQL Approach

Initially I was somewhat flummoxed why it was even written the way it was as there appeared to be no concurrency issues in play at all, it was a single service doing the writing, but I later discovered that an accident of the implementation meant there were two writers internally competing and they chose to resolve this in the database rather than remove the root source of concurrency in the service.

The upsert was basically written like this:

  • Try SELECTing the existing row.
  • If it exists, UPDATE it.
  • If it doesn’t exist, INSERT it.

In the service code there were a number of comments describing why the transaction level was being bumped up to “serializable” – it was effectively to deal with the concurrency they had introduced within the code by creating two competing writers. On top of that the initial SELECT statement in the upsert applied a HOLDLOCK which also effectively makes the transaction serializable because it puts a range lock on the row’s key (even if that key doesn’t exist yet).

The Document DB Approach

The last few years away from the relational world meant that I was used to dealing with these kinds of conflicts at a slightly lower level. Also, dealing with document updates in the service rather than writing them as SQL mean that updates were done in a server-side loop rather than pushing the concurrency issue down into the database, hence it would look more like this:

  • Try selecting the document.
  • If it exists, update it and try writing it back.
  • If it doesn’t exist, try creating it.
  • If any write fails start over from the beginning.

Due to the lack of transactions and locking, write conflicts are commonly detected by using a version number attribute that gets used in the update predicate [2]. (A write failure, via a “document not found” error, means the predicate failed to match the specific document and version and therefore a conflict has occurred.)

Another SQL Approach

So what does all this have to do with upserts in SQL?

Well, what I found interesting was that my gut reaction was to question why there is the initial select there as I would have written it as:

  • Try to UPDATE the row.
  • If no rows were updated, then INSERT it.

This particular order makes an assumption that updates are more prevalent than inserts and as a I rule I’d say that checking @@ROWCOUNT to see if anything was written is far less ugly than adding a TRY…CATCH block in T-SQL and attempting to verify that the insert failed due to a primary key violation.

That all seemed fairly obvious but I had forgotten that with the document DB approach you tend to expect, and handle, write failures as part of handling concurrency, but in this case if two connections both attempted the insert concurrently it’s theoretically possible that they could both fail the UPDATE step and then one of the INSERTs would succeed and the other would fail resulting in a primary key violation. However the code in the service was not written to detect this and retry the operation (as you would with a document DB) which is why the initial SELECT is there – to lock the “unwritten row” up front which ensures that another transaction is blocked until the row is then inserted or updated. This way no client logic needs to handle the concurrency problem.

However I believe we can still achieve the same effect by adding the same HOLDLOCK hint to our initial UPDATE so that if the row does not exist other writers will be blocked by the range lock until the subsequent INSERT goes through. Hence the initial SELECT is, I believe, redundant.

The MERGE Approach

At this point I remembered that way back in the past SQL Server introduced the MERGE operation which effectively allows you to write an upsert with a single statement as you factor both the hit and miss logic into different branches of the statement. This caused me to go looking to see what the start of the art in upsert techniques were, possibly with performance comparisons to see how much faster this must be (given that SQL Server clearly has the potential to optimise the query plan as it better knows our intent).

I started digging and was somewhat surprised when I came across the page “Performance of the SQL MERGE vs. INSERT/UPDATE”. I was expecting to have my hypothesis validated but discovered that the answer was far from clear cut. Naturally I then googled “SQL Server upsert performance” to see what else had been written on the subject and I discovered this wasn’t an anomaly so much as a misunderstanding about what problem the MERGE statement is really intended to solve.

You should of course never take performance improvements at face value but “measure, measure, measure” yourself. I wasn’t avoiding doing that, I was looking to see if there might be any pitfalls I needed to be wary of when benchmarking the approach.

At this point I haven’t gone any further with this as it’s more of a personal investigation (there is no actual performance issue to solve) but it just goes to show that writing SQL is as much an art as it’s always been.

 

[1] Some document databases, such as Couchbase, do support locking of documents, but there are heavy restrictions so you tend to find another way.

[2] In the particular example I was looking at no version number was needed in the SQL predicate because the data had a total ordering independent of the write order (it was tracking the minimum and maximum of a value over the day).

Super Simple Named Boolean Parameters

Quite often we come across interfaces with multiple boolean parameters, like this:

cake make_cake (bool with_dairy, bool chocolate_sauce, bool poison);

A call to this function might look like:

auto c = make_cake(true, true, false);

Unfortunately, this is not very descriptive. We need to look at the declaration of the function to know what each of those bools mean, and it would be way too easy to accidentally flip the wrong argument during maintainance and end up poisoning ourselves rather than having a chocolatey treat.

There are many solutions which people use, from just adding comments to each argument, to creating tagged types. I recently came across a very simple trick which I had not seen before:

#define K(name) true

Yep, it’s a macro which takes an argument and gets replaced by true. This may look strange, but it enables us to write this:

auto c = make_cake(K(with_dairy), !K(chocolate_sauce), !K(poison));

// or using `not` if you prefer
auto c = make_cake(K(with_dairy), not K(chocolate_sauce), not K(poison));

Of course, it’s up to you whether you think using a macro for this kind of thing is worth it, or if you think that macro should have a name which is less likely to collide. There’s also the issue that this looks a bit like Python named parameters, where you can pass them in any order, but the order matters here. There are much better solutions when you own the interface, but for the times when you don’t, I think this is a neat trick to solve the issue in a low-overhead manner.

A guess I need to come up with a marketing name for this. Since we already have X Macros, I hereby dub this trick “K Macros”.

Even the smallest job deserves dedication!

We don’t just create industry-changing software for big businesses, we can also pick up smaller jobs that need a quick result. Naked Element recently completed a minor job for Thyngs, a digital consumer engagement company based in Norwich. With our flexible team, experienced in development and design, we were able to turn the work around in a matter of hours.

Niall, head of marketing at Thyngs, said “we needed some urgent, short-term support with some updates we needed to make to our website. We know Paul and the team, and he was responsive and eager to help.”

Our designer/developer Shelley stepped in to get the job done. “Thyngs wanted some new pages added to their existing website, all in the same format, and they wanted their homepage amended slightly. Niall was lovely, easy to talk to and clear in what he wanted, which meant there were no snags or issues to worry about!”

Even though it was only a few hours work, Naked Element still have a reputation for quality. “Everyone was very professional and proactive” said Niall, “I would absolutely recommend Naked Element to companies in need of similar support”

Even the smallest job deserves dedication!

Support-Friendly Tooling

One of the techniques I briefly mentioned in my last post “Treat All Test Environments Like Production” was how constraining the test environments by adhering to the Principle of Least Privilege drove us to add diagnostic specific features to our services and tools.

In some cases that might be as simple as exposing some existing functionality through an extra command line verb or service endpoint. For example a common technique these days is to add a “version” verb or “–-version” switch to allow you to check which build of a particular tool or service you have deployed [1].

As Bertrand Meyer suggests in his notion of Command/Query Separation (CQS) any behaviour which is a query in nature should have no side-effects and therefore could also be freely available to use for diagnostic purposes – security and performance issues notwithstanding. Naturally these queries would be over-and-above any queries you might run directly against your various data stores, i.e. databases, file-system, etc. using the vendors own lower-level tools.

Where it gets a little more tricky is on the “command” side as we might need to investigate the operation but without disturbing the current state of the system. In an ideal world it should be possible to execute them against a part of the system reserved for such eventualities, e.g. a special customer or schema that looks and acts like a real one but is owned by the team and therefore its side-effects are invisible to any real users. (This is one of the techniques that falls under the in-vogue term of “testing in production”.)

If the issue can be isolated to a particular component then it’s probably more effective to focus on that part of the system by replaying the operation whilst simultaneously redirecting the side-effects somewhere else (or avoiding them altogether) so that the investigation can be safely repeated. One technique here is to host the component in another type of process, such as a GUI or command line tool and provide a variety of widgets or switches to control the input and output locations. Alternatively you could use the Null Object pattern to send the side-effects into oblivion.

In its most simplest form it might be a case of adding a “--ReadOnly” switch that disables all attempts to write to back-end stores (but leaves logging intact if that won’t interfere). This would give you the chance to safely debug the process locally using production inputs. As an aside this idea has been formalised in the PowerShell world via the “-WhatIf” switch which allows you to run a script whilst disabling (where supported) the write actions of any cmdlets.

If the operation requires some form of bulk processing where there is likely to be far too much output for stdout or because you need a little more structure to the data then you can add multiple switches instead, such as the folder to write to and perhaps even a different format to use which is easier to analyse with the usual UNIX command line tools. If implementing a whole different persistence mechanism for support is considered excessive [2] you could just allow, say, an alternative database connection string to be provided for the writing side and point to a local instance.

Earlier I mentioned that the Principle of Least Privilege helped drive out the need for these customisations and that’s because restricting your access affects you in two ways. The first is that by not allowing you to make unintentional changes you cannot make the situation worse simply through your analysis. For example if you happened to be mistaken that a particular operation had no side-effects but it actually does now, then they would be blocked as a matter of security and an error reported. If done in the comfort of a test environment you now know what else you need to “mock out” to be able to execute the operation safely in future. And if the mocking feature somehow gets broken, your lack of privilege has always got your back. This is essentially just the principle of Defence in Depth applied for slightly different reasons.

The second benefit you get is a variation of yet another principle – Design for Testability. To support such features we need to be able to substitute alternative implementations for the real ones, which effectively means we need to “program to an interface, not an implementation”. Of course this will likely already be a by-product of any unit tests we write, but it’s good to know that it serves another purpose outside that use case.

What I’ve described might seem like a lot of work but you don’t have to go the whole hog and provide a separate host for the components and a variety of command-line switches to enable these behaviours, you could probably get away with just tweaking various configuration settings, which is the approach that initially drove my 2011 post “Testing Drives the Need for Flexible Configuration”. What has usually caused me to go the extra step though is the need to use these features more than just once in a blue moon, often to automate their use for longer term use. This is something I covered in much more detail very recently in “Libraries, Console Apps & GUIs”.

 

[1] Version information has been embedded in Windows binaries since the 3.x days back in the ‘90s but accessing it easily usually involved using the GUI shell (i.e. Explorer) unless the machine is remote and has limited access, e.g. the cloud. Luckily PowerShell provides an alternative route here and I’m sure there are plenty of third party command line tools as well.

[2] Do not underestimate how easy it is these days to serialise whole object graphs into JSON files and then process them with tools like JQ.

Instructions that cpus don’t need to support

What instructions can computers do without (an earlier post covered instructions they should support)?

The R in RISC was supposed to stand for Reduced, but in practice almost all the instructions you would expect were supported. What was missing were the really complicated instructions that machines of the time (last 1980s), like the VAX, supported (analysis of instruction set usage showed that these complicated instructions were rarely used; from the compiler perspective the combination sequence of operations supported by these instructions rarely occurred in code).

One instruction that was often missing from the early RISC processors was integer multiply. Compilers were expected to generate a series of instructions that had the same effect. Some of the omitted ‘basic’ instructions got added to later versions of the processors that survived commercially (e.g., SPARC).

The status register is still a common omission from RISC designs (at least for the integer operations). Where is the data showing that in the grand scheme of things (i.e., processor performance running real programs), status registers slow things down? I know that hardware designers don’t like them because they introduce bottlenecks. I don’t recall ever having seen an analysis of instruction set usage targeted at the impact of status registers on generated code. Pointers welcome.

These days, nobody seems to analyze instruction set usage like they did in times past. Perhaps Intel’s marketing and the demise of almost every cpu vendor has dampened enthusiasm for researching new cpu designs. These days most new cpu designs seem to be fashion driven, rather than data driven.

Do computers need registers? An issue that once attracted lots of research was the optimal number of registers for a processor. The minimum number of registers (or temporary storage locations) needed to evaluate an expression was known by 1970. There were various studies of the impact, on code generation, of increasing/decreasing the number of registers available to the compiler. But these studies were done using 1990s era compilers and modern compilers do many more optimizations; whole program optimization ought to be able to make use of many more registers than are probably available on today’s processors (at least I think so, until somebody does a study that shows otherwise). There is a register-less processor that is supposed to be taking the world by storm, sometime soon.

Do computers need to support the IEEE floating-point representation? Logarithmic number systems are starting to be used in various devices, but accuracy remains an issue for some applications.

Treat All Test Environments Like Production

One of the policies I pushed for from the start when working on a greenfield system many years ago was the notion that we were going to treat all test environments (e.g. dev and UAT) like the production environment.

As you can probably imagine this was initially greeted with a heavy dose of scepticism. However all the complaints I could see against the idea were dysfunctional behaviours of the delivery process. All the little workarounds and hacks that were used to back-up their reasons for granting unfettered access to the environments seemed to be the result of poorly thought out design, inadequate localised testing or organisational problems. (See “Testing Drives the Need for Flexible Configuration” for how we addressed one of those concerns.)

To be clear, I am not suggesting that you should completely disable all access to the environment; on the contrary I believe that this is required even in production for those rare occasions when you just cannot piece together the problem from your monitoring and source code alone. No, what I was suggesting was that we employ the same speed bumps and privileges in our test environments that we would in production. And that went for the database too.

The underlying principle I was trying to enshrine here was that shared testing environments, by their very nature, should be treated with the utmost care to ensure a smooth delivery of change. In the past I have worked on systems where dev and test environments were a free-for-all. The result is that you waste so much time investigating issues that are orthogonal to your actual problem because someone messed with it for their own use and just left it in a broken state. (This is another example of the “Broken Windows” syndrome.)

A secondary point I was trying to make was that your test environments are also, by definition, your practice runs at getting things right. Many organisations have a lot of rigour around how they deploy to production but very little when it comes to the opportunities leading up to it. In essence your dev and test environments give you two chances to get things right before the final performance – if you’re not doing dress rehearsals beforehand how can you expect it to go right on the day? When production deployments go wrong we get fearful of them and then risk aversion kicks in meaning we do them less often and a downward spiral kicks in.

The outcome of this seemingly “draconian” approach to managing the development and test environments was that we also got to practice supporting the system in two other environments, and in a way that prepared us for what we needed to do when the fire was no longer just a drill. In particular we quickly learned what diagnostic tools we should already have on the box and, most importantly, what privileges we needed to perform certain actions. It also affected what custom tools we built and what extra features we added to the services and processes to allow safe use for analysis during support (e.g. a --ReadOnly switch).

The Principle of Least Privilege suggests that for our incident analysis we should only require read access to any resource, such as files, the database, OS logs, etc. If you know that you are protected from making accidental mistakes you can be more aggressive in your approach as you feel confident that the outcome of any mistake will not result in breaking the system any further [1][2]. Only at the point at which you need to make a change to the system configuration or data should the speed bumps kick in and you elevate yourself temporarily, make the change and immediately drop back to mere mortal status again.

The database was an area in particular where we had all been bitten before by support issues made worse through the execution of ad-hoc SQL passed around by email or pasted in off the wiki. Instead we added a new schema (i.e. namespace) specifically for admin and support stored procedures that were developed properly, i.e. they were written test-first. (See “You Write Your SQL Unit Tests in SQL” for more on how and why we did it this way.) This meant applying certain kinds of workarounds were easier to administer because they were essentially part of the production codebase, not just some afterthought that nobody maintained.

On the design front this also started to have an interesting effect as we found ourselves wanting to leverage our production service code in new ways to ensure that we avoided violating invariants by hosting the underlying service components inside new containers, i.e. command line tools or making them scriptable. (See “Building Systems as Toolkits”.)

The Interface Segregation Principle is your friend here as it pushes you towards having separate interfaces for reading and writing making it clearer which components you can direct towards a production service if you’re trying to reproduce an issue locally. For example our calculation engine support tool allowed you to point any “readers” towards real service endpoints whilst redirecting the the writers to /dev/null (i.e. using the Null Object pattern) or to some simple in-memory implementation (think Dictionary) to pass data from one internal task to the next.

I find it somewhat annoying that we went to a lot of effort to give ourselves the best chance of designing and building a supportable system that also provided traceability only for the infrastructure team to disallow our request for personal per-environment support accounts, saying instead that we needed to share a single one! Even getting them to give us a separate account for dev, UAT and production was hard work. It sometimes feel like the people who complain most about a lack of transparency and rigour are the same ones that deny you access to exactly that.

I know there were times when it felt as though we could drop our guard in dev or UAT “just this once” but I don’t remember us ever doing that. Instead we always used it as an opportunity to learn more about what the real need was and how it could become a bona fide feature rather than just a hack.

 

[1] That’s not entirely true. A BA once concocted a SQL query during support that ended up “bug checking” SQL Server and brought the entire system to a grinding halt. They then did it again by accident after it was restarted :o).

[2] A second example was where someone left the Sysinternals DebugView tool running overnight on a server whereupon it filled up the log window and locked up a service due to the way OutputDebugString works under the covers.

Cubic Line Division – a.k.

Last time we took a look at how we can use linear interpolation to approximate a function from a set of points on its graph by connecting them with straight lines. As a consequence the result isn't smooth, meaning that its derivative isn't continuous and is undefined at the x values of the points, known as the nodes of the interpolation.
In this post we shall see how we can define a smooth interpolation by connecting the points with curves rather than straight lines.

Software engineering is fertile ground for the belief in silver bullets

The idea that there exists some wonderful technique or methodology, which solves one or more perceived software engineering problems, was given a name in 1986; the title of Brooks’ paper No Silver Bullet is a big clue that the author does not think it exists. Indeed, over the years a steady stream of papers have attempted to dispel the idea that silver bullets exist. These attempts have two things in common: the use of reasoning and facts to make their case, and failure to dispel the idea that there are no silver bullets in software engineering.

Now, I am a great fan of reasoning using facts, but I am also a fan of evidence driven approaches to solving problems. There is now over 30 years of evidence that reasoning using facts is not an effective means of convincing people that silver bullets don’t exist.

Belief in silver bullets will not go away until it ceases to be in some peoples’ interest for them to exist.

If you have something to sell, there is a benefit to having customers believe in silver bullets: the product/research will dramatically improve performance, time to market, costs, profitability, etc…

Belief in silver bullets is not unfounded. Computing has a 70-year history of things going faster, getting cheaper and systems doing what was once thought impossible. The press has bought into this and amazing success stories abound. Having worked on a few projects that delivered faster/cheaper/impossible systems, I know that no silver bullets were involved, just lots of hard work and sometimes being in the right place at the right time. Hard work and happenstance don’t make for feel-good headlines, and rarely get mentioned in the press.

When faced with a problem, the young and inexperienced tend to be optimists; there must be a silver bullet better way of doing this that is fast/cheap/efficient. The computing field has been evolving so rapidly that many of those involved are young and inexperienced; fertile ground for belief in silver bullets to flourish.

Consequences of a belief in silver bullets in industry include, time/cost overruns on projects and money wasted on tools that are never used. In academia a belief in silver bullets results in the pointless invention of new programming languages, methodologies, programming techniques, etc.

The belief in silver bullets will not fade away until the rate of change in computing slows to a crawl and most of those involved have gained substantial experience from which they can see that results come from hard work (and some amount of luck).

Organizational structure in the Digital and Agile age

iStock_000003002725XSmall-2018-07-3-18-18.jpg

Someone ask the other day: how should a organisation be designed?

There are two potential answers, which actually aren’t as contradictory as they look at first site.

The first is very simple: Don’t.

That is, don’t design your organization, don’t set out an organizational chart, don’t set out a plan and aim to restructure your organization to that plan. Rather create the conditions to let a structure emerge.

I suppose its the difference between “design” meaning “create a plan for the way you want things to be” and “design” meaning “the way things are arranged.” To differentiate them the first might be called “intentional design” and the latter “emergent design.”

That does not necessarily imply all emergent structures are good. As we see in code sometimes emergent designs are not always the best and over time they need refactoring. Which implies at some point there needs to be intentional design.

Put it like this: I’d rather your organization pulls the design rather than you push a design on the organization.

Organizational structure is itself a function of business strategy. And both need to be part emergent and part intentional. Although you might have noticed I tend towards emergent while most of the world tends towards intentional!

Thus it helps to have a reference model of how you think the organization should be, maybe something to steer the organization towards.

So the second answer to the question would be longer:

  • Create standing delivery teams which are embedded in the business line itself. This is sometimes call stream teams, or stream based development, or “teams aligned to the value stream”, or several other names I can’t think of just now.
  • Each business line is itself a stream of work and digital delivery teams support that work.
  • Teams contain all the skills and authority to do the work that is required for that business stream.
  • The team is part of the stream so the business/technical divide should dissolve. Something I call BusTech.
  • Teams are value seeking and value creating: the team seeks opportunities to create value for the business and delivers on the most valuable ones.
  • Devolve authority to the teams whenever you can. Teams are mini-businesses. (Notice I deliberately don’t use the word empowerment.)
  • Teams grow when the business is successful and more digital capability is needed. And teams shrink when money is tight or less capability is needed.
  • Teams may split (Amoeba style) from time to time. New teams may be in the same business line (addressing another question) or part of another, possibly new, business line.
  • Active – or Agile – Portfolio Management sits on top to monitor progress, provide extra resources, remove resources, etc. There may even be multiple portfolio processes, one at the business line level and perhaps one above multiple business lines.
  • Minimally Viable Teams are started to explore new initiatives, sometimes these go on to be full standing teams but they may also be dissolved if the idea doesn’t validate.
  • Seek to minimise common services between teams because these create bottlenecks, conflicts and delays. Each team should stand alone. This may mean some duplication, and therefore some extra costs, but accept that. Once you have your model working you can fine tune such things later.
  • Don’t worry about planning and synchronisation between teams to much, worry more about getting the teams to release more often and deal with synchronisation issues when they become a problem.

They are the main points at any rate. If you’d like to know more Continuous Digital contains a longer discussion of the topic. (Continuous Digital actually builds on Xanpan in this regard, and the (never finished) Xanpan Appendix discusses the same idea.)

Sign-up to receive these posts by e-mail and free eBook of Xanpan

The post Organizational structure in the Digital and Agile age appeared first on Allan Kelly Associates.

#NoProject #NoEstimates workshop

MilkCartons-2018-07-3-17-57.png

In August I’m running a 1-day workshop in Zurich with Vasco Duarte on the bleeding edge of Agile: #NoProjects and #NoEstimates for Digital First companies.

This is a pre-conference event for the ALE 2018 conference which is happening the same week in Zurich. Everyone is welcome, you don’t need to attend the conference.

If you book in the next two weeks you get it for cheap, after July 20 the price goes up – although its still only a few hundred euros.

Book now, save money and secure your place – places are limited!

For those ho can’t get to Zurich in August I’ve got a Continuous Digital workshop of my own and a half-day management briefing. Right now you can book either of these for private in-house delivery. I’m looking at offering these as public courses here in London, if you are interested get in touch and help me fix a date.

(I have a love hate relationship with #NoProjects, I’d love to retire the name but it resonates with so many people. So I tend to use #NoProjects when I’m discussing my critique of the project model and Continuous Digital when I’m setting out my preferred alternative.)

The post #NoProject #NoEstimates workshop appeared first on Allan Kelly Associates.

Time taken to compile a source file

How long will it take to compile a source file?

When computers were a lot slower than they are today, this question was of general interest. Job scheduling is more effective when reliable runtime estimates are available, and developers want to know if there is enough time to get a coffee before the compile finishes.

An embarrassing fact about compile time performance, used to be that a large percentage of compile time was spent doing lexical analysis [“The cost of lexical analysis”, I cannot find an online copy]. Why was this embarrassing? Compiler writers like to boast about all the fancy optimizations their compiler does; but doing fancy stuff consumes lots of resources, so why were compilers spending so much of their time doing simple things like lexical analysis? The reality was that fancy compiler optimizations were not commercially viable until developer computers contained tens of megabytes of memory, i.e., very few pre-1990 compilers did any real optimization (people are still fussing over lexer performance).

An analysis of the data in Captain Dennis Miller’s Masters thesis (late Rome period), finds compile time is proportional to the square root of the number of tokens in the source (code+data); more complicated models are a slightly better fit. Where did square root come from? I expected a linear relationship, but would be willing to go with log. The measurements are from Ada compilers in the mid 1980s. I know several people who worked on Ada compilers during that time, and they were implementing the latest fancy optimizations (Ada was going to be the next big thing and the venture capital was flowing; big companies, with big computers were going to be paying lots of money to use Ada, but then microcomputers came along). I think that square root is driven by OS resource limitations, the compilers are using lots of memory and a noticeable amount of time is spent swapping.

So computers got a lot faster and people lost interest in estimates of how long it would take to compile individual files. I have not seen any interest in predicting how long it would take to compile whole projects (just complaints about how long it takes). There has been some work on progress indicators, updated as compilation progresses, which is a step in the right direction. Perhaps somebody has recorded compile time information and thrown machine learning at it; I usually ignore machine learning papers applied to software engineering and perhaps I have missed something. Pointers to project compile time prediction work welcome.

Then along came just-in-time compilation. Now people want to estimate how long it will take to generate machine code from some intermediate form, that is being interpreted.

The plot below (thanks to Rafael Auler for kindly supplying the data from his paper) shows the time taken to generate code from functions containing a given number of LLVM instructions (an intermediate code), at optimization level O3. The red line is a regression fit to one of the ‘arms’ and shows constant time for less than 100’ish instructions and then a linear relationship. I have no idea why the time is roughly constant for a large number of functions.

Time taken to convert functions containing a given number of LLVM instructions to machine code

There is a lot of variation for function containing the same number of instructions. This is to be expected when lots of different optimizations are being tried; sometimes a function will contain lots of the kind of code that a particular optimization spends lot of times process and sometimes the code will not contain anything interesting (i.e., no optimizations are found).

Find out how the employer voice can influence technical & vocational training.


When: Monday 2nd July @ 5.30pm

Where: The Library Restuarant, 4A Guildhall Hill, Norwich NR2 1JH

RSVP: https://www.meetup.com/Norfolk-Developers-NorDev/events/251982755/

How the employer voice can influence technical & vocational training and apprenticeships in the digital and IT skills sector over the next 5 years.

Jerry White - Deputy Principal at City College Norwich

The Government’s agenda for the wider field of technical and vocational education alongside apprenticeships is changing. As a large FE College and apprenticeship provider we believe that employers are central to this process and our aim this evening is to start meaningful dialogue into how we can enable employers to both support and challenge us as we launch the new T Level curriculum and apprenticeship standards. We want to hear from you about the challenges you have in recruiting employees and how we can work together to fill the gap and make a difference to the next generation.

The goal of the session is to begin to investigate where local businesses could help us trial the industrial placement model and how we can develop our working relationship over a number of years and ask you as employers what should our curriculum look like?'

Undergraduates and learning to program

I last looked at the research on teaching programming around 10 years ago and I have been catching up with what has been going on; in brief: same old, same old. One of the best papers on the subject is still: Language-independent conceptual “Bugs”

The research activity is still focused on making the tools and language ‘better’. There is a defining silence on the possibility that those doing the teaching could not teach their way out of a paper bag. Nobody is brave enough to suggest that teacher training might be a worthwhile investment, or that lectures oriented to what is useful (rather than what the lecturer finds interesting) would be appreciated by students.

I have always thought that researching the teaching programming had no practical purpose, other than possibly helping universities increase the number of students graduating with computing degrees (some universities are solving the problem students have with programming by offering degrees that don’t involve being able to program). I still think that teaching programming to school children is at best a waste of time.

My experience with students learning to program is from a very long time ago. The process involved listening to confusing and disjoint lectures, reading books and figuring out what worked by trial and error. Students were not taught to program, they got thrown in at the deep and were expected to survive. Anybody who could handle this stood some chance of being able to handle developing software in the ‘real world'; universities were (accidentally) graduating people with the skills industry needed. However, these days universities are supposed to be customer focused, what industry needs to irrelevant (my experience of sitting on departmental industry panels is that the head of department tells us what they are thinking of doing {i.e., new courses for which there will be lots of paying students} and we try to talk him/her out of the sillier ideas); too many fee paying students find programming too hard, let’s offer computing degrees that don’t require any programming.

Would you hire a recent graduate, for a development role, who had trouble figuring out how to fix syntax errors in their code? Surely, the minimum requirement is somebody who gets some pleasure from coding, even if they don’t want to spend lots of time doing it.

There is a shortage of software developers and flying turkeys are still with us.

Best practices considered harmfull

NoBestPractice-2018-06-20-16-53.jpg

I’ve long worried about “Best Practices”. Sure I usually play along at the time but lurking in the back of my mind, waiting for a suitable opportunity are two questions:

  • Who decided this was best practice?
  • Who says this practice can’t be bettered?

I was once told by someone from the oil industry that it was common for contracts to specify “best practice” should be used. But seldom was the actual practice specified. Instead each party to the contract would interpret best practice as they wished, until something went wrong. At that point, after an accident, after money was lost they would go to court and a judge would decide what was best practice.

Sure practice X might be the best know way of doing things at the moment but how much better could it be? By declaring something “best practice” you can be self limiting and potentially preventing innovation.

Now a piece in MIT Sloan Management Review (Why Best Practices Often Fall Short, Jérôme Barthélemy, February 2018) adds to the debate and highlights a few more problems.

Just for openers, sometimes people mistakenly identify the practice creating the benefits. Apparently some people looked at Pixar animation and decided that having rest rooms (toilets to us English speakers) in the centre of an office floor enhances creativity. They might do, but there is so much else happening at Pixar that moving all the toilets in your organization will probably make no difference at all.

But it is worse than that.

Adopting best practice from elsewhere does not mean it will be best practice in your environment but adopting that “best practice” will be disruptive. Think of all the money you will need to spend relocating the toilets, all the people who will be upset by a desk move they don’t want, all the lost productivity while the work is going on.

The author suggests that in some cases that disruption costs are so high the “best practice” will never cover the costs of the change. Organizations are better shunning the best practice and carrying on as they are. (ERP anyone?)

It gets worse.

There is risk in those best practices. Risk that they will cost more, risk that they won’t be implemented correctly and risk that they will backfire. What was best practice at one organization might not be best practice in yours. (Which might imply you need even more change, even more disruption at even more cost.)

In fact, some best practices – like stock options for executives – can go horrendously wrong and induce behaviours you most definitely don’t want.

So what is a poor company to do?

Well, the author suggests something that does work: copying good practices. Not best but “just OK”. That works. Copy the mundane stuff, the proven stuff. The costs and risks of a big change are avoided. (This sounds a bit like In Search of Mediocracy.)

In my world that means you want to be getting better at doing Agile instead of trying to leapfrog Agile and move to DevOps in one bound.

The author also suggests that where your competitive advantage is concerned keep your cards close to your chest. Do thinks yourself. Work out what your best practice is, work out how you can improve yourself.

I’ve long argued that I want teams to learn and learn for themselves rather than have change done to them. But I also want teams to steal. When they see other teams – at home or elsewhere – doing good things they should steal practices. The important thing from my point of view is for the teams to decide for themselves.

Sign-up to receive these posts by e-mail and free eBook of Xanpan

The post Best practices considered harmfull appeared first on Allan Kelly Associates.

Learn WordPress & build a website in ONE day


When: 28 June 2018, 9am to 4.45pm

Where: The Kings Centre, Kings Street, Norwich, NR1 1PH

How much: £150

RSVP: https://www.meetup.com/Norfolk-Developers-NorDev/events/250241910/

WordPress is the world’s best and most popular website builder and this hands-on course takes you through from the basics, including installation and set up, to cover all the most useful features and tools WordPress offers. Whether you already have a site and want to manage it properly or are starting completely from scratch, this is the course for you.

You will learn to

  • set up and run a great website of your own
  • add content, images and videos
  • add structure and navigation menus
  • apply an attractive design using easy templates
  • make the site search engine friendly
  • add contact forms, maps and take payments
  • add social networking and track visitorslearn to add all the features and functionality you need to run and develop a fantastic website
  • and much, much more…


How the course works


  • Please bring your own laptop: PC, Mac or Chromebook, any is fine. Or you can hire a laptop for the day here.
  • WiFi and power sockets are provided
  • No experience is needed – WordPress is incredibly easy to pick up and you will be free to go at your own pace throughout the day.
  • All training materials will be provided after the course, so there need be no fear of “falling behind”.This is an intensive course and assumes a reasonable working knowledge of using computers and the internet, even if you have little or no prior knowledge of WordPress. If you are comfortable with using email, copy/paste, saving files/folders and navigating the internet, you should be fine! (see more advice in our FAQ here)
  • After the course you are welcome to stay around for further discussion with your trainer Toby and with each other, about WordPress, about your website and about your business.
  • After the course, you will be sent all the course materials and clear instructions for setting up your site on its own domain name (old or new). You will have a year of free hosting, after which time it is from just £8/mo for unlimited space and bandwidth.


More details here (https://wpcourses.co.uk/wordpress-training-courses/?gclid=EAIaIQobChMI0bSL8tTR1wIVSjobCh2A9gVdEAAYASAAEgJLnfD_BwE).

Number of parameters vs. accessing globals

I spend a lot of time looking at software engineering data, asking, what is the story here?

In a previous post I suggested that the distribution of the number of functions defined to have a given number of parameters, might be a signature of developer beliefs about the relative cost of parameter passing vs accessing globals.

Looking at the data that Iran Rodrigues Gonzaga Junior made available (good man), as part of his thesis Empirical Studies on Fine-Grained Feature Dependencies, I saw it contained information about the number of parameters in a function definition and whether functions accessed a global (Gonzaga’s research question is in another direction; I am always repurposing data).

Are functions that access globals, defined with fewer parameters, compared to those that do not contain any such access? The plot below shows a count of the number of functions defined to have a given number of parameters, for four systems written in C; the solid lines are functions that did not access globals, the dashed lines are functions that accessed globals (code+data).

Number of functions defined to have a given number of parameters; four systems, written in C

Over all 50 projects measured, functions that don’t access globals are defined, on average, to have an extra 0.7 parameters (the fitted Poisson regression models are better than a poke in the eye {i.e., the distribution is not really Poisson}, it’s more informative to look at the plotted data).

There is a lot of variation between projects (I picked these four because they were the larger projects and showed variation in behaviors). While the shape of the distributions varies a lot, there is always a noticeable difference in the mean.

Is this difference between projects a difference in developer beliefs, a difference in application requirements, a difference in developer coding habits (and parameter usage is a side effect; are there really that many getters and setters)?

I was hoping for a simple answer, and could not find one. Since I am writing a book and not researching individual issues in detail, it’s time to move on.

Ideas welcome.

Further On Natural Analogarithms – student

My fellow students and I have of late been thinking upon an equivalence between the roots of rational numbers and an infinite dimensional rational vector space, which we have named -space, that we discovered whilst defining analogues of logarithms that were expressed purely in terms of rationals.
We were particularly intrigued by the possibility of defining functions of such numbers by applying linear algebra operations to their associated vectors, which we began with a brief consideration of that given by their magnitudes. We have subsequently spent some time further exploring its properties and it is upon our findings that I shall now report.

Connecting to Slack from an IRC client using slirc

I tried to get back to an IRC interface to Slack using Matrix, and it had some problems. Thanks to Colin Watson’s comment on that post, I tried Daniel Beer’s slirc, and so far it seems to be working pretty well.

Here’s what I did:

Get a Slack legacy token which slirc will use to connect to Slack as you. Follow the instructions given at that link, and you should end up with a token that looks something like “abcd-123456768-ETC-ETC”. Keep a note of it.

Install the prerequisites for slirc, and download it:

sudo cpan AnyEvent AnyEvent::HTTP AnyEvent::Socket AnyEvent::WebSocket::Client URI::Encode Data::Dumper JSON
mkdir slirc
cd slirc
wget -q 'https://www.dlbeer.co.nz/articles/slirc/slirc-20180515.pl'
chmod +x slirc-20180515.pl

Create a file in the slirc directory you created above, called rc.conf, and make it look like this:

slack_token=abcd-123456768-ETC-ETC
password=somepassword
port=6667

Replace “abcd-123456768-ETC-ETC” with the Slack legacy token you noted down earlier.

Replace “somepassword” with something you’ve made up (not your Slack password) – this is what you will type as the password in your IRC client.

Run slirc and leave it running:

./slirc-20180515.pl rc.conf

(Make sure you are inside the slirc dir when you run that.)

Start your IRC client (e.g. HexChat) and add a server with address “localhost” and port 6667, with your slack username and the password you added in the rc.conf (which you wrote instead of “somepassword”).

This mostly works for me, except it has a tendency to open a load of ad-hoc chats as channels, so I have to close them all to get a usable list.

Main memory: the crucial component that vendors don’t mention

CPU performance hogs the limelight when people discuss the year-on-year increases in computing power that used to occur.

This focus on cpu performance was/is driven by marketing, the people with the money either don’t want customers thinking about the performance impact of main memory size or speed, or want them to treat the processor as the most important component of a computer. Vendors want processor performance to drive customer purchase decisions.

Hardware manufacturers used to entice new customers with low cost machines, containing minimal memory. Once a customer started to use their shiny new computer, they found that it did save them lots of time and money, but also they needed more memory (which could only be brought from the manufacturer and was not cheap).

The plot below shows the prices IBM charged for System 360s, in 1966. Anti-trust investigations uncover all kinds of interesting data, like selling low-spec equipment at a loss to entice customers and make life difficult for competitors (code+data for all plots).

Profit margin on IBM 360s sold with various memory sizes

The plot below (data from the 19 Aug 1985 issue of ComputerWorld) shows how the price of computers increased as the minimum about of memory they supported increased.

Yes, in 1985 top end computers came with over 50M of memory; but most customers thought themselves lucky if they had a few megabytes.

If the processor is slow, it just takes longer for programs to run. If the computer does not have enough memory, programs cannot run. For most applications memory requirements are addressed first, followed by processor performance; memory requirements is the number one issue. The optimizations that commercial compilers could perform were limited by the memory capacity of developer machines.

List price of computers, in 1985, supporting the given minimum amount of  memory

Intel’s main line of business used to be selling memory chips, but these chips became commodity items as more companies entered the market; Intel bet the farm on selling processors and the rest is history. As a seller of a unique product it was/is in Intel’s interest to spend lots of money on marketing the benefits of processor performance; sellers of commodity items (such as memory chips) don’t have nearly as much to gain from generic product marketing, because customers may choose to buy from other sellers (in such markets sellers have to concentrate on marketing themselves).

Memory capacity/speed and cpu speed are two aspects of system performance; they need to be balanced to meet customer drive application requirements. The plot below shows the SPEC cpu integer performance of 4,332 systems running at various clock rates; the colors denote the different peak memory transfer rates of the memory chips in these systems (code+data).

SPEC cpu integer performance vs. cpu clock rate

These days (and perhaps in the past, I don’t have any data), memory performance is a much better predictor of system performance, but vendors don’t have an incentive to market this fact.

Introduction to Laravel Workshop



When: Wednesday, 20 June 20 - 9:00am to 4:45pm

Where: Kings Centre, King Street, Norwich, NR1 1PH

How much: £85

RSVP: https://www.meetup.com/Norfolk-Developers-NorDev/events/250343445/

If you would like to start using the worlds most popular PHP framework, this is the workshop for you.

We will cover from installing Laravel to building a basic application. Everyone will learn along as we go from a freshly installed Laravel application up to creating a basic web app that interacts with a database.

Course Details

We will aim to cover the following (subject to change and class experience)

  • Routing
  • Using Blade (templating)
  • Controllers
  • Validation
  • Config/Env
  • Using Eloquent (database ORM)
  • Laravel inbuilt Auth
  • Artisan (CLI commands)
  • File Storage

Instructor

Simon Bennett is a Software Consultant who works with Laravel daily, helping clients update there development practices and coding software for startups. He also runs his own SAAS for backing up DigitalOcean servers which of course is all built with basic Laravel.

Prerequisites

This beginners workshop will run through simple code as we learn about Laravel.

Basic knowledge of PHP is required, I would recommend you know you know basic OOP.

To save time on the day, you will be contacted directly before the workshop with a guide on setting up docker. The reason we use docker is to make sure we are using identical environments on Windows/Mac/Linux.

Since Laravel used to build web-based applications (mostly), it would be helpful to a basic understanding of developing websites in PHP and MySQL.

  • Bring your Laptop
  • Have Docker installed
  • A good IDE like PHPStorm is strongly recommended
  • Working command terminal
  • A GIT client installed

TechNorwich: The End. A Story & New beginnings


TechNorwich: The End. A Story & New beginnings

Whitespace is sadly now having to close. Started in 2013, it was the first and largest co-working space in Norwich. Its been home to many great & successful businesses, plus home to Barclays Eagle Labs and birthplace of TechVelocity, the first accelerator for the region. Come along to this event to have a drink to remember some of the history, catch up with some of the great businesses that have lived here and discuss on the future of the amazing digital creative & tech sector in our fine city.

Get your ticket: https://technorwich.eventbrite.co.uk

Using Matrix to connect to Slack from an IRC client on Ubuntu

Update: I found a better solution using slirc.

I like using HexChat to talk to my colleagues, like one other guy. It is fast, and it pops up a new window when someone sends me a direct/private message.

Recently, Slack shut down their IRC gateway, forcing me to use their slow UI, and making me unable to be responsive to direct messages without spending my entire life checking said UI, or being disturbed by notifications.

So, I decided to set up a Matrix bridge, because how hard can it be?

My system is Ubuntu MATE 18.04.

What I did

Install Synapse

Install Synapse, a Matrix homeserver.

wget -qO - https://matrix.org/packages/debian/repo-key.asc | sudo apt-key add -
sudo apt-add-repository http://matrix.org/packages/debian/
sudo apt update
sudo apt install matrix-synapse

Edit /etc/matrix-synapse/homeserver.yaml to change the line containing “registration_shared_secret” to look something like:

registration_shared_secret: "some secret password you made up"

(Replacing “some secret password you made up” with something secret.)

Make a new user:

register_new_matrix_user -c /etc/matrix-synapse/homeserver.yaml http://localhost:8008

Enter a username and password, and take a note of them.

Install Riot

(Technically, this step is optional, but I definitely recommend it to check everything is working.)

Install Riot, a client that can talk to a homeserver.

curl https://riot.im/packages/debian/repo-key.asc | sudo apt-key add -
sudo apt-add-repository https://riot.im/packages/debian/
sudo apt update
sudo apt install riot-web

Start Riot from the menu or by typing riot-web in a console.

Choose to log in to a Custom Server (not create a new user), and enter the Home Server URL as http://localhost:8008, leave the Identity Server URL as it is, and enter the username and password you entered in the previous step.

Riot should be able to connect to the Synapse server, even though it won’t have any rooms in it yet.

If this doesn’t work, try running riot-web in a console, and check the Synapse log file at /var/log/matrix-synapse/homeserver.log

Install matrix-puppet-slack

(Note: after this step, the homeserver can only be for you now, because it is logged into Slack as you. If you want to make a new Matrix server for lots of people, I found that matrix-appservice-slack works, but this only connects channels, not direct messages.)

Get a Slack legacy token which matrix-puppet-slack will use to connect to Slack as you. Follow the instructions given at that link, and you should end up with a token that looks something like “abcd-123456768-ETC-ETC”. Keep a note of it.

Install matrix-puppet-slack which allows you to connect a homeserver to a Slack instance, pretending it is you:

cd
git clone https://github.com/matrix-hacks/matrix-puppet-slack.git
cd matrix-puppet-slack/
npm install
cp config.sample.json config.json

Edit config.json so it looks something like this:

{
  "slack": [{
    "team_name": "Name of my team",
    "user_access_token": "abcd-123456768-ETC-ETC"
  }],
  "registrationPath": "slack-registration.yaml",
  "port": 8090,
  "bridge": {
    "homeserverUrl":"http://localhost:8008",
    "domain": "localhost",
    "registration": "slack-registration.yaml"
  }
}

Now generate a config file that Synapse will use, called slack-registration.yaml, by typing this command (still in the matrix-puppet-slack directory):

node index.js -r -u "http://localhost:8090"

When asked, type in the username and password you entered in the first step (when you ran the register_new_matrix_user command).

This creates a config file telling Synapse that matrix-puppet-slack is running on the local machine on port 8090.

Now, run it:

node index.js

and edit Synapse’s config file /etc/matrix-synapse/homeserver.yaml to point at the new config file you just made, by editing the line mentioning app_service_config_files so that it looks like this:

app_service_config_files: [
    "/path/to/matrix-puppet-slack/slack-registration.yaml"
]

Make sure you replace “/path/to/matrix-puppet-slack” in the above with the directory where you put matrix-puppet-slack, which is /home/yourusername/matrix-puppet-slack if you followed the instructions exactly.

Restart Synapse, and when you go back into Riot, you should see your Slack channels appear gradually (as people talk in them):

sudo service matrix-synapse restart

If not, check the output from the node index.js command, and the Synapse log file at /var/log/matrix-synapse/homeserver.log.

Install matrix-ircd

To allow connecting to Matrix from an IRC client, install matrix-ircd:

cd
git clone https://github.com/matrix-org/matrix-ircd.git
cd matrix-ircd
cargo build
cargo run -- --url "http://localhost:8008"

(Note this works on Ubuntu 18.04, but operating systems with older versions of Rust may need to get the latest first – see the matrix-ircd page for details.)

Connect from an IRC client

Now use an IRC client to connect to matrix-ird. I recommend HexChat if you’re looking for one.

matrix-ircd runs on port 5999 by default, so you should be able to connect to it from your IRC client by setting the server to localhost/5999 and using the username and password from the first step (when you ran the register_new_matrix_user command). Use plain username/password authentication.

If it works, you should see your Slack channels appear in your IRC client. If not check the logs mentioned before, and the error messages and logs from your IRC client.

What works and doesn’t work

  • Messages typed by me and others into IRC, Riot and Slack appear in all the other places, including direct messages.
  • Synapse and Riot are installed fully, so Synapse will be running after a reboot, and Riot is available in the menus, but matrix-puppet-slack and matrix-ircd are just running in a console, and have to be manually started. It should be reasonably simple to make custom systemd files to start them automatically.
  • matrix-puppet-slack crashed once and I had to restart it, so this may not be very reliable. I logged an issue.
  • The Slack channel names look terrible in my IRC client. I think matrix-ircd needs to find the pretty names (which are displayed correctly in Riot) and use them instead of the coded names used under the cover in Matrix.
  • My first ever direct message to someone does not work, even though I can choose them and attempt to send a message. As soon as they say something to me, they appear as a channel in Riot and IRC, and I can talk back and forth no problem from then on.
  • Direct messages look like normal channels in Matrix, which means I can’t use HexChat to pop up notifications for them, so this was all pointless.

Please leave comments below if you know how I can fix any of these problems.

Conclusions

  • Matrix seems really cool
  • It is basically possible to bridge IRC-Matrix-Slack as I wanted
  • But, some remaining bugs mean I have to keep using Slack’s UI for now :-(
  • Please comment telling me how to do this better

Function Template Partial Ordering: Worked Examples

C++ function overloading rules are complex. C++ template rules are complex. Put the two together, and you unfortunately do not get something simple; you get a hideous monster of standardese which requires great patience and knowledge to overcome. However, since C++ is mostly corner-cases, it can pay to understand how the rules apply for those times where you just can’t work out why your code won’t compile. This post will present a few step-by-step examples of how partial ordering of function templates works in order to arm you for these times of need.

Partial ordering of function templates is a step of overload resolution. It occurs when you call a function template which is overloaded and the compiler needs to decide which one is more specialized than the other. Consider this code:

template<class T> void f(T);        //(1)
template<class T> void f(T const*); //(2)

int const* p = nullptr;
f(p);

We expect f(p) to call (2), because p is a int const*. In order to decide that (2) is more specialized than (1), the compiler needs to follow the function template partial ordering rules. Let’s see what the standard has to say.

Partial ordering selects which of two function templates is more specialized than the other by transforming each template in turn (see next paragraph) and performing template argument deduction using the function type.

This is not so complicated, even if the terms may be unfamiliar. Unfortunately, the “next paragraph” and the sections which it references are almost impossible to parse without a lot of background knowledge and re-readings, so I shall step through the algorithm rather than pasting the rules for you to cry over.

There are four steps which we to have work through:

  1. Transforming (1).
  2. Performing deduction on (2) with the transformed template from step 1.
  3. Transforming (2).
  4. Performing deduction on (1) with the transformed template from step 3.

If one and only one of the deductions succeeds, then the template with which the deduction was performed is more specialized than the other.

Step 1: The rules state that for each template parameter, we create some unique type to use in its stead1. Let’s call this unique type type_0. You can pretend that this was defined somewhere like class type_0{};. Now we take our function template template <class T> void f(T) and substitute in type_0 for T. This gives us void f(type_0). The transformation is complete.

Step 2: Now that we have transformed template <class T> void f(T) into void f(type_0), we will perform deduction on (2) using the transformed function type. To do this, we imagine a call to (2) where the arguments have the type of the parameters for (1). Concretely, it would look like this:

template <class T> void func_2(T const*);
func_2(type_0{}); //derived from void f(type_0)

Would this call succeed? We can put it into our compiler to find out. GCC 8.1 says:

<source>: In function 'int main()':
<source>:4:18: error: no matching function for call to 'func_2(type_0)'
   func_2(type_0{});
                  ^
<source>:1:25: note: candidate: 'template<class T> void func_2(const T*)'
 template <class T> void func_2(T const*);
                         ^~~~~~
<source>:1:25: note:   template argument deduction/substitution failed:
<source>:4:18: note:   mismatched types 'const T*' and 'type_0'
   func_2(type_0{});
                  ^   

So deduction from (1) to (2) fails, because the invented type type_0 cannot be used to deduce const T*.

Step 3: Let’s try from (2) to (1). Again, we’ll transform (2) from template <class T> void f(T const*) to void f(type_0 const*).

Step 4: Now we attempt deduction:

template <class T> void func_1(T);
type_0 const* arg = nullptr;
func_1(arg);

This succeeds because a type_0 const* can be used to deduce T. Since deduction from (1) to (2) fails, but deduction from (2) to (1) succeeds, (2) is more specialized than (1) and will be chosen by overload resolution.


Let’s try a different example. How about:

template<class T> void g(T);  //(1)
template<class T> void g(T&); //(2)
int i = 0;
g(i);

(1) transforms to void g(type_0). Before we try deduction, we need to apply one of the numerous additional rules from the standard, which says we need to replace references with the type being referred to. So template <class T> void g(T&) becomes template <class T> void g(T). Deduction time:

template<class T> void func_2(T);
func_2(type_0{});

This succeeds.

Now the other direction. template<class T> void g(T&) transforms to void g(type_0&), then we remove the reference to get void g(type_0). Our second deduction:

template<class T> void func_1(T);
func_1(type_0{});

This is effectively identical to the previous one, so of course it succeeds.

Since deduction succeeded in both directions, the call is ambiguous. Sure enough, GCC diagnoses:

<source>: In function 'int main()':
<source>:5:8: error: call of overloaded 'g(int&)' is ambiguous
     g(i);
        ^
<source>:1:24: note: candidate: 'void g(T) [with T = int]'
 template<class T> void g(T);  //(1)
                        ^
<source>:2:24: note: candidate: 'void g(T&) [with T = int]'
 template<class T> void g(T&); //(2)
                        ^ 

This is why the algorithm is a partial ordering: sometimes two function templates are not ordered.


I’ll give one more example. This one has multiple parameters and is a bit more subtle.

template<class T>struct identity { using type = T; };
template<class T>struct A{};

template<class T, class U> void h(typename identity<T>::type, U); //(1)
template<class T, class U> void h(T, A<U>);                       //(2)
h<int>(0,A<void>{});

identity here just evaluates to its template argument, but the important thing to note is that typename identity<T>::type is a non-deduced context, so T cannot be deduced from the argument for that parameter.

(1) transforms to void h(typename identity<type_0>::type, type_0), which is void h(type_0, type_0). Attempt deduction on (2):

template<class T, class U> void func_2(T, A<U>);
func_2(type_0{}, type_0{});

This fails because we can’t match type_0 against A<U>.

(2) transforms to void h(type_0, A<type_1>). Try deduction against (1):

template<class T, class U> void func_1(typename identity<T>::type, U);
func_1(type_0{}, A<type_1>{});

This fails because typename identity<T>::type is a non-deduced context, so we can’t deduce T.

In the example from the last section deduction succeeded both ways so the call was ambiguous. In this example, deduction fails both ways, which is also an ambiguous call.


That’s the last of the examples. Of course, there are a bunch of rules which I didn’t cover here, like Concepts, parameter packs, non-type/template template parameters, and cases where both the argument and parameter types are references. Hopefully you now have enough of an intuition that you can understand what the standard says when you inevitably hit those corner cases. If you have any partial ordering conundrums, drop them down in the comments below or send them to me on Twitter.


  1. I’ll ignore non-type template parameters and template template parameters for simplicity, but the rules are essentially the same. 

Because your “competitors have it” IS NOT STRATEGY

iStock-514378725medium-2018-06-5-17-04.jpg

“We need a product that does X because our competitors have a product that does X”
“Our product needs feature Y because our competitors product has feature Y.”

It makes me want to cry.

Let me clear: building something because your competitors have it IS NOT A STRATEGY.

Neither is it a particularly good tactic.

Stop obsessing about your competitors and think about your customers.

I don’t doubt that your people are being told that customers are buying the competitor product because it has X or Y and I don’t doubt that some of your people feel that if you only matched the competitors feature for feature you would win but I just can’t see it myself.

For a start, is feature Y really the only thing loosing the sale? Are the products so well balanced that this one small thing is it? And is there really nothing that your product does better?

Try this simple experiment: tell the customer that feature Y will be delivered next month and see if they decide to buy yours there and then or find something else that makes the competition better.

Now lets suppose you decide to build Y. Before you make any plans ask yourself:

While you are building feature Y what are your competitors going to be doing?
Will they stand still or will they be adding feature Z?
And once they have feature Z will you need to play catch up?

Chances are that tomorrow you get to where you want to be (where your competitors are today) only to find your competitors have something else you don’t have either.

I’ll agree this is a good strategy if you have deliberately chosen to be a Fast Follower – you can play Android to your competitors iOS. Just make sure you know why your customers will choose your Android over the competitor iOS.

Will you be cheaper?
Or better?
Or will you bundle some other goodies with it?

Before you run to where your competitors are today ask yourself: where will your competitors be tomorrow?

If you still insist on building this feature you need to

  • Make sure you do a much better job (easier to use, more intuitive, faster to produce results, better quality results, or some such)
  • OR you need to do it fast and cheap so you can spend your precious resources on building something the competitor doesn’t have
  • OR you being overwhelming resources to the table so you are going to stand a chance. Every day you delay the competitor gets further ahead, so don’t try half measures

A better approach is to find out what your customers actually need. Stop looking at the features, go back to first principles: what is the problem your customers face? what is the job they are attempting to make progress with?

How can you help your customers with this job?
How can you make them faster?
How can you help them achieve their work more cheaply? Or at better quality? – in fact, what do “better” and “quality” look like to them.

Someone – I honestly forget who – told me earlier this year that they wanted to catch-up with their competitor and overtake them.

One small flaw there: if you build features to match your competitors you can never overtake them because you won’t know what to build once you reach parity.

Put it another way, you add all the features they have today, and all the features they add while you are catching up. What do you build next? Until they build their next version (and recapture the lead) you don’t know what to build. And if you build something different you just lost feature parity.

So, go back and examine what your customers are using your tool for. Look at the job to be done, look at how your customers are doing their job and using your tool and work out for yourself how you can help customers do a better job.

Celebrate the difference, explain why you are better.

And please forget about matching the competition.

I’m old enough to remember the days when WordStar was fighting WordPerfect, AmiPro was fighting them both, and all were better than Microsoft Word. Adverts and magazine reviews would compare them feature to feature. Someone somewhere thought people bought word processors based on the number of features.

Then Microsoft launched Windows and everybody went over to Microsoft Word for Windows almost overnight.

Don’t focus on your competitors. Focus on your customers. Unfortunately that requires more work and some original thinking.

The post Because your “competitors have it” IS NOT STRATEGY appeared first on Allan Kelly Associates.

Shooting with Flash (And Motofest 2018)

Talking about shooting with an on-camera flash at Coventry Motofest 2018.

I’ve never really used my flashgun before, mostly because on the odd occasion that I have used it at a shoot I ended up with under, or over exposed images and thus never wanted to risk wasting time and shots just to practice.

Last Saturday I decided to seriously try out my flash at an event where my photos were purely for myself, rather than with the intent to sell them.
I was shocked to find my results to be fantastic!

“The prettiest girl is riding in the ‘Stang!”

The day was probably quite appropriate for shooting with an on camera flash.
It was a little overcast and although bright-ish, things just were not popping very much.
So adding some light from a flash worked well.

The subjects seemed to work for flash too, shiny cars!

A row of MGs

I found that to get the exposure right I had to dial up the shutter speed or aperture so my camera was showing a couple 0.1 stops over exposure, otherwise the image would look too dark. Despite using i-TTL mode.
I wonder if this is because of using a diffuser and angling the flash at about 60 degrees, some of the light would be directed upwards and not hit the subject.

Using the flash without a diffuser was terrible, all images just had hugely over-exposed sections where the flash hit.
The regular hard-plastic diffuser that came with my flash is ‘fine’ — but the new one is certainly better. It results in a softer light that is much more directed towards the subject than the plastic one.
Instead, I used a diffuser I’d got from Amazon a few days before.

Flash comparison — Both same shutter, aperture and ISO. The left is lit with the new diffuser, the right with the hard plastic one.

From the above comparison it is clear to see that the new diffuser provides more light more widely spread across the image.

Example of a portrait photo where the flash is being bounced towards the subject well by rotating the diffuser.

The added benefit of this item is that it has a reflector in the back on the inside — allowing me to easily shoot portrait with on-camera flash and just rotate the diffuser so that light is still being bounced toward the subject from the front. Rather than the light ended up hitting the left or right side of the subject.

Below are a bunch of my favourite images from the event. But I got about 100 good ones.

It was a lot of fun being able to try out new equipment without the pressure of having to produce good images.

I now feel confident to get good photos with an on-camera flash.

Shooting different subject for once was also fun! When shooting static subjects one has a lot more time to choose the composition of an image. Although I note now that most of my images see the subject square in the centre of the frame.

60% of the time, works every time
Whatever level of zip ties you’re on, you’re not on this level of zip ties.
Lego block engine cover in a Nissan Cube
An Aston Martin V8 Vantage in Gulf livery
More Gulf livery, this time on a VW Golf.
Jacuar
I also got to try out my 100mm macro lens — although it is certainly too long a focal length for shooting cars.
Ford V8 with Holley carburettor.
I call this one “I like to chop up pedestrians” — also thats the smallest number plate I’ve ever seen.
More lights = Better
Skele says Hi.

Free books and other news

3Books-2018-06-4-10-31.jpg

Many of you are reading this because you signed-up for a free copy of my Xanpan book.

Thank you so much! – I hope you are enjoying my thoughts, reflections and tips.

Now, can I ask a favour, please? – a few minutes of your time.

If you have a copy of Xanpan would you mind writing me an Amazon review? (thats .com, Amazon UK has a separate list of reviews – yes, it is a pain).

Please, please, please 🙂

Amazon reviews make a big difference to sales and I’d be most grateful. (Even more so for 5-star reviews!)

And I will happily give a free review copy of “Little Book of Requirements and User Stories” to anyone who would like to review that book too – mail me, allan@allankelly.net. (And if you already have a copy of Little Book please suggest some other way I can thank you for your review.)

I’m also working on getting Continuous Digital and Project Myopia onto Amazon. Both will get new professional covers and a proper copy edit

Finally, as some of you know, I’ve started writing a companion to Little Book: Product Ownership. Again I’m using the LeadPub system so you can buy the book now and get free updates as I add to the book and edit it. And I am most grateful to those of you who have already bought Product Ownership.

The post Free books and other news appeared first on Allan Kelly Associates.

EDG and Github are both logical purchases for Microsoft

It looks like my prediction that Microsoft buys Github may be about to come true.

Microsoft has been sluggish in integrating their LinkedIn purchase into their identity management system. Lots of sites have verify identity using Github options (or at least the kind of sites I visit do), so perhaps LinkedIn identity will be trialed via Github.

A Github purchase will also allow Microsoft to directly connect lots of developers to Azure. Being able to easily build and execute Github code on Azure is the bait, customer data is where the money is; making Github more data friendly is an obvious first priority for new owners.

Who else should Microsoft buy? As a protective move, I think they should snap up Edison Design Group (EDG) before somebody else does. Readers outside of the compiler/static analysis/C++ standards world are unlikely to have heard of EDG. They sell C/C++ front ends (plus other languages) that support all the historical features/warts supported by other C/C++ compilers. The features only found in Microsoft’s compilers is what make it very costly/time-consuming for many companies to port their applications to other platforms; developer use of Microsoft compiler dependent features is a moat that makes it difficult for many companies to leave the Microsoft ecosystem. EDG have been in the business a long time and have built up an extensive knowledge of vendor specific compiler features; the kind of knowledge that can only be obtained by having customers tell you what language constructs they are using that your current product does not handle (and what those constructs actually mean).

What would happen if a very large company bought EDG, and open sourced its code (to make it easier for Windows developers to switch platforms, not to make any money off compiler related tools)? Somebody would have to bolt on a back-end, to generate code; but that would not be hard (EDG have designed their product to make this easy). A freely available compiler, supporting all/most of the foibles of the Microsoft C++ compiler, would tempt many Windows only developers to give it a go. A free compiler removes management from the loop, developers can try things out as a side project, without having to get management approval to spend money on a compiler (from practical experience I know how hard it is to sell compatible compiler products, i.e., there is no real money to be made by anybody doing this commercially).

Is this risk, to Microsoft, really worth the (relatively) low cost of buying EDG? The EDG guys are not getting any younger, why wouldn’t they be willing sell?

Chalk The Lines – a.k.

Given a set of points (xi,yi), a common problem in numerical analysis is trying to estimate values of y for values of x that aren't in the set. The simplest scheme is linear interpolation, which connects points with consecutive values of x with straight lines and then uses them to calculate values of y for values of x that lie between those of their endpoints.
On the face of it implementing this would seem to be a pretty trivial business, but doing so both accurately and efficiently is a surprisingly tricky affair, as we shall see in this post.

Gitlab certificates

On Ubuntu, cloning a repo from a machine you don't have a certificate for will give the error:

fatal: unable to access 'https://servername': server certificate verification failed. CAFuile /etc/ssl/certs/your_filename CRLfile: None

You can work around this by tell git clone not to use the certificate e.g.

git config --system http.sslverify false


which is asking for trouble. However you can install the certificate, so you don't need to keep doing this. 
Using an answer here: https://stackoverflow.com/questions/21181231/server-certificate-verification-failed-cafile-etc-ssl-certs-ca-certificates-c  looks to have worked, by trying things one step at a time:
hostname=gitlab.city.ac.uk
port=443
trust_cert_file_location=`curl-config --ca`
sudo bash -c "echo -n | openssl s_client -showcerts -connect $hostname:$port \
    2>/dev/null  | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'  \
    >> $trust_cert_file_location"
I did try this first – so errors don’t end up in dev null:

openssl s_client -showcerts -connect $hostname:$port


Also, I first got the error sed: unrecognised option '--ca'
It took a moment to realise the --ca came from curl-config, which I needed to install.

The age of the Algorithm is long gone

I date the age of the Algorithm from roughly the 1960s to the late 1980s.

During the age of the Algorithms, developers spent a lot of time figuring out the best algorithm to use and writing code to implement algorithms.

Knuth’s The Art of Computer Programming (TAOCP) was the book that everybody consulted to find an algorithm to solve their current problem (wafer thin paper, containing tiny handwritten corrections and updates, was glued into the library copies of TAOCP held by my undergraduate university; updates to Knuth was news).

Two developments caused the decline of the age of the Algorithm (and the rise of the age of the Ecosystem and the age of the Platform; topics for future posts).

  • The rise of Open Source (it was not called this for a while), meant it became less and less necessary to spend lots of time dealing with algorithms; an implementation of something that was good enough, was available. TAOCP is something that developers suggest other people read, while they search for a package that does something close enough to what they want.
  • Software systems kept getting larger, driving down the percentage of time developers spent working on algorithms (the bulk of the code in commercially viable systems deals with error handling and the user interface). Algorithms are still essential (like the bolts holding a bridge together), but don’t take up a lot of developer time.

Algorithms are still being invented and some developers spend most of their working with algorithms, but peak Algorithm is long gone.

Perhaps academic researchers in software engineering would do more relevant work if they did not spend so much time studying algorithms. But, as several researchers have told me, algorithms is what people in their own and other departments think computing related research is all about. They remain shackled to the past.

Emacs 26.1 has been released (and it’s already on Homebrew)

Saw the announcement on on the GNU Emacs mailing list this morning. Much to my surprise, it’s also already available on homebrew. So my Mac is now sporting a new fetching version of Emacs as well :). I’ve been running the release candidate on several Linux machines already and was very happy with it, so […]

The post Emacs 26.1 has been released (and it’s already on Homebrew) appeared first on The Lone C++ Coder's Blog.

DSLinux on a DSLite with an M3DS Real card and SuperCard SD

DSLinux running on a Nintendo DSLite

I recently bought a gorgeous pink Nintendo DSLite with the sole purpose of running DSLinux on it.
When I posted about my success on Mastodon , someone helpfully asked “Has it have any use tho?”.
Lets answer that right away: Running Linux on a Nintendo DSLite is at best a few hours entertainment for the masochistic technologist, and at worst a waste of your time.

Running Linux on a Nintendo DSLite is at best a few hours entertainment for the masochistic technologist, and at worst a waste of your time.

But, I do rather enjoy running Linux on things that should not be running Linux, or at least attempting to do so. So heres what I did!

Hardware:

  • Nintendo DSLite
  • SuperCard SD (Slot 2)
  • M3DS Real (Slot 1)
  • R4 Card (Knockoff, says R4 SDHC Revolution for DS on the card)

DSLinux runs on a bunch of devices, luckily we had some R4 cards and an M3DS Real around the place which are both supported by DSLinux.
I purchased a SuperCard SD from Ebay to provide some extra RAM, which apparently is quite useful, since the DSLite has only 2mB of it on it’s own.The SuperCard SD I bought had 32mB extra RAM bringing the total up to some 34mB, wowee.

R4 Cards

The first cards I tried were the R4 cards we had.
They’re popular and supported by DSLinux. Unfortunately, it seems the ones we’ve got are knockoffs and therefore proved challenging to find firmware for.
I spent a long while searching around the internet and trying various firmwares for R4 cards — None of them I tried did anything except show the Menu? screen on boot.

Finally, finding this post on GBATemp.net from a user with a card that looks exactly the same as mine lead me to give up on the R4 card and move on to the M3DS Real. Although the post did prove useful later.

It should be noted that the R4 card I had had never been tested anyway, so it might never have worked.

M3DS Real

Another card listed as supported on the DSLinux site, so seemed a good one to try.
We had a Micro-SD Card in the M3Real anyway, with the M3 Sakura firmware on it so it seemed reasonable to just jump in there.

I copied the firmware onto another SD Card (because we didn’t want to loose the data on the original card). It was only 3 folders, SYSTEM, NDS and SKINS in the root of the card. The NDS file containing ‘games’.

In this case, I put the DSLinux files (dslinux.nds, dslinuxm.nds and ‘linux’, a folder) into the NDS folder and stuck it in my DSLITE.

After selecting DSLinux from the menu, I got the joy of….a blank screen.

Starting DSLinux from M3 Sakura results in a white screen

Some forum posts which are the first results when searching the issue on DuckDuckGo suggest that something called DLDI is the issue.

The DSLinux ‘Running DSLinux’ does mention patching the ‘dslinux.nds’ file with DLDI if the device one is using doesnt support auto-dldi. At the time this was all meaningless jargon to me, since I’ve never done any Nintendo DS homebrew before.

Turns out, DLDI is a library that allows programs to “read and write files on the memory card inserted into one of the system’s slots”.
Homebrew games must be ‘patched’ for whatever device you’re using to allow them to read/write to the storage device.
Most of the links on the DSLinux page to DLDI were broken, but we descovered the new home of DLDI and it’s associated tools to be www.chishm.com/DLDI/ .

I patched the dslinux.nds file using the linux command line tool and saw no change to the behaviour of the DSLite, still white screens.

Upon reading the DSLinux wiki page for devices a little closer, I noticed that the listing for the M3DS Real notes that one should ‘Use loader V2.7d or V2.8’.

What is a loader??
It means the card’s firmware/menu.

Where do I find it?
On the manufacturer’s website, or, bringing back the post mentioned earlier with the R4 card user on GBATemp.net, one can find lots of firmware’s for lots of different cards here: http://www.linfoxdomain.com/nintendo/ds/

Under the listing on the above site for ‘M3/G6 DS Real and M3i Zero’ one can find a link to firmware versions V2.7d and V2.8 listed as ‘M3G6_DS_Real_v2.8_E15_EuropeUSAMulti.zip’.

Upon installing this firmware to the SD Card (by copying ‘SYSTEM’ folder to the root of a FAT32 formatted card, I extracted the DSLinux files again (thus, without the DLDI patching I’d done earlier) and placed the files ‘dslinux.nds, ‘dslinuxm.nds’ and the folder ‘linux’ to an ‘NDS’ folder, also in the root of the drive.
This is INCORRECT.
Upon loading the dslinux.nds file through the M3DS Real menu it did indeed boot Linux, but dropped me into a single-user mode, with essentially no binaries in the PATH.
This is conducive to the Linux kernel having booted successfully, but not being able to find any userland. Hence the single-user and lack of programs.

Progress at least!

I re-read the DSLinux instructions and caught the clear mention to ‘Both of these must be extracted to the root directory of the CF or SD card.’ when talking about the DSLinux files.

Upon moving the DSLinux files to the root of the directory and starting ‘dslinux.nds’ from the M3DS Real menu I had a working Linux system!!

I type `uname -a` on DSLinux. Its running Kernel 2.6.

Notice the ‘DLDI compatible’ that pops up when starting DSLinux — That means that the M3DS Real auto-patches binaries when it runs them. Nice.

What Next?

Probably trying to compile a newer kernel and userspace to start with.
Kernel 2.6, at time of writing, is 2 major versions out of date.

After that, I’d like to understand how DSLinux is handling the multiple screens and multiple processors.
The DS has an ARM7 and an ARM9 processor and two screens, which I think are not connected to the same processor, the buttons are split between the chips too.

Lastly, I’d like to write something for linux on the DS.
Probably something silly, but I’d like to give it a try!

Don’t ask me questions about DSLinux, I don’t really know anything more than what I’ve mentioned here. I just read some Wiki’s, solved some problems and did some searching.

Thanks to the developers of DSLinux and DLDI for making this silliness possible.

Test the Code, Not the Mock

About 18 months or so ago I wrote a post about how I’d seen tests written that were self-reinforcing (“Tautologies in Tests”). The premise was about the use of the same production code to verify the test outcome as that which was supposedly under test. As such any break in the production code would likely not get picked up because the test behaviour would naturally change too.

It’s also possible to see the opposite kind of effect where the test code really becomes the behaviour under test rather than the production code. The use of mocking within tests is a magnet for this kind of situation as a developer mistakenly believes they can save time [1] by writing a more fully featured mock [2] that can be reused across tests. This is a false economy.

Example - Database Querying

I recently saw an example of this in some database access code. The client code (under test) first configured a filter where it calculated an upper and lower bound based on timestamps, e.g.

// non-trivial time based calculations
var minTime = ...
var maxTime = ...

query.Filter[“MinTime”] = minTime;  
query.Filter[“MaxTime”] = maxTime;

The client code then executed the query and performed some additional processing on the results which were finally returned.

The test fixture created some test data in the form of a simple list with a couple of items, presumably with one that lies inside the filter and another that lies outside, e.g.

var orders = new[]
{
  new Order { ..., Timestamp = “2016-05-12 18:00:00” },
  new Order { ..., Timestamp = “2018-05-17 02:15:00” },
};

The mocked out database read method then implemented a proper filter to apply the various criteria to the list of test data, e.g.

{
  var result = orders;

  if (filter[“MinTime”])
    ...
  if (filter[“MaxTime”])
    ...
  if (filter[...])
    ...

  return result;
}

As you can imagine this starts out quite simple for the first test case but as the production code behaviour gets more complex, so does the mock and the test data. Adding new test data to cater for the new scenarios will likely break the existing tests as they all share a single set and therefore you will need to go back and understand them to ensure the test still exercises the behaviour it used to. Ultimately you’re starting to test whether can actually implement a mock that satisfies all the tests rather than write individual tests which independently validate the expected behaviours.

Shared test data (not just placeholder constants like AnyCustomerId) is rarely a good idea as it’s often not obvious which piece of data is relevant to which test. The moment you start adding comments to annotate the test data you have truly lost sight of the goal. Tests are not just about verifying behaviour either they are a form of documentation too.

Roll Back

If we reconsider the feature under test we can see that there are a few different behaviours that we want to explore:

  • Is the filter correctly formed?
  • Are the query results correctly post-processed?

Luckily the external dependency (i.e. the mock) provides us with a seam which allows us to directly verify the filter configuration and also to control the results which are returned for post-processing. Consequently rather than having one test that tries to do everything, or a few tests that try and cover both aspect together we can separate them out, perhaps even into separate test fixtures based around the different themes, e.g.

public static class reading_orders 
{
  [TestFixture]
  public class filter_configuration    
  ...    
  [TestFixture]
  public class post_processing    
  ...
}

The first test fixture now focuses on the logic used to build the underlying query filter by asserting the filter state when presented to the database. It then returns, say, an empty result set as we wish to ignore what happens later (by invoking as little code as possible to avoid false positives).

The following example attempts to define what “yesterday” means in terms of filtering:

[Test]
public void filter_for_yesterday_is_midnight_to_midnight()
{
  DateTime? minTime = null;
  DateTime? maxTime = null;

  var mockDatabase = CreateMockDatabase((filter) =>
  {
    minTime = filter[“MinTime”];
    maxTime = filter[“MaxTime”];
  });
  var reader = new OrderReader(mockDatabase);
  var now = new DateTime(2001, 2, 3, 9, 32, 47);

  reader.FindYesterdaysOrders(now);

  Assert.That(minTime, Is.EqualTo(
                new DateTime(2001, 2, 2, 0, 0, 0)));
  Assert.That(maxTime, Is.EqualTo(
                new DateTime(2001, 2, 3, 0, 0, 0)));
}

As you can hopefully see the mock in this test is only configured to extract the filter state which we then verify later. The mock configuration is done inside the test to make it clear that the only point of interest is the the filter’s eventual state. We don’t even bother capturing the final output as it’s superfluous to this test.

If we had a number of tests to write which all did the same mock configuration we could extract it into a common [SetUp] method, but only if we’ve already grouped the tests into separate fixtures which all focus on exactly the same underlying behaviour. The Single Responsibility Principle applies to the design of tests as much as it does the production code.

One different approach here might be to use the filter object itself as a seam and sense the calls into that instead. Personally I’m very wary of getting too specific about how an outcome is achieved. Way back in 2011 I wrote “Mock To Test the Outcome, Not the Implementation” which showed where this rabbit hole can lead, i.e. to brittle tests that focus too much on the “how” and not enough on the “what”.

Mock Results

With the filtering side taken care of we’re now in a position to look at the post-processing of the results. Once again we only want code and data that is salient to our test and as long as the post-processing is largely independent of the filtering logic we can pass in any inputs we like and focus on the final output instead:

[Test]
public void upgrade_objects_to_latest_schema_version()
{
  var anyTime = DateTime.Now;
  var mockDatabase = CreateMockDatabase(() =>
  {
    return new[]
    {
      new Order { ..., Version = 1, ... },
      new Order { ..., Version = 2, ... },
    }
  });
  var reader = new OrderReader(mockDatabase);

  var orders = reader.FindYesterdaysOrders(anyTime);

  Assert.That(orders.Count, Is.EqualTo(2));
  Assert.That(orders.Count(o => o.Version == 3),
              Is.EqualTo(2));
}

Our (simplistic) post-processing example here ensures that all re-hydrated objects have been upgraded to the latest schema version. Our test data is specific to verifying that one outcome. If we expect other processing to occur we use different data more suitable to that scenario and only use it in that test. Of course in reality we’ll probably have a set of “builders” that we’ll use across tests to reduce the burden of creating and maintaining test data objects as the data models grow over time.

Refactoring

While reading this post you may have noticed that certain things have been suggested, such as splitting out the tests into separate fixtures. You may have also noticed that I discovered “independence” between the pre and post phases of the method around the dependency being mocked which allows us to simplify our test setup in some cases.

Your reaction to all this may well be to suggest refactoring the method by splitting it into two separate pieces which can then be tested independently. The current method then just becomes a simple composition of the two new pieces. Additionally you might have realised that the simplified test setup probably implies unnecessary coupling between the two pieces of code.

For me those kind of thoughts are the reason why I spend so much effort on trying to write good tests; it’s the essence of Test Driven Design.

 

[1] My ACCU 2017 talk “A Test of Strength” (shorter version) shows my own misguided attempts to optimise the writing of tests.

[2] There is a place for “heavier” mocks (which I still need to write up) but it’s not in unit tests.

Visual Lint 6.5.2.295 has been released

This is a recommended maintenance update for Visual Lint 6.5. The following changes are included:
  • Added basic support for Qt Creator projects (.pro/.pro.user files). Note that the implementation does not yet support subprojects or read preprocessor and include folder properties. As such, if the analysis tool you are using requires preprocessor or include folders to be defined (as PC-lint and PC-lint Plus do) for the time being they must be defined manually (e.g. written as -D and -i directives within a PC-lint/PC-lint Plus std.lnt indirect file).
  • The "Analysis Tool" Options page now recognises PC-lint Plus installations containing only a 64 bit executable if the "Use a 64 bit version of PC-lint if available" option is set.
  • When the PC-lint Plus installation folder is selected in the "Analysis Tool" Options page the PC-lint Plus manual (<installation folder>/doc/manual.pdf) is now correctly configured.
  • Added a workaround to the Eclipse plug-in for an issue identified with some Code Composer Studio installations which source plug-in startup and shutdown events in different threads.
  • Fixed a crash which affected some machines when the "Analysis Tool" Options page was activated when PC-lint was the active analysis tool.
  • Fixed a bug which caused the Visual Studio plug-in to be incorrrectly configured in Visual Studio 2017 v15.7.
  • Fixed a bug which could cause the PC-lint/PC-lint Plus environment file to reset to "Defined in std.lnt".
  • Updated the "Example PC-lint/PC-lint Plus project.lnt file" help topic and those relating to supported project types.
Download Visual Lint 6.5.2.295

Windows batch files

I've been writing a batch file to run some mathematical models over a set of inputs.
The models are software reliability growth models, described here.

We are using
  • du: Duane
  • go: Goel and Okumto
  • jm: Jelinski and Moranda
  • kl: Keiller and Littlewood
  • lm: Littlewood model
  • lnhpp: Littlewood non-homogeneous Poisson process
  • lv: Littlewood and Verrall
  • mo: Musa and Okumoto
Littlewood appears many times: he founded the group where I currently work. 

So, far too much background. I have one executable for each model, after making a make file; yet another story. And a folder of input files, named as f3[some dataset]du.dat, f3[some dataset]go.dat,... f3[some dataset]mo.dat. I also have some corresponding output files someone else produced a while ago, so in theory I can check I get the same numbers.I don't but that's going to be yet another story.

You can also use the original file and generated file to recalibrate, giving yet another file. Which I have previously generated results from. Which also don't match. 

I wanted to be able to run this on Ubuntu and Windows, and managed to make a bash script easily enough. Then I tried to make a Windows batch file to do the same thing. I'll just put my final result here, and point out the things I tripped up on several times.


ECHO OFF
setlocal EnableDelayedExpansion
setlocal 


for %%m in (du go jm kl lm lnhpp lv mo) do (
  echo %%m
  for %%f in (*_%%m.dat) do (
    echo %%~nf
    set var=%%~nf
    echo var: !var!
    set var=!var:~2!
    echo var now: !var!

    swrelpred\%%m.exe %%~nf.dat "f4!var!"
    swrelpred\%%mcal.exe %%~nf.dat "f4!var!" "f9!var!"
  )
)


1. First, turn the echo off because there's way too nosie otherwise.
2. Next, enable delayed expansion, otherwise things in blocks get expanded on sight and therefore don't change in the loop: "Delayed expansion causes variables delimited by exclamation marks (!) to be evaluated on execution"  from stack exchanges' Superuser site
3. Corollary: Use ! in the variables in the block not % for delayed expansion.
4.  But we're getting ahead of ourselves. The setlocal at the top means I don't set the variables back at my prompt. Without this, as I changed my script to fix mistakes it did something different between two runs, since a variable I had previously set might end up being empty when I broke stuff.
5. "Echo is off" spewed to the prompt means I was trying to echo empty variables, so the var: etc tells me which line something is coming from.
6. !var:~2! gives me everything from the second character, so I could drop the f3 at the start of the filename and make f4 and f9 files to try a diff on afterwards. Again pling for delayed expansion.




I suspect I could improve this, but it's six importnat things to remember another time.

Writing this in Python might have been easier. Or perhaps I should learn Powrshell one day.