Author: Andy Balaam

My Address Book 1.9.0 – rewritten from scratch

Ordinarily, my motivations for doing open source work are clear: peer recognition and the satisfaction of knowing people are using my work.

However, I’ve been distracted from that stuff recently because of my desire to scratch my own itch, by re-writing My Address Book from scratch to allow sharing the same address book between the web interface and the various email programs I use on different computers.

The only way to make it work with those email programs is to store the data in an LDAP server, instead of the custom MySQL database I had used in the original version (which is still available, of course: My Address Book version 1).

While I was doing it, I took the opportunity (or made the mistake) of re-writing in my favourite language, Python. Joel would not approve, but that’s the fun of open source: I can do whatever I like, no matter how unwise.

I learned a lot of things on the way:

  1. I like web.py. It is a simple, and helpful library in the Python tradition of being concise but powerful.
  2. Setting up Python running as FastCGI on either Apache or lighttpd is much harder than creating a page of PHP. However, once it’s done, it’s done. (The results of my research are shown at the bottom of the Install Guide.)
  3. Setting up an LDAP server to act as a little address book is unbelievably complicated. See the Install Guide for how to do it. One day, I or someone else will turn all those instructions into the preinstall step of a .deb, and no-one will ever have to worry about it again. Volunteers, please.
  4. Templetor, web.py’s templating system is fine, but too slow for large pages. I had to reimplement the main address list rendering in plain Python, which made me sad.
  5. I am compulsive about getting my work out into the world.

It has been a struggle to write all the documentation and think through the installation procedure and all the other stuff that comes with making a public release, but I have been unable to move on to other things until I have got it done. I hope I’ve done an ok job – the installation procedure is far too complex and should be automated, but if the volume of documentation is any indicator of how helpful it is, it should be reasonable.

I think I found it more difficult than normal because, unlike normal, getting it out to other people was not my main motivation for doing the project. I’m glad I’ve worked through it though, and I really hope some people get interested enough to help me make it much easier to use.

I think this project is a good example of how it can be much better to live in the open source world than the proprietory one. If I want to create a small address book for my family in the open source world, I end up with an industrial-strength LDAP server providing it, meaning that so long as someone does the work to make it simple to use for my simple use case, it can scale to solve pretty much any problem I might have in the future. So if I start a small business that eventually grows and needs to track millions of addresses, I can keep using the same LDAP server, on the same hardware, as my original address book.

Of course, My Address Book would have broken long before that. It doesn’t even do paging, and the home page lists all your addresses in one page.

I must write a rant one day about how I hate paging.

Switching workspace in GNOME via the command line

I use rdesktop to connect to some Windows machines, and it works beautifully. I like to allow it to grab keyboard input so I can switch and close windows on my Windows desktop without fear of accidentally doing something to the rest of my Linux desktop.

However, I have never found a way of letting some keystrokes escape – specifically Ctrl-Alt-Left, Right, Up, Down so that I can switch workspaces (virtual desktops) away from the workspace containing rdesktop.

I have finally found what I hope is a good workaround – I have installed easystroke and defined some mouse gestures I can use to switch desktop. These mouse gestures are not swallowed by rdesktop, so they still work, even when it has focus.

When I asked easystroke to send the Ctrl-Alt-Left etc. keystrokes, they got swallowed by rdesktop, so all I needed to be able to do complete the story was a command-line tool to switch workspace.

That turned out to be trickier than you might imagine, but here is my solution, using the amazing wmctrl.

This should work on GNOME with the metacity window manager, and the standard GNOME workspace switcher. It should be easy to adapt to other window managers and workspace tools.

First, install some tools. On Ubuntu you need:

sudo apt-get install bc wmctrl coreutils grep bash

Now create a file with gedit ~/bin/workspace-switcher & and edit it to look like this:

#!/bin/bash

CMD="$1"

NUM_WORKSPACES=`gconftool-2 --get /apps/metacity/general/num_workspaces`
NUM_COLS=`gconftool-2 --get /apps/panel/applets/workspace_switcher_screen0/prefs/num_rows`

NUM_ROWS=`echo "$NUM_WORKSPACES / $NUM_COLS" | bc`

CURRENT_WS=`wmctrl -d | grep \* | cut -d " " -f 1`

MOVE_LEFT="- $NUM_ROWS"
MOVE_RIGHT="+ $NUM_ROWS"
MOVE_UP="-1"
MOVE_DOWN="+1"

case $CMD in

"Left" )
	NEW_WS=`echo $CURRENT_WS "-" $NUM_ROWS | bc`
	if [[ $NEW_WS -lt 0 ]]; then NEW_WS=$CURRENT_WS; fi
	;;

"Right" )
	NEW_WS=`echo $CURRENT_WS "+" $NUM_ROWS | bc`
	if [[ $NEW_WS -ge $NUM_WORKSPACES ]]; then NEW_WS=$CURRENT_WS; fi
	;;

"Up" )
	WS_COL=`echo $CURRENT_WS "%" $NUM_ROWS | bc`
	if [[ $WS_COL -eq 0 ]]; then
	{
		NEW_WS=$CURRENT_WS
	}
	else
	{
		NEW_WS=`echo $CURRENT_WS "- 1" | bc`
	}; fi
	;;

"Down" )
	NEW_WS=`echo $CURRENT_WS "+ 1" | bc`
	NEW_WS_COL=`echo $NEW_WS "%" $NUM_ROWS | bc`
	if [[ $NEW_WS_COL -eq 0 ]]; then NEW_WS=$CURRENT_WS; fi
	;;

* )
	NEW_WS=$CMD

esac

wmctrl -s $NEW_WS

Make it executable with chmod +x ~/bin/workspace-switcher and make sure ~/bin is in your PATH.

You can run it like this:

switch-workspace Left

to move left – the other possiblities are, obviously, Right, Up and Down.

Or like this:

switch-workspace 3

to move to a workspace by number.

If you want to use it with easystroke, create an action, and for the command simply enter switch-workspace Left and similar as the command.

Easystroke can be installed on Ubuntu like this:

sudo apt-get install easystroke

Anatomy of an interpreter: the Evaluator

Posts in this series: Lexer, Parser, Evaluator

I’m still really enjoying writing my Scheme interpreter Subs, which can now succesfully run all the example code from SICP up to section 2.3.4. I’ve made the changes I mentioned I would in the Lexer article, so now the Lexer returns Tokens that contain information about their basic types, and I’ve gone through a significant refactoring to replace one of the several massive switch statements with a virtual function call (Martin Fowler would be proud).

Last time I explained how the Parser takes the stream of tokens coming from the Lexer and returns a hierarchical tree of Values, each of which represents an operation or thing in the program.

The Evaluator takes in a tree of Values, “evaluates” it, and returns another Value object, which is the answer. The Evaluator class is by far the most complex part of Subs, so in this post we’ll start with an overview of how it works. Future posts will break down the different parts in more detail.

The most interesting parts of the Evaluator class interface look like this:

class Evaluator
{
public:
    std::auto_ptr<Value> EvalInContext( const Value* value,
        boost::shared_ptr<Environment>& environment );
};

The EvalInContext method takes in a Value to evaluate, and an “environment” *in which to evaluate it. Note that the in the real code it takes a couple more arguments, including a mysterious and annoying boolean called is_tail_call which will be explained later.

* More on environments later. All you need to know for now is that they provide a way of keeping hold of all the things we currently know about, identified by name.

A very simplified version of EvalInContext would look like this:

std::auto_ptr<Value> Evaluator::EvalInContext( const Value* value,
    boost::shared_ptr<Environment>& environment )
{
    if( is_symbol( value ) )
    {
        return eval_symbol( value, environment );
    }

    if( !is_combination( value ) )
    {
        return auto_ptr<Value>( value->Clone() );
    }

    const CombinationValue* combo = to_combination( value );
    CombinationValue::const_iterator it = combo->begin();

    auto_ptr<Value> evaldoptr = EvalInContext( *it, environment );

    if( special_symbol( evaldoptr ) )
    {
        return process_special_symbol( evaldoptr, combo );
    }
    else
    {
        ++it;

        CombinationValue argvalues;
        for( ; it != combo->end(); ++it )
        {
            argvalues.push_back( EvalInContext( *it, environment ).release() );
        }

        return run_procedure( evaldoptr.get(), &argvalues, *this, environment );
    }
}

If the Value to be evaluated is just a symbol, we call eval_symbol which basically looks up the symbol’s name in the environment and returns the value it finds.

If the Value is not a combination (i.e. the root of a tree of other values) it must be a basic type such as a string or an integer. In this case we simple copy the Value and return it.

Otherwise, it’s a combination. To evaluate a combination, we follow the “eval-apply” pattern. The principle is to evaluate all the Values in the combination separately, and then “apply” (run) the first value (the “operator”) as a procedure, using the other values as arguments. The first value must evaluate to something that is recognisable as a procedure, or this doesn’t make sense and we will throw an error.

In practice it’s a tiny bit more complicated. We evaluate the first Value in the combination (by calling EvalInContext recursively), then we check whether it’s a special symbol such as if or let and if so, deal with it separately. Otherwise, we evaluate all the other Values (calling EvalInContext recursively again) and put them into a new CombinationValue, and pass the operator and the arguments to run_procedure, which looks something like this:

std::auto_ptr<Value> run_procedure( const Value* operator,
    const CombinationValue* args, Evaluator& ev,
    boost::shared_ptr<Environment>& environment )
{
    if( is_builtin_procedure( operator ) )
    {
        return handle_builtin_procedure( operator, args, environment );
    }
    else
    {
        std::auto_ptr<Value> ret;

        const CompoundProcedureValue* proc = to_compound_procedure( operator );

        boost::shared_ptr<Environment> new_env =
            proc->ExtendEnvironmentWithArgs( args );

        for( CombinationValue::const_iterator it = proc->GetBody()->begin();
            it != proc->GetBody()->end(); ++it )
        {
            ret = ev.EvalInContext( *it, new_env );
        }

        return ret;
    }
}

Running a procedure means either doing something built-in (for example adding up two numbers and returning the result) or evaluating some other code, which comes from the definition of the procedure being run. First we call ExtendEnvironmentWithArgs to create a new Environment, which contains the argument Values that were supplied, and then we loop through all the sections of the body of the procedure, evaluating each one. Note that we throw away the returned Values for all sections except the last one (this is how Scheme works).

Once we’ve evaluated and applied our procedure, which of course potentially includes numerous recursive calls to EvalInContext, we end up with a Value that is returned, and we’re done.

Simple eh?

But now I must make a confession: almost everything I have written above is lie. Why would I lie to you? Because I missed out something wonderful and strange called “tail-call optimisation”. I’ll explain next time.

How to ask technical questions in person

In a healthy team performing a technical task, there will be a lot of questions. Those questions will sometimes be asked by those with less technical knowledge, but (in a healthy team) there will be plenty of questions going back the other way too. Questions that come up in the normal course of your work are the main way knowledge is going to be transferred amongst your team members. They are to be encouraged, and you should never feel like a nuisance, or a technical wimp, because you’re asking them.

It’s easy to imagine, when you’re asking someone a question, that whether you get a good answer depends mainly on the person you’re asking.

In fact, an awful lot depends on you.

To get a good answer to a question, you need to prepare, to manage the Zone, and above all, to make the person you are asking feel clever.

What follows are some tips on how to ask a technical person a technical question when you’re in the same room as them. Some of them may also apply to asking on the phone, or even via instant messaging.

1. Prepare

You’re about to ask someone a question.

Breathe.

Think about what you are hoping to find out. This might sound obvious, but when you’re in the thick of trying to solve a problem, you may find you’re actually quite confused about what you’re doing. If you can, you want to avoid wasting the other person’s time with your confusion. Even worse, you might confuse them, and then they won’t feel clever.

This doesn’t need to take long – it may take half a second, but just let it cross your mind – what do I want to find out?

Think about the other person: what are they doing right now? Is it related? What do they know about? From what angle will they be approaching your question? Again, this will probably take almost no time at all, but it will help you ask your question well.

Now you need to zoom out. You are deep into a problem, so deep in fact that you’re stuck on a very specific part of it. This part of it is burning into your consciousness, obscuring the wider picture of what you’re doing. The person you’re asking is going to need that wider picture, so remind yourself what it is.

Once you’ve zoomed out, you will need to engage with the Zone.

2. Manage the Zone

The biggest problem you have when asking a technical person a technical question is that they may well be in the Zone. The Zone is a state of intense concentration in which you get really good work done (Rands elaborates in A Nerd in a Cave). The thing about the Zone is that it is difficult (sometimes even painful) to come out of.

If the person you’re asking is in the Zone, they will need to be coaxed out. When you start talking, even if they turn and look at you and say something encouraging, they are somewhere far away, solving a problem. Your problem is an unwelcome intrusion.

The first lesson about the Zone is: don’t say much at first.

When you start talking the other person doesn’t know whether you’re going to offer them a cup of tea, talk politics, or ask them how to implement quicksort. They haven’t disengaged at all from the problem they are working on. You need to let them know you are going to ask them a technical question, and start the painful process of pulling them out of their problem, and into yours.

Often the best thing to do is pick a single word or phrase that lets them know the general area you are talking about: for example, “import”, or “the C++ standard”, or “garbage collection”. If you say something like that, and watch their eyes, you can almost see the glaze of the Zone lifting.

If you can see they’re still elsewhere, give them a second – maybe they’ll even say something like, “I just want to finish this line.” If they do, let them – step back, take the pressure off – they will be much better able to answer you when the thing they need to remember is written down.

Now your job is to zoom them in on your problem, at their pace. It’s very important to watch and listen to the other person while you do this. Give them a problem area, and check they understood what you meant. Now give them closer context – maybe the directory you’re talking about, or the DLL you’re working on. Now tell them what class you’re editing, or what file.

As soon as you can see it’s needed, be quick to show them the exact thing you’re doing on your screen or paper – that way they can add any extra context they need themselves. Be very wary of drawing vague diagrams on paper or a whiteboard – they confuse more often than they enlighten – much better to show them the real code or document you are working on. They can handle it.

All the time, watch how they are reacting – are they remembering what you’re talking about, or completely in the dark? You may need to choose a different way into the problem. Remember, at this stage it’s your job to explain what you mean, not their job to guess.

If you do this well, they’re slightly in the Zone again, but this time focussed on your problem area. Now, and only now, hit them with your question. They’ll know exactly what you’re talking about, and feel really clever, especially if they know the answer.

There are a couple other things you need to know about the Zone. As we know, it can be painful to be pulled out of the Zone. That means that the person you’re talking to, no matter how nice they are, and how much they like you, may feel a distinct feeling of irritation when you start talking. To make matters worse, they were concentrating, so probably frowning already. Even if they don’t feel any irritation at all, they may look like they do. What can you do about this? Not a lot, except don’t take it personally – it’s normally momentary, and experienced technical people will recognise it’s a false feeling, almost a physical reaction, and put it to one side immediately.

Finally, understanding the Zone explains why preparation is so important. It’s very likely you are in the Zone yourself at the beginning of this process. You are submerged in a difficult problem, and quite possibly deeply irritated by your inability to solve it. To ask a question effectively (and thus get a good answer) you need to pull yourself out of the Zone and back into human interaction mode. This includes ensuring your annoyance is fully dissipated, or you’re going to say something you shouldn’t.

Throughout all this you need to make it your mission to make the other person feel clever.

3. Make them feel clever

Why would you want to make the person you’re asking feel clever?

I certainly don’t mean flatter them. If they’re deep in their work they don’t want you to waste their time telling them how good they are at something.

Think about it this way: how likely is it that you’ll get an answer if you’ve made them feel stupid? I don’t mean you won’t get an answer because they’re too annoyed with you to bother – I mean they’re feeling stupid because you’ve confused them, and so they have no idea what the answer is.

Bear in mind that it’s really easy to confuse someone. In your technical problem area there are thousands, or possibly even millions, of tiny micro-contexts that make sense within themselves, but sound like gibberish if you’re outside that context.

Turning to someone and saying something like, “How do I Frobnish the Pernicator?” without first reminding them that Pernicators get created every time the TunableBeadnest is RePreppered is very hard work for them: they need to ask you a series of questions before they can work out what you’re asking. Making them do this work makes them feel like they aren’t clever enough to hold the entire knowledge of your arena in the front of their mind simultaneously. Of course, no-one is able to do this, but everyone feels like they ought to be. Don’t make them feel stupid.

Even worse is to say something like, “How do I call this method?” In this case, even if they had total front-of-brain memory of the entire arena, they still couldn’t answer this question, without interrogating you about what you meant.

You goal should be that the first thing they have to say in the whole interaction is when they give you the answer, or at least ask a question that shows they have grasped what you’re asking, and makes them feel clever.

Of course, making them feel clever also makes them more inclined to talk to you next time you have a question, but that’s merely a side benefit. The real reason you want to make them feel clever is that when they feel like that, it means you have given them the information they need, in the order they want, with timing that works for them, so that they can give you a great answer.

Anatomy of an interpreter: the Parser

Posts in this series: Lexer, Parser, Evaluator

Subs has reached version 1.3.4, which means that it can successfully run all the tests from chapter 1 of SICP. This is very exciting.

Last time I explained a bit about the Lexer, which takes in a stream of characters and emits a stream of tokens: individual elements of code such as a bracket, a keyword or a symbol.

Generally, parsers emit some kind of tree structure – they understand the raw tokens as a hierarchical structure which (conceptually, at least) will be executed from the bottom up, with each branch-point in the tree being an operation of some kind.

Our parser takes in a stream of tokens, and emits a stream of parsed trees.

Parsing Scheme is very easy, because (except for a couple of exceptions I haven’t implemented yet) there is essentially one rule: start with an open bracket, see a list of things, and then find a close bracket. Of course, one of the “things” you see may itself be another bracketted list, so after parsing you get a tree structure of nested lists.

The parser in Subs looks like this:

class Parser
{
public:
    Parser( ILexer& lexer );
    std::auto_ptr<Value> NextValue();
private:
    ILexer& lexer_;
};

We supply a Lexer in the constructor, which we know will provide us with tokens when we need them via its NextToken() method. The Parser’s NextValue() method returns a pointer to a Value, which is the base class for all the “things” in the Subs interpreter.

There are lots of types of things that inherit from the Value class, but the “parse tree” (the output of the parser) will only consist of a very small subset of them:

  • CombinationValue
  • DecimalValue
  • IntegerValue
  • StringValue
  • SymbolValue

The CombinationValue class forms the tree structure. Its declaration looks like this:

class CombinationValue : public Value, public std::vector<Value*>
{
    // ...
};

It is simply a list of other Values.

Note that it “owns” those Values in the sense that it deletes them when it is deleted. I have recently made the jump to make Subs depend on BOOST, so it’s on my TODO list to make containers like this use the BOOST smart containers to manage this job for me.

DecimalValue, IntegerValue and StringValue are relatively self-explanatory: they contain numbers and strings that were found as literals in the source code.

SymbolValue is essentially everything else – if the code that recognises the type of a token can’t recognise it as a bracket, a number or a string, we assume it is a symbol, and tuck it away in a SymbolValue to be understood later.

The core of the Parser looks like this (with some error-checking removed):

std::auto_ptr<Value> next_value( ILexer& lexer, Token token )
{
    if( token.Name() == "(" )
    {
        auto_ptr<CombinationValue> ret( new CombinationValue );
        while( true )
        {
            token = lexer.NextToken();
            if( token.Name() == ")" )
            {
                break;
            }
            // Recursive call
            ret->push_back( next_value( lexer, token ).release() );
        }
        return auto_ptr<Value>( ret.release() );
    }
    else
    {
        return ValueFactory::CreateValue( token );
    }
}

(Full code here: Parser.cpp) It’s a simple recursive function that creates a CombinationValue whenever it finds a bracket, and otherwise uses a ValueFactory to create an individual value.

Side note: the wisdom of using recursion could certainly be questioned, since it limits the depth of bracketting we can handle to the size of the C++ stack, but the only other way to get the same result would be to keep our own manual stack of unfinished combinations, and it just seems perverse to re-implement language features like that. What might well be more interesting would be to consider whether we can actually evaluate parts of the tree as we go, without parsing it all at once. This might make the whole setup scale rather better, but would most likely be quite complex. The implementation presented here will work fine for almost any imaginable program – remember we would need not just code whose execution is deeply nested, but whose expression in code had thousands of levels of nesting before the parser would fail.

The ValueFactory uses some basic rules such as “starts and ends with a quote” or “consists of only numbers and a decimal point” to recognise what type of Value to create, and if no rules match it defaults to a SymbolValue.

When we have completed a bracketted expression, we return a complete tree of numbers, strings and symbols, and it is ready to be evaluated, which you can think of as simply expanding the tree we already have into the full expression of itself, and then reducing it back down again to an answer.

Next time, the Evaluator and the famous eval-apply loop.