Planet Andy

Planet Gnome : Sankarshan Mukhopadhyay: For the win !

Saturday 04 July 2009 02:07 MST

?What does it take to be good at something at which failure is so easy,so effortless ? ? : a quote from Better: A Surgeon?s Notes on Performance by Atul Gawande which is a highly recommended reading for those who have not read it yet (that?s a link to the flipkart.com entry for those who are local).

Last evening over dinner, among other things, Runa and me got talking about translations and, translation quality. That is one of our favorite shop-talk items and, since the morning blog had bits about my performance with spellings, it was a bit more significant. It is a somewhat known issue that most translation teams measure the length of the sprint, that is, how many strings were completed or, the percentage of the coverage for a particular project. Some projects attach badges like ?supported? / ?unsupported?, ?main? / ?beta? to the coverage and thus make the rush to the tape more important. At some point in time, it is important for the teams to sit down, understand and make notes about the quality of translations. Left to itself, the phrase ?quality of translations? doesn?t mean anything does it ? For example, if the phrase was ?Disconnect from VPN?? and, you were required to translate it ? how wrong can you go ?

It seems you can go wrong, and, most often do.

• One of the reasons that I have observed is that translating strings in application and, translating content like documentation/release_notes/guides require different kind of mind patterns.
• The second reason is the lack of fluency in the source language. So, if you are a translator/reviewer for any language, if you are using English source files (as most of us do), you need to be extremely proficient in the language. The way the sentences, phrases and sub-phrases arrange themselves in English may or may not lend themselves to direct translations
• The third reason is that most translators do not take time out to first use the application in English (or, read the documentation completely in English) and, use it again (or, read it again) after translation. That is a recipe for disaster. English is a funny language and, sometimes, due to the structure of the source files, the context of the content is lost. What does look like a simple word might have a funny implication if the comprehension about how it is placed within the UI or, the user-interaction flow is not made a note of.

Now that most projects have some kind of ?localization steering committees? it would be a good small project to observe which locales are coming up with the highest quality of translations and, attempting to understand what they are doing. Asking the language teams about the reasons that inhibit them from maintaining a high quality would also enable deeper understanding of how a project can help itself become a better one (in a somewhat strange loop way). Such discussions would enable coming up with Guidelines for Quality which are important to have. I firmly believe that all developers desire that their applications be consumed by the largest number of audience possible and, at heart, they are willing to sit down and listen to constructive suggestions about how best they can help the localization teams make it happen. That is the sweet spot the ?LSCo? folks need to converge on and get going. In fact, for projects like OLPC, where a lot of new paradigms are being created, understanding translation processes and, chipping away at improving translation quality is highly requested.

Translation is still an activity that requires a fanatical attention to detail and, that little bit of ingenuity. There is something not right about committing a translation that smacks of a ?letting go of the disciplined focus on detail? and, does not contain anything new. The job is made somewhat more hard when it comes to documentation. One cannot (and, perhaps should not) go beyond what the author has written and yet, it has to be made available in the local language after ?stepping into the shoes? (or, ?getting into the mind?) of the original author while making it aligned with the natural flow of the target language. This is also the place where the ?translator memory?, as opposed to the ?Translation Memory? becomes important. The mind should be supple enough to recall how similar idioms were translated earlier or, if an error that was already reported has cropped up again. Translators have a significant bit to contribute towards making the translation source files better, cleaner, well-maintained and, well documented. And, they have to do it right every time.

All this would come together to produce high quality translations and, wider usage of applications and documentation. Collaboration for the win !

The post is brought to you by lekhonee v0.6

Planet Python : Evan Fosmark: Creating fake words

Saturday 04 July 2009 02:02 MST

I was trying to figure out how to create a word that's not a word. What I ended up doing was creating a way of generating a random syllable, and then simply appending 2 or 3 of them together. It seems to work well enough. Here's what I got in Python:

import random
 
vowels = ["a", "e", "i", "o", "u"]
consonants = ['b', 'c', 'd', 'f', 'g', 'h', 'j', 'k', 'l', 'm', 'n', 'p', 'q', 
              'r', 's', 't', 'v', 'w', 'x', 'y', 'z']
 
def _vowel():
    return random.choice(vowels)
 
def _consonant():
    return random.choice(consonants)
 
def _cv():
    return _consonant() + _vowel()
 
def _cvc():
    return _cv() + _consonant()
 
def _syllable():
    return random.choice([_vowel, _cv, _cvc])()
 
def create_fake_word():
    """ This function generates a fake word by creating between two and three
        random syllables and then joining them together.
    """
    syllables = []
    for x in range(random.randint(2,3)):
        syllables.append(_syllable())
    return "".join(syllables)
 
if __name__ == "__main__":
    print create_fake_word()

The first four functions are for generating an individual type of syllable (V, CV, or CVC) and then _syllable() just chooses one of them at random. Finally, create_fake_word() calls _syllable() a few times and joins them together. Here is some example output:

hojocina
eliphaa
deaketyed
ciboa
tiuzi

I haven't a clue whether or not there is a better means of generating words that look somewhat real. If you know of a better method, I'd love to hear it!

kuro5hin : VirtualBox 3.0: Oracle Wants VMWare's Market and VirtualBox Makes Rapid Progress

Saturday 04 July 2009 02:00 MST

Planet Gnome : Hubert Figuiere: Half a decade

Saturday 04 July 2009 01:30 MST

Half a decade ago I landed in Canada. Not sure what happened since, but that's surely 5 years.

OSNews : PearC Mac Cloner Expands to France, The Netherlands, Belgium

Saturday 04 July 2009 00:40 MST

Planet Gnome : Leonardo Ferreira Fontenelle: Unsubscribe spam

Saturday 04 July 2009 00:23 MST

This has got to be the weirdest spam ever. Today I received this email with ?unsubscribe? as subject and content, from an unknown sender. Could anyone think I would ever reply to this email?

Planet Gnome : Nat Friedman: Nerds and Jocks

Friday 03 July 2009 23:47 MST

Growing up without any noticeable athletic skills, the nerd-jock duality was a pretty important part of my childhood.  Nerds were the kids who carried calculators, wore glasses, dressed poorly, read books for fun, liked to be right in class, and had few friends.  Jocks were athletic, well dressed, and popular, but probably stupid as well.  Every person in my class could have listed, by name, the ?nerds? and the ?jocks? among our classmates, and if we?d transferred to a different school, we could have identified them on sight.  It was, for me, and I suspect for many other kids like me, the primary sorting system for my peers (I guess there was also ?goth? and ?punk,? but we only had one of each at the entire school, so they didn?t count).

Both these terms are pejorative, but ?nerd? was my stigma.  At dinner one evening in 3rd grade, I explained to my parents that my friends and I were the nerds, and that we were proud of it.  I still remember my father?s horrified reaction. ?You?re not a nerd!? he said.

Of course as you get older you find that the labels that dominated your childhood don?t make any sense - but early childhood perspectives sometimes linger, lensing your experiences in ways you don?t notice.

So when I moved to Germany, and found myself having to explain this whole concept to bewildered friends and colleagues, I started to think about the nerd-jock duality a little deeper.  What I realized is that, in Germany, engineering is not stigmatized in the same way that it is in the US.  It is possible to self-identify as an engineer, even at a very early age, without being a nerd.

Germany is, in fact, a country of engineers.  It has to be.  Think about it: a cold, cloudy country ranked only 62nd in land mass, 14th in population, and yet in 2008 Germany was #1 in the world in exports by dollars!  Yes, ahead of the US and ahead of China.  How is that possible?  Nerds!  Oops, I mean engineers; engineers who design and build high-quality cars, engines, tools, machinery, scientific equipment.  This is what happens when you don?t stigmatize engineers: you get a country full of engineers, self-identifying as engineers, growing up dreaming of being engineers.

But what kind of country do you get  when you do stigmatize nerds?  I?m afraid you get a country of importers.  A country of investment bankers and ?famous for being famous? celebrities and television ?news? shows that are frighteningly reminiscent of some of my worst memories of grade school.  A country of people who don?t make things.

My 20 year old sister informs me that the ?nerd? thing has softened a bit in recent years, but maybe not always for the right reasons.  Lots more people spend time with technological devices now, and to be part of the priesthood that creates them, tweaks them, hacks them is more impactful than it used to be.

But one of the reasons ?nerd? isn?t such a dirty word now is because some nerds get rich.  And that?s the wrong reason to appreciate nerds.  Because only very few nerds will get rich, but we need lots of engineers to build our society.

The archetypes that you have as a country matter.  They affect the kind of society you create.  We have a lot of good archetypes in the US.  We have the pioneer, the frontiersman, the individualist, the entrepreneur.  Let?s keep those.  But we can do without the whole nerd/jock thing.  It isn?t helping.

And I think we?d do well to celebrate the engineer archetype again.  I hear it was a big thing in the 50s.  Can we bring it back?

Planet Python : Shannon -jj Behrens: Rails: Validating URLs

Friday 03 July 2009 23:37 MST

require 'uri'

class Film < ActiveRecord::Base

  VALID_URI_SCHEMES = ['http', 'https']

  validates_presence_of :url
  validate :url_must_be_valid

protected

  def url_must_be_valid
    parsed = URI.parse(url)
    if !VALID_URI_SCHEMES.member?(parsed.scheme)
      raise URI::InvalidURIError
    end
  rescue URI::InvalidURIError => e
    errors.add(:url, 'is not a valid URL')
  end
end

Charlie Brooker : Charlie Brooker's screen burn: 4 July 2009

Friday 03 July 2009 23:01 MST

Neil Armstrong was the first man on the moon. Pretty impressive. So impressive that 40 years later, people still make documentaries wondering what that must've been like. Being Neil Armstrong (Sun, 9pm, BBC4) is the latest.

Its premise is this: Neil Armstrong has become a recluse. He never signs autographs and doesn't speak to the press. Why? Why don't you want to come out and talk to us, Neil? Why Neil? Why? Why? Why? What's the matter with you Neil? What's your problem? OI, NEIL! WHY?

Since we're repeatedly told that Neil Armstrong effectively now lives a hermit-like existence in which he scarcely acknowledges the existence of humankind, an interview seems unlikely, so presenter Andrew Smith has to find different ways of discovering what makes him tick. He goes to Neil's home town and talks to a woman who used to be friends with his sister. She reveals that he wasn't a particularly unusual or talkative character.

The woman now runs a model airplane shop, so Smith buys one, goes back to his motel, assembles it, and throws it out of the window. Maybe Neil Armstrong used to do stuff like when he was a kid, he says.

Then he chats to one of Neil's old schoolfriends who reveals that, yes, Neil did play with model planes. Brilliant. We're getting somewhere. When he wasn't making planes, did he like to jump up and down yelping and pointing excitedly at the moon? No. The erstwhile schoolfriend also recalls Neil as fairly subdued person.

We see photos of Neil at school, looking quiet. "Who would have guessed this quiet boy would one day become one of the most famous men on the planet?" ponders Smith.

Nobody did. Perhaps if he'd spent his childhood bellowing, "I LOVE THE MOON," or, "ONE DAY I'LL GO TO THE MOON," or simply shrieking the word "MOON!" at passers-by, maybe someone might've guessed. But he didn't, so they didn't.

Next Smith's in his car, thinking. "Maybe he was just an ordinary, nice man," he says. As you may have gathered by now, not much is happening in this documentary. He drives to a house in the middle of nowhere where Neil used to live. Can he have a look round? No, because he doesn't have permission. He's not even allowed up the driveway. Someone else is in there, though: a couple being shown round by an estate agent. As they leave, Smith, still standing outside, stops them. Did they know this used to be Neil Armstrong's house?

No they didn't.

Thankfully, before things devolve to the point where Smith is looking at a napkin on the basis that Neil Armstrong probably once looked at a napkin, we get to the part of the story where Neil goes to the moon, and there's lots of thrilling footage of that and some good interviews with other former astronauts. The pressures of fame would overwhelm Neil, they reckon. When you've been an astronaut, everyone asks you to repeat the story of how you walked on the moon again and again until you're not even sure of the details yourself. For Neil, the pioneer, it would be intolerable. We meet a barber who once sold a bag of Neil's hair sweepings for $3,000. Little wonder the poor man became a recluse. Little wonder he lives in a lightless cave, shunning all contact with the world outside. It's a sobering moment.

So imagine my surprise when, after the credits roll, I visit Wikipedia in search of some more facts about this solitary, mankind-dodging loner and quickly discover that as recently as 2005 he approved the release of an official biography called First Man: The Life Of Neil Armstrong. There's also a photograph of him happily receiving a platinum disc of Fly Me To The Moon from Quincy Jones at a Nasa anniversary gala in 2008. He doesn't look like a man crushed by the weight of human expectation, but a normal guy who probably couldn't be arsed talking to the 7,000th film crew to contact him that week. Is that right Neil? Neil? Is it? Is it, Neil? WELL, NEIL? IS IT?

OSNews : AmigaForever 2009 Released

Friday 03 July 2009 21:32 MST

OSNews : iPhone Hole Found, Getting Patched

Friday 03 July 2009 21:23 MST

Planet Gnome : Sam Thursfield: Autohell, part 995

Friday 03 July 2009 21:22 MST

• I'm putting in a little time today on my windows branch of jhbuild. Running git now works (using a .bat file to call MSYSgit in its own shell, it's all messy but works fine once it's set up).
  
  
• I spent the past hour or so wondering why ACLOCAL_FLAGS was being ignored. I finally realised that it's not actually honoured by aclocal at all and never has been. autogen.sh scripts tend to execute aclocal $ACLOCAL_FLAGS which make it work often enough that I assumed it was meant to.
  
  Now I wonder whether autoreconf would accept a patch to make it honour $ACLOCAL_FLAGS, or if I should patch Pixman's autogen.sh to call autoreconf $ACLOCAL_FLAGS .. and any others that don't ..
  
  
• Highlights of Glastonbury were definitely Blur, and a more obscure band called Edward II who I last saw aged about 12.
  
  Best wishes for everyone in Gran Canaria!

OSNews : Thoughts of a Linux Game Porter

Friday 03 July 2009 20:56 MST

OSNews : The Bing Thing

Friday 03 July 2009 20:24 MST

JJ : Senior Developer at Clark & Parsia LLC (Washington, DC 20001)

Friday 03 July 2009 19:27 MST

LWN : Tiemann: Open Source Incentives

Friday 03 July 2009 18:57 MST

LWN : Would You Like Linux With Your Jello? (Linux Journal)

Friday 03 July 2009 17:53 MST

LWN : Security advisories for Friday

Friday 03 July 2009 17:51 MST

Debian has updated nagios (arbitrary program execution).

Gentoo has updated libwmf (pointer use-after-free flaw), modsecurity (denial of service).

Red Hat has updated ruby (denial of service).

SUSE has updated java (multiple vulnerabilities), optipng, cups, quagga, pango, strongswan, perl-DBD-Pg, irssi, openssl/libopenssl-devel, net-snmp, ImageMagick/GraphicsMagick, perl, ipsec-tools/novell-ipsec-tools, poppler/libpoppler3/libpoppler4, yast2-ldap-server, tomcat6, gstreamer-plugins/gstreamer010-plugins-bad, apache2-mod_php5 (various issues).

Ubuntu has updated perl (buffer overflow), nagios (arbitrary program execution).

neuro : TechCrunch Has Disgraced Mrs. Slocombe’s Pussy

Friday 03 July 2009 15:31 MST

Dear oh dear. The well-loved and well-respected actress Mollie Sugden has died, aged 86. In tribute to Ms. Sugden’s most famous character, Mrs. Slocombe, and to the constant running jokes about her pet pussy cat Tiddles, Jonathan Ross sent out a tweet encouraging one and all to use the Twitter hashtag #MrsSlocombesPussy in their tweets. Unbelievably rude, but also staggeringly apt! However, Twitter has decided (perhaps algorithimically) not to display search results for that hashtag: that, in and of itself, is somewhat disappointing. The hashtag became so immediately popular it appeared in Twitter’s list of trending topics, dominated in recent days by topics like Michael Jackson, and Glastonbury.

What’s more disappointing, however, is how US technology gossip blogs TechCrunch and Mashable dealt with this information. They considered it an attempt to poison the trending topics list with spam, neither bothering for an instant before publication to check and see if perhaps it was legitimate in some way.

Both sites have since been put right by blog commenters, and they’ve updated their posts to reflect that, but their knee jerk reaction was to condemn the tag as spam. $deity forbid that a territory outwith the US with a better sense of humour, and with less instinct to consider mild double entendres as nasty in some way, would gather up the power to invade the hallowed Temple of Twitter’s Trending Topics.

The blogs’ concerns were that the system could be gamed, but are we saying that those clicking through the trending topics list are stupid, and can’t tell the difference between targeted spam, and legitimate trends?

Planet Python : Isotoma: Black Box BlackBerry

Friday 03 July 2009 14:42 MST

Debugging software is best done using the scientific method: gather evidence about the effects of the bug, conjure up hypotheses to explain the behaviour, experiment to test the hypotheses and modify the code to change the behaviour. Rinse and repeat. If you can’t consistently reproduce the bug though, it can get tricky.

Recently, while developing a site targeted at mobile devices, we came across an intermittent problem when using a BlackBerry device. Testing mobile sites with desktop browsers and emulators can only take you so far. Eventually you reach the point where real devices begin to exhibit their own peccadillos and so we use DeviceAnywhere to access a whole host of remote-controlled physical devices.

Using the BlackBerry Curve, occasionally, our login page wouldn’t proceed to the home page after successful authentication. But we could never reproduce the this in our development environments, only on live; sometimes.

One major difference between the two environments was that the live one had dozens of servers behind a load-balancer which used a URL parameter for session affinity (we couldn’t assume all mobile devices would support cookies), whereas the development environment was a single server. We also had a staging environment which closely reproduced the live environment, although there were only a couple of servers behind its load-balancer. Initial tests on the staging environment indicated that the problem didn’t appear there either.

To rule out the mobile network provider, we installed the excellent Opera Mini browser on the BlackBerry and it worked every time. This also ruled out any issues with pages being cached by Akamai, the content delivery network. So we were now looking for a problem with our code interacting with the BlackBerry browser, but only behind our live load-balancer; sometimes.

After painstakingly tracing through the live Apache logs we closed in on the unexpected cause: a bug in the BlackBerry browser. When a server tells a browser to redirect it sends the full URL, including in our case the all-important session parameter. This URL was being tampered with before the browser navigated to it. The parameter name was being converted to lower-case (if it wasn’t preceded by a slash). This meant that the load-balancer didn’t use it for server affinity so the home page server probably didn’t have a logged-in session, and so it would bounce back to the login page.

The reason this problem had been so hard to reproduce was that in development there was only one server so affinity wasn’t an issue and the server software didn’t care about the case of the session parameter. Also the site URL was different and so the session parameter always had a preceding slash which didn’t trigger the BlackBerry URL tampering, so it never appeared as lower-case in the development logs. And on the staging environment, because there were only two servers, the device would hit the same server, notwithstanding any affinity failure caused by the lower-casing, half of the time by chance alone. The live environment was more likely to fail, but even it gave a sizeable probability of hitting the same server successively by chance alone.

We built a test server and, using some black box reverse-engineering (because the BlackBerry browser is closed-source), we reckon the logic inside the browser’s redirect code goes something like this: "lower-case all the characters in the location URL up to the first slash" presumably with the intention of making the DNS name lower-case. But it should be: "... up to the first slash or ?" to preserve the case of any query parameters.

Googling for this issue returns a number of other sites having redirect and login issues with BlackBerrys. I wonder how many are caused by this subtle, case-sensitive bug?

We’ve since searched our logs and found the bug across this wide range of BlackBerry devices/versions:

• BlackBerry8100/4.2.0
• BlackBerry8100/4.5.0.52
• BlackBerry8110/4.3.0
• BlackBerry8120/4.5.0.52
• BlackBerry8310/4.2.2
• BlackBerry8700/4.2.1
• BlackBerry8800/4.2.1
• BlackBerry8820/4.2.2
• BlackBerry8830/4.2.2
• BlackBerry8900/4.6.1.101
• BlackBerry8900/4.6.1.109
• BlackBerry9000/4.6.0.125
• BlackBerry9000/4.6.0.221

We’ve logged it with BlackBerry. I’ll post an update if we receive any response.

Planet Python : Tarek Ziade: Distutils nighlty builds

Friday 03 July 2009 13:57 MST

LWN : Milepost GCC released

Friday 03 July 2009 13:16 MST

Planet Python : Python African Tour: We?re in Senegal next week

Friday 03 July 2009 12:38 MST

As already announced, the second “Python African Tour” stop is Dakar, next week. A set of events are planned all over the week, thanks to the unstoppable contribution from the DakarLUG guys (Thomas, Ousmane, Serguei, and the others) managing the organization with love. What’s more amazing is that we will be training and showing Python-related stuff to more than 60 people ! Thanks to the companies and groups/organizations sponsoring these events.

Here is an overview of our busy agenda:

Trainings and workshops (monday, tuesday, thursday)

• Python tutorial
  
• Advanced Python training (objects, classes, iterators, decorators, etc)
  
• Scientific Python training
  
• Django tutorial
  
• Django workshop and Scientific Python workshop (taking place at the same time on thursday)
  

The conference day (wednesday) ; we will have various talks and discussions on topics including:

• The “Python African Tour” project
  
• The Python language for newcomers
  
• Using Python for scientific applications
  
• Demos on tools and applications (DemoCamp)
  

Other activities

• A sprint on friday and saturday, for those interested. 
  
• A PloneGov meeting (friday morning).
  

We’re really excited about this, and hope Python will really take off in West Africa !

For more information and following our activities:

• DakarLUG PAT microsite
  
• PythonAfricanTour-related tweets

Planet Classpath : Christian Thalinger (Adv): Turning off format=flowed in Thunderbird

Friday 03 July 2009 10:15 MST

I just noticed that I'm sending emails with format=flowed in Thunderbird, which is not intentional.  A quick search revealed that both sending and displaying emails in flowed format can be turned off. Just change the following preferences to:

mailnews.display.disable_format_flowed_support = true
mailnews.send_plaintext_flowed = false

Better.

Planet Classpath : Roman Kennke: Back in Ubuntu land

Friday 03 July 2009 10:13 MST

After some weeks of having to deal with Windows, I finally upgraded my laptop to Ubuntu. What a relief! Here’s a (not so) short list of immediate advantages that I missed badly on Windows:

• Virtual desktops. Really, can’t live without this. Desktop becomes utterly cluttered when more than 3 windows are open. And I usually have 20 or so. I know there are tools that do similar on Windows (my laptop had one from NVidia) but this is really not the same.
• All important software and drivers on board. Installation was a breeze (a few clicks and 20 minutes waiting) and when finished, _everything_ just worked. Including wifi, printer, scanner. Maybe I was just lucky, or maybe Linux dramatically improved over the last couple of years. And of course, no day long searching web for stuff, installing tons of crapware, etc.
• More consistent, usable and clean UI. On Windows, every app seems to be proud of looking and behaving differently. This is nice for the bling, but makes using the computer unnecessarily difficult. And then there’s small but significant details that make life easier on Ubuntu.
• Worldclock applet. Extremely useful when your team is all over the world.
• Language support. I always considered this obvious, but apparently, it’s not.
• No need to run millions of apps in the system tray to keep your system half up to date. (The other half still needs manual updates ).
• No need to run all kinds of security software, firewalls (they are still a good idea, but I usually don’t  use them on my laptop and never had any problem), virus scanner.
• It uses all 4 GB of my RAM. Which is actually quite important when you make heavy use of virtual machines (for running Windows ).
• Cultural stuff: Apps on Linux usually install themselves, and not a handful of other pieces of software. Recent example on Windows: After having installed some codecs (yay, I was thinking that codecs are a non-problem on Windows, but it’s not the case..) I found that I now have Google Chrome as well. Not that I object trying Chrome, but I prefer to be in control of what gets installed or not.

I probably could continue this list, but I leave that task to you readers if you like ;-). The point is, I have absolutely no pressing need (as in Hardware or Software that only works with Windows) to run Windows as base OS. It’s actually much better to run it in VirtualBox, where I can easily reset it to a clean state when I want to. This is cool for testing anyway. And working on Ubuntu instantly gives me a productivity boost of at least 50-100%.

Phoronix : Open-Source ATI R600/700 3D Driver Almost Working

Friday 03 July 2009 09:30 MST

Phoronix : Fluendo Releases Its Own Linux DVD Player

Friday 03 July 2009 06:26 MST

Phoronix : An Interview With A Linux Game Porter

Friday 03 July 2009 05:00 MST

XKCD : Extrapolating

Friday 03 July 2009 04:00 MST

LWN : Stable kernels 2.6.30.1, 2.6.29.6, and 2.6.27.26

Friday 03 July 2009 03:12 MST

kuro5hin : Playing With Its Wii: Nintendo's 8G Console

Friday 03 July 2009 02:00

Stuart Langridge : Not blocking the UI in tight JavaScript loops

Friday 03 July 2009 00:55 MST

Everyone’s written a JavaScript loop that just loops over all the {LIs, links, divs} on a page*, and it’s pretty standard. Something like

var lis = document.getElementsByTagName("li");
for (var i=0; i<lis.length; i++) { // yes this could be more efficient, don't care
  // do something here to lis[i]
};

or, if you’re using jQuery:

$("li").each(function() {
  // do something here to this
});

This is problematic if there are, say, 2000 LI elements on the page, and what you’re doing in the loop is semi-intensive (imagine you’re creating a couple of extra elements to append to each of those LIs, or something like that). The reason this is a problem is that JavaScript is single-threaded. A tight loop like this hangs the browser until it’s finished, you get the “this script has been running for a long time” dialog, and the user interface doesn’t update while you’re in this kind of loop. You might think: aha, this will take a long time, so I’ll have some sort of a progress monitor thing:

var lis = document.getElementsByTagName("li");
for (var i=0; i<lis.length; i++) { // yes this could be more efficient, don't care
  // do something here to lis[i]
  progressMonitor.innerHTML = "processing list item " + i; // fail
};

but that doesn’t work. What happens is that the browser freezes until the loop finishes. Annoying, but there it is.

One approach to getting around this is with timeouts rather than a for loop.

var lis = document.getElementsByTagName("li");
var counter = 0;
function doWork() {
  // do something here to lis[i]
  counter += 1;
  progressMonitor.innerHTML = "processing list item " + counter;
  if (counter < lis.length) {
    setTimeout(doWork, 1);
  }
};
setTimeout(doWork, 1);

so you move the bit of work you need to do into a function, and that function re-schedules itself repeatedly, using setTimeout. This time, your user interface will indeed update, and your progress monitor will show where you’re up to. There are a couple of caveats with this: it’ll take a bit longer, and you’re no longer guaranteed to have things processed in the order you expect, but they’re minor issues.

For doing this in jQuery, a tiny plugin:

jQuery.eachCallback = function(arr, process, callback) {
    var cnt = 0;
    function work() {
        var item = arr[cnt];
        process.apply(item);
        callback.apply(item, [cnt]);
        cnt += 1;
        if (cnt < arr.length) {
            setTimeout(work, 1);
        }
    }
    setTimeout(work, 1);
};
jQuery.fn.eachCallback = function(process, callback) {
    var cnt = 0;
    var jq = this;
    function work() {
        var item = jq.get(cnt);
        process.apply(item);
        callback.apply(item, [cnt]);
        cnt += 1;
        if (cnt < jq.length) {
            setTimeout(work, 1);
        }
    }
    setTimeout(work, 1);
};

and now you can do

$.eachCallback(someArray, function() {
  // "this" is the array item, just like $.each
}, function(loopcount) {
  // here you get to do some UI updating
  // loopcount is how far into the loop you are
});

$("li").eachCallback(function() {
  // do something to this
}, function(loopcount) {
  // update the UI
});

Not always a useful technique, but when you need it, you need it.

Phoronix : NVIDIA Issues An Open-Source Driver Update

Thursday 02 July 2009 21:01 MST

JJ : Software Developer at Leap Marketing Technologies (Vancouver, BC)

Thursday 02 July 2009 20:47 MST

Planet Classpath : David Gilbert: Apache Harmony and JFreeChart

Thursday 02 July 2009 15:11 MST

Coding Horror : Oh, You Wanted "Awesome" Edition

Thursday 02 July 2009 07:59 MST

We recently upgraded our database server to 48 GB of memory -- because hardware is cheap, and programmers are expensive.

Imagine our surprise, then, when we rebooted the server and saw only 32 GB of memory available in Windows Server 2008. Did we install the memory wrong? No, the BIOS screen reported the full 48 GB of memory. In fact, the system information applet even reports 48 GB of memory:

But there's only 32 GB of usable memory in the system, somehow.

Did you feel that? A great disturbance in the Force, as if 17 billion bytes simultaneously cried out in terror and were suddenly silenced. It's so profoundly sad.

That's when I began to suspect the real culprit: weasels.

No. Not the cute weasels. I'm referring to angry, evil marketing weasels.

That's more like it. Those marketing weasels are vicious.

We belatedly discovered post-upgrade that we are foolishly using Windows Server 2008 Standard edition. Which has been arbitrarily limited to 32 GB of memory. Why? So the marketing weasels can segment the market.

It's sort of like if you were all set to buy that new merino wool sweater, and you thought it was going to cost $70, which is well worth it, and when you got to Banana Republic it was on sale for only $50! Now you have an extra $20 in found money that you would have been perfectly happy to give to the Banana Republicans!

Yipes!

That bothers good capitalists. Gosh darn it, if you're willing to do without it, well, give it to me! I can put it to good use, buying a SUV or condo or Mooney or yacht one of those other things capitalists buy!

In economist jargon, capitalists want to capture the consumer surplus.

Let's do this. Instead of charging $220, let's ask each of our customers if they are rich or if they are poor. If they say they're rich, we'll charge them $349. If they say they're poor, we'll charge them $220.

Now how much do we make? Back to Excel. Notice the quantities: we're still selling the same 233 copies, but the richest 42 customers, who were all willing to spend $349 or more, are being asked to spend $349. And our profits just went up! from $43K to about $48K! NICE!

Capture me some more of that consumer surplus stuff!

How many versions of WIndows Server 2008 are there? I count at least six. They're capturing some serious consumer surplus, over there in Redmond.

• Datacenter Edition
• Enterprise Edition
• Standard Edition
• Foundation
• Web
• HPC

Already, I'm confused. Which one of these versions allows me to use all 48 GB of my server's memory? There are no less than six individual "compare" pages to slice and dice all the different features each version contains. Just try to make sense of it all. I dare you. No, I double dog dare you! Oh, and by the way, there's zero pricing information on any of these pages. So open another browser window and factor that into your decisionmaking, too.

I don't mean to single out Microsoft here; lots of companies use this segmented pricing trick. Even Web 2.0 darlings 37 Signals.

Heck, our very own product segments the market.

37signals just does it .. prettier, that's all. They're still asking you if you're poor or rich, and charging you more if you're rich.

Eric Sink also advocates the same "rich customer, poor customer" software pricing policy:

In an ideal world, the price would be different for every customer. The "perfect" pricing scheme would charge every customer a different amount, extracting from each one the maximum amount they are willing to pay.

• The IT guy at Podunk Lutheran College has no money: Gratis.
• The IT guy at a medium-sized real estate agency has some money: $500.
• The IT guy at a Fortune 100 company has tons of money: $50,000.

You can never make your pricing "perfect," but you can do much better than simply setting one constant price for all situations. By carefully tuning all these details, you can find ways to charge more money from the people who are willing to pay more.

This sort of pricing seems exploitative, but it can also be an act of public good -- remember that the poorest customers are paying less; with a one-size-fits-all pricing policy, they might not be able to afford the product at all. Drug companies often follow the same pricing model when selling life-saving drugs to third-world countries. First-world countries end up subsidizing the massive costs of drug development, but the whole world benefits.

What I object to isn't the money involved, but the mental overhead. The whole thing runs so contrary to the spirit of Don't Make Me Think. Sure, don't make us customers think. Unless you want us to think about how much we'd like to pay you, that is.

And what are we paying for? The privilege of flipping the magic bits in the software that say "I am blah edition!" It's all so.. anticlimactic. All that effort, all that poring over complex feature charts and stressing out about pricing plans, and for what? Just to get the one simple, stupid thing I care about -- using all the memory in my server.

Perhaps these complaints, then, point to one unsung advantage of open source software:

Open source software only comes in one edition: awesome.

The money is irrelevant; the expensive resource here is my brain. If I choose open source, I don't have to think about licensing, feature matrices, or recurring billing. I know, I know, we don't use software that costs money here, but I'd almost be willing to pay for the privilege of not having to think about that stuff ever again.

Now if you'll excuse me, I'm having trouble deciding between Windows 7 Smoky Bacon Edition and Windows 7 Kenny Loggins Edition. Bacon is delicious, but I also love that Footloose song..

[advertisement] Interested in agile? See how a world-leading software vendor is practicing agile.

Phoronix : A Root-less X Server Nears Reality

Thursday 02 July 2009 06:31 MST

Planet Classpath : Jonathan Gibbons: Minor updates for jtreg

Thursday 02 July 2009 02:38 MST

It includes the following minor fixes and updates:

1. Do not follow symlinks when cleaning a directory.
2. Avoid potential NPE when cleaning a directory.
3. Allow annotation processing of class files in jtreg directives.
   New option /process for @compile tag disables requirement that one or more *.java files must be given to @compile.
4. Set test.src and test.classes properties for @compile tasks.
   These properties can now be used by annotation processors invoked by the @compile task.
5. Recategorize -e option as a general option instead of a JDK one.
6. Set MKS to use case-sensitive compare in launcher script.
7. Remove obsolete reference to JAVA_HOME from launcher script.
8. (jtdiff) Avoid NPE in standard jtdiff if no output file given.

kuro5hin : The Unified Theory of Conservative Healthcare

Thursday 02 July 2009 02:00

JJ : Software Developer at IMC Financial Markets (Chicago, IL)

Wednesday 01 July 2009 23:10 MST

Martin Fowler : RequestStreamMap

Wednesday 01 July 2009 18:58 MST

Hang around my colleagues at ThoughtWorks and you soon get the impression that the only good Enterprise Service Bus (ESB) is a dead ESB. Jim Webber refers to them as Egregious Spaghetti Boxes. So it's not uncommon to hear tales of attempts to get them out of systems that don't need them.

Battle was joined at one client and it brought to mind my younger days playing D&D. Webber swings but misses as the ESB is AC 2, Evan gets a hit and rolls 2d8 for 6 damage. Erik finally kills it by casting "Summon Request Stream Map".

So what was Erik's decisive spell? Essentially the idea was to take a simple request and show how the data for the request and response made their way through the layers of the application. Erik printed out all the code that you needed to read to understand how this would work - which ran to several pages. He also produced this diagram.

It's currently fashionable in agile circles to do Value Stream Mapping as a way to uncover waste in a software development process. I think of this as a request stream map because it similarly takes a request and shows how it moves through the layers allowing us to visualize what's going on and think about the cost and value of the layers.

Layering is an essential tool for building software applications. But like most essential things in life, excess can be almost as much of a problem as too little. A visualization like this (or the multiple pages of code) can help you find where "just enough" is.

One hazard, however. If you do need to transform data from one form to another, it's usually better to a few little transformations than one big transformation. You want to avoid unnecessary transformations not compress the ones you need.

UKJJS : Java Developer at hedgehog lab (Newcastle Upon Tyne, United Kingdom)

Wednesday 01 July 2009 18:37 MST

hedgehog lab is a software company in the North East of England that specialises in products and services for the web, mobile and desktop. Due to growth in the business in the past few months, we are looking to recruit a new team member to join our R&D team working on Java-based products.

Ideally, you should feel very comfortable across Java and related technologies. You would generally be involved in software design, infrastructure and architecture design, solving scalability problems, adding new functionality and, of course, writing code. Our products use a full open source stack focusing mainly on Java/J2EE, MySQL, Tomcat, Hibernate, Spring MVC, Apache Lucene/Solr etc.

Applicants should have

- Permanent legal right to work in the United Kingdom.
- Available to work at our office in Gateshead, UK. Remote/telecommuting is not ideal for this role.

No recruiters, consulting engagements or solicitations.

Interested? E-mail recruitment@hedgehoglab.com with a plain text CV or PDF.

JJ : Java Developer at hedgehog lab (Newcastle Upon Tyne, United Kingdom)

Wednesday 01 July 2009 18:37 MST

Planet Classpath : Roman Kennke: Java is a Doom Trojan Horse

Wednesday 01 July 2009 17:02 MST

At least according to Symantec.

I want to make a screenshot, but I don’t know how to do this in Windows XP. I probably need the Ultimate Edition or so. God, this crappy OS doesn’t even have the most basic functionality, or it hides it so well that nobody finds it. No, the print key doesn’t work.

Folks, I really gave this OS another serious chance. Actually I was more or less forced to, due to my new job and new hardware and lack of Linux install CDs for 2 weeks in the US. But it so gets in the way of everything and is generally so unusable, I think I really do the same update that David Gilbert (*) mentioned in the article above.

Interesting side detail: I tried a couple of games in Windows that I tried in Wine before, it must be said that nowadays almost all games I would want to play work perfect in Wine (Heroes series, Starcraft, Baldur’s Gate, Settlers series), some show the same buggy behaviour in Windows as in Wine (SimCity4) and some others actually work better in Wine (!!), namely Settlers IV and Baldur’s Gate I. As far as I found out it’s a bug in the NVidia driver not supporting DirectDraw properly.

* David, is that you: http://en.wikipedia.org/wiki/David_Gilbert ?

Update: Ok, thanks to Robert I managed to make the print key work. Here is the screenshot:

UKJJS : Web Developer at Wonder Associates Ltd (Manchester, United Kingdom)

Wednesday 01 July 2009 16:18 MST

Web Developer/£20-25,000. Permanent. Full-time.

Design consultancy with enviable client list and reputation, seeking Web Developer to support digital growth. This is a rare opportunity to not only work on exciting client projects alongside studio graphic designers and marketeers, but to help shape our ambitious plans for the future. We require a degree-qualified Web Developer with at least 2 years commercial experience (freelance, client or agency) in both front-end and back-end development. We're looking for technical excellence in HTML, CSS, JavaScript & PHP, with the skills to develop software in-house (e.g. CMS) meeting W3C standards. As a fast paced, creative consultancy, you should be a team player with outstanding capabilities, high work ethic, flexible attitude and great communication skills. You must also be able to demonstrate a wider commitment to the digital industry, with a passion for applying the latest innovations.

Interested? Interested? Please send your CV and examples of your work ASAP to lisa.gardiner@wonderassociates.com for review.

JJ : Web Developer at Wonder Associates Ltd (Manchester, United Kingdom)

Wednesday 01 July 2009 16:18 MST

Uraeus : Syntax Era

Wednesday 01 July 2009 15:54 MST

So the BBC is making a new drama series about the battle between the ZX spectrum and the BBC Micro. Currently codenamed ‘Syntax Era’. As it turns out Clive Sinclair, the creator of the ZX Spectrum, had his offices very close to the current Collabora office here in Cambridge. And due to our own Edward Hervey knowing some of the people involved, the production team behind this new series came by our offices to do some location scouting some weeks ago. It is a little bit up in the air if they are going to use our offices or not in the end, but there is a chance they will, and if that happens there is also chance you might catch some familiar faces as extras in this new series

XKCD : Qwertial Aphasia

Wednesday 01 July 2009 04:00 MST

kuro5hin : Apple Will Never Replace Darwin With Linux

Wednesday 01 July 2009 02:00

UKJJS : Senior PHP Developer at English360 (New York, NY / Cambridge, United Kingdom)

Tuesday 30 June 2009 22:19 MST

We’re a unique combination: an e-learning startup looking to shake things up through innovation, recently partnered with one of the oldest, most prestigious educational publishers in the world, Cambridge University Press.

Our flagship web application is now at version 1, and we are looking for a senior PHP developer with the desire and attitude to take the technology lead within English360. Have no fear of a poor code base: the initial application has been developed using CakePHP, by the people who wrote much of the framework itself: Cake Development Corporation.

Now we are looking for you:

You’re a world-class PHP developer who has previous experience with CakePHP, or the ambition to pick it up from the folks who wrote it. You are a testing / Q&A fanatic, proficient in jQuery/HTML/CSS, and a SysAdmin wizard.

After a 6-month, full-time contractor role, you would take full responsibility for the technology side of the company, manage contractors, build your department, and collaborate with senior management on the future direction of our products. Ideally you would also be located in, or willing to relocate to either our New York City or Cambridge UK offices, after successful completion of your contract.

Interested? Resume, and three references who can detail your experience are required for consideration. Email e360jobs@gmail.com. You will be contacted if we are interested in an interview.

Martin Fowler : IllustrativeProgramming

Tuesday 30 June 2009 19:23 MST

What's the most common programming language in the world?

I'm not sure how you could go about measuring this, but one thing you'd need to do is consider what we mean by programming. My candidate answer considers that the most popular programming language is one used widely by people who do not consider themselves as programmers. This language is Excel, or more generally spreadsheets.

Spreadsheets are easily used for small tasks, but are also used for surprisingly complex and important things. Often I've seen professional programmers gulp when they realize that some vital business function is being run off some spreadsheet that they'd find too complicated to muck with.

In general, we've not had much success with programming languages for these kind of LayProgrammers. Whenever someone talks about some new environment that's going to allow people to specify complex behavior "without programming" I mention COBOL, which was originally designed to get rid of programmers. So it's important to consider what Excel can teach us about programming environments.

One property of spreadsheets, that I think is important, is its ability to fuse the execution of the program together with its definition. When you look at a spreadsheet, the formulae of the spreadsheet are not immediately apparent, instead what you see is the calculated numbers - an illustration of what the program does.

Using examples as a first class element of a programming environment crops up in other places - UI designers also have this. Providing a concrete illustration of the program output helps people understand what the program definition does, so they can more easily reason about behavior.

So why do I feel we need this particular Neologism? Essentially because I think it deserves more thought. We pass by illustrative programming examples without really thinking about them or what makes them special - or even that they are special in some way. We've used illustrative programming for years, but we've not paid enough attention to it. We've not thought enough about what are its essential qualities and what its strengths and weaknesses are.

I've chosen the term "Illustrative Programming" to describe this, partly because "example" is so heavily used (and illustration isn't) but also because the term "illustration" reinforces the explanatory nature of the example execution. Illustrations are meant to help explain a concept by giving you a different way of looking at it - similarly an illustrative execution is there to help you see what your program does as you change it.

When trying to make a concept explicit like this, it's useful to think about the boundary cases. One boundary is the notion of using projections of program information during editing, such as an IDE that shows you the class hierarchy while you are working on the code. In some ways this is similar, as the hierarchy display is continuously updated as you modify the program, but the crucial difference is that the hierarchy can be derived from static information about the program. Illustrative programming requires information from the actual running of the program.

I also see illustrative programming as a concept beyond the classic REPL loop of dynamic languages. REPL loops allow you to explore execution, but they don't make the examples front and center in the way that a spreadsheet does its values. Illustrative programming techniques put the illustration in the foreground of your editing experience. The program retreats to the background, peeping out only when we want to explore a part of the illustration.

I don't think that illustrative programming is all goodness. One problem I've seen with spreadsheets and with GUI designers is that they do a good job of revealing what a program does, but de-emphasizes program structure. As a result complicated spreadsheets and UI panels are often difficult to understand and modify. They are often riven with uncontrolled copy-and-paste programming.

This strikes me as a consequence of the fact that the program is de-emphasized in favor of the illustrations. As a result the programmers don't think to take care of it. We suffer enough from a lack of care of programs even in regular programming, so it's hardly shocking that this occurs with illustrative programs written by lay programmers. But this problem leads us to create programs that quickly become unmaintainable as they grow. The challenge for future illustrative programming environments is to help develop a well structured program behind the illustrations - although the illustrations may also make us rethink what a well structured program is.

The hard part of this may well be the ability to easily create new abstractions. One of my observations of rich client UI software is that they get tangled because the UI builders think only in terms of screens and controls. My experiments here suggest to me that you need to find the right abstractions for you program, which will take a different form. But these abstractions won't be supported by the screen builder as it can only illustrate the abstractions it knows about.

My colleagues Rebecca Parsons and Neal Ford have been spending a lot of time involved in thinking along these lines too. So here's some thoughts that Neal had in an email exchange

One of the few people to take illustrative programming seriously is Jonathan Edwards. He's come up with many very imaginative ideas as to what such an environment should look like. His vision of illustrative programming is also closely bound to the notions of projectional editing and controlled copy-and-paste.

The trigger for me in wanting to coin a term here, is the use of illustrative programming by Language Workbenches by people like IntentionalSoftware. These Language Workbenches encourage you to build illustrative DSLs. Using illustration is important in this case since this should help engage lay-programmers, which is one of the aims of using DSLs. The challenge is to do this without falling into the trap of poor program structure.

Coding Horror : All Abstractions Are Failed Abstractions

Tuesday 30 June 2009 13:01 MST

In programming, abstractions are powerful things:

Joel Spolsky has an article in which he states

All non-trivial abstractions, to some degree, are leaky.

This is overly dogmatic - for example, bignum classes are exactly the same regardless of the native integer multiplication. Ignoring that, this statement is essentially true, but rather inane and missing the point. Without abstractions, all our code would be completely interdependent and unmaintainable, and abstractions do a remarkable job of cleaning that up. It is a testament to the power of abstraction and how much we take it for granted that such a statement can be made at all, as if we always expected to be able to write large pieces of software in a maintainable manner.

But they can cause problems of their own. Let's consider a particular LINQ to SQL query, designed to retrieve the most recent 48 Stack Overflow questions.

var posts = 
  (from p in DB.Posts
  where 
  p.PostTypeId == PostTypeId.Question &&
  p.DeletionDate == null &&
  p.Score >= minscore
  orderby p.LastActivityDate descending
  select p).
  Take(maxposts);

The big hook here is that this is code the compiler actually understands. You get code completion, compiler errors if you rename a database field or mistype the syntax, and so forth. Perhaps best of all, you get an honest to goodness post object as output! So you can turn around and immediately do stuff like this:

foreach (var post in posts.ToList())
{
    Render(post.Body);
}

Pretty cool, right?

Well, that Linq to SQL query is functionally equivalent to this old-school SQL blob. More than functionally, it is literally identical, if you examine the SQL string that LINQ generates behind the scenes:

string query = 
  "select top 48 * from Posts
  where 
  PostTypeId = 1 and 
  DeletionDate is null and 
  Score >= -4
  order by LastActivityDate desc";

This text blob is of course totally opaque to the compiler. Fat-finger a syntax error in here, and you won't find out about it until runtime. Even if it does run without a runtime error, processing the output of the query is awkward. It takes row level references and a lot of tedious data conversion to get at the underlying data.

var posts = DB.ExecuteQuery(query);

foreach (var post in posts.ToList());
{
   Render(post["Body"].ToString());
}

So, LINQ to SQL is an abstraction -- we're abstracting away raw SQL and database access in favor of native language constructs and objects. I'd argue that Linq to SQL is a good abstraction. Heck, it's exactly what I asked for five years ago.

But even a good abstraction can break down in unexpected ways.

Consider this optimization, which is trivial in the old-school SQL blob code: instead of pulling down every single field in the post records, why not pull just the id number? Makes sense, if that's all I need. And it's faster -- much faster!

select top 48 * from Posts	827 ms
select top 48 Id from Posts	260 ms

Selecting all columns with the star (*) operator is expensive, and that's what LINQ to SQL always does by default. Yes, you can specify lazy loading, but not on a per-query basis. Normally, this is a non-issue, because selecting all columns for simple queries is not all that expensive. And you'd think pulling down 48 measly little post records would be squarely in the "not expensive" category!

So let's compare apples to apples. What if we got just the id numbers, then retrieved the full data for each row?

select top 48 Id from Posts	260 ms
select * from Posts where Id = 12345	3 ms

Now, retrieving 48 individual records one by one is sort of silly, becase you could easily construct a single where Id in (1,2,3..,47,48) query that would grab all 48 posts in one go. But even if we did it in this naive way, the total execution time is still a very reasonable (48 * 3 ms) + 260 ms = 404 ms. That is half the time of the standard select-star SQL emitted by LINQ to SQL!

An extra 400 milliseconds doesn't sound like much, but slow pages lose users. And why in the world would you perform a slow database query on every single page of your website when you don't have to?

It's tempting to blame Linq, but is Linq really at fault here? These seem like identical database operations to me:

1. Give me all columns of data for the top 48 posts.

or

1. Give me just the ids for the top 48 posts.
2. Retrieve all columns of data for each of those 48 ids.

So why in the wide, wide world of sports would one of these seemingly identical operations be twice as slow as the other?

The problem isn't Linq to SQL. The problem is that we're attempting to spackle a nice, clean abstraction over a database that is full of highly irregular and unusual real world behaviors. Databases that:

• may not have the right indexes
• may misinterpret your query and generate an inefficient query plan
• are trying to perform an operation that doesn't fit well in available memory
• are paging data from disks which might be busy at that particular moment
• might contain irregularly sized column datatypes

That's what's so frustrating. We can't just pretend all our data is formatted into neat, orderly data structures sitting there in memory, lined up in convenient little queues for us to reach out and casually scoop them up. As I've demonstrated, even trivial queries can have bizarre behavior and performance characteristics that are not at all clear.

To its credit, Linq to SQL is quite flexible: we can use strongly typed queries, or we can use SQL blob queries that we cast to the right object type. That flexibility is critical, because so much of our performance depends on these quirks of the database. We default to the built-in Linq language constructs, and drop down to hand-tuning ye olde SQL blobs where the performance traces tell us we need to.

Either way, it's clear that you've got to know what's happening in the database every step of the way to even begin understanding the performance of your application, much less troubleshoot it.

I think you could make a fairly solid case that Linq to SQL is, in fact, a leaky and failed abstraction. Exactly the kind of thing Joel was complaining about. But I'd also argue that virtually all good programming abstractions are failed abstractions. I don't think I've ever used one that didn't leak like a sieve. But I think that's an awfully architecture astronaut way of looking at things. Instead, let's ask ourselves a more pragmatic question:

Does this abstraction make our code at least a little easier to write? To understand? To troubleshoot? Are we better off with this abstraction than we were without it?

It's our job as modern programmers not to abandon abstractions due to these deficiencies, but to embrace the useful elements of them, to adapt the working parts and construct ever so slightly less leaky and broken abstractions over time. Like desperate citizens manning a dike in a category 5 storm, we programmers keep piling up these leaky abstractions, shoring up as best we can, desperately attempting to stay ahead of the endlessly rising waters of complexity.

As much as I may curse Linq to SQL as yet another failed abstraction, I'll continue to use it. Yes, I may end up soggy and irritable at times. But it sure as heck beats drowning.

[advertisement] Interested in agile? See how a world-leading software vendor is practicing agile.

UKJJS : Systems Developer (PHP 5, MySQL 5) at 7city Learning (London, United Kingdom)

Tuesday 30 June 2009 12:26 MST

Do you thrive on developing innovative technical solutions to business problems? 7city Learning is looking for smart, well-rounded developers such as you to join a development team responsible for delivering award-winning e-learning and business systems.

As a candidate for this London-based role you will have drive, passion and enthusiasm coupled with commercial PHP 5 development experience. You will thrive on taking responsibility for important technical projects that help the business in its goal to deliver leading-edge training.

As a systems developer you will undertake the following key responsibilities:

• Understanding and translating business and e-learning requirements into robust technical solutions
• Developing code for business and e-learning systems, including maintaining and enhancing our MVC-based development framework
• Designing and implementing relational databases for these systems
• Building sophisticated client and management reports
• Assisting and supporting our global user-base of clients and staff
• Writing clear and concise end-user and systems documentation

To succeed you should have the following skills and experience:

• Computer science or IT degree, or equivalent industry experience
• Practical web application development experience using object-oriented PHP 5, including exposure to MVC-based frameworks
• Good relational database analysis and design skills, including diagramming techniques
• Experience in data querying and manipulation with SQL, including ‘enterprise’ functionality such as triggers, transactions and stored procedures
• Broad experience with markup and stylesheet languages, including XHTML, CSS, XML and XSL
• Solid written and verbal communication skills

It will be great if you can demonstrate the following additional skills:

• Experience with source code management systems such as Subversion and CVS
• Experience with software project management and defect tracking systems such as FogBugz and Trac
• Experience with systems integration technologies, including web services and APIs from vendors such as Salesforce
• Linux, Apache and MySQL server management, including shell scripting and task scheduling
• Exposure to e-learning technologies such as learning management systems (LMS)

As well as the opportunity to work with a diverse group of professional, innovative and passionate people, we offer a competitive salary of up to £45,000 depending on experience and excellent benefits package.

Interested? If you like the sound of this role, email your CV to recruitment@7city.com. In the email subject, please include the label “PHP Systems Developer (London)” along with your name.

kuro5hin : Lost frames of reference... Part 1: reach out and touch someone.

Tuesday 30 June 2009 02:00

MozillaZine : Mozilla Firefox 3.5 Released

Tuesday 30 June 2009 00:49 MST

Mozilla Firefox 3.5 has been released. Available for Windows, Mac OS X and Linux in more than 70 different languages, Firefox 3.5 is the most recent major Mozilla browser release since the launch of Firefox 3 in June 2008.

Firefox 3.5 can be downloaded from the Firefox product page or via software update. More information can be found in the Firefox 3.5 Release Notes.

The Firefox Features page at mozilla.com lists the new and improved features in Firefox 3.5

Talkback

Martin Fowler : Revitalizing Enterprise Software

Monday 29 June 2009 23:14 MST

Allan Kelly : Agile FAQ - Kicking off another Agile team

Monday 29 June 2009 12:02 MDT

GingerDog : Timberhonger 10k Race - June 2009

Monday 29 June 2009 06:38 MST

I came 29th - with a time of 42 minutes 50 seconds - it was pretty hot, and I was therefore quite pleased with my time - which was better than last year.

thanks to the organisers; more info here

XKCD : Idiocracy

Monday 29 June 2009 04:00 MST

neuro : 2009-06-29: neuro’s Tweets from Last Week

Monday 29 June 2009 01:41 MST

• Watching Hantuchova v Robson, blissfully free from shrieks and grunts #wimbledon #
• A very promising seniors debut from Laura Robson, hopefully she'll have more confidence next year #wimbledon #
• RT @stephenfry: One of my all time heroes Alan Turing was born today 1912! Find out more here: http://www.bletchleypark.org.uk #bpark #
• Tonight there's gonna be a(n iPhone) jailbreak, somewhere in this (unlocked) town! #redsn0w #ultrasn0w #
• iPhone now says "T-Mobile" at top left of screen … ah, sim unlocking is fun! #ultrasn0w #
• Jeff Goldblum death reports sourced from fake news site; if you're at mediafetcher.com change name at start of the URL! #
• BBC News just trotted out their old hackneyed "give us a sense of … " question, can we get some real journalism on the go please? #
• BBC News: "[Michael Jackson's] 50-date tour, which will now of course not happen", really?! #
• RT @demonbaby: If you have any doubt left that Perez Hilton is a worthless shitbag, let me fix that for you: http://twitpic.com/8et2f #
• RT @demonbaby: Show the world you value culture, taste, and intelligence, not mindless hateful garbage: #unfollowperezhilton – RT this! #
• shops on a friday, mental – Photo: http://bkite.com/08VFk #
• BBC on Andy Murray: "An hour and thirty minutes of controlled brilliance" #wimbledon #
• Requesting an invite for Tweetboard Alpha (http://tweetboard.com) by @140ware, for my site: http://neuro.me.uk #

Charlie Brooker : Michael Jackson's death hit Glastonbury hard ? and the news channels harder | Charlie Brooker

Sunday 28 June 2009 23:05 MST

Festival-goers did the moonwalk in tribute to Jacko. But for the BBC, ITV and Sky, the news demanded much, much more ?

I was at Glastonbury when Jacko died. That's not a factual statement, but a T-shirt slogan. The day after his death, souvenir tops with "I was at Glasto 09 when Jacko died" printed on them were already on sale around the site. In fact, when Jacko died, I was at home playing Grand Theft Auto: Chinatown Wars on a Nintendo DSi. I am 38 years old.

Many festival-goers apparently discovered the news when DJs around the site began playing Michael Jackson records simultaneously. Music combined with word of mouth. That's a nice way to find out. I learned it via a harsh electric beep, bringing my attention to a text message that simply proclaimed "Jackson's dead" in stark pixelated lettering. Clearly it's the sort of information you have to mindlessly share with the rest of the herd the moment you hear about it. But first I needed confirmation. I occasionally text people to say there's been a massive nuclear explosion in Canada, or David Cameron's gone mad and launched his own breakfast cereal shaped like little swastikas or whatever, in the hope they'll pass it on without checking. I didn't want to fall for my own jape.

I switched on the TV. Jackson was still alive on BBC News 24, where they seemed to be reporting he was in hospital following a heart attack. That wasn't good enough, so I flicked over to Sky News, which tends to blab stuff out while the Beeb drags its feet tediously checking the facts. He was bound to be dead on Sky. But he wasn't; he was possibly in a coma. In desperation, I turned to Fox. They would already be attempting to communicate with him via the spirit realm, surely. But they weren't. If anything, they were being more cautious than the Beeb. Boo.

Back to Sky, which was now reporting that a website was announcing his death. That'd do for now. I beamed a few texts out: "Michael Jackson apparently dead". "Piss off" came the reply. It was my own fault. I'd texted a few weeks earlier to say Huw Edwards had just vomited live on the news.

Confirmation of his death gradually spread across the news networks, but the main terrestrial channels were still obliviously broadcasting their scheduled programmes. ITV won the newsflash race, diving straight in after Trial and Retribution. Alastair Stewart abruptly shouted "MICHAEL JACKSON HAS DIED" down the lens like a man standing on the shoreline trying to get the attention of someone on the deck of a passing ferry during gale-force winds. Fair enough. Whenever I hear the phrase, "And now a special news report", I automatically start scanning the room for blunt objects to club myself to death with in case they're about to announce nuclear war. Since this wasn't the apocalypse, but an unexpected celebrity death ? sad, but not worth killing yourself with a paperweight over ? Stewart was right to blurt it out as fast as he could.

After watching the news long enough to assess that, yes, he was dead, and the circumstances all seemed rather tragic, long enough for them to play a bit of Billie Jean and Beat It and Smooth Criminal and Blame it on the Boogie and so on, reminding me that he was a bona fide musical genius, I went to bed.

The next day he was still dead, but somehow deader than the day before. He was all over the radio and papers. The TV had clips of Thriller on heavy rotation, which seemed a tad inappropriate, what with him playing a decomposing corpse in it. If Bruce Willis died falling from a skyscraper, I doubt they'd illustrate his life story by repeatedly showing that bit from Die Hard where he ties a firehose round his waist and jumps off the building.

Across all the networks, a million talking heads shared their thoughts and feelings on his death. They had rung everyone in the universe and invited them on the show. On This Morning, a Coronation Street actor revealed he had once had tickets for a Michael Jackson concert but couldn't go because of the traffic. It was a sad day indeed. At 3pm, his death was still "BREAKING NEWS" according to Sky, which has to be some kind of record. Even 9/11 didn't "break" that long.

Next day, the news was apparently still sinking in around the globe. The BBC went live to Emily Maitlis as she stood on Hollywood Boulevard (at 1am local time) waiting for two young Latinos to perform a breakdance tribute to the King of Pop. Something went wrong with the iPod hooked up to their speakers so she had to stand there for a full two minutes, awkwardly filling in while they fiddled with the settings. Sky had flown Kay Burley out to LA too, to hear the fans' pain and pull concerned faces. This continued into the following day. It's probably still going on now.

But the news is not the place to "celebrate" Jackson's music. The Glastonbury stage, the pub, the club, the office stereo, the arts documentary: that's the place. The news should report his death, then piss off out of the way, leaving people to moonwalk and raise a toast in peace.

If I was God, here's what I'd do now. I'd force all the rolling networks to cover nothing but the death of Michael Jackson, 24 hours a day, for the next seven years. Glue up the studio doors and keep everyone inside, endlessly "reporting" it, until they start going mad and developing their own language ? not just verbal, but visual. And I'd encourage viewers to place bets on which anchor would be the first to physically end it all live on air.

And while that was happening, I'd create some other stations that covered other stuff. Current affairs type stuff. I think I'd call them "news channels". They might catch on.

This week Charlie was saddened to read of the death of former NME writer Steven "Swells" Wells: "I disagreed with 85% of what he wrote, but I always wished I could hurl sentences together like him ? he tossed words around like a demented cartoon chef. He seemed hilarious and furious, music journalism's very own Sadowitz. Never met or spoke to him; now I wish I had. RIP."

grifferz : Links for 2009-06-27

Saturday 27 June 2009 22:00 MST

• Ladies and Gentlemen, [jwz has] ported Dali Clock to the Palm Pre.
• Phil Hands explains how to do passphrase-less ssh properly.

Don Marti : Friday MLP...Friday!

Saturday 27 June 2009 17:00 MST

Student Statement on The Right to Research: "Learning and inquiry are impeded when scholars lack access to fellow researchers? work, and when students lack access to the work of scholars before them."

Seth Godin: Textbook rant. Andy Oram: Four roles for publishers: staying relevant when you are no longer a gatekeeper

Script and Screen

Worried about SHA-1? Read The code monkey's guide to cryptographic hashes for content-based addressing.

...even if you pay your premiums for years, if you ever fall seriously ill, they will do their best to find a way to cancel your insurance.

a hardware hacker ponders flu viruses

Advice from 2005, still not yet applied to GPL enforcement: "Registered trademarks, copyrights, and certain trade names may be recorded with the U.S. Customs Service."

So there's probably an anti-Linux back channel for Digg users. I wonder how many private lists are devoted just to promulgating URLs to pump or slag on other services.

Uraeus : Transmageddon 0.11 ?GUADEC Edition?

Saturday 27 June 2009 13:10 MST

So in preparation for heading of to Gran Canaria and GUADEC on Thursday I pushed a new Transmageddon release today. 0.11 is actually the first release I posted to sites such as Gnomefiles and Freshmeat so in some sense I guess I feel more confident about this version that earlier ones. A lot of new features included, like multipass encoding, videoflipping (so if your video is 90 degrees tilted you can correct that during transcoding) and better profiles. Still kinda rough at the edges though, and the very latest GStreamer releases are needed for everything to work. Still expecting quite a few bugs to be reported still.

In some sense development have stood still for a while as I have focused on testing various devices and GStreamer features. Remuxing is still on my todo list, but discovered that for instance an ac3parser is still needed in GStreamer to
do nice DVD conversions.

Was kinda nice to successfully use Transmageddon for a business related need at work, we had gotten a request at Collabora Multimedia which meant I needed to transcode a video into quicktime+h264+aac from a .vob file. Worked like a charm I mean while I have of course done hundred of transcodes as part of development, it was nice to do one for something ‘real’

grifferz : Web developers who don?t understand how email works

Friday 26 June 2009 23:53 MST

More annoying that companies who don’t believe ‘+’ in an email address is valid, are ones that did until their site got redesigned, leaving you unable to log in and having to explain the problem to people that can only read scripts. I’m looking at you, MBNA.

Update:

  CustomerService@MBNA.co.uk
    SMTP error from remote mail server after RCPT TO:
    <CustomerService@MBNA.co.uk>:
    host mbna.co.uk.s7a1.psmtp.com [64.18.6.14]: 554 No
    relaying allowed - psmtp

Update:

From: autoreply@customerservice.mbna.co.uk
Subject: Customer Service Reply

Thank you for replying to this email.

Any emails sent to this inbox are not
responded to. If you have a query about
your account please check your account
details online.

Charlie Brooker : Charlie Brooker's screen burn: 27 June 2009

Friday 26 June 2009 23:01 MST

This is not a freak show but one of the most remarkable natural history shows I've seen

If you were to compile a list of 100 things you wouldn't really want to see on TV, "watching someone methodically dissect the corpse of an elephant" would probably feature somewhere around the mid-30s point, sandwiched between "Simon Bates investigates naturism" and "toddler being sick against a butcher's shop window".

Certainly my initial reaction on hearing about Inside Nature's Giants (Mon, 9pm, C4) was one of incredulity balls-deep in glee. Once I'd got over the title, I thought: they're ACTUALLY chopping up an elephant? For an HOUR? Bless their sensationalist socks. That'll be fun to write about. Maybe they'll use a chainsaw on the trunk. Maybe there'll be a bit where they get 28 dwarves to climb inside the skin and form a human pachyderm, walking around like a giant pantomime horse while the producers play Baby Elephant Walk on the soundtrack. Maybe they'll pull one of its eyes out and demonstrate how tough it is by asking Vernon Kay to jump up and down on it 'til it bursts, except it won't burst - it'll be like jumping on a giant squash ball, so he'll slip over and land face-first in its guts.

None of that happens. Make no mistake, they take the poor creature apart. There's not a bit of that elephant you don't get to see. They pull the skin off, drag the intestines out, saw the legs into segments ... and yet, and yet ...

And yet the overwhelming sense you're left with is one of towering respect for the wonder of nature, for the excitement of science and its role in explaining the world. This is categorically not an empty freak show, but one of the most remarkable natural history programmes I've ever seen. The gore may sound off-putting but it isn't really. It's fine once you're over the initial shock - like jumping in an unheated swimming pool that feels cold for 10 seconds until your body gets used to it.

The first thing to understand is that the elephant wasn't killed for the sake of the programme. It was dead anyway. Secondly, these dissections take place regularly, for the benefit of trainee veterinary surgeons (there's a large number of them watching proceedings throughout). Thirdly, and perhaps most significantly, the programme takes each segment of the elephant - literally - and uses it as a springboard for a fairly in-depth VT sequence detailing how said part works and why it evolved that way.

So we get an entire section on the digestive system, one on the trunk, another on the feet, and so on, all illustrated with bespoke reports from Africa, archive footage, explanatory CGI animations and even Richard Dawkins, who pops up a couple of times to share his awe of nature (and appears so delighted and enthused by the process of evolution, he manages to talk for several minutes without once calling all organised religion a bastard).

At every turn, you learn new things about elephants - and not just things you didn't know, but things you hadn't even thought of questioning. Take the feet. I always thought of elephants' feet as simply being stumps with toenails. In fact I scarcely thought of them as "feet" at all, but legs that ended arbitrarily at the point they met the ground. I now know that, inside, the skeletal structure of an elephant's foot is surprisingly human. They're effectively walking around on tip-toes: the rear of each foot is a kind of fatty pad, a shock absorber, like a spongy wedge heel. It evolved to help them cope with their massive weight. That's a small example, but one that's genuinely changed the way I'll look at elephants forever. And it's precisely the sort of detail that might simply wash over you in a more traditional nature documentary.

This is a rare thing - a hardcore biological science documentary that will both entertain and enlighten almost anyone who watches.

It's also strangely moving. Because they chop that elephant to pieces all right - but they do so with palpable love. Watch it. It's amazing.

Don Marti : Productivity Paradox 2.0?

Friday 26 June 2009 21:28 MST

Auren Hoffman points out that, thanks to free libraries and tools, companies are able to do more software engineering and build more stuff with fewer people, but the salaries for those increasingly productive software people haven't kept pace. (Rands explained: "It?s this pile of high-value, well-maintained code that is helping shrink the average size of the engineering team because it?s allowing us to focus less on writing new code and more on integrating existing code to get the job done with fewer people and in less time.")

Another way to put it is that you can look at an IT project as made up of two complementary goods: code and maintenance programming. If the price of one good falls, thanks to open source and the price pressure it puts on the whole industry, then the other good's price should go up. Software company stock prices are on the way down, reflecting the lower value of a drive full of source code, but where are all the pay raises for programmers?

We used to have a productivity paradox of increasing IT spending but no productivity to show for it. Now we have more programmer productivity but without the increased pay. Auren suggests two possible explanations: offshoring and decreased need.

Yes, IT employers in the USA have an indentured servitude program in the form of H-1B visas, and yes, there's some shake-out of code monkeys if not every project has to implement every little piece. But still, more value per software developer year should mean higher wages, even if developers have to give up on constantly reimplementing the same stuff. And the H-1B program is full enough that for most raise-seeking programmers, replacing you with an H-1B employee isn't a credible threat.

Other possibilities: in most cases, the employment market for software developers is a Market for Lemons. Badly informed employers won't pay more for a value-creating developer than for a code monkey. If that's the problem, then developers who contribute to open projects and can show their contributions accepted upstream should have an advantage, just from the signaling value. "Hey, if he were a bozo, would his name show up so much in 'git log' on example.org?"

Maybe programmers just don't negotiate well. If that's the problem, we should see substantially higher salaries for less negotiation-averse programmers, such as the people who have to work both for a company and with an upstream project.

I don't know. I think we're in the process of moving from high-value software brands and mostly anonymous hackers to a system more like professional sports, where both franchises and individuals have brand names and revenue.

Martin Fowler : Agilists and Architects: Allies not Adversaries

Friday 26 June 2009 15:24 MST

Stuart Langridge : Why not to use domain sockets for a desktop CouchDB

Friday 26 June 2009 09:47 MST

The obvious idea that pops into everyone’s head, including mine, when talking about having a running CouchDB that’s specific to me is: why use TCP for it? Why not just use a unix domain socket? Then you don’t have to worry about other people on the same machine trying to access it. Everyone thinks this, and on balance it’s not the way to go. This is why.

1. You can’t browse to a unix domain socket in your web browser. This, by itself, is enough to kill the idea for me. I genuinely love the idea that applications store their data and then I can see that data in my browser using Futon, the CouchDB web UI. I can edit that data. That’s fantastic. Using unix sockets would break that.
2. One other nice thing about CouchDB is that you can do replication between two different databases; I like this because I can have the same data on my laptop and my netbook if I want. That doesn’t work if you use domain sockets, because the two CouchDBs can’t see one another.
3. As far as I can tell, Mochiweb, the underlying Erlang HTTP library that Couch uses, doesn’t do domain sockets anyway. (Obviously fixing this is only a Small Matter Of Programming.)

This has been a public service broadcast on behalf of the Write These Reasons Down So I Have Them To Hand Next Time Someone Suggests Unix Domain Sockets party.

XKCD : Overstimulated

Friday 26 June 2009 04:00 MST

Otaku : Soft asserts

Thursday 25 June 2009 17:24 MST

Allan Kelly : Agile Governance model

Thursday 25 June 2009 16:23 MDT

Coding Horror : The iPhone Software Revolution

Thursday 25 June 2009 07:59 MST

The original iPhone was for suckers hard-core gadget enthusiasts only. But as I predicted, 12 months later, the iPhone 3G rectified all the shortcomings of the first version. And now, with the iPhone 3GS, we've reached the mythical third version:

A computer industry adage is that Microsoft does not make a successful product until version 3. Its Windows operating system was not a big success until the third version was introduced in 1990 and, similarly, its Internet Explorer browsing software was lackluster until the third version.

The platform is now so compelling and polished that even I took the plunge. For context, this is the first Apple product I've owned since 1984. Literally.

I am largely ambivalent towards Apple, but it's impossible to be ambivalent about the iPhone -- and in particular, the latest and greatest iPhone 3GS. It is the Pentium to the 486 of the iPhone 3G. A landmark, genre-defining product, no longer a mere smartphone but an honest to God fully capable, no-compromises computer in the palm of your hand.

Here's how far I am willing to go: I believe the iPhone will ultimately be judged a more important product than the original Apple Macintosh.

Yes, I am dead serious. Just check back here in fifteen to twenty years to see if I was right. (Hint: I will be.)

There's always been a weird tension in Apple's computer designs, because they attempt to control every nuance of the entire experience from end to end. For the best Apple^tm experience, you run custom Apple^tm applications on artfully designed Apple^tm hardware dongles. That's fundamentally at odds with the classic hacker mentality that birthed the general purpose computer. You can see it in the wild west, anything goes Linux ecosystem. You can even see it in the Wintel axis of evil, where a million motley mixtures of hardware, software, and operating system variants are allowed to bloom, like little beige stickered flowers, for a price.

But a cell phone? It's a closed ecosystem, by definition, running on a proprietary network. By a status quo of incompetent megacorporations who wouldn't know user friendliness or good design if it ran up behind them and bit them in the rear end of their expensive, tailored suits. All those things that bugged me about Apple's computers are utter non-issues in the phone market. Proprietary handset? So is every other handset. Locked in to a single vendor? Everyone signs a multi-year contract. One company controlling your entire experience? That's how it's always been done. Nokia, Sony/Ericsson, Microsoft, RIM -- these guys clearly had no idea what they were in for when Apple set their sights on the cell phone market -- a market that is a nearly perfect match to Apple's strengths.

Apple was born to make a kick-ass phone. And with the lead they have, I predict they will dominate the market for years to come.

Consider all the myriad devices that the iPhone 3GS can sub for, and in some cases, outright replace:

• GPS
• Netbook (for casual web browsing and email)
• Gameboy
• Watch
• Camera
• MP4 Video Recorder
• MP3 player
• DVD player
• eBook reader

Oh yeah, and I heard you can make phone calls with it, too. Like any general purpose computer, it's a jack of all trades.

As impressive as the new hardware is, the software story is even bigger. If you're a software developer, the iPhone can become a career changing device, all thanks to one little teeny-tiny icon on the iPhone home screen:

The App Store makes it brainlessly easy to install, upgrade, and purchase new applications. But more importantly, any software developer -- at the mild entry cost of owning a Mac, and signing up for the $99 iPhone Developer Program -- can build an app and sell it to the worldwide audience of iPhone users. Apple makes this stuff look easy, when historically it has been anything but. How many successful garage developers do you know for Nintendo DS? For the Motorola Razr? For Palm? For Windows Mobile?

Apple has never been particularly great at supporting software developers, but I have to give them their due: with the iPhone developer program, they've changed the game. Nowhere is this more evident than in software pricing. I went on a software buying spree when I picked up my iPhone 3GS, ending up with almost three pages of new applications from the App Store. I was a little worried that I might rack up a substantial bill, but how can I resist when cool stuff like ports of the classic Amiga Pinball Dreams are available, or the historic Guru Meditation? The list of useful (and useless) apps is almost endless, and growing every day.

My total bill for 3 screens worth of great iPhone software applications? About fifty bucks. I've paid more than that for Xbox 360 games I ended up playing for a total of maybe three hours! About half of the apps were free, and the rest were a few bucks. I think the most I paid was $9.99, and that was for an entire library. What's revolutionary here isn't just the development ecosystem, but the economics that support it, too. At these crazy low prices, why not fill your phone with cool and useful apps? You might wonder if developers can really make a living selling apps that only cost 99 cents. Sure you can, if you sell hundreds of thousands of copies:

Freeverse, one of the leading developers and publishers of iPhone games, sold the millionth copy of its Flick Fishing game over the weekend, making Flick Fishing the first paid application to reach the one million download milestone. Flick Fishing, which costs 99 cents, allows iPhone and iPod touch users to take a virtual fishing trip with the flick of a wrist. The game uses the iPhone's accelerometer to recreate a casting motion, then a combination of bait choice and fishing skill helps players land the big fish.

Preliminary weekly reports for the period from 23 March to 19 April indicate that Flight Control sold a total of 587,485 units during this time. We estimate total sales are now over 700,000 units, with the bulk of sales occurring in a 3 week period. Flight Control

That's an honorable way to get rich programming, and a nice business alternative to the dog-eat-dog world of advertising subsidized apps.

I love nothing more than supporting my fellow software developers by voting with my wallet. it does my heart good to see so many indie and garage developers making it big on the iPhone. (Also, I'm a sucker for physics games, and there are a bunch of great ones available in the App Store). I'm more than happy to pitch in a few bucks every month for a great new iPhone app.

If this has all come across as too rah-rah, too uncritical a view of the iPhone, I apologize. There are certainly things to be critical about, such as the App Store's weird enforcement policies, the lack of support for emulators, or Flash, or anything else that might somehow undermine the platform as decided in some paranoid, secretive Apple back room. Not that we'd ever hear about it.

I didn't write this to kiss Apple's ass. I wrote this because I truly feel that the iPhone is a key inflection point in software development. We will look back on this as the time when "software" stopped being something that geeks buy (or worse, bootleg), and started being something that everyone buys, every day. You'd have to be a jaded developer indeed not to find something magical and transformative in this formula, and although others will clearly follow, the iPhone is leading the way.

"There's an app for that." Kudos, Apple. From the bottom of my hoary old software developer heart.

[advertisement] Interested in agile? See how a world-leading software vendor is practicing agile.

Coding Horror : Scaling Up vs. Scaling Out: Hidden Costs

Wednesday 24 June 2009 07:59 MST

In My Scaling Hero, I described the amazing scaling story of plentyoffish.com. It's impressive by any measure, but also particularly relevant to us because we're on the Microsoft stack, too. I was intrigued when Markus posted this recent update:

Last monday we upgraded our core database server after a power outage knocked the site offline. I haven't touched this machine since 2005 so it was a major undertaking to do it last minute. We upgraded from a machine with 64 GB of ram and 8 CPUs to a HP ProLiant DL785 with 512 GB of ram and 32 CPUs ...

The HP ProLiant DL785 G5 starts at $16,999 -- and that's barebones, with nothing inside. Fully configured, as Markus describes, it's kind of a monster:

• 7U size (a typical server is 2U, and mainstream servers are often 1U)
• 8 CPU sockets
• 64 memory sockets
• 16 drive bays
• 11 expansion slots
• 6 power supplies

It's unclear if they bought it pre-configured, or added the disks, CPUs, and memory themselves. The most expensive configuration shown on the HP website is $37,398 and that includes only 4 processors, no drives, and a paltry 32 GB memory. When topped out with ultra-expensive 8 GB memory DIMMs, 8 high end Opterons, 10,000 RPM hard drives, and everything else -- by my estimates, it probably cost closer to $100,000. That might even be a lowball number, considering that the DL785 submitted to the TPC benchmark website (pdf) had a "system cost" of $186,700. And that machine only had 256 GB of RAM. (But, to be fair, that total included another major storage array, and a bunch of software.)

At any rate, let's assume $100,000 is a reasonable ballpark for the monster server Markus purchased. It is the very definition of scaling up -- a seriously big iron single server.

But what if you scaled out, instead -- Hadoop or MapReduce style, across lots and lots of inexpensive servers? After some initial configuration bumps, I've been happy with the inexpensive Lenovo ThinkServer RS110 servers we use. They're no match for that DL785 -- but they aren't exactly chopped liver, either:

Lenovo ThinkServer RS110 barebones	$600
8 GB RAM	$100
2 x eBay drive brackets	$50
2 x 500 GB SATA hard drives, mirrored	$100
Intel Xeon X3360 2.83 GHz quad-core CPU	$300

Grand total of $1,150 per server. Plus another 10 percent for tax, shipping, and so forth. I replace the bundled CPU and memory that the server ships with, and then resell the salvaged parts on eBay for about $100 -- so let's call the total price per server $1,200.

Now, assuming a fixed spend of $100,000, we could build 83 of those 1U servers. Let's compare what we end up with for our money:

	Scaling Up	Scaling Out
CPUs	32	332
RAM	512 GB	664 GB
Disk	4 TB	40.5 TB

Now which approach makes more sense?

(These numbers are a bit skewed because that DL785 is at the absolute extreme end of the big iron spectrum. You pay a hefty premium for fully maxxing out. It is possible to build a slightly less powerful server with far better bang for the buck.)

But there's something else to consider: software licensing.

	Scaling Up	Scaling Out
OS	$2,310	$33,200*
SQL	$8,318	$49,800*

(If you're using all open source software, then of course these costs will be very close to zero. We're assuming a Microsoft shop here, with the necessary licenses for Windows Server 2008 and SQL Server 2008.)

Now which approach makes more sense?

What about the power costs? Electricity and rack space isn't free.

	Scaling Up	Scaling Out
Peak Watts	1,200w	16,600w
Power Cost / Year	$1,577	$21,815

Now which approach makes more sense?

I'm not picking favorites. This is presented as food for thought. There are at least a dozen other factors you'd want to consider depending on the particulars of your situation. Scaling up and scaling out are both viable solutions, depending on what problem you're trying to solve, and what resources (financial, software, and otherwise) you have at hand.

That said, I think it's fair to conclude that scaling out is only frictionless when you use open source software. Otherwise, you're in a bit of a conundrum: scaling up means paying less for licenses and a lot more for hardware, while scaling out means paying less for the hardware, and a whole lot more for licenses.

* I have no idea if these are the right prices for Windows Server 2008 and SQL Server 2008, because reading about the licensing models makes my brain hurt. If anything, it could be substantially more.

[advertisement] Interested in agile? See how a world-leading software vendor is practicing agile.

The Webbs : What I Am Currently Reading.

Wednesday 24 June 2009 06:11 MDT

Joel On Software : The eternal optimism of the Clear mind

Wednesday 24 June 2009 03:32 MST

Clear just closed down.

Here?s how it worked while it was in business. You paid $200 for a one-year membership. You underwent a big, complicated background check to prove that you were extra-super-trustworthy.

In exchange, in a few big airports, you got to skip to the front of the TSA line for screening.

Now, you didn?t skip the screening itself. You still went through the X-ray machine and had to remove your shoes, belt, pocket contents, laptops, and plastic quart ziplock bag of toiletries.

You just got to cut to the front of the line.

A few people signed up. In certain airports, it was, indeed, worth actual money to cut to the front of the line.

This wasn?t Clear?s actual business plan. The actual business plan was that Clear would do detailed background checks on travellers, who would then be trusted to bypass security completely because they were extra-super-trustworthy.

Now, the TSA doesn?t even trust pilots, who go through the same screening as the rest of us to make sure they?re not bringing something extraordinarily dangerous onto a plane like a 3.5 oz bottle of shampoo. Because, of course, with a little bottle of shampoo, they could make a bomb, which they could use to fly the plane they are piloting into a building, something that is impossible for mere pilots sitting at the controls of the jet.

So as it turns out, the TSA never actually agreed to go along with this skipping-the-screening thing, and ultimately, all Clear was allowed to do was get you to the front of the line.

At this point, and here?s the interesting part, at this point, a rational businessperson would say, ?Well, does the Clear idea still make sense if we can?t actually let you skip the screening??

OK, maybe it still makes sense to charge to skip to the front of the line. Maybe there?s a business model in that.

In that case, though, why did they still do background checks? It doesn?t make any sense.

The environment changed. It turns out that Clear?s business model of prescreening wasn?t going to be possible. But they kept doing it anyway. What kind of organizational dysfunction does it take to completely ignore the changed circumstances and keep at a money-losing business?

What?s even funnier is that Clear could probably have been profitable if they had just skipped the one unnecessarily stupid part of their business model: the detailed background checks on all their customers.

Nobody at Clear did any thinking. They had a business model, the business model wasn?t actually possible, everybody knew it, and they still plugged away at it. Thoughtless optimism. I don?t know whether to salute ?em or laugh.

Need to hire a really great programmer? Want a job that doesn't drive you crazy? Visit the Joel on Software Job Board: Great software jobs, great people.

Michele Simionato : The Adventures of a Pythonista in Schemeland/25

Michele Simionato : The Adventures of a Pythonista in Schemeland/26

Michele Simionato : The Adventures of a Pythonista in Schemeland/27

Michele Simionato : The Adventures of a Pythonista in Schemeland/28

Michele Simionato : The Adventures of a Pythonista in Schemeland/29