Friday 12 March 2010 19:13 MST
Version 1.2 of PyPy - an alternative implementation of the Python interpreter - has been released. "This version 1.2 is a major milestone and it is the first release to ship a Just-in-Time compiler that is known to be faster than CPython (and unladen swallow) on some real-world applications (or the best benchmarks we could get for them). The main theme for the 1.2 release is speed." It's still not quite ready for production use, but it appears to be getting a lot closer.Friday 12 March 2010 18:55 MST
____ _________|\___\_ |\.-------| | |\ | \\ \|_[]|\\ | \\ \\ | \\ ____________\\ | +---------------+ | | _____ ____ | TAKE THIS YOU IGNORANT MOTHERFUCKERS!!! | | |_ _|| o | | | | | | | .--' | | | |_| |_| | \// ______ _ ___ | | |Friday 12 March 2010 18:53 MST
Debian has updated Egroupware (multiple vulnerabilities) and MoinMoin (multiple vulnerabilities).
Fedora has updated nss (F12: TLS man-in-the-middle plaintext injection) and cups (fix for earlier denial of service fix).
Mandriva has updated ncpfs (multiple vulnerabilities).
Ubuntu has updated MoinMoin (multiple vulnerabilities).
Friday 12 March 2010 18:38 MST
We are pleased to announce PyPy's 1.2 release. This version 1.2 is a major milestone and it is the first release to ship a Just-in-Time compiler that is known to be faster than CPython (and unladen swallow) on some real-world applications (or the best benchmarks we could get for them). The main theme for the 1.2 release is speed.
The JIT is stable and we don't observe crashes. Nevertheless we would recommend you to treat it as beta software and as a way to try out the JIT to see how it works for you.
Highlights:
Known JIT problems (or why you should consider this beta software) are:
If you want to try PyPy, go to the download page on our excellent new site and find the binary for your platform. If the binary does not work (e.g. on Linux, because of different versions of external .so dependencies), or if your platform is not supported, you can try building from the source.
The PyPy release team,
Armin Rigo, Maciej Fijalkowski and Amaury Forgeot d'Arc
Together with
Antonio Cuni, Carl Friedrich Bolz, Holger Krekel, Samuele Pedroni and many others.
Friday 12 March 2010 18:16 MST
Over at opensource.com, OpenNMS's Tarus Balog looks at the process of starting an open source business. This article covers much of the same material as his recent SCALE 8x keynote. "You might think that I was motivated by some sort of idealistic love of open source software. Nothing could be further from the truth. At the time, I was still running a Windows desktop. I undertook the OpenNMS project because I believed one thing: in the area of network management, open source represents the best business solution."Friday 12 March 2010 18:07 MST
Marius contributed quite a few changes (restructuring to support VirtualEnv-ing, enhanced percentage reporting), the source-code-view is also there, the home-view is more likely to be reasonable (though it can still get messed up with threaded views). SquareMap is now explicitly a dependency and has been restructured to work with `setuptools develop` in a virtualenv. Available off the project page or via easy_install RunSnakeRun (note that you need wxPython installed).Friday 12 March 2010 18:00 MST
In annotation processing there are three distinct roles, the author of the annotation types, the author of the annotation processor, and the client of the annotations. The third role includes the responsibility to configure the compiler correctly, such as setting the source, target, and encoding options and setting the source and class file destination for annotation processing. The author of the annotation processor shares a related responsibility: property returning the source version supported by the processor.
Most processors can be written against a particular source version and always return that source version, such as by including a @SupportedSourceVersion annotation on the processor class.
In principle, the annotation processing infrastructure could tailor the view of newer-than-supported language constructs to be more compatible with existing processors. Conversely, processors have the flexibility to implement
their own policies when encountering objects representing newer-than-supported structures.
In brief, by extending version-specific abstract visitor classes, such as
AbstractElementVisitor6 and
AbstractTypeVisitor6, the visitUnknown method will be called on entities newer than the version in question.
Just as regression tests inside the JDK itself should by default follow a dual policy of accepting the default source and target settings rather than setting them explicitly like other programs, annotation processors used for testing with the JDK should generally support the latest source version and not be constrained to a particular version. This allows any issues or unexpected interactions of new features to be found more quickly and keeps the regression tests exercising the most recent code paths in the compiler.
This dual policy is now consistently implemented in the langtools regression tests as of build 85 of JDK 7 (6926699).
Friday 12 March 2010 17:50 MST
Friday 12 March 2010 16:01 MST
ABOUT GROUPSPACES
GroupSpaces is developing technology to help real-world groups and communities manage themselves online. Today our software helps manage over ½ million memberships for sports clubs, charities, university societies, national associations and many other groups, and we’re only just getting started.
We are located in the Old Street area at the heart of London’s tech scene, sharing cool offices with other startups including MOO, Tweetdeck and AMEE. We’re a small, dedicated team of entrepreneurs and startup folk with a great collection of VC and Angel investor backers.
If you like a challenge, have what it takes to succeed and want to help create something your friends and family can use and enjoy, come join us!
BENEFITS OF WORKING AT GROUPSPACES
* Be one of the first 10 employees, getting in at an early stage of an exciting company
* Join a small team of smart, driven, fun-loving people, passionate about helping millions of groups around the world
* Flexibility to choose the tools you want to get the job done
* Competitive salary
* Early employee stock options
* Flexible hours
RESPONSIBILITIES
You will be working within a small team to develop the GroupSpaces.com web application, with varied responsibilities including the following:
* Develop new features for the GroupSpaces.com web application, turning high level descriptions into things users will love
* Develop the supporting infrastructure and in-house development processes
* Diagnose performance bottlenecks and implement optimizations
REQUIREMENTS
* Solid object-oriented design, programming, and debugging skills
* A demonstrated expert-level proficiency in PHP 5
* A strong understanding of SQL and database design
* Experienced in the use of appropriate development tools including source code management and the Linux command line
* Track record of producing high-quality web applications through the full product lifecycle
* Excellent technical communication, both verbal and written
* Ridiculous attention to detail
* Focus on customers’ needs and desire to build technology solutions to address them.
* Degree in Computer Science or a related subject
ABOUT YOU
We are looking for the kind of people that thrive in a startup environment – those who love taking responsibility for their work, understanding and solving problems and can support their teammates as necessary.
* You are a highly motivated self-starter who can work independently with minimal supervision
* You have a 'can do' attitude and don't give up at challenging tasks
* You care about code quality and design and take pride ensuring your code is stable and secure
* You know when a project is finished and are surprised on the rare occasion that someone finds a bug once you have released.
* You are eager to learn from others and to share skills with colleagues
* You want to have fun producing software for the consumer internet market where people you know will want to use the things you create
EXTRA CREDIT
* Active leader or committee member of one or more groups, clubs, societies or other organisations.
* Previous experience in a startup or similarly agile company
* Experience with the Propel ORM framework – have you seen what’s coming with the new v1.5?
* Active in the community – open source etc.
Interested? Please fill in our short online application form at http://groupspaces.com/c/join-us/apply/
Friday 12 March 2010 16:01 MST
Friday 12 March 2010 15:54 MST
As most of you will know, Microsoft lost the patent lawsuit filed against the Redmond giant by i4i, while also losing the first appeal. Microsoft was forced to remove the infringing functionality from Word for US customers, but they also filed a second appeal. This appeal, too, has been lost.Friday 12 March 2010 15:44 MST
Based off the work Lennart did, let me introduce you to the speaker testing UI in gnome-volume-control.
Friday 12 March 2010 15:30 MST
New Zealand's internet filtering system went live last month - but the government forgot to mention this to its electorate until its hand was forced by online freedom campaign, Tech Liberty. Thomas Beagle, a spokesman for the group, said he was "very disappointed that the filter is now running" and that its launch had been conducted in such a "stealthy mode". He added: "It's a sad day for the New Zealand internet." It's sad indeed, that things like this are possible in modern democracies.Friday 12 March 2010 15:20 MST
Following in the success of the Phoronix Test Suite, last month we launched Phoromatic as a remote test management system targeted for enterprise users of the Phoronix Test Suite that allows the automatic scheduling of tests, remote installation of new tests, and the management of multiple test systems all through an intuitive, easy-to-use web interface. Whether you are looking to build a test farm or just benchmark systems around the world, Phoromatic can turn this otherwise taxing work into a really easy process with turn-key deployment capabilities. As an extension of Phoromatic, we then wrote Phoromatic Tracker that is designed to track any software component (either on a timed or per-commit basis) and automatically execute a set of tests each time around all in an autonomous matter and then pump the data back to the Phoromatic server and showcase it on the Phoromatic Tracker interface...
Friday 12 March 2010 15:09 MST
After spent some time improving cally, reviewing mx new focusable/focus-manager objects and several days configuring my environment (karmic upgrade, broken linkage in my jhbuild environment, etc) I started to check again how to use cally on gnome shell (first look here).
One of the entries in my TODO is start to make the module loading more that a hacking patch. A first solution proposal and a gratuitous rant here.
Other point on my TODO list is check why accerciser and orca froze gnome-shell. Well, accerciser still frozes the shell, but, fortunately it seems that orca now works (more or less) fine without doing anything special (black magic probably):
I know that it would be more useful with sound, but as the gnome shell screencast recording feature doesn?t record audio, and I wasn?t able to use recordmydesktop or istanbul, finally I just recorded the sound with my N900, and I was too lazy to create a video with both. If you are curious enough, you can hear the audio of the previous video screencast here.
How I run gnome-shell
During this environment configuration time, I was also looking for a convenient way to run gnome shell. On live gnome, there are two proposed options to run gnome shell:
As I said, running some of the accessibility tools leads to froze the gnome-shell. Additionally in my case, running it on xephyr had a horrible performance, so both options were not really useful to me. Finally I chose a mixed option. I just launch a second X server, and launch the gnome-shell here.
So, in brief:
xinit -- :1jhbuild shell./gnome-shell --replace
Then you can just use Ctrl+F9 and Ctrl+F7 to move between your "normal environment" and your "gnome-shell" environment. Probably someone can wonder why it is required to run gnome-session, and not execute directly gnome-shell (without the replace). Well, for any reason if I do that, the performance is also as horrible as with the --xephyr option. In the same way gnome-session load all my configuration, etc. Not a big issue anyway.
Friday 12 March 2010 14:59 MST
The Fedora board has, in response to ongoing discussions about updates to its releases (as covered in the March 11 Weekly Edition), adopted a "vision statement" on how Fedora releases should be maintained. "Stable releases should provide a consistent user experience throughout the lifecycle, and only fix bugs and security issues. Stable releases should not be used for tracking upstream version closely when this is likely to change the user experience beyond fixing bugs and security issues."Friday 12 March 2010 14:37 MST
OK, after A MONTH of trying to figure out how to get the Grails mail plugin to talk to authsmtp, and *utterly* failing, in a puddle of tears this afternoon at my 1,745th attempt at finding out anything about this on the web, I went into deep hack mode and cracked it in 15 minutes flat.
The answer… write my own f*!%$ing email service.
If you can bear to read to the end I have a bigger Grails lesson, but here’s the solution for any poor person who may be following me down this path.
First off, why AuthSMTP.com? Well, in a production envronment, for a real world application which is going to be generating email in response to user actions (such as notifications), then your gmail account will not cut it and your local ISP will almost certainly block the IP address of your server – especially if you deploy into the cloud at either Google or AWS. That means you have to have a real SMTP host to take care of the relaying and send. AuthSMTP is one of many out there, but their pricing policy is reasonable and they are production strength. I have used them for personal accounts for POP3 and they have been good, although their technical support demonstrated the worst sort of brainless responses possible and was deeply disappointing.
So, following the mantra which got me here, of “how hard can it be?”, here’s the answer…
First off go to your grails prompt and create a service with a name of your choice using
grails create-service Mymail
Next in the scaffolded code, put the following – I decided that I would just use old fashioned Java since I sort of felt I know what I was doing and I am still not comfortable enough with Groovy that I would risk chasing my own stupid errors for another month.
import java.util.Properties;
import javax.mail.internet.*;
import javax.mail.*;
class MymailService
{
boolean transactional = false
public boolean sendMessage(String to, String msgSubject, String msgText)
{
String host = "mail.authsmtp.com";
String username = "ac99999"; // your authsmtp username
String password = "xxxxxxxxxx" // your authsmtp password
String from = "no-reply@yourdomain.com";
Properties props = System.getProperties();
props.put("mail.smtp.host", host);
props.put("mail.smtp.user", username);
props.put("mail.smtp.password", password);
props.put("mail.smtp.port", "2525"); // thish is the port recommended by authsmtp
props.put("mail.smtp.auth", "true");
Session session = Session.getDefaultInstance(props, null);
MimeMessage message = new MimeMessage(session);
message.setFrom(new InternetAddress(from));
InternetAddress to_address = new InternetAddress(to);
message.addRecipient(Message.RecipientType.TO, to_address);
message.setSubject(msgSubject);
message.setText(msgText);
Transport transport = session.getTransport("smtp");
transport.connect(host, username, password);
transport.sendMessage(message, message.getAllRecipients());
transport.close();
return true;
}
}
It’s worst fault is that the thread waits for a response form the server and I confess the error handling could be added improved, and OK, you can’t attach files and you can’t send to multiple recipients, although adding that would be really simple, and you can’t do back-flips and set properties on the fly and blah blah. BUT… you can send a basic message from one email address to another, which I am betting is what 99% of automatically generated email does.
Now for the lesson, and look away if you have already been baptised by a Grails evangelist. The conclusion I am quickly coming to is that all the benefit I got on the swings of a rapid start with Grails I have more than lost on the ugly, nasty, dizzying, vomity, downright dangerous roundabout of debugging it when something doesn’t work.
Even having gone as far as downloading and installing the STS Eclipse IDE – which remains the slowest piece of software on my desktop by a factor of 5 – and figuring out how to actually debug a session, stepping through the code I found myself in a completely impenetrable marass of codeless call stacks as deep as the mariana trench. It remains about as bad a development experience as I have had in the 20-some years I’ve been doing it. The only worse thing is the silent fail of a Javascript library as the browser refuses it. It’s about time the browser was replaced with something better, but that’s a whole other rant.
Friday 12 March 2010 14:25 MST
Google has announced the release of its RE2 library under a BSDish license. "At Google, we use regular expressions as part of the interface to many external and internal systems, including Code Search, Sawzall, and Bigtable. Those systems process large amounts of data; exponential run time would be a serious problem. On a more practical note, these are multithreaded C++ programs with fixed-size stacks: the unbounded stack usage in typical regular expression implementations leads to stack overflows and server crashes. To solve both problems, we've built a new regular expression engine, called RE2, which is based on automata theory and guarantees that searches complete in linear time with respect to the size of the input and in a fixed amount of stack space." More information can be found on the RE2 project page.Friday 12 March 2010 14:23 MST
Friday 12 March 2010 14:10 MST
After knowing about Grilo, Bastien proposed to use it as a helper library to implement Rygel?s MediaServer D-Bus API spec. This D-Bus API specifies how content providers can expose content over D-Bus that Rygel can consume and then export to the UPnP world. Although this spec was developed for Rygel in particular, it aims to be generic enough so other applications could use it to consume content over D-Bus directly.
The idea is to decouple providers and consumers, this way one does not have to worry about the language they are written in and providers don?t have to be loaded in the consumer?s address space, instead they are communicated through D-Bus.
So, what could be Grilo?s role in this context? As you know, Grilo is a pluggable framework that provides a single, high-level API to consume contents from various sources (Youtube, Jamendo, SHOUTcast, etc) which are implemented as plugins for the framework. As such, the role of Grilo would be that of a content provider. Juan has been working on a daemon that would use Grilo to get access to all the content exposed by the framework and expose it over D-Bus according to Rygel?s MediaServer spec. This will enable Rygel (or any other application), to get access to all this content over D-Bus.
As of today, I think only Rygel has implemented the consumer side of this API, but the idea is that in the future other applications like Totem or Rhythmbox would have plugins to consume it too. Also, since Rygel is a UPnP server, and it is feeding on Grilo through this D-Bus interface, it is also exposing Grilo?s contents to the UPnP world, so any UPnP client should be able to access these contents over UPnP thanks to Rygel. Totem for example, has a UPnP plugin already, and because of that it can consume the contents from Grilo through Rygel even when it does not implement the consumer API of the MediaServer spec.
If you are interested in more details check Juan?s post here. There you will also find a screencast showcasing a Grilo-powered daemon that?s exposing content from various of its plugins over D-Bus, then Rygel feeding on that and exposing them over UPnP, and finally Totem consuming all the content though its UPnP plugin. Of course, you may ask why wouldn?t Totem feed on D-Bus directly, well just because it has the UPnP plugin already and not the D-Bus one, but it will come 
.
Talking about that, another thing that Bastien proposed and on which we will be working too, is a set of helper libraries to ease development of content producers and consumers. On the producer side (backends) the idea would be to hide all the D-Bus stuff and provide a clean, easy to use interface for exposing content to the bus (for those not wanting to deal with D-Bus directly), and on the consumer side (frontends), we could do the same and maybe add some extra bonus stuff, like widgets to manipulate available sources and the like.
Friday 12 March 2010 13:49 MST
Friday 12 March 2010 13:33 MST
Is there a way to find the memory usage of python processes?
Friday 12 March 2010 12:36 MST
Now that David Airlie's vga_switcheroo has went upstream in the Linux 2.6.34 kernel that provides hybrid graphics support and delayed GPU switching, David went on to look for something new to work on in his downtime when not busy with tasks at Red Hat. This new work is on GPU offloading / multi-GPU rendering...
Friday 12 March 2010 11:14 MST
Reading a couple of books from the folks at Packt Publishing. They've made some chapters available online so I figured I'd simply include the links.
Friday 12 March 2010 11:13 MST
I've just finished my copy of Python Testing: Beginner's Guide, by Daniel Arbuckle. While I'll fully admit I didn't type each and every code sample into a text editor, I did read this one cover-to-cover. Overall, it serves as a good introduction to Test Driven Development from a Python angle.
Friday 12 March 2010 09:54 MST
I?m no audiophile, but I?m in absolute lust over these ceramic speakers from Joey Roth.
Do want.
(via GeekBrief.tv #720)
Friday 12 March 2010 09:30 MST
I’m back on Shark, after a four month hiatus. A minor milestone: it can build itself again.
Friday 12 March 2010 06:00 MST
Following the release of Fedora 13 Alpha this week we delivered Intel graphics benchmarks looking at the performance of an Intel Atom Netbook using the very latest kernel, DRM, and Mesa packages that Fedora is known for carrying. There are regressions in the Intel stack worth noting, but in this article, we are continuing in our Fedora 13 benchmarking by looking at the general system performance of the Linux desktop.
Friday 12 March 2010 05:00 MST
Friday 12 March 2010 02:10 MST
I just wanted to let you good people of the Internet know that I will be delivering a brand new talk entitled Unwrapping The Community Manager at OSBC in San Francisco at 11.40am on Thu 18th March 2010.
In the presentation I will be deconstructing the role of community manager and discussing:
I will also be sharing my experiences working with Ubuntu and other organizations. I hope to see some of you there!
Friday 12 March 2010 01:10 MST
...I would be demanding this to the EU right now:
The misconception here is that the so called ballot only fixes a symptom of a deeper and more important problem. The real problem here is Microsoft's monopoly over the consumer computing market, and the root cause of that monopoly is their draconian deals with the OEMs. I don't even have the choice to buy a Windows-less laptop, let alone a Linux preloaded one.
The EU should act immediately on that situation, and it is outrageos that it allows it while at the same time, they put so much energy in deciding what applications goes inside of Microsoft's own product.
I'm against this, because I think it totally goes against good principles when it comes to designing a good user experience for a product and I somewhat illustrate that with the picture above. By the way, who chooses which browsers goes there? Based on which criteria? How can the EU citizens revise that process?
I'm against the EU having the power to decide which apps or user workflow goes in a software product, I find it outrageous on Windows, I would find it outrageous in Mac OS X and I would find it outrageous in a Linux distribution.
IE embedded in every copy on Windows is not the problem a copy of Windows embedded with every single computer we buy is.
(well, and maybe IE's total lack of respect for web standards up to some extend too)
How about this ballot for a real change:
Friday 12 March 2010 00:40 MST
With a duo of fixes in JDK 7 build 85, one by Jon
(6927797)
and another by me (6926703), the langtools repository has reached another milestone in testing robustness: all the tests pass with assertions (-ea) and system assertions (-esa) enabled. This adds to other useful langtools testing properties, such as being able to successufully run in the speedy
same vm testing mode.
Jon's fix was just updating a test so that some code would always be run with assertions disabled while my fix corrected an actual buggy assert I included in apt. Addressing such problems helps simplify analyzing test results; if there is a failure, there is a problem!
These fixes have also been applied in the forthcoming OpenJDK 6 build 19 so it too will have the same assertive testing quality.
Thursday 11 March 2010 22:59 MST
Common wisdom has it that Flash is a resource hog, and that HTML5 will prevent your processor from having to work really, really hard to show animations of videos. Well, a number of people have conducted benchmarks with the latest browsers and Flash betas, and common wisdom is starting to show serious signs of crackage.Thursday 11 March 2010 22:43 MST
Or “things you would do to not use Windows”...
Here is the recipe:
1. A decent Operating System with a sane Desktop Environment (in other words Fedora 12 + Gnome because it has gvfs, which is the coolest thing in the World)
2. VPN/SSH access to a Linux host that has a share to all the Windows based toolchain.
3. Some scripts to automate compilation for the test files (in my case, all the usual make machinery for Jamaica).
4. A Windows shell that is used only to invoke the build machinery (and only that!).
5. Samba/NFS to share the sane Linux directories with the borked Windows machine.
6. FTP whatever to upload the OS9 binaries to the target device.
7. MAUI and OS9 documentation.
Preparation (for 1 person):
Just open a connection with nautilus to the remote Linux machine. On a separate terminal ssh into the same Linux machine and link the toolchain header files directory on a directory that is local to the mounted home (ugh, that‘s a complex sentence). This is needed only if the toolchain location is not on the local machine (like in my case is mounted via nfs), the reason is that the gvfs share will only see the remote Linux machine but not all that is mounted remotely there, so the trick of linking is necessary. Create the project in NetBeans telling it that the location of the header files is the shared folder that was made visible via gvfs. Edit code as necessary. Now the tricky part. When you compile huge amount of code gvsf crashes because NetBeans polls every file that is modified and make modifies them (it set the timestamps!) so never, ever ever, never use the same virtual desktop to edit code and to issue the make command. Just go back coding when make finishes (it‘s windows, so is supposed to be slow). Remember this again, don‘t focus NetBeans when windows compiles code. I think this is a weird bug in gvfs, and hopefully it will be fixed. If gvfs crashes, you need to restart and reaload the project in NetBeans, which takes more time that waiting for the compilation to finish… Ah, make the customers that provide a windows only toolchain to pay twice is a good thing to do also.
End result: Jamaica runs on OS9 with MAUI :)
Thursday 11 March 2010 22:17 MST
Thursday 11 March 2010 18:56 MST
Version 0.4.2 of the little-known microkernel-based multiserver operating system HelenOS has been released. See their official release notes to find out what other bleeding-edge features besides an experimental and highly modular networking stack this release brings.Thursday 11 March 2010 18:55 MST
Motorola will start loading Microsoft's search and map services onto its Android smartphones in China, bringing more non-Google services to the phones amid a row between Google and China.Thursday 11 March 2010 18:55
After weeks of e-mails and days of phone calls, yesterday I drove to San Jose for a demo of Michael David Crawford's secret project. Michael was eager to show off his work, but urged me not to share my interest with the K5 community.Thursday 11 March 2010 18:00 MST
Encryption on the Internet has come a long, long way from the oft-ignored little yellow key in the lower left corner of your Netscape Navigator status bar. Today, cryptography is a vital part of all of our Internet lives, whether we realize it or not. Now, if you’re reading this article on Nerdland, chances are that you’re well aware of that, and I don’t need to explain why you need to be sure your online banking is done over an HTTPS connection, and why connecting your laptop to an open, unsecured wireless network is usually a bad idea.
But the little stuff can trip you up just as easily, and if you don’t have a solid understanding of the different facets of cryptography, you may well think that a system meets your security requirements when it does not. After all, modern cryptography is just mathematics. There’s no inherent application for it. Security isn’t a tangible property either; it’s an umbrella term for a whole class of goals. Rather, privacy, authentication, identification, trust, and verification — mechanisms of applied cryptography — are what provide the most commonly desired types of security. Understanding what these terms really mean, how they are implemented, and how they are different is essential to a true understanding of how encryption works to assure your security on the Internet, and even within a single computer.
This article assumes you are familiar with the fundamentals of cryptography: that you know what constitutes encryption, that you know what a key is, and that you know the basic difference between symmetric key cryptography and public key cryptography. I am concerned with describing and clearing up some misconceptions about the practical applications of cryptography to modern computing.
Privacy (or “secrecy”) is the cornerstone of applied cryptography. A commonly desired form of security is making data readable only by certain intended recipients. Whether symmetric or public key cryptography is in use, a person (or machine) proves that they are an intended recipient by possessing the key that can be used to decrypt the message. In the case of simply achieving privacy, it really doesn’t matter whether symmetric or public key encryption is used; public key encryption is very slow, so in practice, it’s only used to encrypt a symmetric key that is used to encrypt the rest of the data.
Privacy is commonly desired when sensitive data is being transmitted. In the case of web browsing, this is one of the purposes of the Secure HTTP (HTTPS) protocol. When communicating with, for example, your bank’s website, it is important that the information being transacted is private. It is highly undesirable for any other person, even a professional network administrator at your ISP, who happens to control a computer on the Internet through which the data between you and your bank passes, to be able to look at your account numbers and balances.
Similarly, if you store sensitive corporate information or highly personal documents on a laptop, you would want to make sure that these documents remain private if the laptop were ever lost or stolen. For this, you would encrypt the files (or better yet the entire hard drive) and either keep the decryption key outside of the laptop, or keep it protected with a strong passphrase. In the latter case, the passphrase itself is the key to a cryptographic algorithm will provide the unencrypted version of the decryption key for your files or hard drive, and the passphrase is ideally stored only in your head.
This is privacy: no third parties can read your data. No more, and no less. A common problem is that users, even technically savvy users, often make the false assumption that privacy implies authentication and verification. While the ability to create privacy is a prerequisite for authentication and verification, and they are often used in conjunction, it is not the case that obtaining privacy implies that the other two types of security have also been obtained.
Authentication is the act of proving who you are, or challenging someone else to prove who they are. The underlying technology for modern authentication schemes is public key cryptography. I said earlier that I was assuming familiarity with public key cryptography, but let me reiterate the most salient aspect of it for the purposes of authentication: In public key cryptography, only Alice’s private key is able to decrypt messages that have been encrypted with Alice’s public key, and only Alice’s private key is able to create encrypted messages that can be decrypted by Alice’s public key. Specifically, a message encrypted with any other private key will produce different (usually meaningless) unencrypted data if Bob attempts to decrypt it using Alice’s public key.
The fundamentals of authentication consist of a challenge-response exchange. If Bob presents (“challenges”) Alice with a piece of arbitrary data, and Alice responds with a piece of encrypted data that decrypts to Bob’s original arbitrary data when decrypted using Alice’s public key, this proves that Alice possesses Alice’s private key. Nobody else other than the person who possesses Alice’s private key (presumably only Alice) could produce encrypted data that would decrypt back to Bob’s initial data using Alice’s public key. If Bob presented Mallory with arbitrary data, and Mallory wanted to impersonate Alice, he could not; without Alice’s private key, he would not be able to produce the expected response that Bob was looking for.
It is clear from this, however, that authentication is only useful if you already know the public key of the person you are hoping to communicate with. One common application of cryptographic authentication on computer networks is Secure Shell (SSH) logins. Commonly, a user will install his or her public key on a server that they wish to log into via SSH, and will keep his or her private key on a personal machine. When logging into the server, the server challenges the client to prove that it holds the private key corresponding to the username that the client is trying to log in as. If the client satisfies the challenge with an appropriate response, the login is allowed without requiring a password for the user.
This is more secure and often more convenient than prompting for a password, since the private key is much harder to steal or guess than a password, and the same public key can be used on multiple servers with none of the security risks that apply to re-using the same password in multiple places. The same sort of thing can be done with web servers using something a little more complicated called a client-side certificate (see below about certificates), although these are uncommon on the public Internet and more often used on corporate intranets.
This is authentication: you can know with certainty who you are talking to. That is all; no more, no less. Note that this carries no implication of privacy. It is perfectly possible to authenticate your counterpart in a conversation and then proceed to have a non-private conversation. That wouldn’t be a common choice, but there’s nothing that prevents it.
More importantly, it is perfectly possible to have a private conversation without authenticating your counterpart. This is where a danger of a false sense of security lies. Bob could be talking over a perfectly private, encrypted connection, but if the person on the other end is Mallory and not Alice, Bob would never know that he is sending his sensitive data to, or receiving critical information from, a different and potentially malicious person.
In other words, just because you are sending your credit card number over a private, encrypted connection, doesn’t mean you aren’t unknowingly sending it directly to a criminal.
Identification is the aspect of applied cryptography that addresses the flaw in the above-described authentication process wherein you must know a priori the public key of the person you wish to communicate with. Perhaps surprisingly, this is the most complex common application of cryptography to security. If Alice and Bob wish to authenticate each other over the Internet, they must first exchange public keys. But they can’t just send them to each other over the Internet! If Bob received a message that purports to be from Alice and to contain Alice’s public key, he has no way to authenticate that the message actually came from Alice (and not from Mallory pretending to be Alice) without already knowing Alice’s public key. It’s a chicken-and-egg problem.
The direct solution to the problem is for Alice and Bob to exchange public keys off-line; to meet at Starbucks and hand each other CDs with their respective public keys on them. But this is not practical if Alice and Bob live thousands of miles apart, it is not practical if Alice is a banking institution and not a person, and it is still not practical if Alice and Bob do not already know each other.
If Alice and Bob are strangers (but still wish to authenticate one another) meeting to exchange CDs at Starbucks still, even if physically feasible, still isn’t secure. Mallory could show up at Starbucks a few minutes before Alice and, pretending to be Alice, give her public key to Bob, and now Bob will authenticate Mallory as Alice in future conversations. A way to fix this loophole is to have Bob check Alice’s driver’s license before accepting the CD. This is identification: you can know that a public key purporting to belong to a particular person or entity actually does.
Now, meeting in person and checking driver’s licenses is a human solution to a computing problem. There are of course computer-based solutions to this same problem that will also avoid the impracticalities of first having to meet in person with everyone whom you wish to authenticate later. But these solutions are based on the same principle as the driver’s license check: trust. The reason that Bob is willing to accept Alice’s driver’s license as proof that Alice is who she says she is because Bob trusts that the state government would not issue a license in a false name or with a false photograph (ignoring for the moment the possibility that the license itself is a fake and not issued by the state). Computational identification is based on the same notion of trust.
Ultimately, to accept that a public key belongs to the person it claims to, you must trust that it does. Trust can be simple, if for example the key was given to you in person by your friend Charlie who you are sure is not being impersonated by a shape-shifting alien. Trust can also be more indirect. If Charlie gives you his brother Dan’s public key, and you trust your that Charlie is honest and has good reason himself to trust that the key legitimately belong to Dan, then you can accept Charlie’s assertion that the key belongs to Dan as identification of Dan’s public key.
Computationally, this identification process is based on signatures and certificates. A certificate is like a driver’s license: it identifies a public key as belonging to a named individual, entity, company, or organization. The fundamentals of a certificate are simple. The person wishing to be certified generates a file with their identifying information (in a standardized format), and appends to it their public key. That’s all. But, of course, this certificate is worthless without trust. If a stranger just handed me a card saying “I am Alice, my public key is …”, I would not accept that as their identification, would you?
To be worth anything, certificates must be signed. I’ll get to the mechanics of signatures in the next section, but suffice to say that the goal of a cryptographic signature is to use a private key to produce a non-forgeable endorsement. If Dan produces a certificate for himself, and Charlie signs the certificate using his own private key, this functions as an assertion by Charlie that the contents of Dan’s certificate are accurate. Then, since I already trust my friend Charlie, Dan can simply present me with the signed certificate containing his public key to identify himself to me. I can check Charlie’s signature against Charlie’s public key (which I already have), and from that know that Charlie asserts that Dan’s certificate is accurate, and therefore that Dan’s purported public key actually belongs to him.
This is trust: you can know that a public key belongs to who it purports to by means of endorsement by a third party. What’s important is that this can all be done without ever actually contacting Charlie, beyond once to obtain and identify his public key in the first place.
Further yet, let’s say that Erin presents a certificate with her public key to me and this certificate is signed by Dan. If I trust that Charlie would only sign Dan’s certificate if Dan himself were trustworthy, then I can trust that Erin’s certificate is valid as well. This sort of peer-to-peer trust acquisition, where an identity certificate can be signed by any number of other individuals who trust the holder (with varying levels of expressed trust), is known as a web of trust, and is commonly used for personal communications amongst security-sensitive Internet users.
But most Internet users never encounter a web of trust explicitly, and don’t really need to know how it works. What they do encounter frequently, however, is the similar notion of a public key infrastructure. This is used to establish Secure HTTP (or, more generally, TLS) connections. When establishing a secure connection to, say, Bank of America, it really does no good just to make the connection private. You must authenticate that the server you are communicating with really does belong to Bank of America. The server will send your browser its public key for authentication, but in order for the authentication to mean anything, the public key itself must first be identified. To facilitate identification, the server will send you a certificate.
In order to be identifiable, the certificate will be signed by a “certificate authority“. A certificate authority is a company who sells certificate endorsements and who has the responsibility to do whatever is necessary to assure that the contents of the certificates they are signing is truthful. Part of this process may be to ask for a faxed-in copy of a driver’s license, or to call the company’s well-known phone number and check with their IT department. The price of the endorsement can itself be a means of ensuring that an applicant is not fraudulent; a large company will have no problem paying over a thousand dollars annually for an endorsement, but to a small-time impersonator, this might be prohibitive.
A public key infrastructure (PKI) differs from a web of trust in two major ways. First, in a PKI, a certificate is signed by only one endorser, while in a web of trust a certificate may have multiple endorsers. Second, while in a web of trust a user is interested in tracing the endorsement chain back to someone that he or she knows personally, in a PKI the browser is interested in tracing the endorsement chain back to a “root” Certificate Authority. What makes a certificate authority a functional “root” in the context of HTTPS is that the root authorities’ certificates and public keys are pre-installed in the browser, and signed only by themselves. And so, ultimately, you are trusting that the manufacturer of your browser (Microsoft, the Mozilla foundation, Apple, Google, Opera, etc) is pre-installing root certificates only for trustworthy certifying authorities.
By now, you should know enough about privacy, authentication, and identification to understand what those HTTPS certificate error messages you receive from your browser mean. A browser error or warning message about an HTTPS certificate almost always indicates that a problem was encountered while attempting to use the certificate to identify the remote server (the actual authentication or encryption of the data almost never fails). The most common errors encountered are that a certificate has expired, or that a certificate’s chain of endorsements cannot be traced back to a known root certifying authority. A special case of the latter is a self-signed certificate, which is not signed by any certifying authority, root or otherwise.
These errors are important because they mean that the certificate presented by the server cannot be trusted as identification. You should afford them the same level of trust as identification that you would afford the “I am Alice” card that was handed to you; that is to say, none. And without identification of the public key, any authentication you attempt to perform on the remote server is equally worthless. The person handing you the “I am Alice” card could easily be Mallory and you would never know the difference. Note, however, that this says nothing about the compromise of privacy.
An HTTPS (or TLS) connection using an expired, self-signed or otherwise untrusted certificate allows for private communication, but does not provide authenticated communication.
That is, your data is protected against third-party snoopers on its transit through the Internet, but it is most certainly not protected against your counterpart being a malicious imposter.
I took so much space writing about trust and certificates largely to get to that point, because it is perhaps the most widespread and dangerous misconception about cryptography on the Internet. It is perfectly possible to have a cryptographically private conversation with a cryptographically unauthenticated, unidentified, and untrusted server. Just because you have obtained the “privacy” form of security does not imply that you have all of these other forms of security that you may also desire, so you shouldn’t assume that you do.
This will almost seem like a post-script considering how simple it is compared to identification and trust, and really it should logically appear between identification and trust, since it is the basis for signatures, but I didn’t want to break up the narrative.
Above, I glossed over the fact that a person (in a web of trust) or a certifying authority (in a public key infrastructure) is able to endorse a certificate by “signing” it. But what does that mean, exactly? Cryptographic signatures provide verification, the final common form of cryptographic security in modern computing.
Suppose that Bob writes a will leaving half his estate to Alice and half to Charlie, and disinheriting Mallory. Suppose then that Mallory sneaks into Bob’s home office, finds his will in his desk drawer, and modifies it such that it now leaves the entire estate to Mallory and disinherits Alice and Charlie. When Bob dies and the will is read, how can the executor verify that the will is what Bob wrote and has not been tampered with? In this non-computing situation, the will will have been signed by a witness or a notary public, and the executor will trust the witness or notary to inform him if the document differs from the document that they signed.
In computing, things work essentially the same way. If an e-mail (or document, or certificate) needs to be verified as having not been tampered with, it will be cryptographically signed, and the public key of the signer will be used to verify that the contents of the e-mail, document, or certificate have not changed since the signature was applied. This is verification: you have assurance that the data has not changed since a trusted party signed it. Again, don’t infer that this means more than it does. The document need not be private, and it is important that the signature be authenticated with an identified, trusted key in order to mean anything.
The mechanics of a cryptographic signature are simple. First, a cryptographically secure hash function is applied to the document to obtain a relatively short sequence of bytes. Normally, the function used today is SHA-1. The important part about the sequence of bytes produced is that it would be incredibly difficult to create a meaningful document with different contents which would generate the same sequence when the cryptographic hash is applied to it. The output of the cryptographic hash is then encrypted using the signer’s private key and attached to the document.
The recipient can then use an identified and trusted public key belonging to the signer to decrypt the output of the cryptographic hash. If the recipient re-computes the hash on the data and compares it to the decrypted hash output, he can be assured that the document was not tampered with if the outputs match. In the case of certificates, the certifying authority’s signature of the certificate verifies that the identifying information contained within the certificate has not been altered since the time at which the certifying authority validated that the information was true.
In cases other than certificates, for example documents and e-mails, data is usually signed by its own author. For example, Alice sends an e-mail to Bob and signs it with her own private key. Then, presuming Bob already has an identified, trusted copy of Alice’s public key, he can not only verify that the message has not been tampered with, but he can also authenticate Alice as the author of the message, since no one but Alice could have produced a signature that would decrypt properly using Alice’s public key. If the message or document were signed by someone other than Alice, Bob would have to trust that the signer was being honest when endorsing that the message came from Alice.
What’s important to note is that if Alice simply sends a private message to Bob, this provides neither verification that the message has not been altered nor authentication that the message is actually from Alice. When Alice sends a private message to Bob, she encrypts it using Bob’s public key. This provides privacy and ensures that only the intended recipient (Bob) can read the message. But to provide verification and authentication, Alice must also sign the message with her own private key.
Hopefully, this article has helped the reader understand the similarities, differences, and interrelations between the five most common applications of cryptography to modern computing. To wrap up, I’ll repeat the most salient points about each:
Ideally, you want all of these things at once, and that’s exactly what HTTPS (or other protocols on top of TLS) give you. That’s why it’s completely secure to give your credit card number and personal details to a bank or other merchant over the Internet, so long as you are using HTTPS and you are not otherwise worried that the bank or merchant will misuse or mishandle this information in some way completely unrelated to having transmitted it over the Internet.
The certificate given by the web server is trusted by your browser because it is identified by a certificate which has its contents verified by a certifying authority’s signature. Thus, the certificate can be used to authenticate that you are communicating with the server that the certificate describes. Once all of that that is ascertained, the cryptographic key in the certificate is used to ensure that the conversation between you and the web server is private with respect to third parties along the route of data transit.
But, of course, to be truly secure, all of these aspects must be present, and a savvy Internet user must recognize that an HTTPS error displayed by the browser indicates that that is not the case. Moreover, when using or devising security systems that are not as well automated as TLS, one must be sure that each desired aspect of security is in place, and not make the assumption that one aspect implies the others, which is most certainly not the case.
Thursday 11 March 2010 17:30 MST
An opportunity to join a UK software product company that is a leader in the fastest growing IT sector of mobile computing. The company is going through a rapid stage of growth and is seeking high calibre individuals who relish the challenge of working in a fast moving and dynamic environment. You will be part of a highly skilled team that is empowered to come up and follow through with innovative ideas, features, designs, process improvements etc. The team use Agile methodology to deliver exciting regular new product releases of very high quality. We are seeking a Senior .NET 3.5 Software Engineer who is motivated and has the flexibility to adapt the changing situations and circumstances that arise in a young company going through rapid expansion.
You will join a highly skilled Product Development Team designing and developing the core technology that enables the remote management of mobile devices. You must have excellent commercial experience of designing and developing software with Microsoft development technologies including .NET, Web services, ASP.NET and C#. You must be experienced in OO and design patterns and demonstrate a good understanding of Enterprise system requirements including scalability, resilience, reliability, etc. Extensive experience and understanding of the full software development lifecycle, unit testing, build automation and continuous integration is also required. B2M offers an excellent package including equity which is awarded through the tax favoured Enterprise Management Incentive ("EMI") Share Option Scheme.
See http://www.b2m-solutions.com/about/vacancies.asp for more details
Interested? Email to careers@b2m-solutions.com or call our HR Consultant, Teresa Hughes on +44 (0)1235 432754
Thursday 11 March 2010 16:57 MST
While we are still waiting for the hardware drivers to mature for Gallium3D (particularly the Nouveau stack for all NVIDIA GPUs, the Intel 965 driver, the ATI R300g driver to mature, and then the R600g driver to come about), VMware has been working on their a software rasterizer as well through a Gallium3D state tracker. This new software rasterizer looks like it's finally coming about and is already delivering great performance compared to Mesa's existing software rasterizer that is rather crippled...
Thursday 11 March 2010 16:46 MST
Back in 2007 we reported on Valve looking for a senior software engineer to port their Windows-base games to Linux, then in 2008 we said the Source Engine would be coming to Linux based upon our sources (something that we still believe in), later that year we also found a few Linux libraries with the Left 4 Dead game...
Thursday 11 March 2010 05:13 MST
Here is a very interesting take on the concept of Integrated Development Environment.
As opposed to traditional IDE’s, which work at the same level as the Java language itself (classes and packages), this IDE, called Code Bubbles, allows you to work at a much finer granularity: methods, fragments of code and whatever you need for the resolution of a specific task. All these tasks are linked to each other in a workspace, thus allowing you to stay focused only on what is relevant for your current task.
Of course, the concept is not new since it’s exactly what Mylyn is trying to achieve, but to be honest, every time I’ve tried to get into Mylyn (and I tried several times over the past years), I ended up giving up in frustration. This is not to say that Mylyn is a bad product, just that retrofitting such an idea on a traditional IDE, no matter how flexible, is probably impossible.
Still, I can’t shake this impression that it should be possible to mix both approaches, and considering the mindsharing that Eclipse has, being able to offer an intuitive and lightweight add-on that would enable the kind of unit of work granularity that Code Bubbles enables could be very interesting.
And this thought led me to git, but I’ll need to make a digression first.
One of the strengths of git is its branching model: branches are so cheap that you find yourself branching all the time and then switching, merging and committing very often.
Another interesting aspect of source control systems (not limited to git) is that the diffs that you are creating capture the unit of work that is relevant to you. And a git branch is actually very similar to a Code Bubbles Workspace.
So how about an Eclipse perspective that would be based on git branches?
The perspective wouldn’t just show the diffs, an information that is in itself not very interesting, but it would be a bit smarter than that and be able to infer that if you modified a couple of lines in the method init(), the that whole method should become a bubble in that perspective. Intelligent linking between bubbles could also be provided by looking at the chronological order in which the methods have been edited: git would only know that you added two lines in the method init() and that you then renamed a field in the class Foo, but the perspective would note that the two events are related since they followed each other, and it would reflect this by linking the bubbles.
Thoughts?
Thursday 11 March 2010 04:49 MST
This article describing the early days of Windows was a very interesting read, and even though I know this part of the computer history pretty well, I did learn a couple of things that I’d like to share.
The first is that after Windows 2.0 came out, Apple sued Microsoft for copying the look and feel of its MacIntosh:
In 1988, Apple decided to sue Microsoft over Windows 2.0?s ?look and feel?, claiming it infringed on Apple?s visual copyrights. Having been a principal manager in charge during development of Windows 2.0, I was now caught up in the maelstrom and over the next year I got a thorough education on the US legal process as I briefed the Microsoft legal team, created exhibits for them, and was grilled by deposition by the other side. To me the allegation clearly had no merit as I had never intended to copy the Macintosh interface, was never given any directive to do that, and never directed my team to do that.
Interestingly, the suit ended up being dropped because:
Apple had previously granted a license to Microsoft to use any part the interface included in its applications for the Mac.
I’m guessing a few heads must have rolled in the Apple legal department when they realized that they filed a suit that they had already signed themselves out of.
But the more interesting part comes in the next paragraph:
However, I can recall that within my first year at Microsoft, Gates had acquired a Xerox Star, and encouraged employees to try it out because he thought it exemplified the future of where the PC would be headed and this was long before Microsoft even saw a Mac or even a Lisa from Apple. Gates believed in WYSIWYG (What You See Is What You Get?i.e. fidelity between the screen and document output) and the value of a graphical user interface as far back as I can remember. And prototypes of Windows existed long before the first appearance of the Macintosh.
Intrigued about the timing, I did some digging and I found out that Gates bought that Xerox Star in 1981:
Among the developers of the Gypsy editor, Larry Tesler left Xerox to join Apple in 1980 and Charles Simonyi left to join Microsoft in 1981 (whereupon Bill Gates spent $100,000 on a Xerox Star and laser printer)
This was just a few months after Steve Jobs himself got his epiphany about graphical user interfaces and the mouse during a visit to Xerox PARC:
In the early 1980s, Jobs was among the first to see the commercial potential of the mouse-driven graphical user interface
This story about Steve Jobs is well known but the fact that just a few months later, Bill Gates himself envisioned the same future of computing is news to me.
Wednesday 10 March 2010 19:29 MST
I’ve blogged in the past about my exploits with MythTV, and playing around with Personal Video Recorders. Over Christmas we bought ourselves a new TV and DVD player, to replace our aged tv, freeview box and dvd player, all of which had faults. However, our video recorder is also showing some signs of age – it was a wedding present – and I recently saw that eBuyer now have the Acer Revo “nettop” PCs for only £120, and I began investigating the world of PVRs again.
Here’s what I would really like:
Other items (picture viewing, other video plugins, etc, etc) are a bonus that is a consideration, but not a deal breaker.
So I started to look around at some of the current popular offerings. As far as I can tell, MythTV is still the one to beat for straight PVR, although there’s a certain amount of confusing online as to whether or a not a Revo could handle it as a combined backend/frontend. (The dual-core model seems like it might, unsure about the ‘bargain’ box).
When it comes to the online content, there are 3 big boys out there – XBMC, Plex (which is only for MacOS) and Boxee. I’ve looked at all three, and here are my conclusions:
It had a whole load of “social” stuff built into it, but, frankly, I don’t think I want/need that. (Reminds me a bit of the Flock browser)I _want_ to like Boxee, but I find myself leaning towards XBMC more at the moment. Alternatively, there are a load of plugins for MythTV itself, so maybe that is a better option?
I’d love to hear from anybody who has similar needs, and also anyone who’s running Myth on a Revo.
mrBen
Wednesday 10 March 2010 18:55
Two trends in automotive development, value engineering and drive-by-wire have the potential to reduce driver safety. Toyota's recent travails provide some food for thought.Wednesday 10 March 2010 16:57 MST
When making a programming tool or a virtual machine getting the tool running perfectly stable without any crash bugs are always on a higher priority than gaining more speed. A crashing tool are a broken tool so I will share some tricks that I have practised to find and fix Shark LLVM JIT CodeGen crash bugs. The main trick are to be able generate reproducable testcases that can be reported to the LLVM developers bugzilla bugtracker by using what you can extract from the Shark LLVM JIT CodeGen crashes. Here is how I do it, enjoy!
How to provoke hard to find Shark LLVM JIT bugs
Some Shark LLVM JIT bugs are hard to find because they only occour after the Shark JIT enabled JVM have been running for a long time, this are because the Shark Hotspot JVM takes advantage of the fact that a given running application spends about 90% of its time running only 10% of the applications code. Hotspot profiles the running code and only JITs the most frequently used methods of the program. Hotspot uses a threshold to determine which methods to JIT. When a method have been used more than 100000 times then it are scheduled to be optimized by the JIT. JIT bugs can stay undetected if they are located in unfrequently executed methods, those methods that makes up the 90%, of the unfrequently executed application code.
A easy trick to provoke unfrequently executed JIT bugs are to lower the JIT threshold in Hotspot so that Hotspot JITs everything. The JIT threshold can be controlled by using the -XX:CompileThreshold=1 option and -Xbatch option. -Xbatch prevents the hotspot from running the JIT in background and will make hotspot reproduce JIT bugs more determistic.
Using a low JIT threshold will of course make the program startup magnitudes slower but it will also eventually find and hit all JIT bugs for a given application. Try pass -XX:+PrintCompilation to Hotspot as well so that you can observe all the java methods that Hotspot are JITting and find out which method that failed to JIT if Hotspot hits a JIT crash bug.
java -XX:CompileThreshold=1 -Xbatch -XX:+PrintCompilation JavaApplication
1 b java.lang.Thread:: (49 bytes)
…
10 b java.lang.String::getChars (66 bytes)
*crash*
/home/xerxes/llvm/include/llvm/CodeGen/MachineFrameInfo.h:289: int64_t
llvm::MachineFrameInfo::getObjectOffset(int) const: Assertion
`!isDeadObjectIndex(ObjectIdx) && “Getting frame offset for a dead object?”‘
failed.
Huh.. no logfile??
Most Shark LLVM JIT CodeGen crash bugs makes the JVM instantainiously exit without producting a hs_err_pid*.log file. Whats usefull are that the JVM output will contain a Assertion, Unreachable or Unimplemented keyword and a LLVM code line numer.
So what do we do now?
Thanks by using -XX:+PrintCompilation makes us aware that the last method JITed was the java.lang.String::getChars method and that caused the Assertion in the LLVM CodeGen when running the Shark JIT so the next step are to dump the LLVM IR that Shark have generated for the method.
Extract the LLVM IR for the java method that makes the Shark JIT crash.
Ok so we got a crash and we know that it was JITing of java.lang.String::getChars that caused it.
a) shark debug build -XX:SharkPrintBitcodeOf= method:
If you have built a debuggable “Mixtech” Shark build then Shark will contain some extra usefull debug runtimeoptions where one of the more usefull are
-XX:SharkPrintBitcodeOf=java.package.name::MethodName
use it and Shark will dump the LLVM IR bitcode to stdout just before jitting it.
b) gdb call F->dump() method:
I personally prefer dumping LLVM IR from inside the gnu gdb debugger since this method can be used using release Shark build in combination with release llvm builds so lets jump into the gdb debugger!
Start gdb and attach it to the java application with all the options that triggered the JIT CodeGen bug!
$ gdb -args java -XX:CompileThreshold=1 -Xbatch -XX:+PrintCompilation JavaApplication
(gdb) run
...
Segmentation fault
$
Ick gdb crashed why? This are because the JVM launcher “java” first sets up the system environment and then forks off in a new process using execve(). gdb gets killed by the linux kernel when it are trying to read memory across process boundarys so we must stop java from forking!
The easiest way to prevent java from forking are to setup the system environments before launching the application. And all this can be done from inside gdb so lets try again!
$ gdb -args java -XX:CompileThreshold=1 -Xbatch -XX:+PrintCompilation JavaApplication
(gdb) break execve
Breakpoint 1 at 0x93b8
(gdb) run
(gdb) call puts(getenv("LD_LIBRARY_PATH"))
/media/disk/4mar-shark-1.8pre-b18-llvm-2.7svn.so-npplugin/jre/lib/arm/server:/media/disk/4mar-shark-1.8pre-b18-llvm-2.7svn.so-npplugin/jre/lib/arm:/media/disk/4mar-shark-1.8pre-b18-llvm-2.7svn.so-npplugin/jre/../lib/arm
$1 = 220
Ok now we know what the LD_LIBRARY_PATH should look like and if we set it before running the java launcher will prevent java from forking using execve, this LD_LIBRARY_PATH and execve madness are thankfully gone in JDK7!
(gdb) set env LD_LIBRARY_PATH=/media/disk/4mar-shark-1.8pre-b18-llvm-2.7svn.so-npplugin/jre/lib/arm/server:/media/disk/4mar-shark-1.8pre-b18-llvm-2.7svn.so-npplugin/jre/lib/arm:/media/disk/4mar-shark-1.8pre-b18-llvm-2.7svn.so-npplugin/jre/../lib/arm
I will do one more thing namely set a gdb breakpoint inside java_md.c:652 right after the hotspot library libjvm.so have been loaded by the java launcher.
(gdb) break java_md.c:652
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
...
Breakpoint 2, LoadJavaVM ... java_md.c:652
652 if (libjvm == NULL) {
This are a good spot to setup new gdb breakpoints inside the loaded libjvm.so that contains the Shark JIT. Finally we are able to place a breakpoint on the line where the Shark JIT failed inside LLVM.
(gdb) break MachineFrameInfo.h:289
(gdb) continue
Continuing.
...
10 b java.lang.String::getChars (66 bytes)
[Switching to Thread 0x67ed96a490 (LWP 21127)]
Breakpoint 3, … at … MachineFrameInfo.h:289
Get a backtrace and try to locate the frame where Shark calls getPointerToFunction
(gdb) bt
...
#9 0x40d4ee68 in llvm::JIT::getPointerToFunction (this=0x9e138, F=0xda6f0)
...
Switch to the getPointerToFunction stack frame
(gdb) frame 9
and finnaly dump the LLVM IR for the function by calling the functions own method dump() !
(gdb) call F->dump()
define internal void @"java.lang.String::getChars"([84 x i8]* %method, i32 %base_pc, [788 x i8]* %thread) {
%1 = getelementptr inbounds [788 x i8]* %thread, i32 0, i32 756 ; [#uses=1]
%zero_stack = bitcast i8* %1 to [12 x i8]* ; [12 x i8]*> [#uses=1]
%2 = getelementptr inbounds [12 x i8]* %zero_stack, i32 0, i32 8 ; [#uses=1]
%stack_pointer_addr = bitcast i8* %2 to i32* ; [#uses=1]
%3 = load i32* %stack_pointer_addr ; [#uses=1]
…
%142 = getelementptr inbounds [17 x i32]* %frame, i32 0, i32 12 ; [#uses=1]
store i32 %31, i32* %142
call void inttoptr (i32 13839116 to void ([788 x i8]*, i32)*)([788 x i8]* %thread, i32 7)
ret void
}
Horray! we have successfully dumped the Shark generated LLVM IR for the problematic method-call. Now simply copy the dump output from the terminal into a file named bug.ll and continue reading.
Check for LLVM CodeGen bugs by testing if the dumped LLVM IR bug.ll file can reproduce the bug using llc
After you have extracted LLVM IR for the problematic method check if you can reproduce the bug using llc..
$ llvm-as < bug.ll | llc
.syntax unified
.eabi_attribute 20, 1
.eabi_attribute 21, 1
.eabi_attribute 23, 3
.eabi_attribute 24, 1
.eabi_attribute 25, 1
.file "”
llc:
/wd/buildbot/llvm-arm-linux/llvm/include/llvm/CodeGen/MachineFrameInfo.h:289:
int64_t llvm::MachineFrameInfo::getObjectOffset(int) const: Assertion
`!isDeadObjectIndex(ObjectIdx) && “Getting frame offset for a dead object?”‘
failed.
0 llc 0×01368414
1 llc 0×01368ccc
2 libc.so.6 0×4021cc10 __default_sa_restorer_v2 + 0
Stack dump:
0. Program arguments: /wd/r96575/Debug/bin/llc -march=arm
1. Running pass ‘Prolog/Epilog Insertion & Frame Finalization’ on function
‘@”java.lang.String::getChars”‘
Aborted
If it crashes using llc then cheer up because you now got a reproducable CodeGen bug and thats great! These kind of crash bugs are on LLVM developers top wanted list because they can fire on any tool that uses LLVM code generation. The best way to report this kind of bugs are to first generate a compact testcase for LLVM that triggers the bug that can be used by the LLVM developers to fix it. It can also be run by the LLVM developers daily regression testing to make sure this bug never hits again.
If it fails to crash with an Aborted like above then you are probably observing a JIT CodeEmitter runtime bug, stay tuned and look forward to my next blog post on “How to fix Shark LLVM JIT CodeEmitter bugs”!
How to generate a bugpoint-reduced-simplified.bc from the bug.ll using bugpoint for CodeGen crash bugs
LLVM ships with a clever tool called bugpoint that are designed to convert dumped blocks of LLVM IR into a compact bugpoint-reduced-simplified.bc LLVM bitcode testcase file that only contains the instructions needed to reproduce the bug.
$ bugpoint -run-llc bug.ll --tool-args -march=arm
Bugpoint work by using deductive logic to break down and remove parts of the bug.ll file and automatically narrow down the LLVM IR lines needed to reproduce the bug. It can take some minutes so be patient but bugpoint will eventually stop and give you a bugpoint-reduced-simplified.bc and print some information on how to reproduce the bug.
File a LLVM bugreport containing the bugpoint-reduced-simplified.bc file
An example of a Shark JIT LLVM bug that have been fixed after submitting a bugpoint-reduced-simplified.bc produced from a dumped Shark metod are :
LLVM PR6478 ARM CodeGen Running pass ‘Prolog/Epilog Insertion & Frame Finalization’ on function ‘@”java.lang.String::getChars”‘
I hope this post have given you some inspiration on how to get Shark LLVM JIT CodeGen crash bugs fixed!
If you want to know more about how bugpoint works and how to officially prepare LLVM bugreports then take a peek at the LLVM documentation: http://llvm.org/docs/HowToSubmitABug.html its great.
Xerxes
Wednesday 10 March 2010 05:00 MST
Tuesday 09 March 2010 19:40 MST
Those of you with geek leanings, and possibly some of you who aren’t, will have experienced the problem – you have a great idea for a new project/product/business/etc and you’ve come up with a name for it and it’s beginning to come together, and then you go to buy the domain, only to discover that .com, .net and .org are all taken. And suddenly you have to decide – do you go for a marginally more obscure ending – .co.uk, org.uk, .info, .me – or do you change the name of the project?
The happened for me with both linkpot.net and fosstr.org (both of which I handed over to a friend when I realised that I wasn’t going to be able to maintain them). In both of those, I took the option to choose a new name and get a ‘decent’ domain. However, it’s happened to me again, and, in light of the recent hubbub from ReadWriteWeb which seems to indicate that a significant proportion of the web use Google rather than actually type in addresses, am I better off keeping the name I like and having a longer TLD, or is there still value in having a short TLD?
Thoughts welcome (although let’s not have a discussion about whether someone in the UK should be using .***.uk rather than a US TLD )
mrBen
Tuesday 09 March 2010 18:55
He did it for love and he is a model employee with his employer The Rutgers University according to his friends and colleagues And the TSA security officer that should have been stationed where the 'trespass' occurred is on administrative leave. So why aren't the security guard and his supervisor being charged with negligence for leaving a gaping security hole and causing this whole mess? It is an equal embarrassment for TSA , not an embarrassment for the lover man who was only too stupid to know better not to let love blind him Isn't US Justice system badly need an overhaul?Tuesday 09 March 2010 15:28 MST
The current App made the top 10 iTunes chart and we are about to launch several new and exciting products in the coming months.
We are actively seeking iPhone and Android Developers with a keen interest in mobile devices and platforms to join an established but growing team.
iPhone Developer
You will be an experienced developer with skills in Objective-C, C/C++, GDB, Cocoa/Cocoa Touch Frameworks, Subversion and Git and possess experience of Unit Testing. An excellent
communicator with a desire to meet deadlines and exceed expectations, you will have a proven track record of writing applications and will already have an App in the Apple App Store or one you can demonstrate.
Android Developer
You will be an experienced developer with skills in Java, C/C++ Android SDK, Subversion and Git and possess experience of Unit Testing. An excellent communicator with a desire to meet deadlines and exceed expectations, you will have a proven track record of writing applications and will already have an App in the Google Market Place or one you can demonstrate.
In return for your skills and commitment, you can be assured of a role in an organisation that will enhance your knowledge and provide further career broadening opportunities.
Please note: We have several openings for both iPhone and Android developers. Please state clearly in your application which role you are applying for.
Interested? Please send your CV (preferably Stack Overflow CVs) to cvs@nyxdigital.com or contact me on Twitter: http://twitter.com/rjstelling
Tuesday 09 March 2010 09:00 MST
This is an exciting position for a highly-motivated and talented software engineer to lead the software development of an imaginative new social media product targeted at the UK youth market. Reporting directly to the managing director of this new spin-out venture, the successful individual will have outstanding demonstrable skills in the following:
- PHP
- MySQL
- Object-oriented programming
- Hudson CI Server
- SVN
- Linux & Apache
The position will be based in our modern open-plan development offices in Paisley, next to Glasgow, with the individual working in a team of highly skilled and dedicated individuals as part of a disciplined but enjoyable environment.
In return the successful candidate can expect to be renumerated with an exceptional benefits package including a starting salary of up to £30,000 a year as well as possible employee share opportunities, subsidised lunches and more.
Interested? Apply by contacting Christopher McCann at chris.mccann@sparkofgenius.co.uk or on 0141 587 2710.
Tuesday 09 March 2010 00:05 MST
We were struck by the lack of a critical response to the research cited by Larry Elliott and Heather Stewart on Africa and the millennium development goals, (Africa begins to make poverty history, 3 March). The article covers a key policy goal for governments in Africa and for the Department for International Development, other donors and the UN, so it is important that the research is properly understood and correctly represented.
The original paper does three things that suggest extreme caution. First, it manufactures 1,800 data points on inequality from surveys that cover only 118 data points: in other words, 94% of the inequality numbers are extrapolations from other countries and other years. Second, the poverty estimates rely heavily on government-reported GDP, when we know that GDP data from national income accounts do not match income levels recorded from household surveys. Third, using GDP per capita and the manufactured inequality data, the authors construct poverty rates for 48 African countries for each year between 1970 and 2006. The authors find few correlations between their manufactured poverty rates and structural features of the countries in the sample. This insensitivity to structural features either means that poverty has been reduced in every single location (unlikely) or that the data do not reflect reality.
The authors are fairly critical of their results and it is important that the media also takes a critically engaged view.
The countries of sub-Saharan Africa should be applauded for making substantial progress in numerous policy areas including growth and poverty reduction. But a triumph of elegant methods over reality may lead to an allocation of resources away from sub-Saharan Africa just when they are most needed. Five years after the Commission for Africa and with five years to go to the 2015 MDG target, interpreting the research correctly is more important than ever.
Lawrence Haddad and Andy Sumner
Institute of Development Studies
? It is important to be clear about what the BBC has reported (BBC stands firm over Ethiopia fund claim, 8 March). Last week's Assignment documentary on the BBC World Service examined evidence that, in the mid-1980s, the main rebel group in Tigray in northern Ethiopia diverted relief aid from western donors to support its military campaign. It did not suggest that the larger part of overall famine relief funds was used improperly.
The critical response from relief agencies has focused on a comment by Aregawi Berhe, the Tigray People's Liberation Front (TPLF) military commander in the mid-1980s. He told the programme that the relief society connected to the TPLF received about $100m and that a decision was made that only 5% should be spent helping famine victims. The balance, he said, was used to fund the TPLF and a linked political party. The programme made clear that the assertion that 95% of the aid routed through the TPLF relief society was diverted was made by a once high-ranking TPLF figure, now in exile. It is entirely correct to report these comments. They relate only to aid for areas then held by TPLF rebels and not to the total famine relief effort for Ethiopia.
The evidence presented in the programme that relief aid was used for military purposes included the testimony of a second former senior member of the TPLF, as well as the contesting accounts of two western aid workers. One stated: "If we were being conned, I think it was on a very small scale."
This was a well-researched programme and the BBC stands by the journalism. We are happy to repeat that there is no suggestion that any relief agency was complicit in any diversion of funds.
Andrew Whitehead
Editor, news and current affairs, BBC World Service
? As an aid agency working on the ground in the Democratic Republic of Congo, we believe that talk of withdrawing the peacekeeping force is premature (Report, 5 March). The decision should be based on the needs of ordinary people. Many communities are still at risk.
In Province Orientale, recent months have seen a resurgence in attacks by the Lord's Resistance Army, forcing thousands to flee their homes. Communities in North and South Kivu continue to face attacks. Women have been raped and homes burned. Oxfam has been critical of the performance of the peacekeeping force in the past, but has never doubted the need for it. Sections of the Congolese army still prey on the civilian population. Any withdrawal of Congo's Monuc peacekeepers needs to be closely linked with significant progress on reforming the Congolese army.
Kirsty Hughes
Head of policy and advocacy, Oxfam
? Blessing-Miles Tendi is wrong to advocate an end to EU sanctions on the thugs in Robert Mugabe's Zanu-PF (Zuma's right on Zimbabwe, 4 March). The last thing Zimbabwe needs now is to let these people off the hook.
The Zimbabwe Congress of Trade Unions reports continuing harassment, beatings and arrests by the parts of Zimbabwe's government that Zanu-PF still control. Last month, the leader of the rural workers' union, Gertrude Hambira, was forced to flee to South Africa. Two of her colleagues were arrested. Their crime was to reveal the extent of farm invasions still going on.
Zimbabwe's people are well aware of the difference between sanctions on Zimbabwe as a whole and sanctions on those who abuse the human rights of their own citizens as they loot the country of its natural resources ? not just farms, but now diamond mines as well. Relaxing sanctions would only encourage Zanu-PF to step up its brutality.
President Zuma should not revert to the failed appeasement that characterised President Mbeki's handling of Zimbabwe. The EU is right to maintain the sanctions against Zimbabwe's human rights abusers.
Brendan Barber
General secretary, TUC
Monday 08 March 2010 21:00 MST
Ndeze and Ndakasi, symbols of hope in the struggle to rescue an endangered species in eastern Africa
Monday 08 March 2010 19:02 MST
When I discussed VersionControlTools I said that it was an unscientific agglomeration of opinion. As I was doing it I realized that I could add some spurious but mesmerizing numbers to my analysis by doing a survey. Google's spreadsheet makes the mechanics of conducting a survey really simple, so I couldn't resist.
I conducted the survey from February 23 2010 until March 3 2010 on the ThoughtWorks software development mailing list. I got 99 replies. In the survey I asked everyone to rate a number of version control tools using the following options:
The results were this:
| Tool | Best | OK | Problematic | Dangerous | No Opinion | Active Responses | Approval % |
|---|---|---|---|---|---|---|---|
| Subversion | 20 | 72 | 6 | 1 | 0 | 99 | 93% |
| git | 65 | 19 | 1 | 0 | 14 | 85 | 99% |
| Mercurial | 33 | 27 | 2 | 0 | 36 | 62 | 97% |
| ClearCase | 0 | 3 | 14 | 41 | 41 | 58 | 5% |
| TFS | 0 | 0 | 32 | 22 | 44 | 54 | 0% |
| CVS | 0 | 14 | 59 | 11 | 15 | 84 | 17% |
| Bazaar | 1 | 13 | 3 | 0 | 80 | 17 | 82% |
| Perforce | 1 | 26 | 16 | 1 | 54 | 44 | 61% |
| VSS | 1 | 1 | 11 | 64 | 22 | 77 | 3% |
As well as the raw summary values, I've added two calculated columns here to help summarize the results.
The graph shows a scatter plot of approval percentage and active responses. As you can see there's a clear cluster around Subversion, git, and Mercurial with high approval and a large amount of responses. It's also clear that there's a big divide in approval between those three, together with Bazaar and Perforce, versus the rest.
Although the graph captures the headline information well, there's a couple of other subtleties I should mention.
Some caveats. This is a survey of opinion of ThoughtWorkers who follow our internal software development discussion list, nothing more. It's possible some of them may have been biased by my previous article (although unlikely, since I've never managed to get my ThoughtBot opinion-control software to work reliably). Opinions of tools are often colored by processes that are more about the organization than the tool itself. But despite these, I think it's an interesting data point.
I should also stress the important point to take away from this isn't the comparison between those close in the numbers, eg comparing git and Mercurial or comparing TFS and ClearCase. Any survey like this has a certain amount of noise in it, and I suspect the noise here is greater than such a difference. The important point is the big approval gap between the leading tools (Subversion, git, and Mercurial) and the laggards - essentially the point in VersionControlTools.
Monday 08 March 2010 18:55
Having discovered that I'm not the only pipe-faggot on Kuro5hin I decided to see whether I could get my loose shag as moist as a 14 year old baptist girl at her first Jonas brothers concert. Unfortunately my girlfriend doesn't like handlebar moustaches around her lady-garden, so, thus rebuffed, I decided to see whether I could replicate Mr Tiber's success in the improvement of my tobacco. Similarly to Ghost of Tiber, I had issues with knowing precisely the humidity of my jarmidor, which I eventually decided to ignore; and with a bit of serendipity, I was able to get a result which was, if not correct to a scientific degree of accuracy, at least was palatable and a definite improvement on the moisture content of tobacco as bought.Monday 08 March 2010 18:36 MST
PopJam is focused on innovating in the chat/IM space and we're looking for a talented software engineer to join the team.
As employee #1, you will be fundamental to the company and the development of the product. You will be at the forefront of the action, building new features and solving challenging problems every day.
With backing from the founders of Bebo, Firebox and Moshi Monsters, you'll work alongside experienced web entrepreneurs in taking PopJam from simple app to fully-fledged consumer internet business.
Must have:
* Working knowledge of Python and Javascript
* Experience with some RDBMS (e.g. MySQL/PostgreSQL)
Nice to have:
* Experience with the Twisted framework
* Familiarity with XMPP/Jabber
* Experience with the Facebook platform
We're more interested in your raw talent than we are your experience. Send us your CV with some examples of your own projects.
Interested? Get in touch with Alex: joeloverflow@popjam.com
Monday 08 March 2010 09:27 MST
As an appendix to the previous post on the subject:
Rock Paper Shotgun: Ubisoft?s servers have been down/overloaded for around the last ten hours, making it impossible for people in some parts of the world to play Assassin?s Creed II. Which is certainly not amusing if you?re someone who bought the game despite the DRM (that requires constant connection to their servers), and trusted that Ubisoft would not allow something like this to happen. Especially not in the first week. An enraged forum thread appeared on Ubi?s site, which eventually led to a post from Community Manager ?Ubi.Vigil?, who explained that the situation was, ?unacceptable?.
And the pirates?
Play on.
Monday 08 March 2010 05:00 MST
Monday 08 March 2010 00:05 MST
There are 76 rules broadcasters have to follow for the debates. But I've found a loophole...
So: those televised prime ministerial debates will definitely be happening in the runup to the election. The excitement is hard to contain: three separate primetime shows on Sky, ITV and the Beeb in which Brown, Cameron and Clegg will get the opportunity to talk and talk and talk and talk and talk. And possibly jig. But mainly talk.
Depending on your point of view, this is either a refreshing opportunity for politicians to connect with the electorate, or the least sexy hour of television since that Channel 4 documentary where they chopped up an elephant.
Even though its power and influence are in decline, TV still fascinates and horrifies politicians in equal measure. They're attracted by its potential to hypnotise and pacify millions, but repelled by its laser-like ability to magnify physical flaws or tonal cock-ups. It's like a magic amulet that can sometimes control the masses, but also might explode in the user's hand at any time.
Obviously image is paramount. On TV, no matter how eloquent you are, 75% of the audience can't even hear what you're saying: they're too busy making subconscious judgments about the tone of your voice or the angle of your lips. Conventional wisdom would have it that Gordon Brown is clearly at a massive disadvantage here, since he's slowly come to resemble a lumbering, doomy Mr Snuffaluffagus with all the carefree joie de vivre of the Kursk submarine disaster. But Cameron and Clegg are, if anything, a bit too telegenic, a bit too slick, a bit too clean-cut and heigh-ho. They've tried too hard to appeal in soundbite pop-up form: stretched over an hour, they may start to grate, their smooth appearances unexpectedly conspiring against them.
Cameron in particular looks like a boring dot-eyed "nice" neighbour from an underwhelming Christian soap opera. He's a replicant; an Auton; a humanoid; a piece of adaptive software that's learned to appeal to your likes and dislikes ? "customers who bought Tony Blair also bought the following" ? but inadvertently creeped you out in the process. Let's face it: if you discovered he doesn't have a belly button or any pubic hair, and spends one night each week lying semi-conscious, face-down, "recharging" inside a giant white laboratory pod filled with amniotic fluid, you wouldn't be entirely surprised. And voters are likely to sense that eerie unearthliness. He'd better stutter or fluff a few times, just to throw them off the scent.
But even if all three manage to flawlessly imitate human beings, defeat may still be snatched from the jaws of victory: if Nick Clegg spends the first 50 minutes rousing the audience with his fiery, lyrical rhetoric ? as per usual ? only to sneeze unexpectedly five minutes before the end, leaving a giant pendulum of mucus dangling off the end of his conk, the unfortunate mishap would be looped and repeated ad nauseam on every rolling news bulletin for weeks to come. He'd be Mr Snot. And do you want to vote for Mr Snot? No way. What if he sneezed on the nuclear button? He's out of the running. Which leaves you choosing between a haunted elephant or the humanoid.
(There are other parties you could vote for, obviously. But they're excluded from the debates and therefore no longer exist ? a terrible blow for Nick Griffin, who was hoping to win over the public with his devilish good looks and impish personality.)
So: mammoth or android. Which is it to be? To help you choose, the news networks will doubtless offer post-match analysis of each nanosecond. Professional Westminster spods will deconstruct each sentence in search of hidden meanings, like scientists translating garbled messages from space. A body-language expert will discuss Cameron's eyebrows for 38 minutes. A fashionista will tell us who wore the best shirt. And every other citizen in the country will be asked to deliver their opinion via vox pop, email, tweet, phone poll or synchronised Mexican wave. Eventually a consensus will form regarding who won, at which point the lucky victor will be given the keys to 10 Downing Street, a fly-drive holiday for two courtesy of Virgin Atlantic, a five-album recording contract with Sony BMG, and an ITV2 reality show of their very own.
So terrifying-yet-alluring is the prospect of the debates, the parties have only consented to take part provided each broadcaster adheres to a series of 76 rules, drawn up in advance. Every aspect will be controlled, from the time allocated to each question, to the layout of the set ? even the framing of audience cutaway shots is crucial. Presumably spin doctors from all three parties will be lurking ominously on the sidelines, ready to run in and kick the cameramen to death if their candidate starts looking too sweaty. You can already picture Andy Coulson in the wings, chewing gum and eavesdropping on the gallery audio feed, which has been illegally tapped by a private detective and routed directly into Andy Coulson's earpiece without Andy Coulson's knowledge.
Curiously, one thing that's left open to the broadcaster is the opening and closing credits. Rule 68 states that "each broadcaster [is] responsible for their own titles, music, branding etc". If I was running ITN ? which, at the time of writing, I'm not ? I'd make the most of this sole crumb of freedom by creating an insanely inappropriate title sequence in which a claymation Brown, Cameron and Clegg take turns performing sex acts on a cow, a kettle and a hole in the ground, all of it backed by the old It's a Knockout theme tune. Then it abruptly cuts live to the studio, where all three leaders have been waiting to speak, watching with mounting horror as this sickening cartoon unfolded on the monitors. As they storm out, a body language expert analyses their facial expressions, and the studio audience waves giant foam hands around. It might not affect the election either way, but who cares: that's entertainment.
Sunday 07 March 2010 22:00 MST
I took my newly purchased 50mm f1.8 lens to the Chinese New Year celebrations in Liverpool, I have uploaded a few shots to flickr. I'm still getting used to getting my composition right without being able to alter the focal length.
Update: Fixed the white balance on the images to remove the blue tint.
Sunday 07 March 2010 21:54 MST
One of the ideas I talked about in my Jax London presentation is something I call the Scrum hegemony and it deserves a few notes.
Sunday 07 March 2010 21:46 MST
The slides from my Future of Agile presentation to this weeks Jax conference in London are now online. Although this talk started as a revision of last year?s Future of Agile (at ACCU and BCS Bristol) it ended up as a rewrite. The essential message is largely the same (key message: The future is lean) it brings out some new themes (e.g. software craftsmanship).
Sunday 07 March 2010 16:41 MST
Sunday 07 March 2010 16:41 MST
Sunday 07 March 2010 00:06 MST
An Observer investigation reveals how rich countries faced by a global food shortage now farm an area double the size of the UK to guarantee supplies for their citizens
? Read the expert's view
We turned off the main road to Awassa, talked our way past security guards and drove a mile across empty land before we found what will soon be Ethiopia's largest greenhouse. Nestling below an escarpment of the Rift Valley, the development is far from finished, but the plastic and steel structure already stretches over 20 hectares ? the size of 20 football pitches.
The farm manager shows us millions of tomatoes, peppers and other vegetables being grown in 500m rows in computer controlled conditions. Spanish engineers are building the steel structure, Dutch technology minimises water use from two bore-holes and 1,000 women pick and pack 50 tonnes of food a day. Within 24 hours, it has been driven 200 miles to Addis Ababa and flown 1,000 miles to the shops and restaurants of Dubai, Jeddah and elsewhere in the Middle East.
Ethiopia is one of the hungriest countries in the world with more than 13 million people needing food aid, but paradoxically the government is offering at least 3m hectares of its most fertile land to rich countries and some of the world's most wealthy individuals to export food for their own populations.
The 1,000 hectares of land which contain the Awassa greenhouses are leased for 99 years to a Saudi billionaire businessman, Ethiopian-born Sheikh Mohammed al-Amoudi, one of the 50 richest men in the world. His Saudi Star company plans to spend up to $2bn acquiring and developing 500,000 hectares of land in Ethiopia in the next few years. So far, it has bought four farms and is already growing wheat, rice, vegetables and flowers for the Saudi market. It expects eventually to employ more than 10,000 people.
But Ethiopia is only one of 20 or more African countries where land is being bought or leased for intensive agriculture on an immense scale in what may be the greatest change of ownership since the colonial era.
An Observer investigation estimates that up to 50m hectares of land ? an area more than double the size of the UK ? has been acquired in the last few years or is in the process of being negotiated by governments and wealthy investors working with state subsidies. The data used was collected by Grain, the International Institute for Environment and Development, the International Land Coalition, ActionAid and other non-governmental groups.
The land rush, which is still accelerating, has been triggered by the worldwide food shortages which followed the sharp oil price rises in 2008, growing water shortages and the European Union's insistence that 10% of all transport fuel must come from plant-based biofuels by 2015.
In many areas the deals have led to evictions, civil unrest and complaints of "land grabbing".
The experience of Nyikaw Ochalla, an indigenous Anuak from the Gambella region of Ethiopia now living in Britain but who is in regular contact with farmers in his region, is typical. He said: "All of the land in the Gambella region is utilised. Each community has and looks after its own territory and the rivers and farmlands within it. It is a myth propagated by the government and investors to say that there is waste land or land that is not utilised in Gambella.
"The foreign companies are arriving in large numbers, depriving people of land they have used for centuries. There is no consultation with the indigenous population. The deals are done secretly. The only thing the local people see is people coming with lots of tractors to invade their lands.
"All the land round my family village of Illia has been taken over and is being cleared. People now have to work for an Indian company. Their land has been compulsorily taken and they have been given no compensation. People cannot believe what is happening. Thousands of people will be affected and people will go hungry."
It is not known if the acquisitions will improve or worsen food security in Africa, or if they will stimulate separatist conflicts, but a major World Bank report due to be published this month is expected to warn of both the potential benefits and the immense dangers they represent to people and nature.
Leading the rush are international agribusinesses, investment banks, hedge funds, commodity traders, sovereign wealth funds as well as UK pension funds, foundations and individuals attracted by some of the world's cheapest land.
Together they are scouring Sudan, Kenya, Nigeria, Tanzania, Malawi, Ethiopia, Congo, Zambia, Uganda, Madagascar, Zimbabwe, Mali, Sierra Leone, Ghana and elsewhere. Ethiopia alone has approved 815 foreign-financed agricultural projects since 2007. Any land there, which investors have not been able to buy, is being leased for approximately $1 per year per hectare.
Saudi Arabia, along with other Middle Eastern emirate states such as Qatar, Kuwait and Abu Dhabi, is thought to be the biggest buyer. In 2008 the Saudi government, which was one of the Middle East's largest wheat-growers, announced it was to reduce its domestic cereal production by 12% a year to conserve its water. It earmarked $5bn to provide loans at preferential rates to Saudi companies which wanted to invest in countries with strong agricultural potential .
Meanwhile, the Saudi investment company Foras, backed by the Islamic Development Bank and wealthy Saudi investors, plans to spend $1bn buying land and growing 7m tonnes of rice for the Saudi market within seven years. The company says it is investigating buying land in Mali, Senegal, Sudan and Uganda. By turning to Africa to grow its staple crops, Saudi Arabia is not just acquiring Africa's land but is securing itself the equivalent of hundreds of millions of gallons of scarce water a year. Water, says the UN, will be the defining resource of the next 100 years.
Since 2008 Saudi investors have bought heavily in Sudan, Egypt, Ethiopia and Kenya. Last year the first sacks of wheat grown in Ethiopia for the Saudi market were presented by al-Amoudi to King Abdullah.
Some of the African deals lined up are eye-wateringly large: China has signed a contract with the Democratic Republic of Congo to grow 2.8m hectares of palm oil for biofuels. Before it fell apart after riots, a proposed 1.2m hectares deal between Madagascar and the South Korean company Daewoo would have included nearly half of the country's arable land.
Land to grow biofuel crops is also in demand. "European biofuel companies have acquired or requested about 3.9m hectares in Africa. This has led to displacement of people, lack of consultation and compensation, broken promises about wages and job opportunities," said Tim Rice, author of an ActionAid report which estimates that the EU needs to grow crops on 17.5m hectares, well over half the size of Italy, if it is to meet its 10% biofuel target by 2015.
"The biofuel land grab in Africa is already displacing farmers and food production. The number of people going hungry will increase," he said. British firms have secured tracts of land in Angola, Ethiopia, Mozambique, Nigeria and Tanzania to grow flowers and vegetables.
Indian companies, backed by government loans, have bought or leased hundreds of thousands of hectares in Ethiopia, Kenya, Madagascar, Senegal and Mozambique, where they are growing rice, sugar cane, maize and lentils to feed their domestic market.
Nowhere is now out of bounds. Sudan, emerging from civil war and mostly bereft of development for a generation, is one of the new hot spots. South Korean companies last year bought 700,000 hectares of northern Sudan for wheat cultivation; the United Arab Emirates have acquired 750,000 hectares and Saudi Arabia last month concluded a 42,000-hectare deal in Nile province.
The government of southern Sudan says many companies are now trying to acquire land. "We have had many requests from many developers. Negotiations are going on," said Peter Chooli, director of water resources and irrigation, in Juba last week. "A Danish group is in discussions with the state and another wants to use land near the Nile."
In one of the most extraordinary deals, buccaneering New York investment firm Jarch Capital, run by a former commodities trader, Philip Heilberg, has leased 800,000 hectares in southern Sudan near Darfur. Heilberg has promised not only to create jobs but also to put 10% or more of his profits back into the local community. But he has been accused by Sudanese of "grabbing" communal land and leading an American attempt to fragment Sudan and exploit its resources.
Devlin Kuyek, a Montreal-based researcher with Grain, said investing in Africa was now seen as a new food supply strategy by many governments. "Rich countries are eyeing Africa not just for a healthy return on capital, but also as an insurance policy. Food shortages and riots in 28 countries in 2008, declining water supplies, climate change and huge population growth have together made land attractive. Africa has the most land and, compared with other continents, is cheap," he said.
"Farmland in sub-Saharan Africa is giving 25% returns a year and new technology can treble crop yields in short time frames," said Susan Payne, chief executive of Emergent Asset Management, a UK investment fund seeking to spend $50m on African land, which, she said, was attracting governments, corporations, multinationals and other investors. "Agricultural development is not only sustainable, it is our future. If we do not pay great care and attention now to increase food production by over 50% before 2050, we will face serious food shortages globally," she said.
But many of the deals are widely condemned by both western non-government groups and nationals as "new colonialism", driving people off the land and taking scarce resources away from people.
We met Tegenu Morku, a land agent, in a roadside cafe on his way to the region of Oromia in Ethiopia to find 500 hectares of land for a group of Egyptian investors. They planned to fatten cattle, grow cereals and spices and export as much as possible to Egypt. There had to be water available and he expected the price to be about 15 birr (75p) per hectare per year ? less than a quarter of the cost of land in Egypt and a tenth of the price of land in Asia.
"The land and labour is cheap and the climate is good here. Everyone ? Saudis, Turks, Chinese, Egyptians ? is looking. The farmers do not like it because they get displaced, but they can find land elsewhere and, besides, they get compensation, equivalent to about 10 years' crop yield," he said.
Oromia is one of the centres of the African land rush. Haile Hirpa, president of the Oromia studies' association, said last week in a letter of protest to UN secretary-general Ban Ki-moon that India had acquired 1m hectares, Djibouti 10,000 hectares, Saudi Arabia 100,000 hectares, and that Egyptian, South Korean, Chinese, Nigerian and other Arab investors were all active in the state.
"This is the new, 21st-century colonisation. The Saudis are enjoying the rice harvest, while the Oromos are dying from man-made famine as we speak," he said.
The Ethiopian government denied the deals were causing hunger and said that the land deals were attracting hundreds of millions of dollars of foreign investments and tens of thousands of jobs. A spokesman said: "Ethiopia has 74m hectares of fertile land, of which only 15% is currently in use ? mainly by subsistence farmers. Of the remaining land, only a small percentage ? 3 to 4% ? is offered to foreign investors. Investors are never given land that belongs to Ethiopian farmers. The government also encourages Ethiopians in the diaspora to invest in their homeland. They bring badly needed technology, they offer jobs and training to Ethiopians, they operate in areas where there is suitable land and access to water."
The reality on the ground is different, according to Michael Taylor, a policy specialist at the International Land Coalition. "If land in Africa hasn't been planted, it's probably for a reason. Maybe it's used to graze livestock or deliberately left fallow to prevent nutrient depletion and erosion. Anybody who has seen these areas identified as unused understands that there is no land in Ethiopia that has no owners and users."
Development experts are divided on the benefits of large-scale, intensive farming. Indian ecologist Vandana Shiva said in London last week that large-scale industrial agriculture not only threw people off the land but also required chemicals, pesticides, herbicides, fertilisers, intensive water use, and large-scale transport, storage and distribution which together turned landscapes into enormous mono-cultural plantations.
"We are seeing dispossession on a massive scale. It means less food is available and local people will have less. There will be more conflict and political instability and cultures will be uprooted. The small farmers of Africa are the basis of food security. The food availability of the planet will decline," she says. But Rodney Cooke, director at the UN's International Fund for Agricultural Development, sees potential benefits. "I would avoid the blanket term 'land-grabbing'. Done the right way, these deals can bring benefits for all parties and be a tool for development."
Lorenzo Cotula, senior researcher with the International Institute for Environment and Development, who co-authored a report on African land exchanges with the UN fund last year, found that well-structured deals could guarantee employment, better infrastructures and better crop yields. But badly handled they could cause great harm, especially if local people were excluded from decisions about allocating land and if their land rights were not protected.
Water is also controversial. Local government officers in Ethiopia told the Observer that foreign companies that set up flower farms and other large intensive farms were not being charged for water. "We would like to, but the deal is made by central government," said one. In Awassa, the al-Amouni farm uses as much water a year as 100,000 Ethiopians.
Saturday 06 March 2010 19:38 MST
It's Sunday afternoon and I'm sitting here in my study writing this blog to the backdrop of another torrential rain shower. It seems like we've had a considerable amount of rain this summer, which the farmers appreciate but would prefer in the winter. I've had to move stuff around in my office to prevent losing things to water damage like I did a few weeks ago.
Saturday 06 March 2010 18:32 MST
Annoyingly the current version of the google-news-sitemap plugin for Wordpress (v1.4) doesn’t work with some silly XML namespace error reported by google.
See http://wordpress.org/support/topic/364929 and effectively the ‘patch’ on the Google Support forum thing which works fine (there are two bits of the plugin which need updating – whcih correlate to the two parts mentioned in the posting etc)
Bit annoyed that the fix is so easy – yet the plugin hasn’t been updated yet. Grr.
Saturday 06 March 2010 00:08 MST
'Stone is such a tool it's a wonder the cameras didn't explode out of horrified glee'
Reader, I apologise in advance. Words can't describe the exquisite mix of pain, fury and joy that is Pineapple Dance Studios (Sun, 6pm, Sky1). Yet words must suffice. I can't just sit here silently popping my mouth open and shut like a surprised mute, although that's precisely the reaction it provokes. You know how every so often the natural history unit throws up a documentary about hallucinogenically weird organisms that live 15 miles down in the deep, during which some undulating avant garde cross between a jellyfish, a diagram and an inside-out seahorse will wobble across the screen, defying any rational attempt at description? This is the docusoap equivalent of that.
Yes, it's a docusoap. That much we can cling to. It's a docusoap about the various characters around Pineapple Dance Studios in Covent Garden. The most immediately noticeable example is a berk called Louie Spence, a creature so theatrically camp he seems perpetually on the verge of turning into a disco-dancing peacock. God knows what his job at the studio actually consists of: you could watch for a thousand years and never find out. All he does is mug for the cameras, perpetually striking poses, pulling arch faces, cracking lurid innuendo, shrieking, mincing and generally behaving in a way no fictional gay character has been permitted to do for decades. Given the right narrator, this could be a heartbreaking doc about an incurable mental condition whose sufferers lose their minds at the sight of a film crew and turn into a 1978 sitcom homosexual.
And incredibly it has been given the right narrator: former BBC news anchor Michael Buerk. You'd be hard pressed to find a more sobering voice of authority. Instant gravitas. Each time there's an establishing shot of a building exterior, I fully expect to hear him say: "Dawn ? and as the sun breaks through the piercing chill of night on the plain outside Korem, it lights up a biblical famine; now, in the 20th century." But he doesn't. Instead he says something like: "9am ? and Louie is pirouetting in a stairwell." Cut to Louie pirouetting in a stairwell. It's upsetting and funny and wrong and right. It's everything. This is madness.
Louie is a maddening show-off, but at least he isn't Andrew Stone. In reality, Andrew Stone is one of the resident dance teachers. In his head, he's a global pop superstar. The show focuses heavily on the ups and downs of his derivative, deeply uninspiring band Starman, which he fronts with a level of egomaniacal self-assurance hitherto undocumented on British TV. Seriously, they've captured lightning in a bottle here: the man is a tool of such breathtaking immensity, it's a wonder the cameras didn't simply explode out of horrified glee. One of life's sorest tragedies is that the people who brim with confidence are always the wrong people. This is the clearest possible illustration of that truth ever committed to videotape. Show this to your children. Make them learn from it.
On and on the show goes, swerving effortlessly from fist-chewingly mundane office-management sequences straight out of The Day Today's famous docusoap spoof The Pool one moment, into bizarre choreographed dance sequences the next. Yes: they've thrown in occasional fourth-wall-smashing musical numbers just to baffle you to death. One minute Louie is complaining to the builders next door about noise and then suddenly ? boom! ? they unexpectedly start dancing, as though he's stumbled into a dream sequence. And this breakdown of reality isn't acknowledged in Michael Buerk's voiceover at all. No, it simply occurs. And then the show moves on as if it hadn't. As though the TV fakery scandals never happened. And suddenly you question the veracity of everything you're watching. Except the rest of it is real. It just doesn't ? just shouldn't ? feel that way.
But that's Pineapple Dance Studios. A show designed to trigger life-threatening cognitive dissonance. As mundane as a breadbin; more outlandish than Avatar. As horrible as war; as funny as a guffing cartoon donkey. Words don't even graze the surface.
Friday 05 March 2010 17:45 MST
Suppose you have a sparse RAM API, something along the lines of:
People use this API for things like running a simulated CPU:
Suppose this API becomes a runaway success, with a whopping 10 programmers using it (very little irony here, >95% of the APIs in this world are used exclusively by their designer). Then chances are that 9 of the 10 programmers are API users, and 1 of them is an API wrapper. Here’s what they do.
API users
The first thing the first API user does is call you. “How do I use this sparse thing of yours?” You point him to the short tutorial with the sample code. He says “Uhmm. Errm…”, which is userish for “Come on, I know you know that I’m lazy, and you know I know that docs lie. Come over here and type the code for me.” And you insist that it’s actually properly documented, but you will still come over, just because it’s him, and you personally copy the sample code into a source file of his:
add_range(0x100000, 6) # input range add_range(0x200000, 6) # output range write_ram(0x100000, "abcdef") # run a program converting the input to uppercase print read_ram(0x200000, 6) # should print "ABCDEF"
It runs. You use the opportunity to point out how your documentation is better than what he’s perhaps used to assume (though you totally understand his frustration with the state of documentation in this department, this company and this planet). Anyway, if he has any sort of problem or inconvenience with this thing, he can call you any time.
The next 8 API users copy your sample code themselves, some of them without you being aware that they use or even need this API. Congratulations! Your high personal quality standards and your user-centric approach have won you a near-monopoly position in the rapidly expanding local sparse RAM API market.
Then some time later you stumble upon the following code:
add_range(0x100000,256) add_range(0x200000,1024) add_range(0x300000,1024) ... add_range(0xb00000,128) ... add_range(0x2c00000,1024) ...
Waitaminnit.
You knew the API was a bit too low-level for the quite common case where you need to allocate a whole lot of objects, doesn’t matter where. In that case, something like base=allocate_range(size) would be better than add_range(base,size) - that way users don’t have to invent addresses they don’t care about. But it wasn’t immediately obvious how this should work (Nth call to allocate_range() appends a range to the last allocated address, but where should the first call to allocate_range() put things? What about mixing add_range() and allocate_range()? etc.)
So you figured you’d have add_range(), and then whoever needed to allocate lots of objects, doesn’t matter where, could just write a 5-line allocate_range() function good enough for him, though not good enough for a public API.
But none of them did. Why? Isn’t it trivial to write such a function? Isn’t it ugly to hard-code arbitrary addresses? Doesn’t it feel silly to invent arbitrary addresses? Isn’t it actually hard to invent constant addresses when you put variable-sized data there, having to think about possible overlaps between ranges? Perhaps they don’t understand what a sparse RAM is? Very unlikely, that, considering their education and experience.
Somehow, something makes it very easy for them to copy sample code, but very hard to stray from that sample code in any syntactically substantial way. To them, it isn’t a sparse RAM you add ranges to. Rather, they think of it as a bunch of add_range() calls with hexadecimal parameters.
And add_range() with hex params they promptly will, just as it’s done in the sample. And they’ll complain about how this API is a bit awkward, with all these hex values and what-not.
API wrappers
If there’s someone who can see right through syntax deep into semantics, it’s the tenth user of your API, or more accurately, its first wrapper. The wrapper never actually uses an API directly in his “application code” as implied by the abbreviation, standing for “Application Programming Interface”. Rather, he wraps it with another (massive) layer of code, and has his application code use that layer.
The wrapper first comes to talk to you, either being forced to use your API because everybody else already does, or because he doesn’t like to touch something as low-level as “RAM” so if there’s already some API above it he prefers to go through that.
In your conversation, or more accurately, his monologue, he points out some admittedly interesting, though hardly pressing issues:
When you manage to terminate the monologuish conversation, he walks off to implement his sparse RAM API on top of yours. He calls it SParser (layer lovers, having to invent many names, frequently deteriorate into amateur copywriters).
When he’s done (which is never; let’s say “when he has something out there”), nobody uses SParser but him, though he markets it heavily. Users won’t rely on the author who cares about The Right Thing but not about their problems. Other wrappers never use his extra layers because they write their own extra layers.
However, even with one person using it, SParser is your biggest headache in the sparse RAM department.
For example, your original implementation used a list of ranges you (slowly) scanned through to find the range containing a given address. Now you want to replace this with a page table, so that, given an address, you simply index into a page array with its high bits and either find a page with the data or report a bad address error.
But this precludes “shadowing”, where you have overlapping segments, one hiding the other’s data. You thought of that as a bug in the user code your original implementation didn’t detect. The wrapper thought it was a feature, and SParser uses it all over to have data used at some point and then “hidden” later in the program.
So you can’t deploy your new implementation, speeding up the code of innocent users, without breaking the code of this wrapper.
What to do
Add an allocate_range() API ASAP, update the tutorial, walk over to your users to help replace their hex constants with allocate_range() calls. Deploy the implementation with the page table, and send the complaining wrapper to complain upwards along the chain of command.
Why
Your users will switch to allocate_range() and be happy, more so when they get a speed-up from the switch to page tables. The wrapper, constituting the unhappy 10% of the stakeholders, will have no choice but fix his code.
Ivan drank half a bottle of vodka and woke up with a headache. Boris drank a full bottle of vodka and woke up with a headache. Why drink less?
Users are many, they follow a predictable path (copy sample code) and are easily satisfied (just make it convenient for them to follow that path). Wrappers are few, they never fail to surprise (you wouldn’t guess what and especially why their layers do), and always fail to be satisfied (they never use APIs and always wrap them). Why worry about the few?
The only reason this point is worth discussing at all is that users offend programmers while wrappers sweet-talk them, thus obscuring the obvious. It is natural to feel outrage when you give someone an add_range() function and a silly sample with hex in it, and not only do they mindlessly multiply hex numbers in their code, but they blame you for the inconvenience of “your API with all the hex in it”. It is equally natural to be flattered when someone spends time to discuss your work with you, at a level of true understanding (”sparse RAM”) rather than superficial syntactic pattern matching (”add_range(hex)”).
He who sees through this optical illusion will focus on the satisfaction of the happy many who couldn’t care less, securing the option to ignore the miserable few who think too much.
Friday 05 March 2010 16:11 MST
I spent ages trying to decide which was the best way for me to implement file upload in my grails app. What I wanted was the ability for a user to select an image from their local machine and for it to be uploaded into a tag on my page. The biggest issue I faced was that the upload button, and in fact the whole DIV that contained the image and other stuff, is generated at runtime. The other mandatory condition was that it had to happen asynchronously and without re-loading the page.
I tried out 3 or 4 solutions, including several grails plug-ins (google for “grails file upload” and you’ll find them), several ajax javascript solutions, which I have now lost, and a couple of Flash uploader implementations. The trouble with all of them was a combination of complexity and size. In the end I found a really simple alternative which seems to work just fine and builds on adding the Grails UI plugin into my application.
My solution was in two parts, first was to give myself the ability to have a modal dialog box on my page and second was to utilise the Yahoo GUI library to make an asynchronous connection back to the server.
The modal dialog came directly from the Grails UI plugin, which has a very rich set of excellent features and I recommend it. Installing that plugin gave me a dialog box, which was as simple as including a resource reference and a GSP tag in a hidden div on my page…
<gui:resources components="['dialog']"/>
<div>
<gui:dialog
id="dlgFileUpload"
title="Upload Image"
draggable="false"
modal="true"
buttons="[[text:'Upload', handler: 'onUploadButtonClick', isDefault: true],[text:'Cancel', handler: 'function() {this.cancel();}', isDefault: false]]">
<form action="uploadfile" enctype="multipart/form-data" method="post" id="uploadForm">
<input type="file" name="testFile"/>
</form>
</gui:dialog>
</div>
The important thing to notice in here is that the dialog box contains a form which defines the multipart enctype, the method as POST. The action refers to the code in my Grails controller which handled persistence of the file. The gui:resources tag is exploded by Grails into a set of script includes for the Yahoo UI libraries. This is important because the YUI library is very large and there are many interdependencies, and this single tag hides all that, which is a major time saver.
The other thing that the Grails UI plugin gave me was the whole of the YUI library, so the next bit was to write some code to handle the file upload itself. Rather than lots of fancy components I realised that I could just use the YAHOO.util.Connect object to make an asynchronous call back to the server, in much that same way that I might otherwise have used an XMLHTTPRequest object in Javascript. Because the whole YUI library is included with the plugin I already had the code. To get at the Connect object I had to include the appropriate script because the gui:resources tag seemed not to add it as a dependence…
<script type="text/javascript" src="/js/yui/2.7.0/utilities/utilities.js" ></script>
From that point it was a simple matter of writing a function to handle the click on the upload button in my dialog which made the request back to the server, and handled the response…
function onUploadButtonClick(e)
{
var uploadHandler =
{
upload: function(o)
{
refreshActiveImage(o.responseText);
setDirty();
hideWaitCursor();
}
};
showWaitCursor();
//the second argument of setForm is crucial,
//which tells Connection Manager this is an file upload form
YAHOO.util.Connect.setForm('uploadForm', true);
YAHOO.util.Connect.asyncRequest('POST', 'uploadfile', uploadHandler);
this.cancel();
}
The important thinhg to notice in here is that the setForm call requires the second argument to be set to true so it recognises the form settings for the POST. Otherwise, the two lines of code invoking YAHOO connect do everything you need to send a file back to the server.
My server side code processes the multipart file upload and persists the file into a known location which suits my application. I then render the file as a virtual location relative to the current page back into my page so I can simply update the src property of the right IMG tag on my page – that’s what refreshActiveImage(o.responseText); does.
Eh voila! A common plug-in, a great utility library, some really basic Javascript and a form and I have the simplest file upload I could imagine.
Friday 05 March 2010 12:52 MST
From the Lord of the Rings Online dev tracker:
Vastin says: Tsk. You know, Aragorn brought a 5-man team into his solo Weathertop instance and just ended up having to carry the whole thing anyway.
Worst… pug… ever!
As for hitting the solo story wall, I think several folks have some reason to be pretty bitter on that front. I mean, seriously:
Thorin organized an entire 14-man raid before he realized that the first half of the Lonely Mountain instance was solo-only – and pretty much impossible for anyone but a burglar.
The rest of the raid ended up having to sit and twiddle their thumbs for days until the idiot burg finally beat the damn quest and opened up the storyline they needed to move on to the next stage of the encounter.
Almost as bad as when the devs forced Frodo and Sam to disband their two-man questing fellowship in the middle of Cirith Ungol and made them run several hours of crazy statted solo-blocker instances before they could reform and move into the new Mordor expansion.
[shakes head]
Friday 05 March 2010 12:05 MST
Wow ow wow ow wow?
Friday 05 March 2010 05:00 MST
Thursday 04 March 2010 19:26 MST
Jono Bacon, Alan Pope, and many others have written, yesterday we published a new visual story and style for Ubuntu. The core design work was lead by Marcus Haslam, Otto Greenslade and Dominic Edmunds, who are the three visual artists leading our efforts in the Canonical Design team. Once we had the base ideas in place we invited some anchor members of the Ubuntu Art community to a design sprint, to test that the concept had the legs to work with the full range of forums, websites, derivatives and other pieces of this huge and wonderful project. And apparently, it does!
Here are some additional thoughts.
Embracing both Ubuntu and Canonical
One of the real challenges for us has been to find a branding and design strategy which spans the spectrum of audiences, forums and dialogues that we cover. With Ubuntu, it’s my specific dream to find a constructive blend of commercial and community interests, not only for Canonical but for other companies. That has made our design and branding work difficult – the distinctive look of Ubuntu lent itself well to pure community messaging, but it was hard to do a brochure for Canonical data center services for Ubuntu on servers. We have not only Ubuntu, but also Kubuntu and an important range of derivatives that all have a role in our ecosystem.
So we spent a lot of time trying to distill the requirements down into a set of three dimensions:
We found a set of ideas which each represent those spectrums, and which work together.
For example, we identified a palette which includes both a fresh, lively Orange, and a rich, mature Aubergine, which work together. The use of Aubergine indicates Commercial involvement of one form or another, while Orange is a signal of community engagement. The Forums will use the Orange elements more strongly, and a formal product brochure, with descriptions of supporting services, would use more of the Aubergine.
On the consumer/enterprise spectrum, we took inspiration from the aerospace industry, and identified a texture of closely spaced dots. When you see more of that, it means we’re signalling that the story is more about the enterprise, less of that, and it’s more about the consumer. Of course, there are cross-overs, for example when we are talking about the corporate desktop, where we’ll use that closely space dot texture as a boundary area, or separator. We also identified shades of Aubergine that are more consumer, or more enterprise – the darker shades mapping to a stronger emphasis on enterprise work.
And on the end-user / engineer spectrum, we took inspiration from graph paper and engineering blue prints. When you see widely spaced patterns of dots, or outline images and figures, that’s signalling that the content is more engineering-oriented than end-user oriented.
And finally, we found a number of themes which enhanced and echoed those ideas. We use a warm gray supporting colour to give shape to pages and documents, and we built on the dots and circles to create a whole style for figures, illustrations and pictograms.
The beauty of this is that we can now publish content that spans the full range, and we generally know when we start the design process what sorts of visual cues we want to be signalling. Instead of having these different mental domains fight with one another, we can now convey quite subtle collaboration between community and corporate, or work which is aimed at engineers and developers from enterprises as opposed to developers working with consumers. Time will tell how it shapes up, but for now I’m celebrating the milestone and the efforts of the team that pulled it together. There’s something there for everyone who wants to participate in the great hubbub of Ubuntuness that is our shared experience of free software.
So, for example, here’s a conference banner. The strong use of Aubergine suggests that it’s more corporate messaging (Canonical is heavily involved). Orange is used here more as a highlight. The Aubergine is darker, and there’s quite a lot of the fine dot pattern. Below the image is a set of scales showing where on those spectra this work is pitched.
As another example, here’s a brochure with an emphasis on end-users who are thinking about adopting Ubuntu’s cloud infrastructure. Again, the fine dot patterns suggests a more enterprise focus, as does the use of the dark aubergine. You can see the circle metaphor used in the quote callout.
And here’s a similar brochure, but with a more developer or engineering oriented focus: note the use of the graph-paper theme with wide spaced dots, and outline shapes.
Finally, here’s an example of a brochure and CD cover for Ubuntu:
As you can see the idea is to signal a mix of both community and Canonical involvement in the message, addressing consumer audiences with a mix of developers and end-users.
A new Ubuntu font
We have commissioned a new font to be developed both for the logo’s of Ubuntu and Canonical, and for use in the interface. The font will be called Ubuntu, and will be a modern humanist font that is optimised for screen legibility. It will be published under an open font license, and considered part of the trade dress of Ubuntu, which will limit its relevance for software interfaces outside of Ubuntu but leave it free for use across the web and in printed documents.
It will take a few months for the font to be finalised, initial elements will be final in the next week which will be sufficient for the logo and other bits and pieces, but I expect to see that font widely used in 10.10. The work has been commissioned from world-renowned fontographers Dalton Maag, who have expressed excitement at the opportunity to publish an open font and also a font that they know will be used daily by millions of people.
Initial coverage will be Western, Arabic, Hebrew and Cyrillic character sets, but over time we may be able to extend that to being a full Unicode font, with great kerning and hinting for print and screen usage globally. We are considering an internship program, to support aspiring fontographers from all corners of the world to visit London and work with Dalton Maag to extend the font to their own regional glyph set.
The critical test of the font is screen efficiency and legibility, and its character and personality are secondary to its fitness for that purpose. Nevertheless, our hope is that the font has a look that is elegant and expresses the full set of values for both Canonical and Ubuntu: adroitness, accountability, precision, reliability, freedom and collaboration. We’ll publish more as soon as we have it.
A good start
It’s been an exciting process, but I have the sense that we are just getting started. The language will get richer, we will find new things that we want to communicate, and new treatments and visual themes that resonate well with these starting points. We’ll find new ways to integrate this on the web, and on the desktop (look out for the two new themes, Radiance and Ambiance). I hope we’ll see the language being used to good effect across everything we do, both commercial and community oriented. There’s a range of expression here that should be useful to artists across the spectrum. Let me know how it works for you.
Thursday 04 March 2010 14:43 MST
While GStreamer has been working on Windows for a long time and one can compile GStreamer using Visual Studio, the lack of pre-made binaries for Windows developers has been a bit of an issue. Various groups and people have tried providing windows binaries for a while, but most efforts have stalled after a short while. The GStreamer winbuilds project however seems quite solid however and have now been doing good windows packages for quite a while. If you have been looking for Windows builds for GStreamer this is a good place to start. They already have a list of users on Windows and the reason I became aware is that the jokosher guys are using it for their windows porting effort.
Thursday 04 March 2010 12:46 MST
Peacekeepers on world's biggest mission may leave Democratic Republic of the Congo by mid-2011
The UN has begun talks with the Democratic Republic of the Congo about ending its controversial peacekeeping mission in the country.
The mission, which has been in Congo for 11 years, is the world's biggest with 18,500 "blue helmet" troops at an annual cost of $1.35bn (£895m).
The discussions were announced by Alain Le Roy, the UN under secretary-general, after a meeting with the Congolese president, Joseph Kabila.
Le Roy told the BBC that a UN team had been given one month to assess how the mission, known by a French acronym, Monuc, could start pulling out troops.
He said peacekeepers would be ready for withdrawal from central and south-east Congo by June. But he added it was still vital for Monuc to support Congolese military offensives against Rwandan Hutu rebels in the east and the Uganda-based Lord's Resistance Army in the north.
Congo's government has said it wants all the peacekeepers gone before the presidential election due next year.
Lambert Mende, a government spokesman, said there should be no UN troops in Congo, other than in the troubled eastern regions of North and South Kivu, by the end of this year. "Withdrawal must be completed by mid-2011," he added.
In the east, Monuc has been providing rations, transport, fuel and firepower to the Congolese army in its operations against an exiled Rwandan Hutu militia group, some of whose leaders took part in the 1994 genocide in Congo's neighbour.
The Congolese army has been accused of atrocities against the people it is meant to be protecting. Human Rights Watch and other NGOs have documented hundreds of killings and thousands of rapes and questioned why Monuc is supporting the army responsible.
Writing in the Guardian last month, Alan Doss, the head of the UN mission in Congo, defended its actions: "We have concentrated more than 95% of our troops in the eastern provinces, which cover an area almost three times the size of France. The terrain is heavily forested. There are few roads and communities are isolated. Because of these conditions quick access is not always possible when reports of violence reach us."
Monuc's current mandate from the UN security council is due to expire at the end of May. The Congolese government is keen for a withdrawal timeline to be announced before 30 June, when the country celebrates 50 years of independence from Belgium.
Thursday 04 March 2010 09:32 MST
Sam made some changes to a SOAP service one customer has – and suddenly our automated tests kept failing. “WTF?” we thought.
We persistently got the same error (e.g. Fatal error: SOAP-ERROR: Encoding: object hasn’t ‘SortResults’ property in ….) yet the generated WSDL file (when viewed through a web browser etc) no longer has SortResults in it.
We checked :
Then it turned out the SoapClient PHP object caches the WSDL file by default in e.g. /tmp/wsdl-blahblahblah. (Where blahblahblah looks very much like an md5). And it doesn’t make much of an effort to check it’s validity. Obviously the documentation does state this, but it does seem like the wrong default behaviour to me.
Solution: Add something to the automated tests to delete /tmp/wsdl-*. OR pass into the SoapClient constructor an array of options like :
$client = new SoapClient('http://somewhere/blah.php?wsdl", array(cache_wsdl => WSDL_CACHE_NONE)
Time wasted: Too much.
Wednesday 03 March 2010 17:44 MST
I have a dream.
My dream is that one day, a giant carrot carved into the shape of a submarine will sail down the Thames before sinking below the waves to take back America using only the power of latin.
But also, I want for machines that are on my local network to be accessible as “$hostname.d.water.gkhs.net” to everyone else on the same local network. That’s a more technical dream, and this is how I did it:
first, we google “smoothwall dhcp to dns”. The first result seems to be exactly what we need, so we click it, and find outselves on Kryogenix, the website of Aquarius, who I have known for somewhere close to a decade, which is an aeon in internet time. The article is now close seven years old, and while its lost its styling, it is (a) entirely what I want to do (b) comprehensive and (c) now completely broken.
The new page that Douglas Warner’s dhcp2dnrd script lives is now somewhere else on the site, and appears to be having some kind of formatting problem, but can still be downloaded from this direct link. At the bottom of this is a link to my own version of this file, with all these changes already made.
Although the the class::date problem no longer exists, a few other things that have changed since the article was released. So, this is what to do to get it working. Most of this is built on the stuff sill said already in his article, just updated for Smoothwall 3.0:
Log in to your Smoothwall box over ssh (If you cannot do this, you need to go to the web interface, Services, Remote Access, and tick SSH. Then, using your favourite terminal, log in to the same IP, port 222. Username root, password whatever you chose when you set up the firewall so long ago. I do hope you remember it.
mkdir dhcp2dnrd; cd dhcp2dnrd # (Being neat and tidy is good)
wget http://www.silfreed.net/download/progs/dhcp2dnrd.pl
wget http://search.cpan.org/CPAN/authors/id/D/DL/DLUX/Class-Date-1.1.9.tar.gz
tar xzvf Class-Date-1.1.9.tar.gz # to extract the perl module.
mv Class-Date-1.1.9/Date* /usr/lib/perl5/5.8.8/Class/ # to copy the perl module in place
vim dhcp2dnrd.pl # Or use your personal favourite editor. Unless it’s emacs or something, because I don’t think that’s installed.
Personally, I change the “home.net” line to “d.water.gkhs.net”, because it fits my network model better. You do need to change the “$dhcpdpath” to “/usr/etc/dhcpd.leases”, however.
Finally, smoothwall no longer uses dnrd, so either comment out the entire bottom of the file after “# restart dnrd”, or rewrite that to work. I’ve modified the code in mine to “work”, but it’s mostly cargo culty.
Downloading Douglas’ script, I found it had windows line endings, which confused me. You can convert it back to unix format in vim with “:set fileformat=unix”. If you’re using mine you shouldn’t need to.
Finally, run it, check the output of /etc/hosts is roughly what you expect, then throw the script into cron like this:
cp dhcp2dnrd.pl /etc/cron.often/
And that appears to work. You can grab my copy of the code from github should you want to.
Wednesday 03 March 2010 17:32 MST
A few people asked me what I was doing with Lua, so here it is: http://beust.com/ucq.
It’s a simple add-on called “Ultimate Craft Queue” that I wrote to help me craft glyphs. This will probably only mean something to WoW players, but in short, with some discipline, it’s possible to make a lot of money with this profession in WoW, but there are a lot of repeated operations involved in this process, so any automation you can come up with helps. This add-on is simply helping me streamline my process.
Back to Lua, I have to admit that the problems I described in my previous entry were pretty minor overall, and the hardest part of writing this add-on was the WoW Lua API itself, not the language. It is extremely powerful, if the number of add-ons in existence is any indication, but it takes quite a bit of effort to navigate through the scarce and sometimes nonexistent docs. The usage of extra libraries (such as AceGUI) is pretty much mandatory and there are quite a few holes (such as the tooltip API) that make a WoW add-on developer’s life absolutely miserable.
But once it works, it’s really rewarding…
Wednesday 03 March 2010 17:19 MST
Decided to join the early adopters crowd today and use the desktop of the future by switching to using GNOME Shell on my desktop. Luckily with Fedora its dead simple, you just yum install gnome-shell and then switch using the desktop effects widget under Preferences. Scarily simple.
So far GNOME shell has been very stable for me and the user experience has been mostly good. Still feels a little alien compared to what I was used to before, but nothing annoyingly alien. Only irritant so far is that the clock on the shell is using the luddite AM/PM time system instead of the proper 24H clock and I can’t figure out how/where to fix it. 
Will report back next week if I decided GNOME shell is here to stay on my desktop or if its still needs some more love before I am ready to let it rule my life.
Wednesday 03 March 2010 00:45 MST
One of the arguments used to support the adoption of lean techniques in software is the success of Toyota. So do Toyota's recent quality failings undermine the case for lean software development?
One answer for this is to take a sense of proportion. Lean manufacturing techniques were the underpinning of Toyota's rise from an insignificant company in the 1950's to a global giant in the 2000's. By the 1990's other car companies, and many other manufacturers, were busily copying Toyota's techniques. The general sense is that copying these techniques did much to raise the overall quality of cars in the last decade or so. I would be very surprised if the recent problems at Toyota are enough negate that half-century of success.
But a better answer is to remember that Lean manufacturing is about manufacturing not software. The application of lean ideas to software development is a consequence of MetaphoricQuestioning. Lean ideas can help us come up with better ideas for software development, and as such are valuable. But in the end their usefulness lies with how they are used in software and they should be judged on their record here. Their history in manufacturing, both good and bad, is another industry.
Tuesday 02 March 2010 23:40 MST
I don’t normally do anything with Wordpress from a work point of view – I’ve always left such work to ‘designer’ types…
Anyway, yesterday I had a referral for someone who has two fairly busy websites (anorak.co.uk / whoateallthepies.tv) sat on a fairly beefy server (8 core, 16g of ram… oh in a few years that’ll be entry level… but I digress)… anyway, they were having performance difficulties with one site – a bit of investigation found the problem to be related to their migration from one server to another – rinetd was directing traffic from the old server, but had filled the filesystem up and was consuming all cpu time …… Easy enough to fix. Job done. Everything started working again.
After a bit more investigation I found that the two sites needed updates applying and plugins upgraded, and they had no backup job in place *doh* …. Clone the site, whizz through the wordpress upgrade routine on the clone, get the customer to OK it (he did) and then we did it on the live server…. and it looked like a success. Until an hour after I’d done the update and the customer realised part of his front page was missing….
Great. Just what I’d hoped to avoid – delving into wordpress’s code.
On opening up the theme’s index.php file it was easy to see where the content should be – add in some debugging on the clone – and “Oh look – that ‘thing’ is empty.. it should contain ’stuff’….”
Turns out there’s a WP_Query class; and it seems Wordpress 2.9.x treats it’s query slightly differently to previous versions – ‘they’ used category_name=blah as a parameter – this no longer works, instead it needed changing to cat=1234 .. bingo, data returned; site fixed; customer happy.
I breathed a big sigh of relief. I was worried that they previous developers had made some weird customisation to wordpress core which I’d have to forward port and debug/fix.
Being the nice chap I am, I also installed xcache onto the server to help PHP out – I suspect they could cut their hardware ‘allocation’ by half and still have ample capacity to serve the sites. A few days with munin running and I’ll know for certain. Perhaps they’ll appreciate the cost saving?
Tuesday 02 March 2010 22:15 MST
How did I not know about the Sourcecode tags? I have spent years trying to figure out how to post nice looking code to my blog and was misled by all the plug-in nonsense which only applies if you are hosting your own wordpress instance – which I’m not. Here is the link to the docs for formatting source code in your wordpress.org or wordpress.com hosted blog.
I’m a bit disappointed that there is not an HTML language definition, but that’s OK, I write XHTML in any case so XML works just fine.
function makeRooms(edit)
{
var t = getTag("roomsBody", "div");
if (t != null)
{
var div = "<div id='rooms'>";
//if (rooms === undefined) rooms = [];
for (var i = 0; i < rooms.length; i++)
{
if (edit)
{
div += makeRoomDivEdit(rooms[i], imgpath, i);
}
else
{
div += makeRoomDiv(rooms[i], imgpath);
}
}
div += "</div>";
if (edit) div += "<a style='padding:5px;' class='instruction' href='addRoom()'>Add Room</a>";
t.innerHTML = div;
}
}
Tuesday 02 March 2010 22:05 MST
The grails-ui plugin has a very rich feature set, being based on the Yahoo’s YUI library. I have been trying to avoid it, but eventually I have caved and I need a modal dialog box for my web page. I say caved because I think that it is not an intuitive element of a web application and there are enough cross-browser coding issues to make the introduction of a modal window on a web page a bit of a nightmare. Of course that’s all taken care of by libraries such as YUI, but they come at a price; there’s a lot of code to download.
Anyhow, I caved and decided I would install the ui plugin to my project and use the dialog box. It still feels like a bit of a sledgehammer to crack my little nut, but it works and it is not as buggy as the modal DIV I started writing myself.
The trouble is that out of the box you get none of the default styling from the YUI skins. According to the grails documentation it should be sufficient to enclose your dialog in a tag and set the class of that tag to refer to the YUI skins…
<body>
<div class="yui-skin-sam">
<gui:dialog blah blah></gui:dialog>
</div>
</body>
However, this doesn’t work. And neither does this…
<body class="yui-skin-sam">
<div>
<gui:dialog blah blah></gui:dialog>
</div>
</body>
In fact the only way I could get this to work was by adding the “yui-skin-sam” class reference to the body tag in layout/main.gsp.
But that comes with a set of problems of its own, not least of which is that styling the body tag in the main layout means that every generated page gets that styling. If, like me, you had gone to some lengths to define your own styling, that is a real pain because the chance of you not having weird clashes in styles – and yui-skin-sam not overriding them – is almost nil for any semi-serious web app.
So, some words to the wise:
Tuesday 02 March 2010 16:38 MST
I have been writing a lot of Lua recently, and it hasn’t been a very pleasant experience.
The first thing that took a little while to get used to is the fact that table indices start at 1, not 0. Admittedly, it is possible to configure this, but the default is certainly confusing and I have been battling nil errors in tables that I knew couldn’t be empty.
But the worst was the following:
function f()
print("f()")
end
f(2)
function f(n)
print("f(n):" .. n)
end
This snippet will print "f()", indicating that when the interpreter read f(2) and couldn’t find a matching function (since it wasn’t defined yet), it decided to get rid of my parameter and call f() instead. And all this without any error or warning.
It doesn’t stop here:
function f(n)
if n then print("f(n):" .. n)
else print("f(nil)")
end
end
f()
This will print… f(nil).
Here again, the interpreter couldn’t find a signature matching f() so it decided to pick f(n) and simply pass nil as parameter.
I am willing to suffer some discomfort when using a dynamically typed language, but this is really terrible and I dread the fact that it’s inevitably going to cause more bug hunting in the future….
A couple more things of interest:
--[[ This is commented out --]] ---[[ ... but this is not --]]
I like the practicality of this idea.
What I find disappointing is that Lua is a language that’s more recent than Python, so it’s hard to understand why it’s being so lackdaisical about signaling errors. Having said that, there is no question that Lua is alive and well and used increasingly more often as an embedded languages, especially in video games, so it’s most likely here to stay…