Open Formats and Open Source for Better Government

The Government of Canada is currently reliant on proprietary file formats and proprietary software applications, which lock it into a licensing bind with a single software manufacturer — Microsoft. There is not only a question of cost — as we pay a monopoly corporation for per-seat licenses to run software that already dominates the market — but more importantly, there is the question of future access to our own data. In this post, I'd like to share my thoughts on both issues.

Before you dismiss the idea of a major institution losing access to its stored data as ludicrous, consider this quote from Natalie Ceeney, chief executive of the UK National Archives:

"If you put paper on shelves, it's pretty certain it is going to be there in a hundred years. If you stored something on a floppy disc just three or four years ago [2003-04], you'd have a hard time finding a modern computer capable of opening it. Digital information is in fact inherently far more ephemeral than paper. The pace of software and hardware developments means we are living in the world of a ticking time bomb when it comes to digital preservation."

The UK National Archives includes a collection of 900 years of written material. As of 2007 they estimated that 580 terabytes of their data (the equivalent of 580,000 encyclopedias) was stored in file formats which have since become extinct.

While Ceeney's estimate of "three or four years ago" strikes me as somewhat hyperbolous, changes in technology have certainly changed our selection of media storage and electronic file formats. 5.25" floppy disks are nearly absent from current use, and 3.5" disk drives are disappearing as standard equipment on modern computers. Information on high-density floppies isn't necessarily difficult to access, though doing so may require the purchase of an external USB floppy drive and compatible application software. I've already experienced some electronic data loss due to incompatible, extinct file formats. Much of the material I produced in the 90's, both academic and personal, was saved in AppleWorks or pfs: Professional Write format.

For me, the most uncomfortable aspect of the source article I've quoted above is that it announces the "solution" to the National Archives' problem in the form of a partnership with Microsoft, who were contracted to ensure future compatibility to read old formats, and ostensibly, to convert and write them into a modern, proprietary format. So, while access to data may be restored, the issue of reliance on a monopoly corporation remains (or, in the case of the National Archives, begins).

The Government of Canada's relationship with Microsoft initially came by way of a competitive process, but in my opinion it has become a relationship of dependence, because Government data is saved in an electronic format wholly owned and controlled by a private, foreign corporation.

Storing files in a proprietary format like Word .DOC, Excel .XLS or PowerPoint .PPT places the Government in a position of perpetual dependence on private enterprise. We require a certain office suite (Micosoft Office) that is available from only one company to access our own data — the peoples' data. Why?

While the lack of a suitable alternative may have once made this a necessary concession, other formats presently exist which will allow all the various types of information we use to be preserved in a format that can be opened by a variety of software packages, indefinitely, without concern for legal, financial or technical constraints.

In a recent exchange I had with a colleague, two important questions were raised: (1) is the mandatory use of Microsoft Office by employees anti-competitive? and (2) in the competitive process that led to the Government's use of Microsoft Office, were free/open source options in the running? Given Microsoft's monopoly of the office software industry, and their trials for engagement in anti-competitive business practices in court systems worldwide, these are legitimate concerns.

One of the Microsoft practices that has been criticized is the control and regular alteration of file formats, including those used for documents, presentations, spreadsheets, databases and websites. This creates software and version incompatibilities, regular upgrade cycles, and difficulties for open source and other private competitors who wish to support the proprietary standard. This business practice is known as "embrace, extend and extinguish", and it was a subject of the United States v. Microsoft antitrust trial.

OpenDocument, an ISO and IEC International Standard registered as ISO/IEC 26300:2006, is the leading open standard used by governments in the United Kingdom, Germany, Australia, Japan, the United States, and over a dozen other nations. In response to this increasing popularity, Microsoft (via an interview with Tom Robertson, GM of Interoperability & Standards) stated that if OpenDocument becomes a requirement then Microsoft would implement it, but extend it. Later, Microsoft presented the ISO/IEC with a competing 'open format' called Office Open XML.

While Microsoft should clearly remain eligible to compete in future competitive processes, in my opinion the Government should adopt open data file formats that are independent of Microsoft, or any other proprietary vendor/product. When external contractors are required for future business, full compatibility with a list of Government-approved open standard formats would be a condition that bidders must guarantee as part of the competitive process.

Eliminating proprietary formats increases opportunities for other corporations offering service solutions to the Government. No company's ability to compete with or replace a prior service / software provider will be impaired by their lack of the previous company's internal knowledge or control of a closed format. Correspondingly, it increases choice for government, increasing the pool of companies that can knowledgeably manage information (now stored in an open format).

In North America, the Commonwealth (state) of Massachusetts was an early adopter of open formats. In 2005, Eric Kriss, Secretary of Administration and Finance said:

"It is an overriding imperative of the American democratic system that we cannot have our public documents locked up in some kind of proprietary format, perhaps unreadable in the future, or subject to a proprietary system license that restricts access."

Their data is now accessible by a variety of free and open source office suites including OpenOffice, KOffice, and StarOffice and free stand-alone word processors like AbiWord. This not only ensures the State’s continued access to the data into the future, but also offers it in a format that can is usable by their citizens regardless of their ability to pay for proprietary software like Microsoft Windows or Microsoft Office.

Consider the Canadian people I serve. At best, the inability for people to afford software creates a technology gap between classes, where the have-nots do not possess the tools they need to be productive in an information society. At worst, this gap fosters software piracy as people feel compelled to illegally share and crack copies of MS Office.

While it is feasible for a struggling family to obtain an older computer that runs some version of Windows, it is of limited use without an office suite. OpenOffice is compatible with Windows, Linux and OSX, is publicly downloadable from the Internet, and free of charge or other licensing fee. For those without broadband access, copying and distribution on CD or USB drive through friends at school or at work is not only allowed, but encouraged.

Now consider the government I serve. 2009 was a year characterized by the state of the economy: shrinking expenditures by cutbacks, and shrinking staff levels by attrition. Could a government initiative to reduce dependence on proprietary software and formats contribute significantly to a future economic action plan, reducing financial burden on both the Government and its citizens? I believe so.

OpenOffice's lack of cost does not reflect in any way on it's value or utility. As I've already mentioned, OpenOffice and the OpenDocument group of file formats are presently being used by the governments of Massachusetts, Belgium, Finland, the National Archives of Australia, and the Allahabad High Court of India, among many others.

The adoption of open data formats is a logical first step towards the use of free and open source software (FOSS) applications (like OpenOffice suite), which will reduce or completely eliminate per-user licensing fees on the thousands of computers where the Government of Canada currently uses proprietary software. In an ideal world, my government would also roll out an open source operating system, but... one step at a time.

If we do hope to update all the desktops in the Government of Canada — many of which were recently "upgraded" to Windows XP after having run Windows 2000 for many years — an open source operating system like Linux or BSD would offer the same savings, stability, and control over our information infrastructure that an open format offers to the information itself.

Linux is free, and its ability as a server is demonstrated whenever you are on the Internet (60% of the web is power by Linux; 30% by Microsoft). As a graphical desktop operating system, Linux has replaced Microsoft Windows at Google, Novell, Panasonic, Tommy Hilfiger, Toyota Motor Sales U.S.A, the U.S. Army, the U.S. Federal Court System, and the U.S. Postal Service.

Just some food for thought. To conclude:

Freeing data: Open file formats ensure the government's perpetual access to its own data well into the future, regardless of the software used in the coming decades. While applications change over time, the standard evolves and endures independently. More importantly, the data remains in an open format (beyond plain text) usable by citizens, regardless of their ability to pay for software.

Cost savings: OpenOffice (and other open source office software) is free of charge.

Conversion ability: At present, OpenOffice, StarOffice, and other suites can load and save documents created by Microsoft Office. No special software or expertise is needed to change them to OpenDocument (or, to re-save them back into Microsoft format if necessary). NOTE: Due to the nature of the Microsoft proprietary format, complex formatting may not convert exactly, which is why the use of an open format like OpenDocument is preferable.

Choice: OpenOffice and the OpenDocument format is something that the government can adopt and support internally. Additionally, we can choose a proprietary office suite that uses open file formats (like StarOffice) if a commercial product is preferred.