Up Till 3

Zipping up OOXML files

adam — Sat, 19 Jun 2010 08:17:06 +0000

For future records, when creating OOXML files from scratch, make sure to use the ‘-D’ option to not store directory entries in the ZIP archive. This makes the Office apps choke (at least on OS X).

Here’s an example of an Excel created XLSX file:
Monstar:n adam$ unzip -l finances Archive: finances Length Date Time Name -------- ---- ---- ---- 1432 01-01-80 00:00 [Content_Types].xml 588 01-01-80 00:00 _rels/.rels 972 01-01-80 00:00 xl/_rels/workbook.xml.rels 828 01-01-80 00:00 xl/workbook.xml 5547 01-01-80 00:00 xl/styles.xml 6646 01-01-80 00:00 xl/worksheets/sheet2.xml 7562 01-01-80 00:00 xl/theme/theme1.xml 22807 01-01-80 00:00 xl/worksheets/sheet1.xml 5595 01-01-80 00:00 xl/sharedStrings.xml 2375 01-01-80 00:00 xl/calcChain.xml 615 01-01-80 00:00 docProps/core.xml 829 01-01-80 00:00 docProps/app.xml -------- ------- 55796 12 files

To recreate that after editting, use the following command:
Monstar:n adam$ zip -rD finances.xlsx * adding: [Content_Types].xml (deflated 75%) adding: _rels/.rels (deflated 60%) adding: docProps/app.xml (deflated 52%) adding: docProps/core.xml (deflated 49%) adding: xl/_rels/workbook.xml.rels (deflated 73%) adding: xl/calcChain.xml (deflated 77%) adding: xl/sharedStrings.xml (deflated 86%) adding: xl/styles.xml (deflated 85%) adding: xl/theme/theme1.xml (deflated 79%) adding: xl/workbook.xml (deflated 46%) adding: xl/worksheets/sheet1.xml (deflated 82%) adding: xl/worksheets/sheet2.xml (deflated 76%)

This will yield a file like:
Monstar:n adam$ unzip -l finances.xlsx Archive: finances.xlsx Length Date Time Name -------- ---- ---- ---- 1432 01-01-80 00:00 [Content_Types].xml 588 01-01-80 00:00 _rels/.rels 829 01-01-80 00:00 docProps/app.xml 615 01-01-80 00:00 docProps/core.xml 972 01-01-80 00:00 xl/_rels/workbook.xml.rels 2375 01-01-80 00:00 xl/calcChain.xml 5595 01-01-80 00:00 xl/sharedStrings.xml 5547 01-01-80 00:00 xl/styles.xml 7562 01-01-80 00:00 xl/theme/theme1.xml 828 01-01-80 00:00 xl/workbook.xml 22807 01-01-80 00:00 xl/worksheets/sheet1.xml 6646 01-01-80 00:00 xl/worksheets/sheet2.xml -------- ------- 55796 12 files

If you skip the ‘-D’ option, you end up with a zip file that looks like this:
Monstar:n adam$ unzip -l finances2.xlsx Archive: finances2.xlsx Length Date Time Name -------- ---- ---- ---- 1432 01-01-80 00:00 [Content_Types].xml 0 06-19-10 04:05 _rels/ 588 01-01-80 00:00 _rels/.rels 0 06-19-10 04:05 docProps/ 829 01-01-80 00:00 docProps/app.xml 615 01-01-80 00:00 docProps/core.xml 16222 06-19-10 04:13 finances 13477 06-19-10 04:08 finances.xlsx 0 06-19-10 04:05 xl/ 0 06-19-10 04:05 xl/_rels/ 972 01-01-80 00:00 xl/_rels/workbook.xml.rels 2375 01-01-80 00:00 xl/calcChain.xml 5595 01-01-80 00:00 xl/sharedStrings.xml 5547 01-01-80 00:00 xl/styles.xml 0 06-19-10 04:05 xl/theme/ 7562 01-01-80 00:00 xl/theme/theme1.xml 828 01-01-80 00:00 xl/workbook.xml 0 06-19-10 04:05 xl/worksheets/ 22807 01-01-80 00:00 xl/worksheets/sheet1.xml 6646 01-01-80 00:00 xl/worksheets/sheet2.xml -------- ------- 85495 20 files
All of those ‘0′ length directory entries make Excel puke.

Additionally, note that you have to add the file extension ‘.xlsx’ (or appropriate for your OOXML file type) in order for Office to open the file after you’ve created it (even though the Office created files don’t have extensions by default). I think there is a way to add metadata that determines the MIME type, and would solve this problem, but that may be a OS X Office problem only.

Amazon EC2 Pricing Analysis

adam — Thu, 03 Jun 2010 01:39:49 +0000

This is part 1 of a 4 part series on EC2.

Part 1 – EC2 Background

What is EC2?

EC2, or ‘Elastic Compute Cloud’ is a primary component of AWS (‘Amazon Web Services’). EC2 is Amazon’s market offering for cloud computing. EC2 allows individuals and organizations the ability to run a full instance of their favorite version of Linux or Windows on Amazon’s gear – the same gear, ostensibly, that runs Amazon.com itself. Users are given the opportunity to host their systems, applications, and files in Amazon’s data centers, on Amazon’s bandwidth and Internet connectivity, and run everything on Amazon’s server hardware (including redundant storage, power, etc.).

What is an AMI?

EC2 is primarily competing with the colocation and shared hosting providers. Users that pick EC2 typically do so because of the previously mentioned platform benefits, as well as it’s extreme flexibility in deployment capabilities. Unlike traditional colocation or shared hosting, where putting 30 new servers into production is an arduous, expensive task – possibly involving paying remote help for racking, configuring all the gear, burning in all of the equipment, etc., EC2 uses an entirely virtual setup. Bringing up 300 new servers (called ‘instances’) is no more difficult or time consuming than deploying 1. This is accomplished by way of what Amazon calls an AMI, or ‘Amazon Machine Image’. AMIs are basically special instances of disk images that are copied and booted on demand by the EC2 system. If you are familiar with VMWare, it is much like the VMDK files used – except it is cloned and booted, rather than run directly (this is what gives you the flexibility to stand up multiple instances all in one go).

Who uses EC2?

What companies and individuals would use EC2? A number of high profile websites, including the New York Times, Reddit.com, and SmugMug utilize EC2 for a variety of reasons. Some companies chose to use EC2 for expanding their data-crunching needs (the New York Times does this), while others host their entire site on it (Reddit.com does this). Another significant user of AWS are individual developers or small teams – the overhead and initial cost to get a server up and running, while retaining full control of it, are extremely minimal – a few hours of work to prep an AMI (or, no time at all to use an AMI from the AMI library!), and a cost of pennies an hour in most cases.

EC2 Limitations

EC2 is not without it’s limitations however – operating system choice is limited, as are some network services (such as email – EC2 is on many RBLs, unfortunately). Despite being on Amazon’s highly redundant cloud, there are still outages (including some fairly long ones) – nobody is perfect.

To work around such limitations, and to leverage an extended amount of reliability and Amazon engineering there are 2 major concepts that must be understood – regions and availability zones.

Regions

Regions are basically exactly what they sound like – a specific geographic region used by Amazon to host EC2. Amazon currently offers EC2 hosting in 4 regions around the world – Northern Virginia, Northern California, Northern Ireland, and Asia Pacific (Singapore).

Availability Zones

Within these regions, there are different availability zones. Availability zones allow users to run instances in the same region, but on physically diverse infrastructure – separate diesel generators, power racks, disk storage, etc. This allows for users to work around potential outages and problems, by leveraging Amazon’s large scale data center engineering, and provides another big benefit over colocation or shared hosting (how easy is it to guarantee 2 of your servers with one provider are on different generators?).

AWS API

Amazon allows you to fully utilize all of this flexibility and capability via their AWS API. This is a SOAP based web service that provides all management functionality – everything from starting new instances to changing security keys. The use of the API allows many otherwise time consuming and arduous system administration tasks to be completely automated.

Part 2 will continue on with an overview of the EC2 pricing model – what a specific server instance will cost you is not quite as straight forward as it could be.

A Sneak Peak

adam — Tue, 20 Apr 2010 06:58:37 +0000

I’m working on an extensive post about the market dynamics and pricing of Amazon’s AWS EC2 Spot Instance market. Here’s a sneak peak:

iPhone Forensics Sans iPhone

adam — Tue, 20 Apr 2010 06:27:36 +0000

This is a presentation I developed and gave at HRGeeks ][, on March 14. The bulk of this was original research.

Grab the PDF here.

iPhone forensics, without the iPhone

Python MySQLdb snippet

adam — Tue, 15 Jan 2008 06:10:51 +0000

So, for the past few weeks, I’ve been hacking around with Python’s MySQLdb DBAPI v2 driver. After getting so used to mysqli in PHP, the move to MySQLdb was a mixed bag – some things are awesome, others not so much.

First the awesome – it’s so much cleaner to do safe, simple queries with python’s dbapi, than with the PHP mysqli stuff. Just call cursor.execute(query, arguments) and pass in your SQL and it’s arguments, and all the escaping and safety stuff is taken care of behind the scenes. No ugly hacks to check to see if something is a digit or a string and to properly escape/format the final query for you. In addition to that, connecting and setting the character set and forcing the driver to use unicode is all just 2 little options passed in the connect() method.

Now, the not so awesome – I really liked mysqli_fetch_assoc() for returning data from the database in a easy to use form – an associate array (in case it wasn’t obvious!). DBAPI drivers for python do not provide such a feature out of the box – your data comes back as a tuple, rather than a dict. While this makes some sense, the lack of a way to get a dict back seems to be rather un-pythonic. It violates the ‘Explicit is better than implicit’ and ‘Beautiful is better than ugly’ Zen of Python aphorisms. It requires implicit accessing of your data, as you must use tuple slices/subscripts instead of keywords to the dict, and that requires ugly numbers dropped into the otherwise easily readable code.

In an effort to save myself some trouble, here’s a quick way to emulate the fetch_assoc functionality, built in to a generator function so it can be used easily in ‘for’ statements and the like:

#!/usr/bin/env python
# Requires a version of python that supports generator functions (2.3+)
import  MySQLdb

connection = MySQLdb.connect(host="localhost", user="username", passwd="passwd", db="myInstance", use_unicode=True, charset="utf8")
cursor = connection.cursor()
def query(query, args):
        numRows = cursor.execute(query, args)
        tries = 0
        dbKeys = cursor.description
        row = {}
        while tries < numRows:
                tries = tries + 1
                dbRow = cursor.fetchone()
                for item, key in zip(dbRow, dbKeys):
                        row[key[0]] = item
                yield row

sql = """SELECT foo, bar FROM baz WHERE id = %(id)s"""
args = {'id':'adam'}

for rows in query(sql, args):
        print rows

That outputs:

	{'foo': u'fooValue', 'bar':'barValue'}
	{'foo': u'fooValue2', 'bar':'barValue2'}