Discussing the nuts and bolts of software development

Wednesday, July 30, 2008


Snake On A Phone

For a few years now, my main task at work has been working on the firmware of an IP phone. The phone runs VxWorks on a MIPS32 CPU; the firmware is written in C and C++.

For slightly less time, I've been dabbling in python on my own time. Freedom from explicit typing was a refreshing change, and python's tendency to Just Work was a nice bonus.

It was perhaps inevitable that I would one day try to combine phone and language. (why? because they were both there)

It wasn't obvious that the idea stood a chance. VxWorks is a bit off the OS beaten track, and might not provide all the functionality needed by Python's "core" (not with the same names, anyway). There might be some processor-specific pieces that would rule out MIPS32. And even if I could get something built, would it fit in the 2 or 3 MB of RAM (and even less flash) I could spare?

As it turns out, there was very little to worry about. Python's code is impressively (if perhaps unsurprisingly) portable, only needing a couple of tweaks to its build system and none at all to its source code. There doesn't seem to be anything CPU-dependant; and in the end, adding python to my firmware only cost me 1MB. It took me only a few evenings of tinkering to get a libpython built, linked into my firmware, and loaded on my phone, to the point that I could run this little experiment at the VxWorks shell:

-> Py_Initialize()
value = 42 = 0x2a = '*'
-> PyRun_SimpleString("print 'Hello, World!'\n")
Hello, World!
value = 0 = 0x0

(the VxWorks shell being a peculiar animal that allows calling C functions by name, in this case giving me access to Python's C Extension API for a near-REPL experience)

For my purposes, that's enough; I know it can work, and that's all I wanted. I don't expect to ever go further than this. But as little as it is, publishing how I got there might help someone get started on a real project; so here goes:

Porting python in 10 easy* steps

*for a suitable definition of easy

  1. As far as embedding Python in an existing application (or firmware) is concerned, Python's own documentation should give you most of what you need
  2. You'll need a cross-compiling toolchain, i.e. a compiler that can be used on one platform (e.g. x86) and produces executables for use on a different platform (e.g. MIPS32). GCC is your best bet; it's what will make python's build system happiest, and there's lots of resources on getting a GCC cross-compiler working on the web, though it looks a bit daunting to me. I was fortunate in that, since I was already set up to build firmwares, I already had all the needed tools; I would guess that most people engaging on a similar project would be in the same position.

  3. In addition to the compiler (and assembler, linker, etc), you'll want to have a Unix-like environment to run Python's configure script and makefiles. If you're on Windows, cygwin will serve nicely.

  4. The 'configure' script needs some tweaking: it contains a few uses of AC_TRY_RUN, which will fail when cross-compiling.
    • If you have a working 'autoconf', the simplest is to edit the 'configure.in' file. You can either remove the AC_TRY_RUN tests altogether or replace them by the newer, more cross-compiler-friendly AC_RUN_IFELSE. Then run 'autoconf' to regenerate the 'configure' script.
    • If you don't (as I didn't), you can brace yourself and go edit the 'configure' script directly. Running the script produces error messages that gives something to search for. The fix is actually simple: just remove the calls to 'exit' to allow the error to get ignored.

  5. The makefile also needs tweaking: just like 'configure', at some point it tries to compile and run a program. This appears to be in order to autogenerate some source files, which fortunately are already provided in the source distribution; so it's safe to disable this step. The simplest way:
    • open "Makefile.pre.in"
    • find the place where "$(PGEN)" shows up AFTER a ':'
    • remove "$(PGEN)"

    (this will only prevent the executable from getting built. The makefile will still attempt to run it, but it's written so that the resulting failure is ignored)

  6. The configure script and makefile try to guess at the name of tools to use; you can give them a hint with environment variables. In my case I needed to set CC (the C Compiler) and AR (the "archiver", ie. what creates static libraries)

  7. If you need to specify special command-line options to the compiler, environment variables can also be used. Annoyingly, 'configure' and the makefiles use different variable; you'll want to set CFLAGS and BASECFLAGS to the same thing.

  8. Finally you'll be ready to run the 'configure' script. You need to give it the special options --build and --host to tell it you're cross-compiling, something like:

    $ ./configure --build=win32 --host=vxworks

    (win32 and vxworks were a wild guess that happened to work for me. I got the impression the specific values didn't particularly matter)

  9. You can then run 'make' to compile everything. If, like me, all you need is a static library, this will do it:

    $ make libpython2.5.a

  10. There's a good chance some files under Modules/ will fail to compile (in my case, posixmodule.c). The file Module/Setup specifies (in a rather well-documented way) which Python modules (written in C) should be built into the python library; comment out the failing one, and re-run 'make'. I only had to disable posixmodule and pwdmodule; YMMV.

And for me, that was it; nothing else needed manual intervention. If you run into more troubles (e.g. trying to build the actual python.exe), I'm afraid you're on your own.

My next step was to figure out how to integrate the python library into my firmware; you'll have to figure out the corresponding steps for your own firmware/embedded application/whatever. Start with the 'embedding' link for how to access python code from your code.

If you want to be able to load python source files with 'import', pay particular attention to what that page says about PYTHONHOME; as for me, I put a putenv(PYHONHOME=/whatever") before the Py_Initialize call, letting me import /whatever/python2.5/*.py files (and possibly, though I haven't tried, .py files contained in a /whatever/python2.5/libpython2.5.zip)

Happy cross-compiling!

Labels: , , ,

Thursday, July 24, 2008


Outlook Entry IDs Made Easy

When developing for Outlook you will quickly realize that there's more than one way of doing things. For example, take a look at the number of APIs for interacting with Outlook:That doesn't even include the six libraries included in the Outlook 2003 Integration API or the excellent 3rd-party Redemption library. I'm positive there are others. Usually you won't use all these APIs in the same project. However, if you want to do anything significant in Outlook, you'll need to use the OOM and MAPI libraries.

It's important to understand the relationship between MAPI and the OOM:When you work with these APIs, you'll hit on the problem of linking them together. Outlook Entry IDs are a good example. Entry IDs uniquely identify most objects in the OOM and MAPI. Being very flexible, Entry IDs can take many shape or form and can cause headaches when you try to handle them.

At its simplest, an Entry ID is a variable length byte buffer, sometimes represented with a simple byte array. In MAPI, most Entry IDs are represented by the ENTRYID structure:


typedef struct { 
BYTE abFlags[4];
This structure usually comes wrapped in the SBinary structure.


typedef struct _SBinary { 
} SBinary, FAR *LPSBinary;
In the Outlook Object Model things take a turn for the worst:Is your head spinning? Mine was.

I solved the problem by creating a class (yet another format!) to facilitate manipulation of all these formats:
class EntryId
EntryId(EntryId const& entryId);

// All explicit so that it's always clear what we are doing.
explicit EntryId(BSTR const* str, bool hexEncoded = true);
explicit EntryId(SBinary const* binary);
explicit EntryId(SAFEARRAY const* array);
explicit EntryId(unsigned long count, unsigned char const* bytes);

// Again, no implicit conversion
CComBSTR toString(bool hexEncoded = true) const;
CComSafeArray toSafeArray() const;
std::vector toByteArray() const;
// ... add more as you need them

// TODO: Add more custom helper functions.
bool isLongTermId() const;

// TODO: Add comparison operators for convenience and ordered
// container support (such as std::map keys). When comparing
// Entry Ids, remember you need to use the
// IMAPISupport::CompareEntryIDs method.

std::vector<unsigned char> m_buffer;
Now whenever I get an Entry ID in any format, the first thing I do is convert it to an EntryId instance. This adds many convenient functionalities:

Labels: , ,

Wednesday, July 23, 2008


Software Testers - Don't Underestimate Their Worth for Success

Sometimes, testing your application isn’t enough: sometimes your test cases need to be tested! Testing doesn’t always give you what you’d expect. In extreme cases, it can even give you a false sense of confidence in a product that is a complete failure.

Recently, I was managing an offshore team developing a component for our client's core enterprise application. We wrote extensive test cases and, because we were concerned about a lack of domain knowledge, we even had architects write some of the test cases. Things were looking good as we approached the end of the development cycle: the product was behaving as it was supposed to; we were passing test cases; we were greenlighted by the QC ‘process’.

Our confidence was high, and we were ready to move on to automated testing. Then we found the problem. It turns out that our test cases were written with a single user in mind. The minute we started to use this component with concurrent user access, the system would pretty much lock up. Our confidence evaporated.

This was a big mistake. Architects and technical leads both reviewed the test cases. Nobody found the oversight at the time. This was compounded by an ‘if it passes the test cases then it works’ mentality.

Unfortunately, a solution to this problem isn’t as easy as ‘follow these three steps’. But if we can take away a lesson learned from this, it is that testing the right way is not only very important to the success of a product, but it also shouldn’t be taken for granted. If you haven’t noticed, QC/QA and testers were not included in the writing of test cases, thus bypassing all kinds of valuable experience that would most likely caught our newbie mistakes.

Bottom line - Don’t underestimate the worth of your testers and their experience, it might just come back to bite you in the future.

Labels: , , ,

Thursday, July 17, 2008


Maintenance Nightmares: Commented out code

I was working on an estimate for a prospective customer, and luckily I had the code to examine beforehand to see what it was doing.

While examining the code, I noticed code like this:

// Fix for bug 1021892
// some code that was commented out.

There were instances of this throughout the code.

When I see this I instinctively cringe, I immediately think that there is no proper source control or defect tracking procedures (or tools) in place. While there may be the odd corner case where commented out code is acceptable, in general it is not and should be avoided.

There are many better ways of dealing with this, the first way is to just delete the line of code, and when you commit/check-in the code, make sure you put a meaningful comment in the commit logs that ties it to the defect it was changed for. You can often search the commit logs, and if you put the bug id in the commit log, you will get the change in your search.

Another easy way to do this is to delete the line of code, make a patch, and when resolving the bug in your bug tracking software, attach the patch file to that issue. Then you can always see what you changed to fix the bug.

Does anyone else having any suggestions for this?


Friday, July 11, 2008


Alignment Matters

When do these two snippets behave differently?

void* p = something();
int i = *(int*)p;

int i;
void* p = something();
memcpy(&i, p, sizeof(int));

Answer: when 'something' returns a value that's not a multiple of "sizeof(int)".

Some processor architectures only allow loading memory into N-byte registers from memory addresses that are a multiple of N (e.g. multiple of 4 for 32-bit registers). Using misaligned addresses can have some various interesting consequences, depending on the CPU; I've heard of at least these:

Last week, I got lucky: I discovered that the MIPS32 CPU on the phone I was programming falls in the "crash" category. (why lucky? because an "address load exception" message is much easier to debug than some corrupted data)

Usually, we don't have to worry about such alignment issues; the compiler and runtime make sure that all objects it allocates go at addresses that have the right alignment for their type (e.g. malloc must return memory "suitably aligned" for all possible types).

Trouble comes when we lie to the compiler, such as telling it by a cast that "p" points to an "int" when such is not the case. This is what happened to me: I was parsing a file, and the 4-byte-value-that-should-be-put-in-an-int followed an arbitrary-length string. It ended up on an odd address, and boom. (Here's another way I was lucky: it COULD have been a nice multiple of 4 in all my tests, only to come out odd on a client's desk)

Functions like memcpy, of course, are required to work with all addresses (as expressed by taking void* parameters, which require no cast).

Lesson of the day? Don't lie to your compiler!
(alternate lesson: "casts: evil AND chaotic"?)

Labels: , ,

Thursday, July 10, 2008


Compile Time Semantic Checking

C# is a TypeSafe language, what about SemanticSafe?

In a project I'm working on we have a lot of database records ids being passed around as integers. Today I needed to add a new record id to this list. Adding the new item to the function signature is easy. The hard part comes in making sure the right value is being assigned.
Public void MyFunction(int TableA_ID, int TableB_ID, String someOtherValue);

int A_ID = 5;
int B_ID = 4;
MyFunction(B_ID, A_ID, "Hello World");

So what happens when I compile? Nothing, it works fine. Yet that line where I called the function has just corrupted the entire database because those database ID's were transposed. Which kinda sucks.

It's one thing to guarantee type safety, what what about semantic safety? An idea popped into my head to use the existing type-checking to accomplish this.
    public struct UserID
private int value;
public static implicit operator int(UserID source)
{ return source.value; }

public UserID(int intValue)
value = intValue;

private void TryToFail(UserID test) { }

UserId works = new UserId(5);
int fails = works;

TryToFail(works) //compiles fine
TryToFail(fails) //Doesn't compile

Even though both works and fails contain the exact same value, the function can be made to fail if the values are put in the wrong order. I used an implicit operator so that I didn't lose the convince of having a simple integer, neato.

Obviously this technique is not something you would want to use for every possible field as there is one extra step involved in getting the value. But if you only did this for the record IDs I think it would be enough gain to make it worthwhile.

I would like to think of a way to do this so that it has compile time checking, but once compiled only contains the primitive value. Any ideas?

Labels: ,

Wednesday, July 09, 2008


Trouble Getting Your Custom Protector Called?

You might be getting a lot of grey hair by developing your own custom protector for SharePoint. Until recently, I was right there with you. Having troubles getting your custom protector to be called by SharePoint? I'm here to help. I'm happy to tell you that it could be a simple fix. A simple flag as a matter of fact. If you haven't heard of the IrmAddInsEnabled flag, then this is what you must do.

You need to set the IrmAddInsEnabled flag to true in the Central Admin page to get SharePoint to call any of the protector function calls (HrIsProtected, HrProtect, HrUnprotect). This sets a Boolean value that specifies whether to enable Information Rights Management (IRM) addins.

You can do this directly on the MOSS server from the 12 Hive (C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\BIN) by calling the following:

On the MOSS server:

'stsadm -o setproperty -propertyname irmaddinsenabled -propertyvalue "yes"'

Alternatively, you can do this programmatically by using SharePoint via the IRM Settings by using SPIrmSettings.IrmAddins. This can be done at the farm level or as granular as a per-list basis.

IRM Settings for the FARM:

SPWebService svc = SPFarm.Local.Services.GetValue();
SPIrmSettings irmSettings = svc.IrmSettings;

//Specifies whether to enable Information Rights Management (IRM)
//locally for sites
irmSettings.IrmRMSEnabled = true;

//Specifies whether to enable Information Rights Management (IRM)
irmSettings.IrmAddinsEnabled = true;

//Sets the number of times the Information Rights Management (IRM)
//settings have been changed
irmSettings.IrmChanges += 2;


IRM Settings for the Individual Lists on FeatureActivated:

SPWeb web = (SPWeb)properties.Feature.Parent;
foreach (SPList list in web.Lists)
// Specifies whether to enable Information Rights Management (IRM)
// for the list
list.IrmEnabled = true;

// Specifies whether to allow users to upload documents that do
// not support IRM
list.IrmReject = true;


After you have done this, your custom protector should be called when the documents are checked in and out of your SharePoint system via the HrIsProtected, HrProtect, HrUnprotect methods.


Labels: , ,

Monday, July 07, 2008


Mining Project Scope with Python + SVN

After developing a component for a fairly large enterprise application, I was tasked with coordinating a complete review of all code touched by our team. As the project involved hundreds of commits across every tier of the application, this was not a trivial task. I didn’t want to make developers hunt through SVN logs and JIRA - even if the result was accurate, producing meta-documentation in the time allotted for actual documentation seemed wasteful. There had to be a way to automate it. After failing to find an appropriate SVN tool via Google, I weighed the pros and cons [1] then decided that using an SVN API would be easy enough.

To make it interesting As part of my commitment to Macadamian’s core value of continual improvement, I decided to try out Python. The following script was maybe an hour of entertaining work that saved at least a day of tedium.

import pysvn

# Script variables. Could be read from a command line or config. file.
username = "jlennon"
password = "yoko"
projectFirstRevision = 103991
svnRoot = "http://path/to/svn/trunk/"

# Team members’ names for SVN commits.
users = set([

# Set up SVN callbacks.
def get_login(realm, username, may_save):
return True, username, password, False;

def get_log_message() :
return True, "log message called"

client = pysvn.Client()
client.callback_get_login = get_login
client.callback_get_log_message = get_log_message

# Convert the revision number into a pysvn revision.
projectFirstRevision = pysvn.Revision(pysvn.opt_revision_kind.number, projectFirstRevision)

# Get all commits since the project's first commit.
items = client.log(

# Set up variables for storing commit information
allPaths = set([])
removedPaths = set([])

# For each commit entry *made by one of this project's users*
# Add any deleted files in the commit to removedPaths
# Add any added/updated files to the list of all paths
for item in [ e for e in items if e.author in users ]:

for changed_path in item.changed_paths:

if changed_path.action == "D":

# Remove deleted files from allPaths, then display
allPaths = allPaths - removedPaths

for path in sorted(allPaths):
print path

Overall, I was impressed by Python. My biggest problem was with the structure of the documentation, but that’s to be expected with any new language. My biggest surprise was that the indentation rules didn’t bother me – why is this cited as a show-stopper by so many developers? I’ll definitely be coming back to Python to test its OO facilities.

[1] The main con being that a developer would much rather write a neat SVN spider than write documentation. In a leadership role, you have to think twice before eagerly firing up your compiler to solve a problem.

Wednesday, July 02, 2008


Getting RIL!!

Have you ever wanted to mess around with calling functionality of a mobile phone running Windows Mobile?

I have this nifty phone but how do I control it programmatically?

Well, first of all, the Windows Mobile Radio Interface Layer works by having specific modules control the cellular component. Since we are in North America there are two different cellular protocol implementations: GSM and CDMA. Above this layer is an abstraction layer that hides the implementation of the underlying protocol so that applications/OEM phone developers can use the functionality of call handling (and other useful information) without getting down to how things work under the hood. So how do we, as an application, talk to this abstraction layer, which we'll just clump together and call it the RIL( Radio Interface Layer )? Good Question!

Since cellular devices are really just modems that use AT commands, the RIL operates mainly from an asynchronous callback system. In fact most of the RIL functions really just map to AT commands. The application accessing the RIL makes a request to the RIL, and the RIL takes some time to process this request and sends back a result code indicating the success or failure of the request. In addition the RIL spits out notifications which our application can hook into and be informed of status changes related to the cellular network. So let's dig a little with some RIL examples.

There are a few pitfalls by going down this path. Firstly you might run into situations where you're fighting for control with the resident dialer (the default phone application supplied to the mobile device; developed by Microsoft). There may also be some undocumented internal states that need to be obeyed or else it can lead to unintended behavior (i.e. device lock up).

There are also some differences to note between CDMA and GSM RIL implementations. With GSM devices there can be multiple lines, and as such the RIL can fetch call information for each line. RIL can also manage the GSM lines to perform actions such as putting the call on hold or switching lines. In the CDMA environment, there is only one "real" line and one "virtual" line. That is, one line is used to represent two lines. Access between the two lines is achieved by sending a "flash" command. Call management is also not possible with CDMA. So when you call "RIL_GetCallList" for GSM, for example, you may be returned information for one or more lines depending on the call state. For CDMA, you will only see at most one line.

You will have to get your hands on the ril.h and ril.lib files. Both of these files are floating around somewhere on the internet (try here). If you can get both of these files, link your application with ril.lib and away you go. If you can’t find ril.lib but have ril.h, you can dynamically link with the ril.dll file (contained in the “windows” folder on the device) since you know what some of the prototypes are.

First we need to initialize our application with the RIL so that we can receive both results and notifications callbacks. Here’s some code to do this:


// keep this handle handy as we need it for further RIL commands
HRIL rilHandle;


HRESULT hr = RIL_Initialize(
1, // index of the RIL port to use (e.g., 1 for RIL1:)
&resultCallBack, // this is a pointer to your result call back method
&notifyCallback, // this is a pointer to your notify call back method
RIL_NCLASS_ALL, // all notification (except device specific)
(DWORD) this, // custom param (could be a pointer to an instance of a class)
&rilHandle); // returned handle to RIL instance


void CALLBACK resultCallback(
DWORD dwCode,
const void *lpData,
DWORD cbData,
DWORD dwParam)

// handle the results


void CALLBACK notifyCallback(
DWORD dwCode,
const void *lpData,
DWORD cbData,
DWORD dwParam)
// handle the notification


Note that we’ve initialize the RIL with a reference to our class as the user parameter which is passed to the result and notification callbacks. The parameter is optional; however having the class reference would be especially useful if you would like to perform further actions after receiving a result or notification. More on this later.

OK so what - I've just initialized myself with RIL - what can I do now?
To answer that question, anything under the Sun related to call handling, and then some!!

For example if you wish to check the call status of the phone you can do the following. First send a request for call list:



void getCallList()
// call RIL_GetCallList with RIL handle from the initialization
cmdID = RIL_GetCallList(rilHandle);

Next we have to wait for the results:


void CALLBACK resultCallback(
DWORD dwCode,
const void *lpData,
DWORD cbData,
DWORD dwParam)
If (cmdID == hrCmdID && RIL_RESULT_OK == (dwCode & 0xff))
// lpData points to an array of structures.
// extract the call info from the structure and do something...

Notice how the cmdID return value of RIL_GetCallList matches hrCmdID in the callback. In a realistic scenario, multiple commands are sent and the results do not necessary return in the same order the commands were sent. As a result we would need to queue up the command IDs in order to properly identify the command represented by each callback

Now what is notifiyCallback useful for? This function is called basically whenever an event occurs that is of a notification class that was registered during initialization (we registered with RIL_NCLASS_ALL in this case). For example, for an incoming call you would receive and handle the “ring” notification as follows:

void CALLBACK notifyCallback(
DWORD dwCode,
const void *lpData,
DWORD cbData,
DWORD dwParam)
switch(dwCode & (0xff | RIL_NCLASS_CALLCTRL))
// do something here

Since the phone is ringing, it is probably best to get more information about the incoming call. If you remember from initialization we use the class reference as the custom parameter. This gives us the flexibility to send further RIL commands by either passing in the RIL handle (HRIL) or point to a function in the class. In this case we can point back to the getCallList function created earlier.

MyClass* myClass = (MyClass*)dwParam;

If we don’t have a function to point to, we can always send an RIL command on the fly by passing in the RIL handle from our class. For example to answer the call right away:

MyClass* myClass = (MyClass*)dwParam;
HRESULT hr = RIL_Answer(myClass->rilHandle);

As a note, if we're not using the interface anymore we'll need to de-register ourselves:


To learn more about RIL and all its wonders, you should closely examine the ril.h header. There is some information on the msdn site as well. The best way to learn about it, of course, is to just try it out yourself.

(Originally written and concept by Quan Nguyen; modified and edited by Henry Yi)

Labels: ,

This page is powered by Blogger. Isn't yours?