Wednesday, July 30, 2008
Snake On A Phone
For a few years now, my main task at work has been working on the firmware of an IP phone. The phone runs VxWorks on a MIPS32 CPU; the firmware is written in C and C++.
For slightly less time, I've been dabbling in python on my own time. Freedom from explicit typing was a refreshing change, and python's tendency to Just Work was a nice bonus.
It was perhaps inevitable that I would one day try to combine phone and language. (why? because they were both there)
It wasn't obvious that the idea stood a chance. VxWorks is a bit off the OS beaten track, and might not provide all the functionality needed by Python's "core" (not with the same names, anyway). There might be some processor-specific pieces that would rule out MIPS32. And even if I could get something built, would it fit in the 2 or 3 MB of RAM (and even less flash) I could spare?
As it turns out, there was very little to worry about. Python's code is impressively (if perhaps unsurprisingly) portable, only needing a couple of tweaks to its build system and none at all to its source code. There doesn't seem to be anything CPU-dependant; and in the end, adding python to my firmware only cost me 1MB. It took me only a few evenings of tinkering to get a libpython built, linked into my firmware, and loaded on my phone, to the point that I could run this little experiment at the VxWorks shell:
-> Py_Initialize()
value = 42 = 0x2a = '*'
-> PyRun_SimpleString("print 'Hello, World!'\n")
Hello, World!
value = 0 = 0x0
(the VxWorks shell being a peculiar animal that allows calling C functions by name, in this case giving me access to Python's C Extension API for a near-REPL experience)
For my purposes, that's enough; I know it can work, and that's all I wanted. I don't expect to ever go further than this. But as little as it is, publishing how I got there might help someone get started on a real project; so here goes:
Porting python in 10 easy* steps
*for a suitable definition of easy
- As far as embedding Python in an existing application (or firmware) is concerned, Python's own documentation should give you most of what you need
- You'll need a cross-compiling toolchain, i.e. a compiler that can be used on one platform (e.g. x86) and produces executables for use on a different platform (e.g. MIPS32). GCC is your best bet; it's what will make python's build system happiest, and there's lots of resources on getting a GCC cross-compiler working on the web, though it looks a bit daunting to me. I was fortunate in that, since I was already set up to build firmwares, I already had all the needed tools; I would guess that most people engaging on a similar project would be in the same position.
- In addition to the compiler (and assembler, linker, etc), you'll want to have a Unix-like environment to run Python's configure script and makefiles. If you're on Windows, cygwin will serve nicely.
- The 'configure' script needs some tweaking: it contains a few uses of AC_TRY_RUN, which will fail when cross-compiling.
- If you have a working 'autoconf', the simplest is to edit the 'configure.in' file. You can either remove the AC_TRY_RUN tests altogether or replace them by the newer, more cross-compiler-friendly AC_RUN_IFELSE. Then run 'autoconf' to regenerate the 'configure' script.
- If you don't (as I didn't), you can brace yourself and go edit the 'configure' script directly. Running the script produces error messages that gives something to search for. The fix is actually simple: just remove the calls to 'exit' to allow the error to get ignored.
- If you have a working 'autoconf', the simplest is to edit the 'configure.in' file. You can either remove the AC_TRY_RUN tests altogether or replace them by the newer, more cross-compiler-friendly AC_RUN_IFELSE. Then run 'autoconf' to regenerate the 'configure' script.
- The makefile also needs tweaking: just like 'configure', at some point it tries to compile and run a program. This appears to be in order to autogenerate some source files, which fortunately are already provided in the source distribution; so it's safe to disable this step. The simplest way:
- open "Makefile.pre.in"
- find the place where "$(PGEN)" shows up AFTER a ':'
- remove "$(PGEN)"
(this will only prevent the executable from getting built. The makefile will still attempt to run it, but it's written so that the resulting failure is ignored) - open "Makefile.pre.in"
- The configure script and makefile try to guess at the name of tools to use; you can give them a hint with environment variables. In my case I needed to set CC (the C Compiler) and AR (the "archiver", ie. what creates static libraries)
- If you need to specify special command-line options to the compiler, environment variables can also be used. Annoyingly, 'configure' and the makefiles use different variable; you'll want to set CFLAGS and BASECFLAGS to the same thing.
- Finally you'll be ready to run the 'configure' script. You need to give it the special options --build and --host to tell it you're cross-compiling, something like:
$ ./configure --build=win32 --host=vxworks
(win32 and vxworks were a wild guess that happened to work for me. I got the impression the specific values didn't particularly matter) - You can then run 'make' to compile everything. If, like me, all you need is a static library, this will do it:
$ make libpython2.5.a - There's a good chance some files under Modules/ will fail to compile (in my case, posixmodule.c). The file Module/Setup specifies (in a rather well-documented way) which Python modules (written in C) should be built into the python library; comment out the failing one, and re-run 'make'. I only had to disable posixmodule and pwdmodule; YMMV.
And for me, that was it; nothing else needed manual intervention. If you run into more troubles (e.g. trying to build the actual python.exe), I'm afraid you're on your own.
My next step was to figure out how to integrate the python library into my firmware; you'll have to figure out the corresponding steps for your own firmware/embedded application/whatever. Start with the 'embedding' link for how to access python code from your code.
If you want to be able to load python source files with 'import', pay particular attention to what that page says about PYTHONHOME; as for me, I put a putenv(PYHONHOME=/whatever") before the Py_Initialize call, letting me import /whatever/python2.5/*.py files (and possibly, though I haven't tried, .py files contained in a /whatever/python2.5/libpython2.5.zip)
Happy cross-compiling!
Labels: c++, cross-compiling, embedded, python
Thursday, July 24, 2008
Outlook Entry IDs Made Easy
- Outlook Object Model (OOM)
- MAPI
- CDO
- Office/Outlook COM Add-in (_IDTExtensibility2)
- Exchange Client Extensions
It's important to understand the relationship between MAPI and the OOM:
- The MAPI API dates back to the early Outlook days. It's used for the messaging and storage subsystems in Outlook.
- The Outlook Object Model sits on top of MAPI and wraps a minimal amount of its functionality. It also exposes some of the Outlook UI to the developer.
At its simplest, an Entry ID is a variable length byte buffer, sometimes represented with a simple byte array. In MAPI, most Entry IDs are represented by the ENTRYID structure:
mapidefs.h
typedef struct {This structure usually comes wrapped in the SBinary structure.
BYTE abFlags[4];
BYTE ab[MAPI_DIM];
} ENTRYID, FAR *LPENTRYID;
mapidefs.h
typedef struct _SBinary {In the Outlook Object Model things take a turn for the worst:
ULONG cb;
LPBYTE lpb;
} SBinary, FAR *LPSBinary;
- Most times you see Entry IDs as a BSTR that is hex encoded (i.e. the strings look like this: "000F1329EC29A0382BC...").
- Sometimes it still uses a BSTR, but it doesn't encode the buffer, instead using the BSTR as a binary blob. Although inconsistent, this is perfectly valid.
- Sometimes it's useful to represent the Entry ID as a SAFEARRAY of VARIANT VT_UI1 (this is the format VB uses for its byte arrays). For example, I've needed to have a function that can return a property any type from a MAPI object that needs to be called through automation (i.e. VB). For example, the IMAPIUtils::HrGetOneProp method from the Redemption Library does this.
I solved the problem by creating a class (yet another format!) to facilitate manipulation of all these formats:
class EntryIdNow whenever I get an Entry ID in any format, the first thing I do is convert it to an EntryId instance. This adds many convenient functionalities:
{
public:
EntryId(EntryId const& entryId);
// All explicit so that it's always clear what we are doing.
explicit EntryId(BSTR const* str, bool hexEncoded = true);
explicit EntryId(SBinary const* binary);
explicit EntryId(SAFEARRAY const* array);
explicit EntryId(unsigned long count, unsigned char const* bytes);
// Again, no implicit conversion
CComBSTR toString(bool hexEncoded = true) const;
CComSafeArraytoSafeArray() const;
std::vectortoByteArray() const;
// ... add more as you need them
// TODO: Add more custom helper functions.
bool isLongTermId() const;
// TODO: Add comparison operators for convenience and ordered
// container support (such as std::map keys). When comparing
// Entry Ids, remember you need to use the
// IMAPISupport::CompareEntryIDs method.
private:
std::vector<unsigned char> m_buffer;
}
- Standardized interface to all ids.
- Easy comparison of entry ids.
- EntryId is a copyable class and can be passed by value.
- EntryId can be held in a STL container.
- Your other helper classes and methods can use the EntryId as parameter instead of other formats.
Wednesday, July 23, 2008
Software Testers - Don't Underestimate Their Worth for Success
Recently, I was managing an offshore team developing a component for our client's core enterprise application. We wrote extensive test cases and, because we were concerned about a lack of domain knowledge, we even had architects write some of the test cases. Things were looking good as we approached the end of the development cycle: the product was behaving as it was supposed to; we were passing test cases; we were greenlighted by the QC ‘process’.
Our confidence was high, and we were ready to move on to automated testing. Then we found the problem. It turns out that our test cases were written with a single user in mind. The minute we started to use this component with concurrent user access, the system would pretty much lock up. Our confidence evaporated.
This was a big mistake. Architects and technical leads both reviewed the test cases. Nobody found the oversight at the time. This was compounded by an ‘if it passes the test cases then it works’ mentality.
Unfortunately, a solution to this problem isn’t as easy as ‘follow these three steps’. But if we can take away a lesson learned from this, it is that testing the right way is not only very important to the success of a product, but it also shouldn’t be taken for granted. If you haven’t noticed, QC/QA and testers were not included in the writing of test cases, thus bypassing all kinds of valuable experience that would most likely caught our newbie mistakes.
Bottom line - Don’t underestimate the worth of your testers and their experience, it might just come back to bite you in the future.
Labels: QA, QC, software testing, testing
Thursday, July 17, 2008
Maintenance Nightmares: Commented out code
While examining the code, I noticed code like this:
// Fix for bug 1021892
// some code that was commented out.
There were instances of this throughout the code.
When I see this I instinctively cringe, I immediately think that there is no proper source control or defect tracking procedures (or tools) in place. While there may be the odd corner case where commented out code is acceptable, in general it is not and should be avoided.
There are many better ways of dealing with this, the first way is to just delete the line of code, and when you commit/check-in the code, make sure you put a meaningful comment in the commit logs that ties it to the defect it was changed for. You can often search the commit logs, and if you put the bug id in the commit log, you will get the change in your search.
Another easy way to do this is to delete the line of code, make a patch, and when resolving the bug in your bug tracking software, attach the patch file to that issue. Then you can always see what you changed to fix the bug.
Does anyone else having any suggestions for this?
Labels: code quality
Friday, July 11, 2008
Alignment Matters
void* p = something();
int i = *(int*)p;
int i;
void* p = something();
memcpy(&i, p, sizeof(int));
Answer: when 'something' returns a value that's not a multiple of "sizeof(int)".
Some processor architectures only allow loading memory into N-byte registers from memory addresses that are a multiple of N (e.g. multiple of 4 for 32-bit registers). Using misaligned addresses can have some various interesting consequences, depending on the CPU; I've heard of at least these:
- It works fine
- It works, but slowly (e.g. because the misalignment is detected and alternate instructions are used to load 1 byte at a time - I've heard that x86 works this way)
- The program may crash due on an "invalid address" trap
- The CPU may load data from the wrong address (e.g. if only multiples of 4 are valid addresses, the instruction might ignore the bottom 2 bits of the address)
Last week, I got lucky: I discovered that the MIPS32 CPU on the phone I was programming falls in the "crash" category. (why lucky? because an "address load exception" message is much easier to debug than some corrupted data)
Usually, we don't have to worry about such alignment issues; the compiler and runtime make sure that all objects it allocates go at addresses that have the right alignment for their type (e.g. malloc must return memory "suitably aligned" for all possible types).
Trouble comes when we lie to the compiler, such as telling it by a cast that "p" points to an "int" when such is not the case. This is what happened to me: I was parsing a file, and the 4-byte-value-that-should-be-put-in-an-int followed an arbitrary-length string. It ended up on an odd address, and boom. (Here's another way I was lucky: it COULD have been a nice multiple of 4 in all my tests, only to come out odd on a client's desk)
Functions like memcpy, of course, are required to work with all addresses (as expressed by taking void* parameters, which require no cast).
Lesson of the day? Don't lie to your compiler!
(alternate lesson: "casts: evil AND chaotic"?)
Labels: c++, compiler, optimization
Thursday, July 10, 2008
Compile Time Semantic Checking
In a project I'm working on we have a lot of database records ids being passed around as integers. Today I needed to add a new record id to this list. Adding the new item to the function signature is easy. The hard part comes in making sure the right value is being assigned.
Public void MyFunction(int TableA_ID, int TableB_ID, String someOtherValue);
int A_ID = 5;
int B_ID = 4;
MyFunction(B_ID, A_ID, "Hello World");
So what happens when I compile? Nothing, it works fine. Yet that line where I called the function has just corrupted the entire database because those database ID's were transposed. Which kinda sucks.
It's one thing to guarantee type safety, what what about semantic safety? An idea popped into my head to use the existing type-checking to accomplish this.
public struct UserID
{
private int value;
public static implicit operator int(UserID source)
{ return source.value; }
public UserID(int intValue)
{
value = intValue;
}
}
private void TryToFail(UserID test) { }
UserId works = new UserId(5);
int fails = works;
TryToFail(works) //compiles fine
TryToFail(fails) //Doesn't compile
Even though both works and fails contain the exact same value, the function can be made to fail if the values are put in the wrong order. I used an implicit operator so that I didn't lose the convince of having a simple integer, neato.
Obviously this technique is not something you would want to use for every possible field as there is one extra step involved in getting the value. But if you only did this for the record IDs I think it would be enough gain to make it worthwhile.
I would like to think of a way to do this so that it has compile time checking, but once compiled only contains the primitive value. Any ideas?
Wednesday, July 09, 2008
Trouble Getting Your Custom Protector Called?
You need to set the IrmAddInsEnabled flag to true in the Central Admin page to get SharePoint to call any of the protector function calls (HrIsProtected, HrProtect, HrUnprotect). This sets a Boolean value that specifies whether to enable Information Rights Management (IRM) addins.
You can do this directly on the MOSS server from the 12 Hive (C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\BIN) by calling the following:
On the MOSS server:
'stsadm -o setproperty -propertyname irmaddinsenabled -propertyvalue "yes"'
Alternatively, you can do this programmatically by using SharePoint via the IRM Settings by using SPIrmSettings.IrmAddins. This can be done at the farm level or as granular as a per-list basis.
IRM Settings for the FARM:
SPWebService svc = SPFarm.Local.Services.GetValue();
SPIrmSettings irmSettings = svc.IrmSettings;
//Specifies whether to enable Information Rights Management (IRM)
//locally for sites
irmSettings.IrmRMSEnabled = true;
//Specifies whether to enable Information Rights Management (IRM)
//addins
irmSettings.IrmAddinsEnabled = true;
//Sets the number of times the Information Rights Management (IRM)
//settings have been changed
irmSettings.IrmChanges += 2;
svc.Update();
IRM Settings for the Individual Lists on FeatureActivated:
SPWeb web = (SPWeb)properties.Feature.Parent;
foreach (SPList list in web.Lists)
{
// Specifies whether to enable Information Rights Management (IRM)
// for the list
list.IrmEnabled = true;
// Specifies whether to allow users to upload documents that do
// not support IRM
list.IrmReject = true;
list.Update();
}
After you have done this, your custom protector should be called when the documents are checked in and out of your SharePoint system via the HrIsProtected, HrProtect, HrUnprotect methods.
Goodluck!
Labels: .NET, IRM, SharePoint
Monday, July 07, 2008
Mining Project Scope with Python + SVN
import pysvn
# Script variables. Could be read from a command line or config. file.
username = "jlennon"
password = "yoko"
projectFirstRevision = 103991
svnRoot = "http://path/to/svn/trunk/"
# Team members’ names for SVN commits.
users = set([
"gharrison",
"rstarkey",
"jlennon",
"pmccartney",
"gmartin",
"pbest"
])
# Set up SVN callbacks.
def get_login(realm, username, may_save):
return True, username, password, False;
def get_log_message() :
return True, "log message called"
client = pysvn.Client()
client.callback_get_login = get_login
client.callback_get_log_message = get_log_message
# Convert the revision number into a pysvn revision.
projectFirstRevision = pysvn.Revision(pysvn.opt_revision_kind.number, projectFirstRevision)
# Get all commits since the project's first commit.
items = client.log(
svnRoot,
pysvn.Revision(pysvn.opt_revision_kind.head),
projectFirstRevision,
discover_changed_paths=True,
)
# Set up variables for storing commit information
allPaths = set([])
removedPaths = set([])
# For each commit entry *made by one of this project's users*
# Add any deleted files in the commit to removedPaths
# Add any added/updated files to the list of all paths
for item in [ e for e in items if e.author in users ]:
for changed_path in item.changed_paths:
if changed_path.action == "D":
removedPaths.add(changed_path.path)
else:
allPaths.add(changed_path.path)
# Remove deleted files from allPaths, then display
allPaths = allPaths - removedPaths
for path in sorted(allPaths):
print path
Overall, I was impressed by Python. My biggest problem was with the structure of the documentation, but that’s to be expected with any new language. My biggest surprise was that the indentation rules didn’t bother me – why is this cited as a show-stopper by so many developers? I’ll definitely be coming back to Python to test its OO facilities.
[1] The main con being that a developer would much rather write a neat SVN spider than write documentation. In a leadership role, you have to think twice before eagerly firing up your compiler to solve a problem.
Wednesday, July 02, 2008
Getting RIL!!
I have this nifty phone but how do I control it programmatically?
Well, first of all, the Windows Mobile Radio Interface Layer works by having specific modules control the cellular component. Since we are in North America there are two different cellular protocol implementations: GSM and CDMA. Above this layer is an abstraction layer that hides the implementation of the underlying protocol so that applications/OEM phone developers can use the functionality of call handling (and other useful information) without getting down to how things work under the hood. So how do we, as an application, talk to this abstraction layer, which we'll just clump together and call it the RIL( Radio Interface Layer )? Good Question!
Since cellular devices are really just modems that use AT commands, the RIL operates mainly from an asynchronous callback system. In fact most of the RIL functions really just map to AT commands. The application accessing the RIL makes a request to the RIL, and the RIL takes some time to process this request and sends back a result code indicating the success or failure of the request. In addition the RIL spits out notifications which our application can hook into and be informed of status changes related to the cellular network. So let's dig a little with some RIL examples.
There are a few pitfalls by going down this path. Firstly you might run into situations where you're fighting for control with the resident dialer (the default phone application supplied to the mobile device; developed by Microsoft). There may also be some undocumented internal states that need to be obeyed or else it can lead to unintended behavior (i.e. device lock up).
There are also some differences to note between CDMA and GSM RIL implementations. With GSM devices there can be multiple lines, and as such the RIL can fetch call information for each line. RIL can also manage the GSM lines to perform actions such as putting the call on hold or switching lines. In the CDMA environment, there is only one "real" line and one "virtual" line. That is, one line is used to represent two lines. Access between the two lines is achieved by sending a "flash" command. Call management is also not possible with CDMA. So when you call "RIL_GetCallList" for GSM, for example, you may be returned information for one or more lines depending on the call state. For CDMA, you will only see at most one line.
You will have to get your hands on the ril.h and ril.lib files. Both of these files are floating around somewhere on the internet (try here). If you can get both of these files, link your application with ril.lib and away you go. If you can’t find ril.lib but have ril.h, you can dynamically link with the ril.dll file (contained in the “windows” folder on the device) since you know what some of the prototypes are.
First we need to initialize our application with the RIL so that we can receive both results and notifications callbacks. Here’s some code to do this:
...
// keep this handle handy as we need it for further RIL commands
HRIL rilHandle;
...
HRESULT hr = RIL_Initialize(
1, // index of the RIL port to use (e.g., 1 for RIL1:)
&resultCallBack, // this is a pointer to your result call back method
¬ifyCallback, // this is a pointer to your notify call back method
RIL_NCLASS_ALL, // all notification (except device specific)
(DWORD) this, // custom param (could be a pointer to an instance of a class)
&rilHandle); // returned handle to RIL instance
...
void CALLBACK resultCallback(
DWORD dwCode,
HRESULT hrCmdID,
const void *lpData,
DWORD cbData,
DWORD dwParam)
{
// handle the results
...
void CALLBACK notifyCallback(
DWORD dwCode,
const void *lpData,
DWORD cbData,
DWORD dwParam)
{
// handle the notification
...
Note that we’ve initialize the RIL with a reference to our class as the user parameter which is passed to the result and notification callbacks. The parameter is optional; however having the class reference would be especially useful if you would like to perform further actions after receiving a result or notification. More on this later.
OK so what - I've just initialized myself with RIL - what can I do now?
To answer that question, anything under the Sun related to call handling, and then some!!
For example if you wish to check the call status of the phone you can do the following. First send a request for call list:
HRESULT cmdID;
...
void getCallList()
{
// call RIL_GetCallList with RIL handle from the initialization
cmdID = RIL_GetCallList(rilHandle);
}
Next we have to wait for the results:
...
void CALLBACK resultCallback(
DWORD dwCode,
HRESULT hrCmdID,
const void *lpData,
DWORD cbData,
DWORD dwParam)
{
If (cmdID == hrCmdID && RIL_RESULT_OK == (dwCode & 0xff))
{
// lpData points to an array ofstructures.
RILCALLINFO *lpCallInfo = (RILCALLINFO *)lpData;
// extract the call info from the structure and do something...
...
Notice how the cmdID return value of RIL_GetCallList matches hrCmdID in the callback. In a realistic scenario, multiple commands are sent and the results do not necessary return in the same order the commands were sent. As a result we would need to queue up the command IDs in order to properly identify the command represented by each callback
Now what is notifiyCallback useful for? This function is called basically whenever an event occurs that is of a notification class that was registered during initialization (we registered with RIL_NCLASS_ALL in this case). For example, for an incoming call you would receive and handle the “ring” notification as follows:
void CALLBACK notifyCallback(
DWORD dwCode,
const void *lpData,
DWORD cbData,
DWORD dwParam)
{
if(dwCode & RIL_NCLASS_CALLCTRL)
{
switch(dwCode & (0xff | RIL_NCLASS_CALLCTRL))
{
case RIL_NOTIFY_RING:
// do something here
...
Since the phone is ringing, it is probably best to get more information about the incoming call. If you remember from initialization we use the class reference as the custom parameter. This gives us the flexibility to send further RIL commands by either passing in the RIL handle (HRIL) or point to a function in the class. In this case we can point back to the getCallList function created earlier.
MyClass* myClass = (MyClass*)dwParam;
myClass->getCallList();
If we don’t have a function to point to, we can always send an RIL command on the fly by passing in the RIL handle from our class. For example to answer the call right away:
MyClass* myClass = (MyClass*)dwParam;
HRESULT hr = RIL_Answer(myClass->rilHandle);
As a note, if we're not using the interface anymore we'll need to de-register ourselves:
RIL_Deinitialize(rilHandle);
To learn more about RIL and all its wonders, you should closely examine the ril.h header. There is some information on the msdn site as well. The best way to learn about it, of course, is to just try it out yourself.
(Originally written and concept by Quan Nguyen; modified and edited by Henry Yi)
Labels: RIL, Windows Mobile