Twitter was mentioned during Macadamian's corporate blogging presentation. "Useless! Bah! Humbug!" were the first thoughts to enter my head. Then the speaker pointed out that corporate bloggers should strive to see the positive side of any situation, and I figured making Twitter useful to a neo-Luddite was a good challenge; there must be an itch to scratch here, right?

All I knew about Twitter was that people think it's slow, and it's written in Ruby (on Rails.) After exploring a little more, I found out that it can provide private Atom feeds of your messages, and exposes a REST API (ooh, new, shiny!) That was enough to get me coding.

To keep things simple I decided to be boring and test out Twitter using Ruby (normally, I plan epic posts with 5 different languages, communicating through at least 2 protocols... and a database for no good reason.) The process I followed was:

• Install Ruby (one-click installer)
• In the Gems package manager: gem install rest-client
• Install the Ruby Development Tools for Eclipse
• Read the Twitter API
  

require 'date'
require 'rest_client'
require 'rexml/document'

username = "your_twitter_id_here"
password = "your_twitter_password_here"

twitter = RestClient::Resource.new 'http://%s:%s@twitter.com' % [ username, password ]

# Send a message to twitter
result = twitter['statuses/update.xml'].post(
   :status => "This is your computer.  You have no life as of %s" % DateTime.now()
)
# print result

timelineXML = twitter['statuses/user_timeline.xml'].get({ :count => 5 });

# Make the XML usable
# (Use XmlSimple for even easier handling)
timeline = REXML::Document.new(timelineXML)

# Display the text element of every status returned by the service
# (print timelineXML to learn more about the structure of the result.)
puts "%s's timeline:" % username
timeline.elements.each('statuses/status/text') { |message|
   puts message.text
}

As Twitter restricts you to 70 calls per hour, it's a good job Ruby (on REST) made this so easy. REST is a nice change from the usual URL gobbledygook that frameworks throw up - I can't wait to try it out in Struts2.

So, what can people with no interest in socializing do with Twitter?

• Free yourself from your desk by monitoring that 45 minute long J2EE build from your iPhone
• Monitor your computers by receiving disk capacity warnings, CPU performance reports, and IP address changes
• Find out when someone is breaking into your home
• Know when your plants need to be watered
• Search for signs of life in the Mars lander

One final note, until the authentication framework is more robust, don’t rely on anything you twit staying private.

Labels: rest, ruby, twitter

"Look Ma! I'm inventing my own buzzwords!"

The problem

A little while ago, some of us at Macadamian held discussions on behaviors we would like to encourage among younger developers. One thing we agreed was that "stubbing" (coming up with methods containing as little code as possible, in order to stabilize the API interactions faster) was becoming a "lost art". And so we resolved to start encouraging "top-down development", as we were used to call it.

So far, we've met with little success. Despite our explicit encouragements, it seems hard for people to adopt a "top-down" approach.

And lately, I've been wondering: what if we're not communicating right? After all, "top-down" is a pretty generic term. After all, it's even used to describe an entire programming methodology which predates Object-Oriented Design. So perhaps some of the people in our teams are thinking: "Of course I'm going from top to bottom, what are they complaining about!?"

That's when I thought of something...

Mandatory flashback to the author's younger days

Back at school, I followed this neat AI course which taught me some general problem-solving heuristics. Overall, it showed two ways to attack a problem by brute force: breadth-first and depth-first.

Let's say we're looking for a file within a directory structure. A depth-first approach will recursively explore each solution by first going to the lowest node possible. Only then will it work its way back up, before going down again.



A breadth-first approach, on the other hand, would attempt to fully explore one level of nodes before jumping to the next one.



For additional details, you check out breadth-first search and depth-first search on Wikipedia (where I borrowed these graphs from).

And so...

I'm beginning to think that younger developers show a natural tendency to code depth-first. They pick a single feature/functionality/API call, then implement it down to the lowest level, at which point they consider to have a "valid iteration". Then they go back up a few levels, and start again. Doing things that way can make you feel good because you've added a lot of code, but it makes things harder to test, and might complicate future integration.

We'd like people to try out breadth-first coding, which would mean getting a wider range of partly-implemented methods in the early stages.



First, you'd define the APIs of the first layer, providing a "bare-bone" implementation of each method. Then you'd define the second layer, once again with a minimal implementation, at which point you'd be able to properly implement the first layer. Repeat for each layer of implementation.

So, as my first step, I'll start using the term "breadth-first" when talking about this approach. I'm hoping that by using this term instead of "top-down", I'll get a few "Huh? WTF?" responses, which may be just what we need in order to break some old habits...

As we have found out in part 1, SharePoint can manipulate data using the user profile management objects and can accept data from external sources. Using these two features apart we have two limited tools with limited usage, but using them together gives us a powerful, flexible and efficient method of storing and using data from any internal or external resource.

It’s time to see this solution to our problems in action.

Code example of getting/setting the profile property from a C# application

The concept is the same for any source of information (web services, web applications, etc.):
Get the user’s profile->Get the required profile property->Get/Set profile property value

Code example: (note that to access profile properties, this code must run with elevated privileges)

string value;
try
{
 SPSecurity.RunWithElevatedPrivileges(delegate()
 {
  
     // Change the site address for different deployment environments
     // Note: WIN2003STD is a place holder for your environment, usually your server name
     SPSite site = new SPSite("http://WIN2003STD/");
     SPWeb web = site.OpenWeb();

     // Get the profile manager object for the site
     UserProfileManager profileManager = new UserProfileManager(ServerContext.GetContext(site));

     // Use the username from the User Information Item to get the full profile of an user
     UserProfile user_profile = profileManager.GetUserProfile(System.Web.HttpContext.Current.User.Identity.Name);

     // Get the required profile property value from the profile
     value = user_profile["MyProperty"].Value.ToString();

  
 });
}
catch (Exception e)
{
 return "Error getting user info: " + e.Message.ToString();
}

user_profile["MyProperty"].Value gets or sets the profile property value.

This is a very convenient way of using profile properties to store required information for each user. Properties can be set as read-only or not to appear in the user’s profile, which gives you even more control.

System.Web.HttpContext.Current.User.Identity.Name returns the username (the login username) and it is used in the code example above to get the profile of the currently logged-in user.

Tools You Never Knew You Had… (and what to do with them now that you’ve wised up!)

With this simple solution to getting/setting a user’s profile property value, endless opportunities are now at hand. This small code snippet helps developers control, validate and use values stored for each user, without corrupting the database or resorting to other more complex and error-prone solutions.
For those not needing to use an external application to get information from the users, a custom SharePoint web application can be developed and deployed on SharePoint. This way all the controls offered by ASP .NET or custom controls can be used to perform required operations on the data before it’s stored.

External applications running on the server can access this information the same way, so a bridge between SharePoint and applications like web sites, game servers, messaging apps, etc. can be easily created.

SharePoint Complications, As Usual

Watch out that the profile property might be set as read-only in SharePoint. Even if it’s the case however, the above code should still be able to access the profile because it’s running with elevated privileges. So if there is a situation in which the user is allowed to see the data but only modify it by using a service (like a web application or web service), this is a good way of doing it.

Another gotcha is that the code above can only be used on the SharePoint server machine. This is due to the framework that SharePoint uses. To get information or to change data from a network on internet location, a SharePoint custom web service or web application can come in handy. Other ways of passing information will work too, like server-client applications, as long as the part running the above code is on the SharePoint server machine.

So we now have a way to store data and manipulate it according to our needs. We can control it, we can validate it and most important of all, we decide how the user interacts with the data.

We’re now ready to start doing some serious SharePoint development!

Labels: c#, code quality, SharePoint, string conversion, web development, Win32

There is a constant need in SharePoint feature development to bridge SharePoint 2007 with other web applications. In order to create such a bridge, a variety of user information must be stored in the SharePoint database.

Warning: Do not tamper with the SharePoint database! No matter how tempting it is!

Although this might sound as simple as a small SQL application, tampering with SharePoint’s database could lead to disastrous results and even complete and irreversible server crashes, since many of SharePoint’s recovery features rely on the database being intact. “Feeding” information to SharePoint through its database is unadvisable and sometimes very hard to do, mainly due to the complex structure of the database.

Fortunately there is a built-in solution to this problem that is both simple and efficient: User Profiles and User Profile Properties.

In this post we will see how SharePoint’s user profiles and profile properties can help us do our job fast and easy without worrying about crashing the server or breaking more features than we create.

Let’s start with the storing information problem.
Many developers have already considered using Profile Properties as way to store the required data, but have given up on using them due to their limitations (limited validation, control, variety). Controlling the user’s input and making sure that the values are correct is essential in any bridge, and often auto-generated values based on that data are necessary. SharePoint can only do so much, and this is usually the main reason why developers avoid using the custom profile properties.

Solution: Manage information using the Microsoft.Office.Server.UserProfiles namespaces.

SharePoint Server 2007 stores user profiles in SQL Server, but the information can be imported from other data sources, such as:
· Active Directory
· Lightweight Directory Access Protocol directories (which is not Active Directory)
· Databases
· Enterprise applications (such as SAP or PeopleSoft) by defining a Business Data Catalog connection
· Web applications
· Custom SharePoint web services
· Standard .NET applications, etc.
The main classes are found in the Microsoft.Office.Server.UserProfiles namespaces. The assembly is Microsoft.Office.Server.dll found in C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\ISAPI.
The main objects to handle information within the User Profile store are:
UserProfile
Allows you to access profile properties, the My Site, personalization links, and so on.

UserProfileManager
Gives access to a collection of UserProfile objects and allows you to create, edit, and retrieve user profile objects and properties from the user profile store.

UserProfileConfigManager
Manages the user profile configuration.

Note: In order to manage data in a custom user profile property, the property must be first created. You can use the following steps to create a profile property:

1. Start > Administrative Tools > SharePoint 3.0 Central Administrator
2. In the left panel, under Shared Services Administration, click SharedServices1
3. Click User Profiles and Properties
4. Under User Profile Properties, click Add Profile Property
5. Fill in the required data and click OK

Now that we know that SharePoint accepts data from external sources and that it stores individual user values in user profile properties, it’s time to put the two together and find out how to store user data in profile properties from external data sources. This is the topic for part two of this post: Getting/Storing user data from external sources.

Labels: .NET, Authentication, SharePoint, testing, web development

ERROR: Only Content controls are allowed directly in a content page that contains Content controls.

By default, SharePoint doesn't seem willing to share some of its internal errors with developers opting instead to use a generic "Unknown Error" page. The informative (and I use the term loosely) error string above wasn't displayed in the browser until the file C:\Inetpub\wwwroot\wss\VirtualDirectories\80\web.config was modified to disable custom error messages.

By changing <SafeMode CallStack="false" ...> to <SafeMode CallStack="true" ...> and changing <customErrors mode="On" /> to <customErrors mode="Off" />, the generic error page is replaced with a more detailed error message and maybe even a callstack.

As for the error above? It was caused by a comment that had been added to an .aspx file. For some reasons, comments were not allowed because they didn't qualify as "content controls".

When I work on a web application, I usually have the luxury of running everything that I need locally on my developer machine. My browser, Web-server and database all working together without ever having to put a single packet on a real network.

With today's web applications growing more complex and making use of asynchronous calls to the web server, this idealistic development environment makes it very difficult for the developer to assess the performance characteristics of the application. This can lead to surprises when the application is deployed to a real environment where users actually access it from remote locations.

A good way to bring reality back to this equation is to make use of a proxy server. I like to use Don's Proxy. It is simple Java-based application that takes 1 minute to setup and lets you inject latency, errors and throttle your connection bandwidth. All of this is made in a manner that is independent of your browser or your web server. Most importantly. it does not require any changes to your application's code or setup.

To use Don's Proxy, simply download the package, unzip and double-click the jar file. A simple GUI will open (screenshot) that will prompt you for a port for the proxy to listen to and a host/port for the destination of your test web server. In the example screenshot, the test web server is running locally at port 8080 and the proxy is setup to listen on port 9090.

Once the proxy is started, you just redirect your browser to the proxy's port and everything should work as before. The difference is that now, you can inject realistic network parameters like latency and bandwidth limitations. Don's Proxy also allows you to capture traffic as it goes through and perform other diagnostics without the use of a packet-sniffer.

Hopefully, a more common use of tools like this will help curb developer enthusiasm for flashy Ajax behavior. Your users will thank you.

Labels: testing, web development

For a few years now, my main task at work has been working on the firmware of an IP phone. The phone runs VxWorks on a MIPS32 CPU; the firmware is written in C and C++.

For slightly less time, I've been dabbling in python on my own time. Freedom from explicit typing was a refreshing change, and python's tendency to Just Work was a nice bonus.

It was perhaps inevitable that I would one day try to combine phone and language. (why? because they were both there)

It wasn't obvious that the idea stood a chance. VxWorks is a bit off the OS beaten track, and might not provide all the functionality needed by Python's "core" (not with the same names, anyway). There might be some processor-specific pieces that would rule out MIPS32. And even if I could get something built, would it fit in the 2 or 3 MB of RAM (and even less flash) I could spare?

As it turns out, there was very little to worry about. Python's code is impressively (if perhaps unsurprisingly) portable, only needing a couple of tweaks to its build system and none at all to its source code. There doesn't seem to be anything CPU-dependant; and in the end, adding python to my firmware only cost me 1MB. It took me only a few evenings of tinkering to get a libpython built, linked into my firmware, and loaded on my phone, to the point that I could run this little experiment at the VxWorks shell:

  -> Py_Initialize()
  value = 42 = 0x2a = '*'
  -> PyRun_SimpleString("print 'Hello, World!'\n")
  Hello, World!
  value = 0 = 0x0

(the VxWorks shell being a peculiar animal that allows calling C functions by name, in this case giving me access to Python's C Extension API for a near-REPL experience)

For my purposes, that's enough; I know it can work, and that's all I wanted. I don't expect to ever go further than this. But as little as it is, publishing how I got there might help someone get started on a real project; so here goes:

Porting python in 10 easy* steps

*for a suitable definition of easy

1. As far as embedding Python in an existing application (or firmware) is concerned, Python's own documentation should give you most of what you need
   
2. You'll need a cross-compiling toolchain, i.e. a compiler that can be used on one platform (e.g. x86) and produces executables for use on a different platform (e.g. MIPS32). GCC is your best bet; it's what will make python's build system happiest, and there's lots of resources on getting a GCC cross-compiler working on the web, though it looks a bit daunting to me. I was fortunate in that, since I was already set up to build firmwares, I already had all the needed tools; I would guess that most people engaging on a similar project would be in the same position.
   
   
3. In addition to the compiler (and assembler, linker, etc), you'll want to have a Unix-like environment to run Python's configure script and makefiles. If you're on Windows, cygwin will serve nicely.
   
   
4. The 'configure' script needs some tweaking: it contains a few uses of AC_TRY_RUN, which will fail when cross-compiling.
   
   • If you have a working 'autoconf', the simplest is to edit the 'configure.in' file. You can either remove the AC_TRY_RUN tests altogether or replace them by the newer, more cross-compiler-friendly AC_RUN_IFELSE. Then run 'autoconf' to regenerate the 'configure' script.
     
   • If you don't (as I didn't), you can brace yourself and go edit the 'configure' script directly. Running the script produces error messages that gives something to search for. The fix is actually simple: just remove the calls to 'exit' to allow the error to get ignored.
     
   
   
   
5. The makefile also needs tweaking: just like 'configure', at some point it tries to compile and run a program. This appears to be in order to autogenerate some source files, which fortunately are already provided in the source distribution; so it's safe to disable this step. The simplest way:
   
   • open "Makefile.pre.in"
     
   • find the place where "$(PGEN)" shows up AFTER a ':'
     
   • remove "$(PGEN)"
     
   
   (this will only prevent the executable from getting built. The makefile will still attempt to run it, but it's written so that the resulting failure is ignored)
   
   
6. The configure script and makefile try to guess at the name of tools to use; you can give them a hint with environment variables. In my case I needed to set CC (the C Compiler) and AR (the "archiver", ie. what creates static libraries)
   
   
7. If you need to specify special command-line options to the compiler, environment variables can also be used. Annoyingly, 'configure' and the makefiles use different variable; you'll want to set CFLAGS and BASECFLAGS to the same thing.
   
   
8. Finally you'll be ready to run the 'configure' script. You need to give it the special options --build and --host to tell it you're cross-compiling, something like:
   

   
   $ ./configure --build=win32 --host=vxworks
   

   
   
   (win32 and vxworks were a wild guess that happened to work for me. I got the impression the specific values didn't particularly matter)
   
   
9. You can then run 'make' to compile everything. If, like me, all you need is a static library, this will do it:
   

   
   $ make libpython2.5.a
   

   
   
   
10. There's a good chance some files under Modules/ will fail to compile (in my case, posixmodule.c). The file Module/Setup specifies (in a rather well-documented way) which Python modules (written in C) should be built into the python library; comment out the failing one, and re-run 'make'. I only had to disable posixmodule and pwdmodule; YMMV.
    

And for me, that was it; nothing else needed manual intervention. If you run into more troubles (e.g. trying to build the actual python.exe), I'm afraid you're on your own.

My next step was to figure out how to integrate the python library into my firmware; you'll have to figure out the corresponding steps for your own firmware/embedded application/whatever. Start with the 'embedding' link for how to access python code from your code.

If you want to be able to load python source files with 'import', pay particular attention to what that page says about PYTHONHOME; as for me, I put a putenv(PYHONHOME=/whatever") before the Py_Initialize call, letting me import /whatever/python2.5/*.py files (and possibly, though I haven't tried, .py files contained in a /whatever/python2.5/libpython2.5.zip)

Happy cross-compiling!

Labels: c++, cross-compiling, embedded, python