Posts filed under: Coding
Small pieces loosely joined - to a ski hill
I’ve always been a fan of the “small pieces loosely joined” approach to building simple web apps. A little bit of something from here, hook it up to there, and voilà! Sometimes something useful can spring into existence.
With the coming ski season (which I’m really excited about) I was spending some time on the Whistler Blackcomb website checking out the conditions, and I happened to notice a few public data feeds. One of these feeds contains the status of all the lifts on Whistler and Blackcomb, so of course, I had to hook it up to Twitter. The WhistlerBot updates its twitter stream every time a lifts status changes, and from my visit to the ski hill last weekend, it seems to do so in pretty close to real time.
When I was on the hill, I turned on mobile notifications for just this account (so I don’t get distracted with other Twitter noise) and now my pocket will vibrate whenever the Peak chair changes from standby to open, for example. No one likes to ski in a tracked out bowl, right?…
While I was at it, I also threw up the slightly hilarious, but still useful isthepeakopen.com and gave the WhistlerBot its own home.
Hope other folks find these useful too.
Is Magento right for your next ecommerce project
I recently launched a new ecommerce site using the open source package Magento. This is the first site I’ve worked on that uses Magento, so I thought I would jot down some of my early impressions.
- Tried the online demo, and loved the user experience and admin dashboard. Very polished UI for an Open Source project. (+1 for Magento)
- Huge download (-1 for Magento)
- Convoluted download process and scant details for the SVN checkout (-1 for Magento)
- The Zend framework. Magento is built on the Zend Framework, and in my opinion, the Zend Framework is like a hole in the head. Everyone has their favorite flavour of ice cream, so this is really just my personal preference, but coming from something like Rails to the Zend Framework is like running full speed into a brick wall. Does one really need an XML configuration file to point to the location of other XML configuration files? Yeeessh (-2 for Magento)
- Decent web admin tools to add products and create a catalog. Clients could (mostly) grok it. (+1 for Magento)
- Pretty straight forward to theme, although it could be easier. (+1 for Magento)
Totally subjective and largely meaningless score: -1 (Magento’s weaknesses slightly outweigh positives, in my opinion)
So in summary, Magento is a large, (overly?) complex code base with a great admin and frontend user interface. It’s fairly easy to skin, but I found developing custom functionality a drag. Keep in mind, this was the first Magento site I’ve built, and there’s always a learning curve for every project.
My biggest complaint though is something that I have trouble even articulating. It’s a “vibe”, if you will. Having worked on the Drupal project for a number of years, I have seen first hand how an open source project can (and should) be run. The Magento team could learn a lot from looking at the development processes that projects like Drupal and WordPress employ. For example, with Magento extensions, in most cases, it seemed to not be possible to download actual source code, but rather, you had to enter a key and then Magento would download the package for you. In almost all cases, it feels like the actual source code is kept away from end users. Where is the place to file, and track bugs against various extensions? Where can I browse the source code online? How can I contribute bug fixes? To me, an open source project should be more then trial ware, with an option to upgrade to a premium edition. Open source software is largely about community development, and I don’t see an active developer community hacking away on Magento.
Other alternatives: (open source) Ubercart, Spree (Monthly fee) Shopify, FoxyCart
Introducing: SXSW Gig Guide
With around 1600 bands playing at SXSW, planning your time is critical. With the inevitable slipping of gig schedules, knowing who else is playing at a given venue is really important, and so I threw together this tool to help myself plan my evenings entertainment. You might also find it helpful:
I screen scraped the SXSW site, plotted the venue data on a map and then scrapped artist MP3s from the SXSW site, and hooked those into the new Yahoo Media Player. The end result is you can click on a venue marker, and then listen to the bands that are playing there that evening.
It was also interesting in how long each part of this app took to build:
- Using Ruby and the totally awesome hpricot HTML parser to write the screen scraper to grab HTML from sxsw.com and convert data to JSON - 20 minutes
- Writing the JS to show the JSON data on the map - 1 hour
- Fussing with CSS and HTML layout (which is still broken!) - 3+ hours!
Moral of the story - Either learn more CSS, or surround myself with talented developers and designers so I don’t have to. I’m inclined to do the later…
Next up is to incorporate this data into Rilli so it’s possible to build your own calendar and see where your friends are going.
Thanks to Jeremy Keith for inspiration, and Mike Purvis for some Map code.
UPDATE: Now working in IE7. Thanks to Lucas for the bug report.
UPDATE 2: A few folks have asked for the data I’ve scraped from the SXSW site. The (huge) JSON file can be found here. Go nuts…
Introducing prankmail.org (and the Drupal way VS roll’yer own)
A few rainy weekends ago, I had the urge to roll up my sleeves and build something. Every web application I’ve built in the last few years has been built on Drupal, and I wanted to see (remind myself) what it would be like to develop an application from scratch (where “scratch” = a random collection of open source components combined with bits from Drupal). I also had given myself the timeframe of a weekend, so building “Basecamp right” was out. I wanted a simple coding project.
Since I’m extremely childish, immature, and a fan of the practical joke, I decided to build a web app that let you send an email to someone, but make it appear as if it comes from someone else. I decided to call it Prankmail. Frivolous, slightly dangerous, and perfect for a rainy weekend indoors.
Skipping to the good part, you can check it out at:
http://prankmail.org
And if you would like to read about how I built it, and how the
“build from scratch” process compares to building a Drupal site, continue on:
Before I really started with application specific code, I wanted to grab a few Open Source pieces and get a simple and efficient framework to build on (We’re not using Drupal, after all).
I wanted to have clean URL’s (none of that goofy query string nonsense) so I took a look at how Drupal uses MOD_REWRITE and .htaccess files and cobbled together a simple solution. I’ve also been looking at the new Zend framework, and I like the way it maps URL’s to controllers. I am also a fan of Drupal menu system, so I hacked up the arg() function from Drupal, and made it route URL’s to controllers. I like Ruby’s “convention over configuration” and so by default, I set it up so a URL of ‘/message/123′ gets passed to a function ‘controller_message’ with an arg of ‘123′, if that function exists.
I really like Drupal’s very light weight DB abstraction layer, and so I grabbed that as well (mostly because the syntax is so natural to me at this point), and stuck it in my web app.
For templating, I decided to give Smarty one last try. Every time I struggled with the syntax, or found myself fighting along, I could hear Rasmus saying “PHP *is* a templating language, stupid!” Next time, I think I will just stick with a ‘pure’ PHP templating solution.
Setting up this simple framework let me write the rest of the application with speed and ease. Included libraries aside, the actual application itself is quite small.
For help on the UI, I used JQuery and specifically thickbox. I also found the visual JQuery site to be a great resource when I got tripped up on syntax. JQuery is truly fun to code with, and it’s a great feeling to have so much control over the DOM, with such simple and beautiful syntax.
I mean, how is the following snippet not poetry in it’s simplicity:
//add a token to the form that we will check to
//make sure the form processor only looks at forms
//submitted from this page...
$.get("/token",function(txt){
$("#mailform").append('
‘);
});
The above snippet leads into how I was concerned with spammers and bots submitting mail, so I borrowed a technique from Jack Born and used JQuery to append a token (MD5′d salt + timestamp) to the submission forms DOM, and at the same time inserting the same token into the Session. When the form is submitted, I check to make sure that the timestamp is “fresh” (within 20 minutes), and that the two tokens match. This should prevent, or at least make it more difficult for a bot to make a post directly to my form processor (and I don’t have to use a silly CAPTCHA).
So how did the whole process work, compared to using my usual framework of choice, Drupal?
Setting up the whole framework from scratch was definitely the fun part. I really enjoyed having a chance to “do things my way”, rather then hunt through documentation or lines of code in search of an answer. I think many people code because it feels good to build something, and while Drupal gives you many things “for free”, it was very enjoyable and satisfying to dive in and do it myself.
When it got to the point of adding comments and RSS feeds to the site, I have to admit that it was proving to get a bit tedious, and it would have been nice just to “turn comments on”, in Drupal. But all that being said, the whole exercise reminds me that there are many ways to get a job done. Drupal is a fantastic tool for a specific set of problems, but it’s just that.
Moral of the story is?
There are no silver bullets, and building silly websites is easy and fun.
Now go send some mail!
Back from Brussels
So I’m back in San Francisco after returning from Brussels and one insane week packed with conferences. (EuroOSCON, GovCamp, DrupalCon, and BarCampBrussels). I’m exhausted, but buzzing with excitement. There are just too many people doing interesting things in this world.
Next up: San Fran Drupaler’s should unite and throw something serious down at the upcoming Yahoo! Hack Day. I’d like to go, so if there are other Drupal heads in attendance, I’d love to work on any kind of hack involving Drupal, Yahoo! maps, upcoming.org concert data, and last.fm. Send me a shout if you’re interested!
Speeding up XML-RPC calls in Drupal
Through Bryght, I’ve been working on an interesting project that really exercises the “toolkit” and “web services platform” like nature of Drupal. The final architecture of our project ended up having many distributed processes, all sending data to Drupal through XML-RPC, and we were using Drupal mainly to aggregate and display the results.
Performance was very important, and we were having a huge bottle neck with Drupal’s XML-RPC library. XML-RPC calls were talking a long fricken time, and we were making tons of them.
Fortunately I had help from Walkah and Moshe, and I thought I would pass on what we did to drastically increase XML-RPC performance. This worked for us, so maybe it might help someone else out.
The problem is of course that Drupal does a full bootstrap for every XML-RPC call.
Our solution:
The first, simple (and huge) win was to utilize a PHP opcode cacher - this drastically reduces this bootstrapping time. We used eAccelerator, but there are many others.
Secondly, since we were working in a controlled environment (we knew what modules were going to be called via XML-RPC) we were able to hack the xmlrpc.php file to avoid a full bootstrap. The downside is we had to hardcode a few path things into the file, but again, this is for a particular app/site, and the performance gains were worth it.
Courtesy of Moshe, here is our revised xmlrpc.php file (you might want to rename it to xmlrpcs.php or something):
< ?php
// $Id$
/**
* @file
* Optimized php page for handling xml-rpc requests. Your module must use declare the hook_init() or hook_exit()
* so that it is loaded during the boostrap. Otherise, use hook_xmlrpc as usual. Note that only required modules are loaded at this point.
*
* @author
* Moshe Weitzman
*/
// CONFIGURE
$path = ‘./sites/mwpb-9.local/modules/’;
$module = ‘my-xmlrpc-module’; //the name of the module where your xml-rpc calls are
// END CONFIGURE
include_once ‘./includes/bootstrap.inc’;
drupal_bootstrap(DRUPAL_BOOTSTRAP_DATABASE);
include_once ‘./includes/xmlrpc.inc’;
include_once ‘./includes/xmlrpcs.inc’;
require_once $path. $module. ‘.module’;
$function = $module. ‘_xmlrpc’;
$callbacks = $function();
// print_r($callbacks);
xmlrpc_server($callbacks);
// put any common.inc or other core functions here that your module relies upon.
function t($string) {
return $string;
}
?>
This was enough to get a huge increase in performance. (Yeah!)
Further thoughts:
One could use “mysql_query()” and avoid bootstrap all together, although I don’t think this is very much overhead, and the benefits of using Drupal’s “Database abstraction layer” probably out way any performance overhead here.
I wonder if there is any way to get Drupal to intelligently load necessary code instead of doing a full bootstrap. The “hardcoding” method above certainly works, but I am craving a more elegant and universal solution. Any thoughts?
Playlist module released
Over the last little while I’ve been working on a playlist module for Drupal. Farsheed was also working on a playlist module. So we have combined forces and now present to you the fruit of our labours: The new, improved, ass kicking playlist module.
Features include:
- support for mulitple playlist formats, including xspf, m3u, pls
- podcastable playlists
- AJAX ordering of audio tracks
- multiple ways to add an audio track to a playlist
- For the geeks in the crowd, playlists are implemented as a new node type.
Farsheed has set up a nice demo site where you can go and kick tires a bit, but you have to sign up first to create your own playlists. Enjoy!
Micro Formats and Audio = Peanut Butter and Chocolate
Boris has an interesting post summarizing a discussion we had on micro formats and audio files. The problem is that audio found on the internet has its meta information (title, artists, album, etc..) obfuscated by the binary nature of the file. Jonathan left a comment on Boris’ post saying that it’s pretty straight forward to grab this information from the file itself. But how much easier (not to mention computationally less expensive) would it be for crawlers and bots to find audio if meta information is embedded right in the glorious page itself; no need to parse another large audio file again! I think a micro format that is a one-to-one representation of ID3 meta information will be quite useful.
I think it’s also important to point out that I’m not saying this is the only way this information should be stored and retrieved. The audio module (shameless plug) I have just released for Drupal stores all of it’s meta information in a database and will soon be query-able via a web service, but I also think it would be great if this rich meta information was available right in the XHTML.
There is some good preliminary information on the micro format wiki, and in related news Lucas Gonze has also recently been talking about a playlist micro format based on XSPF.
I think audio meta information is one of the most compelling use cases for micro formats yet, and I look forward to more discussion on this!
AJAX side effects
I’ve been busy adding some AJAX code to a new album/playlist module for Drupal. Since this is the first time I have really worked with the XMLHttpRequest stuff, I was surprised at how adding these calls actually made my code a bit cleaner. I felt like I was really able to base my code on the actions my module was going to perform, and as a bonus, I basically have a RESTful interface to my module as well! I don’t know why I never really thought about it, but someone would have to work actively to not let AJAX-ing some code open up a web service. You’ve got to like it when technology that is chosen (primarily) for user interface reasons, comes bundled with a few nice architectural bonuses.
It seems like as AJAX adoption continues, we should see a corresponding increase in RESTful web services being offered.