(Ken Struys' Blog)

web-developer, serious schemer

Mobile Web Development

Recently I've been doing a lot of Mobile Web Development, and I decided I would write down a couple of the lessons learned buildng mobile web applications.

Layout

Mobile is generally a very different style of layout. You're restricted with the amount of information you want to put on a page and you have to consider things like browser resolution. Mobile web apps are portrait site with little to no fixed width/padded/margined elements. You avoid fixed width to make sure your product works on as many phones as possible. By fixing the width of an element, you create a minimum width of the page. You also want to make sure you fill the browser window and make use of the entire screen.

Most of your layout is going to be done with percentage widths. This isn't that hard, the only place where it becomes a little tricky is when you have a mix of floats and percentage widths. The browsers won'talways calculate the percentage correctly. Generally, if you're good and don't have any unnecessary containers around everything, you shouldn't have a problem.

Decide up front, before even wireframes, what phones/resolutions you're targeting. Does this product have to work on really low width phones like the Blackberry Pearl? Are you just targeting the iPhone? maybe you want to make it look native, checkout jQTouch. How about Androids? Well then you might not want the apps look/feel similar to the iPhone. If you get a requirement like: "As a user, the product must work on every phone in the market, so the app maximizes reach", then remember to consider all resolutions (even low width screens). You are going to be restricted even more on how much you can fit and it might hinder the UX of iPhone.

Here are a few examples of sites that are/aren't targeted for low res phones:

I visited all of the mobile sites and resized the window down to 240px width to find errors

Progressive Enhancements

Some of the errors you see above can be fixed.

  • When I visit digg 4, one of the really iconic features are the huge tabs. The tabs are part of the design of the original site and I understand why they had to be there with fixed width/padding/etc. Thoora has a similar problem when you resize smaller then 240px; the catch phrase hides behind navigation bar. The tabs for digg are actually there, but are now floating below the logo. One thing they could change is simply setting the background colour of the navigation to their brand blue. When it's sitting over the header it would appear normal and when it floats below, the header just looks taller. This slight change is a huge advantage because you can actually see the text for My News/Upcoming.
  • Flickr actually did a really nice job with handling resizing. When on a wide profile screen, the search box floats to the left so you don't have to scroll to get what you want. At the same time the layout shows off a nice picture from their product. If you get a chance try out resizing on Flickr's mobile site. The only restriction on Flickr's site, is they set a min-width on the body to 320px. It's reasonable, but will mess up on <320px wide phones. Again they picked their market and they have a link to an old version of the mobile site made for older phones. One thing they might do is make a list of models that have <320px width (the list shouldn't be growing) and do redirection to the old mobile site.
  • Answers just didn't accommodate for small phones. One thing I really don't like about their mobile site, is it doesn't include original design features of answers.com, instead of including their logo they just typed Answers.com in plain text. Even worse, on all small profile phones, their nav is overlapping the name of their company. Easily fixed, get the branding in place (add the company's green somewhere), and move the navigation below the logo.

Meta/Link Tags

There are tons of smartphone specific meta/link tags make your product cleaner. Here are the ones I commonly throw in:

Phone Number Detection

Phones will try to auto detect phone numbers and make them hyperlinks. If something is a phone number, I'll make it look like a button, so I disable this feature.

<meta name="format-detection" content="telephone=no"/>

Mobile Icons

When a user bookmarks the page on their phone, meaning they have it on their home screen, you want to have a nice icon on iPhones/Andriod to make your webapp more appish.

<link rel="apple-touch-icon" href="/link-to-image.png"/>

<link rel="apple-touch-icon-precomposed" href="/link-to-image.png"/>

<link rel="shortcut icon" href="/link-to-image.png" >

Note: apple-touch-icon-precomposed works for Android.

Disable Scrolling/Zoom

If you build your app correctly the user won't need to scroll left/right or zoom. Disable these features:

<meta name="viewport" content="width=device-width; initial-scale=1.0; maximum-scale=1.0; user-scalable=0;" />

Status Bar Color

Make the status bar on the iPhone either black or grey (default).

<meta name="apple-mobile-web-app-status-bar-style" content="black"/>

Hide the Address Bar

This one is just for the iPhone and it's actually JavaScript but it will hide the address bar on the iPhone giving you an extra 60px of height. Any scrolling on the iPhone will cause the phone to hide the address bar so scroll a pixel down the page on load.

<script type="text/javascript">

(function() {

    addEventListener('load', function() {

        setTimeout(hideAddressBar, 0);}, false);



    function hideAddressBar() {

        window.scrollTo(0, 1);

    }

})();

</script>

DOCTYPE

There are a couple different DOCTYPEs you can use or surprisingly, you can just not include it. I would go into the explanation of what doctype to use by Andrew Hedges of Digg did a really good job see his point here.

Redirection

Redirection is something you really need. Your users aren't going to know about the mobile site without it. I typically use a simple cookie to handle correct redirection behaviour. The user should be able to switch between the mobile and full website when they wish and if you don't follow this chart for handling redirection then you will confuse them. In the chart we have three states for the cookie, not-set, mobile preferred and full site preferred. In theory you only need two states, but assuming you're using a reverse proxy cache you’ll need to know when to miss the cache and set cookies.

DeviceRequest URLRequest CookieResponse CookieResponse
Mobilesite.comNONEMOBILE_PREFredirect: m.site.com
Mobilesite.comMOBILE_PREFMOBILE_PREFredirect: m.site.com
Mobilesite.comFULLSITE_PREFFULLSITE_PREFsite.com
Mobilem.site.comNONEMOBIILE_PREFm.site.com
Mobilem.site.comMOBILE_PREF MOBILE_PREFm.site.com
Mobilem.site.comFULLSITE_PREFMOBILE_PREFm.site.com
Non-Mobilesite.comNONEFULLSITE_PREFsite.com
Non-Mobilesite.comFULLSITE_PREFFULLSITE_PREFsite.com
Non-Mobilem.site.comFULLSITE_PREFFULLSITE_PREFm.site.com

Gotchas

Couple note's on things that might give you headaches. Some browsers especially older ones will try to "optimize". By optimize I mean remove "unneeded" elements and even stop scripts from running. You don't have firebug on these phones and it's hard to tell when it's happening. If you do have empty elements (I know you shouldn't have these anyways) stop browsers from removing them with a non-breaking space:

<div class="background">&nbsp;</div>

In terms of scripts, jQuery is a script, it's parsed and evaluated. Older phones might choke on jQuery and randomly stop the loading the library. I typically end up writing all my js from scratch if I have the slow phone requirement but there are more modern solutions like using lightweight js libraries.

You're going to want to make sure you design things to be clickable. This often means making block elements with anchors wrapped around them. This is invalid HTML and some browser will drop the anchor element making nothing clickable. My solution is putting onclicks on the block elements, and then inserting anchors as well on inline elements. This way everything stays clickable and will still work with JavaScript disabled (anchor as fallback). You can also set your inline elements to block and avoid the JavaScript completely but it sometimes doesn't layout correctly on Blackberry.

Blackberry <=4.6, JSONP is not supported correctly. It seems to work when you do a JSONP request from the same domain (in-which case you can just use AJAX) but you will have to proxy any cross domain JSONP requests to get any blackberry (pre 2008ish models) to work.

Tools

There are some good tools out there for mobile. I love using the user-agent switch add-on for Firefox. You can use it to set your browser to have an iPhone user-agent and will be redirected to mobile versions of sites. I often forget to disable it and end up browsing sites all day in mobile version just to see how everything is done. Also assuming you have a Mac, the iPhone simulator is excellent for testing mobile web apps. Also firebug's net panel is useful for ensuring cache headers are working correctly and that you’re not downloading any massive assets. Generally each page of your mobile site should be under 40KB (with an empty cache).

You also have some really terrible tools out there. With the exception of the iPhone simulator, I haven't found a simulator I ever want to use again, especially the blackberry simulators, they are slow and absolutley awful. Luckily I'm usually able to borrow a blackberry and android from someone for testing.

Setting up PLT-Racket Hosting

Hosting for PLT-Racket (formally PLT-Scheme) can be very hard to find. Most hosting companies only support languages like PHP/Perl/Ruby and Python. I had a very difficult time finding a hosting that would actually support this site. In the end the solution was a Virtual Private Server (VPS). With a VPS you are given a VM with root access on a server that is shared by other VMs. Usually VPS is a really expensive option for hosting but I found real cheap VPS for $17US/m from Ultra Hosting.

    The VPS included:
  1. Root Access
  2. 256Mb Ram
  3. 5 Gb Storage/200Gb Bandwidth
  4. SSH
  5. Dedicated IP
  6. CentOS/Fedora/Ubuntu

Within 24 hours of signing up I got an email with my machine's IP and root password. From there it just just a matter of installing my plt-racket webserver. The services has been pretty reasonable, I've had my VM randomly die once but they managed to fix the issue immediately.

Building a Scalable Web Architecture

A lot of people talk about building a scalable web architecture but don't actually explain what they mean. "Oh well you know use memcache and stuff", "No, that doesn't explain anything, how do you actually do it?". I'm sick of hearing it, so I'm going to explain how I do it. The best part about it is, other then server cycles/memory all the software I use are free. Here's a diagram of everything in the stack and that I will cover:

Let's start from the first entry point. First we have some kind of Load Balancer. In terms of software there are two great products out there, SQUID and Varnish. SQUID is the one I'm familiar with so I'm going to go over that one, but Varnish is being used by companies like Facebook, Twitter and Flickr it's really awesome and hopefully I will soon get the chance to try it out.

There are two main services SQUID provides. The first service is reverse proxying. SQUID receives a request from the web and send it's own request to one of the application servers (usually following round robin). That satisfies the load balancing of application servers. The other service SQUID provides is caching, when SQUID receives the response from the application server it caches the result by URL. You save a lot with SQUID's cache because application servers running things like PHP can be crazy CPU heavy. Caching generated results for even minutes we can handle a lot more load.

SQUID is policies are controlled by the squid.conf file. I like the configuration file because it allows you to control the caching policies. You can do things like disallow cache access of logged in users (acl aclname req_header Cookie [-i] user_cookie) and we can create rewrite programs to do url rewriting (url_rewrite_program /etc/squid/rewrite.py). I won't go through everything related policy by here's a starting configuration that I think makes sense. When you get the conf from SQUID it comes with an entire manual inline and it's hard to actually see what your configuration is doing either use mine, or remove the manual, it's available online.

Next we have our actual webservers, I usually just go with vanilla apache running whatever MVC framework I fancy. You might also want a webserver that better handles loading of assets (like images), for that I suggest nginx.

The final step that pretty much everyone takes these days is using some form in memory data cache. Usually I use memcached (distributed memcache). memcache isn't magic it's actually the simplest thing in the world: key value storage (max 1 MB values). You only really need two functions it provides, get(key) and set(key, value). The distributed aspect is that each webserver gets it's own memcache. When you add another webserver you add another memcache to the pool and all webservers can see the entire pool of memcaches. Why do the webservers need access to the entire pool? Let's say two requests come for the same page and we have a architecture decribed above. First request comes in and SQUID makes the request to the first webserver. The webserver tries to get data from memcache and misses so it pulls data from the database and stores it in it's memcache. Second request comes in, SQUID makes the request to the second webserver, this webserver checks and it hits because the first webserver stored it in the cache, no database call. If the second webserver wasn't able to see the entire distributed cache, it would have required a call to the database.

That's about all you really need to know to have pretty reasonable performance. Don't get me wrong, companies like Twitter and Facebook do far more then what I've described, but until your product requires fail whales stay lean, you can hire a scalability expert later. To give you idea of what companies actually use, I compiled the following table:

CompanyLoad BalancerWeb ServerApp ServerMemcacheDatabaseOther Cool Stuff
TwitterVarnishMongrelRuby on Rails/ScalaMemcachedCassandraMurder
WikipediaOver 40 SQUIDs and physical load balancersApacheMediaWiki/PHPMemcached (over 80 memcache servers, >3gb storage)MySQL
FlickrServerIron/SQUIDApachePHPMemcachedMySQLGanglia
DiggPhysical Load BalancerApachePHP/PythonMemcachedLazyboy/CassandraRabbitMQ
RedditElastic Load BalancingTornadoPython/DjangoMemcachedb, tons of stuff precomputedPostgreSQL (mostly denormalized)All on EC2
facebookVarnishHipHop embeded , over 10,000 serversPHP/C++ via Hip-HopMemcached over 800 instancesCassandraHipHop, BigPipe, Haystack, Hadoop/Hive
YoutubeNetScaler/VarnishApache, lighttpd for videoPython with psycoMemcached a lot of pre-rendered pagesMySQLCDN for most popular videos per locale

PLT-Scheme Web Development

I really hate having another blog on the web that explains how to make a blog but I do think mine is a little unique. My entire site is written in PLT-Scheme. I'm a really big fan of scheme especially for web development and I thought it would be hypocritical to write about how great scheme is for web development and have it hosted on wordpress.

Unfortunately this was probably the least lean product I've ever had to create. I had to implement and search index, and MVC framework to keep it organized and I really wanted s-expression based css so I implemented a css compiler as well. I also had to deal with a major lack of documentation/examples. The PLT-Scheme website is great but there is a lack of bloggers/external sources talking about scheme web development. Try understand anything non-trivial without using google, it's really hard and we've become way to dependent on it.

So onto the meat, why scheme? There are a lot of reasons but it comes down to two major concepts Pattern Matching and Hygienic Macros. Show me any other languages that provide those two features and I would probably switch. Another big one is s-expression html templating.

S-expression HTML

S-expression syntax is a method of display tree structured data that has been around since the 1950s. XML (and HTML) also attempted to solve this problem. A lot of web programmers claim they follow the "don't repeat yourself" principle but XML is designed in a way that causes programmers to constantly repeat themselves e.g. closing tags. I do understand the argument for readability of closing tags but a lot of people want a more concise syntax. [1, 2, 3]

S-expression are used in scheme for both language syntax (cause programs are tree) and structured data (lists/trees). Here is HTML markup in scheme:

Fairly concise and includes both expressions; anything in a list that starts with a comma ,(get-post-body id) and literals; anything in the list that is not unquoted. Here's the same example in php:

Okay not bad, s-expressions manage to cut some of the fat but we still have a lot of repeating code. For example, our script tags have a repeating pattern. We can solve this problem with a function call (as we can in other languages): <?= script("foo.js", "bar.js") ?> which is fine but I think scheme provides a better solution via Pattern Matching.

Pattern Matching

Pattern matching is a the act of checking for the presence of a pattern. Most languages provide some form of pattern matching (eg. regular expressions). Scheme has pattern matching for not just strings but for tree structured data. The pattern matching is fairly similar to context free grammar. The main function I use is called match-lambda. match-lambda is a function that takes pairs of (pattern,result) and returns a function that takes one parameter. When you call the returned function it tries to match the argument to the one of the patterns. When it finds a match, it returns the corresponding result. Here's a simple example:

Note: The ... notation in the pattern describes match one or more

So how does this apply to html and our script example? We'll we can write a pattern that removes repetition and will expand into valid html.

I really don't want to write: script (blah blah blah (src filename))) (script (blah (src otherfile)) ...

I would prefer (js filename otherfile ...)

So let's code the ideal and then write a program that rewrites it in a syntax that the browser understands. I'm going to use a library created by Gary Baumgartner called rewrite. Rewrite uses Rewrite Systems to abstract away from recursion when rewriting trees. The main function he provides is called rewrite which takes to parameters, a unary function which either "rewrites" it's input or returns it unmodified, and the thing we need rewritten. The rewrite function then iteratively applies the rewrite rules until nothing changes, meaning there is nothing else to rewrite. First we have our rewrite rules:

Then we just simple write html as we want and call rewrite on it

We can take this even further if we want. html is riddled with repetition, view the source of any website and you'll see it. If you use clearing divs wouldn't something like (clear) be nicer? or if one day you decide to change the path all images files in a large repo, wouldn't it be nice if you could just write a 3 line program do it for you?...

Not only did I use match/rewrite on all of my html, but I used it to convert an s-expression syntax css into standard css. It was about 5 lines of code and I could probably implement the features included in SASS within a couple hours.

You can find a lot more examples of Pattern Matching on the PLT-Scheme siteThe final thing worth mentioning is Hygienic Macros, if you though match was kinda cool you're gonna love this.

Hygienic Macros

Hygienic macros are macros that guarantee that their expansion will not collide with any existing symbol definition. Here's a simple C example (un-hygienic macro):

Here's the output:

Expected:a=2, b=1
Actual:a=2, b=1
Expected:a=10, _c=2
Actual:a=2, _c=10

Obviously someone is being a jerk but we can stop jerks with hygiene, here's the same example in scheme:

Here's the output:

Expected:a=2, b=1
Actual:a=2, b=1
Expected:a=10, c=2
Actual:a=10, c=2

Success! It did what we expected because the compiler recognized the collision (It can also be done at runtime for runtime macros). Now how does this apply to web development? One thing I've noticed with the web development community is the openness to work with and integrate with a wide variety of languages. We'll work with anything that will accomplish whatever it might be were working on, whether it's PHP, ruby, python, flash, javascript, C++, SQL, css, etc. we use whatever is the easiest/natural/most suitable. Unfortunately the syntax and semantics of these languages are mostly kept separate. There might be some overlap but when you're working in PHP and you wish you had some feature in python it just isn't going to happen. What if it could? We are allowed to create namespaces/classes/functions, why not allow us to create language constructs. Working in a language that doesn't have the ternary operator (<boo_expr>?<true>:false)? Why not allow the programer to add it? Scheme is the only language that I know that will actually let you and make it reasonably easy to do so.

Let's try an example:

Okay fine, but I've had people claim they can do this in any language, just define functions plus_plus and ternary:

It doesn't work. Think about why. What if I use both of those functions together?

In Scheme:

In PHP:

PHP outputs 1 because it evaluated plus_plus before calling ternary. This might seem incredibly obvious, but a number of people haven't understood it when I attempt to explain.

So where do we go from here? We make languages libraries, allow syntax/semantics to be imported into a project. Have some AI program you wrote in Prolog? Implement the semantics and write a program to convert your prolog program into s-expressions. Nicole Allard and Gary Baumgartner managed to do this exact thing in about three weeks. Throw it on the scheme webserver and boom Prolog to the browser.

That's about it, that's why I actually use scheme it allows me to express more then any other language and if I can't find a feature in the language I add it. Check out the PLT-scheme website and comment if you have any questions.