An Experiment in Developing Small Mobile Phone Applications Comparing On-Phone to Off-Phone Development

TouchDevelop represents a radically new mobile application development model, as TouchDevelop enables mobile application development on a mobile device. I.e., with TouchDevelop, the task of programming say a Windows Phone is shifted from the desktop computer to the mobile phone itself. We describe a first experiment on independent, non-expert subjects to compare programmer productivity using TouchDevelop vs. using a more traditional approach to mobile application development.

Download: PDF

Example task 2 solution in Android ("Convert degrees Fahrenheit to degrees Celsius"). Example task 2 solution in TouchDevelop ("Convert degrees Fahrenheit to degrees Celsius").

(1) Corpus of TouchDevelop applications (as of 17 February 2012)

The following zip file contains all TouchDevelop apps that have been published to the TouchDevelop cloud as of 17 February 2012: TouchDevelop_src.zip

(2) Program used to build TouchDevelop application corpus

We developed the following tool that downloads from the TouchDevelop cloud all apps that have ever been submitted: TouchDevelopAllAppsReport.jar

Disclaimer: The design of the TouchDevelop cloud web services is still under heavy development. The APIs may change at any time and and may thereby break.

Our tool also counts LOC as described below. Usage:

> java -jar TouchDevelopAllAppsReport.jar > report.txt

(3) How large are TouchDevelop applications?

To answer that question, we used our TouchDevelopAllAppsReport tool to download from the TouchDevelop cloud all apps that have ever been submitted. From the LOC counting we exclude all prior versions of an app published by the app's original author under the same app name. By doing that we only count the current version of each app. However we do not exclude such apps if the prior and current app are by different authors or have different names.

Size in LOC of the 2,081 TouchDevelop apps published in the TouchDevelop cloud as of 17 February 2012. Not shown are the two largest apps of 1,742 and 1,675 LOC. Each dot represents all apps in one bin of 5 LOC. E.g., the left-most dot represents the 674 apps from 0 to 4 LOC.

As of 17 February 2012, the largest TouchDevelop app is Photo Dog with 1742 LOC.

(4) Counting LOC

To make counting reproducible and comparable between TouchDevelop and Android, we first normalized all TouchDevelop and Android apps and then counted their logical source statements (LSS) http://www.sei.cmu.edu/library/abstracts/reports/92tr020.cfm. We did not count the content of configuration files such as the xml files each Android app uses to define layout, styling, etc.

To count individual TouchDevelop apps of each subject, we first publish them to the TouchDevelop cloud and then use our following TouchDevelopService tool to download them and count their LOC. You can download the TouchDevelopService tool at TouchDevelopService.jar. TouchDevelopService takes as parameter a list of subjects' userid and counts their apps' LOC. For example:

> java -jar TouchDevelopService.jar wivo fvfi mzxo dwns ybqh askf kgbv lavd snlg kzhv

To count individual Android apps, we create a custom Eclipse code formatter that imitates the layout rules we used for counting the TouchDevelop LSS. The formatter configure file can be download at org.eclipse.jdt.core.prefs. In order to apply the formatter configure file on Android_src, first download Eclipse IDE for Java Developers, then run the below command inside the Eclipse application folder. 

> eclipse -application org.eclipse.jdt.core.JavaCodeFormatter -config "[PATH_TO_COFIGURATION_FILE]/org.eclipse.jdt.core.prefs" "PATH_TO_ANDROID_SRC"

- You can also download the formatted source code at Formatted_Android_src.zip

- After source code is formatted, we develop a tool which uses JavaNCSS (http://javancss.codehaus.org) to count the Java LOC in each Android project. You can download the tools at AndroidCodeCounter.zip and run the command below on the folder that contains the extracted Formatted_Android_src.zip.

> java -jar AndroidCodeCounter.jar Formatted_Android_src

(5) Lab experiment comparing on-phone to off-phone development

We defined a set of 11 simple programming tasks. For the experiment, Microsoft Research loaned us 10 mobile phones, which allowed us to have subjects develop TouchDevelop apps on a phone. We randomly assigned 10 subjects to one phone each and the control-group of the remaining 17 subjects to a lab Windows PC each that had installed an Android IDE (Eclipse) and SDK. The Android IDE contains an interactive wizard that generates a working hello-world Android app. The experiment used TouchDevelop v2.4.0.0 beta and Android v1.6.

Instructions for Android subjects:
https://docs.google.com/document/pub?id=1zwrEMO-C5FWOGEmh3B2MWb5iZfo76bvY5vFbA-qJgtE
Instructions for TouchDevelop subjects:
https://docs.google.com/document/pub?id=1kSVh94ZrfL33-5CfiYx8u7-khqYyutlxghdl1mb8BCc
Post-experiment questionnaire:
https://docs.google.com/document/pub?id=1h6QC5Z8a0SAxuVhl5uaNef37GgowebkDpMBzCRKjUEQ

TouchDevelop subjects submitted the following solutions. Each link in the following list links to all solutions submitted by one of our 10 TouchDevelop subjects: wivo, fvfi, mzxo, dwns, ybqh, askf, kgbv, lavd, snlg, kzhv.

Android subjects submitted the following solutions: Android_src.zip

Acknowledgments

This material is based upon work supported by the National Science Foundation under Grants No. 1017305 and 1117369. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Contact

Tuan Nguyen, Christoph Csallner, SERC lab in the Computer Science and Engineering Department at the University of Texas at Arlington