Abstract—When developing the user interface code of a mobile application, in practice a big gap exists between the digital conceptual drawings of graphic artists and working user interface code. Currently, programmers bridge this gap manually, by reimplementing the conceptual drawings in code, which is cumbersome and expensive. To bridge this gap, we introduce the first technique to automatically Reverse Engineer Mobile Application User Interfaces (REMAUI). On a given input bitmap REMAUI identifies user interface elements such as images, texts, containers, and lists, via computer vision and optical character recognition (OCR) techniques. In our experiments on 488 screenshots of over 100 popular third-party Android and iOS applications, REMAUI-generated user interfaces were similar to the originals, both pixel-by-pixel and in terms of their runtime user interface hierarchies. REMAUI’s average overall runtime on a standard desktop computer was 9 seconds.
Download: PDF, ASE Presentation, ACM SIGSOFT Distinguished Paper Award
Example REMAUI use: The UI designer provides a conceptual UI drawing (left). REMAUI identifies UI elements such as lists of text and images and arranges them in a suitable UI~hierarchy. REMAUI then exports the inferred UI as source code and resource files, compiles them, and runs them on a phone.
Overview of REMAUI processing steps: (1) Locate and extract candidate words and lines with OCR; (2) locate and extract candidate UI elements as a hierarchy of nested bounding boxes using computer vision; (3) merge the results to improve recognition quality; (4) identify repeated items and summarize them as collections; (5) export the constructed UI as a mobile application for a given platform; (6) compile and execute.
Runtime of REMAUI's seven main processing steps on the 488 subjects, shown by group, from left (A) to right (E).
|Normalized pixel-by-pixel screenshot similarity between REMAUI input and generated application on the 488 subjects, shown by group A-E from left to right. Higher values are better.||Image, text, and overall UI element precision (p) and recall (r) for groups B (left) and C (right). Higher values are better.|
Running REMAUI on a screenshot of the Google Chrome app
Running REMAUI on a screenshot of Apple Settings app
Running REMAUI on a screenshot of Yahoo Mail app
Running REMAUI on a screenshot of Facebook app's Settings
This material is based upon work supported by the National Science Foundation under Grants No. 1117369. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.