Turning GUI into HTML source code with Image processing and Deep learning

Abstract

Web development is one of the top emerging fields in the IT industry. It is the process of developing web-based content, hosting them and accessing via internet or intranet based on the requirement. Drafting and prototyping the design mock-up of a webpage is an important step in web development procedure. The User Interface (UI) design of a website plays an essential role in attracting new users and engaging them on the web pages helps businesses in boosting up sales and revenue. A unique design template will help to increase the user interaction among the visitors of the page and also assure value for the time and resources spent on a page. Web designers design the websites either by using an existing template or by creating the design from scratch. Once the UI design of a website has been created, it is given to a developer to implement in code. This process is challenging and time consuming since the design and implementation are carried out by different teams and therefore costly. The overall outcome of the design template of a website almost depends on the design skills of the web designer and there are differences between two source codes even implemented in the same UI by two different developers. What if there is a mechanism or system to identify the HTML components on GUI image and generate source code automatically. If this system or tool can generate the source code that way, developers can be more focused on functionalities instead of wasting time on front-end development, in an efficient manner. Therefore, this research introduces an approach which automatically turns UI design of a website into HTML source code, end to end with image processing and deep learning. This research was conducted in an experimental approach to achieve two separate scenarios which are implemented as image processing module and deep learning module. Image processing module is to extract the HTML components from a UI design of a website, and deep learning module is to train a convolutional neural network (CNN) which generates the source code according to the constructed HTML tag tree hierarchy. For the evaluation process, the tag tree created from image processing was compared with the original HTML tag tree, and original source code implemented for particular GUI was compared with generated code lines. The system was implemented using MATLAB, Image Processing Toolbox, TensorFlow and Python.