Cost-Effective Infrastructure Solution for Document Scanning in a Web-Browser: R&D and Open Source Customization on a Tight Schedule

Provided services

Custom Software Development, Early-Stage Innovation, R&D Services

Client

Developer of a system for electronic document management and automation of business processes of various organizations.

Product

The product is a Java application for scanning documents with various parameters: selection of color mode, document size, and quality, integrated into the customer’s system. Scanning is performed at the software level, without the use of third-party software supplied by scanner manufacturers. It is enough to have a scanner driver installed on the user’s computer that is compatible with the most common TWAIN protocol, on which the vast majority of the world’s scanner manufacturers operate. The application is installed and launched on the user’s computer and works in the listener mode. As soon as a request for scanning is received from the system, our application starts, performs the necessary operations with the specified parameters, and sends the result directly to the client’s system.

Challenge

At the beginning of the project, our team faced the task of finding a solution to meet the client’s requirements. There were already offers on the market that solved the problem, but these solutions were quite expensive and had advanced functionality that the client did not need. Our team needed to research open-source solutions in this domain area, perform extensive R&D, and create an MVP that can be scaled to meet the specific needs of the client.

It was also important to create an application that could be used on computers with modest specifications.

The project evolved in an environment where there were multiple stakeholders and each had a different idea of the ideal outcome. We had to be a bridge – to listen, negotiate, and find a common language to move forward. At the same time, it was critical to clearly distinguish which part of the work was research and development (R&D) and could be unstable or changeable, and where we could already talk about production-ready solutions. This allowed us to build transparent expectations and avoid misunderstandings between the teams and the business.

The project implementation timeframe was limited to 1 month.

JazzTeam Challenge

While testing the prototype, our team encountered the problem of excessive memory consumption when the application was running. While scanning color documents in multi-page mode, too much memory was consumed – at peak times, the value exceeded 1GB, which contradicted the idea of using the application on computers with modest specifications. The main reason for this behavior was the fact that all scanned data was saved to RAM.
We faced an additional task – to find out the reason for such behavior of the application and find ways to solve the problem as soon as possible.

Solution

Our team immediately got to work. Thanks to our extensive experience in R&D and building working prototypes, we researched open source solutions, selected those that suited us, and analyzed them by their main characteristics: lightweight, wide choice of parameters, scalability, and support for a large number of scanners.

In the end, we decided on a library that works with most devices based on the TWAIN protocol. Our work was accompanied by a thorough study of the source code and undocumented features of the library in order to effectively overcome the emerging obstacles and think more deeply about the architecture of our solution, as well as further scaling.

Especially for internal testing of prototypes, we organized the developer’s workplace so that he had a choice of different scanning devices: direct connection of the scanner to the PC, network connection of the scanner, and virtual scanner.

The entire work process was planned and divided into iterations. The completion of each iteration was accompanied by a demonstration of the results to the customer and a detailed report for all stakeholders. Moreover, during the first 2 weeks of work, we already had a prototype ready, which we handed over to the customer for intermediate testing. In other words, we worked on the MVP in parallel and received feedback from the customer to make the necessary improvements.

To design the MVP, we also engaged an architect with many years of experience who followed the development process throughout. He suggested ideas, delved into problems, and participated in brainstorming sessions.

Technological Overview

The project is an application implemented using Jetty service, Servlet API, and TWAIN library. The application itself is installed on the user’s PC using our installer and runs in the background. After installation, three endpoints become available on a special port: a method to get the list of available scanners, a method to get the available DPI for a particular scanner, and a method to scan. All methods establish communication with the scanners and send requests to them. Thus, if a client has its own system that runs in a browser, it looks like a simple Ajax request via JavaScript, accessed via a URL.

The scanning method is implemented with customizability – when requested, we can specify DPI, scanning type (color or black and white), as well as select the scanning method (multi-page or single sheet). As a result of scanning, all data is saved in temporary files to the hard disk, after which the files are read from the file system and sent in the response. Once the response has been sent to the end user, the temporary files are deleted from the hard disk.

Our application allows us to work with scanners using our endpoints. It is capable of scanning large amounts of data without requiring a large memory.

Result

Our team performed extensive R&D and prepared an MVP that can be scaled and extended to meet the customer’s specific requirements.
The work was completed slightly earlier than the deadline, and with the implementation of additional functionality that was not specified in the initial requirements of the customer.

Thanks to the ability of JazzTeam to attract highly qualified specialists, the development process was clearly planned and competently organized.

After the delivery of the application, we continued to support the application during its implementation into the customer’s system and further productization.

JazzTeam Achievement

To solve the problem with excessive RAM consumption, we conducted additional research and, as a result, we applied a method of saving data in temporary files on the hard disk. This way, we were able to stabilize memory consumption and generally improve the performance of the application. RAM consumption at peak times did not exceed 50MB.