Automated Competitors Price Monitoring and Self-optimizing System

Duration: 1 year
Industries: E-Commerce, Marketing, Advertising, Sales
Services: Product Development, Backend Development, Frontend Development, Software Reengineering, Refactoring and Modernization. Cloud Migration
Software Categories and Types: Social Media Analysis/Crowling, Analytics Systems, Data Analysis & Visualization, B2B Solutions
IT Architecture Paradigms and Approaches: Enterprise Integration Patterns, Plugins Architecture and Development, Architectural Patterns
Technical Expertise: Java Server Side Development, Web Development, Databases
DevOps Expertise: Advanced build organization, CI/CD Basics, CI/CD Advanced, Virtual Machines
Technologies: Jenkins, Bootstrap (Twitter Bootstrap), Git, Apache Maven, GitHub, Data Bases, RabbitMQ, JUnit / TestNG, Apache Tomcat, Apache Spark, Project Management, Collaboration and Bug Tracking, Jira, CI/CD Automation Servers, Hibernate, MySQL, Java, Java Enterprise Edition (EE), CSS, Keycloak, Frontend, Apache Solr, Build and Dependency Platforms and Tools, HttpUnit, Backend, JVM Languages, Scala, JavaScript, JavaScript Frameworks, Spring, ExtJS, CSS Frameworks, Containers and DevOps Tools, Puppet, Source Code Management (SCM), Chrome DevTools, Identity and Access Management Solutions, Test Automation, jQuery, Jackson, Open SSH, CI/CD and DevOps, Putty, Message Queues and Service Buses, Languages, Protocols, APIs, Network Tools, Network Tools, Software Engineering and Management Tools, ORM & Persistence, FileZilla, Data Science and Machine Learning Infrastructure, Various Platforms, Solutions and Engines, Java Libraries, Web and J2EE Servers, Java Runtimes, Firebug, HipChat

Team size (2):
1 Project Manager
1 Backend

Project summary

The system provides meticulously customized monitoring of competitors’ stores and allows automatic optimizing the prices of the goods according to certain rules in real time.

For instance, you can set a rule so that the prices in your store are always lower than the competitors’ prices by a certain value. Let’s say 1$ or 1 %. In this case the system will monitor your competitors’ prices regularly and promptly underbid prices on similar products.

The competitors’ websites are scanned with the specific application (crawler), which collects necessary information and saves all information on found products in system database. Crawler can be run either on schedule or on system client’s request. There’s the whole store and certain products scan support.

While scanning a marketplace (for example: Amazon, Ebay, Pricerunner) the system automatically finds your competitors, who have analogous products as you do. The system informs a user not just about existing competitors, but also about new ones. After that the user is able to add a new competitor into the system and customize new price changing rules for user’s products.

The system provides access to charts, graphs, which allow to analyze current situation easily and to build a business development strategy, based on collected information and the history of changes.

The system consists of two independent applications: web application and crawler.

1. Web application is an application which the user interacts with: sets up his dashboard, adds competitors, sets products price regulation rules, controls product scanning.

Backend is implemented on Java using Spring. Database MySQL is used in the system, Apache Solr is used for quick access to data. The application is divided into modules, which could be expanded on separate servers, RabbitMQ is used for connection between the modules.

The frontend is developed with ExtJS and Bootstrap.

KeyCloak server provides protection.

2. Crawler is a separate application, which scans internet stores. It is developed in Scala with using Apache Spark framework.
Parsing scripts are written for most cms stores. These scripts analyze a store structure, find information about products, save information in database. For parsing new and non-standard cms there’s a possibility to specify individual parsing rules using a certain pattern.

Our team was responsible for the development of new highly loaded system components, legacy code refactoring and unit tests application coverage.

Technologies

Backend: Java EE, Hibernate, JPA, Spring, KeyCloak, MySQL, Solr, RabbitMQ, Jackson, Apache Tomcat.

Frontend: ExtJS, JQuery, JavaScript, Bootstrap.

Crawler: JavaScript, Spark, Scala.

Jira, HipChat, GitHub, Jenkins, Maven.

Screenshots

Project features

Joint work in a distributed international team.
Quick entry into the project and fast, short stages of development.
Frequently changing functional requirements.
Work on the product functioning on high risky market.
Need of immersion into the peculiar qualities of the major global trading platforms functioning.

Project results

Work of the existing functionality was stabilized.
The new functionality was realized.
Work with database at high loads was optimized.

Company’s achievements during the project:

The project is successfully finished in conditions of dynamically changing requirements.
We initiated and took part in the database architecture development.