Challenge

After initial research, FortySeven engineers outlined the following key requirements for the required solution: Extensive reporting on application usage data collected from users' devices. The amount of reported data is expected to increase. Ability to calculate existing/new stat types based on all collected data. In other words, data storage must be provided for life. Ability to filter calculated statistics and create charts based on specific criteria. The ability to automatically perform statistical calculations based on specific periods.

FortySeven developers implemented a multi-stage data line:

• First stage: a collection of primary data. To lower the cost of the reporting API and protect it from unauthorized access, the web development team used the Amazon Cognito service, which provides accurate permission management for both authorized users and guests. The reports are collected from the Android apps on the devices and sent directly to the S3 group with Cognito-controlled write-only access. This is an entirely secure transaction because it is impossible to steal information from the Amazon S3 bucket that is accessed only by writing. The data received as a result of the above operations contains a large number of small report files in a special repository S3
• The second stage: data compression. Before storing data in the main bucket, we convert it to larger, compressed, vertically formatted files that are optimized for partial reading of data. We enable the conversion with the help of a Spark icon that runs on the Amazon Web Services EMR Spot Instance Hadoop block that automatically turns on and off. Here we see the benefit of the AWS dashboard it brings to the project because the EMR service provides Hadoop group management automatically without engineering efforts
• The third stage: statistics. FortySeven engineers manually launched a similar EMR-based suite and ran the distributed Spark software to calculate the required statistics. The statistical output is saved in an extensive SQL database hosted by Amazon. The analytical result contains several SQL tables optimized to search in a date range
• Fourth stage: visualization of statistics. The Apache Tomcat web interface uses the D3 Javascript library to draw graphs based on the data results stored in the SQL database defined according to specific criteria.
• An app’s user can pick out the object by scanning a barcode or QR code