The Power of Data-Driven Applications

A breakdown of our internal tool, the Hyperion Website Auditing App

A few months ago, I started work on an auditing tool that would help speed up the website auditing process. The problem I kept having was that Lighthouse, Google's premier web page auditing software, is limited by only letting you take a look at one web page at a time. Its a powerful tool, but I wanted to be able to get a picture of the entire website's performance and other metrics, all in one go.

So I found a way to programmatically batch these lighthouse audits that would speed up the data-gathering portion, but this generated a new problem. I now had mountains of data to make sense of and organize in a meaningful way! Some websites have hundreds of thousands of pages, and some have separate applications on subdomains that deserve their own audit. I realized quickly that I would need to engineer a data-driven application that could take the output of these many individual web page audits and turn them into a concise picture of the four Lighthous metrics- performance, accessibility, best practices and search engine optimization (seo).

The Hyperion Audits App

The result: the Hyperion Audits App.

This is the tool I now use to make deep-dive website audits happen in a fraction of the time I used to do them in. Now, this is an ever-evolving project, and we have plans to release a public-facing version of the app in the near future so that customers can make audits of their own sites.

I wanted to briefly break down the underlying data infrastructure of the app to give the common business owner an understanding of what goes into an app like this. It may look simple from the outside looking in, which in a lot of ways is the desired result, but there is a lot going on underneath when it comes to data handling on this scale.

The Input Data

The auditing process begins by invoking a batching program that produces the data that will be the input to the application. Right now, this data is dropped into the drag and drop feature of the app and the app reacts by performing certain calculations and data-handling. Here is a look at what the data structure of the input data file looks like:

This data file is called a "JSON" file, or javascript object notation. Its an efficient way of describing any type of repeatable data in a the language that modern web applications are written in.

Component-Based Architecture

When you open the data file in the browser, the data is nice and human-readable. The trick though, is making sense of it on large scales. The modern approach to web applications that are data-driven is to break the app down into components. This is known as "component-based architecture".

The app first requires the user to drop the JSON data file into the app, or select it from the user computer's file system.

Once the data has been dropped into the field, the app makes the data available to all components that need the data to make the necessary calculations and visual updates, for example, updating the table with all the data.

The data cascades through components, like the Scoreboard Component, and is passed to the Gauge Component which programmatically averages the categorical data for each web page. At the same time, the same data is processed differently when it is passed to the Table Component, where the data is used to first build the table and then the cell background color updates based on the value of each cell. This is possible by letting each Cell Component have its own logic.

The application is designed to handle the variations in data - and respond to any issues with specific error-handling instructions - as the data cascades through the application.

Some Extra Features

I needed to be able to make compelling reports for client's whose websites I audit. This is done by adding an export to PDF feature. Implementing this feature required the integration of specialized javascript libraries that handle the conversion of the web page to the pdf format. This again utilizes the power of components to abstract away the complicated logic behind the button that triggers the web-page-to-pdf conversion event.

The result is a handy, nicely formatted pdf that can be shared anywhere.

Conclusion

Just like that, we have taken a data file, sometimes with thousands of data points, and turned it into usable insights at a glance that can be sent to clients to show them what is going on behind the scenes of their website.

I hope this gives you some insights into the power of data-driven applications. While this is just a small example, the use cases for data driven apps is nearly unlimited. They can be used to automate and enhance just about any mundane task you can think of!

Thanks for reading, and keep an eye out for updates to the app coming soon - features like enhanced data visualization, email sharing of reports and hosted results pages for ongoing monitoring of a website.