How to Connect the ExtraHop REST API to Highcharts for Killer Metric Visualizations

We put a lot of effort into the UI of our products. We want you to use them - but we also recognize that there will be times when you want to use something else. You're in luck when such a time arises though, as we have a variety of solutions for getting data out of the ExtraHop platform, and into whatever other system you want to use. Want to integrate some of our charts into your favorite "manager of managers?" Or perhaps you need to pull metrics from an application and programmatically feed it into a business process? These are just a couple of things that you can accomplish using our REST API.

Getting started with a new API can be intimidating, so I'm collecting some examples to show how to get going. When I heard that Nick Braun from the ExtraHop engineering team had an example of how to wire up ExtraHop metrics to Highcharts, it seemed like a perfect example to start off with.

Getting Started with Highcharts and The ExtraHop REST API

A warning: I did this all in Linux, and I think it should more-or-less work in Mac OS X with these instructions. Getting it working in Windows is probably possible, but you might need to venture out on your own to get there. With that out of the way, let's see how this works!

First, download the code from the ExtraHop Github. The repo is called extrahop-platform-showcase. Extract it to your home directory, then look in the HighchartsPOC directory for this example. Change into that directory, as any relative file paths I mention from here on out will be with respect to this directory.

There are two paths to running the example: use a node.js backend based on the Express framework, or host the files on a webserver of your choosing. The big difference between the two is what actually performs the REST calls to the ExtraHop Discover Appliance; in the case of the Express backend, this is handled by the included server, whereas the client (your browser) does it directly when using a general purpose HTTP server. Later on we'll get into cross-origin resource sharing, but be aware that if you're not running at least version 5.2 of the ExtraHop software on your discover appliance, you'll probably want to use the Express backend.

Client-Side Via HTTP Server

If you already have a web server (I used lighttpd that I already had running on my workstation, but most anything should work), drop the files some place where they can be served up (for me that was under /var/www/). You'll need to configure a few variables in the code to tell it how to reach an ExtraHop Discover Appliance (or an ExtraHop Command Appliance), as well as configure the ExtraHop to enable cross-origin resource sharing (CORS).

First, edit public/js/main.js within the code you downloaded from us. Towards the top of the file you'll see a couple of variables specifying how this should work. At a minimum set EXTRAHOP_API_KEY to your key (Don't know what an API key is? We have a doc for that), and EXTRAHOP_HOST to the value of an EDA with "https://" prepended. For example:

// API POST Parameters
// Application OID to display stats from
// can be extracted from the ExtraHop URL by navigating to:
//   Metrics > Applications and clicking on the application you want
// OID will appear in the url as: (example: applicationOid=<id_here>)
var APPLICATION_OID = 0, // TODO: Populate this!
// API Key can be generated from the ExtraHop Admin panel
// for more information,
// please visit: https://docs.extrahop.com/current/admin-ui-users-guide/#api-access
// Populate if not using Express backend
EXTRAHOP_API_KEY = 'WHATEVER_YOUR_KEY_IS',
// Populate EXTRAHOP_HOST if not using Express backend and you've added your host as a
// 'trusted origin' inside the ExtraHop admin
EXTRAHOP_HOST = 'https://my-extrahop-eda.example.com',
LOOKBACK_HOURS = 6;

Feel free to also change APPLICATION_OID to be the ID for one of your applications, but if you're working against an EDA (as opposed to an ECA) the default of 0 represents "All Activity" and should be fine. If you are connecting to an ECA you should set this to the OID of one of your applications (these are exposed via the API, but it's easiest to navigate to "Metrics", then "Applications", pick an app, and then look at the URL for the applicationOid query string variable). Similarly, LOOKBACK_HOURS can be tweaked as well.

If you tried to access this right now using your browser, things probably wouldn't work so well due to a lack of cross-origin resource sharing. The code accesses the REST API through use of XMLHttpRequest (sometimes called AJAX) and by default XHR is only allowed to connect to the same domain as the parent web page - in this case that's localhost or at least something other than your EDA, and so the browser won't feel obliged to do it. The trick is to configure the Discover appliance to permit this. As of version 5.2 of the ExtraHop software, this is now exposed as part of the API Access portion of the product UI (There's a doc for that too). If you're running an older version, you should probably be using the Express backend covered in a bit.

Once you've done that, you should be able to fire up a browser, connect to your web server requesting the public/index.html file and see something like this:

Click image to zoom

Those are a pair of charts showing metrics for HTTP responses by status codes, and by server addresses. If you see two charts resembling that, it's working! More about how it works later.

One final thing that might come up: your browser will be connecting to your ExtraHop appliance via a secure connection, and that requires it to trust the certificate from the appliance. By default ExtraHop appliances ship with a self-signed certificate that will not be trusted, requiring the user to acknowledge this before it will continue. If you try this and it doesn't work, one quick thing to try is to open another tab within the same browser instance and connect to the web UI of the appliance using the same IP or hostname that was used for the EXTRAHOP_HOST variable. You'll probably have to acknowledge the self-signed certificate, but doing so here might be enough to get the browser to allow connections via AJAX to the ExtraHop. I know I ran into this!

Express Backend Via Node.js

If you don't have a web server already, or don't want to mess with CORS, there is another way to get this working: spin up a Node.js instance with Express and a few other modules, and let the backend perform the REST calls rather than your browser.

So, to do that you won't need to mess with a web server or worry about placing the files anywhere specifically - your home directory should be fine. First, you'll need to install Node.js. Use whatever package manager is appropriate for your distro, or build from source, but get Node.js installed. Next, install the dependencies that our code needs. As an unprivileged user (as in, not root) run:

$ npm install

You need to be in the root of the HighchartsPOC directory and there should be a file named package.json in case that command isn't successful.

Next, edit app.js. Near the top of this file, look for the lines with EXTRAHOP_HOST and EXTRAHOP_API_KEY and change them so that they have your API key and EDA address. For example:

// API access variables
// If using backend instead of accessing ExtraHop directly from client-side,
// populate the following fields:
var EXTRAHOP_HOST = 'my-extrahop-eda.example.com',
    EXTRAHOP_API_KEY = 'WHATEVER_YOUR_KEY_IS';

It's worth noting that in this file you do not have to prepend the IP/hostname of your EDA with "https://" like in the clientside version. Also, if you edited public/js/main.js for the clientside example but are now trying this one, revert your changes to that file - having the host variable configured in both places could cause some strangeness (your browser might try to connect to the ExtraHop directly rather than from the backend - and if CORS isn't configured it won't work).

Once the code has been adjusted, you're ready to start the server:

$ node app.js

At that point you should be able to aim your browser at http[:]//localhost:3030/ and see something like:

Click image to zoom

That should look about the same as the other one - and that's because it's accessing the same data, just accessed slightly differently.

How It Works

If you managed to get it working in one of those two ways, you're probably asking how it works now. Good question. The core of the code lies in public/js/main.js so let's crack that open. This file consists of an anonymous function that is executed whenever a set of data needs pulled from your ExtraHop appliance and fed to Highcharts. I won't be explaining how Highcharts works, but instead just how the data is retrieved from our REST API.

The top of the file consists of some constants that are defined to configure how to connect and how much data to pull. We customized that in the section titled Client-Side Via HTTP Server. Let's skip past that for now, and instead pick apart of how REST calls are made.

The first interesting parts are when the BY_STATUS_CODE_PARAMS and BY_SERVER_PARAMS variables are defined. The ExtraHop REST API accepts JSON objects when using the POST HTTP method, and these two objects represent the objects that are directly sent to an ExtraHop API endpoint. It's worth pointing out that the object_ids attribute in each object is defined as an array consisting of a single value: the application OID as a Number primitive. The from attribute is calculated inline to be the desired amount of time to include in the query in milliseconds (ms) - the negative sign indicates it should be relative, but if it were positive that would indicate absolute time from the UNIX epoch (12:00:00 AM January 1, 1970). until is set to 0 which means now. Just like from, if until were a positive number that would indicate ms since the UNIX epoch as well. cycle is set to "auto" which indicates that the backend should choose the granularity of data to send back, but you could also specify "1sec," "30sec," "5min," "1hr," or "24hr."

Both objects have metric_category and metric_specs attributes, though different values. The object_type for both is "application." To understand where the values from the code come from, I recommend consulting the Metric Catalog in the product UI (it's under the Settings menu). Find a metric that you want to use, click on it, then scroll the right side of the dialog down to where you see "REST API Parameters" - that's a subset of the Javascript that you'll need to use in your own code. To see the BY_SERVER_PARAMS example, type "http_server_addr_detail" into the search box of the Metric Catalog, then click on the entry for "HTTP - Requests by Server". Look familiar? That same code is mixed in with other attributes in the Highcharts code.

Click image to zoom

The next chunk of code where HORIZONTAL_BAR_GRAPH_OPTIONS is defined is just a definition for how Highcharts will display the graphs once it's time to render. Nothing too interesting there (feel free to customize this to your liking), but since it's not related to making queries against the REST API I'm going to skip it for now.

Let's move down to where the authKey variable is defined. The purpose of this code is to determine what is going to perform the REST query: the Express backend, or your browser. Remember from before when I said to only define EXTRAHOP_API_KEY and EXTRAHOP_HOST variables in public/js/main.js if you were using the pure clientside implementation? Well this is why… if those variables are defined in this file, the next part of the code will make HTTP requests to that location (your ExtraHop) using your API key - but if they're left as empty values, the HTTP requests will be made to the same endpoint that is serving them up, which for us means the Express backend. I'll get to how the Express backend makes those queries when using it later, but for now let's focus on what your browser is doing, remembering that the HTTP requests might be going to either the ExtraHop appliance directly, or to the Express backend.

There is a pair of code blocks beginning with $.ajax (this is a jQuery thing if you want to read up on it). This bit of code is what actually causes your browser to issue HTTP POST queries. In both cases, the URL path that is requested is "/api/v1/metrics/total". This path corresponds to fetching metrics from the API (see the later section on using the REST API Explorer for how to look at other valid paths). We specify that we will be sending JSON data (text in Javascript Object Notation) and indicate the auth info (which might be empty). The data attribute represents the values of BY_STATUS_CODE_PARAMS and BY_SERVER_PARAMS serialized as JSON text. Furthermore, a callback function is specified that whenever the requests are complete to call either populateByStatusChart() or populateByServerTable() functions.

Those two functions are pretty simple. First, you'll see definitions for two other functions (parseByServerResponse() and parseByStatusCodeResponse()) which are actually called at the start of the two "populate" functions. These two functions take the results from the REST API and convert them to what the Highcharts API needs in order to actually render it as a chart. The comments in the code show what format the data needs converted to - a two dimensional array of key/value pairs where the keys are either the status code or the IP/hostname depending upon which function it is. Once the ExtraHop response is converted to the Highcharts format, the results are sorted, the top 8 results are kept (and the rest discarded), and a data structure (format dictated by Highcharts) is populated containing some formatting data along with the top 8 results of the sorted data, before being passed into the Highcharts code for rendering as charts.

If you're using the Express backend, that all still happens, only rather than your browser directly contacting the ExtraHop's REST API endpoint, it instead sends the same requests to Express running under Node.js. Take a look at app.js to see how that works. This code acts as a simple web server with explicit support for serving up the static files (/index.html and the files under /public) and to act as a proxy for calls with a path of '/api/v1/metrics/total'. That path should look familiar - it's the one for retrieving metrics from an ExtraHop, so when it receives a POST request for that path, it turns around and issues the same request to the ExtraHop on the user's behalf along with the API key. When the ExtraHop returns the result, it streams it back to the web browser which transforms and displays it in the same way as before.

Pretty cool, right?

Using the REST API Explorer

So the code we just discussed covers a pair of calls to a specific URL for retrieving metrics from an ExtraHop. But what if you wanted to do something else via the REST API? The easiest way to see how the entirety of the REST API works is by using the interactive ExtraHop REST API Explorer which can be accessed directly from the product's UI under the user menu of the system - that's the button towards the upper right of the UI labeled with the current user. I'm being lazy here and using the setup user, so it looks like this:

Click image to zoom

Choose API Access, then copy the key you want to use (or create one if there isn't a suitable one yet). Click the link on this page for ExtraHop API Explorer, then paste the API key that you copied into the box at the top of the new screen and click Explore. You can now browse the different methods that can be called, what their parameters are, and even experiment with making calls right from this interface - plus it even has code snippets for a couple of programming languages to serve as a starting place once you get some results you want to use elsewhere.

Conclusion

So that's been a whirlwind example of some code that calls the ExtraHop REST API and how to look up your own metrics and even make other types of calls against a Discover appliance. Hopefully it gets you thinking of the sorts of things you might want to do with the API. Stay tuned for a few other examples of how to use our APIs to do interesting things.

Already want more? Check out our post on How to Access the ExtraHop REST API using Java.