How to Get ExtraHop Metrics in AWS Cloudwatch

It's been awhile folks, but I'm back with some more ExtraHop trigger goodness. We have had several users ask for the next iteration of the CloudWatch integration that debuted a few years back, marrying AWS machine data with ExtraHop wire data. We're going to be using Open Data Stream (ODS) functionality, specifically the Remote.HTTP method which lets us make RESTful API calls directly from the our triggers. If you're new to trigger writing, I recommend checking out the online training as well as my Trigger Optimization 101 series.

You can take this metric and shove it (into CloudWatch)

We're going to be using the PutMetricData method from the CloudWatch API, but before we get into the details, let's configure Open Data Stream so it knows how to talk to AWS.

Click image to zoom

Configuring ODS

Information you'll need:

• Your Region (I'll be using US West 2)
• Your AWS endpoint to send your API requests, which can be found on the AWS Regions and Endpoints page. For US West 2 and CloudWatch, we'll use monitoring.us-west-2.amazonaws.com
• Your AWS Access Key ID
• Your AWS Secret Key

In the ExtraHop Admin UI under System Configuration, click Open Data Streams. We're going to configure a new ODS using the information from above. Select Amazon AWS under Authentication and the ExtraHop platform will handle signing every API request.

Click image to zoom

Success!

Click image to zoom

If the configuration test fails, a good first step is to make sure that outbound connections on port 80 are open for the ExtraHop appliance. You can also use HTTPS (and port 443) in conjunction with the same AWS endpoint. Otherwise hover over the status indicator (colored ball) for more information as to why the test failed.

Configuring the Trigger

The first step to configuring the trigger is to determine which metrics you want to send to CloudWatch. In the following example I'm going to send the unidirectional bytes sent from a device and include the device's MAC address as a dimension. We could just as easily send processing time, error counts, or any other data which we could use in conjunction with the AWS resource metrics like CPU and memory utilization.

Rinse and repeat for additional metrics and dimensions. The for-loop handles structuring the payload properly.

var namespace = 'extrahop';

/*
Valid metric units

Seconds | Microseconds | Milliseconds | Bytes | Kilobytes | Megabytes | Gigabytes
Terabytes | Bits | Kilobits | Megabits | Gigabits | Terabits | Percent | Count 
Bytes/Second | Kilobytes/Second | Megabytes/Second | Gigabytes/Second 
Terabytes/Second | Bits/Second | Kilobits/Second | Megabits/Second | Gigabits/Second
Terabits/Second | Count/Second | None

Metric is a list of objects, where each object takes the same form

{
    'MetricName' : [name],
    'Unit': [unit],
    'Value' : [metric],

    // Dimensions are totally optional, but help slice up data.
    // THEY CREATE A NEW METRIC FOR EVERY NEW KEY-VALUE PAIR ie you can end up with
    // a million metrics if you aren't careful. And you pay for metrics. 
    'Dimensions' : {
        [key1] : [dimension1],
        ...
        [keyN] : [dimensionN]
    }
}
*/

var metrics = [
    {
        'MetricName' : 'testMetric',
        'Unit': 'Count',
        'Value' : Flow.bytes1,
        'Dimensions' : {
            'MacAddr1' : Flow.device1.hwaddr
        }
    }
];

// Format the payload based on our metrics
// Request structure is defined by the PutMetricData method located at
// docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_PutMetricData.html
var payload ='Action=PutMetricData&Version=2010-08-01&Namespace=' + namespace;

for (m_index in metrics) {
    var i = parseInt(m_index) + 1;
    for (aspect in metrics[m_index]) {
        if (aspect === 'Dimensions') {
            var count = 1;
            for (dimension in metrics[m_index][aspect]) {
                payload += '&MetricData.member.' + i + '.' + aspect + 
                           '.member.' + count + '.Name=' + dimension;
                payload += '&MetricData.member.' + i + '.' + aspect + 
                           '.member.' + count + '.Value=' + 
                           metrics[m_index][aspect][dimension];
                count++;
            }
        } else {
            payload += '&MetricData.member.' + i + '.' + aspect + '=' + 
                       metrics[m_index][aspect];
        }
    }
}

var headers = {"Content-Type":"application/x-www-form-urlencoded"};

var obj = {
    'path' : '/',
    'headers' : headers,
    'payload' : encodeURI(payload)
};

debug(JSON.stringify(obj));

//  'cloudwatch' is the name we configured in the Admin UI
Remote.HTTP('cloudwatch').request('POST', obj);

Mix and Match Your Data

With the trigger assigned and firing, you should be seeing metrics in CloudWatch. From here, you can easily chart ExtraHop and CloudWatch metrics together, as well as expand on the metrics you want to send to CloudWatch. If you end up using the script to send metrics to CloudWatch, share your experience in the comments sections below.

Click image to zoom