• /
  • Log in
  • Start now

Getting started with New Relic and Terraform

30 min

Terraform is a popular infrastructure-as-code software tool built by HashiCorp. You use it to provision all kinds of infrastructure and services, including New Relic dashboards and alerts.

In this guide, you learn how to set up New Relic alerts with Terraform. More specifically, you provision an alert policy, four alert conditions, and a notification channel. The four alert conditions are based on the four golden signals of monitoring introduced in Google’s Site Reliability Engineering book:

  • Latency: The amount of time it takes your application to service a request.
  • Traffic: The amount of requests your system receives.
  • Errors: The rate of requests that fail.
  • Saturation: The stress on resources to meet the demands of your application.

Before you begin

To use this guide, you should have some basic knowledge of both New Relic and Terraform. If you haven't deployed a New Relic open source agent yet, install New Relic for your application. Also, install the Terraform CLI.

Step 1 of 5

Bootstrap Terraform and the New Relic provider

Start by initializing a working directory and creating a Terraform configuration file:

bash
$
mkdir terraform-project && cd terraform-project
$
touch main.tf

Next, instruct Terraform to install and use the New Relic provider, by setting the terraform and required_providers blocks in main.tf:

terraform {
# Require Terraform version 0.13.x (recommended)
required_version = "~> 0.13.0"
# Require the latest 2.x version of the New Relic provider
required_providers {
newrelic = {
source = "newrelic/newrelic"
version = "~> 2.12"
}
}
}

In this code block, you're setting the required version of Terraform to 0.13.x and setting the New Relic provider to the latest 2.x version. Using the right version constraints for your setup will provide better stability with your Terraform runs.

Now that you've set your Terraform and New Relic provider versions, you need to configure the New Relic provider.

Step 2 of 5

Configure the New Relic provider

With terraform all set, configure the New Relic provider with the following items:

  1. Your New Relic Account ID.
  2. Your New Relic user key. Most user keys begin with the prefix NRAK-.
  3. Your New Relic region. Your region is US if your account settings page is located at one.newrelic.com, and EU if your account is located at one.eu.newrelic.com.

In main.tf, set those values on the provider:

provider "newrelic" {
account_id = 12345 # Your New Relic account ID
api_key = "NRAK-***" # Your New Relic user key
region = "US" # US or EU (defaults to US)
}

By setting these values on the New Relic provider, you're configuring that provider to make changes on behalf of your account through New Relic APIs.

Tip

You can also configure the New Relic provider using environment variables. This is a useful way to set default values for your provider configuration.

For more information about configuring the New Relic provider, please feel free to check out our official provider documentation.

With your New Relic provider configured, initialize Terraform:

bash
$
terraform init

When Terraform finishes installing and registering the New Relic provider, you'll receive a success message and some actionable next steps, such as running terraform plan. Before you can run terraform plan, however, you need to create your resources.

Step 3 of 5

Create a New Relic alert policy with the golden signal alerts

With the New Relic provider configured and initialized, you can define an alerting strategy for your application.

Since you're targeting a specific application, use a newrelic_entity to fetch the application information from New Relic and allow us to reference that data elsewhere in the configuration:

data "newrelic_entity" "example_app" {
name = "Your App Name" # Must be an exact match to your application name in New Relic
domain = "APM" # or BROWSER, INFRA, MOBILE, SYNTH, depending on your entity's domain
type = "APPLICATION"
}

Next, create a newrelic_alert_policy. Give the policy a dynamic name based on your application's name. This helps specify the scope of the policy:

resource "newrelic_alert_policy" "golden_signal_policy" {
name = "Golden Signals - ${data.newrelic_entity.example_app.name}"
}

At this point, you should be able to test your configuration with a dry run:

bash
$
terraform plan

You should see output that displays Terraform's execution plan. The plan contains the actions Terraform performs when your run terraform apply:

bash
# Example output
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# newrelic_alert_policy.golden_signal_policy will be created
+ resource "newrelic_alert_policy" "golden_signal_policy" {
+ account_id = (known after apply)
+ id = (known after apply)
+ incident_preference = "PER_POLICY"
+ name = "Golden Signals - Your App Name"
}
Plan: 1 to add, 0 to change, 0 to destroy.
------------------------------------------------------------------------

In this case, the plan shows you that Terraform will create a new alert policy when you run terraform apply. After verifying the details, execute the plan to provision the alert policy resource in your New Relic account:

bash
$
terraform apply

Every time you apply changes, Terraform asks you to confirm the actions you've told it to run. Type "yes".

While it's running, Terraform sends logs to your console:

bash
# Example output of `terraform apply`
newrelic_alert_policy.golden_signal_policy: Creating...
newrelic_alert_policy.golden_signal_policy: Creation complete after 1s [id=111222333]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Log in to New Relic and navigate to Alert Policies to confirm that Terraform created your new policy.

As you move through the next steps of creating alert conditions, you can run terraform apply after configuring each resource. Refresh your alert policy webpage to see the new resources.

Step 4 of 5

Provision alert conditions based on the four golden signals

Next, add alert conditions for your application based on the four golden signals: latency, traffic, errors, and saturation. Apply these alert conditions to the alert policy you created in the previous step.

Latency

Most folks want to avoid slow response times. You can create a newrelic_alert_condition that triggers if the overall response time of your application rises above five seconds for five minutes:

# Response time
resource "newrelic_alert_condition" "response_time_web" {
policy_id = newrelic_alert_policy.golden_signal_policy.id
name = "High Response Time (Web) - ${data.newrelic_entity.example_app.name}"
type = "apm_app_metric"
entities = [data.newrelic_entity.example_app.application_id]
metric = "response_time_web"
runbook_url = "https://www.example.com"
condition_scope = "application"
term {
duration = 5
operator = "above"
priority = "critical"
threshold = "5"
time_function = "all"
}
}

Note that you're linking this alert condition to the previously configured alert policy with policy_id.

Traffic

Traffic represents how much demand is placed on your system at any given moment. Throughput is a metric that measures how much traffic goes to your application. Create a newrelic_alert_condition that triggers if the overall response rate of your application falls below five requests per minute for five minutes:

# Low throughput
resource "newrelic_alert_condition" "throughput_web" {
policy_id = newrelic_alert_policy.golden_signal_policy.id
name = "Low Throughput (Web)"
type = "apm_app_metric"
entities = [data.newrelic_entity.example_app.application_id]
metric = "throughput_web"
condition_scope = "application"
# Define a critical alert threshold that will
# trigger after 5 minutes below 5 requests per minute.
term {
priority = "critical"
duration = 5
operator = "below"
threshold = "5"
time_function = "all"
}
}

This type of alert is useful when you expect a constant baseline of traffic throughout the day — a drop off in traffic can indicate a problem.

Errors

If your application's error rate spikes, you need to know about it. Create a newrelic_alert_condition that triggers if your application's error rate rises above 5% for five minutes:

# Error percentage
resource "newrelic_alert_condition" "error_percentage" {
policy_id = newrelic_alert_policy.golden_signal_policy.id
name = "High Error Percentage"
type = "apm_app_metric"
entities = [data.newrelic_entity.example_app.application_id]
metric = "error_percentage"
runbook_url = "https://www.example.com"
condition_scope = "application"
# Define a critical alert threshold that will trigger after 5 minutes above a 5% error rate.
term {
duration = 5
operator = "above"
threshold = "5"
time_function = "all"
}
}

Saturation

Saturation represents how "full" your service is and can take many forms, such as CPU time, memory allocation, or queue depth. In this example, assume you already have a New Relic Infrastructure agent installed on the hosts serving your application, and you want to configure an alert for when CPU utilization spikes above a certain threshold:

# High CPU usage
resource "newrelic_infra_alert_condition" "high_cpu" {
policy_id = newrelic_alert_policy.golden_signal_policy.id
name = "High CPU usage"
type = "infra_metric"
event = "SystemSample"
select = "cpuPercent"
comparison = "above"
runbook_url = "https://www.example.com"
where = "(`applicationId` = '${data.newrelic_entity.example_app.application_id}')"
# Define a critical alert threshold that will trigger after 5 minutes above 90% CPU utilization.
critical {
duration = 5
value = 90
time_function = "all"
}
}

For the Infrastructure alert, you created a newrelic_infra_alert_condition that triggers if the aggregate CPU usage on these hosts rises above 90% for five minutes.

Step 5 of 5

Get notified when an alert triggers

Now that you've configured some important alert conditions, add a notification channel to your alert policy to ensure the proper folks get notified when an alert triggers. To do so, use a newrelic_alert_channel.

To begin, create an email notification channel to send alert notifications to your email. Use this when you want to notify a specific person or team when alerts are triggered:

resource "newrelic_alert_channel" "team_email" {
name = "example"
type = "email"
config {
recipients = "yourawesometeam@example.com"
include_json_attachment = "1"
}
}

If you want to specify multiple recipients, use a comma-delimited list of emails.

Last, but not least, in order to apply the notification channel to your alert policy, create a newrelic_alert_policy_channel:

resource "newrelic_alert_policy_channel" "golden_signals" {
policy_id = newrelic_alert_policy.golden_signal_policy.id
channel_ids = [newrelic_alert_channel.team_email.id]
}

A newrelic_alert_policy_channel links the notification channel you just created to your alert policy.

To finalize your golden signal alerts configuration, run terraform apply one last time to make sure all of your configured resources are up to date.

Extra Credit

new_relic_alert_channel supports several types of notification channels, including email, Slack, and PagerDuty. So, if you want to explore this more, try creating an alert channel for a second channel type, such as Slack:

# Slack notification channel
resource "newrelic_alert_channel" "slack_notification" {
name = "slack-example"
type = "slack"
config {
# Use the URL provided in your New Relic Slack integration
url = "https://hooks.slack.com/services/XXXXXXX/XXXXXXX/XXXXXXXXXX"
channel = "your-slack-channel-for-alerts"
}
}

Before you apply this change, you need to add the New Relic Slack App to your Slack account and select a Slack channel to send the notification. With this new alert channel, triggered alerts send notifications to the Slack channel of your choice.

Conclusion

As your team evaluates the alerting system you’ve put in place, you’ll find that you may need to tweak configuration values, such as the alert threshold and duration. If you manage your Terraform project in a remote repository, you can submit a pull request so your team can review these changes alongside the rest of your code contributions.

Tip

You may also want to consider automating this process in your CI/CD pipeline. Use Terraform's recommended practices guide to learn more about their recommended workflow and how to evolve your provisioning practices.

Congratulations! You're officially practicing observability-as-code. Review the New Relic Terraform provider documentation to learn how you can take your configuration to the next level.

Edit this pageCreate an issue
Copyright © 2021 New Relic Inc.

This site uses cookies 🍪

We use cookies and other similar technologies ("Cookies") on our websites and services to enhance your experience and to provide you with relevant content. By using our websites and services you are agreeing to the use of cookies. You can read more here. If you consent to our cookies, please click Yes.