Blog - MegaMonit |

Know When to Alert on What Matters

It is extremely important to set up automated alerts to assist monitoring. The biggest benefit of automated alerts is to enable spotting problems anywhere in your infrastructure so that you can easily determine the cause and minimize service degradation and disruption. But do you think alerts are reliable? Certainly not. You could miss out real problems in the sea of noisy alarms. If you are looking for effective ways for alerting, this article is for you. In this write-up, we will be explaining a simple yet effective approach to alerting.

In simple terms, an alert should communicate specific about your system. With the help of automatic alerting, you can respond quickly to issues and provide your customers better service. It also helps you save time by freeing from manual inspection of metrics.

important issues

Alerts are created on varied degrees of urgencies. Some alerts may require immediate human intervention, on the other hand, others may point to the areas that may require attention in the future. It is advisable to manage all alerts at a sole destination because it will be easy to correlate with other metrics and events.

Many alerts are not associated with a major problem. For example, when a data store starts serving queries much slower, it should generate a low-urgency alert. This low-urgency alert is recorded in the monitoring system for future reference or investigation. The alert-based data will provide invaluable context for your investigation.

alert monitoring

Notification Alerts of Moderate Severity

Alerts of moderate severity require intervention. If the data store is running low on disk space, it could be scaled out in the next several days. Sending an email in the service owner’s chat room is a perfect way to deliver the alerts.

High Severity Alerts

High severity alerts are given special treatment. They require immediate human attention because failing to do so can worsen the problem. For example, if the response time for your web application is exceeding your internal SLA, immediate action should be taken.

When to create alerts?

Before you set up an alert, it is important to ask yourself several questions to determine the urgency level of the alert. The most common questions to ask yourself are given below:

Is it real?

The first and foremost thing you need to do is, know if the issue is real. Always generate an alert for the real issue. It is upto you if you want to set up notifications but always record alerts within your monitoring system for later correlation and analysis.

404 alert

Does it require attention?

Another important question you need to ask yourself is if it requires your attention. It can be problematic calling someone from work or sleep to fix the problem. If the issue is real and it requires attention, you can generate an alert to notify someone who can investigate and fix the problem. The notification should be sent via email, chat or ticketing system to help recipients prioritize their response.

Is it Urgent?

Not all issues are urgent. Therefore, you should understand if the issues are urgent to handle. If a key system stops performing, you should for an engineer immediately.

Pages on Symptoms

Pages on symptoms are effective for delivering useful information, but these symptoms could be disruptive if overused. If some kind of errors or symptoms are being depicted through page symptoms, you should know what to do immediately. The fact your system has stopped is itself a symptom and it can have a number of causes. Let’s say for example- if the loading speed of your website is quite low, it is a symptom. The possible cause of this symptom could be – high latency database latency, high load, failed application servers, etc.

Pages on symptoms are real and can cause internal problems. No matter how underlying system architectures may change, you will get an appropriate page even without updating your alert definitions. It is crucial to call human attention to a handful of metrics when the system performs adequately. Early warning metrics show the unacceptably high probability that serious symptoms will be developed and may require immediate intervention.

The Verdict

That was everything you need to know about alerting, types of alerting and how to create alerts to handle the complexities. Every issue or problem exhibits several kinds of symptoms. It is important to pay attention to the symptoms to ensure the problem does not affect the overall performance.

Monitoring Amazon’s Application Load Balancer Became Easy with Datadog

Amazon Web Services offers Elastic Load Balancing services to route traffic to multiple targets to ensure each target does a similar amount of work. Elastic Load Balancing distributes incoming traffic across multiple instances. The purpose of ELB is to enhance the fault tolerance of your applications. It is a single point of contact for clients to enhance the availability of their applications. Elastic Load Balancing offers high availability, robust security and automatic scaling to enhance the fault tolerance capability of your applications.

Amazon’s Application Load Balancing

Different Types of Elastic Load Balancer

The most common types of Elastic Load Balancer are given below:

Application Load Balancer – This balancer is best suited for load balancing of HTTP and HTTP’s traffic. Application Load Balancer routes traffic within Amazon Virtual Private Cloud based on the request of the content.

Network Load Balancer – Network Load Balancing is suited for balancing TCP traffic where extreme performance is required. It routes traffic to targets within Amazon Virtual Private Cloud. It is capable of handling millions of requests per second. Network Load Balancer can also optimize volatile traffic patterns.

Classic Load Balancer– Classic Load Balancer is used to provide basic load balancing across multiple Amazon EC2 instances. Classic Load Balancer is used for applications that were built within the EC2 classic network.

Elastic Load Balancing

Impressive Features of Amazon Application Load Balancer

HTTPS Support -An application load balancer is responsible for termination between the clients and the load balancer. It also offers management of SSL certificates through AWS and IAM for pre-defined security policies.

Server Name Indication – SNI is an extension to the TLS protocol by which a client connects to at the start of the TLS handshake. Application load balancer also supports a smart certificate selection algorithm, with SNI.

Higher Availability – An application load balancer is required to distribute incoming traffic across your targets in multiple availability zones. It automatically scales its request handling capacity in response to incoming application traffic.

Safety– Amazon Application load balancer is secure. You can create and manage security groups associated with elastic load balancing to provide additional networking and security options.

Health checks – An application load balancer routes traffic to healthy targets. You can get insight into the health of your application by configuring detailed error codes and analyzing each service on an EC2 instance.

Datadog application load balancer

Why is Monitoring Load Balancer Important?

It is extremely important to monitor your load balancer because it serves as the primary gateway for clients to your application. If not monitored, unexpected issues with your load balancer could result in a target being overwhelmed with requests. The different metrics of load balancer such as end-to-end application latency, error code responses, high-value metrics, etc can be prepared to combat issues coming your way. Customers can also see key-metrics such as healthy and unhealthy hosts along with a full breakdown of HTTP response codes.

Track the number of healthy Hosts

As we mentioned, customers can track the number of healthy hosts to handle the volume of incoming requests. Once a target exceeds the threshold of unhealthy hosts, it stops routing requests to it. If there are unhealthy targets in your system, it can become difficult for the healthy targets to keep up the increased workload leading to dropping of increased workload. Moreover, relating the health of instances with old metrics can help you diagnose points of failure and enhance the performance of the application. You can also setup alerts to get notified about the unhealthy hosts.

Assess Target Performance

If you want to ensure how well your backend is performing, gauge the target performance. To do this, the time elapsed after a request leaves the load balance until a response is received is tracked. If its value comes high, the reasons could be network problems, overloaded targets, and issues on your application infrastructure or database. To troubleshoot the problem ALB latency can be correlated with metrics from your backend infrastructure.

Track the amount of Traffic

The amount of traffic your application receives can be tracked with the number of requests handled by the load balancer. Monitoring the number of requests will help you adjust the number of targets in your backend to handle the changes in the load. HTTP error codes generated by the ALB in response to incoming requests can also be monitored. In case, your ALB is transmitting too many 5xx errors, you can troubleshoot the infrastructure issues you might be having.

Sign Up Today!

If you are using Datadog, start monitoring your ALB’s and the rest of AWS infrastructure by following the integration guide. If you are not using Datadog, register today for a free trial.

Filtering the Noise and Collecting the Right Data

There are a number of structures the monitoring data is available in. While some systems provide data on a regular basis, the others just provide it rarely, only when an event occurs. Depending on the type of data, its importance also varies. Some data are crucial for identifying the issues and the same type, the other data is important for resolving it. However, no matter what the type is, it is only the monitoring data which lets you observe the inner working of the system.

As all the data is not useful, you need to collect it and classify it so that the meaningful data can be retrieved and used for locating and fixing the performance issues. You need to remember that though collecting data is not costly, it becomes just priceless when you need it but don’t actually have it. In this write-up, you will get to know about the types of monitoring data you need to collect and the ways to classify it.

Metrics

Metrics are collected at regular intervals because they capture a particular value related to your system, at a particular time. It depends on the values being captured that whether the metrics will be collected every second, every minute or at any other regular interval. Basically, the metrics are classified into two main types- Resource Metrics and Work Metrics. What you need to do is analyze your software infrastructure, inspect that which metrics are available and then, collect them.

Metrics

1. Work Metrics

These refer to the useful output of your system which indicates its performance and top-level health. You can divide these metrics into four types which are:

Throughput

Generally, recorded as an absolute number, throughput is the amount of work your system does in the particular time unit. Better it is if the throughput is high.

Success

The proportion of work that is executed successfully, without any issue comes under the success metrics.

Error

It is the count of the results which came with an error. Generally, it is measured as the rate of errors in a particular time unit. It can also be expressed as throughput to yield errors in a particular work unit. In the case there are many potential errors, the error metrics are captured separately from other metrics. However, not all the errors are equally serious; some are more actionable than others.

Performance

These metrics are the measurement of the efficiency of a particular component as they are a record that how efficiently a particular component is doing its job. Latency is the most common performance metric; it is the measure of the time a particular unit requires to perform a particular task.

It is clear that all these metrics are crucial to know about the efficiency and the performance of the system that whether it is doing the job it is designed for and if it is delivering the quality work.

Types of monitoring data

2. Resource Metrics

Many components of your software infrastructure act as a resource for other systems, though it depends whether they are the low-level components or the high-level. In general, the resource metrics can help you with the diagnosis and investigation of the problems. While collecting resource metrics, you need to cover the following areas:

Utilization

It represents the percentage of time the particular resource was in use.

Saturation

It is the amount of work that is queued i.e. it has been requested from the particular resource but is not yet serviced.

Errors

These are the errors which are not shown in the work a particular resource produces.

Availability

It is only for those resources which can be regularly checked for availability. It is the percentage of time a particular resource reacted to the work requests.

Monitoring data

3. Other Metrics

These are the metrics which neither comes in the work metrics nor in the resource metrics, like database lock or the count of cache hits. These need to be captured when you have any doubt or want to make the complex system more observable.

Events

These are the infrequent occurrences that can provide you important information about the changes in your system’s behavior. Examples of events include:

Alerts
Changes
Scaling events

Events provide the information that can be interpreted and is meaningful on its own. Though most often they are used for investigating issues and correlating across systems, they can also be used for generating alerts.

Characteristics of Good Monitoring Data

Whether it is the metrics or the events, you should always capture the good data. The characteristics of good data include:

Easy to understand
Granular i.e. collected at a frequency
Tagged by multiple scopes
Long-lived

Conclusion

So, collect as many metrics and events you can but make sure to collect that with sufficient granularity. Also, in order to maximize the value of your data, you can tag events and metrics with different scopes and finally, retain them for at least 15 months.

How to do Monitoring in the Kubernetes era?

What is Kubernetes?

Kubernetes is a powerful system that has been launched by Google for the management of containerized applications running in a clustered environment. It makes the management easier by deploying and scaling the containers that run service-oriented applications. The development of Kubernetes has been supported by a number of renowned partners, including CoreOS, VMWare, Meteor, Red Hat and some others.

kubernetes monitoring

Kubernetes manages the containers by creating, starting, stopping and destroying them automatically so that the applications can run seamlessly and ensure optimal performance. The containerized applications are distributed across the groups of nodes and are orchestrated by:

Automating the deployment of the containers
Scaling the containers in or out, on the fly
Providing load balancing between the containers by organizing them in groups
Rolling out the new application container versions
Providing container flexibility

No matter whether your containers run in AWS, Azure, Google Cloud Platform or in a hybrid or on-premise infrastructure, Kubernetes can deploy them effectively.

kubernetes performance monitorning

Working of Kubernetes

Though containers are used for the application deployment, the workloads which define the types of work are particular to Kubernetes. Here is how the Kubernetes works behind the scenes.

Pods

A pod is the smallest unit in the Kubernetes that you can create and deploy. Pods are basically of two types:

Pods that consist of a single container

In this, the pod acts as a wrapper of the container and rather than managing the containers, Kubernetes manages the pods directly.

2. Pods that contain more than one container

In this, the co-located containers (which need to share resources) are tightly coupled in a pod that wraps them along with the storage resources as a single manageable identity.

Each pod runs a single instance of an application. In the case you want to scale your application horizontally, you need to use multiple pods, one pod for each instance. In Kubernetes model, it is known as “replication”. And these replicated pods are created and managed as a group by a controller.

Nodes and clusters

Nodes are the virtual or physical machines which are grouped into the clusters. And every cluster has at least one master. However, it is better to have more than one master as it will ensure high availability.

Pods run on nodes and each node has one kubelet. This kubelet makes sure that all the containers in the PodSpec are running efficiently.

Also, in Kubernetes, there can be multiple virtual clusters backed by physical clusters, which are termed as namespaces. Thus, you can spin up a cluster and use its resources for different environments. This will help in saving resources, time and cost.

Replica sets and deployments

Replicated sets are the work units that create and destroy pods. Replica sets make sure that the replicas (pods) keep on running all the times and thus, preserve the continuity. In the case, the pods are terminated or failed, the replica set will replace them automatically.

And for the management of replica set, Kubernetes introduced deployments. You just need to describe a state in the deployment object and it will handle the replica sets.

Services

As pods are constantly created and destroyed, their IP addresses do not remain stable and reliable. The communication between pods cannot be carried out by using these addresses. This is where services play their role. A service acts as an endpoint and exposes a stable IP address for a set of pods.

How Kubernetes monitoring differs?

While monitoring Kubernetes, you need to rethink and reorient your strategies because monitoring Kubernetes differs from monitoring the physical machines as well as the traditional hosts, like VMs. Let’s make you know about these differences.

Tags and labels have become essential

Previously, it was not possible to know where your applications are running but now, it has been made possible by the tags and labels. They have become the only way to identify the containers and the pods. Actually, they have made it possible for you to look at every aspect of the containerized infrastructure.

More components are available for monitoring

In the previous infrastructure, there were only two layers that needed monitoring- applications and the hosts which run them. Now, the containers and Kubernetes also need to be monitored and collect metrics from.

Applications move among the containers

Due to the dynamical scheduling of the applications, you always don’t get to know where your applications are running but they still need to be monitored. And for that, you will need a monitoring system that will collect the metrics and will help in the interruption-free monitoring of the applications.

Though you need to think again and change your approach while monitoring Kubernetes, you can easily keep the infrastructure well-orchestrated if you are clear that what you have to observe, how you can track it and how to interpret the data.

Know When to Alert on What Matters

It is extremely important to set up automated alerts to assist monitoring. The biggest benefit of automated alerts is to enable spotting problems anywhere in your infrastructure so that you can easily determine the cause and minimize service degradation and disruption. But do you think alerts are reliable? Certainly not. You could miss out real problems in the sea of noisy alarms. If you are looking for effective ways for alerting, this article is for you. In this write-up, we will be explaining a simple yet effective approach to alerting.

In simple terms, an alert should communicate specific about your system. With the help of automatic alerting, you can respond quickly to issues and provide your customers better service. It also helps you save time by freeing from manual inspection of metrics.

important issues

Alerts are created on varied degrees of urgencies. Some alerts may require immediate human intervention, on the other hand, others may point to the areas that may require attention in the future. It is advisable to manage all alerts at a sole destination because it will be easy to correlate with other metrics and events.

Many alerts are not associated with a major problem. For example, when a data store starts serving queries much slower, it should generate a low-urgency alert. This low-urgency alert is recorded in the monitoring system for future reference or investigation. The alert-based data will provide invaluable context for your investigation.

alert monitoring

Notification Alerts of Moderate Severity

Alerts of moderate severity require intervention. If the data store is running low on disk space, it could be scaled out in the next several days. Sending an email in the service owner’s chat room is a perfect way to deliver the alerts.

High Severity Alerts

High severity alerts are given special treatment. They require immediate human attention because failing to do so can worsen the problem. For example, if the response time for your web application is exceeding your internal SLA, immediate action should be taken.

When to create alerts?

Before you set up an alert, it is important to ask yourself several questions to determine the urgency level of the alert. The most common questions to ask yourself are given below:

Is it real?

The first and foremost thing you need to do is, know if the issue is real. Always generate an alert for the real issue. It is upto you if you want to set up notifications but always record alerts within your monitoring system for later correlation and analysis.

404 alert

Does it require attention?

Another important question you need to ask yourself is if it requires your attention. It can be problematic calling someone from work or sleep to fix the problem. If the issue is real and it requires attention, you can generate an alert to notify someone who can investigate and fix the problem. The notification should be sent via email, chat or ticketing system to help recipients prioritize their response.

Is it Urgent?

Not all issues are urgent. Therefore, you should understand if the issues are urgent to handle. If a key system stops performing, you should for an engineer immediately.

Pages on Symptoms

Pages on symptoms are effective for delivering useful information, but these symptoms could be disruptive if overused. If some kind of errors or symptoms are being depicted through page symptoms, you should know what to do immediately. The fact your system has stopped is itself a symptom and it can have a number of causes. Let’s say for example- if the loading speed of your website is quite low, it is a symptom. The possible cause of this symptom could be – high latency database latency, high load, failed application servers, etc.

Pages on symptoms are real and can cause internal problems. No matter how underlying system architectures may change, you will get an appropriate page even without updating your alert definitions. It is crucial to call human attention to a handful of metrics when the system performs adequately. Early warning metrics show the unacceptably high probability that serious symptoms will be developed and may require immediate intervention.

The Verdict

That was everything you need to know about alerting, types of alerting and how to create alerts to handle the complexities. Every issue or problem exhibits several kinds of symptoms. It is important to pay attention to the symptoms to ensure the problem does not affect the overall performance.

Monitoring Amazon’s Application Load Balancer Became Easy with Datadog

Amazon Web Services offers Elastic Load Balancing services to route traffic to multiple targets to ensure each target does a similar amount of work. Elastic Load Balancing distributes incoming traffic across multiple instances. The purpose of ELB is to enhance the fault tolerance of your applications. It is a single point of contact for clients to enhance the availability of their applications. Elastic Load Balancing offers high availability, robust security and automatic scaling to enhance the fault tolerance capability of your applications.

Amazon’s Application Load Balancing

Different Types of Elastic Load Balancer

The most common types of Elastic Load Balancer are given below:

Application Load Balancer – This balancer is best suited for load balancing of HTTP and HTTP’s traffic. Application Load Balancer routes traffic within Amazon Virtual Private Cloud based on the request of the content.

Network Load Balancer – Network Load Balancing is suited for balancing TCP traffic where extreme performance is required. It routes traffic to targets within Amazon Virtual Private Cloud. It is capable of handling millions of requests per second. Network Load Balancer can also optimize volatile traffic patterns.

Classic Load Balancer– Classic Load Balancer is used to provide basic load balancing across multiple Amazon EC2 instances. Classic Load Balancer is used for applications that were built within the EC2 classic network.

Elastic Load Balancing

Impressive Features of Amazon Application Load Balancer

HTTPS Support -An application load balancer is responsible for termination between the clients and the load balancer. It also offers management of SSL certificates through AWS and IAM for pre-defined security policies.

Server Name Indication – SNI is an extension to the TLS protocol by which a client connects to at the start of the TLS handshake. Application load balancer also supports a smart certificate selection algorithm, with SNI.

Higher Availability – An application load balancer is required to distribute incoming traffic across your targets in multiple availability zones. It automatically scales its request handling capacity in response to incoming application traffic.

Safety– Amazon Application load balancer is secure. You can create and manage security groups associated with elastic load balancing to provide additional networking and security options.

Health checks – An application load balancer routes traffic to healthy targets. You can get insight into the health of your application by configuring detailed error codes and analyzing each service on an EC2 instance.

Datadog application load balancer

Why is Monitoring Load Balancer Important?

It is extremely important to monitor your load balancer because it serves as the primary gateway for clients to your application. If not monitored, unexpected issues with your load balancer could result in a target being overwhelmed with requests. The different metrics of load balancer such as end-to-end application latency, error code responses, high-value metrics, etc can be prepared to combat issues coming your way. Customers can also see key-metrics such as healthy and unhealthy hosts along with a full breakdown of HTTP response codes.