Why is network visibility in the cloud more critical than ever?

In 2016, I attended the AWS re:Invent summit. It’s the biggest AWS event of the year, in Las Vegas, hosting more than 50K participants. I remember attending one of the sessions, and the speaker said something that stuck with me… “It’s no longer a question if an organization will migrate to the cloud, the question is why they are not already there.”

A lot has happened since 2016, and that sentence couldn’t be more relevant. COVID-19 had a huge impact on digital transformation, which accelerated cloud adoption. Enterprises continue to leverage Infrastructure-as-a-Service (IaaS) for the ease of application deployment and IT resource scalability. Additional cloud offerings, such as Platform-as-a-Service (PaaS) and Function-as-a-Service (FaaS), are enabling organizations to optimize services and applications for small functions. These can be considered as small-scale services, hosted standalone and running autonomously, which can be triggered by events.

One of the biggest challenges IT professionals experience when migrating their assets to the public cloud.

As IT professionals migrate their applications, services, assets, and networks to the cloud to gain the cloud benefits that include scalability, continuity, performance, and “pay-as-you-go” and “pay for only what is used” for cost optimization, they are losing one vital ability, which is visibility and control of their infrastructure.

Prior to the cloud era, everything was simple. IT managers had full network visibility and control of their network. Networks were centralized with traffic from remote users or branches aggregated and consolidated to the centralized location, which was usually in the organization’s HQ or data center where all applications and services were hosted and protected behind the security stack comprised of the firewall, IPS/IDS, NDR, anti-malware, DLP, and many others. The concept was that, as long as you’re in the corporate network, you’re protected. Network administrators had full control and awareness of the network ingress and egress traffic and branch connectivity over private links, such as MPLS/IP-VPN or SD-WAN, remote access to employees over VPNs, etc. The IT admin had full control over the network, prioritizing applications and implementing policies to achieve business objectives, productivity, QoE, and security.

In the past years, with the cloud adoption of Software-as-a-Service (SaaS) utilization, applications such as Office 365, salesforce, and many others, organizations are utilizing the public cloud to offload applications and services from private data centers to public clouds including Amazon Web Services (AWS), Microsoft Azure, and the Google Cloud Platform (GCP). IT managers face a new challenge, which they never encountered before – How to maintain adequate visibility in the public cloud, not just for security purposes and “the blind spots” due to usage of ad hoc service containers, but also for network monitoring, performance monitoring, and troubleshooting, all of which they had on their local networks?

There is no doubt that the public cloud introduces many advantages, and organizations understand this. They also understand the shared responsibility model with its pros and cons. If one wants to gain the public cloud advantages and lose some control over the network, then for some it will be a fair exchange. But, for others, this is unacceptable.

Public cloud visibility challenges and considerations

Limited control and blind spots for cloud data transfers

Migrating workloads and services to a public cloud (or to multiple public clouds) means losing many of the controls IT and security teams had in their on-premise deployments. Especially noteworthy is the loss of visibility into data transfers in the cloud, which has a significant financial impact on the organization. The public cloud’s shared infrastructure and responsibility means that IT managers need to trust the cloud providers for certain aspects of the shared infrastructure. Some of the controls weren’t even possible until a few years ago. Utilizing security and performance tools like TAP or vTAP in the cloud wasn’t even possible until AWS released virtual private cloud (VPC) traffic mirroring in 2019.

Limited security and performance monitoring

Even if the applications/services are fully hosted in the public cloud or hybrid cloud,  performance issues may occur and impact cloud platforms just as they would in the traditional on-premise environment.  Performance issues from overall availability, network latency, and application processing will impact user experience. Cloud providers furnish native tools, like logs and smart KPI metrics, for the ongoing operation of cloud assets, which I will explain in more detail further. These tools are just covering metadata, lacking contextual understanding of fault.  Deep analysis is required to locate the root cause of a security incident or performance issue. Without complete, context-based data analysis, it’s a very challenging task.

The main visibility tools

Public cloud providers also understand these challenges and provide native tools to enable IT managers to have visibility and control of their cloud assets. I will cover some of the main visibility tools of AWS, which have similar counterpart services found in Azure and GCP.

AWS Flow logs

Flow logs capture information about the IP traffic going to and from network interfaces in the VPC. Flow log data can be published to several AWS locations, such as Amazon CloudWatch Logs, Amazon S3, and others. Once a flow log is created, it can be retrieved and the flow log records in the log group, bucket, or delivery stream configured can be viewed.

Flow log advantages:

• Diagnosing overly restrictive security group rules
• Monitoring the traffic that is reaching the instance
• Determining the direction of the traffic to and from the network interfaces

AWS flow logs are a great starting point to have better visibility of the traffic traversing in your VPC. Eventually, a log is a log and, if an IT manager wants to gain contextual understanding of faults, performance issues, suspicious activity, data exfiltration, and much more, then they need to dig into the log and analyze it, which is complex and time consuming. Below is an example of flow log output. Now, try to understand, from the log, if you have any problems in your network.

AWS CloudWatch

Amazon CloudWatch monitors AWS resources and the applications running on AWS in real-time. CloudWatch can be used to collect and track metrics to measure resources and applications.

IT managers can create alarms that watch metrics and send notifications or automatically make changes to the resources being monitored when a threshold is breached. For example, an IT manager can monitor the CPU usage and disk reads and writes of the Amazon EC2 instances and then use that data to determine if additional instances should be launched to handle the increased load. IT managers can also use these data to stop under-used instances to save money.

With CloudWatch, IT managers gain system-wide visibility into resource utilization, application performance, and operational health.

IT managers can consider CloudWatch as an operation tool to monitor applications and cloud assets. Its main usage is for ongoing operational aspects of the organization’s cloud assets. But it still can’t provide rich visibility into data transfer traffic within the organization’s cloud instances and ingress/egress traffic. Below is an example showing CloudWatch dashboard matrices.

Addressing the problems and challenges faced by IT professionals

Both AWS Flow logs and CloudWatch are great native AWS tools to monitor the organization’s cloud assets. Flow logs provide a rich log of all IP traffic going to and from network interfaces in the VPC, and AWS CloudWatch provides operational metrics of applications. So, everything is OK, right? What’s the problem?

The problem or the challenge is that IT professionals need simple and contextual visibility of the traffic traversing in their VPC. It isn’t realistic to expect an IT manager to analyze flow logs should they get a complaint of a fault or during troubleshooting. They need an immediate understanding of their cloud transfer data, including classification of applications, protocols used, any machine-to-machine API communications, as well as east/west and north/south traffic visibility.

Allot Traffic Intelligence & Monitoring Tools are the trusted visibility and control solutions used by IT professionals throughout the world. For over 20 years, they have been providing rich visibility into networks. In order to achieve this rich visibility into networks on the public cloud, Allot solutions are deployed inline and integrated with the AWS transit gateway, allowing all ingress/egress traffic to be inspected, classified, and monitored. Additionally, these solutions can be deployed out-of-path monitoring the VPC by analyzing a copy of the data when using AWS VPC mirroring. In any deployment option, the outcome is clear, rich, contextual visibility of all data transferred on the cloud. IT managers get an instant view of the cloud network, bottlenecks, insights on data spikes or anomalies. They also get an understanding of which application, service, or API call caused the anomaly that, in some cases, contributes to an increase in data transfer costs and security considerations in case of data exfiltration. In short, the Allot Traffic Intelligence solution enables IT managers to gain back their control over the network, much as they had in the private data center.

For more information, the Allot Enterprise team and I created a Traffic Intelligence and Monitoring Tools portal on our website.