Secure Management Design Options

In the previous sections, you learned about the protocols and the broad categories of tool types that can help with network management. Much of that might have been review if you've been working in security management for some time. This section shows how the protocols and tools just described can be deployed on a security-sensitive network. Both secure and insecure options are provided. Your own security policies will help drive the type of secure management you wish to deploy. Throughout these designs, the medium network examples from Chapter 13, "Edge Security Design," and Chapter 14, "Campus Security Design," are used because they represent the majority of deployments today. The principles of these different designs can be easily applied to smaller or larger networks. Most organizations use some combination of the following: cleartext in-band, cryptographically secure in-band (session and application layer), cryptographically secure in-band (network layer), out-of-band (OOB), or hybrid management design.

Cleartext In-Band

The most insecure management option available today sadly is the management option used by the majority of organizations. All management takes place in-band, meaning the management traffic travels across the same logical links as the production traffic. This is contrasted with out-of-band (OOB), in which a separate logical, and sometimes physical, network is built exclusively for management traffic. Additionally, this management traffic is cleartext, so not only are passwords sent in the clear but so are configuration details and alarm data as they pass between the management systems and the managed devices.

Supported Platforms

Nearly all platforms support this form of management.

Multisite Considerations

The main consideration for managing multiple sites is the security of the data if it transits an untrusted network such as the Internet. For example, managing a remote router using Telnet across the Internet is not advisable.

Attack Mitigation

By using cleartext in-band management, the security requirements fall back to IP address filtering and the mitigation of sniffing. The following technologies are key in this management type:

• Ingress/egress/uRPF filtering RFC 2827 filtering should be implemented throughout your network per Chapter 6. This will limit an attacker's ability to spoof management traffic.
• Management channel filtering As shown in Chapter 5, most devices can limit access to the management channel through some form of filtering. Such filtering should identify a specific IP address or range of addresses that is able to access a particular management interface. Allowing your entire internal network is not advisable. Depending on the device and protocol, some management functions must be filtered by using traditional ACLs as opposed to some access control tied directly to the management daemon.
• Stateless/stateful management network filtering Although not practical for smaller networks, putting all management systems off of a dedicated subnet on your network allows you to filter access between this network and the rest of the devices. Ideally, this filtering is stateful, as provided by a dedicated firewall, but stateless filtering is still better than no filtering.
• Layer 2 best practices To lessen the likelihood of an attacker sniffing your management communications, the L2 best practices as identified in Chapter 6 should be implemented throughout your network as identified in Chapters 13 and 14.
• Host/application security The host-hardening and host security capabilities (HIDS, AV, and so on) should be strongly considered for all management systems. After all, the management network is often one of the most attractive targets for an attacker because from there the attacker can operate from a position of relative trust with the rest of the network.

Best Deployment Practices

Figure 16-1 shows a typical example of cleartext in-band management. A firewall is shown as optional, though some form of L3 filtering should be required if not implemented at the firewall. The filtering should be configured to allow only designated management traffic into the management network (Syslog, SNMP traps, TFTP, and so on). Likewise, the same restrictions should apply from the management network outbound. Traffic can be restricted to required protocols (Telnet, SNMP, HTTP, and so on). Keep in mind that, without a stateful firewall, filtering must be very porous to ensure that all protocols can work properly (see the TFTP example earlier in this chapter).

Figure 16-1. Cleartext In-Band Management Design

To the extent possible, all management and certainly all cleartext management should occur from the management network. If someone in IT needs access to a system, the best option is to have that individual SSH to a host on the management network and then access the system from there. Otherwise, the management channel filtering described earlier must be too porous.

Some managed devices reside in less and less trusted environments. This increases the risk of using cleartext protocols in-band. Most organizations deal with this risk by running fewer management protocols on edge devices when compared to the campus. This is a good option to a point. If you disable too many management functions, though, as described earlier, your network can become unmanageable and security can suffer instead of be improved.

NOTE

A AAA server could be considered part of the management network or the production network, depending on its use. Chapter 9, "Identity Design Considerations," recommends that large networks adopt a separate AAA server for management access. This server should be housed on the management network. Even if a dual-purpose AAA server is used, it can still be housed on the management network for greater security.

Cryptographically Secure In-Band (Session and Application Layer)

This option is different than the previous design option only in that the management communications are cryptographically secure using either session or application layer crypto.

Supported Platforms

More and more platforms support some form of secure management as identified earlier. Many Cisco routers, switches, and security appliances, for example, support SSH. Others support HTTPS. Most modern OSs support some form of secure configuration through SSH or some other protocol.

The downside to this form of management is that certain protocols that you might need for management do not yet have secure versions. Syslog, described earlier, is still UDP based and cleartext, as is TFTP and all versions of SNMP prior to v3.

Multisite Considerations

There are no unique multisite considerations. If you are using a secure management protocol, you can easily manage remote devices at another site.

Attack Mitigation

All the same attack mitigation techniques described in the cleartext in-band management section apply here, only with slightly less criticality. For example, if you are using SSH properly, the threat posed by traffic sniffing is eliminated. Likewise, if you are using SSH, limiting management channel access to specific IP address is still useful, but your security is by no means totally compromised if it is configured in a more permissive way.

Best Deployment Practices

Although this option and the previous design are separated for the purposes of this chapter, they can be considered two parts of the same design in your network. Use secure management protocols where you can and cleartext management protocols where you must. Figure 16-2 shows both designs merged together. Secure management protocols where a client talks to the managed device (such as SSH) are allowed from the entire campus network, but insecure management protocols are sent through the firewall between the sender and recipient.

Figure 16-2. In-Band Management Design (Cleartext and Secure)

Arrows in this diagram denote the "management listener" in the communication (SSH daemon, Syslog daemon, and so on). For example, although an IT administrator in the user subnet can SSH to a device to make configuration changes, the administrator should not run a Syslog server that accepts traffic originated by the managed devices. Traffic to the management network can go in both directions. Although a firewall is still listed as optional here, if you are taking the time to implement secure management on many of your devices, a firewall will greatly aid in limiting the scope of attack on the cleartext protocols and the management servers. As mentioned earlier, RFC 2827 filtering is still critical.

Ideally, all management will still come from the management network where more robust tools might be available to do change control and configuration rollback. Using managment tools over secure channels from the user domain should be used for quick troubleshooting or minor changes by administrators when necessary.

Cryptographically Secure In-Band (Network Layer)

As discussed in the previous section, sometimes it is not possible to encrypt all the types of traffic you might want to use for management. Either a secure option is not available (as in Syslog) or a management tool you need does not yet support the secure alternative (as in SNMP). In these cases, an IPsec tunnel (Chapter 10) can be built between the managed device and the management host or, better yet, between the managed device and the management firewall. Figure 16-3 shows the two options.

Figure 16-3. IPsec Management Tunneling Options

In option 1, the firewall simply passes IPsec traffic between the management host and the managed device. Option 2 allows the management host to not be aware of the IPsec process but just to attempt to communicate directly with the managed devices. The management firewall handles the encrypt and decrypt functions on behalf of the management network. Not only does option 2 ease configuration requirements on the management hosts, it also allows one IPsec tunnel from the managed device potentially to carry traffic to and from several different management hosts.

Supported Platforms

In principle, any device that supports IPsec should be able to implement this form of management. Be sure to review the IPsec considerations in Chapter 10. In practice, using IPsec in this way is not a mainstream use of the technology. There is no reason it shouldn't work, just be aware that this is less common. I've personally tested this with Cisco IOS routers and it works fine, but the configuration isn't trivial. The crypto ACLs can be easily set at the IP layer as the following abbreviated configuration shows (from the managed device's perspective):

crypto map crypto-map-name 100 ipsec-isakmp set peer mgmnt-fw-ip-addr set transform-set transform-name match address 120 access-list 120 permit ip host managed-device-ip management-subnet-ip-addr management-subnet-mask

After setting the L3 IPsec filtering, it is useful to set up ACLs to limit the types of management traffic that can go in either direction. This could also be done with L4 specifics in the crypto ACL. You also should be sure that the management traffic you are generating is using the IP address configured in the crypto map and not some other IP address on the device. For example, Syslog traffic can be sourced from a particular interface on Cisco routers by typing the command logging source-interface interface-name.

Finally, it is important to bear in mind the performance impact the IPsec process will have. It is by no means trivial to encrypt all management traffic on a low-power network device or end host.

Multisite Considerations

Using this option with remote sites is easy, and you might even be able to take advantage of existing IPsec tunnels between VPN-connected sites.

One alternative is to use IPsec tunnels to encrypt the traffic between management networks at different major sites. Traffic can be sent from the remote sites to the central sites over the management network as needed, and local devices can be managed from local management devices using the in-band option described previously. Figure 16-4 shows this option.

Figure 16-4. Multisite IPsec-Connected Management Networks

If you are already using IPsec between sites, you can try to take advantage of those tunnels rather than build a separate set just for management. Oftentimes, though, it is easier to do so on the management firewalls directly and have the traffic encrypted twice as it transits to a remote site. This lets the management traffic not care whether the remote site is reachable by leased line, frame relay, or IPsec VPN. Whether to take advantage of existing tunnels is a risk analysis decision (see Chapter 2, "Security Policy and Operations Life Cycle"). Just be aware that if the network designed to facilitate the management of systems itself needs a lot of management, you are asking for trouble. Building large, full-mesh IPsec networks just for management is probably overkill.

Attack Mitigation

The same attack mitigation techniques exist here as in any of the management designs. Refer to the list presented previously in the cleartext management section. The difference is that all management traffic is protected against sniffing, modification, and so forth at the network level.

Firewalls in front of the management network should be required in this case to adequately protect the management network from attack. If someone compromises a remote device, you don't want that person to have direct access to the management network over an IPsec tunnel. A firewall can enforce your policy to ensure that only necessary management protocols are permitted and only in certain directions.

TIP

A management firewall in the case of the network encrypted management design is one of the situations in which a firewall integrated with IPsec is beneficial. Using separate devices to encrypt the management traffic and filter it is unnecessary.

Best Deployment Practices

This option should not be used for all devices because it will be too difficult to manage. Instead, use it in high-risk environments where you need management protocols that you can't secure in other ways. For example, if you need TFTP, Syslog, and SNMPv2c on your Internet edge router, using an IPsec tunnel allows these protocols to transit the untrusted Internet edge back to the management network in a secure way. Figure 16-5 shows the in-band design with the IPsec option used to manage a single device on the edge network.

Figure 16-5. In-Band Management (Cleartext and Secure [Network and SessionApplication])

Out of Band (OOB)

One of the most secure ways to route your management traffic is to send it on a separate logical or physical network. This alleviates many of the security issues found in management networks but creates a few new issues that need attention. Figure 16-6 shows the classic OOB management model. Production traffic flows on production links, and management traffic uses a separate interface.

Figure 16-6. Basic OOB Management

NOTE

The designs discussed here should not be confused with "console" access to a device using a terminal server, modem, or serial cable. Console access is necessary for fault recovery and other needs but is limited in function. The OOB access described in this chapter is L3 and can support all management functions. Console access is not suitable for Syslog, SNMP, FTP, and so on.

OOB is common in specific applications in the network. NIDS appliances, because of the security risks, often separate management and attack detection into two different interfaces. This management interface is typically on a separate management VLAN or separate physical network. Likewise, managing L2 switches on the Internet edge is a good application for OOB access.

Using OOB management to manage an entire site's network or multiple sites is far less common. The principal reason this hasn't been considered is the trust-level disparity among the managed devices. For example, in Figure 16-6, consider what would happen if the two managed devices were in completely different trust domains. If an attacker were somehow able to compromise one device, the attacker could use the management network to attack another device. This traditionally could be mitigated by using dedicated routed interfaces to each management device with basic filtering or firewalls between subnets. This approach doesn't scale, though.

The advent of private VLANs (PVLANs) changes the way this can be done. You might want to reacquaint yourself with PVLAN capabilities (see Chapter 6) if it isn't fresh in your mind. Each managed device can be configured as an isolated port with only the management hosts as promiscuous ports. This prevents the management channel from being used to allow one managed device to attack another through the management network. This still leaves the management hosts vulnerable to attack, but it can be mitigated with a firewall between the managed devices and the management network. This design is shown in Figure 16-7.

Figure 16-7. Out-of-Band Management with PVLANs and Firewall

The main benefits of an OOB management design are as follows:

• Production traffic is not impacted by management traffic (and vice versa).
• An attacker on the production network, or accessing the production network, has no ability to access the management network without first compromising a device that is managed OOB, and even then, the only directly reachable IP on the OOB network is the firewall interface.
• Because of this lack of attacker access, insecure management protocols can be used on the OOB network with a much higher degree of assurance.

Although the OOB design is as secure a management network as I can think of, there are certainly downsides that prevent its comprehensive use for most networks:

• An additional interface is required on every managed device (see the next section).
• There is significant added cost in building out an OOB network.
• Not all devices support OOB management (systems that are interface constrained such as WLAN APs, for example).
• A compromise or implementation flaw of PVLANs breaks down the security of the OOB network.
• Topology-sensitive network management systems cannot run OOB because all systems would look directly reachable. Faults in the production network might never be detected.
• Multisite OOB management is problematic (more on this later).
• Managing routing protocols to prevent OOB network advertisements on the production network is required (more detail later).

Supported Platforms

To support OOB management, a device must have an interface that can be dedicated to management traffic. Most firewalls, switches, routers, NIDS, and VPN gateways support such connectivity. Servers can also easily be configured to use an additional network interface card (NIC). The cost of this additional interface will vary widely. Often a network device at a minimum will have two interfaces. Adding the third can significantly increase the cost of the device.

There is one main consideration regarding OOB management and Cisco L2 Ethernet switches. When managing an L2 switch, there is only one interface that is identified as the management interface. This interface has an IP address and a default gateway but does not route. Because there can be only one of these interfaces on an L2 switch, all of the elements the L2 switch must reach must be accessible from that one interface. A AAA server, for example, is often connected to the production network, but if the L2 switch must make use of it, the server will need to be moved, dual-homed, or replicated on the OOB network.

Multisite Considerations

Only the most well-funded and paranoid organizations are able to maintain the OOB management model across several sites. For most networks, building a separate physical or logical WAN just for management is too much.

In some limited situations, you might want to implement multisite OOB management. Say, for example, you have two primary connections to the Internet in different geographic areas. The management concerns around the Internet edge are the same in both cases, and after a thorough risk assessment, you decide on OOB management for both sites. You would prefer to have only one operations center for both locations. In this case, a topology similar to the one in Figure 16-4 would work well. The only difference would be using OOB management for the devices in each location and then securing the communication between the two OOB networks with IPsec over the production network.

This topology could be repeated for multiple networks. The main requirement is a connection between the OOB management network and the production network. Because not all management functions can be OOB, this will be desirable anyway, as you will learn later in this section.

Threats and Attack Mitigation

The technologies to mitigate the threats in an OOB network are slightly adjusted from the previous designs. In addition to the techniques described for the other management designs, a new concern must be dealt with: keeping the OOB network and the management network separate. There are several ways to achieve this:

• Separate address space By using separate address space for the management network, you can make it easy to filter production traffic from the management network and vice versa. An easy way to do this is to use dedicated RFC 1918 private addresses for the OOB network.
• Private VLANs Use private VLANs to stop traffic from one managed device from getting to another over the OOB network.
• Aggressively filter between OOB and production By having the OOB network on separate address space, you can easily set up filters at all choke points to block any traffic coming from or going to the OOB network.
• Use distribution lists to prevent route propagation To prevent the OOB network from being advertised or accepted by any other device, configure outbound and inbound distribution lists on all devices running routing protocols that are managed OOB.
• Filter within the OOB network Each OOB-managed device should have an inbound and an outbound ACL on the OOB-managed interface. These filters should allow only the specific management communications necessary between the management network and the managed devices. In addition, the management firewall should filter traffic to specific management flows.

As an example of managed host filtering, the following is the filtering that might be required on a Cisco IOS router managed by OOB. The 172.16.1.0/24 network is used for the managed device's OOB interfaces, and the 172.16.128.0/24 network is used for the management hosts. The topology is shown in Figure 16-8.

! Sample ACLs for R1 ! Permit ICMP on the management network (you could be more restrictive if desired) access-list 101 permit icmp any any ! Permit established TCP session from the management network to the router !(TACACS+ responses). access-list 101 permit tcp 172.16.128.0 0.0.0.255 host 172.16.1.25 established ! Permit UDP high ports, necessary for TFTP (see earlier in this chapter) access-list 101 permit udp 172.16.128.0 0.0.0.255 host 172.16.1.25 gt 1023 ! Permit SSH access to the router (telnet could be permitted as well if desired) access-list 101 permit tcp 172.16.128.0 0.0.0.255 host 172.16.1.25 eq 22 ! Permit SNMP and TFTP requests from specific hosts within the management network access-list 101 permit udp host 172.16.128.107 host 172.16.1.25 eq snmp access-list 101 permit udp host 172.16.128.110 host 172.16.1.25 eq tftp ! Permit NTP responses from the NTP server in the management network access-list 101 permit udp host 172.16.128.99 host 172.16.1.25 eq ntp access-list 101 deny ip any any log ! As you learned earlier in the book, ACLs on a router do not apply to ! traffic originated by the router. This means you only need one entry ! in the outbound ACL denying all traffic. This will stop traffic on ! the production network from being routed to the management network. access-list 102 deny ip any any log ! interface FastEthernet0/0 ip address 172.16.1.25 255.255.255.0 ip access-group 101 in ip access-group 102 out

Figure 16-8. Out-of-Band Management Example

Best Deployment Uses

OOB management is best used in high-risk networks where insecure management protocols are essential. Internet edge designs are a good place to consider OOB. Within a campus, OOB can be used, but its costs should be considered in comparison to the risk associated with the internal network.

Some organizations save money on OOB networks by using a logical separation as opposed to a physical separation. By using a VLAN dedicated to management, the traffic can be separated back to the management network without requiring a separate infrastructure. I would be comfortable with this option in small networks and contained deployments. If you have a large OOB network, the impact on the stability of your network if you bridge management traffic across a number of switches is significant. Spanning tree issues become a big concern as with any large L2 network.

I've seen logical separation work well within a given trust domain, which then moves traffic to a physically separate backhaul to connect to the management network. This creates small pockets of managed devices in a single logical OOB network and then links them together to form a larger OOB network.

In most cases, even when some of your devices are OOB managed, you will also need in-band management. This option is discussed in the next section.

Hybrid Management Design

Most networks do not have the explicit goal of using only one type of management infrastructure. The overall goal is to make management as secure as possible, given the realities of the protocols used, the capabilities of the devices, and the layout of the network. In keeping with that, most networks use some combination of the management techniques outlined so far, depending on the requirements. Sessionapplication layer crypto is the most appropriate for all management functions because it is the most flexible without sacrificing security. Cleartext management protocols should be avoided when possible, but if they are absolutely necessary, use one of the following options:

• If there is a low risk of attack, use filtering and L2 best practices as identified in the cleartext management design section.
• If the risk is high, use either IPsec tunnels or OOB management. Keep in mind that OOB management has other benefits besides protecting cleartext protocols.

Ideally, you want to do this with a single management network for all communications. Figure 16-9 shows all the management types merged into a single design.

Figure 16-9. Hybrid Management Design

Here you can see some cleartext management from the management network, secure management from the management network and the user network, IPsec-tunneled management through the management firewall, and OOB management for the NIDS appliances. This will work fine; the main consideration is the IP ranges you use. Using a separate IP range allows the production devices to filter any OOB traffic very easily. However, you might need the same server in the management network to do both in-band and OOB management.

If you use a separate IP range (as discussed in the OOB section), the in-band management will likely be filtered on the production network. The solution, I'm sorry to say, is Network Address Translation (NAT). You can choose to translate the management network traffic either before it gets to the production network or before it gets to the OOB network. Make your selection based on which management function you use most often. If most of your network is OOB managed, NAT for your in-band management is preferred. If most of your management traffic is in-band, do the opposite.

This allows you a high degree of flexibility. You can manage devices in any way you choose, on any part of the network. The firewall can be used to build IPsec tunnels to other management networks as required or to allow remote management systems inbound.

Secure Network Management Optional Components

Depending on your level of risk, NIDS might be appropriate inside your management network. I recommend this for most customers with large management environments and a large management staff to go along with it. After all, the management network is often the most trusted location in a network. It has privileged access to both network event data and configurations on network devices. If an attacker found a way into your management network, serious harm could be done.

Another option is more of a convenience for operators. If you want to give remote administrators access to the management environment directly, you can do one of two things. Option 1, described earlier, has SSH permitted from the campus network into a specific management host, which can then allow the administrator to manage the network from the management subnet. Option 2 is to install a remote user VPN gateway into the management network with a connection on the Internet edge (in the same area as the rest of your VPN traffic; see Chapter 14). This allows a remote administrator to have a machine with an IP address on the management subnet.

WARNING

Be sure to configure remote management access carefully because you don't want to create a huge backdoor for attackers. NIDS, firewalls, and any other security best practices you've already learned about should be considered if you are providing this kind of connection. Because the user community that will access the network is small, you might consider digital certificates as well.

Figure 16-10 shows the hybrid design with both NIDS and remote user VPN added.

Figure 16-10. Hybrid Management Design with NIDS and Remote VPN Access