With systems, this all is quite easy as there are almost no restrictions (besides compatibility), but network elements like routers or switches have limited resources and updates are not broken down to smaller components, but are usually just one big file containing everything from kernel to all the "applications" or processes and supporting utilities.
In this post, I would try to describe the process of planning such upgrade.
Inventory phase
First step is to find out what needs to be upgraded, as in later phases this information is quite important for selecting the software version as well as the process of the upgrade.
Following information needs to be collected about each router/switch/firewal/etc.:
- Hardware type and version (not just the type printed on the device chassis, but also slot/port count information. For example "Cisco Nexus 56128P Switch" or Catalyst 2950T 24 Switch)
- Memory information (RAM as well as storage or flash memory size is important)
- Management IP (or IP address via which the software is going to be uploaded, as pcmcia or console modem options are usually not very fast)
Memory information is needed to find out if the software can run in the memory the hardware provides as well as if the software can be stored on the memory provided. Some devices have only one memory and split it in the two types (like Cisco 7200 routers) , while others have dedicated storage memory and RAM.
Software selection phase
After having all the information from previous phase, we can move on to finding the appropriate software version, that each device can support and contains the fixes needed.
Each software vendor has at least one web-tool that provides the information (or even the software download) needed:
Sometimes the vendors have links directly from security advisories or notifications, but it's not necessarily there, so the safest way to get the software and information about it is via the download pages.
Some vendor make it easy to select the latest version, while others have a set of sub-versions indicating feature upgrades or just patches; standard or extended support; early deployment or limited deployment; etc. Each vendor has a document describing what each part of the version means, and it can also be different for each product series.
Besides having the choice of software to download, there's also release notes or readme document for each version, where the vendor describes:
- how to perform the upgrade
- what are the pre-requisites (which platforms and current software versions are compatible)
- what new features are introduced and old ones removed
- what issues/bugs/problems were resolved with that software version
- what caveats were identified with this version
If the current version is way too old (by 1 or several major version releases), it might be needed to perform several upgrades in order to ensure that configuration is properly translated to new syntax or with new features. This should be described in the pre-requisites in order to ensure trouble-free upgrade. This phase has to be repeated for each of the versions that need to be installed before the latest one can be applied.
With constant change and improvements in the network field, features come and go, so it's necessary to watch out for removal or modification of features used (default deny could change to allow any; or statically configured IPSEC local networks might be auto-negotiated in newer version).
List of bugs resolved is a good source for identifying whether the new version would fix the recent vulnerabilities flowing in the wild. This might help with vulnerability management tickets or anomaly reports that are overdue.
And the caveats are good to know problems that were identified during vendor testing of the new version. When the local conditions are similar as those described in caveat, this might put a stop to the installation of that version (or the upgrade).
Software validation
With all the information collected from previous phases, only very brave people would install the software right away into production.A lot of companies have labs, where new versions can be tested before installing them into production. In larger data-centers there could be canary elements for testing, where this could be done.
Goal of validation should be to ensure:
- current configuration syntax is fine under new version
- all required features are going to work as expected (with the same licences)
- redundancy mechanisms would work (no timer defaults or protocol defaults changed)
- monitoring functions get the same format of data as before (no snmp OID or syslog message format or API changes)
- migration/upgrade plan is not going to cause an impact (some systems require same version of clustered elements to work)
Whether all this is automated or done manually by verification team with defined validation test-cases, it's up to each company to decide, but what most of the IT managers wouldn't like is to have total outage of core network after software upgrade of central router or switch.
And let's not forget to verify the hash of the downloaded software (if the vendor offers it on the download website), as network elements are the best place for MiTM attacks.
If you know of anything else I missed, let me know and I'll update the post.
