Purpose

This document describes in detail how patches are chosen, tested and deployed by Unix Operations

Patch Teams

There will be 3 groups within the patch team: the patch selection committee, the patch planning team and the patch execution team.

The patch selection committee will be charged with the bundle selection, testing and documenting the procedures.  The patch selection committee will consist of 2 to 3 members in the US Unix ops team. 

There will be a patch selection committee for both Solaris and Linux.  The committees are responsible for clear and accurate procedures being well documented and handed off to the patch execution team. 

The Patch planning team will be the US Application / System owners and they will be in charge of planning and scheduling the patches.  There will be 2 US Unix admins per Application / System to complete this, and they will work closely with their assigned SME team(s).

The execution teams will be 3 admins per Application / System group from India.  They will be responsible for following the timings and plans laid out by the other 2 teams.

Bundle Selection

Patch bundles will be downloaded on the first and 15^th of each month up to 3 months prior to the patch rollout (or the first business day after if this is on a weekend) and kept for consideration in choosing a patch bundle in the next round of patching.

Patch bundles downloaded by engineering for build processes will be included in the selection process as they are what engineering will be releasing.

The bundle selection will be completed by the UNIX Ops patch selection committee and will be completed a minimum of 1 month prior to the next patching cycle.

The following criteria are used in selecting a Solaris patch bundle.

• The bundle must be older than one month old and have a proven track record as described in the next two qualifications.
• It must not include recalled patches
• The kernel patch must not have been updated multiple times shortly before or after the downloaded release.
• Other considerations that are deemed important or necessary by the selection committee

              Linux patching management is addressed by the following areas:

• Consolidate patches advised by Red Hat network, including security, bug fix, and enhancement rpms.
• Evaluate, test, and combine patches from non-Redhat third parties such as hardware and java vendors.
• Maintain and manage repositories or channels that can handle gigabytes of patches on different versions of Linux system
• Other considerations that are deemed important or necessary by the selection committee

Testing

The testing will be completed and documented a minimum of 1 week before the start of the next patching cycle.

The bundle to be installed will be tested and the following will be checked and documented by the US planning team.

• What configuration files have been updated and do they need to be merged or replaced with the old config files?
• How much space does the installed bundle take?
• Is backing out the patches easy or difficult?  What is/are the process(es)?
• Are there any recently recalled patches
• Are there any anomalies from the OS that we need to be aware of or fix before the patch is released?
• Any other determinates that the committee deems necessary.  These additions will be documented in the hand off to the execution team.  Any permanent additions will be updated in this document.

Planning

Patching Cycles

All execution plans MUST take into account the Change freeze periods.  Do NOT try and schedule production servers within the freeze period.

Each bundle will have an effective life of 7 months starting from the initial test installation.

The schedule of these should be on going.  Each Application / System Owner owner should work out with their Application / System’s a schedule for these to occur during these months.  The change control tickets should be put in then as well, and the schedule will be made available to the DBA’s for resource allocation.  The July – December schedule should be in place before the patching starts.  The work to plan the next set should start once the patching effort is under way.

Java related patches will need to be tested and verified by the AppServer Services team.  Coordination and resource request will need to be planed and sent to the AppServer Services team at least a week ahead of the planned execution date.

Execution

In order to keep us relatively up to date and not fall behind and to keep sox compliant , a strict timeline must be kept. 

Patching of the previous bundle will conclude on June 15^th and December 15^th or the business day preceding it if it lands on a weekend.

All dev boxes must be patched by the end of August/February.

All QA systems must be patched by the end of October / April.  During this patching period, all SOX approvals for production (Including UAT approval) should be obtained and attached in the ticket using Change Control standards.

All Prod systems must be patched by December 15^th/June 15^th.