files: 72

This data as json

rowid	image	timestamp	memento	first_capture	last_capture	current_status	text	mime	status	url	urlkey	digest	length	file_path
72		20120507040004	https://web.archive.org/web/20120507040004/http://www.defence.gov.au/dgta/Documents/DAVENG/Software%20Symposium%20documents/2008/Presentations/Failure%20to%20Transition%20(Feodoroff).ppt	2012-05-07	2012-05-07	404	Page 1 BAE SYSTEMS IN CONFIDENCE Failure to Transition Continued Airworthiness in the context of Software Maintenance and Support of AP-3C Weapon System Page 2 BAE SYSTEMS IN CONFIDENCE Topics • Introduction • Revision of previous analysis on DMO Category of Support requirements • Revision on Technical Frameworks for SM&S • DT&E and IT&E across the Transitional boundary • Supportability analysis of Software, SAE Supportability Concepts et al • Program v Project Risk Management – who pays, who is left holding the bag Page 3 BAE SYSTEMS IN CONFIDENCE Introduction Page 4 BAE SYSTEMS IN CONFIDENCE Introduction • The Contractor Support Facility (CSF) supplying Software Maintenance and Support (SM&S) services for the AP-3C Weapon System is housed in the Integration, Test and Training Facility (ITTF) at RAAF Edinburgh. • The 3rd Party Non-Associative Software Support Agency (SSA) working within the ITTF is utilising capability delivered into the CSF under Project AIR 5276 to provide Indigenous Support of the AP-3C Weapon System. • The Support Concept out of Project AIR 5276 included requirements for DMO Category Support Level 3 which is described by the AEO for the SSA as: – Fault Rectifications, and – Minor Enhancements • Project AIR 5276 was pre-TAMM which now pushes Continued Airworthiness requirements onto SM&S activities. Some latitude is taken to interpret this also as Mission Worthiness and Business Worthiness as the SSA has to deal with Mission Criticality and Business Criticality in addition to Technical Airworthiness (X-Worthiness). Page 5 BAE SYSTEMS IN CONFIDENCE Revision of previous analysis on DMO Category of Support requirements Page 6 BAE SYSTEMS IN CONFIDENCE DMO Category of Support requirements overview • The Category of Support (CATSUP) is a DMO vehicle for specifying, at a high level, the degree of capability and the extent of the capacity required to provide software support. • Four categories [1..4]. • Main Capability themes of the Category of Support model are: – Control and maintain configuration; – Validate installation; – Provide data; – Provide capability to modify software; – Provide capability to analyse, integrate and test modifications to software; and – Monitor or predict performance. Page 7 BAE SYSTEMS IN CONFIDENCE Themes allocated per Category of Support level • The inauguration of capability maps to the CATSUP levels in the following manner: – Category 1 • A) Control and maintain configuration • B) Validate Installation • C) Provide data – Category 2 • D) Provide capability to modify software • E) Provide capability to analyse, integrate and test modifications to software – Category 3 • F) Monitor Performance – Category 4 • G) Predict Performance Page 8 BAE SYSTEMS IN CONFIDENCE Example aggregations of themes under CATSUP levels • For example Category 2 will provide the following: – A) Control and maintain configuration (from CATSUP 1) – B) Validate Installation (from CATSUP 1) – C) Provide data (from CATSUP 1) – D) Provide capability to modify software – E) Provide capability to analyse, integrate and test modifications to software • While Category 3 would provide the following – Category 3 • A) Control and maintain configuration (from CATSUP 1) • B) Validate Installation (from CATSUP 1) • C) Provide data (from CATSUP 1) • D) Provide capability to modify software (from CATSUP 2) • E) Provide capability to analyse, integrate and test modifications to software (from CATSUP 2) • F) Monitor Performance Page 9 BAE SYSTEMS IN CONFIDENCE CATSUP provided in CSF by CoA • The table above captures the exec summary of coarse audit of SM&S capability owned by the CoA within the CSF. • The most notable omission is Column F or Monitor Performance (Column G is not relevant as the CSF was supposedly targeting CATSUP 3). This is a relatively large omission given it is the core capability that distinguishes CATSUP 3 from CATSUP 2. • Note the impact of the lack of a Performance Monitoring framework may be a Capability Capping of the CSF to CATSUP 2 which is colloquially Fault Corrections rather than Minor Enhancements. Mission System CATSUP 1 2 3 4 A B C D E F G Intercommunications Subsystem    See later slides   Navigation Subsystem      Radar Subsystem      Acoustic Processing Subsystem      Acoustic Trainer Subsystem      Data Management Subsystem      Page 10 BAE SYSTEMS IN CONFIDENCE Capability verses Ability • To tackle the confusion that reigns between Capability and Ability: – The SSA has the Ability to do, say, performance monitoring because engineers can (through best endeavours) exercise their engineering prowess. They have the Ability to use the equipment to carry out performance monitoring because it can run software, they can instrument software, they can collate data, they can analyse data. – The Performance Monitoring Capability would have been infrastructure, plans, tools, data, issue management etc; delivered into the CSF as part of the technical frameworks. • The scope of Technical Framework artefacts will include the following artefacts: – Planning artefacts. – Specification artefacts. – Construction artefacts. – Execution artefacts. – Analysis artefacts. – Reporting artefacts. – Problem resolution artefacts. • No Technical Framework resembling system or sub-system level Performance Monitoring exists with the SM&S capability owned by the CoA nor has the SSA been tasked to develop such a Capability. Some tools exist and there are reporting artefacts out of Project AIR 5276 coffers but no substantive framework to support impact assessment to the system or software architecture of modifications. Page 11 BAE SYSTEMS IN CONFIDENCE Circular Warrantee problem • The Circular Warrantee problem arises when CoA requires the SSA to warrant delivery of CATSUP 3 services utilising Capability supplied CoA owned CSF. – The problem for the SSA arises when the CoA warrantee does not stand up to close scrutiny, or the CoA do not warrant the GFM. – Note, GFI is not literally warranted but is bundled under the implied warrantee that the CSF is fit for purpose. • Using the effects of Capability Capping (and the missing Performance Monitoring framework as an example) there are risks to Continued Airworthiness/Mission Worthiness/Business Worthiness (X-worthiness) if modifications to the Weapon System affect software or system architecture. • Continued X-worthiness may be defended by capping modifications to Minor Enhancements or below (untenable); or tasks may be required to redeem Capability (requires non-Operational funding stream); or terminate Indigenous Support and pass back to OEM (NAV, ACO, RADAR). Page 12 BAE SYSTEMS IN CONFIDENCE TAMM Category of Support requirements in comparison • Rectification may be interpreted as a deliberate or an accidental capping of Development through capping of Capability. • Note absences of “Full design disclosure and IPR” clause against Rectification. Rectification. Capability to analyse and rectify faults. The necessary tools to make a software change shall be included, however, testing and other related activities may require the direct use of aircraft. Development. A full development and test capability similar in scope to the initial development environment, that can produce significant software changes to the system. The capability to do off-aircraft DT&E and provide simulated stimulus for testing purposes shall be provided. Full design disclosure and IPR shall also be provided. Enhancement. A full development environment with additional simulation, modelling, and analysis tools that would allow significant enhancement of system operation. The development and test environment shall allow analysis of actual system performance using appropriate tools and techniques. The testing facilities shall enable environment testing of real time performance where appropriate to the systems. Full design disclosure and IPR shall also be provided. Page 13 BAE SYSTEMS IN CONFIDENCE Revision on Technical Frameworks for SM&S Page 14 BAE SYSTEMS IN CONFIDENCE Scope covered by Technical Frameworks • The scope of AEO for the AP-3C Weapon System SSA currently is the following (with Resultant Change Capability and CATSUP mapping): – Fault Correction ( ≡ Rectification or CATSUP 2) – Minor Enhancement ( ≡ Development or CATSUP 3) – Integration Support ( ≡ Rectification or CATSUP 2) • Note no Enhancement or CATSUP 4 • The technical frameworks required to support this scope are the following: – System Software Qualification Framework – Software Integration Test Framework – CSCI Test Framework – Unit Test Framework – Software Construction Framework Page 15 BAE SYSTEMS IN CONFIDENCE Goals of Technical Frameworks • Recapping: the scope of technical framework artefacts to support these goals will include the following artefacts at each level: – Planning artefacts. – Specification artefacts. – Construction artefacts. – Execution artefacts. – Analysis artefacts. – Reporting artefacts. – Problem resolution artefacts. Technical Framework Components Rectification (Integration Support and Fault Correction) Goals Development (Minor Enhancement) Goals System Software Qualification Framework REPRODUCTION Problem reproduced at System Interface? QUALIFICATION Did we build the right thing? Software Integration Test Framework LOCALISATION Offending CSCI discovered? ASSEMBLY Did the system components go together correctly? CSCI Test Framework CHARACTERISATION Behaviour of offending CSCI determined? CHECKOUT Is the component sound? Unit Test Framework INSTRUMENTATION What does it look like under the microscope? VERIFICATION Did we build it right? Software Construction Framework RESOLUTION Construct solution! RESOLUTION Construct solution! Page 16 BAE SYSTEMS IN CONFIDENCE Exec Summary of coarse audit of Technical Frameworks at CSF AP3C Weapon System - Mission System A B C D E Intercommunications Subsystem  SEL    Navigation Subsystem  SEL [1]   Radar Subsystem  SEL RSDL [2]  Acoustic Processing Subsystem  SEL [3]   Acoustic Trainer Subsystem  SEL [4]   Data Management Subsystem  SEL  [5]  KEY: A) System Software Qualification Framework B) Software Integration Test Framework C) CSCI Test Framework D) Unit Test Framework E) Software Construction Framework SEL) Provides Ability, dearth of Technical Framework artefacts related to Software Integration Testing from PA5276 NOTES: [1] SDE with CSF has less capability than full SDE within OEM. [2] RSDL can be utilised for CSCI and UNIT testing support. [3] OEM utilised separate hardware rigs for CSCI and UNIT testing that were not delivered. The mission box can be utilised for these activities but with restricted access because of SEL resource utilisation clashes and also with reduced throughput due to restrictions in interfaces with box internals. Hardware obsolescence has also broken the capability to connect the mission system to a high speed LAN. [4] OEM utilised separate hardware rigs for CSCI and UNIT testing that were not delivered. The mission box can be utilised for these activities but with restricted access because of SEL resource utilisation clashes and also with reduced throughput due to restrictions in interfaces with box internals. Hardware obsolescence has also broken the capability to connect the mission system to a high speed LAN. Page 17 BAE SYSTEMS IN CONFIDENCE High level Technical Framework discussion System Software Qualification Framework Using the AP-3C as the example, the experience is that system software qualification frameworks are part of sell off to customer so are generally available. The delivered soft copy test procedures are often out of sync with the as-run test masters but the data is generally available – though not as ‘re-useable’ an asset as some would proffer. Regardless, the high level qualification data is available. Software Integration Test Framework The ITTF SEL is the software integration test facility; however, very few of the other technical framework artefacts related to a software integration test framework has been delivered. The SEL was delivered “assembled” and so much of the data reflects the destination rather than the journey. CSCI Test Framework, Unit Test Framework In most cases, the OEM will deliver the mission boxes with at least a serial port and claimed this covers CSCI and UNIT testing. However, none of the other technical framework artefacts related to CSCI or UNIT testing are generally delivered. So the Ability to run software (test or otherwise) exists as a consequence of having mission equipment, rather than a full testing framework as a capability. In terms introduced in the reference material, the CSCU or UNIT testing delivered is generally: • Un-managed • Decentralised • Un-metered Software Construction Framework Some issues exist with capacity of software construction frameworks falling short of requirements of the relevant CATSUP. Expansion of software construction frameworks to meet CATSUP capacity requirements is problematic where quality goals related to Portability or Maintainability of the SDE are not met, namely: • CHANGEABILITY • ADAPTABILITY • INSTALLABILITY • CONFORMANCE • REPLACEABILITY Many of the SDE were delivered turn key without substantive software installation and configuration planning material that would allow building the capability from vendor media. This coupled with hardware obsolescence (usually realised prior to delivery to the CSF) means that “hardware failures ≡ lost Capability”. This generally makes the process of replicating or expanding the SDE more costly; especially where there may be certification issues. Page 18 BAE SYSTEMS IN CONFIDENCE Impact of fractured Technical Frameworks • With the shortfall in the technical framework artefacts delivered that address the CSCI and Unit Testing aspects (columns C and D of table), issues for continued certification arise out of discontinuous DT&E and IT&E across the transitional boundary from Developing agency to Maintenance agency (see next slide section below). • Problems arise from fractured history and missing data (information) related to the journey that the software travelled on its way to delivery. • This is compounded by being expected to work at a higher, or at least different, certification level without the framework artefacts delivered or the redemption of any gap in the frameworks sponsored by customer. Page 19 BAE SYSTEMS IN CONFIDENCE DT&E and IT&E across the Transitional boundary Page 20 BAE SYSTEMS IN CONFIDENCE Traditional V&V model Page 21 BAE SYSTEMS IN CONFIDENCE V&V Approaches mapped to Technical Frameworks • The exec summary, continued airworthiness requires DT&E and IT&E, across the transitional boundaries from Developing agency into Maintaining agency and the 3rd Party SSA will generally be disadvantaged unless particular care is place on the supportability requirements on projects to ensure transition of solid DT&E and IT&E frameworks. Technical Framework Components Development & Maintenance Operation A B C D E F G H I J K L M System Software Qualification Framework 6 8 9 9 9 Software Integration Test Framework 1 1 4 5 7 8 CSCI Test Framework 1 1 4 5 7 8 Unit Test Framework 1 5 Software Construction Framework 1 1 1 2 3 4 Key: A) Checklist-based inspection B) Perspective-based inspection C) Fagan-based inspection D) Complexity Measures E) Language Compilers F) Design Measures G) Path Testing H) Scenario-based Testing I) Module Interface Testing J) User Interface Testing K) User Discovered L) System Administration M) Environmental Page 22 BAE SYSTEMS IN CONFIDENCE TAMM Category of Support requirements recapped • Rectification may be interpreted as a deliberate or an accidental capping of Development through capping of Capability. • In terms of Technical Frameworks delivered it can be interpreted to mean incomplete or missing Unit, CSCI and Software Integration testing frameworks. Rectification. Capability to analyse and rectify faults. The necessary tools to make a software change shall be included, however, testing and other related activities may require the direct use of aircraft. Development. A full development and test capability similar in scope to the initial development environment, that can produce significant software changes to the system. The capability to do off-aircraft DT&E and provide simulated stimulus for testing purposes shall be provided. Full design disclosure and IPR shall also be provided. Enhancement. A full development environment with additional simulation, modelling, and analysis tools that would allow significant enhancement of system operation. The development and test environment shall allow analysis of actual system performance using appropriate tools and techniques. The testing facilities shall enable environment testing of real time performance where appropriate to the systems. Full design disclosure and IPR shall also be provided. Page 23 BAE SYSTEMS IN CONFIDENCE Supportability analysis of Software, SAE Supportability Concepts et al Page 24 BAE SYSTEMS IN CONFIDENCE Support Analysis for Software (SAS) Tailoring • Support Analysis for Software (SAS) has been devised as a consistent methodology that seeks the achievement of system and software supportability throughout requirements, specification and design, in order to define the most cost effective support concept that meets the operational requirements, and to ensure that the necessary support infrastructure is in place before the system enters into service. • The goals of Support Analysis for Software (SAS) are the following: – Evaluate sustainment capability – Demonstrate expansion and growth capacity – Verify logistic management of software – Address operational support of software – Support Concept planning • Such a process (support concept planning and execution) has therefore to: – determine those (support concept) requirements – influence (product) design so that supportability is built into the product – establish the Support Concept and ensure it is implemented be itself validated Page 25 BAE SYSTEMS IN CONFIDENCE SAS – Specification, Assessment, Qualification • It is convenient, in order to ensure software supportability, to frame the management of software supportability around two key components: – Software Supportability Plan: As part of the System Supportability Plan, it describes the activities to be undertaken in order to achieve the software supportability objectives. It also describes activities to be undertaken to demonstrate achievement of those objectives – Software Supportability Case: A written documentation about how product supportability was verified/developed at each stage of software development as per the SW Supportability Plan • These two elements are described in detail the SAE suite of Support Concept Standards: – JA 1004 Software Supportability Program Standard – JA 1005 Software Supportability Program Implementation Guide – JA 1006 Software Support Concept Page 26 BAE SYSTEMS IN CONFIDENCE SAS – Full scope of Specification, Assessment, Qualification • Overall Functional Architecture • Functional architecture at LRU Level • Functional architecture of related ground systems • Expansion and growth capacity information for each LRU • Expansion of data buses • Product Information • Identification Data • Management Data • Process Information • Software Loadable Units • Problem investigation capability • Operational support of Mission, Engineering and Diagnostic Data • Maintenance support indicators • CSCI inherent characteristics • CSCI Maintenance Support Resources • System-Level maintenance support • Proposed maintenance support options • Life-Cycle costs • Documentation Available Page 27 BAE SYSTEMS IN CONFIDENCE CATSUP Tailoring of SAS Page 28 BAE SYSTEMS IN CONFIDENCE Technical Framework Tailoring of SAS Page 29 BAE SYSTEMS IN CONFIDENCE SAS4SSA – Vehicle for assessing Supportability • Thus, the modified SAS, coupled with the audit tool for technical frameworks, and the CATSUP tailoring, attempts to provide both a depth and a breadth of support concept expression (covering Specification, Assessment and Qualification) for elements such as the data requirements and the quality of enabling sets (combined as Capability) of utility to the SSA (colloquially SAS4SSA). • Certainly, in the light of changing AP-3C Support Concept requirements, the SSA needs a basis for: – providing guidance to customer and projects in and around support concept specification and sustainment goals; and – vetting (assessment of) delivered products for sustainability (including continuing certification). • In the hands of the SSA, SAS4SSA is a Assessment tool for developing Sustainment risk models based on a Supportability taxonomy. • In the hands of the Acquirer it provides an approach to for Specification and Qualification of a Support Concept. Page 30 BAE SYSTEMS IN CONFIDENCE Program v Project Risk Management – who pays, who is left holding the bag Page 31 BAE SYSTEMS IN CONFIDENCE History repeats - tomorrow and tomorrow and tomorrow … • The root cause of sustainment problems is as much right sizing the support concept, but it is also specifying the support concept deep enough to ensure goals such as maintenance of the certification continuum are assured. • The CATSUP approach to support concept specification has tended to be too high a level of specification and has allowed quite wide ranging interpretations by OEM and Prime Contractors. • The previous paper by the author was a working document which was part of a SSA driven reaction to the support concept fracturing seen in the AP-3C. • Various mechanisms are culpable; the one that the author is trying to address is the lack of software sustainment/maintenance framework understanding – by stakeholders on both sides of the contract. Page 32 BAE SYSTEMS IN CONFIDENCE … and tomorrow • Issues in an around the AP-3C enabling sets, and the quality of the technical frameworks, create a tension between the requirements of the regulatory body of the SGTA and the users of 92WG – with the SSA caught in the middle. • The aging capability is under tension to “grow” from a small software maintenance and support organisation into a new sustainment organisation – but appears to be capped by legacy issues. • Much of the discussion focuses on models gleaned from standards and best practice -all the models being developed by industry as part of the failure mode analysis of software intensive system sustainment. • Continued Airworthiness (X-Worthiness) is, like many of the other goals of the AP-3C Weapon System sustainment organisation, impacted by risks injected from outside the Sustainment organisation. Page 33 BAE SYSTEMS IN CONFIDENCE Program Level Risk Management? • Sustainment of a weapon system requires a program level risk management model. • Someone Else’s Problem (SEP): Development Project view of Sustainment Issues. • The techniques to manage the risk fall into one or more of these four major categories: – Avoidance (eliminate) – Reduction (mitigate) – Transference (outsource or insure) – Retention (accept and budget) Page 34 BAE SYSTEMS IN CONFIDENCE Someone Else’s Problem • Def: SEP – Support concepts defined at too high a level. – Possibly too small a support facility established. – Too little in the way of significant work passing through the doors to develop and maintain system comprehension or program understanding in a changing resource pool. – Fracturing of AIC by CoA going directly to OEM. – OEM dis-engagement, particularly where OEM is being driven by poor market to product line approaches, rather than managing independent requirements of un-coordinated acquirers. – Un-warranted GFM provided to SSA. – Un-financed changes in support concept. Obsolete enabling toolsets supporting obsolete technology. Un-financed expectations to back fit regulatory frameworks requirements. Page 35 BAE SYSTEMS IN CONFIDENCE DoD speaks … • DoD touts 80% of a system’s cost life is in Operation and Support. • No stats appear to exist to account for the contribution of SEP to that cost. • Best practice approaches offer intuitive notions of cost savings if certain practices are avoided. • As mentioned in the preceding discussion, none of this is new. The standards embodied industry best practice yet lesson learned are un-learned. • The concern is that 80% of the sustainment budget needs to include the fat in schedules for additional V&V effort to cover unwarranted GFM in the CSF; and additional fat in schedules for system comprehension tasks due to shortfalls in the quality of the GFM and lack of tool support to recover information about the system; aimed to back fit the regulatory frameworks. • This is the risk avoided by projects realised. Page 36 BAE SYSTEMS IN CONFIDENCE Finale • The imperative however, as always, is to avoid injecting the issues into Sustainment to allow the SSA to provide Development services up to CATSUP 3, where required, and to maintain Continued Airworthiness in that context. • Extant systems and their sustainment issues aside, continued Airworthiness amongst them, with the expectation of future programs within the ADF, can we ever learn to better spend that 80%? Page 37 BAE SYSTEMS IN CONFIDENCE Summary • Revision of previous analysis on DMO Category of Support requirements • Revision on Technical Frameworks for SM&S • DT&E and IT&E across the Transitional boundary • Supportability analysis of Software, SAE Supportability Concepts et al • Program v Project Risk Management – who pays, who is left holding the bag Page 38 BAE SYSTEMS IN CONFIDENCE Questions	application/vnd.ms-powerpoint	200	http://www.defence.gov.au/dgta/Documents/DAVENG/Software%20Symposium%20documents/2008/Presentations/Failure%20to%20Transition%20(Feodoroff).ppt	au,gov,defence)/dgta/documents/daveng/software%20symposium%20documents/2008/presentations/failure%20to%20transition%20(feodoroff).ppt	QOLGPKLFL44YR3ALYYX4EWHECVCHJXLK	748568	domains/defence-gov-au/powerpoints/original/au-gov-defence-dgta-documents-daveng-software-20symposium-20documents-2008-presentations-failure-20to-20transition-20-feodoroff-ppt-20120507040004.ppt