files: 69

This data as json

rowid	image	timestamp	memento	first_capture	last_capture	current_status	text	mime	status	url	urlkey	digest	length	file_path
69		20120507035606	https://web.archive.org/web/20120507035606/http://www.defence.gov.au/dgta/Documents/DAVENG/Software%20Symposium%20documents/2008/Presentations/Considerations%20in%20the%20Preference%20for%20and%20Application%20of%20RTCA_DO-178B%20in%20the%20Australian%20Military%20A.ppt	2012-05-07	2012-05-07	404	1 DGTA-ADF Directorate General Technical Airworthiness Considerations in the Preference for and Application of RTCA/DO-178B in the Australian Military Avionics Context SQNLDR Derek Reinhardt Systems Certification and Integrity (SCI) Directorate of Aircraft Engineering (DAIRENG) 2 DGTA-ADF Directorate General Technical Airworthiness Overview • Introduction to criticisms of RTCA/DO-178B • Background and structure of DO-178B • Criticisms that RTCA/DO-178B is insufficient • Criticisms that RTCA/DO-178B is too onerous • ADF’s preference for RTCA/DO-178B • Application issues of RTCA/DO-178B in the Australian Military Avionics Context 3 DGTA-ADF Directorate General Technical Airworthiness Introduction • RTCA/DO-178B is the centre of much debate or criticism – insufficient, too onerous, etc • Avoidable software failures have already been responsible for aircraft mishaps – cockpit displays go blank – engines throttle back during takeoff – contradictory airspeed readings • This presentation/paper – examines these criticisms – how do they influence the ADF’s preference for and application of RTCA/DO-178B 4 DGTA-ADF Directorate General Technical Airworthiness ADF Preference for RTCA/DO-178B • RTCA/DO-178B is the ADF’s preferred software assurance standard for safety critical and safety related airborne software development – Refer AAP7001.054 Airworthiness Design Requirements Manual • ADF recognises the FAA processes and standards – widely used and accepted by many international airworthiness authorities • Many military aviation systems have a civil heritage with software developed under RTCA/DO-178B – AEW&C B737, MRTT A330, etc 5 DGTA-ADF Directorate General Technical Airworthiness Background of RTCA/DO-178B • 14 CFR 25.1309 – often referred to as FAR 25.1309 – SAE ARP 4754 and SAE ARP 4761 • aircraft level FHA, FTA • system level FMEA, FTA, etc. • common cause analysis – PRA, ZSA, CMA – RTCA/DO-178B for software design assurance – RTCA/DO-254 for hardware design assurance • Five failure condition severities are assigned design assurance levels (DALs) – Catastrophic (A), Hazardous / Severe Major (B), Major (C), Minor (D), No Effect (E) 6 DGTA-ADF Directorate General Technical Airworthiness Structure of RTCA/DO-178B • 66 objectives in 10 tables – + several implicit objectives – satisfaction tailored by software level – several prescriptive objectives • statement coverage, decisions coverage, modified condition decision coverage • Lifecycle phases – planning, development, requirements, design, coding and integration, verification • Integral processes – configuration management, quality assurance • Certification Authority liaison • RTCA/DO-248B, FAA Order 8110.49, CAST5 7 DGTA-ADF Directorate General Technical Airworthiness Intricacies of DO-178B • Common misconception that RTCA/DO-178B completely specifies the process and activities • Yes – objectives are coupled to the software lifecycle • Yes – they don’t distinguish between requirements – functional, software safety, etc • Fidelity of the objectives presents challenges for COTS and PDS • With exception of 3 objectives – flexibility is permitted in how the objectives are satisfied • Objectives can be broadly classified as contributing to requirements validity, requirements satisfaction, and requirements traceability 8 DGTA-ADF Directorate General Technical Airworthiness Criticisms of RTCA/DO-178B • Divided into several positions – those that believe RTCA/DO-178B is insufficient • academics, researchers, or consultants – those that believe RTCA/DO-178B is too onerous • development contractors with cost and schedule driven imperative • Criticisms are at odds with each other – central to the criticisms, then is it about right? • Widely accepted by FAA, EASA – NTSB reports that it is effective in those contexts • How does the ADF address the criticisms? 9 DGTA-ADF Directorate General Technical Airworthiness RTCA/DO-178B is Insufficient • Absence of mandatory formal methods • Absence of mandatory static code analysis • Ineffectiveness of MC/DC testing • Absence of specific requirements or objectives relating to software safety analysis and software safety requirements • Assumptions underlying the design assurance level definition are questionable 10 DGTA-ADF Directorate General Technical Airworthiness Absence of Mandatory Formal Methods • Does not prohibit formal methods – acceptable method to satisfy objectives • Application to problem domain – not universally applicable to problem domains and technologies used in critical systems – only partially applicable to problem scope – are closed languages, no inherent real world meaning, natural language is still required • Formal methods and safety – does not address significant sources of error WRT the safety of systems – little evidence that it would prevent a number of mishaps attributable to software • Complementary to testing – testing is required to demonstrate real world behaviours, on real hardware, in the target environment • Formal methods is not the ‘silver bullet’ for safety software 11 DGTA-ADF Directorate General Technical Airworthiness Absence of Mandatory Static Code Analysis • Does not prohibit static code analysis – it is an acceptable method used to satisfy objectives • Static analysis won’t find all the faults (requirements and implementation) most related to the safety of the software • Permits greater focus on those activities related to identifying requirements validity and satisfaction problems affecting safety – prevents developers being overwhelmed in code reviews and testing identifying inadvertent implementation problems, which static code analysis tools readily detect • Limitations to applying static analysis retrospectively 12 DGTA-ADF Directorate General Technical Airworthiness Ineffectiveness of MC/DC Testing • Exercise each data flow that directly affects a control flow to identify as many faults as possible • Widely debated objective – studies confirm that MCDC does find faults that other DO- 178B testing approaches do not find – other studies found “no significant difference” • RCDC address some of these limitations • MCDC not finding additional faults is not the concern – if normal and robustness testing has been comprehensive, then it is possible that the gap in MCDC will be small, and NOT safety related – adequacy of normal and robustness testing 13 DGTA-ADF Directorate General Technical Airworthiness Absence of Specific Requirements or Objectives Relating to Software Safety Analysis and Software Safety Requirements • Is not explicit in objectives relating to software safety analysis and software safety requirements – but they are not absent! – number of objectives contribute to requirements validity, including that of safety requirements • However, systematic software safety analysis are not always proposed to show that the identified and allocated set of software safety requirements is complete and correct • DGTA recommends an IEEE1228 Software Safety Plan be used to document the planned software safety activities – and their outcomes – or the RTCA/DO-178B PSAC can be used 14 DGTA-ADF Directorate General Technical Airworthiness Assumptions Underlying the Design Assurance Level Definition are Questionable • Issues with integrity/assurance levels – little evidence that software of different integrity levels does have failure rates of integrity level order – debate regarding the philosophy and rules for allocating integrity levels • significant differences in the processes recommended by standards • Inconsistent application – misunderstanding of the differences between reviews versus analyses • some objectives being presumed to be satisfied solely by reviews • intent is combination of analysis and reviews of the outputs – variable • normal and robustness testing • software architecture is appropriate – avoids design constructs that would not comply with system safety objectives • software safety requirements are identified/allocated • Apportionment and adequacy of objectives – objectives fundamental to the validity and satisfaction – from Level C – Level A and B provide additional evidence - trustworthiness 15 DGTA-ADF Directorate General Technical Airworthiness RTCA/DO-178B is Too Onerous • Excessive requirements specification and traceability fidelity requirements • Excessive verification requirements 16 DGTA-ADF Directorate General Technical Airworthiness Excessive Requirements Specification and Traceability Fidelity Requirements • RTCA/DO-178B motivation for fidelity – each behaviour that constitutes the requirements at level of abstraction is systematically accounted for – design tool to assist developer produce a good design • Why should all the behaviours be accounted for? – evidence that all behaviours of the software are acceptable – do not lead to unacceptable failure conditions – all the behaviours of the software should be disclosed – permits reasoning about their suitability for the safety of the system – arguing non-interference with the behaviours that are important to the safety of the system • Questionable disagreements – Intellectual Property constraints – software development is a creative process, not itself compatible with such rigour requirements 17 DGTA-ADF Directorate General Technical Airworthiness Excessive Verification Requirements • Testing will always be required to gain confidence in the behaviour of software on the target hardware in the intended operating environment • Completion of testing – defensible engineering argument as to why testing is complete – with evidence to support it • Not based on the following factors: – cost and schedule – consensus of program managers – broad consensus of the programmers and testers – any other non-engineering based arguments • RTCA/DO-178B provides a defensible argument as to when testing is complete by specifying: – requirements completeness criteria – requirements coverage criteria • extent of normal and robustness tests • extensiveness of requirements based low level testing – coupled with additional implementation related coverage criteria (structural coverage) to elicit certain properties 18 DGTA-ADF Directorate General Technical Airworthiness Application of RTCA/DO-178B • RTCA/DO-178B is not applied in isolation • Test coverage objectives • Use of RTCA/DO-178B as a benchmark for assessing software assurance practices • COTS with RTCA/DO-178B • Migrating to RTCA/DO-178B 19 DGTA-ADF Directorate General Technical Airworthiness Not Applied in Isolation • System Safety Program - MIL-STD-882C or FAR 25.1309 • Key Issues – A Software Safety Program (SwSP) should be established to coordinate hazard identification and mitigation efforts for hazards with software-related causal factors. • IEEE1228 Software Safety Plan • Software Safety Analysis, Generic Software Safety Requirements – A Software Assurance standard is required for the development of all software that is safety related – Software process standards should be applied to the development of software for airborne and related systems • An assessment of the applicant’s software development capability, including domain knowledge, should be conducted as part of the tender evaluation and contract negotiation process – examines the safety culture – determine if non-systematic (experience) activities can be trusted 20 DGTA-ADF Directorate General Technical Airworthiness Test coverage objectives • Some organisations believe that DGTA does not require structural code analysis – by default – THIS IS NOT TRUE! • In cases where DGTA has not required it – additional activities to ensure that the coverage is comprehensive • fidelity of requirements • explicitness of traceability • extent of normal and robustness testing • additional activity to identify dead and deactivated code – often exception related code – mission systems, with no series hazards – extenuating circumstances – legacy constraints • Negotiate through the PSAC –expect to have a compelling argument if you are going to proposed not doing it 21 DGTA-ADF Directorate General Technical Airworthiness Software Assurance Benchmark • Assess software products and their development practices – software agencies that do not use RTCA/DO-178B – confused that DGTA is applying DO-178B where not contracted • RTCA/DO-178B objectives help with the assessment of – requirements validity, satisfaction, and traceability • RTCA/DO-178B provide clear measures of – fidelity in the specification and traceability of requirements • no gap between required behaviours and executable code • all software behaviours are appropriate with respect to safety – extent of test based verification of requirements and implementation on the target computer in the intended environment – development and review rigour • independence and oversight • assures that the evidence presented contains an acceptably small number of faults 22 DGTA-ADF Directorate General Technical Airworthiness COTS with RTCA/DO-178B • COTS fall under the scope of PDS • PDS criteria are demanding, but not excessive – but often the evidence is just not available • Alternate proposals for use of COTS • Provide evidence to demonstrate the following types of properties – Partitioning (containment and/or mediation) – Non-Interference – Acceptable Behaviours 23 DGTA-ADF Directorate General Technical Airworthiness Migrating to DO-178B • Acquired where no software assurance standard applied – DOD-STD-2167A, MIL-STD-498, IEEE12207 • Two options – transition to RTCA/DO-178B or develop a Software Assurance Task Matrix • FAA guidelines – Notice 8110.49 Chapter 10 – Intended for systems developed to RTCA/DO-178 and RTCA/DO-178A – Careful management of the scope of retrospective production of software assurance evidence is required – DGTA has assessed the approach as acceptable 24 DGTA-ADF Directorate General Technical Airworthiness Summary • Introduction to criticisms of RTCA/DO-178B • Background and Structure of DO-178B • Criticisms that RTCA/DO-178B is insufficient • Criticisms that RTCA/DO-178B is too onerous • ADF’s preference for RTCA/DO-178B • Application of RTCA/DO-178B in the Australian Military Avionics Context – RTCA/DO-178B applied within the ADF framework addresses relevant criteria for producing safety software systems 25 DGTA-ADF Directorate General Technical Airworthiness Questions	application/vnd.ms-powerpoint	200	http://www.defence.gov.au/dgta/Documents/DAVENG/Software%20Symposium%20documents/2008/Presentations/Considerations%20in%20the%20Preference%20for%20and%20Application%20of%20RTCA_DO-178B%20in%20the%20Australian%20Military%20A.ppt	au,gov,defence)/dgta/documents/daveng/software%20symposium%20documents/2008/presentations/considerations%20in%20the%20preference%20for%20and%20application%20of%20rtca_do-178b%20in%20the%20australian%20military%20a.ppt	GDC2THWRIGUGO6DZ25KMTBKV3PTB25HQ	68372	domains/defence-gov-au/powerpoints/original/au-gov-defence-dgta-documents-daveng-software-20symposium-20documents-2008-presentations-considerations-20in-20the-20preference-20for-20and-20application-20of-20rtca-do-178b-20in-20the-20australian-20military-20a-ppt-20120507035606.ppt