Building an expert system for software quality evaluation

Building an expert system for software quality evaluation

North-Holland Microprocessing and Microprogramming 28 (1989) 179-182 179 BUILDING AN EXPERT SYSTEM FOR SOFTWARE QUALITY EVALUATION Nikolay S. Bukov...

295KB Sizes 1 Downloads 111 Views

North-Holland Microprocessing and Microprogramming 28 (1989) 179-182

179

BUILDING AN EXPERT SYSTEM FOR SOFTWARE QUALITY EVALUATION

Nikolay S. Bukovsky R e s e a r c h and D e v e l o p m e n t I n s t i t u t e I n t e r p r o g r a m a 62 S t a m b o l i i s k i Street, S o f i a 1303, B U L G A R I A

A u t o m a t e d s o f t w a r e tools are n e e d e d to i m p r o v e the a c c u r a c y and o b j e c t i v i t y of the s o f t w a r e q u a l i t y e v a l u a t i o n . The s o f t w a r e quality evaluation expert system that is currently being d e v e l o p e d at I n t e r p r o g r a m a I n s t i t u t e is d e s i g n e d to p r o v i d e s u c h support. The p a p e r d w e l l s on the first p h a s e in b u i l d i n g this system knowledge acquisition. Gathering and s y s t e m a t i z i n g e m p i r i c a l t e c h n i q u e s and h e u r i s t i c s , i n t e n d e d to f a c i l i t a t e q u a l i t y m e a s u r e m e n t and to i m p r o v e its a c c u r a c y and o b j e c t i v i t y , are d i s c u s s e d .

1.

INTRODUCTION

Mature software quality assurance r e l i e s on e s t a b l i s h e d s y s t e m s of q u a l i t y r e q u i r e m e n t s and m e a s u r e ment criteria to guide quality control a c t i v i t i e s [I]. There are p l e n t y of s t u d i e s and r e s e a r c h in the field of software quality e v a l u a t i o n . As a result, d i f f e r e n t software quality characteristics are d e f i n e d w h i c h are m e a s u r a b l e , non-overlapping and sufficiently e x h a u s t i v e to permit accurate q u a l i t y e v a l u a t i o n [2]. The quality assurance group w i t h i n the R e s e a r c h and D e v e l o p ment Institute Interprograma is chiefly concerned with software quality control. The latter is based on measuring (evaluating) software product attributes - more then 200 quality metrics (QM), hierarchically structured into a 4-1evel set; t h e s e QM are m e a s u r e d by q u a l i t y c o n t r o l e x p e r t s in each development phase. Considerable results have been achieved in quality control of the Interp r o g r a m a ' s p r o d u c t s . Yet we t h i n k that quality c o u l d be evaluated more o b j e c t i v e l y and a c c u r a t e l y if the f o l l o w i n g p r o b l e m s are solved: * the set of QM is c o m p r e h e n s i v e , covering all important quality attributes. However, measurement of v e r y f e w QM can be f o r m a l i z e d by a c c u r a t e a l g o r i t h m s or formulas. The m a j o r i t y of the QM is

e v a l u a t e d t h r o u g h h e u r i s t i c s and e m p i r i c a l t e c h n i q u e s p r o v i d e d by the q u a l i t y experts. These, however, differ in e x p e r i e n c e and qualification, and their knowledge is r a r e l y c o m p l e t e . As a result, q u a l i t y e v a l u a t i o n is to a certain extent subjective b e i n g d e t e r m i n e d by p e r s o n a l expertise; * q u a l i t y e v a l u a t i o n is l a b o u r - i n tensive and time-consuming (i.e. m o r e than 200 QM have to be m e a s u r e d ) a n d q u a l i t y e x p e r t s s h o u l d be quite c o n s i s t e n t and thorough in their day-to-day work. T h e y are o n l y human, however, and due to o v e r s i g h t or omissions, quality measurements are not a l w a y s p r e c i s e . Obviously automated software tools are needed to provide a s s i s t a n c e to the q u a l i t y experts. The software quality evaluation expert system (SQEES) that is currently being developed by Interprograma is designed to p r o v i d e such s u p p o r t a n d a u t o m a t e quality evaluation. B a s e d on the existent QM and integrating (in its k n o w l e d g e b a s e ) the e x p e r t ' s experience and knowledge, SQEES should facilitate quality meas u r e m e n t and i m p r o v e its a c c u r a c y and o b j e c t i v i t y . This p a p e r d e s c r i b e s our initial steps and l e s s o n s l e a r n e d in b u i l d i n g SQEES.

N.S. Bukovsky / An Expert System for Software Quality Evaluation

180

2. INITIAL

CONSIDERATIONS

The expert system approach to quality e v a l u a t i o n has b e e n c h o s e n for the f o l l o w i n g r e a s o n s [3]: * q u a l i t y e v a l u a t i o n k n o w l e d g e is s u b j e c t e d to o n - g o i n g e x p a n s i o n to include a d d i t i o n a l e v a l u a t i o n c r i t e r i a as q u a l i t y e x p e r t s discover them. With the inherent flexibility of the expert systems, such u p d a t e s can be done fast a n d easily; * when evaluating QM, the data available are rarely complete and certain, so the reasoning mechanism and uncertainties handling capability of the expert systems make them highly suitable; * e v a l u a t i n g s o f t w a r e q u a l i t y is a d i a g n o s t i c process: p r o b l e m s and a n o m a l i e s are i d e n t i f i e d t h r o u g h the use of e m p i r i c a l criteria and rules. Expert systems characteristics make them highly suitable for such an application. The SQEES being developed at Interprograma Institute is intended to be used both by the quality experts - in e v a l u a t i n g software quality, and by the software d e v e l o p e r s - in d e t e c t i n g (and even p r e v e n t i n g ) errors and anomalies. SQEES is designed to p e r f o r m the f o l l o w i n g f u n c t i o n s : * c u s t o m i z e the QM set and their weight (priority) according to the specific product category and q u a l i t y r e q u i r e m e n t s ; * guide quality experts through the r e s u l t i n g QM set, p r o v i d i n g c o n s u l t a t i o n , if n e c e s s a r y ; * e v a l u a t e each QM t h r o u g h g u i d e lines (questions), representing d i f f e r e n t QM a s p e c t s and g u i d i n g e x p e r t s to m e a s u r e the QM stepby-step. Completeness of the q u e s t i o n s as well as the c h o s e n e v a l u a t i o n rule (based on these questions) are a prerequisite for a c c u r a t e QM m e a s u r e m e n t . S Q E E S is d e v e l o p e d in 2 phases: knowledge acquisition (including SQEES p r o t o t y p i n g ) a n d S Q E E S imp l e m e n t a t i o n [4].

3. K N O W L E D G E

ACQUISITION

Knowledge acquisition (already started and expected to e n d by S e p t e m b e r 1989) includes:

1) R e f i n i n g the set of QM. Ease of use by h u m a n e x p e r t s and comprehensiveness of the QM are mutually exclusive requirements. Hence effective QM application requires the achievement of a reasonable and p r a c t i c a l balance b e t w e e n them. P r o c e e d i n g f r o m our five-year experience in a p p l y i n g the set of QM, we r e d u c e d it into a three-level structure of about one h u n d r e d QM [5]. Most signific a n t l y w e r e r e d u c e d the QM r e l a t e d to "Modifiability" as being s c a r c e l y a p p l i e d a n d not important requirements for Interprograma's products. 2) D e f i n i n g Q M w e i g h t s for each c a t e g o r y of s o f t w a r e p r o d u c t s developed at Interprograma Institute. The d e s i r a b l e q u a l i t i e s of a product vary with the characteristics of its category. So different sets of typical QM weights were defined for each p r o d u c t s c a t e g o r y to q u a n t i f y its specific quality requirements. Subsequently, in e v a l u a t i n g quality, the score g i v e n to each QM is m u l t i p l i e d by the a s s o c i a t e d QM weight. 5) D e v e l o p i n g quality evaluation algorithm. In e v a l u a t i n g the quality of a product, only the lowest-level (primitive) QM are evaluated by straightforward analysis of the product itself (hence only these QM need acquiring knowledge about their evaluation). Being more general quality characteristics, the higher-level QM rely on the prim i t i v e QM as n e c e s s a r y c o n d i t i o n s for their e v a l u a t i o n . So an algorithm (formula) was developed which calculates the h i g h e r - l e v e l QM scores by m e a n s of the primitive QM scores and the QM weights.

4) D e t e r m i n i n g the applicability of the QM to the d e v e l o p m e n t phases. This a c t i v i t y results in d e f i n i n g a d i f f e r e n t set of QM for each d e v e l o p m e n t phase. 5) G a t h e r i n g heuristics and other empirical techniques on evaluation of the p r i m i t i v e QM, applied by quality experts or recommended in r e f e r e n t i a l literature. (These techniques are u s u a l l y in the f o r m of e v a l u a t i o n criteria).

N.S. Bukovsky / An Expert System for Software QualRy Evaluation 6) For each primitive QM, the information thus acquired about its e v a l u a t i o n is s y s t e m a t i z e d in the f o l l o w i n g way: * b r e a k i n k g down of the e v a l u a t i o n criteria into two groups in acordance with their output: i n c r e a s i n g / d e c r e a s i n g QM score; * the e v a l u a t i o n c r i t e r i a of e a c h group are represented in the form of w e l l - d e f i n e d , detailed and n o n - o v e r l a p p i n g g u i d e l i n e s . These are i n t e n d e d to a i d h u m a n e x p e r t s in e v a l u a t i n g the different a s p e c t s of the QM, and also to be u s e d b y d e v e l o p e r s as g u i d a n c e for r e d u c i n g a n d even preventing occurrences of errors. Examples of such guidelines are: "Data-base-oriented products not containing checkpoint-restart capability have decreased recoverability" "Network products not providing procedures for recovery from h a r d w a r e e r r o r s are not s u f f i ciently reliable" "End-user products not allowing the use of f r e e - f o r m input are not c o n v e n i e n t to use"; * q u a n t i f y i n g the g u i d e l i n e s output, i.e. d e t e r m i n i n g the quant i t a t i v e r e s u l t of a p p l y i n g the g u i d e l i n e . E a c h g u i d e l i n e is assigned a numerical value indicating the degree (in p e r c e n tage) to w h i c h the a v e r a g e QM score should be increased/decreased in case the evaluated product conforms to the g u i d e line. C o n s e q u e n t l y , a numerical r a t i n g of the g u i d e l i n e s of e a c h QM group is established with r e s p e c t to their c o n t r i b u t i o n to the QM score. For instance: "The e x i s t e n c e of g r a p h i c a l representation of the system structure leads to a 20% increase of the system structuredness" " C o n s i s t e n t use of s p e c i a l n a m i n g conventions, requiring the use of mnemonic, self-descriptive and uniform names for all system elements variables, modules, blocks, functions leads to a 10% i n c r e a s e of the system maintainability" "Test c a s e s not c o v e r i n g all p e r missible combinations of the p r o d u c t f u n c t i o n s or not v e r i f y ing all b o u n d a r y v a l u e s of the input d a t a d e c r e a s e s the p r o d u c t functional capability (operability) by 20%"

181

"Product results (like reports, m e s s a g e s ) not d e s i g n e d in a uniform manner and not using a single notation l e a d to a 20% d e c r e a s e of the u s a b i l i t y of the user i n t e r f a c e " "Device-dependent and operatingsystem-dependent parameters of the p r o d u c t not s p e c i f i e d in a table-driven way, but embedded in the p r o g r a m code, r e d u c e the p r o d u c t m o d i f i a b i l i t y by 15%" "Self-modifying program code is p e r m i s s i b l e o n l y for h i g h l y efficient real-time programs; o t h e r w i s e it d e c r e a s e s the product m o d i f i a b i l i t y by 35%" "The excessive usage of common blocks by different functional components decreases functional c h a n g e f l e x i b i l i t y of the product by 15%" " D o c u m e n t a t i o n w h i c h a l l o w s tracing program implementation of the p r o d u c t functions improves its m a i n t a i n a b i l i t y by 10%"

7) Determining the permissible deviation (in p e r c e n t a g e ) f r o m the a v e r a g e QM score. The o b j e c t i v e of this is to i d e n t i f y d e f i c i e n c i e s and a n o m a l i e s in the p r o d u c t w h i c h the d e v e l o p e r n e e d s to deal w i t h rather than just indicate a numerical value of the quality deviation. For that purpose, in case of low QM score, exceeding the permissible deviation, the expert system reasoning mechanism s h o u l d b a c k t r a c k the c a u s e and its importance, i.e. indicate the evaluation guideline which contrib u t e s m o s t to d e c r e a s i n g the QM score. This s h o u l d m a k e the e x p e r t system effective in error d e t e c t i o n , not just in q u a n t i f y n g the QM. In g a t h e r i n g k n o w l e d g e on the QM e v a l u a t i o n it t u r n e d out that many QM are easy to evaluate, consequently evaluation guidelines are not so m u c h required. This pertains to such QM like " C o m p l e t e n e s s of the p r o g r a m implementation", "Completeness of the d o c u m e n t a t i o n " . On the other h a n d QM r e l a t e d to " R e l i a b i l i t y " and " S t r u c t u r e d n e s s " i n v o l v e h i g h d e g r e e of a n a l y s i s w h i c h is quite complex, h e n c e about f i f t y g u i d e lines w e r e d e v e l o p e d to a i d eval u a t i o n of t h e s e QM. Together with knowledge acquisition, the SQEES prototype is

N.S. Bukovsky / An Expert System for Software Quality Evaluation

182

b e i n g d e v e l o p e d - u s i n g an expert s y s t e m shell ( r u n n i n g on IBM PC) and i m p l e m e n t i n g a subset of the QM (about t w e n t y QM). D e v e l o p m e n t of the p r o t o t y p e is a i m e d at: * getting acquainted with the c h o s e n shell and a s s e s s i n g its suitability; * d e t e r m i n i n g a n d t u n i n g the user i n t e r f a c e to SQEES; * providing (through the simplified prototype version) experience and k n o w l e d g e about the SQEES necessary when implementing it ( p l a n n e d to end by m i d 1990).

assessing the QM completeness and, if necessary, further d e f i n i n g t h e m in detail; * if the QM g u i d e l i n e s are detailed, non-overlapping and quantified ( r a t e d in a c c o r d a n c e with their output), they are easily transformed into IF-THEN rules. (In fact, e a c h g u i d e l i n e is represented as a single rule). H e n c e a n a l y z i n g and systematizing QM g u i d e l i n e s in a thorough way is required to facilatate SQEES implementation.

REFERENCES 4.

CONCLUSIONS

Based on our e x p e r i e n c e of expert system application in software quality evaluation, the f o l l o w i n g c o n c l u s i o n s can be drawn: * at I n t e r p r o g r a m a S Q E E S is b e i n g d e v e l o p e d by the q u a l i t y e x p e r t s t h e m s e l v e s w h e r e b y e l i c i t i n g and transferring knowledge f r o m exp e r t s to e n g i n e e r s is s i g n i f i c a n t l y s i m p l i f i e d . This a p p r o a c h of a c q u i r i n g quality knowledge is strongly recommended to others i n v o l v e d in the d e v e l o p ment of s y s t e m s like SQEES; * having a reasonably comprehensive set of QM is a p r e c o n d i t i o n for successful knowledge acquisition - if detailed, QM are easily provided with g u i d e l i n e s for their e v a l u a t i o n . So the first step in q u a l i t y knowledge acquisition s h o u l d be

[I] C a r d , D . N . , P r o c e e d i n g s of the Second IEE/BCS Conference S o f t w a r e E n g i n e e r i n g 88 (1988) 33. [2] B o e h m , B . W . , Brown,J.R., Kaspar,H., Lipow,M., MacLeod, G.J. and M e r r i t , M . J . , Characterictics of S o f t w a r e Quality (North-Holland, Amsterdam, 1978 ). [3] B r i d g e s , M . C . , Proceedings of the S e c o n d I E E / B C S C o n f e r e n c e S o f t w a r e E n g e n e e r i n g 88 (1988) 178. [4] H a y e s - R o t h , F . , Waterman,D.A. and Lenat, D.B., B u i l d i n g Expert Systems (Addison-Wesley P u b l i s h i n g Company, Inc., Mass a c h u s e t t s , 1983). [5] B u k o v s k y , N . S . , Proceedings of the I n t e r n a t i o n a l S y m p o s i u m on Information Technology Standardization INSITS, Braunschweig, (1989) in print.