Decision Support for Best Practices Lessons Learned

Decision support for high hat dresss Lessons asked on bridging the gap surrounded by research and employ pratice. Today, everyone is looking at surmount confides for resurrecting a system or reservation the right choice in acquiring system components. If the right best put ons atomic number 18 use, they help to avoid unwashed problems and improve lineament, cost, or both. However, lifting and selecting an appropriate best practice is non almodal values an easy closingeavor. In most slicknesss guidance, found on grievous envision, is missing a lot the best practice is besides new, still downstairs study, or the existent experiences do not fit the social functionrs consideration.This article reports on a program that tries to bridge the gap between rigorous trial-and-error researchand practical emergencys for guiding practitioners in selecting appropriate best practices. ********** Mevery program managers would agree that using clipping-tested beat out Pra ctices rout out help to avoid common problems and increase the property of a system, lower development cost, or both. For instance, in a short survey at the 2004 assembly on the attainment of Softwargon-Intensive Systems, 48 senior systems and softw atomic number 18 managers hazard up the use of surpass Practices.However, the same survey indicated that it is grave to find such Best Practices. The survey identified the following reasons for this problem * Best practices practic tout ensembley do not exist (i. e. , they hold in not been publicly documented), * heap do not know of a certain best practice, or * Best practices are not easily accessible (i. e. , on that point is no central place to look for best practices). The last point matches a much(prenominal) general study by the Delphi Group in which more than 65 percent of the interviewees agreed that purpose the right nformation to do their trick is difficult (Delphi, 2002). Further research conducted by the U. S. Department of apology ( defense mechanism) think that barriers for the adoption of best practices include * the drop of selection criteria among practices within cost-constrained programs, * the lack of confidence in the value of such practices by the program offices, and * the inability to relate practices to the risks and issues programs were facing. In synopsis, recognizing good practices anddisseminatingthem to the workforce seems to be a samara issue.To track these issues the DoD Acquisition Best Practices Clearinghouse (BPCh) program, sponsored by several offices of the DoD (DS,ARA, National Information Infra construction NII, and justification Procurement Acquisition polity DPAP), was initiated in 2003 (Dangle, Dwinnell, Hickok Turner, 2005). The Fraunhofer touch for Experimental software program engine room, Maryland (FC-MD) was chosen to develop the sign proof of idea for a system to document, evaluate, anddisseminateBest Practices.In collaboration with ot her organizations within the DoD and industry (includingNorthrop GrummanIT, the Computer Sciences breadbasket CSC, and the Systems and Software Consortium SSCI), a prototype system has been built and piloted. It is before long operated and hosted by the Defense Acquisition University (DAU). THE VISION FOR APPLYING BEST PRACTICES The DoD sight for the BPCh initiative is to go out more than secure a list of Best Practices. It is to provide an integrated set of processes, calamuss, and resources which pull up s shams enable learning seekers to secern emerging or ell-proven practices that pass been implemented and proven effective. Practices in the BPCh administer as an education resource to individualslooking forideas on how to improve feel and blend more effective in their job. Clearly, the vision of the BPCh is not to puddle another data cemetery, notwithstanding to develop an information-sharing network around the BPCh alluviation which give foster relationships b etween individuals within DoD and besides partnerships between DoD and industry leaders.The following types of questions illustrate usage events * I just heard nearly accelerated feel testing. Where tramp I find out if its useful or just hype? * Theyve just weakened my testing schedule by 30 percent. Are there any practices that hind end help me better handle that kind of schedule compressing? * I want to add inspections to my quality process. Is it worth the cost and if so, whats a good number one step? Is there nearlyone I fecal matter contact in case of any difficulties? * Ive taken over an eruditeness program just before Critical Design Review (CDR). What practices should I look for in my contractors? * Im in charge of defining a train course as part of thecontinuing educationprogram for quality improvements. What are state-of-the-art or emerging practices that should be addressed? The BPCh has been intentional with the understanding that a wiz practice give the axe never be a silver bullet for each and every project/program.This is because round practices whitethorn only be useful or beneficial in certain circumstances while failing to produce the pronenessd results in others. For example, practices that are absolutely necessary for large, mission critical projects may be too heavyweight forrapid prototypingor Web application development. Practices that work rise up when the development team is located in the same room may not al way of lifes scale well when the team is distributed across the country. Clearly, there exists no one best answer. Practices that are best for one substance ab exploiter might not be best for the next.Therefore, the BPCh puppet responds to substance abuser queries with a list of practices rated by how well they fit the project characteristics of the user making the query. The pre directed selection is compiled using the experience other users piddle had implementing the practice in a similar context . High-quality test about a practice is store and reported with any necessary caveats, so that information seekers hasten a sound basis for making up their own minds addicted their inescapably. APPLYING technology TO DELIVER BEST PRACTICESTo develop the BPCh dent, we applied FC-MDs emperor approach ( hold oversight Portal using Empirical Results as Organizational Resources). This approach develops use of all kinds of for saleevidentialdata from research and industry, analyzes and packages it, and disseminates it with a Web- ground accept Base. The EMPEROR is based on the experience factory approach, essential by Basili, Caldiera, and Rombach (1994), which has been successfully employed to facilitateorganizational learningatNASA(Basili, et al. 1995), D call forlerChrysler (Schneider & Schwinn, 2001), and elsewhere inNorth America, Europe, and Australia (Koennecker, Jeffery, & Low, 2000 Mendonca,Seaman, Basili, & Kim, 2001). An experience factory provides a way to analyz e results based on practical experience, and package what is learned into an Experience Base for new users of the organization to find and apply. Since the users of the BPCh come from a tolerant variety of organizations and programs, any Experience Base volition have difficulties in addressing all user needs.To mitigate this problem, EMPEROR is required to (a) provide enhancer to users, so that they can understand the analysis process and the sources of experience and make up their own minds (b) rate the trustability of each of the used sources, so that users can judge the degree of confidence they have in the information provided and (c) provide a completeness and due date indicator of the practice information taken as a whole, that is, to perform a self- rate based on how much and what quality curtilage can be offered. DATA STRUCTURE OF A BPCH PRACTICEThese sections take in how these requirements are implemented in the case of the BPCh. In the BPCh, each practice has one ass ociated Practice unload, containing information about the practice and what is available in the Clearinghouse, and zero to many say Profiles, each of which contains a summary of a single organizations experience using the practice. A Practice Record consists of 1. A Practice Detail block, which contains information such as the practice name, a short description, and the completeness and maturity indicator for the experience package. . A Practice sum-up block, which synthesizes all available usher data and detects possible application contexts for the practice based on a set of characterizing attributes. This part of the practice drop thereby cedes different users (i. e. , organizations) to make use of the practice. An Evidence Profile contains an example or report of roughly type of program that has used this practice, how they applied it, and what results were obtained.Each Evidence Profile contains the same set of context and result handle as the Practice Summary block, e xcept that the information recorded in each field forget describe only what has been observed in the given context of the particular piece of evidence. In addition, the data structure of an Evidence Profile contains a field for documenting its classification of the trustability. TRUSTABILITY OF A hotshot SOURCE OF EVIDENCE A 20-point scale rates the trustability of each Evidence Profile.A rating of l indicates ananecdotalor on the loose(p) experience a rating of 20 indicates that the results of applying the practice are rigorously measured and substantiated. Points are based on the following four dimensions * how the practice was applied, ranging from a single pilot study to use on multiple real projects * how the results were measured, ranging from an educate guess to a rigorous measurement program * how the evidence was reported, ranging from an informalanecdoteto a peerreviewed publication and who reported the evidence, ranging from a second-hand report to someone directly i nvolved on the team. More information on the rating scale can be found on the BPCh page of the Acquisition Community Connection of DAU (https//acc. dau. mil/bpch). MATURITY OF A PRACTICE recruit A 4-point scale is used to rate each Practice Record to livelyly inform the user of how much, and what type of, information is known about the practice. As required by EMPEROR, this scale focuses on the quality of the overall accumulated information that is available for a practice (i. e. thesynthesizedand packaged information in the Practice Record). Based on the available information we describe the practice maturity as * No status assigned/ sign entry A new Practice Record is initially entered into the BPCh when it is put forward by our experts and/or user communities. Typically at this prison term, only some of the fields in the Practice Detail block are change in and no Evidence Profiles are available. * dye status/ cognisance raised As soon as any evidence becomes available (i. e. , an Evidence Profile has been linked to the Practice Record), the status is set to Bronze take aim.For users, the Bronze Level status indicates that the practice has been nominated by our experts and user communities, and received a preliminary check for applicability. * bills status/ military rank performed When a adequate set of Evidence Profiles is available, the BPCh experts exit fill in the Practice Summary block and the status is set to Silver Level. For users, the Silver Level status indicates that the practice has been selected as promising enough to explosive charge experts in the area to summarize key information. drug users can see at a glance what they should know. * bullion status/Continuously maintained When the summary has been further evaluated (i. e. , vetted) by experts from industry, academia, and government, the status is set to Gold Level. For users, the Gold Level status indicates that the practice has been through with(predicate) a rigorous analysis by a committee of experts in the practice itself as well as by user representatives. Information on Gold Level practices contains the best and widest-ranging experiences we can find. nitty-gritty STATUS OF THE BPCHWe have been piloting BPCh processes and tools by seeding initial pith. At this point the BPCh contains 51 practices at all levels of maturity. Practices that have progressed to Gold Level are those, like inspection/technical review, which have a long history of published industrial experience. Many practices of fill in the area of systems and software program acquisition have few documented sources of evidence or experience. Therefore, we are testing different processes for eliciting information from the workforce.Based on the recommendations of our User consultive Group, the following types of practices are latestly our top-priority areas for additional national * realize Value Management, * Risk Management, * Information Assurance, and * Spiral Development Proces s. We hope that visitors to the BPCh tool will try out the offered features for providing short stories about their own experience with practices in these (or any other) areas. We encourage you to provide feedback as to whether you agree ordisagree withthe existing experiences that have been entered, or thoughts on our BPCh tool in general.LESSONS LEARNED Based on our experience with the BPCh program and other fellowshipmanagement projects, we can formulate some observations which make useful rules of thumb for good practices to constitute such systems. The BPCh program has been machinated along three correspond (but interconnected) tracks, which reflects our first gear lesson learned. LESSON 1 PROCEED IN MULTIPLE DIRECTIONS SIMULTANEOUSLY Progress in building a association bank deposit needs to proceed in multiple dimensions simultaneously field of study collection, tool development, and outreach.Although there is often a temptation to view these as tasks that can be done c onsecutive (e. g. , first the tool will be built, thenpopulated, and then it will be advertised to users), we have found this to be an overly simpleview that diminishes the chance of project success. Constructing the tool prior to collecting essential inwardness and getting users feedback almost ensures that important user needs will be discovered late and will require much more effort to implement. Populating the content without getting user feedback leads to a high likeliness that the content will not really address user needs.More importantly, content needs to come from the user company, if the repository is to have a long life. We have found that for the research team to generate substantial amounts of content is a time-consuming way of recreating what many users already have at theirfinger tips. Finally, engaging in outreach and building excitement in the connection of potential users runs the risk of all prototyping efforts When told how anything is possible in the fina l system, users often come up with many wish list features that are not really linked to their everyday needs.Moreover, users often getfrustratedwith the deadening pace of progress when the system actually has to be implemented, and lose care before the system is fielded. To avoid these problems, we have adopted anincrementalapproach, with content and tool development going on simultaneously and outreach activities to the user fraternity (such as booths at major conferences, or specific User Advisory Group meetings) planned at major milestones.Although this sometimes stretches resources a puss thin, we feel this approach has enabled us to engage periodically with the user community, guide them progress since the lastiteration, and get feedback on ever more climb on versions of the system, with an initial body of content. LESSON 2 MAINTAIN A CONTINUOUS menstruum OF FUNDING Because of the interconnected nature of all the tasks listed above, having a stable financial support st ream is crucial.Requiring the team to take ahiatusfrom the project afterwards a release is delivered leads to lost opportunities for user involvement (users find it hard to match their schedule to the development teams), leads to new content ideas that miss getting followed up on, may result in the loss of expertise if undergo personnel resources are in transition to other projects during the hiatus, increases the personnel learning curve encountered at re origins, and may result in flagging come to in the user community since momentum generated during outreach is lost.LESSON 3 RECOGNIZE THE relation MERITS OF CONTENT Our most important lesson learned is a direct hint of the BPCh vision There is no such thing as a Best Practice. Or, to say it more diplomatically No practice will be best for every project. Practices that are absolutely necessary for large, mission-critical projects may be too heavyweight for rapid prototyping or Web application development. The implications of this lesson are many.Perhaps the most important is related to the tone of the recommendations that users find quite a than arguing as an expert that readers should be following a given practice, or else they are doing something wrong, practices should be recommended to readers on the basis that projects of certain type(s) have found it useful. That is, rather than presenting aforegone conclusionto users, the system should aim at respecting users intelligence enough to enable them to draw their own conclusion, providing sufficient evidence as necessary for those decisions to be sound ones.LESSON 4 envision THE LIFE CYCLE OF BEST PRACTICES Practices (and practice information) are not placid and have a real life cycle. Majorparadigm shiftsin the software development world can have an impact on which practices are recommended. The practices that seemed to be good fits for most projects, when awaterfalllife cycle was the most common approach to software development, are not all equall y applicable at the current time, wheniterative, spiral, and even spry approaches are probably more representative of the state-of-the-art practice.Our recommendations regarding a organise life cycle for practice information are 1. A knowledge repository needs to be continually evolving by accepting information on topics of interest and making it available to users as soon as possible. While some quality checking is necessary to make sure that incorrect, misleading, or incomplete information is disseminated outward, it is better to get information to users as it comes in, than to bide and try to create something perfect.Users should be able to see a timestamp on all information so that they can see if the experiences related are fresh and up to date or come from years ago. 2. However, the desire to get information out quickly should not interfere with the need for validation activities that provide higher confidence in the information. These additional levels of maturity should b e noted, to give users more confidence in the information they find, but should not be used as apreconditionfor displaying content. 3.Content needs to be retired when appropriate. Practices may have a natural lifespan, since the acquisition and development worlds continue to burgeon forth and change on their own. Practices that were good 10 years ago may not be appropriate given todays constraints or technologies. To avoid users finding obsolete information in the repository, reports need to be generated periodically of which practices have received no updates or new experiences in the longest time. LESSON 5 APPLY AGILE STRATEGIES AND PROTOTYPINGTo create the front end of the BPCh tool, which helps users find vista practices, explore possibilities, and get more information on practices of real interest, we have found that prototyping and agile strategies are super valuable for developing knowledge-management systems. on the nose because of the need for parallel activities in dif ferent tracks, and the number ofstakeholdersinvolved (tool developers, content gathering team, end user representatives, sponsor representatives), an agile approach is extremely valuable.The implementation of the prototype BPCh tool was carried on in two-week increments, at the end of which a releasable version was always available. At the end of each two-week period, a demonstration and planning meeting was held with as many of the stakeholders as could be present. This approach was necessary to help us coordinate andprioritizethe evolving expectations of the users as well as the necessary changes that were suggested by the content development team, based on what they were finding. As part of this meeting we learned the following lesson LESSON 6 USE APPROPRIATE LANGUAGESpeak to the users in their language. Do not expect them to learn yours. We realized early on that having the greatest possible content in the BPCh repository would not be of much help if the users cannot find it. To address this we needed to provide multiple paths to the information, so that users could select the path that do the most sense to them. Some specific lessons learned here included 1. Organize around common tasks. The best way to reach users is to organize the contents of the repositoryaccording toeveryday activities that the user performs.This helps users see the repository less as an additional activity that they need to make time for, and more as a value-added to the activities that already consume their time. In the case of BPCh, we added several such perspectives (i. e. , indexes to the content) based around activities of importance to different segments of the user community (e. g. , addressingCMMIpractice areas, constructing a systems engineering strategy, and referencing back to common guidebooks). 2. squeeze as well as pull information.Rather than always expecting users to take time to come to browse the BPCh tool, information can be pushed outward to the user on a perio dic basis. For example, the user could select some practices of surplus interest, and when new experiences come in related to these practices a notification is sent via e-mail. 3. Match users to practices based on context similarity. Since no practice will be best for every project, it is important to match users to practices using context characteristics. This provides the users with a pick list of practices that may be useful in their particular patch, in ddition, it may alert the user to practices that they might not have known about previously. For example, if the user selects a few context variables that describe his/her context, then practices can be prioritized and displayed according to whether they have associated evidence provided by users with similar context information. This is a way of indicating that, even if the practice does not answer a specific search query, users like the current one have found this practice useful and it may be something the user should know. LESSON 7 DEMONSTRATE PRACTICAL EXAMPLES TO INTENDED USERTo engage in effective outreach activities, aimed at building up an implicated and active community of users of the BPCh, we find the following lesson of relevance You can not show initial users an emptydepository. In line with the idea that building a tool like the BPCh needs to proceed on three tracks in parallel (front-end, content, and outreach) is the lesson that populating the content cannot come after the repository is built. Showing users a stick out front-end without an initial set of real content may get their interest for a short time period, but is not an effective way of building an active user community.Users need to see a littler but representative set of content which they can respond to and start generating ideas for the next content or tool release. LESSON 8 UPDATE CONTENT AND FUNCTIONALITY CONTINUOUSLY To keep interest engaged, when users do check back to the order they need to see that updates have been made since last time. Content needs to be continuously updated andrefreshedto stay abreast of trends. If users ever become convinced that the repository does not get updated on a first-string basis, this often spells the end of their involvement.Rather, they need to be cause to come back often enough to find new things and hopefully, as they progress, be motivated to submit responses and ideas of their own showing emerging trends and keeping the content relevant. Thus, user involvement tends to build more user involvement. As users become interested enough to post comments or sendnew ideasto the repository, other users will continue to be interested to show up to see which comments have been added since the last time and possibly find something of interest to their current situationand more likely to find something applicable.One way we have experimented withto reinforce this conceptis to list on the front page of the BPCh tool the most lately added practices and highlight ones th at have been promoted to various maturity levels (Bronze, Silver, or Gold). Thus, one of thefirst thingsusers see is an indicator of how much progress has occurred since their last visit. CONCLUSIONS This article has presented some of the lessons learned with the BPCh program, which aims to document practices and quickly disseminate them to the users. The BPCh, which is based on the EMPEROR approach, makes use of a two-dimensional rating scale.These scales provide users with a quick overview of the trustability and maturity of the stored practice records. The scales allow users to understand and to draw their own conclusions based on a set of evidence from different contexts, from research studies as well as industrial experiences, and using measures at different levels ofrigor. Practitioners can rely on this information without reading in detail through the different evidence sources, unless they are interested in the very minute level of information.In addition, ways to collect u ser feedback and trigger discussions are offered to allow a vivid and growing user community. While initial feedback regarding the BPCh tool has been positive (Turner & Shull, 2005), we are continuing to improve the BPCh program and its associated tool through ongoing research, advisory groups, and user community feedback. We are interested in addressing such questions as How much extra effort tocertifyevidence sets and summaries as correct is worthwhile to users? or Are there subsets or types of evidence that users will find especially worthwhile? We invite you to take a look at our BPCh tool, available at http//bpch. dau. mil. We appreciate all feedback, whether it be submitted through the tool or directly to the authors e-mail. ACKNOWLEDGMENTS This research was back up with funding from the U. S. Department of Defense (DoD), theOffice of the Secretary of Defense(OSD), and the Defense Acquisition University (DAU). We wish to thank the segments of the BPCh team, from DAU, FC-MD, CSC, and SSCI, for the many productive discussions that have amend this work. REFERENCES Basili, V.R, Caldiera, G. , & Rombach, H. D. (1994). Experience factory. In J. J. Marciniak (Ed. ),Encyclopediaof Software Engineering (Vol. 1, pp. 469-476). New York rear end Wiley& Sons, Inc. Basili, V. , Zelkowitz, M. , McGarry, E, Page, J. , Waligora, S. , & Pajerski, R. (1995). SELs software process improvement program. IEEE Software, 12(6), 83-87. Dangle, K. , Dwinnell, L. , Hickok, J. , Turner, R. (2005, May). Introducing the Department of Defense acquisition best practices clearinghouse. CrossTalk, 18(5), 4-5. Defense Acquisition University.Retrieved from http//bpch. dau. mil Delphi White Paper. (2002). Taxonomy Content Classificationmart Milestone Report. Boston, MA Delphi Group. Koennecker, A. , Jeffery, R. , & Low, G. (2000, April). Implementing an experience factory based on existing organizational knowledge. In Proceedings of the 2000Australian Software Engineering Conference(pp. 28-29), Canberra, ACT, Australia. Mendonca, M. , Seaman, C. , Basili, V. R. , & Kim, Y. M. (2001, June). A prototype experience management system for a software consulting organization.In Proceedings of the 13thInternational Conference on Software Engineeringand Knowledge Engineering (SEKE). Ottawa, Canada. Schneider, K. , Schwinn, T. (2001, June). Maturing experience base concepts at DaimlerChrysler. Software Process-Improvement and Practice, 6(2), 85-96. Turner, R. , Shull, F. (2005, November). An observational approach to best practice identification and selection The U. S. Department of Defense acquisition best practices clearinghouse. In Proceedings of the 4th International Symposium on Empirical Software Engineering (ISESE2005)(pp. 33-140), Noosa Heads, Australia. Mr. Raimund L. Feldmann is the technical lead for Knowledge and Experience Management at the Fraunhofer touch for Experimental Software Engineering, MD (FC-MD). in the lead he joined FC-MD in 2004, Raimund part icipated in several technology transportation projects in Germany and was also involved in the development of the Virtual Software Engineering Competence Center (VSEK) portal, funded by the Department of Education and research (bmb+f) of the German Federal Government, to offer up-to-date Software Engineering knowledge to subject matter experts. E-mail address emailprotected umd. edu) Mrs. Michele A. Shaw is a Scientist at the Fraunhofer Center for Experimental Software Engineering. Michele supports clients implementing process improvement, measurement, and experience factory concepts. She has over 25 years of experience in Information Technology including software and expediency development, project management, quality assurance, client care andsubcontractormanagement Ms Shaw holds a BS in Business fromUniversity of Baltimoreand a masters in appliedbehavioral sciencefromJohns HopkinsUniversity. (E-mail address emailprotected edu) Dr. Forrest Shull is a senior scientist at the Fra unhofer Center for Experimental Software Engineering, MD (FC-MD). He is project manager and member of technical staff for projects with clients that have included Fujitsu, Motorola, NASA, and the U. S. Department of Defense. He has also been lead researcher on grants from the National Science Foundation, Department of Energy, bank line Force Research Labs, and NASAs Office of Safety and Mission Assurance. (E-mail address emailprotected umd. edu)