Expert Systems with Applications 38 (2011) 80548065

Contents lists available at ScienceDirect

Expert Systems with Applications

journal homepage: www.elsevier.com/locate/eswa

Benchmarking enhancements to a decision support system for global crop production assessments
Wim van Leeuwen a,b,, Chuck Hutchinson a, Sam Drake a, Brad Doorn c, Verne Kaupp d, Tim Haithcoat d, Vladislav Likholetov d, Ed Sheffner e, Dave Tralli f
a

School of Natural Resources and the Environment/Ofce of Arid Lands Studies, University of Arizona 1955E. 6th St., Tucson, AZ 85721, United States School of Geography and Development, University of Arizona, Tucson, AZ 85721, United States USDA/FAS/OGU/IPAD, Washington, DC 20250, United States d University of Missouri, Columbia, MO 65211, United States e NASA Ames Research Center, Moffett Field, CA 94035, United States f National Space Technology Applications, Earth Science and Technology Directorate, Jet Propulsion Laboratory, CA, United States
b c

a r t i c l e

i n f o

a b s t r a c t
The Ofce of Global Analysis/International Production Assessment Division (OGA/IPAD) of the United States Department of Agriculture Foreign Agricultural Service (USDA-FAS) has been assimilating new data and information products from agencies such as the National Aeronautics and Space Administration (NASA) into its operational decision support system (DSS). The FAS mission is to improve monthly estimates of global production of major agricultural commodities and provide US Government senior decision makers and the public the most accurate, timely, and objective assessment of the global food supply situation possible. These estimates are ultimately captured as the US governments ofcial assessments of world food supply for the commodity markets and policy makers. The goal of this research was to measure changes in the quality and accuracy of decision support information resulting from the assimilation of new NASA products in the DSS. We gathered both qualitative and quantitative information through questionnaires and interviews to benchmark these changes. We used an interactive project lifecycle risk management tool developed for NASA mission spaceight design and quality assurance (DDP Defect Detection and Prevention) to do this. In this case, we used it to (1) quantify the change in DSS Objectives attained after assimilation of new products, and (2) evaluate the effectiveness of various Mitigation options against potential Risks. The change in Objectives attainment was considered the most important benchmarking indicator for examining the effectiveness of the assimilation of NASA products into OGA/IPADs DSS. From this research emerged a novel model for benchmarking DSSs that (1) promotes continuity and synergy within and between government agencies, (2) accommodates scientic, operational and architectural dynamics, and (3) facilitates transfer of knowledge among research, management, and decision-making agencies. 2010 Elsevier Ltd. All rights reserved.

Keywords: Agricultural production Risk assessment Project management Cross agency investments World food security

1. Introduction The use of Earth science data, models and geographic information systems in agricultural monitoring and assessment continues to expand our ability to understand the impacts that climate variability, landscape change, and anthropogenic and economic forces have on global agricultural production (CCSP, 2008). The most important responsibilities of the Ofce of Global Analysis/International Production Assessment Division (OGA/IPAD; formerly the Production Estimates and Crop Assessment Division PECAD) of
Corresponding author at: School of Natural Resources and the Environment/ Ofce of Arid Lands Studies, University of Arizona 1955E. 6th St., Tucson, AZ 85721, United States. Tel.: +1 520 626 0058; fax: +1 520 621 3816. E-mail address: leeuw@email.arizona.edu (W. van Leeuwen).
0957-4174/$ - see front matter 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.eswa.2010.12.145

the United States Department of Agriculture Foreign Agricultural Service (USDA-FAS; created in 1953) are producing assessments of global crop conditions and monthly estimates of planted area, yield, and production for selected commodities like soybeans, wheat, corn, rice, cotton, and oilseeds (Hammond, 1975; Hutchinson, 1991). IPADs assessments are intended to promote the development of new initiatives directed at expanding US agricultural exports, combating world food insecurity, monitoring global agricultural change, and improving US crop condition and disaster assessments. IPADs main goal is to collect and analyze global climate, biophysical, crop, economic and eld reference data, produce the most accurate production estimates and decision support information possible from these data, and then disseminate timely, objective, useful, and cost-effective global crop condition and agricultural

W. van Leeuwen et al. / Expert Systems with Applications 38 (2011) 80548065

8055

market intelligence information with a high level of condence in the production estimates. Low condence forecasts can translate into more volatile markets where food shortages and over-stocks are more likely to occur. Stability of food prices requires a delicate balance between food supply and demand. Accurate assessments and forecasts of the global food supply help achieve this balance. These food supply estimates are arguably the most scrutinized and comprehensive in the world. The collaboration between USDA and the National Aeronautics and Space Administration (NASA) to improve the accuracy and timeliness of global crop production forecasts is an important component in helping to achieve market stability. This collaboration continues to introduce new NASA Earth science and measurements, model predictions, and information technology to enhance decision support for the IPAD. IPAD has developed a complex data- and model-driven decision support system that increasingly provides structure to the decision-making process and leads to more consistent and reliable crop production estimates for all countries potentially inuencing US agricultural industries and policies. Enhanced decision-support derived from Earth science and the satellite perspective and based on its comprehensive view provides the objective, global, and farm-level information necessary to assess world-wide production throughout the growing season and improves the USDAs capability to serve its management and policy responsibilities to society. As US and global socio-economic pressures increase due to globalization, population pressure, resource depletion and global climate change, IPAD analysts are facing rapidly increasing information demands. As a result, IPAD is gradually incorporating more advanced data, model, and technology systems to enhance the efciency and accuracy of global commodity estimates. Since 1974, NASA and the USDA have collaborated intermittently on research in remote sensing of agriculture (Hammond, 1975; Macdonald & Hall, 1980). More recently NASA and academic research partners have been collaborating with the USDA through a series of projects that have worked to assimilate NASA products to improve IPADs decision support system (DSS). Operational use of new products (data, information, models) within an organization is not purely a technical issue. A decision support tool operates within a broader decision making system that is based on data and technology but is driven by the experience, preferences and perceptions of individuals and groups inside the organization who operate it. It is within this conceptual structure that any integration effort must be benchmarked to assess adverse effects, probability of success, and operational hurdles that must be overcome technical or organizational. In this study, systematic benchmarking techniques (Lucertini, Nicol, & Telmon, 1995; Spendolini, 1992) were used to evaluate the effectiveness of risk-reducing mitigations like the assimilation and utilization of NASA Earth Observation System (EOS) (Justice et al., 1998) data. The Defect Detection and Prevention (DDP) risk management software tool developed by NASAs Jet Propulsion Laboratory (JPL) (Cornford, 1998; Feather, 2001; Feather, Cornford, & Gibbel, 2000; Gindorf & Cornford, 1999) was employed to quantify the effectiveness of the enhancements, using risk balance and attainment of objectives as performance indicators. The focus of this research was the application of a novel benchmarking approach to assess the impact of enhancements on a complex operational decision support system used by IPAD. The benchmarking process measures the change or delta between pre and post-enhancement states of the DSS by using metrics that quantify changes in the DSS after infusion of NASA science, data and technology. The outcome of this benchmarking is intended to quantify how investments from USDA and NASA have enhanced the performance of IPADs DSS. It is increasingly important for NASAs Earth Sciences Division to integrate NASA solutions into DSSs (McCuistion & Birk, 2005). Although other

agencies have also contributed to the enhancements of IPADs DSS, benchmarked in this study, NASA products are the focus to illustrate how environmental remote sensing science and data assist IPAD to attain its mission objectives and requirements. 2. Production assessment decisions support system and benchmark tools The following section describes the IPAD DSS. Subsequent sections provide a description of the general benchmarking steps, the risk management tool, and the data surveys needed to quantify the DSS enhancements in terms of risks and attainment of IPADs objectives. A fundamental assumption underlying this benchmarking effort was that risk-management techniques could be adapted to measure and quantify the degree of improvement in Objectives attainment. Within this context, NASA Earth science products incorporated in the upgraded DSS were considered principal factors for mitigating risk and enhancing objectives attainment for the enhanced state of the DSS. 2.1. Global agricultural production DSS characterization and USDA framework 2.1.1. Background and perspective After the 1972 launch of the rst of the Landsat series of multispectral scanners on sun-synchronous orbiting satellites, NASA and USDA began a collaborative research program for using data from these sensors to estimate crop yield and crop condition of the worlds major agricultural regions. This collaboration was initiated in the mid-1970s with the Large Area Crop Inventory Experiment (LACIE) (MacDonald, Hall, & Erb, 1975) and continued with the Agriculture and Resources Inventory Surveys Through Aerospace Remote Sensing (AgRISTARS, 1981) program. The models developed under the LACIE program formed the foundation for the current operational USDA program. Crop-yield and crop-condition estimation models were developed under the AgRISTARS program. The operational renement of these historic programs is a much enhanced DSS known today as the OGA/IPAD; a derivative of the long-standing PECAD. These joint efforts evolved to enable USDAs current operational programs to monitor, for example, rice, cotton, and soybeans in the Mississippi Delta and Brazil; corn in the American Midwest and South Africa; and wheat on the American Prairie and the plains of Central Asia. The USDAs FAS, with responsibility to monitor the world production of economically important crops, has become a major consumer of Landsat data. Several other satellite data sets, which complement the Landsat series, are also important to the FAS (e.g. Advanced Very High Resolution Radiometer [AVHRR] from NOAA) and are heavily used as well. Congress, agribusiness, the State Department, the US Agency for International Development and the United Nations are among routine users of these satellite-based analyses. 2.1.2. IPAD DSS characterization and description An overview of this DSS for assessing worldwide agricultural production is provided in Fig. 1. Fig. 1 shows how global data and model input sources and analysis tools for estimating crop production drive this decision support system that informs stakeholders and benets global citizens and societies. DSS enhancements are continuously being implemented to reduce risks and accomplish the mission of IPAD and USDA. Fig. 1 shows how multiple data, information and model results are the basis for the processing, analysis and visualization techniques that lead by way of a convergence of evidence approach to monthly, country specic commodity production estimates. It also indicates the groups of

8056

W. van Leeuwen et al. / Expert Systems with Applications 38 (2011) 80548065

Fig. 1. IPAD A DSS for estimating global crop production from disparate data.

experts that engage in this complex system of systems. Moreover, Fig. 1 also highlights how feedback mechanisms between DSS development, enhancement and risk mitigation activities accomplish DSS objectives, and have value and benet for society. An important rst step in benchmarking improvements to the IPAD DSS was to develop a baseline characterization of the initial state, its working environment and constraints, including the identication of issues and potential solutions. IPAD management, information technology staff, and analysts were interviewed during a 2-week period in 2003 and again in 2005. We developed several questionnaires ($50 questions) in collaboration with IPAD personnel to both characterize the DSS and provide benchmark information. Interviews and paper surveys provided information about personnel and stakeholders involved, DSS functioning, input data and model requirements, and provided insight into current

and future Earth observation needs, including temporal and spatial coverage needs, timeliness considerations, and accuracy and consistency requirements. Twelve IPAD regional analysts each use the convergence of independent evidence approach to assess, estimate and forecast national-level crop production for a selected set of commodities within major agricultural regions of the world (Fig. 1). This methodology, in effect, minimizes risk of error or bias and maximizes reliability by integrating multiple independent sources of information with past trends so that estimates are not driven by a single data source. IPADs DSS, thus, assimilates current satellite remote sensing data, meteorological data, crop models, economic data, ground observation and external data sources (e.g. newspaper reports) to derive information about general crop condition and converge to preliminary production estimates based on observed crop

W. van Leeuwen et al. / Expert Systems with Applications 38 (2011) 80548065

8057

areas and projected yields (production = yield area). Production assessments based on biometeorological and environmental data are compared to FAS country attach reports, in-country economic information, wire services, and analysts personal knowledge (e.g. eld reconnaissance trips) and rened by comparing the results with recent trending and similar historical periods to provide a formal estimate of national and regional crop production on or about the 10th day of each month. The FAS has a global network of attachs that provide on-the-ground reports of observed crop and contextual information. The FAS/IPAD regional analysts travel extensively in the countries they cover to more fully develop the context and constraints within which their assessments are made. Each analyst uses the available data sources in a variety of ways, with differences in priority and frequency related to factors such as temporal and spatial coverage, timeliness, accuracy, consistency, quality, crop specic phenology and other regional and commodity traits. IPADs DSS allows analysts to spend less time on repetitive analysis tasks and more time on utilizing all available data for their monthly crop production assessments. The needs for and utility of satellite and other remotely sensed data are increasing. Most new DSS developments are related to automation of algorithms, analysis tools, and visualization of spatial and time series of environmental data. These developments will make data, model results and ancillary data more easily accessible, thus improving responsiveness and consistency. IPADs DSS has several user-friendly web interfaces, which provide intuitive decision support tools to the analysts, such as CropExplorer (http://www.pecad.fas.usda.gov/cropexplorer/) and CADRE (Crop Assessment Data Retrieval & Evaluation). These tools provide communicative crop condition information for most agricultural regions in the world based on weather data and satellite imagery. CropExplorer is an interactive spatially explicit webbased tool that provides timely time series of satellite NDVI imagery and graphics for worldwide crop regions. CADRE (Crop Condition Data Retrieval and Evaluation) is one of the principal decision support tools used by IPAD analysts. CADRE is a grid-based, geospatial data base management system (DBMS), which stores numerous land surface and weather data sets and performs various modeling functions. CADRE has been redesigned over time to integrate new data sources and applications, and has changed to reect new technologies (e.g. grid size representation, regionalization of databases, etc.). Most IPAD analysts use a number of historical comparisons to create a production assessment. Therefore, the legacy data and systems represented within CADRE continue to be integrated into new congurations so that IPAD can leverage that investment in the future. Through this process, IPAD has developed a historical database stretching back over 20 years (CADRE; Fig. 1), enabling their expert analysts to compare current and prior conditions and bring a long-term perspective to their evaluations. From the range of available data, IPAD can also provide early warning of unusual crop conditions or changes in the production outlook for a country or region. This provides the USDA with an information edge and assists the marketplace in price determination and adjustments. The World Agricultural Outlook Board (WAOB) and Interagency Commodity Estimates Committee (ICEC) must verify and endorse all nal decisions about the global production forecasts. At this stage, other contextual information such as ofcial governmental reports, trade and news sources play a significant role in interpreting how these factors will affect price and policies and other econometric analysis conducted. The DSS also has the additional duty, through intra-USDA agreements and the USDA charter, to support domestic USDA agencies with early warning and condition assessment of US crops affected by events such as ooding and hurricanes. The USDA is managing the Satellite Imagery Archive to support these assessments. The

FAS also provides expertise, satellite and crop data to the USAID Famine Early Warning System (FEWS) for Africa. Many stakeholders benet from the crop production estimates as well as the environmental observations that went into the decision making process to estimate the monthly agricultural commodity production. Several examples of outcomes and benets to society are shown in Fig. 1. The global crop production estimates impact global food production policies (Fig. 1; right side column), the US Agricultural industry and commodity traders worldwide. The priorities of the Ofce of Global Analysis (OGA) are driven by events around the world (e.g. climate, politics, wars, natural hazards and severe meteorological events) and their effects on the global agricultural market. Although the focus is non-US, global production forecasts for positioning domestic interests in the world markets of major grains and oil seeds are key components to the success of US agriculture. The aim of FAS is to provide reliable and accurate information on US competitors, US strategic interests, and food-insecure regions around the world.

2.1.3. IPAD DSS continuing development During the course of this research, a more structured process was being pursued for evaluation, prioritization, and standardization of IT (Information Technology) development that supports the IPAD group. In the past (and to a certain extent now) tasking was ad hoc resulting in major projects being incomplete in their design while more trivial tasks were being managed and taken care of. The rst attempt at using a more structured approach is represented by the Crop Explorer website development. The application developers worked to synthesize and optimize the interactions with the database while responding to user and analyst requests for implementation within the web tool. Future enhancement phases to Crop Explorer are targeted to be more user-dened. Stakeholders of new information products are expected to reach and document a consensus as a product evolves through the application process. It should be emphasized that the DSS development and benchmarking process is best done by including all aspects of the DSS (management, users, tools, information technology, and data sources) as the assimilation of a product will affect overall system performance and individual components (e.g. user interface, knowledge and problem processing system) of the DSS (Fig. 1). Recent development plans to migrate from AVHRR (Advanced Very High Resolution Radiometer) to MODIS (Moderate Resolution Imaging Spectroradiometer) satellite vegetation data included two major components image processing and GIS. Both must be enabled to move into and support these new applications. A new image analysis framework was developed to process all images in an integrated environment to minimize replication and disparity and to optimize automation. Other important system specications continue to be important and include scalability, robustness, and the need for near and off-line archiving and storage associated with a DBMS. The analysis framework supports different media and maintains an US Federal Geographic Data Committee (FGDC) compliant archive. Other reasons for the infusion of new data and technology are the need for (1) redundancy in case a satellite system fails, and (2) continuity of Earth system observation and science, which are pertinent to the use, and extension of the historical data archives. Enhanced soil moisture, crop area and crop yield models (Jame & Cutforth, 1996; Reynolds et al., 2000) are being integrated into IPADs DSS. Most of these models currently depend on weather and climate data rather than satellite vegetation data. Although more satellite data is expected to be included in these models, it remains important to crop production analysts to maintain a stream of independent sources of environmental data and informa-

8058

W. van Leeuwen et al. / Expert Systems with Applications 38 (2011) 80548065

tion that can be used towards the convergence of evidence and production estimates. The perceived benets of IPADs evolving DSS are: improved quality of crop assessment and production estimates and decisions, improved communication among its users, cost reduction, increased productivity, timesavings, and improved customer and employee satisfaction. IPADs DSS continues to improve in userfriendliness, and its capacity to conduct special analysis. 2.2. Defect detection and prevention (DDP) as a benchmarking tool The outcome of the IPAD benchmarking process was intended to show how investments from USDA and NASA change IPADs DSS. The DSS benchmarking process used metrics to indicate performance improvements and changes of the DSS from State 1 based on current input of NASAs science, data and technology in comparison with the State 2 DSS incorporating new NASA data, science and technology (e.g. MODIS products) (Kaupp, Hutchinson, Van Leeuwen, Drake, & Tuyahov, 2003). Usability indicators and dened performance metrics are critical for assessing changes to the DSS. Both qualitative and quantitative information were gathered to benchmark the DSS. This involved the use of questionnaires and interviews, and risk assessment software (i.e. DDP). The IPAD-DSS description (Hutchinson, van Leeuwen, Drake, Kaupp, & Haithcoat, 2003) provides additional information about the current characteristics of IPADs DSS and aided in the set-up of a benchmarking process and selection of performance indicators. At about the time that the benchmarking team, consisting of IPAD analysts, NASA investigators and academics, was examining options for an objective approach to benchmarking, the Jet Propulsion Laboratory was expanding the applications of the Defect Detection and Prevention (DDP) (Cornford, 1998; Feather, 2001; Feather et al., 2000; Gindorf & Cornford, 1999) risk-based approach for planning and managing technology development. DDP is a project lifecycle risk management decision-support software tool for the design of complex space borne systems and mission quality assurance. Herein, the concept was advanced in that NASA data assimilation into a DSS is a form of technology development and planning that could be served by expanding the application of DDP to programmatic risk balancing. The use of DDP was further suggested by the fact that the data assimilation would likely entail architectural changes in how OGA/IPAD (then PECAD) approached the assimilation process itself into its decision support system. 2.2.1. DDP implementation and analysis approach DDP is an interactive software tool that establishes the relative signicance of specic Risk elements by evaluating the impact of their occurrence on system requirements or Objectives, and quanties the effectiveness of particular Mitigations for offsetting the Risks to attain the Objectives. Because of its interactive nature, DDP can also be considered a process, mediated by the software tool. The DDP process of eliciting and structuring Objectives, Risks and Mitigations relies on an initial phase of data collection from a critical mass of experts on the system under evaluation. Another example of knowledge elicitation in agriculture from multiple experts is described by Lger and Olivier (2009). In this case, OGA/ IPADs crop analysts acted as experts on the DSS. The critical mass of expertise can be harvested either through open meetings involving point-by-point discussion and consensus building among the experts, or through survey questionnaires that collect individual expert opinions for aggregation. The analysis team (authors) chose the latter option given the difculty of assembling all the analysts for any meaningful length of time. The intent of using the DDP tool was to systematically quantify the performance differences between DSS States. The strength of the DDP tool was realized by performing before-and-after analyses,

quantifying and visualizing the Risk proles and Objectives attainment. For benchmarking the OGA/IPAD DSS, the relevant analyses were those based on the different combinations of Mitigations (NASA inputs in particular) in operational use at State 0 (no NASA inputs), State 1 (only older-generation inputs such as AVHRR data composites), State 2 (older- and new-generation inputs including MODIS products and reservoir heights). The implemented DDP process was accomplished in six processing steps: 1. Identify all Objectives of the DSS for State 2 and rate their relative importance (Fig. 1). 2. List the Risk factors (their a priori likelihood of occurring is initially assumed to be equal) which affect attainment of Objectives or limit DSS effectiveness (e.g. lack of data, usefulness of tools) (Table 2). 3. Establish State 2 Mitigations (e.g. assimilated NASA products) that reduce the Risks and improve attainment of Objectives (Table 3). 4. Appraise the impacts of the Risks on the DSS Objectives (if a Risk occurs, what is the loss in attainment of Objectives). 5. Assess the effectiveness of the Mitigations for offsetting the impact of the Risks. 6. Examine and compare residual-Risk proles and Objectivesattainment scores with different selections of Mitigations representing States 0, 1 and 2. Fig. 2 shows a schematic diagram of the DDP tool. Impact of a particular Risk is quantied as the percentage or fraction of a particular Objective that is not attained, and ranges from 0 for no impact to 1 for catastrophic impact. The impact of a risk is comprised of the likelihood of the risk occurring and the severity of such occurrence. Typically, it is assumed that a Risk has an a priori likelihood of 100% it will occur if nothing is done. Severity or impact

Fig. 2. Defect detection and prevention roadmap towards risk management and DSS benchmarking.

W. van Leeuwen et al. / Expert Systems with Applications 38 (2011) 80548065

8059

of each Risk on Objectives was provided by OGA/IPAD Analysts. Impacts are additive and an Objective can potentially be more than 100% at Risk depending on the number and impact values of Risks that are imposed on it. Typically in the application of DDP, it is assumed that the severity of a Risk cannot be changed and, therefore, Mitigations are measures to reduce their likelihood of occurrence. The effectiveness of each Mitigation is quantied as the percentage or fraction of each Risk that is offset, again ranging from 0, a Mitigation with no effect on (usually, not applicable to) a particular Risk, to 1, a Mitigation that fully offsets or retires a particular Risk. Numbers to populate these matrices were obtained from the responses of OGA/IPAD analysts to a specically tailored DDP questionnaire. This survey was designed to capture the changes improvements analysts asserted accompanied the assimilation of NASA products. This assessment captured changes resulting from assimilation of products such as MODIS vegetation data and Topex/Jason reservoir heights. This is referred to as the transition from State 1 to State 2 of the OGA/IPAD DSS. Not all risks and mitigation factors were clearly associated with all requirements. That is, the impact of some risks on some requirements could be marked as none or unknown. Similarly, the effectiveness of particular mitigation factors to reduce particular risks could be none or unknown as well. An example of one of the 37 questions and answers used to establish relationships among Objectives, Risks and Mitigations is given below: What is the impact of the Lack of continuity of MODIS data on crop area estimates: None (a) Early in the crop growing-season? (b) Late in the crop growing-season? Low Medium High Unknown

By simple mathematical operations on the matrices, the main DDP output provided two key integrated DSS performance metrics: (1) an Objectives attainment score, and (2) a residual Risk balance prole (Fig. 2). Analysis, summary and comparison of these metrics at States 0, 1, and 2 provided a measure of the difference or delta between States, and demonstrated the value of the modications that moved the DSS from baseline to target. These can be displayed by the DDP tool in bar charts or spider charts similar to the graphs in this manuscript. Another branch of analysis to measure change between States 0, 1, and 2 was pursued using the DDP tool to simulate the States, using the data collected in the 2005 survey. The DDP tool lends itself to such objective simulations and the exploration of a range of scenarios. Any of the listed Objectives, Risks and Mitigations can be turned off and on by the user of the DDP tool to evaluate attainment of certain objectives and determine which risk impact objectives and evaluate the effectiveness of certain mitigations. The DDP tool can also be set up to run through all these combinations and perform a system-wide sensitivity analysis. The visualizations generated by the DDP tool graphically portray progress towards attaining system Objectives or requirements. In all cases, the outcomes are dependent on the matrix linkages between Objectives and Risks, and Mitigations and Risks. Therefore, the architecture of the system and the functionality of DDP provide a comprehensive systems engineering perspective to the benchmarking activity. 3. Results and discussion The following sections demonstrate how the DDP tool was congured to systematically quantify the performance differences between States 0, 1 and State 2 of the DSS development history. Integration of the interview and survey results within the DDP environment shows how the DDP is utilized to prioritize the Objectives, weight the impact of the Risks, and evaluate the effect of various Mitigation combinations at reducing these Risks while attaining Objectives. 3.1. Benchmarking IPADs DSS DDP structure and baseline Weaknesses (i.e. Risks) in the DSS are addressed through improvement (i.e. Mitigations) of data continuity and data-latency, and through the development of data and tools that can accomplish requirements (i.e. Objectives) listed in Table 1. IPADs primary objectives are to provide accurate monthly production estimates by commodity and country, and to present evidence to the World Agricultural Outlook Board (WAOB) to support these estimates.

The answers to this survey were translated into relative values of 0, 0.1, 0.5, 0.9, and for None, Low, Medium, High and Unknown respectively. Consequently, the geometric means of these values were entered into the DDP tool, resulting in an Objective x Risk impact matrix, and a Mitigation x Risk effectiveness matrix (Fig. 2). The matrices were used to evaluate risk balance scenarios and attainment of Objectives. The relative importance of IPADs requirements and objectives was determined through discussion with IPAD collaborators.
Table 1 IPAD requirements that are mandated by the USDA/FAS. Weights 10 5 2.5 2.5 5 1 1 1 1 1 5 5 5 2.5 2.5 5 Objectives Requirements

1. Provide accurate monthly production numbers by commodity and country using: 1.1. Crop area estimates: 1.1.1. Early in the crop growing season 1.1.2. Late in the crop growing season 1.2. Crop yield estimates: 1.2.1. Pre-planting stage 1.2.2. Planting stage 1.2.3. Vegetative growth stage 1.2.4. Reproductive stage 1.2.5. Grain lling stage 2. Provide credible supporting evidence at lock-up 3. Provide bi-weekly web updates for countries/regions during the growing season 4. Provide automated analysis products to users (analysts, external customers) and extend tools (e.g. Crop Explorer): 4.1. Deliver timely data through automated applications such as Crop Explorer and Reservoir Monitor 4.2. Maintain historical archives for crop area/yield/ production (PS&D), weather data (CADRE), and satellite imagery (Archive Explorer) 5. Have an ad hoc analysis capability, including US crop condition assessment

8060

W. van Leeuwen et al. / Expert Systems with Applications 38 (2011) 80548065

IPAD is also required to provide and archive web updates and automated analytical products to external customers (i.e. the rest of the world) and have an ad hoc analysis capability to disseminate timely global crop condition and production information (Table 1). The relative importance of each of the Objectives and sub-objectives are listed in the Weight column in Table 1, and represent priorities related to IPADs mission. The sum of the weights of the subobjectives (indented) that apply to State 0,1 and 2 is equal to the total weight of the main Objective (without indentation) (Table 1). Risks that could impede attainment of IPADs requirements are listed in Table 2. Risks that we identied include: lack of continuous access to national- and regional-level multispectral observations (e.g. AVHRR, MODIS); delay in receiving timely remote sensing data; obstacles to operational use of remote sensing data within IPAD (i.e. facilities, tools, and training); the lack of baseline or historical information specic to agricultural regions (Table 2). Recently developed experimental products are often too costly to introduce to the decision making system (integration of hardware and software, staff training). These products are considered ad hoc and temporary solutions to ll data gaps. Discontinuity in the availability of consistent global LANDSAT-like data (e.g. LANDSAT

5-7, SPOT, ALI and IRS) used, among others, for detecting changing land use and land cover types is resulting in the loss of an important line of evidence. The impact of these Risks on IPADs mission can be reduced through integration of mitigation factors such as IPAD resources (analysts, attach eld data, ground truth, and external data sources) and NASA data products. The state of Earth observations and their assimilation in decision making systems for agricultural efciency is in continuous transition due to development efforts by IPAD, NASA and academic collaborations. Table 3 gives an overview of the satellite products that have been and are being assimilated in IPADs DSS. IPADs development plan included the migration from AVHRR to MODIS data, requiring investments in new image processing and GIS technologies. Both must be enabled to support adoption of new remotely sensed products. New precipitation and soil moisture (e.g. AMSR-E [Advanced Microwave Scanning Radiometer Earth Observing System]) and crop type and area products that exploit new medium and high resolution satellite technology are being developed and evaluated, including tools for integration of these products in IPADs decision making system. Input product accuracy, temporal frequency, latency, coverage and

Table 2 Risks that were perceived to threaten accomplishment of DSS objectives. Risks 1. 2. 3. 4. 5. 6. 7. Lack of access to multi-date eld-level multispectral observations and information (e.g. Landsat-like) Lack of image processing and/or analysis tools for Landsat-like data (e.g. web services; GIS tools; limitations of hardware, software and network capacity) Inadequate timeliness and/or quality of Landsat-like data Lack of continuous access to national level multispectral observations (e.g. AVHRR/MODIS-like) Lack of image processing and/or analysis tools for AVHRR/MODIS-like data (e.g. web services, GIS tools; limitations of hardware, software and network capacity) Inadequate timeliness and/or quality of AVHRR/MODIS-like data Inadequate or lack of crop and soil moisture models (country- and crop-specic): 7.1. Crop stage models 7.2. Crop yield models 7.3. Soil moisture model/data 8. Inadequate accuracy, timeliness, reliability and usefulness of precipitation/weather data and analysis tools 9. Lack of a historical country- and commodity-specic area or yield data base or contextual reference data (e.g. crop regions, countries, provinces)

Table 3 Mitigations that were put in place at different times/states to enhance the DSS, mitigate risks and attain the objectives. The State in rst columns describes the relative timing sequence of implemented mitigations. State Mitigations 1. Land remote sensing products (with associated characteristics of timeliness and quality): 1.1. MODIS products: 1.1.1. MODIS Rapid Response products (daily; false color, NDVI) 1.1.2. MODIS Vegetation Index product (NDVI and EVI) and analysis tools (16-day composite standard product) 1.1.3. MODIS-derived Crop Mask product 1.2. AVHRR 1 km (daily, 10-day) 1.3. AVHRR 8 km NDVI 10-day composite (GIMMS) 1.4. SPOT-VEGETATION 10-day NDVI composite 1.5. LANDSAT- or ASTER-like data 1.6. Reservoir heights (Topex/Jason) 1.7. Surface wetness (e.g. NOAA/SSMI) 2. Multiple sources of ground-based weather data: 2.1. WMO (precipitation, Tair data; weather stations) 2.2. AFWA (precipitation data derived from weather stations) 3. FAS attach ground truth 4. External data sources (incl. Reuters, economic statistics, internet, PS&D online, etc.) 5. OGU/IPAD-obtained ground truth from crop travel 6. Having an experienced analyst for each crop and region 7. Availability of user-friendly and integrated tools and/or DBMS (continuously being maintained and extended): 7.1. CADRE DBMS as an aid in decision making (NDVI and precipitation) 7.2. Crop Explorer as a tool in aiding decision making (imagery and graphing of time series or difference imagery) including tools for integration of NASA data (e.g. MODIS RR, DBMS, MODIS NDVI time series graphing) 8. Current crop yield and soil moisture models 8.1. Crop stage models 8.2. Crop yield models 8.3. Soil moisture model

2 2 2 1 1 1 1 2 2 1 1 0 0 0 0 1 2 2 2 2 2

W. van Leeuwen et al. / Expert Systems with Applications 38 (2011) 80548065

8061

interoperability are key requirement metrics for the successful use of national and international sources of Earth observations in the convergence of evidence strategy and pursuit of timely estimates of global production and assessment of crop health.

State 0 State 1 State 2 A Priori 9 1 0 8 -1 -2 7.3 2

1 2

3.2. Benchmarking IPADs DSS DDP results The assessments from the paper surveys and interviews were implemented in the DDP tool to examine the effectiveness of the Mitigations to retire Risks and attain Objectives. Benchmarking performance results for the attainment of the objectives and the risk balance for States 0, 1 and 2 are presented in Figs. 3 and 4. State 0 represents the DSS when mitigations were mostly based on eld-based data, reports and a regional IPAD analyst with knowledge of the crops. State 1 shows the effect of a combination of additional mitigating factors such as satellite data and historical data bases and State 0 Mitigations. State 2 represents the enhancements from NASA based data and tool mitigations on top of State 1 Mitigations. The attainment of the Objectives for any of the states is computed as a portion of the total weight of all the Objectives (30) by applying the appropriate Mitigation options for States 0, 1, and 2, respectively. In other words, the level of attainment of Objectives is a measure of the effectiveness of the Mitigations in retiring Risk. Fig. 3 is a spider chart showing overall attainment of IPADs objectives for different mitigation categories and States after applying the DDP tool. Each axis or ray of the diagram represents an objective, and the total possible Objectives attainment (30) is represented as the outer weight envelope. As seen in Fig. 3, the attainment of Objectives increases for each successive State, with State 2 being the best, almost reaching the maximum possible attainment of objectives. The closer the polygons that represent different DSS States are to the outer envelope, the more fully the Objectives are attained. Clearly, this metric shows State 2 to be a very signicant improvement over previous States. A more detailed treatment of Objectives attainment is provided in Fig. 5. Fig. 4 shows the risk balance for each of the Risks as listed in Table 2. The A priori Risk balance decreased with the application of mitigations. The amount of risk for each of the Risks was very close to zero for State 2. Within a class of Mitigations such as land remote sensing data or weather data, it is common for the lack of

7.2

7.1

Fig. 4. Log scaled risk balance after applying Mitigations that reduced the Risks for States 0, 1, and 2. Short descriptions of the Risks and associated codes (1, 2, .., 7.1, .., 8, 9) are provided in Table 2.

one dataset to be partly or completely mitigated by the availability of other similar datasets, in the OGA/IPADs convergence-of-evidence approach to decision making. The addition of Mitigations at each successive DSS State accounts for most of the risk reduction in this way. At State 0 there is much unmitigated Risk due to a sparse set of available Mitigations, while at State 2, with all Mitigations active, the residual risk is vanishingly small at the log scale resolution of the graphic in Fig. 4. In summary, the attainment of Objectives was shown for each of the three progressive developmental phases or States of IPAD to be as follows, with 30 signifying 100% Objective attainment (Table 4). 3.3. Benchmarking IPADs DSS DDP analysis summary and discussion Fig. 5 lists the Mitigations that are applied to retire Risks and attain the Objectives. The when column indicates the State (N) at which a Mitigation is applied. The Attainment of Objectives column shows the attainment due to various Mitigation option sets. The intent is to identify the relative contributions of different Mitigations and Mitigation classes towards achieving State 2 without suggesting that any such combination is a realistic or realizable option. The total attainment of Objectives for State 2 (second questionnaire) is 28.3 out of 30 (column a, Fig. 5). Column a therefore is comprised of three mitigation classes: (1) Various data types, (2) analysis and visualization tools, and (3) models. The 30 is based on a top-down weighting of the 5 top-level requirements. The absolute weighting is arbitrary and the Objectives attainment would scale according to the total weighting applied. What is important is the relative weighting between Objectives, not the absolute weighting. Discussions with IPAD management and analysts led to the rst Objective (accurate crop production estimates) being weighted 2 to 1 in importance compared to the other four at the same hierarchy level in the Objectives tree (Fig. 5). Hence user requirement (Objective) 1 has a weight of 10 while the other four have a weight of 5. The absolute numeric value is arbitrary so long as the values represent a 2:1 ratio. In the top-down weighting used in DDP, the software assigns the corresponding weights to the objectives tree branches.

1.1.1 5 5 4 3 4.2 2 1 4.1 0 1.1.2

State 0 State 1 State 2 Weight 1.2.1

1.2.2

1.2.3

2 1.2.5

1.2.4

Fig. 3. Linearly scaled attainment levels of IPAD objectives for all mitigation categories for States 0, 1 and 2. The maximum Weight is equal to the total possible attainment for each of the objectives identied by the codes (1.1.1, 1.1.2, .., 4.2, 5) at each of the spider chart axes (see Table 1 for an explanation of these codes).

8062

W. van Leeuwen et al. / Expert Systems with Applications 38 (2011) 80548065

Mitigations
MODIS Rapid Response products (daily; false color, NDVI) MODIS Vegetation Index product (NDVI and MODIS EVI) and analysis tools (16-day composite products standard product) MODIS Surface Reflectance product (8-day composites)

When (N) a b

Objective Attainment c d e f

h 11.4

15.1 2 17.0 19.7 21.3 10.8

12.1 * 9.6 8.8 10.4 * 9.3

Land remote sensing products (with associated characteristics of timeliness and quality)

MODIS-derived Crop Mask product

Reservoir heights (Topex/Jason) Surface wetness (e.g., NOAA/SSMI) NASA satellite-derived precipitation (e.g., TRMM) AVHRR 1 km (daily, 10-day) AVHRR 8 km NDVI 10-day composite (GIMMS) SPOT-VEGETATION 10-day NDVI composite LANDSAT- or ASTER-like data

10.2 1 26.4 28.3 1 13.2 18.0 14.7

9.2 12.1 9.6 11.6 10.2

WMO (precipitation, Tair data; weather stations) Multiple sources of ground-based AFWA (precipitation data derived from weather weather data stations) FAS attache ground truth External data sources (incl. Reuters, economic statisticss, internet, PS&D online, etc.) PECAD-obtained ground truth from crop travel Having an experienced analyst for each crop and region CADRE DBMS as an aid in decision making (NDVI and precipitation) Crop Explorer as a tool in aiding decision making (imagery and graphing of time series or difference imagery) including tools for intregration of NASA data (e.g., MODIS RR, DBMS, MODIS NDVI time series graphing) Crop stage models Crop yield models Soil moisture model

State 0 = 8.4 STATE0 = 8.4

16.5

Availability of user-friendly and integrated tools and/or DBMS (continuously being maintained and extended)

1+

20.3 15.4 11.3 16.2 13.2

Current crop yield and soil moisture models

1+

20.9

Fig. 5. Summary of the benchmarking results based on the attainment of Objectives for each mitigation or combination of mitigations beyond State 0 as indicated in the when column.

Table 4 Objectives attainment (OA) results for Mitigations associated with States 0, 1, and 2 based on the DDP analyses. Absolute OA values obtained are provided for reference, only; results derived from comparisons showing the percentage of relative attainment. DDP Analysis 30 = 100% Attainment OA State 0 State 1 State 2 8.4 22.4 28.3 % OA 28.0 74.7 94.3 Ground-based data and information Addition of AVHRR, SPOT-VEGETATION and Landsat data, use of crop yield and stage, and soil moisture models Addition of MODIS data, TOPEX/Jason and NOAA/SSMI, and subsequent enhancements in crop yield, stage and soil moisture models Mitigations (State N)

Column b shows the Objective Attainment (OA), without using the crop models, to be 26.4. The OA for the data product alone is 21.3 (column c), and 20.3 for tools and DBMS. Of the 20.3 OA for tools and DBMS, 16.5 and 15.4 are their respective contributions. Recall from Section 3.1 that these Mitigation options are not simply additive because of the escape probability of their effectiveness on risk. The OA of the data product can be further decomposed into NASA and non-NASA elements with OAs of 17 and 18, respectively. Similarly for tools and DBMS, the OA is 16.5 and 15.4, respectively. Each value in Fig. 5 is only for the indicated Mitigations. No other Mitigation is employed in this exercise. This permits us to look at the respective contributions of any combination relative to State 2, recognizing that in some cases a combination may not be a realizable or meaningful set. The data OA of 21.3 listed in the table can also be disaggregated into a remote sensing component and a meteorological component with OAs of 19.7 and 13.2, respectively

the remote sensing component containing both NASA and nonNASA assets. A further comparison can be made between the OAs of State 1 and 2 for data products. State 1 consists of AVHRR, SPOT, Landsat with OA of 14.7 and 13.2 for WMO and AFWA meteorological data, respectively. The NASA-only data product set with an OA of 17 (column d) shows 15.1 for MODIS and 10.8 for NOAA/SMMI (i.e. surface wetness data). The individual MODIS, TOPEX, surface wetness, AVHRR SPOT, and LANDSAT, OA values can be found in Column h of Fig. 5. Finally, Fig. 5 shows that crop yield models, with all other Mitigations turned off, yield an OA of 16.2 compared with crop stage and soil moisture models with OAs of 11.3 and 13.2, respectively. Fig. 6 shows the attainment of IPADs objectives for different mitigation categories after applying the DDP tool. The sum of the attainment of the six mitigation categories exceeds the attainment

W. van Leeuwen et al. / Expert Systems with Applications 38 (2011) 80548065

8063

Attainment of Objectives (Max=30)

30 25 20 15 10 5 0 2

28.3 All Mitigations 13.2 14.7 20.3 Tools, CADRE, Cropexplorer 20.9 10.8 15.1

Crop Models, Soil moisture models

models. At the same time the smooth progression of the lack of glaringly high sensitivity suggests that the system architecture, namely the linkages between Objectives, Mitigations and Risks t the situation being tested, and further substantiate the principal outcomes of the benchmarking as discussed above in the previous sections.

AVHRR, Landsat, SPOT

Height, Precip., Wetness

MODIS products

Weather data

4. Institutional realities The way forward FAS/IPAD is an operational division requiring continuous and consistent data products. One thread of this dynamic data stream includes redundant Earth observation data sets from AVHRR, SPOT-VEGETATION, MODIS, and future satellites like VIIRS. Weather now-casting based on GOES and Meteosat and climate forecasting models are also pertinent to IPADs mission. The use of decision support systems and tools often results in more informed decision making and better communication among stakeholders (Sivakami & Karthikeyan, 2009), managers and decision makers. However, to efciently develop and adapt decision support tools assimilating satellite data (e.g. MODIS, ASTER, AMSR) and other products, partnerships between stakeholders and product developers are essential to communicate user and system requirements (Pyke et al., 2007) (Fig 2). The collaborative benchmarking activities provided not only feedback about the benets of DSS enhancement to USDA/FAS and NASA, but facilitated communication among DSS users, developers, and USDA management that helped to suggest future avenues for system development as well as improved intra- and interagency collaboration. It is important to note that production estimates for each country, commodity and season are derived differently based on the usability and availability of information sources for each country and the unique requirements presented to its analyst. The benets demonstrated in this benchmarking exercise of the assimilation of NASA data in IPADs evolving DSS are: improved quality of crop assessment and production estimates and decisions, cost reduction and timesavings. For example, benchmarking through the evaluation of performance metrics showed the effectiveness of the assimilation of MODIS VI (Vegetation Index) products into IPADs DSS. The MODIS vegetation products (as well as some ASTER data) are one of multiple lines of evidence that are used in a convergence of evidence methodology to estimate agricultural production at a global scale. The MODIS VI product is shared online through Cropexplorer (http://www.pecad.fas.usda.gov/cropexplorer/). Results from validation and verication efforts can provide analysts a measure of condence in the products. The development of data elds that provide the end-user community with better quality information associated with satellite data products will nurture enhanced condence and could move users from visual examination to more sophisticated uses and better decision making (Bahill & Gissing, 1998). The degree of structure and problem complexity for the IPAD DSS was favorably altered, changing from an unstructured, novel problem to a semi-structured, modeled problem by the introduction of products derived from NASA observations and research. Introducing such products for use in a DSS serving semi- or unstructured, complex problems can improve the quality of information on which human judgment is based by providing not only new and different information for the decision maker but also alternative solutions with their potential impacts. Improving the quality of information available from selected NASA knowledge, capacity, and systems for input to the IPAD DSS then has both strengthened the structure of the IPAD DSS problem by making it more amenable to systematic (i.e. algorithmic) solutions, and improved the information and evidence available on which to base decisions. This has a number of important ramications, offering such prospects

1+ State

1+

Fig. 6. Attainment levels of IPADs objectives for several mitigation categories.

of all mitigations (28.3) applied at the same time, even when the attainment based on State 0 is subtracted (13.2 + 14.7 + 20.3 + 20.9 + 10.8 + 15.1 = 95 (6 8.4) = 44.6). Basically, simultaneous use of mitigations provides a marginal benet depending on the number of mitigations applied and their effectiveness. Mitigations scenarios can also work synergistically when the benet (attainment of objectives) of the sum of several single mitigations is less than the benet of the mitigations used simultaneously. For example, the attainment of objectives by mitigating risks with trained analysts and the use of a data base (CADRE) and data visualization tools (CropExplorer) is 4.1 out of 30. If MODIS data are used to mitigate as well, the attainment is 11.8 out of 30. However, the attainments for the single mitigations are less than 1 for all single mitigation elements. It should be noted that the use of the data is most enhanced when the data visualization and extraction tools are integrated in the system. These results suggest that the relative contributions of data products, tools and models are generally equivalent. Further, non-NASA and NASA products are equivalent in terms of their respective OAs, suggesting that if but one set is available, they are interchangeable but they are not necessarily redundant with each providing a satellite perspective but different information for application towards the convergence of evidence approach used in IPAD. Moreover, MODIS products are interchangeable in the same sense with the AVHRR, SPOT, and Landsat products, but each has its unique contribution to the whole. However, it is also apparent that relative contributions or value of data products are highly cumulative (positive correlation) and not independent. That is, the value of MODIS is essential with no other information, adding ancillary information (as in ground truth) increases the value of the MODIS information more then simply adding more MODIS data. At this stage we have not taken into consideration the acquisition costs of the data products which would be expected to inuence the cost-benet or cost-OA. Finally, the DBMS has a slightly higher OA contribution than the integrated tools, it is commensurate with crop yield models, but it is noticeably greater than crop stage and soil moisture models. A similar disaggregation of the OA contribution for the transition from State 0 to State 1 was not done. Recall that State 0 is dened to consist only of ground-based analyst and attach-based Mitigations, and State 1 is the benchmarking baseline with State 0 Mitigations carried throughout. 3.4. Benchmarking IPADs DSS DDP sensitivity analysis The DDP tool can also be used for sensitivity analyses measuring all possible combinations of Mitigations across the complete range of input/mitigation effectiveness values obtained from the analysts. The sensitivity analysis shows the top Mitigations as a percentage change in OA. The Risk element most impacted is lack of historical data. . . with Mitigations ranging from data to tools to

8064

W. van Leeuwen et al. / Expert Systems with Applications 38 (2011) 80548065

as: (a) Enhanced consistency of results; (b) Increased accuracy; (c) Increased throughput speed; and (d) Increased socio-economic benet. The novel application of the DDP tool provided a means of quantifying information about critical aspects of the OGA/IPAD decision support system. For example, it became immediately obvious that the Objectives, Risks and Mitigations needed to be succinctly descriptive, dene the complete but not overlapping set of affective elements, and must be to the point. Critical decisions involved determining the weight or importance of each of the Objectives identied for meeting the institutional mission, the impact of specic Risk elements on the Objectives, and the utility of various Mitigations for countering the Risks and attaining the Objectives. The DDP proved to be a robust tool for measuring and portraying changes in a complex DSS. Increasing the quality or variety of suitable Mitigations leads to greater Objectives attainment, as does simply increasing the number of Mitigations (but not for the same reasons). Objectives attainment shows a generally asymptotic trend approaching 100% OA with additional remote sensing Mitigations. Despite this behavior, the DDP data permit teasing apart the relative value of particular Mitigations and combinations of them. The DDP software provides a powerful tool to consistently and quantitatively manage and analyze risk and benchmark IPADs DSS. The DDP tool elicited many constructive discussions and helped direct the benchmarking thought process in a risk management context. Since NASA enhancements to IPADs DSS are expected to be at various stages of implementation (e.g. between State 2 and State 3), the results presented here can be complemented with annual repeat surveys or when most State 3 enhancements are implemented. The same benchmarking approach could be used to measure the difference in performance of the DSS between State 2 and State 3 and subsequent States, and can be based on performance indicators like risk balance and objective attainment using the current IPAD-DDP application as a baseline. This benchmarking of IPADs DSS also provides the International Earth Observing System (IEOS) with a better understanding of the observational needs of the national agricultural efciency applications area. This work demonstrates how requirements must be solicited not only from the USDA but also with respect to global priorities, research agenda and needs, and around the alignment of international collaboration across many agencies, institutions and local user communities. Although new technology such as LIDAR currently lacks global coverage, it could provide many insights in the growth patterns and structure of crops and vegetation. RADAR is another promising technology for retrieving soil moisture and can be used to monitor crops under cloud cover. Strategic IEOS requirements could be derived from a comprehensive examination of IPADs integrated data, modeling, tools and user community requirements. This benchmarking approach could also provide insights to the US Ofce of Management and Budget (OMB) about how investments are used and aid decision making that result in improved and independent global commodity production intelligence that equally benets US and world societies. This research resulted in a model for benchmarking Decision Support Systems within a larger framework of system solutions. It has implications for national and international agricultural policies, for agricultural production and efciency management, for societal benets and impacts. This benchmarking model (1) promotes intra- and inter-agency collaboration and communication, (2) accommodates multiple model results and data sources such as Earth satellite and eld observations and economic data, (3) integrates scientic, visualization, system and operational requirements, (4) facilitates transfer of current and historical data and knowledge among research, management, and decision making organizations and agencies, (5) provides a risk management prole

and levels of objective attainment that are associated with enhancements and changes to decision support systems. This DSS benchmarking approach is exible; it can be applied to other decision support systems such as the Invasive Species Forecasting System (ISFS), the National Integrated Drought Information System (NIDIS), land use planning (Witlox, 2005), or other ecological or hazard forecasting systems and applications that inform management decisions. Acknowledgements Benchmarking the performance characteristics of the IPAD decision support system was a collaborative effort among the USDA Foreign Agricultural Service, the Universities of Missouri and Arizona, the NASA Jet Propulsion Laboratory at Pasadena, California, and the Applied Sciences Directorate at NASAs John C. Stennis Space Center, Mississippi. The University of Arizona participated in this effort under NASA Grants NAG5-12484 and NNG06GF1G. The University of Missouri participated in this effort under NASA Grants NAG5-12482 and NNG06GF16G. The DDP development team that contributed to the benchmarking was led by David Tralli at the NASA Jet Propulsion Laboratory and is conducted under Research and Technology Operating Plans 613-08-02-05 and 613-2552-40. Consultation with the DDP development team led by S.L. Cornford, M.S. Feather and S.L. Prusha in the Strategic Systems Technology Program at JPL was invaluable. Without their help, the success and insight gained herein for the benet of USDA/ FAS/IPAD and the NASA Applied Sciences Directorate would not have been possible. Special thanks are extended to Drs. Curt Reynolds and Bob Tetrault for sharing their perspectives and insights during the benchmarking process. Appendix A. Acronyms AFWA AgRISTARS Air Force Weather Agency Agriculture and Resources Inventory Surveys Through Aerospace Remote Sensing Advanced Very High Resolution Radiometer Crop Condition Data Retrieval and Evaluation Centre National dEtudes Spatiales (the French space agency) Database management system LAI MODIS Leaf Area Index Moderate Resolution Imaging Spectroradiometer

AVHRR

NASA

CADRE

NASS

CNES

NDVI

National Aeronautics and Space Administration National Agricultural Statistics Service (USDA) Normalized Difference Vegetation Index National Integrated Drought Information System National Oceanic and Atmospheric Administration National Polar-

DBMS

NIDIS

DDP

Defect Detection and Prevention Decision Support

NOAA

DSS

NPOESS

W. van Leeuwen et al. / Expert Systems with Applications 38 (2011) 80548065

8065

System

EVI

Enhanced Vegetation Index

NPP

FAO

FAS

FEWS

FGDC

Food and Agriculture Organization Foreign Agricultural Service Famine Early Warning System (USAID) Federal Geographic Data Committee Farm Service Agency (USDA) Global Area Coverage (8-km pixels) Global Inventory Monitoring and Modeling Studies Geographic Information Systems Global Reservoir and Lake Monitor Ground sample distance

OA

orbiting Operational Environmental Satellite System NPOES (National Polar Orbiting Environmental Satellite) Preparatory Project Objective Attainment Ofce of Global Analysis Ofce of Management and Budget Production Estimates and Crop Assessment Division Production, Supply and Distribution Quality Assurance/Quality Control Special Sensor Microwave/ Imager) United States Department of Agriculture United States Geological Survey Satellite Probatoire dObservation de la Terre Thematic Mapper (Landsat) US Department of Agriculture

LACIE

Large Area Crop Inventory Experiment

WMO

World Meteorological Organization

References
AgRISTARS, (1981). Agriculture and Resources Inventory Surveys Through Aerospace Remote Sensing: AgRISTARS: National Aeronautics and Space Administration. Houston, TX. Bahill, A. T., & Gissing, B. (1998). Re-evaluating systems engineering concepts using systems thinking. Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, 28(4), 516527. CCSP, (2008). Uses and limitations of observations, data, forecasts, and other projections in decision support for selected sectors and regions. A Report by the U.S. Climate Change Science Program (CCSP) and the Subcommittee on Global Change Research. Cornford, S. (1998). Managing risk as a resource using the defect detection and prevention process. Paper presented at the international conference on probabilistic safety assessment and management. Feather, M. S. (2001). Risk reduction using DDP (Defect Detection and Prevention): Software support and software applications, Proceedings of the 5th IEEE international symposium on requirements engineering: IEEE Computer Society. Feather, M. S., Cornford, S. L., & Gibbel, M. (2000). Scalable Mechanisms for Requirements Interaction Management. Paper presented at the Proceeding 4th IEEE international conference on requirements engineering, Schaumburg, Illinois. Gindorf, T., & Cornford, S. (1999). Defect Detection and Prevention (DDP): A tool for failure mode risk management. Paper presented at the 50th international astronautical congress, Amsterdam, The Netherlands. 48 Oct. 1999. Hammond, A. L. (1975). Crop forecasting from space Towards a global food watch. Science, 188(4187), 434436. Hutchinson, C., van Leeuwen, W., Drake, S., Kaupp, V., & Haithcoat, T. (2003). Characterization of PECADs DSS: a zeroth-order assessment and benchmarking preparation. Report submitted to Ed Sheffner (NASA HQ) and Brad Doorn (PECAD) U.S. Department of Agriculture Foreign Agricultural Service FAS production estimates and crop assessment division PECAD 1400 Independence Avenue, SW Room 6053-S Washington, DC 20250. Hutchinson, C. F. (1991). Uses of satellite data for famine early warning in subSaharan Africa. International Journal of Remote Sensing, 12(6), 14051421. Jame, Y. W., & Cutforth, H. W. (1996). Crop growth models for decision support systems. Canadian Journal of Plant Science, 76(1), 919. Justice, C. O., Vermote, E., Townshend, J. R. G., Defries, R., Roy, D. P., Hall, D. K., et al. (1998). The Moderate Resolution Imaging Spectroradiometer (MODIS): Land remote sensing for global change research. Geoscience and Remote Sensing, IEEE Transactions on, 36(4), 12281249. Kaupp, V., Hutchinson, C., Van Leeuwen, W., Drake, S., & Tuyahov, A. (2003). Assimilation of NASA Earth science results and data in national decision support systems. Paper presented at the geoscience and remote sensing symposium, 2003. IGARSS 03. Proceedings. 2003 IEEE International. Lger, B., & Naud, O. (2009). Experimenting statecharts for multiple experts knowledge elicitation in agriculture. Expert Systems with Applications, 36(8), 1129611303. Lucertini, M., Nicol, F., & Telmon, D. (1995). Integration of benchmarking and benchmarking of integration. International Journal of Production Economics, 38(1), 5971. Macdonald, R. B., & Hall, F. G. (1980). Global crop forecasting. Science, 208(4445), 670679. MacDonald, R. B., Hall, F. G., & Erb, R. B. (1975). The use of Landsat Data in a Large Area Crop Inventory Experiment (LACIE). Houston, TX: National Aeronautics and Space Administration. McCuistion, J. D., & Birk, R. (2005). From observations to decision support: The new paradigm for satellite data. Acta Astronautica, 56(12), 58. Pyke, C. R., Bierwagen, B. G., Furlow, J., Gamble, J., Johnson, T., Julius, S., et al. (2007). A decision inventory approach for improving decision support for climate change impact assessment and adaptation. Environmental Science & Policy, 10(78), 610621. Reynolds, C. A., Yitayew, M., Slack, D. C., Hutchinson, C. F., Huete, A., & Petersen, M. S. (2000). Estimating crop yields and production by integrating the FAO Crop specic Water Balance model with real-time satellite data and ground-based ancillary data. International Journal of Remote Sensing, 21(18), 34873508. Sivakami, S., & Karthikeyan, C. (2009). Evaluating the effectiveness of expert system for performing agricultural extension services in India. Expert Systems with Applications, 36(6), 96349636. Spendolini, M. J. (1992). The benchmarking book. New York, NY: Amacom. Witlox, F. (2005). Expert systems in land-use planning: An overview. Expert Systems with Applications, 29(2), 437445.

OGA

OMB

PECAD

FSA

PS&D

GAC

QA/QC

GIMMS

SSM/I

GIS

USDA

GRLM GSD

USGS SPOT

GSFC GOES

IPAD

ISFS

IT

Goddard Space Flight Center Geostationary Operational Environmental Satellites International Production Assessment Division Invasive Species Forecasting System Information Technology Jet Propulsion Laboratory

TM USDA

USGS

US Geological Survey

VI

Vegetation Index

VIIRS

JPL

WAOB

Visible/Infrared Imager/ Radiometer Suite World Agricultural Outlook Board