A Security Evaluation and Testing Methodology

for Open Source Software Embedded

Information Security System

Sung-ja Choi, Yeon-hee Kang, and Gang-soo Lee

Hannam University, Dept. of Computer Science,

Daejon, 306-791, Korea
irecomm@dreamwiz.com
{dusi82@se, gslee@eve}.hannam.ac.kr

Abstract. Many of Information Security Systems (ISS) have been de-

veloped by using and embedding Open Source Software(OSS) such as
OpenSSL. The “OSS-embedded ISS” should be tested and evaluated
when it will be used as a security product or system for an organization.
In this paper,we present a test and evaluation procedure for an OSS-
embedded ISS, and ROSEM(real-time OpenSSL execution monitoring
system) that is a testing tool in according to presented methodology.
The main function of ROSEM such as an execution path generator for
OpenSSL is useful for test case generation in the CC evaluation scheme.

1 Introduction
Recently, it has been being increased to introduce an Open Source Software
(OSS) such as Apache, Linux, BSD, Mozilla, MySQL, OpenSSL, Crypto++
and so on, that contains security functions and cryptography modules, for the
purpose of shorten the development duration of Information Security System
(ISS)[1]. OpenSSL which is a well known OSS of cryptography component is
mostly used for IDS or VPN development[11]. OSS based components (e.g.,
cryptography component, communication functions) embodied in various forms,
and they are offered as a form of OSS. It is possible that most of ISS developers
use components, which are in the form of OSS, without a formal analysis to
shorten the period of development.Thus, they could be loaded and embedded
to source code of ISS without the assurance of security. Therefore, the safety
and security of OSS-embedded ISS is not guaranteed. Also, the most developers
and security evaluators in Common Criteria(CC, ISO/IEC 15408) evaluation
scheme should know the details about inner structure and source code as well as
development information of OSS, because they use and evaluate some cryptog-
raphy components in OSS. However, it is very hard, because most of OSS do not
have any documentation and development information. Thus, we should obtain
deliverables for evaluation by means of reverse-engineering from OSS.
From those backgrounds, we have researched and developed the following
topics for developers and evaluators in CC evaluation scheme:

O. Gervasi et al. (Eds.): ICCSA 2005, LNCS 3481, pp. 215–224, 2005.


c Springer-Verlag Berlin Heidelberg 2005
216 S.-j. Choi, Y.-h. Kang, and G.-s. Lee

• Research of a new test method and paradigm for an OSS embedded ISS.
• Development of a ROSEM as a test case generation for cryptography com-
ponents in OpenSSL.
• As a case study, generation of test case for testing cryptography function
such as rc4 in OpenSSL by using ROSEM.

Next section presents a test method for OSS embedded ISS. Section 3 presents
development of ROSEM. Section 4 presents the result of test case generation for
OpenSSL by using ROSEM, as a case study. In Section 5, we summarize and
conclude.

2 Evaluation and Testing Method for an OSS Embedded

ISS
2.1 Procedure of Development and Evaluation
Fig. 1 presents the paradigm for development and evaluation of TOE that is
consisted of three process such as OSS development process, TOE development
process and TOE evaluation process.

OSS developing process

req. implemen- delivery of evaluation
evaluation
evaluationof
of
ofOSS
OSS
OSS
design test
analysis tation OSS (CC,CMVP)
(CC,CMVP)
(CC,CMVP)
"Upper version"

TOE developing process

browse include
"Using OSS" browsering
useful OSS
req.
analysis
design implementation test
"On development"

"deliverables"

TOE evaluation process TOE

(CC based) testing mechanism security function
(blackbox,whitebox
KAT so on) OSS

vulnerability configuration
manage template for non-security
analysis
evaluation OSS function

the result of evaluation

Fig. 1. The paradigm of development and evaluation of TOE

• OSS development process: The most of OSS does not have the development
and evaluation process. Because it has been modified by developers without
uniform development process and evaluation process whenever new function
is required. Then, new version of OSS has been distributed through network.
Note that, the integrity of configuration of the OSS should be preserved and
 A Security Evaluation and Testing Methodology 217

validated in case of insecure distribution environment such as internet. So,

some cryptography component in OSS such as OpenSSL and Crypto++ have
been evaluated under CMVP (Cryptography Module Validation Program)
scheme according to FIPS 140-2 of NIST[3]. Linux which is a famous OSS
has been evaluated and certified as EAL2+ by CC scheme[12].
• TOE development process: OSS have been greatly used to reduce the de-
velopment time and cost. A development process of OSS-embedded TOE is
consisted of two part, self development and development by using OSS.
• TOE evaluation process: An evaluator need not consider whether a TOE
is built by using some OSS or not, because he evaluate all part of TOE.
Therefore, if he has evaluated OSS-embedded TOE, the OSS part in TOE
would be not evaluated. Our paradigm will provides evaluation methodology
of OSS-embedded TOE to be consider evaluation of OSS part.

2.2 Test Process

A test process for evaluation of OSS-embedded ISS has below test process phase.
It is test planning, test specification, test execution, test result recording and
test result validation [4].
[Step 1] Test planning
Test planning phase has be consist of the establishment test plan and the building
test bed for evaluation / authentication.

• [Step 1.1] Establishment of test plan : In the establishment of test plan,

it should be described to test the OSS-embedded ISS such as the name of
vendor, test date and version information of OSS which has been used, and
so on.
• [Step 1.2] Build of test bed : It could be specified details information to test
such as environment construction, the kind of OS and forms of compile so
on. It should be built of the test bed according to specified environments.

[Step 2] Test specification

The tester should specify the test mechanism which is software test techniques
such as component test methodology [4], security test, and test validation tool
[5] in test specification phase. Black-box[2], white-box and KAT testing are well
known software test techniques. Then, it could be applied to test, also the tester
should derived the test item from the function classes in order to test OSS part
in a TOE as like Fig.2. The slicing of OSS-embedded ISS is very important
and hard but we have applied the slicing method to test each component. It is
possible to test independently.

• [Step 2.1] Derivation of test item: The phase of function specification has
the test items. It is regarded a test item as a class have been divided according
to function approximately.
• [Step 2.2] Family and component (function A1, A2..) specific : All of the
function could be divided sub-function in details. The several sub-functions
218 S.-j. Choi, Y.-h. Kang, and G.-s. Lee

OSS embedde ISS

component A1 element A1
function calss A1
function calss A2 component A2 element A2

TOE requested
evaluation component An element An

Fig. 2. A Test of OSS and An Evaluation Procedure

could be makes a component to separate OSS-embedded ISS, also it could be

classified family unit to slice according to function requirements. It could be
details module in order to test modules after components had been gotten
[9]. It may be the structure design in reverse engineering.
• [Step 2.3] Detail module (security module) : Each component is divided
element units and these units present each API modules as like OpenSSL.
Note that, it may be called non-divided function unit. It may be the detail
design in reverse engineering.
• [Step 2.4] Derivation of interface : Each component has inputs and outputs
and it could be drawn a state transition because of interaction among them.
Interface among components would be operated independently because a
module is possible to extract from each component. It have been presented
mapping table or call graph.
• [Step 2.5] Test cases generation : Test cases are generated by each elements
has been independent function unit. A test case, non-divided function, could
be operated independently. The test case should be tested by real-time test-
ing tool(as like ROSEM), the result of testing could be used for evaluation
OSS-embedded ISS.

[Step 3] Test execution

In test execution, it could be applied according to black-box technique and
KAT[7][8][9] so on, which is based on test specification. It could be classified
the evaluation/authentication level by analyzing the test execution resulting.
We offer the real-time monitoring tool, ROSEM, which had been applied the
harness mechanism to test and show the table of execution resulting and the
monitor log screen. It will be generate a call graph in the elements testing and
it could be validated OSS-embedded ISS.
• [Step 3.1] Test execution : The generated test case should be execute ac-
cording to specified test mechanism at test specific step[10]. We apply a state
transition and show a table from a state transition. Each test case gener-
ates a table for the test specification. State transition table includes state
number, API number, Function name, Function family(module) and Func-
tion form(it states related calls which are divided into security modules, SSL
connections, and applications).
 A Security Evaluation and Testing Methodology 219

Supported "ROSEM Manual Test spec. for

template type of test test spec. templet Test case of TOE for
specification test case of TOE evaluation document
test methodology
element A1 and document" element A1 - test plan
element A2 element A2 - test spec.
................. "evaluation request ................. - test execution
evaluation / & test document" - test reporting
authentication evaluation
agency requestor - test fault report

Fig. 3. The evaluation method of TOE test case

• [Step 3.2] Test coverage calculation : Test coverage could be calculated by

the result of test execution of test cases. Test coverage could be applied
effectively by applying test measurement technique.
Test coverage = number of test cases completed / number of total test
cases x 100
This numerical formula is calculated in order to analyze test range, and the
aim of the test execution is to test all of the specified test cases.

[Step 4] Test result recording

The recording of the test result, whenever every single test is operated, should
be recorded about information such as component version of test, component
test specific, test date, number of test cases, number of test discordant, range
measurement, fault report, and so on. Using testing in order to compare between
actual test result and expected test result should check a test log file. And a fault
report should be presented about the differences.

[Step 5] Test result validation

In test result validation, the result of testing execution is used to evaluate and
classified evaluation level for OSS-embedded ISS and then should be report eval-
uation of TOE to ISP.

3 ROSEM – A Real Time OpenSSL Evaluation

Monitoring System
We have developed a real-time monitoring tool for the purpose of evaluating and
testing “OpenSSL embedded ISS” to which “harness code for the monitor” is
included, in order to monitor the calling sequence of set of functions(or modules)
during the operation of OpenSSL.
This is a technique of so called the “test harness”. The harness code plays
a role as a proxy and it has report all sorts of information to be collected to
the analyzer in ROSEM through window messaging mechanism. The analyzer
has enabled to analyze whatever the tester wants to test. We suggest “OpenSSL
+” by inserting harness code into OpenSSL in order to monitor as like Fig 4.
ROSEM has a function which receives and analyzes information from harness
220 S.-j. Choi, Y.-h. Kang, and G.-s. Lee

OpenSSL+
>
"start" “St Module "inserted harness
Module code"
Analyser
command Module >
execution Module >
Module >
Module >
"execution
"finish" path" Module

result
(execution etc.)

Fig. 4. An operation theory of ROSEM

correcting analyser module

Analyser module present
module
preservation
module
OpenSSL+
(OpenSSL Module
inserted harness code
for monitor)

GUI engine

Fig. 5. The structure design of ROSEM

Fig. 6. Class design

code of OpenSSL+, and the result of analyzed information have been output
which certain orders are being operated. The ROSEM shows the monitoring
procedure of the operation for OpenSSL embedded ISS.
This method have been perfectly executed the every test case on real time.
ISS which used certain function of OpenSSL will be call a set of function library
in OpenSSL and the called function library sends its own operation information
to monitoring tools and returns the operating result to the next code. After it
chould be analyzed the received logging information through log analyzer, also
the monitoring tool shows that on the screen by using GUI engine.
ROSEM have been made up correcting module, analyzing module, preserva-
tion module and presentation module of OpenSSL-embedded ISS as like Fig. 5.
We have designed classes for implementation of ROSEM. In Fig 6. is a sample
of generated class design.
 A Security Evaluation and Testing Methodology 221

Below clause shows a generated test case and an evaluation method by ap-
plying monitor tools had been developed.

4 Case Study – Test Case for rc4 in OpenSSL

By applying test method and ROSEM for evaluation OpenSSL-embedded ISS,
we show the example of testing for OpenSSL. RC2 function testing in the
OpenSSL is operated through test bed already had been built up. It could be pos-
sible to analyze testing module while RC2 application program is being run. RC2
cryptography program has been run on the Console window. In Fig 7, testfile.txt
had been cryptography RC2 by cbc and then it had been created testfile.rc2.

Fig. 7. The captured hex value by RC2 Function Cryptograph

The cryptography resulting of the testfile.txt had been done the gathering
of hex value. It is certain that the cryptography function of RC2 had been
achieved normally. The RC2 cryptography had been used EVP as well. The
testing of Module, which had been used in this cryptography function, it could
be analyzed on the monitor tool with the information of logging, too. It had been
occurred about 15,600 logs during running this RC2 cryptography function. We
present logs screen which is captured the testing result as like Fig. 8.

Fig. 8. The captured of RC2 function test result log screen (TS-37)

We can see that mainly five modules are operating apart from basic modules
to cryptography RC2. That is, BIO module for input/output and user interface
module to ask codes and take them, RAND module which is generating random
222 S.-j. Choi, Y.-h. Kang, and G.-s. Lee

RC2 component test specification

test purpose : RC2 component function test

test case : TS-37
component test specification : SP-37

Element
(API NUMBER Function) Kind of function type Function form

16873 2365 RC2_set_key RC2 CRYPT_API CRYPT

16874 715 BIO_push Bio_BASIC CRYPT_API CRYPT
16875 711 BIO_ctrl Bio_BASIC CRYPT_API CRYPT
16879 1869 EVP_CipherUpdate EVP_BASIC CRYPT_API CRYPT

16880 1876 EVP_EncryptUpdate EVP_ENCRYPT CRYPT_API CRYPT

16881 2361 RC2_cbc_encrypt RC2 CRYPT_API CRYPT
16882 2362 RC2_encrypt RC2 CRYPT_API CRYPT

Function call 16873 -16882 : the test omitted.

Fig. 9. RC2 fuction test case - test specification

numbers to encode, MD for message digest, and SHA module, and RC module
for cryptography.
An evaluation requestor has to generate the specification of test through test
bed as like Fig. 9 and should be evaluated it with other deliverables of TOE
when he applies to evaluation. It is similar to procedure of existing certification,
the OSS part in TOE should be compared the template of OSS testing with
test specification of TOE in deliverable which provided from the requestor of
evaluation. It has been assured to be correctly loaded the OSS part in TOE and
to be trusted OSS-embedded TOE.

5 Summary and Conclusion

In this paper, we present testing methodology for correctness using of Open
source and evaluation of OSS-embedded ISS. It could be achieved reliability
and trust in using OSS. Also, we had been developed real-time monitoring too,
ROSEM, it had been applied a example in order to test OpenSSL-embedded ISS
which is used to develop ISP which has short sort code like smart card or RFID.
Main findings of this study are as follows.

• Investigation open source related security: By investigating and ana-

lyzing security test tool of open source form and down location and character
of cryptography library, it makes easy to use them.
• Investigation and analysis of software component test method: We
have investigated and analyzed general software test methodology and ex-
plicated BS 7925 of English which is the standard of this field. From this
finding, the product developer and the evaluator can get general knowledge
for the test and have the ability of choosing the most suitable test method.
• Investigation of test of cryptography module and ISP and eval-
uation method: CMVP, the system inspecting cryptography module of
open source, has been investigated carefully and evaluation standard of open
source-embedded product which shown in CC/CEM has been investigated
as well. The result from these investigations can be used as a guide in the
CC evaluation system.
 A Security Evaluation and Testing Methodology 223

• Analysis of OpenSSL version 0.9.7: We have analyzed module structure

and function of OpenSSL and presented each security module component and
interface. It has also presented instruction codes with option and presented
19 weaknesses well known. And it has presented security function classi-
fication result inside of OpenSSL and a countermeasure with CC security
function.
• Test of OpenSSL product and presentation evaluation methodol-
ogy: We have presented the body of test and evaluation methodology and
how to draw report from OpenSSL in each step.

As a conclusion, we provide a draft of “The Guide line for evaluation of OSS-

embedded ISS” to the evaluation executor and the vendor for evaluation. It is
expected to use as follows. On the developer’s side, this study is information
needed when the report is drawn for development and evaluation of ISP through
OpenSSL and all sorts of open source. The concept of open source, software
test technique, structure analysis of OpenSSL, and ROSEM will be very useful
information and a tool for developer. On the side of evaluator, it can be used as
a guide and a tool for the test and evaluation of ISP that is developed through
not only OpenSSL but all sorts of open source. However, the function of ROSEM
should be extended to apply to the other security function of open source. Various
counterattacking tools those are drawn more various information for evaluation
from counterattack against from open source like OpenSSL should be developed.
There are a few similar tools to these already. So, methods have to be studied
to use them. Just a few functions of OpenSSL have been studied about running
condition but more functions should be studied. Also, the environment which is
applied evaluation technology of CMVP should be expanded.

Acknowledgements
This work was supported by KISA(Korea Information Security Agency)&RRC(a
grand No.R12-2003-004-01001-0 from Ministry of Commerce a Industry and
Energy).

References
1. A. Wheeler, Why Open Source Software / Free Software (OSS/FS)? Look at the
Numbers!. July 23, 2004, http://www.dwheeler.com.
2. B. Beizer, Black-Box Testing, John Wiley & Sons, 1995.
3. NIST Special Publication 800-29, A Comparison of the Security Requirements for
Cryptographic Modules, FIPS 140-1 AND FIPS 140-2, Ray Snouffer, Annabelle Lee
and Arch Oldehoeft, NIST, June, 2001.
4. British Computer Society Specialist Interest Group in Software Testing(BCS
SIGIST), Standard for Software Component Testing, Working Draft 3.4, April
2001. 2001.
224 S.-j. Choi, Y.-h. Kang, and G.-s. Lee

5. C. Kaner, Quality Cost Analysis: Benefits and Risks, http://www.kaner.com/

qualcost. htm.
6. Frank Tip, A survey of program slicing technique, Journal of Programming Lan-
guages, Vol.3, No.3, pp.121-189, September 1995.
7. W. E. Perry, Effective Methods for Soft-ware Testing, 2nd Ed., John Wiley & Sons,
Inc., 1999.
8. G. Myers, The Art of Software Testing, John Wiley & Sons, Inc., 1979. 1979.
9. K. J. Ross, Practical Guide to Software System Testing, K. J. Ross & Associates
Pty. Ltd., 1998. 1998.
10. C. Kaner, J. Falk, Q. Nguyen, Testing Computer Software, 2nd Ed., Thomson
Computer Press, 1993. 1993.
11. OpenSSL, http://www.openssl.org/ .
12. Common Creteria for Information Technology Security Evaluation(CC), CCIMB-
2004-01-003, version 2.2, ISO/IEC 15408, Jan. 2004.