Grid Architecture

Grid Architecture
Prof. Ruay-Shiung Chang March 2004
Course Contents
Grid Architecture Open Grid Services Architecture Resource and Service Management Building Reliable Clients and Services Instrumentation and Monitoring
Grid Architecture
Grid Architecture
Why Discuss Architecture?
Descriptive
Provide a common vocabulary for use when describing Grid systems
Guidance
Identify key areas in which services are required
Prescriptive
Define standard Intergrid protocols and APIs to facilitate creation of interoperable Grid systems and portable applications
Water Garden in England
One View of Requirements
Identity & authentication Authorization & policy Resource discovery Resource characterization Resource allocation (Co-)reservation, workflow Distributed algorithms Remote data access High-speed data transfer Performance guarantees Monitoring
Adaptation Intrusion detection Resource management Accounting & payment Fault management System evolution etc. etc.
Another View: Three Obstacles to Making Grid Computing Routine

1) New approaches to problem solving Data Grids, distributed computing, peer-topeer, collaboration grids, 2) Structuring and writing programs Abstractions, tools 3) Enabling resource sharing across
10
distinct institutions
Resource discovery, access, reservation, allocation; authentication, authorization, policy; communication; fault detection and notification;
11
13
The Systems Problem: Resource Sharing Mechanisms That
Address security and policy concerns of resource owners and users Are flexible enough to deal with many resource types and sharing modalities Scale to large number of resources, many participants, many program components Operate efficiently when dealing with large amounts of data & computation
1) Need for interoperability when different
Aspects of the Systems Problem
14
groups want to share resources
Diverse components, policies, mechanisms E.g., standard notions of identity, means of communication, resource descriptions
2) Need for shared infrastructure services
to avoid repeated development, installation
E.g., one port/service/protocol for remote access to computing, not one per tool/app E.g., Certificate Authorities: expensive to run
A common need for protocols & services
Hence, a Protocol-Oriented View of Grid Architecture, that Emphasizes
15
Development of Grid protocols & services

Protocol-mediated access to remote resources New services: e.g., resource brokering On the Grid = speak Intergrid protocols Mostly (extensions to) existing protocols
Development of Grid APIs & SDKs

Interfaces to Grid protocols & services Facilitate application development by supplying higher-level abstractions
The (hugely successful) model is the Internet
16
Layered Grid Architecture (By Analogy to Internet Architecture)

Application
Coordinating multiple resources: ubiquitous infrastructure services, app-specific distributed services Sharing single resources: negotiating access, controlling use Talking to things: communication (Internet protocols) & security Controlling things locally: Access to, & control of, resources Internet Protocol Architecture
Collective
Application
Resource
Connectivity Fabric
Transport Internet
Link
17
Protocols, Services, and APIs Occur at Each Level

Applications Languages/Frameworks
Collective Service APIs and SDKs Collective Services
Collective Service Protocols
Resource APIs and SDKs Resource Services
Resource Service Protocols
Connectivity APIs
Connectivity Protocols Local Access APIs and Protocols
Fabric Layer
18
Important Points
Built on Internet protocols & services

Communication, routing, name resolution, etc.
Layering here is conceptual, does not imply constraints on who can call what
Protocols/services/APIs/SDKs will, ideally, be largely self-contained Some things are fundamental: e.g., communication and security But, advantageous for higher-level functions to use common lower-level functions
19
The Hourglass Model
Focus on architecture issues
Applications Diverse global services
Propose set of core services as basic infrastructure Use to construct high-level, domain-specific solutions Core
Design principles
Keep participation cost low Enable local control Support for adaptation IP hourglass model
services
Local OS
20
Hourglass
21
Where Are We With Architecture?
No official standards exist But:

Globus Toolkit has emerged as the de facto standard for several important Connectivity, Resource, and Collective protocols GGF has an architecture working group Technical specifications are being developed for architecture elements: e.g., security, data, resource management, information Internet drafts submitted in security area
22
Just what you would expect: the diverse mix of resources that may be shared
Individual computers, Condor pools, file systems, archives, metadata catalogs, networks, sensors, etc., etc.
Fabric Layer Protocols & Services
Few constraints on low-level technology: connectivity and resource level protocols form the neck in the hourglass Defined by interfaces not physical characteristics
23
Fabrics in general
24
Communication
Connectivity Layer Protocols & Services
Internet protocols: IP, DNS, routing, etc.
Security: Grid Security Infrastructure (GSI)

Uniform authentication, authorization, and message protection mechanisms in multiinstitutional setting Single sign-on, delegation, identity mapping Public key technology, SSL, X.509, GSS-API Supporting infrastructure: Certificate Authorities, certificate & key management,
25
Not too many connections!
26
Grid Resource Allocation Management (GRAM)

Remote allocation, reservation, monitoring, control of compute resources
Resource Layer Protocols & Services
GridFTP protocol (FTP extensions)

High-performance data access & transport
Grid Resource Information Service (GRIS)

Access to structure & state information
Others emerging: Catalog access, code repository access, accounting, etc. All built on connectivity layer: GSI & IP
27
Index servers aka metadirectory services

Custom views on dynamic resource collections assembled by a community
Collective Layer Protocols & Services
Resource brokers (e.g., Condor Matchmaker)

Resource discovery and allocation
Replica catalogs Replication services Co-reservation and co-allocation services Workflow management services etc. Condor: www.cs.wisc.edu/condor
28
Collectives: The Borgs
Resistance is futile. You will be assimilated.
29
Example: High-Throughput Computing System

API SDK
App
High Throughput Computing System

job management,
C-point Protocol Checkpoint Repository
Collective Dynamic checkpoint, (App) failover, staging
Collective Brokering, certificate authorities (Generic)

API
Resource Access to data, access to computers, access to network performance data
SDK Access Protocol Compute Resource
Connect Communication, service discovery (DNS), authentication, authorization, delegation

Fabric Storage systems, schedulers
30
Example: Data Grid Architecture

App
Discipline-Specific Data Grid Application
Collective Coherency control, replica selection, task management, (App) virtual data catalog, virtual data code catalog,
Collective Replica catalog, replica management, co-allocation, (Generic) certificate authorities, metadata catalogs,
Resource Access to data, access to computers, access to network performance data,
Communication, service discovery (DNS), Connect authentication, authorization, delegation Fabric Storage systems, clusters, networks, network caches,
31
CERNs Data Grid Application
32
The Compact Muon Solenoid
Open Grid Service Architecture

Minds are like parachutes, they only function when they are open. Thomas Robert Dewar
34
Service-Oriented Architecture
Service: an entity that provides some capability to its clients by exchanging messages Service: defined by identifying sequences of specific message exchanges that cause the service to perform some operations
36
Service-oriented Architecture
Great flexibility in implementation and location: because all operations are defined in terms of message exchanges
SOAP: Simple Object Access Protocol
37
Service-oriented Architecture
Definition: a service-oriented architecture is one in which all entities are services, and thus any operation visible to the architecture is the result of message exchange
Message exchanges
Underlying infrastructure
services
38
Service-oriented architecture
Examples:
A storage service A data transfer service A troubleshooting service
low-level service
high-level service
Two important themes
39
But, first four personality types
40
Common behaviors can reoccur in different contexts

A goal of OGSA design is to allow these behaviors to be expressed in standard ways regardless of contexts, so as to simplify application design and encourage code reuse
41
42
A higher-level service behavior (data transfer) can be implemented via the composition of simpler behaviors (storage service)
Ease of composition is a second major design goal for OGSA
43
Similarity in protocols
44
By encapsulating service operations behind a common message-oriented service interface, service-oriented architecture encourages service virtualization, isolating users from details of service implantation and location
45
Virtualization
46
Virtualization
Everything is becoming virtual: virtual stores, virtual workspace, virtual organization, virtual networks, More than 60 million US workers work remotely
47
Interaction with a given service is facilitated by using a standard interface definition language (IDL), such as WSDL, to describe the services interfaces.
48
Web service description language
49
IDL: a cornerstone of interoperability and transparency
50
An IDL defines the operations supported by a service, by specifying the messages that a service consumes and produces An interface specification describes the messages the service expects but does not define what the service does in response to those messages (i.e., its behavior)
51
52
53
A well-defined interface definition language and a separation of concerns between service interface and implementation simplify the manipulation and management of services in four important respects: service discovery, composition, specialization, and interface extension.
54
Service discovery
55
Service discovery
Service Location Protocol (SLP) Jini: A service discovery architecture based on Java Universal Plug and Play (UPnP): Microsoft's solution to service discovery Salutation: A light weight network protocol independent service discovery protocol
56
Service composition
57
Service specialization: use of different implementations of a service interface on different platforms
58
Interface extension: allow for specialized implementations to add additional, implementationspecific functionality, while still supporting the common A value-added extension interface
59
Objectives of OGSA
Manage resources across distributed heterogeneous platforms Deliver seamless quality of service Provide a common base for autonomic management solutions Define open, published interfaces Exploit industry standard integration technologies
60
Three principal elements of OGSA: OGSI, OGSA services, OGSA applications

OGSA main architecture
61
Open Grid Service Infrastructure
OGSI: provides a uniform way for
software developers to model and interact with grid services by providing interfaces for discovery, life cycle, state management, creation and destruction, event notification, and reference management.
62
Requesting a service
63
Important OGSI concepts and interactions service factory

create service
resource allocation
grid service handle
service requester
service data, keep-alive, notifications, service invocation
service discovery
service instances
register service
service registry
64
OGSI and web services: OGSA architecture enhances Web services to accommodate requirements of the grid. These enhancements are specified in OGSI. Over time, it's expected that much of the OGSI functionality will be incorporated in Web services standards.
65
Implementation of OGSI: The Globus

Toolkit 3 (GT3) is the first full-scale implementation of the OGSI standard.
66
OGSA Architected Services
OGSA architected services
67
Grid core services
68
Grid program execution and data services
69
Grid program execution and data services hosting
70
Naming
Because Grid services are dynamic and stateful, we need a way to distinguish one dynamically created service instance from another. Thus, we need a naming scheme for Grid service instances.
71
72
Naming
73
Naming
74
Naming
75
Naming
76
Naming
77
Naming
OGSI defines a two-level naming scheme for Grid service instances based on simple, abstract, long-lived Grid service handles (GSH). GSH can be mapped by handle resolution services to concrete but potentially short-lived Grid service references (GSR).
78
Naming
A GSH is a globally unique name that distinguishes that specific Grid service instance from all other Grid service instances that have existed, exist now, or will exist in the future. A GSH is represented using a Uniform Resource Identifier.
79
Naming
A GSH carries no protocol- or instance-specific information such as network address or supported protocol bindings. All other instance-specific information is encapsulated into a single abstraction called a Grid service reference (GSR).
80
Naming
Resolve (GSH) client time<T GSR1 time>T GSR2 handle resolver
service instance
migrate at time T
service instance
81
Service lifetime management
The introduction of transient service instances raises the issue of determining the services lifetime, that is, determining when a service can or should be terminated so that associated resources can be recovered.
82
Happiness for a lifetime

If you want happiness for an hour -take a nap. If you want happiness for a day -- go fishing. If you want happiness for a month -- get married. If you want happiness for a year -inherit a fortune. If you want happiness for a lifetime -- help someone else. - Chinese Proverb
83
Service lifetime management
OGSA addresses this problem through a soft-state approach in which Grid service instances are created with a specified lifetime. Three steps:
Negotiating an initial lifetime Explicit terminating Requesting a lifetime modification
84
Soft-state used in RSVP
What is RSVP? (Resource reSerVation Protocol)
85
86
87

Path message
88
Merge the reservation
Resv message
89
OGSA Implementations
Container (good for code reuse) demarshaling/decoding protocol termination adaptation layer
Grid service implementation
Grid service invocation
protocol termination
protocol termination
90
91
92
93
Future Directions
Service and tools Implementation Semantics Scalability

Grid Architecture

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Grid Architecture

Uploaded by

Copyright:

Available Formats

Grid Architecture

Prof. Ruay-Shiung Chang March 2004

Why Discuss Architecture?

Water Garden in England

One View of Requirements

Another View: Three Obstacles to Making Grid Computing Routine

The Systems Problem: Resource Sharing Mechanisms That

1) Need for interoperability when different

Aspects of the Systems Problem

groups want to share resources

2) Need for shared infrastructure services

to avoid repeated development, installation

A common need for protocols & services

Hence, a Protocol-Oriented View of Grid Architecture, that Emphasizes

Development of Grid protocols & services

Development of Grid APIs & SDKs

The (hugely successful) model is the Internet

Layered Grid Architecture (By Analogy to Internet Architecture)

Protocols, Services, and APIs Occur at Each Level

Collective Service APIs and SDKs Collective Services

Collective Service Protocols

Resource APIs and SDKs Resource Services

Resource Service Protocols

Built on Internet protocols & services

The Hourglass Model

Focus on architecture issues

Applications Diverse global services

Where Are We With Architecture?

No official standards exist But:

Fabric Layer Protocols & Services

Connectivity Layer Protocols & Services

Internet protocols: IP, DNS, routing, etc.

Security: Grid Security Infrastructure (GSI)

Not too many connections!

Grid Resource Allocation Management (GRAM)

Resource Layer Protocols & Services

GridFTP protocol (FTP extensions)

Grid Resource Information Service (GRIS)

Index servers aka metadirectory services

Collective Layer Protocols & Services

Resource brokers (e.g., Condor Matchmaker)

Collectives: The Borgs

Resistance is futile. You will be assimilated.

Example: High-Throughput Computing System

High Throughput Computing System

C-point Protocol Checkpoint Repository

Collective Dynamic checkpoint, (App) failover, staging

Collective Brokering, certificate authorities (Generic)

Resource Access to data, access to computers, access to network performance data

SDK Access Protocol Compute Resource

Connect Communication, service discovery (DNS), authentication, authorization, delegation

Example: Data Grid Architecture

Discipline-Specific Data Grid Application

CERNs Data Grid Application

The Compact Muon Solenoid

Open Grid Service Architecture

Two important themes

But, first four personality types

Common behaviors can reoccur in different contexts

Web service description language

IDL: a cornerstone of interoperability and transparency

Service specialization: use of different implementations of a service interface on different platforms

Open Grid Service Architecture

Open Grid Service Architecture