From: Tatu Ylonen <ylo@cs.hut.fi>
Date: Thu, 23 Nov 95 06:44:09 PST
To: cypherpunks@toad.com
Subject: Intelligence Community Massive Digital Data Systems Initiative
Message-ID: <199511231246.NAA00407@trance.olari.clinet.fi>
MIME-Version: 1.0
Content-Type: text/plain


Below is some information about the Intelligence Community Massive
Digital Data Systems Initiative.

Summary:
 - new data 2 - 5 terabytes (10^12 bytes) per day
 - total size about 20 petabytes (20 * 10^15 bytes)
 - 300 terabytes on-line, the rest accessible in a few minutes
 - funding (for the research initiative, not for the final system): 
   3-5 million USD per year estimated for investments

Now, how much is 2 - 5 terabytes per day?
  - 20 - 50.000.000 jpeg images (100kB/image, relatively high-quality) per day
  - 20 - 50.000.000 minutes of GSM-quality phone intercepts per day
  - 1.000.000 - 2.500.000 minutes of compressed (256kbit/sec) video per day
  - 1.000.000.000 - 3.000.000.000 e-mail messages per day
  - you can continue the list; most available data sets turn out to be
    much smaller

How much is 20 petabytes?  Assuming you want to collect
information about 100.000.000 people worldwide, this makes 200
megabytes per person (on the average for each of those 100 million
people).

200 megabytes per person on the average is quite a lot, since for many
of those people you probably don't have all that much data.  Maybe 90%
of the data for 10% of the people?  

(Of course, in a database like this you might also have a lot of data
like aerial imaginery, satellite imaginery, economical information,
etc., so it is a little exaggarated to talk about all of it being on
individual people.)

The full text is below.

Crypto relevance?  Makes you think whether you should protect your data.

    Tatu

From: dbowner@cs.wisc.edu ( Dbowner)
To: bal@mitre.org, mike@nobozo.CS.Berkeley.EDU, shosani@csr.lbl.gov,
        gray@sfbay.enet.dec.com, livny@cs.wisc.edu, ragrawal@almaden.ibm.com,
        manola@gte.com, heiler@gte.com, dayal@hplabs.hpl.hp.com,
        shan@hplabs.hpl.hp.com, toby@almaden.ibm.com, reiner@ksr.com,
        jag@allegra.att.com, randy@allspice.berkeley.edu, mcleod@vaxa.isi.edu,
        nick@MIMSY.CS.UMD.EDU, ake@purdue.edu, laney@ccr-p.ida.org,
        darema@watson.ibm.com, grossman@math.uic.edu, dbusa@cs.wisc.edu,
        metadata@llnl.gov, jmaitan@mosaic.uncc.edu, whm@thumper.bellcore.com
Cc: susan@mitre.org, connie@mitre.org
Subject: Call For Papers MDDS
Date: Thu, 18 Nov 93 11:08:03 EST
Resent-To: dbworld-people@cs.wisc.edu
Comments: IF YOU REPLY TO THIS MESSAGE, BE SURE TO EDIT THE to: AND cc: LISTS.
 The dbworld alias reaches many people, and should only be used for
 messages of general interest to the database community.  Mail sent
 to dbedu goes to the subset of addresses with a .edu suffix; mail
 sent to dbusa goes to the subset of US addresses.  Please use
 the smaller lists when appropriate.  Requests to
 get on or off dbworld should go to dbworld-request@cs.wisc.edu.
Reply-To: (Susan L. Hanlon) <susan@linus.mitre.org>
Resent-Reply-To: (Susan L. Hanlon) <susan@linus.mitre.org>



									3 November 1993

Dear Colleague:

Subject:  Call for Abstracts for Massive Digital Data Systems



	Future intelligence systems must effectively manage massive amounts of digital
data (i.e., multi-terabytes or greater).  Issues such as scalability, design,
and integration need to be addressed to realize a wide spectrum of intelligence
systems ranging from centralized terabyte and petabyte systems comprised of many
large objects (e.g., images) to distributed heterogeneous databases that contain
many small and large objects (e.g., text).  The Community Management Staff's
Massive Digital Data Systems (MDDS) Working Group on behalf of the intelligence
community, is sponsoring a two day invitation-only unclassified workshop on the
data management of massive digital data systems with government, industry, and
academia.

	The workshop will be held on the 1st and 2nd of February 1994 in Reston,
Virginia.  The objective of the workshop is to make industry and academia aware
of intelligence community needs, stimulate discussion of the technical issues
and possible solutions, and identify potential research efforts that warrant
further investigation for possible government funding.  The amount of funding
estimated for investments is three to five million dollars per year over the
next 2-3 years.

	Last July, a one-day, classified, government-only workshop was held to
characterize the magnitude of the problem and identify the major challenges. 
The needs, issues, and in some cases, lessons learned, were presented for
different data types including Imagery, Text, Voice, Video, and Multi-media. 
Enclosure 1, "Massive Digital Data System Issues", is an unclassified
description of the consolidated challenges.

	The Massive Digital Data Systems Working Group is soliciting one-page abstracts
related to the issues of the data management of massive digital systems
including (but not limited to) scalability, architecture and data models, and
database management functions.  The focus of the abstract should be on potential
solutions for the longer term research challenges (i.e., 5-10 years out) that
must be addressed today in order to effectively manage data of massive
proportions in the future.  The solutions need not be limited to proven
approaches today but can foster new approaches and paradigms.  Issues relating
to the storage media and analysis tools, while important to the intelligence
community, are not within the scope of the workshop.  Selection for attendance
will be based upon technical relevance, clarity, and quality of the proposed
solution.

Call for Abstracts								Page 2



	Each one-page abstract should follow the abstract format enclosed (Enclosure
2).  All submissions must be UNCLASSIFIED.  To allow enough time for proper
evaluation of each abstract, the deadline for submission is 01 December 1993. 
You will be notified of acceptance to attend by 17 December 1993.  Abstracts
should be forwarded to one of the following:

	Jackie Booth, P.O. Box 9146, Rosslyn Station, Arlington, VA   22219
	Jackie Booth, ORD/SETA, fax number (703) 351-2629
	boothj@mcl.saic.com (Internet)

	Please pass this call for abstracts on to other colleagues that are working on
solutions in this area.

	Sincerely,



	Dr. David Charvonia
	Director, Advanced Technology Office
	Community Management Staff

Enclosures:
  1.  Massive Digital Data Systems Issues
  2.  Abstract Format
	
Enclosure 2
ABSTRACT FORMAT

Title:


Author(s):

Organization/Affiliation:

Address:

Phone:					FAX:


Description:


















Status:  (Research, Prototype, Operational)


Scope:  (Size of effort in terms of dollars and/or staff months;  Size of system
in terms of amount of data, number of databases, nodes, users, etc.)


Customer:  (if applicable)


Operational Use:  (if applicable)


********************************************************************
Forward to one of the following:
	Jackie Booth, P.O. Box 9146, Rosslyn Station, Arlington, VA   22219
	Jackie Booth, ORD/SETA, fax number (703) 351-2629
	boothj@mcl.saic.com (Internet)

-------------------------------------------------------------------
MASSIVE DIGITAL DATA SYSTEMS ISSUES


EXECUTIVE SUMMARY

Future intelligence systems must effectively manage massive amounts of digital
data (i.e., multi-terabytes or greater). Issues such as scalability, design, and
integration need to be addressed to realize a wide spectrum of intelligence
systems ranging from centralized terabyte and petabyte systems comprising many
large objects (e.g. images) to distributed heterogeneous databases that contain
many small and large objects (e.g. text). Consequently, Massive Digital Data
Systems (MDDS) are needed to store, retrieve, and manage this data for the
intelligence community (IC). While several advances have been made in database
management technology, the complexity and the size of the database as well as
the unique needs of the IC require the development of novel approaches. This
paper identifies a set of data management issues for MDDS. In particular,
discussions of the scalability issues, architectural and data modeling issues,
and functional issues are given. The architectures for MDDS could be
centralized, distributed, parallel, or federated. The functions of MDDS include
query processing, browsing, transaction management, metadata management,
multimedia data processing, integrity maintenance, and realtime data processing.
Representing complex data structures, developing appropriate architectures,
indexing multimedia data, optimizing queries, maintaining caches, minimizing
secondary storage access and communications costs, enforcing integrity
constraints, meeting realtime constraints, enforcing concurrency control,
recovery, and backup mechanisms, and integrating heterogeneous schemas, are some
of the complex tasks for massive database management. The issues identified in
this paper will provide the basis for stimulating efforts in massive database
management for the IC.

1.0  INTRODUCTION	
1.1  The Challenge
The IC is challenged to store, retrieve, and manage massive amounts of digital
information. Massive Digital Data Systems (MDDS), which range from centralized
terabyte and petabyte systems containing many large objects (e.g., images) to
distributed heterogeneous databases that contain many small and large objects
(e.g., open source), are needed to manage this information. Although
technologies for storage, processing, and transmission are rapidly advancing to
support centralized and distributed database applications, more research is
still needed to handle massive databases efficiently. This paper describes
issues on data management for MDDS including scalability, architecture, data
models, and database management functions. Issues related to storage media,
analysis tools, and security while important to the IC are not within the scope
of this paper. 

The key set of data management issues for MDDS include:  

	Developing architectures for managing massive databases
	Utilizing data models for representing the complex data structures
	Formulating and optimizing queries
	Developing techniques for concurrency control and recovery
	Integrating heterogeneous schemas
	Meeting timing constraints for queries and transactions
	Indexing multimedia data
	Maintaining caches and minimizing secondary storage access and communications
costs
	Enforcing integrity constraints.

1.2  Background
The IC provides analysis on current intelligence priorities for policy makers
based upon new and historical data collected from intelligence sources and open
sources (e.g., news wire services, magazines). Not only are activities becoming
more complex, but changing demands require that the IC process different types
as well as larger volumes of data. Factors contributing to the increase in
volume include continuing improvements in collection capabilities, more
worldwide information, and open sources. At the same time, the IC is faced with
decreasing resources, less time to respond, shifting priorities, and wider
variety of interests. Consequently, the IC is taking a proactive role in
stimulating research in the efficient management of massive databases and
ensuring that IC requirements can be incorporated or adapted into commercial
products. Because the challenges are not unique to any one agency, the Community
Management Staff (CMS) has commissioned a Massive Digital Data Systems Working
Group to address the needs and to identify and evaluate possible solutions.

1.3  Assumptions and Project Requirements
Future intelligence systems must provide a full suite of services for gathering,
storing, processing, integrating, retrieving, distributing, manipulating,
sharing and presenting intelligence data. The information to be shared is
massive including multimedia data such as documents, graphics, video, and audio.
  It is desired that the systems be adapted to handle new data types.  

The goal is to be able to retain the data for potential future analysis in a
cost effective manner. The more relevant data would remain on-line, say for 5
years, organized with the most relevant data accessible in the least amount of
time. It is expected that 2 to 5 terabytes of new data has to be processed each
day.  Thus, the total size of the database (both on-line and off-line) could be
as large as 20 petabyes with about 300 terabytes of data stored on-line. It is
assumed that storage devices (primary, secondary, and even tertiary) for the
large multimedia databases as well as data pathways with the required capacity
will exist. The access times are about 5 seconds for the data less than a week
old, about 30 seconds for data under two months old, and on the order of minutes
for data up to 10 years old. 

2.0  SCALABILITY ISSUES
A particular data management approach can be scaled to manage larger and larger
databases. That is, a database can often sustain a certain amount of growth
before it becomes too large for a particular approach. For example, more memory,
storage, and processors could be added, a new hardware platform or an operating
system could be adopted, or a different microprocessor could be used (e.g. using
a 32 bit microprocessor instead of a 16 bit microprocessor). Once the size of
the database has achieved its limit with a particular approach, then a new
approach is required. This new approach could be a new architecture, a new data
model, or new algorithms to implement one or more of the functions of the
database management system (DBMS), or a combination of these features.
Discussions of these three features are given below. 

 	Architectures:  The type of architecture impacts the size and response time
of the DBMS.  
	Centralized approaches are being migrated to distributed and parallel
approaches to handle large databases.  Some architectures such as a the shared
nothing parallel architectures are scalable to thousands of processors, but will
have multiprocessor communication issues.  Current approaches need to be
assessed to determine their scalability limits.  New approaches may be required
for handling massive databases.


 	Data Models:  Data models which support a rich set of constructs are desired
for next generation database applications.  However, the search and access time
of the DBMS would depend on the data model used. For example DBMSs which support
complex data structures use large caches, access data through pointers, and work
well with large main memory in general, while DBMSs based on simpler data models
maintain index files and provide associative access to the secondary storage.
The limits of these models within the context of massive databases need to be
understood.  New or modified approaches may be required.

	DBMS Functions:  The techniques to implement the DBMS functions have to be
modified to handle massive databases.  For example, as the size of the database
increases, new approaches for  query optimization, concurrency control,
recovery, and backup, access methods and indexing, and metadata management will
be required.  

The architectural, data modeling, and functional issues that need to be
addressed for MDDS will be elaborated in sections 3 and 4. 

3.0  ARCHITECTURAL AND DATA MODELING ISSUES
3.1  Architectural Issues
This section describes some of the architectural issues that need to be
addressed for an MDDS. In the case of the centralized approach, a major issue is
managing the data transfer between the main memory and secondary storage. One
could expect the data that is a week old to be cached in main memory, the data
that is less than two months old to be in secondary memory, and data that is a
few years old to be in tertiary storage. In designing the data management
techniques (such as those for querying, updating, and transaction processing),
data transfer between the main and secondary memories needs to be minimized.
There is also a need to reflect patterns of use (e.g., in migrating items to
lower/higher levels of storage hierarchy). Another issue is the relationship
between the size of the cache and the size of the database. 

When one migrates to distributed and parallel architectures, a goal is to
maintain a larger number of smaller databases. It is assumed that processors and
storage devices are available. A  major issue is the communication between the
processors. In designing the data management mechanisms, an objective would be
to minimize the communication between the different processors. For example, in
the case of a join operation between several relations in a relational DBMS,
each fragmented across multiple sites, an issue is whether to merge all of the
fragments of a relation and then perform the join operation or whether to do
several join operations between the fragments and then merge the results to form
the final result. Different configurations of the distributed and parallel
architectures also need to be examined. For example, there could be
point-to-point communication between every processor, or the processors could be
arranged in clusters and communication between clusters is carried out by
designated processors. Another issue in migrating to a distributed architecture
is handling data distribution. For example, if the data model is relational,
then how could one fragment the various relations across the different sites? 
If the relations are to be replicated for availability, then how could
consistency of the replicated copies be maintained?  Another issue is what data
could be cached within the distributed system, how could data be cached, and for
how long could the cache be maintained. 
   
While distributed and parallel architectures are being investigated for managing
massive databases, federated architectures are needed to integrate the existing
different and disparate databases.  The existing databases could be massive
centralized databases or they could be distributed databases. Furthermore, they
could be relational, object-oriented and even legacy systems.  An issue in
heterogeneous database integration is developing standard uniform interfaces
which can be accessed via an integration backplane.  If the environment is a
federated one, where the nodes have some autonomy, then a major issue is the
ability to share each other's data while maintaining the autonomy of the
individual DBMSs. This is hard because cooperation and autonomy are conflicting
goals.  The techniques to implement the DBMS functions for data retrieval,
updates, and maintaining integrity have to be adapted or new approaches have to
be developed for federated architectures.  

Extensible architectures are also being investigated for massive databases. 
With such architectures, DBMSs are extended with inferencing modules which make
deductions from data already in the database.  This way, one need not store all
of the data in the database explicitly. Instead, appropriate inference rules are
used to make deductions and derive new data.  This way the size of the database
is reduced.  The issues include determining what data is to be stored in the
database and what data is to be stored in the knowledge base manipulated by the
inferencing module, effective management of the knowledge base, and adapting the
functions of the DBMS to handle extensible architectures.  
   
3.2  Data Modeling Issues

In selecting an appropriate data model for massive databases, several issues
must be considered. Providing a data model powerful enough to support the
representation of complex data must be addressed.  For example, with a
multimedia document, one may need to devise a scheme to represent the entire
document in such a way to facilitate browsing and updating. Since the age of a
document could be used to move it between different storage media, it is
desirable for the data model to support the representation of temporal
constructs. The representation of different types of multimedia devices and
grouping of documents are also important considerations in selecting a data
model. The data model chosen has an impact on the techniques to implement the
functions of a DBMS. For example, DBMSs based on some models use associative
access while those based on some other models use pointer traversal. 

In migrating to a distributed/parallel architecture, if it is assumed that the
data model is the same for all databases, then a major issue is whether it is
feasible to provide a conceptual view of the entire massive database to the
user. However, in the case of a federated architecture, since it is generally
assumed that the individual data models are different, several additional issues
need to be considered. For example, could the users have a global view of the
massive database or could they have their own individual views?  In either case,
it would be desirable for the users to access the distributed databases in a
transparent manner. If a global view is enforced, the query processor could
transform the queries on the global view to the views of the individual
databases. If each user has his own view, then the query processor could
transform the users view into the views of the individual databases. Other
issues for a federated architecture include the representation of the individual
schemas (which describe the data in the databases), determining which schemas to
be exported to the federation, filtering appropriate information from the
schemas at different echelons, integrating the schemas to provide a global view,
and generating the external schemas for the users. In integrating the different
schemas, the semantic and syntactic inconsistencies between the different
representations need to be resolved. For example, the address in database A
could include the house number and the street name while in database B it could
just be the city and the state. 

4.0  FUNCTIONAL ISSUES   
The techniques to implement the functions of MDDS will be impacted by the
architectures and data models as well as requirements such as integrity and
multimedia data processing. Therefore some of the functional issues have already
been addressed in section 3. This section provides a more detailed overview of
the functional issues.  First the basic functional issues for MDDS (such as
issues on query processing and transaction management) will be discussed and
then the impact of maintaining integrity, realtime processing, and multimedia
data processing will be given. 

4.1 Querying, Browsing, and Filtering
The query operation is a means by which users can retrieve data from the
database. Closely related is the browsing operation where users traverse various
links and subsequently scan multiple documents either sequentially or
concurrently. To determine if the new information warrants viewing by the
analyst and/or to enforce access control, automatic filtering of the data is
needed. Some issues in query management for massive databases are using an
appropriate language for specifying queries and developing optimization
techniques for the various operations involved in a query. The goals are to make
it easier for users to formulate queries and also to minimize data transfer
between primary and secondary storage. 

Query management in a federated environment must provide the means for
formulating and processing queries seamlessly and efficiently. This involves
designing an interface for formulating queries over multiple sources. There is a
need for query optimization, in order to prevent degradation in performance in
the distributed system. In addition to determining the execution strategy for a
query, query optimization techniques could also determine which portion of the
query processing is to remain under direct and unshared control at the analyst's
workstation. Methods need to be developed for browsing the integrated
information space and for displaying results obtained from multiple sources.
Finally, data from local databases have to be filtered according to the various
constraints (such as security constraints) and enforced before sending it to the
remote sites. 

Query processing algorithms in an extensible architecture need to incorporate
inferencing techniques.  The usefulness of inferencing techniques for
intelligence applications can best be illustrated with a simple example. Suppose
parts A, B, C and D are needed to build a nuclear weapon, and also suppose that
the following constraint is enforced:  " if three of the four parts are shipped
to country X, then the fourth part should not be shipped to X."   Therefore, if
parts A, B, and C are already shipped to X and there is a request from X for
part D, then the inferencing module will determine that this part cannot be
shipped.  An issue in developing an inference module is determining the
deduction strategies to be implemented. These strategies could be just logical
deduction or could include more sophisticated techniques such as reasoning under
uncertainty and inductive inference. With most inference strategies one runs
into the problem of an infinite loop; therefore appropriate time limits must be
enforced to control the computation.

In general, the issues to be addressed in query management will include:

	Query optimization.
	Handling data distribution
	Making intelligent deductions 
	Uniform vs. user-tailored query language 

4.2  Update Transaction Processing
Multi-user updates are supported in general to improve performance. The goal is
for multiple users to be able to update the database concurrently. A major issue
here is ensuring that the consistency of the database is maintained. The
techniques that ensure consistency are concurrency control techniques. Often
update requests are issued as part of transactions. A transaction is a program
unit that must be executed in its entirety or not executed at all. Therefore, if
the transaction aborts due to some error, such as system failure, then the
database is recovered to a consistent state.

Several concurrency control algorithms have been designed and developed for
different environments. Some algorithms are suitable for short transactions in
business processing applications and some others are suitable for long
transactions which often involve multimedia data. To handle long transactions
efficiently, weaker forms of consistency conditions have been formulated.
Several recovery techniques have also been developed to maintain the consistency
of the database. If the transaction is long, then the log files that record the
actions of the transaction may be quite large. Efficient management of log files
becomes an issue. As the size of the database increases, a transaction would
take a longer time for execution. Adapting the concurrency control and recovery
algorithms or developing new algorithms to work with the massive databases
becomes an issue.  

Update transaction processing gets more complicated in distributed and federated
environments. For example, if replicated copies are to be maintained, then
making them consistent will have an impact on the performance. Therefore, an
issue here is whether to maintain strict consistency or select a subset of the
copies and make them consistent immediately so that the remaining copies could
be updated at a later time. One of the problems with a federated environment is
the different concurrency control and recovery algorithms used by the individual
DBMSs. In such a situation synchronizing the different techniques becomes a
major issue. 

4.3  Access Methods and Index Strategies
To enhance the performance of query and update algorithms, efficient access
methods and index strategies have to be enforced. That is, in generating
strategies for executing query and update requests, the access methods and index
strategies that are used need to be taken into consideration. The access methods
used to access the database would depend on the indexing methods. Therefore
creating and maintaining appropriate index files is a major issue in a DBMS.
Usually, the size of the index file grows with the size of the database. In some
cases, the index file could be larger than the database itself. Some of the
issues include determining what type of indexes are to be maintained for massive
databases.  Is it feasible to have dense indexing where there is an entry in the
index file for every entry in the database?  If so, the index file could have as
many entries as there are in the database. Is it better to have sparse indexing
so that the size of the index file could be reduced?  If so, is there a strategy
to determine which entries in the database are to be indexed?  For multimedia
data, indexing could be done not only by content but by type, language, context
(i.e., where, how, when it was collected), author (i.e., for documents), and
speaker (i.e., for voice). The challenge is how to index and to provide improved
mechanisms for extraction of the information used for indexing.   For example,
the ability to automatically index voice is desired.  Additionally, the ability
to index voice and video (with associated voice) with their transcriptions
(i.e., time alignment) is necessary.

Various storage structures have been proposed. These include B-Trees and
Parent-Child links. The question is, are these methods suitable for massive
databases? Voice and video data require segmentation into logical units for
storage and access. Additionally, the ability for automatic segmentation within
documents of embedded drawings and figures and their interpretation (via
seamless integration with image handling tools) is needed. Other challenges
include providing user transparent hierarchical storage management (i.e., store
the most relevant or most recent information on the fastest media) and the
ability to reposition data in the storage hierarchy based upon changing
importance, migration mechanisms for transferring information to newer storage
media or a new architecture (failure to do so can lead to exorbitant costs to
maintain discontinued storage media drives or inaccessible data), archival
technology/policies for older/less important information, and synchronization of
information distributed across multiple repositories

Compression can decrease the costs of storage and transmission especially for
the larger objects such as vector and raster spatial data types, voice, imagery,
and video. Real-time conversion of heterogeneous voice and video compression and
file formats in network broadcasts/multicasts is an issue.  For imagery, a
capability such as pyramidal decomposition for providing reduced resolution
images is needed for browsing purposes.

4.4  Managing the Metadata
The metadata includes a description of the data in the database (also referred
to as the schemas), the index strategies and access methods used, the integrity
mechanisms enforced, and other information for administrative purposes. 
Metadata management functions include representing, querying, and updating, the
metadata.  In massive databases, if the metadatabase is much smaller than the
database, then the traditional techniques could be applied to manage the
metadata. If the metadatabase becomes massive, then new techniques need to be
developed. An issue here is whether the techniques for massive databases could
be applied for massive metadatabases also. Support for schema evolution is
desired in many new generation applications.  For example, the structures of the
entities in the database could change with time.  An entity could acquire new
attributes or existing attributes could be deleted. The metadata needs to be
represented in a manner that would facilitate schema evolution. That is,
appropriate models to represent the metadata are desired. Since the metadata has
to be accessed for all of the functions of a DBMS, the module that is
responsible for accessing the metadata needs to communicate with all the other
modules. Efficient implementation of this module is necessary to avoid
performance bottlenecks. 

Certain types of metadata, such as the schemas, are usually accessible to the
external users.  An issue here is whether to provide a view to the users that is
different from the system's view of the metadata.  For example, a different
representation of the metadata could be sued for the users.  Also, if the
metadatabase is massive, then subsets of it could be presented to the users.    

4.5  Integrity
Concurrency control and recovery issues discussed in section 4.3 are some of the
issues that need to be dealt with in order to maintain the integrity  (i.e.
consistency) of the database. Other types of integrity include maintaining the
referential integrity of entities and enforcing application dependent integrity
constraints. Referential integrity mechanisms must ensure that the entities
referenced exist. The question is, how could the references to an entity be
deleted when an entity itself is deleted?  If the databases are massive, then
there will probably be more references to the deleted entity. Deleting all these
references in a timely manner is an issue.

In the case of application specific integrity constraints, they could trigger a
series of updates when one or more items in the database gets updated. Again, as
the size of the database increases, the number of updates that are triggered
could also increase. The issue here is ensuring that the updates are carried out
in a timely manner. 	

4.6  Realtime / Near Realtime Processing
Within a massive digital data system, the challenges of realtime or near
realtime processing will be compounded.  For realtime or near realtime
applications, timing constraints may be enforced on the transactions and/or the
queries. In the case of a hard realtime environment, meeting the timing
constraints may cause the integrity of the data to suffer. In the case of soft
realtime constraints (also referred to as near real-time), there is greater
flexibility in meeting the deadlines. The issues for real-time processing
include:

 	If a transaction misses its deadline, then what are the actions that could be
taken?   
	Could a value function be associated with a transaction which can be used to
determine whether the transaction should continue after it misses its deadline?
	Could the transaction be aborted if the value of the data approaches zero?  
	What is the impact on the scheduling algorithms when timing constraints are
present?  
	How can the techniques be extended for a distributed/federated architecture? 
	In the case of realtime updates in a distributed replicated environment, is it
possible to maintain the consistency of the replicated copies and still meet the
timing constraints?  
	What is the impact on the techniques for multimedia data processing?
4.7  Multimedia Data Processing
By nature, multimedia data management has to deal with many of the requirements
for indexing, browsing, retrieving, and updating of the individual media types. 
Implementing multimedia data types will require new paradigms for representing,
storing, processing, accessing, manipulating, visualizing, and displaying data
from various sources in different media. One of the major issues here is
synchronizing the display of different media types such as voice and video. 
Other issues include selecting/developing appropriate data models for
representing the multimedia data and developing appropriate indexing techniques
such as maintaining indexes on textual, voice, and video patterns.  For example,
the ability to index voice and video simultaneously may be desired.  

In addition to the manipulation of multimedia data, frameworks for the
integration of multimedia objects as well as handling different granularity of
multimedia objects (i.e., 1 hour video clip versus a spreadsheet cell) need to
be considered. A flexible environment has to be provided so that the linked and
embedded distributed multimedia objects can accommodate geographic/network
changes.  Finally, the data manipulation techniques as well as the frameworks
need to be extensible to support new and diverse data types.  

4.8  Backup and Recovery
On-line backup procedures are being used for massive databases. This is because
off-line procedures will consume too much time for massive databases. Even if
the backup procedures are carried out on-line, the system could be slowed down
and therefore the performance of other data management functions would suffer.
The issue here is to develop improved techniques for backup so that it will not
impact functions such as querying, browsing, and updating. 

Recovery issues for transaction management were discussed in section 4.3. Other
recovery issues include whether to maintain multiple copies of the database, and
if so, the number of copies to be maintained, and whether the checkpointing,
roll-back and recovery procedures proposed for traditional databases could be
used for massive databases or is there a need to develop special mechanisms?

5.0  SUMMARY

Massive digital data systems will require effective management, retrieval, and
integration of databases which are possibly heterogeneous in nature. Achieving
this concept of massive intelligence information systems will require new
technologies and novel approaches for data management. While hardware is rapidly
advancing to provide massive data storage, processing, and transmission, the
software necessary for the retrieval, integration, and management of data
remains an enormous challenge.

This paper has identified a set of issues for managing the data in massive
digital data systems with a focus on intelligence applications. First, an
overview of the current approaches to data management and the scalability of the
current approaches were discussed  Then some architectural and data modeling
issues were given. Finally, a discussion of the issues for the various functions
of MDDS were given. The set of issues identified is by no means considered a
complete list. As the progression of research, prototyping, and deployments
continue, new or hidden challenges will arise.